Failure to Disclose Sample Size: A Flaw at Multiple Levels

Yesterday, I revealed that a new study published in the journal Tobacco Control which concluded that e-cigarettes are a "one-way bridge" to youth smoking failed to disclose the sample size upon which its major conclusion was based. It turns out that the paper's sweeping conclusion was based on only 4 kids who had tried an e-cigarette at baseline and went on to try a cigarette in the next year. That sample size was not even revealed in the online supplement to the article. I had to calculate it from the raw data. The supplement did reveal that there were only 13 nonsmokers in the study who had vaped in the past month. Thus, I argued that the paper really should have concluded not that vaping is a gateway to smoking but that e-cigarette experimentation among nonsmokers doesn't appear to even be a gateway to regular vaping.

A further analysis of the paper reveals that among the 4 youth who had supposedly progressed to smoking, none of them had smoked more than two cigarettes in the past year. According to the paper: "Among the group of new smokers at follow-up who had recently vaped at baseline, all reported that they had smoked cigarettes at the level of ‘once or twice’ in the past 12 months at follow-up."

What the paper doesn't reveal is that this "group" of new smokers consists of only about 4 kids.

This raises the question of why the paper doesn't share this information with the reader. To talk about a "group" of new smokers who had vaped at baseline, but without informing the reader that there are only 4 kids in this "group" seems misleading.

The Rest of the Story

It appears to me that there were three levels of failure which explain why this paper violated what is perhaps the most important aspect of scientific reporting: revealing the sample size upon which your major conclusion is based.

First, the paper itself should have disclosed this sample size. It seems critical for readers to understand that the paper's major conclusion - that there was a 4.8 times higher rate of "smoking initiation" among nonsmokers who had vaped in the past month - was based on 4 kids having tried a cigarette or two in the past year.

This failure has already led to deceptive headlines, such as this one in the Daily Mail: "E-cigarettes are a One-Way Bridge to Tobacco." A more accurate headline would have said something like: "Researchers Can't Find More than 4 Kids Who Progressed from Vaping to Smoking, and Even Those Four Had Only Smoked Once or Twice in their Lifetimes." Or: "E-cigarette Experimentation among Nonsmokers Found Not to Be a One-Way Bridge to Vaping, Much Less Smoking."

The second failure is upon the peer reviewers of the manuscript. How could the reviewers possibly have not caught the glaring omission from the paper of the sample size underlying its most important conclusion?

The third failure is that of the journal itself. It, too, should have caught the glaring omission from the paper of the sample size underlying its most important conclusion. I used to be a statistical/methodological editor for Tobacco Control and I can tell you that I never would have let a paper through that did not reveal the sample size upon which its major conclusion was drawn. It's difficult for me to understand how this occurred.

The rest of the story is that it is difficult not to agree with Professor Robert West's conclusion that e-cigarette researchers in the United States are "waging a ‘moral crusade’ against e-cigarettes" and that they are "exaggerating their findings." This may not be a conscious decision, but may reflect a deeper underlying bias against e-cigarettes.