Some of the researchers and advocates who opposed our crowdfunding campaign to raise money for a randomized behavioral study of the effects of electronic cigarettes on smoking behavior argued that randomized clinical trials (RCTs) are simply not appropriate to study e-cigarettes because they cannot simulate the real-life situation, where smokers have many choices of different types of products, can engage with social networks, and can experiment over time, change products, advance from one type of product to another, etc.
Instead, these advocates argued that surveys are the best way to study the potential benefits of e-cigarettes. Surveys measure the real-life situation of how e-cigarettes and vapor products are actually used.
According to the argument, surveys produce valid results, while RCTs produce invalid results.
The Rest of the Story
Unfortunately, it's just not that simple.
The above argument is convenient for advocates who want to suppress "negative" or "unfavorable" findings by discouraging RCTs - which they believe will "underestimate" the effectiveness of e-cigarettes for smoking cessation and encouraging survey studies - which they believe will show the effects of vapor products in all their possible glory. However, the argument carries with it a lack of scientific validity.
The truth is that there are major advantages and disadvantages of both surveys and clinical trials to examine the potential benefits of electronic cigarettes. Both types of study designs have inherent biases which can produce results that are more "favorable" or "unfavorable." This is why the methodology of published research needs to be examined so carefully. It is not so simple as to dismiss RCT evidence and accept survey evidence. One needs to consider the strengths and limitations inherent in both approaches. And at the end of the day, the only defensible approach, I believe, is that both types of studies are necessary, and we must carefully interpret and synthesize the findings from studies of multiple designs, using the totality of the evidence to make final judgments about the benefits of electronic cigarettes.
This is why I find it so troubling that some major voices in the e-cigarette community are arguing that RCTs should not be conducted and only survey studies are of value.
The randomized study provides a number of important benefits that can never be realized in a survey study. Most importantly, the RCT can equalize between study groups the known and unknown confounding variables that may lead to invalid study results.
For example, one of the major problems in some of the previous survey studies of electronic cigarettes (the ones cited by Stan Glantz, for example) is that they fail to control for a major confounder: the level of a smoker's addiction to smoking and motivation to quit. Some of the studies that Glantz has cited to show that e-cigarettes are ineffective for smoking cessation compare users of e-cigarettes and non-users of e-cigarettes. In two of these studies, those who did not use e-cigarettes had higher quit rates.
The problem is that the smokers who used e-cigarettes were likely more heavily addicted and more resistant to quitting. The reason they tried e-cigarettes was probably related to their failure to quit smoking using other types of therapy, such as NRT or other drugs. E-cigarettes, after all, are not generally the first smoking cessation approach that a smoker will try.
The impact of this confounding is that it will bias the results towards finding that e-cigarettes are much less effective than other approaches. In fact, this is exactly what happened and it is the reason why the evidence that Glantz and others are citing to support their contention that e-cigarettes hinder smoking cessation is invalid.
The beauty of a randomized study is that it can equalize the levels of smoking addiction, reasons for wanting to quit, and motivation to quit across groups. In addition, it equalizes other confounders, including unknown confounders which could affect the study results. There is no way for a survey study to accomplish this. Thus, to simply throw out the RCT is quite unscientific, in my opinion. It throws out one of the most valid pieces of evidence that is necessary to make an informed judgment about the effect of these products: the differences in effectiveness of the products under conditions in which confounding cannot throw off the results.
A second major advantage of a clinical trial is that it can examine the potential effectiveness of interventions in which the use of a product is promoted for use among smokers who are interested in quitting. A survey cannot do this, because it can only examine the use of products under current conditions. It provides no information on what would happen if the product was actively promoted to a group of smokers, as it is in a clinical trial.
A third major advantage of a clinical trial is that conditions are controlled as carefully as possible, minimizing potential biases. Both sampling and measurement bias are greatly reduced, if not eliminated. In contrast, survey studies are generally subject to significant sampling and measurement bias.
There are some specific problems with survey studies that could lead to an overestimation of the benefits of electronic cigarettes. One major problem is that a survey study of the use of advanced vapor products would result in a severe bias towards finding a high level of effectiveness of these products. The reason? By the time smokers advance to use products like open-ended systems, they have already experienced success, fulfillment, and enjoyment with vaping. In other words, limiting the sample to users of advanced vapor products filters out the majority of e-cigarette users, who do not experience great success and therefore don't go on to the more advanced products.
Although the survey approach described above would lead to "invalid" results, I would never argue that survey studies should be thrown out and that we should only be conducting RCTs. My point is that there are strengths and limitations of various study designs, and causal inference is enhanced by having research evidence from multiple study designs. On balance, the strengths and weaknesses balance each other out, leading to a much greater ability to draw valid conclusions.
Neither the survey approach or the RCT approach is perfect. Both have major strengths and significant limitations. The best way to proceed is to conduct both types of studies and to synthesize the results across study designs, keeping in mind the methodological strengths and limitations of each individual study. Only by looking at this totality of the evidence can we draw valid scientific conclusions.
And this is why when some responded to our proposal for a randomized behavioral study by arguing that such an approach was invalid and that we should do a survey instead, I viewed those responses as being unscientific and unsound. Instead, I believe what is truly behind these draconian opinions (draconian because they would throw out an entire line of potential evidence) is a bias towards electronic cigarettes. I'm not arguing that it is a conscious bias. It may be subconscious. But I don't believe that any objective scientist would argue for completely throwing out a randomized clinical study design and relying solely on survey evidence to draw conclusions about the effectiveness of a product such as electronic cigarettes.
Finally, while it is true that the typical RCT is limited because it does not simulate the real-life situation where smokers can choose between different products, change products over time, and engage in social networks to support their vaping, the study we had proposed would have allowed for all of these things. We would have given smokers a wide range of products to choose from and allowed them to experiment with different products and make changes if they desired, and we would have referred them to a number of vaping web sites and support groups. In fact, we had decided to include a training and information session with an actual vaper as part of the study. This would still not have simulated the real-life situation exactly, but it would have significantly helped to make the study more meaningful by making it more similar to what happens in "real life."
I believe that such a modified, randomized clinical study is essential to add another line of evidence with which to evaluate the potential benefits of electronic cigarettes. Yes, there are limitations to RCTs, but it makes no sense to throw out the baby with the bath water.
Disclosure: I have not received any funding or compensation from
the tobacco, electronic cigarette, or pharmaceutical industries.
However, I am seeking funding from several electronic cigarette
companies to conduct a behavioral study on the effects of electronic
cigarettes on smoking behavior.
No comments:
Post a Comment