Tuesday, August 21, 2012

New Study on Effectiveness of Smoking Cessation Medications is Really a Retrospective, Rather than Prospective Cohort Study; Bias in Defining Exposure Groups Threatens Validity of Results

Yesterday, I reported on a new study published online ahead of print in the journal Addiction which presented survey data purporting to show that quit attempts using medication were more successful than unaided attempts. The study used data from an international survey conducted in the United Kingdom, Canada, Australia and the United States to study smoking and quitting behavior.

I criticized the study on the basis of a number of biases towards finding an effect of smoking cessation drugs, coupled with a financial conflict of interest of one of the study authors, who has served as a paid pharmaceutical consultant and speaker, writing: "the study introduces two sources of potential bias, both of which would bias the results toward finding a higher rate of effectiveness of medication. The more troubling of the two is the exclusion of medication users based on a question, asked retrospectively, about the reasons for their use of stop smoking medication. In my view, once you start excluding certain users of medication, you are no longer playing fair. Smoking cessation drugs are smoking cessation drugs and to exclude failures based on smokers retrospectively reporting that they actually didn't have the desire to completely quit smoking is not a fair and balanced analytic approach."

Today, I comment on a less important but interesting aspect of the study - namely, what type of study was it? According to the article, this was a "prospective cohort" study. However, as I will argue, the study design was really more of a retrospective cohort study.

The Rest of the Story

A cohort study is one in which subjects are grouped by their exposure to the variable of interest and then followed over time to determine an outcome of interest and its relationship to the exposure. In this study, the exposure variable is making a quit attempt with medication versus making a quit attempt without medication. The outcome is successful cessation for one month or six months. Thus, there is no question that it is a cohort study. Groups are defined based on their exposure (medication or no medication) and then quit rates are estimated and compared between the two groups.

The key to distinguishing a prospective versus a retrospective cohort study is whether subjects with the exposure were identified before or after the follow-up period began to determine their outcomes.

For example, suppose we conduct a longitudinal survey, following a group of 1000 smokers over two years. We interview them at baseline and then again two years later. Of course, some of them will have quit after the two years. We are interested in examining the differences in quit rates for smokers who quit using medication and for those who quit unassisted.

There are two possibilities for how this study can be conducted:

#1: We can determine the exposure variable prior to starting the follow-up period. In other words, we can take the 1000 smokers and divide them into two groups. One group is instructed to try to quit using medication. Another groups is instructed to try to quit without medication. Since the exposure is determined prior to the follow-up period (subjects are enrolled into exposure groups prior to the follow-up period), this is a prospective cohort design.

This design does not require that we instruct subjects whether to use medication or not. They could, instead, make their own decisions about how they choose to quit. But to be a prospective cohort study, the determination of which exposure group they are in needs to be made prior to the observation period of the outcome.

#2: Suppose we do not know which smokers will try to quit. Since our interest is only in comparing smokers who try to quit using medication and those who try to quit without medication, we cannot identify our exposure groups at the start of the study. We have to wait until the study is concluded and the second wave of data have been collected to define the exposure groups. Specifically, in the wave 2 survey, we would ask subjects if they made a quit attempt between waves 1 and 2, and if so, we would then assess what method they used to try to quit.

Because we are assessing the exposure after the observation period, this is a retrospective cohort design.

The study of interest is a bit more complicated because there are multiple waves of data, but essentially it is like a combination of mini-studies, each consisting of a two-wave follow-up study as per the above example. The key point is that exposures are being assessed only after the subjects have been followed up and the observation period has begun.

In any given wave, we do not know who is going to try to quit and whether they will use medication or not. Thus, exposures are not known prior to the observation period. The way that the investigators determine exposure to form the groups is to retrospectively assess (in the following wave) which subjects made a quit attempt and if so, what method they recall using to quit.

Because the exposure assessment is being made retrospectively - after the observation period has begun - this study is most accurately described as a retrospective cohort study.

Why This is So Important

The reason this is important is that it has major implications for the introduction of bias into the study.

If the study were conducted using a true prospective cohort design, there would be much less opportunity to introduce bias into the classification of exposure (i.e., whether or not someone was in the medication group or the non-medication group). Just prior to, or at the time of their quit attempts, subjects could easily be separated into groups based on the mode being used to quit. The assessment of the mode of quitting would be quite accurate as it would not depend at all upon recall.

However, with the retrospective cohort design of the present study, there is ample opportunity for the introduction of bias, and in fact, that is exactly what happened.

For one, the study re-defined what subjects were to be included in the exposure group after the fact. So rather than including all attempting quitters who reported using medication in their quit attempt, the study decided to throw out subjects who - retrospectively - reported that their original intention was not actually to quit completely.

Second, the study most likely misclassified exposure because at wave 5 and earlier, the study classified users of medication based on whether they had used any smoking cessation medication since the previous survey, not necessarily whether they used medication in their most recent quit attempt. The method used to quit during the most recent quit attempt does not appear to have been ascertained in the survey until wave 6. Therefore, it seems entirely possible that for wave 5 and earlier, a person might have tried and failed a few times to quit using NRT and then decided to go cold turkey for their most recent quit attempt. But because this person is not asked to report the method used in their most recent quit attempt, they would be classified as a medication user. If they were successful in their cold turkey attempt, that would go down in the results as a success for the use of drugs, not a success for unassisted quitting.

Third, because exposure is being assessed via recall (that is, retrospectively), it opens the door to the very bias that the study argues is operating here: differential recall of quit attempts based on whether the attempt was medication-assisted or unassisted. Were this a prospective cohort study as claimed by the article, then there would be no recall bias because exposure would have been determined at baseline, prior to the follow-up or observation period. It is precisely because the study uses a retrospective cohort design that there is a perceived need to control for recall bias.

Finally, the retrospective determination of the exposure groups allowed the study to introduce another source of bias: the inclusion of unaided quitters who tried to quit by cutting down gradually (a method known to be much less effective than abrupt, cold-turkey quitting).

A contrast to the current study is the study recently reported by the UK National Health Service. In that study, callers to a national quitline were assigned to receive or not receive medication and then followed up to observe their cessation rates. The results, as I reported last week, showed no significant advantage to the use of smoking cessation drugs. The advantage of this truly prospective cohort design is that exposure was determined more accurately as it did not depend on recall. In addition, the assessment of exposure status was not subject to bias. It was quite clear whether or not a client was receiving medication. No recall was required. The study could not retroactively drop a huge portion of the medication users out of the study by arguing that they didn't really intend to quit completely.

The rest of the story is that:

1. The retrospective nature of this cohort study opens it to substantial exposure classification bias.

2. There are indeed several aspects of substantial bias present in the exposure classification, each of which would lead to an overestimate of the relative efficacy of medication-assisted compared to unassisted and cold turkey quitting.

3. The presence of a significant financial conflict of interest of one of the study authors - who has been a paid consultant and served on the speakers bureau of pharmaceutical companies that manufacture smoking cessation drugs - creates the appearance that the biases introduced may be attributable to the conflict of interest (although I do not believe they were consciously introduced; conflicts of interest usually create bias subconsciously).

No comments: