IN MY VIEW: Study on Effect of England's Smoking Ban on Quit Rates Not Only Represents Science by Press Release; It Also Appears to Be Shoddy Science

I have already discussed why the study on the effect of England's smoking ban on quit rates is problematic on the grounds of it representing science by press release. Today, I posit, based on the limited information available about the study, that its conclusions are questionable as well. In light of science by press release approach, this is extremely problematic, because the conclusion has already been widely disseminated by the media.

Based on the little information we have available (i.e., the limited information that was provided by Cancer Research UK in its press release and which is publicly available on the internet), it appears that the conclusion of this study - that the smoking ban substantially reduced the quit rate in England - is based on a comparison of two data points: the quit rate during the nine months preceding the smoking ban (which was 1.6%) and the quit rate during the nine months following the smoking ban (which was 5.5%).

While it may appear, on the surface, that the smoking ban had a dramatic effect on the quit rate, increasing it from 1.6% to 5.5%, one cannot conclude that there was a causal effect of the smoking ban on the quit rate unless one can rule out the possibilities that the change in quit rate represents: (1) random variation in the quit rate over time; (2) a secular change in the quit rate that would have occurred in the absence of the smoking ban; and (3) a change that is attributable to some other factor.

In order to accomplish the above, one needs to do three important things:

(1) calculate the quit rate for several years prior to the smoking ban, preferably for about 8 years or so, in order to establish the baseline variability in the quit rate over time;

(2) compare the change in quit rates over time to changes in the quit rate in a comparison location, presumably a nearby country or countries without a smoking ban; and

(3) somehow rule out the possibility that there is another factor which may explain the change in quit rate -- in this case, that includes the implementation of another important policy change in October 2007: an increase in the legal age of purchase of tobacco from 16 to 18.

From what I can tell, the study did none of these three important things.

First, its conclusion appears to be based on a comparison of just two data points: the quit rate in 2006-2007 and 2007-2008. It doesn't even appear that data for the entire year following the smoking ban were used. It doesn't appear that the quit rate was determined for the years prior to the smoking ban. Rather than going back 8 years or so, it doesn't even appear that the study went back for a full year prior to the smoking ban.

While the Smoking Toolkit Study made an attempt to estimate historical quit rates, these rates were not actual rates but were very rough estimates, and they cannot be compared with the quit rate as calculated after the smoking ban. The methods are completely different and cannot be compared directly. The investigators themselves acknowledge that the historical quit rate calculations are very rough and should be taken with a grain of salt: "We do not have the data to be able to calculate background quit rates in the UK population but we can estimate historic rates and rates in the past year indirectly as long as certain assumptions hold true. ... Because of the assumptions involved in the estimates and the sample size used to calculate these figures, these figures must be treated with caution."

Even more troubling is the fact that there has been an impressive trend of declining smoking prevalence in England. In fact, one article pointed out that there was a reduction of 400,000 smokers between 2003 and 2004 -- the same reduction that between 2007 and 2008 is being attributed to the smoking ban. The article reports that there was a decline of 1.2 million smokers in England from 1998 to 2003. It was in 1998 that the Smoking Kills White Paper was released.

Second, it does not appear that any comparison country or location was used. This makes it very difficult, if not impossible, to conclude that any observed changes in quit rate would not have occurred anyway in the absence of the smoking ban.

Third, it does not appear that other potential causes of an increase in quit rate were ruled out, including the potential effects of the increase in the legal age of purchase of cigarettes and all the media attention to tobacco produced by the debate over the proposed smoking ban.

It is unfortunate that the investigators have issued this press release and garnered widespread media coverage of their conclusions without making the study available for review, so that we could ask precisely the above questions and review the methodology to examine how these three important issues were addressed, if at all.

This is now extremely problematic because if I am right and these conclusions are not justified based on the study's major limitations, it is really too late. The conclusions have already been widely disseminated to the public. Are newspapers going to run a special article that says: "Remember that story last year about how the smoking ban dramatically increased quit rates in England? Well, it turns out that the conclusion was invalid because the investigators could not rule out the possibility that the observed increase in the quit rate was due to random variation, rather than to a causal effect of the smoking ban."


(Thanks to GreatScot and Tim Clarke for ideas used in this post).