Tuesday, November 20, 2007

How the Wrong Statistical Analysis Can Produce an Invalid Study Conclusion: The Indiana Smoking Ban/Heart Attack Study

According to a study published this month in the Journal of Drug Education, the smoking ban enacted in Monroe County, Indiana in August 2003 has led to a 70% decline in heart attacks among nonsmokers with no previous history of cardiac disease (see: Seo D-C, Torabi MR. Reduced admissions for acute myocardial infarction associated with a public smoking ban: Matched controlled study. Journal of Drug Education 2007; 37:217-226).

As I explained yesterday, the study compared the number of heart attacks among patients without prior cardiac disease or major cardiac risk factors in Monroe County with the number of heart attacks among similar patients in Delaware County, a county in Indiana with similar demographics but no smoking ban. The data were compared for a 22 month period prior to the ban (August 2001 through May 2003) and a 22 month period following the ban (August 2003 through May 2005). The smoking ban was implemented in August 2003 and included all workplaces and restaurants. Bars were initially exempt, but became smoke-free in January 2005. Analysis was stratified by smoking status of the heart attack patients.

The paper reports a 70% decline in heart attacks among nonsmoking patients in Monroe County, compared to only an 11% decline in heart attacks among nonsmoking patients in Delaware County. There was no significant change in heart attacks among smoking patients in either county.

The authors conclude that the substantial reduction in observed heart attacks in Monroe County was attributable to the smoking ban.

If you examine the details from the paper, it turns out that the total number of nonsmoking heart attack patients upon which the study conclusions are drawn is 22 in Monroe County and 34 in Delaware County. That is, the number of heart attacks in Monroe County decreased from 17 to 5, and in Delaware County decreased from 18 to 16.

The paper reports the decline in heart attacks in Monroe County to be -12, with a 95% confidence interval of -21.19 to -2.81. This means the decline is statistically significant. One can conclude that the true number of heart attacks in Monroe County definitely went down, not up.

The paper reports the decline in heart attacks in Delaware County to be -2, with a 95% confidence interval of -13.43 to +9.43. This means the decline is not statistically significant. The confidence interval crosses zero, which means that the true number of heart attacks in Delaware County could have gone down or it could have gone up.

Based on the finding that the decline in heart attacks in Monroe County was statistically significant and the decline in Delaware County was not statistically significant, the paper concludes that there was a significant difference in the decline in heart attacks between the two counties.

The Rest of the Story

Unfortunately, the paper relies on the wrong analysis to analyze these data. The mere fact that the decline in heart attacks in Monroe County was statistically significant and the decline in heart attacks in Delaware County was not statistically significant does not mean that the degree of change in heart attacks between the two counties was statistically different. That is the wrong statistical analysis to apply to these data to answer the primary research question at hand.

The primary research question at hand is whether or not the change in heart attacks in Monroe County was statistically different from the change in heart attacks in Delaware County. To test that, one needs to do 2 things:

(1) generate an estimate of the difference in the change in heart attacks between the two counties; and

(2) generate an estimate of the degree of uncertainty (the "standard error") in that estimate of the difference in the change in heart attacks between the two counties.

Unfortunately, the paper only does the first of these. It does not do the second.

The paper provides an estimate that the difference between the change in heart attacks in Monroe County and Delaware County was -10. That is, the paper estimates that the decline in heart attacks in Monroe County was 10 greater than the decline in heart attacks in Delaware County (or to be more accurate, that the change in heart attacks in Monroe County was 10 more negative than the change in Delaware County - it could have been that the true number of heart attacks in Delaware County actually went up).

However, the paper does not provide a confidence interval around this estimate of -10. It does not provide the reader of an idea how much variability there is around this estimate. With 95% certainty, can we be sure that this estimate of -10 is not actually greater than -20? Or, can we be sure with 95% certainty that this estimate is not actually positive (greater than zero). We need to know whether the confidence interval includes zero or not. If it does, then the estimate is not statistically significant.

Because the paper does not provide this estimate of the confidence interval around the estimate of a -10 difference in the change in heart attacks between the two counties, we are unable to conclude (and the paper is unable to conclude, despite its conclusion) that there was a significant difference in the change in heart attacks between the two counties, with Monroe County experiencing a greater drop in heart attacks.

We can, however, make a rough estimate of the likely 95% confidence interval around the estimate of -10 by calculating the variance of the estimates for the change in heart attacks in each county, since those confidence intervals are provided. If one does this, one finds that the likely 95% confidence interval around the -10 actually includes zero. In fact, the upper limit of the confidence interval appears to reach as high as +4.7.

In other words, the study findings are entirely consistent with the possibility that the change in heart attacks in Monroe County was actually 4.7 lower than the change in Delaware County. It is entirely consistent with the results of the paper that the heart attacks in Delaware County dropped by more than the heart attacks in Monroe County.

For example, it is entirely possible that the true number of heart attacks in Monroe County fell by 3 (which is within the 95% confidence interval for the change in heart attacks in Monroe County) and that the true number of heart attacks in Delaware County fell by 8 (which is well within the 95% confidence interval for the change in heart attacks in Delaware County).

This means that the study cannot conclude that the true number of heart attacks in Monroe County did not decline by more than the true number of heart attacks in Delaware County.

The rest of the story, then, is that based on the very data presented in the paper, it is entirely possible that the smoking ban was associated with a lowering of the decline in heart attacks in Monroe County.

Why is the confidence interval around the estimate of -10 so wide that it includes zero? The reason is that the number of heart attacks in the study is so very small.

Even though the number of heart attacks in Delaware County seemed to stay about the same (18 and then 16), the numbers are so small that the paper can not rule out that the true number of heart attacks might actually have gone down by as much as 13, or up by as much as 9.

In other words, there is so much uncertainty about whether heart attacks in Delaware County went up, down, or stayed the same, that it becomes impossible to conclude that the difference of 10 in the change in heart attacks between the two counties was statistically significant.

I hope I explained that well enough. I tried to make it as clear as possible, but if I need to clarify some of this in the comments, I'll be happy to do that.

The bottom line is that the paper presents the wrong statistical analysis. Based on the data presented in the paper, the conclusion that the number of heart attacks among nonsmokers in Monroe County declined by more than the number of heart attacks in Delaware County is not supported, and therefore is invalid.

It is difficult for me to imagine that none of the reviewers caught this and asked the authors to present such an estimate of variability around this key statistic, but it happened. Sometimes these errors, even when there are sentinel errors like this one, can fall through the cracks.

By the way, I'm not arguing that the smoking ban failed to reduce the number of heart attacks in Monroe County. I'm simply arguing that this particular paper provides essentially no basis for drawing such a conclusion.

Legislators will soon be considering legislation that would eliminate smoking in workplaces, including bars, restaurants, and casinos in Indiana. While I strongly support this legislation to protect the health of nonsmokers, I would hope that anti-smoking advocacy groups in Indiana will not rely on this particular study to back up their support for this bill. I think that health advocates actually hurt their cause more than help it when they rely on junk science like this to promote public policy.

No comments: