Meta-Analysis of Smoking Ban and Heart Attack Studies Seems to Have Cherry-Picked Its Data, Excluding the Second Largest Study Because It Failed to Find an Effect

As I reported here yesterday, in the largest meta-analysis to date of studies examining the effect of smoking bans on heart attack admissions, a paper published this week in the journal Circulation concludes that smoking bans lead to an immediate 15% decline in heart attack admissions or deaths. I noted that the authors excluded from the analysis a study which found no effect of smoking bans on heart disease deaths in six states. After further examination of the reasons provided by the authors for the exclusion of that study, I have concluded that there was no valid reason for the study's exclusion and it appears to be motivated by a desire to exclude the negative findings than by a valid scientific justification.

To remind you, in that study (Rodu B, Peiper N, Cole P. Acute myocardial infarction mortality before and after state-wide smoking bans. J Community Health 2011), the authors examined age-adjusted rates of heart attack mortality during the 3 years before implementation of the smoking ban and during the first year after the smoking ban was implemented in the eight states that implemented smoking bans between 1995 and 2003. These trends were also compared with those in the 44 other states without smoking bans.

The results were that in four of the six states (California, Utah, Delaware, and South Dakota), the smoking bans were not associated with any significant short-term decline in heart attack mortality. In one of these states - South Dakota - there was an 8.9% increase in heart attack mortality during the first year of the smoking ban which was significantly different from the expected decline of 7.2%. These results certainly seem to refute the assertion that smoking bans lead to a dramatic, immediate decline in heart attacks.

The authors of the meta-analysis give two major reasons for excluding the Rodu study:

1. The study "did not report or present data that permitted estimating relative risk and confidence intervals."

2. "In addition, the analysis was based on a very small number of data points...".

The Rest of the Story

Let's examine each of these supposed justifications for excluding the Rodu study.

1. The study "did not report or present data that permitted estimating relative risk and confidence intervals."

The raw data to calculate these relative risks and confidence intervals would gladly have been provided by Dr. Rodu had the meta-analysis authors simply asked him for the data. The authors acknowledge that they contacted the authors of another study (the "Malta" study) to provide additional information, but they apparently made no attempt to contact Dr. Rodu. Given the large size and importance of the Rodu study, its exclusion is not scientifically justified on these grounds, especially since the data were only one email away.

2. "In addition, the analysis was based on a very small number of data points...".

This is untrue. The Rodu study was actually the second largest study ever conducted. It examined heart attack death rates in six different states. Most of the other studies examined heart attacks in only one community, and many of those communities were small. Dr. Glantz' own study, in which he concluded that a smoking ban in Helena, Montana reduced heart attack admissions, was based on a decrease of only 16 cases.

A number of the studies included in the meta-analysis had very small sample sizes. As Rodu et al. point out in their paper: "In Helena there were 40 and 24 admissions for AMI before and after the smoke-free ordinance; in Pueblo the corresponding numbers were 399 and 291; in Bowling Green, 36 and 22; and in Monroe County, 17 and 5." In contrast, the Rodu study was based on a total of nearly 40,000 heart attack deaths.

In my view, there is no justification for excluding such an important study: the study with the second largest sample size of all. Moreover, if one is going to exclude studies based on low sample size, then you need to be consistent. If the authors wanted to exclude studies with low sample size, then they should have set a threshold and excluded all studies below that threshold. Of course, if the threshold was set above the sample size of the Rodu study, that would have resulted in throwing out most of the studies they reviewed.

It is therefore my opinion that underneath the surface, the real reason for the exclusion of the Rodu study was that it showed negative results among a systematic sample of large populations. The study is fatal to the notion that smoking bans produce dramatic, immediate, and measurable reductions in heart attacks.

Most worrisome about the cherry-picking that is evident in this meta-analysis is that this is precisely what the authors of the meta-analysis have attacked the tobacco companies for doing in their own analyses in the past.

The rest of the story is that the scientific quality of research in the area of smoking bans and heart attacks has deteriorated to such a level that we are now using techniques that we attacked tobacco companies for using in the past: failing to include comparison groups in our analyses, being inconsistent in our criteria for including studies, and employing a systematic bias in our interpretation of the results so as to produce favorable findings.