Tuesday, I reported here that yet another study with severely faulty methods has concluded that an observed decline in cardiovascular disease admissions is attributable to a smoking ban. The study, which was published online ahead of print in the Canadian Medical Association Journal, purported to show that the smoking ban resulted in a decrease in cardiovascular disease admissions by reporting a large decline in admissions in Toronto, which the study said did not occur in the comparison cities (Durham Region and Thunder Bay).
The study has received widespread media attention, with a typical headline reading something like that in the Toronto Globe and Mail (Toronto Smoking Ban Leads to Decline in Hospitalizations) and text reading something like: "A ban on smoking in restaurants is being credited for a precipitous drop in hospital admissions for cardiovascular and respiratory problems."
Today, I report an analysis of the actual data. The data on cardiovascular disease admission rates for Toronto are reported in the study and the data for the two comparison cities are reported in an online appendix.
The Rest of the Story
The truth is that the data show that while there was indeed a huge decline in cardiovascular disease admissions in Toronto, there were also huge declines in cardiovascular disease admission rates in both of the comparison cities during the same time period. If one actually examines and analyzes the data, one sees that the magnitude of the decline in cardiovascular disease admission rates in Toronto is similar to that observed in the comparison cities. Therefore, this paper does not provide any evidence that the smoking ban led to a decline in cardiovascular disease hospitalizations.
To see why the study conclusions are invalid, consider the following data.
1. Decline in angina hospital admission rates, 2001-2006
Durham Region: 59%
Conclusion: There was no difference in the rate of decline in hospital admissions for angina in Toronto compared to Durham Region from 2001-2006.
2. Decline in heart attack hospital admission rates, 2001-2006
Thunder Bay: 28%
Conclusion: There was no difference in the rate of decline in hospital admissions for heart attacks in Toronto compared to Thunder Bay from 2001-2006.
3. Decline in stroke hospital admission rates, 2001-2006
Durham Region: 35%
Conclusion: There was no difference in the rate of decline in hospital admissions for stroke in Toronto compared to Durham Region from 2001-2006.
4. Decline in overall cardiovascular disease hospital admission rates, 2001-2006
Durham Region: 38%
Conclusion: There was almost no difference in the rate of decline in hospital admissions for cardiovascular disease in Toronto compared to Durham Region from 2001-2006.
Even if you just examine the period from 2001-2004, when the smoking ban in restaurants was in effect (but before the bar smoking ban took effect), the results are similar:
5. Decline in overall cardiovascular disease hospital admission rates, 2001-2004
Durham Region: 31%
Conclusion: There was no difference in the rate of decline in hospital admissions for cardiovascular disease in Toronto compared to Durham Region from 2001-2006.
What This All Means
What this all means is that the study was correct in reporting a large decline in cardiovascular disease admissions in Toronto that occurred shortly after the smoking ban. However, where the study went awry was in simply telling the reader that there was no similar decline in the comparison cities, when in fact these cities experienced very similar declines in cardiovascular disease rates to those observed in Toronto.
The most important point I want to emphasize is that these trends are readily evident simply from looking at the actual data. And although the paper claims to have compared hospitalization trends in Toronto with those in the comparison cities, it actually does not analyze the data from Durham Region or Thunder Bay. These data are summarily dismissed, without any question asked. The paper justifies this by arguing that there is not a consistent trend.
Well it looks pretty consistent to me. Nevertheless, consistent or not, a study should actually analyze the data it collects. That's a most basic concept in data analysis. You collect data and then you analyze it.
Strikingly, while this paper presents and analyzes the data for trends in cholecystitis, appendicitis, and bowel obstruction in the comparison cities, it does not do the same for cardiovascular disease. But that is the most important part of the analysis, which is central to the study's ability to determine whether the observed changes in heart disease in Toronto were attributable to the smoking ban or due to a secular decline that was occurring anyway.
In other words, this paper does not actually perform the critical analysis that is central to the research question it poses. I have no idea how the paper got past the peer review process without having carried out the central analysis required to answer its basic research question. Nor do I understand how the paper could just leave the data for the control cities sitting there and not analyze them.
I can't answer those questions, but I can tell you that the actual data collected in this study demonstrate that the declines in cardiovascular disease associated with the smoking ban in Toronto were roughly identical to those observed in the comparison cities during the same time period. Thus, the study does not support the conclusion that the smoking ban resulted in a decline in cardiovascular disease hospitalizations.