Faulty Scientific Conclusions in Published Research: Top Two Papers of the Week

Paper #1 -  Restaurant Smoking Bans and Youth Smoking Rates

Our first paper is a study of the relationship between national restaurant smoking bans and youth smoking rates in 15 European countries. The purpose of the study was to determine whether the enactment of restaurant smoking bans at a national level leads to a reduction in youth smoking.

(see: Vuolo M. Placing deviance in a legal and local context: A multilevel analysis of cigarette use in the European Union. Social Forces 2012; 90(4): 1377-1402.)

The investigator used data from the Eurobarometer survey, administered in 2002, to determine the smoking status of a probability sample of 450 youth ages 15-24 in each of the 15 countries. At the same time, he used data from the World Health Organization to determine whether or not each of these countries had a restaurant smoking ban in place in 2002. Such bans were in place in 7 of the 15 countries.

The paper uses a multilevel model to determine the relationship between the presence of a smoking ban and the proportion of youth smokers, while accounting for the clustering of respondents by country and while controlling for a range of individual-level, regional, and national factors that might influence youth smoking.

Using a logistic regression, the paper finds a statistically significant odds ratio of 0.65 for smoking associated with living in a country with a smoking ban. This means that the sampled youth were 0.65 times as likely to smoke (i.e., they were significantly less likely to be regular cigarette smokers) if they lived in a country with a restaurant smoking ban.

The paper concludes that restaurant smoking bans cause a reduction in youth smoking.

The conclusion of this study was disseminated through the media, with the help of a press release entitled "Restaurant smoking bans help snuff out European youth smoking." The press release stated: "Teenagers and young adults are less likely to smoke when faced with restaurant smoking bans."

The Rest of the Story

Does anybody see the flaw in the study conclusion? Do you see why this conclusion is flawed? Can you identify a plausible alternative explanation for the observed findings?

The answer lies in the classic distinction between correlation and causation. While the paper undoubtedly demonstrated a correlation between the presence of restaurant smoking bans and lower smoking prevalence among youth, one cannot conclude that smoking bans cause lower smoking rates. 

Why? Because this is a cross-sectional study and the direction of causation is not clear. It is certainly possible that restaurant smoking bans lead to a reduction in youth smoking, but is it not also plausible that countries with lower smoking rates are the ones which are more likely to enact restaurant smoking bans in the first place? In other words, is it not possible that countries with stronger anti-smoking sentiment (as evidenced by having fewer smokers) are the ones most likely to lead the pack in banning smoking in restaurants?

There is, in fact, evidence that localities with stronger anti-smoking sentiment and lower smoking prevalence are indeed more likely to adopt smoking bans. We found this to be the case in our study of the diffusion of restaurant smoking bans in Massachusetts.

Unfortunately, the cross-sectional nature of the study precludes the opportunity to determine the direction of causality: do the smoking bans precede a drop in youth smoking, or do low rates of youth smoking predict which countries will adopt smoking bans?

The paper does not mention this limitation. Despite its fancy statistical analyses and use of complex, hierarchical, generalized linear models, the interpretation of the study findings fails to take into consideration the basic distinction between correlation and causation.

By the way, there is strong evidence that restaurant smoking bans do reduce youth smoking. In a study I published with colleagues in the Archives of Pediatrics & Adolescent Medicine in 2008, we reported the results of a longitudinal study which examined the behavior of nonsmokers over a 4-year period in towns with and without restaurant smoking bans. We also controlled for the baseline levels of anti-smoking sentiment in each town. Because of the longitudinal nature of the study, we were able to rule out the possibility that any observed association between smoking bans and youth smoking was due to the tendency of towns with lower youth smoking to begin with to adopt such bans. We found that youth were about 0.6 times as likely to have progressed to established smoking over the 4-year follow-up period if they lived in towns with strong restaurant smoking bans. 

Paper #2 - Using Cell Phones in the Middle of the Night and Youth Depression

Our second paper is a study of the relationship between the use of cell phones in the middle of the night and depression and suicidal thoughts among youth. While not related to smoking, it demonstrates the same principles as our first case study.

In this study, the authors conducted a cross-sectional investigation of the use of cell phones after having gone to sleep and the presence of depression or suicidal ideation among a sample of adolescents using a self-administered questionnaire.

(see: Oshima N, et al. The suicidal feelings, self-injury, and mobile phone use after lights out in adolsecents. Journal of Pediatric Psychology 2012.) 

Nocturnal mobile phone use was defined as use of a cell phone to talk or send e-mails after lights out. Self-reported symptoms of anxiety or depression were measured, as were the presence of suicidal feelings or self-harm behaviors.

The paper reports a significant association between nocturnal mobile phone use and mental health symptoms, suicidal feelings, and self-harm behaviors, even after controlling for length of sleep. It reports that: "Logistic regression showed significant associations of the nocturnal mobile phone use with poor mental health, suicidal feelings, and self-injury after controlling for sleep length and other confounders."

How do the authors explain this relationship? They conclude that it is the cell phone use that causes the poor mental health, suicidal feelings, and self-harm. Specifically, they hypothesize that: 

1. "looking at the bright display of mobile phone might have critical physiologic effects on sleep."

2. "The combination of looking at a bright display and doing an exciting task (e.g., playing a shooting game) may change the secretion of melatonin and therefore the quality of sleep."

3. "the electromagnetic field ... emitted from mobile phones could have an effect on noctural melatonin secretion."

4. "Another mechanism could be negative emotions or stress by the mobile phone use."

The paper concludes: "The present study indicates that the mobile phone use might make a focus of the psychological education to improve mental health in adolescents. ... focusing on change in nocturnal use might be feasible, for example, through health education in high schools."

The article's conclusions were disseminated by the media, and included headlines such as: "Using cell phones after lights out may put teens at risk for depression, suicide."

The Rest of the Story

Before we start invoking theories regarding the effect of electromagnetic fields and melatonin secretion, do you see an alternative explanation for the study findings?

Once again, the answer lies in the distinction between correlation and causation. And once again, the cross-sectional nature of the study introduces the possibility that reverse causation is present here. It might be that cell phone use in the middle of the night causes depression and anxiety, but might it also be the case that youths who are depressed or anxious have more trouble sleeping and are therefore more likely to be awake in the middle of the night to use their cell phones?

In other words, might nocturnal cell phone use be a signal of mental health problems such as depression or anxiety, rather than the cause?

We do know that disturbed sleep is a very common symptom of depression and anxiety. Thus, it is entirely plausible that youths with mental health problems would be more likely to exhibit nocturnal cell phone use because of their depression or anxiety.

The distinction here is of great importance because if we get the direction of causation wrong, we may deliver the wrong intervention. If the cell phone use is the cause of the depression, then yes, taking the cell phones away from teenagers may cure their depression problems and lower the rates of self-injury among adolescents. However, if it is the underlying depression or anxiety that is signalling nocturnal cell phone use, then taking the cell phones away is not going to have any effect and we risk wasting resources on an intervention that won't work and which may distract us away from addressing the real problem.

Thus, in this case, making a causal attribution error could actually have substantial adverse public health consequences.