Last week, I posted about the American Legacy Foundation's claim that its "truth" campaign will save "hundreds of thousands of lives" (or less modestly, "millions of young lives"). I opined that this is an unsubstantiated and unfounded claim that is based on a single paper which Legacy itself authored and that it was reported in a public statement by Legacy's President and CEO which failed to disclose that the paper being cited in support of the claim was actually authored by her and paid for by Legacy.
Since then, a number of readers have asked me to explain why I find Legacy's claim to be unsubstantiated, in light of the conclusions made in the AJPH paper.
There are 5 basic points that I think readers should be aware of to understand my reasoning:
1. The Paper Found No Linear Relationship Between Exposure to the "truth" Campaign and Smoking Prevalence
The first test the paper made to see whether higher exposure to the "truth" campaign was associated with lower youth smoking prevalence was whether youth smoking decreased linearly with "truth" campaign exposure over the range of observed exposures in the sample. The paper found that there was no significant linear relationship between the predictor and outcome variable.
This itself would not be the expected outcome if the media campaign was as effective as Legacy claims that it has been. Why would one not find an increasing effect of the campaign on smoking prevalence as exposure to the campaign increased?
The paper suggests that perhaps an explanation for this is that the relationship between exposure and outcome was not linear, but L-shaped. Perhaps the campaign's effects increased as exposure increased, but only up to a point - higher levels of exposure did not have any increased value after that point.
The problem is that this is not what the paper found. It actually reported a U-shaped relationship, whereby the campaign's purported effect on youth smoking increased up to moderate levels of exposure, but then actually decreased as exposure became very high. In fact, at the very highest levels of exposure, there was no effect of the campaign on smoking behavior.
While the paper concludes that this is evidence for a diminishing returns model (whereby increasing exposure effects youth smoking, but only up to a certain point, then further increases in exposure have no further effect on smoking), I believe that the data best support the conclusion that the campaign simply did not work as intended.
Had the campaign really worked, then one would have expected to see the absolutely greatest effect at the highest levels of exposure. Since the highest levels of exposure were actually associated with no effect, I find it difficult to conclude from this single study that the "truth" campaign was effective. But even if I concluded that it was effective, I certainly would not go so far as to quantify the effects in precise terms (i.e., attributing "22%" of the observed decline in smoking prevalence to the campaign), given some of the inconsistencies in the observed findings.
It's kind of like finding that smoking 1-2 packs per day of cigarettes caused lung cancer but smoking 3 packs per day did not. Yes - it is possible that there is some sort of threshold by which additional smoking would not increase the lung cancer risk, but there is no plausible reason why it would take away the risk. From results such as these, one would certainly pause before concluding that a causal connection existed, and more research would certainly be warranted to clarify what is going on.
2. The Paper Did Not Measure Exposure to the "truth" Campaign Among a Single Individual
The paper was essentially a cross-sectional, ecological study. It did not measure exposure to the "truth" campaign among individuals. Instead, it compared rates of youth smoking in regions of the country that had varying levels of intensity of exposure to the "truth" ads. But the study cannot document that the youth subjects were or were not in fact exposed to the campaign, and to what extent.
This does not invalidate the study. However, it does suggest that extreme caution should be used in drawing causal conclusions. It means that the research is very susceptible to the potential effects of confounding variables, since there are a large number of differences between youths in different regions of the country - not just differences in their exposure to the "truth" campaign. It would really require multiple studies to ensure that the observed results of the study are not due to a spurious relation between exposure to the campaign and youth smoking (i.e., that they are not due to some confounding variable that is related both to campaign exposure and to youth smoking rates in different areas of the country). At very least, I would be hesitant to take the results of a single cross-sectional, ecological study and use them to draw a conclusion of causality and then to go so far as to quantify such an effect and apply it to future generations of youths.
3. This is a Single, Cross-Sectional Study
Even if the results of the study were completely consistent with an effect of the "truth" campaign on youth smoking behavior, I would almost never take the results of such a cross-sectional study and: (1) draw a conclusion of a causal relationship; (2) quantify such a relationship in precise terms; and (3) extrapolate those results to what effects one could expect from the campaign in future years and express those results in terms of the number of lives that will be saved by the campaign.
There are methods available to assess the effects of anti-smoking media campaigns using a longitudinal design, and such methods have been used to document the effectiveness of anti-smoking media campaigns in Massachusetts and Florida.
In my own research, I was unwilling to draw a conclusion that the Massachusetts anti-smoking media campaign resulted in a decrease in youth smoking until I found that in a longitudinal study, youths exposed to the ads were more likely to start smoking (over a four-year follow-up period) than youths who were not exposed to the ads. I would never have drawn such a conclusion from a single cross-sectional study of regional exposure differences and regional differences in youth smoking.
4. The Study was Paid for and Authored by the American Legacy Foundation
The simple fact that the study was funded by and authored by Legacy does not in any way invalidate the study. It needs to be judged on its scientific validity. However, in drawing a causal conclusion about whether the "truth" campaign reduces youth smoking (and certainly in quantifying the number of lives saved by such a campaign), one would prefer to see at least one independent study confirm the results reported by the Foundation. Actually, in this case, one would want to see that the results were not confirmed, and that instead, the very highest levels of exposure to the campaign were associated with the strongest effects on youth smoking.
5. The Paper's Findings Rely Upon a Conclusion of Diminishing Campaign Effects
Even if it were sound to conclude from the paper that the campaign did reduce youth smoking, such a conclusion would rely upon the conclusion that the campaign's impact had a diminishing effects dose-response relationship. In other words, campaign exposure was effective up to a point, but at some point, increased exposure to the ads had no further effect on smoking behavior, and may have even been counter-productive. If true, that is hardly what I would call an effective advertising campaign.
But more importantly, even if these results supported a conclusion that the campaign was effective in its early years, it would actually suggest that the campaign's effectiveness may have worn out and that future exposure might not be expected to have similar effects on smoking. Thus, even if a conclusion that the campaign was effective in its first two years were reasonable, it would be inconsistent with the paper's findings to claim that the campaign will have similar effects in the future. In fact, the research would actually suggest that the campaign would not be effective, or would have limited effectiveness, in the future. One would certainly not want to go around claiming that the campaign was going to save "millions of young lives."
The Rest of the Story
In summary, I think there are a number of important reasons why Legacy's claims about the impact of its own "truth" campaign are wildly exaggerated. Perhaps most importantly, I don't find that the results, as reported in the original AJPH article, actually support a conclusion that the campaign was effective in reducing youth smoking. But even if they did, I think it is not particularly responsible from a scientific standpoint to draw a conclusion of causality from a single cross-sectional, ecological-style study. And it certainly doesn't seem reasonable to use a single cross-sectional study to quantify a precise campaign effect, and further, to extrapolate that finding to infer what the campaign's effects will be in the future (especially since the AJPH paper itself concludes that there were diminishing effects of increased campaign exposure).
Note that I am in no way concluding that the "truth" campaign did not reduce youth smoking; I am simply opining that the evidence does not support a conclusion that it did. It may or may not have a significant effect on youth smoking. More research is necessary to make this determination. But until that evidence is available, I do not think it is reasonable or particularly responsible for Legacy to be making these wildly exaggerated claims of its campaign's effectiveness, especially in the setting of trying to convince a federal court to give it billions of dollars.
No comments:
Post a Comment