A new, comprehensive, 51-page review of multiple meta-analyses summarizing the effectiveness of nicotine replacement therapy (NRT) and other drugs for smoking cessation concludes that these drugs are highly effective in helping smokers quit.
(See: Cahill K, Stevens S, Perera R, Lancaster T. Pharmacological
interventions for smoking cessation: an overview and network
meta-analysis. Cochrane Database of Systematic Reviews 2013, Issue 5. Art. No.:CD009329.DOI: 10.1002/14651858.CD009329.pub2.)
The review was extensive, covering hundreds of studies and tens of thousands of subjects: "The authors combined the findings of existing Cochrane reviews on the
subject, using all the available data from across the individual
reviews. In total, they collected evidence from 267 studies, which
together involved a total of 101,804 people. The studies covered a wide
variety of licensed and unlicensed smoking cessation medications,
comparing the treatments with placebo, and the three main treatments
with each other. If a person stopped smoking for six months or longer,
this was considered a successful quit attempt."
The study results were summarized - correctly - by one news outlet, as follows: "Participants were 80% more likely to quit when using a single NRT or
taking bupropion compared to those using a placebo. Those
using varenicline as well as an NRT had two to three times greater odds."
The Rest of the Story
If I gave my students an assignment to review the scientific evidence on the effectiveness of NRT and other drugs for smoking cessation and they handed in this comprehensive, 51-page review, I would not be able to give them a passing grade on the assignment.
Because the article leaves out the single most important piece of information that the review should provide. It fails to answer the single most important research question:
What was the success rate for these products?
How could any review of this topic claim to be "comprehensive" when it doesn't answer the most basic question: What percentage of people who use NRT or other drugs to quit smoking are successful? If the paper doesn't provide the answer to that question, how can we consider it to be a solid review of the scientific evidence on this topic? That's why I would have to give a student passing in such a paper a failing grade.
This review reports only the relative risk of smoking cessation with NRT/drugs, not the absolute
prevalence of abstinence following treatment. Knowing that the relative risk for cessation associated with NRT compared to placebo is 1.8 doesn't actually answer the question of whether these drugs are effective.
Suppose you went to the doctor for pneumonia. She prescribed an antibiotic and told you that the antibiotic is 500 times more likely than placebo to cure your infection. Sounds great, right? But suppose that the absolute cure rate with placebo for bacterial pneumonia is only 0.001%. That would mean that this antibiotic has a success rate of 0.5%. This means it will work only 5 out of 1,000 times. Your proper course of action is not to fill your prescription for this antibiotic with a relative risk of 500 compared with placebo but to head to your lawyer's office to file malpractice papers.
Well, sadly, this is the precise error that this review commits. It reveals the relative risk of treatment but hides from the reader any information about the absolute success rates. In other words, it doesn't actually tell us whether these treatments work or not!
The truth is that placebo success rates are only about 5%, so at best, use of NRT in real world settings produces about a 10% long-term quit rate.
How can a treatment with a 90% probability of complete failure be considered effective?
Instead of concluding that "Participants were 80% more likely to quit when using a single NRT or
taking bupropion compared to those using a placebo," the review could have just as easily concluded that "Participants using NRT or
taking bupropion had a 90% chance of failing to quit smoking."
To me, a 90% failure rate is not the hallmark of an "effective" medication.