How to Read Beyond the Headline: 9 Essential Questions to Evaluate Medical News

Ben Goldacre writing in Bad Science classified science reporting as falling into three categories – wacky stories, scare stories and breakthrough stories; the last of which he views as ”a more subtly destructive category of science story”. Whether you get your news through digital or traditional means, you can’t fail to notice the regularity with which journalists report on the latest medical breakthroughs. Some of these reports are sensationalist (“coffee causes cancer”) and fairly easy to dismiss; but do you know how to separate fact from fiction when it comes to less sensationalist headlines?

The foundation of empowered patient-hood is built on reliable health information. This means not only knowing where to find medical information, but being able to evaluate it and knowing how it can be applied to your own, or your loved-ones’ particular circumstances. Headlines often mislead people into thinking a certain substance or activity will prevent or cure chronic disease. As patient advocates we must learn to read beyond the headlines to filter out the good, the bad, and the questionable. The following questions are designed to help sort the signal from the noise next time you read the latest news story heralding a medical breakthrough.

1. Does the article support its claims with scientific research?

Your first concern should be the research behind the news article. If an article contains no link to scientific research to support its claims, then be very wary about treating those claims as scientifically credible.

2. What is the original source of the article?

If the article cites scientific research you should still treat the findings with caution. Always consider the source. Find out where the study was done. Who paid for and conducted the study? Is there a potential conflict of interest?

3. Does the article contain expert commentary to back up claims?

Look for expert independent commentary from doctors or other healthcare providers to explain the findings (there should be an independent expert source quoted – someone not directly connected with the research).

4. Is this a conference presentation?

Journalists frequently report on research presented at large scientific meetings. It’s important to realize that this research may only be at a preliminary stage and may not fulfill its early promise.

5. What kind of clinical trial is being reported on?

If the news relates to results from a clinical trial, it’s important you understand how, or even if, the results apply to you. Quite often, news publications report on trials which have not yet been conducted on humans. Many drugs that show promising results in animals don’t work in humans. Cancer.Net and American Cancer Society have useful guides to understanding the format of cancer research studies.

6. What stage is the trial at?

Research studies must go through several phases before a treatment can be considered safe and effective; but many times journalists report on early phase trials as if these hold all the answers. The testing process in humans is divided into several phases:

•  Phase I trials: Researchers test a new drug or treatment in a small group of people for the first time to evaluate its safety, determine a safe dosage range, and identify side effects.
• Phase II trials: The drug or treatment is given to a larger group of people to see if it is effective and to further evaluate its safety.
• Phase III trials: The drug or treatment is given to large groups of people to confirm its effectiveness, monitor side effects, compare it to commonly used treatments, and collect information that will allow the drug or treatment to be used safely.

Source: ClinicalTrials.gov

7. How many people did the research study include?

In general, the larger a study the more you can trust its results. Small studies may miss important differences because they lack statistical power.

8. Did the study include a control group?

A control group allows researchers to compare outcomes in those who receive a treatment with those who don’t. The gold standard is a “randomised controlled trial”, a study in which participants are randomly allocated to receive (or not receive) a particular intervention (e.g. a treatment or a placebo).

9. What are the study’s limitations?

Many news stories fail to point out the limitations of the evidence. The limitations of a study are the shortcomings, conditions or influences that cannot be controlled by the researcher. Any limitations that might influence the results should be mentioned in the study’s findings, so always read the original study where possible.

Useful Resources

• Sense about Science works with scientists and members of the public to equip people to make sense of science and evidence. It responds to hundreds of requests for independent advice and questions on scientific evidence each year.
• Trust It or Trash is a tool to help you think critically about the quality of health information (including websites, handouts, booklets, etc.).
• Understanding Health Research (UHR) is a free service created with the intention of helping people better understand health research in context. It gives clear and understandable explanations of important considerations like sampling, bias, uncertainty and replicability.