Can you spot the problem with these headlines Level 1 Jeff Leek Lucy McGowan
“New drug may cure cancer.”
“Aspirin may reduce risk of
heart attacks.”
“Eating breakfast can help
you lose weight.”
Health headlines like these
flood the news,
often contradicting each other.
So how can you figure out what’s a
genuine health concern
or a truly promising remedy,
and what’s less conclusive?
In medicine,
there’s often a disconnect between
news headlines
and the scientific research they cover.
That’s because a headline is designed
to catch attention—
it’s most effective when
it makes a big claim.
By contrast,
many scientific studies produce
meaningful results
when they focus on a narrow,
specific question.
The best way to bridge this gap
is to look at the original research
behind a headline.
We’ve come up with a simplified
research scenario
for each of these three headlines
to test your skills.
Keep watching for the explanation
of the first study;
then pause at the headline
to figure out the flaw.
Assume all the information you need
to spot the flaw is included.
Let’s start with this hypothetical
scenario:
a study using mice to test
a new cancer drug.
The study includes two groups of mice,
one treated with the drug,
the other with a placebo.
At the end of the trial,
the mice that receive the drug are cured,
while those that received
the placebo are not.
Can you spot the problem
with this headline:
“Study shows new drug
could cure cancer”
Since the subjects of the study were mice,
we can’t draw conclusions about
human disease based on this research.
In real life, early research on new drugs
and therapies is not conducted on humans.
If the early results are promising,
clinical trials follow to determine
if they hold up in humans.
Now that you’ve warmed up,
let’s try a trickier example:
a study about the impact of aspirin
on heart attack risk.
The study randomly divides a pool
of men into two groups.
The members of one group
take aspirin daily,
while the others take a daily placebo.
By the end of the trial,
the control group suffered significantly
more heart attacks
than the group that took aspirin.
Based on this situation, what’s wrong
with the headline:
“Aspirin may reduce risk of heart attacks”
In this case, the study shows evidence
that aspirin reduces heart attacks in men,
because all the participants were men.
But the conclusion “aspirin reduces risk
of heart attacks” is too broad;
we can’t assume that results found in
men would also apply to women.
Studies often limit participants based on
geographic location, age, gender,
or many other factors.
Before these findings can be generalized,
similar studies need to be run
on other groups.
If a headline makes a general claim,
it should draw its evidence from a diverse
body of research, not one study.
Can you take your skills from the first
two questions to the next level?
Try this example about the impact
of eating breakfast on weight loss.
Researchers recruit a group of people
who had always skipped breakfast
and ask them to start
eating breakfast everyday.
The participants include men and women
of a range of ages and backgrounds.
Over a year-long period,
participants lose an average
of five pounds.
So what’s wrong with the headline:
“Eating breakfast can help
you lose weight”
The people in the study started eating
breakfast and lost weight—
but we don’t know that they lost weight
because they started eating breakfast;
perhaps having their weight tracked
inspired them to change their eating
habits in other ways.
To rule out the possibility that
some other factor caused weight loss,
we would need to compare
these participants
to a group who didn’t eat breakfast
before the study
and continued to skip it during the study.
A headline certainly shouldn’t claim the
results of this research
are generally applicable.
And if the study itself made such
a claim without a comparison group,
then you should question its credibility.
Now that you’ve battle-tested your skills
on these hypothetical studies
and headlines,
you can test them on real-world news.
Even when full papers aren’t available
without a fee,
you can often find summaries of
experimental design and results
in freely available abstracts,
or even within the text
of a news article.
Individual studies have results
that don’t necessarily correspond
to a grabby headline.
Big conclusions for human health issues
require lots of evidence accumulated
over time.
But in the meantime,
we can keep on top of the science,
by reading past the headlines.