This one weird trick will help you spot clickbait Jeff Leek Lucy McGowan
One simple vitamin can reduce
your risk of heart disease.
Eating chocolate reduces
stress in students.
New drug prolongs lives of
patients with rare disease.
Health headlines like these
are published every day,
sometimes making opposite claims
from each other.
There can be a disconnect between broad,
attention-grabbing headlines
and the often specific,
incremental results of the medical
research they cover.
So how can you avoid being
misled by grabby headlines?
The best way to assess a headline’s
credibility
is to look at the original
research it reports on.
We’ve come up with a hypothetical research
scenario
for each of these three headlines.
Keep watching for the explanation
of the first example;
then pause at the headline to
answer the question.
These are simplified scenarios.
A real study would detail many more
factors and how it accounted for them,
but for the purposes of this exercise,
assume all the information
you need is included.
Let’s start by considering the
cardiovascular effects
of a certain vitamin, Healthium.
The study finds that participants taking
Healthium
had a higher level of healthy cholesterol
than those taking a placebo.
Their levels became similar to those of
people with naturally high levels
of this kind of cholesterol.
Previous research has shown that people
with naturally high levels
of healthy cholesterol have lower
rates of heart disease.
So what makes this headline misleading:
“Healthium reduces risk of heart disease.”
The problem with this headline is that the
research didn’t actually investigate
whether Healthium reduces heart disease.
It only measured Healthium’s impact
on levels of a particular
kind of cholesterol.
The fact that people with naturally high
levels of that cholesterol
have lower risk of heart attacks
doesn’t mean that the same
will be true of people
who elevate their cholesterol
levels using Healthium.
Now that you’ve cracked the
case of Healthium,
try your hand at a particularly alluring
mystery:
the relationship between eating chocolate
and stress.
This hypothetical study
recruits ten students.
Half begin consuming a
daily dose of chocolate,
while half abstain.
As classmates, they all follow
the same schedule.
By the end of the study, the chocolate
eaters are less stressed
than their chocolate-free counterparts.
What’s wrong with this headline:
“Eating chocolate reduces
stress in students”
It’s a stretch to draw a conclusion about
students in general from a sample of ten.
That’s because the fewer participants are
in a random sample,
the less likely it is that the sample will
closely represent
the target population as a whole.
For example, if the broader population of
students is half male and half female,
the chance of drawing a sample of 10
that’s skewed 70% male and
30% is about 12%.
In a sample of 100 that would be less than
a .0025% chance,
and for a sample of 1000,
the odds are less than 6 x 10^-36.
Similarly, with fewer participants,
each individual’s outcome has a larger
impact on the overall results—
and can therefore skew big-picture trends.
Still, there are a lot of good reasons for
scientists to run small studies.
By starting with a small sample,
they can evaluate whether the results are
promising enough
to run a more comprehensive,
expensive study.
And some research requires very specific
participants
that may be impossible to
recruit in large numbers.
The key is reproducibility—
if an article draws a conclusion
from one small study,
that conclusion may be suspect—
but if it’s based on many studies
that have found similar results,
it’s more credible.
We’ve still got one more puzzle.
In this scenario, a study tests a new drug
for a rare, fatal disease.
In a sample of 2,000 patients,
the ones who start taking the drug upon
diagnosis
live longer than those who
take the placebo.
This time, the question
is slightly different.
What’s one more thing you’d like to know
before deciding if the headline,
“New drug prolongs lives of patients
with rare disease”, is justified?
Before making this call,
you’d want to know how much the drug
prolonged the patients’ lives.
Sometimes, a study can have results that,
while scientifically valid, don’t have
much bearing on real world outcomes.
For example, one real-life clinical trial
of a pancreatic cancer drug
found an increase in life expectancy—
of ten days.
The next time you see a surprising medical
headline,
take a look at the science
it’s reporting on.
Even when full papers aren’t
available without a fee,
you can often find summaries of
experimental design
and results in freely available abstracts,
or even within the text
of a news article.
It’s exciting to see scientific research
covered in the news,
and important to understand
the studies’ findings.