Not all scientific studies are created equal David H. Schwartz
Studies have shown that
taking vitamins is good for your health
and bad for your health.
That newly discovered herb can improve your memory
or destroy your liver.
Headlines proclaim a promising new cancer treatment
and never mention it again.
On a daily basis,
we are bombarded with attention-grabbing news,
backed up by scientific studies,
but what are these studies?
How are they performed?
And how do we know whether they’re reliable?
When it comes to dietary or medical information,
the first thing to remember
is that while studies on animals or individual cells
can point the way towards further research,
the only way to know how something will affect humans
is through a study involving human subjects.
And when it comes to human studies,
the scientific gold standard is
the randomized clinical trial, or RCT.
The key to RCTs is that the subjects are randomly assigned
to their study groups.
They are often blinded to make them more rigorous.
This process attempts to ensure
that the only difference between the groups
is the one the researchers are attempting to study.
For example,
when testing a new headache medication,
a large pool of people with headaches
would be randomly divided into two groups,
one receiving the medication
and another receiving a placebo.
With proper randomization,
the only significant overall difference
between the two groups
will be whether or not they received the medication,
rather than other differences that could affect results.
Randomized clinical trials are incredible tools,
and, in fact, the US Food and Drug Administration
often requires at least two to be conducted
before a new drug can be marketed.
But the problem is that an RCT is not possible
in many cases,
either because it’s not practical
or would require too many volunteers.
In such cases,
scientists use an epidemiological study,
which simply observes people going about their usual behavior,
rather than randomly assigning active participants
to control invariable groups.
Let’s say we wanted to study
whether an herbal ingredient on the market
causes nausea.
Rather than deliberately giving people something
that might make them nauseated,
we would find those who already take the ingredient
in their everyday lives.
This group is called the cohort.
We would also need a comparison group
of people who do not have exposure to the ingredient.
And we would then compare statistics.
If the rate of nausea is higher in the herbal cohort,
it suggests an association
between the herbal supplement and nausea.
Epidemiological studies are great tools
to study the health effects of almost anything,
without directly interfering in people’s lives
or assigning them to potentially dangerous exposures.
So, why can’t we rely on these studies
to establish causal relationships
between substances and their effects on health?
The problem is
that even the best conducted epidemiological studies
have inherent flaws.
Precisely because the test subjects
are not randomly assigned to their groups.
For example, if the cohort in our herbal study
consisted of people who took the supplement
for health reasons,
they may have already had higher rates of nausea
than the other people in the sample.
Or the cohort group could’ve been composed of
people who shop at health food stores
and have different diets
or better access to healthcare.
These factors that can affect results,
in addition to the factor being studied,
are known as confounding variables.
These two major pitfalls,
combined with more general dangers,
such as conflicts of interest or selective use of data,
can make the findings of any particular epidemiological study suspect,
and a good study must go out of its way
to prove that its authors have taken steps
to eliminate these types of errors.
But even when this has been done,
the very nature of epidemiological studies,
which examine differences between preexisting groups,
rather than deliberately inducing changes within the same individuals,
means that a single study
can only demonstrate a correlation
between a substance and a health outcome,
rather than a true cause and effect relationship.
At the end of the day,
epidemiological studies have served as excellent guides to public health,
alerting us to critical health hazards,
such as smoking, asbestos, lead, and many more.
But these were demonstrated through
multiple, well-conducted epidemiological studies,
all pointing in the same direction.
So, the next time you see a headline
about a new miracle cure
or the terrible danger posed by an everyday substance,
try to learn more about the original study
and the limitations inherent in any epidemiological study or clinical trial
before jumping to conclusions.