Battling bad science Ben Goldacre
so I’m a doctor but I kind of slipped
sideways into research and now I’m an
epidemiologist and nobody really knows
what epidemiology is epidemiology is the
science of how we know in the real world
if something is good for you or bad for
you and it’s best understood through
example as the science of those crazy
wacky newspaper headlines and these are
just some of the examples these are from
the Daily Mail every country in the
world has a newspaper like this it has
this kind of bizarre ongoing
philosophical project of dividing all
the objects in the world really into the
ones that either cause or prevent cancer
so here are some of the things they’ve
said cause cancer recently divorced
Wi-Fi toiletries and coffee here are
some of the things they say prevent
cancer crushed red-pepper licorice and
coffee so already you can see there are
contradictions here coffee both causes
and prevents cancer and as you start to
read on you can see that maybe there’s
some kind of political valence behind
some of this so for women housework
preventive rest cancer but for men
shopping could make you impotent so we
know that we need to start unpicking the
science behind this and what I hope to
show is that I’m picking dodgy claims
I’m picking the evidence behind dodgy
claims isn’t a kind of nasty carping
activity it’s socially useful but it’s
also a kind of an extremely valuable
explanatory tool because real science is
all about critically appraising the
evidence for somebody else’s position
that’s what happens in academic journals
that’s what happens and academic
conferences the Q&A session after a
postdoc presents data is often a
bloodbath and nobody minds that we
actively welcome it it’s like a kind of
consenting intellectual S&M activity so
what I’m going to show you is all of the
main themes all of the main features of
my discipline evidence-based medicine
and I will talk you through all of these
and demonstrate how they work
exclusively using examples of people
getting stuff wrong so we’ll start with
the absolute weakest form of evidence
known to man and that is Authority in
science
don’t care how many letters you have
after your name in science we want to
know what your reasons are for believing
something how do you know that something
is good for us or bad for us but we’re
also unimpressed by Authority because
it’s so easy to contrive this is
somebody called dr. Gillian McKeith PhD
would you give her full medical title
Gillian McKeith again every country has
somebody like this she is our TV diet
guru she has massive kind of five series
of primetime television giving out very
lavish and exotic health advice she
turns out has a non-accredited
correspondence course PhD from somewhere
in America she also boasts that she’s a
certified professional member of the
American Association of nutritional
consultants which sounds very glamorous
and exciting you get a certificate and
everything this one belongs to my dead
cat Hetty she was a horrible cat you
just go to the website fill out the form
give them $60 and arrives in the post
now that’s not the only reason that we
think this person is an idiot she also
goes on and says things like you should
eat lots of dark green leaves because
they contain lots of chlorophyll and
that will be oxygenate your blood and
anybody who’s done school biology
remembers that chlorophyll in
chloroplasts only makes oxygen in
sunlight and it’s quite dark in your
bowels after you’ve eaten spinach next
we need proper science proper evidence
so red wine can help prevent breast
cancer there’s a headline from The Daily
Telegraph in the UK a glass of red wine
a day could help prevent breast cancer
so you can find this paper and what you
find is it is a real piece of science
it’s a description of the changes in the
behavior of one enzyme when you drip a
chemical extracted from some red grape
skin onto some cancer cells in a dish on
a bench in a laboratory somewhere and
that’s a really useful thing to describe
in a scientific paper but on the
question of your own personal risk of
getting breast cancer if you drink red
wine it tells you absolutely bugger-all
okay actually turns out that your risk
of breast cancer actually increases
slightly with every amount of alcohol
that you drink so what we want is
studies in real human people and here’s
another example this is from Britain’s
leading diet and nutritionist in The
Daily Mirror which is our second biggest
selling newspaper an Australian study in
2001 found that olive oil in combination
with fruits vegetables and pulses offers
measurable protection against skin
wrinkling so then they give the advice
if you eat olive oil and vegetables
you’ll have fewer skin wrinkles and they
very helpfully tell you how to
paper so you go find the paper and what
you find is an observational study right
obviously nobody has ever been able to
go back to like 1930 get all of the
people born in one motor nity unit and
half of them eat lots of fruit and veg
and olive oil and then half of them eat
McDonald’s and then we see how many
wrinkles you’ve got later you have to
take a snapshot of how people are now
and what you find is of course people
are eat fruit and veg and olive oil have
fewer skin wrinkles but that’s because
people are eating fruit and veg in olive
oil they’re freaks okay they’re not
normal they’re like you they come to
events like this right
they are posh they’re wealthy they’re
less likely to have outdoor jobs they’re
less likely to do manual labour they
have better social support they’re less
like to smoke so for a whole host of
fascinating interlocking social
political and cultural reasons they are
less likely to have skin wrinkles that
doesn’t mean that it’s the vegetables or
the olive oil
so ideally what you want to do is a
trial and everybody thinks they’re very
familiar with the idea of a trial trials
are very old the first trials in the
Bible Daniel 1:12 it’s very
straightforward you take a bunch of
people you split them in half you treat
one group one way you treat the other
group the other way and then a little
while later you fold them up and see
what happened to each of them so I’m
going to tell you throughout about one
trial which is probably the most well
reported trial in the UK news media over
the past decade and this is trial of
fish oil pills and the claim was fish
oil pills improved school performance
and behavior in mainstream children and
they said we’ve done a trial all the
previous trials were positive and we
know this one’s going to be - that
should always ring alarm bells right
because if you’ve already know the
answer to your trial you shouldn’t be
doing one either you’ve rigged it by
design or you’ve got enough data so
there’s no need to randomize people
anymore so this is what they were going
to do in their trial they were taking
3,000 children they were going to give
them all these huge fish oil pills six
of them a day and then a year later they
were going to measure their school exam
performance and compare their exam
performance against what they predicted
their exam performance would have been
if they hadn’t had the pills
now can anybody spot a flaw in this
design and no professors of clinical
trial methodology are allowed to answer
this question so there’s no control okay
there’s no control great but that sounds
really techy right that sounds really no
that’s a technical term but the kids got
the pills and then their performance
improved what else could it possibly be
if it wasn’t the pills they got older
okay we all develop over time and of
course also there’s the placebo effect
recibo effect is one of those
fascinating things in the whole of
medicine it’s not just about taking a
pill and your performance and your pain
getting better
it’s about our beliefs and expectations
it’s about the cultural meaning of a
treatment and this has been demonstrated
in a whole raft of fascinating studies
comparing one kind of placebo against
another so we know for example that two
sugar pills a day are a more effective
treatment for getting rid of gastric
ulcers than one sugar pill a day two
sugar pills a day beats one sugar pill a
day and that’s an outrageous and
ridiculous finding but it’s true we know
from three different studies all three
different types of pain that a saltwater
injection is a more effective treatment
for pain than taking a sugar pill taking
a dummy pill that has no medicine in it
not because the injection or the pill do
anything physically to the body but
because an injection feels like a much
more dramatic intervention so we know
that our beliefs and expectations can be
manipulated which is why we do trials
where we control
against a placebo where one half of the
people get the real treatment and the
other half get placebo but that’s not
enough what I’ve just shown you are
examples of the very simple and
straightforward ways that journalists
and food supplement pill peddlers and
naturopaths can distort evidence for
their own purposes what I find really
fascinating is that the pharmaceutical
industry use exactly the same kinds of
tricks and devices but slightly more
sophisticated versions of them in order
to distort the evidence that they give
to doctors and patients in which we use
to make vitally important decisions so
firstly trials against placebo everybody
thinks they know that a trial should be
a comparison of your new drug against
placebo but actually in a lot of
situations that’s wrong because often we
already have a very good treatment that
is currently available so we don’t want
to know that your alternative new
treatment is better than nothing we want
to know that it’s better than the best
currently available treatment that we
have and yet repeatedly you consistently
see people doing trials still against
placebo and you can get licensed to
bring your drug to market with only data
showing that it’s better than nothing
which is useless for a doctor like me
trying to make a decision but that’s not
the only way that you can rig your data
you can also rig your data by can making
the thing that you compare your new drug
against really rubbish you can give the
competing drug in to lower dose so that
people aren’t properly treated you can
give the competing drug in to higher
dose so that people get side effects and
this is exactly what happened with
anti-psychotic medication for
schizophrenia twenty years ago a new
generation of antipsychotic drugs were
brought in and the promise was that they
would have fewer side-effects so people
set about doing trials of these new
drugs against the old drugs but they
gave the old drugs in ridiculously high
doses twenty milligrams a day of
haloperidol and it’s a foregone
conclusion if you give a drug at that
higher dose that it will have more side
effects and that your new drug will look
better
ten years ago history repeated itself
interestingly when risperidone which was
the first of the new generation
anti-psychotic drugs came off copyright
so anybody could make copies everybody
wanted to show that their drug was
better than risperidone so you see a
bunch of trials comparing new
anti-psychotic drugs against risperidone
at 8 milligrams a day again not an
insane dose not an illegal dose but very
much at the high end of normal until
you’re bound to make your new drug look
better and so it’s no surprise that
overall industry-funded trials are four
times more likely to give a positive
result than independently sponsored
trials but and it’s a big but
it turns out when you look at the
methods used by industry-funded trials
that they’re actually better than
independently sponsored trials and yet
they always managed to get the result
that they want so how does this work how
can we explain this strange phenomenon
well it turns out that what happens is
the negative data goes missing in action
it’s withheld from doctors and patients
and this is the most important aspect of
the whole story it’s at the top of the
pyramid of evidence we need to have all
of the data on a particular treatment to
know whether or not it really is
effective and there are two different
ways that you can spot whether some data
has gone missing in action you can use
statistics or you can use stories I
personally prefer statistics so that’s
what I’m going to do first this is
something called a funnel plot and a
funnel plot is a very clever way of
spotting if small negative trials have
disappeared have gone missing in action
so this is a graph of all of the trials
that have been done on a particular
treatment and as you go up towards the
top of the graph what you see is each
dot is a trial and as you got to the top
those are the bigger chance so they’ve
got less error in them so they’re less
likely to be randomly false positives
randomly false negatives so they will
cluster together the big trials are
closer to the true answer then as you go
further down at the bottom what you can
see is over on this side spurious false
negatives and over on this side the
spurious false positives if there is
publication bias if small negative
trials have gone missing action you can
see it on one of these graphs so you can
see here that the small negative trials
that should be on the bottom left have
disappeared this is a graph
demonstrating the presence of
publication bias in studies of
publication bias and I think that’s the
funniest epidemiology joke that you will
ever hear that’s how you can prove it
statistically but what about stories
well they’re heinous they really are
this is a drug called reboxetine and
this is a drug which I myself have
prescribed to patients and I’m a very
nerdy doctor I hope I go out of my way
to try and read and understand all the
literature I read the trials on this
they were all positive they were all
well conducted I found no flaw
unfortunately it turned out that many of
these trials were withheld in fact 76%
of all of the trials that were done on
this drug were withheld from doctors and
patients now if you think about it if I
toss a coin a hundred times and I’m
allowed to withhold from you the answers
half the times
then I can convince you that I have a
coin with two heads okay
if we remove half of the data we can
never know what the true effect size of
these medicines is and this is not an
isolated story around half of all of the
trial data on antidepressants has been
withheld but it goes way beyond that the
Nordic Cochrane group we’re trying to
get hold of the data on that to bring it
all together the Cochrane groups are an
international nonprofit collaboration
that produce systematic reviews of all
of the data that has ever been shown and
they need to have access to all of the
trial data but the companies withheld
that data from them and so did the
European Medicines Agency for three
years this is a problem that is
currently lacking a solution and to show
how big it goes this is a drug called
Tamiflu which governments around the
world have spent billions and billions
of dollars on and they spend that money
on the promise that this is a drug which
will reduce the rate of complications
with flu we already have the data
showing that it reduces the duration of
your flu by a few hours but I don’t
really care about that governments don’t
care about that I’m very sorry if you
have the flu I know it’s horrible but
we’re not going to spend billions of
dollars trying to reduce the duration of
your flu symptoms by half a day we
prescribe these drugs we stockpile them
for emergencies on the understanding
they will reduce the number of
complications which means pneumonia and
which means death the infectious
diseases Cochrane group which are based
in Italy have been trying to get the
full data in a usable form out of the
drug company so that they can make a
full decision about whether this drug is
effective or not and they’ve not been
able to get that information this is
undoubtedly the single biggest ethical
problem facing medicine today we cannot
make decisions in the absence of all of
the information so it’s a little bit
difficult from there to spin in some
kind of positive conclusion
but I would say this I think that
sunlight is the best disinfectant all of
these things are happening in plain
sight and they’re all protected by a
kind of force field of tediousness and I
think with all of the problems in
science one of the best things that we
can do is to lift up the lid finger
around at the mechanics and peirong
thank you very much