What doctors dont know about the drugs they prescribe Ben Goldacre
hi uh so um this chap here he thinks he
can tell you the future his name is
Nostradamus although here the son have
made him look a little bit like Sean
Connery and like most of you I suspect I
don’t really believe that people can
sense the future I don’t believe in
precognition and every now and then you
hear that somebody has been able to
predict something that happened in the
future and that’s probably because it
was a fluke and we only hear about the
flukes and about the freaks we don’t
hear about all the times that people got
stuff wrong now he expects that to
happen with silly stories about
precognition but the problem is we have
exactly the same problem in academia and
in medicine and in this environment it
costs lives so firstly thinking just
about precognition as it turns out just
last year a researcher called Daryl BEM
conducted a piece of research where he
found evidence of precognitive powers in
undergraduate students and this was
published in a peer-reviewed academic
journal and most of the people who read
this just said okay well fair enough but
I think that’s a fluke that’s a freak
because I know that if I did a study
where I found no evidence that
undergraduate students had precognitive
powers it probably wouldn’t get
published in a journal and in fact we
know that that’s true
because several different groups of
research scientists tried to replicate
the findings of this precognition study
and when they submitted it to the exact
same journal the journal said no we’re
not interested in publishing replication
we’re not interested in your negative
data so this is already evidence of how
in the academic literature we will see a
biased sample of the true picture of all
of the scientific studies that have been
conducted but it doesn’t just happen in
the dry academic field of psychology it
also happens in for example cancer
research so in March 2012 just one month
ago some researchers reported in the
journal Nature how they had tried to
replicate 53 different basic science
studies looking at potential treatment
targets in cancer and out of those 53
studies they were only able to success
we replicate 647 out of those 53 were
unlockable and they say in their
discussion that this is very likely
because freaks get published people will
do lots and lots and lots of different
studies and the occasions when it works
they will publish in the ones where it
doesn’t work they won’t and their first
recommendation of how to fix this
problem because it is a problem because
it sends us all down blind alleys their
first recommendation of how to fix this
problem is to make it easier to publish
negative results in science and to
change the incentives so that scientists
are encouraged to post more of their
negative results in public but it
doesn’t just happen in the very dry
world of preclinical basic science
cancer research it also happens in the
very real flesh and blood of academic
medicine so in 1980 some researchers did
a study on a drug called Lorcan ID and
this was an anti-arrhythmic drug a drug
that suppresses abnormal heart rhythms
and the idea was after people have had a
heart attack they’re quite likely to
have abnormal heart rhythms so if we
give them a drug that suppresses
abnormal heart rhythms this will
increase the chances of them surviving
early on in its development they did a
very small trial just under a hundred
patients 50 patients got Lorcan ID and
of those patients ten died another 50
patients got a dummy placebo sugar pill
with no active ingredient and only one
of them died so they rightly regarded
this drug as a failure and it’s
commercial development was stopped and
because it’s commercial development was
stopped this trial was never published
unfortunately over the course of the
next 5-10 years other companies had the
same idea about drugs that would prevent
arrhythmias in people who’ve had heart
attacks these drugs were brought to
market they were prescribed very widely
because heart attacks are a very common
thing and it took so long for us to find
out that these drugs also caused an
increased rate of death but before we
detected that safety signal over a
hundred thousand people died
unnecessarily in America from the
prescription of anti-arrhythmic drugs
now actually in 1993
the researchers who did that 1980 study
that early study published a mayor culpa
an apology to the scientific community
in which they said when we carried out
our study in 1980 we thought that the
increased death rate that occurred in
the law an ID group was an effective
chance the development of local ID was
abandoned for commercial reasons and
this study was never published it’s now
a good example of publication bias
that’s the technical term for the
phenomenon where unflattering data gets
gets lost gets unpublished it is left
missing in action and they say the
results described here might have
provided an early warning of trouble
ahead now these are stories from basic
science these are stories from twenty
thirty years ago the academic publishing
environment is very different now there
are academic journals like trials the
open access journal which will publish
any trial conducted in humans regardless
of whether it has a positive or a
negative result but this problem of
negative results that go missing in
action is still very prevalent in fact
it’s so prevalent that it cuts to the
core of evidence-based medicine so this
is a drug called reboxetine and this is
a drunk that I myself have prescribed
it’s an antidepressant and I’m a very
nerdy doctor so I read all of the
studies that I could on this drug I read
the one study that was published that
showed that reboxetine was better than
placebo and I read the other three
studies that were published that showed
that reboxetine was just as good as any
other antidepressant and because this
patient hadn’t done well on those other
antidepressants I thought were robots
teams just as good it’s one to try but
it turned out that I was misled in fact
seven trials were conducted comparing
the Box 13 against a dummy placebo sugar
pill one of them was positive and that
was published but six of them were
negative and they were left unpublished
three trials were published comparing
reboxetine against other antidepressants
in which reboxetine was just as good and
they were published but three times as
many patients worth of data was
collected which showed that reboxetine
was worse than those other treatments
and those trials were not published
I felt misled now you might say well
that’s an extreme and unusual example
and I wouldn’t want to
guilty of the same kind of
cherry-picking and selectively
referencing that I’m accusing other
people of but it turns out that this
phenomenon of publication bias has
actually been very very well studied so
here is one example of how you approach
it the classic model is you get a bunch
of studies where you know that they’ve
been conducted and completed and then
you go and see if they’ve been published
anywhere in the academic literature so
this took all of the trials that have
ever been conducted on antidepressants
that were approved over a 15-year period
by the FDA they took all of the trials
which was submitted to the FDA as part
of the approval package so that’s not
all of the trials that were ever
conducted on these drugs because we can
never know if we have those but it is
the ones that were conducted in order to
get the marketing authorization and then
they went to see if these trials had
been published in the peer-reviewed
academic literature and this is what
they found it was pretty much a 50/50
split half of these trials were positive
half of them were negative in reality
but when they went to look for these
trials in the peer-reviewed academic
literature what they found was a very
different picture only three of the
negative trials were published but all
but one of the positive trials were
published now if we just flick back and
forth between those two you can see what
a staggering difference there was
between reality and what doctors
patients commissioners of health
services and academics were able to see
in the peer-reviewed academic literature
we were misled and this is a systematic
flaw in the core of medicine in fact
there have been so many studies
conducted on publication bias now over a
hundred that they’ve been collected in a
systematic review published in 2010 that
took every single study on publication
that bias but they could find
publication bias affects every field of
medicine about half of all trials on
average go missing in action and we know
that positive findings are around twice
as likely to be published as negative
findings this is a cancer at the core of
evidence-based medicine if I flipped a
coin a hundred times but then withheld
the results from you from half of those
tosses I can make it look as if I had a
coin that always came up heads but that
wouldn’t mean that
a two-headed coin that would mean that I
was a chancer and you were an idiot for
letting me get away with it but this is
exactly what we blindly tolerate in the
whole of evidence-based medicine and to
me this is research misconduct if I
conducted one study and I withheld half
of the data points from that one study
you would rightly accuse me essentially
of research fraud and yet for some
reason if somebody conducts ten studies
but only publishes the five that give
the result that they want we don’t
consider that to be research misconduct
and when that responsibility is diffused
between a whole network of researchers
academics industry sponsors journal
editors for some reason we find it more
acceptable but the effect on patients is
damning and this is happening right now
today this is the drug called Tamiflu
tell if Louise a drug which governments
around the world have spent billions and
billions of dollars on stockpiling and
we’ve stockpiled Tamiflu in panic in the
belief that it will reduce the rate of
complications of influenza complications
is a medical euphemism for pneumonia and
death
now when the Cochrane systematic
reviewers were trying to collect
together all of the data from all of the
trials that had ever been conducted on
whether Tamiflu actually did this or not
they found that several of those trials
were unpublished the results were
unavailable to them and when they
started obtaining the write-ups of those
trials through various different means
through Freedom of Information Act
requests through harassing various
different organizations what they found
was inconsistent and when they tried to
get hold of the clinical study reports
the 10,000 page long documents that have
the that the best possible rendition of
the information they were told they
weren’t allowed to have them and if you
want to read the full correspondence and
the excuses and the explanations given
by the drug company you can see that
written up in this week’s edition of
PLoS Medicine and the most staggering
thing of all of this
to me is that not only is this a problem
not only do we recognize that this is a
problem but we’ve had to suffer fake
fixes we’ve had people pretend that this
is a problem that’s been fixed first of
all we had trials registers and
everybody said oh it’s okay we’ll get
everyone to register their trials
whether they’ll post the protocol
they’ll say what they’re going to do
before they do it
and then afterwards we’ll be able to
check and see if all the trials which
have been conducted and completed have
been published but people didn’t bother
to use those registers and so then the
International Committee of medical
journal editors came along and they said
oh well we will hold the line we weren’t
publish any journals we weren’t
published any trials unless they’ve been
registered before they began but they
didn’t hold the line in 2008 a study was
conducted which showed that half of all
of trials published by journals edited
by members of the ICM je weren’t
properly registered and a quarter of
them weren’t registered at all and then
finally the FDA Amendment Act was passed
a couple of years ago say that everybody
who conducts a trial must post the
results of that trial within one year
and in the BMJ in the first edition of
January 2012 you can see a study which
looks to see if people kept to that
ruling and it turns out that only one in
five have done so this is a disaster we
cannot know the true effects of the
medicines that we prescribe if we do not
have access to all of the information
and this is not a difficult problem to
fix we need to force people to publish
all trials conducted in humans including
the older trance because the FDA
Amendment Act only asks that you publish
the trials conducted after 2008 and I
don’t know what world it is in which
were only practicing medicine on the
basis of trials that completed in the
past two years
we need to publish all trials in humans
including the older trials for all drugs
in current use and you need to tell
everyone you know that this is a problem
and that it has not been fixed thank you
very much
you