Battling bad science Ben Goldacre

so I’m a doctor but I kind of slipped

sideways into research and now I’m an

epidemiologist and nobody really knows

what epidemiology is epidemiology is the

science of how we know in the real world

if something is good for you or bad for

you and it’s best understood through

example as the science of those crazy

wacky newspaper headlines and these are

just some of the examples these are from

the Daily Mail every country in the

world has a newspaper like this it has

this kind of bizarre ongoing

philosophical project of dividing all

the objects in the world really into the

ones that either cause or prevent cancer

so here are some of the things they’ve

said cause cancer recently divorced

Wi-Fi toiletries and coffee here are

some of the things they say prevent

cancer crushed red-pepper licorice and

coffee so already you can see there are

contradictions here coffee both causes

and prevents cancer and as you start to

read on you can see that maybe there’s

some kind of political valence behind

some of this so for women housework

preventive rest cancer but for men

shopping could make you impotent so we

know that we need to start unpicking the

science behind this and what I hope to

show is that I’m picking dodgy claims

I’m picking the evidence behind dodgy

claims isn’t a kind of nasty carping

activity it’s socially useful but it’s

also a kind of an extremely valuable

explanatory tool because real science is

all about critically appraising the

evidence for somebody else’s position

that’s what happens in academic journals

that’s what happens and academic

conferences the Q&A session after a

postdoc presents data is often a

bloodbath and nobody minds that we

actively welcome it it’s like a kind of

consenting intellectual S&M activity so

what I’m going to show you is all of the

main themes all of the main features of

my discipline evidence-based medicine

and I will talk you through all of these

and demonstrate how they work

exclusively using examples of people

getting stuff wrong so we’ll start with

the absolute weakest form of evidence

known to man and that is Authority in

science

don’t care how many letters you have

after your name in science we want to

know what your reasons are for believing

something how do you know that something

is good for us or bad for us but we’re

also unimpressed by Authority because

it’s so easy to contrive this is

somebody called dr. Gillian McKeith PhD

would you give her full medical title

Gillian McKeith again every country has

somebody like this she is our TV diet

guru she has massive kind of five series

of primetime television giving out very

lavish and exotic health advice she

turns out has a non-accredited

correspondence course PhD from somewhere

in America she also boasts that she’s a

certified professional member of the

American Association of nutritional

consultants which sounds very glamorous

and exciting you get a certificate and

everything this one belongs to my dead

cat Hetty she was a horrible cat you

just go to the website fill out the form

give them $60 and arrives in the post

now that’s not the only reason that we

think this person is an idiot she also

goes on and says things like you should

eat lots of dark green leaves because

they contain lots of chlorophyll and

that will be oxygenate your blood and

anybody who’s done school biology

remembers that chlorophyll in

chloroplasts only makes oxygen in

sunlight and it’s quite dark in your

bowels after you’ve eaten spinach next

we need proper science proper evidence

so red wine can help prevent breast

cancer there’s a headline from The Daily

Telegraph in the UK a glass of red wine

a day could help prevent breast cancer

so you can find this paper and what you

find is it is a real piece of science

it’s a description of the changes in the

behavior of one enzyme when you drip a

chemical extracted from some red grape

skin onto some cancer cells in a dish on

a bench in a laboratory somewhere and

that’s a really useful thing to describe

in a scientific paper but on the

question of your own personal risk of

getting breast cancer if you drink red

wine it tells you absolutely bugger-all

okay actually turns out that your risk

of breast cancer actually increases

slightly with every amount of alcohol

that you drink so what we want is

studies in real human people and here’s

another example this is from Britain’s

leading diet and nutritionist in The

Daily Mirror which is our second biggest

selling newspaper an Australian study in

2001 found that olive oil in combination

with fruits vegetables and pulses offers

measurable protection against skin

wrinkling so then they give the advice

if you eat olive oil and vegetables

you’ll have fewer skin wrinkles and they

very helpfully tell you how to

paper so you go find the paper and what

you find is an observational study right

obviously nobody has ever been able to

go back to like 1930 get all of the

people born in one motor nity unit and

half of them eat lots of fruit and veg

and olive oil and then half of them eat

McDonald’s and then we see how many

wrinkles you’ve got later you have to

take a snapshot of how people are now

and what you find is of course people

are eat fruit and veg and olive oil have

fewer skin wrinkles but that’s because

people are eating fruit and veg in olive

oil they’re freaks okay they’re not

normal they’re like you they come to

events like this right

they are posh they’re wealthy they’re

less likely to have outdoor jobs they’re

less likely to do manual labour they

have better social support they’re less

like to smoke so for a whole host of

fascinating interlocking social

political and cultural reasons they are

less likely to have skin wrinkles that

doesn’t mean that it’s the vegetables or

the olive oil

so ideally what you want to do is a

trial and everybody thinks they’re very

familiar with the idea of a trial trials

are very old the first trials in the

Bible Daniel 1:12 it’s very

straightforward you take a bunch of

people you split them in half you treat

one group one way you treat the other

group the other way and then a little

while later you fold them up and see

what happened to each of them so I’m

going to tell you throughout about one

trial which is probably the most well

reported trial in the UK news media over

the past decade and this is trial of

fish oil pills and the claim was fish

oil pills improved school performance

and behavior in mainstream children and

they said we’ve done a trial all the

previous trials were positive and we

know this one’s going to be - that

should always ring alarm bells right

because if you’ve already know the

answer to your trial you shouldn’t be

doing one either you’ve rigged it by

design or you’ve got enough data so

there’s no need to randomize people

anymore so this is what they were going

to do in their trial they were taking

3,000 children they were going to give

them all these huge fish oil pills six

of them a day and then a year later they

were going to measure their school exam

performance and compare their exam

performance against what they predicted

their exam performance would have been

if they hadn’t had the pills

now can anybody spot a flaw in this

design and no professors of clinical

trial methodology are allowed to answer

this question so there’s no control okay

there’s no control great but that sounds

really techy right that sounds really no

that’s a technical term but the kids got

the pills and then their performance

improved what else could it possibly be

if it wasn’t the pills they got older

okay we all develop over time and of

course also there’s the placebo effect

recibo effect is one of those

fascinating things in the whole of

medicine it’s not just about taking a

pill and your performance and your pain

getting better

it’s about our beliefs and expectations

it’s about the cultural meaning of a

treatment and this has been demonstrated

in a whole raft of fascinating studies

comparing one kind of placebo against

another so we know for example that two

sugar pills a day are a more effective

treatment for getting rid of gastric

ulcers than one sugar pill a day two

sugar pills a day beats one sugar pill a

day and that’s an outrageous and

ridiculous finding but it’s true we know

from three different studies all three

different types of pain that a saltwater

injection is a more effective treatment

for pain than taking a sugar pill taking

a dummy pill that has no medicine in it

not because the injection or the pill do

anything physically to the body but

because an injection feels like a much

more dramatic intervention so we know

that our beliefs and expectations can be

manipulated which is why we do trials

where we control

against a placebo where one half of the

people get the real treatment and the

other half get placebo but that’s not

enough what I’ve just shown you are

examples of the very simple and

straightforward ways that journalists

and food supplement pill peddlers and

naturopaths can distort evidence for

their own purposes what I find really

fascinating is that the pharmaceutical

industry use exactly the same kinds of

tricks and devices but slightly more

sophisticated versions of them in order

to distort the evidence that they give

to doctors and patients in which we use

to make vitally important decisions so

firstly trials against placebo everybody

thinks they know that a trial should be

a comparison of your new drug against

placebo but actually in a lot of

situations that’s wrong because often we

already have a very good treatment that

is currently available so we don’t want

to know that your alternative new

treatment is better than nothing we want

to know that it’s better than the best

currently available treatment that we

have and yet repeatedly you consistently

see people doing trials still against

placebo and you can get licensed to

bring your drug to market with only data

showing that it’s better than nothing

which is useless for a doctor like me

trying to make a decision but that’s not

the only way that you can rig your data

you can also rig your data by can making

the thing that you compare your new drug

against really rubbish you can give the

competing drug in to lower dose so that

people aren’t properly treated you can

give the competing drug in to higher

dose so that people get side effects and

this is exactly what happened with

anti-psychotic medication for

schizophrenia twenty years ago a new

generation of antipsychotic drugs were

brought in and the promise was that they

would have fewer side-effects so people

set about doing trials of these new

drugs against the old drugs but they

gave the old drugs in ridiculously high

doses twenty milligrams a day of

haloperidol and it’s a foregone

conclusion if you give a drug at that

higher dose that it will have more side

effects and that your new drug will look

better

ten years ago history repeated itself

interestingly when risperidone which was

the first of the new generation

anti-psychotic drugs came off copyright

so anybody could make copies everybody

wanted to show that their drug was

better than risperidone so you see a

bunch of trials comparing new

anti-psychotic drugs against risperidone

at 8 milligrams a day again not an

insane dose not an illegal dose but very

much at the high end of normal until

you’re bound to make your new drug look

better and so it’s no surprise that

overall industry-funded trials are four

times more likely to give a positive

result than independently sponsored

trials but and it’s a big but

it turns out when you look at the

methods used by industry-funded trials

that they’re actually better than

independently sponsored trials and yet

they always managed to get the result

that they want so how does this work how

can we explain this strange phenomenon

well it turns out that what happens is

the negative data goes missing in action

it’s withheld from doctors and patients

and this is the most important aspect of

the whole story it’s at the top of the

pyramid of evidence we need to have all

of the data on a particular treatment to

know whether or not it really is

effective and there are two different

ways that you can spot whether some data

has gone missing in action you can use

statistics or you can use stories I

personally prefer statistics so that’s

what I’m going to do first this is

something called a funnel plot and a

funnel plot is a very clever way of

spotting if small negative trials have

disappeared have gone missing in action

so this is a graph of all of the trials

that have been done on a particular

treatment and as you go up towards the

top of the graph what you see is each

dot is a trial and as you got to the top

those are the bigger chance so they’ve

got less error in them so they’re less

likely to be randomly false positives

randomly false negatives so they will

cluster together the big trials are

closer to the true answer then as you go

further down at the bottom what you can

see is over on this side spurious false

negatives and over on this side the

spurious false positives if there is

publication bias if small negative

trials have gone missing action you can

see it on one of these graphs so you can

see here that the small negative trials

that should be on the bottom left have

disappeared this is a graph

demonstrating the presence of

publication bias in studies of

publication bias and I think that’s the

funniest epidemiology joke that you will

ever hear that’s how you can prove it

statistically but what about stories

well they’re heinous they really are

this is a drug called reboxetine and

this is a drug which I myself have

prescribed to patients and I’m a very

nerdy doctor I hope I go out of my way

to try and read and understand all the

literature I read the trials on this

they were all positive they were all

well conducted I found no flaw

unfortunately it turned out that many of

these trials were withheld in fact 76%

of all of the trials that were done on

this drug were withheld from doctors and

patients now if you think about it if I

toss a coin a hundred times and I’m

allowed to withhold from you the answers

half the times

then I can convince you that I have a

coin with two heads okay

if we remove half of the data we can

never know what the true effect size of

these medicines is and this is not an

isolated story around half of all of the

trial data on antidepressants has been

withheld but it goes way beyond that the

Nordic Cochrane group we’re trying to

get hold of the data on that to bring it

all together the Cochrane groups are an

international nonprofit collaboration

that produce systematic reviews of all

of the data that has ever been shown and

they need to have access to all of the

trial data but the companies withheld

that data from them and so did the

European Medicines Agency for three

years this is a problem that is

currently lacking a solution and to show

how big it goes this is a drug called

Tamiflu which governments around the

world have spent billions and billions

of dollars on and they spend that money

on the promise that this is a drug which

will reduce the rate of complications

with flu we already have the data

showing that it reduces the duration of

your flu by a few hours but I don’t

really care about that governments don’t

care about that I’m very sorry if you

have the flu I know it’s horrible but

we’re not going to spend billions of

dollars trying to reduce the duration of

your flu symptoms by half a day we

prescribe these drugs we stockpile them

for emergencies on the understanding

they will reduce the number of

complications which means pneumonia and

which means death the infectious

diseases Cochrane group which are based

in Italy have been trying to get the

full data in a usable form out of the

drug company so that they can make a

full decision about whether this drug is

effective or not and they’ve not been

able to get that information this is

undoubtedly the single biggest ethical

problem facing medicine today we cannot

make decisions in the absence of all of

the information so it’s a little bit

difficult from there to spin in some

kind of positive conclusion

but I would say this I think that

sunlight is the best disinfectant all of

these things are happening in plain

sight and they’re all protected by a

kind of force field of tediousness and I

think with all of the problems in

science one of the best things that we

can do is to lift up the lid finger

around at the mechanics and peirong

thank you very much