The science doesnt lie Cognitive contamination in forensics
Transcriber: Paulina Kaniewska
Reviewer: Rhonda Jacobs
When I was pregnant with my son,
I was diagnosed with gestational diabetes,
and that meant
a lot of extra tests and scans,
and checking my blood sugars
four times a day.
But the funny thing was
every time I got a new test result,
or a data point, if you like,
it seemed to show
that I wasn’t a diabetic.
But when I raised this
with the doctor at the hospital,
he was adamant that I was.
A few months went by, and I asked
for another diagnostic test,
and it was negative.
So I went to the hospital
and asked to speak with the doctor.
And I’m sitting in the doctor’s office,
waiting for him to come in,
bracing myself.
If you’ve ever watched
a modern day murder mystery,
you’ll know that forensic scientists
go to great lengths
to prevent their evidence
becoming contaminated.
You’ll have seen them wearing the suits,
and the booties, and the gloves.
And we wear those things
to stop our physical evidence,
things like DNA swabs,
fingerprints, shoe marks,
coming into contact with people or things
that aren’t relevant to the case.
Now, I’m a forensic scientist,
but today I want to talk about
a totally different type of contamination,
and that’s cognitive contamination.
And what that means
is how our scientific thinking
can become contaminated
by details about the case that we learn.
Of course, we all love to think
that we’re purely objective,
and we always make
a 100 percent scientific decision.
But sometimes things
that have nothing to do do with science
can creep in to our decision-making,
and we’re not even aware
that it’s happened.
My area of expertise
is forensic toxicology,
and that means I’m interested
in drugs and alcohol,
well, professionally interested
in drugs and alcohol,
and the role they play
in deaths and crimes.
And a really common type of case
a forensic toxicologist might deal with
is driving under the influence of drugs.
In those cases, typically, we’re sent
a single blood sample from a driver
and asked to test it.
So one day one of these samples
arrives in my lab,
and the police officer in charge
of the case phones me up and says:
“We’ve already caught
this 21-year-old male driver nine times
for driving around on drugs,
so we’re pretty sure he’s guilty,
and the previous nine times
he’d taken methamphetamine,
which you may know as crystal meth.
So the first step in any tox case
is to decide which test to do.
Despite what you might have seen on CSI,
there is no one amazing “magic machine”
that can test for every drug
and poison ever.
There is usually some element
of picking and choosing a test,
and we want to choose the test
that will be most useful for the case.
So, which test do you think
would be most useful on this case?
Well, some of you
might be sitting there thinking:
“Your job sounds pretty easy.
Just test the blood for methamphetamine.”
And that is the most obvious decision.
But it’s also a biased decision
because it’s been influenced
by what I’ve been told
about the previous behavior of the driver.
And we call this
“expected frequency bias.”
And it’s the reason jurors are not told
about previous convictions.
But even if I know nothing
about the previous behavior
of the driver,
just knowing a few personal details
can lead to the same kind
of biased thinking.
Toxicologists will sometimes choose
tests based on assumptions
about the sorts of people who use drugs.
Assumptions based on age, gender,
ethnicity, even sexuality.
And the problem with these assumptions
is that they are just that,
they are just guesses,
and they’re very often not based
on anything particularly scientific.
And the danger in using them
is that it’s really easy
to miss something.
I could just look for
methamphetamine in this case.
But what if that result is negative?
What if the driver’s taken
something else, and I miss it
because I don’t look properly,
because I’ve been swayed
by what the police have told me.
So I need to be more open-minded,
and I decide to do a screening test
that, yes, can see methamphetamine,
but can also see other drugs,
like cannabis and cocaine.
A few days later,
I get my test results back,
and they’re positive for methamphetamine.
“Surprise, surprise,”
you might be thinking,
but remember I said
it was a screening test.
What that means is that all I have
is a simple “yes” or “no,”
and the result is preliminary.
It’s very like
a lateral flow test for COVID-19.
The negatives are OK, but the positives
need to be confirmed by another method.
In forensic toxicology,
we call that method “mass spectrometry.”
And just like your PCR test for COVID-19,
it takes longer to do
and it’s more expensive.
And it also gives us
a more complex answer.
So instead of that simple “yes” or “no,”
we get a picture,
a picture on a computer screen,
and it’s my job as a toxicologist
to compare that picture to another one
and do a sort of a “spot the difference.”
So let’s look at our first picture,
and we call this “a mass spectrum.”
Now, this is a picture
of methamphetamine powder,
something I know
is definitely methamphetamine.
You can see in the picture
we have three vertical lines,
and each of those lines
represents a number.
On the left - 91, in the middle - 119,
on the right - 150.
Those three numbers together
tell me that this is methamphetamine.
There is another important detail,
and that is how tall each of the lines
is compared to each other.
You can see that the tallest is actually
the one in the middle, the 119.
And the next tallest
is the one on the right, 150.
Shortest is the 91.
Now, let’s look at our case.
Hopefully, you can see straight away
that there are some extra lines in here
that are not in the methamphetamine.
So we’ve got a really big one there at 88,
another one at 60, and so on.
What do those lines mean?
Are they important?
Are they just some rubbish
that I can ignore?
If you’re eagle-eyed,
you will have also seen
the ratio between the lines isn’t right,
but that 150 out there on the right -
that is shorter than it should be.
But yet, I do have my three numbers.
And this is the problem
with real case samples.
They’re messy and hard to interpret.
And this is what we call
in technical terms “an iffy match.”
And when results are iffy,
that’s when that case information
can start to creep in.
I think that this sample comes from
a 21-year-old male driver
who’s already been caught
using methamphetamine nine times.
And knowing that
I might subconsciously upgrade this
from “iffy” to “definitely positive.”
“Yes, OK, there are some
extra lines in there,
but they’re probably
just some rubbish that I can ignore,
and I have all three numbers.
I’m going to say it’s positive.”
Now, what if I think this is being taken
from a 92-year-old grandma in a rest home?
I might subconsciously downgrade this
from “iffy” to “definitely negative.”
“What about extra lines at the start?
I really don’t think I can ignore those.
And the ratio between the lines
is definitely off.
I’m going to say it’s negative.”
And if I did either of those two things,
it would be called “contextual bias”.
The fact is that when the exact same data
is interpreted as positive
for a 21-year-old male
but negative for a 92-year-old female,
that’s a real problem
for the criminal justice system.
It’s also important to know
that not all of the information
I get is correct.
People don’t always tell the police
the truth in the aftermath of a crime.
Some of the stuff I’m told
is basically just hearsay
that would never be allowed in court,
but yet it’s allowed to influence
my scientific decision-making.
So I phone the police officer,
and I give him the news.
“I can’t be sure
there is methamphetamine
in this sample.”
Now, that doesn’t mean it isn’t there.
It doesn’t mean the driver didn’t do it.
What it means is there isn’t
enough scientific proof to say
there is methamphetaminein the sample.
That might not seem to you
like the best outcome
for the police or the prosecution,
but it’s the one that’s based on science,
not the one that’s based on
what I’ve been told about the driver.
In the title of my talk,
I mention a battle,
and there are many battles going on
in forensic science right now
over these very issues.
There’s a particularly fierce one
raging in forensic pathology right now.
And the pathologist works very closely
with the toxicologist
because it’s their job
to work out the cause
and sometimes the manner of death.
But in my field,
colleagues will say to me:
“We use machines to get our data.
And machines are objective, right?
So how could our results be biased?”
But as you’ve seen today, the problem
exists between the keyboard and the chair.
Yes, machines give us objective data,
but it’s we humans with all our biases
who decide what that data means.
My colleagues will also say to me:
“I’ll just forget what I’ve been told.
I just won’t use that when I come
to make a scientific decision.”
Sadly, we just don’t have that kind
of level of control over our subconscious.
So let’s go back to that doctor’s office
where I’m sitting, waiting,
heavily pregnant.
On the one hand,
I have a single positive
result for diabetes;
on the other, I have a mountain
of negative data.
Then something
I was not expecting happened,
and a new doctor walked into the room.
And she looked at the data,
and then she looked at me and said:
“This is a misdiagnosis.
You don’t have gestational diabetes.”
And I was so relieved,
but it got me thinking.
“What was happening here?
Why were these two medical professionals
looking at the same data
and coming to opposite conclusions?”
Or, to put it another way:
“What was stopping the first doctor
seeing the mountain of negative data?”
And what was stopping him
was that he knew something about me
that the second doctor didn’t.
He knew that diabetes runs in my family.
And because he knew that,
he couldn’t forget it.
And he was convinced that because others
in my family have had diabetes,
I must have it too.
And this is called “confirmation bias,”
and it’s the last piece
of the cognitive bias puzzle.
Because once we learn something
about a patient or a case,
we can’t unlearn it,
and we start to look for data or evidence
that is consistent with the thing
that we’ve learned.
And we start to ignore evidence
that is not consistent
with the thing that we’ve learned.
So I want to leave you
with a question now.
Given what we know
about how cognitive contamination
can affect forensic toxicology,
isn’t it about time we started
protecting our evidence from it?
Thank you.
(Applause)