Visualizing the medical data explosion Anders Ynnerman
I will start by posting a little bit of
a challenge the challenge of dealing
with data data that we have to deal with
in medical situations it’s really a huge
challenge for us and this is our beast
of burden this is a computer tomography
machine CT machine it’s a fantastic
device it uses x-rays x-ray beams that
are rotating very fast around the human
body it takes about 30 seconds to go
through the whole machine and is
generating enormous amounts of
information that comes out of the
machine so this is a fantastic machine
that we can use for improving healthcare
but as I said it’s also a challenge for
us and the challenge is really found in
this picture here it’s the medical data
explosion that we’re having right now
we’re facing this problem and let me
step back in time let’s just go back a
few years in time and see what happened
back then these machines that came out
they started coming in the 1970s they
would scan off human bodies and they
would generate about a hundred images of
the human body and I’ve taken the
liberty just for clarity to translate
that to data sizes that would correspond
to about 15 megabytes of data which is
small when you think about the data that
we can handle today just on normal
mobile devices if you translate that to
phone books it’s about one litre of
phone books in the pile looking at what
we’re doing today with these machines
that we have we can just in a few
seconds get 24,000 images out of the
body and that will correspond to about
20 gigabytes of data or 800 phone books
and the pod will then be 200 meters of
phone books what’s about to happen and
we’re seeing this it’s beginning a
technology trend that’s happening right
now is that we’re starting to look at
time-resolved situations as well so
we’re getting the dynamics out of the
body as well and just assume that we
will be collecting data during five
seconds and that will correspond to
about one terabyte of data that’s
800,000 books and 16 kilometres of phone
books there’s one patient one data set
and this is what we have to deal with so
this is really the enormous challenge
that we have
today this is 25,000 images you know
imagine that the days when we had
radiologists
doing this they would put up 25,000
images that would go like this 25 there
there is the problem right they can’t do
that anymore that’s impossible so we
have to do something that’s a little bit
more intelligent than doing this okay so
what we do is that we put all these
slices together
imagine that you slice your body in all
these directions and then you try to put
the slices back together again into a
pile of data into a block of data so
this is really what we’re doing so this
gigabyte a terabyte of data we’re
putting it into this block but of course
the block of data just contains the
amount of x-ray that’s been absorbed in
each point in the human body so what we
need to do is to figure out a way of
looking at the things that we do want to
look at and make things transparent that
we don’t want to look at so transforming
the data set into something that looks
like this
and this is a challenge this is a huge
challenge for us to do that using
computers even though they’re getting
faster and better all the time it’s a
challenge to deal with gigabytes of data
or terabytes of data and extracting the
relevant information I want to look at
the heart I want to look at the blood
vessels I want to look at the liver
maybe even finally the tumor in some
cases okay so this is where this little
dude comes into play this is my daughter
this is a as of 9:00 a.m. this morning
she’s playing a computer game she’s only
2 years old and she’s having a blast so
she’s really the the driving force
behind the development of graphics
processing units as long as kids are
playing computer games graphics is
getting better and better and better so
please go back home tell your kids to
play more games because that’s what I
need so what’s inside of this machine is
what enables me to do the things that
I’m doing with the medical data so
really what I’m doing is using these
fantastic little devices and you know
going back maybe 10 years in time when I
got the funding to buy my first graphics
computer
it was a huge machine it was cabinets of
processors and storage and everything I
paid about 1 million dollars for that
machine that machine is today about as
fast as my iPhone
so every month there are new graphics
cards coming out and here’s a few of the
latest ones from the vendors and Nvidia
ATI Intel is out there as well and you
know for a few hundred bucks you can get
these things and put them into your
computer and you can do fantastic things
with these graphics cards so this is
really what’s enabling us to deal with
the explosion of data in medicine
together with some really nifty work in
terms of algorithms compressing data
extracting the relevant information that
people are doing research on so I’m
going to show you a few examples of what
we can do this is the data set that was
captured using a CT scanner you can see
this is a full data it’s a it’s a woman
you can see the hair you can see the
individual structures and of the woman
you can see that there is scattering or
x-rays on the teeth the metal in the
teeth that’s that’s what those artifacts
are coming from but fully interactively
on standard graphics cards I on a normal
computer I can just put in a clip plane
and of course all the data is inside so
I can start rotating I can look at it
from different angles and I can see that
this woman had a problem there was a she
had a bleeding up in the brain and
that’s been fixed with a little stent a
metal clamp that’s tightening up the
vessel and just by changing the
functions and I can decide what’s going
to be transparent and what’s going to be
visible I can look at the skull
structure and I can see that okay this
is where they open up the skull on this
woman and that’s where they went in so
these are fantastic images they’re
really high resolution and they’re
really showing us what we can do with
with standard graphics cards today
now we have really made use of this and
we have tried to squeeze a lot of data
into the system and one of the
applications that we’ve been working on
and this has gotten a little bit of
traction worldwide is the application of
virtual autopsies so again looking at
very very large datasets and you saw
those full-body scans that we can do
we’re just pushing the body through the
whole CT scanner and just in a few
seconds we can get a full-body data set
so so this is from a virtual autopsy and
you can see how I’m gradually peeling
off first you saw the body bag that the
body came in that I’m peeling off the
skin you can see the muscles and
eventually
you can see the bone structure of this
woman now at this point I would also
like to emphasize that with the greatest
respect for the people that I’m now
going to show I’m going to show you a
few cases of virtual autopsies so it’s a
great respect for the people that have
died under violent circumstances that
I’m showing these pictures to you in the
forensic case and this is something that
there’s been approximately 400 cases so
far just in the part of Sweden that I
come from that has been undergoing
virtual autopsies in the past four years
so this will be the typical workflow
situation that police would decide in
the evening when there’s a case coming
in they would decide okay this is the
case where we need to do an autopsy so
in the morning between 6:00 and 7:00 in
the morning the board is then
transported inside of the body bag to
our Center and is being scanned through
one of the CT scanners and then the
radiologist together with the
pathologist and sometimes the forensic
scientist looks at the data that’s
coming out and they have a joint session
and then they decide what to do in the
real physical autopsy after that now
looking at a few cases here’s one of the
first cases that we had you can really
see the details of the data set it’s
very high resolution and it’s our
algorithms that allow us to zoom in on
all the details
and again it’s fully interactive so you
can rotate and you can look at things in
real time all these systems here without
saying too much about this case this is
a traffic accident and a drunk driver
that hit a woman and it’s very very easy
to see the damages on the bone structure
and the course of death is the broken
neck and this woman also ended up under
the car so she’s quite badly beaten up
by by this injury here’s another case a
knifing and this is also again showing
us what we can do it’s very easy to look
at metal artifacts that we can show
inside of the body you can also see some
of the some of the artifacts from from
the teeth that’s actually the feeling in
the teeth but because I’ve set the
functions to show me metal and make
everything else transparent here’s
another violin case this really didn’t
kill the person the person was killed by
stabs in the heart but they just
deposited the knife by putting it
through one of the eyeballs here’s
another case it’s very very interesting
for us to
be able to look at things like knife
stabbings here you can see that knife
went through the heart it’s very easy to
see how air has been leaking from one
part to another part which it’s
difficult to do in a normal standard
physical autopsy so it really really
helps the the criminal investigation to
establish the cause of death and in some
cases also directing investigation in
the right direction to find out who the
killer really wasn’t
here’s another case that I think is
interesting here we you can see a bullet
that has lodged just next to the spine
on this person and what we’ve done is
that we’ve turned the bullet into a
light source so the the bullet is
actually shining and it makes it really
easy to find these fragments during a
physical autopsy if you have to dig
through the body to find these fragments
that’s actually quite hard to do one of
the things that I’m really really happy
to be able to show you here today is our
virtual autopsy table it’s a touch
device that we have developed based on
these algorithms using standard graphics
GPUs it actually looks like this just to
give you a feeling for what it looks
like it’s really an it really just works
like a huge iPhone so we’ve implemented
all the gestures that you can do on the
table
and you can think of it as an enormous
touch interface so if you were thinking
of buying an iPad forget about it this
is what you want instead Steve I hope
you’re listening to this way okay so
it’s a very nice little device so if you
have the opportunity please try it out
it’s it’s really a hands-on experience
so it gained some traction and we’re
trying to roll this out and trying to
use it for educational purposes but also
perhaps in the future in a more clinical
situation that’s a YouTube video that
you can download and look at this if you
want to convey the information to other
people about virtual autopsies okay now
we’re talking about touch let’s let me
move on to really touching data and this
is a bit of science fiction now so so
we’re moving into the really the future
this is not really what the medical
doctors are using right now but I hope
they will in the future so what you’re
seeing on the left is a touch device
it’s a it’s a little mechanical pen that
has very very fast depth motors inside
of the tongue so I can generate a force
feedback so when I perch
we touched data it will generate forces
independent so I get a feedback okay so
in this particular situation is it’s a
scan or the living person I have this
pen and I look at the data and I moved
the pen towards the head and all of a
sudden my feet of assistance okay so I
can feel the skin if I push a little bit
harder I’ll go through the skin and I
can feel the bone structure inside if I
push even harder I’ll go through the
bone structure especially close to the
ear where the bone is very soft and then
I can feel the brain inside and this
will be a slushie weightless so this is
really nice and you know to take that
even further this is at heart and this
is also due to these fantastic new
scanners that justing in 0.3 seconds I
can scan the whole heart and I can do
that with time resolution so just
looking at this heart I can playback a
video here and this is Collier on one of
my graduate students is been working on
this project and he’s sitting there in
front of the haptic device the force
feedback system and he’s moving his pen
towards the heart and the heart is not
beating in front of him so you can see
how the heart’s beating he’s taking the
pen and he’s moving it towards the heart
and he’s putting it on the heart and
then he feels the heartbeats from the
real living patient and he can examine
how the heart is moving they can go
inside push inside of the heart and
really feel how the valves are moving
and this I think is really the future
for for heart surgeons so I mean it’s
probably the wet dream for a heart
surgeon to be able to go inside of the
patients heart before we actually do
surgery and do that with high quality
resolution data so this is really neat
now we’re going even further into
science fiction and we heard a little
bit about functional MRI now this is
really really an interesting project MRI
is using magnetic fields and radio
frequencies to scan off the brain or any
part of the body so what we’re really
getting out of this is information of
the structure of the brain but we can
also measure the difference in magnetic
properties of blood that’s oxygenated
and blood as depleted of oxygen that
means that it’s possible to map out the
activity of the brain so this is
something that we’ve been working on and
you just saw Matt’s the research
engineer they’re going into the MRI
system and he was wearing goggles
they could actually see things in the
goggle so I could present things to him
while he’s in the scanner and this is a
little bit freaky
because what Matt’s is seeing is
actually this he’s seeing his own brain
so much is doing something here probably
he’s going like this with his right hand
because the left side is is activated
with the motor cortex and then you can
see that at the same time these
visualizations are brand new and this is
something that we’ve been researching
for a little while this is another
sequence of Matz’s brain and here we
asked much to go to calculate backwards
from 100 so he’s going 100 97 94 and
then he’s going backwards and you can
see how little math processor is working
up here in his brain and is lighting up
the whole brain this is fantastic we can
do this in real time we can investigate
things we can tell him to do things you
can also see that his visual cortex is
activated in the back of the head
because that’s where he’s seeing he’s
seeing his own brain and it’s also
hearing our instructions when we tell
him to do things the signal is really
deep inside of the brain as well and
it’s shining through because all of the
data is inside of this volume and in
just a second here you will see ok here
mots ok now move your left foot okay so
it’s going like this ok but for 20
seconds it’s going like that and all of
a sudden it lights up up here so we get
motor cortex activation up there so this
is really really nice and and I think
this is a great tool and connecting also
to the previous talk here this is
something that we could use as a tool to
really understand how the neurons are
working how the brain is working and we
can do this was a very very high visual
quality and very fast resolution now
we’re also having a bit of fun at the
center so this is a cat scan computed
tomography so this is a lion from the
local Zoo outside of norrkoping called
Morden Elsa so she came to the center
and they sedated her and then put it
straight into the scanner and of course
I get the whole data set from the lion
and I can do very nice images like this
I can peel off the layer of the lion I
can look inside of it and you know we’ve
been experimenting with this and I think
this is a great application for the
future of this technology because
there’s very little known about the
animal Anatomy what’s known out there
for veterinarians it’s kind of basic
information now we can scan all sorts of
things all sorts of animals
the only problem is to fit it into the
machine okay so here’s a bear it was
kind of hard to get it in and you know
the bear is a cuddly friendly animal and
here it is here’s the nose of the bear
and you know you might want to cuddle
this one until you’ve changed the
functions look at this so be aware of
the bear alright so with that I’d like
to to thank all the people that have
helped me to generate these images it’s
a huge effort that goes into doing this
gathering the data and developing the
algorithms writing all the software so
some very talented people my motto is
always I only hire people that are
smarter than I am and most of these are
smarter than I am so thank you very much