Visualizing the medical data explosion Anders Ynnerman

I will start by posting a little bit of

a challenge the challenge of dealing

with data data that we have to deal with

in medical situations it’s really a huge

challenge for us and this is our beast

of burden this is a computer tomography

machine CT machine it’s a fantastic

device it uses x-rays x-ray beams that

are rotating very fast around the human

body it takes about 30 seconds to go

through the whole machine and is

generating enormous amounts of

information that comes out of the

machine so this is a fantastic machine

that we can use for improving healthcare

but as I said it’s also a challenge for

us and the challenge is really found in

this picture here it’s the medical data

explosion that we’re having right now

we’re facing this problem and let me

step back in time let’s just go back a

few years in time and see what happened

back then these machines that came out

they started coming in the 1970s they

would scan off human bodies and they

would generate about a hundred images of

the human body and I’ve taken the

liberty just for clarity to translate

that to data sizes that would correspond

to about 15 megabytes of data which is

small when you think about the data that

we can handle today just on normal

mobile devices if you translate that to

phone books it’s about one litre of

phone books in the pile looking at what

we’re doing today with these machines

that we have we can just in a few

seconds get 24,000 images out of the

body and that will correspond to about

20 gigabytes of data or 800 phone books

and the pod will then be 200 meters of

phone books what’s about to happen and

we’re seeing this it’s beginning a

technology trend that’s happening right

now is that we’re starting to look at

time-resolved situations as well so

we’re getting the dynamics out of the

body as well and just assume that we

will be collecting data during five

seconds and that will correspond to

about one terabyte of data that’s

800,000 books and 16 kilometres of phone

books there’s one patient one data set

and this is what we have to deal with so

this is really the enormous challenge

that we have

today this is 25,000 images you know

imagine that the days when we had

radiologists

doing this they would put up 25,000

images that would go like this 25 there

there is the problem right they can’t do

that anymore that’s impossible so we

have to do something that’s a little bit

more intelligent than doing this okay so

what we do is that we put all these

slices together

imagine that you slice your body in all

these directions and then you try to put

the slices back together again into a

pile of data into a block of data so

this is really what we’re doing so this

gigabyte a terabyte of data we’re

putting it into this block but of course

the block of data just contains the

amount of x-ray that’s been absorbed in

each point in the human body so what we

need to do is to figure out a way of

looking at the things that we do want to

look at and make things transparent that

we don’t want to look at so transforming

the data set into something that looks

like this

and this is a challenge this is a huge

challenge for us to do that using

computers even though they’re getting

faster and better all the time it’s a

challenge to deal with gigabytes of data

or terabytes of data and extracting the

relevant information I want to look at

the heart I want to look at the blood

vessels I want to look at the liver

maybe even finally the tumor in some

cases okay so this is where this little

dude comes into play this is my daughter

this is a as of 9:00 a.m. this morning

she’s playing a computer game she’s only

2 years old and she’s having a blast so

she’s really the the driving force

behind the development of graphics

processing units as long as kids are

playing computer games graphics is

getting better and better and better so

please go back home tell your kids to

play more games because that’s what I

need so what’s inside of this machine is

what enables me to do the things that

I’m doing with the medical data so

really what I’m doing is using these

fantastic little devices and you know

going back maybe 10 years in time when I

got the funding to buy my first graphics

computer

it was a huge machine it was cabinets of

processors and storage and everything I

paid about 1 million dollars for that

machine that machine is today about as

fast as my iPhone

so every month there are new graphics

cards coming out and here’s a few of the

latest ones from the vendors and Nvidia

ATI Intel is out there as well and you

know for a few hundred bucks you can get

these things and put them into your

computer and you can do fantastic things

with these graphics cards so this is

really what’s enabling us to deal with

the explosion of data in medicine

together with some really nifty work in

terms of algorithms compressing data

extracting the relevant information that

people are doing research on so I’m

going to show you a few examples of what

we can do this is the data set that was

captured using a CT scanner you can see

this is a full data it’s a it’s a woman

you can see the hair you can see the

individual structures and of the woman

you can see that there is scattering or

x-rays on the teeth the metal in the

teeth that’s that’s what those artifacts

are coming from but fully interactively

on standard graphics cards I on a normal

computer I can just put in a clip plane

and of course all the data is inside so

I can start rotating I can look at it

from different angles and I can see that

this woman had a problem there was a she

had a bleeding up in the brain and

that’s been fixed with a little stent a

metal clamp that’s tightening up the

vessel and just by changing the

functions and I can decide what’s going

to be transparent and what’s going to be

visible I can look at the skull

structure and I can see that okay this

is where they open up the skull on this

woman and that’s where they went in so

these are fantastic images they’re

really high resolution and they’re

really showing us what we can do with

with standard graphics cards today

now we have really made use of this and

we have tried to squeeze a lot of data

into the system and one of the

applications that we’ve been working on

and this has gotten a little bit of

traction worldwide is the application of

virtual autopsies so again looking at

very very large datasets and you saw

those full-body scans that we can do

we’re just pushing the body through the

whole CT scanner and just in a few

seconds we can get a full-body data set

so so this is from a virtual autopsy and

you can see how I’m gradually peeling

off first you saw the body bag that the

body came in that I’m peeling off the

skin you can see the muscles and

eventually

you can see the bone structure of this

woman now at this point I would also

like to emphasize that with the greatest

respect for the people that I’m now

going to show I’m going to show you a

few cases of virtual autopsies so it’s a

great respect for the people that have

died under violent circumstances that

I’m showing these pictures to you in the

forensic case and this is something that

there’s been approximately 400 cases so

far just in the part of Sweden that I

come from that has been undergoing

virtual autopsies in the past four years

so this will be the typical workflow

situation that police would decide in

the evening when there’s a case coming

in they would decide okay this is the

case where we need to do an autopsy so

in the morning between 6:00 and 7:00 in

the morning the board is then

transported inside of the body bag to

our Center and is being scanned through

one of the CT scanners and then the

radiologist together with the

pathologist and sometimes the forensic

scientist looks at the data that’s

coming out and they have a joint session

and then they decide what to do in the

real physical autopsy after that now

looking at a few cases here’s one of the

first cases that we had you can really

see the details of the data set it’s

very high resolution and it’s our

algorithms that allow us to zoom in on

all the details

and again it’s fully interactive so you

can rotate and you can look at things in

real time all these systems here without

saying too much about this case this is

a traffic accident and a drunk driver

that hit a woman and it’s very very easy

to see the damages on the bone structure

and the course of death is the broken

neck and this woman also ended up under

the car so she’s quite badly beaten up

by by this injury here’s another case a

knifing and this is also again showing

us what we can do it’s very easy to look

at metal artifacts that we can show

inside of the body you can also see some

of the some of the artifacts from from

the teeth that’s actually the feeling in

the teeth but because I’ve set the

functions to show me metal and make

everything else transparent here’s

another violin case this really didn’t

kill the person the person was killed by

stabs in the heart but they just

deposited the knife by putting it

through one of the eyeballs here’s

another case it’s very very interesting

for us to

be able to look at things like knife

stabbings here you can see that knife

went through the heart it’s very easy to

see how air has been leaking from one

part to another part which it’s

difficult to do in a normal standard

physical autopsy so it really really

helps the the criminal investigation to

establish the cause of death and in some

cases also directing investigation in

the right direction to find out who the

killer really wasn’t

here’s another case that I think is

interesting here we you can see a bullet

that has lodged just next to the spine

on this person and what we’ve done is

that we’ve turned the bullet into a

light source so the the bullet is

actually shining and it makes it really

easy to find these fragments during a

physical autopsy if you have to dig

through the body to find these fragments

that’s actually quite hard to do one of

the things that I’m really really happy

to be able to show you here today is our

virtual autopsy table it’s a touch

device that we have developed based on

these algorithms using standard graphics

GPUs it actually looks like this just to

give you a feeling for what it looks

like it’s really an it really just works

like a huge iPhone so we’ve implemented

all the gestures that you can do on the

table

and you can think of it as an enormous

touch interface so if you were thinking

of buying an iPad forget about it this

is what you want instead Steve I hope

you’re listening to this way okay so

it’s a very nice little device so if you

have the opportunity please try it out

it’s it’s really a hands-on experience

so it gained some traction and we’re

trying to roll this out and trying to

use it for educational purposes but also

perhaps in the future in a more clinical

situation that’s a YouTube video that

you can download and look at this if you

want to convey the information to other

people about virtual autopsies okay now

we’re talking about touch let’s let me

move on to really touching data and this

is a bit of science fiction now so so

we’re moving into the really the future

this is not really what the medical

doctors are using right now but I hope

they will in the future so what you’re

seeing on the left is a touch device

it’s a it’s a little mechanical pen that

has very very fast depth motors inside

of the tongue so I can generate a force

feedback so when I perch

we touched data it will generate forces

independent so I get a feedback okay so

in this particular situation is it’s a

scan or the living person I have this

pen and I look at the data and I moved

the pen towards the head and all of a

sudden my feet of assistance okay so I

can feel the skin if I push a little bit

harder I’ll go through the skin and I

can feel the bone structure inside if I

push even harder I’ll go through the

bone structure especially close to the

ear where the bone is very soft and then

I can feel the brain inside and this

will be a slushie weightless so this is

really nice and you know to take that

even further this is at heart and this

is also due to these fantastic new

scanners that justing in 0.3 seconds I

can scan the whole heart and I can do

that with time resolution so just

looking at this heart I can playback a

video here and this is Collier on one of

my graduate students is been working on

this project and he’s sitting there in

front of the haptic device the force

feedback system and he’s moving his pen

towards the heart and the heart is not

beating in front of him so you can see

how the heart’s beating he’s taking the

pen and he’s moving it towards the heart

and he’s putting it on the heart and

then he feels the heartbeats from the

real living patient and he can examine

how the heart is moving they can go

inside push inside of the heart and

really feel how the valves are moving

and this I think is really the future

for for heart surgeons so I mean it’s

probably the wet dream for a heart

surgeon to be able to go inside of the

patients heart before we actually do

surgery and do that with high quality

resolution data so this is really neat

now we’re going even further into

science fiction and we heard a little

bit about functional MRI now this is

really really an interesting project MRI

is using magnetic fields and radio

frequencies to scan off the brain or any

part of the body so what we’re really

getting out of this is information of

the structure of the brain but we can

also measure the difference in magnetic

properties of blood that’s oxygenated

and blood as depleted of oxygen that

means that it’s possible to map out the

activity of the brain so this is

something that we’ve been working on and

you just saw Matt’s the research

engineer they’re going into the MRI

system and he was wearing goggles

they could actually see things in the

goggle so I could present things to him

while he’s in the scanner and this is a

little bit freaky

because what Matt’s is seeing is

actually this he’s seeing his own brain

so much is doing something here probably

he’s going like this with his right hand

because the left side is is activated

with the motor cortex and then you can

see that at the same time these

visualizations are brand new and this is

something that we’ve been researching

for a little while this is another

sequence of Matz’s brain and here we

asked much to go to calculate backwards

from 100 so he’s going 100 97 94 and

then he’s going backwards and you can

see how little math processor is working

up here in his brain and is lighting up

the whole brain this is fantastic we can

do this in real time we can investigate

things we can tell him to do things you

can also see that his visual cortex is

activated in the back of the head

because that’s where he’s seeing he’s

seeing his own brain and it’s also

hearing our instructions when we tell

him to do things the signal is really

deep inside of the brain as well and

it’s shining through because all of the

data is inside of this volume and in

just a second here you will see ok here

mots ok now move your left foot okay so

it’s going like this ok but for 20

seconds it’s going like that and all of

a sudden it lights up up here so we get

motor cortex activation up there so this

is really really nice and and I think

this is a great tool and connecting also

to the previous talk here this is

something that we could use as a tool to

really understand how the neurons are

working how the brain is working and we

can do this was a very very high visual

quality and very fast resolution now

we’re also having a bit of fun at the

center so this is a cat scan computed

tomography so this is a lion from the

local Zoo outside of norrkoping called

Morden Elsa so she came to the center

and they sedated her and then put it

straight into the scanner and of course

I get the whole data set from the lion

and I can do very nice images like this

I can peel off the layer of the lion I

can look inside of it and you know we’ve

been experimenting with this and I think

this is a great application for the

future of this technology because

there’s very little known about the

animal Anatomy what’s known out there

for veterinarians it’s kind of basic

information now we can scan all sorts of

things all sorts of animals

the only problem is to fit it into the

machine okay so here’s a bear it was

kind of hard to get it in and you know

the bear is a cuddly friendly animal and

here it is here’s the nose of the bear

and you know you might want to cuddle

this one until you’ve changed the

functions look at this so be aware of

the bear alright so with that I’d like

to to thank all the people that have

helped me to generate these images it’s

a huge effort that goes into doing this

gathering the data and developing the

algorithms writing all the software so

some very talented people my motto is

always I only hire people that are

smarter than I am and most of these are

smarter than I am so thank you very much