Haptography Digitizing our sense of touch Katherine Kuchenbecker
Translator: Andrea McDonough
Reviewer: Morton Bast
I’m a mechanical engineering professor
at the University of Pennsylvania,
and my favorite hobby is photography.
And as I travel around the world,
I love taking photographs like these,
so I can remember all
the beautiful and interesting things
that I’ve seen.
But what I can’t do is record and share
how these objects feel to touch.
And that’s kind of surprising,
because your sense of touch
is really important.
It’s involved in every physical
interaction you do every day,
every manipulation task,
anything you do in the world.
So the sense of touch
is actually pretty interesting.
It has two main components.
The first is tactile sensations,
things you feel in your skin.
And the second is kinesthetic sensations.
This has to do with the position
of your body and how it’s moving,
and the forces you encounter.
And you’re really good at incorporating
both of these types of sensations together
to understand the physical interactions
you have with the world
and understand as you touch a surface:
is it a rock, is it a cat,
is it a bunny, what is it?
And so, as an engineer,
I’m really fascinated
and I have a lot of respect
for how good people are with their hands.
And I’m intrigued and curious
about whether we could
make technology better
by doing a better job at leveraging
the human capability
with the sense of touch.
Could I improve the interfaces
to computers and machines
by letting you take
advantage of your hands?
And indeed, I think we can,
and that’s at the core
of a field called haptics,
and this is the area that I work in.
It’s all about interactive
touch technology.
And the way it works is,
as you move your body through the world,
if, as an engineer, I can make a system
that can measure that motion,
and then present to you
sensations over time
that kind of make sense,
that match up with what you might feel
in the real world,
I can fool you into thinking
you’re touching something
even though there’s nothing there.
So here are three examples
and these are all done
from research in my lab at Penn.
The first one is all about that same
problem that I was showing you:
how can we capture how objects feel
and recreate those experiences?
So the way we solve this problem
is by creating a hand-held tool
that has many different sensors inside.
It has a force sensor, so we can tell
how hard you’re pushing;
it has motion tracking, so we can tell
exactly where you’ve moved it;
and it has a vibration sensor,
an accelerometer, inside,
that detects the shaking
back and forth of the tool
that lets you know
that’s a piece of canvas
and not a piece of silk or something else.
Then we take the data we record
from these interactions.
Here’s ten seconds of data.
You can see how the vibrations
get larger and smaller,
depending on how you move.
And we make a mathematical
model of those relationships
and program them into a tablet computer
so that when you take the stylus
and go and touch the screen,
that voice-coil actuator
in the white bracket
plays vibrations to give you the illusion
that you’re touching the real surface,
just like if you touched, dragged
back and forth, on the real canvas.
We can create very compelling illusions.
We can do this for all kinds of surfaces
and it’s really a lot of fun.
We call it haptography –
haptic photography.
And I think it has potential
benefits in all sorts of areas
like online shopping,
maybe interactive museum exhibits,
where you’re not supposed
to touch the precious artifacts,
but you always want to.
The second example I want to tell you
about comes from a collaboration I have
with Dr. Margrit Maggio
at the Penn Dental School.
Part of her job is to teach
dental students
how to tell where in a patient’s mouth
there are cavities.
Of course they look at X-rays,
but a large part of this clinical judgment
comes from what they feel
when they touch your teeth
with a dental explorer.
You’ve all had this happen,
they go across.
What they’re feeling for is if the tooth
is really hard, then it’s healthy,
but if it’s kind of soft and sticky,
that’s a signal that the enamel
is starting to decay.
These types of judgments are hard
for a new dental student to make,
because they haven’t touched
a lot of teeth yet.
And you want them to learn this
before they start practicing
on real human patients.
So what we do is add an accelerometer
on to the dental explorer,
and then we record what Dr. Maggio feels
as she touches different extracted teeth.
And we can play it back for you as a video
with a touch track –
not just a sound track,
but also a touch track,
that you can feel by holding
that repeating tool.
You feel the same things the dentist
felt when they did the recording,
and practice making judgments.
So here’s a sample one.
Here’s a tooth that looks
kind of suspicious, right?
It has all those brown stains.
You might be thinking, “We should
definitely put a filling in this tooth.”
But if you pay attention to how it feels,
all the surfaces of this tooth
are hard and healthy,
so this patient does not need a filling.
And these are exactly the kind
of judgments doctors make every day
and I think this technology
we’ve invented has a lot of potential
for many different things in medical
training, because it’s really simple
and it does a great job at recreating
what people feel through tools.
I think it could also help make games
more interactive and fun
and more realistic
in the sensations that you feel.
The last example I want to tell you about
is again about human movement.
So if any of you have ever learned sports,
how do you get good
at something like surfing?
You practice.
You practice some more and more, right?
Making small corrections,
maybe getting some input from a coach,
learning how to improve your motions.
I think we could use computers
to help make that process
more efficient and more fun.
And so here, for example,
if I have six different arm movements
that I want you to learn,
you come into my lab at Penn
and try out our system.
We use a Kinect to measure your motions,
we show graphics on the screen,
and then we also give you touch cues,
haptic feedback on your arm,
delivered by these haptic arm bands
which have motors inside,
and guide you as you move.
So, if we put it together,
as you’re trying to track this motion,
if you deviate – say, maybe,
your arm is a little too high –
we turn on the motors
right there on the skin
to let you know you should move down,
almost like a coach gently guiding you
and helping you master
these movements more quickly
and make more precise corrections.
We developed this system
for use in stroke rehabilitation,
but I think there are a lot
of applications,
like maybe dance training
or all sorts of sports training as well.
So now you know a little bit
about the field of haptics,
which I think you’ll hear more
about in the coming years.
I’ve shown you three examples.
I just want to take a moment
to acknowledge the great students
who work with me in my lab at Penn
and my collaborators.
They’re a great group.
I also want to thank you
for your kind attention.
(Applause)