How we can teach computers to make sense of our emotions Raphael Arar
Translator: Ivana Korom
Reviewer: Joanna Pietrulewicz
I consider myself one part artist
and one part designer.
And I work at an artificial
intelligence research lab.
We’re trying to create technology
that you’ll want to interact with
in the far future.
Not just six months from now,
but try years and decades from now.
And we’re taking a moonshot
that we’ll want to be
interacting with computers
in deeply emotional ways.
So in order to do that,
the technology has to be
just as much human as it is artificial.
It has to get you.
You know, like that inside joke
that’ll have you and your best friend
on the floor, cracking up.
Or that look of disappointment
that you can just smell from miles away.
I view art as the gateway to help us
bridge this gap between human and machine:
to figure out what it means
to get each other
so that we can train AI to get us.
See, to me, art is a way
to put tangible experiences
to intangible ideas,
feelings and emotions.
And I think it’s one
of the most human things about us.
See, we’re a complicated
and complex bunch.
We have what feels like
an infinite range of emotions,
and to top it off, we’re all different.
We have different family backgrounds,
different experiences
and different psychologies.
And this is what makes life
really interesting.
But this is also what makes
working on intelligent technology
extremely difficult.
And right now, AI research, well,
it’s a bit lopsided on the tech side.
And that makes a lot of sense.
See, for every
qualitative thing about us –
you know, those parts of us that are
emotional, dynamic and subjective –
we have to convert it
to a quantitative metric:
something that can be represented
with facts, figures and computer code.
The issue is, there are
many qualitative things
that we just can’t put our finger on.
So, think about hearing
your favorite song for the first time.
What were you doing?
How did you feel?
Did you get goosebumps?
Or did you get fired up?
Hard to describe, right?
See, parts of us feel so simple,
but under the surface,
there’s really a ton of complexity.
And translating
that complexity to machines
is what makes them modern-day moonshots.
And I’m not convinced that we can
answer these deeper questions
with just ones and zeros alone.
So, in the lab, I’ve been creating art
as a way to help me
design better experiences
for bleeding-edge technology.
And it’s been serving as a catalyst
to beef up the more human ways
that computers can relate to us.
Through art, we’re tacking
some of the hardest questions,
like what does it really mean to feel?
Or how do we engage and know
how to be present with each other?
And how does intuition
affect the way that we interact?
So, take for example human emotion.
Right now, computers can make sense
of our most basic ones,
like joy, sadness,
anger, fear and disgust,
by converting those
characteristics to math.
But what about the more complex emotions?
You know, those emotions
that we have a hard time
describing to each other?
Like nostalgia.
So, to explore this, I created
a piece of art, an experience,
that asked people to share a memory,
and I teamed up with some data scientists
to figure out how to take
an emotion that’s so highly subjective
and convert it into something
mathematically precise.
So, we created what we call
a nostalgia score
and it’s the heart of this installation.
To do that, the installation
asks you to share a story,
the computer then analyzes it
for its simpler emotions,
it checks for your tendency
to use past-tense wording
and also looks for words
that we tend to associate with nostalgia,
like “home,” “childhood” and “the past.”
It then creates a nostalgia score
to indicate how nostalgic your story is.
And that score is the driving force
behind these light-based sculptures
that serve as physical embodiments
of your contribution.
And the higher the score,
the rosier the hue.
You know, like looking at the world
through rose-colored glasses.
So, when you see your score
and the physical representation of it,
sometimes you’d agree
and sometimes you wouldn’t.
It’s as if it really understood
how that experience made you feel.
But other times it gets tripped up
and has you thinking
it doesn’t understand you at all.
But the piece really serves to show
that if we have a hard time explaining
the emotions that we have to each other,
how can we teach a computer
to make sense of them?
So, even the more objective parts
about being human are hard to describe.
Like, conversation.
Have you ever really tried
to break down the steps?
So think about sitting
with your friend at a coffee shop
and just having small talk.
How do you know when to take a turn?
How do you know when to shift topics?
And how do you even know
what topics to discuss?
See, most of us
don’t really think about it,
because it’s almost second nature.
And when we get to know someone,
we learn more about what makes them tick,
and then we learn
what topics we can discuss.
But when it comes to teaching
AI systems how to interact with people,
we have to teach them
step by step what to do.
And right now, it feels clunky.
If you’ve ever tried to talk
with Alexa, Siri or Google Assistant,
you can tell that it or they
can still sound cold.
And have you ever gotten annoyed
when they didn’t understand
what you were saying
and you had to rephrase what you wanted
20 times just to play a song?
Alright, to the credit of the designers,
realistic communication is really hard.
And there’s a whole branch of sociology,
called conversation analysis,
that tries to make blueprints
for different types of conversation.
Types like customer service
or counseling, teaching and others.
I’ve been collaborating
with a conversation analyst at the lab
to try to help our AI systems
hold more human-sounding conversations.
This way, when you have an interaction
with a chatbot on your phone
or a voice-based system in the car,
it sounds a little more human
and less cold and disjointed.
So I created a piece of art
that tries to highlight
the robotic, clunky interaction
to help us understand, as designers,
why it doesn’t sound human yet
and, well, what we can do about it.
The piece is called Bot to Bot
and it puts one conversational
system against another
and then exposes it to the general public.
And what ends up happening
is that you get something
that tries to mimic human conversation,
but falls short.
Sometimes it works and sometimes
it gets into these, well,
loops of misunderstanding.
So even though the machine-to-machine
conversation can make sense,
grammatically and colloquially,
it can still end up
feeling cold and robotic.
And despite checking all the boxes,
the dialogue lacks soul
and those one-off quirks
that make each of us who we are.
So while it might be grammatically correct
and uses all the right
hashtags and emojis,
it can end up sounding mechanical
and, well, a little creepy.
And we call this the uncanny valley.
You know, that creepiness factor of tech
where it’s close to human
but just slightly off.
And the piece will start being
one way that we test
for the humanness of a conversation
and the parts that get
lost in translation.
So there are other things
that get lost in translation, too,
like human intuition.
Right now, computers
are gaining more autonomy.
They can take care of things for us,
like change the temperature
of our houses based on our preferences
and even help us drive on the freeway.
But there are things
that you and I do in person
that are really difficult
to translate to AI.
So think about the last time
that you saw an old classmate or coworker.
Did you give them a hug
or go in for a handshake?
You probably didn’t think twice
because you’ve had so many
built up experiences
that had you do one or the other.
And as an artist, I feel
that access to one’s intuition,
your unconscious knowing,
is what helps us create amazing things.
Big ideas, from that abstract,
nonlinear place in our consciousness
that is the culmination
of all of our experiences.
And if we want computers to relate to us
and help amplify our creative abilities,
I feel that we’ll need to start thinking
about how to make computers be intuitive.
So I wanted to explore
how something like human intuition
could be directly translated
to artificial intelligence.
And I created a piece
that explores computer-based intuition
in a physical space.
The piece is called Wayfinding,
and it’s set up as a symbolic compass
that has four kinetic sculptures.
Each one represents a direction,
north, east, south and west.
And there are sensors set up
on the top of each sculpture
that capture how far away
you are from them.
And the data that gets collected
ends up changing the way
that sculptures move
and the direction of the compass.
The thing is, the piece doesn’t work
like the automatic door sensor
that just opens
when you walk in front of it.
See, your contribution is only a part
of its collection of lived experiences.
And all of those experiences
affect the way that it moves.
So when you walk in front of it,
it starts to use all of the data
that it’s captured
throughout its exhibition history –
or its intuition –
to mechanically respond to you
based on what it’s learned from others.
And what ends up happening
is that as participants
we start to learn the level
of detail that we need
in order to manage expectations
from both humans and machines.
We can almost see our intuition
being played out on the computer,
picturing all of that data
being processed in our mind’s eye.
My hope is that this type of art
will help us think differently
about intuition
and how to apply that to AI in the future.
So these are just a few examples
of how I’m using art to feed into my work
as a designer and researcher
of artificial intelligence.
And I see it as a crucial way
to move innovation forward.
Because right now, there are
a lot of extremes when it comes to AI.
Popular movies show it
as this destructive force
while commercials
are showing it as a savior
to solve some of the world’s
most complex problems.
But regardless of where you stand,
it’s hard to deny
that we’re living in a world
that’s becoming more
and more digital by the second.
Our lives revolve around our devices,
smart appliances and more.
And I don’t think
this will let up any time soon.
So, I’m trying to embed
more humanness from the start.
And I have a hunch that bringing art
into an AI research process
is a way to do just that.
Thank you.
(Applause)