Making Visual Design Adapt to Emotions
[Music]
the other day
i was running late for an important
meeting and i just
missed my train i took out my phone
i opened the app to look for an
alternative connection and it took
forever
i just wanted to go and search for an
alternative connection
but the you know i
misspelled the name of the station i
want i
wanted to go to and i needed to start
over
again even the bright yellow
part of the brand identity of the
mobility providers app
which i usually liked for its boldness
and freshness
was aggressive in a way and was getting
on my nerves
the entire experience was neither
helpful
nor fresh it was just stressful
imagine in this situation
that the app recognizes your fear
your anxiety your stress and starts
adapting to it
by giving you intelligent corrections
by guiding you intelligently and by
preventing you from misspelling things
by even
giving you a neatly arranged map showing
you
the fastest connection and the estimated
time of arrival
if what i would have that kind of
experience
probably i wouldn’t be that stressed and
maybe i realized that i get in time to
my meeting
and would have relaxed a little bit
this is a glimpse into the future of
visual design as i see it
a future where
design is based on actual users
needs and emotions and effects
it does a design that helps you achieve
your goals faster and with more delight
if you think that this is a far-fetched
scenario
on the one hand you’re mistaken
the technology that these experiences
are based on is already here it’s
already available to designers
it’s called emotion ai or effective
computing
and it’s a very powerful tool if you
think that
this is a far fetched fetched scenario
on the other hand you’re wrong
you’re right because designers they
don’t have a rule book they don’t have
guidelines to follow in order to create
these kind of
kind of experiences
if we could help develop these kind of
rules and principles
this would have far reaching
consequences for the design community
because designers would change it would
change the way designers tackle design
problems
but it would also change the way
people use and perceive the design
in general
i want you to take a step back with me
and see
where this kind of design fits in
since centuries now design has been
informed by a fixed set of static design
principles
take the golden ratio for example
which defines a very balanced canvas
by dividing the canvas into two sections
but it’s a fixed it’s a fixed principle
and 100 years ago the design the grid
system
was developed based on this golden ratio
rule
which gave designers more control over
the canvas but also
uh gave gave more structure and and
determined how to use white space
patterns fonts images
and subsequently
determine how to how to come up with
hierarchy and
emphasis balance and unity rhythm and
contrast
rhythm and movement and contrast finally
you can see in the slide here
that this
is informed by the rigid grid system
and modernism basically the philosophy
to this kind of uh rigid concept
was propelled and made popular the idea
of design as a universal constant
meaning that design should not have any
regard
for cultural or social backgrounds of
its users
nor for any personal aesthetic
preferences the famous bauhaus
propelled and popularized the idea of a
narrow set
of rationalistic design principles that
guide designers up until this day
and it’s no wonder that af 30 years
into digital design we still have
predominantly grid-based
static web pages and one-size-fits-all
experiences there are only two ways
in which this kind of static design is
challenged the first is
generative design generative design
introduces
the idea of design variations
handing over a certain amount of design
action to a design software by
predefining
certain parameters but these parameters
are not
based on users and users actual needs
and goals
and the second one is the second one
these kind of static designs are
challenged
is dynamic web content so agencies begin
to experiment
with dynamic web content by combining
machine learning models
and design systems and create some sort
of
personalized design based on user types
so if you’re a magazine type of reader
you will be presented if you visit a
website
with bigger pictures more pictures
bigger headline and less text if you are
a newspaper kind of reader
you get presented with more text
smaller headlines and fewer images
but these kind of dynamic web pages
they are still static in a way because
they
rely on static preferences on
fixed likes and dislikes of people
the concept i want to talk about is a
concept that
hyper individualizes design that makes
it
micro adaptable and highly situational
because it is based on actual users
on actual moods and emotions from users
so here’s this little model i want to
explain where that explains this kind of
idea
you have an emotion detection algorithm
which is a face recognition algorithm
that
feeds a design software which then
gives users based on their emotions
based on their situations based on the
current
environment they’re in that they give
users kind of individualized designs
layouts logo designs and they can be
they can be used for all kinds of
scenarios
for example uh the scenario from the
beginning for rush
travelers for stressed drivers
in cars up to
frustrated gamers where the game design
is actually adapting to the
level of frustration a gamer has
and in all in all this is an experience
that is
tailored to personal emotional needs
i as a designer set out to
test this radical idea with the means
that i have i used
the open source software face api
that recognizes emotions and i used
the open source design software p5.js
and i created a simple shape which
uh is a prototype which could be
uh which is a prototype that is
interestingly able to
for to work iteratively and quickly to
prevent me from
becoming too complex in a in in it into
too much in into a small of time
so i i called this i called this
prototype
the shape blobby because it resembles a
blob
basically and blobby changes color
blobby changes
shape lobby changes motion and here’s
how blobby works um look at the video
i brought um so if users
are happy lobby is getting happy too
he uh it it is getting bigger and and
the colors get brighter
and blobby is
moving around more lively and if you’re
sad
blobby’s getting sad too getting grayer
and
inanimate in a way and if you are
surprised
lobby’s getting surprised too by
changing colors rapidly
and if you’re angry basically lobby’s
getting spiky
okay i tested this kind of
prototype with with a lot of different
people
and here’s what i learned first of all
users really liked this kind of
prototype they thought it’s playful
they thought it’s fun to play around
and they quickly imitated emotions
coming up with facial expressions to see
how this
prototype reacts and we could easily
come up with use
cases for this kind of prototype
beginning from social avatars
where you have your social
state of mind represented by blobby for
other people to see in a remote
situation
to social games to break the ice
basically when people
are in a setting where they don’t know
each other
making funny faces and getting to know
each other better better
digital support animals that basically
soothe a patient like lobby could soothe
people who suffer from
anxiety up to
uh settings in wellness where you could
um project blobby uh
in a bathing or a showering setting
and support the wellness feeling
but this was not what i had in mind
originally
for me it was too game-like people
playing around with blobby was not
that what i thought it should be i
thought it’s
lobby is working more silently in the
background because i thought it’s
becoming designed eventually so for me
the next step would be
to create something that resembles more
a design situation
for example a logo that changes or
typography or even an entire web page
a second hurdle i stumbled across was
that emotion detection
is not working at all so
i use it but it’s not working so here’s
an example
with smiling for example so if you smile
it’s not always indicative of happiness
and well-being
you can easily see that by smiling you
can
hide stress or discomfort you can
disguise
insecurities you can reassure other
people who talking to you
you can even use smiling sarcastically
or in a demeaning fashion
last not least in smiling is a
stereotypical gender norm
women are said to be to smile
more this could all be uh
this could all uh be a situation
but the algorithm does not detect these
kind of subtleties
lisa feldman barrett um from the
northeastern university
did uh very thorough research on this
topic and she found out that
almost all of the algorithms they don’t
account for
situational personal or conversational
aspects
of emotion hence failing completely at
detecting
genie wine emotions in all kinds of
situations
so to cope with this dilemma it means we
have to get more data
more sensors more information on people
which means to invade body mind and
privacy of people
which raises serious ethical concerns
and even if we mitigate these kinds of
concerns we create maybe create
applications that yeah
actually evolve into this kind of
emotion
filter bubble that keep people in this
emotion filter bubble or
and and uh persuasive black boxes
and the third and final hurdle i
discovered is
that we as designers lack a decent
amount of knowledge how to design these
kinds of
uh interaction these kind of experiences
so to give you just a quick lay of the
land here are some questions
that come up if you want to start using
this technology
for example how do you change colors
if people are stressed and miss their
train you know remember
the the small scenario from the
beginning
how do you how do you change um
typography in a
in a virtual museum if people get bored
how do you create
interactive layouts adaptable layouts if
people can find certain items
in shops and get frustrated
and how does it all align with brand
identity rules and guidelines with
which are said to be static and and
fixed
so yeah this is uh there are many
variables in this equation which we have
to solve
in conclusion this
is all uncharted territory
but these are interesting times for
designers at least that’s what
what i think we are called upon as
designers to develop
these new rules and principles for
adaptive
emotion centered design rules which on
the one side
account for emotion bubbles and
persuasive black boxes
rules that on the other hand usher in
this new kind of design era
a design era in which design is highly
situational
highly personalized and interwoven with
effective and emotional states in other
words
emotion centered design that is
delivering
unique user experience
thank you very much
you