Making Visual Design Adapt to Emotions

[Music]

the other day

i was running late for an important

meeting and i just

missed my train i took out my phone

i opened the app to look for an

alternative connection and it took

forever

i just wanted to go and search for an

alternative connection

but the you know i

misspelled the name of the station i

want i

wanted to go to and i needed to start

over

again even the bright yellow

part of the brand identity of the

mobility providers app

which i usually liked for its boldness

and freshness

was aggressive in a way and was getting

on my nerves

the entire experience was neither

helpful

nor fresh it was just stressful

imagine in this situation

that the app recognizes your fear

your anxiety your stress and starts

adapting to it

by giving you intelligent corrections

by guiding you intelligently and by

preventing you from misspelling things

by even

giving you a neatly arranged map showing

you

the fastest connection and the estimated

time of arrival

if what i would have that kind of

experience

probably i wouldn’t be that stressed and

maybe i realized that i get in time to

my meeting

and would have relaxed a little bit

this is a glimpse into the future of

visual design as i see it

a future where

design is based on actual users

needs and emotions and effects

it does a design that helps you achieve

your goals faster and with more delight

if you think that this is a far-fetched

scenario

on the one hand you’re mistaken

the technology that these experiences

are based on is already here it’s

already available to designers

it’s called emotion ai or effective

computing

and it’s a very powerful tool if you

think that

this is a far fetched fetched scenario

on the other hand you’re wrong

you’re right because designers they

don’t have a rule book they don’t have

guidelines to follow in order to create

these kind of

kind of experiences

if we could help develop these kind of

rules and principles

this would have far reaching

consequences for the design community

because designers would change it would

change the way designers tackle design

problems

but it would also change the way

people use and perceive the design

in general

i want you to take a step back with me

and see

where this kind of design fits in

since centuries now design has been

informed by a fixed set of static design

principles

take the golden ratio for example

which defines a very balanced canvas

by dividing the canvas into two sections

but it’s a fixed it’s a fixed principle

and 100 years ago the design the grid

system

was developed based on this golden ratio

rule

which gave designers more control over

the canvas but also

uh gave gave more structure and and

determined how to use white space

patterns fonts images

and subsequently

determine how to how to come up with

hierarchy and

emphasis balance and unity rhythm and

contrast

rhythm and movement and contrast finally

you can see in the slide here

that this

is informed by the rigid grid system

and modernism basically the philosophy

to this kind of uh rigid concept

was propelled and made popular the idea

of design as a universal constant

meaning that design should not have any

regard

for cultural or social backgrounds of

its users

nor for any personal aesthetic

preferences the famous bauhaus

propelled and popularized the idea of a

narrow set

of rationalistic design principles that

guide designers up until this day

and it’s no wonder that af 30 years

into digital design we still have

predominantly grid-based

static web pages and one-size-fits-all

experiences there are only two ways

in which this kind of static design is

challenged the first is

generative design generative design

introduces

the idea of design variations

handing over a certain amount of design

action to a design software by

predefining

certain parameters but these parameters

are not

based on users and users actual needs

and goals

and the second one is the second one

these kind of static designs are

challenged

is dynamic web content so agencies begin

to experiment

with dynamic web content by combining

machine learning models

and design systems and create some sort

of

personalized design based on user types

so if you’re a magazine type of reader

you will be presented if you visit a

website

with bigger pictures more pictures

bigger headline and less text if you are

a newspaper kind of reader

you get presented with more text

smaller headlines and fewer images

but these kind of dynamic web pages

they are still static in a way because

they

rely on static preferences on

fixed likes and dislikes of people

the concept i want to talk about is a

concept that

hyper individualizes design that makes

it

micro adaptable and highly situational

because it is based on actual users

on actual moods and emotions from users

so here’s this little model i want to

explain where that explains this kind of

idea

you have an emotion detection algorithm

which is a face recognition algorithm

that

feeds a design software which then

gives users based on their emotions

based on their situations based on the

current

environment they’re in that they give

users kind of individualized designs

layouts logo designs and they can be

they can be used for all kinds of

scenarios

for example uh the scenario from the

beginning for rush

travelers for stressed drivers

in cars up to

frustrated gamers where the game design

is actually adapting to the

level of frustration a gamer has

and in all in all this is an experience

that is

tailored to personal emotional needs

i as a designer set out to

test this radical idea with the means

that i have i used

the open source software face api

that recognizes emotions and i used

the open source design software p5.js

and i created a simple shape which

uh is a prototype which could be

uh which is a prototype that is

interestingly able to

for to work iteratively and quickly to

prevent me from

becoming too complex in a in in it into

too much in into a small of time

so i i called this i called this

prototype

the shape blobby because it resembles a

blob

basically and blobby changes color

blobby changes

shape lobby changes motion and here’s

how blobby works um look at the video

i brought um so if users

are happy lobby is getting happy too

he uh it it is getting bigger and and

the colors get brighter

and blobby is

moving around more lively and if you’re

sad

blobby’s getting sad too getting grayer

and

inanimate in a way and if you are

surprised

lobby’s getting surprised too by

changing colors rapidly

and if you’re angry basically lobby’s

getting spiky

okay i tested this kind of

prototype with with a lot of different

people

and here’s what i learned first of all

users really liked this kind of

prototype they thought it’s playful

they thought it’s fun to play around

and they quickly imitated emotions

coming up with facial expressions to see

how this

prototype reacts and we could easily

come up with use

cases for this kind of prototype

beginning from social avatars

where you have your social

state of mind represented by blobby for

other people to see in a remote

situation

to social games to break the ice

basically when people

are in a setting where they don’t know

each other

making funny faces and getting to know

each other better better

digital support animals that basically

soothe a patient like lobby could soothe

people who suffer from

anxiety up to

uh settings in wellness where you could

um project blobby uh

in a bathing or a showering setting

and support the wellness feeling

but this was not what i had in mind

originally

for me it was too game-like people

playing around with blobby was not

that what i thought it should be i

thought it’s

lobby is working more silently in the

background because i thought it’s

becoming designed eventually so for me

the next step would be

to create something that resembles more

a design situation

for example a logo that changes or

typography or even an entire web page

a second hurdle i stumbled across was

that emotion detection

is not working at all so

i use it but it’s not working so here’s

an example

with smiling for example so if you smile

it’s not always indicative of happiness

and well-being

you can easily see that by smiling you

can

hide stress or discomfort you can

disguise

insecurities you can reassure other

people who talking to you

you can even use smiling sarcastically

or in a demeaning fashion

last not least in smiling is a

stereotypical gender norm

women are said to be to smile

more this could all be uh

this could all uh be a situation

but the algorithm does not detect these

kind of subtleties

lisa feldman barrett um from the

northeastern university

did uh very thorough research on this

topic and she found out that

almost all of the algorithms they don’t

account for

situational personal or conversational

aspects

of emotion hence failing completely at

detecting

genie wine emotions in all kinds of

situations

so to cope with this dilemma it means we

have to get more data

more sensors more information on people

which means to invade body mind and

privacy of people

which raises serious ethical concerns

and even if we mitigate these kinds of

concerns we create maybe create

applications that yeah

actually evolve into this kind of

emotion

filter bubble that keep people in this

emotion filter bubble or

and and uh persuasive black boxes

and the third and final hurdle i

discovered is

that we as designers lack a decent

amount of knowledge how to design these

kinds of

uh interaction these kind of experiences

so to give you just a quick lay of the

land here are some questions

that come up if you want to start using

this technology

for example how do you change colors

if people are stressed and miss their

train you know remember

the the small scenario from the

beginning

how do you how do you change um

typography in a

in a virtual museum if people get bored

how do you create

interactive layouts adaptable layouts if

people can find certain items

in shops and get frustrated

and how does it all align with brand

identity rules and guidelines with

which are said to be static and and

fixed

so yeah this is uh there are many

variables in this equation which we have

to solve

in conclusion this

is all uncharted territory

but these are interesting times for

designers at least that’s what

what i think we are called upon as

designers to develop

these new rules and principles for

adaptive

emotion centered design rules which on

the one side

account for emotion bubbles and

persuasive black boxes

rules that on the other hand usher in

this new kind of design era

a design era in which design is highly

situational

highly personalized and interwoven with

effective and emotional states in other

words

emotion centered design that is

delivering

unique user experience

thank you very much

you