The Multisensory Gap Towards Audiovisual Design

[Music]

we live in a multi-sensory world

our senses are constantly being

stimulated by an integrated assault

of sounds images

sensory impressions of different kind

we all remember walking down a pier and

hearing this

water splash around us our brain can

fill in gaps

when sensory information is not present

when we ought watch a suggestive image

such as this one

that will evoke sounds and other sensory

impressions

an example of this connection between

senses is the kikibu

effect it was first observed in the

early 20th century

participants were asked which shape is

called kiki

and which shape is called bubba

the answer is consistent the vast

majority of respondents

associate the rounded shape with bubba

and the spiky shape with kiki

this means that our brain attaches

abstract meanings to shapes and sounds

in a consistent way

michel xion states that there is no

sensory given

that is isolated from the start the

senses are channels

highways more than territories or

domains

when kinetic sensations organized into

art are transmitted through a single

sensory channel

they can convey all other senses via

that channel

exemplifies this with inherent visuality

of music concrete

and the implied sound behind silent

movies

the rise of radio and tv and the

escalation of the mechanical

reproduction of media in the 20th

century

changed how media was consumed for

example

music partly became a commodity often

dissociated from performance

which is highly visual this has led to a

multi-sensory gap

packaging and music videos for example

have partly compensated for that but the

gap remains

different waves of digitization have

further affected how we consume media

in the 21st century for example

streaming audio grew by

one third in the first half of 2019

the portability of our digital devices

and the social context in which we

interact with them

have inhibited the use of sound

furthering the multi-sensory gap

digital technologies are distancing us

from the richness of real world

experiences

and they are limiting how media can

convey them

how will our digital future look like

if this multi-sensory gap continues how

will we evolve

will we become sensorily desensitized

babies and young children sense their

environment as a blend of light

sound smell and other impressions

education however has created sensory

silos

we are taught that visual design and

sound design

are very different fields often taught

in different departments or faculties

when sound design and visual design are

combined often there is a lack of

careful look at the correspondences

between them

how can we invert this trend bring more

sensoriality to design

and to digital experiences i have been

working with audiovisual art and design

for 15 years

i’d like to show you a video of one of

my projects

called av clash from 2010

av class is a netart project you can

check it online in avclash.com

av clash visualizes one of the largest

sound databases in the world called

freesound.org

but instead of navigating that database

as image and static audio

images you

navigate a series of four interactive

audio visual objects

that you can manipulate and they will

create animations for you

based on the sounds that they visualize

[Music]

[Music]

i believe there are two main fields that

lead the way in exploring the

reintegration of senses

in digital experiences one is audio

visual performance

audio visual performances are electronic

music concerts

usually where artists are generating

visuals

tightly connected with music creating

integrated auto visual

experiences in the image you can see my

latest

project for artificial performance

called av zones

another important field is video games

the game’s industry has an increased

awareness

regarding the importance of

multi-sensory experience

and how it contributes to engagement and

immersion

in this image you can see an audio

visual memory game i have developed

called shape tones

inspired by kikibuba

accessibility also benefits from a

multi-sensory design approach

persons with impairment in one sense

will benefit

if related information is conveyed

through another sense

multi-sensory design brings specific

challenges how to match and map

sensory modalities how much space to

leave for interpretation

for this reason we need audio visual

design and multi-sensory approaches

taught in schools and at universities

we also need to bring awareness

regarding multi-sensory user experience

to the design community

we are immersed in digital technologies

and this is distancing us from how we

experience the world

a multi-sensory approach to design can

help create digital experiences

that are more meaningful accessible

engaging and human thank you