The Multisensory Gap Towards Audiovisual Design
[Music]
we live in a multi-sensory world
our senses are constantly being
stimulated by an integrated assault
of sounds images
sensory impressions of different kind
we all remember walking down a pier and
hearing this
water splash around us our brain can
fill in gaps
when sensory information is not present
when we ought watch a suggestive image
such as this one
that will evoke sounds and other sensory
impressions
an example of this connection between
senses is the kikibu
effect it was first observed in the
early 20th century
participants were asked which shape is
called kiki
and which shape is called bubba
the answer is consistent the vast
majority of respondents
associate the rounded shape with bubba
and the spiky shape with kiki
this means that our brain attaches
abstract meanings to shapes and sounds
in a consistent way
michel xion states that there is no
sensory given
that is isolated from the start the
senses are channels
highways more than territories or
domains
when kinetic sensations organized into
art are transmitted through a single
sensory channel
they can convey all other senses via
that channel
exemplifies this with inherent visuality
of music concrete
and the implied sound behind silent
movies
the rise of radio and tv and the
escalation of the mechanical
reproduction of media in the 20th
century
changed how media was consumed for
example
music partly became a commodity often
dissociated from performance
which is highly visual this has led to a
multi-sensory gap
packaging and music videos for example
have partly compensated for that but the
gap remains
different waves of digitization have
further affected how we consume media
in the 21st century for example
streaming audio grew by
one third in the first half of 2019
the portability of our digital devices
and the social context in which we
interact with them
have inhibited the use of sound
furthering the multi-sensory gap
digital technologies are distancing us
from the richness of real world
experiences
and they are limiting how media can
convey them
how will our digital future look like
if this multi-sensory gap continues how
will we evolve
will we become sensorily desensitized
babies and young children sense their
environment as a blend of light
sound smell and other impressions
education however has created sensory
silos
we are taught that visual design and
sound design
are very different fields often taught
in different departments or faculties
when sound design and visual design are
combined often there is a lack of
careful look at the correspondences
between them
how can we invert this trend bring more
sensoriality to design
and to digital experiences i have been
working with audiovisual art and design
for 15 years
i’d like to show you a video of one of
my projects
called av clash from 2010
av class is a netart project you can
check it online in avclash.com
av clash visualizes one of the largest
sound databases in the world called
freesound.org
but instead of navigating that database
as image and static audio
images you
navigate a series of four interactive
audio visual objects
that you can manipulate and they will
create animations for you
based on the sounds that they visualize
[Music]
[Music]
i believe there are two main fields that
lead the way in exploring the
reintegration of senses
in digital experiences one is audio
visual performance
audio visual performances are electronic
music concerts
usually where artists are generating
visuals
tightly connected with music creating
integrated auto visual
experiences in the image you can see my
latest
project for artificial performance
called av zones
another important field is video games
the game’s industry has an increased
awareness
regarding the importance of
multi-sensory experience
and how it contributes to engagement and
immersion
in this image you can see an audio
visual memory game i have developed
called shape tones
inspired by kikibuba
accessibility also benefits from a
multi-sensory design approach
persons with impairment in one sense
will benefit
if related information is conveyed
through another sense
multi-sensory design brings specific
challenges how to match and map
sensory modalities how much space to
leave for interpretation
for this reason we need audio visual
design and multi-sensory approaches
taught in schools and at universities
we also need to bring awareness
regarding multi-sensory user experience
to the design community
we are immersed in digital technologies
and this is distancing us from how we
experience the world
a multi-sensory approach to design can
help create digital experiences
that are more meaningful accessible
engaging and human thank you