Making Machines Creative.
[Music]
good morning um
today i want to talk about visual
creativity
and what it can teach us about our own
incredible brains
take a look at this painting what is it
about da vinci’s masterful brushstrokes
that evokes a living breathing
three-dimensional object
in our minds
how about this painting which despite
its very different
and perhaps unusual style still looks
kind of real
or this one or even this one
by three-year-old teresa it still looks
like a face right
how does that work now
we humans we’ve been using physical
tools like
brushes and chisels to reconstruct
our sensory observations in the form of
paintings and sculptures
for at least 60 000 years this
incredible cave painting for instance is
thought to be 17 000 years old
that’s thousands of years before we had
domesticated sheep and goats even
in fact we’ve been doing this
abstracting away from observations and
creating them in the form of paintings
for so long that it’s thought to be well
before
the earliest signs of any meaningful
written language even
take a look at this cave painting where
the artist
depicts this battle scene using a few
brush strokes
no more than say 10 or 20.
now many people believe that it is
precisely this ability to abstract away
from reality
and to conceptualize that reality that
underpins
the cognitive abilities that make us
uniquely intelligent as human beings
but it largely remains a mystery how
exactly we do this
and the field of artificial intelligence
or ai
concerns itself with this question
basically
what is intelligence what is it composed
of
how does it come to be and how can we
recreate it
i’m an ai scientist myself and i’ve
spent most of my career thinking about
how visual conceptualization
and visual abstraction can be recreated
now as you know and as you heard ai has
been making great progress over the past
couple of decades
we now have computer programs that can
play chess or go as well as the best
humans
and even self-driving cars seem within
reach
what i’m going to talk about today is
some of the early steps we’ve taken
towards getting computers to be visually
creative as well
now to do this we use computers heavily
and i’d like to begin by going over what
a computer actually is
so a computer in my mind is a simple
device that transforms
inputs to outputs and
the way we program a computer is by
giving it a list of instructions
or an algorithm to specify how we want
those inputs to be transformed into
outputs
in this example the inputs are
two numbers and what we want the
computer to do
is to output the sum and the algorithm
that’s run on the computer
is the summation algorithm now what
makes computers really useful
is that they are extremely fast and they
are extremely accurate
so they do those instructions very very
quickly and very accurately
but it turns out that there are certain
input output transformations
or certain tasks that we really care
about but no computer scientist has
succeeded in writing an
algorithm for it from scratch and only
from their own minds
for instance consider the case of
translation
where you want to go from one language
to another language now of course there
are many
human beings that know these two
languages and can translate
but none of them would be able to
describe how they do it with sufficient
formality to be able to express it to a
computer
and that’s where the idea of learning
comes in you will have heard of the term
machine learning
where the idea is that if we give the
computer enough examples of sentences in
both languages
it will by itself learn how to do this
transformation
and indeed today we now have systems
that can classify
pictures of cats and dogs and translate
sentences from languages into each other
with
fairly good accuracy now
the question that i asked um with my
colleagues a couple of years ago
was could we use this paradigm to get
computers to be creative
so if i consider the paintbrush and
paint
to be inputs to a computer can i get it
to
output visually interesting conceptually
abstract
outputs at the other end
now we know that computers can be used
to create images that’s what digital art
is all about
but in normal digital art the human is
specifying exactly what they want the
computer to do the algorithm is actually
being provided by the human
the question here is can the computer
learn that algorithm by itself
through trial and error kind of figure
out
what kinds of images are visually
appealing and conceptually abstract
so i’m going to dive into the setup that
we actually used
let’s begin with a painter in the bottom
right so the painter is a computer
program
that sends actions to another computer
program
in the form of brush strokes this
second thing which i call the canvas
here think of it as something like adobe
photoshop or microsoft paint it’s a
simulated painting environment
the painter’s job is to produce an image
that looks as real as possible
the final state of this painting
environment is sent as an image
to another computer program in the top
which i call the critic
the critic’s job is to determine how
real that image looks
and to express it in the form of a
reward
which is sent back to the painter the
reward is a single number that expresses
how
real the critic thinks that image looks
and this cycle is repeated millions or
billions of times
until the painter gets really good at
producing images
and the critic gets good at determining
what looks real and what looks fake
and this process is called adversarial
reinforcement learning
and is powered by machine learning
that’s how the critic and the painter
update themselves over time
more concretely the two computer
programs are implemented by what we call
neural networks
so in each one of these sections i’ve
shown
the neural network corresponding to the
painter and the neural network
corresponding to the critic
a neural network is basically a tunable
mathematical function that transforms
inputs to desirable outputs and it
updates itself slightly over time to do
better better
mappings of this kind
so the painter network takes a partially
complete image as input and it outputs
the actions that it wants the
environment to execute next
and the critic network as i said takes a
complete image as input in the bottom
and it outputs a single number which
expresses how real or fake the image
looks
and the action that the painter sends is
a single brush stroke
so this specifies how thick the brush
stroke is
how much pressure is applied what color
it is and
where it starts and where it ends and by
producing a sequence of these actions it
produces the complete image
now if you train this well enough and
for long enough
you can end up with computer programs
that
produce relatively good looking images
so here we’ve trained the program to
reconstruct the image on the left and
you can see that it’s using this
real robotic arm and a real paintbrush
with real ink
to create reconstructions of that image
that’s not too bad right
now naturally we wanted to see whether
we could scale this idea up even further
so we collected a data set of images of
human faces
and we asked the neural painter
to produce images that look like these
photographs
so all the computer is given is some
virtual paint
it’s given a fixed amount of time and is
given these photographs and is asked to
create something that looks like them
now keep in mind we never show the
computer how a human would draw a
portrait
and we never tell the computer how
humans judge
between real and fake looking portraits
all it’s given is the
the opportunity to try and it’s given
images that look real
and this is what it produces at the end
of that training process
so i’ll give you a few minutes to just
observe what it’s doing
you’ll notice that it starts off with
relatively simple shapes just to convey
the
approximate contours of a face
but it manages to use this brush quite
accurately to
gradually fix over the mistakes it makes
early on
and to eventually end up with an image
that looks kind of real
and the physical properties of the brush
are such that the
final image actually kind of resembles
the kinds of portraits we would create
if we were given just as much time
here’s another example of a portrait
painted by this painter
so you’ll see that it produces a variety
of images
and that it can vary the pose of the
person being painted
and that as i said it can control the
brush with sufficient accuracy
to create a variety of shapes and
textures
now in this example the computer is
generating the image
entirely from its own imagination so to
speak which means that we’re not asking
it to reconstruct a certain painting
we’re just saying give us an image that
looks real to you any image that you
want
and here’s another example of an image
produced by this model
now we were all very excited by these
results but what i really wanted to know
was instead of giving the computer
hundreds of brush strokes to create the
image what if we limited it to only 10
or 20 brush strokes
could it learn to be visually abstract
could it learn to
conceptualize what it means for
something to look like a face
could it be creative in other words
and so that’s what we did we retrained
the painter and the critic
but this time we only allowed it 10 or
20 interactions with the environment
and then this is the kind of portrait it
painted for us
i was super excited when i saw this
result for the first time
you’ll see that the painter uses
very succinct brush strokes to depict
the eyes of the face
it uses a single brush stroke to depict
the red lips
and look at the way it draws the
contours of the face
it’s actually quite reminiscent of how i
would draw a face if i were asked to do
so
you might even notice why it draws that
um
red red line in the third panel but if
you pay attention you’ll see that that
red line eventually becomes the cheek
and the faces
the blush in the piece faces cheeks at
the end
this is pretty remarkable and it shows
the agent’s ability to kind of make
long-term plans
and here are some more of the painters
images notice the picasso-esque image in
the top
left or the really super simple
minimal image in the bottom right one of
my favorites is this one in the left
where it
literally uses just a single dot for
each one of the faces eyes
remember none of this has ever been in
the data given to the
painter during training
all of this not because we made the
networks bigger or gave it more data
but because we gave it a set of tools
namely paint brushes
and we gave it an environment and we
gave it a fixed amount of time
and we asked it to do the best job that
it can
now an interesting question to consider
is whether this
computer program is creative it
certainly produces
interesting looking images i would say
that some of them are even aesthetically
pleasing i’d be happy to have them
on my own walls and it can produce an
infinite variety of them
so you can ask it to continuously paint
paintings and will produce more and more
images of this kind
but is this what we mean by creativity
is this what we
mean when we say an artist is being
creative now regardless of your answer
to that question
i think these demonstrations are
important because
they shed light on the cognitive
capabilities that are actually important
for our own creativity
so what what is a human artist doing
when they’re painting a painting what
objective function are they optimizing
for
a chess player who’s playing chess
what’s going on in their mind
or a driver when driving what’s going on
in their mind
and what i find most exciting is when
ai can discover solutions that were
previously hidden in plain sight
so for instance many of you will have
heard of alphago which was this
state-of-the-art goal playing neural
network
and how in a fabled move 37 it
discovered
a go move that has literally transformed
the way we humans play go now
now this back and forth between human
and machine
is tremendously powerful in my opinion
and we’re beginning to see this even in
the arts so
on this slide i’m showing the works of a
number of painters and artists
who use ai in their day-to-day work to
create pieces that previously would not
have been able to
be created and i’m very much looking
forward to seeing how more of these
innovations
will continue to create human ai
co-creations of new kinds
thank you