Making Machines Creative.

[Music]

good morning um

today i want to talk about visual

creativity

and what it can teach us about our own

incredible brains

take a look at this painting what is it

about da vinci’s masterful brushstrokes

that evokes a living breathing

three-dimensional object

in our minds

how about this painting which despite

its very different

and perhaps unusual style still looks

kind of real

or this one or even this one

by three-year-old teresa it still looks

like a face right

how does that work now

we humans we’ve been using physical

tools like

brushes and chisels to reconstruct

our sensory observations in the form of

paintings and sculptures

for at least 60 000 years this

incredible cave painting for instance is

thought to be 17 000 years old

that’s thousands of years before we had

domesticated sheep and goats even

in fact we’ve been doing this

abstracting away from observations and

creating them in the form of paintings

for so long that it’s thought to be well

before

the earliest signs of any meaningful

written language even

take a look at this cave painting where

the artist

depicts this battle scene using a few

brush strokes

no more than say 10 or 20.

now many people believe that it is

precisely this ability to abstract away

from reality

and to conceptualize that reality that

underpins

the cognitive abilities that make us

uniquely intelligent as human beings

but it largely remains a mystery how

exactly we do this

and the field of artificial intelligence

or ai

concerns itself with this question

basically

what is intelligence what is it composed

of

how does it come to be and how can we

recreate it

i’m an ai scientist myself and i’ve

spent most of my career thinking about

how visual conceptualization

and visual abstraction can be recreated

now as you know and as you heard ai has

been making great progress over the past

couple of decades

we now have computer programs that can

play chess or go as well as the best

humans

and even self-driving cars seem within

reach

what i’m going to talk about today is

some of the early steps we’ve taken

towards getting computers to be visually

creative as well

now to do this we use computers heavily

and i’d like to begin by going over what

a computer actually is

so a computer in my mind is a simple

device that transforms

inputs to outputs and

the way we program a computer is by

giving it a list of instructions

or an algorithm to specify how we want

those inputs to be transformed into

outputs

in this example the inputs are

two numbers and what we want the

computer to do

is to output the sum and the algorithm

that’s run on the computer

is the summation algorithm now what

makes computers really useful

is that they are extremely fast and they

are extremely accurate

so they do those instructions very very

quickly and very accurately

but it turns out that there are certain

input output transformations

or certain tasks that we really care

about but no computer scientist has

succeeded in writing an

algorithm for it from scratch and only

from their own minds

for instance consider the case of

translation

where you want to go from one language

to another language now of course there

are many

human beings that know these two

languages and can translate

but none of them would be able to

describe how they do it with sufficient

formality to be able to express it to a

computer

and that’s where the idea of learning

comes in you will have heard of the term

machine learning

where the idea is that if we give the

computer enough examples of sentences in

both languages

it will by itself learn how to do this

transformation

and indeed today we now have systems

that can classify

pictures of cats and dogs and translate

sentences from languages into each other

with

fairly good accuracy now

the question that i asked um with my

colleagues a couple of years ago

was could we use this paradigm to get

computers to be creative

so if i consider the paintbrush and

paint

to be inputs to a computer can i get it

to

output visually interesting conceptually

abstract

outputs at the other end

now we know that computers can be used

to create images that’s what digital art

is all about

but in normal digital art the human is

specifying exactly what they want the

computer to do the algorithm is actually

being provided by the human

the question here is can the computer

learn that algorithm by itself

through trial and error kind of figure

out

what kinds of images are visually

appealing and conceptually abstract

so i’m going to dive into the setup that

we actually used

let’s begin with a painter in the bottom

right so the painter is a computer

program

that sends actions to another computer

program

in the form of brush strokes this

second thing which i call the canvas

here think of it as something like adobe

photoshop or microsoft paint it’s a

simulated painting environment

the painter’s job is to produce an image

that looks as real as possible

the final state of this painting

environment is sent as an image

to another computer program in the top

which i call the critic

the critic’s job is to determine how

real that image looks

and to express it in the form of a

reward

which is sent back to the painter the

reward is a single number that expresses

how

real the critic thinks that image looks

and this cycle is repeated millions or

billions of times

until the painter gets really good at

producing images

and the critic gets good at determining

what looks real and what looks fake

and this process is called adversarial

reinforcement learning

and is powered by machine learning

that’s how the critic and the painter

update themselves over time

more concretely the two computer

programs are implemented by what we call

neural networks

so in each one of these sections i’ve

shown

the neural network corresponding to the

painter and the neural network

corresponding to the critic

a neural network is basically a tunable

mathematical function that transforms

inputs to desirable outputs and it

updates itself slightly over time to do

better better

mappings of this kind

so the painter network takes a partially

complete image as input and it outputs

the actions that it wants the

environment to execute next

and the critic network as i said takes a

complete image as input in the bottom

and it outputs a single number which

expresses how real or fake the image

looks

and the action that the painter sends is

a single brush stroke

so this specifies how thick the brush

stroke is

how much pressure is applied what color

it is and

where it starts and where it ends and by

producing a sequence of these actions it

produces the complete image

now if you train this well enough and

for long enough

you can end up with computer programs

that

produce relatively good looking images

so here we’ve trained the program to

reconstruct the image on the left and

you can see that it’s using this

real robotic arm and a real paintbrush

with real ink

to create reconstructions of that image

that’s not too bad right

now naturally we wanted to see whether

we could scale this idea up even further

so we collected a data set of images of

human faces

and we asked the neural painter

to produce images that look like these

photographs

so all the computer is given is some

virtual paint

it’s given a fixed amount of time and is

given these photographs and is asked to

create something that looks like them

now keep in mind we never show the

computer how a human would draw a

portrait

and we never tell the computer how

humans judge

between real and fake looking portraits

all it’s given is the

the opportunity to try and it’s given

images that look real

and this is what it produces at the end

of that training process

so i’ll give you a few minutes to just

observe what it’s doing

you’ll notice that it starts off with

relatively simple shapes just to convey

the

approximate contours of a face

but it manages to use this brush quite

accurately to

gradually fix over the mistakes it makes

early on

and to eventually end up with an image

that looks kind of real

and the physical properties of the brush

are such that the

final image actually kind of resembles

the kinds of portraits we would create

if we were given just as much time

here’s another example of a portrait

painted by this painter

so you’ll see that it produces a variety

of images

and that it can vary the pose of the

person being painted

and that as i said it can control the

brush with sufficient accuracy

to create a variety of shapes and

textures

now in this example the computer is

generating the image

entirely from its own imagination so to

speak which means that we’re not asking

it to reconstruct a certain painting

we’re just saying give us an image that

looks real to you any image that you

want

and here’s another example of an image

produced by this model

now we were all very excited by these

results but what i really wanted to know

was instead of giving the computer

hundreds of brush strokes to create the

image what if we limited it to only 10

or 20 brush strokes

could it learn to be visually abstract

could it learn to

conceptualize what it means for

something to look like a face

could it be creative in other words

and so that’s what we did we retrained

the painter and the critic

but this time we only allowed it 10 or

20 interactions with the environment

and then this is the kind of portrait it

painted for us

i was super excited when i saw this

result for the first time

you’ll see that the painter uses

very succinct brush strokes to depict

the eyes of the face

it uses a single brush stroke to depict

the red lips

and look at the way it draws the

contours of the face

it’s actually quite reminiscent of how i

would draw a face if i were asked to do

so

you might even notice why it draws that

um

red red line in the third panel but if

you pay attention you’ll see that that

red line eventually becomes the cheek

and the faces

the blush in the piece faces cheeks at

the end

this is pretty remarkable and it shows

the agent’s ability to kind of make

long-term plans

and here are some more of the painters

images notice the picasso-esque image in

the top

left or the really super simple

minimal image in the bottom right one of

my favorites is this one in the left

where it

literally uses just a single dot for

each one of the faces eyes

remember none of this has ever been in

the data given to the

painter during training

all of this not because we made the

networks bigger or gave it more data

but because we gave it a set of tools

namely paint brushes

and we gave it an environment and we

gave it a fixed amount of time

and we asked it to do the best job that

it can

now an interesting question to consider

is whether this

computer program is creative it

certainly produces

interesting looking images i would say

that some of them are even aesthetically

pleasing i’d be happy to have them

on my own walls and it can produce an

infinite variety of them

so you can ask it to continuously paint

paintings and will produce more and more

images of this kind

but is this what we mean by creativity

is this what we

mean when we say an artist is being

creative now regardless of your answer

to that question

i think these demonstrations are

important because

they shed light on the cognitive

capabilities that are actually important

for our own creativity

so what what is a human artist doing

when they’re painting a painting what

objective function are they optimizing

for

a chess player who’s playing chess

what’s going on in their mind

or a driver when driving what’s going on

in their mind

and what i find most exciting is when

ai can discover solutions that were

previously hidden in plain sight

so for instance many of you will have

heard of alphago which was this

state-of-the-art goal playing neural

network

and how in a fabled move 37 it

discovered

a go move that has literally transformed

the way we humans play go now

now this back and forth between human

and machine

is tremendously powerful in my opinion

and we’re beginning to see this even in

the arts so

on this slide i’m showing the works of a

number of painters and artists

who use ai in their day-to-day work to

create pieces that previously would not

have been able to

be created and i’m very much looking

forward to seeing how more of these

innovations

will continue to create human ai

co-creations of new kinds

thank you