Jeff Dean AI isnt as smart as you think but it could be TED
Hi, I’m Jeff.
I lead AI Research and Health at Google.
I joined Google more than 20 years ago,
when we were all wedged
into a tiny office space,
above what’s now a T-Mobile store
in downtown Palo Alto.
I’ve seen a lot of computing
transformations in that time,
and in the last decade, we’ve seen AI
be able to do tremendous things.
But we’re still doing it
all wrong in many ways.
That’s what I want
to talk to you about today.
But first, let’s talk
about what AI can do.
So in the last decade,
we’ve seen tremendous progress
in how AI can help computers see,
understand language,
understand speech better than ever before.
Things that we couldn’t do
before, now we can do.
If you think about computer vision alone,
just in the last 10 years,
computers have effectively
developed the ability to see;
10 years ago, they couldn’t see,
now they can see.
You can imagine this has had
a transformative effect
on what we can do with computers.
So let’s look at a couple
of the great applications
enabled by these capabilities.
We can better predict flooding,
keep everyone safe,
using machine learning.
We can translate over 100 languages
so we all can communicate better,
and better predict and diagnose disease,
where everyone gets
the treatment that they need.
So let’s look at two key components
that underlie the progress
in AI systems today.
The first is neural networks,
a breakthrough approach to solving
some of these difficult problems
that has really shone
in the last 15 years.
But they’re not a new idea.
And the second is computational power.
It actually takes a lot
of computational power
to make neural networks
able to really sing,
and in the last 15 years,
we’ve been able to halve that,
and that’s partly what’s enabled
all this progress.
But at the same time,
I think we’re doing several things wrong,
and that’s what I want
to talk to you about
at the end of the talk.
First, a bit of a history lesson.
So for decades,
almost since the very
beginning of computing,
people have wanted
to be able to build computers
that could see, understand language,
understand speech.
The earliest approaches
to this, generally,
people were trying to hand-code
all the algorithms
that you need to accomplish
those difficult tasks,
and it just turned out
to not work very well.
But in the last 15 years,
a single approach
unexpectedly advanced all these different
problem spaces all at once:
neural networks.
So neural networks are not a new idea.
They’re kind of loosely based
on some of the properties
that are in real neural systems.
And many of the ideas
behind neural networks
have been around since the 1960s and 70s.
A neural network is what it sounds like,
a series of interconnected
artificial neurons
that loosely emulate the properties
of your real neurons.
An individual neuron
in one of these systems
has a set of inputs,
each with an associated weight,
and the output of a neuron
is a function of those inputs
multiplied by those weights.
So pretty simple,
and lots and lots of these work together
to learn complicated things.
So how do we actually learn
in a neural network?
It turns out the learning process
consists of repeatedly making
tiny little adjustments
to the weight values,
strengthening the influence
of some things,
weakening the influence of others.
By driving the overall system
towards desired behaviors,
these systems can be trained
to do really complicated things,
like translate
from one language to another,
detect what kind
of objects are in a photo,
all kinds of complicated things.
I first got interested in neural networks
when I took a class on them
as an undergraduate in 1990.
At that time,
neural networks showed
impressive results on tiny problems,
but they really couldn’t scale to do
real-world important tasks.
But I was super excited.
(Laughter)
I felt maybe we just needed
more compute power.
And the University of Minnesota
had a 32-processor machine.
I thought, “With more compute power,
boy, we could really make
neural networks really sing.”
So I decided to do a senior thesis
on parallel training of neural networks,
the idea of using processors in a computer
or in a computer system
to all work toward the same task,
that of training neural networks.
32 processors, wow,
we’ve got to be able
to do great things with this.
But I was wrong.
Turns out we needed about a million times
as much computational power
as we had in 1990
before we could actually get
neural networks to do impressive things.
But starting around 2005,
thanks to the computing progress
of Moore’s law,
we actually started to have
that much computing power,
and researchers in a few universities
around the world started to see success
in using neural networks for a wide
variety of different kinds of tasks.
I and a few others at Google
heard about some of these successes,
and we decided to start a project
to train very large neural networks.
One system that we trained,
we trained with 10 million
randomly selected frames
from YouTube videos.
The system developed the capability
to recognize all kinds
of different objects.
And it being YouTube, of course,
it developed the ability
to recognize cats.
YouTube is full of cats.
(Laughter)
But what made that so remarkable
is that the system was never told
what a cat was.
So using just patterns in data,
the system honed in on the concept
of a cat all on its own.
All of this occurred at the beginning
of a decade-long string of successes,
of using neural networks
for a huge variety of tasks,
at Google and elsewhere.
Many of the things you use every day,
things like better speech
recognition for your phone,
improved understanding
of queries and documents
for better search quality,
better understanding of geographic
information to improve maps,
and so on.
Around that time,
we also got excited about how we could
build hardware that was better tailored
to the kinds of computations
neural networks wanted to do.
Neural network computations
have two special properties.
The first is they’re very tolerant
of reduced precision.
Couple of significant digits,
you don’t need six or seven.
And the second is that all the
algorithms are generally composed
of different sequences
of matrix and vector operations.
So if you can build a computer
that is really good at low-precision
matrix and vector operations
but can’t do much else,
that’s going to be great
for neural-network computation,
even though you can’t use it
for a lot of other things.
And if you build such things,
people will find amazing uses for them.
This is the first one we built, TPU v1.
“TPU” stands for Tensor Processing Unit.
These have been used for many years
behind every Google search,
for translation,
in the DeepMind AlphaGo matches,
so Lee Sedol and Ke Jie
maybe didn’t realize,
but they were competing
against racks of TPU cards.
And we’ve built a bunch
of subsequent versions of TPUs
that are even better and more exciting.
But despite all these successes,
I think we’re still doing
many things wrong,
and I’ll tell you about three
key things we’re doing wrong,
and how we’ll fix them.
The first is that most
neural networks today
are trained to do one thing,
and one thing only.
You train it for a particular task
that you might care deeply about,
but it’s a pretty heavyweight activity.
You need to curate a data set,
you need to decide
what network architecture you’ll use
for this problem,
you need to initialize the weights
with random values,
apply lots of computation
to make adjustments to the weights.
And at the end, if you’re lucky,
you end up with a model
that is really good
at that task you care about.
But if you do this over and over,
you end up with thousands
of separate models,
each perhaps very capable,
but separate for all the different
tasks you care about.
But think about how people learn.
In the last year, many of us
have picked up a bunch of new skills.
I’ve been honing my gardening skills,
experimenting with vertical
hydroponic gardening.
To do that, I didn’t need to relearn
everything I already knew about plants.
I was able to know
how to put a plant in a hole,
how to pour water, that plants need sun,
and leverage that
in learning this new skill.
Computers can work
the same way, but they don’t today.
If you train a neural
network from scratch,
it’s effectively like forgetting
your entire education
every time you try to do something new.
That’s crazy, right?
So instead, I think we can
and should be training
multitask models that can do
thousands or millions of different tasks.
Each part of that model would specialize
in different kinds of things.
And then, if we have a model
that can do a thousand things,
and the thousand and first
thing comes along,
we can leverage
the expertise we already have
in the related kinds of things
so that we can more quickly be able
to do this new task,
just like you, if you’re confronted
with some new problem,
you quickly identify
the 17 things you already know
that are going to be helpful
in solving that problem.
Second problem is that most
of our models today
deal with only a single
modality of data –
with images, or text or speech,
but not all of these all at once.
But think about how you
go about the world.
You’re continuously using all your senses
to learn from, react to,
figure out what actions
you want to take in the world.
Makes a lot more sense to do that,
and we can build models in the same way.
We can build models that take in
these different modalities of input data,
text, images, speech,
but then fuse them together,
so that regardless of whether the model
sees the word “leopard,”
sees a video of a leopard
or hears someone say the word “leopard,”
the same response
is triggered inside the model:
the concept of a leopard
can deal with different
kinds of input data,
even nonhuman inputs,
like genetic sequences,
3D clouds of points,
as well as images, text and video.
The third problem
is that today’s models are dense.
There’s a single model,
the model is fully activated
for every task,
for every example
that we want to accomplish,
whether that’s a really simple
or a really complicated thing.
This, too, is unlike
how our own brains work.
Different parts of our brains
are good at different things,
and we’re continuously calling
upon the pieces of them
that are relevant for the task at hand.
For example, nervously watching
a garbage truck
back up towards your car,
the part of your brain that thinks
about Shakespearean sonnets
is probably inactive.
(Laughter)
AI models can work the same way.
Instead of a dense model,
we can have one
that is sparsely activated.
So for particular different tasks,
we call upon different parts of the model.
During training, the model can also learn
which parts are good at which things,
to continuously identify what parts
it wants to call upon
in order to accomplish a new task.
The advantage of this is we can have
a very high-capacity model,
but it’s very efficient,
because we’re only calling
upon the parts that we need
for any given task.
So fixing these three things, I think,
will lead to a more powerful AI system:
instead of thousands of separate models,
train a handful of general-purpose models
that can do thousands
or millions of things.
Instead of dealing with single modalities,
deal with all modalities,
and be able to fuse them together.
And instead of dense models,
use sparse, high-capacity models,
where we call upon the relevant
bits as we need them.
We’ve been building a system
that enables these kinds of approaches,
and we’ve been calling
the system “Pathways.”
So the idea is this model
will be able to do
thousands or millions of different tasks,
and then, we can incrementally
add new tasks,
and it can deal
with all modalities at once,
and then incrementally learn
new tasks as needed
and call upon the relevant
bits of the model
for different examples or tasks.
And we’re pretty excited about this,
we think this is going
to be a step forward
in how we build AI systems.
But I also wanted
to touch on responsible AI.
We clearly need to make sure
that this vision of powerful AI systems
benefits everyone.
These kinds of models raise
important new questions
about how do we build them with fairness,
interpretability, privacy and security,
for all users in mind.
For example, if we’re going
to train these models
on thousands or millions of tasks,
we’ll need to be able to train them
on large amounts of data.
And we need to make sure that data
is thoughtfully collected
and is representative of different
communities and situations
all around the world.
And data concerns are only
one aspect of responsible AI.
We have a lot of work to do here.
So in 2018, Google published
this set of AI principles
by which we think about developing
these kinds of technology.
And these have helped guide us
in how we do research in this space,
how we use AI in our products.
And I think it’s a really helpful
and important framing
for how to think about these deep
and complex questions
about how we should
be using AI in society.
We continue to update these
as we learn more.
Many of these kinds of principles
are active areas of research –
super important area.
Moving from single-purpose systems
that kind of recognize patterns in data
to these kinds of general-purpose
intelligent systems
that have a deeper
understanding of the world
will really enable us to tackle
some of the greatest problems
humanity faces.
For example,
we’ll be able to diagnose more disease;
we’ll be able to engineer better medicines
by infusing these models
with knowledge of chemistry and physics;
we’ll be able to advance
educational systems
by providing more individualized tutoring
to help people learn
in new and better ways;
we’ll be able to tackle
really complicated issues,
like climate change,
and perhaps engineering
of clean energy solutions.
So really, all of these kinds of systems
are going to be requiring
the multidisciplinary expertise
of people all over the world.
So connecting AI
with whatever field you are in,
in order to make progress.
So I’ve seen a lot
of advances in computing,
and how computing, over the past decades,
has really helped millions of people
better understand the world around them.
And AI today has the potential
to help billions of people.
We truly live in exciting times.
Thank you.
(Applause)
Chris Anderson: Thank you so much.
I want to follow up on a couple things.
This is what I heard.
Most people’s traditional picture of AI
is that computers recognize
a pattern of information,
and with a bit of machine learning,
they can get really good at that,
better than humans.
What you’re saying is those patterns
are no longer the atoms
that AI is working with,
that it’s much richer-layered concepts
that can include all manners
of types of things
that go to make up a leopard, for example.
So what could that lead to?
Give me an example
of when that AI is working,
what do you picture happening in the world
in the next five or 10 years
that excites you?
Jeff Dean: I think
the grand challenge in AI
is how do you generalize
from a set of tasks
you already know how to do
to new tasks,
as easily and effortlessly as possible.
And the current approach of training
separate models for everything
means you need lots of data
about that particular problem,
because you’re effectively trying
to learn everything
about the world
and that problem, from nothing.
But if you can build these systems
that already are infused with how to do
thousands and millions of tasks,
then you can effectively
teach them to do a new thing
with relatively few examples.
So I think that’s the real hope,
that you could then have a system
where you just give it five examples
of something you care about,
and it learns to do that new task.
CA: You can do a form
of self-supervised learning
that is based on remarkably
little seeding.
JD: Yeah, as opposed to needing
10,000 or 100,000 examples
to figure everything in the world out.
CA: Aren’t there kind of terrifying
unintended consequences
possible, from that?
JD: I think it depends
on how you apply these systems.
It’s very clear that AI
can be a powerful system for good,
or if you apply it in ways
that are not so great,
it can be a negative consequence.
So I think that’s why it’s important
to have a set of principles
by which you look at potential uses of AI
and really are careful and thoughtful
about how you consider applications.
CA: One of the things
people worry most about
is that, if AI is so good at learning
from the world as it is,
it’s going to carry forward
into the future
aspects of the world as it is
that actually aren’t right, right now.
And there’s obviously been
a huge controversy about that
recently at Google.
Some of those principles
of AI development,
you’ve been challenged that you’re not
actually holding to them.
Not really interested to hear
about comments on a specific case,
but … are you really committed?
How do we know that you are
committed to these principles?
Is that just PR, or is that real,
at the heart of your day-to-day?
JD: No, that is absolutely real.
Like, we have literally hundreds of people
working on many of these
related research issues,
because many of those
things are research topics
in their own right.
How do you take data from the real world,
that is the world as it is,
not as we would like it to be,
and how do you then use that
to train a machine-learning model
and adapt the data bit of the scene
or augment the data with additional data
so that it can better reflect
the values we want the system to have,
not the values that it sees in the world?
CA: But you work for Google,
Google is funding the research.
How do we know that the main values
that this AI will build
are for the world,
and not, for example, to maximize
the profitability of an ad model?
When you know everything
there is to know about human attention,
you’re going to know so much
about the little wriggly,
weird, dark parts of us.
In your group, are there rules
about how you hold off,
church-state wall
between a sort of commercial push,
“You must do it for this purpose,”
so that you can inspire
your engineers and so forth,
to do this for the world, for all of us.
JD: Yeah, our research group
does collaborate
with a number of groups across Google,
including the Ads group,
the Search group, the Maps group,
so we do have some collaboration,
but also a lot of basic research
that we publish openly.
We’ve published more
than 1,000 papers last year
in different topics,
including the ones you discussed,
about fairness, interpretability
of the machine-learning models,
things that are super important,
and we need to advance
the state of the art in this
in order to continue to make progress
to make sure these models
are developed safely and responsibly.
CA: It feels like we’re at a time
when people are concerned
about the power of the big tech companies,
and it’s almost, if there was ever
a moment to really show the world
that this is being done
to make a better future,
that is actually key to Google’s future,
as well as all of ours.
JD: Indeed.
CA: It’s very good to hear you
come and say that, Jeff.
Thank you so much for coming here to TED.
JD: Thank you.
(Applause)