A camera that can see around corners David Lindell
Transcriber: Ivana Korom
Reviewer: Krystian Aparta
In the future,
self-driving cars will be safer
and more reliable than humans.
But for this to happen,
we need technologies
that allow cars to respond
faster than humans,
we need algorithms
that can drive better than humans
and we need cameras
that can see more than humans can see.
For example, imagine a self-driving car
is about to make a blind turn,
and there’s an oncoming car
or perhaps there’s a child
about to run into the street.
Fortunately, our future car
will have this superpower,
a camera that can see around corners
to detect these potential hazards.
For the past few years as a PhD student
in the Stanford Computational Imaging Lab,
I’ve been working on a camera
that can do just this –
a camera that can image objects
hidden around corners
or blocked from direct line of sight.
So let me give you an example
of what our camera can see.
This is an outdoor experiment we conducted
where our camera system is scanning
the side of this building with a laser,
and the scene that we want to capture
is hidden around the corner
behind this curtain.
So our camera system
can’t actually see it directly.
And yet, somehow,
our camera can still capture
the 3D geometry of this scene.
So how do we do this?
The magic happens here
in this camera system.
You can think of this
as a type of high-speed camera.
Not one that operates
at 1,000 frames per second,
or even a million frames per second,
but a trillion frames per second.
So fast that it can actually capture
the movement of light itself.
And to give you an example
of just how fast light travels,
let’s compare it to the speed
of a fast-running comic book superhero
who can move at up to three times
the speed of sound.
It takes a pulse of light
about 3.3 billionths of a second,
or 3.3 nanoseconds,
to travel the distance of a meter.
Well, in that same time,
our superhero has moved
less than the width of a human hair.
That’s pretty fast.
But actually, we need to image much faster
if we want to capture light
moving at subcentimeter scales.
So our camera system can capture photons
at time frames of just
50 trillionths of a second,
or 50 picoseconds.
So we take this ultra-high-speed camera
and we pair it with a laser
that sends out short pulses of light.
Each pulse travels to this visible wall
and some light scatters
back to our camera,
but we also use the wall
to scatter light around the corner
to the hidden object and back.
We repeat this measurement many times
to capture the arrival times
of many photons
from different locations on the wall.
And after we capture
these measurements, we can create
a trillion-frame-per-second
video of the wall.
While this wall may look
ordinary to our own eyes,
at a trillion frames per second,
we can see something truly incredible.
We can actually see waves of light
scattered back from the hidden scene
and splashing against the wall.
And each of these waves
carries information
about the hidden object that sent it.
So we can take these measurements
and pass them into
a reconstruction algorithm
to then recover the 3D geometry
of this hidden scene.
Now I want to show you one more example
of an indoor scene that we captured,
this time with a variety
of different hidden objects.
And these objects
have different appearances,
so they reflect light differently.
For example, this glossy dragon statue
reflects light differently
than the mirror disco ball
or the white discus thrower statue.
And we can actually see the differences
in the reflected light
by visualizing it as this 3D volume,
where we’ve just taken the video frames
and stacked them together.
And time here is represented
as the depth dimension of this cube.
These bright dots that you see
are reflections of light
from each of the mirrored
facets of the disco ball,
scattering against the wall over time.
The bright streaks of light that you see
arriving soonest in time
are from the glossy dragon statue
that’s closest to the wall,
and the other streaks of light come from
reflections of light from the bookcase
and from the statue.
Now, we can also visualize
these measurements frame by frame,
as a video,
to directly see the scattered light.
And again, here we see, first,
reflections of light from the dragon,
closest to the wall,
followed by bright dots
from the disco ball
and other reflections from the bookcase.
And finally, we see the reflected
waves of light from the statue.
These waves of light illuminating the wall
are like fireworks that last
for just trillionths of a second.
And even though these objects
reflect light differently,
we can still reconstruct their shapes.
And this is what you can see
from around the corner.
Now, I want to show you one more example
that’s slightly different.
In this video, you see me
dressed in this reflective suit
and our camera system is scanning the wall
at a rate of four times every second.
The suit is reflective,
so we can actually capture enough photons
that we can see where I am
and what I’m doing,
without the camera
actually directly imaging me.
By capturing photons that scatter
from the wall to my tracksuit,
back to the wall and back to the camera,
we can capture this indirect
video in real time.
And we think that this type
of practical non-line-of-sight imaging
could be useful for applications
including for self-driving cars,
but also for biomedical imaging,
where we need to see
into the tiny structures of the body.
And perhaps we could also put
similar camera systems on the robots
that we send to explore other planets.
Now you may have heard
about seeing around corners before,
but what I showed you today
would have been impossible
just two years ago.
For example, we can now image large,
room-sized hidden scenes outdoors
and at real-time rates,
and we’ve made significant advancements
towards making this a practical technology
that you could actually see
on a car someday.
But of course, there’s still
challenges remaining.
For example, can we image
hidden scenes at long distances
where we’re collecting
very, very few photons,
with lasers that are low-power
and that are eye-safe.
Or can we create images from photons
that have scattered around many more times
than just a single bounce
around the corner?
Can we take our prototype system
that’s, well, currently large and bulky,
and miniaturize it into something
that could be useful
for biomedical imaging
or perhaps a sort of improved
home-security system,
or can we take this new imaging modality
and use it for other applications?
I think it’s an exciting new technology
and there could be other things
that we haven’t thought of yet
to use it for.
And so, well, a future
with self-driving cars
may seem distant to us now –
we’re already developing the technologies
that could make cars safer
and more intelligent.
And with the rapid pace
of scientific discovery and innovation,
you never know what new
and exciting capabilities
could be just around the corner.
(Applause)