A camera that can see around corners

To work safely, self-driving cars must avoid obstacles -- including those just out of sight. And for this to happen, we need technology that sees better than humans can, says electrical engineer David Lindell. Buckle up for a quick, groundbreaking tech demo as Lindell explains the significant and versatile potential of a high-speed camera that can detect objects hidden around corners.

In the future,

self-driving cars will be safer
and more reliable than humans.

But for this to happen,

we need technologies
that allow cars to respond

faster than humans,

we need algorithms
that can drive better than humans

and we need cameras
that can see more than humans can see.

For example, imagine a self-driving car
is about to make a blind turn,

and there's an oncoming car

or perhaps there's a child
about to run into the street.

Fortunately, our future car
will have this superpower,

a camera that can see around corners
to detect these potential hazards.

For the past few years as a PhD student

in the Stanford Computational Imaging Lab,

I've been working on a camera
that can do just this --

a camera that can image objects
hidden around corners

or blocked from direct line of sight.

So let me give you an example
of what our camera can see.

This is an outdoor experiment we conducted

where our camera system is scanning
the side of this building with a laser,

and the scene that we want to capture

is hidden around the corner
behind this curtain.

So our camera system
can't actually see it directly.

And yet, somehow,

our camera can still capture
the 3D geometry of this scene.

So how do we do this?

The magic happens here
in this camera system.

You can think of this
as a type of high-speed camera.

Not one that operates
at 1,000 frames per second,

or even a million frames per second,

but a trillion frames per second.

So fast that it can actually capture
the movement of light itself.

And to give you an example
of just how fast light travels,

let's compare it to the speed
of a fast-running comic book superhero

who can move at up to three times
the speed of sound.

It takes a pulse of light
about 3.3 billionths of a second,

or 3.3 nanoseconds,

to travel the distance of a meter.

Well, in that same time,

our superhero has moved
less than the width of a human hair.

That's pretty fast.

But actually, we need to image much faster

if we want to capture light
moving at subcentimeter scales.

So our camera system can capture photons

at time frames of just
50 trillionths of a second,

or 50 picoseconds.

So we take this ultra-high-speed camera

and we pair it with a laser
that sends out short pulses of light.

Each pulse travels to this visible wall

and some light scatters
back to our camera,

but we also use the wall
to scatter light around the corner

to the hidden object and back.

We repeat this measurement many times

to capture the arrival times
of many photons

from different locations on the wall.

And after we capture
these measurements, we can create

a trillion-frame-per-second
video of the wall.

While this wall may look
ordinary to our own eyes,

at a trillion frames per second,
we can see something truly incredible.

We can actually see waves of light
scattered back from the hidden scene

and splashing against the wall.

And each of these waves
carries information

about the hidden object that sent it.

So we can take these measurements

and pass them into
a reconstruction algorithm

to then recover the 3D geometry
of this hidden scene.

Now I want to show you one more example
of an indoor scene that we captured,

this time with a variety
of different hidden objects.

And these objects
have different appearances,

so they reflect light differently.

For example, this glossy dragon statue
reflects light differently

than the mirror disco ball

or the white discus thrower statue.

And we can actually see the differences
in the reflected light

by visualizing it as this 3D volume,

where we've just taken the video frames
and stacked them together.

And time here is represented
as the depth dimension of this cube.

These bright dots that you see
are reflections of light

from each of the mirrored
facets of the disco ball,

scattering against the wall over time.

The bright streaks of light that you see
arriving soonest in time

are from the glossy dragon statue
that's closest to the wall,

and the other streaks of light come from
reflections of light from the bookcase

and from the statue.

Now, we can also visualize
these measurements frame by frame,

as a video,

to directly see the scattered light.

And again, here we see, first,
reflections of light from the dragon,

closest to the wall,

followed by bright dots
from the disco ball

and other reflections from the bookcase.

And finally, we see the reflected
waves of light from the statue.

These waves of light illuminating the wall

are like fireworks that last
for just trillionths of a second.

And even though these objects
reflect light differently,

we can still reconstruct their shapes.

And this is what you can see
from around the corner.

Now, I want to show you one more example
that's slightly different.

In this video, you see me
dressed in this reflective suit

and our camera system is scanning the wall
at a rate of four times every second.

The suit is reflective,

so we can actually capture enough photons

that we can see where I am
and what I'm doing,

without the camera
actually directly imaging me.

By capturing photons that scatter
from the wall to my tracksuit,

back to the wall and back to the camera,

we can capture this indirect
video in real time.

And we think that this type
of practical non-line-of-sight imaging

could be useful for applications
including for self-driving cars,

but also for biomedical imaging,

where we need to see
into the tiny structures of the body.

And perhaps we could also put
similar camera systems on the robots

that we send to explore other planets.

Now you may have heard
about seeing around corners before,

but what I showed you today
would have been impossible

just two years ago.

For example, we can now image large,
room-sized hidden scenes outdoors

and at real-time rates,

and we've made significant advancements
towards making this a practical technology

that you could actually see
on a car someday.

But of course, there's still
challenges remaining.

For example, can we image
hidden scenes at long distances

where we're collecting
very, very few photons,

with lasers that are low-power
and that are eye-safe.

Or can we create images from photons

that have scattered around many more times

than just a single bounce
around the corner?

Can we take our prototype system
that's, well, currently large and bulky,

and miniaturize it into something
that could be useful

for biomedical imaging

or perhaps a sort of improved
home-security system,

or can we take this new imaging modality
and use it for other applications?

I think it's an exciting new technology

and there could be other things
that we haven't thought of yet

to use it for.

And so, well, a future
with self-driving cars

may seem distant to us now --

we're already developing the technologies

that could make cars safer
and more intelligent.

And with the rapid pace
of scientific discovery and innovation,

you never know what new
and exciting capabilities

could be just around the corner.

(Applause)