1
00:00:15,088 --> 00:00:21,142
So over the past few centuries,
microscopes have revolutionized our world.

2
00:00:23,204 --> 00:00:27,563
They revealed to us a tiny world
of objects, life and structures

3
00:00:27,563 --> 00:00:30,567
that are too small for us
to see with our naked eyes.

4
00:00:30,567 --> 00:00:34,064
They are a tremendous contribution
to science and technology.

5
00:00:34,064 --> 00:00:37,839
Today I'd like to introduce you
to a new type of microscope,

6
00:00:37,839 --> 00:00:40,230
a microscope for changes.

7
00:00:40,230 --> 00:00:43,438
It doesn't use optics
like a regular microscope

8
00:00:43,438 --> 00:00:45,350
to make small objects bigger,

9
00:00:45,350 --> 00:00:49,673
but instead it uses a video camera
and image processing

10
00:00:49,673 --> 00:00:54,387
to reveal to us the tiniest motions
and color changes in objects and people,

11
00:00:54,526 --> 00:00:58,368
changes that are impossible
for us to see with our naked eyes.

12
00:00:58,906 --> 00:01:02,970
And it lets us look at our world
in a completely new way.

13
00:01:03,295 --> 00:01:05,715
So what do I mean by color changes?

14
00:01:07,093 --> 00:01:09,896
Our skin, for example,
changes its color very slightly

15
00:01:09,896 --> 00:01:12,211
when the blood flows under it.

16
00:01:12,211 --> 00:01:14,443
That change is incredibly subtle,

17
00:01:14,443 --> 00:01:16,872
which is why, when you look
at other people,

18
00:01:16,872 --> 00:01:19,171
when you look at the person
sitting next to you,

19
00:01:19,171 --> 00:01:21,920
you don't see their skin
or their face changing color.

20
00:01:21,920 --> 00:01:27,150
When we look at this video of Steve here,
it appears to us like a static picture,

21
00:01:27,849 --> 00:01:31,292
but once we look at this video
through our new, special microscope,

22
00:01:31,292 --> 00:01:34,550
suddenly we see
a completely different image.

23
00:01:35,276 --> 00:01:39,200
What you see here are small changes
in the color of Steve's skin,

24
00:01:39,200 --> 00:01:43,087
magnified 100 times
so that they become visible.

25
00:01:43,539 --> 00:01:46,259
We can actually see a human pulse.

26
00:01:46,554 --> 00:01:49,729
We can see how fast
Steve's heart is beating,

27
00:01:49,729 --> 00:01:53,994
but we can also see the actual way
that the blood flows in his face.

28
00:01:55,152 --> 00:01:58,055
And we can do that not just
to visualize the pulse,

29
00:01:58,055 --> 00:02:01,448
but also to actually recover
our heart rates,

30
00:02:01,448 --> 00:02:03,510
and measure our heart rates.

31
00:02:03,510 --> 00:02:07,692
And we can do it with regular cameras
and without touching the patients.

32
00:02:07,692 --> 00:02:12,768
So here you see the pulse and heart rate
we extracted from a neonatal baby

33
00:02:12,768 --> 00:02:16,250
from a video we took
with a regular DSLR camera,

34
00:02:16,250 --> 00:02:18,364
and the heart rate measurement we get

35
00:02:18,364 --> 00:02:22,588
is as accurate as the one you'd get
with a standard monitor in a hospital.

36
00:02:23,384 --> 00:02:26,457
And it doesn't even have to be
a video we recorded.

37
00:02:26,457 --> 00:02:29,391
We can do it essentially
with other videos as well.

38
00:02:29,391 --> 00:02:32,565
So I just took a short clip
from "Batman Begins" here

39
00:02:32,565 --> 00:02:35,226
just to show Christian Bale's pulse.

40
00:02:35,226 --> 00:02:37,282
(Laughter)

41
00:02:37,282 --> 00:02:39,417
And you know, presumably
he's wearing makeup,

42
00:02:39,417 --> 00:02:41,389
the lighting here is kind of challenging,

43
00:02:41,389 --> 00:02:44,392
but still, just from the video,
we're able to extract his pulse

44
00:02:44,392 --> 00:02:46,224
and show it quite well.

45
00:02:46,224 --> 00:02:47,995
So how do we do all that?

46
00:02:47,995 --> 00:02:52,315
We basically analyze the changes
in the light that are recorded

47
00:02:52,315 --> 00:02:54,928
at every pixel in the video over time,

48
00:02:54,928 --> 00:02:56,648
and then we crank up those changes.

49
00:02:56,648 --> 00:02:59,494
We make them bigger
so that we can see them.

50
00:02:59,494 --> 00:03:01,576
The tricky part is that those signals,

51
00:03:01,576 --> 00:03:04,359
those changes that we're after,
are extremely subtle,

52
00:03:04,359 --> 00:03:07,353
so we have to be very careful
when you try to separate them

53
00:03:07,353 --> 00:03:10,240
from noise that always exists in videos.

54
00:03:10,240 --> 00:03:13,682
So we use some clever
image processing techniques

55
00:03:13,682 --> 00:03:17,736
to get a very accurate measurement
of the color at each pixel in the video,

56
00:03:17,736 --> 00:03:20,569
and then the way
the color changes over time,

57
00:03:20,569 --> 00:03:23,227
and then we amplify those changes.

58
00:03:23,227 --> 00:03:27,081
We make them bigger to create those types
of enhanced videos, or magnified videos,

59
00:03:27,081 --> 00:03:29,552
that actually show us those changes.

60
00:03:32,007 --> 00:03:36,227
But it turns out we can do that
not just to show tiny changes in color,

61
00:03:36,227 --> 00:03:38,379
but also tiny motions,

62
00:03:38,379 --> 00:03:42,064
and that's because the light
that gets recorded in our cameras

63
00:03:42,064 --> 00:03:45,189
will change not only if the color
of the object changes,

64
00:03:45,189 --> 00:03:47,305
but also if the object moves.

65
00:03:47,905 --> 00:03:52,543
So this is my daughter
when she was about two months old.

66
00:03:56,157 --> 00:03:59,397
It's a video I recorded
about three years ago.

67
00:03:59,397 --> 00:04:02,807
And as new parents, we all want
to make sure our babies are healthy,

68
00:04:02,807 --> 00:04:05,373
that they're breathing,
that they're alive, of course.

69
00:04:05,373 --> 00:04:07,465
So I too got one of those baby monitors

70
00:04:07,465 --> 00:04:10,072
so that I could see my daughter
when she was asleep.

71
00:04:10,072 --> 00:04:13,590
And this is pretty much what you'll see
with a standard baby monitor.

72
00:04:13,590 --> 00:04:15,686
You can see the baby's sleeping,

73
00:04:15,686 --> 00:04:17,729
but there's not too much information
there.

74
00:04:17,729 --> 00:04:19,516
There's not too much we can see.

75
00:04:19,516 --> 00:04:22,358
Wouldn't it be better,
or more informative, or more useful,

76
00:04:22,358 --> 00:04:25,261
if instead we could look
at the view like this.

77
00:04:25,261 --> 00:04:30,310
So here I took the motions
and I magnified them 30 times,

78
00:04:31,217 --> 00:04:33,708
and then I could clearly see
that my daughter

79
00:04:33,708 --> 00:04:35,428
was indeed alive and breathing.

80
00:04:35,428 --> 00:04:37,565
(Laughter)

81
00:04:38,092 --> 00:04:39,891
Here is a side-by-side comparison.

82
00:04:39,891 --> 00:04:42,440
So again, in the source video,
in the original video,

83
00:04:42,440 --> 00:04:44,260
there's not too much we can see,

84
00:04:44,260 --> 00:04:48,212
but once we magnify the motions,
the breathing becomes much more visible.

85
00:04:48,212 --> 00:04:50,801
And it turns out,
there's a lot of phenomena

86
00:04:50,801 --> 00:04:54,474
we can reveal and magnify
with our new motion microscope.

87
00:04:54,474 --> 00:04:58,909
We can see how our veins and arteries
are pulsing in our bodies.

88
00:04:59,752 --> 00:05:02,560
We can see that our eyes
are constantly moving

89
00:05:02,560 --> 00:05:04,776
in this wobbly motion.

90
00:05:04,776 --> 00:05:06,354
And that's actually my eye,

91
00:05:06,354 --> 00:05:09,414
and again this video was taken
right after my daughter was born,

92
00:05:09,414 --> 00:05:13,103
so you can see I wasn't getting
too much sleep. (Laughter)

93
00:05:13,539 --> 00:05:16,403
Even when a person is sitting still,

94
00:05:16,403 --> 00:05:18,997
there's a lot of information
we can extract

95
00:05:18,997 --> 00:05:21,989
about their breathing patterns,
small facial expressions.

96
00:05:22,672 --> 00:05:24,623
Maybe we could use those motions

97
00:05:24,623 --> 00:05:27,488
to tell us something about
our thoughts or our emotions.

98
00:05:29,003 --> 00:05:32,385
We can also magnify
small mechanical movements,

99
00:05:32,385 --> 00:05:34,337
like vibrations in engines,

100
00:05:34,337 --> 00:05:38,017
that can help engineers detect
and diagnose machinery problems,

101
00:05:40,130 --> 00:05:45,547
or see how our buildings and structures
sway in the wind and react to forces.

102
00:05:45,547 --> 00:05:50,333
Those are all things that our society
knows how to measure in various ways,

103
00:05:50,333 --> 00:05:52,875
but measuring those motions is one thing,

104
00:05:52,875 --> 00:05:55,479
and actually seeing those motions
as they happen

105
00:05:55,479 --> 00:05:57,614
is a whole different thing.

106
00:05:58,450 --> 00:06:02,021
And ever since we discovered
this new technology,

107
00:06:02,021 --> 00:06:03,723
we made our code available online

108
00:06:03,723 --> 00:06:06,269
so that others could use
and experiment with it.

109
00:06:08,005 --> 00:06:09,809
It's very simple to use.

110
00:06:09,809 --> 00:06:11,853
It can work on your own videos.

111
00:06:11,853 --> 00:06:15,357
Our collaborators at Quantum Research
even created this nice website

112
00:06:15,357 --> 00:06:18,036
where you can upload your videos
and process them online,

113
00:06:18,036 --> 00:06:21,603
so even if you don't have any experience
in computer science or programming,

114
00:06:21,603 --> 00:06:24,509
you can still very easily experiment
with this new microscope.

115
00:06:24,509 --> 00:06:26,941
And I'd like to show you
just a couple of examples

116
00:06:26,941 --> 00:06:28,919
of what others have done with it.

117
00:06:32,363 --> 00:06:37,259
So this video was made
by a YouTube user called Tamez85.

118
00:06:37,259 --> 00:06:38,807
I don't know who that user is,

119
00:06:38,807 --> 00:06:41,105
but he, or she, used our code

120
00:06:41,105 --> 00:06:43,910
to magnify small belly movements
during pregnancy.

121
00:06:44,933 --> 00:06:46,420
It's kind of creepy.

122
00:06:46,420 --> 00:06:48,818
(Laughter)

123
00:06:48,818 --> 00:06:52,782
People have used it to magnify
pulsing veins in their hands.

124
00:06:53,532 --> 00:06:56,699
And you know it's not real science
unless you use guinea pigs,

125
00:06:58,037 --> 00:07:00,584
and apparently this guinea pig
is called Tiffany,

126
00:07:00,584 --> 00:07:04,007
and this YouTube user claims
it is the first rodent on Earth

127
00:07:04,007 --> 00:07:05,780
that was motion-magnified.

128
00:07:06,604 --> 00:07:08,811
You can also do some art with it.

129
00:07:08,811 --> 00:07:12,123
So this video was sent to me
by a design student at Yale.

130
00:07:12,123 --> 00:07:14,516
She wanted to see
if there's any difference

131
00:07:14,516 --> 00:07:16,072
in the way her classmates move.

132
00:07:16,072 --> 00:07:20,361
She made them all stand still,
and then magnified their motions.

133
00:07:20,361 --> 00:07:23,457
It's like seeing still pictures
come to life.

134
00:07:23,714 --> 00:07:26,077
And the nice thing with all those examples

135
00:07:26,077 --> 00:07:28,315
is that we had nothing to do with them.

136
00:07:28,315 --> 00:07:32,165
We just provided this new tool,
a new way to look at the world,

137
00:07:32,165 --> 00:07:36,683
and then people find other interesting,
new and creative ways of using it.

138
00:07:37,735 --> 00:07:39,620
But we didn't stop there.

139
00:07:40,943 --> 00:07:44,597
This tool not only allows us
to look at the world in a new way,

140
00:07:44,597 --> 00:07:47,034
it also redefines what we can do

141
00:07:47,034 --> 00:07:50,232
and pushes the limits
of what we can do with our cameras.

142
00:07:50,232 --> 00:07:52,611
So as scientists, we started wondering,

143
00:07:52,611 --> 00:07:56,299
what other types of physical phenomena
produce tiny motions

144
00:07:56,299 --> 00:07:59,212
that we could now use
our cameras to measure?

145
00:07:59,212 --> 00:08:02,635
And one such phenomenon
that we focused on recently is sound.

146
00:08:03,664 --> 00:08:05,963
Sound, as we all know,
is basically changes

147
00:08:05,963 --> 00:08:08,134
in air pressure
that travel through the air.

148
00:08:08,134 --> 00:08:11,857
Those pressure waves hit objects
and they create small vibrations in them,

149
00:08:11,857 --> 00:08:14,519
which is how we hear
and how we record sound.

150
00:08:14,519 --> 00:08:18,294
But it turns out that sound
also produces visual motions.

151
00:08:18,579 --> 00:08:21,303
Those are motions
that are not visible to us

152
00:08:21,303 --> 00:08:24,229
but are visible to a camera
with the right processing.

153
00:08:24,229 --> 00:08:26,045
So here are two examples.

154
00:08:26,045 --> 00:08:29,374
This is me demonstrating
my great singing skills.

155
00:08:30,845 --> 00:08:33,603
(Singing)

156
00:08:33,603 --> 00:08:34,710
(Laughter)

157
00:08:34,710 --> 00:08:37,706
And I took a high-speed video
of my throat while I was humming.

158
00:08:37,706 --> 00:08:39,355
Again, if you stare at that video,

159
00:08:39,355 --> 00:08:41,387
there's not too much
you'll be able to see,

160
00:08:41,387 --> 00:08:45,793
but once we magnify the motions 100 times,
we can see all the motions and ripples

161
00:08:45,793 --> 00:08:49,103
in the neck that are involved
in producing the sound.

162
00:08:49,103 --> 00:08:51,528
That signal is there in that video.

163
00:08:51,528 --> 00:08:54,228
We also know that singers
can break a wine glass

164
00:08:54,228 --> 00:08:56,274
if they hit the correct note.

165
00:08:56,274 --> 00:08:58,325
So here, we're going to play a note

166
00:08:58,325 --> 00:09:00,849
that's in the resonance frequency
of that glass

167
00:09:00,849 --> 00:09:03,125
through a loudspeaker that's next to it.

168
00:09:03,125 --> 00:09:07,568
Once we play that note
and magnify the motions 250 times,

169
00:09:07,568 --> 00:09:10,789
we can very clearly see
how the glass vibrates

170
00:09:10,789 --> 00:09:13,623
and resonates in response to the sound.

171
00:09:14,132 --> 00:09:16,545
It's not something you're used to seeing
every day.

172
00:09:16,545 --> 00:09:19,408
And we actually have the demo
right outside set up,

173
00:09:19,408 --> 00:09:21,300
so I encourage you to stop by,

174
00:09:21,300 --> 00:09:24,347
and just play with it yourself,
you can actually see it live.

175
00:09:24,608 --> 00:09:27,768
But this made us think.
It gave us this crazy idea.

176
00:09:28,078 --> 00:09:32,865
Can we actually invert this process
and recover sound from video

177
00:09:33,454 --> 00:09:37,581
by analyzing the tiny vibrations
that sound waves create in objects,

178
00:09:37,581 --> 00:09:41,898
and essentially convert those
back into the sounds that produced them.

179
00:09:42,548 --> 00:09:46,472
In this way, we can turn
everyday objects into microphones.

180
00:09:47,958 --> 00:09:49,595
So that's exactly what we did.

181
00:09:49,595 --> 00:09:52,462
So here's an empty bag of chips
that was lying on a table,

182
00:09:52,462 --> 00:09:55,234
and we're going to turn that bag of chips
into a microphone

183
00:09:55,234 --> 00:09:57,145
by filming it with a video camera

184
00:09:57,145 --> 00:10:00,914
and analyzing the tiny motions
that sound waves create in it.

185
00:10:01,479 --> 00:10:04,242
So here's the sound
that we played in the room.

186
00:10:04,242 --> 00:10:07,634
(Music: "Mary Had a Little Lamb")

187
00:10:12,476 --> 00:10:15,426
And this is a high-speed video
we recorded of that bag of chips.

188
00:10:15,426 --> 00:10:16,528
Again it's playing.

189
00:10:16,528 --> 00:10:19,886
There's no chance you'll be able
to see anything going on in that video

190
00:10:19,886 --> 00:10:21,000
just by looking at it,

191
00:10:21,000 --> 00:10:23,962
but here's the sound we were able
to recover just by analyzing

192
00:10:23,962 --> 00:10:26,273
the tiny motions in that video.

193
00:10:27,127 --> 00:10:30,494
(Music: "Mary Had a Little Lamb")

194
00:10:44,607 --> 00:10:46,458
I call it -- Thank you.

195
00:10:46,458 --> 00:10:49,328
(Applause)

196
00:10:53,834 --> 00:10:56,140
I call it the visual microphone.

197
00:10:56,140 --> 00:10:59,251
We actually extract audio signals
from video signals.

198
00:10:59,251 --> 00:11:02,435
And just to give you a sense
of the scale of the motions here,

199
00:11:02,435 --> 00:11:06,696
a pretty loud sound will cause
that bag of chips

200
00:11:06,696 --> 00:11:09,266
to move less than a micrometer.

201
00:11:09,807 --> 00:11:12,485
That's one thousandth of a millimeter.

202
00:11:12,485 --> 00:11:16,179
That's how tiny the motions are
that we are now able to pull out

203
00:11:16,179 --> 00:11:19,282
just by observing how light
bounces off objects

204
00:11:19,282 --> 00:11:21,704
and gets recorded by our cameras.

205
00:11:22,208 --> 00:11:25,358
We can recover sounds
from other objects, like plants.

206
00:11:25,986 --> 00:11:29,183
(Music: "Mary Had a Little Lamb")

207
00:11:34,153 --> 00:11:36,456
And we can recover speech as well.

208
00:11:36,456 --> 00:11:38,817
So here's a person speaking in a room.

209
00:11:38,817 --> 00:11:43,632
Voice: Mary had a little lamb
whose fleece was white as snow,

210
00:11:43,632 --> 00:11:47,570
and everywhere that Mary went,
that lamb was sure to go.

211
00:11:48,722 --> 00:11:51,486
Michael Rubinstein: And here's
that speech again recovered

212
00:11:51,486 --> 00:11:54,220
just from this video
of that same bag of chips.

213
00:11:54,220 --> 00:11:59,211
Voice: Mary had a little lamb
whose fleece was white as snow,

214
00:11:59,211 --> 00:12:03,731
and everywhere that Mary went,
that lamb was sure to go.

215
00:12:04,352 --> 00:12:06,907
MR: We used "Mary Had a Little Lamb"

216
00:12:06,907 --> 00:12:09,252
because those are said to be
the first words

217
00:12:09,252 --> 00:12:13,053
that Thomas Edison spoke
into his phonograph in 1877.

218
00:12:13,053 --> 00:12:16,565
It was one of the first
sound recording devices in history.

219
00:12:16,565 --> 00:12:19,842
It basically directed the sounds
onto a diaphragm

220
00:12:19,842 --> 00:12:24,270
that vibrated a needle that essentially
engraved the sound on tinfoil

221
00:12:24,270 --> 00:12:26,565
that was wrapped around the cylinder.

222
00:12:26,565 --> 00:12:29,679
Here's a demonstration of recording

223
00:12:29,679 --> 00:12:32,446
and replaying sound
with Edison's phonograph.

224
00:12:33,549 --> 00:12:36,487
(Video) Voice: Testing, testing,
one two three.

225
00:12:36,487 --> 00:12:39,654
Mary had a little lamb
whose fleece was white as snow,

226
00:12:39,654 --> 00:12:43,491
and everywhere that Mary went,
the lamb was sure to go.

227
00:12:43,491 --> 00:12:46,014
Testing, testing, one two three.

228
00:12:46,014 --> 00:12:50,103
Mary had a little lamb
whose fleece was white as snow,

229
00:12:50,103 --> 00:12:54,235
and everywhere that Mary went,
the lamb was sure to go.

230
00:12:55,719 --> 00:12:59,081
MR: And now, 137 years later,

231
00:13:00,334 --> 00:13:03,492
we're able to get sound
in pretty much similar quality

232
00:13:03,492 --> 00:13:07,559
but by just watching objects
vibrate to sound with cameras,

233
00:13:07,853 --> 00:13:09,952
and we can even do that when the camera

234
00:13:09,952 --> 00:13:13,631
is 15 feet away from the object,
behind soundproof glass.

235
00:13:14,178 --> 00:13:17,475
So this is the sound that we were able
to recover in that case.

236
00:13:17,475 --> 00:13:22,282
Voice: Mary had a little lamb
whose fleece was white as snow,

237
00:13:22,282 --> 00:13:26,993
and everywhere that Mary went,
the lamb was sure to go.

238
00:13:28,111 --> 00:13:31,711
MR: And of course, surveillance is
the first application that comes to mind.

239
00:13:31,711 --> 00:13:33,993
(Laughter)

240
00:13:33,993 --> 00:13:38,055
But it might actually be useful
for other things as well.

241
00:13:38,095 --> 00:13:41,196
Maybe in the future,
we'll be able to use it, for example,

242
00:13:41,196 --> 00:13:43,557
to recover sound across space,

243
00:13:43,557 --> 00:13:46,569
because sound can't travel
in space, but light can.

244
00:13:47,166 --> 00:13:49,525
We've only just begun exploring

245
00:13:49,525 --> 00:13:52,509
other possible uses
for this new technology.

246
00:13:52,509 --> 00:13:55,288
It lets us see physical processes
that we know are there

247
00:13:55,288 --> 00:13:59,575
but that we've never been able
to see with our own eyes until now.

248
00:14:00,677 --> 00:14:01,917
This is our team.

249
00:14:01,917 --> 00:14:04,767
Everything I showed you today
is a result of a collaboration

250
00:14:04,767 --> 00:14:06,864
with this great group
of people you see here,

251
00:14:06,864 --> 00:14:10,484
and I encourage you and welcome you
to check out our website,

252
00:14:10,484 --> 00:14:12,017
try it out yourself,

253
00:14:12,017 --> 00:14:15,263
and join us in exploring
this world of tiny motions.

254
00:14:15,263 --> 00:14:16,700
Thank you.

255
00:14:16,700 --> 00:14:18,726
(Applause)