0:00:00.000,0:00:01.976
We grew up

0:00:02.000,0:00:04.976
interacting with the physical[br]objects around us.

0:00:05.000,0:00:08.400
There are an enormous number of them[br]that we use every day.

0:00:09.293,0:00:11.976
Unlike most of our computing devices,

0:00:12.000,0:00:14.253
these objects are much more fun to use.

0:00:15.920,0:00:17.976
When you talk about objects,

0:00:18.000,0:00:20.976
one other thing automatically[br]comes attached to that thing,

0:00:21.000,0:00:22.976
and that is gestures:

0:00:23.000,0:00:24.976
how we manipulate these objects,

0:00:25.000,0:00:27.976
how we use these objects in everyday life.

0:00:28.000,0:00:30.976
We use gestures not only to interact[br]with these objects,

0:00:31.000,0:00:33.286
but we also use them[br]to interact with each other.

0:00:33.310,0:00:36.976
A gesture of "Namaste!",[br]maybe, to respect someone, or maybe,

0:00:37.000,0:00:39.429
in India I don't need to teach[br]a kid that this means

0:00:39.453,0:00:40.976
"four runs" in cricket.

0:00:41.000,0:00:43.523
It comes as a part[br]of our everyday learning.

0:00:44.456,0:00:47.976
So, I am very interested,[br]from the beginning,

0:00:48.000,0:00:51.976
how our knowledge[br]about everyday objects and gestures,

0:00:52.000,0:00:53.976
and how we use these objects,

0:00:54.000,0:00:56.976
can be leveraged to our interactions[br]with the digital world.

0:00:57.000,0:00:59.976
Rather than using a keyboard and mouse,

0:01:00.000,0:01:02.976
why can I not use my computer

0:01:03.000,0:01:05.976
in the same way that I interact[br]in the physical world?

0:01:06.000,0:01:08.976
So, I started this exploration[br]around eight years back,

0:01:09.000,0:01:11.976
and it literally started[br]with a mouse on my desk.

0:01:12.000,0:01:17.976
Rather than using it for my computer,[br]I actually opened it.

0:01:18.000,0:01:20.191
Most of you might be aware[br]that, in those days,

0:01:20.215,0:01:22.215
the mouse used to come with a ball inside,

0:01:22.239,0:01:23.976
and there were two rollers

0:01:24.000,0:01:26.976
that actually guide the computer[br]where the ball is moving,

0:01:27.000,0:01:29.096
and, accordingly,[br]where the mouse is moving.

0:01:29.120,0:01:31.976
So, I was interested in these two rollers,

0:01:32.000,0:01:35.381
and I actually wanted more, so I borrowed[br]another mouse from a friend --

0:01:35.405,0:01:36.976
never returned to him --

0:01:37.000,0:01:38.976
and I now had four rollers.

0:01:39.000,0:01:41.976
Interestingly, what I did[br]with these rollers is,

0:01:42.000,0:01:46.976
basically, I took them off of these mouses[br]and then put them in one line.

0:01:47.000,0:01:49.976
It had some strings[br]and pulleys and some springs.

0:01:50.000,0:01:52.976
What I got is basically[br]a gesture-interface device

0:01:53.000,0:01:56.976
that actually acts[br]as a motion-sensing device

0:01:57.000,0:01:58.976
made for two dollars.

0:01:59.000,0:02:01.976
So, here, whatever movement[br]I do in my physical world

0:02:02.000,0:02:04.976
is actually replicated[br]inside the digital world

0:02:05.000,0:02:08.096
just using this small device[br]that I made, around eight years back,

0:02:08.120,0:02:09.976
in 2000.

0:02:10.000,0:02:12.667
Because I was interested[br]in integrating these two worlds,

0:02:12.691,0:02:13.976
I thought of sticky notes.

0:02:14.000,0:02:16.976
I thought, "Why can I not connect

0:02:17.000,0:02:19.143
the normal interface[br]of a physical sticky note

0:02:19.167,0:02:20.976
to the digital world?"

0:02:21.000,0:02:23.148
A message written[br]on a sticky note to my mom,

0:02:23.172,0:02:24.376
on paper,

0:02:24.400,0:02:25.976
can come to an SMS,

0:02:26.000,0:02:27.976
or maybe a meeting reminder

0:02:28.000,0:02:30.191
automatically syncs[br]with my digital calendar --

0:02:30.215,0:02:32.976
a to-do list that automatically[br]syncs with you.

0:02:33.000,0:02:35.976
But you can also search[br]in the digital world,

0:02:36.000,0:02:37.976
or maybe you can write a query, saying,

0:02:38.000,0:02:39.976
"What is Dr. Smith's address?"

0:02:40.000,0:02:42.191
and this small system[br]actually prints it out --

0:02:42.215,0:02:44.692
so it actually acts like a paper[br]input-output system,

0:02:44.716,0:02:47.501
just made out of paper.

0:02:50.000,0:02:51.976
In another exploration,

0:02:52.000,0:02:54.976
I thought of making a pen[br]that can draw in three dimensions.

0:02:55.000,0:02:58.976
So, I implemented this pen[br]that can help designers and architects

0:02:59.000,0:03:00.976
not only think in three dimensions,

0:03:01.000,0:03:02.976
but they can actually draw,

0:03:03.000,0:03:05.048
so that it's more intuitive[br]to use that way.

0:03:05.072,0:03:07.120
Then I thought,[br]"Why not make a Google Map,

0:03:07.144,0:03:08.976
but in the physical world?"

0:03:09.000,0:03:11.976
Rather than typing a keyword[br]to find something,

0:03:12.000,0:03:13.976
I put my objects on top of it.

0:03:14.000,0:03:17.191
If I put a boarding pass, it will show me[br]where the flight gate is.

0:03:17.215,0:03:19.976
A coffee cup will show[br]where you can find more coffee,

0:03:20.000,0:03:21.976
or where you can trash the cup.

0:03:22.000,0:03:24.976
So, these were some[br]of the earlier explorations I did

0:03:25.000,0:03:28.000
because the goal was to connect[br]these two worlds seamlessly.

0:03:29.000,0:03:30.976
Among all these experiments,

0:03:31.000,0:03:32.976
there was one thing in common:

0:03:33.000,0:03:36.505
I was trying to bring[br]a part of the physical world

0:03:36.529,0:03:38.027
to the digital world.

0:03:38.051,0:03:39.976
I was taking some part of the objects,

0:03:40.000,0:03:42.977
or any of the intuitiveness of real life,

0:03:43.001,0:03:45.190
and bringing them to the digital world,

0:03:45.214,0:03:49.239
because the goal was to make[br]our computing interfaces more intuitive.

0:03:49.263,0:03:53.976
But then I realized that we humans[br]are not actually interested in computing.

0:03:54.000,0:03:56.976
What we are interested in is information.

0:03:57.000,0:03:58.976
We want to know about things.

0:03:59.000,0:04:01.381
We want to know about[br]dynamic things going around.

0:04:01.405,0:04:05.976
So I thought, around last year --[br]in the beginning of the last year --

0:04:06.000,0:04:09.477
I started thinking, "Why can I not take[br]this approach in the reverse way?"

0:04:10.119,0:04:12.176
Maybe, "How about I take my digital world

0:04:12.200,0:04:16.976
and paint the physical world[br]with that digital information?"

0:04:18.154,0:04:21.776
Because pixels are actually, right now,[br]confined in these rectangular devices

0:04:21.800,0:04:23.547
that fit in our pockets.

0:04:23.571,0:04:25.976
Why can I not remove this confine

0:04:26.000,0:04:28.976
and take that to my everyday[br]objects, everyday life

0:04:29.000,0:04:31.143
so that I don't need[br]to learn the new language

0:04:31.167,0:04:32.964
for interacting with those pixels?

0:04:34.214,0:04:36.976
So, in order to realize this dream,

0:04:37.000,0:04:39.976
I actually thought of putting[br]a big-size projector on my head.

0:04:40.000,0:04:43.239
I think that's why this is called[br]a head-mounted projector, isn't it?

0:04:43.263,0:04:44.976
I took it very literally,

0:04:45.000,0:04:46.976
and took my bike helmet,

0:04:47.000,0:04:50.381
put a little cut over there so that[br]the projector actually fits nicely.

0:04:50.405,0:04:51.976
So now, what I can do --

0:04:52.000,0:04:55.805
I can augment the world around me[br]with this digital information.

0:04:56.658,0:04:57.876
But later,

0:04:57.900,0:05:00.059
I realized that I actually[br]wanted to interact

0:05:00.083,0:05:01.676
with those digital pixels, also.

0:05:01.700,0:05:04.976
So I put a small camera over there[br]that acts as a digital eye.

0:05:05.000,0:05:06.976
Later, we moved to a much better,

0:05:07.000,0:05:09.000
consumer-oriented pendant version of that,

0:05:09.024,0:05:11.976
that many of you now know[br]as the SixthSense device.

0:05:12.000,0:05:14.976
But the most interesting thing[br]about this particular technology

0:05:15.000,0:05:18.976
is that you can carry[br]your digital world with you

0:05:19.000,0:05:20.976
wherever you go.

0:05:21.000,0:05:23.976
You can start using any surface,[br]any wall around you,

0:05:24.000,0:05:25.976
as an interface.

0:05:26.000,0:05:28.976
The camera is actually tracking[br]all your gestures.

0:05:29.000,0:05:30.976
Whatever you're doing with your hands,

0:05:31.000,0:05:32.976
it's understanding that gesture.

0:05:33.000,0:05:35.576
And, actually, if you see,[br]there are some color markers

0:05:35.600,0:05:38.076
that in the beginning version[br]we are using with it.

0:05:38.100,0:05:39.976
You can start painting on any wall.

0:05:40.000,0:05:42.976
You stop by a wall,[br]and start painting on that wall.

0:05:43.000,0:05:45.143
But we are not only tracking[br]one finger, here.

0:05:45.167,0:05:48.976
We are giving you the freedom[br]of using all of both of your hands,

0:05:49.000,0:05:52.143
so you can actually use both of your hands[br]to zoom into or zoom out

0:05:52.167,0:05:54.143
of a map just by pinching all present.

0:05:54.167,0:05:57.976
The camera is actually doing --[br]just, getting all the images --

0:05:58.000,0:06:00.976
is doing the edge recognition[br]and also the color recognition

0:06:01.000,0:06:03.976
and so many other small algorithms[br]are going on inside.

0:06:04.000,0:06:06.000
So, technically,[br]it's a little bit complex,

0:06:06.024,0:06:09.500
but it gives you an output which is more[br]intuitive to use, in some sense.

0:06:09.524,0:06:12.376
But I'm more excited that you can[br]actually take it outside.

0:06:12.400,0:06:14.976
Rather than getting your camera[br]out of your pocket,

0:06:15.000,0:06:17.976
you can just do the gesture[br]of taking a photo,

0:06:18.000,0:06:19.976
and it takes a photo for you.

0:06:20.000,0:06:23.976
(Applause)

0:06:24.000,0:06:25.000
Thank you.

0:06:25.599,0:06:27.976
And later I can find a wall, anywhere,

0:06:28.000,0:06:29.976
and start browsing those photos

0:06:30.000,0:06:32.676
or maybe, "OK, I want to modify[br]this photo a little bit

0:06:32.700,0:06:34.686
and send it as an email to a friend."

0:06:34.710,0:06:36.976
So, we are looking for an era

0:06:37.000,0:06:39.976
where computing will actually merge[br]with the physical world.

0:06:40.000,0:06:42.976
And, of course,[br]if you don't have any surface,

0:06:43.000,0:06:45.976
you can start using your palm[br]for simple operations.

0:06:46.000,0:06:48.477
Here, I'm dialing a phone number[br]just using my hand.

0:06:51.880,0:06:54.976
The camera is actually not[br]only understanding your hand movements,

0:06:55.000,0:06:56.176
but, interestingly,

0:06:56.200,0:06:59.439
is also able to understand what objects[br]you are holding in your hand.

0:07:00.009,0:07:03.976
For example, in this case,

0:07:04.000,0:07:05.976
the book cover is matched

0:07:06.000,0:07:08.976
with so many thousands,[br]or maybe millions of books online,

0:07:09.000,0:07:10.976
and checking out which book it is.

0:07:11.000,0:07:12.476
Once it has that information,

0:07:12.500,0:07:14.376
it finds out more reviews about that,

0:07:14.400,0:07:16.976
or maybe New York Times[br]has a sound overview on that,

0:07:17.000,0:07:19.096
so you can actually hear,[br]on a physical book,

0:07:19.120,0:07:20.976
a review as sound.

0:07:21.000,0:07:23.176
(Video) Famous talk[br]at Harvard University --

0:07:23.200,0:07:26.976
This was Obama's visit last week to MIT.

0:07:27.000,0:07:30.465
(Video) And particularly I want[br]to thank two outstanding MIT --

0:07:30.489,0:07:33.523
Pranav Mistry: So, I was seeing[br]the live [video] of his talk,

0:07:33.547,0:07:35.489
outside, on just a newspaper.

0:07:36.000,0:07:38.976
Your newspaper will show you[br]live weather information

0:07:39.000,0:07:41.606
rather than having it updated.

0:07:41.630,0:07:44.477
You have to check your computer[br]in order to do that, right?

0:07:44.501,0:07:48.976
(Applause)

0:07:49.000,0:07:51.976
When I'm going back,[br]I can just use my boarding pass

0:07:52.000,0:07:54.096
to check how much my flight[br]has been delayed,

0:07:54.120,0:07:55.976
because at that particular time,

0:07:56.000,0:07:57.976
I'm not feeling like opening my iPhone,

0:07:58.000,0:07:59.976
and checking out a particular icon.

0:08:00.000,0:08:03.134
And I think this technology[br]will not only change the way --

0:08:03.158,0:08:04.134
(Laughter)

0:08:04.158,0:08:05.176
Yes.

0:08:05.200,0:08:07.678
It will change the way[br]we interact with people, also,

0:08:07.702,0:08:09.217
not only the physical world.

0:08:09.241,0:08:11.976
The fun part is, I'm going[br]to the Boston metro,

0:08:12.000,0:08:16.976
and playing a pong game inside the train[br]on the ground, right?

0:08:17.000,0:08:18.076
(Laughter)

0:08:18.100,0:08:20.196
And I think the imagination[br]is the only limit

0:08:20.220,0:08:21.976
of what you can think of

0:08:22.000,0:08:24.476
when this kind of technology[br]merges with real life.

0:08:24.500,0:08:26.376
But many of you argue, actually,

0:08:26.400,0:08:29.076
that all of our work is not[br]only about physical objects.

0:08:29.100,0:08:32.076
We actually do lots[br]of accounting and paper editing

0:08:32.100,0:08:34.391
and all those kinds of things;[br]what about that?

0:08:34.415,0:08:37.976
And many of you are excited[br]about the next-generation tablet computers

0:08:38.000,0:08:39.976
to come out in the market.

0:08:40.000,0:08:41.976
So, rather than waiting for that,

0:08:42.000,0:08:44.976
I actually made my own,[br]just using a piece of paper.

0:08:45.000,0:08:47.000
So, what I did here[br]is remove the camera --

0:08:47.024,0:08:50.976
All the webcam cameras have[br]a microphone inside the camera.

0:08:51.000,0:08:53.976
I removed the microphone from that,

0:08:54.000,0:08:55.976
and then just pinched that --

0:08:56.000,0:08:58.976
like I just made a clip[br]out of the microphone --

0:08:59.000,0:09:02.976
and clipped that to a piece of paper,[br]any paper that you found around.

0:09:03.000,0:09:05.976
So now the sound of the touch

0:09:06.000,0:09:08.976
is getting me when exactly[br]I'm touching the paper.

0:09:09.000,0:09:12.976
But the camera is actually tracking[br]where my fingers are moving.

0:09:13.000,0:09:15.976
You can of course watch movies.

0:09:16.000,0:09:18.976
(Video) Good afternoon.[br]My name is Russell,

0:09:19.000,0:09:21.976
and I am a Wilderness[br]Explorer in Tribe 54."

0:09:22.000,0:09:24.976
PM: And you can of course play games.

0:09:25.000,0:09:27.976
(Car engine)

0:09:28.000,0:09:31.334
Here, the camera is actually understanding[br]how you're holding the paper

0:09:31.358,0:09:32.976
and playing a car-racing game.

0:09:33.000,0:09:36.000
(Applause)

0:09:37.396,0:09:40.174
Many of you already must have[br]thought, OK, you can browse.

0:09:40.198,0:09:42.476
Yeah. Of course you can[br]browse to any websites

0:09:42.500,0:09:45.176
or you can do all sorts[br]of computing on a piece of paper

0:09:45.200,0:09:46.376
wherever you need it.

0:09:46.400,0:09:48.976
So, more interestingly,

0:09:49.000,0:09:51.976
I'm interested in how we can[br]take that in a more dynamic way.

0:09:52.000,0:09:54.976
When I come back to my desk,[br]I can just pinch that information

0:09:55.000,0:09:56.976
back to my desktop

0:09:57.000,0:09:59.976
so I can use my full-size computer.

0:10:00.000,0:10:01.976
(Applause)

0:10:02.000,0:10:04.976
And why only computers?[br]We can just play with papers.

0:10:05.000,0:10:07.976
Paper world is interesting to play with.

0:10:08.000,0:10:09.976
Here, I'm taking a part of a document,

0:10:10.000,0:10:13.976
and putting over here a second part[br]from a second place,

0:10:14.000,0:10:18.976
and I'm actually modifying the information[br]that I have over there.

0:10:19.000,0:10:23.976
Yeah. And I say, "OK, this looks nice,[br]let me print it out, that thing."

0:10:24.000,0:10:26.381
So I now have a print-out of that thing.

0:10:26.405,0:10:28.729
So the workflow is more intuitive,

0:10:28.753,0:10:31.976
the way we used to do it[br]maybe 20 years back,

0:10:32.000,0:10:34.976
rather than now switching[br]between these two worlds.

0:10:35.000,0:10:37.976
So, as a last thought,

0:10:38.000,0:10:42.376
I think that integrating[br]information to everyday objects

0:10:42.400,0:10:45.976
will not only help us to get rid[br]of the digital divide,

0:10:46.000,0:10:47.976
the gap between these two worlds,

0:10:48.000,0:10:49.976
but will also help us, in some way,

0:10:50.000,0:10:51.976
to stay human,

0:10:52.000,0:10:55.000
to be more connected[br]to our physical world.

0:10:58.408,0:11:00.976
And it will actually help us[br]not end up being machines

0:11:01.000,0:11:02.718
sitting in front of other machines.

0:11:03.507,0:11:05.976
That's all. Thank you.

0:11:06.000,0:11:19.976
(Applause)

0:11:20.000,0:11:21.176
Thank you.

0:11:21.200,0:11:23.976
(Applause)

0:11:24.000,0:11:27.976
Chris Anderson: So, Pranav,[br]first of all, you're a genius.

0:11:28.000,0:11:30.976
This is incredible, really.

0:11:31.000,0:11:34.100
What are you doing with this?[br]Is there a company being planned?

0:11:34.124,0:11:35.976
Or is this research forever, or what?

0:11:36.000,0:11:38.276
Pranav Mistry: So, there are[br]lots of companies,

0:11:38.300,0:11:41.296
sponsor companies of Media Lab[br]interested in taking this ahead

0:11:41.320,0:11:42.506
in one or another way.

0:11:42.530,0:11:44.503
Companies like mobile-phone operators

0:11:44.527,0:11:47.401
want to take this in a different way[br]than the NGOs in India,

0:11:47.425,0:11:49.601
thinking, "Why can we only[br]have 'Sixth Sense'?

0:11:49.625,0:11:53.076
We should have a 'Fifth Sense'[br]for missing-sense people who cannot speak.

0:11:53.100,0:11:56.391
This technology can be used for them[br]to speak out in a different way

0:11:56.415,0:11:57.691
maybe a speaker system."

0:11:57.715,0:12:00.176
CA: What are your own plans?[br]Are you staying at MIT,

0:12:00.200,0:12:02.276
or are you going to do[br]something with this?

0:12:02.300,0:12:04.829
PM: I'm trying to make this[br]more available to people

0:12:04.853,0:12:07.529
so that anyone can develop[br]their own SixthSense device,

0:12:07.553,0:12:10.976
because the hardware is actually[br]not that hard to manufacture

0:12:11.000,0:12:12.976
or hard to make your own.

0:12:13.000,0:12:15.572
We will provide all the open source[br]software for them,

0:12:15.596,0:12:16.976
maybe starting next month.

0:12:17.000,0:12:18.976
CA: Open source? Wow.

0:12:19.000,0:12:23.976
(Applause)

0:12:24.000,0:12:27.429
CA: Are you going to come back to India[br]with some of this, at some point?

0:12:27.453,0:12:28.976
PM: Yeah. Yes, yes, of course.

0:12:29.000,0:12:30.976
CA: What are your plans? MIT? India?

0:12:31.000,0:12:33.476
How are you going to split[br]your time going forward?

0:12:33.500,0:12:35.976
PM: There is a lot of energy here.[br]Lots of learning.

0:12:36.000,0:12:39.976
All of this work that you have seen[br]is all about my learning in India.

0:12:40.000,0:12:42.976
And now, if you see, it's more about[br]the cost-effectiveness:

0:12:43.000,0:12:44.976
this system costs you $300

0:12:45.000,0:12:47.976
compared to the $20,000 surface tables,[br]or anything like that.

0:12:48.000,0:12:53.976
Or maybe even the $2 mouse gesture system[br]at that time was costing around $5,000?

0:12:54.000,0:12:59.976
I showed that, at a conference,[br]to President Abdul Kalam, at that time,

0:13:00.000,0:13:03.524
and then he said, "OK, we should use this[br]in Bhabha Atomic Research Centre

0:13:03.548,0:13:04.976
for some use of that."

0:13:05.000,0:13:08.096
So I'm excited about how I can bring[br]the technology to the masses

0:13:08.120,0:13:11.120
rather than just keeping that technology[br]in the lab environment.

0:13:11.144,0:13:14.976
(Applause)

0:13:15.000,0:13:17.118
CA: Based on the people we've seen at TED,

0:13:17.142,0:13:19.476
I would say you're truly[br]one of the two or three

0:13:19.500,0:13:21.376
best inventors in the world right now.

0:13:21.400,0:13:22.976
It's an honor to have you at TED.

0:13:23.000,0:13:24.976
Thank you so much.

0:13:25.000,0:13:26.176
That's fantastic.

0:13:26.200,0:13:30.000
(Applause)