0:00:00.811,0:00:04.364
We've evolved with tools,[br]and tools have evolved with us.

0:00:04.388,0:00:09.407
Our ancestors created these[br]hand axes 1.5 million years ago,

0:00:09.431,0:00:12.453
shaping them to not only[br]fit the task at hand

0:00:12.477,0:00:13.945
but also their hand.

0:00:14.747,0:00:16.327
However, over the years,

0:00:16.351,0:00:18.881
tools have become[br]more and more specialized.

0:00:19.293,0:00:23.130
These sculpting tools[br]have evolved through their use,

0:00:23.154,0:00:26.716
and each one has a different form[br]which matches its function.

0:00:26.740,0:00:29.408
And they leverage[br]the dexterity of our hands

0:00:29.432,0:00:32.562
in order to manipulate things[br]with much more precision.

0:00:33.338,0:00:36.404
But as tools have become[br]more and more complex,

0:00:36.428,0:00:40.306
we need more complex controls[br]to control them.

0:00:40.714,0:00:45.158
And so designers have become[br]very adept at creating interfaces

0:00:45.182,0:00:48.920
that allow you to manipulate parameters[br]while you're attending to other things,

0:00:48.944,0:00:51.823
such as taking a photograph[br]and changing the focus

0:00:51.847,0:00:53.183
or the aperture.

0:00:53.918,0:00:58.137
But the computer has fundamentally[br]changed the way we think about tools

0:00:58.161,0:01:00.175
because computation is dynamic.

0:01:00.564,0:01:02.715
So it can do a million different things

0:01:02.739,0:01:05.088
and run a million different applications.

0:01:05.112,0:01:08.857
However, computers have[br]the same static physical form

0:01:08.881,0:01:10.817
for all of these different applications

0:01:10.841,0:01:13.618
and the same static[br]interface elements as well.

0:01:13.976,0:01:16.396
And I believe that this[br]is fundamentally a problem,

0:01:16.420,0:01:19.416
because it doesn't really allow us[br]to interact with our hands

0:01:19.440,0:01:22.867
and capture the rich dexterity[br]that we have in our bodies.

0:01:24.026,0:01:28.562
And my belief is that, then,[br]we must need new types of interfaces

0:01:28.586,0:01:32.345
that can capture these[br]rich abilities that we have

0:01:32.369,0:01:34.739
and that can physically adapt to us

0:01:34.763,0:01:37.013
and allow us to interact in new ways.

0:01:37.037,0:01:39.624
And so that's what I've been doing[br]at the MIT Media Lab

0:01:39.648,0:01:40.972
and now at Stanford.

0:01:41.901,0:01:45.512
So with my colleagues,[br]Daniel Leithinger and Hiroshi Ishii,

0:01:45.536,0:01:46.925
we created inFORM,

0:01:46.949,0:01:49.447
where the interface can actually[br]come off the screen

0:01:49.471,0:01:51.747
and you can physically manipulate it.

0:01:51.771,0:01:54.514
Or you can visualize[br]3D information physically

0:01:54.538,0:01:58.052
and touch it and feel it[br]to understand it in new ways.

0:02:03.889,0:02:07.965
Or you can interact through gestures[br]and direct deformations

0:02:07.989,0:02:10.286
to sculpt digital clay.

0:02:14.474,0:02:17.555
Or interface elements can arise[br]out of the surface

0:02:17.579,0:02:18.951
and change on demand.

0:02:18.975,0:02:21.483
And the idea is that for each[br]individual application,

0:02:21.507,0:02:24.807
the physical form can be matched[br]to the application.

0:02:25.196,0:02:27.301
And I believe this represents a new way

0:02:27.325,0:02:29.275
that we can interact with information,

0:02:29.299,0:02:30.728
by making it physical.

0:02:31.142,0:02:33.357
So the question is, how can we use this?

0:02:33.810,0:02:37.500
Traditionally, urban planners[br]and architects build physical models

0:02:37.524,0:02:40.334
of cities and buildings[br]to better understand them.

0:02:40.358,0:02:44.573
So with Tony Tang at the Media Lab,[br]we created an interface built on inFORM

0:02:44.597,0:02:49.580
to allow urban planners[br]to design and view entire cities.

0:02:49.604,0:02:53.861
And now you can walk around it,[br]but it's dynamic, it's physical,

0:02:53.885,0:02:55.585
and you can also interact directly.

0:02:55.609,0:02:57.347
Or you can look at different views,

0:02:57.371,0:03:00.188
such as population or traffic information,

0:03:00.212,0:03:01.774
but it's made physical.

0:03:02.996,0:03:06.806
We also believe that these dynamic[br]shape displays can really change

0:03:06.830,0:03:09.790
the ways that we remotely[br]collaborate with people.

0:03:09.814,0:03:12.117
So when we're working together in person,

0:03:12.141,0:03:13.799
I'm not only looking at your face

0:03:13.823,0:03:16.861
but I'm also gesturing[br]and manipulating objects,

0:03:16.885,0:03:20.675
and that's really hard to do[br]when you're using tools like Skype.

0:03:21.905,0:03:24.891
And so using inFORM,[br]you can reach out from the screen

0:03:24.915,0:03:27.027
and manipulate things at a distance.

0:03:27.051,0:03:30.226
So we used the pins of the display[br]to represent people's hands,

0:03:30.250,0:03:34.756
allowing them to actually touch[br]and manipulate objects at a distance.

0:03:38.519,0:03:42.793
And you can also manipulate[br]and collaborate on 3D data sets as well,

0:03:42.817,0:03:46.486
so you can gesture around them[br]as well as manipulate them.

0:03:46.510,0:03:50.900
And that allows people to collaborate[br]on these new types of 3D information

0:03:50.924,0:03:54.535
in a richer way than might[br]be possible with traditional tools.

0:03:55.870,0:03:58.623
And so you can also[br]bring in existing objects,

0:03:58.647,0:04:01.861
and those will be captured on one side[br]and transmitted to the other.

0:04:01.885,0:04:04.671
Or you can have an object that's linked[br]between two places,

0:04:04.695,0:04:06.779
so as I move a ball on one side,

0:04:06.803,0:04:08.730
the ball moves on the other as well.

0:04:10.278,0:04:13.381
And so we do this by capturing[br]the remote user

0:04:13.405,0:04:16.210
using a depth-sensing camera[br]like a Microsoft Kinect.

0:04:16.758,0:04:19.775
Now, you might be wondering[br]how does this all work,

0:04:19.799,0:04:23.450
and essentially, what it is,[br]is 900 linear actuators

0:04:23.474,0:04:25.760
that are connected to these[br]mechanical linkages

0:04:25.784,0:04:29.530
that allow motion down here[br]to be propagated in these pins above.

0:04:29.554,0:04:33.121
So it's not that complex[br]compared to what's going on at CERN,

0:04:33.145,0:04:35.471
but it did take a long time[br]for us to build it.

0:04:35.495,0:04:37.750
And so we started with a single motor,

0:04:37.774,0:04:39.189
a single linear actuator,

0:04:39.816,0:04:42.979
and then we had to design[br]a custom circuit board to control them.

0:04:43.003,0:04:45.055
And then we had to make a lot of them.

0:04:45.079,0:04:48.693
And so the problem with having[br]900 of something

0:04:48.717,0:04:51.837
is that you have to do[br]every step 900 times.

0:04:51.861,0:04:54.218
And so that meant that we had[br]a lot of work to do.

0:04:54.242,0:04:57.974
So we sort of set up[br]a mini-sweatshop in the Media Lab

0:04:57.998,0:05:01.710
and brought undergrads in and convinced[br]them to do "research" --

0:05:01.734,0:05:02.748
(Laughter)

0:05:02.772,0:05:05.790
and had late nights[br]watching movies, eating pizza

0:05:05.814,0:05:07.642
and screwing in thousands of screws.

0:05:07.666,0:05:08.864
You know -- research.

0:05:08.888,0:05:10.435
(Laughter)

0:05:10.459,0:05:13.777
But anyway, I think that we were[br]really excited by the things

0:05:13.801,0:05:15.497
that inFORM allowed us to do.

0:05:15.521,0:05:19.721
Increasingly, we're using mobile devices,[br]and we interact on the go.

0:05:19.745,0:05:22.424
But mobile devices, just like computers,

0:05:22.448,0:05:24.759
are used for so many[br]different applications.

0:05:24.783,0:05:26.776
So you use them to talk on the phone,

0:05:26.800,0:05:29.956
to surf the web, to play games,[br]to take pictures

0:05:29.980,0:05:31.689
or even a million different things.

0:05:31.713,0:05:34.697
But again, they have the same[br]static physical form

0:05:34.721,0:05:36.839
for each of these applications.

0:05:36.863,0:05:40.226
And so we wanted to know how can we take[br]some of the same interactions

0:05:40.250,0:05:41.933
that we developed for inFORM

0:05:41.957,0:05:43.845
and bring them to mobile devices.

0:05:44.427,0:05:48.074
So at Stanford, we created[br]this haptic edge display,

0:05:48.098,0:05:51.275
which is a mobile device[br]with an array of linear actuators

0:05:51.299,0:05:52.646
that can change shape,

0:05:52.670,0:05:56.567
so you can feel in your hand[br]where you are as you're reading a book.

0:05:57.058,0:06:00.795
Or you can feel in your pocket[br]new types of tactile sensations

0:06:00.819,0:06:02.621
that are richer than the vibration.

0:06:02.645,0:06:05.880
Or buttons can emerge from the side[br]that allow you to interact

0:06:05.904,0:06:07.612
where you want them to be.

0:06:09.334,0:06:12.731
Or you can play games[br]and have actual buttons.

0:06:13.786,0:06:15.302
And so we were able to do this

0:06:15.326,0:06:20.080
by embedding 40 small, tiny[br]linear actuators inside the device,

0:06:20.104,0:06:22.159
and that allow you not only to touch them

0:06:22.183,0:06:24.087
but also back-drive them as well.

0:06:24.911,0:06:29.089
But we've also looked at other ways[br]to create more complex shape change.

0:06:29.113,0:06:32.505
So we've used pneumatic actuation[br]to create a morphing device

0:06:32.529,0:06:36.394
where you can go from something[br]that looks a lot like a phone ...

0:06:36.418,0:06:38.648
to a wristband on the go.

0:06:39.720,0:06:42.559
And so together with Ken Nakagaki[br]at the Media Lab,

0:06:42.583,0:06:45.137
we created this new[br]high-resolution version

0:06:45.161,0:06:51.111
that uses an array of servomotors[br]to change from interactive wristband

0:06:51.135,0:06:54.263
to a touch-input device

0:06:54.287,0:06:55.532
to a phone.

0:06:55.556,0:06:57.214
(Laughter)

0:06:58.104,0:07:00.276
And we're also interested[br]in looking at ways

0:07:00.300,0:07:02.927
that users can actually[br]deform the interfaces

0:07:02.951,0:07:05.839
to shape them into the devices[br]that they want to use.

0:07:05.863,0:07:08.271
So you can make something[br]like a game controller,

0:07:08.295,0:07:10.925
and then the system will understand[br]what shape it's in

0:07:10.949,0:07:12.568
and change to that mode.

0:07:14.052,0:07:15.624
So, where does this point?

0:07:15.648,0:07:17.576
How do we move forward from here?

0:07:17.600,0:07:20.195
I think, really, where we are today

0:07:20.219,0:07:22.973
is in this new age[br]of the Internet of Things,

0:07:22.997,0:07:24.788
where we have computers everywhere --

0:07:24.812,0:07:26.930
they're in our pockets,[br]they're in our walls,

0:07:26.954,0:07:30.520
they're in almost every device[br]that you'll buy in the next five years.

0:07:30.544,0:07:33.425
But what if we stopped[br]thinking about devices

0:07:33.449,0:07:35.843
and think instead about environments?

0:07:35.867,0:07:38.379
And so how can we have smart furniture

0:07:38.403,0:07:41.719
or smart rooms or smart environments

0:07:41.743,0:07:44.835
or cities that can adapt to us physically,

0:07:44.859,0:07:49.090
and allow us to do new ways[br]of collaborating with people

0:07:49.114,0:07:51.352
and doing new types of tasks?

0:07:51.376,0:07:54.760
So for the Milan Design Week,[br]we created TRANSFORM,

0:07:54.784,0:07:58.608
which is an interactive table-scale[br]version of these shape displays,

0:07:58.632,0:08:01.815
which can move physical objects[br]on the surface; for example,

0:08:01.839,0:08:04.096
reminding you to take your keys.

0:08:04.120,0:08:08.602
But it can also transform[br]to fit different ways of interacting.

0:08:08.626,0:08:09.943
So if you want to work,

0:08:09.967,0:08:12.959
then it can change to sort of[br]set up your work system.

0:08:12.983,0:08:14.934
And so as you bring a device over,

0:08:14.958,0:08:17.696
it creates all the affordances you need

0:08:17.720,0:08:22.520
and brings other objects[br]to help you accomplish those goals.

0:08:25.139,0:08:26.700
So, in conclusion,

0:08:26.724,0:08:30.723
I really think that we need to think[br]about a new, fundamentally different way

0:08:30.747,0:08:32.905
of interacting with computers.

0:08:33.551,0:08:36.505
We need computers[br]that can physically adapt to us

0:08:36.529,0:08:39.130
and adapt to the ways[br]that we want to use them

0:08:39.154,0:08:43.701
and really harness the rich dexterity[br]that we have of our hands,

0:08:43.725,0:08:47.996
and our ability to think spatially[br]about information by making it physical.

0:08:48.663,0:08:52.659
But looking forward, I think we need[br]to go beyond this, beyond devices,

0:08:52.683,0:08:56.076
to really think about new ways[br]that we can bring people together,

0:08:56.100,0:08:59.118
and bring our information into the world,

0:08:59.142,0:09:03.095
and think about smart environments[br]that can adapt to us physically.

0:09:03.119,0:09:04.683
So with that, I will leave you.

0:09:04.707,0:09:05.858
Thank you very much.

0:09:05.882,0:09:09.474
(Applause)