My colleagues and I are fascinated
by the science of moving dots.

So what are these dots?

Well, it's all of us.

And we're moving in our homes,
in our offices, as we shop and travel

throughout our cities 
and around the world.

And wouldn't it be great if
we could understand all this movement?

If we could find patterns and meaning
and insight in it.

And luckily for us, we live in a time
where we're incredibly good

at capturing information about ourselves.

So whether it's through 
sensors or videos, or apps,

we can track our movement
with incredibly fine detail.

So it turns out one of the places
where we have the best data

about movement is sports.

So whether it's basketball or baseball,
or football or the other football,

we're instrumenting our stadiums 
and our players to track their movements

every fraction of a second.

So what we're doing is turning our 
athletes into -- you probably guessed it

moving dots.

So we've got mountains of moving dots
and like most raw data,

it's hard to deal with 
and not that interesting.

But there are things that -- for example 
basketball coaches want to know.

And the problem is they can't know them
because they'd have to watch every second

of every game, remember it 
and process it.

And a person can't do that...
but a machine can.

The problem is a machine can't see
the game with the eye of a coach.

At least they couldn't until now.

So what have we taught the machine to see?

So, we started simply.
We taught it things like passes,

shots and rebounds.

Things that most casual fans would know.
And then we moved on to things

slightly more complicated.

Events like post-ups,
and pick-and-rolls, and isolations.

And if you don't know them, that's okay.
Most casual players probably do.

Now, we've gotten to a point where today,
the machine understands complex events

like down screens and wide pins.
Basically things only professionals know.

So we have taught a machine to see
with the eyes of a coach.

So how have we been able to do this?
If I asked a coach to describe something

like a pick-and-roll, they would
give me a description and

if I encoded that as an algorithm,
it would be terrible.

The pick-and-roll happens to be the stance 
in basketball between four players,

two on offense and two on defense.

And here's kind of how it goes.

So there's the guy on offense without 
the ball and he goes next to the guy

guarding the guy with the ball,
and he kind of stays there

and they both move and stuff happens, 
and ta-da, it's a pick-and-roll.

(Laughter)

So that is also an example of
a terrible algorithm.

So, if the player who's the interferer
-- he's called the screener,

you know, goes close by,
but he doesn't stop.

It's probably not a pick-and-roll.

Or if he does stop,
but he doesn't stop close enough,

it's probably not a pick-and-roll.

Or, if he does go close by 
and he does stop but they do it

under the basket,
it's probably not a pick-and-roll.

Or I could be wrong.
They could all be pick-and-rolls.

It really depends on the exact timing,
the distances, the locations

and that 's what makes it hard.

So, luckily with machine learning
we can go beyond our own ability

to describe the things we know.

So how does this work?
Well, it's by example.

So we go to the machine and say,
"Good morning, machine."

"Here are some pick-and-rolls,
and here are somethings that are not."

"Please find a way to tell a difference."

And the key to all of this is to find
features that enable it to separate.

So if I was trying to teach it 
the difference between an apple and orange,

I might say, "Why don't you use color,
or shape?"

And the problem that we're solving is,
what are those things?

What are the key features that let a 
computer navigate the world of moving dots?

So figuring out all these relationships
with relative, absolute, location,

distance, timing, velocities.

That's really the key to the science
of moving dots, or as we like to call it

spatiotemporal pattern recognition,
in academic vernacular.

Because the first thing is,
you have to make it sound hard

and... because it is.

The key thing is for NBA coaches,
it's not that they want to know

whether a pick-and-roll happened or not.

It's that they want to know how it happened.

And why is it so important to them?
So here's a little insight.

It turns out in modern basketball, this 
pick-and-roll is perhaps

the most important play.

And knowing how to run it,
and knowing how to defend it,

is basically a key to winning 
and losing most games.

So it turns out that the dance has 
a great many variations

and identifying the variations are really
the things that matter,

and that's why we need it to be
really, really good.

So, here's an example.
There's two offensive players

getting ready to do the pick-and-roll dance.

So the guy with ball can either take,
or he can reject.

His teammate can either roll or pop. The 
guy guarding the ball can go over or under.

His teammate can either show 
or play up to touch, or play soft

and together they can either
switch or blitz

and I didn't know most of the things
when I started and it would be

lovely if everybody moved according to 
those arrows.

It would make our lives a lot easier,
but it turns out movement is very messy.

People wiggle a lot and getting these
variations identified with very, very

high accuracy, both in precision and recall
is tough because that's what it takes

to get a professional coach 
to believe in you.

And despite all the difficulties with
the right spatiotemoporal features

we have been able to do that.

Coaches trust are ability of our machine 
to identify these variations.

We're at the point where almost every 
single contender for an NBA championship

this year is using our software, which is 
built on a machine that understands

the moving dots of basketball.

So, not only that, we have given advice 
that has changed strategies,

that have helped teams win 
very important games

and it's very exciting because you have
coaches who've been in the league for

30 years, that are willing to take advice 
from a machine.

And it's very exciting.
It's much more than the pick-and-roll.

Our computer have started with simple 
things and learnt more and more

complex things and now it knows 
so many things.

Frankly, I don't understand much of what
it does and while it's not special

to be smarter than me,
we were wondering,

can a machine know more than a coach?
Could it know more than person could know?

Turns out the answer is yes.

Coaches want players to take good shots.
So if I'm standing near the basket

and there's nobody near me,
it's a good shot.

If I'm standing far away and surrounded
by defenders, that's generally a bad shot.

But we never knew how good "good" was,
or how bad "bad" was quantitatively.

Until now.

So what we can do, again, 
using spatiotemporal features.

We looked at every shot. We can see where 
is the shot? What's the angle to the basket?

Where are the defenders standing?
What are their distances?

What are there angles?

For multiple defenders, we can look at how
players move and predict the shot type.

We can look at all their velocities 
and we can build a model that predicts

what is the likelihood that this shot 
would go in under these circumstances?

So why is this important?
We can take something that was shooting,

that was one thing before, and turn it 
into two things.

The quality of the shot
and the quality of the shooter.

So here's a bubble chart because
what's TED without a bubble chart?

Those are NBA players. 
The size is the size of the player

and the color is the position.

On the x-axis, we've the shot probability.
People on the left take difficult shots,

on the right, they take easy shots.

On the right is their shooting ability.
People who are good at the top,

bad at the bottom.

So for example, if there was a player who
generally made 47% of their shots

that's all you knew before.

But today, I can tell you that player 
takes shots that an average NBA player

would make 49% of the time
and they were 2% worse.

And the reason that's important,
is that there are lots of 47s out there.

And so it's really important to know
if the 47 that you're considering

giving 100 million dollars to,
is a good shooter who takes bad shots

or bad shooter who takes good shots.

Machine understanding doesn't 
change how we look at players,

it changes how we look at the game.
So there was this very exciting game

a couple of years ago, in the NBA finals.

Miami was down by three, 
there was 20 seconds left.

They were about to lose the championship.
A gentleman named Lebron James

came up and he took a three to tie.

He missed.
His teammate Chris Bosh got a rebound,

passed it to another teammate
named Ray Allen.

He sank a three.
It went into overtime.

They won the game.
They won the championship.

It was one of the most exciting 
games in basketball.

And our ability to know the shot 
probability for every player

at every second, and the likelihood 
of them getting a rebound at every second

can illuminate this moment in a way
that we never could before.

Now unfortunately, I can't show you that 
video, but for you we recreated

that moment at our weekly basketball game
about 3 weeks ago.

(Laughter)

And we recreated the tracking 
that led to the insights.

So, here is us.
This is Chinatown in Los Angeles,

a park we play every week at and that's us
recreating the Ray Allen moment

and all the tracking that's associated.

So, here's the shot.
I'm going to show you that moment

and all the insights of that moment.

The only difference is, instead of the 
professional players -- it's us

and instead of a professional 
announcer, it's me.

So, bare with me.

Miami. Down three. 20 seconds left.

Jeff brings up the ball... Josh catches, 
puts up a three!

Won't go! Rebound Noel(??), back to Daria.

Her 3-pointer -- bang!
Tied game with five seconds left.

The crowd goes wild.

(Laughter)

That's roughly how it happened.

(Applause)

Roughly.

I'm not going to -- that moment had about
a 9% chance of happening in the NBA

and we know that and a great many other things.

I'm not going to tell you how many times
it took us to make that happen.

(Laughter)

Okay, I will!
It was four, it was four.

Way to go Doug(??).

But the important thing about that video 
and the insights we have for every second

of every NBA game, It's not that.

It's the fact you don't have to be a 
professional team to track movement.

You do not have to be a professional player
to get insights about movement.

In fact, it doesn't even have to be about 
sports because we're moving everywhere.

We're moving in our homes.

In our offices.

As we shop and we travel, throughout 
our cities and around our world.

What will we know? What will we learn?
Perhaps, instead of identifying

pick-and-rolls, a machine can identify 
the moment and let me know when

my daughter takes her first steps.

Which could literally be happening 
any second now.

Perhaps we can learn to better use 
our buildings, better plan our cities.

I believe that with the development
of the science of moving dots,

we will move better, we will move smarter,
we will move forward.

Thank you very much.

(Applause)