My colleagues and I are fascinated
by the science of moving dots.

So what are these dots?

Well, it's all of us.

And we're moving in our homes,
in our offices, as we shop and travel

throughout our cities 
and around the world.

And wouldn't it be great if
we could understand all this movement?

If we could find patterns and meaning
and insight in it.

And luckily for us, we live in a time
where we're incredibly good

at capturing information about ourselves.

So whether it's through 
sensor,s or videos, or apps,

we can track our movement
with incredibly fine detail.

So it turns out one of the places
where we have the best data

about movement is sports.

So whether it's basketball or baseball,
or football or the other football,

we're instrumenting our stadiums 
and our players to track their movements

every fraction of a second.

So what we're doing is turning our 
athletes into -- you probably guessed it

moving dots.

So we've got mountains of moving dots
and like most raw data,

it's hard to deal with 
and not that interesting.

But there are things that -- for example 
basketball coaches want to know.

And the problem is they can't know them
because they'd have to watch every second

of every game, remember it 
and process it.

And a person can't do that...
but a machine can.

The problem is a machine can't see
the game with the eye of a coach.

At least they couldn't until now.

So what have we taught the machine to see?

So, we started simply.
We taught it things like passes,

shots and rebounds.

Things that most casual fans would know.
And then we moved on to things

slightly more complicated.

Events like post-ups,
and pick-and-rolls, and isolations.

And if you don't know them, that's okay.
Most casual players probably do.

Now, we've gotten to a point where today,
the machine understands complex events

like down screens and wide pins.
Basically things only professionals know.

So we have taught a machine to see
with the eyes of a coach.

So how have we been able to do this?
If I asked a coach to describe something

like a pick-and-roll, they would
give me a description and

if I encoded that as an algorithm,
it would be terrible.

The pick-and-roll happens to be the stance 
in basketball between four players,

two on offense and two on defense.

And here's kind of how it goes.

So there's the guy on offense without 
the ball and he goes next to the guy

guarding the guy with the ball,
and he kind of stays there

and they both move and stuff happens, 
and ta-da, it's a pick-and-roll.

(Laughter)

So that is also an example of
a terrible algorithm.

So, if the player who's the interferer
-- he's called the screener,

you know, goes close by,
but he doesn't stop.

It's probably not a pick-and-roll.

Or if he does stop,
but he doesn't stop close enough,

it's probably not a pick-and-roll.

Or, if he does go close by 
and he does stop but they do it

under the basket,
it's probably not a pick-and-roll.

Or I could be wrong.
They could all be pick-and-rolls.

It really depends on the exact timing,
the distances, the locations

and that 's what makes it hard.

So, luckily with machine learning
we can go beyond our own ability

to describe the things we know.

So how does this work?
Well, it's by example.

So we go to the machine and say,
"Good morning, machine."

"Here are some pick-and-rolls,
and here are somethings that are not."

"Please find a way to tell a difference."

And the key to all of this is to find
features that enable it to separate.

So if I was trying to teach it 
the difference between an apple and orange,

I might say, "Why don't you use color,
or shape?"

And the problem that we're solving is,
what are those things?

What are the key features that let a 
computer navigate the world of moving dots?

So figuring out all these relationships
with relative, absolute, location,

distance, timing, velocities.

That's really the key to the science
of moving dots, or as we like to call it

spatiotemporal patter recognition,
in academic vernacular.

Because the first thing is,
you have to make it sound hard

and... because it is.

The key thing is for NBA coaches,
it's not that they want to know

whether a pick-and-roll happened or not.

It's that they want to know how it happened.

And why is it so important to them?
So here's a little insight.

It turns out in modern basketball, this 
pick-and-roll is perhaps

the most important play.

And knowing how to run it,
and knowing how to defend it,

is basically a key to winning 
and losing most games.

So it turns out that the stance has 
a great many variations

and identifying the variations are really
the things that matter,

and that's why we need it to be
really, really good.

So, here's an example.
There's two offensive players

getting ready to do the pick-and-roll dance.

So the guy with ball can either take,
or he can reject.

His teammate can either roll or pop. The 
guy guarding the ball can go over or under.

His teammate can either show 
or play up to touch, or play soft

and together they can either
switch or blitz

and I didn't know most of the things
when I started and it would be

lovely if everybody moved according to 
those arrows.

It would make our lives a lot easier,
but it turns out movement is very messy.

People wiggle a lot and getting these
variations identified with very, very

high accuracy, precision and recall
is tough because that's what it takes

to get a professional coach 
to believe in you.

And despite t