The birth of virtual reality as an art form

When I was a kid,

I experienced something so powerful
I spent the rest of my life

searching for it,

and in all the wrong places.

What I experienced wasn't virtual reality.

It was music.

And this is where the story begins.

That's me

listening to the Beatles' "White Album,"

and the look on my face

is a feeling that I've been
searching for ever since.

Music goes straight
to the emotional vein,

into your bloodstream,

and right into your heart.

It deepens every experience.

Fellows?

(Music)

This is the amazing McKenzie Stuckard
and Joshua Roman.

Music.

(Applause)

Yep.

Music makes everything
have more emotional resonance.

Let's see how it goes with this talk.

The right piece of music
at the right time fuses with us

on a cellular level.

When I hear that one song

from that one summer

with that one girl

I'm instantly transported
back there again.

Hey, Stacey.

Here's a part of the story, though,
where I got a little greedy.

I thought if I added more layers
on top of the music,

I could make the feelings
even more powerful,

so I got into directing music videos.

Here's what they looked like.

That's my brother, Jeff.

Sorry about this, Jeff.

Here's me, just so we're even.

Tried all the moves.

Should've been a dancer.

These experiments grew and in time
started to look more like this.

In both, I'm searching
for the same thing, though,

to capture that lightning in a bottle.

Except, I'm not.

Adding moving pictures over the music
added narrative dimension, yes,

but never quite equated
to the power that just raw music

had for me on its own.

This is not a great thing to realize
when you've devoted your life

and professional career to becoming
a music video director.

I kept asking myself,
did I take the wrong path?

So I started thinking, if I could
involve you, the audience, more,

I might be able to make you
feel something more as well.

So Aaron Koblin and I

began auditioning new technologies
that could put more of you

inside of the work,

like your childhood home
in "The Wilderness Downtown,"

your hand-drawn portraits,
in "The Johnny Cash Project,"

and your interactive dreams

in "Three Dreams of Black."

We were pushing beyond the screen,

trying to connect more deeply

to people's hearts and imaginations,

but it wasn't quite enough.

It still didn't have the raw
experiential power of pure music for me.

So I started chasing a new technology
that I only had read about

in science fiction,

and after years of searching,
I found a prototype.

It was a project from Nonny de la Peña
in Mark Bolas's lab in USC,

and when I tried it,

I knew I'd found it.

I could taste the lightning.

It was called virtual reality.

This was at five years ago
when I ran into it.

This is what it looks like now.

And I quickly started building things
in this new medium,

and through that process
we realized something:

that VR is going to play
an incredibly important role

in the history of mediums.

In fact, it's going to be the last one.

I mean this because it's the first medium
that actually makes the jump

from our internalization
of an author's expression

of an experience

to our experiencing it firsthand.

You look confused.
I'll explain. Don't worry.

(Laughter)

If we go back to the origins of mediums,

by all best guesses,
it starts around the fire

with a good story.

Our clan leader is telling us
about how he hunted

the woolly mammoth
on the tundra that day.

We hear his words

and translate them
into our own internal truths.

Same thing happens
when we look at the cave painting

version of the story,

the book about the mammoth hunt,

the play,

the radio broadcast,
the television show,

or the movie.

All of these mediums require
what we call suspension of disbelief,

because there's a translation gap
between the reality of the story

and our consciousness
interpreting the story

into our reality.

I'm using the word consciousness
as a feeling of reality that we get

from our senses experiencing
the world around us.

Virtual reality bridges that gap.

Now, you are on the tundra
hunting with the clan leader,

or you are the clan leader,

or maybe you're even the woolly mammoth.

(Music)

So here's what special about VR.

In all other mediums,

your consciousness interprets the medium.

In VR, your consciousness is the medium.

So the potential for VR is enormous,

but where are we now?

What is the current state of the art?

Well, you are here.

We are the equivalent
of year one of cinema.

This is the Lumière brothers film
that allegedly sent a theater

full of people running for their lives

as they thought a train
was coming towards them.

Similar to this early stage
of this medium,

in VR, we also have to move past
the spectacle

and into the storytelling.

It took this medium decades
to figure out its preferred language

of storytelling in the form
of a feature film.

In VR today, we're more learning grammar

than writing language.

We've made 15 films in the last year

at our VR company, Vrse,
and we've learned a few things.

We found that we have a unique,
direct path into your senses,

your emotions, even your body.

So let me show you some things,

and for the purpose of this demo,
we're going to take every direction

that you could possibly look

and stretch it into this giant rectangle.

Okay, here we go.

So, first,

camera movement is tricky in VR.

Done wrong, it can actually make you sick.

We found if you move the camera
at a constant speed

in a straight line,

you can actually get away with it, though.

So the first day in film school,
they told me that you have

to learn every single rule
before you can break one.

We have not learned every single rule.
We've barely learned any at all,

but we're already trying to break them
to see what kind of creative things

we can accomplish.

In this shot here, where we're moving up
off the ground, I added acceleration,

and I did that because
I wanted to give you

a physical sensation
of moving up off the ground,

and in VR, I can give that to you.

(Music)

So not surprisingly,
music matters a lot

in this medium as well.

It guides us how to feel.

In this project we made
with the New York Times' Zach Richter

and our friend J.R.,

we take you up in a helicopter,

and even though you're flying
2,000 feet above Manhattan,

you don't feel afraid,
you feel triumphant

for J.R.'s character.

The music guides you there.

Contrary to popular belief,
there is composition in virtual reality,

but it's completely different than in film
where you have a rectangular frame.

Composition is now
where your consciousness exists

and how the world moves around you.

In this film, "Waves of Grace,"

which was a collaboration between Vrse,
the United Nations,

Gabo Arora, and Imraan Ismail,

we also see the changing role
of the closeup in virtual reality.

A closeup in VR means you're
actually close up to someone.

It brings that character inside
of that personal space,

a space that we'd usually reserve
for the people that we love,

and you feel an emotional closeness
to the character

because of what you feel
to be a physical closeness.

Directing VR is not like directing
for the rectangle.

It's more of a choreography
of the viewer's attention.

One tool we can use
to guide your attention

is called spatialized sound.

So I can put a sound anywhere
in front of you, to left or right,

even behind you,

and when you turn your head,
the sound will rotate accordingly.

So I can use that to direct your attention
to where I want you to see.

Next time you hear someone
singing over your shoulder,

it might be Bono.

(Laughter)

VR makes us feel like we
are part of something.

For most of human history,
we lived in small family units.

We started in caves

and moved to clans and tribes,

then villages and towns,

and now we're all global citizens.

But I believe that we are still hardwired
to care the most about the things

that are local to us.

And VR makes anywhere
and anyone feel local.

That's why it works
as an empathy machine.

Our film "Clouds Over Sidra"
takes you to a Syrian refugee camp,

and instead of watching a story
about people over there,

it's now a story about us here.

But where do we go from here?

The tricky thing is that
with all previous mediums,

the format is fixed at its birth.

Film has been a sequence of rectangles.
From Muybridge and his horses to now,

the format has never changed.

But VR as a format, as a medium,

isn't complete yet.

It's not using physical celluloid
or paper or TV signals.

It actually employs what we use

to make sense of the world.

We're using your senses
as the paints on the canvas,

but only two right now.

Eventually, we can see if we will have
all of our human senses employed,

and we will have agency to live
the story in any path we choose.

And we call it virtual reality right now,

but what happens when we move
past simulated realities.

What do we call it then?

What if instead of verbally
telling you about a dream,

I could let you live inside that dream?

What if instead of just experiencing
visiting some reality on Earth,

you could surf gravitational waves
on the edge of a black hole,

or create galaxies from scratch,

or communicate with each other
not using words

but using our raw thoughts.

That's not a virtual reality anymore,

and honestly I don't know
what that's called,

but I hope you see where we're going.

But here I am intellectualizing
a medium I'm saying is experiential,

so let's experience it.

In your hands, you hopefully hold
a piece of cardboard,

and let's open the flap,

tap on the power button
to unlock the phone,

and for the people watching at home,

we are going to put up a card right now
to show you how to download

this experience on your phone yourself,

and you even get a Google cardboard
of your own to try it with.

We played in cardboard boxes as kids,

and as adults, I'm hoping we can all find
a little bit of that lightning

by sticking our head in one again.

So you're about to participate
in the largest collective VR viewing

in history,

and in that classic old-timey style
of yesteryear,

we're all going to watch something

at the exact same time together.

And let's hope it works.

What's the countdown now? I can't see.

Crowd: 16... 15... 14...
13... 12... 11... 10... 9...

8... 7... 6... 5... 4... 3... 2... 1...

(Music)

(Train engine)

Voice: Let me tell you how I shot
the cover of the New York Times Magazine

walking New York.

I just constructed [???]
outside [the little tent],

and I had to be perfectly verticle

so I could grab it,

and when I was perfectly above,
you know, with the wind,

we had to redo it a few times,

then I keep shooting.

Voice: Dear Lord,

protect us from evil,

for your are the Lord,

the light.

You who gave us life took it away.

Let your will be done.

Please bring peace to the many
who have lost loved ones.

Help us to live again.

(Music)

Voice: There are more kids in Zaatari
than adults right now.

Sometimes I think

we are the ones in charge.

Chris Milk: How was it?

(Applause)

That was a cheap way of getting you
to do a standing ovation.

I just made you all stand.
I knew you'd applaud at the end of it.

(Applause)

I believe that everyone on Earth
needs to experience

what you just experienced.

That way we can collectively
start to shape this

not as a tech platform,

but as a humanity platform,

and to that end, in November of last year,

the New York Times and Vrse made
a VR project called "The Displaced."

It launched with one million
Google cardboards sent out

to every Sunday subscriber
with their newspaper.

But a funny thing happened
that Sunday morning.

A lot of people got them
that were not the intended recipients

on the mailing label,

and we started seeing this
all over Instagram.

Look familiar?

Music led me on a path
of searching for what looked like

the unattainable for a very long time.

Now, millions of kids just had
the same formative experience

in their childhood

that I had in mine,

only I think this one

surpasses it.

Let's see where this leads them.

Thank you.

(Applause)