The birth of virtual reality as an art form

When I was a kid,

I experienced something so powerful,

I spent the rest of my life
searching for it,

and in all the wrong places.

What I experienced wasn't virtual reality.

It was music.

And this is where the story begins.

That's me,

listening to the Beatles' "White Album."

And the look on my face is the feeling

that I've been searching for ever since.

Music goes straight to the emotional vein,

into your bloodstream

and right into your heart.

It deepens every experience.

Fellas?

(Music)

This is the amazing McKenzie Stubbert

and Joshua Roman.

Music --

(Applause)

Yep.

Music makes everything
have more emotional resonance.

Let's see how it does for this talk.

The right piece of music
at the right time fuses with us

on a cellular level.

When I hear that one song

from that one summer

with that one girl,

I'm instantly transported
back there again.

Hey, Stacey.

Here's a part of the story, though,
where I got a little greedy.

I thought if I added more layers
on top of the music,

I could make the feelings
even more powerful.

So I got into directing music videos.

Here's what they looked like.

That's my brother, Jeff.

Sorry about this, Jeff.

(Laughter)

Here's me, just so we're even.

Incredible moves.

Should've been a dancer.

(Laughter)

These experiments grew,

and in time, started
to look more like this.

In both, I'm searching
for the same thing, though,

to capture that lightning in a bottle.

Except, I'm not.

Adding moving pictures over the music
added narrative dimension, yes,

but never quite equated the power

that just raw music had for me on its own.

This is not a great thing to realize
when you've devoted your life

and professional career
to becoming a music video director.

I kept asking myself,
did I take the wrong path?

So I started thinking: if I could
involve you, the audience, more,

I might be able to make you
feel something more as well.

So Aaron Koblin and I began
auditioning new technologies

that could put more of you
inside of the work,

like your childhood home
in "The Wilderness Downtown,"

your hand-drawn portraits,
in "The Johnny Cash Project,"

and your interactive dreams

in "3 Dreams of Black."

We were pushing beyond the screen,

trying to connect more deeply

to people's hearts and imaginations.

But it wasn't quite enough.

It still didn't have the raw
experiential power of pure music for me.

So I started chasing a new technology

that I only had read about
in science fiction.

And after years of searching,
I found a prototype.

It was a project from Nonny de la Peña
in Mark Bolas's lab in USC.

And when I tried it, I knew I'd found it.

I could taste the lightning.

It was called virtual reality.

This was it five years ago
when I ran into it.

This is what it looks like now.

I quickly started building things
in this new medium,

and through that process
we realized something:

that VR is going to play
an incredibly important role

in the history of mediums.

In fact, it's going to be the last one.

I mean this because it's the first medium
that actually makes the jump

from our internalization
of an author's expression

of an experience,

to our experiencing it firsthand.

You look confused.
I'll explain. Don't worry.

(Laughter)

If we go back to the origins of mediums,

by all best guesses,

it starts around a fire,
with a good story.

Our clan leader is telling us

about how he hunted the woolly mammoth
on the tundra that day.

We hear his words

and translate them
into our own internal truths.

The same thing happens

when we look at the cave painting
version of the story,

the book about the mammoth hunt,

the play,

the radio broadcast,

the television show,

or the movie.

All of these mediums require
what we call "suspension of disbelief,"

because there's a translation gap
between the reality of the story

and our consciousness
interpreting the story

into our reality.

I'm using the word "consciousness"
as a feeling of reality that we get

from our senses experiencing
the world around us.

Virtual reality bridges that gap.

Now, you are on the tundra
hunting with the clan leader,

or you are the clan leader,

or maybe you're even the woolly mammoth.

(Laughter)

So here's what special about VR.

In all other mediums,

your consciousness interprets the medium.

In VR, your consciousness is the medium.

So the potential for VR is enormous.

But where are we now?

What is the current state of the art?

Well,

we are here.

We are the equivalent
of year one of cinema.

This is the Lumière Brothers film

that allegedly sent a theater full
of people running for their lives

as they thought a train
was coming toward them.

Similar to this early stage of ths medium,

in VR, we also have to move
past the spectacle

and into the storytelling.

It took this medium decades

to figure out its preferred
language of storytelling,

in the form of a feature film.

In VR today, we're more learning grammar

than writing language.

We've made 15 films in the last year
at our VR company, Vrse,

and we've learned a few things.

We found that we have a unique,
direct path into your senses,

your emotions, even your body.

So let me show you some things.

For the purpose of this demo,

we're going to take every direction
that you could possibly look,

and stretch it into this giant rectangle.

OK, here we go.

So, first: camera movement
is tricky in VR.

Done wrong, it can actually make you sick.

We found if you move the camera
at a constant speed in a straight line,

you can actually get away with it, though.

The first day in film school,

they told me you have to learn
every single rule

before you can break one.

We have not learned every single rule.

We've barely learned any at all,

but we're already trying to break them

to see what kind of creative things
we can accomplish.

In this shot here, where we're moving up
off the ground, I added acceleration.

I did that because I wanted
to give you a physical sensation

of moving up off the ground.

In VR, I can give that to you.

(Music)

Not surprisingly, music matters a lot
in this medium as well.

It guides us how to feel.

In this project we made
with the New York Times's Zach Richter

and our friend, JR,

we take you up in a helicopter,

and even though you're flying
2,000 feet above Manhattan,

you don't feel afraid.

You feel triumphant for JR's character.

The music guides you there.

(Music)

Contrary to popular belief,

there is composition in virtual reality,

but it's completely
different than in film,

where you have a rectangular frame.

Composition is now
where your consciousness exists

and how the world moves around you.

In this film, "Waves of Grace,"
which was a collaboration between Vrse,

the United Nations, Gabo Arora,
and Imraan Ismail,

we also see the changing role
of the close-up in virtual reality.

A close-up in VR means
you're actually close up to someone.

It brings that character inside
of your personal space,

a space that we'd usually reserve
for the people that we love.

And you feel an emotional
closeness to the character

because of what you feel
to be a physical closeness.

Directing VR is not like
directing for the rectangle.

It's more of a choreography
of the viewer's attention.

One tool we can use
to guide your attention

is called "spatialized sound."

I can put a sound anywhere
in front of you, to left or right,

even behind you,

and when you turn your head,
the sound will rotate accordingly.

So I can use that to direct your attention
to where I want you to see.

Next time you hear someone
singing over your shoulder,

it might be Bono.

(Laughter)

VR makes us feel
like we are part of something.

For most of human history,
we lived in small family units.

We started in caves,

then moved to clans and tribes,
then villages and towns,

and now we're all global citizens.

But I believe that we are still
hardwired to care the most

about the things that are local to us.

And VR makes anywhere
and anyone feel local.

That's why it works as an empathy machine.

Our film "Clouds Over Sidra"
takes you to a Syrian refugee camp,

and instead of watching a story
about people over there,

it's now a story about us here.

But where do we go from here?

The tricky thing is that
with all previous mediums,

the format is fixed at its birth.

Film has been a sequence of rectangles,

from Muybridge and his horses to now.

The format has never changed.

But VR as a format, as a medium,

isn't complete yet.

It's not using physical celluloid
or paper or TV signals.

It actually employs what we use
to make sense of the world.

We're using your senses
as the paints on the canvas,

but only two right now.

Eventually, we can see if we will have
all of our human senses employed,

and we will have agency to live
the story in any path we choose.

And we call it virtual reality right now,

but what happens when we move
past simulated realities?

What do we call it then?

What if instead of verbally
telling you about a dream,

I could let you live inside that dream?

What if instead of just experiencing
visiting some reality on Earth,

you could surf gravitational waves
on the edge of a black hole,

or create galaxies from scratch,

or communicate with each other
not using words,

but using our raw thoughts?

That's not a virtual reality anymore.

And honestly I don't know
what that's called.

But I hope you see where we're going.

But here I am, intellectualizing
a medium I'm saying is experiential.

So let's experience it.

In your hands, you hopefully hold
a piece of cardboard.

Let's open the flap.

Tap on the power button
to unlock the phone.

For the people watching at home,

we're going to put up a card right now

to show you how to download
this experience on your phone yourself,

and even get a Google cardboard
of your own to try it with.

We played in cardboard boxes as kids,

and as adults, I'm hoping we can all find
a little bit of that lightning

by sticking our head in one again.

You're about to participate

in the largest collective
VR viewing in history.

And in that classic old-timey
style of yesteryear,

we're all going to watch something

at the exact same time, together.

Let's hope it works.

What's the countdown
look like? I can't see.

Audience: ...15, 14, 13, 12, 11, 10, 9,

8, 7, 6, 5, 4, 3, 2, 1

(Birds singing)

(Train engine)

Audience: (Shreiks)

(Video) JR: Let me tell you

how I shot the cover
of the New York Times Magazine,

"Walking New York."

I just got strapped on
outside the helicopter,

and I had to be perfectly
vertical so I could grab it.

And when I was perfectly above --

you know, with the wind,
we had to redo it a few times --

then I kept shooting.

(Video) Woman's voice: Dear Lord,

protect us from evil,

for you are the Lord,

the light.

You who gave us life took it away.

Let your will be done.

Please bring peace to the many
who have lost loved ones.

Help us to live again.

(Music)

(Video)(Children's voices)

Child's voice: There are more kids
in Zaatari than adults right now.

Sometimes I think

we are the ones in charge.

Chris Milk: How was it?

(Applause)

That was a cheap way of getting you
to do a standing ovation.

I just made you all stand.
I knew you'd applaud at the end.

(Applause)

I believe that everyone on Earth
needs to experience

what you just experienced.

That way we can collectively
start to shape this,

not as a tech platform,

but as a humanity platform.

And to that end, in November of last year,

the New York Times and Vrse made
a VR project called "The Displaced."

It launched with one million
Google cardboards

sent out to every Sunday subscriber
with their newspaper.

But a funny thing happened
that Sunday morning.

A lot of people got them

that were not the intended recipients
on the mailing label.

And we started seeing this
all over Instagram.

Look familiar?

Music led me on a path

of searching for what seemed
like the unattainable

for a very long time.

Now, millions of kids just had
the same formative experience

in their childhood

that I had in mine.

Only I think this one

surpasses it.

Let's see

where this

leads them.

Thank you.

(Applause)