Valve's "Secret Weapon"

About a year into the development of Portal,

playtesters kept giving 
Valve very similar feedback.

Something along the lines of "that was a great 
tutorial, I can't wait to play the actual game".

*VCR Pause Sound Effect*

Uh. Slight problem. That was the actual game.

Despite playing through 14 or so 
hand-crafted puzzles, players were

clearly missing some vital element that would 
tell them this was, indeed, a video game.

And so after a lot of discussion, Valve 
decided that the game needed an antagonist:

someone to push back against the player,

to provide motivation to keep moving, 
and to put these puzzles into context:

this was all training in order to defeat a boss.

The end result was, of course, GLaDOS:

a loopy AI overlord with a 
biting, passive-aggressive wit.

She teases you, taunts you, tricks you.

And the whole game leads up to a climactic 
one-on-one showdown in her central chamber.

GLaDOS: "Well, you found me. 
Congratulations. Was it worth it?

Because despite your violent behaviour, the only 
thing you've managed to break so far is my heart."

Thanks to sharp writing and killer voice acting,

she has become one of the most iconic 
video game villains of all time.

But, says Valve's Robin Walker, "her 
genesis begins with a straightforward

process of us trying to solve the 
core gameplay problem in Portal".

Now if you ask me, the fact that 
GLaDOS may never have existed if

it wasn't for this feedback goes 
to show the value of playtesting.

Not to be confused with quality 
assurance - that's for rooting out bugs.

Or focus testing - that's for market research.

No: playtesting is simply watching 
people play through a chunk of the game

- sometimes with a questionnaire 
or interview at the end -

and then using what you see and hear 
to drive changes in the game's design.

For example, if Valve sees that 
players are routinely getting

themselves killed when trying 
to redirect these energy balls

- perhaps they should make a change -

like having it so you can only place portals on 
walls that are way above the player's height.

And so on Portal, this approach was used to 
touch up almost every aspect of the game.

Valve used playtesting to make sure 
the learning curve was perfect,

they used it to remove moments of frustration,

to make sure players noticed key objects,

to improve the pacing, to tweak the difficulty,

and to ensure the storyline was coherent.

Playtesting even led to the game's iconic visual 
style, with those sterile white walls and floor.

Originally, the game featured 
cluttered and grungy environments,

but playtesters found it hard to 
identify the key puzzle elements.

In one test, a player spent half an hour trying to

push a shelf onto a button - while 
completely ignoring a nearby box.

Kim Swift - one of the key developers 
on the project - calls playtesting

"the most important thing that we 
learned since coming to Valve".

You see, if you didn't know, Portal 
started life as Narbacular Drop:

a student project made at 
Digipen University in Washington.

hey were invited to show the game off at 
Valve's headquarters and about halfway

through the demo, CEO Gabe Newell stopped the 
presentation to offer the entire team a job.

Their goal was to remake the game in the source 
engine, and within the Half-Life universe.

And to help, Valve introduced the small 
student team to its game development process.

Here's how it works.

You start with a goal: perhaps, in this case,

to make a puzzle that is clearly 
readable, and satisfying to solve.

You then take a stab at reaching that goal,

by designing something - in 
this case, a test chamber.

Next, you evaluate whether your design 
reached that goal by doing a playtest.

And if it doesn't meet the grade, 
change the design and repeat.

Valve keeps going with this 
iterative process until it

is "no longer excruciatingly painful to watch 
the playtests", says developer David Speyrer.

GLaDOS: "Fantastic! You remained 
resolute and resourceful in an

atmosphere of extreme pessimism."

Perhaps, given the subjects 
of its most famous games,

its apt that Valve sees game development as 
an engineering problem or a scientific study.

Valve's former in-house psychologist, 
Mike Ambinder, says "we see our game

designs as hypotheses and our playtests as 
experiments to validate these hypotheses".

Now, it should be said that playtesting 
is by no means exclusive to Valve.

And indeed, every game developer sources 
player feedback at some point in the process.

But the difference is that Valve is obsessed 
with it, to an almost religious degree.

Ambinder calls playtesting "the most important 
part of the game development process".

Newell calls it "our secret weapon".

And a designer who joined the studio in 2018 said

“I always heard that Valve does a lot of 
playtesting before I came to work here,

and I was not prepared for the amount 
of playtesting this company does".

Valve does playtesting early.

Swift and her team started running playtests of 
Portal after just one week of starting to build

the game at Valve, even though they only 
had one half-finished room to show people.

And then do it often.

Portal was playtested practically every week.

They'd do a playtest on Friday, 
discuss the results on Monday,

apply the lessons the rest of the 
week, and test again on Friday.

It sounds extreme.

But I think this devotion to playtesting 
makes a lot more sense when you learn

about the disastrous development of 
Valve's very first game, Half-Life.

About two months before 
Half-Life was supposed to ship,

Valve realised that the game... well, sucked.

The developer said "you couldn't play the 
game all the way through, none of the levels

tied together well, and there were serious 
technical problems with most of the game.

As a whole the game just wasn't working."

The only real option on the table was to 
scrap the whole thing and start from scratch.

But this time, they would do things 
differently - which lead to two game

development philosophies that are 
still present in the studio today.

One is the idea of the cabal: where 
the staff is split up into tiny,

multi-disciplined teams who each take complete 
ownership of a small chunk of the game.

But the other is frequent playtesting, from 
the very earliest point of development.

This time, if the game sucked, 
they wanted to know right away.

So, about three months into development of the 
restarted Half-Life, they started playtesting.

They found random gamers in video game shops,

and contacted people who filled in those little 
registration cards you got in old PC boxes.

They'd sit them down in front of 
the game and just watch them play.

Each test would result in dozens 
of things that needed to be fixed,

changed, added, or deleted from the game.

Like, after watching some playtesters break 
every crate in a level, Valve realised it

probably needed to stuff some of those boxes 
with goodies like ammo and health packs.

And this new process obviously worked: this 
version of Half-Life was a huge success,

and is now seen as one of the most 
important and influential games ever made.

And, as such, Valve would use playtesting 
extensively in all of its future games.

On Half-Life 2, Valve planned to give out 
the gravity gun towards the end of the game,

but players loved using it so much, the team 
decided to make it available much sooner.

On Left 4 Dead, playtesters found it hard 
to find teammates who were in trouble,

which led to the x-ray outlines 
of your fellow survivors.

And on Portal 2, a paint type that 
let you walk on walls was scrapped,

when it made multiple playtesters feel queasy.

The arrival of Steam allowed Valve 
to go further - converting millions

of online players into post-launch playtesters.

When Steam's data showed that lots of 
players were getting stuck in Episode One,

they released a patch to reduce the 
difficulty of a tricky siege battle.

Plus, playtesting was instrumental 
when exploring an all-new technology,

virtual reality, for Half-Life: Alyx.

For instance, Valve learned that a player's 
tolerance for standing around watching people

talk was significantly lower in virtual 
reality, so had to speed up the game's pace.

"Player behaviour helped us navigate VR 
development," says Valve's Christine Phelan.

"You could almost think of the player as another 
designer. There is barely a moment in the game

that wasn't improved by what playtesters told 
us and showed us through their play behaviour".

So, from Half-Life 1 to Half-Life: Alyx, 
playtesting has been an invaluable tool for Valve.

So I want to share some tips from the developers, 
to show you how to playtest more effectively.

Though it should be stated: Valve 
is not like other companies.

From its unorthodox organisational 
structure to its near infinite resources,

there are very few companies on 
Earth who can copy what Valve does.

And also: this approach may work best on super 
linear, extremely hand-crafted story games

- and other techniques, like heat maps, 
may work better for other types of game.

But, with those caveats out of the way,

perhaps their experience can still 
provide some useful guidance.

So, tip number one is test early.

Valve says "playtesting is where we make

the vast majority of our most 
important changes to our game.

So we try to do it as early as possible on 
the project, to get the most value out of it."

When you identify a problem early 
on, you have time to go back to

the drawing board and develop an 
effective solution to the problem.

But do it too late, and sometimes the only way to 
fix a problem is with a flimsy band-aid solution.

Like, uh, I dunno, having the characters just 
tell you the solution to a crummy puzzle.

Valve may start testing within a few days of 
prototyping a mechanic or designing a level.

Even if it looks hideously ugly, with programmer 
art or bright orange textures on all the walls.

But this actually helps them avoid 
wasting time on other aspects:

there's no point investing heavily 
in art or audio if the game mechanic

is gonna be completely changed 
- or cut from the game entirely.

Number two is test often.

Valve typically tests every single week,

to make sure that they are constantly 
iterating on player feedback.

This also leads to a huge amount of data.

Each Half-Life 2 chapter had about 
100 playtesters, for example.

And that mass of information can be invaluable.

By having a huge number of people look 
at the game, Valve can look for common

trends in the feedback, and avoid tweaking 
the game based on a few weird outliers.

And the more experience you have seeing 
people interact with your designs,

the better you'll get at preempting 
this feedback in the future.

Gabe Newell says "after you've watched a couple 
hundred playtests, you start to develop a much

better sense of what are successful 
and unsuccessful design strategies".

So, some famous lessons learnt 
at Valve include "players don't

learn when stressed" and "players don't look up".

Number three is shut up and watch.

A playtest should try to simulate the 
actual experience of playing the game.

Which means the observers need to stay 
quiet for the duration of the test:

no hints, no guidance, no 
answers to burning questions.

"Nothing is quite so humbling as being forced 
to watch in silence as some poor playtester

stumbles around your level for 20 minutes, 
unable to figure out the 'obvious' answer

that you now realise is completely arbitrary and 
impossible to figure out", the studio has said.

Post-game interviews and 
questionnaires can certainly happen

- those were instrumental in solving 
the GLaDOS problem after all -

but you'll often learn more by watching 
players than by talking to them.

Swift says "they may tell you later 
that they like the game but you'll

really tell by their body language whether 
or not they actually enjoyed themselves".

It's also a common game design trope that

playtesters like to offer potential 
solutions to the problems they face

- but as they don't have insider 
knowledge of your vision or constraints,

those solutions are usually better left ignored.

Number four is designers should run playtests.

Valve does not outsource testing to a special 
department or leave it with a publisher:

playtesting is done by the people 
who are actually responsible for the

design and execution of the 
level or mechanic in question.

The cabal, from earlier.

This shows the developers 
exactly what needs fixing,

and can give much-needed 
motivation to improve the game.

And player behaviour might inspire new puzzle 
solutions or ideas for the designers watching.

Like, in Half-Life Alyx, players would 
instinctively cover their real-life mouth

to stop Alyx from coughing, and alerting 
a gigantic blind zombie named Jeff.

So Valve turned covering your 
mouth into an actual game mechanic.

Number five is get the right people.

Valve gets feedback from a huge 
variety of different playtesters,

from fellow staff members to 
little kids to pro gamers.

But its learned to always have a target 
audience in mind when making changes.

Like, when Valve was figuring out how to 
make a climactic boss fight against GLaDOS,

they initially got feedback 
from hardcore FPS players,

who said the level needed more action, 
more challenge, and more skill.

But that idea was a dud:

when facing this tricky boss 
fight, says Erik Wolpaw,

"the vast majority of playtesters who 
had gotten used to the slower-paced,

cerebral nature of Portal were just 
frustrated, confused, and dissatisfied".

They still wanted to satisfy 
those hardcore FPS players,

but did so through optional content, like 
Portal's advanced chambers and challenge maps.

Number six is to challenge your assumptions.

Okay, so maybe the GLaDOS boss fight 
didn't need to be a test of skill.

So, perhaps it needed to be the most 
complex puzzle in the entire game?

Surely that would be a suitable climax.

But, here's the thing - playtesters 
thought the game's mid-point escape,

that bit where you use a portal to 
avoid falling into a pit of fire

- players thought that was 
incredibly climactic and satisfying.

Even though it's basically 
the easiest puzzle in Portal.

But Valve realised the time 
pressure, the visual impact,

and the high drama all made it 
way more exciting to players.

So maybe they were overthinking 
the final boss, too.

"We've been holding on to this idea that 
we need a complex puzzle at the end and

it simply wasn't true", says Swift, and so 
they settled on a pretty simple sequence.

It's very easy for game designers to make 
wonky assumptions about how players will act

- I recently made a whole video about 
doing this in my own puzzle game.

I'll drop a link to that one in 
the end screen of this video.

Perhaps the most important tip, though, 
is that playtesting feedback is just data,

and it's up to the designer how that data 
is interpreted, filtered, and applied.

For example, Half-Life 2 originally 
had a very short introduction before

Gordon Freeman grabbed a gun and started shooting.

Playtesters loved it.

Who wouldn't be excited when 
jumping into the action?

And so, given this good feedback, it would be 
easy to keep the game exactly like that.

But, despite the positive sentiment, writer 
Marc Laidlaw says Valve decided to keep working.

They thought it would be better to delay combat.

"We wanted you to witness the cops doing 
something horrible, and feel like what you

were doing was a response to that - not that 
you were just a killing machine," says Laidlaw.

And they thought it would lead to 
a better emotional pay-off if you

had to wait a much longer time before 
finally getting Gordon's iconic crowbar.

BARNEY: "Oh, and before I forget. I think 
you dropped this back in Black Mesa.

Good luck out there, buddy."

Here's the thing: playtesting 
is as useful as you make it.

If you just bend the game to the whims and 
desires of every playtester who comes through,

you'll end up with a dumbed-down game, 
or bland, design-by-committee sludge.

But if you go in with a clear goal.

A specific game you want to make, for a 
specific audience you want to entertain.

And then use playtesting to validate 
whether you are hitting that goal,

you can use feedback to unlock the 
incredible game you are trying to make.

Oh, don't click off.

I've got one last Portal story 
while the Patreon credits roll.

So, the invention of GlaDOS is actually not 
the most dramatic change that Valve has made,

in response to player feedback.

During an internal game jam, held 
shortly after the release of Portal 1,

a bunch of Valve developers came up with 
an experimental puzzle game called F-Stop.

In this game, you use a camera 
to take photos of objects.

Then, you can spawn that object elsewhere in the world 
- perhaps at a completely different scale.

Gabe Newell loved the concept, and said it 
should be developed as a follow-up to Portal

- meaning each game in the series would feature

a different piece of technology 
developed at Aperture Science.

But, after nearly a year of development,

playtesters were very vocal in their feedback: 
Portal without portals just didn't work.

And so Valve decided to scrap the game 
and start Portal 2 all over again.

In recent years, we've been able to 
see what F-Stop might have looked like,

as fans have used leaked assets and code 
to make various recreations of the game.

But Valve never returned to the concept.

If you want to play something similar,

check out Superliminal, or wait for the 
upcoming photography puzzler, Viewfinder.

Thanks for watching.