Spotify Engineering Culture - Part 1

An attempt to describe our engineering culture.

This is a journey in progress, not a journey completed. So the stuff in the video isn't all true for all squads, but it appears to be mostly true for most squads.

(NOTE - Part 2 hasn't been recorded yet. Stay tuned!)

One of the big success factors here

at Spotify is our Agile Engineering Culture.

Culture tends to be invisible.

We don't notice it

because it's there all the time.

It's kind of like the air we breathe.

But, if everyone understands the culture,

we're more likely to be able to keep it ..

.. and even strengthen it as we grow.

So, that's the purpose of this video.

When our first music player

was launched in 2008.

We were pretty much just a scrum company.

Scrum is a well established

agile development approach ..

and it gave us a nice team-based culture.

However, a few years later,

we had grown into a bunch of teams

and found that some of

standard scrum practices

were actually getting into the way.

So, we decided to make all these optional.

Rules are good start, but then

break them when needed.

We decided Agile matters

more than Scrum and ...

... Agile Principles matter

more than specific practices.

So, we renamed

the Scrum Master role to Agile Coach.

Because, we wanted Servant Leaders

more than a Process Masters.

We also started using the term

Squad instead of Scrum Team.

And our key driving force became Autonomy.

So, what is an Autonomous Squad?

A Squad is a small cross-functional

self-organizing team ..

usually less than eight people.

They sit together ..

and they have ene-to-end responsibility

for the stuff they build,

design, commit, deploy,

maintenance, operations,

the whole thing.

each Squad has a long-term mission ..

such as "Makes Spotify

the best place to discover music."

or internal stuff like

infrastructure for A/B Testing

Autonomy basically means that

the squad decides what to build,

.. how to build it and

.. how to work together

while doing it?

There are of cause

some boundaries to this such as

the squad mission,

the overall product strategy

for whatever area they are working on

and short-term goals

that are re-negotiated every quarter.

Our office is optimized for collaboration.

Here is a typical squad area.

Squad members work together here

with adjustable desks

and easy access to each other's screens.

We gather over here

at the lounge for thing like

planning session and retrospectives.

and back, there is a huddle room

for smaller meetings or

just get some quite time.

Almost all walls are whiteboards.

So, why is autonomy so important.

Well, because it's motivating

and motivated people

build better stuff.

Also, autonomy makes us fast

by letting decisions happened locally

in the squad instead of

via a bunch of mangers

and committees and stuff.

It helps us minimize hand-offs

and waiting.

So, we can scale

without block out with dependencies

and coordination.

Although, each squad has

its own mission.

They need to be aligned

with product strategy,

company priorities,

and other squads.

Basically, be a good citizen

in this Spotify eco-system.

Spotify's overall mission is

more important than

any individual squad.

So, the key principle is

really be autonomous

but don't sub-optimized.

It's a kind of like a Jazz man.

Although, each musician is autonomous

and plays his own instrument

They listen to each other

and focus on the whole song

together

That's how great

music is created.

So, our goal is loosely coupled

but tightly aligned squads

We're not all there yet.

But, we have experiment

a lot with different ways

of getting closer

In fact, that applies

to most things in this video

This culture description is

a mix of what we are today

and what we are trying to become

in the future.

Alignment and Autonomy

may seem like

different end of a scale.

as in more automy

equals less alignment.

However, we think of it more like a

two different dimensions.

Down here is

low alignment and low autonomy,

a micro-management culture.

No high-level purpose

, just shut up and

follow orders.

Up here is

high alignment, but

still low autonomy.

So, leaders are

good at communicating

what problem needs

to be solved.

But, they also tell

people how to solve it.

High alignment and high autonomy

means leaders focus on

what problem to solve.

But, let the team

figure out how to solve it.

What about down here then?

Low alignment

and high autonomy

means team do whatever they want

and basically all running in

different directions.

Leaders are hopeless

and our product

becomes a Frankenstein.

We are trying hard

to be up here.

aligned autonomy

and we keep experimenting

with different ways

of doing them.

So, alignment enables autonomy.

The stronger alignment we have

, the more autonomy we can effort to grant.

That means the leader's job

is to communicate

what problem needs to be solved

and why?

and the squads collaborate with each other

to find the best solution.

One consequence of autonomy

is that we have

very little standardization.

When people ask things like

which code editor that you use

or how do you plan?

The answer is mostly

depends on which Squad

some do Scrum sprints

others do Kanban

some estimate story

and measure velocity

others don't

it's really up to

each squad

Instead of formal standards

we have strong culture of

cross-pollination

when enough Squads use

a specific practice

or tool such as

Git. That becomes the path of

least resistant

and other Squads tend to pick

the same tool

Squad starts supporting

that tool and helping

each other

and it becomes

like a de facto standard.

This informal approach

gives us a healthy balance

between consistency and

flexibility.

Our architecture is based on

over a hundred separated systems,

coded and deployed

independently.

There are plenty of interactions.

But, each system focuses on

one specific need

such as play list management

search or monitoring.

We tried to keep them

small and de-coupled

with clear interfaces and protocols.

Technically, each system is owned by one Squad.

In fact, most Squads own several.

But, we have an

internal Open-source model

and our culture is more about

sharing than owning

suppose Squad one here needs

something done in System B

and Squad 2 knows that code best

They typically asked Squad 2 to do it.

However, if Squad 2 doesn't have time.

or they have other priorities.

Then, Squad 1 doesn't necessary need to wait

We hate waiting.

Instead, they're welcome

to go ahead edit

the code themselves

and asked Squad 2

to review the changes.

So, anyone can edit any code.

But, we have a culture of

peer code review.

This improve quality and

more importantly spread knowledge.

Over time, we evolved design guidelines

, code standards, and other things

to reduce engineering friction.

But, only when badly needed.

So, on a scale from

Authoritative to Liberal

We're definitely on

the Liberal side.

Now, none of this would work

if it wasn't for the people.

We have a really strong culture

of mutual respect.

I keep hearing comments like

my colleagues are awesome.

People often give credit to each other

for great work and

seldom take credit for themselves.

Considering how much talent

we have here

There is surprisingly little ego.

One big Ah-ha for new hires

is that autonomy

is kind of scary at first.

You and your Squad mates

are expected to find

you own solution.

No one will tell you

what to do.

But, it turns out

if you asked for help

you get lots of it

and fast.

There is genuine respect

for the fact that

We're all in this boat together

and need to help

each others succeed.

We focus a lot on motivation.

Here is an example ...

an actual email from the Head of People Operations

Hi Everyone,

Our employee satisfaction survey

says 91% enjoy working here

and 4% don't.

Now that may seem like

a pretty high satisfaction rate

especially considering our growth pain.

from 2006 to 2013

we've doubled every year.

and now have over 1200 people.

But, then it continues

This is of course not satisfactory

and we want to fix it.

If you're one of those unhappy 4%,

please contact us.

We're here for your sake,

and nothing else.

So, good enough isn't

good enough.

Half a year later

things have improved

and satisfaction rate was up to 94%.

This strong focus on motivation

has helped us build up

a pretty good reputation

as a workplace.

But, we still have plenty of problems

to deal with. So, yeah

we need to keep improving.

Okay, so we have over 50 Squads

spread across 4 cities.

Some kind of structure is needed.

Currently, Squads are grouped

into Tribes.

A Tribe is

a light-weight matrix

Each person is member of a Squad

as well as a Chapter.

The Squad is a primary dimension,

focusing on product delivery

and quality.

While the Capter is a competency area.

such as Quality Assitant, Agile Coaching, ..

or Web Development.

As Squad member,

my Chapter lead is my formal line manager

is servant leader focusing on

coaching and mentoring me as an engineer.

So, I can switch Squad without getting a new manager.

It's a pretty picture, hah?

Except that it's not really true.

In reality, the lines aren't nice and straight

and things keep changing.

Here is a real-life example.

from one moment in time

for one Tribe.

and of cause it's all different by now.

and that's ok.

The most valuable communication happens

informal and un-predictable ways.

To support this, we also have Guilds.

A Guild is a lightweight community of interest.

where people across the whole company gather

and share knowledge within a specific area.

For example Leadership, Web Development,

or Continuous Delivery.

Anyone can join or leave the Guild at anytime.

Guilds typically have a mailing list,

bi-annual un-conferences, and ...

other informal communcation methods.

Most organizational charts are illusion.

So, our main focus is community

rather than hierarchical structures.

We found that a strong enough community

can get away with

an informal volatile structure.

If you always need to know exactly

who is making decisions.

You're in the wrong place.

One thing that matters a lot for autonomy

is how easily get our stuff into Production?

If releasing is hard,

we'll be tempted to release it seldom

to avoid the pain.

That means each release is bigger

and therefore even harder.

It's vicious cycle.

But if releasing is easy,

we can release often.

That means each release is smaller

and therefore easier.

To stay in this loop

and avoid that one.

We encourage small and frequent releases

and invest heavily in Test Automation

and Continuous Delivery infrastructure.

Release should be routine not drama.

Sometimes, we make big investments

to make releasing easier.

For example, the original Spotify desktop client

was a single monolithic application.

In the early days,

we've just a handful of developers.

That was fine.

But, as we grew, this became a huge problem.

Dozen of Squads have to synchonize

with each other for each release.

And it could take months to get

a stable version.

Instead of creating lots of process

and rules and stuff to manage this.

We changed the architecture

to enable Decupled Releases.

Using Chromium embedded framework,

the client is now basically a web browser

in disguised. Each section is

like a frame on the website

and Squad can release

their own stuff directly.

As part of this architectural change,

we started seeing each client platform

as a client app

and evolve three different flavors of Squads.

Client App Squads

Feature Squads

and Infrastructure Squads.

A Feature squad focuses on

one feature area

such as Search.

This squad will build, ship,

and maintain search related features

of our platform.

A Client App squad focuses on

making release easy

on one specific client platform

such as desktop, iOS, or Android.

An Infrastructure squad focuses on

making other squads more effective.

They provide tools and routines

for thing like continuous delivery

A/B Testing, Monitoring, and Operations.

Regardless of the current structure,

we always strive for a Self-service model.

A kind of like a buffet,

the restaurant staffs

don't serve you directly.

They enable you to serve yourself.

So, we avoid hand-offs like the plague.

For example, an operation squad

or client app squad does not

put code into production

for people.

Instead, their job is to make it easy

for feature squad to put their own code

into production.

Despite the self-service model, we sometimes

need a bit of sync between squads

when doing releases.

We manage this using

Release Trains and Feature Toggles.

Each client app has a release train

that departs a regular schedule.

Typically every week or every three weeks

depends on which client.

Just like in physical world

if trains depart frequently and reliably,

you don't need much up-front planning.

Just show up and take the next train.

Suppose these three squads

are building stuff and

when the next release train arrives

feature A, B, and C are done

while D is still in progress.

The release train will include all 4 features,

but the unfinished one is hidden

using a feature toggle.

It may sound weird

to release unfinished features

and hide them.

But, it's nice because

it exposes integration problem early

and minimize the need for code branches.

Un-mearged code hides problems

and it's a form of technical debt.

Feature toggles let us dynamically

show and hide stuff in test

as well as production.

In addition to hiding unfinished work,

we use this to A/B Testing

gradually roll-out finished features.

All in all our release processes is better

than it's use to be.

But, we still see plenty of improvement areas

so we'll keep experimenting.

This may seem like a scary model

letting each Squad put their own stuff

into production

without any formal centralized control.

and we do screw up sometimes.

But we learned that trust

is more important than control.

Why would we hire someone

who we don't trust.

Agile at scale requires

trust at scale

and that means that no politics.

It also means no fear.

Fear doesn't just kill trust.

It kills innovation

because the fear to get punished

people won't dare try new things.

So, let's talk about failure.

Actually, no

let's take a break.

Get on your feet,

get some coffee.

Let this stuff sink in for a bit

and then come back

when you're ready for Part 2. Okay?