"How NOT to Measure Latency" by Gil Tene

Hi everyone, I'm Gil Tene.

I'm going to be talking about this subject
that I call "How NOT to Measure Latency".

It's a subject that I've been talking
about for 3 years or so.

I keep the title and change all
the slides every time.

A bunch of this stuff is new.

So if you've seen any of my previous "How NOT to",
you'll see only some things that are common.

A nickname for the subject is this...

Because I often will get that reaction
from some people in the audience.

Ever since I've told people that it's a
nickname,

They feel free to actually exclaim,
"Oh S@%#!".

And feel free to do that here in this talk.

I'll prompt you in a couple of places
where it is natural.

But if just have the urge, go ahead.

So just a tiny bit about me.

I am the co-founder of Azul Systems.

I play around with garbage collection a lot.

Here is some evidence of me playing around
with garbage collection in my kitchen.

That's a trash compactor.

The compaction function wasn't working right,
so I had to fix it.

I thought it'd be funny to take a picture
with a book.

I've also built a lot of things.

I've been playing with computers since
the early 80's.

I've built hardware.

I've helped design chips.

I've built software at many
different levels.

Operating systems, drivers...
JVM's obviously.

And lots of big systems at the system level.

Built our own app server in the late 90's
because web logic wasn't around yet.

So, I've made a lot of mistakes,
and I've learned from a few of them.

This is actually a combination of a bunch
of those mistakes looking at latency.

I do have this hobby of depressing people
by pulling the wool up from over your eyes,

and this is what this talk is about.

So, I need to give you a choice right here.

There's the door.

You can take the blue pill,
and you can leave.

Tomorrow you can keep believing whatever
it is you want to believe.

But if you stay here and take the red pill,
I will show you a glimpse of how

far down the rabbit hole goes,
and it will never be the same again.

Let's talk about latency.

And when I say latency, I'm talking about
latency response time, any of those things

where you measure time from 'here to here',
and you're interested in how long it took.

We do this all the time, but I see a lot
of mish-mash in how people

treat the data, or think about it.

Latency is basically the time it took
something to happen once.

That one time, how long did it take.

And when we measure stuff, like we did
a million operations in the last hour,

we have a million latencies. Not one,
we have a million of them.

Our actual goal is to figure out how to
describe that million.

How did the million behave?

For example, 'they're all really good, and
they're all exactly the same', would be a

behavior that you will never see,
but that would be a great behavior.

So we need to talk about how things behave,
communicate, think, evaluate,

set requirements for, talk to other people,
but these are all common things around that.

To do that, we have to describe the
distribution, the set, the behavior,

but not the one.

For example, the behavior that says "the
the common case was x" is a piece of

information about the behavior,
but it's a tiny sliver.

Usually the least relevant one.

Well, there's some less relevant ones,
but not a strongly relevant one,

and one that people often focus on.

To take a look at what we actually do
with this stuff, almost on a daily basis,

this is a snapshot from a monitoring system.

A small dashboard on a big screen
in a monitoring system.

Where you're watching the response time of
a system over time.

This is a two hour window.

These lines that are 95th percentile,
90, 75, 50, and 25th percentiles,

you can look at how they behave over time.

We're a small audience here, if you look at
this picture, what draws your eye?

What do you want to go investigate here
or pay attention to ?

It's the big red spike there, right?

So we could look at the red spike,
cause it's different,

and say, "Woah, the 95th percentile shot up
here. And look, the 90th percentile

shot up at about the same time.

The rest of them didn't shoot up,
so maybe something happened here

that affected that much, I should probably
pay attention to it

because it's a monitoring system, and
I like things to be calm."

You could go investigate the why.

At this point, I've managed to waste
about 90 seconds of your life,

looking at a completely meaningless chart,
which unfortunately you do

every day, all the time.

This chart is the chart you want to show
somebody if you want to

hide the truth from them.

If you want to pull the wool
over their eyes.

This is the chart of the good stuff.

What's not on this chart?

The 5% worse things that happened during
this two hours.

They're not here.

This is only the good things that happened
during the things.

And to get this spike, that 5% had to be
so bad that it even pulled

the 95th percentile all up.

There is zero information here at all about
what happened bad during this two hours,

which makes it a bad fit for
a monitoring system.

It's a really good thing for
a marketing system.

It's a great way to get the bonus from your boss, even though you didn't do the work.

If you want to learn how to do that,
we can do another talk about that.

But this is not a good way to look at latency.

It's the opposite of good.

Unfortunately, this is one of the most
common tools used for

server monitoring on earth right now.

That's where the snapshot is from,
and this is what people look at.

I find this chart to be a goldmine
of information.

When I first showed it in another talk
like this, I had this really cool experience.

Somebody came up to me and said, "Hey,
as I was sitting here, I was texting one

of our guys, and he was saying,

'look, we have this issue with
our 95th percentile'."

And I got this chart from him!

So I went and said, "Hey, what does the
rest of the spectrum look like?"

This is the actual chart they got.

And when they look at the rest of the
spectrum, it looked like that.

That's what was hiding.

I noticed the scales are a little different.

That yellow line is that yellow line.

So that's a much more representative number.

Is it? Is that good enough?

That's the 99th percentile.

We still have another 1% of really bad
stuff that's hiding above the blue line.

I wonder how big that is?

I don't know because he didn't have the data.

So a common problem that we have is that
we only plot what's convenient.

We only plot what gives us nice,
colorful graphs.

And often, when we have to choose between
the stuff that hides the rest of the data,

and the stuff that is noise, we choose
the noise to display.

I like to rant about latency.

This is from a blog that I don't write
enough in, but the format for it was simple.

I tweet a single tweet about latency,
latency tip of the day,

and then I rant about my own tweet.

As an example, this chart is a goldmine
of information because it has so many

different things that are wrong in it,
but we won't get into all of them.

You can read it online.

Anyway, this is one to take away from
what we just said.

If you are not measuring and showing the
maximum value, what is it you are hiding?

And from whom?

If you're job is to hide the truth from
others, this is a good way to do it.

But if actually are interested in what's
going on, the number one indicator

you should never get rid of is the
maximum value.

That is not noise, that is the signal.

The rest of it is noise.

Okay, let's look at this chart for some
more cool stuff.

I'm gonna zoom in to a small part
of the chart, and ask you what that means.

What is the average of the 95th percentile
over 2 hours mean?

What is the math that does that?

What does it do?

Let's look at that, and I'll give you
an example with another percentile.

The 100th percentile. The max, right?

Let's take a data set.

Suppose this was the maximum every minute
for 15 minutes.

What does it mean to say that the average
max over the last 15 minutes was 42?

I specifically chose the data to
make that happen.

It's a meaningless statement.

It's a completely meaningless statement.

But when you see 95th percentile,
average 184, you think that the 95th

percentile for the last two hours
was around 184.

It makes you think that.

Putting this on a piece of paper is not
just noise and irrelevant,

it's a way to mislead people.

It's a way to mislead yourself, because
you'll start to believe your own mistruths.

This is true for any percentile.

There is no percentile that you could do
this math on.

Another tip, you cannot average percentiles.

That math doesn't happen.

But percentiles do matter. You really
want to know about them.

And a common misperception is that we want
to look at the main part of the spectrum,

not those outliers and perfection stuff.

Only people that actually bet their house
every day, or the bank on it,

need to know about the "five-nine's",
and all those.

The 99th percentile is a pretty
good number.

Is 99% really rare?

Let's look at some stuff, because we can
ask questions like, "If I were looking

at a webpage, what is the chance of me
hitting the 99th percentile?"

Of things like this: a search engine node,
or a key value store,

or a database, or a CDN, right?

Because they will report their 99th percentile.

They won't tell you anything above that,
but how many of the

webpages that we go to
actually experience this?

You want to say 1%, right?