-
Not Synced
Hi everyone, I'm Gil Tene.
-
Not Synced
I'm going to be talking about this subject
that I call "How NOT to Measure Latency".
-
Not Synced
It's a subject that I've been talking
about for 3 years or so.
-
Not Synced
I keep the title and change all
the slides every time.
-
Not Synced
A bunch of this stuff is new.
-
Not Synced
So if you've seen any of my previous "How NOT to",
you'll see only some things that are common.
-
Not Synced
A nickname for the subject is this...
-
Not Synced
Because I often will get that reaction
from some people in the audience.
-
Not Synced
Ever since I've told people that it's a
nickname,
-
Not Synced
They feel free to actually exclaim,
"Oh S@%#!".
-
Not Synced
And feel free to do that here in this talk.
-
Not Synced
I'll prompt you in a couple of places
where it is natural.
-
Not Synced
But if just have the urge, go ahead.
-
Not Synced
So just a tiny bit about me.
-
Not Synced
I am the co-founder of Azul Systems.
-
Not Synced
I play around with garbage collection a lot.
-
Not Synced
Here is some evidence of me playing around
with garbage collection in my kitchen.
-
Not Synced
That's a trash compactor.
-
Not Synced
The compaction function wasn't working right,
so I had to fix it.
-
Not Synced
I thought it'd be funny to take a picture
with a book.
-
Not Synced
I've also built a lot of things.
-
Not Synced
I've been playing with computers since
the early 80's.
-
Not Synced
I've built hardware.
-
Not Synced
I've helped design chips.
-
Not Synced
I've built software at many
different levels.
-
Not Synced
Operating systems, drivers...
JVM's obviously.
-
Not Synced
And lots of big systems at the system level.
-
Not Synced
Built our own app server in the late 90's
because web logic wasn't around yet.
-
Not Synced
So, I've made a lot of mistakes,
and I've learned from a few of them.
-
Not Synced
This is actually a combination of a bunch
of those mistakes looking at latency.
-
Not Synced
I do have this hobby of depressing people
by pulling the wool up from over your eyes,
-
Not Synced
and this is what this talk is about.
-
Not Synced
So, I need to give you a choice right here.
-
Not Synced
There's the door.
-
Not Synced
You can take the blue pill,
and you can leave.
-
Not Synced
Tomorrow you can keep believing whatever
it is you want to believe.
-
Not Synced
But if you stay here and take the red pill,
I will show you a glimpse of how
-
Not Synced
far down the rabbit hole goes,
and it will never be the same again.
-
Not Synced
Let's talk about latency.
-
Not Synced
And when I say latency, I'm talking about
latency response time, any of those things
-
Not Synced
where you measure time from 'here to here',
and you're interested in how long it took.
-
Not Synced
We do this all the time, but I see a lot
of mish-mash in how people
-
Not Synced
treat the data, or think about it.
-
Not Synced
Latency is basically the time it took
something to happen once.
-
Not Synced
That one time, how long did it take.
-
Not Synced
And when we measure stuff, like we did
a million operations in the last hour,
-
Not Synced
we have a million latencies. Not one,
we have a million of them.
-
Not Synced
Our actual goal is to figure out how to
describe that million.
-
Not Synced
How did the million behave?
-
Not Synced
For example, 'they're all really good, and
they're all exactly the same', would be a
-
Not Synced
behavior that you will never see,
but that would be a great behavior.
-
Not Synced
So we need to talk about how things behave,
communicate, think, evaluate,
-
Not Synced
set requirements for, talk to other people,
but these are all common things around that.
-
Not Synced
To do that, we have to describe the
distribution, the set, the behavior,
-
Not Synced
but not the one.
-
Not Synced
For example, the behavior that says "the
the common case was x" is a piece of
-
Not Synced
information about the behavior,
but it's a tiny sliver.
-
Not Synced
Usually the least relevant one.
-
Not Synced
Well, there's some less relevant ones,
but not a strongly relevant one,
-
Not Synced
and one that people often focus on.
-
Not Synced
To take a look at what we actually do
with this stuff, almost on a daily basis,
-
Not Synced
this is a snapshot from a monitoring system.
-
Not Synced
A small dashboard on a big screen
in a monitoring system.
-
Not Synced
Where you're watching the response time of
a system over time.
-
Not Synced
This is a two hour window.
-
Not Synced
These lines that are 95th percentile,
90, 75, 50, and 25th percentiles,
-
Not Synced
you can look at how they behave over time.
-
Not Synced
We're a small audience here, if you look at
this picture, what draws your eye?
-
Not Synced
What do you want to go investigate here
or pay attention to ?
-
Not Synced
It's the big red spike there, right?
-
Not Synced
So we could look at the red spike,
cause it's different,
-
Not Synced
and say, "Woah, the 95th percentile shot up
here. And look, the 90th percentile
-
Not Synced
shot up at about the same time.
-
Not Synced
The rest of them didn't shoot up,
so maybe something happened here
-
Not Synced
that affected that much, I should probably
pay attention to it
-
Not Synced
because it's a monitoring system, and
I like things to be calm."
-
Not Synced
You could go investigate the why.
-
Not Synced
At this point, I've managed to waste
about 90 seconds of your life,
-
Not Synced
looking at a completely meaningless chart,
which unfortunately you do
-
Not Synced
every day, all the time.
-
Not Synced
This chart is the chart you want to show
somebody if you want to
-
Not Synced
hide the truth from them.
-
Not Synced
If you want to pull the wool
over their eyes.
-
Not Synced
This is the chart of the good stuff.
-
Not Synced
What's not on this chart?
-
Not Synced
The 5% worse things that happened during
this two hours.
-
Not Synced
They're not here.
-
Not Synced
This is only the good things that happened
during the things.
-
Not Synced
And to get this spike, that 5% had to be
so bad that it even pulled
-
Not Synced
the 95th percentile up.