[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Hi everyone, I'm Gil Tene.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'm going to be talking about this subject\Nthat I call "How NOT to Measure Latency".
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a subject that I've been talking now\Nabout for 3 years or so.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I keep the title and change all\Nthe slides every time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A bunch of this stuff is new.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So if you've seen any of my previous "How NOT to",\Nyou'll see only some things that are common.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A nickname for the subject is this...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Because I often will get that reaction\Nfrom some people in the audience.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ever since I've told people that it's a\Nnickname,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They feel free to actually exclaim,\N"Oh S@%#!".
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And feel free to do that here in this talk.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'll prompt you in a couple of places\Nwhere it is natural.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if just have the urge, go ahead.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So just a tiny bit about me.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I am the co-founder of Azul Systems.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I play around with garbage collection a lot.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Here is some evidence of me playing around\Nwith garbage collection in my kitchen.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's a trash compactor.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The compaction function wasn't working right,\Nso I had to fix it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I thought it'd be funny to take a picture\Nwith a book.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've also built a lot of things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've been playing with computers since\Nthe early 80's.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've built hardware.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've helped design chips.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've built software at many \Ndifferent levels.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Operating systems, drivers...\NJVM's obviously.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And lots of big systems at the system level.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Built our own app server in the late 90's\Nbecause web logic wasn't around yet.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, I've made a lot of mistakes,\Nand I've learned from a few of them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is actually a combination of a bunch\Nof those mistakes looking at latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I do have this hobby of depressing people\Nby pulling the wool up from over your eyes,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and this is what this talk is about.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, I need to give you a choice right here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's the door.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can take the blue pill, \Nand you can leave.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Tomorrow you can keep believing whatever\Nit is you want to believe.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if you stay here and take the red pill, \NI will show you a glimpse of how
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,far down the rabbit hole goes, \Nand it will never be the same again.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's talk about latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And when I say latency, I'm talking about\Nlatency response time, any of those things
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,where you measure time from 'here to here',\Nand you're interested in how long it took.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We do this all the time, but I see a lot \Nof mish-mash in how people
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,treat the data, or think about it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Latency is basically the time it took\Nsomething to happen once.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That one time, how long did it take.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And when we measure stuff, like we did \Na million operations in the last hour,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,we have a million latencies. Not one,\Nwe have a million of them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our actual goal is to figure out how to\Ndescribe that million.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How did the million behave?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For example, 'they're all really good, and\Nthey're all exactly the same', would be a
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,behavior that you will never see, \Nbut that would be a great behavior.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we need to talk about how things behave,\Ncommunicate, think, evaluate,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,set requirements for, talk to other people,\Nbut these are all common things around that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,To do that, we have to describe the \Ndistribution, the set, the behavior,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but not the one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For example, the behavior that says "the \Nthe common case was x" is a piece of
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,information about the behavior,\Nbut it's a tiny sliver.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Usually the least relevant one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well, there's some less relevant ones, \Nbut not a strongly relevant one,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and one that people often focus on.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,To take a look at what we actually do \Nwith this stuff, almost on a daily basis,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,this is a snapshot from a monitoring system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A small dashboard on a big screen \Nin a monitoring system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Where you're watching the response time of\Na system over time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a two hour window.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,These lines that are 95th percentile, \N90, 75, 50, and 25th percentiles,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you can look at how they behave over time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We're a small audience here, if you look at\Nthis picture, what draws your eye?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What do you want to go investigate here\Nor pay attention to ?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's the big red spike there, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we could look at the red spike,\Ncause it's different,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and say, "Woah, the 95th percentile shot up\Nhere. And look, the 90th percentile
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,shot up at about the same time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The rest of them didn't shoot up, \Nso maybe something happened here
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that affected that much, I should probably\Npay attention to it
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,because it's a monitoring system, and \NI like things to be calm."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You could go investigate the why.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,At this point, I've managed to waste \Nabout 90 seconds of your life,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,looking at a completely meaningless chart,\Nwhich unfortunately you do
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,every day, all the time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This chart is the chart you want to show \Nsomebody if you want to
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,hide the truth from them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you want to pull the wool \Nover their eyes.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the chart of the good stuff.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What's not on this chart?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The 5% worse things that happened during\Nthis two hours.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They're not here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is only the good things that happened\Nduring the things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And to get this spike, that 5% had to be\Nso bad that it even pulled
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the 95th percentile all up.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There is zero information here at all about\Nwhat happened bad during this two hours,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which makes it a bad fit for \Na monitoring system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a really good thing for \Na marketing system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a great way to get the bonus from your boss, even though you didn't do the work.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you want to learn how to do that, \Nwe can do another talk about that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But this is not a good way to look at latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's the opposite of good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Unfortunately, this is one of the most\Ncommon tools used for
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,server monitoring on earth right now.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's where the snapshot is from,\Nand this is what people look at.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I find this chart to be a goldmine\Nof information.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When I first showed it in another talk \Nlike this, I had this really cool experience.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Somebody came up to me and said, "Hey, \Nas I was sitting here, I was texting one
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of our guys, and he was saying,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,'look, we have this issue with \Nour 95th percentile'."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And I got this chart from him!
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So I went and said, "Hey, what does the \Nrest of the spectrum look like?"
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the actual chart they got.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And when they look at the rest of the\Nspectrum, it looked like that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's what was hiding.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I noticed the scales are a little different.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That yellow line is that yellow line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So that's a much more representative number.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Is it? Is that good enough?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's the 99th percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We still have another 1% of really bad \Nstuff that's hiding above the blue line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I wonder how big that is?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't know because he didn't have the data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So a common problem that we have is that\Nwe only plot what's convenient.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We only plot what gives us nice,\Ncolorful graphs.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And often, when we have to choose between\Nthe stuff that hides the rest of the data,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and the stuff that is noise, we choose \Nthe noise to display.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I like to rant about latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is from a blog that I don't write \Nenough in, but the format for it was simple.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I tweet a single tweet about latency, \Nlatency tip of the day,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and then I rant about my own tweet.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,As an example, this chart is a goldmine\Nof information because it has so many
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,different things that are wrong in it, \Nbut we won't get into all of them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can read it online.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Anyway, this is one to take away from \Nwhat we just said.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you are not measuring and showing the\Nmaximum value, what is it you are hiding?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And from whom?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you're job is to hide the truth from\Nothers, this is a good way to do it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if actually are interested in what's\Ngoing on, the number one indicator
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you should never get rid of is the \Nmaximum value.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That is not noise, that is the signal.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The rest of it is noise.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, let's look at this chart for some\Nmore cool stuff.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'm gonna zoom in to a small part\Nof the chart, and ask you what that means.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What is the average of the 95th percentile\Nover 2 hours mean?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What is the math that does that?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What does it do?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's look at that, and I'll give you\Nan example with another percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The 100th percentile. The max, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's take a data set.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Suppose this was the maximum every minute\Nfor 15 minutes.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What does it mean to say that the average \Nmax over the last 15 minutes was 42?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I specifically chose the data to\Nmake that happen.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a meaningless statement.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a completely meaningless statement.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But when you see 95th percentile,\Naverage 184, you think that the 95th
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,percentile for the last two hours\Nwas around 184.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It makes you think that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Putting this on a piece of paper is not \Njust noise and irrelevant,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it's a way to mislead people.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a way to mislead yourself, because \Nyou'll start to believe your own mistruths.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is true for any percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There is no percentile that you could do\Nthis math on.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Another tip, you cannot average percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That math doesn't happen.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But percentiles do matter. You really\Nwant to know about them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And a common misperception is that we want\Nto look at the main part of the spectrum,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,not those outliers and perfection stuff.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Only people that actually bet their house\Nevery day, or the bank on it,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,need to know about the "five-nine's", \Nand all those.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The 99th percentile is a pretty\Ngood number.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Is 99% really rare?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's look at some stuff, because we can\Nask questions like, "If I were looking
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,at a webpage, what is the chance of me\Nhitting the 99th percentile?"
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Of things like this: a search engine node,\Nor a key value store,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or a database, or a CDN, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Because they will report their 99th percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They won't tell you anything above that,\Nbut how many of the
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,webpages that we go to \Nactually experience this? \N
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You want to say 1%, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well, I went to some webpages and I counted\Nhow many "http" requests were generated
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,by one click into that webpage,\Nand here are the numbers.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I ended that about a year ago.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They've probably gone up since then.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now that translates into this math.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the likelihood of one click seeing\Nthe 99th percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And the only page where that is less than\N50% is the clean google search page.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Where only a quarter will see the\N99th percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The 99th percentile is the thing that most\Nof your webpages will see.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Most of them will be there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, we could look at other things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We can pick which things to focus on.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's say I had to pick between the 95th\Npercentile, and the three 9's (99.9%).
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The three 9's is way into perfection mode\Nfor most people, or they think.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Which one of those represents our \Ncommunity better?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our population?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our users?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our experience?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's run a hypothetical.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Suppose we don't have that many pages, \Nand that many resources like we said before.\N
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We'll be much more conservative.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A user session will only go through five\Nclicks, and each click will only bring up
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,up to 40 things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A lot less, and they're all as clean\Nas the google page.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How many of the users will not experience\Nsomething worse than the 95th percentile?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Because that's what the 95th percentile\Nis good for, the people who see that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Anybody above that, is that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What are the chances of not seeing it?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's an interesting number.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So you're watching a number that is \Nrelevant to 0.003% of your users.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,99.997% of your users are going to \Nsee worse than this number.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why are you looking at it?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why are you spending time\Nthinking about it?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In reverse, we could say how many people\Nare going to see something
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,worse than the three 9's (99.9%)?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's going to be 18%.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In reverse, 82% of the people will see\Nthe three 9's (99.9%) or better.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's a slightly better representation.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Probably not good enough either.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We could look at some more math with them, \Nsame kind of scenario.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What percentile of http response time \Nwill be the thing that 95%
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of people experience in this scenario?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's the 99.97 percentile that 95% \Nof people see.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And if you want to know what 99%\Nof the people see,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that's four and a half 9's (99.995%).
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You want to know that number from Akamai\Nif you want to predict what 1%
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of your users are going to experience.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When you know the 99th percentile, \Nyou kind of know a tiny bit.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So here's another tip.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And this is not an exaggeration,\Nby the way.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The median, which is a much smaller\Npercentile, has that minuscule a chance
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of ever being the number that \Nanybody experiences.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the chance of getting worse\Nthan the median.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Which makes the median an irrelevant \Nnumber to look at.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Unfortunately, it's probably the most \Ncommon one looked at.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When people say "the typical",\Nthey look at the thing that
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,everything will be worse than.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, I'm sorry about that part.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We'll do some other parts.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, why is it that when we look at these\Nmonitoring systems, we don't see
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,data with a lot of 9's?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why do we stop at the\N90, 95, 99th percentile?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why don't we look further?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, some of it is because people think, \N"Well that's perfection, I don't need it."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The other part is that it's hard.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's hard because you can't\Naverage percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We already talked about that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But you also can't derive your \Nfive 9's (99.999%) out of a lot
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of 10 second samples of percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And the reason for that is, "Hey, in 10 \Nseconds, maybe I only had 1,000 things."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I could take all the 10 seconds in the \Nworld, there's no way to say what the
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,hour five 9's (99.999%) were, what the \Nminutes five 9's were
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,if I'm collecting just this data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And unfortunately, the data being collected\Nand reported to the back ends of monitoring
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,is usually summarized at a second,\N5 seconds, 10 seconds, etc.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Basically throwing away all the good data,\Nand leaving you with absolutely no way
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,to compute large 9's for longer\Nperiods of time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, this is where you might want to look\Nat HDR Histogram.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's an open source thing I've created\Na few years ago.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I did it in Java, and know there's a\NC, C-Sharp, Python, Erlang,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and Go ports of this that I didn't create.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And it lets you actually get an entire\Npercentile spectrum.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Some of you here I know are \Nalready using it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you can look at all the percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Any number of 9's that's in the data, if \Nyou just keep it right and report it right,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it's got a log format, you can \Nstore things forever.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well, for a long time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, so it lets you have nice things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Enough for that advertisement.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, latency... Well, I think this is\Nslightly out of order.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Yeah, sorry.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the red/blue pill part, so I warn\Nyou, this is your last chance.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's a problem I call the \Ncoordinated omission problem.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The coordinated omission problem is \Nbasically a conspiracy.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a conspiracy that we're all part of.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't think anybody actually meant\Nto do it, but once I've noticed it,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,everywhere I look, there it is.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, I've been using a specific way of\Nshowing you numbers so far.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Has anybody here noticed how\NI spell percentile?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,(Audience Member): "You put lie at the\Nend of the percent sign."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Yeah, good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So coordinated omission problem is the\N"lie" in %lies.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And this is how it works.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,One common way to do this is\Nto use a load generator.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Pretty much all load generator's\Nhave this problem.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There are two that I know of that don't.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What you do with a load generator,\Nis you test.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You issue requests, or send packets.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you measure how long something took.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And as long as the numbers go right, \Nmeasure them, put them in a bucket,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,study them later, and get your \Npercentiles from it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But what if the thing that you are\Nmeasuring took longer than the time
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it would've taken until you send \Nthe next thing?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You're supposed to send something \Nevery second,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but this one took a second and a half.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well you've got to wait before\Nyou send the next one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You just avoided measuring something \Nwhen the system was problematic.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You've coordinated with it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You weren't looking at it then.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's common scenario A: You've backed\Noff, and avoided measuring when it was bad.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Another way, is you measure inside your code.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We all do this. We all have to do this,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,where we measure time, do something, \Nthen measure time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The delta between them is how long it took.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We can then put it in a stats bucket,\Nand then do the percentiles in that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Unfortunately, if the system freezes right\Nhere, for any reason,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,an interrupted contact switch,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,a cash buffer flushed to disk,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,a garbage collection,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,a re-indexing of your database,\Nthis is a database.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is Cassandra by the way, \Nmeasuring itself.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In any of the above, then you will\Nhave one bad report
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,while 10,000 things are waiting in line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And when they come in, they will look\Nreally, really good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Even though each one of them has had\Na really bad experience.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,\NIt can even get worse, where maybe the\Nfreeze happened outside the timing,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and you won't even know there was a freeze.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now these are examples of admitting data\Nthat is bad on a very selective basis.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's not random sampling.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's, "I don't like bad data",
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or "I couldn't handle it",
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or "I don't know about it",
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so we'll just talk about the good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What does that do to your data?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Because it often makes people feel like,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Okay, yeah, I understand,\Nbut it's a little bit of noise."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's run some hypotheticals, \Nand I'll show you some real numbers.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Imagine a perfect system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's doing 100 requests a second, \Nat exactly a millisecond each.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But we go and freeze the system, \Nafter 100 seconds of perfect operations
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,for 100 seconds, and then repeat.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, I'm going to describe how the system\Nbehaves in terms that should mean something,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and then we'll measure it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If we actually wanted to describe the\Nsystem,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,on the left we have an average\Nof one millisecond by the finish,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and on the right we have an\Naverage of 50 seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why 50? Because if I randomly came in \Nin that 100 seconds,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'll get anything from 0 to 100\Nwith even distribution.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The overall average over 200 seconds\Nis 25 seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If I just came in here and said, \N"Surprise, how long did this take?"
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,On average, it will be 25.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I can also do the percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,50th percentile will be really good, \Nand then it'll get really bad.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The four 9's is terrible.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a fair honest description of\Nthis system if this is what it did.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you can make the system do that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's what Control Z is good for.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can make any of your systems do that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now lets go measure this system with \Na load generator,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or with a monitoring system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The common ones.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The ones everybody does.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,On the left, we're going to get 10,000 \Nresults of one millisecond each.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Great.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And we're going to get one result of\N100 seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Wow, really big response time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is our data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is OUR data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So now you go do math with it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The average of that is 10.9 milliseconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A little less than 25 seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And here are the percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Your load generator monitoring system\Nwill tell you that this system is perfect.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You could go to production with it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You like what you see.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Look at that, four 9's.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It is lying to you.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,To your face.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you can catch it doing that with a \NControl Z-Test.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But people tend to not want to do that,\Nbecause then what are they going to do?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you just do that test, and calibrate \Nyour system, and you find it
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,telling you that, about this, the next \Nstep should be to throw all the numbers away.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Don't believe anything else it says.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If it lies this big, what else did it do?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Don't waste your time on numbers\Nfrom uncalibrated systems.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now the problem here was, that if you\Nwant to measure the system,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you have to measure at random rates, \Nor same rates.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you measure 10,000 things in 100 seconds,\Nthere should be another 10,000 things here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you measure them, you would've gotten\Nall the right numbers.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Coordinated omission is the simple act of\Nerasing all that bad stuff.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The conspiracy here is that we all do it\Nwithout meaning to.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't know who put that in our systems,\Nbut it happens to all of us .
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, I often get people saying,\N"Okay, I get it. All the numbers are wrong,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but at least for my job where I tune \Nperformance, and I try to make things
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,faster, I can use the numbers to figure\Nout if I'm going in the right direction."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Is it better, or is it worse? Let me \Ndispel that for you for a second.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Suppose I went and took this system,\Nand improved it dramatically.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Rather than freezing for 100 seconds, \Nit will now answer every question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It'll take a little longer,\N5 milliseconds instead of one,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but it's much better than freezing, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So let's measure that system that we spent\Nweeks and weeks improving,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and see if it's better.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's the data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If we do the percentiles, it'll tell us \Nthat we just really hurt the four 9's.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We made it go 5 times worse than before.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We should revert this change, go back to\Nthat much better system we had before.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So this is just to make sure that you\Ndon't think that you can have
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,any intuition based on any of these numbers.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They go backwards sometimes.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You don't know which way is good or bad.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you'll never know which way is good\Nor bad with a system that lies like that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The other cool technique is\Nwhat I call "Cheating Twice".
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You have a constant load generator,\Nand it needs to do 100 per second.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When it woke up after 200 seconds, \Nit says,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Woah, were 9,999 behind.\NWe've got to issue those requests."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So it issues those requests.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,At this point, not only did it get rid of\Nall the bad requests,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it replaced every one of them with \Na perfect request.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Coining the four 9's (99.99%), all the way\Nto four and a half 9's (99.995%),
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it's twice as wrong as dropping them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So these are all cool things that\Nhappen to you.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'm not going to spend much time on how\Nto fix those and avoid those.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's a lot of other material that you\Ncan find with me
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,talking about that, in longer talks.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But this is pretty bad.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And like I said...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That should've been up there before.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How did this repeat itself?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Did I create a loop in the\Npresentation somehow?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't know how to do that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's see if I can get through here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Hopefully editing later will take it out.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we have the cheats twice.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There, okay.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, after we look at coordinated\Nomission that way,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,we should also look at response time, \Nand service time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Coordinated omission, what it really is\Nachieving for you, unfortunately,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,is that it makes something that you think\Nis response time, and only shows you
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the service time component of latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a simple depiction of what service\Ntime and response times are.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This guy is taking a certain amount of\Ntime to take payment
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or make a cup of coffee.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's service time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How long does it take to do the work?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This person has experienced\Nthe response time,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which includes the amount of time they \Nhave to wait before they
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,get to the person that does the work.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And the difference between those\Ntwo is immense.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The coordinated omission problem makes\Nsomething that you think is
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,response time, only measure the \Nservice time,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and basically hide the fact that things \Nstalled, waited in line,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that this guy might've taken a lunch break,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and now we have line around, \Nbuilding three times.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Service time stays the same.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the backwards part...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, let's look at what it\Nactually looks like.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In a load generator that I fixed,\NI measured both
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,response time and service time,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,this happens to be Casandra,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,at a very low load.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you can see that they're very very \Nsimilar, at a very low load.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why? Because there's nobody in line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This thing is really fast.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We're not asking for too much.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Casandra's pretty fast,\Nso they're the same.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if I increase the load, we \Nstart seeing gaps.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If I increase the load a little more,\Nthe gap grows.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If I increase the load a little more,\Nthe gap grows.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now this is not the failure point yet.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If I actually increase it all the way past\Nthe point where the system
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,can't even do the work I want, \Nservice time stays the same,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,response time goes through the roof.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This was when it was 100 and something\Nmilliseconds, now it's 7 and a half seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why 7 and a half seconds?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Cause you're waiting in line that long\Nto go around the block.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The guy just can't serve as many people\Nas are showing up in line, you fall behind.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a virtual world reaction to this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I really like this slide, it's where I came\Nup with the notion of a blue/red pill.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When you actually measure reality, people\Ntend to have this reaction when
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,they compare the two.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And if we actually look at these on the\Ntwo sides of a collapse point of a system,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,this specific system can only do 87,000 \Nthings a second.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,No matter how hard you press it,\Nthat's all it'll do.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The service time on the two sides of \Nthe collapse looks virtually identical,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which it would.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if you compare the response time, \Nyou have a very different picture.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And I'm showing this picture so you get \Na feeling for what to look at
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,on whether or not you're measuring\Nthe right one.