[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Hi everyone, I'm Gil Tene.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'm going to be talking about this subject\Nthat I call "How NOT to Measure Latency".
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a subject that I've been talking\Nabout now for 3 years or so.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I keep the title and change all\Nthe slides every time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A bunch of this stuff is new.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So if you've seen any of my previous "How NOT to",\Nyou'll see only some things that are common.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A nickname for the subject is this...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Because I often will get that reaction\Nfrom some people in the audience.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Ever since I've told people that it's a\Nnickname,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They feel free to actually exclaim,\N"Oh S@%#!".
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And feel free to do that here in this talk.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'll prompt you in a couple of places\Nwhere it is natural.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if just have the urge, go ahead.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So just a tiny bit about me.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I am the co-founder of Azul Systems.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I play around with garbage collection a lot.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Here is some evidence of me playing around\Nwith garbage collection in my kitchen.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's a trash compactor.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The compaction function wasn't working right,\Nso I had to fix it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I thought it'd be funny to take a picture\Nwith a book.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've also built a lot of things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've been playing with computers since\Nthe early 80's.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've built hardware.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've helped design chips.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I've built software at many \Ndifferent levels.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Operating systems, drivers...\NJVM's obviously.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And lots of big systems at the system level.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Built our own app server in the late 90's\Nbecause web logic wasn't around yet.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, I've made a lot of mistakes,\Nand I've learned from a few of them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is actually a combination of a bunch\Nof those mistakes looking at latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I do have this hobby of depressing people\Nby pulling the wool up from over your eyes,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and this is what this talk is about.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, I need to give you a choice right here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's the door.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can take the blue pill, \Nand you can leave.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Tomorrow you can keep believing whatever\Nit is you want to believe.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if you stay here and take the red pill, \NI will show you a glimpse of how
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,far down the rabbit hole goes, \Nand it will never be the same again.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's talk about latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And when I say latency, I'm talking about\Nlatency response time, any of those things
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,where you measure time from 'here to here',\Nand you're interested in how long it took.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We do this all the time, but I see a lot \Nof mish-mash in how people
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,treat the data, or think about it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Latency is basically the time it took\Nsomething to happen once.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That one time, how long did it take.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And when we measure stuff, like we did \Na million operations in the last hour,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,we have a million latencies. Not one,\Nwe have a million of them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our actual goal is to figure out how to\Ndescribe that million.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How did the million behave?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For example, 'they're all really good, and\Nthey're all exactly the same', would be a
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,behavior that you will never see, \Nbut that would be a great behavior.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we need to talk about how things behave,\Ncommunicate, think, evaluate,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,set requirements for, talk to other people,\Nbut these are all common things around that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,To do that, we have to describe the \Ndistribution, the set, the behavior,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but not the one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For example, the behavior that says "the \Nthe common case was x" is a piece of
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,information about the behavior,\Nbut it's a tiny sliver.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Usually the least relevant one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well, there's some less relevant ones, \Nbut not a strongly relevant one,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and one that people often focus on.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,To take a look at what we actually do \Nwith this stuff, almost on a daily basis,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,this is a snapshot from a monitoring system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A small dashboard on a big screen \Nin a monitoring system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Where you're watching the response time of\Na system over time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a two hour window.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,These lines that are 95th percentile, \N90, 75, 50, and 25th percentiles,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you can look at how they behave over time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We're a small audience here, if you look at\Nthis picture, what draws your eye?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What do you want to go investigate here\Nor pay attention to ?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's the big red spike there, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we could look at the red spike,\Ncause it's different,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and say, "Woah, the 95th percentile shot up\Nhere. And look, the 90th percentile
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,shot up at about the same time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The rest of them didn't shoot up, \Nso maybe something happened here
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that affected that much, I should probably\Npay attention to it
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,because it's a monitoring system, and \NI like things to be calm."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You could go investigate the why.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,At this point, I've managed to waste \Nabout 90 seconds of your life,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,looking at a completely meaningless chart,\Nwhich unfortunately you do
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,every day, all the time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This chart is the chart you want to show \Nsomebody if you want to
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,hide the truth from them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you want to pull the wool \Nover their eyes.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the chart of the good stuff.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What's not on this chart?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The 5% worse things that happened during\Nthis two hours.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They're not here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is only the good things that happened\Nduring the things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And to get this spike, that 5% had to be\Nso bad that it even pulled
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the 95th percentile all up.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There is zero information here at all about\Nwhat happened bad during this two hours,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which makes it a bad fit for \Na monitoring system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a really good thing for \Na marketing system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a great way to get the bonus from your boss, even though you didn't do the work.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you want to learn how to do that, \Nwe can do another talk about that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But this is not a good way to look at latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's the opposite of good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Unfortunately, this is one of the most\Ncommon tools used for
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,server monitoring on earth right now.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's where the snapshot is from,\Nand this is what people look at.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I find this chart to be a goldmine\Nof information.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When I first showed it in another talk \Nlike this, I had this really cool experience.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Somebody came up to me and said, "Hey, \Nas I was sitting here, I was texting one
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of our guys, and he was saying,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,'look, we have this issue with \Nour 95th percentile'."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And I got this chart from him!
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So I went and said, "Hey, what does the \Nrest of the spectrum look like?"
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the actual chart they got.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And when they look at the rest of the\Nspectrum, it looked like that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's what was hiding.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I noticed the scales are a little different.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That yellow line is that yellow line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So that's a much more representative number.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Is it? Is that good enough?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's the 99th percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We still have another 1% of really bad \Nstuff that's hiding above the blue line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I wonder how big that is?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't know because he didn't have the data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So a common problem that we have is that\Nwe only plot what's convenient.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We only plot what gives us nice,\Ncolorful graphs.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And often, when we have to choose between\Nthe stuff that hides the rest of the data,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and the stuff that is noise, we choose \Nthe noise to display.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I like to rant about latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is from a blog that I don't write \Nenough in, but the format for it was simple.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I tweet a single tweet about latency, \Nlatency tip of the day,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and then I rant about my own tweet.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,As an example, this chart is a goldmine\Nof information because it has so many
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,different things that are wrong in it, \Nbut we won't get into all of them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can read it online.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Anyway, this is one to take away from \Nwhat we just said.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you are not measuring and showing the\Nmaximum value, what is it you are hiding?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And from whom?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you're job is to hide the truth from\Nothers, this is a good way to do it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if actually are interested in what's\Ngoing on, the number one indicator
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you should never get rid of is the \Nmaximum value.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That is not noise, that is the signal.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The rest of it is noise.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, let's look at this chart for some\Nmore cool stuff.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'm gonna zoom in to a small part\Nof the chart, and ask you what that means.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What is the average of the 95th percentile\Nover 2 hours mean?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What is the math that does that?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What does it do?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's look at that, and I'll give you\Nan example with another percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The 100th percentile. The max, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's take a data set.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Suppose this was the maximum every minute\Nfor 15 minutes.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What does it mean to say that the average \Nmax over the last 15 minutes was 42?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I specifically chose the data to\Nmake that happen.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a meaningless statement.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a completely meaningless statement.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But when you see 95th percentile,\Naverage 184, you think that the 95th
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,percentile for the last two hours\Nwas around 184.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It makes you think that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Putting this on a piece of paper is not \Njust noise and irrelevant,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it's a way to mislead people.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a way to mislead yourself, because \Nyou'll start to believe your own mistruths.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is true for any percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There is no percentile that you could do\Nthis math on.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Another tip, you cannot average percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That math doesn't happen.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But percentiles do matter. You really\Nwant to know about them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And a common misperception is that we want\Nto look at the main part of the spectrum,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,not those outliers and perfection stuff.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Only people that actually bet their house\Nevery day, or the bank on it,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,need to know about the "five-nine's", \Nand all those.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The 99th percentile is a pretty\Ngood number.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Is 99% really rare?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's look at some stuff, because we can\Nask questions like, "If I were looking
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,at a webpage, what is the chance of me\Nhitting the 99th percentile?"
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Of things like this: a search engine node,\Nor a key value store,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or a database, or a CDN, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Because they will report their 99th percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They won't tell you anything above that,\Nbut how many of the
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,webpages that we go to \Nactually experience this? \N
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You want to say 1%, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well, I went to some webpages and I counted\Nhow many "http" requests were generated
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,by one click into that webpage,\Nand here are the numbers.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I ended that about a year ago.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They've probably gone up since then.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now that translates into this math.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the likelihood of one click seeing\Nthe 99th percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And the only page where that is less than\N50% is the clean google search page.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Where only a quarter will see the\N99th percentile.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The 99th percentile is the thing that most\Nof your webpages will see.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Most of them will be there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, we could look at other things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We can pick which things to focus on.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's say I had to pick between the 95th\Npercentile, and the three 9's (99.9%).
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The three 9's is way into perfection mode\Nfor most people, or they think.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Which one of those represents our \Ncommunity better?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our population?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our users?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Our experience?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's run a hypothetical.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Suppose we don't have that many pages, \Nand that many resources like we said before.\N
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We'll be much more conservative.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A user session will only go through five\Nclicks, and each click will only bring up
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,up to 40 things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A lot less, and they're all as clean\Nas the google page.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How many of the users will not experience\Nsomething worse than the 95th percentile?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Because that's what the 95th percentile\Nis good for, the people who see that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Anybody above that, is that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What are the chances of not seeing it?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's an interesting number.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So you're watching a number that is \Nrelevant to 0.003% of your users.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,99.997% of your users are going to \Nsee worse than this number.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why are you looking at it?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why are you spending time\Nthinking about it?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In reverse, we could say how many people\Nare going to see something
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,worse than the three 9's (99.9%)?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's going to be 18%.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In reverse, 82% of the people will see\Nthe three 9's (99.9%) or better.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's a slightly better representation.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Probably not good enough either.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We could look at some more math with them, \Nsame kind of scenario.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What percentile of http response time \Nwill be the thing that 95%
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of people experience in this scenario?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's the 99.97 percentile that 95% \Nof people see.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And if you want to know what 99%\Nof the people see,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that's four and a half 9's (99.995%).
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You want to know that number from Akamai\Nif you want to predict what 1%
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of your users are going to experience.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When you know the 99th percentile, \Nyou kind of know a tiny bit.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So here's another tip.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And this is not an exaggeration,\Nby the way.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The median, which is a much smaller\Npercentile, has that minuscule a chance
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of ever being the number that \Nanybody experiences.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the chance of getting worse\Nthan the median.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Which makes the median an irrelevant \Nnumber to look at.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Unfortunately, it's probably the most \Ncommon one looked at.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When people say "the typical",\Nthey look at the thing that
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,everything will be worse than.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, I'm sorry about that part.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We'll do some other parts.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, why is it that when we look at these\Nmonitoring systems, we don't see
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,data with a lot of 9's?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why do we stop at the\N90, 95, 99th percentile?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why don't we look further?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, some of it is because people think, \N"Well that's perfection, I don't need it."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The other part is that it's hard.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's hard because you can't\Naverage percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We already talked about that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But you also can't derive your \Nfive 9's (99.999%) out of a lot
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of 10 second samples of percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And the reason for that is, "Hey, in 10 \Nseconds, maybe I only had 1,000 things."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I could take all the 10 seconds in the \Nworld, there's no way to say what the
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,hour five 9's (99.999%) were, what the \Nminutes five 9's were
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,if I'm collecting just this data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And unfortunately, the data being collected\Nand reported to the back ends of monitoring
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,is usually summarized at a second,\N5 seconds, 10 seconds, etc.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Basically throwing away all the good data,\Nand leaving you with absolutely no way
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,to compute large 9's for longer\Nperiods of time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, this is where you might want to look\Nat HDR Histogram.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's an open source thing I've created\Na few years ago.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I did it in Java, and know there's a\NC, C-Sharp, Python, Erlang,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and Go ports of this that I didn't create.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And it lets you actually get an entire\Npercentile spectrum.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Some of you here I know are \Nalready using it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you can look at all the percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Any number of 9's that's in the data, if \Nyou just keep it right and report it right,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it's got a log format, you can \Nstore things forever.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well, for a long time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, so it lets you have nice things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Enough for that advertisement.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, latency... Well, I think this is\Nslightly out of order.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Yeah, sorry.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the red/blue pill part, so I warn\Nyou, this is your last chance.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's a problem I call the \Ncoordinated omission problem.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The coordinated omission problem is \Nbasically a conspiracy.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a conspiracy that we're all part of.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't think anybody actually meant\Nto do it, but once I've noticed it,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,everywhere I look, there it is.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, I've been using a specific way of\Nshowing you numbers so far.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Has anybody here noticed how\NI spell percentile?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,(Audience Member): "You put lie at the\Nend of the percent sign."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Yeah, good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So coordinated omission problem is the\N"lie" in %lies.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And this is how it works.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,One common way to do this is\Nto use a load generator.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Pretty much all load generator's\Nhave this problem.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There are two that I know of that don't.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What you do with a load generator,\Nis you test.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You issue requests, or send packets.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you measure how long something took.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And as long as the numbers go right, \Nmeasure them, put them in a bucket,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,study them later, and get your \Npercentiles from it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But what if the thing that you are\Nmeasuring took longer than the time
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it would've taken until you send \Nthe next thing?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You're supposed to send something \Nevery second,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but this one took a second and a half.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Well you've got to wait before\Nyou send the next one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You just avoided measuring something \Nwhen the system was problematic.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You've coordinated with it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You weren't looking at it then.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's common scenario A: You've backed\Noff, and avoided measuring when it was bad.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Another way, is you measure inside your code.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We all do this. We all have to do this,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,where we measure time, do something, \Nthen measure time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The delta between them is how long it took.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We can then put it in a stats bucket,\Nand then do the percentiles in that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Unfortunately, if the system freezes right\Nhere, for any reason,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,an interrupted contact switch,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,a cash buffer flushed to disk,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,a garbage collection,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,a re-indexing of your database,\Nthis is a database.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is Cassandra by the way, \Nmeasuring itself.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In any of the above, then you will\Nhave one bad report
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,while 10,000 things are waiting in line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And when they come in, they will look\Nreally, really good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Even though each one of them has had\Na really bad experience.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,\NIt can even get worse, where maybe the\Nfreeze happened outside the timing,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and you won't even know there was a freeze.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now these are examples of admitting data\Nthat is bad on a very selective basis.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's not random sampling.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's, "I don't like bad data",
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or "I couldn't handle it",
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or "I don't know about it",
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so we'll just talk about the good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What does that do to your data?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Because it often makes people feel like,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Okay, yeah, I understand,\Nbut it's a little bit of noise."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's run some hypotheticals, \Nand I'll show you some real numbers.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Imagine a perfect system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's doing 100 requests a second, \Nat exactly a millisecond each.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But we go and freeze the system, \Nafter 100 seconds of perfect operations
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,for 100 seconds, and then repeat.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, I'm going to describe how the system\Nbehaves in terms that should mean something,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and then we'll measure it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If we actually wanted to describe the\Nsystem,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,on the left we have an average\Nof one millisecond by the finish,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and on the right we have an\Naverage of 50 seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why 50? Because if I randomly came in \Nin that 100 seconds,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'll get anything from 0 to 100\Nwith even distribution.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The overall average over 200 seconds\Nis 25 seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If I just came in here and said, \N"Surprise, how long did this take?"
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,On average, it will be 25.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I can also do the percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,50th percentile will be really good, \Nand then it'll get really bad.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The four 9's is terrible.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a fair honest description of\Nthis system if this is what it did.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you can make the system do that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's what Control Z is good for.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can make any of your systems do that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now lets go measure this system with \Na load generator,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or with a monitoring system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The common ones.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The ones everybody does.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,On the left, we're going to get 10,000 \Nresults of one millisecond each.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Great.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And we're going to get one result of\N100 seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Wow, really big response time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is our data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is OUR data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So now you go do math with it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The average of that is 10.9 milliseconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A little less than 25 seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And here are the percentiles.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Your load generator monitoring system\Nwill tell you that this system is perfect.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You could go to production with it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You like what you see.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Look at that, four 9's.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It is lying to you.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,To your face.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you can catch it doing that with a \NControl Z-Test.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But people tend to not want to do that,\Nbecause then what are they going to do?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you just do that test, and calibrate \Nyour system, and you find it
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,telling you that, about this, the next \Nstep should be to throw all the numbers away.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Don't believe anything else it says.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If it lies this big, what else did it do?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Don't waste your time on numbers\Nfrom uncalibrated systems.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now the problem here was, that if you\Nwant to measure the system,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you have to measure at random rates, \Nor same rates.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you measure 10,000 things in 100 seconds,\Nthere should be another 10,000 things here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you measure them, you would've gotten\Nall the right numbers.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Coordinated omission is the simple act of\Nerasing all that bad stuff.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The conspiracy here is that we all do it\Nwithout meaning to.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't know who put that in our systems,\Nbut it happens to all of us .
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, I often get people saying,\N"Okay, I get it. All the numbers are wrong,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but at least for my job where I tune \Nperformance, and I try to make things
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,faster, I can use the numbers to figure\Nout if I'm going in the right direction."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Is it better, or is it worse? Let me \Ndispel that for you for a second.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Suppose I went and took this system,\Nand improved it dramatically.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Rather than freezing for 100 seconds, \Nit will now answer every question.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It'll take a little longer,\N5 milliseconds instead of one,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but it's much better than freezing, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So let's measure that system that we spent\Nweeks and weeks improving,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and see if it's better.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's the data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If we do the percentiles, it'll tell us \Nthat we just really hurt the four 9's.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We made it go 5 times worse than before.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We should revert this change, go back to\Nthat much better system we had before.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So this is just to make sure that you\Ndon't think that you can have
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,any intuition based on any of these numbers.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They go backwards sometimes.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You don't know which way is good or bad.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you'll never know which way is good\Nor bad with a system that lies like that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The other cool technique is\Nwhat I call "Cheating Twice".
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You have a constant load generator,\Nand it needs to do 100 per second.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When it woke up after 200 seconds, \Nit says,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Woah, were 9,999 behind.\NWe've got to issue those requests."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So it issues those requests.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,At this point, not only did it get rid of\Nall the bad requests,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it replaced every one of them with \Na perfect request.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Coining the four 9's (99.99%), all the way\Nto four and a half 9's (99.995%),
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it's twice as wrong as dropping them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So these are all cool things that\Nhappen to you.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'm not going to spend much time on how\Nto fix those and avoid those.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's a lot of other material that you\Ncan find with me
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,talking about that, in longer talks.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But this is pretty bad.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And like I said...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That should've been up there before.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How did this repeat itself?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Did I create a loop in the\Npresentation somehow?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't know how to do that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's see if I can get through here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Hopefully editing later will take it out.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So we have the cheats twice.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There, okay.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, after we look at coordinated\Nomission that way,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,we should also look at response time, \Nand service time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Coordinated omission, what it really is\Nachieving for you, unfortunately,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,is that it makes something that you think\Nis response time, and only shows you
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the service time component of latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a simple depiction of what service\Ntime and response times are.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This guy is taking a certain amount of\Ntime to take payment
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or make a cup of coffee.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's service time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How long does it take to do the work?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This person has experienced\Nthe response time,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which includes the amount of time they \Nhave to wait before they
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,get to the person that does the work.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And the difference between those\Ntwo is immense.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The coordinated omission problem makes\Nsomething that you think is
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,response time, only measure the \Nservice time,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and basically hide the fact that things \Nstalled, waited in line,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that this guy might've taken a lunch break,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and now we have line around, \Nbuilding three times.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Service time stays the same.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the backwards part...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, let's look at what it\Nactually looks like.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In a load generator that I fixed,\NI measured both
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,response time and service time,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,this happens to be Casandra,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,at a very low load.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you can see that they're very very \Nsimilar, at a very low load.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why? Because there's nobody in line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This thing is really fast.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We're not asking for too much.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Casandra's pretty fast,\Nso they're the same.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if I increase the load, we \Nstart seeing gaps.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If I increase the load a little more,\Nthe gap grows.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If I increase the load a little more,\Nthe gap grows.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now this is not the failure point yet.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If I actually increase it all the way past\Nthe point where the system
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,can't even do the work I want, \Nservice time stays the same,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,response time goes through the roof.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This was when it was 100 and something\Nmilliseconds, now it's 7 and a half seconds.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why 7 and a half seconds?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Cause you're waiting in line that long\Nto go around the block.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The guy just can't serve as many people\Nas are showing up in line, you fall behind.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a virtual world reaction to this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I really like this slide, it's where I came\Nup with the notion of a blue/red pill.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When you actually measure reality, people\Ntend to have this reaction when
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,they compare the two.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And if we actually look at these on the\Ntwo sides of a collapse point of a system,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,this specific system can only do 87,000 \Nthings a second.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,No matter how hard you press it,\Nthat's all it'll do.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The service time on the two sides of \Nthe collapse looks virtually identical,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,which it would.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But if you compare the response time, \Nyou have a very different picture.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And I'm showing this picture so you get \Na feeling for what to look at
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,on whether or not you're measuring\Nthe right one.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Whenever you push, you try and push load\Nbeyond what the system can do,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you are falling behind over time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is a 250 second run,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,where at the end of it\Nyou are waiting for 8 seconds in line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Why? Because for every second \Nthat goes by, there are
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,3,000 more things that are\Nadded to the line.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The interesting thing that happens when \Nyou cross the threshold limit,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or capability of the system, is that\Nresponse time grows over time linearly.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It doesn't happen if you're below.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Only if you're above.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's the point where that happens, and \Nany load generator that doesn't show
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that line when you try pushing harder \Nthan you can, is lying to you.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's a simple sanity check.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If your load generator shows you that, \Nit didn't push.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Or it pushed, but it didn't\Nreport correctly,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,whichever it is.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If we draw that to scale...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Just to make sure, this was not to scale, \Nthis is the scale, I just zoomed in
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so you could see that it was \Nrelatively stable.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So... I don't know what happened to the\Norder of the slides.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's like looping and randoming.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's some conspiracy going on there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, latency doesn't live on it's own.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You do need to look at latency in the \Ncontext of load.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Cause as I showed you, as you're nearly\Nidle, things are nearly perfect.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Even these mistakes won't show up.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But as you start pressing, things start\Ncracking or behaving differently.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And usually when you want to know how much\Nyour system can handle,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the answer is not 87,000 things a second,\Nbecause nobody wants the
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,response time that comes with that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's how many things can I handle so \Nthat I don't get angry phone calls.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So I do get my bonus, and so my\Ncompany stays above ground.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is not sustainable speed.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Running this experiment is really\Ninteresting with software,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,because it actually doesn't hurt, but\Nspending the next 6 months of your time
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,repeating this experiment, trying to\Nchange the shape of the bumper
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,every time you hit the thing\Nis a waste of your time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Your goal when you're trying to figure\Nout sustainable speed throughput,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,whatever it is, is to see how fast you can\Ngo without this happening,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and then to try and engineer\Nto improve that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Meaning, can I make it go faster\Nwithout this happening?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Measuring what happens after you\Nhit the pole is useless for that exercise.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The only thing that matters about hitting\Nthe pole, is that you hit the pole.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When you go and study the behavior\Nof latency, at saturation,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you are doing this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You're looking at this and saying, "That\Nbumper, I don't like the shape of that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's measure it closely and do this 100\Ntimes to see if we can vary it."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's what it means to look at latency \Nat saturation,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and repeat, and repeat, and change,\Nand tune, and see if you can do it again.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you're pressing it to the wall,\Nit should look like this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And it shouldn't be a surprise that it's\Na 7 and a half second response time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,In fact, if it's not, something is\Nterribly wrong with what you're measuring.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You should look at that instead.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So don't do this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Try to minimize the number of times\Nthat you actually run red cars
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,into poles in your testing.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'm not saying don't do it, but use it\Nto establish the end.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And then you need to test all the speeds,\Nand we need to see when you hit the pole.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Maybe you hit the pole at 100 mph,\Nbut maybe you also hit the pole at 70 mph.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Maybe you don't hit it at 20.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,We should find out how fast is safe.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,When you have data, you can compare\Nit like this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is what I would say a recommended \Nway to look at it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Plot requirements, that's the hitting\Nthe pole.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And some things hit the pole, \Nand some things don't.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And you run different scenarios, \Ndifferent loads,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,different configurations,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,different settings,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and see what works, and what doesn't.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Your goal is to stay here, and carry \Nmore while staying there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Usually.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's very useful for figuring out how many\Nmachines I need to carry a certain thing.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you don't know this, you don't know\Nhow many machines to deploy.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, I'm going to run through\Nsome comparisons of
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,latency or response time behaviors\Nbetween different configurations
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,to show you some of the places\Npeople look, and some of the
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,intuitive and non-intuitive\Nthings to do with them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The common thing,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and again, this is that Casandra thing,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,comparing two systems, A and B.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'll let you guess which one is A,\Nand which one is B.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's two systems, and saying\Nwhich is better, what can I do with this?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And we're measuring here at two \Nthroughputs, 85 and 90k.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,As I said in here, 90k is past the\Ncapability of the system.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can sort of see it here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,See, 85 for both of them is here,\Nand 90k is here.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So you could look at this and say,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Look. when the car hits the pole,\Nthe blue system is better."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's half as bad, but that's just\Nthe wrong place to look.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,They both suck.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You do not want to be doing this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The fact that this system is better\Nthan that system
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,doesn't make you want to use it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is the wrong place to measure.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is where latency is irrelevant.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How they behave past this point\Ndoesn't matter.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What we should be doing is saying,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Well, then don't measure here. \NLet's look there."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So if we zoom just at the 85k's on these\Ntwo systems, okay, they're different.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And now...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The red and the blue alternate here,\Nwhatever that is.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And now you look at this,\Nand okay, it's better.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But we're still in the wrong place,\Nbecause we are 1.5% from hitting the pole.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It is not where you will be\Nrunning in production.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's not the interesting place\Nto study latency.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's the place that if you're anywhere\Nclose to that, you should be on the phone
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,getting more servers now, rather than\Ntrying to figure out the latency behaves.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You know it's going to collapse if just\Na little bit of noise happens.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What you should be doing is looking\Nfar away from the need,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,far away from that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For example, let's go to half the\Nthroughput that causes collapse,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and see what things happen there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And here you can see,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,okay, these are two systems, and one \Nof them does better.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can say that this percentile is better,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that percentile, whatever these are.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It is interesting, but what\Ncan we do with this?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How do we tell our boss what this means?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Or how do we translate this into, \Nhow many machines do I need?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now so far, I've been comparing\Nthings at the same throughput,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and looking at latencies.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And that's good for pass/fail kind of\Nthings, or getting quantitative things,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but once you get to this point,\Nyou can start saying,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Wait, what if I do it at\Ndifferent throughputs?"
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,How slow do I need to make this blue thing\Nto make it look closer to the red thing,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,or the other way around.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't want to move this fast to 3-L too,\NI want to move this to be there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For example, slow that one up by 4X, \Nand look,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the two 9's are actually starting\Nto look similar.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you slow it by...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So you can make a statement like this:
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The 99th percentile, if you had a goal\Nlike this,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and now you've passed the goal,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You'd say, "Both of them passed the goal,\Nbut system B does it at 4 times the load."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That drives a choice, right?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can make a harsher goal, and say,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I need the three 9's to be below\N10 milliseconds,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so you'll slow these down even further.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,At this point, you can make this statement:
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you want those, one of them is \N10 times better.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Meaning, not that the system is\N10 times faster,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but I can carry 10 times the load\Nbefore I fail, before I have to pull.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What I'm trying to demonstrate here,\Nis that how much more, or not,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you can can get out of a system depends\Non you're requirements,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and whether or not you need to meet them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Without setting those requirements,\Nlooking at the percentile spectrum
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of response time, not service time,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you'll never know how much you\Nneed or not.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can do a lot of other things, \Nthese are just demonstrations
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,of how to look at data sets.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You make measure at a lot of levels.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can look for systemic behaviors.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,For example, this is one system, but\Nat varying levels.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can sort of see that as you increase\Nthe load, the percentiles move to the left.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,That's a good observation.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's not all systems that'll do it, but \Nfor this system it'll be that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can also see that even though\Nthis didn't totally collapse,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,it's completely out of whack with\Nthe rest,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so that kind of tells you let's not\Nlook there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So throw away the behavior...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You just know not to go to 80.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,No need to study it much.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now that's the remaining set.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You could look at that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You could look at the set from the other\Nsystem and compare them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Maybe put them next to\Neach other like this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Or if you actually can fit enough lines,\Nwith enough colors on a chart,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you can try and do stuff like that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,These are all good ways to actually\Nlook at latencies,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,actually study them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And notice that in all these cases, \NI didn't pick a number.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Oh, let's compare the 99.9 percentile,"\Nbecause I won't get
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,any feeling for the shapes if I did that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You want to look at the entire spectrum.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And that is what and HDR histogram \Nis very good for.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So, you know... You get those.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Wow, we're actually doing okay on time.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Now, this is one of my favorite ways to \Ndepict things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Remember I told you that if you don't plot\Nthe max, what are you hiding?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It turns out that if you plot the max,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,usually it's the number one\Nsignal to look at over time,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,these are just those two systems.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And with a simple visual, you get\Na great intuition.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Same load, one of them's noisy, one's not.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You can look at the response time\Nand service time,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and all of the numbers of different\Nsamples of percentiles,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but if you actually want to show a CEO\Nsomething,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,this is a pretty good thing to show them.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"Look what I did over the weekend."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Before the weekend it looked like that,\Nand I fixed it."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,"I deserve a prize."
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,With that, a simple thing to remember\Nis that this is your load on system A,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,this your load on system B.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Any questions?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is from an anti-drug commercial \Nin the 80's,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I don't know if anybody can remember.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So with that, we're ready\Nfor any questions.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Any questions?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Wow, that bad?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,(Laughing) Dreadful.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, I have one here, and\None back there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Let's start with the back.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,(Audience Member): You said that there are\Nall these tools that you could use
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that give you reasonable numbers, \Nand reasonable answers as far as
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,latency is concerned, so what are those\Ntools that you use?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So the question was, there a couple of\Ntools I mentioned that could give you
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,better information, and I used some\Nto chart here,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,let me see, there are a lot of tools.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I used HDR histogram to plot\Nall these charts
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,with the continuous percentile curves.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I highly recommend you look at using it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Just go to HDR histogram.org\Nand read stuff.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Or google it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's a bunch of people using it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And the basic thing it does, is that it\Ngives you a tool that
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,allows you a practical way to have\Nthis kind of
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,fidelity, dynamic range, and resolution\Nto even look at the shapes.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,The other way to do it is to keep\Nall the data.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You don't have to have histograms\Nif you kept every single result,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but it many places that's not practical,\Nor makes it harder for the system to run.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you can do that, that's even better.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,And then run it through an HDR histogram\Nfor analysis.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So that's as far as viewing things.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,If you have data viewing it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Unfortunately, HDR histogram is not\Ngoing to make the data good.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It's just going to show you\Nthe data you have.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,One of the things I would highly \Nrecommend you try to do,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I'm going backwards, and hopefully\NI'll hit what I wanted.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I highly recommend you look at your \Ndata sets, and remember
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,that in this visual,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,one strong tip I will give you, is that \Nany time you see a vertical
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,rise like that, you have a 99.9% chance\Nof looking at coordinated omission.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This is what coordinated omission\Nlooks like.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There's a couple of other things that\Ncan also look like that.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,I haven't seen them in awhile, but \NI can make them artificially happen,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so it's not conclusive that this is \Ncoordinated omission,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,but suspect it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Suspect it hard.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So if you plot your data with\Ncoordinated omission,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,you will get a view of whether or not\Nyou have this other problem.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,But honestly, there's a much simpler\Nway to do it.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Run your control Z test, and see \Nif you have the problem.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,This will just show you how it works.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,A non-omitted, a sane response time\Ntest, or latency test,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,tends to have these more smooth humps\Nof curves transitioning between numbers.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Any vertical rise tends to\Nindicate omission.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,So that's one thing there.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,As far as the tools actually\Nmeasuring correctly,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,remember I told you what the name of \Nthe talk is,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,so let me rattle off some tools.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Actually, let's do this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You guys measure stuff here,\NI assume.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Could you rattle off some tools \Nthat you use?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,What do you use for load generation\Nand measurement right now?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Volunteers?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,JMeter?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, JMeter.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Gatling.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Anybody else?
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Okay, anybody with Grinder, WRK, \Nsome of the commercial...
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Oh, well yeah.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,Gatling is the only tool I know of\Nright now, that is an actual tool
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,people use, not a demo, that has fixed\Na coordinated omission
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,problem in its measurement.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,There was actually a bug filed against it,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,and the control Z edition in it\Nwas fixed.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,It is actually possible to perfectly \Nfix this.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,You don't have to correct your guess,\Nyou can actually correctly compute
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the exact response time in any load\Ngenerator on earth,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,if you just do it right.
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,All the other tools,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,JMeter, Grinder, WRK,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,the commercial tools that I won't mention,
Dialogue: 0,9:59:59.99,9:59:59.99,Default,,0000,0000,0000,,they all do this wrong, unfortunately.