[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:17.24,0:00:20.03,Default,,0000,0000,0000,,AUSTIN PUTMAN: This is the last session before\Nhappy hour.
Dialogue: 0,0:00:20.03,0:00:24.56,Default,,0000,0000,0000,,I appreciate all of you for hanging around
Dialogue: 0,0:00:24.56,0:00:27.72,Default,,0000,0000,0000,,this long. Maybe you're here because,
Dialogue: 0,0:00:27.72,0:00:28.94,Default,,0000,0000,0000,,you don't know, there's a bar on the
Dialogue: 0,0:00:28.94,0:00:32.25,Default,,0000,0000,0000,,first floor of this hotel. I think that
Dialogue: 0,0:00:32.25,0:00:35.39,Default,,0000,0000,0000,,is where the main track is currently taking\Nplace.
Dialogue: 0,0:00:35.39,0:00:37.66,Default,,0000,0000,0000,,I am Austin Putman. I am the VP of
Dialogue: 0,0:00:37.66,0:00:41.41,Default,,0000,0000,0000,,engineering for Omada Health. At Omada, we\Nsupport people
Dialogue: 0,0:00:41.41,0:00:44.94,Default,,0000,0000,0000,,at risk of chronic disease, like diabetes,\Nmake crucial
Dialogue: 0,0:00:44.94,0:00:49.19,Default,,0000,0000,0000,,behavior changes and have longer, healthier\Nlives. So, it's
Dialogue: 0,0:00:49.19,0:00:49.91,Default,,0000,0000,0000,,pretty awesome.
Dialogue: 0,0:00:49.91,0:00:52.68,Default,,0000,0000,0000,,I'm gonna start with some spoilers, because\NI want
Dialogue: 0,0:00:52.68,0:00:55.34,Default,,0000,0000,0000,,you to have an amazing RailsConf. So if this
Dialogue: 0,0:00:55.34,0:00:57.35,Default,,0000,0000,0000,,is not what you're looking for, don't be shy
Dialogue: 0,0:00:57.35,0:01:00.77,Default,,0000,0000,0000,,about finding that bar track. We're gonna\Nspend some
Dialogue: 0,0:01:00.77,0:01:04.75,Default,,0000,0000,0000,,quality time with Capybara and Cucumber, whose\Nflakiness is
Dialogue: 0,0:01:04.75,0:01:07.49,Default,,0000,0000,0000,,legendary, for very good reasons.
Dialogue: 0,0:01:07.49,0:01:10.67,Default,,0000,0000,0000,,Let me take your temperature. Can I see hands?
Dialogue: 0,0:01:10.67,0:01:12.91,Default,,0000,0000,0000,,How many people have had problems with random\Nfailures
Dialogue: 0,0:01:12.91,0:01:18.23,Default,,0000,0000,0000,,in Cucumber or Capybara? Yeah. Yeah. This\Nis reality,
Dialogue: 0,0:01:18.23,0:01:19.02,Default,,0000,0000,0000,,folks.
Dialogue: 0,0:01:19.02,0:01:21.88,Default,,0000,0000,0000,,We're also gonna cover the ways that Rspec\Ndoes
Dialogue: 0,0:01:21.88,0:01:24.63,Default,,0000,0000,0000,,and does not help us track down test pollution.
Dialogue: 0,0:01:24.63,0:01:26.95,Default,,0000,0000,0000,,How many folks out there have had a random
Dialogue: 0,0:01:26.95,0:01:28.71,Default,,0000,0000,0000,,failure problem in the Rspec suite, like in\Nyour
Dialogue: 0,0:01:28.71,0:01:32.07,Default,,0000,0000,0000,,models or your controller tests? OK, still\Na lot
Dialogue: 0,0:01:32.07,0:01:34.95,Default,,0000,0000,0000,,of people, right. It happens. But we don't\Ntalk
Dialogue: 0,0:01:34.95,0:01:36.78,Default,,0000,0000,0000,,about it.
Dialogue: 0,0:01:36.78,0:01:39.52,Default,,0000,0000,0000,,So in between, we're gonna review some problems\Nthat
Dialogue: 0,0:01:39.52,0:01:42.52,Default,,0000,0000,0000,,can dog any test suite. This is like, random
Dialogue: 0,0:01:42.52,0:01:46.57,Default,,0000,0000,0000,,data, times zone heck, external dependencies.\NAll this leads
Dialogue: 0,0:01:46.57,0:01:48.49,Default,,0000,0000,0000,,to pain. There was a great talk before about
Dialogue: 0,0:01:48.49,0:01:49.72,Default,,0000,0000,0000,,external dependencies.
Dialogue: 0,0:01:49.72,0:01:52.99,Default,,0000,0000,0000,,Just, here's just a random one. How many people
Dialogue: 0,0:01:52.99,0:01:54.22,Default,,0000,0000,0000,,here have had a test fail due to a
Dialogue: 0,0:01:54.22,0:01:59.69,Default,,0000,0000,0000,,daylight savings time issue? Yeah. Ben Franklin,\Nyou are
Dialogue: 0,0:01:59.69,0:02:02.36,Default,,0000,0000,0000,,a menace.
Dialogue: 0,0:02:02.36,0:02:05.78,Default,,0000,0000,0000,,Let's talk about eliminating inconsistent\Nfailures in your tests,
Dialogue: 0,0:02:05.78,0:02:09.82,Default,,0000,0000,0000,,and on our team, we call that fighting randos.
Dialogue: 0,0:02:09.82,0:02:11.92,Default,,0000,0000,0000,,And I'm here to talk about this, because I
Dialogue: 0,0:02:11.92,0:02:15.67,Default,,0000,0000,0000,,was stupid and short-sighted, and random failures\Ncaused us
Dialogue: 0,0:02:15.67,0:02:18.25,Default,,0000,0000,0000,,a lot of pain. I chose to try to
Dialogue: 0,0:02:18.25,0:02:21.91,Default,,0000,0000,0000,,hit deadlines instead of focusing on build\Nquality, and
Dialogue: 0,0:02:21.91,0:02:24.06,Default,,0000,0000,0000,,our team paid a terrible price.
Dialogue: 0,0:02:24.06,0:02:26.25,Default,,0000,0000,0000,,Anybody out there paying that price? Anybody\Nout there
Dialogue: 0,0:02:26.25,0:02:30.03,Default,,0000,0000,0000,,feel me on this? Yeah. It's, it sucks.
Dialogue: 0,0:02:30.03,0:02:31.95,Default,,0000,0000,0000,,So let's do some science. Some problems seem\Nto
Dialogue: 0,0:02:31.95,0:02:34.75,Default,,0000,0000,0000,,have more random failure problems than others.\NI want
Dialogue: 0,0:02:34.75,0:02:37.12,Default,,0000,0000,0000,,to gather some data. So first, if you write
Dialogue: 0,0:02:37.12,0:02:40.75,Default,,0000,0000,0000,,tests on a regular basis, raise your hand.\NRight?
Dialogue: 0,0:02:40.75,0:02:44.53,Default,,0000,0000,0000,,Wow. I love RailsConf. Keep your hand up if
Dialogue: 0,0:02:44.53,0:02:48.36,Default,,0000,0000,0000,,you believe you have experienced a random\Ntest failure.
Dialogue: 0,0:02:48.36,0:02:49.82,Default,,0000,0000,0000,,The whole room.
Dialogue: 0,0:02:49.82,0:02:52.28,Default,,0000,0000,0000,,Now, if you think you're likely to have one
Dialogue: 0,0:02:52.28,0:02:55.56,Default,,0000,0000,0000,,in the next, like, four weeks. Who's out there?
Dialogue: 0,0:02:55.56,0:02:57.88,Default,,0000,0000,0000,,It's still happening, right. You're in the\Nmiddle of
Dialogue: 0,0:02:57.88,0:03:00.79,Default,,0000,0000,0000,,it. OK, so this is not hypothetical for this
Dialogue: 0,0:03:00.79,0:03:03.32,Default,,0000,0000,0000,,audience. This is a widespread problem. But\NI don't
Dialogue: 0,0:03:03.32,0:03:05.12,Default,,0000,0000,0000,,see a lot of people talking about it.
Dialogue: 0,0:03:05.12,0:03:08.16,Default,,0000,0000,0000,,And the truth is, while being a great tool,
Dialogue: 0,0:03:08.16,0:03:11.56,Default,,0000,0000,0000,,a comprehensive integration suite, is like\Na breeding ground
Dialogue: 0,0:03:11.56,0:03:13.81,Default,,0000,0000,0000,,for baffling Heisenbugs.
Dialogue: 0,0:03:13.81,0:03:18.40,Default,,0000,0000,0000,,So, to understand how test failures become\Na chronic
Dialogue: 0,0:03:18.40,0:03:20.56,Default,,0000,0000,0000,,productivity blocker, I want to talk a little\Nbit
Dialogue: 0,0:03:20.56,0:03:25.01,Default,,0000,0000,0000,,about testing culture, right. Why is this\Neven bad?
Dialogue: 0,0:03:25.01,0:03:28.39,Default,,0000,0000,0000,,So, we have an automated CI machine that runs
Dialogue: 0,0:03:28.39,0:03:30.62,Default,,0000,0000,0000,,our full test suite every time a commit is
Dialogue: 0,0:03:30.62,0:03:33.65,Default,,0000,0000,0000,,pushed. And every time the bill passes, we\Npush
Dialogue: 0,0:03:33.65,0:03:36.62,Default,,0000,0000,0000,,the new code to a staging environment for\Nacceptance.
Dialogue: 0,0:03:36.62,0:03:38.51,Default,,0000,0000,0000,,Right, that's our process. How many people\Nout there
Dialogue: 0,0:03:38.51,0:03:40.97,Default,,0000,0000,0000,,have a setup that's kind of like that? OK.
Dialogue: 0,0:03:40.97,0:03:43.41,Default,,0000,0000,0000,,Awesome. So a lot of people know what I'm
Dialogue: 0,0:03:43.41,0:03:44.22,Default,,0000,0000,0000,,talking about.
Dialogue: 0,0:03:44.22,0:03:47.41,Default,,0000,0000,0000,,So, in the fall of 2012, we started seeing
Dialogue: 0,0:03:47.41,0:03:52.19,Default,,0000,0000,0000,,occasional, unreproducible failures of the\Ntest suite in Jenkins.
Dialogue: 0,0:03:52.19,0:03:54.46,Default,,0000,0000,0000,,And we were pushing to get features out the
Dialogue: 0,0:03:54.46,0:03:58.63,Default,,0000,0000,0000,,door for January first. And we found that\Nwe
Dialogue: 0,0:03:58.63,0:04:01.47,Default,,0000,0000,0000,,could just rerun the build and the failure\Nwould
Dialogue: 0,0:04:01.47,0:04:03.26,Default,,0000,0000,0000,,go away.
Dialogue: 0,0:04:03.26,0:04:05.63,Default,,0000,0000,0000,,And we got pretty good at spotting the two
Dialogue: 0,0:04:05.63,0:04:09.40,Default,,0000,0000,0000,,or three tests where this happened. So, we\Nwould
Dialogue: 0,0:04:09.40,0:04:11.50,Default,,0000,0000,0000,,check the output of a failed build, and if
Dialogue: 0,0:04:11.50,0:04:14.65,Default,,0000,0000,0000,,it was one of the suspect tests, we would
Dialogue: 0,0:04:14.65,0:04:17.72,Default,,0000,0000,0000,,just run the build again. Not a problem. Staging
Dialogue: 0,0:04:17.72,0:04:20.79,Default,,0000,0000,0000,,would deploy. We would continue our march\Ntowards the
Dialogue: 0,0:04:20.79,0:04:22.79,Default,,0000,0000,0000,,launch.
Dialogue: 0,0:04:22.79,0:04:24.28,Default,,0000,0000,0000,,But by the time spring rolled around, there\Nwere
Dialogue: 0,0:04:24.28,0:04:28.48,Default,,0000,0000,0000,,like seven or eight places causing problems\Nregularly. And
Dialogue: 0,0:04:28.48,0:04:29.95,Default,,0000,0000,0000,,we would try to fix them, you know, we
Dialogue: 0,0:04:29.95,0:04:33.36,Default,,0000,0000,0000,,wouldn't ignore them. But the failures were\Nunreliable. So
Dialogue: 0,0:04:33.36,0:04:35.06,Default,,0000,0000,0000,,it was hard to say if we had actually
Dialogue: 0,0:04:35.06,0:04:36.84,Default,,0000,0000,0000,,fixed anything.
Dialogue: 0,0:04:36.84,0:04:39.02,Default,,0000,0000,0000,,And eventually we just added a gem called\NCucumber-Rerun.
Dialogue: 0,0:04:39.02,0:04:46.02,Default,,0000,0000,0000,,Yeah. And this just reruns the failed specs\Nif
Dialogue: 0,0:04:46.39,0:04:48.54,Default,,0000,0000,0000,,there's a problem. And when it passed the\Nsecond
Dialogue: 0,0:04:48.54,0:04:54.23,Default,,0000,0000,0000,,time, it's good. You're fine. No big deal.
Dialogue: 0,0:04:54.23,0:04:55.75,Default,,0000,0000,0000,,And then some people on our team got ambitious,
Dialogue: 0,0:04:55.75,0:04:58.47,Default,,0000,0000,0000,,and they said, we could make it faster. We
Dialogue: 0,0:04:58.47,0:05:01.63,Default,,0000,0000,0000,,could make CI faster with the parallel_test\Ngem, which
Dialogue: 0,0:05:01.63,0:05:07.17,Default,,0000,0000,0000,,is awesome. But Cucumber-Rerun and parallel_test\Nare not compatible.
Dialogue: 0,0:05:07.17,0:05:08.100,Default,,0000,0000,0000,,And so we had a test suite that ran
Dialogue: 0,0:05:08.100,0:05:13.02,Default,,0000,0000,0000,,three times faster, but failed twice as often.
Dialogue: 0,0:05:13.02,0:05:15.80,Default,,0000,0000,0000,,And as we came into the fall, we had
Dialogue: 0,0:05:15.80,0:05:20.63,Default,,0000,0000,0000,,our first Bad Jenkins week. On a fateful Tuesday,
Dialogue: 0,0:05:20.63,0:05:24.21,Default,,0000,0000,0000,,4 PM, the build just stopped passing. And\Nthere
Dialogue: 0,0:05:24.21,0:05:27.39,Default,,0000,0000,0000,,were anywhere from like thirty to seventy\Nfailures. And
Dialogue: 0,0:05:27.39,0:05:30.23,Default,,0000,0000,0000,,some of them were our usual suspects, and\Ndozens
Dialogue: 0,0:05:30.23,0:05:34.02,Default,,0000,0000,0000,,of them were, like, previously good tests.\NTests we
Dialogue: 0,0:05:34.02,0:05:34.51,Default,,0000,0000,0000,,trusted.
Dialogue: 0,0:05:34.51,0:05:37.37,Default,,0000,0000,0000,,And, so none of them failed in isolation,\Nright.
Dialogue: 0,0:05:37.37,0:05:39.64,Default,,0000,0000,0000,,And after like two days of working on this,
Dialogue: 0,0:05:39.64,0:05:42.71,Default,,0000,0000,0000,,we eventually got a clean Rspec build, but\NCucumber
Dialogue: 0,0:05:42.71,0:05:45.44,Default,,0000,0000,0000,,would still fail. And the failures could not\Nbe
Dialogue: 0,0:05:45.44,0:05:47.67,Default,,0000,0000,0000,,reproduced on a dev machine, or even on the
Dialogue: 0,0:05:47.67,0:05:50.80,Default,,0000,0000,0000,,same CI machine, outside of the, the whole\Nbuild
Dialogue: 0,0:05:50.80,0:05:51.93,Default,,0000,0000,0000,,running.
Dialogue: 0,0:05:51.93,0:05:55.31,Default,,0000,0000,0000,,So, over the weekend, somebody pushes a commit\Nand
Dialogue: 0,0:05:55.31,0:05:58.73,Default,,0000,0000,0000,,we get a green build. And there's nothing\Nspecial
Dialogue: 0,0:05:58.73,0:06:00.99,Default,,0000,0000,0000,,about this commit, right. Like, it was like,\Na
Dialogue: 0,0:06:00.99,0:06:04.55,Default,,0000,0000,0000,,comment change. And we had tried a million\Nthings,
Dialogue: 0,0:06:04.55,0:06:07.90,Default,,0000,0000,0000,,and no single change obviously lead to the\Npassing
Dialogue: 0,0:06:07.90,0:06:09.35,Default,,0000,0000,0000,,build.
Dialogue: 0,0:06:09.35,0:06:10.93,Default,,0000,0000,0000,,And the next week, we were back to like,
Dialogue: 0,0:06:10.93,0:06:14.89,Default,,0000,0000,0000,,you know, fifteen percent failure rate. Like,\Npretty good.
Dialogue: 0,0:06:14.89,0:06:18.58,Default,,0000,0000,0000,,So, we could push stories to staging again,\Nand
Dialogue: 0,0:06:18.58,0:06:23.29,Default,,0000,0000,0000,,we're still under the deadline pressure, right.\NSo, so
Dialogue: 0,0:06:23.29,0:06:28.40,Default,,0000,0000,0000,,we shrugged. And we moved on, right. And maybe
Dialogue: 0,0:06:28.40,0:06:32.40,Default,,0000,0000,0000,,somebody wants to guess, what happened next?\NRight?
Dialogue: 0,0:06:32.40,0:06:35.25,Default,,0000,0000,0000,,Yeah. It happened again, right. A whole week\Nof
Dialogue: 0,0:06:35.25,0:06:38.09,Default,,0000,0000,0000,,just no tests pass. The build never passes.\NSo
Dialogue: 0,0:06:38.09,0:06:40.40,Default,,0000,0000,0000,,we turn of parallel_tests, right. Because\Nwe can't even
Dialogue: 0,0:06:40.40,0:06:42.42,Default,,0000,0000,0000,,get like a coherent log of which tests are
Dialogue: 0,0:06:42.42,0:06:45.99,Default,,0000,0000,0000,,causing errors, and then we started commenting\Nout the
Dialogue: 0,0:06:45.99,0:06:49.72,Default,,0000,0000,0000,,really problematic tests, and there were still\Nthese like
Dialogue: 0,0:06:49.72,0:06:55.34,Default,,0000,0000,0000,,seemingly innocuous specs that failed regularly\Nbut not consistently.
Dialogue: 0,0:06:55.34,0:06:57.47,Default,,0000,0000,0000,,So these are tests that have enough business\Nvalue
Dialogue: 0,0:06:57.47,0:07:00.73,Default,,0000,0000,0000,,that we are very reluctant to just, like,\Ndelete
Dialogue: 0,0:07:00.73,0:07:01.37,Default,,0000,0000,0000,,them.
Dialogue: 0,0:07:01.37,0:07:07.73,Default,,0000,0000,0000,,And so we reinstated Cucumber-Rerun, and its\Nbuddy Rspec-Rerun.
Dialogue: 0,0:07:07.73,0:07:12.53,Default,,0000,0000,0000,,And this mostly worked, right. So we were\Nmaking
Dialogue: 0,0:07:12.53,0:07:15.31,Default,,0000,0000,0000,,progress. But the build issues continued to\Nshow up
Dialogue: 0,0:07:15.31,0:07:18.52,Default,,0000,0000,0000,,in the negative column in our retrospectives.\NAnd that
Dialogue: 0,0:07:18.52,0:07:21.40,Default,,0000,0000,0000,,was because there were several problems with\Nthis situation,
Dialogue: 0,0:07:21.40,0:07:25.61,Default,,0000,0000,0000,,right. Like, reduced trust. When build failures\Nhappen four
Dialogue: 0,0:07:25.61,0:07:27.52,Default,,0000,0000,0000,,or five times a day, those aren't a red
Dialogue: 0,0:07:27.52,0:07:30.42,Default,,0000,0000,0000,,flag. Those are just how things go. And everyone
Dialogue: 0,0:07:30.42,0:07:33.44,Default,,0000,0000,0000,,on the team knows that the most likely explanation
Dialogue: 0,0:07:33.44,0:07:35.56,Default,,0000,0000,0000,,is a random failure.
Dialogue: 0,0:07:35.56,0:07:38.14,Default,,0000,0000,0000,,And the default response to a build failure\Nbecomes,
Dialogue: 0,0:07:38.14,0:07:41.69,Default,,0000,0000,0000,,run it again. So, just run it again, right.
Dialogue: 0,0:07:41.69,0:07:47.78,Default,,0000,0000,0000,,The build failed. Whatever. So then, occasionally,\Nwe break
Dialogue: 0,0:07:47.78,0:07:50.42,Default,,0000,0000,0000,,things for real. But we stopped noticing because\Nwe
Dialogue: 0,0:07:50.42,0:07:54.34,Default,,0000,0000,0000,,started expecting CI to be broken. Sometimes\Nother pairs
Dialogue: 0,0:07:54.34,0:07:56.45,Default,,0000,0000,0000,,would pull the code and they would see the
Dialogue: 0,0:07:56.45,0:07:59.75,Default,,0000,0000,0000,,legitimate failures. Sometimes we thought\Nwe were having a
Dialogue: 0,0:07:59.75,0:08:02.32,Default,,0000,0000,0000,,Bad Jenkins week, and on the third or fourth
Dialogue: 0,0:08:02.32,0:08:06.28,Default,,0000,0000,0000,,day we realized we were having actual failures.
Dialogue: 0,0:08:06.28,0:08:08.32,Default,,0000,0000,0000,,This is pretty bad, right.
Dialogue: 0,0:08:08.32,0:08:12.04,Default,,0000,0000,0000,,So our system depends on green builds to mark
Dialogue: 0,0:08:12.04,0:08:14.19,Default,,0000,0000,0000,,the code that can be deployed to staging and
Dialogue: 0,0:08:14.19,0:08:17.24,Default,,0000,0000,0000,,production, and without green builds, stories\Ncan't get delivered
Dialogue: 0,0:08:17.24,0:08:22.02,Default,,0000,0000,0000,,and reviewed. So we stopped getting timely\Nfeedback. Meanwhile,
Dialogue: 0,0:08:22.02,0:08:24.52,Default,,0000,0000,0000,,the reviewer gets, like, a week's worth of\Nstories.
Dialogue: 0,0:08:24.52,0:08:26.19,Default,,0000,0000,0000,,All at once. Big clump.
Dialogue: 0,0:08:26.19,0:08:27.75,Default,,0000,0000,0000,,And that means they have less time to pay
Dialogue: 0,0:08:27.75,0:08:31.72,Default,,0000,0000,0000,,attention to detail on each delivered feature.\NAnd that
Dialogue: 0,0:08:31.72,0:08:34.81,Default,,0000,0000,0000,,means that the product is a little bit crappier
Dialogue: 0,0:08:34.81,0:08:39.81,Default,,0000,0000,0000,,every week. So, maybe you need a bug fix.
Dialogue: 0,0:08:39.81,0:08:42.51,Default,,0000,0000,0000,,Fast. Forget about that. You've got, like,\Na twenty
Dialogue: 0,0:08:42.51,0:08:44.41,Default,,0000,0000,0000,,percent chance your bug fix build is gonna\Nfail
Dialogue: 0,0:08:44.41,0:08:46.07,Default,,0000,0000,0000,,for no reason.
Dialogue: 0,0:08:46.07,0:08:47.87,Default,,0000,0000,0000,,Maybe the code has to ship, because the app
Dialogue: 0,0:08:47.87,0:08:51.53,Default,,0000,0000,0000,,is mega busted. In this case, we would rerun
Dialogue: 0,0:08:51.53,0:08:53.92,Default,,0000,0000,0000,,the failed tests on our local machine, and\Nthen
Dialogue: 0,0:08:53.92,0:08:58.66,Default,,0000,0000,0000,,cross our fingers and deploy. So, in effect,\Nour
Dialogue: 0,0:08:58.66,0:09:03.13,Default,,0000,0000,0000,,policy was, if the code works on my machine,
Dialogue: 0,0:09:03.13,0:09:07.19,Default,,0000,0000,0000,,it can be deployed to production.
Dialogue: 0,0:09:07.19,0:09:11.34,Default,,0000,0000,0000,,So. At the most extreme, people lose faith\Nin
Dialogue: 0,0:09:11.34,0:09:13.69,Default,,0000,0000,0000,,the build, and eventually they just forget\Nabout testing.
Dialogue: 0,0:09:13.69,0:09:15.69,Default,,0000,0000,0000,,And this didn't happen to us, but I had
Dialogue: 0,0:09:15.69,0:09:18.16,Default,,0000,0000,0000,,to explain to management that key features\Ncouldn't be
Dialogue: 0,0:09:18.16,0:09:21.84,Default,,0000,0000,0000,,shipped because of problems with the test\Nserver. And
Dialogue: 0,0:09:21.84,0:09:24.31,Default,,0000,0000,0000,,they wanted to know a lot more about the
Dialogue: 0,0:09:24.31,0:09:26.68,Default,,0000,0000,0000,,test server. And it was totally clear that\Nwhile
Dialogue: 0,0:09:26.68,0:09:30.02,Default,,0000,0000,0000,,a working test server has their full support,\Nan
Dialogue: 0,0:09:30.02,0:09:33.48,Default,,0000,0000,0000,,unreliable test server is a business liability\Nand needs
Dialogue: 0,0:09:33.48,0:09:34.80,Default,,0000,0000,0000,,to be resolved.
Dialogue: 0,0:09:34.80,0:09:37.66,Default,,0000,0000,0000,,So, the test server is supposed to solve problems
Dialogue: 0,0:09:37.66,0:09:39.56,Default,,0000,0000,0000,,for us, and that is the only story that
Dialogue: 0,0:09:39.56,0:09:43.36,Default,,0000,0000,0000,,I like to tell about it. So, we began
Dialogue: 0,0:09:43.36,0:09:48.31,Default,,0000,0000,0000,,to fight back. And we personified the random\Nfailures.
Dialogue: 0,0:09:48.31,0:09:53.09,Default,,0000,0000,0000,,They became randos. A rando attack. A rando\Nstorm.
Dialogue: 0,0:09:53.09,0:10:00.09,Default,,0000,0000,0000,,And most memorably, Rando Backstabbian. Intergalactic\Nrandomness villain.
Dialogue: 0,0:10:01.65,0:10:03.29,Default,,0000,0000,0000,,We had a pair working on the test suite
Dialogue: 0,0:10:03.29,0:10:05.49,Default,,0000,0000,0000,,full time for about three months trying to\Nresolve
Dialogue: 0,0:10:05.49,0:10:08.29,Default,,0000,0000,0000,,these issues. We tried about a thousand things,\Nand
Dialogue: 0,0:10:08.29,0:10:09.64,Default,,0000,0000,0000,,some of them worked. And I'm gonna pass along
Dialogue: 0,0:10:09.64,0:10:12.44,Default,,0000,0000,0000,,the answers we found, and my hypothesis that\Nwe
Dialogue: 0,0:10:12.44,0:10:16.72,Default,,0000,0000,0000,,didn't disprove. Honestly, I'm hoping that\Nyou came to
Dialogue: 0,0:10:16.72,0:10:18.94,Default,,0000,0000,0000,,this talk because you've had similar problems\Nand you
Dialogue: 0,0:10:18.94,0:10:22.68,Default,,0000,0000,0000,,found better solutions. So, this is just what\Nwe
Dialogue: 0,0:10:22.68,0:10:23.60,Default,,0000,0000,0000,,found.
Dialogue: 0,0:10:23.60,0:10:26.38,Default,,0000,0000,0000,,I, I have a very important tool for this
Dialogue: 0,0:10:26.38,0:10:29.42,Default,,0000,0000,0000,,section of the talk. It's the finger of blame.
Dialogue: 0,0:10:29.42,0:10:31.05,Default,,0000,0000,0000,,We use this a lot when we were like,
Dialogue: 0,0:10:31.05,0:10:34.92,Default,,0000,0000,0000,,hey, could the problem be Cucumber? And then\Nwe
Dialogue: 0,0:10:34.92,0:10:37.55,Default,,0000,0000,0000,,would go after that. So here comes finger\Nof
Dialogue: 0,0:10:37.55,0:10:39.21,Default,,0000,0000,0000,,blame.
Dialogue: 0,0:10:39.21,0:10:46.16,Default,,0000,0000,0000,,Cucumber! Capybara. Poltergeist. Definitely\Npart of the problem. I've
Dialogue: 0,0:10:46.16,0:10:48.96,Default,,0000,0000,0000,,talked to enough other teams that use these\Ntools
Dialogue: 0,0:10:48.96,0:10:51.74,Default,,0000,0000,0000,,extensively, and the evidence from our audience,\Njust to
Dialogue: 0,0:10:51.74,0:10:53.63,Default,,0000,0000,0000,,know that the results are just not as deterministic
Dialogue: 0,0:10:53.63,0:10:57.25,Default,,0000,0000,0000,,as we want. When you're using multiple threads\Nand
Dialogue: 0,0:10:57.25,0:11:00.94,Default,,0000,0000,0000,,you're asserting against a browser environment,\Nyou're gonna have
Dialogue: 0,0:11:00.94,0:11:02.47,Default,,0000,0000,0000,,so issues, right.
Dialogue: 0,0:11:02.47,0:11:06.67,Default,,0000,0000,0000,,And one of those is browser environment, right.\NBrowser
Dialogue: 0,0:11:06.67,0:11:09.71,Default,,0000,0000,0000,,environment is a euphemism for, like, a complicated\Npiece
Dialogue: 0,0:11:09.71,0:11:12.97,Default,,0000,0000,0000,,of software that itself is a playground for\Nnetwork
Dialogue: 0,0:11:12.97,0:11:17.43,Default,,0000,0000,0000,,latency issues and rendering hiccups and a\Ncallback soup.
Dialogue: 0,0:11:17.43,0:11:18.81,Default,,0000,0000,0000,,So your tests have to be written in a
Dialogue: 0,0:11:18.81,0:11:21.49,Default,,0000,0000,0000,,very specific way to prevent all the threads\Nand
Dialogue: 0,0:11:21.49,0:11:23.49,Default,,0000,0000,0000,,all the different layers of code from getting\Nconfused
Dialogue: 0,0:11:23.49,0:11:24.77,Default,,0000,0000,0000,,and smashing into each other.
Dialogue: 0,0:11:24.77,0:11:26.91,Default,,0000,0000,0000,,You know, some of you maybe are lucky, and
Dialogue: 0,0:11:26.91,0:11:29.01,Default,,0000,0000,0000,,you use the right style most of the time
Dialogue: 0,0:11:29.01,0:11:31.88,Default,,0000,0000,0000,,by default. Maybe you don't see that many\Nproblems.
Dialogue: 0,0:11:31.88,0:11:35.15,Default,,0000,0000,0000,,A few things you gotta never assume.
Dialogue: 0,0:11:35.15,0:11:38.25,Default,,0000,0000,0000,,Never assume the page has loaded. Never assume\Nthe
Dialogue: 0,0:11:38.25,0:11:42.17,Default,,0000,0000,0000,,markup you are asserting against exists. Never\Nassume your
Dialogue: 0,0:11:42.17,0:11:46.39,Default,,0000,0000,0000,,AJAX request actually finished, and never\Nassume the speed
Dialogue: 0,0:11:46.39,0:11:48.71,Default,,0000,0000,0000,,at which things happen, because until you\Nbolt it
Dialogue: 0,0:11:48.71,0:11:52.41,Default,,0000,0000,0000,,down, you just don't know.
Dialogue: 0,0:11:52.41,0:11:57.07,Default,,0000,0000,0000,,So, always make sure the markup exists before\Nyou
Dialogue: 0,0:11:57.07,0:11:59.82,Default,,0000,0000,0000,,assert against it. New Capybara is supposed\Nto be
Dialogue: 0,0:11:59.82,0:12:03.38,Default,,0000,0000,0000,,better at this, and it, it's improved. But\NI
Dialogue: 0,0:12:03.38,0:12:06.06,Default,,0000,0000,0000,,do not trust them. I am super paranoid about
Dialogue: 0,0:12:06.06,0:12:08.51,Default,,0000,0000,0000,,this stuff. This is a good example of a
Dialogue: 0,0:12:08.51,0:12:11.71,Default,,0000,0000,0000,,lurking rando, due to a race condition, in\Nyour
Dialogue: 0,0:12:11.71,0:12:12.59,Default,,0000,0000,0000,,browser.
Dialogue: 0,0:12:12.59,0:12:14.47,Default,,0000,0000,0000,,Capybara is supposed to wait for the page\Nto
Dialogue: 0,0:12:14.47,0:12:17.47,Default,,0000,0000,0000,,load before it continues after the visit method,\Nbut
Dialogue: 0,0:12:17.47,0:12:19.64,Default,,0000,0000,0000,,I find it has sort of medium success with
Dialogue: 0,0:12:19.64,0:12:24.08,Default,,0000,0000,0000,,doing that. Bolt it down, right. We used to
Dialogue: 0,0:12:24.08,0:12:26.07,Default,,0000,0000,0000,,have something called the wait_until block,\Nand that would
Dialogue: 0,0:12:26.07,0:12:28.41,Default,,0000,0000,0000,,stop execution until a condition was met.\NAnd that
Dialogue: 0,0:12:28.41,0:12:31.31,Default,,0000,0000,0000,,was great. Cause it replaced, like, sleep\Nstatements, which
Dialogue: 0,0:12:31.31,0:12:33.12,Default,,0000,0000,0000,,is what we used before that.
Dialogue: 0,0:12:33.12,0:12:37.43,Default,,0000,0000,0000,,Modern Capybara, no more wait_until block.\NIt's inside the
Dialogue: 0,0:12:37.43,0:12:42.11,Default,,0000,0000,0000,,have_cc and have_content matcher. So, always\Nassert that something
Dialogue: 0,0:12:42.11,0:12:46.37,Default,,0000,0000,0000,,exists before you try to do anything with\Nit.
Dialogue: 0,0:12:46.37,0:12:49.05,Default,,0000,0000,0000,,And sometimes it might take a long time. The
Dialogue: 0,0:12:49.05,0:12:51.52,Default,,0000,0000,0000,,default timeout for that, for those Capybara\Nassertions, is
Dialogue: 0,0:12:51.52,0:12:55.72,Default,,0000,0000,0000,,like five seconds. And sometimes, you need\Ntwenty seconds.
Dialogue: 0,0:12:55.72,0:12:57.52,Default,,0000,0000,0000,,Usually, for us, that's because we're doing\Nlike a
Dialogue: 0,0:12:57.52,0:13:01.16,Default,,0000,0000,0000,,file upload or another lengthy operation.\NBut, again, never
Dialogue: 0,0:13:01.16,0:13:03.83,Default,,0000,0000,0000,,assume that things are gonna take a normal\Namount
Dialogue: 0,0:13:03.83,0:13:07.06,Default,,0000,0000,0000,,of time.
Dialogue: 0,0:13:07.06,0:13:09.94,Default,,0000,0000,0000,,Race conditions. I would be out of line to
Dialogue: 0,0:13:09.94,0:13:13.30,Default,,0000,0000,0000,,give this talk without talking explicitly\Nabout race conditions,
Dialogue: 0,0:13:13.30,0:13:16.45,Default,,0000,0000,0000,,right. Whenever you get, create a situation\Nwhere a
Dialogue: 0,0:13:16.45,0:13:20.47,Default,,0000,0000,0000,,sequence of key events doesn't happen in a\Npredetermined
Dialogue: 0,0:13:20.47,0:13:24.56,Default,,0000,0000,0000,,order, you've got a potential race condition.
Dialogue: 0,0:13:24.56,0:13:27.57,Default,,0000,0000,0000,,So the winner of the race is random. And
Dialogue: 0,0:13:27.57,0:13:31.71,Default,,0000,0000,0000,,that can create random outcomes in your test\Nsuite.
Dialogue: 0,0:13:31.71,0:13:36.45,Default,,0000,0000,0000,,So what's an example of one of those? AJAX.
Dialogue: 0,0:13:36.45,0:13:42.57,Default,,0000,0000,0000,,Right? In AJAX, your JavaScript running in\NFirefox may
Dialogue: 0,0:13:42.57,0:13:46.11,Default,,0000,0000,0000,,or may not complete its AJAX call and render
Dialogue: 0,0:13:46.11,0:13:49.63,Default,,0000,0000,0000,,the response before the test thread makes\Nits assertions.
Dialogue: 0,0:13:49.63,0:13:54.06,Default,,0000,0000,0000,,Now, Capybara tries to fix this by retrying\Nto
Dialogue: 0,0:13:54.06,0:13:57.66,Default,,0000,0000,0000,,assertions. But that doesn't always work.\NSo, say you're
Dialogue: 0,0:13:57.66,0:14:00.74,Default,,0000,0000,0000,,clicking a button to submit a form, and then
Dialogue: 0,0:14:00.74,0:14:05.34,Default,,0000,0000,0000,,you're going to another page or refreshing\Nthe page.
Dialogue: 0,0:14:05.34,0:14:07.92,Default,,0000,0000,0000,,This might cut off that post request, whether\Nit's
Dialogue: 0,0:14:07.92,0:14:10.34,Default,,0000,0000,0000,,from a regular form or an AJAX form, but
Dialogue: 0,0:14:10.34,0:14:13.29,Default,,0000,0000,0000,,especially if it's an AJAX request. As soon\Nas
Dialogue: 0,0:14:13.29,0:14:16.44,Default,,0000,0000,0000,,you say, visit, all the outstanding AJAX requests\Ncancel
Dialogue: 0,0:14:16.44,0:14:18.00,Default,,0000,0000,0000,,in your browser.
Dialogue: 0,0:14:18.00,0:14:20.74,Default,,0000,0000,0000,,So, you can fix this by adding an explicit
Dialogue: 0,0:14:20.74,0:14:23.98,Default,,0000,0000,0000,,wait into your Cucumber step, right. When\Nyou need
Dialogue: 0,0:14:23.98,0:14:27.33,Default,,0000,0000,0000,,to rig the race, jQuery provides this handy\Ncounter,
Dialogue: 0,0:14:27.33,0:14:30.53,Default,,0000,0000,0000,,dollar dot active. That's all the XHR requests\Nthat
Dialogue: 0,0:14:30.53,0:14:33.12,Default,,0000,0000,0000,,are outstanding. So, it's really not hard\Nto keep
Dialogue: 0,0:14:33.12,0:14:36.72,Default,,0000,0000,0000,,an eye on what's going on.
Dialogue: 0,0:14:36.72,0:14:41.09,Default,,0000,0000,0000,,Here's another offender. Creating database\Nobjects from within the
Dialogue: 0,0:14:41.09,0:14:44.71,Default,,0000,0000,0000,,test thread, right. What's wrong with this\Napproach? Now,
Dialogue: 0,0:14:44.71,0:14:49.40,Default,,0000,0000,0000,,if you're using mySQL, maybe nothing's wrong\Nwith this,
Dialogue: 0,0:14:49.40,0:14:53.75,Default,,0000,0000,0000,,right. And that's because mySQL has the transaction\Nhygiene
Dialogue: 0,0:14:53.75,0:14:59.59,Default,,0000,0000,0000,,of a roadside diner, right. There's no separation.\NIf
Dialogue: 0,0:14:59.59,0:15:02.32,Default,,0000,0000,0000,,you're using Postgres, which we are, it has\Nstricter
Dialogue: 0,0:15:02.32,0:15:04.57,Default,,0000,0000,0000,,rules about the transactions. And this can\Ncreate a
Dialogue: 0,0:15:04.57,0:15:06.72,Default,,0000,0000,0000,,world of pain.
Dialogue: 0,0:15:06.72,0:15:09.86,Default,,0000,0000,0000,,So, the test code and the Rails server are
Dialogue: 0,0:15:09.86,0:15:13.05,Default,,0000,0000,0000,,running in different threads. And this effectively\Nmeans different
Dialogue: 0,0:15:13.05,0:15:17.02,Default,,0000,0000,0000,,database connections, and that means different\Ntransaction states. Now
Dialogue: 0,0:15:17.02,0:15:20.82,Default,,0000,0000,0000,,there is some shared database connection code\Nout there.
Dialogue: 0,0:15:20.82,0:15:23.89,Default,,0000,0000,0000,,And I've had sort of mixed results with it.
Dialogue: 0,0:15:23.89,0:15:26.81,Default,,0000,0000,0000,,I've heard this thing, right, about shared\Nmutable resources
Dialogue: 0,0:15:26.81,0:15:32.20,Default,,0000,0000,0000,,between threads being problematic. Like, they\Nare. So let's
Dialogue: 0,0:15:32.20,0:15:34.64,Default,,0000,0000,0000,,say you're lucky, and both threads are in\Nthe
Dialogue: 0,0:15:34.64,0:15:38.91,Default,,0000,0000,0000,,same database transaction. Both the test thread\Nand the
Dialogue: 0,0:15:38.91,0:15:43.36,Default,,0000,0000,0000,,server thread are issuing check points and\Nrollbacks against
Dialogue: 0,0:15:43.36,0:15:47.86,Default,,0000,0000,0000,,the same connection. So sometimes one thread\Nwill reset
Dialogue: 0,0:15:47.86,0:15:50.35,Default,,0000,0000,0000,,to a checkpoint after the other thread has\Nalready
Dialogue: 0,0:15:50.35,0:15:54.12,Default,,0000,0000,0000,,rolled back the entire transaction. Right?\NAnd that's how
Dialogue: 0,0:15:54.12,0:15:56.19,Default,,0000,0000,0000,,you get a rando.
Dialogue: 0,0:15:56.19,0:15:59.03,Default,,0000,0000,0000,,So, you want to create some state within your
Dialogue: 0,0:15:59.03,0:16:01.08,Default,,0000,0000,0000,,application to run your test against, but\Nyou can't
Dialogue: 0,0:16:01.08,0:16:03.63,Default,,0000,0000,0000,,trust the test thread and the server thread\Nto
Dialogue: 0,0:16:03.63,0:16:07.38,Default,,0000,0000,0000,,read the same database state, right. What\Ndo you
Dialogue: 0,0:16:07.38,0:16:09.85,Default,,0000,0000,0000,,do?
Dialogue: 0,0:16:09.85,0:16:11.82,Default,,0000,0000,0000,,So in our project, we use a single set
Dialogue: 0,0:16:11.82,0:16:14.27,Default,,0000,0000,0000,,of fixture data that's fixed at the beginning\Nof
Dialogue: 0,0:16:14.27,0:16:16.73,Default,,0000,0000,0000,,the test run. And, essentially, the server\Nthread, or
Dialogue: 0,0:16:16.73,0:16:20.28,Default,,0000,0000,0000,,the test thread, sorry, treats the database\Nas immutable.
Dialogue: 0,0:16:20.28,0:16:22.97,Default,,0000,0000,0000,,It is read only, and any kind of verification
Dialogue: 0,0:16:22.97,0:16:25.68,Default,,0000,0000,0000,,of changes has to happen via the browser.
Dialogue: 0,0:16:25.68,0:16:29.41,Default,,0000,0000,0000,,So, we do this using RyanD's fixture_builder\Ngem, to
Dialogue: 0,0:16:29.41,0:16:34.33,Default,,0000,0000,0000,,combine the maintainable characteristics of\Nfactoried objects with the,
Dialogue: 0,0:16:34.33,0:16:37.53,Default,,0000,0000,0000,,like, set it and forget it simplicity of fixtures.
Dialogue: 0,0:16:37.53,0:16:40.69,Default,,0000,0000,0000,,So, any state that needs to exist across multiple
Dialogue: 0,0:16:40.69,0:16:43.13,Default,,0000,0000,0000,,tests is stored in a set of fixtures, and
Dialogue: 0,0:16:43.13,0:16:45.32,Default,,0000,0000,0000,,those are used throughout the test suite.
Dialogue: 0,0:16:45.32,0:16:50.91,Default,,0000,0000,0000,,And this is great, except it's also terrible.\NUnfortunately,
Dialogue: 0,0:16:50.91,0:16:56.29,Default,,0000,0000,0000,,our fixture_builder definition file is like\N900 lines long.
Dialogue: 0,0:16:56.29,0:16:59.17,Default,,0000,0000,0000,,And it's as dense as a Master's thesis, right.
Dialogue: 0,0:16:59.17,0:17:01.72,Default,,0000,0000,0000,,It takes about two minutes to rebuild the\Nfixture
Dialogue: 0,0:17:01.72,0:17:03.25,Default,,0000,0000,0000,,set. And this happens when we rebundle, change\Nthe
Dialogue: 0,0:17:03.25,0:17:07.43,Default,,0000,0000,0000,,factories, change the schema. Fortunately,\Nthat only happens a
Dialogue: 0,0:17:07.43,0:17:09.69,Default,,0000,0000,0000,,couple of times a day, right. So mostly we're
Dialogue: 0,0:17:09.69,0:17:12.12,Default,,0000,0000,0000,,saving time with it. But seriously? Two minutes\Nas
Dialogue: 0,0:17:12.12,0:17:14.54,Default,,0000,0000,0000,,your overhead to run one test is brutal.
Dialogue: 0,0:17:14.54,0:17:17.45,Default,,0000,0000,0000,,So, at our stage, we think the right solution
Dialogue: 0,0:17:17.45,0:17:20.90,Default,,0000,0000,0000,,is to use fixture_builder sparingly, right.\NUse it for
Dialogue: 0,0:17:20.90,0:17:24.09,Default,,0000,0000,0000,,Cucumber tests, because they need an immutable\Ndatabase. And
Dialogue: 0,0:17:24.09,0:17:27.72,Default,,0000,0000,0000,,maybe use it for core shared models for Rspec,
Dialogue: 0,0:17:27.72,0:17:30.12,Default,,0000,0000,0000,,but whatever you do, do not create like a
Dialogue: 0,0:17:30.12,0:17:33.50,Default,,0000,0000,0000,,DC Comics multiverse in your fixture setup\Nfile, with
Dialogue: 0,0:17:33.50,0:17:36.94,Default,,0000,0000,0000,,like different versions for everything, because\Nthat leads to
Dialogue: 0,0:17:36.94,0:17:37.60,Default,,0000,0000,0000,,pain.
Dialogue: 0,0:17:37.60,0:17:40.80,Default,,0000,0000,0000,,Another thing you want to do is Mutex it.
Dialogue: 0,0:17:40.80,0:17:43.90,Default,,0000,0000,0000,,So, a key technique we've used to prevent\Ndatabase
Dialogue: 0,0:17:43.90,0:17:46.93,Default,,0000,0000,0000,,collisions is to put a Mutex on access to
Dialogue: 0,0:17:46.93,0:17:50.78,Default,,0000,0000,0000,,the database. And this is crazy, but, you\Nknow,
Dialogue: 0,0:17:50.78,0:17:53.82,Default,,0000,0000,0000,,an app running in the browser can make more
Dialogue: 0,0:17:53.82,0:17:56.29,Default,,0000,0000,0000,,than one connection to the server at once\Nover
Dialogue: 0,0:17:56.29,0:17:59.44,Default,,0000,0000,0000,,AJAX. And that's a great place to breed race
Dialogue: 0,0:17:59.44,0:18:00.28,Default,,0000,0000,0000,,conditions.
Dialogue: 0,0:18:00.28,0:18:03.69,Default,,0000,0000,0000,,So, unless you have a Mutex, to ensure the
Dialogue: 0,0:18:03.69,0:18:07.22,Default,,0000,0000,0000,,server only responds to one request at a time,
Dialogue: 0,0:18:07.22,0:18:08.72,Default,,0000,0000,0000,,you don't necessarily know the order in which\Nthings
Dialogue: 0,0:18:08.72,0:18:10.31,Default,,0000,0000,0000,,are gonna happen, and that means you're gonna\Nget
Dialogue: 0,0:18:10.31,0:18:11.46,Default,,0000,0000,0000,,unreproducible failures.
Dialogue: 0,0:18:11.46,0:18:13.72,Default,,0000,0000,0000,,In effect, we use a Mutex to rig, rig
Dialogue: 0,0:18:13.72,0:18:15.77,Default,,0000,0000,0000,,the race. You can check it out on GitHub.
Dialogue: 0,0:18:15.77,0:18:17.54,Default,,0000,0000,0000,,It's just a sketch of the code we're using.
Dialogue: 0,0:18:17.54,0:18:24.15,Default,,0000,0000,0000,,It's on omadahealth slash capybara_sync.
Dialogue: 0,0:18:24.15,0:18:30.77,Default,,0000,0000,0000,,Faker. Some of the randomness in our test\Nsuite
Dialogue: 0,0:18:30.77,0:18:33.40,Default,,0000,0000,0000,,was due to inputs that we gave it. Our
Dialogue: 0,0:18:33.40,0:18:38.26,Default,,0000,0000,0000,,code depends on factories. And the factories\Nused randomly
Dialogue: 0,0:18:38.26,0:18:41.63,Default,,0000,0000,0000,,generated fake data to fill in names, zip\Ncodes,
Dialogue: 0,0:18:41.63,0:18:44.01,Default,,0000,0000,0000,,all the text fields. And there are good reasons
Dialogue: 0,0:18:44.01,0:18:45.01,Default,,0000,0000,0000,,to use random data.
Dialogue: 0,0:18:45.01,0:18:49.27,Default,,0000,0000,0000,,It regularly exercises your edge cases. Engineers\Ndon't have
Dialogue: 0,0:18:49.27,0:18:51.55,Default,,0000,0000,0000,,to think of all possible first names you could
Dialogue: 0,0:18:51.55,0:18:54.36,Default,,0000,0000,0000,,use. The code should work the same regardless\Nof
Dialogue: 0,0:18:54.36,0:18:57.76,Default,,0000,0000,0000,,what zip code someone is in. But sometimes\Nit
Dialogue: 0,0:18:57.76,0:18:59.56,Default,,0000,0000,0000,,doesn't.
Dialogue: 0,0:18:59.56,0:19:03.65,Default,,0000,0000,0000,,For example, did you know that Faker includes\NGuam
Dialogue: 0,0:19:03.65,0:19:05.70,Default,,0000,0000,0000,,and Porto Rico in the states that it might
Dialogue: 0,0:19:05.70,0:19:09.05,Default,,0000,0000,0000,,generate for someone? And we didn't include\Nthose in
Dialogue: 0,0:19:09.05,0:19:12.69,Default,,0000,0000,0000,,our states dropdown. So when a Cucumber test\Nedits
Dialogue: 0,0:19:12.69,0:19:15.13,Default,,0000,0000,0000,,an account for a user that Faker placed in
Dialogue: 0,0:19:15.13,0:19:18.22,Default,,0000,0000,0000,,Guam, and their state is not entered when\Nyou
Dialogue: 0,0:19:18.22,0:19:20.41,Default,,0000,0000,0000,,try to click save. And that leads to a
Dialogue: 0,0:19:20.41,0:19:23.38,Default,,0000,0000,0000,,validation failure, and that leads to Cucumber\Nnot seeing
Dialogue: 0,0:19:23.38,0:19:26.47,Default,,0000,0000,0000,,the expected results, and a test run with,\Nfrom
Dialogue: 0,0:19:26.47,0:19:33.47,Default,,0000,0000,0000,,a new factory will not reproduce that failure,\Nright.
Dialogue: 0,0:19:37.21,0:19:40.36,Default,,0000,0000,0000,,Something crazy happened. Here we go.
Dialogue: 0,0:19:40.36,0:19:45.55,Default,,0000,0000,0000,,Times and dates. Oh, we're out of sync. Let
Dialogue: 0,0:19:45.55,0:19:52.55,Default,,0000,0000,0000,,me just. Momentary technical difficulties.\NMhmm.
Dialogue: 0,0:20:02.31,0:20:09.17,Default,,0000,0000,0000,,Cool.
Dialogue: 0,0:20:09.17,0:20:16.17,Default,,0000,0000,0000,,OK. Times and dates. Another subtle input\Nto your
Dialogue: 0,0:20:19.46,0:20:23.85,Default,,0000,0000,0000,,code is the current time. Our app sets itself
Dialogue: 0,0:20:23.85,0:20:26.72,Default,,0000,0000,0000,,up to be on the user's time zone, to
Dialogue: 0,0:20:26.72,0:20:29.74,Default,,0000,0000,0000,,prevent time-dependent data, like which week\Nof our program
Dialogue: 0,0:20:29.74,0:20:32.74,Default,,0000,0000,0000,,you are on in the middle of Saturday night.
Dialogue: 0,0:20:32.74,0:20:34.74,Default,,0000,0000,0000,,And this was policy. We all knew about this.
Dialogue: 0,0:20:34.74,0:20:38.35,Default,,0000,0000,0000,,We always used zone-aware time calls.
Dialogue: 0,0:20:38.35,0:20:40.80,Default,,0000,0000,0000,,Except that we didn't. Like, when I audited\Nit,
Dialogue: 0,0:20:40.80,0:20:43.03,Default,,0000,0000,0000,,I found over a hundred places where we neglected
Dialogue: 0,0:20:43.03,0:20:46.65,Default,,0000,0000,0000,,to use zone-aware time calls. So most of these
Dialogue: 0,0:20:46.65,0:20:50.91,Default,,0000,0000,0000,,are fine. There's usually nothing wrong with\Nepic seconds.
Dialogue: 0,0:20:50.91,0:20:53.60,Default,,0000,0000,0000,,But it only takes one misplaced call to time
Dialogue: 0,0:20:53.60,0:20:56.63,Default,,0000,0000,0000,,dot now to create a failure. It's really best
Dialogue: 0,0:20:56.63,0:20:59.11,Default,,0000,0000,0000,,to just forget about time dot now. Search\Nyour
Dialogue: 0,0:20:59.11,0:21:01.55,Default,,0000,0000,0000,,code base for it and eliminate it. Always\Nuse
Dialogue: 0,0:21:01.55,0:21:04.10,Default,,0000,0000,0000,,time dot zone dot now. Same thing for date
Dialogue: 0,0:21:04.10,0:21:07.05,Default,,0000,0000,0000,,dot today. That's time zone dependent. You\Nwant to
Dialogue: 0,0:21:07.05,0:21:09.06,Default,,0000,0000,0000,,use time dot zone dot today.
Dialogue: 0,0:21:09.06,0:21:11.60,Default,,0000,0000,0000,,Unsurprisingly, I found a bunch of this class\Nof
Dialogue: 0,0:21:11.60,0:21:15.24,Default,,0000,0000,0000,,failure when I was at RubyConf in Miami. So
Dialogue: 0,0:21:15.24,0:21:19.25,Default,,0000,0000,0000,,these methods create random failures. Because\Nyour database objects
Dialogue: 0,0:21:19.25,0:21:21.28,Default,,0000,0000,0000,,can be in a different time zone than your
Dialogue: 0,0:21:21.28,0:21:25.58,Default,,0000,0000,0000,,machine's local time zone.
Dialogue: 0,0:21:25.58,0:21:28.92,Default,,0000,0000,0000,,External dependencies. Any time you depend\Non a third
Dialogue: 0,0:21:28.92,0:21:32.26,Default,,0000,0000,0000,,party service in your test, you introduce\Na possible
Dialogue: 0,0:21:32.26,0:21:37.19,Default,,0000,0000,0000,,random element, right. S3, Google Analytics,\NFacebook. Any of
Dialogue: 0,0:21:37.19,0:21:39.40,Default,,0000,0000,0000,,these things can go down. They can be slow.
Dialogue: 0,0:21:39.40,0:21:42.67,Default,,0000,0000,0000,,They can be broken. Additionally, they all\Ndepend on
Dialogue: 0,0:21:42.67,0:21:46.10,Default,,0000,0000,0000,,the quality of your local internet connection.
Dialogue: 0,0:21:46.10,0:21:49.87,Default,,0000,0000,0000,,So, I'm gonna suggest that if you are affected
Dialogue: 0,0:21:49.87,0:21:54.57,Default,,0000,0000,0000,,by random failures, it's important to reproduce\Nthe failure.
Dialogue: 0,0:21:54.57,0:21:56.13,Default,,0000,0000,0000,,It is possible. It is possible. It is not
Dialogue: 0,0:21:56.13,0:21:59.24,Default,,0000,0000,0000,,only possible. It is critical. And any problem\Nthat
Dialogue: 0,0:21:59.24,0:22:04.12,Default,,0000,0000,0000,,you can reproduce, reliably, can be solved.\NWell, at
Dialogue: 0,0:22:04.12,0:22:06.13,Default,,0000,0000,0000,,least, if you can reproduce it, you have a
Dialogue: 0,0:22:06.13,0:22:10.21,Default,,0000,0000,0000,,heck of a lot better chance of solving it.
Dialogue: 0,0:22:10.21,0:22:14.32,Default,,0000,0000,0000,,So, you have to bolt it all down. How
Dialogue: 0,0:22:14.32,0:22:16.66,Default,,0000,0000,0000,,do you fix the data? When you're trying to
Dialogue: 0,0:22:16.66,0:22:19.05,Default,,0000,0000,0000,,reproduce a random failure, you're gonna need\Nthe same
Dialogue: 0,0:22:19.05,0:22:22.41,Default,,0000,0000,0000,,database objects used by the failing test.\NSo if
Dialogue: 0,0:22:22.41,0:22:24.15,Default,,0000,0000,0000,,you used factories, and there's not a file\Nsystem
Dialogue: 0,0:22:24.15,0:22:27.03,Default,,0000,0000,0000,,record when a test starts to fail randomly,\Nyou're
Dialogue: 0,0:22:27.03,0:22:29.11,Default,,0000,0000,0000,,gonna want to document the database state\Nat the
Dialogue: 0,0:22:29.11,0:22:30.25,Default,,0000,0000,0000,,time of failure.
Dialogue: 0,0:22:30.25,0:22:33.32,Default,,0000,0000,0000,,And that's gonna mean yml fixtures or, like,\Nand
Dialogue: 0,0:22:33.32,0:22:35.54,Default,,0000,0000,0000,,SQL dump, or something else clever. You have\Nto
Dialogue: 0,0:22:35.54,0:22:38.12,Default,,0000,0000,0000,,find a way to re-establish that same state\Nthat
Dialogue: 0,0:22:38.12,0:22:41.88,Default,,0000,0000,0000,,was created at the moment that you had the
Dialogue: 0,0:22:41.88,0:22:44.92,Default,,0000,0000,0000,,failure. And the network. Great talk before\Nabout how
Dialogue: 0,0:22:44.92,0:22:48.05,Default,,0000,0000,0000,,to nail down the network. API calls and responses
Dialogue: 0,0:22:48.05,0:22:52.28,Default,,0000,0000,0000,,are input for your code. Web-mock, vcr, other\Nlibraries
Dialogue: 0,0:22:52.28,0:22:55.52,Default,,0000,0000,0000,,exist to replay third party service responses.
Dialogue: 0,0:22:55.52,0:22:57.91,Default,,0000,0000,0000,,So, if you're trying to reproduce a failure\Nin
Dialogue: 0,0:22:57.91,0:23:00.57,Default,,0000,0000,0000,,a test that has any third party dependencies,\Nyou're
Dialogue: 0,0:23:00.57,0:23:03.38,Default,,0000,0000,0000,,gonna wanna use a library to capture and replay
Dialogue: 0,0:23:03.38,0:23:05.73,Default,,0000,0000,0000,,those responses.
Dialogue: 0,0:23:05.73,0:23:09.82,Default,,0000,0000,0000,,Also, share buttons, right. In your Cucumber\Ntests, you're
Dialogue: 0,0:23:09.82,0:23:12.78,Default,,0000,0000,0000,,gonna wanna remove the calls to Google Analytics,\NFacebook
Dialogue: 0,0:23:12.78,0:23:15.68,Default,,0000,0000,0000,,lite buttons, all that stuff from the browser.\NThese
Dialogue: 0,0:23:15.68,0:23:18.23,Default,,0000,0000,0000,,slow down your page load time, and they create
Dialogue: 0,0:23:18.23,0:23:21.67,Default,,0000,0000,0000,,unnecessary failures because of that.
Dialogue: 0,0:23:21.67,0:23:24.41,Default,,0000,0000,0000,,But, if you're replaying all your network\Ncalls, how
Dialogue: 0,0:23:24.41,0:23:27.55,Default,,0000,0000,0000,,do you know the external API hasn't changed,\Nright?
Dialogue: 0,0:23:27.55,0:23:29.20,Default,,0000,0000,0000,,You want to test the services that your code
Dialogue: 0,0:23:29.20,0:23:31.41,Default,,0000,0000,0000,,depends on, too. So you need a build that
Dialogue: 0,0:23:31.41,0:23:34.85,Default,,0000,0000,0000,,does that. But it shouldn't be the main build.
Dialogue: 0,0:23:34.85,0:23:36.39,Default,,0000,0000,0000,,Purpose of the main build is to let the
Dialogue: 0,0:23:36.39,0:23:40.00,Default,,0000,0000,0000,,team know when their code is broken, when\Ntheir
Dialogue: 0,0:23:40.00,0:23:42.84,Default,,0000,0000,0000,,code is broken. And it should do that as
Dialogue: 0,0:23:42.84,0:23:44.74,Default,,0000,0000,0000,,quickly as possible.
Dialogue: 0,0:23:44.74,0:23:46.96,Default,,0000,0000,0000,,And then we have a separate, external build\Nthat
Dialogue: 0,0:23:46.96,0:23:50.54,Default,,0000,0000,0000,,tests the interactions with third party services.\NSo, essentially,
Dialogue: 0,0:23:50.54,0:23:52.57,Default,,0000,0000,0000,,external communication is off and then on,\Nand we
Dialogue: 0,0:23:52.57,0:23:54.11,Default,,0000,0000,0000,,check build results for both.
Dialogue: 0,0:23:54.11,0:23:57.99,Default,,0000,0000,0000,,So, I want to talk about another reason that
Dialogue: 0,0:23:57.99,0:24:01.36,Default,,0000,0000,0000,,tests fail randomly. Rspec runs all your tests\Nin
Dialogue: 0,0:24:01.36,0:24:06.20,Default,,0000,0000,0000,,a random order every time. And obviously this\Nintroduces
Dialogue: 0,0:24:06.20,0:24:09.45,Default,,0000,0000,0000,,randomness. But, there is a reason for that,\Nand
Dialogue: 0,0:24:09.45,0:24:11.13,Default,,0000,0000,0000,,the reason is to help you stay on top
Dialogue: 0,0:24:11.13,0:24:13.67,Default,,0000,0000,0000,,of test pollution.
Dialogue: 0,0:24:13.67,0:24:16.02,Default,,0000,0000,0000,,Test pollution is when state that is changed\Nin
Dialogue: 0,0:24:16.02,0:24:19.87,Default,,0000,0000,0000,,one test persists and influences the results\Nof other
Dialogue: 0,0:24:19.87,0:24:22.95,Default,,0000,0000,0000,,tests. Changed state can live in process memory,\Nin
Dialogue: 0,0:24:22.95,0:24:26.46,Default,,0000,0000,0000,,a database, on the file system, in an external
Dialogue: 0,0:24:26.46,0:24:29.24,Default,,0000,0000,0000,,service. Right. Lots of places.
Dialogue: 0,0:24:29.24,0:24:32.75,Default,,0000,0000,0000,,Sometimes, the polluted state causes the subsequent\Ntest to
Dialogue: 0,0:24:32.75,0:24:37.07,Default,,0000,0000,0000,,fail incorrectly. And sometimes it causes\Nthe subsequent test
Dialogue: 0,0:24:37.07,0:24:40.79,Default,,0000,0000,0000,,to pass incorrectly. And this was such a rampant
Dialogue: 0,0:24:40.79,0:24:43.55,Default,,0000,0000,0000,,issue in the early days of Rspec that the
Dialogue: 0,0:24:43.55,0:24:45.71,Default,,0000,0000,0000,,Rspec team made running the tests in a random
Dialogue: 0,0:24:45.71,0:24:49.74,Default,,0000,0000,0000,,order the default as of Rspec 2. So, thank
Dialogue: 0,0:24:49.74,0:24:51.22,Default,,0000,0000,0000,,you Rspec.
Dialogue: 0,0:24:51.22,0:24:56.42,Default,,0000,0000,0000,,Now, any test pollution issues should stand\Nout. But
Dialogue: 0,0:24:56.42,0:24:59.39,Default,,0000,0000,0000,,what do you think happens if you ignore random
Dialogue: 0,0:24:59.39,0:25:04.67,Default,,0000,0000,0000,,test failures for like a year or so? Yeah.
Dialogue: 0,0:25:04.67,0:25:08.08,Default,,0000,0000,0000,,Here's some clues that your issue might be\Ntest
Dialogue: 0,0:25:08.08,0:25:10.49,Default,,0000,0000,0000,,pollution, right.
Dialogue: 0,0:25:10.49,0:25:14.09,Default,,0000,0000,0000,,With test pollution, the effected tests never\Nfail when
Dialogue: 0,0:25:14.09,0:25:17.60,Default,,0000,0000,0000,,they're run in isolation. Not ever. And rather\Nthan
Dialogue: 0,0:25:17.60,0:25:21.97,Default,,0000,0000,0000,,throwing an unexpected exception, a test pollution\Nfailure usually
Dialogue: 0,0:25:21.97,0:25:24.76,Default,,0000,0000,0000,,takes the form of returning different data\Nthan what
Dialogue: 0,0:25:24.76,0:25:26.48,Default,,0000,0000,0000,,you expected.
Dialogue: 0,0:25:26.48,0:25:29.46,Default,,0000,0000,0000,,And finally, the biggest clue that you might\Nhave
Dialogue: 0,0:25:29.46,0:25:31.73,Default,,0000,0000,0000,,a test pollution issue is that you haven't\Nreally
Dialogue: 0,0:25:31.73,0:25:36.91,Default,,0000,0000,0000,,been checking for test pollution. So, we gotta\Nreproduce
Dialogue: 0,0:25:36.91,0:25:39.06,Default,,0000,0000,0000,,test pollution issues. Which means we have\Nto run
Dialogue: 0,0:25:39.06,0:25:40.98,Default,,0000,0000,0000,,the tests suite in the same order, and we
Dialogue: 0,0:25:40.98,0:25:43.93,Default,,0000,0000,0000,,have to use the fixture or database data and
Dialogue: 0,0:25:43.93,0:25:46.54,Default,,0000,0000,0000,,the network data from the failed build.
Dialogue: 0,0:25:46.54,0:25:49.90,Default,,0000,0000,0000,,So, first you have to identify the random\Nseed.
Dialogue: 0,0:25:49.90,0:25:52.34,Default,,0000,0000,0000,,Maybe you've seen this cryptic line at the\Nend
Dialogue: 0,0:25:52.34,0:25:55.74,Default,,0000,0000,0000,,of your Rspec test output. This is not completely
Dialogue: 0,0:25:55.74,0:25:59.99,Default,,0000,0000,0000,,meaningless. 22164 is your magic key to rerun\Nthe
Dialogue: 0,0:25:59.99,0:26:01.82,Default,,0000,0000,0000,,test in the same order as the build that
Dialogue: 0,0:26:01.82,0:26:05.21,Default,,0000,0000,0000,,just ran. So you want to modify your dot
Dialogue: 0,0:26:05.21,0:26:08.93,Default,,0000,0000,0000,,Rspec file to include the seed value. Be sure
Dialogue: 0,0:26:08.93,0:26:11.50,Default,,0000,0000,0000,,to format, to change the format to documentation\Nas
Dialogue: 0,0:26:11.50,0:26:13.47,Default,,0000,0000,0000,,well as adding the seed. That will make it
Dialogue: 0,0:26:13.47,0:26:15.64,Default,,0000,0000,0000,,more readable, for you, so that you can start
Dialogue: 0,0:26:15.64,0:26:17.26,Default,,0000,0000,0000,,to think about the order that things are running
Dialogue: 0,0:26:17.26,0:26:19.53,Default,,0000,0000,0000,,in and what could possibly be causing your\Npollution
Dialogue: 0,0:26:19.53,0:26:21.31,Default,,0000,0000,0000,,problem.
Dialogue: 0,0:26:21.31,0:26:24.98,Default,,0000,0000,0000,,So, the problem with test pollution is fundamentally\Nabout
Dialogue: 0,0:26:24.98,0:26:28.48,Default,,0000,0000,0000,,incorrectly persisted state, so that the state\Nthat's being
Dialogue: 0,0:26:28.48,0:26:31.60,Default,,0000,0000,0000,,persisted is important. You want to ensure\Nthat the
Dialogue: 0,0:26:31.60,0:26:35.32,Default,,0000,0000,0000,,data is identical to the failed build. And\Nthere's
Dialogue: 0,0:26:35.32,0:26:39.01,Default,,0000,0000,0000,,lots of ways to do this.
Dialogue: 0,0:26:39.01,0:26:41.68,Default,,0000,0000,0000,,So you've got your random seed. You've got\Nyour
Dialogue: 0,0:26:41.68,0:26:43.94,Default,,0000,0000,0000,,data from the failed build, and then you rerun
Dialogue: 0,0:26:43.94,0:26:47.11,Default,,0000,0000,0000,,the specs. And if you see the failure repeated,
Dialogue: 0,0:26:47.11,0:26:50.22,Default,,0000,0000,0000,,you should celebrate, right. You've correctly\Ndiagnosed that the
Dialogue: 0,0:26:50.22,0:26:52.37,Default,,0000,0000,0000,,issue is test pollution and you are on your
Dialogue: 0,0:26:52.37,0:26:53.44,Default,,0000,0000,0000,,way to fixing it.
Dialogue: 0,0:26:53.44,0:26:56.37,Default,,0000,0000,0000,,And if you don't see the failure, maybe it's
Dialogue: 0,0:26:56.37,0:26:59.33,Default,,0000,0000,0000,,not test pollution. Maybe there's another\Naspect of your
Dialogue: 0,0:26:59.33,0:27:03.16,Default,,0000,0000,0000,,build environment that needs to be duplicated,\Nright. But
Dialogue: 0,0:27:03.16,0:27:07.70,Default,,0000,0000,0000,,even then, say you've reproduced the problem.\NNow what?
Dialogue: 0,0:27:07.70,0:27:10.57,Default,,0000,0000,0000,,You still have to diagnose what is causing\Nthe
Dialogue: 0,0:27:10.57,0:27:12.91,Default,,0000,0000,0000,,pollution. You know that running the tests\Nin a
Dialogue: 0,0:27:12.91,0:27:16.95,Default,,0000,0000,0000,,particular order creates a failure. The problem\Nwith test
Dialogue: 0,0:27:16.95,0:27:19.80,Default,,0000,0000,0000,,pollution is that there is a non-obvious connection\Nbetween
Dialogue: 0,0:27:19.80,0:27:22.77,Default,,0000,0000,0000,,where the problem appears in the failed test\Nand
Dialogue: 0,0:27:22.77,0:27:25.83,Default,,0000,0000,0000,,its source in another test case.
Dialogue: 0,0:27:25.83,0:27:28.28,Default,,0000,0000,0000,,And you can find out about the failure using
Dialogue: 0,0:27:28.28,0:27:31.98,Default,,0000,0000,0000,,print statements or debugger, using whatever\Ntools you want.
Dialogue: 0,0:27:31.98,0:27:34.77,Default,,0000,0000,0000,,But, maybe you get lucky and you are able
Dialogue: 0,0:27:34.77,0:27:37.10,Default,,0000,0000,0000,,to just figure it out. But in a complex
Dialogue: 0,0:27:37.10,0:27:41.08,Default,,0000,0000,0000,,code base with thousands of tests the source\Nof
Dialogue: 0,0:27:41.08,0:27:44.71,Default,,0000,0000,0000,,the pollution can be tricky to track down.
Dialogue: 0,0:27:44.71,0:27:48.94,Default,,0000,0000,0000,,So, just running through the suite to reproduce\Nthe
Dialogue: 0,0:27:48.94,0:27:52.06,Default,,0000,0000,0000,,failure might take ten minutes. And this is\Nactually
Dialogue: 0,0:27:52.06,0:27:56.09,Default,,0000,0000,0000,,terrible, right. Waiting ten minutes for feedback?\NThis is
Dialogue: 0,0:27:56.09,0:28:00.66,Default,,0000,0000,0000,,a source of cognitive depletion. All of the\Nstack
Dialogue: 0,0:28:00.66,0:28:02.65,Default,,0000,0000,0000,,you've built up in your brain to solve this
Dialogue: 0,0:28:02.65,0:28:05.91,Default,,0000,0000,0000,,problem is disintegrating over that ten minutes.\NYou're gonna
Dialogue: 0,0:28:05.91,0:28:08.98,Default,,0000,0000,0000,,work on other problems. You're gonna check\NFacebook while
Dialogue: 0,0:28:08.98,0:28:11.46,Default,,0000,0000,0000,,those tests are running. And you're gonna\Nlose your
Dialogue: 0,0:28:11.46,0:28:16.28,Default,,0000,0000,0000,,focus, right. And that is, essentially, how\Nrando wins.
Dialogue: 0,0:28:16.28,0:28:20.96,Default,,0000,0000,0000,,Fortunately, we can discard large amounts\Nof complexity and
Dialogue: 0,0:28:20.96,0:28:23.18,Default,,0000,0000,0000,,noise, by using a stupid process that we don't
Dialogue: 0,0:28:23.18,0:28:28.91,Default,,0000,0000,0000,,have to think about. Binary search. In code,\Ndebugging
Dialogue: 0,0:28:28.91,0:28:32.44,Default,,0000,0000,0000,,via binary search is a process of repeatedly\Ndividing
Dialogue: 0,0:28:32.44,0:28:36.03,Default,,0000,0000,0000,,the search space in half, until you locate\Nthe
Dialogue: 0,0:28:36.03,0:28:41.19,Default,,0000,0000,0000,,smallest coherent unit that exhibits the desired\Nbehavior.
Dialogue: 0,0:28:41.19,0:28:44.90,Default,,0000,0000,0000,,OK. So we have the output of a set
Dialogue: 0,0:28:44.90,0:28:47.63,Default,,0000,0000,0000,,of specs that we ran in documentation mode.\NThis
Dialogue: 0,0:28:47.63,0:28:49.73,Default,,0000,0000,0000,,is sort of a high level overview that you
Dialogue: 0,0:28:49.73,0:28:52.38,Default,,0000,0000,0000,,might see in Sublime, right. And in the middle
Dialogue: 0,0:28:52.38,0:28:56.22,Default,,0000,0000,0000,,here, this red spot is where the failure occurs.
Dialogue: 0,0:28:56.22,0:28:59.69,Default,,0000,0000,0000,,So we know the cause has to happen before
Dialogue: 0,0:28:59.69,0:29:05.07,Default,,0000,0000,0000,,the failure, because causality. So in the\Ngreen block,
Dialogue: 0,0:29:05.07,0:29:08.77,Default,,0000,0000,0000,,at the top, is, that's the candidate block,\Nor
Dialogue: 0,0:29:08.77,0:29:10.70,Default,,0000,0000,0000,,the search space.
Dialogue: 0,0:29:10.70,0:29:16.79,Default,,0000,0000,0000,,So, practically, we split the search space\Nin half,
Dialogue: 0,0:29:16.79,0:29:19.63,Default,,0000,0000,0000,,and remove half of it. And if the failure
Dialogue: 0,0:29:19.63,0:29:23.40,Default,,0000,0000,0000,,reoccurs when we rerun with this configuration,\Nwe know
Dialogue: 0,0:29:23.40,0:29:28.37,Default,,0000,0000,0000,,that the cause is in that remaining block,\Nright.
Dialogue: 0,0:29:28.37,0:29:30.67,Default,,0000,0000,0000,,But sometimes you've got more problems than\Nyou know.
Dialogue: 0,0:29:30.67,0:29:32.24,Default,,0000,0000,0000,,So it's good to test the other half of
Dialogue: 0,0:29:32.24,0:29:35.69,Default,,0000,0000,0000,,the search space as well.
Dialogue: 0,0:29:35.69,0:29:37.98,Default,,0000,0000,0000,,So if you're failure appeared in step zero,\Nyou
Dialogue: 0,0:29:37.98,0:29:41.10,Default,,0000,0000,0000,,expect not to see the failure here. If you
Dialogue: 0,0:29:41.10,0:29:44.79,Default,,0000,0000,0000,,also see the failure here, you might have\Nmultiple
Dialogue: 0,0:29:44.79,0:29:48.87,Default,,0000,0000,0000,,sources of test pollution or, more likely,\Ntest pollution
Dialogue: 0,0:29:48.87,0:29:51.28,Default,,0000,0000,0000,,isn't really your problem, and the problem\Nis actually
Dialogue: 0,0:29:51.28,0:29:55.09,Default,,0000,0000,0000,,outside of the search space.
Dialogue: 0,0:29:55.09,0:29:58.84,Default,,0000,0000,0000,,So here's a hiccup. Binary search requires\Nus to
Dialogue: 0,0:29:58.84,0:30:01.15,Default,,0000,0000,0000,,remove large segments of the test suite to\Nnarrow
Dialogue: 0,0:30:01.15,0:30:04.03,Default,,0000,0000,0000,,in on the test that causes the pollution.\NAnd
Dialogue: 0,0:30:04.03,0:30:07.22,Default,,0000,0000,0000,,this creates a problem, because random ordering\Nin the
Dialogue: 0,0:30:07.22,0:30:11.29,Default,,0000,0000,0000,,test suite changes when you remove tests.\NCompletely. Remove
Dialogue: 0,0:30:11.29,0:30:14.46,Default,,0000,0000,0000,,one test, the whole thing reshuffles on the\Nsame
Dialogue: 0,0:30:14.46,0:30:17.55,Default,,0000,0000,0000,,seeds. So there's no way to effectively perform\Na
Dialogue: 0,0:30:17.55,0:30:21.21,Default,,0000,0000,0000,,binary search using a random seed.
Dialogue: 0,0:30:21.21,0:30:23.71,Default,,0000,0000,0000,,So here's the good news. It is possible to
Dialogue: 0,0:30:23.71,0:30:27.12,Default,,0000,0000,0000,,manually declare the ordering of your Rspec\Ntests, using
Dialogue: 0,0:30:27.12,0:30:34.12,Default,,0000,0000,0000,,this undocumented configuration option, order_examples.\NSo, config dot order_examples
Dialogue: 0,0:30:35.09,0:30:37.75,Default,,0000,0000,0000,,takes a block, and that'll get the whole collection
Dialogue: 0,0:30:37.75,0:30:41.18,Default,,0000,0000,0000,,of Rspec examples after Rspec has loaded the\Nspecs
Dialogue: 0,0:30:41.18,0:30:43.43,Default,,0000,0000,0000,,to be run. And then you just reorder the
Dialogue: 0,0:30:43.43,0:30:45.37,Default,,0000,0000,0000,,examples in whatever order you want them to\Nbe
Dialogue: 0,0:30:45.37,0:30:48.62,Default,,0000,0000,0000,,ordered in and return that set from the block.
Dialogue: 0,0:30:48.62,0:30:50.73,Default,,0000,0000,0000,,So, that sounds simple.
Dialogue: 0,0:30:50.73,0:30:56.69,Default,,0000,0000,0000,,I, I made a little proto-gem for this. It's
Dialogue: 0,0:30:56.69,0:31:00.83,Default,,0000,0000,0000,,called rspec_manual_order, and basically it\Ntakes the output of
Dialogue: 0,0:31:00.83,0:31:04.11,Default,,0000,0000,0000,,the documentation format from the test that\Nyou ran
Dialogue: 0,0:31:04.11,0:31:08.82,Default,,0000,0000,0000,,earlier, and turns that into an ordering list.\NSo,
Dialogue: 0,0:31:08.82,0:31:11.02,Default,,0000,0000,0000,,if you, if you log the output of, of
Dialogue: 0,0:31:11.02,0:31:13.58,Default,,0000,0000,0000,,your Rspec suite with the failure to a file,
Dialogue: 0,0:31:13.58,0:31:16.21,Default,,0000,0000,0000,,you'll be able to replay it using rspec_manual_order,\Nand
Dialogue: 0,0:31:16.21,0:31:18.93,Default,,0000,0000,0000,,you can check that out on GitHub.
Dialogue: 0,0:31:18.93,0:31:25.21,Default,,0000,0000,0000,,So it's possible to reduce the search space\Nand
Dialogue: 0,0:31:25.21,0:31:27.42,Default,,0000,0000,0000,,do a binary search on Rspec. And once you've
Dialogue: 0,0:31:27.42,0:31:30.08,Default,,0000,0000,0000,,reduced the search space to a single spec\Nor
Dialogue: 0,0:31:30.08,0:31:32.50,Default,,0000,0000,0000,,a suite of examples that all cause the problem,
Dialogue: 0,0:31:32.50,0:31:35.51,Default,,0000,0000,0000,,you put your monkey in the position to shine
Dialogue: 0,0:31:35.51,0:31:38.49,Default,,0000,0000,0000,,against your test pollution issue, right.\NThis is where
Dialogue: 0,0:31:38.49,0:31:40.57,Default,,0000,0000,0000,,it actually becomes possible to figure it\Nout by
Dialogue: 0,0:31:40.57,0:31:43.50,Default,,0000,0000,0000,,looking at the context.
Dialogue: 0,0:31:43.50,0:31:45.72,Default,,0000,0000,0000,,I've gone in depth into test pollution, because\Nit's
Dialogue: 0,0:31:45.72,0:31:50.86,Default,,0000,0000,0000,,amenable to investigation using simple techniques,\Nright. Binary search
Dialogue: 0,0:31:50.86,0:31:54.10,Default,,0000,0000,0000,,and reproducing the failure state are key\Ndebugging skills
Dialogue: 0,0:31:54.10,0:31:56.99,Default,,0000,0000,0000,,that you will improve with practice. When\NI started
Dialogue: 0,0:31:56.99,0:31:59.47,Default,,0000,0000,0000,,looking into our random failures, I didn't\Nknow we
Dialogue: 0,0:31:59.47,0:32:02.85,Default,,0000,0000,0000,,had test pollution issues. Turned out we weren't\Nresetting
Dialogue: 0,0:32:02.85,0:32:06.10,Default,,0000,0000,0000,,the global time zone correctly between tests.
Dialogue: 0,0:32:06.10,0:32:08.11,Default,,0000,0000,0000,,This was far from the only problem I found.
Dialogue: 0,0:32:08.11,0:32:11.14,Default,,0000,0000,0000,,But without fixing this one, our suite would\Nnever
Dialogue: 0,0:32:11.14,0:32:16.40,Default,,0000,0000,0000,,be clean. So, every random failure that you\Nare
Dialogue: 0,0:32:16.40,0:32:19.79,Default,,0000,0000,0000,,chasing has its own unique story. There are\Nsome
Dialogue: 0,0:32:19.79,0:32:22.24,Default,,0000,0000,0000,,in our code that we haven't figured out yet,
Dialogue: 0,0:32:22.24,0:32:23.62,Default,,0000,0000,0000,,and there are some in your code that I
Dialogue: 0,0:32:23.62,0:32:26.93,Default,,0000,0000,0000,,hope I never see.
Dialogue: 0,0:32:26.93,0:32:30.61,Default,,0000,0000,0000,,The key to eliminating random test failures\Nis don't
Dialogue: 0,0:32:30.61,0:32:33.88,Default,,0000,0000,0000,,give up, right. Today we've covered things\Nthat go
Dialogue: 0,0:32:33.88,0:32:36.90,Default,,0000,0000,0000,,wrong in Cucumber and Capybara. Things that\Ngo wrong
Dialogue: 0,0:32:36.90,0:32:39.87,Default,,0000,0000,0000,,in Rspec and just general sources of randomness\Nin
Dialogue: 0,0:32:39.87,0:32:42.10,Default,,0000,0000,0000,,your test suite. And hopefully you're walking\Nout of
Dialogue: 0,0:32:42.10,0:32:44.20,Default,,0000,0000,0000,,here with at least one new technique to improve
Dialogue: 0,0:32:44.20,0:32:46.40,Default,,0000,0000,0000,,the reliability of your tests.
Dialogue: 0,0:32:46.40,0:32:48.26,Default,,0000,0000,0000,,We've been working with ours for about eight\Nmonths,
Dialogue: 0,0:32:48.26,0:32:50.08,Default,,0000,0000,0000,,and we're in a place where random failures\Noccur
Dialogue: 0,0:32:50.08,0:32:53.79,Default,,0000,0000,0000,,like, less than five percent of the time.\NAnd
Dialogue: 0,0:32:53.79,0:32:55.62,Default,,0000,0000,0000,,we set up a tiered build system to run
Dialogue: 0,0:32:55.62,0:32:58.82,Default,,0000,0000,0000,,the tests sequentially when the fast parallel\Nbuild fails.
Dialogue: 0,0:32:58.82,0:33:02.15,Default,,0000,0000,0000,,So, the important thing is that when new random
Dialogue: 0,0:33:02.15,0:33:04.57,Default,,0000,0000,0000,,failures occur, we reliably assign a team\Nto hunt
Dialogue: 0,0:33:04.57,0:33:05.40,Default,,0000,0000,0000,,them down.
Dialogue: 0,0:33:05.40,0:33:08.05,Default,,0000,0000,0000,,And if you keep working on your build, eventually
Dialogue: 0,0:33:08.05,0:33:10.05,Default,,0000,0000,0000,,you'll figure out a combination of tactics\Nthat will
Dialogue: 0,0:33:10.05,0:33:12.80,Default,,0000,0000,0000,,lead to a stable, reliable test suite, that\Nwill
Dialogue: 0,0:33:12.80,0:33:15.74,Default,,0000,0000,0000,,have the trust of your team. So thank you.