0:00:18.330,0:00:18.950
LUKE MELIA: All right, folks.

0:00:18.950,0:00:24.040
Thanks for coming out. So you are at Lightning[br]Fast

0:00:24.040,0:00:26.180
Deployment of Your Rails-Baked Javascript[br]APP.

0:00:26.180,0:00:29.349
Hopefully you're in the right place.

0:00:29.349,0:00:32.369
My name is Luke. I, Luke Melia. I live

0:00:32.369,0:00:35.540
in Manhattan in New York City. Got a couple

0:00:35.540,0:00:39.140
little girls who are learning Ruby and Javascript[br]using

0:00:39.140,0:00:44.050
Code Academy and KidsRuby. And I have a company

0:00:44.050,0:00:46.980
called Yapp that I co-founded. We're one of[br]these

0:00:46.980,0:00:51.110
kind of hybrid product and consulting companies.[br]And when

0:00:51.110,0:00:54.080
I'm not doing Dad stuff or coding, I really

0:00:54.080,0:00:57.000
love to play beach volleyball and have recently[br]taken

0:00:57.000,0:00:58.809
up parkour.

0:00:58.809,0:01:01.549
So Yapp Labs is the consulting side of our

0:01:01.549,0:01:04.290
business. We do Ember.js consulting and training[br]based out

0:01:04.290,0:01:08.000
of New York and Seattle, if, if that's interesting,

0:01:08.000,0:01:09.900
happy to talk with you about that.

0:01:09.900,0:01:12.850
So, by way of introducing this topic, I want

0:01:12.850,0:01:16.130
to tell you a story. And it's a story

0:01:16.130,0:01:21.840
of when deployments were driving me crazy.[br]You know,

0:01:21.840,0:01:24.530
kind of like, tear my hair out crazy.

0:01:24.530,0:01:27.390
We had an app, and the app consisted, you

0:01:27.390,0:01:29.280
know, it was pretty straightforward for a[br]modern app.

0:01:29.280,0:01:30.510
And it was a Rails app that had a

0:01:30.510,0:01:33.510
home page. It had your terms and conditions[br]page.

0:01:33.510,0:01:35.570
You can't have a site without that. It had

0:01:35.570,0:01:38.450
a Javascript app, which in this case was an

0:01:38.450,0:01:40.660
Ember app, but, you know, you can substitute[br]any

0:01:40.660,0:01:44.280
kind of rich Javascript MVC app that you,[br]style

0:01:44.280,0:01:46.680
you'd like, for the purposes of this talk.[br]I

0:01:46.680,0:01:49.450
then, of course, had a JSON API.

0:01:49.450,0:01:50.960
And so these, this was kind of the bullet

0:01:50.960,0:01:53.870
points, but in terms of the amount of code,

0:01:53.870,0:01:55.280
the complexity, and how much time that it[br]took

0:01:55.280,0:01:57.870
working on it, it was more like this, right.

0:01:57.870,0:02:00.090
We had a lot of work on the Javascript

0:02:00.090,0:02:02.409
app. Some, a bunch more in the JSON API.

0:02:02.409,0:02:04.440
And the rest of the site was, you know,

0:02:04.440,0:02:07.200
pretty trivial.

0:02:07.200,0:02:09.470
But in terms of deployments, every time we[br]wanted

0:02:09.470,0:02:11.840
to, every time I made a change and wanted

0:02:11.840,0:02:15.340
to deploy it, we would package everything[br]up and

0:02:15.340,0:02:18.430
deploy it. And so I have a question for

0:02:18.430,0:02:21.260
you folks. Hopefully everybody's, in the room[br]has worked

0:02:21.260,0:02:22.930
with Rails. How long does it take to deploy

0:02:22.930,0:02:24.930
a Rails app? We're gonna do a show of

0:02:24.930,0:02:26.950
hands, and by the end, I hope everybody will

0:02:26.950,0:02:28.959
have their Rails app, sorry, will have their[br]hands

0:02:28.959,0:02:29.800
up.

0:02:29.800,0:02:32.480
And so, please start by raising your hand[br]if

0:02:32.480,0:02:34.730
your Rails app is deployed in less than thirty

0:02:34.730,0:02:37.890
seconds. OK. Good awesome. One, I want to[br]talk

0:02:37.890,0:02:42.000
with you later. How about less than one minute?

0:02:42.000,0:02:47.000
Cool. A few folks. Less than three minutes?[br]A

0:02:47.000,0:02:50.520
bunch more. Less than five minutes? Keep,[br]keep your

0:02:50.520,0:02:51.730
hands up even if you were in the early

0:02:51.730,0:02:53.060
group. Less than five minutes.

0:02:53.060,0:02:56.069
OK, so we're probably at a majority now. Less

0:02:56.069,0:03:01.750
than ten minutes? Keep your hands up. OK.[br]And

0:03:01.750,0:03:03.569
less than twenty minutes? I hope that's everybody[br]in

0:03:03.569,0:03:06.580
the room. Please, please, mercy. OK. Cool.

0:03:06.580,0:03:09.650
So, the, I think the, the answer is, it

0:03:09.650,0:03:11.180
takes at least a few minutes to deploy a

0:03:11.180,0:03:14.030
Rails app, unless you're one of an exceptional[br]few

0:03:14.030,0:03:16.920
folks in the audience. And I get it. There's

0:03:16.920,0:03:19.530
a lot, you know, there's files to transfer.[br]There's

0:03:19.530,0:03:22.980
dependencies to install. Most modern Rails[br]apps, you know,

0:03:22.980,0:03:25.010
that I run into, that we create, have a

0:03:25.010,0:03:28.150
lot of gem dependencies. It takes some time[br]to

0:03:28.150,0:03:30.520
boot the app with all those dependencies and[br]just

0:03:30.520,0:03:33.090
with, just with your app code.

0:03:33.090,0:03:35.620
And so, that's fine, except for, I was going

0:03:35.620,0:03:38.760
days just working on the Javascript app. Right,[br]I

0:03:38.760,0:03:41.950
was just making changes in Javascript, and[br]every time

0:03:41.950,0:03:44.770
I wanted to deploy, I was waiting five minutes

0:03:44.770,0:03:50.540
in our case to just deploy static Javascript[br]changes.

0:03:50.540,0:03:52.050
And it really made me want to throw something

0:03:52.050,0:03:53.959
across the room. Why was I doing this to

0:03:53.959,0:03:55.030
myself?

0:03:55.030,0:03:58.510
And it wasn't just me that I was annoying.

0:03:58.510,0:04:02.430
I was also annoying our users, because, in[br]most,

0:04:02.430,0:04:06.680
in most Rails deployment scenarios, there[br]are some, there

0:04:06.680,0:04:09.200
can be some hiccups in each deploy. And we'll,

0:04:09.200,0:04:11.230
so let's talk a little bit about this, this

0:04:11.230,0:04:16.339
kind of hiccups and deployments and zero downtime[br]deploys.

0:04:16.339,0:04:19.060
So if your Rails app takes several seconds[br]to

0:04:19.060,0:04:22.449
boot, which is probably about average, obviously[br]it can't

0:04:22.449,0:04:25.830
serve requests during that time. And so, under[br]high

0:04:25.830,0:04:28.650
load in most architectures, most requesters[br]are just gonna

0:04:28.650,0:04:30.620
be queued, waiting for the app to be ready

0:04:30.620,0:04:34.090
to handle requests. And then once it boots[br]up,

0:04:34.090,0:04:37.139
it's gonna start handling those requests,[br]and eventually flush

0:04:37.139,0:04:41.039
that queue and hopefully catch up to the requests

0:04:41.039,0:04:43.210
as they're coming in.

0:04:43.210,0:04:45.919
And so users that are hitting the, hitting[br]your

0:04:45.919,0:04:49.139
app during this time may experience at, at[br]best

0:04:49.139,0:04:51.289
case just a couple of seconds of downtime.[br]At

0:04:51.289,0:04:52.889
worst case, kind of a feeling like that, that

0:04:52.889,0:04:55.330
this site is not responsive.

0:04:55.330,0:04:57.520
And so it disappoints me that we don't yet

0:04:57.520,0:05:01.800
have a kind of conventional solution for zero[br]downtime

0:05:01.800,0:05:03.990
deploys. But it kind of makes sense because,[br]by

0:05:03.990,0:05:08.490
definition, Rails runs inside of other web[br]servers, and

0:05:08.490,0:05:10.550
so, and that, this is really a concern kind

0:05:10.550,0:05:11.889
of at that web server layer.

0:05:11.889,0:05:15.300
So, Heroku, for example, has an experimental[br]solution. Heroku

0:05:15.300,0:05:20.330
Labs is, Heroku's kind of unsupported experimental[br]area features.

0:05:20.330,0:05:23.539
And you can run heroku labs:enable preboot,[br]which will

0:05:23.539,0:05:25.830
start up new servers or dynos with your new

0:05:25.830,0:05:28.669
code, wait three minutes to give your Rails[br]app

0:05:28.669,0:05:30.979
plenty of time to boot, and then switch traffic

0:05:30.979,0:05:32.650
over.

0:05:32.650,0:05:36.680
For, if you're using Puma or Unicorn, there[br]are

0:05:36.680,0:05:39.419
facilities to start one worker at a time,[br]or

0:05:39.419,0:05:42.169
groups of workers by sending signals to the[br]master

0:05:42.169,0:05:45.020
process. HAProxy is a tool that I've used[br]in

0:05:45.020,0:05:48.229
the past to kind of split traffic, give yourself

0:05:48.229,0:05:52.389
time to boot up another, another set of servers.

0:05:52.389,0:05:54.499
HAProxy is nice because you can do health[br]checks

0:05:54.499,0:05:57.150
against those new servers and say, am I ready

0:05:57.150,0:06:02.289
to deploy? And Passenger also has some solutions[br]around

0:06:02.289,0:06:04.589
this.

0:06:04.589,0:06:06.339
In terms of kind of the full scope of

0:06:06.339,0:06:08.619
zero downtime stuff, it gets more complicated[br]when you

0:06:08.619,0:06:12.569
talk about database migrations and what's[br]safe and what's

0:06:12.569,0:06:16.430
not safe for these, these types of, these[br]types

0:06:16.430,0:06:18.949
of deployments. And I'm happy to chat about[br]that

0:06:18.949,0:06:21.039
with anybody later, but that's out of scope[br]for,

0:06:21.039,0:06:22.729
for this particular talk.

0:06:22.729,0:06:24.360
What I do want to drill into a little

0:06:24.360,0:06:28.080
bit is issues with static assets and zero[br]downtime

0:06:28.080,0:06:30.219
deploys, because that's the thing that was,[br]you know,

0:06:30.219,0:06:31.589
kind of at the heart of what I was

0:06:31.589,0:06:35.099
doing, was really dealing with these static[br]Javascript assets.

0:06:35.099,0:06:37.539
And I think that these issues aren't discussed[br]often.

0:06:37.539,0:06:38.839
So I kind of want to drill down, kind

0:06:38.839,0:06:41.479
of at a detailed level, and talk about them

0:06:41.479,0:06:43.419
here.

0:06:43.419,0:06:46.279
So when a browser makes an initial request[br]to

0:06:46.279,0:06:49.479
your server for, to load your, your rich Javascript

0:06:49.479,0:06:52.099
app, it's loading the index dot html file[br]and

0:06:52.099,0:06:54.899
by index.html, I'm gonna refer to the html[br]file

0:06:54.899,0:06:58.089
that's the bootstrapping, it's bootstrapping[br]your Javascript app. It's

0:06:58.089,0:07:00.069
the thing that has a little bit of code

0:07:00.069,0:07:01.629
to fire things up and it pulls in the

0:07:01.629,0:07:03.839
right Javascript and CSS assets.

0:07:03.839,0:07:06.189
So the request comes into your servers, your[br]server

0:07:06.189,0:07:09.679
responds with the HTML file, with the text[br]slash

0:07:09.679,0:07:13.580
html mine type, and typically the, your asset[br]files,

0:07:13.580,0:07:15.729
the Javascript and CSS that are gonna be referenced

0:07:15.729,0:07:19.479
here, are gonna be fingerprinted. Right, so[br]we do,

0:07:19.479,0:07:21.699
take a hash of the contents of the Javascript

0:07:21.699,0:07:24.909
file, we set it as a, we include that

0:07:24.909,0:07:27.409
hash in the file name, and we're then are

0:07:27.409,0:07:30.149
able to set far features expires headers on[br]those

0:07:30.149,0:07:32.520
files, so that when the file changes, we don't

0:07:32.520,0:07:35.809
have to worry about cache expire or anything.[br]We

0:07:35.809,0:07:38.749
just gotta new file that's gonna come through[br]as

0:07:38.749,0:07:41.179
if the browser's never seen it before.

0:07:41.179,0:07:43.599
And so in this case, our html file might

0:07:43.599,0:07:47.599
contain something like assets slash app dash[br]abc123 dot

0:07:47.599,0:07:51.779
js, where abc123 is this fingerprint we're[br]talking about.

0:07:51.779,0:07:54.830
And so the browser takes that html, parses[br]the

0:07:54.830,0:07:57.469
page, a short time later makes the request[br]for

0:07:57.469,0:08:01.550
app dash abc123. Server says, here you go,[br]some

0:08:01.550,0:08:05.330
text Javascript. Browser parses that, boots[br]up the app.

0:08:05.330,0:08:06.369
All is well.

0:08:06.369,0:08:08.849
Hopefully this is very clear to everybody[br]who's in

0:08:08.849,0:08:12.029
the room. What's maybe less clear, unless[br]you've thought

0:08:12.029,0:08:14.740
about it in detail is that during deployments,[br]this

0:08:14.740,0:08:17.300
idea can break down a little bit. And so

0:08:17.300,0:08:20.459
imagine that we've got our deployment and[br]we've got

0:08:20.459,0:08:23.319
two kind of sets of server. The top set

0:08:23.319,0:08:25.300
that we're looking at here on the screen is

0:08:25.300,0:08:27.749
the existing code. The bottom set is the new

0:08:27.749,0:08:29.869
code that you're deploying.

0:08:29.869,0:08:31.020
And in this case, there's a change to the

0:08:31.020,0:08:35.080
Javascript file, so there's the new fingerprinted[br]filename there.

0:08:35.080,0:08:38.120
So when our initial request for our page comes

0:08:38.120,0:08:40.830
in, it was, it will go to the old

0:08:40.830,0:08:43.020
code, because we haven't yet switched traffic[br]over to

0:08:43.020,0:08:46.890
the, to the new deploy. And just like in

0:08:46.890,0:08:49.209
our first example, it's gonna come back with[br]the

0:08:49.209,0:08:54.220
index file references app dash abc123 dot[br]js.

0:08:54.220,0:08:56.490
Now, what if, in that moment, where, as the

0:08:56.490,0:08:59.710
page is being parsed, before this request[br]comes back,

0:08:59.710,0:09:02.000
we then switch traffic over to pointing to[br]the

0:09:02.000,0:09:05.510
new server? Well, request is gonna come in[br]for

0:09:05.510,0:09:09.060
app dash abc123 dot js, the new server gets

0:09:09.060,0:09:11.640
the request and says, ah, I don't know what

0:09:11.640,0:09:13.950
you're talking about. I don't have a Javascript[br]file

0:09:13.950,0:09:18.230
with that name. And so it says, 404 Not

0:09:18.230,0:09:18.900
Found.

0:09:18.900,0:09:24.130
And this is a challenging problem to, to address,

0:09:24.130,0:09:26.240
because there's a, because of a few reasons.[br]One

0:09:26.240,0:09:28.280
is that most simply if I, at this point,

0:09:28.280,0:09:30.900
now hit refresh in the browser, of course[br]everything

0:09:30.900,0:09:33.460
works fine. Right, because now both of those[br]requests

0:09:33.460,0:09:35.380
are going to the new server and the world

0:09:35.380,0:09:36.200
is good.

0:09:36.200,0:09:39.640
It can be further kind of shadowed, this issue,

0:09:39.640,0:09:43.090
because, if you are serving your assets up[br]through

0:09:43.090,0:09:46.000
an assets host at CBN, you might have some,

0:09:46.000,0:09:48.730
some, some of their, your edge nodes might[br]have

0:09:48.730,0:09:51.510
that old page cached. That old Javascript[br]cached. In

0:09:51.510,0:09:54.450
which case, those nodes will return it just[br]fine.

0:09:54.450,0:09:57.850
And so, to, to know that you're, you know,

0:09:57.850,0:10:00.910
totally impervious to this, you know, might[br]be a

0:10:00.910,0:10:02.620
little bit fuzzy, and also to be able to

0:10:02.620,0:10:05.350
reproduce it reliably is a challenging thing[br]to do.

0:10:05.350,0:10:08.320
But, in short, to avoid these hiccups, both[br]the

0:10:08.320,0:10:10.660
old versions of your assets and the new versions

0:10:10.660,0:10:12.420
have to be available for at least a few

0:10:12.420,0:10:15.670
minutes during your deployment in order to,[br]to make

0:10:15.670,0:10:19.450
this zero downtime approach really work well[br]on the

0:10:19.450,0:10:21.730
static asset front.

0:10:21.730,0:10:23.330
And so this was one of the things I

0:10:23.330,0:10:25.670
was thinking about during these many five[br]minute deploys,

0:10:25.670,0:10:29.540
where I was, you know, wishing that I had

0:10:29.540,0:10:32.340
a solution. And so, in thinking about that,[br]I

0:10:32.340,0:10:34.040
said, well, we could figure out how to do

0:10:34.040,0:10:35.810
this on our app servers. Kind of keep the

0:10:35.810,0:10:38.830
old versions of the, of the Javascript and[br]the

0:10:38.830,0:10:40.610
new versions together. Or we could move the[br]assets

0:10:40.610,0:10:43.220
elsewhere. And the idea of moving the assets[br]elsewhere

0:10:43.220,0:10:45.780
really appealed to me, because that meant,[br]if they

0:10:45.780,0:10:47.870
weren't on the Rails, my Rails servers, then[br]maybe

0:10:47.870,0:10:50.060
I could avoid doing Rails deploys when I just

0:10:50.060,0:10:52.580
had static asset changes.

0:10:52.580,0:10:55.140
So I started to sketch out an idea. We've

0:10:55.140,0:10:56.580
got our Rails server at the top, and we

0:10:56.580,0:10:59.220
had this separate static asset server at the[br]bottom.

0:10:59.220,0:11:01.590
Let's deploy our Rails app code to the Rails

0:11:01.590,0:11:05.860
servers, our Javascript, CSS, and images to[br]these static

0:11:05.860,0:11:08.800
assets servers, and then we would deploy our[br]index

0:11:08.800,0:11:13.070
file. Where would we deploy our index file?

0:11:13.070,0:11:14.880
And so, I started to think about, well, what

0:11:14.880,0:11:16.000
is the index file? It is kind of this

0:11:16.000,0:11:18.240
thing that bridges the two? What are the requirements

0:11:18.240,0:11:20.490
around it? What do we know about it? And

0:11:20.490,0:11:24.070
this index, the html file points to fingerprinted[br]Javascript

0:11:24.070,0:11:27.250
and CSS, but it's not fingerprinted itself.[br]That's obviously

0:11:27.250,0:11:30.600
important, because it needs to be at, at a

0:11:30.600,0:11:34.530
consistent location for browsers to locate,[br]to load. It

0:11:34.530,0:11:38.260
contains Javascript urls and code to boot[br]the Javascript

0:11:38.260,0:11:40.470
app to load CSS and such in the right

0:11:40.470,0:11:44.080
order. It's a good place to provide environment-specific[br]configuration

0:11:44.080,0:11:46.960
to Javascript. Maybe you have some differences[br]between dev

0:11:46.960,0:11:49.520
and stage in production.

0:11:49.520,0:11:51.150
One thing that I knew was key, because I

0:11:51.150,0:11:53.250
had struggled with it, is that when you serve

0:11:53.250,0:11:55.480
this html off of the same domain as your

0:11:55.480,0:11:59.090
API, life is way simpler with respect to cores

0:11:59.090,0:12:03.910
and cross origin security issues. And finally,[br]if I

0:12:03.910,0:12:06.140
wanted, if you wanted to be able to deploy

0:12:06.140,0:12:08.920
changes quickly to your users, caching on[br]this particular

0:12:08.920,0:12:11.960
page should be minimal to none, so that you

0:12:11.960,0:12:14.810
can pretty instantly switch over.

0:12:14.810,0:12:17.920
So my conclusion, from thinking about this,[br]is that

0:12:17.920,0:12:20.190
the html page ideally should be managed and[br]thought

0:12:20.190,0:12:23.290
about as part of your static asset deployment[br]process,

0:12:23.290,0:12:25.350
but it should be served off of your Rails

0:12:25.350,0:12:26.240
server.

0:12:26.240,0:12:28.720
And, as importantly, it should be served off[br]your

0:12:28.720,0:12:33.210
Rails server without requiring a Rails reboot[br]or redeploying

0:12:33.210,0:12:36.760
the entire Rails app. And so, so we were

0:12:36.760,0:12:39.000
able to start to refine this sketch and say,

0:12:39.000,0:12:41.620
OK, our Rails server's gonna be serving up[br]our

0:12:41.620,0:12:45.100
API requests, our, kind of, traditional, dynamic[br]Rails pages

0:12:45.100,0:12:47.160
as part of the html for a Javascript app,

0:12:47.160,0:12:49.490
and our static asset server's gonna be serving[br]up

0:12:49.490,0:12:51.860
the Javascript, CSS and images.

0:12:51.860,0:12:53.740
And so that means we need to somehow deploy

0:12:53.740,0:12:58.520
our html up to the Rails server without a

0:12:58.520,0:13:00.360
full, a full reboot. And so, how could we

0:13:00.360,0:13:01.070
do this?

0:13:01.070,0:13:03.400
Well, the most obvious thing to me was, well,

0:13:03.400,0:13:06.330
take a html file and put it on the

0:13:06.330,0:13:09.590
file system of each Rails server. And this[br]has

0:13:09.590,0:13:11.610
a few things that aren't great about it. You

0:13:11.610,0:13:15.310
can probably make this work in some configurations.[br]In

0:13:15.310,0:13:19.370
many deployment environments, disk is ephemeral,[br]and so relying

0:13:19.370,0:13:22.290
on, you know, on copying some things up might

0:13:22.290,0:13:24.120
not be a great idea. It's also a little

0:13:24.120,0:13:27.440
bit weird to mix assets, files deployed from[br]a

0:13:27.440,0:13:31.030
particular gitshaw, with files deployed from[br]somewhere else, kind

0:13:31.030,0:13:33.700
of in the same file system.

0:13:33.700,0:13:35.450
And so, we said, well, what if there's something

0:13:35.450,0:13:37.780
that we could all see and talk to? Well,

0:13:37.780,0:13:40.900
what about uploading to S3? So then all the

0:13:40.900,0:13:44.860
Rails servers can, can see S3, read from it,

0:13:44.860,0:13:47.010
be able to serve up that html. And this

0:13:47.010,0:13:48.930
could kind of work, but reading from S3 is

0:13:48.930,0:13:52.310
a little bit slow. And we wanted this page

0:13:52.310,0:13:55.790
to be fast, obviously. No Javascript or CSS[br]is

0:13:55.790,0:13:57.930
gonna start being loaded until this page is[br]loaded

0:13:57.930,0:13:59.790
in the browser. And so the fastest we could,

0:13:59.790,0:14:01.060
the faster we could get this page to the

0:14:01.060,0:14:03.370
user, the better.

0:14:03.370,0:14:05.620
And so, then we said, well, what about redis?

0:14:05.620,0:14:09.560
Redis is persistent. It's fast. For us, it[br]was

0:14:09.560,0:14:13.190
already in our environment. We liked this[br]idea a

0:14:13.190,0:14:16.000
lot. We decided to, to dig in. This is

0:14:16.000,0:14:17.750
not the, as you'll see, this is not the

0:14:17.750,0:14:21.029
only way to do it. It's totally possible to

0:14:21.029,0:14:23.750
user other systems besides redis. But redis[br]kind of

0:14:23.750,0:14:26.960
firt the bill for us and works quite well,

0:14:26.960,0:14:27.710
as you'll see.

0:14:27.710,0:14:29.880
So, the general idea was we're gonna deploy[br]into

0:14:29.880,0:14:32.490
redis and then serve out of redis via a

0:14:32.490,0:14:35.960
Rails controller. So here's the simplest possible[br]kind of

0:14:35.960,0:14:39.470
deploy code that we had. It's a rig task,

0:14:39.470,0:14:42.760
and we're going to generate our html for the

0:14:42.760,0:14:46.380
current assets, and that's, that's kind of[br]an exercise

0:14:46.380,0:14:48.110
for your build tooling, which we'll talk about[br]a

0:14:48.110,0:14:50.440
little bit later. And then once we had this

0:14:50.440,0:14:52.090
html, we're actually going to set it as a,

0:14:52.090,0:14:56.029
at a, as a redis key-value store. So the

0:14:56.029,0:14:57.880
html is the value and the key would be

0:14:57.880,0:15:02.730
something well-known like jsapp colon index,[br]for example. And,

0:15:02.730,0:15:04.840
and this is a redis connection that's connecting[br]directly

0:15:04.840,0:15:07.779
to your charted deployment environment. So[br]that's staging more

0:15:07.779,0:15:10.440
production.

0:15:10.440,0:15:13.090
Once it's there in redis, our controller,[br]again, the

0:15:13.090,0:15:16.470
most simplest version, is get the value out[br]of

0:15:16.470,0:15:21.350
redis. Render text.html. Now, when I first[br]looked at,

0:15:21.350,0:15:22.910
looked at this code or wrote this code, I

0:15:22.910,0:15:24.410
said, is that gonna be served up with the

0:15:24.410,0:15:26.570
right mine type? Seems a little strange. And[br]it

0:15:26.570,0:15:29.920
turns out that, yes, if you do render text

0:15:29.920,0:15:33.070
and some string, Rails serves that up with[br]text

0:15:33.070,0:15:36.690
slash html, if you don't specify. So, it's[br]a

0:15:36.690,0:15:38.520
little, I think a little bit of a confusing

0:15:38.520,0:15:41.430
API, but it does what we want.

0:15:41.430,0:15:44.370
So, we can now continue to refine this approach.

0:15:44.370,0:15:46.370
We know we're deploying, when we need to deploy

0:15:46.370,0:15:49.160
Rails app code, we're doing a deployment to[br]our

0:15:49.160,0:15:52.370
Rails server. When we're deploying Javascript,[br]CSS and images,

0:15:52.370,0:15:54.870
we're deploying to the static assets server,[br]and then

0:15:54.870,0:15:58.160
we're deploying html by connecting to redis[br]and deploying

0:15:58.160,0:15:59.430
into it.

0:15:59.430,0:16:03.430
And we can make things a little bit nicer

0:16:03.430,0:16:06.520
by dropping cloud front right in front of,[br]by

0:16:06.520,0:16:08.810
using S3 as our static assets server, and[br]then

0:16:08.810,0:16:10.940
dropping cloud front instead of in front of[br]our,

0:16:10.940,0:16:13.350
in front of S3. So, for very little effort

0:16:13.350,0:16:15.620
and very little money, we've got now CBN distribution

0:16:15.620,0:16:18.170
for our static assets.

0:16:18.170,0:16:21.710
Now, there's a few things about this deployment[br]to

0:16:21.710,0:16:25.160
S3, in terms of making it fast. Getting a

0:16:25.160,0:16:28.720
file list from S3 can be somewhat expensive,[br]particularly

0:16:28.720,0:16:31.839
the more files that you have. And so the

0:16:31.839,0:16:34.440
approach that we took was to generate a manifest

0:16:34.440,0:16:38.620
file of our current assets and store the copy

0:16:38.620,0:16:42.000
of that manifest on S3 so we, we're basically

0:16:42.000,0:16:44.500
gonna read the remote manifest, compare it[br]to our

0:16:44.500,0:16:46.310
local manifest, and know we only need to deploy

0:16:46.310,0:16:48.520
what's different. And so this means that if[br]I

0:16:48.520,0:16:51.540
make one Javascript, one line of Javascript[br]change, it's

0:16:51.540,0:16:54.510
just the file that that's concatenated into[br]that needs

0:16:54.510,0:16:56.620
to be updated, and not all of my images

0:16:56.620,0:17:00.260
and CSS, as we're doing our deploy, our assets

0:17:00.260,0:17:02.110
deploy to S3.

0:17:02.110,0:17:05.260
Now, purging has been on our to-do list for

0:17:05.260,0:17:08.209
this architecture for quite a while, right.[br]After a

0:17:08.209,0:17:12.000
deploy is successfully completed, we can,[br]in theory, remove

0:17:12.000,0:17:15.039
stuff from S3. We never really got around[br]to

0:17:15.039,0:17:17.959
that. Mostly because it's so incredibly cheap[br]to store

0:17:17.959,0:17:20.730
these small files on S3, so, still on our

0:17:20.730,0:17:23.638
to-do list. I would probably not recommend[br]you prioritizing

0:17:23.638,0:17:26.839
it too high for yourself. The code for this

0:17:26.839,0:17:29.730
S3 sink is here at this link. I will

0:17:29.730,0:17:31.399
make these slides available. You don't have[br]to worry

0:17:31.399,0:17:34.100
about copying it down. This repo is open source

0:17:34.100,0:17:35.570
and contains a lot of the code that we're

0:17:35.570,0:17:37.860
looking at today. And it's the actual code[br]that

0:17:37.860,0:17:41.169
we use for our, for our production environments.

0:17:41.169,0:17:45.230
So, once you start thinking about this architecture,[br]it

0:17:45.230,0:17:47.799
paves the way to do something a little bit

0:17:47.799,0:17:50.119
more fundamental with your rich Javascript[br]app and your

0:17:50.119,0:17:53.179
Rails app, which is that you pull them, tease

0:17:53.179,0:17:56.179
them apart into separate repositories. And[br]now why would

0:17:56.179,0:17:57.340
you want to do that?

0:17:57.340,0:17:59.470
Well, one of the reasons is, you know, thinking

0:17:59.470,0:18:02.289
about tagging, do, you know, kind of tagging[br]a

0:18:02.289,0:18:04.840
deployed version of your code. Since you've[br]got these

0:18:04.840,0:18:08.360
independent deploy processes, it makes sense[br]to be able

0:18:08.360,0:18:11.230
to tag a Javascript deploy separate from a[br]Rails

0:18:11.230,0:18:13.559
deploy, because they really are now independent[br]of each

0:18:13.559,0:18:14.619
other.

0:18:14.619,0:18:17.679
And I find also that thinking of your Javascript

0:18:17.679,0:18:21.570
app as a separate, independent client of your[br]API,

0:18:21.570,0:18:22.850
works really well. Kind of puts it on the

0:18:22.850,0:18:25.249
same level as a native app, for example, maybe

0:18:25.249,0:18:26.769
if you've got an iPhone app that communicates[br]to

0:18:26.769,0:18:29.289
your API as well.

0:18:29.289,0:18:33.289
It also opens up the realm of possibility[br]to

0:18:33.289,0:18:35.029
having a lot of flexibility with what kind[br]of

0:18:35.029,0:18:37.200
build tools you want to use with your Javascript

0:18:37.200,0:18:40.899
app. You may choose to use sprockets in your

0:18:40.899,0:18:44.119
separate standalone repo. Or you may choose[br]to use

0:18:44.119,0:18:49.119
grunt, gulp, broccoli, brunch. You name it,[br]there's obviously

0:18:49.119,0:18:52.509
a lot of innovation and creativity happening[br]around build

0:18:52.509,0:18:55.619
tools in the Javascript environment. And my[br]guess would

0:18:55.619,0:18:59.369
be that you're, we're gonna see faster, you[br]know,

0:18:59.369,0:19:03.749
innovation, iteration in the Javascript environment[br]for building Javascript

0:19:03.749,0:19:06.019
apps than we will in the Ruby environment[br]for

0:19:06.019,0:19:07.340
building Javascript apps.

0:19:07.340,0:19:11.710
So, we've now got an approach that works pretty

0:19:11.710,0:19:14.700
well. But I think the, the question is, is

0:19:14.700,0:19:16.639
it worth doing the work to set this in

0:19:16.639,0:19:18.820
place? Like, how fast is this in the real

0:19:18.820,0:19:20.559
world? And so I took one of our apps,

0:19:20.559,0:19:22.980
and this isn't, was not a scientific bench[br]mark,

0:19:22.980,0:19:27.039
so consider it directional. And our builds[br]took about

0:19:27.039,0:19:29.909
six and a half seconds. This particular app[br]is

0:19:29.909,0:19:31.669
using Rake Pipeline as a build tool for the

0:19:31.669,0:19:37.799
Javascript side. Our transferring assets to[br]S3 using this

0:19:37.799,0:19:40.269
differential approach was about a second,[br]and then uploading

0:19:40.269,0:19:44.330
html into redis was about two and a half

0:19:44.330,0:19:47.649
seconds. And so, instead of a five minute[br]deploy,

0:19:47.649,0:19:50.289
I was now able, we were now able to

0:19:50.289,0:19:53.759
deploy this, our Javascript apps in under[br]ten seconds.

0:19:53.759,0:19:58.139
So just by that, that was a big win.

0:19:58.139,0:20:00.429
And stopped me from wanting to throw things[br]across

0:20:00.429,0:20:03.149
the office. But, I think that, you know, in

0:20:03.149,0:20:06.619
any kind of architectural choices like this,[br]you learn

0:20:06.619,0:20:09.200
if this is a good idea or not over

0:20:09.200,0:20:12.200
time, right, based on, how does, how does[br]this

0:20:12.200,0:20:15.039
architecture respond to changes. What kind[br]of possibilities does

0:20:15.039,0:20:16.909
it enable? So I want to talk a little

0:20:16.909,0:20:19.639
bit about the kind of emergent behavior that[br]we've

0:20:19.639,0:20:21.580
seen around, now that we've had this in production

0:20:21.580,0:20:23.580
for awhile.

0:20:23.580,0:20:26.239
The first thing is, the idea of preview. And

0:20:26.239,0:20:28.950
so this is an actual command line session[br]for

0:20:28.950,0:20:31.929
deploying an app. In this case, it's yapp-prefs,[br]which

0:20:31.929,0:20:36.259
is our, kind of, account settings app. We[br]first

0:20:36.259,0:20:40.779
run rake dist. This is the build. And with

0:20:40.779,0:20:42.580
the build completes, in this case, as we saw

0:20:42.580,0:20:44.289
in the pie chart before, in about six seconds

0:20:44.289,0:20:47.600
or so. And it says, OK, to deploy these

0:20:47.600,0:20:52.529
assets to S3, run rake deploy:assets with[br]this, what

0:20:52.529,0:20:55.070
we call a manifest idea, this b35b.

0:20:55.070,0:20:57.769
So what's a manifest id? We talked earlier[br]about

0:20:57.769,0:21:00.820
fingerprinting assets and we talked about[br]creating a manifest

0:21:00.820,0:21:03.289
file. So what we do also is we fingerprint

0:21:03.289,0:21:05.129
the manifest file. So we take a hash of

0:21:05.129,0:21:07.570
the contents of that manifest file and we[br]say,

0:21:07.570,0:21:10.210
OK, that is the manifest id for this deploy.

0:21:10.210,0:21:12.249
And that's, it's, it's kind of a unique identifier

0:21:12.249,0:21:15.399
as it's going through its unique deploy process.

0:21:15.399,0:21:17.489
And so we run rake deploy:assets, which does[br]the

0:21:17.489,0:21:20.580
differential upload to S3 and it's gonna show[br]us

0:21:20.580,0:21:23.029
what it uploads. It's gonna spit out, OK,[br]I've

0:21:23.029,0:21:28.899
uploaded these four files. JS, CSS, two yaml[br]files

0:21:28.899,0:21:31.149
for the manifest. We, actually, these are[br]two copies

0:21:31.149,0:21:33.070
of the same thing. One is a, has a

0:21:33.070,0:21:35.139
file name with the id, and one just is,

0:21:35.139,0:21:38.470
hey I am the latest. And it's going to

0:21:38.470,0:21:40.989
then tell us the next command to run is

0:21:40.989,0:21:45.659
deploy:generate_index for this manifest id.[br]And what this is

0:21:45.659,0:21:49.049
gonna do is going to connect to redis and

0:21:49.049,0:21:51.509
set this at a key named for the manifest

0:21:51.509,0:21:54.369
id. So in the previous simplistic example[br]we looked

0:21:54.369,0:21:57.289
at, it was just updating jsapp index. Now[br]it's

0:21:57.289,0:21:59.779
updating a key at prefs, in this case, prefs:index

0:21:59.779,0:22:04.039
b35b something, you know, named for the manifest[br]id.

0:22:04.039,0:22:06.989
And why is this awesome? Well, this command[br]line

0:22:06.989,0:22:10.429
tool can now tell us, hey, to preview this,

0:22:10.429,0:22:12.610
this asset change, go ahead and take a look

0:22:12.610,0:22:16.509
at your at, your site with the query param

0:22:16.509,0:22:21.629
manifest id equals b35 et cetera. And what[br]this

0:22:21.629,0:22:24.110
is gonna do is it's going to pull the

0:22:24.110,0:22:27.080
new html file from redis. Gonna show, which[br]is

0:22:27.080,0:22:29.899
loading up the new Javascript. It's connecting[br]to the

0:22:29.899,0:22:33.190
production API. So you're able to smoke test[br]this

0:22:33.190,0:22:36.039
in production. Everything is working just[br]as the user

0:22:36.039,0:22:38.659
will see it, except for your users don't see

0:22:38.659,0:22:40.879
it yet. So if you screwed something up, you've

0:22:40.879,0:22:42.580
got a chance before kind of pulling the trigger

0:22:42.580,0:22:44.889
and switching it life to go.

0:22:44.889,0:22:47.720
And then, finally, one more command to kind[br]of

0:22:47.720,0:22:49.779
active that redis key and make it the current

0:22:49.779,0:22:53.659
key. And so what does this code look like?

0:22:53.659,0:22:55.519
It's actually not that much more complicated[br]than what

0:22:55.519,0:22:59.200
we saw before. We invoke our rake task with

0:22:59.200,0:23:04.730
the manifest id. Generate our html file. And[br]then,

0:23:04.730,0:23:07.549
instead of setting jsapp index, we'll set[br]a jsapp

0:23:07.549,0:23:09.529
key based on the manifest name, or redis key

0:23:09.529,0:23:13.299
based on the manifest name. And then spit[br]out

0:23:13.299,0:23:15.700
something to give the developer the url to[br]take

0:23:15.700,0:23:20.480
a look at, to preview, to preview the app.

0:23:20.480,0:23:23.409
On the server-side, we're gonna add one more[br]redis

0:23:23.409,0:23:27.529
request to the mix. If there's a manifest[br]id

0:23:27.529,0:23:30.899
param, then we'll just use that. If it's blank,

0:23:30.899,0:23:33.269
then we'll go and we'll connect to a current

0:23:33.269,0:23:35.509
key. Grab the manifest id, then use that to

0:23:35.509,0:23:40.070
serve up the current version of, of the site.

0:23:40.070,0:23:41.659
Of the, of this index file.

0:23:41.659,0:23:46.119
And so, that has been pretty powerful, and[br]it's,

0:23:46.119,0:23:48.320
it's a super useful tool that we use in

0:23:48.320,0:23:51.289
almost every single deploy. The developer's[br]just gonna do

0:23:51.289,0:23:53.789
a quick smoke test and say, yes, everything[br]looks

0:23:53.789,0:23:56.429
good, before they flip the switch.

0:23:56.429,0:24:00.999
The next interesting aspect that this kind[br]of enabled

0:24:00.999,0:24:05.109
is around dynamic html rewriting. And so what[br]we

0:24:05.109,0:24:08.749
realize is that as html was passing through[br]from

0:24:08.749,0:24:11.950
redis through the controller back to the browser,[br]we

0:24:11.950,0:24:14.899
had the opportunity to make adjustments if[br]we wanted

0:24:14.899,0:24:18.929
to. And one category of adjustment that we[br]ended

0:24:18.929,0:24:22.259
up making, as you see in this example, is

0:24:22.259,0:24:24.889
injecting some information about the current[br]user.

0:24:24.889,0:24:27.960
Now, obviously in a Rails controller we know[br]typically

0:24:27.960,0:24:31.409
who the current user is. Most apps will have

0:24:31.409,0:24:35.100
a current_user method available to any controller[br]that they

0:24:35.100,0:24:38.590
can, it can grab. In contrast, when you're[br]booting

0:24:38.590,0:24:41.950
up a Javascript MVC app, at that point, you

0:24:41.950,0:24:44.159
know, most apps don't know who the current[br]user

0:24:44.159,0:24:46.320
is. And if, in most, you know mostly there

0:24:46.320,0:24:49.330
is some XHR request that's involved in figuring[br]out,

0:24:49.330,0:24:51.499
is this user logged in and, if so, what

0:24:51.499,0:24:53.970
is their role in the system? And during that

0:24:53.970,0:24:56.869
time, the user is sitting there waiting, right.[br]The

0:24:56.869,0:24:58.999
Javascript is just kind of, maybe it's rendering[br]a

0:24:58.999,0:25:01.889
loading spinner for the user or something.[br]But it's

0:25:01.889,0:25:04.489
a little bit annoying and it also makes the

0:25:04.489,0:25:07.799
boot process for the, for your Javascript[br]app more

0:25:07.799,0:25:08.179
complicated.

0:25:08.179,0:25:11.200
So we said, well what if the app could

0:25:11.200,0:25:15.129
have, at boot time, have access to this information?

0:25:15.129,0:25:17.600
So what we're doing here is, in the controller

0:25:17.600,0:25:20.509
action at the top, between the time we get

0:25:20.509,0:25:24.519
the html out of redis and return, render it,

0:25:24.519,0:25:27.669
we're going to inject current_user information.[br]We're gonna grab

0:25:27.669,0:25:31.919
the current_user and then user our AcitveModel[br]serializer to

0:25:31.919,0:25:35.249
convert it to JSON, escape it, and stick it

0:25:35.249,0:25:37.159
into the head tag.

0:25:37.159,0:25:39.690
And you'll notice that the method that we're[br]using

0:25:39.690,0:25:42.720
to add this to the html, you might find

0:25:42.720,0:25:46.419
a little crude. And certainly when I first[br]started

0:25:46.419,0:25:48.720
doing this, I said, OK, we'll use Nokogiri.[br]We'll

0:25:48.720,0:25:51.799
parse the html. And then we'll insert, you[br]know,

0:25:51.799,0:25:55.399
insert a node, and then we'll render, you[br]know,

0:25:55.399,0:25:58.279
convert that back to text. And it turns out

0:25:58.279,0:26:00.889
that really what we're doing is so simple[br]that,

0:26:00.889,0:26:02.929
as fast as Nokogiri is, and it's pretty fast

0:26:02.929,0:26:05.820
for an XML or html parser, it is not

0:26:05.820,0:26:08.759
faster than string manipulation and string[br]indexing.

0:26:08.759,0:26:10.830
And so, in this case, we're just looking at,

0:26:10.830,0:26:12.059
where's the end of the head tag or the

0:26:12.059,0:26:15.309
beginning of the head tag, inject this meta[br]tag

0:26:15.309,0:26:18.350
in there. And this works great.

0:26:18.350,0:26:20.739
The, some other use cases for this same approach

0:26:20.739,0:26:24.950
might be injecting csrf tokens if you, you[br]know,

0:26:24.950,0:26:27.669
need to interact with Rails forms from, from[br]your

0:26:27.669,0:26:32.309
Javascript app. Including dynamic analytics[br]params, we had a

0:26:32.309,0:26:36.210
case where the, the Javascript app was kind[br]of

0:26:36.210,0:26:39.129
the, the, a goal page for a Google analytics

0:26:39.129,0:26:41.960
kind of flow. And we needed to set some,

0:26:41.960,0:26:45.220
certain Javascript only in some conditions.[br]So this was

0:26:45.220,0:26:46.499
a nice way to do that.

0:26:46.499,0:26:49.049
If you're using feature flags through something[br]like the

0:26:49.049,0:26:53.980
excellent rollout gem, it's great that that's[br]available throughout

0:26:53.980,0:26:55.450
Rails. But how do you make it available inside

0:26:55.450,0:26:57.789
of your Javascript app also, this is a nice

0:26:57.789,0:27:00.389
solution to be able to kind of inject variables

0:27:00.389,0:27:03.799
along those lines.

0:27:03.799,0:27:07.039
Another pretty awesome thing that we've been[br]able to

0:27:07.039,0:27:09.529
do using this approach is doing A/B testing[br]within

0:27:09.529,0:27:13.639
our Javascript app. And we've experimented[br]with two different

0:27:13.639,0:27:17.220
kinds. One is, kind of, setting some flags[br]based

0:27:17.220,0:27:19.840
on which bucket the user ends up in, A

0:27:19.840,0:27:23.359
or B. This is pretty similar to what we

0:27:23.359,0:27:26.950
just described. And then the, the second is[br]serving

0:27:26.950,0:27:29.600
up wholly different html based on the A/B[br]bucket.

0:27:29.600,0:27:33.249
We'll, I'll talk, take you through each of[br]those.

0:27:33.249,0:27:36.859
So this is using the split gem, which, if

0:27:36.859,0:27:39.210
you haven't seen, is a great A/B tool, A/B

0:27:39.210,0:27:42.419
test tool for Ruby frameworks, web frameworks.[br]It has

0:27:42.419,0:27:45.590
a Rails integration that gives us the A/B[br]test

0:27:45.590,0:27:48.619
method, which you see on line eleven there.[br]And

0:27:48.619,0:27:52.470
so we're doing some injection into our html[br]where

0:27:52.470,0:27:54.649
we're saying, if the user is part of the

0:27:54.649,0:27:58.840
show walkthrough experiment, then we're going[br]to inject the

0:27:58.840,0:28:01.720
script tag that just sets a global variable.[br]And

0:28:01.720,0:28:04.429
then our Javascript app as it boots in is

0:28:04.429,0:28:06.229
running, is, is, can consult that global variable[br]to

0:28:06.229,0:28:08.989
decide whether to show the walkthrough that[br]you're doing,

0:28:08.989,0:28:10.090
doing some testing on.

0:28:10.090,0:28:13.759
And then later, elsewhere in your app, you[br]would

0:28:13.759,0:28:16.830
indicate that, that the goal was achieved,[br]that the

0:28:16.830,0:28:19.720
user signed up or published or whatever it[br]is

0:28:19.720,0:28:22.950
that was kind of the, the goal for you

0:28:22.950,0:28:25.830
A/B test. And so this was great. This is

0:28:25.830,0:28:28.249
excellent for the kinds of A/B tests where[br]both

0:28:28.249,0:28:32.830
paths are supported by your, by a given incarnation

0:28:32.830,0:28:34.700
of your Javascript app.

0:28:34.700,0:28:38.649
We had another case, though, where, this was[br]right

0:28:38.649,0:28:41.259
around the time that iOS7 released and there[br]was,

0:28:41.259,0:28:43.289
if you recall, there was a lot of excitement

0:28:43.289,0:28:46.389
in, around flat design. And so we did a

0:28:46.389,0:28:49.139
redesign of our Javascript app. We weren't[br]immune to

0:28:49.139,0:28:54.559
the hype. And, but as we got near completion,

0:28:54.559,0:28:56.789
we started to wonder, well, we're gonna release[br]this.

0:28:56.789,0:28:58.009
How do we know that this is any better

0:28:58.009,0:29:00.809
than our existing site? Is this going to improve

0:29:00.809,0:29:04.119
our metrics or hurt them? And we wanted to

0:29:04.119,0:29:06.129
try to get some confidence one way or the

0:29:06.129,0:29:06.769
other.

0:29:06.769,0:29:08.929
So we said, well, we've been developing this[br]in

0:29:08.929,0:29:10.950
a branch, and we already know that we can

0:29:10.950,0:29:13.669
preview different versions of our Javascript[br]app, because we

0:29:13.669,0:29:16.090
had this preview mechanism in place. Could[br]we use

0:29:16.090,0:29:18.350
this for A/B testing, also?

0:29:18.350,0:29:21.249
And so, it turns out that we were able

0:29:21.249,0:29:23.840
to just update our deploy scripts to be able

0:29:23.840,0:29:27.450
to deploy from a branch into an experiment[br]kind

0:29:27.450,0:29:30.129
of redis key, and then use the same A/B

0:29:30.129,0:29:33.929
test mechanism to determine, for the new design[br]experiment,

0:29:33.929,0:29:36.190
should the user use the current manifest or[br]the

0:29:36.190,0:29:38.989
experimental manifest?

0:29:38.989,0:29:41.019
Once we made that decision, we then got the

0:29:41.019,0:29:45.440
appropriate manifest id from redis, got the[br]html, rendered

0:29:45.440,0:29:48.499
it, and users either saw our old app or

0:29:48.499,0:29:52.029
the new flat-design app. Turns out, flat design,[br]about

0:29:52.029,0:29:55.919
nine percent better. So good news.

0:29:55.919,0:29:58.259
So this is suitable for changes where your[br]development's

0:29:58.259,0:30:00.210
capping at a branch, and you want to A/B

0:30:00.210,0:30:03.070
between the branches. Not a common scenario,[br]but if

0:30:03.070,0:30:05.429
it's, when it's, it's useful, it was great[br]to

0:30:05.429,0:30:09.830
see how this approach supported that architecture.

0:30:09.830,0:30:11.960
One more possibility that I didn't cover here,[br]but

0:30:11.960,0:30:16.759
you can imagine how this might work, is around

0:30:16.759,0:30:20.239
doing rollback. So what if every time that[br]we

0:30:20.239,0:30:23.590
did this deploy and updated our html in redis

0:30:23.590,0:30:26.259
we pushed into a redis list and said, hey,

0:30:26.259,0:30:29.309
so and so user deployed such and such manifest

0:30:29.309,0:30:31.940
id at this time, and then we were able

0:30:31.940,0:30:33.979
to have a rake task that reads that list

0:30:33.979,0:30:37.299
and lets you rollback to any particular version.[br]Hopefully,

0:30:37.299,0:30:39.460
at this point, you can see how straightforward[br]that

0:30:39.460,0:30:42.200
would, that kind of thing would be as well.

0:30:42.200,0:30:44.039
So, with that, I want to say thank you

0:30:44.039,0:30:47.409
to my colleagues at Yapp Labs who helped create

0:30:47.409,0:30:50.769
this. Kris Seldon, Stefan Penner and Ray Cohen.[br]And

0:30:50.769,0:30:52.229
while we were working on this, we had heard

0:30:52.229,0:30:56.769
some rumors about some Square engineers doing[br]a, using

0:30:56.769,0:30:59.369
a similar approach at Square. So we took some

0:30:59.369,0:31:01.899
inspiration from those rumors as well, and[br]so thank

0:31:01.899,0:31:05.210
you, nameless Square engineers, or if anybody's[br]here. Love,

0:31:05.210,0:31:07.879
love to chat with you about it.

0:31:07.879,0:31:11.820
We've got some time for questions, and so[br]I

0:31:11.820,0:31:14.409
want to open it up to all of you.

0:31:14.409,0:31:15.789
Question in the middle?

0:31:15.789,0:31:17.009
All right, cool. Thank you all so much. Appreciate

0:31:17.009,0:31:19.429
it. Enjoy the rest of the conference.