LUKE MELIA: All right, folks.
Thanks for coming out. So you are at Lightning
Fast
Deployment of Your Rails-Baked Javascript
APP.
Hopefully you're in the right place.
My name is Luke. I, Luke Melia. I live
in Manhattan in New York City. Got a couple
little girls who are learning Ruby and Javascript
using
Code Academy and KidsRuby. And I have a company
called Yapp that I co-founded. We're one of
these
kind of hybrid product and consulting companies.
And when
I'm not doing Dad stuff or coding, I really
love to play beach volleyball and have recently
taken
up parkour.
So Yapp Labs is the consulting side of our
business. We do Ember.js consulting and training
based out
of New York and Seattle, if, if that's interesting,
happy to talk with you about that.
So, by way of introducing this topic, I want
to tell you a story. And it's a story
of when deployments were driving me crazy.
You know,
kind of like, tear my hair out crazy.
We had an app, and the app consisted, you
know, it was pretty straightforward for a
modern app.
And it was a Rails app that had a
home page. It had your terms and conditions
page.
You can't have a site without that. It had
a Javascript app, which in this case was an
Ember app, but, you know, you can substitute
any
kind of rich Javascript MVC app that you,
style
you'd like, for the purposes of this talk.
I
then, of course, had a JSON API.
And so these, this was kind of the bullet
points, but in terms of the amount of code,
the complexity, and how much time that it
took
working on it, it was more like this, right.
We had a lot of work on the Javascript
app. Some, a bunch more in the JSON API.
And the rest of the site was, you know,
pretty trivial.
But in terms of deployments, every time we
wanted
to, every time I made a change and wanted
to deploy it, we would package everything
up and
deploy it. And so I have a question for
you folks. Hopefully everybody's, in the room
has worked
with Rails. How long does it take to deploy
a Rails app? We're gonna do a show of
hands, and by the end, I hope everybody will
have their Rails app, sorry, will have their
hands
up.
And so, please start by raising your hand
if
your Rails app is deployed in less than thirty
seconds. OK. Good awesome. One, I want to
talk
with you later. How about less than one minute?
Cool. A few folks. Less than three minutes?
A
bunch more. Less than five minutes? Keep,
keep your
hands up even if you were in the early
group. Less than five minutes.
OK, so we're probably at a majority now. Less
than ten minutes? Keep your hands up. OK.
And
less than twenty minutes? I hope that's everybody
in
the room. Please, please, mercy. OK. Cool.
So, the, I think the, the answer is, it
takes at least a few minutes to deploy a
Rails app, unless you're one of an exceptional
few
folks in the audience. And I get it. There's
a lot, you know, there's files to transfer.
There's
dependencies to install. Most modern Rails
apps, you know,
that I run into, that we create, have a
lot of gem dependencies. It takes some time
to
boot the app with all those dependencies and
just
with, just with your app code.
And so, that's fine, except for, I was going
days just working on the Javascript app. Right,
I
was just making changes in Javascript, and
every time
I wanted to deploy, I was waiting five minutes
in our case to just deploy static Javascript
changes.
And it really made me want to throw something
across the room. Why was I doing this to
myself?
And it wasn't just me that I was annoying.
I was also annoying our users, because, in
most,
in most Rails deployment scenarios, there
are some, there
can be some hiccups in each deploy. And we'll,
so let's talk a little bit about this, this
kind of hiccups and deployments and zero downtime
deploys.
So if your Rails app takes several seconds
to
boot, which is probably about average, obviously
it can't
serve requests during that time. And so, under
high
load in most architectures, most requesters
are just gonna
be queued, waiting for the app to be ready
to handle requests. And then once it boots
up,
it's gonna start handling those requests,
and eventually flush
that queue and hopefully catch up to the requests
as they're coming in.
And so users that are hitting the, hitting
your
app during this time may experience at, at
best
case just a couple of seconds of downtime.
At
worst case, kind of a feeling like that, that
this site is not responsive.
And so it disappoints me that we don't yet
have a kind of conventional solution for zero
downtime
deploys. But it kind of makes sense because,
by
definition, Rails runs inside of other web
servers, and
so, and that, this is really a concern kind
of at that web server layer.
So, Heroku, for example, has an experimental
solution. Heroku
Labs is, Heroku's kind of unsupported experimental
area features.
And you can run heroku labs:enable preboot,
which will
start up new servers or dynos with your new
code, wait three minutes to give your Rails
app
plenty of time to boot, and then switch traffic
over.
For, if you're using Puma or Unicorn, there
are
facilities to start one worker at a time,
or
groups of workers by sending signals to the
master
process. HAProxy is a tool that I've used
in
the past to kind of split traffic, give yourself
time to boot up another, another set of servers.
HAProxy is nice because you can do health
checks
against those new servers and say, am I ready
to deploy? And Passenger also has some solutions
around
this.
In terms of kind of the full scope of
zero downtime stuff, it gets more complicated
when you
talk about database migrations and what's
safe and what's
not safe for these, these types of, these
types
of deployments. And I'm happy to chat about
that
with anybody later, but that's out of scope
for,
for this particular talk.
What I do want to drill into a little
bit is issues with static assets and zero
downtime
deploys, because that's the thing that was,
you know,
kind of at the heart of what I was
doing, was really dealing with these static
Javascript assets.
And I think that these issues aren't discussed
often.
So I kind of want to drill down, kind
of at a detailed level, and talk about them
here.
So when a browser makes an initial request
to
your server for, to load your, your rich Javascript
app, it's loading the index dot html file
and
by index.html, I'm gonna refer to the html
file
that's the bootstrapping, it's bootstrapping
your Javascript app. It's
the thing that has a little bit of code
to fire things up and it pulls in the
right Javascript and CSS assets.
So the request comes into your servers, your
server
responds with the HTML file, with the text
slash
html mine type, and typically the, your asset
files,
the Javascript and CSS that are gonna be referenced
here, are gonna be fingerprinted. Right, so
we do,
take a hash of the contents of the Javascript
file, we set it as a, we include that
hash in the file name, and we're then are
able to set far features expires headers on
those
files, so that when the file changes, we don't
have to worry about cache expire or anything.
We
just gotta new file that's gonna come through
as
if the browser's never seen it before.
And so in this case, our html file might
contain something like assets slash app dash
abc123 dot
js, where abc123 is this fingerprint we're
talking about.
And so the browser takes that html, parses
the
page, a short time later makes the request
for
app dash abc123. Server says, here you go,
some
text Javascript. Browser parses that, boots
up the app.
All is well.
Hopefully this is very clear to everybody
who's in
the room. What's maybe less clear, unless
you've thought
about it in detail is that during deployments,
this
idea can break down a little bit. And so
imagine that we've got our deployment and
we've got
two kind of sets of server. The top set
that we're looking at here on the screen is
the existing code. The bottom set is the new
code that you're deploying.
And in this case, there's a change to the
Javascript file, so there's the new fingerprinted
filename there.
So when our initial request for our page comes
in, it was, it will go to the old
code, because we haven't yet switched traffic
over to
the, to the new deploy. And just like in
our first example, it's gonna come back with
the
index file references app dash abc123 dot
js.
Now, what if, in that moment, where, as the
page is being parsed, before this request
comes back,
we then switch traffic over to pointing to
the
new server? Well, request is gonna come in
for
app dash abc123 dot js, the new server gets
the request and says, ah, I don't know what
you're talking about. I don't have a Javascript
file
with that name. And so it says, 404 Not
Found.
And this is a challenging problem to, to address,
because there's a, because of a few reasons.
One
is that most simply if I, at this point,
now hit refresh in the browser, of course
everything
works fine. Right, because now both of those
requests
are going to the new server and the world
is good.
It can be further kind of shadowed, this issue,
because, if you are serving your assets up
through
an assets host at CBN, you might have some,
some, some of their, your edge nodes might
have
that old page cached. That old Javascript
cached. In
which case, those nodes will return it just
fine.
And so, to, to know that you're, you know,
totally impervious to this, you know, might
be a
little bit fuzzy, and also to be able to
reproduce it reliably is a challenging thing
to do.
But, in short, to avoid these hiccups, both
the
old versions of your assets and the new versions
have to be available for at least a few
minutes during your deployment in order to,
to make
this zero downtime approach really work well
on the
static asset front.
And so this was one of the things I
was thinking about during these many five
minute deploys,
where I was, you know, wishing that I had
a solution. And so, in thinking about that,
I
said, well, we could figure out how to do
this on our app servers. Kind of keep the
old versions of the, of the Javascript and
the
new versions together. Or we could move the
assets
elsewhere. And the idea of moving the assets
elsewhere
really appealed to me, because that meant,
if they
weren't on the Rails, my Rails servers, then
maybe
I could avoid doing Rails deploys when I just
had static asset changes.
So I started to sketch out an idea. We've
got our Rails server at the top, and we
had this separate static asset server at the
bottom.
Let's deploy our Rails app code to the Rails
servers, our Javascript, CSS, and images to
these static
assets servers, and then we would deploy our
index
file. Where would we deploy our index file?
And so, I started to think about, well, what
is the index file? It is kind of this
thing that bridges the two? What are the requirements
around it? What do we know about it? And
this index, the html file points to fingerprinted
Javascript
and CSS, but it's not fingerprinted itself.
That's obviously
important, because it needs to be at, at a
consistent location for browsers to locate,
to load. It
contains Javascript urls and code to boot
the Javascript
app to load CSS and such in the right
order. It's a good place to provide environment-specific
configuration
to Javascript. Maybe you have some differences
between dev
and stage in production.
One thing that I knew was key, because I
had struggled with it, is that when you serve
this html off of the same domain as your
API, life is way simpler with respect to cores
and cross origin security issues. And finally,
if I
wanted, if you wanted to be able to deploy
changes quickly to your users, caching on
this particular
page should be minimal to none, so that you
can pretty instantly switch over.
So my conclusion, from thinking about this,
is that
the html page ideally should be managed and
thought
about as part of your static asset deployment
process,
but it should be served off of your Rails
server.
And, as importantly, it should be served off
your
Rails server without requiring a Rails reboot
or redeploying
the entire Rails app. And so, so we were
able to start to refine this sketch and say,
OK, our Rails server's gonna be serving up
our
API requests, our, kind of, traditional, dynamic
Rails pages
as part of the html for a Javascript app,
and our static asset server's gonna be serving
up
the Javascript, CSS and images.
And so that means we need to somehow deploy
our html up to the Rails server without a
full, a full reboot. And so, how could we
do this?
Well, the most obvious thing to me was, well,
take a html file and put it on the
file system of each Rails server. And this
has
a few things that aren't great about it. You
can probably make this work in some configurations.
In
many deployment environments, disk is ephemeral,
and so relying
on, you know, on copying some things up might
not be a great idea. It's also a little
bit weird to mix assets, files deployed from
a
particular gitshaw, with files deployed from
somewhere else, kind
of in the same file system.
And so, we said, well, what if there's something
that we could all see and talk to? Well,
what about uploading to S3? So then all the
Rails servers can, can see S3, read from it,
be able to serve up that html. And this
could kind of work, but reading from S3 is
a little bit slow. And we wanted this page
to be fast, obviously. No Javascript or CSS
is
gonna start being loaded until this page is
loaded
in the browser. And so the fastest we could,
the faster we could get this page to the
user, the better.
And so, then we said, well, what about redis?
Redis is persistent. It's fast. For us, it
was
already in our environment. We liked this
idea a
lot. We decided to, to dig in. This is
not the, as you'll see, this is not the
only way to do it. It's totally possible to
user other systems besides redis. But redis
kind of
firt the bill for us and works quite well,
as you'll see.
So, the general idea was we're gonna deploy
into
redis and then serve out of redis via a
Rails controller. So here's the simplest possible
kind of
deploy code that we had. It's a rig task,
and we're going to generate our html for the
current assets, and that's, that's kind of
an exercise
for your build tooling, which we'll talk about
a
little bit later. And then once we had this
html, we're actually going to set it as a,
at a, as a redis key-value store. So the
html is the value and the key would be
something well-known like jsapp colon index,
for example. And,
and this is a redis connection that's connecting
directly
to your charted deployment environment. So
that's staging more
production.
Once it's there in redis, our controller,
again, the
most simplest version, is get the value out
of
redis. Render text.html. Now, when I first
looked at,
looked at this code or wrote this code, I
said, is that gonna be served up with the
right mine type? Seems a little strange. And
it
turns out that, yes, if you do render text
and some string, Rails serves that up with
text
slash html, if you don't specify. So, it's
a
little, I think a little bit of a confusing
API, but it does what we want.
So, we can now continue to refine this approach.
We know we're deploying, when we need to deploy
Rails app code, we're doing a deployment to
our
Rails server. When we're deploying Javascript,
CSS and images,
we're deploying to the static assets server,
and then
we're deploying html by connecting to redis
and deploying
into it.
And we can make things a little bit nicer
by dropping cloud front right in front of,
by
using S3 as our static assets server, and
then
dropping cloud front instead of in front of
our,
in front of S3. So, for very little effort
and very little money, we've got now CBN distribution
for our static assets.
Now, there's a few things about this deployment
to
S3, in terms of making it fast. Getting a
file list from S3 can be somewhat expensive,
particularly
the more files that you have. And so the
approach that we took was to generate a manifest
file of our current assets and store the copy
of that manifest on S3 so we, we're basically
gonna read the remote manifest, compare it
to our
local manifest, and know we only need to deploy
what's different. And so this means that if
I
make one Javascript, one line of Javascript
change, it's
just the file that that's concatenated into
that needs
to be updated, and not all of my images
and CSS, as we're doing our deploy, our assets
deploy to S3.
Now, purging has been on our to-do list for
this architecture for quite a while, right.
After a
deploy is successfully completed, we can,
in theory, remove
stuff from S3. We never really got around
to
that. Mostly because it's so incredibly cheap
to store
these small files on S3, so, still on our
to-do list. I would probably not recommend
you prioritizing
it too high for yourself. The code for this
S3 sink is here at this link. I will
make these slides available. You don't have
to worry
about copying it down. This repo is open source
and contains a lot of the code that we're
looking at today. And it's the actual code
that
we use for our, for our production environments.
So, once you start thinking about this architecture,
it
paves the way to do something a little bit
more fundamental with your rich Javascript
app and your
Rails app, which is that you pull them, tease
them apart into separate repositories. And
now why would
you want to do that?
Well, one of the reasons is, you know, thinking
about tagging, do, you know, kind of tagging
a
deployed version of your code. Since you've
got these
independent deploy processes, it makes sense
to be able
to tag a Javascript deploy separate from a
Rails
deploy, because they really are now independent
of each
other.
And I find also that thinking of your Javascript
app as a separate, independent client of your
API,
works really well. Kind of puts it on the
same level as a native app, for example, maybe
if you've got an iPhone app that communicates
to
your API as well.
It also opens up the realm of possibility
to
having a lot of flexibility with what kind
of
build tools you want to use with your Javascript
app. You may choose to use sprockets in your
separate standalone repo. Or you may choose
to use
grunt, gulp, broccoli, brunch. You name it,
there's obviously
a lot of innovation and creativity happening
around build
tools in the Javascript environment. And my
guess would
be that you're, we're gonna see faster, you
know,
innovation, iteration in the Javascript environment
for building Javascript
apps than we will in the Ruby environment
for
building Javascript apps.
So, we've now got an approach that works pretty
well. But I think the, the question is, is
it worth doing the work to set this in
place? Like, how fast is this in the real
world? And so I took one of our apps,
and this isn't, was not a scientific bench
mark,
so consider it directional. And our builds
took about
six and a half seconds. This particular app
is
using Rake Pipeline as a build tool for the
Javascript side. Our transferring assets to
S3 using this
differential approach was about a second,
and then uploading
html into redis was about two and a half
seconds. And so, instead of a five minute
deploy,
I was now able, we were now able to
deploy this, our Javascript apps in under
ten seconds.
So just by that, that was a big win.
And stopped me from wanting to throw things
across
the office. But, I think that, you know, in
any kind of architectural choices like this,
you learn
if this is a good idea or not over
time, right, based on, how does, how does
this
architecture respond to changes. What kind
of possibilities does
it enable? So I want to talk a little
bit about the kind of emergent behavior that
we've
seen around, now that we've had this in production
for awhile.
The first thing is, the idea of preview. And
so this is an actual command line session
for
deploying an app. In this case, it's yapp-prefs,
which
is our, kind of, account settings app. We
first
run rake dist. This is the build. And with
the build completes, in this case, as we saw
in the pie chart before, in about six seconds
or so. And it says, OK, to deploy these
assets to S3, run rake deploy:assets with
this, what
we call a manifest idea, this b35b.
So what's a manifest id? We talked earlier
about
fingerprinting assets and we talked about
creating a manifest
file. So what we do also is we fingerprint
the manifest file. So we take a hash of
the contents of that manifest file and we
say,
OK, that is the manifest id for this deploy.
And that's, it's, it's kind of a unique identifier
as it's going through its unique deploy process.
And so we run rake deploy:assets, which does
the
differential upload to S3 and it's gonna show
us
what it uploads. It's gonna spit out, OK,
I've
uploaded these four files. JS, CSS, two yaml
files
for the manifest. We, actually, these are
two copies
of the same thing. One is a, has a
file name with the id, and one just is,
hey I am the latest. And it's going to
then tell us the next command to run is
deploy:generate_index for this manifest id.
And what this is
gonna do is going to connect to redis and
set this at a key named for the manifest
id. So in the previous simplistic example
we looked
at, it was just updating jsapp index. Now
it's
updating a key at prefs, in this case, prefs:index
b35b something, you know, named for the manifest
id.
And why is this awesome? Well, this command
line
tool can now tell us, hey, to preview this,
this asset change, go ahead and take a look
at your at, your site with the query param
manifest id equals b35 et cetera. And what
this
is gonna do is it's going to pull the
new html file from redis. Gonna show, which
is
loading up the new Javascript. It's connecting
to the
production API. So you're able to smoke test
this
in production. Everything is working just
as the user
will see it, except for your users don't see
it yet. So if you screwed something up, you've
got a chance before kind of pulling the trigger
and switching it life to go.
And then, finally, one more command to kind
of
active that redis key and make it the current
key. And so what does this code look like?
It's actually not that much more complicated
than what
we saw before. We invoke our rake task with
the manifest id. Generate our html file. And
then,
instead of setting jsapp index, we'll set
a jsapp
key based on the manifest name, or redis key
based on the manifest name. And then spit
out
something to give the developer the url to
take
a look at, to preview, to preview the app.
On the server-side, we're gonna add one more
redis
request to the mix. If there's a manifest
id
param, then we'll just use that. If it's blank,
then we'll go and we'll connect to a current
key. Grab the manifest id, then use that to
serve up the current version of, of the site.
Of the, of this index file.
And so, that has been pretty powerful, and
it's,
it's a super useful tool that we use in
almost every single deploy. The developer's
just gonna do
a quick smoke test and say, yes, everything
looks
good, before they flip the switch.
The next interesting aspect that this kind
of enabled
is around dynamic html rewriting. And so what
we
realize is that as html was passing through
from
redis through the controller back to the browser,
we
had the opportunity to make adjustments if
we wanted
to. And one category of adjustment that we
ended
up making, as you see in this example, is
injecting some information about the current
user.
Now, obviously in a Rails controller we know
typically
who the current user is. Most apps will have
a current_user method available to any controller
that they
can, it can grab. In contrast, when you're
booting
up a Javascript MVC app, at that point, you
know, most apps don't know who the current
user
is. And if, in most, you know mostly there
is some XHR request that's involved in figuring
out,
is this user logged in and, if so, what
is their role in the system? And during that
time, the user is sitting there waiting, right.
The
Javascript is just kind of, maybe it's rendering
a
loading spinner for the user or something.
But it's
a little bit annoying and it also makes the
boot process for the, for your Javascript
app more
complicated.
So we said, well what if the app could
have, at boot time, have access to this information?
So what we're doing here is, in the controller
action at the top, between the time we get
the html out of redis and return, render it,
we're going to inject current_user information.
We're gonna grab
the current_user and then user our AcitveModel
serializer to
convert it to JSON, escape it, and stick it
into the head tag.
And you'll notice that the method that we're
using
to add this to the html, you might find
a little crude. And certainly when I first
started
doing this, I said, OK, we'll use Nokogiri.
We'll
parse the html. And then we'll insert, you
know,
insert a node, and then we'll render, you
know,
convert that back to text. And it turns out
that really what we're doing is so simple
that,
as fast as Nokogiri is, and it's pretty fast
for an XML or html parser, it is not
faster than string manipulation and string
indexing.
And so, in this case, we're just looking at,
where's the end of the head tag or the
beginning of the head tag, inject this meta
tag
in there. And this works great.
The, some other use cases for this same approach
might be injecting csrf tokens if you, you
know,
need to interact with Rails forms from, from
your
Javascript app. Including dynamic analytics
params, we had a
case where the, the Javascript app was kind
of
the, the, a goal page for a Google analytics
kind of flow. And we needed to set some,
certain Javascript only in some conditions.
So this was
a nice way to do that.
If you're using feature flags through something
like the
excellent rollout gem, it's great that that's
available throughout
Rails. But how do you make it available inside
of your Javascript app also, this is a nice
solution to be able to kind of inject variables
along those lines.
Another pretty awesome thing that we've been
able to
do using this approach is doing A/B testing
within
our Javascript app. And we've experimented
with two different
kinds. One is, kind of, setting some flags
based
on which bucket the user ends up in, A
or B. This is pretty similar to what we
just described. And then the, the second is
serving
up wholly different html based on the A/B
bucket.
We'll, I'll talk, take you through each of
those.
So this is using the split gem, which, if
you haven't seen, is a great A/B tool, A/B
test tool for Ruby frameworks, web frameworks.
It has
a Rails integration that gives us the A/B
test
method, which you see on line eleven there.
And
so we're doing some injection into our html
where
we're saying, if the user is part of the
show walkthrough experiment, then we're going
to inject the
script tag that just sets a global variable.
And
then our Javascript app as it boots in is
running, is, is, can consult that global variable
to
decide whether to show the walkthrough that
you're doing,
doing some testing on.
And then later, elsewhere in your app, you
would
indicate that, that the goal was achieved,
that the
user signed up or published or whatever it
is
that was kind of the, the goal for you
A/B test. And so this was great. This is
excellent for the kinds of A/B tests where
both
paths are supported by your, by a given incarnation
of your Javascript app.
We had another case, though, where, this was
right
around the time that iOS7 released and there
was,
if you recall, there was a lot of excitement
in, around flat design. And so we did a
redesign of our Javascript app. We weren't
immune to
the hype. And, but as we got near completion,
we started to wonder, well, we're gonna release
this.
How do we know that this is any better
than our existing site? Is this going to improve
our metrics or hurt them? And we wanted to
try to get some confidence one way or the
other.
So we said, well, we've been developing this
in
a branch, and we already know that we can
preview different versions of our Javascript
app, because we
had this preview mechanism in place. Could
we use
this for A/B testing, also?
And so, it turns out that we were able
to just update our deploy scripts to be able
to deploy from a branch into an experiment
kind
of redis key, and then use the same A/B
test mechanism to determine, for the new design
experiment,
should the user use the current manifest or
the
experimental manifest?
Once we made that decision, we then got the
appropriate manifest id from redis, got the
html, rendered
it, and users either saw our old app or
the new flat-design app. Turns out, flat design,
about
nine percent better. So good news.
So this is suitable for changes where your
development's
capping at a branch, and you want to A/B
between the branches. Not a common scenario,
but if
it's, when it's, it's useful, it was great
to
see how this approach supported that architecture.
One more possibility that I didn't cover here,
but
you can imagine how this might work, is around
doing rollback. So what if every time that
we
did this deploy and updated our html in redis
we pushed into a redis list and said, hey,
so and so user deployed such and such manifest
id at this time, and then we were able
to have a rake task that reads that list
and lets you rollback to any particular version.
Hopefully,
at this point, you can see how straightforward
that
would, that kind of thing would be as well.
So, with that, I want to say thank you
to my colleagues at Yapp Labs who helped create
this. Kris Seldon, Stefan Penner and Ray Cohen.
And
while we were working on this, we had heard
some rumors about some Square engineers doing
a, using
a similar approach at Square. So we took some
inspiration from those rumors as well, and
so thank
you, nameless Square engineers, or if anybody's
here. Love,
love to chat with you about it.
We've got some time for questions, and so
I
want to open it up to all of you.
Question in the middle?
All right, cool. Thank you all so much. Appreciate
it. Enjoy the rest of the conference.