LUIGI MONTANEZ: So, thank you everyone for
coming.
It's late on Friday and I really appreciate
all of you sticking around. Obviously Tenderlove
is next. So-
AUDIENCE: Talk louder!
L.M.: Talk louder? All right. That's be- is
that
better? All right. So, before I get started,
I
just want to actually do a quick plug, cause
I saw that the session behind me is actually
on the impostor syndrome. And since you folks
are
not gonna be there to, to see that, I
just wanted to actually tell you a little
bit
about it, cause actually it's a, it's a really
important concept. So, essentially, the impostor
syndrome is when
successful people like software developers
feel like they might
not deserve all the success they've gotten.
And it's
actually very common. When I learned about
it a
few years ago, it was actually really helpful
for
me and my career. So when you go to
conferences like these and you see members
of the
Rails core team or the Ember core team or
all the speakers and you think, wow, I am
not worthy, actually you are. You, you really
do
belong here, and, and people who are successful
usually
do deserve it. So Google the impostor syndrome
when
you get home or, or watch the video when
it gets posted.
So to our talk. So. You will never believe
which web framework powers Upworthy. I, I
can't believe
the conference organizers have made everyone
wait all week
to figure this out, to find this out. So,
at Upworthy we aim to please. So we're just
gonna spill the beans right away. Obviously
the answer
is Java Struts.
AUDIENCE: [applause]
L.M.: Awesome. Now. So. To introduce myself,
my name's
Luigi. I'm the founding engineer at Upworthy.
I was
the first engineer hired. I've always been
a software
developer involved in politics and advocacy.
I got really
into this guy who screened Howard Dean. And
so
I worked for his political campaign. There
was this
political action committee. I worked for other
campaigns and
non-profits, and then before coming to Upworthy
I worked
for the Sunlight Foundation, which is a great
non-profit
in D.C. that focuses on transparency and government,
and
open, open government data.
RYAN RESELLA: I'm Ryan Resella. I'm the senior
engineer
at Upworthy. Before this in 2011 I was a
Code for America fellow, the first class of
fellows.
I came on as technical lead full time staff
there. And then last year I was on the
Obama for America tech team, or, I guess 2012,
working as a software engineer. And I ran
out
of for America organizations to work for so
I
joined Upworthy.
L.M.: So, at Upworthy, our mission, and we,
this
is something we truly really believe at the
company
is to drive massive amounts of attention to
the
topics that matter most. And that will kind
of
inform the engineering decisions we made as
the, as
the tech team. So, just to kind of give
people a little peak at what Upworthy does,
for
those who aren't too familiar, so this might
be
a bit hard to read, but when we say
that topics that matter most, these are kind
of
the topics we've covered in the last year,
the
topics that have gotten the most attention.
So I'll
just read some of them aloud.
There's a lot of women's issues, like body
image,
gender inequality, standards of beauty, a
lot of economic
issues like income inequality, health policy.
We also cover
a lot of stuff about disability, mental health.
Also
bullying, bigotry, racial profiling and, and
race issues. And
when we say that we want to drive massive
amounts of attention to these things, what
we really
mean as web developers, as web engineers is
we
want to drive massive amounts of traffic.
So here's a look at our growth for the
last two years. So we launched a little over
two years ago. We started off at, around one
point seven million uniques per month. That
was in
our first ever month in April 2012. And then
we went up to, in November of 2013, 190
million page views. So, this has made us probably
around the top forty, a top forty site in
the U.S. We're maybe one of the larger Rails,
more trafficked Rails apps out there.
To give you a sense of what kind of
traffic we actually deal with, here's a twenty-four
hour
graph of what our page view data looks like.
So starting at midnight all the way on the
left, and then ending midnight the next day,
you
can kind of say how during the, when the
daytime happens, when, when work hours start,
we get
these spikes, these peaks of, of, of traffic.
This
is essentially viral, the viral cycle being
visualized.
So, we have handled, at most, about 130,000
concurrent
visitors. This is a screenshot from Google
analytics during
one of the traffic spikes. So, you know, we,
we are handling large amounts of traffic in
very
spiky ways.
So here is a example post from Upworthy. This
was really popular about, a few months ago,
in
the fall really. Who here remembers this post?
Just
curious. Cool. A few of you.
So this is what Upworthy content looks like.
So
it's really just static content. So See Why
We
Have An Absolutely Ridiculously Standard of
Beauty in Just
37 Seconds. It's a video about how a, a
woman, a model gets photoshopped and looks
essentially like
something, like the standard of beauty that
doesn't really
exist. So that, that's kind of the content,
or
angle we were going for. And here you see,
we have the content, which is basically static
content
on the left side of the screen. We have
this sidebar with recommended content on the
right side
of the screen. And then scrolling down, we
have
what we call asks. We, we also do kind
of some testing on different kinds of content,
different
kinds of headlines. You see that down there
with
that John Stewart video. And then we have
asks
around, do you want to subscribe to our email
list? Do you want to like us on Facebook?
We also have kind of pop-ups that happen after
you watch a video, after you share, also asking
you to do stuff.
So those are the things we, we, the technical
things we, the technical concerns we have
at Upworthy.
We're pretty much a static site. We're a public
site. And then we have a CMS backing that,
and then we have this system of dynamic pop-ups
and asks around the site that kind of optimize
our subscriber base, get us more subscribers,
get, get
folks to share content.
So, the topic of this talk will really be
about managing the growth of our start up's
web
app in the face of very high traffic. And
I actually remember maybe five years ago sitting
at
a RailsConf, maybe it was in Baltimore, and
I
was sitting where you're sitting and it was
a
talk, it was a talk by the guys from
YellowPages dot com. And YellowPages dot com
was and
still is one of the larger Rails sites in
the world. Obviously YellowPages is such a
strong brand.
Everyone goes to the YellowPages, a lot of
people
still use YellowPages dot com to find out
local
businesses and stuff like that. And they were
talking
about how they scaled their app, their Rails
app.
And I was thinking, I was sitting there in
the audience, thinking well, this is really
interesting, but
I'm not sure this is ever gonna apply to
me. I don't really, I work on these small
things. No one ever really sees them. But
fast
forward a few years and, you know, here I
am. I'm, I'm working on this, this app that
millions of people see every day. So it can
really happen to you too.
So, let's start from the beginning. We launched
in
early 2012, March 26th, 2012 to be exact.
And
at the time there was one engineer, me, and
our CTO who is a programmer, but was not
really a Ruby or Rails developer. And we actually
launched not on Rails but on Padrino. Who
here
is familiar with Padrino? Cool. Who here has
production
apps in Padrino? No one. OK. That's what I
thought.
So Padrino, this is, it's, it kind of builds
itself as the elegant web framework. It is
essentially
just Sinatra with more Rails-like things on
top of
it. So he, who here has used Sinatra? More
hands. Of course. And who here actually has
Sinatra
in production? A few, a few hands. Yeah.
So, essentially, when you're working with
Sinatra, it's a,
it's a more low-level library. More closer
to Rack.
And Padrino, essentially, adds the things
that you miss
from Rails into Sinatra. It also freely borrowed
some
really great ideas from Djando. The first
one being
first-class mountable apps. So in Rails we
have engines.
But it seems like not, people don't really
use
engines that often. Like, you might use it
for
Rails admin. It's, you kind of might have
a
larger Rails app and then break it up by
putting it into separate engines. But with
Padrino, all
code actually lives in a mountable app. So
you
have to use this mountable app system to use
it.
It's also middleware centric. So, those of
you who
are familiar with Rack, know that there's
a lot
of kind of, there's this concept of middleware
-
it's also in Rails - where you can kind
of write these small bits of code that are
Rack compatible, that sit on the stack of
what
you do, of how requests and responses come
into
your Rails app, or your Sinatra app or any
Rack app.
And there's also this built-in Admin area.
And that
Admin area is actually an app. That's just
another
mountable app. So this is, this is something
that
Django has. I know we have Rails Admin here
in, in the Rails world. But with Padrino,
this
is a thing that is built into the framework
itself.
So, why Padrino? Why did I use Padrino in
the beginning? Essentially, at the time, I
was a
seasoned Rails developer. I was probably developing
for about
five years on Rails. And during that time
I
started to form my own opinions about Rails.
And
some of those opinions were not compatible
with what
the, the, the Rails way prescribed. And I
saw
Sinatra and Padrino as this way that I could
still write Ruby, still, I still loved writing
Ruby,
but I could also make my own choices. And
there's this kind of epiphany that seasoned
web developers
kind of ultimately have, which is, I'm writ-
I'm
using this web framework, whether it be Rails
or
Django or Sinatra or Node, that all it's doing
at the end, is it's just, it's really just
a web server. Cause at the very end, you're
just getting in responses, you're just emitting
HTML or
CSS or JSON, if it's an API, or JavaScript.
That's all you're really doing. And all this
talk
about TDD and good object-ori- domain, object-driven
design, they're,
they're very important. They help us manage
complexity. But
in the end, what does the framework physically
do
in the air quote physical way is it, it
really just takes in HTTP responses, or, excuse
me,
takes in HTTP requests and then responds with
HTTP
responses.
So, while Rails will give you the full foundation
of how to build, build a skyscraper, Padrino
gives
you a foundation, but it also lets you choose
some of the plumbing and make some choices
for
yourself. So the good parts about Padrino
are, is,
it really is just Ruby. It's just Rack. And
if you're a fan of thinking in that mindset,
you'll, you'll really enjoy it. There is less
magic.
Things are more explicit. It's unopinionated.
All the generators,
when you, when you generate a Padrino app,
you
specifically say what you want. That you want
ActiveRecord
versus DataMapper versus Monoid or you want
Sass versus
Lest versus ERB. Whatever, whatever you want,
you can
specify it.
I actually enjoy the process of writing middleware.
I
like thinking about web apps like that. I
think
it's a much more performant way to think about
writing web apps. And Padrino itself, unlike
Rails is,
it's light, it's performant. It's really just
Sinatra with
a few more libraries on top of it.
So this is what our architecture looked like
when
we launched. So the whole big box is a
Padrino, is the Padrino app. And we had two
mounted apps inside it: main, the public site;
so
when you visited Upworthy dot com, this is
what
the public would see, those, those content
pages; and
then we had the admin tool, the built-in Admin
app in Padrino, which essentially functioned
as our CMS.
And keep in mind that we, we were hoping
that we're gonna launch this thing and then
we're
gonna get lots of traffic. So we needed to
figure out how to scale it right away, right
from the get go.
So, I kind of devised this, this idea called
explicit caching, and I remembered back in
the early
2000s, there was this blogging framework called,
or blogging
tool called Moveable Type. And Moveable Type
was what
all the big, those early blogs used. And Moveable
Type essentially, the way it worked is, it
was
a CMS. So when you saved your post to
the database, Moveable Type would actually
save, obviously, see
that you saved something to the database.
And then
it render the HTML right then, and then write
the HTML to disc. So when your, when people
visited your blog that was hosted on Moveable
Type,
they weren't hitting that Perl app and going
through
the database. They were actually just hitting
these rendered
HTML files and CSS, JavaScript, that were
just living
on the file system of your server.
So I, I actually was drawn to that idea.
I liked that idea. So I kind of re,
re-made it a little bit here in Padrino. So
in the Admin app, there was this publisher
object.
And the publisher object essentially did that,
where once
anything was saved to the CMS, any content
was
saved to the CMS, the publisher object would
see
that. It would actually make a request to
the
Main app, which was actually rendering the
public site,
and it would write that rendered response
to Redis.
And so RedisCache was a middleware layer - I
talked about middleware earlier - that sat
very close
to the front of this Padrino app.
So, when we had a million or so pages
in that first month, they were all really
being
served from Redis. Essentially the website
was just in
memory in Redis. And so that worked, you know,
it scaled well.
So, around this time, June, 2012, we hired
a
second Rails engineer, Josh French. So, he
kind of
joined the team, and then a few weeks later
he said guys, I think we should switch to
Rails. And he was, he was probably right.
Because
there were pain points that were not related,
really,
to the technical performance of Padrino, but
actually more
about social aspects of it. The first one
being
that the ecosystem for libraries, while, while
pretty good,
because, again, Padrino is just Sinatra, just
Rack, was
not as strong as Rails. There's just libraries
for
everything you want to do in Rails. We, there
were many things we could do in Padrino, but
the, the kind of quality of those libraries
was
not as high. A part of that is because
Padrino isn't as popular, it wasn't very frequently
maintained.
The actual Admin app was not very pleasant
to
look at. It was its own HTML, CSS styling.
I put a star here because, literally, once,
the
sec, the day we moved off of Padrino fully,
they released a new, a new release which was,
which had the Admin system in Bootstrap, which
was
what we wanted all along. And there was no
community and, as a start up, it's like actually
easier to hire Rails developers, cause we
can't really
go hey, we know you're a great Rails developer
but you're gonna have to work on this thing
called Padrino. That wasn't really a, a strong
sell.
So we decided to move. We wanted to move
to Rails. But at the same time, we're, we're
a growing start up. We're getting lots of
traffic.
So how do we kind of balance this desire
to move to, move our architecture while still,
you
know, maintaining a running app, while still
having to
maintain a stable, running app that is serving
a
growing traffic base. And Ryan's gonna continue.
R.R.: So, we started our Rails migration in
October
of 2012, and so this is a timeline. This
is October 2012. And basically the way it
started,
we generated a new Rails app, moved all the,
we mounted it inside the routes.rb, so we
just
basically mounted, and you can do the same
thing
with Sinatra, since it's Rack. And then we
slowly
the migrated the models and utilities into
the app.
So, when I joined, we were kind of in
this weird hybrid state. I joined in January
of
2013, after taking a nice long siesta, re-electing
the
president stuff. And so, we had to figure
out,
how could we accelerate and just, you know,
get
us over the file hurdle and just move us
onto Rails completely and get out of Padrino.
So
the first step that I did was migrating assets,
and so we activated the Rails asset pipeline
which
had been turned off in our app. Migrated the
frontend assets, the backend assets. That
took us about
a week in February, 2013. The next step was
deciding if we wanted to do the admin area
or our frontend area first. So we decided
to
do the hard part first and do the frontend.
So we migrated all the frontend code. The
views
and controllers, which took another two weeks.
And then
lastly we, we did the entire backend CMS system
as a final push, and we added, we changed
it to bootstrap. Moved all the Rails controllers.
That
took us another two weeks.
So, here at this point, the entire migration
is
eight months. But it's really, it really ends
up
getting accelerated in the last few weeks
just because
we wanted to get to that, that final push.
And at this point, we're at three Rails developers:
me, myself, Luigi, and, sorry, myself, Josh,
and Luigi,
and our CTO goes back to actually doing CTO
work, which is great.
So now here we are. We're in a monolith.
We're in a big huge monorail for the entire
2013, we were able to do things the Rails
way. We were able to increase our velocity
and
just able to add lots of features. We were
able to program the way we wanted to and
really get things moving. We didn't have to
rebuild
helpers that weren't, weren't existing. So
we could just
have this one huge monolithic Rails app. We
had
the backend CMS, and then we had our frontend
and we had all our Ajax endpoints.
But one of the things, when you're looking
at
this monorail, is how are you scaling for
virility?
So on the campaign, there was a lot of
traffic, and it was pretty easy to know that,
you know, in November is gonna be really busy,
or in October there's a voter deadline so
it's
gonna be very specific of when traffic's gonna
hit.
You could, you could tell. In, in the virile
media world, you don't know if your post is
gonna be a hit or not. You don't know
when it's gonna get traction. And so you can't
have someone sitting there monitoring twenty-four
hours a day,
when to scale, when to not scale. So, we
had to think about how we were gonna do
that.
So a lot of it was just pretty simple
basic stuff here. We added action caching.
So we
took, we removed the homegrown publisher system,
just turned
on action caching in Rails, and it's backed
by
memcache. So people would hit the page, you
would
hit the memcached instance of the page instead
of
going out, hitting our database, pulling in
the page.
We were able to just do that.
The second part was just moving our assets
on
S3 and Cloudfront, so our app is hosted on
Heroku. There's actually, it's a really easy
tutorial on
Heroku on how to do this. You just set
up a Cloudfront instance and then you just
point
your config host for the, the CDM for your
assets to that and it magically works. It's
great.
And then the third thing is, we have lots
of Heroku dynos. So at the time we were
running up to forty Heroku dynos, and these
were
1x dynos at the time.
So mainly these are for our Ajax requests
so
that those asks, those popups, and the different
things
that ask you around the site, we were able
to scale with those. So we ran this for
awhile, and we, we had some slowness on the
site sometimes, so we tried to figure out
what
could we do to make sure that our site
would be stable and not have to worry about
these viral traffic spikes, having to scale
up and
down. So we actually implemented a CDN in
front
of it. We took some time and tried to
figure out what CDN we wanted. Because we
wanted
to do faster posts and just different things,
and
at the time we ended up on Fastly. So
Fastly is a reverse proxy. It runs on Varnish.
It's made, you guys could check it out. It's
great.
We, we moved all of that to, all our
HTML, CSS, and images to Fastly, and then
we
turned off the Rails action caching. And the
way
Fastly works is, is it reads your headers
on
your page, so you just set, set expire four
hours from now. So our site could literally
be
down for four hours and Fastly would continue
to
serve the pages.
From there, we were able to dial down our
Heroku dynos. So we went to, we switched to
the 2x dynos, and we only needed about half
as many dynos, because we were only serving,
off
the Heroku dyno for Ajax requests.
Probably the biggest thing that we learned
from Fastly
was the mobile performance gain. So Fastly
has all
these different locations around the world.
If I'm in
California requesting a page from Upworthy
dot com, it's
gonna go to the closest point of presence,
the
CDN location in California instead of going
out to
Heroku in Virginia, pulling from the data
center and
bringing it back. So the, the load times went
way down, and we're able to fully cache the
site, and we have, we've had zero downtime
since
implementing Fastly. So it's just been a great
performance
gain for us.
So, with having a big monorail, there's huge
pain
points that come along with it. We have, so
we have one app that deals with our www
service, Upworthy dot com, and then we have
a
backend CMS that our, our curators log into
and
do their work. So we had to figure out
the concerns with that. So it's, it's really
painful
there.
When there's traffic spikes on the public
site, it
could basically, it makes our CMS unstable,
so the
curator would log into our site, try to do
their work, and they couldn't navigate and
we'd just
have to tell them, you know, come back later
or come and do the work later. And when
the traffic spike comes down you'll be able
to
use it. And as our team grows, the code
base was becoming very large. So the classes
would
get huge and, you know, the frontend didn't
care
about some of the stuff the backend did. So
it just got harder and harder to maintain.
So, of course, what did we do? We break
up here. Fun fact. This is actually a film
in Chicago. In December, 2013, our buddy Josh
French
has another great idea. He says hey, I think
we should really start splitting up the Rails
app.
And so, if you look how this timeline is,
it's pretty evenly spaced. We didn't just
jump into
different things and start building them.
We like took
some time on each of these sections and really
focused on, on that narrow gap.
So, one of the things, when you're trying
to
decide how to break up your Rails app into
services, how do you do it? There's plenty
of
different ways you can do it. This is the
way we did it. This is not the perfect
prescription for everybody. I think you have
to look
at your app and see, like, where the clear
dividing lines are. So we basically, we just
chose
two right now.
So we have our www site and we have
our backend CMS. So there's a clear dividing
line
between the two. What we ended up doing is
cloning each repo into its own separate repository,
so
we could maintain the git repo history, and
then
from there we started breaking everything
up, right. So
this is what we need for this side and
this is what we need for this side and
let's start chopping.
Which was, ended up being a lot of deleting
and removing namespaced objects. So, once
we did that,
we deployed each of these apps to, to separate
Heroku applications. The nice thing about
Heroku is, they
have this function called Heroku fork, so
they'll just
take your current app, it'll fork it into
another
Heroku app, pull over all your plugins, everything
you
need. So we did that. We forked the app,
our main, our main app into two separate applications.
Removed the plugins that we didn't need on
each
side, and then we pushed out our applications
to
those Heroku sites. Everything's working great.
And all we
had to do was switch our Fastly endpoint to
point at the new Heroku app at origin and
we're done. So. Zero downtime really helped
there.
And then we continued to de-duplicate the,
the two
apps. So we created this core gem that is
holding a lot of the code that shares between
the two apps. So, our CMS runs on about
two, two 2x Heroku dynos, and now our frontend
app runs about twenty-five to thirty 2x dynos.
This is pretty much what this looks like.
So
we just have an app called www, and then
we have an app called CMS. And then this
gem shares the code between it. People will
hit
the Fastly endpoint between www and, and our
app.
So, what are the real benefits of service
oriented
architecture? I think there's, there's plenty
if you look
and think about it. One of the big things
is, we talked about the instability. You know,
if
our curators couldn't do their work, they,
you know,
we can't have like articles go out, so it
gets kind of annoying. So if there's a problem
on one app, it's, it's easier to fix and
maintain.
Each of them have different scaling needs.
So the
interesting thing is, our CMS, you know, when
we
have like twenty users that use our CMS, so
we could you know have is at 2x dynos,
instead of like having thirty dynos serving
all these
different pages. So the scaling needs was
really beneficial
here.
And that also divides our teamwork more naturally.
So,
we can have people, we can have engineers
on
a team decide to, you know, work on different
parts or features, and we don't have to wait
for something to ship or something else to
finish.
But of course, there's, you know, there's
a bunch
of drawbacks running an SOA. If you're on
the
full stack and you want to make a feature
on our CMS that needs to carry all the
way through the funnels of our frontend, you
have
to have both, all three environments running
on our
system to make sure that, you know, your change
goes and it funnels all the way through.
Coordinating deploys is also a huge pain.
So if
you need something that's in the core gem
plus
the CMS plus the www app, that means you
have to deploy each three, coordinate them
and make
sure that all of them are happening at the
same time, or all of them go in a
certain order, which, when you're on the monolith,
it's
really easy to just push one big Rails app
out.
And then migrating to a fully DRY set of
codebases, it's really tedious. Trying to
figure out what,
what needs to go where and where we put
stuff is, it's just been a really hard thing
to do.
So, some of the future work that we're gonna
continue to do on the SOA. We're gonna continue
to add stuff to our core gem. Remove dupli-
de-duplication, which is, it's kind of a pain
sometimes
to figure out where, you know, what things
go
where. And then we're considering making,
breaking up the
app even more. So, right now we have two
apps. We have this third app that actually
uses
up almost all our Heroku dynos. So we're thinking
about making that its own separate app and
service,
and we can ping that.
And then, you know, they all communicate with
these
different data stores. Should we just use,
a lot
of times an SOA has that service layer, and
everything communicates with that service
layer. So maybe we
should move to that.
L.M.: Cool. So. Just to wrap up here, some
lessons learned. So, first really, when you're,
when you're
working on an app for a company and you're,
you're, there's a feature request and there's
all these
other things going on in your company, really
do
wait to make these big technical architectural
changes until
everything starts to really, really hurt.
Until you really
feel the pain. And once you do decide to
make these changes, it's OK to take a really
long time. We're probably gonna take eight
months to
do that fully SOA system that we envisioned
back
in the beginning of the year. And that's just
because we have other feature requests from
the rest
of the company to fulfill. And luckily as,
since
we're on Ruby, it kind of makes it easier
to do that. It really made it easier when
we were moving from Padrino to Rails.
Serve everything you possibly can from a CDN.
I
don't think people use CDN's enough. They're
just hugely
beneficial to the performance of your system.
They're great
for end users, particularly mobile users.
At Upworthy, we
have well over fifty percent of our traffic
comes
from phones and tablets. And CDNs really,
really help
there.
So remember that you're just dealing with
HTML, CSS,
JavaScript, JSON, maybe, at the end of the
day.
Right. That's all you're serving. So think
about how
you can most efficiently serve those resources
from your
app. And if you are, you're doing your own
start up, if you're working on your own project,
I hope you can also experience this kind of
high traffic growth, because it's been a hugely
fun
and rewarding experience for the both of us.
So, with that, I'm Luigi, he's Ryan. Our slides
and our write up is already on our GitHub
engineering blog. And we will be happy to
answer
questions.