LUIGI MONTANEZ: So, thank you everyone for
coming.

It's late on Friday and I really appreciate

all of you sticking around. Obviously Tenderlove
is next. So-

AUDIENCE: Talk louder!

L.M.: Talk louder? All right. That's be- is
that

better? All right. So, before I get started,
I

just want to actually do a quick plug, cause

I saw that the session behind me is actually

on the impostor syndrome. And since you folks
are

not gonna be there to, to see that, I

just wanted to actually tell you a little
bit

about it, cause actually it's a, it's a really

important concept. So, essentially, the impostor
syndrome is when

successful people like software developers
feel like they might

not deserve all the success they've gotten.
And it's

actually very common. When I learned about
it a

few years ago, it was actually really helpful
for

me and my career. So when you go to

conferences like these and you see members
of the

Rails core team or the Ember core team or

all the speakers and you think, wow, I am

not worthy, actually you are. You, you really
do

belong here, and, and people who are successful
usually

do deserve it. So Google the impostor syndrome
when

you get home or, or watch the video when

it gets posted.

So to our talk. So. You will never believe

which web framework powers Upworthy. I, I
can't believe

the conference organizers have made everyone
wait all week

to figure this out, to find this out. So,

at Upworthy we aim to please. So we're just

gonna spill the beans right away. Obviously
the answer

is Java Struts.

AUDIENCE: [applause]

L.M.: Awesome. Now. So. To introduce myself,
my name's

Luigi. I'm the founding engineer at Upworthy.
I was

the first engineer hired. I've always been
a software

developer involved in politics and advocacy.
I got really

into this guy who screened Howard Dean. And
so

I worked for his political campaign. There
was this

political action committee. I worked for other
campaigns and

non-profits, and then before coming to Upworthy
I worked

for the Sunlight Foundation, which is a great
non-profit

in D.C. that focuses on transparency and government,
and

open, open government data.

RYAN RESELLA: I'm Ryan Resella. I'm the senior
engineer

at Upworthy. Before this in 2011 I was a

Code for America fellow, the first class of
fellows.

I came on as technical lead full time staff

there. And then last year I was on the

Obama for America tech team, or, I guess 2012,

working as a software engineer. And I ran
out

of for America organizations to work for so
I

joined Upworthy.

L.M.: So, at Upworthy, our mission, and we,
this

is something we truly really believe at the
company

is to drive massive amounts of attention to
the

topics that matter most. And that will kind
of

inform the engineering decisions we made as
the, as

the tech team. So, just to kind of give

people a little peak at what Upworthy does,
for

those who aren't too familiar, so this might
be

a bit hard to read, but when we say

that topics that matter most, these are kind
of

the topics we've covered in the last year,
the

topics that have gotten the most attention.
So I'll

just read some of them aloud.

There's a lot of women's issues, like body
image,

gender inequality, standards of beauty, a
lot of economic

issues like income inequality, health policy.
We also cover

a lot of stuff about disability, mental health.
Also

bullying, bigotry, racial profiling and, and
race issues. And

when we say that we want to drive massive

amounts of attention to these things, what
we really

mean as web developers, as web engineers is
we

want to drive massive amounts of traffic.

So here's a look at our growth for the

last two years. So we launched a little over

two years ago. We started off at, around one

point seven million uniques per month. That
was in

our first ever month in April 2012. And then

we went up to, in November of 2013, 190

million page views. So, this has made us probably

around the top forty, a top forty site in

the U.S. We're maybe one of the larger Rails,

more trafficked Rails apps out there.

To give you a sense of what kind of

traffic we actually deal with, here's a twenty-four
hour

graph of what our page view data looks like.

So starting at midnight all the way on the

left, and then ending midnight the next day,
you

can kind of say how during the, when the

daytime happens, when, when work hours start,
we get

these spikes, these peaks of, of, of traffic.
This

is essentially viral, the viral cycle being
visualized.

So, we have handled, at most, about 130,000
concurrent

visitors. This is a screenshot from Google
analytics during

one of the traffic spikes. So, you know, we,

we are handling large amounts of traffic in
very

spiky ways.

So here is a example post from Upworthy. This

was really popular about, a few months ago,
in

the fall really. Who here remembers this post?
Just

curious. Cool. A few of you.

So this is what Upworthy content looks like.
So

it's really just static content. So See Why
We

Have An Absolutely Ridiculously Standard of
Beauty in Just

37 Seconds. It's a video about how a, a

woman, a model gets photoshopped and looks
essentially like

something, like the standard of beauty that
doesn't really

exist. So that, that's kind of the content,
or

angle we were going for. And here you see,

we have the content, which is basically static
content

on the left side of the screen. We have

this sidebar with recommended content on the
right side

of the screen. And then scrolling down, we
have

what we call asks. We, we also do kind

of some testing on different kinds of content,
different

kinds of headlines. You see that down there
with

that John Stewart video. And then we have
asks

around, do you want to subscribe to our email

list? Do you want to like us on Facebook?

We also have kind of pop-ups that happen after

you watch a video, after you share, also asking

you to do stuff.

So those are the things we, we, the technical

things we, the technical concerns we have
at Upworthy.

We're pretty much a static site. We're a public

site. And then we have a CMS backing that,

and then we have this system of dynamic pop-ups

and asks around the site that kind of optimize

our subscriber base, get us more subscribers,
get, get

folks to share content.

So, the topic of this talk will really be

about managing the growth of our start up's
web

app in the face of very high traffic. And

I actually remember maybe five years ago sitting
at

a RailsConf, maybe it was in Baltimore, and
I

was sitting where you're sitting and it was
a

talk, it was a talk by the guys from

YellowPages dot com. And YellowPages dot com
was and

still is one of the larger Rails sites in

the world. Obviously YellowPages is such a
strong brand.

Everyone goes to the YellowPages, a lot of
people

still use YellowPages dot com to find out
local

businesses and stuff like that. And they were
talking

about how they scaled their app, their Rails
app.

And I was thinking, I was sitting there in

the audience, thinking well, this is really
interesting, but

I'm not sure this is ever gonna apply to

me. I don't really, I work on these small

things. No one ever really sees them. But
fast

forward a few years and, you know, here I

am. I'm, I'm working on this, this app that

millions of people see every day. So it can

really happen to you too.

So, let's start from the beginning. We launched
in

early 2012, March 26th, 2012 to be exact.
And

at the time there was one engineer, me, and

our CTO who is a programmer, but was not

really a Ruby or Rails developer. And we actually

launched not on Rails but on Padrino. Who
here

is familiar with Padrino? Cool. Who here has
production

apps in Padrino? No one. OK. That's what I

thought.

So Padrino, this is, it's, it kind of builds

itself as the elegant web framework. It is
essentially

just Sinatra with more Rails-like things on
top of

it. So he, who here has used Sinatra? More

hands. Of course. And who here actually has
Sinatra

in production? A few, a few hands. Yeah.

So, essentially, when you're working with
Sinatra, it's a,

it's a more low-level library. More closer
to Rack.

And Padrino, essentially, adds the things
that you miss

from Rails into Sinatra. It also freely borrowed
some

really great ideas from Djando. The first
one being

first-class mountable apps. So in Rails we
have engines.

But it seems like not, people don't really
use

engines that often. Like, you might use it
for

Rails admin. It's, you kind of might have
a

larger Rails app and then break it up by

putting it into separate engines. But with
Padrino, all

code actually lives in a mountable app. So
you

have to use this mountable app system to use

it.

It's also middleware centric. So, those of
you who

are familiar with Rack, know that there's
a lot

of kind of, there's this concept of middleware
-

it's also in Rails - where you can kind

of write these small bits of code that are

Rack compatible, that sit on the stack of
what

you do, of how requests and responses come
into

your Rails app, or your Sinatra app or any

Rack app.

And there's also this built-in Admin area.
And that

Admin area is actually an app. That's just
another

mountable app. So this is, this is something
that

Django has. I know we have Rails Admin here

in, in the Rails world. But with Padrino,
this

is a thing that is built into the framework

itself.

So, why Padrino? Why did I use Padrino in

the beginning? Essentially, at the time, I
was a

seasoned Rails developer. I was probably developing
for about

five years on Rails. And during that time
I

started to form my own opinions about Rails.
And

some of those opinions were not compatible
with what

the, the, the Rails way prescribed. And I
saw

Sinatra and Padrino as this way that I could

still write Ruby, still, I still loved writing
Ruby,

but I could also make my own choices. And

there's this kind of epiphany that seasoned
web developers

kind of ultimately have, which is, I'm writ-
I'm

using this web framework, whether it be Rails
or

Django or Sinatra or Node, that all it's doing

at the end, is it's just, it's really just

a web server. Cause at the very end, you're

just getting in responses, you're just emitting
HTML or

CSS or JSON, if it's an API, or JavaScript.

That's all you're really doing. And all this
talk

about TDD and good object-ori- domain, object-driven
design, they're,

they're very important. They help us manage
complexity. But

in the end, what does the framework physically
do

in the air quote physical way is it, it

really just takes in HTTP responses, or, excuse
me,

takes in HTTP requests and then responds with
HTTP

responses.

So, while Rails will give you the full foundation

of how to build, build a skyscraper, Padrino
gives

you a foundation, but it also lets you choose

some of the plumbing and make some choices
for

yourself. So the good parts about Padrino
are, is,

it really is just Ruby. It's just Rack. And

if you're a fan of thinking in that mindset,

you'll, you'll really enjoy it. There is less
magic.

Things are more explicit. It's unopinionated.
All the generators,

when you, when you generate a Padrino app,
you

specifically say what you want. That you want
ActiveRecord

versus DataMapper versus Monoid or you want
Sass versus

Lest versus ERB. Whatever, whatever you want,
you can

specify it.

I actually enjoy the process of writing middleware.
I

like thinking about web apps like that. I
think

it's a much more performant way to think about

writing web apps. And Padrino itself, unlike
Rails is,

it's light, it's performant. It's really just
Sinatra with

a few more libraries on top of it.

So this is what our architecture looked like
when

we launched. So the whole big box is a

Padrino, is the Padrino app. And we had two

mounted apps inside it: main, the public site;
so

when you visited Upworthy dot com, this is
what

the public would see, those, those content
pages; and

then we had the admin tool, the built-in Admin

app in Padrino, which essentially functioned
as our CMS.

And keep in mind that we, we were hoping

that we're gonna launch this thing and then
we're

gonna get lots of traffic. So we needed to

figure out how to scale it right away, right

from the get go.

So, I kind of devised this, this idea called

explicit caching, and I remembered back in
the early

2000s, there was this blogging framework called,
or blogging

tool called Moveable Type. And Moveable Type
was what

all the big, those early blogs used. And Moveable

Type essentially, the way it worked is, it
was

a CMS. So when you saved your post to

the database, Moveable Type would actually
save, obviously, see

that you saved something to the database.
And then

it render the HTML right then, and then write

the HTML to disc. So when your, when people

visited your blog that was hosted on Moveable
Type,

they weren't hitting that Perl app and going
through

the database. They were actually just hitting
these rendered

HTML files and CSS, JavaScript, that were
just living

on the file system of your server.

So I, I actually was drawn to that idea.

I liked that idea. So I kind of re,

re-made it a little bit here in Padrino. So

in the Admin app, there was this publisher
object.

And the publisher object essentially did that,
where once

anything was saved to the CMS, any content
was

saved to the CMS, the publisher object would
see

that. It would actually make a request to
the

Main app, which was actually rendering the
public site,

and it would write that rendered response
to Redis.

And so RedisCache was a middleware layer - I

talked about middleware earlier - that sat
very close

to the front of this Padrino app.

So, when we had a million or so pages

in that first month, they were all really
being

served from Redis. Essentially the website
was just in

memory in Redis. And so that worked, you know,

it scaled well.

So, around this time, June, 2012, we hired
a

second Rails engineer, Josh French. So, he
kind of

joined the team, and then a few weeks later

he said guys, I think we should switch to

Rails. And he was, he was probably right.
Because

there were pain points that were not related,
really,

to the technical performance of Padrino, but
actually more

about social aspects of it. The first one
being

that the ecosystem for libraries, while, while
pretty good,

because, again, Padrino is just Sinatra, just
Rack, was

not as strong as Rails. There's just libraries
for

everything you want to do in Rails. We, there

were many things we could do in Padrino, but

the, the kind of quality of those libraries
was

not as high. A part of that is because

Padrino isn't as popular, it wasn't very frequently
maintained.

The actual Admin app was not very pleasant
to

look at. It was its own HTML, CSS styling.

I put a star here because, literally, once,
the

sec, the day we moved off of Padrino fully,

they released a new, a new release which was,

which had the Admin system in Bootstrap, which
was

what we wanted all along. And there was no

community and, as a start up, it's like actually

easier to hire Rails developers, cause we
can't really

go hey, we know you're a great Rails developer

but you're gonna have to work on this thing

called Padrino. That wasn't really a, a strong
sell.

So we decided to move. We wanted to move

to Rails. But at the same time, we're, we're

a growing start up. We're getting lots of
traffic.

So how do we kind of balance this desire

to move to, move our architecture while still,
you

know, maintaining a running app, while still
having to

maintain a stable, running app that is serving
a

growing traffic base. And Ryan's gonna continue.

R.R.: So, we started our Rails migration in
October

of 2012, and so this is a timeline. This

is October 2012. And basically the way it
started,

we generated a new Rails app, moved all the,

we mounted it inside the routes.rb, so we
just

basically mounted, and you can do the same
thing

with Sinatra, since it's Rack. And then we
slowly

the migrated the models and utilities into
the app.

So, when I joined, we were kind of in

this weird hybrid state. I joined in January
of

2013, after taking a nice long siesta, re-electing
the

president stuff. And so, we had to figure
out,

how could we accelerate and just, you know,
get

us over the file hurdle and just move us

onto Rails completely and get out of Padrino.
So

the first step that I did was migrating assets,

and so we activated the Rails asset pipeline
which

had been turned off in our app. Migrated the

frontend assets, the backend assets. That
took us about

a week in February, 2013. The next step was

deciding if we wanted to do the admin area

or our frontend area first. So we decided
to

do the hard part first and do the frontend.

So we migrated all the frontend code. The
views

and controllers, which took another two weeks.
And then

lastly we, we did the entire backend CMS system

as a final push, and we added, we changed

it to bootstrap. Moved all the Rails controllers.
That

took us another two weeks.

So, here at this point, the entire migration
is

eight months. But it's really, it really ends
up

getting accelerated in the last few weeks
just because

we wanted to get to that, that final push.

And at this point, we're at three Rails developers:

me, myself, Luigi, and, sorry, myself, Josh,
and Luigi,

and our CTO goes back to actually doing CTO

work, which is great.

So now here we are. We're in a monolith.

We're in a big huge monorail for the entire

2013, we were able to do things the Rails

way. We were able to increase our velocity
and

just able to add lots of features. We were

able to program the way we wanted to and

really get things moving. We didn't have to
rebuild

helpers that weren't, weren't existing. So
we could just

have this one huge monolithic Rails app. We
had

the backend CMS, and then we had our frontend

and we had all our Ajax endpoints.

But one of the things, when you're looking
at

this monorail, is how are you scaling for
virility?

So on the campaign, there was a lot of

traffic, and it was pretty easy to know that,

you know, in November is gonna be really busy,

or in October there's a voter deadline so
it's

gonna be very specific of when traffic's gonna
hit.

You could, you could tell. In, in the virile

media world, you don't know if your post is

gonna be a hit or not. You don't know

when it's gonna get traction. And so you can't

have someone sitting there monitoring twenty-four
hours a day,

when to scale, when to not scale. So, we

had to think about how we were gonna do

that.

So a lot of it was just pretty simple

basic stuff here. We added action caching.
So we

took, we removed the homegrown publisher system,
just turned

on action caching in Rails, and it's backed
by

memcache. So people would hit the page, you
would

hit the memcached instance of the page instead
of

going out, hitting our database, pulling in
the page.

We were able to just do that.

The second part was just moving our assets
on

S3 and Cloudfront, so our app is hosted on

Heroku. There's actually, it's a really easy
tutorial on

Heroku on how to do this. You just set

up a Cloudfront instance and then you just
point

your config host for the, the CDM for your

assets to that and it magically works. It's
great.

And then the third thing is, we have lots

of Heroku dynos. So at the time we were

running up to forty Heroku dynos, and these
were

1x dynos at the time.

So mainly these are for our Ajax requests
so

that those asks, those popups, and the different
things

that ask you around the site, we were able

to scale with those. So we ran this for

awhile, and we, we had some slowness on the

site sometimes, so we tried to figure out
what

could we do to make sure that our site

would be stable and not have to worry about

these viral traffic spikes, having to scale
up and

down. So we actually implemented a CDN in
front

of it. We took some time and tried to

figure out what CDN we wanted. Because we
wanted

to do faster posts and just different things,
and

at the time we ended up on Fastly. So

Fastly is a reverse proxy. It runs on Varnish.

It's made, you guys could check it out. It's

great.

We, we moved all of that to, all our

HTML, CSS, and images to Fastly, and then
we

turned off the Rails action caching. And the
way

Fastly works is, is it reads your headers
on

your page, so you just set, set expire four

hours from now. So our site could literally
be

down for four hours and Fastly would continue
to

serve the pages.

From there, we were able to dial down our

Heroku dynos. So we went to, we switched to

the 2x dynos, and we only needed about half

as many dynos, because we were only serving,
off

the Heroku dyno for Ajax requests.

Probably the biggest thing that we learned
from Fastly

was the mobile performance gain. So Fastly
has all

these different locations around the world.
If I'm in

California requesting a page from Upworthy
dot com, it's

gonna go to the closest point of presence,
the

CDN location in California instead of going
out to

Heroku in Virginia, pulling from the data
center and

bringing it back. So the, the load times went

way down, and we're able to fully cache the

site, and we have, we've had zero downtime
since

implementing Fastly. So it's just been a great
performance

gain for us.

So, with having a big monorail, there's huge
pain

points that come along with it. We have, so

we have one app that deals with our www

service, Upworthy dot com, and then we have
a

backend CMS that our, our curators log into
and

do their work. So we had to figure out

the concerns with that. So it's, it's really
painful

there.

When there's traffic spikes on the public
site, it

could basically, it makes our CMS unstable,
so the

curator would log into our site, try to do

their work, and they couldn't navigate and
we'd just

have to tell them, you know, come back later

or come and do the work later. And when

the traffic spike comes down you'll be able
to

use it. And as our team grows, the code

base was becoming very large. So the classes
would

get huge and, you know, the frontend didn't
care

about some of the stuff the backend did. So

it just got harder and harder to maintain.

So, of course, what did we do? We break

up here. Fun fact. This is actually a film

in Chicago. In December, 2013, our buddy Josh
French

has another great idea. He says hey, I think

we should really start splitting up the Rails
app.

And so, if you look how this timeline is,

it's pretty evenly spaced. We didn't just
jump into

different things and start building them.
We like took

some time on each of these sections and really

focused on, on that narrow gap.

So, one of the things, when you're trying
to

decide how to break up your Rails app into

services, how do you do it? There's plenty
of

different ways you can do it. This is the

way we did it. This is not the perfect

prescription for everybody. I think you have
to look

at your app and see, like, where the clear

dividing lines are. So we basically, we just
chose

two right now.

So we have our www site and we have

our backend CMS. So there's a clear dividing
line

between the two. What we ended up doing is

cloning each repo into its own separate repository,
so

we could maintain the git repo history, and
then

from there we started breaking everything
up, right. So

this is what we need for this side and

this is what we need for this side and

let's start chopping.

Which was, ended up being a lot of deleting

and removing namespaced objects. So, once
we did that,

we deployed each of these apps to, to separate

Heroku applications. The nice thing about
Heroku is, they

have this function called Heroku fork, so
they'll just

take your current app, it'll fork it into
another

Heroku app, pull over all your plugins, everything
you

need. So we did that. We forked the app,

our main, our main app into two separate applications.

Removed the plugins that we didn't need on
each

side, and then we pushed out our applications
to

those Heroku sites. Everything's working great.
And all we

had to do was switch our Fastly endpoint to

point at the new Heroku app at origin and

we're done. So. Zero downtime really helped
there.

And then we continued to de-duplicate the,
the two

apps. So we created this core gem that is

holding a lot of the code that shares between

the two apps. So, our CMS runs on about

two, two 2x Heroku dynos, and now our frontend

app runs about twenty-five to thirty 2x dynos.

This is pretty much what this looks like.
So

we just have an app called www, and then

we have an app called CMS. And then this

gem shares the code between it. People will
hit

the Fastly endpoint between www and, and our
app.

So, what are the real benefits of service
oriented

architecture? I think there's, there's plenty
if you look

and think about it. One of the big things

is, we talked about the instability. You know,
if

our curators couldn't do their work, they,
you know,

we can't have like articles go out, so it

gets kind of annoying. So if there's a problem

on one app, it's, it's easier to fix and

maintain.

Each of them have different scaling needs.
So the

interesting thing is, our CMS, you know, when
we

have like twenty users that use our CMS, so

we could you know have is at 2x dynos,

instead of like having thirty dynos serving
all these

different pages. So the scaling needs was
really beneficial

here.

And that also divides our teamwork more naturally.
So,

we can have people, we can have engineers
on

a team decide to, you know, work on different

parts or features, and we don't have to wait

for something to ship or something else to
finish.

But of course, there's, you know, there's
a bunch

of drawbacks running an SOA. If you're on
the

full stack and you want to make a feature

on our CMS that needs to carry all the

way through the funnels of our frontend, you
have

to have both, all three environments running
on our

system to make sure that, you know, your change

goes and it funnels all the way through.

Coordinating deploys is also a huge pain.
So if

you need something that's in the core gem
plus

the CMS plus the www app, that means you

have to deploy each three, coordinate them
and make

sure that all of them are happening at the

same time, or all of them go in a

certain order, which, when you're on the monolith,
it's

really easy to just push one big Rails app

out.

And then migrating to a fully DRY set of

codebases, it's really tedious. Trying to
figure out what,

what needs to go where and where we put

stuff is, it's just been a really hard thing

to do.

So, some of the future work that we're gonna

continue to do on the SOA. We're gonna continue

to add stuff to our core gem. Remove dupli-

de-duplication, which is, it's kind of a pain
sometimes

to figure out where, you know, what things
go

where. And then we're considering making,
breaking up the

app even more. So, right now we have two

apps. We have this third app that actually
uses

up almost all our Heroku dynos. So we're thinking

about making that its own separate app and
service,

and we can ping that.

And then, you know, they all communicate with
these

different data stores. Should we just use,
a lot

of times an SOA has that service layer, and

everything communicates with that service
layer. So maybe we

should move to that.

L.M.: Cool. So. Just to wrap up here, some

lessons learned. So, first really, when you're,
when you're

working on an app for a company and you're,

you're, there's a feature request and there's
all these

other things going on in your company, really
do

wait to make these big technical architectural
changes until

everything starts to really, really hurt.
Until you really

feel the pain. And once you do decide to

make these changes, it's OK to take a really

long time. We're probably gonna take eight
months to

do that fully SOA system that we envisioned
back

in the beginning of the year. And that's just

because we have other feature requests from
the rest

of the company to fulfill. And luckily as,
since

we're on Ruby, it kind of makes it easier

to do that. It really made it easier when

we were moving from Padrino to Rails.

Serve everything you possibly can from a CDN.
I

don't think people use CDN's enough. They're
just hugely

beneficial to the performance of your system.
They're great

for end users, particularly mobile users.
At Upworthy, we

have well over fifty percent of our traffic
comes

from phones and tablets. And CDNs really,
really help

there.

So remember that you're just dealing with
HTML, CSS,

JavaScript, JSON, maybe, at the end of the
day.

Right. That's all you're serving. So think
about how

you can most efficiently serve those resources
from your

app. And if you are, you're doing your own

start up, if you're working on your own project,

I hope you can also experience this kind of

high traffic growth, because it's been a hugely
fun

and rewarding experience for the both of us.

So, with that, I'm Luigi, he's Ryan. Our slides

and our write up is already on our GitHub

engineering blog. And we will be happy to
answer

questions.