0:00:16.849,0:00:20.700
LUIGI MONTANEZ: So, thank you everyone for[br]coming.

0:00:20.779,0:00:24.509
It's late on Friday and I really appreciate

0:00:24.509,0:00:30.750
all of you sticking around. Obviously Tenderlove[br]is next. So-

0:00:30.750,0:00:32.439
AUDIENCE: Talk louder!

0:00:32.439,0:00:35.879
L.M.: Talk louder? All right. That's be- is[br]that

0:00:35.879,0:00:40.480
better? All right. So, before I get started,[br]I

0:00:40.480,0:00:42.539
just want to actually do a quick plug, cause

0:00:42.539,0:00:45.190
I saw that the session behind me is actually

0:00:45.190,0:00:48.129
on the impostor syndrome. And since you folks[br]are

0:00:48.129,0:00:50.010
not gonna be there to, to see that, I

0:00:50.010,0:00:51.719
just wanted to actually tell you a little[br]bit

0:00:51.719,0:00:53.670
about it, cause actually it's a, it's a really

0:00:53.670,0:00:56.929
important concept. So, essentially, the impostor[br]syndrome is when

0:00:56.929,0:01:01.649
successful people like software developers[br]feel like they might

0:01:01.649,0:01:04.890
not deserve all the success they've gotten.[br]And it's

0:01:04.890,0:01:07.729
actually very common. When I learned about[br]it a

0:01:07.729,0:01:09.280
few years ago, it was actually really helpful[br]for

0:01:09.280,0:01:11.159
me and my career. So when you go to

0:01:11.159,0:01:14.159
conferences like these and you see members[br]of the

0:01:14.159,0:01:16.670
Rails core team or the Ember core team or

0:01:16.670,0:01:19.950
all the speakers and you think, wow, I am

0:01:19.950,0:01:22.090
not worthy, actually you are. You, you really[br]do

0:01:22.090,0:01:25.939
belong here, and, and people who are successful[br]usually

0:01:25.939,0:01:28.939
do deserve it. So Google the impostor syndrome[br]when

0:01:28.939,0:01:31.740
you get home or, or watch the video when

0:01:31.740,0:01:32.770
it gets posted.

0:01:32.770,0:01:37.460
So to our talk. So. You will never believe

0:01:37.460,0:01:41.299
which web framework powers Upworthy. I, I[br]can't believe

0:01:41.299,0:01:45.590
the conference organizers have made everyone[br]wait all week

0:01:45.590,0:01:49.939
to figure this out, to find this out. So,

0:01:49.939,0:01:52.100
at Upworthy we aim to please. So we're just

0:01:52.100,0:01:55.740
gonna spill the beans right away. Obviously[br]the answer

0:01:55.740,0:01:57.499
is Java Struts.

0:01:57.499,0:01:59.490
AUDIENCE: [applause]

0:01:59.490,0:02:04.789
L.M.: Awesome. Now. So. To introduce myself,[br]my name's

0:02:04.789,0:02:06.840
Luigi. I'm the founding engineer at Upworthy.[br]I was

0:02:06.840,0:02:11.510
the first engineer hired. I've always been[br]a software

0:02:11.510,0:02:14.930
developer involved in politics and advocacy.[br]I got really

0:02:14.930,0:02:18.390
into this guy who screened Howard Dean. And[br]so

0:02:18.390,0:02:21.020
I worked for his political campaign. There[br]was this

0:02:21.020,0:02:23.970
political action committee. I worked for other[br]campaigns and

0:02:23.970,0:02:26.360
non-profits, and then before coming to Upworthy[br]I worked

0:02:26.360,0:02:30.000
for the Sunlight Foundation, which is a great[br]non-profit

0:02:30.000,0:02:33.550
in D.C. that focuses on transparency and government,[br]and

0:02:33.550,0:02:35.230
open, open government data.

0:02:35.230,0:02:38.070
RYAN RESELLA: I'm Ryan Resella. I'm the senior[br]engineer

0:02:38.070,0:02:40.440
at Upworthy. Before this in 2011 I was a

0:02:40.440,0:02:42.600
Code for America fellow, the first class of[br]fellows.

0:02:42.600,0:02:45.460
I came on as technical lead full time staff

0:02:45.460,0:02:48.220
there. And then last year I was on the

0:02:48.220,0:02:53.140
Obama for America tech team, or, I guess 2012,

0:02:53.140,0:02:55.650
working as a software engineer. And I ran[br]out

0:02:55.650,0:02:57.670
of for America organizations to work for so[br]I

0:02:57.670,0:02:59.200
joined Upworthy.

0:02:59.200,0:03:04.230
L.M.: So, at Upworthy, our mission, and we,[br]this

0:03:04.230,0:03:06.430
is something we truly really believe at the[br]company

0:03:06.430,0:03:09.060
is to drive massive amounts of attention to[br]the

0:03:09.060,0:03:12.530
topics that matter most. And that will kind[br]of

0:03:12.530,0:03:15.500
inform the engineering decisions we made as[br]the, as

0:03:15.500,0:03:18.250
the tech team. So, just to kind of give

0:03:18.250,0:03:21.980
people a little peak at what Upworthy does,[br]for

0:03:21.980,0:03:25.050
those who aren't too familiar, so this might[br]be

0:03:25.050,0:03:26.300
a bit hard to read, but when we say

0:03:26.300,0:03:29.220
that topics that matter most, these are kind[br]of

0:03:29.220,0:03:31.310
the topics we've covered in the last year,[br]the

0:03:31.310,0:03:33.430
topics that have gotten the most attention.[br]So I'll

0:03:33.430,0:03:35.450
just read some of them aloud.

0:03:35.450,0:03:39.180
There's a lot of women's issues, like body[br]image,

0:03:39.180,0:03:43.640
gender inequality, standards of beauty, a[br]lot of economic

0:03:43.640,0:03:48.560
issues like income inequality, health policy.[br]We also cover

0:03:48.560,0:03:51.480
a lot of stuff about disability, mental health.[br]Also

0:03:51.480,0:03:57.260
bullying, bigotry, racial profiling and, and[br]race issues. And

0:03:57.260,0:03:59.440
when we say that we want to drive massive

0:03:59.440,0:04:03.260
amounts of attention to these things, what[br]we really

0:04:03.260,0:04:07.360
mean as web developers, as web engineers is[br]we

0:04:07.360,0:04:10.970
want to drive massive amounts of traffic.

0:04:10.970,0:04:13.650
So here's a look at our growth for the

0:04:13.650,0:04:15.540
last two years. So we launched a little over

0:04:15.540,0:04:22.540
two years ago. We started off at, around one

0:04:24.460,0:04:30.820
point seven million uniques per month. That[br]was in

0:04:30.820,0:04:36.660
our first ever month in April 2012. And then

0:04:36.660,0:04:41.720
we went up to, in November of 2013, 190

0:04:41.720,0:04:46.500
million page views. So, this has made us probably

0:04:46.500,0:04:48.930
around the top forty, a top forty site in

0:04:48.930,0:04:53.280
the U.S. We're maybe one of the larger Rails,

0:04:53.280,0:04:56.570
more trafficked Rails apps out there.

0:04:56.570,0:04:57.640
To give you a sense of what kind of

0:04:57.640,0:05:00.030
traffic we actually deal with, here's a twenty-four[br]hour

0:05:00.030,0:05:03.180
graph of what our page view data looks like.

0:05:03.180,0:05:04.810
So starting at midnight all the way on the

0:05:04.810,0:05:08.370
left, and then ending midnight the next day,[br]you

0:05:08.370,0:05:10.280
can kind of say how during the, when the

0:05:10.280,0:05:13.230
daytime happens, when, when work hours start,[br]we get

0:05:13.230,0:05:18.090
these spikes, these peaks of, of, of traffic.[br]This

0:05:18.090,0:05:24.370
is essentially viral, the viral cycle being[br]visualized.

0:05:24.370,0:05:31.320
So, we have handled, at most, about 130,000[br]concurrent

0:05:31.320,0:05:34.900
visitors. This is a screenshot from Google[br]analytics during

0:05:34.900,0:05:37.750
one of the traffic spikes. So, you know, we,

0:05:37.750,0:05:40.240
we are handling large amounts of traffic in[br]very

0:05:40.240,0:05:43.560
spiky ways.

0:05:43.560,0:05:46.470
So here is a example post from Upworthy. This

0:05:46.470,0:05:50.430
was really popular about, a few months ago,[br]in

0:05:50.430,0:05:53.310
the fall really. Who here remembers this post?[br]Just

0:05:53.310,0:05:56.210
curious. Cool. A few of you.

0:05:56.210,0:05:59.530
So this is what Upworthy content looks like.[br]So

0:05:59.530,0:06:01.440
it's really just static content. So See Why[br]We

0:06:01.440,0:06:03.990
Have An Absolutely Ridiculously Standard of[br]Beauty in Just

0:06:03.990,0:06:07.280
37 Seconds. It's a video about how a, a

0:06:07.280,0:06:12.669
woman, a model gets photoshopped and looks[br]essentially like

0:06:12.669,0:06:15.310
something, like the standard of beauty that[br]doesn't really

0:06:15.310,0:06:17.590
exist. So that, that's kind of the content,[br]or

0:06:17.590,0:06:20.230
angle we were going for. And here you see,

0:06:20.230,0:06:24.139
we have the content, which is basically static[br]content

0:06:24.139,0:06:26.110
on the left side of the screen. We have

0:06:26.110,0:06:28.669
this sidebar with recommended content on the[br]right side

0:06:28.669,0:06:31.450
of the screen. And then scrolling down, we[br]have

0:06:31.450,0:06:34.919
what we call asks. We, we also do kind

0:06:34.919,0:06:39.110
of some testing on different kinds of content,[br]different

0:06:39.110,0:06:41.860
kinds of headlines. You see that down there[br]with

0:06:41.860,0:06:44.919
that John Stewart video. And then we have[br]asks

0:06:44.919,0:06:47.960
around, do you want to subscribe to our email

0:06:47.960,0:06:50.020
list? Do you want to like us on Facebook?

0:06:50.020,0:06:53.430
We also have kind of pop-ups that happen after

0:06:53.430,0:06:57.480
you watch a video, after you share, also asking

0:06:57.480,0:06:59.389
you to do stuff.

0:06:59.389,0:07:01.940
So those are the things we, we, the technical

0:07:01.940,0:07:05.310
things we, the technical concerns we have[br]at Upworthy.

0:07:05.310,0:07:08.169
We're pretty much a static site. We're a public

0:07:08.169,0:07:11.190
site. And then we have a CMS backing that,

0:07:11.190,0:07:14.300
and then we have this system of dynamic pop-ups

0:07:14.300,0:07:17.639
and asks around the site that kind of optimize

0:07:17.639,0:07:20.480
our subscriber base, get us more subscribers,[br]get, get

0:07:20.480,0:07:23.690
folks to share content.

0:07:23.690,0:07:28.160
So, the topic of this talk will really be

0:07:28.160,0:07:31.840
about managing the growth of our start up's[br]web

0:07:31.840,0:07:37.260
app in the face of very high traffic. And

0:07:37.260,0:07:41.120
I actually remember maybe five years ago sitting[br]at

0:07:41.120,0:07:44.230
a RailsConf, maybe it was in Baltimore, and[br]I

0:07:44.230,0:07:45.980
was sitting where you're sitting and it was[br]a

0:07:45.980,0:07:47.820
talk, it was a talk by the guys from

0:07:47.820,0:07:51.620
YellowPages dot com. And YellowPages dot com[br]was and

0:07:51.620,0:07:54.669
still is one of the larger Rails sites in

0:07:54.669,0:07:58.150
the world. Obviously YellowPages is such a[br]strong brand.

0:07:58.150,0:07:59.940
Everyone goes to the YellowPages, a lot of[br]people

0:07:59.940,0:08:02.990
still use YellowPages dot com to find out[br]local

0:08:02.990,0:08:04.810
businesses and stuff like that. And they were[br]talking

0:08:04.810,0:08:07.870
about how they scaled their app, their Rails[br]app.

0:08:07.870,0:08:09.790
And I was thinking, I was sitting there in

0:08:09.790,0:08:11.960
the audience, thinking well, this is really[br]interesting, but

0:08:11.960,0:08:14.570
I'm not sure this is ever gonna apply to

0:08:14.570,0:08:16.360
me. I don't really, I work on these small

0:08:16.360,0:08:19.340
things. No one ever really sees them. But[br]fast

0:08:19.340,0:08:21.250
forward a few years and, you know, here I

0:08:21.250,0:08:24.300
am. I'm, I'm working on this, this app that

0:08:24.300,0:08:27.090
millions of people see every day. So it can

0:08:27.090,0:08:29.669
really happen to you too.

0:08:29.669,0:08:36.000
So, let's start from the beginning. We launched[br]in

0:08:36.000,0:08:41.120
early 2012, March 26th, 2012 to be exact.[br]And

0:08:41.120,0:08:44.300
at the time there was one engineer, me, and

0:08:44.300,0:08:47.569
our CTO who is a programmer, but was not

0:08:47.569,0:08:52.290
really a Ruby or Rails developer. And we actually

0:08:52.290,0:08:54.589
launched not on Rails but on Padrino. Who[br]here

0:08:54.589,0:08:59.269
is familiar with Padrino? Cool. Who here has[br]production

0:08:59.269,0:09:04.809
apps in Padrino? No one. OK. That's what I

0:09:04.809,0:09:04.899
thought.

0:09:04.899,0:09:06.819
So Padrino, this is, it's, it kind of builds

0:09:06.819,0:09:10.579
itself as the elegant web framework. It is[br]essentially

0:09:10.579,0:09:14.839
just Sinatra with more Rails-like things on[br]top of

0:09:14.839,0:09:17.540
it. So he, who here has used Sinatra? More

0:09:17.540,0:09:20.899
hands. Of course. And who here actually has[br]Sinatra

0:09:20.899,0:09:24.269
in production? A few, a few hands. Yeah.

0:09:24.269,0:09:27.240
So, essentially, when you're working with[br]Sinatra, it's a,

0:09:27.240,0:09:32.269
it's a more low-level library. More closer[br]to Rack.

0:09:32.269,0:09:36.300
And Padrino, essentially, adds the things[br]that you miss

0:09:36.300,0:09:43.300
from Rails into Sinatra. It also freely borrowed[br]some

0:09:43.819,0:09:48.639
really great ideas from Djando. The first[br]one being

0:09:48.639,0:09:52.670
first-class mountable apps. So in Rails we[br]have engines.

0:09:52.670,0:09:55.110
But it seems like not, people don't really[br]use

0:09:55.110,0:09:57.279
engines that often. Like, you might use it[br]for

0:09:57.279,0:10:01.730
Rails admin. It's, you kind of might have[br]a

0:10:01.730,0:10:04.779
larger Rails app and then break it up by

0:10:04.779,0:10:10.610
putting it into separate engines. But with[br]Padrino, all

0:10:10.610,0:10:13.399
code actually lives in a mountable app. So[br]you

0:10:13.399,0:10:15.670
have to use this mountable app system to use

0:10:15.670,0:10:18.619
it.

0:10:18.619,0:10:21.910
It's also middleware centric. So, those of[br]you who

0:10:21.910,0:10:24.050
are familiar with Rack, know that there's[br]a lot

0:10:24.050,0:10:26.089
of kind of, there's this concept of middleware[br]-

0:10:26.089,0:10:29.360
it's also in Rails - where you can kind

0:10:29.360,0:10:32.749
of write these small bits of code that are

0:10:32.749,0:10:36.230
Rack compatible, that sit on the stack of[br]what

0:10:36.230,0:10:40.059
you do, of how requests and responses come[br]into

0:10:40.059,0:10:43.420
your Rails app, or your Sinatra app or any

0:10:43.420,0:10:44.910
Rack app.

0:10:44.910,0:10:49.439
And there's also this built-in Admin area.[br]And that

0:10:49.439,0:10:52.639
Admin area is actually an app. That's just[br]another

0:10:52.639,0:10:56.160
mountable app. So this is, this is something[br]that

0:10:56.160,0:10:59.119
Django has. I know we have Rails Admin here

0:10:59.119,0:11:01.600
in, in the Rails world. But with Padrino,[br]this

0:11:01.600,0:11:03.100
is a thing that is built into the framework

0:11:03.100,0:11:04.790
itself.

0:11:04.790,0:11:09.040
So, why Padrino? Why did I use Padrino in

0:11:09.040,0:11:14.209
the beginning? Essentially, at the time, I[br]was a

0:11:14.209,0:11:16.519
seasoned Rails developer. I was probably developing[br]for about

0:11:16.519,0:11:20.689
five years on Rails. And during that time[br]I

0:11:20.689,0:11:23.970
started to form my own opinions about Rails.[br]And

0:11:23.970,0:11:27.749
some of those opinions were not compatible[br]with what

0:11:27.749,0:11:31.910
the, the, the Rails way prescribed. And I[br]saw

0:11:31.910,0:11:34.720
Sinatra and Padrino as this way that I could

0:11:34.720,0:11:38.949
still write Ruby, still, I still loved writing[br]Ruby,

0:11:38.949,0:11:43.689
but I could also make my own choices. And

0:11:43.689,0:11:50.679
there's this kind of epiphany that seasoned[br]web developers

0:11:50.679,0:11:54.079
kind of ultimately have, which is, I'm writ-[br]I'm

0:11:54.079,0:11:56.459
using this web framework, whether it be Rails[br]or

0:11:56.459,0:12:00.300
Django or Sinatra or Node, that all it's doing

0:12:00.300,0:12:02.970
at the end, is it's just, it's really just

0:12:02.970,0:12:04.889
a web server. Cause at the very end, you're

0:12:04.889,0:12:08.559
just getting in responses, you're just emitting[br]HTML or

0:12:08.559,0:12:12.569
CSS or JSON, if it's an API, or JavaScript.

0:12:12.569,0:12:14.480
That's all you're really doing. And all this[br]talk

0:12:14.480,0:12:21.480
about TDD and good object-ori- domain, object-driven[br]design, they're,

0:12:23.230,0:12:26.589
they're very important. They help us manage[br]complexity. But

0:12:26.589,0:12:30.149
in the end, what does the framework physically[br]do

0:12:30.149,0:12:33.339
in the air quote physical way is it, it

0:12:33.339,0:12:37.889
really just takes in HTTP responses, or, excuse[br]me,

0:12:37.889,0:12:41.739
takes in HTTP requests and then responds with[br]HTTP

0:12:41.739,0:12:42.300
responses.

0:12:42.300,0:12:45.769
So, while Rails will give you the full foundation

0:12:45.769,0:12:51.189
of how to build, build a skyscraper, Padrino[br]gives

0:12:51.189,0:12:52.879
you a foundation, but it also lets you choose

0:12:52.879,0:12:56.170
some of the plumbing and make some choices[br]for

0:12:56.170,0:12:59.739
yourself. So the good parts about Padrino[br]are, is,

0:12:59.739,0:13:01.679
it really is just Ruby. It's just Rack. And

0:13:01.679,0:13:05.230
if you're a fan of thinking in that mindset,

0:13:05.230,0:13:09.509
you'll, you'll really enjoy it. There is less[br]magic.

0:13:09.509,0:13:15.869
Things are more explicit. It's unopinionated.[br]All the generators,

0:13:15.869,0:13:18.029
when you, when you generate a Padrino app,[br]you

0:13:18.029,0:13:22.850
specifically say what you want. That you want[br]ActiveRecord

0:13:22.850,0:13:28.459
versus DataMapper versus Monoid or you want[br]Sass versus

0:13:28.459,0:13:33.360
Lest versus ERB. Whatever, whatever you want,[br]you can

0:13:33.360,0:13:34.860
specify it.

0:13:34.860,0:13:37.660
I actually enjoy the process of writing middleware.[br]I

0:13:37.660,0:13:39.639
like thinking about web apps like that. I[br]think

0:13:39.639,0:13:41.759
it's a much more performant way to think about

0:13:41.759,0:13:46.449
writing web apps. And Padrino itself, unlike[br]Rails is,

0:13:46.449,0:13:49.470
it's light, it's performant. It's really just[br]Sinatra with

0:13:49.470,0:13:53.399
a few more libraries on top of it.

0:13:53.399,0:13:55.989
So this is what our architecture looked like[br]when

0:13:55.989,0:13:58.129
we launched. So the whole big box is a

0:13:58.129,0:14:05.049
Padrino, is the Padrino app. And we had two

0:14:05.049,0:14:07.499
mounted apps inside it: main, the public site;[br]so

0:14:07.499,0:14:09.879
when you visited Upworthy dot com, this is[br]what

0:14:09.879,0:14:13.699
the public would see, those, those content[br]pages; and

0:14:13.699,0:14:17.350
then we had the admin tool, the built-in Admin

0:14:17.350,0:14:23.360
app in Padrino, which essentially functioned[br]as our CMS.

0:14:23.360,0:14:26.290
And keep in mind that we, we were hoping

0:14:26.290,0:14:28.089
that we're gonna launch this thing and then[br]we're

0:14:28.089,0:14:32.100
gonna get lots of traffic. So we needed to

0:14:32.100,0:14:34.429
figure out how to scale it right away, right

0:14:34.429,0:14:35.459
from the get go.

0:14:35.459,0:14:41.899
So, I kind of devised this, this idea called

0:14:41.899,0:14:46.850
explicit caching, and I remembered back in[br]the early

0:14:46.850,0:14:50.279
2000s, there was this blogging framework called,[br]or blogging

0:14:50.279,0:14:53.110
tool called Moveable Type. And Moveable Type[br]was what

0:14:53.110,0:14:57.220
all the big, those early blogs used. And Moveable

0:14:57.220,0:14:59.509
Type essentially, the way it worked is, it[br]was

0:14:59.509,0:15:01.790
a CMS. So when you saved your post to

0:15:01.790,0:15:07.009
the database, Moveable Type would actually[br]save, obviously, see

0:15:07.009,0:15:09.519
that you saved something to the database.[br]And then

0:15:09.519,0:15:12.699
it render the HTML right then, and then write

0:15:12.699,0:15:17.939
the HTML to disc. So when your, when people

0:15:17.939,0:15:20.279
visited your blog that was hosted on Moveable[br]Type,

0:15:20.279,0:15:22.679
they weren't hitting that Perl app and going[br]through

0:15:22.679,0:15:25.949
the database. They were actually just hitting[br]these rendered

0:15:25.949,0:15:30.619
HTML files and CSS, JavaScript, that were[br]just living

0:15:30.619,0:15:33.699
on the file system of your server.

0:15:33.699,0:15:35.920
So I, I actually was drawn to that idea.

0:15:35.920,0:15:39.549
I liked that idea. So I kind of re,

0:15:39.549,0:15:42.489
re-made it a little bit here in Padrino. So

0:15:42.489,0:15:45.439
in the Admin app, there was this publisher[br]object.

0:15:45.439,0:15:48.699
And the publisher object essentially did that,[br]where once

0:15:48.699,0:15:51.399
anything was saved to the CMS, any content[br]was

0:15:51.399,0:15:53.999
saved to the CMS, the publisher object would[br]see

0:15:53.999,0:15:57.670
that. It would actually make a request to[br]the

0:15:57.670,0:16:02.559
Main app, which was actually rendering the[br]public site,

0:16:02.559,0:16:09.559
and it would write that rendered response[br]to Redis.

0:16:09.899,0:16:12.410
And so RedisCache was a middleware layer - I

0:16:12.410,0:16:15.069
talked about middleware earlier - that sat[br]very close

0:16:15.069,0:16:17.779
to the front of this Padrino app.

0:16:17.779,0:16:21.100
So, when we had a million or so pages

0:16:21.100,0:16:23.239
in that first month, they were all really[br]being

0:16:23.239,0:16:27.160
served from Redis. Essentially the website[br]was just in

0:16:27.160,0:16:30.149
memory in Redis. And so that worked, you know,

0:16:30.149,0:16:32.959
it scaled well.

0:16:32.959,0:16:38.629
So, around this time, June, 2012, we hired[br]a

0:16:38.629,0:16:43.639
second Rails engineer, Josh French. So, he[br]kind of

0:16:43.639,0:16:47.109
joined the team, and then a few weeks later

0:16:47.109,0:16:49.679
he said guys, I think we should switch to

0:16:49.679,0:16:52.869
Rails. And he was, he was probably right.[br]Because

0:16:52.869,0:16:55.489
there were pain points that were not related,[br]really,

0:16:55.489,0:16:58.230
to the technical performance of Padrino, but[br]actually more

0:16:58.230,0:17:02.249
about social aspects of it. The first one[br]being

0:17:02.249,0:17:05.069
that the ecosystem for libraries, while, while[br]pretty good,

0:17:05.069,0:17:09.179
because, again, Padrino is just Sinatra, just[br]Rack, was

0:17:09.179,0:17:11.609
not as strong as Rails. There's just libraries[br]for

0:17:11.609,0:17:14.359
everything you want to do in Rails. We, there

0:17:14.359,0:17:17.679
were many things we could do in Padrino, but

0:17:17.679,0:17:22.260
the, the kind of quality of those libraries[br]was

0:17:22.260,0:17:24.589
not as high. A part of that is because

0:17:24.589,0:17:28.220
Padrino isn't as popular, it wasn't very frequently[br]maintained.

0:17:28.220,0:17:30.789
The actual Admin app was not very pleasant[br]to

0:17:30.789,0:17:33.610
look at. It was its own HTML, CSS styling.

0:17:33.610,0:17:36.419
I put a star here because, literally, once,[br]the

0:17:36.419,0:17:39.440
sec, the day we moved off of Padrino fully,

0:17:39.440,0:17:43.100
they released a new, a new release which was,

0:17:43.100,0:17:44.889
which had the Admin system in Bootstrap, which[br]was

0:17:44.889,0:17:49.470
what we wanted all along. And there was no

0:17:49.470,0:17:52.980
community and, as a start up, it's like actually

0:17:52.980,0:17:55.159
easier to hire Rails developers, cause we[br]can't really

0:17:55.159,0:17:58.019
go hey, we know you're a great Rails developer

0:17:58.019,0:17:59.470
but you're gonna have to work on this thing

0:17:59.470,0:18:06.470
called Padrino. That wasn't really a, a strong[br]sell.

0:18:06.970,0:18:09.710
So we decided to move. We wanted to move

0:18:09.710,0:18:11.880
to Rails. But at the same time, we're, we're

0:18:11.880,0:18:15.260
a growing start up. We're getting lots of[br]traffic.

0:18:15.260,0:18:17.690
So how do we kind of balance this desire

0:18:17.690,0:18:23.620
to move to, move our architecture while still,[br]you

0:18:23.620,0:18:27.360
know, maintaining a running app, while still[br]having to

0:18:27.360,0:18:29.480
maintain a stable, running app that is serving[br]a

0:18:29.480,0:18:33.500
growing traffic base. And Ryan's gonna continue.

0:18:33.500,0:18:39.590
R.R.: So, we started our Rails migration in[br]October

0:18:39.590,0:18:43.399
of 2012, and so this is a timeline. This

0:18:43.399,0:18:47.570
is October 2012. And basically the way it[br]started,

0:18:47.570,0:18:51.850
we generated a new Rails app, moved all the,

0:18:51.850,0:18:53.960
we mounted it inside the routes.rb, so we[br]just

0:18:53.960,0:18:56.750
basically mounted, and you can do the same[br]thing

0:18:56.750,0:19:00.320
with Sinatra, since it's Rack. And then we[br]slowly

0:19:00.320,0:19:04.289
the migrated the models and utilities into[br]the app.

0:19:04.289,0:19:06.429
So, when I joined, we were kind of in

0:19:06.429,0:19:09.080
this weird hybrid state. I joined in January[br]of

0:19:09.080,0:19:11.620
2013, after taking a nice long siesta, re-electing[br]the

0:19:11.620,0:19:17.620
president stuff. And so, we had to figure[br]out,

0:19:17.620,0:19:20.250
how could we accelerate and just, you know,[br]get

0:19:20.250,0:19:21.919
us over the file hurdle and just move us

0:19:21.919,0:19:24.490
onto Rails completely and get out of Padrino.[br]So

0:19:24.490,0:19:28.320
the first step that I did was migrating assets,

0:19:28.320,0:19:30.419
and so we activated the Rails asset pipeline[br]which

0:19:30.419,0:19:32.820
had been turned off in our app. Migrated the

0:19:32.820,0:19:35.750
frontend assets, the backend assets. That[br]took us about

0:19:35.750,0:19:39.010
a week in February, 2013. The next step was

0:19:39.010,0:19:41.760
deciding if we wanted to do the admin area

0:19:41.760,0:19:45.360
or our frontend area first. So we decided[br]to

0:19:45.360,0:19:47.909
do the hard part first and do the frontend.

0:19:47.909,0:19:49.570
So we migrated all the frontend code. The[br]views

0:19:49.570,0:19:54.340
and controllers, which took another two weeks.[br]And then

0:19:54.340,0:19:59.169
lastly we, we did the entire backend CMS system

0:19:59.169,0:20:01.730
as a final push, and we added, we changed

0:20:01.730,0:20:03.860
it to bootstrap. Moved all the Rails controllers.[br]That

0:20:03.860,0:20:06.490
took us another two weeks.

0:20:06.490,0:20:10.100
So, here at this point, the entire migration[br]is

0:20:10.100,0:20:12.850
eight months. But it's really, it really ends[br]up

0:20:12.850,0:20:16.110
getting accelerated in the last few weeks[br]just because

0:20:16.110,0:20:18.889
we wanted to get to that, that final push.

0:20:18.889,0:20:21.809
And at this point, we're at three Rails developers:

0:20:21.809,0:20:26.269
me, myself, Luigi, and, sorry, myself, Josh,[br]and Luigi,

0:20:26.269,0:20:29.909
and our CTO goes back to actually doing CTO

0:20:29.909,0:20:33.080
work, which is great.

0:20:33.080,0:20:34.549
So now here we are. We're in a monolith.

0:20:34.549,0:20:38.139
We're in a big huge monorail for the entire

0:20:38.139,0:20:40.159
2013, we were able to do things the Rails

0:20:40.159,0:20:43.100
way. We were able to increase our velocity[br]and

0:20:43.100,0:20:45.919
just able to add lots of features. We were

0:20:45.919,0:20:49.399
able to program the way we wanted to and

0:20:49.399,0:20:51.309
really get things moving. We didn't have to[br]rebuild

0:20:51.309,0:20:54.639
helpers that weren't, weren't existing. So[br]we could just

0:20:54.639,0:20:57.529
have this one huge monolithic Rails app. We[br]had

0:20:57.529,0:20:59.460
the backend CMS, and then we had our frontend

0:20:59.460,0:21:02.830
and we had all our Ajax endpoints.

0:21:02.830,0:21:05.440
But one of the things, when you're looking[br]at

0:21:05.440,0:21:09.679
this monorail, is how are you scaling for[br]virility?

0:21:09.679,0:21:12.730
So on the campaign, there was a lot of

0:21:12.730,0:21:15.549
traffic, and it was pretty easy to know that,

0:21:15.549,0:21:17.980
you know, in November is gonna be really busy,

0:21:17.980,0:21:20.259
or in October there's a voter deadline so[br]it's

0:21:20.259,0:21:22.630
gonna be very specific of when traffic's gonna[br]hit.

0:21:22.630,0:21:25.169
You could, you could tell. In, in the virile

0:21:25.169,0:21:27.409
media world, you don't know if your post is

0:21:27.409,0:21:28.769
gonna be a hit or not. You don't know

0:21:28.769,0:21:30.580
when it's gonna get traction. And so you can't

0:21:30.580,0:21:33.610
have someone sitting there monitoring twenty-four[br]hours a day,

0:21:33.610,0:21:37.370
when to scale, when to not scale. So, we

0:21:37.370,0:21:38.620
had to think about how we were gonna do

0:21:38.620,0:21:38.730
that.

0:21:38.730,0:21:40.549
So a lot of it was just pretty simple

0:21:40.549,0:21:44.080
basic stuff here. We added action caching.[br]So we

0:21:44.080,0:21:47.070
took, we removed the homegrown publisher system,[br]just turned

0:21:47.070,0:21:49.889
on action caching in Rails, and it's backed[br]by

0:21:49.889,0:21:51.740
memcache. So people would hit the page, you[br]would

0:21:51.740,0:21:54.159
hit the memcached instance of the page instead[br]of

0:21:54.159,0:21:56.700
going out, hitting our database, pulling in[br]the page.

0:21:56.700,0:21:58.539
We were able to just do that.

0:21:58.539,0:22:01.580
The second part was just moving our assets[br]on

0:22:01.580,0:22:04.710
S3 and Cloudfront, so our app is hosted on

0:22:04.710,0:22:07.190
Heroku. There's actually, it's a really easy[br]tutorial on

0:22:07.190,0:22:08.880
Heroku on how to do this. You just set

0:22:08.880,0:22:12.710
up a Cloudfront instance and then you just[br]point

0:22:12.710,0:22:16.299
your config host for the, the CDM for your

0:22:16.299,0:22:18.960
assets to that and it magically works. It's[br]great.

0:22:18.960,0:22:22.169
And then the third thing is, we have lots

0:22:22.169,0:22:24.090
of Heroku dynos. So at the time we were

0:22:24.090,0:22:27.039
running up to forty Heroku dynos, and these[br]were

0:22:27.039,0:22:30.250
1x dynos at the time.

0:22:30.250,0:22:32.919
So mainly these are for our Ajax requests[br]so

0:22:32.919,0:22:35.460
that those asks, those popups, and the different[br]things

0:22:35.460,0:22:37.769
that ask you around the site, we were able

0:22:37.769,0:22:42.220
to scale with those. So we ran this for

0:22:42.220,0:22:45.600
awhile, and we, we had some slowness on the

0:22:45.600,0:22:49.009
site sometimes, so we tried to figure out[br]what

0:22:49.009,0:22:51.639
could we do to make sure that our site

0:22:51.639,0:22:55.509
would be stable and not have to worry about

0:22:55.509,0:22:57.870
these viral traffic spikes, having to scale[br]up and

0:22:57.870,0:23:00.889
down. So we actually implemented a CDN in[br]front

0:23:00.889,0:23:03.049
of it. We took some time and tried to

0:23:03.049,0:23:05.440
figure out what CDN we wanted. Because we[br]wanted

0:23:05.440,0:23:08.269
to do faster posts and just different things,[br]and

0:23:08.269,0:23:10.460
at the time we ended up on Fastly. So

0:23:10.460,0:23:14.190
Fastly is a reverse proxy. It runs on Varnish.

0:23:14.190,0:23:16.919
It's made, you guys could check it out. It's

0:23:16.919,0:23:18.080
great.

0:23:18.080,0:23:22.630
We, we moved all of that to, all our

0:23:22.630,0:23:25.980
HTML, CSS, and images to Fastly, and then[br]we

0:23:25.980,0:23:29.559
turned off the Rails action caching. And the[br]way

0:23:29.559,0:23:33.070
Fastly works is, is it reads your headers[br]on

0:23:33.070,0:23:36.850
your page, so you just set, set expire four

0:23:36.850,0:23:39.649
hours from now. So our site could literally[br]be

0:23:39.649,0:23:42.149
down for four hours and Fastly would continue[br]to

0:23:42.149,0:23:44.870
serve the pages.

0:23:44.870,0:23:46.539
From there, we were able to dial down our

0:23:46.539,0:23:49.289
Heroku dynos. So we went to, we switched to

0:23:49.289,0:23:52.720
the 2x dynos, and we only needed about half

0:23:52.720,0:23:56.970
as many dynos, because we were only serving,[br]off

0:23:56.970,0:23:59.450
the Heroku dyno for Ajax requests.

0:23:59.450,0:24:02.870
Probably the biggest thing that we learned[br]from Fastly

0:24:02.870,0:24:07.169
was the mobile performance gain. So Fastly[br]has all

0:24:07.169,0:24:10.830
these different locations around the world.[br]If I'm in

0:24:10.830,0:24:13.779
California requesting a page from Upworthy[br]dot com, it's

0:24:13.779,0:24:16.330
gonna go to the closest point of presence,[br]the

0:24:16.330,0:24:18.659
CDN location in California instead of going[br]out to

0:24:18.659,0:24:21.309
Heroku in Virginia, pulling from the data[br]center and

0:24:21.309,0:24:24.320
bringing it back. So the, the load times went

0:24:24.320,0:24:27.899
way down, and we're able to fully cache the

0:24:27.899,0:24:30.669
site, and we have, we've had zero downtime[br]since

0:24:30.669,0:24:33.659
implementing Fastly. So it's just been a great[br]performance

0:24:33.659,0:24:37.440
gain for us.

0:24:37.440,0:24:42.950
So, with having a big monorail, there's huge[br]pain

0:24:42.950,0:24:46.580
points that come along with it. We have, so

0:24:46.580,0:24:49.500
we have one app that deals with our www

0:24:49.500,0:24:51.320
service, Upworthy dot com, and then we have[br]a

0:24:51.320,0:24:55.159
backend CMS that our, our curators log into[br]and

0:24:55.159,0:24:57.519
do their work. So we had to figure out

0:24:57.519,0:24:59.799
the concerns with that. So it's, it's really[br]painful

0:24:59.799,0:25:01.570
there.

0:25:01.570,0:25:04.549
When there's traffic spikes on the public[br]site, it

0:25:04.549,0:25:07.850
could basically, it makes our CMS unstable,[br]so the

0:25:07.850,0:25:10.269
curator would log into our site, try to do

0:25:10.269,0:25:12.529
their work, and they couldn't navigate and[br]we'd just

0:25:12.529,0:25:14.409
have to tell them, you know, come back later

0:25:14.409,0:25:17.120
or come and do the work later. And when

0:25:17.120,0:25:18.960
the traffic spike comes down you'll be able[br]to

0:25:18.960,0:25:22.559
use it. And as our team grows, the code

0:25:22.559,0:25:25.330
base was becoming very large. So the classes[br]would

0:25:25.330,0:25:27.779
get huge and, you know, the frontend didn't[br]care

0:25:27.779,0:25:30.480
about some of the stuff the backend did. So

0:25:30.480,0:25:32.899
it just got harder and harder to maintain.

0:25:32.899,0:25:36.450
So, of course, what did we do? We break

0:25:36.450,0:25:39.440
up here. Fun fact. This is actually a film

0:25:39.440,0:25:46.940
in Chicago. In December, 2013, our buddy Josh[br]French

0:25:46.950,0:25:48.909
has another great idea. He says hey, I think

0:25:48.909,0:25:52.360
we should really start splitting up the Rails[br]app.

0:25:52.360,0:25:54.299
And so, if you look how this timeline is,

0:25:54.299,0:25:56.370
it's pretty evenly spaced. We didn't just[br]jump into

0:25:56.370,0:26:00.169
different things and start building them.[br]We like took

0:26:00.169,0:26:03.279
some time on each of these sections and really

0:26:03.279,0:26:07.149
focused on, on that narrow gap.

0:26:07.149,0:26:10.289
So, one of the things, when you're trying[br]to

0:26:10.289,0:26:13.799
decide how to break up your Rails app into

0:26:13.799,0:26:15.970
services, how do you do it? There's plenty[br]of

0:26:15.970,0:26:17.620
different ways you can do it. This is the

0:26:17.620,0:26:19.259
way we did it. This is not the perfect

0:26:19.259,0:26:22.029
prescription for everybody. I think you have[br]to look

0:26:22.029,0:26:23.860
at your app and see, like, where the clear

0:26:23.860,0:26:26.820
dividing lines are. So we basically, we just[br]chose

0:26:26.820,0:26:28.279
two right now.

0:26:28.279,0:26:30.149
So we have our www site and we have

0:26:30.149,0:26:32.990
our backend CMS. So there's a clear dividing[br]line

0:26:32.990,0:26:35.480
between the two. What we ended up doing is

0:26:35.480,0:26:38.570
cloning each repo into its own separate repository,[br]so

0:26:38.570,0:26:41.190
we could maintain the git repo history, and[br]then

0:26:41.190,0:26:44.730
from there we started breaking everything[br]up, right. So

0:26:44.730,0:26:45.940
this is what we need for this side and

0:26:45.940,0:26:47.399
this is what we need for this side and

0:26:47.399,0:26:48.809
let's start chopping.

0:26:48.809,0:26:50.509
Which was, ended up being a lot of deleting

0:26:50.509,0:26:57.509
and removing namespaced objects. So, once[br]we did that,

0:26:57.789,0:27:00.929
we deployed each of these apps to, to separate

0:27:00.929,0:27:03.909
Heroku applications. The nice thing about[br]Heroku is, they

0:27:03.909,0:27:05.809
have this function called Heroku fork, so[br]they'll just

0:27:05.809,0:27:07.970
take your current app, it'll fork it into[br]another

0:27:07.970,0:27:10.379
Heroku app, pull over all your plugins, everything[br]you

0:27:10.379,0:27:13.350
need. So we did that. We forked the app,

0:27:13.350,0:27:16.759
our main, our main app into two separate applications.

0:27:16.759,0:27:18.580
Removed the plugins that we didn't need on[br]each

0:27:18.580,0:27:21.450
side, and then we pushed out our applications[br]to

0:27:21.450,0:27:25.179
those Heroku sites. Everything's working great.[br]And all we

0:27:25.179,0:27:27.679
had to do was switch our Fastly endpoint to

0:27:27.679,0:27:29.720
point at the new Heroku app at origin and

0:27:29.720,0:27:35.970
we're done. So. Zero downtime really helped[br]there.

0:27:35.970,0:27:38.860
And then we continued to de-duplicate the,[br]the two

0:27:38.860,0:27:42.340
apps. So we created this core gem that is

0:27:42.340,0:27:44.519
holding a lot of the code that shares between

0:27:44.519,0:27:48.679
the two apps. So, our CMS runs on about

0:27:48.679,0:27:52.450
two, two 2x Heroku dynos, and now our frontend

0:27:52.450,0:27:56.830
app runs about twenty-five to thirty 2x dynos.

0:27:56.830,0:27:59.360
This is pretty much what this looks like.[br]So

0:27:59.360,0:28:01.120
we just have an app called www, and then

0:28:01.120,0:28:03.249
we have an app called CMS. And then this

0:28:03.249,0:28:06.070
gem shares the code between it. People will[br]hit

0:28:06.070,0:28:12.909
the Fastly endpoint between www and, and our[br]app.

0:28:12.909,0:28:17.669
So, what are the real benefits of service[br]oriented

0:28:17.669,0:28:20.059
architecture? I think there's, there's plenty[br]if you look

0:28:20.059,0:28:22.929
and think about it. One of the big things

0:28:22.929,0:28:27.320
is, we talked about the instability. You know,[br]if

0:28:27.320,0:28:30.749
our curators couldn't do their work, they,[br]you know,

0:28:30.749,0:28:32.700
we can't have like articles go out, so it

0:28:32.700,0:28:36.559
gets kind of annoying. So if there's a problem

0:28:36.559,0:28:40.169
on one app, it's, it's easier to fix and

0:28:40.169,0:28:42.950
maintain.

0:28:42.950,0:28:44.799
Each of them have different scaling needs.[br]So the

0:28:44.799,0:28:46.909
interesting thing is, our CMS, you know, when[br]we

0:28:46.909,0:28:51.070
have like twenty users that use our CMS, so

0:28:51.070,0:28:53.679
we could you know have is at 2x dynos,

0:28:53.679,0:28:56.950
instead of like having thirty dynos serving[br]all these

0:28:56.950,0:29:00.259
different pages. So the scaling needs was[br]really beneficial

0:29:00.259,0:29:01.159
here.

0:29:01.159,0:29:04.999
And that also divides our teamwork more naturally.[br]So,

0:29:04.999,0:29:06.850
we can have people, we can have engineers[br]on

0:29:06.850,0:29:09.590
a team decide to, you know, work on different

0:29:09.590,0:29:12.210
parts or features, and we don't have to wait

0:29:12.210,0:29:13.490
for something to ship or something else to[br]finish.

0:29:13.490,0:29:16.049
But of course, there's, you know, there's[br]a bunch

0:29:16.049,0:29:18.580
of drawbacks running an SOA. If you're on[br]the

0:29:18.580,0:29:21.110
full stack and you want to make a feature

0:29:21.110,0:29:23.110
on our CMS that needs to carry all the

0:29:23.110,0:29:24.960
way through the funnels of our frontend, you[br]have

0:29:24.960,0:29:27.159
to have both, all three environments running[br]on our

0:29:27.159,0:29:29.759
system to make sure that, you know, your change

0:29:29.759,0:29:33.129
goes and it funnels all the way through.

0:29:33.129,0:29:35.580
Coordinating deploys is also a huge pain.[br]So if

0:29:35.580,0:29:37.330
you need something that's in the core gem[br]plus

0:29:37.330,0:29:40.580
the CMS plus the www app, that means you

0:29:40.580,0:29:43.129
have to deploy each three, coordinate them[br]and make

0:29:43.129,0:29:44.490
sure that all of them are happening at the

0:29:44.490,0:29:46.080
same time, or all of them go in a

0:29:46.080,0:29:49.169
certain order, which, when you're on the monolith,[br]it's

0:29:49.169,0:29:50.840
really easy to just push one big Rails app

0:29:50.840,0:29:53.029
out.

0:29:53.029,0:29:54.549
And then migrating to a fully DRY set of

0:29:54.549,0:29:59.029
codebases, it's really tedious. Trying to[br]figure out what,

0:29:59.029,0:30:00.590
what needs to go where and where we put

0:30:00.590,0:30:04.840
stuff is, it's just been a really hard thing

0:30:04.840,0:30:06.710
to do.

0:30:06.710,0:30:08.549
So, some of the future work that we're gonna

0:30:08.549,0:30:11.779
continue to do on the SOA. We're gonna continue

0:30:11.779,0:30:14.690
to add stuff to our core gem. Remove dupli-

0:30:14.690,0:30:17.129
de-duplication, which is, it's kind of a pain[br]sometimes

0:30:17.129,0:30:19.519
to figure out where, you know, what things[br]go

0:30:19.519,0:30:23.889
where. And then we're considering making,[br]breaking up the

0:30:23.889,0:30:25.830
app even more. So, right now we have two

0:30:25.830,0:30:28.120
apps. We have this third app that actually[br]uses

0:30:28.120,0:30:30.440
up almost all our Heroku dynos. So we're thinking

0:30:30.440,0:30:32.990
about making that its own separate app and[br]service,

0:30:32.990,0:30:35.830
and we can ping that.

0:30:35.830,0:30:37.159
And then, you know, they all communicate with[br]these

0:30:37.159,0:30:39.399
different data stores. Should we just use,[br]a lot

0:30:39.399,0:30:41.590
of times an SOA has that service layer, and

0:30:41.590,0:30:44.190
everything communicates with that service[br]layer. So maybe we

0:30:44.190,0:30:46.029
should move to that.

0:30:46.029,0:30:51.409
L.M.: Cool. So. Just to wrap up here, some

0:30:51.409,0:30:58.409
lessons learned. So, first really, when you're,[br]when you're

0:30:59.110,0:31:02.190
working on an app for a company and you're,

0:31:02.190,0:31:03.909
you're, there's a feature request and there's[br]all these

0:31:03.909,0:31:06.539
other things going on in your company, really[br]do

0:31:06.539,0:31:11.090
wait to make these big technical architectural[br]changes until

0:31:11.090,0:31:13.389
everything starts to really, really hurt.[br]Until you really

0:31:13.389,0:31:16.629
feel the pain. And once you do decide to

0:31:16.629,0:31:20.110
make these changes, it's OK to take a really

0:31:20.110,0:31:24.210
long time. We're probably gonna take eight[br]months to

0:31:24.210,0:31:26.869
do that fully SOA system that we envisioned[br]back

0:31:26.869,0:31:29.080
in the beginning of the year. And that's just

0:31:29.080,0:31:32.369
because we have other feature requests from[br]the rest

0:31:32.369,0:31:36.529
of the company to fulfill. And luckily as,[br]since

0:31:36.529,0:31:39.799
we're on Ruby, it kind of makes it easier

0:31:39.799,0:31:41.480
to do that. It really made it easier when

0:31:41.480,0:31:46.529
we were moving from Padrino to Rails.

0:31:46.529,0:31:48.450
Serve everything you possibly can from a CDN.[br]I

0:31:48.450,0:31:51.820
don't think people use CDN's enough. They're[br]just hugely

0:31:51.820,0:31:56.179
beneficial to the performance of your system.[br]They're great

0:31:56.179,0:32:00.149
for end users, particularly mobile users.[br]At Upworthy, we

0:32:00.149,0:32:02.809
have well over fifty percent of our traffic[br]comes

0:32:02.809,0:32:07.230
from phones and tablets. And CDNs really,[br]really help

0:32:07.230,0:32:10.289
there.

0:32:10.289,0:32:13.100
So remember that you're just dealing with[br]HTML, CSS,

0:32:13.100,0:32:15.289
JavaScript, JSON, maybe, at the end of the[br]day.

0:32:15.289,0:32:17.690
Right. That's all you're serving. So think[br]about how

0:32:17.690,0:32:21.159
you can most efficiently serve those resources[br]from your

0:32:21.159,0:32:24.559
app. And if you are, you're doing your own

0:32:24.559,0:32:26.429
start up, if you're working on your own project,

0:32:26.429,0:32:29.090
I hope you can also experience this kind of

0:32:29.090,0:32:31.970
high traffic growth, because it's been a hugely[br]fun

0:32:31.970,0:32:34.899
and rewarding experience for the both of us.

0:32:34.899,0:32:41.149
So, with that, I'm Luigi, he's Ryan. Our slides

0:32:41.149,0:32:44.429
and our write up is already on our GitHub

0:32:44.429,0:32:47.210
engineering blog. And we will be happy to[br]answer

0:32:47.210,0:32:47.679
questions.