1 00:00:16,849 --> 00:00:20,700 LUIGI MONTANEZ: So, thank you everyone for coming. 2 00:00:20,779 --> 00:00:24,509 It's late on Friday and I really appreciate 3 00:00:24,509 --> 00:00:30,750 all of you sticking around. Obviously Tenderlove is next. So- 4 00:00:30,750 --> 00:00:32,439 AUDIENCE: Talk louder! 5 00:00:32,439 --> 00:00:35,879 L.M.: Talk louder? All right. That's be- is that 6 00:00:35,879 --> 00:00:40,480 better? All right. So, before I get started, I 7 00:00:40,480 --> 00:00:42,539 just want to actually do a quick plug, cause 8 00:00:42,539 --> 00:00:45,190 I saw that the session behind me is actually 9 00:00:45,190 --> 00:00:48,129 on the impostor syndrome. And since you folks are 10 00:00:48,129 --> 00:00:50,010 not gonna be there to, to see that, I 11 00:00:50,010 --> 00:00:51,719 just wanted to actually tell you a little bit 12 00:00:51,719 --> 00:00:53,670 about it, cause actually it's a, it's a really 13 00:00:53,670 --> 00:00:56,929 important concept. So, essentially, the impostor syndrome is when 14 00:00:56,929 --> 00:01:01,649 successful people like software developers feel like they might 15 00:01:01,649 --> 00:01:04,890 not deserve all the success they've gotten. And it's 16 00:01:04,890 --> 00:01:07,729 actually very common. When I learned about it a 17 00:01:07,729 --> 00:01:09,280 few years ago, it was actually really helpful for 18 00:01:09,280 --> 00:01:11,159 me and my career. So when you go to 19 00:01:11,159 --> 00:01:14,159 conferences like these and you see members of the 20 00:01:14,159 --> 00:01:16,670 Rails core team or the Ember core team or 21 00:01:16,670 --> 00:01:19,950 all the speakers and you think, wow, I am 22 00:01:19,950 --> 00:01:22,090 not worthy, actually you are. You, you really do 23 00:01:22,090 --> 00:01:25,939 belong here, and, and people who are successful usually 24 00:01:25,939 --> 00:01:28,939 do deserve it. So Google the impostor syndrome when 25 00:01:28,939 --> 00:01:31,740 you get home or, or watch the video when 26 00:01:31,740 --> 00:01:32,770 it gets posted. 27 00:01:32,770 --> 00:01:37,460 So to our talk. So. You will never believe 28 00:01:37,460 --> 00:01:41,299 which web framework powers Upworthy. I, I can't believe 29 00:01:41,299 --> 00:01:45,590 the conference organizers have made everyone wait all week 30 00:01:45,590 --> 00:01:49,939 to figure this out, to find this out. So, 31 00:01:49,939 --> 00:01:52,100 at Upworthy we aim to please. So we're just 32 00:01:52,100 --> 00:01:55,740 gonna spill the beans right away. Obviously the answer 33 00:01:55,740 --> 00:01:57,499 is Java Struts. 34 00:01:57,499 --> 00:01:59,490 AUDIENCE: [applause] 35 00:01:59,490 --> 00:02:04,789 L.M.: Awesome. Now. So. To introduce myself, my name's 36 00:02:04,789 --> 00:02:06,840 Luigi. I'm the founding engineer at Upworthy. I was 37 00:02:06,840 --> 00:02:11,510 the first engineer hired. I've always been a software 38 00:02:11,510 --> 00:02:14,930 developer involved in politics and advocacy. I got really 39 00:02:14,930 --> 00:02:18,390 into this guy who screened Howard Dean. And so 40 00:02:18,390 --> 00:02:21,020 I worked for his political campaign. There was this 41 00:02:21,020 --> 00:02:23,970 political action committee. I worked for other campaigns and 42 00:02:23,970 --> 00:02:26,360 non-profits, and then before coming to Upworthy I worked 43 00:02:26,360 --> 00:02:30,000 for the Sunlight Foundation, which is a great non-profit 44 00:02:30,000 --> 00:02:33,550 in D.C. that focuses on transparency and government, and 45 00:02:33,550 --> 00:02:35,230 open, open government data. 46 00:02:35,230 --> 00:02:38,070 RYAN RESELLA: I'm Ryan Resella. I'm the senior engineer 47 00:02:38,070 --> 00:02:40,440 at Upworthy. Before this in 2011 I was a 48 00:02:40,440 --> 00:02:42,600 Code for America fellow, the first class of fellows. 49 00:02:42,600 --> 00:02:45,460 I came on as technical lead full time staff 50 00:02:45,460 --> 00:02:48,220 there. And then last year I was on the 51 00:02:48,220 --> 00:02:53,140 Obama for America tech team, or, I guess 2012, 52 00:02:53,140 --> 00:02:55,650 working as a software engineer. And I ran out 53 00:02:55,650 --> 00:02:57,670 of for America organizations to work for so I 54 00:02:57,670 --> 00:02:59,200 joined Upworthy. 55 00:02:59,200 --> 00:03:04,230 L.M.: So, at Upworthy, our mission, and we, this 56 00:03:04,230 --> 00:03:06,430 is something we truly really believe at the company 57 00:03:06,430 --> 00:03:09,060 is to drive massive amounts of attention to the 58 00:03:09,060 --> 00:03:12,530 topics that matter most. And that will kind of 59 00:03:12,530 --> 00:03:15,500 inform the engineering decisions we made as the, as 60 00:03:15,500 --> 00:03:18,250 the tech team. So, just to kind of give 61 00:03:18,250 --> 00:03:21,980 people a little peak at what Upworthy does, for 62 00:03:21,980 --> 00:03:25,050 those who aren't too familiar, so this might be 63 00:03:25,050 --> 00:03:26,300 a bit hard to read, but when we say 64 00:03:26,300 --> 00:03:29,220 that topics that matter most, these are kind of 65 00:03:29,220 --> 00:03:31,310 the topics we've covered in the last year, the 66 00:03:31,310 --> 00:03:33,430 topics that have gotten the most attention. So I'll 67 00:03:33,430 --> 00:03:35,450 just read some of them aloud. 68 00:03:35,450 --> 00:03:39,180 There's a lot of women's issues, like body image, 69 00:03:39,180 --> 00:03:43,640 gender inequality, standards of beauty, a lot of economic 70 00:03:43,640 --> 00:03:48,560 issues like income inequality, health policy. We also cover 71 00:03:48,560 --> 00:03:51,480 a lot of stuff about disability, mental health. Also 72 00:03:51,480 --> 00:03:57,260 bullying, bigotry, racial profiling and, and race issues. And 73 00:03:57,260 --> 00:03:59,440 when we say that we want to drive massive 74 00:03:59,440 --> 00:04:03,260 amounts of attention to these things, what we really 75 00:04:03,260 --> 00:04:07,360 mean as web developers, as web engineers is we 76 00:04:07,360 --> 00:04:10,970 want to drive massive amounts of traffic. 77 00:04:10,970 --> 00:04:13,650 So here's a look at our growth for the 78 00:04:13,650 --> 00:04:15,540 last two years. So we launched a little over 79 00:04:15,540 --> 00:04:22,540 two years ago. We started off at, around one 80 00:04:24,460 --> 00:04:30,820 point seven million uniques per month. That was in 81 00:04:30,820 --> 00:04:36,660 our first ever month in April 2012. And then 82 00:04:36,660 --> 00:04:41,720 we went up to, in November of 2013, 190 83 00:04:41,720 --> 00:04:46,500 million page views. So, this has made us probably 84 00:04:46,500 --> 00:04:48,930 around the top forty, a top forty site in 85 00:04:48,930 --> 00:04:53,280 the U.S. We're maybe one of the larger Rails, 86 00:04:53,280 --> 00:04:56,570 more trafficked Rails apps out there. 87 00:04:56,570 --> 00:04:57,640 To give you a sense of what kind of 88 00:04:57,640 --> 00:05:00,030 traffic we actually deal with, here's a twenty-four hour 89 00:05:00,030 --> 00:05:03,180 graph of what our page view data looks like. 90 00:05:03,180 --> 00:05:04,810 So starting at midnight all the way on the 91 00:05:04,810 --> 00:05:08,370 left, and then ending midnight the next day, you 92 00:05:08,370 --> 00:05:10,280 can kind of say how during the, when the 93 00:05:10,280 --> 00:05:13,230 daytime happens, when, when work hours start, we get 94 00:05:13,230 --> 00:05:18,090 these spikes, these peaks of, of, of traffic. This 95 00:05:18,090 --> 00:05:24,370 is essentially viral, the viral cycle being visualized. 96 00:05:24,370 --> 00:05:31,320 So, we have handled, at most, about 130,000 concurrent 97 00:05:31,320 --> 00:05:34,900 visitors. This is a screenshot from Google analytics during 98 00:05:34,900 --> 00:05:37,750 one of the traffic spikes. So, you know, we, 99 00:05:37,750 --> 00:05:40,240 we are handling large amounts of traffic in very 100 00:05:40,240 --> 00:05:43,560 spiky ways. 101 00:05:43,560 --> 00:05:46,470 So here is a example post from Upworthy. This 102 00:05:46,470 --> 00:05:50,430 was really popular about, a few months ago, in 103 00:05:50,430 --> 00:05:53,310 the fall really. Who here remembers this post? Just 104 00:05:53,310 --> 00:05:56,210 curious. Cool. A few of you. 105 00:05:56,210 --> 00:05:59,530 So this is what Upworthy content looks like. So 106 00:05:59,530 --> 00:06:01,440 it's really just static content. So See Why We 107 00:06:01,440 --> 00:06:03,990 Have An Absolutely Ridiculously Standard of Beauty in Just 108 00:06:03,990 --> 00:06:07,280 37 Seconds. It's a video about how a, a 109 00:06:07,280 --> 00:06:12,669 woman, a model gets photoshopped and looks essentially like 110 00:06:12,669 --> 00:06:15,310 something, like the standard of beauty that doesn't really 111 00:06:15,310 --> 00:06:17,590 exist. So that, that's kind of the content, or 112 00:06:17,590 --> 00:06:20,230 angle we were going for. And here you see, 113 00:06:20,230 --> 00:06:24,139 we have the content, which is basically static content 114 00:06:24,139 --> 00:06:26,110 on the left side of the screen. We have 115 00:06:26,110 --> 00:06:28,669 this sidebar with recommended content on the right side 116 00:06:28,669 --> 00:06:31,450 of the screen. And then scrolling down, we have 117 00:06:31,450 --> 00:06:34,919 what we call asks. We, we also do kind 118 00:06:34,919 --> 00:06:39,110 of some testing on different kinds of content, different 119 00:06:39,110 --> 00:06:41,860 kinds of headlines. You see that down there with 120 00:06:41,860 --> 00:06:44,919 that John Stewart video. And then we have asks 121 00:06:44,919 --> 00:06:47,960 around, do you want to subscribe to our email 122 00:06:47,960 --> 00:06:50,020 list? Do you want to like us on Facebook? 123 00:06:50,020 --> 00:06:53,430 We also have kind of pop-ups that happen after 124 00:06:53,430 --> 00:06:57,480 you watch a video, after you share, also asking 125 00:06:57,480 --> 00:06:59,389 you to do stuff. 126 00:06:59,389 --> 00:07:01,940 So those are the things we, we, the technical 127 00:07:01,940 --> 00:07:05,310 things we, the technical concerns we have at Upworthy. 128 00:07:05,310 --> 00:07:08,169 We're pretty much a static site. We're a public 129 00:07:08,169 --> 00:07:11,190 site. And then we have a CMS backing that, 130 00:07:11,190 --> 00:07:14,300 and then we have this system of dynamic pop-ups 131 00:07:14,300 --> 00:07:17,639 and asks around the site that kind of optimize 132 00:07:17,639 --> 00:07:20,480 our subscriber base, get us more subscribers, get, get 133 00:07:20,480 --> 00:07:23,690 folks to share content. 134 00:07:23,690 --> 00:07:28,160 So, the topic of this talk will really be 135 00:07:28,160 --> 00:07:31,840 about managing the growth of our start up's web 136 00:07:31,840 --> 00:07:37,260 app in the face of very high traffic. And 137 00:07:37,260 --> 00:07:41,120 I actually remember maybe five years ago sitting at 138 00:07:41,120 --> 00:07:44,230 a RailsConf, maybe it was in Baltimore, and I 139 00:07:44,230 --> 00:07:45,980 was sitting where you're sitting and it was a 140 00:07:45,980 --> 00:07:47,820 talk, it was a talk by the guys from 141 00:07:47,820 --> 00:07:51,620 YellowPages dot com. And YellowPages dot com was and 142 00:07:51,620 --> 00:07:54,669 still is one of the larger Rails sites in 143 00:07:54,669 --> 00:07:58,150 the world. Obviously YellowPages is such a strong brand. 144 00:07:58,150 --> 00:07:59,940 Everyone goes to the YellowPages, a lot of people 145 00:07:59,940 --> 00:08:02,990 still use YellowPages dot com to find out local 146 00:08:02,990 --> 00:08:04,810 businesses and stuff like that. And they were talking 147 00:08:04,810 --> 00:08:07,870 about how they scaled their app, their Rails app. 148 00:08:07,870 --> 00:08:09,790 And I was thinking, I was sitting there in 149 00:08:09,790 --> 00:08:11,960 the audience, thinking well, this is really interesting, but 150 00:08:11,960 --> 00:08:14,570 I'm not sure this is ever gonna apply to 151 00:08:14,570 --> 00:08:16,360 me. I don't really, I work on these small 152 00:08:16,360 --> 00:08:19,340 things. No one ever really sees them. But fast 153 00:08:19,340 --> 00:08:21,250 forward a few years and, you know, here I 154 00:08:21,250 --> 00:08:24,300 am. I'm, I'm working on this, this app that 155 00:08:24,300 --> 00:08:27,090 millions of people see every day. So it can 156 00:08:27,090 --> 00:08:29,669 really happen to you too. 157 00:08:29,669 --> 00:08:36,000 So, let's start from the beginning. We launched in 158 00:08:36,000 --> 00:08:41,120 early 2012, March 26th, 2012 to be exact. And 159 00:08:41,120 --> 00:08:44,300 at the time there was one engineer, me, and 160 00:08:44,300 --> 00:08:47,569 our CTO who is a programmer, but was not 161 00:08:47,569 --> 00:08:52,290 really a Ruby or Rails developer. And we actually 162 00:08:52,290 --> 00:08:54,589 launched not on Rails but on Padrino. Who here 163 00:08:54,589 --> 00:08:59,269 is familiar with Padrino? Cool. Who here has production 164 00:08:59,269 --> 00:09:04,809 apps in Padrino? No one. OK. That's what I 165 00:09:04,809 --> 00:09:04,899 thought. 166 00:09:04,899 --> 00:09:06,819 So Padrino, this is, it's, it kind of builds 167 00:09:06,819 --> 00:09:10,579 itself as the elegant web framework. It is essentially 168 00:09:10,579 --> 00:09:14,839 just Sinatra with more Rails-like things on top of 169 00:09:14,839 --> 00:09:17,540 it. So he, who here has used Sinatra? More 170 00:09:17,540 --> 00:09:20,899 hands. Of course. And who here actually has Sinatra 171 00:09:20,899 --> 00:09:24,269 in production? A few, a few hands. Yeah. 172 00:09:24,269 --> 00:09:27,240 So, essentially, when you're working with Sinatra, it's a, 173 00:09:27,240 --> 00:09:32,269 it's a more low-level library. More closer to Rack. 174 00:09:32,269 --> 00:09:36,300 And Padrino, essentially, adds the things that you miss 175 00:09:36,300 --> 00:09:43,300 from Rails into Sinatra. It also freely borrowed some 176 00:09:43,819 --> 00:09:48,639 really great ideas from Djando. The first one being 177 00:09:48,639 --> 00:09:52,670 first-class mountable apps. So in Rails we have engines. 178 00:09:52,670 --> 00:09:55,110 But it seems like not, people don't really use 179 00:09:55,110 --> 00:09:57,279 engines that often. Like, you might use it for 180 00:09:57,279 --> 00:10:01,730 Rails admin. It's, you kind of might have a 181 00:10:01,730 --> 00:10:04,779 larger Rails app and then break it up by 182 00:10:04,779 --> 00:10:10,610 putting it into separate engines. But with Padrino, all 183 00:10:10,610 --> 00:10:13,399 code actually lives in a mountable app. So you 184 00:10:13,399 --> 00:10:15,670 have to use this mountable app system to use 185 00:10:15,670 --> 00:10:18,619 it. 186 00:10:18,619 --> 00:10:21,910 It's also middleware centric. So, those of you who 187 00:10:21,910 --> 00:10:24,050 are familiar with Rack, know that there's a lot 188 00:10:24,050 --> 00:10:26,089 of kind of, there's this concept of middleware - 189 00:10:26,089 --> 00:10:29,360 it's also in Rails - where you can kind 190 00:10:29,360 --> 00:10:32,749 of write these small bits of code that are 191 00:10:32,749 --> 00:10:36,230 Rack compatible, that sit on the stack of what 192 00:10:36,230 --> 00:10:40,059 you do, of how requests and responses come into 193 00:10:40,059 --> 00:10:43,420 your Rails app, or your Sinatra app or any 194 00:10:43,420 --> 00:10:44,910 Rack app. 195 00:10:44,910 --> 00:10:49,439 And there's also this built-in Admin area. And that 196 00:10:49,439 --> 00:10:52,639 Admin area is actually an app. That's just another 197 00:10:52,639 --> 00:10:56,160 mountable app. So this is, this is something that 198 00:10:56,160 --> 00:10:59,119 Django has. I know we have Rails Admin here 199 00:10:59,119 --> 00:11:01,600 in, in the Rails world. But with Padrino, this 200 00:11:01,600 --> 00:11:03,100 is a thing that is built into the framework 201 00:11:03,100 --> 00:11:04,790 itself. 202 00:11:04,790 --> 00:11:09,040 So, why Padrino? Why did I use Padrino in 203 00:11:09,040 --> 00:11:14,209 the beginning? Essentially, at the time, I was a 204 00:11:14,209 --> 00:11:16,519 seasoned Rails developer. I was probably developing for about 205 00:11:16,519 --> 00:11:20,689 five years on Rails. And during that time I 206 00:11:20,689 --> 00:11:23,970 started to form my own opinions about Rails. And 207 00:11:23,970 --> 00:11:27,749 some of those opinions were not compatible with what 208 00:11:27,749 --> 00:11:31,910 the, the, the Rails way prescribed. And I saw 209 00:11:31,910 --> 00:11:34,720 Sinatra and Padrino as this way that I could 210 00:11:34,720 --> 00:11:38,949 still write Ruby, still, I still loved writing Ruby, 211 00:11:38,949 --> 00:11:43,689 but I could also make my own choices. And 212 00:11:43,689 --> 00:11:50,679 there's this kind of epiphany that seasoned web developers 213 00:11:50,679 --> 00:11:54,079 kind of ultimately have, which is, I'm writ- I'm 214 00:11:54,079 --> 00:11:56,459 using this web framework, whether it be Rails or 215 00:11:56,459 --> 00:12:00,300 Django or Sinatra or Node, that all it's doing 216 00:12:00,300 --> 00:12:02,970 at the end, is it's just, it's really just 217 00:12:02,970 --> 00:12:04,889 a web server. Cause at the very end, you're 218 00:12:04,889 --> 00:12:08,559 just getting in responses, you're just emitting HTML or 219 00:12:08,559 --> 00:12:12,569 CSS or JSON, if it's an API, or JavaScript. 220 00:12:12,569 --> 00:12:14,480 That's all you're really doing. And all this talk 221 00:12:14,480 --> 00:12:21,480 about TDD and good object-ori- domain, object-driven design, they're, 222 00:12:23,230 --> 00:12:26,589 they're very important. They help us manage complexity. But 223 00:12:26,589 --> 00:12:30,149 in the end, what does the framework physically do 224 00:12:30,149 --> 00:12:33,339 in the air quote physical way is it, it 225 00:12:33,339 --> 00:12:37,889 really just takes in HTTP responses, or, excuse me, 226 00:12:37,889 --> 00:12:41,739 takes in HTTP requests and then responds with HTTP 227 00:12:41,739 --> 00:12:42,300 responses. 228 00:12:42,300 --> 00:12:45,769 So, while Rails will give you the full foundation 229 00:12:45,769 --> 00:12:51,189 of how to build, build a skyscraper, Padrino gives 230 00:12:51,189 --> 00:12:52,879 you a foundation, but it also lets you choose 231 00:12:52,879 --> 00:12:56,170 some of the plumbing and make some choices for 232 00:12:56,170 --> 00:12:59,739 yourself. So the good parts about Padrino are, is, 233 00:12:59,739 --> 00:13:01,679 it really is just Ruby. It's just Rack. And 234 00:13:01,679 --> 00:13:05,230 if you're a fan of thinking in that mindset, 235 00:13:05,230 --> 00:13:09,509 you'll, you'll really enjoy it. There is less magic. 236 00:13:09,509 --> 00:13:15,869 Things are more explicit. It's unopinionated. All the generators, 237 00:13:15,869 --> 00:13:18,029 when you, when you generate a Padrino app, you 238 00:13:18,029 --> 00:13:22,850 specifically say what you want. That you want ActiveRecord 239 00:13:22,850 --> 00:13:28,459 versus DataMapper versus Monoid or you want Sass versus 240 00:13:28,459 --> 00:13:33,360 Lest versus ERB. Whatever, whatever you want, you can 241 00:13:33,360 --> 00:13:34,860 specify it. 242 00:13:34,860 --> 00:13:37,660 I actually enjoy the process of writing middleware. I 243 00:13:37,660 --> 00:13:39,639 like thinking about web apps like that. I think 244 00:13:39,639 --> 00:13:41,759 it's a much more performant way to think about 245 00:13:41,759 --> 00:13:46,449 writing web apps. And Padrino itself, unlike Rails is, 246 00:13:46,449 --> 00:13:49,470 it's light, it's performant. It's really just Sinatra with 247 00:13:49,470 --> 00:13:53,399 a few more libraries on top of it. 248 00:13:53,399 --> 00:13:55,989 So this is what our architecture looked like when 249 00:13:55,989 --> 00:13:58,129 we launched. So the whole big box is a 250 00:13:58,129 --> 00:14:05,049 Padrino, is the Padrino app. And we had two 251 00:14:05,049 --> 00:14:07,499 mounted apps inside it: main, the public site; so 252 00:14:07,499 --> 00:14:09,879 when you visited Upworthy dot com, this is what 253 00:14:09,879 --> 00:14:13,699 the public would see, those, those content pages; and 254 00:14:13,699 --> 00:14:17,350 then we had the admin tool, the built-in Admin 255 00:14:17,350 --> 00:14:23,360 app in Padrino, which essentially functioned as our CMS. 256 00:14:23,360 --> 00:14:26,290 And keep in mind that we, we were hoping 257 00:14:26,290 --> 00:14:28,089 that we're gonna launch this thing and then we're 258 00:14:28,089 --> 00:14:32,100 gonna get lots of traffic. So we needed to 259 00:14:32,100 --> 00:14:34,429 figure out how to scale it right away, right 260 00:14:34,429 --> 00:14:35,459 from the get go. 261 00:14:35,459 --> 00:14:41,899 So, I kind of devised this, this idea called 262 00:14:41,899 --> 00:14:46,850 explicit caching, and I remembered back in the early 263 00:14:46,850 --> 00:14:50,279 2000s, there was this blogging framework called, or blogging 264 00:14:50,279 --> 00:14:53,110 tool called Moveable Type. And Moveable Type was what 265 00:14:53,110 --> 00:14:57,220 all the big, those early blogs used. And Moveable 266 00:14:57,220 --> 00:14:59,509 Type essentially, the way it worked is, it was 267 00:14:59,509 --> 00:15:01,790 a CMS. So when you saved your post to 268 00:15:01,790 --> 00:15:07,009 the database, Moveable Type would actually save, obviously, see 269 00:15:07,009 --> 00:15:09,519 that you saved something to the database. And then 270 00:15:09,519 --> 00:15:12,699 it render the HTML right then, and then write 271 00:15:12,699 --> 00:15:17,939 the HTML to disc. So when your, when people 272 00:15:17,939 --> 00:15:20,279 visited your blog that was hosted on Moveable Type, 273 00:15:20,279 --> 00:15:22,679 they weren't hitting that Perl app and going through 274 00:15:22,679 --> 00:15:25,949 the database. They were actually just hitting these rendered 275 00:15:25,949 --> 00:15:30,619 HTML files and CSS, JavaScript, that were just living 276 00:15:30,619 --> 00:15:33,699 on the file system of your server. 277 00:15:33,699 --> 00:15:35,920 So I, I actually was drawn to that idea. 278 00:15:35,920 --> 00:15:39,549 I liked that idea. So I kind of re, 279 00:15:39,549 --> 00:15:42,489 re-made it a little bit here in Padrino. So 280 00:15:42,489 --> 00:15:45,439 in the Admin app, there was this publisher object. 281 00:15:45,439 --> 00:15:48,699 And the publisher object essentially did that, where once 282 00:15:48,699 --> 00:15:51,399 anything was saved to the CMS, any content was 283 00:15:51,399 --> 00:15:53,999 saved to the CMS, the publisher object would see 284 00:15:53,999 --> 00:15:57,670 that. It would actually make a request to the 285 00:15:57,670 --> 00:16:02,559 Main app, which was actually rendering the public site, 286 00:16:02,559 --> 00:16:09,559 and it would write that rendered response to Redis. 287 00:16:09,899 --> 00:16:12,410 And so RedisCache was a middleware layer - I 288 00:16:12,410 --> 00:16:15,069 talked about middleware earlier - that sat very close 289 00:16:15,069 --> 00:16:17,779 to the front of this Padrino app. 290 00:16:17,779 --> 00:16:21,100 So, when we had a million or so pages 291 00:16:21,100 --> 00:16:23,239 in that first month, they were all really being 292 00:16:23,239 --> 00:16:27,160 served from Redis. Essentially the website was just in 293 00:16:27,160 --> 00:16:30,149 memory in Redis. And so that worked, you know, 294 00:16:30,149 --> 00:16:32,959 it scaled well. 295 00:16:32,959 --> 00:16:38,629 So, around this time, June, 2012, we hired a 296 00:16:38,629 --> 00:16:43,639 second Rails engineer, Josh French. So, he kind of 297 00:16:43,639 --> 00:16:47,109 joined the team, and then a few weeks later 298 00:16:47,109 --> 00:16:49,679 he said guys, I think we should switch to 299 00:16:49,679 --> 00:16:52,869 Rails. And he was, he was probably right. Because 300 00:16:52,869 --> 00:16:55,489 there were pain points that were not related, really, 301 00:16:55,489 --> 00:16:58,230 to the technical performance of Padrino, but actually more 302 00:16:58,230 --> 00:17:02,249 about social aspects of it. The first one being 303 00:17:02,249 --> 00:17:05,069 that the ecosystem for libraries, while, while pretty good, 304 00:17:05,069 --> 00:17:09,179 because, again, Padrino is just Sinatra, just Rack, was 305 00:17:09,179 --> 00:17:11,609 not as strong as Rails. There's just libraries for 306 00:17:11,609 --> 00:17:14,359 everything you want to do in Rails. We, there 307 00:17:14,359 --> 00:17:17,679 were many things we could do in Padrino, but 308 00:17:17,679 --> 00:17:22,260 the, the kind of quality of those libraries was 309 00:17:22,260 --> 00:17:24,589 not as high. A part of that is because 310 00:17:24,589 --> 00:17:28,220 Padrino isn't as popular, it wasn't very frequently maintained. 311 00:17:28,220 --> 00:17:30,789 The actual Admin app was not very pleasant to 312 00:17:30,789 --> 00:17:33,610 look at. It was its own HTML, CSS styling. 313 00:17:33,610 --> 00:17:36,419 I put a star here because, literally, once, the 314 00:17:36,419 --> 00:17:39,440 sec, the day we moved off of Padrino fully, 315 00:17:39,440 --> 00:17:43,100 they released a new, a new release which was, 316 00:17:43,100 --> 00:17:44,889 which had the Admin system in Bootstrap, which was 317 00:17:44,889 --> 00:17:49,470 what we wanted all along. And there was no 318 00:17:49,470 --> 00:17:52,980 community and, as a start up, it's like actually 319 00:17:52,980 --> 00:17:55,159 easier to hire Rails developers, cause we can't really 320 00:17:55,159 --> 00:17:58,019 go hey, we know you're a great Rails developer 321 00:17:58,019 --> 00:17:59,470 but you're gonna have to work on this thing 322 00:17:59,470 --> 00:18:06,470 called Padrino. That wasn't really a, a strong sell. 323 00:18:06,970 --> 00:18:09,710 So we decided to move. We wanted to move 324 00:18:09,710 --> 00:18:11,880 to Rails. But at the same time, we're, we're 325 00:18:11,880 --> 00:18:15,260 a growing start up. We're getting lots of traffic. 326 00:18:15,260 --> 00:18:17,690 So how do we kind of balance this desire 327 00:18:17,690 --> 00:18:23,620 to move to, move our architecture while still, you 328 00:18:23,620 --> 00:18:27,360 know, maintaining a running app, while still having to 329 00:18:27,360 --> 00:18:29,480 maintain a stable, running app that is serving a 330 00:18:29,480 --> 00:18:33,500 growing traffic base. And Ryan's gonna continue. 331 00:18:33,500 --> 00:18:39,590 R.R.: So, we started our Rails migration in October 332 00:18:39,590 --> 00:18:43,399 of 2012, and so this is a timeline. This 333 00:18:43,399 --> 00:18:47,570 is October 2012. And basically the way it started, 334 00:18:47,570 --> 00:18:51,850 we generated a new Rails app, moved all the, 335 00:18:51,850 --> 00:18:53,960 we mounted it inside the routes.rb, so we just 336 00:18:53,960 --> 00:18:56,750 basically mounted, and you can do the same thing 337 00:18:56,750 --> 00:19:00,320 with Sinatra, since it's Rack. And then we slowly 338 00:19:00,320 --> 00:19:04,289 the migrated the models and utilities into the app. 339 00:19:04,289 --> 00:19:06,429 So, when I joined, we were kind of in 340 00:19:06,429 --> 00:19:09,080 this weird hybrid state. I joined in January of 341 00:19:09,080 --> 00:19:11,620 2013, after taking a nice long siesta, re-electing the 342 00:19:11,620 --> 00:19:17,620 president stuff. And so, we had to figure out, 343 00:19:17,620 --> 00:19:20,250 how could we accelerate and just, you know, get 344 00:19:20,250 --> 00:19:21,919 us over the file hurdle and just move us 345 00:19:21,919 --> 00:19:24,490 onto Rails completely and get out of Padrino. So 346 00:19:24,490 --> 00:19:28,320 the first step that I did was migrating assets, 347 00:19:28,320 --> 00:19:30,419 and so we activated the Rails asset pipeline which 348 00:19:30,419 --> 00:19:32,820 had been turned off in our app. Migrated the 349 00:19:32,820 --> 00:19:35,750 frontend assets, the backend assets. That took us about 350 00:19:35,750 --> 00:19:39,010 a week in February, 2013. The next step was 351 00:19:39,010 --> 00:19:41,760 deciding if we wanted to do the admin area 352 00:19:41,760 --> 00:19:45,360 or our frontend area first. So we decided to 353 00:19:45,360 --> 00:19:47,909 do the hard part first and do the frontend. 354 00:19:47,909 --> 00:19:49,570 So we migrated all the frontend code. The views 355 00:19:49,570 --> 00:19:54,340 and controllers, which took another two weeks. And then 356 00:19:54,340 --> 00:19:59,169 lastly we, we did the entire backend CMS system 357 00:19:59,169 --> 00:20:01,730 as a final push, and we added, we changed 358 00:20:01,730 --> 00:20:03,860 it to bootstrap. Moved all the Rails controllers. That 359 00:20:03,860 --> 00:20:06,490 took us another two weeks. 360 00:20:06,490 --> 00:20:10,100 So, here at this point, the entire migration is 361 00:20:10,100 --> 00:20:12,850 eight months. But it's really, it really ends up 362 00:20:12,850 --> 00:20:16,110 getting accelerated in the last few weeks just because 363 00:20:16,110 --> 00:20:18,889 we wanted to get to that, that final push. 364 00:20:18,889 --> 00:20:21,809 And at this point, we're at three Rails developers: 365 00:20:21,809 --> 00:20:26,269 me, myself, Luigi, and, sorry, myself, Josh, and Luigi, 366 00:20:26,269 --> 00:20:29,909 and our CTO goes back to actually doing CTO 367 00:20:29,909 --> 00:20:33,080 work, which is great. 368 00:20:33,080 --> 00:20:34,549 So now here we are. We're in a monolith. 369 00:20:34,549 --> 00:20:38,139 We're in a big huge monorail for the entire 370 00:20:38,139 --> 00:20:40,159 2013, we were able to do things the Rails 371 00:20:40,159 --> 00:20:43,100 way. We were able to increase our velocity and 372 00:20:43,100 --> 00:20:45,919 just able to add lots of features. We were 373 00:20:45,919 --> 00:20:49,399 able to program the way we wanted to and 374 00:20:49,399 --> 00:20:51,309 really get things moving. We didn't have to rebuild 375 00:20:51,309 --> 00:20:54,639 helpers that weren't, weren't existing. So we could just 376 00:20:54,639 --> 00:20:57,529 have this one huge monolithic Rails app. We had 377 00:20:57,529 --> 00:20:59,460 the backend CMS, and then we had our frontend 378 00:20:59,460 --> 00:21:02,830 and we had all our Ajax endpoints. 379 00:21:02,830 --> 00:21:05,440 But one of the things, when you're looking at 380 00:21:05,440 --> 00:21:09,679 this monorail, is how are you scaling for virility? 381 00:21:09,679 --> 00:21:12,730 So on the campaign, there was a lot of 382 00:21:12,730 --> 00:21:15,549 traffic, and it was pretty easy to know that, 383 00:21:15,549 --> 00:21:17,980 you know, in November is gonna be really busy, 384 00:21:17,980 --> 00:21:20,259 or in October there's a voter deadline so it's 385 00:21:20,259 --> 00:21:22,630 gonna be very specific of when traffic's gonna hit. 386 00:21:22,630 --> 00:21:25,169 You could, you could tell. In, in the virile 387 00:21:25,169 --> 00:21:27,409 media world, you don't know if your post is 388 00:21:27,409 --> 00:21:28,769 gonna be a hit or not. You don't know 389 00:21:28,769 --> 00:21:30,580 when it's gonna get traction. And so you can't 390 00:21:30,580 --> 00:21:33,610 have someone sitting there monitoring twenty-four hours a day, 391 00:21:33,610 --> 00:21:37,370 when to scale, when to not scale. So, we 392 00:21:37,370 --> 00:21:38,620 had to think about how we were gonna do 393 00:21:38,620 --> 00:21:38,730 that. 394 00:21:38,730 --> 00:21:40,549 So a lot of it was just pretty simple 395 00:21:40,549 --> 00:21:44,080 basic stuff here. We added action caching. So we 396 00:21:44,080 --> 00:21:47,070 took, we removed the homegrown publisher system, just turned 397 00:21:47,070 --> 00:21:49,889 on action caching in Rails, and it's backed by 398 00:21:49,889 --> 00:21:51,740 memcache. So people would hit the page, you would 399 00:21:51,740 --> 00:21:54,159 hit the memcached instance of the page instead of 400 00:21:54,159 --> 00:21:56,700 going out, hitting our database, pulling in the page. 401 00:21:56,700 --> 00:21:58,539 We were able to just do that. 402 00:21:58,539 --> 00:22:01,580 The second part was just moving our assets on 403 00:22:01,580 --> 00:22:04,710 S3 and Cloudfront, so our app is hosted on 404 00:22:04,710 --> 00:22:07,190 Heroku. There's actually, it's a really easy tutorial on 405 00:22:07,190 --> 00:22:08,880 Heroku on how to do this. You just set 406 00:22:08,880 --> 00:22:12,710 up a Cloudfront instance and then you just point 407 00:22:12,710 --> 00:22:16,299 your config host for the, the CDM for your 408 00:22:16,299 --> 00:22:18,960 assets to that and it magically works. It's great. 409 00:22:18,960 --> 00:22:22,169 And then the third thing is, we have lots 410 00:22:22,169 --> 00:22:24,090 of Heroku dynos. So at the time we were 411 00:22:24,090 --> 00:22:27,039 running up to forty Heroku dynos, and these were 412 00:22:27,039 --> 00:22:30,250 1x dynos at the time. 413 00:22:30,250 --> 00:22:32,919 So mainly these are for our Ajax requests so 414 00:22:32,919 --> 00:22:35,460 that those asks, those popups, and the different things 415 00:22:35,460 --> 00:22:37,769 that ask you around the site, we were able 416 00:22:37,769 --> 00:22:42,220 to scale with those. So we ran this for 417 00:22:42,220 --> 00:22:45,600 awhile, and we, we had some slowness on the 418 00:22:45,600 --> 00:22:49,009 site sometimes, so we tried to figure out what 419 00:22:49,009 --> 00:22:51,639 could we do to make sure that our site 420 00:22:51,639 --> 00:22:55,509 would be stable and not have to worry about 421 00:22:55,509 --> 00:22:57,870 these viral traffic spikes, having to scale up and 422 00:22:57,870 --> 00:23:00,889 down. So we actually implemented a CDN in front 423 00:23:00,889 --> 00:23:03,049 of it. We took some time and tried to 424 00:23:03,049 --> 00:23:05,440 figure out what CDN we wanted. Because we wanted 425 00:23:05,440 --> 00:23:08,269 to do faster posts and just different things, and 426 00:23:08,269 --> 00:23:10,460 at the time we ended up on Fastly. So 427 00:23:10,460 --> 00:23:14,190 Fastly is a reverse proxy. It runs on Varnish. 428 00:23:14,190 --> 00:23:16,919 It's made, you guys could check it out. It's 429 00:23:16,919 --> 00:23:18,080 great. 430 00:23:18,080 --> 00:23:22,630 We, we moved all of that to, all our 431 00:23:22,630 --> 00:23:25,980 HTML, CSS, and images to Fastly, and then we 432 00:23:25,980 --> 00:23:29,559 turned off the Rails action caching. And the way 433 00:23:29,559 --> 00:23:33,070 Fastly works is, is it reads your headers on 434 00:23:33,070 --> 00:23:36,850 your page, so you just set, set expire four 435 00:23:36,850 --> 00:23:39,649 hours from now. So our site could literally be 436 00:23:39,649 --> 00:23:42,149 down for four hours and Fastly would continue to 437 00:23:42,149 --> 00:23:44,870 serve the pages. 438 00:23:44,870 --> 00:23:46,539 From there, we were able to dial down our 439 00:23:46,539 --> 00:23:49,289 Heroku dynos. So we went to, we switched to 440 00:23:49,289 --> 00:23:52,720 the 2x dynos, and we only needed about half 441 00:23:52,720 --> 00:23:56,970 as many dynos, because we were only serving, off 442 00:23:56,970 --> 00:23:59,450 the Heroku dyno for Ajax requests. 443 00:23:59,450 --> 00:24:02,870 Probably the biggest thing that we learned from Fastly 444 00:24:02,870 --> 00:24:07,169 was the mobile performance gain. So Fastly has all 445 00:24:07,169 --> 00:24:10,830 these different locations around the world. If I'm in 446 00:24:10,830 --> 00:24:13,779 California requesting a page from Upworthy dot com, it's 447 00:24:13,779 --> 00:24:16,330 gonna go to the closest point of presence, the 448 00:24:16,330 --> 00:24:18,659 CDN location in California instead of going out to 449 00:24:18,659 --> 00:24:21,309 Heroku in Virginia, pulling from the data center and 450 00:24:21,309 --> 00:24:24,320 bringing it back. So the, the load times went 451 00:24:24,320 --> 00:24:27,899 way down, and we're able to fully cache the 452 00:24:27,899 --> 00:24:30,669 site, and we have, we've had zero downtime since 453 00:24:30,669 --> 00:24:33,659 implementing Fastly. So it's just been a great performance 454 00:24:33,659 --> 00:24:37,440 gain for us. 455 00:24:37,440 --> 00:24:42,950 So, with having a big monorail, there's huge pain 456 00:24:42,950 --> 00:24:46,580 points that come along with it. We have, so 457 00:24:46,580 --> 00:24:49,500 we have one app that deals with our www 458 00:24:49,500 --> 00:24:51,320 service, Upworthy dot com, and then we have a 459 00:24:51,320 --> 00:24:55,159 backend CMS that our, our curators log into and 460 00:24:55,159 --> 00:24:57,519 do their work. So we had to figure out 461 00:24:57,519 --> 00:24:59,799 the concerns with that. So it's, it's really painful 462 00:24:59,799 --> 00:25:01,570 there. 463 00:25:01,570 --> 00:25:04,549 When there's traffic spikes on the public site, it 464 00:25:04,549 --> 00:25:07,850 could basically, it makes our CMS unstable, so the 465 00:25:07,850 --> 00:25:10,269 curator would log into our site, try to do 466 00:25:10,269 --> 00:25:12,529 their work, and they couldn't navigate and we'd just 467 00:25:12,529 --> 00:25:14,409 have to tell them, you know, come back later 468 00:25:14,409 --> 00:25:17,120 or come and do the work later. And when 469 00:25:17,120 --> 00:25:18,960 the traffic spike comes down you'll be able to 470 00:25:18,960 --> 00:25:22,559 use it. And as our team grows, the code 471 00:25:22,559 --> 00:25:25,330 base was becoming very large. So the classes would 472 00:25:25,330 --> 00:25:27,779 get huge and, you know, the frontend didn't care 473 00:25:27,779 --> 00:25:30,480 about some of the stuff the backend did. So 474 00:25:30,480 --> 00:25:32,899 it just got harder and harder to maintain. 475 00:25:32,899 --> 00:25:36,450 So, of course, what did we do? We break 476 00:25:36,450 --> 00:25:39,440 up here. Fun fact. This is actually a film 477 00:25:39,440 --> 00:25:46,940 in Chicago. In December, 2013, our buddy Josh French 478 00:25:46,950 --> 00:25:48,909 has another great idea. He says hey, I think 479 00:25:48,909 --> 00:25:52,360 we should really start splitting up the Rails app. 480 00:25:52,360 --> 00:25:54,299 And so, if you look how this timeline is, 481 00:25:54,299 --> 00:25:56,370 it's pretty evenly spaced. We didn't just jump into 482 00:25:56,370 --> 00:26:00,169 different things and start building them. We like took 483 00:26:00,169 --> 00:26:03,279 some time on each of these sections and really 484 00:26:03,279 --> 00:26:07,149 focused on, on that narrow gap. 485 00:26:07,149 --> 00:26:10,289 So, one of the things, when you're trying to 486 00:26:10,289 --> 00:26:13,799 decide how to break up your Rails app into 487 00:26:13,799 --> 00:26:15,970 services, how do you do it? There's plenty of 488 00:26:15,970 --> 00:26:17,620 different ways you can do it. This is the 489 00:26:17,620 --> 00:26:19,259 way we did it. This is not the perfect 490 00:26:19,259 --> 00:26:22,029 prescription for everybody. I think you have to look 491 00:26:22,029 --> 00:26:23,860 at your app and see, like, where the clear 492 00:26:23,860 --> 00:26:26,820 dividing lines are. So we basically, we just chose 493 00:26:26,820 --> 00:26:28,279 two right now. 494 00:26:28,279 --> 00:26:30,149 So we have our www site and we have 495 00:26:30,149 --> 00:26:32,990 our backend CMS. So there's a clear dividing line 496 00:26:32,990 --> 00:26:35,480 between the two. What we ended up doing is 497 00:26:35,480 --> 00:26:38,570 cloning each repo into its own separate repository, so 498 00:26:38,570 --> 00:26:41,190 we could maintain the git repo history, and then 499 00:26:41,190 --> 00:26:44,730 from there we started breaking everything up, right. So 500 00:26:44,730 --> 00:26:45,940 this is what we need for this side and 501 00:26:45,940 --> 00:26:47,399 this is what we need for this side and 502 00:26:47,399 --> 00:26:48,809 let's start chopping. 503 00:26:48,809 --> 00:26:50,509 Which was, ended up being a lot of deleting 504 00:26:50,509 --> 00:26:57,509 and removing namespaced objects. So, once we did that, 505 00:26:57,789 --> 00:27:00,929 we deployed each of these apps to, to separate 506 00:27:00,929 --> 00:27:03,909 Heroku applications. The nice thing about Heroku is, they 507 00:27:03,909 --> 00:27:05,809 have this function called Heroku fork, so they'll just 508 00:27:05,809 --> 00:27:07,970 take your current app, it'll fork it into another 509 00:27:07,970 --> 00:27:10,379 Heroku app, pull over all your plugins, everything you 510 00:27:10,379 --> 00:27:13,350 need. So we did that. We forked the app, 511 00:27:13,350 --> 00:27:16,759 our main, our main app into two separate applications. 512 00:27:16,759 --> 00:27:18,580 Removed the plugins that we didn't need on each 513 00:27:18,580 --> 00:27:21,450 side, and then we pushed out our applications to 514 00:27:21,450 --> 00:27:25,179 those Heroku sites. Everything's working great. And all we 515 00:27:25,179 --> 00:27:27,679 had to do was switch our Fastly endpoint to 516 00:27:27,679 --> 00:27:29,720 point at the new Heroku app at origin and 517 00:27:29,720 --> 00:27:35,970 we're done. So. Zero downtime really helped there. 518 00:27:35,970 --> 00:27:38,860 And then we continued to de-duplicate the, the two 519 00:27:38,860 --> 00:27:42,340 apps. So we created this core gem that is 520 00:27:42,340 --> 00:27:44,519 holding a lot of the code that shares between 521 00:27:44,519 --> 00:27:48,679 the two apps. So, our CMS runs on about 522 00:27:48,679 --> 00:27:52,450 two, two 2x Heroku dynos, and now our frontend 523 00:27:52,450 --> 00:27:56,830 app runs about twenty-five to thirty 2x dynos. 524 00:27:56,830 --> 00:27:59,360 This is pretty much what this looks like. So 525 00:27:59,360 --> 00:28:01,120 we just have an app called www, and then 526 00:28:01,120 --> 00:28:03,249 we have an app called CMS. And then this 527 00:28:03,249 --> 00:28:06,070 gem shares the code between it. People will hit 528 00:28:06,070 --> 00:28:12,909 the Fastly endpoint between www and, and our app. 529 00:28:12,909 --> 00:28:17,669 So, what are the real benefits of service oriented 530 00:28:17,669 --> 00:28:20,059 architecture? I think there's, there's plenty if you look 531 00:28:20,059 --> 00:28:22,929 and think about it. One of the big things 532 00:28:22,929 --> 00:28:27,320 is, we talked about the instability. You know, if 533 00:28:27,320 --> 00:28:30,749 our curators couldn't do their work, they, you know, 534 00:28:30,749 --> 00:28:32,700 we can't have like articles go out, so it 535 00:28:32,700 --> 00:28:36,559 gets kind of annoying. So if there's a problem 536 00:28:36,559 --> 00:28:40,169 on one app, it's, it's easier to fix and 537 00:28:40,169 --> 00:28:42,950 maintain. 538 00:28:42,950 --> 00:28:44,799 Each of them have different scaling needs. So the 539 00:28:44,799 --> 00:28:46,909 interesting thing is, our CMS, you know, when we 540 00:28:46,909 --> 00:28:51,070 have like twenty users that use our CMS, so 541 00:28:51,070 --> 00:28:53,679 we could you know have is at 2x dynos, 542 00:28:53,679 --> 00:28:56,950 instead of like having thirty dynos serving all these 543 00:28:56,950 --> 00:29:00,259 different pages. So the scaling needs was really beneficial 544 00:29:00,259 --> 00:29:01,159 here. 545 00:29:01,159 --> 00:29:04,999 And that also divides our teamwork more naturally. So, 546 00:29:04,999 --> 00:29:06,850 we can have people, we can have engineers on 547 00:29:06,850 --> 00:29:09,590 a team decide to, you know, work on different 548 00:29:09,590 --> 00:29:12,210 parts or features, and we don't have to wait 549 00:29:12,210 --> 00:29:13,490 for something to ship or something else to finish. 550 00:29:13,490 --> 00:29:16,049 But of course, there's, you know, there's a bunch 551 00:29:16,049 --> 00:29:18,580 of drawbacks running an SOA. If you're on the 552 00:29:18,580 --> 00:29:21,110 full stack and you want to make a feature 553 00:29:21,110 --> 00:29:23,110 on our CMS that needs to carry all the 554 00:29:23,110 --> 00:29:24,960 way through the funnels of our frontend, you have 555 00:29:24,960 --> 00:29:27,159 to have both, all three environments running on our 556 00:29:27,159 --> 00:29:29,759 system to make sure that, you know, your change 557 00:29:29,759 --> 00:29:33,129 goes and it funnels all the way through. 558 00:29:33,129 --> 00:29:35,580 Coordinating deploys is also a huge pain. So if 559 00:29:35,580 --> 00:29:37,330 you need something that's in the core gem plus 560 00:29:37,330 --> 00:29:40,580 the CMS plus the www app, that means you 561 00:29:40,580 --> 00:29:43,129 have to deploy each three, coordinate them and make 562 00:29:43,129 --> 00:29:44,490 sure that all of them are happening at the 563 00:29:44,490 --> 00:29:46,080 same time, or all of them go in a 564 00:29:46,080 --> 00:29:49,169 certain order, which, when you're on the monolith, it's 565 00:29:49,169 --> 00:29:50,840 really easy to just push one big Rails app 566 00:29:50,840 --> 00:29:53,029 out. 567 00:29:53,029 --> 00:29:54,549 And then migrating to a fully DRY set of 568 00:29:54,549 --> 00:29:59,029 codebases, it's really tedious. Trying to figure out what, 569 00:29:59,029 --> 00:30:00,590 what needs to go where and where we put 570 00:30:00,590 --> 00:30:04,840 stuff is, it's just been a really hard thing 571 00:30:04,840 --> 00:30:06,710 to do. 572 00:30:06,710 --> 00:30:08,549 So, some of the future work that we're gonna 573 00:30:08,549 --> 00:30:11,779 continue to do on the SOA. We're gonna continue 574 00:30:11,779 --> 00:30:14,690 to add stuff to our core gem. Remove dupli- 575 00:30:14,690 --> 00:30:17,129 de-duplication, which is, it's kind of a pain sometimes 576 00:30:17,129 --> 00:30:19,519 to figure out where, you know, what things go 577 00:30:19,519 --> 00:30:23,889 where. And then we're considering making, breaking up the 578 00:30:23,889 --> 00:30:25,830 app even more. So, right now we have two 579 00:30:25,830 --> 00:30:28,120 apps. We have this third app that actually uses 580 00:30:28,120 --> 00:30:30,440 up almost all our Heroku dynos. So we're thinking 581 00:30:30,440 --> 00:30:32,990 about making that its own separate app and service, 582 00:30:32,990 --> 00:30:35,830 and we can ping that. 583 00:30:35,830 --> 00:30:37,159 And then, you know, they all communicate with these 584 00:30:37,159 --> 00:30:39,399 different data stores. Should we just use, a lot 585 00:30:39,399 --> 00:30:41,590 of times an SOA has that service layer, and 586 00:30:41,590 --> 00:30:44,190 everything communicates with that service layer. So maybe we 587 00:30:44,190 --> 00:30:46,029 should move to that. 588 00:30:46,029 --> 00:30:51,409 L.M.: Cool. So. Just to wrap up here, some 589 00:30:51,409 --> 00:30:58,409 lessons learned. So, first really, when you're, when you're 590 00:30:59,110 --> 00:31:02,190 working on an app for a company and you're, 591 00:31:02,190 --> 00:31:03,909 you're, there's a feature request and there's all these 592 00:31:03,909 --> 00:31:06,539 other things going on in your company, really do 593 00:31:06,539 --> 00:31:11,090 wait to make these big technical architectural changes until 594 00:31:11,090 --> 00:31:13,389 everything starts to really, really hurt. Until you really 595 00:31:13,389 --> 00:31:16,629 feel the pain. And once you do decide to 596 00:31:16,629 --> 00:31:20,110 make these changes, it's OK to take a really 597 00:31:20,110 --> 00:31:24,210 long time. We're probably gonna take eight months to 598 00:31:24,210 --> 00:31:26,869 do that fully SOA system that we envisioned back 599 00:31:26,869 --> 00:31:29,080 in the beginning of the year. And that's just 600 00:31:29,080 --> 00:31:32,369 because we have other feature requests from the rest 601 00:31:32,369 --> 00:31:36,529 of the company to fulfill. And luckily as, since 602 00:31:36,529 --> 00:31:39,799 we're on Ruby, it kind of makes it easier 603 00:31:39,799 --> 00:31:41,480 to do that. It really made it easier when 604 00:31:41,480 --> 00:31:46,529 we were moving from Padrino to Rails. 605 00:31:46,529 --> 00:31:48,450 Serve everything you possibly can from a CDN. I 606 00:31:48,450 --> 00:31:51,820 don't think people use CDN's enough. They're just hugely 607 00:31:51,820 --> 00:31:56,179 beneficial to the performance of your system. They're great 608 00:31:56,179 --> 00:32:00,149 for end users, particularly mobile users. At Upworthy, we 609 00:32:00,149 --> 00:32:02,809 have well over fifty percent of our traffic comes 610 00:32:02,809 --> 00:32:07,230 from phones and tablets. And CDNs really, really help 611 00:32:07,230 --> 00:32:10,289 there. 612 00:32:10,289 --> 00:32:13,100 So remember that you're just dealing with HTML, CSS, 613 00:32:13,100 --> 00:32:15,289 JavaScript, JSON, maybe, at the end of the day. 614 00:32:15,289 --> 00:32:17,690 Right. That's all you're serving. So think about how 615 00:32:17,690 --> 00:32:21,159 you can most efficiently serve those resources from your 616 00:32:21,159 --> 00:32:24,559 app. And if you are, you're doing your own 617 00:32:24,559 --> 00:32:26,429 start up, if you're working on your own project, 618 00:32:26,429 --> 00:32:29,090 I hope you can also experience this kind of 619 00:32:29,090 --> 00:32:31,970 high traffic growth, because it's been a hugely fun 620 00:32:31,970 --> 00:32:34,899 and rewarding experience for the both of us. 621 00:32:34,899 --> 00:32:41,149 So, with that, I'm Luigi, he's Ryan. Our slides 622 00:32:41,149 --> 00:32:44,429 and our write up is already on our GitHub 623 00:32:44,429 --> 00:32:47,210 engineering blog. And we will be happy to answer 624 00:32:47,210 --> 00:32:47,679 questions.