1 00:00:17,039 --> 00:00:18,769 ERIC SAXBY: Thanks for coming. 2 00:00:18,769 --> 00:00:21,279 So, I am Eric Saxby. Today I'm gonna be talking 3 00:00:21,279 --> 00:00:24,609 about iterating towards service-oriented 4 00:00:24,609 --> 00:00:28,869 architecture. I may have time for questions 5 00:00:28,869 --> 00:00:30,449 at the end, but I may not. 6 00:00:30,449 --> 00:00:31,380 This might go forty minutes, 7 00:00:31,380 --> 00:00:32,598 so I'm just gonna get started. 8 00:00:32,598 --> 00:00:35,829 So, if you're like me, you can't really pay 9 00:00:35,829 --> 00:00:38,500 attention in talking about programming 10 00:00:38,500 --> 00:00:39,250 unless there are pictures 11 00:00:39,250 --> 00:00:44,559 of cats. So really, we're going from this to 12 00:00:44,559 --> 00:00:48,579 something a little bit more like this. But working 13 00:00:48,579 --> 00:00:50,729 together, as a family. 14 00:00:50,729 --> 00:00:53,229 So why should you care? Why should you listen 15 00:00:53,229 --> 00:00:57,869 to me? So you may not know this, but 16 00:00:57,869 --> 00:01:01,799 I'm kind of a big deal. But, but really, 17 00:01:01,799 --> 00:01:04,519 I've, I've actually not been doing this for that 18 00:01:04,519 --> 00:01:07,490 long. I've been doing Rails for about six years. 19 00:01:07,490 --> 00:01:11,490 Before and after that, I had been using various 20 00:01:11,490 --> 00:01:17,149 different technologies. And I have been fortunate to work 21 00:01:17,149 --> 00:01:19,540 with some very smart people and be able to 22 00:01:19,540 --> 00:01:21,750 learn from them and to break a lot of 23 00:01:21,750 --> 00:01:24,900 things really quickly. And right now I work at 24 00:01:24,900 --> 00:01:28,030 Wanelo. Also, I'm trying to collect all of the 25 00:01:28,030 --> 00:01:30,770 buzzwords on my resume. I have more. I have 26 00:01:30,770 --> 00:01:32,070 more than just this. 27 00:01:32,070 --> 00:01:37,040 But, so, why is Wanelo important? We're like a 28 00:01:37,040 --> 00:01:40,940 social network for shopping, but really it's because we 29 00:01:40,940 --> 00:01:44,670 have many millions of users, active users, and databases 30 00:01:44,670 --> 00:01:46,840 with billions of records. And we've gone through the 31 00:01:46,840 --> 00:01:50,040 pain of getting there and keeping the site actually 32 00:01:50,040 --> 00:01:54,390 running. So, you can save any product from any 33 00:01:54,390 --> 00:01:56,750 store on the internet into your own collections and 34 00:01:56,750 --> 00:01:59,890 have your own wishlist. That's what we do. 35 00:01:59,890 --> 00:02:04,260 More importantly, we've gone from having this as one 36 00:02:04,260 --> 00:02:08,199 main Rails application, doing all of this, to a 37 00:02:08,199 --> 00:02:11,870 central Rails application that's still fairly large, but supported 38 00:02:11,870 --> 00:02:15,200 by a lot of services. We've done as much 39 00:02:15,200 --> 00:02:17,340 as we could open source. Some of the, the 40 00:02:17,340 --> 00:02:20,120 business domain logic, it's really hard to open source. 41 00:02:20,120 --> 00:02:23,080 But, but we try as much as possible. 42 00:02:23,080 --> 00:02:26,290 We've done almost all of this in Ruby, including 43 00:02:26,290 --> 00:02:29,999 some things that people who prefer other languages say 44 00:02:29,999 --> 00:02:32,549 can't be done in Ruby. And we've done this 45 00:02:32,549 --> 00:02:36,439 with a very, very small team very quickly. 46 00:02:36,439 --> 00:02:39,450 If you're like me, though, you're really not so 47 00:02:39,450 --> 00:02:42,469 interested in the success story as you're interested in, 48 00:02:42,469 --> 00:02:45,659 how did you screw up, how did you, how 49 00:02:45,659 --> 00:02:50,099 did you break. So, let me take you on 50 00:02:50,099 --> 00:02:56,260 a journey to another company that a friend of 51 00:02:56,260 --> 00:03:03,260 mine once, recently, called "The Anti-Pattern Goldmine." Completely hypothetical 52 00:03:03,739 --> 00:03:06,400 company. Not naming any names. That I may or 53 00:03:06,400 --> 00:03:07,909 may not have worked for. Some of you in 54 00:03:07,909 --> 00:03:12,359 the audience may or may not have worked for. 55 00:03:12,359 --> 00:03:13,620 After some of this story, you might think you 56 00:03:13,620 --> 00:03:18,169 did. Come in, it's a startup. It's new. Small 57 00:03:18,169 --> 00:03:20,230 team. And come in and say, wow. For a 58 00:03:20,230 --> 00:03:26,730 startup, you have a lot of code that's really, 59 00:03:26,730 --> 00:03:31,099 really tangled. It's all Rails 2 code base. You 60 00:03:31,099 --> 00:03:33,519 know, if you remember vendoring Rails, we, we did 61 00:03:33,519 --> 00:03:36,379 that. If you remember how vendoring Rails can go 62 00:03:36,379 --> 00:03:38,519 wrong? Yeah, yeah. Yeah, that was there. That was 63 00:03:38,519 --> 00:03:40,159 there. 64 00:03:40,159 --> 00:03:41,599 And I think a lot of this might have 65 00:03:41,599 --> 00:03:45,709 come down to the fact that at least early 66 00:03:45,709 --> 00:03:50,569 on, success of a product, success of a feature, 67 00:03:50,569 --> 00:03:53,689 really was launching it as quickly as possible. And 68 00:03:53,689 --> 00:03:56,219 no, no, no, no. Don't worry about that stuff. 69 00:03:56,219 --> 00:03:59,849 Don't worry about design. And we have thirty engineers 70 00:03:59,849 --> 00:04:03,279 doing that as rapidly as possible. Like, five or 71 00:04:03,279 --> 00:04:06,809 six teams all doing that as rapidly as possible, 72 00:04:06,809 --> 00:04:10,059 trying to get it into production. And releases were 73 00:04:10,059 --> 00:04:12,999 a mess, you know. I'm sure a lot of 74 00:04:12,999 --> 00:04:19,029 you can relate to this. Deployments. Multiple hours with 75 00:04:19,029 --> 00:04:21,449 all of this engineer, all these engineers trying to 76 00:04:21,449 --> 00:04:23,449 put all of this code in as quickly as 77 00:04:23,449 --> 00:04:29,410 possible. Invariably, deployments went wrong. 78 00:04:29,410 --> 00:04:34,320 And eventually this got a little bit faster from, 79 00:04:34,320 --> 00:04:39,050 from monthly releases to weekly releases. But then we 80 00:04:39,050 --> 00:04:42,900 would have code freezes, where for three months, everyone's 81 00:04:42,900 --> 00:04:44,960 trying to cram in all the features without deploying, 82 00:04:44,960 --> 00:04:46,510 and you can imagine where that goes. 83 00:04:46,510 --> 00:04:50,140 And, over time, really, things are just getting worse. 84 00:04:50,140 --> 00:04:54,070 We, our rough estimates that we're trying to give 85 00:04:54,070 --> 00:04:57,380 to see how long things are gonna last come 86 00:04:57,380 --> 00:04:59,000 back, and we say, oh, that means we were 87 00:04:59,000 --> 00:05:00,680 deploying on this date. You know, hey, we told 88 00:05:00,680 --> 00:05:02,930 the board, great. That's awesome. You're gonna deploy on 89 00:05:02,930 --> 00:05:06,940 this date. Ah. Whoo. With a deadline like that, 90 00:05:06,940 --> 00:05:09,450 the only really way to meet it is to 91 00:05:09,450 --> 00:05:13,230 make serious shortcuts. And then as soon as a 92 00:05:13,230 --> 00:05:18,410 product finishes, because we're invariably missing those deadlines, and 93 00:05:18,410 --> 00:05:20,580 there's the next project that is supposed to be 94 00:05:20,580 --> 00:05:22,930 out in, in a week, based on the estimates 95 00:05:22,930 --> 00:05:26,620 that you were supposed to do. Team gets dispersed. 96 00:05:26,620 --> 00:05:30,580 Finish these new projects. And no matter how worse 97 00:05:30,580 --> 00:05:32,310 case we are in our estimates, it's just never 98 00:05:32,310 --> 00:05:35,380 worst case enough. It just, it just keeps getting 99 00:05:35,380 --> 00:05:36,510 worse. 100 00:05:36,510 --> 00:05:39,910 So some of you might, might be familiar with 101 00:05:39,910 --> 00:05:44,340 this story. So, during the course of this, I, 102 00:05:44,340 --> 00:05:47,500 I, I, I think I've learned a lot. You 103 00:05:47,500 --> 00:05:49,580 know, programming is one of the most fun careers 104 00:05:49,580 --> 00:05:51,960 that I've, that I've had, and when it's not 105 00:05:51,960 --> 00:05:55,710 fun, you know, you know something's wrong. And keep 106 00:05:55,710 --> 00:05:58,860 reading about service-oriented architecture, and, you know, a lot 107 00:05:58,860 --> 00:06:01,970 of us latch onto this. This is the solution 108 00:06:01,970 --> 00:06:03,690 to all of our problems. And you know at 109 00:06:03,690 --> 00:06:06,420 dev ops, that's pretty cool over there. You know, 110 00:06:06,420 --> 00:06:07,530 dev ops is the answer. That's how we're gonna 111 00:06:07,530 --> 00:06:10,310 do services. It's just gonna, it's a done deal. 112 00:06:10,310 --> 00:06:12,370 All we have to do is do services, dev 113 00:06:12,370 --> 00:06:15,110 opsy, and like, we're done. 114 00:06:15,110 --> 00:06:19,150 So around this time, I move into the operations, 115 00:06:19,150 --> 00:06:22,670 not, not the operations team, cause dev ops. Not 116 00:06:22,670 --> 00:06:25,220 the operations team. Not the team over there that 117 00:06:25,220 --> 00:06:27,160 you just throw the things over the wall and 118 00:06:27,160 --> 00:06:29,170 they just make it work. No, no, no. Not, 119 00:06:29,170 --> 00:06:32,600 not the operations team at all. 120 00:06:32,600 --> 00:06:36,010 And certainly me, a number of other people in 121 00:06:36,010 --> 00:06:40,600 the engineering team, I, I could say, really decided 122 00:06:40,600 --> 00:06:44,490 services are the only way forward. And really quickly, 123 00:06:44,490 --> 00:06:47,510 product, and, and I don't mean an individual project 124 00:06:47,510 --> 00:06:49,460 manager or anything like that. By product, I mean 125 00:06:49,460 --> 00:06:52,120 really, all of the people, all of the teams 126 00:06:52,120 --> 00:06:56,080 going into designing the product, planning the product, really 127 00:06:56,080 --> 00:06:59,560 how it was going forward, they quickly come to 128 00:06:59,560 --> 00:07:01,720 the conclusion that services are really just that, just 129 00:07:01,720 --> 00:07:03,940 that, that thing getting in the way of us 130 00:07:03,940 --> 00:07:08,040 cranking out these features. Cause features is success. 131 00:07:08,040 --> 00:07:10,130 But at first, if at first you can't succeed, 132 00:07:10,130 --> 00:07:13,220 I have, I have, I have learned, you know, 133 00:07:13,220 --> 00:07:15,180 there's, there's a few things. You can take a 134 00:07:15,180 --> 00:07:21,930 firm stance, breathe deeply, and become the loudest person 135 00:07:21,930 --> 00:07:27,330 in the room. Really, really helps. Also, if there's 136 00:07:27,330 --> 00:07:29,740 anything that I've learned from my cats, it's throw 137 00:07:29,740 --> 00:07:34,490 shit, you know. Done. It really, really helps in 138 00:07:34,490 --> 00:07:36,340 situations like this. 139 00:07:36,340 --> 00:07:39,100 So, we, we have a few things coming up. 140 00:07:39,100 --> 00:07:43,850 A few products. These, these aren't necessarily concurrent, these, 141 00:07:43,850 --> 00:07:46,860 these features. These projects. But we have a new 142 00:07:46,860 --> 00:07:51,530 login feature and login, in this application, is as 143 00:07:51,530 --> 00:07:55,400 homegrown as you could possibly imagine it could go 144 00:07:55,400 --> 00:07:59,750 wrong. And it's this core, core login functionality, and 145 00:07:59,750 --> 00:08:01,810 we're like, no, no, no, no, no, no. No, 146 00:08:01,810 --> 00:08:03,430 no, no, no, no, no. That's not going in 147 00:08:03,430 --> 00:08:06,710 this code base. That's gonna be a new application, 148 00:08:06,710 --> 00:08:10,520 we're. That's the only way we're gonna do this. 149 00:08:10,520 --> 00:08:12,930 Otherwise we're just not gonna do this. It's gonna 150 00:08:12,930 --> 00:08:14,980 fail. We know it's gonna fail. So let's just 151 00:08:14,980 --> 00:08:16,930 skip to not doing it. 152 00:08:16,930 --> 00:08:22,230 And, also, we have this enterprise software over here, 153 00:08:22,230 --> 00:08:24,889 and we have this homegrown Rails application, and all 154 00:08:24,889 --> 00:08:28,080 the data really needs to kind of be synced 155 00:08:28,080 --> 00:08:30,830 between it in order for the company to actually 156 00:08:30,830 --> 00:08:35,700 work. So a lot of iterations on this, but 157 00:08:35,700 --> 00:08:37,880 really this time, we're gonna do it right, and 158 00:08:37,880 --> 00:08:40,870 it's gonna be a, a data service. We're gonna 159 00:08:40,870 --> 00:08:43,179 have our Enterprise software in our Rails app and 160 00:08:43,179 --> 00:08:45,589 that is totally just gonna make this an Enterprise-Rails 161 00:08:45,589 --> 00:08:46,190 app. It's gonna be amazing. 162 00:08:46,190 --> 00:08:49,930 And I remember saying this a lot to a 163 00:08:49,930 --> 00:08:52,129 lot of different people. Like, we're gonna screw this 164 00:08:52,129 --> 00:08:55,149 up. We're gonna fail. I know we're gonna fail. 165 00:08:55,149 --> 00:08:56,930 But it's the only option that we have. It's 166 00:08:56,930 --> 00:08:59,230 the only way that we're actually gonna be able 167 00:08:59,230 --> 00:09:01,470 to get to be able to do this and 168 00:09:01,470 --> 00:09:05,360 succeed at some point in the future. 169 00:09:05,360 --> 00:09:10,309 OK, so on, on a side track, hopefully this 170 00:09:10,309 --> 00:09:13,749 is not new to very many of you, but 171 00:09:13,749 --> 00:09:15,610 really there's a lot of different ways to do 172 00:09:15,610 --> 00:09:17,959 service-oriented architecture. It can mean a lot of different 173 00:09:17,959 --> 00:09:21,779 things. Read blog posts. You get a lot of 174 00:09:21,779 --> 00:09:24,550 different ideas. It can be synchronous, asynchronous. Use a 175 00:09:24,550 --> 00:09:28,279 message bus. Maybe some services require actually hitting them 176 00:09:28,279 --> 00:09:31,680 over HTTP, TCP sockets. You know, there's a lot 177 00:09:31,680 --> 00:09:34,589 of ways of different doing, of doing this kind 178 00:09:34,589 --> 00:09:35,670 of thing. 179 00:09:35,670 --> 00:09:37,680 But why would you really want to? So, you 180 00:09:37,680 --> 00:09:39,139 know, scale the organization. You have a lot of 181 00:09:39,139 --> 00:09:41,769 different teams. A lot of different engineers. Maybe you 182 00:09:41,769 --> 00:09:43,740 really want this team over here to just have 183 00:09:43,740 --> 00:09:47,240 their own code base that they deploy separately. Also, 184 00:09:47,240 --> 00:09:48,959 maybe you want to outsource this to a different 185 00:09:48,959 --> 00:09:50,740 team, and you really don't want to give all 186 00:09:50,740 --> 00:09:53,050 of the code to the other team. You really 187 00:09:53,050 --> 00:09:55,699 just want to isolate this and say, you know, 188 00:09:55,699 --> 00:10:00,939 you guys over here, you know, just, here, here's 189 00:10:00,939 --> 00:10:03,889 this. Sorry. I'm, I'm actually trying to explain it 190 00:10:03,889 --> 00:10:09,459 all gendered pronounce from my vocabulary but it's really 191 00:10:09,459 --> 00:10:11,860 hard. Sorry. 192 00:10:11,860 --> 00:10:15,540 So, you can also scale the code base, you 193 00:10:15,540 --> 00:10:20,019 know. Performance of this thing over here might be 194 00:10:20,019 --> 00:10:23,809 completely different than this thing over here, and that 195 00:10:23,809 --> 00:10:25,709 might actually be easier in a service. You can 196 00:10:25,709 --> 00:10:28,439 really tune the code to the work load. You 197 00:10:28,439 --> 00:10:31,369 might have this really complicated code that, for various 198 00:10:31,369 --> 00:10:33,439 reasons, needs to be complicated. It might be really 199 00:10:33,439 --> 00:10:36,029 complicated functionality. But you can, you might be able 200 00:10:36,029 --> 00:10:40,980 to hide that behind a, a clean interface. An 201 00:10:40,980 --> 00:10:41,559 API. 202 00:10:41,559 --> 00:10:46,019 And tests, usually, not always, but a small code 203 00:10:46,019 --> 00:10:49,470 base can mean fast tests. And if you're sitting 204 00:10:49,470 --> 00:10:52,110 around for hours waiting for tests to complete, you 205 00:10:52,110 --> 00:10:54,009 know, that can be, that can really eat into 206 00:10:54,009 --> 00:10:56,459 your productivity. But it, but it comes at a 207 00:10:56,459 --> 00:10:58,499 cost. All of this comes at a cost. 208 00:10:58,499 --> 00:11:00,720 And I think one of the things that, that 209 00:11:00,720 --> 00:11:04,180 I've been learning is that sometimes the cost of 210 00:11:04,180 --> 00:11:06,779 not doing this is greater than the cost of 211 00:11:06,779 --> 00:11:10,149 doing. The cost of the infrastructure, new servers, how 212 00:11:10,149 --> 00:11:12,740 they talk together. It's really complicated. Things will go 213 00:11:12,740 --> 00:11:15,129 wrong. But sometimes not doing this is gonna, is 214 00:11:15,129 --> 00:11:17,269 gonna mean that your productivity is going down and 215 00:11:17,269 --> 00:11:20,439 down and down and, and it actually is more 216 00:11:20,439 --> 00:11:21,149 costly. 217 00:11:21,149 --> 00:11:24,269 OK, so back to these projects. We have this 218 00:11:24,269 --> 00:11:27,920 data service. It's sometimes, I think six engineers, sometimes 219 00:11:27,920 --> 00:11:31,899 eight. I don't, I don't really remember. And nine 220 00:11:31,899 --> 00:11:35,749 months of, of work. It's really complicated, like state 221 00:11:35,749 --> 00:11:38,740 transactions, and this is really critical data that really 222 00:11:38,740 --> 00:11:43,920 needs to be done right. And there are, so, 223 00:11:43,920 --> 00:11:46,369 there, there might have been, when deploying it, some, 224 00:11:46,369 --> 00:11:48,369 some slight data things. But they were fixable. Really 225 00:11:48,369 --> 00:11:50,189 quickly. It was fine. There were actually no major 226 00:11:50,189 --> 00:11:52,449 data problems. 227 00:11:52,449 --> 00:11:53,959 And there were some applications that were built on 228 00:11:53,959 --> 00:11:57,490 top of this data service. And depending on who 229 00:11:57,490 --> 00:12:00,519 you talk to, they, they were more or less 230 00:12:00,519 --> 00:12:02,779 successful. Some people using that, these applications are like, 231 00:12:02,779 --> 00:12:05,740 oh, thank you, this did exactly what, what I 232 00:12:05,740 --> 00:12:07,269 need. It's actually helping me with my workflow. Some 233 00:12:07,269 --> 00:12:11,449 people are like, oh, OK. 234 00:12:11,449 --> 00:12:13,329 But engineering, you know what, this was really critical 235 00:12:13,329 --> 00:12:16,639 data. It was really hard. Totally new for us. 236 00:12:16,639 --> 00:12:19,910 This was a success. We did not break anything. 237 00:12:19,910 --> 00:12:26,410 Product is like nine months? Nine months. Eight engineers, 238 00:12:26,410 --> 00:12:29,709 nine months. How could you possibly call that a 239 00:12:29,709 --> 00:12:30,740 success? 240 00:12:30,740 --> 00:12:37,019 OK. So different application. Login flow. Depending on the 241 00:12:37,019 --> 00:12:39,509 time period, two engineers, four engineers, plus the dev 242 00:12:39,509 --> 00:12:46,410 ops team. Dev ops. Three months, you know. You 243 00:12:46,410 --> 00:12:50,399 know, we were figuring out, like, the system's automation 244 00:12:50,399 --> 00:12:55,209 was really tangled and in, in a few weeks 245 00:12:55,209 --> 00:12:59,790 we had some staging servers. And about two months 246 00:12:59,790 --> 00:13:02,980 later, someone comes up to me and says, hey, 247 00:13:02,980 --> 00:13:05,290 where, where, where are the servers gonna, gonna be? 248 00:13:05,290 --> 00:13:08,179 Can we get those? And I'm like, deep breath. 249 00:13:08,179 --> 00:13:12,269 Deep breath. Which servers? Oh, this, those staging servers. 250 00:13:12,269 --> 00:13:16,649 But it went, we worked it all out. 251 00:13:16,649 --> 00:13:19,029 Something that I learned about dev ops. Everyone actually 252 00:13:19,029 --> 00:13:22,959 needs to be invested and it's new to people. 253 00:13:22,959 --> 00:13:24,999 So, and it was released with a feature flag. 254 00:13:24,999 --> 00:13:27,499 Only a small amount of people were actually running 255 00:13:27,499 --> 00:13:31,059 through it. So it, we had some really good 256 00:13:31,059 --> 00:13:34,290 time to break it in production and really figure 257 00:13:34,290 --> 00:13:37,249 out how, how it was really gonna run. 258 00:13:37,249 --> 00:13:38,939 And we launched it to all of our users, 259 00:13:38,939 --> 00:13:41,929 and it was actually, I would say, very successful. 260 00:13:41,929 --> 00:13:46,529 It worked exactly intended. It was very resilient to 261 00:13:46,529 --> 00:13:49,389 failures. We had to do some unexpected database maintenance. 262 00:13:49,389 --> 00:13:52,230 Restart master databases and we're like, you know what, 263 00:13:52,230 --> 00:13:54,980 just do it. You know, just, just restart the 264 00:13:54,980 --> 00:13:56,629 database. No, no, no, no, no. Don't turn off 265 00:13:56,629 --> 00:13:59,369 the service. It's gonna recover. It's gonna be fine. 266 00:13:59,369 --> 00:14:02,269 Maybe like one user will notice. 267 00:14:02,269 --> 00:14:04,220 And the company's actually sort of figuring out that 268 00:14:04,220 --> 00:14:07,980 user metrics might be more important than dates as 269 00:14:07,980 --> 00:14:12,209 metrics, and the user metrics were generally successful. So 270 00:14:12,209 --> 00:14:15,040 engineering, like, this, this is good. This is good. 271 00:14:15,040 --> 00:14:17,329 This is, this is how we do this. This 272 00:14:17,329 --> 00:14:19,720 is, you know, the next project, this is how 273 00:14:19,720 --> 00:14:22,009 we're gonna, we're gonna have to figure this. 274 00:14:22,009 --> 00:14:25,199 And products is like three months? What are you 275 00:14:25,199 --> 00:14:28,300 talking about? Success? That's not success. We have these 276 00:14:28,300 --> 00:14:30,739 other product features that needed to be done. And 277 00:14:30,739 --> 00:14:32,300 we have these four engineers that are sitting in 278 00:14:32,300 --> 00:14:35,739 the corner, not able to do these things. 279 00:14:35,739 --> 00:14:41,629 OK. So I would say that this is a 280 00:14:41,629 --> 00:14:44,350 really important question to ask. What is, what is 281 00:14:44,350 --> 00:14:47,639 success? If engineering says it's a success, and product 282 00:14:47,639 --> 00:14:52,300 says it's a failure, who really wins? So, this 283 00:14:52,300 --> 00:14:55,639 is actually a trick question, because it's a zero-sum 284 00:14:55,639 --> 00:14:59,449 game. Like, nobody wins in this interaction. 285 00:14:59,449 --> 00:15:00,800 So let's go a little bit deeper to ask 286 00:15:00,800 --> 00:15:04,829 why we needed SOA in the first place. And, 287 00:15:04,829 --> 00:15:07,920 I, this is really complicated. There's lots of moving 288 00:15:07,920 --> 00:15:09,800 parts, but some of the things that, that I 289 00:15:09,800 --> 00:15:14,619 think now were because engineering didn't trust product, and 290 00:15:14,619 --> 00:15:16,670 we actually didn't, what that really means is we 291 00:15:16,670 --> 00:15:21,699 didn't trust that we could do our jobs well 292 00:15:21,699 --> 00:15:24,189 given the shortcuts that we needed to do to 293 00:15:24,189 --> 00:15:28,040 actually do our job and actually meet our metrics. 294 00:15:28,040 --> 00:15:30,439 And product didn't trust engineering to meet the products 295 00:15:30,439 --> 00:15:32,959 we were given, and we, we couldn't actually meet 296 00:15:32,959 --> 00:15:37,720 those promises. And, again, over time, this was changing, 297 00:15:37,720 --> 00:15:40,980 but product was accountable for features, not or quality, 298 00:15:40,980 --> 00:15:44,989 and, and the actual how users interacted with them. 299 00:15:44,989 --> 00:15:47,529 And this is probably the subject of much more 300 00:15:47,529 --> 00:15:49,730 discussion, and would love to continue this and, and 301 00:15:49,730 --> 00:15:51,429 learn more myself, but I think that a lot 302 00:15:51,429 --> 00:15:55,850 of this comes down to trust. If, if, if 303 00:15:55,850 --> 00:15:57,589 you can't trust that you can do your job 304 00:15:57,589 --> 00:15:59,689 successfully, how, how can you actually do your job 305 00:15:59,689 --> 00:16:02,040 successfully? If, if product, if different parts of the 306 00:16:02,040 --> 00:16:04,290 company don't trust each other to do their jobs 307 00:16:04,290 --> 00:16:07,249 well, how can they actually do their jobs well? 308 00:16:07,249 --> 00:16:10,689 So, what did, what did we learn? You know, 309 00:16:10,689 --> 00:16:13,540 fuck it, next time it's SOA from the beginning. 310 00:16:13,540 --> 00:16:15,689 Like, before we even do Rails new it's gonna 311 00:16:15,689 --> 00:16:19,220 be RabbitMQ. We're gonna have a cluster. It's gonna 312 00:16:19,220 --> 00:16:22,670 be amazing. So, no. No, no, no. Not, not, 313 00:16:22,670 --> 00:16:26,170 not the right answer. I almost went here, and 314 00:16:26,170 --> 00:16:29,959 thankfully I worked, then, with some very humble, empathetic 315 00:16:29,959 --> 00:16:32,649 people, who were also very convincing. No. 316 00:16:32,649 --> 00:16:36,649 So, I think what I've, what I have really 317 00:16:36,649 --> 00:16:39,480 taken from this, is Agile's not just one sprint 318 00:16:39,480 --> 00:16:42,110 after the next. It's like four miles is great. 319 00:16:42,110 --> 00:16:44,569 You just break it up into a hundred meter 320 00:16:44,569 --> 00:16:46,850 increments and you just, each one is a sprint. 321 00:16:46,850 --> 00:16:48,329 It's gonna be fine. 322 00:16:48,329 --> 00:16:52,470 What this really is about iterating. Deploying things, small 323 00:16:52,470 --> 00:16:54,209 things, really quickly, using the data that you get 324 00:16:54,209 --> 00:16:57,319 from that to figure out what to do next, 325 00:16:57,319 --> 00:17:00,670 and refactoring. Refactoring is really, really important. You might 326 00:17:00,670 --> 00:17:03,980 be able to do a small thing quickly, and 327 00:17:03,980 --> 00:17:07,990 with some shortcuts, not really needing to know how 328 00:17:07,990 --> 00:17:09,990 it needs to be designed. But when you see 329 00:17:09,990 --> 00:17:13,000 the pattern, you, you have to fix it. And 330 00:17:13,000 --> 00:17:16,670 SOA's not gonna solve larger organizational problems. 331 00:17:16,670 --> 00:17:19,980 It's not gonna fix your code. Really what it 332 00:17:19,980 --> 00:17:24,180 is is another tool at our disposal to solve 333 00:17:24,180 --> 00:17:27,459 our pain points, and our, and our organization's, our 334 00:17:27,459 --> 00:17:31,090 company's pain points. And how we do that is 335 00:17:31,090 --> 00:17:38,040 through iteration. So, small changes deployed, deployed quickly, using 336 00:17:38,040 --> 00:17:40,480 feature flags so that you can actually get code 337 00:17:40,480 --> 00:17:43,320 into production as soon as possible, knowing that it's 338 00:17:43,320 --> 00:17:46,390 off and it's not gonna affect users. And prioritizing 339 00:17:46,390 --> 00:17:48,490 this when it's necessary. 340 00:17:48,490 --> 00:17:51,630 So when might it be necessary? I would say 341 00:17:51,630 --> 00:17:53,370 performance. That's a really big thing. This is actually 342 00:17:53,370 --> 00:17:57,770 driving a lot of what we're doing. Code complexity 343 00:17:57,770 --> 00:17:59,740 might be less in a service than it would 344 00:17:59,740 --> 00:18:02,380 be outside of a service. So if you have 345 00:18:02,380 --> 00:18:03,800 these two things and you're like, this is actually 346 00:18:03,800 --> 00:18:05,490 gonna be easier to do over here than to 347 00:18:05,490 --> 00:18:07,810 put in this tangled mess, maybe, maybe it is 348 00:18:07,810 --> 00:18:08,340 better. 349 00:18:08,340 --> 00:18:10,960 May, and also maybe, sometimes you have a new 350 00:18:10,960 --> 00:18:14,220 feature and it is completely unrelated to everything else 351 00:18:14,220 --> 00:18:18,370 that you've already done. With the caveat that you, 352 00:18:18,370 --> 00:18:19,320 in the short term, you might be able to 353 00:18:19,320 --> 00:18:20,710 put it in with all the rest of that 354 00:18:20,710 --> 00:18:24,850 mess, trusting that when it becomes a problem you 355 00:18:24,850 --> 00:18:27,540 will be empowered to fix that problem. 356 00:18:27,540 --> 00:18:31,750 OK. So, performance. As I said, this is driving 357 00:18:31,750 --> 00:18:35,180 a lot of, of our work. So, Winelow is 358 00:18:35,180 --> 00:18:38,430 getting slower. We're running into some problems. We just 359 00:18:38,430 --> 00:18:41,010 got ring that the databases are becoming I/O bound 360 00:18:41,010 --> 00:18:42,890 on disk. This is a really, really bad place 361 00:18:42,890 --> 00:18:47,360 to be. But our user growth is increasing dramatically. 362 00:18:47,360 --> 00:18:50,850 The, the activity is increasing dramatically. We're starting to 363 00:18:50,850 --> 00:18:54,820 see exponential tendencies in our graphs, and if you 364 00:18:54,820 --> 00:18:59,350 see exponential tendencies in your user graphs, it's kind 365 00:18:59,350 --> 00:19:02,300 of like giving a talk to hundreds of people. 366 00:19:02,300 --> 00:19:06,210 You're like, why are you all looking at me?! 367 00:19:06,210 --> 00:19:09,900 It's, it can be scary. It can be really 368 00:19:09,900 --> 00:19:13,760 scary. But we have, so we have one table 369 00:19:13,760 --> 00:19:16,630 that's really outstripping the others, and that's causing problems. 370 00:19:16,630 --> 00:19:19,240 That's, it's, we discover, because of our data, because 371 00:19:19,240 --> 00:19:22,150 of our graphs, that it's really one table that's 372 00:19:22,150 --> 00:19:24,860 really destroying the performance of the rest of the 373 00:19:24,860 --> 00:19:28,230 table. And we're in the cloud, cause we have 374 00:19:28,230 --> 00:19:31,630 all those buzzwords. So there's really a maximum size 375 00:19:31,630 --> 00:19:34,010 of a host that we can get. So there's 376 00:19:34,010 --> 00:19:36,080 really, like, an upper limit of how much we 377 00:19:36,080 --> 00:19:38,550 can actually solve this with one database. 378 00:19:38,550 --> 00:19:42,630 Even after read-write splitting, we've already done that, and 379 00:19:42,630 --> 00:19:45,450 we, and, and pretty soon, we realized that the 380 00:19:45,450 --> 00:19:48,760 site is gonna break. If we don't do anything, 381 00:19:48,760 --> 00:19:51,250 we're just not gonna have a company anymore. It's 382 00:19:51,250 --> 00:19:55,830 just gonna, it's gonna fall down. But, we have 383 00:19:55,830 --> 00:20:00,080 really, really, really active committed users joining us right 384 00:20:00,080 --> 00:20:03,080 now. Now is not the time to stop feature-development. 385 00:20:03,080 --> 00:20:05,410 Now's the time to really learn from what our 386 00:20:05,410 --> 00:20:07,960 users are doing, double down on it, really tweak 387 00:20:07,960 --> 00:20:10,250 our features and figure out what is gonna make 388 00:20:10,250 --> 00:20:11,560 our business successful. 389 00:20:11,560 --> 00:20:14,040 And we only have ten engineers at this point. 390 00:20:14,040 --> 00:20:16,170 We don't really have that much, that many resources 391 00:20:16,170 --> 00:20:19,150 to, to, to work with this. 392 00:20:19,150 --> 00:20:24,670 So, our first step of iteration is realizing that 393 00:20:24,670 --> 00:20:26,710 this is one problem. How do we solve this 394 00:20:26,710 --> 00:20:29,530 one problem? And this is maybe going to be 395 00:20:29,530 --> 00:20:31,280 a service. How do we get to a point 396 00:20:31,280 --> 00:20:32,990 where it can be a service? So first step 397 00:20:32,990 --> 00:20:37,900 is isolating the data. So ActiveRecord gives you these 398 00:20:37,900 --> 00:20:42,290 really nice things. Associations, has_many, has_one. Things that really 399 00:20:42,290 --> 00:20:45,870 make the joins between your tables easier. 400 00:20:45,870 --> 00:20:48,290 When you have a service, you don't have joins. 401 00:20:48,290 --> 00:20:50,110 So these need to go away. These, these just 402 00:20:50,110 --> 00:20:52,430 don't exist. But it's actually really easy to get 403 00:20:52,430 --> 00:20:57,010 rid of these, honestly. A product has saves, you 404 00:20:57,010 --> 00:20:59,920 know. You save this product into a collection. You 405 00:20:59,920 --> 00:21:02,560 know, we could just take that where clause that, 406 00:21:02,560 --> 00:21:04,770 that ActiveRecord was gonna sort of do for us 407 00:21:04,770 --> 00:21:08,240 with, with a join, and just pull it out 408 00:21:08,240 --> 00:21:10,510 as a, as a where. 409 00:21:10,510 --> 00:21:14,190 Really not that hard, actually. And you know what, 410 00:21:14,190 --> 00:21:16,990 ActiveRecord also gives us ways of talking to a 411 00:21:16,990 --> 00:21:20,160 different database. But, you know, we can actually use 412 00:21:20,160 --> 00:21:22,510 this to just pretend it's in a different database. 413 00:21:22,510 --> 00:21:26,810 Establish connection allows you to have this model live 414 00:21:26,810 --> 00:21:28,880 in this database and all the rest of your 415 00:21:28,880 --> 00:21:31,060 stuff live in your main frames. It's really not 416 00:21:31,060 --> 00:21:32,310 that hard. 417 00:21:32,310 --> 00:21:34,080 And, one of the key things is that each 418 00:21:34,080 --> 00:21:37,390 step of this, each slight change, is deployable, and 419 00:21:37,390 --> 00:21:40,330 it's testable. And you can deploy it to staging. 420 00:21:40,330 --> 00:21:42,620 You can click around to see where, where your 421 00:21:42,620 --> 00:21:45,200 test coverage is maybe missing. Figure out where, where 422 00:21:45,200 --> 00:21:46,380 it breaks. And one thing is that, that I 423 00:21:46,380 --> 00:21:48,300 will say, is you might be doubling your database 424 00:21:48,300 --> 00:21:51,480 connections at this point. And when you database hits 425 00:21:51,480 --> 00:21:56,300 the max connections, just everything stops. It just stops 426 00:21:56,300 --> 00:21:57,160 working. 427 00:21:57,160 --> 00:22:04,160 So, we've learned this lesson the hard way. But 428 00:22:04,640 --> 00:22:07,060 now that we have this code deployed that pretends 429 00:22:07,060 --> 00:22:08,460 like it's in a different database, we can make 430 00:22:08,460 --> 00:22:11,970 it a different database. Without actually that much work. 431 00:22:11,970 --> 00:22:15,760 You have a Postgres - we love Postgres - 432 00:22:15,760 --> 00:22:17,750 we have a master database. And we spin up 433 00:22:17,750 --> 00:22:20,860 a replica over here, and we put the site 434 00:22:20,860 --> 00:22:27,620 into maintenance mode. If you have more critical interactions 435 00:22:27,620 --> 00:22:29,880 with your company that maintenance mode might, might not 436 00:22:29,880 --> 00:22:33,400 be possible, brain tree has some really great blogs 437 00:22:33,400 --> 00:22:37,540 and, I think, talks about this. But, for us, 438 00:22:37,540 --> 00:22:40,140 the operational overhead of making it so we can 439 00:22:40,140 --> 00:22:42,690 do this without the site being down was way 440 00:22:42,690 --> 00:22:44,000 more than it was worth. 441 00:22:44,000 --> 00:22:45,780 So we just take the site down. Push in 442 00:22:45,780 --> 00:22:48,200 new database dot yml, saying now this connection is 443 00:22:48,200 --> 00:22:50,310 talking to this database. And we just promote that 444 00:22:50,310 --> 00:22:52,960 to master and restart everything, bring the site back, 445 00:22:52,960 --> 00:22:57,510 and now we have two databases. Five minutes of 446 00:22:57,510 --> 00:23:02,350 downtime. Not, not that bad, actually. And you know, 447 00:23:02,350 --> 00:23:03,910 after the fact, you can clean up. You know, 448 00:23:03,910 --> 00:23:05,940 we have a redundant table over here. A bunch 449 00:23:05,940 --> 00:23:07,580 of redundant tables over here. You just truncate them 450 00:23:07,580 --> 00:23:11,590 and delete them. Very carefully. 451 00:23:11,590 --> 00:23:13,820 Not that hard. And actually, you know what, at 452 00:23:13,820 --> 00:23:16,290 this point, you might not need a service. You 453 00:23:16,290 --> 00:23:18,830 might be done. Your site might just be working. 454 00:23:18,830 --> 00:23:23,500 It's fine. For us, we, we knew, based on 455 00:23:23,500 --> 00:23:25,920 how things were going, that we were gonna have 456 00:23:25,920 --> 00:23:27,500 to horizontally shared this data. Now it's in a 457 00:23:27,500 --> 00:23:31,000 different database. That is gonna have to be many 458 00:23:31,000 --> 00:23:32,330 databases. 459 00:23:32,330 --> 00:23:34,130 And we want to hide that complexity. We don't, 460 00:23:34,130 --> 00:23:36,210 we do not want our main application to have 461 00:23:36,210 --> 00:23:40,470 any of that code. So, we know we're gonna 462 00:23:40,470 --> 00:23:43,110 have a service. Now isolate the interface. And now 463 00:23:43,110 --> 00:23:45,350 what this, by this, I mean, how are you 464 00:23:45,350 --> 00:23:48,150 actually accessing this data? And what is your long-term 465 00:23:48,150 --> 00:23:52,190 goal? You know, ours is sharding, and Wanelo is 466 00:23:52,190 --> 00:23:54,150 saves, and anytime we're actually getting saves, we're either 467 00:23:54,150 --> 00:23:55,960 looking at by product - a product has these 468 00:23:55,960 --> 00:23:57,430 saves - or we have it by user. A 469 00:23:57,430 --> 00:23:58,770 user has saved these things. 470 00:23:58,770 --> 00:24:01,610 So, this is actually really helpful, how to plan 471 00:24:01,610 --> 00:24:04,630 out what, your DSL, what your API is gonna 472 00:24:04,630 --> 00:24:07,080 look like, you know. We know that things are 473 00:24:07,080 --> 00:24:09,060 gonna have to get to it via product or 474 00:24:09,060 --> 00:24:12,880 via user. And so, you have some where clauses, 475 00:24:12,880 --> 00:24:15,780 you know. Where is ActiveRecord? We are not gonna 476 00:24:15,780 --> 00:24:16,680 have ActiveRecord. 477 00:24:16,680 --> 00:24:20,170 So, instead a save has a by_product method that 478 00:24:20,170 --> 00:24:23,340 is also. Oh, so one thing I will say 479 00:24:23,340 --> 00:24:26,310 is, at this point it's really helpful to remove 480 00:24:26,310 --> 00:24:28,840 redundancy. If you have different ways of accessing kind 481 00:24:28,840 --> 00:24:31,500 of the same data, do you really need that? 482 00:24:31,500 --> 00:24:33,200 You know. Can you change the product to mean 483 00:24:33,200 --> 00:24:35,150 that you don't actually have as many of these 484 00:24:35,150 --> 00:24:37,820 fingers as you have. And very soon, things are 485 00:24:37,820 --> 00:24:39,850 gonna break. So if you don't have tests, this 486 00:24:39,850 --> 00:24:44,200 is a really good place to add tests. 487 00:24:44,200 --> 00:24:49,470 So, now, we have a, a small sort of 488 00:24:49,470 --> 00:24:52,200 Ruby DSL. How do we actually pull that out? 489 00:24:52,200 --> 00:24:54,350 And I would say, right now, it actually doesn't 490 00:24:54,350 --> 00:24:56,110 need to be a service. Really what you need 491 00:24:56,110 --> 00:24:57,490 is a client. And how do you build the 492 00:24:57,490 --> 00:25:01,410 client out? And that's where adapters really come in. 493 00:25:01,410 --> 00:25:03,410 So, we use the module. This could be a 494 00:25:03,410 --> 00:25:05,190 base class. Some of the reasons why we thought 495 00:25:05,190 --> 00:25:07,080 we needed a module. Maybe we didn't Maybe we 496 00:25:07,080 --> 00:25:08,940 could have actually done this as, as a base 497 00:25:08,940 --> 00:25:12,630 class. But now, a save is a SaveClient. And 498 00:25:12,630 --> 00:25:16,710 that SaveClient is where your fingers go as, their 499 00:25:16,710 --> 00:25:19,920 class methods. And one thing I would point out 500 00:25:19,920 --> 00:25:23,860 is that that, that finder is calling through to 501 00:25:23,860 --> 00:25:27,090 an adapter. A database adapter. 502 00:25:27,090 --> 00:25:29,190 And really, that's what's wrapping up all your database 503 00:25:29,190 --> 00:25:32,960 calls. Hiding them from, from your application. And really 504 00:25:32,960 --> 00:25:34,910 one of the core pieces of this is that 505 00:25:34,910 --> 00:25:38,160 your database, your, your adapter is your feature flag. 506 00:25:38,160 --> 00:25:41,000 It's also deployable in place, you know. You, you 507 00:25:41,000 --> 00:25:42,950 can have this in your lib folder. You can 508 00:25:42,950 --> 00:25:44,710 actually start a gem now. And you can just 509 00:25:44,710 --> 00:25:47,720 deploy it and it's, your main application is still 510 00:25:47,720 --> 00:25:50,610 talking directly to this other database. But you're starting 511 00:25:50,610 --> 00:25:52,910 to build out your client. And later, when you 512 00:25:52,910 --> 00:25:54,350 have a service, you can replace it with a 513 00:25:54,350 --> 00:25:55,630 different adapter. 514 00:25:55,630 --> 00:25:58,940 So that adapter gives you back, when you, when 515 00:25:58,940 --> 00:26:02,750 you call, like, a, you know, you get save.by_product 516 00:26:02,750 --> 00:26:04,800 and you call all on it, that's actually, when 517 00:26:04,800 --> 00:26:11,080 you call by_product, you're getting back a relation. And 518 00:26:11,080 --> 00:26:13,930 this is something that we thought we didn't need, 519 00:26:13,930 --> 00:26:16,800 but turns out ActiveRecord does this for very good 520 00:26:16,800 --> 00:26:20,640 reason, and that's because threads and because state. 521 00:26:20,640 --> 00:26:23,370 If you say I want this type of data 522 00:26:23,370 --> 00:26:25,590 and you save it away to a variable, and 523 00:26:25,590 --> 00:26:28,640 now you call some other method, order.by_this, and it 524 00:26:28,640 --> 00:26:31,380 changes state, you might do something else on this 525 00:26:31,380 --> 00:26:34,340 variable over here, not realizing that you've altered its 526 00:26:34,340 --> 00:26:36,510 state later. So any time you make a change, 527 00:26:36,510 --> 00:26:38,130 a state change on these objects, you really want 528 00:26:38,130 --> 00:26:40,360 to get back a new instance. A different instance 529 00:26:40,360 --> 00:26:43,410 with different state. And when you call, you know, 530 00:26:43,410 --> 00:26:45,780 all, first, or pluck any of these things, what 531 00:26:45,780 --> 00:26:49,400 you're really calling it on is your relation instance. 532 00:26:49,400 --> 00:26:52,100 And the, the key thing that we learned is 533 00:26:52,100 --> 00:26:55,000 that that relation is sharable between all of your 534 00:26:55,000 --> 00:26:57,150 adapters. So the actual work done to get the 535 00:26:57,150 --> 00:27:00,990 data is on your adapters. So the relation delegates 536 00:27:00,990 --> 00:27:03,030 back. 537 00:27:03,030 --> 00:27:07,980 So, in our database adapter, the thing actually getting 538 00:27:07,980 --> 00:27:11,560 the data, is ActiveRecord. We've just moved our ActiveRecord 539 00:27:11,560 --> 00:27:14,080 class into the library and hidden it from our 540 00:27:14,080 --> 00:27:18,500 application. We, in this case, we were using ActiveRecord, 541 00:27:18,500 --> 00:27:20,970 so you could just, we'd do ActiveRecord. If you 542 00:27:20,970 --> 00:27:23,730 have another favorite database adapter, great. 543 00:27:23,730 --> 00:27:30,590 So you call Save.by_product. You, you get an adapter 544 00:27:30,590 --> 00:27:34,530 that, or so the, sorry, that calls through to 545 00:27:34,530 --> 00:27:38,010 the adapter and gives you back relation. You call 546 00:27:38,010 --> 00:27:40,860 Relation dot all and that just delegates back to 547 00:27:40,860 --> 00:27:44,760 the adapter, which calls through to an ActiveRecord, gets 548 00:27:44,760 --> 00:27:48,380 to your data, takes the attributes out of it 549 00:27:48,380 --> 00:27:50,260 and gives you back an instance of, of your 550 00:27:50,260 --> 00:27:52,460 core class. Because you're hiding ActiveRecord. You don't want 551 00:27:52,460 --> 00:27:55,250 to get back an ActiveRecord class. 552 00:27:55,250 --> 00:27:56,740 And I would say it's critical to deploy at 553 00:27:56,740 --> 00:28:00,200 this steps, because you've made mistakes. I guarantee you've 554 00:28:00,200 --> 00:28:02,570 made mistakes. And the cost of fixing this is 555 00:28:02,570 --> 00:28:06,050 really low right now, as opposed to spending a 556 00:28:06,050 --> 00:28:08,160 lot of time trying to design your server, how 557 00:28:08,160 --> 00:28:09,940 that's going to interact, and realize, whoa, whoa, whoa, 558 00:28:09,940 --> 00:28:11,750 whoa, we screwed up the client. All of this 559 00:28:11,750 --> 00:28:13,780 work we've done on the service, we have to 560 00:28:13,780 --> 00:28:17,470 through away because we did it wrong. 561 00:28:17,470 --> 00:28:20,940 Now, you have a client. Now you need a 562 00:28:20,940 --> 00:28:24,680 server. And it doesn't matter. What, whatever you want. 563 00:28:24,680 --> 00:28:27,210 It's fine. It's actually the, the cost of writing 564 00:28:27,210 --> 00:28:31,440 this is really, really low. Because the server is 565 00:28:31,440 --> 00:28:34,220 the simplest part of this, and if you did 566 00:28:34,220 --> 00:28:36,310 it wrong, if you chose the wrong transport method, 567 00:28:36,310 --> 00:28:38,720 mechanism, you know what? You build a new adapter. 568 00:28:38,720 --> 00:28:41,400 That's actually really quite easy. 569 00:28:41,400 --> 00:28:44,350 So let me take a moment to just reiterate, 570 00:28:44,350 --> 00:28:47,310 why should we have deployed by now? And I'd 571 00:28:47,310 --> 00:28:49,860 say it's because the client is much, much more 572 00:28:49,860 --> 00:28:52,610 complicated than the server, so your bugs are in 573 00:28:52,610 --> 00:28:54,510 the client. Your bugs are not in the server 574 00:28:54,510 --> 00:28:57,930 at this point. And the server is going to 575 00:28:57,930 --> 00:28:59,700 be dictated by the choices you've made in the 576 00:28:59,700 --> 00:29:02,970 client. So if you've made wrong choices and you 577 00:29:02,970 --> 00:29:07,090 build your server, you've built the wrong server. 578 00:29:07,090 --> 00:29:09,640 We used Sinatra and Oj, just cause it's awesome. 579 00:29:09,640 --> 00:29:12,150 It just, it really just works. It's small and 580 00:29:12,150 --> 00:29:15,710 it's useful. We thought we would need to move 581 00:29:15,710 --> 00:29:17,800 away from HTTP, but we've grown a lot and 582 00:29:17,800 --> 00:29:19,150 we haven't had to do this yet. It just 583 00:29:19,150 --> 00:29:21,720 works. Things that we thought we would have to 584 00:29:21,720 --> 00:29:24,610 change immediately, you know, it's almost a year later 585 00:29:24,610 --> 00:29:26,850 and it's just. 586 00:29:26,850 --> 00:29:30,380 OK, so now we use the service. And that's 587 00:29:30,380 --> 00:29:32,530 really a feature flag. You just write a new 588 00:29:32,530 --> 00:29:37,210 adapter that talks to the service instead of the 589 00:29:37,210 --> 00:29:39,900 database. So now you call, you know, by_product, you 590 00:29:39,900 --> 00:29:42,540 get a HTTP, it calls through to the HTTPAdapter 591 00:29:42,540 --> 00:29:45,050 which gets you back the same time of relation. 592 00:29:45,050 --> 00:29:47,430 When you call all on it, you know, it 593 00:29:47,430 --> 00:29:49,290 calls adapter dot all, which now goes to an 594 00:29:49,290 --> 00:29:52,300 HTTPClass that actually gets the JSON and takes the 595 00:29:52,300 --> 00:29:54,100 attributes out of it and gives you back your 596 00:29:54,100 --> 00:29:56,530 save class. You're getting back the same object you're 597 00:29:56,530 --> 00:29:59,090 getting back as save. 598 00:29:59,090 --> 00:30:03,330 So, retrospective. Great. We've isolated the data. We've isolated 599 00:30:03,330 --> 00:30:06,420 the interface. We started to build our DSL. We've 600 00:30:06,420 --> 00:30:09,760 pulled that DSL out into a gem. Now we've, 601 00:30:09,760 --> 00:30:11,150 now that we actually kind of understand what that 602 00:30:11,150 --> 00:30:13,330 gem needs to do, we can launch the service 603 00:30:13,330 --> 00:30:15,380 and then just build a new adapter to switch 604 00:30:15,380 --> 00:30:16,430 to this. 605 00:30:16,430 --> 00:30:18,880 If, I would say that if we hadn't, if 606 00:30:18,880 --> 00:30:20,550 we had realized that this was the order that 607 00:30:20,550 --> 00:30:21,950 we needed to do it on, we, we would 608 00:30:21,950 --> 00:30:25,330 have done this in two weeks. Instead. So that 609 00:30:25,330 --> 00:30:27,370 first part of it was like a day worth 610 00:30:27,370 --> 00:30:31,020 of work. That second part of it was like, 611 00:30:31,020 --> 00:30:34,470 three hours worth of work. And deployed immediately. 612 00:30:34,470 --> 00:30:37,630 The, the harder part was realizing that we needed 613 00:30:37,630 --> 00:30:39,710 an adapter. And at this point, people, we didn't 614 00:30:39,710 --> 00:30:42,360 really see anything about hexagonal architecture. This might have 615 00:30:42,360 --> 00:30:45,080 been before some of those talks and papers have 616 00:30:45,080 --> 00:30:49,040 been coming out. But, but it's actually really useful. 617 00:30:49,040 --> 00:30:52,980 Tests. We, we use Sunspot for some of our 618 00:30:52,980 --> 00:30:54,970 Solar things. And we're already used to spinning up 619 00:30:54,970 --> 00:30:58,880 a Solar instance using a gem, trap-it-up. You know 620 00:30:58,880 --> 00:31:03,150 you can do that for your integration tests. But 621 00:31:03,150 --> 00:31:05,450 for unit tests, you know what, we have tests 622 00:31:05,450 --> 00:31:08,410 around all of this. So we can have tests 623 00:31:08,410 --> 00:31:11,030 around a fake adapter that proves that it behaves 624 00:31:11,030 --> 00:31:14,170 the right way, and then that just saves, saves 625 00:31:14,170 --> 00:31:18,400 data in, in memory, in your application. 626 00:31:18,400 --> 00:31:21,640 And redundant tests, you know, you might say, eh, 627 00:31:21,640 --> 00:31:24,730 do I really need this test? Yes. Because one 628 00:31:24,730 --> 00:31:26,809 thing, you can delete your tests later that are 629 00:31:26,809 --> 00:31:29,940 redundant, and you want to be really confident that, 630 00:31:29,940 --> 00:31:34,840 that when you do switch over, it's gonna work. 631 00:31:34,840 --> 00:31:38,700 Foreman and Subcontractor are, are really helpful for this 632 00:31:38,700 --> 00:31:42,100 kind of thing. So subcontractor is a gem that 633 00:31:42,100 --> 00:31:45,650 really says, for this service, change directory over here 634 00:31:45,650 --> 00:31:48,670 and run this command in a completely isolated Bundler 635 00:31:48,670 --> 00:31:51,590 environment. Cause you really don't want to mix your, 636 00:31:51,590 --> 00:31:55,200 your dependencies. You don't want your server dependencies and 637 00:31:55,200 --> 00:31:58,340 the versions to be conflicting with your main application 638 00:31:58,340 --> 00:32:00,170 dependencies. You want to be able to, to change 639 00:32:00,170 --> 00:32:01,220 those separately. 640 00:32:01,220 --> 00:32:03,540 OK. So what about a new app? I'm spinning 641 00:32:03,540 --> 00:32:07,350 up a completely new thing. Not extracting. Totally new. 642 00:32:07,350 --> 00:32:09,390 How do I do that? How do I iterate 643 00:32:09,390 --> 00:32:12,220 on something that doesn't exist yet? And I would 644 00:32:12,220 --> 00:32:13,950 say that some of the lessons that, that we've 645 00:32:13,950 --> 00:32:16,070 learned from this, and actually from just doing our 646 00:32:16,070 --> 00:32:20,470 product development in general is iterate. Find a way 647 00:32:20,470 --> 00:32:23,430 to get to the smallest deployable thing as quickly 648 00:32:23,430 --> 00:32:27,590 as possible, and whatever tool you use to deploy, 649 00:32:27,590 --> 00:32:30,710 to spin up infrastructure, one of the sort of 650 00:32:30,710 --> 00:32:35,080 heuristics that, that, that we've found is really focus 651 00:32:35,080 --> 00:32:37,770 on organizing that code around being able to change 652 00:32:37,770 --> 00:32:41,710 things easily, and understand how this thing is different 653 00:32:41,710 --> 00:32:42,750 from this thing. 654 00:32:42,750 --> 00:32:45,490 You know, the Chef makes it really easy to 655 00:32:45,490 --> 00:32:48,760 define a giant hash of global state over here 656 00:32:48,760 --> 00:32:50,800 and then just run this thing over here and 657 00:32:50,800 --> 00:32:53,830 it's magic and it'll just do it. When you 658 00:32:53,830 --> 00:32:56,420 actually start to spin up different services, this thing 659 00:32:56,420 --> 00:32:58,940 is gonna need to be slightly different than this 660 00:32:58,940 --> 00:33:03,050 thing. So how can your code make that as 661 00:33:03,050 --> 00:33:05,970 easily understandable as possible? 662 00:33:05,970 --> 00:33:10,210 So feature flags, also, on or off. Do customers 663 00:33:10,210 --> 00:33:14,440 see this, or they don't. But you know what? 664 00:33:14,440 --> 00:33:18,860 Maybe it's just kind of half on. Maybe everything, 665 00:33:18,860 --> 00:33:22,670 every request that's gonna actually call through this, use 666 00:33:22,670 --> 00:33:26,950 Sidekiq to just every time, just spin off a 667 00:33:26,950 --> 00:33:30,400 job to hammer your, your service. And if it 668 00:33:30,400 --> 00:33:33,170 breaks, you've learned that before you've launched it to, 669 00:33:33,170 --> 00:33:35,430 to your users. There's a lot of other ways 670 00:33:35,430 --> 00:33:36,700 you can do this. 671 00:33:36,700 --> 00:33:39,450 On to these five years users, and let's see 672 00:33:39,450 --> 00:33:41,930 how we can break it, see how the interaction 673 00:33:41,930 --> 00:33:44,680 feels. It's, it's really useful. 674 00:33:44,680 --> 00:33:48,330 Also, one thing that we found is, it's often 675 00:33:48,330 --> 00:33:52,680 helpful to inter, integrate very deeply before you go 676 00:33:52,680 --> 00:33:56,240 widely. So, if you have a code-interaction that goes 677 00:33:56,240 --> 00:33:59,240 through all this process, do it kind of once 678 00:33:59,240 --> 00:34:01,970 through all the process, without worrying so much about 679 00:34:01,970 --> 00:34:04,630 the edge cases or the errors, because those are, 680 00:34:04,630 --> 00:34:06,230 you're gonna find those. Those are gonna, those are 681 00:34:06,230 --> 00:34:08,639 gonna come up. But it's re- really useful to 682 00:34:08,639 --> 00:34:11,069 kind of go all the way through the interaction 683 00:34:11,069 --> 00:34:13,050 knowing that it's really kind of sketch and doesn't 684 00:34:13,050 --> 00:34:17,760 do everything, and really let that drive the design. 685 00:34:17,760 --> 00:34:19,969 And this is also a thing of like letting 686 00:34:19,969 --> 00:34:23,250 the feature kind of drive the design and figuring 687 00:34:23,250 --> 00:34:25,310 out, what is the pattern for how you really 688 00:34:25,310 --> 00:34:28,020 need to organize the code, before you have. Don't 689 00:34:28,020 --> 00:34:29,739 just like whiteboard everything and say this is gonna 690 00:34:29,739 --> 00:34:32,340 be perfect. You know. 691 00:34:32,340 --> 00:34:34,590 Production is going to destroy every single design that 692 00:34:34,590 --> 00:34:39,619 you think you have. Also, if something seems clever, 693 00:34:39,619 --> 00:34:41,899 it's bad. If you're like, eh, this is kind 694 00:34:41,899 --> 00:34:44,139 of cool. No. No, no, no. It's bad. Complexity 695 00:34:44,139 --> 00:34:47,550 is gonna come to you. Don't, don't seek it 696 00:34:47,550 --> 00:34:51,239 out. It's, it's, it's evil. 697 00:34:51,239 --> 00:34:53,110 So if there are any kind of take aways 698 00:34:53,110 --> 00:34:57,360 from this, I would say you know, hexagonal architecture 699 00:34:57,360 --> 00:34:59,930 is really cool, but you don't have to design 700 00:34:59,930 --> 00:35:03,390 it from the start. If you have trust, if 701 00:35:03,390 --> 00:35:06,880 everyone in the organization has trust that you can 702 00:35:06,880 --> 00:35:09,110 do your job and that you're all going, working 703 00:35:09,110 --> 00:35:13,660 together to build an awesome product, an awesome company, 704 00:35:13,660 --> 00:35:16,400 you can fix this later. You can actually let 705 00:35:16,400 --> 00:35:20,780 the needs of your product determine where your boundaries 706 00:35:20,780 --> 00:35:21,830 are. 707 00:35:21,830 --> 00:35:27,119 So, thank you. I actually have a few minutes 708 00:35:27,119 --> 00:35:29,880 of, of questions, to answer questions.