WEBVTT 00:00:17.039 --> 00:00:18.769 ERIC SAXBY: Thanks for coming. 00:00:18.769 --> 00:00:21.279 So, I am Eric Saxby. Today I'm gonna be talking 00:00:21.279 --> 00:00:24.609 about iterating towards service-oriented 00:00:24.609 --> 00:00:28.869 architecture. I may have time for questions 00:00:28.869 --> 00:00:30.449 at the end, but I may not. 00:00:30.449 --> 00:00:31.380 This might go forty minutes, 00:00:31.380 --> 00:00:32.598 so I'm just gonna get started. 00:00:32.598 --> 00:00:35.829 So, if you're like me, you can't really pay 00:00:35.829 --> 00:00:38.500 attention in talking about programming 00:00:38.500 --> 00:00:39.250 unless there are pictures 00:00:39.250 --> 00:00:44.559 of cats. So really, we're going from this to 00:00:44.559 --> 00:00:48.579 something a little bit more like this. But working 00:00:48.579 --> 00:00:50.729 together, as a family. 00:00:50.729 --> 00:00:53.229 So why should you care? Why should you listen 00:00:53.229 --> 00:00:57.869 to me? So you may not know this, but 00:00:57.869 --> 00:01:01.799 I'm kind of a big deal. But, but really, 00:01:01.799 --> 00:01:04.519 I've, I've actually not been doing this for that 00:01:04.519 --> 00:01:07.490 long. I've been doing Rails for about six years. 00:01:07.490 --> 00:01:11.490 Before and after that, I had been using various 00:01:11.490 --> 00:01:17.149 different technologies. And I have been fortunate to work 00:01:17.149 --> 00:01:19.540 with some very smart people and be able to 00:01:19.540 --> 00:01:21.750 learn from them and to break a lot of 00:01:21.750 --> 00:01:24.900 things really quickly. And right now I work at 00:01:24.900 --> 00:01:28.030 Wanelo. Also, I'm trying to collect all of the 00:01:28.030 --> 00:01:30.770 buzzwords on my resume. I have more. I have 00:01:30.770 --> 00:01:32.070 more than just this. 00:01:32.070 --> 00:01:37.040 But, so, why is Wanelo important? We're like a 00:01:37.040 --> 00:01:40.940 social network for shopping, but really it's because we 00:01:40.940 --> 00:01:44.670 have many millions of users, active users, and databases 00:01:44.670 --> 00:01:46.840 with billions of records. And we've gone through the 00:01:46.840 --> 00:01:50.040 pain of getting there and keeping the site actually 00:01:50.040 --> 00:01:54.390 running. So, you can save any product from any 00:01:54.390 --> 00:01:56.750 store on the internet into your own collections and 00:01:56.750 --> 00:01:59.890 have your own wishlist. That's what we do. 00:01:59.890 --> 00:02:04.260 More importantly, we've gone from having this as one 00:02:04.260 --> 00:02:08.199 main Rails application, doing all of this, to a 00:02:08.199 --> 00:02:11.870 central Rails application that's still fairly large, but supported 00:02:11.870 --> 00:02:15.200 by a lot of services. We've done as much 00:02:15.200 --> 00:02:17.340 as we could open source. Some of the, the 00:02:17.340 --> 00:02:20.120 business domain logic, it's really hard to open source. 00:02:20.120 --> 00:02:23.080 But, but we try as much as possible. 00:02:23.080 --> 00:02:26.290 We've done almost all of this in Ruby, including 00:02:26.290 --> 00:02:29.999 some things that people who prefer other languages say 00:02:29.999 --> 00:02:32.549 can't be done in Ruby. And we've done this 00:02:32.549 --> 00:02:36.439 with a very, very small team very quickly. 00:02:36.439 --> 00:02:39.450 If you're like me, though, you're really not so 00:02:39.450 --> 00:02:42.469 interested in the success story as you're interested in, 00:02:42.469 --> 00:02:45.659 how did you screw up, how did you, how 00:02:45.659 --> 00:02:50.099 did you break. So, let me take you on 00:02:50.099 --> 00:02:56.260 a journey to another company that a friend of 00:02:56.260 --> 00:03:03.260 mine once, recently, called "The Anti-Pattern Goldmine." Completely hypothetical 00:03:03.739 --> 00:03:06.400 company. Not naming any names. That I may or 00:03:06.400 --> 00:03:07.909 may not have worked for. Some of you in 00:03:07.909 --> 00:03:12.359 the audience may or may not have worked for. 00:03:12.359 --> 00:03:13.620 After some of this story, you might think you 00:03:13.620 --> 00:03:18.169 did. Come in, it's a startup. It's new. Small 00:03:18.169 --> 00:03:20.230 team. And come in and say, wow. For a 00:03:20.230 --> 00:03:26.730 startup, you have a lot of code that's really, 00:03:26.730 --> 00:03:31.099 really tangled. It's all Rails 2 code base. You 00:03:31.099 --> 00:03:33.519 know, if you remember vendoring Rails, we, we did 00:03:33.519 --> 00:03:36.379 that. If you remember how vendoring Rails can go 00:03:36.379 --> 00:03:38.519 wrong? Yeah, yeah. Yeah, that was there. That was 00:03:38.519 --> 00:03:40.159 there. 00:03:40.159 --> 00:03:41.599 And I think a lot of this might have 00:03:41.599 --> 00:03:45.709 come down to the fact that at least early 00:03:45.709 --> 00:03:50.569 on, success of a product, success of a feature, 00:03:50.569 --> 00:03:53.689 really was launching it as quickly as possible. And 00:03:53.689 --> 00:03:56.219 no, no, no, no. Don't worry about that stuff. 00:03:56.219 --> 00:03:59.849 Don't worry about design. And we have thirty engineers 00:03:59.849 --> 00:04:03.279 doing that as rapidly as possible. Like, five or 00:04:03.279 --> 00:04:06.809 six teams all doing that as rapidly as possible, 00:04:06.809 --> 00:04:10.059 trying to get it into production. And releases were 00:04:10.059 --> 00:04:12.999 a mess, you know. I'm sure a lot of 00:04:12.999 --> 00:04:19.029 you can relate to this. Deployments. Multiple hours with 00:04:19.029 --> 00:04:21.449 all of this engineer, all these engineers trying to 00:04:21.449 --> 00:04:23.449 put all of this code in as quickly as 00:04:23.449 --> 00:04:29.410 possible. Invariably, deployments went wrong. 00:04:29.410 --> 00:04:34.320 And eventually this got a little bit faster from, 00:04:34.320 --> 00:04:39.050 from monthly releases to weekly releases. But then we 00:04:39.050 --> 00:04:42.900 would have code freezes, where for three months, everyone's 00:04:42.900 --> 00:04:44.960 trying to cram in all the features without deploying, 00:04:44.960 --> 00:04:46.510 and you can imagine where that goes. 00:04:46.510 --> 00:04:50.140 And, over time, really, things are just getting worse. 00:04:50.140 --> 00:04:54.070 We, our rough estimates that we're trying to give 00:04:54.070 --> 00:04:57.380 to see how long things are gonna last come 00:04:57.380 --> 00:04:59.000 back, and we say, oh, that means we were 00:04:59.000 --> 00:05:00.680 deploying on this date. You know, hey, we told 00:05:00.680 --> 00:05:02.930 the board, great. That's awesome. You're gonna deploy on 00:05:02.930 --> 00:05:06.940 this date. Ah. Whoo. With a deadline like that, 00:05:06.940 --> 00:05:09.450 the only really way to meet it is to 00:05:09.450 --> 00:05:13.230 make serious shortcuts. And then as soon as a 00:05:13.230 --> 00:05:18.410 product finishes, because we're invariably missing those deadlines, and 00:05:18.410 --> 00:05:20.580 there's the next project that is supposed to be 00:05:20.580 --> 00:05:22.930 out in, in a week, based on the estimates 00:05:22.930 --> 00:05:26.620 that you were supposed to do. Team gets dispersed. 00:05:26.620 --> 00:05:30.580 Finish these new projects. And no matter how worse 00:05:30.580 --> 00:05:32.310 case we are in our estimates, it's just never 00:05:32.310 --> 00:05:35.380 worst case enough. It just, it just keeps getting 00:05:35.380 --> 00:05:36.510 worse. 00:05:36.510 --> 00:05:39.910 So some of you might, might be familiar with 00:05:39.910 --> 00:05:44.340 this story. So, during the course of this, I, 00:05:44.340 --> 00:05:47.500 I, I, I think I've learned a lot. You 00:05:47.500 --> 00:05:49.580 know, programming is one of the most fun careers 00:05:49.580 --> 00:05:51.960 that I've, that I've had, and when it's not 00:05:51.960 --> 00:05:55.710 fun, you know, you know something's wrong. And keep 00:05:55.710 --> 00:05:58.860 reading about service-oriented architecture, and, you know, a lot 00:05:58.860 --> 00:06:01.970 of us latch onto this. This is the solution 00:06:01.970 --> 00:06:03.690 to all of our problems. And you know at 00:06:03.690 --> 00:06:06.420 dev ops, that's pretty cool over there. You know, 00:06:06.420 --> 00:06:07.530 dev ops is the answer. That's how we're gonna 00:06:07.530 --> 00:06:10.310 do services. It's just gonna, it's a done deal. 00:06:10.310 --> 00:06:12.370 All we have to do is do services, dev 00:06:12.370 --> 00:06:15.110 opsy, and like, we're done. 00:06:15.110 --> 00:06:19.150 So around this time, I move into the operations, 00:06:19.150 --> 00:06:22.670 not, not the operations team, cause dev ops. Not 00:06:22.670 --> 00:06:25.220 the operations team. Not the team over there that 00:06:25.220 --> 00:06:27.160 you just throw the things over the wall and 00:06:27.160 --> 00:06:29.170 they just make it work. No, no, no. Not, 00:06:29.170 --> 00:06:32.600 not the operations team at all. 00:06:32.600 --> 00:06:36.010 And certainly me, a number of other people in 00:06:36.010 --> 00:06:40.600 the engineering team, I, I could say, really decided 00:06:40.600 --> 00:06:44.490 services are the only way forward. And really quickly, 00:06:44.490 --> 00:06:47.510 product, and, and I don't mean an individual project 00:06:47.510 --> 00:06:49.460 manager or anything like that. By product, I mean 00:06:49.460 --> 00:06:52.120 really, all of the people, all of the teams 00:06:52.120 --> 00:06:56.080 going into designing the product, planning the product, really 00:06:56.080 --> 00:06:59.560 how it was going forward, they quickly come to 00:06:59.560 --> 00:07:01.720 the conclusion that services are really just that, just 00:07:01.720 --> 00:07:03.940 that, that thing getting in the way of us 00:07:03.940 --> 00:07:08.040 cranking out these features. Cause features is success. 00:07:08.040 --> 00:07:10.130 But at first, if at first you can't succeed, 00:07:10.130 --> 00:07:13.220 I have, I have, I have learned, you know, 00:07:13.220 --> 00:07:15.180 there's, there's a few things. You can take a 00:07:15.180 --> 00:07:21.930 firm stance, breathe deeply, and become the loudest person 00:07:21.930 --> 00:07:27.330 in the room. Really, really helps. Also, if there's 00:07:27.330 --> 00:07:29.740 anything that I've learned from my cats, it's throw 00:07:29.740 --> 00:07:34.490 shit, you know. Done. It really, really helps in 00:07:34.490 --> 00:07:36.340 situations like this. 00:07:36.340 --> 00:07:39.100 So, we, we have a few things coming up. 00:07:39.100 --> 00:07:43.850 A few products. These, these aren't necessarily concurrent, these, 00:07:43.850 --> 00:07:46.860 these features. These projects. But we have a new 00:07:46.860 --> 00:07:51.530 login feature and login, in this application, is as 00:07:51.530 --> 00:07:55.400 homegrown as you could possibly imagine it could go 00:07:55.400 --> 00:07:59.750 wrong. And it's this core, core login functionality, and 00:07:59.750 --> 00:08:01.810 we're like, no, no, no, no, no, no. No, 00:08:01.810 --> 00:08:03.430 no, no, no, no, no. That's not going in 00:08:03.430 --> 00:08:06.710 this code base. That's gonna be a new application, 00:08:06.710 --> 00:08:10.520 we're. That's the only way we're gonna do this. 00:08:10.520 --> 00:08:12.930 Otherwise we're just not gonna do this. It's gonna 00:08:12.930 --> 00:08:14.980 fail. We know it's gonna fail. So let's just 00:08:14.980 --> 00:08:16.930 skip to not doing it. 00:08:16.930 --> 00:08:22.230 And, also, we have this enterprise software over here, 00:08:22.230 --> 00:08:24.889 and we have this homegrown Rails application, and all 00:08:24.889 --> 00:08:28.080 the data really needs to kind of be synced 00:08:28.080 --> 00:08:30.830 between it in order for the company to actually 00:08:30.830 --> 00:08:35.700 work. So a lot of iterations on this, but 00:08:35.700 --> 00:08:37.880 really this time, we're gonna do it right, and 00:08:37.880 --> 00:08:40.870 it's gonna be a, a data service. We're gonna 00:08:40.870 --> 00:08:43.179 have our Enterprise software in our Rails app and 00:08:43.179 --> 00:08:45.589 that is totally just gonna make this an Enterprise-Rails 00:08:45.589 --> 00:08:46.190 app. It's gonna be amazing. 00:08:46.190 --> 00:08:49.930 And I remember saying this a lot to a 00:08:49.930 --> 00:08:52.129 lot of different people. Like, we're gonna screw this 00:08:52.129 --> 00:08:55.149 up. We're gonna fail. I know we're gonna fail. 00:08:55.149 --> 00:08:56.930 But it's the only option that we have. It's 00:08:56.930 --> 00:08:59.230 the only way that we're actually gonna be able 00:08:59.230 --> 00:09:01.470 to get to be able to do this and 00:09:01.470 --> 00:09:05.360 succeed at some point in the future. 00:09:05.360 --> 00:09:10.309 OK, so on, on a side track, hopefully this 00:09:10.309 --> 00:09:13.749 is not new to very many of you, but 00:09:13.749 --> 00:09:15.610 really there's a lot of different ways to do 00:09:15.610 --> 00:09:17.959 service-oriented architecture. It can mean a lot of different 00:09:17.959 --> 00:09:21.779 things. Read blog posts. You get a lot of 00:09:21.779 --> 00:09:24.550 different ideas. It can be synchronous, asynchronous. Use a 00:09:24.550 --> 00:09:28.279 message bus. Maybe some services require actually hitting them 00:09:28.279 --> 00:09:31.680 over HTTP, TCP sockets. You know, there's a lot 00:09:31.680 --> 00:09:34.589 of ways of different doing, of doing this kind 00:09:34.589 --> 00:09:35.670 of thing. 00:09:35.670 --> 00:09:37.680 But why would you really want to? So, you 00:09:37.680 --> 00:09:39.139 know, scale the organization. You have a lot of 00:09:39.139 --> 00:09:41.769 different teams. A lot of different engineers. Maybe you 00:09:41.769 --> 00:09:43.740 really want this team over here to just have 00:09:43.740 --> 00:09:47.240 their own code base that they deploy separately. Also, 00:09:47.240 --> 00:09:48.959 maybe you want to outsource this to a different 00:09:48.959 --> 00:09:50.740 team, and you really don't want to give all 00:09:50.740 --> 00:09:53.050 of the code to the other team. You really 00:09:53.050 --> 00:09:55.699 just want to isolate this and say, you know, 00:09:55.699 --> 00:10:00.939 you guys over here, you know, just, here, here's 00:10:00.939 --> 00:10:03.889 this. Sorry. I'm, I'm actually trying to explain it 00:10:03.889 --> 00:10:09.459 all gendered pronounce from my vocabulary but it's really 00:10:09.459 --> 00:10:11.860 hard. Sorry. 00:10:11.860 --> 00:10:15.540 So, you can also scale the code base, you 00:10:15.540 --> 00:10:20.019 know. Performance of this thing over here might be 00:10:20.019 --> 00:10:23.809 completely different than this thing over here, and that 00:10:23.809 --> 00:10:25.709 might actually be easier in a service. You can 00:10:25.709 --> 00:10:28.439 really tune the code to the work load. You 00:10:28.439 --> 00:10:31.369 might have this really complicated code that, for various 00:10:31.369 --> 00:10:33.439 reasons, needs to be complicated. It might be really 00:10:33.439 --> 00:10:36.029 complicated functionality. But you can, you might be able 00:10:36.029 --> 00:10:40.980 to hide that behind a, a clean interface. An 00:10:40.980 --> 00:10:41.559 API. 00:10:41.559 --> 00:10:46.019 And tests, usually, not always, but a small code 00:10:46.019 --> 00:10:49.470 base can mean fast tests. And if you're sitting 00:10:49.470 --> 00:10:52.110 around for hours waiting for tests to complete, you 00:10:52.110 --> 00:10:54.009 know, that can be, that can really eat into 00:10:54.009 --> 00:10:56.459 your productivity. But it, but it comes at a 00:10:56.459 --> 00:10:58.499 cost. All of this comes at a cost. 00:10:58.499 --> 00:11:00.720 And I think one of the things that, that 00:11:00.720 --> 00:11:04.180 I've been learning is that sometimes the cost of 00:11:04.180 --> 00:11:06.779 not doing this is greater than the cost of 00:11:06.779 --> 00:11:10.149 doing. The cost of the infrastructure, new servers, how 00:11:10.149 --> 00:11:12.740 they talk together. It's really complicated. Things will go 00:11:12.740 --> 00:11:15.129 wrong. But sometimes not doing this is gonna, is 00:11:15.129 --> 00:11:17.269 gonna mean that your productivity is going down and 00:11:17.269 --> 00:11:20.439 down and down and, and it actually is more 00:11:20.439 --> 00:11:21.149 costly. 00:11:21.149 --> 00:11:24.269 OK, so back to these projects. We have this 00:11:24.269 --> 00:11:27.920 data service. It's sometimes, I think six engineers, sometimes 00:11:27.920 --> 00:11:31.899 eight. I don't, I don't really remember. And nine 00:11:31.899 --> 00:11:35.749 months of, of work. It's really complicated, like state 00:11:35.749 --> 00:11:38.740 transactions, and this is really critical data that really 00:11:38.740 --> 00:11:43.920 needs to be done right. And there are, so, 00:11:43.920 --> 00:11:46.369 there, there might have been, when deploying it, some, 00:11:46.369 --> 00:11:48.369 some slight data things. But they were fixable. Really 00:11:48.369 --> 00:11:50.189 quickly. It was fine. There were actually no major 00:11:50.189 --> 00:11:52.449 data problems. 00:11:52.449 --> 00:11:53.959 And there were some applications that were built on 00:11:53.959 --> 00:11:57.490 top of this data service. And depending on who 00:11:57.490 --> 00:12:00.519 you talk to, they, they were more or less 00:12:00.519 --> 00:12:02.779 successful. Some people using that, these applications are like, 00:12:02.779 --> 00:12:05.740 oh, thank you, this did exactly what, what I 00:12:05.740 --> 00:12:07.269 need. It's actually helping me with my workflow. Some 00:12:07.269 --> 00:12:11.449 people are like, oh, OK. 00:12:11.449 --> 00:12:13.329 But engineering, you know what, this was really critical 00:12:13.329 --> 00:12:16.639 data. It was really hard. Totally new for us. 00:12:16.639 --> 00:12:19.910 This was a success. We did not break anything. 00:12:19.910 --> 00:12:26.410 Product is like nine months? Nine months. Eight engineers, 00:12:26.410 --> 00:12:29.709 nine months. How could you possibly call that a 00:12:29.709 --> 00:12:30.740 success? 00:12:30.740 --> 00:12:37.019 OK. So different application. Login flow. Depending on the 00:12:37.019 --> 00:12:39.509 time period, two engineers, four engineers, plus the dev 00:12:39.509 --> 00:12:46.410 ops team. Dev ops. Three months, you know. You 00:12:46.410 --> 00:12:50.399 know, we were figuring out, like, the system's automation 00:12:50.399 --> 00:12:55.209 was really tangled and in, in a few weeks 00:12:55.209 --> 00:12:59.790 we had some staging servers. And about two months 00:12:59.790 --> 00:13:02.980 later, someone comes up to me and says, hey, 00:13:02.980 --> 00:13:05.290 where, where, where are the servers gonna, gonna be? 00:13:05.290 --> 00:13:08.179 Can we get those? And I'm like, deep breath. 00:13:08.179 --> 00:13:12.269 Deep breath. Which servers? Oh, this, those staging servers. 00:13:12.269 --> 00:13:16.649 But it went, we worked it all out. 00:13:16.649 --> 00:13:19.029 Something that I learned about dev ops. Everyone actually 00:13:19.029 --> 00:13:22.959 needs to be invested and it's new to people. 00:13:22.959 --> 00:13:24.999 So, and it was released with a feature flag. 00:13:24.999 --> 00:13:27.499 Only a small amount of people were actually running 00:13:27.499 --> 00:13:31.059 through it. So it, we had some really good 00:13:31.059 --> 00:13:34.290 time to break it in production and really figure 00:13:34.290 --> 00:13:37.249 out how, how it was really gonna run. 00:13:37.249 --> 00:13:38.939 And we launched it to all of our users, 00:13:38.939 --> 00:13:41.929 and it was actually, I would say, very successful. 00:13:41.929 --> 00:13:46.529 It worked exactly intended. It was very resilient to 00:13:46.529 --> 00:13:49.389 failures. We had to do some unexpected database maintenance. 00:13:49.389 --> 00:13:52.230 Restart master databases and we're like, you know what, 00:13:52.230 --> 00:13:54.980 just do it. You know, just, just restart the 00:13:54.980 --> 00:13:56.629 database. No, no, no, no, no. Don't turn off 00:13:56.629 --> 00:13:59.369 the service. It's gonna recover. It's gonna be fine. 00:13:59.369 --> 00:14:02.269 Maybe like one user will notice. 00:14:02.269 --> 00:14:04.220 And the company's actually sort of figuring out that 00:14:04.220 --> 00:14:07.980 user metrics might be more important than dates as 00:14:07.980 --> 00:14:12.209 metrics, and the user metrics were generally successful. So 00:14:12.209 --> 00:14:15.040 engineering, like, this, this is good. This is good. 00:14:15.040 --> 00:14:17.329 This is, this is how we do this. This 00:14:17.329 --> 00:14:19.720 is, you know, the next project, this is how 00:14:19.720 --> 00:14:22.009 we're gonna, we're gonna have to figure this. 00:14:22.009 --> 00:14:25.199 And products is like three months? What are you 00:14:25.199 --> 00:14:28.300 talking about? Success? That's not success. We have these 00:14:28.300 --> 00:14:30.739 other product features that needed to be done. And 00:14:30.739 --> 00:14:32.300 we have these four engineers that are sitting in 00:14:32.300 --> 00:14:35.739 the corner, not able to do these things. 00:14:35.739 --> 00:14:41.629 OK. So I would say that this is a 00:14:41.629 --> 00:14:44.350 really important question to ask. What is, what is 00:14:44.350 --> 00:14:47.639 success? If engineering says it's a success, and product 00:14:47.639 --> 00:14:52.300 says it's a failure, who really wins? So, this 00:14:52.300 --> 00:14:55.639 is actually a trick question, because it's a zero-sum 00:14:55.639 --> 00:14:59.449 game. Like, nobody wins in this interaction. 00:14:59.449 --> 00:15:00.800 So let's go a little bit deeper to ask 00:15:00.800 --> 00:15:04.829 why we needed SOA in the first place. And, 00:15:04.829 --> 00:15:07.920 I, this is really complicated. There's lots of moving 00:15:07.920 --> 00:15:09.800 parts, but some of the things that, that I 00:15:09.800 --> 00:15:14.619 think now were because engineering didn't trust product, and 00:15:14.619 --> 00:15:16.670 we actually didn't, what that really means is we 00:15:16.670 --> 00:15:21.699 didn't trust that we could do our jobs well 00:15:21.699 --> 00:15:24.189 given the shortcuts that we needed to do to 00:15:24.189 --> 00:15:28.040 actually do our job and actually meet our metrics. 00:15:28.040 --> 00:15:30.439 And product didn't trust engineering to meet the products 00:15:30.439 --> 00:15:32.959 we were given, and we, we couldn't actually meet 00:15:32.959 --> 00:15:37.720 those promises. And, again, over time, this was changing, 00:15:37.720 --> 00:15:40.980 but product was accountable for features, not or quality, 00:15:40.980 --> 00:15:44.989 and, and the actual how users interacted with them. 00:15:44.989 --> 00:15:47.529 And this is probably the subject of much more 00:15:47.529 --> 00:15:49.730 discussion, and would love to continue this and, and 00:15:49.730 --> 00:15:51.429 learn more myself, but I think that a lot 00:15:51.429 --> 00:15:55.850 of this comes down to trust. If, if, if 00:15:55.850 --> 00:15:57.589 you can't trust that you can do your job 00:15:57.589 --> 00:15:59.689 successfully, how, how can you actually do your job 00:15:59.689 --> 00:16:02.040 successfully? If, if product, if different parts of the 00:16:02.040 --> 00:16:04.290 company don't trust each other to do their jobs 00:16:04.290 --> 00:16:07.249 well, how can they actually do their jobs well? 00:16:07.249 --> 00:16:10.689 So, what did, what did we learn? You know, 00:16:10.689 --> 00:16:13.540 fuck it, next time it's SOA from the beginning. 00:16:13.540 --> 00:16:15.689 Like, before we even do Rails new it's gonna 00:16:15.689 --> 00:16:19.220 be RabbitMQ. We're gonna have a cluster. It's gonna 00:16:19.220 --> 00:16:22.670 be amazing. So, no. No, no, no. Not, not, 00:16:22.670 --> 00:16:26.170 not the right answer. I almost went here, and 00:16:26.170 --> 00:16:29.959 thankfully I worked, then, with some very humble, empathetic 00:16:29.959 --> 00:16:32.649 people, who were also very convincing. No. 00:16:32.649 --> 00:16:36.649 So, I think what I've, what I have really 00:16:36.649 --> 00:16:39.480 taken from this, is Agile's not just one sprint 00:16:39.480 --> 00:16:42.110 after the next. It's like four miles is great. 00:16:42.110 --> 00:16:44.569 You just break it up into a hundred meter 00:16:44.569 --> 00:16:46.850 increments and you just, each one is a sprint. 00:16:46.850 --> 00:16:48.329 It's gonna be fine. 00:16:48.329 --> 00:16:52.470 What this really is about iterating. Deploying things, small 00:16:52.470 --> 00:16:54.209 things, really quickly, using the data that you get 00:16:54.209 --> 00:16:57.319 from that to figure out what to do next, 00:16:57.319 --> 00:17:00.670 and refactoring. Refactoring is really, really important. You might 00:17:00.670 --> 00:17:03.980 be able to do a small thing quickly, and 00:17:03.980 --> 00:17:07.990 with some shortcuts, not really needing to know how 00:17:07.990 --> 00:17:09.990 it needs to be designed. But when you see 00:17:09.990 --> 00:17:13.000 the pattern, you, you have to fix it. And 00:17:13.000 --> 00:17:16.670 SOA's not gonna solve larger organizational problems. 00:17:16.670 --> 00:17:19.980 It's not gonna fix your code. Really what it 00:17:19.980 --> 00:17:24.180 is is another tool at our disposal to solve 00:17:24.180 --> 00:17:27.459 our pain points, and our, and our organization's, our 00:17:27.459 --> 00:17:31.090 company's pain points. And how we do that is 00:17:31.090 --> 00:17:38.040 through iteration. So, small changes deployed, deployed quickly, using 00:17:38.040 --> 00:17:40.480 feature flags so that you can actually get code 00:17:40.480 --> 00:17:43.320 into production as soon as possible, knowing that it's 00:17:43.320 --> 00:17:46.390 off and it's not gonna affect users. And prioritizing 00:17:46.390 --> 00:17:48.490 this when it's necessary. 00:17:48.490 --> 00:17:51.630 So when might it be necessary? I would say 00:17:51.630 --> 00:17:53.370 performance. That's a really big thing. This is actually 00:17:53.370 --> 00:17:57.770 driving a lot of what we're doing. Code complexity 00:17:57.770 --> 00:17:59.740 might be less in a service than it would 00:17:59.740 --> 00:18:02.380 be outside of a service. So if you have 00:18:02.380 --> 00:18:03.800 these two things and you're like, this is actually 00:18:03.800 --> 00:18:05.490 gonna be easier to do over here than to 00:18:05.490 --> 00:18:07.810 put in this tangled mess, maybe, maybe it is 00:18:07.810 --> 00:18:08.340 better. 00:18:08.340 --> 00:18:10.960 May, and also maybe, sometimes you have a new 00:18:10.960 --> 00:18:14.220 feature and it is completely unrelated to everything else 00:18:14.220 --> 00:18:18.370 that you've already done. With the caveat that you, 00:18:18.370 --> 00:18:19.320 in the short term, you might be able to 00:18:19.320 --> 00:18:20.710 put it in with all the rest of that 00:18:20.710 --> 00:18:24.850 mess, trusting that when it becomes a problem you 00:18:24.850 --> 00:18:27.540 will be empowered to fix that problem. 00:18:27.540 --> 00:18:31.750 OK. So, performance. As I said, this is driving 00:18:31.750 --> 00:18:35.180 a lot of, of our work. So, Winelow is 00:18:35.180 --> 00:18:38.430 getting slower. We're running into some problems. We just 00:18:38.430 --> 00:18:41.010 got ring that the databases are becoming I/O bound 00:18:41.010 --> 00:18:42.890 on disk. This is a really, really bad place 00:18:42.890 --> 00:18:47.360 to be. But our user growth is increasing dramatically. 00:18:47.360 --> 00:18:50.850 The, the activity is increasing dramatically. We're starting to 00:18:50.850 --> 00:18:54.820 see exponential tendencies in our graphs, and if you 00:18:54.820 --> 00:18:59.350 see exponential tendencies in your user graphs, it's kind 00:18:59.350 --> 00:19:02.300 of like giving a talk to hundreds of people. 00:19:02.300 --> 00:19:06.210 You're like, why are you all looking at me?! 00:19:06.210 --> 00:19:09.900 It's, it can be scary. It can be really 00:19:09.900 --> 00:19:13.760 scary. But we have, so we have one table 00:19:13.760 --> 00:19:16.630 that's really outstripping the others, and that's causing problems. 00:19:16.630 --> 00:19:19.240 That's, it's, we discover, because of our data, because 00:19:19.240 --> 00:19:22.150 of our graphs, that it's really one table that's 00:19:22.150 --> 00:19:24.860 really destroying the performance of the rest of the 00:19:24.860 --> 00:19:28.230 table. And we're in the cloud, cause we have 00:19:28.230 --> 00:19:31.630 all those buzzwords. So there's really a maximum size 00:19:31.630 --> 00:19:34.010 of a host that we can get. So there's 00:19:34.010 --> 00:19:36.080 really, like, an upper limit of how much we 00:19:36.080 --> 00:19:38.550 can actually solve this with one database. 00:19:38.550 --> 00:19:42.630 Even after read-write splitting, we've already done that, and 00:19:42.630 --> 00:19:45.450 we, and, and pretty soon, we realized that the 00:19:45.450 --> 00:19:48.760 site is gonna break. If we don't do anything, 00:19:48.760 --> 00:19:51.250 we're just not gonna have a company anymore. It's 00:19:51.250 --> 00:19:55.830 just gonna, it's gonna fall down. But, we have 00:19:55.830 --> 00:20:00.080 really, really, really active committed users joining us right 00:20:00.080 --> 00:20:03.080 now. Now is not the time to stop feature-development. 00:20:03.080 --> 00:20:05.410 Now's the time to really learn from what our 00:20:05.410 --> 00:20:07.960 users are doing, double down on it, really tweak 00:20:07.960 --> 00:20:10.250 our features and figure out what is gonna make 00:20:10.250 --> 00:20:11.560 our business successful. 00:20:11.560 --> 00:20:14.040 And we only have ten engineers at this point. 00:20:14.040 --> 00:20:16.170 We don't really have that much, that many resources 00:20:16.170 --> 00:20:19.150 to, to, to work with this. 00:20:19.150 --> 00:20:24.670 So, our first step of iteration is realizing that 00:20:24.670 --> 00:20:26.710 this is one problem. How do we solve this 00:20:26.710 --> 00:20:29.530 one problem? And this is maybe going to be 00:20:29.530 --> 00:20:31.280 a service. How do we get to a point 00:20:31.280 --> 00:20:32.990 where it can be a service? So first step 00:20:32.990 --> 00:20:37.900 is isolating the data. So ActiveRecord gives you these 00:20:37.900 --> 00:20:42.290 really nice things. Associations, has_many, has_one. Things that really 00:20:42.290 --> 00:20:45.870 make the joins between your tables easier. 00:20:45.870 --> 00:20:48.290 When you have a service, you don't have joins. 00:20:48.290 --> 00:20:50.110 So these need to go away. These, these just 00:20:50.110 --> 00:20:52.430 don't exist. But it's actually really easy to get 00:20:52.430 --> 00:20:57.010 rid of these, honestly. A product has saves, you 00:20:57.010 --> 00:20:59.920 know. You save this product into a collection. You 00:20:59.920 --> 00:21:02.560 know, we could just take that where clause that, 00:21:02.560 --> 00:21:04.770 that ActiveRecord was gonna sort of do for us 00:21:04.770 --> 00:21:08.240 with, with a join, and just pull it out 00:21:08.240 --> 00:21:10.510 as a, as a where. 00:21:10.510 --> 00:21:14.190 Really not that hard, actually. And you know what, 00:21:14.190 --> 00:21:16.990 ActiveRecord also gives us ways of talking to a 00:21:16.990 --> 00:21:20.160 different database. But, you know, we can actually use 00:21:20.160 --> 00:21:22.510 this to just pretend it's in a different database. 00:21:22.510 --> 00:21:26.810 Establish connection allows you to have this model live 00:21:26.810 --> 00:21:28.880 in this database and all the rest of your 00:21:28.880 --> 00:21:31.060 stuff live in your main frames. It's really not 00:21:31.060 --> 00:21:32.310 that hard. 00:21:32.310 --> 00:21:34.080 And, one of the key things is that each 00:21:34.080 --> 00:21:37.390 step of this, each slight change, is deployable, and 00:21:37.390 --> 00:21:40.330 it's testable. And you can deploy it to staging. 00:21:40.330 --> 00:21:42.620 You can click around to see where, where your 00:21:42.620 --> 00:21:45.200 test coverage is maybe missing. Figure out where, where 00:21:45.200 --> 00:21:46.380 it breaks. And one thing is that, that I 00:21:46.380 --> 00:21:48.300 will say, is you might be doubling your database 00:21:48.300 --> 00:21:51.480 connections at this point. And when you database hits 00:21:51.480 --> 00:21:56.300 the max connections, just everything stops. It just stops 00:21:56.300 --> 00:21:57.160 working. 00:21:57.160 --> 00:22:04.160 So, we've learned this lesson the hard way. But 00:22:04.640 --> 00:22:07.060 now that we have this code deployed that pretends 00:22:07.060 --> 00:22:08.460 like it's in a different database, we can make 00:22:08.460 --> 00:22:11.970 it a different database. Without actually that much work. 00:22:11.970 --> 00:22:15.760 You have a Postgres - we love Postgres - 00:22:15.760 --> 00:22:17.750 we have a master database. And we spin up 00:22:17.750 --> 00:22:20.860 a replica over here, and we put the site 00:22:20.860 --> 00:22:27.620 into maintenance mode. If you have more critical interactions 00:22:27.620 --> 00:22:29.880 with your company that maintenance mode might, might not 00:22:29.880 --> 00:22:33.400 be possible, brain tree has some really great blogs 00:22:33.400 --> 00:22:37.540 and, I think, talks about this. But, for us, 00:22:37.540 --> 00:22:40.140 the operational overhead of making it so we can 00:22:40.140 --> 00:22:42.690 do this without the site being down was way 00:22:42.690 --> 00:22:44.000 more than it was worth. 00:22:44.000 --> 00:22:45.780 So we just take the site down. Push in 00:22:45.780 --> 00:22:48.200 new database dot yml, saying now this connection is 00:22:48.200 --> 00:22:50.310 talking to this database. And we just promote that 00:22:50.310 --> 00:22:52.960 to master and restart everything, bring the site back, 00:22:52.960 --> 00:22:57.510 and now we have two databases. Five minutes of 00:22:57.510 --> 00:23:02.350 downtime. Not, not that bad, actually. And you know, 00:23:02.350 --> 00:23:03.910 after the fact, you can clean up. You know, 00:23:03.910 --> 00:23:05.940 we have a redundant table over here. A bunch 00:23:05.940 --> 00:23:07.580 of redundant tables over here. You just truncate them 00:23:07.580 --> 00:23:11.590 and delete them. Very carefully. 00:23:11.590 --> 00:23:13.820 Not that hard. And actually, you know what, at 00:23:13.820 --> 00:23:16.290 this point, you might not need a service. You 00:23:16.290 --> 00:23:18.830 might be done. Your site might just be working. 00:23:18.830 --> 00:23:23.500 It's fine. For us, we, we knew, based on 00:23:23.500 --> 00:23:25.920 how things were going, that we were gonna have 00:23:25.920 --> 00:23:27.500 to horizontally shared this data. Now it's in a 00:23:27.500 --> 00:23:31.000 different database. That is gonna have to be many 00:23:31.000 --> 00:23:32.330 databases. 00:23:32.330 --> 00:23:34.130 And we want to hide that complexity. We don't, 00:23:34.130 --> 00:23:36.210 we do not want our main application to have 00:23:36.210 --> 00:23:40.470 any of that code. So, we know we're gonna 00:23:40.470 --> 00:23:43.110 have a service. Now isolate the interface. And now 00:23:43.110 --> 00:23:45.350 what this, by this, I mean, how are you 00:23:45.350 --> 00:23:48.150 actually accessing this data? And what is your long-term 00:23:48.150 --> 00:23:52.190 goal? You know, ours is sharding, and Wanelo is 00:23:52.190 --> 00:23:54.150 saves, and anytime we're actually getting saves, we're either 00:23:54.150 --> 00:23:55.960 looking at by product - a product has these 00:23:55.960 --> 00:23:57.430 saves - or we have it by user. A 00:23:57.430 --> 00:23:58.770 user has saved these things. 00:23:58.770 --> 00:24:01.610 So, this is actually really helpful, how to plan 00:24:01.610 --> 00:24:04.630 out what, your DSL, what your API is gonna 00:24:04.630 --> 00:24:07.080 look like, you know. We know that things are 00:24:07.080 --> 00:24:09.060 gonna have to get to it via product or 00:24:09.060 --> 00:24:12.880 via user. And so, you have some where clauses, 00:24:12.880 --> 00:24:15.780 you know. Where is ActiveRecord? We are not gonna 00:24:15.780 --> 00:24:16.680 have ActiveRecord. 00:24:16.680 --> 00:24:20.170 So, instead a save has a by_product method that 00:24:20.170 --> 00:24:23.340 is also. Oh, so one thing I will say 00:24:23.340 --> 00:24:26.310 is, at this point it's really helpful to remove 00:24:26.310 --> 00:24:28.840 redundancy. If you have different ways of accessing kind 00:24:28.840 --> 00:24:31.500 of the same data, do you really need that? 00:24:31.500 --> 00:24:33.200 You know. Can you change the product to mean 00:24:33.200 --> 00:24:35.150 that you don't actually have as many of these 00:24:35.150 --> 00:24:37.820 fingers as you have. And very soon, things are 00:24:37.820 --> 00:24:39.850 gonna break. So if you don't have tests, this 00:24:39.850 --> 00:24:44.200 is a really good place to add tests. 00:24:44.200 --> 00:24:49.470 So, now, we have a, a small sort of 00:24:49.470 --> 00:24:52.200 Ruby DSL. How do we actually pull that out? 00:24:52.200 --> 00:24:54.350 And I would say, right now, it actually doesn't 00:24:54.350 --> 00:24:56.110 need to be a service. Really what you need 00:24:56.110 --> 00:24:57.490 is a client. And how do you build the 00:24:57.490 --> 00:25:01.410 client out? And that's where adapters really come in. 00:25:01.410 --> 00:25:03.410 So, we use the module. This could be a 00:25:03.410 --> 00:25:05.190 base class. Some of the reasons why we thought 00:25:05.190 --> 00:25:07.080 we needed a module. Maybe we didn't Maybe we 00:25:07.080 --> 00:25:08.940 could have actually done this as, as a base 00:25:08.940 --> 00:25:12.630 class. But now, a save is a SaveClient. And 00:25:12.630 --> 00:25:16.710 that SaveClient is where your fingers go as, their 00:25:16.710 --> 00:25:19.920 class methods. And one thing I would point out 00:25:19.920 --> 00:25:23.860 is that that, that finder is calling through to 00:25:23.860 --> 00:25:27.090 an adapter. A database adapter. 00:25:27.090 --> 00:25:29.190 And really, that's what's wrapping up all your database 00:25:29.190 --> 00:25:32.960 calls. Hiding them from, from your application. And really 00:25:32.960 --> 00:25:34.910 one of the core pieces of this is that 00:25:34.910 --> 00:25:38.160 your database, your, your adapter is your feature flag. 00:25:38.160 --> 00:25:41.000 It's also deployable in place, you know. You, you 00:25:41.000 --> 00:25:42.950 can have this in your lib folder. You can 00:25:42.950 --> 00:25:44.710 actually start a gem now. And you can just 00:25:44.710 --> 00:25:47.720 deploy it and it's, your main application is still 00:25:47.720 --> 00:25:50.610 talking directly to this other database. But you're starting 00:25:50.610 --> 00:25:52.910 to build out your client. And later, when you 00:25:52.910 --> 00:25:54.350 have a service, you can replace it with a 00:25:54.350 --> 00:25:55.630 different adapter. 00:25:55.630 --> 00:25:58.940 So that adapter gives you back, when you, when 00:25:58.940 --> 00:26:02.750 you call, like, a, you know, you get save.by_product 00:26:02.750 --> 00:26:04.800 and you call all on it, that's actually, when 00:26:04.800 --> 00:26:11.080 you call by_product, you're getting back a relation. And 00:26:11.080 --> 00:26:13.930 this is something that we thought we didn't need, 00:26:13.930 --> 00:26:16.800 but turns out ActiveRecord does this for very good 00:26:16.800 --> 00:26:20.640 reason, and that's because threads and because state. 00:26:20.640 --> 00:26:23.370 If you say I want this type of data 00:26:23.370 --> 00:26:25.590 and you save it away to a variable, and 00:26:25.590 --> 00:26:28.640 now you call some other method, order.by_this, and it 00:26:28.640 --> 00:26:31.380 changes state, you might do something else on this 00:26:31.380 --> 00:26:34.340 variable over here, not realizing that you've altered its 00:26:34.340 --> 00:26:36.510 state later. So any time you make a change, 00:26:36.510 --> 00:26:38.130 a state change on these objects, you really want 00:26:38.130 --> 00:26:40.360 to get back a new instance. A different instance 00:26:40.360 --> 00:26:43.410 with different state. And when you call, you know, 00:26:43.410 --> 00:26:45.780 all, first, or pluck any of these things, what 00:26:45.780 --> 00:26:49.400 you're really calling it on is your relation instance. 00:26:49.400 --> 00:26:52.100 And the, the key thing that we learned is 00:26:52.100 --> 00:26:55.000 that that relation is sharable between all of your 00:26:55.000 --> 00:26:57.150 adapters. So the actual work done to get the 00:26:57.150 --> 00:27:00.990 data is on your adapters. So the relation delegates 00:27:00.990 --> 00:27:03.030 back. 00:27:03.030 --> 00:27:07.980 So, in our database adapter, the thing actually getting 00:27:07.980 --> 00:27:11.560 the data, is ActiveRecord. We've just moved our ActiveRecord 00:27:11.560 --> 00:27:14.080 class into the library and hidden it from our 00:27:14.080 --> 00:27:18.500 application. We, in this case, we were using ActiveRecord, 00:27:18.500 --> 00:27:20.970 so you could just, we'd do ActiveRecord. If you 00:27:20.970 --> 00:27:23.730 have another favorite database adapter, great. 00:27:23.730 --> 00:27:30.590 So you call Save.by_product. You, you get an adapter 00:27:30.590 --> 00:27:34.530 that, or so the, sorry, that calls through to 00:27:34.530 --> 00:27:38.010 the adapter and gives you back relation. You call 00:27:38.010 --> 00:27:40.860 Relation dot all and that just delegates back to 00:27:40.860 --> 00:27:44.760 the adapter, which calls through to an ActiveRecord, gets 00:27:44.760 --> 00:27:48.380 to your data, takes the attributes out of it 00:27:48.380 --> 00:27:50.260 and gives you back an instance of, of your 00:27:50.260 --> 00:27:52.460 core class. Because you're hiding ActiveRecord. You don't want 00:27:52.460 --> 00:27:55.250 to get back an ActiveRecord class. 00:27:55.250 --> 00:27:56.740 And I would say it's critical to deploy at 00:27:56.740 --> 00:28:00.200 this steps, because you've made mistakes. I guarantee you've 00:28:00.200 --> 00:28:02.570 made mistakes. And the cost of fixing this is 00:28:02.570 --> 00:28:06.050 really low right now, as opposed to spending a 00:28:06.050 --> 00:28:08.160 lot of time trying to design your server, how 00:28:08.160 --> 00:28:09.940 that's going to interact, and realize, whoa, whoa, whoa, 00:28:09.940 --> 00:28:11.750 whoa, we screwed up the client. All of this 00:28:11.750 --> 00:28:13.780 work we've done on the service, we have to 00:28:13.780 --> 00:28:17.470 through away because we did it wrong. 00:28:17.470 --> 00:28:20.940 Now, you have a client. Now you need a 00:28:20.940 --> 00:28:24.680 server. And it doesn't matter. What, whatever you want. 00:28:24.680 --> 00:28:27.210 It's fine. It's actually the, the cost of writing 00:28:27.210 --> 00:28:31.440 this is really, really low. Because the server is 00:28:31.440 --> 00:28:34.220 the simplest part of this, and if you did 00:28:34.220 --> 00:28:36.310 it wrong, if you chose the wrong transport method, 00:28:36.310 --> 00:28:38.720 mechanism, you know what? You build a new adapter. 00:28:38.720 --> 00:28:41.400 That's actually really quite easy. 00:28:41.400 --> 00:28:44.350 So let me take a moment to just reiterate, 00:28:44.350 --> 00:28:47.310 why should we have deployed by now? And I'd 00:28:47.310 --> 00:28:49.860 say it's because the client is much, much more 00:28:49.860 --> 00:28:52.610 complicated than the server, so your bugs are in 00:28:52.610 --> 00:28:54.510 the client. Your bugs are not in the server 00:28:54.510 --> 00:28:57.930 at this point. And the server is going to 00:28:57.930 --> 00:28:59.700 be dictated by the choices you've made in the 00:28:59.700 --> 00:29:02.970 client. So if you've made wrong choices and you 00:29:02.970 --> 00:29:07.090 build your server, you've built the wrong server. 00:29:07.090 --> 00:29:09.640 We used Sinatra and Oj, just cause it's awesome. 00:29:09.640 --> 00:29:12.150 It just, it really just works. It's small and 00:29:12.150 --> 00:29:15.710 it's useful. We thought we would need to move 00:29:15.710 --> 00:29:17.800 away from HTTP, but we've grown a lot and 00:29:17.800 --> 00:29:19.150 we haven't had to do this yet. It just 00:29:19.150 --> 00:29:21.720 works. Things that we thought we would have to 00:29:21.720 --> 00:29:24.610 change immediately, you know, it's almost a year later 00:29:24.610 --> 00:29:26.850 and it's just. 00:29:26.850 --> 00:29:30.380 OK, so now we use the service. And that's 00:29:30.380 --> 00:29:32.530 really a feature flag. You just write a new 00:29:32.530 --> 00:29:37.210 adapter that talks to the service instead of the 00:29:37.210 --> 00:29:39.900 database. So now you call, you know, by_product, you 00:29:39.900 --> 00:29:42.540 get a HTTP, it calls through to the HTTPAdapter 00:29:42.540 --> 00:29:45.050 which gets you back the same time of relation. 00:29:45.050 --> 00:29:47.430 When you call all on it, you know, it 00:29:47.430 --> 00:29:49.290 calls adapter dot all, which now goes to an 00:29:49.290 --> 00:29:52.300 HTTPClass that actually gets the JSON and takes the 00:29:52.300 --> 00:29:54.100 attributes out of it and gives you back your 00:29:54.100 --> 00:29:56.530 save class. You're getting back the same object you're 00:29:56.530 --> 00:29:59.090 getting back as save. 00:29:59.090 --> 00:30:03.330 So, retrospective. Great. We've isolated the data. We've isolated 00:30:03.330 --> 00:30:06.420 the interface. We started to build our DSL. We've 00:30:06.420 --> 00:30:09.760 pulled that DSL out into a gem. Now we've, 00:30:09.760 --> 00:30:11.150 now that we actually kind of understand what that 00:30:11.150 --> 00:30:13.330 gem needs to do, we can launch the service 00:30:13.330 --> 00:30:15.380 and then just build a new adapter to switch 00:30:15.380 --> 00:30:16.430 to this. 00:30:16.430 --> 00:30:18.880 If, I would say that if we hadn't, if 00:30:18.880 --> 00:30:20.550 we had realized that this was the order that 00:30:20.550 --> 00:30:21.950 we needed to do it on, we, we would 00:30:21.950 --> 00:30:25.330 have done this in two weeks. Instead. So that 00:30:25.330 --> 00:30:27.370 first part of it was like a day worth 00:30:27.370 --> 00:30:31.020 of work. That second part of it was like, 00:30:31.020 --> 00:30:34.470 three hours worth of work. And deployed immediately. 00:30:34.470 --> 00:30:37.630 The, the harder part was realizing that we needed 00:30:37.630 --> 00:30:39.710 an adapter. And at this point, people, we didn't 00:30:39.710 --> 00:30:42.360 really see anything about hexagonal architecture. This might have 00:30:42.360 --> 00:30:45.080 been before some of those talks and papers have 00:30:45.080 --> 00:30:49.040 been coming out. But, but it's actually really useful. 00:30:49.040 --> 00:30:52.980 Tests. We, we use Sunspot for some of our 00:30:52.980 --> 00:30:54.970 Solar things. And we're already used to spinning up 00:30:54.970 --> 00:30:58.880 a Solar instance using a gem, trap-it-up. You know 00:30:58.880 --> 00:31:03.150 you can do that for your integration tests. But 00:31:03.150 --> 00:31:05.450 for unit tests, you know what, we have tests 00:31:05.450 --> 00:31:08.410 around all of this. So we can have tests 00:31:08.410 --> 00:31:11.030 around a fake adapter that proves that it behaves 00:31:11.030 --> 00:31:14.170 the right way, and then that just saves, saves 00:31:14.170 --> 00:31:18.400 data in, in memory, in your application. 00:31:18.400 --> 00:31:21.640 And redundant tests, you know, you might say, eh, 00:31:21.640 --> 00:31:24.730 do I really need this test? Yes. Because one 00:31:24.730 --> 00:31:26.809 thing, you can delete your tests later that are 00:31:26.809 --> 00:31:29.940 redundant, and you want to be really confident that, 00:31:29.940 --> 00:31:34.840 that when you do switch over, it's gonna work. 00:31:34.840 --> 00:31:38.700 Foreman and Subcontractor are, are really helpful for this 00:31:38.700 --> 00:31:42.100 kind of thing. So subcontractor is a gem that 00:31:42.100 --> 00:31:45.650 really says, for this service, change directory over here 00:31:45.650 --> 00:31:48.670 and run this command in a completely isolated Bundler 00:31:48.670 --> 00:31:51.590 environment. Cause you really don't want to mix your, 00:31:51.590 --> 00:31:55.200 your dependencies. You don't want your server dependencies and 00:31:55.200 --> 00:31:58.340 the versions to be conflicting with your main application 00:31:58.340 --> 00:32:00.170 dependencies. You want to be able to, to change 00:32:00.170 --> 00:32:01.220 those separately. 00:32:01.220 --> 00:32:03.540 OK. So what about a new app? I'm spinning 00:32:03.540 --> 00:32:07.350 up a completely new thing. Not extracting. Totally new. 00:32:07.350 --> 00:32:09.390 How do I do that? How do I iterate 00:32:09.390 --> 00:32:12.220 on something that doesn't exist yet? And I would 00:32:12.220 --> 00:32:13.950 say that some of the lessons that, that we've 00:32:13.950 --> 00:32:16.070 learned from this, and actually from just doing our 00:32:16.070 --> 00:32:20.470 product development in general is iterate. Find a way 00:32:20.470 --> 00:32:23.430 to get to the smallest deployable thing as quickly 00:32:23.430 --> 00:32:27.590 as possible, and whatever tool you use to deploy, 00:32:27.590 --> 00:32:30.710 to spin up infrastructure, one of the sort of 00:32:30.710 --> 00:32:35.080 heuristics that, that, that we've found is really focus 00:32:35.080 --> 00:32:37.770 on organizing that code around being able to change 00:32:37.770 --> 00:32:41.710 things easily, and understand how this thing is different 00:32:41.710 --> 00:32:42.750 from this thing. 00:32:42.750 --> 00:32:45.490 You know, the Chef makes it really easy to 00:32:45.490 --> 00:32:48.760 define a giant hash of global state over here 00:32:48.760 --> 00:32:50.800 and then just run this thing over here and 00:32:50.800 --> 00:32:53.830 it's magic and it'll just do it. When you 00:32:53.830 --> 00:32:56.420 actually start to spin up different services, this thing 00:32:56.420 --> 00:32:58.940 is gonna need to be slightly different than this 00:32:58.940 --> 00:33:03.050 thing. So how can your code make that as 00:33:03.050 --> 00:33:05.970 easily understandable as possible? 00:33:05.970 --> 00:33:10.210 So feature flags, also, on or off. Do customers 00:33:10.210 --> 00:33:14.440 see this, or they don't. But you know what? 00:33:14.440 --> 00:33:18.860 Maybe it's just kind of half on. Maybe everything, 00:33:18.860 --> 00:33:22.670 every request that's gonna actually call through this, use 00:33:22.670 --> 00:33:26.950 Sidekiq to just every time, just spin off a 00:33:26.950 --> 00:33:30.400 job to hammer your, your service. And if it 00:33:30.400 --> 00:33:33.170 breaks, you've learned that before you've launched it to, 00:33:33.170 --> 00:33:35.430 to your users. There's a lot of other ways 00:33:35.430 --> 00:33:36.700 you can do this. 00:33:36.700 --> 00:33:39.450 On to these five years users, and let's see 00:33:39.450 --> 00:33:41.930 how we can break it, see how the interaction 00:33:41.930 --> 00:33:44.680 feels. It's, it's really useful. 00:33:44.680 --> 00:33:48.330 Also, one thing that we found is, it's often 00:33:48.330 --> 00:33:52.680 helpful to inter, integrate very deeply before you go 00:33:52.680 --> 00:33:56.240 widely. So, if you have a code-interaction that goes 00:33:56.240 --> 00:33:59.240 through all this process, do it kind of once 00:33:59.240 --> 00:34:01.970 through all the process, without worrying so much about 00:34:01.970 --> 00:34:04.630 the edge cases or the errors, because those are, 00:34:04.630 --> 00:34:06.230 you're gonna find those. Those are gonna, those are 00:34:06.230 --> 00:34:08.639 gonna come up. But it's re- really useful to 00:34:08.639 --> 00:34:11.069 kind of go all the way through the interaction 00:34:11.069 --> 00:34:13.050 knowing that it's really kind of sketch and doesn't 00:34:13.050 --> 00:34:17.760 do everything, and really let that drive the design. 00:34:17.760 --> 00:34:19.969 And this is also a thing of like letting 00:34:19.969 --> 00:34:23.250 the feature kind of drive the design and figuring 00:34:23.250 --> 00:34:25.310 out, what is the pattern for how you really 00:34:25.310 --> 00:34:28.020 need to organize the code, before you have. Don't 00:34:28.020 --> 00:34:29.739 just like whiteboard everything and say this is gonna 00:34:29.739 --> 00:34:32.340 be perfect. You know. 00:34:32.340 --> 00:34:34.590 Production is going to destroy every single design that 00:34:34.590 --> 00:34:39.619 you think you have. Also, if something seems clever, 00:34:39.619 --> 00:34:41.899 it's bad. If you're like, eh, this is kind 00:34:41.899 --> 00:34:44.139 of cool. No. No, no, no. It's bad. Complexity 00:34:44.139 --> 00:34:47.550 is gonna come to you. Don't, don't seek it 00:34:47.550 --> 00:34:51.239 out. It's, it's, it's evil. 00:34:51.239 --> 00:34:53.110 So if there are any kind of take aways 00:34:53.110 --> 00:34:57.360 from this, I would say you know, hexagonal architecture 00:34:57.360 --> 00:34:59.930 is really cool, but you don't have to design 00:34:59.930 --> 00:35:03.390 it from the start. If you have trust, if 00:35:03.390 --> 00:35:06.880 everyone in the organization has trust that you can 00:35:06.880 --> 00:35:09.110 do your job and that you're all going, working 00:35:09.110 --> 00:35:13.660 together to build an awesome product, an awesome company, 00:35:13.660 --> 00:35:16.400 you can fix this later. You can actually let 00:35:16.400 --> 00:35:20.780 the needs of your product determine where your boundaries 00:35:20.780 --> 00:35:21.830 are. 00:35:21.830 --> 00:35:27.119 So, thank you. I actually have a few minutes 00:35:27.119 --> 00:35:29.880 of, of questions, to answer questions.