1 00:00:16,640 --> 00:00:20,420 PETE HODGSON: OK. So, thanks for coming. 2 00:00:20,420 --> 00:00:25,800 Today I'm talking about Rails as an SOA client. 3 00:00:25,800 --> 00:00:30,020 So, in the beginning, there was a Rails application. 4 00:00:30,020 --> 00:00:32,710 And in the beginning, most of these Rails applications 5 00:00:32,710 --> 00:00:33,710 were pretty straight-forward. 6 00:00:33,710 --> 00:00:36,760 Pretty simple things. They were normally 7 00:00:36,760 --> 00:00:39,489 just talking to a single database. 8 00:00:39,489 --> 00:00:42,620 And this is how most of our Rails applications start. 9 00:00:42,620 --> 00:00:46,320 They're green fields, and we're talking to 10 00:00:46,320 --> 00:00:47,920 a single database. 11 00:00:47,920 --> 00:00:51,440 Quite often, we, or pretty soon, particularly now days, 12 00:00:51,440 --> 00:00:54,989 we start talking to services. These might be external 13 00:00:54,989 --> 00:00:59,109 services like Twillo or Twitter, or they might be 14 00:00:59,109 --> 00:01:01,129 internal services that we're using as part of doing 15 00:01:01,129 --> 00:01:04,769 our job in a, in a more enterprise-y situation. 16 00:01:04,769 --> 00:01:09,509 And as time has gone on, we've noticed, or 17 00:01:09,509 --> 00:01:12,039 I've noticed, that our Rails applications are depending more 18 00:01:12,039 --> 00:01:14,740 and more on services. So, I think there's kind 19 00:01:14,740 --> 00:01:18,030 of two big forces in play here. Force number 20 00:01:18,030 --> 00:01:21,840 one is, as time has grown and as Rails 21 00:01:21,840 --> 00:01:25,560 has matured and our community's matured, our Rails applications 22 00:01:25,560 --> 00:01:27,380 have been kind of growing into these big monoliths, 23 00:01:27,380 --> 00:01:31,310 mono Rails, these big, bloated applications, and now there's 24 00:01:31,310 --> 00:01:34,430 this kind of movement, quite a widespread movement, meant 25 00:01:34,430 --> 00:01:37,659 to break up our large, monolithic Rails apps into, 26 00:01:37,659 --> 00:01:40,369 into services and, obviously, something needs to talk to 27 00:01:40,369 --> 00:01:42,140 those services. And normally that is a Rails app 28 00:01:42,140 --> 00:01:44,060 fronting those services. 29 00:01:44,060 --> 00:01:47,909 The second big force is, Rails applications, which I 30 00:01:47,909 --> 00:01:49,920 think started off mainly being used by start ups, 31 00:01:49,920 --> 00:01:52,930 have moved more and more into enterprise-y places where 32 00:01:52,930 --> 00:01:56,969 there are a lot, lot, lot of services. And 33 00:01:56,969 --> 00:02:00,360 in these enterprise contexts, Rails apps are normally talking 34 00:02:00,360 --> 00:02:01,530 to a lot of services to get their job 35 00:02:01,530 --> 00:02:02,340 done. 36 00:02:02,340 --> 00:02:05,590 And this is, even goes to the extreme where 37 00:02:05,590 --> 00:02:07,030 we have Rails app that don't actually have any 38 00:02:07,030 --> 00:02:09,310 local data storage at all, are not doing any 39 00:02:09,310 --> 00:02:12,980 kind of persistence. All they're doing is, is interacting 40 00:02:12,980 --> 00:02:15,200 with services to do all of their work. 41 00:02:15,200 --> 00:02:19,310 So, I, I built an application like this fairly 42 00:02:19,310 --> 00:02:24,810 recently, and I'm gonna talk to you guys about 43 00:02:24,810 --> 00:02:27,120 what we built, some of the techniques we used 44 00:02:27,120 --> 00:02:29,610 to be successful in building these kind of services 45 00:02:29,610 --> 00:02:34,530 only application. Yup. So that's, that's gonna be the 46 00:02:34,530 --> 00:02:35,800 talk today. 47 00:02:35,800 --> 00:02:38,450 My name is Pete. I work for this consulting 48 00:02:38,450 --> 00:02:41,840 company called ThoughtWorks. So, we work with clients to 49 00:02:41,840 --> 00:02:44,950 help them build software and get better at building 50 00:02:44,950 --> 00:02:48,700 software. I've been with ThoughtWorks for about four years. 51 00:02:48,700 --> 00:02:50,069 As you can tell from my accent, I'm in 52 00:02:50,069 --> 00:02:53,290 the San Francisco office. 53 00:02:53,290 --> 00:02:55,250 I really am in the San Francisco office. And 54 00:02:55,250 --> 00:02:57,959 I, I, in my time in ThoughtWorks, I've done 55 00:02:57,959 --> 00:02:59,870 a lot of different things. I've done a fair 56 00:02:59,870 --> 00:03:01,989 amount of Ruby and Rails, but I've also done 57 00:03:01,989 --> 00:03:05,480 some Scala, some JavaScript, some, some iOS. And I've 58 00:03:05,480 --> 00:03:08,040 also worked in a lot of different contexts, organizations. 59 00:03:08,040 --> 00:03:10,569 So I've worked in really small start ups all 60 00:03:10,569 --> 00:03:13,349 the way through to huge, lumbering banks. 61 00:03:13,349 --> 00:03:15,629 And, one of the things that I love about 62 00:03:15,629 --> 00:03:17,349 ThoughtWorks is I get to move in all these 63 00:03:17,349 --> 00:03:19,150 different places. And one of the things that I 64 00:03:19,150 --> 00:03:23,319 think ThoughtWorks brings to its clients is ideas from 65 00:03:23,319 --> 00:03:26,790 one place applied in a different place. So, what 66 00:03:26,790 --> 00:03:28,489 we find ourselves doing a lot is taking ideas 67 00:03:28,489 --> 00:03:30,590 from the Rails community and introducing them to the 68 00:03:30,590 --> 00:03:33,709 Scala community. Or taking ideas from mobile applications and 69 00:03:33,709 --> 00:03:36,750 introducing them to our client-side JavaScript applications. 70 00:03:36,750 --> 00:03:39,510 And, indeed, taking ideas from start ups and taking 71 00:03:39,510 --> 00:03:42,980 them to the enterprise. And so, shockingly enough, vice 72 00:03:42,980 --> 00:03:44,599 versa. So, that's part of what I'm gonna be 73 00:03:44,599 --> 00:03:47,010 talking about today is, there's this subtext of kind 74 00:03:47,010 --> 00:03:48,849 of taking ideas from other communities that are successful 75 00:03:48,849 --> 00:03:53,860 and bringing them into the Rails community. 76 00:03:53,860 --> 00:03:55,209 And so this is gonna be a kind of 77 00:03:55,209 --> 00:03:56,760 a talk in two parts. The first part I'm 78 00:03:56,760 --> 00:03:59,209 gonna talk about some kind of more hand wave-y, 79 00:03:59,209 --> 00:04:02,860 how do we live in this ecosystem environment. And 80 00:04:02,860 --> 00:04:04,510 some tools and techniques that can help with that, 81 00:04:04,510 --> 00:04:06,959 particularly in the context of Rails. And then I'm 82 00:04:06,959 --> 00:04:09,599 gonna dive into some more kind of nitty gritty 83 00:04:09,599 --> 00:04:10,810 kind of, how do we actually build these things 84 00:04:10,810 --> 00:04:12,450 into our Rails applications? 85 00:04:12,450 --> 00:04:18,238 So, first off. Co-dependence versus independence. So, as I 86 00:04:18,238 --> 00:04:20,789 said, I worked for a client. We were building 87 00:04:20,789 --> 00:04:23,370 a large, we were building, actually, a pretty small 88 00:04:23,370 --> 00:04:26,039 Rails application in front of a very large set 89 00:04:26,039 --> 00:04:30,050 of services. We were building an online store for, 90 00:04:30,050 --> 00:04:32,580 for a large book retailer. They were going to 91 00:04:32,580 --> 00:04:34,099 a new market, and they didn't want to have 92 00:04:34,099 --> 00:04:37,700 to keep working in Java. So they were taking 93 00:04:37,700 --> 00:04:40,599 this as an opportunity to build a green-filled Rails 94 00:04:40,599 --> 00:04:43,719 app for a new market, which was still gonna 95 00:04:43,719 --> 00:04:46,130 be using all of their existing services, which were 96 00:04:46,130 --> 00:04:50,800 implemented in a variety of languages. 97 00:04:50,800 --> 00:04:52,320 So this is what the home page of this 98 00:04:52,320 --> 00:04:54,779 application kind of looked like. There was this kind 99 00:04:54,779 --> 00:04:57,719 of list of all the products and prices and 100 00:04:57,719 --> 00:04:59,550 descriptions and all that kind of stuff. And then 101 00:04:59,550 --> 00:05:01,089 across the top we had these kind of deals 102 00:05:01,089 --> 00:05:03,580 of the day. 103 00:05:03,580 --> 00:05:06,779 And the way that this worked was, was interesting 104 00:05:06,779 --> 00:05:09,409 or, I, I suppose is, is interesting if you 105 00:05:09,409 --> 00:05:11,479 haven't worked in like a large enterprise like this. 106 00:05:11,479 --> 00:05:14,570 But the information to power this page was coming 107 00:05:14,570 --> 00:05:17,229 from a lot of different places. And our Rails 108 00:05:17,229 --> 00:05:19,450 app was really just responsible for stitching that information 109 00:05:19,450 --> 00:05:21,909 together. So, we had the product service that was 110 00:05:21,909 --> 00:05:24,330 kind of serving up product catalog data, and then 111 00:05:24,330 --> 00:05:26,800 we had the deals service that was kind of 112 00:05:26,800 --> 00:05:28,050 telling us what the deals of the day were. 113 00:05:28,050 --> 00:05:29,349 And, as I said, the Rails app was just 114 00:05:29,349 --> 00:05:30,880 kind of there to kind of, go over here 115 00:05:30,880 --> 00:05:32,529 and get some information. Go over here and get 116 00:05:32,529 --> 00:05:33,820 some information, and then kind of plug it all 117 00:05:33,820 --> 00:05:38,169 together. So when you're building an app like this, 118 00:05:38,169 --> 00:05:42,839 you get this interesting phenomenon, where this app, on 119 00:05:42,839 --> 00:05:45,490 its own, really can't do much at all. In 120 00:05:45,490 --> 00:05:47,620 fact, it can't do anything. We, there wasn't, there 121 00:05:47,620 --> 00:05:51,000 was probably not a single page of this app 122 00:05:51,000 --> 00:05:53,839 that would render if those services weren't there. 123 00:05:53,839 --> 00:05:57,409 So, what, this is what I mean by co-dependence. 124 00:05:57,409 --> 00:06:01,310 This application was incredibly co-dependent on the ecosystem of 125 00:06:01,310 --> 00:06:05,320 services that it lived inside of. But ideally we 126 00:06:05,320 --> 00:06:07,870 don't want to be co-dependent, because we want to 127 00:06:07,870 --> 00:06:09,589 be able to run this thing in isolation, to 128 00:06:09,589 --> 00:06:13,180 test it in isolation, to debug it in isolation. 129 00:06:13,180 --> 00:06:15,779 So, we don't want to have to stand up 130 00:06:15,779 --> 00:06:19,640 the entire enterprise on our laptops just in order 131 00:06:19,640 --> 00:06:20,839 to, to run this thing when we're, when we're 132 00:06:20,839 --> 00:06:23,700 on a train or in a plane. So, we 133 00:06:23,700 --> 00:06:26,659 want independence, but we're in a situation of co-dependence. 134 00:06:26,659 --> 00:06:30,229 And this gets even more interesting when you start 135 00:06:30,229 --> 00:06:32,899 thinking about it in terms of teams. So, in 136 00:06:32,899 --> 00:06:35,839 our case, we had our team, let's call us 137 00:06:35,839 --> 00:06:39,839 the red team. We're building the Rails application. But 138 00:06:39,839 --> 00:06:42,580 we weren't building any of these services. The deals 139 00:06:42,580 --> 00:06:44,950 services might have been maintained by the green team 140 00:06:44,950 --> 00:06:47,159 and the blue team, of course, were working on 141 00:06:47,159 --> 00:06:48,830 the product service. 142 00:06:48,830 --> 00:06:53,080 And these teams were teams we sometimes, we didn't 143 00:06:53,080 --> 00:06:54,899 even know where they, where they were. We had 144 00:06:54,899 --> 00:06:56,520 vague ways of getting in contact with them, but 145 00:06:56,520 --> 00:06:58,959 we, we didn't have a very good relationship with 146 00:06:58,959 --> 00:07:03,659 them. And this leads to lots of interesting side-effects. 147 00:07:03,659 --> 00:07:06,070 So one thing that you might note here is 148 00:07:06,070 --> 00:07:08,640 the shape of the teams. Lines up pretty closely 149 00:07:08,640 --> 00:07:10,930 with the shape of the services. So this gets 150 00:07:10,930 --> 00:07:15,260 to this thing called Conway's Law. Don't tell DHH 151 00:07:15,260 --> 00:07:19,700 that I'm talking about laws. 152 00:07:19,700 --> 00:07:22,260 Conway's Law, coined in the 60s, says that the 153 00:07:22,260 --> 00:07:25,459 communication patterns of a software system tend to mirror 154 00:07:25,459 --> 00:07:28,039 the communication patterns of the people who build that 155 00:07:28,039 --> 00:07:31,719 system. How many people have heard of Conway's Law, 156 00:07:31,719 --> 00:07:34,140 by the way? 157 00:07:34,140 --> 00:07:38,180 So, I'm trying to avoid calling it a law 158 00:07:38,180 --> 00:07:40,250 but I'm not going to be able to do 159 00:07:40,250 --> 00:07:42,630 that. I think Conway's Law is the most important 160 00:07:42,630 --> 00:07:46,709 law for us to understand as software engineers. Oh, 161 00:07:46,709 --> 00:07:53,709 I said. As people. As software writers. Sorry. 162 00:07:55,909 --> 00:07:58,560 Conway's Law says that how people work together effects 163 00:07:58,560 --> 00:08:04,269 system architecture. And, and the flip-side. System architecture effects 164 00:08:04,269 --> 00:08:09,320 how people work together. And, this isn't kind of 165 00:08:09,320 --> 00:08:13,870 a, a possible outcome. This is reality. This is 166 00:08:13,870 --> 00:08:18,140 empirically shown for fifty years, sixty years almost. This 167 00:08:18,140 --> 00:08:19,769 does happen. This is true. 168 00:08:19,769 --> 00:08:21,890 So you can't, like, decide you don't want it 169 00:08:21,890 --> 00:08:23,450 to happen. But what you can do is you 170 00:08:23,450 --> 00:08:26,640 can either wield Conway's Law to your advantage, go 171 00:08:26,640 --> 00:08:29,279 with the grain and, and use it to your 172 00:08:29,279 --> 00:08:30,560 advantage, or you can kind of yield to Conway's 173 00:08:30,560 --> 00:08:32,120 Law and have it beat you up. 174 00:08:32,120 --> 00:08:34,190 So I'm gonna talk a little bit today about 175 00:08:34,190 --> 00:08:37,059 how we wielded Conway's Law to our advantage rather 176 00:08:37,059 --> 00:08:40,330 than have it beat us up. 177 00:08:40,330 --> 00:08:45,010 So, here's our three teams. One of the most 178 00:08:45,010 --> 00:08:48,770 fun aspects of, of working in a system that 179 00:08:48,770 --> 00:08:50,000 has more than one team is, is when you 180 00:08:50,000 --> 00:08:53,780 find a bug. So let's say we've got our 181 00:08:53,780 --> 00:08:55,930 deal, deals of the day section, and we've noticed 182 00:08:55,930 --> 00:08:59,330 that the prices are missing in some of the 183 00:08:59,330 --> 00:09:01,870 deals section when we run the application. So now 184 00:09:01,870 --> 00:09:04,250 we start to play this game, it's called Who's 185 00:09:04,250 --> 00:09:06,330 Bug is it Anyway? 186 00:09:06,330 --> 00:09:09,830 Hands up if you've played this game. Awesome. OK. 187 00:09:09,830 --> 00:09:12,080 I wasn't sure if that was a cultural reference 188 00:09:12,080 --> 00:09:17,200 that people wouldn't get. So, we know that the 189 00:09:17,200 --> 00:09:19,560 deals aren't loading correctly, but is that the deal 190 00:09:19,560 --> 00:09:22,750 service's fault? Maybe the product service isn't returning the 191 00:09:22,750 --> 00:09:25,690 right data for those deals. Maybe we're asking it 192 00:09:25,690 --> 00:09:27,560 the wrong questions. Or maybe we just have a 193 00:09:27,560 --> 00:09:32,140 bug in our rendering code. 194 00:09:32,140 --> 00:09:34,760 The fundamental issue of all of these things is 195 00:09:34,760 --> 00:09:40,560 this tension between co-dependence and independence. We are co-dependent. 196 00:09:40,560 --> 00:09:44,650 We want to be independent. Some people solve a 197 00:09:44,650 --> 00:09:48,640 lot of these problems by embracing co-dependence. So, if 198 00:09:48,640 --> 00:09:50,140 our Rails app needs the services to be up 199 00:09:50,140 --> 00:09:52,010 and running in order to run, we'll stand up 200 00:09:52,010 --> 00:09:54,830 the services. You can do that. You can stand 201 00:09:54,830 --> 00:09:57,210 up copies of the services locally on your laptop. 202 00:09:57,210 --> 00:09:59,940 This is more and more easy to do nowadays 203 00:09:59,940 --> 00:10:01,860 cause we've got kind of trendy dev-ops things like 204 00:10:01,860 --> 00:10:04,750 Vagrant and Chef and Puppet. 205 00:10:04,750 --> 00:10:06,720 That will only get you so far. Try standing 206 00:10:06,720 --> 00:10:10,390 up a bank on your laptop. It's not happening. 207 00:10:10,390 --> 00:10:12,950 Very, very common approach here is to, to use 208 00:10:12,950 --> 00:10:15,950 kind of shared services. So a shared dev environment. 209 00:10:15,950 --> 00:10:20,190 A shared kind of staging environment. That kind of 210 00:10:20,190 --> 00:10:22,280 thing. And that, that works quite well. But, for 211 00:10:22,280 --> 00:10:25,390 our team, we decided to go the alternate route. 212 00:10:25,390 --> 00:10:28,930 We wanted to, as much as possible, push for 213 00:10:28,930 --> 00:10:33,900 independence. And, the main way we did that was 214 00:10:33,900 --> 00:10:38,700 by rem- replacing our dependency on services with fake 215 00:10:38,700 --> 00:10:39,760 services. 216 00:10:39,760 --> 00:10:44,700 So, when we wanted to test our application, we 217 00:10:44,700 --> 00:10:47,030 wanted to test just our application in independence. And 218 00:10:47,030 --> 00:10:49,200 the way that we did that was, for each 219 00:10:49,200 --> 00:10:51,620 of our dependent services, we replaced those services with 220 00:10:51,620 --> 00:10:55,750 kind of a fake version of that service. And 221 00:10:55,750 --> 00:10:57,700 by doing that, we were allowed to run these 222 00:10:57,700 --> 00:11:01,810 things that, that we called bounded integration tests. 223 00:11:01,810 --> 00:11:05,750 So, these were tests that tested the entire stack 224 00:11:05,750 --> 00:11:08,990 of our application, of our Rails application, from top 225 00:11:08,990 --> 00:11:11,140 to bottom, from kind of html all the way 226 00:11:11,140 --> 00:11:16,420 down to the network. But, we weren't, we were 227 00:11:16,420 --> 00:11:19,930 testing that in isolation. We weren't actually hitting those 228 00:11:19,930 --> 00:11:23,960 shared services. 229 00:11:23,960 --> 00:11:26,310 And what's really interesting here is, do you see 230 00:11:26,310 --> 00:11:29,280 the, the boundary of those tests lines up almost 231 00:11:29,280 --> 00:11:31,850 perfectly with the boundary of our team. So this 232 00:11:31,850 --> 00:11:33,960 is a really good example of using Conway's Law 233 00:11:33,960 --> 00:11:37,880 to your advantage. Embracing Conway's Law and shaping your 234 00:11:37,880 --> 00:11:41,550 approach, shaping your software, with the shape of your 235 00:11:41,550 --> 00:11:43,610 team. And this was something that helped us win. 236 00:11:43,610 --> 00:11:46,570 So, if you want to build these fake services, 237 00:11:46,570 --> 00:11:49,280 what are your options? Comes down to, to two 238 00:11:49,280 --> 00:11:52,310 different kind of techniques, really. In-process and out-of-process. So 239 00:11:52,310 --> 00:11:55,200 in-process means you're kind of messing around with the 240 00:11:55,200 --> 00:11:57,870 actual Rails, running Rails process, and like replacing the 241 00:11:57,870 --> 00:12:00,690 network stack with a fake network stack maybe. 242 00:12:00,690 --> 00:12:02,930 Out of process means you're standing up a real 243 00:12:02,930 --> 00:12:05,380 http server. It's an actual, you know, you can 244 00:12:05,380 --> 00:12:08,520 actually talk to it over http. And your application, 245 00:12:08,520 --> 00:12:11,410 your Rails app, rather than pointing to a real 246 00:12:11,410 --> 00:12:13,210 deal service, it's going to point to this fake 247 00:12:13,210 --> 00:12:14,830 deal service. It's gonna look and smell like a 248 00:12:14,830 --> 00:12:16,910 deal service. You send it a request. It gives 249 00:12:16,910 --> 00:12:19,120 you a response. It's not a real deals service. 250 00:12:19,120 --> 00:12:21,830 It's a fake service that we have control over. 251 00:12:21,830 --> 00:12:24,100 So, you can imagine if we wanted to test 252 00:12:24,100 --> 00:12:26,810 unicode rendering, we could stand up a fake product 253 00:12:26,810 --> 00:12:30,090 service that returned a product tied tool that was 254 00:12:30,090 --> 00:12:32,210 all kind of snow man and that kind of 255 00:12:32,210 --> 00:12:35,350 stuff. And, and that allows us to test how 256 00:12:35,350 --> 00:12:38,490 we handle unicode titles without having, then, to need 257 00:12:38,490 --> 00:12:39,460 a unicode title. 258 00:12:39,460 --> 00:12:43,130 So, in process, a couple of popular options, vcr 259 00:12:43,130 --> 00:12:45,040 and webmock. We actually used vcr quite heavily on 260 00:12:45,040 --> 00:12:47,500 this project that I'm talking about. So basically it 261 00:12:47,500 --> 00:12:49,980 just kind of sits, it kind of injects itself 262 00:12:49,980 --> 00:12:53,279 in between your application and the network layer and 263 00:12:53,279 --> 00:12:55,779 records all the interactions between your application and the 264 00:12:55,779 --> 00:12:57,650 network, and then you can kind of flip it 265 00:12:57,650 --> 00:12:59,760 into playback mode, and rather than, next you time 266 00:12:59,760 --> 00:13:01,690 you go and make a, make a network call, 267 00:13:01,690 --> 00:13:03,320 it's not actually gonna call the network. It's going 268 00:13:03,320 --> 00:13:07,900 to just play back the previously recorded interaction. So 269 00:13:07,900 --> 00:13:12,650 that worked out well for us with some caveats. 270 00:13:12,650 --> 00:13:14,350 Out of process, you've got a lot more options, 271 00:13:14,350 --> 00:13:16,440 because you're not running out of process. You don't 272 00:13:16,440 --> 00:13:19,120 have to use Ruby. You can use whatever tool 273 00:13:19,120 --> 00:13:21,440 makes the most sense. So mimic is a really 274 00:13:21,440 --> 00:13:23,529 good option here. This is actually a Ruby gem 275 00:13:23,529 --> 00:13:27,880 written by Luke Redpath. It's a Sinatra application that 276 00:13:27,880 --> 00:13:30,950 pretends to be whatever you want it to be. 277 00:13:30,950 --> 00:13:32,900 Outside of the Ruby community, or outside of the 278 00:13:32,900 --> 00:13:37,060 Ruby language, I suppose, there's a library called moco. 279 00:13:37,060 --> 00:13:41,410 It's quite powerful. Stubby is another one. And then 280 00:13:41,410 --> 00:13:44,950 there's this interesting one called montebank. What makes montebank 281 00:13:44,950 --> 00:13:47,640 interesting is it doesn't just fake out http, it 282 00:13:47,640 --> 00:13:49,920 will fake out whatever protocol you want. So it 283 00:13:49,920 --> 00:13:51,529 can fake out smtp to it, so you can 284 00:13:51,529 --> 00:13:54,090 check emails. It'll fake out web sockets. That kind 285 00:13:54,090 --> 00:13:57,220 of stuff. So that's an interesting one to look 286 00:13:57,220 --> 00:13:59,020 at. 287 00:13:59,020 --> 00:14:01,570 So now we've got these bounded integration tests. And 288 00:14:01,570 --> 00:14:04,920 that means we have some confidence that it wasn't 289 00:14:04,920 --> 00:14:08,339 us that, that's creating this bug. Because our bounded 290 00:14:08,339 --> 00:14:10,660 integration tests are passing. So all was happy and 291 00:14:10,660 --> 00:14:13,020 we can go on with our day. 292 00:14:13,020 --> 00:14:15,740 Well, obviously not, because we actually still don't know 293 00:14:15,740 --> 00:14:18,250 where the bug is. And the goal here is 294 00:14:18,250 --> 00:14:20,600 not for us to kind of, prove that we, 295 00:14:20,600 --> 00:14:22,970 it's not our fault. The goal is to identify 296 00:14:22,970 --> 00:14:24,839 the problem and fix it, because at the end 297 00:14:24,839 --> 00:14:26,660 of the day it's a system. We're a part 298 00:14:26,660 --> 00:14:28,220 of that system and we need to fix the 299 00:14:28,220 --> 00:14:31,279 system and move on. 300 00:14:31,279 --> 00:14:33,420 So, so we're left saying, OK. We know it's 301 00:14:33,420 --> 00:14:35,730 not our fault. We'd like to help our, our 302 00:14:35,730 --> 00:14:40,720 comrades on other teams figure out where the problem 303 00:14:40,720 --> 00:14:43,410 is. And for that we used a technique called 304 00:14:43,410 --> 00:14:49,180 contract tests. So, contract tests, also sometimes referred to 305 00:14:49,180 --> 00:14:55,230 as consumer-driven contracts. The idea with these is we 306 00:14:55,230 --> 00:15:00,300 write test code that expresses what we expect a 307 00:15:00,300 --> 00:15:03,120 dependency, an external service to do, and then we 308 00:15:03,120 --> 00:15:05,050 run those tests against a real version of that 309 00:15:05,050 --> 00:15:07,290 service, and we find out if that service actually 310 00:15:07,290 --> 00:15:10,980 does what we expect. Quite simple. 311 00:15:10,980 --> 00:15:13,339 What's weird is we're testing someone else's code. We're 312 00:15:13,339 --> 00:15:16,540 not testing our code. We're, we're writing tests, but 313 00:15:16,540 --> 00:15:21,580 we're testing another team's services. So here's, again, our 314 00:15:21,580 --> 00:15:27,100 boundary, our bounded integration tests. And, so we verified 315 00:15:27,100 --> 00:15:29,380 that the way that the Rails app is, that 316 00:15:29,380 --> 00:15:30,920 the Rails app does the right thing when it 317 00:15:30,920 --> 00:15:34,480 talks to what we think these services are doing. 318 00:15:34,480 --> 00:15:37,110 Contract tests, once we add these, actually verify that 319 00:15:37,110 --> 00:15:39,020 what the service does is what we think it's 320 00:15:39,020 --> 00:15:39,750 gonna do. 321 00:15:39,750 --> 00:15:42,420 Cause it could be that there's a bug in 322 00:15:42,420 --> 00:15:44,730 the service, and these tests will find those bugs, 323 00:15:44,730 --> 00:15:46,279 hopefully, because we'll ask it to do something and 324 00:15:46,279 --> 00:15:48,460 it won't do what we expect. There could be 325 00:15:48,460 --> 00:15:51,290 a bug in our understanding of the service, and 326 00:15:51,290 --> 00:15:52,839 that's why these are called contract tests, because they 327 00:15:52,839 --> 00:15:55,220 define a contract between our team and the other 328 00:15:55,220 --> 00:15:59,660 team. Again, Conway's Law is coming in here. 329 00:15:59,660 --> 00:16:02,310 How many of you have worked with an external 330 00:16:02,310 --> 00:16:05,360 service where the, the Wiki page that documented the 331 00:16:05,360 --> 00:16:08,730 API was out of date? How many of you 332 00:16:08,730 --> 00:16:10,790 have ever worked on a service where it wasn't 333 00:16:10,790 --> 00:16:16,890 wrong? It's always wrong. And that's OK. Documentation tends 334 00:16:16,890 --> 00:16:19,060 to be stale. API documentation I think is never 335 00:16:19,060 --> 00:16:22,960 not stale. It just is born that way. 336 00:16:22,960 --> 00:16:25,310 These contract tests are a way to, to, to 337 00:16:25,310 --> 00:16:28,950 mitigate that. Because you're expressing your expectations in code 338 00:16:28,950 --> 00:16:32,520 rather than in words. 339 00:16:32,520 --> 00:16:35,860 So we, on, on my, on my team, we 340 00:16:35,860 --> 00:16:38,110 wrote contract tests for every single one of our 341 00:16:38,110 --> 00:16:41,230 dependencies. This was the most productive thing we did 342 00:16:41,230 --> 00:16:44,190 on this team is, in terms of improving efficiency 343 00:16:44,190 --> 00:16:49,460 of, of us creating software. It really, really helped 344 00:16:49,460 --> 00:16:51,490 us nail down, whenever there was an issue, where 345 00:16:51,490 --> 00:16:53,610 that issue was. And it really helped us communicate 346 00:16:53,610 --> 00:16:56,180 with, with our friends on other teams. 347 00:16:56,180 --> 00:16:58,560 So, if you want to do these, you've got 348 00:16:58,560 --> 00:17:00,500 a few different options for contract tests. We actually 349 00:17:00,500 --> 00:17:03,910 just did plain old RSpec. We just used standard 350 00:17:03,910 --> 00:17:07,099 RSpec test runner to make network calls to our 351 00:17:07,099 --> 00:17:09,000 dependencies and then look at the results and make 352 00:17:09,000 --> 00:17:11,299 sure that things looked the way that we expected 353 00:17:11,299 --> 00:17:12,480 them to look. 354 00:17:12,480 --> 00:17:15,409 There's two more kind of sophisticated options out there. 355 00:17:15,409 --> 00:17:17,539 There's a gem called pacto and another gem called 356 00:17:17,539 --> 00:17:20,539 pact. And these are kind of like real power 357 00:17:20,539 --> 00:17:24,589 tools. They're quite sophisticated. They're, they're very fully-featured. If 358 00:17:24,589 --> 00:17:26,339 you want to really kind of get into this, 359 00:17:26,339 --> 00:17:28,910 into this mind set. 360 00:17:28,910 --> 00:17:31,400 So once we had these bounded integration tests and 361 00:17:31,400 --> 00:17:34,410 these contract tests, we were left with this kind 362 00:17:34,410 --> 00:17:38,730 of CI dashboard that looked something like this. So 363 00:17:38,730 --> 00:17:40,540 whenever we checked in code, we'd run our unit 364 00:17:40,540 --> 00:17:42,970 tests, we'd run our functional tests. And assuming those 365 00:17:42,970 --> 00:17:45,140 passed, we'd run our kind of bounded integration tests 366 00:17:45,140 --> 00:17:46,330 up there at the top right. 367 00:17:46,330 --> 00:17:50,900 Assuming those passed, we'd then run our end-to-end tests. 368 00:17:50,900 --> 00:17:54,040 So these were running, testing as much of the 369 00:17:54,040 --> 00:17:56,929 stack as we could possibly stand up. So, our 370 00:17:56,929 --> 00:18:00,390 code, our team's code, maybe our team's dependency's code. 371 00:18:00,390 --> 00:18:02,520 So as much of the stack as possible. So 372 00:18:02,520 --> 00:18:04,690 really verifying, from a user's point of view, that 373 00:18:04,690 --> 00:18:08,510 this, this system worked as we expected. 374 00:18:08,510 --> 00:18:11,390 In theory, this was the picture. In reality it 375 00:18:11,390 --> 00:18:15,240 was almost never green. That was OK. So, this 376 00:18:15,240 --> 00:18:16,850 is what things looked like a lot of the 377 00:18:16,850 --> 00:18:21,110 time. Our tests were passing, the end-to-end tests were 378 00:18:21,110 --> 00:18:23,890 failing. If that was the end of the story, 379 00:18:23,890 --> 00:18:26,650 we'd be left constantly fighting fires of, why is 380 00:18:26,650 --> 00:18:28,590 it not working in product- or, why is it 381 00:18:28,590 --> 00:18:31,220 not working in staging? And is it the deals 382 00:18:31,220 --> 00:18:33,140 service or the product service, blah, blah, blah. 383 00:18:33,140 --> 00:18:35,240 But we had these contract tests. So now we 384 00:18:35,240 --> 00:18:37,860 can see here that service D's contract tests seem 385 00:18:37,860 --> 00:18:40,500 to be failing. So we've got some end-to-end tests 386 00:18:40,500 --> 00:18:44,210 that are failing. This contract is failing. That's where 387 00:18:44,210 --> 00:18:46,420 we can start looking for, for the cause of 388 00:18:46,420 --> 00:18:47,160 this problem. 389 00:18:47,160 --> 00:18:48,620 So this is what we spent a lot of 390 00:18:48,620 --> 00:18:49,910 our time, not a lot of our time doing, 391 00:18:49,910 --> 00:18:52,600 but this was a very frequent occurrence. The build 392 00:18:52,600 --> 00:18:54,429 would go red. We'd have a look at our 393 00:18:54,429 --> 00:18:58,420 dashboard. Oh, it's service D, again. Service D, you 394 00:18:58,420 --> 00:19:00,370 guys. 395 00:19:00,370 --> 00:19:03,309 So we go to our CI system and we 396 00:19:03,309 --> 00:19:06,860 would look at the logs. We'd get the log 397 00:19:06,860 --> 00:19:08,640 data that showed the request we were sending the 398 00:19:08,640 --> 00:19:11,590 response we, we got back. And we'd have a 399 00:19:11,590 --> 00:19:13,420 look and say, yup. It looks like they've broken 400 00:19:13,420 --> 00:19:15,170 it again. They forgot to turn on the database 401 00:19:15,170 --> 00:19:18,840 after doing a deployment, again. And we would write 402 00:19:18,840 --> 00:19:20,340 an email to them saying, hey, I think it's 403 00:19:20,340 --> 00:19:23,220 probably a bug in our code, but we've noticed 404 00:19:23,220 --> 00:19:24,860 when we send this request we get back this 405 00:19:24,860 --> 00:19:28,670 500 error from your service, I don't know what's 406 00:19:28,670 --> 00:19:29,950 going on there. 407 00:19:29,950 --> 00:19:32,640 You can tell I'm a consultant. 408 00:19:32,640 --> 00:19:37,000 And, things moved a lot smoother. It, it wasn't 409 00:19:37,000 --> 00:19:38,960 all kind of rainbows and unicorns, but it was 410 00:19:38,960 --> 00:19:40,280 a lot better than if these tests hadn't have 411 00:19:40,280 --> 00:19:42,730 been there. We would have still ended up telling 412 00:19:42,730 --> 00:19:43,880 them and they would have fixed it, but it 413 00:19:43,880 --> 00:19:46,000 would have sucked a lot more of our time. 414 00:19:46,000 --> 00:19:49,179 So that's all the hand wave-y, let's all work 415 00:19:49,179 --> 00:19:51,900 as teams stuff. Let's talk about how we're actually 416 00:19:51,900 --> 00:19:56,720 gonna build this, this inside of our team. 417 00:19:56,720 --> 00:19:57,800 So I want to talk about this idea called 418 00:19:57,800 --> 00:20:01,480 service gateways. The main thing I'm gonna talk about 419 00:20:01,480 --> 00:20:04,150 with service gateways is this gem called faraday. And 420 00:20:04,150 --> 00:20:06,250 in order to talk about this gem called faraday, 421 00:20:06,250 --> 00:20:08,010 I need to talk about this other gem called 422 00:20:08,010 --> 00:20:08,600 rack. 423 00:20:08,600 --> 00:20:10,820 So, how many of you know what rack is? 424 00:20:10,820 --> 00:20:13,410 Yeah, everyone knows what rack is. How many of 425 00:20:13,410 --> 00:20:15,420 you hate people that ask you raise your hands 426 00:20:15,420 --> 00:20:19,350 when you're in the audience? I have all the 427 00:20:19,350 --> 00:20:22,530 power. You guys have the power but don't realize 428 00:20:22,530 --> 00:20:23,540 that. 429 00:20:23,540 --> 00:20:27,660 So, rack is this awesome abstraction over http servers. 430 00:20:27,660 --> 00:20:31,990 And the main idea, the main thing that makes 431 00:20:31,990 --> 00:20:36,150 rack awesome is, by abstracting over the concept of 432 00:20:36,150 --> 00:20:38,380 a request and the concept of a response, we 433 00:20:38,380 --> 00:20:41,299 can kind of stack these middleware components in between 434 00:20:41,299 --> 00:20:45,929 our application, and the, the underlying http server. 435 00:20:45,929 --> 00:20:49,429 So, as a request comes in, from the outside 436 00:20:49,429 --> 00:20:55,470 world, it travels through this stack of middleware. So, 437 00:20:55,470 --> 00:20:58,299 each piece of middleware can, as this request is 438 00:20:58,299 --> 00:21:03,540 traveling through, has the opportunity to kind of modify, 439 00:21:03,540 --> 00:21:06,040 modify the request and kind of transform it in 440 00:21:06,040 --> 00:21:09,040 some way, add some information to the request, or 441 00:21:09,040 --> 00:21:11,059 to kind of have some side-effect. 442 00:21:11,059 --> 00:21:13,410 And then, a Rails app is gonna deal with 443 00:21:13,410 --> 00:21:16,090 that request and then, and then send back a 444 00:21:16,090 --> 00:21:18,120 response. And, again, as that response is traveling back 445 00:21:18,120 --> 00:21:23,670 up through the pipeline, through the, the middleware stacks, 446 00:21:23,670 --> 00:21:26,890 again, each of those pieces of middleware can modify 447 00:21:26,890 --> 00:21:29,150 that request in some way, can add some information 448 00:21:29,150 --> 00:21:32,320 to it or can have some side effect. 449 00:21:32,320 --> 00:21:35,890 Rails loves rack. So this is the, the kind 450 00:21:35,890 --> 00:21:40,960 of the stop stack of rack middlewares that comes 451 00:21:40,960 --> 00:21:42,970 when you, when you just do rails generate new 452 00:21:42,970 --> 00:21:46,580 project. So, yeah. Pretty, pretty popular. And the reason 453 00:21:46,580 --> 00:21:48,179 that this is popular is because the guys that 454 00:21:48,179 --> 00:21:50,210 built rack are smart and the guys that, guys 455 00:21:50,210 --> 00:21:52,230 and gals, excuse me, that built rack are smart 456 00:21:52,230 --> 00:21:55,980 and, likewise, Rails. And they realized that this is 457 00:21:55,980 --> 00:21:57,880 a very powerful abstraction and so they're really leveraging 458 00:21:57,880 --> 00:22:00,100 this power. 459 00:22:00,100 --> 00:22:02,150 So why was I talking about rack when I 460 00:22:02,150 --> 00:22:04,559 was supposed to be talking about faraday? Faraday is 461 00:22:04,559 --> 00:22:07,600 the exact same idea of rack, but applied for 462 00:22:07,600 --> 00:22:11,200 an http client. So, again, we have this abstraction 463 00:22:11,200 --> 00:22:13,720 over request and response. This time in the context 464 00:22:13,720 --> 00:22:16,170 of making a request and receiving a response rather 465 00:22:16,170 --> 00:22:19,179 than receiving a request and sending back a response. 466 00:22:19,179 --> 00:22:23,120 And, again, we can stack these pieces of middleware 467 00:22:23,120 --> 00:22:27,690 in between our application and the underlying network library. 468 00:22:27,690 --> 00:22:30,260 So, in this case, it's us making a request. 469 00:22:30,260 --> 00:22:32,809 So, as we send the request out to the 470 00:22:32,809 --> 00:22:36,380 network, again, that request travels through all of these 471 00:22:36,380 --> 00:22:38,420 middlewares, and these middlewares, again, have the opportunity to 472 00:22:38,420 --> 00:22:40,850 either modify that request in some way, have a 473 00:22:40,850 --> 00:22:43,720 side effect, add some information to the request. Eventually 474 00:22:43,720 --> 00:22:45,010 it gets on the wire. Eventually it gets to 475 00:22:45,010 --> 00:22:47,290 our dependency. Hopefully not service D because they'll probably 476 00:22:47,290 --> 00:22:50,260 go down again. 477 00:22:50,260 --> 00:22:53,640 And then service D hopefully sends a response, and 478 00:22:53,640 --> 00:22:57,160 again, as that response travels back through the stack, 479 00:22:57,160 --> 00:23:01,110 the stack is able to modify that, that response 480 00:23:01,110 --> 00:23:02,620 as it comes through, add some information. Have a 481 00:23:02,620 --> 00:23:04,690 side effect. 482 00:23:04,690 --> 00:23:07,990 So, how do we use this thing? So here's 483 00:23:07,990 --> 00:23:10,230 a, this is a service gateway. This is a 484 00:23:10,230 --> 00:23:14,360 class that represents a, a connection to the outside 485 00:23:14,360 --> 00:23:19,480 world. Here's us building a faraday middleware stack. So 486 00:23:19,480 --> 00:23:24,160 we're building a connection. We're saying hey, new faraday. 487 00:23:24,160 --> 00:23:26,400 Please use the instrumentation middleware and do some JSON 488 00:23:26,400 --> 00:23:30,840 stuff and follow redirects and do some logging. 489 00:23:30,840 --> 00:23:33,929 And that's it. We've now got a faraday connection. 490 00:23:33,929 --> 00:23:37,190 This is how we use a faraday connection. So, 491 00:23:37,190 --> 00:23:40,490 we say, hey, faraday connection, hey, http client, we 492 00:23:40,490 --> 00:23:43,160 want to get this url, or this path. And 493 00:23:43,160 --> 00:23:46,030 we get a response back. Pretty simple stuff. I 494 00:23:46,030 --> 00:23:47,950 would say this is readable code. DHH would be 495 00:23:47,950 --> 00:23:49,990 pleased with me. 496 00:23:49,990 --> 00:23:51,590 And what's really nice, the real power of, of 497 00:23:51,590 --> 00:23:54,420 faraday, is there's all this extra stuff that we 498 00:23:54,420 --> 00:23:58,840 set up, this middleware, this technical junk going on. 499 00:23:58,840 --> 00:24:01,330 But our, the rest of our code doesn't have 500 00:24:01,330 --> 00:24:03,140 to care about it. So we've kind of abstracted 501 00:24:03,140 --> 00:24:07,970 over all of that stuff of JSON, following redirects 502 00:24:07,970 --> 00:24:11,679 and logging and caching. 503 00:24:11,679 --> 00:24:13,470 So we used faraday a lot in these things 504 00:24:13,470 --> 00:24:19,250 that, that I was calling service gateways. So the, 505 00:24:19,250 --> 00:24:20,540 as I said, the really nice thing about things 506 00:24:20,540 --> 00:24:22,309 like rack and faraday is they allow us to 507 00:24:22,309 --> 00:24:28,559 segregate the boring tech-y stuff, http, logging, caching, instrumentation, 508 00:24:28,559 --> 00:24:30,530 the stuff that computer people talk about at conferences 509 00:24:30,530 --> 00:24:33,580 like this, from the business domain, which is the 510 00:24:33,580 --> 00:24:35,919 stuff that people actually care about. 511 00:24:35,919 --> 00:24:38,890 So, these service gateways acted as the place where 512 00:24:38,890 --> 00:24:43,419 we pushed all of the boring techy goop out 513 00:24:43,419 --> 00:24:45,960 to the boundaries of our system, so the core 514 00:24:45,960 --> 00:24:48,630 of our system, the core of our Rails application, 515 00:24:48,630 --> 00:24:53,630 wasn't talking about JSON requests, urls, caching, logging, deserialization. 516 00:24:53,630 --> 00:24:56,940 It was talking about books and products and prices. 517 00:24:56,940 --> 00:25:01,080 Things that our application really should be caring about. 518 00:25:01,080 --> 00:25:07,090 So this is this idea of hexagonal architectures. Push 519 00:25:07,090 --> 00:25:09,370 all of the goopy boring techy stuff out to 520 00:25:09,370 --> 00:25:11,169 the boundaries of your system, so that the core 521 00:25:11,169 --> 00:25:15,240 of your system can focus on the domain. The 522 00:25:15,240 --> 00:25:17,059 stuff that you really care about. The stuff that 523 00:25:17,059 --> 00:25:22,530 makes your system valuable to other people. 524 00:25:22,530 --> 00:25:24,350 So what kind of things did we put into 525 00:25:24,350 --> 00:25:26,290 these? What kind of stuff, boring techy stuff did 526 00:25:26,290 --> 00:25:30,610 we isolate in these service gateways? One thing that 527 00:25:30,610 --> 00:25:35,760 we did was we isolated serialization. So, serialization means 528 00:25:35,760 --> 00:25:39,710 JSON parsing or, in our case, XML parsing, cause 529 00:25:39,710 --> 00:25:44,190 we were in an enterprise. Yay. 530 00:25:44,190 --> 00:25:47,120 Comes down to the same thing actually. So, when 531 00:25:47,120 --> 00:25:49,320 we talk about parsing JSON or XML, we're actually 532 00:25:49,320 --> 00:25:52,020 talking about two different things. And I think often 533 00:25:52,020 --> 00:25:55,630 we conflate these two. So, step one, is we 534 00:25:55,630 --> 00:25:57,830 have this stream of bytes or this raw string, 535 00:25:57,830 --> 00:25:59,390 and we want to turn it into this kind 536 00:25:59,390 --> 00:26:02,919 of generic structure, like a hash of arrays of, 537 00:26:02,919 --> 00:26:04,929 of objects, right. So that's, that's something, you know. 538 00:26:04,929 --> 00:26:06,910 You do JSON dot parse or whatever and you 539 00:26:06,910 --> 00:26:09,390 get this kind of generic data structure out. 540 00:26:09,390 --> 00:26:13,760 But, if we are following the principles of hexagonal 541 00:26:13,760 --> 00:26:17,179 architecture, we don't want to deal with JSON data 542 00:26:17,179 --> 00:26:18,900 structures or generic data structures. We want to deal 543 00:26:18,900 --> 00:26:21,730 with products and books and deals. So the next 544 00:26:21,730 --> 00:26:23,950 step is to actually take that generic structure and 545 00:26:23,950 --> 00:26:26,520 map it into a domain object that we actually 546 00:26:26,520 --> 00:26:27,299 want to work with. 547 00:26:27,299 --> 00:26:30,240 Cause our goal here is to stop talking about 548 00:26:30,240 --> 00:26:32,580 things like JSON as quickly as possible, at the 549 00:26:32,580 --> 00:26:35,410 boundaries of our system, and start talking about products 550 00:26:35,410 --> 00:26:39,940 and prices and deals. So this is two steps. 551 00:26:39,940 --> 00:26:42,850 So, the other thing that you'll, I noticed working 552 00:26:42,850 --> 00:26:47,120 particularly in larger organizations, is these responses that you 553 00:26:47,120 --> 00:26:49,820 get back are huge. And really, often times, you 554 00:26:49,820 --> 00:26:52,340 only really care about a small subset of the 555 00:26:52,340 --> 00:26:54,470 response you get back. So you get back this 556 00:26:54,470 --> 00:26:56,750 big chunk of bytes. You turn it into this 557 00:26:56,750 --> 00:26:59,450 big generic structure, and then you actually just want 558 00:26:59,450 --> 00:27:00,730 to pluck three or four or five or six 559 00:27:00,730 --> 00:27:03,490 things out of that structure, that represent the product 560 00:27:03,490 --> 00:27:04,980 in your, in your domain. 561 00:27:04,980 --> 00:27:07,240 You don't care about the, the short description. You 562 00:27:07,240 --> 00:27:09,690 certainly don't care about the response time. Really all 563 00:27:09,690 --> 00:27:11,500 you care about is the, you know, the title 564 00:27:11,500 --> 00:27:13,760 or the author, you know, that kind of thing. 565 00:27:13,760 --> 00:27:16,570 So there's some gems that can help you make 566 00:27:16,570 --> 00:27:19,220 this easier. So, very popular one is this thing 567 00:27:19,220 --> 00:27:20,980 called hashie. Hashie will just make it a little 568 00:27:20,980 --> 00:27:22,630 bit easier for you to work with these generic 569 00:27:22,630 --> 00:27:24,470 data structures. But at the end of the day, 570 00:27:24,470 --> 00:27:27,090 they're still generic data structures. 571 00:27:27,090 --> 00:27:31,299 The second step that we took was to take 572 00:27:31,299 --> 00:27:32,880 all of that boring boiler plate that we were 573 00:27:32,880 --> 00:27:35,070 doing of saying, you know, get this dot that 574 00:27:35,070 --> 00:27:37,530 dot the other thing and get me the price. 575 00:27:37,530 --> 00:27:38,940 And then this dot that dot the other thing, 576 00:27:38,940 --> 00:27:41,650 get me the, the offer details. And turn that 577 00:27:41,650 --> 00:27:44,100 into a declarative statement that we could just make 578 00:27:44,100 --> 00:27:46,059 at the top of a class that would make 579 00:27:46,059 --> 00:27:46,250 these things. 580 00:27:46,250 --> 00:27:48,809 So we turn that mapping, that boring boilerplate mapping, 581 00:27:48,809 --> 00:27:52,100 into something that was declarative rather than imperative. We 582 00:27:52,100 --> 00:27:55,580 actually ended up extracting that little library that we 583 00:27:55,580 --> 00:27:58,590 built into this thing called lazy_doc. There's another couple 584 00:27:58,590 --> 00:28:00,970 of gems that are very, that have very similar 585 00:28:00,970 --> 00:28:04,450 goals. Embedded_doc is another one. And there's this really 586 00:28:04,450 --> 00:28:08,570 quite powerful one called representable, which is very good. 587 00:28:08,570 --> 00:28:10,809 It's, it's very powerful. It almost does too many 588 00:28:10,809 --> 00:28:13,179 things for my taste, but it's definitely, definitely a 589 00:28:13,179 --> 00:28:14,960 good option. 590 00:28:14,960 --> 00:28:17,120 The reason, by the way, embedded_doc is called embedded_doc 591 00:28:17,120 --> 00:28:19,230 is cause it's, what we're kind of talking about 592 00:28:19,230 --> 00:28:22,490 here is this embedded document pattern. That's the name 593 00:28:22,490 --> 00:28:24,350 of the pattern. Sorry, now I'm talking about patterns. 594 00:28:24,350 --> 00:28:28,169 I'm really off-message. 595 00:28:28,169 --> 00:28:29,549 So what else did we put in our faraday 596 00:28:29,549 --> 00:28:35,720 stack? Caching. This is the most, for me, the 597 00:28:35,720 --> 00:28:38,919 most exciting thing to talk about. So let me 598 00:28:38,919 --> 00:28:41,450 talk about it. So, back to our example, let's 599 00:28:41,450 --> 00:28:44,000 say we've got our Rails app. And we've got 600 00:28:44,000 --> 00:28:47,200 our product service and we've got this pricing service. 601 00:28:47,200 --> 00:28:48,640 Fun fact that I didn't know before I worked 602 00:28:48,640 --> 00:28:51,450 on this application. Prices can change, like, multiple times 603 00:28:51,450 --> 00:28:54,220 a minute in some of these systems. They're really 604 00:28:54,220 --> 00:28:56,840 trying to optimize for prices. 605 00:28:56,840 --> 00:29:01,820 So, can't really cache pricing information. Product information tends 606 00:29:01,820 --> 00:29:03,530 to be pretty static. The author of a book 607 00:29:03,530 --> 00:29:06,110 doesn't change very often. The title of a book 608 00:29:06,110 --> 00:29:09,650 doesn't change very often. But we're always looking up 609 00:29:09,650 --> 00:29:12,160 this product information, almost every single request. So let's 610 00:29:12,160 --> 00:29:13,850 cache that. Let's make this performant. 611 00:29:13,850 --> 00:29:16,679 So, a lot of you are automatically starting to 612 00:29:16,679 --> 00:29:18,910 think about how we would implement this and, well, 613 00:29:18,910 --> 00:29:20,730 we would have a product cache, and then when 614 00:29:20,730 --> 00:29:22,429 we need a product we'll go to the cache, 615 00:29:22,429 --> 00:29:24,750 and if it's there, then we'll check the freshness. 616 00:29:24,750 --> 00:29:27,160 And if the freshness is up to some configured 617 00:29:27,160 --> 00:29:28,890 thing, then we'll get it. Otherwise we'll go to 618 00:29:28,890 --> 00:29:30,280 the network. And then when we get back from 619 00:29:30,280 --> 00:29:31,950 network, put it in the cache, blah, blah, blah. 620 00:29:31,950 --> 00:29:34,559 All of this stuff, right. And we're, we're back 621 00:29:34,559 --> 00:29:37,090 to thinking about boring technical goop and we're not 622 00:29:37,090 --> 00:29:40,500 talking about our domain anymore. We didn't want to 623 00:29:40,500 --> 00:29:44,789 do this. I'm, honestly, bored with getting caching wrong 624 00:29:44,789 --> 00:29:46,690 over and over again in each of my applications. 625 00:29:46,690 --> 00:29:48,700 And so, we just pretended to be a web 626 00:29:48,700 --> 00:29:50,690 browser. 627 00:29:50,690 --> 00:29:53,559 As web developers, we know that images are very 628 00:29:53,559 --> 00:29:56,049 cacheable, and we should put caching headers on our, 629 00:29:56,049 --> 00:29:59,059 on our images, so that the browser can do 630 00:29:59,059 --> 00:30:00,450 the right thing and cache this so that every 631 00:30:00,450 --> 00:30:02,150 time we want to get this image, if it's 632 00:30:02,150 --> 00:30:04,200 already in the cache, we don't have to go 633 00:30:04,200 --> 00:30:09,200 onto the network. Kind of very common practice. 634 00:30:09,200 --> 00:30:12,820 We don't write JavaScript cache repositories and then go 635 00:30:12,820 --> 00:30:13,980 and check and see if the image is in 636 00:30:13,980 --> 00:30:16,059 there and then, if it's not, we'll go to 637 00:30:16,059 --> 00:30:17,230 the network and then put it in the cache 638 00:30:17,230 --> 00:30:18,320 repository. 639 00:30:18,320 --> 00:30:19,910 The web browser just does this for us. All 640 00:30:19,910 --> 00:30:21,990 we need to do is set the caching headers 641 00:30:21,990 --> 00:30:26,570 on this, this ping. So, my argument is, and 642 00:30:26,570 --> 00:30:28,990 what we did was, just apply the same principle 643 00:30:28,990 --> 00:30:33,000 to our API. So product dot, or product slash 644 00:30:33,000 --> 00:30:35,809 wherever this is at - my pretend url for, 645 00:30:35,809 --> 00:30:39,750 for this product. We had caching headers on this 646 00:30:39,750 --> 00:30:43,980 API call, that said this, this url is cacheable 647 00:30:43,980 --> 00:30:46,590 for twenty minutes. 648 00:30:46,590 --> 00:30:49,789 In our faraday stack, we had a caching middleware. 649 00:30:49,789 --> 00:30:52,880 So, the first time that we requested that url 650 00:30:52,880 --> 00:30:55,240 for that product, first time we wanted that product, 651 00:30:55,240 --> 00:30:57,140 our Rails app would say to our service gateway, 652 00:30:57,140 --> 00:30:58,809 I want this product. The service gateway would say, 653 00:30:58,809 --> 00:31:01,549 OK, it's at this url. I'm gonna make a 654 00:31:01,549 --> 00:31:05,210 request through my faraday stack for that url. 655 00:31:05,210 --> 00:31:08,320 That request would go through our stack, and the 656 00:31:08,320 --> 00:31:09,830 response would go through our response on the way 657 00:31:09,830 --> 00:31:11,309 back, and the caching layer would say, oh look 658 00:31:11,309 --> 00:31:13,820 at that. There's this caching header on this JSON 659 00:31:13,820 --> 00:31:15,440 that says it's gonna be valid for twenty minutes. 660 00:31:15,440 --> 00:31:17,450 Let me just put this response in my little 661 00:31:17,450 --> 00:31:19,400 cache on disk here. 662 00:31:19,400 --> 00:31:23,590 And, five minutes later, another user comes in and 663 00:31:23,590 --> 00:31:26,429 wants to, by chance, look at the same product. 664 00:31:26,429 --> 00:31:30,450 Again, our service gateway, our code doesn't do anything 665 00:31:30,450 --> 00:31:31,950 different. It just says, hey, service gateway, I want 666 00:31:31,950 --> 00:31:34,690 this product. Our service gateway doesn't do anything different. 667 00:31:34,690 --> 00:31:36,630 It just says to faraday, hey, I want to 668 00:31:36,630 --> 00:31:39,080 get this url. 669 00:31:39,080 --> 00:31:41,289 The request comes through to the caching layer. The 670 00:31:41,289 --> 00:31:44,049 caching layer says, oh look. That's the same url 671 00:31:44,049 --> 00:31:46,150 I just pulled five minutes, and the, the caching 672 00:31:46,150 --> 00:31:48,270 headers say it's valid for twenty minutes. I'll just 673 00:31:48,270 --> 00:31:49,940 return what I've got in my local cache. I 674 00:31:49,940 --> 00:31:52,789 won't go on the network. I won't bother with 675 00:31:52,789 --> 00:31:56,059 all that stuff. I'll just read it from the 676 00:31:56,059 --> 00:31:56,730 disc. 677 00:31:56,730 --> 00:31:59,159 This is what web browsers do all the time. 678 00:31:59,159 --> 00:32:01,440 We can do this all this time with, literally, 679 00:32:01,440 --> 00:32:04,530 a single line of code. Sorry. That's not a 680 00:32:04,530 --> 00:32:09,049 single line. It's a single method call. Would be 681 00:32:09,049 --> 00:32:10,340 a single line, I guess everything could be a 682 00:32:10,340 --> 00:32:14,110 single line of code, right. 683 00:32:14,110 --> 00:32:18,960 So, we've just said to our faraday stack, hey, 684 00:32:18,960 --> 00:32:21,030 add some caching middleware. Write it to disc at 685 00:32:21,030 --> 00:32:24,450 this location, and there we go. We, we've now 686 00:32:24,450 --> 00:32:27,000 got all the semantics of a web browser. And, 687 00:32:27,000 --> 00:32:28,750 we didn't have to do any work apart from 688 00:32:28,750 --> 00:32:33,460 this single method call. And what's really powerful is 689 00:32:33,460 --> 00:32:37,610 the team that knows the most about the data, 690 00:32:37,610 --> 00:32:40,289 so the product team in this case, can define 691 00:32:40,289 --> 00:32:43,299 the caching semantics of their data. 692 00:32:43,299 --> 00:32:45,950 So, if they realized, like, you know what, these, 693 00:32:45,950 --> 00:32:47,970 this, the name of the author keeps on changing. 694 00:32:47,970 --> 00:32:51,720 We need to drop down the caching freshness, they 695 00:32:51,720 --> 00:32:53,690 can just say, you know what, it's gonna be 696 00:32:53,690 --> 00:32:56,690 ten minutes. And we just get that semantics, we 697 00:32:56,690 --> 00:32:59,580 just start, we change our caching rules. No, no 698 00:32:59,580 --> 00:33:02,850 code change. No configuration change. No redeployment. It just 699 00:33:02,850 --> 00:33:03,539 happens. 700 00:33:03,539 --> 00:33:06,990 So, again, Conway's Law. The team that knows the 701 00:33:06,990 --> 00:33:10,309 most about the data has control over the caching 702 00:33:10,309 --> 00:33:15,580 of that data. Really, really, really powerful stuff. 703 00:33:15,580 --> 00:33:18,720 This goes back to this principle that, that I'm 704 00:33:18,720 --> 00:33:21,090 really passionate about. I think a lot of times 705 00:33:21,090 --> 00:33:22,559 we, we, we, we think of ourselves as building 706 00:33:22,559 --> 00:33:24,650 these systems on the web. We, we're building our 707 00:33:24,650 --> 00:33:28,240 stuff on top of http. I don't think we 708 00:33:28,240 --> 00:33:29,320 should be doing that. I think we should be 709 00:33:29,320 --> 00:33:32,130 building systems that are of the web. 710 00:33:32,130 --> 00:33:37,110 So, we should be using principles like, like REST, 711 00:33:37,110 --> 00:33:42,880 like hypermedia, like caching, http caching, because these, these 712 00:33:42,880 --> 00:33:46,400 are the principles that have made the most successful, 713 00:33:46,400 --> 00:33:49,539 biggest distributed system in the world work. It's really 714 00:33:49,539 --> 00:33:53,200 messy, and sometimes it's a bit of a pain 715 00:33:53,200 --> 00:33:56,080 to use, but it works really well, and it's 716 00:33:56,080 --> 00:33:58,850 all done for us already. We don't have to 717 00:33:58,850 --> 00:34:01,299 figure this stuff out ourselves. It's all out there 718 00:34:01,299 --> 00:34:03,700 and it's been working for decades. 719 00:34:03,700 --> 00:34:06,320 We use it, in some cases, hypermedia is something 720 00:34:06,320 --> 00:34:08,949 we're starting to understand more and more. We can 721 00:34:08,949 --> 00:34:10,829 use it in more places. Caching is, is an 722 00:34:10,829 --> 00:34:12,969 example of that. 723 00:34:12,969 --> 00:34:17,030 So, some principles. I could have talked about all 724 00:34:17,030 --> 00:34:18,690 this stuff for hours and hours and hours, but 725 00:34:18,690 --> 00:34:20,329 I don't have time to do that. So I'm 726 00:34:20,329 --> 00:34:22,359 just gonna shove a bunch of words at the 727 00:34:22,359 --> 00:34:26,210 end of this talk, and, and hope that, that 728 00:34:26,210 --> 00:34:27,679 if they're new words to you that you go 729 00:34:27,679 --> 00:34:29,929 out and, and research them. 730 00:34:29,929 --> 00:34:32,789 Conway's Law is, is the big one. This is 731 00:34:32,789 --> 00:34:36,429 something I'm really passionate about. If you, software, building 732 00:34:36,429 --> 00:34:38,690 software today is very much a team sport. If 733 00:34:38,690 --> 00:34:40,739 you don't understand how to work with other teams, 734 00:34:40,739 --> 00:34:42,549 then someone else is gonna do it better than 735 00:34:42,549 --> 00:34:45,139 you. So this is important stuff. 736 00:34:45,139 --> 00:34:47,909 Domain-Driven Design, I kind of touched on this talking 737 00:34:47,909 --> 00:34:51,559 about domain objects and talking about kind of bounded 738 00:34:51,559 --> 00:34:53,960 contexts. It's a really, really good book. It's really, 739 00:34:53,960 --> 00:34:57,660 really hard to read. Eric Evans is an incredibly 740 00:34:57,660 --> 00:35:00,200 smart guy who's not necessarily the best at kind 741 00:35:00,200 --> 00:35:02,969 of succinctly expressing ideas. 742 00:35:02,969 --> 00:35:04,769 If you do get this book, read the second 743 00:35:04,769 --> 00:35:07,529 half before you read the first half. He himself 744 00:35:07,529 --> 00:35:09,130 has said he got the order around wrong in 745 00:35:09,130 --> 00:35:11,170 retrospect. The, the second half has loads of really 746 00:35:11,170 --> 00:35:12,940 good stuff in it. The first half does too. 747 00:35:12,940 --> 00:35:14,930 But the second half, I think, is where the 748 00:35:14,930 --> 00:35:17,140 real, the real, real great stuff is. 749 00:35:17,140 --> 00:35:20,529 Postel's Law. I didn't really talk about this at 750 00:35:20,529 --> 00:35:24,359 all. But if you're building SOA systems, this is 751 00:35:24,359 --> 00:35:27,950 the way that you evolve these systems over time, 752 00:35:27,950 --> 00:35:31,089 without having to redeploy your entire enterprise at once. 753 00:35:31,089 --> 00:35:34,450 So, Postel's Law is very, very valuable. Another reason 754 00:35:34,450 --> 00:35:36,710 why the web has been so successful is Postel's 755 00:35:36,710 --> 00:35:41,349 Law. html is incredibly permissive. It draws, it causes 756 00:35:41,349 --> 00:35:42,960 a bunch of pain, but it also is the 757 00:35:42,960 --> 00:35:45,670 reason why html is the system that we still 758 00:35:45,670 --> 00:35:46,609 use today. 759 00:35:46,609 --> 00:35:50,150 Hexagonal architectures. Talked about this a fair bit. This 760 00:35:50,150 --> 00:35:52,160 is a really, really important principle. I'm really excited 761 00:35:52,160 --> 00:35:53,809 that the Rails community is starting to kind of 762 00:35:53,809 --> 00:35:55,779 get, to get excited about this. But it's been 763 00:35:55,779 --> 00:35:58,339 around for awhile. There's loads of really good talks 764 00:35:58,339 --> 00:36:01,130 out there about hexagonal architectures as they apply to 765 00:36:01,130 --> 00:36:02,099 Rails. There's been a couple of talks at the 766 00:36:02,099 --> 00:36:03,849 conference already. 767 00:36:03,849 --> 00:36:05,359 There's a, there's a Birds of a Feather session 768 00:36:05,359 --> 00:36:08,660 tonight about it, I believe. So there's loads of 769 00:36:08,660 --> 00:36:10,579 res- loads of good resources out there. Not just 770 00:36:10,579 --> 00:36:14,099 for Rails specifically, but just, hexagonal architectures in general. 771 00:36:14,099 --> 00:36:16,739 Because it is a general architectural principle. 772 00:36:16,739 --> 00:36:21,489 Finally, we should be embracing the web. It's been 773 00:36:21,489 --> 00:36:23,049 around for a long time. The patterns are very 774 00:36:23,049 --> 00:36:26,440 well understood, very well established. The tooling is amazingly 775 00:36:26,440 --> 00:36:30,160 powerful. We keep on reinventing wheels. Let's not keep 776 00:36:30,160 --> 00:36:32,880 reinventing this particular wheel. When it makes sense to 777 00:36:32,880 --> 00:36:34,680 make the web and the principles of the web, 778 00:36:34,680 --> 00:36:35,739 we should be doing that. 779 00:36:35,739 --> 00:36:37,099 Thank you.