WEBVTT 00:00:18.160 --> 00:00:24.640 RICHARD SCHNEEMAN: All right. OK. Hello everyone. 00:00:24.739 --> 00:00:25.930 AUDIENCE: Hello. 00:00:25.930 --> 00:00:29.180 R.S.: Thank you. Thank you. Welcome to, welcome, let 00:00:29.180 --> 00:00:31.220 me be the first to welcome you to RailsConf. 00:00:31.220 --> 00:00:35.809 So, our, our talk today is Heroku 2014: A 00:00:35.809 --> 00:00:38.390 Year in Review. It is gonna be a play 00:00:38.390 --> 00:00:43.610 in six acts, featuring Terrance Lee and Richard Schneeman. 00:00:43.610 --> 00:00:45.930 So, of course this is a year in review, 00:00:45.930 --> 00:00:49.059 and Heroku does measure their years by RailsConf. So 00:00:49.059 --> 00:00:53.809 this is from Portland to Chicago RailsConf year. The 00:00:53.809 --> 00:00:55.340 Standard RailsConf Year. 00:00:55.340 --> 00:00:57.769 As, as some of you might know, we are 00:00:57.769 --> 00:01:01.059 on the Ruby Task Force, and, in fact, that 00:01:01.059 --> 00:01:06.060 makes us Ruby Task Force members. And, of course, 00:01:06.060 --> 00:01:07.990 this was a big year. We're gonna be talking 00:01:07.990 --> 00:01:12.310 a little bit about app performance, some Heroku features, 00:01:12.310 --> 00:01:15.590 and community features. So, first up to the stage, 00:01:15.590 --> 00:01:19.020 I'm gonna be introducing the one, the only Mister 00:01:19.020 --> 00:01:23.030 Terrance Lee. You might have recognized him in some 00:01:23.030 --> 00:01:27.890 other roles. He hails from Austin, Texas, which has, 00:01:27.890 --> 00:01:31.800 undoubtedly, the best tacos in the entire world. So, 00:01:31.800 --> 00:01:32.489 he- 00:01:32.489 --> 00:01:33.860 AUDIENCE: [indecipherable] 00:01:33.860 --> 00:01:38.640 R.S.: Them's fightin' words, friend. So that he, he's 00:01:38.640 --> 00:01:42.360 also sometimes known as the Chief Taco Officer. Or, 00:01:42.360 --> 00:01:45.500 or the CTO. And something, something very interesting about 00:01:45.500 --> 00:01:48.940 Terrance is, recently, he was inducted into Ruby Core, 00:01:48.940 --> 00:01:52.340 so congratulations to, to Terrance. All right. 00:01:52.340 --> 00:01:56.090 So, without further ado, Act 1: Deploy Speed. 00:01:56.090 --> 00:02:02.479 TERRANCE LEE: Thank you, Richard. So, at the beginning 00:02:02.479 --> 00:02:05.720 of the year Rails Standard Year, we focused a 00:02:05.720 --> 00:02:08.060 lot on deployment speed. We got a lot of 00:02:08.060 --> 00:02:12.060 feedback and realized deployment was not as fast as 00:02:12.060 --> 00:02:14.629 it could be. And we wanted to make it 00:02:14.629 --> 00:02:16.250 faster. So, the first thing we set out to 00:02:16.250 --> 00:02:18.250 do was to actually do a bunch of measurement 00:02:18.250 --> 00:02:22.019 and profiling to look at where things were slow, 00:02:22.019 --> 00:02:24.160 and how we could make it better, and to 00:02:24.160 --> 00:02:27.590 kind of gage, like, the before and after and 00:02:27.590 --> 00:02:30.920 know when the good points were to kind of 00:02:30.920 --> 00:02:34.730 stop and move on to other things. Cause you 00:02:34.730 --> 00:02:37.140 can never make, you can never, you will never 00:02:37.140 --> 00:02:40.230 be done with, like, performance improvements. 00:02:40.230 --> 00:02:44.370 So, after about six months of work on this, 00:02:44.370 --> 00:02:46.930 we managed to cut down the deploy speeds for, 00:02:46.930 --> 00:02:50.780 across the platform for Ruby by about forty percent. 00:02:50.780 --> 00:02:54.220 So it's a pretty decent speed improvement. And, in 00:02:54.220 --> 00:02:56.130 order to do this, we mainly looked at three 00:02:56.130 --> 00:02:59.360 various ways to speed this up. 00:02:59.360 --> 00:03:02.640 The first thing was running code in parallel, so 00:03:02.640 --> 00:03:07.360 running more things, running things, like, more than one 00:03:07.360 --> 00:03:10.290 thing at one time. If you cache stuff you 00:03:10.290 --> 00:03:12.430 don't have to do it again, and, in general, 00:03:12.430 --> 00:03:14.580 just like, cutting out code that doesn't need to 00:03:14.580 --> 00:03:16.010 be there. 00:03:16.010 --> 00:03:19.360 So, with the parallel code, we worked with the 00:03:19.360 --> 00:03:23.930 bundler team on Bundler 1.5. There was a pull 00:03:23.930 --> 00:03:27.010 request sent by CookPad that was sent in to 00:03:27.010 --> 00:03:30.780 add parallel bundler install for Bundler 1.5. So if 00:03:30.780 --> 00:03:33.060 you actually aren't using this yet, I would recommend 00:03:33.060 --> 00:03:37.450 upgrading your bundle to at least Bundler 1.5. And 00:03:37.450 --> 00:03:41.140 the bundler added this dash j option, which allows 00:03:41.140 --> 00:03:44.520 you to specify the number of jobs to run. 00:03:44.520 --> 00:03:47.420 And this is, basically, if you're using MRI, it 00:03:47.420 --> 00:03:50.780 forks and does a num, these number of sub-processes, 00:03:50.780 --> 00:03:53.870 and if you're on JRuby or Rubinius it actually 00:03:53.870 --> 00:03:55.970 just uses threads here. 00:03:55.970 --> 00:03:58.440 And the benefit of doing this is, when you 00:03:58.440 --> 00:04:01.250 actually do bundle install, the dependencies that get installed 00:04:01.250 --> 00:04:03.830 get downloaded in parallel, so you're not waiting on 00:04:03.830 --> 00:04:07.150 network traffic sequentially anymore, and in addition you're also 00:04:07.150 --> 00:04:10.709 install gems in parallel. And this is mostly beneficial, 00:04:10.709 --> 00:04:14.500 especially when you're running native extensions. So if you 00:04:14.500 --> 00:04:16.500 have something like Nokogiri, that takes a long time. 00:04:16.500 --> 00:04:19.759 Oftentimes, you notice, you just like hang and wait 00:04:19.759 --> 00:04:21.228 for it to install and then it installs the 00:04:21.228 --> 00:04:24.130 next thing, so this allows you to install that, 00:04:24.130 --> 00:04:26.259 basically, in the background and then go and install 00:04:26.259 --> 00:04:28.919 other gems at the same time. 00:04:28.919 --> 00:04:34.610 Also, in Bundler 1.5, Richard actually added this function 00:04:34.610 --> 00:04:39.590 that allows people, allows bundler to auto-retry failed commands, 00:04:39.590 --> 00:04:43.659 so initially, before this, we would, when we run 00:04:43.659 --> 00:04:46.990 bundle install and something would fail because of some 00:04:46.990 --> 00:04:49.749 odd network timeout, like during one chance, you would 00:04:49.749 --> 00:04:51.939 have to basically repush again, no matter where you 00:04:51.939 --> 00:04:55.620 were in the build process. So, by default now, 00:04:55.620 --> 00:05:00.490 Bundler actually will retry clones and gem installs for 00:05:00.490 --> 00:05:02.090 up to three times by default. 00:05:02.090 --> 00:05:06.889 So, it will continue going during the deploy process. 00:05:06.889 --> 00:05:10.349 And so is anyone here actually familiar with the 00:05:10.349 --> 00:05:17.349 PIGz command? So, just Richard? So, PIGz is Parallel 00:05:19.089 --> 00:05:23.749 Gzip, and the build and packaging team at Heroku 00:05:23.749 --> 00:05:27.029 worked on this feature, or worked on implementing this 00:05:27.029 --> 00:05:31.029 at Heroku using the PIGz command, and in order 00:05:31.029 --> 00:05:32.969 to understand the kind of benefit of using something 00:05:32.969 --> 00:05:35.710 like this, when you push an app up on 00:05:35.710 --> 00:05:39.020 Heroku during the compile process, it actually builds these 00:05:39.020 --> 00:05:44.029 things at Heroku that are called slugs. And basically 00:05:44.029 --> 00:05:47.659 it's just, like, a tar of your app directory 00:05:47.659 --> 00:05:50.419 of everything after the compile face is run. 00:05:50.419 --> 00:05:54.599 And, originally, we were just using SquashFS, initially, and 00:05:54.599 --> 00:05:57.430 then we moved to kind of just tar files, 00:05:57.430 --> 00:06:00.400 and we noticed that one of the slowest point 00:06:00.400 --> 00:06:03.169 in the actual build process was actually just going 00:06:03.169 --> 00:06:06.419 through and compressing everything in that, that file directory, 00:06:06.419 --> 00:06:09.080 and then pushing it up onto S3 after that 00:06:09.080 --> 00:06:12.139 was done. And so one of the things that 00:06:12.139 --> 00:06:14.139 we looked into was, is there a way we 00:06:14.139 --> 00:06:16.809 can make that faster? So, if you ever push 00:06:16.809 --> 00:06:19.360 a Heroku app and then you basically, like, wait 00:06:19.360 --> 00:06:21.449 when it says, like, compressing and then it goes 00:06:21.449 --> 00:06:23.960 to done, like that's the compressing of the actual 00:06:23.960 --> 00:06:25.219 slug. 00:06:25.219 --> 00:06:29.199 And we managed to use slug, PIGz to now 00:06:29.199 --> 00:06:31.460 improve that by sniffing them out. I don't remember 00:06:31.460 --> 00:06:35.860 the actual performance improvement, but it was pretty significant, 00:06:35.860 --> 00:06:39.400 and the only downside was in certain slugs, the 00:06:39.400 --> 00:06:42.199 slug sizes are a little bit bigger. But the 00:06:42.199 --> 00:06:46.409 performance trade off was worth it at that time. 00:06:46.409 --> 00:06:48.330 The next thing we started doing was looking into 00:06:48.330 --> 00:06:54.150 caching. So anyone here using Rails 4? So pretty 00:06:54.150 --> 00:06:56.529 good amount of the room. So the one thing 00:06:56.529 --> 00:07:00.719 is that we did which differ from Rails 3, 00:07:00.719 --> 00:07:02.259 thanks to a bunch of the work that's happened 00:07:02.259 --> 00:07:04.419 on the Rails Core team with us is that, 00:07:04.419 --> 00:07:07.479 we can now cache assets between deploys. This wasn't 00:07:07.479 --> 00:07:10.779 possible in Rails 3 because the cache was, you 00:07:10.779 --> 00:07:13.289 couldn't actually reuse the cache. There was times when 00:07:13.289 --> 00:07:15.889 the cache would basically be corrupted and then you 00:07:15.889 --> 00:07:18.659 would get, like, assets that wouldn't work between deploys. 00:07:18.659 --> 00:07:20.300 So the fix there was you actually have to 00:07:20.300 --> 00:07:24.330 remove the assets between each deploy in some Rails 00:07:24.330 --> 00:07:27.099 3 builds. But it wasn't consistent, so sometimes it 00:07:27.099 --> 00:07:29.520 would work and sometimes it didn't. And on Heroku 00:07:29.520 --> 00:07:31.379 that's not something we can rely on in an 00:07:31.379 --> 00:07:32.369 automated fashion. 00:07:32.369 --> 00:07:34.569 But luckily a lot of that stuff has been 00:07:34.569 --> 00:07:37.399 fixed for Rails 4, so now we cache assets 00:07:37.399 --> 00:07:39.569 between deploys in Rails 4. And so if we 00:07:39.569 --> 00:07:43.639 look at Rails 3, I guess this got cut 00:07:43.639 --> 00:07:46.379 off, but this is supposed to say about, like, 00:07:46.379 --> 00:07:50.149 thirty-two seconds for a Rails 3 deploy, and then 00:07:50.149 --> 00:07:54.449 on Rails 4 it got, for the average we, 00:07:54.449 --> 00:07:56.680 we measured the steps in the, in the build 00:07:56.680 --> 00:07:59.949 process, and on Rails 4, the perk fifty was 00:07:59.949 --> 00:08:03.449 about fourteen point something seconds. So a pretty significant 00:08:03.449 --> 00:08:07.179 speed improvement there, both due to caching and other 00:08:07.179 --> 00:08:11.149 improvements inside of Rails 4 for the asset pipeline. 00:08:11.149 --> 00:08:12.550 So the other thing we also looked at was 00:08:12.550 --> 00:08:17.199 just, if there's code that is doing extra work, 00:08:17.199 --> 00:08:19.050 if we remove that, it will speed up the 00:08:19.050 --> 00:08:23.789 build process for everyone who's deploying every day. So 00:08:23.789 --> 00:08:25.179 one of the first things that we did was 00:08:25.179 --> 00:08:30.499 actually stop downloading bundler more than once. So, initially, 00:08:30.499 --> 00:08:33.409 when you, when we do the Ruby version detection, 00:08:33.409 --> 00:08:36.659 we actually have to download bundler, and then basically 00:08:36.659 --> 00:08:39.169 run that to get the version of Ruby to 00:08:39.169 --> 00:08:42.210 install on the application. And then again, we would 00:08:42.210 --> 00:08:45.000 then download and install it again because it was 00:08:45.000 --> 00:08:49.779 run in a separate process for the actual, like, 00:08:49.779 --> 00:08:52.279 installing of your dependencies. And one of the things 00:08:52.279 --> 00:08:55.100 we did was to actually just stop doing that 00:08:55.100 --> 00:08:57.649 and we would cache the Bundler gem so we 00:08:57.649 --> 00:09:00.319 don't have to download that two or three times 00:09:00.319 --> 00:09:03.300 during the build process. So, so cutting network IO 00:09:03.300 --> 00:09:06.060 and other things. 00:09:06.060 --> 00:09:09.139 We also started removing, there was like duplicate checks 00:09:09.139 --> 00:09:11.250 between detection of what kind of app you were 00:09:11.250 --> 00:09:14.610 using so, and, bin detects. We would use it 00:09:14.610 --> 00:09:17.060 to figure out what kind of app you have, 00:09:17.060 --> 00:09:18.480 like if it was a Ruby app, a Rack 00:09:18.480 --> 00:09:21.069 app, a Rails 3 app, a Rails 4 app, 00:09:21.069 --> 00:09:24.130 stuff like that. And then, again, since it was 00:09:24.130 --> 00:09:26.190 a separate process in bin compile, we would have 00:09:26.190 --> 00:09:29.740 to do it again. So, Richard actually did a 00:09:29.740 --> 00:09:32.829 bunch of work to refactor both detect and release, 00:09:32.829 --> 00:09:35.589 and so now detect is super simple. It literally 00:09:35.589 --> 00:09:39.029 just checks if you have the gemfile file there, 00:09:39.029 --> 00:09:40.660 and then all the other work is now deferred 00:09:40.660 --> 00:09:43.279 to bin compile. So that means we're only doing 00:09:43.279 --> 00:09:45.589 a bunch of these checks once, like examine your 00:09:45.589 --> 00:09:49.569 jump file, checking what gems you have. So not 00:09:49.569 --> 00:09:52.790 doing that two or more times. 00:09:52.790 --> 00:09:57.230 And, if you haven't watched this talk, he gave 00:09:57.230 --> 00:09:58.930 this talk at Ancient City Ruby. I don't actually 00:09:58.930 --> 00:10:03.170 know if the videos are quite up yet. But 00:10:03.170 --> 00:10:05.110 Richard does a talk about testing the untestable, so 00:10:05.110 --> 00:10:07.230 if you're interested in learning how we test the 00:10:07.230 --> 00:10:11.130 build pack, you should go watch this talk. 00:10:11.130 --> 00:10:15.149 So I'd like to introduce Richard, cause he's gonna 00:10:15.149 --> 00:10:18.790 present on the next section. So Richard loves Ruby 00:10:18.790 --> 00:10:21.829 so much that he got married to her. I 00:10:21.829 --> 00:10:23.170 think he got married last, last year. 00:10:23.170 --> 00:10:24.230 R.S.: Right before our last RailsConf. 00:10:24.230 --> 00:10:28.300 T.L.: Yeah. Right before our last RailsConf. I remember 00:10:28.300 --> 00:10:31.600 that. He's also on the Rails Issue Team, and 00:10:31.600 --> 00:10:35.220 he's one of the top one hundred Rails contributors, 00:10:35.220 --> 00:10:40.149 according to the Rails contributor sites. And you might 00:10:40.149 --> 00:10:43.980 also know him for his, this gem called sextant 00:10:43.980 --> 00:10:48.759 that he released for Rails 3. Basically, I remember 00:10:48.759 --> 00:10:50.940 back in the day, developing Rails apps, when I 00:10:50.940 --> 00:10:52.850 wanted to basically verify routes, I would run the 00:10:52.850 --> 00:10:55.129 rake routes command, and it would, you know, boot 00:10:55.129 --> 00:10:56.430 up the Rails environment and you'd have to wait 00:10:56.430 --> 00:10:58.360 a few seconds and then they would print out 00:10:58.360 --> 00:11:00.360 all the routes. And then if you wanted to, 00:11:00.360 --> 00:11:03.269 like, rerun it using grep, you would keep running 00:11:03.269 --> 00:11:04.389 it again. 00:11:04.389 --> 00:11:07.029 So, a lot of us, when we're doing development, 00:11:07.029 --> 00:11:10.639 already have, like, Rails running in a server while 00:11:10.639 --> 00:11:14.079 we're testing things and whatnot. And so what sextant 00:11:14.079 --> 00:11:17.750 does is it allows it, supports basically looking at 00:11:17.750 --> 00:11:19.940 the routes that are already in memory and just 00:11:19.940 --> 00:11:22.440 allowing you to query against the programmatically, and then 00:11:22.440 --> 00:11:24.709 that has a view for doing this. And this 00:11:24.709 --> 00:11:27.959 was also just merged into Rails 4. So if 00:11:27.959 --> 00:11:29.740 you're using Rails 4 or higher, you actually don't 00:11:29.740 --> 00:11:33.569 need to sextant gem and it's now built in. 00:11:33.569 --> 00:11:36.190 Richard and I both live in Austin, and so 00:11:36.190 --> 00:11:38.190 when people come visit, or actually when I'm in 00:11:38.190 --> 00:11:41.230 town, which isn't often, we have Ruby meet ups 00:11:41.230 --> 00:11:44.220 at Franklin's Barbecue, so if you guys are ever 00:11:44.220 --> 00:11:46.490 in town, let us know and we'd be more 00:11:46.490 --> 00:11:49.620 than happy to take you to a meet up. 00:11:49.620 --> 00:11:54.420 R.S.: All right. So the, for the first part 00:11:54.420 --> 00:11:56.569 of this, this act, we're gonna be talking about 00:11:56.569 --> 00:11:58.389 app speed, but before we talk about app speed, 00:11:58.389 --> 00:12:03.459 we're actually gonna talk about dimensions. So, the, the 00:12:03.459 --> 00:12:09.269 document dimensions are, let me see. Here we go. 00:12:09.269 --> 00:12:12.189 Were originally written in wide-screen, but the screens here 00:12:12.189 --> 00:12:13.230 are standard. 00:12:13.230 --> 00:12:17.730 There we go. So. You're actually gonna get to 00:12:17.730 --> 00:12:20.019 see all of the slides, as opposed to just 00:12:20.019 --> 00:12:22.139 having some of them cut off. So, OK. 00:12:22.139 --> 00:12:24.779 On, on app speed. The first thing I want 00:12:24.779 --> 00:12:27.750 to talk about is, is tail latencies. Is anybody 00:12:27.750 --> 00:12:31.199 familiar with tail latencies? OK. The guys in the 00:12:31.199 --> 00:12:34.269 Heroku t-shirts and somebody else. 00:12:34.269 --> 00:12:38.699 OK. So this is, this is a normalized distribution. 00:12:38.699 --> 00:12:41.990 WE have, on one side, the number of requests. 00:12:41.990 --> 00:12:43.529 On the, on the other side, we have the 00:12:43.529 --> 00:12:46.329 time to respond. So the further out you go, 00:12:46.329 --> 00:12:49.069 the slower it's gonna be. And we can, we 00:12:49.069 --> 00:12:50.990 can see this is the distribution of our requests. 00:12:50.990 --> 00:12:54.550 So over here, it's super fast. Like you love 00:12:54.550 --> 00:12:57.120 to be that customer. You're super happy. 00:12:57.120 --> 00:12:59.769 Over here, we have a super slow request, and 00:12:59.769 --> 00:13:01.160 you don't want to be that customer and you're 00:13:01.160 --> 00:13:05.259 pretty unhappy. So right in the middle is our 00:13:05.259 --> 00:13:09.060 average, and I'm sure they talked a ton about 00:13:09.060 --> 00:13:11.670 why the average is really misleading in the, in 00:13:11.670 --> 00:13:14.720 the last session with Skylight.IO. But, but we're basically 00:13:14.720 --> 00:13:17.600 saying that, roughly fifty percent of your, of your 00:13:17.600 --> 00:13:20.139 customers, fifty percent of your traffic, is going to 00:13:20.139 --> 00:13:24.120 get a response time at this or, or lower. 00:13:24.120 --> 00:13:26.399 So, like, this is, this is pretty decent. We 00:13:26.399 --> 00:13:27.750 can say, like, fifty percent of the people who 00:13:27.750 --> 00:13:29.899 come to our web site get a response before 00:13:29.899 --> 00:13:34.649 then. Moving up the distribution, to something like perk 00:13:34.649 --> 00:13:37.360 ninety-five, we say ninety-five percent of everyone who visits 00:13:37.360 --> 00:13:40.500 our traffic will get a response by now. So 00:13:40.500 --> 00:13:42.339 I'm gonna be using those terms, perk fifty, perk 00:13:42.339 --> 00:13:46.060 ninety-five, that refers to the percentage of, of requests 00:13:46.060 --> 00:13:48.649 that come in that we can respond by. 00:13:48.649 --> 00:13:50.939 So this is kind of theorized. This is an 00:13:50.939 --> 00:13:56.560 actual application. I, and one thing that you'll notice 00:13:56.560 --> 00:13:59.399 is that it's not perfectly normalized. Like, it's not, 00:13:59.399 --> 00:14:01.800 like both sides are not symmetrical. We kind of 00:14:01.800 --> 00:14:04.310 like, steeply shoot up, and then we have this 00:14:04.310 --> 00:14:07.730 really, really long tail, and, and this is kind 00:14:07.730 --> 00:14:09.579 of the, what I'm referring to when I'm saying 00:14:09.579 --> 00:14:10.639 tail latencies. 00:14:10.639 --> 00:14:13.949 So, yes, somebody actually might have gotten a response 00:14:13.949 --> 00:14:17.269 in zero milliseconds. You know, I doubt it, but 00:14:17.269 --> 00:14:20.269 somebody for sure did get a response in 3000 00:14:20.269 --> 00:14:23.220 milliseconds, and that's a really long time to wait 00:14:23.220 --> 00:14:26.160 for your request to actually come in and, and 00:14:26.160 --> 00:14:28.620 get finished. So even though somebody is getting really 00:14:28.620 --> 00:14:31.019 fast responses, and your average isn't bad - your 00:14:31.019 --> 00:14:34.850 average is under 250 milliseconds - one customer might 00:14:34.850 --> 00:14:36.889 be getting a really slow response and a really 00:14:36.889 --> 00:14:39.470 fast response, and, and the net is a bad 00:14:39.470 --> 00:14:40.350 experience. 00:14:40.350 --> 00:14:43.620 So, the net, it, it just, it's a very 00:14:43.620 --> 00:14:47.790 inconsistent experience. So whenever we're talking about application speed, 00:14:47.790 --> 00:14:50.959 we have to consider individual request speed and average, 00:14:50.959 --> 00:14:55.579 but also consistency. How consistent is each request? 00:14:55.579 --> 00:14:58.569 So, how do, how do we do this? What? 00:14:58.569 --> 00:15:00.639 How can we, how can we help with this? 00:15:00.639 --> 00:15:02.529 Well, one of the things that we launched this 00:15:02.529 --> 00:15:06.249 year was PX dynos. So a PX dyno - 00:15:06.249 --> 00:15:09.430 a typical dyno only has 512 megabytes of RAM. 00:15:09.430 --> 00:15:12.529 It's a shared infrastructure. A PX Dyna has six 00:15:12.529 --> 00:15:15.959 gigabytes of RAM and eight CPU cores, which is 00:15:15.959 --> 00:15:17.680 a little, a little nicer, a little better, a 00:15:17.680 --> 00:15:19.819 little bit more room to play. 00:15:19.819 --> 00:15:24.499 And, and it's also real hardware. So, or, it's, 00:15:24.499 --> 00:15:29.430 it's not on the same shared infrastructure. So you 00:15:29.430 --> 00:15:32.560 can, you can scale with Dynos, you, you can 00:15:32.560 --> 00:15:34.680 also scale inside of Dynos. And that's kind of 00:15:34.680 --> 00:15:37.319 two, two important parts that we're gonna, gonna have 00:15:37.319 --> 00:15:40.379 to cover. So, of course, whenever you have more 00:15:40.379 --> 00:15:42.810 requests that you can possibly process, you want to 00:15:42.810 --> 00:15:45.439 scale up and say, I'm gonna have more Dynos. 00:15:45.439 --> 00:15:50.430 But what happens if, if you're not making the 00:15:50.430 --> 00:15:53.120 best use of everything inside of your dyno? Previously 00:15:53.120 --> 00:15:55.160 with 512 megabytes of RAM, you could just, you 00:15:55.160 --> 00:15:56.889 know, throw a couple Unicorn workers in there and 00:15:56.889 --> 00:15:58.910 you're like, oh, I'm probably using most of this. 00:15:58.910 --> 00:16:00.879 Like, if you put two unicorn workers in a 00:16:00.879 --> 00:16:03.370 PX Dyno, you're not making the most use of 00:16:03.370 --> 00:16:04.209 it all. 00:16:04.209 --> 00:16:07.740 So, recently, I am super in love with, with 00:16:07.740 --> 00:16:12.949 Puma. Evan, this is Evan Phoenix's web server that 00:16:12.949 --> 00:16:15.059 was originally written at, to kind of show case 00:16:15.059 --> 00:16:18.050 Rubinius. Guess what? It's really nice with MRI as 00:16:18.050 --> 00:16:22.029 well. Recently we've gotten some Puma docs, and so 00:16:22.029 --> 00:16:23.809 I'm gonna talk about Puma for just, for just 00:16:23.809 --> 00:16:24.540 a little bit. 00:16:24.540 --> 00:16:29.559 So, if you're, if you're not familiar. I was 00:16:29.559 --> 00:16:34.829 totally off on the formatting. So, Puma handles requests 00:16:34.829 --> 00:16:38.279 by running multiple processes, or by multiple threads. And 00:16:38.279 --> 00:16:40.040 it can actually run in something called a hybrid 00:16:40.040 --> 00:16:43.189 mode, where each process has multiple threads. We, we 00:16:43.189 --> 00:16:45.819 recommend this, or I recommend this. E- if one 00:16:45.819 --> 00:16:47.980 of your processes crash, it doesn't crash your entire 00:16:47.980 --> 00:16:51.399 web server. It's kind of nice. 00:16:51.399 --> 00:16:54.110 And so the multiple processes is something that we're 00:16:54.110 --> 00:16:57.959 pretty familiar with. As Rubyists, we're familiar with forking 00:16:57.959 --> 00:17:02.129 processes. We're familiar with Unicorn. But the, the, the 00:17:02.129 --> 00:17:04.220 multiple threads is a little bit different. 00:17:04.220 --> 00:17:07.260 Even with MRI, even with a, something like a 00:17:07.260 --> 00:17:10.359 global interpreter lock, you are still doing enough IO, 00:17:10.359 --> 00:17:13.569 you're still hitting your database frequently enough, maybe making 00:17:13.569 --> 00:17:17.510 API calls to like, Facebook or GitHub status, being 00:17:17.510 --> 00:17:19.970 like, hey, are you still up? 00:17:19.970 --> 00:17:23.099 And, and this will give our threads time to 00:17:23.099 --> 00:17:25.460 kind of jump around and allow others to do 00:17:25.460 --> 00:17:27.050 work. So you, you can get quite an extra 00:17:27.050 --> 00:17:29.110 bit of performance with, there. 00:17:29.110 --> 00:17:31.290 So, we're actually gonna be using Puma to scale 00:17:31.290 --> 00:17:33.470 up inside of our Dynos. So once we give 00:17:33.470 --> 00:17:35.780 you that eight gigs of RAM, we want to 00:17:35.780 --> 00:17:38.280 make sure that, that you can, you can, you 00:17:38.280 --> 00:17:39.830 can make the most use out of it. 00:17:39.830 --> 00:17:43.710 In general, with Puma, more processes means more RAM, 00:17:43.710 --> 00:17:47.360 and more threads are gonna need more CPU consumption. 00:17:47.360 --> 00:17:50.050 So, you want to, you want to maximize your 00:17:50.050 --> 00:17:52.870 processes and maximize your threads, kind of without going 00:17:52.870 --> 00:17:54.370 over. As soon as you start swapping, as soon 00:17:54.370 --> 00:17:56.550 as you go over that RAM limit, your app's 00:17:56.550 --> 00:17:58.930 gonna be really slow and that kind of defeats 00:17:58.930 --> 00:18:02.670 the purpose of trying to add these resources. 00:18:02.670 --> 00:18:06.240 Another issue is that I had kind of never 00:18:06.240 --> 00:18:07.940 heard of until I started looking into all of 00:18:07.940 --> 00:18:11.300 these multiple web servers, is slow-client. So if somebody's 00:18:11.300 --> 00:18:13.940 connecting to your web site via like, a two 00:18:13.940 --> 00:18:17.660 G over like a Nokia candy bar phone, uploading 00:18:17.660 --> 00:18:20.180 like photos or something like that, like that is 00:18:20.180 --> 00:18:22.330 a slow client, and if you're using something like 00:18:22.330 --> 00:18:27.000 Unicorn, it can deDOS your, your site, because each 00:18:27.000 --> 00:18:29.670 one of those requests takes up an entire Unicorn 00:18:29.670 --> 00:18:32.890 worker, whereas Puma has a, has a buffer, and 00:18:32.890 --> 00:18:36.390 it buffers those requests as similar to the way 00:18:36.390 --> 00:18:38.340 NginX does. 00:18:38.340 --> 00:18:40.860 One other thing to consider with Puma is, so 00:18:40.860 --> 00:18:44.260 I'm mentioning threads, I'm talking, talking about threads. Ruby, 00:18:44.260 --> 00:18:48.570 we're not necessarily known as the most thread-safe culture. 00:18:48.570 --> 00:18:52.060 Thread-safe community. And so a lot of apps just 00:18:52.060 --> 00:18:54.540 aren't thread-safe. And so some, you might take a 00:18:54.540 --> 00:18:55.920 look at Puma and be like, hey, that's not 00:18:55.920 --> 00:18:58.820 for me. You can always set your maximum threads 00:18:58.820 --> 00:19:01.400 to one, and then now you're behaving just like 00:19:01.400 --> 00:19:05.270 Unicorn, except you have the slow-client protection, and whenever 00:19:05.270 --> 00:19:08.170 you get that gem that's bad or you, like, 00:19:08.170 --> 00:19:13.180 stop mutating your constants are runtime or something, then 00:19:13.180 --> 00:19:15.570 you can maybe bump up and try multiple threads. 00:19:15.570 --> 00:19:18.980 OK. So, I'm, I'm talking about consistency and I'm 00:19:18.980 --> 00:19:20.990 talking a lot about Puma. How does that all 00:19:20.990 --> 00:19:25.380 kind of boil down and help? So, does anybody 00:19:25.380 --> 00:19:29.630 think that sharing distributed state across multiple machines is 00:19:29.630 --> 00:19:36.630 like really fast? Maybe. I, OK. Good. 00:19:37.150 --> 00:19:39.440 What about sharing state inside of memory on the 00:19:39.440 --> 00:19:44.520 same machine? Is that faster? OK. All right. I 00:19:44.520 --> 00:19:47.550 think we're in, in agreement. So, a, a little 00:19:47.550 --> 00:19:50.120 bit of a point of controversy. You might have 00:19:50.120 --> 00:19:53.750 heard of the, the Heroku router at some point 00:19:53.750 --> 00:19:58.310 in time. And this, the router is actually designed, 00:19:58.310 --> 00:20:00.630 not randomly, but it is, it is designed to 00:20:00.630 --> 00:20:04.300 use a random algorithm. And it basically will try 00:20:04.300 --> 00:20:07.360 to deliver requests as fast as humanly possible, or 00:20:07.360 --> 00:20:11.210 computerly possible, to individual dynos. So it's like, it 00:20:11.210 --> 00:20:12.580 gets the request. It wants to get it to 00:20:12.580 --> 00:20:15.280 your dyno as fast as it possibly can. 00:20:15.280 --> 00:20:18.260 And adding any sort of additional overhead of distributed 00:20:18.260 --> 00:20:24.180 locks or queues is gonna be slowing that down. 00:20:24.180 --> 00:20:26.690 Once inside of your, in your, your process, Puma 00:20:26.690 --> 00:20:30.760 or Unicorn has, in memory, state of all of 00:20:30.760 --> 00:20:33.300 its own processes, and is capable of saying, oh, 00:20:33.300 --> 00:20:35.400 hey, this process is busy. This process is not 00:20:35.400 --> 00:20:40.060 busy. I can do really intelligent routing and, and 00:20:40.060 --> 00:20:43.720 basically, for free. 00:20:43.720 --> 00:20:45.970 It's really fast. It, it took a little bit 00:20:45.970 --> 00:20:49.600 of convincing for me. So does anybody else need 00:20:49.600 --> 00:20:50.460 to be convinced? 00:20:50.460 --> 00:20:51.590 AUDIENCE: Yeah. 00:20:51.590 --> 00:20:54.680 R.S.: OK. Good. Cause otherwise I could totally just 00:20:54.680 --> 00:20:57.280 skip over the next section of slides. 00:20:57.280 --> 00:21:00.570 So, this is, this is a graph produced by 00:21:00.570 --> 00:21:05.760 the fine developers over at, at RapGenius, and on 00:21:05.760 --> 00:21:07.940 one side we will actually see a percentage of 00:21:07.940 --> 00:21:10.740 requests queued, and on the bottom we are gonna 00:21:10.740 --> 00:21:13.160 be seeing number of dynos. So the goal is 00:21:13.160 --> 00:21:15.930 actually to minimize request queuing, like this, this is 00:21:15.930 --> 00:21:18.000 time that your customers are waiting that you're not 00:21:18.000 --> 00:21:19.560 actually doing anything. 00:21:19.560 --> 00:21:21.870 You, so you, you want to minimize that queuing 00:21:21.870 --> 00:21:24.220 with the smallest number of resources, so the smallest 00:21:24.220 --> 00:21:28.020 number of dynos. This top line, we actually have 00:21:28.020 --> 00:21:30.510 is what we've currently got now. The random routing 00:21:30.510 --> 00:21:34.560 with a, a single-threaded server. And, like, this is 00:21:34.560 --> 00:21:36.960 pretty bad. It, like, starts out bad and it, 00:21:36.960 --> 00:21:39.180 like, it doesn't even like trend towards zero. So 00:21:39.180 --> 00:21:41.120 this is probably bad. So this is using something 00:21:41.120 --> 00:21:43.500 like Webrick in production. 00:21:43.500 --> 00:21:47.890 So. Don't use Webrick in production. Or, or like 00:21:47.890 --> 00:21:54.890 even, even thin. In single-threaded mode. So, on the, 00:21:55.150 --> 00:21:58.050 on the very bottom, we actually have a, like, 00:21:58.050 --> 00:22:01.280 mythological, like, if, if we could do all of 00:22:01.280 --> 00:22:05.410 that distributed shared state without, and locks and queues, 00:22:05.410 --> 00:22:08.250 without having any kind of overhead, we can see 00:22:08.250 --> 00:22:11.390 that, basically, it just drops down to zero at 00:22:11.390 --> 00:22:14.380 about, you know, in their case, about seventy-five dynos, 00:22:14.380 --> 00:22:16.350 and then just, you know, it's straight zero. There's 00:22:16.350 --> 00:22:17.190 no queuing. 00:22:17.190 --> 00:22:19.020 And it's great. And this would be amazing if 00:22:19.020 --> 00:22:22.200 we could have it. But unfortunately there is a 00:22:22.200 --> 00:22:24.750 little bit over overhead. What was really interesting to 00:22:24.750 --> 00:22:27.570 me is this second one, which is not nearly 00:22:27.570 --> 00:22:30.900 as nice as that mythological intelligent router, but it's 00:22:30.900 --> 00:22:34.210 kind of not too far off. This is still 00:22:34.210 --> 00:22:36.890 our random routing, and, and this was actually done 00:22:36.890 --> 00:22:41.880 with Unicorn, and workers set to two. So basically, 00:22:41.880 --> 00:22:44.490 once we get the, the request to your operating 00:22:44.490 --> 00:22:46.260 system, it's like one of those two workers is 00:22:46.260 --> 00:22:48.620 free and can immediately start working on it. 00:22:48.620 --> 00:22:51.490 Some, some interesting things known about this is, for 00:22:51.490 --> 00:22:54.220 the non-optimal case, for the, we basically don't have 00:22:54.220 --> 00:22:56.460 enough dynos to handle this, so that might happen 00:22:56.460 --> 00:22:58.910 is, you know, you got on Hacker News or 00:22:58.910 --> 00:23:05.910 whatever, slash dotted, Reddited. SnapChatted. Secreted. I don't know. 00:23:07.620 --> 00:23:11.990 And it does actually, eventually approach ideal state. So, 00:23:11.990 --> 00:23:14.820 it, it gets even better, and unfortunately they kind 00:23:14.820 --> 00:23:17.820 of stopped at, at two processes, but it gets 00:23:17.820 --> 00:23:20.290 better, the more concurrency that you add. So if 00:23:20.290 --> 00:23:23.230 you had three or four workers or, again, if 00:23:23.230 --> 00:23:24.830 you're using something like Puma, and each one of 00:23:24.830 --> 00:23:27.630 those workers is running, like, four threads, now you 00:23:27.630 --> 00:23:29.540 have, like, a massive amount of concurrency that you 00:23:29.540 --> 00:23:31.050 could deal with all of these requests coming in. 00:23:31.050 --> 00:23:35.640 So, the, the, the, if, again, and we're looking 00:23:35.640 --> 00:23:37.920 for consistency. We want that request to get to 00:23:37.920 --> 00:23:40.850 our dyno, immediately be able to process it. So, 00:23:40.850 --> 00:23:43.610 you can use Puma or Unicorn to maximize that 00:23:43.610 --> 00:23:49.700 worker number, and, again, distributed optimi- distributed routing is 00:23:49.700 --> 00:23:54.140 slow. In memory, routing is relatively quick. 00:23:54.140 --> 00:24:00.100 On, again, just in, in the whole context of 00:24:00.100 --> 00:24:04.820 speed, Ruby 2.0 came out and this was awhile 00:24:04.820 --> 00:24:07.250 ago. It's got GC, it's optimized for Copy on 00:24:07.250 --> 00:24:12.180 Write. In, in Ruby, extra processes, process forks actually 00:24:12.180 --> 00:24:14.430 become cheaper. So the first process might take seventy 00:24:14.430 --> 00:24:17.250 megabytes, the second one twenty, and ten and seven 00:24:17.250 --> 00:24:19.900 and six. So, if you get a larger box, 00:24:19.900 --> 00:24:22.620 you can actually run more processes on them. If 00:24:22.620 --> 00:24:24.300 you get eight gigs on one box, you can 00:24:24.300 --> 00:24:26.890 run more processes than you can if you had 00:24:26.890 --> 00:24:28.960 eight gigs across eight boxes. 00:24:28.960 --> 00:24:33.470 So, again, more processes mean more concurrency, and more 00:24:33.470 --> 00:24:38.900 concurrency means consistency. If you are using workers, you 00:24:38.900 --> 00:24:43.400 can, you can also scale out with Resq pool, 00:24:43.400 --> 00:24:46.370 and if your application's still slow, we rolled out 00:24:46.370 --> 00:24:48.480 a couple of really neat platform features. One of 00:24:48.480 --> 00:24:52.040 them is, is called HTTP Request ID. So as 00:24:52.040 --> 00:24:54.860 a request comes into our system, we will actually 00:24:54.860 --> 00:24:57.300 give it a uuid, and you can see this 00:24:57.300 --> 00:25:00.490 in your router log. And then we've got documentation 00:25:00.490 --> 00:25:02.790 on how to configure your Rails app so it 00:25:02.790 --> 00:25:05.410 will actually pick this up and use that uuid 00:25:05.410 --> 00:25:06.590 in tagged logs. 00:25:06.590 --> 00:25:09.590 So, like, how is this useful? So, if you 00:25:09.590 --> 00:25:11.740 are getting, like, and out of memory error, or 00:25:11.740 --> 00:25:13.870 if your request is taking a really long time 00:25:13.870 --> 00:25:16.200 and you're like, ah, like, that request is timing 00:25:16.200 --> 00:25:19.380 out and, you know, Heroku's returning a response and 00:25:19.380 --> 00:25:22.320 we don't even know why. Now, if the request 00:25:22.320 --> 00:25:24.370 id is tagged, you can actually follow along between 00:25:24.370 --> 00:25:26.090 your two logs and be like, oh, it's hitting 00:25:26.090 --> 00:25:28.540 that controller action. Maybe I should be sending that 00:25:28.540 --> 00:25:31.160 email in the background as opposed to having to 00:25:31.160 --> 00:25:34.140 actually block on it. So you can trace specific 00:25:34.140 --> 00:25:34.960 errors. 00:25:34.960 --> 00:25:38.500 We also launched Log Runtime Metrics awhile ago, and 00:25:38.500 --> 00:25:42.950 this is something that we'll actually put your, your 00:25:42.950 --> 00:25:45.240 runtime information directly into your logs. You can check 00:25:45.240 --> 00:25:49.870 it out. Liberato will automatically pick it up for 00:25:49.870 --> 00:25:52.500 you and make you these, these really nice graphs. 00:25:52.500 --> 00:25:55.330 And, again, if you're doing something like Unicorn or, 00:25:55.330 --> 00:25:57.460 or Puma, then you want to get as close 00:25:57.460 --> 00:26:00.570 to your RAM limit without actually going over. 00:26:00.570 --> 00:26:05.400 OK. So, the, the next act in, in our 00:26:05.400 --> 00:26:09.660 play, again, introducing Terence, is, we'll be talking about 00:26:09.660 --> 00:26:12.160 Ruby on the Heroku stack and in the community. 00:26:12.160 --> 00:26:15.630 T.L.: Thank you. So I know we're at RailsConf, 00:26:15.630 --> 00:26:17.690 but I've been doing a bunch of work with 00:26:17.690 --> 00:26:20.940 Ruby, so I wanted to talk about some Ruby 00:26:20.940 --> 00:26:25.420 stuff. So who here is actually using Ruby 1.8.7? 00:26:25.420 --> 00:26:31.890 Wow. No one. That's pretty awesome. Oh, wait. One 00:26:31.890 --> 00:26:34.940 person. You should probably get off of it. 00:26:34.940 --> 00:26:35.880 [laughter] 00:26:35.880 --> 00:26:41.190 But. Who is using Ruby 1.9.2? A few more 00:26:41.190 --> 00:26:48.190 people. 1.9.3? Good amount of people here. 00:26:48.660 --> 00:26:51.160 So, I don't know if you guys were following 00:26:51.160 --> 00:26:54.980 along, but Ruby 1.8.7 and 1.9.2 got end-of-lifed at 00:26:54.980 --> 00:26:59.470 one point. And then there was a security incidence, 00:26:59.470 --> 00:27:03.510 and Zachary Scott and I have volunteered to maintain 00:27:03.510 --> 00:27:07.350 security patches till the end of June. So, if 00:27:07.350 --> 00:27:12.860 you are on 1.8.7 and 1.9.2, I would recommend 00:27:12.860 --> 00:27:16.870 hopefully getting off some time soon, unless you don't 00:27:16.870 --> 00:27:19.190 care about security or want to back port your 00:27:19.190 --> 00:27:22.490 own patches. 00:27:22.490 --> 00:27:25.040 And then Ru- we recently announced that Ruby 1.9.3 00:27:25.040 --> 00:27:27.530 is also getting an end of life in February 00:27:27.530 --> 00:27:32.380 2015, which is coming up relatively quickly. It's a 00:27:32.380 --> 00:27:34.010 little less than a year away now, at this 00:27:34.010 --> 00:27:38.560 point. So, please upgrade to at least 2.0.0 or 00:27:38.560 --> 00:27:40.130 later. 00:27:40.130 --> 00:27:43.880 And, during this past Rails Standard Year, we also 00:27:43.880 --> 00:27:46.200 moved the default Ruby on Heroku from 1.9.2 to 00:27:46.200 --> 00:27:49.710 2 dot 0 dot 0. We believe people should 00:27:49.710 --> 00:27:52.470 be at least using this version of Ruby or 00:27:52.470 --> 00:27:53.560 higher. 00:27:53.560 --> 00:27:56.530 And, if you don't know yet, you can declare 00:27:56.530 --> 00:27:59.790 your Ruby version in the gemfile on Heroku to 00:27:59.790 --> 00:28:05.480 get that version. And, we've also are pretty, pretty 00:28:05.480 --> 00:28:08.740 serious about supporting the latest versions of Ruby. Basically, 00:28:08.740 --> 00:28:10.320 the same day that they come out. So we 00:28:10.320 --> 00:28:15.460 did this for 2.0.0, 2.1.0 and 2.1.1, and in 00:28:15.460 --> 00:28:17.880 addition we also try to support any of the 00:28:17.880 --> 00:28:21.860 preview releases as, whenever they get form release, so 00:28:21.860 --> 00:28:25.410 we can, as a community, help, help find bugs 00:28:25.410 --> 00:28:27.880 and test things like, put your staging app on 00:28:27.880 --> 00:28:30.870 new versions of Ruby. If you find bugs then, 00:28:30.870 --> 00:28:32.900 hopefully, we can fix them before they actually make 00:28:32.900 --> 00:28:35.720 it to the final release. 00:28:35.720 --> 00:28:39.850 And, with regards to security patches, if there are 00:28:39.850 --> 00:28:41.990 any security releases that come out, we make sure 00:28:41.990 --> 00:28:44.850 to release them that day as well. We take 00:28:44.850 --> 00:28:47.280 security pretty seriously. 00:28:47.280 --> 00:28:50.540 So, once a security patch has release and we've 00:28:50.540 --> 00:28:54.390 patched those Rubies, you have to push your app 00:28:54.390 --> 00:28:57.420 again to get that release. And the reason we, 00:28:57.420 --> 00:28:58.890 well, a lot of people ask us, like, why 00:28:58.890 --> 00:29:01.710 we don't do that. Why we don't just automatically 00:29:01.710 --> 00:29:04.640 upgrade peoples' Rubies in place. And the reason it, 00:29:04.640 --> 00:29:07.900 the reasoning here is that, not, there might be 00:29:07.900 --> 00:29:09.790 a regression to the security patch, or maybe the 00:29:09.790 --> 00:29:13.360 patch level is not 100% backwards compatible. There's a 00:29:13.360 --> 00:29:16.190 bug that slipped through. But you probably want to 00:29:16.190 --> 00:29:18.810 be there when you're actually deploying your application, in 00:29:18.810 --> 00:29:20.790 case something does go wrong. 00:29:20.790 --> 00:29:22.740 You probably wouldn't want us to deploy something and 00:29:22.740 --> 00:29:25.270 then have your site go down and then you're 00:29:25.270 --> 00:29:27.290 like, not at your computer at all. You're at 00:29:27.290 --> 00:29:29.350 dinner somewhere and it's like super inconvenient to get 00:29:29.350 --> 00:29:30.870 paged there. 00:29:30.870 --> 00:29:34.800 So, we publish all of this information, all of 00:29:34.800 --> 00:29:37.420 the updates to the platform, but also all of 00:29:37.420 --> 00:29:41.230 the Ruby updates, including security updates to the devcenter 00:29:41.230 --> 00:29:45.150 changelogs. So, if you don't, this is, I think, 00:29:45.150 --> 00:29:48.360 devcenter dot heroku dot com slash changelog. And if 00:29:48.360 --> 00:29:51.490 you don't subscribe to it, I would recommend subscribing 00:29:51.490 --> 00:29:53.860 to it just to keep up to date with 00:29:53.860 --> 00:29:57.010 what is happening on Heroku for platform changes in 00:29:57.010 --> 00:30:02.510 addition to updates to Ruby specifically on Heroku. And, 00:30:02.510 --> 00:30:04.710 there isn't too much traffic. Like, you won't get, 00:30:04.710 --> 00:30:07.380 like, a hundred emails a day. So, I highly 00:30:07.380 --> 00:30:09.940 recommend subscribing to this to just keep up to 00:30:09.940 --> 00:30:14.090 date with things like that here. 00:30:14.090 --> 00:30:15.110 So the next thing I would like to talk 00:30:15.110 --> 00:30:18.590 about is Matz's Ruby team. So if you didn't 00:30:18.590 --> 00:30:22.120 know, back in 2012 we hired three people from 00:30:22.120 --> 00:30:27.410 Ruby core. We hired Matz himself, Koichi and Nobu. 00:30:27.410 --> 00:30:29.850 And, as I've gone around over the last years 00:30:29.850 --> 00:30:32.310 talking and interacting with people, I realize a lot 00:30:32.310 --> 00:30:35.600 of people have no idea who, besides Matz, who 00:30:35.600 --> 00:30:37.130 Koichi and Nobu are. 00:30:37.130 --> 00:30:38.400 So I wanted to take the time to kind 00:30:38.400 --> 00:30:42.100 of update people on who these people were and 00:30:42.100 --> 00:30:44.200 kind of what they've actually, like, we've been paying 00:30:44.200 --> 00:30:46.700 them money, and what they've actually been doing to 00:30:46.700 --> 00:30:50.710 kind of move Ruby forward in a positive direction. 00:30:50.710 --> 00:30:54.820 So, if you run a git log, since 2012, 00:30:54.820 --> 00:30:56.970 since we've hired them, you can see the number 00:30:56.970 --> 00:31:01.720 of commits they've made to Ruby itself. So, the, 00:31:01.720 --> 00:31:05.200 so Nobu here, who we've hired, has basically more 00:31:05.200 --> 00:31:07.970 commits than like the second guy by many, many 00:31:07.970 --> 00:31:13.610 commits. And then, Koichi's the third highest committer as 00:31:13.610 --> 00:31:15.680 well. 00:31:15.680 --> 00:31:17.560 And you're probably wondering why I have six names 00:31:17.560 --> 00:31:21.260 here on a list for the top five. And 00:31:21.260 --> 00:31:23.410 so there is this, so there isn't actually on 00:31:23.410 --> 00:31:25.650 the Ruby Core Team who has the handle svn. 00:31:25.650 --> 00:31:28.590 It's not actually a person. So I find out, 00:31:28.590 --> 00:31:32.870 the hard way, who this person was. So when 00:31:32.870 --> 00:31:35.480 I made my first patch to Ruby after being 00:31:35.480 --> 00:31:40.040 on core, I found out that if, all the 00:31:40.040 --> 00:31:43.160 date information is done in JST, and I, of 00:31:43.160 --> 00:31:45.840 course, did not know that, and but like scumbag 00:31:45.840 --> 00:31:48.610 American dates. And so there's basically this bot that 00:31:48.610 --> 00:31:51.330 will go through and, like, fix your commits for 00:31:51.330 --> 00:31:53.760 you, and like, so he does like another commit, 00:31:53.760 --> 00:31:55.750 and it's like, ah, you actually put the wrong 00:31:55.750 --> 00:31:57.360 date. Let me fix that for you. 00:31:57.360 --> 00:32:00.120 So there's like 710 of those commits. I think 00:32:00.120 --> 00:32:02.680 I did this like a month ago. So these 00:32:02.680 --> 00:32:05.880 are the number of commits from a month ago. 00:32:05.880 --> 00:32:07.630 So, the first person I like to talk about 00:32:07.630 --> 00:32:13.200 is Nobuyoshi Nokada. Also known as Nobu. And he's 00:32:13.200 --> 00:32:15.430 known, I think on Ruby Core as The Patch 00:32:15.430 --> 00:32:21.630 Monster. So, we'll go into why he's known by 00:32:21.630 --> 00:32:22.320 this. 00:32:22.320 --> 00:32:25.540 So, what do you think the result of Time 00:32:25.540 --> 00:32:30.510 dot now equals empty string? I'm sure you thought 00:32:30.510 --> 00:32:35.790 it was an infinite loop, right. Or using the 00:32:35.790 --> 00:32:40.460 rational, like, so if you're using the rational number 00:32:40.460 --> 00:32:42.440 library in standard lib, like, what do you think 00:32:42.440 --> 00:32:44.990 the result of doing this operation? 00:32:44.990 --> 00:32:45.810 AUDIENCE: Segfault. 00:32:45.810 --> 00:32:48.260 T.L.: Yeah. So this is a segfault. 00:32:48.260 --> 00:32:50.090 AUDIENCE: [indecipherable - 00:32:40] 00:32:50.090 --> 00:32:54.760 T.L.: Thank you. Thank you for reporting the bug. 00:32:54.760 --> 00:32:57.230 So these, so Eric Hodel actually reported the other 00:32:57.230 --> 00:32:59.350 bug, the time thing, and he found this in 00:32:59.350 --> 00:33:02.050 RubyGems I believe. But these are real issues that 00:33:02.050 --> 00:33:04.940 are in Ruby itself. So if you actually run 00:33:04.940 --> 00:33:06.990 those two things now and you're using later patch 00:33:06.990 --> 00:33:10.200 levels, you should not see them. But, they're real 00:33:10.200 --> 00:33:14.190 issues, and someone has to go and fix all 00:33:14.190 --> 00:33:14.960 them. 00:33:14.960 --> 00:33:17.760 And so the person who actually does this is 00:33:17.760 --> 00:33:22.160 Nobu. And he actually gets paid full time and, 00:33:22.160 --> 00:33:25.040 to basically do bug fixes for Ruby. So all 00:33:25.040 --> 00:33:27.330 those two thousand seven hundred and some commits are 00:33:27.330 --> 00:33:30.220 bug fixes to Ruby trunk to make Ruby run 00:33:30.220 --> 00:33:34.370 better. And, I thanked him when I was just 00:33:34.370 --> 00:33:36.660 in Japan last week for all the work he's 00:33:36.660 --> 00:33:39.730 done. It's pretty incredible. Like, there's so many times 00:33:39.730 --> 00:33:42.270 when things segfault and other things, and he's basically 00:33:42.270 --> 00:33:44.110 made it better. 00:33:44.110 --> 00:33:46.280 I was at Oweito, and there was actually someone 00:33:46.280 --> 00:33:49.050 giving a presentation about, like, thirty tips of, like, 00:33:49.050 --> 00:33:51.570 how to use Ruby. And someone was talking about 00:33:51.570 --> 00:33:55.180 open uri, and there was code on the screen, 00:33:55.180 --> 00:33:57.770 and he found a bug during, like, the guy's 00:33:57.770 --> 00:34:00.190 presentation, and during it, he committed a patch to 00:34:00.190 --> 00:34:04.500 trunk during that guy's presentation. So, he's pretty awesome. 00:34:04.500 --> 00:34:07.810 He doesn't, he doesn't do, he hasn't done any 00:34:07.810 --> 00:34:09.949 talks, but I think people should know about the 00:34:09.949 --> 00:34:11.619 work he's been doing. 00:34:11.619 --> 00:34:14.589 So, this last bug, actually, that I wanted to 00:34:14.589 --> 00:34:16.980 talk about, that he fixed was, are any of 00:34:16.980 --> 00:34:20.290 you familiar with the regression from, in Ruby 2.1.1, 00:34:20.290 --> 00:34:23.480 which regards to hash? 00:34:23.480 --> 00:34:26.590 So, I'm sure you're familiar with the fact that, 00:34:26.590 --> 00:34:29.469 if you use Ruby 2.1.1 on Rails 4.0.3, it 00:34:29.469 --> 00:34:35.010 just doesn't work. Like, there, in Rails, we, we 00:34:35.010 --> 00:34:38.870 use this, we fetch, in hashes we use objects 00:34:38.870 --> 00:34:41.480 as keys. And if you override the hash and 00:34:41.480 --> 00:34:44.149 equal method, and when you fetch you won't get 00:34:44.149 --> 00:34:48.060 the right result back. So inside of Rails in 00:34:48.060 --> 00:34:50.739 4.0.4, they actually had to work around this bug. 00:34:50.739 --> 00:34:53.139 And, Nobu actually was the one who fixed this 00:34:53.139 --> 00:34:55.370 inside of Ruby itself. 00:34:55.370 --> 00:34:58.980 So, these were just, like, the three most interesting 00:34:58.980 --> 00:35:01.720 bugs that I found from within the last year 00:35:01.720 --> 00:35:05.050 or two of stuff he's worked on. But, if 00:35:05.050 --> 00:35:07.720 you look on site Ruby Core, you can find, 00:35:07.720 --> 00:35:09.860 like, hundreds and hundreds of bugs that he's done 00:35:09.860 --> 00:35:12.960 within the last year of, just like, rip segfaults 00:35:12.960 --> 00:35:16.750 and other things. So he's great to work with. 00:35:16.750 --> 00:35:19.450 So the next person I want to talk about 00:35:19.450 --> 00:35:26.450 is Koichi Sasada. He's also known as ko1, ko1. 00:35:26.800 --> 00:35:30.230 And he doesn't have a nickname in Ruby Core, 00:35:30.230 --> 00:35:32.560 so me and Richard spent a good amount of 00:35:32.560 --> 00:35:34.900 our talk preparation trying to come up with a 00:35:34.900 --> 00:35:36.680 nickname for him. So we came up with the 00:35:36.680 --> 00:35:39.240 Performance Pro. And this is a picture of him 00:35:39.240 --> 00:35:42.650 giving a talk in Japanese. 00:35:42.650 --> 00:35:47.590 So, if you use Ruby 1.9 at all, he 00:35:47.590 --> 00:35:52.850 worked on Yarv. So basically the new VM stuff 00:35:52.850 --> 00:35:55.600 that made Ruby 1.9, I think it was like 00:35:55.600 --> 00:35:59.860 thirty percent faster than 1.8 for like longer-running processes. 00:35:59.860 --> 00:36:04.830 More recently he's worked on the RGenGC. And this 00:36:04.830 --> 00:36:08.400 was introduced in Ruby 2.1.1, and it allows faster 00:36:08.400 --> 00:36:13.260 code execution by having, basically, shorter GC pauses. So 00:36:13.260 --> 00:36:15.360 instead of doing full GC every time, like, you 00:36:15.360 --> 00:36:17.690 can have these minor ones. 00:36:17.690 --> 00:36:22.110 So, just, he spends all of his time thinking 00:36:22.110 --> 00:36:25.030 about performance in Ruby, and that's like what he's 00:36:25.030 --> 00:36:28.640 paid to work on. So if anyone actually cares 00:36:28.640 --> 00:36:31.650 about Ruby performance, you should thank this guy for 00:36:31.650 --> 00:36:33.630 the work he's done. If you've looked at the 00:36:33.630 --> 00:36:37.030 performance of Ruby since, in the last few years, 00:36:37.030 --> 00:36:39.960 like it's improved a lot. A lot due to 00:36:39.960 --> 00:36:41.480 this guy's work. 00:36:41.480 --> 00:36:43.780 And, I was just, I was talking to him, 00:36:43.780 --> 00:36:46.980 and he was telling me that he basically, like, 00:36:46.980 --> 00:36:49.000 when he was working on RGenGC, he like, he 00:36:49.000 --> 00:36:51.200 was just like, walking around the park and he 00:36:51.200 --> 00:36:53.110 had a breakthrough. So he like spends a lot 00:36:53.110 --> 00:36:55.580 of his time, even, off of work hours, just 00:36:55.580 --> 00:36:58.530 thinking about this stuff. 00:36:58.530 --> 00:37:00.070 Other stuff that he's been working on as well 00:37:00.070 --> 00:37:04.500 is profiling work. So, if you've used any of 00:37:04.500 --> 00:37:09.330 the Man stuff for 2.1.1, with the MemProfiler and 00:37:09.330 --> 00:37:12.110 other things, he's been working on, with him to 00:37:12.110 --> 00:37:14.930 introduce hooks into the internal API to make stuff 00:37:14.930 --> 00:37:17.830 like that work. So we, I think we understand 00:37:17.830 --> 00:37:20.420 that profiling, being able to measure your application for 00:37:20.420 --> 00:37:25.100 Ruby is super important. So, if you have basically 00:37:25.100 --> 00:37:29.100 comments or suggestions on things that you need or 00:37:29.100 --> 00:37:31.320 think that you can't improve this thing, like it's 00:37:31.320 --> 00:37:34.090 worth talking, reaching out and talking to Koichi about 00:37:34.090 --> 00:37:35.150 this. 00:37:35.150 --> 00:37:37.180 And some of the stuff he's been working on 00:37:37.180 --> 00:37:40.980 in this vein has been, like, the gc_tracer gem. 00:37:40.980 --> 00:37:44.390 So, using this to basically get more information about 00:37:44.390 --> 00:37:47.800 your garbage collector, an allocation_tracer gem to see how 00:37:47.800 --> 00:37:51.380 long live, like, objects are. And then even in 00:37:51.380 --> 00:37:54.550 2.2, we're, as a team, we're working on, there 00:37:54.550 --> 00:37:58.520 is an incremental GC patch, and then also. Or, 00:37:58.520 --> 00:38:01.060 he's working on making the GC better with incremental 00:38:01.060 --> 00:38:03.960 GC and there is symbol GC for security things, 00:38:03.960 --> 00:38:07.130 which'll be super good for Rails. So we can't 00:38:07.130 --> 00:38:09.530 get, like, DOS because of the symbol table being 00:38:09.530 --> 00:38:11.220 filled up. 00:38:11.220 --> 00:38:14.240 Another, so one of the things, when I was 00:38:14.240 --> 00:38:15.960 in Japan, we had a Ruby Core meeting, and 00:38:15.960 --> 00:38:20.150 we talked about Ruby releases. And releasing Ruby is 00:38:20.150 --> 00:38:23.990 kind of a slow process, and I was, I 00:38:23.990 --> 00:38:26.780 wasn't really sure why it took so long. And 00:38:26.780 --> 00:38:29.770 so I kind of asked the question, and, and 00:38:29.770 --> 00:38:34.980 Naruse, who's the release manager of 2.1 and was 00:38:34.980 --> 00:38:38.000 telling me that it requires lots of human and 00:38:38.000 --> 00:38:41.150 machine resources. Basically, Ruby has to work on many 00:38:41.150 --> 00:38:45.670 configurations, Linux distros, you know, on OS X and 00:38:45.670 --> 00:38:48.070 other things. And in order to release, like, the 00:38:48.070 --> 00:38:50.040 CI server has to pass and, like, you kind 00:38:50.040 --> 00:38:52.840 of have to pass on like various vendors and 00:38:52.840 --> 00:38:54.120 what not. And so like, there's a lot of 00:38:54.120 --> 00:38:58.850 coordination and like checking to like make an actual 00:38:58.850 --> 00:39:01.550 release happen. Which is why things don't release super 00:39:01.550 --> 00:39:02.460 fast. 00:39:02.460 --> 00:39:06.610 So, some of the stuff that Koichi and my 00:39:06.610 --> 00:39:08.110 team and other people on Ruby Core have been 00:39:08.110 --> 00:39:10.550 working on is, like, working on infrastructure and services 00:39:10.550 --> 00:39:13.970 to help with, basically, testing of Ruby, to kind 00:39:13.970 --> 00:39:16.880 of hopefully automate and, like, basically do that per, 00:39:16.880 --> 00:39:19.860 either nightly or per commit or something along those 00:39:19.860 --> 00:39:20.200 lines. 00:39:20.200 --> 00:39:22.780 So hopefully we can get releases that are faster 00:39:22.780 --> 00:39:26.720 and are out to users sooner. 00:39:26.720 --> 00:39:29.690 If you have ideas for Ruby 2.2, like, I 00:39:29.690 --> 00:39:32.940 would love to hear them. We have a meeting 00:39:32.940 --> 00:39:36.070 next month in May, about what is gonna go 00:39:36.070 --> 00:39:39.490 into Ruby 2.2. So I'd be more than happy 00:39:39.490 --> 00:39:42.380 to talk to you about ideas that you have 00:39:42.380 --> 00:39:45.980 that you would like to see there. I'm just 00:39:45.980 --> 00:39:47.620 gonna skip this stuff since I talked about it 00:39:47.620 --> 00:39:49.600 earlier, and we're running short on time. So, here's 00:39:49.600 --> 00:39:51.290 Scheems to actually talk about Rails. 00:39:51.290 --> 00:39:52.460 R.S.: OK. 00:39:52.460 --> 00:39:54.970 Has anybody used Rails? Have we covered that question 00:39:54.970 --> 00:39:59.900 yet? OK. Welcome to RailsConf. OK, so Rails 4.1 00:39:59.900 --> 00:40:00.500 on Heroku. 00:40:00.500 --> 00:40:03.130 A lot of things in a very short amount 00:40:03.130 --> 00:40:05.420 of time. We are secure by default. Have you 00:40:05.420 --> 00:40:08.670 heard of the secrets dot yml file? OK. So 00:40:08.670 --> 00:40:10.360 secrets dot yml file is actually reading out an 00:40:10.360 --> 00:40:13.070 environment variable by default, which is great. We love 00:40:13.070 --> 00:40:17.820 environment variables. It separates your config from your source. 00:40:17.820 --> 00:40:20.790 And, so whenever you push your app, we're gonna 00:40:20.790 --> 00:40:22.770 set this environment variable to just, like, literally a 00:40:22.770 --> 00:40:25.850 random value. And if, for some reason, you ever 00:40:25.850 --> 00:40:28.990 need to like change that, you can do so 00:40:28.990 --> 00:40:32.860 by just setting your, the, the secret key base 00:40:32.860 --> 00:40:35.260 environment variable to, to whatever you want. 00:40:35.260 --> 00:40:37.450 Maybe, you know, like another OpenSSL bug comes out 00:40:37.450 --> 00:40:42.150 or something. So, another thing that was worked on 00:40:42.150 --> 00:40:45.650 a bunch is the database_url environment variable. This is 00:40:45.650 --> 00:40:47.300 something that we have spent a lot of time 00:40:47.300 --> 00:40:49.380 looking at. And it's actually, support has been in 00:40:49.380 --> 00:40:52.010 Rails for a surprisingly large amount of time, to 00:40:52.010 --> 00:40:54.510 just read from the environment variable, but never quite 00:40:54.510 --> 00:40:58.380 worked due to some edge cases and random rake 00:40:58.380 --> 00:41:01.360 tasks and so on and so forth. So this, 00:41:01.360 --> 00:41:03.560 this December, around Christmas time, I spent a lot 00:41:03.560 --> 00:41:05.260 of time getting that to work. 00:41:05.260 --> 00:41:08.910 So I'd like to happily announce that Rails 4, 00:41:08.910 --> 00:41:13.490 4.1 actually does support the database_url environment variable out 00:41:13.490 --> 00:41:18.100 of the box. Whoo! And, so, some, to describe 00:41:18.100 --> 00:41:22.810 a little, like, the behavior is, bears going over. 00:41:22.810 --> 00:41:25.250 If the database_url is present, we're just gonna connect 00:41:25.250 --> 00:41:28.430 to that database. It's, that's pretty simple. Makes sense. 00:41:28.430 --> 00:41:30.800 If the database.yml is present but there's no environment 00:41:30.800 --> 00:41:33.230 variable, then we're gonna use that. That also just 00:41:33.230 --> 00:41:35.190 kind of makes sense. 00:41:35.190 --> 00:41:37.369 If both are present, then we're gonna merge the 00:41:37.369 --> 00:41:41.920 values. Makes sense, right? OK. 00:41:41.920 --> 00:41:46.600 So, we, that sounds crazy. Bear with me. But, 00:41:46.600 --> 00:41:48.369 a lot of people, you, you want to put 00:41:48.369 --> 00:41:52.440 your connection information in your database_url environment variable. But, 00:41:52.440 --> 00:41:54.840 there's also other values you can use inside of 00:41:54.840 --> 00:41:58.860 your database.yml file to configure ActiveRecord itself. Not your 00:41:58.860 --> 00:42:00.990 database. So you can turn off and on prepared 00:42:00.990 --> 00:42:03.230 statements. You can change your pool size. All this 00:42:03.230 --> 00:42:04.790 kind of thing. 00:42:04.790 --> 00:42:07.420 And, we wanted to still enable you to be 00:42:07.420 --> 00:42:10.300 able, able to do this. So the, the results 00:42:10.300 --> 00:42:15.369 are actually merged, and for, for somebody like Heroku 00:42:15.369 --> 00:42:19.310 or, like, if you're using another container, we don't 00:42:19.310 --> 00:42:21.260 have to have as much magic. If you, if 00:42:21.260 --> 00:42:24.460 you didn't know, database_url, we actually had to over, 00:42:24.460 --> 00:42:26.490 whatever your database_url was, we were just writing a 00:42:26.490 --> 00:42:28.850 file over top of it. And it's like, forget 00:42:28.850 --> 00:42:30.110 that. We're gonna write a custom file. 00:42:30.110 --> 00:42:32.580 So people would put stuff in their database_url, or 00:42:32.580 --> 00:42:34.869 their database.yml file, and they'd be surprised when it 00:42:34.869 --> 00:42:37.070 wasn't there. Like, a different file was there. So, 00:42:37.070 --> 00:42:39.390 we no longer, we no longer have to do 00:42:39.390 --> 00:42:41.380 that. And Rails plays a little bit nicer with, 00:42:41.380 --> 00:42:45.110 with this containerized style environment. 00:42:45.110 --> 00:42:47.900 It also means that, you could actually start putting 00:42:47.900 --> 00:42:52.260 your ActiveRecord configuration in that file. Another note, if 00:42:52.260 --> 00:42:55.500 you were manually setting that, your pool size or 00:42:55.500 --> 00:42:59.150 any of those things via a, after reading an 00:42:59.150 --> 00:43:01.840 article on our devcenter, go back and revisit that 00:43:01.840 --> 00:43:04.560 please, before upgrading to Rails 4.1. Some of the 00:43:04.560 --> 00:43:08.060 syntax did change between Rails 4.0 and 4.1. So, 00:43:08.060 --> 00:43:10.619 if you can't connect to a database, then maybe, 00:43:10.619 --> 00:43:12.930 just like, email Schneemz and be like, I hate 00:43:12.930 --> 00:43:14.460 you. What's the link to that thing? And I'll, 00:43:14.460 --> 00:43:16.480 I'll help you out. 00:43:16.480 --> 00:43:20.520 OK. I think, probably, actually, the last thing that 00:43:20.520 --> 00:43:25.110 we have time for, is asset pipeline. Who, like, 00:43:25.110 --> 00:43:29.040 if asked in an interview, would say that their 00:43:29.040 --> 00:43:31.510 favorite thing in the whole world is Rails asset 00:43:31.510 --> 00:43:35.460 pipeline? Oh. Oh. 00:43:35.460 --> 00:43:37.210 AUDIENCE: Just Raphael. 00:43:37.210 --> 00:43:38.880 R.S.: Just Raphael. We have a bunch of, like, 00:43:38.880 --> 00:43:41.510 Rails Core here, by the way. So you should, 00:43:41.510 --> 00:43:44.380 you should come and thank them afterwards. For, for 00:43:44.380 --> 00:43:46.260 other things. Not for the asset pipeline. 00:43:46.260 --> 00:43:47.140 [laughter] 00:43:47.140 --> 00:43:49.650 So, the asset pipeline is the number one source 00:43:49.650 --> 00:43:52.860 of, of Ruby support tickets at Heroku. Just people 00:43:52.860 --> 00:43:55.220 being like, hey, this worked locally, and like, didn't 00:43:55.220 --> 00:43:58.020 work in production. And we're like, yeah, that's just 00:43:58.020 --> 00:44:00.970 how asset pipeline works. That's not Heroku. 00:44:00.970 --> 00:44:04.800 So, so Rails 4.1 added, added a couple things. 00:44:04.800 --> 00:44:07.860 It's gonna warn you in development if you're doing 00:44:07.860 --> 00:44:10.550 something that's gonna break production. Like, if you've ever 00:44:10.550 --> 00:44:13.530 forgotten to add something to your precompile list, well 00:44:13.530 --> 00:44:16.600 now, guess what, you get an error. If you 00:44:16.600 --> 00:44:19.619 are not properly declaring your asset dependencies, then you're 00:44:19.619 --> 00:44:21.730 gonna get an error. 00:44:21.730 --> 00:44:26.440 And this is even better, actually, in Rails 4.2. 00:44:26.440 --> 00:44:28.260 As some of these checks aren't even needed anymore, 00:44:28.260 --> 00:44:30.490 we can just automatically do them for you. But, 00:44:30.490 --> 00:44:34.220 unfortunately, those have, are not in Rails 4.1 yet. 00:44:34.220 --> 00:44:37.340 So, in general, I have a, a personal belief 00:44:37.340 --> 00:44:40.200 that, in programming, or, really in life, the only 00:44:40.200 --> 00:44:47.200 thing that should fail silently is. This. This joke. 00:44:48.369 --> 00:44:54.369 So. Thank you all very much for, for coming. 00:44:54.369 --> 00:44:59.760 We, we have a booth, and later on, what. 00:44:59.760 --> 00:45:00.990 What time, three o' clock? 00:45:00.990 --> 00:45:02.020 T.L.: Between 3:00 and 4:30. 00:45:02.020 --> 00:45:03.480 R.S.: Yeah. From 3:00 to 4:30, we'll actually have 00:45:03.480 --> 00:45:09.510 a bunch of Rails contributors coming to, to talk 00:45:09.510 --> 00:45:14.330 about. Oh yeah, the slides. Yeah. Yeah. 00:45:14.330 --> 00:45:16.750 T.L.: Yeah. 3:00 to 4:30, we'll have community office 00:45:16.750 --> 00:45:20.720 hours with some nice people from Rails Core, contrib. 00:45:20.720 --> 00:45:22.869 R.S.: Yeah. So come ask. 00:45:22.869 --> 00:45:26.590 T.L.: Basically any Rails questions or anything you want. 00:45:26.590 --> 00:45:28.010 And then Schneeman will actually be doing a book 00:45:28.010 --> 00:45:30.990 signing of his Heroku Up & Running book today 00:45:30.990 --> 00:45:33.850 and tomorrow at 2:30. So if you want that. 00:45:33.850 --> 00:45:36.010 R.S.: Yeah. So get a, get a free book, 00:45:36.010 --> 00:45:38.160 and then come and ask questions and just, like, 00:45:38.160 --> 00:45:41.580 hang out. And, any time you stop by the 00:45:41.580 --> 00:45:44.160 booth, feel free to ask Heroku questions. And thank 00:45:44.160 --> 00:45:45.880 you all very much for coming.