0:00:17.300,0:00:19.960
AARON SUGGS: All right. Can people hear OK?

0:00:19.970,0:00:21.369
I'll go ahead and get started.

0:00:21.369,0:00:24.050
So this talk is Rack::Attack and

0:00:24.050,0:00:27.380
how to protect your app with this one weird[br]gem.

0:00:27.380,0:00:30.930
Where does Rack::Attack come from? We built[br]it at

0:00:30.930,0:00:34.400
KickStarter. If you haven't heard of KickStarter,[br]it is

0:00:34.400,0:00:37.610
a funding platform for creative projects.[br]So somebody has

0:00:37.610,0:00:40.690
an idea for a film, a comic book, an

0:00:40.690,0:00:43.829
open source project, a gadget. They, they[br]put their

0:00:43.829,0:00:46.720
project up on our site. They can offer rewards

0:00:46.720,0:00:50.430
for various pledge levels. Their friends,[br]family, strangers on

0:00:50.430,0:00:53.370
the internet come and can, can give them money.

0:00:53.370,0:00:56.050
At the end of the deadline, if they've reached

0:00:56.050,0:00:57.550
their funding goal and so they have enough[br]to

0:00:57.550,0:01:00.750
reach their project, that's when we process[br]the transactions

0:01:00.750,0:01:02.800
and the creators' get the funds they need[br]to,

0:01:02.800,0:01:03.980
to do the project.

0:01:03.980,0:01:06.939
To give you a sense of scale for what

0:01:06.939,0:01:09.000
we do, we, we recently crossed over a billion

0:01:09.000,0:01:11.530
dollars pledged to the site. It's over a million

0:01:11.530,0:01:14.960
dollars a day. And it's gone to over 60,000

0:01:14.960,0:01:17.079
creative projects.

0:01:17.079,0:01:21.759
Quick introduction. My name's Aaron Suggs.[br]I go by

0:01:21.759,0:01:24.909
ktheory on social media. I love dancing in[br]my

0:01:24.909,0:01:29.960
bear outfit. And I'm the operations engineer[br]at KickStarter.

0:01:29.960,0:01:33.409
We, we have a very dev ops-y style workflow.

0:01:33.409,0:01:35.450
So, so it means I end up writing a

0:01:35.450,0:01:36.880
lot of Ruby code, and I love writing Ruby

0:01:36.880,0:01:37.880
code.

0:01:37.880,0:01:41.979
So, so Rack::Attack is, is a tool I wrote,

0:01:41.979,0:01:46.259
and it's Rack middleware for blocking and[br]throttling abusive

0:01:46.259,0:01:49.329
requests. What do we mean by abusive requests?[br]These

0:01:49.329,0:01:52.159
can be things like malicious attackers trying[br]to take

0:01:52.159,0:01:55.369
down your site, doing things like trying to[br]crack

0:01:55.369,0:01:58.509
user accounts or get sensitive information,[br]or it can

0:01:58.509,0:02:02.049
be naively written scrapers, who are just,[br]like, people

0:02:02.049,0:02:04.600
on the internet doing weird things as they[br]are

0:02:04.600,0:02:07.909
prone to do, and that's cool, but sometimes[br]it,

0:02:07.909,0:02:09.788
it is a lot of traffic. It's a lot

0:02:09.788,0:02:12.300
of resources for your app to try to handle,

0:02:12.300,0:02:16.310
and Rack::Attack is a very elegant DSL and,[br]and

0:02:16.310,0:02:18.560
way for dealing with these sorts of things.[br]Sort

0:02:18.560,0:02:22.190
of constraining their behavior so your website[br]stays up.

0:02:22.190,0:02:27.020
Rack::Attack is on GitHub at slash kickstarter[br]slash rack-attack.

0:02:27.020,0:02:30.160
It's an open source Ruby gem. There's a README,

0:02:30.160,0:02:34.050
sort of exactly like what you'd expect.

0:02:34.050,0:02:36.860
So the big wins that KickStarter has gotten[br]from

0:02:36.860,0:02:40.250
using Rack::Attack, and the reason we developed[br]it, was

0:02:40.250,0:02:42.950
we wanted to increase our performance. So,[br]so this

0:02:42.950,0:02:46.390
is like site performance. We, we had problems[br]with

0:02:46.390,0:02:49.720
sort of abusive requests making our website[br]slow because

0:02:49.720,0:02:52.530
they were using up too many app servers CP.

0:02:52.530,0:02:54.500
Too much app server CPU, or too much, too

0:02:54.500,0:02:58.140
many database resources, by sort of constraining[br]them we

0:02:58.140,0:03:00.610
were able to make the website faster for the

0:03:00.610,0:03:03.450
sort of, the most important requests. Like[br]people coming

0:03:03.450,0:03:06.160
on, wanting to watch videos, wanting to pledge[br]money.

0:03:06.160,0:03:07.660
Not people just trying to scrape down the[br]entire

0:03:07.660,0:03:08.270
site.

0:03:08.270,0:03:11.940
We also improved our available. Because sometimes[br]these requests

0:03:11.940,0:03:14.460
were, were so much, there were so many that

0:03:14.460,0:03:15.920
they would take down the site, or there would

0:03:15.920,0:03:19.860
just be some weird incident and, we, right.[br]It,

0:03:19.860,0:03:22.540
it hurt our availability.

0:03:22.540,0:03:25.840
But the biggest win that we had was developer

0:03:25.840,0:03:30.370
happiness. Because dealing with these sort[br]of bad actors

0:03:30.370,0:03:33.940
on the internet especially if it means, like,[br]your,

0:03:33.940,0:03:35.970
your site's going down or like, the, you know,

0:03:35.970,0:03:38.960
you need to scale up because somebody's doing[br]something

0:03:38.960,0:03:41.750
weird, that can really interrupt a lot of[br]developers.

0:03:41.750,0:03:44.230
It can, it can sort of derail your product

0:03:44.230,0:03:47.320
road map. We want to be writing cool features

0:03:47.320,0:03:50.190
and Rack::Attack was a great DSL to let us

0:03:50.190,0:03:53.330
spend less time thinking about that stuff[br]and more

0:03:53.330,0:03:54.850
stuff doing the stuff that we, that we like

0:03:54.850,0:03:55.430
doing.

0:03:55.430,0:03:58.960
So let me talk about the origin story for

0:03:58.960,0:04:01.260
Rack::Attack. Like, what happened at KickStarter[br]that made us

0:04:01.260,0:04:05.060
realize we, we needed this? Let's rewind to[br]the

0:04:05.060,0:04:10.790
summer of 2012.

0:04:10.790,0:04:12.510
And this happened. So this is a story in

0:04:12.510,0:04:15.930
a graph. So the blue line, I hope it

0:04:15.930,0:04:19.850
shows up pretty well. Cool. Is our regular[br]successful

0:04:19.850,0:04:22.000
logins. People typing in an email and password[br]and

0:04:22.000,0:04:24.919
us being like, OK, you are logged in. You

0:04:24.919,0:04:26.830
know, it ebs and flows throughout the day.

0:04:26.830,0:04:30.620
Suddenly, one Sun, one Saturday afternoon,[br]we just get

0:04:30.620,0:04:34.460
so many of these, like, bad login requests,[br]and

0:04:34.460,0:04:36.500
for awhile we're like, what's going on? Did[br]we

0:04:36.500,0:04:39.220
deploy a feature that broke login? No. Somebody[br]is

0:04:39.220,0:04:42.090
trying to, to crack our user accounts. They're[br]just

0:04:42.090,0:04:45.470
like guessing email addresses and passwords[br]as fast as

0:04:45.470,0:04:48.850
they can, from several different IP addresses.

0:04:48.850,0:04:51.810
So, as the ops guy, this is sort of

0:04:51.810,0:04:54.000
on my plate. I'm like, OK, well, I gotta

0:04:54.000,0:04:55.560
stop this. This is bad for the site for

0:04:55.560,0:04:58.650
this to be going on. So I wrote a

0:04:58.650,0:05:01.740
pretty nasty before filter for our login action,[br]that's

0:05:01.740,0:05:04.560
like, you know, keep a counter in memcache[br]and,

0:05:04.560,0:05:07.320
you know, if it's too many like, like, give

0:05:07.320,0:05:11.480
them an error page and it was, it was

0:05:11.480,0:05:14.630
kind of a sucky experience, because I was[br]changing

0:05:14.630,0:05:17.480
a really critical feature of our site, sort[br]of

0:05:17.480,0:05:20.120
under duress of, of knowing that I needed[br]to

0:05:20.120,0:05:22.980
get it out there quickly. And it was sort

0:05:22.980,0:05:24.440
of like a big change, and in the pull

0:05:24.440,0:05:26.620
request I was, I was apologetic, being like,[br]I

0:05:26.620,0:05:28.220
know this is badly tested and it's like a

0:05:28.220,0:05:29.850
nasty code change, but we've got to get it

0:05:29.850,0:05:33.700
out fast because this, this event's going[br]on.

0:05:33.700,0:05:36.930
And, so that, so we did that. And then

0:05:36.930,0:05:39.680
sort of in the cold light of day, I

0:05:39.680,0:05:42.050
reflected a little bit and I thought, we need

0:05:42.050,0:05:47.440
a more elegant way to prevent bad requests.[br]This

0:05:47.440,0:05:50.280
is, it's not just gonna be about this login

0:05:50.280,0:05:51.669
attack. This is gonna be about a whole class

0:05:51.669,0:05:54.250
of problems that we might have on the site.

0:05:54.250,0:05:57.639
You know, I should say, too, with that login

0:05:57.639,0:06:00.310
attack, it was something that we sort of always

0:06:00.310,0:06:02.510
imagined that, like, oh yeah, of course we[br]should,

0:06:02.510,0:06:05.020
like, throttle login requests. We just hadn't[br]ever gotten

0:06:05.020,0:06:06.669
around to it. You know, it was in our

0:06:06.669,0:06:09.770
ticketing system as like a low-priority someday[br]somebody should

0:06:09.770,0:06:13.300
do this thing. And having it actually happen[br]was

0:06:13.300,0:06:15.460
like, OK, now we gotta do it right now.

0:06:15.460,0:06:18.570
So, we realized, like, we need this generic[br]tool

0:06:18.570,0:06:23.810
to stop bad requests. And really, there's[br]already, in

0:06:23.810,0:06:25.810
the Ruby world, a great solution for this,[br]and

0:06:25.810,0:06:29.340
it's Rack middleware. So now we get to the

0:06:29.340,0:06:32.020
code section of the talk. Here comes some[br]code.

0:06:32.020,0:06:33.620
Get ready.

0:06:33.620,0:06:35.840
This is an example of, like, the most basic

0:06:35.840,0:06:38.190
Rack middleware. Just, really quick, for,[br]for people who

0:06:38.190,0:06:41.389
might not be familiar with it. So middleware[br]is

0:06:41.389,0:06:45.650
basically like hugging your application, wrapping[br]around so, so

0:06:45.650,0:06:47.430
you, you have your Rails app or your Sinatra

0:06:47.430,0:06:52.350
app, that is the app in this case. And

0:06:52.350,0:06:53.940
you want to do things, you want to sort

0:06:53.940,0:06:56.440
of be able to do things to the request

0:06:56.440,0:06:58.590
that's coming in from the client. That's the[br]end.

0:06:58.590,0:07:01.560
So every, every request from a client is gonna

0:07:01.560,0:07:03.260
do this call method where you pass in the

0:07:03.260,0:07:06.050
environment, the environment is, like, I don't[br]know, what

0:07:06.050,0:07:09.080
page the client wants or what they're cookie[br]is

0:07:09.080,0:07:11.710
and, and all that information.

0:07:11.710,0:07:14.460
And so the real magic of Rack middleware is

0:07:14.460,0:07:16.740
it lets you do stuff here with, with the

0:07:16.740,0:07:19.010
requests. Like, you can block it in the case

0:07:19.010,0:07:23.040
of Rack::Attack, potentially. Or you can do[br]stuff with

0:07:23.040,0:07:26.340
the response. You can log it. You can cache

0:07:26.340,0:07:27.570
it. Stuff like that.

0:07:27.570,0:07:29.020
So this, so this is just a great pattern

0:07:29.020,0:07:34.100
for managing, for sort of making easy architectures[br]to

0:07:34.100,0:07:38.630
do stuff with HTTP requests. So in Rack::Attack's[br]case,

0:07:38.630,0:07:40.880
this is a sort of simplified version of the

0:07:40.880,0:07:45.440
Rack::Attack call method. We say, for this[br]request, should

0:07:45.440,0:07:47.889
we allow it? If so, go ahead and pass

0:07:47.889,0:07:51.889
it onto your application. Your application[br]is gonna do,

0:07:51.889,0:07:53.330
potentially, a lot of work.

0:07:53.330,0:07:55.960
Maybe it's gonna spend a couple hundred milliseconds,[br]like,

0:07:55.960,0:07:59.300
querying the database and rendering views[br]and stuff like

0:07:59.300,0:08:01.910
that. So that's the expensive work that we[br]want

0:08:01.910,0:08:05.150
to save if the, if this is an abusive

0:08:05.150,0:08:07.580
request. So, so if we shouldn't allow it,[br]then

0:08:07.580,0:08:11.490
we just return back this very fast access-denied[br]as

0:08:11.490,0:08:14.570
a very simple and fast response to render.

0:08:14.570,0:08:18.500
Rack::Attack can do several hundred of these[br]access denied

0:08:18.500,0:08:21.870
requests per, like, thread that you have running.[br]So

0:08:21.870,0:08:25.130
like, per unicorn worker or per Heroku instance[br]or

0:08:25.130,0:08:26.580
something like that.

0:08:26.580,0:08:29.870
But, so, that's what you get for, when you

0:08:29.870,0:08:32.399
just use the Rack middleware for free. So,[br]so

0:08:32.399,0:08:34.828
we don't yet know what this should_allow method[br]should

0:08:34.828,0:08:36.419
be. That's code that you sort of have to

0:08:36.419,0:08:39.299
configure yourself, of what do you want to[br]throttle

0:08:39.299,0:08:40.000
on.

0:08:40.000,0:08:43.229
So that looks like this. This is sort of

0:08:43.229,0:08:46.070
a generic throttle that you might put in your,

0:08:46.070,0:08:51.430
in an initializer to configure Rack::Attack.[br]The important stuff

0:08:51.430,0:08:53.300
that's going on here is we are calling the

0:08:53.300,0:08:57.089
throttle class method on Rack::Attack, so[br]that's just something

0:08:57.089,0:09:00.200
we expose to let you plug into the middleware.

0:09:00.200,0:09:02.149
We give it a name, in this case it's

0:09:02.149,0:09:04.950
the, we, we named the throttle IP. This is

0:09:04.950,0:09:08.080
gonna determine how we track it. And that[br]just

0:09:08.080,0:09:11.210
has to be unique throughout your application.[br]We're gonna

0:09:11.210,0:09:13.140
give it a limit and a period. And so

0:09:13.140,0:09:15.649
that's how much, the, the period is how many

0:09:15.649,0:09:18.480
seconds we're gonna be considering for the[br]throttle, and

0:09:18.480,0:09:20.390
the limit is sort of your quota for how

0:09:20.390,0:09:23.180
many requests you get to make during that[br]time.

0:09:23.180,0:09:25.320
So in this case, it's ten requests every five

0:09:25.320,0:09:30.920
seconds. For the arithmetically inclined,[br]you'll notice that this

0:09:30.920,0:09:33.630
is not like a reduced fraction. We could say

0:09:33.630,0:09:37.040
two requests every one second. The advantage[br]of doing

0:09:37.040,0:09:38.790
a higher multiple is that, like, it allows[br]a

0:09:38.790,0:09:42.649
little burstiness. So these periods are basically[br]dividing time

0:09:42.649,0:09:46.210
up into these, like, five second long buckets.[br]So

0:09:46.210,0:09:49.399
in between zero and, seconds and five seconds[br]after

0:09:49.399,0:09:51.940
the minute, like, in that window, you're allowed[br]to

0:09:51.940,0:09:54.200
make up to ten requests.

0:09:54.200,0:09:57.660
And so by having bigger multiples in bigger[br]windows,

0:09:57.660,0:10:00.839
you can sort of get around some burstiness[br]at,

0:10:00.839,0:10:03.740
but the long-term average stays the same.[br]Like, long

0:10:03.740,0:10:07.300
term, nobody's gonna make more requests that[br]two every

0:10:07.300,0:10:09.240
one second.

0:10:09.240,0:10:12.140
OK, so what's going on? We got the, the

0:10:12.140,0:10:14.690
class method. We got the name. WE have the

0:10:14.690,0:10:17.390
limit and the period. And then to this block,

0:10:17.390,0:10:20.730
we are passing along the request. Now, in[br]the

0:10:20.730,0:10:23.320
earlier middleware expample we talked, we[br]called this the

0:10:23.320,0:10:25.830
end, which was just like the, the environment[br]hash

0:10:25.830,0:10:29.240
that comes from the request. Request is just[br]like

0:10:29.240,0:10:33.740
a light little Rack request object wrapped[br]around the

0:10:33.740,0:10:36.770
environment that just sort of gives you methods,[br]instance

0:10:36.770,0:10:39.510
methods to call, like dot IP or dot host

0:10:39.510,0:10:41.170
or dot path or something like that. It just

0:10:41.170,0:10:46.149
sort of, you use these in Rails controllers,[br]too.

0:10:46.149,0:10:49.820
So it's just like a lightly-wrapped request.[br]And then

0:10:49.820,0:10:52.100
inside the block, what the block returns is[br]the

0:10:52.100,0:10:54.640
sort of really important part. That's the[br]discriminator that

0:10:54.640,0:10:58.140
determines how we're gonna bucket up these[br]throttles. So

0:10:58.140,0:11:00.930
in this case we are gonna say every IP

0:11:00.930,0:11:03.649
address, every distinct IP address is going[br]to get

0:11:03.649,0:11:06.830
its own throttle limit. But we could throttle[br]by

0:11:06.830,0:11:09.620
something else. WE could throttle by a parameter[br]or

0:11:09.620,0:11:14.459
a host name or something like that, or an

0:11:14.459,0:11:16.130
API token.

0:11:16.130,0:11:18.490
And one thing to note with these discriminators,[br]too,

0:11:18.490,0:11:21.459
is like, if this would, this is returning[br]a

0:11:21.459,0:11:24.050
string, so it's always gonna be a truthy value,

0:11:24.050,0:11:26.700
and true values sort of enable the, the throttling.

0:11:26.700,0:11:29.029
Like, we are gonna throttle these requests[br]as long

0:11:29.029,0:11:32.190
as there's an IP address, and there always[br]is.

0:11:32.190,0:11:34.670
If we would return nil or a falsey value,

0:11:34.670,0:11:36.540
we just sort of let the request go through

0:11:36.540,0:11:38.740
and we're not gonna throttle it. I'll talk[br]about

0:11:38.740,0:11:41.790
why we might want to do that later. But,

0:11:41.790,0:11:43.709
so now we have this issue of throttle state.

0:11:43.709,0:11:46.420
Like, we have these counters per IP address[br]that

0:11:46.420,0:11:48.320
we need to track.

0:11:48.320,0:11:50.560
And so, so where do we store that? A

0:11:50.560,0:11:53.120
pretty elegant and simple and obvious place[br]for that

0:11:53.120,0:11:56.880
was our Rails cache. So when you just use

0:11:56.880,0:11:59.100
Rack::Attack by default, if you have a Rails[br]cache,

0:11:59.100,0:12:02.450
it's gonna use it. But, it really works best

0:12:02.450,0:12:05.779
with memcache or redis. So, so I hope you're

0:12:05.779,0:12:09.050
using that as your Rails cache. But if you're

0:12:09.050,0:12:10.890
not, like, there are ways that you can build

0:12:10.890,0:12:12.510
your own, or sort of like plug in a,

0:12:12.510,0:12:14.670
a different cache store.

0:12:14.670,0:12:17.070
The great advantage about memcache and redis[br]is that

0:12:17.070,0:12:21.209
they have really good support for atomically[br]incrementing counters,

0:12:21.209,0:12:22.720
and that's the sort of key feature we'd need

0:12:22.720,0:12:26.269
behind the scenes. So now we're imagining[br]for, for

0:12:26.269,0:12:28.209
every request that comes in, we need to sort

0:12:28.209,0:12:31.050
of increment the counter per IP address.

0:12:31.050,0:12:32.370
And so how do we do that? Like what's,

0:12:32.370,0:12:35.250
what's the algorithm? So this is the nitty[br]gritty

0:12:35.250,0:12:40.100
of how Rack::Attack works. How it constructs[br]that key.

0:12:40.100,0:12:43.060
So remember how we divided the minute up into

0:12:43.060,0:12:46.810
like little buckets depending on our period.[br]So, so

0:12:46.810,0:12:48.790
to do that, we sort of take the current

0:12:48.790,0:12:53.620
second. We construct a key that is the name

0:12:53.620,0:12:57.380
of our request, like IP in this case. We

0:12:57.380,0:12:59.490
take the time divided by the period, so this

0:12:59.490,0:13:02.660
means that that middle component is going[br]to be,

0:13:02.660,0:13:05.350
is going to increment every five seconds.[br]It's gonna,

0:13:05.350,0:13:07.570
so it's, the key's gonna change.

0:13:07.570,0:13:09.360
And then the final part is that block return

0:13:09.360,0:13:12.170
value. So in this case it's the IP address

0:13:12.170,0:13:15.149
of the request. But maybe it's an API token

0:13:15.149,0:13:17.170
or something like that.

0:13:17.170,0:13:18.720
So at the end of it, we have this

0:13:18.720,0:13:21.990
key that changes every couple seconds. Every[br]time, like,

0:13:21.990,0:13:24.720
the period rotates, and this ends up being[br]a

0:13:24.720,0:13:27.450
very efficient use case, a very efficient[br]use of

0:13:27.450,0:13:30.880
memcache or redis. Like, this is, storing[br]all this

0:13:30.880,0:13:34.010
information is gonna take, like, a couple[br]megabytes. It's

0:13:34.010,0:13:36.000
like, don't worry about the impact on your[br]cache

0:13:36.000,0:13:39.100
store in pretty much every scenario.

0:13:39.100,0:13:41.339
To make it even more efficient use of your

0:13:41.339,0:13:45.910
cache store, we set an expire rate, so that

0:13:45.910,0:13:48.110
in that, like, in that bucket window of, say,

0:13:48.110,0:13:50.420
zero to five seconds, we're gonna say that[br]all

0:13:50.420,0:13:53.100
those cache keys expire at five seconds. So[br]at

0:13:53.100,0:13:56.990
the same moment that the cache keys change,[br]they

0:13:56.990,0:13:59.680
also expire. So memcache or redis just ends[br]up

0:13:59.680,0:14:03.200
reusing the same memory blocks over and over.[br]You

0:14:03.200,0:14:06.339
don't have, even though there's changing,[br]they're changing in

0:14:06.339,0:14:08.399
memory, you don't have as much churn as you

0:14:08.399,0:14:11.300
would otherwise.

0:14:11.300,0:14:13.890
And so then the Rack middleware is really[br]doing

0:14:13.890,0:14:16.310
pretty simple stuff of we're saying, for whatever[br]your

0:14:16.310,0:14:19.870
cache is, increment this key with this expire[br]rate.

0:14:19.870,0:14:21.459
That's gonna give us back the count of how

0:14:21.459,0:14:23.610
many requests that have been made that, that[br]match

0:14:23.610,0:14:26.610
that throttle. And if it's more than our limit,

0:14:26.610,0:14:29.680
we're gonna return that access denied response.

0:14:29.680,0:14:32.750
So, we rolled this out. You know, we're able

0:14:32.750,0:14:36.920
to have this global throttle per IP address.[br]We

0:14:36.920,0:14:40.779
start making a couple other, other features,[br]and it

0:14:40.779,0:14:44.029
was about a year later when we had a,

0:14:44.029,0:14:46.930
the sort of redux of, of a new event

0:14:46.930,0:14:48.980
that put Rack::Attack to the test.

0:14:48.980,0:14:52.040
So, a new challenger emerges in the summer[br]of

0:14:52.040,0:14:57.899
2013. This was a script called kicksniper[br]dot py.

0:14:57.899,0:15:02.079
And this revealed a pretty interesting behavior[br]on KickStarter

0:15:02.079,0:15:05.290
that we call reward sniping. Actually, kicksniper[br]dot py

0:15:05.290,0:15:08.790
refers to it in the code as reward sniping.

0:15:08.790,0:15:12.110
And so, this is, this is an, an interesting

0:15:12.110,0:15:14.920
behavior because. So I told you how KickStarter[br]offers

0:15:14.920,0:15:17.800
these rewards. They can be limited rewards.[br]So a

0:15:17.800,0:15:20.170
creator says, I'm only gonna give away, like,[br]a

0:15:20.170,0:15:24.120
hundred of these, and first come, first serve.

0:15:24.120,0:15:26.980
So, there's a, a pretty popular project where[br]it

0:15:26.980,0:15:28.700
was like a video game and, and the video

0:15:28.700,0:15:31.750
game was offering these reward tiers that[br]would be,

0:15:31.750,0:15:33.980
like, for fifty bucks, you get, like, the[br]silver

0:15:33.980,0:15:35.790
level package, and for a hundred bucks you[br]get

0:15:35.790,0:15:37.600
the gold package, and so on and so, like,

0:15:37.600,0:15:41.459
ever more deluxe and expensive packages. And[br]they were

0:15:41.459,0:15:42.720
all very much in demand.

0:15:42.720,0:15:47.230
So the early reward tiers like sold-out super[br]fast.

0:15:47.230,0:15:50.089
And then occasionally, somebody in, who had[br]those early

0:15:50.089,0:15:53.070
reward tiers, would decide they're gonna splurge[br]and they're

0:15:53.070,0:15:54.870
gonna upgrade. They're gonna change their[br]pledge to a

0:15:54.870,0:15:58.600
higher one, and now for that moment, like,[br]there's

0:15:58.600,0:16:01.430
now one available of the lower tier. And so

0:16:01.430,0:16:04.980
people were like hitting refresh, refresh,[br]refresh, hoping that

0:16:04.980,0:16:07.769
they just noticed when somebody, when somebody[br]had changed

0:16:07.769,0:16:09.350
their pledge and now there was one of these

0:16:09.350,0:16:12.410
highly desirable lower-tier pledges available.

0:16:12.410,0:16:18.420
Some entrepreneur, enterprising Python developer,[br]says, I will make

0:16:18.420,0:16:21.959
a script that does this for me. Sure enough,

0:16:21.959,0:16:24.940
so, so he writes kicksniper dot py that's,[br]that's

0:16:24.940,0:16:27.019
in a tight loop, trying to change his pledge

0:16:27.019,0:16:29.089
on our site. Saying, like, let me get that,

0:16:29.089,0:16:32.260
that early reward tier. You know, our ActiveRecord[br]validations

0:16:32.260,0:16:34.050
were working fine and we said, no, you can't

0:16:34.050,0:16:36.470
change your pledge to that the vast majority[br]of

0:16:36.470,0:16:39.730
the time, but, but eventually he got through[br]and

0:16:39.730,0:16:41.100
was able to get the pledge.

0:16:41.100,0:16:43.089
It was such a great success that he goes

0:16:43.089,0:16:45.889
on all the forums and says, hey, everybody[br]just

0:16:45.889,0:16:50.740
run this, like, Python script on your laptop[br]and

0:16:50.740,0:16:53.300
you, too, might look, luck out and get one

0:16:53.300,0:16:56.220
of these highly desirable earlier reward tiers.

0:16:56.220,0:17:01.360
So let's tell this story in a graph. So,

0:17:01.360,0:17:04.089
this is our master database CPU over the course

0:17:04.089,0:17:05.929
of a, of a day or so. We see

0:17:05.929,0:17:07.919
at the very beginning, it starts off between[br]ten

0:17:07.919,0:17:10.760
or fifteen percent. That's my happy place.[br]That's where

0:17:10.760,0:17:12.039
I like it to be. We have plenty of

0:17:12.039,0:17:15.148
head room for like, you know, big projects[br]to

0:17:15.148,0:17:16.949
sort of blow up on the site, as they

0:17:16.949,0:17:18.980
do from time to time.

0:17:18.980,0:17:21.490
And, I honestly didn't really notice that[br]it had

0:17:21.490,0:17:24.339
been creeping up over the course of the day.

0:17:24.339,0:17:28.000
Thursday morning, it crossed thirty percent,[br]and that's when

0:17:28.000,0:17:30.940
I get a CPU alert threshold. So it, so

0:17:30.940,0:17:32.590
in fact, the whole dev team gets this email

0:17:32.590,0:17:34.910
being like, hey, the master database CPU is[br]pretty

0:17:34.910,0:17:37.260
high. You guys should check that out.

0:17:37.260,0:17:41.320
So, what do we, you know, we, we spend

0:17:41.320,0:17:43.070
a little time, we're like, why is the database

0:17:43.070,0:17:44.870
so high? Well, you know, it looks like there

0:17:44.870,0:17:46.870
are a crazy number of requests trying to change

0:17:46.870,0:17:49.600
their pledge for this one project.

0:17:49.600,0:17:52.650
We, we're able to sort of construct this back

0:17:52.650,0:17:54.530
story and, like, see what was happening on[br]the

0:17:54.530,0:17:57.090
database CPU. We see the form request where[br]everybody's

0:17:57.090,0:18:02.280
like, thank you for kicksniper dot py. And[br]so,

0:18:02.280,0:18:03.919
and we're like, all right, so, so how are

0:18:03.919,0:18:06.390
we gonna handle this? Like, is it really that

0:18:06.390,0:18:08.990
important that people are able to try to change

0:18:08.990,0:18:11.400
their pledge like multiple times a second?

0:18:11.400,0:18:13.500
What if they only could change their pledge[br]every

0:18:13.500,0:18:17.049
couple seconds? Right, like, I guess that's[br]fair enough

0:18:17.049,0:18:19.130
to the, like, there's this question of, like,[br]what's

0:18:19.130,0:18:22.030
the fairest way to allocate the scarce resources[br]of,

0:18:22.030,0:18:24.500
of like the pledge as soon as it's available.

0:18:24.500,0:18:26.720
I kind of don't care about the answer. Anybody

0:18:26.720,0:18:28.450
can get it.

0:18:28.450,0:18:32.410
But, but we're like, if we start throttling[br]these

0:18:32.410,0:18:34.600
people, it's like totally fair. They're using[br]an inordinate

0:18:34.600,0:18:37.540
number of resources. And people who are just[br]clicking

0:18:37.540,0:18:40.040
around the site are having a slower experience[br]because

0:18:40.040,0:18:42.400
our database CPU is so high.

0:18:42.400,0:18:44.020
So we decide, like, OK, you can make a

0:18:44.020,0:18:46.250
couple requests per minute to change a pledge.[br]It

0:18:46.250,0:18:49.860
was one line of Rack::Attack code. We deploy[br]it.

0:18:49.860,0:18:52.190
The yellow vertical lines here are deploy[br]lines, so

0:18:52.190,0:18:55.210
you can see that right here, about an hour

0:18:55.210,0:18:57.250
after we get the alert that something was[br]going

0:18:57.250,0:19:01.710
wrong, we deploy and immediately our database[br]CPU drops.

0:19:01.710,0:19:04.780
We're pretty much back to the happy place.

0:19:04.780,0:19:07.640
And so, for us, that was like, revealing the,

0:19:07.640,0:19:09.240
the great success that we could have. Like,[br]it

0:19:09.240,0:19:12.000
was so easy, like, once we figured out what

0:19:12.000,0:19:14.470
was going on, it was so easy for us

0:19:14.470,0:19:17.100
to write code that just, like, solved that[br]problem.

0:19:17.100,0:19:19.030
We didn't have to think about, like, how do

0:19:19.030,0:19:23.330
we optimize the edit pledge flow? Which could[br]have

0:19:23.330,0:19:26.260
been, like, a much bigger product change,[br]and derail,

0:19:26.260,0:19:28.090
like, taken up a lot more developer time.[br]It

0:19:28.090,0:19:30.320
was sort of a cut and dry decision of

0:19:30.320,0:19:32.799
like, most people aren't gonna try to change[br]their

0:19:32.799,0:19:35.020
pledge, like, we're super confused if you're[br]actually trying

0:19:35.020,0:19:36.910
to change your pledge several times a minute.

0:19:36.910,0:19:39.290
That's a, that's a bug we should fix. But

0:19:39.290,0:19:41.360
it's really just these scrapers. It's not[br]big deal

0:19:41.360,0:19:43.040
to say they can try a few times a

0:19:43.040,0:19:43.450
minute.

0:19:43.450,0:19:47.470
So, that was a big win for Rack::Attack at

0:19:47.470,0:19:50.110
KickStarter. We feel like we sort of, we sort

0:19:50.110,0:19:54.230
of cemented that its value in the organization.[br]So

0:19:54.230,0:19:56.250
now I'm gonna shift gears a little bit and

0:19:56.250,0:19:59.500
I'm gonna tell you pro tips of general things

0:19:59.500,0:20:01.820
you can do with Rack::Attack that, that are[br]probably

0:20:01.820,0:20:03.210
useful for your application.

0:20:03.210,0:20:07.040
I just, oh my gosh I'm so glad that

0:20:07.040,0:20:08.410
I got to use this gif. This gif is

0:20:08.410,0:20:12.580
like condensed, pure condensed happiness for[br]me. OK. Back

0:20:12.580,0:20:14.320
to the code.

0:20:14.320,0:20:16.900
So, we talked about how to do, like, a

0:20:16.900,0:20:20.130
general, a, a log, I'm sorry. We talked about

0:20:20.130,0:20:24.179
how to do a throttle for all IP addresses.

0:20:24.179,0:20:25.770
So like each IP has this quota of how

0:20:25.770,0:20:28.710
many requests you can do. But, in our, in

0:20:28.710,0:20:32.500
our origin story about the login attack, we[br]wanted

0:20:32.500,0:20:35.380
to be extra careful about login requests.[br]Like, those

0:20:35.380,0:20:38.169
are something that, that you would want to[br]throttle

0:20:38.169,0:20:40.880
even more strictly than you would throttle[br]many other

0:20:40.880,0:20:43.730
things on your, in your application.

0:20:43.730,0:20:46.860
So this is a new throttle, and so we

0:20:46.860,0:20:49.590
give it a new name of logins per IP.

0:20:49.590,0:20:52.020
And this is saying that if you are making

0:20:52.020,0:20:55.530
a post request to the login url, then we

0:20:55.530,0:20:57.660
want to throttle you by IP to like this

0:20:57.660,0:21:01.600
much, this lower limit. And so this is relying

0:21:01.600,0:21:04.520
on the fact that we mentioned earlier, that[br]if

0:21:04.520,0:21:06.830
the block returns nil, we're not gonna do[br]throttle

0:21:06.830,0:21:08.760
at all. So, so if this is not a

0:21:08.760,0:21:11.740
post to the login action, like, we're not[br]gonna

0:21:11.740,0:21:14.200
check memcache, we're not gonna increment[br]any counters or

0:21:14.200,0:21:15.730
do anything like that. We're just gonna sort[br]of

0:21:15.730,0:21:18.590
allow this request right through.

0:21:18.590,0:21:20.400
But if it is, we're gonna hold you, we're

0:21:20.400,0:21:22.830
gonna say each IP address gets this lower[br]quota

0:21:22.830,0:21:24.650
of how many login requests they can make.

0:21:24.650,0:21:27.190
Thinking of this same problem from a, from[br]a

0:21:27.190,0:21:29.910
kind of different angle, you might want to[br]imagine

0:21:29.910,0:21:32.250
a, a situation where a, an attacker is using

0:21:32.250,0:21:35.580
many different IP addresses to try to crack[br]passwords

0:21:35.580,0:21:39.020
for one particular email address, right. Maybe[br]it's the

0:21:39.020,0:21:41.710
founder's email address or something like[br]that.

0:21:41.710,0:21:44.289
So you, so putting on your security hat, you

0:21:44.289,0:21:45.630
can be like, how am I gonna be safe

0:21:45.630,0:21:48.610
from those kinds of requests? The only change[br]here

0:21:48.610,0:21:51.660
is what we're returning. Instead of the IP[br]address,

0:21:51.660,0:21:54.789
we're returning the value of the email parameter.[br]So

0:21:54.789,0:21:57.780
this is a, a sort of little different way

0:21:57.780,0:22:00.890
of thinking about throttles, of saying, whoever[br]you are,

0:22:00.890,0:22:02.789
if you're trying to login with this one particular

0:22:02.789,0:22:05.809
IP address, you can only do it five times

0:22:05.809,0:22:07.900
every twenty seconds.

0:22:07.900,0:22:11.370
So those are two throttles that pretty much[br]everybody

0:22:11.370,0:22:13.610
should, should have that feature on their[br]website. If

0:22:13.610,0:22:15.960
you haven't been bitten by it yet, it's probably

0:22:15.960,0:22:19.280
just a matter of time.

0:22:19.280,0:22:22.360
Another pretty cool Rack::Attack feature are[br]blacklists. So these

0:22:22.360,0:22:24.059
are requests that you don't even want to throttle.

0:22:24.059,0:22:27.169
Like, you're not gonna allow them at all.[br]Just,

0:22:27.169,0:22:30.309
access denied every time they happen. I kind,[br]I

0:22:30.309,0:22:33.140
was gonna call these blocks, but like blocks,[br]I

0:22:33.140,0:22:35.020
can't call them blocks. Because in Ruby the,[br]like,

0:22:35.020,0:22:37.250
that's already a different thing.

0:22:37.250,0:22:39.090
So hence the term blacklists.

0:22:39.090,0:22:42.130
Here's an example of a pretty handy blacklist.[br]Say

0:22:42.130,0:22:44.610
you have an admin section of your website,[br]and

0:22:44.610,0:22:46.750
you want to restrict access to the admin section

0:22:46.750,0:22:49.630
to just like, your one office IP address.[br]So

0:22:49.630,0:22:52.780
this is, again, it's using the, it's using[br]the

0:22:52.780,0:22:56.780
blacklist class method on Rack::Attack to[br]sort of configure

0:22:56.780,0:22:58.570
this in the middleware. You would, you would[br]put

0:22:58.570,0:23:03.299
this in an initializer, saying that, you're[br]given a

0:23:03.299,0:23:07.010
name like bad_admin_ip, and one of the things,[br]like,

0:23:07.010,0:23:08.669
it's different than throttles in that we don't[br]have

0:23:08.669,0:23:10.799
to pass along a limit of a period, because

0:23:10.799,0:23:13.970
it just like, it doesn't apply to blacklists.

0:23:13.970,0:23:15.720
But it has the same logic where if the

0:23:15.720,0:23:19.480
return value of this block is truthy, we're[br]gonna,

0:23:19.480,0:23:22.450
like, just give them the very fast access[br]denied

0:23:22.450,0:23:24.400
message. If it's false, then we're gonna let[br]the

0:23:24.400,0:23:26.919
request through. So this is saying, if you're[br]making

0:23:26.919,0:23:30.690
a request to a url that starts with admin,

0:23:30.690,0:23:33.510
and you are not from this IP address, we're

0:23:33.510,0:23:38.410
gonna, we're gonna just give you an access[br]denied.

0:23:38.410,0:23:41.260
This is something that KickStarter uses. We[br]call it

0:23:41.260,0:23:46.130
the starve the trolls feature. So this is,[br]if,

0:23:46.130,0:23:48.630
if you're one of our banned IPs that our

0:23:48.630,0:23:52.289
customer support team decides which IPs get[br]banned, you

0:23:52.289,0:23:54.919
cannot make any request that's not a get request.

0:23:54.919,0:23:57.520
Or, put another way, you can only make get

0:23:57.520,0:24:00.370
requests if you're from these IP addresses.

0:24:00.370,0:24:01.919
So let's think about what it's like to use

0:24:01.919,0:24:05.460
a dynamic web application if you're only using[br]gets.

0:24:05.460,0:24:07.730
You can't sign up. You can't log in. You

0:24:07.730,0:24:11.059
can't post comments. These are, these are,[br]we sort

0:24:11.059,0:24:14.640
of use this as a measure of last resort

0:24:14.640,0:24:18.140
for people who are, who are bad actors in

0:24:18.140,0:24:20.700
our community. Any big community has, you[br]know, knows

0:24:20.700,0:24:22.610
that this stuff is sort of inevitable, to[br]have

0:24:22.610,0:24:26.570
a few rotten apples.

0:24:26.570,0:24:28.590
And this has been like really fast and effective

0:24:28.590,0:24:30.970
for our community team to be able to just

0:24:30.970,0:24:34.080
like put these IP addresses into a yaml file.

0:24:34.080,0:24:36.100
They leave them there for about a week or

0:24:36.100,0:24:38.840
so, and you know gives that person sort of

0:24:38.840,0:24:41.169
time to cool off, where they're not gonna[br]go

0:24:41.169,0:24:43.490
around signing up for a bunch of accounts[br]and,

0:24:43.490,0:24:46.789
and maybe doing bad stuff or, like, posting[br]messages

0:24:46.789,0:24:49.419
or stuff like that.

0:24:49.419,0:24:52.190
So this is, I don't, I was really, I

0:24:52.190,0:24:54.380
was sort of struck when we started doing this

0:24:54.380,0:24:57.700
of like how simple this was in code, and

0:24:57.700,0:25:01.419
how much it helped our CSS, or, our community

0:25:01.419,0:25:04.350
support team. So this is another example of,[br]like,

0:25:04.350,0:25:06.250
sort of an area where I wouldn't expect Rack::Attack

0:25:06.250,0:25:08.150
to be very helpful, but it ended up being

0:25:08.150,0:25:10.760
very helpful.

0:25:10.760,0:25:16.880
Another Rack::Attack nice to have feature[br]is ActiveSupport::Notifications. So,

0:25:16.880,0:25:20.870
every time, if, if ActiveSupport::Notifications[br]are in your app,

0:25:20.870,0:25:24.360
and so for any Rails app they're already there,

0:25:24.360,0:25:28.620
we will fire a ActiveSupport notification[br]event every time

0:25:28.620,0:25:32.549
a request gets blocked or throttled. So this[br]means

0:25:32.549,0:25:35.470
you can have a subscriber to these events[br]that's

0:25:35.470,0:25:38.039
gonna log or graph these events and stuff[br]like

0:25:38.039,0:25:40.490
that. There are examples of how to do that

0:25:40.490,0:25:43.970
in the README on GitHub.

0:25:43.970,0:25:48.390
So thinking of where Rack::Attack might fall[br]in the

0:25:48.390,0:25:50.520
set of tools you use to keep your site

0:25:50.520,0:25:54.210
fast and reliable, it is, it's not a silver

0:25:54.210,0:25:57.340
bullet. Like, it very much compliments things[br]like, the

0:25:57.340,0:26:04.130
iptables firewall, or nginx limit_conn_zone,[br]limit conn module to

0:26:04.130,0:26:06.659
limit the number of concurrent requests per[br]IP address.

0:26:06.659,0:26:08.270
Or if you have, like, a CDN or a

0:26:08.270,0:26:11.330
web app firewall. So, like, you know, hardware[br]to,

0:26:11.330,0:26:13.980
to keep your website fast and reliable. Like,[br]keep

0:26:13.980,0:26:14.950
doing those.

0:26:14.950,0:26:18.210
Rack::Attack's not a silver bullet. You know,[br]it's, if

0:26:18.210,0:26:22.940
you have a mtp reflection ddos attack, like,[br]it's

0:26:22.940,0:26:27.460
gonna overwhelm your Unicorn or Heroku processes[br]pretty fast.

0:26:27.460,0:26:30.669
You need something else. But, what Rack::Attack[br]really is

0:26:30.669,0:26:34.409
good at is, it's Ruby. It knows everything[br]about

0:26:34.409,0:26:36.470
your app, like, I mean, because it's in your

0:26:36.470,0:26:41.080
application, you can use other logic from[br]your app.

0:26:41.080,0:26:43.350
Because it's Ruby, it's easy to test. You[br]write

0:26:43.350,0:26:45.730
integration tests for it the same way you[br]write

0:26:45.730,0:26:48.000
tests for the rest of your application.

0:26:48.000,0:26:49.600
And it's easy to deploy, because it's Ruby[br]code.

0:26:49.600,0:26:51.940
I don't know how you deploy changes to a

0:26:51.940,0:26:54.600
CDN or a web app firewall, but it's probably

0:26:54.600,0:26:57.570
a different process than how you deploy your[br]Ruby

0:26:57.570,0:27:01.470
code. And, and this is something that a lot,

0:27:01.470,0:27:05.740
everybody on our engineering team is comfortable[br]doing.

0:27:05.740,0:27:08.659
So that, that's why, that's where Rack::Attack[br]can fit

0:27:08.659,0:27:13.730
in into your application security mindset.

0:27:13.730,0:27:16.010
I also wanted to call out and say thank

0:27:16.010,0:27:19.559
you to my many GitHub contributors. These[br]people are

0:27:19.559,0:27:23.539
really awesome and they've taken Rack::Att-[br]like, added really

0:27:23.539,0:27:26.200
cool features, like allow to ban and fail[br]to

0:27:26.200,0:27:29.299
ban, and they've cleaned up documentation[br]and they've made

0:27:29.299,0:27:32.210
the tests a lot better. They support, added[br]reddis

0:27:32.210,0:27:37.000
support was, it used to be just memcache.[br]But

0:27:37.000,0:27:40.750
these people are doing fantastic things with[br]open source.

0:27:40.750,0:27:44.179
They're from five different continents, too,[br]which, like it

0:27:44.179,0:27:46.529
feels so cool to put code out there and,

0:27:46.529,0:27:49.730
like, people from five different continents[br]contribute to it

0:27:49.730,0:27:51.820
because they find it useful.

0:27:51.820,0:27:56.190
So, more like that please.

0:27:56.190,0:28:02.500
So, sort of wrapping up, the web, weird stuff

0:28:02.500,0:28:06.700
happens on the web. It's inevitable. It's[br]good in

0:28:06.700,0:28:08.600
a lot of cases. I, I like that, you

0:28:08.600,0:28:11.580
know, people write really innovative things[br]and, and stuff

0:28:11.580,0:28:13.049
that I would never would have come up with.

0:28:13.049,0:28:15.520
Like, that's fantastic. So I hope the web[br]stays

0:28:15.520,0:28:17.960
weird. But I also hope that the website stays

0:28:17.960,0:28:21.020
up. And Rack::Attack lets you have the best[br]of

0:28:21.020,0:28:24.470
both worlds.

0:28:24.470,0:28:27.840
So that's all I had. That's, that's Rack::Attack[br]at

0:28:27.840,0:28:31.400
KickStarter. If you have any quest- I'd love[br]to

0:28:31.400,0:28:34.520
answer any questions if people have them.[br]And, if

0:28:34.520,0:28:36.460
you're more comfortable, hit me up on Twitter[br]or

0:28:36.460,0:28:38.740
find me after the talk.