0:00:18.640,0:00:20.259
ABHISHEK PILLAI: Thanks for coming. I know[br]there's

0:00:20.259,0:00:22.800
some other cool talks right now, but you're[br]here so

0:00:22.800,0:00:25.810
that's awesome. Let's get started. You're[br]here to

0:00:25.810,0:00:28.939
learn about how to tame COBRAs.

0:00:28.939,0:00:33.960
JASON SISK: My name is Jason Sisk. I work

0:00:33.960,0:00:37.449
at Groupon. I've been here for a couple of

0:00:37.449,0:00:41.650
years. I work on predominantly Ruby/Rails[br]systems, backend development,

0:00:41.650,0:00:44.460
et cetera, and I do not like onions.

0:00:44.460,0:00:47.629
A.P.: My name is Abi, and I'm at, I've

0:00:47.629,0:00:51.159
been at Groupon for about two years, too.[br]And

0:00:51.159,0:00:52.629
Jason and I work on a team that does

0:00:52.629,0:00:58.269
backend service, basically managing inventory.[br]And I don't like

0:00:58.269,0:00:58.839
fruits.

0:00:58.839,0:01:00.760
J.S.: So part of what we're gonna tell you

0:01:00.760,0:01:03.640
today is a little bit of a history lesson

0:01:03.640,0:01:06.549
about the early pain of Groupon having site[br]outages,

0:01:06.549,0:01:10.530
et cetera, due to Rails scaling. We want to

0:01:10.530,0:01:13.390
tell you about the story of the developers[br]that

0:01:13.390,0:01:15.530
actually handled those problems and some of[br]the decisions

0:01:15.530,0:01:19.810
that they made. So that's that.

0:01:19.810,0:01:22.729
But we want to lead off with one important

0:01:22.729,0:01:23.420
point.

0:01:23.420,0:01:29.679
A.P.: Boom! Pause. You don't have to pause[br]for

0:01:29.679,0:01:32.460
that long. And, yeah.

0:01:32.460,0:01:37.100
J.S.: So. Back, back around 2007, we were[br]doing

0:01:37.100,0:01:39.479
what all the other cool kids were doing. We

0:01:39.479,0:01:43.210
were using a Rails monolith, and to some degree

0:01:43.210,0:01:45.909
still are. Rails 2 is a great framework. Who

0:01:45.909,0:01:48.170
is using Rails 2? Anyone?

0:01:48.170,0:01:49.079
AUDIENCE: Yeah!

0:01:49.079,0:01:50.439
J.S.: All right.

0:01:50.439,0:01:51.340
A.P.: Awesome.

0:01:51.340,0:01:54.880
J.S.: You and us. Rails is a great framework.

0:01:54.880,0:01:57.920
We all love Rails. That's why we're here.[br]We

0:01:57.920,0:02:03.490
still love Rails and that's why we're here.[br]But

0:02:03.490,0:02:04.939
what's great about it is that it's great for

0:02:04.939,0:02:07.700
Agile teams. It's, and for us it was really

0:02:07.700,0:02:12.069
simple. We could make some really quick decisions.[br]We

0:02:12.069,0:02:15.950
could iterate product very quickly. We could[br]iterate new

0:02:15.950,0:02:17.540
features. And we could do it with a small

0:02:17.540,0:02:19.890
team of five to ten devs.

0:02:19.890,0:02:23.159
We had a single repository. We had a single

0:02:23.159,0:02:25.189
test suite. And we had a single deploy process.

0:02:25.189,0:02:26.170
Very simple.

0:02:26.170,0:02:29.269
A.P.: And, most importantly, you, we had like[br]one

0:02:29.269,0:02:31.670
shared, conceptual understanding of the code[br]base. When we

0:02:31.670,0:02:33.129
wanted to make a change, we knew where to

0:02:33.129,0:02:36.510
put it. And things were simple that way.

0:02:36.510,0:02:39.650
J.S.: Also what was great was, and still is,

0:02:39.650,0:02:44.260
about Rails, that integrating components is[br]really easy. The

0:02:44.260,0:02:47.829
convention over configuration, model associations[br]- all of that

0:02:47.829,0:02:50.110
business you can put together things very[br]quickly and

0:02:50.110,0:02:53.200
very easily. But we didn't come here to talk

0:02:53.200,0:02:55.420
to you about Rails.

0:02:55.420,0:02:58.860
A.P.: We came here to tell you about cobras,

0:02:58.860,0:03:02.530
and how to tame them. At Groupon, we actually

0:03:02.530,0:03:04.650
have a mo- monolith, and we call it the

0:03:04.650,0:03:07.500
primary web app. But Jason had a thought for

0:03:07.500,0:03:09.549
the purposes of this talk, we'd come up with

0:03:09.549,0:03:13.260
a more scientifically accurate name for it.

0:03:13.260,0:03:19.540
Yeah. So. Centralized Omnipotent Big-ass Rails[br]Application.

0:03:19.540,0:03:22.030
J.S.: Big-ass. So we want to take you back

0:03:22.030,0:03:26.909
to 2009 for just a minute. So Groupon was

0:03:26.909,0:03:28.780
about two years old, give or take, and we

0:03:28.780,0:03:31.299
were still kind of kicking into gear. People[br]would

0:03:31.299,0:03:34.219
come into the office in Chicago we've got,[br]open

0:03:34.219,0:03:36.750
up New Relic, and they'd see stuff like this.

0:03:36.750,0:03:38.709
A.P.: So as you can see, like, in the

0:03:38.709,0:03:41.629
middle of the night, it's great. Everything's[br]working really

0:03:41.629,0:03:43.780
well. Soon as people woke up and started using

0:03:43.780,0:03:47.709
it - damn people - our performance immediately[br]started

0:03:47.709,0:03:52.150
to drop.

0:03:52.150,0:03:55.340
And then eight months later, we had about[br]thirty

0:03:55.340,0:04:00.319
thousand requests per minute and everything[br]was on fire.

0:04:00.319,0:04:01.950
J.S.: We blame Oprah.

0:04:01.950,0:04:03.540
A.P.: As you do.

0:04:03.540,0:04:07.599
J.S.: It's Oprah's fault. Oprah crashed Groupon.[br]Oprah crashed

0:04:07.599,0:04:12.579
Groupon not once, but at least twice. And[br]also

0:04:12.579,0:04:16.130
the Gap crashed Groupon too. Actually, the[br]truth is,

0:04:16.130,0:04:20.269
Groupon crashed Groupon. We were not scaling[br]properly. Bad.

0:04:20.269,0:04:22.440
Bad Groupon.

0:04:22.440,0:04:27.940
The Cobra was getting fatter and fatter. We[br]were

0:04:27.940,0:04:29.240
up to-

0:04:29.240,0:04:33.780
A.P.: Yeah. So. We were up to, we started,

0:04:33.780,0:04:35.990
we had, like, five to fifty devs. We started

0:04:35.990,0:04:37.940
with about three to five hundred commits per[br]month.

0:04:37.940,0:04:40.229
Slowly, and in a couple of years, as you

0:04:40.229,0:04:42.910
can see, we were averaging about two thousand[br]commits

0:04:42.910,0:04:45.139
in a single month. We had a lot of

0:04:45.139,0:04:46.690
developers developing a lot of things.

0:04:46.690,0:04:49.900
J.S.: This is all one cobra.

0:04:49.900,0:04:54.000
A.P.: And you know, we started thinking about[br]SOA

0:04:54.000,0:04:57.199
at that point. It was already becoming really[br]painful.

0:04:57.199,0:05:01.430
But we looked at the cobra, directly in the

0:05:01.430,0:05:03.139
eyes, and it scared the shit out of us.

0:05:03.139,0:05:05.600
J.S.: We had a lot of scoping problems. And

0:05:05.600,0:05:09.770
a lot of that had to do with model

0:05:09.770,0:05:12.310
coupling. So, one of the biggest things that[br]was

0:05:12.310,0:05:16.810
keeping us from extracting services early[br]was as the,

0:05:16.810,0:05:18.860
as the code grew, you had a lot of

0:05:18.860,0:05:21.699
sort of natural convention coupling that was[br]happening in

0:05:21.699,0:05:22.870
the models.

0:05:22.870,0:05:26.380
So a little bit of a over-simplified example[br]here.

0:05:26.380,0:05:29.900
But you have a, let's say you have, you're

0:05:29.900,0:05:31.759
on the MyGroupon's page. You want to look[br]at

0:05:31.759,0:05:33.960
all of the Groupons that you've bought. And[br]you

0:05:33.960,0:05:35.650
want to see all the titles for all of

0:05:35.650,0:05:37.370
those. So when we go to render the interface

0:05:37.370,0:05:39.880
we want to display all these deal titles.[br]In

0:05:39.880,0:05:41.930
the cobra, you might find a set of dependent

0:05:41.930,0:05:44.490
relationships that are somewhat like this,[br]where you can

0:05:44.490,0:05:47.539
see the cyclical dependencies.

0:05:47.539,0:05:51.199
But building these types of associations was[br]fairly common

0:05:51.199,0:05:56.349
place, which was kind of bad in some ways.

0:05:56.349,0:05:59.090
So in this case, you would instantiate a user,

0:05:59.090,0:06:01.360
which would require a database lookup to the[br]Users

0:06:01.360,0:06:05.470
table, select star, and, and you would map[br]over

0:06:05.470,0:06:07.930
that, that user's orders to get all of the

0:06:07.930,0:06:09.699
deal titles.

0:06:09.699,0:06:12.870
In this, in this case, there is a Demeter

0:06:12.870,0:06:16.830
violation. Demeter violations are bad.

0:06:16.830,0:06:20.060
A.P.: And it looks clean. I mean, it looks

0:06:20.060,0:06:23.310
good. But, what it does is couples our components.

0:06:23.310,0:06:26.180
J.S.: Here is an example of what I was

0:06:26.180,0:06:30.280
talking about. You, you have a basically unnecessarily-[br]unnecessary

0:06:30.280,0:06:33.930
table lookup to Users. Now, if you're designing[br]your

0:06:33.930,0:06:37.400
applications well, you can avoid this right[br]out of

0:06:37.400,0:06:39.800
the gate. But Rails conventions don't, don't[br]encourage you

0:06:39.800,0:06:42.199
to avoid this right out of the gate. And

0:06:42.199,0:06:46.160
ActiveRecord DSL for, for advanced queries[br]aren't something that

0:06:46.160,0:06:48.590
people just tend to do by default. Or at

0:06:48.590,0:06:50.349
least they didn't in 2009.

0:06:50.349,0:06:55.090
A.P.: Yeah. And, I mean. Things got a lot

0:06:55.090,0:06:59.440
worse, because our code base and cobra was[br]just

0:06:59.440,0:07:01.740
getting bigger and bigger. You can see here[br]it's

0:07:01.740,0:07:07.520
almost two million lines of code at this point.

0:07:07.520,0:07:09.400
And, oh yeah, we have to stay up 100%

0:07:09.400,0:07:13.539
of the time. So that's a problem. All right.

0:07:13.539,0:07:16.599
J.S.: Also, the database is completely on[br]fire.

0:07:16.599,0:07:22.310
A.P.: So yeah. We were in quite a pickle.

0:07:22.310,0:07:29.310
It was painful. Testing sucked. I mean, we[br]had

0:07:30.650,0:07:32.770
to wait like forty-five minutes for a build[br]to

0:07:32.770,0:07:36.340
run. You basically ran your tests and then[br]figure

0:07:36.340,0:07:37.690
out something else to do, because you had[br]to

0:07:37.690,0:07:41.370
wait while your tests ran. And a lot of

0:07:41.370,0:07:43.879
our release engineer devoted a lot of effort[br]to

0:07:43.879,0:07:45.590
make those tests run faster.

0:07:45.590,0:07:49.720
J.S.: Deploys were terrible. Deploy, deploy[br]process was somewhere

0:07:49.720,0:07:52.090
on the, on the scale of three hours to

0:07:52.090,0:07:56.150
deploy the, the application. Just a really[br]bad development

0:07:56.150,0:07:58.370
experience, especially as you start to have[br]teams that,

0:07:58.370,0:08:01.599
that split, split ownership. They want to[br]iterate on

0:08:01.599,0:08:03.650
features that matter to their team, and they[br]don't

0:08:03.650,0:08:06.650
want to be held up by this gigantic monolithic

0:08:06.650,0:08:07.050
application.

0:08:07.050,0:08:09.979
And, and it's, you know, the, the deploy's[br]only

0:08:09.979,0:08:12.090
happening once a week. That really hurts the[br]team's

0:08:12.090,0:08:14.439
ability to set, that maybe wants to do continuous

0:08:14.439,0:08:16.289
deployment. So, it sucked.

0:08:16.289,0:08:20.590
A.P.: Yeah. I mean, and development pace was[br]increasing,

0:08:20.590,0:08:23.460
as you saw, and, I mean, what's the best

0:08:23.460,0:08:24.949
place to put the next line of code, as

0:08:24.949,0:08:26.530
I heard in a talk earlier. It's the place

0:08:26.530,0:08:29.740
that you're changes. Models got bloated, and[br]there's a

0:08:29.740,0:08:30.650
lot of cruft.

0:08:30.650,0:08:32.740
J.S.: So all of these things were terrible.[br]It

0:08:32.740,0:08:37.690
was very painful. So, we decided to move towards

0:08:37.690,0:08:41.070
service extraction a little bit more seriously.

0:08:41.070,0:08:44.690
If there's a big take away from this first

0:08:44.690,0:08:47.220
section, we just want you to remember that[br]cobras

0:08:47.220,0:08:54.220
are great. They are great. Until they aren't.

0:08:55.269,0:08:58.440
A.P.: So we needed to alleviate this pain[br]immediately.

0:08:58.440,0:09:00.370
We needed to get that code out of there.

0:09:00.370,0:09:04.180
We needed a quick extraction. So we decided[br]to

0:09:04.180,0:09:06.779
extract a new service and build it on top

0:09:06.779,0:09:09.610
of our current schema. We decided to start[br]with

0:09:09.610,0:09:13.890
the order service, because. I mean. It was[br]causing

0:09:13.890,0:09:16.589
a lot of database contention. We had a lot

0:09:16.589,0:09:18.959
of people buying a lot of Groupons, and, a

0:09:18.959,0:09:20.600
good problem to have, but it was bringing[br]our

0:09:20.600,0:09:21.470
database down.

0:09:21.470,0:09:23.370
So we needed to get that code out of

0:09:23.370,0:09:27.800
it, and also another thing behind the, behind[br]choosing

0:09:27.800,0:09:30.010
orders to start is that, you know, it's gonna

0:09:30.010,0:09:32.060
be a long-lived model, a long living model[br]in

0:09:32.060,0:09:35.120
our domain. We know that for sure.

0:09:35.120,0:09:38.190
So, to illustrate, this is what it looks like

0:09:38.190,0:09:40.660
in the beginning. And this is what we're trying

0:09:40.660,0:09:44.250
to accomplish. You have an orders, you have[br]the

0:09:44.250,0:09:45.920
cobra, and then we're trying to have a separate

0:09:45.920,0:09:49.450
orders codebase, which will have its own database.[br]But

0:09:49.450,0:09:53.070
it continues to have re- a read-only access[br]to

0:09:53.070,0:09:56.180
the cobra's database, because we didn't focus[br]on completely

0:09:56.180,0:10:01.040
making the cobra, the order service, re, stopping,[br]stopping

0:10:01.040,0:10:03.500
it from reaching back into the cobra's database.

0:10:03.500,0:10:08.230
And, I mean, the cobra was really sneaky.[br]It

0:10:08.230,0:10:11.000
was really tough to find all the ways that,

0:10:11.000,0:10:14.269
with Rails callbacks and model associations,[br]all the ways

0:10:14.269,0:10:18.769
that the components were coupled.

0:10:18.769,0:10:21.510
So we built some tools to make that easier.

0:10:21.510,0:10:23.310
This is one of them. The service wall, as

0:10:23.310,0:10:25.360
call it. We're trying to, the main goal here

0:10:25.360,0:10:29.600
is separating the concerns of orders within[br]the application.

0:10:29.600,0:10:33.240
So, you start with having your services in[br]a

0:10:33.240,0:10:37.589
separate directory. Let's see a closer look[br]of it.

0:10:37.589,0:10:40.240
You have the order service in its own directory,

0:10:40.240,0:10:42.600
and you have its own app, its own lib,

0:10:42.600,0:10:45.160
its own specs. The way that works is that

0:10:45.160,0:10:48.260
in environment dot rb file, we iterated through[br]these

0:10:48.260,0:10:50.500
services and added them to the load path.[br]So

0:10:50.500,0:10:53.260
the application to the application looks like[br]it's just

0:10:53.260,0:10:57.339
one big application, but for our purposes,[br]the code

0:10:57.339,0:10:59.000
was separate.

0:10:59.000,0:11:03.060
So, this is like, a small example of how

0:11:03.060,0:11:06.110
service wall works. You have this disable[br]model access

0:11:06.110,0:11:11.579
method that basically, if, if you specify[br]the models

0:11:11.579,0:11:14.510
that you want to, if you specify the service

0:11:14.510,0:11:18.820
that you want to disable or deprecate, and[br]it'll

0:11:18.820,0:11:22.560
figure out the models of that service and[br]add

0:11:22.560,0:11:28.709
it to this do-not-touch list. And basically[br]raise these

0:11:28.709,0:11:31.440
kinds of violations. So if you use the disable

0:11:31.440,0:11:34.440
model access model, when you run your tests,[br]it

0:11:34.440,0:11:37.190
will put up this message saying, you don't[br]have

0:11:37.190,0:11:38.700
access to this method.

0:11:38.700,0:11:41.380
When a deal is trying to access an order,

0:11:41.380,0:11:43.370
we can figure that out just by running our

0:11:43.370,0:11:46.720
tests. If you use the more friendlier, deprecate[br]service

0:11:46.720,0:11:50.269
mo- deprecate model access method, then you[br]can be

0:11:50.269,0:11:52.660
more permissive and it'll just log it to a

0:11:52.660,0:11:54.899
file. You can see that in development mode[br]or

0:11:54.899,0:11:57.630
you can have it on staging, and that'll basically,

0:11:57.630,0:11:59.459
that'll allow you to find all the places where

0:11:59.459,0:12:02.930
you're having service infractions.

0:12:02.930,0:12:04.800
You can't do this in production though, because[br]it

0:12:04.800,0:12:09.240
causes a serious produ- performance hit.

0:12:09.240,0:12:14.360
Oh yeah. So this is how, so this is

0:12:14.360,0:12:18.339
how you actually use the service wall. Use,[br]you,

0:12:18.339,0:12:21.639
at the top of your controller, you disable,[br]use

0:12:21.639,0:12:24.769
the method disable_model_access or deprecate_model_access,[br]depending on what you

0:12:24.769,0:12:27.029
want to do. You tell it what service, and

0:12:27.029,0:12:29.570
it even lets you exempt some actions that[br]you

0:12:29.570,0:12:31.480
don't want to raise violations on yet.

0:12:31.480,0:12:35.649
That way you can comment out that action and

0:12:35.649,0:12:38.269
tackle one action at a time. Which endpoints[br]are

0:12:38.269,0:12:41.019
actually reaching over and causing the service[br]wall infraction.

0:12:41.019,0:12:45.680
J.S.: So, in addition to the service wall,[br]one,

0:12:45.680,0:12:48.420
one other problem with this approach, this[br]extraction approach

0:12:48.420,0:12:51.760
is that, because you necessarily fork the[br]code, you

0:12:51.760,0:12:53.829
get a lot of cruft left over from the

0:12:53.829,0:12:59.740
old, the old domain. So you find yourself[br]asking,

0:12:59.740,0:13:02.339
teams find themselves asking, very often,[br]is this endpoint

0:13:02.339,0:13:04.360
even used? Do we even care about this code

0:13:04.360,0:13:04.990
anymore?

0:13:04.990,0:13:09.740
So, a small team of Groupon developers hacked[br]together

0:13:09.740,0:13:13.139
something called Route 66 that we use internally[br]to

0:13:13.139,0:13:17.079
track down cruft in both our old cobra and

0:13:17.079,0:13:20.610
our new cobra. So it basically answers the[br]question,

0:13:20.610,0:13:22.750
are these endpoints used? I don't know if[br]you

0:13:22.750,0:13:23.990
can see this very well, but this is a

0:13:23.990,0:13:24.899
little bit of a UI.

0:13:24.899,0:13:25.750
A.P.: Yeah.

0:13:25.750,0:13:30.430
J.S.: But what we do is, we analyze log

0:13:30.430,0:13:33.620
files, we analyze, spelunk logs to come up[br]with

0:13:33.620,0:13:37.060
which controller actions are being hit, what's[br]the frequency.

0:13:37.060,0:13:39.370
Is this a route that is hit once a

0:13:39.370,0:13:42.449
week, you know. Once a, once a month? And

0:13:42.449,0:13:45.490
we can very aggressively decruft using this[br]tool as

0:13:45.490,0:13:46.790
well.

0:13:46.790,0:13:52.769
A.P.: All right. So there's definitely pros[br]to this

0:13:52.769,0:13:56.570
approach. Because you're focusing on just[br]separating the models,

0:13:56.570,0:14:00.209
I mean, just separating the code, you can[br]quickly

0:14:00.209,0:14:02.730
and not worry about spinning up a separate[br]database

0:14:02.730,0:14:04.839
schema, separate naming, all of that. You[br]just worry

0:14:04.839,0:14:08.230
about separating the code, and that focuses[br]the abstraction.

0:14:08.230,0:14:11.839
It makes it easier to spin up endpoints. But

0:14:11.839,0:14:13.410
the cons are, you're stilled tied to that[br]legac,

0:14:13.410,0:14:15.540
to that legacy database. Not such a bad thing

0:14:15.540,0:14:16.930
if you really need to get it out of

0:14:16.930,0:14:20.860
there. But, because you're forking this code[br]now, and

0:14:20.860,0:14:23.459
now it's being hit through endpoints, there[br]is still

0:14:23.459,0:14:26.079
a lot of cruft in the, in the, in

0:14:26.079,0:14:28.079
the code base. Because a lot of these endpoints

0:14:28.079,0:14:29.570
are now not being used.

0:14:29.570,0:14:32.220
J.S.: So this was the first extraction pattern[br]that

0:14:32.220,0:14:34.320
we used at Groupon to get out of the

0:14:34.320,0:14:39.480
original cobra, the original Groupon cobra.[br]But teams sort

0:14:39.480,0:14:41.149
of own their own tactics, and there are other

0:14:41.149,0:14:44.040
ways that they're doing it as well. One way

0:14:44.040,0:14:47.009
that, one way that service extraction is also[br]happening

0:14:47.009,0:14:49.940
is by using greenfield services that use a[br]message

0:14:49.940,0:14:53.730
bus. Sometimes you just need to keep that[br]legacy

0:14:53.730,0:14:56.110
API running, because there are a lot of client

0:14:56.110,0:14:58.120
dependencies on it. There's a lot of dependencies[br]on

0:14:58.120,0:15:00.699
the structure of the data.

0:15:00.699,0:15:03.310
But who likes doing greenfield work in here?[br]Raise

0:15:03.310,0:15:05.860
your hand if you like greenfield work. Right.[br]That

0:15:05.860,0:15:10.000
should be all of you. Whatever.

0:15:10.000,0:15:13.170
So, it is possible to do greenfield service[br]extraction,

0:15:13.170,0:15:16.620
and we're doing this as well. So, again, we

0:15:16.620,0:15:21.720
have a similar. Whoops. Juggling between power[br]point and

0:15:21.720,0:15:26.940
preview. Similar type of situation. You have[br]this cobra,

0:15:26.940,0:15:29.639
and then we get to the scenario that we're,

0:15:29.639,0:15:31.610
we're trying to reach with the greenfield[br]extraction, where

0:15:31.610,0:15:34.680
you have, in this case the red, the red

0:15:34.680,0:15:37.730
box represents all new code. There's a gem,[br]a

0:15:37.730,0:15:40.810
client gem that interact, that runs in the[br]original

0:15:40.810,0:15:42.850
cobra, that runs in the green cobra. And when

0:15:42.850,0:15:46.290
this service writes data to its db, a message

0:15:46.290,0:15:49.899
is sent that the green cobra consumes and[br]sends

0:15:49.899,0:15:53.190
over to its own data store, thus satisfying[br]all

0:15:53.190,0:15:56.550
of the legacy API requirements.

0:15:56.550,0:15:58.399
And then what's notable about this is to keep

0:15:58.399,0:16:02.899
everything in sync for service cut-overs,[br]rollouts, et cetera,

0:16:02.899,0:16:05.589
there is a background sync worker that runs,[br]that

0:16:05.589,0:16:08.920
syncs it one way from the old database to

0:16:08.920,0:16:12.620
the new database.

0:16:12.620,0:16:16.079
There are pros and cons to this approach as

0:16:16.079,0:16:19.230
well. Some of the better parts are that you

0:16:19.230,0:16:21.880
can get rid of your legacy data quickly, again.

0:16:21.880,0:16:23.870
Devs like greenfield stuff. You like to design[br]your

0:16:23.870,0:16:29.019
own systems. You also get to minimize the[br]cut-over

0:16:29.019,0:16:32.199
risk with your data sync. So you're not splitting

0:16:32.199,0:16:34.319
the table and you have to have all of

0:16:34.319,0:16:38.500
these API dependencies written on one hand[br]so that

0:16:38.500,0:16:41.649
when you break your database you don't have,[br]you

0:16:41.649,0:16:43.259
don't have failures.

0:16:43.259,0:16:45.120
So you can phase the, you can phase out

0:16:45.120,0:16:48.089
your new, your new endpoints, and you can[br]own

0:16:48.089,0:16:50.579
the timing of when you build out new endpoint

0:16:50.579,0:16:53.500
features. Again. Some of the, or some of the

0:16:53.500,0:16:56.380
cons are that, it is not trivial to build

0:16:56.380,0:17:00.399
synchronization worker, and it is less trivial[br]to build

0:17:00.399,0:17:03.670
a validation engine for the data to make sure

0:17:03.670,0:17:05.180
that you don't get it out of sync when

0:17:05.180,0:17:07.490
you're pulling from the original source. And[br]then there

0:17:07.490,0:17:12.490
are race conditions involved in this as well.

0:17:12.490,0:17:14.540
A.P.: So Jason and I work on a team

0:17:14.540,0:17:18.950
that manages inventory, as I said earlier.[br]One of

0:17:18.950,0:17:21.660
the, looking a little further down the road,[br]one

0:17:21.660,0:17:23.569
of the things we needed to do was get,

0:17:23.569,0:17:25.530
now we needed to get vouchers out of the

0:17:25.530,0:17:30.120
orders service. Another service extraction.[br]And vouchers are actually

0:17:30.120,0:17:33.160
the things that customers redeem.

0:17:33.160,0:17:37.680
So, a simplified example of what a voucher[br]actually

0:17:37.680,0:17:41.480
like would look like, except that now we have

0:17:41.480,0:17:44.650
an id, which is stored in our database. We

0:17:44.650,0:17:47.000
have the price, which is stored in a legacy

0:17:47.000,0:17:51.480
database, and now, Groupon's grown since orders.[br]We now

0:17:51.480,0:17:55.260
have an international platform codebase that[br]serves many different

0:17:55.260,0:17:59.560
countries. We have offices in Berlin, London,[br]Chinai, Korea,

0:17:59.560,0:18:02.630
and many more places. But yeah. Now we've[br]got

0:18:02.630,0:18:05.590
to make it, but our service's responsibility[br]is to

0:18:05.590,0:18:07.480
make it seem like none of that matters. Anyone

0:18:07.480,0:18:10.110
asking for voucher data needs to know about[br]all

0:18:10.110,0:18:11.020
voucher data.

0:18:11.020,0:18:13.490
Our services need to be global as well. So,

0:18:13.490,0:18:16.830
this is what our world looks like. And this

0:18:16.830,0:18:18.320
is how our service needs to be built on

0:18:18.320,0:18:23.580
top of that. What helped, in managing these[br]different

0:18:23.580,0:18:26.780
sources of truth, was this manager accessor[br]pattern in

0:18:26.780,0:18:31.690
our code base. Specifically, oh. Let me check[br]if

0:18:31.690,0:18:37.190
I need to- yeah. Specifically, next slide[br]please, this

0:18:37.190,0:18:38.850
is what, this is how it helped our code

0:18:38.850,0:18:41.290
base. Because in the controller, you could[br]just specify,

0:18:41.290,0:18:44.270
you could talk, talk to this manager object,[br]and

0:18:44.270,0:18:46.090
you'd say, find me this voucher.

0:18:46.090,0:18:48.620
And the manager, can you jump to that? All

0:18:48.620,0:18:50.300
right, it's gonna look like a lot of code,

0:18:50.300,0:18:53.260
but let's go step-by-step. In the manager,[br]that's where

0:18:53.260,0:18:55.760
all the complexity lies. You have the accessor[br]that

0:18:55.760,0:18:58.330
accesses local data. You have an accessor,[br]a separate

0:18:58.330,0:19:01.190
accessor - and accessors are just simply,[br]all they

0:19:01.190,0:19:06.050
do is persistence and finding, and finding[br]data -

0:19:06.050,0:19:09.450
so the accessors for the legacy database here,[br]the

0:19:09.450,0:19:12.290
cobra accessor, you get that price information,[br]and then

0:19:12.290,0:19:15.500
you have an international accessor that goes,[br]it could

0:19:15.500,0:19:18.580
be a database call or, in our case, that's

0:19:18.580,0:19:23.020
a HTTP call across the ocean.

0:19:23.020,0:19:24.840
And then you bring all that together, wrap[br]it

0:19:24.840,0:19:27.480
in a model and have it return that back

0:19:27.480,0:19:30.940
to your controller. Hang on.

0:19:30.940,0:19:35.230
All right. So, definitely pros and cons to[br]this

0:19:35.230,0:19:37.160
approach. One of the things was, it's easy[br]to

0:19:37.160,0:19:40.090
incorporate many different data sources. We[br]call that a

0:19:40.090,0:19:42.910
facade because it kind of hides all of that.

0:19:42.910,0:19:45.840
But the, behind the backend of it is really

0:19:45.840,0:19:47.150
more complex.

0:19:47.150,0:19:51.710
And, but you hide that complexity. That your[br]accessors

0:19:51.710,0:19:54.020
are bound to the schema changes. So, our cobra

0:19:54.020,0:19:57.180
accessor still has to know about the legacy[br]schema.

0:19:57.180,0:20:00.390
And you're, you, you can't really, making[br]changes there

0:20:00.390,0:20:02.350
is not trivial.

0:20:02.350,0:20:05.310
And, sometimes you can use that as a crutch.

0:20:05.310,0:20:07.190
So if someone asks you, can you give me

0:20:07.190,0:20:08.980
this piece of data about a voucher, I really

0:20:08.980,0:20:10.510
need it, and you want to expose it to

0:20:10.510,0:20:12.900
the endpoints, you're like, well, I do have[br]access

0:20:12.900,0:20:15.190
to the database or I could just make a

0:20:15.190,0:20:16.810
call. And now you, now you're serving the[br]end-

0:20:16.810,0:20:20.040
that data, and you're tied to serving that[br]data

0:20:20.040,0:20:21.040
in your API.

0:20:21.040,0:20:23.900
But the important thing there is to be diligent,

0:20:23.900,0:20:25.920
and as soon as you start serving that, they'll

0:20:25.920,0:20:31.080
put a strategy together to, actually on that[br]data.

0:20:31.080,0:20:34.220
Otherwise you're, the complexity in the manager,[br]which is

0:20:34.220,0:20:37.270
both a pro and a con, will always be

0:20:37.270,0:20:40.110
there. The purpose of the manager is that[br]it

0:20:40.110,0:20:43.060
hides that complexity, but as you start owning[br]more

0:20:43.060,0:20:45.560
data, it should become simpler.

0:20:45.560,0:20:50.500
J.S.: So, these, these three extraction patterns[br]that we've

0:20:50.500,0:20:54.780
gone through are just a little bit of, a

0:20:54.780,0:20:57.410
little bit of what's going on. There are different

0:20:57.410,0:21:00.730
service extraction patterns going on, both[br]at Groupon and

0:21:00.730,0:21:05.640
probably in your worlds too. So, again, this[br]is

0:21:05.640,0:21:07.600
just a example of some of the ways that

0:21:07.600,0:21:11.000
we've chosen to do things. There are other[br]interesting

0:21:11.000,0:21:13.450
talks about this this week at RailsConf going[br]on,

0:21:13.450,0:21:15.640
so be, it'd be neat to check those out,

0:21:15.640,0:21:17.410
too, if you want to talk to us about

0:21:17.410,0:21:18.350
them.

0:21:18.350,0:21:21.400
But, you should definitely consider letting[br]your teams own

0:21:21.400,0:21:23.290
their tactics if you're trying to make decisions[br]about

0:21:23.290,0:21:26.840
doing SOA, because you might find some neat[br]things

0:21:26.840,0:21:27.680
that you didn't know about.

0:21:27.680,0:21:30.410
A.P.: Yeah. So I'm gonna stand over here cause

0:21:30.410,0:21:33.140
I feel like I'm just talking to these guys.

0:21:33.140,0:21:35.000
But yeah. So, there's definitely a lot of[br]things

0:21:35.000,0:21:37.630
that we learned from doing these different[br]service extractions.

0:21:37.630,0:21:39.170
Like Jason said, there are a lot of other

0:21:39.170,0:21:43.160
service extractions that happened at Groupon[br]and continue to

0:21:43.160,0:21:45.250
happen today.

0:21:45.250,0:21:48.810
But, taming a cobra is serious business. I[br]mean,

0:21:48.810,0:21:51.980
like I always say, YPAGNIRN. You probably[br]ain't gonna

0:21:51.980,0:21:56.780
need it right now. But, but the, but, like,

0:21:56.780,0:21:59.380
the tipping point on which you need to start

0:21:59.380,0:22:03.800
going towards service-oriented architecture[br]isn't just black or white.

0:22:03.800,0:22:06.530
It's, it's more of an art than a science.

0:22:06.530,0:22:08.460
But as soon as you start talking about service-oriented

0:22:08.460,0:22:11.060
architecture, once you start feeling the pains,[br]you need

0:22:11.060,0:22:13.580
to put, put together a strategy to accomplish[br]that.

0:22:13.580,0:22:15.310
J.S.: Yeah. You don't want to sit around and

0:22:15.310,0:22:16.660
wait for Oprah to blow your site up.

0:22:16.660,0:22:20.750
A.P. But there's also the importance of allowing[br]your

0:22:20.750,0:22:24.870
domain to actually evolve. Models that you[br]think are

0:22:24.870,0:22:27.000
important in the beginning aren't gonna be[br]important later

0:22:27.000,0:22:30.760
on. And it, that's the big benefit of a

0:22:30.760,0:22:33.830
cobra, is that it allows you to iterate quickly.

0:22:33.830,0:22:36.360
J.S.: Something else that we have also learned[br]is

0:22:36.360,0:22:38.470
that when you go into service extraction,[br]it's really

0:22:38.470,0:22:42.390
important that you actually have a strategy.[br]Know what

0:22:42.390,0:22:45.400
you need to break apart. Know what you need

0:22:45.400,0:22:48.270
to leave in the monolith. These are important[br]things

0:22:48.270,0:22:50.780
to consider. Know what the priorities are[br]between those

0:22:50.780,0:22:54.180
things. It's very, it's very tricky to just[br]go

0:22:54.180,0:22:58.480
about service extraction very scattershot[br]and not really understanding

0:22:58.480,0:23:00.810
your business model or what benefits you derive[br]from

0:23:00.810,0:23:03.650
extracting certain pieces over others.

0:23:03.650,0:23:05.460
You should prefer the things that are clearly[br]like

0:23:05.460,0:23:08.830
their own thing, their own components, or[br]things that

0:23:08.830,0:23:13.030
are particular maintenance problems or represent[br]some sort of

0:23:13.030,0:23:17.490
legacy design or, or strange behavior. But[br]the other

0:23:17.490,0:23:19.530
important part of having a strategy is that[br]you

0:23:19.530,0:23:24.360
should expect the unexpected. Scope creep[br]will bite you,

0:23:24.360,0:23:26.490
and you know, as these, as these code bases

0:23:26.490,0:23:29.010
get bigger, pulling out of them becomes a[br]lot

0:23:29.010,0:23:33.910
more of a tricky process than you might envision.

0:23:33.910,0:23:35.860
Another thing that's important is that you,[br]you think

0:23:35.860,0:23:39.470
about your entire service stack. And you should[br]know

0:23:39.470,0:23:42.090
your business, and so you should know, or[br]you

0:23:42.090,0:23:44.860
should at least conceptualize how all of those[br]parts

0:23:44.860,0:23:47.330
of your business are gonna fit together.

0:23:47.330,0:23:48.890
How does the data flow between them? What[br]are

0:23:48.890,0:23:52.970
the service agreements between those, those[br]compartments? That's all

0:23:52.970,0:23:55.070
important to know. You're gonna need to be[br]caching

0:23:55.070,0:23:59.290
between services for, for load. You're gonna[br]need to

0:23:59.290,0:24:05.570
be caching services for, for latency requirements.[br]So you

0:24:05.570,0:24:07.590
have to serve upstream to some kind of complex

0:24:07.590,0:24:10.690
algorithm. That algorithm is gonna need zero[br]latency return

0:24:10.690,0:24:12.030
from your service.

0:24:12.030,0:24:13.410
You need to be thinking about all of these

0:24:13.410,0:24:16.540
kinds of things when you're doing service[br]extraction.

0:24:16.540,0:24:20.250
A.P.: And the way Jason's saying it is, is

0:24:20.250,0:24:23.120
definitely makes it seem like, oh, it's one[br]slide

0:24:23.120,0:24:24.920
on our deck. But each of those topics could

0:24:24.920,0:24:29.160
be a separate talk. And they are. So, definitely,

0:24:29.160,0:24:30.260
there's a lot of learn in that, in that

0:24:30.260,0:24:30.990
domain.

0:24:30.990,0:24:34.730
J.S.: Right. Just in terms of actual topics[br]in

0:24:34.730,0:24:37.040
it, another thing you want to think about[br]is

0:24:37.040,0:24:40.490
messaging. Inter-service messaging, when you're[br]pulling these services apart,

0:24:40.490,0:24:42.290
they do need to talk to each other. You

0:24:42.290,0:24:45.400
should definitely think about what do those[br]messages look

0:24:45.400,0:24:49.790
like. What are their delivery SOAs? Do you[br]guarantee

0:24:49.790,0:24:52.060
that they're delivered? Do you guarantee the[br]order that

0:24:52.060,0:24:54.880
they're delivered in? What are the payloads[br]look like?

0:24:54.880,0:24:57.830
Think about all of this stuff.

0:24:57.830,0:25:01.620
And, you also need to consider your, concern[br]yourself

0:25:01.620,0:25:05.750
with authentication and authorization. These[br]are, these are important

0:25:05.750,0:25:08.470
topics. I think like, there was a talk about

0:25:08.470,0:25:09.330
this yesterday-

0:25:09.330,0:25:09.880
A.P.: There were two.

0:25:09.880,0:25:12.300
J.S.: Oh, there were two talks about this[br]yesterday.

0:25:12.300,0:25:13.680
But you should know what you're, know what[br]you're

0:25:13.680,0:25:16.730
users are doing. Your sites getting bigger.[br]Your users

0:25:16.730,0:25:19.640
are getting more complicated. Know, know what[br]they need

0:25:19.640,0:25:21.680
access to. Know how they get into your, how

0:25:21.680,0:25:23.230
they get into your services, how they get[br]through

0:25:23.230,0:25:26.420
your services. And know what they can do at

0:25:26.420,0:25:28.860
each step of the way.

0:25:28.860,0:25:31.660
A.P.: And you need to create like a supportive,

0:25:31.660,0:25:36.180
supporting environment for services. We were[br]lucky, we had

0:25:36.180,0:25:40.320
entire teams devoted to building tools, to,[br]that make

0:25:40.320,0:25:43.130
it easier to spin up services easily. And[br]a

0:25:43.130,0:25:48.090
release engineering team that made it easier[br]to re,

0:25:48.090,0:25:52.080
deploy these services. All those became really[br]easy for

0:25:52.080,0:25:54.790
us, but if, in your company, you need to

0:25:54.790,0:25:56.830
make sure that, or in your application, you[br]need

0:25:56.830,0:25:58.270
to make sure that you think about these things

0:25:58.270,0:26:01.610
and devote tools and time to making those[br]things

0:26:01.610,0:26:02.280
simpler.

0:26:02.280,0:26:06.240
Also, now is the time to start considering[br]uuids.

0:26:06.240,0:26:09.710
As soon as you start talking about service-oriented[br]architecture,

0:26:09.710,0:26:15.470
go to uuids from the start. This will immediately

0:26:15.470,0:26:17.890
separate you from your database, and that's[br]gonna be

0:26:17.890,0:26:20.430
really important, because you're gonna be[br]moving data from

0:26:20.430,0:26:23.200
one source to another.

0:26:23.200,0:26:26.380
And, you need to write code good. You know,

0:26:26.380,0:26:28.900
like, it's hard to. I mean, it's easy to

0:26:28.900,0:26:31.010
say, say that, but it's hard to do. Think

0:26:31.010,0:26:34.150
about the solid principles. Think about where[br]things belong.

0:26:34.150,0:26:37.000
Ask yourself, am I coupling these two components[br]together

0:26:37.000,0:26:40.970
for the fu- and is that useful enough that

0:26:40.970,0:26:42.720
it's gonna cause me a lot of pain later

0:26:42.720,0:26:43.350
in the future?

0:26:43.350,0:26:45.820
J.S.: So when you're writing your code good,[br]you

0:26:45.820,0:26:48.990
should be thinking about your models. Those[br]models are

0:26:48.990,0:26:51.520
gonna become your APIs. They're gonna become[br]your service

0:26:51.520,0:26:55.680
APIs. So consider your public methods. What[br]are you

0:26:55.680,0:26:58.650
putting in the public space of that model?[br]Is

0:26:58.650,0:27:00.760
it named well? Does it represent what your[br]service

0:27:00.760,0:27:03.110
should be doing?

0:27:03.110,0:27:05.950
Make sure that, while you're building up your[br]cobras,

0:27:05.950,0:27:08.950
that your models are reflective of the way[br]you

0:27:08.950,0:27:11.800
intend for your service APIs to look like,[br]should

0:27:11.800,0:27:15.090
you ever need to go down that road.

0:27:15.090,0:27:19.150
A.P.: And, like I said earlier, avoid tangling[br]those

0:27:19.150,0:27:23.850
components together. Specifically in Rails,[br]when you introduce associations,

0:27:23.850,0:27:26.290
you're kind of expanding that API that Jason[br]was

0:27:26.290,0:27:29.620
talking about. All those, now you're creating[br]ways for

0:27:29.620,0:27:33.630
developers to reach through these models and[br]get data,

0:27:33.630,0:27:36.460
and that'll couple them together and make[br]it harder

0:27:36.460,0:27:38.640
for you to separate them.

0:27:38.640,0:27:42.580
J.S.: Test. Who's here, who here tests? Anyone[br]test?

0:27:42.580,0:27:43.890
A.P.: Not DHH.

0:27:43.890,0:27:46.770
J.S.: Nope. You don't test anymore. You should[br]be

0:27:46.770,0:27:50.490
testing. You should be testing at high levels.[br]Avoid

0:27:50.490,0:27:54.000
the unit tests. If you can avoid the unit

0:27:54.000,0:27:56.960
tests. Especially because once you start doing[br]service extraction,

0:27:56.960,0:28:00.830
you will break assloads of unit tests.

0:28:00.830,0:28:02.450
Make sure you write your high-level tests[br]first. Make

0:28:02.450,0:28:05.050
sure you've got solid coverage on those high-level[br]end

0:28:05.050,0:28:09.750
to end tests. Secondly, as you are doing service

0:28:09.750,0:28:12.260
extraction, it is not trivial to be spinning[br]up

0:28:12.260,0:28:15.330
other services quickly in order to test end[br]to

0:28:15.330,0:28:17.550
end, but you should be thinking about how[br]you

0:28:17.550,0:28:20.370
might be doing that. Because otherwise you're[br]going to

0:28:20.370,0:28:22.610
be doing a lot of stubbing, and that gets

0:28:22.610,0:28:24.740
very painful and gets error-prone.

0:28:24.740,0:28:28.520
A.P.: I mean, when we talked to the developers

0:28:28.520,0:28:29.890
who had to do some of the tougher service

0:28:29.890,0:28:33.500
extractions, they were like, I wish we had[br]more

0:28:33.500,0:28:36.010
integration specs. Because we're gonna be[br]changing a lot

0:28:36.010,0:28:37.920
of this stuff, and we need to know if

0:28:37.920,0:28:40.350
it works. If you've got a good set of

0:28:40.350,0:28:43.170
integrations, integration tests, you can be[br]a lot more

0:28:43.170,0:28:45.880
confident about making those changes.

0:28:46.840,0:28:48.540
Next, over there?

0:28:48.540,0:28:49.040
J.S.: Yup.

0:28:50.220,0:28:53.420
A.P.: Yeah. So, you need to communicate. I[br]mean,

0:28:53.420,0:28:56.750
everyone always says this, but like, when[br]you solve

0:28:56.750,0:28:59.750
a problem, when you're spinning up a service,[br]you're

0:28:59.750,0:29:02.460
gonna, and as more teams are spinning up services,

0:29:02.460,0:29:04.370
a lot of you are gonna be encountering the

0:29:04.370,0:29:07.260
same problems. So when you solve a problem,[br]share

0:29:07.260,0:29:09.040
it. Make it a gem, write it down, put

0:29:09.040,0:29:11.230
it in a wiki, and tell people about it.

0:29:11.230,0:29:14.940
Give talks. Because it's gonna be hard to,[br]I

0:29:14.940,0:29:17.570
mean, you don't want people solving the same[br]problems.

0:29:17.570,0:29:21.700
At Groupon, we have this, Core Architecture[br]Forum, it's

0:29:21.700,0:29:24.220
called, and basically it's got a bunch of[br]people

0:29:24.220,0:29:27.050
who meet, and you can say, I'm gonna spin

0:29:27.050,0:29:28.970
up a new service, or I'm gonna solve this

0:29:28.970,0:29:31.950
problem. Have you seen this before? They're[br]gonna help

0:29:31.950,0:29:35.370
you answer questions like, what's, has someone[br]else solved

0:29:35.370,0:29:38.200
this already? Is there a similar problem?[br]Is there

0:29:38.200,0:29:40.390
a particular technology that would help you[br]solve that

0:29:40.390,0:29:43.690
problem better? All those questions are really[br]important to

0:29:43.690,0:29:48.600
ask so that you don't reinvent the wheel over

0:29:48.600,0:29:50.740
and over again.

0:29:50.740,0:29:54.540
What else? Oh yeah. One more thing. One more

0:29:54.540,0:29:55.060
thing. That sounds like Steve Jobs. One more[br]thing.

0:29:55.060,0:29:57.740
We have the interest, we have interest leagues[br]at

0:29:57.740,0:30:01.900
Groupon, which are just internal user groups[br]for Clojure,

0:30:01.900,0:30:04.760
Java. We even have one for onboarding. You[br]know,

0:30:04.760,0:30:06.930
there's are really cool. And that's another[br]way to

0:30:06.930,0:30:09.640
help communicate, like, what's happening.[br]Once your company gets

0:30:09.640,0:30:14.730
big enough, that's really important.

0:30:14.730,0:30:20.520
J.S.: So. In conclusion, cobras are great.

0:30:20.520,0:30:21.240
A.P.: Yeah. They're awesome.

0:30:21.240,0:30:24.000
J.S.: Rails is great. And cobras do serve[br]a

0:30:24.000,0:30:25.840
useful purpose.

0:30:25.840,0:30:32.150
A.P.: Oh. But beware. It's not so simple.

0:30:32.150,0:30:35.770
J.S.: Once you decide that you're gonna start[br]raising

0:30:35.770,0:30:40.590
up a baby cobra, be ready for what comes

0:30:40.590,0:30:41.220
next.

0:30:41.220,0:30:46.890
A.P.: Oh. Yeah. And. OK, so. Got his part.

0:30:46.890,0:30:50.820
We're hiring. I mean, if you want to come

0:30:50.820,0:30:53.670
help us solve some of these problems, come[br]talk

0:30:53.670,0:30:56.040
to us after the talk. There's a booth downstairs.

0:30:56.040,0:31:00.710
You can go to this website. Tweet at us.

0:31:00.710,0:31:04.340
I'd like that. But yeah. Join us.

0:31:04.340,0:31:06.820
J.S.: And we are standing on other people's[br]shoulders

0:31:06.820,0:31:06.890
here.

0:31:06.890,0:31:07.050
A.P.: Yeah.

0:31:07.050,0:31:09.570
J.S.: A lot of these folks are people who

0:31:09.570,0:31:12.210
helped with the talk or who helped actually[br]do

0:31:12.210,0:31:14.910
a lot of this service extraction work. This[br]does

0:31:14.910,0:31:18.760
not comprise the total list, but we definitely[br]wanted

0:31:18.760,0:31:20.250
to bring attention to these people.

0:31:20.250,0:31:22.060
A.P.: Yeah, and I mean. People like these[br]guys,

0:31:22.060,0:31:24.420
they gave us a lot of feedback when we

0:31:24.420,0:31:27.960
did the talk at, at Groupon. And having people

0:31:27.960,0:31:31.420
who will mentor and, like, spend time to help

0:31:31.420,0:31:34.470
you understand things, I mean, that's the[br]reason I

0:31:34.470,0:31:35.520
work at Groupon.

0:31:35.520,0:31:36.340
J.S.: Thank you all.

0:31:39.240,0:31:42.440
A.P.: [drowned out by applause]