STEPHAN HAGEMANN: Thanks for coming.
You came to Big Rails.
If you did not intend to go to Big Rails.
My wife's actually currently in a plane,
and so for some reason I had to think of a plane metaphor.
And her flight just got canceled.
If you did not intend to go to Big
Rails, which I can understand. It can be messy.
Better get out now, the flight might be bumpy.
So I want to talk about really large Rails
apps, and my buddy Austin suggested, a couple days
ago, let's use GitHub. Search for Rails. Sort by
size. This is officially the biggest Rails app you've
never heard of on GitHub. It's open source. It
also only has eight models. Which is because they
upload all their assets.
It's a hundred and fifty megabytes big, and that's
not the kind of big I'm talking about. I
want to talk about component-based Rails architectures and why
you should refactor towards them. And how to do
that.
Component-based Rails architectures is a term I use. I
tag my Tweets about it as #cbra, and I'm
the only person in the world doing it. But
if you use that too, I'm certainly replying. And,
here's a memo for that.
If I can get one idea in straight off
the bat, then it is that this is, this
stuff is easy. If you can think about your
application by writing, by painting boxes and drawing arrows
for things that depend on each other and how
they interact, then you know everything else you need
to know about component-based architectures and you just need
to start doing them to what I think will
be improving the quality of your application and the
quality of your code.
Since I called this talk refactoring towards component-based Rails
architectures, this talk assumes that you all know what
that is. So, no. I will actually explain what
that is to you. You're not screwed if you
don't know what that is.
So, I started talking and actually writing applications pretty
much exclusively in this kind of odd way, so,
like, two, three years ago. And it, I wrote
this sample app that you find at this address.
I'm shageman with one n on GitHub. And there's
just the_next_big_thing sample app.
It does nothing. Only it shows you the structure
and how to hook certain things up. And let
me quickly go through what that means. So this
is the root of the project. And, as you
can see, it's not a Rails application. But it
is. There's actually no particular prescribed way on how
to do this. In many apps where I've done
this, the Rails application is still at the root
of the project. But in this particular one, I
chose the other version, where Rails moves into a
sub-folder and, as you can see here, so this
looks pretty much like Rails. But it doesn't have
an app folder. There, and you might be asking,
is there nothing happening in this app?
Well, there is. Here, we see there's, at least
one thing is mounted. There is an engine that's
mounted. So, Teaser::Engine. I don't know if anyone has
ever downloaded that from RubyGems. Well, don't try. I
don't know what that gem would be. But I
didn't upload it there.
Let's look at the gem file to find out
what that gem is. So this is the, the
only kind of trick that you need to employ
when you want to write component-based architectures. And that
is, you use gems as not a, not as
a distribution mechanism, but just as a packaging mechanism
for parts of your application.
You leave them in the same source tree and
you just reference them. And then you can do
everything you're used to with those things.
So, as you can see here, if we go
to the components folder, we expect to see a
teaser folder. Ah. Which we do. And there is
an engine. So that's just a Rails. Ah, sorry.
A RubyGem that is pimped a little bit so
it has Active star. All the Actives. So Rails
in it. And this, indeed, now looks like an
application.
If you don't know what an engine is, it's
essentially Rails minus everything you need to actually get
it booted. So, if we look in here, what
we can see, you know, there's a routes file
and this engine is doing something, right. There's. It's
normal. Here we'll look in apps. So the controller.
So you can see this is namespaced under the
name, which is nice. But it's just, you know,
your ordinary controller. But, again, everything's namespaced.
There's assets in here. Yeah. This is pretty normal,
standard stuff. We even have migrations down here in
this engine. What else would I talk about? Well,
ah, yes. Maybe the coolest part. There's specs in
here. I like to use rSpec. That could be
tests in here if you're doing this and liking
those better.
But there's, there's a spec here for this controller
that I just looked at, and the cool thing
about that, and I don't know how to emphasize
it more than to come up to the front
of the stage and say it's very important that
I can run these tests, and the only thing
that these tests are gonna load is the code
from this engine, and not from all the other
stuff in my Big Rails application.
So, just by there being a spec folder in
here, I have a part of which I can
prove that it is small. Or, at least, as
big as only this part, and not depending on
anything else.
I, for, what I typically do, then, to kind
of bind these components together - and there's more
in this folder, as you can see. There's this
event counter signup and whatnot. So I write a
little script here that binds all the tests. As
you can see there's some request specs and some
Jasmine specs, and if you look into the other
folders, you find more of these test files. And
then in the root. I typically put this, this
build script here. And what this does is loops
over all the folders and tries to find test
files and runs them.
So you can still run this as the app,
as the specs to one application. But if you
just change one part, you can just test that
one part and still be sure that it works.
So, now that you know what these applications are,
you don't need to think about all those little
things. How to hook them up. You'll find out
about that when you try to do it. What's
important about them is that you can now talk
about your application as a sum of smaller parts.
For example, the app that I just showed you
is a empty Rails container that mounts an engine
that uses two other engines that I didn't talk
about and another gem.
This is a project that it recently, it o-
that was a tiny project, so this was Tiny
Rails. Empty Rails container. Two engines. The one was
using two gems to talk to APIs, and the
other one was just straight-up, pretty much normal application.
Or this site that was a travel site. It
was so small, we didn't even bother to use
engines. We just put a couple gems in there
that were talking to APIs.
This site was a bit bigger. It started out
like this. We had six engines, one gem, and
it ended up about like this. But this is
a lie, because there were about twice as many
engines, only the other ones are abstracted away for
clarity in this picture. As you can see, it's
a very clear picture.
Now, this looks chaotic. But the fact that I
can draw such a chaotic picture about your application,
but if you, if you were to draw the
arrows a bit straighter, you would see that the
dependencies all go in one direction, and that there
are parts that other parts are based upon, right.
So this app is composed of parts, and it's
no longer a ball of mud.
The fact that you can see that this is
a complicated domain is an improvement, despite this picture
looking a bit weird. This picture is from Ben
Smith's talk about how he architected his Big Rails
app for success, which he gave at Rocky Mountain
Ruby last year. And I recommend you watch that
talk.
So, if you're like, what? Now? Why would I
do that? As I just said. Because it helps.
When applications get big, I believe it's fundamentally important
to be able to think about the parts of
the application independently, as much as possible.
There's a bunch of resources that, you know. I've
given talks about this. Ben's given talks about this.
There's blog posts about this. And there's, most importantly,
that repository that I recommend you look at.
OK. Back to Big Rails. How big? Does anyone
here think they're working on a big Rails application?
OK. That's about a third. So, I've, I think
I've worked on a couple, but I didn't know
what to show, because I work for Pivatol Labs
and we consult and the code is our clients',
so I can't show it here, so I won't.
But I went onto this thing called Google and
I searched for open source Rails apps, and there
they were. So these claim to be big Rails
apps.
How big, I wondered. Well, if you have, currently,
on your laptop, your company's code, please, execute this
script and Tweet the result with a #cbra hashtag,
cause this is just gonna open how many lines
of code your app has.
And here's the result for this list of applications.
So, as you can see, they're not all simi-
not all the same interpretation of big. So we're
going from about 400,000 lines of code to still
in the thousands. If you look at the files,
same picture. Thousands of files for the biggest apps,
and still hundreds for the smaller ones.
I'll accept that as big. And I hope you
will too. Ever wonder what happens to the complexity
within a system as it grows? Something like this
happens. And maybe this is bad as that, and
you just can't take it anymore.
When you introduce structure, you fundamentally change the game.
Oh. I'm supposed to say, first, that this is
because the number of interactions between the elements inside
the system grows somewhat like an exponential, in an
exponential function when you add new, new parts to
this system.
So it explodes in complexity. Well, what if you
were able to split a system and use a
complex web of service objects, maybe, to connect the
two. But you have two smaller parts that can
now, that are now isolated, in some way, what
happens to the complexity?
Well, turns out, it's still this, and probably still
that. But only half as fast. And that's good.
If you. I wish I could run up to
the slide right now. But, you see that green
line at the bottom? It's almost flat. Now imagine
you could split it again, and again, and again,
and you always stay in this flatter part, so
your complexity never really explodes. You get those crazy
diagrams that I was showing earlier, but your complexity
doesn't explode quite as badly. You're still writing a
huge app, and that's something that I'm not gonna
discuss away. But you might be able to manage
it better.
The rich get rich and the poor get. Anyone?
Children. I. This is from a song from the
1920s, and while it's funny, the, the first part
of this alludes to a thing called preferential attachment.
Preferential attachment is a set of processes where, when,
where, god. This is so hard to say. Where,
when you, where the amount of things you have,
whatever that might be - the thing, or you
- defines how much you will get.
So, 2006, Chris Anderson I think, wrote about the
long tail in, in publishing and books. And he,
this, this curve became very popular. He called, I
think he even called the book The Long Tail.
And he was concentrated on the right side of
this tail.
So you have many, many small things that don't
get any traction. But, you know, Netflix allows us
to view all those movies that no one wants,
that no one else wants to see. But there's
just a very few movies at the front that
everyone wants to see. The Blockbusters. And, while the
long tail is very interesting, it's the ones in
the front that harvest most of the benefits, well
I guess in the case of movies.
So, the green to the yellow relationship is about,
80% of the, the tail versus 20% is inverse
in the relationship to how much they have. So
it's 20/80, 80/20.
And, preferential attachment, as I said, happens when you
already have a lot, so you get more. And
I have seen so many code bases that I
have the feeling there's something like that going on
in code. So I looked at these code bases,
and again, I typed in this bash script. First
try of course.
And I urge you to do the same. Now,
you will not be able to Tweet this, because
this lists all the files, all the Ruby files,
in your app by size. And when you do
that, that looks about like this for these applications.
Now, does that not look similar?
There's another way of saying this, you know, the
rich get richer. There's a German proverb. I'm from
Germany. Hi. That proverb is that Der Teufel scheißt
immer auf den größten Haufen. Or, the Devil always
shits on the biggest pile.
And I think something like that is going on
here. And which file might that be? So who
thinks that might be a good guess as to
which file that is? And, lower your, don't lower
your hand, because it's probably actually yours. I said
this two days ago and then the guy next
to me shows me his Ruby, his user file
and it's literally this long and I'm like, oh
god, don't do this to me.
But everyone else knows this problem. The user is
around earlier. He is important to the system, so
he attaches, he attracts functionality. But to the point
where I would say that is a pile of
shit, man.
And I thought that would be something else I
could find. So I looked at associations. Associations are
another way of expressing dependencies. In fact, so when,
I want to talk about has_manys, because I find
them particularly interesting.
So, you have something that has_many. User, maybe. So
since I assumed it was user, I want you
to read this code and, kind of reflecting on
yesterday's keynote, I find this file very readable. The
user has_many :cars, :flowers, :trees, :houses, :moods, :checks. Hopefully.
I think we can agree that it's very readable.
It makes a lot of sense and we can,
we can empathize with this class.
Have you ever wondered what's funny about this when
you look at the table and there is no
mention of :cars, :flowers, :trees, :houses, :moods, and :checks
in that table at all? You don't find it
funny to write that user class, but would you
find it funny if I wrote a color class
like that?
Why do we not write that color class? Why
do we always write that user class? I'll tell
you why. Because users are important to us, and
we always write them first, and then we attach
all sorts of stuff to them that doesn't belong
to them. This table clearly says what belongs to,
sorry, a user. Namely, in this case, a name
and a home.
So I've kind of gotten the opinion that we
shouldn't be using has_many too much. You can check,
and I would be very interested in the results,
again, so there's another gist. Please, try it out
if you can.
Whoops. And guess which file is on the left?
Yeah. He shat on that again. The, it's the
user. And it's ot always actually the user. I
shouldn't be saying that. I should say, it's probably
always, to some extent, the user. But what's your
domain? If your domain is, I don't know, money?
Houses? Properties? I don't know. It's probably that file.
Whatever I just said or whatever is in your
name. That's that file.
And that attracts all the functionality. But the user
is typically not far behind. And, because I want
to. Yeah, I just. I think this is very
enlightening. So, I just opened to the project I
just showed you. I opened all the user files,
and I'm just gonna scroll through them, K.
This is not very intimidating. Eh. This is a
bit better. Ooh. I'm trying to be non-discriminatory here
and go the same speed
all the time. Ooh.
Oh. Oh you wait. Now, I have to speed
up.
And it's, you, this one file looks particularly bad
maybe, but it's probably the case that these user
files are big to the extent that their applications
are big. These files just tend to do that.
These models just attract that.
So, what does that mean?
To recap. We have things that grow, and when
they grow, they get exponentially more, they get more
complex. They get exponentially more complex. And then in
those things, there are certain things that are, you
know, that are the top of the badness scale.
And, and those things that also depend on everything
else in the system.
I don't know. I think we can do better.
We can try to reduce the size of the
parts. Maybe flatten out that curve and reduce the
number of dependencies in between those parts. Something that
has helped me in doing that has been Solid,
the five ideas, principles. Signs, truths. I don't know.
No.
And especially for this problem, the Single Responsibility Principle.
I think if you just apply that as a
rule of thumb, as a, as a starter for
a conversation, it will, over time, on average, improve
the quality of your code. But I think there's
something funny going on when we discuss single responsibility.
Oh. Yes. What's single responsibility for? To find things
that don't belong. Clearly it's the cat with the
green eyes.
Where, where to apply Single Responsibility Principle? If you've
been to any of the refactoring talks - I
didn't have a chance this time - but I'm
pretty sure you refactored some methods. We look at
methods, we want them to be better. We make
them smaller. We find what they are about and
we give them one responsibility. Class. We probably did
the same, right. We look at a class, we're
like, oh, this does a little bit much. But
then we're like, OK, ActiveRecord is cool. I'm like,
allowing it to save itself and soforth. So maybe
that's OK. But it still kind of should be
about this one thing.
Except for user.
So we're, on these two levels, I think we're
really good. Module, and by that I mean a
namespace. If you use module as mixin, I get
like that I don't want to talk about that,
and that's why I continue to say namespace. Why
do I think this about mixins? I should maybe
explain in one sentence. If you mixin stuff into
your objects, the only thing you're doing is hiding
away the fact that you have more complexity than
you can bear to look at on one screen,
so you put it on another so you don't
have to see it. I would not do that.
I mean, I have done it. I have written
apps like that, and now I hate myself for
it. But, I want to not do it anymore.
So I think we shouldn't. So I use modules
a lot, but as namespaces, to group things together.
I think you should do that. You know, look
at modules or namespaces and, and analyze whether they
have one responsibility. But have you done that lately?
It's harder, right? It doesn't mean a lot. You
can, it doesn't, you know, no one cares if
you go across these things. You just, you can
load any class anyways.
So I think we could get better at that.
Let me jump a level. Application. And I think,
again, you should. And the fact that we have
this Big Rails track, and the fact that I'm
talking about components, but pretty much everyone else is
talking about SOAs, shows that we are talking about,
what do applications mean? What does it mean for
an application to do something? Or, yeah. To have
a function in, in something bigger.
So, I encourage you to go to all those
tracks and make your applications better on that level
as well. But I just talked to you about
the component, and I think that sits in the
middle. And in, in a certain way, the component
is, is an improved namespace. Because of that thing
I said earlier, namely, I can run tests and
I can prove that those tests run without ever
touching any code that is not in that namespace.
So, now you can. And I urge you to
do it. Because the cat picture, the discussion of
what doesn't belong. If you are in your Big
Rails app and you're writ- painting all those classes
as boxes and the dependencies they have as arrows,
first off, it's really easy to screw that up.
Because there's no, you can't really prove that you're
not using another class. At least not in run
time with big applications.
So components give you an arbitrarily scalable and provably
independent way to talk about parts of your application.
So I use it. Now, I said you should
look at SOAs, and when, if you're considering refactoring
your application to improve its quality and to make
it more maintainable, I think there are a couple
things you should know about or consider when deciding
between the two. Right, you're kind of feeling pains
right now, or you're not even yet feeling pains.
Great place to be in, too. But there's a
couple things that are really cool about just splitting
up your application into these components.
And that is, you stick with one repo. You
have one test suite. But you can split it
up. You have, still, one deployment. You can, however,
use, like, a load balancer to push traffic to
certain parts of the application to make the one
faster, the other, you know, has less of a
demand or, you know, you can go crazy. But
you don't get these additional versioning constraints of where,
like, if you, if you have this app and
it needs that app and that app doesn't have
the API yet, then I need to add this
stuff here at the same time, and I need
to deploy them at the same time. And if
I didn't do that then everything goes down.
You don't have that because you're still talking about
one application. And also, one other thing that's very
interesting to me, at least, is that, I never
know which parts I need. On this crazy graph
project that I just showed you, it went on
for about six months. And, and there were three
pairs on that. And at some point. Four pairs.
And at some point, there was always one pair
refactoring something. And they created, I think there were
a couple weeks where they created basically a component
a day. But they also removed a component.
Try to do that with a SOA. Try to
move that stuff around quickly. You can do a
SOA, I think, when you, when you're sure you
need pieces. But when you're playing around with parts
of your application and you, you want to just
structure it, and you want to still be able
to move a part from one to the other,
SOA is a bit heavy.
Within a SOA, you can still think about your
applications as being built of these components. There's, you
know, nothing is exclusive there. Just do both.
So, I've now spent about, most of my time
telling you why you should refactor towards this. But
I haven't done what I said in my title
I would do, namely tell you how to refactor
towards it. I'm sorry about that. But I gave
you this idea, right. And you all believed me.
This is easy.
Today's day of the, International Day of the Book.
I looked in these books. Refactoring to Patterns. Refactoring.
Awesome books. But they don't talk about this either.
And I think the reason is that this stuff
is not very important in other languages. In Java,
you do packages every day and you have implicit
imports. In Go, you do that. Other languages do
not have this problem, because other languages do not
allow for this kind of freedom and openness that
we've gotten because we do Ruby.
So, I call these two refactorings, teasing out an
app component and extracting a functional component. And the
reason I don't want to spend much time on
it is because it's actually very simple to do
it. The devil's just in the details.
And there is no book about this. But I'm
trying to write one. And if you, I don't
know, show, if you're interested in this, send a
message there and I might be more inclined to
actually sit down.
All right. But to the first one - teasing
out an app component. So, step zero. Got tests?
If, if you don't have tests, I wouldn't move
as much code as we're about to move. I
mean, virtually. But, you should add some tests around
the stuff that you want to extract.
And then you, for the app component. So that's
this teaser thing we looked at. So we're, a
component of your application, where you almost thing, this
could be its own application, maybe. So you look
for this vertical. Views, controllers, models. You find that
vertical that makes sense on its own and you
namespace it.
A funny thing happens when you just namespace it.
Suddenly, the things in there. Well, first off, what's
awful about it is you have to rename all
your tables. OK. But that aside, you suddenly start
seeing what's actually in your vertical and what's not.
You could actually put colon colon in front of
everything else that's not, and you would get a
really easy view parsing or, or way of seeing
in your files what's not in your vertical.
Those are your external dependencies to, an external to
the app. So when you do this at first,
you're apps gonna be unhappy. So please make your
tests green. And to do that, you will need
to find other parts that you didn't know you
depended upon, and you move them into what's gonna
be the component. Into this vertical.
And when you got this vertical green, you extract
it into an engine. Extracting into an engine is
rails plugin new component dash dash full, dash dash
mountable. In most cases.
It doesn't matter.
And step three, profit. Now, most of the time,
this vertical will not be as easy to find
as I just pretended it would be. And that's
because of, you guessed it, user. Right? The user
is probably somewhere, and other things as well, are
used somewhere, are needed somewhere, but don't really belong
to this verticle.
And what I've found is when you refactor towards
this, if you abstract. If you, if you're coming
from one big piece, if you want to abstract
one thing, you're gonna abstract two. The one thing
you wanted and the common stuff. But please do
not call that stuff common, because that's just gonna
be another pile of shit that you can attach
stuff to. Call it something else. Call it user
if you have to.
So this special case is not a special case.
I think this is, of course there is lots
of meat in how to do this and where
you get hung up on certain gem dependencies and
engines not doing it exactly the way that Rails
does it and blah, blah, blah. But essentially this
is it.
The other side to this is, is, something even
more interesting. It's abstracting functional components. Now your app
is round, suddenly. I don't know why. Again, got
tests? Let's start there.
So you find a functional component, and yes, I
made that much smaller than that verticle. Because typically
you find a piece that is really, really separate.
A very special functionality. Maybe even something mathematically a
function. In and out. Some processing. You find that
and you create a gem.
And you move the tests in there as well,
and your gem is, when you run those tests,
the gem's gonna be unhappy, because of all those
dependencies that you didn't see coming. So, you take
all those and you move them in. And you
make your tests happy.
Well, and now you require that gem in the
main app, and your main app is totally unhappy.
Because it doesn't know how to talk to that
gem anymore. And if there were any ActiveRecord dependencies,
or other dependencies on, on stuff that came from
Rails, I would urge you to try and keep
that out of, of these kinds of gems, because
it just adds such a huge surface area and
stuff that could happen, could be done and could
go wrong.
I, I would urge you to find these connections,
these, the shims, maybe. Call them ports and adapters.
And move them in the app closer to the
surface area of this gem, so that you can
do the processing of whatever that is there, and
that talks to the gem and makes it work
again. So now the app is happy. And you
got this weird kind of lump on the side.
But that's a good thing.
In, in one of the books, actually I think
it's Eric Evans, it's called Bound to Context. Now
you have a thing that is only relevant for
a context. But that's how you take from that,
from that curve that has the huge left side.
That's how you chip away at the top and
put it somewhere at the end. Because this thing,
in a bound context, it only has one meaning.
And that's why it's gonna be smaller.
So, step three. Profit.
I hope that made sense. And I know. It
doesn't work that easily. It's not five steps. It
can take weeks. The first time we tried this,
it took three attempts and four weeks in one
app that we wanted to split into two. And
we ended up with three, and now I think
it's about ten. But, trust me, you can get
there.
And I kind of felt challenged by Far- by
Farrah's keynote yesterday. So, I will help. If you
want to attempt this, and you don't find the
resources and any of those links that I gave
you, Tweet at me, talk to me, send me
an email. I'll respond to all of them. Maybe
publicly so it helps others, too. I just think
this is a great way to structure applications and
to make them better, and that's why I just
will help.
So don't let these guys ruin your day. Make
friends with the #cbra, and thanks for your attention.