Rails Conf 2013 Patterns of Basecamp's Application Architecture by David Heinemeier Hansson

[Introductory Music]

Welcome everybody.

Before I go into my talk about the

patterns of Basecamp's Application Architecture

A little announcement

about an hour and twenty minutes ago

we released release candidate one.
[Audience applause and cheers]

As well as branched for rails/4-0-stable.

So, we're really close to having 4-0 done
and shipped.

And it would be awesome if everybody who's here

who is going to work on Rails code during

the next four days could try it out.

Because that's the best way for us to

figure out what's still missing, or bugs, or
whatever else, have you.

We sort of have, almost, this pattern of
putting out betas and release candidates

and then we ship the final version and
fifty tickets come in with obvious defaults

or things that are broken.

So it'd be great if 4-0 could come out
and for the first week or two at least

we wouldn't have to ship an update.

So, if you guys can help us do that
that'd be amazing.

So, the story of Rails is in many
ways intertwined with the story of Basecamp.

This is a screenshot of the very first
version of Basecamp we released in 2004.

I started working on Basecamp in 2003, before
I had any aspirations of working on Rails.

Before I had any ideas that was I going
to work on was going to turn into a framework.

So, the whole approach with Basecamp
and with the creation of Rails was that

I was interested in an application.
I was interested in building something.

The fact that I had to build my own tools
to get there was sort of secondary.

It's certainly wasn't the primary objective.

And that led to, in some ways unique
style of working, I think.

That we've really embraced as a community,
and in Rails core.

That, frameworks are not inventions,
they're extractions.

We don't sit down and try to come up with
features that we could need, or would want.

We build applications, things that actually
have to run in the real world.

Things that actually have to matter to somebody.
Not example code.

And from there we see, what are the pieces
that we keep using over and over again?

For me, that's why Basecamp is so important.

Because -- just about, for sure, every
single good feature I've put into Rails

have come from Basecamp, or Basecamp
derivative.

A few bad features came exactly
when I thought I could invent good frameworks.

It -- it never really worked.

The question we're asking ourselves is not

what would someone, somewhere, might
want to do with Rails?

Because that sort of speculative development
style -- we've had to work

I've had to work with a lot of tools
that were built like that.

Teams assembled just to make tools, with

influences of listening to people in various
places.

But the core motive to just build tools.

And that is what I was running away from
in many ways as it came to Rails.

I didn't want something built by somebody

whose sole occupation was just to build
tools for me.

Because those tools tended to never really

to match the expectations I had for real-world
development.

So what we do instead is say:

I'm not going to care about anybody else right now.

I'm just going to focus on building Rails

as the perfect framework for building Basecamp.

That's my mental model.

The side effect is that Basecamp, of course,
is not that special.

Most of the things it does is same kind of things

you want to do in a lot of different applications.

But, because we attack the problems

from this sort of concrete approach

we get to extract something that just feels right.

Something that feels like somebody's actually

used it in anger.

I mean, I've sworn and cursed my fair amount at Rails at certain times.

So I've certainly used it in anger, and have changed things in anger.

I find that that's really a lot of where a lot of the magic happens.

When I'm so fucking annoyed that something

takes four lines of code when it really should

just take one, that's when good stuff happens.

And it happens in sort of a different way

than that example code driven development style

that you often run into when people do this for full-time living.

Because it's very easy to make pretty examples.

We have lots of this in Rails where -- for example

the standard scaffold for a long time

it looked like it was not very DRY.

It had a lot of duplication, right?

People would often come to Rails and they would say, as the first thing:

"Well this code you're generating it's verbose, I could optimize it in all these ways"

A few people have tried, with various gems.

And what you find is, yes, you can make example code

really pretty by making a certain set of choices.

But those same set of choices are usually the ones

that makes it horrible for actual real web development.

So, 2003, 2013, I've been working on Ruby on Rails

and Basecamp for 10 years now.

That's about a third of my life.

So, I don't think it's any stretch to say

this is my life's work, at least up until this point.

And, it's funny, because in some ways, it feels familiar.

When I looked at Rails code today, it looks and feels and smells

very much like the Rails code I wrote ten years ago.

Of course it changed in a myriad of different ways

but it's much, much better now.

But the kernals and the principles are very

much still the same.

And that's interesting to reflect on

when you compare it to the things that changed.

In 2003, this was a state of the art phone.
[Audience Laughs]

Seriously. Flip phones was really back then.

It's so funny because that seems so far away, right?

I mean, flip phones? Seems like another era.

Which it is in many ways

because many parts of technology move very fast.

It's interesting that we're here today and we're still

working with and discussing and debating and improving a framework

that is from the time of flip phones.

So, that's sort of like the form factor of actually

a lot of people interfacing with our applications today go through phones.

If we look at other parts of technology

just sort of the hardware we were on

I looked up these prices yesterday

and, it's like, half a gigabyte costs $50.

Now you can just 16 times as much RAM for almost half the price.

That's more than an order of magnitude change.

Which is just, it's one of those things that's also

interesting in terms of what does that change

in how we approach things?

Often times, people will almost lock their concept

of what hardware costs and how it runs to the time

they entered programming.

If something was scarce, and expensive

at the time they got into programming they'll

think that's how it's going to be forever.

Which, of course, this shows you can get

8 gigabytes for $29. How much RAM do you use?

Probably it doesn't matter as much as

a lot of other things. Which is funny

to me because that's always been --

Matz, back in 1993, 20 years ago, 20 years ago Matz started working on Ruby.

He had to care far -- I should've actually looked up

what RAM cost back in 1993.

But I'm sure it was fucking expensive.
[Audience Laughs]

Yet, he sort of had the foresight to think

well, I don't know if he had the foresight,

but it turned out, that the thoughts

that he was thinking about, how we should

make programming languages, and development environments,

like, I can make something now that -- ok, it's going to take more memory,

it's going to be slower, and it's not going to matter

because those things are going to improve

drastically over time.

What's not going to improve is programming brains.

They're pretty much as mush as they were back in 1993.

So we need to optimize for that.

So, that's sort of the hardware side, that that moves really fast.

Ok, not that -- we know that, Moore's Law, and all, right?

Also funny to look at the things from 2003.

Java was actually a real thing that people non-ironically wrote.

[Audience Laughter and Applause]

This was actually -- when Rails first came out,

I was talking to a lot of Java programmers

and they were serious.

[Audience Laughs]

And it's just funny to think back about this sort of battle, or whatever

that we had in the beginning. And this was the relevant foe.

This was who were trying to sort of say, there's a better way to implement

a lot of the good ideas that you have.

Because that was the second funny thing of course.

Because Rails is from 2003, and the bulk of the

thinking of how to structure, and create nicely

factored web applications, and those

types of applications came from the Java world.

It's just that they were trapped inside that beast

and we had to liberate it.

So, there are still people programming Java.

Of course, the JVM lives on and it seems like

that has a long and prosperous future.

And people are putting it to much better use

than the Java language itself.

The other thing, struts, that was actually

a serious framework at that time that we were

sort of trying to convince people why this was better

why XML situps and so on, were not something you should to do.

It's really interesting to think about all of these things

that have changed, and how much the world

of technology has changed in the 10 years Rails has been around.

I've been trying to reflect on why that is.

Why are we still here?

Why didn't Rails go the way of Struts?

Why wasn't it just a footnote in sort of a progression of things?

Where were all these other things that were supposed

to come afterwards. I certainly never

thought, when I started working on Rails

that I'd be here 10 years later and talking to

1,500 people at a sold-out conference.

That's a pretty measureable success.

That's surprising I'd say in technology.

Just like the flip phone is not here anymore

and Struts, thankfully is neither.

You should've thought that Rails would've gone away, right?

I think the stuff I talked about last time -- or last year

that pioneering spirit, I think is what makes the Ruby community

and the Rails community unique.

We are willing to continue west.

We don't just stop when we find a comfortable resting place.

There's a lot of other communities that value

stability far higher than we do.

That's not to say that stability is bad, it's just

a trade-off. If you look at say,

the Python guys, and how long they've taken

to get on Python 3, they're not there yet, by the way.

They're still going, and it's been years, right?

How long did it take the Ruby community to move

from 1.8 to 1.9 to 2.0? [Snap, snap, snap]

And it wasn't because it was painless.

How long did it take the Rails community to move

from Rails 1, to 2, to 3, to now 4?

Again, it wasn't painless.

Anytime you want to make progress like this

it's going to hurt.

But we as a community are willing to take that hurt.

That makes the community unique.

We're willing to sacrifice for progress.

I think what's interesting about that, the pioneering spirit,

that we're willing to continue west.

We don't just stop halfway and say,

"Ah, fuck it, Kansas is good enough"

[Audience Laughs]

A lot of people just -- they stop in Kansas.

Like that's just like, things are fine

I've got my cows and the sun comes up in the morning, great.

I mean, although, some Rails people have stopped in Kansas.

[Audience Laughs]

There are still a few people still running.

This is the half-life of Rails where we are now.

It came out in 2009, that's 4 years ago.

Anyway, if you're still in Kansas, keep

going west. California awaits you.

The west coast awaits you, Portland I should say actually, my bad.

We've continued west. We've gotten closer and closer

to this sort of promise of the west coast.

And I think as we've gotten closer and closer to that

and it's taken years and years, because again

it's a painful journey.

It's a perfectly legit thing to say,

I'm just not interested in any more progress, we're stopping in Kansas.

And I will go on, and I will do Rails 2.3 for the next 4 years.

A lot of people did and it wasn't like they were necessarily

miserable. I mean people do lead good lives in Kansas.

But, I think that there is something more important here

to keep on pressing on, and not just because

we keep our own interests and this is still interesting to go to

if we had just stopped as a community and said

Rails 2.3, that's our final resting place.

Ruby 1.8, what did we end up at? Ruby 1.8.9? 1.8.8? Whatever it was.

That's the final resting place, that's where we're gonna stop, right?

That's just where we are going to park our ambition and say, that's good enough.

I don't think we would be here because then these things would just be tools.

Nothing wrong with just being a tool.
[Audience Laughs]

There's just something more inspiring about pressing on and going further.

About this whole notion of pressing on and going further,

Joel Spolsky, in 2001, had this great essay: "Good Software Takes Ten Years, Get Used To It"

At the time I read that, it was funny, I read that

and was like "what a fucking jackass"

[Audience Laughs]

I can make something in fucking six months

that's going to blow the doors off whatever you've been

working on for ten years.

But I think he actually had the long end of something.

You can build something that's good for you in six months

and it could be "good software".

If you want to build something that has the kind of impact

on community, on the programming community,

on the world, for lack, for less sort of grandious vision.

It does take a long time. We've slowly but surely built this up over time.

That goes both for the work that I've put into Basecamp

and the work that I've put into Rails.

These things compound slowly over time if you keep investing in them.

You can see the sort of progression of investment that we've had with Rails

and the people who've been flowing into it on this chart.

New contributors with their first commit to Rails.

2004, first time I opened the code base and accept patches and let other people commit to it,

a few people, 40 people I think had a patch go into Rails in 2004.

The very next year, big boost, 280 people or whatever.

Then we sort of had like, I don't want to call it flat face

because this is all new people coming into the community

and starting to add something. But, for a long time

we had sort of a steady rate of new people

joining the community. And then something interesting

happens. In 2009 there's a small dip.

Everybody thinks the world is going to end and

everyone is getting fired and alright, I understand

and then after that, what happens?

There's a massive jump.

And this is actually funny. So in 2009, our Kansas,

so there's a baseline there, alright? We keep pushing.

And people respond. A lot of people are interested

in pushing on with us, interested in joining the caravan and continuing on.

Which is really just sort of interesting to see just

sort of the phase we can go through.

By 2009 I've been working on Rails for 6 years.

And we've had this steady inflow and then

all of a sudden lots of people get interested

in getting involved, awesome.

If you look at the same chart, just compounding

all of the committers, you can see it's more

of a steady climb, but the same pattern is there,

from 2009 and onwards, lots of new people got involved

because we kept pressing through.

We kept pressing through the comfortable phase.

Even though, the funny thing is, when you

do that you're sort of -- we have this

saying with Basecamp were you shouldn't be afraid to let customers go.

Sometimes customers are going to outgrow you.

They're going to think, well you no longer represent what they need in a

project management application, right?

A natural reaction to a lot of people, or for a lot of people

is to look, "Shit, we can't let that happen, no customer can leave, lock the doors"

"We will do whatever we can to take them happy"

We had that moment in 2009, where we could've said, let's just lock the doors.

Let's just improve things for people who are already here.

Big enough community, we're comfortable,

we could be self-sustaining for a long time, that could be fine.

We instead said nope, we're going to keep the doors open.

What that's going to mean is that there are going to be some people who are unhappy.

I've heard people say Rails 2.3 is the best version of Rails ever made.

[Audience Laughs]

Which is, it's sort of one of those funny nostalgic things of

you just sort of lock your timezone, like the 80's that's when music peaked.

[Audience Laughs]

It's just a funny phenomenon, and it's real, and I think

that's where a lot of technology circles an ecosystem, they stop.

And then they just stay there for the rest of life.

Anyway, we pushed on.

Here are the stats for the last 12 months running.

More than 1000 people were part of more than 8000 commits

to rails/rails GitHub repo. That is fucking amazing.

1000 people working, and this is just the people who get commits in.

We have so many more people doing all sorts of

other stuff around the codebase.

This is just the people who got a commit into rails/rails itself.

That is really and truly astounding.

And I think that that shows and validates

the approach, the uncomfortable approach that we have

to keep pushing and to keep getting better

even if it will ruffle the feathers of the people we already have.

The customers, the developers, who are already using Rails

that are somewhat uncomfortable with whatever new direction we have.

It's also a funny match to Roger's Innovation Adoption Curve Idea.

This is sort of an analysis describing how products are adopted.

So obviously, from the beginning we had a lot of innovators.

People who would use Rails even though there were

a lot of things that were broken or didn't do whatever in the beginning.

Small slice of the overall market of developers.

Okay, we've got the early adopters.

I think that's sort of our first crest.

And you sort of hit this point where you have to

make the jump from early adoptors to the

early minority. I think that's exactly what that point in 2009 shows us.

We did make that jump. We are now on sort of the next ride up.

Which is, which is pretty cool.

And I think it's also interesting for me in -- the original reason

I got into open source, the original reason

I wanted to release Rails was two-fold,

first, I wanted to give back. I had been

using Ruby, I had been using Linux,

I had been using MySQL, I had been using Apache

I'd been using all these open source projects

so, when I had something myself to share

I felt like, dude you're a dick if you don't release

something like this that other people could find value in,

when you've used all of their stuff.

So there's that. Don't be a dick.

The second part is, I'm having a lot of fun

doing this stuff. Ruby is really a great programming language

Rails enables me to use Ruby to build Basecamp.

There's going to be a lot of other people out there

who could benefit from Ruby so let's

get as much programming joy out there in the world.

So that's why this stuff is interesting because

we've having a bigger impact, we're getting

more people involved with enjoying their work.

And not being stuck in Struts, or whatever painful torture instruments of the time.

Anyway, so I think that's all great.

And it's great to see there are more people getting involved.

But you're also sitting there and have to stop

and say okay, what are we actually doing here?

What is the purpose? Why are we still working on this?

What is the improvement path? Who are we?

Who do we want to be?

So, I think a while back, decided we want

to have a big tent. There should be room

for a lot of different people in the Rails community.

Jugglers, musicians, clowns and elephants all invited, come on.

And we pulled some things in, we did

with Rails 3, we had a influx of good new ideas

from, at the time, a competing framework

called merb, which was sort of a

good reflection of okay, it can be a big tent.

We can have room for a lot of people in here

who might not agree on everything,

they might not all want to build Basecamp

but we have a general principle, a general purpose

that this is where we're going, and

we agree on these things. And once we

sit down and actually work on code together

we realize we share more opinions than we don't.

So, even so, even if you have a big tent

and you have room for jugglers, and elephants and so forth

and you're having fun, you're not a ferris wheel.

There's got to be something you're not.

You have to sort of define it in some sort of opposition

there's limits to what this is, because if

you try to be everything to everyone all the time

you're nothing to nobody none of the time.

Second, the title of this talk is "Patterns of Basecamp's Application Architecture"

The whole idea of patterns is that you see a problem

and there's a solution, but there's a third element,

there's a context. There's a context in which

that problem arises. Because you can encounter

the same problem in two different contexts

and you're going to want two different solutions to that.

So, what is the context? What is the context of Rails?

Of the work that I'm doing on Basecamp?

For me, that context is Dynamic hypertext documents.

What does that mean?

It means that this is sort of what I'm trying to produce.

It's funny because when you see HTML it looks like

it's such a good, there are books like Learn HTML in 4 Days, right?

It seems like such a low-level thing, it seems like such a simple thing,

like sort of beneath us, kind of thing, we are "serious programmers" who do "serious work",

like, all we're doing is just generating HTML, that's -- I mean, c'mon, that's for HTML monkeys, right?

Like, I need something more grand. Well, to me

the grandness is in embracing the simplicity.

Is embracing what the web is, and not just what

the web and HTML as the delivery mechanism,

sort of the web and HTML as sort of a lot of

side effects that are neat, in term of software deployment and development,

don't have to install software, you just get it from one place.

You could deliver a lot of different kinds of software where

you just use HTML as a delivery mechanism.

But, when I've been revisiting and going back through the Basecamp code base

and especially the work I've done with the new verison of Basecamp.

I realize that it's deeper than that, that this is not just a delivery mechanism.

That the actual structure of HTML, the approach to the context of

software development as a document is profound.

That profound approach is not something we should try to escape.

It's something we should try to embrace.

So, you can think about a lot of different applications that are being delivered on the web

on sort of on a contiuum. On the one end

you have something that is basically purely document based

and on the other hand you could have something that's

almost like a GUI using HTML as a delivery mechanism

but you certainly wouldn't describe it as a document.

I think if you look at that sort of continuum from document to GUI and nothing falls perfectly at either hand.

Perhaps these two, I mean Wikipedia is just a collection of documents.

It's a pretty good to, I think, my vision at least of

what I'm interested in, the kinds of applications

I'm interested in, and thus, the kinds of tools

I'm interested in building. I want to build tools

for a document based web. That approach to software development is quite

different from all sorts of other approaches that went before.

If you look at software from before the web

it looks different. I've been struggling to find a word to

describe that style of applications, GUI is the best

I can come up with, even though of course the web is a GUI, too.

You sort of know what I mean.

Maybe it's easier if you look at an example.

If you look at Google maps for example, that doesn't feel

like that's a document. That feels like

this is using HTML as a delivery mechanism for

the application that it provides. It doesn't mean

it's not a great application. It's a great application.

It just means that what we're trying to do is optimize for

a version and vision for web applications that is more

document based, this is not the primary thing

we should optimize for. So, if you're trying to

write the next Google Maps, maybe Rails, at least as

a whole package, as a full-stack framework isn't

the natural fit. You can pick parts out and you can have

a good HTTP processor and you can do all sorts of

other things that's interesting.

So, if that' 100% the camp that you're in, the

sort of not related to documents at all, maybe that's not it.

Then it becomes interesting, because as I said,

a few things are either one thing or another thing completely.

If you look at Basecamp for example, we have this

calendar there, that doesn't feel very much like

a document. It doesn't feel like it fits HTML, what's the section

what's the header, what's the paragraphs here?

It feels like we're using HTML as a delivery mechanism.

But it's a minority thing part of what we do

so we try to fit it in and it should be possible, but it's

not the main thing. The main thing is more

stuff like this. Where it actually does look and

feel like a document. This looks and feels

like something you would not have before the web

this would not be like what an application looked like.

Because this is, that native development style of the web.

If you look at something like GitHub, same thing,

it feels like a document, it feels like a document that's decorated with some dynamic behavior

and that's great, but the key part of it, it's essence and it's being, is the document itself.

It's not just a delivery mechanism. That's the, the slice, and the part of information technology

that I'm interested in. These kinds of document focused

applications are not a bad thing. It's not something

we should be trying to escape, the constraints

that it puts on us to develop an application

in this style, they're liberating.

They're not bad. We don't want to start

with an empty canvas and have to draw everything

from scratch. The fact that HTML is there, and gives us

a sort of semantically relevant containers to put

certain things in, is a benefits, it's great.

And, that's where, I guess, the insight for me is

why is it that I was never interested in Java Applets?

It wasn't just because it was Java, it's horrible enough in itself,

but it was the notion that this was just a blank canvas,

you could draw whatever you want, there are no limits here,

you don't have to fall in line to some predescribed notion of a document.

Or set GUI elements for a document, forms and so forth.

You can do whatever you want.

I don't wanna do whatever I want. I want a container, a base to put my ideas in.

That's when I find, that they flourish.

So, with that insight in mind, and with the insight in mind

that the web is the most successful software development platform of all human history,

that leaves me to think that it's not acutally that big a surprise that this failed.

Because, the architects of this sort of canvas-base, let's

use the web as a sort of delivery mechanism for

the same application architecture that we had on the desktop

they're wrong. That's not the part of the web that's interesting.

It's not just these accidental properties of oh, it's easy

to update software because it runs on my server.

The things that brought the web to it's prominence with users

and developers alike, we're the constraints.

Was the fact that that style of interfacing with information technology is

really user-friendly, really development-friendly, it's different.

Not in a worse way. And that's usually what I --

I had conversation, I think it was Rails Conf, maybe 7, or something,

and some Microsoft guy was coming up and he was going

to tell me about serverlite, and the pitch was

"I'm finally going to free you from HTML"