[Introductory Music]
Welcome everybody.
Before I go into my talk about the
patterns of Basecamp's Application Architecture
A little announcement
about an hour and twenty minutes ago
we released release candidate one.
[Audience applause and cheers]
As well as branched for rails/4-0-stable.
So, we're really close to having 4-0 done
and shipped.
And it would be awesome if everybody who's here
who is going to work on Rails code during
the next four days could try it out.
Because that's the best way for us to
figure out what's still missing, or bugs, or
whatever else, have you.
We sort of have, almost, this pattern of
putting out betas and release candidates
and then we ship the final version and
fifty tickets come in with obvious defaults
or things that are broken.
So it'd be great if 4-0 could come out
and for the first week or two at least
we wouldn't have to ship an update.
So, if you guys can help us do that
that'd be amazing.
So, the story of Rails is in many
ways intertwined with the story of Basecamp.
This is a screenshot of the very first
version of Basecamp we released in 2004.
I started working on Basecamp in 2003, before
I had any aspirations of working on Rails.
Before I had any ideas that was I going
to work on was going to turn into a framework.
So, the whole approach with Basecamp
and with the creation of Rails was that
I was interested in an application.
I was interested in building something.
The fact that I had to build my own tools
to get there was sort of secondary.
It's certainly wasn't the primary objective.
And that led to, in some ways unique
style of working, I think.
That we've really embraced as a community,
and in Rails core.
That, frameworks are not inventions,
they're extractions.
We don't sit down and try to come up with
features that we could need, or would want.
We build applications, things that actually
have to run in the real world.
Things that actually have to matter to somebody.
Not example code.
And from there we see, what are the pieces
that we keep using over and over again?
For me, that's why Basecamp is so important.
Because -- just about, for sure, every
single good feature I've put into Rails
have come from Basecamp, or Basecamp
derivative.
A few bad features came exactly
when I thought I could invent good frameworks.
It -- it never really worked.
The question we're asking ourselves is not
what would someone, somewhere, might
want to do with Rails?
Because that sort of speculative development
style -- we've had to work
I've had to work with a lot of tools
that were built like that.
Teams assembled just to make tools, with
influences of listening to people in various
places.
But the core motive to just build tools.
And that is what I was running away from
in many ways as it came to Rails.
I didn't want something built by somebody
whose sole occupation was just to build
tools for me.
Because those tools tended to never really
to match the expectations I had for real-world
development.
So what we do instead is say:
I'm not going to care about anybody else right now.
I'm just going to focus on building Rails
as the perfect framework for building Basecamp.
That's my mental model.
The side effect is that Basecamp, of course,
is not that special.
Most of the things it does is same kind of things
you want to do in a lot of different applications.
But, because we attack the problems
from this sort of concrete approach
we get to extract something that just feels right.
Something that feels like somebody's actually
used it in anger.
I mean, I've sworn and cursed my fair amount at Rails at certain times.
So I've certainly used it in anger, and have changed things in anger.
I find that that's really a lot of where a lot of the magic happens.
When I'm so fucking annoyed that something
takes four lines of code when it really should
just take one, that's when good stuff happens.
And it happens in sort of a different way
than that example code driven development style
that you often run into when people do this for full-time living.
Because it's very easy to make pretty examples.
We have lots of this in Rails where -- for example
the standard scaffold for a long time
it looked like it was not very DRY.
It had a lot of duplication, right?
People would often come to Rails and they would say, as the first thing:
"Well this code you're generating it's verbose, I could optimize it in all these ways"
A few people have tried, with various gems.
And what you find is, yes, you can make example code
really pretty by making a certain set of choices.
But those same set of choices are usually the ones
that makes it horrible for actual real web development.
So, 2003, 2013, I've been working on Ruby on Rails
and Basecamp for 10 years now.
That's about a third of my life.
So, I don't think it's any stretch to say
this is my life's work, at least up until this point.
And, it's funny, because in some ways, it feels familiar.
When I looked at Rails code today, it looks and feels and smells
very much like the Rails code I wrote ten years ago.
Of course it changed in a myriad of different ways
but it's much, much better now.
But the kernals and the principles are very
much still the same.
And that's interesting to reflect on
when you compare it to the things that changed.
In 2003, this was a state of the art phone.
[Audience Laughs]
Seriously. Flip phones was really back then.
It's so funny because that seems so far away, right?
I mean, flip phones? Seems like another era.
Which it is in many ways
because many parts of technology move very fast.
It's interesting that we're here today and we're still
working with and discussing and debating and improving a framework
that is from the time of flip phones.
So, that's sort of like the form factor of actually
a lot of people interfacing with our applications today go through phones.
If we look at other parts of technology
just sort of the hardware we were on
I looked up these prices yesterday
and, it's like, half a gigabyte costs $50.
Now you can just 16 times as much RAM for almost half the price.
That's more than an order of magnitude change.
Which is just, it's one of those things that's also
interesting in terms of what does that change
in how we approach things?
Often times, people will almost lock their concept
of what hardware costs and how it runs to the time
they entered programming.
If something was scarce, and expensive
at the time they got into programming they'll
think that's how it's going to be forever.
Which, of course, this shows you can get
8 gigabytes for $29. How much RAM do you use?
Probably it doesn't matter as much as
a lot of other things. Which is funny
to me because that's always been --
Matz, back in 1993, 20 years ago, 20 years ago Matz started working on Ruby.
He had to care far -- I should've actually looked up
what RAM cost back in 1993.
But I'm sure it was fucking expensive.
[Audience Laughs]
Yet, he sort of had the foresight to think
well, I don't know if he had the foresight,
but it turned out, that the thoughts
that he was thinking about, how we should
make programming languages, and development environments,
like, I can make something now that -- ok, it's going to take more memory,
it's going to be slower, and it's not going to matter
because those things are going to improve
drastically over time.
What's not going to improve is programming brains.
They're pretty much as mush as they were back in 1993.
So we need to optimize for that.
So, that's sort of the hardware side, that that moves really fast.
Ok, not that -- we know that, Moore's Law, and all, right?
Also funny to look at the things from 2003.
Java was actually a real thing that people non-ironically wrote.
[Audience Laughter and Applause]
This was actually -- when Rails first came out,
I was talking to a lot of Java programmers
and they were serious.
[Audience Laughs]
And it's just funny to think back about this sort of battle, or whatever
that we had in the beginning. And this was the relevant foe.
This was who were trying to sort of say, there's a better way to implement
a lot of the good ideas that you have.
Because that was the second funny thing of course.
Because Rails is from 2003, and the bulk of the
thinking of how to structure, and create nicely
factored web applications, and those
types of applications came from the Java world.
It's just that they were trapped inside that beast
and we had to liberate it.
So, there are still people programming Java.
Of course, the JVM lives on and it seems like
that has a long and prosperous future.
And people are putting it to much better use
than the Java language itself.
The other thing, struts, that was actually
a serious framework at that time that we were
sort of trying to convince people why this was better
why XML situps and so on, were not something you should to do.
It's really interesting to think about all of these things
that have changed, and how much the world
of technology has changed in the 10 years Rails has been around.
I've been trying to reflect on why that is.
Why are we still here?
Why didn't Rails go the way of Struts?
Why wasn't it just a footnote in sort of a progression of things?
Where were all these other things that were supposed
to come afterwards. I certainly never
thought, when I started working on Rails
that I'd be here 10 years later and talking to
1,500 people at a sold-out conference.
That's a pretty measureable success.
That's surprising I'd say in technology.
Just like the flip phone is not here anymore
and Struts, thankfully is neither.
You should've thought that Rails would've gone away, right?
I think the stuff I talked about last time -- or last year
that pioneering spirit, I think is what makes the Ruby community
and the Rails community unique.
We are willing to continue west.
We don't just stop when we find a comfortable resting place.
There's a lot of other communities that value
stability far higher than we do.
That's not to say that stability is bad, it's just
a trade-off. If you look at say,
the Python guys, and how long they've taken
to get on Python 3, they're not there yet, by the way.
They're still going, and it's been years, right?
How long did it take the Ruby community to move
from 1.8 to 1.9 to 2.0? [Snap, snap, snap]
And it wasn't because it was painless.
How long did it take the Rails community to move
from Rails 1, to 2, to 3, to now 4?
Again, it wasn't painless.
Anytime you want to make progress like this
it's going to hurt.
But we as a community are willing to take that hurt.
That makes the community unique.
We're willing to sacrifice for progress.
I think what's interesting about that, the pioneering spirit,
that we're willing to continue west.
We don't just stop halfway and say,
"Ah, fuck it, Kansas is good enough"
[Audience Laughs]
A lot of people just -- they stop in Kansas.
Like that's just like, things are fine
I've got my cows and the sun comes up in the morning, great.
I mean, although, some Rails people have stopped in Kansas.
[Audience Laughs]
There are still a few people still running.
This is the half-life of Rails where we are now.
It came out in 2009, that's 4 years ago.
Anyway, if you're still in Kansas, keep
going west. California awaits you.
The west coast awaits you, Portland I should say actually, my bad.
We've continued west. We've gotten closer and closer
to this sort of promise of the west coast.
And I think as we've gotten closer and closer to that
and it's taken years and years, because again
it's a painful journey.
It's a perfectly legit thing to say,
I'm just not interested in any more progress, we're stopping in Kansas.
And I will go on, and I will do Rails 2.3 for the next 4 years.
A lot of people did and it wasn't like they were necessarily
miserable. I mean people do lead good lives in Kansas.
But, I think that there is something more important here
to keep on pressing on, and not just because
we keep our own interests and this is still interesting to go to
if we had just stopped as a community and said
Rails 2.3, that's our final resting place.
Ruby 1.8, what did we end up at? Ruby 1.8.9? 1.8.8? Whatever it was.
That's the final resting place, that's where we're gonna stop, right?
That's just where we are going to park our ambition and say, that's good enough.
I don't think we would be here because then these things would just be tools.
Nothing wrong with just being a tool.
[Audience Laughs]
There's just something more inspiring about pressing on and going further.
About this whole notion of pressing on and going further,
Joel Spolsky, in 2001, had this great essay: "Good Software Takes Ten Years, Get Used To It"
At the time I read that, it was funny, I read that
and was like "what a fucking jackass"
[Audience Laughs]
I can make something in fucking six months
that's going to blow the doors off whatever you've been
working on for ten years.
But I think he actually had the long end of something.
You can build something that's good for you in six months
and it could be "good software".
If you want to build something that has the kind of impact
on community, on the programming community,
on the world, for lack, for less sort of grandious vision.
It does take a long time. We've slowly but surely built this up over time.
That goes both for the work that I've put into Basecamp
and the work that I've put into Rails.
These things compound slowly over time if you keep investing in them.
You can see the sort of progression of investment that we've had with Rails
and the people who've been flowing into it on this chart.
New contributors with their first commit to Rails.
2004, first time I opened the code base and accept patches and let other people commit to it,
a few people, 40 people I think had a patch go into Rails in 2004.
The very next year, big boost, 280 people or whatever.
Then we sort of had like, I don't want to call it flat face
because this is all new people coming into the community
and starting to add something. But, for a long time
we had sort of a steady rate of new people
joining the community. And then something interesting
happens. In 2009 there's a small dip.
Everybody thinks the world is going to end and
everyone is getting fired and alright, I understand
and then after that, what happens?
There's a massive jump.
And this is actually funny. So in 2009, our Kansas,
so there's a baseline there, alright? We keep pushing.
And people respond. A lot of people are interested
in pushing on with us, interested in joining the caravan and continuing on.
Which is really just sort of interesting to see just
sort of the phase we can go through.
By 2009 I've been working on Rails for 6 years.
And we've had this steady inflow and then
all of a sudden lots of people get interested
in getting involved, awesome.
If you look at the same chart, just compounding
all of the committers, you can see it's more
of a steady climb, but the same pattern is there,
from 2009 and onwards, lots of new people got involved
because we kept pressing through.
We kept pressing through the comfortable phase.
Even though, the funny thing is, when you
do that you're sort of -- we have this
saying with Basecamp were you shouldn't be afraid to let customers go.
Sometimes customers are going to outgrow you.
They're going to think, well you no longer represent what they need in a
project management application, right?
A natural reaction to a lot of people, or for a lot of people
is to look, "Shit, we can't let that happen, no customer can leave, lock the doors"
"We will do whatever we can to take them happy"
We had that moment in 2009, where we could've said, let's just lock the doors.
Let's just improve things for people who are already here.
Big enough community, we're comfortable,
we could be self-sustaining for a long time, that could be fine.
We instead said nope, we're going to keep the doors open.
What that's going to mean is that there are going to be some people who are unhappy.
I've heard people say Rails 2.3 is the best version of Rails ever made.
[Audience Laughs]
Which is, it's sort of one of those funny nostalgic things of
you just sort of lock your timezone, like the 80's that's when music peaked.
[Audience Laughs]
It's just a funny phenomenon, and it's real, and I think
that's where a lot of technology circles an ecosystem, they stop.
And then they just stay there for the rest of life.
Anyway, we pushed on.
Here are the stats for the last 12 months running.
More than 1000 people were part of more than 8000 commits
to rails/rails GitHub repo. That is fucking amazing.
1000 people working, and this is just the people who get commits in.
We have so many more people doing all sorts of
other stuff around the codebase.
This is just the people who got a commit into rails/rails itself.
That is really and truly astounding.
And I think that that shows and validates
the approach, the uncomfortable approach that we have
to keep pushing and to keep getting better
even if it will ruffle the feathers of the people we already have.
The customers, the developers, who are already using Rails
that are somewhat uncomfortable with whatever new direction we have.
It's also a funny match to Roger's Innovation Adoption Curve Idea.
This is sort of an analysis describing how products are adopted.
So obviously, from the beginning we had a lot of innovators.
People who would use Rails even though there were
a lot of things that were broken or didn't do whatever in the beginning.
Small slice of the overall market of developers.
Okay, we've got the early adopters.
I think that's sort of our first crest.
And you sort of hit this point where you have to
make the jump from early adoptors to the
early minority. I think that's exactly what that point in 2009 shows us.
We did make that jump. We are now on sort of the next ride up.
Which is, which is pretty cool.
And I think it's also interesting for me in -- the original reason
I got into open source, the original reason
I wanted to release Rails was two-fold,
first, I wanted to give back. I had been
using Ruby, I had been using Linux,
I had been using MySQL, I had been using Apache
I'd been using all these open source projects
so, when I had something myself to share
I felt like, dude you're a dick if you don't release
something like this that other people could find value in,
when you've used all of their stuff.
So there's that. Don't be a dick.
The second part is, I'm having a lot of fun
doing this stuff. Ruby is really a great programming language
Rails enables me to use Ruby to build Basecamp.
There's going to be a lot of other people out there
who could benefit from Ruby so let's
get as much programming joy out there in the world.
So that's why this stuff is interesting because
we've having a bigger impact, we're getting
more people involved with enjoying their work.
And not being stuck in Struts, or whatever painful torture instruments of the time.
Anyway, so I think that's all great.
And it's great to see there are more people getting involved.
But you're also sitting there and have to stop
and say okay, what are we actually doing here?
What is the purpose? Why are we still working on this?
What is the improvement path? Who are we?
Who do we want to be?
So, I think a while back, decided we want
to have a big tent. There should be room
for a lot of different people in the Rails community.
Jugglers, musicians, clowns and elephants all invited, come on.
And we pulled some things in, we did
with Rails 3, we had a influx of good new ideas
from, at the time, a competing framework
called merb, which was sort of a
good reflection of okay, it can be a big tent.
We can have room for a lot of people in here
who might not agree on everything,
they might not all want to build Basecamp
but we have a general principle, a general purpose
that this is where we're going, and
we agree on these things. And once we
sit down and actually work on code together
we realize we share more opinions than we don't.
So, even so, even if you have a big tent
and you have room for jugglers, and elephants and so forth
and you're having fun, you're not a ferris wheel.
There's got to be something you're not.
You have to sort of define it in some sort of opposition
there's limits to what this is, because if
you try to be everything to everyone all the time
you're nothing to nobody none of the time.
Second, the title of this talk is "Patterns of Basecamp's Application Architecture"
The whole idea of patterns is that you see a problem
and there's a solution, but there's a third element,
there's a context. There's a context in which
that problem arises. Because you can encounter
the same problem in two different contexts
and you're going to want two different solutions to that.
So, what is the context? What is the context of Rails?
Of the work that I'm doing on Basecamp?
For me, that context is Dynamic hypertext documents.
What does that mean?
It means that this is sort of what I'm trying to produce.
It's funny because when you see HTML it looks like
it's such a good, there are books like Learn HTML in 4 Days, right?
It seems like such a low-level thing, it seems like such a simple thing,
like sort of beneath us, kind of thing, we are "serious programmers" who do "serious work",
like, all we're doing is just generating HTML, that's -- I mean, c'mon, that's for HTML monkeys, right?
Like, I need something more grand. Well, to me
the grandness is in embracing the simplicity.
Is embracing what the web is, and not just what
the web and HTML as the delivery mechanism,
sort of the web and HTML as sort of a lot of
side effects that are neat, in term of software deployment and development,
don't have to install software, you just get it from one place.
You could deliver a lot of different kinds of software where
you just use HTML as a delivery mechanism.
But, when I've been revisiting and going back through the Basecamp code base
and especially the work I've done with the new verison of Basecamp.
I realize that it's deeper than that, that this is not just a delivery mechanism.
That the actual structure of HTML, the approach to the context of
software development as a document is profound.
That profound approach is not something we should try to escape.
It's something we should try to embrace.
So, you can think about a lot of different applications that are being delivered on the web
on sort of on a contiuum. On the one end
you have something that is basically purely document based
and on the other hand you could have something that's
almost like a GUI using HTML as a delivery mechanism
but you certainly wouldn't describe it as a document.
I think if you look at that sort of continuum from document to GUI and nothing falls perfectly at either hand.
Perhaps these two, I mean Wikipedia is just a collection of documents.
It's a pretty good to, I think, my vision at least of
what I'm interested in, the kinds of applications
I'm interested in, and thus, the kinds of tools
I'm interested in building. I want to build tools
for a document based web. That approach to software development is quite
different from all sorts of other approaches that went before.
If you look at software from before the web
it looks different. I've been struggling to find a word to
describe that style of applications, GUI is the best
I can come up with, even though of course the web is a GUI, too.
You sort of know what I mean.
Maybe it's easier if you look at an example.
If you look at Google maps for example, that doesn't feel
like that's a document. That feels like
this is using HTML as a delivery mechanism for
the application that it provides. It doesn't mean
it's not a great application. It's a great application.
It just means that what we're trying to do is optimize for
a version and vision for web applications that is more
document based, this is not the primary thing
we should optimize for. So, if you're trying to
write the next Google Maps, maybe Rails, at least as
a whole package, as a full-stack framework isn't
the natural fit. You can pick parts out and you can have
a good HTTP processor and you can do all sorts of
other things that's interesting.
So, if that' 100% the camp that you're in, the
sort of not related to documents at all, maybe that's not it.
Then it becomes interesting, because as I said,
a few things are either one thing or another thing completely.
If you look at Basecamp for example, we have this
calendar there, that doesn't feel very much like
a document. It doesn't feel like it fits HTML, what's the section
what's the header, what's the paragraphs here?
It feels like we're using HTML as a delivery mechanism.
But it's a minority thing part of what we do
so we try to fit it in and it should be possible, but it's
not the main thing. The main thing is more
stuff like this. Where it actually does look and
feel like a document. This looks and feels
like something you would not have before the web
this would not be like what an application looked like.
Because this is, that native development style of the web.
If you look at something like GitHub, same thing,
it feels like a document, it feels like a document that's decorated with some dynamic behavior
and that's great, but the key part of it, it's essence and it's being, is the document itself.
It's not just a delivery mechanism. That's the, the slice, and the part of information technology
that I'm interested in. These kinds of document focused
applications are not a bad thing. It's not something
we should be trying to escape, the constraints
that it puts on us to develop an application
in this style, they're liberating.
They're not bad. We don't want to start
with an empty canvas and have to draw everything
from scratch. The fact that HTML is there, and gives us
a sort of semantically relevant containers to put
certain things in, is a benefits, it's great.
And, that's where, I guess, the insight for me is
why is it that I was never interested in Java Applets?
It wasn't just because it was Java, it's horrible enough in itself,
but it was the notion that this was just a blank canvas,
you could draw whatever you want, there are no limits here,
you don't have to fall in line to some predescribed notion of a document.
Or set GUI elements for a document, forms and so forth.
You can do whatever you want.
I don't wanna do whatever I want. I want a container, a base to put my ideas in.
That's when I find, that they flourish.
So, with that insight in mind, and with the insight in mind
that the web is the most successful software development platform of all human history,
that leaves me to think that it's not acutally that big a surprise that this failed.
Because, the architects of this sort of canvas-base, let's
use the web as a sort of delivery mechanism for
the same application architecture that we had on the desktop
they're wrong. That's not the part of the web that's interesting.
It's not just these accidental properties of oh, it's easy
to update software because it runs on my server.
The things that brought the web to it's prominence with users
and developers alike, we're the constraints.
Was the fact that that style of interfacing with information technology is
really user-friendly, really development-friendly, it's different.
Not in a worse way. And that's usually what I --
I had conversation, I think it was Rails Conf, maybe 7, or something,
and some Microsoft guy was coming up and he was going
to tell me about serverlite, and the pitch was
"I'm finally going to free you from HTML"