Chad: Yes, hello, thank you.
Audience member: Hello!
Chad: Hello!
I am Chad, as he said.
He said I need no introduction
so I won't introduce myself any further.
I may be the biggest non-Indian fan of India
[Hindi speech]
I'll now switch back, sorry.
If you don't understand Hindi, I said nothing of value
and it was all wrong.
But I was saying that my Hindi is bad
and it's because now I'm learning German
so I mixed them together, but I know not everyone
speaks Hindi here.
I just had to show off, you know
So, I am currently working on 6WunderKinder,
and I'm working on a product called Wunderlist.
It is a productivity application.
It runs on every client you can think of.
We have native clients, we have a back-end,
we have millions of active users,
and I'm telling you this not so that you'll go download it -
you can do that too -
but I want to tell you about the challenges that I have
and the way I'm starting to think about system's architecture and design.
That's what I'm gonna talk about today
I'm going to show you some things that are real
and that we're really doing.
I'm going to show you some things that are
just a fantasy that maybe don't make any sense at all.
But hopefully I'll get you think about
how we think about system architecture
and how we build things that can last for a long time.
So the first thing that I want to mention:
this is a graph from the Standish Chaos report
and I've taken the years out
and I've taken some of the raw data out
because it doesn't matter.
If you look at these, this graph,
each one of these bars is a year,
and each bar represents successful projects in green -
software projects.
Challenged projects are in silver or white in the middle
and then failed ones are in red.
But challenged means significantly over time or budget
which to me means failed too.
So basically we're terrible,
all of us here, we're terrible.
We call ourselves engineers but it's a disgrace.
We very rarely actually launch things that work.
Kind of sad,
and I am here to bring you down.
Then once you launch software, anecdotal-y,
and you probably would see this in your own work lives, too,
anecdotal-y, software gets killed after about five years -
business software.
So you barely ever get to launch it, because,
or at least successfully, in a way that you're proud of,
and then in about five years
you end up in that situation where you're doing a big rewrite
and throwing everything away and replacing it.
You know there's always that project to get rid of the junk,
old Java code or whatever that you wrote five years ago,
replace it with Ruby now,
five years from now you'll be replacing your old junk Ruby code
that didn't work with something else.
We create this thing, probably all of you know the term legacy software -
Right, am I right? You know what legacy software is,
and you probably think of it as a negative thing.
You think of it as that ugly code that doesn't work,
that's brittle, that you can't change, that you're all afraid of.
But there's actually also a positive connotation of the word legacy:
it's leaving behind something that future generations can benefit from.
But if we're rarely ever launching successful projects
and then the ones we do launch tend to die within five years
none of us are actually creating a legacy in our work.
We're just creating stuff that gets thrown away.
Kind of sad.
So we create this stuff that's a legacy software.
It's hard to change, that's why it ends up getting thrown away
right, that's, if the software worked
and you could keep changing it to meet the needs of the business
you wouldn't need to do a big rewrite and throw it away.
We create these huge tightly-coupled systems,
and I don't just mean one application,
but like many applications are all tightly coupled.
You've got this thing over here talking to the database of this system over here
so if you change the columns to update the view of a webpage
you ruin your billing system, that kind of thing
this is what makes it so hard to change
and the sad thing about this is the way we work
the way we develop software, this is the default setting
and, what I mean is, if we were robots churning out software
and we had a preferences panel
the default preferences would lead to us creating terrible software that gets thrown away in
five years
that's just how we all work
as human beings when we sit down to write code
our default instincts lead to us to create systems that are tightly coupled
and hard to change and ultimately get thrown away and can't scale
we create, we try doing tests, we try doing TDD
but we create test suites that take forty-five minutes to run
every team has had to deal with this I'm sure
if you've written any kind of meaningful application
and it gets to where you have like a project
to speed up the test suite
like you start focusing your company's resources
on making the test suite faster
or making it like only fail ninety percent of the time
and then you say well if it only fails ninety percent that's OK
right, and right now it's taking forty-five minutes
we want to get it to where it only takes ten minutes to run
so the test suite ends up being a liability instead of a benefit
because of the way you do it
because you have this architect where everything is so coupled
you can't change anything without spending hours working on the stupid test suite
and your terrified to deploy
I know like the last big Java project I was working on
it would take, once a week we did a deploy
it would take fifteen people all night to deploy the thing
and usually it was like copying class files around
and restarting servers
it's much better today but it's still terrifying
you deploy code, you change it in production
you're not sure what might break
cause it's really hard to test these big integrated things together
and actually upgrading the technology component is terrifying
so, how many of you have been doing Rails for more than three years?
do you have, like a Rails 2 app in production, anyone? Yeah?
that's a lot of people, wow, that's terrifying
and I've been in situations, recently, where we had Rails 2 apps in production
security patches are coming out, we were applying our own versions
of those security patches
because we were afraid to upgrade Rails
we would rather hack it than upgrade the thing
because you just don't know what's gonna happen
and then you end up, as you're re-implementing all this stuff yourself
you end up burning yourself out, wasting your time
because you're hacking on stupid Rails 2
or some old struts version
when you should be just taking advantage of the new patches
but you can't because you're afraid to upgrade the software
because you don't know what's going to happen
because the system is too big and too scary
then, and this is really bad, I think this is something
Ruby messes up for all of us
I say this as someone who's been using Ruby for thirteen years now
happily
we create these mountains of abstractions
and the logic ends up being buried inside them
I mean in Java it was like static, or, you know, factories
and design pattern soup
in Ruby its modules and mixins and you know
we have all these crazy ways of hiding what's actually happening from us
but when you go look at the code
it's completely opaque
you have no idea where the stuff actually gets done
because it's in some magic library somewhere
and we do all that because we're trying to save ourselves from the complexity of these
big nasty systems
but like if you look at the rest of the world
this is a software specific problem
these cars are old, they're older than any software that you would ever run
and they're still driving down the street
they're older than software itself, right
but these things still function, they still work
how? why? why do they work?
bodies! my body should not work
I have abused it
I should not be standing here today
I shouldn't have been able to come from Berlin here
without dying somehow by being in the air
you know, by the air pressure changes
but our bodies somehow can survive even when
we don't take care of them
and like it's just the system that works, right
so how do our bodies work?
how do we stay alive
despite this fact
even though we haven't done like some
great design, we don't have any design patterns
like mixed up into our bodies
in biology there is a term called homeostasis
and I literally don't know what this means
other than this definition
so you won't learn about this from me
there's probably at least one biologist in the room
so you can correct me later
but basically the idea of homeostasis is
that an organism has all these different components
that serve different purposes
that regulate it
so they're all kind of in balance
and they work together to regulate the system
if one component, like a liver, does too much
or does the wrong thing
another component kicks in and fixes it
and so our bodies are this well designed system
for staying alive
because we have almost like autonomous agents
internally that take care of the many things that can and do go wrong
on a regular basis
so you have, you know, your brain, your liver
your liver, of course, metabolizes toxic substances
your kidney deals with blood, water level, et cetera
you know all these things work in concert to make you live
the inability to continue to do that is known as homeostatic imbalance
so I was saying, homeostasis is balancing
not being able to do that is when you're out of balance
and that will actually lead to really bad health problems
or probably death, if you fall into homeostatic imbalance
so the good news is you're already dying
like we're all dying all the time
this is the beautiful thing about death
there is, there is an estimate that fifty trillion cells
are in your body, and three million die per second
it's an estimate because it's actually impossible to count
but scientists have figured out somehow that this is probably the right number
so your cells, you've probably heard this all your life
like physically, after some amount of time,
you aren't the same human being that you were, physically
you know, I don't know, you some period of time ago
you're literally not the same organism anymore
but you're the same system
kind of interesting, isn't it
so in a way you can think about software this
you can think about software as a system
if the components could be replaced like these cells
like, if you focus on making death, constant death OK
on a small level
then the system can live on a large level
that's what this talk is about
solution, the solution being to mimic living organisms
and as an aside, I will say many times the word small or tiny in this talk
because I think I'm learning, as I age
that small is good
its, small projects are good
you know how to estimate them
small commitments are good
because you know you can make them
small methods are good
small classes are good
small applications are good
small teams are good
so I don't know, this is sort of a non sequitur
so if we're going to think about software
as like an organism
what is a cell in that context?
this is sort of the key question that you have to ask yourself
and I say that a cell is a tiny component
now, tiny and component are both subjective words
so you can kind of do what you want with that
but it's a good frame of thinking
if you make your software system of tiny components
each one can be like a cell
each one can die and the system is a collection of those tiny components
and what you want is not for your code to live forever
you don't care that each line of code lives forever, right
like if you're trying to develop a legacy in software
it's not important to you that your system dot out dot printline statement
lives for ten years
it's important to you that the function of the system lives for ten years
so like, about exactly ten years ago
we created Ruby gems at the RubyConf 2003 in Austin, Texas
I haven't touched Ruby gems myself in like four or five years
but people are still using it
they hate it because it's software
everybody hates software right
so if you can create software that people hate
you've succeeded
but it still exists
I have no idea if any of the code is the same
I would assume not
you know I think, I'm sure that my name is still in it in a copyright notice
but that's about it
and that's a beautiful thing
people are still using it to install Ruby libraries
and software
and I don't care if any of my existing, or my initial code is still in the system
because the system still lives
so, quite a long time ago now I was researching this kind of question
about Legacy software
and I asked a question on Twitter as I often do at conferences
when I'm preparing
what are some of the old surviving software systems you regularly use
and if you look at this, I mean, one thing is obviously
everyone who answered gave some sort of Unix related answer
but basically all of these things on this list
are either systems that are collections of really well-known split-up components
or they're tiny, tiny programs
so, like, grep is a tiny program, make
it only does one thing
well make is actually also arguably an operating system
but I won't get into that
emacs is obviously an operating system, right
but it's well designed of these tiny little pieces
so a lot of the old systems I know about follow this pattern
this metaphor that I'm proposing
and from my own career
when I was here before in Banglore
I worked for GE and some of the people
we hired even worked on the system there
we had a system called the Bull
and it was a Honeywell Bull mainframe
I doubt any of you have worked on that
but this one I know you didn't work on
because it had a custom operating system
with our own RDVMS
we had created a PCP stack for it
using like custom hardware that we plugged into a Windows MT computer
with some sort of MT queuing system back in the day
it was this terrifying thing
when I started working there the system was already something like twenty-five years old
and I believe even though there have been many, many projects
to try to kill it, like we had a team called the Bull exit team
I believe the system is still in production
not as much as it used to be, there are less and less functions in production
but I believe the system is still in production
the reason for this is that the system was actually made up of these tiny little components
and like really queer interfaces between them
and we kept the system live because every time we tried to replace it
with some fancy new gem, web thing or gooey app
it wasn't as good, and the users hated it
it just didn't work
so we had to use this old, crazy, modified mainframe
for a long time as a result
so, the question I ask myself is now
how do I, how do I approach a problem like this
and build a system that can survive for a long time
I would encourage you
how many of you know of Fred George
this is Fred George
he was at ThoughtWorks for awhile
so he may have, I think he lived in Banglore
for some time with ThoughtWorks, in fact
he is now running a start-up in Silicon Valley
but he has this talk that you can watch online
from the Barcelona Ruby Conference the year before last
called Microservice Architectures
and he talks in great detail about he,
how he implemented a concept at forward
that's very much like what I'm talking about
tiny components that only do one thing and can be thrown away
so Microservice Architecture is kind of the core of what I'm gonna talk about
now I've put together some rules for 6WunderKinder
which I am going to share with you
6WunderKinder is the company I work for
when we're working on Wunderlist
and the rules of the, the goals of these rules
are to reduce coupling, to make it where we can do fear-free deployments
we reduce the chance of "cruft" in our code
like nasty stuff that you're afraid of
that you leave there, kind of broken window problems
we make it literally trivial to change code
so you just never have to ask how do I do that
you just find it easy
and most importantly we give ourselves the freedom to go fast
because I think no developer ever wants to be slow
that's one of the worst things
just toiling away and not actually accomplishing anything
but we go slow because we're constrained by the system
and we're constrained by, sometimes projects
and other, you know, management related things
but often times its the mess of the system that we've created
so some of the rules
I think one thing, and maybe, maybe I'm going to get some push back from this crowd
one rule that is less controversial than it used to be
is that comments are a design smell
does anyone strongly disagree with that?
no?
does anyone strongly agree with that?
OK, so the rest of you have no idea what I'm talking about
so a design smell, I want to define this really quickly
a design smell is something you see in your code or your system
where it doesn't necessarily mean it's bad
but you look at it and you think
hmm, I should look into this a little bit
and ask myself, why are there so many comments in this code?
you know, especially the bottom one
inline comments?
definitely bad, definitely a sign that you should have another method, right
so it's pretty easy to convince people
that comments are a design smell
and I think a lot of people in the industry
are starting to agree
maybe not for like a public library
where you really need to tell someone
here's how you use this class and this is what it's for
but you shouldn't have to document every method
and every argument because the method name and the argument name
should speak for themselves, right
so here's one that you probably won't agree with
tests are a design smell
so this one is probably a little more controversial
especially in an environment where you're maybe still struggling people
struggling with people to actually get them to write tests to begin with, right
you know I went through this period in, like, 2000 and 2001
where I was really heavily into evangelizing TDD
and it was really stressful that you couldn't get anyone to do it
I think you do have to go through that period
and I'm not saying you shouldn't write any tests
but that picture I showed you earlier of the slow, brittle test suite
that's bad, right
that is a bad state to be in
and you're in that state because your tests suck
that's why you get in that state
your tests suck because you're writing bad tests
that don't exercise the right things in your system
and what I've found is whenever I look into one of these
big slow brittle test suites
the tests themselves are indications
and the sheer proliferation of tests
are indications that the system is bad
and the developers are like desperately
fearfully trying to run the code
in every way they can
because it's the only way they can manage
to even think about the complexity
but if you think about it, if you had a tiny trivial system
you wouldn't need to have hundreds of test files
that take ten minutes to run, ever
if you did, you're doing something stupid
you're wasting your time working on tests
and we as software developers obsess about this kind of thing
because we have to fight so hard to get our peers to do it in the first place
and to understand it
we obsess to the point where we focus on the wrong thing
none of us are in the business of writing tests for customers
like we're not launching our tests on the web
and hoping people will buy them, right
it doesn't provide value, it's just a side-effect
that we have focused too heavily on
and we've lost sight of what the actual goal is
so, this one actually requires a visual
I tell the people on my team now
you can write code in any language you want
any framework you want, anything you want to do
as long as the code is this big
so if you want to write the new service in Haskell
and it's this big in a normal size font
you can do it
if you want to do it in Closure or Elixir or Scarla or Ruby
or whatever you want to do
even Python for god's sake
you can do it if it's this big and no bigger
why? because it means I can look at it
and I can understand it
or if I don't I'll just throw it away
because if it's this big it doesn't do very much, right
so the risk is really low
and I really mean the system is that
there are the, the component is that big
and in my world a component means a service
that's running and probably listening on an HTTP board
or some sort of rift or RPC protocol
so it's a standalone thing
it's its own application
it's probably in its own git repository
people do poll requests against it
but it's just tiny
so this big
at the top of this, by the way
is some code by Konstantin Haase
who also lives in Berlin, where I live
this is a rewrite of Sinatra
the web framework
and Konstantin is actually the maintainer of Sinatra
it's not fully compatible, but it's amazingly close
and it all fits right in that
but the font size is kind of small, so I cheated
another rule, our systems are heterogeneous by default
so I say you can write in any language you want
that's not just because I want the developers to be excited
although I think, most of you, if you worked
in an environment where your boss told you
you can use any programming language or tool you want
you would be pretty happy about that, right
anyone unhappy about that? I don't think so
unless it's one of the bosses here
that's like don't tell people that
so that's one thing
the other one is, it leads to a good system design
because think about this
if I write one program in Erlang, one component in Erlang
one program in Ruby
I have to work really, really hard to make tight coupling
between those things
like I have to basically use computer science to do that
I don't even know what I would do
you know it's hard
like I would have to maybe implement Ruby in Erlang
so that it can run in the same BM or vice versa
it's just silly, I wouldn't do it
so if my system is heterogeneous by default
my coupling is very low, at least at a certain level by default
because it's the path of least resistance
is to make the system decoupled
it's easier to make things decoupled than coupled
if they're all running in different languages
so in the past three months, I'll say
I have written production code in objective CRuby, Scala, Closure, Node
I don't know, more stuff, Java
all these different languages
real code for work
and yes, they are not tightly coupled
like I haven't installed JRuby so that I could reach into the internals of my Scala code
because that would be a pain
I don't want to do that
another very important one is
server nodes are disposable
so, back when I was at GE, for example
I remember being really proud when I looked at the up time of one of my servers
and it was like four hundred days or something
it's like, wow, this is awesome
I have this big server, it had all these apps on it
we kept it running for four hundred days
the problem with that is I was afraid to ever touch it
I was really happy it was alive
but I didn't want to do anything to it
I was afraid to update the operating system
in fact you could not upgrade Solaris then without restarting it
so that meant I had not upgrading the operating system
I probably shouldn't have been too proud about it
Nodes that are alive for a long time lead to fear
and what I want is less fear
so I throw them away
and this means I don't have physical servers that I throw away
that would be fun but I'm not that rich yet
we use AWS right now, you could do it with any kind of cloud service
or even internal cloud divider
but every node is disposable
so, we never upgrade software on an existing server
whenever you want to deploy a new version of a service
you create new servers
and you deploy that version
and then you replace them in the load balance or somewhere
that's it
so, you never have to wonder what's on a server
because it was deployed through an automated process
and there's no fear there
you know exactly what it is
you know exactly how to recreate it
because you have a golden master image
and in our case it's actually an Amazon image
that you can just boot more of
scaling is a problem
you just boot ten more servers
boom, done, no problem
so yeah I tell the team, you know, pick your technology
everything must be automated, that's another piece
if you're going to deploy a closure service for the first time
you have to be responsible for figuring out how it fits into our deployment system
so that you have immutable deployments and disposable nodes
if you can do that and you're willing to also maintain it and teach someone else
about the little piece of code that you wrote, then cool
you can do it, any level you want
and then once you deploy stuff
like a lot of us like to just SFH in the machines
and then twiddle with things and replace files
and like try like fixing bugs live on production
why no just throw away the actual keys
because you're going to throw away the system eventually
you don't even need route access to it
you don't need to be able to get to it
except through the port that your service is listening on
so you can't screw it up
you can't introduce entropy and mess things up
if you throw away the keys
so this is actually a practice that you can do
deploy the servers, remove all the credentials
for logging in and the only option you have
is to destroy them when you're done with them
provisioning new services in our world
must also be trivial
so we have actually now thrown away our chef repository
because chef is obsolete and
we have replaced it with shell scripts
and that sounds like I'm an idiot
I know, but when I say chef is obsolete
I don't really mean that
I like to say that so that people will think
because a lot of you are probably thinking
we should move to chef
that would be great
because what you have is a bunch of servers
that are running for a long time
and you need to be able to continue to keep them up to date
chef is really great at that
chef is also good at booting a new server
but really it's just overkill for that
yeah
so if you're always throwing stuff away
I don't think you need chef
do something really, really simple
and that's what we've done
so like whenever we deploy a new type of service
I set up ZooKepper recently, which is a complete change from the other stuff we're deploying
I think it was a five line shell script to do that
I just added it to a get repo and run a command
I've got a cluster of ZooKeeper servers running
you want to always be deploying your software
this is something I learned from Kent Beck early on in the agile extreme programming
world
that if something is hard
or you perceive it to be hard or difficult
the best thing you can do
if you have to do that thing all the time
is to just do it constantly
non-stop all the time
so like deploying in our old world
where it would take all night once a week
if we instituted a new policy
in that team that said
any change that goes to master must be deployed within five minutes
I guarantee you we would have fixed that process, right
and if you're deploying constantly
all day every day
you're never going to be afraid of deployments
because it's always a small change
so always be deploying
every new deploy means you're throwing away old servers
and replacing them with new ones
in our world I would say that the average uptime
of one of our servers is probably something like
seventeen hours and that's because we don't tend to work on the weekend very much
you also, when you have these sorts of systems
that are distributed like this
and you're trying to reduce the fear of change
the big thing that you're afraid of is failure
you're afraid that the service is going to fail
the system is going to go down
one component won't be reachable, that sort of thing
so you just to have assume that that's going to happen
you are not going to build a system that never fails, ever
I hope you don't, because you will have wasted much of your life
trying to get that to happen
instead, assume that the thing, the components are going to fail
and build resiliency in
I have a picture here of Joe Armstrong
who is one of the inventors of Erlang
if you have not studied Erlang philosophy around failure and recovery
you should
and it won't take you long
so I'm just going to leave that as homework for you
and then, you know, I said, the tests are a design pattern
I don't mean don't write any tests
but I also want to be further responsible here
and say you should monitor everything
you want to favor measurement over testing
so I use measurement as a surrogate for testing
or as an enhancement
and the reason I say this is
you can either focus on one of two things
I said assume failure right, so
mean time between failures or mean time to resolution
those are kind of two metrics in the ops world
that people talk about
for measuring their success and their effectiveness
mean time between failures means
you're trying to increase the time between failures
of the system, so basically you're trying to make failures never happen, right
mean time to resolution means
when they happen, I'm gonna focus on bringing them back
as fast as I possibly can
so a perfect example would be a system fails
and another one is already up and just takes over its work
mean time to resolution is essentially zero, right
if you're always assuming that every component can will fail
then mean time to resolution is going to be really good
because you're going to bake it into the process
if you do that, you don't care about when things fail
and back to this idea of favoring measurement over testing
if you're monitoring everything, everything with intelligence
then you're actually focusing on mean time to resolution
and acknowledging that the software is going to be broken sometimes, right
and when I say monitor everything, I mean everything
I don't mean, like your disk space and your memory and stuff there
I'm talking about business metrics
so, at living social we created this thing called rearview
which is now opensource
which allows you do to aberration detection
and aberration means strange behavior, strange change in behavior
so rearview can do aberration detection
on data sets, arbitrary data sets
which means, like in the living social world
we had user sign ups
constantly streaming in
it was a very high volume site
if user sign-ups were weird
we would get an alert
why might they be weird?
one thing could be like the user service is down, right
so then we would get two alerts
user sign ups have gone down
and so has the service
so obviously the problem is the service is down
let's bring it back up
but it could be something like
a front-end developer or a designer
made a change that was intentional
but it just didn't work and no one liked it
so they didn't sign up to the site anymore
that's more important than just knowing that the service is down
right, because what you care about
isn't that the service is up or down
if you could crash the entire system and still be making money
you don't care, right, that's better
throw it away and stop paying for the servers
but if your system is up 100% of the time and performs excellently
but no one's using it, that's bad
so monitoring business metrics gives you a lot more than unit test could ever give you
and then in our world
we focused on experiencing
no, you have to come up to front and say ten!
ok, ten minutes left
when I got to 6WunderKinder in Berlin
everyone was terrified to touch the system
because they hadn't created a really well-designed
but traditional monolithic API
so they had layers of abstractions
it was all kind of in one big thing
they had a huge database
and they were really, really scared to do anything
so there's like one person who would deploy anything
and everyone else was trying to work on other projects
and not touch it
but it was like the production system
you know so it wasn't really an option
so the first thing I did in my first week
is I got these graphs going
and this was, yeah, response time
and the first thing I did is I started turning off servers
and just watching the graphs
and then, as I was turning off the servers
I went to the production database
and I did select, count, star from tasks
and we're a task management app
so we have hundreds of millions of tasks
and the whole thing crashed
and all the people were like AAAAH what's going on
you know, and I said, it's no problem
I did this on purpose, I'll just make it come back
which I did
and from that point on
like, really every day I would do something
which basically crash the system for just a moment
and really, like, we had way too many servers in production
we were spending tens of thousands more Euros per month
than we should have on the infrastructure
and I just started taking things away
and I would usually do it
instead of the responsible way,
like one server at a time
I would just remove all of them and start adding them back
so for a moment everything was down
but after that we go to a point where
everyone on the team was absolutely comfortable
with the worst case scenario
of the system being completely down
so that we could, in a panic free way
just focus on bringing it up when it was bad
so now when you do a deployment
and you have your business metrics being measured
you know the important stuff is happening
and you know what to do when everything is down
you've experienced the worst thing that can happen
well the worst thing is like someone breaks in
and steals all your stuff, steals all your users' phone numbers
and posts them online like SnapChat or something
but you've experienced all these potentially horrible things
and realized, eh, it's not so bad, I can deal with this
I know what do to
it allows you to start making bold moves
and that's what we all want right
we all want to be able to bravely go into our systems
and do anything we think is right
so that's what I've been focusing on
we also do this thing called Canary in the Coal Mine deployments
which removes the fear, also
canary in the coalmine refers to a kind of sad thing
about coal miners in the US
where they would send canaries into the mines
at various levels
and if the canary died they knew there was a problem
with the air
but in the software world
what this means is you have bunch of servers running
or a bunch of, I don't know, clients running a certain version
and you start introducing new version incrementally
and watching the effects
so once you're measuring everything
and monitoring everything
you can also start doing these canary in the coalmine things
where you say OK I have a new version of this service
that I'm going to deploy
and I've got thirty servers running for it
but I'm going to change only five of them now
and see, like, does my error rate increase
or does my performance drop on those servers
or do people actually not successfully complete the task they're trying to do
on those servers
so, this also allows us the combination of monitoring everything
and these immutable deployments and everything
gives us the ability to gradually affect change and not be afraid
so we roll out changes all day every day
because we don't fear that we're just going to destroy the entire system all at once
so I think I have like five minutes left
uh, these are some things we're not necessarily doing yet
but they're some ideas that I have
that given some free time I will work on
and, they're probably more exciting
one is I talked about homeostatic regulation
and homeostasis
so I think we all understand the idea of you know homeostasis
and the fact that systems have different parts that do different roles
and can protect each other from each other
but, so this diagram is actually just some random diagram
I copied and pasted off the AWS website
so it's not necessarily all that meaningful
except to show that every architecture
especially server based architectures
has a collection of services that play different roles
and it almost looks like a person
you've got a brain and a heart and a liver
and all these things, right
what would it mean to actually implement
homeostatic regulation in a web service?
so that you have some controlling system
where the database will actually kill an app server
that is hurting it, for example
just kill it
I don't know yet, I don't know what that is
but some ideas about this stuff
I don't know if you've heard of these
NetFlix, do you have NetFlix in India yet?
probably not, unless you have a VPN, right
NetFlix has a really great cloud based architecture
they have this thing called Chaos Monkey they've created
which goes through their system and randomly destroys Nodes
just crashes servers
and they did this because, when they were, they were early users of AWS
and when they went out initially with AWS, servers were crashing
like it was still immature
so they said OK we still want to use this
and we'll build in stuff so that we can deal with the crashes
but we have to know it's gonna work when it crashes
so let's make crashing be part of production
so they actually have gotten really sophisticated now
and they will crash entire regions
cause they're in multiple data centers
so they'll say like, what would happen if this
data center went down, does the site still stay up?
and they do this in production all the time
like they're crashing servers right now
it's really neat
another one that is inspirational in this way
is Pinterest, they use AWS as well
and they have, AWS has this thing called Spot Instances
and I won't go into too much detail
because I don't have time
but Spot Instances allow you to effectively
bid on servers at a price that you are willing to pay
so like if a usual server costs $0.20 per minute
you can say, I'll give $0.15 per minute
and when excess capacity comes open
it's almost like a stock market
if $0.15 is the going price, you'll get a server
and it starts up and it runs what you want
but here's the cool thing
if the stock market goes and the price goes higher than you're willing to pay
Amazon will just turn off those servers
they're just dead, you don't have any warning
they're just dead
so Pinterest uses this for their production servers
which means they save a lot of money
they're paying way under the average Amazon cost for hosting
but the really cool thing in my opinion
is not the money they save but the fact that
like, what would you have to do to build a full system
where any node can and will die at any moment
and it's not even under your control
that's really exciting
so a simple thing you can do for homeostasis though
is you can just adjust
so in our world we have multiple nodes
and all these little services
we can scale each one independently
we're measuring everything
so Amazon has a thing called Auto Scaling
we don't use it, we do our own scaling
and we just do it based on volume and performance
now when you have a bunch of services like this
like, I don't know, maybe we have fifty different services now
that each play tiny little roles
it becomes difficult to figure out, like, where things are
so we've started implementing zookeeper for service resolution
which means a service can come online and say
I'm the reminder service version 2.3
and then tell a central guardian
and the zookeeper can then route traffic to it
probably too detailed for now
I'm gonna skip over some stuff real quick
but I want to talk about this one
if, did the Nordic Ruby, no, Nordic Ruby talks never go online
so you can never see this talk
sorry
at Nordic Ruby Reginald Braithwaite did a really cool talk
on like challenges of the Ruby language
and he made this statement
Ruby has beautiful but static coupling
which was really strange
but basically he was making the same point that
I was talking about earlier
that, like Ruby creates a bunch of ways that you can couple
your system together
that kind of screw you in the end
but they're really beautiful to use
but, like, Ruby can really lead to some deep crazy coupling
and so he presented this idea of bind by contract
and bind by contract, in a Ruby sense
would be, like, I have a class that has a method
that takes these parameters under these conditions
and I can kind of put it into my VM
and whenever someone needs to have a functionality like that
it will be automatically bound together
by the fact that it can do that thing
and instead of how we tend to use Ruby and Java and other languages
I have a class with a method name I'm going to call it
right, that's coupling
but he proposed this idea of this decoupled system
where you just say I need a functionality like this
that works under the conditions that I have present
so this lead me to this idea
and this may be like way too weird, I don't know
what if in your web application your routes file
for your services read like a functional pattern matching syntax
so like if you've ever used Erlang or Haskell or Scala
any of these things that have functional pattern matching
what if you could then route to different services
across a bunch of different services
based on contract
now I have zero time left
but I'm just gonna keep talking, cause I'm mean
oh wait I'm not allowed to be mean
because of the code of contact
so I'll wrap up
so this is an idea that I've started working on as well
where I would actually write an Erlang service
with this sort of functional pattern matching
but have it be routing in really fast real time
through back end services that support it
one more thing I just want to show you real quick
that I am working on and I want to show you
because I want you to help me
has anyone used JSON schema?
OK, you people are my friends for the rest of the conference
in a system where you have all these things talking to each other
you do need a way to validate the inputs and outputs
but I don't want to generate code that parses and creates JSON
I don't want to do something in real time that intercepts my
kind of traffic, so there's this thing called JSON schema
that allows you to, in a completely decoupled way
specify JSON documents and how they should interact
and I am working on a new thing that's called Klagen
which is the German word for complain
it's written in Scala, so if anyone wants to pair up on some Scala stuff
what it will be is a high performance asynchronous JSON schema validation middleware
so if that's interesting to anyone, even if you don't know Scala or JSON schema
please let me know
and I believe I'm out of time so I'm just gonna end there
am I right? I'm right, yes
so thank you very much, and let's talk during the conference