RailsConf 2014 - Living Social Lightning Talks

??: Hi everyone. So we're gonna do,

we're at Living Social.

We build this consumer marketing platform.

We, you go to LivingSocial dot com and

it looks like it's just deals.

I know it's small and boring.

But there's like, actually a phenomenal

amount of stuff under the water line.

We have, most of our apps are written

in Ruby and Rails. But we also have teams

working in Clojure, Scala. I almost said SmallTalk.

That would be a lie. iOS, like native iOS,

native Android, big data stuff, operations.

All sorts of stuff. So there's tons of interesting,

wei- like, big problems that we come upon.

And we're gonna do a few lightning talks on

those technical things, and then also a

few on how we operate as a distributed team,

and how we just operate as developers.

So, the first person that's gonna speak to
you

is Ed Weng. He's a, he's a graduate from

our Hungry Academy training program,

where we pull basically just, like,

thirty random people off the streets.

No. They, they, it was not,

we did not press gang them.

We, they applied, we put them through a,

was it nine weeks? Nine, nine months?

ED WENG: Six months.

??: Six months.

E.W.: I don't know where you got nine weeks.

??: Nine. It was some number of months. It

was six months. They did a really intensive
training

program, and are all really amazing developers.
And so

he's a Hungry Academy grad. He works on our

mobile consumer apps and if you meet him later,

ask him about his pen pal. So this is

Ed.

E.W.: Hey guys.

[applause]

So, so my name is Ed Weng, and what

I'm gonna talk to you today is about sharing

templates in a service-oriented architecture.

Cool. So, if you've never been to our site

before, if you go to LivingSocial dot com,
this

is the page that you land on. And to

give you some context about our architecture,
about two

and a half, three years ago, we were still

a monolithic Rails app, right. So everything
was inside

this one gigantic Rails app.

Since then, we've been working very hard to
kind

of break things out into smaller micro services.
So

if you come to this page, this is actually

in an app that we call browse, and browse

is responsible for surfacing the browse experience
to our

customers.

So, you can see some of the deals on,

on our site. If you click on, let's say,

the Audible deal, you actually get taken to
a

page that looks like this. So, most of our

users won't be able to tell, but they actually

just left the browse app, and now they're
in

an app that we call sponsors. So, sponsors
is

responsible for surfacing deals that are sponsored.
So this

deal is sponsored by Audible dot com, hence
its

free. And let's say a user then decides to

click on this tab up on the top. The

shop tab.

They'd get taken to here. And this app, again,

is not the sponsors app anymore. It's an app

that we call products, and products is responsible
for

surfacing physical goods that we sell to our
customers.

So, if we were to look back at the

last three slides I just showed you, and I

were to ask you, what is the one common

thing that appears on all three slides? You
might

say it's the nav bar. And you'd be correct.

One of the things that we kind of discovered

along the way of splitting things out into
smaller

services, is things that might be very simple
in

a monolithic Rails app can become quite complex
when

you move towards a service-oriented architecture.
So the big

question that, that we're gonna ask ourselves
today is,

well, when it comes to sharing views, how
exactly

do we keep ourselves dry, right?

So, I don't need to tell you guys that,

you know, out of the box, in a monolithic

Rails app, you have partials and layouts,
right. So,

how much you show the nav-bar in every single

page. Well, you'll throw it into a partial
called

nav-bar, and then you'll just go into your
application

layout, you'll render the nav-bar, and then
it just

shows up everywhere, right. This is great
for a

monolithic Rails app. But as soon as you move

to a service-oriented architecture, it starts
to break down.

Well, why is that? The first thing is that

it's a lot of copy and paste coding, right,

so any time you want to make a change,

you have to copy that nav-bar into a bunch

of different places. The second thing is that
it

requires changes in multiple places, and so
it, it

kind of slows your development down. Any time
you

want to make a tweak, you have to go

to five, ten, fifteen services and issue a,
a

pull request, so that you know, the nav-bar
is

updated in that specific application.

What we've noticed is that, things tend to
get

out of sync very quickly if you, if you

take this approach. So, it leads to a bad

kind of user experience, in that the nav-bar
starts

to change as you go from page to page.

So this doesn't really work for us.

The next thing that you might think of doing

is, well, what about putting everything inside
an engine

or a gem, right? And this is a, a

much better solution, right. Why is it much
better?

Well, the first thing is that it, it eliminates

that copy and paste coding that we were just

talking about, right. So everything is kind
of stored

in this single central repository, and then,
and then

any time you want to make a change, you

just update the gem and kind of release it

into the wild.

One of the great things about putting this
in

an engine or a gem is that you can

share more than just the mark up, right. You

can share translation files, you can share
images, JavaScript,

stylesheets, et cetera, et cetera. So that's
great. The

problem with it, though, is that any time
you

make an update, you still have to go to

every single app and issue a bundle update.

So, you know, it might work fine if you

have a couple of apps. As soon as you

move to five, ten, fifteen apps, it becomes
a

hassle. So this is not quite good enough.
And

at LivingSocial, we, we have a saying, that
if

services aren't solving all of your problems,
you're probably

not using enough services. So, what did we
do

to solve our problem?

Of course. We build another service. And the
service

we built is called Stepford. It was designed
by

a guy named Eric Brody. And Stepford is responsible

for sharing our views across our service-oriented
architecture. So,

when we were first building Stepford, we went
through

kind of a DDD approach. A desi- a dream-driven

development approach. And we asked ourselves,
what do we

really want the interaction with a service
to look

like? And this is kind of what we came

up with.

So, we have a client app. We wanted the

client app to be able to ask Stepford and

say, send me the footer, right. Stepford would
then

go off, do some stuff behind the scenes, and

then just come back and say, here's your package.

So, let's kind of break that down into three

separate steps.

So the first step, send me the footer. We

tried to come up with the simplest possible
thing

that we could think of. And that is just

a GET request to some JSON endpoint, right.
So

this is what the request looks like inside
the

Stepford app. It's a GET request to, to the

packages controller, and we just put the things
that

we want. So, different elements inside the
query string.

So, here we're requesting the footer. We can
also

pass additional variables. Things like city
id, if we

need that footer to be semi-personalized or
modified.

So now you're probably asking, well, what's
inside a

package, right? So a package is actually made
up

of three separate things, right. It's made
up of

markup, it's made up of styles, and it's made

up of scripts. And everything kind of revolves
around

the markup that, that you're requesting. So
you might

imagine that if you're requesting the footer,
there's an

associated set of styles with it, and associated
set

of scripts with it.

If the styles don't come through, then your
footer

looks really ugly. And if the scripts don't'
come

through, then your footer probably isn't gonna
work the

way that you want it to work. So, we

kind of package all of the three things up,

cause they all kind of have to come together.

And we put it in this concept of a

package.

OK, so the next step. So the next step

is, Stepford gets that request, and what does
it

do behind the scenes? So, it's a little bit

complicated, and this is a very simplified
high-level version

of it, but in the top-left corner, we see

that the GET request comes into that package's
end

point. Stepford then tries to make the package.
So

it grabs the mark up, it grabs the styles,

it grabs the scripts, it tries to precompile
the

styles, it precompiles the scripts, it throws
everything into

the database, and then it actually tries to
render

the mark up on the server side. So that's

super interesting. It takes the erb template
that is

stored inside Stepford, and it actually renders
it into

raw html. It then sends it back via JSON.

So that's the last step of the three-step
process.

The here is your package. And I'm about to

show you what a package response looks like.
So,

the package response contains the three things
that we

talked about, right. The styles, the scripts,
and the

mark up. And here, you can see that all

we requested was the footer, right. But you
can

imagine that in our, in our architecture,
we're, an

app generally requests more than just the
footer. It

probably requests the footer, it might request
the nav-bar,

it might request button styles, et cetera,
et cetera.

And that is kind of all going to show

up in that mark up as raw html.

And the nice thing is that Stepford will kind

of package together all of the styles and
all

of the scripts for every single mark up element

that you've requested. So, everything just
kind of boils

down into one precompiled stylesheet and one
precompiled JavaScript.

Cool, so let's spend a little bit of time

talking about the pros and cons of spinning
up

a service like Stepford. The obvious, biggest
pro is

that you keep your views DRY, right. I don't

need to tell you the benefit of, of having

all of your views in a, in a single

central repository, right. There's huge benefits.
It's easy to

maintain. Et cetera, et cetera.

The second thing is that it's super easy to

release changes. So Stepford is all pull-through,
which means

that any time you update a nav-bar or any

time you update the footer, you don't have
to

go to every single app and tell them, there's

a new nav-bar. There's a new footer.

The next request that it makes to Stepford,
it's

actually going to, Stepford is actually going
to send

back the latest version of whatever element
there is

on, on the, in the database.

OK. So, there's obviously some cons. The first
con

is that it's another service. So one thing
that

we've realized by spinning up so many services
at

LivingSocial is that, any time you make a
service,

you're trading application complexity for
network complexity, right.

So that means that your tech-ops team has
to

maintain one more application. They have to
make sure

that it's up all the time, otherwise, you
know,

your styles won't come through. The next thing
is

that, because it's another service, it also
means that

there's another network request going on.
So Stepford does

introduce some amount of latency into your
application, right.

There's another HTTP request that goes on
on the

server side.

At LivingSocial, the way that we kind of have

gotten around that is we put in a lot

of, a lot of caching, right. So we have

bonus caching around Stepford. We also have
Memcache around

it. The reason is is because, a lot of

times, our main UI elements don't change all
that

frequently. And so it's OK to put things like

varnish in front of it. It's OK to put

Memcache in front of it.

And the last thing is that changes have a

much larger impact. So when I said it's easy

to release changes, sometimes it's too easy
to release

changes. You know, there have been instances
where we'll

make the change in JavaScript, and we quite,
and

we can't quite anticipate what the effects
downstream will

exactly be. And so as such, you really need

very close JavaScript monitoring to make sure
that you

understand when errors come through.

So that's it. That's my talk on Stepford.
If

you have any questions, please come find me
after

the talk. And we, we'll chat.

[applause]

??: Thanks Ed. So, the, these guys get to

watch me use a computer. It's great fun. Over

my shoulder. This is Tyler. Tyler works, also,
on

our consumer apps team. He's from Southern
California, which

is, his handsome looks, as you can see.

TYLER MONTGOMERY: Northern California.

??: Nor- I botched it. Sorry. But still handsome.

But he's taken ladies. And I just learned
from

his website that he wants to learn everything,
so

if you see him in the hall, just liter-

tell him anything and it will be the best

part of his day.

T.M.: Howdy. I'm Tyler. I live in Northern
California.

I work remote. So, how do you work a

computer? OK. Let's, let's start at the beginning.
It's

a good place to start. All right. Ready and

play.

All right. So I'm gonna talk about internal
gem

infrastructure. Basically making it easy to
write gems, release

gems. For us, you know, sharing is caring.
Being

able to share code effectively across your
apps makes

things really awesome. It's easy, like we
said, to

extract things into services, break down the
monolith when

you can share code and you can put it

into a gem really easily. It's, reduces the
copy-paste

mayhem that we've had, you know, we put up

a little service and someone writes a little
client

that gets copy-pasted fifty places, and then
we make

changes. It's really hard to track down who's
hitting

what end points, so we've had to go back,

put those things into a gem so we can

share and keep things maintainable.

Building gems and having this, you know, code
in

gems really helps build this culture of internal
open

source. A gem gets its own repo, gets its

own README. It's really easy for anyone to
contribute

to it and people know where things are. You

get all the benefits of pull requests.

It helps with separating concerns. Like, we're
told, like,

hey, you shouldn't make a mess, you know.
But

where do you put that mess in a Rails

app? It's kind of weird, like, where do I

put all these little tiny objects with single
responsibilities?

Where do they live? How do I keep track

of it?

And putting the stuff in a gem is really

nice. It helps you draw boundaries around
your code

to find your domain better. Isolating your
code is,

is actually really great for deleting it later.
It's,

you can track it down. You can find where

it needs to go and get rid of it.

But you don't necessarily have to delete the
code.

You don't have to go, where is that one

commit that one time where I did that one

thing? And I gotta go look through all these

old commits. You just go look at the repo,

but you've deleted all that code out of the

other code bases you still have some of that

history around.

Isolating is good too, because it forces you
into

how does your code interact with Rails? How
do

you interact with the database? How do you
do

logging when you're isolated in your own little
gem?

Do you even need Rails for this little business

project that you're working on? Does it need
to

be part of a Rails app?

This has really helped me, like, try to constrain

my code in the simplest form it needs to

be to get the job done. This helps me

mock and stub boundaries in my tests, cause
it's

clearly defined where it needs to define other,
other

pieces of our architecture.

This has really helped me with writing good
docs,

because I have a README for this thing. It's

really easy. So, a real example, a few weeks

ago I was tasked with automating a daily seatsbe??

(00:15:18) report out of the database, and
uploading it

to a third parties FTP site. They then will

return us a plain text email telling us the

results of processing this data. a coworker
commented, 1997

wants its architecture back.

So we've all done this. We say, hey man,

I'm just gonna drop this in the model, you

know. This FTP goes to CSV, it uploads. Gets

the job done. You know, fat models, bro. And

a couple weeks later you're like what is,
what

is this code? Who put this here? What does

this even do? Or, a year later. Two years

later. Who, who did this?

And, so, you know I was thinking, this is

really gnarly code. It has, just, it has just

kind of a single responsibility. It has just
one

thing it needs to do. So, I'll make a

gem. So, the rest of my talk, I'm just

gonna quickly go through building gems, releasing
gems, and

then hosting your own gems internally.

So, building a gem. This is another crash
course

inside a crash course. This might be review
for

some of you, but this is just kind of

the easy way that I see how to do

it. So, it's a, it's another crash course,
so.

So building a new gem is this easy. You

already have this installed. You already have
bundler installed.

You just type bundle gem your gem name. The

ls on the front there, that's to help us

with namespacing. I'll get into that in a
second.

But bundler will do that for you.

This is what it creates. This is your basic

fresh gem. It's the gem file, a license which

is MIT, you get your README, like I said.

You get your Rakefile and then you get your

lib directory with all your program code in
there.

The most important file there is the bottom
and

that's the gemspec. Without that, you don't
have a

gem. There's plenty of documentation on, you
know, what

goes in a gemspec. But bundle does a good

job of doing defaults for you.

Namespacing. If you hadn't seen this before,
I've done

this with a couple different companies and
it's really

helpful. You basically take your company name,
shorten it

down to two letters, and you put it in

a module and you put all your classes under

this global kind of module. That's how you
use

it there. It's LS for this one, GeminaboxClient,
and

you require it. If you see that, like, ls

slash thing, that's how it's, that's how you're
doing

namespacing.

This is really helpful. We have some early
gems

that didn't do this, and I wasn't sure when

I first started. Is this an open source implementation,

or is this internal? Like, where did this
thing

come from? So, maintaining this namespace
is, it's, it's

helpful, and knowing where your code is coming
from

and that it's an internal project.

So, basic example. This is just kind of an

example that has a few more things in it.

We threw a changes file in there so we

know what happened when the version changes,
and then

we've got our, you know, some more program
code,

and we have our tests. Like, everything is
nicely,

tightly compacted for, for this little project.

A quick review on semantic versioning. You
can go

to SemVer dot org. Yeah. Thanks. Some people
don't

understand it so I wanted to review it. The

last number in a version number is the bug

fix number. You increment this when things
change. Like,

when you fix bugs, but you haven't broken
the

public API. You haven't changed or broken
anything.

The minor version number is when you added
a

new feature. Yay. We have new stuff. But you

haven't removed anything. You haven't broken
the public API.

You haven't forced people into changing things.

And then finally, the major version number
is, you

broke something. You have changed significant
amounts of things

and people beware. Like, this might break
your code.

And you can think of this like how hard

it is to upgrade from Rails 2 to 3

or from Rails 3 to 4. That's a big

change so that is a major version number.

If it's stable, it should be 1 point 0.

If it's not stable, it should not be 1

point 0. This is a good signifier that just

says hey, this has been used in production.
It's

stable. It's, everyone can use this when you
go

to 1 point 0. If it's before then, you

probably see gems that are zero zero twelve-hundred
and

thirty-seven. And it's like, I'm not sure
if I

should use this. That's a lot of version numbers.

So, that's just quick refresher on building
gems and

versioning, namespacing. Now, we need to let
the world

behold your awesomeness, or, at least the
people at

your company. Releasing should be easy. We
say deploying

should be easy. Deploying and releasing gems
should be

an easy thing. This is not easy. Remembering
the

version number and tagging it and rake build
and,

like, this is not easy.

This is easy. Rake release. This is really
easy.

When you use bundler you get this rake command

for free. But, don't accidentally open source
your code.

This happens, and there are bots that mirror
RubyGems

right away. It's very hard to take that back.

So how do go from this to this, but

keeping everything internal?

You can monkey patch, I mean subclass Bundler.
Geminabox

will actually tell you on their Wiki page
to

monkey patch Bundler. Don't do that. So what
we

can, we wrote our own gem that just inherits

from Bundler's gem helper. There's tutorials
on this. But

I'm just gonna go through real quick how this

works.

In your, in your Rake file you see this

bundler/gem_tasks at the top. What that actually
does is

that gives you those three default rake tasks.
The

build, the install, and the release. By default,
it's

going to push it to RubyGems, and we don't

want that.

So, when we subclass it, what we want to

do is we want to say, we use geminabox

as our gem server at Living Social. We want

to say, hey, don't push to RubyGems. Push
to

geminabox. So we've subclassed Bundler and
said hey, don't

push to RubyGems. If you try to, if you

even try to call the private, the protected
method

there, you're gonna get an error. So just
don't

do it.

And then once we release that gem, it's just

a really small gem, we put ls/gem_tasks in
place

of bundler/gem_tasks in all of our other gems.
So,

by default this goes to our gem server when

you release. It's really easy.

So, gem servers. You, you know how to get

it out there and now you need some place

to put it. So there's three options. There's
geminabox,

stickler, and then gemfury. So if you're hosting
your

own, it should be behind the firewall. It's
internal

code. You don't want anyone else accessing
it.

Geminabox provides a nice web interface for
managing your

code. It has authentication. You have to build
your

own Rack middleware to do the authentication.
It does

pull through mirroring, which is kind of cool.
You

can tell it to mirror everything from RubyGems
as

you install things, so you don't need to depend

on RubyGems being up when you deploy. Geminabox
will

pull down everything for you.

And it has a nice command line client. It's

pretty simple. Patches gem with this inabox
command and

allows you to push and setup your own machine

to talk to your server.

The other option for open source thing, gem
servers

is stickler. Provides authentication in the
same kind of

way. It does selective mirroring, so you can
tell

it, you know, mirror ActiveRecord 3 point
2 and

it'll pull it from RubyGems. You can also
give

it a whole gem file, gem file lock, and

it'll take everything in there and it'll mirror
all

of it for you. Which is handy.

It has a really full-featured command line
client. You

can do everything that you can do from their

web interface. And then it's, the author wanted
me

to note that it's going to support the bundler

dependency API in the next release coming
soon.

For a hosted server that's not behind the
firewall,

it's on someone else's machine, there's only
one option

I know about. It's gemfury dot com. They provide

authentication obviously because it's not
on your machine. They

don't appear to do mirroring, which is, you
know,

if you're still rolling on someone else's
service you're

gonna probably be relying on RubyGems, too.
So, they

don't do mirroring.

They have command line client, and they do
support

node.js and Python packages. So that's it.
My name's

Tyler Montgomery, and I'm @ubermajestix on
Twitter.

??: Thank you, Tyler. Next up we have Dan

Mayer. He's also on our consumer app team.
Not

everyone is on the consumer app team. He does

a lot of teaching in the D.C. area, teaching

people Ruby. And I just learned that he can

fix bugs from ski lifts. Over the phone. So

that is a useful parlor trick. And take it

away Dan.

DAN MAYER: All right. Hello. I'm gonna be
talking

to you about production code analysis. Obviously
as you

build up all these services and split out
all

your code, it gets much harder to debug and

start finding out things about your code.
Like a

small application is easy to reason about
in your

head, but as your systems grow and the architecture

grows it gets more difficult.

Oh. Could somebody fix the projector and get
us

all on? Shrink down. Is that one on? Or

is that off as well? If somebody could fix

that one too, that'd be great.

You can't improve if you don't measure. You
can't

improve what you don't know. We often focus
on

performance and exception monitoring. Some
fixate on test code

coverage. There's a lot to learn from what
your

code actually does in production. Ruby tools
aren't quite

as good as, like, the Java tools, so we

really are still trying, as a community, to
improve

those tools. So that's some things we're working
on

internally.

And then, I wanted to mention Etsy. They have

some great posts all about measuring production
systems. Graphing

everything and actually release Statsd, which
we rely on

quite heavily. So I wanted to just say thanks

for that.

All right. So, I like to focus, as we

split things out and kind of forked off code

bases on getting rid of unnecessary code,
because I

want to be able to only think about and

reason about the code that we actually care
about

and what's actually being used in a system.

That code ends up in production for a whole

lot of reasons. I'd list up a whole bunch

there, but we don't really need to go through

every one. If you've ever worked on a team

as it grows, you eventually find code that
you're

like, this was written three years ago and
nobody's

using it. Why is it here?

I think it's important to try to track that

down and get rid of it as soon as

possible. Especially as you're adding a lot
of code

very quickly to systems.

So there's various ways to find that code.
Some's

as simple as just using New Relic, or formally

trace Linux. You can do custom stats instrumentation,
which

I'm gonna go, going to talk about a little

bit. There's production code coverage. There's
the gem I've

been working on that does code coverage, but
not

on your test suite. It's actually live code
running

in your system.

And after seeing some of the talks by Tilde,

I also realized I could make it much more

performant in Ruby 2.1.1, so I'll be working
on

doing that, cause right now there's a, a high

cost, which I'll talk about a little later.

You can use logs. You should definitely have
your

logs all searchable. And we'll go into that.
Using

stats in implementation and production code
coverage we've deleted

twenty plus thousand lines of app code. And
then

hundreds of thousands of lines of code, if
you

start including our assets, our JavaScript,
test files and

that sort of thing. We've really been able
to

shrink down our code bases as we expand out.

So, third-party, real quick. If you use New
Relic,

you can go to the transactions. Go look at

the last seven days. Sort by count. The things

at the bottom, likely, are dead, deprecated,
or you

can start working on killing them. They have,
like,

one request in the last seven days. You might

want to go look at what those are.

It's really easy but it's still useful. If
you're

ever finding an end point and you wonder,
wait

a minute, what is this? Go check. It might

not be hit in the last seven days. You

can probably kill it. It's really easy. It's
actually

even easier, because we have a gem, newrelicroutecheck,
which

basically you can download the CSV from New
Relic.

This generates your Rails routes file and
then compares

your routes versus the output of New Relic,
and

then starts telling you, this was in your
routes

file but never actually hit. You can go look

at those end points.

Stats instrumentation. We're gonna step through
all of these

fairly quickly. But basically there's a lot
of different

things you can do with stats to help you

find code and tell what's going on in your

systems. So, background events. You want to
know which

jobs are being executed. You want to know
if

they're completing successfully. You might
want to know how

long they take. This is an example for Resq.

We actually have our own wrapper around most
of

our background jobs, so we do it at a

different layer.

Statsd. Before preform. After preform. You
can throw in

whatever else you want there. If you want
timers.

Anything. But it's really easy to instrument
that. Then

you have your nice graphite graphs, where
you can

kind of follow and see the, see the charts

and everything. You can also look at graphite
and

see if a job's no longer ever being performed.

Emails. Email templates, especially transaction
emails. Eventually, you have

a one-off transactional email for some holiday
thing. Some

special product. Something that's no longer
being triggered. You

can check all your emails. There's no, no
reason

to work on updating them or fixing them, especially

when you're doing, say, a Rails update and
you

have to change all your ActionMailer code.
This will

let you know if it's worth your time.

Views rendered. Eventually your view layer
will get very

complicated, with layers and layers of partials.
If you

have layers and layers of partials, it's hard
to

reason about which ones are still in use,
which

ones have been refactored away entirely. You
can actually,

really quickly, using ActiveSupport::Notifications,
find all your layouts and

all your partials and templates and track
them. This

is an example showing it to Statsd, or you

can just render it into your Rails logger
and

then you can search it in something like Splunk,

ElasticSearch, see if it exists.

Again, we made a gem for this to make

it easier. You can see flatfoot. Basically,
you set

up flatfoot. It does this for you. And then

it gives you some helpers to find used views,

unused views. You can reset it on each deploy.

It makes it really easy to kind of automate

this process and generate reports on it.

One off trackers. Sometimes you just find
some code

and you're not quite sure what's going on.
It's

very easy, with Statsd, to just throw on one-off

trackers. This is an example of a controller.
We're

trying to figure out which of the types, or

paths in the controller are still being hit.
You

can go ahead and throw increment on each of

the formats. On that request dot XHR. Now
you're

going to be able to say, oh, actually we

stopped using the, you know, JSON format months
ago.

Let's get rid of all the related code.

Production performance comparisons. This one
I owe thanks to

uberjemef- ubermajestix over here, Tyler.
He actually just threw

this out and didn't really make any fanfare
of

it on one of our applications. Then I saw

this and I was like, that's really cool. We

had two different options for stripping out
some html,

and we wanted to know what's faster. You could

benchmark it locally on some fake data, but
the

production data is very different than what
you're going

to test with. And it, it will vary a

whole lot over all our applications.

He threw this in there. Splits it fifty percent.

Throws the timer around it. Measures both
one implementation

of the algorithm versus the other. Couple
days later,

it's incredibly obvious that nokogiri is much
faster than

just using the regular strip tags from Rails.

And that, that I, I just thought was so

interesting that I've kind of been carrying
around to

point out to everybody I can find that's interested.

Translation usage. This one I owe Chris Morris
some

credit. He is another engineer at Living Social.
He

works a lot on our payment systems. But he

used to do a lot of our translation and

internationalization work.

And eventually, your translation files get
really

large. And they actually take up a lot of

memory in your running Ruby processes. So
if you're

trying to shrink down your memory so you can

foot, fit more workers on a box, you might

want to look at your translation files. They
tend

to be a pain point for a lot of

companies that have a lot of languages.

This lets us track which keys are actually
being

used. What are we still translating in production?
Or

what is it we no longer use the translations

for. As you delete all these views, it's easy

to forget about all the translations that
were in

the views. This actually makes it very simple
to

go and find those.

Production code coverage. This is a little
bit more

experimental at the moment. We do run it on

production. I have it on a lot of systems.

I cannot use coverage, which is what, like,
simple

cov and things use in your test suite, because

there's a bug in Ruby. If you sample and

turn it on and off, it will crash. Which

luckily I learned prior to production.

So I switched to using set trace func, and

this lets us see each line as it's being

run live, and then we do sampling to cover

the fact that set trace func is extremely
slow.

Sometimes four hundred, eight hundred percent
slower. So we

can sample a very small rate of requests,
but

it still will, over a whole lot of volume,

give you a pretty good idea of what's used

in production.

And, like I said, in 2.1.1, I just learned

that you can actually get the CPU profiler,
is

a sampling profiler, and you can get the line-number

data out of it with a very small C

extension. So I'll probably be updating this,
and that

will only help 2.1.1. It'll still have to
use

set trace func on 1.9.3 and 2.0.0. But you

can get some really interesting data and delete
a

lot of code this way.

Here's a quick example of the setup, but really,

go to the gem's home page. You can see.

You can do ignores on things. It'll check
your

app path. It tracks in Redis. I use a

startup delay, because Rails is very slow
on the

first few requests, because it actually builds
a lot

of code in memory at that time. And you

really don't want to be checking your code
coverage

as it's building new methods. It's just not
a

good use of CPU.

Yeah. And this shows how to set it up

in your Rakefile. And real quick I'll, I'll
-

I was gonna have a demo. I think I

missed the slide. But we're, we've got other
speakers

so I'm gonna skip ahead real quick.

On logs, I just want to mention. Make sure

to get all your logs from all your applications

in one place. ElasticSearch, Splunk, Cabana,
doesn't really matter.

You just want everything in one place so you

can actually see how requests are going across
your

system.

I've been working on a gem called imprint.
But

also you can look at Zipkin, which is, I

guess, some of like Twitter's work. Mostly
they do

Scala, but I guess they have stuff for Ruby

as well. I just learned about that at this

conference. But basically, if you put a trace
id

on your requests, you can see as they're coming

in your system, and then as you make multiple

API requests to your backend, you can see
how

that fanned out in your system.

You can also drop it to all your events.

So any event that fails, you can track it

to the incoming request. If you have an exception,

it's going to be in the headers. It'll be

on your exception. You can actually take the
exception

and go see all the logs for all the,

the requests and debug what was coming in
that

was so awkward to cause this exception.

And, I think all these tools can help you

clean up your code and actually understand
what's happening

in production, much better. And that's all
I've got.

??: Thanks Dan. Right. Are, are you going
next?

Or is, K. I don't know.

I, I'm telling Rodrigo to sit down. Nick.
Nick

is gonna go next. Nick is a, a Minnesotan.

He's on our Mail tools team. A JRuby contributor.

He, and for one, he's six foot four inches

tall and can help you get things off of

tall shelves. And he's gonna talk to us a

little about meditation.

NICK SIGER: [French]

Which is French for, I think, therefore I
am,

of course, by the famous philosopher Renee
Descartes. I'd

like us to ponder today a little bit about

whether that's really an appropriate frame
of mind. You

can interpret that statement to mean that
we identify

ourselves, our entire existence, by the fact
that we

can think. And I personally think that's a
really

bad idea.

Early in my career, I was obsessed with my

capacity of work done. I wanted to work all

the time. And I was constantly searching for
that

sort of state, that elusive state of flow.
But,

in fact, what I think I really got stuck

in, most often, was a, an oft more common

state of rat hole.

So, when you think of yourself during the
work

day, you know, how often do you find yourself

getting distracted? Running down a rat hole,
and then

once you get done with that hole, that hole,

once you finally unwind the stack get all
the

way out of that rat hole, how many times

do you find you saying, wow. That felt great.

Like, no. It probably did not. And I'm arguing

that one of the reasons why it doesn't feel

great is because we all too often let our

minds run wild. We take every thought that
comes

into our head and we believe it's truth or

we believe there's some element of truth to
it.

We might even have perceptions about ourselves
and the

way other people perceive ourselves, and we
think that

for some reason those are true.

And I'm here to tell you that they're not.

I think that we have, all of us, inside

of us, we have a sort of a silent

watcher-observer of our lives that's behind
all the thoughts.

And if we can all take a little bit

of time in our day to access that part

of ourselves, we find that we're gonna be
happier.

We're gonna be healthier. And that's kind
of a

little bit of what I'm gonna, what I'm gonna

outline today.

So, as far as meditation goes, in terms of

discussion for today, I'm gonna say that meditation
is

sort of spending some quiet time, you know,
attempting

to find sort of an absence of thought in

your head. And if you've ever tried meditation
you

know this is actually extremely hard. People
have been

meditating for thousands of thousands of years,
and it

is most-certainly a life-long practice. And
when you realize

how hard, you think, oh my gosh, how am

I gonna actually- will I ever get to a

state where I feel comfortable with this?
Is this

something that's really worth while to do?

And I would advise up front that meditation,
like

many other things in our lives that we find

worth while to do, all it requires is just

a little bit of time and making it a

habit to get the most out of it. So

don't make, don't, don't listen to the voices
in

your head again. Don't listen to your thoughts
that

are saying, oh my gosh, I have no time

to do this. I'm, I'm not gonna do this

right. I don't know how to do this. It's

not gonna be perfect. So therefore I'm not
gonna

try. Just ignore all those and just try to

do it.

There's a number of studies out there, meditation
does

have actual tangible, scientific health benefits.
There's one study

that had people meditate for thirty minutes
a day

for eight weeks, and they did brain scans
on

the people before and after and found that
their

gray matter in brain regions associated with
memory and

stress and empathy actually increased in just
eight weeks.

So you can actually, you know, even as an

adult, you know, we're long passed puberty,
even as

adult you can spend time and change the composition

of your mind and the composition of your body.

Another study at UC Davis foc- said that focusing

on the present, doing activities like meditation,
rather than

letting the mind run away, adrift, actually
lowered levels

of the stress hormone cortisol in your body.
So,

meditation also will allow you to get on top

of your, your hormones, and get a better hormonal

balance. You'll feel a lot better about yourself.

So, unfortunately, a lightning talk is probably
a bad

place to actually try to, try to lead a

meditation session, since I think you, to
do it

justice, you'd want to probably have just
at least

a five minutes quiet time to yourself. But
instead,

what I'm gonna do is take just a few

moments and actually ask all of you to, you

don't have to do this yet, but what I'm

gonna ask you to do in a minute here

is close your eyes and just try to get

comfortable, and try to clear, clear your
mind of

any thoughts.

And then what I'm gonna ask you to do

is to just spend some time trying to obtain

that, that, that clear head, and watch your
mind

for the very next thought that comes out.
And

watch your mind like a cat would watch a

mouse hole. So I'm gonna ask you all now

to close your eyes, and watch your mind, and

just try to watch for the very next thought

that comes in. And just try to be focused

and attentive. Don't try to follow it, don't
try

to lull yourself to sleep or anything. Just
be

focused and attentive and try to look for
that

very next thought.

And I'll just give you a few moments. And

when that thought comes in go ahead and open

your eyes. OK. Some of you actually lasted
quite

awhile there. That's pretty good. So what
you may

be finding with this little exercise is that
when

you take the time to be really attentive about

having a clear mind, it actually takes a little

while longer for that thought to come in.

So this is, this is actually the very beginning

of the meditation practice. You're actually
trying to, rather

than being completely, completely lifeless
or, or almost like

a, well, you, you're actually trying to be
very,

very attentive to the immediate present moment.

So, when you're doing this, it's important
to not

be self-critical, right. So maybe a thought
rushed in

right away, again, and you feel like, oh gosh,

I have no control of it. Well, don't try

to control what's happening. Don't try to
be on

top of it. We, you know, we're all control

freaks a lot. We're in technology. We like
things

to work. We like to control computers. Don't
try

to be in control of the situation. Just observe

where your thoughts are jumping to. And just,
just

try to be that passive observer of what's
happening

in your mind.

And then, so, once you start to get in

this habit of taking time to at least check

in with yourself, maybe you start a meditation
practice

or maybe you just want to take moments out

of your day to just take a break from

things and be a little bit more mindful and

a little more self-aware. Ask yourself throughout
the day,

am I at ease at this moment? You know,

ask yourself, how do I feel at this moment?

Am I at ease? Or what's going on inside

me. And just, just pay attention to those
feelings

and those, what's going on inside you without
judging

or, or trying to make any changes about it.

We're just trying to cultivate self-awareness
here.

I also find that doing meditation has lead
me

to doing other things really mindfully as
well. I

like to, I like to cook. I like to

clean. I like to do all these little kind

of tasks that don't require a huge amount
of

mental energy. And I like just focusing on
them

intently. And I find that even doing things
like

chopping vegetables or, or clean- or doing
dishes or

any task which you think normally would be
completely,

you know, oh, what a waste of time. I

don't want to do this right now, I really

don't want to do these chores. You can actually

find a little bit of pleasure in just focusing

completely on the task.

And that's another way to just kind of open

yourself up and be, be more at ease. And,

you know, of course, we all have screens on

us at all, at all times of the days

these days, and that's another, you know,
where it's

like, looking at a screen is, you're putting
your

attention out into the ether, out somewhere
else from

where you are right now. So when you're with

someone, even when you're by yourself, maybe
every now

and then put the screen down and just focus

on, on just what you're doing.

And I think, for me, my opinion of, of

the, you know, this, this lifelong process
of trying

to be more mindful, trying to meditate, trying
to

make, make ourselves better through meditation,
is actually a

contagious thing and one that we can contribute
to

the well-being of the entire planet. So I
hope

you will consider, you know, reducing the
crazy noise

in your mind and maybe check out what meditation

can do for you and kind of follow up

on some of those studies I, I mentioned.

I will have, I'll Tweet a link later on

it. I was planning on posting it before this

talk but I forgot to do it. I have

to get back to my computer. But I will,

I will Tweet a link later. My, my Twitter

handle is @nicksiegr, N-I-C-K-S-I-E-G-R, and
I'll Tweet a link

later with some of the, the articles and links

to thinks that I, that I look, that I

refer to during the course of my talk. So,

meditate. Be happy. Make the world better.
Thanks.

RODRIGO FRANCO: Can I go now?

??: Now's your time, Rodrigo. So, Rodrigo
is a

developer on our merchant team. And he's an
avid

game, both board and video, right?

R.F.: Yes.

??: And you're from Brazil, right?

R.F.: Yes.

??: Definitely south of the Equator.

R.F.: Yes.

??: And, if you see him in the hall,

ask him about his goat.

R.F.: They're gonna, they're gonna see them.

??: All right.

R.F.: But yeah.

??: Take it away.

R.F.: Thank you. Is this one? You're strong.
All

right. So, hello everyone. My name is Rodrigo.
And

I'm here to give you one trick to boost

your productivity when working from home.
My Twitter handle

is @caffo. My wife call me that. So you

can call me that, too. Or Rodrigo. Whatever
you

want.

I wrote an article for, or tech blog, about

how I do work from home, and how I

try to focus on things while doing that. The

most important thing that people talk about
was the

pet goats, and that they were in the article,

like, briefly. I'm gonna show you some.

Here's me with my pet goat in a hangout

for work. If you want to read the article,

it's available here. It's a sign in link.
And

I have been working remotely from home for
about

ten years. So, I had to find my way

to focus on stuff. Otherwise I'd just play
video

games all the time. And, that's why we have

this unrelated cat picture.

And, this article name is One Weird Trick,
because

while I was browsing the internet without
ad block,

I saw a lot of these, these like ads,

like. One tip for a flat belly. And, like

DHH said, like, there's no way we can have

like a quick flat belly. So, what, what would

be, like, this one trick that we can get

to have focus? And I think it's boundaries.

Like, I had a couple things I do that

limit my work day and my personal day, because

at home they are very together. And I'm just

gonna briefly show you some of these things,
like

everyday I start my day with a startup ritual.

I get presentable, because we do hangouts
and I

need to be properly dressed. Not undressed.
Whatever. And

then I do what I call a, I open

the cafeteria. I get, like, some fizzy water
done.

I put coffee in the press and I grind

the beans. And then I have a very good

cup of coffee.

The other thing that I think it's important,
after

all of that done, I get my cup of

coffee. I sit down. I start working. And I

think you really, you really need to eliminate
distractions.

You need to find a way to be in

your mind office while in your home. And what

I found out was very good for me was

having a good pair of head phones. The head

phones are nice because they allow you to
focus

on stuff, and like people around you, like,
my

wife knows when I'm with the headphones, she
can't

talk to me. The dog don't care about, a

lot about it, and pet goats just get sad

because he can't get my attention.

And I also have all sorts of gizmos. Like,

I plug the headphones there, so I can have

even more like better sound. And that way
my

wife just, like, needs to throw stuff at me.

Like, wake up, get out of here. I need

something. And I call that my focus flag.
It's

like a way I can show to the world

and to myself that it's time to work. I'm

not here. I may be here, in my body,

but my mind's doing other things for my company.

So, OK. You got there. You had your startup

ritual. You got your computer. Your headset.
Goats away.

So, you can really start working. And then
you're

done, because you do a lot of multitasking.
And

you can't do that. Like, why we spend all

the time in Hacker News, on Reddit. Because
it's

way more fun than do work. Like, you have

that huge list of things to do, but you

want to be on Reddit.

So we need to cut down multitasking. You need

to find ways to not change what you're doing.

So if you're coding, you're coding. You don't
have

like, if gmail is loaded, you should not open

a new browser tab. Oh, but I'm just gonna

browse a second. Your day is gone.

So, what I done, for example, I had these,

these tiny Arduino plugged into my computer,
so if

someone sends me an IM message and they are

in a white list, I would get a light

that says you need to answer them because
it's

your boss or someone from your team. If it's

not, like, you can answer whenever you have
time.

So you need to find ways to not multitask.

Another important thing is, don't work all
the time.

When I started working remotely, I said like,
yeah.

I need to make sure I don't play video

games all the time. And I started working
all

the time. And I would never stop. So, you

need to find a good balance between one thing

and the other. You should also spend time
with

others. Like, don't spend all the time at
the

computer, like just working mindless. Like,
find time to

spend time with other people. And also find
time

to learn new stuff, like challenge yourself
on Code

Wars. Do some Khan Academy stuff. Like, you
need

to have a good balance in your life to

make sure your work is, works well, and everything

goes fine. Yeah.

That's it. Thank you so much.