GoGaRuCo 2014- The Scientific Method of Troubleshooting

First of all, I just want to start off
by thanking the organizer.

They really put in a lot of effort

to make me feel really special as a speaker.

Also as an attendee,
watching these other talks

so I just want to give them a huge thanks

Can I get an extra round of
applaud for the organizers

And also, thanks everyone for being here.

So, I'm Blithe

You can find me @blithe
on Twitter and Github

I work for a company call Big Nerd Ranch

It's based in Atlanta and

some people know it. Alright!

It's based in Atlanta and
I work remotely full time in Oakland

We do a couple of things at
Big Nerd Ranch

We develop apps, we write books
and we also teach courses.

And of course, we're hiring
good people, always.

As you heard, I used to be a scientist

I have a PhD in Physical Chemistry

And a lot of people ask me
what is physical chemistry?

what does that mean?

Well, if you think about it
It's right on the border between

what you know as physics and
what you know as chemistry

That means I spent a
long time in grad school

It also means I've done a lot
of experiments

A lot of people asked me
"How did you do it?

How did you make it
through 6 years of grad school

make it to the end?"

And I did it by using the scientific method

and I'm still using it everyday

I'm a developer, but
I'm still using the scientific method everyday

How many of you have gotten an error

Google it

pull the first answer straight from StackOverflow

and put it in your code and run it

I think we all tried it

And did it solve your problem?

Maybe

Maybe it did.
Maybe if you were lucky

But probably not

Most of the time, probably not

and it was pretty inefficient

and the more important question is

Did you learn anything from that?

The most important part of troubleshooting is learning

If you solved a problem without learning
from it,

It's only a temporary fix

It's either going to come back later or
you're going to make the same mistake in the future

Learning is how you level up as a developer

by learning from your experiences

And that bring me back to the scientific method

How can using the scientific method
help you level up as a developer?

Well, What is it?

Well, if you Google it and go straight to
Wikipedia you'll see this definition

It's a set of techniques for acquiring knowledge

And I love this definition because it can apply to anything

any career, including being a developer

The scientific method, it's methodical

It's an established procedure and
scientists have been using it since the 17th century

It's also systematic

It's a step-by-step process

and it's for gaining knowledge
and incorporating that into your process

and the time for reflection is built right into the
scientific method

The other day, a coworker of mine popped up
in the company chat room and said

"Has anyone seen Ruby string interpolation fail?"

and I thought that's a pretty interesting question

Why do you ask? Why are you asking that?

So we dug a little deeper and they were
using factory girl sequencing method

to create a series of users
with unique email addresses.

So email 1, email 2, email 3...

and it wasn't working. It was failing.

It was trying to create users
with the same email address

and it was failing and
they couldn't figure out why

and their solution,

they ended up just rolling
their own sequencing method

Well, that may have solved
their problem temporarily,

that wasn't really the
most elegant solution

and it turned out,

the real problem was that they
were missing the hash symbol

that was required for interpolation

so simple mistakes, but we all
make simple mistakes like that

We all make them all the times

But maybe using the scientific
method would have helped in this case

The first step of the scientific
method is defining the problem

As a scientist, this means saying what is
the information that you're trying to gain?

What knowledge are you
trying to gain or learn?

What's important to keep in mind
when you're defining the problem is

forget what you think you know
and question everything

Just think about open up your mind, question
everything and just define the problem

It's also important to get rid of
your emotions at this point

You want to, maybe you're on a tight deadline
maybe you think you know what the answer is

I'm just going to go do this. I already
know what's wrong blah blah blah

Get rid of all that and
just define the problem

You can do that by
answering a few questions

What is behavior that
you're expecting to happen?

What's actually happening?

and how are those two things different?

and how will you know
when you solved your problem?

What is your criteria for success?

This is particularly important when

when other people come and
ask you to help them troubleshoot

Don't listen to their assumptions.

Get rid of that out of the way.

and just start from scratch and
answer these simple questions

What do you want to happen?
What's actually happening?

How will you know when
you fixed the problem?

Once you defined your problem,

you can start by doing
some background research

and in reality, you guys are already doing this.

Everyone's doing it.

In grad school, we would just be constantly reading

Before we would ever start an experiment,

we would just be reading for days and days and days

Reading recent articles

or sometimes really old article that we would have to go to the library and dig up

We were just constantly reading

And the reason is because you want
to have enough background information

So that you know you're starting
with the right experiment

As a scientist, it's really expensive both

It cost a lot of time and money to do an experiment

So you want to make sure you're starting with the right experiment

and you do that by making sure you have enough background information

As a developer, you can do that by knowing your environment

So you need to know the language that you're working in

and you can always become a little more familiar with it

go dig in that dark area that you're unfamiliar with

or how does this work? What's going on here?

When you're working on a specific application,

you can look at what are the dependencies that you're using

Get to know those and even which version of those dependency are you using

For the case of the factory girl sequencing,

probably just taking a look at the documentation might have helped

Oh, I'm missing the hash symbol. That's it.

You also want to be reading the literature

You want to stay up to date with blog post

Always be reading what's coming out,
what's new

There's always a lot of content out there,
so you want to be reading it

For your specific dependency,

you can look at the README

or the changelog for the most recent version

What changed, what's different

You can also look at the specific source code
of the version you're using

Either on GitHub or using bundle open and the gem name

You can pull it right up,
right on your computer

And dig around in there.
Look at what's going on.

And also a helpful thing is to look at the test suite

of this dependency that you're looking at

Sometimes that helps you tell
the expect behavior of the gem

What's suppose to be happening

Another helpful source of background information is
just discussion

with your friends and colleagues

Maybe they have already encounter your problem

and they know the perfect solution

The best first experiment for you to start with

That's really hellpful

Particularly, when you're struggling on a problem

maybe you been working on it for a long time

and you're not getting anywhere

it's time to get up and go ask somebody for help

Have you seen anything like this?

And that's exactly what my colleague did

They popped in the company chatroom

"Hey, I need some help on this"

We have a rule at Big Nerd Ranch

If you're working on something for more than an hour

and you're not making progress

Don't!

It's time to stop and go ask somebody for help

It's better to get help from other people

And of course, sometime the good
old rubber duck is a good helpful solution

Maybe the rubber duck doesn't talk back,

but just by explaining your problem

and how you got there is a good way to get ideas

You also want to make sure you can
replicate the problem that you're having

Make it fail.

I mean can you even reproduce it?
Is it an error you can reproduce?

And what are the steps that it takes to do that

Once you have that information,

you can go ahead and write an automated test

to help you solve that problem

After you gather a lot of background information,

you can then establish a hypothesis

A hypothesis needs to be educated and based in evidence

So this goes back to all the research
that you've been doing

And it also need to be something that you can test

So this goes back to defining your problem

and your criteria for success

So once you have a hypothesis,

then you can start to design an experiment

And the most important part of an experiment design

is that it test your hypothesis, of course.

Don't do an experiment that doesn't test your hypothesis

Couple of tips for designing an experiment

One good thing to start with is just dividing and conquering your code base

Well first of all, hopefully you've
written an automated test

So you can just run that test over and over again

to focus on the condition that you're working on

So that isolate a lot other behavior that you don't
need to test immediately right now

Of course, you can always comment
out big chunks of codes

If you're like "I know this stuff is working.

Let me kind of comment it out

and just run the part of the code
that I need to look at right now."

Breakpoints are excellent helpful way to do it

Say your code has to go through 10 steps

and only the last one isn't working

Go ahead and enter a breakpoint

or a binding .pry right in there

You can stop and just look at the code is broken,
that is not working

You can also

Say you suspect there's a problem
with the dependency or a gem

What you can do is maybe create a new app

and only use that one dependency or API

Are you still having the problem?

OK, well maybe you just got rid of all
of the other code that you have written

and you know it just with that

you can replicate it in a fresh app

That's another way you can isolate the code

Another important thing is to limit the variables

that you're testing

When you look at beginner troubleshoot sometimes

They try to change many variables at once

Maybe it's a problem with the controller over here

and the view or maybe it's in the model. I don't know.

and sometimes they get down a road
that's really hard to come back from

and they can't even answer the question
"what has changed?"

So you really don't want to do that

You want to limit your experiment
to one variable at a time.

Let's test one variable

and finish our experiment before we start another one

so that we know

and we need to make sure we do
the analysis part of the experiment

Don't be afraid to try something weird

Sometimes, you know you think, you're thinking

"Ah this never gonna work
but I'll try it anyway."

And you try it and you find your solution

Don't be afraid to do that.
Try something strange.

You can also try maybe

try something that you know is
going to break your program

Go throw some gibberish in there.
Type it in there.

And you know it's going to break it.

You know it's going to fail.

What happen when you run your program
and it doesn't fail?

Maybe that code was never even getting run

Maybe you're not including that file or something

So try something strange

A friend of mine told me a story the other day

about how they've created a table

anytime they were trying to do a join with that table,

it was throwing some obscure error
and they couldn't figure it out

What's wrong with this one table?
What's wrong with it?

So they ended up, they just

they got rid of the table

They created a new one and they named it Bananas.

Just something weird.

Like just do it weird.

And that actually fixed their problem.

So Bananas worked.

And they're just like "What? What is going on?

Why? Why did that work"

and it turned out they had named
their original table attributes

So of course that's a protected name
so it didn't work

So trying something weird,

naming the table Bananas,
that worked for them.

How many of you have ever
done something like this?

Where you dig, you super deep dive. Yeah.

You're like oh it's way down here.

There's some weird problem.

And it ends up being
just right on the surface.

Something simple.

When you're designing your experiment,

you want to focus on

Think about the hierarchy of blame.

Think about OK,

what is the most likely
candidate for this problem?

Once you eliminated
the most likely candidates,

then you can go on
to the harder problems

For example,

more recently changed code

is probably more likely to be at fault

than codes that's been in
production for a long time

and something that you wrote

is probably more likely to
be the problem than say,

something in Ruby

But through your experiments go ahead
and eliminate the most likely candidate

and then work your way down

Don't start by trying to figure out
what's wrong with Ruby

After you designed your experiment,

go ahead and carry it out

and you're going to start
gathering your data

that you're going to use for your analysis

So you need to think about OK,

what's the current status of your program now?

What's happening now?

Were there any changes?

Are there any new messages?

Are there any new results from your experiment?

What else changed recently?

Were there any other things that you changed?

What does the log file say?

Gather all that information

And this one may seem obvious

but it need to be said
even for senior developers

Read the error message!

Just read it.

Read it out loud.

Even that alone will help it.

Just read it.

Take a look at the stack trace.

Look at the line number.

What's going on?

That's very useful information for a reason

So gather that information

as part of your experiment.

Once you have all your data,

you can start doing your analysis

and you really don't
want to skip this part

This is the most important part

because this where the learning happens

This is where you level up as a developer

and it's built right into the scientific method

So now that you've done your experiment

Did it solve your problem?

That test that you wrote, is it passing now?

and what about all the rest of your test suite?

Make sure it's also passing

Make sure you solved your problem
without breaking something else

And make sure you met all
your criteria for success

And you also want to think about

what did you learn from that experiment?

What information can you get from that?

Don't just move on to the next thing and say

"OK, it's fixed now. Let's go"

And take a moment,

Take the time to reflect on your experiment.

What went well?
What didn't go well?

Think about that

And most importantly,

you need to understand

why you solved your problem

Have you ever said,

"Well, I don't why it works,
but it works."

I have news for you.
You didn't fix it.

You didn't solve your problem.

It's the understanding

that cement the knowledge in your brain.

That's what make you a better developer.

And for those of you in leadership positions,

maybe you're teaching a newer developer

It's important that you make sure
that they understand the why

So make them explain it to you,

Why did this change solved your problem?

Your results from your experiment

will also help you determine

what's going to happen in your next experiment.

Maybe you solved your problem already

and you don't have to do another experiment

but it will influence your experiment in the future

Maybe now you know
a better first experiment to start with

"I know.

I know the fix.
I'll start with this next time."

Or if you didn't solve your problem,

maybe you have a new error message

"Ok, you can work with that.

Now what's the new experiment?"

Or "Oh maybe it's a problem with this gem.

I should do more
background research on that"

Another important part of the analysis

is embracing your successes.

This is really important.

A lot of people they just,

they sort of skip over this part

and go back to work

But it's really important.

If you did solve your problem,

let that sink in

because that's going to drive you

for future troubleshooting problems

You need to remember all those problems

that you solved before

and know that you can do it

and that you can keep going

And even if you didn't solved
your problem this time

and you're still working on it

maybe you got a new error message.

That's progress and that's
something to be celebrated

Another important part of being a scientist

is keeping a good lab notebook

and the reason you do this is because

it tells the story of your experiences

either to you in the future when
you're going back to look at it

or to somebody else

maybe another developer in the future

So this is the picture of one of my
lab notebook from grad school.

I probably filled about 10 of these
and they're like 300 pages each

So I did a lot of documenting in grad school.

The reason I included this is just to say

" it's OK to have a paper lab notebook
and write the things that you're doing

That's OK. Digital work also."

And the reason you keep
a lab notebook is because

you're just not going to remember it

you're not going to remember

As a grad student,

we probably did about 20 to
30 experiments a day

each just changing one variable at a time.

And there's just no way we would've remember

all of the things that we tried

And think about how many experiments
you do as a developer each day

when you're troubleshooting

And maybe you think

"OK, well I'm already just
recording all this stuff in the log.

It's all saved over there.

I can always go back and look

But that's not an sufficient lab notebook

Too much information can be just
as bad as too little information

So it's up to you

to glean the meaning from
these experiment that you're doing

and record that

One of the best first step for
keeping a good lab notebook

is writing a good commit message

It's all fresh in your brain

and so you can maybe you solve the problem

so you can commit that change and push it

So write some explanation in that

that commit message and let
people know what's going on

and people will see this in the future

Either you'll come back and look at it

or maybe another developer who's working
in the codebase will come back and look

Additionally , if you added test for
the scenario that you're working on,

that's also a good documentation for what's going on.

"OK, this was the the problem.

We added a test for it now."

Also, right after you solve a problem,

you want to update the documentation.

Is it missing anything?

Did you have that error because

the documentation wasn't good enough?

So go ahead and go update the README.

Or maybe you were missing
some environment variable

Go ahead and set up an example
environment variable file

so people know "Hey, you need this thing."

to save them that time

So that's all part of good documentation.

so keep that

Set that up for other people.

You can also contribute to open source.

So say you find a problem in
someone else software or gem,

go ahead and let them know

So open up an issue on Github
if it doesn't already exist.

Let them know the step that
it takes to recreate this problem.

If you were able to write a test for it,

go ahead and submit a pull request with that test

Sometimes that's helpful even if you
don't know how to solve the problem.

That's helpful for maintainers to say

"Oh, this is the test that's failing."

And of course,

if you do know how how to solve the problem,

go ahead and submit a pull request for that

You just got some open source contribution

and that's awesome

You can also write about your experiences

Have you ever written something up and then

later down the road and you're like
googling for that same problem

and you come back and
you find your own blog post?

Yes. Yes.

And you're like "Man,
thank you so much old me.

That was really helpful.

Thank you for writing that down."

So think about all those other people out there

that are getting the same reaction
when they find your blog post

and they're like "Man, that was really helpful blog post.

Thank you for writing that down."

So if you're not doing this already,
you should be doing this

Anyone can start a blog.

If you don't already have one, go start it.

You can also write failed experiment.

Don't be afraid to do that.

In grad school, we always joked about
starting a journal of failed experiments

That way we would publish a million papers that way

Like well, these are the million things
we tried and they didn't work

But that's OK. If you...

Sometimes that's also helpful

and maybe somebody comes along
and read this and can help you

and say "Hey,

you might try doing it this way.

That might solve your problem."

The most important part

about keeping a lab notebook
is sharing the the knowledge

In academia, it's all about

gaining knowledge and
then sharing that knowledge

either through publication
or through teaching

and I think developers can
really learn a lot from that

So as you're doing these experiments,

go ahead and share that information

Maybe tweet about it

or blog about it, like I said

or you can even come and present
a talk at a conference like this

"Hey, here's this weird
thing that I figured out.

Let me tell you about it."

So that's really the most important part

is sharing that knowledge with other people

and teaching it to other people

And so I'm really looking forward to
seeing all of your lab notebooks online.

Thanks very much