PAVAN SUDARSHAN: Hi. My name is Pavan.

ANANDHA KRISHNAN: And I'm Anandha Krishnan.
I'm

also called Jake. Not Anandha. I know. We
work

at MavenHive technologies.

P.S.: This is probably the only

talk where there are two speakers, and we
haven't

rehearsed who says what, so we'll be stepping
on

each other's toes, so yeah. Bear with us.
So

yeah. Quick disclaimer: Most of what we are
going

to talk about is actually platform and language
independent,

or at least, in a sense, learned. But the

reason we are talking here in a Ruby Conf

is because of the Ruby community. We really
think

that a lot of things we are going to

talk about resonates really well with the
community, and

we are hoping to, you know, drive a good

discussion, conversation, whatever it is,
from this audience. So

that's really why we are talking here on this

topic.

A.K.: And a lot of the points that

we're going to talk about kind of naturally
apply

to Rails and Ruby. And most of these we

learned, and we sort of experience in our
projects,

which are mostly in Ruby and Rails, so.

P.S.: Yeah, so let's start with condition.
We have screwed

up a lot in our careers. Right, between me

and Jake, we have no idea how many mistakes

we have made. And on those rare occassions,
we

have actually learned from it, or at least
we

would like to think so. So yeah, this talk

is about one such mistake from which we learned,

and yeah.

A.K.: And, yes, I think most of

it, we just put it up front, just based

on projects, and you know, as we were talking

about what happened with each of us, and things

like that. So yeah, just trying to make it,

you know, presentable and that stuff. But
we-

P.S.: Yeah, like, though we screwed up, we
would like

to believe that no employers or paranoid androids
were

hurt in the process of our mistakes, so yeah.

OK, about three months back, so why pharamcist
and

doctors, right? So about three months about
I was

in this pharmacy buying diapers for my daughter,
and

in walks this guy - he just goes straight

to the pharmacist and he's like, hey, can
you

give me something for a toothache? There was
something

very interesting and weird about this, and
Jake and

I, we carpool. So the next morning I was

just telling Jake, and between the two of
us,

like, between the two of us, we realized that

we have seen people ask for all sorts of

medicines in a pharmacy. Right. Headaches,
fever, like, like,

true story, I even once saw this guy with

a lot of cuts on his face from a

knife, and yeah. Insane. Insane, right. So
about, when

we were talking about this, we thought there
was

something fundamentally wrong with this. Is
there anyone here

who thinks that it's totally OK for you to

just walk up to a pharmacist and ask for

a medicine? Oh yeah, cool. Yeah, so. Nice.
OK.

Hold that thought. Yeah, but like. So what
we

think, yes, a pharmacy potentially has a cure
for

pretty much any, most ailments that you could
think

of, and but the important thing is, though,
you

need to know what ailment you have. Right,
there's

that small implementation detail, right. And
if it was

that easy for you to just go to the

right medicine and get it, this world would
be

filled with only pharmacists and not doctors,
right. Yeah,

so that's, so that's where the whole analogy
starts,

and then we'll get to how we connect to

what we're going to talk about.

A.K.: Yeah, and that's, that's where we, in
a sort of thought

that we use this metaphor to drive home that

point. Of course, a lot of you might have

your opinions about self-medication and the
whole thing. So

we'll stop it at this, and we will give

us, our definition of what we think about
these

two sort of mindsets actually are and, you
know.

So starting off with like doctors, right.
They don't

treat, rarely, they don't try to treat the
symptoms,

it's about how you deal with them. So they

go about just figuring out what the problem
could

be, you know, and then probably, you know,
a

lot of tests, make you run through a few

tests or try and figure out what's what, if

indeed it is the problem, and then try and,

based on that, prescribe a treatment and,
of course,

make sure that you're OK at the end of

it, right. The symptoms are gone.

P.S.: And we didn't take a look through a
medical textbook, so

we don't know if this is right, but assuming

it is, though, this is what worked for us,

so.

A.K.: And, again, in contrast, a pharmacist's
job,

we think very different, should be more about
understanding

the medicines, the medicines themselves. Probably
even figure out

what the disease is based on the medicine,
right.

But definitely they don't really think about,
you know,

what was the problem originally or what are
we

prescribing the treatment for. And they usually
don't do

it. Hopefully they don't do it.

P.S.: Yeah. OK.

So now with this context of what we mean

by a doctor and a pharmacist and medicines
and

self-medication, all right. Let's get back
to our mistake,

which we want to talk about. Right. So our

mistake that we want to talk about is a

way we have dealt with, or rather we used

to deal with bad symptoms on our code bases

and on our projects, right. So you, a lot

of times you see these in your code bases.

There's the symptom, or there are some issues,
right.

So we obviously had a lot of those, in

every single project we have worked on, and
this

is about how we dealt with that, right.

A.K.: Let's start off with one very simple
one, or

at least the one which was the most easily

expressible.

P.S.: Yeah, as in, Tejas Dinkar, who has

already been mentioned several times in different
talks, he

threatened us to throw off the stage if we

took anything more than twenty-nine minutes,
fifty-nine seconds. So

we had to like really dumbify our, you know,

anecdotes. But you learn, we have like a quick

mention of different things which we would
love to

talk about offline, but yeah. So thanks Dejas.

A.K.: So we'll get started on the first one.
So

this was a project where we had a familiar

problem of regression bugs. We added new features,
and

that kept breaking things. So this is what
we

designed. We want this down. We want the number

of bugs down from 10 to 5, you know.

At that time it was, like, let's set up

this goal, let's try and achieve this. What
did

we do? Oh, before that. Some facts about the

project, right. This was not a project we
started

on from scratch, it was a lot of legacy

code, a lot of code that we did not

understand, and probably that's why we thought
it was

bad. And the test coverage was-

P.S.: How many of you have taken over a code
base from

another team and thought the code base was
awesome?

Very small samples, so you realize what we
mean

when we thought it was bad. So.

A.K.: Sure.

So test-coverage was low, which was probably
one of

the reasons why people complained about the
code base,

of course. So what's your guess?

P.S.: OK, so the problem we had was every
time we checked

in something, built a new feature, touch any
part

of the code base, we ended up breaking a

whole bunch of other things. And we would
not

even know it right away, we would know it

over a period of time. Right, so this was

a regression problem. And given the facts,
what would

you probably have done?

A.K.: I'll try and just

go back again, and then hopefully forward
- we're

done [00:08:04]??.

P.S.: Yeah, based on some facts.

A.K.: Sure, so the low coverage was definitely
a problem.

We thought, I mean, everybody agreed that
we need

to start working on that, and you know, fix

the coverage. So we went in there, put the

coverage tool in place, you know. And then
we

decided we will write tests for every bug
we

caught. We got the coverage up, not very surprising,

I mean, we didn't manage to- you know, improving

the coverage-

P.S.: Yeah like, so we spend like the whole
time-

A.K.: over a period of time.

P.S.: OK, so, like so this was a problem,

right. When you look at that, when you try

to abstract the, into, like what our thought
process

was. It was something like this, right. Check.
Yeah.

So, we had a symptom. In this case it

was low test-coverage, and - Jake - and we

had, we decided on what metric and tool to

use. In our case it was a simple line

coverage, right and, archive?? [00:09:10],
this was back in

the days. And then we started solving that
problem

that we had, and then, you know, hopefully,
for

example, we were TDDing most of the new code

that we wrote, so coverage was good on that,

then start writing tests for things, which
were, any

part of the code base which we touched where

there were no tests, we started adding tests
there.

You know, a bunch of different things. So
basically,

like, the idea was to improve the coverage
and

keep on writing, right. So cool, so. And what

was the result? We ended up with, of course,

drastically improving our test coverage, so
we were in

the late '90s for most the part of the

code base, which was awesome.

A.K.: A hundred, man, a hundred.

P.S.: A hundred, yeah. Or, yeah, sure.

Very good coverage. But things got only marginally
better.

At this point, this was when we realized that

inspite of putting so much effort into actually
improving

our test coverage, our actual goal was to
actual

reduce the number of regression bugs. So we,
we

were still no better than what we started
off,

about two months back. So the developers were
generally

very happy, now they were doing a lot more

TDD, they have a lot, they had manag- successfully

convinced the manag- you know the product
manager and

the stake holders to spend more time on the

tech deck?? [00:10:29] and things like that.
They they

also were happy. But the project manager was
extremely

frustrated, because in spite of spending so
much effort,

there was no real benefit from any of it,

right. So it's like one of those very classic

moments where you know, in Dilbert where,
you know,

the, OK, in Dilbert developers are never happy,
but

at least here we were happy and the project

manager was sad, right. A.K.: And we weren't
happy

with the project managers as well.

P.S.: Yeah, and eventually we feel there was
something wrong,

it's not like we take pleasure out of it.
So, we

think this is a very big mistake, where we

spent almost two months of time without really
realizing

where we were going wrong, right. So this
mistake

and several mistakes across different projects
which Dejas won't

let us go into, ended up making us realize

something very basic, right. And that's, this
is basically

what, this is what we were going to say.

A.K.: OK, tell us a little bit.

P.S.: If we had like a lightning talk, this
is what

we probably the only thing we would have put

up and left. So never improve a metric. Solving

a problem should automatically imrpve the
metric that you're

measuring, right. So the focus is on, is never

on making a metric better. It's always about
solving

the problem. And, the metric-

A.K.: This is like

one of those, one of those things that is,

it's very easily said and-

P.S.: Yeah, and it-

A.K.: You have to, at least for us, we

always fell in that trap of-

P.S.: Yeah, like

it almost sounds like 'do the right thing,'
but,

yeah, like, it's very, it fits your common
sense

very well, but then when you're caught up
in

the daily, the day-to-day stuff in what you
do

in a project, it becomes very easy for you

to miss the point here. So yeah, like this

is really what, what is essence of what we

are trying to say, right.

A.K.: So, so what really happened here.

Let's go a little bit more

into what we were trying earlier and what
we

think we should have probably done. Instead
of something

like this, which, which actually ended up
attacking the

symptoms, or you know, targeting the symptom,
we want

to do something like this: There is a symptom,

so just like always-

P.S.: This is where the

whole, like, the pharmacist and the doctor
approach, yeah,

like, it's a very long-shot metaphor, we agree,
but-

A.K.: The doctor thinking, which we hopefully
want to

do, is first is just try and take a

guess at least at the problem, at least in

our context, maybe not the doctor's. But in
our

context, take a quess at the problem, right.
Think

what it might be. Then that hopefully will
tell

you what you could do to probably solve the

problem, solve, you know, that could be the
solution

which hope- will hopefully fix the problem,
right. So

this kind of very similar, we are iterating??[00:13:00]
over

this and hopefully, you know, improving on
what we

thought was the problem. And how do you know,

then, that you know we are in fact improving

on the problem? How do we know that is

the problem?

P.S.: Like the whole, how do you

define them, right. So, how do we define them?

A.K.: And that's where we think, that's where
we

think the metrics come in. Again, not metric,
hopefully

metrics, because that lets us measure the
problem. It

tells us at every point that, you know, you're

doing better, you know, it's improving. And
hopefully you

also, when it gets done, right. So, yeah.
So

this is, this is probably the approach that
we

would like to try on, try from now on,

also. And, right, there, you know, the problem
may

not be the, what is the one that we

started out to fix. Like, like, probably in
our

previous case, you know, it, you should always
be

open to the idea that the problem will, what

you thought was the problem was never the
case,

and it was not showing up. I mean, you

were trying to, you were seeing the metrics
improve,

but then the symptom never went away, right.
So

be open to the notion that the problem could

be different, in which case, the important
thing is

the solution is different and the metrics
are different,

right. So yeah.

P.S.: Any guesses on what could

have been the problem on that project? Where
the

regression bugs were written very high? It
was duplication,

as in there was, like, rampant duplication
all over

the place. And we would change something but
forget

to change the same thing in some other place.

But, and because we didn't know about the
code

base, yeah. We were just blindly adding tests.
And

incrimentally going through different places,
where each place where

we found that there were no tests, we added

a test, right. But that didn't really help
us

with actually solving the problem of duplication.
Yeah. So-

A.K.: The coverage number is something which
is, it

easily drives you to just keep adding the
specs,

and we will talk about, more about that soon.

P.S.: Yeah, so, if you really think about
it,

of, basically what, at least, we would love
to

believe that we have stopped doing this, right.
So

Yogi mentioned this in the panel discussion
yesterday, block-force

driven decisions, right. So that was really
what we

were doing, essentially. Like, we were a bunch
of

kids on this project, who'd see a problem
at

the first, the, at the first, trigger we would

just start crawling the web, start crawling
block force,

see different github projects, find a gem,
install it,

start, like, monitoring it, measuring it,
try to improve

it, you know. We don't really spending too
much

time into figuring out what was really the
problem,

right. And, especially, so in, OK - where
does-

Especially so in Rails projects, where, you
know, or

Ruby projects, where we believe that the number
of

gems which actually bundle best practices
is actually very

high, right. Here it's very easy for us to

fall into the trap of, OK, just choose a

gem, start using it, and then two months or

three months down the way you have no idea

where you used it in the first place. But

it's just there in your process, right. Yeah.
Like

at least we used to find ourselves in that

trap all the time. So yeah. This is basically

what we stopped doing. OK, at this-

A.K.: [indecipherable]

P.S.: So this, this dude is basically Hari,

he's sitting way over there. Yesterday we
were doing

a write-in??[00:16:14], and the only dude
that, after this

point it's fine, but it's getting monotonous,
it's very

black and white. And you need more images.
And,

like, Jake and I were really, we really think

that-

A.K.: That's our image, Hari!

P.S.: We really think that we don't know how
to add images

to a presentation. And we were like, OK fine

Hari, we'll just add your picture. And, yeah,
so

thanks for- and as you notice, we are very

receptive of feedback, so.

A.K.: He hasn't spoken yet,

but his is full of interesting ones.

P.S.: Like it would have been funny if like
his was

the presentation before we got the schedule,
but yeah,

anyway. He'll be talking next. So yeah, when
you

look at test coverage, right, how many of
you

think you understand test coverage very well?
Well enough.

OK, I mean, sure, yeah, like-

A.K.: This was one thing which at least took
us as a

very obvious metrics thing, which we always
get into,

and-

P.S.: When we were young and stupid as

against now old and stupid, right, we were,
we

used to think oh, test coverage, what's that,
it's

just - meh. It's so easy, right. But then

we realized even something so seemingly obvious
had, like,

so many different shades of details. And once
you

start understanding it and interpreting it
in context is

when you really understand the, the complexity
of that,

of that thing that you're measuring, right.
Now think

of all the things that people at Flipkart
measure.

I don't even know if they have a rational

behind every one of it. I'm hoping they do,

but you know like, it becomes very OK, we

just need to monitor these ten things. Why?
Why

are you doing it? Right. So it should not

be like a checklist at every project on start.

You just start using it. So yeah, test coverage

is definitely one thing that we found where,
on

start of every project we just set up a

coverage tool. We just wanted to talk about
some

details on what we learned when doing that,
so

yeah.

A.K.: So first we want to start off

the obvious one. The measuring. So controller
specs versus

model specs coverage. I- I'm guessing it,
does it

ring any bells? I'm- so, so I'm, think of

it this way, like, he has a question for

you, right? You have, let's take a simple
case.

There is a single controller, you have a spec

around it, there's a corresponding module,
you have a

spec around it, right. And then, you, with
these

two tests you run your coverage, right.

P.S.: And you get some coverage from-

A.K.: I'm guessing it's going to be a hundred
percent.

P.S.: Do you see a problem with this? Could
there, rather, OK,

could there be a problem with this?

A.K.: What if you just removed the model spec?

P.S.: What will the coverage for model be?
Is there a

chance that model's coverage is not zero?

A.K.: Your controller spec is still gonna
be loading the model.

So your coverage-

P.S.: Depends on-

A.K.: -is still in question.

P.S.: The answer is it depends, right,

like, really depends on how you are testing
your

controller. But most of all things, what we
have

seen is, not every model is mocked out in

the controller. Well, it's a totally different
debate, whether

should you mock your models or not, but if

you are not modelin- or, mocking, your models
are

being loaded in your controller. So the controller
spec,

when it is tested for, like when the coverage

is reported your models are being reported
as well.

A.K.: While, while the, or the point of, the

point we are trying to make is, here, is

not whether you should, how you should test
your

model specs and controller specs. What we
do implore

you to do is make sure that when you're

doing, when you're looking at your coverage,
you do

have in mind your testing strategy, which
is, am

I actually mocking the model out or is it

also getting wrote as a part of my controller

spec? Is my controller spec also hitting the
models,

right? Think about these things when, when
you're looking

at these numbers, right. Or something that
worked for

us which we tried to do was we started

monitoring the model specs coverage independently,
and then started

looking at the controller specs in light of
the,

in light of the model spec coverage. We wanted

the model spec coverage to be high, because
at

least we wanted all, hopefully all our business
logic

was in the model specs, and you know, that's

what we were keen on. Yes. Yeah, and then

the next one, the line coverage itself, I
think

most commonly when we talk about coverage
we just

talk about line coverage. But then there is
a

bunch of other things as well, branch coverage,
and

then unique path coverage.

P.S.: How many of you

pay attention to branch coverages?

A.K.: Or even monitor it?

P.S.: How many of you don't think it's

important? How many of you have no opinions?
Cool.

Yeah. I mean, sure. We have it on projects

where it's been important, it's not been important,
it's

fine. But it's just that, you just need to

know that it exists, and you need to train

your data, right, so.

A.K.: Just, hopefully it should

not be a single metric. Something usually
seems wrong

if it is just gonna be about that one

metric. Next one. So reporting. Yeah, this
one is

a bit tricky. What I usually don't like about

coverage tools and these tools in general
is they

sometimes miss out this aspect of it. And
they,

in an attempt to try and be nice to

you when you are very simple answered is try

and give you a number which inherently makes
it

good or bad. There's nothing in between, and
then

the focus is lost. Like you either start liking

it or you don't like it. You don't really

think about what is there in between, right.
Yeah.

So the focus on, focus on some of the

aspects of where is the coverage, you know,
what

does it mean, right. One thing that worked
for

us was code climate in the region projects.
I

really like it because they put a lot of

focus into the presentation aspect of it.
Not just

the collection aspect of it. It really, you
know,

takes you down to the, to the code, which

is missing the specs. Of course, they do also

other metrics like code quality, which I really
like,

by the way. They have some notification things
like

that, like, on, like Lexus a lot, whenever
this

climate goes in, poof the coverage goes down
or

something like that. [00:22:19] It doesn't
break the builder

part, it doesn't break the build, but it lets

you know, and then you can deal with it

if you think it is important to deal with.

P.S.: OK, speaking of breaking build, how
many of

you know what racheting, in builds? OK. So
the

idea of racheting is basically you will never
leave

your code base worse than what it already
was,

right. So every commit basically makes sure,
even if

it doesn't do any good, it doesn't do any

bad to your code base. So for example, if

you're, your current code coverage is at 70%,
and

if this check-in makes it 69%, it will break

the build. Even though there's nothing functionally
wrong with

it, you know, it's bad, right. We really think

it's a double-edged sword. I will, this is
one

of those things which I have a, in theory

sounds very, very good, and direct, but in
practice,

what it typically ends up doing is people
end

up fretting the actual metric, and never about
what

the problem is, right. Becau- this exactly
does what

we said in the previous slide, which is coverage

is never red or green, right. Sometimes you
are

OK with taking this hit because you want to

do something. I mean there are all, there
are

so many reasons why which, in practic- in
reality

you may have to do some bad things, and

eventually have to pay for it, but it's OK,

it's a conscious decision, right. But racheting
invariably stops

that. It makes it very, you know, black and

white, right.

A.K.: Difficult for you to proceed at

that very moment.

P.S.: Yeah and it has a

more behavioral impact on the team, which
is your,

your team members start hating either the
racheting, or

they start hating the metric, or you know,
we

had this one person who did not commit for

four days because they thought they did not
have

enough test coverage. And, like, when we breached
the

whole, OK, frequent check-ins, call check-in,
and this person

was actually scared because they were about,
they would

break the build, is actually a, very, very
bad

signal, of, a bad sign from your team, right.

Now well, you can always argue, OK, this person

totally missed the point, we can say a bunch

of things, right, but this is the reality.
So

we think that it's a good idea, so that

you might want to do it at certain points

in time. But yeah, be very, very careful about

the freakanomic implication that it has on
your team,

right, always keep that in mind.

A.K.: And then

the very other popular thing, which is, you
just

write that test to bump up the coverage, if

you're not just-

P.S.: Yeah, so there was, there

was just one more philosophy of full-on coverage??
[00:24:50],

which is saying, OK, you just checked-in code,
but

you don't have a test for this, but that's

OK. There's this other class which does not
have

a test for it, which is easy, so let's

start this there and keep my coverage maintained
right

there. There's so many, like, weird things
people end

up doing, just because they are now worried
about

coverage and not really worried about what
it means

to- you know, what it means with what they

are doing, right. So it's-

is done with the best of intentions, but then

you get really, really bad, cause, like-

P.S.: Yeah.

So, OK, how, how did we go, I mean,

how do we now improve, like, so we still

take on a lot of projects, customer, we are,

we are mainly now in consulting, so we do

end up taking worker code bases, right. So
what

do we do to improve coverage if we have

a bad one? One thing we realized is adding

unit tests to classes, existing classes, could
be a

very dangerous thing, right. What that essentially
means, is

you have, you know, you are cementing the
current

design. Now I won't say, like prejudiously,
it might

be bad to say not a good design, right.

But you are cementing it, right. If ever you

want to cement something, cement features.
Cement functionality, right.

Which means you might want to write like a

much higher level test of, and, you know,
ensure

that the functionality is the same, so that
you

can go and refactor later. On this one project,

which Yogi, me, and Jake worked together on
in

ThoughtWorks, it worked beautifully for us,
where we were

strangling ?? to Hibernate [00:26:14]. Which
meant that all

that entire unit tests and database level
tests were

completely invalidated, they were useless,
right. And because of

the way, and because we wanted to turn on

to caches?? [00:26:24], transactions changed.
Like that meant the

whole app was rendered useless from a testing
point

of view, right, from a safety net point of

view. But what came to our help was controller

and beyond level tests. So our biggest, we
had

such good coverage there, that we went in
and

we just modified a whole bunch of things,
and

we like started deleting tests, you know.
You get

a lot more flexibility and freedom inside
your code

base when you know that you're not breaking
any

functionality. So yeah like it's a really,
really, like,

good thing, so definitely think about it when
you're

inheriting a legacy codebase.

A.K.: Unit tests, unit tests

are great, but do keep in mind that they

might also come in the way of-

P.S.: Changes.

A.K.: Refactoring, and changes.

P.S.: Yeah.

A.K.: Big, big refactoring-

P.S.: So, OK. What - the whole measurement

reporting, racheting, improvement, all of
this is basically saying,

always keep the problem you're solving in
mind, right.

Rachet don't rachet - how do you decide? Well,

is it helping you to achieve your goal, or

the problem you have at hand? Sure, you know,

so do it. Otherwise don't do it, right. It's-

Yeah.

A.K.: I think-

P.S.: So, the second-

A.K.: Let's-

P.S.: -anecdote- I'll just be, quickly talk
about-

A.K.: We have five minutes, so-

P.S.: Yeah, like,

the second anecdote was basically, OK, we
had this

server which was under a very heavy load.
And,

like thousands of requests, a minute, and
only about

5% of those requests, in very seemingly arbitrary
periods

of times, and arbirtrary controllers and actions,
would pause,

and it would take a very long time, right.

So the- this problem was way more technical.
It

had nothing to do with the behavior or, you

know, like, or practices part of, or side
of

things. It was pure technical problem. And
goal was

for us to find the root cause of this

unpredictable behavior and fix it, right.
And, yeah, like

we were, like we can definitely talk about
how

we went into, like a lot of different solving,

a lot of different symptoms, at one point
even

suspecting JRuby. You know, like, so it's
very, like,

it becomes very hard for you to figure out,

OK, what is a problem, what is a symptom,

and like, go very methodical about it, right.
So

that's what this, this problem was going to
be

about. But let's take this offline. At this
point

we can-

A.K.: I mean, before that, yeah, let's

get out some questions.

P.S.: Yeah, let's take some

questions if you have any.

A.K.: Cool.

P.S.: Cool, thanks.

A.K.: Thanks.

P.S.: Thanks guys.