PAVAN SUDARSHAN: Hi. My name is Pavan.
ANANDHA KRISHNAN: And I'm Anandha Krishnan. I'm
also called Jake. Not Anandha. I know. We work
at MavenHive technologies.
P.S.: This is probably the only
talk where there are two speakers, and we haven't
rehearsed who says what, so we'll be stepping on
each other's toes, so yeah. Bear with us. So
yeah. Quick disclaimer: Most of what we are going
to talk about is actually platform and language independent,
or at least, in a sense, learned. But the
reason we are talking here in a Ruby Conf
is because of the Ruby community. We really think
that a lot of things we are going to
talk about resonates really well with the community, and
we are hoping to, you know, drive a good
discussion, conversation, whatever it is, from this audience. So
that's really why we are talking here on this
topic.
A.K.: And a lot of the points that
we're going to talk about kind of naturally apply
to Rails and Ruby. And most of these we
learned, and we sort of experience in our projects,
which are mostly in Ruby and Rails, so.
P.S.: Yeah, so let's start with condition. We have screwed
up a lot in our careers. Right, between me
and Jake, we have no idea how many mistakes
we have made. And on those rare occassions, we
have actually learned from it, or at least we
would like to think so. So yeah, this talk
is about one such mistake from which we learned,
and yeah.
A.K.: And, yes, I think most of
it, we just put it up front, just based
on projects, and you know, as we were talking
about what happened with each of us, and things
like that. So yeah, just trying to make it,
you know, presentable and that stuff. But we-
P.S.: Yeah, like, though we screwed up, we would like
to believe that no employers or paranoid androids were
hurt in the process of our mistakes, so yeah.
OK, about three months back, so why pharamcist and
doctors, right? So about three months about I was
in this pharmacy buying diapers for my daughter, and
in walks this guy - he just goes straight
to the pharmacist and he's like, hey, can you
give me something for a toothache? There was something
very interesting and weird about this, and Jake and
I, we carpool. So the next morning I was
just telling Jake, and between the two of us,
like, between the two of us, we realized that
we have seen people ask for all sorts of
medicines in a pharmacy. Right. Headaches, fever, like, like,
true story, I even once saw this guy with
a lot of cuts on his face from a
knife, and yeah. Insane. Insane, right. So about, when
we were talking about this, we thought there was
something fundamentally wrong with this. Is there anyone here
who thinks that it's totally OK for you to
just walk up to a pharmacist and ask for
a medicine? Oh yeah, cool. Yeah, so. Nice. OK.
Hold that thought. Yeah, but like. So what we
think, yes, a pharmacy potentially has a cure for
pretty much any, most ailments that you could think
of, and but the important thing is, though, you
need to know what ailment you have. Right, there's
that small implementation detail, right. And if it was
that easy for you to just go to the
right medicine and get it, this world would be
filled with only pharmacists and not doctors, right. Yeah,
so that's, so that's where the whole analogy starts,
and then we'll get to how we connect to
what we're going to talk about.
A.K.: Yeah, and that's, that's where we, in a sort of thought
that we use this metaphor to drive home that
point. Of course, a lot of you might have
your opinions about self-medication and the whole thing. So
we'll stop it at this, and we will give
us, our definition of what we think about these
two sort of mindsets actually are and, you know.
So starting off with like doctors, right. They don't
treat, rarely, they don't try to treat the symptoms,
it's about how you deal with them. So they
go about just figuring out what the problem could
be, you know, and then probably, you know, a
lot of tests, make you run through a few
tests or try and figure out what's what, if
indeed it is the problem, and then try and,
based on that, prescribe a treatment and, of course,
make sure that you're OK at the end of
it, right. The symptoms are gone.
P.S.: And we didn't take a look through a medical textbook, so
we don't know if this is right, but assuming
it is, though, this is what worked for us,
so.
A.K.: And, again, in contrast, a pharmacist's job,
we think very different, should be more about understanding
the medicines, the medicines themselves. Probably even figure out
what the disease is based on the medicine, right.
But definitely they don't really think about, you know,
what was the problem originally or what are we
prescribing the treatment for. And they usually don't do
it. Hopefully they don't do it.
P.S.: Yeah. OK.
So now with this context of what we mean
by a doctor and a pharmacist and medicines and
self-medication, all right. Let's get back to our mistake,
which we want to talk about. Right. So our
mistake that we want to talk about is a
way we have dealt with, or rather we used
to deal with bad symptoms on our code bases
and on our projects, right. So you, a lot
of times you see these in your code bases.
There's the symptom, or there are some issues, right.
So we obviously had a lot of those, in
every single project we have worked on, and this
is about how we dealt with that, right.
A.K.: Let's start off with one very simple one, or
at least the one which was the most easily
expressible.
P.S.: Yeah, as in, Tejas Dinkar, who has
already been mentioned several times in different talks, he
threatened us to throw off the stage if we
took anything more than twenty-nine minutes, fifty-nine seconds. So
we had to like really dumbify our, you know,
anecdotes. But you learn, we have like a quick
mention of different things which we would love to
talk about offline, but yeah. So thanks Dejas.
A.K.: So we'll get started on the first one. So
this was a project where we had a familiar
problem of regression bugs. We added new features, and
that kept breaking things. So this is what we
designed. We want this down. We want the number
of bugs down from 10 to 5, you know.
At that time it was, like, let's set up
this goal, let's try and achieve this. What did
we do? Oh, before that. Some facts about the
project, right. This was not a project we started
on from scratch, it was a lot of legacy
code, a lot of code that we did not
understand, and probably that's why we thought it was
bad. And the test coverage was-
P.S.: How many of you have taken over a code base from
another team and thought the code base was awesome?
Very small samples, so you realize what we mean
when we thought it was bad. So.
A.K.: Sure.
So test-coverage was low, which was probably one of
the reasons why people complained about the code base,
of course. So what's your guess?
P.S.: OK, so the problem we had was every time we checked
in something, built a new feature, touch any part
of the code base, we ended up breaking a
whole bunch of other things. And we would not
even know it right away, we would know it
over a period of time. Right, so this was
a regression problem. And given the facts, what would
you probably have done?
A.K.: I'll try and just
go back again, and then hopefully forward - we're
done [00:08:04]??.
P.S.: Yeah, based on some facts.
A.K.: Sure, so the low coverage was definitely a problem.
We thought, I mean, everybody agreed that we need
to start working on that, and you know, fix
the coverage. So we went in there, put the
coverage tool in place, you know. And then we
decided we will write tests for every bug we
caught. We got the coverage up, not very surprising,
I mean, we didn't manage to- you know, improving
the coverage-
P.S.: Yeah like, so we spend like the whole time-
A.K.: over a period of time.
P.S.: OK, so, like so this was a problem,
right. When you look at that, when you try
to abstract the, into, like what our thought process
was. It was something like this, right. Check. Yeah.
So, we had a symptom. In this case it
was low test-coverage, and - Jake - and we
had, we decided on what metric and tool to
use. In our case it was a simple line
coverage, right and, archive?? [00:09:10], this was back in
the days. And then we started solving that problem
that we had, and then, you know, hopefully, for
example, we were TDDing most of the new code
that we wrote, so coverage was good on that,
then start writing tests for things, which were, any
part of the code base which we touched where
there were no tests, we started adding tests there.
You know, a bunch of different things. So basically,
like, the idea was to improve the coverage and
keep on writing, right. So cool, so. And what
was the result? We ended up with, of course,
drastically improving our test coverage, so we were in
the late '90s for most the part of the
code base, which was awesome.
A.K.: A hundred, man, a hundred.
P.S.: A hundred, yeah. Or, yeah, sure.
Very good coverage. But things got only marginally better.
At this point, this was when we realized that
inspite of putting so much effort into actually improving
our test coverage, our actual goal was to actual
reduce the number of regression bugs. So we, we
were still no better than what we started off,
about two months back. So the developers were generally
very happy, now they were doing a lot more
TDD, they have a lot, they had manag- successfully
convinced the manag- you know the product manager and
the stake holders to spend more time on the
tech deck?? [00:10:29] and things like that. They they
also were happy. But the project manager was extremely
frustrated, because in spite of spending so much effort,
there was no real benefit from any of it,
right. So it's like one of those very classic
moments where you know, in Dilbert where, you know,
the, OK, in Dilbert developers are never happy, but
at least here we were happy and the project
manager was sad, right. A.K.: And we weren't happy
with the project managers as well.
P.S.: Yeah, and eventually we feel there was something wrong,
it's not like we take pleasure out of it. So, we
think this is a very big mistake, where we
spent almost two months of time without really realizing
where we were going wrong, right. So this mistake
and several mistakes across different projects which Dejas won't
let us go into, ended up making us realize
something very basic, right. And that's, this is basically
what, this is what we were going to say.
A.K.: OK, tell us a little bit.
P.S.: If we had like a lightning talk, this is what
we probably the only thing we would have put
up and left. So never improve a metric. Solving
a problem should automatically imrpve the metric that you're
measuring, right. So the focus is on, is never
on making a metric better. It's always about solving
the problem. And, the metric-
A.K.: This is like
one of those, one of those things that is,
it's very easily said and-
P.S.: Yeah, and it-
A.K.: You have to, at least for us, we
always fell in that trap of-
P.S.: Yeah, like
it almost sounds like 'do the right thing,' but,
yeah, like, it's very, it fits your common sense
very well, but then when you're caught up in
the daily, the day-to-day stuff in what you do
in a project, it becomes very easy for you
to miss the point here. So yeah, like this
is really what, what is essence of what we
are trying to say, right.
A.K.: So, so what really happened here.
Let's go a little bit more
into what we were trying earlier and what we
think we should have probably done. Instead of something
like this, which, which actually ended up attacking the
symptoms, or you know, targeting the symptom, we want
to do something like this: There is a symptom,
so just like always-
P.S.: This is where the
whole, like, the pharmacist and the doctor approach, yeah,
like, it's a very long-shot metaphor, we agree, but-
A.K.: The doctor thinking, which we hopefully want to
do, is first is just try and take a
guess at least at the problem, at least in
our context, maybe not the doctor's. But in our
context, take a quess at the problem, right. Think
what it might be. Then that hopefully will tell
you what you could do to probably solve the
problem, solve, you know, that could be the solution
which hope- will hopefully fix the problem, right. So
this kind of very similar, we are iterating??[00:13:00] over
this and hopefully, you know, improving on what we
thought was the problem. And how do you know,
then, that you know we are in fact improving
on the problem? How do we know that is
the problem?
P.S.: Like the whole, how do you
define them, right. So, how do we define them?
A.K.: And that's where we think, that's where we
think the metrics come in. Again, not metric, hopefully
metrics, because that lets us measure the problem. It
tells us at every point that, you know, you're
doing better, you know, it's improving. And hopefully you
also, when it gets done, right. So, yeah. So
this is, this is probably the approach that we
would like to try on, try from now on,
also. And, right, there, you know, the problem may
not be the, what is the one that we
started out to fix. Like, like, probably in our
previous case, you know, it, you should always be
open to the idea that the problem will, what
you thought was the problem was never the case,
and it was not showing up. I mean, you
were trying to, you were seeing the metrics improve,
but then the symptom never went away, right. So
be open to the notion that the problem could
be different, in which case, the important thing is
the solution is different and the metrics are different,
right. So yeah.
P.S.: Any guesses on what could
have been the problem on that project? Where the
regression bugs were written very high? It was duplication,
as in there was, like, rampant duplication all over
the place. And we would change something but forget
to change the same thing in some other place.
But, and because we didn't know about the code
base, yeah. We were just blindly adding tests. And
incrimentally going through different places, where each place where
we found that there were no tests, we added
a test, right. But that didn't really help us
with actually solving the problem of duplication. Yeah. So-
A.K.: The coverage number is something which is, it
easily drives you to just keep adding the specs,
and we will talk about, more about that soon.
P.S.: Yeah, so, if you really think about it,
of, basically what, at least, we would love to
believe that we have stopped doing this, right. So
Yogi mentioned this in the panel discussion yesterday, block-force
driven decisions, right. So that was really what we
were doing, essentially. Like, we were a bunch of
kids on this project, who'd see a problem at
the first, the, at the first, trigger we would
just start crawling the web, start crawling block force,
see different github projects, find a gem, install it,
start, like, monitoring it, measuring it, try to improve
it, you know. We don't really spending too much
time into figuring out what was really the problem,
right. And, especially, so in, OK - where does-
Especially so in Rails projects, where, you know, or
Ruby projects, where we believe that the number of
gems which actually bundle best practices is actually very
high, right. Here it's very easy for us to
fall into the trap of, OK, just choose a
gem, start using it, and then two months or
three months down the way you have no idea
where you used it in the first place. But
it's just there in your process, right. Yeah. Like
at least we used to find ourselves in that
trap all the time. So yeah. This is basically
what we stopped doing. OK, at this-
A.K.: [indecipherable]
P.S.: So this, this dude is basically Hari,
he's sitting way over there. Yesterday we were doing
a write-in??[00:16:14], and the only dude that, after this
point it's fine, but it's getting monotonous, it's very
black and white. And you need more images. And,
like, Jake and I were really, we really think
that-
A.K.: That's our image, Hari!
P.S.: We really think that we don't know how to add images
to a presentation. And we were like, OK fine
Hari, we'll just add your picture. And, yeah, so
thanks for- and as you notice, we are very
receptive of feedback, so.
A.K.: He hasn't spoken yet,
but his is full of interesting ones.
P.S.: Like it would have been funny if like his was
the presentation before we got the schedule, but yeah,
anyway. He'll be talking next. So yeah, when you
look at test coverage, right, how many of you
think you understand test coverage very well? Well enough.
OK, I mean, sure, yeah, like-
A.K.: This was one thing which at least took us as a
very obvious metrics thing, which we always get into,
and-
P.S.: When we were young and stupid as
against now old and stupid, right, we were, we
used to think oh, test coverage, what's that, it's
just - meh. It's so easy, right. But then
we realized even something so seemingly obvious had, like,
so many different shades of details. And once you
start understanding it and interpreting it in context is
when you really understand the, the complexity of that,
of that thing that you're measuring, right. Now think
of all the things that people at Flipkart measure.
I don't even know if they have a rational
behind every one of it. I'm hoping they do,
but you know like, it becomes very OK, we
just need to monitor these ten things. Why? Why
are you doing it? Right. So it should not
be like a checklist at every project on start.
You just start using it. So yeah, test coverage
is definitely one thing that we found where, on
start of every project we just set up a
coverage tool. We just wanted to talk about some
details on what we learned when doing that, so
yeah.
A.K.: So first we want to start off
the obvious one. The measuring. So controller specs versus
model specs coverage. I- I'm guessing it, does it
ring any bells? I'm- so, so I'm, think of
it this way, like, he has a question for
you, right? You have, let's take a simple case.
There is a single controller, you have a spec
around it, there's a corresponding module, you have a
spec around it, right. And then, you, with these
two tests you run your coverage, right.
P.S.: And you get some coverage from-
A.K.: I'm guessing it's going to be a hundred percent.
P.S.: Do you see a problem with this? Could there, rather, OK,
could there be a problem with this?
A.K.: What if you just removed the model spec?
P.S.: What will the coverage for model be? Is there a
chance that model's coverage is not zero?
A.K.: Your controller spec is still gonna be loading the model.
So your coverage-
P.S.: Depends on-
A.K.: -is still in question.
P.S.: The answer is it depends, right,
like, really depends on how you are testing your
controller. But most of all things, what we have
seen is, not every model is mocked out in
the controller. Well, it's a totally different debate, whether
should you mock your models or not, but if
you are not modelin- or, mocking, your models are
being loaded in your controller. So the controller spec,
when it is tested for, like when the coverage
is reported your models are being reported as well.
A.K.: While, while the, or the point of, the
point we are trying to make is, here, is
not whether you should, how you should test your
model specs and controller specs. What we do implore
you to do is make sure that when you're
doing, when you're looking at your coverage, you do
have in mind your testing strategy, which is, am
I actually mocking the model out or is it
also getting wrote as a part of my controller
spec? Is my controller spec also hitting the models,
right? Think about these things when, when you're looking
at these numbers, right. Or something that worked for
us which we tried to do was we started
monitoring the model specs coverage independently, and then started
looking at the controller specs in light of the,
in light of the model spec coverage. We wanted
the model spec coverage to be high, because at
least we wanted all, hopefully all our business logic
was in the model specs, and you know, that's
what we were keen on. Yes. Yeah, and then
the next one, the line coverage itself, I think
most commonly when we talk about coverage we just
talk about line coverage. But then there is a
bunch of other things as well, branch coverage, and
then unique path coverage.
P.S.: How many of you
pay attention to branch coverages?
A.K.: Or even monitor it?
P.S.: How many of you don't think it's
important? How many of you have no opinions? Cool.
Yeah. I mean, sure. We have it on projects
where it's been important, it's not been important, it's
fine. But it's just that, you just need to
know that it exists, and you need to train
your data, right, so.
A.K.: Just, hopefully it should
not be a single metric. Something usually seems wrong
if it is just gonna be about that one
metric. Next one. So reporting. Yeah, this one is
a bit tricky. What I usually don't like about
coverage tools and these tools in general is they
sometimes miss out this aspect of it. And they,
in an attempt to try and be nice to
you when you are very simple answered is try
and give you a number which inherently makes it
good or bad. There's nothing in between, and then
the focus is lost. Like you either start liking
it or you don't like it. You don't really
think about what is there in between, right. Yeah.
So the focus on, focus on some of the
aspects of where is the coverage, you know, what
does it mean, right. One thing that worked for
us was code climate in the region projects. I
really like it because they put a lot of
focus into the presentation aspect of it. Not just
the collection aspect of it. It really, you know,
takes you down to the, to the code, which
is missing the specs. Of course, they do also
other metrics like code quality, which I really like,
by the way. They have some notification things like
that, like, on, like Lexus a lot, whenever this
climate goes in, poof the coverage goes down or
something like that. [00:22:19] It doesn't break the builder
part, it doesn't break the build, but it lets
you know, and then you can deal with it
if you think it is important to deal with.
P.S.: OK, speaking of breaking build, how many of
you know what racheting, in builds? OK. So the
idea of racheting is basically you will never leave
your code base worse than what it already was,
right. So every commit basically makes sure, even if
it doesn't do any good, it doesn't do any
bad to your code base. So for example, if
you're, your current code coverage is at 70%, and
if this check-in makes it 69%, it will break
the build. Even though there's nothing functionally wrong with
it, you know, it's bad, right. We really think
it's a double-edged sword. I will, this is one
of those things which I have a, in theory
sounds very, very good, and direct, but in practice,
what it typically ends up doing is people end
up fretting the actual metric, and never about what
the problem is, right. Becau- this exactly does what
we said in the previous slide, which is coverage
is never red or green, right. Sometimes you are
OK with taking this hit because you want to
do something. I mean there are all, there are
so many reasons why which, in practic- in reality
you may have to do some bad things, and
eventually have to pay for it, but it's OK,
it's a conscious decision, right. But racheting invariably stops
that. It makes it very, you know, black and
white, right.
A.K.: Difficult for you to proceed at
that very moment.
P.S.: Yeah and it has a
more behavioral impact on the team, which is your,
your team members start hating either the racheting, or
they start hating the metric, or you know, we
had this one person who did not commit for
four days because they thought they did not have
enough test coverage. And, like, when we breached the
whole, OK, frequent check-ins, call check-in, and this person
was actually scared because they were about, they would
break the build, is actually a, very, very bad
signal, of, a bad sign from your team, right.
Now well, you can always argue, OK, this person
totally missed the point, we can say a bunch
of things, right, but this is the reality. So
we think that it's a good idea, so that
you might want to do it at certain points
in time. But yeah, be very, very careful about
the freakanomic implication that it has on your team,
right, always keep that in mind.
A.K.: And then
the very other popular thing, which is, you just
write that test to bump up the coverage, if
you're not just-
P.S.: Yeah, so there was, there
was just one more philosophy of full-on coverage?? [00:24:50],
which is saying, OK, you just checked-in code, but
you don't have a test for this, but that's
OK. There's this other class which does not have
a test for it, which is easy, so let's
start this there and keep my coverage maintained right
there. There's so many, like, weird things people end
up doing, just because they are now worried about
coverage and not really worried about what it means
to- you know, what it means with what they
are doing, right. So it's-
is done with the best of intentions, but then
you get really, really bad, cause, like-
P.S.: Yeah.
So, OK, how, how did we go, I mean,
how do we now improve, like, so we still
take on a lot of projects, customer, we are,
we are mainly now in consulting, so we do
end up taking worker code bases, right. So what
do we do to improve coverage if we have
a bad one? One thing we realized is adding
unit tests to classes, existing classes, could be a
very dangerous thing, right. What that essentially means, is
you have, you know, you are cementing the current
design. Now I won't say, like prejudiously, it might
be bad to say not a good design, right.
But you are cementing it, right. If ever you
want to cement something, cement features. Cement functionality, right.
Which means you might want to write like a
much higher level test of, and, you know, ensure
that the functionality is the same, so that you
can go and refactor later. On this one project,
which Yogi, me, and Jake worked together on in
ThoughtWorks, it worked beautifully for us, where we were
strangling ?? to Hibernate [00:26:14]. Which meant that all
that entire unit tests and database level tests were
completely invalidated, they were useless, right. And because of
the way, and because we wanted to turn on
to caches?? [00:26:24], transactions changed. Like that meant the
whole app was rendered useless from a testing point
of view, right, from a safety net point of
view. But what came to our help was controller
and beyond level tests. So our biggest, we had
such good coverage there, that we went in and
we just modified a whole bunch of things, and
we like started deleting tests, you know. You get
a lot more flexibility and freedom inside your code
base when you know that you're not breaking any
functionality. So yeah like it's a really, really, like,
good thing, so definitely think about it when you're
inheriting a legacy codebase.
A.K.: Unit tests, unit tests
are great, but do keep in mind that they
might also come in the way of-
P.S.: Changes.
A.K.: Refactoring, and changes.
P.S.: Yeah.
A.K.: Big, big refactoring-
P.S.: So, OK. What - the whole measurement
reporting, racheting, improvement, all of this is basically saying,
always keep the problem you're solving in mind, right.
Rachet don't rachet - how do you decide? Well,
is it helping you to achieve your goal, or
the problem you have at hand? Sure, you know,
so do it. Otherwise don't do it, right. It's-
Yeah.
A.K.: I think-
P.S.: So, the second-
A.K.: Let's-
P.S.: -anecdote- I'll just be, quickly talk about-
A.K.: We have five minutes, so-
P.S.: Yeah, like,
the second anecdote was basically, OK, we had this
server which was under a very heavy load. And,
like thousands of requests, a minute, and only about
5% of those requests, in very seemingly arbitrary periods
of times, and arbirtrary controllers and actions, would pause,
and it would take a very long time, right.
So the- this problem was way more technical. It
had nothing to do with the behavior or, you
know, like, or practices part of, or side of
things. It was pure technical problem. And goal was
for us to find the root cause of this
unpredictable behavior and fix it, right. And, yeah, like
we were, like we can definitely talk about how
we went into, like a lot of different solving,
a lot of different symptoms, at one point even
suspecting JRuby. You know, like, so it's very, like,
it becomes very hard for you to figure out,
OK, what is a problem, what is a symptom,
and like, go very methodical about it, right. So
that's what this, this problem was going to be
about. But let's take this offline. At this point
we can-
A.K.: I mean, before that, yeah, let's
get out some questions.
P.S.: Yeah, let's take some
questions if you have any.
A.K.: Cool.
P.S.: Cool, thanks.
A.K.: Thanks.
P.S.: Thanks guys.