How to Stop Hating Your Tests

Alright, high energy I love it

The doors are closing
and now we're covered

Alright, great

Can I get my slides up on the monitors?

Alright, great

Let me start my timer

Where's my phone? Uh oh

Who has my phone? Where's my timer?

Alright we'll start

Alright.

So, there's this funny thing where
every year conference season lines up with

Apple's operating system release schedule
and I'm a big Apple fanboy

So I, on one hand really want to upgrade,

and on the other hand,
really want my slide deck to work

So, this year, because they announced
the iPad Pro I was pretty excited,

I was like, maybe this year finally OS9
is going to be ready for me

to give a real talk out of
and to build my entire talk out of it

So this talk was built entirely in OS9

So let's just start it up, see how it goes

I'm a little nervous

Alright, so, it's a little retro

It takes a while to start up

I built my entire presentation in AppleWorks

So I gotta open up my AppleWorks presentation

Ok, there it is

I gotta find the play button

And here we go

And good, alright

So this talk is,
"How to stop hating your tests"

My name is Justin

I play a guy named "searls" on the Internet

And I work at the best software
agency in the world: Test Double

So, why do people hate their tests?

Well, I think a lot of teams start off
in experimentation mode

Like everything's fun, free,
they're "pivoting" all the time

And having a big test suite would really
just slow down their rate of change

and discovery

But eventually we get to a point where
we're worried that if we

create a new change we might break things

and it's important that
things stay working

so people start writing some test suites

so they have a build so that
when they push new code

they know whether they just broke stuff

But if we write out tests in a haphazard,
unorganized way

they tend to be slow, convoluted and every
time that you want to change a thing

we spend all day just updating tests

and eventually teams get to this point

where they just yearn for
the good old days

where they got to change stuff
and move quickly

And I see this pattern repeat so much
that I'm starting to believe

that an ounce of prevention
is worth a pound of cure in this instance

'Cause once you get to the end
there's not much you can do

You can say,
"our test approach isn't working"

And a lot of people would be like

"Well I guess we're just
not testing hard enough"

And when you see a problem
over and over again

I really personally,
I don't believe that the

"Work harder comrade"
approach is appropriate

You should be always inspecting your
workflow and your tools

and trying to make them better
if you keep running into the same issue

Some other people they might say,
"Ok well let's just buckle down,

"remediate, testing is job one,
let's really focus on testing for a while"

But from the perspective of the people
who pay us to build stuff

testing is not job one

It's at best job two

From their perspective they want to
see us shipping stuff

Shipping new features

And the longer we go with that
impedance mismatch

the more friction and tension
we're going to have

So that's not sustainable

I said we're talking about prevention

but if you're working in a big,
legacy, monolithic, application, you know

and you're not greenfield

this is not a problem at all 'cause
I've got this cool thing to show you

There's this one weird trick
to starting fresh with your test suite

That's right, you're gonna learn_
what the one weird trick is

Basically just move your tests
into a new directory

aAnd then you make another directory

Aand then you have two directories

Aand you can write this thing
called a shell script

get this

that runs both test suites

And then, you know, eventually
port them over

and you're able to decommission the
old test suite

But, I would hesitate to even
give a talk about testing

because I am the worst kind
of expert

I have too much experience,
navel-gazing about testing

building open-sourcing tools
around testing

I've been on many many teams

as the guy who cared just a little
bit more about testing that everyone else

and lots of high falutin, philosophical
and nuanced Twitter arguments

that really are not pertinent
to anyone's life

So my advice is toxic

I'm overly cynical, I'm very risk averse

and if told you what I really thought
about testing

it would just discourage all of you

So, instead, my goal here today is to
distill my advice down into

just a few component parts

The first part, we're going to talk
about structure,

the physicality of our tests

like what the lines
and files look like on disk

We're going to talk about isolation
'cause I really believe that

how we choose to isolate
the code that we're testing

is the best way to
communicate the concept

and the value that we hope
to get out of a test.

And we're gonna talk about feedback like,

do our tests make us happy or sad?

Are they fast or are they slow?

Do they make us more or less productive?

And keep in mind we're thinking about
this from the perspective of prevention

Because these are all things that are much
easier to do on day one

than to try to shoehorn in on day 100.

So at this point,
in keeping with the Apple theme,

my brother dug up this Apple II
copy of Family Feud

and it turns out it's really hard
to make custom artwork in AppleWorks 6

so I just ripped off the artwork
from this Family Feud board

We're going to use that to organize our slides

It's a working board, that means if
I point at the screen and say

"Show me potato salad!"

I get an X

But unfortunately I didn't have
100 people to survey

I just surveyed myself 100 times
so I know all the answers already

So, uh, first round, we're going to
talk about test structure

And I'm going to say,
"Show me 'Too Big To Fail'"

People hate tests of big code

In fact, have you ever noticed that
the people who are really into testing

and TDD, they really seem to hate
big objects and big functions

More than normal people

I mean we all understand big objects
are harder to deal with than small objects

But one thing I've learned over the years

is that tests actually make big objects
even harder to manage

which is counterintuitive,
you'd expect the opposite

And I think part of the reason is that
when you've got big objects

they might have many dependencies

which means you have lots of test setup

They might have multiple side effects
in addition to whatever they return

which means you have lots of verifications

But what's most interesting is they have
lots of logical branches

Like depending on the arguments
and the state

there's a lot of test cases that you
have to write

And this is the one that I think
is most significant

So, uh, let's take a look at some code

At this point I realize that OS9 is not unix

So, I found a new terminal

It's actually a cool new one

It just came out this week

So let's boot that up

Yep, here we go

Alright, we're almost there

It's a little slow

Alright, so, this is a
fully operational terminal

Alright, so we're gonna type like
an arbitrary Unix command

That works fine

I'm gonna start a new test

It's a validation method
of a timesheet object

To see whether or not people
have notes entered

And so, we're gonna say like

if you have notes, and you're an admin

and it's an invoice week or an off week

and whether or not you've
entered time or not

All of those four boolean attributes

they factor into whether or not

that record is considered valid

And at this point I wrote the first test

but I'm like, "Uh, I've got
a lot of other context to write"

Let's start planning those out

And like "Damn, this is a lot of tests
that I would need to write"

"to cover this case of just four booleans"

And what I fell victim to there

is a thing called the Rule of Product

which is a thing from the
school of combinatorics and math

It's a real math thing
because it has a wikipedia page

And what is says essentially is that

if you've got a method with four arguments

you need to take each of those arguments

and the number of possible values
of each of them

multiply them together

and that gives you the total
number of potential combinations

or the total number of upper bound

of test cases you might need to write

So in this case with all booleans

it's 2^4, so we have 16 test cases

that we may have to write in this case

And if you're a team that used to

writing a lot of big objects,
big functions

you're probably in the habit of thinking

"Oh I have some new functionality,

"I'll just add one little more argument

"what more harm could that do?"

Other than double the number of
test cases that I have to write!

And so, as a result as somebody

who trains people on testing a lot

I'm not surprised at all to see

like a lot of team who are used
to big objects

want to get serious about testing

and then they're like,

"Wow, this is really hard, I quit"

So if you want to get serious about
testing and have a lot of tests of code

I encourage you, stop the bleeding.

Don't keep adding on to your big objects.

I try to limit objects to 1 public method

and at most 3 dependencies

which, to that particular audience
is shocking

The first thing they'll say is

But then we'll have too many small things!

How will we possibly deal with all the
organized and carefully named

comprehensible small things?

And, you know, people get off on their
own complexity right?

So that's what makes them feel like a
serious software developer.

That's how hard their job is.

They're like, "that sounds like
programming on easy mode"

and I'm like, "It is easy!"

It's actually like, you know, not

rocket science to build an enterprise
crud application but

you're making it that way.

Just, write small stuff. It works.

So, Next I want to talk about about

how we hate when our tests go off script

Code can do anything.

Our programs should be
unique and creative

special unicorns of awesomeness,

but tests can and should
do only three things.

They all follow the same script.

Every test ever sets stuff up,

invokes a thing,

and then verifies a behavior.

We're writing the same things
over and over again.

And it has these three phases:
Arrange, Act, and Assert.

A more English-natural way to say that
would be: Given, When, Then.

And when I'm writing a test I always

intentionally call out all those phases
really clearly and consistently.

For example if I'm writing this is a
MiniRest method,

I always put exactly two empty new lines

in every single x-unit style
tests that I write.

One after my Arrange,
one after my action.

Then it's really clear at a glance

what's my arrange, what's my act,
what's my assert.

I always make sure to go in the
correct order as well

which is something
people get wrong a lot.

If I'm using something like RSpec

and I have a lot of constructs available

to me to specify what the intent is

so I can say like 'let' and give a value

to do a set up, so 'let' says
I'm setting up a new thing.

I can use 'before' to call out like

this is an action with a side effect
this is my act

and that allows me to split up
if I so choose

those assertions into two separate blocks.

So now at a glance
somebody that knows RSpec

they'll know exactly what phase
each of those lines belongs in.

I also try to minimize each phase
to 1 action per line.

so the test scope logic doesn't sneak in.

The late great Jim Weirich wrote an
awesome Ruby gem

I hope you check it out

it's called RSpec Given.

I help maintain it now.

He and Mike Moore ported it to
MiniTest as well.

I ported it a few years ago to Jasmine.

Somebody else has taken it on
and ported it Mocha.

It's a really cool given-when-then
conscious testing API.

And what it does is

you start from the same place as RSpec
as you may have before,

and we'll just say
'given' instead of 'let'

because that's more straightforward.

'when' instead of 'before'
so it's clear.

But what really shines is that you'll see

'then' is just just a little one-liner

and I don't have a custom assertions API

because it's actually interpreting
the Ruby inside of that

and able to split it up to give you
great error messages.

So it's a really really terse and yet
successfully expressive testing API.

Now you don't have to
use that tool though

to just write your test in a way
that is conscious of given-when-then.

They're easier to read regardless.

They point out superfluous bits of
test code that don't

fit one of those test phases.

And they can highlight certain
design smells for instance-

if you've got a lot of given steps

maybe you have too many
dependencies on your subject

or too complex of arguments.

If it takes more than one 'when' steps,

then its probably the case where
your API is confusing or hard to evoke.

There's something awkward in
how you use that object.

And if you got too many of "then" steps

then your code is probably doing too much.

Or it's returning too complex of a type.

Next step I want to talk about
hard-to-read, hard-to-skim code.

Some people are fond to saying
'test code is code'

but test code is untested code.

So I try to minimize it.

I try to make it as boring as possible
for that reason.

Because what I find is that

a good test tells me a story of
what the code undertest should look like.

But if there's logic in the test
it confuses that story

and I'm spending most of
my time reading that logic

and making sure I got it right.

Because I know there there's
no test of that test.

So test-scoped logic is
not only hard to read,

but if there are any errors
they're very easy to miss.

Maybe it's passing green
for fantasy reasons.

Maybe only the last item
in this loop of data

is actually executing
over and over again.

A lot of times people have this impulse.

They say

"Hey I've got a lot of
"redundancy in my test."

" I could really dry this up by"

"just generating all of my test cases".

You know, for example,

this person did a
roman numeral arcata.

And they can see really clearly

"On I could just
have a data structure"

"and make it really much more terse"

looping over that data structure

and then generating
using defined method

a new test method that would
like, give a good message

and it's a perfectly reasonable test

and in this case it totally works fine.

But, I still think its problematic.

And the reason is,
that person experienced test pain

and their reaction was to go
and make the test cleaner.

Usually when we experience test pain
the first thing I look at is

Maybe there's something wrong with

my production code that lead me there.

And so, if you look at
that person's production code

you can see all that data is hiding
ifs and elses.

They've got all this dense logic in there.

I would much rather take
a look at the same thing

and extract the same sort of
data structure from that,

so that I could then,
instead of having all that if and else

I'm looping over the same data structure

and figuring out whatever rule I have to.

So now I only need a few test cases

in fact, i can just keep adding
additional keys to that hash.

And now I've covered
a lot of test cases

without needing a whole bunch of
really explicit test cases.

It's much cleaner this way.

Sandi Metz, who's around.

Is she here?

Sandi, where are you at?

Hey Sandi!

So she's got a thing

called the Squint Test.

It helps her understand and cope
with really big file listings.

And she can draw a few conclusions.

I don't have anything nearly fancy,

but when I'm reading your test suite
I really hope that I'm able to

at a glance understand what's
the thing under test.

Specifically, where are all the methods?

Are they in order,
and are they symmetrical?

Is it easy for me to find
all the tests of one method?

And I like to use,
if I'm using Rspec for example,

I like to use context
to point out every logical branch

and all the subordinate behavior
underneath each logical branch.

It's very easy to organize this way.

And when you do it consistently,
its easy to read tests.

Additionally, as I said,

Arrange, Act and Search
should really pop in a consistent way.

Now if I'm using a x-unit style
testing tool like MiniTest,

at least I like to see
Arrange, Act and Assert

straightforwardly throughout
every single file listing

and the names of the tests
should mean something.

Alright, next step.

Lets talk about tests that are too magic.

A lot of people hate tests
that are too magic.

Or - not magic enough.

As it turns out because all software
is, right?, a balancing act,

and test libraries are no different.

Expressiveness of our testing APIs
exist along a long spectrum.

You know, smaller APIs
generally are

slightly less expressive
than things that have a larger API

because they have more features.

But you have to learn those features.

So if you look at something like MiniTest

it's very cool because
it's classes and methods.

We know that.

So every test is a class

we overwrite 'setup' and 'teardown'
to override that behavior.

Every new test is another method.

Assert is very easy to use.

Ryan's a funny guy so he's got
some fun ones like

i_suck_and _my _tests_are_order_dependent

to get some custom behavior.

But when you compare that to RSpec
it's night and bay.

RSpec has 'describe' and 'context'
and they're are synonyms.

'subject' and 'let' and they're similar.

'before' 'after' and 'around'

and 'each' 'suite' and 'all'
for each of those.

You've got 'it', you've got 'specify'
which are similar.

You've got 'object.should have_'
and all those matches.

You've got 'expect().to be_'
and the mostly similar matchers.

You've got shared_examples groups

tagging, advanced CLI groups.

There's a lot to learn in RSpec.

Jim tried to write both ways
when he designed *-given.

He wanted a terse API:
'Given' 'When' 'Then'

with just a handful of other
things he came across:

'And' 'Invariant' and his
'natural assertion' API.

So it's very very terse
but it's also

sufficiently expressive
for most people's tests.

Now, because its not a
standalone testing library,

you're still standing on top of
all of MiniTest or Rspec.

So it is still physically complicated.

But it's still nice to live from
a day to day basis.

I'm not here to say there's some
right or wrong testing library

or level of expressiveness.

You just have to keep yourself aware
of the trade-offs.

Smaller testing APIs:

they're easier to learn but
they might encourage

more one-off test helpers
that we write

and that, you carry that complexity.

Whereas, a bigger testing API
something like RSpec

might help you yield
really terse tests

but to an uninitiated person

they're just going to
look like magic

and you'll have to eat that
onboarding cost

if someone doesn't know RSpec.

Finally this category -
people hate tests that are

accidentally creative.

Because, in testing
consistency is golden.

If we look at a similar test
as what we had before

we're going to use 'let' to set up
an author, a blog, and a comment

but it's not clear at all
what the test is.

So I'm going to rename it 'subject'

I always call the thing
under test 'subject'

and I always call the thing I get back
from the thing I'm gonna assert on

I always call that 'result' or 'result'

100% of the time.

So if I'm reading a
really big nasty test

at least i know what's being tested
and what's being asserted on.

This is a surprisingly daunting task
in a lot of people's test suites.

so if learn one thing today

and you start calling the
thing you're testing 'subject'

this would have been worth
all of this preparation.

And when you're consistent

inconsistency can actually
carry nuance meaning.

For example if I've got a
handful of tests here

I'm gonna look at em
and be like

Oh, wait!
There's something weird about Test C

That implies there must be something
interesting about object C

I should look into that.

That's really useful.
That really speeds me up.

But when every test is inconsistent
when every test looks way different

I'll have to bring that same level of
screw needed to each and every test

and I have to read very carefully
to understand what's going on -

understand the story of the test.

So, as a result
if I'm adopting your test suite,

I would much rather see
hundreds of consistent tests

even if they're mediocre
even if they're crappy

then even a handful of
beautifully crafted brilliant

custom tests that are way different.

Because, every time I fix anything
it's just a one-off thing.

Also, readers are silly, right?

They've got this funny habit of
assuming all of code has meaning

but especially in testing
very often the stuff we put in our tests

is just pluming to
make our code execute properly.

So i try to point out meaningless stuff
to help my reader out.

In particular,
I make unimportant test code

obviously silly and meaningless
to the reader.

In this instance I'm setting up
a new author object

and he's got a fancy name,
phone number, and an email

and they're all validatable,

but it's not necessary for this method.

So here, I'll just
change his name to 'pants'

and I'll remove his phone number
because it's not necessary

and I'll just change his email
to 'pantsmail' and update my assertion

and now, everyone in the room,

before, you might have assumed you
needed a real author but you didn't

now everyone in the room can
implement this method

understanding exactly what it
needs to really do.