-
Title:
Reporting Bugs - Software Testing
-
Description:
-
But now I would like to talk a little bit about the art of reporting bugs.
-
So let's assume that you're successful in creating some sort of really nice
-
random tester that starts crashing utilities.
-
Let's say for example that you want to do your own fuzzing study.
-
So if you recall a fuzzing paper was written in 1990, one was written in 1995,
-
then another in 2000, another in 2006.
-
So now it's 2012 and is basically time for a new one.
-
You do it and so what you do is go take whatever operating system you choose,
-
and you start crashing software and hopefully
-
instead of just being happy about what a successful effort you've had,
-
you'll report the bugs to the maintainers so they can start
-
improving the robustness of the software. Let's talk about how to do that.
-
The first rule and it's really simple is don't report a duplicate bug.
-
To avoid reporting duplicate bugs is to go to a bug reporting systems that most open source projects
-
maintain and search on some symptom of the bug that you've found and see if it's been reported.
-
Let's say we've found a way to get a segmentation fault
-
out of all the m and we'll see if anybody else has that way.
-
Well okay, so we only have one reported here and so it's really nice.
-
What we would do then if we want to report a new segmentation fault to an LVM is
-
we'll go ahead and look at this and see if the new one that we're going to report
-
seems to be sufficiently different and if it is, we'll probably go ahead and report it.
-
The next golden rule for reporting bugs is to respect whatever the local conventions are
-
of the community that you're reporting to and so the best thing to do here is just look for some bugs
-
that have been reported and see what kind of discussions they generate and try to take sort of a
-
similar tone to what people have already done.
-
The third thing is to report a small test case. I'm going to say here a small stand alone test case.
-
The test case manipulation stuff that we have talked about a little bit earlier is really important
-
for purposes of bug reporting and that kind of a process should always be a part of
-
your bug reporting and the second cause here a stand alone test case is also really important.
-
This means for example, don't report a bug that depends on some other file
-
that only exists in your machine because that's not very useful to the people who are going to try to
-
reproduce the bug because of course they don't have the other file.
-
Do what you can to make a bug report stand alone.
-
The next rule is when we report valid test cases and so this referring back
-
exactly to the input validity problem that we've been talking about
-
that I'd talked about a number of times in the material in random testing and if the software
-
to which your reporting bugs isn't supposed to programmatically reject invalid inputs
-
then you shouldn't report any valid inputs and so an example of this system is for example
-
a C compiler isn't suppose to reject invalid inputs and another example might be something like
-
internal communication between the JavaScript engine and a web browser
-
there may not be total input checking on some sort of an internal interface like that,
-
and so basically, we just want to be sensitive to the fact that some systems can't do
-
or aren't equipped to do full validity check of their input and for those systems
-
we often need to make sure the input is valid ourselves by hand.
-
Now of course other systems and we've talked about things like Acrobat reader or whatever.
-
These are suppose to reject invalid input and they are not suppose to crash
-
in some sort a buffer overflow even on invalid inputs.
-
For those kind of programs, this issue doesn't really exist.
-
The next thing you need to do is tell the people who are the recipients for your bug report
-
what the expected output of the system was that is what you hope would happen
-
if the system wasn't buggy and what the actual output was.
-
That is to say what really happen on your system and so the temptation is to not supply these things
-
because you think it was insulting or you think that's really obvious, but in general
-
it's a good idea to err on this side of caution and include these things.
-
So you might include a bug report to the Python maintainers saying that when I added 1 + 1,
-
I was really expected to but then this implementation,
-
this particular version of the implementation printed 3.
-
The next thing you need is to make the failure reproducible because nothing will get a bug report
-
ignored faster than the incompletely irreducible and so you need to include things like platform details.
-
For example this probably crucial information that your running Firefox on windows
-
as oppose to Linux if you are trying to report some sort of Firefox crash.
-
The exact version of the software under test is important
-
and of course, if this is some sort of an open source project where you've hacked yourself
-
before compiling you will definitely need to include details about that in the bug report.
-
But if this little bug is your fault, then you don't want to waste people's time trying to track in down.
-
So in summary, if we're able to come up with good solid answers to all of these requirements,
-
that's a great idea to report a bug and so let's go ahead and do that right now.
