YouTube

Got a YouTube account?

New: enable viewer-created translations and captions on your YouTube channel!

English subtitles

← Reporting Bugs - Software Testing

Get Embed Code
1 Language

Showing Revision 3 created 05/25/2016 by Udacity Robot.

  1. But now I would like to talk a little bit about the art of reporting bugs.
  2. So let's assume that you're successful in creating some sort of really nice
  3. random tester that starts crashing utilities.
  4. Let's say for example that you want to do your own fuzzing study.
  5. So if you recall a fuzzing paper was written in 1990, one was written in 1995,
  6. then another in 2000, another in 2006.
  7. So now it's 2012 and is basically time for a new one.
  8. You do it and so what you do is go take whatever operating system you choose,
  9. and you start crashing software and hopefully
  10. instead of just being happy about what a successful effort you've had,
  11. you'll report the bugs to the maintainers so they can start
  12. improving the robustness of the software. Let's talk about how to do that.
  13. The first rule and it's really simple is don't report a duplicate bug.
  14. To avoid reporting duplicate bugs is to go to a bug reporting systems that most open source projects
  15. maintain and search on some symptom of the bug that you've found and see if it's been reported.
  16. Let's say we've found a way to get a segmentation fault
  17. out of all the m and we'll see if anybody else has that way.
  18. Well okay, so we only have one reported here and so it's really nice.
  19. What we would do then if we want to report a new segmentation fault to an LVM is
  20. we'll go ahead and look at this and see if the new one that we're going to report
  21. seems to be sufficiently different and if it is, we'll probably go ahead and report it.
  22. The next golden rule for reporting bugs is to respect whatever the local conventions are
  23. of the community that you're reporting to and so the best thing to do here is just look for some bugs
  24. that have been reported and see what kind of discussions they generate and try to take sort of a
  25. similar tone to what people have already done.
  26. The third thing is to report a small test case. I'm going to say here a small stand alone test case.
  27. The test case manipulation stuff that we have talked about a little bit earlier is really important
  28. for purposes of bug reporting and that kind of a process should always be a part of
  29. your bug reporting and the second cause here a stand alone test case is also really important.
  30. This means for example, don't report a bug that depends on some other file
  31. that only exists in your machine because that's not very useful to the people who are going to try to
  32. reproduce the bug because of course they don't have the other file.
  33. Do what you can to make a bug report stand alone.
  34. The next rule is when we report valid test cases and so this referring back
  35. exactly to the input validity problem that we've been talking about
  36. that I'd talked about a number of times in the material in random testing and if the software
  37. to which your reporting bugs isn't supposed to programmatically reject invalid inputs
  38. then you shouldn't report any valid inputs and so an example of this system is for example
  39. a C compiler isn't suppose to reject invalid inputs and another example might be something like
  40. internal communication between the JavaScript engine and a web browser
  41. there may not be total input checking on some sort of an internal interface like that,
  42. and so basically, we just want to be sensitive to the fact that some systems can't do
  43. or aren't equipped to do full validity check of their input and for those systems
  44. we often need to make sure the input is valid ourselves by hand.
  45. Now of course other systems and we've talked about things like Acrobat reader or whatever.
  46. These are suppose to reject invalid input and they are not suppose to crash
  47. in some sort a buffer overflow even on invalid inputs.
  48. For those kind of programs, this issue doesn't really exist.
  49. The next thing you need to do is tell the people who are the recipients for your bug report
  50. what the expected output of the system was that is what you hope would happen
  51. if the system wasn't buggy and what the actual output was.
  52. That is to say what really happen on your system and so the temptation is to not supply these things
  53. because you think it was insulting or you think that's really obvious, but in general
  54. it's a good idea to err on this side of caution and include these things.
  55. So you might include a bug report to the Python maintainers saying that when I added 1 + 1,
  56. I was really expected to but then this implementation,
  57. this particular version of the implementation printed 3.
  58. The next thing you need is to make the failure reproducible because nothing will get a bug report
  59. ignored faster than the incompletely irreducible and so you need to include things like platform details.
  60. For example this probably crucial information that your running Firefox on windows
  61. as oppose to Linux if you are trying to report some sort of Firefox crash.
  62. The exact version of the software under test is important
  63. and of course, if this is some sort of an open source project where you've hacked yourself
  64. before compiling you will definitely need to include details about that in the bug report.
  65. But if this little bug is your fault, then you don't want to waste people's time trying to track in down.
  66. So in summary, if we're able to come up with good solid answers to all of these requirements,
  67. that's a great idea to report a bug and so let's go ahead and do that right now.