English subtitles

← Difficulties With Bug Triage - Software Testing

Get Embed Code
1 Language

Showing Revision 3 created 05/25/2016 by Udacity Robot.

  1. Unfortunately, not all bugs resolve to nice assertion violation messages,
  2. and bug disambiguation can be trickier when all we have
  3. is something like a core dump of a stack trace.
  4. What these things are going to give is--at least if we have good debugging symbols--
  5. probably some sort of an indication of what part of the code failed,
  6. and then it's going to give an indication of the stack frames leading up to the failure.
  7. Sometimes these are going to be useful for doing disambiguation and sometimes not.
  8. They provide qualitatively at some level similar information to assertion violation messages,
  9. and if they're giving us some clues relating the symptom to the source code
  10. but is not going to be as easy to find and it's not going to be as readable.
  11. A third weapon we have in the arsenal when doing bug triage
  12. is actually pretty interesting,
  13. which is that we can do a search over the revision history of the software under test--
  14. that is to say, of course, if we have access to the revision control symptom we can do this.
  15. Let's go back to our example here where we have all these bugs.
  16. If it's the case that all of these bugs here appear just one revision back,
  17. and these other ones appeared in different revisions maybe weeks or months ago
  18. then it's a good bet that all of these new inputs are triggering the same error--
  19. that is to say, an error that somebody just committed the other day--
  20. and these probably trigger older bugs.
  21. This is a pretty powerful weapon and some automated tools for revision control
  22. even build in functionality to do this search over version history.
  23. For example, git has a git_bisect command
  24. that can be used to do a binary search over revision history in an automated way.
  25. Even if that kind of thing isn't available it's not too hard to script this up yourself
  26. given an automated way to detect if a particular version has a particular bug.
  27. Our last way we can triage bugs
  28. is to examine a test case.
  29. Often it's the case that test cases that trigger the same bugs have similar features.
  30. In the compiler bug finding example that I've been using
  31. it's often the case that I'll look at a bunch of failure-triggering programs--
  32. that is to say, inputs to the compiler that cause it to crash or otherwise fail--
  33. and contains some sort of a smoking gun like maybe 64-bit integer divides or something like that.
  34. There would be some sort of common feature that tips us off
  35. that some collection of test inputs all maps to the same bug.
  36. Then if we see another group of test inputs
  37. that contain some totally different feature-
  38. maybe really deeply nested parentheses or really long identifiers--
  39. then it's a good bet that those map to a different bug.
  40. The problem is looking over large, randomly-generated test cases is really painful.
  41. This is going to directly lead to our next topic,
  42. which is a really important one, which is test-case reduction or test-case minimization.
  43. That's an automated process for taking a large failure-inducing input
  44. and turning it into a small one, and we're going to cover that next.