Got a YouTube account?

New: enable viewer-created translations and captions on your YouTube channel!

English subtitles

← Complaints About Random Testing - Software Testing

Get Embed Code
1 Language

Showing Revision 3 created 05/25/2016 by Udacity Robot.

  1. So now that you've seen how easy it is to build a random tester,
  2. I hope that you build many of them in the future and most of the rest of this course is going to be spent
  3. going into more details about how we build effective random testers.
  4. But before we do that, what I'd like to do is address an issue that eventually confronts
  5. almost everybody who works on random testing, which is that random testing gets no respect.
  6. And what I mean by that is people often think that it's a really stupid way to test software.
  7. For example, one of the classic references on software testing
  8. is a book called The Art of Software Testing.
  9. In this book, the author talks about random testing and here's what he has to say.
  10. "Probably the poorest methodology of all is random random input testing...
  11. What we look for is a set of thought processes that allow one to select
  12. a set of test data more intelligently."
  13. So basically what Myers is saying is random input testing is not a way that allows one
  14. to select a set of test data intelligently.
  15. And so my opinion and this is after writing at least a dozen testers
  16. what he should've said is something like what we look for in testing is a set of thought processes
  17. that allow one to randomly generate a set of test data more intelligently.
  18. And so what we can do is get around this fact that
  19. testing requires thought about the software under test.
  20. But my experience and the experience of a lot of other people is that if we think hard
  21. about a random test case generator that's as good or better in many cases
  22. than thinking hard about generating the test cases themselves.
  23. So another classic piece of work is a book chapter called Random Testing by Richard Hamlet.
  24. And in this chapter he says, "Most criticism of random testing is really objection
  25. to misapplication of the method using inappropriate input distribution."
  26. And so what Hamlet is really talking about here is the input validity problem
  27. that we've just been discussing and what he means is that if you ignore the input validity problem
  28. and you just test randomly using completely random garbage, you're going to get a bad
  29. impression of the method because it's not going to work very well.
  30. You're going to experience a phenomenon that we talked about a little bit ago
  31. where all of the test cases get rejected by very early parts of the software under test.
  32. And so I would actually say what Hamlet said a little bit differently.
  33. Most criticism in random testing is really objection to random testing done badly.
  34. So let's take a look at one more quote.
  35. So what the author of Testing for Zero Bugs has to say is, "The introduction of random testing
  36. practically eliminated user bug reports on released back ends.
  37. To our amazement, RIG (and this is their test case generator) was able to find over half of all bugs
  38. reported by customers on the code generator in just one night of run time."
  39. And so this clearly was written by somebody who did a really good job creating a random tester.
  40. And also probably applied in the domain where random testing happened to work really well.
  41. The rest of this Testing for Zero Bugs is worth reading as well,
  42. and we will include a link to that in the supplemental material for the course.