English subtitles

← Random Testing Loop - Software Testing

Get Embed Code
1 Language

Showing Revision 3 created 05/25/2016 by Udacity Robot.

  1. Just go back to the diagram--what we just saw was exactly an example of this general
  2. outline that I showed you, so CCNet was the random test case generator.
  3. It was given seeds by a driver's script--the driver's script run the tools in a loop
  4. and the oracle in this case was just simply looking for compiler crashes,
  5. so it wasn't even running the compiler output.
  6. Of course, when the compiler runs, it generates code.
  7. We weren't even running that code or even looking at it--all we we're doing is waiting for the compiler
  8. to tell us that it fail, and the reason the compiler could tell us that it failed is because the
  9. compiler developers have conveniently included lots of assertion violations.
  10. If those compiler developers hadn't included all of those assertions, then these compilers
  11. wouldn't have failed in such an obvious way, but they still had encountered bugs and they
  12. probably would've failed in different ways later and it might have failed in the worse possible way
  13. which is where the compiler generates code and that code actually executes improperly at runtime.
  14. And that's bad because we, as compiler consumers,
  15. don't want the compiler giving us wrong code.
  16. So while those assertions can be annoying as compiler uses because the compiler crashes
  17. actually very good and we wanted them to be there.
  18. So the driver's script are then checking the output, taking the test cases wherein this case
  19. the seed in the log file and then go back and do it again.
  20. And so if you remember, we got about 2 days of testing time, this loop would've executed about
  21. 150,000 times on a fast day core machine, and of course, we we're testing
  22. are simpler systems and compilers.
  23. We might have executed a much larger number of test than that and if we're executing the
  24. very slowest software under test, then we might've only execute a much smaller number of iterations