Got a YouTube account?

New: enable viewer-created translations and captions on your YouTube channel!

English subtitles

← How Random Testing Should Work - Software Testing

Get Embed Code
1 Language

Showing Revision 3 created 05/25/2016 by Udacity Robot.

  1. The first reason, I think, is that it's because the developers of the commercial software systems
  2. aren't doing it or at least not doing it enough and I think it's pretty clear that the kind of bugs found,
  3. for example, in the fuzzing papers, or in Charlie Miler's talk would'nt have been there in the software
  4. if the developers of Adobe Acrobat or the Unix utilities have
  5. had a reasonably aggressive random testing program, and remember that
  6. some of those bugs Charlie Norris was talking about were secured liabilities.
  7. These are things that they really don't want in their software, and so what I'd argue--
  8. Software development efforts that don't make proper use of random testing are flawed,
  9. and the reason that these efforts are flawed is because
  10. modern software systems are so large and so complicated
  11. that test cases produced by nonrandom processes are simply unlikely to find
  12. the kind of bugs that are lurking in these software systems.
  13. What that leaves us with is a question: what should they have done?
  14. How is random testing supposed to work? Let me give some ideas about that.
  15. What I'm going to do here is show sort of a rough software development timeline
  16. with a releasing software over here and early development stages over here.
  17. What we've looked at mainly so far in this course is random unit testing.
  18. We're developing these software units and what we're trying to do is make sure that they are robust enough.
  19. Then we start composing them together later, they'll be a solid foundation.
  20. We looked at several cases of, for example, with a queue--bounded queue here.
  21. We looked at fuzzing the interfaces that it provides.
  22. We also looked at an example of random fault injection and that was for the read_all function,
  23. and if you remember, that was the function that was supposed
  24. to cope with the fact that the Unix read system call can display partial success.
  25. They were doing fault injections that were fuzzing the interface used
  26. by the read_all call not the interface that it provides.
  27. And so, like I said, what we want to do is ensure as we're developing the modules
  28. that we're creating robust pieces of software whose interfaces we understand
  29. and that are going to be solid foundation for future work,
  30. so I would start developing more elaborate software stacks.
  31. It's going to be the case that some of our random testers become useless.
  32. For example, if we have a Q instantiated here that is used by some
  33. more sophisticated piece of software, we no longer are interested in the ability to randomly test
  34. the interface provided by the Q because it's simply being used by the rest of the software.
  35. On the other hand, other kinds of random testers such as those that come in at the top level
  36. and those that perform system-level fault injection are absolutely still useful.
  37. In fact, fault injection of things like erroneous responses to system calls
  38. are really important things to test larger pieces of software with
  39. because typically, those kind of errors can result in failures propagating
  40. all the way through our software stack and we'd really like to understand
  41. how our system responds to that sort of thing.
  42. To be part of something that's more of a complete product
  43. our focus should be on external interfaces provided, so this is going to be things like
  44. file I/O and the graphical user interface, and so, if you recall those fuzzing papers,
  45. we're fuzzing exactly these sorts of things.
  46. Here, we're delivering random bits to the file interface and they were delivering
  47. random gooey events to applications and knocking over
  48. a pretty large proportion of applications that they tested.