English subtitles

← 03-02 How Much Testing Is Enough

03-02 How Much Testing Is Enough

Get Embed Code
1 Language

Showing Revision 2 created 07/16/2014 by Udacity Robot.

  1. One of the trickiest things about testing software is that it's hard to know when you've done
  2. enough testing and the fact is, it is really easy to spend a lot of time testing
  3. and to start to believe that you did a good job and then to have some really nasty bug show up
  4. that are triggered by parts of the input space that we just didn't think to test.
  5. So let's look what's really going on, so we're going to start with the
  6. domain range diagrams that we known off.
  7. We're going to have some test cases into domain and after a while
  8. we're going to start pretty good about what we're been doing.
  9. What's going to turn out is without really knowing it, our testing is being compliant to some small part
  10. of the input domain and the problem, is that even the small part of the domain may contain
  11. an infinite number of test cases or else a finite numbers of test cases
  12. that is large enough that its for practical purposes.
  13. It's not distinguishable from infinity and of course what's going on is we're going to do test cases
  14. other parts of the domain we didn't think to test, putting the results and outputs that are not okay.
  15. It really depends on how we've broken up the input domain.
  16. For example, let's think about the case that we're testing Therac-25 radiation
  17. therapy machine that I used an example in the last lecture.
  18. It's not might be the case but all of the inputs that we are going to test, the once in this region
  19. are the once where we happened to type the input to the machines slowly.
  20. We simply didn't realize that there's a whole other part of the input space
  21. that has to be treated differently that happens when we type the input fast and of course
  22. its in that region where we triggered the raised conditions that we're leading to massive overdoses.
  23. Similarly, if you remember from lecture one,
  24. we have a small Python program called the Python runtime to crash.
  25. The distinguishing feature of it seem to be a large number of cascaded if statements.
  26. It's pretty easy if we're testing Python programs to remain in the part of this space where for example
  27. we have <5 nested in these statements.
  28. Over here is another region containing programs with 5 of more if statement nested and these are ones
  29. that cause for example, the Python virtual machine to crash.
  30. To take an even more extreme example, let's say that we're testing some software
  31. that somebody has inserted in the back door in.
  32. Well in that case, there's going to be an absolute infinite test of all part of the input domain.
  33. Maybe wherever here that triggers the back door cause remember if your putting a backdoor in code
  34. you don't want to trigger it accidentally and that's going to lead to something
  35. extremely bad happening over here.
  36. We didn't tested inputs triggering the back door because we just didn't know it was there.
  37. So what we'd really like is some sort of a tool or some sort of a methodology
  38. that if we are in fact testing only a small part of the input domain for a system what we really like
  39. is some sort of an automated scoring system that looks at our testing effort and says to us
  40. something like, your score is 14 out of 100.
  41. Your not doing a good job testing the system. Keep trying.
  42. And that's what today's lecture is going to be about.
  43. It turns out there's a lot of reasons we do not want to know the sign of score to our testing effort.
  44. Probably the main one is that this helps us find part of the input domain that need more testing.
  45. So for example, if we can increase our score by testing this part of the domain,
  46. we're naturally going to be led to do that and our testing efforts will improve.
  47. Other reasons to assign a score to a testing effort are that we might be able to argue to a boss
  48. which is some sort of a certifying authority that we've done enough testing.
  49. Similarly, we might want to argue that the system we're developing has not been tested well enough
  50. but its not yet safe to deploy and that it needs more testing and a testing score can provide
  51. a numerical justification of that sort of way.
  52. Probably, it would be nicer if we could take a large test suite one maybe that takes several days to run
  53. and identify parts of the test suite that are completely redundant.
  54. That is to say parts of the test suite that tests exactly the same parts of the input domain.
  55. Even though they occupy parts of the input domain have roughly the same testing effect on the system
  56. to assigning a score to our testing efforts and let us do that as well.