English subtitles

← Building A Test Suite - Software Testing

Get Embed Code
1 Language

Showing Revision 3 created 05/24/2016 by Udacity Robot.

  1. Alright. So I'd like to talk just really briefly about building a test
  2. suite for a piece of software. So test suite is just a collection
  3. of tests. And it's often the case that the test suite can be
  4. run automatically. It's also often the case
  5. that test suite gets run periodically. So
  6. for example, perhaps nightly, on every
  7. commit. Although in many cases that's infeasible,
  8. since, if commits are frequent and the test case is slow. The goal of
  9. a test suite is to show that some software under test has some
  10. desired properties, namely passing all of the
  11. tests. Although it's very common for real
  12. software to almost always be in a state of partial failure. But what
  13. we hope is that most of the time, most of these failures are
  14. not particularly severe ones. So the
  15. question is, if you're maintaining a software
  16. project, what goes into the test suite? And to a large extent, this is
  17. a matter of taste and preference, but on the other hand there's some
  18. pretty common features of nearly all test
  19. suites. It's very common, first of all,
  20. to have a lot of unit tests or at least feature-specific tests that
  21. are small tests that exercise very specialized
  22. behaviors. So, for example, if we're developing
  23. some sort of a web browser, we might have tested effect that different
  24. HTML elements render correctly and that sort of thing. It's also very common
  25. for a test suite to contain large realistic inputs. So, for example, if
  26. we're testing some sort of a microprocessor, maybe we'd boot up Linux and
  27. run it for a couple of hours, and the purpose of these kind
  28. of inputs is to provide realistic stresses on the system. And to exercise
  29. a lot of features and combination. And make sure that things are up
  30. to scuff. It's nearly, always a good idea to include regression tests in
  31. a test suite. Now, regression tests is
  32. basically any input that's cause any version
  33. in the software in the test, to fail at any time. And, there are several
  34. reasons regression tests exist. The main one of which, is we want to make sure
  35. that the software in your test doesn't regress. That is to say that it doesn't
  36. go back into a state in which it fails on a bug that we
  37. already fixed. There are a number of
  38. reasons why that could happen. First of all,
  39. regression tests are useful because whatever the
  40. defect was in the software that caused the
  41. bug in the first place, we might not have gotten rid of all the instances
  42. of that defect in a source code. So, for example, a buggy piece of
  43. code might have been cut and pasted to several other places, and those other
  44. locations might not be causing our system
  45. to fail currently, but some other change
  46. might enable the buggy code to fire, and make the bug happen again. Another
  47. reason is it is pretty easy through, for example, mess ups with the revision
  48. control system. To accidently go back to an old version of a file, before
  49. we fix the bug; if that happens we want to catch it as soon
  50. as possible, because, because some regression test
  51. trips. Third reasons is that defect in software
  52. come from errors in people's thinking, it's
  53. pretty often the case, that the person
  54. who introduce the defect into the software;
  55. didn't actually correct the error that they
  56. had in their thinking. Rather maybe somebody else fix the defect. And the person
  57. retains their mistaken assumption about some sort
  58. of an API or something. And due
  59. to this latent error in somebody's head, they can go ahead and start adding
  60. similar defects to the system later on. And if we have good regression tests,
  61. we stand more of a chance of catching those kind of things. And something
  62. that usually doesn't go into a test suite is a random tester. And for whatever
  63. reason I'm not totally sure that I
  64. understand all the reasons even. Random testing
  65. is often treated as a separate activity. This is related to the fact that random
  66. tests often are non deterministic unless we're
  67. being careful to preserve the same seed. They
  68. don't have a clear correctness criterion. And
  69. perhaps more importantly, random tests always have a
  70. possibility of showing us something new. That
  71. is to say, they have the possibility of
  72. introducing a test case that we haven't
  73. seen before. In fact, that's what we hope
  74. will happen. And the reason that might
  75. be undesirable, is the test suite is supposed
  76. to be predictable. It's supposed to consist of things that we know to test for.
  77. Now, if all of a sudden, the test
  78. starts containing new and different tests, then that's
  79. not necessarily good. So, for whatever combination of
  80. these reasons, random testing is often a separate activity.