Return to Video

Hard Testing Problems - Software Testing

  • 0:00 - 0:02
    So, sometimes we find ourselves faced
  • 0:02 - 0:04
    with, really hard software testing problems.
  • 0:04 - 0:06
    Let me go over some of the characteristics of this problem. Lack
  • 0:06 - 0:09
    of a specification is common, or perhaps only, lack of a good
  • 0:09 - 0:13
    specification. If there are no comparable implementations of the kind of system
  • 0:13 - 0:15
    we're implementing, that is to say of the first system of its
  • 0:15 - 0:18
    sort, that makes testing quite hard. Because what it probably means is
  • 0:18 - 0:21
    we are debugging the specification or even developing the specification as we
  • 0:21 - 0:26
    go. Big systems are hard to test, large highly structured input spaces,
  • 0:26 - 0:28
    like testing quite hard and so if you want to imagine sort
  • 0:28 - 0:31
    of a hard testing problem with a large highly structured input space.
  • 0:31 - 0:34
    Consider for example, the flight control computer is on a space craft
  • 0:34 - 0:37
    or an airplane, these things take sort of an enormous variety of input
  • 0:37 - 0:40
    from all sort of different redundant sensors. The time at which these inputs
  • 0:40 - 0:44
    arrive is significant. The spacecraft or the airplane has all sorts of
  • 0:44 - 0:46
    physical properties likes its altitude, its
  • 0:46 - 0:47
    attitude, the positions of various control
  • 0:47 - 0:50
    surfaces that all affected dynamics. These kind of systems are really, really,
  • 0:50 - 0:53
    sort of truly hard to test. Non-determinism can make a system very hard
  • 0:53 - 0:56
    to test. And, the issue here is that, we'll play a test case
  • 0:56 - 0:59
    against the system once, and it succeeds. But then, at some later time,
  • 0:59 - 1:02
    some variable that's not under our control will cause the system to fail on
  • 1:02 - 1:05
    that same input. Lots of hidden statement make systems hard to test. Some
  • 1:05 - 1:08
    systems that I've heard of either extremely hard to test or for example, Java
  • 1:08 - 1:10
    Virtual Machine. That are run by,
  • 1:10 - 1:12
    for example, financial organizations on lots of
  • 1:12 - 1:16
    cores with huge amounts of memory. These things have so, so much internal state,
  • 1:16 - 1:17
    that when something goes wrong, it's almost
  • 1:17 - 1:19
    impossible to make any inferences about what
  • 1:19 - 1:22
    was going on inside of it. And, you end up needing to try to
  • 1:22 - 1:23
    reproduce the problem. But, of course, this
  • 1:23 - 1:25
    is also extremely hard because the problem
  • 1:25 - 1:27
    probably happened Three hours into some sort
  • 1:27 - 1:29
    of a massive processing task. Finally, if
  • 1:29 - 1:30
    we lack strong oracles testing can be
  • 1:30 - 1:32
    really hard. And so, for example, something
  • 1:32 - 1:35
    like a large molecular simulation might be very hard to test. So ,we have
  • 1:35 - 1:38
    some sort of a new simulation code, we have no idea what the right
  • 1:38 - 1:41
    answer is supposed to be probably. It's running on some sort of a large
  • 1:41 - 1:44
    parallel machine, so we have a very hard time reproducing problems. Problems
  • 1:44 - 1:46
    might take a long time to occur, something like an autopilot might be
  • 1:46 - 1:49
    extremely hard to test, because the response of the thing is going to
  • 1:49 - 1:53
    be inherently in terms of the stability. And good behavior of the airplane.
  • 1:53 - 1:56
    And of course this is an incredibly large complex physical object that's
  • 1:56 - 1:59
    very hard to model and simulate in a reliable fashion. If you think
  • 1:59 - 2:01
    about picking a strong test for
  • 2:01 - 2:03
    an autopilot, it's almost inconceivable. Okay, like
  • 2:03 - 2:06
    I said, some sort of a giant JVM running with lots of cores
  • 2:06 - 2:09
    for a long time using a huge amount of heat. Very, very hard to
  • 2:09 - 2:11
    test the behavior of something in that kind of a state. So, the question we
  • 2:11 - 2:14
    want to ask ourselves is, how do we get a handle on these situations?
  • 2:14 - 2:15
    How should we test these things? And
  • 2:15 - 2:17
    often, there aren't really any easy answers. But
  • 2:17 - 2:20
    what we can do is, we can, leverage weak oracles to the maximum extent
  • 2:20 - 2:22
    possible if any of these things, the
  • 2:22 - 2:24
    simulation, the auto-pilot or the JVM actually crashes
  • 2:24 - 2:27
    in a bad fashion. Then we definitely know something's gone wrong. We can try to
  • 2:27 - 2:28
    bootstrap some degree of confidence in the
  • 2:28 - 2:31
    software under test by taking small test inputs.
  • 2:31 - 2:34
    For which we've hand-checked the output and trying to argue that, for example,
  • 2:34 - 2:38
    somehow, the auto-pilot if it responds well for these inputs also responds well
  • 2:38 - 2:41
    for other test inputs. And in the end, if we're thwarted in our
  • 2:41 - 2:44
    attempts to do really good testing, we probably have to rely on non-testing
  • 2:44 - 2:47
    methods. And so, of course we, we should be doing code inspections and
  • 2:47 - 2:50
    using formal methods on our systems in any case, if we care about
  • 2:50 - 2:53
    the reliability. But what's happening here, if we really can't test the system
  • 2:53 - 2:56
    effectively, we might have to rely on these things more than we would've liked.
  • 2:56 - 2:58
    Okay, so that's just a quick survey of,
  • 2:58 - 3:00
    things that can make testing really hard in practice.
Tytuł:
Hard Testing Problems - Software Testing
Opis:

more » « less
Video Language:
English
Team:
Udacity
Projekt:
CS258: Software Testing
Duration:
03:01

English subtitles

Revisions Compare revisions