Return to Video

03-27 When Coverage Doesnt Work

  • 0:00 - 0:02
    How do we boost a number of different coverage metrics?
  • 0:02 - 0:06
    Iâd like to return to a topic that I discussed at the beginning of this unit which is the input domain.
  • 0:06 - 0:11
    What coverage is letting us do is divide up the input domain into regions in such a way,
  • 0:11 - 0:16
    for any given region, any test input in that region, will accomplish some specific coverage task.
  • 0:16 - 0:19
    Any input that we select within this particular region of the input domain
  • 0:19 - 0:23
    will cause a particular statement or line to execute, cause the branch to execute in some
  • 0:23 - 0:27
    specific direction, execute a loop zero times, one time, or more, etc.
  • 0:27 - 0:30
    And so, the obvious problem is, if we partition the input domain this way
  • 0:30 - 0:35
    and we go ahead and test, it is easier for me to reach the region in the input domain,
  • 0:35 - 0:39
    weâre going to get good coverage, but weâre not going to be able to do is find areas of omission.
  • 0:39 - 0:44
    That is to say, nothing about this process of selecting points in the input domain and testing them
  • 0:44 - 0:47
    in order to achieve good coverage is going to let us discover of what we havenât implement.
  • 0:47 - 0:51
    So, let me give a quick example--so, typically, as we talk about unit too, the software under test
  • 0:51 - 0:53
    is using APIs provided by the system.
  • 0:53 - 0:56
    Perhaps, the system under test is creating files on the hard disc.
  • 0:56 - 1:02
    One extremely possible kind of bug that we would put in to the system under test
  • 1:02 - 1:06
    is failing to check error codes that could be returned from file creation operations
  • 1:06 - 1:10
    that happen when the disc is full or when there is a hard disc failure or something like that.
  • 1:10 - 1:12
    And so, what I really mean here is that, for example,
  • 1:12 - 1:17
    we got a full branch coverage of the system under test but there just isnât a branch in the test
  • 1:17 - 1:20
    that should have been taken when the hard disc returns an error code.
  • 1:20 - 1:23
    And so, if the branch that should be there isnât there, when the disc does return an error code,
  • 1:23 - 1:25
    something weird is going to happen, the software is going to fail.
  • 1:25 - 1:29
    So, the fundamental fact here is that coverage metrics are not particularly good
  • 1:29 - 1:32
    at discovering areas of omission like missing error checks.
  • 1:32 - 1:34
    To discover those, we need to use other methods.
  • 1:34 - 1:37
    So, for example, we discussed fault injection where we make the disc fail,
  • 1:37 - 1:40
    we will make it send something bad up to the system and we see what happens,
  • 1:40 - 1:44
    and in that case, if weâre missing an error check, then the system should actually do the wrong thing
  • 1:44 - 1:47
    and will be able to discover this by watching the system misbehave.
  • 1:47 - 1:50
    Another thing we could have done is partition the input domain in a different way.
  • 1:50 - 1:54
    That is to say not partition the input domain using an automated coverage metrics
  • 1:54 - 1:56
    rather partition the input domain using the specification.
  • 1:56 - 2:00
    So, if we partitioning input domain based on the specification
  • 2:00 - 2:05
    and the specification mentions the need for our system to respond gracefully to disc errors,
  • 2:05 - 2:07
    there is going to be some little corner of the input space
  • 2:07 - 2:10
    that is triggered only when disc fail and while we test that.
  • 2:10 - 2:12
    So the point that Iâm getting to here is that there are multiple ways
  • 2:12 - 2:15
    of partitioning the input domain for purposes of testing.
  • 2:15 - 2:18
    Partitioning the input domain based on features of the code
  • 2:18 - 2:20
    is one way and it happens to be quite effective in general,
  • 2:20 - 2:24
    but since I canât find the areas of omission, we also want sample the input domain in different ways.
  • 2:24 - 2:26
    We also want to sample the input domain in other ways
  • 2:26 - 2:30
    and thatâs the team that we will return to a full force in units four, five, and six.
  • 2:30 -
    All right, so that wraps up our quick survey of test coverage metrics.
Title:
03-27 When Coverage Doesnt Work
Description:

03-27 When Coverage Doesnt Work

more » « less
Team:
Udacity
Project:
CS258: Software Testing
Duration:
02:34
Udacity Robot edited English subtitles for cs258 unit2 22 l When Coverage Doesnt Work
Amara Bot added a translation

English subtitles

Incomplete

Revisions Compare revisions