1 00:00:00,000 --> 00:00:02,000 Let's take a little bit different view of the same problem. 2 00:00:02,000 --> 00:00:06,000 So, what I want to do is draw a graph here, so in the level of code coverage 3 00:00:06,000 --> 00:00:09,000 with a random test cases are inducing on the system under test. 4 00:00:09,000 --> 00:00:12,000 So again, we're testing a web browser, and on the x axis, 5 00:00:12,000 --> 00:00:14,000 this is going to be a little bit fuzzy unfortunately. 6 00:00:14,000 --> 00:00:17,000 But what I want to show is how far into the web browser we execute. 7 00:00:17,000 --> 00:00:22,000 Which is just checking, for example, if the incoming data is even valid HTTP. 8 00:00:22,000 --> 00:00:26,000 Once we get valid HTTP, the browser is going to scan and make sure it got valid HTML. 9 00:00:26,000 --> 00:00:29,000 It's going to be doing lexical analysis and checking of HTML. 10 00:00:29,000 --> 00:00:34,000 If the input fails to be rejected by this kind of code, it's going to go on the rendering engine 11 00:00:34,000 --> 00:00:36,000 and finally, it might have sort of some more advanced processing 12 00:00:36,000 --> 00:00:40,000 which is dealing with things like forms, cookies, scripting and such. 13 00:00:40,000 --> 00:00:42,000 Okay so, we have this graph. 14 00:00:42,000 --> 00:00:46,000 So, now let's see what happens when we fuzz the web browser using totally random bits. 15 00:00:46,000 --> 00:00:50,000 Well, what's kind of most likely happen is most of those bits that come in, 16 00:00:50,000 --> 00:00:52,000 are now even going to be valid HTTP responses. 17 00:00:52,000 --> 00:00:56,000 So, we're going to get coverage rapidly drops off 18 00:00:56,000 --> 00:01:01,000 and what's left is almost always going to fail somewhere else. 19 00:01:01,000 --> 00:01:04,000 What we're going to see is, we're going to spend the bulk of our testing effort 20 00:01:04,000 --> 00:01:08,000 rejecting random sequences of bits very early on 21 00:01:08,000 --> 00:01:11,000 and very little of our testing effort, testing code here. 22 00:01:11,000 --> 00:01:13,000 Again, as I said, if that's what we're trying to do, 23 00:01:13,000 --> 00:01:16,000 if we really want to be stressing the early parts of the web browser code, then that's great. 24 00:01:16,000 --> 00:01:18,000 And random testing is perfectly good at that, but on the other hand, 25 00:01:18,000 --> 00:01:24,000 if we are interested in is for the broad coverage of the software under test, then we're going to fail. 26 00:01:24,000 --> 00:01:27,000 The red color indicates random bits. 27 00:01:27,000 --> 00:01:31,000 The next thing we can do is we could go random input generator 28 00:01:31,000 --> 00:01:34,000 totally respects the constraints of the HTTP protocol. 29 00:01:34,000 --> 00:01:38,000 Furthermore, we can adapt it, so the text that it generates contains valid lexical elements of HTML. 30 00:01:38,000 --> 00:01:42,000 That is to say it's composed of things like races with tags in them, 31 00:01:42,000 --> 00:01:45,000 other kinds of tags so this was the directive, but this isn't too hard to do. 32 00:01:45,000 --> 00:01:48,000 So, if we do something like that, I think I'm going to call that is, 33 00:01:48,000 --> 00:01:51,000 use green to represent protocol credit code. 34 00:01:51,000 --> 00:01:56,000 I'm using sort of fuzzy terms of LAN devising here, I'm not trying to use any kind of standard terms. 35 00:01:56,000 --> 00:02:00,000 So now what's going to happen is, hopefully we'll get pretty good coverage of the protocol code still. 36 00:02:00,000 --> 00:02:04,000 To get quite good coverage of lexical HTML processing and we're going to fall off the cliff again. 37 00:02:04,000 --> 00:02:07,000 Because as soon as we get to the render, it's going to become apparent, 38 00:02:07,000 --> 00:02:10,000 and we didn't try hard enough to generate valid HTML 39 00:02:10,000 --> 00:02:12,000 and we're going to get something to render very often. 40 00:02:12,000 --> 00:02:15,000 So now, what we've accomplished here while we pushed the coverage that we're getting 41 00:02:15,000 --> 00:02:21,000 on the software under test farther into the system, it is farther into the HTML processing chain, 42 00:02:21,000 --> 00:02:23,000 but still haven't pushed it very far. 43 00:02:23,000 --> 00:02:26,000 So, the next thing that we can do is use some sort of a grammar 44 00:02:26,000 --> 00:02:30,000 or some sort of a structural definition of HTML to generate random but valid HTML. 45 00:02:30,000 --> 00:02:33,000 The next thing is valid HTML. 46 00:02:33,000 --> 00:02:39,000 And so what's going to happen is, a coverage of the protocol code and the lexer may decrease, 47 00:02:39,000 --> 00:02:41,000 while on the other hand, we're going to be able to push 48 00:02:41,000 --> 00:02:45,000 into the HTML processing code quite deeply before falling off a cliff. 49 00:02:45,000 --> 00:02:48,000 So what've we done? We've traded off coverage in the early parts of the web browser 50 00:02:48,000 --> 00:02:53,000 which may well be so simple that we don't care about much about them for coverage farther in. 51 00:02:53,000 --> 00:02:59,000 And so finally, what we could do is, generate random code that includes 52 00:02:59,000 --> 00:03:04,000 elements of scripting, forms, whatever else that we're interested in testing 53 00:03:04,000 --> 00:03:08,000 and we can run that through and now we can start randomly testing our browser with this. 54 00:03:08,000 --> 00:03:10,000 What's going to happen now is, our coverage might decrease 55 00:03:10,000 --> 00:03:14,000 even a little bit more in the early parts because we're spending more time doing other things, 56 00:03:14,000 --> 00:03:16,000 but we're probably not going to fall off a cliff at all. 57 00:03:16,000 --> 00:03:20,000 And so, you can see that in most cases when we do random testing, 58 00:03:20,000 --> 00:03:23,000 what we're looking for is something like this kind of flat line, 59 00:03:23,000 --> 00:03:25,000 and what this flat line indicates is that we're covering 60 00:03:25,000 --> 00:03:28,000 all parts of the software under test roughly equally. 61 00:03:28,000 --> 00:03:31,000 What we're going to see is as we look through more random testing examples, 62 00:03:31,000 --> 00:03:35,000 is getting sort of a coverage curve like this often requires quite a lot of work, quite a lot of sensitivity 63 00:03:35,000 --> 00:03:38,000 to the structure of the input demand, but on the other hand, 64 00:03:38,000 --> 00:03:42,000 we get paid back for that work with random tests that can exercise the entire software under test 65 00:03:42,000 --> 00:03:46,000 and that's going to be a valuable thing in many cases.