English subtitles

← Problems With Random Tests - Software Testing

Get Embed Code
1 Language

Showing Revision 3 created 05/25/2016 by Udacity Robot.

  1. Now here we have below a code from Wikipedia that does the same thing.
  2. And I'm not going to go through the logic here.
  3. But what you can see is this is quite a bit more idiomatic Python.
  4. It's actually quite a bit nicer than the code that I wrote.
  5. So if you like that better then use this as a model instead of the code that I wrote.
  6. The code that I wrote is pretty kind of dumb and obvious.
  7. We have equivalently a Luhn valid check sum using the Wikipedia algorithm,
  8. which just does the obvious thing.
  9. And then what I have here is a random tester, which generates a random credit card number
  10. with a certain prefix and 15 digits and then ensures is that it's valid.
  11. The validity checking function for credit card numbers simply makes sure
  12. that it has the right length, that it has the right prefix, and that the checksum comes out to be 0.
  13. That is to say, the is-Luhn-valid function returns true. So that's all there is to it.
  14. But what I want to do finally is take a look at one other issue.
  15. I'm going to comment out my code here and comment in some different code.
  16. What we're doing here is generating completely random 15-digit numbers.
  17. What I'm doing is generating a random integer between 0 and the smallest 16-digit number.
  18. The largest number that could be generated here is the largest 15-digit number.
  19. And then we're going to zero-fill, convert that to a string, and do a zero-fill operation,
  20. which adds leading zeros.
  21. Now we have a completely random number that is 15 digits long.
  22. And if that checks out to be a valid credit card number,
  23. we're going to increment our validity checker and then finally after doing this 100 thousand times
  24. we're going to print the number of valid credit card numbers that we got.
  25. So let's run this and see what happens. Okay.
  26. We got no valid credit card numbers out of 100 thousand.
  27. So the problem is the prefix was too long.
  28. With a 6-digit prefix, the chance is one in a million that we'll generate just this prefix
  29. and then it goes down to one in 10 million that will meet the prefix and the checksum requirement.
  30. So if we start off with a much smaller prefix like just 37 and this is basically anything
  31. in the American Express system I think, now let's see what happens.
  32. We're going to generate 100 thousand credit card numbers and 104 of them came out to be valid.
  33. So even with just a two-digit prefix, it's pretty unlikely that we generate valid credit card numbers.
  34. And so what that means is if we're generating lots of invalid credit card numbers
  35. of course we're stressing only a very small bit of a transaction processing logic
  36. that checks for valid credit card numbers and we're not stressing the rest of it.
  37. So what I hoped I accomplished here is first of all motivated the fact that this generation of valid data
  38. is a real one and second of all, to give you a little bit of a feel for what code looks like
  39. that we usually have to write to generate valid inputs.
  40. And so, if we go back to our web browser example, you can see that we will be doing a
  41. similar exercise but it'd just be quite a bit more sophisticated to generate
  42. for example a valid HTML or a valid HTML with scripts and other things.
  43. That it would take to actually do meaningful fuzzing of a web browser as shown by the blue line here.
  44. And so now to do this, instead of spending half an hour or however long you spend
  45. on the PR quiz now maybe you're going to be spending many weeks.
  46. But on the other hand, what we're going to get out of this if we do it right
  47. is a lot of high-value bugs including security bugs in our web browser and strongly possible
  48. but of course not guaranteed that the value we get out of those bugs
  49. that we find in a web browser but the effort would've been worth it.