English subtitles

← Mystery Test Solution - Software Debugging

Get Embed Code
3 Languages

Showing Revision 3 created 05/25/2016 by Udacity Robot.

  1. Here we are again, we have a mystery test function and what is it actually in here that causes the input?
  2. What is it that causes mystery test to fail? Well, we can again use delta debugging to figure this out.
  3. These are implementation of delta debugging.
  4. I'm going to extend this such that the test now comes as a parameter.
  5. And then, we can invert ddmin with an appropriate test just as we'd like.
  6. And I'm going to invert the ddmin with this input for which you already know
  7. that it causes mystery test to fail and then passing mystery test as a parameter.
  8. Let's see what's in here that causes mystery test to fail.
  9. Press 1, we press 1 and we see a single character, a single dot suffices.
  10. A single dot suffices to cause mystery test to fail. So what we have here is classical set.
  11. First, we use a fuzz tester to test a program and then we repeat
  12. and repeat this until the program breaks.
  13. Then we feed this into delta debugging which again runs the program again and again
  14. until we get a minimal failing input such as a single dot.
  15. Such minimal failing inputs are very important as it comes to convincing other people to fix a bug.
  16. Suppose your program for testing is a huge SQL server.
  17. Suppose you use a first tester to generate extremely long
  18. and complex SQL query that you sent to the SQL server.
  19. And then, the SQL server is going to choke on one of these super, super complex inputs
  20. and then you say to the SQL developers, "Hey, I have this very, very long and complicated
  21. SQL query and I can use this to break the server," but the developers are going to tell you,
  22. "Ha, a big query like this one, never ever happens in practice".
  23. So maybe this will crash our server but it's really not the top priority right now.
  24. We'd rather care about real queries, thank you very much.
  25. So what you'd do then is you run this big, big complicated query to delta debugging
  26. which then will give you some minimal failing input.
  27. which now will be way, way smaller and actually look
  28. like something so simple that it can actually happen in practice.
  29. And all of a sudden, the SQL developers will take you very seriously because all of a sudden
  30. this looks like something that could happen in practice.
  31. True story from Microsoft.