Basics of creating a reftest

Firefox development tutorial showing you how to create an automated reftest
http://codefirefox.com

In the last video, I gave an introduction to reftests.

Which is basically just taking 2 web pages, rendering them, and then comparing them visually to see if they match or not.

In this video I'll go over how to create your own reftest.

So one thing that I wanted to do first is just show you how reftests are run.

And we looked at the mach command in the last video, but I just wanted to show you this reftest.py file.

So all mach does basically is run this reftest python file with some specified command line parameters.

And if we look at this add_options right here

we can see there's a bunch of different command line parameters that you can run if you wanted to run these manually.

It's a lot easier just to use mach, so I would just stick with that.

I'm going to close that.

So the first thing you want to do when creating a reftest is to list it in your .list file.

So I'm just going to pretend that I want to add it to content/html/content/reftests/reftest.list

OK, and I'm going to add it to a new line here.

I'm going to start it with == to mean that I want these pages to match.

And I'm going to call them test1.html and test2.html

The next thing I'm going to do is create 2 files in that directory.

One called test1.html

And this page is going to be basically a simple html page, and it's going to have strong markup here.

Which is basically the same as bold.

So what I want to do is compare that this strong element right here is going to match the equivalent b, which is bold element.

So that's test1.html.

And then I'm going to copy that to test2.html

And then I'm going to open test2.html

And this strong tag right here, I'm going to change that to bold, to b.

Save and quit.

So now we created our reftest, it's basically that simple.

Now to run it we're going to type mach reftest content/html/content/reftests/reftest.list.

So I'm just specifying the list file that I added it to.

And I'm going to add a filter here, just so it only runs the one I just added, and not everything in that file.

And when I run this, we should see that it's going to be successful.

And if we look right up here, we can see that successful is 1.

So that passed, which is what we were expecting, so that's good.

Now I want to show you what will happen if there's actually a mismatch.

So I'm going to open up test2.html

And let's just say we took out this b right there.

So the first one, if you recall has strong.

The second one has no tag, so these should render a little bit differently.

And the test should fail

So I'm just going to run it again.

And if I scroll up here, we'll see unexpected is 1, successful is 0.

So yes it definitely did fail.

And you can actually just scroll up here a little bit more.

And anytime that there's a failure or a mismatch with reftests.

You'll see that it gives you the actual data URL of both images.

So this is image 1, up here above it.

And this is image2 right here.

And if you paste those data URLs in a web browser URL bar.

You're going to actually see the images rendered.

So you can look at it visually to see if you have any intuition of why that test failed.

One more thing that I wanted to add, is if you're adding a brand new .list file that doesn't exist yet.

The way that reftests work, is that it recursively includes other list files.

So the master list file that you want to add to is just layout/reftests/reftest.list

And for example if I look for the reftest [list] that I had

content/htmlcontent... reftest.list you can find it right there.

I'm going to quit.

And that's all I wanted to show you for this video.