wiki:Writing Reftests

Version 17 (modified by danakj@chromium.org, 12 years ago) ( diff )

--

Writing Reftests

Reftests in WebKit

Reftest uses a HTML file ('-expected.html') to represent an expected rendering result, instead of pre-generated expected files (such as '-expected.txt', '-expected.checksum' and '-expected.png').

Let's see the following sample reftest.

  • hello.html 
    <html><head><title>reftest0001</title>
<body><strong>Hello!</strong></body>
</html>
  • hello-expected.html 
    <html><head><title>reftest0001</title>
<body><b>Hello!</b></body>
</html>

A new-run-webkit-tests recognizes this is a reftest and runs the test as if this is a normal layout test. You can think that a '-expected.html' file is used to produce a '-expected.checksum' and a '-expected.png' on the fly. If there is a difference between the rendering results of two html files, the test fails.

There is yet another type of reftests, 'a mismatch reftest'.

Let's see the following sample 'mismatch' reftest.

  • foo.html 
    <html><head><title>reftest0001</title>
<body><strong>Hello!</strong></body>
</html>
  • foo-expected-mismatch.html 
    <html><head><title>reftest0001</title>
<body>Hello!</body>
</html>

In this case, the test fails if both HTML files produce the same results. The test passes if there is any differences between them.

You can see the sample reftest here: ​http://trac.webkit.org/changeset/81644

File naming convention

  • (normal) reftests: {testname}.html and {testname}-expected.html
  • 'mismatch' reftests: {testname}.html and {testname}-expected-mismatch.html

FAQ

  • Q. Can we use a platform-specific expected html file?
  • A. Yes. Reftests try to obey the current layout test mechanism as possible as we can.
  • Q. What happens if we 'rebase' for reftests?
  • A. A 'Rebasing' doesn't make sense in reftests. A rebase line tool simply skips reftests.
  • Q. If we have hello.html, hello-expected.html and hello-expected-mismatch.html files, what happens? Does that run two reftests?
  • A. No. That should be considerd a wrong configuration. Don't use one test html file for both a (normal) reftest and a 'mismatch' reftest. Please choose the different file name for each test.
  • Q. Can we use '-expected.html' and '-expected.{checksum,png}' at the same time for one test file?
  • A. No.
  • Q. What kind of failures reftests produce?
  • A. If an image comparison failure happens in reftests, that is marked as a 'ImageOnlyFailure'. If either of two html files crashes, it is considered as 'Crash'. So does 'Timeout'.

References

Note: See TracWiki for help on using the wiki.