wiki:Writing Reftests
Last modified 3 years ago Last modified on 10/22/12 17:36:12

Writing Reftests

Reftests in WebKit

Reftest uses a HTML file ('-expected.html') to represent an expected rendering result, instead of pre-generated expected files (such as '-expected.txt', '-expected.checksum' and '-expected.png').

Let's see the following sample reftest.

  • hello.html
  • hello-expected.html

A new-run-webkit-tests recognizes this is a reftest and runs the test as if this is a normal layout test. You can think that a '-expected.html' file is used to produce a '-expected.checksum' and a '-expected.png' on the fly. If there is a difference between the rendering results of two html files, the test fails.

There is yet another type of reftests, 'a mismatch reftest'.

Let's see the following sample 'mismatch' reftest.

  • foo.html
  • foo-expected-mismatch.html

In this case, the test fails if both HTML files produce the same results. The test passes if there is any differences between them.

You can see the sample reftest here:

File naming convention

  • (normal) reftests: {testname}.html and {testname}-expected.html
  • 'mismatch' reftests: {testname}.html and {testname}-expected-mismatch.html


  • Q. Can we use a platform-specific expected html file?
  • A. Yes. Reftests try to obey the current layout test mechanism as possible as we can.
  • Q. What happens if we 'rebase' for reftests?
  • A. A 'Rebasing' doesn't make sense in reftests. A rebase line tool simply skips reftests.
  • Q. If we have hello.html, hello-expected.html and hello-expected-mismatch.html files, what happens? Does that run two reftests?
  • A. No. That should be considerd a wrong configuration. Don't use one test html file for both a (normal) reftest and a 'mismatch' reftest. Please choose the different file name for each test.
  • Q. Can we use '-expected.html' and '-expected.{checksum,png}' at the same time for one test file?
  • A. No.
  • Q. What kind of failures reftests produce?
  • A. If an image comparison failure happens in reftests, that is marked as a 'ImageOnlyFailure'. If either of two html files crashes, it is considered as 'Crash'. So does 'Timeout'.