wiki:Writing Reftests

Version 8 (modified by, 13 years ago) (diff)


Writing Reftests

Reftest is not supported yet in WebKit. Don't check in any reftests until an announcement comes. Please see for the current status.


Currently, reftests are run on only platforms that use new-run-webkit-tests by default. Please be aware this limitation when you decide to write new tests using reftest.

See for the current status.

Reftests in WebKit

Reftest uses a HTML file ('-expected.html') to represent an expected rendering result, instead of pre-generated expected files, such as '-expected.txt', '-expected.checksum' and '-expected.png'.

Let's see the following sample reftest.

  • hello.html
  • hello-expected.html

A new-run-webkit-tests recognizes this is a reftest and run the test as if this is a normal layout test. You can think that a '-expected.html' file is used to produce '-expected.checksum' and '-expected.png' on the fly. If there is a difference between the rendering results of two html files, the test fails.

Screenshot: Todo

There is yet another type of reftests, 'a mismatch reftest'.

Let's see the following sample 'mismatch' reftest.

  • foo.html
  • foo-expected-mismatch.html

In this case, the test fails if both produces the same results. The test passes if there is any differences between them.

Screenshot: Todo

File naming convention

  • (normal) reftests: {testname}.html and {testname}-expected.html
  • 'mismatch' reftests: {testname}.html and {testname}-expected-mismatch.html


  • Q. Can we use a platform-specific expected html file?
  • A. Yes. Reftests try to obey the current layout test mechanism as possible as we can.
  • Q. What happens if we 'rebase' for reftests?
  • A. A 'Rebasing' doesn't make sense in reftests. A rebase line tool simply skips reftests.
  • Q. If we have hello.html, hello-expected.html and hello-expected-mismatch.html files, what happens? Does that run two reftests?
  • A. No. That should be considerd a wrong configuration. Don't use one test html file for both a (normal) reftest and a 'mismatch' reftest. Please choose the different file name for each test.
  • Q. Can we use '-expected.html' and '-expected.{checksum,png}' at the same time for one test file?
  • A. No.
  • Q. What kind of failures reftests produce?
  • A. If an image comparison failure happens in reftests, that is marked as a 'IMAGE' failure. If either of two html files crashes, it is considered as 'CRASH'. So does 'TIMEOUT'.