= Writing Reftests = == Disclaimer == Currently, reftests run only with new-run-webkit-tests. That means a patch which is only tested with reftests might not be allowed to be checked in. Anyone not working in the chromium port will not run the reftests and can not notice regressions. Please be aware this limitation and use reftests carefully at your responsibility until reftests are available on all platforms. There are on-going efforts that try to run new-run-webkit-tests on all platforms. https://bugs.webkit.org/show_bug.cgi?id=34984 == Reftests in WebKit == Reftest uses a HTML file ('-expected.html') to represent an expected rendering result, instead of pre-generated expected files, such as '-expected.txt', '-expected.checksum' and '-expected.png'. Let's see the following sample reftest. * hello.html {{{ reftest0001 Hello! }}} * hello-expected.html {{{ reftest0001 Hello! }}} A new-run-webkit-tests recognizes this is a reftest and run the test as if this is a normal layout test. You can think that a '-expected.html' file is used to produce '-expected.checksum' and '-expected.png' on the fly. If there is a difference between the rendering results of two html files, the test fails. There is yet another type of reftests, 'a mismatch reftest'. Let's see the following sample 'mismatch' reftest. * foo.html {{{ reftest0001 Hello! }}} * foo-expected-mismatch.html {{{ reftest0001 Hello! }}} In this case, the test fails if both produces the same results. The test passes if there is any differences between them. == File naming convention == * (normal) reftests: {testname}.html and {testname}-expected.html * 'mismatch' reftests: {testname}.html and {testname}-expected-mismatch.html == FAQ == * Q. Can we use a platform-specific expected html file? * A. Yes. Reftests try to obey the current layout test mechanism as possible as we can. * Q. What happens if we 'rebase' for reftests? * A. A 'Rebasing' doesn't make sense in reftests. A rebase line tool simply skips reftests. * Q. If we have hello.html, hello-expected.html and hello-expected-mismatch.html files, what happens? Does that run two reftests? * A. No. That should be considerd a wrong configuration. Don't use one test html file for both a (normal) reftest and a 'mismatch' reftest. Please choose the different file name for each test. * Q. Can we use '-expected.html' and '-expected.{checksum,png}' at the same time for one test file? * A. No. * Q. What kind of failures reftests produce? * A. If an image comparison failure happens in reftests, that is marked as a 'IMAGE' failure. If either of two html files crashes, it is considered as 'CRASH'. So does 'TIMEOUT'. == References == * https://bugs.webkit.org/show_bug.cgi?id=36065 for the current status. * https://developer.mozilla.org/en/Creating_reftest-based_unit_tests * http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/README.txt