| Version 14 (modified by , 14 years ago) ( diff ) |
|---|
Writing Reftests
Disclaimer
Currently, reftests run only with new-run-webkit-tests. That means a patch which is only tested with reftests might not be allowed to be checked in. Anyone not working in the chromium port will not run the reftests and can not notice regressions. Please be aware this limitation and use reftests carefully at your responsibility until reftests are available on all platforms.
[Update]: Reftests can not work on Mac build of WebKit. See https://bugs.webkit.org/show_bug.cgi?id=58858.
There are on-going efforts that try to run new-run-webkit-tests on all platforms. https://bugs.webkit.org/show_bug.cgi?id=34984
Reftests in WebKit
Reftest uses a HTML file ('-expected.html') to represent an expected rendering result, instead of pre-generated expected files (such as '-expected.txt', '-expected.checksum' and '-expected.png').
Let's see the following sample reftest.
- hello.html
<html><head><title>reftest0001</title> <body><strong>Hello!</strong></body> </html>
- hello-expected.html
<html><head><title>reftest0001</title> <body><b>Hello!</b></body> </html>
A new-run-webkit-tests recognizes this is a reftest and runs the test as if this is a normal layout test. You can think that a '-expected.html' file is used to produce a '-expected.checksum' and a '-expected.png' on the fly. If there is a difference between the rendering results of two html files, the test fails.
There is yet another type of reftests, 'a mismatch reftest'.
Let's see the following sample 'mismatch' reftest.
- foo.html
<html><head><title>reftest0001</title> <body><strong>Hello!</strong></body> </html>
- foo-expected-mismatch.html
<html><head><title>reftest0001</title> <body>Hello!</body> </html>
In this case, the test fails if both HTML files produce the same results. The test passes if there is any differences between them.
You can see the sample reftest here: http://trac.webkit.org/changeset/81644
File naming convention
- (normal) reftests: {testname}.html and {testname}-expected.html
- 'mismatch' reftests: {testname}.html and {testname}-expected-mismatch.html
FAQ
- Q. Can we use a platform-specific expected html file?
- A. Yes. Reftests try to obey the current layout test mechanism as possible as we can.
- Q. What happens if we 'rebase' for reftests?
- A. A 'Rebasing' doesn't make sense in reftests. A rebase line tool simply skips reftests.
- Q. If we have hello.html, hello-expected.html and hello-expected-mismatch.html files, what happens? Does that run two reftests?
- A. No. That should be considerd a wrong configuration. Don't use one test html file for both a (normal) reftest and a 'mismatch' reftest. Please choose the different file name for each test.
- Q. Can we use '-expected.html' and '-expected.{checksum,png}' at the same time for one test file?
- A. No.
- Q. What kind of failures reftests produce?
- A. If an image comparison failure happens in reftests, that is marked as a 'IMAGE' failure. If either of two html files crashes, it is considered as 'CRASH'. So does 'TIMEOUT'.
- Q. It seems reftests are skipped. Why?
- A. In current implementation, reftests run only when you enable 'pixel tests'. In some platforms, pixel tests are disabled in default. You can use '--pixel-tests' option to enable pixel tests. I agree this is confusing. We might change this behavior in the future.
