Version 6 (modified by 14 years ago) ( diff ) | ,
---|
Chromium (for now) Rebaselining Tool
The rebaselining tool automatically produces new expected results using the layout test results from buildbots. The goal is that after running this tool, the working copy will be ready to create a change containing all new baselines.
The tool is written in Python and lives in WebKitTools/Scripts
(http://trac.webkit.org/browser/trunk/WebKitTools/Scripts/rebaseline-chromium-webkit-tests).
How does this tool work?
The script does the following for each platform:
- Compile a list of tests that need rebaseline.
- Download test result archive from buildbot for the platform.
- Extract baselines from the archive file for all identified files.
- Add new baselines to SVN or Git repository.
- For each test that has been rebaselined, remove this platform option from the test in _test_expectation.txt_. If no other platform options remaining after removal, delete the test from the file.
After new baselines are generated, the tool launches a html page to compare the old and new baselines. The html can be disabled with --no_html_results
option.
How to use it?
- Make sure the test files exist in repository and the build bots have cycled at least once after they were committed. If you are not sure, check the Release version of buildbot layout archive to see whether your test results are available. By default, the rebaselining tool retrieves new baselines from the release version of buildbot layout test archive.
- Update
test_expectations.txt
: add flagREBASELINE
to the tests that need to be rebaselined. - Run rebaselining script:
WebKitTools/Scripts/rebaseline-chromium-webkit-tests
. If you want to pull new baselines from the webkit.org canary, use--webkit_canary
(-w
) option. - If the script runs successfully, new baselines (expected files) are added to SVN repository and the rebaselined tests are removed from
test_expectations.txt
. - The tool launches a html page to compare the old and new baselines. Check the new baseline changes are expected.
- Prepare a patch with the new baselines and
test_expectations.txt
, review the changes and then commit the patch.
Examples
We need rebaseline test LayoutTests/foo1.html
for all three platforms and test LayoutTests/foo2.html
for Chromium Win and Linux.
- If these are tests just added, make sure the layout tests ran at buildbot and the test results have been archived.
- Update
test_expectations.txt
to include the following lines:BUG1234 REBASELINE : LayoutTests/foo1.html = FAIL BUG5678 REBASELINE WIN LINUX : LayoutTests/foo2.html = FAIL
- Run
WebKitTools/Scripts/rebaseline-chromium-webkit-tests
. - Check that tests
foo1
andfoo2
were removed from test_expectations.txt.
- Create a patch, review the changes and commit.
NOTE: Don't check in with any tests still marked REBASELINE, as this will prevent others from cleanly using the tool.
Notes to GIT user
Though the script add modified files to the index,
you need to add LayoutTest/ChangeLog manually before commit. prepare-ChangeLog --git-index
might help.
FAQ
Rebaseline tool complains ImageDiff is missing
The rebaseline tool requires ImageDiff binary to compare pixel baselines. If you don't have ImageDiff binary in WebKitBuild/[Debug or Release], run WebKitTools/Scripts/build-dumprendertree from the webkit checkout generates ImageDiff.