Changes between Version 19 and Version 20 of Rebaseline


Ignore:
Timestamp:
Jul 12, 2012, 3:10:33 PM (13 years ago)
Author:
ojan@chromium.org
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Rebaseline

    v19 v20  
    1 = Chromium (for now) Rebaselining Tool =
     1= Rebaselining Tool =
    22
    33The rebaselining tool automatically produces new expected results using the layout test results from buildbots. The goal is that after running this tool, the working copy will be ready to create a change containing all new baselines.
    44
    5 The tool is part of the webkit-patch suite of utilities
     5You can also use "webkit-patch garden-o-matic" to run garden-o-matic locally and do rebaselines from the GUI.
    66
    77== How does this tool work? ==
    8 The script does the following for each platform:
    9   * Compile a list of tests that need rebaseline.
    10   * Download test result archive from buildbot for the platform.
    11   * Extract baselines from the archive file for all identified files.
    12   * Add new baselines to SVN or Git repository.
    13   * For each test that has been rebaselined, remove this platform option from the test in [wiki:TestExpectations TestExpectations files]. If no other platform options remaining after removal, delete the test from the file.
    14 
    15 After new baselines are generated, the tool launches a html page to compare the old and new baselines. (EDIT: No it doesn't. See https://lists.webkit.org/pipermail/webkit-dev/2011-November/018527.html ) The html can be disabled with `--no_html_results` option.
     8  * Grabs the results off the buildbot and puts them in the right place in your local repository.
     9  * Updates TestExpectations to remove the appropriate lines for rebaselined tests.
     10  * Runs webkit-patch optimize-baselines to remove duplicate baselines across ports.
    1611
    1712== How to use it? ==
    18   * Make sure the test files exist in repository and the build bots have cycled at least once after they were committed. If you are not sure, check the Release version of [http://build.chromium.org/buildbot/layout_test_results/ buildbot layout archive] to see whether your test results are available. By default, the rebaselining tool retrieves new baselines from the release version of buildbot layout test archive.
    19   * Update `TestExpectations`: add flag `REBASELINE` to the tests that need to be rebaselined.
    20   * Run rebaselining script: `Tools/Scripts/webkit-patch rebaseline-expectations`
    21   * If the script runs successfully, new baselines (expected files) are added to SVN repository and the rebaselined tests are removed from `TestExpectations`.
    22   * The tool launches a html page to compare the old and new baselines. (EDIT: No it doesn't. See https://lists.webkit.org/pipermail/webkit-dev/2011-November/018527.html ) Check the new baseline changes are expected.
    23   * Prepare a patch with the new baselines and `TestExpectations`, review the changes and then commit the patch.
     13Make sure the test files exist in repository (i.e. you've synced) and the build bots have cycled at least once. If you are not sure, check [http://trac.webkit.org/export/HEAD/trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/garden-o-matic.html garden-o-matic] or the buildbot waterfall to see whether your test results are available.
    2414
    25 == Examples ==
    26 We need rebaseline test `LayoutTests/foo1.html` for all three platforms and test `LayoutTests/foo2.html` for Chromium Win and Linux.
    27 
    28   * If these are tests just added, make sure the layout tests ran at buildbot and the test results have been archived.
    29   * Update `TestExpectations` to include the following lines:
     15Gives you a list of bots to select from, then a list of failing tests on each selected bot:
    3016{{{
    31 BUG1234 REBASELINE : foo1.html = FAIL
    32 BUG5678 REBASELINE WIN LINUX : foo2.html = FAIL
     17webkit-patch rebaseline
    3318}}}
    3419
    35   * Run `Tools/Scripts/webkit-patch rebaseline-expectations`.
    36   * Check that tests `foo1` and `foo2` were removed from TestExpectations.
     20Gives you a list of bots to select from then rebaselines the two tests on those bots:
     21{{{
     22webkit-patch rebaseline path/to/test1.html path/to/test2.html
     23}}}
    3724
    38   * Create a patch, review the changes and commit.
     25Finds all the tests with a REBASELINE modifier in TestExpectations and rebaselines appropriately:
     26{{{
     27webkit-patch rebaseline-expectations
     28}}}
     29
     30For example,
     31{{{
     32BUG1234 REBASELINE : path/to/test1.html = IMAGE
     33BUG1234 REBASELINE : path/to/test2.html = IMAGE+TEXT
     34BUG5678 REBASELINE WIN LINUX : path/to/test3.html = TEXT
     35}}}
     36
     37Rebaselines just the png for test1.html, the png and txt for test2.html and just the txt for test3.html.
    3938
    4039'''NOTE''': Don't check in with any tests still marked REBASELINE, as this will prevent others from cleanly using the tool.
    41 
    42 == Notes to GIT user ==
    43 
    44 Though the script add modified files to the index,
    45 you need to add LayoutTest/ChangeLog manually before commit. {{{prepare-ChangeLog --git-index}}} might help.
    46 
    47 == FAQ ==
    48 === Rebaseline tool complains ImageDiff is missing ===
    49 The rebaseline tool requires ImageDiff binary to compare pixel baselines. If you don't have ImageDiff binary in WebKitBuild/[Debug or Release], `Tools/Scripts/update-webkit  && Tools/Scripts/build-webkit` from the webkit checkout generates ImageDiff.