Changes between Version 20 and Version 21 of Rebaseline


Ignore:
Timestamp:
Aug 3, 2012 6:33:36 PM (9 years ago)
Author:
pkasting@chromium.org
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Rebaseline

    v20 v21  
    1 = Rebaselining Tool =
     1= Rebaselining =
    22
    3 The rebaselining tool automatically produces new expected results using the layout test results from buildbots. The goal is that after running this tool, the working copy will be ready to create a change containing all new baselines.
     3There are three common ways to rebaseline.  Using garden-o-matic is generally the easiest and safest.  It's also possible to rebaseline by marking your local TestExpectations file and then using "webkit-patch rebaseline".  Both of these methods use baselines created by the buildbots and are described below.  The third method, which may be easier for mass rebaselines (such as when an antialiasing change affects hundreds of tests), is to use RebaselineServer, which relies on locally-generated baselines.
    44
    5 You can also use "webkit-patch garden-o-matic" to run garden-o-matic locally and do rebaselines from the GUI.
    6 
    7 == How does this tool work? ==
     5Both garden-o-matic and the rebaselining tool perform the same core actions:
    86  * Grabs the results off the buildbot and puts them in the right place in your local repository.
    9   * Updates TestExpectations to remove the appropriate lines for rebaselined tests.
     7  * Updates TestExpectations to remove the appropriate lines for rebaselined tests.  (This may not always work with garden-o-matic.)
    108  * Runs webkit-patch optimize-baselines to remove duplicate baselines across ports.
    119
    12 == How to use it? ==
    13 Make sure the test files exist in repository (i.e. you've synced) and the build bots have cycled at least once. If you are not sure, check [http://trac.webkit.org/export/HEAD/trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/garden-o-matic.html garden-o-matic] or the buildbot waterfall to see whether your test results are available.
     10Notably, these tools do NOT actually "svn add" any new baseline files or prepare the ChangeLog; you must still do both these steps before landing your local changes.
     11
     12== Rebaselining with garden-o-matic ==
     13
     14You can use garden-o-matic to rebaseline tests with a convenient GUI.  First update your checkout (or baseline optimization may do the wrong thing), then launch the tool:
     15{{{
     16update-webkit
     17webkit-patch garden-o-matic
     18}}}
     19
     20You should start on the "Unexpected failures" tab.  Look at the "Latest revision processed by every bot" line.  If this revision does not contain all tests and changes you want reflected in the new baselines, do not proceed; you must wait until the bots cycle sufficiently, or you may check in erroneous baselines.
     21
     22To examine the current results for a particular test, either hover the test name on the "Unexpected failures" tab and click the blue "Examine" button that appears to the right, or switch to the "Expected failures" tab and hover the appropriate test directory and click "Examine".  Either one will take you to the "Results" tab where you can select the test you're interested in from the list at the top.  Be sure that your test does not appear in both the "expected" and "unexpected" lists, as may happen if your TestExpectations already list some failures for the test, but you're now seeing additional ones; if this is the case, then each "Results" set will only reflect the expected or unexpected failing results, respectively.  This can lead to generating incorrect baselines.  To resolve this, update your test expectations to list all the failures, commit, and wait for the bots to cycle, so that all your failures will be "expected".
     23
     24Once you have the results up for a test, you can use the arrow buttons on the right side just below the test list to quickly scan through the current results for the test.  (Going past the last result for a test will skip to the next test in the list.)  Make sure all the results appear (and are as you expect); this ensures that the tool can actually reach all the bots and fetch the expected results.  (Unreachable bots can be another source of incorrect baseline generation.)
     25
     26When everything looks good, click the "Rebaseline" button.  A small panel should appear at the bottom noting that the rebaselining is in process.  While this is occurring, you can skip to other tests and examine their results.  Once this process finishes, you'll see a "done" message.
     27
     28After you've generated all the baselines necessary, do an "svn status" so you can find and add all newly-added baselines.  Also open your test expectations file and remove any lines relating to the tests you've changed -- garden-o-matic sometimes does this automatically, but not always.
     29
     30== Rebaselining with the rebaselining tool ==
     31
     32You can also rebaseline without using garden-o-matic.  First ensure your checkout is up to date and your test results are all available -- see the garden-o-matic section above for a way to check what revision the buildbots have cycled through.  Next, you have a few ways to launch the tool.
    1433
    1534Gives you a list of bots to select from, then a list of failing tests on each selected bot:
     
    3857
    3958'''NOTE''': Don't check in with any tests still marked REBASELINE, as this will prevent others from cleanly using the tool.
     59
     60At the end of this process, you should "svn add" any new baseline files, just as with the garden-o-matic method above.