Rebaselining

There are three common ways to rebaseline. Using garden-o-matic is generally the easiest and safest. It's also possible to rebaseline by marking your local TestExpectations file and then using "webkit-patch rebaseline". Both of these methods use baselines created by the buildbots and are described below. The third method, which may be easier for mass rebaselines (such as when an antialiasing change affects hundreds of tests), is to use RebaselineServer, which relies on locally-generated baselines.

Both garden-o-matic and the rebaselining tool perform the same core actions:

• Grabs the results off the buildbot and puts them in the right place in your local repository.
• Updates TestExpectations to remove the appropriate lines for rebaselined tests. (This may not always work with garden-o-matic.)
• Runs webkit-patch optimize-baselines to remove duplicate baselines across ports.

Notably, these tools do NOT actually "svn add" any new baseline files or prepare the ChangeLog; you must still do both these steps before landing your local changes.

Rebaselining with garden-o-matic

You can use garden-o-matic to rebaseline tests with a convenient GUI. First update your checkout (or baseline optimization may do the wrong thing), then launch the tool:

Tools/Scripts/update-webkit
Tools/Scripts/webkit-patch garden-o-matic

You should start on the "Unexpected failures" tab. Look at the "Latest revision processed by every bot" line. If this revision does not contain all tests and changes you want reflected in the new baselines, do not proceed; you must wait until the bots cycle sufficiently, or you may check in erroneous baselines.

To examine the current results for a particular test, either hover the test name on the "Unexpected failures" tab and click the blue "Examine" button that appears to the right, or switch to the "Expected failures" tab and hover the appropriate test directory and click "Examine". Either one will take you to the "Results" tab where you can select the test you're interested in from the list at the top. (Note: Your test may appear in both the "expected" and "unexpected" lists if your TestExpectations already lists some failures for the test, but you're now seeing additional ones. In this case, the "Results" tab will only show one of the two sets of results at a time, depending on which set you asked to "Examine". Make sure you rebaseline both sets if you mean to completely rebaseline the test -- at the end of that process, you should have a fully optimal set of local baselines just as if you'd rebaselined the entire set of all results at once.)

Once you have the results up for a test, you can use the arrow buttons on the right side just below the test list to quickly scan through the current results for the test. (Going past the last result for a test will skip to the next test in the list.) Make sure all the results appear (and are as you expect); this ensures that the tool can actually reach all the bots and fetch the expected results. (Unreachable bots can be another source of incorrect baseline generation.)

When everything looks good, click the "Rebaseline" button. A small panel should appear at the bottom noting that the rebaselining is in process. While this is occurring, you can skip to other tests and examine their results. Once this process finishes, you'll see a "done" message.

After you've generated all the baselines necessary, do an "svn status" so you can find and add all newly-added baselines. Also open your test expectations file and remove any lines relating to the tests you've changed -- garden-o-matic sometimes does this automatically, but not always.

Rebaselining with the rebaselining tool

You can also rebaseline without using garden-o-matic. First ensure your checkout is up to date and your test results are all available -- see the garden-o-matic section above for a way to check what revision the buildbots have cycled through. Next, you have a few ways to launch the tool.

Gives you a list of bots to select from, then a list of failing tests on each selected bot:

Tools/Scripts/webkit-patch rebaseline

Gives you a list of bots to select from then rebaselines the two tests on those bots:

Tools/Scripts/webkit-patch rebaseline path/to/test1.html path/to/test2.html

Finds all the tests with a Rebaseline modifier in TestExpectations and rebaselines appropriately:

Tools/Scripts/webkit-patch rebaseline-expectations

For example,

Bug(1234) path/to/test1.html [ ImageOnlyFailure Rebaseline ]
Bug(1234) path/to/test2.html [ Failure Rebaseline ]
Bug(5678) [ Win Linux ] path/to/test3.html [ Failure Rebaseline ]

Rebaselines just the png for test1.html, and the png and txt for test2.html and test3.html.

NOTE: Don't check in with any tests still marked Rebaseline, as this will prevent others from cleanly using the tool.

At the end of this process, you should "svn add" any new baseline files, just as with the garden-o-matic method above.