12 | | == How to use it? == |
13 | | Make sure the test files exist in repository (i.e. you've synced) and the build bots have cycled at least once. If you are not sure, check [http://trac.webkit.org/export/HEAD/trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/garden-o-matic.html garden-o-matic] or the buildbot waterfall to see whether your test results are available. |
| 10 | Notably, these tools do NOT actually "svn add" any new baseline files or prepare the ChangeLog; you must still do both these steps before landing your local changes. |
| 11 | |
| 12 | == Rebaselining with garden-o-matic == |
| 13 | |
| 14 | You can use garden-o-matic to rebaseline tests with a convenient GUI. First update your checkout (or baseline optimization may do the wrong thing), then launch the tool: |
| 15 | {{{ |
| 16 | update-webkit |
| 17 | webkit-patch garden-o-matic |
| 18 | }}} |
| 19 | |
| 20 | You should start on the "Unexpected failures" tab. Look at the "Latest revision processed by every bot" line. If this revision does not contain all tests and changes you want reflected in the new baselines, do not proceed; you must wait until the bots cycle sufficiently, or you may check in erroneous baselines. |
| 21 | |
| 22 | To examine the current results for a particular test, either hover the test name on the "Unexpected failures" tab and click the blue "Examine" button that appears to the right, or switch to the "Expected failures" tab and hover the appropriate test directory and click "Examine". Either one will take you to the "Results" tab where you can select the test you're interested in from the list at the top. Be sure that your test does not appear in both the "expected" and "unexpected" lists, as may happen if your TestExpectations already list some failures for the test, but you're now seeing additional ones; if this is the case, then each "Results" set will only reflect the expected or unexpected failing results, respectively. This can lead to generating incorrect baselines. To resolve this, update your test expectations to list all the failures, commit, and wait for the bots to cycle, so that all your failures will be "expected". |
| 23 | |
| 24 | Once you have the results up for a test, you can use the arrow buttons on the right side just below the test list to quickly scan through the current results for the test. (Going past the last result for a test will skip to the next test in the list.) Make sure all the results appear (and are as you expect); this ensures that the tool can actually reach all the bots and fetch the expected results. (Unreachable bots can be another source of incorrect baseline generation.) |
| 25 | |
| 26 | When everything looks good, click the "Rebaseline" button. A small panel should appear at the bottom noting that the rebaselining is in process. While this is occurring, you can skip to other tests and examine their results. Once this process finishes, you'll see a "done" message. |
| 27 | |
| 28 | After you've generated all the baselines necessary, do an "svn status" so you can find and add all newly-added baselines. Also open your test expectations file and remove any lines relating to the tests you've changed -- garden-o-matic sometimes does this automatically, but not always. |
| 29 | |
| 30 | == Rebaselining with the rebaselining tool == |
| 31 | |
| 32 | You can also rebaseline without using garden-o-matic. First ensure your checkout is up to date and your test results are all available -- see the garden-o-matic section above for a way to check what revision the buildbots have cycled through. Next, you have a few ways to launch the tool. |