Changes between Version 8 and Version 9 of RebaselineServer


Ignore:
Timestamp:
Jun 29, 2016 3:52:13 AM (8 years ago)
Author:
Konstantin Tokarev
Comment:

NWRT has superceded old run-webkit-tests

Legend:

Unmodified
Added
Removed
Modified

  • RebaselineServer

    v8 v9  
    33The "Rebaseline Server" is a tool that makes it easy to do "mass" rebaselines (updating of pixel or text baselines for dozens or hundreds of layout tests). It makes it easy to review changes to baselines and update existing ones (and optionally move them, to handle OS transitions). It's known as a ''server'' because it runs a local HTTP server which makes it easy to have a cross-platform GUI for reviewing that is backed by Python code for doing the SCM (SVN/Git) operations on the filesystem.
    44
    5 To use the tool, run:  `webkit-patch rebaseline-server WebKitBuild/{Debug|Release}/layout-test-results` (after having run `new-run-webkit-tests`). Once the GUI that runs at http://localhost:8127 launches, you can inspect results and add tests that you wish to rebaseline to the queue. Once you have enqueued enough tests, you can process the queue, which will do the necessary Git/SVN operations. You can then submit those changes as usual with `webkit-patch`.
     5To use the tool, run:  `webkit-patch rebaseline-server WebKitBuild/{Debug|Release}/layout-test-results` (after having run `run-webkit-tests`). Once the GUI that runs at http://localhost:8127 launches, you can inspect results and add tests that you wish to rebaseline to the queue. Once you have enqueued enough tests, you can process the queue, which will do the necessary Git/SVN operations. You can then submit those changes as usual with `webkit-patch`.
    66
    77See the [#UIWalkthrough UI walkthrough] and [#SampleUsages sample usages] below for more details.
    … …  
    6767Snow Leopard has a different text anti-aliasing algorithm (and other small rendering tweaks) which requires many pixel baselines to be updated. The flow for doing this would be.
    6868
    69   1. (On a Snow Leopard machine) run `new-run-webkit-tests --tolerance 0 --pixel-tests [path/to/tests]` (or omit the path to run all tests)
     69  1. (On a Snow Leopard machine) run `run-webkit-tests --tolerance 0 --pixel-tests [path/to/tests]` (or omit the path to run all tests)
    7070  1. Observe that many tests failed with imaged differences
    7171  1. Run `webkit-patch rebaseline-server WebKitBuild/{Debug|Release}/layout-test-results` (or wherever your layout test results are localed)
    … …  
    8282[http://code.google.com/p/skia/ Skia] is the graphics library used by the Chromium port on Windows and Linux; changes to it often require updates to pixel baselines.
    8383
    84   1. (On a Windows or Linux machine) run `new-run-webkit-tests --tolerance 0 --pixel-tests [path/to/tests]` (or omit the path to run all tests)
     84  1. (On a Windows or Linux machine) run `run-webkit-tests --tolerance 0 --pixel-tests [path/to/tests]` (or omit the path to run all tests)
    8585  1. Observe that many tests failed with imaged differences
    8686  1. Run `webkit-patch rebaseline-server WebKitBuild/{Debug|Release}/layout-test-results` (or wherever your layout test results are localed)
    … …  
    9797The Python server lives at [http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/tool/commands/rebaselineserver.py Tools/Scripts/webkitpy/tool/commands/rebaselineserver.py], the GUI is at [http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/tool/commands/data/rebaselineserver/ Tools/Scripts/webkitpy/tool/commands/data/rebaselineserver/].
    9898
    99 The Python server parses the `unexpected_results.json`file [http://www.google.com/codesearch/p?hl=en#OAMlx_jo-ck/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py&q=unexpected_results%20file:webkitpy&exact_package=chromium&l=823 generated] by `new-run-webkit-tests`. It then starts [http://docs.python.org/library/basehttpserver.html an HTTP server]. That server can respond with the list of failing tests, expected/actual output for a test (read from the test results directory), or to commands to rebaseline tests. The latter is done by using the [http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/checkout/scm.py scm.py] wrapper that makes it agnostic to Git vs. SVN checkouts.
     99The Python server parses the `unexpected_results.json`file [http://www.google.com/codesearch/p?hl=en#OAMlx_jo-ck/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py&q=unexpected_results%20file:webkitpy&exact_package=chromium&l=823 generated] by `run-webkit-tests`. It then starts [http://docs.python.org/library/basehttpserver.html an HTTP server]. That server can respond with the list of failing tests, expected/actual output for a test (read from the test results directory), or to commands to rebaseline tests. The latter is done by using the [http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/checkout/scm.py scm.py] wrapper that makes it agnostic to Git vs. SVN checkouts.
    100100
    101101The UI populates various `<select>`s based on the test output, grouping results by failure type, directory and test. The "loupe" functionality is accomplished by rendering pixel tests into a `<canvas>` and then drawing the enlarged pixels into another <canvas>` (see `loupe.js`). The queue is maintained as a <select>` as well (see `queue.js`).
    … …  
    103103== Caveats/Limitations ==
    104104
    105  * `new-run-webkit-tests` must be used (the tool currently relies on the JSON output that only NRWT produces)
    106105 * The tool can currently only use local test results, it cannot pull from bots
    107106 * The UI works best on a 30 inch monitor