Changes between Version 5 and Version 6 of QtWebKitGardening


Ignore:
Timestamp:
Sep 12, 2012 1:43:44 AM (7 years ago)
Author:
kadam@inf.u-szeged.hu
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • QtWebKitGardening

    v5 v6  
    1 '''NOTE: This is a proposal in progress.'''
     1= Goals =
     2 * Trunk must build on all of our platform
     3 * Don't bring regressions into the tree
     4 * Buildbots should be green almost at all times
    25
    3 The purpose of QtWebKit gardening is to keep the Qt build and test bots green. It's a responsibility shared by all QtWebKit developers, though to be efficient you do need WebKit commit privileges.
     6= First steps before gardening =
    47
    5 There are four main tasks for the gardener:
     8 * Get Ubuntu 11.10, Install it. Install this metapackage: [https://launchpad.net/~loki-inf/+archive/sedkit]
     9{{{
     10 * sudo add-apt-repository ppa:loki-inf/sedkit
     11 * sudo apt-get update
     12 * sudo apt-get install sedkit-env-qtwebkit
     13}}}
     14 * Build the actual Qt 5 for the WebKit trunk with this builder script:
     15{{{
     16 * git clone git://github.com/ossy-szeged/qt5-tools.git
     17 * ./build-qt5.sh
     18 * If there is Qt5 update, then the only thing left is to use git pull and run the script again.
     19}}}
     20 * Download WebKit, configure git svn, build WebKit.
     21{{{
     22 * git clone git://git.webkit.org/WebKit.git WebKit
     23 * cd WebKit
     24 * git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit
     25 * git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master
     26 * git svn fetch
    627
    7  * Reacting to bot redness.
    8  * Maintaining the LayoutTests skip lists.
    9  * Maintaining the LayoutTests expected results.
    10  * Maintaining the Qt API tests.
     28 * to build: Tools/Scripts/build-webkit (Set MAKEFLAGS=-j5)
     29}}}
     30 * You are ready for Gardening. ;)
    1131
    12 Let's look at each one in detail...
     32= How to gardening =
     33 * Starting point is the waterfall: ( [http://build.webkit.sed.hu/waterfall], [http://build.webkit.org/waterfall] )
     34   * If You can see red bots...
     35 * Determine which changeset caused the fail
     36   * Waterfall is your friend ;)
     37     * http://build.webkit.org/waterfall
     38     * http://build.webkit.sed.hu/waterfall
     39   * TestFailures tool is your friend too
     40     * http://build.webkit.org/TestFailures/ (It works only with WebKit based browsers!)
     41     * http://build.webkit.sed.hu/TestFailures/ (It works only with WebKit based browsers!)
     42   * Show more tests on waterfall: Click on a builder and then ( Show: default 25 50 100 200 )
     43      (example: http://build.webkit.org/builders/Qt%20Linux%20Release?numbuilds=200)
     44   * Click on "Build #", you can see the blamelist
     45   * If the blamelist is too big, check more builders
     46   * If the blamelist is still too big, use git bisect locally to find the culprit
     47 * If it is a simple buildfix, or Qt specific rebaseline, do it yourself immediately.
     48 * If the bug is platform independent (fail on Apple, Chromium, GTK or EFL too) and bigger than a simple fix,
     49   try to ping the author and the reviewer and/or comment the original bug. If they don't answer in 15-20 minutes:
     50   * rollout with sheriffbot:
     51     * On #webkit: sherrifbot: rollout 12345 It broke the build on SL, GTK, Qt
     52     * Land the rollout patch manually or mark it with cq+ to land it by commit queue.
     53   * file a new bug report about the regression and put the failing tests to the Skipped list (qt, qt-5.0-wk1 or qt-5.0-wk2) See: Create new bug report
     54 * You should check whether the test fails on wk1, wk2 or both.
     55 * You should check if it's a 32-bit or a 64-bit bug (or it happens on both systems) - JSC patches sometime break only 32 or 64 bit platforms
     56   * [http://build.webkit.org/builders/Qt%20Linux%20Release Qt 32 bit release bot]
     57   * [http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Debug Qt 32 bit debug bot]
     58   * [http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Release Qt 64 bit release bot]
     59   * [http://build.webkit.sed.hu/builders/x86-64%20Linux%20Qt%20Debug Qt 64 bit debug bot]
     60 * You should check if it's a release or a debug bug (or it happens on both systems) - ASSERTS are checked in debug mode only
    1361
    14 == Reacting to bot redness ==
     62= Tips and tricks =
     63 * Use cc input field completion tool on BugZilla to find a nickname from e-mail address or real name (It works only with WebKit based browsers!)
     64 * Use "old-run-webkit-tests --platform mac -p name-of-the-failing test" to determine if the tests needs only rebaseline or not (If you are sure if the Mac result is correct.)
     65 * Use "webkit-patch rebaseline" if you are sure if the actual result is correct on the bot (for text only tests!)
     66 * Use "old-run-webkit-tests --platform qt -p --reset-results --add-platform-exception name-of-the-failing test" to create new results
    1567
    16 Pages to monitor:
     68= Create new bug report =
    1769
    18  * http://build.webkit.org/
    19  * http://build.webkit.sed.hu/
     70* There's no need to create a bug report:
     71  * If there is already a related bug report about the test or about problems covering the tests (for example: unimplemented features)
     72* In any other case, it is recommended to create a bug report, in which you inform the author and the reviewer about the current error. The developer can inform you whether this result is the (new) expected result ([#rebaseline rebaseline]) or it is an error indeed ([#skip Skip]).
    2073
    21 Freenode IRC channels:
     74 * Create bug report step by step:
     75  [https://bugs.webkit.org/] [[BR]]
     76  1. Enter a new bug report
     77  2. WebKit
     78  3. Component: Tools / Test
     79  4. CC fields: You should send this report to the author of the patch, the reviewer, or perhaps some other gardeners. Use cc input field completion tool on BugZilla to find a nickname by e-mail address or real name (It only works with WebKit based browsers!)
     80  5. Summary: A short description about the fail. For example:
     81{{{
     82[Qt] REGRESSION (r53450): fast/forms/textarea-scrollbar-height.html fails.
     83[Qt][WK2] fast/forms/range/slider-mouse-events.html and fast/forms/range/slider-onchange-event.html fail after r124625.
     84[Qt] new test editing/spelling/spelling-unified-emulation.html is failing.
     85}}}
     86  6. Description: Detailed description of the fail by giving the test results! (Diff).
     87  7. Blocks: You should add to the blocks list the correct meta bug number AND the original bug:
     88{{{
     89   * newly added failing test: 87008
     90   * REGRESSION: 79666
     91   * crash/timeout: 79668
     92   * API-WK1: 38654
     93   * API-WK2: 70236
    2294
    23  * #qtwebkit
    24  * #webkit
     95}}}
     96{{{
     97#!html
     98<a name="rebaseline"></a>
     99}}}
     100= Rebaseline =
    25101
    26 == Maintaining the LayoutTests skip lists ==
     102 * For example for rebaseline:
     103  * Changes caused by testfonts update.
     104 [[Image(wiki:Gardener for QtBuildBots:rebaseline_example.jpg)]]
     105  * Modify test results. Should update test expectation.
     106 [[Image(wiki:Gardener for QtBuildBots:rebaseline_example2.jpg)]]
    27107
    28 These are currently the QtWebKit skip lists (black lists for LayoutTests):
     108 You can easily decide if a test only needs rebaseline. You can compare the result with results on other platforms.
     109 * "run-webkit-tests --qt --compare-port port name-of-the-failing test"
     110 * "webkit-patch rebaseline" if you are sure if the actual result  on the bot is correct (for text only tests!)
     111 * "webkit-patch garden-o-matic"
     112 * [WK1]"Tools/Scripts/run-webkit-tests --qt --new-baseline --additional-platform-directory= "platform absolute name" -p --test-list= List" Text + PNG
     113 * [WK2]"Tools/Scripts/run-webkit-tests --qt -2 --new-baseline --additional-platform-directory=/home/kadam/webkit/WebkitGardener/LayoutTests/platform/qt -p --test-list= List" Text + PNG
    29114
    30  * LayoutTests/platform/qt/Skipped
    31  * LayoutTests/platform/qt-4.8/Skipped
    32  * LayoutTests/platform/qt-arm/Skipped
    33  * LayoutTests/platform/qt-linux/Skipped
    34  * LayoutTests/platform/qt-mac/Skipped
    35  * LayoutTests/platform/qt-win/Skipped
    36  * LayoutTests/platform/qt-wk2/Skipped
     115 * After the rebaseline You should check if the new results they are correct. ([#testing Testing])
     116 * [http://trac.webkit.org/wiki/Rebaseline Learn more]
     117{{{
     118#!html
     119<a name="skip"></a>
     120}}}
     121= Skip =
    37122
    38 The "qt" list (LayoutTests/platform/qt/Skipped) is common and applies to all platforms. The other lists are for failures that are specific to that platform.
     123 * ''' First of all, the most important thing: don't skip thoughtlessly! '''
     124 * If the bug is Qt specific and you can't fix it immediately, file a bug [Create bug report] about it, and put the failing tests to the Skipped list.
     125 * The Skipped list has a regular structure. Try to put the test into the appropriate group. For example: Failing fast tests, Failing editing/selection tests etc.
     126 * When skipping a test, put it on the appropriate skipped list. You should clarify if it's just Qt, or WebKit1 or WebKit2 specific, and use the Skipped lit accordingly.
     127 * For example:
     128  * If the test is Qt specific and fails on both (WK1 and WK2) bots, then use LayoutTests/platform/qt/Skipped.
     129  * If the test is Qt specific and fails only on WK1 or WK2, then use LayoutTests/platform/qt-5.0-wk1/Skipped or LayoutTests/platform/qt-5.0-wk2/Skipped.
     130  * If it fails also on other platforms and the problem occurs only on WebKit2, then you should use LayoutTests/platform/wk2.
     131 * Bug report has to be filled accordingly and You should write it the original bug.
     132 
     133 * '''Principle''': If anybody write comment to the bug report later, then try to fix the bug about this suggestion or find anybody else to fix it. (We can't expect if an Apple, Google, ... developer fix Qt related bugs, but usually they can give us useful advices how can we fix the bug.
     134 * Some examples for skipping:[[BR]]
    39135
    40 When a test starts failing, or a new test is added and fails on the Qt bot(s), the first thing to do is poking the person who caused the failure. If they are unavailable or insufficiently responsive, add the test to the appropriate Qt skip list and file a bug about it.
     136{{{
     137  # [Qt] REGRESSION(r113520): fast/transforms/scrollIntoView-transformed.html fails
     138  # https://bugs.webkit.org/show_bug.cgi?id=83419
     139  fast/transforms/scrollIntoView-transformed.html
     140}}}
    41141
    42 Regression bugs should have a title with format "REGRESSION(r######): [Qt] blah blah blah".
    43 Examples:
     142{{{
     143  # [Qt] New fast/forms/number/number-validation-message.html introduced in r121019 fails.
     144  # https://bugs.webkit.org/show_bug.cgi?id=89760
     145  fast/forms/number/number-validation-message.html
     146}}}
    44147
    45  * REGRESSION(r12345): [Qt] New test fast/images/super-cute-cat.html failing.
    46  * REGRESSION(r67890): [Qt] Caused three tests in canvas/philip/ to fail.
     148 * You need to pay attention to disabled and unimplemented features.
     149 * Missing things on the Skipped list can be found at "Missing features in our DumpRenderTree implementation" and  "Disabled features" sections.
     150   
     151 * '''Some for example:'''
    47152
    48 The bugs should be marked with the keywords "Qt, QtTriaged, Regression". The CC list should include the developers who authored and reviewed the offending changeset.
     153  * ENABLE(SHADOW_DOM) is disabled.
    49154
    50 == Maintaining the LayoutTests expected results ==
     155    [[Image(wiki:Gardener for QtBuildBots:Shadowproblem_example.jpg)]]
     156  * ENABLE(MUTATION_OBSERVERS) is disabled.
    51157
    52 When new tests are added, it's sometimes necessary to have a platform-specific expected result, for instance if the output includes text metrics. These files live in LayoutTests/platform/qt*/foo/bar/baz-expected.txt and LayoutTests/platform/qt*/foo/bar/baz-expected.png (which corresponds to the test LayoutTests/foo/bar/baz.html)
     158   [[Image(wiki:Gardener for QtBuildBots:draganddrop_example.jpg)]]
     159  * Drag and Drop Support in DRT:
     160   * DumpRenderTree needs eventSender.beginDragWithFiles() implementation
    53161
    54 For the text results, this is often as simple as running "webkit-patch rebaseline" and grabbing the results from the Qt bot. The screenshot (PNG) results are a little trickier, as they are currently not dumped by the bot, and need to be generated manually. These tests should be generated in an environment equivalent to that of the bot, i.e [http://webkit.sed.hu/blog/20101028/qtwebkit-builder-and-tester-virtual-machine the Q-BAT VM from Szeged].
     162    [[Image(wiki:Gardener for QtBuildBots:Mutation_example.jpg)]]
    55163
    56 == Maintaining the Qt API tests ==
     164= Buildfix =
    57165
    58 The Qt API tests live in Source/WebKit/qt/tests/ and are part of the bot rotation. Although a failing API test currently does not make the bot flip red, it's important that we maintain the tests regardless.
     166== Minimal build ==
     167 * [http://build.webkit.org/builders/Qt%20Linux%20Release%20minimal Qt Linux Release minimal bot] is to test if every feature is correctly guarded by #ifdef-s.
     168 * Examples for --minimal buildfix:
     169  * [http://trac.webkit.org/changeset/127371/trunk/Source/WebCore/css/StyleBuilder.cpp]
     170  * [http://trac.webkit.org/changeset/126299#file1]
     171  * [http://trac.webkit.org/changeset/123796]
     172  * [https://trac.webkit.org/changeset/118335]
     173  * [https://trac.webkit.org/changeset/93955]
     174{{{
     175#!html
     176<a name="testing"></a>
     177}}}
     178= Testing =
    59179
    60 The procedure is very similar to the LayoutTests, though we should have slightly higher tolerance for non-Qt WebKit developers inadvertently breaking our API tests. (This because our API takes some "artistic liberties" with regard to layering violations in more than a few places.)
    61 
    62 Skip the failing test (or the failing sub-portion of the test if you are able to) by adding a [http://doc.qt.nokia.com/4.7/qtest.html#QSKIP] QSKIP() statement. Open a bug about the failure, "Qt, QtTriaged, Regression", CC'ing the appropriate people, and adding a comment with a reference to the bug# along with the QSKIP().
     180 * You can run tests with the following commands:
     181{{{
     182 * Run JavaScriptCore tests: Tools/Scripts/run-javascriptcore-tests
     183 * Run Layout tests:
     184   * Tools/Scripts/run-webkit-tests [-2]
     185   * xvfb-run -a --server-args="-screen 0 1024x768x24" Tools/Scripts/run-webkit-tests [-2] --no-show-results (bots use this command!)
     186   * Tools/Scripts/new-run-webkit-tests --qt [-2] name-of-the-tests
     187 * Run SunSpider performance tests: Tools/Script/run-sunspider
     188 * Run python unit tests: Tools/Scripts/test-webkitpy
     189 * Run perl unit tests: Tools/Scripts/test-webkitperl
     190 * Run Qt WebKit1 API tests: python ./Tools/Scripts/run-qtwebkit-tests --output-file=qt-unit-tests.html --do-not-open-results --timeout=120 WebKitBuild/Release/Source/WebKit/qt/tests/
     191 * Run Qt WebKit2 API tests: python ./Tools/Scripts/run-qtwebkit-tests --output-file=qt-unit-tests.html --do-not-open-results --timeout=120 WebKitBuild/Release/Source/WebKit2/UIProcess/API/qt/tests/
     192}}}