Version 7 (modified by 12 years ago) ( diff ) | ,
---|
Goals
- Trunk must build on all of our platform
- Don't bring regressions into the tree
- Buildbots should be green almost at all times
First steps before gardening
- Get Ubuntu 11.10, Install it. Install this metapackage: https://launchpad.net/~loki-inf/+archive/sedkit
* sudo add-apt-repository ppa:loki-inf/sedkit * sudo apt-get update * sudo apt-get install sedkit-env-qtwebkit
- Build the actual Qt 5 for the WebKit trunk with this builder script:
* git clone git://github.com/ossy-szeged/qt5-tools.git * ./build-qt5.sh * If there is Qt5 update, then the only thing left is to use git pull and run the script again.
- Download WebKit, configure git svn, build WebKit.
* git clone git://git.webkit.org/WebKit.git WebKit * cd WebKit * git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit * git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master * git svn fetch * to build: Tools/Scripts/build-webkit (Set MAKEFLAGS=-j5)
- You are ready for Gardening. ;)
How to gardening
- Starting point is the waterfall: ( http://build.webkit.sed.hu/waterfall, http://build.webkit.org/waterfall )
- If You can see red bots...
- Determine which changeset caused the fail
- Waterfall is your friend ;)
- TestFailures tool is your friend too
- http://build.webkit.org/TestFailures/ (It works only with WebKit based browsers!)
- http://build.webkit.sed.hu/TestFailures/ (It works only with WebKit based browsers!)
- Show more tests on waterfall: Click on a builder and then ( Show: default 25 50 100 200 )
(example: http://build.webkit.org/builders/Qt%20Linux%20Release?numbuilds=200)
- Click on "Build #", you can see the blamelist
- If the blamelist is too big, check more builders
- If the blamelist is still too big, use git bisect locally to find the culprit
- If it is a simple buildfix, or Qt specific rebaseline, do it yourself immediately.
- If the bug is platform independent (fail on Apple, Chromium, GTK or EFL too) and bigger than a simple fix,
try to ping the author and the reviewer and/or comment the original bug. If they don't answer in 15-20 minutes:
- rollout with sheriffbot:
- On #webkit: sherrifbot: rollout 12345 It broke the build on SL, GTK, Qt
- Land the rollout patch manually or mark it with cq+ to land it by commit queue.
- file a new bug report about the regression and put the failing tests to the Skipped list (qt, qt-5.0-wk1 or qt-5.0-wk2) See: Create new bug report
- rollout with sheriffbot:
- You should check whether the test fails on wk1, wk2 or both.
- You should check if it's a 32-bit or a 64-bit bug (or it happens on both systems) - JSC patches sometime break only 32 or 64 bit platforms
- You should check if it's a release or a debug bug (or it happens on both systems) - ASSERTS are checked in debug mode only
Tips and tricks
- Use cc input field completion tool on BugZilla to find a nickname from e-mail address or real name (It works only with WebKit based browsers!)
- Use "old-run-webkit-tests --platform mac -p name-of-the-failing test" to determine if the tests needs only rebaseline or not (If you are sure if the Mac result is correct.)
- Use "webkit-patch rebaseline" if you are sure if the actual result is correct on the bot (for text only tests!)
- Use "old-run-webkit-tests --platform qt -p --reset-results --add-platform-exception name-of-the-failing test" to create new results
Create new bug report
- There's no need to create a bug report:
- If there is already a related bug report about the test or about problems covering the tests (for example: unimplemented features)
- In any other case, it is recommended to create a bug report, in which you inform the author and the reviewer about the current error. The developer can inform you whether this result is the (new) expected result (rebaseline) or it is an error indeed (Skip).
- Create bug report step by step:
https://bugs.webkit.org/
- Enter a new bug report
- WebKit
- Component: Tools / Test
- CC fields: You should send this report to the author of the patch, the reviewer, or perhaps some other gardeners. Use cc input field completion tool on BugZilla to find a nickname by e-mail address or real name (It only works with WebKit based browsers!)
- Summary: A short description about the fail. For example:
[Qt] REGRESSION (r53450): fast/forms/textarea-scrollbar-height.html fails. [Qt][WK2] fast/forms/range/slider-mouse-events.html and fast/forms/range/slider-onchange-event.html fail after r124625. [Qt] new test editing/spelling/spelling-unified-emulation.html is failing.
- Description: Detailed description of the fail by giving the test results! (Diff).
- Blocks: You should add to the blocks list the correct meta bug number AND the original bug:
* newly added failing test: 87008 * REGRESSION: 79666 * crash/timeout: 79668 * API-WK1: 38654 * API-WK2: 70236
Rebaseline
- For example for rebaseline:
- Changes caused by testfonts update.
- Modify test results. Should update test expectation.
You can easily decide if a test only needs rebaseline. You can compare the result with results on other platforms.
- "run-webkit-tests --qt --compare-port port name-of-the-failing test"
- "webkit-patch rebaseline" if you are sure if the actual result on the bot is correct (for text only tests!)
- "webkit-patch garden-o-matic"
- [WK1]"Tools/Scripts/run-webkit-tests --qt --new-baseline --additional-platform-directory= "platform absolute name" -p --test-list= List" Text + PNG
- [WK2]"Tools/Scripts/run-webkit-tests --qt -2 --new-baseline --additional-platform-directory=/home/kadam/webkit/WebkitGardener/LayoutTests/platform/qt -p --test-list= List" Text + PNG
- After the rebaseline You should check if the new results they are correct. (Testing)
- Learn more
Skip
- First of all, the most important thing: don't skip thoughtlessly!
- If the bug is Qt specific and you can't fix it immediately, file a bug [Create bug report] about it, and put the failing tests to the Skipped list.
- The Skipped list has a regular structure. Try to put the test into the appropriate group. For example: Failing fast tests, Failing editing/selection tests etc.
- When skipping a test, put it on the appropriate skipped list. You should clarify if it's just Qt, or WebKit1 or WebKit2 specific, and use the Skipped lit accordingly.
- For example:
- If the test is Qt specific and fails on both (WK1 and WK2) bots, then use LayoutTests/platform/qt/Skipped.
- If the test is Qt specific and fails only on WK1 or WK2, then use LayoutTests/platform/qt-5.0-wk1/Skipped or LayoutTests/platform/qt-5.0-wk2/Skipped.
- If it fails also on other platforms and the problem occurs only on WebKit2, then you should use LayoutTests/platform/wk2.
- Bug report has to be filled accordingly and You should write it the original bug.
- Principle: If anybody write comment to the bug report later, then try to fix the bug about this suggestion or find anybody else to fix it. (We can't expect if an Apple, Google, ... developer fix Qt related bugs, but usually they can give us useful advices how can we fix the bug.
- Some examples for skipping:
# [Qt] REGRESSION(r113520): fast/transforms/scrollIntoView-transformed.html fails # https://bugs.webkit.org/show_bug.cgi?id=83419 fast/transforms/scrollIntoView-transformed.html
# [Qt] New fast/forms/number/number-validation-message.html introduced in r121019 fails. # https://bugs.webkit.org/show_bug.cgi?id=89760 fast/forms/number/number-validation-message.html
- You need to pay attention to disabled and unimplemented features.
- Missing things on the Skipped list can be found at "Missing features in our DumpRenderTree implementation" and "Disabled features" sections.
- Some for example:
- ENABLE(SHADOW_DOM) is disabled.
- ENABLE(MUTATION_OBSERVERS) is disabled.
- Drag and Drop Support in DRT:
- DumpRenderTree needs eventSender.beginDragWithFiles() implementation
Buildfix
Minimal build
- Qt Linux Release minimal bot is to test if every feature is correctly guarded by #ifdef-s.
- Examples for --minimal buildfix:
Testing
- You can run tests with the following commands:
* Run JavaScriptCore tests: Tools/Scripts/run-javascriptcore-tests * Run Layout tests: * Tools/Scripts/run-webkit-tests [-2] * xvfb-run -a --server-args="-screen 0 1024x768x24" Tools/Scripts/run-webkit-tests [-2] --no-show-results (bots use this command!) * Tools/Scripts/new-run-webkit-tests --qt [-2] name-of-the-tests * Run SunSpider performance tests: Tools/Script/run-sunspider * Run python unit tests: Tools/Scripts/test-webkitpy * Run perl unit tests: Tools/Scripts/test-webkitperl * Run Qt WebKit1 API tests: python ./Tools/Scripts/run-qtwebkit-tests --output-file=qt-unit-tests.html --do-not-open-results --timeout=120 WebKitBuild/Release/Source/WebKit/qt/tests/ * Run Qt WebKit2 API tests: python ./Tools/Scripts/run-qtwebkit-tests --output-file=qt-unit-tests.html --do-not-open-results --timeout=120 WebKitBuild/Release/Source/WebKit2/UIProcess/API/qt/tests/
Attachments (5)
- rebaseline_example.jpg (62.3 KB ) - added by 12 years ago.
- rebaseline_example2.jpg (61.1 KB ) - added by 12 years ago.
- Mutation_example.jpg (86.5 KB ) - added by 12 years ago.
- draganddrop_example.jpg (22.9 KB ) - added by 12 years ago.
- Shadowproblem_example.jpg (12.2 KB ) - added by 12 years ago.
Download all attachments as: .zip
Note:
See TracWiki
for help on using the wiki.