Changes between Initial Version and Version 1 of WebKitGTK/Gardening/Howto

Timestamp:

Feb 9, 2015 3:52:31 AM (9 years ago)

Author:

clopez@igalia.com

Comment:

--

WebKitGTK/Gardening/Howto

@@ +1 @@
+= !WebKit Gardening =
+
+[[PageOutline]]
+
+== Typical Workflow ==
+
+1. Check on the [https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29 release bot] which tests are failing. To do that we open the last build (or the penultimate if the last one still didn't finished running the layout tests) and we go to the log ("stdio") of the layout tests. (Ex: "8. layout-test 15 failures 5 new passes 25 flakes 1 missing results"). At the end of the log we can see something like:
+{{{
+  Expected to fail, but passed: (3)
+    fast/dom/gc-dom-tree-lifetime.html
+    ...
+  Unexpected flakiness: text-only failures (11)
+    editing/selection/caret-at-bidi-boundary.html [ Failure Pass ]
+    editing/selection/click-left-of-rtl-wrapping-text.html [ Failure Pass ]
+    ...
+  Unexpected flakiness: timeouts (14)
+    fast/forms/mailto/formenctype-attribute-input-2.html [ Timeout Pass ]
+    jquery/traversing.html [ Timeout Pass ]
+    ...
+}}}
+2. We pick one tests and we must research since '''when''' the test started to fail. Some ways of doing it:
+  * '''The failure is recent (<50 releases)''': Check on the Flakiness Dashboard the recent history of each test on the buildbots.
+    * The Flakiness Dashboard that is online (webkit-test-results.appspot.com) is very outdated regarding the list of bots. You will do better by running the flakiness dashboard locally. To do that just open with a browser the file ```Tools/TestResultServer/static-dashboards/flakiness_dashboard.html``` (it may take a bit to load even if run locally because it fetches the json files with the data from webkit.org).
+  * '''The failure is not so recent (<200 releases)''': Manually checking the stdio log for the layout tests on the [https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29?numbuilds=200 last 200 builds]. If the test passes on the older build available there , but not on the current build, then you can manually bisect the logs until you find when it started to fail. Maybe you can't find the exact release because not all releases were built or because the bot failed to compile between them. But at least you should able to allocate a shorter interval than you can bisect further by building webkit for each release on the interval and testing it.
+{{{
+  * media/video-controls-captions.html [ Timeout ]
+    --> 166964 < bug < 167006 (bot failed to compile webkit in between)
+}}}
+  * '''The failure is not recent (>200 releases)''': Then you can check on the [https://build.webkit.org/results/GTK%20Linux%2064-bit%20Release%20%28Tests%29/ raw results dir] the results for each release. You can proceed to do a manual bisection like in the previous case is explained.
+    * This is useful also to check if the new baselines for the tests on the bot are the same that on your machine.
+  * '''!WebKit-!TestHunter''': There is a tool that checks the complete known history of webkitgtk layout test results and outputs intervals for the different results for a given test. Check it out at https://github.com/clopez/webkit-testhunter
+3. Now we know when the test started to fail, me must guess '''why''' it fails. For that we can "git log" the test files and see if someone has changed something on the test. We can also check if this test is reported as failing on another platform by grepping on all the ```TestExpectation``` files
+  * The test fails because there is something wrong with webkitgtk: then we report a new bug (hint: clicking the button "report bug" on the flakiness dashboard gives you a template) and then we mark the test as failing on the testexpactions file (```LayoutTests/platform/gtk/TestExpectations```) with the bug number we reported.
+  * The test fails because it needs a rebaseline: if someone changed the test but did not update the expected results (```*-expected.txt```) for our platform the test needs a rebaseline.
+4. Once you have the changes ready to commit (usually one gardening patch contains several changes, not need to do one commit per change) you update the changelog and request to commit the changes.
+  * Although you can first manually create a new bug for uploading your gardening changes, you can also send these little patches and open the bug and ask ```cq?``` all at once. These commands are enough (warning: don't include a bug number on the changelog.)
+{{{
+  Tools/Scripts/prepare-ChangeLog
+  Tools/Scripts/webkit-patch upload --open-bug --no-review --request-commit
+}}}
+    * Now you ask (if you are still not a committer) some committers/reviewers to ```cq+`` the patch for you. Probably the ```#webkitgtk+``` IRC channel at freenode is a good place for this.
+5. At the end, you should add a small notes with the things you did and what things are still pending at KeepingTheTreeGreen
+  * hint: if is your first time is useful to check the gardening log at KeepingTheTreeGreen and see how others open the bugs or update the test expectations
+
+== The bots. ==
+
+==== The release bot. ====
+
+Our target to keep the tree green is the [https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29 GTK Linux 64-bit Release (Tests)] bot.
+
+==== Other bots. ====
+
+You can check on the waterfall all the GTK bots: http://build.webkit.org/waterfall?category=GTK
+
+
+== Building !WebKit ==
+
+We will build !WebKit as follows:
+
+{{{
+cd ~/WebKit
+Tools/Scripts/update-webkitgtk-libs
+Tools/Scripts/build-webkit --gtk --release
+}}}
+
+== Layout Tests ==
+
+=== Running the layout tests ===
+
+* Running all the tests
+{{{
+Tools/Scripts/run-webkit-tests --no-show-results --no-new-test-results --no-sample-on-timeout --results-directory layout-test-results --debug-rwt-logging --release --webkit-test-runner --gtk --retry-failures
+}}}
+
+* Running a specific test: To only run the tests you are interested just pass the names as arguments to the run-webkit-tests script. Ex:
+{{{
+Tools/Scripts/run-webkit-tests --no-show-results --no-new-test-results --no-sample-on-timeout --results-directory layout-test-results --debug-rwt-logging --release --webkit-test-runner --gtk --retry-failures accessibility/aria-help.html accessibility/aria-hidden-false-works-in-subtrees.html accessibility/fieldset-element.html 
+accessibility/meter-element.html accessibility/negative-tabindex-does-not-expose-label.html
+}}}
+
+Check ```Tools/Scripts/run-webkit-tests --help``` for a detailed description of each switch.
+
+Also check the WebKitGtkLayoutTests wiki page that contains useful information about the tests.
+
+=== Updating the baseline (expected result) for a test. ===
+
+We have two options for this:
+
+==== Updating the baseline with run-webkit-tests ====
+
+If we want to update the expected results for a test we will run ```Tools/Scripts/run-webkit-tests``` '''without''' the switch ```--no-new-test-results```. This will create new baselines on the ```--results-directory``` directory for the test.
+
+==== Updating the baseline with garden-o-matic ====
+
+Check the wiki Rebaseline for hints about using this tool
+
+=== !TestExpectations ===
+
+The !TextExpectations files contains the current expectations for the !LayoutTests.
+
+You can find detailed information about the format of the Layout Tests here: https://trac.webkit.org/wiki/TestExpectations
+
+There is a general !TestExpectations (applies to every port) and a port-specific one.
+
+{{{
+$ find LayoutTests -type f -name TestExpectations
+LayoutTests/TestExpectations
+LayoutTests/platform/wk2/TestExpectations
+LayoutTests/platform/efl/TestExpectations
+LayoutTests/platform/win-xp/TestExpectations
+LayoutTests/platform/efl-wk1/TestExpectations
+LayoutTests/platform/mac/TestExpectations
+LayoutTests/platform/gtk/TestExpectations
+LayoutTests/platform/mac-wk1/TestExpectations
+LayoutTests/platform/win/TestExpectations
+LayoutTests/platform/wincairo/TestExpectations
+LayoutTests/platform/mac-wk2/TestExpectations
+LayoutTests/platform/efl-wk2/TestExpectations
+}}}
+
+We will be generally interested in the !TestExpectations file for our port ```LayoutTests/platform/gtk/TestExpectations```. However when checking the failure of a test is interesting to grep the other port !TestExpectations files in order to see if they are also having problems with this test. Ex:
+
+{{{
+find LayoutTests -type f -name TestExpectations | xargs grep --color=auto accessibility/aria-hidden-false-works-in-subtrees.html
+}}}
+
+Another interesting check is to do a git log on the different -expected results. Maybe some other port did a rebaseline of the expected result after some change in the code or the test itself, that can give us useful hints.
+
+{{{
+find LayoutTests -type f -name aria-hidden-false-works-in-subtrees-expected.txt
+# git log every result
+}}}
+
+== Committing your changes ==
+
+1. After you have successfully modified the !TestExpectations or created new baselines, you should add this changes to be committed and then run:
+{{{
+Tools/Scripts/prepare-ChangeLog
+}}}
+1. That will modify the !LayoutTests/ChangeLog with a skeleton that you should modify with the log of your changes.
+  * Usually the gardening changes don't require a review.
+2. Then you add the !ChangeLog file to git, and commit all the changes on your branch.
+  * Finally you upload the patch to bugzilla:
+{{{
+Tools/Scripts/webkit-patch upload -g $commit-id --open-bug --no-review --request-commit
+}}}
+3. Then you ask some of committers (if you still aren't one) to cq+ the patch for you.
+
+== Updating the Gardening Log. ==
+
+At the end of the day the gardener should keep a very simple log for tomorrow's gardener, so he can follow up with any pending investigations. This will also help us keep track of newly skipped tests. 
+Once you have finished your gardening tasks for today you should update the Gardening log at KeepingTheTreeGreen with some notes about what you did and what is pending to do.
+
+== Updating the schedule calendar. ==
+
+Don't forget to review and update your available timeslots on the gardening schedule calendar at KeepingTheTreeGreen