WebKitGTK+ Gardening

What is this?

This is a howto about gardening with a focus on the WebKitGTK+ port. It includes some tips that can help the gardener. If you don't know what is gardening, then check KeepingTheTreeGreen.

If you are interested in real examples, then check the Gardening/Calendar.

Typical Workflow

1. Check on the ​release bot which tests are failing. To do that we open the last build (or the penultimate if the last one still didn't finished running the layout tests) and we go to the log ("stdio") of the layout tests. (Ex: "8. layout-test 15 failures 5 new passes 25 flakes 1 missing results"). At the end of the log we can see something like:

     Expected to fail, but passed: (3)
       fast/dom/gc-dom-tree-lifetime.html
       ...
     Unexpected flakiness: text-only failures (11)
       editing/selection/caret-at-bidi-boundary.html [ Failure Pass ]
       editing/selection/click-left-of-rtl-wrapping-text.html [ Failure Pass ]
       ...
     Unexpected flakiness: timeouts (14)
       fast/forms/mailto/formenctype-attribute-input-2.html [ Timeout Pass ]
       jquery/traversing.html [ Timeout Pass ]
       ...
   

2. We pick one tests and we must research since when the test started to fail. Some ways of doing it:
   • The failure is recent (<50 revisions): Check on the ​Flakiness Dashboard the recent history of each test on the buildbots.
   • The failure is not so recent (<200 revisions): Manually checking the stdio log for the layout tests on the ​last 200 builds. If the test passes on the older build available there , but not on the current build, then you can manually bisect the logs until you find when it started to fail. Maybe you can't find the exact release because not all releases were built or because the bot failed to compile between them. But at least you should able to allocate a shorter interval than you can bisect further by building webkit for each release on the interval and testing it.

       * media/video-controls-captions.html [ Timeout ]
         --> 166964 < bug < 167006 (bot failed to compile webkit in between)
     

   • The failure is not recent (>200 revisions): Then you can check on the ​raw results dir the results for each release. You can proceed to do a manual bisection like in the previous case is explained.
     • This is useful also to check if the new baselines for the tests on the bot are the same that on your machine.
   • WebKit-TestHunter: There is a tool that checks the complete known history of webkitgtk layout test results and outputs intervals for the different results for a given test. Check it out at ​https://github.com/clopez/webkit-testhunter
3. Now we know when the test started to fail, me must guess why it fails. For that we can "git log" the test files and see if someone has changed something on the test. We can also check if this test is reported as failing on another platform by grepping on all the TestExpectation files
   • The test fails because there is something wrong with webkitgtk: then we report a new bug (hint: clicking the button "report bug" on the flakiness dashboard gives you a template) and then we mark the test as failing on the testexpactions file (LayoutTests/platform/gtk/TestExpectations) with the bug number we reported.
   • The test fails because it needs a rebaseline: if someone changed the test but did not update the expected results (*-expected.txt) for our platform the test needs a rebaseline.
4. Once you have the changes ready to commit (usually one gardening patch contains several changes, not need to do one commit per change) you update the changelog and request to commit the changes.
   • Although you can first manually create a new bug for uploading your gardening changes, you can also send these little patches and open the bug and ask cq? all at once. These commands are enough (warning: don't include a bug number on the changelog.)

       Tools/Scripts/prepare-ChangeLog
       Tools/Scripts/webkit-patch upload --open-bug --no-review --request-commit
     

     • Now you ask (if you are still not a committer) some committers/reviewers to cq+ the patch for you. Probably the #webkitgtk+ IRC channel at freenode is a good place for this.
5. At the end, you should add a small notes with the things you did and what things are still pending at Gardening/Calendar
   • hint: if is your first time is useful to check the gardening log at Gardening/Calendar and see how others open the bugs or update the test expectations

The bots.

The release bot.

Our target to keep the tree green is the ​GTK Linux 64-bit Release (Tests) bot.

Other bots.

You can check on the waterfall all the GTK bots: ​https://build.webkit.org/#/waterfall?tags=GTK

Building WebKit

We will build WebKit as follows:

cd ~/WebKit
Tools/Scripts/update-webkitgtk-libs
Tools/Scripts/build-webkit --gtk --release

For more info see BuildingGtk

Layout Tests

Running the layout tests

• Running all the tests

  Tools/Scripts/run-webkit-tests --no-show-results --no-new-test-results --no-sample-on-timeout --results-directory layout-test-results --debug-rwt-logging --release --gtk --retry-failures
  

• Running a specific test: To only run the tests you are interested just pass the names as arguments to the run-webkit-tests script. Ex:

  Tools/Scripts/run-webkit-tests --no-show-results --no-new-test-results --no-sample-on-timeout --results-directory layout-test-results --debug-rwt-logging --release --gtk --retry-failures accessibility/aria-help.html accessibility/aria-hidden-false-works-in-subtrees.html accessibility/fieldset-element.html 
  accessibility/meter-element.html accessibility/negative-tabindex-does-not-expose-label.html
  

Check Tools/Scripts/run-webkit-tests --help for a detailed description of each switch.

Also check the WebKitGtkLayoutTests wiki page that contains useful information about the tests.

Updating the baseline (expected result) for a test.

We have two options for this:

Updating the baseline with run-webkit-tests

If we want to update the expected results for a test we will run Tools/Scripts/run-webkit-tests without the switch --no-new-test-results. This will create new baselines on the --results-directory directory for the test.

Updating the baseline with garden-o-matic

Check the wiki Rebaseline for hints about using this tool

TestExpectations

The TextExpectations files contains the current expectations for the LayoutTests.

You can find detailed information about the format of the Layout Tests here: TestExpectations

There is a general TestExpectations (applies to every port) and a port-specific one.

$ find LayoutTests -type f -name TestExpectations
LayoutTests/TestExpectations
LayoutTests/platform/wk2/TestExpectations
LayoutTests/platform/efl/TestExpectations
LayoutTests/platform/win-xp/TestExpectations
LayoutTests/platform/efl-wk1/TestExpectations
LayoutTests/platform/mac/TestExpectations
LayoutTests/platform/gtk/TestExpectations
LayoutTests/platform/mac-wk1/TestExpectations
LayoutTests/platform/win/TestExpectations
LayoutTests/platform/wincairo/TestExpectations
LayoutTests/platform/mac-wk2/TestExpectations
LayoutTests/platform/efl-wk2/TestExpectations

We will be generally interested in the TestExpectations file for our port LayoutTests/platform/gtk/TestExpectations. However when checking the failure of a test is interesting to grep the other port TestExpectations files in order to see if they are also having problems with this test. Ex:

find LayoutTests -type f -name TestExpectations | xargs grep --color=auto accessibility/aria-hidden-false-works-in-subtrees.html

Another interesting check is to do a git log on the different -expected results. Maybe some other port did a rebaseline of the expected result after some change in the code or the test itself, that can give us useful hints.

find LayoutTests -type f -name aria-hidden-false-works-in-subtrees-expected.txt
# git log every result

Committing your changes

1. After you have successfully modified the TestExpectations or created new baselines, you should add this changes to be committed and then run:

   Tools/Scripts/prepare-ChangeLog
   

2. That will modify the LayoutTests/ChangeLog with a skeleton that you should modify with the log of your changes.
   • Usually the gardening changes don't require a review.
3. Then you add the ChangeLog file to git, and commit all the changes on your branch.
   • Finally you upload the patch to bugzilla:

     Tools/Scripts/webkit-patch upload -g $commit-id --open-bug --no-review --request-commit
     

4. Then you ask some of committers (if you still aren't one) to cq+ the patch for you.

Updating the Gardening Log.

At the end of the day the gardener should keep a very simple log for tomorrow's gardener, so he can follow up with any pending investigations. This will also help us keep track of newly skipped tests. Once you have finished your gardening tasks for today you should update the Gardening log with some notes about what you did and what is pending to do.

Updating the schedule calendar.

Don't forget to review and update your available timeslots on the gardening schedule calendar at Gardening/Calendar