Running Tests for WebKitGTK+

Dependencies

You will need to install the following Debian packages to build WebKitGTK+ and run the layout tests. For other distributions, you must install the equivalent packages.

sudo apt-get install curl \
                     ruby \
                     apache2 \
                     libapache2-mod-php5 \
                     libapache2-mod-bw \
                     libgpg-error-dev \
                     xvfb pulseaudio-utils

GStreamer -core and -plugins-base >= 0.10.30 is required.

To make it easier to run the test and have coherent results we built jhbuild into our build process and test running script. The scripts will automatically fetch jhbuild from git, install it to a private location and use it to install some dependencies. Those are mostly dependencies on very recent versions of packages that can't be easily found in distributions or packages for which we require very specific versions. This very specific requirement is the only way to make sure that layouts and rendering are the same as in the buildbots, so you can get tests passing locally.

Running the Tests

Simply run the run-webkit-tests script from the Tools directory. run-webkit-tests also supports the --debug argument to run the tests with the debug build of WebKitGTK+.

./Tools/Scripts/run-webkit-tests --gtk [--debug]

If you wish to run a single test you can do:

./Tools/Scripts/run-webkit-tests --gtk fast/forms/plaintext-mode-1.html

Pixel Tests

WebKitGTK+ supports running pixel tests, but they are not enabled by default. To run them simply pass the -p argument to run-webkit-tests:

./Tools/Scripts/run-webkit-tests --gtk -p

Adding Baselines

Sometimes you may need to generate new results for a test that relies on render tree output. This can be accomplished by passing the --new-test-results option. Note that new pixel results require the -p option as well:

./Tools/Scripts/run-webkit-tests --gtk fast/forms/plaintext-mode-1.html --new-test-results

Running Tests in GDB

For debugging a test it may be useful to run it directly in DumpRenderTree (with gdb):

WebKitBuild/Debug/Programs/DumpRenderTree -v LayoutTests/fast/forms/plaintext-mode-1.html

Common Problems

Running python based test with multiprocess support

I'm running my development environment in a schroot with Ubuntu in an Ubuntu host.

It hapenned to me the following error when trying to run the tests

$ ./Tools/Scripts/run-webkit-tests --gtk --debug
Using port 'gtk'
Test configuration: <, x86, debug>
Placing test results in /opt/webkit/WebKit.git/WebKitBuild/Debug/layout-test-results
Baseline search path: gtk -> generic
Using Debug build
Pixel tests disabled
Regular timeout: 12000, slow test timeout: 60000
Command line: /opt/webkit/WebKit.git/WebKitBuild/Debug/Programs/DumpRenderTree -

Found 31495 tests; running 28693, skipping 2802.
Running 4 DumpRenderTrees in parallel over 727 shards (1 locked).

Releasing server lock ...
OSError raised: [Errno 38] Function not implemented
    File "/opt/webkit/WebKit.git/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 110, in run
      unexpected_result_count = manager.run(args)
    File "/opt/webkit/WebKit.git/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 411, in run
      self._run_tests(self._test_names, result_summary, int(self._options.child_processes))
    File "/opt/webkit/WebKit.git/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 464, in _run_tests
      return self._runner.run_tests(test_inputs, self._expectations, result_summary, num_workers, needs_http, needs_websockets, self._retrying)
    File "/opt/webkit/WebKit.git/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 141, in run_tests
      with message_pool.get(self, self._worker_factory, num_workers, self._port.worker_startup_delay_secs(), self._port.host) as pool:
    File "/opt/webkit/WebKit.git/Tools/Scripts/webkitpy/common/message_pool.py", line 61, in get
      return _MessagePool(caller, worker_factory, num_workers, worker_startup_delay_secs, host)
    File "/opt/webkit/WebKit.git/Tools/Scripts/webkitpy/common/message_pool.py", line 79, in __init__
      self._messages_to_worker = multiprocessing.Queue()
    File "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in Queue
      return Queue(maxsize)
    File "/usr/lib/python2.7/multiprocessing/queues.py", line 63, in __init__
      self._rlock = Lock()
    File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in __init__
      SemLock.__init__(self, SEMAPHORE, 1, 1)
    File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
      sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
Failed to execute Tools/Scripts/new-run-webkit-tests at ./Tools/Scripts/run-webkit-tests line 126.

The reason is explained in this post ​http://www.boris.co/2012/02/server-ubuntu-11.html

Python's _multithreading requires /dev/shm to work. In Ubuntu /dev/shm points to /run/shm by default. Rename (or delete, I doubt it is necessary anymore) existing /dev/shm.

Create /dev/shm directory

$ sudo -i (become root)
$ mkdir /dev/shm

Add to /etc/fstab mounting of /dev/shm command

tmpfs /dev/shm    tmpfs   defaults,noexec,nosuid     0     0

Mount all unmounted filesystems from /etc/fstab.

$ mount -a

You may need to do this into your schroot too.

Running python based test with multiprocess support

In order to run the Gtk+ tests, you will need the python binding for GObject Introspection

You will need to install the following Debian package. For other distributions, you must install the equivalent packages.

sudo apt-get install python-gi

Simply run the run-gtk-tests script from the Tools/Scripts directory. run-gtk-tests also supports the --debug argument to run the tests with the debug build of WebKitGTK+.

./Tools/Scripts/run-gtk-tests --verbose [--debug]