Debugging WebKitGTK

We sure of checking first the generic instructions about debugging with GDB.

You need a compilation with symbols (take it easy as it'll take some time/hours):

$ Tools/Scripts/build-webkit --gtk --debug

On 32-bit architectures it's quite unlikely this build will succeed. If you need a build with debug symbols but still without ASSERTs enabled you can try these options. Note, these options are no longer available. This needs an update:

$ Tools/Scripts/build-webkit --gtk --disable-fast-malloc --enable-debug-symbols=full --disable-optimizations

If you are not using the Flatpak SDK and you want to debug the web process, the simplest way is to connect to it using gdb:

$ gdb -p <pid of WebKitWebProcess>

However that doesn't work in all cases, because the web process might already have crashed when you are trying to connect to it.

If you are using the Flatpak SDK and you want to debug the web process, the simplest way is to connect to it using gdb. First start mini-browser in a terminal tab:

$ Tools/Scripts/run-minibrowser --gtk --debug <url>

In a second terminal tab, enter in the Flatpak sandbox and run gdb:

$ flatpak ps
Instance   PID    Application              Runtime
953119635  616738 org.webkit.Webkit        org.webkit.Sdk
$ sudo flatpak enter 616738 /bin/bash
# gdb attach $(/bin/pidof /app/webkit/WebKitBuild/Debug/bin/WebKitWebProcess)
(gdb) ...

You can use the WEB_PROCESS_CMD_PREFIX environment variable (only works on Developer builds, build-webkit or regular CMake enabled with -DDEVELOPER_MODE=ON) for that purpose. If that variable is defined the web process will be run using its value as a prefix.

Example:

WEB_PROCESS_CMD_PREFIX='/usr/bin/gdbserver --no-startup-with-shell localhost:9090' Tools/Scripts/run-minibrowser --gtk --debug

and in a different terminal:

$ Tools/Scripts/webkit-flatpak --debug --command=gdb /app/webkit/WebKitBuild/Debug/bin/WebKitWebProcess
Reading symbols from bin/WebKitWebProcess...done.
(gdb) target remote localhost:9090
(gdb) continue

If you want to debug the network process you can use NETWORK_PROCESS_CMD_PREFIX in a similar way.

An alternative to using a command prefix is to set the WEBKIT2_PAUSE_WEB_PROCESS_ON_LAUNCH environment variable. This will pause the web process for 30 seconds, giving you time to attach in a debugger before it starts.

Yet another option is to use the WEBKIT_MINI_BROWSER_PREFIX env var to run MiniBrowser inside gdb:

$ WEBKIT_MINI_BROWSER_PREFIX=gdb Tools/Scripts/run-minibrowser --gtk ...
...
(gdb) set follow-fork-mode child
(gdb) b SomeSymbol::InTheWebProcess
(gdb) c

Getting a backtrace from a coredump

This works both with release and debug builds. If you are trying to get a backtrace from the WebKitGTK packages shipped by your distribution, ensure you have the corresponding dbgsym packages installed.

Without Systemd

• Open a terminal/shell, and enable coredumps

  ulimit -c unlimited
  

• From that shell, run the browser and make it crash.
• After it has crashed a file named "core" should have appeared on your working directory.
  • If it wasn't the case ensure that you have not set the sysctl kernel.core_pattern set to another directory/pipe

    $ sudo sysctl kernel.core_pattern=core
    

• Once you have the core, get a backtrace from it as follows:

  gdb --batch -ex "thread apply all bt full" /full/path/to/WebKitWebProcess core &> backtrace.txt
  

  • If you get a ​crash on the GDB demangler, you can workaround it as follows:
    • Edit (or create) the file ${HOME}/.gdbinit
    • Write this line on it

      set demangle-style none
      

    • And repeat the operation. You should not have got a backtrace with the symbols mangled.
      • You can now manually demangle those symbols by piping the output as follows

        cat backtrace.txt | c++filt
        

With Systemd and coredumpctl enabled

First, make sure systemd-coredump will not truncate its coredumps, add a /etc/systemd/coredump.conf.d/tune.conf file containing

[Coredump]
ProcessSizeMax=12G
ExternalSizeMax=12G

$ Tools/Scripts/webkit-flatpak --gdb --debug -m /app/webkit/WebKitBuild/Debug/WebKitWebProcess
(gdb) t a a bt

This should also work:

$ Tools/Scripts/webkit-flatpak --gdb-stack-trace --debug -m /app/webkit/WebKitBuild/Debug/WebKitWebProcess
$ Tools/Scripts/webkit-flatpak --gdb-stack-trace --debug -m <crashed-process-pid>

Logging support

Logging and other output/behaviors support is activated when the -DENABLE_JOURNALD_LOG=ON CMake option is used.

In addition to having the logging support activated in the compilation, we need also to turn on the proper logging channels when running.

These channels are defined in the ​Source/WebCore/platform/Logging.h and ​Source/WebKit/Platform/Logging.h headers in the code:

For passing the wanted channels to the running process we need to use the WEBKIT_DEBUG env variable as it is seen in the ​WebCore/platform/unix/LoggingUnix.cpp.

• Notice also the DISABLE_NI_WARNING env variable referred there.

It is important to export the WEBKIT_DEBUG env variable because you will want it to affect also the WebProcess. Known log levels associated to log channels are in this order: error, warning, info, debug. So if you enable debug for a specific channel, you will also get the logs for the preceding levels (error, warning, info).

• This is an example for turning on the logging int he Network and Media channels:

  $ export WEBKIT_DEBUG="Network=debug,Media=debug"
  $ Tools/Scripts/run-minibrowser --gtk
  

  • To enable all logging channels use WEBKIT_DEBUG="all" above

To watch the logs, use journalctl like in this example:

$ journalctl -f WEBKIT_SUBSYSTEM=WebKitGTK WEBKIT_CHANNEL=Media

It is also possible to turn on the logging and other output/behaviors support in a Release build by setting the proper C Macros. You may want to check the ​Source/WTF/wtf/Assertions.h header.

• This is a Release build example in which we want to turn on the logging support

  $ export CXXFLAGS="-DLOG_DISABLED=0"
  $ export WEBKIT_DEBUG="all"
  $ Tools/Scripts/build-webkit --gtk
  

Network analysis

Although libsoup should log everything we need about the network traffic in the GTK port, we may want to use other mechanisms to get that information because, for example, we don't have a build with logging support available in that moment.

Tools like ​tcpdump or ​WireShark can be handy in this case.

This is an example of tcpdump usage to get all the headers of the HTTP traffic going to an HTTP server running in the port 8000 of the loopback interface:

$ tcpdump -vvvs 1024 -l -A -i lo0 'tcp port 8000'

Debugging multimedia stuff

You can use the environment variable GST_DEBUG:

$ export GST_DEBUG="3,webkit*:6" GST_DEBUG_FILE="$HOME/gst.log" GST_DEBUG_NO_COLOR=1 WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1

WEBKIT_DISABLE_SANDBOX_THIS_IS_DANGEROUS=1 is required to ensure the debug log is saved in your real home directory, not the sandbox's home directory.

To debug using flatpak, be sure to use flatpak run --filesystem=home to disable the flatpak sandbox as well.

To debug a multimedia layout test:

$ GST_DEBUG=webkit*:6 GST_DEBUG_NO_COLOR=1 Tools/Scripts/run-webkit-tests --gtk  http/tests/media/video-play-stall.html

Once the MiniBrowser opens showing the results, look for the stderr hyper-text link. The GStreamer logs will be in that file.

Related

• Tracking memory errors in WebKitGTK
• List of WebKit environment variables
• Debugging with RR