Changes between Initial Version and Version 1 of WebKitGTK/SpeedUpBuild


Ignore:
Timestamp:
Jun 19, 2013 5:05:26 AM (11 years ago)
Author:
Andres Gomez
Comment:

Initial editing

Legend:

Unmodified
Added
Removed
Modified
  • WebKitGTK/SpeedUpBuild

    v1 v1  
     1[[PageOutline]]
     2
     3= Speeding up build times =
     4
     5== ccache ==
     6
     7With ccache, you can speed up your compilations by having a way to reuse object files when they do not change across different compilations, thanks to its cache system.
     8
     9 * Install it in your distro of choice:
     10   * debian / ubuntu:
     11{{{
     12#!sh
     13$ sudo apt-get install ccache
     14}}}
     15
     16   * Fedora:
     17{{{
     18#!sh
     19$ sudo yum install ccache
     20}}}
     21
     22* Make sure it provides wrappers for gcc/g++:
     23{{{
     24#!sh
     25$ which gcc
     26/usr/lib64/ccache/gcc
     27$ which g++
     28/usr/lib64/ccache/g++
     29}}}
     30
     31 * If that's not the case you can just create symlinks in a bin directory with precedence over all the other ones, like ''~/bin'':
     32{{{
     33#!sh
     34$ mkdir ~/bin
     35$ ln -s /usr/bin/ccache gcc
     36$ ln -s /usr/bin/ccache g++
     37$ export PATH=$(HOME)/bin:$PATH
     38}}}
     39
     40I personally would recommend to set ccache's cache size to something between 4GB and 8GB:
     41{{{
     42#!sh
     43$ ccache --max-size=8G
     44}}}
     45
     46Check ccache's stats:
     47{{{
     48#!sh
     49$ ccache -s
     50cache directory                     /home/user/.ccache
     51cache hit (direct)                     5
     52cache hit (preprocessed)               0
     53cache miss                          3637
     54called for link                      100
     55called for preprocessing              14
     56compile failed                         2
     57bad compiler arguments                 5
     58autoconf compile/link                 14
     59no input file                         12
     60files in cache                     10880
     61cache size                         426.5 Mbytes
     62max cache size                       8.0 Gbytes
     63}}}
     64
     65
     66== distcc (WIP) ==
     67
     68With distcc, you can offload your machine from building some files and let distribute the work among other (hopefully more powerful) machines in your local network. Ideally, you should be connected with a wired connection for maximum awesomeness.
     69
     70To make it work, you need to install the distcc client in your local machine and the distcc server in all the machines that you want to use as remote nodes for your "build farm".
     71
     72=== distcc server(s) ===
     73
     74You need to install and configure it in every remote machine you want to use from your local machine (the client):
     75
     76 * Install it in your distro of choice:
     77   * debian / ubuntu server:
     78{{{
     79#!sh
     80$ sudo apt-get install distcc
     81}}}
     82
     83   * Fedora server:
     84{{{
     85#!sh
     86$ sudo yum install distcc-server
     87}}}
     88
     89 * Run the server with something like this (will run on port 3632 by default):
     90{{{
     91#!sh
     92$ distccd --daemon --allow 127.0.0.1 --allow 192.168.10.0/24 --listen 192.168.10.107 --nice 10 --jobs 12
     93}}}
     94(You can also configure it to start on boot. Check distcc documentation)
     95
     96 * Check it's running and listening:
     97{{{
     98#!sh
     99$ # netstat -nuta | grep LISTEN
     100   tcp        0      0 192.168.10.107:3632     0.0.0.0:*               LISTEN     
     101}}}
     102
     103
     104=== distcc client ===
     105
     106You need to install and configure it in your work local machine:
     107
     108 * Install it in your distro of choice:
     109   * debian / ubuntu server:
     110{{{
     111#!sh
     112$ sudo apt-get install distcc
     113}}}
     114
     115   * Fedora server:
     116{{{
     117#!sh
     118$ sudo yum install distcc
     119}}}
     120
     121 * Make sure you integrate it seamlessly with ccache by defining the CCACHE_PREFIX variable:
     122{{{
     123#!sh
     124$ export CCACHE_PREFIX=distcc
     125}}}
     126
     127 * Define the list of hosts (see also "Using a dynamic set of hosts") you want to compile in (order is important, first ones are always preferred):
     128{{{
     129#!sh
     130$ export DISTCC_HOSTS='remote-powerful-machine localhost'  # DON'T USE 127.0.0.1 but 'localhost' instead!!!
     131}}}
     132
     133 * Now you just build webkit specifying an "interesting" number in '-j' (typically -j2N, where N is the number of available cores):
     134{{{
     135#!sh
     136$ Tools/Scripts/build-webkit --makeargs=-j24 --gtk
     137}}}
     138
     139
     140Some considerations:
     141
     142 * By default distcc only sends at most *four* jobs per host. If a distcc server can handle more processes you can increase the number like this: DISTCC_HOST='remote-powerful-machine/8'.
     143 * If the connection to the distcc server is slow you can compress the data: DISTCC_HOST='remote-powerful-machine/8,lzo'. Depending on your case this can actually make things slower, so try first to see if it's worth it in your case.
     144 * Versions of gcc/g++ should *match* in the client and the servers. Otherwise remote compilation would probably fail, falling back to building locally instead.
     145 * You need to perform a full rebuild before having distccpower available in webkit. So, you rm -rf WebKitBuild/Release (or Debug) first.
     146 * It's interesting to have distccmon-gnome installed to check if it's properly distributing the work among the servers.
     147   * debian / ubuntu:
     148{{{
     149#!sh
     150$ sudo apt-get install distccmon-gnome
     151}}}
     152   * Fedora: already included in the distcc package
     153
     154==== Using a dynamic set of hosts ====
     155
     156This is very useful if you're not working always at the same location. distcc comes with avahi support (at least in Debian) which means that it can automatically discover distccd servers. So instead of defining the DISTCC_HOSTS variable just edit your ~/.distcc/hosts (or /etc/distcc hosts if you want system wide changes) and add something like this:
     157{{{
     158#!sh
     159+zeroconf
     160localhost
     161}}}
     162
     163That will instruct distcc to search for distccd servers using avahi.