Speeding up builds with distcc on Ubuntu

distcc is a very clever utility aimed at speeding up builds for large projects.  It distributes compilations across a range of machines, piggybacking on make’s ability to multithread builds.  If you have multiple machines of fairly comparable processing power, distcc can vastly decrease the amount of time a large build takes.  Here, we look at how to use distcc on an open network with machines running the same operating system.  Note that you can also use distcc with a cross-compile environment, as described here.

For the purposes of this tutorial, we’ll call the machine that runs the build locally the build host and the other machines compile hosts.

(Optional) Enable secure access from the build host to all the compile hosts. This is done with SSH. If you have a private network behind a firewall, this can be omitted, as SSH does incur some overhead.

Install SSH server on all the compile hosts.

$ sudo apt-get install openssh-server

On the build host, we generate an SSH keypair.

$ ssh-keygen -t dsa -f ~/.ssh/id_dsa -C “user@buildhost”

Next we copy the public keys to the build host’s authorized_keys file:

$ cat ~/.ssh/id_dsa.pub | ssh user@buildhost ‘cat – >> ~/.ssh/authorized_keys’

Finally, we enable ssh-agent on the build host by adding the following to .profile:

SSHAGENT=/usr/bin/ssh-agent
SSHAGENTARGS="-s"
if [ -z "$SSH_AUTH_SOCK" -a -x "$SSHAGENT" ]; then
eval `$SSHAGENT $SSHAGENTARGS`
trap "kill $SSH_AGENT_PID" 0
fi

Now when you log in to the build host, all you have to do is add your identity to the SSH keychain:

$ ssh-add ~/.ssh/id_dsa


Next we must set up distcc. We start by installing it on all hosts – build and compile:

$ sudo apt-get install distcc distcc-pump

On the build host, we need to tell distcc what other machines it should attempt to compile on. Replace the last line in /etc/distcc/hosts with a space delimited list of your hosts, starting with localhost. Each compile machine should be preceded by ‘@’ (to tell distcc to use SSH). Faster hosts should be first in the list. Any machine with more than two processors should have /numproc appended. Here’s an example where localhost has 4 cores, 192.168.1.5 has 6, and 192.168.1.3 has only 2.

localhost/4 @192.168.1.5/6 @192.168.1.3

To silence warnings about cork sock operations, you also need to add a line to /etc/environment on all compile hosts.

$ sudo echo ‘DISTCC_TCP_CORK=0′ >> /etc/environment

Now we’re ready to build. Note that all machines must have the same version of the compiler and that compiler must be called by the same name. In other words if you’re using clang 3.2 on your build host, and you invoke it as clang-3.2, then you’ll need that same version and name on your compile hosts.

The typical way to invoke distcc is simply to prepend the standard compiler command with distcc. This works well with Make, as you can set or override the $(CC) variable easily.

With projects using GNU autotools and configure scripts, the process is simply:

$ export CC=”distcc gcc”
$ ./configure
$ make -jN

Here n is the number of subprocesses you want to spawn.

 

Comments are closed.