release
dev builds
extras
themes manual
wiki
device status forums
mailing lists
IRC bugs
patches
dev guide



Wiki > Main > BuildClient (compare)

Difference: BuildClient (r11 vs. r10)

Setting Up a Build Client

Commit Builds

On every commit, we build all targets of Rockbox and present the build status in a build table. This is done in a distributed fashion where build clients gets assigned a target at a time to complete and returns the results back.

Build Client

For this to work as good as possible and to keep the total build time to a minimum, we use a set of build clients. The more clients, the shorter build times and the more appreciated it'll be by the developers.

Who can join?

Literally anyone can join. You don't need a monster machine. The buildmaster distributes builds according to each client's speed, so even relatively slow machines are able to contribute.

You also don't have to keep the client running all the time. The system dynamically adjusts to clients coming and going. So you can keep it running 24/7, or turn it on for a while every now and then. It's all up to you. Every little helps.

Requirements

  • One or more (preferably all) of the cross-compilers, a native gcc and SDL
  • At least 5GB disk space available
  • Preferrably 1Mbit/s or better network network uplink. We have servers doing fine with a 384kps upload link, but much lower will not be very useful.
  • git, ccache, curl and zip installed

Install Procedure

Why not distcc

distcc is a great and fun project, but unsuitable for use by us. Since distcc still preprocesses each file locally, hand over the file to the remote servers to get build and then gets the object file back, it requires slow compiles and fast networks to be really efficient.

In contrast we have relatively fast compiles and a slow network, to the extent that transfering the file over and getting an object file back is slower than simply compiling it locally. This is of course also due to the somewhat slow network between our build servers.

Credits

The build system is written by BjornStenberg and DanielStenberg.

How it works

  • The rbmaster.pl server runs on buildmaster.rockbox.org
  • Each client maintains a constant connection to buildmaster, waiting for build commands.
  • Every time a commit to the main source is done, the git server (git.rockbox.org) connects to buildmaster server and requests a new build.
  • The buildmaster loads the list of targets ("builds") and starts distributing them to the connected clients.
    • The past performance of every client is used to calculate its' speed.
    • The sum of all client speeds is used to calculate a fastest possible build time.
    • The builds are then allocated to the clients, giving each client as much work as he can do within the time.
    • Clients without performance history (i.e. new clients) get a single small build, for benchmarking.
  • When all builds have been handed out, and a client reports ready for more work, speculative building begins:
    • The uncompleted builds are handed out again, so multiple clients are now building the same target.
    • When a client completes a build, that same target is cancelled on all other clients building it.
    • This means that as the end of the build round approaches, more and more builds are being cancelled. This is intentional.

If you want every nitty gritty detail, look at the source code to the build master script.

Fun stuff to do with a client

Using the -commandhook option in recent clients, it's possible to run a script every time the server sends a command. This can e.g. be used to get notified whenever a build is started. See hook.sh for an example script that shows current build status in the title of the terminal window, and that also talks to you to keep you informed (using the festival TTS engine).

r14 - 15 May 2013 - 21:35:22 - BertrikSikken

Revision r11 - 09 Mar 2012 - 15:35 - BjornStenberg
Revision r10 - 24 Jan 2012 - 12:40 - MarcinBukat
Copyright by the contributing authors.