Remake of the Distributed Build System

Goals

• Faster builds without sacrificing reliability
• Stats per server
• Build clients (BC) connect to server
• Support cross-compiled win32 sim builds
• Upgrade ability from the master to the clients == keep local configs separate from the main build script

Would be nice

• Use persistent SSH sessions instead of opening and closing multiple per build
• Support rbutil builds?
• Support manual builds?

Ideas

Things we want (to consider) in the new build system.

A Speculative parallel builds

All BCs pick an unfinished build as long as not all builds are done. This means that towards the end, more than one BC will build specific builds and the fastest one will be used. This also means that the slower BCs need to be told to kill their build when another BC already completed an upload.

B Allow More than One Admin

We need to allow for a multitude of admins to be able to allow new clients to join the fun.

C Prioritize builds recently done on same BC

Allow BCs to ask for "preferred builds" so that they can ask for builds it recently did, and thus make better use of ccache.

D Allow updates from server

One problem of the existing solution is that we allow a bit too much configuration of the main build script in a way that it made it impossible for us to upgrade the remote builds scripts.

E Auto-ranking of build clients

Use build and uploads speeds (over time) for particular BCs to calculate which BC that ought to get which builds. May of course conflict with (C)

F Uploading in the background

Make the BC start the next build while uploading the previous one.

---

Client/server protocol

Each client keeps a control channel open against the build server, for commands and information.

Control channel is a SSL connection with unique certificate for each build client

Server commands

> BUILD [build-id] [conf-options] [revision] [zip] [multithreaded] [output]
< _BUILD [build-id]

• build-id: numeric id, created by server
• conf-options: tools/configure options
• revision: the subversion revision number to build
• zip: zip=send zip, nozip=don't send zip
• multithreaded: mt=build is multithreaded (-j), st=build is single-thread
• output: the main output file name this builds create

Several BUILD commands can be sent without waiting for completion. (Client queuing)

>CANCEL [build-id] [revision]
<_CANCEL [build-id]

• build: all = cancel all

> PING [seqnum]
< _PING [seqnum]

> UPDATE [revision]
< _UPDATE

Update client script to [revision] and reload.

Client messages:

> HELLO [version] [archlist] [username] [clientname] [cpu] [32/64bit] [os] [bogomips]
< _HELLO [status]

• version: client script version
• archlist: sh,arm,m68k,mipsel,linuxsim,winsim,manual,rbutil
• username: owner of machine/client
• clientname: user-unique name of this running instance
• cpu: output of uname -m
• 32/64: 32 or 64
• os: output of uname -o
• bogomips: how many bogomips the host boasts

• status: ok=ok, baduser=unknown user

> COMPLETED [build-id]
< _COMPLETED [build-id]

Build completed, log and optional zip uploaded.

Build order

• Build list is time sorted, longest build first.
• Client list is bogomips sorted, highest first.

• Pop the list, one build per client.
• Loop the client list until all builds are assigned.

• When a client completes all its' builds, give it the first (longest) unfinished build.
• When a build is finished, cancel other clients building it.

Statistics

• compile times: wallclock time, cpu time, iowait time?
• upload time
• sysinfo: cpu, disk, 64-bit (uname?)

Test run

• Check out (or copy) a clean rockbox repo to a new directory. Make sure the tree is clean, i.e. no modified files.
• Give a go at tools/rockboxdev.sh (probably as root user) and be sure to extend the $PATH if you haven't already on that machine.
• Download https://git.rockbox.org/cgit/www.git/plain/buildserver/rbclient.pl into it
• Download https://git.rockbox.org/cgit/www.git/plain/buildserver/runclient.sh into it
• Edit runclient.sh:
  • Invent a personal (unique) name for -clientname=
  • Make up any value for -username and -password. Those are not yet used.
  • Parameter -archlist specifies which builds you can build, i.e. which compilers you have. It should be a comma separated list such as "arm,sh,m68k,mipsel".
• Start runclient.sh