Rockbox mail archiveSubject: Remaking distributed builds
Remaking distributed builds
From: Daniel Stenberg <daniel_at_haxx.se>
Date: Mon, 22 Jun 2009 10:44:10 +0200 (CEST)
I just wanted to bring people up to speed on where we are in the remaking of
our fancy ditributed build system to become even more fancy.
Me and Björn worked fiercly on this most of the devcon weekend and towards the
end of it, we did actual build rounds involving some 11 present laptops as
build clients and it seemed to work fairly well.
We did find out a few flaws during these tests that we will continue to fix
and improve before this goes live. We also didn't do all the concepts we had
in mind in those tests (like bg uploading while starting the next build).
In a few days, I'll start running this new build server on one of our project
servers alongside with the existing system. During this "beta period" I hope
that a bunch of you guys will volunteer to endure some extra pain by running
early build clients on your machines. Then this will hand out builds and run
"for real" for a while to allow me to monitor it, check the logs and restart
it as often as I need to smoothen off the bugs that prevent it from doing
So to sum up: with the new system you run clients in your end, and each client
connects to the server and keeps that connection alive. It gets and sends
commands over that connection, and it uploads data to the server using HTTP.
There's no SSH involved anymore and no actual login is required anywhere. We
will use a simple user+password scheme for clients just to reduce the risk of
A build client can be run by anyone who can build Rockbox, as it's just a perl
script that builds rockbox and uses curl to upload data.
The new system is hopefully more allowing of slower machines, so my hope is
that more people will be able to participate in this now - it is also less
hassles involved to join in. The system is designed to deal with clients
popping in or out at any given time without that causing trouble.
I intend to let the server keep all builds logged with plent of details to
allow for fancy stats, toplists and more to make it more fun to "compete" with
contributing cpu cycles etc.
I want to wait until Björn gets back from his vacation abroad until we replace
the existing build system with this. So it won't happen for at least another
I'll also write down the concepts of this new system with more details in a
couple of days to fully describe how it works so that you all can understand
it and contribute with ideas on how we can improve it further.
Also, I ask that we can wait a little longer with the addition of the new
builds that have been suggested to me the recent days and instead get the new
system up and then add the builds in that.
Of course, the sources are already committed to SVN (www/buildserver) so you
can start peeking at it whenever you want! :-)
-- / daniel.haxx.seReceived on 2009-06-22