|
Note those instructions are deprecated, see
Gnome Buildbot Installation
OlavVitters: Some script to handle below things would be really nice to get say 50+ slaves provides by various users 
Gnome Buildbot Slave Installation
0. Ensure you have python >= 2.3 installed in your system.
- You will also need the development package installed.
1. Create a user account for the slave (i.e. gnomeslave or buildslave).
OlavVitters: Should decide on a username and use that for all machines.
2. Create the following directory structure (this is not mandatory, but helps with keeping things clean and easy to maintain):
# Directory where Gnome-Buildbot scripts will be deployed (<gnome>) $ mkdir -p ~/gnome # Directories for the actual build process (<work>) $ mkdir -p ~/gnome/work/src $ mkdir -p ~/gnome/work/bin $ mkdir -p ~/gnome/work/slaves # Directory to keep downloaded sources (<sources>) $ mkdir ~/sources
3. Download and unpack Buildbot to the <sources> directory.
We suggest to use buildbot 0.7.5 release tarball because the buildbot API is not stable yet, so the scripts we have developed on top of it might not work with current svn trunk.
You can get buildot from http://buildbot.sourceforge.net
4. Download and unpack Twisted to the <sources> directory.
OlavVitters: I want a RHEL5 RPM from a trusted source.
We suggest to use twisted 2.5.0 release tarball, which is the one we've used and tested but higher versions should work too.
You can get twisted from http://twistedmatrix.com
5. Install Twisted in the slave user account:
$ cd ~/<sources>/Twisted-2.5.0/zope.interface-3.3.0/ $ python ./setup.py build $ python ./setup.py install --home=~ $ cat << END >> ~/.bashrc export PATH=~/bin:$PATH export PYTHONPATH=$PYTHONPATH:~/lib/python export LANG=C export LANGUAGE=C END $ . ~/.bashrc $ cd .. $ python ./setup.py build $ python ./setup.py install --home=~
- NOTE: on some systems I've experienced that Twisted is installed in ~/lib64, not in ~/lib. Consider adding this path to PYTHONPATH if this is your case.
6. Install Buildbot in the slave user account.
The steps for a regular installation are detailed in the buildbot manual: http://buildbot.sourceforge.net/manual-0.7.5.html (Chapter 2)
- But basically, we only need to do this:
$ cd ~/<sources>/buildbot-0.7.5/ $ python ./setup.py build $ python ./setup.py install --home=~
7. Download jhbuild in <sources>:
$ cd ~/<sources> $ svn co http://svn.gnome.org/svn/jhbuild/trunk ./jhbuild
8. Install jhbuild and edit ~/.jhbuildrc to fit your deployment.
$ cd ~/<sources>/jhbuild $ make -f Makefile.plain install
- Some settings that should be used:
makeclean = 'yes' alwaysautogen = True checkout_mode = 'clobber' os.environ['CFLAGS'] = '-g3 -O0 -fprofile-arcs -ftest-coverage' os.environ['CXXFLAGS'] = '-g -O0 -fprofile-arcs -ftest-coverage' os.environ['CPPFLAGS'] = '-g -O0 -fprofile-arcs -ftest-coverage' # Uncomment this if you are using gcc 4.1 (not tested with gcc 4.0) # os.environ['LDFLAGS'] = '-lgcov'
- Some other interesting considerations:
- In order to keep things clean I suggest that jhbuildrc parameters named "checkoutroot" and "prefix" are set pointing to subdirectories of a dedicated directory, for example:
checkoutroot: <work>/src
prefix: <work>/bin
- In order to keep things clean I suggest that jhbuildrc parameters named "checkoutroot" and "prefix" are set pointing to subdirectories of a dedicated directory, for example:
- Try to avoid the "~" symbol in your jhbuildrc, use absolute paths instead.
9. Run jhbuild bootstrap as the slave user.
10. Install lcov in your system. This is used for code coverage statistics.
11. Install xvfb in your system. This is used to provide an X server for those modules which run tests that require one.
- In Debian this package provides a script called xvfb-run, looks like Fedora does not provide it, so maybe we'd need to provide it to you.
OlavVitters: Need a RHEL5 script.
12. Download Gnome-Buildbot scripts:
$ cd <sources> $ cvs -d :pserver:anonymous@cvs.igalia.com:/var/publiccvs co jhbuild-buildbot-scripts
- It would be better if this is hosted in Gnome svn.
OlavVitters: Please provide your SVN username and a CVS dump of above and I'll set it up. Send an email to svnmaster@gnome.org and provide a LINK to the CVS dump.
13. Copy the files under the "slave" subdirectory to the <gnome> directory. Also, copy the files under the "slave/scripts" subdir to the ~/bin directory.
OlavVitters: I want to avoid above&below step. I want to be able to only update the SVN checkout and have the slave do the right thing.
14. Copy the buildslave.list file to the <gnome> directory.
- This file provides the list of projects the master is building and the ports that should be used to communicate slave and master for each one.
- This file must be provided by the master administrator.
OlavVitters: Why can't this file also be in SVN?
15. Setup the multislave.cfg file to fit your deployment.
slavecommonname: will be used in the master to identify your slave in thew web view so it should state the kind of OS or arch used in the slave, something like "Sarge", "RH6", etc. Please avoid using the "-" char in the name.
slavesbasedir: a directory were the build process will be done (set it to <work>/slaves)
host: the master name or address.
passwd: a password that identifies your slave in the master. Should be provided by the master admin.
16. Edit ~/bin/module-reports.sh script.
- This script is used for the code coverage reports.
Set the value of REPORTSDIR to a directory where the HTML files with the code coverage reports will be served by a web server.
Set the value of REPORTSURL to an URL that will be used as the base for
- code coverage pages, for example:
export REPORTSDIR="/var/www/gnomebuild"
export REPORTSURL="http://buildbot.gnome.org/gnomebuild"With this example, when the system creates code coverage statistics for the module named "mymodule", it will create a subdir of /var/www/gnomebuild/ named "mymodule/lcov/" and will move the HTML files there, which could be access through http://buildbot.gnome.org/gnomebuild/mymodule/lcov/.
17. Setup a webserver that properly servers REPORTSURL. The web view in the master will link that kind of URLs to access code coverage statistics.
18. Start the slaves with make start and stop them with make stop.
$ cd <gnome>
$ make start && tail -f twistd.log
...
$ make stop19. Under ~/<gnome> directory, Twisted will create log files that should be deleted after some time to limit the amount of disk being used. You'd probably want to create a cron task to remove all obsolete log files. Something like this should do the trick:
DAYS=14
cd <work>/slaves && find . -mindepth 3 -type f -mtime +${DAYS} -exec rm {} \;
cd <gnome> && find twistd.log.* -mtime +${DAYS} -exec rm {}\;20. Install check >= 0.9.2. Check is probably included in your Linux distro, so, for example, in the case of debian, just get it with apt-get install check.
Gnome Buildbot Master Installation
0. Ensure you have python >= 2.3 installed in your system.
- You will also need the development package installed.
1. Create a user account for the master (i.e. gnomemaster or buildmaster).
2. Create the following directory structure (this is not mandatory, but helps with keeping things clean and easy to maintain):
# Directory where Gnome-Buildbot scripts will be deployed (<gnome>) $ mkdir -p ~/gnome # Directories for the master work (<work>) $ mkdir -p ~/gnome/work/src $ mkdir -p ~/gnome/work/bin $ mkdir -p ~/gnome/work/masters # Directory to keep downloaded sources (<sources>) $ mkdir ~/sources # Directories to keep www resources (<www>) $ mkdir -p ~/www/css $ mkdir -p ~/www/images
3. Download and unpack Buildbot to the <sources> directory.
We suggest to use buildbot 0.7.5 release tarball because the buildbot API is not stable yet, so the scripts we have developed on top of it might not work with current svn trunk.
You can get buildot from http://buildbot.sourceforge.net
4. Download and unpack Twisted to the <sources> directory.
We suggest to use twisted 2.5.0 release tarball, which is the one we've used and tested but higher versions should work too.
You can get twisted from http://twistedmatrix.com
5. Install Twisted in the master user account:
$ cd ~/<sources>/Twisted-2.5.0/zope.interface-3.3.0/ $ python ./setup.py build $ python ./setup.py install --home=~ $ cat << END >> ~/.bashrc export PATH=~/bin:$PATH export PYTHONPATH=$PYTHONPATH:~/lib/python export LANG=C export LANGUAGE=C END $ . ~/.bashrc $ cd .. $ python ./setup.py build $ python ./setup.py install --home=~
- NOTE: on some systems I've experienced that Twisted is installed in ~/lib64, not in ~/lib. Consider adding this path to PYTHONPATH if this is your case.
6. Install Buildbot in the master user account.
The steps for a regular installation are detailed in the buildbot manual: http://buildbot.sourceforge.net/manual-0.7.5.html (Chapter 2)
- But basically, we only need to do this:
$ cd ~/<sources>/buildbot-0.7.5/
- Apply this minor patch (just a cosmetic enhancement, not mandatory):
diff -ruP ./buildbot/status/html.py ../buildbot-0.7.5.patched/buildbot/status/html.py
--- ./buildbot/status/html.py 2006-12-11 06:40:11.000000000 +0100
+++ ../buildbot-0.7.5.patched/buildbot/status/html.py 2007-05-10 18:11:37.000000000 +0200
@@ -1114,7 +1114,7 @@
return self.phase0(request, (changeNames + builderNames),
timestamps, eventGrid)
# start the table: top-header material
- data += '<table border="0" cellspacing="0">\n'
+ data += '<table border="0" cellspacing="2" cellpadding="4">\n'
if projectName and projectURL:
# TODO: this is going to look really ugly- Build and install
$ python ./setup.py build $ python ./setup.py install --home=~
7. Download and install jhbuild:
$ cd ~/<sources> $ svn co http://svn.gnome.org/svn/jhbuild/trunk jhbuild $ cd jhbuild $ make && make install
8. Setup your ~/.jhbuildrc file to fit your deployment.
- The master does not execute any build, jhbuild is used just to get a sorted list of modules to be built.
- So, the .jhbuildrc file just needs the following parameters:
modules: Only if you want to customize the list of modules to build.
checkout: Set it to ~/<work>/src
prefix: Set it to ~/<work>/bin
9. Download Gnome-Buildbot scripts:
$ cd ~/<sources> $ cvs -d :pserver:anonymous@cvs.igalia.com:/var/publiccvs co jhbuild-buildbot-scripts
- It would be nice if this was hosted in Gnome svn.
10. Copy the files under the "master" directory to the <gnome> directory. Also, overwrite default buildbot css file with the one under "master/style":
$ cp ~/<sources>/jhbuild-buildbot-scripts/master/* ~/<gnome> $ cp ~/<sources>/jhbuild-buildbot-scripts/master/style/classic.css ~/<www>/css/classic.css $ cp ~/<sources>/jhbuild-buildbot-scripts/master/style/*.png ~/<www>/images/
11. Setup the ~/<gnome>/multimaster.cfg file to fit your deployment.
buildslaveslist: The path to the file describing the modules to be built and the ports to be used. This file will be generated later using the update-buildmasters script. Set it to ~/<gnome>/buildslave.list
mastersdir: The work directory for the master (set it to ~/<work>/masters)
moduleset: The moduleset to use. This will be used by the slaves to build the proper moduleset. We suggest to use an URI that's accesible by all the slaves instead of a local file path (because this would force you to copy the moduleset file to the same path in all the slaves). For example, http://svn.gnome.org/svn/jhbuild/trunk/modulesets/gnome-2.20.modules would work for a Gnome moduleset.
baseportslave: For each module in the moduleset, a port must be opened to communicate the master and the slave. Set this to the first port you want to open for this communication. For example, if your moduleset contains 100 modules and you set this parameter ro 12000, then ports from 12000 to 12099 will be used. As these ports are used for master<->slave communication, slaves should be allowed to connect to those ports.
baseporthttp: For each module in the moduleset, a port must be opened to get an HTTP frontend with its build details (the so called Waterfall view). For example, if your moduleset contains 100 modules and you set this parameter to 13000, then ports from 13000 to 13099 will be used. You should take care that this range of ports does not conflict with the one defined by the baseportslave parameter.
multimasterhttp: A port where the web frontend that provides the home page with the summary of the current state of all modules will be hosted. Make sure this port is not in the ranges defined by the baseportslave and baseporthttpparameters.
slaves: a list of slave names that are allowed to connect to this master. The names in the list must be the same that the ones used in the slavecommonname parameter of the multislave.cfg file used in the slaves.
slavepasswd: The password that slaves should use to be allowed to connect to this master. You must provide this password to all your slaves.
periodicBuildTimer: Seconds between two consecutive builds of the complete moduleset. Make sure this time is big enough to execute a complete build of the moduleset.
12. Execute the update-buildmasters script:
$ cd ~/<gnome> $ ./update-buildmasters
This script will create the file buildslave.list as defined in the buildslaveslist parameter of multimaster.cfg, with all the modules to be built and the ports to be used for the master<->slave communications and the web frontends. This file should also be provided to the slaves, as they will need it to work properly.
13. Start the master with make start and stop it with make stop.
$ cd ~/<gnome>
$ make start && tail -f twistd.log
...
$ make stop14. You can access your web frontend using http://<host>:<port>/, where:
<host>: Is the value of the host parameter of the multimaster.cfg file.
<port>: Is the value of the multimasterhttp parameter of the multimaster.cfg file.
15. Under ~/<gnome> directory, Twisted and Buildbot will create log files that should be deleted after some time to limit the amount of disk being used. You'd probably want to create a cron task to remove all obsolete log files. Something like this should do the trick:
DAYS=14
cd <work>/masters && find . -mindepth 3 -type f -mtime +${DAYS} -exec rm {} \;
cd <gnome> && find twistd.log.* -mtime +${DAYS} -exec rm {}\;