Buildbot
From charlesreid1
Buildbot creates a framework for automated code testing.
Installation
Tarballs can be obtained from here: http://trac.buildbot.net/
Download/installation instructions are here: http://trac.buildbot.net/wiki/DownloadInstall
Since I was building Buildbot on a system where I had no administrative permissions, I used virtualenv (see Python#Virtual Python: virtualenv for more information).
I downloaded the master and slave tarballs, extracted them, and ran:
$ ptyhon setyp.py install --prefix=/path/to/virtualenv
After doing this, I can use the python binary in /path/to/virtualenv/bin/python to load the buildbot library through python.
Once setup.py is finished, I ran:
$ buildbot --version Buildbot version: 0.8.3p1 Twisted version: 11.0.0 $ buildslave --version Buildslave version: 0.8.3 Twisted version: 11.0.0
Mock Package
In order to run BuildBot, you will also need the Mock package, which is also related to testing.
Obtain Mock here: http://pypi.python.org/pypi/mock#downloads
Creating Buildbots
Unit Test Suite
export PYTHONPATH=".:${PYTHONPATH}"; trial buildbot.test
export PYTHONPATH=".:${PYTHONPATH}"; trial buildslave.test
(Note that although you can build Buildbot without Mock, the above test suites will fail without mock.)
For me, the slave test fails with:
$ export PYTHONPATH=".:${PYTHONPATH}"; trial buildslave.test
buildslave
test ... [ERROR]
===============================================================================
[ERROR]
Traceback (most recent call last):
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/
Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/trial/runner.py", line 677, in loadByNames
things.append(self.findByName(name))
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/
Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/trial/runner.py", line 487, in findByName
return reflect.namedAny(name)
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/
Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/python/reflect.py", line 475, in namedAny
obj = getattr(obj, n)
exceptions.AttributeError: 'module' object has no attribute 'test'
buildslave.test
Creating a Buildmaster
Documentation on creating buildmaster: http://buildbot.net/buildbot/docs/current/Creating-a-buildmaster.html#Creating-a-buildmaster
I created a buildmaster like this:
$ buildbot create-master -r experiment_master
This creates master.cfg.sample, which is a sample configuration file
Creating a Buildslave
Documentation on creating buildslave: http://buildbot.net/buildbot/docs/current/Creating-a-buildslave.html#Creating-a-buildslave
I created a buildslave like this:
$ buildslave create-slave project_directory host:port slavename password
Buildslave creation options: http://buildbot.net/buildbot/docs/current/Buildslave-Options.html#Buildslave-Options
I'm still not clear on what the hostname:port significance is... like, what if the master and slave are on the same machine?
Running Buildbots
Daemon
You can stat the buildmaster daemon by running:
buildbot start /path/to/buildmaster/directory
and likewise, yo ucan start the buildslave daemon by running:
buildslave start /path/to/buildslave/directory
Errors?
I'm seeing errors like:
$ buildbot start experiment_master
Following twistd.log until startup finished..
Removing stale pidfile /home/u0552682/pkg/buildbot/experiment_master/twistd.pid
2011-05-11 20:08:03-0600 [-] Log opened.
2011-05-11 20:08:03-0600 [-] twistd 11.0.0 (/usr/bin/python 2.4.3) starting up.
2011-05-11 20:08:03-0600 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-05-11 20:08:03-0600 [-] Creating BuildMaster -- buildbot.version: 0.8.3p1
2011-05-11 20:08:03-0600 [-] loading configuration from /home/u0552682/pkg/buildbot/experiment_master/master.cfg
2011-05-11 20:08:03-0600 [-] unable to import dnotify, so Maildir will use polling instead
2011-05-11 20:08:03-0600 [-] creating adbapi pool: pysqlite2.dbapi2 ('/home/u0552682/pkg/buildbot
/experiment_master/state.sqlite',) {'check_same_thread': False, 'cp_noisy':
True, 'cp_reconnect': True}
2011-05-11 20:08:03-0600 [-] twisted.spread.pb.PBServerFactory starting on 8099
2011-05-11 20:08:03-0600 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x2ab7da8ab248>
2011-05-11 20:08:03-0600 [-] adding new builder runtests for category None
2011-05-11 20:08:03-0600 [-] trying to load status pickle from /home/u0552682/pkg/buildbot/
experiment_master/runtests/builder
2011-05-11 20:08:03-0600 [-] /home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/
Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/persisted/styles.py:160:
exceptions.DeprecationWarning: twisted.python.reflect.allYourBase was deprecated in
Twisted 11.0.0; please use inspect.getmro instead
2011-05-11 20:08:03-0600 [-] added builder runtests in category None
2011-05-11 20:08:03-0600 [-] setBuilders._add: [<buildbot.util.loop.DelegateLoop instance at 0x2ab7d9fa4908>,
<BuildSlave 'slavename', current builders: >] [<Builder ''runtests'' at 46969133905824>]
2011-05-11 20:08:03-0600 [-] adding IStatusReceiver <WebStatus on port tcp:8099 at 0x2ab7da312248>
2011-05-11 20:08:03-0600 [-] configuration update started
2011-05-11 20:08:03-0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/application/service.py", line 184, in setServiceParent
self.parent.addService(self)
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/application/service.py", line 303, in addService
service.privilegedStartService()
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/application/internet.py", line 357, in privilegedStartService
self._waitingForPort = self.endpoint.listen(self.factory)
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/internet/endpoints.py", line 187, in listen
interface=self._interface)
--- <exception caught here> ---
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/internet/defer.py", line 104, in execute
result = callable(*args, **kw)
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/internet/posixbase.py", line 419, in listenTCP
p.startListening()
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/internet/tcp.py", line 867, in startListening
raise CannotListenError, (self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:8099: (98, 'Address already in use').
The buildmaster took more than 10 seconds to start, so we were unable to
confirm that it started correctly. Please 'tail twistd.log' and look for a
line that says 'configuration update complete' to verify correct startup.
It looks like it started OK: running netstat -tulpn shows that there is a python process using port 8099, and the twistd.log and twistd.pid files both exist. The pid using port 8099 matches the pid in the twistd.pid file.
Further, running ps ax | grep python shows the buildbot start process running...???
Configuring Buildbots
Conceptualization
Documentation on configuration files: http://buildbot.net/buildbot/docs/current/Configuration.html#Configuration
<graphviz> digraph G { "Master Configuration File" -> "Scheduler" "Master Configuration File" -> "Periodic Scheduler"
"Scheduler" -> "Changes" "Periodic Scheduler" -> "Changes"
"Periodic Scheduler" -> "Periodic Builds" "Periodic Builds" -> "BuildSet"
"Scheduler" -> "BuildSet"
"BuildSet" -> "BuildMaster"
"BuildMaster" -> "BuildRequest1" -> "Associated BuildSlaves"
"BuildMaster" -> "BuildRequest2" -> "Associated BuildSlaves"
"BuildMaster" -> "BuildRequest3" -> "Associated BuildSlaves"
"BuildMaster" -> "BuildRequest4" -> "Associated BuildSlaves"
"Associated BuildSlaves" -> "Builder1" -> "BuildFactory1" -> "Build1"
"Associated BuildSlaves" -> "Builder2" -> "BuildFactory2" -> "Build2"
"Associated BuildSlaves" -> "Builder3" -> "BuildFactory3" -> "Build3"
"Associated BuildSlaves" -> "Builder4" -> "BuildFactory4" -> "Build4"
} </graphviz>
In words:
The BuildMaster has Scheduler objects. These scheduler objects create BuildSet objects and give them to the BuildMaster. The BuildMaster then determines how to construct BuildRequests out of the BuildSet objects, and passes the BuildRequests to individual Builders (or BuildSlaves).
Each Builder shares a common operating system, libraries, compilers, headers, etc. If any of these are different, there should be a different builder.
A Builder can be part of multiple BuildSlaves. This is for redundancy; e.g. if there is a single Builder that is part of two BuildSlaves, and one of them goes offline, the Builder still receives instructions from the BuildSlave that has not gone offline.
A Build is a single compilation or test run of a particular version of code. (This is typically something like a checkout, configure, make, make check sequence).
The Builder uses a BuildFactory to specify the steps involved in the Build.