Buildbot
From charlesreid1
Buildbot creates a framework for automated code testing.
Installation
Tarballs can be obtained from here: http://trac.buildbot.net/
Download/installation instructions are here: http://trac.buildbot.net/wiki/DownloadInstall
Prerequisites
VirtualEnv
Since I was building Buildbot on a system where I had no administrative permissions, I used virtualenv (see Python#Virtual Python: virtualenv for more information).
I downloaded the master and slave tarballs, extracted them, and ran:
$ ptyhon setyp.py install --prefix=/path/to/virtualenv
After doing this, I can use the python binary in /path/to/virtualenv/bin/python to load the buildbot library through python.
Mock
In order to run BuildBot, you will also need the Mock package, which is also related to testing.
Obtain Mock here: http://pypi.python.org/pypi/mock#downloads
Installing Buildbot
I downloaded Buildbot (both the master and the slave) from http://trac.buildbot.net/wiki/DownloadInstall
Master
I installed the Buildbot master by running
$ python setup.py build
The next thing I had to do was add the future installation directory to my $PYTHONPATH variable (otherwise the python installer throws an error). I put this in one of my Dot files (.bashrc):
export PYTHONPATH="${HOME}/pkg/buildbot/0.8.3_master/lib/python2.7/site-packages:${PYTHONPATH}"
then made the directory:
$ mkdir -p ${HOME}/pkg/buildbot/0.8.3_master/lib/python2.7/site-packages
and then ran setup's installer:
$ python setup.py install --prefix=${HOME}/pkg/buildbot/0.8.3_master
Slave
I basically repeated the master build process. First, I downloaded the tarball, then untarred it.
Then I ran:
$ python setup.py build
Then I added the path to the Python package to my $PYTHONPATH variable:
export PYTHONPATH="${HOME}/pkg/buildbot/0.8.3_slave/lib/python2.7/site-packages:${PYTHONPATH}"
then made the directory:
$ mkdir -p ${HOME}/pkg/buildbot/0.8.3_slave/lib/python2.7/site-packages
and then ran setup's installer:
$ python setup.py install --prefix=${HOME}/pkg/buildbot/0.8.3_slave
Confirming Installation
Once both installs finished, I added both to my path:
export PATH="${HOME}/pkg/buildbot/0.8.3_master/bin:${PATH}"
export PATH="${HOME}/pkg/buildbot/0.8.3_slave/bin:${PATH}"
and then ran:
$ buildbot --version Buildbot version: 0.8.3p1 Twisted version: 11.0.0 $ buildslave --version Buildslave version: 0.8.3 Twisted version: 11.0.0
Creating Buildbots
Creating a Master
First, create the build master with the create-master command:
$ cd /path/to/buildbot/stuff/
$ buildbot create-master master
This makes a directory called "master".
Second, create the build master config file by creating /path/to/buildbot/stuff/master/master.cfg (see http://buildbot.net/buildbot/docs/latest)
If you're looking for instant gratification, just do this:
$ mv master/master.cfg.sample master/master.cfg
This will use an example git repository for the pyflakes project. However, since I was running on a system that already had a Buildbot running on it, and I just wanted to leave it alone and use my own, I had to change the default HTTP port (8010) to something else, and change the build slave port (9989) to something else.
Third, start the build master:
buildbot start master
You should see something like this:
$ buildbot start master
Following twistd.log until startup finished..
2011-06-03 16:20:09-0600 [-] Log opened.
2011-06-03 16:20:09-0600 [-] twistd 11.0.0 (/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python 2.7.1) starting up.
2011-06-03 16:20:09-0600 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-06-03 16:20:09-0600 [-] Creating BuildMaster -- buildbot.version: 0.8.3p1
2011-06-03 16:20:09-0600 [-] loading configuration from /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/master.cfg
2011-06-03 16:20:09-0600 [-] unable to import dnotify, so Maildir will use polling instead
2011-06-03 16:20:09-0600 [-] creating adbapi pool: sqlite3 ('/uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/state.sqlite',) {'check_same_thread': False, 'cp_noisy': True, 'cp_reconnect': True}
2011-06-03 16:20:09-0600 [-] twisted.spread.pb.PBServerFactory starting on 9999
2011-06-03 16:20:09-0600 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x10231c290>
2011-06-03 16:20:09-0600 [-] adding new builder runtests for category None
2011-06-03 16:20:09-0600 [-] trying to load status pickle from /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/runtests/builder
2011-06-03 16:20:09-0600 [-] added builder runtests in category None
2011-06-03 16:20:09-0600 [-] setBuilders._add: [<buildbot.util.loop.DelegateLoop instance at 0x101e11ea8>, <BuildSlave 'example-slave', current builders: >] [<Builder ''runtests'' at 4331784312>]
2011-06-03 16:20:09-0600 [-] adding IStatusReceiver <WebStatus on port tcp:8888 at 0x101f755a8>
2011-06-03 16:20:09-0600 [-] buildbot.status.web.baseweb.RotateLogSite starting on 8888
2011-06-03 16:20:09-0600 [-] Starting factory <buildbot.status.web.baseweb.RotateLogSite instance at 0x10231cbd8>
2011-06-03 16:20:09-0600 [-] Setting up http.log rotating 10 files of 10000000 bytes each
2011-06-03 16:20:09-0600 [-] WebStatus using (/uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/public_html)
2011-06-03 16:20:09-0600 [-] removing 0 old schedulers, adding 1 new ones
2011-06-03 16:20:09-0600 [-] configuration update started
2011-06-03 16:20:09-0600 [-] adbapi connecting: sqlite3 ('/uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/state.sqlite',){'check_same_thread': False}
2011-06-03 16:20:09-0600 [-] scheduler 'all' got id 1
2011-06-03 16:20:09-0600 [-] adding 1 new changesources, removing 0
2011-06-03 16:20:09-0600 [-] gitpoller: polling git repo at git://github.com/buildbot/pyflakes.git
2011-06-03 16:20:09-0600 [-] configuration update complete
The buildmaster appears to have (re)started correctly.
See #Errors below for error messages and resolution.
Creating a Slave
First, create the build slave with the create-slave command:
buildslave create-slave /tmp/experimental_buildslave .. .. ..
Second, start the build slave:
buildslave start /tmp/experimental_buildslave
Errors
Couldn't listen on any XXXX: Address already in use
If you are seeing error messages about an address already in use:
$ buildbot start master
Following twistd.log until startup finished..
2011-06-03 16:08:17-0600 [-] Log opened.
2011-06-03 16:08:17-0600 [-] twistd 11.0.0 (/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python 2.7.1) starting up.
2011-06-03 16:08:17-0600 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-06-03 16:08:17-0600 [-] Creating BuildMaster -- buildbot.version: 0.8.3p1
2011-06-03 16:08:17-0600 [-] loading configuration from /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/master.cfg
2011-06-03 16:08:18-0600 [-] unable to import dnotify, so Maildir will use polling instead
2011-06-03 16:08:18-0600 [-] creating adbapi pool: sqlite3 ('/uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/state.sqlite',) {'check_same_thread': False, 'cp_noisy': True, 'cp_reconnect': True}
2011-06-03 16:08:18-0600 [-] twisted.spread.pb.PBServerFactory starting on 9989
2011-06-03 16:08:18-0600 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x10231c290>
2011-06-03 16:08:18-0600 [-] adding new builder runtests for category None
2011-06-03 16:08:18-0600 [-] trying to load status pickle from /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/runtests/builder
2011-06-03 16:08:18-0600 [-] no saved status pickle, creating a new one
2011-06-03 16:08:18-0600 [-] added builder runtests in category None
2011-06-03 16:08:18-0600 [-] setBuilders._add: [<buildbot.util.loop.DelegateLoop instance at 0x101e11ea8>, <BuildSlave 'example-slave', current builders: >] [<Builder ''runtests'' at 4331782584>]
2011-06-03 16:08:18-0600 [-] adding IStatusReceiver <WebStatus on port tcp:8010 at 0x101f745a8>
2011-06-03 16:08:18-0600 [-] configuration update started
2011-06-03 16:08:18-0600 [-] Unhandled Error
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/application/service.py", line 184, in setServiceParent
self.parent.addService(self)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/application/service.py", line 303, in addService
service.privilegedStartService()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/application/internet.py", line 357, in privilegedStartService
self._waitingForPort = self.endpoint.listen(self.factory)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/internet/endpoints.py", line 187, in listen
interface=self._interface)
--- <exception caught here> ---
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/internet/defer.py", line 104, in execute
result = callable(*args, **kw)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/internet/posixbase.py", line 419, in listenTCP
p.startListening()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/internet/tcp.py", line 867, in startListening
raise CannotListenError, (self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:8010: [Errno 48] Address already in use.
then it is likely there is already a Buildbot server running (or some other service that is using that port). In this case, you can edit your master.cfg configuration file and change the port 8010 to a different and non-standard port.
It is a good idea to make sure the Buildbot is stopped if you are changing the configuration file. Stop the buildbot by running:
buildbot stop master
and edit master.cfg to use non-standard ports (not 8010 and not 9989).
Next, start the buildbot again:
buildbot start master
Running Buildbots
Daemon
You can stat the buildmaster daemon by running:
buildbot start /path/to/buildmaster/directory
and likewise, yo ucan start the buildslave daemon by running:
buildslave start /path/to/buildslave/directory
Errors?
I'm seeing errors like:
$ buildbot start experiment_master
Following twistd.log until startup finished..
Removing stale pidfile /home/u0552682/pkg/buildbot/experiment_master/twistd.pid
2011-05-11 20:08:03-0600 [-] Log opened.
2011-05-11 20:08:03-0600 [-] twistd 11.0.0 (/usr/bin/python 2.4.3) starting up.
2011-05-11 20:08:03-0600 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-05-11 20:08:03-0600 [-] Creating BuildMaster -- buildbot.version: 0.8.3p1
2011-05-11 20:08:03-0600 [-] loading configuration from /home/u0552682/pkg/buildbot/experiment_master/master.cfg
2011-05-11 20:08:03-0600 [-] unable to import dnotify, so Maildir will use polling instead
2011-05-11 20:08:03-0600 [-] creating adbapi pool: pysqlite2.dbapi2 ('/home/u0552682/pkg/buildbot
/experiment_master/state.sqlite',) {'check_same_thread': False, 'cp_noisy':
True, 'cp_reconnect': True}
2011-05-11 20:08:03-0600 [-] twisted.spread.pb.PBServerFactory starting on 8099
2011-05-11 20:08:03-0600 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x2ab7da8ab248>
2011-05-11 20:08:03-0600 [-] adding new builder runtests for category None
2011-05-11 20:08:03-0600 [-] trying to load status pickle from /home/u0552682/pkg/buildbot/
experiment_master/runtests/builder
2011-05-11 20:08:03-0600 [-] /home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/
Twisted-11.0.0-py2.4-linux-x86_64.egg/twisted/persisted/styles.py:160:
exceptions.DeprecationWarning: twisted.python.reflect.allYourBase was deprecated in
Twisted 11.0.0; please use inspect.getmro instead
2011-05-11 20:08:03-0600 [-] added builder runtests in category None
2011-05-11 20:08:03-0600 [-] setBuilders._add: [<buildbot.util.loop.DelegateLoop instance at 0x2ab7d9fa4908>,
<BuildSlave 'slavename', current builders: >] [<Builder ''runtests'' at 46969133905824>]
2011-05-11 20:08:03-0600 [-] adding IStatusReceiver <WebStatus on port tcp:8099 at 0x2ab7da312248>
2011-05-11 20:08:03-0600 [-] configuration update started
2011-05-11 20:08:03-0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/application/service.py", line 184, in setServiceParent
self.parent.addService(self)
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/application/service.py", line 303, in addService
service.privilegedStartService()
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/application/internet.py", line 357, in privilegedStartService
self._waitingForPort = self.endpoint.listen(self.factory)
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/internet/endpoints.py", line 187, in listen
interface=self._interface)
--- <exception caught here> ---
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/internet/defer.py", line 104, in execute
result = callable(*args, **kw)
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/internet/posixbase.py", line 419, in listenTCP
p.startListening()
File "/home/u0552682/pkg/virtualenv/lib/python2.4/site-packages/Twisted-11.0.0-py2.4-linux-x86_64.egg
/twisted/internet/tcp.py", line 867, in startListening
raise CannotListenError, (self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:8099: (98, 'Address already in use').
The buildmaster took more than 10 seconds to start, so we were unable to
confirm that it started correctly. Please 'tail twistd.log' and look for a
line that says 'configuration update complete' to verify correct startup.
It looks like it started OK: running netstat -tulpn shows that there is a python process using port 8099, and the twistd.log and twistd.pid files both exist. The pid using port 8099 matches the pid in the twistd.pid file.
Further, running ps ax | grep python shows the buildbot start process running...???
Configuring Buildbots
Conceptualization
Documentation on configuration files: http://buildbot.net/buildbot/docs/current/Configuration.html#Configuration
<graphviz> digraph G { "Master Configuration File" -> "Scheduler" "Master Configuration File" -> "Periodic Scheduler"
"Scheduler" -> "Changes" "Periodic Scheduler" -> "Changes"
"Periodic Scheduler" -> "Periodic Builds" "Periodic Builds" -> "BuildSet"
"Scheduler" -> "BuildSet"
"BuildSet" -> "BuildMaster"
"BuildMaster" -> "BuildRequest1" -> "Associated BuildSlaves"
"BuildMaster" -> "BuildRequest2" -> "Associated BuildSlaves"
"BuildMaster" -> "BuildRequest3" -> "Associated BuildSlaves"
"BuildMaster" -> "BuildRequest4" -> "Associated BuildSlaves"
"Associated BuildSlaves" -> "Builder1" -> "BuildFactory1" -> "Build1"
"Associated BuildSlaves" -> "Builder2" -> "BuildFactory2" -> "Build2"
"Associated BuildSlaves" -> "Builder3" -> "BuildFactory3" -> "Build3"
"Associated BuildSlaves" -> "Builder4" -> "BuildFactory4" -> "Build4"
} </graphviz>
In words:
The BuildMaster has Scheduler objects. These scheduler objects create BuildSet objects and give them to the BuildMaster. The BuildMaster then determines how to construct BuildRequests out of the BuildSet objects, and passes the BuildRequests to individual Builders (or BuildSlaves).
Each Builder shares a common operating system, libraries, compilers, headers, etc. If any of these are different, there should be a different builder.
A Builder can be part of multiple BuildSlaves. This is for redundancy; e.g. if there is a single Builder that is part of two BuildSlaves, and one of them goes offline, the Builder still receives instructions from the BuildSlave that has not gone offline.
A Build is a single compilation or test run of a particular version of code. (This is typically something like a checkout, configure, make, make check sequence).
The Builder uses a BuildFactory to specify the steps involved in the Build.