Revision as of 06:57, 4 June 2011

Buildbot creates a framework for automated code testing.

Installation

Tarballs can be obtained from here: http://trac.buildbot.net/

Download/installation instructions are here: http://trac.buildbot.net/wiki/DownloadInstall

Prerequisites

VirtualEnv

Since I was building Buildbot on a system where I had no administrative permissions, I used virtualenv (see Python#Virtual Python: virtualenv for more information).

I downloaded the master and slave tarballs, extracted them, and ran:

$ ptyhon setyp.py install --prefix=/path/to/virtualenv

After doing this, I can use the python binary in /path/to/virtualenv/bin/python to load the buildbot library through python.

Mock

In order to run BuildBot, you will also need the Mock package, which is also related to testing.

Obtain Mock here: http://pypi.python.org/pypi/mock#downloads

Installing Buildbot

I downloaded Buildbot (both the master and the slave) from http://trac.buildbot.net/wiki/DownloadInstall

Master

I installed the Buildbot master by running

$ python setup.py build

The next thing I had to do was add the future installation directory to my $PYTHONPATH variable (otherwise the python installer throws an error). I put this in one of my Dot files (.bashrc):

export PYTHONPATH="${HOME}/pkg/buildbot/0.8.3_master/lib/python2.7/site-packages:${PYTHONPATH}"

then made the directory:

$ mkdir -p ${HOME}/pkg/buildbot/0.8.3_master/lib/python2.7/site-packages

and then ran setup's installer:

$ python setup.py install --prefix=${HOME}/pkg/buildbot/0.8.3_master

Slave

I basically repeated the master build process. First, I downloaded the tarball, then untarred it.

Then I ran:

$ python setup.py build

Then I added the path to the Python package to my $PYTHONPATH variable:

export PYTHONPATH="${HOME}/pkg/buildbot/0.8.3_slave/lib/python2.7/site-packages:${PYTHONPATH}"

then made the directory:

$ mkdir -p ${HOME}/pkg/buildbot/0.8.3_slave/lib/python2.7/site-packages

and then ran setup's installer:

$ python setup.py install --prefix=${HOME}/pkg/buildbot/0.8.3_slave

Confirming Installation

Once both installs finished, I added both to my path:

export PATH="${HOME}/pkg/buildbot/0.8.3_master/bin:${PATH}"
export PATH="${HOME}/pkg/buildbot/0.8.3_slave/bin:${PATH}"

and then ran:

$ buildbot --version
Buildbot version: 0.8.3p1
Twisted version: 11.0.0

$ buildslave --version
Buildslave version: 0.8.3
Twisted version: 11.0.0

Creating Buildbots

Creating a Master

First, create the build master with the create-master command:

$ cd /path/to/buildbot/stuff/
$ buildbot create-master master

This makes a directory called "master".

Second, create the build master config file by creating /path/to/buildbot/stuff/master/master.cfg (see http://buildbot.net/buildbot/docs/latest)

If you're looking for instant gratification, just do this:

$ mv master/master.cfg.sample master/master.cfg

This will use an example git repository for the pyflakes project. However, since I was running on a system that already had a Buildbot running on it, and I just wanted to leave it alone and use my own, I had to change the default HTTP port (8010) to something else, and change the build slave port (9989) to something else.

Third, start the build master:

buildbot start master

(or, if you are not in the same directory as your buildbot master directory,)

buildbot start /path/to/master

You should see something like this:

$ buildbot start master
Following twistd.log until startup finished..
2011-06-03 16:20:09-0600 [-] Log opened.
2011-06-03 16:20:09-0600 [-] twistd 11.0.0 (/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python 2.7.1) starting up.
2011-06-03 16:20:09-0600 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-06-03 16:20:09-0600 [-] Creating BuildMaster -- buildbot.version: 0.8.3p1
2011-06-03 16:20:09-0600 [-] loading configuration from /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/master.cfg
2011-06-03 16:20:09-0600 [-] unable to import dnotify, so Maildir will use polling instead
2011-06-03 16:20:09-0600 [-] creating adbapi pool: sqlite3 ('/uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/state.sqlite',) {'check_same_thread': False, 'cp_noisy': True, 'cp_reconnect': True}
2011-06-03 16:20:09-0600 [-] twisted.spread.pb.PBServerFactory starting on 9999
2011-06-03 16:20:09-0600 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x10231c290>
2011-06-03 16:20:09-0600 [-] adding new builder runtests for category None
2011-06-03 16:20:09-0600 [-] trying to load status pickle from /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/runtests/builder
2011-06-03 16:20:09-0600 [-] added builder runtests in category None
2011-06-03 16:20:09-0600 [-] setBuilders._add: [<buildbot.util.loop.DelegateLoop instance at 0x101e11ea8>, <BuildSlave 'example-slave', current builders: >] [<Builder ''runtests'' at 4331784312>]
2011-06-03 16:20:09-0600 [-] adding IStatusReceiver <WebStatus on port tcp:8888 at 0x101f755a8>
2011-06-03 16:20:09-0600 [-] buildbot.status.web.baseweb.RotateLogSite starting on 8888
2011-06-03 16:20:09-0600 [-] Starting factory <buildbot.status.web.baseweb.RotateLogSite instance at 0x10231cbd8>
2011-06-03 16:20:09-0600 [-] Setting up http.log rotating 10 files of 10000000 bytes each
2011-06-03 16:20:09-0600 [-] WebStatus using (/uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/public_html)
2011-06-03 16:20:09-0600 [-] removing 0 old schedulers, adding 1 new ones
2011-06-03 16:20:09-0600 [-] configuration update started
2011-06-03 16:20:09-0600 [-] adbapi connecting: sqlite3 ('/uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/state.sqlite',){'check_same_thread': False}
2011-06-03 16:20:09-0600 [-] scheduler 'all' got id 1
2011-06-03 16:20:09-0600 [-] adding 1 new changesources, removing 0
2011-06-03 16:20:09-0600 [-] gitpoller: polling git repo at git://github.com/buildbot/pyflakes.git
2011-06-03 16:20:09-0600 [-] configuration update complete
The buildmaster appears to have (re)started correctly.

See #Errors below for error messages and resolution.

Creating a Slave

To create the build slave, you'll need some information from master.cfg:

c['slavePortnum'] = PPPP
c['slaves'] = [BuildSlave("example-slave", "pass")]

You can create the build slave by running:

$ buildslave create-slave slave localhost:PPPP example-slave pass

which should give the output:

$ buildslave create-slave slave localhost:PPPP example-slave pass
mkdir /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/slave
chdir /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/slave
mkdir /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/slave/info
Creating info/admin, you need to edit it appropriately
Creating info/host, you need to edit it appropriately
Not creating info/access_uri - add it if you wish
Please edit the files in /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/slave/info appropriately.
buildslave configured in /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/slave

Second, start the build slave:

buildslave start slave

(or, if you are not in the same directory as your buildbot slave,)

buildslave start /path/to/slave

which should return the output:

$ buildslave start slave
Following twistd.log until startup finished..
2011-06-03 22:39:28-0600 [-] Log opened.
2011-06-03 22:39:28-0600 [-] twistd 11.0.0 (/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python 2.7.1) starting up.
2011-06-03 22:39:28-0600 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-06-03 22:39:28-0600 [-] Starting factory <buildslave.bot.BotFactory instance at 0x101b007a0>
2011-06-03 22:39:28-0600 [-] Connecting to localhost:9999
2011-06-03 22:39:28-0600 [Broker,client] message from master: attached
The buildslave appears to have (re)started correctly.

Webpage View

You should be able to visit hostname:8010 (or whatever port you used) and see a message that indicates success with setting up the project:

Running Buildbots

Starting Buildbot Daemons

You can start the buildmaster daemon by running:

buildbot start /path/to/master

and likewise, you can start the buildslave daemon by running:

buildslave start /path/to/slave

(Note: if you experience errors, see the #Errors section. If you don't see the error you are experiencing listed there, please email me at root(at)charlesmartinreid.com, and I will add your problem/resolution to the page.)

Restarting/Reconfiguring Buildbot

If you edit the configuration file (e.g. master.cfg), then you can restart the buildbot and it will re-load the configuration file:

buildbot reconfig /path/to/master

Checking Buildbot Config File

You can check the buildbot config file by running

$ buildbot checkconfig master.cfg

Buildbot Configuration

Conceptualization of Configure Files

Documentation on configuration files: http://buildbot.net/buildbot/docs/current/Configuration.html#Configuration

<graphviz> digraph G { "Master Configuration File" -> "Scheduler" "Master Configuration File" -> "Periodic Scheduler"

 "Scheduler" -> "Changes"
 "Periodic Scheduler" -> "Changes"

 "Periodic Scheduler" -> "Periodic Builds"
   "Periodic Builds" -> "BuildSet"

 "Scheduler" -> "BuildSet"

   "BuildSet" -> "BuildMaster"

     "BuildMaster" -> "BuildRequest1" -> "Associated BuildSlaves" 
     "BuildMaster" -> "BuildRequest2" -> "Associated BuildSlaves"
     "BuildMaster" -> "BuildRequest3" -> "Associated BuildSlaves"
     "BuildMaster" -> "BuildRequest4" -> "Associated BuildSlaves"

       "Associated BuildSlaves" -> "Builder1" -> "BuildFactory1" -> "Build1"
       "Associated BuildSlaves" -> "Builder2" -> "BuildFactory2" -> "Build2"
       "Associated BuildSlaves" -> "Builder3" -> "BuildFactory3" -> "Build3"
       "Associated BuildSlaves" -> "Builder4" -> "BuildFactory4" -> "Build4"

} </graphviz>

In words:

The BuildMaster has Scheduler objects. These scheduler objects create BuildSet objects and give them to the BuildMaster. The BuildMaster then determines how to construct BuildRequests out of the BuildSet objects, and passes the BuildRequests to individual Builders (or BuildSlaves).

Each Builder shares a common operating system, libraries, compilers, headers, etc. If any of these are different, there should be a different builder.

A Builder can be part of multiple BuildSlaves. This is for redundancy; e.g. if there is a single Builder that is part of two BuildSlaves, and one of them goes offline, the Builder still receives instructions from the BuildSlave that has not gone offline.

A Build is a single compilation or test run of a particular version of code. (This is typically something like a checkout, configure, make, make check sequence).

The Builder uses a BuildFactory to specify the steps involved in the Build.

Errors

Running Buildbot

Couldn't listen on any XXXX: Address already in use

If you are seeing error messages about an address already in use:

$ buildbot start master
Following twistd.log until startup finished..
2011-06-03 16:08:17-0600 [-] Log opened.
2011-06-03 16:08:17-0600 [-] twistd 11.0.0 (/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python 2.7.1) starting up.
2011-06-03 16:08:17-0600 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-06-03 16:08:17-0600 [-] Creating BuildMaster -- buildbot.version: 0.8.3p1
2011-06-03 16:08:17-0600 [-] loading configuration from /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/master.cfg
2011-06-03 16:08:18-0600 [-] unable to import dnotify, so Maildir will use polling instead
2011-06-03 16:08:18-0600 [-] creating adbapi pool: sqlite3 ('/uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/state.sqlite',) {'check_same_thread': False, 'cp_noisy': True, 'cp_reconnect': True}
2011-06-03 16:08:18-0600 [-] twisted.spread.pb.PBServerFactory starting on 9989
2011-06-03 16:08:18-0600 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x10231c290>
2011-06-03 16:08:18-0600 [-] adding new builder runtests for category None
2011-06-03 16:08:18-0600 [-] trying to load status pickle from /uufs/chpc.utah.edu/common/home/u0552682/codes/buildbot/master/runtests/builder
2011-06-03 16:08:18-0600 [-] no saved status pickle, creating a new one
2011-06-03 16:08:18-0600 [-] added builder runtests in category None
2011-06-03 16:08:18-0600 [-] setBuilders._add: [<buildbot.util.loop.DelegateLoop instance at 0x101e11ea8>, <BuildSlave 'example-slave', current builders: >] [<Builder ''runtests'' at 4331782584>]
2011-06-03 16:08:18-0600 [-] adding IStatusReceiver <WebStatus on port tcp:8010 at 0x101f745a8>
2011-06-03 16:08:18-0600 [-] configuration update started
2011-06-03 16:08:18-0600 [-] Unhandled Error
	Traceback (most recent call last):
	  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/application/service.py", line 184, in setServiceParent
	    self.parent.addService(self)
	  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/application/service.py", line 303, in addService
	    service.privilegedStartService()
	  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/application/internet.py", line 357, in privilegedStartService
	    self._waitingForPort = self.endpoint.listen(self.factory)
	  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/internet/endpoints.py", line 187, in listen
	    interface=self._interface)
	--- <exception caught here> ---
	  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/internet/defer.py", line 104, in execute
	    result = callable(*args, **kw)
	  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/internet/posixbase.py", line 419, in listenTCP
	    p.startListening()
	  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Twisted-11.0.0-py2.7-macosx-10.6-x86_64.egg/twisted/internet/tcp.py", line 867, in startListening
	    raise CannotListenError, (self.interface, self.port, le)
	twisted.internet.error.CannotListenError: Couldn't listen on any:8010: [Errno 48] Address already in use.

then it is likely there is already a Buildbot server running (or some other service that is using that port). In this case, you can edit your master.cfg configuration file and change the port 8010 to a different and non-standard port.

It is a good idea to make sure the Buildbot is stopped if you are changing the configuration file. Stop the buildbot by running:

buildbot stop master

and edit master.cfg to use non-standard ports (not 8010 and not 9989).

Next, start the buildbot again:

buildbot start master

Reconfiguring

Reconfiguration Failed

If you make a syntax error in your buildbot and try to reconfigure the buildbot, you will see an error like this:

2011-06-01 1:52:22 [-] loading configuration from /path/to/buildbot/master/master.cfg
2011-06-01 1:52:22 [-] error while parsing config file
2011-06-01 1:52:22 [-] error during loadConfig
2011-06-01 1:52:22 [-] Unhandled Error
      Traceback (most recent call last):
        File "path/to/buildbot/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/application/app.py", line 348, in runReactorWithLogging
          reactor.run()
        File "/path/to/buildbot/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 1128, in run
          self.mainLoop()
        File "/path/to/buildbot/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 1137, in mainLoop
          self.runUntilCurrent()
        File "/path/to/buildbot/lib/python2.6/site-packages/Twisted-8.2.0-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 757, in runUntilCurrent
            call.func(*call.args, **call.kw)
        --- <exception caught here> ---
          File "/path/to/buildbot/src/buildbot/master.py", line 511, in loadTheConfigFile
            self.loadConfig(f)
          File "/path/to/buildbot/src/buildbot/master.py", line 529, in loadConfig
            exec f in localDict
        exceptions.SyntaxError: EOL while scanning string literal (master.cfg, line 191)

2011-06-01 1:52:22 [-] The new config file is unusable, so I'll ignore it.
2011-06-01 1:52:22 [-] I will keep using the previous config file instead.

Reconfiguration failed. Please inspect the master.cfg file for errors,
correct them, then try 'buildbot reconfig' again.

Project Source Code

Specifying Project Source Code

You can specify a "poller" that will check the version control logs for any changes, and test out a build if there are changes detected.

I use the SVNPoller object, so the following goes in my master.cfg configuration file:

from buildbot.changes.svnpoller import SVNPoller
svn_poller = SVNPoller(
                project='Project Name',
                svnurl='http://project_url/svn/trunk'
                pollinterval=600,
                svnbin='/bin/svn'
)               
c['change_source'] = [ svn_poller ]

Information about detecting changes in a project source code are here: http://buildbot.net/buildbot/docs/0.8.3/Change-Sources.html#Change-Sources

Buildbot: Difference between revisions

From charlesreid1