CVS & Bugzilla integration

Last updated 11 May 2004
Steve McIntyre <steve@einval.com>


1. Introduction

In the software group at Plasmon (my ex-employer), we settled on a very useful combination of free software to help us manage development. We used

We made some small changes to these packages locally to help integrate them into our development process. With these small changes, we had diverse open source tools linked almost seamlessly:

The work we have done here was not rocket science, but was quite fiddly and time-consuming. We hope that others find it useful - please feel free to mail me with comments.

NOTE! I have used example.com throughout these pages. Be aware that the links I'm using here won't work! Example.com is reserved for use in documentation and not available for registration. See RFC 2606, Section 3. You'll (obviously) need to change config files to make this lot work for you!

2. CVS configuration

CVS is (probably) the world's most commonly-used software for source code control. It is a tried-and-tested stable piece of open source software that has been continuously developed and maintained since 1989.

On our internal network, we use the pserver access method, as this is most easily configured for the range of client machines that we commonly use (Linux, FreeBSD and Windows). For external access, we use the ext method and cvs over ssh - this is useful as ssh provides a simple secure connection that is both authenticated and encrypted.

Most of the configuration of CVS is done through the files in the CVSROOT module, which is itself managed through CVS (of course!).

2.1 loginfo

The first thing to do is make grab the logs from CVS checkin messages. To do this, modify the file CVSROOT/loginfo. The version we use looks (something) like:

^genstr        (echo ""; echo $USER; date; cat) | mail -s "CVS commit -- genstr" developer1 developer2
^twedmon       (echo ""; echo $USER; date; cat) | mail -s "CVS commit -- twedmon" developer1 developer2
^udo_demo      (echo ""; echo $USER; date; cat) | mail -s "CVS commit -- udo_demo" developer2 developer3
^bugzilla-test (echo ""; echo $USER; date; cat) | mail -s "CVS commit -- bugzilla-test" developer4
^moinmoin      (echo ""; echo $USER; date; cat) | mail -s "CVS commit -- moinmoin" developer2 developer1 developer4
CVSROOT        (echo ""; echo $USER; date; cat) | mail -s "CVS commit -- CVSROOT" cvs-watchers
DEFAULT        (echo ""; echo $USER; date; cat) | mail -s "CVS commit -- Other modules" developer1
ALL            (echo ""; echo $USER; date; cat) >> $CVSROOT/CVSROOT/commitlog
ALL            (echo ""; echo "Date: "`date`; cat) | $CVSROOT/CVSROOT/bugzilla-watcher $USER %{SVv}

This warrants a little explanation!

In the above list, project members are mailed about checkins on the various modules that they work on (genstr, twedmon, udo_demo, bugzilla-test and moinmoin). CVSROOT is special, and the commits are sent to a local mailing list called cvs-watcheers. The first ALL line simply appends all commit messages to a logfile that we find helpful for searching.

The last line is most important here. It passes all commit messages through a locally-written script called bugzilla-watcher. The variable $USER will evaluate to the name of the user running the script (i.e. the user running the checkin). The macro %{SVv} expands to a comma-separated list of files checked in, of the form:

<DIRECTORY> "<FILE1>",<OLDVER1>,<NEWVER1> "<FILE2>",<OLDVER2>,<NEWVER2> ... "<FILEn>",<OLDVERn>,<NEWVERn>

From this information, the bugzilla-watcher script can (with some work) extract all the information it needs to log all the changes (filename, old version number and new version number) along with their commitlog entries. On additions and removals, the version number shows up as "NONE".

NOTE! The %S entry in the macro is not standard in CVS! We have added it in to our version of CVS (patch for 1.11 ; patch for 1.12.7). It behaves exactly as the %s macro, but it adds quotes around the file name which makes dealing with silly filenames containing spaces much easier. As I'm also the Debian CVS maintainer, current Debian CVS packages also have this patch applied. If you can't or don't want to apply this patch and build your own CVS binary, use %s instead and modify the bugzilla-watcher script - you'll just have problems with those silly filenames. See the cvs documentation for more information about the macros available.

2.2 rcsinfo

CVSROOT/rcsinfo controls templates used on commits and imports. The format of the rcsinfo file is similar to the loginfo file - the first column is a regexp; when it matches the checkin directory, use the filename on the rest of the line as a template. We have added a simple single template called rcstemplate; if you want different messages for different modules, this is easy enough. Our rcsinfo file looks like:

DEFAULT                 $CVSROOT/CVSROOT/rcstemplate

2.3 rcstemplate

CVSROOT/rcstemplate is the simple template that is given to the user when they are asked for a CVS commit message. What we use is:

BugID:   

CVS: Please fill in the above fields with information about your checkin.
CVS: Make sure that you know for certain what you are checking in.

As in the default CVS commit message, lines beginning with "CVS:" are stripped from the message before checkin. We use the BugID: tag later in the bugzilla-watcher script; developers are told to add the Bugzilla bug number(s) on that line, and this is how we track where the commit log messages are meant to go.

NOTE! If people check in using cvs commit -m "message", then they will not see this template. Also, the templates will only be applied to new checkouts; existing checkouts will not start to use it, even if people cvs update.

2.4 checkoutlist

CVSROOT/checkoutlist contains a list of files that should be tracked in the CVSROOT special modules alongside the files that CVS knows about internally. Ours lists

rcstemplate
bugzilla-watcher

so that these local files are tracked too.

2.5 bugzilla-watcher

CVSROOT/bugzilla-watcher is a locally-written perl script. It accepts the information about each changed file that is passed in from CVS loginfo, and reformats that information into something more useful. It then packages this up into an email which is sent to the bugzilla system. I didn't write the script, and it's grotty perl :-). It's here.

This outputs email like:

To: bugzilla-interface@example.com
Subject: [Bug 522] CVS Checkin
From: CVS User <user@example.com>

CVS COMMIT
LOG MESSAGE:
  Make the stripspaces code work in preview mode too. Needs changes to
  
  send_editor() - modify the contents of the edit box
  send_page() - modify the contents of the preview pane
  do_savepage() - make sure we use the stripspaces option on preview too
BRANCH: HEAD
FILES CHECKED IN:
  moinmoin/Page.py  1.12    1.13
  moinmoin/PageEditor.py    1.6    1.7
  moinmoin/wikiaction.py    1.5    1.6

which is sent to the bugzilla system, ready to be further parsed and dealt with.

3. Bugzilla configuration

Bugzilla is the defect-tracking system, or bug tracker, that was written to support the development of Netscape products and more recently Mozilla. It is a powerful Open Source tool that compares very favourably with the expensive proprietary systems available today. It is especially popular among users and developers of other Open Source projects, but also has a growing following in commercial environments.

Bugzilla is a large perl application that implements a web interface on top of a backend SQL database. As well as the web frontend that most users are aware of, there has also been some work done to add an email interface that can be used to manipulate bugs. In the contrib directory of the bugzilla distribution, read the file README.Mailif for instructions on how to set this up. We have done exactly as specified here, and set up a user account with access to the bugzilla database. Procmail for that user pipes incoming mail into the bugzilla_email_append.pl script. We don't use the bug_email.pl script normally.

For reference, there is a bug open against bugzilla right now that covers the patches here, see http://bugzilla.mozilla.org/show_bug.cgi?id=199116.

3.1 bugzilla_email_append.pl

We have slightly modified the input script bugzilla_email_append.pl from the original. We have added the capability to enter the same commit message against multiple bug numbers. We also cope with the case where the user checking in does not have a bugzilla account (e.g. on a build machine with a common login used by many users). The bugzilla_email_append.pl will simply append the body of any email it receives to the bug number(s) specified in the Subject: line, which is quite straightforward.

Our version of this script is here.

3.2 globals.pl

Now that we have CVS commit messages added into the appropriate bug reports, the logical next step is to link those commit messages back into CVS itself, so that it is possible to easily view the changes committed at each step. We have ViewCVS installed on the CVS server, which is a simple-to-use web front end for CVS. By making bugzilla parse the comments it prints when viewing bug reports, we can turn the files and versions in the checkin comments into links into the CVS repository. The code that talks to the database and turns bug reports into HTML is in globals.pl. We hook into the code in quoteUrls() that already parses the comments, and add a chunk of code that converts the format in the email above into something like:

CVS COMMIT
LOG MESSAGE:
  Make the stripspaces code work in preview mode too. Needs changes to
  
  send_editor() - modify the contents of the edit box
  send_page() - modify the contents of the preview pane
  do_savepage() - make sure we use the stripspaces option on preview too
BRANCH: HEAD
FILES CHECKED IN:
moinmoin/Page.py 1.12 1.13 View diff
moinmoin/PageEditor.py 1.6 1.7 View diff
moinmoin/wikiaction.py 1.5 1.6 View diff

Simply clicking on the links here shows you all the changes that were committed for a given bug. The patch is quite small, but not especially pretty. There are two patches to choose from here:

4 ViewCVS configuration

ViewCVS is a Python package that allows web access to CVS repositories. Configuration is quite easy, so I won't cover it here. We have made one simple change to the code - in the code that converts source and logs into HTML for printing, we have added a new regexp that matches the BugID tag we use and converts the bug numbers following it into links back to bugzilla.

HINT:We use several different Apache virtual host configurations in the system described here. This is thoroughly recommended, as it makes life much easier for users and admins alike. Users only have to remember service names (and no trailing /cgi-bin/foo/bar?code=wibble etc.), and it becomes possible to rearrange services onto multiple different machines easily without end-users noticing if real machine names and paths are not hard-coded everywhere. We currently use one machine to host all of the services here, but it is aliased as

5 MoinMoin configuration

See the MoinMoin project page for now. I have quite a lot of local changes that I'd like to be merged upstream, but upstream has moved on from the point where I did this work. I'll need to find some time to merge some of it forward and push it upstream.

Arbitrary diffs

This added support for diffing betweeen any two versions of a file, and was merged in upstream quite a while ago.

PDF output

Great for documentation! (needs more work before I release it).

Bugzilla integration

Simply a tiny modification to linkify Bugxxxx - patch. You'll need to configure moin to tell it where your bugzilla installation lives, of course.

Multi-coloured comments

Support for multi-coloured comments in pages to aid in discussions - patch. Simply enclose your text in "<X<" and ">>>" to change the background colour. We find it very useful for highlighting comments when people are collaborating on documents. Replace X above with one of the following:
< or y for yellow
r for red
g for green
b for blue
o for orange
m for mauve
Of course, expanding the choices further is easy.

Steve McIntyre
steve@einval.com
"Can't keep my eyes from the circling sky,
Tongue-tied & twisted, Just an earth-bound misfit, I..."