 |
CC's Build System
Other Information
| Maven Definitions |
| Jelly Scripting Hints |
Automatic Repository Building
Automatic Maven repository building can be done in a couple of ways. One way is at a timed interval, like every couple of hours, once a day, once a week, etc. Another way is to have the automatic build start whenever there is a change to the code base. Normally, an organization will run a nightly build of their source code during off-peak hours, but I choose to run the build when code changed for a few reasons.1. The code base is relatively small, with projects split up into very small libraries typically with no more than one developer working on the same module at the same time. If the code base was large and would possibly lock out a number of developers then this would not have been practical.
2. No unit testing is done yet. Unit testing is one of the primary reasons for running nightly builds, so that all projects can be tested to make sure a dependency change hasn't broken another project.
3. CC needs almost instantaneous deployment for end users and development. CC doesn't really have a strict release policy. After the initial release of a project, updates will be done at anytime in an unscheduled manner. Although this is dangerous and has impacted other projects/development, using versioned releases is intended to minimize this effect, but release schedules will most likely continue to be random.
After deciding on the triggered build system, I implemented the following steps:
- Created a directory to hold marker files for cvs modules that had changed. This queue directory holds a single empty file named appropriately for the module that had changed in the cvs tree. If necessary, the file could hold the name of the specific file in the module that changed, it's version, the user, etc., if needed at a later time.
- Modified the CVSROOT/loginfo file to create the queue file
or update it for any changes:
ALL (echo ""; cat; umask 0011; touch /home/buildspace/queue/cvschange;) >> /dev/null
^/?Projects/Framework\(/.*\)*$ (echo ""; cat; umask 0011; touch /home/buildspace/queue/framework;) >> /dev/null
^/?Projects/Security\(/.*\)*$ (echo ""; cat; umask 0011; touch /home/buildspace/queue/security;) >> /dev/null
^/?Projects/Jasmin\(/.*\)*$ (echo ""; cat; umask 0011; touch /home/buildspace/queue/jasmin;) >> /dev/null
The first line will touch a queue file called 'cvschange' for any changes in the entire cvs tree. I'm in the process of moving some of the cvs repository to a new server and in the mean time, I sync the old and new cvs trees for some modules, so the 'cvschange' file triggers the sync. The build system is on the new server and the queue is part of the sync.
The next few lines are samples of the entries for specific submodules, the only changes between them being the subproject name and the queue file. Basically, it traps changes in the module marked by the beginning regular expression to trigger the commands in the same line that follows the regex match. The command that follows must accept input, hence the dummy echo and cat commands. The only thing that is important is the touch command. - Once the trigger file is created (and the sync), I created a script that runs as a per minute cronjob. The script locks the build scripts or exits if there is already a build script running. It then checks the timestamp on the trigger file against a build marker file if there was ever a previous build of the subproject. If the trigger file is newer than the build marker file or no marker file exists, the build will be run.
- There are a couple of different builds that can be triggered, like creating web documents, building and deploying jar files, building, deploying and installing plugins, and building and deploying jnlp distributions. When a project is added to the auto-build system, it is known what type of build is used on that project, the most common being a jar and site deployment, but regardless of the build type, a Maven build needs a project.xml file to start, so for every project that will be built, I create a bare bones project.xml file:
- For the jar builds, the following process is followed:
- Checkout the project using the scm:checkout-project goal
- Auto-licensing - Check for the licensing of the java
files and a main license file for the project. The licensing program is
a perl script that can be
run seperately from the build process to replace licensing or run
licensing on a source directory offline. Files may be excluded with an
'.exclude_license' document at the top of
the project tree and the license type can be defined. In the auto-build
sequence, the auto-licensing script will check if there are some
licensing components missing, add them and check in the changes. This
will cause the trigger file to be updated again and the build script
will build the files the next time around. This was done this way
because some processes, namely creating a file catalog,
requires CVS version numbers.
- If the licensing check returns no changes are necessary, the build continues by running the ant:generate-build goal to create an Ant build.xml file. Creating the build.xml file is for people who may download our open source software and may not be familiar with Maven. Unfortunately, the ant:generate-build goal is not full-proof as it would always put dependency locations as being at the main Maven repository at Ibiblio, whereas our local dependencies are hosted locally. So, there is a script that fixes that and runs after the build.xml file is created.
- Finally, the jar file is built and installed for the user account used for auto-building, then deployed along with the project's web site and source code
- After the deployment, a build marker file is created or updated.
- After the build script is run, control is passed back to the main cron script that will update the top-level web site, remove the locking file, and format a report for mailing.
<?xml version="1.0" encoding="UTF-8"?>
<project>
<!-- Inherit main project xml file -->
<extend>project_master.xml</extend>
<!-- Project specific parameters -->
<artifactId>domain</artifactId>
<name>Domain</name>
<groupId>${pom.artifactId}</groupId>
<currentVersion>1.0</currentVersion>
</project>
This is an example of the project.xml file. It has no dependency info or any other relevent info, the only important things are the artifactid and name elements. It does inherit the project_master.xml file, though, which has info regarding the repository for checking out the project.
Only the jar build is described here because it is the most common and complicated, but the process is basically the same for all types of builds.
When first developing and testing this system, it was set up to build the SNAPSHOT and the current version of the project, assuming that developers would want to bump up the version everytime a change was made. But as we discovered, bumping the version is not practical for CC's current release process and the client application update system was never setup for checking newer versions of dependencies. We are in the process of changing that (see the ccjnlp goal) and we now have an interface to add projects to the auto-building and an auto-deploy interface for developers to release and deploy projects when they are ready. Another problem with the current scheme is that it doesn't use the power of Maven's multiproject goals, therefore, it loses the cascading changes made to lower level dependencies and the build order. The auto-deploy interface also fixes that, but it if it didn't, we could always have the build run the parseproject goal and then a multiproject deploy goal to build complete projects.
It's worth mentioning that sometimes a dependency is not always available for download because it is not a CC project or it is not at the Ibiblio site. For instance, one specific problem that occurred when building the Projects/Multimedia project was the requirement of building against Apple's QuickTime jar file. That jar file is not distributable by CC so I had to put a copy on the server and use the maven.jar override property:
maven -Dmaven.jar.override=on -Dmaven.jar.QTjava=~/lib/java/thirdparty/jars/QTjava.zip ...