Introduction

CC's Build System

Introduction

Basic individual project build and deployment

More advanced project build and deployment

Extracting commonality

Repository building

Automatic Repository building

Adding a Project to the Process

CC's specific Maven goals

Auto-versioning deployment system

Automatic deployment interface

Adding unit testing

Other Information

Maven Definitions

Jelly Scripting Hints

Introduction:

Concord Consortium uses Java as it's primary language, creating cross-platform educational software for Windows, Mac, and Linux. As a research company, CC's release schedules are slightly unique in that partial releases are not uncommon so that data may be gathered at the earliest possible time in the project's life cycle, with software improvements implemented and delivered later. For that reason, most projects are delivered to client machines using WebStart, a Java technology that provides automatic software updates and CC's own download clients.

Like all programming environments, CC's programmers wanted to simplify their build processes, maintain versioning of their source code, and automate their release processes. In addition, as an open source development organization, the source code needed to be published for the rest of the community. There is still alot of work to complete all of the goals of the organization, but this documentation is meant to help internal and external developers improve their build processes.

Here is an image from a presentation I did about the basic build system I envisioned for CC:

Build system overview image

Basic Requirements:

The basic requirements for the programming environment are separate, but obviously related, to the build system requirements and are standard requirements for any software development. These requirements are:

Central repository to store source code
Tracking of source code changes
An automatic, scheduled build system
Unit testing
Easy to use or automatic deployment system of final executables
Publicly available source code for specific projects

The basic requirements of the build system was to create a deployable build system that would allow developers to version, build, deploy, and recreate projects without having to go through a lengthy process requiring in-depth knowledge of build and source code management systems. Assume in the following documentation that all dependencies are also projects and all projects may have multiple subprojects. The following is a list of further requirements from discussions with our development team:

All dependency jar files should include a version number in it's final executable name.
All versions of a project must be recreateable.
A catalog of all versioned files used to build a project should be created to mark a snapshot of the project when the catalog is created.
If the first requirement is implemented, a recursive catalog of all versioned dependencies for a project should be created to mark a snapshot.
All projects and dependencies should be easily deployable to any user or predefined location.
The system should be OS independent.
If a project is open-source, all source code should be accessible by anyone wishing to get it.

CVS was our source code management system, with an eventual migration to Subversion planned. Initially, Ant was to be used for the the build system, but after Maven was brought to our attention, we chose it as our build system for the project deployment and web site documentation generation. All of our servers are linux based as are a good portion of our developer's systems, so this documentation is similarly skewed.

CVS:

The most obvious way to satisfy most of the requirements would be to use CVS tagging, but there are a number of problems with that.

Developers would need to understand the concepts of tagging and branching. Given project time constraints, developers are amenable to the learning curve, but with short-term interns and contract developers, it would require a substantial commitment to retraining.
Developers may not be the only users of this system as some scripters, documenters, etc. may also be required to mark and deploy releases.
Tagging information can be lost by accidents such as deleting or moving tags, or forgetting to tag appropriately.

Maven:

Having tried Maven and seeing it's potential, it seemed relatively easy to handle most of the requirements with the following three built-in capabilities:

Extensive use of versioning
Dependency retrieval system
Backward compatibility with Ant

It also seemed reasonable to use this project as a good opportunity to learn more about Maven to determine whether it could help fulfill our requirements. Unfortunately, since Maven is relatively new, the documentation is scarce and in it's preliminary stages, so I wrote plugins to handle functionality I couldn't see implemented elsewhere. That's basically what this document is for, to describe and document the plugins and processes for our users, but maybe this document may help others to understand Maven and it's usefulness.
New Maven documentation is available, so look for any answers there before looking here.

CVS Repository Structure:

Java projects in Concord Consortium's repository are set up in a parallel structure under a main "Projects" directory:

Image of cvs repository

Like many software development group projects, these projects are created according to functionality and each project may be a standalone application or a library module for any number of other projects. The deployed system for many of the top-tier projects is also set up in this manner, so library paths are relative to the main class jar file. So, for instance, the main class is in the DataGraph jar here:

Image of Library Structure

and depends on the Data library jar file in a parallel project directory. So in the manifest of the DataGraph jar file, we can put a relative reference to the Data jar file or any other libraries that may be required and we always know they will be located at ../../<subproject>/lib/<subproject>.jar. This directory structure setup was in place before our adoption of Maven, but it is the structure that Maven uses for multiproject goals. After CC's Maven repository was set up and the build system in place, we changed the manifest references to reference the repository, <groupId>/jars/<artifactId>-<version>.jar instead of the lib directories. This repository structure is also useful for JNLP downloads where the codebase can be at the top of the 'Projects' space and each dependency jar file is specified as a relative path to top:

<?xml version="1.0" encoding="UTF-8"?>
<jnlp href="regressiondatagraph.jnlp" spec="1.0+" codebase="http://source.concord.org/repository">
    <information>
        <title>Regression Data Graph Example</title>
        <vendor>The Concord Consortium</vendor>
        <homepage href="index.html" />
        <description />
        <description kind="short" />
        <offline-allowed />
    </information>
    <resources>
        <j2se version="1.4+" max-heap-size="128m" initial-heap-size="32m" />
        <jar href="thirdparty/jars/vecmath-cc-2.0.jar" />
        <jar href="framework/jars/framework-1.43.jar" />
        <jar href="math/jars/math-1.1.jar" />
        <jar href="data/jars/data-2.3.jar" />
        <jar href="datagraph/jars/datagraph-2.3.jar" />
        <jar href="datagraphaddon/jars/datagraphaddon-2.3.5.jar" />
        <jar href="graph/jars/graph-4.3.jar" />
        <jar main="true" href="concord/jars/regressiondatagraph-1.0.jar" />
    </resources>
    <application-desc main-class="org.concord.examples.datagraphaddon.RegressionDataGraph" />
</jnlp>

I suspect that was the main motivation for the directory structure.

The next couple of pages will describe Maven and how I used it for our customized goals. Then I will describe our build and deploy system, the interaction with CVS and Maven, the Perl and shell scripts used as glue, and the web interfaces for developer use.