Time and time again, we have seen a resistance to the use of Autoconf because of its (seemingly) arcane language constructs. While macro languages are difficult to use and sh(1) syntax is not immediately intuitive, neither tool can be considered arcane. Indeed, sh(1) is a much more powerful language than most other shell scripting languages, but because of its unique syntax, people tend to shy away from its use.

Further complicating the issue is the complexity of Autoconf in general. It contains many, many macros, and it defines rules for the order of executing those macros. In order to use Autoconf effectively, build system developers must read a fair amount of documentation (all of which is readily available). In our experience, however, many people feel that development build system software should be immediately obvious^[1] and should not require much effort to learn.

As a result of these issues, we have attempted to make Doozer++ easier to use than “raw” Autoconf-based <>configure scripts. Doozer++ macros hide some common details that often trip up configure script authors. Many utility macros are provided to reduce code duplication between configure.in files. For example, code for verifying that an existing installation of a software package meets a version requirement is provided. The code for performing the check is modularized to separate the common tasks involved with this. Namely, Doozer++ provides macros comparing two version numbers, acting on the results of the comparison, and caching the results to speed up future tasks. Each macro is used as the basis for the next so that users have the freedom to choose how they want their code to behave. We feel that this offers a level of flexibility not available with Autoconf 2.13.

Because VR Juggler has always been a cross-platform tool, we have had the need for a cross-platform way to build it. The VR Juggler 1.0 build system achieved this to a limited degree, and Doozer++ goes a step further toward true platform independence. At this time, Doozer++ makes use of software utilities found on all modern operating systems. The list of utilities includes m4(1), sh(1), Perl, Autoconf, and GNU make. We have avoided the use of platform-specific tools because it is all too easy for the platform-specific parts of a build system to get out of sync with each other. Furthermore, we have tried to avoid the use of platform-specific code whenever possible. This is a more difficult goal to achieve, but later sections will address the extent to which platform-specific code can be reduced.

As a cross-platform C++ library, VR Juggler must be compatible with the prevailing C++ compiler on a given platform. In the case of IRIX, that is SGI's MIPSpro Compilers. Similarly, the use of Microsoft Visual C++ is crucial on Win32 platforms. Open source operating systems such as FreeBSD and Linux use GCC, so we have not ignored that compiler whatsoever. Due to our limited needs, however, we have focused on compilers for the C, C++, Java, and IDL languages. Nothing prevents Doozer++ from being extended to allow compiling of code written in FORTRAN, Ada, Pascal, etc.

In Doozer++, an m4 macro sets up the basic platform-specific pieces. User-level code (in an Autoconf configure.in file) then uses that information to execute other macros that pick the appropriate compiler. This builds upon the foundation provided by Autoconf for detecting installed programs, but it goes further by allowing users to associate one or more compilers with a given platform. The end result is a “fallback” system wherein users specify the preferred compiler and zero or more alternatives if the preferred compiler is not available. To achieve this flexibility, the Doozer++ m4 macros must have no compiler-specific bits.

Language freedom is slightly more complicated than compiler freedom. As will be discussed more fully in the following section, Doozer++ separates its work into two pieces: configuration and compilation. During configuration, the tools for compilation are chosen; during compilation, the chosen tools are put to use.

The primary project building tool is GNU make. We chose GNU make initially because of its portability and because it offered useful features over most basic make(1) implementations. Other implementations, such as BSD make, offer even more features, but they are not as portable as GNU make.

Because Doozer++ uses GNU make, users must write makefiles. Doozer++ provides a number of modular makefile “stubs” that collect common functionality. In particular, these stubs provide support for compiling code written in Java, C, C++, and IDL. Adding support for other languages can be done within Doozer++ or within user-level makefiles.

The design of Doozer++ involves a distinct separation between the tools used and the tasks performed. As a result of this separation, there exists the possibility to extend Doozer++ to use tools other than Autoconf and GNU make. Moreover, different combinations are allowed. For example, we have found that make(1) does not deal well with compiling Java code in general. A more appropriate tool for this task has come out of the Apache Jakarta Project called Ant. Within Doozer++, we could make use of Ant instead of or in addition to GNU make to compile Java software. Nothing restricts users to Autoconf and/or GNU make.

The basic idea behind the separation is the following: detect and configure the static information needed to execute tools that compile a (potentially complex) software system. This leads to a two-step process: configuration and compilation. These steps are described in the following subsections.

#if defined(HAVE_HASH_MAP)
#include <hash_map>
#elif defined(HAVE_EXT_HASH_MAP)
#include <ext/hash_map>
#elif defined(HAVE_HASH_MAP_H)
#include <hash_map.h>
#else
#error "std::hash_map is not available with this compiler"
#endif

#if defined(__GNUC__)
#  if __GNUC__ < 3 && __GNUC__ >= 2 && __GNUC_MINOR__ >= 95
#     include <hash_map>
#  elif __GNUC__ >= 3
#     include <ext/hash_map>
#  else
#     include <hash_map.h>
#  endif
#elif defined(__MSVC_VER__)
#  if __MSVC_VER__ >= 7
#     include <hash_map>
#  else
#     error "std::hash_map is not available with this compiler"
#  endif
#elif defined(__sgi__)
#  include <hash_map>
#else
#  error "std::hash_map is not available with this compiler"
#endif

There are many long-term goals for Doozer++ that are beyond the scope of this document. As should be evidenced from the previous sections, however, some near-term goals include the use of project builder tools besides make(1).

Due to the size of the VR Juggler 2.0 source base (nearly 500,000 lines of C++ and Java by one estimate), it is important that the complexity of compiling be centralized rather than spread out over the entire source tree. Centralizing the complexity allows most of the work to be done once, the results of which can then be reused by the rest of the build code. This is the key tenet for the other goals.

Centralization of complexity allows the VR Juggler build to follow the goal of Doozer++ wherein tools and tasks are separated based on what they do. The result is that each of the module's has an Autoconf- and Doozer++-based configure script that bears the brunt of the work needed to hide complexity. Then, each module has one or more “glue” makefiles that pull everything together. These glue makefiles typically direct a given module's build process so that steps occur in the correct order. Because they oversee the whole compilation process, they are often large files with many targets, each of which is responsible for a specific task. With a configure script and a glue makefile, all that remains is a listing of source files that must be compiled. The makefiles that list source files are often very, very short and easy for anyone to extend.

With this foundation, the complexity is separated thusly:

Other steps have been taken to offload complexity or repeated tasks into centralized components. For example, the directory juggler/macros contains Autoconf/m4 macros that can be used in configure scripts. Most of the macros are used to detect and provide information about the various modules that comprise VR Juggler. There are a few helper macros that build on top of Doozer++ and common idioms we have used to simplify configure scripts further.

As has been discussed at great length previously, it is important that users of the build system be freed from remembering many command-line arguments or special steps that must be taken to build a given piece of software. With several modules to configure and build, there is a lot to know about compiling VR Juggler. Most modules have external dependencies, the paths to which must be specified using command-line arguments. Furthermore, some modules may be compiled differently based on configuration-time settings. The so-called “global build” manages all of this, and so it has the responsibility of reducing the amount of information users must remember.

To that end, the global build offers some features for remembering command-line arguments so that users do not have to. Many of the configure scripts for the individual modules are written such that the default values for arguments are reasonable enough that users rarely have to pass them. The combination of these two features usually allows the typical developer to run the global configure script with no arguments. While this is not always the case (for example, on Win32, the path to NSPR must be specified unless the user has done something special in his or her environment), it happens often enough to keep people reasonably happy.

VR Juggler as a whole is a cross-platform software system. Porting to new operating systems is a non-trivial task, and spending time porting the build system to new platforms is time that could be spent more effectively. Speaking less abstractly, before the source code can be ported to a new platform, the build system has to be capable of compiling the source code on the new platform. For that reason, it is important that the entire build system can be put to use right away so that the attention can be devoted to the more difficult task of porting C++ code.

The VR Juggler build system is limited in its portability by the portability of Doozer++. To its credit, Doozer++ is more portable than VR Juggler at this time, so the VR Juggler team still has some room to move. Even so, the VR Juggler build has its own quirks, and thus, people writing the code to build VR Juggler must always have portability in mind. For example, “BASH-isms” must never appear in VR Juggler configure scripts or makefiles. Most Linux distributions may use BASH for /bin/sh, but that certainly does not meant hat all operating systems vendors follow that unfortunate trend.

In keeping with the Doozer++ goal of separating tools and tasks, the VR Juggler build system offers good portability by putting all of the platform- and software-specific pieces in Autoconf configure scripts. In so doing, the makefiles rarely, if ever, have to be modified when a new platform is added to the list. Furthermore, makefiles for GNU make lack sufficient programmatic constructs to provide developers with the ability to write tests that provide more than limited portability. While the Doozer software is relatively portable through its use of GNU make and platform-specific makefiles, it is missing the expressiveness of something based on Autoconf (or a similar tool). As discussed in the previous chapter, the use of make(1) alone for portability requires much effort to make hard-coded, platform- and site-specific makefiles.

In order for the global build to work, the modules it wraps must follow certain rules. If a module does not comply with all the rules, there is no guarantee that it will be able to compile under all circumstances. In other words, “rogue” modules that only implement a few pieces of the puzzle quickly become the weak link in the chain, and module build system authors who want to live outside the rule set complicate matters for everyone else.

Because the VR Juggler build system is a complicated dance, there are many rules that must be followed. For example, certain targets must be defined so that recursive make(1) calls can proceed through the entire source tree. These targets include 'release', 'install-debug', 'links', and 'buildworld'. Other rules include restrictions on the names of files or directories, use of platform-specific conventions, the presence of a working -config script (see the section called “The *-config Scripts and the *.m4 Macro Files”), and provisions for the detection of a usable installation. The full list of rules are provided in the following subsections.

#include <defines.h>
#include <vpr/Sync/Mutex.h>
#include <vrj/Kernel/Kernel.h>

The script configure.pl is a Perl script written to act as part of a build wrapper around an arbitrary collection of software modules. The modules are linked through some sort of (conceptual) dependency graph, in this case specified by a simple configuration file (see the section called “Build Configuration File”). configure.pl reads the configuration file and proceeds to configure the individual modules in an order that satisfies the dependencies. Along the way, environment variables are set or extended so that each subsequent module is configured with the correct settings. The processes of managing the dependencies and performing the configuration are the topics of this section.

module VPR                                                 
{
   external;                                               
   modules/vapor;                                          
}

module Tweek                                               
{
   depend VPR;                                             
   modules/tweek;                                          
}

module VPR
{
   external;
   modules/vapor: VPR_CONFIG=vpr-config, VPR_BASE_DIR=instlinks;          
}

module Tweek
{
   depend VPR;
   modules/tweek: TWEEK_CONFIG=tweek-config, TWEEK_BASE_DIR=instlinks;    
}

The various *-config scripts (vpr-config, tweek-config, vrjuggler-config, etc.) play a vital role in the design of the global build. Unfortunately, this is also where the global build gets complex. Here, code that is intended for use by users of VR Juggler and associated modules is put to use by the code that compiles everything. It makes use of a strict set of behaviors wherein various environment settings and command-line options form a hierarchy of preferences and fallbacks. If anything goes wrong with a user's configuration process, it is almost always related to a .m4 file or a -config script giving unexpected results because of a misused command-line option or a “dirty” environment.

At various times in the history of the VR Juggler project, the developer “pseudo-installation” has been a topic of great controversy. Questions (or arguments, depending on your perspective) regarding its usefulness and its differences from a release installation come up repeatedly. The reason for its existence is quite simple: to simplify the lives of developers. The differences between the developer installation are also fairly simple. In a nutshell, a developer installation uses debugging libraries by default and links applications statically. A release installation, on the other hand, uses optimized libraries by default and links applications dynamically. The reasoning behind this is a little more difficult to nail down, and for that reason, we will say that it is beyond the scope of the document.

To satisfy the only goal of the developer installation (simplification of developers' lives), the developer installation must act exactly like a release installation, but it must be inside the build tree. The developer installation is created automatically as part of the build process, and ideally, its construction is faster than that of a full release installation. In any case, if all goes well, a developer can treat this pseudo-installation as if it were a real installation for the purposes of running tests.

Prior to early August 2002, the developer installation was created separately from a release installation. This was done through the use of symlinks on UNIX-based platforms and file copies on Win32. Since August 2002, the developer installation still uses symlinks or file copies in the same manner, but there is no longer a separation between creation of the developer installation and the release installation. In other words, the release installation is used to make the developer installation, but it is directed to install into the build tree.

The decision to use symlinks or file copies is based on the host platform and on the use of the 'links' target that every module must define. Using a custom Perl script, bsd-install.pl, written to be fully compatible with BSD install(1), symlinks may be created instead of using file copies. (The bsd-install.pl script comes with Doozer++ 1.5 and beyond.) Within the script, a test is performed to determine if the host platform is a Win32 system. If so, copies are always used because there are no symlinks on Win32 file systems. All of these decisions had been made in each module's build system, but they have since been offloaded into bsd-install.pl. This is in keeping with the Doozer++ goal of centralizing complexity.

At one time, there was a long-term goal for this global build script, or “project builder”. In conjunction with cvs-gather.pl, a semi-arbitrary collection of software packages would be downloaded and compiled. The commonality between them would be the use of configure scripts that would be invoked by the project builder. The dependencies would be specified through some configuration file, possibly written in XML, that would be constructed on the fly based on the cvs-gather.pl dependency file.

These lofty plans have not materialized, and it is unclear whether the need for such a tool still exists. Nonetheless, the idea of a highly generalized project builder played a key role in the way that configure.pl was written. In particular, its highly generic nature was motivated by the potential for downloading arbitrary source code and running a configure script. The makefile generated by configure.pl initially followed this goal, too, but it has since had its scope narrowed to deal with the specific conventions of the VR Juggler project.

Any module with DocBook-based documentation can be added to the documentation build. As with the source code build, there are rules that must be followed. With this smaller, less complicated build system, there are fewer things to deal with.

At this time, most of the documentation in the Juggler Project is written using DocBook. Because of that, the settings for building documents from DocBook files are centralized in the file juggler/doc/docbook.mk. This file is parameterized to the extreme so that including makefiles can override its default settings easily. Usually, makefiles that include docbook.mk direct the build to use OpenJade, Saxon, or Xalan to process the DocBook input and PassiveTeX, FOP, or XEP to create PDF files. Further configuration can be done that chooses different versions or installations of the DocBook style sheets, Saxon, Xalan, FOP, etc. The following is the makefile used to generate HTML and PDF versions of this document:

default: html

docs: html chunk-html pdf                                            
install-docs: install-html install-chunk-html install-pdf            

NAME=		build.system                                                  

XML_FILES=	$(NAME).xml                                               
HTML_FILES=	$(NAME).html                                             
PDF_FILES=	$(NAME).pdf                                               

XSLT_TOOL=	Saxon                                                     

# Fill these in!!  Together, they make up the installation prefix.
webroot=	$(HOME)/public_html/jugglerweb                              
instdir=	docs/juggler.build.system                                   

prefix=		$(webroot)/$(instdir)                                       
INSTALL_FILES=	$(webroot)/base_style.css                             

NEED_DB_IMAGES=	1                                                    

$(NAME).html: $(NAME).xml                                            
$(NAME).pdf: $(NAME).xml $(NAME).fo                                  

include ../docbook.mk