Automated building and testing

Workflow

The spack-stack repository now contains scripts for implementing scheduled, automated building, unit testing, and build caching of packages. These scripts live under utils/weekly_build/. They download their own copy of spack-stack (the develop branch, by default) and build one or more environments under that directory structure, which may be transient or reused.

Script failures are detected by non-zero exit codes.

The list of packages to be unit tested is stored in the $PACKAGES_TO_TEST variable in ShellSetup.sh. As unit testing needs are identified, this list should be updated to be as comprehensive as possible, see below.

Platform-specific configuration

Use a clean environment. Do NOT run setup.sh in the directory containing the build scripts (util/weekly_build/) to be run. Nothing in spack-stack outside of the build scripts is needed to get the automated build workflow running, as the workflow will take care of download a copy of spack-stack to be used for doing the automated builds themselves.

The ShellSetup.sh script points to a script in the sites/ subdirectory based on the site name as set by the third argument of each build script (stored and used as $PLATFORM throughout the scripts). Note that a number of variables and functions can be set or overridden here:

• $COMPILERS: Space-delimited list of compilers to be built against on a given system.
• $TEMPLATES: Space-delimited list of templates to install. Default is 'unified-env'.
• $PACKAGES_TO_TEST: Space-delimited list of packages for which unit tests should be run.
• $PACKAGES_TO_INSTALL: Space-delimited list of packages to be installed. For many systems it can be left empty so as to install the entire Unified Environment.
• alert_cmd(): This function should be set for each system. It is invoked when any step fails (non-zero exit code somewhere in the workflow), and so can be used to send emails, post to Slack or GitHub, etc.
• spack_wrapper(): This function runs the Spack command, where the first argument is a log file name. It should not typically need to be redefined by platform.
• spack_install_wrapper(): This function determines the means by which the spack command is invoked for the install steps and can be overridden for each platform. If testing and installation will be performed outside of a batch scheduler, i.e., directly on a login node, then the default can probably be used. However, overriding this function allows, for instance, for job schedulers to be invoked (srun, qsub, etc.), as well as for parallel installations to be performed. Note that the first argument is a log file name, so custom versions of this function will most likely need to use shift to properly access the remaining arguments.
  • $INSTALL_OPTS: Additional spack install flags to be supplied to spack_install_wrapper().
  • $TEST_<APPNAME>: Set this variable to ON in order to enable the corresponding test under util/weekly_build/apptests/.
  • $BUILD_CACHE: Name of or path to (beginning with '\file://') build cache directory. Defaults to 'local-binary' mirror defined in site's mirrors.yaml.
• $SOURCE_CACHE: Name of or path to (beginning with '\file://') source cache directory. Defaults to 'local-source' mirror defined in site's mirrors.yaml.
• $PADDED_LENGTH: Padded length setting for Spack build cache generation. It should be as long as possible without the build failing.
• $KEEP_WEEKLY_BUILD_DIR: Set to 'YES' to use a persistent directory structure for automated builds.
• $DISABLE_TESTING: Set to 'YES' to switch off build-time package testing regardless of whether $PACKAGES_TO_TEST is set.

Setting up weekly builds: cron

Prior to running, obtain a current copy of the build scripts by cloning the spack-stack repository to the platform of interest (/path/to/spack-stack/auto-build/).

If /path/to/spack-stack is the local spack-stack root directory containing spack-stack-x.y.z installations, run mkdir -p /path/to/spack-stack/weekly_build/logs.

In crontab, add:

0 1 * * SUN /path/to/spack-stack-auto-build/util/weekly_build/SpackStackBuildCache_AllSteps.sh $(date +\%y\%m\%d) /path/to/spack-stack/weekly_build acorn > /path/to/spack-stack/weekly_build/logs/cron.$(date +\%y\%m\%d).out 2>&1

Platform-specific testing requirements

Platform overview

Platform name	Maintaining org.	Job system (Jenkins, cron, etc.)	Key applications to test	Notes
Acorn	EMC	cron	UFS WM	builds as user (no role account)
Orion	EPIC	Jenkins	UFS WM	role.epic account
...	...	...	...	...

Unit testing requirements

Package	Has Spack-accessible unit tests	Work needed (link to issues)
bacio	yes	version upgrade for unit testing
g2	yes	none
g2c	yes	none
jasper	yes	none
libaec	yes	none
libpng	yes	none
scotch	yes	none
w3emc	yes	none
hdf5	yes	Set HDF5_TEST_PARALLEL through Spack *
ip	yes	https://github.com/NOAA-EMC/NCEPLIBS-ip/issues/250

* May be needed for systems where testing with MPI is inconvenient. Alternatively, maybe HDF5 can be modified to support job schedulers.

Note that spack install --test does not fail if a package has no make test or make check targets. Therefore it is strongly recommended that each package recipe override Spack's built-in check() function so as to ensure that unit tests must successfully run:

    def check(self):
        with working_dir(self.builder.build_directory):
            make("test")