Skip to content

Spack - A package manager

Spack automates the download-build-install process for software - including dependencies - and provides convenient management of versions and build configurations.

NERSC provides Spack via module load spack. For most uses, the module for the newest available version is recommended.

The default Spack instance is configured to build on software installed by NERSC. By default, this instance of Spack will install new software into $HOME/spack-workspace/perlmutter, though we recommend using a Spack Environment to control where and how Spack installs software.

Using Spack at NERSC - An example

See the Spack documentation for detailed information about using Spack. What follows here is a summary of a recommended process for using spack at NERSC. We also have a glossary of Spack terms further below to help you get started.

To install software with Spack, perform the following steps. In this example we will install GNU Octave into $HOME/octave:

  1. Load the Spack module:

    perlmutter$ module load cpu spack

    Note the "cpu" module used here: this is to remove the cudatoolkit module from your environment. Spack can find the cuda installation for itself, and having the cudatoolkit module loaded complicates that process.

  2. Create, enter and activate a Spack Environment to work in. This will allow you to set configuration preferences for, eg, where you would like the software installed to. (Also see the Spack documentation about environments).

    At NERSC, you cannot create "named" environments with the provided Spack instances (as those are always written into the Spack instance itself), so you will need to specify a path for the environment with -d <path>. When you activate an environment, the -p option is also useful as it modifies your bash prompt to show the active environment.

    perlmutter$ mkdir my-octate-env && cd my-octate-env
perlmutter$ spack env create -d .
==> Created environment in /pscratch/sd/s/sleak/spack-example/my-octave-env
==> You can activate this environment with:
==>   spack env activate -p -d /pscratch/sd/s/sleak/spack-example/my-octave-env
perlmutter$ spack env activate -p -d .
[my-octave-env] perlmutter$

  3. Check and tweak the environment settings. Especially, config:install_tree:root: is where the software will be installed to, and projections describes the pathnames Spack will create for each package it installs. You may also wish to create a filesystem view for you environment, which is a single tree with bin/, lib/ and so on directories. You can get the current config with spack config get config and edit it by editing spack.yaml in your environment directory.

    Tip

    Making a view is an effective way to make your collection of software easy to use: for example:

    spack:
  view: /global/homes/e/elvis/octave # (update this for your own location)

  4. Check that Spack has a package for the software you want to install, and read about the versions and variants available. spack list <word1> <word2> searches where <word1> or <word2> appear in the name, for example:

    [my-octave-env] perlmutter$ spack list gnu octave
==> 12 packages.
dejagnu  gnuradio  octave-optim    perl-term-readline-gnu
gnupg    gnutls    octave-splines  py-gnuplot
gnuplot  octave    octave-struct   ruby-gnuplot

    In this case, octave looks like what we want, and we can find out about it with spack info octave

    Tip

    You can also check whether there is an already-installed version of what you want with spack find. For example, to check for installed instances of the Spack package octave:

    perlmutter$ spack find -vpl octave

    The -v here shows the variants used in each install, the -p shows the path to where it is installed and the -l shows the hash.

    If you are in an environment, spack find will search only in that environment. To search things installed upstream, use spack -E to momentarily ignore the environment, eg spack -E find -vpl octave

    Tip

    There is a full list of the packages Spack knows about at https://spack.readthedocs.io/en/latest/package_list.html.

  5. Check the list of dependencies Spack will install:

    [my-octave-env] perlmutter$ spack spec -Il octave
...
 -   5g4qm6p  octave@7.3.0%gcc@11.2.0 ~arpack+bz2~curl~fftw~fltk~fontconfig~freetype...
[^]  rdin2vk      ^bzip2@1.0.8%gcc@11.2.0 ~debug~pic+shared build_system=generic arc...
[^]  s3d64h7          ^diffutils@3.8%gcc@11.2.0  build_system=autotools arch=linux-s...
[^]  jrgqsiu              ^libiconv@1.16%gcc@11.2.0 build_system=autotools libs=shar...
[^]  3vaa6y4      ^cray-libsci@23.02.1.1%gcc@11.2.0 ~mpi~openmp+shared build_system=...

    Things Spack sees already installed in your installation tree will have a [+] in the first column, and things Spack found installed upstream will have [^]. A - means Spack did not find it, and will (in most cases) build it.

    You can specify on the command-line the version, compiler, variants, etc that you want for the package, for example spack spec -Il octave%gcc means "Octave built with the gcc compiler". Run spack help --spec to see syntax and examples.

  6. When you're satisfied with the predicted outcome of installing this spec, add it to the environment, concretize the enviroment, and install it. (Recent versions of Spack no longer support installing in an environment directly, you must spack add first).

    [my-octave-env] perlmutter$ spack add octave
...
[my-octave-env] perlmutter$ spack concretize
...
[my-octave-env] perlmutter$ spack install -v

    If you specified a view, the software will now be in that view (otherwise, it will be under your install_tree directory, with a different subdirectory per package)

Using software installed with Spack

If you specified a view, you can point your $PATH, $LD_LIBRARY_PATH, etc to the view. If you did not, you can use spack load to load individual packages into your environment, ready to use.

What to do when spack install fails

  1. Check if the error output suggests an easy solution. You can increase the verbosity of spack install with spack -dv install, which might help in identifying the cause of a failure.

    Some failures can be avoided by:

    • building a different (eg earlier) version of the application
    • building with a different compiler (try %gcc or %cray)
    • disabling a variant
    • modifying the spec of a dependency (see Specs and Dependencies in the official Spack documentation)

    Other errors might be fixed by modifying the package - see Debugging, developing or just reading Spack packages for tips on this.

  2. Seek help: you can check the official Spack documentation, open a ticket at https://help.nersc.gov or ask the Spack community via the Spack Slack. (The Spack Slack community is very active and helpful)

Debugging, developing or just reading Spack packages

spack edit octave will open the package.py file in your default editor. In most cases this file is read-only, but you can make a local repo in your environment, and copy the package to there to edit and modify:

[my-octave-env] perlmutter$ spack repo create local-repo
[my-octave-env] perlmutter$ spack repo add $PWD/local-repo
[my-octave-env] perlmutter$ cp -vr $(spack location -p octave) local-repo/packages/

And after this, spack edit octave will edit your local copy of the package, and spack spec and spack install will also preferentially use your copy of octave .

Spack Locations

In this document we use the following environment variables (which are set in the modulefile for spack) to refer to specific locations:

  • $SPACK_ROOT is the location where Spack itself is installed. This is significant because the builtin repo is stored within it.

  • $SPACK_BASE is the location where Spack will look first for packages and modulefiles, and where it will install software to. By default this is at $HOME/spack-workspace/$NERSC_HOST, but you can change this by setting $SPACK_USER_PREFERRED_BASE to an alternative location.

    For example, if you are mostly building software for your team, you might want to add this line to your .bashrc (and encourage your team to do the same):

    # I'm in project m1234 and want my team to share software:
export SPACK_USER_PREFERRED_BASE /global/common/software/m1234/sw

Spack Glossary

The diagram below illustrates the words describing some key Spack concepts:

diagram of relationships between Spack words

Package

"Source code" describing a piece of software and how to build it (actually a Python class, defined in package.py), in a directory with any patches that might need to be applied first.

elvis@cori$ ls -l $SPACK_ROOT/var/spack/repos/builtin/packages/zlib
total 7
drwxrwxr-x+ 2 swowner swowner  512 May 26 13:00 __pycache__
-rw-rw-r--+ 1 swowner swowner 2074 Apr  3 13:40 package.py
-rw-rw-r--+ 1 swowner swowner  564 Mar 30 19:16 w_patch.patch

The Spack Packaging Guide has detail about creating and maintaining packages.

Repo

A collection ("repository") of packages. Almost everything is in the "builtin" repo, but Spack has a "repos" config section where you can specify locations and order of repos to search.

See the Spack package repository documentation for more about Spack repos.

Spec

Spack defines a Domain-Specific Language (DSL) for describing the build parameters and dependencies with which a package was, or should be, built. The Spack Specs and Dependencies documentation describes this in detail, and the example below shows the main elements of a spec:

elements of a Spack Spec

Variant

A variant is a selectable build option for a package. Available variants for a package are defined in the Spack package. You can, in the spec, enable (+), disable (- or ~) or set (name=value) a variant. Variants usually correspond to a ./configure or cmake setting.

Hash

Often directory or modulefile names created by Spack include a string of arbitrary characters, such as x2icaez in the example below:

elvis@cori:~> module av cmake

----------------------- /global/common/sw/modulefiles/cray-cnl7-haswell -----------------------
cmake/3.18.4-gcc-8.3.0-cg2udpv  cmake/3.20.5-gcc-10.1.0-x2icaez

This is a "hash": Spack identifies a unique build spec by computing a hash of the package build settings and dependencies. This is a convenient way to refer to a unique instance of software without typing the full specification.

One common use for a hash is to instruct Spack to build software on a specific, already installed library, e.g. here we can see 2 instances of METIS:

elvis@cori$ spack find -vl metis arch=$(spack arch)
==> 2 installed packages
-- cray-cnl7-haswell / intel@19.0.3.199 ------------------------..
kmwfurn metis@5.1.0~gdb~int64~real64+shared build_type=Release p..

-- cray-cnl7-haswell / intel@19.1.2.254 ------------------------..
lzrewvi metis@5.1.0~gdb~int64~real64+shared build_type=Release p..

Spack will prefer the one with the hash lzrewvi:

elvis@cori$ spack spec -Il mumps+metis
[snip]
Concretized
--------------------------------
 -   my5bvud  mumps@5.3.3%intel@19.1.2.254+complex+double+float~..
 -   t7ue4px      ^cray-libsci@19.06.1%intel@19.1.2.254~mpi~open..
[^]  lzrewvi      ^metis@5.1.0%intel@19.1.2.254~gdb~int64~real64..
 -   yezois3          ^cmake@3.20.5%intel@19.1.2.254~doc+ncurses..
[^]  wcezuim      ^mpich@3.1%intel@19.1.2.254~argobots+fortran+h..

But we can specify a preference to use the other one:

elvis@cori$ spack spec -Il mumps+metis ^metis/kmwfurn
[snip]
Concretized
--------------------------------
==> Warning: Intel's compilers may or may not optimize to the sa..
 -   d5q6rxk  mumps@5.3.3%intel@19.1.2.254+complex+double+float~..
 -   t7ue4px      ^cray-libsci@19.06.1%intel@19.1.2.254~mpi~open..
[^]  kmwfurn      ^metis@5.1.0%intel@19.0.3.199~gdb~int64~real64..
 -   2t6titn          ^cmake@3.20.5%intel@19.0.3.199~doc+ncurses..
[^]  wcezuim      ^mpich@3.1%intel@19.1.2.254~argobots+fortran+h..

Installation tree

A directory for Spack-installed software. Spack allows specification of a directory structure and naming convention for installed software. At NERSC this defaults to a location in your $HOME and the directory structure starts with the target architecture, e.g. cray-cnl7-haswell for Cori Haswell software. Spack uses the keyword install_tree to refer to the installation tree.

Upstream

Spack can only install software to a place where you have write permission, but it can use software (to satisfy dependencies) in other locations, called "upstream" locations. At NERSC, Spack modules are each configured to use as upstreams some locations where NERSC staff have installed software, as a convenience.

To see the list of upstreams Spack will use, run spack config get upstreams:

$ spack config get upstreams
upstreams:
  nersc-installs:
    install_tree: /global/common/software/nersc/cori-2021q4/spack
    modules:
      tcl: /global/common/software/nersc/cori-2021q4/spack-modulefiles
  e4s-2105:
    install_tree: /global/common/software/spackecp/e4s-21.05/software
    modules:
      tcl: /global/common/software/spackecp/e4s-21.05/modules

Buildcache

As a way to avoid repeatedly building the same software, Spack supports packaging up built software for reuse. These pre-built packages are held in a "buildcache", and a subsequent spack install can simply unpack the cached package, instead of rebuilding it. NERSC puts many packages it builds into buildcaches. You can see the list of buildcaches Spack can find, with spack config get mirrors.

Environment

The NERSC Spack configuration aims to make installing software simple, so spack install foo should, in many cases, "just work", and install software under your $SPACK_USER_PREFERRED_BASE. However in most cases it is better to create and use a Spack Environment to fine-tune the Spack configuration, for example:

  • you want to install different software to different locations (for example, you build software for two different projects, and want to install some software to /global/common/software/project1 and some to /global/common/software/project2.
  • you want to install a collection of software into a single tree, where $my_tree/bin has all of the binaries, and $my_tree/lib has all of the libraries, etc. This can be easily done with a Spack View, for which you need an environment.
  • you have software built outside of Spack, and need Spack to build software on your external build, rather than its default approach of building each dependency package.
  • you want to use different conventions for directory structure or modulefile names than the defaults set by NERSC.

A Spack environment is a directory with a spack.yaml file that outlines software to build, and configuration modifications to use while building it.

To create and activate an environment:

nersc$ mkdir my_spack_env
nersc$ cd my_spack_env
nersc$ spack env create -d $PWD
[snip]
==> You can activate this environment with:
==>   spack env activate /path/to/my_spack_env
nersc$ spack env activate -d $PWD

By default the environment will install things to your regular installation tree, but you could change this in spack.yaml, for example:

spack:
  config:
    install_tree:
      # install things into the `install` subdirectory of this environment:
      root: ${SPACK_ENV}/install
      # alternatively: install things into my project's /global/common/software:
      # root: /global/common/software/my_project
    # put modulefiles generated by Spack in this environment directory:
    module_roots:
      tcl:    ${SPACK_ENV}/modules
      lmod:   ${SPACK_ENV}/lmod
  view: /path/to/my_tree
  specs: []

If your software stack needs something that was installed external to Spack you can add a packages section, e.g.:

spack:
  # [snip]
  packages:
    # use the ncl from "module load ncl"
    ncl:
      externals:
      - spec: ncl@6.5.0 
      modules: [ncl/6.5.0] 
    # use my local install of fftw 3.3.8, as the preferred fftw version:
    fftw:
      version: [3.3.8]
      externals:
      - spec: fftw@3.3.8
        prefix: /path/to/my/fftw/install

For further details about using Spack environments see https://spack.readthedocs.io/en/latest/environments.html

Spack Resources