Finding and using software on Perlmutter¶

The modules environment¶

Perlmutter's modules environment is similar to Cori's but using Lmod in place of the (earlier) TCL modules used on Cori¹.

Lmod works in the same way as the TCL environment modules familiar to NERSC users, and provides the same user interface and commands - module list, module avail, module load etc all do the same as they do on Cori. Lmod also can use the same TCL-based modulefiles as are used on Cori (though its native and preferred modulefile language is Lua).

Lmod adds some capabilities over TCL modules, and these lead to some differences in behavior:

• module avail doesn't show everything, module spider shows the rest
• Loading a module can automatically unload a conflicting one
• A new shortcut command, ml

Module hierarchies¶

Lmod supports a module hierarchy, where sets of modules with a common dependency are "under" the required module, and do not appear in the output of module avail. Lmod provides another command, module spider, which also shows modules that can be made available, by loading the dependency module first.

This is visible particularly in the Cray Programming Environment (NERSC-deployed modulefiles currently use a flat structure). For example, the Cray Programming Environment provides netcdf, which uses HDF5 for lower-level data access. If HDF5 is built with support for parallel I/O, then netcdf can also support parallel I/O. So Perlmutter has two modules for netcdf: one dependent on cray-hdf5 and one on cray-hdf5-parallel. In your initial environment you see neither:

perlmutter> module avail cray-netcdf
No module(s) or extension(s) found!
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Using module spider:

perlmutter> module --terse spider cray-netcdf
cray-netcdf/4.7.4.3
perlmutter> module spider cray-netcdf/4.7.4.3

---------------------------------------------------------------------------------------------------------------------------------------------------------
  cray-netcdf: cray-netcdf/4.7.4.3
---------------------------------------------------------------------------------------------------------------------------------------------------------

    You will need to load all module(s) on any one of the lines below before the "cray-netcdf/4.7.4.3" module is available to load.

      aocc/2.2.0.1  cray-hdf5/1.12.0.3
      cce/11.0.4  cray-hdf5/1.12.0.3
      gcc/10.2.0  cray-hdf5/1.12.0.3
      gcc/9.3.0  cray-hdf5/1.12.0.3

Module families¶

Lmod has a concept of a "family", which can be thought of as a "niche": only one thing can occupy it at a time. As an example, the craype-hugepages* modules all belong to the "craype_hugepages" family. If you have craype-hugepages2M and run module load craype-hugepages16M, then craype-hugepages2M will be automatically unloaded first.

The ml command¶

Lmod provides an extra command called ml. ml can be used like a short-form of module for most module subcommands, and with no arguments will perform module list.

The Perlmutter software stack¶

GPU-ready applications¶

Please check the Popular applications section for information on the software available or to be available on Perlmutter.

Libraries¶

• MPI / mpi4py (CPU/GPU)
• GASNet (CPU)
• BLAS / LAPACK / ScaLAPACK (CPU/GPU)
• FFTW/cuFFT (CPU/GPU)
• HDF5 / h5py (CPU)
• NetCDF (CPU)
• Spark (CPU/GPU)
• R (CPU/GPU)
• Numpy (CPU/GPU)
• pyCUDA (CPU/GPU)
• Scipy (CPU/GPU)
• Scikit-Learn (CPU/GPU)
• Numba (CPU/GPU)
• TensorFlow (CPU/GPU)
• Keras (CPU/GPU)
• Caffe (CPU/GPU)
• pytorch (CPU/GPU)
• Hypre (CPU/GPU)
• SuperLU-dist (CPU/GPU)
• Sundials (CPU/GPU)
• SLATE (CPU/GPU)
• MAGMA (CPU/GPU)

Programming models¶

• MPI
• OpenMP
• OpenACC
• Kokkos
• Raja
• UPC

Tools¶

Tools supporting GPU development:

• TotalView
• DDT
• ATP
• CUDA-GDB
• CUDA-MEMcheck
• PAPI / cupti
• Nsight

CPU-based tools:

• Cmake
• Spack
• CrayPAT
• Reveal