Skip to content

NPE and PrgEnv-llvm

NPE or NERSC Programming Environment complements and extends the Cray Programming Environment provided by HPE.

NPE and PrgEnv-LLVM are currently Experimental

These are new efforts under active development. Long term support options are being planned and evaluated, but for now significant breaking changes may occur at any time.

Your feedback is welcome!

Usage

NPE is not yet part of the default set of modules and must be added via:

module use /global/cfs/cdirs/nstaff/cookbg/pe/modulefiles

after which NPE can be loaded

ml npe

Note

Additional versions of the npe stack will be made available as additional npe modules.

PrgEnv-llvm

LLVM compilers with support for OpenMP offload and SYCL with support for MPI via MPICH.

ml PrgEnv-llvm

If GPU support is desired then

ml cudatoolkit craype-accel-nvidia80
ml PrgEnv-llvm

Note

Use mpic++, mpicc to compile, not cc or CC.

Note

Fortran (flang) is not supported at this time.

intel-llvm is the default version and supports OpenMP offload and SYCL for A100. Nightly versions are also available (see ml avail intel-llvm).

llvm is based on llvm.org released versions and supports OpenMP offload for A100.

MPICH

The mpich module is available for:

  • PrgEnv-gnu
  • PrgEnv-nvidia
  • PrgEnv-llvm

Tip

mpic++, mpicc and mpif90 should be used instead of cc, CC, and ftn in this release.

Tip

The cudatoolkit and craype-accel-nvidia80 modules must be loaded before mpich to enable CUDA aware MPI.

flux

Flux is a flexible framework for resource management consisting of a suite of projects, tools, and libraries which may be used to build site-custom resource managers for HPC centers.

Warn

Reminder that everything in NPE is considered "experimental" NERSC's current workflow tool recommendations.

Start multi-node interactive job

salloc -C gpu -A <account> -N 2 --ntasks-per-node=1 --gpus-per-node 4 -q debug -t 30

Load the flux-sched module

module load flux-sched

Start flux

srun --mpi=pmi2 --pty flux start

Verify that flux can see gpu resources

$ flux resource list
     STATE NNODES   NCORES    NGPUS NODELIST
      free      2      128        8 nid[001704-001705]
 allocated      0        0        0
      down      0        0        0

Support

Please report any issues, comments or feedback via the NERSC Help Desk.

Release notes

22.06

  • flux support added (flux-sched/0.22.0)
  • Reframe test for flux
  • also build llvm/14.0.0

22.05

Initial "alpha" release includes

  • mpich/4.0 for PrgEnv-gnu, PrgEnv-nvidia and PrgEnv-llvm mpich source
  • PrgEnv-llvm/0.1 a new LLVM based PrgEnv
  • llvm/14.0.3 from llvm.org llvm.org source with OpenMP offload support for A100.
  • intel-llvm/2021-12 from Intel's fork of LLVM with OpenMP offload and SYCL support for A100.
  • intel-llvm/sycl-nightly/<date> Access to nightly builds of intel-llvm.
  • ninja/1.10.2 Ninja build tool
  • cmake/3.22.1 CMake build system
  • reframe/3.10.1 HPC testing framework