Install PINACLES on Perlmutter

Load required modules

module load python
module load cray-fftw

(Do we need to load the hdf5 and netcdf4 modules?)

Loading the python module gets us into the NERSC's default "nersc-python" conda environment, which does not have all the dependencies of PINACLES.

Create a conda environment for PINACLES

WACCEM members can use the copy in the shared directory global/common/software/m1867/python, so this step is not necessary. If one is interested in having her/his own conda environment to run PINACLES, follow the commands below.

It is probably better to copy the conda environment "nersc-python" (or "nersc-mpi4py") that already has mpi4py, and add other packages used by PINACLES

conda create --clone nersc-python --prefix /global/common/software/m1867/python/pinacles_share 

#conda config --append envs_dirs /global/common/software/m1867/python
# This command is for having the short name appear on the conda env list so that
# we don't need to type the full path to the environment

mamba install -c conda-forge xarray zarr termcolor pysolar pyFFTW

pip install mpi4py_fft  #not available with Conda

conda activate /global/common/software/m1867/python/pinacles_share

We (WACCEM) usually install python packages in the project directory under /global/common/software instead of the default location of $HOME directory, following the guidance from NERSC.

Install PINACLES

The following commands were used to install PINACLES in /global/common/software/m1867/PINACLES. One can repeat the same procedure to have her/his own local copy.

Download PINACLES source code

cd /global/common/software/m1867 #replace with your own path

git clone --recurse-submodules ssh://git@gitlab-data.pnnl.gov:2222/pinacles/PINACLES.git

#the following git commands are not necessary if the option "--recurse-submodules" is used with git clone
#git submodule init
#git submodule update

#checkout the RCE branch
git checkout -t origin/waccem/RCE

export PINACLESdir="/global/common/software/m1867/PINACLES"  
#this is for convenience; again replace with your own path

Compile the external physics source code

We need to compile the Fortran source codes for the physics parameterizations as shared libraries. PINACLES uses them as external modules. To compile, we run the python script build_external.py in the top directory of the PINACLES code. It takes about 20 minutes to compile all the physics modules.

It is better NOT to have the climate-utils module loaded, which could overwrite $PATH variables with older ones that are not valid anymore.

cd $PINACLESdir

python build_external.py #assume python module is already loaded

After this compilation, PINACLES is ready to be run.