Skip to content

Getting Started

About this page

This document will guide you through the basics of using NERSC's supercomputers, storage systems, and services.

Welcome to the National Energy Research Scientific Computing Center (NERSC)! If you are a new or existing user, please see links to slides and video for overview of NERSC.

Computing Resources

Perlmutter

Perlmutter is a HPE Cray EX supercomputer that is currently being built. Its phase 1 system has over 1500 GPU-accelerated compute nodes.

Cori

Cori is a Cray XC40 supercomputer with approximately 12000 compute nodes.

Storage Resources

File systems are configured for different purposes. Each machine has access to at least three different file systems with different levels of performance, permanence and available space.

Community File System (CFS)

The Community File System (CFS) is a global file system available on all NERSC computational systems. It allows sharing of data between users, systems, and the "outside world".

High Performance Storage System (HPSS) Archival Storage

The High Performance Storage System (HPSS) is a modern, flexible, performance-oriented mass storage system. It has been used at NERSC for archival storage since 1998. HPSS is intended for long term storage of data that is not frequently accessed.

NERSC Accounts

In order to use the NERSC facilities, you need to obtain a NERSC account and your user account must be tied to a NERSC Allocation in order to run jobs. Please see our password rules if you need help creating account or recovering passwords. Once you have account, you can login to Iris to manage account details.

For more details on account support see following links below:

With Iris you can

  • Check allocation balances
  • Change passwords
  • Run reports
  • Update contact information
  • Clear login failures
  • Change login shell
  • and more!

Connecting to NERSC

MFA is required for NERSC users

Software

NERSC and its vendors supply a rich set of compilers, HPC utilities, programming libraries, development tools, debuggers/profilers, data and visualization tools.

We provide a list of application specific documentation that are built by NERSC staff. NERSC software stack will be provided via spack package manager that helps automate software builds for popular HPC tools. To learn more about our setup see NERSC Spack Setup.

NERSC will provide software stack via Extreme-Scale Scientific Software Stack (E4S) which is a collection of open source software packages for HPC system built via spack. NERSC will provide periodic updates (quarterly/semi-annual) for e4s stacks which can be accessed via module load e4s.

Software can be accessed via module command current we support lmod for Perlmutter, and environment-modules for Cori.

Please see our Software Policy outlining our software support model for NERSC provided software stack.

Something missing?

If there is something missing that you would like to have on our systems, please submit a request and we will evaluate it for appropriateness, cost, effort, and benefit to the community.

Python

The Python we provide at NERSC is Anaconda Python.

Machine Learning

NERSC supports a variety of software for Machine Learning and Deep Learning on our systems.

Computing Environment

Info

$HOME directories are shared across all NERSC systems (except HPSS)

Compiling/ building software

Running Jobs

Typical usage of the system involves submitting scripts (also referred to as "jobs") to a batch system such as Slurm.

Interactive Computing

NERSC provides interactive computing via interactive slurm jobs and Jupyter service.

Debugging and Profiling

NERSC provides many popular debugging and profiling tools.

Data Ecosystem

NERSC Production Data Stack includes support for Data Transfer + Access, Workflows, Data Management Tools, Data Analytics, and Data Visualization.

Data Sharing

Security and Data Integrity

Sharing data with other users must be done carefully. Permissions should be set to the minimum necessary to achieve the desired access. For instance, consider carefully whether it's really necessary before sharing write permissions on data. Be sure to have archived backups of any critical shared data. It is also important to ensure that private login secrets (like SSH private keys or apache htaccess files) are not shared with other users (either intentionally or accidentally). Good practice is to keep things like this in a separate directory that is as locked down as possible.

Sharing with Other Members of Your Project

NERSC's Community file system is set up with group read and write permissions and is ideal for sharing with other members of your project. There is a directory for every active project at NERSC and all members of that project should have access to it by default.

Sharing with NERSC Users Outside of Your Project

You can share files and directories with NERSC users outside of your project by adjusting the unix file permissions. We have an extensive write-up of unix file permissions and how they work.

NERSC provides two commands: give and take which are useful for sharing small amounts of data between users.

To send a file or path to <receiving_username>:

give -u <receiving_username> <file or directory>

To receive a file sent by <sending_username>:

take -u <sending_username> <filename>

To take all files from <sending_username>:

take -a -u <sending_username>

To see what files <sending_username> has sent to you:

take -u <sending_username>

For a full list of options pass the --help flag.

Warning

Files that remain untaken 12 weeks after being given will be purged from the staging area.

Sharing Data outside of NERSC

You can easily and quickly share data over the web using our Science Gateways framework.

You can also share large volumes of data externally by setting up a Globus Sharing Endpoint.

Data Transfers

NERSC partners with ESNet to provide a high speed connection to the outside world. NERSC also provides several tools and systems optimized for data transfer.

External Data Transfer

Tip

NERSC recommends transferring data to and from NERSC using Globus

Globus is a web-based service that solves many of the challenges encountered moving data between systems. Globus provides the most comprehensive, efficient, and easy to use service for most NERSC users.

However, there are other tools available to transfer data between NERSC and other sites:

  • scp: standard Linux utilities suitable for smaller files (<1GB)
  • GridFTP: parallel transfer software for large files

Transferring Data Within NERSC

Tip

"Do you need to transfer at all?" If your data is on NERSC Global File Systems (/global/cfs, /global/projecta, /global/cscratch), data transfer may not be necessary because these file systems are mounted on almost all NERSC systems. However, if you are doing a lot of I/O with these files, you will benefit from staging them on the most performant file system. Usually that's the local scratch file system or the Burst Buffer.

  • Use the the unix command cp, tar or rsync to copy files within the same computational system. For large amounts of data use Globus to leverage the automatic retry functionality

Data Transfer Nodes

The Data Transfer Nodes (DTNs) are servers dedicated for data transfer based upon the ESnet Science DMZ model. DTNs are tuned to transfer data efficiently, optimized for bandwidth and have direct access to most of the NERSC file systems. These transfer nodes are configured within Globus as managed endpoints available to all NERSC users.

NERSC FTP Upload Service

NERSC maintains an FTP upload service designed for external collaborators to be able to send data to NERSC staff and users.

Getting Help

NERSC places a very strong emphasis on enabling science and providing user-oriented systems and services.

Help Desk

Availability

Account support is available 8-5 Pacific Time on business days.

The online help desk is the preferred method for contacting NERSC.

Attention

NERSC Consultants handle thousands of support requests per year. In order to ensure efficient timely resolution of issues include as much of the following as possible when making a request:

  • error messages
  • jobids
  • location of relevant files
    • input/output
    • job scripts
    • source code
    • executables
  • output of module list
  • any steps you have tried
  • steps to reproduce

Tip

You can make code snippets, shell outputs, etc in your ticket much more readable by inserting a line with:

[code]<pre>

before the snippet, and another line with:

</pre>[/code]

after it. While these are the most useful, other options to improve formatting can be found in the full list of formatting options.

Access to the online help system requires logging in with your NERSC username, password, and one-time password. If you are an existing user unable to log in, you can send an email to accounts@nersc.gov for support.

If you are not a NERSC user, you can reach NERSC with your queries at accounts@nersc.gov or allocations@nersc.gov.

Operations

For critical system issues only.

  • Please check the Online Status Page before calling 1-800-666-3772 (USA only) or 1-510-486-8600, Option 1

Phone support

Consulting and account-support phone services have been suspended.

To report an urgent system issue, you may call NERSC at 1-800-66-NERSC (USA) or 510-486-8600 (local and international).

FAQ

Please see the following links for common issues that can be addressed, if you are still having issues please create a ticket in help desk.

Appointments with NERSC User-Support Staff

NERSC provides 25-minute appointments with NERSC expert staff. Before you schedule your appointment consult the list of available topics described below.

Tip

To make the most use of an appointment, we strongly encourage you to try some things on your own and share them with NERSC staff ahead of time using the appointment intake form.

Attention

Appointments use Google Hangouts for video conferencing. If you are using Google Chrome, please make sure that the “enable screen share” Security setting is enabled.

NERSC 101

This category is good for basic questions, and you could not find the answer in our documentation. Or when you just don't know where to start.

KNL Optimization

Advice on how to optimize code and compilers to make use of the KNL compute nodes on Cori. Possible discussion topics include:

  1. Compiling code
  2. Thread affinity
  3. Batch script setup
  4. Profiling your code
  5. Refactoring your code

Containers at NERSC

Advice on deploying containerized workflows at NERSC using Shifter. We recommend that you share your Dockerfile, the image name (after downloading it to Cori using shifterimg) before the appointment if possible.

NERSC Filesystems

Advice on I/O optimization and Filesystems at NERSC. Possible discussion topics include:

  1. Optimal file system choices
  2. Quota and file-permission issues
  3. I/O profiling
  4. Refactoring your code

GPU Basics

Advice on programming GPUs for users that are new to the topic. This category is good for when you have started developing your GPU code, but are encountering problems.

Using GPUs in Python

Advice on how to use GPUs from Python, eg. CuPy, RAPIDS, or PyCUDA.

Checkpoint/Restart using MANA

Advice on how to use MANA to enable automatic checkpoint/restart in MPI applications.