Testing Code

Omega development requires both standalone testing and coupled testing:

  • Standalone testing: CTest unit tests and the Polaris omega_pr regression suite.

  • Coupled testing: CIME system tests and coupled cases.

Important

Coupled testing is just as important as standalone testing. Omega is the ocean component of E3SM, and changes that pass standalone testing can still break coupled simulations (e.g., causing restart ERS or PEM tests to fail). Developers are expected to validate both standalone and coupled behavior on at least one CPU and one GPU machine before submitting pull requests.

For instructions on how to build Omega and run CTest unit tests, see Building and testing Omega.

Running the Polaris omega_pr Test Suite

The CTest unit tests on their own are not sufficient to ensure that changes to Omega behave as expected. Developers also need to run the regression tests in the form of the omega_pr test suite from Polaris. The tests in the suite are defined in omega_pr.txt in the Polaris repository.

You will run the suite twice: once using the Omega version vendored in Polaris under e3sm_submodules/Omega (the official baseline) and once using your development branch. This means you will need to build Omega from both sources before running the suite. You may wish to use the Polaris CTest Utility to automate the build and run the CTests for each.

Note

While the Omega develop branch often matches the Polaris submodule bit-for-bit (b4b), the submodule at e3sm_submodules/Omega is the authoritative baseline for regression comparison.

Prerequisites: build Omega for baseline and PR

Before setting up and running the omega_pr suite, you must build Omega twice:

  • Build the baseline from the Omega sources vendored by Polaris at <path-to-polaris-repo>/e3sm_submodules/Omega. Record the CMake build directory path as BASELINE_BUILD (used below with polaris suite -p).

  • Build from your development (PR) branch. Record the CMake build directory path as PR_BUILD (also used with -p).

The -p option passed to polaris suite should point to the CMake build directory for the corresponding branch (the directory where you ran CMake and invoked ./omega_build.sh).

Note

Run both baseline and PR suite on the same machine, CPU/GPU partition, compiler, and Omega build type (Debug/Release) to avoid false positive/negative diffs.

Setting up the suite

If you have not yet set up Polaris, you will need to follow its quick start including cloning the repo, configuring the software environment, and activating the environment.

Then, with the Polaris environment active, you can set up the baseline omega_pr suite as follows:

BASELINE_BUILD=/path/to/build/from/polaris/e3sm_submodules/Omega
POLARIS_SCRATCH=/path/to/scratch/polaris_testing/
polaris suite -c ocean -t omega_pr --model omega \
  -w $POLARIS_SCRATCH/baseline_omega_pr \
  -p $BASELINE_BUILD

where $POLARIS_SCRATCH is a directory within your scratch space that is useful for setting up and running the Polaris test suites and $BASELINE_BUILD is the directory where you built Omega from the Omega sources located at <polaris-repo>/e3sm_submodules/Omega, or where the Polaris CTest utility built it for you (e.g. build_omega/build_chrysalis_intel).

Next, you can set up the omega_pr for your development branch, using the baseline set up above:

PR_BUILD=/path/to/omega/pr/branch/build
POLARIS_SCRATCH=/path/to/scratch/polaris_testing/
polaris suite -c ocean -t omega_pr --model omega \
    -b $POLARIS_SCRATCH/baseline_omega_pr \
    -w $POLARIS_SCRATCH/my_branch_omega_pr \
  -p $PR_BUILD

Here, $POLARIS_SCRATCH is the same scratch directory as above, $PR_BUILD is the directory where you build Omega from your development branch (or where the CTest utility built for you), and my_branch_omega_pr can be any name that helps you keep track of what you were testing.

Running the suite

First, run the baseline, e.g. for Slurm systems:

cd $POLARIS_SCRATCH/baseline_omega_pr
sbatch job_script.omega_pr.sh

Note the job ID for the next step. Next, run the suite for the test branch, waiting until the baseline is done:

cd $POLARIS_SCRATCH/my_branch_omega_pr
sbatch --kill-on-invalid-dep=yes --dependency=afterok:$JOBID job_script.omega_pr.sh

where $JOBID is the JOB ID of the baseline slurm job.

Monitor the jobs (e.g. squeue -j $JOBID) and take a look at combined stdout/stderr log in polaris_omega_pr.o$JOBID in each work directory to see whether the tests passed. Report the results as part of any (non-trivial) Omega PR.

Note

For other schedulers (e.g. PBS on Aurora), take a look at the machine’s documentation for equivalent commands.

Handling test failures

What should you do if you see test failures?

Test execution errors for baseline (Polaris e3sm_submodules/Omega)

If you see test execution failures in the baseline run from Polaris e3sm_submodules/Omega, these need to be reported as issues on the Omega repo, preferably mentioning a relevant Omega developer who may be able to investigate the cause of the failure. Please provide the contents of the relevant log file for the failing test(s) and optionally the location of the tests (and log files) if other Omega developers will have access.

Test execution errors for your branch

If you see execution failures on the tests for your development branch, please take a look at the log files and attempt to debug the issues yourself first, then reach out to the Omega team if you need help. In many cases, the test failures may related to changes that are required in Polaris, see Updating Polaris below.

Diffs between your branch and baseline

If diffs are expected:

  • Document the diffs (in Omega PR comment)

  • Update e3sm_submodules/Omega in Polaris (after merge of Omega PR)

If not expected:

  • Debug and eliminate

If you see errors related to comparisons with the baseline, you will need to determine if these are expected based on the code changes or not. If they are expected, you will need to document (in a PR comment) how large the differences are in the tests that show differences. In addition, you will need to make a pull request to the Polaris repository to update the Omega submodule in e3sm_submodules/Omega once your branch has been merged so that the changes become the new baseline in Polaris.

Note

We do not update e3sm_submodules/Omega after every branch is merged into develop on Omega to reduce maintenance burden.

If you see unexpected differences between the results for your development branch and those from the baseline, you should attempt to debug the cause yourself and then reach out to the Omega team if you need help.

You may want to look at Polaris’ bisect utility to help debug which commit introduced the unexpected differences.

Updating Polaris

Many Omega changes are likely to also require Polaris changes. In such cases, it makes sense to co-develop your Omega branch and a Polaris branch. The Omega branch will need to be merged first, then the e3sm_submodules/Omega submodule will need to be updated to include the Omega changes as part of the Polaris pull request.

Here are some examples of Omega changes that will require corresponding Polaris changes.

Adding new Omega dimensions or variables

If you add new Omega variables that have a corresponding MPAS-Ocean name, you need to add the mapping between the names to mpaso_to_omega.yaml. For more details, see the Polaris Ocean Framework documentation.

For example:

dimensions:
  Time: time
  nVertLevels: NVertLayers

variables:
  temperature: Temperature
  salinity: Salinity
  ssh: SshCell

In each case, the keys are the MPAS-Ocean names and the values are the corresponding Omega names.

Updates to Default.yml

If you add or remove sections, config options, streams, etc. in Default.yml, you may need to make corresponding changes to the YAML files for Polaris test cases for them to continue to work as expected.

Whenever possible, we try to use a mapping between MPAS-Ocean namelist options and Omega config options within mpaso_to_omega.yaml. For example:

config:
- section:
    time_management: TimeIntegration
  options:
    config_start_time: StartTime
    config_stop_time: StopTime
    config_run_duration: RunDuration
    config_calendar_type: CalendarType

If you have added new Omega config sections or options that have corresponding MPAS-Ocean namelist sections or options, please add them to the map. In each case, the keys are the MPAS-Ocean names and the values are the corresponding Omega names.

Frequently, there is no MPAS-Ocean equivalent to an Omega config option, and we currently do not try to map streams from MPAS-Ocean to Omega (their syntax is too different). In such cases, we have do define Omega YAML config options and streams in each test case, and you may need to update each one with your changes. Here is an example exerpt from the manufactured_solution test:

Omega:
  Tendencies:
    VelDiffTendencyEnable: false
    VelHyperDiffTendencyEnable: false
    UseCustomTendency: true
    ManufacturedSolutionTendency: true
  IOStreams:
    InitialVertCoord:
      Filename: init.nc
    InitialState:
      UsePointerFile: false
      Filename: init.nc
      Contents:
      - NormalVelocity
      - PseudoThickness
    RestartRead: {}
    History:
      Filename: output.nc
      Freq: {{ output_freq }}
      FreqUnits: Seconds
      IfExists: append
      # effectively never
      FileFreq: 9999
      FileFreqUnits: years
      Contents:
      - NormalVelocity
      - PseudoThickness
      - SshCell

For more details on updating either the map or the YAML files for individual tests, see the Polaris Ocean Framework documentation.

Coupled Testing with CIME

Because Omega is the ocean component of E3SM, changes must be validated in coupled configurations using E3SM’s CIME test harness.

Note

All CIME creation commands shown below (create_test, create_newcase, etc.) are executed from the cime/scripts/ directory relative to the top-level E3SM repository root, not from within the components/omega/ subdirectory.

For general CIME testing concepts, options, and advanced workflows, refer to the comprehensive CIME System Testing documentation. This section focuses on the specific configurations, conventions, and knowledge an Omega developer needs.

Why Coupled Testing is Essential

Coupled testing exercises critical behaviors that cannot be validated in standalone runs:

  • Flux exchanges: Validates ocean-atmosphere and ocean-ice boundary flux exchanges (SfcCoupling and the MCT/MOAB bridges).

  • Coupler clock stepping: Ensures ocean time stepping advances correctly under coupler clock alarms and coupling intervals.

  • Exact restart reproducibility (ERS): Verifies bit-for-bit restart capability under CIME driver control.

  • PE layout independence (PEM/PET): Verifies bit-for-bit results across different processor element (PE, i.e., MPI task) layouts and thread counts.

Sourcing the CIME Environment

Important

Required environment for running Omega under CIME: Prior to Omega being added to E3SM, developers did not need to source a special environment before running CIME commands; legacy components relied exclusively on the host system’s base Python interpreter.

However, Omega uses YAML configuration files instead of traditional Fortran namelists. To generate omega.yml, CIME invokes Omega’s components/omega/cime_config/buildnml in-process using Python. This requires the pyyaml package, which is not part of the Python standard library.

This generation step occurs during case setup and inside batch jobs during case.run (including between segments of ERS and PEM tests). Sourcing the machine’s environment script beforehand ensures pyyaml remains available at all stages of execution.

A proposed standard developer environment (E3SM Discussion #8906) aims to make sourcing an environment a standard, required step for running all E3SM v4 components. Until that is fully adopted in E3SM, developers must explicitly source the CIME environment script for their machine before invoking any CIME command:

Machine

Environment Script

aurora

source /lus/flare/projects/E3SMinput/soft/e3sm-unified/load_latest_cime_env.sh

chrysalis

source /lcrc/soft/climate/e3sm-unified/load_latest_cime_env.sh

frontier

source /ccs/proj/cli115/software/e3sm-unified/load_latest_cime_env.sh

perlmutter (pm-cpu, pm-gpu)

source /global/common/software/e3sm/anaconda_envs/load_latest_cime_env.sh

Omega Compsets and Grid Compatibility

Omega compsets are grouped into three main tiers (defined in components/omega/cime_config/config_compsets.xml and cime_config/allactive/config_compsets.xml):

  • C Cases (Data Atmosphere & Data Sea Ice): COMEGA-IAF, COMEGA-JRA1p5

    • Lightweight tests for atmospheric and runoff forcing ingestion and ocean state evolution without active ice or atmosphere components.

  • G Cases (Active Sea-Ice MPASSI): GOMEGA-IAF, GOMEGA-JRA1p5

    • Tests bidirectional ice-ocean coupling, ice-ocean fluxes, and frazil ice.

  • B Cases (Fully Coupled): WCYCL1850NS-OMEGA

    • Fully active simulation across all Earth system components (EAM, ELM, MOSART, MPASSI, OMEGA).

Important

Data atmosphere (DATM) grid and forcing compatibility: In C and G compsets, the atmosphere component is a data atmosphere (DATM) reading prescribed atmospheric forcing files, not a prognostic atmosphere component (such as EAM, which runs on spectral element grids like ne4pg2 or ne30pg2). For DATM, the atmosphere grid identifier specifies the resolution of the prescribed input data files:

  • IAF (Interannual Forcing) compsets use the T62 atmospheric forcing grid (e.g., T62_oQU240).

  • JRA1p5 compsets use the TL319 atmospheric forcing grid (e.g., TL319_EC30to60E2r2).

Attempting to pair T62 with a JRA1p5 compset (or TL319 with IAF) is invalid and will fail during case setup or input data checking. Refer to the e3sm_omega_developer test suite in cime_config/tests.py for verified compset, resolution, and test modifier combinations (such as .omega-jra_1958).

Running Coupled Tests with create_test

Automated coupled tests are run using CIME’s create_test script located in cime/scripts.

Test Name Syntax

CIME encodes the test configuration into a standardized name format (see CIME Test Name Syntax):

TESTTYPE[_MODIFIERS].GRID.COMPSET[.MACHINE_COMPILER][.GROUP-TESTMODS]
  • TESTTYPE: The core test type without underscores (e.g., ERS, PEM, SMS). See CIME Test Types.

  • _MODIFIERS: Optional modifiers prefixed with underscores to alter settings (e.g., _D enables DEBUG=TRUE, _Ld3 sets run length to 3 days). See CIME Modifiers.

  • GRID: Atmosphere and ocean grid alias or longname (e.g., T62_oQU240, ne4pg2_oQU240).

  • COMPSET: The compset alias (e.g., COMEGA-IAF, GOMEGA-IAF, WCYCL1850NS-OMEGA).

  • MACHINE_COMPILER: Optional machine and compiler joined by an underscore (e.g., chrysalis_intel, pm-gpu_gnugpu). Can alternatively be passed via CLI flags (-m <machine> -c <compiler>).

  • GROUP-TESTMODS: Optional directory with namelist or XML modifications (see CIME Test Mods). Omega’s test mods are located under components/omega/cime_config/testdefs/testmods_dirs/omega/ (for example, omega-jra_1958 sets the atmospheric and runoff forcing start/end years to 1958 for JRA-55 simulations).

Note

Machine and compiler detection: Specifying the machine and compiler is optional. If omitted, create_test automatically detects the host machine and uses that machine’s default compiler.

The primary exception is platforms like NERSC Perlmutter, where both pm-cpu and pm-gpu machine profiles are accessible from the same login environment. In that case, you must deliberately specify the machine (e.g., append .pm-gpu or pass -m pm-gpu) to target GPUs rather than CPU nodes.

The e3sm_omega_developer Test Suite

The e3sm_omega_developer test suite defined in cime_config/tests.py is maintained primarily for automated nightly testing in E3SM and serves as a living catalog of valid compset, grid, and modifier combinations. Developers are not expected to run the entire suite for every PR given the variety of supported platforms. However, it can be run when comprehensive multi-case coverage is desired on a supported machine:

cd cime/scripts
./create_test e3sm_omega_developer
# Or explicitly specifying machine and compiler if desired:
./create_test e3sm_omega_developer -m <machine> -c <compiler>

Monitoring, Diagnosing, and Iterating on Failed Tests

Each test generated by create_test creates a standalone case directory under your configured test scratch root (named after the test, e.g., ERS_D.T62_oQU240.COMEGA-IAF.<test-id>). Test status across all phases is tracked in TestStatus and TestStatus.log.

If a test fails, you do not need to set up a new manual case to diagnose or fix the issue:

  1. Navigate directly into the generated test case directory.

  2. Inspect TestStatus to identify the failing phase (e.g., MODEL_BUILD, RUN, COMPARE_base_rest).

  3. Check the logs:

    • For build errors: inspect bld/ocn.bldlog.*.

    • For runtime or comparison errors: inspect logs in the test’s run/ directory (e3sm.log.*, ocn.log.* , cpl.log.*).

  4. Fix the bug directly in the repository source code checkout used to set up the test.

  5. Rebuild and resubmit right from within the existing test case directory:

    # Recompile only modified components
    ./case.build -m
    
    # Rerun the test
    ./case.submit
    

Interactive Coupled Cases

For active feature development or exploring new configurations, creating an interactive case from scratch on Chrysalis is straightforward:

# 1. Source the CIME environment on Chrysalis
source /lcrc/soft/climate/e3sm-unified/load_latest_cime_env.sh

# 2. From the E3SM repository root, navigate to cime/scripts and create a case on Chrysalis
cd cime/scripts
./create_newcase --case /lcrc/group/e3sm/ac.username/cases/omega_test --compset COMEGA-IAF --res T62_oQU240 --mach chrysalis

# 3. Setup the case (invokes Omega's buildnml to generate omega.yml)
cd /lcrc/group/e3sm/ac.username/cases/omega_test
./case.setup

# 4. (Optional) Enable debug compiler flags or customize configuration
./xmlchange DEBUG=TRUE

# You can customize Omega configuration by editing user_nl_omega:
# (see user guide: {ref}`omega-user-config`)
./preview_namelists

# 5. Build the case
./case.build

# 6. Submit the case
./case.submit
  • Fast rebuilds: Run ./case.build -m to recompile only modified components without rebuilding external libraries.

  • Resetting setup: Run ./case.setup --reset after modifying grid resolutions, component layouts, or processor element (PE) counts.

What to Include in Your PR Testing Comment

Please add a PR comment titled “Testing” that summarizes what you ran and the outcomes. Both standalone and coupled testing must be reported for non-trivial PRs:

Important

At a minimum, testing must be performed on at least one CPU and one GPU machine/compiler combination, covering:

  1. CTest unit tests

  2. Polaris omega_pr regression suite (standalone)

  3. Targeted coupled CIME tests (such as ERS or PEM tests)

  • CTest unit tests:

    • Machine(s) and partition/queue (if applicable)

    • Compiler(s), and build type(s) (Debug/Release)

    • Result: either “All tests passed” or a short list of failing tests with a one-line note each

  • Polaris omega_pr regression suite (standalone):

    • Baseline run (Polaris e3sm_submodules/Omega) build path (-p) and workdir (-w)

    • PR branch run build path (-p) and workdir (-w)

    • Machine/partition, compiler, and build type (should match between baseline and PR)

    • Result: “All tests passed” or a concise list of diffs/failures with a brief note

    • Log paths (e.g., polaris_omega_pr.o$JOBID) if issues occur

  • Coupled CIME tests:

    • Machine(s) and compiler(s) (highlighting GPU platforms if applicable)

    • Test name(s) (e.g., ERS_D.T62_oQU240.COMEGA-IAF, PEM_D.T62_oQU240.COMEGA-IAF)

    • Result: “All tests passed (PASS)” or note on failures/phases

    • Log paths (e.g., TestStatus.log) if issues occur

  • Tests added/modified/impacted:

    • Briefly list any new or updated CTest, Polaris, or CIME tests and what they cover

  • Performance PRs (if applicable):

    • Link to the relevant PACE experiment and summarize before/after metrics

Use the following template for your PR comment:

## Testing

CTest unit tests
- Machine: <machine>[, <partition>]
- Compiler: <compiler>
- Build type: <Debug|Release>
- Result: All tests passed | Failures (X of Y): <name> (<one-line note>), ...

Polaris omega_pr regression suite (standalone)
- Baseline build (-p): <path/to/build of Polaris e3sm_submodules/Omega>
- Baseline workdir (-w): <path/to/baseline_omega_pr>
- PR build (-p): <path/to/pr/build>
- PR workdir (-w): <path/to/my_branch_omega_pr>
- Machine/partition: <machine>[, <partition>]
- Compiler/build type: <compiler>, <Debug|Release>
- Result: All tests passed | Diffs/Failures: <brief summary>
- Logs (if applicable): <path/to/polaris_omega_pr.o$JOBID>

Coupled CIME tests
- Machine: <machine>
- Compiler: <compiler>
- Test(s): <e.g. ERS_D.T62_oQU240.COMEGA-IAF, PEM_D.T62_oQU240.COMEGA-IAF>
- Result: All tests passed | Failures (X of Y): <test name> (<phase>), ...
- Logs (if applicable): <path/to/TestStatus.log>

Tests added/modified/impacted
- CTest: <name> — <one-line purpose>
- Polaris: <path or name> — <one-line purpose>
- CIME: <test or compset> — <one-line purpose>

Performance (if applicable)
- PACE: <link>
- Before: <metric(s)>
- After: <metric(s)>
- Delta: <summary>