Testing zppy

Unit tests

Unit tests live in tests/test_*.py. Run them with:

pip install .  # or: python -m pip install .
pytest tests/test_*.py

Integration tests

Integration tests can be run on Chrysalis, Compy, or Perlmutter. They require an active E3SM Unified environment and access to simulation data.

Machine-specific directions (generated setup commands, expected output files, etc.) live in: tests/integration/generated/

Run all integration tests:

pip install .
pytest tests/integration/test_*.py

Note

Image comparison tests require a compute node:

salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
pytest tests/integration/test_images.py
cat test_images_summary.md

Automated tests (CI)

Unit tests run automatically via GitHub Actions on every pull request and push to main. Integration tests must be run manually on a supported machine.

Full release testing directions

This section describes the complete process for testing zppy before making a release. See also Testing zppy for E3SM Unified release for a legacy tutorial.

Overview

Three testing scenarios exist:

  1. test dev — run before making a new zppy RC (no new Unified RC yet)

  2. test new Unified RC — run when a new E3SM Unified RC has been made

  3. test final Unified — run after E3SM Unified is finalized

Tests should be run on three machines: Chrysalis, Compy, and Perlmutter.

Step 1: Set up zppy-interfaces (if needed)

Update zppy-interfaces to the latest main branch:

cd ~/ez/zppy-interfaces
git status         # should be clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 1
# Verify matches https://github.com/E3SM-Project/zppy-interfaces/commits/main

bash  # isolated subshell
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>_chrysalis.sh  # e.g., 1.13.0rc10

# Verify zppy-interfaces tests
pytest tests/unit/global_time_series/test_*.py
pytest tests/unit/pcmdi_diags/test_*.py

Step 2: Check out the zppy branch

cd ~/ez/zppy
git status  # ensure clean workspace

git fetch upstream main
git checkout -b test-zppy-<description>-<machine> upstream/main
git log --oneline | head -n 3
# Verify matches https://github.com/E3SM-Project/zppy/commits/main

Step 3: Set up the environment

  1. test dev: create a new dev environment

    mamba clean --all
mamba env create -f conda/dev.yml -n zppy_dev_pre_<version>rc<N>
conda activate zppy_dev_pre_<version>rc<N>
pip install .

  2. test new Unified RC: activate the Unified RC

    • Chrysalis: source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>rc<N>_chrysalis.sh

    • Compy: source /share/apps/E3SM/conda_envs/test_e3sm_unified_<VERSION>rc<N>_compy.sh

    • Perlmutter: source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_<VERSION>rc<N>_pm-cpu.sh

  3. test final Unified: activate the latest released Unified

    • Chrysalis: source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh

    • Compy: source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh

    • Perlmutter: source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh

Step 4: Configure tests/integration/utils.py

Edit tests/integration/utils.py to set the correct environment and unique ID. Example TEST_SPECIFICS dictionary:

TEST_SPECIFICS: Dict[str, Any] = {
    # NCO path: "" uses production-version NCO commands
    # Set to a specific path to use development-version NCO commands
    "nco_path": "",
    # Environment commands for specific tasks.
    # Never set these to ""; that would override the task's environment
    # with a blank string (no environment).
    "e3sm_to_cmip_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>rc<N>_chrysalis.sh",
    "diags_environment_commands":         "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>rc<N>_chrysalis.sh",
    "mpas_analysis_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>rc<N>_chrysalis.sh",
    "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>rc<N>_chrysalis.sh",
    "livvkit_environment_commands":       "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>rc<N>_chrysalis.sh",
    "pcmdi_diags_environment_commands":   "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>rc<N>_chrysalis.sh",
    # General environment for other tasks.
    # Leave as "" to use the latest Unified environment.
    "environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>rc<N>_chrysalis.sh",
    # For a complete test, run the set of latest cfgs AND at least one set of legacy cfgs
    "cfgs_to_run": [
        "weekly_bundles",
        "weekly_comprehensive_v2",
        "weekly_comprehensive_v3",
        "weekly_legacy_3.1.0_bundles",
        "weekly_legacy_3.1.0_comprehensive_v2",
        "weekly_legacy_3.1.0_comprehensive_v3",
        "weekly_legacy_3.0.0_bundles",
        "weekly_legacy_3.0.0_comprehensive_v2",
        "weekly_legacy_3.0.0_comprehensive_v3",
    ],
    "tasks_to_run": [
        "e3sm_diags",
        "mpas_analysis",
        "global_time_series",
        "ilamb",
        "livvkit",
        "pcmdi_diags",
    ],
    "unique_id": "zppy_<branch>_<version>rc<N>_<description>",
}

Step 5: Run unit tests

pytest tests/test_*.py

All tests should pass. Failures in scenario (a) require code fixes. Failures in scenario (c) may indicate a patch release of E3SM Unified is needed.

Step 6: Generate cfg files and check quota

python tests/integration/utils.py
# Review output to confirm settings

# Check quota before launching
mmlsquota -u <username> --block-size T fs2
lcrc-quota

ls tests/integration/generated/test_weekly_*_chrysalis.cfg

Step 7: Launch integration jobs

To ensure no conflicts, wait for all jobs from one zppy run to finish before launching another. Start with the most important:

# Launch all cfg files
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg

zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg

# Monitor job queue
sq | wc -l  # subtract 1 for the header row

Step 8: Run bundles a second time

After all initial jobs complete, check for errors and run the bundles a second time to complete remaining work:

# Check for errors first
cd /path/to/output/.../post/scripts
grep -v "OK" *status   # Good if no output

# Re-run bundles
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg

Step 9: Review all finished runs

For each output directory, check for failures:

cd /path/to/zppy_weekly_comprehensive_v2_output/.../post/scripts
grep -v "OK" *status

cd /path/to/zppy_weekly_comprehensive_v3_output/.../post/scripts
grep -v "OK" *status

cd /path/to/zppy_weekly_bundles_output/.../post/scripts
grep -v "OK" *status

# Repeat for all legacy output directories

Step 10: Run Python integration tests

pytest tests/integration/test_bash_generation.py
pytest tests/integration/test_campaign.py
pytest tests/integration/test_defaults.py
pytest tests/integration/test_bundles.py
pytest tests/integration/test_last_year.py

For image comparison tests (requires a compute node):

salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_<VERSION>_chrysalis.sh
pytest tests/integration/test_images.py
cat test_images_summary.md

Step 11: Wrap up

After successful testing:

  1. test dev: proceed to create the zppy RC.

  2. test new Unified RC: proceed to create the zppy production release.

  3. test final Unified: run the “Commands to generate official expected results for a release” from the machine-specific directions. Then clean up branches and environments:

    git branch
git branch -D <branch-to-delete>

conda env list
conda remove -n <env-name> --all

    Update expected test result files if there are known/expected differences: use the “Commands to run to replace outdated expected files” from the machine-specific directions, then re-run the tests.

Step 12: Record test results

Document test results in the zppy testing discussion for the relevant E3SM Unified release. Include:

  • Machine name and date

  • E3SM Unified version tested

  • zppy version/branch tested

  • Results summary (pass/fail per cfg)

  • Any known issues or caveats