Field guide · post-processing

Running zppy on EAMxx output

EAMxx (SCREAMv1) writes its history files with different variable names, different stream names, and a different vertical grid than EAM. This guide shows what to change in your zppy cfg to post-process it — and what zppy and e3sm_diags now do under the hood to make EAM-era diagnostics work on EAMxx data.

Machine Perlmutter Example post.v3.eamxx.cfg zppy #424 · #827 · #828 e3sm_diags main

What's actually different about EAMxx

Four things break the EAM assumptions baked into the old post-processing path. Everything else in this guide follows from them.

Naming

New variable names

Descriptive, not cryptic: SeaLevelPressure instead of PSL, T_mid_at_500hPa instead of T500.

Files

New stream names

History files are *.AVERAGE.nmonths_x1, .ndays_x1, .nhours_x6 — not eam.h0.

Metadata

No ps / hybrid auto-extract

ncclimo --prc_typ=eamxx won't pull out ps, hyai/hyam/hybi/hybm for you the way cam/eam does.

Vertical

L128 grid, no topo hint

Vertical remap needs an explicit vert_L128.nc, and topography_file="NONE" defeats grid inference.

The pipeline, and where the env comes from

Same five zppy tasks as an EAM run. Only the first task's component tag and a handful of parameters change; each task activates the right environment automatically.

task

climo

ncclimo climatologies from monthly & diurnal streams.

env · E3SM-Unified

task

ts

Time series + selective vertical regrid of 3-D vars.

env · E3SM-Unified

optionaltask

e3sm_to_cmip

CMORize the time series to CMIP variables. Skip it unless you need CMIP-format output — e3sm_diags doesn't depend on it.

env · E3SM-Unified

task

tc_analysis

TempestExtremes TC tracking from 6-hourly data.

env · E3SM-Unified

task

e3sm_diags

Model-vs-obs diagnostics, incl. COSP & TC sets.

env · e3sm_diags_main

What changes in your cfg

These are the only edits an EAM cfg needs to become an EAMxx cfg. Each block is lifted from examples/post.v3.eamxx.cfg; the why notes explain the mechanism behind it.

Tag the component as EAMxx

input_component

[climo] / [ts] subsectioneamxx

input_component = "eamxx"
input_files = "AVERAGE.nmonths_x1"
input_subdir = "run/"

whyzppy's set_component_and_prc_typ() now accepts eamxx as a first-class value → component=atm, prc_typ=eamxx. That single tag drives ncclimo --prc_typ=eamxx and gates the EAMxx-specific remap flags downstream — no more inferring the component from a fragile eam.h0 filename convention.

List ps & hybrid coefficients explicitly

extra_vars

[ts] atm_monthly_180x360_aave

# ncclimo --prc_typ=eamxx does NOT auto-extract these the way cam/eam does.
# ps is load-bearing: the vertical-remap step passes --ps_nm=${file}/ps.
extra_vars = "ps,hyai,hyam,hybi,hybm,area,landfrac,ocnfrac"

whySurface pressure and the hybrid-coordinate coefficients define the model's vertical levels. EAM's ncclimo carries them along automatically; EAMxx's does not, so you carry them yourself. If ncclimo is fixed upstream to mirror cam/eam, this line can be dropped.

Vertically regrid 3-D vars inside the ts task

vrt_remap_vars

[ts] atm_monthly_180x360_aavenew in #827

vars = "...,U,V,T_mid,omega,RelativeHumidity,qv"
vrt_remap_vars = "U,V,T_mid,z_mid,omega,RelativeHumidity,p_mid,qv"

whyListed 3-D vars are vertically remapped (model levels → pressure levels) and written to a sibling ts_vrt_remap/ directory alongside the standard ts/. When prc_typ=eamxx, zppy auto-emits --ps_nm=${file}/ps and --vrt_in=vert_L128.nc to ncremap. Downstream e3sm_to_cmip then sees one clean ts subsection instead of a split 2-D/3-D setup.

Give TC analysis a grid and its seven vars

input_gridtc_vars

[tc_analysis]new in #828

input_grid = "ne30pg2"   # --res and the pg2 flag are derived from this
# Order: SLP, T@200, T@500, U@bot, V@bot, U@850, V@850
tc_vars = "SeaLevelPressure,T_mid_at_200hPa,T_mid_at_500hPa,
          U_at_model_bot,V_at_model_bot,U_at_850hPa,V_at_850hPa"

whyThe old TC path inferred resolution & the pg2 flag from a sample file's topography_file attribute — but EAMxx sets it to "NONE", so inference fails. input_grid replaces that guess. tc_vars maps EAMxx's seven tracking variables onto the canonical PSL,T200,T500,UBOT,VBOT,U850,V850 order TempestExtremes expects. EAM cfgs keep working: the legacy inference and default tc_vars are still there.

Fix days-per-file for daily & high-frequency streams

dpftpd

[ts] atm_daily_180x360_aave

# Each ndays_x1 file holds ONE daily average (1 day, 1 timestep).
# Override zppy's monthly-h0 defaults (dpf=30, tpd=1), or ncclimo
# treats each daily file as 30 days and mislabels the ts output.
dpf = 1
tpd = 1

whyzppy's defaults (dpf=30, tpd=1) assume EAM monthly h0 files. Any non-monthly EAMxx stream — daily (ndays_x1) or high-frequency (nhours_x3/x6) — packs far fewer days per file, so days_per_file and timesteps_per_day must be told the truth or ncclimo mislabels the output.

The variable translation

You mostly don't write these mappings — e3sm_diags derives the canonical diagnostic variable from EAMxx names. Handy for reading logs, choosing vars lists, and understanding what a set needs.

EAMxx variable(s)	Canonical	How e3sm_diags gets there
SeaLevelPressure	→ PSL	rename
T_mid_at_200hPa · T_mid_at_500hPa	→ T200 · T500	rename (pressure-level slices carried through)
precip_liq_surf_mass_flux + precip_ice_surf_mass_flux	→ PRECT	summed & unit-converted (liquid + ice surface flux)
ShortwaveCloudForcing · LongwaveCloudForcing	→ SWCF · LWCF	rename
SW_flux_dn_at_model_top	→ SOLIN	rename; also feeds an albedo derivation with the up flux
VapWaterPath	→ PREH2O	rename
surf_radiative_T (+ ocnfrac)	→ TS	derived surface temperature
isccp_cldtot · cosp_prs · cosp_htmisr	→ COSP histograms	v2 COSP: `cosp_prs` in Pa (not hPa), `cosp_htmisr` in m (not km) — handled in the set

What changed under the hood

You don't have to touch any of this to run the example — but here's what the two packages had to learn so EAM-era diagnostics run on EAMxx data.

zppy

Post-processing & task wiring

zppy/utils.py · templates/ts.bash · templates/tc_analysis.bash · e3sm_to_cmip.py

eamxx as a real component#424

set_component_and_prc_typ() recognizes eamxx → atm/prc_typ=eamxx, replacing filename-based detection.

ts-level vertical regrid#827

vrt_remap_vars writes remapped 3-D vars to ts_vrt_remap/; when prc_typ=eamxx, ncremap gets --ps_nm=${file}/ps and --vrt_in=vert_L128.nc automatically.

EAMxx-aware CMORization#827

e3sm_to_cmip supplies the explicit L128 source vertical grid when prc_typ=eamxx, since EAMxx has no auto-carried vertical metadata.

Robust TC tracking#828

input_grid derives --res + pg2 (no topography inference); tc_vars remaps the seven tracking vars. Both EAM and EAMxx supported.

e3sm_diags

Reading & deriving EAMxx variables

derivations/derivations.py · formulas_cosp.py · driver/utils/dataset_xr.py

EAMxx naming & monthly averages#712

Teaches the reader EAMxx's variable-name convention and its time stamping (a January average stamped at Feb 1 is shifted back).

Derivation coverage#880#983#1041

Adds EAMxx source names across the derivation table (PRECT, TS, SWCF/LWCF, SOLIN, AODvis…) plus numpy-2.4 and missing-units fixes.

More sets on EAMxx#982#1046

tropical_subseasonal and precip_pdf updated to consume EAMxx output.

COSP histograms#1035

The cosp_histogram set understands EAMxx / COSP v2 fields and their unit differences (Pa vs hPa, m vs km).

Run it yourself

The example is your template: download it, point it at your own case and environments, then submit. It targets Perlmutter, but everything you edit lives in a few places, so it ports easily.

Get the example config

Grab examples/post.v3.eamxx.cfg from the zppy repository and keep your own copy to edit:

https://github.com/E3SM-Project/zppy/blob/main/examples/post.v3.eamxx.cfg

shell

# from a zppy checkout
cp examples/post.v3.eamxx.cfg my.eamxx.cfg
# …or download it straight from the repo, then edit my.eamxx.cfg

Point it at your run

Edit [default] for your case, set the [e3sm_diags] environment, and turn off any task you don't need — e3sm_to_cmip is optional, since e3sm_diags doesn't consume its output.

my.eamxx.cfg

[default]
input  = /path/to/your/eamxx/case
output = /path/to/your/post-processing/output
www    = /your/web/portal/dir
case   = <your_case_name>

[e3sm_to_cmip]
active = False   # optional — only for CMIP-format output

[e3sm_diags]
environment_commands = "source <conda>.sh; conda activate <your_e3sm_diags_env>"

also checkthe mapping_file for your grid, the years / reference-year ranges, and the e3sm_diags sets list — drop any set whose input stream you don't produce.

Use a local zppy build and submit

The EAMxx support (PRs #424 · #827 · #828) isn't in a released E3SM-Unified yet, so you need a zppy built from main. On Perlmutter you can use the shared build zppy_dev_main directly. Each task still activates its own env automatically.

shell — shared build on Perlmutter

source /global/cfs/cdirs/e3sm/zhang40/miniconda3/etc/profile.d/conda.sh
conda activate zppy_dev_main
zppy -c my.eamxx.cfg

shell — or build your own

git clone https://github.com/E3SM-Project/zppy.git
cd zppy
mamba env create -f conda/dev.yml -n zppy_dev
conda activate zppy_dev
pip install .
zppy -c ../my.eamxx.cfg

Re-running is safe. zppy skips any task whose .status is already OK. To force one task to rerun, delete its .status file and resubmit.

Minimum viable cfg — diagnostics only

climo · ts · e3sm_diags

Just enough for model-vs-obs plots — no CMORization, no TC tracking. Fill in your paths, env, and mapping file.

min.eamxx.cfgeamxx

[default]
input  = /path/to/your/eamxx/case
output = /path/to/output
www    = /your/web/dir
case   = <your_case_name>

[climo]
active = True
years  = "1995:2004:10",
  [[ atm_monthly_180x360_aave ]]
  input_component = "eamxx"
  input_files = "AVERAGE.nmonths_x1"
  mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc

[ts]
active = True
years  = "1995:2004:5"
ts_num_years = 5
  [[ atm_monthly_180x360_aave ]]
  input_component = "eamxx"
  input_files = "AVERAGE.nmonths_x1"
  mapping_file = map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  # EAMxx ncclimo won't auto-extract these:
  extra_vars = "ps,hyai,hyam,hybi,hybm,area,landfrac,ocnfrac"

[e3sm_diags]
active = True
years  = "1995:2004:10",
# local e3sm_diags build (main branch — has EAMxx support):
environment_commands = "source /global/cfs/cdirs/e3sm/zhang40/miniconda3/etc/profile.d/conda.sh; conda activate e3sm_diags_main"
  [[ atm_monthly_180x360_aave ]]
  sets = "lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram",
  grid = '180x360_aave'