Inside SamudrACE C1A0

Circuits, law witnesses and error agreement for the E3SMv3 coupled emulator (SFNO atmosphere + Samudra ocean), finished at scale on Perlmutter against E3SM truth, scored only on years the model never saw (1995–1999, 2040–2049). Rules were preregistered before results (PREREG.md). This page is rebuilt from the result JSONs. Participants: start with HACKATHON.md and the repo.

Phase 0 ✔F1 truth test ✔F1 model vs truth ✔F2 localisation ✔F3 ablation ✔Idea 3 gain ✔Idea 1 coupled de-leak · Idea 2: Phase 2

Headline so far

verified P0-1 · The laptop FLUS-thermometer number reproduces on Perlmutter (A100, fp32).
-2.042 K per 100 W/m² of warming flux [-2.076, -2.012] over 73 states of 1985; wrong sign on 99.85% of open ocean (laptop: −2.04 [−2.08, −2.01], 99.8%)
Control: slab law gives +0.33; FLDS/FSNS have the law's sign (0.33 / 0.28)
Caveat: 1985 is a TRAINING year; uniform open-ocean shift (a 10×10° box gives ~2×)
verified I1-F1a · In E3SM truth, the ocean's FLUS input is a near-exact thermometer of the SST it must predict (a target leak in the data).
conditional slope -17.0 K per 100 W/m² of warming flux (literal 1/(4εσT³): −17.3); thermometer sign on 99.5% of open ocean; per-cell R² median 0.9983 (2045–49)
Control: the other four fluxes have conditional slopes ≈ 0 (medians |·| < 0.003)
Caveat: per-cell OLS with intercept; the preregistered alignment check (end vs start of window, pooled RMS) does NOT show the 20% margin (22.21 vs 22.28 W/m²) because a spatial offset dominates pooled RMS
verified I1-F1b · C1A0 does not copy the truth thermometer: it uses about a tenth of it, with an unrelated spatial pattern, and keeps lawful-sign responses to the other fluxes.
model/truth slope ratio 0.109; pattern correlation +0.03 (-0.02 without ice-shelf cavities); model FLUS -1.84 vs truth -16.93 K per 100 W/m² (2045 states vs 2045–49 truth)
Control: same result on 1995 states (ratio 0.109, r = 0.07); FLDS/FSNS model slopes +0.32/+0.29 vs truth ≈ 0
Caveat: model states are coupled-run inputs from truth ICs (not truth-forced); uniform shifts only for this comparison
verified I1-F2a · Below “skip.0 + layer 0”: FLUS splits about evenly between layer 0's linear 1×1 skip_module and its InstanceNorm-ed convblock, whereas an SST perturbation goes 100% through the 1×1 skip_module.
FLUS uniform: skip.0 0.86, skip_module 0.53, convblock 0.47; box: skip.0 0.91, skip_module 0.35, convblock 0.65; sst: skip_module 1.00
Control: projection metric agrees within 0.05 for FLUS. Skeptic pass: an independent implementation (exege's new samudra Patchable adapter, EdgePatch semantics) gives skip.0 share 0.854 on 7 states of 2045 (localize.py: 0.86)
Caveat: near-linear response: sufficiency = necessity, shares sum to ≈1 (one result per split)
narrowed I1-F2b · InstanceNorm statistics account for the box-vs-uniform difference of the FLUS response.
freezing the convblock's IN statistics raises its uniform-shift share 0.47 → 0.94 (box: 0.65 → 0.65); predicted un-normalised uniform slope -2.65 vs measured box -2.58 K per 100 W/m² (uniform -1.80)
Control: the same freeze leaves box shares unchanged, as expected if a box barely moves whole-field statistics
Caveat: holds for FLUS only: for FLDS the same prediction gives 1.34× vs measured box/uniform 1.07×; box ratio here is 1.43× (laptop said ~2×)

Setup, data and checks

Model and lineage

CheckpointHF mahf708/e3smv3-lr-hist-samudrace@rc1 checkpoints/C1A0.tar, md5 a5082a9aec1243e37e0820a99e77d059
Training runE35-CFT S01: 25 epochs coupled fine-tune (4 coupled steps, ensemble CRPS), EMA weights
AtmosphereNoiseConditioned SFNO, 8 blocks × 384, big skip, isotropic noise (32 ch); init E02-FT S01. 6 h step. Corrector: dry air, moisture budget, positivity
OceanSamudra U-Net 280/380/480/520, InstanceNorm; init E11-FT S01 (= the hackathon E11 archive ocean). 5-day step; FLUS is an input; corrector never touches sst
Couplingocean fluxes = atmosphere outputs averaged over 20 inner steps; the atmosphere sees the ocean's current SST
Other release checkpointsC0A0 = E30-CFT S03 (no CO₂; ocean E11-FT S03); C1A1 = E31-CFT S01 (CO₂ + aerosol; atmosphere E05-FT S01, ocean E11-FT S01, same as C1A0)

Truth data and splits

Runv3.LR.historical_0101.aigo: EAM 6-hourly, MPAS-O/SI 5-day means, 1940–2065, 1° Gaussian (C1A0's grid)
Train1940-01-06 … 1990-01-01 and 2000-01-06 … 2040-01-01
Validation1990–1994 (also used to pick the checkpoint)
Held out (scored here)1995–1999 and 2040–2049; fit years 1995–97 / 2040–44, score years 1998–99 / 2045–49
Laptop states1985, 2015 = training years; used only to reproduce laptop numbers

Exactness and reproducibility checks

checkresult
common.Graph vs Samudra.forward (A100)max |diff| 0.0 (bitwise)
SamudraReplay vs Samudra.forwardmax |diff| 0.0 (bitwise)
same-seed fme coupled steps (all tensors, all outputs)bitwise identical
de-leak switch at load time (CombinedFluxConfig FLUS := 5.67e-8·sst⁴)works without editing the checkpoint; ocean FLUS / emission median 0.9997; control − de-leak FLUS 5.6 W/m² RMS (step 0), 7.3 (step 1)
TF32 (CUDA default) vs fp32TF32: batch-of-4 vs single 0.021 K max; fp32: 4.3e-05 K; fp32 vs deployed TF32 output 0.06 K max
truth-forced ocean input builder vs fme packing of the 1995-01-01 truth ICall state channels identical; only the 9 flux channels differ (truth vs coupled-atmosphere fluxes), as expected
Reproduce
bash setup_env.sh
python -m hc.capture data/smoke/A --steps 2 --atm-pred
python -m hc.capture data/smoke/B --steps 2 --atm-pred
python -m hc.capture data/smoke/D --steps 2 --deleak
python phase0/smoke.py data/smoke/A data/smoke/B data/smoke/D
python phase0/numerics_check.py data/smoke/A

Results: smoke.json numerics_check.json flus_slope.json

Idea 1 · The ocean reads FLUS as an SST thermometer

Question. Raising upward longwave (FLUS) at the ocean's input warms its next SST (wrong sign on 99.8% of open ocean). Is this a target leak, i.e. is the training-time FLUS (a 5-day mean over the window being predicted, the coupler's emission from that window's SST) a legitimate predictor of the target? If so, does the model's response match the truth relation, and can it be removed without breaking the lawful fluxes?

verified I1-F1a · In E3SM truth, the ocean's FLUS input is a near-exact thermometer of the SST it must predict (a target leak in the data).
conditional slope -17.0 K per 100 W/m² of warming flux (literal 1/(4εσT³): −17.3); thermometer sign on 99.5% of open ocean; per-cell R² median 0.9983 (2045–49)
Control: the other four fluxes have conditional slopes ≈ 0 (medians |·| < 0.003)
Caveat: per-cell OLS with intercept; the preregistered alignment check (end vs start of window, pooled RMS) does NOT show the 20% margin (22.21 vs 22.28 W/m²) because a spatial offset dominates pooled RMS

F1 (a) · Alignment of FLUS with εσT⁴ in truth

FLUS[w] is the 5-day mean the ocean receives with window w (MPAS = what training fed; EAM = the atmosphere's own). ε fitted on fit years over open ocean, scored on score years. Five SST definitions: previous-window mean (what the ocean reads), target-window mean, window-mean TS⁴ (6-hourly EAM), instantaneous TS at window start and end.

blockFLUS sourceSST definitionεRMS (W/m²)pooled R²per-cell R² (median)
1990sFLUSprev_window_mean_sst[w-1]0.988322.280.84490.744
1990sFLUStarget_window_mean_sst[w]0.988422.210.84590.770
1990sFLUSwindow_mean_TS^40.994816.500.91490.944
1990sFLUSwindow_start_TS0.995616.770.91220.950
1990sFLUSwindow_end_TS0.995716.860.91130.951
1990sFLUS_eamprev_window_mean_sst[w-1]0.99579.170.96730.937
1990sFLUS_eamtarget_window_mean_sst[w]0.99588.830.96970.961
1990sFLUS_eamwindow_mean_TS^41.00031.120.99951.000
1990sFLUS_eamwindow_start_TS1.00104.500.99210.988
1990sFLUS_eamwindow_end_TS1.00114.480.99220.988
2040sFLUSprev_window_mean_sst[w-1]0.988722.370.85160.781
2040sFLUStarget_window_mean_sst[w]0.988722.290.85270.806
2040sFLUSwindow_mean_TS^40.994617.030.91400.946
2040sFLUSwindow_start_TS0.995417.270.91150.952
2040sFLUSwindow_end_TS0.995517.360.91060.954
2040sFLUS_eamprev_window_mean_sst[w-1]0.99618.910.97080.948
2040sFLUS_eamtarget_window_mean_sst[w]0.99628.530.97320.970
2040sFLUS_eamwindow_mean_TS^41.00031.150.99951.000
2040sFLUS_eamwindow_start_TS1.00104.630.99210.989
2040sFLUS_eamwindow_end_TS1.00114.590.99230.989
1980s_train_contrastFLUSprev_window_mean_sst[w-1]0.988322.290.84420.746
1980s_train_contrastFLUStarget_window_mean_sst[w]0.988322.220.84520.771
1980s_train_contrastFLUSwindow_mean_TS^40.994916.630.91320.946
1980s_train_contrastFLUSwindow_start_TS0.995716.880.91070.953
1980s_train_contrastFLUSwindow_end_TS0.995816.960.90970.954
1980s_train_contrastFLUS_eamprev_window_mean_sst[w-1]0.99579.000.96820.937
1980s_train_contrastFLUS_eamtarget_window_mean_sst[w]0.99578.660.97060.961
1980s_train_contrastFLUS_eamwindow_mean_TS^41.00031.120.99951.000
1980s_train_contrastFLUS_eamwindow_start_TS1.00104.520.99200.989
1980s_train_contrastFLUS_eamwindow_end_TS1.00114.490.99210.988

Reading: EAM's FLUS is exactly εσ⟨TS⁴⟩ over its window (RMS 1.1 W/m²). MPAS FLUS carries a spatially varying offset against every SST definition (pooled RMS 16–22 W/m²) that hides the time relation; per-cell R² favours the target window (0.770 vs 0.744 in the 1990s). The preregistered supporting criterion (end beats start by > 20% pooled RMS) is not met.

F1 (b) · The conditional slope in truth

Per open-ocean cell (ice-free in every window): ΔSST = sst[w] − sst[w−1] regressed on the five window-w fluxes, sst[w−1], annual + semiannual harmonics and an intercept (OLS, Newey–West lag 6). K per 100 W/m² of warming flux (FLUS, LHFLX, SHFLX enter with −1), so the thermometer sign is negative. The Frisch–Waugh restatement agrees to 5e-9 (one number, not two).

blockyearsfluxarea meanmedian5–95%area frac. negativearea frac. |z|>2
1990sscoreFLUS-17.17-16.952[-20.36, -15.87]0.9950.995
1990sscoreFLDS+0.02+0.003[-0.03, +0.05]0.3820.177
1990sscoreFSNS-0.00+0.002[-0.01, +0.02]0.2900.232
1990sscoreLHFLX-0.12+0.002[-0.01, +0.02]0.3370.280
1990sscoreSHFLX+0.08-0.000[-0.05, +0.05]0.4980.215
1990sfitFLUS-17.24-16.931[-20.36, -15.91]0.9950.995
1990sfitFLDS+0.00+0.003[-0.03, +0.03]0.4000.152
1990sfitFSNS-0.06+0.002[-0.01, +0.01]0.3260.196
1990sfitLHFLX-0.10+0.002[-0.01, +0.02]0.3330.273
1990sfitSHFLX+0.14-0.000[-0.04, +0.06]0.5070.190
2040sscoreFLUS-17.01-16.770[-20.18, -15.72]0.9950.995
2040sscoreFLDS-0.02+0.002[-0.03, +0.04]0.4260.171
2040sscoreFSNS+0.00+0.001[-0.01, +0.01]0.3430.200
2040sscoreLHFLX-0.18+0.002[-0.01, +0.02]0.2990.358
2040sscoreSHFLX+0.12-0.000[-0.06, +0.06]0.5120.246
2040sfitFLUS-17.13-16.816[-20.23, -15.72]0.9950.995
2040sfitFLDS-0.10+0.002[-0.04, +0.04]0.4190.190
2040sfitFSNS-0.02+0.001[-0.01, +0.01]0.3270.201
2040sfitLHFLX+0.56+0.002[-0.01, +0.02]0.2840.361
2040sfitSHFLX-0.21+0.000[-0.05, +0.06]0.4840.251
1980s_train_contrastscoreFLUS-17.25-16.988[-20.37, -15.86]0.9950.995
1980s_train_contrastscoreFLDS-0.05+0.003[-0.03, +0.04]0.3950.170
1980s_train_contrastscoreFSNS+0.00+0.002[-0.01, +0.01]0.3390.236
1980s_train_contrastscoreLHFLX+0.45+0.002[-0.01, +0.02]0.3290.247
1980s_train_contrastscoreSHFLX-0.19+0.000[-0.05, +0.05]0.4840.223
1980s_train_contrastfitFLUS-17.20-16.967[-20.27, -15.91]0.9950.995
1980s_train_contrastfitFLDS+0.01+0.001[-0.03, +0.03]0.4390.165
1980s_train_contrastfitFSNS-0.02+0.002[-0.01, +0.01]0.3300.190
1980s_train_contrastfitLHFLX-0.24+0.002[-0.01, +0.02]0.2970.310
1980s_train_contrastfitSHFLX+0.13-0.001[-0.06, +0.05]0.5510.200

F1 (c) · Model vs truth, and the preregistered verdict

verified I1-F1b · C1A0 does not copy the truth thermometer: it uses about a tenth of it, with an unrelated spatial pattern, and keeps lawful-sign responses to the other fluxes.
model/truth slope ratio 0.109; pattern correlation +0.03 (-0.02 without ice-shelf cavities); model FLUS -1.84 vs truth -16.93 K per 100 W/m² (2045 states vs 2045–49 truth)
Control: same result on 1995 states (ratio 0.109, r = 0.07); FLDS/FSNS model slopes +0.32/+0.29 vs truth ≈ 0
Caveat: model states are coupled-run inputs from truth ICs (not truth-forced); uniform shifts only for this comparison

Model: per-node slope of next SST to a uniform open-ocean shift of each flux (bare network, fp32, every 4th state of a 1-year coupled run started from truth). Verdict rule fixed in PREREG 1.1: leak supported iff truth thermometer area > 75% and model/truth ratio in [0.33, 3] and pattern correlation > 0.3; refuted if truth has the lawful sign over > 50%; otherwise something else to explain.

statesfluxmodel meantruth meanmodel/truthpattern rpattern r (no cavities)
t1995FLUS-1.865-17.0990.109+0.066-0.017
t1995FLDS+0.319+0.01422.430-0.022-0.019
t1995FSNS+0.286-0.004-72.567+0.005-0.001
t1995LHFLX+0.079-0.012-6.346+0.007+0.020
t1995SHFLX+0.133+0.001137.006-0.000+0.028
t2045FLUS-1.838-16.9350.109+0.029-0.024
t2045FLDS+0.315-0.014-22.622+0.018+0.017
t2045FSNS+0.289-0.002-144.371+0.006-0.008
t2045LHFLX+0.075-0.014-5.528+0.022+0.002
t2045SHFLX+0.137+0.0275.064+0.011+0.009
Left: truth conditional FLUS slope (≈ −1/(4εσT³): stronger in cold water). Middl
Left: truth conditional FLUS slope (≈ −1/(4εσT³): stronger in cold water). Middle/right: C1A0's input-response to FLUS and FLDS (t1995). Note the colour ranges: the model uses ~1/10 of the truth relation and its map is nearly flat.
Left: truth conditional FLUS slope (≈ −1/(4εσT³): stronger in cold water). Middl
Left: truth conditional FLUS slope (≈ −1/(4εσT³): stronger in cold water). Middle/right: C1A0's input-response to FLUS and FLDS (t2045). Note the colour ranges: the model uses ~1/10 of the truth relation and its map is nearly flat.

Interpretation (open). The data contain the leak in its strongest form, so a model that fit the linear truth relation would answer ~17 K per 100 W/m² to FLUS and ~0 to everything else. C1A0 does neither. One candidate: its ocean was fine-tuned in coupled mode (E35-CFT), where FLUS comes from the atmosphere driven by the ocean's start-of-window SST, so the leak is absent there. The replication across checkpoints (E11-FT = the same ocean before coupled fine-tuning; E11 seeds; E12–E16) tests this directly.

Reproduce
python idea1_flus/f1_extract.py 1995 1996 1997 1998 1999 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 1985 1986 1987 1988 1989
python idea1_flus/f1_analyze.py
python -m hc.evaluate data/states/t1995 --start 1995-01-01 --steps 73 --capture --atm-spectra && python -m hc.meta data/states/t1995
python -m hc.evaluate data/states/t2045 --start 2045-01-01 --steps 73 --capture --atm-spectra && python -m hc.meta data/states/t2045
python idea1_flus/f1_model_compare.py data/states/t1995 1990s t1995
python idea1_flus/f1_model_compare.py data/states/t2045 2040s t2045

Results: f1.json f1_compare_t1995.json f1_compare_t2045.json

F2 · Where the response goes: exact patching below “skip.0 + layer 0”

Layer 0 is the network's only reader of the input: skip_module (1×1 conv, no norm) + convblock (3×3 conv → InstanceNorm → GELU → 3×3 conv → InstanceNorm → GELU → 1×1 conv). Its output feeds the full-resolution skip (skip.0) and the deep path. Exact clean→corrupted patching over the factorial {skip_module, convblock, convblock with clean InstanceNorm statistics} × {skip.0, deep}; perturbations: each flux, sst, sst + 0–20 m temperature; uniform open-ocean and 10×10° boxes on the preregistered lattice.

t1995|uniform

perturbationslope (K/100 W/m²; K/K for sst)/FLDSskip.0 onlydeep onlyskip_module onlyconvblock onlyconvblock, IN frozen
FLUS-1.829 [-1.845, -1.808]-5.790.86 / 0.860.14 / 0.140.52 / 0.500.48 / 0.500.94 / 1.19
FLDS+0.316 [+0.309, +0.322]+1.000.75 / 0.740.25 / 0.260.78 / 0.770.22 / 0.230.55 / 0.55
FSNS+0.286 [+0.282, +0.289]+0.911.08 / 1.06-0.08 / -0.061.09 / 0.97-0.09 / 0.03-0.29 / -0.14
LHFLX+0.073 [+0.068, +0.080]+0.231.36 / 1.01-0.35 / -0.010.29 / 0.300.71 / 0.711.81 / 0.89
SHFLX+0.129 [+0.125, +0.133]+0.410.57 / 0.630.43 / 0.371.83 / 1.16-0.83 / -0.16-2.30 / -1.00
sstT0+0.499 [+0.498, +0.500]–0.89 / 0.890.11 / 0.111.00 / 0.990.00 / 0.01-0.04 / -0.03
sst+0.222 [+0.222, +0.223]–0.89 / 0.890.11 / 0.111.00 / 1.00-0.00 / 0.00-0.06 / -0.06

t1995|box

perturbationslope (K/100 W/m²; K/K for sst)/FLDSskip.0 onlydeep onlyskip_module onlyconvblock onlyconvblock, IN frozen
FLUS-2.572 [-2.971, -2.271]-7.840.91 / 0.910.09 / 0.090.35 / 0.300.65 / 0.700.65 / 0.70
FLDS+0.332 [+0.246, +0.443]+1.000.93 / 0.920.07 / 0.080.52 / 0.500.48 / 0.500.49 / 0.50
FSNS+0.255 [+0.164, +0.352]+0.770.99 / 0.950.01 / 0.061.31 / 0.67-0.31 / 0.33-0.31 / 0.33
LHFLX+0.115 [-0.001, +0.245]+0.330.89 / 0.940.11 / 0.060.05 / 0.280.95 / 0.720.95 / 0.72
SHFLX-0.132 [-0.269, -0.027]-0.401.12 / 0.90-0.13 / 0.10-1.91 / -0.402.90 / 1.402.91 / 1.40
sstT0+0.468 [+0.419, +0.516]–0.90 / 0.890.10 / 0.111.05 / 1.05-0.05 / -0.05-0.05 / -0.05
sst+0.205 [+0.183, +0.228]–0.89 / 0.890.11 / 0.111.08 / 1.07-0.08 / -0.07-0.08 / -0.07

t2045|uniform

perturbationslope (K/100 W/m²; K/K for sst)/FLDSskip.0 onlydeep onlyskip_module onlyconvblock onlyconvblock, IN frozen
FLUS-1.801 [-1.822, -1.786]-5.770.86 / 0.850.15 / 0.150.53 / 0.510.47 / 0.490.94 / 1.22
FLDS+0.312 [+0.306, +0.317]+1.000.75 / 0.740.25 / 0.260.79 / 0.780.21 / 0.220.55 / 0.56
FSNS+0.289 [+0.285, +0.293]+0.931.07 / 1.05-0.07 / -0.051.09 / 0.96-0.09 / 0.04-0.29 / -0.13
LHFLX+0.068 [+0.064, +0.076]+0.221.41 / 1.01-0.40 / -0.000.28 / 0.290.72 / 0.711.92 / 0.88
SHFLX+0.133 [+0.126, +0.139]+0.430.58 / 0.640.42 / 0.361.77 / 1.14-0.77 / -0.14-2.24 / -0.99
sstT0+0.500 [+0.498, +0.502]–0.89 / 0.890.11 / 0.110.99 / 0.990.01 / 0.01-0.04 / -0.03
sst+0.223 [+0.222, +0.224]–0.89 / 0.890.11 / 0.111.00 / 1.000.00 / 0.00-0.06 / -0.06

t2045|box

perturbationslope (K/100 W/m²; K/K for sst)/FLDSskip.0 onlydeep onlyskip_module onlyconvblock onlyconvblock, IN frozen
FLUS-2.582 [-2.947, -2.226]-7.820.91 / 0.900.09 / 0.090.35 / 0.300.65 / 0.700.65 / 0.70
FLDS+0.334 [+0.247, +0.458]+1.000.93 / 0.910.07 / 0.090.52 / 0.500.48 / 0.500.48 / 0.50
FSNS+0.256 [+0.170, +0.348]+0.770.98 / 0.940.01 / 0.061.31 / 0.66-0.31 / 0.34-0.31 / 0.34
LHFLX+0.117 [+0.018, +0.251]+0.340.89 / 0.940.10 / 0.06-0.46 / 0.271.46 / 0.731.47 / 0.73
SHFLX-0.140 [-0.239, -0.028]-0.421.10 / 0.90-0.11 / 0.09-1.73 / -0.402.72 / 1.402.73 / 1.40
sstT0+0.467 [+0.417, +0.530]–0.90 / 0.890.10 / 0.111.06 / 1.05-0.06 / -0.05-0.06 / -0.05
sst+0.204 [+0.179, +0.236]–0.89 / 0.890.11 / 0.111.08 / 1.08-0.08 / -0.08-0.08 / -0.08
verified I1-F2a · Below “skip.0 + layer 0”: FLUS splits about evenly between layer 0's linear 1×1 skip_module and its InstanceNorm-ed convblock, whereas an SST perturbation goes 100% through the 1×1 skip_module.
FLUS uniform: skip.0 0.86, skip_module 0.53, convblock 0.47; box: skip.0 0.91, skip_module 0.35, convblock 0.65; sst: skip_module 1.00
Control: projection metric agrees within 0.05 for FLUS. Skeptic pass: an independent implementation (exege's new samudra Patchable adapter, EdgePatch semantics) gives skip.0 share 0.854 on 7 states of 2045 (localize.py: 0.86)
Caveat: near-linear response: sufficiency = necessity, shares sum to ≈1 (one result per split)
narrowed I1-F2b · InstanceNorm statistics account for the box-vs-uniform difference of the FLUS response.
freezing the convblock's IN statistics raises its uniform-shift share 0.47 → 0.94 (box: 0.65 → 0.65); predicted un-normalised uniform slope -2.65 vs measured box -2.58 K per 100 W/m² (uniform -1.80)
Control: the same freeze leaves box shares unchanged, as expected if a box barely moves whole-field statistics
Caveat: holds for FLUS only: for FLDS the same prediction gives 1.34× vs measured box/uniform 1.07×; box ratio here is 1.43× (laptop said ~2×)

Cells: perturbed-area-mean share / projection share. Because the network is near-linear in dQ, sufficiency = necessity and skip.0 + deep ≈ 1: one result, not converging lines.

Reproduce
python idea1_flus/localize.py data/states/t1995 t1995 4
python idea1_flus/localize.py data/states/t2045 t2045 4
python idea1_flus/localize_summary.py t1995 t2045

Results: localize_summary_t1995.json localize_summary_t2045.json

F3 · Can FLUS be removed without breaking FLDS/FSNS?

verified I1-F3 · At layer-0 channel granularity the FLUS shortcut cannot be removed without breaking FLDS; at the input it can.
greedy-32 channels (selected on fit states 1985 + 1995, scored on 2045): FLUS ×0.08 but FLDS ×-0.06; replacing the FLUS input with 5.67e-8·sst⁴ keeps FLDS/FSNS ×1.00/×1.02 and shifts the clean next SST by +0.17 K (open-ocean mean; RMS 0.49 K)
Control: 20 random size-matched sets per size: FLUS ×0.97 (10) / ×0.87 (32) median, 0 successes
Caveat: mean ablation at one place (layer-0 output); box shifts move SHFLX by +67% under input replacement; the one-step +0.17 K is what the coupled de-leak tests over years
shiftarmFLUS ×FLDS ×FSNS ×LHFLX ×SHFLX ×PREREG success
uniformgreedy100.510.540.295.56-0.00no
uniformgreedy320.08-0.060.796.24-1.06no
uniformdeleak_input-0.001.001.020.981.02yes
uniformrandom10 ×200.97 [0.85, 1.02]0 successes
uniformrandom32 ×200.87 [0.75, 1.01]0 successes
boxgreedy100.510.330.284.021.07no
boxgreedy320.08-0.150.904.361.58no
boxdeleak_input-0.001.100.881.151.67yes
boxrandom10 ×200.97 [0.85, 1.04]0 successes
boxrandom32 ×200.88 [0.75, 1.04]0 successes

Ratios are ablated / unablated slope per state, averaged (LHFLX ratios are unstable because its unablated slope is small). Greedy channels at layer 0: 254, 51, 203, 179, 33, 14, 256, 127, 30, 57 … (channel 254 first, as on the laptop). Greedy curve on fit states: 10 channels 0.48, 32 channels 0.91 of the FLUS response.

Reproduce
python idea1_flus/ablate.py l0 data/states/ic1985,data/states/t1995 data/states/t2045 v1
python idea1_flus/ablate_summary.py ablate_l0_v1

Results: ablate_l0_v1_summary.json

Replication across checkpoints

verified I1-R · Every ocean checkpoint has the FLUS thermometer; its strength tracks how leaky the FLUS it was last trained on was.
ocean-only E11 seeds -4.23…-3.99; after more ocean-only fine-tuning (E11-FT) -7.07…-6.85; after coupled fine-tuning (C1A0, C1A1, C0A0, E30/E31 seeds) -2.49…-1.71 K per 100 W/m² (uniform, 1995 truth-forced inputs)
Control: truth's conditional slope is −17: every model is far below it; in this sample, coupled fine-tuning also turns LHFLX from the wrong sign to the lawful one
Caveat: C1A1 shares C1A0's ocean init and E11-FT S01 is C1A0's ocean before coupled fine-tuning (not independent); E17 (1-day ocean) excluded: 5-day inputs
checkpointtrainingFLUSFLDSFSNSLHFLXSHFLXFLUS boxthermometer areashortcut?
E11-FT_S02ocean-only + fine-tune (leaky FLUS)-7.07+0.52+0.08-0.56+0.33-9.080.989yes
E11-FT_S01ocean-only + fine-tune (leaky FLUS)-6.97+0.26+0.07-0.51+0.34-8.840.988yes
E11-FT_S03ocean-only + fine-tune (leaky FLUS)-6.85+0.35+0.06-0.49+0.32-8.790.988yes
E12_W1ocean-only, loss-weight variant-5.50+0.22+0.15-0.14+0.14-7.870.992yes
E13_W2ocean-only, loss-weight variant-5.24+0.16+0.09-0.21+0.23-7.600.989yes
E11_S02ocean-only, truth-forced (leaky FLUS)-4.23+0.43+0.19-0.21+0.14-5.810.996yes
E14_W4ocean-only, loss-weight variant-4.14+0.13+0.17-0.10+0.16-6.090.995yes
E16_W3ocean-only, loss-weight variant-4.05+0.18+0.19-0.16+0.23-5.970.996yes
E11_S03ocean-only, truth-forced (leaky FLUS)-4.03+0.28+0.18-0.05+0.07-5.600.993yes
E11_S01ocean-only, truth-forced (leaky FLUS)-3.99+0.15+0.17-0.17+0.17-5.900.994yes
E30-CFT_S01coupled fine-tune (C0A0 family seed)-2.49+0.28+0.26+0.03+0.22-3.450.995yes
E30-CFT_S02coupled fine-tune (C0A0 family seed)-2.02+0.55+0.30+0.08+0.17-2.810.998yes
C1A1coupled fine-tune (release)-1.98+0.31+0.28+0.07+0.17-2.760.997yes
E31-CFT_S02coupled fine-tune (C1A1 family seed)-1.98+0.56+0.33+0.08+0.15-2.740.998yes
E31-CFT_S03coupled fine-tune (C1A1 family seed)-1.87+0.44+0.31+0.03+0.33-2.730.996yes
C1A0coupled fine-tune (release)-1.84+0.32+0.29+0.08+0.13-2.570.997yes
C0A0coupled fine-tune (release)-1.71+0.43+0.31+0.05+0.24-2.510.997yes
Reproduce
python idea1_flus/replicate.py C1A0=... E11-FT_S01=... (see slurm/queue scripts 08_replicate_*.sh)

Results: replicate_C1A0.json replicate_E11-FT_S02.json replicate_E12_W1.json replicate_E30-CFT_S01.json

Idea 2 · Do explanations keep the emulator's real errors?

Plan (preregistered, Phase 2). One-step forecasts from truth states (atmosphere 6 h, 4 seeds; ocean 5 days, truth-forced). Error = model − E3SM. Tolerance τ = max(2 × same-seed SD, 1% of spatial SD), sweep k ∈ {1, 2, 3}. Explanations: ocean skip.0-only, atmosphere blocks-only / big-skip-only (corrector in/out), PCA-k / SAE at SFNO blocks. Scores: A_ok, A_err, three-way split (reproduces / corrects / other), scalar-offset and per-component-gain controls, empty-explanation and seed-ceiling anchors, block bootstrap.

Starter result (hackathon track T2; not a finished result)

One-step forecasts from 12 truth states (fit: 1996, 2042; score: 4 dates of 2047), 4 seeds, pre-corrector, τ = max(2σ_seed, 1% spatial SD). Cells: A_ok / A_err.

variableerror areaseed ceilingempty (PCA-0)PCA-8 L7PCA-32 L7PCA-128 L7scalar offsetaffine persistence
TS0.330.83 / 0.800.23 / 0.170.32 / 0.250.50 / 0.390.88 / 0.770.29 / 0.190.33 / 0.16
T_70.210.83 / 0.790.22 / 0.160.30 / 0.220.65 / 0.520.95 / 0.880.41 / 0.280.39 / 0.28
U_70.250.83 / 0.780.19 / 0.130.37 / 0.260.58 / 0.430.91 / 0.780.33 / 0.230.32 / 0.23
PS1.00–––––––
surface_precipitation_rate0.680.82 / 0.820.31 / 0.280.38 / 0.340.65 / 0.620.84 / 0.81––
FLUT0.250.83 / 0.770.30 / 0.220.40 / 0.310.65 / 0.510.94 / 0.87––
FSUTOA0.360.82 / 0.800.31 / 0.180.30 / 0.210.40 / 0.250.74 / 0.56––

Latent explanations beat the output-level controls by a wide margin; PCA-128 at block 7 reaches the seed ceiling for T_7 and FLUT. PS is 'error' everywhere and precipitation on 68% because this is pre-corrector output: the corrector column is the first thing to add. No bootstrap; 4 score dates.

Reproduce
python idea2_errors/strata.py data/states/atm12 --layer 7

Results: strata_L7_K2.json

Idea 3 · How much of each spatial scale survives one step?

Method. JVP of the 6-hour atmosphere network (pre-corrector) at 12 truth states (Jan/Apr/Jul/Oct × 1996, 2042, 2047) × 3 noise draws, noise held fixed by passing our own draw to the conditional model. Probes: band-limited random fields (flat power over the band, 8 per band, physical units: K, m/s, Pa); gain(band) = √(output power in band / input power in band). Baselines: persistence (1); E3SM's own per-degree one-step AR(1) transfer of anomalies (score years) and the damped-linear fit (fit years). These two are the same statistic on different years, a restatement, shown to bound sampling noise.

E3SM's own 6-hour per-degree persistence (AR(1) coefficient of anomaly SH coeffi
E3SM's own 6-hour per-degree persistence (AR(1) coefficient of anomaly SH coefficients about a fit-year smoothed climatology). Large scales persist (≈0.99 at l ≤ 8); T_7 falls to 0.44 and U_7 to 0.27 at l = 80. This is the bar the model's gain is compared against.
Reproduce
python idea3_gain/truth_transfer.py coeffs 1995 1996 1997 1998 1999 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049
python idea3_gain/truth_transfer.py transfer

Results: truth_transfer.json

C1A0 6-h single-variable gain per SH band (preregistered probe; blue: mean, band
C1A0 6-h single-variable gain per SH band (preregistered probe; blue: mean, band 5–95% over 12 states × 3 noise draws) vs persistence and E3SM's per-degree persistence. Caveat: a single-variable, single-level perturbation is unbalanced and the model rejects part of it, so these gains sit below persistence even at l ≤ 2; 30–50% of the response lands outside the probed band for T_4…V_7 (1–5% for PS, T_1). T_1 and PS gains rise at small scales. The balanced-probe measurement below is the apples-to-apples comparison with truth.

Linearity (finite difference / JVP, per probe): PS|band0-2|eps0.01: 0.998–1.001; PS|band0-2|eps0.1: 0.974–1.005; PS|band14-17|eps0.01: 1.000–1.001; PS|band14-17|eps0.1: 0.994–1.005; PS|band30-33|eps0.01: 1.000–1.000; PS|band30-33|eps0.1: 0.998–1.002; PS|band46-49|eps0.01: 1.000–1.000; PS|band46-49|eps0.1: 0.997–1.001; PS|band62-65|eps0.01: 1.000–1.000; PS|band62-65|eps0.1: 1.000–1.000; PS|band78-81|eps0.01: 1.000–1.000; PS|band78-81|eps0.1: 1.000–1.000; T_7|band0-2|eps0.01: 0.984–1.006; T_7|band0-2|eps0.1: 0.905–1.131; T_7|band14-17|eps0.01: 0.998–1.000; T_7|band14-17|eps0.1: 0.987–1.005; T_7|band30-33|eps0.01: 1.000–1.000; T_7|band30-33|eps0.1: 0.999–1.001; T_7|band46-49|eps0.01: 1.000–1.000; T_7|band46-49|eps0.1: 0.999–1.000; T_7|band62-65|eps0.01: 1.000–1.000; T_7|band62-65|eps0.1: 1.000–1.000; T_7|band78-81|eps0.01: 1.000–1.004; T_7|band78-81|eps0.1: 1.000–1.000; U_7|band0-2|eps0.01: 0.999–1.000; U_7|band0-2|eps0.1: 0.993–1.005; U_7|band14-17|eps0.01: 1.000–1.000; U_7|band14-17|eps0.1: 0.999–1.001; U_7|band30-33|eps0.01: 1.000–1.000; U_7|band30-33|eps0.1: 1.000–1.000; U_7|band46-49|eps0.01: 1.000–1.000; U_7|band46-49|eps0.1: 1.000–1.000; U_7|band62-65|eps0.01: 1.000–1.003; U_7|band62-65|eps0.1: 1.000–1.000; U_7|band78-81|eps0.01: 1.000–1.010; U_7|band78-81|eps0.1: 1.000–1.000.

Reproduce
python -m hc.evaluate data/states/atm12 --steps 1 --capture-atm 1 --no-ocn-pred --capture --start 1996-01-16 ... (12 dates)
python idea3_gain/gain_atm.py data/states/atm12 a 0 1 2  # ... b 3 4 5, c 6 7 8, d 9 10 11
python idea3_gain/gain_summary.py a b c d

Results: gain_atm_summary.json

Ocean, 5-day step

C1A0 ocean single-channel gain per SH band (truth-forced inputs, 12 dates; band
C1A0 ocean single-channel gain per SH band (truth-forced inputs, 12 dates; band = min–max over dates). Flat in degree to l = 89: no scale selectivity at 1° within one step. Single-channel gains understate persistence: sst and the 0–20 m layer describe the same water (sst alone 0.22 K/K; sst + 0–20 m together 0.50 K/K in the localisation runs).
Reproduce
python idea3_gain/gain_ocn.py v1

Results: gain_ocn_v1.json

Balanced probes (exploratory, PREREG change log 2026-11-20)

narrowed I3-1 · With balanced perturbations, C1A0 keeps the amount of each scale about as E3SM does, but decorrelates the pattern faster, increasingly so at small scales.
U_7 projection vs truth AR(1): l≈8 0.83 vs 0.98; l≈31 0.52 vs 0.73; l≈55 0.27 vs 0.45; magnitude at l≈55 0.43
Control: truth per-degree AR(1) on fit and score years (agree within 0.02); preregistered single-variable probes shown for contrast
Caveat: probe = one 5-day truth difference direction per state (12 states × 3 noise draws); truth AR(1) averages all anomalies; T_1 is the exception (small scales persist more than in truth); l ≤ 2 includes global means
Green: C1A0's one-step response to balanced perturbations (truth 5-day differenc
Green: C1A0's one-step response to balanced perturbations (truth 5-day differences, all prognostic channels, band-passed): solid = projection on the perturbation (phase-sensitive, comparable to truth's AR(1)), dashed = magnitude. Red/orange: E3SM's own per-degree 6-h AR(1). Blue dotted: the preregistered single-variable gain (unbalanced, hence low).
Reproduce
python -m hc.evaluate data/states/atm12m5 --steps 1 --capture-atm 1 --no-ocn-pred --start 1996-01-11 ... (12 dates − 5 d)
python idea3_gain/gain_atm_bal.py data/states/atm12 data/states/atm12m5 v2

Results: gain_atm_bal_v2.json

Rollout spectra (starter, track T3)

Power of 1-year C1A0 rollouts (started from truth in 1995, 2040, 2045) relative
Power of 1-year C1A0 rollouts (started from truth in 1995, 2040, 2045) relative to E3SM over months 7–12, per SH degree; dashed: balanced one-step projection. Temperature power matches truth at all l; winds lose 11–17% at l = 80–89. Pre-corrector, 06 vs 00 UTC sampling.
Reproduce
python idea3_gain/rollout_spectra.py

Results: rollout_spectra.json

Pending: other checkpoints; ViT comparison

Numbers that are restatements of one another

For participants and their agents

Start with HACKATHON.md: setup in ~20 min, what is ready, six open tracks, and a prompt to hand your agent.

Updated 2026-11-20 06:57 . Every number links to its JSON under “Reproduce”; machine-readable: manifest.json.