NERSC Perlmutter · 16 October 2026 · run full1
E3SM Agent Co-Discovery
Ten AI agents investigated E3SM code and simulation data to find where agent-driven discovery works and where it fails. Their answers were scored against a hand-verified answer key, and every citation was checked automatically against the files.
Correct (scored questions)
46 / 47
97.9 % · 44 / 47 as first scored
Wrong answers
0
1 partial (provenance)
Valid citations
100 %
file exists, line in range, quote matches
Message-board posts
0
agents never shared findings
Run time
21 min
1 CPU node, ~0.35 node-hours
Model usage
2.6 M
input tokens; 150 k output; $5.39 reported
Setup
Code and data
- Code: an E3SMv3 checkout (v3.1.0-beta), all components (EAM, EAMxx, ELM, MPAS-Ocean, MPAS-SeaIce, coupler).
- Data: the v3.LR historical run (
v3.LR.historical_0101.aigo, 1940–2065) and the S2S2D hindcasts (78 starts × 10 members). - Model:
claude-sonnet-5-5through the AmSC Model Access Gateway; agents run with Academy and the Claude Agent SDK.
Agents and tools
- 8 investigators: EAM, EAMxx, ELM, MPAS-Ocean, sea ice + coupler, configuration, data, code–data tracer.
- Skeptic re-checks every answer; coordinator writes the summary; a shared message board.
- Read-only tools only: list, find, grep and read code; netCDF headers and small variables. Paths outside the code and data trees are refused.
55 questions: 35 lookups (tier 1), 12 multi-step links including deliberate traps (tier 2), 8 open-discovery prompts (tier 3, judged by the skeptic, not scored). Question text and answers are kept private so the set stays usable as a test.
Results by tier
| Tier | Questions | Correct | Partial | Wrong | Notes |
|---|---|---|---|---|---|
| 1 · Lookup | 35 | 35 | 0 | 0 | Exact file and line for fields, units, defaults, classes, counts |
| 2 · Multi-step link | 12 | 11 | 1 | 0 | Code → configuration → data; includes the traps below |
| 3 · Open discovery | 8 | not scored | Skeptic: 5 supported, 3 partly | ||
Results by agent
| Agent | Questions | Correct | Partial | Open (tier 3) |
|---|---|---|---|---|
| EAM (atmosphere physics, parameterizations) | 16 | 14 | 0 | 2 |
| MPAS-Ocean | 9 | 7 | 0 | 2 |
| Code–data tracer | 8 | 5 | 1 | 2 |
| Configuration | 7 | 6 | 0 | 1 |
| Data | 7 | 6 | 0 | 1 |
| EAMxx | 3 | 3 | 0 | 0 |
| Sea ice + coupler | 3 | 3 | 0 | 0 |
| ELM | 2 | 2 | 0 | 0 |
Skeptic verdicts over all 55 answers: 48 supported, 7 partly supported, 0 unsupported.
The traps
| What the trap tests | Outcome |
|---|---|
| Choosing the right one of several configuration-dependent namelist defaults (ZM convection) | Correct |
| Case-sensitive parameter names in CLUBB (Fortran namelists ignore case; a case-sensitive search finds nothing) | Correct |
| A namelist variable used by the run that was renamed in the checkout (ZM trigger) | Correct |
| Ocean parameterization values that look tuned in one defaults file but are defaults in another (GM / Redi) | Correct |
| Output code the run used that does not exist in the checkout: the agent must say so, not invent a file | Correct — searched, then reported it absent |
| A run namelist enabling options the checkout does not have | Correct |
| Which code version actually built the run | Partial — found the source tree and commit, not the branch; did not compare commits |
What worked
- Exact lookups across every component: 35 of 35, each with a file and line that verified.
- Configuration-dependent defaults resolved correctly, often confirmed against the run's generated namelists.
- Admitting absence: when code was missing from the checkout, agents said so instead of inventing paths.
- Stating limits: agents said what they had not checked (gaps, time axes, other files).
- Second-agent review works: the skeptic caught an investigator's false claim that a convection parameter was absent from the run's namelist.
What failed or was weak
- No co-discovery happened. The shared message board received 0 posts: every agent worked alone despite instructions, so this run measures individual agents, not collaboration.
- Data-scale tasks. Auditing the S2S2D archive for gaps took 52 turns and 7 minutes, sampled only part of it, and ended at confidence 0.35.
- Provenance tracing stopped early: the run's commit was found but never compared with the checkout's.
- Unchecked prose: citations are verified automatically, but text outside them is not — one answer gave a wrong path in prose.
- Tool limits: a header-size cap and search timeouts narrowed some answers (reported by the coordinator; not re-verified).
- The coordinator stopped inventing anomalies (a dry run's did) but misreported one question's tier and did not use the tools it had.
Confidence
Agents gave a stated probability with each answer: mean 0.87 on correct answers and 0.70 on the single partial one. With only one answer that was not correct, calibration cannot be judged yet. The run's lowest confidence (0.35, the archive audit) did go with its weakest answer.
Candidate discoveries
Found by the agents beyond the answer key; they still need human verification.
- A third custom ocean analysis member (
fmeVerticalReduce) enabled in the run's ocean namelist. - The run was created from a separate E3SM fork with a custom output configuration, not from the checkout studied.
- A second, lower-case registration of the reference-height temperature field in the EAM history defaults.
- Completeness of the S2S2D archive is not established.
How it was scored
- Each scored answer must contain every required fact (patterns built from the verified answers) and cite files that exist, with the line in range and the quoted text present.
- Two answers were rescored after reading them: both stated an absence in words the first patterns missed ("has no …", "found none …"). Scores before and after are both kept.
- Limitations: one run, one model, no repeats; questions written by one author and reviewed by the user; pattern-based scoring; the skeptic is the same model; lookups dominate the set.
Next steps
- Make collaboration real (a coordinator that splits hard questions across agents and requires board posts), then compare with and without the board.
- Harder questions: EAMxx/Kokkos templates, MAM4 aerosols, macro-heavy Fortran, ELM parameter files, coupling chains.
- Tool fixes (header filtering, scoped search), repeat runs for variance, and a second model for comparison.