Findings — 2026-05-10¶
Consolidated scientific record for Track 1 v1.6 = v1.5 + notebook 17 (ensemble-disagreement trust map). The v1.5 findings (notebooks 10–16) are recorded in 2026_05_09.md and remain unchanged. This doc adds notebook 17, updates the cluster-readiness narrative, and ties everything to the next phase.
Track 1 v1.6 closed on local single-GPU hardware (RTX 5090, 32 GB). All numbers below come from executed notebooks merged to main; reproducible by re-running.
Brief recap of v1.5 (see 2026_05_09.md for full detail)¶
- Carroll-6 recovery against real Darwin v5 across 3 basins × 3 targets. Strongest single-fit: N Pacific Darwin NO₃ with DINN per-cell r = 0.979, loss ratio Global / DINN = 23.8×. Mid-Atlantic underperforms in both Chl and NO₃; Equatorial Pacific FeT is the SST-only baseline's weakest fit at r=0.337.
- Structural-ceiling argument is universal: the global-scalar Green's-functions parametric class produces a constant prediction (mathematically inevitable for one parameter vector + uniform initial state) in every basin × target combination. DINN per-cell gets non-trivial r in every fit. Per-cell parameters can express what global-scalar parameters cannot, and Carroll's published calibration is bounded by the global-scalar ceiling.
- Cross-basin parameter consistency: recovered Carroll-6 means are stable to within ~25% across Mid-Atl + N Pacific. Same systematic offsets vs Carroll's published values appear in every basin (
alpfe~0.5–0.7×,scav_rat~2–3×, etc.). Stronger claim than any single fit because it shows reproducibility — rules out random fitting noise as the explanation, points at the box-model proxy as the source of the systematic offset. alpfeconverges to Carroll's 0.928 when the target constrains it (nb14 Eq Pacific FeT, recovered 0.959, 3% off). Iron pair is identifiable in HNLC regions with iron-relevant data, not in iron-replete regions.- Network capacity is NOT the recovery ceiling (nb15). DINNDeep (4-channel input, 9.4K params, 21× the baseline) drives Eq Pacific FeT r from 0.337 → 1.000 with ~3000× lower loss. But recovered Carroll-6 means do NOT get closer to Carroll's published values — some get worse. The 5-tracer box-model proxy is the recovery ceiling.
- DINNDeep's r=1.000 is interpolation, not extrapolation (nb16). Random hold-out r = 0.995 (passes), block hold-out (W 2/3 → E 1/3) r = 0.301 (fails). DINNDeep is not memorising in the trivial sense, but it IS overfitting in the spatial-extrapolation sense.
- v04 = v05 calibration: Carroll 2022 (Darwin 3 / v05) inherits Carroll 2020's six calibrated values bit-for-bit, verified against the source namelists. The publicly-accessible v05 Darwin output is the active recovery target.
New in v1.6 — Notebook 17: ensemble disagreement as inference-time trust map¶
Question¶
After nb16 showed DINNDeep fails to extrapolate across spatial blocks, the natural follow-up is: can we detect, at inference time, which cells the model shouldn't be trusted on, without needing held-out ground truth?
The cheapest hypothesis from the literature (cf. Salman & Liu 2019, arXiv:1901.06566, originally for classification under cross-entropy + softmax) is consensus across multiple seeds: train an ensemble, look at per-cell disagreement, and trust cells where the ensemble agrees. We adapt to regression by using per-cell prediction stdev across seeds as the disagreement signal.
Two sub-experiments to test the idea separately for in-domain quality scoring and extrapolation detection:
| Section | Setup | Question |
|---|---|---|
| A | 10-seed DINNDeep ensemble on FULL Eq Pacific FeT AOI (no hold-out) | Does per-cell stdev correlate with per-cell \|error\| of the ensemble mean prediction? |
| B | 5-seed DINNDeep ensemble on western 2/3 only (replicates nb16's block-CV) | Is per-cell stdev systematically higher on held-out (eastern, never-seen) cells than on training (western) cells? |
Same architecture, hyperparameters, data as nb15 / nb16. Only torch.manual_seed() differs across seeds.
Section A result — outlier flag, not fine-grained quality score¶
10 DINNDeep seeds trained for 1500 epochs each (~7 min/seed on RTX 5090, ~75 min total). Stack predictions → shape [N=10, H, W]; compute per-cell stdev = ensemble disagreement = trust map. Compare against per-cell |error| of the ensemble-mean prediction vs Darwin truth.
| Metric | Value |
|---|---|
| Ensemble mean r vs Darwin truth | 0.99992 (saturating, consistent with nb15) |
| Pearson r(per-cell stdev, |error|) | +0.869 (p < 1e-300) |
| Spearman ρ(per-cell stdev, |error|) | −0.417 (p = 2.6e-46) |
Sign disagreement. Pearson is strongly positive but Spearman is negative — they aren't compatible signals about a uniform monotonic relationship. The diagnostic interpretation: the relationship is outlier-driven.
- Pearson is sensitive to the tail. A few cells with large stdev coincide with the few cells of large prediction error — they jointly dominate the linear fit.
- Spearman looks at ranks. In the well-predicted bulk (most cells, where both stdev and
|error|are small), the rank order between the two is actually inverted.
Practical reading: ensemble stdev is a usable flag for cells with extreme prediction errors (the long tail of the error distribution), but is not a reliable fine-grained quality ranking within the well-predicted bulk. A user querying "which cells should I trust?" gets a useful answer; a user querying "rank these cells by trustworthiness" does not.
Section B result — extrapolation detection FAILS¶
5 DINNDeep seeds trained on the western 2/3 of the AOI only (loss masked to training cells; predictions still produced everywhere by the per-cell network). Compute per-cell stdev across 5 seeds; compare distributions on training (western, n=714) vs held-out (eastern, n=357) cells.
Per-seed reproducibility of nb16's block-CV result:
| seed | train r | held-out r |
|---|---|---|
| 0 | 1.000 | 0.301 |
| 1 | 1.000 | 0.288 |
| 2 | 1.000 | 0.358 |
| 3 | 1.000 | 0.278 |
| 4 | 1.000 | 0.330 |
| ensemble mean | 1.000 | 0.325 |
nb16's r=0.301 is squarely in the middle of the distribution. Block-CV failure is highly reproducible — not noise, not seed-luck.
Stdev distribution train vs held-out:
| train cells (n=714) | held-out cells (n=357) | |
|---|---|---|
| mean stdev | 3.59e-6 | 4.21e-6 |
| median stdev | 3.91e-6 | 4.17e-6 |
| 90th pct stdev | 4.79e-6 | 6.32e-6 |
- Held-out / training stdev ratio = 1.17 (only 17% higher)
- Mann-Whitney U test (one-sided, train_stdev < test_stdev): p = 1.4e-10 — statistically significant, but the effect size is tiny relative to the held-out r=0.32 catastrophe.
Verdict: FAILS. The ensemble does NOT meaningfully disagree more in cells where it's clearly wrong (held-out r=0.32 vs train r=1.00). The model is overconfident in extrapolation territory — it fails in the held-out block but doesn't know it's failing. Abstention via disagreement does not rescue cross-basin transfer.
Why both halves of the question matter¶
| Section A | Section B | Combined reading |
|---|---|---|
| outlier flag works | extrapolation detection fails | Ensemble disagreement is useful only for in-distribution cells with extreme errors; useless for telling whether you're in or out of distribution. |
Translated for the next phase: cheap solutions (more seeds, more capacity, more training) do not rescue the cross-basin generalisation gap. The path forward has to add real information that the model currently lacks — physics constraints (carbonate-extended box model + multi-tracer joint loss) or different observations (depth-resolved iron, GEOTRACES) — not more network sophistication.
What v1.6 changes about the next phase¶
The v1.5 doc concluded that "next phase deliverables ... need cluster compute." v1.6 sharpens that ask: the cluster work needs to be physics-constrained, not just more of the same compute.
Three specific workloads now justified by quantitative evidence:
- Box-model carbonate extension (DIC + ALK + carbonate chemistry, plus the full 5 PFT + 2 zoo + DOM ecosystem). Justified by the v1.5 finding that the 5-tracer proxy is the recovery ceiling (nb15) and the v1.6 finding that more network capacity does not close the gap. Lets us fit Darwin's air-sea CO₂ flux pattern directly — the most direct Carroll 2022 reproduction target.
- Multi-tracer joint loss (NO₃ + Chl + DIC + FeT simultaneously). Reduces the parameter degeneracy DINNDeep exposed by giving the optimisation multiple loss surfaces to satisfy at once. All four tracers are present in the LLC270 monthly tree on local hardware.
- Time-resolved fitting across 300+ monthly snapshots instead of climatology. Opens Track 2 emulator territory and removes the "smooth-field" interpolation slack DINNDeep currently uses (per nb16/nb17). Infeasible on single-GPU.
The codebase is Linux-portable (uv / pyproject.toml-managed deps; fresh-venv tested on a clean worktree; 104 passing tests). Cluster prep doc at docs/cluster_setup.md. Notebook path-refactor (env-var-driven DARWIN_DATA_ROOT) and SLURM job templates land alongside this doc in the same PR.
Reproducibility¶
Notebook 17 is at notebooks/17_dinn_deep_ensemble_disagreement.ipynb. Per-seed checkpoints (seedA_NN.npz ×10, seedB_NN.npz ×5) live in notebooks/nb17_results/ and are gitignored (regeneratable from the notebook in ~2 hours). The summary JSONs (summary_A.json, summary_B.json) and figures (fig_A_trust_map_4panel.png, fig_A_scatter.png, fig_B_block_cv_4panel.png) are committed and serve as the durable record of the run.
To re-run the ensemble:
export DARWIN_DATA_ROOT="/path/to/ecco_darwin_v5" # or leave unset for local D:\ default
uv sync --all-extras
uv run jupyter nbconvert --execute --to notebook --inplace \
--ExecutePreprocessor.timeout=14400 \
notebooks/17_dinn_deep_ensemble_disagreement.ipynb
The per-seed cache makes the run resumable: completed seeds short-circuit, only missing seeds train.
What v1.6 does NOT close¶
- Multi-tracer joint loss — DINNDeep's r=1.000 is on FeT alone; the parameter degeneracy nb15 + nb17 expose may collapse when multiple tracer fields constrain the same parameter set simultaneously. Untested locally because of memory ceiling.
- Time-resolved fitting — all fits to date are on 23-year monthly climatologies. Whether DINNDeep can fit time-varying Darwin output (and whether the recovered Carroll-6 maps stabilise across time) is the natural Track 2 entry point.
- Cross-basin transfer with proper architectural constraints — nb17 ruled out cheap ensemble-based abstention. Whether physics-augmented or symbolic-constraint approaches recover cross-basin generalisation is the open scientific question for the cluster phase.
- Iron-pair
scav_ratidentifiability — remains 1.8–3× off Carroll's value in every fit, including DINNDeep. Likely requires depth-resolved observations (depth-resolved iron + POC export). Not addressable through architecture or training alone.
Notebook ledger (v1.6)¶
- 09 — methodology demo on real GLODAP NO₃ proxy
- 10 — Carroll-6 recovery against Darwin Chl, Mid-Atl
- 11 — cross-basin Mid-Atl + N Pacific Chl
- 12 — LLC270 native tracer loader
- 13 — cross-basin Mid-Atl + N Pacific NO₃
- 14 — iron-pair recovery via FeT, Eq Pacific
- 15 — DINNDeep architecture upgrade test (Track 1 v1.4)
- 16 — cross-validation honesty check on nb15 (Track 1 v1.5)
- 17 — ensemble disagreement as inference-time trust map (Track 1 v1.6)
Track 1 v1.6 = closed on local hardware. Awaiting cluster allocation for the next phase (box-model carbonate extension + multi-tracer joint loss + time-resolved fitting + Track 2 emulator).