Skip to content

v3.1 closeout — 3-AOI Basin C + PER_AOI_DINN at scale

⚠ SUPERSEDED FRAMING (2026-06-27). This is a point-in-time record; its data and negative results stand, but its framing is corrected by STATUS.md and docs/research_notes/2026-06-27_box_homogenization_DEFINITIVE.md. Specifically: R_PICPOC is NOT a "6/6 wall" and is NOT cluster-gated — it recovers at 1° box scale given a real calcite anchor (Daniels CP:PP / MODIS PIC) plus the RATIO_MAX=2 fix for the contaminated Southern-Ocean ratio target; the differentiable Darwin calcite port and native resolution were tested and did not help. The project is reframed as a surrogate-to-model identifiability study over 4 OBSERVABLE params {alpfe, scav_rat, diatomgraz, R_PICPOC}; the growth pair {Smallgrow, Biggrow} is unobservable by construction (excluded, not failed). The surrogate gap is dimensional (the 0-D box homogenizes spatial structure, tracer CV→1e-15), so box-vs-Darwin spatial-pattern correlations are not fidelity metrics — identifiability comes from real absolute anchors.

Paper-prep quality writeup of the v3.1 work. Companion to STATUS.md (current state) and docs/dinn_design.md (architecture).

Summary

v3.1 extends v3.0's multi-AOI joint training from 2 to 3 AOIs (Equatorial Pacific + N Atlantic Subpolar + Southern Ocean Pacific) and adds a per-AOI DINN architecture with cross-AOI consistency penalty. The v3.1 observation set is broad: per-AOI Chl1–Chl5, per-AOI POC + DIC + ALK + CO₂ flux, GEOTRACES surface + subsurface dissolved iron, GEOTRACES subsurface POC, and GEOTRACES surface biogenic silica (bSi) via the diagnostic in src/darwindiff/silica.py (POSI_W env var; active in every Basin C config). Across 86 configs and 856 seeds on a single NVIDIA RTX 5090 Laptop, the v3.0 5/6 calibration-grade ceiling holds at a 0.23% break rate (2/856 seeds). Both break events are single-seed and unreproduced at scale; their recovery profiles are complementary (one lands diatomgraz, the other R_PICPOC), and Wave 6's composition test of the two intervention families produces 0/10 at 5/6 — confirming the structural ceiling.

The robust v3.1 contribution is the Basin C iron-pair recovery at 38/40 (95%) across n=40 in four independent 10-seed batches, plus a binary mutex characterization at low PIC anchor doses. The structural framing of the 5/6 ceiling — parameter conservation under ~5 effective observational constraints on 6 parameters — survives the v3.1 sweep set unchanged and is strengthened by the failure of architectural and lever-tuning approaches to produce reproducible 5/6.

What v3.1 adds over v3.0

Component v3.0 v3.1
Number of AOIs 2 (Eq Pac + N Atl Subpolar) 3 (adds Southern Ocean Pacific)
Per-AOI architecture Shared DINN + AOI-ID input channel Optional PER_AOI_DINN=1 with per-AOI DINNs + cross-AOI consistency penalty CONSISTENCY_LAMBDA=λ
Positivity loss Optional POSI_W Same, with added POSI_DEPTH_MAX env var
PIC/POC anchors PIC_ABS_W, POC_ABS_W (paired tested) Same, dose-response down to 0.02
Diatom Chl loss CHL1_W_EXTRA (z-score boost) Same, fine-grid response curve [2.0, 7.0]
Sweep orchestration Per-PR ad-hoc Five-wave chained sweeps with thermal cooldown, STOP sentinel, resume logic, crash recovery from .log files

Sweep design (five waves)

Each wave is one orchestration script; each config is one runner invocation with 10 batched seeds.

Wave Purpose Configs Result root
V2 (Wave 1) 9-arc Basin C refinement: n=20 baseline + PIC/POC dose-response + paired anchors + CHL1_W response + SO weight + architecture + carbonate + longer training 33 D:\runs\bcr_20260520_0121\
Wave 2 Orthogonal-lever sensitivity: GEOTRACES_W, POC_SUB_W, PINN_W, FET_W, CONSISTENCY_LAMBDA, n=30 robustness 24 D:\runs\bcr_w2_20260520_1657\
Wave 3 2D grids (POSI × AOI weight, CHL1 × POSI) + n=40 robustness + POSI_DEPTH variants 18 D:\runs\bcr_w3_20260520_2035\
Wave 4 N_STEPS slow-tracer integration test + N_STEPS × PIC anchor combo 6 (1 timeout — 6/10 seeds completed for a_nsteps_1000) D:\runs\bcr_w4_20260520_2302\
Wave 5 Drill into the 5/6 breakthrough: CONSISTENCY_LAMBDA dose around 0.10, combos with best Wave 2 levers (ep2000, fetW5, geoW0.1), n=20 robustness at w2e_peraoi_lam0.1 8 D:\runs\bcr_w5_20260521_0039\

Each wave auto-chains the next via the CHAIN_NEXT_WAVE subprocess invocation, gated by absence of a STOP sentinel.

Headline results

Result Verified count Source
Basin C iron-pair recovery (F2 config, four 10-seed batches) 38/40 (95%) Cal+ on both alpfe and scav_rat bcr_*/seeds0-9, /seeds10-19, /seeds20-29, /seeds30-39
5/6 Cal-grade single-seed events 2 / 856 = 0.23% w2e_peraoi_lam0.1 seed 3, c_chl40_posi15 seed 9
6/6 Cal-grade events 0 / 856 —
Configs preserving iron pair at 10/10 27 / 84 —
Configs preserving iron pair at ≥ 9/10 40 / 84 —
Binary mutex confirmation (PIC anchor presence) 8/8 PIC-bearing configs → iron pair 0/10 All Arc 1 + Arc 7 configs in V2
PER_AOI_DINN at 2-AOI vs 3-AOI Falsified at 2-AOI (0/40 at 6/6 per PR #58); produces 1/10 at 5/6 at 3-AOI with CONSISTENCY_LAMBDA=0.1 bcr_w2_/w2e_peraoi_lam0.1

The two 5/6 events

w2e_peraoi_lam0.1 seed 3 (Wave 2):

  • Config: PER_AOI_DINN=1, CONSISTENCY_LAMBDA=0.1, Basin C base.
  • Recovery: alpfe Excellent + scav_rat Cal+ + Smallgrow Cal+ + Biggrow Cal+ + diatomgraz Cal+; R_PICPOC drifts.
  • Reproducibility: Wave 5 dose-response on CONSISTENCY_LAMBDA ∈ {0.05, 0.15, 0.20, 0.30} → 0 additional 5/6. Wave 5 n=20 extension at the exact winning config (seeds 10-19) → 0 additional 5/6. Combined n=20 rate: 1/20.

c_chl40_posi15 seed 9 (Wave 3):

  • Config: CHL1_W_EXTRA=4.0, POSI_W=1.5, Basin C base. No architectural change (shared DINN with AOI-ID channel).
  • Recovery: alpfe Cal+ + scav_rat Excellent + Smallgrow Cal+ + Biggrow Cal+ + R_PICPOC Cal+; diatomgraz drifts.
  • Reproducibility: n=20 retest at seeds 10-19 produced 0 additional 5/6 (mean_cal 2.40, iron-pair 9/10). Combined rate: 1/20.

The two are complementary: one lands diatomgraz, the other R_PICPOC. Both retain the iron pair.

Methodology contributions

Binary mutex in PIC anchor presence

Any nonzero PIC_ABS_W wipes iron-pair recovery → 0/10, regardless of dose or POC pairing. Tested down to PIC_ABS_W=0.02 (Wave 1 Arc 1) and with paired POC ratios from 1:2 to 1:1. Magnitude does not soften the effect; the basin flip is binary in PIC presence.

Mechanism interpretation: the PIC absolute anchor constrains R_PICPOC × mort_total jointly, pulling the optimizer into a basin where R_PICPOC is calibration-grade but the iron pair must drift to balance the loss. POC anchor alone (Wave 1 Arc 2) also kills iron pair, but with the opposite mean_cal trend: paired PIC+POC gives mean_cal 2.20 → 1.60 with increasing dose (worse), while POC alone gives mean_cal 0.80 → 2.80 (better at higher dose, just not into Basin A or C).

This refines v3.0's "basin mutex" framing from feedback_basin_mutex_robust.md: the boundary is binary in PIC presence, not a smooth gradient in PIC magnitude.

PER_AOI_DINN at 3-AOI is qualitatively different from 2-AOI

PR #58 falsified the per-AOI DINN + consistency penalty architecture at 2-AOI: 0/40 seeds at 6/6 across CONSISTENCY_LAMBDA ∈ {0.01, 0.1, 1.0, 10.0} × n=10; best λ=0.1 produced only 3/10 at 5/6 — below the v3.0 baseline of 7/15.

At 3-AOI with the Southern Ocean Pacific added, the same architecture (PER_AOI_DINN=1, CONSISTENCY_LAMBDA=0.1) produces 1/10 at 5/6 — the only architectural break of the 5/6 ceiling observed in the project. The mechanism is not yet established. Candidate explanations:

  1. The 3rd AOI provides cross-regime constraint that 2-AOI cannot, regardless of architecture; PER_AOI_DINN happens to be the right capacity.
  2. The Southern Ocean Pacific has particle-flux dynamics that the per-AOI parameter freedom can exploit; shared-DINN cannot.
  3. The result is statistical noise at the n=10 scale; n=20 dose-response (Wave 5) does not reproduce, supporting this interpretation.

Wave 5 leans toward (3): the dose-response across CONSISTENCY_LAMBDA ∈ {0.05, 0.10, 0.15, 0.20, 0.30} shows no sweet spot, and the n=20 extension at λ=0.10 produces 0 additional 5/6.

CHL1_W_EXTRA response is a wide flat plateau

Fine-grid response across CHL1_W_EXTRA ∈ {2.0, 2.5, 3.0, 3.5, 4.0, 5.0, 7.0} (V2 Arc 3) shows iron-pair recovery in the 8–10/10 range across [2.0, 3.5] with mean_cal essentially flat at 2.50–2.80. Degradation begins at 4.0 (iron 8/10, mean_cal 2.20). The originally-published Basin C optimum of CHL1_W_EXTRA=3.0 is not a sharp peak; the basin is a plateau approximately 1.5 wide on the lever axis.

Per-AOI Southern Ocean weight

AOI_W_SOUTHERNOCEANPAC=3.0 (V2 Arc 4) marginally improves mean_cal (2.50 → 2.90) without degrading iron pair (10/10). Confirms the Southern Ocean is the binding constraint on the path toward 5/6 in 3-AOI Basin C training, but is not sufficient alone.

Longer training has a non-monotonic effect

V2 Arc 5 (NB23_N_EPOCHS ∈ {2000, 2500, 3000}) shows ep2000 best (3/10 at 4+, mean_cal 2.90), ep2500 worst (1/10 at 4+, mean_cal 2.70), ep3000 recovers partially (2/10, mean_cal 2.90). Interpretation: overfitting onset between 2000 and 2500 epochs; 3000 epochs may exit the overfit regime via Adam adaptive-LR dynamics. The default of 1500 epochs is below the optimal but the difference is marginal.

Operational lessons captured

  • Windows MAX_PATH=260 crashed JSON writes after training succeeded in two Wave 1 Arc 7 configs (arc7_paired_pic0.05_poc0.025 and arc7_paired_pic0.10_poc0.05). The training log contained the full per-seed recovery values. A log-file parser at scripts/recover_failed_config_log.py reconstructs compatible JSONs from the log; sweep orchestration switched to short OUTPUT_DIR prefixes (D:\runs\bcr_<stamp>\) to stay under the limit.
  • Laptop sleep on Windows suspends background Python processes. Overnight runs require powercfg /change standby-timeout-ac 0 for AC-power never-sleep.
  • Thermal cooldown injection (60s base + 4-min deep every 5 configs + nvidia-smi-aware extension above 75°C) added to Waves 2–5 prevents sustained GPU bursts on the laptop. Cost: ~15% wall-clock overhead.
  • torch.compile amortizes JIT across batched seeds: a 10-seed batch takes ~7 min vs ~70 min sequential. Always batch.
  • Resume logic via per-config OUTPUT_DIR + glob count check survives sweep crashes and laptop-sleep interruptions. Sweep scripts skip configs that already have all expected seeds on disk.

Wave 6 — composition test of the two 5/6 lever families

Result: composition fails. Parameter-conservation framing strengthened.

The natural 6/6 candidate stacks the two complementary 5/6 interventions:

Config w6_combo_per_aoi_chl40_posi15
Levers from w2e_peraoi_lam0.1 PER_AOI_DINN=1, CONSISTENCY_LAMBDA=0.1
Levers from c_chl40_posi15 CHL1_W_EXTRA=4.0, POSI_W=1.5
Otherwise Basin C base (3-AOI; AOI_W_NATL=2.0, AOI_W_SO=2.0, hd=32, POC_SUB_W=0.5, GEOTRACES_POC_SUB_W=0.5, 1500 epochs)

n=10 batched seeds (D:\runs\bcr_w6_20260521_0841\w6_combo_per_aoi_chl40_posi15\):

Metric Combined w2e_peraoi_lam0.1 parent c_chl40_posi15 parent
6+/n 0/10 0/10 0/10
5+/n 0/10 1/10 1/10
4+/n 0/10 1/10 1/10
iron-pair Cal+ 9/10 10/10 10/10
R_PICPOC Cal+ 0/10 0/10 1/10
diatomgraz Cal+ 0/10 1/10 0/10
mean_cal 2.00 2.40 2.70

The combination keeps iron-pair recovery essentially intact (9/10) but neither R_PICPOC nor diatomgraz recovers — both 5/6 mechanisms fail simultaneously. Mean_cal regresses below either parent config alone (2.00 vs 2.40 / 2.70); Smallgrow and Biggrow, which were Cal+ in both parent configs, now drift to Loose in 9/10 seeds.

This is the pessimistic outcome predicted by v3.0's parameter-conservation framing. The two intervention families (PER_AOI architecture + low coupling penalty; CHL1+POSI loss reweighting) interfere; the optimizer redistributes the residual to a new binding region (small-cell phyto params) rather than letting both target recoveries hold.

The structural 5/6 ceiling now has three independent pieces of evidence:

  1. 856 seeds across 86 configs with 0 at 6/6 and 2 single-seed 5/6 events.
  2. The 5/6 events are unreproduced at extended seeds (w2e: 1/20; c_chl40: 1/20 on retest at seeds 10-19; Wave 6 also destroys the recipe).
  3. The composition test of the two complementary 5/6 recipes fails to even reach 5/6.

Across 856 total seeds in 86 configs, the v3.1 sweep set has 0 seeds at 6/6 and 2 single-seed 5/6 events. The structural ceiling is the headline finding.

Open questions

  1. Did c_chl40_posi15 reproduce at n=20? No — the retest at seeds 10-19 produced 0 additional 5/6 (mean_cal 2.40, iron-pair 9/10; combined 1/20, see §"The two 5/6 events"). Both 5/6 paths are now confirmed non-reproducing.
  2. Native-resolution Carroll-6 recovery. The current box-model proxy operates at the 1° AOI scale. Cluster-scale recovery at full LLC270 (~1M ocean cells per AOI, ~1000× the data per AOI) may dissolve the 5/6 ceiling if the parameter-conservation framing is a small-cell artifact. Cluster-gated.
  3. Time-resolved fitting. All current fits use 23-year time-mean Darwin output. Time-resolved fitting opens Track 2 emulator territory. Cluster-gated.
  4. POSi (biogenic silica) observation channel. GEOTRACES IDP2025 includes ~1280 finite bSi values — a diatom-specific tracer that directly constrains diatomgraz without competing with the iron-pair budget. Currently shelved for the leapfrog phase. May be the cleanest 6/6 path if cluster-resolution scaling does not dissolve the ceiling.

Known limitations

  • Box model is a 5-tracer proxy of full Darwin 3. The 5-PFT + 2-layer extensions narrow the gap but the structural ceiling on Carroll-6 recovery still has the box-model proxy as a contributing factor alongside parameter conservation.
  • DINN is per-cell, not spatially coupled. Advection / diffusion between cells is ignored. Appropriate for parameter recovery; not appropriate for the Track 2 emulator.
  • DINNDeep does not extrapolate spatially. Block CV gives held-out r=0.301 vs in-distribution r=1.000 on FeT. Use DINN baseline for cross-AOI extrapolation claims.
  • Climatology, not time-resolved. All current fits use 23-year time-mean Darwin output.

Cross-references