Skip to content

Argo-like 3D observations and ocean DA problem redesign#1

Open
glwagner wants to merge 6 commits intomainfrom
argo-observations-and-research
Open

Argo-like 3D observations and ocean DA problem redesign#1
glwagner wants to merge 6 commits intomainfrom
argo-observations-and-research

Conversation

@glwagner
Copy link
Copy Markdown
Member

Summary

  • Replace surface-only observations with Argo-like 3D profile sampling in the ETKF: 76 profiles/cycle, 31 vertical levels in upper 1000m, ~4,700 total obs (T+S) -- down from 56K surface-only
  • Add daily 3D tracer output to nature run for subsurface observation extraction
  • Add RESEARCH_NOTES.md documenting literature review on ocean DA for hurricane prediction

Key finding: our current problem is too easy

Surface perturbations (0.3 C bias + 0.5 C noise) decay naturally under atmospheric forcing in ~11-14 days. The 32% RMSE reduction we observed is mostly the atmosphere doing the work, not the DA.

The scientifically meaningful problem is recovering subsurface structure (thermocline depth, ocean heat content at 50-300m), which:

  • Persists for months-years without DA
  • Is poorly observed (Argo only, every 10 days)
  • Directly controls hurricane intensity (OHC > 60 kJ/cm2 supports intensification)

Changes

Argo-like observation network (ensemble_da.jl)

  • 76 random profile locations per cycle (Argo density: ~1 per 3-degree box, profiling every ~10 days)
  • Each profile observes all 31 grid levels in upper 1000m (k=20:50)
  • Total obs (T+S): ~4,700 per cycle (was 56K surface-only)
  • Obs-to-ensemble ratio: 470:1 (10 members), 47:1 (100 members)
  • MPI-aware: local extraction + single Allreduce for full ensemble-obs matrix
  • Land profiles filtered using nature run surface T mask

Nature run 3D output (nature_run.jl)

  • Daily 3D T/S snapshots via JLD2Writer (TimeInterval 1 day)
  • ~5 GB total for 14-day run (very manageable)
  • Required for subsurface observation extraction

Research notes (RESEARCH_NOTES.md)

  • Literature review on ocean DA for hurricane prediction
  • Operational DA window lengths (3-7 days for 4DVAR)
  • Perturbation persistence analysis: surface decays in days, subsurface persists months-years
  • Recommended redesign: subsurface perturbations, multi-platform obs, hurricane-relevant metrics

Next steps

  1. Subsurface perturbations: zero at surface, max ~1.5 C at 150m, spatially correlated (~200 km)
  2. Multi-platform obs: Argo profiles + satellite SST + SSH altimetry
  3. Localization (via Manteia.jl) to prevent ensemble collapse
  4. Scale to 100 ensemble members
  5. Hurricane-relevant metrics: OHC, mixed layer depth, thermocline depth

Test plan

  • Re-run nature run with 3D output (SLURM currently in maintenance)
  • Run ETKF with Argo-like observations
  • Verify land masking works (profiles on land filtered out)
  • Compare spread/RMSE evolution with previous surface-only run

Generated with Claude Code

Gregory LeClaire Wagner and others added 6 commits February 18, 2026 12:31
Replace surface-only thinned grid observations with Argo-like profile
sampling in the ETKF:
- 76 random profiles per cycle (~770 Argo floats / 10-day repeat)
- Each profile observes 31 vertical levels in upper 1000m
- Total obs: ~4,700 (T+S) per cycle vs 56K surface-only before
- Obs-to-ensemble ratio: 470:1 (10 members) or 47:1 (100 members)
- MPI-aware: local extraction + single Allreduce for efficiency

Add daily 3D tracer output to nature run (T and S at 1-day intervals)
for subsurface observation extraction.

Add RESEARCH_NOTES.md documenting literature review on ocean DA for
hurricane prediction: current perturbations are too easy (surface
errors decay under atmospheric forcing in days), the real challenge is
subsurface structure (thermocline depth, OHC) which persists for months
and controls hurricane intensity.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Documents ensemble Kalman filter variants (ETKF, EnKF, LETKF, DEnKF,
SEEK), operational ocean DA systems (GLORYS12, TOPAZ, ORAS5, RTOFS,
FOAM, ROMS 4DVAR), and implications for our variational approach.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace surface-focused perturbations (0.3C bias in upper 20 levels) with
  depth-dependent profile: zero at surface, max 1.5C at thermocline (~150m),
  decaying below. These persist months-years, making the DA problem meaningful.

- Free run uses bias-only (no noise) to represent ensemble mean trajectory.
  Ensemble members get bias + noise, both scaled by depth profile.

- Load free run surface fields in ensemble_da.jl and compute free run RMSE
  against nature run at each DA cycle.

- Three-line RMSE plot: free run (red) vs DA forecast (orange) vs DA analysis
  (green). Gap between red and green = demonstrated DA value.

- Add free run prerequisite check in SLURM DA script.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Weak-scaled from 4-GPU (1/8 deg) to 64 GPUs (16 nodes):
- 3200x2240x50 global grid, Partition(8,8), 400x280x50 per GPU
- 240 free-surface substeps (scaled with resolution)
- Free run uses bias+gradient perturbation (matching single_gpu_da)
- Array(znodes(...)) fix to avoid GPU scalar indexing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Distributed ensemble DA via SLURM job dependency chains (da_utils.jl,
  initialize_ensemble.jl, ensemble_member.jl, etkf_analysis.jl)
- LETKF with Gaspari-Cohn localization (1000 km radius)
- Fixed critical inflation bug (inflate around analysis mean, not forecast)
- Fixed observation errors (representativeness error for 1/4 deg grid)
- Vertical subsampling (every 4th level) to reduce overdetermination
- 8-GPU and 64-GPU scaling benchmarks
- Multi-node Julia ENV iteration bug fix (fix_multinode_env.jl)
- Progress report with development history and cost estimates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant