Skip to content

SDM with AMR#3181

Draft
debog wants to merge 89 commits into
erf-model:developmentfrom
llnl:dg/sdm_w_AMR
Draft

SDM with AMR#3181
debog wants to merge 89 commits into
erf-model:developmentfrom
llnl:dg/sdm_w_AMR

Conversation

@debog
Copy link
Copy Markdown
Contributor

@debog debog commented May 15, 2026

AMR support for the Lagrangian (super-droplet) moisture model. SuperDropletsMoist operates per-level: each level owns its super-droplet population, initializes against its own BoxArray, advances and undergoes phase change independently, and contributes to the Eulerian moisture state via covered-coarse-cell masking and conservation-preserving average-down at sync points. Coalescence is kept invariant across levels by scaling per-level bin volume with the cumulative refinement ratio. Tagging is extended to all moisture mixing ratios (qv, qc, qi, qr, qs, qg, qt = qc + qi + qr + qs + qg) and to any particle-deposited Eulerian field of the form <species>_<mesh_var> (e.g., super_droplets_moisture_mass_density).

  • LagrangianMicrophysics carries a per-level m_moist_model vector and exposes Define(lev), InitializeParticles(lev, ...), Advance(lev, dt, ...), Update_State_Vars_Lev(lev, cons, z_phys_nd), and Update_Micro_Vars_Lev(lev, ...). NullMoist/NullMoistLagrangian and the Microphysics virtual interface are updated to match. Update_State_Vars is collapsed to a single 2-argument form (cons_in, z_phys_nd); the cached z_phys_nd pointer in SuperDropletsMoist is removed.
  • SuperDropletPC is constructed with the multi-level ParGDB; InitializeParticles(lev, ...) seeds each level against its BoxArray. AdvectParticles guards against out-of-range k on partial-z fine tiles and NaN velocity near level boundaries.
  • EvolveParticles uses per-level Redistribute(0,0) on L0 and Redistribute(a_lev, a_lev) on fine levels, with ExtractAndRouteOORParticles(a_lev) routing fine-level out-of-range particles back to L0 via DefaultAssignor + filterParticles/copyParticles. This prevents mid-coarse-step cross-level moves that previously caused double-advection on the fine level.
  • phaseChange and Copy_Micro_to_State mask covered coarse cells with makeFineMask so cells already updated by a finer level are not re-updated by the coarse advance. Copy_State_to_Micro reads qc and qr from state on coarse/fine boundaries for q_t consistency; qc is recomputed from particles before q_t in phaseChange. fine_mask is built with the microphysics-fab ghost width.
  • AverageDownMicroVars(finest_level) runs amrex::average_down (via ERF::AverageDownTo, do_perturbational_and_momenta=false) on RhoTheta and per-component RhoQ_n so the coarse moisture state stays conservation-consistent with the fine state. Called pre-regrid, before plotfile writes (ERF_Plotfile.cpp), and after each level's advance_microphysics.
  • Coalescence bin volumes are scaled by the level's cumulative refRatio in ERF_SuperDropletPCCoalescence.cpp, so physical bin volume is level-invariant.
  • ERF_Tagging.cpp extends field_name refinement to all moisture mixing ratios (aborts at setup if unsupported by the active moisture model) and dispatches <species>_<mesh_var> field names to ERFPC::computeMeshVar with per-level deposit + level-by-level average down, matching the existing _count tagging pattern.
  • ERF::estTimeStep guards against empty fine levels and sets dz_min per level; ClearLevel guards advflux_reg->reset() against null. Fine-level ghost cells are FillPatchFineLevel-ed before each microphysics advance, and z_phys_nd access in particle boundary treatment is guarded for fine levels.
  • New example inputs in Exec/RegTests/Bubble: inputs_BF02_dry_bubble_AMR{1,2} and inputs_3DBF02_dry_bubble{,_AMR1,_AMR2}.
  • 9 new ctests with gold files: SDM_Bubble{2,3}D_Adv_AMR{1,2}, SDM_Bubble3D_Adv, SDM_MoistBubble{2,3}D_AMR{1,2}.

debog and others added 30 commits April 14, 2026 11:17
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…l_distribution_type

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SuperDropletPC was constructed with single-level (Geom,DMap,BA)
constructor, leaving its internal ParGDB unaware of fine levels.
After regrid, FixKIndexAMR and Redistribute silently skipped L1+,
causing ParIter crashes on fine levels. Re-define the PC with
ERF's AmrParGDB after construction.

Made-with: Cursor
advflux_reg is only allocated for TwoWay coupling. ClearLevel
unconditionally called reset() on it, segfaulting with OneWay.

Made-with: Cursor
On fine AMR levels, z_phys_nd covers only the refined sub-region.
Accessing ground (k=0) or roof (k=domain.bigEnd+1) indices read
garbage, falsely deactivating all particles. Check zheight_box
containment before access; fall back to domain extents otherwise.
Also skip forEachParticleTile on levels with no particles.

Made-with: Cursor
Replace bare Redistribute() with Redistribute(0,0) on L0 and
ExtractAndRouteOORParticles on fine levels to avoid cross-level
interference during sub-stepping. Update k-indices on all levels
0..a_lev after redistribution (particles may move between levels).
Guard per-tile advection work with has_particles check.

Made-with: Cursor
Fixes CUDA warning erf-model#611-D (partially overridden virtual function).
Morrison now overrides the 3-arg version (delegates to 2-arg);
SuperDropletsMoist now overrides the 2-arg version (aborts, since
it requires a level argument).

Made-with: Cursor
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…itializes against its own BoxArray

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ame-hiding warning

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursor-agent@cursor.sh>
…cle mesh ops

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
# Conflicts:
#	Source/Microphysics/ERF_EulerianMicrophysics.H
#	Source/Microphysics/ERF_LagrangianMicrophysics.H
#	Source/Microphysics/SuperDropletsMoist/ERF_SuperDropletsMoistAdvance.cpp
#	Source/Microphysics/SuperDropletsMoist/ERF_SuperDropletsMoistInit.cpp
#	Source/Microphysics/SuperDropletsMoist/ERF_SuperDropletsMoistPhaseChange.cpp
#	Source/Microphysics/SuperDropletsMoist/ERF_SuperDropletsMoistUtils.cpp
debog added 30 commits May 1, 2026 15:21
…qs/qg) plus qt; abort at setup if requested field is unsupported by the active moisture model
…ain metric, and pos_phys/pos_comp helpers for the upcoming computational-coordinate particle position refactor
…e floor formula (uses AMReX DefaultAssignor), drop AoS k slot, drop terrain-correction dance (FixKIndexAMR now just Redistribute), drop ExtractAndRouteOORParticles, swap mapped-z interp with plain interp; uniform-z bubble cases pass; ParticleWoA terrain and ParticleAdvect_AMR1_box mass-density gold need follow-up
…particles from crossing levels mid-coarse-step (which caused double-advection on the fine level); reintroduce a simplified ExtractAndRouteOORParticles that routes fine-level OOR particles back to L0 using DefaultAssignor [run-ci]
…rticle species (e.g., super_droplets_moisture_mass_density). field_name = <species>_<mesh_var> dispatches to ERFPC::computeMeshVar; per-level deposit + average down matches the existing _count pattern
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants