From c2bd8fc9c9c4e79854683044d0f9aa2131cb4d16 Mon Sep 17 00:00:00 2001 From: Emily Boudreaux Date: Thu, 17 Sep 2026 17:06:26 -0400 Subject: [PATCH] Document verified Schur coverage and MPI fixture limitation --- .../GRAVITY_CONSISTENT_SCHUR_2026-09-18.md | 66 ++++++++++++++++--- 1 file changed, 58 insertions(+), 8 deletions(-) diff --git a/experiments/GRAVITY_CONSISTENT_SCHUR_2026-09-18.md b/experiments/GRAVITY_CONSISTENT_SCHUR_2026-09-18.md index 4c2150b..6768c34 100644 --- a/experiments/GRAVITY_CONSISTENT_SCHUR_2026-09-18.md +++ b/experiments/GRAVITY_CONSISTENT_SCHUR_2026-09-18.md @@ -108,12 +108,62 @@ The added FEM integration test compares sparse and matrix-free reduced divergenc and transpose actions on three dense vectors, compares the full refined action against the matrix-free coupling implementation, and checks zero-sweep exact equivalence using identical child inverses. It repeats these checks after geometry -and discretization revisions and tests stale rejection, cache rebuilding, and -collective option mismatch rejection when run on multiple ranks. +and discretization revisions and tests stale rejection and cache rebuilding. +It also contains collective option-mismatch checks for multiple ranks, but those +checks have not executed successfully in the actual FEM fixture; see the MPI +limit below. -Tests are provided for the coordinated build/test campaign; no benchmark result -or production improvement is claimed by this implementation note. Compare the -new policy against the default using the same mass backend, AMG cycles, material -inverse, full-J operator, saved right-hand sides, and scalar-border sources. -Evaluate independent and two-way stellar-structure policies separately so a -coupling change is not mistaken for a gravity-inverse improvement. +No benchmark result or production improvement is claimed by this implementation +note. Compare the new policy against the default using the same mass backend, +AMG cycles, material inverse, full-J operator, saved right-hand sides, and +scalar-border sources. Evaluate independent and two-way stellar-structure +policies separately so a coupling change is not mistaken for a gravity-inverse +improvement. + +### Executed coverage and MPI limit + +The retained `tests_v1` executable built from commit +`dc8d4b0501e2775b0d1938005e142411319d78b1` passed the focused serial selection: +**20 cases and 441,978 assertions**, including the new real-FEM Schur refinement +and cached stellar cross-action checks. Its SHA-256 is +`e11a49ebadc85206fad5a998b5cc6ee3ac074b498d09b3578d9ef5e0c5d0c04b`. + +The same executable passed **13 cases and 213 assertions per rank** on two ranks +with selection +`[gravity_block][unit],[preconditioning][stellar_structure][unit],[coupled_shape_coarse]`. +This verifies the selected algebra and distributed coarse tests. It does not +establish distributed FEM Schur assembly, refresh, or option-mismatch handling. + +The broader two-rank selection aborted before reaching the new Schur FEM test. +The existing `Prepared Gravity Block Diagonal Is Legacy Equivalent And Allocation +Stable` fixture constructs `PreparedMappedHDivMassOperator`, whose constructor +requires both a stellar element and a compactified vacuum element **on each +rank**. Rank 1 has no vacuum element in the production partition of +`sandbox.smesh`, so construction calls `MPI_ABORT` at that pre-existing check. + +`fem::setup_fem` currently calls `GeneratePartitioning(rankCount, 1)` internally; +there is no partition override in the existing setup interface. A lightweight, +read-only reconstruction of the mesh dual graph and installed METIS 4 partition +call, using MFEM's descending adjacency order, reproduced the following split: + +| Mesh | Rank | Core | Envelope | Vacuum | +| --- | ---: | ---: | ---: | ---: | +| `sandbox.smesh` | 0 | 0 | 220 | 384 | +| `sandbox.smesh` | 1 | 448 | 164 | 0 | +| Existing one-level refined mesh | 0 | 0 | 1,673 | 3,072 | +| Existing one-level refined mesh | 1 | 3,584 | 1,399 | 0 | + +The refined mesh therefore has the same obstruction; it was inspected through +its partition graph, not subjected to another FEM run. Selecting only the new +test cannot remove its shared geometry-construction prerequisite. A deliberate +partition distributing every material across both ranks could exercise that +fixture, but adding a partition-injection seam is outside this preconditioner +experiment. Production mesh setup and partitioning remain unchanged. + +The precise current validation claim is **serial FEM plus two-rank algebra/coarse +coverage**. Actual FEM Schur refinement, its distributed sparse-cache lifecycle, +and its collective option-rejection branches remain unvalidated on multiple +ranks. Logs, executable hashes, and commands are retained in the workspace under +`diagnostics/preconditioner_improvement_2026-09-18/tests_v1/`, specifically +`provenance.json`, `serial.log`, `mpi2.log`, `mpi2_units.json`, and +`mpi2_units.log`.