ENH: Bound remaining Laplacian smoothing scratch memory for OOC #1835

Summary

Extend Laplacian Smoothing so its out-of-core (OOC) path can process the remaining per-vertex working data within a fixed working-memory allowance. Address this in a separate OOC-focused branch and PR, while preserving the in-memory path and its results.

The current OOC work in PR #1764 bounds the new coordinate, edge, and node-type copies. It also uses temporary coordinate storage when the coordinate cache does not fit. Three older working arrays still require RAM proportional to the entire vertex count. This issue tracks that remaining capacity limit; it does not request reopening the completed coordinate-buffer fixes.

Current memory cost

Source checked at d1210b4dc3cd5bd74e6c562c5e0c42c079e8ee7b, the verified head of PR #1764 on October 19, 2026:

The lambda allocation and connection-count and delta allocations are ordinary vectors. They are outside the working-memory reservations used for the newer buffers. The arrays are reused between passes, so increasing the iteration count does not multiply this allocation.

These are decimal units and payload estimates. They exclude allocator overhead, coordinate/edge buffers, caches, edge preparation, and other application memory. The current low-memory coordinate fallback also uses a temporary file of 12 * vertexCount bytes for float32 XYZ coordinates. The total temporary-file cost of the proposed extension remains to be determined.

Forcing OOC storage or reducing the cache allowance does not move these three vectors onto disk. An input mesh can be larger than RAM while these arrays still fit, but the arrays eventually become a RAM-based limit. Paging can make the operation very slow, and insufficient memory can cause failure. The current OOC path catches C++ allocation failures; that does not remove the size limit or make the cache setting a process-wide RAM cap.

The earlier 65,536-vertex timing fixture needed about 2 MiB for these arrays in addition to its 64 KiB allowance for the new buffers. That test did not establish bounded total smoothing memory. The costs above are derived from the current types and allocations; this issue does not claim a newly reproduced large-mesh out-of-memory failure.

Proposed approach — final storage and batching design remains open

• Keep the default in-memory path unchanged. Retain an efficient OOC fast path when all simultaneous working allocations fit an admitted allowance.
• Include the remaining working state in the memory plan. Avoid requests that needlessly evict cached data, honor partial grants, and return unused reservations promptly. A low budget must select bounded work or return a clear error, without falling back to unaccounted whole-mesh vectors.
• Consider deriving smoothing strengths from bounded node-type pages during coordinate updates, so a separate whole-mesh lambda vector is unnecessary. Store or compute connection counts and movement sums in bounded blocks, using existing public temporary-record/store interfaces where suitable. Choose the final layout and access strategy after measurement.
• Preserve float32 coordinate subtraction, double accumulation, the order of contributions to each vertex, and the current coordinate-update rules. Complete each pass's movement calculation before updating its coordinates. A simple per-edge disk read/modify/write replacement could cause excessive small I/O and is not an acceptable performance assumption.
• Review final coordinate publication. Its fitting-cache path currently reuses the completed RAM delta allocation to hold coordinate bytes while releasing other buffers. Removing that allocation must preserve the ability to make imported arrays writable without starving the store of working memory.
• Preserve completed-pass cancellation, error and warning propagation, and temporary-file cleanup. Calculation failures must leave the original coordinates unchanged. A failure during final publication may leave a completed prefix written; do not claim atomic rollback without a separate design decision.
• Keep the implementation storage-neutral in simplnx. Put concrete HDF5/provider tests in SimplnxOoc where needed. Do not add concurrency, locking, or a new storage framework without a demonstrated requirement.

Testing and acceptance needed

• Demonstrate the remaining memory limit with safe generated inputs and a deliberately small working allowance. The regression must detect whole-mesh scratch allocation; small I/O transfers or an OOC build label alone are not sufficient evidence. Do not require deliberately exhausting the machine's RAM.
• Verify actual disk-backed stores, including writable and imported read-only HDF5 arrays, compressed and uncompressed inputs, partial grants, insufficient minimum grants, and final partial blocks. Check peak simultaneous owned scratch, reservation release, and temporary-file size/cleanup. Record process memory separately from the application's buffer accounting.
• Use an independent numerical oracle. Reuse the complete four-vertex graph (K4) fixtures: for uniform smoothing strength lambda and centroid C, one pass satisfies v_next = C + (1 - 4*lambda/3) * (v - C). For lambda = 1/4, the factor is 2/3; a positive/negative pair with mu factor -1 gives 8/9. Give graphs distinct positions and scales so transfers from the wrong graph cannot pass. Include multiple iterations, Taubin smoothing, and node classifications with different or zero smoothing strengths. Agreement with the old output alone is not the oracle.
• Cover edges that cross page boundaries, sparse and irregular connectivity, different vertex degrees, and shapes that do not divide evenly into blocks. Review and explicitly test the existing behavior for isolated vertices without silently changing the scientific result contract.
• Cover cancellation before execution and after the positive and negative passes. Check the algorithm's warning contract and the filter's cancellation-result contract separately. Cover empty meshes, zero iterations, and invalid connectivity.
• Inject temporary-store creation, read, short-read, write, and final-publication failures. Check error priority when cancellation and an I/O failure occur together, unchanged original coordinates before publication, the documented completed-prefix behavior afterward, and cleanup on every exit.
• Recheck imported-array writable copies under memory pressure, including failure during that copy. Reopen source and output files independently to verify source preservation and correct results.
• Benchmark the current implementation and the proposed one with identical inputs. Include fitting and constrained allowances, normal and crowded caches, several mesh sizes and layouts, and iteration counts. Report repeated timings, peak RAM, transfer counts/bytes, temporary disk use, and cleanup. Agree on the performance tradeoff from measurements rather than choosing an arbitrary cutoff.
• Run the relevant in-memory and OOC configurations in Release and Debug on Windows, Linux, and macOS. Confirm that the in-memory path and fitting OOC path do not acquire an unintended slowdown.
• Measure end-to-end peak memory, including edge preparation, before describing the whole filter as bounded. Document any remaining mesh-sized allocations or limits explicitly.

Existing starting points are LaplacianSmoothingNumericalTest.cpp, LaplacianSmoothingTransferTest.cpp, and the companion SimplnxOoc/test/PluginTests/SimplnxCore/MeshWorkingMemoryOocTest.cpp coverage. Reuse these contracts and extend them for the remaining scratch state.

This enhancement is deferred from fix/ooc-review-findings to a separate OOC branch. It is separate from closed issue #1422, which concerned leftover edge data. Searches of open and closed issues for Laplacian, smoothing, OOC working memory, and per-vertex memory found no matching issue.