[jmarquisbq]

Summary

Correct OOC storage contracts, selected imports, pipeline completion, and filter behavior. Add bounded array reads and working buffers, preserve geometry references, and check results against independent expectations. 197 files changed, +32,125 / −1,657 lines.

Companion PRs:

• BlueQuartzSoftware/SimplnxOoc#4 — BUG: Fix OOC Review Findings enables disk-backed execution and validation of these contracts.
• BlueQuartzSoftware/DREAM3DNX#1171 — BUG: Fix OOC Review Findings presents storage diagnostics and improves large-data visualization. Validate the coordinated OOC configuration with both companions. The public in-memory build remains independent of the private backend.

Publication status, October 23: The tested rebase onto current develop is published. All branch patches are preserved. Final native CI must verify the latest coordinated revisions.

Storage, import, and completion contracts

• src/simplnx/Common/BoundedRead.hpp defines bounded reads with explicit scratch accounting and separate unavailable, failed, and successful outcomes. HDF5 reads validate selections before transfer.
• src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp plans storage and geometry references for each selected destination and alias before publication.
• src/simplnx/DataStructure/DataStructure.cpp owns import publication and rollback. It validates object identities and parent placements before mutation, preserves existing objects, and retains warnings and notifications.
• src/simplnx/Pipeline/Pipeline.cpp propagates checked store-flush failures through pipeline completion. Recovery writes and nxrunner retain the corresponding errors and warnings.

Numeric copies and storage conversions retain growth initialization. Parameter clones retain storage-location restrictions. Large-data thresholds reject or correct invalid values with diagnostics. The preferences file stores them in GB with two decimals as large_data_size_gb and large_datastructure_size_gb. A legacy byte value loads on the 0.01-GB grid, and the next save writes the GB key. An invalid GB value uses the default with warning -592; an invalid legacy byte value keeps warning -591. Memory and the public API keep the values in bytes.

Filter correctness and bounded working memory

• src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.cpp computes periodic centroids from unwrapped cell centers. Bounded occupancy scans and a defined ambiguous-seam fallback support the documented numerical contract.
• src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.cpp batches YZ transfers. Mesh normals, angles, winding, shared faces, and smoothing also use bounded transfers with checked connectivity and partial grants.
• src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/OrientationTupleLookup.hpp admits face buffers and orientation lookups together. A bounded page route handles metadata that does not fit a complete cache.
• src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/LaplacianSmoothing.cpp supports temporary coordinate storage under constrained memory. Calculation failures preserve original coordinates; final-write failures can leave a completed prefix.

KMeans defaults to one cluster and rejects zero in both implementations. Smoothing, winding, and Zero Crossing honor shared algorithm-override precedence while retaining the storage checks required for direct access. Avizo writers check reads, writes, flushes, closes, and size arithmetic; their coordinate conventions and binary separators remain unchanged.

Regression coverage and maintainability

• The tests use analytical centroid and mesh expectations, literal image values, complete output checks, and injected transfer failures. They cover cancellation, block tails, aliases, rollback, and override precedence.
• test/UnitTestCommon/include/simplnx/UnitTest/ProcessTestProfile.hpp isolates process settings. Test scopes restore the incoming resolver and check cleanup.
• Floating-array comparisons reject mismatched NaNs. Platform guards admit the reviewed standard-library configurations, and test-only interfaces remain conditional on their instrumentation.

Test plan

• Coordinated Windows review builds passed. Affected selections passed 616/616 in-memory Release tests and 1,404/1,404 tests in each OOC Release and Debug configuration.
• Both Windows Release aggregate builds passed after the local rebase. The rebase selections passed 191/191 in-memory and 199/199 OOC tests, with no skips.
• Numerical checks include independent centroid, triangle, smoothing, and writer-output expectations. Disk-backed selections verify actual storage and outputs; an OOC algorithm flag alone is not treated as proof.
• The complete three-PR C/C++ selection passed clang-format 19.1.1. Scoped static-analysis findings were triaged; no clean whole-project clang-tidy result is claimed.
• The GB threshold change passed on macOS (Apple silicon) in the coordinated OOC configuration. The complete [ooc] selection passed 924/924 tests, and the new PreferencesSizeGBTest and the changed preference tests passed in OOC Release and OOC Debug.
• Final Windows, Linux, and macOS CI passes on the published revisions, including the coordinated OOC configuration.

Laplacian smoothing still retains approximately 32 bytes per vertex in older resident working arrays. The whole filter is not yet bounded by its OOC allowance. The full all-filter coverage audit, general preference-type validation, array-kind policy, and library-version-gate redesign remain separate work.

Related Issue

Related to #1532 — expands algorithm and failure-path tests without completing the coverage inventory.

Related to #481 — expands geometry-reference, import, and persistence coverage without completing the HDF5 audit.

Related to #1768 — includes compatibility repairs; the allocation-policy redesign remains separate.

Related to #1835 — bounds newer smoothing buffers; the older resident arrays remain.

[joeykleingers]

Commit b8a1a88 adds the fixes from a review of this pull request together with BlueQuartzSoftware/DREAM3DNX#1171 and BlueQuartzSoftware/SimplnxOoc#4.

• Identify Sample (YZ mode, out-of-core storage): the first batch is published without extra planes. The first write to an imported read-only mask can need working memory from the same allowance as the extra planes; with an exact fill (for example four 64 MiB planes and a 1 GiB budget) that write failed. A regression test with a real imported mask failed before the fix.
• Triangle normals (out-of-core output): the normal of triangle 0 is computed and written before the compute reservation is kept, and the plan is made again after that write. The step never reads the stored normals back. New error codes: -65817 (input read), -65818 (output write) and -65819 (the second reservation gets no memory). Cancellation after that write stops before the next reservation. When a later step fails, triangle 0 already has its new normal; the test TriangleNormalFilter: real HDF5 invalid connectivity now expects that.
• DataStructure::flushChecked() stops at the first object that fails. It returns that object's errors with the warnings collected before it. The later objects are not flushed in that call and keep their changes in memory.
• DiagnosticRepresentationSupported() accepts Apple libc++ 220106 (Xcode 27.0) with ABI 1, C++20 and ordinary new, after a review of the installed headers. The native support tests pass with it.
• Compute Feature Centroids and Verify Triangle Winding use std::unique_ptr<std::vector<T>> buffers.

Verification on macOS arm64 Release: the affected selections pass in the in-core and out-of-core builds, and the complete out-of-core CTest suite passes except one load-sensitive visualization test in DREAM3DNX (see the comment on BlueQuartzSoftware/DREAM3DNX#1171). Windows, Linux and Debug builds were not run.

[imikejackson]

@jmarquisbq I request changes before this pull request merges. The bug fixes are good. My request is about the memory accounting in the bounded reads.

Request: count only data payload bytes against the memory budget

The bounded-read code tries to prove that each operation stays inside its memory grant, down to the byte. It counts Result objects, error vectors, message strings, empty std::vector headers, locks, spans and expected-lite temporaries. That proof depends on how each standard library allocates memory inside std::vector and std::string. For this reason, DiagnosticRepresentationSupported() (src/simplnx/Common/BoundedRead.hpp:110) is an allow-list of reviewed standard-library versions, and the bounded read refuses on any other compiler.

Please change the rule: the budget counts only buffers whose size changes with the data (compressed and decoded chunks, tiles, planes, working arrays). Fixed costs such as diagnostics, path copies, locks and spans are overhead, and the budget does not count them.

Reasons:

1. The CacheMemoryBudgetManager header (src/simplnx/Utilities/CacheMemoryBudgetManager.hpp:32) says that the budget is not a total-application limit. Byte-exact proof of the error path gives more precision than the budget has.
2. The allow-list breaks with each compiler update. This branch already has approximately six commits that only add a compiler to the list.
3. DataStore::readExtentIntoBufferBounded() (src/simplnx/DataStructure/DataStore.hpp:472) allocates no scratch memory, but it refuses when the budget is less than 4096 bytes or when the compiler is not on the list.
4. The accounting code, its comments and its tests are a large part of this pull request.

Changes in this repository

• src/simplnx/Common/BoundedRead.hpp: keep Status, Code, Multiply(), Add() and ValidateExtent(). Remove DiagnosticRepresentationSupported(), EmptyVectorProxyBytes(), EmptyResultControlBytes(), EmptyInvalidResultControlBytes(), TerminalDiagnosticBoundBytes() with its static_assert (:521), ResultCapacityBytes(), ObserveTerminalForTesting() and k_DiagnosticBytes. Make FreshVectorBytes() a checked count * width, or remove it. Use the usual MakeErrorResult() path for diagnostics.
• src/simplnx/DataStructure/DataStore.hpp:472: remove the first if block. The in-memory read ignores the budget and always reads.
• src/simplnx/Utilities/Parsing/HDF5/BoundedChunkRead.cpp: remove the control-size sums (:295–:299), the path-capacity accounting (:492) and the allow-list checks (:499, :592). Do not subtract k_DiagnosticBytes (:597). Required bytes = compressed + decoded bytes of the active chunks. If the budget is too small, return Unavailable with an Insufficient warning that gives the required and available bytes.
• Allocate data buffers with ScratchBuffer<T> or std::unique_ptr<std::byte[]>, so that the payload size is exact on every compiler.
• Tests: remove test/DataStoreBulkTest.cpp:607 and the ResultCapacityBytes() assertions. Keep the value, validation, failure and cancellation tests. Add these budget tests: required − 1 gives Unavailable and an unchanged destination; the exact required bytes give Complete; an in-memory store with budget 0 reads correctly.

The companion pull requests BlueQuartzSoftware/SimplnxOoc#4 and BlueQuartzSoftware/DREAM3DNX#1171 need the same change in the out-of-core storage plugin and the histogram sampler. The complete design note for all three pull requests is in BlueQuartzSoftware/SimplnxOoc#4 (https://github.com/BlueQuartzSoftware/SimplnxOoc/pull/4#issuecomment-6504925290). Make the simplnx change first, because the two companions use this API.

#1768

After this change, there is no allow-list to replace. Please add Closes BlueQuartzSoftware/simplnx#1768 to the ## Related Issue section, and add a short comment to #1768 that tells why no replacement is necessary.