Repository navigation
BUG: Enforce component shapes and types that filters read - #1766
imikejackson merged 23 commits into
Conversation
joeykleingers
left a comment
There was a problem hiding this comment.
Reviewed at 25ba5f78. The fixes are right; the changes below are small.
Requested changes
1. A StringArray mask now makes preflight throw (DBSCAN, ComputeKMeans, ComputeKMedoids).
ArraySelectionParameter::validatePath accepts any IArray and only runs the type and component-shape checks on an IDataArray, so a StringArray passes the bool/uint8 {1} declaration. When its tuple count differs from Attribute Array to Cluster, CheckArraysHaveSameTupleCount detects it (it reads IArray), but the new error message calls getDataRefAs<IDataArray>(maskPath), which throws std::bad_cast. IFilter::preflight does not catch it.
src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp:159src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMeansFilter.cpp:135src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp:140
Fix: use getDataRefAs<IArray> in those three messages. (The StringArray gap in ArraySelectionParameter itself predates this PR and can be a separate fix.)
2. DBSCAN documentation.
src/Plugins/SimplnxCore/docs/DBSCANFilter.md:191states the 1-component rule unconditionally and omits the new tuple-count rule (the KMeans and KMedoids pages include it). Suggested text: "When Use Mask Array is enabled, Cell Mask Array must have 1 component and the same number of tuples as Attribute Array to Cluster."src/Plugins/SimplnxCore/vv/deviations/DBSCANFilter.md:138names the testsSimplnxCore::DBSCANFilter: Reject …; the registered names areSimplnxCore::DBSCAN: Reject multi-component mask in preflightandSimplnxCore::DBSCAN: Reject short mask in preflight(DBSCANTest.cpp:681,:705).- The PR says the DBSCAN V&V documents now reference 6.5.171 only, but the linked
src/Plugins/SimplnxCore/vv/provenance/dbscan_test.md:106still describes a 6.5.172 comparison.
3. PR description.
- The diff is +649/−30, not +535/−30.
- Behavior changes omits the ComputeKMeans (−7588) and ComputeKMedoids (−7589) mask tuple-count checks.
- "Each new test fails on develop" is too broad: the valid-phase counting case in
ComputeNumFeaturesTest.cpppasses ondevelop. AllowedComponentShapes{{1}}/{{3}}compare shapes exactly, so{1,1}and{1,3}arrays are now rejected although the algorithms read them correctly. That matches the usual simplnx convention, but "each of these inputs was already read incorrectly" is not strictly true; a sentence noting it would cover it.
Checked and correct
- The
{1}and{3}constraints match the strides the algorithms read. - Mask validation is skipped when masking is off, including an empty mask path; bool and uint8 masks are still accepted.
- The GBCD shape check tests the rank before indexing dimension 5 and matches the layout Compute GBCD produces.
- ComputeNumFeatures −9740 rejects negative phases and phases ≥ the Ensemble tuple count, skips feature 0, and runs before any counts are written.
- The removed
DBSCANScanlineguards are unreachable through the filter: Attribute Array to Cluster is limited to{2}/{3}and the mask shape and tuple count are now checked in preflight. - New error codes do not collide; no parameter-version bump is needed; no checked-in example pipeline or Python example is broken by the new rules.
- All changed C++ files match clang-format 19.1.1.
Pre-existing gaps (not from this PR; possible follow-up issue)
WriteGBCDGMTFile.cpp:155indexes Crystal Structures with Phase of Interest without checking that array's length or a negative phase.- GBCD dimension products are computed in int32 and overflow for a large but valid shape (e.g.
{1024,1024,1024,1,1,2}). - Unchecked tuple counts between related inputs: normals vs. triangles (ComputeVertexToTriangleDistances), centroids vs. the output Feature Attribute Matrix (ComputeShapesTriangleGeom), Feature Ids vs. mask (ComputeFeaturePhasesBinary), Feature Ids vs. the geometry's cell count (ComputeShapes).
- ComputeNumFeatures: the int32 counter can overflow, and neither loop checks cancellation or reports progress.
Not verified: no build or test run was done for this review, so the test-plan items (local suite, failure on develop, CI) rely on the PR's own evidence.
25ba5f7 to
d0724ec
Compare
|
@joeykleingers All items are fixed as requested |
joeykleingers
left a comment
There was a problem hiding this comment.
Re-reviewed at 65a3038e. The three StringArray fixes and their tests are right, and the DBSCAN page, test names and PR description are fixed. One item still needs a change; the rest are small.
Requested change
The DBSCAN provenance record now names a legacy version the evidence does not support.
src/Plugins/SimplnxCore/vv/provenance/dbscan_test.md:101 and :106 relabel the August 5 comparison from 6.5.172 to 6.5.171. Everywhere else, the DBSCAN report describes that run as a local legacy proof build:
src/Plugins/SimplnxCore/vv/DBSCANFilter.md:22- the Legacy runner line in its Phase 9 setup
src/Plugins/SimplnxCore/vv/deviations/DBSCANFilter.md:22
docs/vv_templates/report_gates.md:27 says to describe a patched local build exactly that way and not to give it a release version. As written, the record says a stock 6.5.171 binary produced those results. (My last review only noted the mismatch; correcting the version in this direction isn't supported.)
Fix: use version-neutral wording, e.g. "The historical local legacy proof build, using the DREAM3DReview DBSCAN filter, was run against the same six sklearn datasets.", with a matching heading.
Small items
src/Plugins/SimplnxCore/vv/DBSCANFilter.md:20: Tests today says 14 tests plus 1 hidden. There are now 15 plus 1 hidden, andSimplnxCore::DBSCAN: Reject short StringArray mask in preflightis not in the coverage table (it belongs with the tuple-count row,Reject short mask in preflight).src/Plugins/SimplnxCore/docs/ComputeKMeansFilter.md:15andsrc/Plugins/SimplnxCore/docs/ComputeKMedoidsFilter.md:67still state the 1-component rule unconditionally, the same wording just fixed inDBSCANFilter.md. Suggested text: "When Use Mask Array is enabled, Cell Mask Array (KMedoids: Mask Array) must have 1 component and the same number of tuples as Attribute Array to Cluster."- PR description, Note for reviewers: #1763 is already in the base (
8bed04ce9), so "conflicts will be resolved when those merge" no longer applies to it.
Checked and correct
- The rebase onto the current
developkept both develop's #1763 changes and this PR's additions in the ComputeKMeans page, filter and tests. The 12 original commits are otherwise unchanged. getDataRefAs<IArray>is safe at all three call sites: parameter validation rejects missing paths and non-arrays beforepreflightImplruns.- The three new tests reach the branch that threw at
25ba5f78, and they are deterministic. - Each of −7587, −7588, −7589, −9740, −34645 and −96713 is emitted at exactly one place in the source.
- The diff is 34 files, +743/−32, matching the description.
Separate follow-up (not this PR)
A StringArray mask with the right number of tuples still passes preflight, as it does on develop. Execute then returns an error in most paths, but ComputeKMeans Scanline treats it as no mask and silently ignores it. That fits with the existing ArraySelectionParameter StringArray gap as a separate fix.
Correction to my last review: WriteGBCDGMTFile preflight does reject a negative phase. A short Crystal Structures array is still unchecked there.
V&V documents and test comments named DREAM3D 6.5.172, which is a private proof branch and was never released. Each reference now names DREAM3D 6.5.171 or describes the locally patched legacy build; no sentence claims that 6.5.171 contains a fix that only the patched build had. The FillBadData summary no longer cites the withdrawn three-way A/B run as evidence. The DBSCAN files are left for BlueQuartzSoftware#1766. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* DBSCANFilter.md: keep only the template sections and fields. Remove the Phase 2 to Phase 13 working notes. Correct the Algorithm Relationship evidence: PR BlueQuartzSoftware#1421 did not change the filter UUID. * Record Michael Jackson as the second engineer (2026-08-28). * Give stable IDs to the Random-seed defect (DBSCAN-D7) and the all-false-mask defect (DBSCAN-D8), and state the affected releases for D5 to D8. * Record the comparison against DREAM3D 6.5.171 (2026-10-20). It gives the same label for every point as the earlier local proof version. Put the three-way table in DBSCAN-D1 as its evidence. * deviations/DBSCANFilter.md: split into Part 1 (D1 to D3) and Part 2 (D4 to D8), and remove the incorrect note about a UUID change. * provenance/dbscan_test.md: cite the DREAM3D 6.5.171 run and record the second-engineer review. * The record describes tests and DBSCAN-D6 from PR BlueQuartzSoftware#1766, so merge this change after that PR. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
V&V documents and test comments named DREAM3D 6.5.172, which is a private proof branch and was never released. Each reference now names DREAM3D 6.5.171 or describes the locally patched legacy build; no sentence claims that 6.5.171 contains a fix that only the patched build had. The FillBadData summary no longer cites the withdrawn three-way A/B run as evidence. The DBSCAN files are left for BlueQuartzSoftware#1766. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* DBSCANFilter.md: keep only the template sections and fields. Remove the Phase 2 to Phase 13 working notes. Correct the Algorithm Relationship evidence: PR BlueQuartzSoftware#1421 did not change the filter UUID. * Record Michael Jackson as the second engineer (2026-08-28). * Give stable IDs to the Random-seed defect (DBSCAN-D7) and the all-false-mask defect (DBSCAN-D8), and state the affected releases for D5 to D8. * Record the comparison against DREAM3D 6.5.171 (2026-10-20). It gives the same label for every point as the earlier local proof version. Put the three-way table in DBSCAN-D1 as its evidence. * deviations/DBSCANFilter.md: split into Part 1 (D1 to D3) and Part 2 (D4 to D8), and remove the incorrect note about a UUID change. * provenance/dbscan_test.md: cite the DREAM3D 6.5.171 run and record the second-engineer review. * The record describes tests and DBSCAN-D6 from PR BlueQuartzSoftware#1766, so merge this change after that PR. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
V&V documents and test comments named DREAM3D 6.5.172, which is a private proof branch and was never released. Each reference now names DREAM3D 6.5.171 or describes the locally patched legacy build; no sentence claims that 6.5.171 contains a fix that only the patched build had. The FillBadData summary no longer cites the withdrawn three-way A/B run as evidence. The DBSCAN files are left for BlueQuartzSoftware#1766. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* DBSCANFilter.md: keep only the template sections and fields. Remove the Phase 2 to Phase 13 working notes. Correct the Algorithm Relationship evidence: PR BlueQuartzSoftware#1421 did not change the filter UUID. * Record Michael Jackson as the second engineer (2026-08-28). * Give stable IDs to the Random-seed defect (DBSCAN-D7) and the all-false-mask defect (DBSCAN-D8), and state the affected releases for D5 to D8. * Record the comparison against DREAM3D 6.5.171 (2026-10-20). It gives the same label for every point as the earlier local proof version. Put the three-way table in DBSCAN-D1 as its evidence. * deviations/DBSCANFilter.md: split into Part 1 (D1 to D3) and Part 2 (D4 to D8), and remove the incorrect note about a UUID change. * provenance/dbscan_test.md: cite the DREAM3D 6.5.171 run and record the second-engineer review. * The record describes tests and DBSCAN-D6 from PR BlueQuartzSoftware#1766, so merge this change after that PR. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
V&V documents and test comments named DREAM3D 6.5.172, which is a private proof branch and was never released. Each reference now names DREAM3D 6.5.171 or describes the locally patched legacy build; no sentence claims that 6.5.171 contains a fix that only the patched build had. The FillBadData summary no longer cites the withdrawn three-way A/B run as evidence. The DBSCAN files are left for BlueQuartzSoftware#1766. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* DBSCANFilter.md: keep only the template sections and fields. Remove the Phase 2 to Phase 13 working notes. Correct the Algorithm Relationship evidence: PR BlueQuartzSoftware#1421 did not change the filter UUID. * Record Michael Jackson as the second engineer (2026-08-28). * Give stable IDs to the Random-seed defect (DBSCAN-D7) and the all-false-mask defect (DBSCAN-D8), and state the affected releases for D5 to D8. * Record the comparison against DREAM3D 6.5.171 (2026-10-20). It gives the same label for every point as the earlier local proof version. Put the three-way table in DBSCAN-D1 as its evidence. * deviations/DBSCANFilter.md: split into Part 1 (D1 to D3) and Part 2 (D4 to D8), and remove the incorrect note about a UUID change. * provenance/dbscan_test.md: cite the DREAM3D 6.5.171 run and record the second-engineer review. * The record describes tests and DBSCAN-D6 from PR BlueQuartzSoftware#1766, so merge this change after that PR. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
75f656e to
8d993b1
Compare
8d993b1 to
ad3341f
Compare
V&V documents and test comments named DREAM3D 6.5.172, which is a private proof branch and was never released. Each reference now names DREAM3D 6.5.171 or describes the locally patched legacy build; no sentence claims that 6.5.171 contains a fix that only the patched build had. The FillBadData summary no longer cites the withdrawn three-way A/B run as evidence. The DBSCAN files are left for BlueQuartzSoftware#1766. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6d5ee82 to
b4753c2
Compare
* CS-1: the Triangle Normals parameter declared no component shape, but the algorithm reads 3 component(s) per tuple, so a DataArray with another component shape was read with the wrong stride (wrong output or a read past the end of the array). The parameter now declares a component shape of 3, and parameter validation rejects other shapes with -208 * Document the requirement and add a preflight test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CS-2: the Face Feature Centroids parameter declared no component shape, but the algorithm reads 3 component(s) per tuple, so a DataArray with another component shape was read with the wrong stride (wrong output or a read past the end of the array). The parameter now declares a component shape of 3, and parameter validation rejects other shapes with -208 * Document the requirement and add a preflight test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CS-4: the Cell Feature Ids parameter declared no component shape, but the algorithm reads 1 component(s) per tuple, so a DataArray with another component shape was read with the wrong stride (wrong output or a read past the end of the array). The parameter now declares a component shape of 1, and parameter validation rejects other shapes with -208 * Document the requirement and add a preflight test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CS-6: the Cell Feature Ids parameter declared no component shape, but the algorithm reads 1 component(s) per tuple, so a DataArray with another component shape was read with the wrong stride (wrong output or a read past the end of the array). The parameter now declares a component shape of 1, and parameter validation rejects other shapes with -208 * Document the requirement and add a preflight test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CS-7: the Cell Mask Array parameter declared no component shape, but the algorithm reads 1 component(s) per tuple, so a DataArray with another component shape was read with the wrong stride (wrong output or a read past the end of the array). The parameter now declares a component shape of 1, and parameter validation rejects other shapes with -208 * Document the requirement and add a preflight test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CS-8: the Mask Array parameter declared no component shape, but the algorithm reads 1 component(s) per tuple, so a DataArray with another component shape was read with the wrong stride (wrong output or a read past the end of the array). The parameter now declares a component shape of 1, and parameter validation rejects other shapes with -208 * Document the requirement and add a preflight test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CS-3: the Feature Phases parameter declared no component shape, but execute reads one value per feature. It now declares a component shape of 1 (parameter validation rejects others with -208) * CS-14: the parameter accepted every DataType, but execute reads it as an Int32Array, so any other DataType was dereferenced as null. The parameter now accepts only int32 (-207 otherwise) * CS-15: a Feature Phases value below 0 or not less than the Ensemble Attribute Matrix tuple count was used directly as an index for a write. Execute now checks every value first and returns -9740 * Document the requirements and add tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CS-11: the Input GBCD DataArray had no component-shape check. The algorithm reads 5 bin dimensions from the component shape and then 2 hemisphere values per bin, so a DataArray with another shape read past the end of the shape vector or of the data. Preflight now requires 6 component dimensions, the last equal to 2 and the others positive, and returns -34645 otherwise. The bin dimensions depend on the GBCD resolution, so a fixed parameter shape cannot express this * Document the requirement and add tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CS-12: the GBCD DataArray had no component-shape check. The algorithm reads 5 bin dimensions from the component shape and then 2 hemisphere values per bin, so a DataArray with another shape read past the end of the shape vector or of the data. Preflight now requires 6 component dimensions, the last equal to 2 and the others positive, and returns -96713 otherwise. The bin dimensions depend on the GBCD resolution, so a fixed parameter shape cannot express this * Document the requirement and add tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CS-13: the Cell Mask Array parameter declared no component shape, and
preflight did not check that the Mask Array has the same number of
tuples as the Attribute Array to Cluster. The Scanline algorithm
rejected both at execute (-54062), but the Direct algorithm read the
mask by flat element, so the two algorithms disagreed
- The parameter now declares a component shape of 1 (-208)
- Preflight returns -7587 when Use Mask is on and the tuple counts
differ
- The Scanline execute checks -54062 and -54060 are removed, because
parameter validation and preflight now make them unreachable
* Add V&V deviation DBSCAN-D6 and the two new tests to the V&V report.
The deviations document now names DREAM3D 6.5.171, the reference
release for V&V documents
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When masking is enabled, preflight now rejects a Cell Mask Array whose number of tuples differs from the Attribute Array to Cluster (-7588). Previously the mismatch reached execute, where the Direct path could throw std::out_of_range and the Scanline path failed with a DataStore range error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When masking is enabled, preflight now rejects a Mask Array whose number of tuples differs from the Attribute Array to Cluster (-7589). Previously the mismatch reached execute, where both the Direct and Scanline eligibility pre-scans failed with a DataStore range error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mask parameter accepts any IArray, and a StringArray passes it. When its tuple count differs from Attribute Array to Cluster, the tuple-count error message cast the mask to IDataArray, which threw std::bad_cast out of preflight. The message now reads the tuple count through IArray, so preflight returns error -7587. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mask parameter accepts any IArray, and a StringArray passes it. When its tuple count differs from Attribute Array to Cluster, the tuple-count error message cast the mask to IDataArray, which threw std::bad_cast out of preflight. The message now reads the tuple count through IArray, so preflight returns error -7588. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The mask parameter accepts any IArray, and a StringArray passes it. When its tuple count differs from Attribute Array to Cluster, the tuple-count error message cast the mask to IDataArray, which threw std::bad_cast out of preflight. The message now reads the tuple count through IArray, so preflight returns error -7589. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The filter doc now states the mask tuple-count rule and that both rules apply only when Use Mask Array is enabled. The deviation doc uses the registered test names. The provenance note refers to DREAM3D 6.5.171. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Restore the DBSCAN V&V report, deviations, and provenance files to their develop versions. * This PR now changes no V&V record. The DBSCAN V&V update moves to topic/vv_report_cleanup, which merges after this PR. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
* DBSCANScanline can be called without DBSCANFilter. Its out-of-core path again checks the coordinate component count (-54060) and the mask component count and tuple count (-54062) before it reads them. Without these checks, a 4-component input writes past a 3-element array and a 2-component mask is read as one value per tuple. * The check that compares the component count with the selected dimensionality is not restored, because it cannot fail. * The error messages now give the actual counts. * Add a test that calls DBSCANScanline directly on its out-of-core path. A scoped replacement of the HDF5 I/O manager supplies the external-sort and record-store providers, and restores the original manager when the test ends, also when an assertion fails. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
71b9845 to
05a7bd1
Compare
… rule condition Add UnitTest::LoadPlugins() as the first statement of the six new component shape TEST_CASEs in the DBSCAN, Compute GBCD Pole Figure and Write GBCD GMT File tests. State in the K Means and K Medoids docs that the mask component and tuple rules apply only when Use Mask Array is enabled.
Add UnitTest::LoadPlugins() as the first statement of the ten DBSCAN TEST_CASEs that predate this branch, so every TEST_CASE in the file starts with it.
- Remove UnitTest::LoadPlugins() from the TEST_CASEs this branch adds and from the ten older DBSCAN TEST_CASEs. Each one constructs its filter or algorithm directly and never looks a filter up by UUID, so it needs no plugins. The SIMPL backwards-compatibility tests, which load pipelines, keep the call. - Give the resident DBSCAN Scanline path the same -54060 message as the out-of-core path, with the actual component count. - Move the TriangleGeom and VertexGeom includes into the simplnx include group.
Summary
This change makes filter parameters declare the component shape or DataType that their algorithms read. Before, these filters accepted a DataArray with any component shape and then read it with a fixed stride, so the result was wrong output, a read past the end of an array, or a crash. Parameter validation now rejects such a DataArray before the filter runs. Where a fixed parameter shape cannot express the requirement, preflight checks it. Each fix has its own commit and a regression test. No test data archive is added or changed.
Behavior changes
Saved pipelines that select one of the inputs below with another shape or DataType now fail parameter validation or preflight. In most cases such an input was read incorrectly or crashed. The shape check is exact, so a
{1,1}or{1,3}DataArray is now also rejected, even though the algorithm read it correctly. This follows the usual simplnx convention.std::bad_cast. (The mask parameter itself still accepts a StringArray; that is an older gap inArraySelectionParameterand is left for a separate fix.)Test plan
developbecause preflight accepts the wrong shape, DataType or tuple count, or because the out-of-range phase is not rejected. The ComputeNumFeatures valid-phase case also passes ondevelop; it guards against over-rejection.