From c6e144f937bdaea4bf50e0b8600021951e23448f Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 27 Oct 2026 12:16:20 -0400 Subject: [PATCH 01/13] DOC: Add coding rules, a knowledge base and a shared AGENTS.md * Add rules to docs/Code_Style_Guide.md that a careful C++ developer could still break in simplnx. An algorithm checks its own inputs, so it runs correctly without its filter. The rules tell where each check goes, how to check component shapes, tuple counts and indices that come from data, and why preflight must not throw. They also cover result propagation, saturating casts, static header caches, out-of-core stores, working-memory reservations, error codes, tests and the Python interpreter of the tests. * Add docs/Coding_Knowledge_Base.md with 19 lessons from pull requests and issues since March 2026. Each entry gives the rule, the pattern to use and to avoid, the cause, the pull request or issue that showed it, and how to recognize it next time. * Link the knowledge base from CONTRIBUTING.md and from the style guide. * Add AGENTS.md for all coding agents. It names the required skills, tells where a new rule goes, and gives the repository layout and the build and test commands. Remove AGENTS.md from .gitignore so that the file is shared. Signed-off-by: Joey Kleingers --- .gitignore | 1 - AGENTS.md | 83 ++++++ CONTRIBUTING.md | 2 +- docs/Code_Style_Guide.md | 268 +++++++++++++++++ docs/Coding_Knowledge_Base.md | 545 ++++++++++++++++++++++++++++++++++ 5 files changed, 897 insertions(+), 2 deletions(-) create mode 100644 AGENTS.md create mode 100644 docs/Coding_Knowledge_Base.md diff --git a/.gitignore b/.gitignore index 88ca8081a5..3d9cf5354c 100644 --- a/.gitignore +++ b/.gitignore @@ -33,7 +33,6 @@ Workspace CLAUDE.md .claude docs/superpowers -AGENTS.md AGENTS.override.md .codex/ .codex diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000000..b210765d61 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,83 @@ +# AGENTS.md + +Guidance for every coding agent working in this repository. Coding rules live in the `bluequartz-skills` skills +below; this file holds what a skill cannot know about this checkout. + +## Required skills + +Every agent that writes, changes or reviews C++ here (the main session and each subagent) first loads these +`bluequartz-skills` skills and applies them. A subagent does not inherit them: it loads them itself, and every +dispatch prompt says so. + +- `bluequartz-skills:cpp-best-practices`: all C++. +- `bluequartz-skills:simplnx-best-practices`: simplnx and simplnx plugin code. + +Task skills own the details: `filter-implementation` for filters and algorithms, `implement-filter-tests` and +`exemplar-testing` for tests, `publish-exemplars` for test-data archives, `thread-safety` for parallel code, +`ooc-architecture` for out-of-core storage, `pipeline-files` for `.d3dpipeline` files, and `github-workflow` for +commits and pull requests. + +## Where new rules go + +When work teaches something a later agent or developer needs, record it in the same change, in exactly one place. +Search all of them first and update an existing rule instead of adding a second one. + +- `docs/Coding_Knowledge_Base.md`: a lesson from a real defect in simplnx or its plugins. State the rule, the evidence + (what broke and the pull request or issue that fixed it) and how to recognize it next time, using the entry format + in that file. +- `docs/Code_Style_Guide.md`: a convention of this project that well-written C++ could still break (names, layout, + project APIs, filter and test structure). No defect history is needed. +- This repository is public: an entry may name a private repository but contains no code from it. +- A `bluequartz-skills` skill: a general C++ rule that applies to every BlueQuartz repository + (`cpp-best-practices`), or a procedure (how to implement, test, review or publish). Skills point to the two + documents above; they do not copy their rules. Skill changes go to the `bluequartz-skills` plugin repository, not + to this one. +- This file: only facts about this checkout that no skill can know, such as layout and build and test commands. +- A rule that is true only on your own machine (tool paths, presets, build directories, personal habits): your personal + `~/.codex/AGENTS.md`, in a section for this project headed by its checkout path. Never in this file or in an + `AGENTS.override.md` here: the override replaces this file instead of adding to it. + +## Project + +simplnx is the C++20 data-processing library behind DREAM3D-NX. It has no UI dependencies. + +- `src/simplnx/`: the core library (DataStructure, filters, parameters, pipelines, utilities). +- `src/Plugins//`: in-tree filter plugins. Each holds its filters, `Filters/Algorithms/`, `docs/` (one Markdown + page per filter) and `test/`. +- `src/nxrunner/`: the command-line pipeline runner. `test/`: core library tests. `docs/`: developer documentation, + including `docs/Code_Style_Guide.md`. +- Out-of-tree filter plugins, such as SimplnxReview, live in their own repositories and join the build through + `SIMPLNX_PLUGIN_SEARCH_DIRS` and `SIMPLNX_EXTRA_PLUGINS`. These variables do not enable SimplnxOoc. When you need + out-of-core storage, follow the build instructions of the SimplnxOoc repository. +- Leave `scripts/`, `conda/` and build trees alone unless the task needs them. +- DREAM3D-NX builds this checkout as a subdirectory, so a change here can break the application build. + +## Build and test + +- Configure with a preset from `CMakePresets.json` (CI) or your `CMakeUserPresets.json`. Use the build directory whose + cache points at this source tree; a DREAM3D-NX build that includes this checkout counts. +- With `SIMPLNX_DOWNLOAD_TEST_FILES=ON`, build the `Fetch_Remote_Data_Files` target to download test data. The target + does not exist when that option is OFF. +- With a multi-config generator (Visual Studio), pass `--config ` to `cmake --build` and + `-C ` to ctest. +- After you add or remove an entry in a CMake source list, reconfigure the build directory. This can rerun vcpkg, so run + it in the foreground with a long timeout. Treat a vcpkg dependency failure as a problem of the environment until you + show otherwise. +- Never pipe a build or test command into `tail` or another command that hides its exit status. +- Run tests only through `ctest`, never the test executable: ctest extracts and removes the test data. Never run tests + in parallel; they share data files. Replace `` with the verified build directory and `` with the + test-name regular expression: + `ctest --test-dir "" -R "" -j 1 --output-on-failure --timeout 1500 --no-tests=error`. +- `SIMPLNX_TEST_ALGORITHM_PATH` picks which algorithm paths the tests run (0 both, 1 out-of-core only, 2 in-core only). + Validate in-core and out-of-core together with `bluequartz-skills:dual-build-protocol`. When out-of-core behavior + changes, run the whole `[ooc]` set, not one narrow tag. +- Format only the files you changed, with clang-format 19.1.1 from inside the repository so `.clang-format` applies + (`bluequartz-skills:clang-format` finds the binary). A whole-tree format belongs in its own commit. + +## Repository + +- Pull requests target `develop` on `BlueQuartzSoftware/simplnx`. +- This repository is public. Issues, pull requests, comments and reviews may name or link private repositories + (SimplnxOoc, DREAM3D-NX) but contain no code from them. +- A published test-data archive is immutable once merged code references it; publish changed data under a new + versioned name. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d23c790191..834b67f3b2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -115,7 +115,7 @@ Willing to Collaborate with Other Contributors on These Changes: Y/N ### Code Styleguide -For information about our specifc naming conventions and styling see our [coding style guide](/docs/Code_Style_Guide.md). +For information about our specifc naming conventions and styling see our [coding style guide](/docs/Code_Style_Guide.md). For lessons learned from past defects, see the [coding knowledge base](/docs/Coding_Knowledge_Base.md). For the basic code styling simplnx uses `clang-format`, the formatting file is included at the root level in the repository. diff --git a/docs/Code_Style_Guide.md b/docs/Code_Style_Guide.md index 8d929897c9..428ad9ad10 100644 --- a/docs/Code_Style_Guide.md +++ b/docs/Code_Style_Guide.md @@ -2,6 +2,9 @@ This is the style guide for the SIMPLNX library. When contributing sources to the simplnx repository we ask that all contributions follow this style guide. These general rules have been developed over the years in an effort to allow efficient coding practices. +This guide holds project conventions. A lesson learned from a specific defect, with its evidence, goes in the +[coding knowledge base](Coding_Knowledge_Base.md) instead. + ## Source Code Line Endings All source code line endings should be in "Unix" style "\n". Every text editor understands these line endings on every platform **except** the "Notepad" application on Windows. @@ -103,6 +106,271 @@ When using ANSI C one should use a char* for constant strings: Using this type of approach allows for quicker code updates when constant values need to be changed. +## Filters and Algorithms + +### Algorithms Are Callable Without Their Filter + +An algorithm class (the class a filter's `executeImpl()` constructs and runs) must be safe to call directly, without its filter. Other code may construct and run an algorithm without going through the filter's parameter validation and `preflightImpl()`. Examples include another algorithm, a test, or a future API. + +The algorithm therefore checks every input property it relies on, and returns an error `Result` when one does not hold. This applies even when the filter's parameters or preflight already enforce the same rule. Check at least: + +- component counts and component shapes +- tuple counts that must match between arrays +- data types +- index ranges that come from data, such as feature ids or phases used to index another array + +Filter parameters and preflight give the user early feedback. They do not replace the checks in the algorithm. Do not remove a check from an algorithm because the filter makes it unreachable. Test each of these checks by calling the algorithm directly. + +When an algorithm dispatches to more than one implementation (for example an in-memory branch and an out-of-core branch), each branch makes these checks before its first read. A check that is present in one branch only is not enough. + +Write the algorithm's preconditions as properties of its inputs (`@pre Epsilon and MinPoints are positive.`). Do not write them as checks that a filter made (`@pre Filter preflight has validated Epsilon.`), because a direct caller does not run that filter. + + // In the algorithm, before any fixed-size buffer is indexed by component + if(numComponents != 2 && numComponents != 3) + { + return MakeErrorResult(-54060, fmt::format("Input array has {} components but only 2 or 3 are accepted.", numComponents)); + } + +### Put Each Check at the Earliest Point That Can See Its Data + +A filter checks its inputs at three levels. Put each check at the earliest level that has the information it needs: + +1. **Parameter.** Structure that does not depend on other inputs: the allowed `DataType`s and component shapes (`AllowedTypes`, `AllowedComponentShapes`). Parameter validation runs before `preflightImpl()`. +2. **`preflightImpl()`.** Parameter values, relationships between inputs (tuple counts, geometry element counts), geometry and file metadata. Preflight cannot see array values. +3. **Execute.** Checks that need array values, such as feature ids or phases used as indices. + +Do not repeat a parameter or preflight check inside the filter's `executeImpl()`. The algorithm class still keeps its own checks (see above). + +### Declare the Component Shape the Algorithm Reads + +If an algorithm assumes a component count for an input `DataArray`, the parameter that selects that array declares it with `AllowedComponentShapes`. Without it, a 3-component array with the correct number of tuples passes preflight and the algorithm reads it with the wrong stride. + + params.insert(std::make_unique(k_FeatureIdsPath_Key, "Cell Feature Ids", "...", DataPath{}, + ArraySelectionParameter::AllowedTypes{DataType::int32}, ArraySelectionParameter::AllowedComponentShapes{{1}})); + +The match is exact: `AllowedComponentShapes{{1}}` rejects a `{1, 1}` array, and `AllowedComponentShapes{{3}}` rejects a `{1, 3}` array. When a change adds a component-shape or type constraint to an existing parameter, its pull request lists the shapes and types that saved pipelines can no longer use. + +When the code means "number of tuples", call `getNumberOfTuples()`. `getSize()` returns tuples times components. + +### Every Accepted Type and Value Must Work in Execute + +Every `DataType` that a parameter allows must be handled by the execute code, and every value that preflight accepts must be usable. Do not allow `GetAllDataTypes()` when the algorithm cannot handle `bool`; do not allow `int8` when the algorithm stores face indices in the label type. Either narrow the parameter or support the type. + +### Check Tuple Counts in Preflight + +Tuple-count rules for input arrays: + +- **An element array that the filter only reads** must have one tuple per geometry element (cells, faces, edges or vertices). It may live in any Attribute Matrix: selecting an array with the right tuple count from a different Attribute Matrix is a feature. For Image and RectGrid cells use `ValidateCellTupleCount()`. For node geometries compare `getNumberOfTuples()` with `getNumberOfFaces()`, `getNumberOfEdges()` or `getNumberOfVertices()`. +- **Sibling inputs** that the algorithm reads with the same index must have the same tuple count. Use `CheckArraysHaveSameTupleCount()`. Include an optional input (for example a mask) only when its option is on. +- **Outputs sized from an Attribute Matrix**: every input that sets the loop count must have the same tuple count as that Attribute Matrix. +- **A spatial filter that modifies Feature Ids, its sibling cell arrays, or the Feature Attribute Matrix** requires Feature Ids to be in the geometry's Cell Attribute Matrix. Use `ValidateArrayInGeometryCellData()`, `ValidateArraysInGeometryCellData()` and `ValidateFeatureAttributeMatrixInGeometry()`. + +Every message names both paths and both counts: + + "The selected DataArray 'Face Labels' has 12 tuples, but the Triangle Geometry 'Mesh' has 24 faces." + +### Range-Check Every Index That Comes From Data + +A feature id, phase, face label or `NeighborList` entry used as an index into another array must be checked before the access. Preflight cannot see these values, so the check runs at the start of execute. For an unmasked 1-component `Int32Array`, call the overload of `ValidateFeatureIdsToFeatureAttributeMatrixIndexing()` that takes `shouldCancel`. That function checks every tuple and has no mask argument. For other inputs, scan the minimum and maximum and return an error. A filter with a mask checks only the ids that execute uses, with the same mask and sentinel rules as execute. A negative value is out of range unless the code skips negative values before it indexes. + +An index that comes from a parameter value (for example **Phase Index**) is checked in preflight, because preflight can see it. + +### Validate Every Parameter Value in Preflight + +Most numeric parameters have no built-in range. `preflightImpl()` rejects every value the algorithm cannot use, before any cast. The documented domain of the algorithm sets the bounds of each parameter: + +- spacings, steps, scales and distances: finite. Require a value greater than 0 only where 0 is not valid. +- counts, dimensions and sizes: a whole number. Require 1 or more only where 0 is not valid. A supported no-op value, such as 0 iterations, stays valid. +- indices: `0 <= index < size` +- related values: legal combinations of options. Accept equal bounds (`min <= max`) when the operation defines them; require `min < max` only when the algorithm needs strict order. +- values that will be cast to an array element type: inside the range of that type + +Write each comparison so that NaN fails it: + + const float64 dim = rowData[idx]; + const float64 upperExclusive = std::ldexp(1.0, std::numeric_limits::digits); + if(!std::isfinite(dim) || !(dim >= 1.0) || !(dim < upperExclusive) || std::floor(dim) != dim) + { + return MakePreflightErrorResult(k_InvalidTupleDim, fmt::format("Tuple dimension at index {} must be a whole number in [1, {}), but was {}", idx, upperExclusive, dim)); + } + tupleDims.push_back(static_cast(dim)); + +This example requires `` and ``. + +Validation follows the parameter linking: validate a value only when the algorithm uses it, and always validate a value that the algorithm always reads. Every parameter reaches the algorithm's input values and changes its behavior, or it is removed. When a filter stores the random seed it used, the algorithm uses that same seed, so the stored seed reproduces the output. + +A direction or axis parameter must not default to `{0, 0, 0}`, and preflight rejects a zero-length or non-finite vector. + +### Preflight Must Not Throw + +`IFilter::preflight()` does not convert exceptions into errors, so an exception in `preflightImpl()` can end the application. In preflight: + +- Use `getDataRefAs()` only when a selection parameter or an earlier check guarantees that the object exists with that type. The numeric-type list of an `ArraySelectionParameter` does not establish a numeric array kind. `MultiArraySelectionParameter::AllowedTypes` can establish that kind, but its `AllowedDataTypes` list does not establish the element type. See entry 4 of the [coding knowledge base](Coding_Knowledge_Base.md). Do not repeat a check that parameter validation already does. Otherwise use `getDataAs()` and check for `nullptr`. `getDataRefAs()` throws `std::out_of_range` when the path does not resolve and `std::bad_cast` when the object is a different type (for example a `float64` array where the code assumes `float32`, or a `StringArray` inside an Attribute Matrix that the code reads as `IDataArray`). +- Use `getCellData()`, `getVertexAttributeMatrix()`, `getFaceAttributeMatrix()` and similar accessors and check for `nullptr`. A geometry can have lost its Attribute Matrix. +- Use non-throwing library calls (`std::filesystem::directory_iterator(path, errorCode)`, `nlohmann::json::parse(text, nullptr, false)`), or catch the exception and return an error `Result`. +- Check `Result::invalid()` and `std::optional::has_value()` before calling `.value()`. +- When code walks every child of an Attribute Matrix, handle each array kind (`IDataArray`, `StringArray`, `INeighborList`) or return an error that names the array. + +### Preflight and Execute Make the Same Decisions + +Preflight and execute must reach the same decision from the same inputs: which arrays are processed, output names, crop or resample indices, which outputs are enabled. Put that decision in one function and call it from both places. Do not copy the logic. + +- Check every mode, every geometry type and every item of a list (every scan, every file, every input geometry), not only the first one. +- Execute writes every output that preflight creates, with the tuple and component shape that preflight gave it. +- Preflight declares every object that execute deletes or changes in place. +- Preflight does not leave temporary objects (for example a root-level temporary mask) in the `DataStructure`. + +### In-Core and Out-of-Core Algorithms Give the Same Result + +A filter that dispatches between an in-core (Direct) algorithm and an out-of-core (Scanline) algorithm has two copies of its logic. Both must give the same output for the same input. A fix, a guard or an initial value in one path goes into the other path too, preferably through a shared helper. Test both paths (see Testing). + +### Build DataPaths From DataPaths + +Build each `DataPath` from the path that the user selected or that preflight created. Do not build a path from a name string or with string operations. + +- New child: `parentPath.createChildPath(name)` +- Sibling in the same parent: `path.replaceName(newName)` +- Same relative path under a different geometry: `childPath.rebase(oldGeometryPath, newGeometryPath)` +- A copied or moved subtree: rebase each descendant from the selected source to its final destination with `DataPath::rebase(oldPrefix, newPrefix)`, as `CopyDataObjectFilter` does. The destination can have a different path depth. + +`DataPath({geometryName, "Cell Data", name})` is wrong when the geometry is inside a `DataGroup`. `StringUtilities::replace(path.toString(), oldName, newName)` changes every occurrence of the name: `Grain/Grain Data` becomes `New/New Data`. A parent or child whose name contains the same text is renamed too; copy and geometry operations that did this failed with error -5102 (pull request #1765). + +### Never Drop a Result + +Return every `Result` or merge it into the `Result` that the function returns. A later warning or error must not replace an earlier one. + + if(suppressedFaceCount == 0) + { + return MergeResults(std::move(windingResult), MakeNoFacesPrunedWarning(geometryPath)); + } + +- An error from a sub-call keeps its own code and message. Add context to it; do not replace it with a generic code. +- Every `MakePreflightErrorResult(...)` is returned. +- A file writer checks the stream state (`fail()` or `bad()`) after the last write and after `flush()`. On failure it returns an error before `AtomicFile::commit()`, so a truncated file is not committed. +- A parallel task that can fail stores its first failed `Result`, and the caller returns it after the tasks finish. +- In execute, a loop over arrays, objects, tuples or chunks returns the first invalid `Result`, with the warnings collected so far. It does not collect one error for each item: when one cause (a full disk, a missing file) fails every item, hundreds of errors hide that cause. This applies also to a checked flush of a `DataStructure`. Preflight can merge the errors of independent checks. Execute can also merge a primary failure with the failure of the cleanup that follows it. +- When several independent calls add errors to one list, collect them in a local `std::vector` and return `{nonstd::make_unexpected(std::move(errors))}` only when the list is not empty. Do not pass `result.errors()` of a valid `Result` as that list: a valid `Result` holds no error vector, so the call asserts in a Debug build and is undefined behavior in Release. + +### Give Empty Features a Defined Output + +A feature with no cells or faces, and the reserved feature 0, get a defined value in every per-feature output. Do not compute 0/0 for them, and do not leave the value of an earlier feature in place. + +### Write Floating-Point Output Through a Saturating Cast + +A filter whose output type can be an integer type (for example an output type that follows the input type) writes each computed floating-point value through `ImageProcessing::detail::SaturateCastFromDouble()`. For an integral `U`, it maps NaN to 0 and clamps to the range of `U`. For a floating-point `U`, it does an ordinary cast without saturation. A plain `static_cast` of NaN or an out-of-range value to an integer type is undefined behavior. + +### Check Every Final Output Path in Preflight + +For outputs that must stay distinct from their inputs, preflight resolves each output to its final file path or `DataPath`, after prefixes and suffixes, and rejects: + +- two outputs that resolve to the same path (ignore case for file names) +- an output that is equal to an input +- a destination that is equal to or inside its source +- a selection that contains a path twice, or a path and its ancestor + +An intentional in-place update, or a deferred replacement of an input, follows the documented contract of the filter and keeps the input until the replacement is ready. For example, with **Perform in Place** on, **Convert AttributeArray DataType** writes a temporary array, then deletes the input and renames the temporary array to the input path in deferred actions. + +## Concurrency + +### Static Header Caches Need a Mutex + +`preflightImpl()` and `executeImpl()` are `const` and cannot directly modify non-`mutable` data members. By project convention, data that preflight reads (for example a file header) and execute reuses lives in a file-level static cache, keyed by an instance id. `IFilter::execute()` runs preflight on the same instance just before `executeImpl()`, so a successful preflight must leave every entry that execute needs ready for use. `ReadNIfTIFileFilter` shows the cache and its mutex; it uses the cache only in preflight. That cache is shared by every instance of the filter, and the GUI preflights filters on one thread while a pipeline executes on another. Lock one mutex around every access: in the constructor, the destructor, `preflightImpl()` and `executeImpl()`. + + std::mutex s_HeaderCacheMutex; + + MyReaderFilter::MyReaderFilter() + : m_InstanceId(s_InstanceId.fetch_add(1)) + { + std::lock_guard lock(s_HeaderCacheMutex); + s_HeaderCache[m_InstanceId] = {}; + } + +### Out-of-Core Stores Are Written Serially + +Out-of-core stores do not support concurrent writes. Threads that write to different tuples, or even to different chunks, of the same store can still race. + +- Parallelize the computation on in-memory buffers, join the workers, then write the results to the store from one thread. +- When parallel writes matter for in-core speed, write separate in-core (parallel) and out-of-core (serial) algorithms and select one with the algorithm dispatch (see `docs/AlgorithmDispatch.md`). +- A `ParallelDataAlgorithm` or `ParallelTaskAlgorithm` whose workers access a store passes every array the workers read or write to `requireArraysInMemory()` (or `requireStoresInMemory()`). This does not load the data into memory: it runs the algorithm serially when any listed array is out-of-core. Each call replaces the previous decision, so call it once with all arrays, and call `setParallelizationEnabled(false)` after it, not before. + +### Keep Optional Working-Memory Reservations Short + +`CacheMemoryBudgetManager::reserveWorkingMemory()` grants memory from one working allowance that every caller shares (a quarter of the cache memory budget). The grant can be smaller than the request, and it can be 0. + +- Use the size that was granted, not the size that was requested. The algorithm must also work with a smaller grant. +- Hold an optional reservation (extra slices for batching, cached coordinates) only while the algorithm uses it, so that it does not block required work of the storage backend. +- A `std::unique_ptr>` owner makes two allocations: the vector object and its elements. An exact charge includes `sizeof(std::vector)` for each owner. +- Test with a budget that the request fills exactly. + +The SimplnxOoc repository documents the ordering rules of its out-of-core stores. + +### Long Loops Check for Cancel + +Every loop that can run for a long time checks `shouldCancel` (or `getCancel()`), and has a progress guarantee: it counts real changes, caps the number of iterations, or applies a timeout. + +Check `shouldCancel` also between the steps that come before the loop. After a step that can take a long time (a write, a read of a complete array), check it again before the next reservation, allocation or complete read. + +## Data Stores + +### Check Shape Products Before You Allocate + +The product of a tuple shape or a component shape can be too large for `usize`. For `{2^63, 2}` an unchecked `std::accumulate` wraps the count to 0, and the store then has a buffer that does not match its shape. Every store that computes a count from a shape (`DataStore`, `EmptyDataStore`, `StringStore`, `ListStore`): + +- looks for a zero dimension first. A shape with a zero dimension has a count of 0 whatever its other dimensions are, so `{SIZE_MAX, 2, 0}` and `{0, SIZE_MAX, 2}` give the same result. `EmptyDataStore::CheckedPlaceholderProduct()` shows this order. +- checks each multiplication for overflow, then the element count (tuples times components) and the byte count (elements times `sizeof(T)`). +- does all checks before it allocates memory or changes a member. + +`resizeTuples()` returns an error for an overflow and keeps the store unchanged. `DataStore`, `StringStore` and `ListStore` return error -6035 for a resize failure. `EmptyDataStore` returns `EmptyDataStore::k_ResizeError` (-10603). A constructor throws, because it cannot return a `Result`. Each overflow message names the count that does not fit ("tuple count", "element count" or "byte count"), so that a test can tell the checks apart. Issues #1883 and #1884 track the stores that do not follow this rule yet. + +## Geometry Conventions + +- The Image Geometry origin is the minimum corner of cell 0. The center of cell `i` is `origin + (i + 0.5) * spacing`. Use `ImageGeom::getCoordsf()` or `getCoords()` for cell centers instead of a local formula. +- A reader that creates a geometry sets its length unit from the file or from the **Length Unit** parameter. +- Do not `static_cast` a choice index to an enum. The order of the choice list does not have to match the enum values (`IGeometry::LengthUnit::Unspecified` is 100, but it is choice index 27). Map the index through a lookup. + +## Error Codes and Messages + +### Error Codes Are Unique + +- Each condition gets its own error code; each warning condition gets its own warning code. Do not copy the code from the check above. +- Filter error codes are more negative than -10000. Each filter uses one range of 100 codes: if the first code is -47500, all codes of that filter are in -47500..-47599. +- Shared utilities use codes that no filter uses. +- Declare each code as a named constant (`constexpr int32 k_ZDimMismatch = -47512;`) and use the same constant in preflight and execute when they report the same condition. +- Before you use a new code, check that it is unique in the repository and in open pull requests: + + git grep -nE -- "-47512([^0-9]|$)" + + Do not rely on `\b`; it is not portable extended-regex syntax. + +### Messages Name the Right Values + +- Each message names the correct values in the correct order, and fires only on the condition it describes. Advice in a message must be something the user can do. +- A per-cell loop tests the mask first. Cells that the mask excludes are not validated, counted or grown. + +## Testing + +- **Expected values come from an independent oracle**: a hand calculation, a closed-form result, a reference implementation or verified DREAM3D 6.5.171 output. Never regenerate an exemplar only to absorb a change in output; show with the oracle that the new value is correct. +- **Generated and expected data come from different `DataStructure`s.** A comparison of an array with itself cannot fail. +- **Test every mode.** A test that only runs a mode is not enough; it compares the output values of that mode with expected values. +- **Test both algorithm paths.** For a filter with in-core and out-of-core algorithms, run each test for both with `GENERATE(from_range(UnitTest::SelectAlgorithmTestScenariosForInMemoryStores()))` and `UnitTest::AlgorithmTestScope`. +- **Test every axis and plane** for code that has one block per axis or plane. +- **Test error paths by code.** Assert the exact error code, not only `REQUIRE_NOTHROW` or an invalid result. +- **Test degenerate and invalid inputs**: empty geometries, out-of-range feature ids, NaN and limit values (`2^63`, `2^64`, `-1` and `256` for `uint8`), inputs in a `DataGroup` instead of at the root, and arrays from a different Attribute Matrix. +- **Tell apart conditions that share an error code.** Assert the code of the store that owns the condition: -6035 for a `DataStore`, `StringStore` or `ListStore` resize failure, and -10603 (`EmptyDataStore::k_ResizeError`) for an `EmptyDataStore` resize failure. When a contract gives several conditions the same code, the test also asserts a phrase that only the message of its condition contains. +- **Print the errors of a failed Result.** Use `SIMPLNX_RESULT_REQUIRE_VALID(result)` instead of `REQUIRE(result.valid())`. It prints each error code and message when the check fails. +- **Apply an action in both modes.** An `IDataAction` creates different objects in preflight and in execute. Test it with `GENERATE(IDataAction::Mode::Preflight, IDataAction::Mode::Execute)` and an `INFO` that names the mode. +- **Test a size failure without the allocator.** The nightly `asan` job runs with the default AddressSanitizer options. A request above the AddressSanitizer limit (1 TiB on 64-bit platforms) stops the test process instead of throwing `std::bad_alloc` (#1885). Use a size that the code rejects before it allocates, such as a shape product that overflows. Test a recoverable allocation failure through a deterministic failure seam. Test an intentional allocator abort in a separate process, with a timeout and a check of its exit result. An AddressSanitizer build alone does not make a real allocation failure recoverable. +- **No network access in CTest.** A test that needs a network service fails when the service is slow or not available. Do not register it with CTest. Keep the script so that it can be run by hand. +- **Call `UnitTest::LoadPlugins();` only when the test uses plugin code** that the `Application` must load, for example a test that gets filters from the filter list or runs a pipeline that does. A test that constructs a filter or algorithm class directly and builds its data in memory does not call it. +- **Test an algorithm's own checks by calling the algorithm directly**, once for each dispatch branch. To reach an out-of-core branch in the in-core build, use a store whose `getStoreType()` returns `IDataStore::StoreType::OutOfCore` (the ImageProcessing tests do this). When that branch needs external sort or temporary record stores, register a test `IDataIOManager` that provides them, as described in the next item. +- **Restore the global I/O state.** A test that registers an `IDataIOManager` changes the `DataIOCollection` for the rest of the process. `DataIOCollection` has no remove call, so replace an existing manager under its format name and re-add the saved original in a destructor, which also runs when an assertion fails (see `ScopedImportSelectionManager` in `test/Dream3dLoadingApiTest.cpp`). Do not rely on ctest running each `TEST_CASE` in its own process: the test executable run directly runs many cases in one process, and a leaked manager makes later tests fail or skip their checks. + +## Build Configuration + +### The Python Interpreter Comes From the CMake Preset + +Set the interpreter paths in the selected project or user preset. With `SIMPLNX_BUILD_PYTHON=ON`, `AddPythonTest()` gives `PYTHON_EXECUTABLE` to its test driver. Otherwise, it runs `Python3_EXECUTABLE` directly. Set the variable that the selected path uses before tests are registered. Keep machine-specific paths in `CMakeUserPresets.json`. + ## Clang Formatting `clang-format` is a tool to automatically format C++ code. You should run 'clang-format' on your code before opening a Pull Request to make sure it passes the *clang-format pr* check. You can install `clang-format` and `git-clang-format` with `npm install -g clang-format`. To automatically format a file according to a project's C++ code style, run `clang-format -i path/to/simplnx/file`, which is supported on macOS/Linux/Windows. If you want to run `clang-format` on all the changed code on your latest git commit (HEAD), you can run `git-clang-format HEAD~1`. Run `git-clang-format -h` for extra help. diff --git a/docs/Coding_Knowledge_Base.md b/docs/Coding_Knowledge_Base.md new file mode 100644 index 0000000000..35ab2c6f26 --- /dev/null +++ b/docs/Coding_Knowledge_Base.md @@ -0,0 +1,545 @@ +# Coding Knowledge Base + +This file collects lessons learned from real defects in simplnx and its plugins. Each entry records a mistake that was +made, why it happened, how it was found, and the rule that prevents it. The audience is developers and the coding +agents that work in this repository. Read the rules before you write or review code in the area they cover. + +Add a new entry when a defect taught something that the code itself does not make obvious. Keep the entries short. +State the rule first, then the evidence. A project convention that does not come from a defect belongs in +[the code style guide](Code_Style_Guide.md) instead. + +## Index + +1. [Record an object in the writer ID map only when the written DataStructure owns it][kb-1] +2. [Check the HDF5 file close before you report a successful save][kb-2] +3. [Copy every field that sets behavior when you clone an object][kb-3] +4. [Do not treat the type list of a selection parameter as proof of a numeric DataArray][kb-4] +5. [Return null from a geometry pointer getter when the geometry is detached][kb-5] +6. [Do not mix a circular mean with an unwrapped arithmetic centroid][kb-6] +7. [Use the same population in every pass of a statistic](#7-use-the-same-population-in-every-pass-of-a-statistic) +8. [Size storage for sparse IDs from the largest ID](#8-size-storage-for-sparse-ids-from-the-largest-id) +9. [Do not reject a generated mesh in preflight for its placeholder size][kb-9] +10. [Open an HDF5 child by its target name below the parent group][kb-10] +11. [Apply a geometry transformation once when a filter runs a subfilter][kb-11] +12. [Give all ghost cells one label](#12-give-all-ghost-cells-one-label) +13. [Compute a shared intersection point from a canonical endpoint order][kb-13] +14. [Key a clipped vertex by its source vertex and its clip point][kb-14] +15. [Convert an optional legacy SIMPL key only when it is present][kb-15] +16. [Check the binary STL file size before you use attribute byte counts][kb-16] +17. [Give a geometry without physical coordinates an unspecified length unit][kb-17] +18. [Apply the hexagonal axis correction to both hexagonal Laue classes][kb-18] +19. [Use the search grid only to find neighbor candidates](#19-use-the-search-grid-only-to-find-neighbor-candidates) + +[kb-1]: #1-record-an-object-in-the-writer-id-map-only-when-the-written-datastructure-owns-it +[kb-2]: #2-check-the-hdf5-file-close-before-you-report-a-successful-save +[kb-3]: #3-copy-every-field-that-sets-behavior-when-you-clone-an-object +[kb-4]: #4-do-not-treat-the-type-list-of-a-selection-parameter-as-proof-of-a-numeric-dataarray +[kb-5]: #5-return-null-from-a-geometry-pointer-getter-when-the-geometry-is-detached +[kb-6]: #6-do-not-mix-a-circular-mean-with-an-unwrapped-arithmetic-centroid +[kb-9]: #9-do-not-reject-a-generated-mesh-in-preflight-for-its-placeholder-size +[kb-10]: #10-open-an-hdf5-child-by-its-target-name-below-the-parent-group +[kb-11]: #11-apply-a-geometry-transformation-once-when-a-filter-runs-a-subfilter +[kb-13]: #13-compute-a-shared-intersection-point-from-a-canonical-endpoint-order +[kb-14]: #14-key-a-clipped-vertex-by-its-source-vertex-and-its-clip-point +[kb-15]: #15-convert-an-optional-legacy-simpl-key-only-when-it-is-present +[kb-16]: #16-check-the-binary-stl-file-size-before-you-use-attribute-byte-counts +[kb-17]: #17-give-a-geometry-without-physical-coordinates-an-unspecified-length-unit +[kb-18]: #18-apply-the-hexagonal-axis-correction-to-both-hexagonal-laue-classes + +--- + +## 1. Record an object in the writer ID map only when the written DataStructure owns it + +**Rule:** `DataStructureWriter` maps each written object ID to its HDF5 path, so that a second location of the same +object becomes a link. Record an object in that map only when it belongs to the `DataStructure` that is being written. +Return an error when an ID is already recorded at a different path; do not overwrite the entry. + +**Pattern to use:** `DataStructureWriter::addWriter()` in open pull request #1859 ignores objects outside +`m_WrittenStructure`, obtains the object ID and writer path, and inserts them with `m_IdMap.try_emplace()`. If the ID +already maps to a different path, it returns error -6055 instead of replacing that mapping. + +**Pattern to avoid:** send the objects of a temporary `DataStructure` through the writer of the main structure, and +record them by their numeric ID. + +**Why:** an ID is unique only inside one `DataStructure`. Every `DataStructure` starts its IDs at 1, so a temporary +structure and the written structure can both hold an object with the same ID. + +**How it showed up:** the NeighborList writer sends a temporary counts array through the main writer. That array +replaced the map entry of an unrelated object that had two locations, and the second location became a link to the +counts dataset. Issue #1831. Fixed by open PR #1859, not merged yet. + +**How to recognize it next time:** + +- A serializer creates a temporary `DataStructure` and writes its objects through the writer of the main structure. +- An ID map that is filled with `operator[]` or `insert_or_assign()` instead of a checked insert. + +--- + +## 2. Check the HDF5 file close before you report a successful save + +**Rule:** a writer that owns an HDF5 file closes it explicitly and checks the result before it returns success. A close +failure is returned before `AtomicFile::commit()` replaces the old file. + +**Pattern to use:** `FileIO::closeChecked()` in open pull request #1859. For example, the path-based write of +`DataStructureWriter` ends with `return MergeResults(std::move(writeResult), fileWriter.closeChecked());`. + +**Pattern to avoid:** return success while the `FileIO` object is still in scope, so that its destructor closes the +file and discards the result of `H5Fclose()`. + +**Why:** HDF5 can write buffered metadata when it closes the file. Successful dataset writes do not prove that the file +is complete. + +**How it showed up:** an injected close failure still gave a valid `Result`, so **Write DREAM3D-NX File** could replace +the old output file with an incomplete one. Issue #1832. Fixed by open PR #1859, not merged yet. + +**How to recognize it next time:** + +- A path-based writer returns while its file object is still in scope, and an atomic-file commit follows. + +--- + +## 3. Copy every field that sets behavior when you clone an object + +**Rule:** the `clone()` of an action or a parameter keeps all of its configuration. When a constructor gets a new +argument, update `clone()` in the same change, and test the clone with a value that is not the default. + +**Pattern to use:** `CreateImageGeometryAction::clone()` passes the length unit: + +```cpp +return std::make_unique(getCreatedPath(), m_Dims, m_Origin, m_Spacing, m_CellDataName, + m_Units); +``` + +Test a clone by applying the original and the clone to separate `DataStructure` objects and comparing the results. + +**Pattern to avoid:** a constructor call in `clone()` that leaves out an argument, so the clone silently takes the +default value. + +**Why:** the framework copies actions and parameters through `clone()` (see `AnyCloneable`) without a visible call at +the filter. A field that the copy loses changes the output with no error. + +**How it showed up:** PR #1838 added length units to geometry creation, but a cloned image-geometry action reverted to +Micrometer. The review of that pull request asked for a direct clone test. + +**How to recognize it next time:** + +- A constructor gets a new argument and `clone()` is not part of the change. + +--- + +## 4. Do not treat the type list of a selection parameter as proof of a numeric DataArray + +**Rule:** distinguish array kind from numeric element type. `ArraySelectionParameter` accepts any `IArray`, but applies +its numeric-type and component-shape restrictions only to `IDataArray`. `MultiArraySelectionParameter` checks +`AllowedTypes` and required component shapes, but does not check `AllowedDataTypes`. Use `getDataRefAs()` only when +parameter validation or an earlier check establishes the required type. Otherwise use `getDataAs()` and reject +`nullptr`. + +**Pattern to use:** `ComputeSurfaceAreaToVolumeFilter::preflightImpl()` gets its inputs with +`getDataAs()` and checks each pointer. + +**Pattern to avoid:** rely on the numeric-type list of `ArraySelectionParameter` to establish `IDataArray`, or on +`MultiArraySelectionParameter::AllowedDataTypes` to establish a concrete element type. + +**Why:** `getDataRefAs()` throws `std::bad_cast` for an object of a different type, and `IFilter::preflight()` does +not convert exceptions into errors. + +**How it showed up:** selections that the parameters accepted caused `std::bad_cast` in preflight, or reached a Boolean +type that the algorithm dispatch did not support. Issues #1821 and #1833. PRs #1817 and #1763 fixed the affected +filters. Open PR #1766 tightens the parameter declarations and preflight checks of individual filters; it does not fix +the shared parameter-validation gaps and is not merged yet. + +**How to recognize it next time:** + +- A typed reference access in preflight that relies only on the type list of a selection parameter. + +--- + +## 5. Return null from a geometry pointer getter when the geometry is detached + +**Rule:** a geometry getter that returns a pointer checks the geometry's `DataStructure` pointer before it resolves an +associated object, and returns `nullptr` when the geometry is detached. + +**Pattern to use:** `IGridGeometry::getCellData()`: + +```cpp +DataStructure* dataStructurePtr = getDataStructure(); +if(nullptr == dataStructurePtr) +{ + return nullptr; +} +return dataStructurePtr->getDataAs(m_CellDataId); +``` + +**Pattern to avoid:** dereference `getDataStructure()`, or use a throwing reference accessor, inside a pointer getter. + +**Why:** an external `shared_ptr` can keep a geometry alive after its `DataStructure` is destroyed. The destructor of +the `DataStructure` detaches the objects that survive it. + +**How it showed up:** a geometry getter that ran during application shutdown threw and ended the application. PR #1809. + +**How to recognize it next time:** + +- A geometry that something outside the `DataStructure` owns while the structure is cleared or destroyed. + +--- + +## 6. Do not mix a circular mean with an unwrapped arithmetic centroid + +**Rule:** state which periodic centroid the code computes, and make the code and the filter documentation agree. For an +unwrapped arithmetic centroid, choose the seam (the largest gap), unwrap the positions, keep their weights, average +them, then wrap the result back into the domain. + +**Pattern to use:** the file-local `PeriodicMean1D()` in `src/simplnx/Utilities/GeometryHelpers.cpp` unwraps vertex +positions at the largest gap. + +**Pattern to avoid:** add half of the domain to an ordinary mean, or compute `atan2(sumSin, sumCos)` and document the +result as an arithmetic average after unwrapping. + +**Why:** the circular mean and the unwrapped arithmetic mean are different for an asymmetric population. + +**How it showed up:** PR #1670 (issue #1665) removed constant-offset corrections from the triangle and feature centroid +filters. The arithmetic centroid for cells in `ComputeFeatureCentroids` is fixed by open PR #1764, not merged yet. + +**How to recognize it next time:** + +- Periodic centroid code with a constant offset or `atan2(sumSin, sumCos)`, while the documentation promises an + arithmetic average. + +--- + +## 7. Use the same population in every pass of a statistic + +**Rule:** counting, gathering, summing and variance passes use the same eligibility test. Reset the denominator of each +group before the code processes that group. + +**Pattern to use:** the counting and gather passes in `ComputeAvgOrientations` both require a positive phase smaller +than `numEnsembles`. The gather pass also requires the voxel's feature ID to match the feature being averaged. + +**Pattern to avoid:** exclude phase-0 cells from the count but include their quaternion, or a placeholder value, in +another pass. + +**Why:** the numerator, the denominator and the deviations must describe the same population. + +**How it showed up:** PR #1664 (issue #1659) stopped phase-0 quaternions from entering the vMF and Watson averages in +**Compute Feature Average Orientations**. PR #1631 made the neighbor-average denominators of +`ComputeFeatureNeighborMisorientations` and `ComputeFeatureNeighborCAxisMisalignments` local to each feature. + +**How to recognize it next time:** + +- A statistic with several passes whose `if` conditions are different. + +--- + +## 8. Size storage for sparse IDs from the largest ID + +**Rule:** when external IDs can have gaps, size the storage for the largest ID, or map the IDs explicitly to dense +indices. Do not size it from the number of distinct IDs. + +**Pattern to use:** `ReadAngDataFilter::preflightImpl()` sizes the ensemble Attribute Matrix from the largest phase +index: + +```cpp +// Size ensemble arrays from the largest phase index because phase indices can have gaps. +tupleDims = {std::max(maxPhaseIndex, angPhases.size()) + 1}; +``` + +`SilhouetteDirect` maps sparse cluster IDs to dense indices with a `std::unordered_map`. + +**Pattern to avoid:** allocate `numberOfDistinctIds + 1` entries and index them with the unchanged external IDs. + +**Why:** two declared phases can have the IDs 1 and 5. + +**How it showed up:** PR #1657 fixed out-of-bounds ensemble writes for ANG files with sparse phase IDs. PR #1765 fixed +sparse cluster IDs in **Silhouette**. + +**How to recognize it next time:** + +- The allocation uses the `size()` of a container, but the writes use IDs read from its elements. + +--- + +## 9. Do not reject a generated mesh in preflight for its placeholder size + +**Rule:** when an upstream filter computes the size of a mesh from the data, accept the placeholder size in preflight. +Check for an empty mesh in execute. + +**Pattern to use:** `SliceTriangleGeometryFilter` accepts the placeholder in preflight. In `executeImpl()` it adds +warning -62105 when the Triangle Geometry has no triangles. + +**Pattern to avoid:** a preflight check that rejects every geometry with zero faces. + +**Why:** mesh-producing filters such as `QuickSurfaceMeshFilter` create their geometry in preflight with zero faces, +because they compute the real mesh in execute. + +**How it showed up:** the review of PR #1763 found that a new empty-mesh check broke pipelines that use **Create +Surface Mesh (QuickMesh)**, **Create Surface Mesh (Surface Nets)** and M3C before they ran. + +**How to recognize it next time:** + +- A new preflight check that rejects a count that an upstream action sets to 0 on purpose. + +--- + +## 10. Open an HDF5 child by its target name below the parent group + +**Rule:** when a reader receives the parent group of an object, open the object by its target name. + +**Pattern to use:** `INodeGeom0dIO` and `RectGridGeomIO`: + +```cpp +auto groupReader = dataStructureGroup.openGroup(dataPath.getTargetName()); +``` + +**Pattern to avoid:** open the complete `DataPath` below a group that already is the parent of the object. + +**Why:** for an object at the root, the two forms give the same path, so the mistake stays hidden. For a nested object, +the path repeats the parent and does not resolve. + +**How it showed up:** importing a geometry inside a `DataGroup` crashed. PR #1676. + +**How to recognize it next time:** + +- A helper that takes both a parent HDF5 group and a complete `DataPath`. + +--- + +## 11. Apply a geometry transformation once when a filter runs a subfilter + +**Rule:** before a filter runs a subfilter, take into account the geometry changes that its own preflight actions have +already applied. + +**Pattern to use:** `ReadH5Ebsd` sets `keep_input_geometry_origin` when it runs **Rotate Sample Reference Frame**, +because preflight already assigned the transformed origin through `UpdateImageGeomAction`. + +**Pattern to avoid:** transform the origin in an outer action, then let the subfilter transform that origin again. + +**Why:** `IFilter::execute()` applies the preflight actions before it calls `executeImpl()`, so the subfilter sees their +result. + +**How it showed up:** the automatic sample rotation of **Read H5EBSD File** rotated the origin twice, and the equivalent +manual pipeline gave a different origin. PR #1836, issue #1788. + +**How to recognize it next time:** + +- An outer preflight writes transformed metadata, and execute runs a filter that transforms the same metadata. + +--- + +## 12. Give all ghost cells one label + +**Rule:** give one exterior region one label. Keep metadata about where a boundary cell is in separate storage. + +**Pattern to use:** `M3CSurfaceMeshing` writes `k_GhostLabel` to every cell of the ghost shell. + +**Pattern to avoid:** different sentinel labels that record which side of the box a ghost cell is on. + +**Why:** the mesher treats two different labels as a material interface. + +**How it showed up:** six ghost sentinels made false interfaces inside the shell, and M3C made triangles outside the +volume. PR #1718. + +**How to recognize it next time:** + +- Sentinel values that encode both material identity and location. + +--- + +## 13. Compute a shared intersection point from a canonical endpoint order + +**Rule:** every calculation of the same intersection point uses the same endpoint order, and sets the constrained +coordinate exactly. + +**Pattern to use:** `IntersectTriangleWithPlane()` in `src/simplnx/Utilities/GeometryUtilities.cpp`: + +```cpp +// Interpolate from the lower-z endpoint so shared edges produce identical points. +if(s[2] > en[2]) +{ + std::swap(s, en); + std::swap(d1, d2); +} +const float t = d1 / (d1 - d2); +Point3Df p = s + (en - s) * t; +p[2] = planeZ; +``` + +**Pattern to avoid:** let the winding of each triangle choose the interpolation order. + +**Why:** floating-point calculations that are equal in algebra can give different results. Vertex deduplication with +exact equality then leaves two vertices where there should be one. + +**How it showed up:** **Slice Triangle Geometry** made disconnected outlines along edges that two triangles share. PR +#1763. + +**How to recognize it next time:** + +- Adjacent primitives that compute a shared point independently, followed by a merge with exact equality. + +--- + +## 14. Key a clipped vertex by its source vertex and its clip point + +**Rule:** one source vertex can give several output vertices when an operation clips. Include the intersection point +in the key of a clipped vertex, and keep a separate output-to-source map for the copied attributes. + +**Pattern to use:** `CropEdgeGeometry` keys its clipped vertices with +`std::map, uint64>`, and copies vertex attributes through a separate source map. + +**Pattern to avoid:** cache one clip point for each source vertex outside the crop box. + +**Why:** two edges from the same outside vertex can cross the crop boundary at different points. + +**How it showed up:** in **Crop Geometry (Edge)**, two crossing edges shared one substituted end point, which moved one +edge off its original line. PR #1765. + +**How to recognize it next time:** + +- A topology-changing operation that assumes one output vertex for each source vertex. + +--- + +## 15. Convert an optional legacy SIMPL key only when it is present + +**Rule:** in a `FromSIMPLJson()` conversion, keep the default when an optional key that later SIMPL versions added is +absent. When the key is present, convert it and keep its errors. + +**Pattern to use:** `ErodeDilateBadDataFilter::FromSIMPLJson()`: + +```cpp +// IgnoredDataArrayPaths was added to the legacy filter after 6.5.49; older pipelines omit the key. +if(json.contains(SIMPL::k_IgnoredDataArrayPathsKey)) +{ + results.push_back(SIMPLConversion::ConvertParameter( + args, json, SIMPL::k_IgnoredDataArrayPathsKey, k_IgnoredDataArrayPaths_Key)); +} +``` + +**Pattern to avoid:** expect a default argument to prevent a missing-key error, or discard every failed conversion +result. + +**Why:** `SIMPLConversion::ConvertParameter()` returns error -571 for a missing key. Discarding all errors also hides a +malformed value that the file does supply. + +**How it showed up:** older pipelines that did not have the later ignore-list and randomization keys failed to convert. +PR #1682. + +**How to recognize it next time:** + +- A converter that sets defaults and then converts optional keys unconditionally, or that adds a result only when it is + valid. + +--- + +## 16. Check the binary STL file size before you use attribute byte counts + +**Rule:** before you use the attribute field of a binary STL triangle as a payload length, check that the file has +room for attribute payloads. Keep the overrides for known vendors. + +**Pattern to use:** `StlUtilities::SanityCheckFile()` compares the file size with the size of the fixed records and +excludes Magics and VXelements files. `ReadStlFile` checks its result before it seeks. + +**Pattern to avoid:** decide the layout of the whole file from triangle 0, or always seek by the attribute field. + +**Why:** some writers store color bits or random data in that field without writing payload bytes. + +**How it showed up:** an uncolored first triangle hid a Magics color encoding, and later triangles caused invalid seeks +and a corrupt geometry in **Read STL File**. PR #1696. + +**How to recognize it next time:** + +- A format field that vendors interpret in different ways, and a heuristic that samples only the first record. + +--- + +## 17. Give a geometry without physical coordinates an unspecified length unit + +**Rule:** pass `IGeometry::LengthUnit::Unspecified` when the coordinates of a geometry are orientation-space values, +projection values or image pixels without a physical scale. + +**Pattern to use:** the geometry actions in `ConvertOrientationsToVertexGeometryFilter`, +`ComputeGBCDPoleFigureFilter` and `WritePoleFigureFilter` pass the unit explicitly. + +**Pattern to avoid:** accept the physical default unit of a geometry action for coordinates that are not lengths. + +**Why:** the numbers can be correct while their unit metadata is false. + +**How it showed up:** the Rodrigues geometry reported Meter, and pole-figure images reported Micrometer. PR #1838, +issue #1803. + +**How to recognize it next time:** + +- A plotting or orientation filter that creates a geometry without a unit argument. + +--- + +## 18. Apply the hexagonal axis correction to both hexagonal Laue classes + +**Rule:** the Oxford-to-EDAX hexagonal axis correction applies to `Hexagonal_Low` and `Hexagonal_High`. Express the 30 +degree offset in the angular unit of the source data. + +**Pattern to use:** `EbsdReaderUtilities::IsHexagonalCrystalStructure()` with `k_HexagonalAlignmentDegrees` or +`k_HexagonalAlignmentRadiansD`. `ReadChannel5Data`, `ReadCtfData` and `ReadH5OinaData` use them. + +**Pattern to avoid:** test only `Hexagonal_High`, or add the literal 30 to angles in radians. + +**Why:** the a/a* axis convention belongs to the hexagonal lattice, not only to its high-symmetry Laue class. + +**How it showed up:** PR #1728 fixed a correction of 30 radians. PR #1781 added the missing correction to **Read Oxford +Instr. Channel 5 (.cpr/.crc)** and included `Hexagonal_Low`. + +**How to recognize it next time:** + +- Reader alignment code that selects a coordinate convention by one symmetry enum value. + +--- + +## 19. Use the search grid only to find neighbor candidates + +**Rule:** compute acceptance radii from the selected neighborhood definition. In diameter-based mode, use each +feature's own diameter. In fixed-radius mode, use the user-supplied radius for every feature. Spatial bins find +candidates; they do not change the acceptance test. + +**Pattern to use:** in diameter-based mode, `ComputeNeighborhoods` uses the average diameter for the bin size and each +feature's diameter for its radius: + +```cpp +radii[i] = equivalentDiameters[i] * multiplesOfAverage; +binSize = avgDiameter; +``` + +**Pattern to avoid:** replace the per-feature radii of the diameter-based mode with a global radius, or make its +directional neighbor relation symmetric. + +**Why:** an acceleration structure must not change which neighbors qualify. + +**How it showed up:** a performance rewrite used `avgDiameter * multiples / 2` as the radius, and **Compute Feature +Neighborhoods** found far fewer neighbors. PR #1652. + +**How to recognize it next time:** + +- A performance rewrite that uses the bin width as the scientific acceptance threshold. + +--- + +## Entry template + +Copy this block for a new entry, number it, and add it to the index. + +```markdown +## N. + +**Rule:** what to do and what not to do, in one or two sentences. + +**Pattern to use:** a short code example, or the file and function that already does it right. + +**Pattern to avoid:** a short code example of the mistake. + +**Why:** the mechanism that makes the mistake fail. + +**How it showed up:** the symptom, the filter or test, and the pull request or issue that fixed it. + +**How to recognize it next time:** + +- the code shape or the error message that gives it away. +``` From aac0da218c1084fdc9e5b163558eaf55ccb87d50 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 27 Oct 2026 14:34:59 -0400 Subject: [PATCH 02/13] DOC: Cite pull requests in the coding documents without their status * Remove every statement of whether a pull request or issue is open, merged, closed or not merged yet from the knowledge base and the style guide. Keep the pull request and issue numbers. * Add the rule to the introduction of the knowledge base: cite pull requests and issues by number, but not their status, because the status changes and the entry would go stale. Signed-off-by: Joey Kleingers --- docs/Code_Style_Guide.md | 2 +- docs/Coding_Knowledge_Base.md | 19 ++++++++++--------- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/Code_Style_Guide.md b/docs/Code_Style_Guide.md index 428ad9ad10..7a517546ab 100644 --- a/docs/Code_Style_Guide.md +++ b/docs/Code_Style_Guide.md @@ -320,7 +320,7 @@ The product of a tuple shape or a component shape can be too large for `usize`. - checks each multiplication for overflow, then the element count (tuples times components) and the byte count (elements times `sizeof(T)`). - does all checks before it allocates memory or changes a member. -`resizeTuples()` returns an error for an overflow and keeps the store unchanged. `DataStore`, `StringStore` and `ListStore` return error -6035 for a resize failure. `EmptyDataStore` returns `EmptyDataStore::k_ResizeError` (-10603). A constructor throws, because it cannot return a `Result`. Each overflow message names the count that does not fit ("tuple count", "element count" or "byte count"), so that a test can tell the checks apart. Issues #1883 and #1884 track the stores that do not follow this rule yet. +`resizeTuples()` returns an error for an overflow and keeps the store unchanged. `DataStore`, `StringStore` and `ListStore` return error -6035 for a resize failure. `EmptyDataStore` returns `EmptyDataStore::k_ResizeError` (-10603). A constructor throws, because it cannot return a `Result`. Each overflow message names the count that does not fit ("tuple count", "element count" or "byte count"), so that a test can tell the checks apart. Issues #1883 and #1884 describe stores that break this rule. ## Geometry Conventions diff --git a/docs/Coding_Knowledge_Base.md b/docs/Coding_Knowledge_Base.md index 35ab2c6f26..fb1253e791 100644 --- a/docs/Coding_Knowledge_Base.md +++ b/docs/Coding_Knowledge_Base.md @@ -5,8 +5,9 @@ made, why it happened, how it was found, and the rule that prevents it. The audi agents that work in this repository. Read the rules before you write or review code in the area they cover. Add a new entry when a defect taught something that the code itself does not make obvious. Keep the entries short. -State the rule first, then the evidence. A project convention that does not come from a defect belongs in -[the code style guide](Code_Style_Guide.md) instead. +State the rule first, then the evidence. Cite pull requests and issues by number, but do not say whether they are +open, merged or closed: that changes, and the entry would go stale. A project convention that does not come from a +defect belongs in [the code style guide](Code_Style_Guide.md) instead. ## Index @@ -54,7 +55,7 @@ State the rule first, then the evidence. A project convention that does not come object becomes a link. Record an object in that map only when it belongs to the `DataStructure` that is being written. Return an error when an ID is already recorded at a different path; do not overwrite the entry. -**Pattern to use:** `DataStructureWriter::addWriter()` in open pull request #1859 ignores objects outside +**Pattern to use:** `DataStructureWriter::addWriter()` in pull request #1859 ignores objects outside `m_WrittenStructure`, obtains the object ID and writer path, and inserts them with `m_IdMap.try_emplace()`. If the ID already maps to a different path, it returns error -6055 instead of replacing that mapping. @@ -66,7 +67,7 @@ structure and the written structure can both hold an object with the same ID. **How it showed up:** the NeighborList writer sends a temporary counts array through the main writer. That array replaced the map entry of an unrelated object that had two locations, and the second location became a link to the -counts dataset. Issue #1831. Fixed by open PR #1859, not merged yet. +counts dataset. Issue #1831. Fixed by PR #1859. **How to recognize it next time:** @@ -80,7 +81,7 @@ counts dataset. Issue #1831. Fixed by open PR #1859, not merged yet. **Rule:** a writer that owns an HDF5 file closes it explicitly and checks the result before it returns success. A close failure is returned before `AtomicFile::commit()` replaces the old file. -**Pattern to use:** `FileIO::closeChecked()` in open pull request #1859. For example, the path-based write of +**Pattern to use:** `FileIO::closeChecked()` in pull request #1859. For example, the path-based write of `DataStructureWriter` ends with `return MergeResults(std::move(writeResult), fileWriter.closeChecked());`. **Pattern to avoid:** return success while the `FileIO` object is still in scope, so that its destructor closes the @@ -90,7 +91,7 @@ file and discards the result of `H5Fclose()`. is complete. **How it showed up:** an injected close failure still gave a valid `Result`, so **Write DREAM3D-NX File** could replace -the old output file with an incomplete one. Issue #1832. Fixed by open PR #1859, not merged yet. +the old output file with an incomplete one. Issue #1832. Fixed by PR #1859. **How to recognize it next time:** @@ -146,8 +147,8 @@ not convert exceptions into errors. **How it showed up:** selections that the parameters accepted caused `std::bad_cast` in preflight, or reached a Boolean type that the algorithm dispatch did not support. Issues #1821 and #1833. PRs #1817 and #1763 fixed the affected -filters. Open PR #1766 tightens the parameter declarations and preflight checks of individual filters; it does not fix -the shared parameter-validation gaps and is not merged yet. +filters. PR #1766 tightens the parameter declarations and preflight checks of individual filters; it does not fix +the shared parameter-validation gaps. **How to recognize it next time:** @@ -199,7 +200,7 @@ result as an arithmetic average after unwrapping. **Why:** the circular mean and the unwrapped arithmetic mean are different for an asymmetric population. **How it showed up:** PR #1670 (issue #1665) removed constant-offset corrections from the triangle and feature centroid -filters. The arithmetic centroid for cells in `ComputeFeatureCentroids` is fixed by open PR #1764, not merged yet. +filters. The arithmetic centroid for cells in `ComputeFeatureCentroids` is fixed by PR #1764. **How to recognize it next time:** From cc5277ca100cc6ab6a6166737680e4a8f0b5ed40 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 27 Oct 2026 15:45:38 -0400 Subject: [PATCH 03/13] DOC: Point error and warning message guidance to the error-messages skill Every rule that says how to word an error or warning message now names the bluequartz-skills error-messages skill, and the message examples follow it. --- AGENTS.md | 6 +++--- docs/Code_Style_Guide.md | 14 ++++++++------ 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b210765d61..595c1fc070 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,9 +13,9 @@ dispatch prompt says so. - `bluequartz-skills:simplnx-best-practices`: simplnx and simplnx plugin code. Task skills own the details: `filter-implementation` for filters and algorithms, `implement-filter-tests` and -`exemplar-testing` for tests, `publish-exemplars` for test-data archives, `thread-safety` for parallel code, -`ooc-architecture` for out-of-core storage, `pipeline-files` for `.d3dpipeline` files, and `github-workflow` for -commits and pull requests. +`exemplar-testing` for tests, `publish-exemplars` for test-data archives, `error-messages` for error and warning +messages, `thread-safety` for parallel code, `ooc-architecture` for out-of-core storage, `pipeline-files` for +`.d3dpipeline` files, and `github-workflow` for commits and pull requests. ## Where new rules go diff --git a/docs/Code_Style_Guide.md b/docs/Code_Style_Guide.md index 7a517546ab..ef9d1ea9e0 100644 --- a/docs/Code_Style_Guide.md +++ b/docs/Code_Style_Guide.md @@ -128,7 +128,7 @@ Write the algorithm's preconditions as properties of its inputs (`@pre Epsilon a // In the algorithm, before any fixed-size buffer is indexed by component if(numComponents != 2 && numComponents != 3) { - return MakeErrorResult(-54060, fmt::format("Input array has {} components but only 2 or 3 are accepted.", numComponents)); + return MakeErrorResult(-54060, fmt::format("Input array '{}' has {} components, but only 2 or 3 are accepted.", inputArrayPath.toString(), numComponents)); } ### Put Each Check at the Earliest Point That Can See Its Data @@ -165,9 +165,9 @@ Tuple-count rules for input arrays: - **Outputs sized from an Attribute Matrix**: every input that sets the loop count must have the same tuple count as that Attribute Matrix. - **A spatial filter that modifies Feature Ids, its sibling cell arrays, or the Feature Attribute Matrix** requires Feature Ids to be in the geometry's Cell Attribute Matrix. Use `ValidateArrayInGeometryCellData()`, `ValidateArraysInGeometryCellData()` and `ValidateFeatureAttributeMatrixInGeometry()`. -Every message names both paths and both counts: +Every message names both paths and both counts, as the `bluequartz-skills:error-messages` skill requires: - "The selected DataArray 'Face Labels' has 12 tuples, but the Triangle Geometry 'Mesh' has 24 faces." + "The selected DataArray 'Mesh/Face Data/Face Labels' has 12 tuples, but the Triangle Geometry 'Mesh' has 24 faces." ### Range-Check Every Index That Comes From Data @@ -209,7 +209,7 @@ A direction or axis parameter must not default to `{0, 0, 0}`, and preflight rej - Use `getCellData()`, `getVertexAttributeMatrix()`, `getFaceAttributeMatrix()` and similar accessors and check for `nullptr`. A geometry can have lost its Attribute Matrix. - Use non-throwing library calls (`std::filesystem::directory_iterator(path, errorCode)`, `nlohmann::json::parse(text, nullptr, false)`), or catch the exception and return an error `Result`. - Check `Result::invalid()` and `std::optional::has_value()` before calling `.value()`. -- When code walks every child of an Attribute Matrix, handle each array kind (`IDataArray`, `StringArray`, `INeighborList`) or return an error that names the array. +- When code walks every child of an Attribute Matrix, handle each array kind (`IDataArray`, `StringArray`, `INeighborList`) or return an error that names the array. Write that message by the `bluequartz-skills:error-messages` skill. ### Preflight and Execute Make the Same Decisions @@ -237,7 +237,7 @@ Build each `DataPath` from the path that the user selected or that preflight cre ### Never Drop a Result -Return every `Result` or merge it into the `Result` that the function returns. A later warning or error must not replace an earlier one. +Return every `Result` or merge it into the `Result` that the function returns. A later warning or error must not replace an earlier one. The text of each error and warning, and how to read the errors of a `Result`, follow the `bluequartz-skills:error-messages` skill. if(suppressedFaceCount == 0) { @@ -320,7 +320,7 @@ The product of a tuple shape or a component shape can be too large for `usize`. - checks each multiplication for overflow, then the element count (tuples times components) and the byte count (elements times `sizeof(T)`). - does all checks before it allocates memory or changes a member. -`resizeTuples()` returns an error for an overflow and keeps the store unchanged. `DataStore`, `StringStore` and `ListStore` return error -6035 for a resize failure. `EmptyDataStore` returns `EmptyDataStore::k_ResizeError` (-10603). A constructor throws, because it cannot return a `Result`. Each overflow message names the count that does not fit ("tuple count", "element count" or "byte count"), so that a test can tell the checks apart. Issues #1883 and #1884 describe stores that break this rule. +`resizeTuples()` returns an error for an overflow and keeps the store unchanged. `DataStore`, `StringStore` and `ListStore` return error -6035 for a resize failure. `EmptyDataStore` returns `EmptyDataStore::k_ResizeError` (-10603). A constructor throws, because it cannot return a `Result`. Each overflow message names the count that does not fit ("tuple count", "element count" or "byte count"), so that a test can tell the checks apart; the rest of its text follows the `bluequartz-skills:error-messages` skill. Issues #1883 and #1884 describe stores that break this rule. ## Geometry Conventions @@ -330,6 +330,8 @@ The product of a tuple shape or a component shape can be too large for `usize`. ## Error Codes and Messages +Write every error and warning message by the `bluequartz-skills:error-messages` skill: it says which values, paths and corrective actions a message shows and how it formats them. The code-allocation rules below supplement that skill. + ### Error Codes Are Unique - Each condition gets its own error code; each warning condition gets its own warning code. Do not copy the code from the check above. From 364826f89e59d955dcff2f6b0ba3a17254839ea2 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 27 Oct 2026 16:05:54 -0400 Subject: [PATCH 04/13] DOC: Share one input check between preflight and the algorithm An algorithm class keeps its input checks in one static ValidateInputs() that preflightImpl() and operator()() both call, instead of a second copy in preflight. The rule covers inputs that the filter's own actions move, placeholder counts from earlier filters, dispatch to several algorithm classes, and how to test a rejection that returns before the dispatch. --- docs/Code_Style_Guide.md | 54 ++++++++++++++++++++++++++++++++++------ 1 file changed, 47 insertions(+), 7 deletions(-) diff --git a/docs/Code_Style_Guide.md b/docs/Code_Style_Guide.md index ef9d1ea9e0..203f9ffb92 100644 --- a/docs/Code_Style_Guide.md +++ b/docs/Code_Style_Guide.md @@ -121,15 +121,55 @@ The algorithm therefore checks every input property it relies on, and returns an Filter parameters and preflight give the user early feedback. They do not replace the checks in the algorithm. Do not remove a check from an algorithm because the filter makes it unreachable. Test each of these checks by calling the algorithm directly. -When an algorithm dispatches to more than one implementation (for example an in-memory branch and an out-of-core branch), each branch makes these checks before its first read. A check that is present in one branch only is not enough. +Write the checks that preflight can make once, in a static `ValidateInputs()` function of the algorithm class, and call it from both places. `preflightImpl()` calls it before it builds its output actions, and the algorithm's `operator()()` calls it before its first read. Do not write a second copy of a check in `preflightImpl()`: two copies drift apart in their conditions, error codes and messages. Build the input values in one function of the filter that `preflightImpl()` and `executeImpl()` both call, so that both check the same inputs with the same options. + +- `ValidateInputs()` checks every input condition that parameter values and metadata can answer: scalar ranges and combinations of options, the existence and type of each input object, component shapes, tuple and geometry element counts, and the header of an input file. It checks an optional input only when its option is on, as execute does. +- It checks inputs only. Outputs, and other objects that the filter's own actions create, do not exist yet during preflight. +- When the filter's own actions move an input, preflight and the algorithm find it at different paths. `ExtractComponentAsArrayFilter` renames the selected array to a temporary name, and its algorithm reads the array under that name. The caller passes the path where the input is at that point (before the actions in preflight, after them in execute); `ValidateInputs()` does not guess the phase from which path exists. +- A count that an earlier filter computes in execute is only a placeholder during preflight: `QuickSurfaceMeshFilter` creates its Triangle Geometry with 0 faces (see entry 9 of the [coding knowledge base](Coding_Knowledge_Base.md)). `ValidateInputs()` can compare such counts with each other. A check that needs the final count, such as a check for an empty mesh, runs in execute, like the checks of array values (for example the index-range check below). +- A check that a parameter already makes (for example `AllowedComponentShapes`) still belongs in it. Through the filter that check runs twice, but a direct caller of the algorithm needs it. +- Build it from the shared utilities, such as `CheckArraysHaveSameTupleCount()` and `ValidateCellTupleCount()`. A reader checks the file header with the same function in both places; execute still handles a failed read, because the file can change after preflight. +- Checks that only the filter needs stay in `preflightImpl()`, for example checks of output names and of parameters that the algorithm does not read. + +When an algorithm dispatches to more than one implementation (for example an in-memory branch and an out-of-core branch), call `ValidateInputs()` before the dispatch, so that every branch is covered. When a filter dispatches straight to two algorithm classes, as `ComputeGBCDPoleFigureFilter` does, one class owns `ValidateInputs()` and both call it first. A check that only one branch needs stays in that branch, before its first read. A filter whose actions do all of its work, such as `DeleteDataFilter`, needs no algorithm class. Write the algorithm's preconditions as properties of its inputs (`@pre Epsilon and MinPoints are positive.`). Do not write them as checks that a filter made (`@pre Filter preflight has validated Epsilon.`), because a direct caller does not run that filter. - // In the algorithm, before any fixed-size buffer is indexed by component - if(numComponents != 2 && numComponents != 3) + // Algorithms/ExampleAlgorithm.cpp + Result<> ExampleAlgorithm::ValidateInputs(const DataStructure& dataStructure, const ExampleAlgorithmInputValues& inputValues) + { + const auto* inputArray = dataStructure.getDataAs(inputValues.InputArrayPath); + if(inputArray == nullptr) + { + return MakeErrorResult(k_InputArrayMissing, fmt::format("The input array '{}' does not exist or is not a DataArray.", inputValues.InputArrayPath.toString())); + } + const usize numComponents = inputArray->getNumberOfComponents(); + if(numComponents != 2 && numComponents != 3) + { + return MakeErrorResult(k_InputComponentCount, fmt::format("Input array '{}' has {} components, but only 2 or 3 are accepted.", inputValues.InputArrayPath.toString(), numComponents)); + } + return {}; + } + + Result<> ExampleAlgorithm::operator()() + { + Result<> validationResult = ValidateInputs(m_DataStructure, *m_InputValues); + if(validationResult.invalid()) + { + return validationResult; + } + // ... dispatch and compute, merging each Result into validationResult + return validationResult; + } + + // ExampleFilter.cpp, in preflightImpl() + const ExampleAlgorithmInputValues inputValues = CreateInputValues(filterArgs); + Result resultOutputActions = ConvertResultTo(ExampleAlgorithm::ValidateInputs(dataStructure, inputValues), {}); + if(resultOutputActions.invalid()) { - return MakeErrorResult(-54060, fmt::format("Input array '{}' has {} components, but only 2 or 3 are accepted.", inputArrayPath.toString(), numComponents)); + return {std::move(resultOutputActions)}; } + // ... append the output actions; keep these warnings when a later check fails ### Put Each Check at the Earliest Point That Can See Its Data @@ -139,7 +179,7 @@ A filter checks its inputs at three levels. Put each check at the earliest level 2. **`preflightImpl()`.** Parameter values, relationships between inputs (tuple counts, geometry element counts), geometry and file metadata. Preflight cannot see array values. 3. **Execute.** Checks that need array values, such as feature ids or phases used as indices. -Do not repeat a parameter or preflight check inside the filter's `executeImpl()`. The algorithm class still keeps its own checks (see above). +Do not repeat a parameter or preflight check inside the filter's `executeImpl()`. The algorithm still runs its `ValidateInputs()` (see above); that is the same check, not a second copy. ### Declare the Component Shape the Algorithm Reads @@ -354,7 +394,7 @@ Write every error and warning message by the `bluequartz-skills:error-messages` - **Expected values come from an independent oracle**: a hand calculation, a closed-form result, a reference implementation or verified DREAM3D 6.5.171 output. Never regenerate an exemplar only to absorb a change in output; show with the oracle that the new value is correct. - **Generated and expected data come from different `DataStructure`s.** A comparison of an array with itself cannot fail. - **Test every mode.** A test that only runs a mode is not enough; it compares the output values of that mode with expected values. -- **Test both algorithm paths.** For a filter with in-core and out-of-core algorithms, run each test for both with `GENERATE(from_range(UnitTest::SelectAlgorithmTestScenariosForInMemoryStores()))` and `UnitTest::AlgorithmTestScope`. +- **Test both algorithm paths.** For a filter with in-core and out-of-core algorithms, run each test that reaches the dispatch for both with `GENERATE(from_range(UnitTest::SelectAlgorithmTestScenariosForInMemoryStores()))` and `UnitTest::AlgorithmTestScope`. - **Test every axis and plane** for code that has one block per axis or plane. - **Test error paths by code.** Assert the exact error code, not only `REQUIRE_NOTHROW` or an invalid result. - **Test degenerate and invalid inputs**: empty geometries, out-of-range feature ids, NaN and limit values (`2^63`, `2^64`, `-1` and `256` for `uint8`), inputs in a `DataGroup` instead of at the root, and arrays from a different Attribute Matrix. @@ -364,7 +404,7 @@ Write every error and warning message by the `bluequartz-skills:error-messages` - **Test a size failure without the allocator.** The nightly `asan` job runs with the default AddressSanitizer options. A request above the AddressSanitizer limit (1 TiB on 64-bit platforms) stops the test process instead of throwing `std::bad_alloc` (#1885). Use a size that the code rejects before it allocates, such as a shape product that overflows. Test a recoverable allocation failure through a deterministic failure seam. Test an intentional allocator abort in a separate process, with a timeout and a check of its exit result. An AddressSanitizer build alone does not make a real allocation failure recoverable. - **No network access in CTest.** A test that needs a network service fails when the service is slow or not available. Do not register it with CTest. Keep the script so that it can be run by hand. - **Call `UnitTest::LoadPlugins();` only when the test uses plugin code** that the `Application` must load, for example a test that gets filters from the filter list or runs a pipeline that does. A test that constructs a filter or algorithm class directly and builds its data in memory does not call it. -- **Test an algorithm's own checks by calling the algorithm directly**, once for each dispatch branch. To reach an out-of-core branch in the in-core build, use a store whose `getStoreType()` returns `IDataStore::StoreType::OutOfCore` (the ImageProcessing tests do this). When that branch needs external sort or temporary record stores, register a test `IDataIOManager` that provides them, as described in the next item. +- **Test an algorithm's own checks by calling the algorithm directly.** Test each failure of `ValidateInputs()` through `operator()()` and assert its error code; a test of `ValidateInputs()` alone does not show that the algorithm calls it. Such a failure returns before the dispatch, so its test does not use `UnitTest::AlgorithmTestScope`, which requires a branch to run. Test a check that is inside a dispatch branch on that branch. To reach an out-of-core branch in the in-core build, use a store whose `getStoreType()` returns `IDataStore::StoreType::OutOfCore` (the ImageProcessing tests do this). When that branch needs external sort or temporary record stores, register a test `IDataIOManager` that provides them, as described in the next item. - **Restore the global I/O state.** A test that registers an `IDataIOManager` changes the `DataIOCollection` for the rest of the process. `DataIOCollection` has no remove call, so replace an existing manager under its format name and re-add the saved original in a destructor, which also runs when an assertion fails (see `ScopedImportSelectionManager` in `test/Dream3dLoadingApiTest.cpp`). Do not rely on ctest running each `TEST_CASE` in its own process: the test executable run directly runs many cases in one process, and a leaked manager makes later tests fail or skip their checks. ## Build Configuration From 8f1d0a4e4dd55ba1407d269b40a6273770954f0c Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 27 Oct 2026 16:35:27 -0400 Subject: [PATCH 05/13] DOC: Add a rule that preflight does not allocate or read array data Preflight runs on every parameter change and on placeholder stores, so its memory may grow with metadata and short previews but not with the element data of an array. The rule allows counts as output shapes and reads past a file header when the format needs them, and names CreateStringArrayAction as a current exception (issue #1892). --- docs/Code_Style_Guide.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/Code_Style_Guide.md b/docs/Code_Style_Guide.md index 203f9ffb92..09ee8f0cf2 100644 --- a/docs/Code_Style_Guide.md +++ b/docs/Code_Style_Guide.md @@ -251,6 +251,17 @@ A direction or axis parameter must not default to `{0, 0, 0}`, and preflight rej - Check `Result::invalid()` and `std::optional::has_value()` before calling `.value()`. - When code walks every child of an Attribute Matrix, handle each array kind (`IDataArray`, `StringArray`, `INeighborList`) or return an error that names the array. Write that message by the `bluequartz-skills:error-messages` skill. +### Preflight Does Not Allocate or Read Array Data + +Preflight runs every time a parameter changes in the GUI, on arrays whose stores are usually placeholders. It must stay cheap: its memory may grow with metadata and short previews, but not with the element data of an array. + +- Declare every output through an action. In preflight an action does not allocate element storage: `CreateArrayAction` and the array-creation paths of the geometry actions create an `EmptyDataStore`, and `CreateNeighborListAction` creates an `EmptyListStore`. Do not create an array with real storage in `preflightImpl()` or in a helper that it calls. `CreateStringArrayAction` allocates every string in preflight and breaks this rule (issue #1892). +- `CopyDataObjectAction` and the copy paths of the geometry actions copy the store of their source. In pipeline preflight the source is usually a placeholder, so the copy is one too. `CopyArrayInstanceAction` creates a new array with the shape and type of its source, a placeholder in preflight. +- Use tuple, element and file-derived counts to compute output shapes. A shape stores dimension counts, not element values. +- Metadata may grow with phases, columns, files, tiles, output objects (even one output per input tuple, as in `SplitDataArrayByTupleFilter`) and the coordinates along one axis (`RegularizeZSpacingFilter` reads the Z + 1 slice bounds). A preview that preflight builds from parameter values holds a few tuples and may grow with the component count, as in `InitializeDataFilter`. Apart from these, do not allocate a scratch buffer, vector or set with one entry per cell, vertex, face or tuple. +- Do not read element values from a `DataStructure` array; read only its shape, type and counts. During preflight its store is usually an `EmptyDataStore`, and an element read from it throws. +- A reader may read past the file header when the format requires it to find or check the output structure. `ReadCSVFileFilter` scans the whole file to count its lines, `ReadDeformKeyFileV12Filter` reads the first record of each array to find its component count, and `ReadGrainMapper3DFilter` reads whole HDF5 metadata datasets. Read only what preflight needs. When a scan passes over array data, hold at most a fixed-size buffer or the one record that it needs, keep none of the scanned records, check `shouldCancel` during a long scan, and cache the result (see "Static Header Caches Need a Mutex"). Do not decode image pixels or load the data of an array only to find metadata. + ### Preflight and Execute Make the Same Decisions Preflight and execute must reach the same decision from the same inputs: which arrays are processed, output names, crop or resample indices, which outputs are enabled. Put that decision in one function and call it from both places. Do not copy the logic. From e552c8abf32ab9d2f0be53fbabc23baac194f802 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 27 Oct 2026 16:44:07 -0400 Subject: [PATCH 06/13] DOC: State when a test loads the plugins Name what needs UnitTest::LoadPlugins(): a filter looked up by UUID through the filter list, a loaded pipeline, or a filter that gets other filters from the list. A test that constructs its filter directly does not call it, also when it reads its data from a .dream3d file. Remove the call from the AlgorithmDispatch example test, which does not need it. --- docs/AlgorithmDispatch.md | 1 - docs/Code_Style_Guide.md | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/AlgorithmDispatch.md b/docs/AlgorithmDispatch.md index 68ae804ea5..f35d54ea1a 100644 --- a/docs/AlgorithmDispatch.md +++ b/docs/AlgorithmDispatch.md @@ -226,7 +226,6 @@ RAII guard that sets `ForceOocAlgorithm()` on construction and restores the prev TEST_CASE("MyFilter", "[Plugin][MyFilter]") { - UnitTest::LoadPlugins(); bool forceOocAlgo = GENERATE(false, true); const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); diff --git a/docs/Code_Style_Guide.md b/docs/Code_Style_Guide.md index 09ee8f0cf2..806842c683 100644 --- a/docs/Code_Style_Guide.md +++ b/docs/Code_Style_Guide.md @@ -414,7 +414,7 @@ Write every error and warning message by the `bluequartz-skills:error-messages` - **Apply an action in both modes.** An `IDataAction` creates different objects in preflight and in execute. Test it with `GENERATE(IDataAction::Mode::Preflight, IDataAction::Mode::Execute)` and an `INFO` that names the mode. - **Test a size failure without the allocator.** The nightly `asan` job runs with the default AddressSanitizer options. A request above the AddressSanitizer limit (1 TiB on 64-bit platforms) stops the test process instead of throwing `std::bad_alloc` (#1885). Use a size that the code rejects before it allocates, such as a shape product that overflows. Test a recoverable allocation failure through a deterministic failure seam. Test an intentional allocator abort in a separate process, with a timeout and a check of its exit result. An AddressSanitizer build alone does not make a real allocation failure recoverable. - **No network access in CTest.** A test that needs a network service fails when the service is slow or not available. Do not register it with CTest. Keep the script so that it can be run by hand. -- **Call `UnitTest::LoadPlugins();` only when the test uses plugin code** that the `Application` must load, for example a test that gets filters from the filter list or runs a pipeline that does. A test that constructs a filter or algorithm class directly and builds its data in memory does not call it. +- **Call `UnitTest::LoadPlugins();` only when the test looks up filters by UUID.** Loading the plugins fills the `Application`'s filter list. A test needs it when it calls `createFilter()` on the filter list, loads a pipeline with `Pipeline::FromJson()`, `FromFile()`, `FromSIMPLJson()` or `FromSIMPLFile()` (every SIMPL backwards-compatibility test does), or runs a filter whose code gets other filters from the filter list (`ReadH5Ebsd` does). A test that constructs a filter or algorithm class directly does not call it, whether it builds its data in memory or reads it with `UnitTest::LoadDataStructure()`. The call loads every plugin library and fails the test when one does not load, so an unneeded call makes the test fail for a plugin it does not use. A test that needs the plugins calls `LoadPlugins()` itself: ctest runs each `TEST_CASE` in its own process, so no other test has loaded them. - **Test an algorithm's own checks by calling the algorithm directly.** Test each failure of `ValidateInputs()` through `operator()()` and assert its error code; a test of `ValidateInputs()` alone does not show that the algorithm calls it. Such a failure returns before the dispatch, so its test does not use `UnitTest::AlgorithmTestScope`, which requires a branch to run. Test a check that is inside a dispatch branch on that branch. To reach an out-of-core branch in the in-core build, use a store whose `getStoreType()` returns `IDataStore::StoreType::OutOfCore` (the ImageProcessing tests do this). When that branch needs external sort or temporary record stores, register a test `IDataIOManager` that provides them, as described in the next item. - **Restore the global I/O state.** A test that registers an `IDataIOManager` changes the `DataIOCollection` for the rest of the process. `DataIOCollection` has no remove call, so replace an existing manager under its format name and re-add the saved original in a destructor, which also runs when an assertion fails (see `ScopedImportSelectionManager` in `test/Dream3dLoadingApiTest.cpp`). Do not rely on ctest running each `TEST_CASE` in its own process: the test executable run directly runs many cases in one process, and a leaked manager makes later tests fail or skip their checks. From b48493a36b2bf1bb765e08b4b6b9e9c3056ca60f Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 27 Oct 2026 16:45:45 -0400 Subject: [PATCH 07/13] DOC: Show AlgorithmTestScope in the algorithm dispatch test example The guide tests both algorithm paths with AlgorithmTestScope and SelectAlgorithmTestScenariosForInMemoryStores(). Show that pattern in place of a bare ForceOocAlgorithmGuard, and describe ForceInCoreAlgorithm() with ForceOocAlgorithm(). --- docs/AlgorithmDispatch.md | 39 ++++++++++++++++++--------------------- 1 file changed, 18 insertions(+), 21 deletions(-) diff --git a/docs/AlgorithmDispatch.md b/docs/AlgorithmDispatch.md index f35d54ea1a..b5f64e16e0 100644 --- a/docs/AlgorithmDispatch.md +++ b/docs/AlgorithmDispatch.md @@ -193,47 +193,44 @@ Filters/Algorithms/ 5. **Register new files**: Add the new algorithm names to the plugin's `AlgorithmList` in `CMakeLists.txt`. -6. **Test both paths**: Use `ForceOocAlgorithmGuard` with Catch2 `GENERATE` to exercise both algorithm paths in every build configuration (see below). +6. **Test both paths**: Use `UnitTest::AlgorithmTestScope` with Catch2 `GENERATE` to exercise both algorithm paths in every build configuration (see below). ## Testing Both Algorithm Paths -CI typically only builds the in-core configuration, so the OOC algorithm path would never be exercised without an explicit override. The `ForceOocAlgorithm()` flag and `ForceOocAlgorithmGuard` RAII class solve this. +CI typically only builds the in-core configuration, so the OOC algorithm path would never be exercised without an explicit override. The `ForceOocAlgorithm()` and `ForceInCoreAlgorithm()` flags provide that override, and filter tests set them through `UnitTest::AlgorithmTestScope`. -### ForceOocAlgorithm +### ForceOocAlgorithm and ForceInCoreAlgorithm ```cpp bool& ForceOocAlgorithm(); +bool& ForceInCoreAlgorithm(); ``` -Returns a reference to a static bool. When set to `true`, `DispatchAlgorithm` always selects the OOC algorithm class regardless of whether the data arrays use chunked storage. Defaults to `false`. +Each returns a reference to a static bool that defaults to `false`. When `ForceOocAlgorithm()` is `true`, `DispatchAlgorithm` selects the OOC algorithm class even when no array uses chunked storage. `ForceInCoreAlgorithm()` takes precedence over both the storage check and `ForceOocAlgorithm()`. -### ForceOocAlgorithmGuard +### AlgorithmTestScope -```cpp -class ForceOocAlgorithmGuard -{ -public: - ForceOocAlgorithmGuard(bool force); - ~ForceOocAlgorithmGuard(); - // non-copyable, non-movable -}; -``` - -RAII guard that sets `ForceOocAlgorithm()` on construction and restores the previous value on destruction. Use with Catch2 `GENERATE` to run each test case twice — once with the in-core algorithm and once with the OOC algorithm: +`UnitTest::AlgorithmTestScope` (`simplnx/UnitTest/AlgorithmTestScope.hpp`) runs a test on one algorithm path with in-memory stores. `UnitTest::SelectAlgorithmTestScenariosForInMemoryStores()` returns the paths that the build's `SIMPLNX_TEST_ALGORITHM_PATH` setting allows, so `GENERATE` runs the test once for each: ```cpp -#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/UnitTest/AlgorithmTestScope.hpp" TEST_CASE("MyFilter", "[Plugin][MyFilter]") { - bool forceOocAlgo = GENERATE(false, true); - const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const auto scenario = GENERATE(from_range(UnitTest::SelectAlgorithmTestScenariosForInMemoryStores())); + CAPTURE(scenario); + UnitTest::AlgorithmTestScope scope(scenario); + + // ... build the DataStructure and the Arguments ... + + auto executeResult = scope.executeFilter(filter, dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - // ... test body runs identically for both algorithm paths ... + // ... compare the outputs with the expected values ... } ``` -This ensures both code paths are tested in every build configuration (in-core and OOC). The OOC algorithm class works correctly on in-core `DataStore` data because the chunk API methods are no-ops for `DataStore` (1 chunk spanning the full volume). +`scope.executeFilter()` and `scope.execute()` set the force flags only for the call they wrap, so the filters that build the test data run normally. After the call, the scope requires that the selected algorithm path ran and the other path did not. The OOC algorithm class works correctly on in-core `DataStore` data because the chunk API methods are no-ops for `DataStore` (1 chunk spanning the full volume). ## Performance Expectations From 393ef3f674df6a86b511e0af51d7de3d4bf243e9 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Wed, 28 Oct 2026 16:21:59 -0400 Subject: [PATCH 08/13] DOC: Add knowledge-base lessons and testing, HDF5 and public-code rules Knowledge base: validate an in-place array by the object ID of its Attribute Matrix; read a count-only check as IArray; keep the legacy conversions when a filter moves; set a distinct mtime in a file-change test; test geometry formulas with unequal spacing; test a transfer failure after an earlier batch. Entries cite code by file and function, not by line number. Style guide: simplnx holds only public code; compare a choice with its named value; keep error codes through a refactoring; check the stores before resident pointers or spans; a dataset from openDataset() is not valid until it is used. Testing: core tests in the core target with a direct test for each exported helper, tests pass with optional plugins off, plugin-prefixed test names, hidden known-defect tests, and unequal dimensions and spacings. --- docs/Code_Style_Guide.md | 27 +++++++ docs/Coding_Knowledge_Base.md | 145 ++++++++++++++++++++++++++++++++-- 2 files changed, 167 insertions(+), 5 deletions(-) diff --git a/docs/Code_Style_Guide.md b/docs/Code_Style_Guide.md index 806842c683..16a73741d7 100644 --- a/docs/Code_Style_Guide.md +++ b/docs/Code_Style_Guide.md @@ -106,6 +106,10 @@ When using ANSI C one should use a char* for constant strings: Using this type of approach allows for quicker code updates when constant values need to be changed. +## Public Code + +simplnx is public. Proprietary code stays in the private repository that owns it (for example SimplnxOoc or DREAM3D-NX). simplnx holds only public interfaces and integration points, and their names, comments and documentation do not describe a private implementation. + ## Filters and Algorithms ### Algorithms Are Callable Without Their Filter @@ -321,6 +325,10 @@ For outputs that must stay distinct from their inputs, preflight resolves each o An intentional in-place update, or a deferred replacement of an input, follows the documented contract of the filter and keeps the input until the replacement is ready. For example, with **Perform in Place** on, **Convert AttributeArray DataType** writes a temporary array, then deletes the input and renames the temporary array to the input path in deferred actions. +### Compare a Choice With Its Named Value + +When the values of an enum are the indices of a `ChoicesParameter`, compare the selected value, and pass default values and parameter links, with `to_underlying(Enum::Value)`, as `RemoveFlaggedFeaturesFilter` does. Do not compare with a raw index such as `== 2`. Keep the order of the choices and the enum values stable: a saved pipeline stores the index, so reordering the choices changes what an existing pipeline does. When the enum values are not the indices, map the index through a lookup (see Geometry Conventions). + ## Concurrency ### Static Header Caches Need a Mutex @@ -373,6 +381,16 @@ The product of a tuple shape or a component shape can be too large for `usize`. `resizeTuples()` returns an error for an overflow and keeps the store unchanged. `DataStore`, `StringStore` and `ListStore` return error -6035 for a resize failure. `EmptyDataStore` returns `EmptyDataStore::k_ResizeError` (-10603). A constructor throws, because it cannot return a `Result`. Each overflow message names the count that does not fit ("tuple count", "element count" or "byte count"), so that a test can tell the checks apart; the rest of its text follows the `bluequartz-skills:error-messages` skill. Issues #1883 and #1884 describe stores that break this rule. +### Check the Stores Before You Use Resident Pointers or Spans + +A path that reads or writes through a raw pointer or a span into store memory checks every store it uses: each store reports `IDataStore::StoreType::InMemory` and casts to the required `DataStore`. An algorithm-selection flag such as `ForceInCoreAlgorithm()` does not show that a store is in memory. When a check fails, use the store API instead (`ZeroCrossingEngine` checks both stores and both casts). Test that fallback with a store that is not a `DataStore`. + +## HDF5 Files + +### A Dataset From `openDataset()` Is Not Valid Until It Is Used + +`GroupIO::openDataset()` checks that the dataset exists and returns a `DatasetIO` that opens it only on first use, so `isValid()` returns false until then. Do not use `isValid()` to test that a dataset exists. Check `isDataset(name)` on the parent group, and check the `Result` of the first read. An empty `getDimensions()` does not mean that the dataset is missing: a scalar dataset has no dimensions. + ## Geometry Conventions - The Image Geometry origin is the minimum corner of cell 0. The center of cell `i` is `origin + (i + 0.5) * spacing`. Use `ImageGeom::getCoordsf()` or `getCoords()` for cell centers instead of a local formula. @@ -395,6 +413,10 @@ Write every error and warning message by the `bluequartz-skills:error-messages` Do not rely on `\b`; it is not portable extended-regex syntax. +### Keep Error Codes Through a Refactoring + +A refactoring keeps the error code of every condition whose meaning it does not change, also when the check moves to another function or class. Tests, documentation and users rely on the codes. When a condition must get a new code, the pull request description lists the old and the new code. + ### Messages Name the Right Values - Each message names the correct values in the correct order, and fires only on the condition it describes. Advice in a message must be something the user can do. @@ -413,6 +435,11 @@ Write every error and warning message by the `bluequartz-skills:error-messages` - **Print the errors of a failed Result.** Use `SIMPLNX_RESULT_REQUIRE_VALID(result)` instead of `REQUIRE(result.valid())`. It prints each error code and message when the check fails. - **Apply an action in both modes.** An `IDataAction` creates different objects in preflight and in execute. Test it with `GENERATE(IDataAction::Mode::Preflight, IDataAction::Mode::Execute)` and an `INFO` that names the mode. - **Test a size failure without the allocator.** The nightly `asan` job runs with the default AddressSanitizer options. A request above the AddressSanitizer limit (1 TiB on 64-bit platforms) stops the test process instead of throwing `std::bad_alloc` (#1885). Use a size that the code rejects before it allocates, such as a shape product that overflows. Test a recoverable allocation failure through a deterministic failure seam. Test an intentional allocator abort in a separate process, with a timeout and a check of its exit result. An AddressSanitizer build alone does not make a real allocation failure recoverable. +- **Test core code in the core test target.** A test of code in `src/simplnx/` goes in the core `test/` directory and carries the `[simplnx]` tag. Core code must not depend on an optional plugin's tests for its only coverage. Each new exported helper gets its own direct test, also when a filter test already runs it. +- **Pass with optional plugins turned off.** Every test passes when an optional plugin is not built. A test that needs filters from an optional plugin skips them when the filter list does not have them, as `test/PrebuiltPipelineTest.cpp` does for shipped pipelines. +- **Name a filter test after its plugin.** The test name starts with the plugin name and its first tag is the plugin: `TEST_CASE("ImageProcessing::AbsImageFilter: ...", "[ImageProcessing][AbsImageFilter]")`. When a filter moves to another plugin, change both. +- **Hide a test of a known defect.** A test that shows an open defect asserts the correct result, carries the hidden tags `[.][BlockedDefect]` and names the issue in a comment. The change that fixes the defect removes the hidden tags. +- **Use unequal dimensions and spacings** in tests of an algorithm that depends on the geometry: different cell counts and spacings along X, Y and Z, so that a swapped axis changes the result (knowledge base entry 23). - **No network access in CTest.** A test that needs a network service fails when the service is slow or not available. Do not register it with CTest. Keep the script so that it can be run by hand. - **Call `UnitTest::LoadPlugins();` only when the test looks up filters by UUID.** Loading the plugins fills the `Application`'s filter list. A test needs it when it calls `createFilter()` on the filter list, loads a pipeline with `Pipeline::FromJson()`, `FromFile()`, `FromSIMPLJson()` or `FromSIMPLFile()` (every SIMPL backwards-compatibility test does), or runs a filter whose code gets other filters from the filter list (`ReadH5Ebsd` does). A test that constructs a filter or algorithm class directly does not call it, whether it builds its data in memory or reads it with `UnitTest::LoadDataStructure()`. The call loads every plugin library and fails the test when one does not load, so an unneeded call makes the test fail for a plugin it does not use. A test that needs the plugins calls `LoadPlugins()` itself: ctest runs each `TEST_CASE` in its own process, so no other test has loaded them. - **Test an algorithm's own checks by calling the algorithm directly.** Test each failure of `ValidateInputs()` through `operator()()` and assert its error code; a test of `ValidateInputs()` alone does not show that the algorithm calls it. Such a failure returns before the dispatch, so its test does not use `UnitTest::AlgorithmTestScope`, which requires a branch to run. Test a check that is inside a dispatch branch on that branch. To reach an out-of-core branch in the in-core build, use a store whose `getStoreType()` returns `IDataStore::StoreType::OutOfCore` (the ImageProcessing tests do this). When that branch needs external sort or temporary record stores, register a test `IDataIOManager` that provides them, as described in the next item. diff --git a/docs/Coding_Knowledge_Base.md b/docs/Coding_Knowledge_Base.md index fb1253e791..49dbf07e3f 100644 --- a/docs/Coding_Knowledge_Base.md +++ b/docs/Coding_Knowledge_Base.md @@ -4,10 +4,11 @@ This file collects lessons learned from real defects in simplnx and its plugins. made, why it happened, how it was found, and the rule that prevents it. The audience is developers and the coding agents that work in this repository. Read the rules before you write or review code in the area they cover. -Add a new entry when a defect taught something that the code itself does not make obvious. Keep the entries short. -State the rule first, then the evidence. Cite pull requests and issues by number, but do not say whether they are -open, merged or closed: that changes, and the entry would go stale. A project convention that does not come from a -defect belongs in [the code style guide](Code_Style_Guide.md) instead. +Add a new entry when a defect taught something that the code itself does not make obvious. Keep the entries short. State +the rule first, then the evidence. Cite pull requests and issues by number, but do not say whether they are open, merged +or closed: that changes, and the entry would go stale. Refer to code by file and function or test name, not by line +number, because line numbers change with every edit; a line number belongs only in a link that is pinned to a commit. A +project convention that does not come from a defect belongs in [the code style guide](Code_Style_Guide.md) instead. ## Index @@ -30,6 +31,11 @@ defect belongs in [the code style guide](Code_Style_Guide.md) instead. 17. [Give a geometry without physical coordinates an unspecified length unit][kb-17] 18. [Apply the hexagonal axis correction to both hexagonal Laue classes][kb-18] 19. [Use the search grid only to find neighbor candidates](#19-use-the-search-grid-only-to-find-neighbor-candidates) +20. [Validate an in-place array by the object ID of its Attribute Matrix, not by its path][kb-20] +21. [Keep the legacy conversions when a filter moves or a plugin is retired][kb-21] +22. [Set a distinct modification time in a file-change test][kb-22] +23. [Test geometry formulas with unequal spacing along each axis][kb-23] +24. [Test a transfer failure after an earlier batch succeeded][kb-24] [kb-1]: #1-record-an-object-in-the-writer-id-map-only-when-the-written-datastructure-owns-it [kb-2]: #2-check-the-hdf5-file-close-before-you-report-a-successful-save @@ -46,6 +52,11 @@ defect belongs in [the code style guide](Code_Style_Guide.md) instead. [kb-16]: #16-check-the-binary-stl-file-size-before-you-use-attribute-byte-counts [kb-17]: #17-give-a-geometry-without-physical-coordinates-an-unspecified-length-unit [kb-18]: #18-apply-the-hexagonal-axis-correction-to-both-hexagonal-laue-classes +[kb-20]: #20-validate-an-in-place-array-by-the-object-id-of-its-attribute-matrix-not-by-its-path +[kb-21]: #21-keep-the-legacy-conversions-when-a-filter-moves-or-a-plugin-is-retired +[kb-22]: #22-set-a-distinct-modification-time-in-a-file-change-test +[kb-23]: #23-test-geometry-formulas-with-unequal-spacing-along-each-axis +[kb-24]: #24-test-a-transfer-failure-after-an-earlier-batch-succeeded --- @@ -137,7 +148,9 @@ parameter validation or an earlier check establishes the required type. Otherwis `nullptr`. **Pattern to use:** `ComputeSurfaceAreaToVolumeFilter::preflightImpl()` gets its inputs with -`getDataAs()` and checks each pointer. +`getDataAs()` and checks each pointer. A check that needs only the tuple count, such as a mask-size check, +reads the selection as `IArray` (`DBSCANFilter` does). A `StringArray` that the parameter accepted then gets the +tuple-count error instead of a `std::bad_cast`. Reading it as `IArray` does not make it a numeric array for execute. **Pattern to avoid:** rely on the numeric-type list of `ArraySelectionParameter` to establish `IDataArray`, or on `MultiArraySelectionParameter::AllowedDataTypes` to establish a concrete element type. @@ -523,6 +536,128 @@ Neighborhoods** found far fewer neighbors. PR #1652. --- +## 20. Validate an in-place array by the object ID of its Attribute Matrix, not by its path + +**Rule:** one DataObject can have several DataPaths. A filter that changes cell arrays in place checks that Feature Ids +and the ignored arrays it uses are children of the geometry's Cell Attribute Matrix by comparing object IDs, and checks +the ignored arrays only when the selected operation uses them. A Feature Attribute Matrix that a removal uses is a +direct child of the selected geometry and differs from its Cell Attribute Matrix. After validation, map the paths onto +the geometry's Cell Attribute Matrix path before the algorithm compares paths or finds sibling arrays. + +**Pattern to use:** `RemoveFlaggedFeaturesFilter` checks membership with `ValidateArrayInGeometryCellData()`, +`ValidateArraysInGeometryCellData()` and `ValidateFeatureAttributeMatrixInGeometry()`, then calls +`MapToGeometryCellDataPath()` (all in `simplnx/Utilities/DataArrayUtilities.hpp`). + +**Pattern to avoid:** compare the selected DataPaths as strings, or take the arrays to change from +`FeatureIdsArrayPath.getParent()` without checking that parent. + +**Why:** a path comparison misses an ignored array that was selected through another path to the same Attribute +Matrix, and an unrelated parent selects the wrong arrays to change. + +**How it showed up:** PR #1829 added the membership checks and the path mapping to the filters that change the cell +data of a geometry in place. + +**How to recognize it next time:** + +- A spatial in-place algorithm that compares selected paths or derives the arrays to change from the parent of the + selected Feature Ids. + +--- + +## 21. Keep the legacy conversions when a filter moves or a plugin is retired + +**Rule:** every legacy SIMPL filter UUID has one conversion owner. Several legacy UUIDs can map to the same current +filter. Each mapping has a fixture with non-default values, and its test checks the target filter UUID and the +converted arguments. When a filter or plugin is retired, move its conversions to the replacement first. + +**Pattern to use:** `src/Plugins/ImageProcessing/test/SimplConversionCoverageTest.cpp` reads the live conversion map, +requires a fixture for every entry, checks the target UUID, and rejects a conversion that produces only the default +arguments. + +**Pattern to avoid:** a copied table of expected mappings, or a test that checks only that a pipeline loads. + +**Why:** a missing mapping stops an old pipeline from converting, and a conversion that produces only the defaults +loses every saved value without an error. + +**How it showed up:** removing the retired image-processing plugin dropped 59 SIMPL conversions, and no test failed. +PR #1752 restored the mappings and added the coverage test. + +**How to recognize it next time:** + +- A change that removes or moves a filter or plugin without moving its conversion map and fixtures. + +--- + +## 22. Set a distinct modification time in a file-change test + +**Rule:** when a test rewrites a file and expects a size-and-modification-time check to notice the change, set a +modification time that differs from the cached one. + +**Pattern to use:** `test/Dream3dPreflightCacheTest.cpp`: + +```cpp +const auto cachedMtime = fs::last_write_time(filePath); +// ... rewrite the file ... +fs::last_write_time(filePath, cachedMtime - std::chrono::seconds(20)); +``` + +**Pattern to avoid:** rewrite the file and assume that its size or time changed. + +**Why:** HDF5 can give different contents the same file size, and the timestamp resolution of the file system can hide +a fast rewrite. + +**How it showed up:** the stale-file test of the preflight metadata cache needed an explicit timestamp to detect the +rewrite reliably. PR #1668. + +**How to recognize it next time:** + +- A cache-invalidation test that rewrites a file without setting its modification time. + +--- + +## 23. Test geometry formulas with unequal spacing along each axis + +**Rule:** test a formula that depends on axis spacing with unequal spacing, with a case along each axis, and with +expected values derived independently. Check a dimensionless result, such as sphericity, under uniform scaling. + +**Pattern to use:** the analytical pair and scale-invariant sphericity cases in +`src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp`. + +**Pattern to avoid:** test only with equal spacing, or use agreement between two implementations as the only check. + +**Why:** equal spacing hides a swapped axis, and two implementations can share the same error. + +**How it showed up:** **Compute Surface Area to Volume** used the wrong face weights for X- and Y-normal faces, which +is wrong only when the X and Y spacings differ. PR #1817, issues #1773 and #1780. + +**How to recognize it next time:** + +- A test of an axis-dependent formula whose fixtures all use equal spacing or a symmetric shape. + +--- + +## 24. Test a transfer failure after an earlier batch succeeded + +**Rule:** when an algorithm reads or writes in batches, inject a failure after at least one batch has succeeded. Assert +the original error, the values that the earlier batches wrote, and that the regions after the failure stay unwritten. + +**Pattern to use:** the "later metric I/O failures preserve prior batches" case in +`src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp`, which fails each operation of the second batch. + +**Pattern to avoid:** test only a failure of the first transfer, or assume that a failure undoes the earlier writes. + +**Why:** a later failure leaves the earlier batches written, and only a test of that state shows whether the error and +the partial output are what the caller expects. + +**How it showed up:** PR #1817 bounded the transfers of **Compute Surface Area to Volume** into batches and added these +failure cases. + +**How to recognize it next time:** + +- An algorithm that writes output before its last operation that can fail, with tests that fail only the first one. + +--- + ## Entry template Copy this block for a new entry, number it, and add it to the index. From bc6383d1a0a9e6719af6576ea3d6e63341574e15 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Wed, 28 Oct 2026 16:54:38 -0400 Subject: [PATCH 09/13] DOC: Add a knowledge-base rule for Catch2 file-time comparisons * Entry 25 says to give CHECK and REQUIRE a bool when a test compares std::filesystem::file_time_type values or file-clock durations. * Catch2 converts the operands to text, and libc++ then needs std::to_chars from macOS 13.3. The DREAM3D-NX macOS CI builds target macOS 11.7, so two tests in PR #1764 did not compile there. Signed-off-by: Joey Kleingers --- docs/Coding_Knowledge_Base.md | 36 +++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/docs/Coding_Knowledge_Base.md b/docs/Coding_Knowledge_Base.md index 49dbf07e3f..2aee43ecf2 100644 --- a/docs/Coding_Knowledge_Base.md +++ b/docs/Coding_Knowledge_Base.md @@ -36,6 +36,7 @@ project convention that does not come from a defect belongs in [the code style g 22. [Set a distinct modification time in a file-change test][kb-22] 23. [Test geometry formulas with unequal spacing along each axis][kb-23] 24. [Test a transfer failure after an earlier batch succeeded][kb-24] +25. [Give Catch2 a bool when you compare file times][kb-25] [kb-1]: #1-record-an-object-in-the-writer-id-map-only-when-the-written-datastructure-owns-it [kb-2]: #2-check-the-hdf5-file-close-before-you-report-a-successful-save @@ -57,6 +58,7 @@ project convention that does not come from a defect belongs in [the code style g [kb-22]: #22-set-a-distinct-modification-time-in-a-file-change-test [kb-23]: #23-test-geometry-formulas-with-unequal-spacing-along-each-axis [kb-24]: #24-test-a-transfer-failure-after-an-earlier-batch-succeeded +[kb-25]: #25-give-catch2-a-bool-when-you-compare-file-times --- @@ -658,6 +660,40 @@ failure cases. --- +## 25. Give Catch2 a bool when you compare file times + +**Rule:** in `CHECK` and `REQUIRE`, put an extra pair of parentheses around a comparison of +`std::filesystem::file_time_type` values or of file-clock durations, so that Catch2 gets one `bool`. + +**Pattern to use:** the legacy-file conversion case in `test/PreferencesSizeGBTest.cpp`: + +```cpp +// Evaluate to bool before Catch2 tries to stream libc++'s file-clock time point. +CHECK((fs::last_write_time(files.input) == modifiedBefore)); +``` + +**Pattern to avoid:** + +```cpp +CHECK(fs::last_write_time(files.input) == modifiedBefore); +``` + +**Why:** Catch2 splits `a == b` into its two operands and compiles code that converts each operand to text for the +failure message. For a file-clock value, libc++ formats a floating-point number, which needs `std::to_chars` from +macOS 13.3. The macOS CI builds of DREAM3D-NX set the deployment target to macOS 11.7, so the test does not compile +there. A local build with a newer deployment target compiles it without an error. + +**How it showed up:** `test/SelectiveImportReferencesTest.cpp` and `test/PreferencesSizeGBTest.cpp` did not compile on +the macOS CI jobs. PR #1764 fixed both. + +**How to recognize it next time:** + +- A `CHECK` or `REQUIRE` whose operand is `fs::last_write_time(...)`, a `file_time_type`, or a `clock::now() - ...` + duration. +- The compile error `'to_chars' is unavailable: introduced in macOS 13.3` in `__format/formatter_floating_point.h`. + +--- + ## Entry template Copy this block for a new entry, number it, and add it to the index. From 25ae69a0106d54bc523a19a54be1f19e03d2cd13 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Thu, 29 Oct 2026 16:26:34 -0400 Subject: [PATCH 10/13] DOC: Add feature-0, single-implementation and shared-error-code rules Style guide: feature 0 is not a feature, so code computes and stores nothing for it unless the user asks; the feature-id range check may run inside the algorithm's main pass; write one bulk-transfer implementation instead of in-core and out-of-core copies only when it is measured to be no slower in core; a test of one of several conditions that share an error code also asserts a phrase of that condition's message. AlgorithmDispatch.md: describe out-of-core storage without private names, show a copyIntoBuffer slice loop, state the ForceInCore precedence and correct the IdentifySample memory descriptions. Signed-off-by: Joey Kleingers --- docs/AlgorithmDispatch.md | 60 +++++++++++++++++++-------------------- docs/Code_Style_Guide.md | 16 +++++++++-- 2 files changed, 42 insertions(+), 34 deletions(-) diff --git a/docs/AlgorithmDispatch.md b/docs/AlgorithmDispatch.md index b5f64e16e0..c133919290 100644 --- a/docs/AlgorithmDispatch.md +++ b/docs/AlgorithmDispatch.md @@ -10,33 +10,30 @@ Some filter algorithms that perform well on in-memory data become extremely slow In-core arrays store data in a single contiguous memory buffer. Any element can be accessed in O(1) time via pointer arithmetic. Algorithms that use random access patterns (BFS flood fill, graph traversal, etc.) work efficiently because every `operator[]` call is a simple memory read. -### Out-of-Core Storage (ZarrStore) +### Out-of-Core Storage -Out-of-core arrays partition data into compressed chunks stored on disk. A fixed-size FIFO cache (6 chunks) keeps recently accessed chunks in memory. When code accesses an element, the system: - -1. Determines which chunk contains that element -2. If the chunk is cached, returns the value directly -3. If not cached, evicts the oldest chunk (writing it to disk if dirty) and loads the needed chunk from disk +Out-of-core arrays keep their data on disk in compressed chunks and hold a bounded number of decoded chunks in memory. When code reads or writes one element whose chunk is not in memory, the store loads and decodes that whole chunk, and it may first have to evict another chunk, writing it back when it was changed. This means algorithms with random access patterns cause **chunk thrashing**: each random jump may load a new chunk and evict another, turning O(1) memory reads into O(disk) operations. A BFS flood fill that visits neighbors across chunk boundaries can trigger thousands of chunk load/evict cycles, making a sub-second algorithm take 10+ minutes. -### The Solution: Chunk-Sequential Algorithms +### The Solution: Sequential Bulk Transfers -Algorithms designed for OOC data process chunks sequentially using the chunk API: +Algorithms designed for OOC data read and write contiguous ranges in order, through the bulk transfers of `AbstractDataStore` (`copyIntoBuffer()`, `copyFromBuffer()`, `readExtent()`), and keep only a bounded window in local buffers. A common form reads one XY slice at a time: ```cpp -for(uint64 chunkIdx = 0; chunkIdx < store.getNumberOfChunks(); chunkIdx++) +std::vector slice(dimX * dimY); +for(usize z = 0; z < dimZ; z++) { - store.loadChunk(chunkIdx); - auto lower = store.getChunkLowerBounds(chunkIdx); - auto upper = store.getChunkUpperBounds(chunkIdx); - for(z = lower[0]; z <= upper[0]; z++) - for(y = lower[1]; y <= upper[1]; y++) - for(x = lower[2]; x <= upper[2]; x++) { /* process voxel */ } + Result<> readResult = featureIdsStore.copyIntoBuffer(z * slice.size(), nonstd::span(slice.data(), slice.size())); + if(readResult.invalid()) + { + return readResult; + } + // ... process the slice from the local buffer ... } ``` -This pattern ensures sequential disk access and keeps the working set within the cache. However, chunk-sequential algorithms often require different data structures (e.g., union-find instead of BFS queues) and may use more auxiliary memory. +This pattern gives sequential disk access and a bounded working set. However, such algorithms often require different data structures (e.g., union-find instead of BFS queues) and may use more auxiliary memory. ## When to Use Algorithm Dispatch @@ -49,7 +46,7 @@ Use the dispatch pattern when: Do **not** use dispatch when: - The algorithm already uses sequential access patterns (a single algorithm with chunk-aware loops suffices) -- The only difference is minor (e.g., adding `loadChunk()` calls) rather than a fundamentally different approach +- The only difference is minor (e.g., reading in larger blocks) rather than a fundamentally different approach - The data is small enough that OOC overhead is negligible ## The DispatchAlgorithm Utility @@ -66,8 +63,7 @@ bool IsOutOfCore(const IArray& array); `IDataArray` uses its `IDataStore::StoreType`; `INeighborList` uses its storage-neutral `IListStore::isOutOfCore()` query. Other `IArray` types, such as `StringArray`, are conservatively in-memory until they expose a residency -contract. The private HDF5 list store overrides that query without adding a -SimplnxOoc type dependency to public simplnx. +contract. ### AnyOutOfCore @@ -76,7 +72,7 @@ bool AnyOutOfCore(std::initializer_list arrays); bool AnyOutOfCore(const AlgorithmArrayTargets& targets); ``` -Returns `true` if **any** of the given arrays uses OOC storage. Null pointers in the list are skipped. Use this when a filter operates on multiple input/output arrays and any one of them being OOC should trigger the chunk-sequential algorithm path. +Returns `true` if **any** of the given arrays uses OOC storage. Null pointers in the list are skipped. Use this when a filter operates on multiple input/output arrays and any one of them being OOC should trigger the out-of-core algorithm path. The original `IDataArray` initializer-list overload is retained for source compatibility, including named `std::initializer_list` @@ -90,10 +86,12 @@ arrays, which must remain alive for the call. ```cpp template Result<> DispatchAlgorithm(std::initializer_list arrays, ArgsT&&... args); + +template Result<> DispatchAlgorithm(const AlgorithmArrayTargets& targets, ArgsT&&... args); ``` -Checks whether any array in `arrays` uses OOC storage, or if the global `ForceOocAlgorithm()` flag is set. If either condition is true, constructs `OocAlgo(args...)` and calls `operator()()`. Otherwise, constructs `InCoreAlgo(args...)` and calls `operator()()`. +When the global `ForceInCoreAlgorithm()` flag is set, constructs `InCoreAlgo(args...)` and calls `operator()()`. Otherwise, when any array in `arrays` uses OOC storage or the global `ForceOocAlgorithm()` flag is set, constructs `OocAlgo(args...)` and calls `operator()()`; when neither is true, it constructs `InCoreAlgo(args...)`. See "ForceOocAlgorithm and ForceInCoreAlgorithm" below. For existing data-array-only callers, continue using the familiar form: @@ -121,7 +119,7 @@ The IdentifySample filter identifies the largest contiguous region of "good" vox ### IdentifySampleBFS (In-Core) Uses BFS flood fill with `std::vector` tracking arrays: -- Memory: 2 bits per voxel (checked + sample flags) +- Memory: 2 bits per voxel (checked + sample flags), plus an int64 queue that grows with the largest component - Access pattern: Random (BFS visits neighbors in all 6 directions) - In-core performance: Fast (random access is O(1) in memory) - OOC performance: Extremely slow (random neighbor access thrashes chunk cache) @@ -129,10 +127,10 @@ Uses BFS flood fill with `std::vector` tracking arrays: ### IdentifySampleCCL (Out-of-Core) Uses scanline Connected Component Labeling with union-find: -- Memory: 8 bytes per voxel (int64 label array) + union-find overhead -- Access pattern: Sequential (processes chunks in order, only checks backward neighbors) -- In-core performance: Good but uses more RAM than BFS -- OOC performance: Fast (sequential chunk access, no thrashing) +- Memory: two rolling slices of int64 labels plus the equivalence storage; a replay pass derives the labels again instead of storing one label per voxel +- Access pattern: Sequential (reads XY slices in Z order with `copyIntoBuffer()`, and checks only the x-1, y-1 and z-1 neighbors) +- In-core performance: Good, at the cost of an extra sequential read for the replay +- OOC performance: Fast (sequential slice reads instead of random neighbor access; the cost of the equivalence lookups depends on the input) ### Dispatcher Code @@ -163,7 +161,7 @@ Filters/Algorithms/ IdentifySampleBFS.hpp # BFS algorithm class declaration IdentifySampleBFS.cpp # BFS flood-fill implementation IdentifySampleCCL.hpp # CCL algorithm class declaration - IdentifySampleCCL.cpp # Chunk-sequential CCL implementation + IdentifySampleCCL.cpp # Slice-sequential CCL implementation IdentifySampleCommon.hpp # Shared code (VectorUnionFind, slice-by-slice functor) ``` @@ -171,10 +169,10 @@ Filters/Algorithms/ 1. **Identify the bottleneck**: Profile the filter in OOC mode. If it is orders of magnitude slower than in-core, random access patterns are likely the cause. -2. **Design the OOC algorithm**: Replace random access with chunk-sequential processing. Common techniques: +2. **Design the OOC algorithm**: Replace random access with sequential bulk transfers. Common techniques: - Scanline CCL with union-find instead of BFS flood fill - - Worklists sorted by chunk instead of global queues - - Multi-pass algorithms where each pass processes chunks sequentially + - Worklists sorted by position instead of global queues + - Multi-pass algorithms where each pass reads the data in order 3. **Create separate algorithm classes**: Both must share the same `InputValues` struct and constructor signature. Follow the existing pattern: ``` @@ -230,7 +228,7 @@ TEST_CASE("MyFilter", "[Plugin][MyFilter]") } ``` -`scope.executeFilter()` and `scope.execute()` set the force flags only for the call they wrap, so the filters that build the test data run normally. After the call, the scope requires that the selected algorithm path ran and the other path did not. The OOC algorithm class works correctly on in-core `DataStore` data because the chunk API methods are no-ops for `DataStore` (1 chunk spanning the full volume). +`scope.executeFilter()` and `scope.execute()` set the force flags only for the call they wrap, so the filters that build the test data run normally. After the call, the scope requires that the selected algorithm path ran and the other path did not. The OOC algorithm class also works on in-core `DataStore` data, because the bulk transfers work on allocated in-memory stores as well as on out-of-core stores. A placeholder store (`EmptyDataStore`) holds no values and rejects them. ## Performance Expectations diff --git a/docs/Code_Style_Guide.md b/docs/Code_Style_Guide.md index 16a73741d7..e3e0f0f9f4 100644 --- a/docs/Code_Style_Guide.md +++ b/docs/Code_Style_Guide.md @@ -217,6 +217,8 @@ Every message names both paths and both counts, as the `bluequartz-skills:error- A feature id, phase, face label or `NeighborList` entry used as an index into another array must be checked before the access. Preflight cannot see these values, so the check runs at the start of execute. For an unmasked 1-component `Int32Array`, call the overload of `ValidateFeatureIdsToFeatureAttributeMatrixIndexing()` that takes `shouldCancel`. That function checks every tuple and has no mask argument. For other inputs, scan the minimum and maximum and return an error. A filter with a mask checks only the ids that execute uses, with the same mask and sentinel rules as execute. A negative value is out of range unless the code skips negative values before it indexes. +The check can instead run inside the algorithm's main pass, when that pass reads each value before it uses it as an index and writes no output until the pass has ended. An invalid index then stops the filter with the output unchanged, and the algorithm saves a separate read of the whole input, which matters for out-of-core data. The check in the pass applies the same mask and sentinel rules as execute. + An index that comes from a parameter value (for example **Phase Index**) is checked in preflight, because preflight can see it. ### Validate Every Parameter Value in Preflight @@ -279,6 +281,8 @@ Preflight and execute must reach the same decision from the same inputs: which a A filter that dispatches between an in-core (Direct) algorithm and an out-of-core (Scanline) algorithm has two copies of its logic. Both must give the same output for the same input. A fix, a guard or an initial value in one path goes into the other path too, preferably through a shared helper. Test both paths (see Testing). +When the two algorithms would differ only in how they read and write the data, write one implementation that reads and writes through bulk transfers (or takes its I/O as a policy) instead of two copies of the computation, but only when that one implementation is not slower on in-core data than the in-core algorithm it replaces. Measure that in an in-core Release build and record the result in the pull request. When it is slower, keep the two algorithms and share their computation through helpers. + ### Build DataPaths From DataPaths Build each `DataPath` from the path that the user selected or that preflight created. Do not build a path from a name string or with string operations. @@ -306,6 +310,12 @@ Return every `Result` or merge it into the `Result` that the function returns. A - In execute, a loop over arrays, objects, tuples or chunks returns the first invalid `Result`, with the warnings collected so far. It does not collect one error for each item: when one cause (a full disk, a missing file) fails every item, hundreds of errors hide that cause. This applies also to a checked flush of a `DataStructure`. Preflight can merge the errors of independent checks. Execute can also merge a primary failure with the failure of the cleanup that follows it. - When several independent calls add errors to one list, collect them in a local `std::vector` and return `{nonstd::make_unexpected(std::move(errors))}` only when the list is not empty. Do not pass `result.errors()` of a valid `Result` as that list: a valid `Result` holds no error vector, so the call asserts in a Debug build and is undefined behavior in Release. +### Feature 0 Is Not a Feature + +Feature Id 0 marks cells that belong to no feature: mount material around the sample, or bad data. Code does not compute anything for feature 0 and does not store results in it, unless the user explicitly asked for that. A cell with Feature Id 0 does not take part in a calculation as a feature either: for example, it is not a neighbor feature. The feature-0 tuple of a per-feature output gets the output's default value (normally 0) without a calculation (see the next rule). + +A filter that does compute or store something for feature 0 because the user asked for it says so on its documentation page. A review flags every computation for feature 0, and every value stored for it, and asks the author what feature 0 means in that code. + ### Give Empty Features a Defined Output A feature with no cells or faces, and the reserved feature 0, get a defined value in every per-feature output. Do not compute 0/0 for them, and do not leave the value of an earlier feature in place. @@ -431,7 +441,7 @@ A refactoring keeps the error code of every condition whose meaning it does not - **Test every axis and plane** for code that has one block per axis or plane. - **Test error paths by code.** Assert the exact error code, not only `REQUIRE_NOTHROW` or an invalid result. - **Test degenerate and invalid inputs**: empty geometries, out-of-range feature ids, NaN and limit values (`2^63`, `2^64`, `-1` and `256` for `uint8`), inputs in a `DataGroup` instead of at the root, and arrays from a different Attribute Matrix. -- **Tell apart conditions that share an error code.** Assert the code of the store that owns the condition: -6035 for a `DataStore`, `StringStore` or `ListStore` resize failure, and -10603 (`EmptyDataStore::k_ResizeError`) for an `EmptyDataStore` resize failure. When a contract gives several conditions the same code, the test also asserts a phrase that only the message of its condition contains. +- **Tell apart conditions that share an error code.** When several conditions return the same code, a test of one of them also asserts a phrase that only the message of that condition contains; a test that asserts only the shared code cannot show which check fired. Assert the code of the store that owns the condition: -6035 for a `DataStore`, `StringStore` or `ListStore` resize failure (in `DataStore`, the tuple, element and byte count checks and an allocation failure all return it), and -10603 (`EmptyDataStore::k_ResizeError`) for an `EmptyDataStore` resize failure. - **Print the errors of a failed Result.** Use `SIMPLNX_RESULT_REQUIRE_VALID(result)` instead of `REQUIRE(result.valid())`. It prints each error code and message when the check fails. - **Apply an action in both modes.** An `IDataAction` creates different objects in preflight and in execute. Test it with `GENERATE(IDataAction::Mode::Preflight, IDataAction::Mode::Execute)` and an `INFO` that names the mode. - **Test a size failure without the allocator.** The nightly `asan` job runs with the default AddressSanitizer options. A request above the AddressSanitizer limit (1 TiB on 64-bit platforms) stops the test process instead of throwing `std::bad_alloc` (#1885). Use a size that the code rejects before it allocates, such as a shape product that overflows. Test a recoverable allocation failure through a deterministic failure seam. Test an intentional allocator abort in a separate process, with a timeout and a check of its exit result. An AddressSanitizer build alone does not make a real allocation failure recoverable. @@ -441,8 +451,8 @@ A refactoring keeps the error code of every condition whose meaning it does not - **Hide a test of a known defect.** A test that shows an open defect asserts the correct result, carries the hidden tags `[.][BlockedDefect]` and names the issue in a comment. The change that fixes the defect removes the hidden tags. - **Use unequal dimensions and spacings** in tests of an algorithm that depends on the geometry: different cell counts and spacings along X, Y and Z, so that a swapped axis changes the result (knowledge base entry 23). - **No network access in CTest.** A test that needs a network service fails when the service is slow or not available. Do not register it with CTest. Keep the script so that it can be run by hand. -- **Call `UnitTest::LoadPlugins();` only when the test looks up filters by UUID.** Loading the plugins fills the `Application`'s filter list. A test needs it when it calls `createFilter()` on the filter list, loads a pipeline with `Pipeline::FromJson()`, `FromFile()`, `FromSIMPLJson()` or `FromSIMPLFile()` (every SIMPL backwards-compatibility test does), or runs a filter whose code gets other filters from the filter list (`ReadH5Ebsd` does). A test that constructs a filter or algorithm class directly does not call it, whether it builds its data in memory or reads it with `UnitTest::LoadDataStructure()`. The call loads every plugin library and fails the test when one does not load, so an unneeded call makes the test fail for a plugin it does not use. A test that needs the plugins calls `LoadPlugins()` itself: ctest runs each `TEST_CASE` in its own process, so no other test has loaded them. -- **Test an algorithm's own checks by calling the algorithm directly.** Test each failure of `ValidateInputs()` through `operator()()` and assert its error code; a test of `ValidateInputs()` alone does not show that the algorithm calls it. Such a failure returns before the dispatch, so its test does not use `UnitTest::AlgorithmTestScope`, which requires a branch to run. Test a check that is inside a dispatch branch on that branch. To reach an out-of-core branch in the in-core build, use a store whose `getStoreType()` returns `IDataStore::StoreType::OutOfCore` (the ImageProcessing tests do this). When that branch needs external sort or temporary record stores, register a test `IDataIOManager` that provides them, as described in the next item. +- **Call `UnitTest::LoadPlugins();` only when the test looks up filters by UUID.** Loading the plugins fills the `Application`'s filter list. A test needs it when it calls `createFilter()` on the filter list, loads a pipeline with `Pipeline::FromJson()`, `FromFile()`, `FromSIMPLJson()` or `FromSIMPLFile()` (every SIMPL backwards-compatibility test does), or runs a filter whose code gets other filters from the filter list (`ReadH5Ebsd` does). A test that constructs a filter or algorithm class directly does not call it, whether it builds its data in memory or reads it with `UnitTest::LoadDataStructure()`. A test of real out-of-core stores does not call it for the storage either: the out-of-core test setup registers the storage before the first test runs. The call loads every plugin library and fails the test when one does not load, so an unneeded call makes the test fail for a plugin it does not use. A test that needs the plugins calls `LoadPlugins()` itself: ctest runs each `TEST_CASE` in its own process, so no other test has loaded them. +- **Test an algorithm's own checks by calling the algorithm directly.** Test each failure of `ValidateInputs()` through `operator()()` and assert its error code; a test of `ValidateInputs()` alone does not show that the algorithm calls it. A test through the filter cannot reach most of these checks, because `IFilter::preflight()` validates the parameters first and returns their errors (for example -204 for a selected array that does not exist) before `preflightImpl()` runs. That is expected: `ValidateInputs()` keeps every check because other code runs the algorithm without the filter, so test the checks through the algorithm, not through the filter. Such a failure returns before the dispatch, so its test does not use `UnitTest::AlgorithmTestScope`, which requires a branch to run. Test a check that is inside a dispatch branch on that branch. To reach an out-of-core branch in the in-core build, use a store whose `getStoreType()` returns `IDataStore::StoreType::OutOfCore` (the ImageProcessing tests do this). When that branch needs external sort or temporary record stores, register a test `IDataIOManager` that provides them, as described in the next item. - **Restore the global I/O state.** A test that registers an `IDataIOManager` changes the `DataIOCollection` for the rest of the process. `DataIOCollection` has no remove call, so replace an existing manager under its format name and re-add the saved original in a destructor, which also runs when an assertion fails (see `ScopedImportSelectionManager` in `test/Dream3dLoadingApiTest.cpp`). Do not rely on ctest running each `TEST_CASE` in its own process: the test executable run directly runs many cases in one process, and a leaked manager makes later tests fail or skip their checks. ## Build Configuration From 0490e9123b0c825d223d7e7e35f618b9591e59d2 Mon Sep 17 00:00:00 2001 From: Michael Jackson Date: Fri, 30 Oct 2026 10:05:36 -0400 Subject: [PATCH 11/13] Cleaned up and organized the 'docs' directory Signed-off-by: Michael Jackson --- Code_Review/vv_work/OoC_VV_Review.md | 173 -- docs/AnacondaPackages.md | 27 +- docs/{ => deprecated}/CMakeLists.txt | 0 docs/{ => deprecated}/Doxyfile.in | 0 docs/{ => deprecated}/conf.py | 0 docs/documentation_review_tracker.md | 1950 ----------------- .../ebsd_segment_grains.d3dstate | 429 ---- .../ebsd_segment_ipf.d3dstate | 429 ---- .../ebsd_segment_pilot.d3dpipeline | 130 -- docs/filter-figure-style.md | 2 +- .../2026-04-12-documentation-review-design.md | 327 --- docs/upstream-filter-sources.md | 73 - 12 files changed, 4 insertions(+), 3536 deletions(-) delete mode 100644 Code_Review/vv_work/OoC_VV_Review.md rename docs/{ => deprecated}/CMakeLists.txt (100%) rename docs/{ => deprecated}/Doxyfile.in (100%) rename docs/{ => deprecated}/conf.py (100%) delete mode 100644 docs/documentation_review_tracker.md delete mode 100644 docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_grains.d3dstate delete mode 100644 docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_ipf.d3dstate delete mode 100644 docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_pilot.d3dpipeline delete mode 100644 docs/superpowers/specs/2026-04-12-documentation-review-design.md delete mode 100644 docs/upstream-filter-sources.md diff --git a/Code_Review/vv_work/OoC_VV_Review.md b/Code_Review/vv_work/OoC_VV_Review.md deleted file mode 100644 index d1a5d8b45a..0000000000 --- a/Code_Review/vv_work/OoC_VV_Review.md +++ /dev/null @@ -1,173 +0,0 @@ -The independent V&V Catch2 assertions remain preserved. Sections 4.2 and 4.3 are closed: all 40 audited filters now have representative, output-sensitive real-HDF5 boundary evidence. Further semantic-test and report-review items are recorded separately below. - -## Initial audit results (historical) - -- Full in-core build: passed. -- Full OOC build: passed. -- In-core CTest: 503/503 passed in 63.82 seconds. -- OOC CTest: 530/531 passed in 286.30 seconds. -- Hidden real-OOC contracts passed: - - FeatureSizes: 3 cases / 129 assertions - - CopyFeatureArray and MultiThreshold store tests: 5 cases / 368 assertions - - GroupMicroTextureRegions 200³: 642 assertions - - CreateFeatureArrayFromElementArray 200³: 3,495 assertions - - MultiThresholdObjects 200³: 11 assertions - -The initial audit changed no files. The subsequent Section 1 remediation is retained in the current branch. The Section 4.2 tests, reports, and CTest registration are local working-tree changes. - -## Five-gate assessment - -1. Oracle preservation [FIXED] - -All 40 filters retain their independent-oracle assertions. No oracle tolerance using Approx().margin() or epsilon() was removed or relaxed. - -Two reorganizations were confirmed rather than counted as losses: - -- [X] MultiThresholdObjects’ V&V cases moved verbatim into MultiThresholdObjectsVvTest.cpp, gaining dispatch witnesses. -- [X] NeighborOrientationCorrelation’s Small IN100 Class-4 test is restored to the public dual-build suite, and its duplicate private implementation is removed. The latest focused runs pass in-core in 4.41 seconds and with explicitly asserted HDF5-OOC storage in 202.30 seconds. The complete selections pass 19/19 in-core and 20/20 OOC; the OOC selection retains the separate 200³ boundary case. - -2. Dual-build runtime [FIXED] - -The initial in-core suite was clean. The initial OOC suite had one deterministic failure, since corrected by reapplying the guard fix: - -- /Users/mjackson/Workspace5/SimplnxOoc/test/PluginTests/OrientationAnalysis/ComputeAvgOrientationsOocTest.cpp:1083 segfaults in the “in-range - feature without a selected operation” section. - -- Isolated reproduction exits 139, with 22/23 assertions passing before SIGSEGV. -- src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp:1182 dereferences featureOps[feature] - before the intended -54677 validation. - -- Local fix commit 04292a66a moves that guard ahead of the dereference. It was reapplied as 180c2cf05; the focused selections subsequently passed 15/15 in-core and 22/22 OOC. - -This was an observable crash regression, not an oracle-output mismatch or flaky test. - -3. Material algorithm implementations [PASS] - -Twelve filters have distinct Direct/Scanline, BFS/CCL, Worklist/Scanline, or manual in-core/OOC implementations. All twelve execute an oracle- -backed test through runtime dispatch witnesses: - -BadDataNeighborOrientationCheck, ComputeAvgOrientations, ComputeIPFColors, ComputeKernelAvgMisorientations, ComputeFeatureNeighbors, -ComputeFeatureSizes, CopyFeatureArrayToElementArray, DBSCAN, FillBadData, IdentifySample, MultiThresholdObjects, and RotateSampleRefFrame. - -The normal oracle comparisons pass for both implementations. The ComputeAvgOrientations guard crash above is fixed. - -4. Real-HDF5 boundary evidence - -4.1: Confirmed meaningful boundary/full-scale evidence exists for 40 of 40: [PASS] - -- ComputeAvgCAxes — 4,097-cell real-HDF5 fixture with a 4,096-tuple block plus one-cell tail and literal +30° output. -- BadDataNeighborOrientationCheck — real-HDF5 Bool/UInt8 masks propagate backward across five 3×3 slices; exact output masks and Scanline dispatch are asserted. -- ComputeFeatureCentroids — real-HDF5 block-plus-tail contributions yield exact physical centroids with non-unit spacing and origin; periodic off/on pass. -- ComputeFeatureNeighbors — existing 5×5×5 analytical fixture runs on HDF5 FeatureIds and all five OOC outputs across five 25-cell slices. -- ComputeGroupingDensity — HDF5 neighbor-list chunks straddle features 2/3; all four option combinations retain exact union-volume ratios and feature claims. -- ComputeIPFColors — Bool/UInt8 HDF5 masks and exact black/red/green colors cross a 65,536-cell block and partial tail. -- ComputeNeighborhoods — HDF5 output list chunks straddle features 2/3; both radius modes retain exact directed neighbors and counts. -- ComputeSchmids — physical 65,536-quaternion chunk plus one-feature tail gives analytical factors 0.5/0.48 and distinct poles. -- ConvertOrientations — 4,096-quaternion block plus one-tuple tail gives stereographic [0,0,1/3] and [1/3,0,0]. -- FillBadData — connected defect spans slices at the retention threshold; an isolated cell copies the correct feature, phase, and payload, with new-phase option off/on. -- ReadAngData — 65,537 input rows preserve exact Euler interleaving, scalar values, and phase-zero remapping into HDF5 output. -- ReadCtfData — 65,537 input rows preserve exact double-intermediate Euler conversions and unindexed phase zero across the block tail. -- RegularizeZSpacing — HDF5 plane copies implement the analytical [0,1,2,2,3] source-plane map in new-geometry and in-place modes. -- RotateEulerRefFrame — HDF5 full-block end and one-tuple tail preserve independent identity/wraparound Euler rotation values. -- RotateSampleRefFrame — full output permutation crosses a 1 MiB input page and partial second page in both whole-volume and slice modes. -- WriteDREAM3D — raw HDF5 readback preserves every uint32 value across a physical 1 MiB input chunk and tail, with compression off/on. -- WritePoleFigure — two orientations straddle the extraction block; exact discrete family counts and a full analytical <001> image verify both contributions. -- ComputeCAxisLocations — 65,537-cell real-HDF5 fixture with distinct literal outputs at tuple 65,535 and the one-cell tail. -- ComputeFaceIPFColoring — 65,537-face real-HDF5 fixture with distinct exact cubic-red and hex-green outputs at tuple 65,535 and the one-face tail. -- ComputeFeatureFaceMisorientations — 65,537-face real-HDF5 fixture with exact 15-degree and 30-degree outputs at tuple 65,535 and the one-face tail. -- ComputeFeatureNeighborCAxisMisalignments — 65,537 features; verified 65,536-tuple quaternion chunk shape and independent 15-degree/30-degree pair and average results across the tail. -- ComputeFeatureNeighborMisorientations — 65,537 features; verified 65,536-tuple quaternion chunk shape and independent cubic 15-degree/30-degree pair and average results across the tail. -- ComputeFeatureReferenceCAxisMisorientations — 17×19×4 real-HDF5 cells; contributors across 323-cell slice buffers verify cell angles, mean, and population standard deviation. -- ComputeFeatureReferenceMisorientations — both modes cross a 65,536-cell block plus one-cell tail; analytical angles, means, and tied-center coordinates pass. -- CopyFeatureArrayToElementArray — 65,537 real-HDF5 cells; literal three-component tuples verify the full-block end and one-cell tail. -- IdentifySample — 129×129×3 HDF5 mask creates over 8,000 labels, crosses 4,096-record equivalence pages and rolling slices, and checks the full expected mask with hole filling off/on. -- ReadH5OinaData — two 65,537-point scans; exact values in all nine HDF5 cell arrays verify batch tails, scan offsets, phase widening, and hexagonal alignment. -- DBSCAN — 65,538 real-HDF5 points; adjacent core grids span the 65,536-point batch boundary and two-point tail. The complete expected cluster-label array passes. -- CAxisSegmentFeatures -- ComputeAvgOrientations -- ComputeKernelAvgMisorientations -- GroupMicroTextureRegions -- NeighborOrientationCorrelation -- ComputeFeaturePhases -- ComputeFeatureSizes -- CreateFeatureArrayFromElementArray -- ErodeDilateBadData -- MultiThresholdObjects -- ReplaceElementAttributesWithNeighborValues -- RequireMinNumNeighbors - -4.2: Closed for all 12 filters in this group (2026-09-18). ComputeAvgCAxes was the first case; all 11 subsequent cases now have an independent, output-sensitive HDF5 block, chunk, page, or slice-boundary regression. [PASS] - -| Filter | In-core CTest | OOC CTest | New hidden boundary case | -|--------|---------------|-----------|--------------------------| -| ComputeAvgCAxes | 5/5 | 5/5 | 38 assertions | -| ComputeCAxisLocations | 7/7 | 7/7 | 25 assertions | -| ComputeFaceIPFColoring | 5/5 | 5/5 | 38 assertions | -| ComputeFeatureFaceMisorientations | 3/3 | 3/3 | 25 assertions | -| ComputeFeatureNeighborCAxisMisalignments | 7/7 | 7/7 | 37 assertions | -| ComputeFeatureNeighborMisorientations | 7/7 | 7/7 | 37 assertions | -| ComputeFeatureReferenceCAxisMisorientations | 8/8 | 8/8 | 34 assertions | -| ComputeFeatureReferenceMisorientations | 9/9 | 9/9 | 63 assertions; both reference modes | -| CopyFeatureArrayToElementArray | 24/24 | 24/24 | 22 assertions | -| IdentifySample | 5/5 | 6/6 | 99,875 assertions; hole filling off/on | -| ReadH5OinaData | 20/20 | 21/21 | 166 assertions; all nine imported cell arrays | -| DBSCAN | 12/12 | 19/19 | 65,555 assertions; ExternalGDCF runtime witness | - -The per-filter counts exclude the hidden tests. The hidden tests require the HDF5-OOC manager and are included in the OOC-only `OrientationAnalysisOocStoreContracts` and `SimplnxCoreOocStoreContracts` CTest entries. The latter also contains three pre-existing contracts. Run them serially: - -```sh -ctest --test-dir /Users/mjackson/Workspace5/DREAM3D-Build/NX-Com-Qt69-Vtk96-OoC-Rel -j 1 --output-on-failure -R '^(OrientationAnalysis|SimplnxCore)OocStoreContracts$' -``` - -The in-core build is `NX-Com-Qt69-Vtk96-Rel`. The OOC build is `NX-Com-Qt69-Vtk96-OoC-Rel`. Final combined CTest logs are under each build's `Testing/Section42-20260918.log`. - -Final combined serial selections: **112/112 passed in-core (18.87 s)** and **123/123 passed OOC (162.31 s)**. The OOC total includes both contract groups, so all 12 new boundary tests are covered through CTest. The OrientationAnalysis group passed all nine cases in 144.61 s. Both test targets built successfully in both configurations. - -No production filter algorithm changed in this campaign. Existing oracle values and tolerances remain intact. The reference-misorientation scaffold now bulk-initializes the same zero/identity values to keep the large HDF5 fixture inexpensive. Reports retain their historical statuses and sign-offs and use the report template's dashboard, coverage, inventory, and defect sections. - -Final preservation check: all 91 upstream/develop test-case declarations in the 12 touched test files remain present. The removed result-macro lines in the branch diff are equivalent named-result wrappers; no numerical oracle assertion or tolerance was removed or relaxed. All 12 reports contain the template's eight dashboard rows and required main sections. Their status and sign-off rows match HEAD exactly. Pinned clang-format 19.1.1 and `git diff --check` pass. - -Additional semantic gaps are recorded, without expanding this boundary campaign: the two feature-neighbor filters can reuse their smallest analytical pair fixtures with optional averaging disabled, check unchanged neighbor lists, and assert that no average array is created. These are separate from the now-closed Section 4.2 storage-boundary gaps. - -4.3: Closed for all 16 filters (2026-09-21). The tests were handled in the listed order. Existing Section 4.2 working-tree changes are preserved. - -| Filter | In-core CTest | OOC CTest | New hidden case assertions | -|--------|---------------|-----------|----------------------------| -| BadDataNeighborOrientationCheck | 33/33 | 33/33 | 139 (two mask types) | -| ComputeFeatureCentroids | 5/5 | 5/5 | 35 | -| ComputeFeatureNeighbors | 37/37 | 37/37 | 73 | -| ComputeGroupingDensity | 7/7 | 7/7 | 143 | -| ComputeIPFColors | 9/9 | 9/9 | 87 | -| ComputeNeighborhoods | 5/5 | 5/5 | 79 | -| ComputeSchmids | 6/6 | 6/6 | 37 | -| ConvertOrientations | 5/5 | 5/5 | 16 | -| FillBadData | 15/15 | 15/15 | 1,053 | -| ReadAngData | 8/8 | 8/8 | 36 | -| ReadCtfData | 12/12 | 12/12 | 38 | -| RegularizeZSpacing | 5/5 | 5/5 | 67 | -| RotateEulerRefFrame | 5/5 | 5/5 | 18 | -| RotateSampleRefFrame | 12/12 | 12/12 | 528,425 | -| WriteDREAM3D | 22/22 | 22/22 | 27 | -| WritePoleFigure | 4/4 | 4/4 | 29 | - -The 16 additions instantiate 17 hidden cases because BadDataNeighborOrientationCheck covers both Bool and UInt8. They are included in the existing OOC-only store-contract groups. The final serial selection passes **190/190 in-core (30.69 s)** and **192/192 OOC (143.12 s)**; the OOC total includes both groups and therefore all new HDF5 cases. Logs are `Testing/Section43-20260921.log` in the two DREAM3DNX build trees. - -Both test targets compile in both builds. All **186 test declarations from the original pre-OOC baseline** and all **192 declarations from the current local upstream/develop reference** in the 16 touched files remain present. No pre-existing REQUIRE/CHECK/Approx assertion was removed by this campaign. Historical DREAM3D reader-result checks moved into the checked UnitTest loader or separate pipeline/DataStructure Result checks; their numerical comparisons remain intact. The apparent IPF red-corner assertion change is comment-only. No production filter or algorithm changed. All 16 reports retain their exact historical status and sign-off rows. - -Boundary evidence uses the implementation's relevant transfer size: numeric blocks and partial tails, rolling image slices, HDF5 NeighborList chunks, or bounded source-cache pages. These are correctness regressions, not cache-eviction performance benchmarks. New expected values come from existing independent fixtures or explicit analytical constructions, never from observed filter output. Existing legacy deviations are retained; no fresh legacy binary comparison is claimed. - -There are **no remaining Section 4.3 storage gaps**. The two optional-average semantic regressions identified under Section 4.2 remain possible follow-ups. - -5. Observable changes and deviations - -- The initial audit removed or changed no V&V report or deviation sidecar. The Section 4.2 and 4.3 campaigns update 28 reports with current inventories and boundary evidence; sidecars are retained. -- No covered numerical output discrepancy was found. -- ComputeAvgOrientations' initial crash is corrected as described above. -- The reference-c-axis report now cites D2 for its precision deviation, matching the existing sidecar; its former D4 cross-reference was stale. -- Historical legacy comparison results were retained; this campaign does not claim fresh legacy binary runs. The DBSCAN report now distinguishes its local legacy proof build from a stock DREAM3D 6.5.171 comparison. - -- MultiThresholdObjects needs a report inventory refresh for its relocated and renamed tests. - -The initial audit used tip e6ca39e1c and baseline 286dfa90b. Section 4.2 used HEAD efb036625. Section 4.3 runs on develop at squashed HEAD 271a72192, whose committed tree is identical, plus local test/report changes. The original pre-OOC comparison baseline is 3e026500e7d8cf8e0debedd9a249ce887f0ad334. The current local upstream/develop reference is 4ec21e93f2d848159905aca95cde037c1ce0db08; its committed tree also matches HEAD. No remote fetch was performed. - -Recommended next step: review the remaining meaningful semantic gaps and the MultiThresholdObjects report inventory. Storage-boundary work in Sections 4.2 and 4.3 is complete for all 40 filters; historical sign-offs have not been renewed by this test campaign. diff --git a/docs/AnacondaPackages.md b/docs/AnacondaPackages.md index a25708fe22..1bc1a798e1 100644 --- a/docs/AnacondaPackages.md +++ b/docs/AnacondaPackages.md @@ -1,19 +1,12 @@ # Building Anaconda Packages -## IMPORTANT NOTE ABOUT PACKAGING TIMES - -These builds can on the order of **multiple hours** to complete on x86 systems that predate Intel 13th Generation processors. Be patient. The build looks like it is stuck but python is just taking it's time to recursively figure out the needed DLLs because ITK and VTK create a ton of DLL libraries. This is especially slow on Windows systems. - ## Python Variations -- 3.9 -- 3.10 - 3.11 - 3.12 ## Operating System Variations -- MacOS x64 - MacOS Arm64 - Windows x64 - Linux x64 @@ -24,9 +17,9 @@ Use a release of Anaconda after NOV 2023. This is because the `conda` program no ```shell # Create the Python Virtual Environment -(base) [user@host] $ conda create --name nx-build python=3.10 conda-build -(base) [user@host] $ conda activate nx-build -(nx-build) [user@host] $ cd DREAM3DNX/conda +(base) [user@host] $ conda create --name nxbuild python=3.10 conda-build +(base) [user@host] $ conda activate nxbuild +(nxbuild) [user@host] $ cd DREAM3DNX/conda ``` if you do a `conda list` from this new environment you should get the following versions: @@ -35,17 +28,3 @@ if you do a `conda list` from this new environment you should get the following conda 23.11.0 py310hbe9552e_1 conda-forge conda-build 3.28.2 py310hbe9552e_0 conda-forge ``` - -If you are on an operating system that requires `mambabuild` you will also need to run the following command inside the `nx-build` virtual environment. - -```shell -(nx-build) [user@host] $ conda install mamba boa -``` - -| Operating System | Build Command | -|------------------|---------------| -| MacOS Arm64 | conda build . | -| MacOS x64 | conda mambabuild | -| Windows x64 (10/11) | conda build . | -| Linux x64 (Ubuntu 22.04) | conda mambabuild . | - diff --git a/docs/CMakeLists.txt b/docs/deprecated/CMakeLists.txt similarity index 100% rename from docs/CMakeLists.txt rename to docs/deprecated/CMakeLists.txt diff --git a/docs/Doxyfile.in b/docs/deprecated/Doxyfile.in similarity index 100% rename from docs/Doxyfile.in rename to docs/deprecated/Doxyfile.in diff --git a/docs/conf.py b/docs/deprecated/conf.py similarity index 100% rename from docs/conf.py rename to docs/deprecated/conf.py diff --git a/docs/documentation_review_tracker.md b/docs/documentation_review_tracker.md deleted file mode 100644 index 5e965436e3..0000000000 --- a/docs/documentation_review_tracker.md +++ /dev/null @@ -1,1950 +0,0 @@ -# DREAM3DNX Filter Documentation Review Tracker - -**Design Spec:** [docs/superpowers/specs/2026-04-12-documentation-review-design.md](superpowers/specs/2026-04-12-documentation-review-design.md) -**Style Guide:** [docs/style_palette_final.svg](style_palette_final.svg) -**Branch:** topic/documentation_update (will submit as single PR when complete) - ---- - -## Progress Summary - -| Batch | Category | Total Filters | Triaged | Rewritten | Status | -|-------|----------|---------------|---------|-----------|--------| -| 1 | Orientation / Crystallography Statistics | 17 | 17 | 17 | Complete | -| 2 | Alignment Filters | 5 | 5 | 5 | Complete | -| 3 | Segmentation / Feature Identification | 10 | 10 | 10 | Complete | -| 4 | Neighbor / Kernel Operations | 11 | 11 | 11 | Complete | -| 5 | Geometry Creation / Manipulation | 15 | 15 | 15 | Complete | -| 6 | Data Manipulation (Copy, Create, Delete, Rename) | 21 | 21 | 21 | Complete | -| 7 | I/O Filters (Read/Write) | 44 | 44 | 44 | Complete | -| 8 | Image Processing (ITK wrappers) | 88 | 88 | 88 | Complete | -| 9 | Remaining SimplnxCore filters | 73 | 73 | 73 | Complete | - ---- - -## Batch 1: Orientation / Crystallography Statistics - -**Plugin:** OrientationAnalysis -**Filters:** 17 (15 crystallography + 2 morphological) - -### Tier 1 — Critical - -| Filter | Plugin | Status | -|--------|--------|--------| -| ComputeAvgCAxes | OrientationAnalysis | Done | -| ComputeCAxisLocations | OrientationAnalysis | Done | -| ComputeGBCD | OrientationAnalysis | Done | -| ComputeSchmids | OrientationAnalysis | Done | - -- [x] **ComputeAvgCAxesFilter** (OrientationAnalysis) - - **Clarity:** Assumed deep quaternion/transform knowledge; no explanation of what C-axis is - - **Figures Needed:** Hexagonal unit cell with C-axis labeled - - **Real-World Viz:** IPF color map showing C-axis orientations - - **Concept Links:** C-axis, hexagonal materials, quaternions, reference frames - - **Changes Made:** Added "What is the C-Axis?" section with example materials, plain-language algorithm, hexagonal-only explanation. Added hexagonal unit cell figure. - -- [x] **ComputeCAxisLocationsFilter** (OrientationAnalysis) - - **Clarity:** Very sparse, just restated algorithm - - **Figures Needed:** Same hexagonal C-axis figure (shared with ComputeAvgCAxes) - - **Real-World Viz:** C-axis direction map colored by orientation - - **Concept Links:** C-axis, hexagonal materials, quaternions, reference frames - - **Changes Made:** Added "What is the C-Axis?" section, upper-hemisphere convention note, comparison with ComputeAvgCAxes to help users pick the right filter. Shared hexagonal unit cell figure. - -- [x] **ComputeGBCDFilter** (OrientationAnalysis) - - **Clarity:** 2-sentence stub; non-expert cannot understand what GBCD is - - **Figures Needed:** Conceptual diagram of 5D boundary space (deferred -- complex) - - **Real-World Viz:** GBCD pole figure output screenshot - - **Concept Links:** grain boundaries, misorientation, boundary normals, MRD units - - **Changes Made:** Added "What is the GBCD?" section explaining 5 parameters, MRD units, algorithm steps, resolution parameter. Added Rohrer citations [1][2]. Cross-reference to metric-based filter. - -- [x] **ComputeSchmidsFilter** (OrientationAnalysis) - - **Clarity:** Formula presented but no explanation of slip systems or physical meaning - - **Figures Needed:** Diagram showing tensile axis, slip plane, slip direction, and angles φ/λ - - **Real-World Viz:** Microstructure colored by Schmid factor - - **Concept Links:** slip systems, crystal plasticity, loading axis, CRSS - - **Changes Made:** Added "What is the Schmid Factor?" section with plain-language slip explanation, 0-0.5 range meaning, 4-step algorithm, CRSS limitation note. Added Schmid factor geometric diagram. - -### Tier 2 — Important - -| Filter | Plugin | Status | -|--------|--------|--------| -| ComputeAvgOrientations | OrientationAnalysis | Done | -| ComputeBoundaryStrengths | OrientationAnalysis | Done | -| ComputeFeatureNeighborCAxisMisalignments | OrientationAnalysis | Done | -| ComputeFeatureNeighborMisorientations | OrientationAnalysis | Done | -| ComputeFeatureReferenceCAxisMisorientations | OrientationAnalysis | Done | -| ComputeFeatureReferenceMisorientations | OrientationAnalysis | Done | -| ComputeSlipTransmissionMetrics | OrientationAnalysis | Done | -| ComputeTwinBoundaries | OrientationAnalysis | Done | -| ComputeShapes | OrientationAnalysis | Done | - -- [x] **ComputeAvgOrientationsFilter** (OrientationAnalysis) - - **Clarity:** Comprehensive but very math-heavy; 3 methods with dense statistical notation - - **Figures Needed:** Diagram showing orientations being averaged within a grain; vMF distribution concept - - **Real-World Viz:** IPF color map showing average orientation assignment - - **Concept Links:** quaternions, Fundamental Zone, symmetry operators, Euler angles, EM algorithm - - **Changes Made:** Added introductory paragraph explaining what average orientation means and why it matters for downstream filters. Existing method descriptions were already thorough; kept intact. - -- [x] **ComputeBoundaryStrengthsFilter** (OrientationAnalysis) - - **Clarity:** Relies on citations for metric definitions; readers need access to papers - - **Figures Needed:** Slip transmission across grain boundary concept diagram - - **Real-World Viz:** Feature boundary map colored by boundary strength metrics - - **Concept Links:** slip transmission, Luster-Morris parameter, fracture initiation, grain boundaries - - **Changes Made:** Clarified relationship to ComputeSlipTransmissionMetrics (same metrics, per-Face storage). Added cross-reference for detailed explanations. Explained M' value meaning. - -- [x] **ComputeFeatureNeighborCAxisMisalignmentsFilter** (OrientationAnalysis) - - **Clarity:** Good basic description but "misalignment" vs "misorientation" distinction unclear - - **Figures Needed:** Diagram showing C-axis misalignment angle between two grains - - **Real-World Viz:** Neighbor misalignment values on microstructure - - **Concept Links:** C-axis, misalignment angle, hexagonal crystals, neighbor relationships - - **Changes Made:** Distinguished C-axis misalignment from full misorientation. Explained flexible neighbor list concept. Cross-referenced ComputeAvgCAxes for hexagonal-only explanation. - -- [x] **ComputeFeatureNeighborMisorientationsFilter** (OrientationAnalysis) - - **Clarity:** Straightforward but minimal - - **Figures Needed:** Diagram showing misorientation angle concept between neighboring grains - - **Real-World Viz:** Microstructure with neighbor misorientations displayed - - **Concept Links:** misorientation, orientation angle, grain boundaries - - **Changes Made:** Added plain-language explanation of misorientation. Explained NaN constraint (different crystal structures not comparable). Clarified optional average output. - -- [x] **ComputeFeatureReferenceCAxisMisorientationsFilter** (OrientationAnalysis) - - **Clarity:** Clear what it does but doesn't explain why this metric matters - - **Figures Needed:** Diagram showing cell-to-average C-axis deviation concept - - **Real-World Viz:** Misalignment map within a feature showing spatial variation - - **Concept Links:** C-axis misalignment, hexagonal materials, orientation variation - - **Changes Made:** Reframed around intragranular orientation gradients. Explained what average/std dev outputs tell you (low = uniform, high = internal variation). Cross-referenced ComputeAvgCAxes. - -- [x] **ComputeFeatureReferenceMisorientationsFilter** (OrientationAnalysis) - - **Clarity:** Good. Two reference orientation options clearly explained with use cases - - **Figures Needed:** Legend for existing color scale images would help interpretation - - **Real-World Viz:** Already has 3 images showing IPF colors and two reference orientation results - - **Concept Links:** misorientation, plastic deformation, reference orientation, IPF colors - - **Changes Made:** Added purpose statement about detecting deformation. Rewrote reference orientation options with physical reasoning for when to use each. - -- [x] **ComputeSlipTransmissionMetricsFilter** (OrientationAnalysis) - - **Clarity:** Relies on citations; metrics not explained in the documentation itself - - **Figures Needed:** Slip transmission concept diagram (same as ComputeBoundaryStrengths) - - **Real-World Viz:** Feature pairs with transmission metrics visualized - - **Concept Links:** slip transmission, slip planes/directions, grain boundaries, Luster-Morris - - **Changes Made:** Added introductory explanation of slip transmission concept. Explained M' value meaning (1.0 = perfect alignment). Added cross-reference to ComputeBoundaryStrengths. - -- [x] **ComputeTwinBoundariesFilter** (OrientationAnalysis) - - **Clarity:** Good sigma-3 identification logic but "incoherence" concept needs definition - - **Figures Needed:** Sigma-3 twin relationship diagram; incoherence concept - - **Real-World Viz:** Microstructure with twin boundaries highlighted - - **Concept Links:** twin boundaries, sigma-3, misorientation axis, CSL, incoherence - - **Changes Made:** Added "What is a Twin Boundary?" section explaining Sigma-3 twins and their significance. Explained incoherence in plain terms (0 = perfect coherent twin). - -- [x] **ComputeShapesFilter** (OrientationAnalysis) - - **Clarity:** Good algorithmic steps but missing interpretation guidance - - **Figures Needed:** Best-fit ellipsoid concept; Omega3 sphericity measure - - **Real-World Viz:** Features colored by aspect ratio or Omega3 - - **Concept Links:** principal moments, ellipsoid fitting, eigenvalues, Omega3, aspect ratios - - **Changes Made:** Added "What This Filter Produces" section explaining each output. Defined Omega3 (1.0 = sphere). Cross-referenced triangle geometry version. - -### Tier 3 — Polish - -| Filter | Plugin | Status | -|--------|--------|--------| -| ComputeGBCDMetricBased | OrientationAnalysis | Done | -| ComputeGBPDMetricBased | OrientationAnalysis | Done | -| ComputeShapesTriangleGeom | OrientationAnalysis | Done | - -- [x] **ComputeGBCDMetricBasedFilter** (OrientationAnalysis) - - **Clarity:** Excellent technical documentation with clear metric-based vs bin-based distinction - - **Figures Needed:** Maybe one more diagram showing metric distances visually - - **Real-World Viz:** Already has 3 excellent figures (GBCD result, sampling points, error maps) - - **Concept Links:** GBCD, metrics, boundary space, MRD units - - **Changes Made:** Restructured dense metric explanation into numbered two-stage process. Added note about advantages over binning (avoids discretization artifacts). - -- [x] **ComputeGBPDMetricBasedFilter** (OrientationAnalysis) - - **Clarity:** Good. References GBCD Metric-Based filter effectively - - **Figures Needed:** Plane normal vs misorientation sampling differences - - **Real-World Viz:** Already has 1 good figure - - **Concept Links:** GBPD, grain boundary plane distribution, boundary normal - - **Changes Made:** Added opening explaining what GBPD is and how it differs from GBCD (2D plane normals vs full 5D). Added MRD interpretation (1.0 = random, >1.0 = preferred). - -- [x] **ComputeShapesTriangleGeomFilter** (OrientationAnalysis) - - **Clarity:** Very technical but thorough. Excellent caveats section and validation table - - **Figures Needed:** Tetrahedron construction from mesh; watertight vs non-watertight mesh - - **Real-World Viz:** Comparison of voxelized vs triangle geometry shape results - - **Concept Links:** triangle geometry, mesh watertightness, Euler characteristic, winding - - **Changes Made:** Consolidated caveats into "Differences from Image Geometry Version" section. Merged watertight warning and Euler characteristic into "Mesh Quality Requirements" section. Cross-referenced Image Geometry version for output descriptions. - -### Already Complete (Prior to Triage) - -| Filter | Plugin | Status | -|--------|--------|--------| -| ComputeKernelAvgMisorientations | OrientationAnalysis | Done | - -- [x] **ComputeKernelAvgMisorientationsFilter** (OrientationAnalysis) - - **Changes Made:** Full rewrite with kernel radius explanation, 6 figures (formula, 1D radius 1 & 2, 2D radius (1,1,0) & (1,2,0), quick reference table). Serves as template for rewrite style. - ---- - -## Batch 4: Neighbor / Kernel Operations - -**Plugin:** SimplnxCore + OrientationAnalysis -**Filters:** 11 (cell-level cleanup, morphology, neighbor-list utilities) - -### Tier 1 — Critical - -| Filter | Plugin | Status | -|--------|--------|--------| -| AddBadData | SimplnxCore | Done | - -- [x] **AddBadDataFilter** (SimplnxCore) - - **Clarity:** Single dense paragraph with one run-on sentence; the filter's purpose (introduce realism into synthetic structures) is buried. - - **Completeness:** No example imagery, no parameter guidance. Mentions "Manhattan distances" with no context. Requires the reader to know about the *Convert Attribute Data Type* filter as a prerequisite. Group is "Synthetic Building (Misc)" which is an outlier vs. the other Batch 4 cleanup filters but is correct given the filter's purpose. - - **Accessibility:** Volume-fraction semantics are easy to misread (the *0.2* applies only to the affected cells, not the whole volume — a subtle but important point). - - **Figures Needed:** Diagram showing the two noise modes (random voxels scattered through volume vs. boundary voxels along feature surfaces). Visual of how a 0.2 volume fraction translates to actual changed-cell counts. - - **Real-World Viz:** Before/after of a synthetic microstructure with random noise applied; same with boundary noise applied. - - **Concept Links:** Manhattan distance, feature boundary cells, synthetic microstructure realism - - **Changes Made:** Added "Why Use This Filter?" section motivating the experimental-realism use case (matched cleanup pipelines for apples-to-apples comparison). Added "Random vs. Boundary Noise" section explaining the two independent modes. Added "Volume Fraction Semantics" section clarifying the per-eligible-cell meaning. Added "What 'Bad' Means in Output" section. Added explicit two-step prerequisite recipe (Compute Euclidean Distance Map → Convert Data Type). Added Required Input Sources. - -### Tier 2 — Important - -| Filter | Plugin | Status | -|--------|--------|--------| -| FillBadData | SimplnxCore | Done | -| ErodeDilateBadData | SimplnxCore | Done | -| ErodeDilateMask | SimplnxCore | Done | -| ErodeDilateCoordinationNumber | SimplnxCore | Done | -| ReplaceElementAttributesWithNeighborValues | SimplnxCore | Done | -| NeighborOrientationCorrelation | OrientationAnalysis | Done | -| BadDataNeighborOrientationCheck | OrientationAnalysis | Done | - -- [x] **FillBadDataFilter** (SimplnxCore) - - **Changes Made:** Rewrote lead-in to distinguish "small noise (filled)" vs "large defects (preserved)". Restructured implementation-detail Phase 1-4 into a concise "How This Filter Works" 3-step summary. Added "Minimum Defect Size Units" section stating cells/integer voxel count with conversion formula and typical ranges (5-50 for noise, 500-5000 for preserving real pores). Promoted "Store Defects as New Phase" option to its own section. Trimmed performance discussion to a brief note. Added Required Input Sources. - -- [x] **ErodeDilateBadDataFilter** (SimplnxCore) - - **Changes Made:** Removed stray backtick. Restructured into Dilation/Erosion/When-to-Use/Iterations-and-Direction sections. Named the morphological *opening* (erode-then-dilate) and *closing* (dilate-then-erode) patterns explicitly. Stated iterations are in **cell-layers**. Documented the X/Y/Z directional toggle for anisotropic morphology. Added Required Input Sources. - -- [x] **ErodeDilateMaskFilter** (SimplnxCore) - - **Changes Made:** Defined "mask" inline as a boolean cell-level array with cross-link to Multi-Threshold Objects. Added "When to Use" with concrete examples (erode = discard unreliable boundary cells; dilate = recover over-thresholded valid cells). Stated iterations are in **cell-layers**. Documented X/Y/Z directional toggle motivation (anisotropic serial-section resolution). Added Required Input Sources. - -- [x] **ErodeDilateCoordinationNumberFilter** (SimplnxCore) - - **Changes Made:** Stated coordination-number range is **0 to 6** with worked examples (CN=0 inside, CN=6 isolated voxel, CN=4-5 thin protrusion). Cross-linked "isotropic coarsening" definition to RequireMinimumSizeFeatures. Promoted "Loop Until Gone" to its own section with single-pass-vs-full-smoothing guidance. Added "When to Use This Filter" section calling out salt-and-pepper cleanup. Added Required Input Sources. - -- [x] **ReplaceElementAttributesWithNeighborValuesFilter** (SimplnxCore) - - **Changes Made:** Added generic "How This Filter Works" section before EBSD vendor-specific examples. Promoted "Comparison Operator" to its own subsection with use-case guidance for higher-is-better (CI, IQ) vs lower-is-better (MAD, Error) scalars. Added "Loop Until Gone" subsection with caution about flood-fill behavior. Promoted the "too much replacement" warning to its own "Caution: Flood Fill Behavior" section. Reframed EBSD examples as "Example Use Cases". Added Required Input Sources. - -- [x] **NeighborOrientationCorrelationFilter** (OrientationAnalysis) - - **Changes Made:** Added lead-in distinguishing this filter from sister filter ReplaceElementAttributesWithNeighborValues. Restructured algorithm into 4 numbered steps. Promoted *Cleanup Level* to its own subsection with **1-6 range** stated explicitly and worked guidance for each value (6=conservative, 4-5=moderate, 2-3=aggressive, 1=flood-fill). Cross-linked sister filters and EBSD readers. Added Required Input Sources. - -- [x] **BadDataNeighborOrientationCheckFilter** (OrientationAnalysis) - - **Changes Made:** Verified the doc title matches humanName() ("Neighbor Orientation Comparison (Bad Data)"); kept as-is. Added lead-in distinguishing this filter (mask-only update) from NeighborOrientationCorrelation (full-attribute replacement) and recommending sequencing. Promoted *Required Number of Neighbors* to its own section with **1-6 range** stated explicitly. Added cross-references to Multi-Threshold Objects, EBSD readers, and the sister filter. Added Required Input Sources. - -### Tier 3 — Polish - -| Filter | Plugin | Status | -|--------|--------|--------| -| RequireMinNumNeighbors | SimplnxCore | Done | -| ComputeNeighborhoods | SimplnxCore | Done | -| ComputeNeighborListStatistics | SimplnxCore | Done | - -- [x] **RequireMinNumNeighborsFilter** (SimplnxCore) - - **Changes Made:** Rewrote lead with explicit motivation (isolated single-feature islands from segmentation). Cross-linked "isotropic coarsening" to the shared definition in RequireMinimumSizeFeatures. Stated threshold is in **count of contiguous neighbors**. Added inspection-recommendation (look at *Number of Neighbors* output before choosing). Added Required Input Sources. - -- [x] **ComputeNeighborhoodsFilter** (SimplnxCore) - - **Changes Made:** Added lead-in motivating the use case (clustering, second-nearest-neighbor analysis). Defined Equivalent Sphere Diameter inline with cross-link to Compute Feature Sizes. Stated *Multiplier* is **dimensionless**. Added typical-value guidance for the multiplier (1.0 immediate neighbors, 2.0-3.0 second-nearest, 5.0+ broader clustering). Added Required Input Sources. - -- [x] **ComputeNeighborListStatisticsFilter** (SimplnxCore) - - **Changes Made:** Added concrete motivating example (summarizing per-feature neighbor misorientations from ComputeFeatureNeighborMisorientations). Restructured statistics list with inline descriptions. Stated outputs inherit units from input. Added Required Input Sources listing the four common NeighborList producers across both plugins. - ---- - -## Batch 5: Geometry Creation / Manipulation - -**Plugin:** SimplnxCore -**Filters:** 15 (geometry creation, transformation, resampling, cropping/padding, mesh generation, partitioning) - -### Tier 1 — Critical - -None. No filter in this batch has a non-expert blocker. - -### Tier 2 — Important - -| Filter | Plugin | Status | -|--------|--------|--------| -| CreateGeometry | SimplnxCore | Done | -| ApplyTransformationToGeometry | SimplnxCore | Done | -| RotateSampleRefFrame | SimplnxCore | Done | -| ResampleImageGeom | SimplnxCore | Done | -| ResampleRectGridToImageGeom | SimplnxCore | Done | -| PadImageGeometry | SimplnxCore | Done | -| QuickSurfaceMesh | SimplnxCore | Done | -| InitializeImageGeomCellData | SimplnxCore | Done | - -- [x] **CreateGeometryFilter** (SimplnxCore) - - **Clarity:** Comprehensive but very long (~19KB); the 8-geometry-type taxonomy is buried in dense prose. - - **Completeness:** Excellent — covers all 8 geometry types with examples, including a complete worked example for importing from text files. Possibly too much for a single filter doc. - - **Accessibility:** Heavy DREAM3D-NX terminology assumed (Element, Attribute Matrix, shared vertex list). - - **Figures Needed:** Side-by-side infographic showing all 8 geometry types (Image, Rectilinear Grid, Vertex, Edge, Triangle, Quadrilateral, Tetrahedral, Hexahedral) with their defining elements illustrated. - - **Real-World Viz:** None additional needed; the text import workflow could use a quick diagram. - - **Units Clarity:** Spacing units ("microns per pixel" example) called out — good. - - **Concept Links:** Geometry types, Element type, shared vertex list, winding/right-hand rule, Attribute Matrix - - **Notes:** "DREAM3D Review (Geometry)" group is unusual — verify this is intended. Typo "dimenionality" appears multiple times. Each geometry-type section duplicates content from per-type filters. Consider promoting the geometry-type taxonomy into a shared concept page and trimming this doc to filter-specific guidance. - -- [x] **ApplyTransformationToGeometryFilter** (SimplnxCore) - - **Clarity:** Strong before/after image sequences explaining the successive-rotation artifact problem. Excellent caveat. - - **Completeness:** Has duplicate transformation-type listing (bullets at lines 96-103 and the same content as a table at lines 107-114). One needs to go. - - **Accessibility:** Node vs Image distinction is critical and well-explained; row-major matrix encoding explained. - - **Figures Needed:** Already has many. Could add a diagram explaining the four-element axis-angle representation. - - **Real-World Viz:** Already excellent. - - **Units Clarity:** Rotation angle units (degrees) explicit; translation in geometry's coordinate units; scale dimensionless. - - **Concept Links:** transformation matrix, axis-angle, row-major matrix order, interpolation modes - - **Notes:** Remove the duplicated transformation-type listing. Convert "[Combine Transformation Matrices](CombineTransformationMatricesFilter.md)" link to a proper MyST link (already is). Add Required Input Sources. The relative `./Filter.md` should be checked — current uses both `./` and bare forms. - -- [x] **RotateSampleRefFrameFilter** (SimplnxCore) - - **Clarity:** Verified-only-for-axis-aligned-90/180 warning is at the top — good. Distinction from ApplyTransformation (sample reference frame vs geometric rotation) is not made explicit. - - **Completeness:** Has rotation matrix equivalent shown. Note about origin shift after rotation is valuable. - - **Accessibility:** "Sample reference frame" jargon assumed; the term needs first-use definition. - - **Figures Needed:** Diagram showing sample frame rotation (the axes of the sample relabeled) vs geometric rotation (the data physically rotated). - - **Real-World Viz:** Already has EBSD rotation example. - - **Units Clarity:** Rotation angle in degrees; axis is dimensionless unit vector. - - **Concept Links:** sample reference frame, EBSD frame convention, axis-angle rotation - - **Notes:** Add explicit "When to Use vs ApplyTransformation" guidance. Make verified-axis-only banner more prominent. Cross-link to SetImageGeomOriginScaling for the origin reset workflow. Add Required Input Sources. - -- [x] **ResampleImageGeomFilter** (SimplnxCore) - - **Clarity:** Three resampling modes (Spacing, Scaling, Exact Dimensions) each with worked numerical examples. Clear. - - **Completeness:** "No interpolation, closest cell wins" behavior noted. Renumber Features + NeighborList removal warning present. - - **Accessibility:** Spacing/dimensions/scaling distinction depends on understanding Image Geometry. - - **Figures Needed:** Before/after voxel-grid diagram showing how a 2x spacing reduction halves the cell count. - - **Real-World Viz:** Cubic Small IN100 sliced at different resolutions side-by-side. - - **Units Clarity:** Spacing values are in **physical units** (same as input geometry's spacing); scaling is **percent**; exact dimensions are in **cells**. - - **Concept Links:** spacing vs scaling vs dimensions, closest-cell resampling, Feature renumbering - - **Notes:** Cross-link to ApplyTransformation is present but uses `./` path; convert to plain MyST form. Add explicit units callouts per mode. Add Required Input Sources. - -- [x] **ResampleRectGridToImageGeomFilter** (SimplnxCore) - - **Clarity:** Brief, just describes the "last one wins" rule. - - **Completeness:** Missing: when to use this, how to specify the target image geometry's spacing/dimensions, what happens when target is finer than source. - - **Accessibility:** Rectilinear Grid concept not introduced. - - **Figures Needed:** Side-by-side diagram of variable-spacing RectGrid → regular Image Geom with the "last one wins" cells highlighted. - - **Real-World Viz:** A real RectGrid resampled to a uniform Image Geom. - - **Units Clarity:** Target spacing in physical units. - - **Concept Links:** Rectilinear Grid vs Image Geometry, downsampling vs upsampling - - **Notes:** Add "When to Use" section. Explain why "last one wins" was chosen (interpolation is wrong for label data). Add Required Input Sources. - -- [x] **PadImageGeometryFilter** (SimplnxCore) - - **Clarity:** Very brief; the figures are helpful but the prose underdocuments the "default padding value" semantics. - - **Completeness:** Doesn't explain what the *Update Origin* option does or when to use it. - - **Accessibility:** "Default padding value" is undefined. - - **Figures Needed:** Already has before/after. Could add a 3-panel diagram showing Update Origin ON vs OFF. - - **Real-World Viz:** Before/after with both Update Origin settings. - - **Units Clarity:** Pad amounts are in **cells/voxels**. Default value is in whatever units the target array uses. - - **Concept Links:** padding, image extension, origin shift - - **Notes:** Expand "default padding value" to "Each padded cell is initialized to the user-specified Default Value, which is interpreted in the same units as each cell-level array being padded". Document *Update Origin* and *Update Spacing* options. Add Required Input Sources. - -- [x] **QuickSurfaceMeshFilter** (SimplnxCore) - - **Clarity:** Deprecation notice at top recommending "Surface Nets". Node Types table is excellent. Triangle-pair-per-cell-face algorithm explained. - - **Completeness:** Has good images for each Node Type. The Verify Triangle Winding reference is implicit; needs a link. - - **Accessibility:** "Stair stepped" surface mesh result not pictured directly. - - **Figures Needed:** Already has plenty. Could use a small diagram of how a single voxel face becomes 2 triangles. - - **Real-World Viz:** Already present. - - **Concept Links:** voxel-face-to-triangle conversion, Node Types, FaceLabels convention, mesh windings - - **Notes:** Improve deprecation banner — make clear when to use this vs Surface Nets. Convert "see Verify Triangle Winding documentation" to a MyST link. Add Required Input Sources (FeatureIds from a segment filter). - -- [x] **InitializeImageGeomCellDataFilter** (SimplnxCore) - - **Clarity:** Three modes (Manual, Random, Random With Range) are listed but the "subvolume" parameter is undefined. - - **Completeness:** Missing: how is the subvolume specified (min/max cell indices? physical bounds?). What data types support each random mode? What about boolean arrays? - - **Accessibility:** Reader has to guess the subvolume specification UI. - - **Figures Needed:** Diagram showing a subvolume highlighted within a larger volume; before/after. - - **Real-World Viz:** Before/after on a real cube showing the initialized subvolume. - - **Units Clarity:** Subvolume bounds in **cells/voxels** vs **physical units** — must be stated. - - **Concept Links:** subvolume, cell-level initialization, random number generation seed - - **Notes:** Document the subvolume specification mode and units. Note random-mode behavior for boolean, integer, and float types separately. Document the random seed parameter. Add Required Input Sources. - -### Tier 3 — Polish - -| Filter | Plugin | Status | -|--------|--------|--------| -| CreateImageGeometry | SimplnxCore | Done | -| CombineTransformationMatrices | SimplnxCore | Done | -| SetImageGeomOriginScaling | SimplnxCore | Done | -| CropImageGeometry | SimplnxCore | Done | -| AppendImageGeometry | SimplnxCore | Done | -| PartitionGeometry | SimplnxCore | Done | -| ComputeCoordinatesImageGeom | SimplnxCore | Done | - -- [x] **CreateImageGeometryFilter** (SimplnxCore) - - **Clarity:** Deprecation banner is right under the title but as a level-2 heading, making it look like its own section. Confusing. - - **Completeness:** Dimensions/Spacing/Origin definitions are good. - - **Accessibility:** Same dense terminology as CreateGeometry. - - **Figures Needed:** Dimensions/origin/spacing diagram (would also serve SetImageGeomOriginScaling and ResampleImageGeom). - - **Real-World Viz:** N/A — filter creates rather than transforms data. - - **Units Clarity:** Already states spacing example "microns per pixel". - - **Concept Links:** Image Geometry dimensions/origin/spacing - - **Notes:** Move deprecation notice into a clear callout. Fix "dimenionality" typo. Cross-link to CreateGeometry filter. Add Required Input Sources note (none — geometry is created from user parameters). - -- [x] **CombineTransformationMatricesFilter** (SimplnxCore) - - **Clarity:** Brief and adequate. Output format documented. - - **Completeness:** Missing motivating context for why you'd combine matrices (the answer is in ApplyTransformation — referenced indirectly there). - - **Accessibility:** Assumes user knows row-major 4x4 convention. - - **Figures Needed:** Optional — a diagram showing two 4x4 matrices multiplying into a single 4x4 result. - - **Real-World Viz:** N/A. - - **Units Clarity:** N/A (matrix is dimensionless apart from translation components). - - **Concept Links:** matrix composition, transformation order, row-major matrix order - - **Notes:** Add a short "Why Use This Filter" pointing to ApplyTransformationToGeometry's caveat about successive image-geometry transformations. Note that matrices are applied left-to-right in the order listed. Add Required Input Sources. - -- [x] **SetImageGeomOriginScalingFilter** (SimplnxCore) - - **Clarity:** Title says "Set Origin & Spacing" but body only describes the origin parameters. Spacing is implicit. - - **Completeness:** "Put Input Origin at the Center of Geometry" option mentioned but not explained in detail. - - **Accessibility:** OK for the limited content provided. - - **Figures Needed:** Diagram showing origin-at-corner vs origin-at-center. - - **Real-World Viz:** N/A. - - **Units Clarity:** Origin coordinates and spacing are in physical units. - - **Concept Links:** Image Geometry origin, spacing - - **Notes:** Document both Origin and Spacing parameter sets. Document the linkable parameters (only-change-origin, only-change-spacing options). Expand the "Put Origin at Center" explanation. Add Required Input Sources. - -- [x] **CropImageGeometryFilter** (SimplnxCore) - - **Clarity:** Three worked examples with images. Inclusive-bounds note is critical and called out. - - **Completeness:** Voxels-vs-physical-coordinates mode is implicit; the "Use Physical Bounds" parameter should be stated explicitly. - - **Accessibility:** Voxel index 0-based convention is implied via examples. - - **Figures Needed:** Already adequate. - - **Real-World Viz:** Already present. - - **Units Clarity:** Bounds are in **cells (0-based, inclusive)** OR **physical units (depending on parameter)** — explicit per-mode. - - **Concept Links:** ROI cropping, inclusive bounds, voxel indexing - - **Notes:** Add explicit "Bounds Mode" subsection naming the parameter that toggles voxels vs physical. State units per mode. Update NeighborList warning to use the standard pattern shared with RequireMinimumSizeFeatures. - -- [x] **AppendImageGeometryFilter** (SimplnxCore) - - **Clarity:** Exhaustive examples for X/Y/Z directions with figures. - - **Completeness:** "Resolution" terminology used in one place — should be "Spacing" per the modern simplnx convention. - - **Accessibility:** Good. - - **Figures Needed:** Already comprehensive. - - **Real-World Viz:** Already present. - - **Units Clarity:** Append happens in cell-count units along the chosen direction; the *Check Spacing* option checks physical-unit match. - - **Concept Links:** geometry concatenation, mirroring, spacing check - - **Notes:** Replace remaining "Resolution" with "Spacing" for consistency. Cross-link to CreateImageGeometry for the geometry-definition concepts. Add Required Input Sources. - -- [x] **PartitionGeometryFilter** (SimplnxCore) - - **Clarity:** Four modes with excellent visual examples and walk-throughs. - - **Completeness:** "Reconstruction (Reconstruction)" group/subgroup is a stutter — verify. - - **Accessibility:** Long doc but well-organized with figures. - - **Figures Needed:** Already excellent. - - **Real-World Viz:** Already present. - - **Units Clarity:** Cell Length is in physical units; Number of Cells Per Axis is in **integer cell counts**; Origin in physical units; Min/Max in physical units. - - **Concept Links:** spatial partitioning, partition grid, out-of-bounds handling, vertex mask - - **Notes:** Verify group "Reconstruction (Reconstruction)". Add a short lead-in explaining why partition (spatial analysis, sub-volume statistics, parallel processing prep). Add Required Input Sources. - -- [x] **ComputeCoordinatesImageGeomFilter** (SimplnxCore) - - **Clarity:** "Implicit vs explicit" framing is jargon; the example output is clear. - - **Completeness:** Three output modes documented with raster scheme. - - **Accessibility:** Reader benefits from concrete example, which is provided. - - **Figures Needed:** Optional — a small voxel-grid diagram with one cell highlighted showing its (i,j,k) index and (x,y,z) physical position. - - **Real-World Viz:** N/A — output is numerical. - - **Units Clarity:** Physical coordinates in geometry's physical units; indices are **integer cell indices (0-based)**. - - **Concept Links:** cell indexing scheme, physical coordinates, ijk vs xyz, raster order - - **Notes:** Rewrite "implicit/explicit" framing in plain language ("makes the per-cell coordinates available as a regular cell-level array"). Add a "When to Use" — typically for CSV/text export workflows. Add Required Input Sources (Image Geometry). - ---- - -## Batch 6: Data Manipulation - -**Plugin:** SimplnxCore -**Filters:** 21 (create / copy / delete / rename / move / combine / split / convert / initialize / calculate) - -### Tier 1 — Critical - -None. No filter in this batch is incomprehensible to a non-expert. - -### Tier 2 — Important - -| Filter | Plugin | Status | -|--------|--------|--------| -| CreateAttributeMatrix | SimplnxCore | Done | -| CreateDataGroup | SimplnxCore | Done | -| DeleteData | SimplnxCore | Done | -| RenameDataObject | SimplnxCore | Done | -| MoveData | SimplnxCore | Done | -| CopyFeatureArrayToElementArray | SimplnxCore | Done | -| CreateFeatureArrayFromElementArray | SimplnxCore | Done | -| ConditionalSetValue | SimplnxCore | Done | -| ConvertColorToGrayScale | SimplnxCore | Done | -| ReshapeDataArray | SimplnxCore | Done | - -- [ ] **CreateAttributeMatrixFilter** (SimplnxCore) - - **Clarity:** Extremely brief (only ~20 lines). Lone example does not explain what an Attribute Matrix conceptually represents. - - **Completeness:** Missing the *why* (tuple-dimensions discipline for arrays in the same matrix), the *when* (typical use cases — creating a Feature Attribute Matrix, an Ensemble Attribute Matrix), and the relationship to DataGroup. - - **Accessibility:** Assumes the user already knows what an Attribute Matrix is. - - **Figures Needed:** Optional — a hierarchy diagram showing Image Geometry → Cell Attribute Matrix → arrays would also serve CreateGeometry/CreateDataGroup. - - **Concept Links:** Attribute Matrix, tuple dimensions, DataGroup vs Attribute Matrix - - **Notes:** Add "What is an Attribute Matrix?" subsection. Cross-link to CreateDataGroup with explicit when-to-use-which guidance. Note tuple-dimension consistency requirement. - -- [ ] **CreateDataGroupFilter** (SimplnxCore) - - **Clarity:** Only 1 paragraph; says "unlike AttributeMatrix, DataGroups are capable of holding any DataObject of any size" but doesn't explain when to choose one over the other. - - **Completeness:** Missing typical use case (organizing related arrays of different sizes, creating an output structure for a future filter). - - **Concept Links:** DataGroup, AttributeMatrix - - **Notes:** Expand. Cross-reference CreateAttributeMatrix and explain the choice: DataGroup for heterogeneous content, Attribute Matrix when all child arrays must share tuple dimensions. - -- [ ] **DeleteDataFilter** (SimplnxCore) - - **Clarity:** Two use cases (memory and name collisions) clearly explained. - - **Completeness:** Missing: cascade behavior (does deleting a Geometry also delete its Attribute Matrices? Does deleting an Attribute Matrix delete its child arrays?), what happens to downstream filters that reference the deleted object. - - **Concept Links:** cascade delete, object lifetimes - - **Notes:** Document the cascade behavior. Add explicit warning: subsequent filters that selected the deleted object will fail preflight. - -- [ ] **RenameDataObjectFilter** (SimplnxCore) - - **Clarity:** Single sentence. - - **Completeness:** Missing: what happens to filter parameters downstream that reference the old name (they won't auto-update — preflight will fail). What name collisions are allowed/disallowed. - - **Notes:** Expand to one paragraph. Warn about downstream filter parameter references not updating automatically. - -- [ ] **MoveDataFilter** (SimplnxCore) - - **Clarity:** Brief but covers the tuple-count requirement. - - **Completeness:** Missing: examples (moving a computed array into a Feature Attribute Matrix; moving an array between DataGroups). What happens to arrays that are children of the moved object. - - **Concept Links:** parent-child hierarchy, tuple validation - - **Notes:** Add concrete examples. Explain that tuple-dimension *shape* doesn't need to match — only the *number of tuples*. - -- [ ] **CopyFeatureArrayToElementArrayFilter** (SimplnxCore) - - **Clarity:** "Xmdf visualization files write only the Element attributes" is jargon and almost the entire rationale. - - **Completeness:** Missing: general "when to use" (broadcasting a per-Feature scalar to every cell of that feature for visualization), parameter description. - - **Notes:** Lead with the general purpose (broadcast Feature-level value back to all cells of the Feature). Explain the Xmdf-export reason as one specific use case rather than the only one. Add Required Input Sources. - -- [ ] **CreateFeatureArrayFromElementArrayFilter** (SimplnxCore) - - **Clarity:** "the value of the *last element copied*" is buried as the central footgun. - - **Completeness:** Missing: warning that this filter is destructive when used on per-cell scalars that vary within a feature (since most cells get discarded). When to use vs. ComputeArrayStatistics (which computes per-feature means). - - **Notes:** Promote the "last element copied" caveat to a Warning section. Cross-reference ComputeArrayStatistics for averaging behavior. Add Required Input Sources. - -- [ ] **ConditionalSetValueFilter** (SimplnxCore) - - **Clarity:** Two modes (conditional mask vs value-replacement) buried in one dense sentence. - - **Completeness:** Mode parameter (*Use Conditional Mask*) is the central control but not promoted. - - **Notes:** Split the two modes into their own subsections. Document *Use Conditional Mask* explicitly. Add Required Input Sources. - -- [ ] **ConvertColorToGrayScaleFilter** (SimplnxCore) - - **Clarity:** Four conversion algorithms documented with formulas. - - **Completeness:** Missing: when to use (preprocessing color images for downstream grayscale-only filters), expected input format (uint8 RGB/RGBA), output type. - - **Notes:** Add "When to Use" lead-in. State input/output array types and component shapes. Add Required Input Sources (ITK image reader). - -- [ ] **ReshapeDataArrayFilter** (SimplnxCore) - - **Clarity:** Critical footgun warning ("DOES NOT MOVE ANY VALUES IN MEMORY") is buried at line 11. - - **Completeness:** Stride-mismatch example is good but the warning needs more prominence given that misuse silently produces wrong results. - - **Concept Links:** strides, row-major / C-order storage, tuple dimensions - - **Notes:** Promote the "no memory rearrangement" warning into a top-level Warning section. Add a "When NOT to Use" callout. Add Required Input Sources. - -### Tier 3 — Polish - -| Filter | Plugin | Status | -|--------|--------|--------| -| CreateDataArray | SimplnxCore | Done | -| CreateDataArrayAdvanced | SimplnxCore | Done | -| CopyDataObject | SimplnxCore | Done | -| CombineAttributeArrays | SimplnxCore | Done | -| ConcatenateDataArrays | SimplnxCore | Done | -| SplitDataArrayByComponent | SimplnxCore | Done | -| SplitDataArrayByTuple | SimplnxCore | Done | -| ExtractComponentAsArray | SimplnxCore | Done | -| ConvertData | SimplnxCore | Done | -| ArrayCalculator | SimplnxCore | Done | -| InitializeData | SimplnxCore | Done | - -- [ ] **CreateDataArrayFilter** (SimplnxCore) - - **Notes:** Already has good data-type ranges table and semicolon-notation example. Minor polish: state that Number of Components must be ≥ 1; explicitly note that this filter creates a *single component dimension* (use CreateDataArrayAdvanced for multi-dimension components). Add Required Input Sources (none). - -- [ ] **CreateDataArrayAdvancedFilter** (SimplnxCore) - - **Notes:** Documents 4 initialization modes (Fill, Incremental, Random, Random With Range) and Step Operation. Mostly good. Polish: state component-dimension product must be ≥ 1; clarify that multi-dimensional components (e.g., 3x3 tensor) are supported here. Cross-link CreateDataArray as the simpler version. - -- [ ] **CopyDataObjectFilter** (SimplnxCore) - - **Notes:** Clear; "deep copy" semantics for BaseGroup containers documented. Polish: cross-link Copy to New Parent option more explicitly; add Required Input Sources (none). - -- [ ] **CombineAttributeArraysFilter** (SimplnxCore) - - **Notes:** Already has worked examples and figures. Polish: cross-link "Concatenate Data Arrays" already present; mention the use case of building a Vertex coordinates array from three separate x/y/z arrays. Add Required Input Sources. - -- [ ] **ConcatenateDataArraysFilter** (SimplnxCore) - - **Notes:** Brief but clear. Cross-link to Combine Attribute Arrays is present. Polish: state that the result is always 1-D (already does); add example. - -- [ ] **SplitDataArrayByComponentFilter** (SimplnxCore) - - **Notes:** Clear worked example. Polish: minor copy-edit ("unput" → "input"); state the "specifying a subset of components" mode more cleanly; cross-link Combine Attribute Arrays as the inverse operation (already does). - -- [ ] **SplitDataArrayByTupleFilter** (SimplnxCore) - - **Notes:** Clear worked example. Polish: cross-link Split Data Array (By Component) present. Polish: state units on tuple counts (integer counts). - -- [ ] **ExtractComponentAsArrayFilter** (SimplnxCore) - - **Notes:** Title in cpp is "Extract/Remove Components" — verify match. Brief but covers the 3 operation modes. Polish: add a worked example showing the 3 modes; add Required Input Sources. - -- [ ] **ConvertDataFilter** (SimplnxCore) - - **Notes:** Already very thorough with up/down casting and signed/unsigned warnings. Polish: cross-link to Reshape Data Array for the related-but-different "interpret data differently" use case. Add Required Input Sources. - -- [ ] **ArrayCalculatorFilter** (SimplnxCore) - - **Notes:** Already extremely thorough — operator tables, multi-component handling, explicit array name escaping, multiple worked examples. Polish: minor only. Cross-link to Convert Angles to Degrees or Radians (already mentioned). Add Required Input Sources. - -- [ ] **InitializeDataFilter** (SimplnxCore) - - **Notes:** Content is good but the prose is rambly. Polish: restructure each initialization mode into its own subsection with a brief intro and bullet list of nuances rather than nested bullet trees. Move boolean entry rules to a single dedicated subsection. - ---- - -## Batches 8-9: Not Yet Triaged - -The following batches have been identified in the design spec but have not yet been triaged. Each batch will be triaged at the start of its work cycle. (Batch 7 was triaged 2026-06-10; see the Batch 7 section below.) - -## Batch 2: Alignment Filters - -**Plugin:** OrientationAnalysis + SimplnxCore -**Filters:** 5 - -### Tier 2 — Important - -| Filter | Plugin | Status | -|--------|--------|--------| -| AlignSectionsMutualInformation | OrientationAnalysis | Done | -| AlignSectionsMisorientation | OrientationAnalysis | Done | -| AlignSectionsList | SimplnxCore | Done | - -- [x] **AlignSectionsMutualInformationFilter** (OrientationAnalysis) - - **Clarity:** Functional but domain-heavy; mutual information concept not explained for non-experts - - **Figures Needed:** Diagram showing 7x7 grid search with high/low MI regions - - **Real-World Viz:** Pipeline screenshot showing input/output shift arrays - - **Concept Links:** mutual information, feature segmentation, 7x7 grid search, local minima - - **Changes Made:** Added "What is Mutual Information?" section. Added "When to Use This Method" guidance. Restructured algorithm into clear numbered steps with named subsections. Explained misorientation tolerance parameter. Cross-referenced other alignment methods. - -- [x] **AlignSectionsMisorientationFilter** (OrientationAnalysis) - - **Clarity:** Clear algorithm steps but non-experts need misorientation definition; background subtraction underdocumented - - **Figures Needed:** Diagram of 7x7 grid search with before/after alignment; background subtraction concept - - **Real-World Viz:** Example output shifts for a real dataset - - **Concept Links:** misorientation angle, cell-to-cell pairing, 7x7 grid search, local minima, background shift removal - - **Changes Made:** Added "When to Use This Method" guidance with cross-references to alternatives. Restructured algorithm with clear step descriptions. Separated local minima warning, masking, and linear background subtraction into named subsections. - -- [x] **AlignSectionsListFilter** (SimplnxCore) - - **Clarity:** Core concept clear but overwhelmed by backwards compatibility cruft and file format specs - - **Figures Needed:** Flowchart showing relative vs cumulative shift interpretation - - **Real-World Viz:** Typical usage showing output from another alignment filter feeding into this one - - **Concept Links:** relative shifts, cumulative shifts, slice ordering, CSV import - - **Changes Made:** Rewrote lead paragraph to explain the three use cases. Clarified relative vs cumulative with concise definitions. Consolidated file import instructions into a single streamlined section. Moved legacy format details into a brief "Note on Legacy Files" section. - -### Tier 3 — Polish - -| Filter | Plugin | Status | -|--------|--------|--------| -| AlignSectionsFeatureCentroid | SimplnxCore | Done | - -- [x] **AlignSectionsFeatureCentroidFilter** (SimplnxCore) - - **Clarity:** Clear and accessible; centroid-based alignment is straightforward - - **Figures Needed:** Diagram showing centroid calculation on adjacent slices before/after alignment - - **Real-World Viz:** Before/after visualization of aligned sections - - **Concept Links:** centroid, cell resolution, reference slice, background shift removal - - **Changes Made:** Restructured into clear subsections (When to Use, How It Works, Reference Slice, Linear Background Subtraction). Added comparison note about local minima advantage over other methods. - -### Tier 4 — Adequate - -| Filter | Plugin | Status | -|--------|--------|--------| -| AlignGeometries | SimplnxCore | No Changes Needed | - -- [x] **AlignGeometriesFilter** (SimplnxCore) - - **Notes:** 32 lines. Two simple alignment methods (Origin and Centroid) clearly explained. No domain jargon. Adequate as-is. - ---- - -## Batch 3: Segmentation / Feature Identification - -**Plugin:** SimplnxCore + OrientationAnalysis -**Filters:** 10 (3 core segmentation + 7 feature identification/post-processing) - -### Tier 1 — Critical - -| Filter | Plugin | Status | -|--------|--------|--------| -| MergeTwins | OrientationAnalysis | Done | - -- [x] **MergeTwinsFilter** (OrientationAnalysis) - - **Clarity:** Very sparse (~27 lines). Cubic-High restriction stated in caps but no context on why twins are merged; Sigma-3 and `<111>` axis terminology used with no definitions. - - **Completeness:** Only one paragraph of description; no output array description; no guidance on typical tolerance values; no before/after imagery. - - **Accessibility:** Assumes reader knows what a twin is, what Sigma-3 notation means, and how to interpret `<111>` direction notation. - - **Figures Needed:** Sigma-3 twin relationship diagram; visual explanation of axis tolerance vs. angle tolerance. - - **Real-World Viz:** Before/after microstructure showing twin variants merged into the parent grain. - - **Concept Links:** twin boundaries, Sigma-3, FCC, `<111>` axis, misorientation - - **Changes Made:** Added "What is a Twin?" section explaining Sigma-3, FCC annealing twins, and CSL notation. Added "Why Merge Twins?" section giving the downstream-analysis motivation (grain counting, size distributions, neighbor stats). Restructured algorithm into numbered steps on the Feature level. Added "Parameter Guidance" with explicit degree units and typical tolerance ranges. Added "Required Input Sources" section listing each upstream filter that must run first. Added "Limitations" clarifying Cubic-High/m3m restriction, FCC-only practical applicability, and Sigma-3-only detection. - -### Tier 2 — Important - -| Filter | Plugin | Status | -|--------|--------|--------| -| ScalarSegmentFeatures | SimplnxCore | Done | -| CAxisSegmentFeatures | OrientationAnalysis | Done | -| EBSDSegmentFeatures | OrientationAnalysis | Done | -| RequireMinimumSizeFeatures | SimplnxCore | Done | -| ComputeFeatureNeighbors | SimplnxCore | Done | - -- [x] **ScalarSegmentFeaturesFilter** (SimplnxCore) - - **Clarity:** Burn algorithm steps clear, but term "burn algorithm" undefined; no introduction to what segmentation accomplishes conceptually. - - **Completeness:** Neighbor scheme well documented with 4 paired figure sets. No guidance on tolerance values (tolerance is in whatever units the scalar array uses). - - **Accessibility:** "Cells" and "Features" used without first-use definition; bold emphasis is stylistic rather than explanatory. - - **Figures Needed:** Neighbor-scheme figures already excellent; could add a simple "what is segmentation" before/after conceptual diagram. - - **Real-World Viz:** Before/after FeatureIds map from a scalar-based segmentation (e.g., image-quality thresholding). - - **Units Clarity:** Tolerance has no explicit units — depends entirely on input array data type; this must be stated. - - **Concept Links:** Feature IDs, burn algorithm, segmentation, Cell vs Feature data - - **Changes Made:** Added lead-in describing the FeatureIds output and cross-referencing the two orientation-based segment filters. Added "What is Feature Segmentation?" conceptual section. Rewrote algorithm into a burn-algorithm step list. Added "Tolerance and Units" section with concrete examples for integer phase maps, 0-1 image quality, and 0-255 grayscale. Added "Mask Array" and "Periodic Option" sections. Added Required Input Sources with cross-plugin links. - -- [x] **CAxisSegmentFeaturesFilter** (OrientationAnalysis) - - **Clarity:** Brief C-axis definition provided, but assumes hexagonal-system knowledge; no guidance on when to use this vs EBSDSegmentFeatures. - - **Completeness:** Shares structure and neighbor scheme figures with other segment filters. Missing: explicit hexagonal-only warning, phase handling, when-to-use guidance. - - **Accessibility:** `<001>`, hexagonal system, C-axis all used without plain-language expansion. - - **Figures Needed:** Reuse hexagonal C-axis figure from ComputeAvgCAxes (Batch 1); add a "C-axis misalignment between two neighboring cells" diagram. - - **Real-World Viz:** Before/after segmentation on hexagonal (e.g., Ti) EBSD data. - - **Units Clarity:** Tolerance is in **degrees** — must be stated explicitly in the parameter description. - - **Concept Links:** C-axis, hexagonal materials, burn algorithm, reference frames - - **Changes Made:** Added explicit hexagonal-only warning at the top with cross-references to alternative segment filters. Added "When to Use This Filter" explaining why C-axis alignment is the right criterion for hexagonal materials and when to use misorientation-based segmentation instead. Rewrote algorithm steps; added "Tolerance and Units" with degree units and typical values (1-3 / 5 / 10+). Added Required Input Sources linking ComputeAvgCAxes for C-axis concept. - -- [x] **EBSDSegmentFeaturesFilter** (OrientationAnalysis) - - **Clarity:** Generic term "misorientation" used without definition; core burn algorithm clear but domain context missing. - - **Completeness:** No guidance on typical tolerance values (5° is the common industry default for grain segmentation); no mention of how multiple phases are handled. - - **Accessibility:** Assumes EBSD workflow and misorientation knowledge. - - **Figures Needed:** Misorientation angle diagram (shared with Batch 1 misorientation concept work); before/after segmentation example. - - **Real-World Viz:** EBSD IPF map → segmented grains visualization — this is one of the highest-value workflow visualizations in DREAM3DNX. - - **Units Clarity:** Tolerance is in **degrees** — must be stated explicitly. - - **Concept Links:** misorientation, EBSD workflow, burn algorithm, reference frames, grain segmentation - - **Changes Made:** Added "What is Misorientation-Based Segmentation?" section explaining grains and misorientation for non-experts. Rewrote algorithm into numbered burn steps including symmetry handling. Added "Typical Tolerance Values" section with 5° industry default, 2-3° for subgrains, 10-15° high-angle cutoff. Added "Phase Handling" explaining different-phase and phase-0 behavior. Added "Mask Array" and "Periodic Option" sections. Added Required Input Sources. - -- [x] **RequireMinimumSizeFeaturesFilter** (SimplnxCore) - - **Clarity:** Core action (remove features below size threshold) clear, but "isotropically coarsened" is unexplained jargon. - - **Completeness:** Has good warnings (feature data invalidation, NeighborList removal); no guidance on typical minimum-size values. - - **Accessibility:** "Ensemble" bolded but not defined; "isotropically coarsened" needs a plain-language explanation. - - **Figures Needed:** Before/after showing small features removed and gaps filled; diagram of isotropic coarsening (neighbors growing outward uniformly). - - **Real-World Viz:** Before/after microstructure with minimum-size filter applied. - - **Units Clarity:** Minimum size is in **cells** (integer voxel count) — must be stated explicitly. - - **Concept Links:** feature cleanup, isotropic coarsening, Ensemble, NeighborList invalidation - - **Changes Made:** Added lead-in explaining typical use (discarding spurious single-cell grains after segmentation). Added "What is Isotropic Coarsening?" section with plain-language explanation. Added "Minimum Size Units" section stating units are **cells** (integer voxel count), with a formula for converting physical volume to cell count. Rewrote the "Single Phase" option more clearly. Kept existing warnings. Added Required Input Sources. - -- [x] **ComputeFeatureNeighborsFilter** (SimplnxCore) - - **Clarity:** Algorithm steps clear, but the multiple output arrays are described inline in running prose and hard to scan. - - **Completeness:** Mentions several outputs (neighbor count, shared surface area, boundary-cell count, surface-feature flag) but all buried in a single paragraph. - - **Accessibility:** Face-sharing neighbor concept not illustrated; implicit assumption that user knows what a NeighborList is. - - **Figures Needed:** Diagram showing feature-to-feature shared boundary with shared surface area highlighted. - - **Real-World Viz:** Grain map colored by number of neighbors; or shared-surface-area visualization. - - **Units Clarity:** Shared surface area is in **cell-face units** (dimensionless count of shared faces, not physical area) — should be stated. - - **Concept Links:** contiguous neighbors, Feature IDs, shared surface area, NeighborList - - **Changes Made:** Rewrote lead-in describing why downstream filters depend on this (misorientation stats, GBCD, twin merging, boundary strengths). Separated algorithm steps from output descriptions. Added "What This Filter Produces" section with bulleted list of the three main outputs plus two optional outputs; explained each output's downstream use. Explicitly stated shared surface area units are **cell-face count**, not physical area. Added Required Input Sources. - -### Tier 3 — Polish - -| Filter | Plugin | Status | -|--------|--------|--------| -| IdentifySample | SimplnxCore | Done | -| ComputeSurfaceFeatures | SimplnxCore | Done | -| ComputeBiasedFeatures | SimplnxCore | Done | -| RemoveFlaggedFeatures | SimplnxCore | Done | - -- [x] **IdentifySampleFilter** (SimplnxCore) - - **Clarity:** Clear purpose (remove overscan border), steps well explained, slice-by-slice option well documented. - - **Completeness:** Good example images showing when to use vs when not to use the filter. - - **Accessibility:** FIB-SEM mentioned but not critical to understanding; typo: "thresheld" should be "thresholded". - - **Figures Needed:** Already adequate; could add a before/after showing cleanup effect on a real overscanned dataset. - - **Real-World Viz:** Already has good/bad-dataset examples. - - **Concept Links:** thresholding, sample identification, contiguous regions - - **Changes Made:** Fixed typo "thresheld" → "thresholded". Rewrote intro prose for readability (shorter sentences, active voice). Added explicit link to Multi-Threshold Objects as the typical mask source. Added Required Input Sources. Title preserved as it matches the filter's humanName(). - -- [x] **ComputeSurfaceFeaturesFilter** (SimplnxCore) - - **Clarity:** Purpose clear; algorithm clear; two WARNING sections are helpful and well-placed. - - **Completeness:** Broken/truncated sentence on line 9: "the Cells that sit at either ." — should be completed or deleted. Intro paragraph and the "This Filter determines..." paragraph are partially redundant. - - **Accessibility:** Feature ID=0 convention referenced in a warning but could use an inline first-use definition. - - **Figures Needed:** Already has good example output images. - - **Real-World Viz:** Already present. - - **Concept Links:** Feature ID=0, surface features, bounding box - - **Changes Made:** Deleted broken truncated sentence. Consolidated the two redundant intro paragraphs into one. Added inline definition of Feature ID = 0 (the "unassigned / outside sample" convention) with cross-link to IdentifySample. Added cross-link to ComputeBiasedFeatures as the more rigorous approach to boundary bias. Added Required Input Sources. - -- [x] **ComputeBiasedFeaturesFilter** (SimplnxCore) - - **Clarity:** Algorithm and rationale well explained; before/after example figure is strong. - - **Completeness:** Good — explains why larger features are more likely to be biased and why bounding-box logic works. - - **Accessibility:** Most terms defined inline; credit to Dave Rowenhorst retained. - - **Figures Needed:** Already adequate. - - **Real-World Viz:** Already present. - - **Concept Links:** stereology, feature centroids, bounding-box bias, unbiased statistics - - **Changes Made:** Added "Why Bias Matters for Statistics" section explaining the size-dependent sampling bias — why excluding only surface-touching features still leaves bias, and why centroid-based bounding-box logic fixes it. Cross-linked ComputeSurfaceFeatures. Added Required Input Sources. - -- [x] **RemoveFlaggedFeaturesFilter** (SimplnxCore) - - **Clarity:** Three operations (Remove / Extract / Extract then Remove) well documented. - - **Completeness:** NeighborList warning present; no Example Pipeline link populated. - - **Accessibility:** "isotropically coarsened" is jargon (same issue as RequireMinimumSizeFeatures). - - **Figures Needed:** Before/after showing each of the three operation modes; at least a side-by-side of Remove vs Extract. - - **Real-World Viz:** Microstructure showing extraction vs removal outcomes. - - **Concept Links:** feature cleanup, isotropic coarsening, feature extraction - - **Changes Made:** Rewrote description to position this filter as the general-purpose flag-based removal tool and cross-reference RequireMinimumSizeFeatures for size-based use. Linked "isotropic coarsening" to the definition in RequireMinimumSizeFeatures rather than repeating. Cleaned up operation list phrasing. Converted NeighborList warning to sub-heading. Added Required Input Sources listing typical flag-producing filters (ComputeBiasedFeatures, ComputeSurfaceFeatures). - ---- - -## Batch 7: I/O Filters (Read/Write) - -**Plugin:** SimplnxCore + OrientationAnalysis -**Filters:** 44 (24 readers + 20 writers) - -> **Scope note:** The design spec estimated ~15 I/O filters. The actual count is **44** once the full set of EBSD/HDF5 readers (Ang, Ctf, Channel5, H5Ebsd, H5Esprit, H5Oim, H5Oina, EnsembleInfo, GrainMapper3D) and the format-specific exporters (Abaqus, LAMMPS, SPParks, LosAlamos FFT, VTK, Avizo, INL, GBCD, StatsGen ODF, etc.) are included. Rewrites will be committed at each tier boundary, and the large Tier 2 group may be split into sub-commits by sub-theme (core data I/O, image/volume readers, simulation exporters, VTK/Avizo, EBSD readers, OrientationAnalysis writers). - -### Tier 1 — Critical - -| Filter | Plugin | Status | -|--------|--------|--------| -| ReadH5OinaData | OrientationAnalysis | Done | -| WriteGBCDGMTFile | OrientationAnalysis | Done | - -- [x] **ReadH5OinaDataFilter** (OrientationAnalysis) - - **Clarity:** Reads a single Oxford Aztec `.h5oina` file; vendor clearly identified. FORMAT VERSION 2.0 limitation and the Read HDF5 Dataset workaround are useful. - - **Completeness:** STRUCTURAL DEFECT (fixed) — the doc had hand-written `## Parameters` and `## Created Objects` tables and was missing the auto-table marker entirely. - - **Accessibility:** Same undefined EBSD jargon (Euler angles, crystal structures, reference frames) as the other EBSD readers. - - **Figures Needed:** Has Figure 1 (Hexagonal alignment) + a UI overview; a sample/crystal reference-frame diagram would help. - - **Real-World Viz:** IPF map of imported Aztec data. - - **Units Clarity:** `.h5oina` angles stated as radians (good); Z Spacing units (microns) now live in the parameter help text (the auto-table). - - **Concept Links:** Euler angles (Bunge Z-X-Z), crystal structures/Ensemble, lattice constants, sample/crystal reference frame, hexagonal symmetry, H5OINA v2.0 format - - **Changes Made:** DELETED the hand-written `## Parameters` and `## Created Objects` tables and INSERTED the `% Auto generated parameter table` marker (so the real parameter table now injects). Added a "What This Filter Produces" prose paragraph explaining the imported EBSD data in plain language (orientation as Euler angles, per-pixel phase, pattern-quality metrics, per-phase Ensemble data) instead of a stale type table. Fixed the phi2 +30° bullet that wrongly referenced "(.ctf) files" → "`.h5oina` files". Reworded the historical reference-frame note from ".ctf file" to "Oxford data". Converted the two `{ref}` rotation links and the bold-only "Threshold Objects" to MyST links; fixed the Read HDF5 Dataset link to the cross-plugin `../SimplnxCore/` form. Added a "Downstream Processing" note cross-linking Convert Orientation Representation. Fixed typo "agment" → "augment". Normalized footer to "## DREAM3D-NX Help". - -- [x] **WriteGBCDGMTFileFilter** (OrientationAnalysis) - - **Clarity:** Was one sentence; never explained what a GBCD pole figure is, what the `.dat` contains, or why an engineer would want one. - - **Completeness:** No parameter explanation beyond "phase index"; the Misorientation Axis-Angle parameter was undocumented. No Required Input Sources. - - **Accessibility:** "GBCD", "GMT", "pole figure", "phase index" all undefined. GMT URL (`gmt.soest.hawaii.edu`) was stale. - - **Figures Needed:** Data-flow diagram: GBCD data (from Compute GBCD) → this filter → `.dat` → GMT → rendered pole figure (shows the external toolchain). - - **Real-World Viz:** Already has a rendered GMT pole figure image (caption now explains MRD contours). - - **Units Clarity:** Phase index now stated as 1-based dimensionless index; misorientation angle stated in degrees, axis (h,k,l) dimensionless. - - **Concept Links:** GBCD, GMT, pole figure, stereographic projection, Ensemble/phase data, MRD - - **Changes Made:** Major rewrite. Added "What is a GBCD Pole Figure?" defining grain boundary, GBCD, misorientation, pole figure, and MRD in plain language. Added "What This Filter Does" making explicit that GMT is an external toolkit and this filter only writes its input `.dat` file (with the Compute GBCD → this filter → GMT toolchain spelled out). Added "Parameter Guidance" documenting Phase of Interest (1-based index), the previously-undocumented Misorientation Axis-Angle (angle in degrees + crystal axis h,k,l, with the Σ3 example), and Output GMT File. Added a Required Input Sources section naming [Compute GBCD](ComputeGBCDFilter.md) (GBCD) and the EBSD readers (Crystal Structures). Fixed the GMT URL to generic-mapping-tools.org. Improved the figure caption to explain the MRD contours. - -### Tier 2 — Important - -| Filter | Plugin | Status | -|--------|--------|--------| -| ReadHDF5Dataset | SimplnxCore | Done | -| WriteBinaryData | SimplnxCore | Done | -| ReadStlFile | SimplnxCore | Done | -| ReadVtkStructuredPoints | SimplnxCore | Done | -| ReadVolumeGraphicsFile | SimplnxCore | Done | -| WriteAbaqusHexahedron | SimplnxCore | Done | -| WriteLAMMPSFile | SimplnxCore | Done | -| WriteSPParksSites | SimplnxCore | Done | -| WriteLosAlamosFFT | SimplnxCore | Done | -| ReadDeformKeyFileV12 | SimplnxCore | Done | -| WriteVtkRectilinearGrid | SimplnxCore | Done | -| WriteVtkStructuredPoints | SimplnxCore | Done | -| WriteAvizoRectilinearCoordinate | SimplnxCore | Done | -| WriteAvizoUniformCoordinate | SimplnxCore | Done | -| ReadDREAM3D | SimplnxCore | Done | -| ReadEnsembleInfo | OrientationAnalysis | Done | -| ReadAngData | OrientationAnalysis | Done | -| ReadCtfData | OrientationAnalysis | Done | -| ReadChannel5Data | OrientationAnalysis | Done | -| ReadH5Ebsd | OrientationAnalysis | Done | -| ReadH5EspritData | OrientationAnalysis | Done | -| ReadH5OimData | OrientationAnalysis | Done | -| WriteGBCDTriangleData | OrientationAnalysis | Done | -| WritePoleFigure | OrientationAnalysis | Done | - -- [x] **ReadHDF5DatasetFilter** (SimplnxCore) - - **Clarity:** Element-count matching rule is explained, but the worked examples confuse more than clarify (Example 1 is an error case; Example 2's arithmetic is internally inconsistent). - - **Completeness:** Missing description of key parameters (which dataset(s) to select, the file path, multi-dataset import in one pass). Only the dimension math is covered. - - **Accessibility:** Assumes the reader knows HDF5 dataset vs group; "component/tuple dimensions", "attribute matrix" undefined. - - **Figures Needed:** Diagram of a flat HDF5 dataset (N elements) reshaped into tuples × components. - - **Real-World Viz:** Screenshot of the HDF5 tree dataset-selection UI. - - **Units Clarity:** N/A (element counts, dimensionless). - - **Concept Links:** HDF5 dataset/group, component dimensions, tuple dimensions, attribute matrix, total element count - - **Changes Made:** Added a one-line definition of HDF5 and first-use definitions of tuple vs component dimensions. Described the dataset-selection behavior and the optional Attribute-Matrix parent placement in prose. Replaced the three inconsistent worked examples (including the '4-dims-labeled-3D' / mismatched-arithmetic case) with three internally consistent ones built on the rule total-element-count = tuples x components. Added Required Input Sources (None -- external .h5/.hdf5 file) and a License & Copyright section. - -- [x] **WriteBinaryDataFilter** (SimplnxCore) - - **Clarity:** Thin — one sentence plus an Endianess section. Does not state output file naming, extension, multi-component behavior, or that no header is written. - - **Completeness:** Only Endianess documented. Missing output directory/naming, raw-no-header behavior, on-disk component ordering, file extension parameter. - - **Accessibility:** The Endianess section itself is clear; the problem is omission. - - **Figures Needed:** None. - - **Real-World Viz:** Optional screenshot of the output files (low priority). - - **Units Clarity:** N/A. - - **Concept Links:** endianness, raw binary (no header), component ordering, output file naming - - **Changes Made:** Expanded the thin description to the actual behavior (one raw file per **Data Array**, `` naming with default `.bin`, interleaved components, no header written). Added the explicit warning that the user must record type/dims/endianness, and a cross-link to read the files back with [Read Raw Binary](ReadRawBinaryFilter.md). Italicized the endianness values; fixed the run-on/period grammar. - -- [x] **ReadStlFileFilter** (SimplnxCore) - - **Clarity:** Dives straight into binary layout/attribute-byte-count without first stating in plain terms that STL is a triangle surface mesh and the output is a Triangle Geometry. - - **Completeness:** Covers binary format, vendor detection, strictness, a Python fix script. Never states ASCII STL is unsupported; no mention of produced arrays/normals. - - **Accessibility:** Heading typo "## IMPORANT NOTES". Strictness warning repeated verbatim twice. Aimed at STL-format experts. - - **Figures Needed:** Illustration of one triangle (normal + 3 vertices) and how triangles form a surface mesh. - - **Real-World Viz:** Screenshot of an STL surface as a Triangle Geometry. - - **Units Clarity:** STL is unitless — one sentence stating coordinates are in the file author's units (dimensionless to the reader). - - **Concept Links:** Triangle Geometry, surface mesh, STL binary format, attribute byte count, face normals - - **Changes Made:** Added a plain-language opening (an STL file is a list of triangles -> **Triangle Geometry**, with Face Normals/Labels). Fixed the 'IMPORANT NOTES' heading typo. Added an explicit binary-only / ASCII-rejected caveat (confirmed against preflight). Collapsed the duplicated attribute-byte-count warning; noted STL coordinates are unitless. Required Input Sources: None; fixed the Wikipedia anchor. - -- [x] **ReadVtkStructuredPointsFilter** (SimplnxCore) - - **Clarity:** Core idea clear (legacy VTK structured-points → Image Geometry) but POINT_DATA vs CELL_DATA and "eight values per voxel" will confuse non-experts without a figure. Uses legacy SIMPL "Data Container" wording. - - **Completeness:** Explains attribute types, binary/ASCII, point vs cell data with an example. Missing: behavior when both POINT_DATA and CELL_DATA present (output naming/paths). - - **Accessibility:** "STRUCTURED_POINTS", "POINT_DATA", "CELL_DATA", "LOOKUP_TABLE" only partly explained. - - **Figures Needed:** A voxel cube showing cell-center value vs the 8 corner (point) values. - - **Real-World Viz:** Screenshot of the example GrainIds dataset as an Image Geometry. - - **Units Clarity:** State spacing/origin come from the file; length units file-defined. - - **Concept Links:** Image Geometry, Cell Attribute Matrix, structured points, point vs cell data, voxel corners vs center, SCALARS/VECTORS - - **Changes Made:** Replaced legacy 'Data Container' with **Image Geometry** throughout. Added a Point Data vs Cell Data subsection and described the two-output naming (`VTK Cell Data` / `VTK Point Data`) when both are present. Added the missing DREAM3D-NX Help footer and a License section, plus a Spacing/Origin/Units note. Required Input Sources: None. - -- [x] **ReadVolumeGraphicsFileFilter** (SimplnxCore) - - **Clarity:** Thin one-paragraph description; a non-expert won't know "Volume Graphics" data is a CT/VGStudio export. - - **Completeness:** Underspecified — no output geometry type, volume data type, cropping support, or created array named. Only caveat is "both files in same directory". - - **Accessibility:** "Volume Graphics data files" is a vendor term used without explanation. - - **Figures Needed:** Optional small diagram of the `.vgi` (metadata) + `.vol` (raw voxels) pairing. - - **Real-World Viz:** Screenshot of a CT volume from a `.vgi`/`.vol` pair. - - **Units Clarity:** Says units come from the `.vgi`; expand to state which units and that spacing/origin derive from it. - - **Concept Links:** Image Geometry, `.vgi`/`.vol` file pair, spacing/dimensions/units, CT volume - - **Changes Made:** Normalized the group to IO (Input). Defined Volume Graphics data (.vgi metadata + .vol raw voxels; a CT/VGStudio export). Stated the output is an **Image Geometry** with a single float32 density Cell array and origin (0,0,0) (confirmed from preflight); noted no sub-volume extraction. Added an Example Pipelines heading and Required Input Sources: None. - -- [x] **WriteAbaqusHexahedronFilter** (SimplnxCore) - - **Clarity:** Non-expert won't know Abaqus is a commercial FEA package or why one exports a voxel grid to it. Jumps into `.inp` files without framing the Image-Geometry → hexahedral-mesh conversion. - - **Completeness:** Five output files listed but their relationship (master `*Include`s the others) only implied. The `Write Dummy Node` parameter referenced at top but never explained. - - **Accessibility:** "Abaqus", ".inp", "hexahedron", "C3D8", "Hourglass Stiffness", "stress-strain" undefined. The huge raw `_elset.inp` dump hurts readability. - - **Figures Needed:** Diagram of a voxel grid mapping to a hexahedral mesh (8 nodes per voxel). - - **Real-World Viz:** Screenshot of the exported mesh in Abaqus/CAE colored by grain. - - **Units Clarity:** Node coordinates are physical units (matching spacing) but never stated; "Hourglass Stiffness 250" has no units/explanation. - - **Concept Links:** Abaqus/FEA, hexahedral mesh, Image Geometry, FeatureIds, element sets (grains), nodes/elements - - **Changes Made:** Defined Abaqus and FEA; framed the filter as converting an **Image Geometry** voxel grid into a C3D8 8-node hexahedral mesh. Documented the five `.inp` files and the master-file `*Include` relationship, the *Write Dummy Node* and *Hourglass Stiffness* parameters, and that node coordinates are in physical units. Trimmed the large `_elset.inp` dump; added Required Input Sources (FeatureIds + Image Geometry). - -- [x] **WriteLAMMPSFileFilter** (SimplnxCore) - - **Clarity:** Non-expert won't know LAMMPS is a molecular-dynamics code. Purpose stated but why/when unclear. - - **Completeness:** No output-format example. "Example Pipelines" heading empty. Names "Insert Atoms" as producer (good, but informal). - - **Accessibility:** "LAMMPS" never expanded. Legacy "Vertex Data Container" wording. Self-reference "Export LAMMPS Filter" is the wrong name. - - **Figures Needed:** Optional before/after of features filled with atoms. - - **Real-World Viz:** Screenshot of the resulting atom configuration (OVITO/VMD). - - **Units Clarity:** Atom-coordinate units undocumented; state them. - - **Concept Links:** LAMMPS/molecular dynamics, atomistic representation, Vertex Geometry, Insert Atoms - - **Changes Made:** Expanded LAMMPS (Sandia molecular-dynamics code); replaced 'Vertex Data Container' with **Vertex Geometry**; fixed the 'Export LAMMPS Filter' self-reference. Added an output-format example and atom-coordinate units. Added Required Input Sources (Vertex Geometry, e.g. from Insert Atoms -- referenced in plain text since no Insert Atoms doc exists in the repo). Removed the empty Example Pipelines heading. - -- [x] **WriteSPParksSitesFilter** (SimplnxCore) - - **Clarity:** Terse and oddly worded ("LINE 4 evidently must be what is shown"); non-expert won't understand a "sites" file or site/value pairs. - - **Completeness:** No Required Input Sources (needs FeatureIds on an Image Geometry). Line-by-line explanation partial (lines 1, 8 unexplained). - - **Accessibility:** "SPPARKS" linked but not contextualized as a Kinetic Monte Carlo grain-growth simulator. "Kinetic Monte Carlo", "max neighbors", "xlo xhi" undefined. Header style `## Description ##` inconsistent. - - **Figures Needed:** None. - - **Real-World Viz:** None. - - **Units Clarity:** Dimension lines (0 200) are cell-index extents (dimensionless cell counts) — clarify vs physical units. - - **Concept Links:** SPPARKS/Kinetic Monte Carlo, sites, FeatureIds, Image Geometry, Cells - - **Changes Made:** Defined SPPARKS (Sandia Kinetic Monte Carlo microstructure simulator); rewrote the tentative 'LINE 4 evidently' wording authoritatively. Explained site/value = cell/FeatureId, clarified the box-extent lines are dimensionless cell-index counts, and explained the blank separator lines. Normalized '## X ##' headings; added Required Input Sources (FeatureIds on an Image Geometry). - -- [x] **WriteLosAlamosFFTFilter** (SimplnxCore) - - **Clarity:** Output column layout (Phi1 Phi Phi2 X Y Z Feature_ID Phase_ID) explained well, but non-expert won't know what an "FFT 3D simulation code" is for or what Euler angles/phases are. - - **Completeness:** Good format detail and a real reference. No Required Input Sources (needs Euler angles, FeatureIds, Phases on an Image Geometry). Empty "Example Pipelines" heading. - - **Accessibility:** "Euler angles", "Phi1/Phi/Phi2", "phase", "FFT" undefined for a general reader. - - **Figures Needed:** None. - - **Real-World Viz:** None. - - **Units Clarity:** Excellent — Euler angles in degrees, X/Y/Z integer indices, IDs from 1. Model for the batch; keep. - - **Concept Links:** FFT/crystal plasticity, Euler angles, phases, FeatureIds, Image Geometry, CellData - - **Changes Made:** Contextualized Euler angles and the Lebensohn FFT crystal-plasticity code; preserved the existing (model) units paragraph. Added Required Input Sources (Euler angles from an EBSD reader, FeatureIds, Phases) and removed the empty Example Pipelines heading. - -- [x] **ReadDeformKeyFileV12Filter** (SimplnxCore) - - **Clarity:** Two sentences. Non-expert won't know DEFORM is a metal-forming FEA package or what a "key file" is; can't tell what data results. - - **Completeness:** Very thin. Lists variable names (stress, strain, ndtmp) without explanation. Does not describe the resulting DataStructure (a Quad Geometry plus cell/node attribute matrices). - - **Accessibility:** "DEFORM v12", "key file", "ndtmp", "quadrilateral mesh", legacy "Data Container" all undefined. - - **Figures Needed:** Optional screenshot of the imported quad mesh colored by a variable. - - **Real-World Viz:** Render of the loaded DEFORM mesh colored by stress/strain. - - **Units Clarity:** N/A (importer); note imported variable units are whatever DEFORM exported. - - **Concept Links:** DEFORM/FEA, key file, Quad Geometry, nodes/cells, cell vs node variables - - **Changes Made:** Defined DEFORM (metal-forming finite-element package) and a v12 key file. Described the created **Quad Geometry** plus node- and cell-level arrays (ndtmp = nodal temperature); replaced 'Data Container' with Quad Geometry. Added an example key-file snippet; Required Input Sources: None. - -- [x] **WriteVtkRectilinearGridFilter** (SimplnxCore) - - **Clarity:** Never says what a "VTK legacy file" or "RECTILINEAR_GRID dataset" is, nor why one wants this (ParaView/VisIt interop). - - **Completeness:** Single sentence. No ASCII vs binary mention, vertex vs cell data, extension, or that input must be an Image Geometry. - - **Accessibility:** "VTK legacy file", "RECTILINEAR_GRID" undefined. - - **Figures Needed:** Optional shared diagram contrasting rectilinear-grid vs structured-points (variable vs uniform spacing). - - **Real-World Viz:** Screenshot of the `.vtk` in ParaView. - - **Units Clarity:** N/A in prose; confirm in param table. - - **Concept Links:** VTK legacy format, ASCII vs binary VTK, RectilinearGrid vs Image Geometry, ParaView/VisIt - - **Changes Made:** Expanded the one-sentence description; defined VTK legacy format and RECTILINEAR_GRID and named ParaView/VisIt. Explained the **Image Geometry** input is written as an explicit-coordinate rectilinear grid; documented ASCII vs binary. Added Required Input Sources (Image Geometry + cell arrays) and a cross-link to the structured-points variant. - -- [x] **WriteVtkStructuredPointsFilter** (SimplnxCore) - - **Clarity:** Assumes the reader knows STRUCTURED_POINTS. The one useful caveat ("only writes cell data") is present. - - **Completeness:** Very thin. Doesn't explain how structured points differs from rectilinear grid, or when to pick this writer. - - **Accessibility:** "VTK legacy file", "STRUCTURED_POINTS" undefined; indistinguishable from the rectilinear writer to a non-expert. - - **Figures Needed:** Shared diagram (uniform structured points vs variable rectilinear spacing). - - **Real-World Viz:** Screenshot in ParaView. - - **Units Clarity:** N/A in prose. - - **Concept Links:** VTK legacy format, structured points vs rectilinear grid, uniform spacing, ParaView/VisIt - - **Changes Made:** Defined STRUCTURED_POINTS (uniform-spacing image data) with the same VTK/ParaView framing; kept the cell-data-only caveat. Added a 'when to use vs [Write Vtk Rectilinear Grid]' note and ASCII/binary documentation; added Required Input Sources. - -- [x] **WriteAvizoRectilinearCoordinateFilter** (SimplnxCore) - - **Clarity:** Confusing opening ("Values should be present from segmentation … cannot be determined by this filter") — vague about what "values" means (the FeatureIds array). - - **Completeness:** Helpful raw AmiraMesh header example, but never says what Avizo is, what FeatureIds are, or that input is an Image Geometry + Int32 FeatureIds. Example contains garbled DateTime bytes. - - **Accessibility:** "Avizo", "AmiraMesh", "Rectilinear Coordinate", "FeatureIds", "segmentation" undefined. - - **Figures Needed:** None. - - **Real-World Viz:** Screenshot of the output in Avizo. - - **Units Clarity:** Example shows "Coordinates microns" — note the Avizo file hardcodes microns regardless of the geometry's actual units (caveat). - - **Concept Links:** Avizo/AmiraMesh format, FeatureIds, segmentation, RectilinearGrid vs Image Geometry, units (microns) - - **Changes Made:** Rewrote the confusing opening to state the **Image Geometry** + Int32 FeatureIds input plainly; defined Avizo and AmiraMesh. Cleaned the garbled DateTime bytes in the header example; added the hardcoded-microns caveat and ASCII/binary note; added a Group line, Required Input Sources, and a link to the Uniform variant. - -- [x] **WriteAvizoUniformCoordinateFilter** (SimplnxCore) - - **Clarity:** Same vague "Values should be present…" sentence; doesn't clarify FeatureIds input or how "Uniform" differs from "Rectilinear". - - **Completeness:** Example header present but contains empty/garbled fields (blank DateTime, truncated Content/BoundingBox). Avizo/AmiraMesh undefined. - - **Accessibility:** Same undefined jargon as the rectilinear variant. - - **Figures Needed:** None. - - **Real-World Viz:** Screenshot in Avizo. - - **Units Clarity:** "Coordinates microns" hardcoded; caveat needed. BoundingBox units/order should tie to spacing/origin. - - **Concept Links:** Avizo/AmiraMesh format, uniform vs rectilinear coordinates, FeatureIds, BoundingBox, units (microns) - - **Changes Made:** Same clarifications as the Rectilinear variant (FeatureIds input, Avizo/AmiraMesh definitions, microns caveat, cleaned garbled example fields). Added the one-line contrast (single uniform spacing/bounding box vs per-axis coordinate arrays) and a link to the Rectilinear variant; added Required Input Sources. - -- [x] **ReadDREAM3DFilter** (SimplnxCore) - - **Clarity:** Core action clear (reads the DataStructure from a `.dream3d` HDF5 file), but omits the most important behavior: selective/partial import. - - **Completeness:** Missing partial-import behavior, merge-into-current-DataStructure and name-collision behavior, pipeline-metadata read. Lacks the "Group (Subgroup)" header other docs carry. No Required Input Sources statement. - - **Accessibility:** ".dream3d", "HDF5", "legacy" used without a one-line gloss. - - **Figures Needed:** None. - - **Real-World Viz:** Screenshot of the import-data tree selector showing partial-import checkboxes. - - **Units Clarity:** N/A. - - **Concept Links:** `.dream3d` file, HDF5, DataStructure import, partial import, legacy SIMPL vs NX format - - **Changes Made:** Added the missing Group header (IO (Input)) and the key selective/partial-import behavior; explained imported objects merge into the current **DataStructure** at their paths (overwrite on collision) and that pipeline metadata may be read. Glossed .dream3d/HDF5/'legacy'; added Required Input Sources (None) and the standard footer. - -- [x] **ReadEnsembleInfoFilter** (OrientationAnalysis) - - **Clarity:** Mechanics (format, sections, keys) well laid out with tables and a worked example, but the *purpose* is buried in EBSD jargon. - - **Completeness:** Strong on format/enum tables/phase-numbering caveat. Missing: what an Ensemble attribute matrix is, that this filter creates one, where the output goes; `.ini`/`.txt` are the same format. - - **Accessibility:** Main blocker — Ensemble, Feature, Cell, Crystal Structure, Phase Type, Laue class all undefined. - - **Figures Needed:** None (tables serve well). - - **Real-World Viz:** None high-value. - - **Units Clarity:** N/A (enumerated integer codes, well documented). - - **Concept Links:** Ensemble Info, Ensemble/phase attribute matrix, Crystal Structure / Laue class, Phase Type, Feature vs Cell, ASCII `.ini`/`.txt` - - **Changes Made:** Added a plain-language opening defining *ensemble* = phase and clarifying the filter reads a .ini/.txt file and writes an **Ensemble Attribute Matrix** into an existing Data Container. Added one-line glosses for Crystal Structure and Phase Type before the enum tables; documented downstream statistics consumers; removed the empty Example Pipelines heading. - -- [x] **ReadAngDataFilter** (OrientationAnalysis) - - **Clarity:** Reads a single EDAX/TSL `.ang` into an Image Geometry. Good, but never explains what the imported data MEANS (Euler angles, phases, CI, IQ named but undefined). - - **Completeness:** Field-ordering block useful but raw jargon; does not say phi1/Phi/phi2 ARE the three Euler angles (Bunge). Mask/threshold guidance present. - - **Accessibility:** Heavy undefined jargon (Euler angles, crystal/sample reference frame, Confidence Index, Image Quality, phase, ODF, IPF, hexagonal grid). - - **Figures Needed:** Sample vs crystal reference-frame diagram. - - **Real-World Viz:** IPF map of a freshly imported Small IN100 `.ang` slice. - - **Units Clarity:** Weak — should state `.ang` angles are in radians; step size/origin in microns; reference-frame rotations in degrees. - - **Concept Links:** Euler angles (Bunge), sample/crystal reference frame, confidence index, image quality, phases, IPF/ODF, hex-vs-square grid - - **Changes Made:** Glossed phi1/Phi/phi2 as the three Euler angles; defined crystal/sample reference frames and Confidence Index/Image Quality. Fixed 'he following'; converted the Hex-Grid `{ref}` and the bold-only 'Threshold Objects' to MyST links; fixed the broken 'Ensemble Attribute Matrix' bold; added Downstream Processing and Required Input Sources. - -- [x] **ReadCtfDataFilter** (OrientationAnalysis) - - **Clarity:** Clearly reads an Oxford/HKL `.ctf` into an Image Geometry. Good vendor ID. - - **Completeness:** Good on HKL default transforms, radians/degrees conversion, hexagonal phi2 +30°. Created-array meanings not explained. - - **Accessibility:** Same undefined jargon; hexagonal section highly expert-oriented (flagged as edge case). - - **Figures Needed:** Has Figure 1 (Hexagonal_Axis_Alignment); a reference-frame diagram would still help. - - **Real-World Viz:** IPF map before/after the +30° phi2 correction, or correct-vs-wrong radians/degrees. - - **Units Clarity:** Better than Ang (notes `.ctf` usually degrees, NX expects radians); still missing origin/step-size (microns). - - **Concept Links:** Euler angles, sample/crystal reference frame, radians vs degrees, hexagonal symmetry/Bravais lattice, pole figure/IPF - - **Changes Made:** Fixed the intro broken bold and the mangled Figure 1 caption; marked the hexagonal-alignment section Advanced. Converted `{ref}` and bold-only filter mentions to MyST links; added Downstream Processing (Convert Orientation Representation) and Required Input Sources. - -- [x] **ReadChannel5DataFilter** (OrientationAnalysis) - - **Clarity:** Reads an Oxford Channel 5 `.cpr`/`.crc` pair into an Image Geometry. Uses `ImageGeometry`/`ImageDataContainer` as code, inconsistent with the **Image Geometry** bolding used elsewhere. - - **Completeness:** Mirrors the Ctf doc; good note that `.cpr`/`.crc` angles are radians and that Error=0 marks valid TRUE. Created-array meanings not explained. - - **Accessibility:** Same undefined jargon; hexagonal section copied verbatim and wrongly says "(.ctf) files". - - **Figures Needed:** Has Figure 1; reference-frame diagram still helpful. - - **Real-World Viz:** IPF map of an imported Channel 5 dataset. - - **Units Clarity:** Good on radians; missing origin/step-size (microns). - - **Concept Links:** Euler angles, sample/crystal reference frame, hexagonal symmetry, pole figures, IPF, `.cpr`/`.crc` pair - - **Changes Made:** Fixed the broken bold and caption; corrected the phi2 +30-degree bullet that wrongly referenced '.ctf' to Channel 5 (.cpr/.crc). Standardized code-style 'ImageGeometry' to **Image Geometry**; removed the empty Example Pipelines heading; converted links to MyST and added Required Input Sources. - -- [x] **ReadH5EbsdFilter** (OrientationAnalysis) - - **Clarity:** Good — states it reads a `.h5ebsd` built by *Import Orientation File(s) to H5EBSD*; the UI screenshot + checkbox/slice-subset/transform description is genuinely helpful. The one reader with a real upstream dependency. - - **Completeness:** Strong on transforms, angle representation, slice subsetting, correct-vs-incorrect IPF example. Created-array meanings rely on auto-table (correct). Could note headless parameter equivalents for UI features. - - **Accessibility:** Same domain jargon undefined, but the IF-steel correct/incorrect example is excellent. - - **Figures Needed:** Already has UI + correct/incorrect + Hexagonal alignment figures; a reference-frame concept diagram would still help. - - **Real-World Viz:** Already present (IF-steel IPF maps). - - **Units Clarity:** Reference-frame rotations in degrees OK; lines 42/46 are missing the word "degrees" ("apparent 30 shifts", "add 30 to phi2"). - - **Concept Links:** H5Ebsd intermediate format, Euler angle representation (deg/rad), sample/crystal reference frame, slice stacking/subsetting, hexagonal symmetry, IPF - - **Changes Made:** Added a Required Input Sources section stating the .h5ebsd must be built by [Import Orientation File(s) to H5EBSD](EbsdToH5EbsdFilter.md). Converted the three `{ref}` links to MyST; fixed the truncated '30' -> '30 degrees' (two places) and 'excellant' typo; marked the hexagonal section Advanced; added Downstream Processing. - -- [x] **ReadH5EspritDataFilter** (OrientationAnalysis) - - **Clarity:** Clearly reads a single Bruker Nano Esprit `.h5`; before/after UI screenshots for Z Spacing and scan selection help. - - **Completeness:** Good — documents Z Spacing, multi-scan selection, X/Y Step=0 → 1.0 fallback, recommended rotation filters. "MAD" used without expansion. - - **Accessibility:** Same undefined jargon; "MAD" (Mean Angular Deviation) not expanded. - - **Figures Needed:** Has two UI screenshots; reference-frame diagram would help. - - **Real-World Viz:** IPF map of an imported multi-scan Bruker volume (illustrate Z-stacking). - - **Units Clarity:** Best of the `.h5` readers — Z Spacing "microns between each layer". Origin units not stated. - - **Concept Links:** Euler angles, sample/crystal reference frame, scan/slice (Z) stacking, MAD/image quality, Z spacing (microns), multi-scan files - - **Changes Made:** Expanded MAD -> Mean Angular Deviation on first use; converted all `{ref}` links to MyST; noted the zero-step fallback is 1.0 micron. Fixed the broken bold; added Downstream Processing and Required Input Sources; removed the empty Example Pipelines heading. - -- [x] **ReadH5OimDataFilter** (OrientationAnalysis) - - **Clarity:** Clearly reads an EDAX OIMAnalysis `.h5`. UI screenshots referenced for Z Spacing / scan selection. - - **Completeness:** Good — multi-slice import, the valuable unique "Perform Slice By Slice Transform" caveat, thresholding. Created-array meanings rely on auto-table (correct). - - **Accessibility:** Same undefined jargon; slice-by-slice transform note is clear and practical. - - **Figures Needed:** Two UI screenshots; reference-frame diagram would help. - - **Real-World Viz:** IPF map of imported multi-slice EDAX `.h5` showing the slice-by-slice transform consequence. - - **Units Clarity:** Weaker than Esprit — Z Spacing units (microns) not stated here; origin units absent. - - **Concept Links:** Euler angles, sample/crystal reference frame, slice-by-slice transform, scan/slice stacking, confidence index/image quality, IPF - - **Changes Made:** Stated Z Spacing and Origin units as microns (confirmed from parameters); cross-linked the 'Perform Slice By Slice Transform' mention to Rotate Sample Reference Frame. Converted `{ref}` links to MyST, fixed the broken bold, defined CI/IQ; added Downstream Processing and Required Input Sources; removed the empty Example Pipelines heading. - -- [x] **WriteGBCDTriangleDataFilter** (OrientationAnalysis) - - **Clarity:** Decent — writes per-triangle GBCD info (inward/outward Euler angles, normals, areas) with sample output. A general reader won't know why this matters or what "inward/outward" means. - - **Completeness:** Good example with column legend, but the legend has a numbering bug ("Column 8: surface area" should be Column 10; cols 7-9 are the normal). No parameter explanation, no Required Input Sources. - - **Accessibility:** "GBCD", "Euler angles", "triangle normals", "right/left hand average orientation" undefined. Cites Rohrer without a reference link. - - **Figures Needed:** Boundary-triangle diagram: two grains, their average orientations (left/right), triangle normal, area — maps to the columns. - - **Real-World Viz:** Optional screenshot of a surface mesh colored by feature. - - **Units Clarity:** Euler angles stated as RADIANS (good). Normals are unitless direction cosines — say so. Area units (square microns?) not stated. - - **Concept Links:** GBCD, Euler angles, triangle/surface mesh, feature average orientation, crystal structures - - **Changes Made:** Rewrote the description in plain language (grain boundary, **Triangle Geometry**, Euler angles, normal, area). Fixed the column-legend numbering bug (surface area is column 10, not 8). Stated units (Euler radians, unitless normals, square-micron areas), defined inward/outward (left/right) orientations, added a Rohrer citation, and added Required Input Sources naming each producer. NOTE: the same column-legend bug also exists in the C++ algorithm's emitted file header. - -- [x] **WritePoleFigureFilter** (OrientationAnalysis) - - **Clarity:** Output Options sections clear and well organized, but never defines "pole figure"; "modified Lambert square", "unit circle interpolation", "Laue Class" appear undefined. - - **Completeness:** Good coverage + example images. Missing Required Input Sources. The "In a practical sense…" bullet list (Cell Euler Angles, Phases, Mask; Ensemble Laue Class, Material Names) is effectively a hand-written input-array list — borderline defect; reframe as Required Input Sources. - - **Accessibility:** "pole figure", "Lambert projection/square", "Laue Class", "Ensemble", "stereographic" undefined. No MyST links. - - **Figures Needed:** Conceptual diagram of the stereographic/Lambert projection (3D orientation → 2D unit circle). - - **Real-World Viz:** Already shows colorized vs discrete examples; optionally a labeled example (color bar, hemisphere label, phase name). - - **Units Clarity:** "Lambert Image Size (Pixels)" and output size are pixels — state in prose. Euler angles radians (good). - - **Concept Links:** pole figure, Lambert projection, stereographic projection, Laue class/crystal structure, Ensemble/Feature data, IPF, GBCD - - **Changes Made:** Added a plain-language pole-figure definition and first-use definitions of Lambert/stereographic projection and Laue Class. Converted the hand-written 'In a practical sense' input-array list into a proper Required Input Sources section; stated pixel units for image sizes; matched the doc to the current refactored parameters; kept the example images. - -### Tier 3 — Polish - -| Filter | Plugin | Status | -|--------|--------|--------| -| ReadCSVFile | SimplnxCore | Done | -| ReadStringDataArray | SimplnxCore | Done | -| ReadTextDataArray | SimplnxCore | Done | -| WriteASCIIData | SimplnxCore | Done | -| WriteFeatureDataCSV | SimplnxCore | Done | -| ReadImage | SimplnxCore | Done | -| ReadImageStack | SimplnxCore | Done | -| ReadNIfTIFile | SimplnxCore | Done | -| ReadZeissTxmFile | SimplnxCore | Done | -| ReadBinaryCTNorthstar | SimplnxCore | Done | -| WriteNodesAndElementsFiles | SimplnxCore | Done | -| WriteStlFile | SimplnxCore | Done | -| ReadGrainMapper3D | OrientationAnalysis | Done | -| WriteINLFile | OrientationAnalysis | Done | -| WriteStatsGenOdfAngleFile | OrientationAnalysis | Done | - -- [x] **ReadCSVFileFilter** (SimplnxCore) - - **Changes Made:** Converted the bold-only 'Combine Attribute Arrays' mention to a MyST link; gave the 10 identical image alt-texts unique descriptive alts; added Required Input Sources (None). - -- [x] **ReadStringDataArrayFilter** (SimplnxCore) - - **Changes Made:** Fixed typos 'Created Arra Path' and 'mulplying'; converted the quoted 'Read CSV Filter' to a MyST link; added Required Input Sources (None). - -- [x] **ReadTextDataArrayFilter** (SimplnxCore) - - **Changes Made:** Fixed typos 'Created Arra Path', 'mulplying', 'Componenets'; converted 'Read CSV Filter' to a MyST link; added a descriptive image alt; confirmed via the cpp there is no 'New Line' delimiter option here; added Required Input Sources (None). - -- [x] **WriteASCIIDataFilter** (SimplnxCore) - - **Changes Made:** Cleaned the cluttered Group line to 'IO (Output)'; added a 'Maximum Tuples Per Line' description (units = tuples per row, multiple-files mode); noted inputs are user-selected Data Arrays. - -- [x] **WriteFeatureDataCSVFilter** (SimplnxCore) - - **Changes Made:** Defined **Feature** on first use; explained the leading output line is the total Feature count; added Required Input Sources naming [Compute Feature Neighbors](ComputeFeatureNeighborsFilter.md) and Compute Feature Neighbor Misorientations. - -- [x] **ReadImageFilter** (SimplnxCore) - - **Changes Made:** Added an up-front 'single 2D image' note with a link to the stack reader; stated Origin/Spacing units; added Required Input Sources (None). - -- [x] **ReadImageStackFilter** (SimplnxCore) - - **Changes Made:** Fixed the title to match humanName 'Read Images [3D Stack]'; converted the bold Read Image mention to a MyST link; added a 'File List and Slice Ordering' subsection; defined the luminosity algorithm; stated spacing/origin units; added Required Input Sources (None). - -- [x] **ReadNIfTIFileFilter** (SimplnxCore) - - **Changes Made:** Removed the hand-written `## Parameters` table that duplicated the auto-table (kept the Caveats list); added plain-language glosses for sform/qform/pixdim/vox_offset/scl_slope/scl_inter; noted spacing units come from the file (often mm); added Required Input Sources (None). - -- [x] **ReadZeissTxmFileFilter** (SimplnxCore) - - **Changes Made:** Expanded 'xCT' to X-ray computed tomography on first use; stated spacing/origin/units are read from the file and named the created output array; normalized trailing-## headings; added Required Input Sources (None). - -- [x] **ReadBinaryCTNorthstarFilter** (SimplnxCore) - - **Changes Made:** Normalized the Group line to 'IO (Input)'; defined CT on first use; named the created output array; noted the subvolume start/end are zero-based inclusive voxels; added Required Input Sources (None). - -- [x] **WriteNodesAndElementsFilesFilter** (SimplnxCore) - - **Changes Made:** Added Required Input Sources (any node-based geometry from a meshing/geometry-creation filter); fixed the 'requried' typo; stated node coordinates are in the geometry's physical units. - -- [x] **WriteStlFileFilter** (SimplnxCore) - - **Changes Made:** Fixed the 'implict' typo; clarified the 2-component Features array holds the two Feature IDs on either side of each triangle; noted vertex coordinates are in the Triangle Geometry's units; added Required Input Sources (Triangle Geometry + Face Labels). - -- [x] **ReadGrainMapper3DFilter** (OrientationAnalysis) - - **Changes Made:** Added a GrainMapper3D/XNovo LabDCT gloss; clarified the two IPF-color options; added a millimeters-vs-microns downstream caveat; added Required Input Sources (None). - -- [x] **WriteINLFileFilter** (OrientationAnalysis) - - **Changes Made:** Annotated the output columns with explicit units (Euler radians, positions/step sizes microns); explained the Symmetry column holds crystal-symmetry codes (43 = cubic m-3m, 62 = hexagonal); added Required Input Sources. - -- [x] **WriteStatsGenOdfAngleFileFilter** (OrientationAnalysis) - - **Changes Made:** Expanded ODF to Orientation Distribution Function on first use; fixed 'can not' -> 'cannot'; noted weight and sigma are dimensionless; added Required Input Sources (Euler angles from an EBSD reader; optional Mask from Multi-Threshold Objects). - -### Tier 4 — Adequate - -| Filter | Plugin | Status | -|--------|--------|--------| -| ReadRawBinary | SimplnxCore | Done (minor) | -| WriteImage | SimplnxCore | Done (minor) | -| WriteDREAM3D | SimplnxCore | Done (minor) | - -- [x] **ReadRawBinaryFilter** (SimplnxCore) - - **Changes Made:** Added a 'Required Input Sources: None' note. No other changes — the doc was already a model for the batch. - -- [x] **WriteImageFilter** (SimplnxCore) - - **Changes Made:** Added a Required Input Sources section naming the image readers / image-processing producers. No other changes — already publication-ready. - -- [x] **WriteDREAM3DFilter** (SimplnxCore) - - **Changes Made:** Added the missing Group (Subgroup) header and a Required Input Sources note (none; overwrites existing file). No other changes — already thorough. - -## Batch 8: Image Processing / ITK Wrappers - -**Plugin:** ITKImageProcessing -**Filters:** 88 (ITK-wrapped image-processing filters) - -> Triaged 2026-06-11. Tier split: T1=14, T2=45, T3=26, T4=3. These descriptions were lifted from ITK; the guiding question is whether a general engineer can understand the filter and set its parameters WITHOUT reading ITK's Doxygen. User decisions: trivial intensity ops get minimal-clean treatment; Required Input Sources is a light generic note (ITK inputs are generic scalar images). -> -> **Recurring ITK defects to sweep:** stale `## DREAM3D Mailing Lists` footer (~17 docs); raw Doxygen math `\f[ \f]`/`\f$ \f$` (BinaryThreshold, RescaleIntensity, Normalize, CurvatureFlow, MinMaxCurvatureFlow); mangled `.* ClassName`/`\sa`/`\li` See-Also artifacts dumped inline (pervasive); C++/developer symbols (`itk::`, `NumericTraits<>`, 'templated over', `Set*()` methods, `m_*`); hand-duplicated parameter tables missing the auto-table marker (IsoContourDistance, GradientMagnitudeRecursiveGaussian, ImportFijiMontage, ThresholdMaximumConnectedComponents). -> -> **Bugs / mismatches found:** ITKGrayscaleErode says 'Erosion takes the maximum' (should be minimum); ITKExpNegative describes a user-provided constant K the filter does not expose; ITKLog has no description text; ITKThreshold lists Above/Below/Outside methods not exposed as params; ITKAdaptiveHistogramEqualization prose may describe alpha/beta that are not exposed — all to verify against the cpp during rewrite. -> -> **Shared concepts to author once and reuse:** morphology (erosion/dilation/opening/closing + the structuring element / kernel radius in pixels); projection-along-an-axis (Projection Dimension); connectivity (Fully Connected = face-only vs face+edge+vertex); distance maps (signed = inside negative; squared-distance and use-image-spacing toggles). **Naming collisions:** ITK Read Image / Read Images [3D Stack] vs the SimplnxCore filters of the same human name — disambiguate and cross-link. - -### Tier 1 — Critical - -| Filter | Plugin | Status | -|--------|--------|--------| -| ITKCurvatureAnisotropicDiffusionImageFilter | ITKImageProcessing | Done | -| ITKGradientAnisotropicDiffusionImageFilter | ITKImageProcessing | Done | -| ITKCurvatureFlowImageFilter | ITKImageProcessing | Done | -| ITKMinMaxCurvatureFlowImageFilter | ITKImageProcessing | Done | -| ITKBinaryOpeningByReconstructionImageFilter | ITKImageProcessing | Done | -| ITKBinaryThinningImageFilter | ITKImageProcessing | Done | -| ITKMorphologicalGradientImageFilter | ITKImageProcessing | Done | -| ITKClosingByReconstructionImageFilter | ITKImageProcessing | Done | -| ITKOpeningByReconstructionImageFilter | ITKImageProcessing | Done | -| ITKBinaryThresholdImageFilter | ITKImageProcessing | Done | -| ITKDoubleThresholdImageFilter | ITKImageProcessing | Done | -| ITKIsoContourDistanceImageFilter | ITKImageProcessing | Done | -| ITKMorphologicalWatershedImageFilter | ITKImageProcessing | Done | -| ITKMorphologicalWatershedFromMarkersImageFilter | ITKImageProcessing | Done | - -- [x] **ITKCurvatureAnisotropicDiffusionImageFilter** (ITKImageProcessing) - - **Changes Made:** Added a plain-language 'what is anisotropic diffusion' lead and an MCDE note; stripped itk:: symbols; documented the previously-undocumented Conductance Scaling Update Interval; kept the TimeStep/Conductance/Iterations guidance with stability values; fixed the stale footer; cleaned the See-Also. - -- [x] **ITKGradientAnisotropicDiffusionImageFilter** (ITKImageProcessing) - - **Changes Made:** Added a plain-language Perona-Malik / edge-stopping lead; stripped itk:: symbols and fixed the copy-paste 'CurvatureND' reference; documented Conductance Scaling Update Interval; promoted the bare URLs to a See-Also; fixed the stale footer. - -- [x] **ITKCurvatureFlowImageFilter** (ITKImageProcessing) - - **Changes Made:** Replaced the raw Doxygen math with a plain contour-curvature explanation; trimmed the solver-internals/padding paragraphs; added a concrete TimeStep typical value; fixed the stale footer; kept the Sethian reference. - -- [x] **ITKMinMaxCurvatureFlowImageFilter** (ITKImageProcessing) - - **Changes Made:** Replaced ALL raw Doxygen math with a plain explanation of the min/max scale switch; defined Stencil Radius in pixels with a typical value; added TimeStep/iterations guidance; fixed the stale footer. - -- [x] **ITKBinaryOpeningByReconstructionImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined reconstruction and the shape-preservation advantage over plain opening; fixed the \sa run-on; stated Kernel Radius in pixels and defined Foreground/Background/Fully Connected; kept the figure and Kernel Type table; light Required Input Sources note. - -- [x] **ITKBinaryThinningImageFilter** (ITKImageProcessing) - - **Changes Made:** Removed the 'parameterized over...' and 'To do: Make this filter ND.*' fragments; defined skeleton/medial axis and restated the 2D-only behavior as a plain caveat; light Required Input Sources note; kept the Gonzalez/Woods reference. - -- [x] **ITKMorphologicalGradientImageFilter** (ITKImageProcessing) - - **Changes Made:** Rewrote the description from scratch: morphological gradient = dilation - erosion = edge magnitude; stripped the ITK boilerplate/cross-ref tail; stated Kernel Radius in pixels; kept the Kernel Type table. - -- [x] **ITKClosingByReconstructionImageFilter** (ITKImageProcessing) - - **Changes Made:** Fixed the garbled 'levels raises' and broken PreserveIntensities sentences; defined Preserve Intensities and Fully Connected; stated Kernel Radius in pixels; explained the reconstruction concept; kept the Soille reference. - -- [x] **ITKOpeningByReconstructionImageFilter** (ITKImageProcessing) - - **Changes Made:** As with Closing-by-Reconstruction: fixed the garbled sentences, defined Preserve Intensities + Fully Connected, stated Kernel Radius in pixels, explained reconstruction. - -- [x] **ITKBinaryThresholdImageFilter** (ITKImageProcessing) - - **Changes Made:** Converted the raw Doxygen \f[...\f] piecewise math to a MyST $$ block; stripped NumericTraits/'templated over' text; stated Lower/Upper are in intensity units (usually set only one) and Inside/Outside are uint8 labels; kept the figure. - -- [x] **ITKDoubleThresholdImageFilter** (ITKImageProcessing) - - **Changes Made:** Mapped Threshold1-4 to the narrow (T2-T3) inside wide (T1-T4) hysteresis ranges with the ordering requirement and intensity units; removed the broken ITK See-Also bullets; defined Fully Connected; fixed the stale footer. - -- [x] **ITKIsoContourDistanceImageFilter** (ITKImageProcessing) - - **Changes Made:** Led with a plain definition (for a level-set/signed image, the distance from near-contour pixels to the zero contour); defined level set/iso-contour/narrowband; DELETED the hand-written Parameters/Required/Created tables and ADDED the auto-table marker; described Far Value correctly as a clamp distance (default 10) — NOTE the cpp FarValue docstring is a copy-paste bug; fixed the stale footer. - -- [x] **ITKMorphologicalWatershedImageFilter** (ITKImageProcessing) - - **Changes Made:** Led with the terrain/flooding analogy; defined catchment basin and watershed line; documented Level (intensity, over-segmentation control), Mark Watershed Line, Fully Connected; noted the over-segmentation tendency, the Relabel follow-up, and the marker-controlled variant; stripped TOutputImage; kept the Soille/IJ reference. - -- [x] **ITKMorphologicalWatershedFromMarkersImageFilter** (ITKImageProcessing) - - **Changes Made:** Removed the redundant title parenthetical; led with the marker-controlled watershed concept; documented Mark Watershed Line and Fully Connected; added a prominent note that the NX wrapper currently exposes only a single input array (no separate marker-image parameter) — flagged for verification; fixed the malformed See-Also and stale footer. - -### Tier 2 — Important - -| Filter | Plugin | Status | -|--------|--------|--------| -| ITKSigmoidImageFilter | ITKImageProcessing | Done | -| ITKIntensityWindowingImageFilter | ITKImageProcessing | Done | -| ITKRescaleIntensityImageFilter | ITKImageProcessing | Done | -| ITKDiscreteGaussianImageFilter | ITKImageProcessing | Done | -| ITKSmoothingRecursiveGaussianImageFilter | ITKImageProcessing | Done | -| ITKMedianImageFilter | ITKImageProcessing | Done | -| ITKAdaptiveHistogramEqualizationImageFilter | ITKImageProcessing | Done | -| ITKBinaryDilateImageFilter | ITKImageProcessing | Done | -| ITKBinaryErodeImageFilter | ITKImageProcessing | Done | -| ITKBinaryMorphologicalClosingImageFilter | ITKImageProcessing | Done | -| ITKBinaryMorphologicalOpeningImageFilter | ITKImageProcessing | Done | -| ITKBinaryContourImageFilter | ITKImageProcessing | Done | -| ITKDilateObjectMorphologyImageFilter | ITKImageProcessing | Done | -| ITKErodeObjectMorphologyImageFilter | ITKImageProcessing | Done | -| ITKGrayscaleDilateImageFilter | ITKImageProcessing | Done | -| ITKGrayscaleErodeImageFilter | ITKImageProcessing | Done | -| ITKGrayscaleMorphologicalClosingImageFilter | ITKImageProcessing | Done | -| ITKGrayscaleMorphologicalOpeningImageFilter | ITKImageProcessing | Done | -| ITKGrayscaleFillholeImageFilter | ITKImageProcessing | Done | -| ITKGrayscaleGrindPeakImageFilter | ITKImageProcessing | Done | -| ITKBlackTopHatImageFilter | ITKImageProcessing | Done | -| ITKWhiteTopHatImageFilter | ITKImageProcessing | Done | -| ITKThresholdImageFilter | ITKImageProcessing | Done | -| ITKOtsuMultipleThresholdsImageFilter | ITKImageProcessing | Done | -| ITKThresholdMaximumConnectedComponentsImageFilter | ITKImageProcessing | Done | -| ITKHConvexImageFilter | ITKImageProcessing | Done | -| ITKHMaximaImageFilter | ITKImageProcessing | Done | -| ITKHMinimaImageFilter | ITKImageProcessing | Done | -| ITKApproximateSignedDistanceMapImageFilter | ITKImageProcessing | Done | -| ITKDanielssonDistanceMapImageFilter | ITKImageProcessing | Done | -| ITKSignedDanielssonDistanceMapImageFilter | ITKImageProcessing | Done | -| ITKSignedMaurerDistanceMapImageFilter | ITKImageProcessing | Done | -| ITKGradientMagnitudeRecursiveGaussianImageFilter | ITKImageProcessing | Done | -| ITKZeroCrossingImageFilter | ITKImageProcessing | Done | -| ITKConnectedComponentImageFilter | ITKImageProcessing | Done | -| ITKRelabelComponentImageFilter | ITKImageProcessing | Done | -| ITKLabelContourImageFilter | ITKImageProcessing | Done | -| ITKMaximumProjectionImageFilter | ITKImageProcessing | Done | -| ITKMeanProjectionImageFilter | ITKImageProcessing | Done | -| ITKMedianProjectionImageFilter | ITKImageProcessing | Done | -| ITKMinimumProjectionImageFilter | ITKImageProcessing | Done | -| ITKStandardDeviationProjectionImageFilter | ITKImageProcessing | Done | -| ITKSumProjectionImageFilter | ITKImageProcessing | Done | -| ITKBinaryProjectionImageFilter | ITKImageProcessing | Done | -| ITKImportFijiMontageFilter | ITKImageProcessing | Done | - -- [x] **ITKSigmoidImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined the sigmoid S-curve remap with MyST math; documented Alpha (transition width), Beta (center), Output Min/Max; light Required Input Sources. - -- [x] **ITKIntensityWindowingImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined the window/level remap; fixed the mangled See-Also fragment and 'RealType'; stated all four params in intensity units. - -- [x] **ITKRescaleIntensityImageFilter** (ITKImageProcessing) - - **Changes Made:** Converted the Doxygen formula to MyST; clarified only Output Min/Max are user-set (input auto-detected); fixed the mangled See-Also. - -- [x] **ITKDiscreteGaussianImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined Gaussian blur; clarified Variance = sigma^2 with units (px^2 / physical^2); documented MaximumKernelWidth/MaximumError/UseImageSpacing; removed the ITK artifact + class list. - -- [x] **ITKSmoothingRecursiveGaussianImageFilter** (ITKImageProcessing) - - **Changes Made:** Led with the constant-time-vs-sigma advantage; Sigma units; explained Normalize Across Scale; fixed the stale footer and stray '|'. - -- [x] **ITKMedianImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined the median filter (salt-and-pepper removal, edge-preserving); Radius in pixels (window 2r+1); removed the operator<()/class-list artifact. - -- [x] **ITKAdaptiveHistogramEqualizationImageFilter** (ITKImageProcessing) - - **Changes Made:** Confirmed alpha/beta ARE exposed in the cpp; added pixels unit to Radius; documented alpha/beta/window; light Required Input Sources. - -- [x] **ITKBinaryDilateImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined dilation (grows foreground); documented Foreground/Background/BoundaryToForeground and Kernel Radius (pixels); stripped C++ symbols + See-Also run-on; cross-linked siblings. - -- [x] **ITKBinaryErodeImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined erosion (shrinks foreground; eroded->Background); replaced the NonpositiveMin() wording; Kernel Radius (pixels); cross-linked siblings. - -- [x] **ITKBinaryMorphologicalClosingImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined closing = dilate-then-erode (fills small holes); Foreground Value, Kernel Radius (pixels), Safe Border; stripped ITK Related-Filters; cross-linked siblings. - -- [x] **ITKBinaryMorphologicalOpeningImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined opening = erode-then-dilate (removes small specks); fixed 'Dilatation'; Foreground/Background, Kernel Radius (pixels); cross-linked siblings. - -- [x] **ITKBinaryContourImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined object-outline extraction; Foreground/Background values and Fully Connected (On=thicker); replaced SetFullyConnected(); cross-linked Label Contour. - -- [x] **ITKDilateObjectMorphologyImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined single-object dilation; Object Value + Kernel Radius (pixels); fixed the See-Also run-on; noted the distinction from Binary Dilate. - -- [x] **ITKErodeObjectMorphologyImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined single-object erosion; Object Value/Background Value + Kernel Radius (pixels); fixed the See-Also run-on; noted the distinction from Binary Erode. - -- [x] **ITKGrayscaleDilateImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined grayscale dilation = local max filter; Kernel Radius (pixels); stripped the ITK cross-ref tail. - -- [x] **ITKGrayscaleErodeImageFilter** (ITKImageProcessing) - - **Changes Made:** FIXED the factual error (erosion is the minimum, not maximum); defined grayscale erosion = local min; Kernel Radius (pixels); stripped the ITK tail. - -- [x] **ITKGrayscaleMorphologicalClosingImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined grayscale closing = dilate-then-erode (removes small dark features); Kernel Radius (pixels), Safe Border; stripped the mismatched ITK tail. - -- [x] **ITKGrayscaleMorphologicalOpeningImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined grayscale opening = erode-then-dilate (removes small bright features); Kernel Radius (pixels), Safe Border; stripped the ITK tail. - -- [x] **ITKGrayscaleFillholeImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined hole-filling (enclosed dark minima); documented Fully Connected; trimmed the geodesic-implementation noise; kept the Soille reference. - -- [x] **ITKGrayscaleGrindPeakImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined peak-removal (enclosed bright maxima; dual of Fillhole); documented Fully Connected; trimmed/fixed the geodesic implementation paragraph. - -- [x] **ITKBlackTopHatImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined black top-hat = closing minus image (small dark detail); Kernel Radius (pixels) tied to feature scale, Safe Border; removed the empty Related Filters heading. - -- [x] **ITKWhiteTopHatImageFilter** (ITKImageProcessing) - - **Changes Made:** Added a real description (image minus opening = small bright detail); Kernel Radius (pixels), Safe Border; removed the empty Related Filters heading. - -- [x] **ITKThresholdImageFilter** (ITKImageProcessing) - - **Changes Made:** Removed the phantom ThresholdAbove/Below/Outside method list; rewrote to the actual Lower/Upper/OutsideValue params (out-of-range -> OutsideValue, in-range kept); intensity units; contrasted with Binary Threshold. - -- [x] **ITKOtsuMultipleThresholdsImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined Otsu (automatic variance-maximizing thresholds; N+1-class label map); removed ITK class/setter/test names and fixed the mangled run-on; per-param guidance (thresholds/bins/offset/valley emphasis/bin midpoint). - -- [x] **ITKThresholdMaximumConnectedComponentsImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined the topological auto-threshold (maximizes object count above a minimum size); removed debug/ITK-method text, the broken sentence, and the personal-email block; fixed the stale footer; documented UpperBoundary/MinimumObjectSizeInPixels/Inside/Outside; replaced the hand Parameters prose. - -- [x] **ITKHConvexImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined h-convex (extracts peaks rising more than h above background); Height in intensity units, FullyConnected; stripped the mangled See-Also tail. - -- [x] **ITKHMaximaImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined h-maxima (suppresses maxima shallower than h); Height in intensity units; removed SetHeight + the mangled See-Also; cross-linked H Convex/H Minima. - -- [x] **ITKHMinimaImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined h-minima (fills minima shallower than h); Height + FullyConnected; stripped the mangled See-Also; replaced the non-NX HConcave reference. - -- [x] **ITKApproximateSignedDistanceMapImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined signed distance map (inside negative) via Chamfer approximation (pixels); concrete Inside/Outside Value example; demoted ITK siblings to See Also; fixed the stale footer. - -- [x] **ITKDanielssonDistanceMapImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined distance map + Voronoi partition + offset-vector outputs in lay terms; documented Input Is Binary / Squared Distance / Use Image Spacing (pixels vs physical); stripped itk::Offset/'4SED'; fixed the stale footer. - -- [x] **ITKSignedDanielssonDistanceMapImageFilter** (ITKImageProcessing) - - **Changes Made:** Stripped the \li markup and the 'parameterized over' text; fixed the broken ITK link; documented Inside Is Positive / Squared Distance / Use Image Spacing; fixed the stale footer. - -- [x] **ITKSignedMaurerDistanceMapImageFilter** (ITKImageProcessing) - - **Changes Made:** Trimmed the C++ type discussion; prominently flagged that Squared Distance defaults TRUE; documented Use Image Spacing / Background Value; fixed run-in sub-headers; kept the Maurer reference. - -- [x] **ITKGradientMagnitudeRecursiveGaussianImageFilter** (ITKImageProcessing) - - **Changes Made:** DELETED the hand Parameters/Required/Created tables and ADDED the missing auto-table marker; defined gradient magnitude (edge strength); emphasized Sigma is in image-spacing (physical) units; rewrote Normalize Across Scale; fixed the stale footer. - -- [x] **ITKZeroCrossingImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined zero-crossing edge detection on a signed input; stripped itk::/operator/NumericTraits text; restated the Foreground/Background label params; Required Input Sources = a signed (Laplacian-of-Gaussian) image; fixed the stale footer. - -- [x] **ITKConnectedComponentImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined connected components / label image; documented Fully Connected (+1-pixel note); noted the Relabel Component follow-up; stripped ITK method names + the useless See-Also; fixed the stale footer. - -- [x] **ITKRelabelComponentImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined relabel (consecutive, size-sorted; optional minimum size); documented Minimum Object Size in pixels and SortByObjectSize; stripped ITK method names + the InPlace paragraph + the mangled See-Also; Required Input Sources = a label image. - -- [x] **ITKLabelContourImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined label-region outline extraction; documented Background Value + Fully Connected (thicker-contours note); stripped SetFullyConnected(); moved the Insight-Journal URL to See Also; Required Input Sources = a label image. - -- [x] **ITKMaximumProjectionImageFilter** (ITKImageProcessing) - - **Changes Made:** Added the shared projection concept (3D->2D, maximum along an axis); documented Projection Dimension (axis index) + Perform In-Place + the dimensionality note; trimmed the Doxygen See-Also. - -- [x] **ITKMeanProjectionImageFilter** (ITKImageProcessing) - - **Changes Made:** Shared projection concept (mean along an axis); cross-linked Median for outlier sensitivity; removed the stray title parenthetical; trimmed the See-Also. - -- [x] **ITKMedianProjectionImageFilter** (ITKImageProcessing) - - **Changes Made:** Shared projection concept (median along an axis; outlier-robust vs mean); removed the stray title parenthetical; trimmed the See-Also. - -- [x] **ITKMinimumProjectionImageFilter** (ITKImageProcessing) - - **Changes Made:** Shared projection concept (minimum along an axis; dark-feature counterpart); removed the stray title parenthetical; trimmed the See-Also. - -- [x] **ITKStandardDeviationProjectionImageFilter** (ITKImageProcessing) - - **Changes Made:** Shared projection concept (standard deviation along an axis; highlights variability); fixed the stale footer; deduped the See-Also. - -- [x] **ITKSumProjectionImageFilter** (ITKImageProcessing) - - **Changes Made:** Shared projection concept (sum along an axis); added an overflow/data-type caution; fixed the stale footer; deduped the See-Also. - -- [x] **ITKBinaryProjectionImageFilter** (ITKImageProcessing) - - **Changes Made:** Refined the projection text; documented the logical-OR rule, Foreground/Background Value, Projection Dimension, Perform In-Place; deduped Related Filters; removed the ITK Doxygen link list. - -- [x] **ITKImportFijiMontageFilter** (ITKImageProcessing) - - **Changes Made:** Title matches humanName 'Read Fiji Montage (ITK)'; rewrote the output description in current DataStructure terms (per the cpp: per-tile Image Geometry -> Cell Attribute Matrix -> Image array); removed the hand-listed Length Unit / Output Data Type enums; fixed typos; replaced the stale Google-Groups footer; corrected the false 'same name' claim. - -### Tier 3 — Polish - -| Filter | Plugin | Status | -|--------|--------|--------| -| ITKAbsImageFilter | ITKImageProcessing | Done | -| ITKAcosImageFilter | ITKImageProcessing | Done | -| ITKAsinImageFilter | ITKImageProcessing | Done | -| ITKAtanImageFilter | ITKImageProcessing | Done | -| ITKCosImageFilter | ITKImageProcessing | Done | -| ITKSinImageFilter | ITKImageProcessing | Done | -| ITKTanImageFilter | ITKImageProcessing | Done | -| ITKExpImageFilter | ITKImageProcessing | Done | -| ITKExpNegativeImageFilter | ITKImageProcessing | Done | -| ITKLogImageFilter | ITKImageProcessing | Done | -| ITKLog10ImageFilter | ITKImageProcessing | Done | -| ITKSqrtImageFilter | ITKImageProcessing | Done | -| ITKNotImageFilter | ITKImageProcessing | Done | -| ITKInvertIntensityImageFilter | ITKImageProcessing | Done | -| ITKBoundedReciprocalImageFilter | ITKImageProcessing | Done | -| ITKNormalizeImageFilter | ITKImageProcessing | Done | -| ITKNormalizeToConstantImageFilter | ITKImageProcessing | Done | -| ITKRegionalMaximaImageFilter | ITKImageProcessing | Done | -| ITKRegionalMinimaImageFilter | ITKImageProcessing | Done | -| ITKValuedRegionalMaximaImageFilter | ITKImageProcessing | Done | -| ITKValuedRegionalMinimaImageFilter | ITKImageProcessing | Done | -| ITKGradientMagnitudeImageFilter | ITKImageProcessing | Done | -| ITKLaplacianRecursiveGaussianImageFilter | ITKImageProcessing | Done | -| ITKMaskImageFilter | ITKImageProcessing | Done | -| ITKImageReaderFilter | ITKImageProcessing | Done | -| ITKMhaFileReaderFilter | ITKImageProcessing | Done | - -- [x] **ITKAbsImageFilter** (ITKImageProcessing) - - **Changes Made:** Replaced the 'itk::Math::abs()' line with a plain one-sentence description (absolute value / magnitude). - -- [x] **ITKAcosImageFilter** (ITKImageProcessing) - - **Changes Made:** Collapsed the templating/cast boilerplate to one sentence: arccosine; input [-1,1]; output in radians. - -- [x] **ITKAsinImageFilter** (ITKImageProcessing) - - **Changes Made:** One sentence: arcsine; input [-1,1]; output in radians. - -- [x] **ITKAtanImageFilter** (ITKImageProcessing) - - **Changes Made:** One sentence: arctangent; output in radians. - -- [x] **ITKCosImageFilter** (ITKImageProcessing) - - **Changes Made:** One sentence: cosine; values interpreted as radians; output [-1,1]. - -- [x] **ITKSinImageFilter** (ITKImageProcessing) - - **Changes Made:** Reworded to plain 'sine (values in radians)'; dropped std::sin; kept the figure. - -- [x] **ITKTanImageFilter** (ITKImageProcessing) - - **Changes Made:** Reworded to plain 'tangent (values in radians)'; dropped std::tan. - -- [x] **ITKExpImageFilter** (ITKImageProcessing) - - **Changes Made:** Reworded to 'natural exponential e^pixel'; dropped std::exp. - -- [x] **ITKExpNegativeImageFilter** (ITKImageProcessing) - - **Changes Made:** Verified NO K parameter is exposed (functor default-constructed, K fixed at 1); corrected the summary and description to exp(-x) instead of the phantom 'user-provided constant K'. - -- [x] **ITKLogImageFilter** (ITKImageProcessing) - - **Changes Made:** Added the missing description sentence (natural logarithm; input must be > 0); kept the figure. - -- [x] **ITKLog10ImageFilter** (ITKImageProcessing) - - **Changes Made:** Reworded to 'base-10 logarithm; input must be > 0'; dropped std::log10. - -- [x] **ITKSqrtImageFilter** (ITKImageProcessing) - - **Changes Made:** Replaced the std::sqrt line with a plain description (square root; non-negative input). - -- [x] **ITKNotImageFilter** (ITKImageProcessing) - - **Changes Made:** Removed the templated/bool text and the raw C++ code block + m_* symbols; plain description (integer image; 0->Foreground, non-zero->Background). - -- [x] **ITKInvertIntensityImageFilter** (ITKImageProcessing) - - **Changes Made:** Replaced 'SetMaximum' with plain Maximum-minus-pixel guidance (set Maximum to the largest intensity, e.g. 255 for 8-bit); trimmed ITK Author/Related-Filters. - -- [x] **ITKBoundedReciprocalImageFilter** (ITKImageProcessing) - - **Changes Made:** Replaced the dimension/scalar-type sentence with the 1/(1+x) description (bounded (0,1] for non-negative input). - -- [x] **ITKNormalizeImageFilter** (ITKImageProcessing) - - **Changes Made:** Converted the Doxygen $-sigma$/$+sigma$ to MyST math; removed the mangled See-Also fragment and internal class names; kept the integral-type caveat. - -- [x] **ITKNormalizeToConstantImageFilter** (ITKImageProcessing) - - **Changes Made:** Dropped the redundant title parenthetical; replaced 'SetConstant()' with the Constant parameter (dimensionless target sum, default 1); fixed the stale footer. - -- [x] **ITKRegionalMaximaImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined regional maxima; documented Flat Is Maxima + Fully Connected; noted Background/Foreground are output labels; fixed the stale footer; dropped ITK Author/See-Also. - -- [x] **ITKRegionalMinimaImageFilter** (ITKImageProcessing) - - **Changes Made:** Defined regional minima; de-duplicated the repeated author block; documented Flat Is Minima + Fully Connected; fixed the stale footer; dropped the non-NX HConcave See-Also. - -- [x] **ITKValuedRegionalMaximaImageFilter** (ITKImageProcessing) - - **Changes Made:** Moved the connectivity paragraph into the Description; trimmed the non-NX Related Filters; kept the FullyConnected explanation; added Required Input Sources. - -- [x] **ITKValuedRegionalMinimaImageFilter** (ITKImageProcessing) - - **Changes Made:** Added the FullyConnected/connectivity explanation (mirroring the Maxima doc); tidied the Related Filters run-on; dropped the non-NX entries. - -- [x] **ITKGradientMagnitudeImageFilter** (ITKImageProcessing) - - **Changes Made:** Replaced the duplicated description with a one-sentence gradient-magnitude definition + the noise-sensitivity contrast vs the Recursive Gaussian variant; removed the meaningless ITK building-block Related-Filters list. - -- [x] **ITKLaplacianRecursiveGaussianImageFilter** (ITKImageProcessing) - - **Changes Made:** Added the Laplacian-of-Gaussian definition + the Zero Crossing follow-up link; documented Sigma (image-spacing units) and Normalize Across Scale; fixed the stale footer. - -- [x] **ITKMaskImageFilter** (ITKImageProcessing) - - **Changes Made:** Rewrote the Description as plain prose (dropped 'templated over', the diff-marked code block, Wiki-Examples cruft); documented Outside Value; fixed the stale footer; kept the figure. - -- [x] **ITKImageReaderFilter** (ITKImageProcessing) - - **Changes Made:** Dropped the 'directly wraps an ITK filter' boilerplate; added a note distinguishing it from the SimplnxCore Read Image filter (same human name) + cross-link. (humanName: Read Image (ITK)) - -- [x] **ITKMhaFileReaderFilter** (ITKImageProcessing) - - **Changes Made:** Fixed the typos; added a cross-link to Read Image (ITK) noting this variant handles the embedded transform; trimmed the orthogonal-matrix Technical Discussion. - -### Tier 4 — Adequate - -| Filter | Plugin | Status | -|--------|--------|--------| -| ITKSquareImageFilter | ITKImageProcessing | Done | -| ITKImageWriterFilter | ITKImageProcessing | Done | -| ITKImportImageStackFilter | ITKImageProcessing | Done | - -- [x] **ITKSquareImageFilter** (ITKImageProcessing) - - **Changes Made:** Added the missing one-sentence description ('square of each pixel'). Otherwise adequate. - -- [x] **ITKImageWriterFilter** (ITKImageProcessing) - - **Changes Made:** No change needed — already clean and complete (humanName: Write Image (ITK)). - -- [x] **ITKImportImageStackFilter** (ITKImageProcessing) - - **Changes Made:** Dropped the 'directly wraps' boilerplate; added a note distinguishing it from the SimplnxCore Read Images [3D Stack] filter + cross-link. Otherwise adequate. (humanName: Read Images [3D Stack] (ITK)) - -## Batch 9: Remaining SimplnxCore Filters - -**Plugin:** SimplnxCore -**Filters:** 73 (catch-all for SimplnxCore filters not covered by Batches 1-7; the ITK plugin is Batch 8, deferred) - -> Triaged 2026-06-11. Tier split: T1=4, T2=35, T3=31, T4=3. Rewrites will be committed per tier (Tier 2 split into sub-theme sub-commits given its size). Recurring defects flagged: hand-duplicated parameter tables (HierarchicalSmooth; CreatePythonSkeleton — also missing the marker), a legacy `@ref` directive (ComputeBoundingBoxStats), Doxygen math/`@image` artifacts (Silhouette, FeatureFaceCurvature, PointSampleTriangleGeometry), legacy 'Data Container' wording (NearestPointFuseRegularGrids, RemoveFlaggedVertices, ExtractInternalSurfaces), and many bold-only filter mentions. - -### Tier 1 — Critical - -| Filter | Plugin | Status | -|--------|--------|--------| -| FlyingEdges3D | SimplnxCore | Done | -| TriangleDihedralAngle | SimplnxCore | Done | -| NearestPointFuseRegularGrids | SimplnxCore | Done | -| PointSampleEdgeGeometry | SimplnxCore | Done | - -- [x] **FlyingEdges3DFilter** (SimplnxCore) - - **Changes Made:** Expanded the one-sentence stub: added 'What is an Isosurface?' (contour-line analogy, isovalue), noted Flying Edges is a fast marching-cubes variant, added 'When to Use' contrasting with Surface Nets / Quick Surface Mesh for label maps, documented the Contour Value units (same as the data array) and Required Input Sources (Image Geometry + scalar array). Captioned the figure. - -- [x] **TriangleDihedralAngleFilter** (SimplnxCore) - - **Changes Made:** Replaced the 'matrix mathematics' stub with a plain explanation of the minimum interior angle as a mesh-quality metric (60 deg = equilateral, near-0 deg = sliver). Stated output units are degrees (0-60). Added Required Input Sources (Triangle Geometry from a surface-meshing filter). - -- [x] **NearestPointFuseRegularGridsFilter** (SimplnxCore) - - **Changes Made:** Replaced all legacy 'Data Container' terminology with Image Geometry. Explained Reference (destination) vs Sampling (source) roles, that assignment is by nearest cell-center using each grid's origin/spacing (no interpolation), and that the grids may differ in spacing/extent. Documented the Use Custom Fill Value behavior for out-of-overlap cells. Added Required Input Sources (the two Image Geometries). - -- [x] **PointSampleEdgeGeometryFilter** (SimplnxCore) - - **Changes Made:** Added the missing Group header (Sampling (Geometry)). Defined Edge Geometry, 'scan vector' (an additive-manufacturing laser-path segment), and 'edge ID'. Stated the Sampling Spacing units are millimeters and explained Cumulative Sample Distance. Added Required Input Sources (Edge Geometry from Slice Triangle Geometry / Create AM Scan Paths). - -### Tier 2 — Important - -| Filter | Plugin | Status | -|--------|--------|--------| -| ComputeArrayStatistics | SimplnxCore | Done | -| ComputeBoundingBoxStats | SimplnxCore | Done | -| ComputeVolumeFractions | SimplnxCore | Done | -| ComputeFeatureClustering | SimplnxCore | Done | -| ComputeBoundaryElementFractions | SimplnxCore | Done | -| ComputeEuclideanDistMap | SimplnxCore | Done | -| ComputeMomentInvariants2D | SimplnxCore | Done | -| ComputeKMeans | SimplnxCore | Done | -| ComputeKMedoids | SimplnxCore | Done | -| Silhouette | SimplnxCore | Done | -| ApproximatePointCloudHull | SimplnxCore | Done | -| MapPointCloudToRegularGrid | SimplnxCore | Done | -| IterativeClosestPoint | SimplnxCore | Done | -| IdentifyDuplicateVertices | SimplnxCore | Done | -| ComputeTriangleAreas | SimplnxCore | Done | -| ComputeTriangleGeomCentroids | SimplnxCore | Done | -| ComputeTriangleGeomVolumes | SimplnxCore | Done | -| TriangleCentroid | SimplnxCore | Done | -| TriangleNormal | SimplnxCore | Done | -| FeatureFaceCurvature | SimplnxCore | Done | -| SharedFeatureFace | SimplnxCore | Done | -| SurfaceNets | SimplnxCore | Done | -| LaplacianSmoothing | SimplnxCore | Done | -| UncertainRegularGridSampleSurfaceMesh | SimplnxCore | Done | -| ExtractInternalSurfacesFromTriangleGeometry | SimplnxCore | Done | -| ReverseTriangleWinding | SimplnxCore | Done | -| VerifyTriangleWinding | SimplnxCore | Done | -| RemoveFlaggedEdges | SimplnxCore | Done | -| RemoveFlaggedTriangles | SimplnxCore | Done | -| CropVertexGeometry | SimplnxCore | Done | -| CombineNodeBasedGeometries | SimplnxCore | Done | -| PointSampleTriangleGeometry | SimplnxCore | Done | -| ExtractFeatureBoundaries2D | SimplnxCore | Done | -| RobustAutomaticThreshold | SimplnxCore | Done | -| ComputeVectorColors | SimplnxCore | Done | - -- [x] **ComputeArrayStatisticsFilter** (SimplnxCore) - - **Changes Made:** Fixed the stale group to 'Statistics'; defined Feature/Ensemble; condensed the redundant Ranges Breakdown Options 1-4; fixed spelling (aproiri/achived/occurance/exisitng); added Required Input Sources (Feature Ids producer). Kept the statistic-type table. - -- [x] **ComputeBoundingBoxStatsFilter** (SimplnxCore) - - **Changes Made:** Converted the legacy @ref directive and the Compute Array Statistics / Compute Feature Bounding Boxes mentions to MyST links; explained the 1-to-many cell-to-box mapping; stated bounds are in physical coordinate units; fixed 'columns'->'tuples'; added Required Input Sources. - -- [x] **ComputeVolumeFractionsFilter** (SimplnxCore) - - **Changes Made:** Reconciled the count-vs-volume-fraction wording (equal-volume cells make them identical); stated output is a per-Ensemble dimensionless fraction in [0,1] summing to 1; added Required Input Sources (Cell Phases); removed the empty Example Pipelines. - -- [x] **ComputeFeatureClusteringFilter** (SimplnxCore) - - **Changes Made:** Defined RDF (normalized histogram of inter-centroid distances) and Ensemble; added when/why (clustering/ordering detection, >1 vs <1 interpretation); stated distances are physical length units; added Required Input Sources (Compute Feature Centroids + Phases). - -- [x] **ComputeBoundaryElementFractionsFilter** (SimplnxCore) - - **Changes Made:** Defined a 'surface element' (one whose Surface Elements input value > 0); stated output is a [0,1] fraction (boundary/total per Feature); added Required Input Sources (Feature Ids + Surface Elements); removed the empty Example Pipelines. - -- [x] **ComputeEuclideanDistMapFilter** (SimplnxCore) - - **Changes Made:** Defined Feature boundary/GB, triple line/TJ, quadruple point/QP; stated the Manhattan output is an integer cell count and the Euclidean output is a physical float distance; added Required Input Sources (Feature Ids). - -- [x] **ComputeMomentInvariants2DFilter** (SimplnxCore) - - **Changes Made:** Added a plain-language definition of 2D moment invariants and what high/low Omega-1/Omega-2 indicate; stated Omega values dimensionless and Z Delta in cells; fixed typos (appllication/particales/'et. al'); fixed the missing newline before the citations; added Required Input Sources (Feature Ids + Feature Rect). - -- [x] **ComputeKMeansFilter** (SimplnxCore) - - **Changes Made:** Defined cluster/centroid/Voronoi tessellation/within-cluster variance; fixed the broken 'Ensemble Attribute Matrix' bold and typos (randomnes, tesselation); added MyST links to Compute K Medoids/DBSCAN/Silhouette and Required Input Sources. - -- [x] **ComputeKMedoidsFilter** (SimplnxCore) - - **Changes Made:** Defined cluster/Voronoi/topology (kept the good medoid definition); fixed the broken bold and typos (randomnes, arbirtary); added MyST links to Compute K Means/DBSCAN/Silhouette and Required Input Sources. - -- [x] **SilhouetteFilter** (SimplnxCore) - - **Changes Made:** Converted the Doxygen math delimiters to MyST $$/$ math; stated the output array location and dimensionless [-1,1] range; added MyST links to Compute K Means/K Medoids and Required Input Sources. - -- [x] **ApproximatePointCloudHullFilter** (SimplnxCore) - - **Changes Made:** Fixed typos (of/or, hve, To/The, 'is able of', 'faster that'); added Parameter Guidance stating grid resolution is in the point cloud's coordinate units and the empty-neighbor threshold is a 0-26 count; added Required Input Sources. - -- [x] **MapPointCloudToRegularGridFilter** (SimplnxCore) - - **Changes Made:** Added a downstream link to Interpolate Point Cloud to Regular Grid; stated Manual-mode dimensions are voxel counts; fixed the uint64-max off-by-one and the 'User may to trace' grammar; added Required Input Sources. - -- [x] **IterativeClosestPointFilter** (SimplnxCore) - - **Changes Made:** Fixed the all-'1.' list to 1-6 and the trailing-period heading; stated the output is a row-major flattened 4x4 transform (translation in coordinate units), reported always but applied only when the apply-transform option is on; cross-linked Apply Transformation to Geometry; added Required Input Sources (two Vertex Geometries). - -- [x] **IdentifyDuplicateVerticesFilter** (SimplnxCore) - - **Changes Made:** Defined 'duplicate' as all-three-coordinate equality within machine epsilon (no user tolerance); reworded the 'visual explanation' claim (the example is text); converted the italic RemoveFlaggedVertices to a MyST link; fixed the 'this filters output' apostrophe. - -- [x] **ComputeTriangleAreasFilter** (SimplnxCore) - - **Changes Made:** Defined Triangle Geometry; added when/why (mesh quality, statistic weighting); stated output area units (length^2); added Required Input Sources. - -- [x] **ComputeTriangleGeomCentroidsFilter** (SimplnxCore) - - **Changes Made:** Defined Feature/Face Labels ('owners')/nodes; stated centroid coordinate units; documented the -1 exterior-label edge case; added Required Input Sources (Triangle Geometry + Face Labels); removed the empty Example Pipelines. - -- [x] **ComputeTriangleGeomVolumesFilter** (SimplnxCore) - - **Changes Made:** Defined Feature/Face Labels/winding; added a watertight-requirement note; stated volume units (length^3); added Required Input Sources (closed Triangle Geometry + Face Labels); removed the empty Example Pipelines. - -- [x] **TriangleCentroidFilter** (SimplnxCore) - - **Changes Made:** Added when/why; stated centroid coordinate units; cross-linked the per-Feature Compute Feature Centroids from Triangle Geometry to disambiguate (this is per-triangle); added Required Input Sources. - -- [x] **TriangleNormalFilter** (SimplnxCore) - - **Changes Made:** Stated the output is a dimensionless unit 3-vector whose direction depends on winding; defined normal/cross-product/winding; fixed 'vertexes'->'vertices'; added Required Input Sources. - -- [x] **FeatureFaceCurvatureFilter** (SimplnxCore) - - **Changes Made:** Removed the stray @image latex Doxygen line; fixed the missing-space bold; replaced the Google-Groups mailing-list footer with the standard DREAM3D-NX Help footer; converted the bold Parameter-Notes filter mentions to a MyST-linked Required Input Sources list; defined curvature; stated units (principal/mean 1/length, Gaussian 1/length^2, directions dimensionless). - -- [x] **SharedFeatureFaceFilter** (SimplnxCore) - - **Changes Made:** Added a prose explanation of the Randomize Face IDs parameter (visualization-only ID permutation); defined Feature/Face Labels/Feature Attribute Matrix; added Required Input Sources (Triangle Geometry + Face Labels). - -- [x] **SurfaceNetsFilter** (SimplnxCore) - - **Changes Made:** Fixed typos (mush/equivelent/Traingle/FaceLables); converted the Verify Triangle Winding and QuickMesh mentions to MyST links; defined isosurface/marching cubes/triple line/winding; stated units (iterations dimensionless, max distance physical length); added Required Input Sources (Image Geometry + Feature Ids); fixed the non-standard footer. - -- [x] **LaplacianSmoothingFilter** (SimplnxCore) - - **Changes Made:** Fixed the corrupted ref [1] author and the malformed Belyaev URL; removed the dangling 'visit the tutorial' line; added alt text to the equation images; cross-linked Hierarchical Smoothing; stated Iteration Steps as a count and lambda/mu as dimensionless; added Required Input Sources (Triangle Geometry + Node Type). - -- [x] **UncertainRegularGridSampleSurfaceMeshFilter** (SimplnxCore) - - **Changes Made:** Defined Cells/rectilinear grid/conformal mesh/Feature; stated uncertainty and resolution/origin values are physical lengths; cross-linked the non-uncertain sibling; added Required Input Sources (Triangle Geometry + Face Labels). - -- [x] **ExtractInternalSurfacesFromTriangleGeometryFilter** (SimplnxCore) - - **Changes Made:** Fixed the two unclosed '**Vertex' bolds; replaced legacy 'Data Container' with 'data group'; added the missing DREAM3D-NX Help footer; cross-linked the Node Type producers (Surface Nets / Quick Surface Mesh); added Required Input Sources. - -- [x] **ReverseTriangleWindingFilter** (SimplnxCore) - - **Changes Made:** Defined 'winding' (vertex ordering that sets normal direction); cross-linked Verify Triangle Winding and Compute Triangle Normals; added Required Input Sources; removed the empty Example Pipelines. - -- [x] **VerifyTriangleWindingFilter** (SimplnxCore) - - **Changes Made:** Converted all bold/italic filter mentions to MyST links; defined 'winding' and 'negative volume'; fixed typos (Traingle/addtionally/preform/untounched/'difference n output'/Zero's/convience); reconciled the group to 'Surface Meshing (Connectivity/Arrangement)'; added Required Input Sources. - -- [x] **RemoveFlaggedEdgesFilter** (SimplnxCore) - - **Changes Made:** Defined Edge Geometry and 'mask'; fixed the group to 'Surface Meshing (Cleanup)'; added Required Input Sources (mask producer); fixed the footer /discussions link. - -- [x] **RemoveFlaggedTrianglesFilter** (SimplnxCore) - - **Changes Made:** Fixed the Vertices->Triangles copy-paste error; defined Triangle Geometry/Face Data/mask; fixed the group to 'Surface Meshing (Cleanup)'; added Required Input Sources; corrected the example-pipeline name. - -- [x] **CropVertexGeometryFilter** (SimplnxCore) - - **Changes Made:** Expanded the thin doc; stated Min/Max are physical coordinates (not cell indices) with inclusive bounds; replaced the stale 'DREAM3D Review (Cropping/Cutting)' group with 'Geometry (Cropping/Cutting)'; cross-linked Crop Edge/Image Geometry; added Required Input Sources and the missing footer. - -- [x] **CombineNodeBasedGeometriesFilter** (SimplnxCore) - - **Changes Made:** Defined node-based and higher/lower-order geometry; clarified vertex-index renumbering and that duplicate vertices are NOT merged; set the group to 'Core (Combining)'; added Required Input Sources. - -- [x] **PointSampleTriangleGeometryFilter** (SimplnxCore) - - **Changes Made:** Added descriptive alt text to the equation PNGs (corrected to the actual barycentric sqrt form); fixed 'beloning'; removed a stale sample-count table that no longer matches the parameter; added a random-sampling/seed note; fixed the stale 'DREAM3D Review (Geometry)' group to 'Sampling (Geometry)'; added Required Input Sources. - -- [x] **ExtractFeatureBoundaries2DFilter** (SimplnxCore) - - **Changes Made:** Defined Feature IDs/Image Geometry/Edge Geometry/overscan; fixed '.d3dpipline'->'.d3dpipeline'; replaced the stale subgroup with 'Surface Meshing (Generation)'; converted bold filter mentions to MyST links; added image alt text and Required Input Sources. - -- [x] **RobustAutomaticThresholdFilter** (SimplnxCore) - - **Changes Made:** Fixed the stale group to 'Threshold'; stated the robust/automatic rationale up front; defined gradient magnitude / 2-norm; replaced the suspect a_i*g_i/g_i equation with a prose weighted-average description; replaced the nonexistent 'Find Derivatives' reference with the real gradient-magnitude producer (ITK Gradient Magnitude Image Filter) as a MyST link and Required Input Sources. - -- [x] **ComputeVectorColorsFilter** (SimplnxCore) - - **Changes Made:** Expanded the description: input is a 3-component float32 vector Cell array, output is uint8 RGB (0-255), each vector is normalized (direction only); documented the mask/bad-voxel behavior; added Required Input Sources. - -### Tier 3 — Polish - -| Filter | Plugin | Status | -|--------|--------|--------| -| ComputeArrayHistogram | SimplnxCore | Done | -| ComputeArrayHistogramByFeature | SimplnxCore | Done | -| ComputeDifferencesMap | SimplnxCore | Done | -| ComputeNumFeatures | SimplnxCore | Done | -| ComputeSurfaceAreaToVolume | SimplnxCore | Done | -| ComputeFeaturePhases | SimplnxCore | Done | -| ComputeFeaturePhasesBinary | SimplnxCore | Done | -| ComputeFeatureBounds | SimplnxCore | Done | -| ComputeFeatureCentroids | SimplnxCore | Done | -| ComputeFeatureRect | SimplnxCore | Done | -| ComputeLargestCrossSections | SimplnxCore | Done | -| ComputeCoordinateThreshold | SimplnxCore | Done | -| DBSCAN | SimplnxCore | Done | -| InterpolatePointCloudToRegularGrid | SimplnxCore | Done | -| ExtractVertexGeometry | SimplnxCore | Done | -| ComputeVertexToTriangleDistances | SimplnxCore | Done | -| LabelTriangleGeometry | SimplnxCore | Done | -| HierarchicalSmooth | SimplnxCore | Done | -| RegularGridSampleSurfaceMesh | SimplnxCore | Done | -| SliceTriangleGeometry | SimplnxCore | Done | -| RemoveFlaggedVertices | SimplnxCore | Done | -| CropEdgeGeometry | SimplnxCore | Done | -| CombineStlFiles | SimplnxCore | Done | -| MultiThresholdObjects | SimplnxCore | Done | -| CreateColorMap | SimplnxCore | Done | -| ChangeAngleRepresentation | SimplnxCore | Done | -| RandomizeFeatureIds | SimplnxCore | Done | -| ExecuteProcess | SimplnxCore | Done | -| CreatePythonSkeleton | SimplnxCore | Done | -| ExtractPipelineToFile | SimplnxCore | Done | -| CreateAMScanPaths | SimplnxCore | Done | - -- [x] **ComputeArrayHistogramFilter** (SimplnxCore) - - **Changes Made:** Trimmed the long raw 'Old Faithful' data dump; defined 'mode' on first use; stated bin bounds inherit the input array units; added Required Input Sources (None). - -- [x] **ComputeArrayHistogramByFeatureFilter** (SimplnxCore) - - **Changes Made:** Defined Feature/Feature Id; added Required Input Sources (Feature Ids producer); removed the empty Example Pipelines. - -- [x] **ComputeDifferencesMapFilter** (SimplnxCore) - - **Changes Made:** Stated the output inherits the input type/shape/location; added a brief Required Input Sources (None -- generic arrays); removed the empty Example Pipelines. - -- [x] **ComputeNumFeaturesFilter** (SimplnxCore) - - **Changes Made:** Defined Ensemble/phase; stated the output is a per-Ensemble count array; added Required Input Sources (Compute Feature Phases). - -- [x] **ComputeSurfaceAreaToVolumeFilter** (SimplnxCore) - - **Changes Made:** Defined 'aliasing'; stated the ratio units (1/length) and that sphericity is dimensionless; linked Compute Boundary Cells; added Required Input Sources (Feature Ids). - -- [x] **ComputeFeaturePhasesFilter** (SimplnxCore) - - **Changes Made:** Defined Feature/Ensemble/Element; stated the per-Feature phase output; added Required Input Sources (Feature Ids + Cell Phases). - -- [x] **ComputeFeaturePhasesBinaryFilter** (SimplnxCore) - - **Changes Made:** Added a generic non-materials framing; stated the boolean-mask input and binary Ensemble output; added Required Input Sources (Multi-Threshold Objects); removed the empty Example Pipelines. - -- [x] **ComputeFeatureBoundsFilter** (SimplnxCore) - - **Changes Made:** Stated bounds are physical coordinates (not cell indices); fixed the 'Dimesions' typo and the doubled ';;'; defined Feature/Cell; added Required Input Sources (Feature Ids). - -- [x] **ComputeFeatureCentroidsFilter** (SimplnxCore) - - **Changes Made:** Stated centroids are physical coordinates (3-component float32, not pixel indices); defined Feature/Cell; added Required Input Sources (Feature Ids). - -- [x] **ComputeFeatureRectFilter** (SimplnxCore) - - **Changes Made:** Fixed 'This values'->'These values'; clarified Pixel coords are zero-based voxel indices; cross-linked Compute Feature Bounding Boxes; added Required Input Sources. - -- [x] **ComputeLargestCrossSectionsFilter** (SimplnxCore) - - **Changes Made:** Stated the area is square physical units (cellCount x in-plane voxel area, confirmed from the cpp); described the per-feature float32 output; added Required Input Sources (Feature Ids). - -- [x] **ComputeCoordinateThresholdFilter** (SimplnxCore) - - **Changes Made:** Converted the prose Remove Flagged Vertices/Edges/Triangles mentions to MyST links; stated the rectangle min/max and sphere center/radius are physical coordinate units. - -- [x] **DBSCANFilter** (SimplnxCore) - - **Changes Made:** Converted eight bold/quoted inter-filter mentions to MyST links; fixed typos (sqaure_root, preforming/preform, 'vise versa'); added Required Input Sources. - -- [x] **InterpolatePointCloudToRegularGridFilter** (SimplnxCore) - - **Changes Made:** Converted the italic Map Point Cloud to Regular Grid to a MyST link; added Required Input Sources (Vertex Geometry producer). - -- [x] **ExtractVertexGeometryFilter** (SimplnxCore) - - **Changes Made:** Standardized the Rectilinear Grid / Vertex Geometry naming; added Required Input Sources. - -- [x] **ComputeVertexToTriangleDistancesFilter** (SimplnxCore) - - **Changes Made:** Fixed the 4x 'Geoemtry' typo; stated distance units (length unit); noted normals must be present/consistent for sign correctness; added Required Input Sources (Vertex + Triangle Geometry). - -- [x] **LabelTriangleGeometryFilter** (SimplnxCore) - - **Changes Made:** Added Required Input Sources with a note that STL is a common CAD mesh format; fixed the footer /discussions link. - -- [x] **HierarchicalSmoothFilter** (SimplnxCore) - - **Changes Made:** Removed the hand-written Parameters table (it duplicated the auto-table), replacing it with prose; added a MyST link to Laplacian Smoothing; added Required Input Sources; normalized the non-standard footer; removed the empty Example Pipelines. - -- [x] **RegularGridSampleSurfaceMeshFilter** (SimplnxCore) - - **Changes Made:** Stated Origin/Spacing physical units and Dimensions as voxel counts; cross-linked the Uncertain variant; added Required Input Sources (Triangle Geometry + Face Labels). - -- [x] **SliceTriangleGeometryFilter** (SimplnxCore) - - **Changes Made:** Fixed typos (geoemtry, bewteen, perimieter); stated Slice Spacing/range/area/perimeter are in the geometry's physical units; added Required Input Sources. - -- [x] **RemoveFlaggedVerticesFilter** (SimplnxCore) - - **Changes Made:** Replaced legacy 'Data Container' wording with NX terminology; added Required Input Sources (the boolean-mask producer). - -- [x] **CropEdgeGeometryFilter** (SimplnxCore) - - **Changes Made:** Added the missing Group header (Core (Conversion)); stated Min/Max are physical units, not indices; removed the empty Example Pipelines; cross-linked Crop Geometry (Image). - -- [x] **CombineStlFilesFilter** (SimplnxCore) - - **Changes Made:** Defined STL on first use; converted the bold Import/Write STL mentions to MyST links. - -- [x] **MultiThresholdObjectsFilter** (SimplnxCore) - - **Changes Made:** Defined Cell/Attribute Matrix/EBSD context; added Required Input Sources naming EBSD readers as typical producers; removed the empty Example Pipelines. - -- [x] **CreateColorMapFilter** (SimplnxCore) - - **Changes Made:** Added the missing Group header (Core (Image)); added Required Input Sources (scalar array + optional mask); noted the preset table is hand-maintained; removed the empty Example Pipelines; fixed the License line. - -- [x] **ChangeAngleRepresentationFilter** (SimplnxCore) - - **Changes Made:** Added a Required Input Sources one-liner naming EBSD readers as typical angle producers (units content already exemplary). - -- [x] **RandomizeFeatureIdsFilter** (SimplnxCore) - - **Changes Made:** Defined Feature Ids/Attribute Matrix/NeighborList; documented that the cpp exposes no seed parameter (not reproducible); added Required Input Sources; fixed the License line. - -- [x] **ExecuteProcessFilter** (SimplnxCore) - - **Changes Made:** Stated the absolute-path requirement; documented the Should Block / Timeout (ms) parameters; added an explicit Required Input Sources: None. - -- [x] **CreatePythonSkeletonFilter** (SimplnxCore) - - **Changes Made:** Removed the hand-written Configuration parameter list and ADDED the missing auto-table marker; added the missing Group header (Core (Generation)); added Required Input Sources: None. - -- [x] **ExtractPipelineToFileFilter** (SimplnxCore) - - **Changes Made:** Added the missing Group header (IO (Output)) and the missing DREAM3D-NX Help footer; noted the no-embedded-pipeline behavior; added Required Input Sources: None; fixed the License line. - -- [x] **CreateAMScanPathsFilter** (SimplnxCore) - - **Changes Made:** Added a Units section (Hatch Spacing/Length in geometry coordinate units, Hatch Rotation Angle in degrees per the cpp label); converted the quoted SliceTriangleGeometry to a MyST link and promoted it to Required Input Sources. - -### Tier 4 — Adequate - -| Filter | Plugin | Status | -|--------|--------|--------| -| ComputeFeatureSizes | SimplnxCore | Done | -| ComputeBoundaryCells | SimplnxCore | Done | -| ComputeGroupingDensity | SimplnxCore | Done | - -- [x] **ComputeFeatureSizesFilter** (SimplnxCore) - - **Changes Made:** Added Required Input Sources (Feature Ids from a segmentation filter). Otherwise adequate as triaged. - -- [x] **ComputeBoundaryCellsFilter** (SimplnxCore) - - **Changes Made:** Stated the output range 0-6 and volume-edge handling; added Required Input Sources (Feature Ids). Otherwise adequate. - -- [x] **ComputeGroupingDensityFilter** (SimplnxCore) - - **Changes Made:** Converted the quoted 'Compute Feature Neighborhoods' to a MyST link (and fixed the dangling quote); added Required Input Sources (Parent IDs, neighbor lists, Feature Volumes). Otherwise model-quality. - ---- - -## Real-World Visualization Wishlist - -These are pipeline-generated screenshots/visualizations that would enhance the documentation but require DREAM3DNX automation tooling (future work). - -| Filter | Desired Visualization | Priority | -|--------|----------------------|----------| -| ComputeAvgCAxes | IPF color map showing C-axis orientations per grain | Medium | -| ComputeCAxisLocations | C-axis direction map colored by orientation | Medium | -| ComputeGBCD | GBCD pole figure output from Write GBCD Pole Figure filter | High | -| ComputeSchmids | Microstructure colored by Schmid factor under a loading direction | High | -| ComputeKernelAvgMisorientations | KAM map showing orientation gradients within grains | High | -| ComputeAvgOrientations | IPF color map showing average orientation assignment per grain | Medium | -| ComputeFeatureReferenceMisorientations | Color scale legend for existing images; additional loading conditions | Low | -| ComputeShapes | Features colored by aspect ratio or Omega3 value | Medium | -| ComputeTwinBoundaries | Microstructure with twin boundaries highlighted and incoherence heat map | Medium | -| ComputeSlipTransmissionMetrics | Feature pairs with high vs low transmission metrics visualized | Medium | -| ComputeBoundaryStrengths | Boundary mesh colored by M' or fip values | Medium | -| AlignSectionsFeatureCentroid | Before/after comparison showing when centroid alignment is appropriate vs when it fails (e.g., sample with consistent vs inconsistent mask regions) | High | -| AlignSectionsMisorientation | Before/after slice alignment showing corrected vs uncorrected sections; side-by-side comparison of alignment with mask vs without mask showing when masking is needed (e.g., sample with mounting material or voids at edges) | High | -| AlignSectionsMutualInformation | Before/after alignment comparison; side-by-side with misorientation method | Medium | -| AlignSectionsList | Typical workflow showing shift output from another alignment filter used as input | Low | -| ITKNormalizeImage | Before/after images showing normalization effect with parameters used to generate result | High | -| ITKRescaleIntensityImage | Before/after images showing rescaling effect with parameters used to generate result | High | -| ScalarSegmentFeatures | Before/after FeatureIds map from a scalar-based segmentation (e.g., image-quality thresholding) | Medium | -| CAxisSegmentFeatures | Before/after segmentation on hexagonal (e.g., Ti) EBSD data | Medium | -| EBSDSegmentFeatures | EBSD IPF map → segmented grains visualization (flagship EBSD workflow) | High | -| MergeTwins | Before/after microstructure showing twin variants merged into parent grain | High | -| IdentifySample | Before/after showing overscan cleanup on a real FIB-SEM dataset | Low | -| RequireMinimumSizeFeatures | Before/after microstructure with minimum-size filter applied | Medium | -| RemoveFlaggedFeatures | Side-by-side comparison of Remove vs Extract vs Extract-then-Remove operation modes | Medium | -| ComputeFeatureNeighbors | Grain map colored by number of neighbors | Medium | -| AddBadData | Synthetic structure before and after random noise; same with boundary noise | High | -| FillBadData | Conceptual diagram: small defects (filled) vs large defects (preserved) at threshold | Medium | -| ErodeDilateBadData | Multi-frame iteration sequence diagram (iter 0 → 3) for both erode and dilate | High | -| ErodeDilateBadData | Erode-then-dilate ("opening") removing isolated single-cell noise but preserving a larger pore | Medium | -| ErodeDilateMask | Directional erosion (X-only vs uniform) on a mask | Medium | -| ErodeDilateCoordinationNumber | 3x3x3 cell cluster diagram with coordination numbers labeled (CN=6 isolated, CN=3 edge, etc.) | High | -| ReplaceElementAttributesWithNeighborValues | Conceptual diagram: flagged cell + neighbors with chosen replacement highlighted by max/min | Medium | -| NeighborOrientationCorrelation | Cell-with-6-neighbors diagram showing Cleanup Level=2,4,6 outcomes side by side | High | -| BadDataNeighborOrientationCheck | Voxel-and-neighbors diagram showing the false→true flip when criterion is met | Medium | -| RequireMinNumNeighbors | Microstructure before/after with isolated low-neighbor-count grains absorbed | Medium | -| ComputeNeighborhoods | Histogram of neighborhood counts for a real microstructure | Low | -| ComputeNeighborListStatistics | Example feature with its neighbor-misorientation list and computed statistics | Low | -| CreateGeometry | Side-by-side infographic of all 8 geometry types (Image, RectGrid, Vertex, Edge, Triangle, Quad, Tet, Hex) | High | -| CreateImageGeometry / SetImageGeomOriginScaling | Image Geometry dimensions/origin/spacing diagram (shared concept) | High | -| ApplyTransformationToGeometry | Axis-angle representation diagram | Medium | -| RotateSampleRefFrame | Sample frame rotation vs geometric rotation comparison | High | -| ResampleImageGeom | Before/after voxel grid showing 2x spacing reduction halving cell count | Medium | -| ResampleRectGridToImageGeom | Variable-spacing RectGrid → uniform Image Geom with "last one wins" cells highlighted | Medium | -| CropImageGeometry | Already adequate | - | -| PadImageGeometry | 3-panel diagram showing Update Origin ON vs OFF | Medium | -| AppendImageGeometry | Already comprehensive | - | -| QuickSurfaceMesh | Single voxel face → 2 triangles diagram | Low | -| PartitionGeometry | Already excellent | - | -| InitializeImageGeomCellData | Subvolume highlight before/after on real cube | Medium | -| CombineTransformationMatrices | Two 4x4 matrices multiplying into a single result | Low | -| ComputeCoordinatesImageGeom | Voxel grid with one cell highlighted showing (i,j,k) and (x,y,z) | Low | -| ReadImageStack | N 2D slices stacking into a 3D volume (Z = slice count) | Medium | -| ReadHDF5Dataset | Flat HDF5 dataset (N elements) reshaped into tuples x components | Medium | -| ReadStlFile | One triangle (normal + 3 vertices); many triangles forming a surface mesh | Low | -| ReadVtkStructuredPoints | Voxel cube showing cell-center value vs 8 corner (point) values | Medium | -| ReadH5Ebsd / ReadAng / ReadCtf | Sample vs crystal reference-frame diagram (why a rotation is needed); IPF map of freshly imported data | High | -| WriteAbaqusHexahedron | Image-geometry voxel grid mapping to a hexahedral element mesh (8 nodes per voxel) | Medium | -| WriteGBCDGMTFile | Data-flow: Compute GBCD -> Write GBCD GMT File -> .dat -> GMT -> rendered pole figure | High | -| WriteGBCDTriangleData | Boundary triangle: two grains, left/right average orientations, normal vector, area | Medium | -| WritePoleFigure | Stereographic/Lambert projection: 3D orientation/pole projected onto the 2D unit circle | High | -| WriteVtkRectilinearGrid / WriteVtkStructuredPoints | Rectilinear grid (variable spacing) vs structured points (uniform spacing) side by side | Medium | -| ReadDREAM3D | Import-data tree selector UI showing partial-import checkboxes | Medium | - ---- - -## Concept Pages Needed (Future Work) - -Domain concepts that appear across 3+ filters and should eventually become shared reference pages: - -| Concept | Filters That Need It | -|---------|---------------------| -| C-Axis / Hexagonal Crystals | ComputeAvgCAxes, ComputeCAxisLocations, ComputeFeatureNeighborCAxisMisalignments, ComputeFeatureReferenceCAxisMisorientations | -| Quaternions & Orientation Representations | ComputeAvgCAxes, ComputeAvgOrientations, ComputeCAxisLocations, ConvertOrientations | -| Misorientation | ComputeKernelAvgMisorientations, ComputeFeatureNeighborMisorientations, ComputeFeatureReferenceMisorientations, ComputeGBCD | -| Slip Systems & Crystal Plasticity | ComputeSchmids, ComputeSlipTransmissionMetrics, ComputeBoundaryStrengths | -| Grain Boundaries | ComputeGBCD, ComputeGBCDMetricBased, ComputeGBPDMetricBased, ComputeTwinBoundaries | -| Feature IDs & Grains | Nearly all filters | -| Reference Frames (Sample vs Crystal) | ComputeAvgCAxes, ComputeCAxisLocations, ComputeSchmids | -| Burn Algorithm / Segmentation | ScalarSegmentFeatures, CAxisSegmentFeatures, EBSDSegmentFeatures | -| Feature Cleanup / Isotropic Coarsening | RequireMinimumSizeFeatures, RemoveFlaggedFeatures | -| Cell vs Feature vs Ensemble Data | ScalarSegmentFeatures, RequireMinimumSizeFeatures, ComputeFeatureNeighbors, and most feature-level filters | -| Morphological Erosion / Dilation / Opening | ErodeDilateBadData, ErodeDilateMask, ErodeDilateCoordinationNumber | -| Coordination Number / Face Neighbors | ErodeDilateCoordinationNumber, NeighborOrientationCorrelation, BadDataNeighborOrientationCheck | -| Equivalent Sphere Diameter | ComputeFeatureSizes, ComputeNeighborhoods, ComputeShapes | -| Geometry Types (Image, RectGrid, Vertex, Edge, Triangle, Quad, Tet, Hex) | CreateGeometry, CreateImageGeometry, and every geometry-aware filter | -| Transformation Matrices (4x4 row-major) | ApplyTransformationToGeometry, CombineTransformationMatrices, RotateSampleRefFrame | -| Sample Reference Frame vs Crystal Reference Frame | RotateSampleRefFrame (already listed above) | -| HDF5 File Format (datasets, groups, tuples vs components) | ReadHDF5Dataset, ReadDREAM3D, WriteDREAM3D, ReadH5Ebsd, ReadH5Esprit, ReadH5Oim, ReadH5Oina, ReadGrainMapper3D | -| VTK Legacy Formats (structured points vs rectilinear grid; ASCII vs binary) | ReadVtkStructuredPoints, WriteVtkStructuredPoints, WriteVtkRectilinearGrid | -| EBSD Vendor File Formats & What They Contain (Euler angles, CI/IQ/MAD, phases) | ReadAngData, ReadCtfData, ReadChannel5Data, ReadH5Ebsd, ReadH5EspritData, ReadH5OimData, ReadH5OinaData | -| External Simulation Exporters (FEA / MD / KMC / FFT input files) | WriteAbaqusHexahedron, WriteLAMMPSFile, WriteSPParksSites, WriteLosAlamosFFT, ReadDeformKeyFileV12 | -| Pole Figures & Projections (stereographic, Lambert, Laue class) | WritePoleFigure, WriteGBCDGMTFile, ComputeGBCD | diff --git a/docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_grains.d3dstate b/docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_grains.d3dstate deleted file mode 100644 index e5c0cfa1bb..0000000000 --- a/docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_grains.d3dstate +++ /dev/null @@ -1,429 +0,0 @@ -{ - "charting_state": { - "0": { - "active_chart_tab": 0, - "active_view": true, - "chart_type_selectors": [ - { - "chart_type": 0, - "chart_type_visible": false - } - ], - "num_chart_tabs": 0 - }, - "dream3d_file_path": "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d" - }, - "pipeline_state": { - "isDisabled": false, - "name": "Untitled Pipeline", - "pinnedParams": [], - "pipeline": [ - { - "args": { - "import_data_object": { - "value": { - "data_paths": [], - "file_path": "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d", - "path_import_policy": 0 - }, - "version": 2 - }, - "parameters_version": 1 - }, - "comments": "", - "filter": { - "name": "nx::core::ReadDREAM3DFilter", - "uuid": "0dbd31c7-19e0-4077-83ef-f4a6459a0e2d" - }, - "isDisabled": false - } - ], - "pipeline_uuid": "4c120bda-d8be-4784-894a-9ec8cc5c41f4", - "version": 1, - "workflowParams": [] - }, - "quadview_state": { - "0": { - "active_quad_view_tab": 0, - "active_view": true, - "num_quad_view_tabs": 1, - "quad_view_tab_states": { - "0": { - "active_array": "", - "active_component": -1, - "ambient": 0.2, - "array_lut_fct": null, - "array_scalar_opacity_fct": null, - "color_map_inverted": false, - "diffuse": 0.8, - "geometry_path": "", - "gi_reach": 0.0, - "length_units": "mm", - "light_follows_camera": true, - "max_intensity_blend": true, - "mean_blend": false, - "min_intensity_blend": false, - "perPaneViewActionState": [ - {}, - {}, - {}, - {} - ], - "reslice_mode": false, - "shading_enabled": false, - "show_annotations": true, - "show_color_legend": true, - "show_plane_widgets": false, - "specular": 0.2, - "specular_power": 20.0, - "ssao_enabled": false, - "thick_mode": false, - "volume_blend_mode": 0, - "volumetric_scattering": 0.0 - } - } - }, - "quad_view_files": [ - "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d" - ] - }, - "spreadsheet_state": { - "0": { - "active_spreadsheet_tab": 0, - "active_view": true, - "num_spreadsheet_tabs": 0, - "spreadsheet_type_selectors": [ - { - "chart_type": 0, - "chart_type_visible": false - } - ] - }, - "dream3d_file_path": "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d" - }, - "state_file_version": 4, - "visualization_state": { - "split_view_state": { - "cells": [ - [ - { - "col_span": 1, - "row_span": 1, - "widget_state": { - "active_view": true, - "block_render": false, - "camera_center_of_rotation_x": -23.625, - "camera_center_of_rotation_y": 25.125, - "camera_center_of_rotation_z": -0.5, - "camera_focal_pt_x": -26.353316907594433, - "camera_focal_pt_y": 22.547708844875523, - "camera_focal_pt_z": -0.5, - "camera_parallel_scale": 33.14477492067222, - "camera_pos_x": -26.353316907594433, - "camera_pos_y": 22.547708844875523, - "camera_pos_z": 160.75006004466653, - "camera_view_angle": 30.0, - "camera_view_up_x": 0.0, - "camera_view_up_y": 1.0, - "camera_view_up_z": 0.0, - "check_context_menu": false, - "global_grid_axes": false, - "global_grid_axes_color": [ - 1.0, - 1.0, - 1.0 - ], - "global_grid_axes_opacity": 1.0, - "global_grid_axes_show_grid_lines": false, - "global_grid_axes_show_labels": true, - "global_grid_axes_spacingx": 0.0, - "global_grid_axes_spacingy": 0.0, - "global_grid_axes_spacingz": 0.0, - "global_grid_axes_title_x": "", - "global_grid_axes_title_y": "", - "global_grid_axes_title_z": "", - "num_render_layers": 2, - "own_context_menu": true, - "parallel_projection": 1, - "split_view_index": 1 - } - } - ] - ], - "col_widths": [ - 1.0 - ], - "grid_cols": 1, - "grid_rows": 1, - "grid_version": 1, - "row_heights": [ - 1.0 - ] - }, - "tree_view_state": [ - { - "item_children": [ - { - "item_children": [ - { - "item_data": { - "core_object_name": "2D Image/Slice View", - "core_object_name_suffix": 1, - "rep_view_implementations": [ - { - "active_array": "FeatureIds", - "active_component": -1, - "animation_duration": 0.5, - "color_map_inverted": false, - "color_preset": { - "Annotations": null, - "ColorSpace": "RGB", - "Creator": "Francesca Samsel, and Alan W. Scott", - "DefaultMap": true, - "IndexedColors": [], - "License": "", - "Name": "Fast", - "NanColor": [ - 0.0, - 1.0, - 0.0 - ], - "RGBPoints": [ - 0.0, - 0.05640000104904175, - 0.05640000104904175, - 0.4699999988079071, - 0.17159223556518555, - 0.24300000071525574, - 0.46035000681877136, - 0.8100000023841858, - 0.2984914779663086, - 0.3568143844604492, - 0.7450246214866638, - 0.9543676972389221, - 0.4321287274360657, - 0.6881999969482422, - 0.9300000071525574, - 0.9179099798202515, - 0.5, - 0.8994959592819214, - 0.944646418094635, - 0.7686567306518555, - 0.5882260203361511, - 0.9571079611778259, - 0.8338184952735901, - 0.5089156031608582, - 0.7061412334442139, - 0.927520751953125, - 0.6214389204978943, - 0.3153570592403412, - 0.8476395010948181, - 0.800000011920929, - 0.35199999809265137, - 0.1599999964237213, - 1.0, - 0.5899999737739563, - 0.07670000195503235, - 0.11947499960660934 - ], - "Source": "" - }, - "custom_range_max": 370.0, - "custom_range_min": 0.0, - "data_grid_visible": false, - "detach_color_settings": true, - "generate_histogram_data": false, - "global_grid_axes_color": [ - 1.0, - 1.0, - 1.0 - ], - "global_grid_axes_opacity": 1.0, - "global_grid_axes_show_grid_lines": false, - "global_grid_axes_show_labels": true, - "global_grid_axes_spacingx": 0.0, - "global_grid_axes_spacingy": 0.0, - "global_grid_axes_spacingz": 0.0, - "global_grid_axes_title_x": "", - "global_grid_axes_title_y": "", - "global_grid_axes_title_z": "", - "indexed_color_preset": { - "Annotations": null, - "ColorSpace": "", - "Creator": "", - "DefaultMap": false, - "IndexedColors": [ - 0.6509804129600525, - 0.8078431487083435, - 0.8901960849761963, - 0.12156862765550613, - 0.47058823704719543, - 0.7058823704719543, - 0.6980392336845398, - 0.8745098114013672, - 0.5411764979362488, - 0.20000000298023224, - 0.6274510025978088, - 0.1725490242242813, - 0.9843137264251709, - 0.6039215922355652, - 0.6000000238418579, - 0.8901960849761963, - 0.10196078568696976, - 0.10980392247438431, - 0.9921568632125854, - 0.7490196228027344, - 0.43529412150382996, - 1.0, - 0.49803921580314636, - 0.0, - 0.7921568751335144, - 0.6980392336845398, - 0.8392156958580017, - 0.4156862795352936, - 0.239215686917305, - 0.6039215922355652, - 1.0, - 1.0, - 0.6000000238418579 - ], - "License": "", - "Name": "Brewer Qualitative Paired", - "NanColor": [ - 1.0, - 1.0, - 0.6000000238418579 - ], - "RGBPoints": [], - "Source": "" - }, - "link_physical_units": false, - "nan_color_b": 0.0, - "nan_color_g": 0.0, - "nan_color_r": 0.0, - "render_window_index": 1, - "rep_linked": false, - "rep_split_id": 1, - "representation_visible": true, - "scalar_bar_visible": false, - "scalar_mapping": 1, - "show_box_tool": false, - "slice_dir": 0, - "slice_num": 0, - "solid_color_b": 255, - "solid_color_g": 255, - "solid_color_r": 255, - "source_position2_x": 0.1, - "source_position2_y": 0.5, - "source_position_x": 0.9, - "source_position_y": 0.175, - "toggle_left_slice_num": 0, - "toggle_right_slice_num": 0, - "transform_box_rotate_x": 0.0, - "transform_box_rotate_y": 0.0, - "transform_box_rotate_z": 0.0, - "transform_box_scale_x": 1.0, - "transform_box_scale_y": 1.0, - "transform_box_scale_z": 1.0, - "transform_box_translate_x": 0.0, - "transform_box_translate_y": 0.0, - "transform_box_translate_z": 0.0, - "use_decimation_during_interactions": false, - "using_custom_range": false, - "using_default_indexed_preset": true, - "using_default_preset": false, - "using_solid_color": false, - "values_as_categories": false - } - ] - }, - "item_type": "VXImageSliceRep " - } - ], - "item_data": { - "core_object_name": "DataContainer", - "current_preset": { - "Annotations": null, - "ColorSpace": "RGB", - "Creator": "Francesca Samsel, and Alan W. Scott", - "DefaultMap": true, - "IndexedColors": [], - "License": "", - "Name": "Fast", - "NanColor": [ - 0.0, - 1.0, - 0.0 - ], - "RGBPoints": [ - 0.0, - 0.05640000104904175, - 0.05640000104904175, - 0.4699999988079071, - 0.17159223556518555, - 0.24300000071525574, - 0.46035000681877136, - 0.8100000023841858, - 0.2984914779663086, - 0.3568143844604492, - 0.7450246214866638, - 0.9543676972389221, - 0.4321287274360657, - 0.6881999969482422, - 0.9300000071525574, - 0.9179099798202515, - 0.5, - 0.8994959592819214, - 0.944646418094635, - 0.7686567306518555, - 0.5882260203361511, - 0.9571079611778259, - 0.8338184952735901, - 0.5089156031608582, - 0.7061412334442139, - 0.927520751953125, - 0.6214389204978943, - 0.3153570592403412, - 0.8476395010948181, - 0.800000011920929, - 0.35199999809265137, - 0.1599999964237213, - 1.0, - 0.5899999737739563, - 0.07670000195503235, - 0.11947499960660934 - ], - "Source": "" - }, - "data_source_color_map_inverted": false, - "data_source_nan_color_b": 0, - "data_source_nan_color_g": 191, - "data_source_nan_color_r": 128, - "data_source_solid_color_b": 255, - "data_source_solid_color_g": 255, - "data_source_solid_color_r": 255, - "data_source_using_default_preset": true, - "data_source_visibility": true, - "imported_data_path": "DataContainer", - "render_window_index": [ - { - "core_object_visibility": true, - "render_window_index": 1 - } - ] - }, - "item_type": "VXDataSource" - } - ], - "item_data": { - "bridge_object_file_path": "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d", - "bridge_object_item_name": "Untitled Pipeline", - "bridge_object_uuid": "4c120bda-d8be-4784-894a-9ec8cc5c41f4" - }, - "item_type": "NXToVXBridge" - } - ] - }, - "window_state": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48UXRBZHZhbmNlZERvY2tpbmdTeXN0ZW0gVmVyc2lvbj0iMSIgVXNlclZlcnNpb249IjEiIENvbnRhaW5lcnM9IjEiPjxDb250YWluZXIgRmxvYXRpbmc9IjAiPjxTcGxpdHRlciBPcmllbnRhdGlvbj0ifCIgQ291bnQ9IjEiPjxTcGxpdHRlciBPcmllbnRhdGlvbj0iLSIgQ291bnQ9IjIiPjxTcGxpdHRlciBPcmllbnRhdGlvbj0ifCIgQ291bnQ9IjIiPjxTcGxpdHRlciBPcmllbnRhdGlvbj0iLSIgQ291bnQ9IjIiPjxBcmVhIFRhYnM9IjEiIEN1cnJlbnQ9Im1fUGlwZWxpbmVEb2NrIj48V2lkZ2V0IE5hbWU9Im1fUGlwZWxpbmVEb2NrIiBDbG9zZWQ9IjEiLz48L0FyZWE+PEFyZWEgVGFicz0iMiIgQ3VycmVudD0ibV9Cb29rbWFya3NEb2NrIj48V2lkZ2V0IE5hbWU9Im1fRmlsdGVyTGlzdERvY2siIENsb3NlZD0iMSIvPjxXaWRnZXQgTmFtZT0ibV9Cb29rbWFya3NEb2NrIiBDbG9zZWQ9IjEiLz48L0FyZWE+PFNpemVzPjAgMCA8L1NpemVzPjwvU3BsaXR0ZXI+PEFyZWEgVGFicz0iNCIgQ3VycmVudD0ibV9WaXN1YWxpemF0aW9uRG9jayI+PFdpZGdldCBOYW1lPSJtX0ZpbHRlclBhcmFtZXRlcnNEb2NrIiBDbG9zZWQ9IjEiLz48V2lkZ2V0IE5hbWU9Im1fV29ya2Zsb3dQYXJhbWV0ZXJzRG9jayIgQ2xvc2VkPSIxIi8+PFdpZGdldCBOYW1lPSJtX1Zpc3VhbGl6YXRpb25Eb2NrIiBDbG9zZWQ9IjAiLz48V2lkZ2V0IE5hbWU9Im1fQ2hhcnRpbmdEb2NrIiBDbG9zZWQ9IjEiLz48L0FyZWE+PFNpemVzPjAgMTA5NCA8L1NpemVzPjwvU3BsaXR0ZXI+PFNwbGl0dGVyIE9yaWVudGF0aW9uPSJ8IiBDb3VudD0iMyI+PEFyZWEgVGFicz0iMSIgQ3VycmVudD0ibV9PdXRwdXREb2NrIj48V2lkZ2V0IE5hbWU9Im1fT3V0cHV0RG9jayIgQ2xvc2VkPSIxIi8+PC9BcmVhPjxBcmVhIFRhYnM9IjEiIEN1cnJlbnQ9Im1fSXNzdWVzRG9jayI+PFdpZGdldCBOYW1lPSJtX0lzc3Vlc0RvY2siIENsb3NlZD0iMSIvPjwvQXJlYT48QXJlYSBUYWJzPSIxIiBDdXJyZW50PSJtX1Bpbm5lZFBhcmFtZXRlcnNEb2NrIj48V2lkZ2V0IE5hbWU9Im1fUGlubmVkUGFyYW1ldGVyc0RvY2siIENsb3NlZD0iMSIvPjwvQXJlYT48U2l6ZXM+MCAwIDAgPC9TaXplcz48L1NwbGl0dGVyPjxTaXplcz43ODkgMCA8L1NpemVzPjwvU3BsaXR0ZXI+PFNpemVzPjEwOTQgPC9TaXplcz48L1NwbGl0dGVyPjwvQ29udGFpbmVyPjwvUXRBZHZhbmNlZERvY2tpbmdTeXN0ZW0+Cg==", - "workflow_mode": false -} \ No newline at end of file diff --git a/docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_ipf.d3dstate b/docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_ipf.d3dstate deleted file mode 100644 index 20791f17c3..0000000000 --- a/docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_ipf.d3dstate +++ /dev/null @@ -1,429 +0,0 @@ -{ - "charting_state": { - "0": { - "active_chart_tab": 0, - "active_view": true, - "chart_type_selectors": [ - { - "chart_type": 0, - "chart_type_visible": false - } - ], - "num_chart_tabs": 0 - }, - "dream3d_file_path": "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d" - }, - "pipeline_state": { - "isDisabled": false, - "name": "Untitled Pipeline", - "pinnedParams": [], - "pipeline": [ - { - "args": { - "import_data_object": { - "value": { - "data_paths": [], - "file_path": "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d", - "path_import_policy": 0 - }, - "version": 2 - }, - "parameters_version": 1 - }, - "comments": "", - "filter": { - "name": "nx::core::ReadDREAM3DFilter", - "uuid": "0dbd31c7-19e0-4077-83ef-f4a6459a0e2d" - }, - "isDisabled": false - } - ], - "pipeline_uuid": "4c120bda-d8be-4784-894a-9ec8cc5c41f4", - "version": 1, - "workflowParams": [] - }, - "quadview_state": { - "0": { - "active_quad_view_tab": 0, - "active_view": true, - "num_quad_view_tabs": 1, - "quad_view_tab_states": { - "0": { - "active_array": "", - "active_component": -1, - "ambient": 0.2, - "array_lut_fct": null, - "array_scalar_opacity_fct": null, - "color_map_inverted": false, - "diffuse": 0.8, - "geometry_path": "", - "gi_reach": 0.0, - "length_units": "mm", - "light_follows_camera": true, - "max_intensity_blend": true, - "mean_blend": false, - "min_intensity_blend": false, - "perPaneViewActionState": [ - {}, - {}, - {}, - {} - ], - "reslice_mode": false, - "shading_enabled": false, - "show_annotations": true, - "show_color_legend": true, - "show_plane_widgets": false, - "specular": 0.2, - "specular_power": 20.0, - "ssao_enabled": false, - "thick_mode": false, - "volume_blend_mode": 0, - "volumetric_scattering": 0.0 - } - } - }, - "quad_view_files": [ - "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d" - ] - }, - "spreadsheet_state": { - "0": { - "active_spreadsheet_tab": 0, - "active_view": true, - "num_spreadsheet_tabs": 0, - "spreadsheet_type_selectors": [ - { - "chart_type": 0, - "chart_type_visible": false - } - ] - }, - "dream3d_file_path": "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d" - }, - "state_file_version": 4, - "visualization_state": { - "split_view_state": { - "cells": [ - [ - { - "col_span": 1, - "row_span": 1, - "widget_state": { - "active_view": true, - "block_render": false, - "camera_center_of_rotation_x": -23.625, - "camera_center_of_rotation_y": 25.125, - "camera_center_of_rotation_z": -0.5, - "camera_focal_pt_x": -26.353316907594433, - "camera_focal_pt_y": 22.547708844875523, - "camera_focal_pt_z": -0.5, - "camera_parallel_scale": 33.14477492067222, - "camera_pos_x": -26.353316907594433, - "camera_pos_y": 22.547708844875523, - "camera_pos_z": 160.75006004466653, - "camera_view_angle": 30.0, - "camera_view_up_x": 0.0, - "camera_view_up_y": 1.0, - "camera_view_up_z": 0.0, - "check_context_menu": false, - "global_grid_axes": false, - "global_grid_axes_color": [ - 1.0, - 1.0, - 1.0 - ], - "global_grid_axes_opacity": 1.0, - "global_grid_axes_show_grid_lines": false, - "global_grid_axes_show_labels": true, - "global_grid_axes_spacingx": 0.0, - "global_grid_axes_spacingy": 0.0, - "global_grid_axes_spacingz": 0.0, - "global_grid_axes_title_x": "", - "global_grid_axes_title_y": "", - "global_grid_axes_title_z": "", - "num_render_layers": 2, - "own_context_menu": true, - "parallel_projection": 1, - "split_view_index": 1 - } - } - ] - ], - "col_widths": [ - 1.0 - ], - "grid_cols": 1, - "grid_rows": 1, - "grid_version": 1, - "row_heights": [ - 1.0 - ] - }, - "tree_view_state": [ - { - "item_children": [ - { - "item_children": [ - { - "item_data": { - "core_object_name": "2D Image/Slice View", - "core_object_name_suffix": 1, - "rep_view_implementations": [ - { - "active_array": "IPFColors", - "active_component": -1, - "animation_duration": 0.5, - "color_map_inverted": false, - "color_preset": { - "Annotations": null, - "ColorSpace": "RGB", - "Creator": "Francesca Samsel, and Alan W. Scott", - "DefaultMap": true, - "IndexedColors": [], - "License": "", - "Name": "Fast", - "NanColor": [ - 0.0, - 1.0, - 0.0 - ], - "RGBPoints": [ - 0.0, - 0.05640000104904175, - 0.05640000104904175, - 0.4699999988079071, - 0.17159223556518555, - 0.24300000071525574, - 0.46035000681877136, - 0.8100000023841858, - 0.2984914779663086, - 0.3568143844604492, - 0.7450246214866638, - 0.9543676972389221, - 0.4321287274360657, - 0.6881999969482422, - 0.9300000071525574, - 0.9179099798202515, - 0.5, - 0.8994959592819214, - 0.944646418094635, - 0.7686567306518555, - 0.5882260203361511, - 0.9571079611778259, - 0.8338184952735901, - 0.5089156031608582, - 0.7061412334442139, - 0.927520751953125, - 0.6214389204978943, - 0.3153570592403412, - 0.8476395010948181, - 0.800000011920929, - 0.35199999809265137, - 0.1599999964237213, - 1.0, - 0.5899999737739563, - 0.07670000195503235, - 0.11947499960660934 - ], - "Source": "" - }, - "custom_range_max": 431.457, - "custom_range_min": 0.0, - "data_grid_visible": false, - "detach_color_settings": true, - "generate_histogram_data": false, - "global_grid_axes_color": [ - 1.0, - 1.0, - 1.0 - ], - "global_grid_axes_opacity": 1.0, - "global_grid_axes_show_grid_lines": false, - "global_grid_axes_show_labels": true, - "global_grid_axes_spacingx": 0.0, - "global_grid_axes_spacingy": 0.0, - "global_grid_axes_spacingz": 0.0, - "global_grid_axes_title_x": "", - "global_grid_axes_title_y": "", - "global_grid_axes_title_z": "", - "indexed_color_preset": { - "Annotations": null, - "ColorSpace": "", - "Creator": "", - "DefaultMap": false, - "IndexedColors": [ - 0.6509804129600525, - 0.8078431487083435, - 0.8901960849761963, - 0.12156862765550613, - 0.47058823704719543, - 0.7058823704719543, - 0.6980392336845398, - 0.8745098114013672, - 0.5411764979362488, - 0.20000000298023224, - 0.6274510025978088, - 0.1725490242242813, - 0.9843137264251709, - 0.6039215922355652, - 0.6000000238418579, - 0.8901960849761963, - 0.10196078568696976, - 0.10980392247438431, - 0.9921568632125854, - 0.7490196228027344, - 0.43529412150382996, - 1.0, - 0.49803921580314636, - 0.0, - 0.7921568751335144, - 0.6980392336845398, - 0.8392156958580017, - 0.4156862795352936, - 0.239215686917305, - 0.6039215922355652, - 1.0, - 1.0, - 0.6000000238418579 - ], - "License": "", - "Name": "Brewer Qualitative Paired", - "NanColor": [ - 1.0, - 1.0, - 0.6000000238418579 - ], - "RGBPoints": [], - "Source": "" - }, - "link_physical_units": false, - "nan_color_b": 0.0, - "nan_color_g": 0.0, - "nan_color_r": 0.0, - "render_window_index": 1, - "rep_linked": false, - "rep_split_id": 1, - "representation_visible": true, - "scalar_bar_visible": false, - "scalar_mapping": 1, - "show_box_tool": false, - "slice_dir": 0, - "slice_num": 0, - "solid_color_b": 255, - "solid_color_g": 255, - "solid_color_r": 255, - "source_position2_x": 0.1, - "source_position2_y": 0.5, - "source_position_x": 0.9, - "source_position_y": 0.175, - "toggle_left_slice_num": 0, - "toggle_right_slice_num": 0, - "transform_box_rotate_x": 0.0, - "transform_box_rotate_y": 0.0, - "transform_box_rotate_z": 0.0, - "transform_box_scale_x": 1.0, - "transform_box_scale_y": 1.0, - "transform_box_scale_z": 1.0, - "transform_box_translate_x": 0.0, - "transform_box_translate_y": 0.0, - "transform_box_translate_z": 0.0, - "use_decimation_during_interactions": false, - "using_custom_range": false, - "using_default_indexed_preset": true, - "using_default_preset": false, - "using_solid_color": false, - "values_as_categories": false - } - ] - }, - "item_type": "VXImageSliceRep " - } - ], - "item_data": { - "core_object_name": "DataContainer", - "current_preset": { - "Annotations": null, - "ColorSpace": "RGB", - "Creator": "Francesca Samsel, and Alan W. Scott", - "DefaultMap": true, - "IndexedColors": [], - "License": "", - "Name": "Fast", - "NanColor": [ - 0.0, - 1.0, - 0.0 - ], - "RGBPoints": [ - 0.0, - 0.05640000104904175, - 0.05640000104904175, - 0.4699999988079071, - 0.17159223556518555, - 0.24300000071525574, - 0.46035000681877136, - 0.8100000023841858, - 0.2984914779663086, - 0.3568143844604492, - 0.7450246214866638, - 0.9543676972389221, - 0.4321287274360657, - 0.6881999969482422, - 0.9300000071525574, - 0.9179099798202515, - 0.5, - 0.8994959592819214, - 0.944646418094635, - 0.7686567306518555, - 0.5882260203361511, - 0.9571079611778259, - 0.8338184952735901, - 0.5089156031608582, - 0.7061412334442139, - 0.927520751953125, - 0.6214389204978943, - 0.3153570592403412, - 0.8476395010948181, - 0.800000011920929, - 0.35199999809265137, - 0.1599999964237213, - 1.0, - 0.5899999737739563, - 0.07670000195503235, - 0.11947499960660934 - ], - "Source": "" - }, - "data_source_color_map_inverted": false, - "data_source_nan_color_b": 0, - "data_source_nan_color_g": 191, - "data_source_nan_color_r": 128, - "data_source_solid_color_b": 255, - "data_source_solid_color_g": 255, - "data_source_solid_color_r": 255, - "data_source_using_default_preset": true, - "data_source_visibility": true, - "imported_data_path": "DataContainer", - "render_window_index": [ - { - "core_object_visibility": true, - "render_window_index": 1 - } - ] - }, - "item_type": "VXDataSource" - } - ], - "item_data": { - "bridge_object_file_path": "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d", - "bridge_object_item_name": "Untitled Pipeline", - "bridge_object_uuid": "4c120bda-d8be-4784-894a-9ec8cc5c41f4" - }, - "item_type": "NXToVXBridge" - } - ] - }, - "window_state": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz48UXRBZHZhbmNlZERvY2tpbmdTeXN0ZW0gVmVyc2lvbj0iMSIgVXNlclZlcnNpb249IjEiIENvbnRhaW5lcnM9IjEiPjxDb250YWluZXIgRmxvYXRpbmc9IjAiPjxTcGxpdHRlciBPcmllbnRhdGlvbj0ifCIgQ291bnQ9IjEiPjxTcGxpdHRlciBPcmllbnRhdGlvbj0iLSIgQ291bnQ9IjIiPjxTcGxpdHRlciBPcmllbnRhdGlvbj0ifCIgQ291bnQ9IjIiPjxTcGxpdHRlciBPcmllbnRhdGlvbj0iLSIgQ291bnQ9IjIiPjxBcmVhIFRhYnM9IjEiIEN1cnJlbnQ9Im1fUGlwZWxpbmVEb2NrIj48V2lkZ2V0IE5hbWU9Im1fUGlwZWxpbmVEb2NrIiBDbG9zZWQ9IjEiLz48L0FyZWE+PEFyZWEgVGFicz0iMiIgQ3VycmVudD0ibV9Cb29rbWFya3NEb2NrIj48V2lkZ2V0IE5hbWU9Im1fRmlsdGVyTGlzdERvY2siIENsb3NlZD0iMSIvPjxXaWRnZXQgTmFtZT0ibV9Cb29rbWFya3NEb2NrIiBDbG9zZWQ9IjEiLz48L0FyZWE+PFNpemVzPjAgMCA8L1NpemVzPjwvU3BsaXR0ZXI+PEFyZWEgVGFicz0iNCIgQ3VycmVudD0ibV9WaXN1YWxpemF0aW9uRG9jayI+PFdpZGdldCBOYW1lPSJtX0ZpbHRlclBhcmFtZXRlcnNEb2NrIiBDbG9zZWQ9IjEiLz48V2lkZ2V0IE5hbWU9Im1fV29ya2Zsb3dQYXJhbWV0ZXJzRG9jayIgQ2xvc2VkPSIxIi8+PFdpZGdldCBOYW1lPSJtX1Zpc3VhbGl6YXRpb25Eb2NrIiBDbG9zZWQ9IjAiLz48V2lkZ2V0IE5hbWU9Im1fQ2hhcnRpbmdEb2NrIiBDbG9zZWQ9IjEiLz48L0FyZWE+PFNpemVzPjAgMTA5NCA8L1NpemVzPjwvU3BsaXR0ZXI+PFNwbGl0dGVyIE9yaWVudGF0aW9uPSJ8IiBDb3VudD0iMyI+PEFyZWEgVGFicz0iMSIgQ3VycmVudD0ibV9PdXRwdXREb2NrIj48V2lkZ2V0IE5hbWU9Im1fT3V0cHV0RG9jayIgQ2xvc2VkPSIxIi8+PC9BcmVhPjxBcmVhIFRhYnM9IjEiIEN1cnJlbnQ9Im1fSXNzdWVzRG9jayI+PFdpZGdldCBOYW1lPSJtX0lzc3Vlc0RvY2siIENsb3NlZD0iMSIvPjwvQXJlYT48QXJlYSBUYWJzPSIxIiBDdXJyZW50PSJtX1Bpbm5lZFBhcmFtZXRlcnNEb2NrIj48V2lkZ2V0IE5hbWU9Im1fUGlubmVkUGFyYW1ldGVyc0RvY2siIENsb3NlZD0iMSIvPjwvQXJlYT48U2l6ZXM+MCAwIDAgPC9TaXplcz48L1NwbGl0dGVyPjxTaXplcz43ODkgMCA8L1NpemVzPjwvU3BsaXR0ZXI+PFNpemVzPjEwOTQgPC9TaXplcz48L1NwbGl0dGVyPjwvQ29udGFpbmVyPjwvUXRBZHZhbmNlZERvY2tpbmdTeXN0ZW0+Cg==", - "workflow_mode": false -} \ No newline at end of file diff --git a/docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_pilot.d3dpipeline b/docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_pilot.d3dpipeline deleted file mode 100644 index d793376c55..0000000000 --- a/docs/figure-generation/EBSDSegmentFeatures/ebsd_segment_pilot.d3dpipeline +++ /dev/null @@ -1,130 +0,0 @@ -{ - "isDisabled": false, - "name": "ebsd_segment_pilot.d3dpipeline", - "pinnedParams": [], - "pipeline": [ - { - "args": { - "cell_attribute_matrix_name": { "value": "Cell Data", "version": 1 }, - "cell_ensemble_attribute_matrix_name": { "value": "Cell Ensemble Data", "version": 1 }, - "input_file": { "value": "/Users/mjackson/Workspace1/DREAM3D_Data/Data/SmallIN100/Slice_1.ang", "version": 1 }, - "output_image_geometry_path": { "value": "DataContainer", "version": 1 }, - "parameters_version": 1 - }, - "comments": "Read single EDAX .ang slice -> DataContainer", - "filter": { "name": "nx::core::ReadAngDataFilter", "uuid": "5b062816-79ac-47ce-93cb-e7966896bcbd" }, - "isDisabled": false - }, - { - "args": { - "euler_angles_array_path": { "value": "DataContainer/Cell Data/EulerAngles", "version": 1 }, - "parameters_version": 1, - "rotation_axis_angle": { "value": [0.0, 0.0, 1.0, 90.0], "version": 1 } - }, - "comments": "Rotate Euler reference frame for EDAX convention", - "filter": { "name": "nx::core::RotateEulerRefFrameFilter", "uuid": "0458edcd-3655-4465-adc8-b036d76138b5" }, - "isDisabled": false - }, - { - "args": { - "input_image_geometry_path": { "value": "DataContainer", "version": 1 }, - "keep_input_geometry_origin": { "value": false, "version": 1 }, - "output_image_geometry_path": { "value": "DataContainer", "version": 1 }, - "parameters_version": 1, - "remove_original_geometry": { "value": true, "version": 1 }, - "rotate_slice_by_slice": { "value": false, "version": 1 }, - "rotation_axis_angle": { "value": [0.0, 1.0, 0.0, 180.0], "version": 1 }, - "rotation_matrix": { "value": [[0.0,0.0,0.0],[0.0,0.0,0.0],[0.0,0.0,0.0]], "version": 1 }, - "rotation_representation_index": { "value": 0, "version": 1 } - }, - "comments": "Rotate sample reference frame for EDAX convention", - "filter": { "name": "nx::core::RotateSampleRefFrameFilter", "uuid": "d2451dc1-a5a1-4ac2-a64d-7991669dcffc" }, - "isDisabled": false - }, - { - "args": { - "array_thresholds_object": { - "value": { - "inverted": false, - "thresholds": [ - { "array_path": "DataContainer/Cell Data/Confidence Index", "comparison": 0, "component_index": 0, "inverted": false, "type": "array", "union": 0, "value": 0.1 } - ], - "type": "collection", - "union": 0 - }, - "version": 2 - }, - "created_mask_type": { "value": 10, "version": 1 }, - "custom_false_value": { "value": 0.0, "version": 1 }, - "custom_true_value": { "value": 1.0, "version": 1 }, - "output_data_array_name": { "value": "Mask", "version": 1 }, - "parameters_version": 1, - "use_custom_false_value": { "value": false, "version": 1 }, - "use_custom_true_value": { "value": false, "version": 1 } - }, - "comments": "Build Mask from Confidence Index >= 0.1", - "filter": { "name": "nx::core::MultiThresholdObjectsFilter", "uuid": "4246245e-1011-4add-8436-0af6bed19228" }, - "isDisabled": false - }, - { - "args": { - "input_orientation_array_path": { "value": "DataContainer/Cell Data/EulerAngles", "version": 1 }, - "input_representation_index": { "value": 0, "version": 1 }, - "output_orientation_array_name": { "value": "Quats", "version": 1 }, - "output_representation_index": { "value": 2, "version": 1 }, - "parameters_version": 1 - }, - "comments": "Convert Euler angles -> Quaternions (needed by Segment Features)", - "filter": { "name": "nx::core::ConvertOrientationsFilter", "uuid": "501e54e6-a66f-4eeb-ae37-00e649c00d4b" }, - "isDisabled": false - }, - { - "args": { - "cell_euler_angles_array_path": { "value": "DataContainer/Cell Data/EulerAngles", "version": 1 }, - "cell_ipf_colors_array_name": { "value": "IPFColors", "version": 1 }, - "cell_phases_array_path": { "value": "DataContainer/Cell Data/Phases", "version": 1 }, - "crystal_structures_array_path": { "value": "DataContainer/Cell Ensemble Data/CrystalStructures", "version": 1 }, - "mask_array_path": { "value": "DataContainer/Cell Data/Mask", "version": 1 }, - "parameters_version": 1, - "reference_dir": { "value": [0.0, 0.0, 1.0], "version": 1 }, - "use_mask": { "value": true, "version": 1 } - }, - "comments": "Compute IPF colors -> IPFColors (the BEFORE coloring)", - "filter": { "name": "nx::core::ComputeIPFColorsFilter", "uuid": "64cb4f27-6e5e-4dd2-8a03-0c448cb8f5e6" }, - "isDisabled": false - }, - { - "args": { - "active_array_name": { "value": "Active", "version": 1 }, - "cell_feature_attribute_matrix_name": { "value": "Cell Feature Data", "version": 1 }, - "cell_mask_array_path": { "value": "DataContainer/Cell Data/Mask", "version": 1 }, - "cell_phases_array_path": { "value": "DataContainer/Cell Data/Phases", "version": 1 }, - "cell_quats_array_path": { "value": "DataContainer/Cell Data/Quats", "version": 1 }, - "crystal_structures_array_path": { "value": "DataContainer/Cell Ensemble Data/CrystalStructures", "version": 1 }, - "feature_ids_array_name": { "value": "FeatureIds", "version": 1 }, - "input_image_geometry_path": { "value": "DataContainer", "version": 1 }, - "is_periodic": { "value": false, "version": 1 }, - "misorientation_tolerance": { "value": 5.0, "version": 1 }, - "neighbor_scheme_index": { "value": 0, "version": 1 }, - "parameters_version": 2, - "randomize_features": { "value": true, "version": 1 }, - "use_mask": { "value": true, "version": 1 } - }, - "comments": "Segment Features (Misorientation) -> FeatureIds (the AFTER coloring). randomize_features=true for visually distinct grain colors.", - "filter": { "name": "nx::core::EBSDSegmentFeaturesFilter", "uuid": "1810c2c7-63e3-41db-b204-a5821e6271c0" }, - "isDisabled": false - }, - { - "args": { - "export_file_path": { "value": "/Users/mjackson/Workspace1/DREAM3D-Build/NX-Com-Qt69-Vtk95-Rel/Bin/Data/Output/DocFigures/ebsd_segment_pilot.dream3d", "version": 1 }, - "parameters_version": 1, - "write_xdmf_file": { "value": false, "version": 1 } - }, - "comments": "Write everything to a regenerable .dream3d (not committed)", - "filter": { "name": "nx::core::WriteDREAM3DFilter", "uuid": "b3a95784-2ced-41ec-8d3d-0242ac130003" }, - "isDisabled": false - } - ], - "version": 1, - "workflowParams": [] -} diff --git a/docs/filter-figure-style.md b/docs/filter-figure-style.md index bd847b5d7b..1edad8c30b 100644 --- a/docs/filter-figure-style.md +++ b/docs/filter-figure-style.md @@ -20,7 +20,7 @@ Visual style and file conventions for figures generated to support filter docume ## Font ``` -'Segoe UI', 'Helvetica Neue', Arial, sans-serif +'Helvetica Neue', Arial, sans-serif ``` | Element | Size | Weight | Color | diff --git a/docs/superpowers/specs/2026-04-12-documentation-review-design.md b/docs/superpowers/specs/2026-04-12-documentation-review-design.md deleted file mode 100644 index cb5aea99d2..0000000000 --- a/docs/superpowers/specs/2026-04-12-documentation-review-design.md +++ /dev/null @@ -1,327 +0,0 @@ -# DREAM3DNX Filter Documentation Review — Design Spec - -**Date:** 2026-04-12 -**Scope:** ~295 filter documentation files across SimplnxCore (154), OrientationAnalysis (53), and ITKImageProcessing (88) -**Branch:** Single PR when all batches are complete - ---- - -## 1. Goals - -### Primary Goals - -1. **Clarity for non-domain-experts** — Documentation should be understandable by a general engineer or scientist (audience B) who understands data processing concepts but may not know EBSD, crystallography, or microstructure terminology. -2. **Accessibility for non-native English speakers** — Straightforward language, short sentences, consistent terminology. Avoid idioms and ambiguous phrasing. -3. **Figures where they help** — Add conceptual diagrams for filters where visual explanation meaningfully aids understanding. Not every filter needs a figure (e.g., unit conversion filters don't). -4. **Parameter guidance** — Help users determine if a filter is appropriate for their use case and what parameter values to use. - -### Non-Goals (Future Work) - -- **Documentation architecture consolidation** — The existing concept docs are scattered across `wrapping/python/docs/source/`, `DREAM3DNX/Documentation/Sphinx/source/Concepts/`, and `Downloads/dream3d_infographics/`. Consolidating these into a single canonical location is a future task, not part of this review. -- **Sphinx format migration** — Current filter docs are Markdown. Converting to RST or standardizing format is a future task. -- **Real-world pipeline visualizations** — Pipeline-generated screenshots require DREAM3DNX automation tooling that is still in progress. This review will build a wishlist of desired visualizations, but generating them is a future task. - ---- - -## 2. Target Audience - -**Audience (B): General engineer/scientist** - -- Understands data processing, arrays, filtering, statistics concepts -- May know something about EBSD or have data they want to process -- Looks to DREAM3D to help with processing and needs help figuring out if a filter is appropriate -- May not know crystallography, quaternions, Euler angles, microstructure terminology -- English may not be their primary language - -The documentation should serve both this audience AND domain experts. Domain experts need parameter guidance and edge case documentation; non-experts need conceptual explanations of what the filter does and when to use it. - ---- - -## 3. Review Process - -### Phase 1: Triage Pass - -Claude autonomously reviews all ~295 filter docs and produces a tracker document (`docs/documentation_review_tracker.md`) that categorizes every filter into a priority tier. - -**Triage Rubric — 5 Dimensions:** - -| Dimension | What We're Assessing | -|-----------|---------------------| -| **Clarity** | Can audience (B) understand what the filter does and when to use it? | -| **Completeness** | Are all parameters explained? Are edge cases/caveats documented? | -| **Accessibility** | Is the language straightforward? Are domain terms explained or defined on first use? | -| **Figures Needed** | Would a conceptual diagram meaningfully help understanding? | -| **Real-World Viz** | Would a pipeline-generated screenshot/visualization add value? (wishlist only) | -| **Units Clarity** | Are all numeric parameters (sizes, distances, radii, tolerances, offsets) documented with explicit units (cells/voxels, degrees, physical units, dimensionless)? | - -Additionally, track: -- **Concept Links** — Domain concepts that should link to shared concept pages (e.g., Quaternions, Feature IDs, Image Geometry). When the same concept appears across 3+ filters, that signals a future shared concept page. - -**Priority Tiers:** - -| Tier | Description | Expected Action | -|------|-------------|-----------------| -| **Tier 1 (Critical)** | Domain-heavy filters with sparse or expert-only docs. Non-expert cannot understand what the filter does. | Major rewrite: add conceptual explanation, define terms, add figures | -| **Tier 2 (Important)** | Docs are functional but assume too much domain knowledge, or would clearly benefit from figures. | Moderate rewrite: add context, simplify language, generate figures | -| **Tier 3 (Polish)** | Docs are mostly clear but could benefit from minor language cleanup, consistency, or term definitions. | Light editing: clarify wording, add term definitions inline | -| **Tier 4 (Adequate)** | Docs are already clear, complete, and accessible. Simple utility/I/O filters that don't need more. | No action needed | - -### Phase 2: Batched Rewrites - -After triage, work through filters in batches grouped by functional category (e.g., all alignment filters, all orientation statistics filters). Working by category ensures: - -- Consistent explanations across related filters -- Shared domain concepts identified naturally -- Cross-references between related filters - -**Batch workflow:** - -1. Claude rewrites the batch of filter docs -2. Claude generates any needed conceptual figures (following the style guide) -3. User reviews the changes -4. User commits the batch when satisfied -5. Move to next batch - -All batches accumulate on one branch and are submitted as a single PR when complete. - -### Phase 3: Future Work - -- Consolidate shared concept pages into a canonical location within this repo -- Generate real-world pipeline visualizations using DREAM3DNX automation -- Consider RST migration for consistency with Sphinx build - ---- - -## 4. Tracker Document Format - -The tracker (`docs/documentation_review_tracker.md`) uses a two-level structure: - -**Level 1: Summary tables** — One table per priority tier for quick scanning. - -```markdown -## Tier 1: Critical - -| Filter | Plugin | Category | -|--------|--------|----------| -| ComputeAvgOrientations | OrientationAnalysis | Statistics | -| ConvertOrientations | OrientationAnalysis | Conversion | -``` - -**Level 2: Detailed checklists** — Below the summary tables, a checklist with full details for each filter, organized by tier. - -```markdown -### Tier 1 — Detailed Checklist - -- [ ] **ComputeAvgOrientations** (OrientationAnalysis) - - **Clarity:** Assumes deep quaternion knowledge; no intuitive explanation - - **Figures Needed:** Diagram showing orientations being averaged within a grain - - **Real-World Viz:** IPF color map showing average orientation assignment - - **Concept Links:** Quaternions, Symmetry Operators, Feature/Grain - - **Notes:** Only 36 lines; needs significant expansion - -- [ ] **ConvertOrientations** (OrientationAnalysis) - - **Clarity:** Lists 8 types but doesn't explain when to choose each - - **Figures Needed:** Visual comparison of orientation representations - - **Real-World Viz:** None - - **Concept Links:** Euler Angles, Quaternions, Rodrigues Vectors - - **Notes:** 113 lines but very dense; needs "which should I use?" guidance -``` - -Filters are checked off as their rewrites are completed and committed. - ---- - -## 5. Documentation Rewrite Guidelines - -### Structure for Each Filter Doc - -Every filter doc should follow this structure (scaled to the filter's complexity): - -1. **Title** — Filter name -2. **Group (Subgroup)** — Category classification -3. **Description** — What the filter does in plain language. Lead with the "what" and "when would I use this?" before diving into algorithm details. If domain terms are needed, define them on first use or link to a concept page. -4. **Algorithm Details** (if applicable) — How it works, with figures where they help -5. **Parameter Guidance** (if applicable) — What the parameters mean, what values are typical, how they affect results -6. **Figures** — Conceptual diagrams embedded as PNG images -7. **Auto-generated parameter table** — `% Auto generated parameter table will be inserted here` -8. **Example Pipelines** — Links to relevant pipelines -9. **License & Copyright** -10. **DREAM3D-NX Help** — Link to discussions - -### Writing Style - -- **Short sentences.** Prefer active voice. -- **Define domain terms on first use.** Example: "the misorientation (the angular difference between two crystal orientations)" -- **Lead with purpose.** Start with what the filter does and why someone would use it, not how it works internally. -- **Use "you" sparingly.** Prefer "the filter" or "this operation" for international clarity. -- **Avoid idioms and colloquialisms.** Write for translation-friendliness. -- **Be specific about parameters.** Don't just name them — explain what values are typical and how changing them affects results. -- **Always specify units for numeric parameters.** Any parameter that accepts a size, distance, radius, offset, tolerance, or similar numeric value must clearly state whether the value is in **cell/voxel units** (integer cell counts), **physical units** (micrometers, degrees, etc.), or **dimensionless** (e.g., a direction vector). This is critical for users who may not know whether a "radius of 3" means 3 cells or 3 micrometers. - -### Figure Decision Framework - -**Add a figure when:** -- The filter operates on spatial data and the spatial relationship matters (kernels, neighborhoods, alignment) -- The filter has a multi-step algorithm that benefits from a flowchart or step diagram -- The filter transforms data in a way that's easier to show than describe (orientation conversions, geometry operations) -- The filter has parameters whose effect is non-obvious (kernel radius, dilation distance) - -**Skip a figure when:** -- The filter performs a simple mathematical operation (radians to degrees, threshold) -- The filter is a basic I/O operation (read file, write file) -- The filter's behavior is fully captured by its parameter names (delete data, rename data) - ---- - -## 6. Figure Style Guide - -All generated figures follow a consistent visual style. The style guide is saved as `docs/style_palette_final.svg` for reference. - -### Font - -``` -'Segoe UI', 'Helvetica Neue', Arial, sans-serif -``` - -| Element | Size | Weight | Color | -|---------|------|--------|-------| -| Figure title | 17-18px | Bold (700) | #2C3E50 | -| Section heading | 14px | Semibold (600) | #2C3E50 | -| Body text | 13px | Regular (400) | #555555 | -| Captions/notes | 11px | Italic | #888888 | - -### Color Palette - -**Primary (Diagram Elements):** - -| Role | Color | Hex | Usage | -|------|-------|-----|-------| -| Focus | Dark Blue | #2874A6 | Center cell, primary element, section headers | -| Active | Light Blue | #AED6F1 | Kernel neighbors, included elements | -| Inactive | Light Gray | #E8ECF0 | Outside kernel, excluded elements | - -**Semantic Accents:** - -| Role | Color | Hex | Usage | -|------|-------|-----|-------| -| Danger/Warning | Red | #E74C3C | Thread-safety warnings, invalid approaches, common mistakes | -| Good/Correct | Green | #27AE60 | Recommended approaches, valid results, correct measurements | -| Annotation | Purple | #8E44AD | Brackets, dimensions, labels, measurements | -| Tip/Highlight | Amber | #E67E22 | Tips, parameters of interest, optional recommendations | - -**Supporting:** - -| Role | Color | Hex | Usage | -|------|-------|-----|-------| -| Primary text | Dark Slate | #2C3E50 | Headings, primary labels | -| Secondary text | Medium Gray | #5D6D7E | Secondary labels, descriptions | -| Card background | White | #FFFFFF | Figure card background | -| Formula background | Pale Blue | #EBF5FB | Formula/equation highlight boxes | -| Warning background | Pale Red | #FDEDEC | Warning callout box fill | -| Success background | Pale Green | #EAFAF1 | Success callout box fill | -| Annotation background | Pale Purple | #F4ECF7 | Note callout box fill | -| Tip background | Pale Amber | #FEF5E7 | Tip callout box fill | -| Page background | Off-white | #F0F2F5 | Outer page fill (when used) | - -### Callout Box Styles - -Four semantic callout box types are available: - -- **Warning (Red)** — `!` icon, #FDEDEC background, #E74C3C border. For dangers and things to watch out for. -- **Good/Correct (Green)** — checkmark icon, #EAFAF1 background, #27AE60 border. For recommended approaches. -- **Note (Purple)** — `i` icon, #F4ECF7 background, #8E44AD border. For general annotations and clarifications. -- **Tip (Amber)** — star icon, #FEF5E7 background, #E67E22 border. For tips and highlights. - -### Section Headers - -Colored bars with white text, using the semantic color that best fits the section content. Default to blue (#2874A6) for neutral sections. - -### Figure File Format - -- Generate as SVG source files -- Convert to PNG at 2x resolution using `rsvg-convert -z 2` -- Keep both SVG and PNG in `docs/Images/` (or the plugin's `docs/Images/` directory) -- Reference the PNG in documentation markdown -- Naming convention: `FilterName_Description.png` (e.g., `ComputeKernelAvgMisorientations_1D_Radius1.png`) - ---- - -## 7. Concept Link Strategy - -During triage and rewrite, track domain concepts that appear across multiple filters. When a concept appears in 3+ filter docs, it is a candidate for a shared concept page. - -**During this review:** -- Note concept links needed in the tracker document -- Define terms inline in filter docs on first use -- Use placeholder links in the format `[concept name](link-tbd)` where a shared page would be ideal - -**Future task:** -- Create shared concept pages in a canonical location within this repo -- Replace placeholder links with real links -- Consolidate existing concept docs from scattered locations - -**Likely concept pages based on initial survey:** -- Quaternions and Orientation Representations -- Feature IDs and Grains -- Image Geometry (Voxels, Dimensions, Spacing, Origin) -- Cell vs. Feature vs. Ensemble Data -- Misorientation -- Crystallographic Symmetry -- Reference Frames and Conventions - ---- - -## 8. Batch Organization - -Rewrites are organized by functional category. Suggested batch order (highest impact first): - -| Batch | Category | Approx. Count | Rationale | -|-------|----------|---------------|-----------| -| 1 | Orientation / Crystallography Statistics | ~15 | Highest domain complexity, most likely Tier 1 | -| 2 | Alignment Filters | ~8 | Spatial concepts benefit from figures | -| 3 | Segmentation / Feature Identification | ~10 | Core workflow, many users encounter these | -| 4 | Neighbor / Kernel Operations | ~12 | Spatial concepts, parameter guidance needed | -| 5 | Geometry Creation / Manipulation | ~15 | Many options, complex parameter interactions | -| 6 | Data Manipulation (Copy, Create, Delete, Rename) | ~20 | Mostly Tier 3-4, quick to process | -| 7 | I/O Filters (Read/Write) | ~15 | Mostly Tier 3-4 | -| 8 | Image Processing (ITK wrappers) | ~88 | Large batch, many follow a template pattern | -| 9 | Remaining SimplnxCore filters | ~varies | Catch-all for uncategorized | - -Exact batch boundaries will be refined after the triage pass reveals the actual distribution across tiers and categories. - ---- - -## 9. Workflow Summary - -``` -Phase 1: Triage - Claude reviews all ~295 filter docs - → Produces docs/documentation_review_tracker.md - → User reviews tracker, adjusts tiers if needed - → User commits tracker - -Phase 2: Batched Rewrites (repeat per batch) - Claude rewrites batch of related filter docs - Claude generates conceptual figures (SVG → PNG) - → User reviews changes - → User commits batch - → Check off completed filters in tracker - -Phase 3: Final - All batches complete - → Submit single PR to develop - → Future: concept page consolidation, real-world visualizations -``` - ---- - -## 10. Files Created During This Design Phase - -| File | Purpose | -|------|---------| -| `docs/style_palette_final.svg` | Visual style guide for all generated figures | -| `docs/style_palette_options.svg` | Palette options explored during design (can be deleted) | -| `kernel_radius_infographic.svg` | Original combined infographic (repo root, can be deleted) | -| `src/Plugins/OrientationAnalysis/docs/Images/ComputeKernelAvgMisorientations_*.svg` | Individual figure SVG sources | -| `src/Plugins/OrientationAnalysis/docs/Images/ComputeKernelAvgMisorientations_*.png` | Individual figure PNGs for documentation | -| `src/Plugins/OrientationAnalysis/docs/ComputeKernelAvgMisorientationsFilter.md` | Updated filter doc (serves as template for rewrite style) | diff --git a/docs/upstream-filter-sources.md b/docs/upstream-filter-sources.md deleted file mode 100644 index 43ac21f919..0000000000 --- a/docs/upstream-filter-sources.md +++ /dev/null @@ -1,73 +0,0 @@ -# Upstream Filter Sources - -Lookup table mapping common input arrays to the filter(s) that produce them. Used when writing the **Required Input Sources** section of a filter's documentation (see the `vv-filter-documentation` skill for the convention). - -This file is intended to be grown over time. As more filters are documented or new filters are added, append rows. The longer-term goal is to derive a dependency graph from this data so that pipeline order constraints can be visualized. - -## How To Use - -When documenting a filter that requires array X, look up X in the table below and link to the producing filter from your Required Input Sources section using MyST link syntax: - -```markdown -- **X** -- produced by [Producer Human Name](ProducerFilter.md). -``` - -For arrays produced by EBSD readers (Cell Quaternions, Cell Phases, ensemble Crystal Structures), name the typical readers explicitly rather than picking one: - -```markdown -- **Cell Quaternions** -- typically read from EBSD data via [Read H5EBSD](ReadH5EbsdFilter.md), [Read CTF Data](ReadCtfDataFilter.md), or [Read ANG Data](ReadAngDataFilter.md); can also be produced from Euler angles by [Convert Orientations](ConvertOrientationsFilter.md). -``` - -## Conventions for Growing This Table - -- **One row per array, multiple producers comma-separated** in the Producer column when the array can come from several filters. -- **Plugin column** lists where the *producer* lives, not where the consuming filter lives. This drives the relative-path direction of MyST links. -- **Cross-plugin links** must use `../OtherPlugin/FilterFile.md`. Within-plugin links use `FilterFile.md` only. -- **Filter names match `humanName()`** in the .cpp, not the file name. The link target file is always `Filter.md`. -- **Notes column** (optional) for caveats: ensemble-level vs feature-level vs cell-level, optional inputs, mode dependencies (e.g., "only when *Find Coherence* is enabled"). - -## Lookup Table - -| Input Array | Producer (humanName) | Plugin | Notes | -|---|---|---|---| -| Cell Quaternions | Read H5EBSD / Read CTF Data / Read ANG Data / Convert Orientations | OrientationAnalysis | From EBSD readers, or computed from Euler angles | -| Cell Eulers | Same EBSD readers | OrientationAnalysis | Bunge convention Z-X-Z | -| Cell Phases | Same EBSD readers | OrientationAnalysis | Read alongside quaternions | -| Crystal Structures | Same EBSD readers / Create Ensemble Info | OrientationAnalysis | Ensemble-level array | -| Cell Feature Ids | Segment Features (Misorientation) | OrientationAnalysis | Misorientation-based grain segmentation | -| Cell Feature Ids | Segment Features (C-Axis Misalignment) | OrientationAnalysis | Hexagonal-only | -| Cell Feature Ids | Segment Features (Scalar) | SimplnxCore | Scalar-based segmentation | -| Cell Parent Ids | Merge Twins | OrientationAnalysis | Cell-level grouping into parent grains | -| Feature Parent Ids | Merge Twins | OrientationAnalysis | Feature-level mapping | -| Feature Phases | Compute Feature Phases | SimplnxCore | Feature-level (one phase per feature) | -| Feature Phases (Binary) | Compute Feature Phases (Binary) | SimplnxCore | Two-phase classification | -| Average Quaternions | Compute Average Orientations | OrientationAnalysis | Per-feature average orientation | -| Average Euler Angles | Compute Average Orientations | OrientationAnalysis | Bunge convention Z-X-Z | -| Average C-Axes | Compute Average C-Axis Orientations | OrientationAnalysis | Hexagonal-only | -| Feature Centroids | Compute Feature Centroids | SimplnxCore | Cell-of-mass per feature | -| Feature Equivalent Diameters | Compute Feature Sizes | SimplnxCore | Diameter of equal-volume sphere | -| Feature Num. Cells (NumElements) | Compute Feature Sizes | SimplnxCore | Cell count per feature | -| Feature Volumes | Compute Feature Sizes | SimplnxCore | Physical volume | -| Surface Features | Compute Surface Features | SimplnxCore | Boolean per feature | -| Surface Features | Compute Feature Neighbors | SimplnxCore | Optional output (enable *Store Surface Features Array*) | -| Biased Features | Compute Biased Features (Bounding Box) | SimplnxCore | Boolean per feature | -| Number of Neighbors | Compute Feature Neighbors | SimplnxCore | Count per feature | -| Contiguous Neighbor List | Compute Feature Neighbors | SimplnxCore | NeighborList of contiguous Feature Ids | -| Shared Surface Area List | Compute Feature Neighbors | SimplnxCore | NeighborList in cell-face-count units | -| Boundary Cells | Compute Feature Neighbors | SimplnxCore | Optional output (enable *Store Boundary Cells Array*) | -| Neighborhoods Neighbor List | Compute Feature Neighborhoods | SimplnxCore | Sphere-radius-based neighbor list | -| Mask | Multi-Threshold Objects | SimplnxCore | Boolean per cell | -| Boundary Euclidean Distances | Compute Euclidean Distance Map | SimplnxCore | Cell-level distance to nearest boundary | -| Triangle Geometry | Quick Surface Mesh | SimplnxCore | Surface mesh from Feature Ids | -| Face Labels | Quick Surface Mesh | SimplnxCore | Per-face Feature ID pair (2-component) | -| Face Normals | Compute Triangle Normals | SimplnxCore | 3-component, on triangle face data | -| Face Areas | Compute Triangle Areas | SimplnxCore | Single-component, on triangle face data | -| Node Types | Quick Surface Mesh | SimplnxCore | Per-vertex node classification | -| Cell Shifts Array (alignment) | Align Sections (Feature Centroid) | SimplnxCore | Output of any Align Sections * filter | -| Cell Shifts Array (alignment) | Align Sections (Misorientation) | OrientationAnalysis | | -| Cell Shifts Array (alignment) | Align Sections (Mutual Information) | OrientationAnalysis | | -| Confidence Index / Image Quality | Read H5EBSD / Read CTF Data / Read ANG Data | OrientationAnalysis | Per-cell scalar quality metric | - -## Future: Graph Representation - -The lookup table is naturally a directed dependency graph: each row is an edge from a producer filter to the input-array node. A separate downstream effort may convert this into a graph (DOT, Mermaid, or networked JSON) to support pipeline-order checking, "what would I need to run before X?" queries, and visual dependency maps. Keep that future use in mind when adding rows -- be precise about producer humanName so the data is machine-parseable. From af284ba5970de5900e5c01bedfc1904570279dcb Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Fri, 30 Oct 2026 11:29:41 -0400 Subject: [PATCH 12/13] TEST: Label the out-of-core test cases so ctest can select them Catch2 2.13.6 has no ADD_TAGS_AS_LABELS, so ctest never saw the [OOC] tag. Each discovering test executable now runs two discoveries with disjoint filters (~[OOC]~[.] and [OOC]~[.]); the second gives the tagged cases the ctest label "ooc". Every visible case is still registered once under the same name. The OocStoreContracts entries get the label too, so `ctest -L "^ooc$"` selects every out-of-core test. Signed-off-by: Joey Kleingers --- cmake/Plugin.cmake | 5 ++++- src/Plugins/OrientationAnalysis/test/CMakeLists.txt | 2 +- src/Plugins/SimplnxCore/test/CMakeLists.txt | 2 +- test/CMakeLists.txt | 4 +++- 4 files changed, 9 insertions(+), 4 deletions(-) diff --git a/cmake/Plugin.cmake b/cmake/Plugin.cmake index 5b044b4511..2c6e53df80 100644 --- a/cmake/Plugin.cmake +++ b/cmake/Plugin.cmake @@ -407,5 +407,8 @@ function(create_simplnx_plugin_unit_test) ${${ARGS_PLUGIN_NAME}_GENERATED_DIR} ) - catch_discover_tests(${UNIT_TEST_TARGET}) + # Two discoveries register every visible test case once. Cases tagged [OOC] (Catch2 tags ignore case) also get the + # ctest label "ooc", so `ctest -L "^ooc$"` selects them. + catch_discover_tests(${UNIT_TEST_TARGET} TEST_SPEC "~[OOC]~[.]") + catch_discover_tests(${UNIT_TEST_TARGET} TEST_SPEC "[OOC]~[.]" TEST_LIST ${UNIT_TEST_TARGET}_OOC_TESTS PROPERTIES LABELS "ooc") endfunction() diff --git a/src/Plugins/OrientationAnalysis/test/CMakeLists.txt b/src/Plugins/OrientationAnalysis/test/CMakeLists.txt index 28e1577566..88808841bf 100644 --- a/src/Plugins/OrientationAnalysis/test/CMakeLists.txt +++ b/src/Plugins/OrientationAnalysis/test/CMakeLists.txt @@ -85,7 +85,7 @@ create_simplnx_plugin_unit_test(PLUGIN_NAME ${PLUGIN_NAME} if(DREAM3DNX_USE_OOC) add_test(NAME OrientationAnalysisOocStoreContracts COMMAND ${PLUGIN_NAME}UnitTest "[.OocStoreContract]" --reporter compact) - set_tests_properties(OrientationAnalysisOocStoreContracts PROPERTIES RUN_SERIAL TRUE TIMEOUT 1500) + set_tests_properties(OrientationAnalysisOocStoreContracts PROPERTIES RUN_SERIAL TRUE TIMEOUT 1500 LABELS "ooc") endif() # Compile the cross-module dispatch probe into the plugin, not the test diff --git a/src/Plugins/SimplnxCore/test/CMakeLists.txt b/src/Plugins/SimplnxCore/test/CMakeLists.txt index 397fc7e159..0d9e73d035 100644 --- a/src/Plugins/SimplnxCore/test/CMakeLists.txt +++ b/src/Plugins/SimplnxCore/test/CMakeLists.txt @@ -182,7 +182,7 @@ create_simplnx_plugin_unit_test(PLUGIN_NAME ${PLUGIN_NAME} if(DREAM3DNX_USE_OOC) add_test(NAME SimplnxCoreOocStoreContracts COMMAND ${PLUGIN_NAME}UnitTest "[.OocStoreContract]" --reporter compact) - set_tests_properties(SimplnxCoreOocStoreContracts PROPERTIES RUN_SERIAL TRUE TIMEOUT 1500) + set_tests_properties(SimplnxCoreOocStoreContracts PROPERTIES RUN_SERIAL TRUE TIMEOUT 1500 LABELS "ooc") endif() # ------------------------------------------------------------------------------ diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 3fc67be1c3..5b556ef702 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -140,7 +140,9 @@ target_compile_options(simplnx_test target_include_directories(simplnx_test PRIVATE ${SIMPLNX_GENERATED_DIR}) -catch_discover_tests(simplnx_test) +# Two discoveries register every visible test case once; cases tagged [OOC] also get the ctest label "ooc". +catch_discover_tests(simplnx_test TEST_SPEC "~[OOC]~[.]") +catch_discover_tests(simplnx_test TEST_SPEC "[OOC]~[.]" TEST_LIST simplnx_test_OOC_TESTS PROPERTIES LABELS "ooc") From 2b473df2b6e055b0bfc54875892f8bc689e777da Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Fri, 30 Oct 2026 11:29:41 -0400 Subject: [PATCH 13/13] DOC: Make the AGENTS.md instructions followable as written Answers the review of this pull request: - select the out-of-core tests with `ctest -L "^ooc$" -LE "benchmark"`, including the store-contract entries; - say which build directory to use, that a normal build downloads the test data, and that SIMPLNX_TEST_ALGORITHM_PATH is set at configure time; - state when dual-build validation is required, that matching task skills are required, and what an agent without access to the private repositories does; - name the agent's personal instruction file generically, make the knowledge base win for defect-derived rules, give concrete triggers for recording a rule, and keep one public-repository rule; - say which tests to run before handing off and what to do when vcpkg fails. The knowledge base introduction now points to the same triggers. Signed-off-by: Joey Kleingers --- AGENTS.md | 93 +++++++++++++++++++++++------------ docs/Coding_Knowledge_Base.md | 11 +++-- 2 files changed, 68 insertions(+), 36 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 595c1fc070..58f1bc943c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,42 +1,52 @@ # AGENTS.md -Guidance for every coding agent working in this repository. Coding rules live in the `bluequartz-skills` skills -below; this file holds what a skill cannot know about this checkout. +Guidance for every coding agent working in this repository. The coding rules are in the `bluequartz-skills` skills +below and in the two documents they point to, the code style guide and the coding knowledge base. This file holds what +they cannot know about this checkout. ## Required skills -Every agent that writes, changes or reviews C++ here (the main session and each subagent) first loads these -`bluequartz-skills` skills and applies them. A subagent does not inherit them: it loads them itself, and every -dispatch prompt says so. +Every agent that writes, changes or reviews code or project files here (C++, CMake, filter documentation, pipeline +files), the main session and each subagent, first loads these `bluequartz-skills` skills and applies them. A subagent +does not inherit them: it loads them itself, and every dispatch prompt says so. -- `bluequartz-skills:cpp-best-practices`: all C++. -- `bluequartz-skills:simplnx-best-practices`: simplnx and simplnx plugin code. +- `bluequartz-skills:cpp-best-practices` +- `bluequartz-skills:simplnx-best-practices` -Task skills own the details: `filter-implementation` for filters and algorithms, `implement-filter-tests` and -`exemplar-testing` for tests, `publish-exemplars` for test-data archives, `error-messages` for error and warning -messages, `thread-safety` for parallel code, `ooc-architecture` for out-of-core storage, `pipeline-files` for -`.d3dpipeline` files, and `github-workflow` for commits and pull requests. +Also load every task skill whose description matches the work that the change contains, for example +`filter-implementation` for filters and algorithms, `implement-filter-tests` and `exemplar-testing` for tests, +`publish-exemplars` for test-data archives, `error-messages` for error and warning messages, `thread-safety` for +parallel code, `ooc-architecture` for out-of-core storage, `pipeline-files` for `.d3dpipeline` files, and +`github-workflow` for commits and pull requests. The plugin's skill list gives each skill's description. + +The skills, DREAM3D-NX and SimplnxOoc are private BlueQuartz repositories. An agent without access to them follows +[the code style guide](docs/Code_Style_Guide.md) and [the coding knowledge base](docs/Coding_Knowledge_Base.md), which +hold the rules of this project, and runs the in-core build and tests. The pull request lists what it could not do: the +skills it could not load and the out-of-core validation it could not run. A maintainer completes that validation. ## Where new rules go -When work teaches something a later agent or developer needs, record it in the same change, in exactly one place. -Search all of them first and update an existing rule instead of adding a second one. +Record a rule when a reviewer or the user had to correct something that no document or skill covers, or when a defect +came from it. Record it in exactly one place. Search all of them first and +update an existing rule instead of adding a second one. - `docs/Coding_Knowledge_Base.md`: a lesson from a real defect in simplnx or its plugins. State the rule, the evidence (what broke and the pull request or issue that fixed it) and how to recognize it next time, using the entry format - in that file. + in that file. When a rule comes from a defect, it goes here even if it is also a convention. - `docs/Code_Style_Guide.md`: a convention of this project that well-written C++ could still break (names, layout, - project APIs, filter and test structure). No defect history is needed. -- This repository is public: an entry may name a private repository but contains no code from it. + project APIs, filter and test structure), with no defect behind it. - A `bluequartz-skills` skill: a general C++ rule that applies to every BlueQuartz repository (`cpp-best-practices`), or a procedure (how to implement, test, review or publish). Skills point to the two - documents above; they do not copy their rules. Skill changes go to the `bluequartz-skills` plugin repository, not - to this one. + documents above; they do not copy their rules. Skill changes go to the `bluequartz-skills` plugin repository, so + they cannot be part of a change here: open a pull request or an issue there and link it from this pull request. + Without access to that repository, describe the proposed skill change in the pull request description. - This file: only facts about this checkout that no skill can know, such as layout and build and test commands. -- A rule that is true only on your own machine (tool paths, presets, build directories, personal habits): your personal - `~/.codex/AGENTS.md`, in a section for this project headed by its checkout path. Never in this file or in an +- A rule that is true only on your own machine (tool paths, presets, build directories, personal habits): your agent's + personal instruction file, in a section for this project headed by its checkout path. Never in this file or in an `AGENTS.override.md` here: the override replaces this file instead of adding to it. +The documents in this repository follow the public-repository rule under "Repository". + ## Project simplnx is the C++20 data-processing library behind DREAM3D-NX. It has no UI dependencies. @@ -49,35 +59,56 @@ simplnx is the C++20 data-processing library behind DREAM3D-NX. It has no UI dep - Out-of-tree filter plugins, such as SimplnxReview, live in their own repositories and join the build through `SIMPLNX_PLUGIN_SEARCH_DIRS` and `SIMPLNX_EXTRA_PLUGINS`. These variables do not enable SimplnxOoc. When you need out-of-core storage, follow the build instructions of the SimplnxOoc repository. -- Leave `scripts/`, `conda/` and build trees alone unless the task needs them. +- Do not edit files in `scripts/` or `conda/` unless the user asks for it. - DREAM3D-NX builds this checkout as a subdirectory, so a change here can break the application build. ## Build and test -- Configure with a preset from `CMakePresets.json` (CI) or your `CMakeUserPresets.json`. Use the build directory whose - cache points at this source tree; a DREAM3D-NX build that includes this checkout counts. -- With `SIMPLNX_DOWNLOAD_TEST_FILES=ON`, build the `Fetch_Remote_Data_Files` target to download test data. The target - does not exist when that option is OFF. +- Use a build directory whose `CMakeCache.txt` points at this checkout: `CMAKE_HOME_DIRECTORY` in a simplnx build, + `simplnx_SOURCE_DIR` in a DREAM3D-NX build. In-core work can use either; when both exist, use the simplnx build. + Out-of-core tests need a DREAM3D-NX build with `DREAM3DNX_USE_OOC=ON`; there is no simplnx-only out-of-core build. + When no in-core build exists, configure one with a preset that sets `SIMPLNX_BUILD_TESTS=ON` (the `ci-` + presets in `CMakePresets.json` do, and build in `build/`). For an out-of-core build, follow the build instructions + of the DREAM3D-NX and SimplnxOoc repositories. Configure, build and test only in a build directory that you created + for this task or that the user named; never delete or clean one. +- A normal build downloads the test data. With `SIMPLNX_DOWNLOAD_TEST_FILES=ON` (the default), the + `Fetch_Remote_Data_Files` target is part of the default build and every plugin unit-test target depends on it; you do + not build it separately. When the option is OFF, the target does not exist and tests that need downloaded data + cannot run. - With a multi-config generator (Visual Studio), pass `--config ` to `cmake --build` and `-C ` to ctest. - After you add or remove an entry in a CMake source list, reconfigure the build directory. This can rerun vcpkg, so run - it in the foreground with a long timeout. Treat a vcpkg dependency failure as a problem of the environment until you - show otherwise. + it in the foreground with a long timeout. When vcpkg fails to build a dependency, stop and report the package and the + path of its log; do not change `vcpkg.json`, the triplet or the toolchain to work around it unless the user asks. - Never pipe a build or test command into `tail` or another command that hides its exit status. - Run tests only through `ctest`, never the test executable: ctest extracts and removes the test data. Never run tests in parallel; they share data files. Replace `` with the verified build directory and `` with the test-name regular expression: `ctest --test-dir "" -R "" -j 1 --output-on-failure --timeout 1500 --no-tests=error`. +- Before you hand off a change, build it and run, in a Release build, the tests of every filter, algorithm or class that + it changes, and the tests of the code that calls a changed base class, store or shared utility. Name the tests that + ran in the pull request. - `SIMPLNX_TEST_ALGORITHM_PATH` picks which algorithm paths the tests run (0 both, 1 out-of-core only, 2 in-core only). - Validate in-core and out-of-core together with `bluequartz-skills:dual-build-protocol`. When out-of-core behavior - changes, run the whole `[ooc]` set, not one narrow tag. + It is a CMake cache variable compiled into the test targets: set it when you configure + (`-DSIMPLNX_TEST_ALGORITHM_PATH=1`) and rebuild the tests. Setting it in the environment when you run ctest does + nothing. +- Validate in-core and out-of-core together with `bluequartz-skills:dual-build-protocol` for every change to code that + runs when a filter executes or a store is read or written: a new or changed filter or algorithm (with or without + `AlgorithmDispatch`), a utility that reads or writes arrays, or a store or I/O class. A change to documentation, + comments or build scripts only does not need it. +- When out-of-core behavior changes, run every out-of-core test except the benchmarks, in the out-of-core build, not + one narrow tag: + `ctest --test-dir "" -L "^ooc$" -LE "benchmark" -j 1 --output-on-failure --timeout 1500 --no-tests=error`. + The `ooc` label covers every test case tagged `[OOC]` (Catch2 tags ignore case), the `OocStoreContracts` + entries, which run the hidden `[.OocStoreContract]` cases, and the out-of-core tests that the SimplnxOoc build adds. + Other hidden cases are not registered with ctest. - Format only the files you changed, with clang-format 19.1.1 from inside the repository so `.clang-format` applies (`bluequartz-skills:clang-format` finds the binary). A whole-tree format belongs in its own commit. ## Repository - Pull requests target `develop` on `BlueQuartzSoftware/simplnx`. -- This repository is public. Issues, pull requests, comments and reviews may name or link private repositories - (SimplnxOoc, DREAM3D-NX) but contain no code from them. +- This repository is public. Its files, issues, pull requests, comments and reviews may name or link private + repositories (SimplnxOoc, DREAM3D-NX) but contain no code from them. - A published test-data archive is immutable once merged code references it; publish changed data under a new versioned name. diff --git a/docs/Coding_Knowledge_Base.md b/docs/Coding_Knowledge_Base.md index 2aee43ecf2..15d069e637 100644 --- a/docs/Coding_Knowledge_Base.md +++ b/docs/Coding_Knowledge_Base.md @@ -4,11 +4,12 @@ This file collects lessons learned from real defects in simplnx and its plugins. made, why it happened, how it was found, and the rule that prevents it. The audience is developers and the coding agents that work in this repository. Read the rules before you write or review code in the area they cover. -Add a new entry when a defect taught something that the code itself does not make obvious. Keep the entries short. State -the rule first, then the evidence. Cite pull requests and issues by number, but do not say whether they are open, merged -or closed: that changes, and the entry would go stale. Refer to code by file and function or test name, not by line -number, because line numbers change with every edit; a line number belongs only in a link that is pinned to a commit. A -project convention that does not come from a defect belongs in [the code style guide](Code_Style_Guide.md) instead. +Add or update an entry when a defect taught a rule, as "Where new rules go" in `AGENTS.md` describes. Keep the entries +short. State the rule first, then the evidence. Cite pull requests and issues by number, but do not say whether they are +open, merged or closed: that changes, and the entry would go stale. Refer to code by file and function or test name, not +by line number, because line numbers change with every edit; a line number belongs only in a link that is pinned to a +commit. A project convention that does not come from a defect belongs in [the code style guide](Code_Style_Guide.md) +instead. ## Index