diff --git a/.cursor/rules/git-commit.mdc b/.cursor/rules/git-commit.mdc new file mode 100644 index 0000000..16ca1f8 --- /dev/null +++ b/.cursor/rules/git-commit.mdc @@ -0,0 +1,11 @@ +--- +description: Git commit message preferences for this project +alwaysApply: true +--- + +# Git Commits + +When creating git commits in this project: + +- Do **not** add `Co-authored-by: Cursor` (or any Cursor co-author trailer) to commit messages. +- Keep commit messages focused on the change itself, following the repository's existing style. diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b692da..44e190a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,81 @@ removed no sooner than the next major (see `docs/API_STABILITY.md`). ### Added +- **Canonical 2D → AI → 3D roadmap and image IO (Epic 83)**: `docs/ROADMAP.md` + reserves Epics 83–90; new `spatialrust-image-io` provides bounded path, + reader/writer, and memory PNG/JPEG/PNM codecs, independently gated TIFF and + OpenEXR, typed pixels, Exif orientation handling, Python/NumPy bindings, + property tests, and 640p/1080p/4K decode benchmarks. + +- **Shared CPU filters (Epic 84A–84B)**: `spatialrust-vision` now exposes a + common `BorderMode`, validated 1D/2D kernels, OpenCV-style filter2D + correlation, explicit convolution, f32-output and separable filters, and + normalized box/Gaussian blur, median and bilateral filters, signed + Sobel/Scharr/Laplacian derivatives, and Gaussian pyramids. The feature + includes strided-view property coverage, Python bindings/stubs, OpenCV + comparison, and 640p/1080p/4K benchmarks. + +- **CPU morphology (Epic 84C)**: validated rectangular, cross, elliptical, + diamond, and custom structuring elements; explicit-anchor erode/dilate; + open/close/gradient/top-hat/black-hat operations; additive feature and meta + feature; u8/u16/f32 and strided-view tests; Python bindings; exact OpenCV + comparisons; and 640p/1080p/4K benchmarks. + +- **CPU image analysis (Epic 84D)**: fixed and adaptive thresholds, u8/u16 + Otsu selection, masked configurable histograms, exact u8 equalization, + contrast-limited adaptive equalization, and checked summed-area tables; + additive Rust/meta features, Python bindings/stubs, strided properties, + OpenCV comparisons, and representative-resolution benchmarks. + +- **Canny edge detection (Epic 84E)**: configurable 3/5/7 Sobel apertures, + L1/L2 gradient magnitude, directional non-maximum suppression, 8-neighbor + hysteresis, and inspectable intermediate stages; additive Rust/meta features, + strided and property tests, Python binding/stub, 640p/1080p/4K benchmark, and + exact OpenCV comparison across all six aperture/magnitude combinations. + +- **Tensor foundation (Epic 85A)**: new dependency-light `spatialrust-tensor` + crate with byte-addressable dtype, arbitrary-rank shape, signed element + strides, checked byte offsets/spans, explicit device identity, safe borrowed + CPU views, and named owned copies. Non-host device memory cannot be exposed as + a Rust byte slice, and the meta-crate integration is opt-in through `tensor`. + +- **Image/spatial tensor bridges (Epic 85B)**: packed interleaved images expose + zero-copy HWC views, packed planar images expose zero-copy CHW views, and + Schema-SoA `f32` point fields expose zero-copy one-dimensional views. Explicit + `pack_*` operations handle padded/ROI images, with feature-alone tests and + 640p/1080p/4K packing benchmarks. + +- **DLPack and Python tensor interoperability (Epic 85C–85D)**: audited + `DLManagedTensorVersioned` major-version 1 CPU import/export, explicit deleter + transfer, read-only/copy flags, signed strides and byte offsets, malformed ABI + rejection, and zero-copy Python `__dlpack__`/`__dlpack_device__`. NumPy and + PyTorch round trips preserve allocations and producer lifetimes; device or + host copy requests remain explicit. + +- **Inference contracts and ONNX Runtime CPU (Epic 86)**: new optional + `spatialrust-ai` crate with named dynamic model metadata, stable backend and + session traits, explicit input/output copy permissions, CPU ONNX Runtime, + separately gated CUDA/TensorRT/DirectML providers, typed zero-copy I/O + Binding, caller-preallocated u8/u16/f32 outputs, runtime-allocation retention, + output-to-input chaining, Python bindings/stubs, reference-runtime comparison, + and 640p/1080p/4K Criterion coverage. Multi-byte raw storage is rejected at + zero-copy boundaries instead of being cast from an under-aligned byte buffer. + +- **Feature2D and ORB matching (Epic 87)**: checked keypoint, binary/float + descriptor, feature-set, and match contracts; Harris and Shi–Tomasi corners; + OpenCV-exact FAST-9/16 detection and scores; deterministic multi-scale ORB + with 256-bit rotated BRIEF; Hamming/L2 brute-force matching with ratio, + cross-check, and distance filters; Python/NumPy bindings and stubs; property + tests, OpenCV comparison, and 640p/1080p/4K Criterion baselines. + +- **Camera geometry, motion, and stereo (Epic 88)**: checked correspondence and + projective contracts; normalized DLT and deterministic RANSAC for homography, + fundamental, and essential matrices; triangulation and essential pose + disambiguation; EPnP-class PnP with iterative refine and RANSAC; sparse + pyramidal Lucas–Kanade tracking; stereo rig, rectify remap grids, SAD block + matching, and disparity-to-depth/XYZ reproject; Python bindings; OpenCV + comparison with documented tolerances; property tests; and Criterion coverage. + - **AI-ready image and vision foundation (Epics 75–79)**: mutable ROI views, planar/interleaved layouts and color metadata in `spatialrust-image`; new feature-gated `spatialrust-vision` preprocessing, warp, detection, mask/RLE, diff --git a/Cargo.toml b/Cargo.toml index f58f0bd..f384303 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -15,6 +15,9 @@ members = [ "crates/spatialrust-transform", "crates/spatialrust-voxelize", "crates/spatialrust-image", + "crates/spatialrust-image-io", + "crates/spatialrust-tensor", + "crates/spatialrust-ai", "crates/spatialrust-camera", "crates/spatialrust-vision", ] @@ -47,6 +50,9 @@ spatialrust-metrics = { path = "crates/spatialrust-metrics", version = "1.0.0", spatialrust-transform = { path = "crates/spatialrust-transform", version = "1.0.0", default-features = false } spatialrust-voxelize = { path = "crates/spatialrust-voxelize", version = "1.0.0", default-features = false } spatialrust-image = { path = "crates/spatialrust-image", version = "1.0.0" } +spatialrust-image-io = { path = "crates/spatialrust-image-io", version = "1.0.0", default-features = false } +spatialrust-tensor = { path = "crates/spatialrust-tensor", version = "1.0.0", default-features = false } +spatialrust-ai = { path = "crates/spatialrust-ai", version = "1.0.0", default-features = false } spatialrust-camera = { path = "crates/spatialrust-camera", version = "1.0.0" } spatialrust-vision = { path = "crates/spatialrust-vision", version = "1.0.0", default-features = false } @@ -63,6 +69,10 @@ thiserror = "2" wgpu = "24" criterion = { version = "0.5", features = ["html_reports"] } proptest = "1" +image = { version = "0.24.9", default-features = false } +kamadak-exif = "0.6.1" +exr = { version = "=1.72.0", default-features = false } +tempfile = "3" [profile.release] lto = "thin" diff --git a/README.md b/README.md index f7e9a02..d8a2a54 100644 --- a/README.md +++ b/README.md @@ -157,8 +157,11 @@ One dataflow, focused crates — each pipeline stage maps to the crate that impl | `spatialrust-core` | Point schema, metadata, execution traits | | `spatialrust-math` | Vec/Mat/Pose math primitives | | `spatialrust-image` | Typed image buffers and zero-copy strided views | +| `spatialrust-image-io` | Bounded PNG/JPEG/PNM codecs; opt-in TIFF/OpenEXR | +| `spatialrust-tensor` | Runtime-independent dtype/shape/stride/device ownership and DLPack | +| `spatialrust-ai` | Explicit-copy inference contracts and opt-in ONNX Runtime providers | | `spatialrust-camera` | Pinhole/Brown–Conrady camera models and RGB-D conversion | -| `spatialrust-vision` | Resize/preprocess, warps, detection postprocess, masks, and dense spatial maps | +| `spatialrust-vision` | CPU filters, Feature2D/ORB matching, resize/preprocess, warps, detection postprocess, masks, and dense spatial maps | | `spatialrust-io` | Point cloud readers/writers (PCD, PLY, LAS, COPC) | | `spatialrust-search` | KD-tree search, k-NN / radius graphs | | `spatialrust-filtering` | Voxel / FPS downsample, outlier removal, crop, MLS | @@ -220,6 +223,26 @@ The reproducible algorithm comparison is in `bench/opencv_vision_comparison/`; the complete synthetic demo is `crates/spatialrust-py/examples/vision_ai_pipeline.py`. +The same feature includes Harris, Shi–Tomasi, exact FAST-9/16, multi-scale ORB, +and checked Hamming/L2 descriptor matching. Python exposes `orb_features` and +NumPy matcher functions; OpenCV is used only by the numerical comparison suite. + +An ONNX Runtime wheel is opt-in (`maturin develop --features onnxruntime`). Its +Python API uses named CPU I/O Binding by default; `copy=True` is the explicit +fallback for inputs that must be repacked: + +```python +session = sr.OnnxRuntimeSession("model.onnx", deterministic=True) +input_tensor = sr.tensor_copy_from_numpy(chw) +outputs = session.run({"images": input_tensor}) +scores = np.from_dlpack(outputs["scores"]) +``` + +The Rust features are `ai`, `ai-onnxruntime`, and separate +`ai-onnxruntime-{cuda,tensorrt,directml}` provider gates. The optional ONNX +Runtime adapter currently has a feature-specific Rust 1.88 MSRV; it does not +raise the default workspace MSRV. +

Top-down view of clusters segmented from the public PCL table_scene_lms400 point cloud via a single Python run_pipeline() call

diff --git a/bench/opencv_vision_comparison/README.md b/bench/opencv_vision_comparison/README.md index 8682aa4..1a47998 100644 --- a/bench/opencv_vision_comparison/README.md +++ b/bench/opencv_vision_comparison/README.md @@ -1,8 +1,14 @@ # OpenCV vision comparison This deterministic harness compares SpatialRust's Python-visible CPU vision -primitives with OpenCV: four resize filters, RGB-to-gray/HSV conversion, -bilinear remap, NMS, and connected-component areas. +primitives with OpenCV: linear, median, and bilateral filters; Sobel, Scharr, +Laplacian, Gaussian pyramids, morphology, thresholds, histograms, CLAHE, integral images, +and Canny across 3/5/7 Sobel apertures and L1/L2 gradients; +Harris, Shi–Tomasi, and FAST-9/16 keypoint coordinates/order (plus exact FAST scores); +Hamming/L2 brute-force nearest matches; ORB keypoint repeatability and descriptor layout; +homography transfer residuals, PnP translation vs OpenCV, and StereoBM center disparity +on a synthetic textured pair; four resize filters; RGB-to-gray/HSV conversion; +bilinear remap; NMS; and connected-component areas. From the repository root, after installing the editable Python extension: diff --git a/bench/opencv_vision_comparison/run.py b/bench/opencv_vision_comparison/run.py index 9c588ae..81b0b95 100644 --- a/bench/opencv_vision_comparison/run.py +++ b/bench/opencv_vision_comparison/run.py @@ -24,6 +24,287 @@ def main() -> None: size = (61, 43) results: dict[str, object] = {"opencv_version": cv2.__version__} + kernel = np.array([[0.0, -0.25, 0.0], [-0.25, 2.0, -0.25], [0.0, -0.25, 0.0]]) + filtered = sr.filter2d_image(image, kernel) + filtered_cv = cv2.filter2D(image, -1, kernel, borderType=cv2.BORDER_REFLECT_101) + filter_error = max_abs(filtered, filtered_cv) + results["filter2d_max_u8_error"] = filter_error + if filter_error > 1: + raise AssertionError(f"filter2D error {filter_error} > 1") + + gaussian = sr.gaussian_blur_image(image, 5, 3, 1.2, 0.8) + gaussian_cv = cv2.GaussianBlur( + image, (5, 3), 1.2, sigmaY=0.8, borderType=cv2.BORDER_REFLECT_101 + ) + gaussian_error = max_abs(gaussian, gaussian_cv) + results["gaussian_blur_max_u8_error"] = gaussian_error + if gaussian_error > 1: + raise AssertionError(f"Gaussian blur error {gaussian_error} > 1") + + median = sr.median_blur_image(image, 5) + median_cv = cv2.medianBlur(image, 5) + median_error = max_abs(median, median_cv) + results["median_blur_max_u8_error"] = median_error + if median_error != 0: + raise AssertionError(f"median blur error {median_error} != 0") + + bilateral = sr.bilateral_filter_image(image, 5, 40.0, 3.0) + bilateral_cv = cv2.bilateralFilter( + image, 5, 40.0, 3.0, borderType=cv2.BORDER_REFLECT_101 + ) + bilateral_error = max_abs(bilateral, bilateral_cv) + results["bilateral_filter_max_u8_error"] = bilateral_error + if bilateral_error > 2: + raise AssertionError(f"bilateral filter error {bilateral_error} > 2") + + gray_derivative = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY) + derivative_cases = { + "sobel_x": ( + sr.sobel_image(gray_derivative, 1, 0, 5), + cv2.Sobel(gray_derivative, cv2.CV_32F, 1, 0, ksize=5), + ), + "scharr_y": ( + sr.scharr_image(gray_derivative, 0, 1), + cv2.Scharr(gray_derivative, cv2.CV_32F, 0, 1), + ), + "laplacian": ( + sr.laplacian_image(gray_derivative, 3), + cv2.Laplacian(gray_derivative, cv2.CV_32F, ksize=3), + ), + } + for name, (actual, expected) in derivative_cases.items(): + error = float(np.max(np.abs(actual - expected))) + results[f"{name}_max_f32_error"] = error + if error > 1e-4: + raise AssertionError(f"{name} error {error} > 1e-4") + + pyramid = sr.pyr_down_image(image) + pyramid_cv = cv2.pyrDown(image) + pyramid_error = max_abs(pyramid, pyramid_cv) + results["pyr_down_max_u8_error"] = pyramid_error + if pyramid_error > 1: + raise AssertionError(f"pyrDown error {pyramid_error} > 1") + pyramid_up = sr.pyr_up_image(pyramid) + pyramid_up_cv = cv2.pyrUp(pyramid_cv) + pyramid_up_error = max_abs(pyramid_up, pyramid_up_cv) + results["pyr_up_max_u8_error"] = pyramid_up_error + if pyramid_up_error > 1: + raise AssertionError(f"pyrUp error {pyramid_up_error} > 1") + + morphology_source = gray_derivative + morphology_cases = { + "erode": cv2.MORPH_ERODE, + "dilate": cv2.MORPH_DILATE, + "open": cv2.MORPH_OPEN, + "close": cv2.MORPH_CLOSE, + "gradient": cv2.MORPH_GRADIENT, + "tophat": cv2.MORPH_TOPHAT, + "blackhat": cv2.MORPH_BLACKHAT, + } + shape_cases = { + "rect": cv2.MORPH_RECT, + "cross": cv2.MORPH_CROSS, + "ellipse": cv2.MORPH_ELLIPSE, + } + for shape_name, shape_code in shape_cases.items(): + element = cv2.getStructuringElement(shape_code, (5, 3)) + for operation_name, operation_code in morphology_cases.items(): + actual = sr.morphology_image( + morphology_source, operation_name, 5, 3, shape_name, 2 + ) + expected = cv2.morphologyEx( + morphology_source, + operation_code, + element, + iterations=2, + borderType=cv2.BORDER_REPLICATE, + ) + error = max_abs(actual, expected) + results[f"morphology_{shape_name}_{operation_name}_max_u8_error"] = error + if error != 0: + raise AssertionError( + f"{shape_name}/{operation_name} morphology error {error} != 0" + ) + + threshold_actual = sr.threshold_image(gray_derivative, 117.0) + _, threshold_expected = cv2.threshold(gray_derivative, 117.0, 255, cv2.THRESH_BINARY) + results["threshold_max_u8_error"] = max_abs(threshold_actual, threshold_expected) + if results["threshold_max_u8_error"] != 0: + raise AssertionError("fixed threshold mismatch") + + otsu_value, otsu_actual = sr.otsu_threshold_image(gray_derivative) + otsu_cv, otsu_expected = cv2.threshold( + gray_derivative, 0, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU + ) + results["otsu_threshold"] = otsu_value + results["otsu_max_u8_error"] = max_abs(otsu_actual, otsu_expected) + if otsu_value != int(otsu_cv) or results["otsu_max_u8_error"] != 0: + raise AssertionError(f"Otsu mismatch: {otsu_value} != {otsu_cv}") + + for method_name, method_code in { + "mean": cv2.ADAPTIVE_THRESH_MEAN_C, + "gaussian": cv2.ADAPTIVE_THRESH_GAUSSIAN_C, + }.items(): + actual = sr.adaptive_threshold_image(gray_derivative, 7, 3.0, method_name) + expected = cv2.adaptiveThreshold( + gray_derivative, 255, method_code, cv2.THRESH_BINARY, 7, 3.0 + ) + error = max_abs(actual, expected) + results[f"adaptive_{method_name}_max_u8_error"] = error + if error != 0: + raise AssertionError(f"adaptive {method_name} mismatch: {error}") + + histogram_actual = sr.histogram_image(gray_derivative) + histogram_expected = cv2.calcHist([gray_derivative], [0], None, [256], [0, 256]).reshape(-1) + histogram_error = int(np.max(np.abs(histogram_actual.astype(np.int64) - histogram_expected.astype(np.int64)))) + results["histogram_max_count_error"] = histogram_error + if histogram_error != 0: + raise AssertionError(f"histogram mismatch: {histogram_error}") + + equalized = sr.equalize_histogram_image(gray_derivative) + equalized_cv = cv2.equalizeHist(gray_derivative) + results["equalize_hist_max_u8_error"] = max_abs(equalized, equalized_cv) + if results["equalize_hist_max_u8_error"] != 0: + raise AssertionError("histogram equalization mismatch") + + clahe_actual = sr.clahe_image(gray_derivative, 2.0, 8, 8) + clahe_expected = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(gray_derivative) + clahe_error = max_abs(clahe_actual, clahe_expected) + results["clahe_max_u8_error"] = clahe_error + if clahe_error > 1: + raise AssertionError(f"CLAHE mismatch: {clahe_error}") + + integral_actual = sr.integral_image_u8(gray_derivative) + integral_expected = cv2.integral(gray_derivative, sdepth=cv2.CV_64F) + integral_error = float(np.max(np.abs(integral_actual - integral_expected))) + results["integral_max_f64_error"] = integral_error + if integral_error != 0.0: + raise AssertionError(f"integral image mismatch: {integral_error}") + + for aperture_size in (3, 5, 7): + for l2_gradient in (False, True): + actual = sr.canny_image( + gray_derivative, 50.0, 100.0, aperture_size, l2_gradient + ) + expected = cv2.Canny( + gray_derivative, + 50.0, + 100.0, + apertureSize=aperture_size, + L2gradient=l2_gradient, + ) + mismatch = int(np.count_nonzero(actual != expected)) + name = f"canny_aperture_{aperture_size}_{'l2' if l2_gradient else 'l1'}" + results[f"{name}_mismatch_pixels"] = mismatch + if mismatch != 0: + raise AssertionError(f"{name} mismatch: {mismatch} pixels") + + for nonmax_suppression in (False, True): + actual = sr.fast_keypoints(gray_derivative, 20, nonmax_suppression) + detector = cv2.FastFeatureDetector_create( + 20, nonmax_suppression, cv2.FAST_FEATURE_DETECTOR_TYPE_9_16 + ) + expected = detector.detect(gray_derivative, None) + actual_rows = [ + (round(point.x), round(point.y), round(point.response)) for point in actual + ] + expected_rows = [ + (round(point.pt[0]), round(point.pt[1]), round(point.response)) + for point in expected + ] + name = f"fast_9_16_{'nms' if nonmax_suppression else 'raw'}" + results[f"{name}_keypoints"] = len(actual_rows) + if actual_rows != expected_rows: + raise AssertionError(f"{name} keypoints or scores differ from OpenCV") + + for use_harris in (False, True): + if use_harris: + actual = sr.harris_keypoints(gray_derivative, 100, 0.01, 1.0, 3, 3, 0.04) + name = "harris" + else: + actual = sr.shi_tomasi_keypoints(gray_derivative, 100, 0.01, 1.0, 3, 3) + name = "shi_tomasi" + expected = cv2.goodFeaturesToTrack( + gray_derivative, + maxCorners=100, + qualityLevel=0.01, + minDistance=1.0, + mask=None, + blockSize=3, + useHarrisDetector=use_harris, + k=0.04, + ) + actual_points = [(round(point.x), round(point.y)) for point in actual] + expected_points = ( + [] + if expected is None + else [(round(point[0][0]), round(point[0][1])) for point in expected] + ) + results[f"{name}_keypoints"] = len(actual_points) + if actual_points != expected_points: + raise AssertionError(f"{name} ordering or coordinates differ from OpenCV") + + binary_query = rng.integers(0, 256, size=(23, 32), dtype=np.uint8) + binary_train = rng.integers(0, 256, size=(31, 32), dtype=np.uint8) + actual_binary = sr.match_binary_descriptors(binary_query, binary_train) + expected_binary = cv2.BFMatcher(cv2.NORM_HAMMING).match(binary_query, binary_train) + actual_binary_rows = [(query, train, distance) for query, train, distance in actual_binary] + expected_binary_rows = [ + (match.queryIdx, match.trainIdx, match.distance) for match in expected_binary + ] + results["hamming_matches"] = len(actual_binary_rows) + if actual_binary_rows != expected_binary_rows: + raise AssertionError("Hamming nearest matches differ from OpenCV BFMatcher") + + float_query = rng.normal(size=(19, 17)).astype(np.float32) + float_train = rng.normal(size=(29, 17)).astype(np.float32) + actual_float = sr.match_float_descriptors(float_query, float_train) + expected_float = cv2.BFMatcher(cv2.NORM_L2).match(float_query, float_train) + float_index_mismatches = sum( + (query, train) != (expected.queryIdx, expected.trainIdx) + for (query, train, _), expected in zip(actual_float, expected_float) + ) + float_distance_error = max( + abs(distance - expected.distance) + for (_, _, distance), expected in zip(actual_float, expected_float) + ) + results["l2_match_index_mismatches"] = float_index_mismatches + results["l2_match_max_distance_error"] = float_distance_error + if float_index_mismatches != 0 or float_distance_error > 1e-5: + raise AssertionError( + f"L2 BFMatcher mismatch: indices={float_index_mismatches}, distance={float_distance_error}" + ) + + orb_image = cv2.resize(gray_derivative, (320, 240), interpolation=cv2.INTER_CUBIC) + actual_orb, actual_descriptors = sr.orb_features( + orb_image, max_features=200, edge_threshold=16 + ) + cv_orb = cv2.ORB_create(nfeatures=200, edgeThreshold=16) + expected_orb, expected_descriptors = cv_orb.detectAndCompute(orb_image, None) + actual_coordinates = np.array([(point.x, point.y) for point in actual_orb], dtype=np.float32) + expected_coordinates = np.array([point.pt for point in expected_orb], dtype=np.float32) + repeatable = 0 + if len(actual_coordinates) and len(expected_coordinates): + nearest_distances = np.sqrt( + np.min( + np.sum( + (actual_coordinates[:, None, :] - expected_coordinates[None, :, :]) ** 2, + axis=2, + ), + axis=1, + ) + ) + repeatable = int(np.count_nonzero(nearest_distances <= 2.0)) + repeatability = repeatable / max(1, min(len(actual_orb), len(expected_orb))) + results["orb_spatialrust_keypoints"] = len(actual_orb) + results["orb_opencv_keypoints"] = len(expected_orb) + results["orb_coordinate_repeatability_2px"] = repeatability + results["orb_descriptor_width"] = actual_descriptors.shape[1] + if actual_descriptors.shape != (len(actual_orb), 32): + raise AssertionError("SpatialRust ORB descriptor layout is not N x 32") + if expected_descriptors is None or repeatability < 0.30: + raise AssertionError(f"ORB repeatability against OpenCV is too low: {repeatability}") + resize_cases = { "nearest": getattr(cv2, "INTER_NEAREST_EXACT", cv2.INTER_NEAREST), "bilinear": cv2.INTER_LINEAR, @@ -101,6 +382,103 @@ def main() -> None: if areas != areas_cv: raise AssertionError(f"component areas mismatch: {areas} != {areas_cv}") + # Geometry: planar homography residual agreement (not scale-normalized identity). + source = np.array( + [[10.0, 12.0], [70.0, 14.0], [18.0, 55.0], [66.0, 60.0], [40.0, 34.0], [28.0, 22.0]], + dtype=np.float64, + ) + homography = np.array( + [[1.04, 0.015, 2.5], [-0.02, 0.97, -1.25], [0.0002, -0.0001, 1.0]], + dtype=np.float64, + ) + target = [] + for point in source: + projected = homography @ np.array([point[0], point[1], 1.0]) + target.append([projected[0] / projected[2], projected[1] / projected[2]]) + target = np.asarray(target, dtype=np.float64) + estimated, inliers, residuals = sr.estimate_homography_ransac( + source, target, threshold=1.0, seed=3 + ) + estimated_cv, mask_cv = cv2.findHomography(source, target, method=0) + max_residual = float(np.max(residuals)) + results["homography_max_residual"] = max_residual + results["homography_inliers"] = int(np.sum(inliers)) + if max_residual > 1e-6 or estimated_cv is None: + raise AssertionError(f"homography residual {max_residual} too large") + # Compare transfer error of both models; allow tiny numeric disagreement. + def transfer_error(matrix: np.ndarray) -> float: + errors = [] + for src, dst in zip(source, target): + projected = matrix @ np.array([src[0], src[1], 1.0]) + errors.append( + np.hypot(projected[0] / projected[2] - dst[0], projected[1] / projected[2] - dst[1]) + ) + return float(np.max(errors)) + + transfer_sr = transfer_error(estimated) + transfer_cv = transfer_error(estimated_cv) + results["homography_transfer_sr"] = transfer_sr + results["homography_transfer_cv"] = transfer_cv + if transfer_sr > 1e-5 or transfer_cv > 1e-5: + raise AssertionError("homography transfer error exceeds tolerance") + + objects = np.array( + [ + [0.0, 0.0, 0.0], + [0.25, 0.0, 0.0], + [0.0, 0.2, 0.0], + [0.0, 0.0, 0.15], + [0.1, 0.1, 0.05], + [0.05, -0.08, 0.02], + [-0.1, 0.05, 0.08], + [0.12, 0.04, -0.03], + ], + dtype=np.float64, + ) + fx = fy = 500.0 + cx = cy = 240.0 + true_r = np.eye(3, dtype=np.float64) + true_t = np.array([0.12, -0.04, 2.4], dtype=np.float64) + images = [] + for point in objects: + camera = true_r @ point + true_t + images.append([fx * camera[0] / camera[2] + cx, fy * camera[1] / camera[2] + cy]) + images = np.asarray(images, dtype=np.float64) + rotation, translation = sr.solve_pnp(objects, images, fx, fy, cx, cy, 480, 480) + ok_cv, rvec, tvec = cv2.solvePnP( + objects.astype(np.float64), + images.astype(np.float64), + np.array([[fx, 0.0, cx], [0.0, fy, cy], [0.0, 0.0, 1.0]]), + None, + flags=cv2.SOLVEPNP_ITERATIVE, + ) + if not ok_cv: + raise AssertionError("OpenCV solvePnP failed") + t_error = float(np.linalg.norm(translation - tvec.reshape(3))) + results["pnp_translation_l2_vs_opencv"] = t_error + if abs(translation[2] - true_t[2]) > 0.05 or t_error > 0.05: + raise AssertionError(f"PnP translation disagreement {t_error}") + _ = rotation # shape already checked by consumer use + + width, height = 128, 96 + disparity = 16 + yy, xx = np.indices((height, width), dtype=np.int32) + left = ((xx * 17 + yy * 29) % 200 + 20).astype(np.uint8) + right = np.zeros_like(left) + right[:, : width - disparity] = left[:, disparity:] + disparity_sr = sr.stereo_block_match( + left, right, window_size=11, min_disparity=1, num_disparities=32, uniqueness_ratio=5.0 + ) + matcher = cv2.StereoBM_create(numDisparities=32, blockSize=11) + disparity_cv = matcher.compute(left, right).astype(np.float32) / 16.0 + center = (height // 2, width // 2) + results["stereo_bm_sr_center"] = float(disparity_sr[center]) + results["stereo_bm_cv_center"] = float(disparity_cv[center]) + if abs(float(disparity_sr[center]) - float(disparity)) > 1.0: + raise AssertionError("SpatialRust StereoBM center disparity incorrect") + if abs(float(disparity_cv[center]) - float(disparity)) > 1.5: + raise AssertionError("OpenCV StereoBM center disparity unexpected for synthetic pair") + results["status"] = "pass" print(json.dumps(results, indent=2, sort_keys=True)) diff --git a/crates/spatialrust-ai/Cargo.toml b/crates/spatialrust-ai/Cargo.toml new file mode 100644 index 0000000..500e756 --- /dev/null +++ b/crates/spatialrust-ai/Cargo.toml @@ -0,0 +1,30 @@ +[package] +name = "spatialrust-ai" +version.workspace = true +edition.workspace = true +license.workspace = true +authors.workspace = true +repository.workspace = true +rust-version.workspace = true +description = "Backend-independent inference contracts for SpatialRust" + +[features] +default = [] +onnxruntime = ["dep:ort"] +onnxruntime-cuda = ["onnxruntime", "ort/cuda"] +onnxruntime-tensorrt = ["onnxruntime-cuda", "ort/tensorrt"] +onnxruntime-directml = ["onnxruntime", "ort/directml"] + +[dependencies] +bytemuck.workspace = true +spatialrust-tensor.workspace = true +thiserror.workspace = true +ort = { version = "=2.0.0-rc.12", optional = true, default-features = false, features = ["std", "ndarray", "download-binaries", "tls-native", "copy-dylibs", "api-24"] } + +[dev-dependencies] +criterion.workspace = true + +[[bench]] +name = "onnxruntime" +harness = false +required-features = ["onnxruntime"] diff --git a/crates/spatialrust-ai/benches/onnxruntime.rs b/crates/spatialrust-ai/benches/onnxruntime.rs new file mode 100644 index 0000000..05476f9 --- /dev/null +++ b/crates/spatialrust-ai/benches/onnxruntime.rs @@ -0,0 +1,71 @@ +use std::{hint::black_box, sync::Arc, time::Duration}; + +use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; +use spatialrust_ai::{ + CopyPolicy, InferenceBackend, IoBinding, ModelSource, NamedTensors, OnnxRuntimeBackend, + OutputBinding, RunOptions, SessionOptions, +}; +use spatialrust_tensor::{DataType, Device, TensorBuffer, TensorDescriptor}; + +const DOUBLE_DYNAMIC: &[u8] = &[ + 8, 8, 18, 16, 115, 112, 97, 116, 105, 97, 108, 114, 117, 115, 116, 45, 116, 101, 115, 116, 58, + 106, 10, 27, 10, 5, 105, 110, 112, 117, 116, 10, 5, 105, 110, 112, 117, 116, 18, 6, 111, 117, + 116, 112, 117, 116, 34, 3, 65, 100, 100, 18, 14, 100, 111, 117, 98, 108, 101, 95, 100, 121, + 110, 97, 109, 105, 99, 90, 28, 10, 5, 105, 110, 112, 117, 116, 18, 19, 10, 17, 8, 1, 18, 13, + 10, 7, 18, 5, 98, 97, 116, 99, 104, 10, 2, 8, 3, 98, 29, 10, 6, 111, 117, 116, 112, 117, 116, + 18, 19, 10, 17, 8, 1, 18, 13, 10, 7, 18, 5, 98, 97, 116, 99, 104, 10, 2, 8, 3, 66, 4, 10, 0, + 16, 13, +]; + +fn benchmark_onnxruntime(c: &mut Criterion) { + let backend = OnnxRuntimeBackend; + let mut session = backend + .create_session(&ModelSource::Bytes(Arc::from(DOUBLE_DYNAMIC)), &SessionOptions::default()) + .expect("embedded model"); + let mut group = c.benchmark_group("onnxruntime_cpu_dynamic_rgb_f32"); + group.sample_size(10); + group.warm_up_time(Duration::from_millis(500)); + group.measurement_time(Duration::from_secs(2)); + + for (label, width, height) in + [("640p", 640_usize, 480_usize), ("1080p", 1920, 1080), ("4k", 3840, 2160)] + { + let pixels = width * height; + let descriptor = TensorDescriptor::contiguous(DataType::F32, vec![pixels, 3], Device::CPU); + let input = TensorBuffer::try_from_f32(vec![0.5; pixels * 3], descriptor).unwrap(); + let mut inputs = NamedTensors::new(); + inputs.insert("input", input).unwrap(); + group.throughput(Throughput::Bytes((pixels * 3 * 4 * 2) as u64)); + + group.bench_with_input(BenchmarkId::new("copy_run", label), &inputs, |b, inputs| { + b.iter(|| { + black_box( + session + .run_with_options( + inputs.clone(), + RunOptions { + input_copy: CopyPolicy::Allow, + output_copy: CopyPolicy::Allow, + }, + ) + .unwrap(), + ) + }); + }); + group.bench_with_input(BenchmarkId::new("io_binding", label), &inputs, |b, inputs| { + b.iter(|| { + let mut binding = IoBinding::try_new( + inputs.clone(), + vec![OutputBinding::Allocate { name: "output".into(), device: Device::CPU }], + ) + .unwrap(); + session.run_with_binding(&mut binding).unwrap(); + black_box(binding.into_results()) + }); + }); + } + group.finish(); +} + +criterion_group!(benches, benchmark_onnxruntime); +criterion_main!(benches); diff --git a/crates/spatialrust-ai/src/lib.rs b/crates/spatialrust-ai/src/lib.rs new file mode 100644 index 0000000..4f64f4e --- /dev/null +++ b/crates/spatialrust-ai/src/lib.rs @@ -0,0 +1,648 @@ +//! Backend-independent model, session, named-I/O, and explicit binding contracts. +//! +//! The default build contains no inference runtime. ONNX Runtime and hardware +//! execution providers are additive features and must preserve the copy/device +//! choices represented by these types. + +#![deny(unsafe_code)] +#![warn(missing_docs)] + +use std::{path::PathBuf, sync::Arc}; + +use spatialrust_tensor::{DataType, Device, TensorBuffer, TensorDescriptor}; + +#[cfg(feature = "onnxruntime")] +mod onnxruntime; +#[cfg(feature = "onnxruntime")] +pub use onnxruntime::{OnnxRuntimeBackend, OnnxRuntimeSession}; + +/// Result type for inference operations. +pub type AiResult = Result; + +/// Errors shared by inference contracts and backend adapters. +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +pub enum AiError { + /// A named input or output appears more than once. + #[error("duplicate tensor name `{0}`")] + DuplicateName(String), + /// A required model input was not supplied. + #[error("missing required input `{0}`")] + MissingInput(String), + /// A tensor name is not declared by the model. + #[error("unexpected tensor `{0}`")] + UnexpectedTensor(String), + /// Actual dtype differs from the model contract. + #[error("tensor `{name}` requires {expected:?}, found {actual:?}")] + DataTypeMismatch { + /// Tensor name. + name: String, + /// Model dtype. + expected: DataType, + /// Supplied dtype. + actual: DataType, + }, + /// Actual rank or dimension differs from the model contract. + #[error("tensor `{name}` shape {actual:?} does not match {expected:?}")] + ShapeMismatch { + /// Tensor name. + name: String, + /// Model dimensions. + expected: Vec, + /// Supplied dimensions. + actual: Vec, + }, + /// A preallocated output does not have enough bytes. + #[error("preallocated output `{name}` needs {required} bytes, found {found}")] + OutputBufferTooSmall { + /// Output name. + name: String, + /// Required allocation bytes. + required: usize, + /// Available allocation bytes. + found: usize, + }, + /// The selected backend cannot honor an operation or option. + #[error("backend `{backend}` does not support {operation}")] + Unsupported { + /// Backend identifier. + backend: String, + /// Unsupported operation. + operation: String, + }, + /// Backend-specific failure with stable outer context. + #[error("backend `{backend}` failed: {message}")] + Backend { + /// Backend identifier. + backend: String, + /// Backend error message. + message: String, + }, + /// Invalid public configuration. + #[error("invalid inference configuration: {0}")] + InvalidConfiguration(String), + /// A backend would need a copy that the caller did not authorize. + #[error( + "{direction} copy is required for tensor `{name}`; opt in explicitly or use I/O binding" + )] + CopyRequired { + /// Input or output transfer direction. + direction: &'static str, + /// Model-visible tensor name. + name: String, + }, +} + +/// One model dimension, fixed, unconstrained, or symbolically dynamic. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub enum Dimension { + /// Exact dimension size. + Fixed(usize), + /// Dynamic dimension without a model-provided symbol. + Dynamic, + /// Dynamic dimension sharing a model-provided symbolic name. + Symbol(String), +} + +impl Dimension { + fn accepts(&self, actual: usize) -> bool { + match self { + Self::Fixed(expected) => *expected == actual, + Self::Dynamic | Self::Symbol(_) => true, + } + } +} + +/// One named model input or output contract. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct TensorSpec { + /// ONNX/model-visible name. + pub name: String, + /// Required scalar/vector dtype. + pub dtype: DataType, + /// Ordered fixed or dynamic dimensions. + pub shape: Vec, +} + +impl TensorSpec { + /// Creates a named tensor specification. + pub fn new(name: impl Into, dtype: DataType, shape: Vec) -> Self { + Self { name: name.into(), dtype, shape } + } + + /// Validates a concrete tensor descriptor against this specification. + pub fn validate(&self, descriptor: &TensorDescriptor) -> AiResult<()> { + if descriptor.dtype() != self.dtype { + return Err(AiError::DataTypeMismatch { + name: self.name.clone(), + expected: self.dtype, + actual: descriptor.dtype(), + }); + } + if descriptor.shape().len() != self.shape.len() + || !self + .shape + .iter() + .zip(descriptor.shape()) + .all(|(expected, &actual)| expected.accepts(actual)) + { + return Err(AiError::ShapeMismatch { + name: self.name.clone(), + expected: self.shape.clone(), + actual: descriptor.shape().to_vec(), + }); + } + Ok(()) + } +} + +/// Model-visible named input and output metadata. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct ModelInfo { + /// Optional producer/model identifier. + pub name: Option, + /// Ordered model inputs. + pub inputs: Vec, + /// Ordered model outputs. + pub outputs: Vec, +} + +impl ModelInfo { + /// Validates uniqueness of all input and output names. + pub fn validate(&self) -> AiResult<()> { + ensure_unique(self.inputs.iter().map(|spec| spec.name.as_str()))?; + ensure_unique(self.outputs.iter().map(|spec| spec.name.as_str()))?; + Ok(()) + } + + /// Validates required names, dtype, and dynamic/fixed shapes for one request. + pub fn validate_inputs(&self, inputs: &NamedTensors) -> AiResult<()> { + for spec in &self.inputs { + let tensor = + inputs.get(&spec.name).ok_or_else(|| AiError::MissingInput(spec.name.clone()))?; + spec.validate(tensor.descriptor())?; + } + for (name, _) in inputs.iter() { + if !self.inputs.iter().any(|spec| spec.name == name) { + return Err(AiError::UnexpectedTensor(name.to_owned())); + } + } + Ok(()) + } +} + +fn ensure_unique<'a>(names: impl IntoIterator) -> AiResult<()> { + let mut seen = Vec::<&str>::new(); + for name in names { + if seen.contains(&name) { + return Err(AiError::DuplicateName(name.to_owned())); + } + seen.push(name); + } + Ok(()) +} + +/// Ordered, uniquely named tensor collection. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct NamedTensors { + values: Vec<(String, TensorBuffer)>, +} + +impl NamedTensors { + /// Creates an empty collection. + pub const fn new() -> Self { + Self { values: Vec::new() } + } + + /// Inserts a unique named tensor while retaining insertion order. + pub fn insert(&mut self, name: impl Into, tensor: TensorBuffer) -> AiResult<()> { + let name = name.into(); + if self.values.iter().any(|(existing, _)| existing == &name) { + return Err(AiError::DuplicateName(name)); + } + self.values.push((name, tensor)); + Ok(()) + } + + /// Returns a tensor by model-visible name. + pub fn get(&self, name: &str) -> Option<&TensorBuffer> { + self.values.iter().find(|(candidate, _)| candidate == name).map(|(_, value)| value) + } + + /// Returns ordered `(name, tensor)` pairs. + pub fn iter(&self) -> impl Iterator { + self.values.iter().map(|(name, tensor)| (name.as_str(), tensor)) + } + + /// Returns the tensor count. + pub fn len(&self) -> usize { + self.values.len() + } + + /// Returns whether no tensors are present. + pub fn is_empty(&self) -> bool { + self.values.is_empty() + } + + /// Consumes the collection into ordered pairs. + pub fn into_values(self) -> Vec<(String, TensorBuffer)> { + self.values + } +} + +/// Model bytes or a filesystem path supplied explicitly at session creation. +#[derive(Clone, Debug)] +pub enum ModelSource { + /// Read model bytes from this path during session creation. + Path(PathBuf), + /// Immutable in-memory model bytes. + Bytes(Arc<[u8]>), +} + +/// Graph optimization level requested from a backend. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub enum GraphOptimization { + /// Disable graph rewrites. + Disabled, + /// Apply safe basic rewrites. + Basic, + /// Apply extended rewrites. + Extended, + /// Apply all backend-supported rewrites. + #[default] + All, +} + +/// Runtime-independent session configuration. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SessionOptions { + /// Intra-operator worker count; `None` delegates to the backend. + pub intra_threads: Option, + /// Inter-operator worker count; `None` delegates to the backend. + pub inter_threads: Option, + /// Graph rewrite level. + pub graph_optimization: GraphOptimization, + /// Request deterministic kernels where supported. + pub deterministic: bool, +} + +/// Whether a run may allocate and copy host tensor data. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub enum CopyPolicy { + /// Fail instead of silently copying. + #[default] + Forbid, + /// Permit a documented host-to-host copy for this run. + Allow, +} + +/// Per-run host copy permissions. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub struct RunOptions { + /// Permission to repack/copy named inputs into backend values. + pub input_copy: CopyPolicy, + /// Permission to copy backend-owned outputs into `TensorBuffer`. + pub output_copy: CopyPolicy, +} + +impl Default for SessionOptions { + fn default() -> Self { + Self { + intra_threads: None, + inter_threads: None, + graph_optimization: GraphOptimization::All, + deterministic: false, + } + } +} + +impl SessionOptions { + /// Rejects zero thread counts. + pub fn validate(&self) -> AiResult<()> { + if self.intra_threads == Some(0) || self.inter_threads == Some(0) { + return Err(AiError::InvalidConfiguration("thread counts must be positive".into())); + } + Ok(()) + } +} + +/// Explicit output destination for an I/O-bound run. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum OutputBinding { + /// Ask the backend to allocate this output on the named device. + Allocate { + /// Model output name. + name: String, + /// Required allocation device. + device: Device, + }, + /// Write into this caller-owned CPU allocation. + PreallocatedCpu(PreallocatedOutput), +} + +/// Caller-owned mutable bytes for an explicitly bound CPU output. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct PreallocatedOutput { + name: String, + descriptor: TensorDescriptor, + storage: Option, +} + +#[derive(Clone, Debug)] +pub(crate) enum PreallocatedStorage { + Bytes(Vec), + U16(Vec), + F32(Vec), +} + +impl PreallocatedStorage { + fn allocation_bytes(&self) -> &[u8] { + match self { + Self::Bytes(values) => values, + Self::U16(values) => bytemuck::cast_slice(values), + Self::F32(values) => bytemuck::cast_slice(values), + } + } + + fn allocation_bytes_mut(&mut self) -> &mut [u8] { + match self { + Self::Bytes(values) => values, + Self::U16(values) => bytemuck::cast_slice_mut(values), + Self::F32(values) => bytemuck::cast_slice_mut(values), + } + } +} + +impl PartialEq for PreallocatedStorage { + fn eq(&self, other: &Self) -> bool { + self.allocation_bytes() == other.allocation_bytes() + } +} + +impl Eq for PreallocatedStorage {} + +impl PreallocatedOutput { + /// Creates a checked preallocated CPU output. + pub fn try_new( + name: impl Into, + descriptor: TensorDescriptor, + bytes: Vec, + ) -> AiResult { + let name = name.into(); + let required = descriptor + .required_byte_range() + .map_err(|error| AiError::InvalidConfiguration(error.to_string()))? + .end; + if !descriptor.device().is_host_accessible() { + return Err(AiError::InvalidConfiguration( + "PreallocatedCpu requires host-accessible storage".into(), + )); + } + if bytes.len() < required { + return Err(AiError::OutputBufferTooSmall { name, required, found: bytes.len() }); + } + Ok(Self { name, descriptor, storage: Some(PreallocatedStorage::Bytes(bytes)) }) + } + + /// Allocates aligned storage for a compact `u8`, `u16`, or `f32` CPU output. + pub fn allocate(name: impl Into, descriptor: TensorDescriptor) -> AiResult { + if !descriptor.is_c_contiguous() || descriptor.byte_offset() != 0 { + return Err(AiError::InvalidConfiguration( + "preallocated output must be compact with byte_offset=0".into(), + )); + } + if descriptor.device() != Device::CPU { + return Err(AiError::InvalidConfiguration( + "preallocated output allocation currently requires Device::CPU".into(), + )); + } + let elements = descriptor + .element_count() + .map_err(|error| AiError::InvalidConfiguration(error.to_string()))?; + let storage = match descriptor.dtype() { + DataType::U8 => PreallocatedStorage::Bytes(vec![0; elements]), + DataType::U16 => PreallocatedStorage::U16(vec![0; elements]), + DataType::F32 => PreallocatedStorage::F32(vec![0.0; elements]), + dtype => { + return Err(AiError::InvalidConfiguration(format!( + "preallocated output dtype {dtype:?} is not supported; use u8, u16, or f32" + ))) + } + }; + Ok(Self { name: name.into(), descriptor, storage: Some(storage) }) + } + + /// Returns the model output name. + pub fn name(&self) -> &str { + &self.name + } + + /// Returns output metadata. + pub const fn descriptor(&self) -> &TensorDescriptor { + &self.descriptor + } + + /// Returns mutable caller-owned output bytes for a backend binding. + pub fn allocation_bytes_mut(&mut self) -> AiResult<&mut [u8]> { + self.storage.as_mut().map(PreallocatedStorage::allocation_bytes_mut).ok_or_else(|| { + AiError::InvalidConfiguration(format!( + "preallocated output `{}` has already been consumed", + self.name + )) + }) + } + + /// Returns whether a bound run has taken ownership of this allocation. + pub fn is_consumed(&self) -> bool { + self.storage.is_none() + } + + /// Converts a completed binding into generic owned tensor storage. + pub fn into_tensor(mut self) -> AiResult { + let storage = self.take_storage()?; + tensor_from_preallocated(storage, self.descriptor) + } + + pub(crate) fn take_storage(&mut self) -> AiResult { + self.storage.take().ok_or_else(|| { + AiError::InvalidConfiguration(format!( + "preallocated output `{}` has already been consumed", + self.name + )) + }) + } +} + +fn tensor_from_preallocated( + storage: PreallocatedStorage, + descriptor: TensorDescriptor, +) -> AiResult { + let result = match storage { + PreallocatedStorage::Bytes(values) => TensorBuffer::try_new(values, descriptor), + PreallocatedStorage::U16(values) => TensorBuffer::try_from_u16(values, descriptor), + PreallocatedStorage::F32(values) => TensorBuffer::try_from_f32(values, descriptor), + }; + result.map_err(|error| AiError::InvalidConfiguration(error.to_string())) +} + +/// Inputs, requested output destinations, and completed named results. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct IoBinding { + inputs: NamedTensors, + outputs: Vec, + results: Option, +} + +impl IoBinding { + /// Creates an explicit binding request. + pub fn try_new(inputs: NamedTensors, outputs: Vec) -> AiResult { + ensure_unique(outputs.iter().map(|output| match output { + OutputBinding::Allocate { name, .. } => name.as_str(), + OutputBinding::PreallocatedCpu(output) => output.name(), + }))?; + Ok(Self { inputs, outputs, results: None }) + } + + /// Returns bound named inputs. + pub const fn inputs(&self) -> &NamedTensors { + &self.inputs + } + + /// Returns requested output destinations. + pub fn outputs(&self) -> &[OutputBinding] { + &self.outputs + } + + /// Returns mutable requested output destinations. + pub fn outputs_mut(&mut self) -> &mut [OutputBinding] { + &mut self.outputs + } + + /// Stores completed named results. Intended for backend implementers. + pub fn set_results(&mut self, results: NamedTensors) { + self.results = Some(results); + } + + /// Clears results before a backend starts another run. + pub fn clear_results(&mut self) { + self.results = None; + } + + /// Returns completed results after a successful bound run. + pub fn results(&self) -> Option<&NamedTensors> { + self.results.as_ref() + } + + /// Consumes completed results, if present. + pub fn into_results(self) -> Option { + self.results + } +} + +/// Stable interface implemented by inference engines. +pub trait InferenceBackend: Send + Sync { + /// Stable backend identifier such as `onnxruntime-cpu`. + fn name(&self) -> &str; + + /// Loads one model and returns an independent mutable session. + fn create_session( + &self, + source: &ModelSource, + options: &SessionOptions, + ) -> AiResult>; +} + +/// Loaded model session with named dynamic I/O. +pub trait ModelSession: Send { + /// Stable backend identifier for diagnostics and capability checks. + fn backend_name(&self) -> &str; + + /// Returns model input/output metadata captured at load time. + fn model_info(&self) -> &ModelInfo; + + /// Runs without authorizing hidden host copies. + fn run(&mut self, inputs: NamedTensors) -> AiResult { + self.run_with_options(inputs, RunOptions::default()) + } + + /// Runs named I/O with explicit host copy permissions. + fn run_with_options( + &mut self, + inputs: NamedTensors, + options: RunOptions, + ) -> AiResult; + + /// Runs with explicit input/output device or allocation bindings. + fn run_with_binding(&mut self, _binding: &mut IoBinding) -> AiResult<()> { + Err(AiError::Unsupported { + backend: self.backend_name().into(), + operation: "explicit I/O binding".into(), + }) + } +} + +#[cfg(test)] +mod tests { + use super::{ + AiError, Dimension, IoBinding, ModelInfo, NamedTensors, OutputBinding, PreallocatedOutput, + TensorSpec, + }; + use spatialrust_tensor::{DataType, Device, TensorBuffer, TensorDescriptor}; + + fn tensor(shape: Vec) -> TensorBuffer { + let len = shape.iter().product::() * 4; + TensorBuffer::try_new( + vec![0; len], + TensorDescriptor::contiguous(DataType::F32, shape, Device::CPU), + ) + .unwrap() + } + + #[test] + fn validates_named_dynamic_inputs() { + let info = ModelInfo { + name: Some("dynamic-batch".into()), + inputs: vec![TensorSpec::new( + "images", + DataType::F32, + vec![Dimension::Symbol("batch".into()), Dimension::Fixed(3), Dimension::Dynamic], + )], + outputs: vec![], + }; + let mut inputs = NamedTensors::new(); + inputs.insert("images", tensor(vec![4, 3, 224])).unwrap(); + info.validate_inputs(&inputs).unwrap(); + let wrong = TensorDescriptor::contiguous(DataType::F32, vec![4, 1, 224], Device::CPU); + assert!(matches!(info.inputs[0].validate(&wrong), Err(AiError::ShapeMismatch { .. }))); + } + + #[test] + fn named_tensors_reject_duplicates_and_missing_inputs() { + let mut inputs = NamedTensors::new(); + inputs.insert("x", tensor(vec![2])).unwrap(); + assert!(matches!(inputs.insert("x", tensor(vec![2])), Err(AiError::DuplicateName(_)))); + let info = ModelInfo { + name: None, + inputs: vec![TensorSpec::new("y", DataType::F32, vec![Dimension::Fixed(2)])], + outputs: vec![], + }; + assert!(matches!(info.validate_inputs(&inputs), Err(AiError::MissingInput(_)))); + } + + #[test] + fn io_binding_requires_explicit_unique_outputs() { + let descriptor = TensorDescriptor::contiguous(DataType::F32, vec![2], Device::CPU); + let output = PreallocatedOutput::allocate("scores", descriptor).unwrap(); + let binding = + IoBinding::try_new(NamedTensors::new(), vec![OutputBinding::PreallocatedCpu(output)]) + .unwrap(); + assert_eq!(binding.outputs().len(), 1); + assert!(IoBinding::try_new( + NamedTensors::new(), + vec![ + OutputBinding::Allocate { name: "y".into(), device: Device::CPU }, + OutputBinding::Allocate { name: "y".into(), device: Device::CPU }, + ], + ) + .is_err()); + } +} diff --git a/crates/spatialrust-ai/src/onnxruntime.rs b/crates/spatialrust-ai/src/onnxruntime.rs new file mode 100644 index 0000000..f414f02 --- /dev/null +++ b/crates/spatialrust-ai/src/onnxruntime.rs @@ -0,0 +1,759 @@ +//! ONNX Runtime CPU execution provider adapter. + +use std::sync::Arc; + +use ort::{ + ep::CPU, + memory::MemoryInfo, + session::{builder::GraphOptimizationLevel, Session}, + value::{DynValue, Tensor, TensorElementType, TensorRef, TensorValueType, ValueType}, +}; +use spatialrust_tensor::{DataType, Device, HostTensorStorage, TensorBuffer, TensorDescriptor}; + +use crate::{ + AiError, AiResult, CopyPolicy, Dimension, GraphOptimization, InferenceBackend, + IoBinding as SpatialIoBinding, ModelInfo, ModelSession, ModelSource, NamedTensors, + OutputBinding, PreallocatedStorage, RunOptions, SessionOptions, TensorSpec, +}; + +const BACKEND_NAME: &str = "onnxruntime-cpu"; + +/// ONNX Runtime backend configured explicitly for the CPU execution provider. +#[derive(Clone, Copy, Debug, Default)] +pub struct OnnxRuntimeBackend; + +impl InferenceBackend for OnnxRuntimeBackend { + fn name(&self) -> &str { + BACKEND_NAME + } + + fn create_session( + &self, + source: &ModelSource, + options: &SessionOptions, + ) -> AiResult> { + options.validate()?; + let mut builder = Session::builder().map_err(ort_error)?; + builder = builder.with_execution_providers([CPU::default().build()]).map_err(ort_error)?; + if let Some(threads) = options.intra_threads { + builder = builder.with_intra_threads(threads).map_err(ort_error)?; + } + if let Some(threads) = options.inter_threads { + builder = builder.with_inter_threads(threads).map_err(ort_error)?; + } + builder = builder + .with_optimization_level(match options.graph_optimization { + GraphOptimization::Disabled => GraphOptimizationLevel::Disable, + GraphOptimization::Basic => GraphOptimizationLevel::Level1, + GraphOptimization::Extended => GraphOptimizationLevel::Level2, + GraphOptimization::All => GraphOptimizationLevel::Level3, + }) + .map_err(ort_error)?; + builder = builder.with_deterministic_compute(options.deterministic).map_err(ort_error)?; + let session = match source { + ModelSource::Path(path) => builder.commit_from_file(path).map_err(ort_error)?, + ModelSource::Bytes(bytes) => builder.commit_from_memory(bytes).map_err(ort_error)?, + }; + let info = model_info(&session)?; + Ok(Box::new(OnnxRuntimeSession { session, info })) + } +} + +/// Loaded ONNX Runtime CPU session. +#[derive(Debug)] +pub struct OnnxRuntimeSession { + session: Session, + info: ModelInfo, +} + +impl ModelSession for OnnxRuntimeSession { + fn backend_name(&self) -> &str { + BACKEND_NAME + } + + fn model_info(&self) -> &ModelInfo { + &self.info + } + + fn run_with_options( + &mut self, + inputs: NamedTensors, + options: RunOptions, + ) -> AiResult { + self.info.validate_inputs(&inputs)?; + if options.input_copy == CopyPolicy::Forbid { + return Err(AiError::CopyRequired { + direction: "input host", + name: self.info.inputs.first().map_or_else(String::new, |spec| spec.name.clone()), + }); + } + if options.output_copy == CopyPolicy::Forbid { + return Err(AiError::CopyRequired { + direction: "output host", + name: self.info.outputs.first().map_or_else(String::new, |spec| spec.name.clone()), + }); + } + let values = inputs + .iter() + .map(|(name, tensor)| Ok((name.to_owned(), to_ort_tensor(name, tensor)?))) + .collect::>>()?; + let outputs = self.session.run(values).map_err(ort_error)?; + let mut named = NamedTensors::new(); + for spec in &self.info.outputs { + let value = outputs.get(&spec.name).ok_or_else(|| AiError::Backend { + backend: BACKEND_NAME.into(), + message: format!("runtime omitted output `{}`", spec.name), + })?; + named.insert(spec.name.clone(), copy_ort_output(spec, value)?)?; + } + Ok(named) + } + + fn run_with_binding(&mut self, binding: &mut SpatialIoBinding) -> AiResult<()> { + binding.clear_results(); + self.info.validate_inputs(binding.inputs())?; + let mut ort_binding = self.session.create_binding().map_err(ort_error)?; + for (name, tensor) in binding.inputs().iter() { + bind_zero_copy_input(&mut ort_binding, name, tensor)?; + } + + let requested = binding + .outputs() + .iter() + .map(|output| match output { + OutputBinding::Allocate { name, .. } => name.clone(), + OutputBinding::PreallocatedCpu(output) => output.name().to_owned(), + }) + .collect::>(); + for output in binding.outputs_mut() { + match output { + OutputBinding::Allocate { name, device } => { + output_spec(&self.info, name)?; + if *device != Device::CPU { + return Err(AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!("bound output device {device:?}"), + }); + } + ort_binding + .bind_output_to_device(name.clone(), &MemoryInfo::default()) + .map_err(ort_error)?; + } + OutputBinding::PreallocatedCpu(output) => { + let spec = output_spec(&self.info, output.name())?; + spec.validate(output.descriptor())?; + if !output.descriptor().is_c_contiguous() + || output.descriptor().byte_offset() != 0 + { + return Err(AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!( + "strided preallocated output `{}`; allocate a compact buffer", + output.name() + ), + }); + } + let name = output.name().to_owned(); + let shape = output.descriptor().shape().to_vec(); + bind_preallocated_output( + &mut ort_binding, + &name, + shape, + output.take_storage()?, + spec.dtype, + )?; + } + } + } + + let mut outputs = self.session.run_binding(&ort_binding).map_err(ort_error)?; + let mut results = NamedTensors::new(); + for name in requested { + let spec = output_spec(&self.info, &name)?; + let value = outputs.remove(&name).ok_or_else(|| AiError::Backend { + backend: BACKEND_NAME.into(), + message: format!("runtime omitted bound output `{name}`"), + })?; + results.insert(name, retain_ort_output(spec, value)?)?; + } + drop(outputs); + binding.set_results(results); + Ok(()) + } +} + +fn output_spec<'a>(info: &'a ModelInfo, name: &str) -> AiResult<&'a TensorSpec> { + info.outputs + .iter() + .find(|spec| spec.name == name) + .ok_or_else(|| AiError::UnexpectedTensor(name.to_owned())) +} + +fn bind_zero_copy_input( + binding: &mut ort::session::IoBinding, + name: &str, + tensor: &TensorBuffer, +) -> AiResult<()> { + let descriptor = tensor.descriptor(); + if !descriptor.is_c_contiguous() || descriptor.byte_offset() != 0 { + return Err(AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!("strided bound input `{name}`; explicitly pack it first"), + }); + } + if let Some(storage) = tensor.host_storage() { + if let Some(storage) = storage.as_any().downcast_ref::() { + return storage.bind_input(binding, name); + } + } + let expected = descriptor + .element_count() + .map_err(|error| AiError::InvalidConfiguration(error.to_string()))?; + let shape = descriptor.shape().to_vec(); + macro_rules! bind_arc { + ($getter:ident, $type:ty) => {{ + let values = tensor.$getter().ok_or_else(|| AiError::CopyRequired { + direction: "input alignment", + name: name.to_owned(), + })?; + if values.len() != expected { + return Err(AiError::InvalidConfiguration(format!( + "bound input `{name}` allocation has {} elements, expected {expected}", + values.len() + ))); + } + let value = TensorRef::<$type>::from_array_view((shape, values)).map_err(ort_error)?; + binding.bind_input(name, &value).map_err(ort_error) + }}; + } + match descriptor.dtype() { + DataType::U8 => bind_arc!(shared_bytes, u8), + DataType::U16 => bind_arc!(shared_u16, u16), + DataType::U32 => bind_arc!(shared_u32, u32), + DataType::I16 => bind_arc!(shared_i16, i16), + DataType::I32 => bind_arc!(shared_i32, i32), + DataType::I64 => bind_arc!(shared_i64, i64), + DataType::F32 => bind_arc!(shared_f32, f32), + DataType::F64 => bind_arc!(shared_f64, f64), + dtype => Err(AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!("zero-copy bound input dtype {dtype:?}"), + }), + } +} + +fn bind_preallocated_output( + binding: &mut ort::session::IoBinding, + name: &str, + shape: Vec, + storage: PreallocatedStorage, + dtype: DataType, +) -> AiResult<()> { + match (storage, dtype) { + (PreallocatedStorage::Bytes(values), DataType::U8) => binding + .bind_output(name, Tensor::::from_array((shape, values)).map_err(ort_error)?) + .map_err(ort_error), + (PreallocatedStorage::U16(values), DataType::U16) => binding + .bind_output(name, Tensor::::from_array((shape, values)).map_err(ort_error)?) + .map_err(ort_error), + (PreallocatedStorage::F32(values), DataType::F32) => binding + .bind_output(name, Tensor::::from_array((shape, values)).map_err(ort_error)?) + .map_err(ort_error), + _ => Err(AiError::CopyRequired { direction: "output alignment", name: name.to_owned() }), + } +} + +#[derive(Debug)] +enum OrtHostStorage { + U8(Tensor), + U16(Tensor), + U32(Tensor), + I8(Tensor), + I16(Tensor), + I32(Tensor), + I64(Tensor), + F32(Tensor), + F64(Tensor), +} + +impl OrtHostStorage { + fn bind_input(&self, binding: &mut ort::session::IoBinding, name: &str) -> AiResult<()> { + macro_rules! bind { + ($value:expr) => { + binding.bind_input(name, $value).map_err(ort_error) + }; + } + match self { + Self::U8(value) => bind!(value), + Self::U16(value) => bind!(value), + Self::U32(value) => bind!(value), + Self::I8(value) => bind!(value), + Self::I16(value) => bind!(value), + Self::I32(value) => bind!(value), + Self::I64(value) => bind!(value), + Self::F32(value) => bind!(value), + Self::F64(value) => bind!(value), + } + } +} + +impl HostTensorStorage for OrtHostStorage { + fn dtype(&self) -> DataType { + match self { + Self::U8(_) => DataType::U8, + Self::U16(_) => DataType::U16, + Self::U32(_) => DataType::U32, + Self::I8(_) => DataType::I8, + Self::I16(_) => DataType::I16, + Self::I32(_) => DataType::I32, + Self::I64(_) => DataType::I64, + Self::F32(_) => DataType::F32, + Self::F64(_) => DataType::F64, + } + } + + fn allocation_bytes(&self) -> &[u8] { + macro_rules! bytes { + ($value:expr) => { + bytemuck::cast_slice($value.extract_tensor().1) + }; + } + match self { + Self::U8(value) => value.extract_tensor().1, + Self::U16(value) => bytes!(value), + Self::U32(value) => bytes!(value), + Self::I8(value) => bytes!(value), + Self::I16(value) => bytes!(value), + Self::I32(value) => bytes!(value), + Self::I64(value) => bytes!(value), + Self::F32(value) => bytes!(value), + Self::F64(value) => bytes!(value), + } + } + + fn as_any(&self) -> &dyn std::any::Any { + self + } +} + +fn retain_ort_output(spec: &TensorSpec, value: DynValue) -> AiResult { + macro_rules! retain { + ($type:ty, $variant:ident) => {{ + let tensor = value.downcast::>().map_err(ort_error)?; + let shape = tensor + .extract_tensor() + .0 + .iter() + .map(|&dimension| usize::try_from(dimension).expect("runtime shape is concrete")) + .collect::>(); + let storage: Arc = Arc::new(OrtHostStorage::$variant(tensor)); + (shape, storage) + }}; + } + let (shape, storage) = match spec.dtype { + DataType::U8 => retain!(u8, U8), + DataType::U16 => retain!(u16, U16), + DataType::U32 => retain!(u32, U32), + DataType::I8 => retain!(i8, I8), + DataType::I16 => retain!(i16, I16), + DataType::I32 => retain!(i32, I32), + DataType::I64 => retain!(i64, I64), + DataType::F32 => retain!(f32, F32), + DataType::F64 => retain!(f64, F64), + dtype => { + return Err(AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!("retaining bound output dtype {dtype:?}"), + }) + } + }; + TensorBuffer::try_from_host_storage( + storage, + TensorDescriptor::contiguous(spec.dtype, shape, Device::CPU), + ) + .map_err(|error| AiError::Backend { backend: BACKEND_NAME.into(), message: error.to_string() }) +} + +fn model_info(session: &Session) -> AiResult { + let inputs = session.inputs().iter().map(outlet_spec).collect::>>()?; + let outputs = session.outputs().iter().map(outlet_spec).collect::>>()?; + let info = ModelInfo { name: None, inputs, outputs }; + info.validate()?; + Ok(info) +} + +fn outlet_spec(outlet: &ort::value::Outlet) -> AiResult { + let ValueType::Tensor { ty, shape, dimension_symbols } = outlet.dtype() else { + return Err(AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!("non-tensor ONNX value `{}`", outlet.name()), + }); + }; + let dtype = decode_ort_dtype(*ty).ok_or_else(|| AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!("ONNX dtype {ty} for `{}`", outlet.name()), + })?; + let dimensions = shape + .iter() + .enumerate() + .map(|(index, &dimension)| { + if dimension >= 0 { + Dimension::Fixed(dimension as usize) + } else { + let symbol = &dimension_symbols[index]; + if symbol.is_empty() { + Dimension::Dynamic + } else { + Dimension::Symbol(symbol.clone()) + } + } + }) + .collect(); + Ok(TensorSpec::new(outlet.name(), dtype, dimensions)) +} + +fn decode_ort_dtype(dtype: TensorElementType) -> Option { + Some(match dtype { + TensorElementType::Float32 => DataType::F32, + TensorElementType::Float64 => DataType::F64, + TensorElementType::Uint8 => DataType::U8, + TensorElementType::Uint16 => DataType::U16, + TensorElementType::Uint32 => DataType::U32, + TensorElementType::Int8 => DataType::I8, + TensorElementType::Int16 => DataType::I16, + TensorElementType::Int32 => DataType::I32, + TensorElementType::Int64 => DataType::I64, + TensorElementType::Float16 => DataType::F16, + TensorElementType::Bfloat16 => DataType::BF16, + TensorElementType::Bool => DataType::BOOL, + _ => return None, + }) +} + +fn to_ort_tensor(name: &str, tensor: &TensorBuffer) -> AiResult { + let descriptor = tensor.descriptor(); + if !descriptor.is_c_contiguous() { + return Err(AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!("strided input `{name}`; explicitly pack it first"), + }); + } + let range = descriptor.required_byte_range().map_err(|error| AiError::Backend { + backend: BACKEND_NAME.into(), + message: error.to_string(), + })?; + let bytes = &tensor.allocation_bytes()[range]; + let shape = descriptor.shape().to_vec(); + macro_rules! typed { + ($type:ty, $width:expr) => {{ + let values = decode_ne::<$type>(bytes, $width, |chunk| { + <$type>::from_ne_bytes(chunk.try_into().expect("fixed chunk width")) + }); + Tensor::<$type>::from_array((shape, values)).map(|value| value.into_dyn()) + }}; + } + let value = match descriptor.dtype() { + DataType::U8 => Tensor::::from_array((shape, bytes.to_vec())).map(|v| v.into_dyn()), + DataType::U16 => typed!(u16, 2), + DataType::U32 => typed!(u32, 4), + DataType::I8 => Tensor::::from_array(( + shape, + bytes.iter().map(|&value| value as i8).collect::>(), + )) + .map(|v| v.into_dyn()), + DataType::I16 => typed!(i16, 2), + DataType::I32 => typed!(i32, 4), + DataType::I64 => typed!(i64, 8), + DataType::F32 => typed!(f32, 4), + DataType::F64 => typed!(f64, 8), + dtype => { + return Err(AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!("input dtype {dtype:?}"), + }) + } + }; + value.map_err(ort_error) +} + +fn decode_ne(bytes: &[u8], width: usize, decode: impl Fn(&[u8]) -> T) -> Vec { + debug_assert_eq!(bytes.len() % width, 0); + bytes.chunks_exact(width).map(decode).collect() +} + +fn copy_ort_output(spec: &TensorSpec, value: &DynValue) -> AiResult { + macro_rules! extract { + ($type:ty, $constructor:ident) => {{ + let (shape, values) = value.try_extract_tensor::<$type>().map_err(ort_error)?; + let descriptor = TensorDescriptor::contiguous( + spec.dtype, + shape.iter().map(|&dimension| dimension as usize).collect(), + Device::CPU, + ); + TensorBuffer::$constructor(values.to_vec(), descriptor) + }}; + } + let result = match spec.dtype { + DataType::U8 => extract!(u8, try_new), + DataType::U16 => extract!(u16, try_from_u16), + DataType::U32 => extract!(u32, try_from_u32), + DataType::I8 => { + let (shape, values) = value.try_extract_tensor::().map_err(ort_error)?; + let descriptor = TensorDescriptor::contiguous( + DataType::I8, + shape.iter().map(|&dimension| dimension as usize).collect(), + Device::CPU, + ); + TensorBuffer::try_new(values.iter().map(|&item| item as u8).collect(), descriptor) + } + DataType::I16 => extract!(i16, try_from_i16), + DataType::I32 => extract!(i32, try_from_i32), + DataType::I64 => extract!(i64, try_from_i64), + DataType::F32 => extract!(f32, try_from_f32), + DataType::F64 => extract!(f64, try_from_f64), + dtype => { + return Err(AiError::Unsupported { + backend: BACKEND_NAME.into(), + operation: format!("output dtype {dtype:?}"), + }) + } + }; + result.map_err(|error| AiError::Backend { + backend: BACKEND_NAME.into(), + message: error.to_string(), + }) +} + +fn ort_error(error: impl std::fmt::Display) -> AiError { + AiError::Backend { backend: BACKEND_NAME.into(), message: error.to_string() } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use super::OnnxRuntimeBackend; + use crate::{ + AiError, CopyPolicy, Dimension, InferenceBackend, IoBinding, ModelSource, NamedTensors, + OutputBinding, PreallocatedOutput, RunOptions, SessionOptions, + }; + use spatialrust_tensor::{DataType, Device, TensorBuffer, TensorDescriptor}; + + const DOUBLE_DYNAMIC: &[u8] = &[ + 8, 8, 18, 16, 115, 112, 97, 116, 105, 97, 108, 114, 117, 115, 116, 45, 116, 101, 115, 116, + 58, 106, 10, 27, 10, 5, 105, 110, 112, 117, 116, 10, 5, 105, 110, 112, 117, 116, 18, 6, + 111, 117, 116, 112, 117, 116, 34, 3, 65, 100, 100, 18, 14, 100, 111, 117, 98, 108, 101, 95, + 100, 121, 110, 97, 109, 105, 99, 90, 28, 10, 5, 105, 110, 112, 117, 116, 18, 19, 10, 17, 8, + 1, 18, 13, 10, 7, 18, 5, 98, 97, 116, 99, 104, 10, 2, 8, 3, 98, 29, 10, 6, 111, 117, 116, + 112, 117, 116, 18, 19, 10, 17, 8, 1, 18, 13, 10, 7, 18, 5, 98, 97, 116, 99, 104, 10, 2, 8, + 3, 66, 4, 10, 0, 16, 13, + ]; + + fn f32_tensor(shape: Vec, values: &[f32]) -> TensorBuffer { + TensorBuffer::try_from_f32( + values.to_vec(), + TensorDescriptor::contiguous(DataType::F32, shape, Device::CPU), + ) + .unwrap() + } + + fn f32_values(tensor: &TensorBuffer) -> Vec { + tensor + .allocation_bytes() + .chunks_exact(4) + .map(|chunk| f32::from_ne_bytes(chunk.try_into().unwrap())) + .collect() + } + + fn dynamic_identity_model_with_onnx_dtype(dtype: u8) -> Arc<[u8]> { + let mut model = DOUBLE_DYNAMIC.to_vec(); + let graph = model + .windows(4) + .position(|window| window == [58, 106, 10, 27]) + .expect("embedded graph and node lengths"); + model[graph + 1] = 104; + let node = graph + 2; + let identity_node = [ + 10, 25, 10, 5, 105, 110, 112, 117, 116, 18, 6, 111, 117, 116, 112, 117, 116, 34, 8, 73, + 100, 101, 110, 116, 105, 116, 121, + ]; + model.splice(node..node + 29, identity_node); + let mut replacements = 0; + for index in 0..model.len().saturating_sub(3) { + if model[index..index + 4] == [8, 1, 18, 13] { + model[index + 1] = dtype; + replacements += 1; + } + } + assert_eq!(replacements, 2, "input and output tensor types must both be replaced"); + Arc::from(model) + } + + #[test] + fn cpu_session_preserves_named_dynamic_contract_and_requires_copy_opt_in() { + let backend = OnnxRuntimeBackend; + let mut session = backend + .create_session( + &ModelSource::Bytes(Arc::from(DOUBLE_DYNAMIC)), + &SessionOptions::default(), + ) + .unwrap(); + assert_eq!(session.model_info().inputs[0].name, "input"); + assert_eq!( + session.model_info().inputs[0].shape, + vec![Dimension::Symbol("batch".into()), Dimension::Fixed(3)] + ); + + let mut inputs = NamedTensors::new(); + inputs.insert("input", f32_tensor(vec![2, 3], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0])).unwrap(); + assert!(matches!(session.run(inputs.clone()), Err(AiError::CopyRequired { .. }))); + let outputs = session + .run_with_options( + inputs, + RunOptions { input_copy: CopyPolicy::Allow, output_copy: CopyPolicy::Allow }, + ) + .unwrap(); + let output = outputs.get("output").unwrap(); + assert_eq!(output.descriptor().shape(), &[2, 3]); + assert_eq!(f32_values(output), &[2.0, 4.0, 6.0, 8.0, 10.0, 12.0]); + } + + #[test] + fn io_binding_allocates_dynamic_output_and_reuses_it_as_zero_copy_input() { + let backend = OnnxRuntimeBackend; + let mut session = backend + .create_session( + &ModelSource::Bytes(Arc::from(DOUBLE_DYNAMIC)), + &SessionOptions::default(), + ) + .unwrap(); + let mut inputs = NamedTensors::new(); + inputs.insert("input", f32_tensor(vec![1, 3], &[1.0, 2.0, 3.0])).unwrap(); + let mut binding = IoBinding::try_new( + inputs, + vec![OutputBinding::Allocate { name: "output".into(), device: Device::CPU }], + ) + .unwrap(); + session.run_with_binding(&mut binding).unwrap(); + let first = binding.results().unwrap().get("output").unwrap(); + assert!(first.host_storage().is_some()); + assert_eq!(f32_values(first), &[2.0, 4.0, 6.0]); + + let mut inputs = NamedTensors::new(); + inputs.insert("input", first.clone()).unwrap(); + let mut chained = IoBinding::try_new( + inputs, + vec![OutputBinding::Allocate { name: "output".into(), device: Device::CPU }], + ) + .unwrap(); + session.run_with_binding(&mut chained).unwrap(); + assert_eq!( + f32_values(chained.results().unwrap().get("output").unwrap()), + &[4.0, 8.0, 12.0] + ); + } + + #[test] + fn io_binding_writes_directly_into_caller_preallocated_f32_storage() { + let backend = OnnxRuntimeBackend; + let mut session = backend + .create_session( + &ModelSource::Bytes(Arc::from(DOUBLE_DYNAMIC)), + &SessionOptions::default(), + ) + .unwrap(); + let mut inputs = NamedTensors::new(); + inputs.insert("input", f32_tensor(vec![2, 3], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0])).unwrap(); + let descriptor = TensorDescriptor::contiguous(DataType::F32, vec![2, 3], Device::CPU); + let mut output = PreallocatedOutput::allocate("output", descriptor).unwrap(); + let allocation = output.allocation_bytes_mut().unwrap().as_ptr(); + let mut binding = + IoBinding::try_new(inputs, vec![OutputBinding::PreallocatedCpu(output)]).unwrap(); + session.run_with_binding(&mut binding).unwrap(); + let result = binding.results().unwrap().get("output").unwrap(); + assert_eq!(result.allocation_bytes().as_ptr(), allocation); + assert_eq!(f32_values(result), &[2.0, 4.0, 6.0, 8.0, 10.0, 12.0]); + } + + #[test] + fn io_binding_rejects_unaligned_raw_f32_storage_instead_of_copying() { + let backend = OnnxRuntimeBackend; + let mut session = backend + .create_session( + &ModelSource::Bytes(Arc::from(DOUBLE_DYNAMIC)), + &SessionOptions::default(), + ) + .unwrap(); + let descriptor = TensorDescriptor::contiguous(DataType::F32, vec![1, 3], Device::CPU); + let input = TensorBuffer::try_new(vec![0; 12], descriptor).unwrap(); + let mut inputs = NamedTensors::new(); + inputs.insert("input", input).unwrap(); + let mut binding = IoBinding::try_new( + inputs, + vec![OutputBinding::Allocate { name: "output".into(), device: Device::CPU }], + ) + .unwrap(); + assert!(matches!( + session.run_with_binding(&mut binding), + Err(AiError::CopyRequired { direction: "input alignment", .. }) + )); + } + + #[test] + fn io_binding_supports_aligned_u8_and_u16_storage() { + let backend = OnnxRuntimeBackend; + + let mut u8_session = backend + .create_session( + &ModelSource::Bytes(dynamic_identity_model_with_onnx_dtype(2)), + &SessionOptions::default(), + ) + .unwrap(); + let mut u8_inputs = NamedTensors::new(); + u8_inputs + .insert( + "input", + TensorBuffer::try_new( + vec![1, 2, 3], + TensorDescriptor::contiguous(DataType::U8, vec![1, 3], Device::CPU), + ) + .unwrap(), + ) + .unwrap(); + let mut u8_binding = IoBinding::try_new( + u8_inputs, + vec![OutputBinding::Allocate { name: "output".into(), device: Device::CPU }], + ) + .unwrap(); + u8_session.run_with_binding(&mut u8_binding).unwrap(); + assert_eq!( + u8_binding.results().unwrap().get("output").unwrap().allocation_bytes(), + &[1, 2, 3] + ); + + let mut u16_session = backend + .create_session( + &ModelSource::Bytes(dynamic_identity_model_with_onnx_dtype(4)), + &SessionOptions::default(), + ) + .unwrap(); + let descriptor = TensorDescriptor::contiguous(DataType::U16, vec![1, 3], Device::CPU); + let mut u16_inputs = NamedTensors::new(); + u16_inputs + .insert( + "input", + TensorBuffer::try_from_u16(vec![10, 20, 30], descriptor.clone()).unwrap(), + ) + .unwrap(); + let output = PreallocatedOutput::allocate("output", descriptor).unwrap(); + let mut u16_binding = + IoBinding::try_new(u16_inputs, vec![OutputBinding::PreallocatedCpu(output)]).unwrap(); + u16_session.run_with_binding(&mut u16_binding).unwrap(); + assert_eq!( + bytemuck::cast_slice::( + u16_binding.results().unwrap().get("output").unwrap().allocation_bytes() + ), + &[10, 20, 30] + ); + } +} diff --git a/crates/spatialrust-core/src/tensor.rs b/crates/spatialrust-core/src/tensor.rs index 0bc8f7e..dfc0f60 100644 --- a/crates/spatialrust-core/src/tensor.rs +++ b/crates/spatialrust-core/src/tensor.rs @@ -37,7 +37,7 @@ impl<'a> SpatialTensor<'a> { /// Returns the underlying point cloud. #[must_use] - pub const fn cloud(&self) -> &PointCloud { + pub const fn cloud(&self) -> &'a PointCloud { self.cloud } diff --git a/crates/spatialrust-image-io/Cargo.toml b/crates/spatialrust-image-io/Cargo.toml new file mode 100644 index 0000000..093bd45 --- /dev/null +++ b/crates/spatialrust-image-io/Cargo.toml @@ -0,0 +1,37 @@ +[package] +name = "spatialrust-image-io" +version.workspace = true +edition.workspace = true +license.workspace = true +authors.workspace = true +repository.workspace = true +rust-version.workspace = true +description = "Feature-gated image codecs and bounded stream IO for SpatialRust" + +[features] +default = ["standard"] +standard = ["png", "jpeg", "pnm", "exif"] +png = ["image/png"] +jpeg = ["image/jpeg"] +pnm = ["image/pnm"] +tiff = ["image/tiff", "exif"] +openexr = ["image/openexr", "dep:exr"] +exif = ["dep:kamadak-exif"] +full = ["standard", "tiff", "openexr"] + +[dependencies] +spatialrust-image.workspace = true +image.workspace = true +kamadak-exif = { workspace = true, optional = true } +exr = { workspace = true, optional = true } +thiserror.workspace = true + +[dev-dependencies] +criterion.workspace = true +proptest.workspace = true +tempfile.workspace = true + +[[bench]] +name = "decode" +harness = false +required-features = ["standard"] diff --git a/crates/spatialrust-image-io/benches/decode.rs b/crates/spatialrust-image-io/benches/decode.rs new file mode 100644 index 0000000..c380aeb --- /dev/null +++ b/crates/spatialrust-image-io/benches/decode.rs @@ -0,0 +1,35 @@ +use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; +use spatialrust_image::Image; +use spatialrust_image_io::{ + decode_bytes, encode_bytes, DecodedPixels, EncodeOptions, ImageFileFormat, +}; + +fn encoded_rgb(width: usize, height: usize, format: ImageFileFormat) -> Vec { + let data = + (0..width * height * 3).map(|index| ((index * 31 + index / 7) & 0xff) as u8).collect(); + let image = Image::try_new(width, height, data).expect("valid benchmark image"); + encode_bytes(&DecodedPixels::Rgb8(image), EncodeOptions::new(format)) + .expect("benchmark fixture encodes") +} + +fn benchmark_decode(c: &mut Criterion) { + let mut group = c.benchmark_group("decode_rgb8"); + group.sample_size(10); + for &(name, width, height) in &[("640p", 640, 480), ("1080p", 1920, 1080), ("4k", 3840, 2160)] { + for format in [ImageFileFormat::Png, ImageFileFormat::Jpeg] { + let encoded = encoded_rgb(width, height, format); + group.throughput(Throughput::Elements((width * height) as u64)); + group.bench_with_input( + BenchmarkId::new(format!("{format}/{name}"), encoded.len()), + &encoded, + |b, bytes| { + b.iter(|| decode_bytes(black_box(bytes), Default::default()).unwrap()); + }, + ); + } + } + group.finish(); +} + +criterion_group!(benches, benchmark_decode); +criterion_main!(benches); diff --git a/crates/spatialrust-image-io/src/lib.rs b/crates/spatialrust-image-io/src/lib.rs new file mode 100644 index 0000000..2573dcf --- /dev/null +++ b/crates/spatialrust-image-io/src/lib.rs @@ -0,0 +1,860 @@ +//! Bounded, feature-gated image decoding and encoding for SpatialRust. +//! +//! All input is staged in CPU memory with an explicit compressed-input limit. +//! This crate never uploads to or reads from a device. Codec dependencies are +//! selected independently from the storage-only `spatialrust-image` crate. +//! +//! ```no_run +//! use spatialrust_image_io::{decode_path, DecodeOptions, DecodedPixels}; +//! +//! let decoded = decode_path("frame.png", DecodeOptions::default())?; +//! match decoded.pixels() { +//! DecodedPixels::Rgb8(image) => assert!(image.width() > 0), +//! other => println!("decoded {other:?}"), +//! } +//! # Ok::<(), spatialrust_image_io::ImageIoError>(()) +//! ``` + +#![deny(unsafe_code)] +#![warn(missing_docs)] + +use std::fs::File; +use std::io::{BufReader, BufWriter, Cursor, Read, Seek, Write}; +use std::path::Path; + +use image::{DynamicImage, GenericImageView, ImageBuffer, ImageFormat, ImageOutputFormat}; +use spatialrust_image::{AlphaMode, ColorRange, ColorSpace, Image, ImageError, ImageMetadata}; + +/// Errors produced by bounded image IO. +#[derive(Debug, thiserror::Error)] +pub enum ImageIoError { + /// Reading or writing the stream failed. + #[error(transparent)] + Io(#[from] std::io::Error), + /// A codec rejected the image or encoded output. + #[error(transparent)] + Codec(#[from] image::ImageError), + /// Constructing a typed SpatialRust image failed. + #[error(transparent)] + Image(#[from] ImageError), + /// The stream exceeded the configured compressed-input limit. + #[error("encoded input exceeds the {maximum} byte limit")] + InputTooLarge { + /// Maximum accepted encoded byte count. + maximum: usize, + }, + /// Decoded dimensions or pixel count exceeded the configured limits. + #[error("decoded image dimensions {width}x{height} exceed configured limits")] + DimensionsTooLarge { + /// Encoded image width. + width: u32, + /// Encoded image height. + height: u32, + }, + /// The detected/requested format is not enabled in this build. + #[error("image format `{0}` is not enabled in this build")] + FormatDisabled(ImageFileFormat), + /// The input format is not one of SpatialRust's image-IO formats. + #[error("unsupported image format: {0}")] + UnsupportedFormat(String), + /// Encoding options were invalid. + #[error("invalid encode option: {0}")] + InvalidEncodeOption(String), + /// A decoded dimension cannot be represented by this platform. + #[error("decoded dimensions cannot be represented by usize")] + DimensionOverflow, +} + +/// File formats recognized by the stable SpatialRust image-IO boundary. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum ImageFileFormat { + /// Portable Network Graphics. + Png, + /// JPEG/JFIF image. + Jpeg, + /// Portable anymap family (PBM/PGM/PPM/PAM). + Pnm, + /// Tagged Image File Format. + Tiff, + /// OpenEXR high-dynamic-range image. + OpenExr, +} + +impl std::fmt::Display for ImageFileFormat { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str(match self { + Self::Png => "png", + Self::Jpeg => "jpeg", + Self::Pnm => "pnm", + Self::Tiff => "tiff", + Self::OpenExr => "openexr", + }) + } +} + +impl ImageFileFormat { + /// Returns whether this codec is compiled into the current crate. + #[must_use] + pub const fn is_enabled(self) -> bool { + match self { + Self::Png => cfg!(feature = "png"), + Self::Jpeg => cfg!(feature = "jpeg"), + Self::Pnm => cfg!(feature = "pnm"), + Self::Tiff => cfg!(feature = "tiff"), + Self::OpenExr => cfg!(feature = "openexr"), + } + } + + fn from_backend(format: ImageFormat) -> Result { + match format { + ImageFormat::Png => Ok(Self::Png), + ImageFormat::Jpeg => Ok(Self::Jpeg), + ImageFormat::Pnm => Ok(Self::Pnm), + ImageFormat::Tiff => Ok(Self::Tiff), + ImageFormat::OpenExr => Ok(Self::OpenExr), + other => Err(ImageIoError::UnsupportedFormat(format!("{other:?}"))), + } + } + + fn require_enabled(self) -> Result<(), ImageIoError> { + if self.is_enabled() { + Ok(()) + } else { + Err(ImageIoError::FormatDisabled(self)) + } + } +} + +/// Strict application-level resource limits for one decode operation. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct DecodeLimits { + /// Maximum compressed input size in bytes. + pub max_input_bytes: usize, + /// Maximum decoded width. + pub max_width: u32, + /// Maximum decoded height. + pub max_height: u32, + /// Maximum decoded pixel count. + pub max_pixels: u64, + /// Best-effort maximum simultaneous allocation performed by the backend. + pub max_alloc_bytes: u64, +} + +impl Default for DecodeLimits { + fn default() -> Self { + Self { + max_input_bytes: 256 * 1024 * 1024, + max_width: 32_768, + max_height: 32_768, + max_pixels: 100_000_000, + max_alloc_bytes: 512 * 1024 * 1024, + } + } +} + +impl DecodeLimits { + fn validate_dimensions(self, width: u32, height: u32) -> Result<(), ImageIoError> { + let pixels = u64::from(width).saturating_mul(u64::from(height)); + if width > self.max_width || height > self.max_height || pixels > self.max_pixels { + return Err(ImageIoError::DimensionsTooLarge { width, height }); + } + Ok(()) + } + + fn backend(self) -> image::io::Limits { + let mut limits = image::io::Limits::default(); + limits.max_image_width = Some(self.max_width); + limits.max_image_height = Some(self.max_height); + limits.max_alloc = Some(self.max_alloc_bytes); + limits + } +} + +/// Options controlling one decode operation. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct DecodeOptions { + /// Resource limits applied before and during decoding. + pub limits: DecodeLimits, + /// Apply Exif orientation to produce canonical top-left pixels. + pub apply_orientation: bool, +} + +impl Default for DecodeOptions { + fn default() -> Self { + Self { limits: DecodeLimits::default(), apply_orientation: true } + } +} + +/// Exif image orientation in encoded-pixel coordinates. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +#[repr(u8)] +pub enum Orientation { + /// No orientation tag was found. + #[default] + Unspecified = 0, + /// Top-left origin; no transform. + Normal = 1, + /// Mirror left-to-right. + FlipHorizontal = 2, + /// Rotate 180 degrees. + Rotate180 = 3, + /// Mirror top-to-bottom. + FlipVertical = 4, + /// Reflect across the top-left to bottom-right diagonal. + Transpose = 5, + /// Rotate 90 degrees clockwise. + Rotate90 = 6, + /// Reflect across the top-right to bottom-left diagonal. + Transverse = 7, + /// Rotate 270 degrees clockwise. + Rotate270 = 8, +} + +impl Orientation { + #[cfg(feature = "exif")] + fn from_exif(value: u32) -> Self { + match value { + 1 => Self::Normal, + 2 => Self::FlipHorizontal, + 3 => Self::Rotate180, + 4 => Self::FlipVertical, + 5 => Self::Transpose, + 6 => Self::Rotate90, + 7 => Self::Transverse, + 8 => Self::Rotate270, + _ => Self::Unspecified, + } + } + + fn changes_pixels(self) -> bool { + !matches!(self, Self::Unspecified | Self::Normal) + } +} + +/// Encoded sample/channel type before conversion into SpatialRust storage. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum SourceColorType { + /// 8-bit grayscale. + Gray8, + /// 8-bit grayscale plus alpha. + GrayAlpha8, + /// 8-bit RGB. + Rgb8, + /// 8-bit RGBA. + Rgba8, + /// 16-bit grayscale. + Gray16, + /// 16-bit grayscale plus alpha. + GrayAlpha16, + /// 16-bit RGB. + Rgb16, + /// 16-bit RGBA. + Rgba16, + /// 32-bit floating-point RGB. + Rgb32Float, + /// 32-bit floating-point RGBA. + Rgba32Float, +} + +impl SourceColorType { + fn from_backend(color: image::ColorType) -> Result { + match color { + image::ColorType::L8 => Ok(Self::Gray8), + image::ColorType::La8 => Ok(Self::GrayAlpha8), + image::ColorType::Rgb8 => Ok(Self::Rgb8), + image::ColorType::Rgba8 => Ok(Self::Rgba8), + image::ColorType::L16 => Ok(Self::Gray16), + image::ColorType::La16 => Ok(Self::GrayAlpha16), + image::ColorType::Rgb16 => Ok(Self::Rgb16), + image::ColorType::Rgba16 => Ok(Self::Rgba16), + image::ColorType::Rgb32F => Ok(Self::Rgb32Float), + image::ColorType::Rgba32F => Ok(Self::Rgba32Float), + other => Err(ImageIoError::UnsupportedFormat(format!( + "unsupported decoded color type {other:?}" + ))), + } + } +} + +/// Owned typed pixels produced by a decoder or accepted by an encoder. +#[derive(Clone, Debug, PartialEq)] +pub enum DecodedPixels { + /// One-channel 8-bit grayscale. + Gray8(Image), + /// Two-channel 8-bit grayscale and alpha. + GrayAlpha8(Image), + /// Three-channel 8-bit RGB. + Rgb8(Image), + /// Four-channel 8-bit RGBA. + Rgba8(Image), + /// One-channel 16-bit grayscale. + Gray16(Image), + /// Two-channel 16-bit grayscale and alpha. + GrayAlpha16(Image), + /// Three-channel 16-bit RGB. + Rgb16(Image), + /// Four-channel 16-bit RGBA. + Rgba16(Image), + /// Three-channel linear-light floating-point RGB. + Rgb32Float(Image), + /// Four-channel linear-light floating-point RGBA. + Rgba32Float(Image), +} + +impl DecodedPixels { + /// Returns pixel width. + #[must_use] + pub fn width(&self) -> usize { + match self { + Self::Gray8(image) => image.width(), + Self::GrayAlpha8(image) => image.width(), + Self::Rgb8(image) => image.width(), + Self::Rgba8(image) => image.width(), + Self::Gray16(image) => image.width(), + Self::GrayAlpha16(image) => image.width(), + Self::Rgb16(image) => image.width(), + Self::Rgba16(image) => image.width(), + Self::Rgb32Float(image) => image.width(), + Self::Rgba32Float(image) => image.width(), + } + } + + /// Returns pixel height. + #[must_use] + pub fn height(&self) -> usize { + match self { + Self::Gray8(image) => image.height(), + Self::GrayAlpha8(image) => image.height(), + Self::Rgb8(image) => image.height(), + Self::Rgba8(image) => image.height(), + Self::Gray16(image) => image.height(), + Self::GrayAlpha16(image) => image.height(), + Self::Rgb16(image) => image.height(), + Self::Rgba16(image) => image.height(), + Self::Rgb32Float(image) => image.height(), + Self::Rgba32Float(image) => image.height(), + } + } + + fn to_dynamic(&self) -> Result { + let width = u32::try_from(self.width()).map_err(|_| ImageIoError::DimensionOverflow)?; + let height = u32::try_from(self.height()).map_err(|_| ImageIoError::DimensionOverflow)?; + macro_rules! buffer { + ($image:expr, $pixel:ty, $variant:path) => {{ + let typed = ImageBuffer::<$pixel, Vec<_>>::from_raw( + width, + height, + $image.as_slice().to_vec(), + ) + .ok_or(ImageIoError::DimensionOverflow)?; + $variant(typed) + }}; + } + Ok(match self { + Self::Gray8(image) => buffer!(image, image::Luma, DynamicImage::ImageLuma8), + Self::GrayAlpha8(image) => { + buffer!(image, image::LumaA, DynamicImage::ImageLumaA8) + } + Self::Rgb8(image) => buffer!(image, image::Rgb, DynamicImage::ImageRgb8), + Self::Rgba8(image) => buffer!(image, image::Rgba, DynamicImage::ImageRgba8), + Self::Gray16(image) => buffer!(image, image::Luma, DynamicImage::ImageLuma16), + Self::GrayAlpha16(image) => { + buffer!(image, image::LumaA, DynamicImage::ImageLumaA16) + } + Self::Rgb16(image) => buffer!(image, image::Rgb, DynamicImage::ImageRgb16), + Self::Rgba16(image) => buffer!(image, image::Rgba, DynamicImage::ImageRgba16), + Self::Rgb32Float(image) => { + buffer!(image, image::Rgb, DynamicImage::ImageRgb32F) + } + Self::Rgba32Float(image) => { + buffer!(image, image::Rgba, DynamicImage::ImageRgba32F) + } + }) + } +} + +/// Metadata describing encoded pixels and orientation handling. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct DecodedMetadata { + /// Detected container/codec format. + pub format: ImageFileFormat, + /// Sample and channel representation reported by the decoder. + pub source_color_type: SourceColorType, + /// Orientation tag found in the encoded container. + pub orientation: Orientation, + /// Whether a non-identity orientation was applied to the returned pixels. + pub orientation_applied: bool, +} + +/// One decoded image with owned typed pixels and source metadata. +#[derive(Clone, Debug, PartialEq)] +pub struct DecodedImage { + pixels: DecodedPixels, + metadata: DecodedMetadata, +} + +impl DecodedImage { + /// Borrows decoded pixels. + #[must_use] + pub fn pixels(&self) -> &DecodedPixels { + &self.pixels + } + + /// Returns source metadata. + #[must_use] + pub const fn metadata(&self) -> DecodedMetadata { + self.metadata + } + + /// Consumes the wrapper and returns pixels. + #[must_use] + pub fn into_pixels(self) -> DecodedPixels { + self.pixels + } + + /// Returns decoded width after orientation handling. + #[must_use] + pub fn width(&self) -> usize { + self.pixels.width() + } + + /// Returns decoded height after orientation handling. + #[must_use] + pub fn height(&self) -> usize { + self.pixels.height() + } +} + +/// Options controlling encoded output. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct EncodeOptions { + /// Output codec/container. + pub format: ImageFileFormat, + /// JPEG quality in the inclusive range 1–100. + pub jpeg_quality: u8, +} + +impl EncodeOptions { + /// Creates options for one format with JPEG quality 90. + #[must_use] + pub const fn new(format: ImageFileFormat) -> Self { + Self { format, jpeg_quality: 90 } + } + + fn output_format(self) -> Result { + self.format.require_enabled()?; + if self.jpeg_quality == 0 || self.jpeg_quality > 100 { + return Err(ImageIoError::InvalidEncodeOption( + "jpeg_quality must be in 1..=100".to_owned(), + )); + } + match self.format { + #[cfg(feature = "png")] + ImageFileFormat::Png => Ok(ImageOutputFormat::Png), + #[cfg(not(feature = "png"))] + ImageFileFormat::Png => Err(ImageIoError::FormatDisabled(self.format)), + #[cfg(feature = "jpeg")] + ImageFileFormat::Jpeg => Ok(ImageOutputFormat::Jpeg(self.jpeg_quality)), + #[cfg(not(feature = "jpeg"))] + ImageFileFormat::Jpeg => Err(ImageIoError::FormatDisabled(self.format)), + #[cfg(feature = "pnm")] + ImageFileFormat::Pnm => { + Ok(ImageOutputFormat::Pnm(image::codecs::pnm::PnmSubtype::ArbitraryMap)) + } + #[cfg(not(feature = "pnm"))] + ImageFileFormat::Pnm => Err(ImageIoError::FormatDisabled(self.format)), + #[cfg(feature = "tiff")] + ImageFileFormat::Tiff => Ok(ImageOutputFormat::Tiff), + #[cfg(not(feature = "tiff"))] + ImageFileFormat::Tiff => Err(ImageIoError::FormatDisabled(self.format)), + #[cfg(feature = "openexr")] + ImageFileFormat::OpenExr => Ok(ImageOutputFormat::OpenExr), + #[cfg(not(feature = "openexr"))] + ImageFileFormat::OpenExr => Err(ImageIoError::FormatDisabled(self.format)), + } + } +} + +/// Decodes an image from an encoded byte slice. +pub fn decode_bytes(bytes: &[u8], options: DecodeOptions) -> Result { + if bytes.len() > options.limits.max_input_bytes { + return Err(ImageIoError::InputTooLarge { maximum: options.limits.max_input_bytes }); + } + let backend_format = image::guess_format(bytes)?; + let format = ImageFileFormat::from_backend(backend_format)?; + format.require_enabled()?; + + let dimensions = + image::io::Reader::with_format(Cursor::new(bytes), backend_format).into_dimensions()?; + options.limits.validate_dimensions(dimensions.0, dimensions.1)?; + + let orientation = read_orientation(bytes); + let mut reader = image::io::Reader::with_format(Cursor::new(bytes), backend_format); + reader.limits(options.limits.backend()); + let dynamic = reader.decode()?; + let source_color_type = SourceColorType::from_backend(dynamic.color())?; + let orientation_applied = options.apply_orientation && orientation.changes_pixels(); + let dynamic = + if orientation_applied { apply_orientation(dynamic, orientation) } else { dynamic }; + let pixels = dynamic_to_pixels(dynamic)?; + Ok(DecodedImage { + pixels, + metadata: DecodedMetadata { format, source_color_type, orientation, orientation_applied }, + }) +} + +/// Reads a bounded encoded stream and decodes it. +pub fn decode_reader( + reader: R, + options: DecodeOptions, +) -> Result { + let maximum = options.limits.max_input_bytes; + let take_limit = u64::try_from(maximum).unwrap_or(u64::MAX).saturating_add(1); + let mut bytes = Vec::new(); + reader.take(take_limit).read_to_end(&mut bytes)?; + if bytes.len() > maximum { + return Err(ImageIoError::InputTooLarge { maximum }); + } + decode_bytes(&bytes, options) +} + +/// Opens and decodes one image path using content-based format detection. +pub fn decode_path( + path: impl AsRef, + options: DecodeOptions, +) -> Result { + decode_reader(BufReader::new(File::open(path)?), options) +} + +/// Encodes typed pixels to a seekable writer. +/// +/// The codec backend receives a temporary owned CPU buffer. No device transfer +/// occurs, and the source image remains unchanged. +pub fn encode_writer( + writer: &mut W, + pixels: &DecodedPixels, + options: EncodeOptions, +) -> Result<(), ImageIoError> { + let dynamic = pixels.to_dynamic()?; + dynamic.write_to(writer, options.output_format()?)?; + Ok(()) +} + +/// Encodes typed pixels into an owned byte vector. +pub fn encode_bytes( + pixels: &DecodedPixels, + options: EncodeOptions, +) -> Result, ImageIoError> { + let mut cursor = Cursor::new(Vec::new()); + encode_writer(&mut cursor, pixels, options)?; + Ok(cursor.into_inner()) +} + +/// Creates or truncates a path and encodes typed pixels into it. +pub fn encode_path( + path: impl AsRef, + pixels: &DecodedPixels, + options: EncodeOptions, +) -> Result<(), ImageIoError> { + let mut writer = BufWriter::new(File::create(path)?); + encode_writer(&mut writer, pixels, options)?; + writer.flush()?; + Ok(()) +} + +fn metadata(color_space: ColorSpace, alpha_mode: AlphaMode, floating: bool) -> ImageMetadata { + ImageMetadata { + color_space, + color_range: if floating { ColorRange::Unspecified } else { ColorRange::Full }, + alpha_mode, + } +} + +fn dimensions(dynamic: &DynamicImage) -> Result<(usize, usize), ImageIoError> { + let (width, height) = dynamic.dimensions(); + Ok(( + usize::try_from(width).map_err(|_| ImageIoError::DimensionOverflow)?, + usize::try_from(height).map_err(|_| ImageIoError::DimensionOverflow)?, + )) +} + +fn dynamic_to_pixels(dynamic: DynamicImage) -> Result { + let (width, height) = dimensions(&dynamic)?; + Ok(match dynamic { + DynamicImage::ImageLuma8(image) => DecodedPixels::Gray8(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::Gray, AlphaMode::None, false), + )?), + DynamicImage::ImageLumaA8(image) => { + DecodedPixels::GrayAlpha8(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::Unknown, AlphaMode::Straight, false), + )?) + } + DynamicImage::ImageRgb8(image) => DecodedPixels::Rgb8(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::Rgb, AlphaMode::None, false), + )?), + DynamicImage::ImageRgba8(image) => DecodedPixels::Rgba8(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::Rgba, AlphaMode::Straight, false), + )?), + DynamicImage::ImageLuma16(image) => DecodedPixels::Gray16(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::Gray, AlphaMode::None, false), + )?), + DynamicImage::ImageLumaA16(image) => { + DecodedPixels::GrayAlpha16(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::Unknown, AlphaMode::Straight, false), + )?) + } + DynamicImage::ImageRgb16(image) => DecodedPixels::Rgb16(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::Rgb, AlphaMode::None, false), + )?), + DynamicImage::ImageRgba16(image) => DecodedPixels::Rgba16(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::Rgba, AlphaMode::Straight, false), + )?), + DynamicImage::ImageRgb32F(image) => { + DecodedPixels::Rgb32Float(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::LinearRgb, AlphaMode::None, true), + )?) + } + DynamicImage::ImageRgba32F(image) => { + DecodedPixels::Rgba32Float(Image::try_new_with_metadata( + width, + height, + image.into_raw(), + metadata(ColorSpace::Unknown, AlphaMode::Straight, true), + )?) + } + _ => { + return Err(ImageIoError::UnsupportedFormat( + "decoder returned an unsupported dynamic image variant".to_owned(), + )) + } + }) +} + +fn apply_orientation(image: DynamicImage, orientation: Orientation) -> DynamicImage { + match orientation { + Orientation::Unspecified | Orientation::Normal => image, + Orientation::FlipHorizontal => image.fliph(), + Orientation::Rotate180 => image.rotate180(), + Orientation::FlipVertical => image.flipv(), + Orientation::Transpose => image.rotate90().fliph(), + Orientation::Rotate90 => image.rotate90(), + Orientation::Transverse => image.rotate90().flipv(), + Orientation::Rotate270 => image.rotate270(), + } +} + +#[cfg(feature = "exif")] +fn read_orientation(bytes: &[u8]) -> Orientation { + use exif::{In, Reader, Tag}; + + let mut cursor = Cursor::new(bytes); + Reader::new() + .read_from_container(&mut cursor) + .ok() + .and_then(|exif| { + exif.get_field(Tag::Orientation, In::PRIMARY).and_then(|field| field.value.get_uint(0)) + }) + .map(Orientation::from_exif) + .unwrap_or(Orientation::Unspecified) +} + +#[cfg(not(feature = "exif"))] +fn read_orientation(_bytes: &[u8]) -> Orientation { + Orientation::Unspecified +} + +#[cfg(test)] +mod tests { + use super::*; + use proptest::prelude::*; + + #[cfg(any(feature = "png", feature = "jpeg", feature = "pnm"))] + fn rgb_fixture() -> DecodedPixels { + DecodedPixels::Rgb8( + Image::try_new_with_metadata( + 3, + 2, + vec![255, 0, 0, 0, 255, 0, 0, 0, 255, 10, 20, 30, 40, 50, 60, 70, 80, 90], + metadata(ColorSpace::Rgb, AlphaMode::None, false), + ) + .unwrap(), + ) + } + + #[cfg(feature = "png")] + #[test] + fn png_memory_roundtrip_is_exact() { + let source = rgb_fixture(); + let bytes = encode_bytes(&source, EncodeOptions::new(ImageFileFormat::Png)).unwrap(); + let decoded = decode_bytes(&bytes, DecodeOptions::default()).unwrap(); + assert_eq!(decoded.pixels(), &source); + assert_eq!(decoded.metadata().format, ImageFileFormat::Png); + assert_eq!(decoded.metadata().source_color_type, SourceColorType::Rgb8); + } + + #[cfg(feature = "pnm")] + #[test] + fn pnm_reader_roundtrip_is_exact() { + let source = rgb_fixture(); + let bytes = encode_bytes(&source, EncodeOptions::new(ImageFileFormat::Pnm)).unwrap(); + let decoded = decode_reader(Cursor::new(bytes), DecodeOptions::default()).unwrap(); + assert_eq!(decoded.pixels(), &source); + } + + #[cfg(feature = "jpeg")] + #[test] + fn jpeg_roundtrip_preserves_shape_and_type() { + let bytes = + encode_bytes(&rgb_fixture(), EncodeOptions::new(ImageFileFormat::Jpeg)).unwrap(); + let decoded = decode_bytes(&bytes, DecodeOptions::default()).unwrap(); + assert_eq!((decoded.width(), decoded.height()), (3, 2)); + assert!(matches!(decoded.pixels(), DecodedPixels::Rgb8(_))); + } + + #[cfg(feature = "png")] + #[test] + fn dimensions_and_input_are_bounded_before_decode() { + let bytes = encode_bytes(&rgb_fixture(), EncodeOptions::new(ImageFileFormat::Png)).unwrap(); + let mut options = DecodeOptions::default(); + options.limits.max_width = 2; + assert!(matches!( + decode_bytes(&bytes, options), + Err(ImageIoError::DimensionsTooLarge { .. }) | Err(ImageIoError::Codec(_)) + )); + + let mut options = DecodeOptions::default(); + options.limits.max_input_bytes = bytes.len() - 1; + assert!(matches!( + decode_reader(Cursor::new(bytes), options), + Err(ImageIoError::InputTooLarge { .. }) + )); + } + + #[test] + fn rotation_and_diagonal_reflection_have_expected_layout() { + let image = image::GrayImage::from_raw(2, 3, vec![1, 2, 3, 4, 5, 6]).unwrap(); + let rotated = + apply_orientation(DynamicImage::ImageLuma8(image.clone()), Orientation::Rotate90); + assert_eq!(rotated.dimensions(), (3, 2)); + assert_eq!(rotated.to_luma8().into_raw(), vec![5, 3, 1, 6, 4, 2]); + + let transposed = apply_orientation(DynamicImage::ImageLuma8(image), Orientation::Transpose); + assert_eq!(transposed.dimensions(), (3, 2)); + assert_eq!(transposed.to_luma8().into_raw(), vec![1, 3, 5, 2, 4, 6]); + } + + #[cfg(feature = "exif")] + #[test] + fn reads_orientation_from_minimal_tiff_exif() { + let bytes = [ + b'I', b'I', 42, 0, 8, 0, 0, 0, // TIFF header and first IFD offset + 1, 0, // one entry + 0x12, 0x01, // Orientation tag + 3, 0, // SHORT + 1, 0, 0, 0, // count + 6, 0, 0, 0, // Rotate 90 clockwise + 0, 0, 0, 0, // next IFD + ]; + assert_eq!(read_orientation(&bytes), Orientation::Rotate90); + } + + #[test] + fn rejects_malformed_input() { + assert!(decode_bytes(b"not an image", DecodeOptions::default()).is_err()); + } + + #[test] + fn disabled_format_reports_feature_boundary() { + if !ImageFileFormat::Tiff.is_enabled() { + assert!(matches!( + EncodeOptions::new(ImageFileFormat::Tiff).output_format(), + Err(ImageIoError::FormatDisabled(ImageFileFormat::Tiff)) + )); + } + } + + proptest! { + #[test] + fn arbitrary_small_input_never_panics(bytes in proptest::collection::vec(any::(), 0..4096)) { + let mut options = DecodeOptions::default(); + options.limits.max_input_bytes = 4096; + options.limits.max_width = 256; + options.limits.max_height = 256; + options.limits.max_pixels = 65_536; + options.limits.max_alloc_bytes = 4 * 1024 * 1024; + let _ = decode_bytes(&bytes, options); + } + } + + #[cfg(feature = "png")] + #[test] + fn path_roundtrip_uses_content_detection() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("image-without-extension"); + let source = rgb_fixture(); + encode_path(&path, &source, EncodeOptions::new(ImageFileFormat::Png)).unwrap(); + let decoded = decode_path(path, DecodeOptions::default()).unwrap(); + assert_eq!(decoded.pixels(), &source); + } + + #[cfg(feature = "tiff")] + #[test] + fn tiff_preserves_sixteen_bit_grayscale() { + let source = DecodedPixels::Gray16( + Image::try_new_with_metadata( + 3, + 1, + vec![0, 1024, u16::MAX], + metadata(ColorSpace::Gray, AlphaMode::None, false), + ) + .unwrap(), + ); + let bytes = encode_bytes(&source, EncodeOptions::new(ImageFileFormat::Tiff)).unwrap(); + let decoded = decode_bytes(&bytes, DecodeOptions::default()).unwrap(); + assert_eq!(decoded.pixels(), &source); + } + + #[cfg(feature = "openexr")] + #[test] + fn openexr_preserves_float_rgb() { + let source = DecodedPixels::Rgb32Float( + Image::try_new_with_metadata( + 2, + 1, + vec![0.0, 0.25, 1.0, 4.0, -0.5, 2.0], + metadata(ColorSpace::LinearRgb, AlphaMode::None, true), + ) + .unwrap(), + ); + let bytes = encode_bytes(&source, EncodeOptions::new(ImageFileFormat::OpenExr)).unwrap(); + let decoded = decode_bytes(&bytes, DecodeOptions::default()).unwrap(); + assert_eq!(decoded.pixels(), &source); + } +} diff --git a/crates/spatialrust-math/src/mat.rs b/crates/spatialrust-math/src/mat.rs index 43918f7..1fc8350 100644 --- a/crates/spatialrust-math/src/mat.rs +++ b/crates/spatialrust-math/src/mat.rs @@ -118,6 +118,46 @@ impl Mat3 { self.m[2][0] * v.x + self.m[2][1] * v.y + self.m[2][2] * v.z, ) } + + /// Matrix multiplication. + #[must_use] + pub fn mul_mat3(self, other: Self) -> Self { + Self::from_rows( + [ + self.m[0][0] * other.m[0][0] + + self.m[0][1] * other.m[1][0] + + self.m[0][2] * other.m[2][0], + self.m[0][0] * other.m[0][1] + + self.m[0][1] * other.m[1][1] + + self.m[0][2] * other.m[2][1], + self.m[0][0] * other.m[0][2] + + self.m[0][1] * other.m[1][2] + + self.m[0][2] * other.m[2][2], + ], + [ + self.m[1][0] * other.m[0][0] + + self.m[1][1] * other.m[1][0] + + self.m[1][2] * other.m[2][0], + self.m[1][0] * other.m[0][1] + + self.m[1][1] * other.m[1][1] + + self.m[1][2] * other.m[2][1], + self.m[1][0] * other.m[0][2] + + self.m[1][1] * other.m[1][2] + + self.m[1][2] * other.m[2][2], + ], + [ + self.m[2][0] * other.m[0][0] + + self.m[2][1] * other.m[1][0] + + self.m[2][2] * other.m[2][0], + self.m[2][0] * other.m[0][1] + + self.m[2][1] * other.m[1][1] + + self.m[2][2] * other.m[2][1], + self.m[2][0] * other.m[0][2] + + self.m[2][1] * other.m[1][2] + + self.m[2][2] * other.m[2][2], + ], + ) + } } impl Mat4 { diff --git a/crates/spatialrust-py/Cargo.toml b/crates/spatialrust-py/Cargo.toml index e0c1838..674ac56 100644 --- a/crates/spatialrust-py/Cargo.toml +++ b/crates/spatialrust-py/Cargo.toml @@ -9,6 +9,10 @@ authors = ["SpatialRust Contributors"] repository = "https://github.com/rsasaki0109/SpatialRust" description = "Python bindings for SpatialRust point cloud processing" +[features] +default = [] +onnxruntime = ["spatialrust/ai-onnxruntime"] + [lib] # Importable module name: `import spatialrust` name = "spatialrust" @@ -42,6 +46,8 @@ spatialrust = { path = "../spatialrust", features = [ "register-fpfh", "camera-rgbd", "vision-full", + "image-io-standard", + "tensor-dlpack", ] } # Keep this crate out of the main Rust workspace so `cargo test --workspace` diff --git a/crates/spatialrust-py/README.md b/crates/spatialrust-py/README.md index 0752a40..91777f9 100644 --- a/crates/spatialrust-py/README.md +++ b/crates/spatialrust-py/README.md @@ -24,6 +24,26 @@ maturin develop --release # builds the Rust extension into the venv maturin build --release --out dist ``` +ONNX Runtime is intentionally not part of the default wheel. Enable its CPU +backend explicitly when building an inference wheel: + +```bash +maturin develop --release --features onnxruntime +``` + +`OnnxRuntimeSession.run()` uses named CPU I/O Binding by default. Pass +`copy=True` only when an explicit host conversion is acceptable. + +Feature2D is available in the default wheel. `harris_keypoints`, +`shi_tomasi_keypoints`, and `fast_keypoints` return immutable keypoint metadata; +`orb_features` returns those keypoints with a `uint8[N, 32]` descriptor matrix. +Use `match_binary_descriptors` for Hamming distance or +`match_float_descriptors` for Euclidean distance, with optional ratio, +cross-check, and maximum-distance filters. + +Geometry bindings expose `estimate_homography_ransac`, `solve_pnp`, and +`stereo_block_match` for NumPy `float64` / grayscale workflows. + ## Test The bindings have a pytest suite (`tests/`) that exercises the NumPy ⇄ Rust diff --git a/crates/spatialrust-py/spatialrust.pyi b/crates/spatialrust-py/spatialrust.pyi index 68f2e24..b9af992 100644 --- a/crates/spatialrust-py/spatialrust.pyi +++ b/crates/spatialrust-py/spatialrust.pyi @@ -11,13 +11,68 @@ import numpy as np from numpy.typing import NDArray __version__: str +__all__: list[str] = [ + "__version__", "ImageMetadata", "Tensor", "Keypoint2", "OnnxRuntimeSession", + "DLPackTensorView", "PointCloud", "PipelineResult", "RegionResult", + "DbscanResult", "GroundResult", "MultiPlaneResult", "SphereResult", + "CylinderResult", "RegistrationResult", "read_image", + "tensor_copy_from_numpy", "tensor_view_from_dlpack", "harris_keypoints", + "shi_tomasi_keypoints", "fast_keypoints", "orb_features", + "estimate_homography_ransac", "solve_pnp", "stereo_block_match", + "match_binary_descriptors", "match_float_descriptors", "write_image", "read", + "write", "voxel_downsample", "crop_box", "pass_through", "iss_keypoints", + "orient_normals", "detect_boundary", "mls_smooth", "farthest_point_sampling", + "statistical_outlier_removal", "radius_outlier_removal", "run_pipeline", + "region_growing", "dbscan", "ground_segmentation", "segment_multi_plane", + "ransac_sphere", "ransac_cylinder", "chamfer_distance", "hausdorff_distance", + "apply_transform", "recenter", "scale", "normalize_unit_sphere", "merge", + "centroid", "bounding_box", "oriented_bounding_box", "voxelize", "range_image", + "rgbd_to_point_cloud", "filter2d_image", "gaussian_blur_image", + "median_blur_image", "bilateral_filter_image", "sobel_image", "scharr_image", + "laplacian_image", "pyr_down_image", "pyr_up_image", "morphology_image", + "threshold_image", "otsu_threshold_image", "adaptive_threshold_image", + "histogram_image", "equalize_histogram_image", "clahe_image", + "integral_image_u8", "canny_image", "resize_image", "letterbox_image", + "normalize_image_chw", "rgb_to_gray_image", "rgb_to_hsv_image", "remap_image", + "nms", "soft_nms", "connected_components_image", "find_mask_contours", + "encode_mask_rle", "decode_mask_rle", "point_map_to_point_cloud", "knn_graph", + "radius_graph", "register_icp", "register_point_to_plane", "register_gicp", + "register_ndt", "register_fpfh_ransac", "register_fpfh_keypoints", +] # Convenient aliases for the array shapes the bindings exchange. _F32Array = NDArray[np.float32] # positions, grids, range images, transforms +_F64Array = NDArray[np.float64] +_BoolArray = NDArray[np.bool_] _I32Array = NDArray[np.int32] # labels, edge_index _U32Array = NDArray[np.uint32] _Vec3 = tuple[float, float, float] _U8Array = NDArray[np.uint8] +_U16Array = NDArray[np.uint16] + +@final +class ImageMetadata: + """Container, sample type, and Exif orientation from image decoding.""" + + @property + def format(self) -> str: ... + @property + def color_type(self) -> str: ... + @property + def orientation(self) -> int: ... + @property + def orientation_applied(self) -> bool: ... + def __repr__(self) -> str: ... + +def read_image( + path: str, apply_orientation: bool = ... +) -> tuple[NDArray[np.uint8] | NDArray[np.uint16] | NDArray[np.float32], ImageMetadata]: ... +def write_image( + path: str, + image: NDArray[np.uint8] | NDArray[np.uint16], + format: str, + jpeg_quality: int = ..., +) -> None: ... @final class PointCloud: @@ -61,6 +116,87 @@ def rgbd_to_point_cloud( # --------------------------------------------------------------------------- # # Image preprocessing, detection, masks, and dense spatial data # --------------------------------------------------------------------------- # +def filter2d_image( + image: _U8Array, kernel: NDArray[np.float64], delta: float = ... +) -> _U8Array: ... +def gaussian_blur_image( + image: _U8Array, + kernel_width: int, + kernel_height: int, + sigma_x: float, + sigma_y: Optional[float] = ..., +) -> _U8Array: ... +def median_blur_image(image: _U8Array, kernel_size: int) -> _U8Array: ... +def bilateral_filter_image( + image: _U8Array, + diameter: int, + sigma_color: float, + sigma_space: float, +) -> _U8Array: ... +def sobel_image( + image: _U8Array, + dx: int, + dy: int, + kernel_size: int = ..., + scale: float = ..., + delta: float = ..., +) -> _F32Array: ... +def scharr_image( + image: _U8Array, + dx: int, + dy: int, + scale: float = ..., + delta: float = ..., +) -> _F32Array: ... +def laplacian_image( + image: _U8Array, + kernel_size: int = ..., + scale: float = ..., + delta: float = ..., +) -> _F32Array: ... +def pyr_down_image(image: _U8Array) -> _U8Array: ... +def pyr_up_image(image: _U8Array) -> _U8Array: ... +def morphology_image( + image: _U8Array, + operation: str, + kernel_width: int, + kernel_height: int, + shape: str = ..., + iterations: int = ..., +) -> _U8Array: ... +def threshold_image( + image: _U8Array, + threshold: float, + max_value: int = ..., + threshold_type: str = ..., +) -> _U8Array: ... +def otsu_threshold_image( + image: _U8Array, max_value: int = ..., threshold_type: str = ... +) -> tuple[int, _U8Array]: ... +def adaptive_threshold_image( + image: _U8Array, + block_size: int, + c: float, + method: str = ..., + max_value: int = ..., + threshold_type: str = ..., +) -> _U8Array: ... +def histogram_image(image: _U8Array) -> NDArray[np.uint64]: ... +def equalize_histogram_image(image: _U8Array) -> _U8Array: ... +def clahe_image( + image: _U8Array, + clip_limit: float = ..., + tiles_x: int = ..., + tiles_y: int = ..., +) -> _U8Array: ... +def integral_image_u8(image: _U8Array) -> NDArray[np.float64]: ... +def canny_image( + image: _U8Array, + low_threshold: float, + high_threshold: float, + aperture_size: int = ..., + l2_gradient: bool = ..., +) -> _U8Array: ... def resize_image( image: _U8Array, width: int, @@ -434,3 +570,138 @@ def register_fpfh_keypoints( ransac_iterations: int = ..., k_neighbors: int = ..., ) -> RegistrationResult: ... +@final +class Tensor: + @property + def shape(self) -> list[int]: ... + @property + def dtype(self) -> str: ... + def __dlpack_device__(self) -> tuple[int, int]: ... + def __dlpack__( + self, + stream: object | None = ..., + *, + max_version: tuple[int, int] | None = ..., + dl_device: tuple[int, int] | None = ..., + copy: bool | None = ..., + ) -> object: ... + def copy(self) -> Tensor: ... + +@final +class Keypoint2: + @property + def x(self) -> float: ... + @property + def y(self) -> float: ... + @property + def size(self) -> float: ... + @property + def angle_degrees(self) -> float | None: ... + @property + def response(self) -> float: ... + @property + def octave(self) -> int: ... + @property + def class_id(self) -> int | None: ... + +def harris_keypoints( + image: _U8Array, + max_corners: int = ..., + quality_level: float = ..., + min_distance: float = ..., + block_size: int = ..., + gradient_size: int = ..., + k: float = ..., +) -> list[Keypoint2]: ... +def shi_tomasi_keypoints( + image: _U8Array, + max_corners: int = ..., + quality_level: float = ..., + min_distance: float = ..., + block_size: int = ..., + gradient_size: int = ..., +) -> list[Keypoint2]: ... +def fast_keypoints( + image: _U8Array, + threshold: int = ..., + nonmax_suppression: bool = ..., +) -> list[Keypoint2]: ... +def orb_features( + image: _U8Array, + max_features: int = ..., + scale_factor: float = ..., + levels: int = ..., + edge_threshold: int = ..., + fast_threshold: int = ..., + patch_size: int = ..., + score_type: str = ..., +) -> tuple[list[Keypoint2], _U8Array]: ... +def estimate_homography_ransac( + source: _F64Array, + target: _F64Array, + threshold: float = ..., + confidence: float = ..., + max_iterations: int = ..., + seed: int = ..., +) -> tuple[_F64Array, _BoolArray, _F64Array]: ... +def solve_pnp( + object_points: _F64Array, + image_points: _F64Array, + fx: float, + fy: float, + cx: float, + cy: float, + width: int = ..., + height: int = ..., +) -> tuple[_F64Array, _F64Array]: ... +def stereo_block_match( + left: _U8Array, + right: _U8Array, + window_size: int = ..., + min_disparity: int = ..., + num_disparities: int = ..., + uniqueness_ratio: float = ..., +) -> _F32Array: ... +def match_binary_descriptors( + query: _U8Array, + train: _U8Array, + cross_check: bool = ..., + ratio: float | None = ..., + max_distance: float | None = ..., +) -> list[tuple[int, int, float]]: ... +def match_float_descriptors( + query: _F32Array, + train: _F32Array, + cross_check: bool = ..., + ratio: float | None = ..., + max_distance: float | None = ..., +) -> list[tuple[int, int, float]]: ... + +@final +class OnnxRuntimeSession: + def __new__( + cls, + path: str, + *, + intra_threads: int | None = ..., + inter_threads: int | None = ..., + deterministic: bool = ..., + ) -> OnnxRuntimeSession: ... + @property + def inputs(self) -> list[tuple[str, str, list[str]]]: ... + @property + def outputs(self) -> list[tuple[str, str, list[str]]]: ... + def run(self, inputs: dict[str, Tensor], *, copy: bool = ...) -> dict[str, Tensor]: ... + +def tensor_copy_from_numpy(array: NDArray[np.generic]) -> Tensor: ... +@final +class DLPackTensorView: + @property + def shape(self) -> list[int]: ... + @property + def dtype(self) -> str: ... + @property + def version(self) -> tuple[int, int]: ... + def copy(self) -> Tensor: ... + +def tensor_view_from_dlpack(producer: object) -> DLPackTensorView: ... diff --git a/crates/spatialrust-py/src/dlpack_capsule.rs b/crates/spatialrust-py/src/dlpack_capsule.rs new file mode 100644 index 0000000..d4c114e --- /dev/null +++ b/crates/spatialrust-py/src/dlpack_capsule.rs @@ -0,0 +1,66 @@ +//! Audited CPython capsule boundary for DLPack ownership transfer. + +use pyo3::{ + exceptions::PyBufferError, + prelude::*, + types::{PyAny, PyDict}, +}; +use spatialrust::tensor::{release_dlpack_raw, DlpackExport, DlpackImport, TensorBuffer}; + +const VERSIONED_NAME: &[u8] = b"dltensor_versioned\0"; +const USED_VERSIONED_NAME: &[u8] = b"used_dltensor_versioned\0"; + +unsafe extern "C" fn capsule_destructor(capsule: *mut pyo3::ffi::PyObject) { + let name = VERSIONED_NAME.as_ptr().cast(); + // SAFETY: CPython calls the destructor with the capsule object. A consumer + // renames consumed capsules, so only the original name retains ownership. + if unsafe { pyo3::ffi::PyCapsule_IsValid(capsule, name) } == 1 { + // SAFETY: validity above proves the name and capsule pointer contract. + let raw = unsafe { pyo3::ffi::PyCapsule_GetPointer(capsule, name) }; + if !raw.is_null() { + // SAFETY: the unconsumed capsule uniquely owns deleter responsibility. + unsafe { release_dlpack_raw(raw) }; + } + } +} + +pub(crate) fn export_tensor(py: Python<'_>, tensor: &TensorBuffer) -> PyResult> { + let export = DlpackExport::from_tensor(tensor) + .map_err(|error| PyBufferError::new_err(error.to_string()))?; + let raw = export.into_raw(); + // SAFETY: `raw` is live and the static nul-terminated name outlives the capsule. + let capsule = unsafe { + pyo3::ffi::PyCapsule_New(raw, VERSIONED_NAME.as_ptr().cast(), Some(capsule_destructor)) + }; + if capsule.is_null() { + // SAFETY: capsule construction failed, so ownership was not transferred. + unsafe { release_dlpack_raw(raw) }; + return Err(PyErr::fetch(py)); + } + // SAFETY: PyCapsule_New returned one new owned Python reference. + Ok(unsafe { Py::from_owned_ptr(py, capsule) }) +} + +pub(crate) fn import_tensor(producer: &Bound<'_, PyAny>) -> PyResult { + let kwargs = PyDict::new_bound(producer.py()); + kwargs.set_item("max_version", (1, 0))?; + kwargs.set_item("copy", false)?; + let capsule = producer.call_method("__dlpack__", (), Some(&kwargs))?; + let name = VERSIONED_NAME.as_ptr().cast(); + // SAFETY: this only asks CPython to validate the exact capsule/name pair. + if unsafe { pyo3::ffi::PyCapsule_IsValid(capsule.as_ptr(), name) } != 1 { + return Err(PyBufferError::new_err("producer did not return a dltensor_versioned capsule")); + } + // SAFETY: capsule validity proves that the payload is non-null for this name. + let raw = unsafe { pyo3::ffi::PyCapsule_GetPointer(capsule.as_ptr(), name) }; + // SAFETY: both names are static nul-terminated strings and capsule is valid. + if unsafe { + pyo3::ffi::PyCapsule_SetName(capsule.as_ptr(), USED_VERSIONED_NAME.as_ptr().cast()) + } != 0 + { + return Err(PyErr::fetch(producer.py())); + } + // SAFETY: renaming transferred exclusive deleter ownership from the capsule. + unsafe { DlpackImport::from_raw(raw) } + .map_err(|error| PyBufferError::new_err(error.to_string())) +} diff --git a/crates/spatialrust-py/src/lib.rs b/crates/spatialrust-py/src/lib.rs index 8fd4d69..98620b8 100644 --- a/crates/spatialrust-py/src/lib.rs +++ b/crates/spatialrust-py/src/lib.rs @@ -6,13 +6,26 @@ // PyO3's `#[pyfunction]` expansion emits `.into()` on already-`PyErr` results. #![allow(clippy::useless_conversion)] +#![deny(unsafe_code)] + +#[allow(unsafe_code)] +mod dlpack_capsule; use numpy::ndarray::{Array2, Array3}; use numpy::{ - IntoPyArray, PyArray1, PyArray2, PyArray3, PyReadonlyArray1, PyReadonlyArray2, PyReadonlyArray3, + IntoPyArray, PyArray1, PyArray2, PyArray3, PyReadonlyArray1, PyReadonlyArray2, + PyReadonlyArray3, PyReadonlyArrayDyn, PyUntypedArrayMethods, }; -use pyo3::exceptions::PyValueError; +use pyo3::exceptions::{PyBufferError, PyRuntimeError, PyTypeError, PyValueError}; use pyo3::prelude::*; +use pyo3::types::PyDict; + +#[cfg(feature = "onnxruntime")] +use spatialrust::ai::{ + CopyPolicy as AiCopyPolicy, InferenceBackend, IoBinding as AiIoBinding, ModelSession, + ModelSource, NamedTensors as AiNamedTensors, OnnxRuntimeBackend, OutputBinding, + RunOptions as AiRunOptions, SessionOptions as AiSessionOptions, +}; use spatialrust::core::{PointBuffer, PointBufferSet, SpatialMetadata}; use spatialrust::features::{ @@ -26,7 +39,11 @@ use spatialrust::filtering::{ StatisticalOutlierConfig, StatisticalOutlierRemoval, VoxelGridDownsample, VoxelGridDownsampleConfig, }; -use spatialrust::math::Mat4; +use spatialrust::image_io::{ + decode_path as decode_image_path, encode_path as encode_image_path, DecodeOptions, + DecodedMetadata, DecodedPixels, EncodeOptions, ImageFileFormat, +}; +use spatialrust::math::{Mat3, Mat4, Vec2, Vec3}; use spatialrust::metrics::{chamfer_distance as chamfer, hausdorff_distance as hausdorff}; use spatialrust::pipeline::{MvpPipeline, MvpPipelineConfig}; use spatialrust::registration::{ @@ -39,19 +56,39 @@ use spatialrust::segmentation::{ MultiPlaneSegmenter, RansacCylinderSegmenter, RansacPrimitiveConfig, RansacSphereSegmenter, RegionGrowingConfig, RegionGrowingSegmenter, }; +use spatialrust::tensor::{ + DataType as TensorDataType, Device as TensorDevice, DlpackImport, TensorBuffer, + TensorDescriptor, +}; use spatialrust::transform::{ apply_transform as apply_tf, bounding_box as bbox, centroid as cloud_centroid, merge_clouds, normalize_unit_sphere as normalize_unit, oriented_bounding_box as obb, recenter as recenter_op, scale_cloud, }; use spatialrust::vision::{ - approximate_polygon as approximate_contour, connected_components as label_components, - decode_rle as decode_mask_runs, encode_rle as encode_mask_runs, - find_contours as trace_contours, letterbox as letterbox_op, nms as nms_op, - pack_chw as pack_chw_op, point_map_to_point_cloud as point_map_to_cloud, remap as remap_op, - resize as resize_op, rgb_to_gray as rgb_to_gray_op, rgb_to_hsv as rgb_to_hsv_op, - soft_nms as soft_nms_op, BinaryMask, BorderMode, BoundingBox2, ConfidenceMap, Connectivity, - Interpolation, MaskRle, PointMap, RleOrder, SoftNmsMethod, + adaptive_threshold as adaptive_threshold_op, approximate_polygon as approximate_contour, + bilateral_filter as bilateral_filter_op, canny as canny_op, clahe as clahe_op, + connected_components as label_components, decode_rle as decode_mask_runs, + detect_and_describe_orb as detect_and_describe_orb_op, detect_fast as detect_fast_op, + detect_harris as detect_harris_op, detect_shi_tomasi as detect_shi_tomasi_op, + encode_rle as encode_mask_runs, equalize_histogram as equalize_histogram_op, + estimate_homography_ransac as estimate_homography_ransac_op, filter2d as filter2d_op, + find_contours as trace_contours, gaussian_blur as gaussian_blur_op, + histogram_u8 as histogram_u8_op, integral_image as integral_image_op, + laplacian as laplacian_op, letterbox as letterbox_op, + match_descriptors as match_descriptors_op, median_blur as median_blur_op, + morphology_ex as morphology_ex_op, nms as nms_op, otsu_threshold_u8 as otsu_threshold_u8_op, + pack_chw as pack_chw_op, point_map_to_point_cloud as point_map_to_cloud, + pyr_down as pyr_down_op, pyr_up as pyr_up_op, remap as remap_op, resize as resize_op, + rgb_to_gray as rgb_to_gray_op, rgb_to_hsv as rgb_to_hsv_op, scharr as scharr_op, + sobel as sobel_op, soft_nms as soft_nms_op, solve_pnp as solve_pnp_op, + stereo_block_match as stereo_block_match_op, threshold as threshold_op, + AdaptiveThresholdMethod, BinaryMask, BorderMode, BoundingBox2, CameraMatrix3, CannyOptions, + ConfidenceMap, Connectivity, CornerSelectionOptions, DescriptorBuffer, FastOptions, + HarrisOptions, Interpolation, Kernel2D, Keypoint2, MaskRle, MatchOptions, + MorphologyOperation, MorphologyShape, ObjectImageCorrespondence, OrbOptions, OrbScoreType, + PointCorrespondence2, PointMap, RobustEstimationOptions, RleOrder, ShiTomasiOptions, + SoftNmsMethod, StereoBmOptions, StructuringElement, ThresholdType, AbsolutePose, }; use spatialrust::voxelize::{ range_image as range_image_proj, voxelize as voxelize_grid, RangeImageConfig, VoxelFill, @@ -77,6 +114,516 @@ fn to_py_err(err: E) -> PyErr { PyValueError::new_err(err.to_string()) } +#[pyclass(name = "Tensor")] +#[derive(Clone)] +struct PyTensor { + inner: TensorBuffer, +} + +#[pyclass(name = "Keypoint2", frozen)] +#[derive(Clone, Copy)] +struct PyKeypoint2 { + inner: Keypoint2, +} + +#[pymethods] +impl PyKeypoint2 { + #[getter] + fn x(&self) -> f32 { + self.inner.x() + } + + #[getter] + fn y(&self) -> f32 { + self.inner.y() + } + + #[getter] + fn size(&self) -> f32 { + self.inner.size() + } + + #[getter] + fn angle_degrees(&self) -> Option { + self.inner.angle_degrees() + } + + #[getter] + fn response(&self) -> f32 { + self.inner.response() + } + + #[getter] + fn octave(&self) -> i32 { + self.inner.octave() + } + + #[getter] + fn class_id(&self) -> Option { + self.inner.class_id() + } + + fn __repr__(&self) -> String { + format!( + "Keypoint2(x={}, y={}, size={}, angle_degrees={:?}, response={})", + self.x(), + self.y(), + self.size(), + self.angle_degrees(), + self.response() + ) + } +} + +#[pyclass(name = "OnnxRuntimeSession", unsendable)] +struct PyOnnxRuntimeSession { + #[cfg(feature = "onnxruntime")] + inner: Box, +} + +#[pymethods] +impl PyOnnxRuntimeSession { + #[new] + #[pyo3(signature = (path, *, intra_threads=None, inter_threads=None, deterministic=false))] + fn new( + path: String, + intra_threads: Option, + inter_threads: Option, + deterministic: bool, + ) -> PyResult { + #[cfg(feature = "onnxruntime")] + { + let options = AiSessionOptions { + intra_threads, + inter_threads, + deterministic, + ..AiSessionOptions::default() + }; + let inner = OnnxRuntimeBackend + .create_session(&ModelSource::Path(path.into()), &options) + .map_err(to_py_err)?; + Ok(Self { inner }) + } + #[cfg(not(feature = "onnxruntime"))] + { + let _ = (path, intra_threads, inter_threads, deterministic); + Err(PyRuntimeError::new_err( + "this SpatialRust Python module was built without the `onnxruntime` feature", + )) + } + } + + /// Returns `(name, dtype, dimensions)` metadata for model inputs. + #[getter] + fn inputs(&self) -> Vec<(String, String, Vec)> { + #[cfg(feature = "onnxruntime")] + { + self.inner.model_info().inputs.iter().map(python_tensor_spec).collect() + } + #[cfg(not(feature = "onnxruntime"))] + { + Vec::new() + } + } + + /// Returns `(name, dtype, dimensions)` metadata for model outputs. + #[getter] + fn outputs(&self) -> Vec<(String, String, Vec)> { + #[cfg(feature = "onnxruntime")] + { + self.inner.model_info().outputs.iter().map(python_tensor_spec).collect() + } + #[cfg(not(feature = "onnxruntime"))] + { + Vec::new() + } + } + + /// Runs named tensors; zero-copy CPU I/O binding is the default. + #[pyo3(signature = (inputs, *, copy=false))] + fn run<'py>( + &mut self, + py: Python<'py>, + inputs: &Bound<'py, PyDict>, + copy: bool, + ) -> PyResult> { + #[cfg(feature = "onnxruntime")] + { + let mut named = AiNamedTensors::new(); + for (name, value) in inputs.iter() { + let name = name.extract::()?; + let tensor = value.extract::>()?; + named.insert(name, tensor.inner.clone()).map_err(to_py_err)?; + } + let outputs = if copy { + self.inner + .run_with_options( + named, + AiRunOptions { + input_copy: AiCopyPolicy::Allow, + output_copy: AiCopyPolicy::Allow, + }, + ) + .map_err(to_py_err)? + } else { + let destinations = self + .inner + .model_info() + .outputs + .iter() + .map(|spec| OutputBinding::Allocate { + name: spec.name.clone(), + device: TensorDevice::CPU, + }) + .collect(); + let mut binding = AiIoBinding::try_new(named, destinations).map_err(to_py_err)?; + self.inner.run_with_binding(&mut binding).map_err(to_py_err)?; + binding.into_results().ok_or_else(|| { + PyRuntimeError::new_err("ONNX Runtime completed without bound results") + })? + }; + let result = PyDict::new_bound(py); + for (name, tensor) in outputs.into_values() { + result.set_item(name, Py::new(py, PyTensor { inner: tensor })?)?; + } + Ok(result) + } + #[cfg(not(feature = "onnxruntime"))] + { + let _ = (py, inputs, copy); + Err(PyRuntimeError::new_err( + "this SpatialRust Python module was built without the `onnxruntime` feature", + )) + } + } +} + +#[cfg(feature = "onnxruntime")] +fn python_tensor_spec(spec: &spatialrust::ai::TensorSpec) -> (String, String, Vec) { + let dimensions = spec + .shape + .iter() + .map(|dimension| match dimension { + spatialrust::ai::Dimension::Fixed(value) => value.to_string(), + spatialrust::ai::Dimension::Dynamic => "?".into(), + spatialrust::ai::Dimension::Symbol(value) => value.clone(), + }) + .collect(); + (spec.name.clone(), tensor_dtype_name(spec.dtype), dimensions) +} + +#[pyclass(name = "DLPackTensorView", unsendable)] +struct PyDlpackTensorView { + inner: DlpackImport, +} + +#[pymethods] +impl PyDlpackTensorView { + #[getter] + fn shape(&self) -> Vec { + self.inner.descriptor().shape().to_vec() + } + + #[getter] + fn dtype(&self) -> String { + tensor_dtype_name(self.inner.descriptor().dtype()) + } + + #[getter] + fn version(&self) -> (u32, u32) { + self.inner.version() + } + + /// Makes ownership independent of the DLPack producer with an explicit copy. + fn copy(&self) -> PyResult { + Ok(PyTensor { inner: self.inner.view().map_err(to_py_err)?.to_owned_copy() }) + } + + fn __repr__(&self) -> String { + format!( + "DLPackTensorView(shape={:?}, dtype='{}', device='cpu', version={:?})", + self.shape(), + self.dtype(), + self.version() + ) + } +} + +fn tensor_dtype_name(dtype: TensorDataType) -> String { + match dtype { + TensorDataType::U8 => "uint8".into(), + TensorDataType::U16 => "uint16".into(), + TensorDataType::F32 => "float32".into(), + _ => format!("{:?}{}x{}", dtype.code(), dtype.bits(), dtype.lanes()), + } +} + +#[pymethods] +impl PyTensor { + #[getter] + fn shape(&self) -> Vec { + self.inner.descriptor().shape().to_vec() + } + + #[getter] + fn dtype(&self) -> String { + tensor_dtype_name(self.inner.descriptor().dtype()) + } + + /// Returns the DLPack CPU device tuple. + fn __dlpack_device__(&self) -> (i32, i32) { + (1, 0) + } + + /// Exports a read-only, zero-copy DLPack major-version 1 capsule. + #[pyo3(signature = (stream=None, *, max_version=None, dl_device=None, copy=None))] + fn __dlpack__( + &self, + py: Python<'_>, + stream: Option>, + max_version: Option<(u32, u32)>, + dl_device: Option<(i32, i32)>, + copy: Option, + ) -> PyResult> { + if stream.is_some() { + return Err(PyBufferError::new_err("CPU DLPack export requires stream=None")); + } + if max_version.is_some_and(|version| version.0 < 1) { + return Err(PyBufferError::new_err( + "consumer does not support the DLPack versioned ABI", + )); + } + if dl_device.is_some_and(|device| device != (1, 0)) { + return Err(PyBufferError::new_err( + "Tensor is CPU-resident; explicit device transfer is required", + )); + } + if copy == Some(true) { + return Err(PyBufferError::new_err( + "implicit DLPack copies are disabled; call Tensor.copy() explicitly", + )); + } + dlpack_capsule::export_tensor(py, &self.inner).map_err(to_py_err) + } + + /// Makes an explicit host-to-host allocation copy. + fn copy(&self) -> Self { + Self { inner: self.inner.to_owned_copy() } + } + + fn __repr__(&self) -> String { + format!("Tensor(shape={:?}, dtype='{}', device='cpu')", self.shape(), self.dtype()) + } +} + +/// Copies a NumPy uint8, uint16, or float32 array into packed CPU tensor storage. +#[pyfunction] +fn tensor_copy_from_numpy(array: &Bound<'_, PyAny>) -> PyResult { + if let Ok(array) = array.extract::>() { + let shape = array.shape().to_vec(); + let bytes = array.as_array().iter().copied().collect::>(); + let descriptor = TensorDescriptor::contiguous(TensorDataType::U8, shape, TensorDevice::CPU); + return Ok(PyTensor { + inner: TensorBuffer::try_new(bytes, descriptor).map_err(to_py_err)?, + }); + } + if let Ok(array) = array.extract::>() { + let shape = array.shape().to_vec(); + let values = array.as_array().iter().copied().collect::>(); + let descriptor = + TensorDescriptor::contiguous(TensorDataType::U16, shape, TensorDevice::CPU); + return Ok(PyTensor { + inner: TensorBuffer::try_from_u16(values, descriptor).map_err(to_py_err)?, + }); + } + if let Ok(array) = array.extract::>() { + let shape = array.shape().to_vec(); + let values = array.as_array().iter().copied().collect::>(); + let descriptor = + TensorDescriptor::contiguous(TensorDataType::F32, shape, TensorDevice::CPU); + return Ok(PyTensor { + inner: TensorBuffer::try_from_f32(values, descriptor).map_err(to_py_err)?, + }); + } + Err(PyTypeError::new_err("expected a NumPy uint8, uint16, or float32 array")) +} + +/// Takes ownership of a producer's CPU DLPack capsule without copying its allocation. +#[pyfunction] +fn tensor_view_from_dlpack(producer: &Bound<'_, PyAny>) -> PyResult { + let inner = dlpack_capsule::import_tensor(producer)?; + Ok(PyDlpackTensorView { inner }) +} + +fn parse_image_format(format: &str) -> PyResult { + match format.to_ascii_lowercase().as_str() { + "png" => Ok(ImageFileFormat::Png), + "jpg" | "jpeg" => Ok(ImageFileFormat::Jpeg), + "pnm" | "pbm" | "pgm" | "ppm" => Ok(ImageFileFormat::Pnm), + other => Err(PyValueError::new_err(format!( + "unsupported image format `{other}` (expected: png, jpeg, or pnm)" + ))), + } +} + +/// Source metadata returned alongside a decoded NumPy image. +#[pyclass(name = "ImageMetadata", frozen)] +#[derive(Clone)] +struct PyImageMetadata { + inner: DecodedMetadata, +} + +#[pymethods] +impl PyImageMetadata { + #[getter] + fn format(&self) -> String { + self.inner.format.to_string() + } + + #[getter] + fn color_type(&self) -> String { + format!("{:?}", self.inner.source_color_type) + } + + #[getter] + fn orientation(&self) -> u8 { + self.inner.orientation as u8 + } + + #[getter] + fn orientation_applied(&self) -> bool { + self.inner.orientation_applied + } + + fn __repr__(&self) -> String { + format!( + "ImageMetadata(format='{}', color_type='{}', orientation={}, orientation_applied={})", + self.format(), + self.color_type(), + self.orientation(), + self.orientation_applied() + ) + } +} + +/// Decodes PNG, JPEG, or PNM into an owned NumPy array and source metadata. +#[pyfunction] +#[pyo3(signature = (path, apply_orientation=true))] +fn read_image<'py>( + py: Python<'py>, + path: &str, + apply_orientation: bool, +) -> PyResult<(Py, PyImageMetadata)> { + let decoded = + decode_image_path(path, DecodeOptions { apply_orientation, ..Default::default() }) + .map_err(to_py_err)?; + let metadata = PyImageMetadata { inner: decoded.metadata() }; + let (height, width) = (decoded.height(), decoded.width()); + macro_rules! array2 { + ($image:expr) => { + Array2::from_shape_vec((height, width), $image.into_vec()) + .map_err(to_py_err)? + .into_pyarray_bound(py) + .into_any() + .unbind() + }; + } + macro_rules! array3 { + ($image:expr, $channels:expr) => { + Array3::from_shape_vec((height, width, $channels), $image.into_vec()) + .map_err(to_py_err)? + .into_pyarray_bound(py) + .into_any() + .unbind() + }; + } + let array = match decoded.into_pixels() { + DecodedPixels::Gray8(image) => array2!(image), + DecodedPixels::GrayAlpha8(image) => array3!(image, 2), + DecodedPixels::Rgb8(image) => array3!(image, 3), + DecodedPixels::Rgba8(image) => array3!(image, 4), + DecodedPixels::Gray16(image) => array2!(image), + DecodedPixels::GrayAlpha16(image) => array3!(image, 2), + DecodedPixels::Rgb16(image) => array3!(image, 3), + DecodedPixels::Rgba16(image) => array3!(image, 4), + DecodedPixels::Rgb32Float(image) => array3!(image, 3), + DecodedPixels::Rgba32Float(image) => array3!(image, 4), + }; + Ok((array, metadata)) +} + +/// Encodes a uint8/uint16 NumPy image as PNG, JPEG, or PNM. +#[pyfunction] +#[pyo3(signature = (path, image, format, jpeg_quality=90))] +fn write_image( + path: &str, + image: &Bound<'_, PyAny>, + format: &str, + jpeg_quality: u8, +) -> PyResult<()> { + let format = parse_image_format(format)?; + let pixels = if let Ok(array) = image.extract::>() { + let view = array.as_array(); + let shape = view.shape(); + DecodedPixels::Gray8( + Image::try_new(shape[1], shape[0], view.iter().copied().collect()) + .map_err(to_py_err)?, + ) + } else if let Ok(array) = image.extract::>() { + let view = array.as_array(); + let shape = view.shape(); + DecodedPixels::Gray16( + Image::try_new(shape[1], shape[0], view.iter().copied().collect()) + .map_err(to_py_err)?, + ) + } else if let Ok(array) = image.extract::>() { + let view = array.as_array(); + let shape = view.shape(); + let packed = view.iter().copied().collect(); + match shape[2] { + 2 => DecodedPixels::GrayAlpha8( + Image::try_new(shape[1], shape[0], packed).map_err(to_py_err)?, + ), + 3 => { + DecodedPixels::Rgb8(Image::try_new(shape[1], shape[0], packed).map_err(to_py_err)?) + } + 4 => { + DecodedPixels::Rgba8(Image::try_new(shape[1], shape[0], packed).map_err(to_py_err)?) + } + channels => { + return Err(PyValueError::new_err(format!( + "expected 2, 3, or 4 channels, found {channels}" + ))) + } + } + } else if let Ok(array) = image.extract::>() { + let view = array.as_array(); + let shape = view.shape(); + let packed = view.iter().copied().collect(); + match shape[2] { + 2 => DecodedPixels::GrayAlpha16( + Image::try_new(shape[1], shape[0], packed).map_err(to_py_err)?, + ), + 3 => { + DecodedPixels::Rgb16(Image::try_new(shape[1], shape[0], packed).map_err(to_py_err)?) + } + 4 => DecodedPixels::Rgba16( + Image::try_new(shape[1], shape[0], packed).map_err(to_py_err)?, + ), + channels => { + return Err(PyValueError::new_err(format!( + "expected 2, 3, or 4 channels, found {channels}" + ))) + } + } + } else { + return Err(PyValueError::new_err( + "expected a uint8 or uint16 NumPy array shaped (H, W) or (H, W, C)", + )); + }; + encode_image_path(path, &pixels, EncodeOptions { format, jpeg_quality }).map_err(to_py_err) +} + fn parse_policy(policy: &str) -> PyResult { match policy.to_lowercase().as_str() { "auto" => Ok(ExecutionPolicy::Auto), @@ -100,6 +647,17 @@ fn parse_interpolation(interpolation: &str) -> PyResult { } } +fn parse_threshold_type(value: &str) -> PyResult { + match value.to_ascii_lowercase().as_str() { + "binary" => Ok(ThresholdType::Binary), + "binary_inv" | "binary-inv" => Ok(ThresholdType::BinaryInv), + "truncate" | "trunc" => Ok(ThresholdType::Truncate), + "to_zero" | "to-zero" => Ok(ThresholdType::ToZero), + "to_zero_inv" | "to-zero-inv" => Ok(ThresholdType::ToZeroInv), + other => Err(PyValueError::new_err(format!("unknown threshold type `{other}`"))), + } +} + fn rgb_image_from_numpy(array: PyReadonlyArray3<'_, u8>) -> PyResult> { let view = array.as_array(); let shape = view.shape(); @@ -1204,6 +1762,760 @@ fn rgbd_to_point_cloud( Ok(PyPointCloud { inner }) } +/// Correlates an RGB image with a 2D float64 kernel using Reflect101 borders. +#[pyfunction] +#[pyo3(signature = (image, kernel, delta=0.0))] +fn filter2d_image<'py>( + py: Python<'py>, + image: PyReadonlyArray3<'_, u8>, + kernel: PyReadonlyArray2<'_, f64>, + delta: f64, +) -> PyResult>> { + let image = rgb_image_from_numpy(image)?; + let kernel_view = kernel.as_array(); + let shape = kernel_view.shape(); + let kernel = Kernel2D::try_new(shape[1], shape[0], kernel_view.iter().copied().collect()) + .map_err(to_py_err)?; + let output = + filter2d_op(image.view(), &kernel, delta, BorderMode::Reflect101).map_err(to_py_err)?; + let array = Array3::from_shape_vec((image.height(), image.width(), 3), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Applies a normalized Gaussian blur to an RGB image using Reflect101 borders. +#[pyfunction] +#[pyo3(signature = (image, kernel_width, kernel_height, sigma_x, sigma_y=None))] +fn gaussian_blur_image<'py>( + py: Python<'py>, + image: PyReadonlyArray3<'_, u8>, + kernel_width: usize, + kernel_height: usize, + sigma_x: f64, + sigma_y: Option, +) -> PyResult>> { + let image = rgb_image_from_numpy(image)?; + let output = gaussian_blur_op( + image.view(), + kernel_width, + kernel_height, + sigma_x, + sigma_y.unwrap_or(sigma_x), + BorderMode::Reflect101, + ) + .map_err(to_py_err)?; + let array = Array3::from_shape_vec((image.height(), image.width(), 3), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Applies an odd-aperture median filter to an RGB image. +#[pyfunction] +fn median_blur_image<'py>( + py: Python<'py>, + image: PyReadonlyArray3<'_, u8>, + kernel_size: usize, +) -> PyResult>> { + let image = rgb_image_from_numpy(image)?; + let output = + median_blur_op(image.view(), kernel_size, BorderMode::Replicate).map_err(to_py_err)?; + let array = Array3::from_shape_vec((image.height(), image.width(), 3), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Applies an RGB bilateral filter using Reflect101 borders. +#[pyfunction] +fn bilateral_filter_image<'py>( + py: Python<'py>, + image: PyReadonlyArray3<'_, u8>, + diameter: usize, + sigma_color: f64, + sigma_space: f64, +) -> PyResult>> { + let image = rgb_image_from_numpy(image)?; + let output = bilateral_filter_op( + image.view(), + diameter, + sigma_color, + sigma_space, + BorderMode::Reflect101, + ) + .map_err(to_py_err)?; + let array = Array3::from_shape_vec((image.height(), image.width(), 3), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Computes a signed float32 Sobel derivative from a grayscale uint8 image. +#[pyfunction] +#[pyo3(signature = (image, dx, dy, kernel_size=3, scale=1.0, delta=0.0))] +fn sobel_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + dx: usize, + dy: usize, + kernel_size: usize, + scale: f64, + delta: f64, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let output = sobel_op(image.view(), dx, dy, kernel_size, scale, delta, BorderMode::Reflect101) + .map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Computes a signed float32 Scharr derivative from a grayscale uint8 image. +#[pyfunction] +#[pyo3(signature = (image, dx, dy, scale=1.0, delta=0.0))] +fn scharr_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + dx: usize, + dy: usize, + scale: f64, + delta: f64, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let output = + scharr_op(image.view(), dx, dy, scale, delta, BorderMode::Reflect101).map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Computes a signed float32 Laplacian from a grayscale uint8 image. +#[pyfunction] +#[pyo3(signature = (image, kernel_size=1, scale=1.0, delta=0.0))] +fn laplacian_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + kernel_size: usize, + scale: f64, + delta: f64, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let output = laplacian_op(image.view(), kernel_size, scale, delta, BorderMode::Reflect101) + .map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Reduces an RGB image with the canonical Gaussian pyramid kernel. +#[pyfunction] +fn pyr_down_image<'py>( + py: Python<'py>, + image: PyReadonlyArray3<'_, u8>, +) -> PyResult>> { + let image = rgb_image_from_numpy(image)?; + let output = pyr_down_op(image.view(), BorderMode::Reflect101).map_err(to_py_err)?; + let array = Array3::from_shape_vec((output.height(), output.width(), 3), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Doubles an RGB image with the canonical Gaussian pyramid kernel. +#[pyfunction] +fn pyr_up_image<'py>( + py: Python<'py>, + image: PyReadonlyArray3<'_, u8>, +) -> PyResult>> { + let image = rgb_image_from_numpy(image)?; + let output = pyr_up_op(image.view(), BorderMode::Reflect101).map_err(to_py_err)?; + let array = Array3::from_shape_vec((output.height(), output.width(), 3), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Applies grayscale erosion, dilation, or a composite morphology operation. +#[pyfunction] +#[pyo3(signature = (image, operation, kernel_width, kernel_height, shape="rect", iterations=1))] +fn morphology_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + operation: &str, + kernel_width: usize, + kernel_height: usize, + shape: &str, + iterations: usize, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let shape = match shape.to_ascii_lowercase().as_str() { + "rect" | "rectangle" => MorphologyShape::Rect, + "cross" => MorphologyShape::Cross, + "ellipse" | "elliptical" => MorphologyShape::Ellipse, + "diamond" => MorphologyShape::Diamond, + other => { + return Err(PyValueError::new_err(format!( + "unknown morphology shape `{other}` (expected: rect, cross, ellipse, diamond)" + ))) + } + }; + let element = + StructuringElement::try_new(shape, kernel_width, kernel_height).map_err(to_py_err)?; + let operation = operation.to_ascii_lowercase(); + let output = match operation.as_str() { + "erode" => { + spatialrust::vision::erode(image.view(), &element, iterations, BorderMode::Replicate) + } + "dilate" => { + spatialrust::vision::dilate(image.view(), &element, iterations, BorderMode::Replicate) + } + "open" => morphology_ex_op( + image.view(), + MorphologyOperation::Open, + &element, + iterations, + BorderMode::Replicate, + ), + "close" => morphology_ex_op( + image.view(), + MorphologyOperation::Close, + &element, + iterations, + BorderMode::Replicate, + ), + "gradient" => morphology_ex_op( + image.view(), + MorphologyOperation::Gradient, + &element, + iterations, + BorderMode::Replicate, + ), + "tophat" | "top-hat" => morphology_ex_op( + image.view(), + MorphologyOperation::TopHat, + &element, + iterations, + BorderMode::Replicate, + ), + "blackhat" | "black-hat" => morphology_ex_op( + image.view(), + MorphologyOperation::BlackHat, + &element, + iterations, + BorderMode::Replicate, + ), + other => { + return Err(PyValueError::new_err(format!("unknown morphology operation `{other}`"))) + } + } + .map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Applies a fixed threshold to a grayscale uint8 image. +#[pyfunction] +#[pyo3(signature = (image, threshold, max_value=255, threshold_type="binary"))] +fn threshold_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + threshold: f64, + max_value: u8, + threshold_type: &str, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let output = threshold_op( + image.view(), + threshold, + f64::from(max_value), + parse_threshold_type(threshold_type)?, + ) + .map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Selects and applies an Otsu threshold, returning `(threshold, image)`. +#[pyfunction] +#[pyo3(signature = (image, max_value=255, threshold_type="binary"))] +fn otsu_threshold_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + max_value: u8, + threshold_type: &str, +) -> PyResult<(u8, Bound<'py, PyArray2>)> { + let image = gray_u8_image_from_numpy(image)?; + let (selected, output) = + otsu_threshold_u8_op(image.view(), max_value, parse_threshold_type(threshold_type)?) + .map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok((selected, array.into_pyarray_bound(py))) +} + +/// Applies mean or Gaussian adaptive thresholding. +#[pyfunction] +#[pyo3(signature = (image, block_size, c, method="mean", max_value=255, threshold_type="binary"))] +fn adaptive_threshold_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + block_size: usize, + c: f64, + method: &str, + max_value: u8, + threshold_type: &str, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let method = match method.to_ascii_lowercase().as_str() { + "mean" => AdaptiveThresholdMethod::Mean, + "gaussian" => AdaptiveThresholdMethod::Gaussian, + other => return Err(PyValueError::new_err(format!("unknown adaptive method `{other}`"))), + }; + let output = adaptive_threshold_op( + image.view(), + max_value, + method, + parse_threshold_type(threshold_type)?, + block_size, + c, + BorderMode::Replicate, + ) + .map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Returns the exact 256-bin grayscale histogram. +#[pyfunction] +fn histogram_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + Ok(histogram_u8_op(image.view()).into_pyarray_bound(py)) +} + +/// Equalizes a grayscale uint8 histogram. +#[pyfunction] +fn equalize_histogram_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let output = equalize_histogram_op(image.view()).map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Applies contrast-limited adaptive histogram equalization. +#[pyfunction] +#[pyo3(signature = (image, clip_limit=2.0, tiles_x=8, tiles_y=8))] +fn clahe_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + clip_limit: f64, + tiles_x: usize, + tiles_y: usize, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let output = clahe_op(image.view(), clip_limit, tiles_x, tiles_y).map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Computes the `(H + 1, W + 1)` float64 summed-area table. +#[pyfunction] +fn integral_image_u8<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let integral = integral_image_op(image.view(), 0).map_err(to_py_err)?; + let array = + Array2::from_shape_vec((integral.height(), integral.width()), integral.as_slice().to_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +/// Detects edges in a grayscale uint8 image with Canny hysteresis. +#[pyfunction] +#[pyo3(signature = (image, low_threshold, high_threshold, aperture_size=3, l2_gradient=false))] +fn canny_image<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + low_threshold: f64, + high_threshold: f64, + aperture_size: usize, + l2_gradient: bool, +) -> PyResult>> { + let image = gray_u8_image_from_numpy(image)?; + let output = canny_op( + image.view(), + CannyOptions { low_threshold, high_threshold, aperture_size, l2_gradient }, + ) + .map_err(to_py_err)?; + let array = Array2::from_shape_vec((image.height(), image.width()), output.into_vec()) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +fn corner_selection_options( + max_corners: usize, + quality_level: f32, + min_distance: f32, + block_size: usize, + gradient_size: usize, +) -> CornerSelectionOptions { + CornerSelectionOptions { + max_corners, + quality_level, + min_distance, + block_size, + gradient_size, + border: BorderMode::Reflect101, + } +} + +/// Detects strongest-first Harris keypoints in a grayscale uint8 image. +#[pyfunction] +#[pyo3(signature = (image, max_corners=0, quality_level=0.01, min_distance=1.0, block_size=3, gradient_size=3, k=0.04))] +#[allow(clippy::too_many_arguments)] +fn harris_keypoints( + image: PyReadonlyArray2<'_, u8>, + max_corners: usize, + quality_level: f32, + min_distance: f32, + block_size: usize, + gradient_size: usize, + k: f32, +) -> PyResult> { + let image = gray_u8_image_from_numpy(image)?; + let options = HarrisOptions { + selection: corner_selection_options( + max_corners, + quality_level, + min_distance, + block_size, + gradient_size, + ), + k, + }; + Ok(detect_harris_op(image.view(), options) + .map_err(to_py_err)? + .into_iter() + .map(|inner| PyKeypoint2 { inner }) + .collect()) +} + +/// Detects strongest-first Shi–Tomasi keypoints in a grayscale uint8 image. +#[pyfunction] +#[pyo3(signature = (image, max_corners=0, quality_level=0.01, min_distance=1.0, block_size=3, gradient_size=3))] +fn shi_tomasi_keypoints( + image: PyReadonlyArray2<'_, u8>, + max_corners: usize, + quality_level: f32, + min_distance: f32, + block_size: usize, + gradient_size: usize, +) -> PyResult> { + let image = gray_u8_image_from_numpy(image)?; + let options = ShiTomasiOptions { + selection: corner_selection_options( + max_corners, + quality_level, + min_distance, + block_size, + gradient_size, + ), + }; + Ok(detect_shi_tomasi_op(image.view(), options) + .map_err(to_py_err)? + .into_iter() + .map(|inner| PyKeypoint2 { inner }) + .collect()) +} + +/// Detects scan-ordered FAST-9/16 keypoints in a grayscale uint8 image. +#[pyfunction] +#[pyo3(signature = (image, threshold=10, nonmax_suppression=true))] +fn fast_keypoints( + image: PyReadonlyArray2<'_, u8>, + threshold: u8, + nonmax_suppression: bool, +) -> PyResult> { + let image = gray_u8_image_from_numpy(image)?; + Ok(detect_fast_op(image.view(), FastOptions { threshold, nonmax_suppression }) + .map_err(to_py_err)? + .into_iter() + .map(|inner| PyKeypoint2 { inner }) + .collect()) +} + +/// Detects ORB keypoints and returns `(keypoints, uint8[N, 32] descriptors)`. +#[pyfunction] +#[pyo3(signature = (image, max_features=500, scale_factor=1.2, levels=8, edge_threshold=31, fast_threshold=20, patch_size=31, score_type="harris"))] +#[allow(clippy::too_many_arguments)] +fn orb_features<'py>( + py: Python<'py>, + image: PyReadonlyArray2<'_, u8>, + max_features: usize, + scale_factor: f32, + levels: usize, + edge_threshold: usize, + fast_threshold: u8, + patch_size: usize, + score_type: &str, +) -> PyResult<(Vec, Bound<'py, PyArray2>)> { + let image = gray_u8_image_from_numpy(image)?; + let score_type = match score_type { + "harris" => OrbScoreType::Harris, + "fast" => OrbScoreType::Fast, + _ => return Err(PyValueError::new_err("score_type must be 'harris' or 'fast'")), + }; + let features = detect_and_describe_orb_op( + image.view(), + OrbOptions { + max_features, + scale_factor, + levels, + edge_threshold, + fast_threshold, + patch_size, + score_type, + }, + ) + .map_err(to_py_err)?; + let keypoints = features + .keypoints() + .iter() + .copied() + .map(|inner| PyKeypoint2 { inner }) + .collect(); + let descriptors = Array2::from_shape_vec( + (features.descriptors().len(), features.descriptors().width()), + features.descriptors().binary_data().expect("ORB descriptors are binary").to_vec(), + ) + .map_err(to_py_err)? + .into_pyarray_bound(py); + Ok((keypoints, descriptors)) +} + +fn correspondences_from_numpy( + source: PyReadonlyArray2<'_, f64>, + target: PyReadonlyArray2<'_, f64>, +) -> PyResult> { + let source = source.as_array(); + let target = target.as_array(); + if source.shape() != target.shape() || source.ndim() != 2 || source.shape()[1] != 2 { + return Err(PyValueError::new_err("source/target must be Nx2 float64 arrays")); + } + source + .outer_iter() + .zip(target.outer_iter()) + .map(|(src, dst)| { + PointCorrespondence2::try_new( + Vec2 { x: src[0], y: src[1] }, + Vec2 { x: dst[0], y: dst[1] }, + ) + .map_err(to_py_err) + }) + .collect() +} + +fn mat3_to_numpy<'py>(py: Python<'py>, matrix: Mat3) -> Bound<'py, PyArray2> { + let mut values = Vec::with_capacity(9); + for row in &matrix.m { + values.extend_from_slice(row); + } + Array2::from_shape_vec((3, 3), values) + .expect("3x3") + .into_pyarray_bound(py) +} + +/// Estimates a homography with deterministic RANSAC. +/// +/// Returns `(matrix[3,3], inliers[N], residuals[N])`. +#[pyfunction] +#[pyo3(signature = (source, target, threshold=1.0, confidence=0.99, max_iterations=2000, seed=0))] +fn estimate_homography_ransac<'py>( + py: Python<'py>, + source: PyReadonlyArray2<'_, f64>, + target: PyReadonlyArray2<'_, f64>, + threshold: f64, + confidence: f64, + max_iterations: usize, + seed: u64, +) -> PyResult<(Bound<'py, PyArray2>, Bound<'py, PyArray1>, Bound<'py, PyArray1>)> { + let pairs = correspondences_from_numpy(source, target)?; + let estimate = estimate_homography_ransac_op( + &pairs, + RobustEstimationOptions { threshold, confidence, max_iterations, seed }, + ) + .map_err(to_py_err)?; + Ok(( + mat3_to_numpy(py, estimate.model().matrix()), + numpy::PyArray1::from_vec_bound(py, estimate.inliers().to_vec()), + numpy::PyArray1::from_vec_bound(py, estimate.residuals().to_vec()), + )) +} + +/// Solves PnP for object points `Nx3` and image points `Nx2`. +/// +/// Camera intrinsics are `fx, fy, cx, cy`. Returns `(rotation[3,3], translation[3])`. +#[pyfunction] +#[pyo3(signature = (object_points, image_points, fx, fy, cx, cy, width=640, height=480))] +#[allow(clippy::too_many_arguments)] +fn solve_pnp<'py>( + py: Python<'py>, + object_points: PyReadonlyArray2<'_, f64>, + image_points: PyReadonlyArray2<'_, f64>, + fx: f64, + fy: f64, + cx: f64, + cy: f64, + width: usize, + height: usize, +) -> PyResult<(Bound<'py, PyArray2>, Bound<'py, PyArray1>)> { + let objects = object_points.as_array(); + let images = image_points.as_array(); + if objects.ndim() != 2 + || images.ndim() != 2 + || objects.shape()[1] != 3 + || images.shape()[1] != 2 + || objects.shape()[0] != images.shape()[0] + { + return Err(PyValueError::new_err( + "object_points must be Nx3 and image_points Nx2 with matching N", + )); + } + let pairs = objects + .outer_iter() + .zip(images.outer_iter()) + .map(|(object, image)| { + ObjectImageCorrespondence::try_new( + Vec3::new(object[0], object[1], object[2]), + Vec2 { x: image[0], y: image[1] }, + ) + .map_err(to_py_err) + }) + .collect::>>()?; + let camera = CameraMatrix3::from_intrinsics( + CameraIntrinsics::try_new(fx, fy, cx, cy, width, height).map_err(to_py_err)?, + ); + let pose: AbsolutePose = solve_pnp_op(&pairs, camera).map_err(to_py_err)?; + let translation = numpy::PyArray1::from_vec_bound( + py, + vec![pose.translation().x, pose.translation().y, pose.translation().z], + ); + Ok((mat3_to_numpy(py, pose.rotation()), translation)) +} + +/// Dense SAD stereo block matching on rectified grayscale images. +#[pyfunction] +#[pyo3(signature = (left, right, window_size=15, min_disparity=0, num_disparities=64, uniqueness_ratio=15.0))] +fn stereo_block_match<'py>( + py: Python<'py>, + left: PyReadonlyArray2<'_, u8>, + right: PyReadonlyArray2<'_, u8>, + window_size: usize, + min_disparity: i32, + num_disparities: i32, + uniqueness_ratio: f32, +) -> PyResult>> { + let left = gray_u8_image_from_numpy(left)?; + let right = gray_u8_image_from_numpy(right)?; + let disparity = stereo_block_match_op( + left.view(), + right.view(), + StereoBmOptions { window_size, min_disparity, num_disparities, uniqueness_ratio }, + ) + .map_err(to_py_err)?; + let array = Array2::from_shape_vec( + (disparity.height(), disparity.width()), + disparity.as_slice().to_vec(), + ) + .map_err(to_py_err)?; + Ok(array.into_pyarray_bound(py)) +} + +fn descriptor_match_tuples( + query: DescriptorBuffer, + train: DescriptorBuffer, + cross_check: bool, + ratio: Option, + max_distance: Option, +) -> PyResult> { + Ok(match_descriptors_op( + &query, + &train, + MatchOptions { cross_check, ratio, max_distance }, + ) + .map_err(to_py_err)? + .into_iter() + .map(|feature_match| { + ( + feature_match.query_index(), + feature_match.train_index(), + feature_match.distance(), + ) + }) + .collect()) +} + +/// Brute-force Hamming matching for two `uint8[N, D]` descriptor matrices. +#[pyfunction] +#[pyo3(signature = (query, train, cross_check=false, ratio=None, max_distance=None))] +fn match_binary_descriptors( + query: PyReadonlyArray2<'_, u8>, + train: PyReadonlyArray2<'_, u8>, + cross_check: bool, + ratio: Option, + max_distance: Option, +) -> PyResult> { + let query_shape = query.shape(); + let train_shape = train.shape(); + let query = DescriptorBuffer::try_binary( + query_shape[0], + query_shape[1], + query.as_array().iter().copied().collect(), + ) + .map_err(to_py_err)?; + let train = DescriptorBuffer::try_binary( + train_shape[0], + train_shape[1], + train.as_array().iter().copied().collect(), + ) + .map_err(to_py_err)?; + descriptor_match_tuples(query, train, cross_check, ratio, max_distance) +} + +/// Brute-force Euclidean matching for two `float32[N, D]` descriptor matrices. +#[pyfunction] +#[pyo3(signature = (query, train, cross_check=false, ratio=None, max_distance=None))] +fn match_float_descriptors( + query: PyReadonlyArray2<'_, f32>, + train: PyReadonlyArray2<'_, f32>, + cross_check: bool, + ratio: Option, + max_distance: Option, +) -> PyResult> { + let query_shape = query.shape(); + let train_shape = train.shape(); + let query = DescriptorBuffer::try_float32( + query_shape[0], + query_shape[1], + query.as_array().iter().copied().collect(), + ) + .map_err(to_py_err)?; + let train = DescriptorBuffer::try_float32( + train_shape[0], + train_shape[1], + train.as_array().iter().copied().collect(), + ) + .map_err(to_py_err)?; + descriptor_match_tuples(query, train, cross_check, ratio, max_distance) +} + /// Resizes an `(H, W, 3)` uint8 RGB image. #[pyfunction] #[pyo3(signature = (image, width, height, interpolation="bilinear"))] @@ -1533,6 +2845,11 @@ fn point_map_to_point_cloud( #[pyo3(name = "spatialrust")] fn spatialrust_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add("__version__", env!("CARGO_PKG_VERSION"))?; + m.add_class::()?; + m.add_class::()?; + m.add_class::()?; + m.add_class::()?; + m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; @@ -1542,6 +2859,19 @@ fn spatialrust_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; + m.add_function(wrap_pyfunction!(read_image, m)?)?; + m.add_function(wrap_pyfunction!(tensor_copy_from_numpy, m)?)?; + m.add_function(wrap_pyfunction!(tensor_view_from_dlpack, m)?)?; + m.add_function(wrap_pyfunction!(harris_keypoints, m)?)?; + m.add_function(wrap_pyfunction!(shi_tomasi_keypoints, m)?)?; + m.add_function(wrap_pyfunction!(fast_keypoints, m)?)?; + m.add_function(wrap_pyfunction!(orb_features, m)?)?; + m.add_function(wrap_pyfunction!(estimate_homography_ransac, m)?)?; + m.add_function(wrap_pyfunction!(solve_pnp, m)?)?; + m.add_function(wrap_pyfunction!(stereo_block_match, m)?)?; + m.add_function(wrap_pyfunction!(match_binary_descriptors, m)?)?; + m.add_function(wrap_pyfunction!(match_float_descriptors, m)?)?; + m.add_function(wrap_pyfunction!(write_image, m)?)?; m.add_function(wrap_pyfunction!(read, m)?)?; m.add_function(wrap_pyfunction!(write, m)?)?; m.add_function(wrap_pyfunction!(voxel_downsample, m)?)?; @@ -1574,6 +2904,24 @@ fn spatialrust_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(voxelize, m)?)?; m.add_function(wrap_pyfunction!(range_image, m)?)?; m.add_function(wrap_pyfunction!(rgbd_to_point_cloud, m)?)?; + m.add_function(wrap_pyfunction!(filter2d_image, m)?)?; + m.add_function(wrap_pyfunction!(gaussian_blur_image, m)?)?; + m.add_function(wrap_pyfunction!(median_blur_image, m)?)?; + m.add_function(wrap_pyfunction!(bilateral_filter_image, m)?)?; + m.add_function(wrap_pyfunction!(sobel_image, m)?)?; + m.add_function(wrap_pyfunction!(scharr_image, m)?)?; + m.add_function(wrap_pyfunction!(laplacian_image, m)?)?; + m.add_function(wrap_pyfunction!(pyr_down_image, m)?)?; + m.add_function(wrap_pyfunction!(pyr_up_image, m)?)?; + m.add_function(wrap_pyfunction!(morphology_image, m)?)?; + m.add_function(wrap_pyfunction!(threshold_image, m)?)?; + m.add_function(wrap_pyfunction!(otsu_threshold_image, m)?)?; + m.add_function(wrap_pyfunction!(adaptive_threshold_image, m)?)?; + m.add_function(wrap_pyfunction!(histogram_image, m)?)?; + m.add_function(wrap_pyfunction!(equalize_histogram_image, m)?)?; + m.add_function(wrap_pyfunction!(clahe_image, m)?)?; + m.add_function(wrap_pyfunction!(integral_image_u8, m)?)?; + m.add_function(wrap_pyfunction!(canny_image, m)?)?; m.add_function(wrap_pyfunction!(resize_image, m)?)?; m.add_function(wrap_pyfunction!(letterbox_image, m)?)?; m.add_function(wrap_pyfunction!(normalize_image_chw, m)?)?; diff --git a/crates/spatialrust-py/tests/test_bindings.py b/crates/spatialrust-py/tests/test_bindings.py index 540a66f..1440ec3 100644 --- a/crates/spatialrust-py/tests/test_bindings.py +++ b/crates/spatialrust-py/tests/test_bindings.py @@ -469,3 +469,246 @@ def test_fpfh_ransac_returns_transform(): res = sr.register_fpfh_ransac(src, tgt, feature_radius=0.5, max_correspondence_distance=0.2, ransac_iterations=500) assert res.transform().shape == (4, 4) +def test_png_image_io_roundtrip(tmp_path): + image = np.arange(5 * 7 * 3, dtype=np.uint8).reshape(5, 7, 3) + path = tmp_path / "roundtrip.png" + sr.write_image(str(path), image[:, ::-1], "png") + decoded, metadata = sr.read_image(str(path)) + np.testing.assert_array_equal(decoded, image[:, ::-1]) + assert metadata.format == "png" + assert metadata.color_type == "Rgb8" + assert metadata.orientation in (0, 1) + assert "ImageMetadata(" in repr(metadata) + + +def test_filter2d_and_gaussian_preserve_rgb_shape(): + image = np.arange(7 * 9 * 3, dtype=np.uint8).reshape(7, 9, 3) + identity = sr.filter2d_image(image[:, ::-1], np.array([[1.0]], dtype=np.float64)) + np.testing.assert_array_equal(identity, image[:, ::-1]) + blurred = sr.gaussian_blur_image(image, 5, 3, 1.2, 0.8) + assert blurred.shape == image.shape + assert blurred.dtype == np.uint8 + + +def test_advanced_filters_and_pyramid_shapes(): + image = np.arange(9 * 11 * 3, dtype=np.uint8).reshape(9, 11, 3) + assert sr.median_blur_image(image[:, ::-1], 3).shape == image.shape + assert sr.bilateral_filter_image(image, 3, 20.0, 2.0).shape == image.shape + gray = image[..., 0] + for derivative in ( + sr.sobel_image(gray, 1, 0), + sr.scharr_image(gray, 0, 1), + sr.laplacian_image(gray), + ): + assert derivative.shape == gray.shape + assert derivative.dtype == np.float32 + down = sr.pyr_down_image(image) + assert down.shape == (5, 6, 3) + assert sr.pyr_up_image(down).shape == (10, 12, 3) + + +def test_morphology_operations_and_noncontiguous_input(): + mask = np.zeros((9, 11), dtype=np.uint8) + mask[2:7, 3:8] = 255 + for operation in ("erode", "dilate", "open", "close", "gradient", "tophat", "blackhat"): + output = sr.morphology_image(mask[:, ::-1], operation, 3, 3, "ellipse", 2) + assert output.shape == mask.shape + assert output.dtype == np.uint8 + + +def test_threshold_histogram_clahe_and_integral_contracts(): + image = np.arange(9 * 11, dtype=np.uint8).reshape(9, 11)[:, ::-1] + assert sr.threshold_image(image, 40).shape == image.shape + selected, otsu = sr.otsu_threshold_image(image) + assert 0 <= selected <= 255 and otsu.shape == image.shape + for method in ("mean", "gaussian"): + assert sr.adaptive_threshold_image(image, 5, 2.0, method).shape == image.shape + histogram = sr.histogram_image(image) + assert histogram.shape == (256,) and int(histogram.sum()) == image.size + assert sr.equalize_histogram_image(image).shape == image.shape + assert sr.clahe_image(image, 2.0, 3, 2).shape == image.shape + integral = sr.integral_image_u8(image) + assert integral.shape == (image.shape[0] + 1, image.shape[1] + 1) + assert integral[-1, -1] == pytest.approx(float(image.sum())) + + +def test_canny_image_binary_output_and_noncontiguous_input(): + image = np.zeros((17, 19), dtype=np.uint8) + image[4:13, 6:14] = 255 + edges = sr.canny_image(image[:, ::-1], 50.0, 100.0, 3, True) + assert edges.shape == image.shape + assert edges.dtype == np.uint8 + assert set(np.unique(edges)).issubset({0, 255}) + assert np.count_nonzero(edges) > 0 + + +def test_feature2d_corner_detectors_and_keypoint_metadata(): + image = np.zeros((25, 29), dtype=np.uint8) + image[5:19, 7:22] = 255 + harris = sr.harris_keypoints(image[:, ::-1], 20, 0.01, 1.0, 3, 3, 0.04) + shi = sr.shi_tomasi_keypoints(image, 20, 0.01, 1.0, 3, 3) + assert len(harris) >= 4 and len(shi) >= 4 + assert all(point.size == 3.0 and point.angle_degrees is None for point in harris) + impulse = np.zeros((9, 9), dtype=np.uint8) + impulse[4, 4] = 255 + fast = sr.fast_keypoints(impulse, 20, True) + assert len(fast) == 1 + assert (fast[0].x, fast[0].y, fast[0].size) == (4.0, 4.0, 7.0) + assert "Keypoint2(" in repr(fast[0]) + + +def test_orb_features_and_descriptor_matchers(): + yy, xx = np.indices((96, 112), dtype=np.int32) + image = ((xx * 37 + yy * 19) ^ (xx * yy * 3) ^ ((xx // 8 + yy // 8) * 127)).astype(np.uint8) + keypoints, descriptors = sr.orb_features(image[:, ::-1], max_features=60, edge_threshold=16) + assert 0 < len(keypoints) <= 60 + assert descriptors.shape == (len(keypoints), 32) + assert descriptors.dtype == np.uint8 + assert all(point.angle_degrees is not None for point in keypoints) + + matches = sr.match_binary_descriptors(descriptors, descriptors, cross_check=True) + assert len(matches) == len(keypoints) + assert all(query == train and distance == 0.0 for query, train, distance in matches) + + query = np.array([[0.0, 0.0], [10.0, 10.0]], dtype=np.float32) + train = np.array([[1.0, 0.0], [3.0, 0.0], [10.0, 9.0]], dtype=np.float32) + assert sr.match_float_descriptors(query, train, cross_check=True, ratio=0.8) == [ + (0, 0, 1.0), + (1, 2, 1.0), + ] + + +def test_geometry_homography_pnp_and_stereo(): + source = np.array( + [[0.0, 0.0], [40.0, 0.0], [0.0, 30.0], [40.0, 30.0], [20.0, 15.0], [10.0, 5.0]], + dtype=np.float64, + ) + h = np.array([[1.05, 0.01, 2.0], [-0.02, 0.98, -1.5], [0.0001, 0.0, 1.0]], dtype=np.float64) + target = [] + for point in source: + projected = h @ np.array([point[0], point[1], 1.0]) + target.append([projected[0] / projected[2], projected[1] / projected[2]]) + target = np.asarray(target, dtype=np.float64) + matrix, inliers, residuals = sr.estimate_homography_ransac(source, target, threshold=1.0) + assert matrix.shape == (3, 3) + assert inliers.dtype == np.bool_ + assert residuals.shape == (source.shape[0],) + assert int(inliers.sum()) >= 4 + + objects = np.array( + [ + [0.0, 0.0, 0.0], + [0.2, 0.0, 0.0], + [0.0, 0.15, 0.0], + [0.0, 0.0, 0.1], + [0.1, 0.05, 0.05], + [0.05, -0.05, 0.02], + ], + dtype=np.float64, + ) + fx = fy = 500.0 + cx, cy = 320.0, 240.0 + rotation = np.eye(3, dtype=np.float64) + translation = np.array([0.1, -0.05, 2.5], dtype=np.float64) + images = [] + for point in objects: + camera = rotation @ point + translation + images.append([fx * camera[0] / camera[2] + cx, fy * camera[1] / camera[2] + cy]) + images = np.asarray(images, dtype=np.float64) + recovered_r, recovered_t = sr.solve_pnp(objects, images, fx, fy, cx, cy) + assert recovered_r.shape == (3, 3) + assert recovered_t.shape == (3,) + assert abs(recovered_t[2] - translation[2]) < 0.05 + + width, height = 96, 64 + disparity = 12 + yy, xx = np.indices((height, width), dtype=np.int32) + left = ((xx * 17 + yy * 29) % 200 + 20).astype(np.uint8) + right = np.zeros_like(left) + right[:, : width - disparity] = left[:, disparity:] + disparity_map = sr.stereo_block_match( + left, right, window_size=11, min_disparity=1, num_disparities=32, uniqueness_ratio=5.0 + ) + assert disparity_map.shape == (height, width) + assert abs(float(disparity_map[height // 2, width // 2]) - float(disparity)) <= 1.0 + + +@pytest.mark.parametrize("dtype", [np.uint8, np.uint16, np.float32]) +def test_tensor_dlpack_zero_copy_export(dtype): + source = np.arange(3 * 5, dtype=dtype).reshape(3, 5)[:, ::-1] + tensor = sr.tensor_copy_from_numpy(source) + first = np.from_dlpack(tensor) + second = np.from_dlpack(tensor) + np.testing.assert_array_equal(first, source) + assert first.dtype == source.dtype + assert first.shape == source.shape + assert first.__array_interface__["data"][0] == second.__array_interface__["data"][0] + assert not first.flags.writeable + assert tensor.__dlpack_device__() == (1, 0) + assert "Tensor(shape=" in repr(tensor) + + +def test_tensor_dlpack_copy_and_device_requests_are_explicit(): + tensor = sr.tensor_copy_from_numpy(np.arange(8, dtype=np.uint8)) + copied = tensor.copy() + assert np.from_dlpack(copied).__array_interface__["data"][0] != np.from_dlpack( + tensor + ).__array_interface__["data"][0] + with pytest.raises(BufferError): + tensor.__dlpack__(copy=True) + with pytest.raises(BufferError): + tensor.__dlpack__(dl_device=(2, 0)) + + +def test_onnxruntime_dynamic_named_binding_matches_reference(tmp_path): + model = bytes( + [ + 8, 8, 18, 16, 115, 112, 97, 116, 105, 97, 108, 114, 117, 115, 116, 45, 116, + 101, 115, 116, 58, 106, 10, 27, 10, 5, 105, 110, 112, 117, 116, 10, 5, 105, + 110, 112, 117, 116, 18, 6, 111, 117, 116, 112, 117, 116, 34, 3, 65, 100, 100, + 18, 14, 100, 111, 117, 98, 108, 101, 95, 100, 121, 110, 97, 109, 105, 99, 90, + 28, 10, 5, 105, 110, 112, 117, 116, 18, 19, 10, 17, 8, 1, 18, 13, 10, 7, + 18, 5, 98, 97, 116, 99, 104, 10, 2, 8, 3, 98, 29, 10, 6, 111, 117, 116, + 112, 117, 116, 18, 19, 10, 17, 8, 1, 18, 13, 10, 7, 18, 5, 98, 97, 116, + 99, 104, 10, 2, 8, 3, 66, 4, 10, 0, 16, 13, + ] + ) + path = tmp_path / "double_dynamic.onnx" + path.write_bytes(model) + try: + session = sr.OnnxRuntimeSession(str(path), deterministic=True) + except RuntimeError as error: + if "without the `onnxruntime` feature" in str(error): + pytest.skip("extension was intentionally built without ONNX Runtime") + raise + + source = np.arange(12, dtype=np.float32).reshape(4, 3) + inputs = {"input": sr.tensor_copy_from_numpy(source)} + assert session.inputs == [("input", "float32", ["batch", "3"])] + bound = np.from_dlpack(session.run(inputs)["output"]) + copied = np.from_dlpack(session.run(inputs, copy=True)["output"]) + expected = source * 2.0 + np.testing.assert_array_equal(bound, expected) + np.testing.assert_array_equal(copied, expected) + + try: + import onnxruntime as reference_runtime + except ImportError: + return + reference = reference_runtime.InferenceSession( + str(path), providers=["CPUExecutionProvider"] + ).run(None, {"input": source})[0] + np.testing.assert_array_equal(bound, reference) + + +@pytest.mark.parametrize("dtype", [np.uint8, np.uint16, np.float32]) +def test_tensor_zero_copy_dlpack_import_retains_producer(dtype): + source = np.arange(12, dtype=dtype).reshape(3, 4) + imported = sr.tensor_view_from_dlpack(source) + assert imported.shape == [3, 4] + assert imported.dtype == np.dtype(dtype).name + assert imported.version[0] == 1 + del source + copied = np.from_dlpack(imported.copy()) + np.testing.assert_array_equal(copied, np.arange(12, dtype=dtype).reshape(3, 4)) + assert "DLPackTensorView(" in repr(imported) diff --git a/crates/spatialrust-tensor/Cargo.toml b/crates/spatialrust-tensor/Cargo.toml new file mode 100644 index 0000000..667ca8d --- /dev/null +++ b/crates/spatialrust-tensor/Cargo.toml @@ -0,0 +1,30 @@ +[package] +name = "spatialrust-tensor" +version.workspace = true +edition.workspace = true +license.workspace = true +authors.workspace = true +repository.workspace = true +rust-version.workspace = true +description = "Small tensor metadata and explicit CPU ownership contracts for SpatialRust" + +[features] +default = [] +dlpack = [] +image = ["dep:spatialrust-image"] +spatial = ["dep:spatialrust-core"] + +[dependencies] +spatialrust-image = { workspace = true, optional = true } +spatialrust-core = { workspace = true, optional = true } +bytemuck.workspace = true +thiserror.workspace = true + +[dev-dependencies] +proptest.workspace = true +criterion.workspace = true + +[[bench]] +name = "image_bridge" +harness = false +required-features = ["image"] diff --git a/crates/spatialrust-tensor/benches/image_bridge.rs b/crates/spatialrust-tensor/benches/image_bridge.rs new file mode 100644 index 0000000..5b5488c --- /dev/null +++ b/crates/spatialrust-tensor/benches/image_bridge.rs @@ -0,0 +1,30 @@ +use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; +use spatialrust_image::{Image, ImageView}; +use spatialrust_tensor::{interleaved_image_view, pack_interleaved_image}; + +fn benchmark_image_bridge(c: &mut Criterion) { + for &(name, width, height) in &[("640p", 640, 480), ("1080p", 1920, 1080), ("4k", 3840, 2160)] { + let packed = Image::::try_new(width, height, vec![127; width * height * 3]).unwrap(); + let row_stride = width * 3 + 16; + let padded = vec![127_u8; row_stride * height]; + let strided = ImageView::::new(width, height, row_stride, &padded).unwrap(); + + let mut zero_copy = c.benchmark_group("tensor_image_zero_copy"); + zero_copy.throughput(Throughput::Elements((width * height) as u64)); + zero_copy.bench_function(BenchmarkId::from_parameter(name), |b| { + b.iter(|| black_box(interleaved_image_view(black_box(&packed)).unwrap())); + }); + zero_copy.finish(); + + let mut packing = c.benchmark_group("tensor_image_pack_strided"); + packing.sample_size(10); + packing.throughput(Throughput::Bytes((width * height * 3) as u64)); + packing.bench_function(BenchmarkId::from_parameter(name), |b| { + b.iter(|| black_box(pack_interleaved_image(black_box(strided)).unwrap())); + }); + packing.finish(); + } +} + +criterion_group!(benches, benchmark_image_bridge); +criterion_main!(benches); diff --git a/crates/spatialrust-tensor/src/dlpack.rs b/crates/spatialrust-tensor/src/dlpack.rs new file mode 100644 index 0000000..4bd2d71 --- /dev/null +++ b/crates/spatialrust-tensor/src/dlpack.rs @@ -0,0 +1,606 @@ +//! Audited DLPack major-version 1 CPU ownership boundary. + +use std::{ + ffi::c_void, + mem::ManuallyDrop, + ptr::{self, NonNull}, + slice, +}; + +use crate::{ + DataType, DataTypeCode, Device, DeviceKind, TensorBuffer, TensorDescriptor, TensorError, + TensorStorage, TensorView, +}; + +/// DLPack ABI major version supported by this boundary. +pub const DLPACK_MAJOR: u32 = 1; +/// Baseline DLPack minor version emitted by this boundary. +pub const DLPACK_MINOR: u32 = 0; + +const FLAG_READ_ONLY: u64 = 1; +const MAX_RANK: usize = 64; + +/// Errors raised while validating a DLPack managed tensor. +#[derive(Debug, thiserror::Error)] +pub enum DlpackError { + /// A null managed-tensor pointer was supplied. + #[error("DLPack managed tensor pointer is null")] + NullManagedTensor, + /// The ABI major version is incompatible. + #[error("unsupported DLPack ABI version {major}.{minor}; expected major {DLPACK_MAJOR}")] + IncompatibleVersion { + /// Producer major version. + major: u32, + /// Producer minor version. + minor: u32, + }, + /// Rank is negative or exceeds the defensive limit. + #[error("invalid DLPack tensor rank {0}")] + InvalidRank(i32), + /// A ranked tensor has no shape pointer. + #[error("DLPack tensor shape pointer is null for non-zero rank")] + NullShape, + /// A dimension is negative or cannot be represented by `usize`. + #[error("invalid DLPack shape dimension {0}")] + InvalidDimension(i64), + /// The data type code is unsupported or not byte-addressable. + #[error("unsupported DLPack dtype code={code}, bits={bits}, lanes={lanes}")] + UnsupportedDataType { + /// Raw DLPack code. + code: u8, + /// Bits per lane. + bits: u8, + /// Vector lanes. + lanes: u16, + }, + /// The device code is unknown to this DLPack minor implementation. + #[error("unsupported DLPack device type {0}")] + UnsupportedDevice(i32), + /// A non-empty host tensor has no data pointer. + #[error("non-empty DLPack host tensor has a null data pointer")] + NullData, + /// A DLPack integer does not fit the host representation. + #[error("DLPack metadata does not fit the host integer representation")] + IntegerConversion, + /// The decoded tensor layout is invalid. + #[error(transparent)] + Tensor(#[from] TensorError), +} + +#[repr(C)] +#[derive(Clone, Copy)] +struct RawVersion { + major: u32, + minor: u32, +} + +#[repr(C)] +#[derive(Clone, Copy)] +struct RawDevice { + device_type: i32, + device_id: i32, +} + +#[repr(C)] +#[derive(Clone, Copy)] +struct RawDataType { + code: u8, + bits: u8, + lanes: u16, +} + +#[repr(C)] +struct RawTensor { + data: *mut c_void, + device: RawDevice, + ndim: i32, + dtype: RawDataType, + shape: *mut i64, + strides: *mut i64, + byte_offset: u64, +} + +#[repr(C)] +struct RawManagedTensorVersioned { + version: RawVersion, + manager_ctx: *mut c_void, + deleter: Option, + flags: u64, + dl_tensor: RawTensor, +} + +struct ExportContext { + _allocation: TensorStorage, + _shape: Box<[i64]>, + _strides: Box<[i64]>, +} + +/// Owner for a versioned DLPack managed tensor exported without copying CPU data. +/// +/// Dropping this value calls the DLPack deleter. [`Self::into_raw`] transfers +/// that responsibility to a capsule or another consumer. +pub struct DlpackExport { + raw: NonNull, +} + +/// Calls the producer deleter for a raw pointer previously returned by +/// [`DlpackExport::into_raw`]. +/// +/// # Safety +/// +/// `raw` must still carry exclusive deleter responsibility for a live +/// `DLManagedTensorVersioned*`. It must not be used after this call. +pub unsafe fn release_dlpack_raw(raw: *mut c_void) { + // SAFETY: forwarded from the public ownership contract above. + unsafe { call_deleter(raw.cast()) }; +} + +impl DlpackExport { + /// Shares an owned CPU tensor allocation with a DLPack consumer without copying. + pub fn from_tensor(tensor: &TensorBuffer) -> Result { + let descriptor = tensor.descriptor(); + if !descriptor.device().is_host_accessible() { + return Err(TensorError::DeviceNotHostAccessible(descriptor.device()).into()); + } + let shape = descriptor + .shape() + .iter() + .map(|&dimension| i64::try_from(dimension).map_err(|_| DlpackError::IntegerConversion)) + .collect::, _>>()? + .into_boxed_slice(); + let strides = match descriptor.strides() { + Some(values) => values + .iter() + .map(|&stride| i64::try_from(stride).map_err(|_| DlpackError::IntegerConversion)) + .collect::, _>>()?, + None => compact_strides(descriptor.shape())?, + } + .into_boxed_slice(); + let ndim = + i32::try_from(descriptor.shape().len()).map_err(|_| DlpackError::IntegerConversion)?; + let byte_offset = + u64::try_from(descriptor.byte_offset()).map_err(|_| DlpackError::IntegerConversion)?; + let allocation = tensor.shared_allocation(); + let data = if allocation.is_empty() { + ptr::null_mut() + } else { + allocation.as_ptr().cast_mut().cast() + }; + let shape_ptr = if shape.is_empty() { ptr::null_mut() } else { shape.as_ptr().cast_mut() }; + let strides_ptr = + if strides.is_empty() { ptr::null_mut() } else { strides.as_ptr().cast_mut() }; + let context = + Box::new(ExportContext { _allocation: allocation, _shape: shape, _strides: strides }); + let manager_ctx = Box::into_raw(context).cast(); + let raw = Box::new(RawManagedTensorVersioned { + version: RawVersion { major: DLPACK_MAJOR, minor: DLPACK_MINOR }, + manager_ctx, + deleter: Some(delete_export), + flags: FLAG_READ_ONLY, + dl_tensor: RawTensor { + data, + device: encode_device(descriptor.device()), + ndim, + dtype: encode_dtype(descriptor.dtype()), + shape: shape_ptr, + strides: strides_ptr, + byte_offset, + }, + }); + Ok(Self { raw: NonNull::from(Box::leak(raw)) }) + } + + /// Returns the opaque managed-tensor pointer without transferring ownership. + pub fn as_raw(&self) -> *mut c_void { + self.raw.as_ptr().cast() + } + + /// Transfers deleter responsibility to an external DLPack consumer. + pub fn into_raw(self) -> *mut c_void { + let this = ManuallyDrop::new(self); + this.raw.as_ptr().cast() + } +} + +impl Drop for DlpackExport { + fn drop(&mut self) { + // SAFETY: `DlpackExport` uniquely owns deleter responsibility until `into_raw`. + unsafe { call_deleter(self.raw.as_ptr()) }; + } +} + +/// Validated owner of a DLPack producer's managed tensor. +#[derive(Debug)] +pub struct DlpackImport { + raw: NonNull, + descriptor: TensorDescriptor, + allocation_len: usize, + data: *const u8, + version: (u32, u32), + flags: u64, +} + +impl DlpackImport { + /// Takes deleter ownership of a DLPack managed tensor and validates its host view. + /// + /// # Safety + /// + /// `raw` must be a live, exclusively transferred `DLManagedTensorVersioned*` + /// produced according to DLPack. The caller must not use or delete it after + /// this call, whether validation succeeds or fails. + pub unsafe fn from_raw(raw: *mut c_void) -> Result { + let raw = NonNull::new(raw.cast::()) + .ok_or(DlpackError::NullManagedTensor)?; + let guard = IncomingGuard { raw: Some(raw) }; + + // SAFETY: the caller promises a live versioned header. DLPack guarantees + // version and deleter positions remain accessible across major mismatch. + let managed = unsafe { raw.as_ref() }; + let version = (managed.version.major, managed.version.minor); + if version.0 != DLPACK_MAJOR { + return Err(DlpackError::IncompatibleVersion { major: version.0, minor: version.1 }); + } + let tensor = &managed.dl_tensor; + if tensor.ndim < 0 || tensor.ndim as usize > MAX_RANK { + return Err(DlpackError::InvalidRank(tensor.ndim)); + } + let rank = tensor.ndim as usize; + if rank != 0 && tensor.shape.is_null() { + return Err(DlpackError::NullShape); + } + let shape_values = if rank == 0 { + &[][..] + } else { + // SAFETY: producer contract supplies `ndim` readable shape entries. + unsafe { slice::from_raw_parts(tensor.shape, rank) } + }; + let shape = shape_values + .iter() + .map(|&dimension| { + usize::try_from(dimension).map_err(|_| DlpackError::InvalidDimension(dimension)) + }) + .collect::, _>>()?; + let strides = if tensor.strides.is_null() { + None + } else { + // SAFETY: producer contract supplies `ndim` readable stride entries. + let values = unsafe { slice::from_raw_parts(tensor.strides, rank) }; + Some( + values + .iter() + .map(|&stride| { + isize::try_from(stride).map_err(|_| DlpackError::IntegerConversion) + }) + .collect::, _>>()?, + ) + }; + let dtype = decode_dtype(tensor.dtype)?; + let device = decode_device(tensor.device)?; + let byte_offset = + usize::try_from(tensor.byte_offset).map_err(|_| DlpackError::IntegerConversion)?; + let descriptor = match strides { + Some(strides) => { + TensorDescriptor::try_strided(dtype, shape, strides, byte_offset, device)? + } + None => { + let mut descriptor = TensorDescriptor::contiguous(dtype, shape, device); + if byte_offset != 0 { + descriptor = TensorDescriptor::try_strided( + dtype, + descriptor.shape().to_vec(), + compact_strides_isize(descriptor.shape())?, + byte_offset, + device, + )?; + } + descriptor + } + }; + let range = descriptor.required_byte_range()?; + if range.end != 0 && tensor.data.is_null() { + return Err(DlpackError::NullData); + } + let raw = guard.disarm(); + Ok(Self { + raw, + descriptor, + allocation_len: range.end, + data: tensor.data.cast(), + version, + flags: managed.flags, + }) + } + + /// Returns producer ABI major and minor versions. + pub const fn version(&self) -> (u32, u32) { + self.version + } + + /// Returns raw DLPack flags. + pub const fn flags(&self) -> u64 { + self.flags + } + + /// Returns validated tensor metadata. + pub const fn descriptor(&self) -> &TensorDescriptor { + &self.descriptor + } + + /// Borrows the imported host allocation without copying it. + pub fn view(&self) -> Result, DlpackError> { + let bytes = if self.allocation_len == 0 { + &[][..] + } else { + // SAFETY: `from_raw` validated the DLPack producer contract and the + // managed tensor remains alive until this owner is dropped. + unsafe { slice::from_raw_parts(self.data, self.allocation_len) } + }; + Ok(TensorView::try_new(bytes, self.descriptor.clone())?) + } +} + +impl Drop for DlpackImport { + fn drop(&mut self) { + // SAFETY: this owner received exclusive deleter responsibility in `from_raw`. + unsafe { call_deleter(self.raw.as_ptr()) }; + } +} + +struct IncomingGuard { + raw: Option>, +} + +impl IncomingGuard { + fn disarm(mut self) -> NonNull { + self.raw.take().expect("incoming pointer is present") + } +} + +impl Drop for IncomingGuard { + fn drop(&mut self) { + if let Some(raw) = self.raw { + // SAFETY: the guard owns the transferred pointer on all error paths. + unsafe { call_deleter(raw.as_ptr()) }; + } + } +} + +unsafe extern "C" fn delete_export(raw: *mut RawManagedTensorVersioned) { + if raw.is_null() { + return; + } + // SAFETY: this function is installed only for allocations built by + // `DlpackExport::from_tensor` and is called exactly once by ownership contract. + let managed = unsafe { Box::from_raw(raw) }; + if !managed.manager_ctx.is_null() { + // SAFETY: manager_ctx was created with Box::into_raw for ExportContext. + drop(unsafe { Box::from_raw(managed.manager_ctx.cast::()) }); + } +} + +unsafe fn call_deleter(raw: *mut RawManagedTensorVersioned) { + if raw.is_null() { + return; + } + // SAFETY: caller owns a live managed-tensor pointer. + if let Some(deleter) = unsafe { (*raw).deleter } { + // SAFETY: deleter belongs to this exact managed tensor. + unsafe { deleter(raw) }; + } +} + +fn compact_strides(shape: &[usize]) -> Result, DlpackError> { + let mut output = vec![0; shape.len()]; + let mut stride = 1_i64; + for (index, &dimension) in shape.iter().enumerate().rev() { + output[index] = stride; + stride = stride + .checked_mul( + i64::try_from(dimension.max(1)).map_err(|_| DlpackError::IntegerConversion)?, + ) + .ok_or(DlpackError::IntegerConversion)?; + } + Ok(output) +} + +fn compact_strides_isize(shape: &[usize]) -> Result, DlpackError> { + compact_strides(shape)? + .into_iter() + .map(|stride| isize::try_from(stride).map_err(|_| DlpackError::IntegerConversion)) + .collect() +} + +fn encode_dtype(dtype: DataType) -> RawDataType { + RawDataType { code: dtype.code() as u8, bits: dtype.bits(), lanes: dtype.lanes() } +} + +fn decode_dtype(raw: RawDataType) -> Result { + let code = match raw.code { + 0 => DataTypeCode::Int, + 1 => DataTypeCode::UInt, + 2 => DataTypeCode::Float, + 4 => DataTypeCode::BFloat, + 5 => DataTypeCode::Complex, + 6 => DataTypeCode::Bool, + _ => { + return Err(DlpackError::UnsupportedDataType { + code: raw.code, + bits: raw.bits, + lanes: raw.lanes, + }) + } + }; + DataType::try_new(code, raw.bits, raw.lanes).map_err(|_| DlpackError::UnsupportedDataType { + code: raw.code, + bits: raw.bits, + lanes: raw.lanes, + }) +} + +fn encode_device(device: Device) -> RawDevice { + let device_type = match device.kind { + DeviceKind::Cpu => 1, + DeviceKind::Cuda => 2, + DeviceKind::CudaHost => 3, + DeviceKind::OpenCl => 4, + DeviceKind::Vulkan => 7, + DeviceKind::Metal => 8, + DeviceKind::Vpi => 9, + DeviceKind::Rocm => 10, + DeviceKind::RocmHost => 11, + DeviceKind::External => 12, + DeviceKind::CudaManaged => 13, + DeviceKind::OneApi => 14, + DeviceKind::WebGpu => 15, + DeviceKind::Hexagon => 16, + DeviceKind::Maia => 17, + DeviceKind::Trainium => 18, + DeviceKind::Tpu => 19, + DeviceKind::TpuHost => 20, + }; + RawDevice { device_type, device_id: device.id } +} + +fn decode_device(raw: RawDevice) -> Result { + let kind = match raw.device_type { + 1 => DeviceKind::Cpu, + 2 => DeviceKind::Cuda, + 3 => DeviceKind::CudaHost, + 4 => DeviceKind::OpenCl, + 7 => DeviceKind::Vulkan, + 8 => DeviceKind::Metal, + 9 => DeviceKind::Vpi, + 10 => DeviceKind::Rocm, + 11 => DeviceKind::RocmHost, + 12 => DeviceKind::External, + 13 => DeviceKind::CudaManaged, + 14 => DeviceKind::OneApi, + 15 => DeviceKind::WebGpu, + 16 => DeviceKind::Hexagon, + 17 => DeviceKind::Maia, + 18 => DeviceKind::Trainium, + 19 => DeviceKind::Tpu, + 20 => DeviceKind::TpuHost, + other => return Err(DlpackError::UnsupportedDevice(other)), + }; + Ok(Device { kind, id: raw.device_id }) +} + +#[cfg(test)] +mod tests { + use super::{DlpackError, DlpackExport, DlpackImport, DLPACK_MAJOR, DLPACK_MINOR}; + use crate::{DataType, Device, TensorBuffer, TensorDescriptor}; + + #[test] + fn owned_cpu_roundtrip_is_zero_copy_and_versioned() { + let tensor = TensorBuffer::try_new( + (0_u8..24).collect(), + TensorDescriptor::contiguous(DataType::F32, vec![2, 3], Device::CPU), + ) + .unwrap(); + let original = tensor.allocation_bytes().as_ptr(); + let allocation = tensor.shared_allocation(); + let export = DlpackExport::from_tensor(&tensor).unwrap(); + assert_eq!(allocation.strong_count(), 3); + // SAFETY: into_raw transfers the live export exactly once. + let imported = unsafe { DlpackImport::from_raw(export.into_raw()) }.unwrap(); + assert_eq!(imported.version(), (DLPACK_MAJOR, DLPACK_MINOR)); + let view = imported.view().unwrap(); + assert_eq!(view.descriptor().shape(), &[2, 3]); + assert_eq!(view.allocation_bytes().as_ptr(), original); + drop(imported); + assert_eq!(allocation.strong_count(), 2); + } + + #[test] + fn negative_stride_and_byte_offset_roundtrip() { + let descriptor = + TensorDescriptor::try_strided(DataType::U8, vec![4], vec![-1], 3, Device::CPU).unwrap(); + let tensor = TensorBuffer::try_new(vec![10, 20, 30, 40], descriptor).unwrap(); + let export = DlpackExport::from_tensor(&tensor).unwrap(); + // SAFETY: the export pointer is transferred exactly once. + let imported = unsafe { DlpackImport::from_raw(export.into_raw()) }.unwrap(); + let view = imported.view().unwrap(); + assert_eq!(view.descriptor().strides(), Some(&[-1][..])); + assert_eq!(view.descriptor().byte_offset(), 3); + assert_eq!(view.allocation_bytes(), &[10, 20, 30, 40]); + } + + #[test] + fn major_mismatch_is_rejected_and_deleted() { + let tensor = TensorBuffer::try_new( + vec![1], + TensorDescriptor::contiguous(DataType::U8, vec![1], Device::CPU), + ) + .unwrap(); + let allocation = tensor.shared_allocation(); + let export = DlpackExport::from_tensor(&tensor).unwrap(); + let raw = export.into_raw().cast::(); + // SAFETY: test exclusively owns this live export and only changes its version header. + unsafe { (*raw).version.major = 99 }; + // SAFETY: the pointer is still live and transferred exactly once. + let error = unsafe { DlpackImport::from_raw(raw.cast()) }.unwrap_err(); + assert!(matches!(error, DlpackError::IncompatibleVersion { major: 99, .. })); + assert_eq!(allocation.strong_count(), 2); + } + + #[test] + fn malformed_dtype_is_rejected_and_deleted() { + let tensor = TensorBuffer::try_new( + vec![1], + TensorDescriptor::contiguous(DataType::U8, vec![1], Device::CPU), + ) + .unwrap(); + let allocation = tensor.shared_allocation(); + let raw = DlpackExport::from_tensor(&tensor) + .unwrap() + .into_raw() + .cast::(); + // SAFETY: test owns the live export and corrupts only metadata under test. + unsafe { (*raw).dl_tensor.dtype.code = 255 }; + // SAFETY: the corrupted but live pointer is transferred exactly once. + let error = unsafe { DlpackImport::from_raw(raw.cast()) }.unwrap_err(); + assert!(matches!(error, DlpackError::UnsupportedDataType { code: 255, .. })); + assert_eq!(allocation.strong_count(), 2); + } + + #[test] + fn null_nonempty_data_is_rejected_and_deleted() { + let tensor = TensorBuffer::try_new( + vec![1, 2], + TensorDescriptor::contiguous(DataType::U8, vec![2], Device::CPU), + ) + .unwrap(); + let allocation = tensor.shared_allocation(); + let raw = DlpackExport::from_tensor(&tensor) + .unwrap() + .into_raw() + .cast::(); + // SAFETY: test owns the live export and corrupts only metadata under test. + unsafe { (*raw).dl_tensor.data = std::ptr::null_mut() }; + // SAFETY: the corrupted but live pointer is transferred exactly once. + let error = unsafe { DlpackImport::from_raw(raw.cast()) }.unwrap_err(); + assert!(matches!(error, DlpackError::NullData)); + assert_eq!(allocation.strong_count(), 2); + } + + #[test] + fn negative_shape_is_rejected_and_deleted() { + let tensor = TensorBuffer::try_new( + vec![1], + TensorDescriptor::contiguous(DataType::U8, vec![1], Device::CPU), + ) + .unwrap(); + let allocation = tensor.shared_allocation(); + let raw = DlpackExport::from_tensor(&tensor) + .unwrap() + .into_raw() + .cast::(); + // SAFETY: exported rank is one and shape points to one writable context entry. + unsafe { *(*raw).dl_tensor.shape = -1 }; + // SAFETY: the corrupted but live pointer is transferred exactly once. + let error = unsafe { DlpackImport::from_raw(raw.cast()) }.unwrap_err(); + assert!(matches!(error, DlpackError::InvalidDimension(-1))); + assert_eq!(allocation.strong_count(), 2); + } +} diff --git a/crates/spatialrust-tensor/src/image.rs b/crates/spatialrust-tensor/src/image.rs new file mode 100644 index 0000000..4ad614d --- /dev/null +++ b/crates/spatialrust-tensor/src/image.rs @@ -0,0 +1,153 @@ +//! Explicit bridges between typed CPU images and generic tensors. + +use bytemuck::{cast_slice, Pod}; +use spatialrust_image::{Image, ImageView, PlanarImage, PlanarImageView}; + +use crate::{DataType, Device, TensorBuffer, TensorDescriptor, TensorError, TensorView}; + +/// A native-endian scalar that has a stable tensor dtype and no invalid bit patterns. +pub trait TensorElement: Pod { + /// Tensor dtype corresponding to this Rust scalar. + const DTYPE: DataType; +} + +macro_rules! tensor_elements { + ($($type:ty => $dtype:expr),+ $(,)?) => { + $(impl TensorElement for $type { + const DTYPE: DataType = $dtype; + })+ + }; +} + +tensor_elements! { + u8 => DataType::U8, + u16 => DataType::U16, + i8 => DataType::I8, + i16 => DataType::I16, + i32 => DataType::I32, + i64 => DataType::I64, + f32 => DataType::F32, + f64 => DataType::F64, +} + +/// Borrows a packed interleaved image as a zero-copy `[height, width, channels]` tensor. +pub fn interleaved_image_view( + image: &Image, +) -> Result, TensorError> { + let descriptor = TensorDescriptor::contiguous( + T::DTYPE, + vec![image.height(), image.width(), CHANNELS], + Device::CPU, + ); + TensorView::try_new(cast_slice(image.as_slice()), descriptor) +} + +/// Borrows a packed planar image as a zero-copy `[channels, height, width]` tensor. +pub fn planar_image_view( + image: &PlanarImage, +) -> Result, TensorError> { + let descriptor = TensorDescriptor::contiguous( + T::DTYPE, + vec![CHANNELS, image.height(), image.width()], + Device::CPU, + ); + TensorView::try_new(cast_slice(image.as_slice()), descriptor) +} + +/// Explicitly packs a possibly strided interleaved image view into an owned HWC tensor. +pub fn pack_interleaved_image( + image: ImageView<'_, T, CHANNELS>, +) -> Result { + let scalar_count = image + .width() + .checked_mul(image.height()) + .and_then(|value| value.checked_mul(CHANNELS)) + .ok_or(TensorError::LayoutOverflow)?; + let byte_count = + scalar_count.checked_mul(T::DTYPE.element_size()).ok_or(TensorError::LayoutOverflow)?; + let mut bytes = Vec::with_capacity(byte_count); + for y in 0..image.height() { + let row = image.row(y).expect("row index is within the image height"); + bytes.extend_from_slice(cast_slice(row)); + } + let descriptor = TensorDescriptor::contiguous( + T::DTYPE, + vec![image.height(), image.width(), CHANNELS], + Device::CPU, + ); + TensorBuffer::try_new(bytes, descriptor) +} + +/// Explicitly packs a possibly strided planar image view into an owned CHW tensor. +pub fn pack_planar_image( + image: PlanarImageView<'_, T, CHANNELS>, +) -> Result { + let scalar_count = image + .width() + .checked_mul(image.height()) + .and_then(|value| value.checked_mul(CHANNELS)) + .ok_or(TensorError::LayoutOverflow)?; + let byte_count = + scalar_count.checked_mul(T::DTYPE.element_size()).ok_or(TensorError::LayoutOverflow)?; + let mut bytes = Vec::with_capacity(byte_count); + for channel in 0..CHANNELS { + for y in 0..image.height() { + for x in 0..image.width() { + let value = image + .get(channel, x, y) + .expect("channel and pixel coordinates are within the image"); + bytes.extend_from_slice(bytemuck::bytes_of(value)); + } + } + } + let descriptor = TensorDescriptor::contiguous( + T::DTYPE, + vec![CHANNELS, image.height(), image.width()], + Device::CPU, + ); + TensorBuffer::try_new(bytes, descriptor) +} + +#[cfg(test)] +mod tests { + use super::{ + interleaved_image_view, pack_interleaved_image, pack_planar_image, planar_image_view, + }; + use spatialrust_image::{Image, ImageRegion, ImageView, PlanarImage, PlanarImageView}; + + #[test] + fn packed_images_are_zero_copy_hwc_and_chw() { + let interleaved = Image::::try_new(2, 2, (0..12).collect()).unwrap(); + let tensor = interleaved_image_view(&interleaved).unwrap(); + assert_eq!(tensor.descriptor().shape(), &[2, 2, 3]); + assert_eq!(tensor.allocation_bytes().as_ptr(), interleaved.as_slice().as_ptr().cast()); + + let planar = + PlanarImage::::try_new(2, 2, (0..12).map(|v| v as f32).collect()).unwrap(); + let tensor = planar_image_view(&planar).unwrap(); + assert_eq!(tensor.descriptor().shape(), &[3, 2, 2]); + assert_eq!(tensor.allocation_bytes().as_ptr(), planar.as_slice().as_ptr().cast()); + } + + #[test] + fn strided_interleaved_roi_is_explicitly_packed() { + let storage = (0_u8..30).collect::>(); + let parent = ImageView::::new(3, 3, 10, &storage).unwrap(); + let roi = parent.subview(ImageRegion::new(1, 1, 2, 2)).unwrap(); + let tensor = pack_interleaved_image(roi).unwrap(); + assert_eq!(tensor.descriptor().shape(), &[2, 2, 3]); + assert_eq!(tensor.allocation_bytes(), &[13, 14, 15, 16, 17, 18, 23, 24, 25, 26, 27, 28]); + } + + #[test] + fn padded_planar_view_is_explicitly_packed() { + let storage = (0_u16..24).collect::>(); + let view = PlanarImageView::::new(2, 2, 3, 12, &storage).unwrap(); + let tensor = pack_planar_image(view).unwrap(); + assert_eq!(tensor.descriptor().shape(), &[2, 2, 2]); + assert_eq!( + bytemuck::cast_slice::(tensor.allocation_bytes()), + &[0, 1, 3, 4, 12, 13, 15, 16] + ); + } +} diff --git a/crates/spatialrust-tensor/src/lib.rs b/crates/spatialrust-tensor/src/lib.rs new file mode 100644 index 0000000..505b49d --- /dev/null +++ b/crates/spatialrust-tensor/src/lib.rs @@ -0,0 +1,763 @@ +//! Small, runtime-independent tensor descriptors and CPU storage views. +//! +//! Shape, element strides, byte offset, device, and ownership are explicit. +//! This crate never uploads, downloads, or otherwise migrates storage. DLPack +//! FFI and image bridges are additive features built on this data model. + +#![deny(unsafe_code)] +#![warn(missing_docs)] + +use std::{any::Any, fmt::Debug, ops::Range, sync::Arc}; + +#[cfg(feature = "image")] +mod image; +#[cfg(feature = "image")] +pub use image::{ + interleaved_image_view, pack_interleaved_image, pack_planar_image, planar_image_view, + TensorElement, +}; +#[cfg(feature = "spatial")] +mod spatial; +#[cfg(feature = "spatial")] +pub use spatial::{spatial_f32_field_view, SpatialTensorBridgeError}; +#[cfg(feature = "dlpack")] +#[allow(unsafe_code)] +mod dlpack; +#[cfg(feature = "dlpack")] +pub use dlpack::{ + release_dlpack_raw, DlpackError, DlpackExport, DlpackImport, DLPACK_MAJOR, DLPACK_MINOR, +}; + +/// Tensor construction and layout errors. +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +pub enum TensorError { + /// The data type has no bits or lanes, or cannot address whole bytes. + #[error("invalid data type: bits and lanes must be non-zero and form whole bytes")] + InvalidDataType, + /// The stride rank differs from the shape rank. + #[error("stride rank {strides} differs from shape rank {shape}")] + RankMismatch { + /// Number of shape dimensions. + shape: usize, + /// Number of strides. + strides: usize, + }, + /// Shape, stride, or byte-range arithmetic overflowed. + #[error("tensor layout overflows addressable memory")] + LayoutOverflow, + /// The byte offset or reachable tensor span lies outside storage. + #[error("tensor byte range {start}..{end} lies outside {available} bytes")] + StorageOutOfBounds { + /// First reachable byte. + start: i128, + /// Exclusive final reachable byte. + end: i128, + /// Available storage size. + available: usize, + }, + /// A host slice was paired with a device that is not host-accessible. + #[error("device {0:?} is not directly host-accessible")] + DeviceNotHostAccessible(Device), + /// Typed storage does not match its descriptor dtype. + #[error("typed storage requires {expected:?}, descriptor declares {actual:?}")] + DataTypeMismatch { + /// Storage dtype. + expected: DataType, + /// Descriptor dtype. + actual: DataType, + }, +} + +/// DLPack-compatible scalar category without depending on a runtime header. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +#[repr(u8)] +pub enum DataTypeCode { + /// Signed integer. + Int = 0, + /// Unsigned integer. + UInt = 1, + /// IEEE floating point. + Float = 2, + /// Brain floating point. + BFloat = 4, + /// Complex floating point. + Complex = 5, + /// Boolean value. + Bool = 6, +} + +/// Scalar category, bit width, and vector lane count. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct DataType { + code: DataTypeCode, + bits: u8, + lanes: u16, +} + +impl DataType { + /// Unsigned 8-bit scalar. + pub const U8: Self = Self::new_unchecked(DataTypeCode::UInt, 8, 1); + /// Unsigned 16-bit scalar. + pub const U16: Self = Self::new_unchecked(DataTypeCode::UInt, 16, 1); + /// Unsigned 32-bit scalar. + pub const U32: Self = Self::new_unchecked(DataTypeCode::UInt, 32, 1); + /// Signed 8-bit scalar. + pub const I8: Self = Self::new_unchecked(DataTypeCode::Int, 8, 1); + /// Signed 16-bit scalar. + pub const I16: Self = Self::new_unchecked(DataTypeCode::Int, 16, 1); + /// Signed 32-bit scalar. + pub const I32: Self = Self::new_unchecked(DataTypeCode::Int, 32, 1); + /// Signed 64-bit scalar. + pub const I64: Self = Self::new_unchecked(DataTypeCode::Int, 64, 1); + /// IEEE binary16 scalar. + pub const F16: Self = Self::new_unchecked(DataTypeCode::Float, 16, 1); + /// Brain floating-point 16-bit scalar. + pub const BF16: Self = Self::new_unchecked(DataTypeCode::BFloat, 16, 1); + /// IEEE binary32 scalar. + pub const F32: Self = Self::new_unchecked(DataTypeCode::Float, 32, 1); + /// IEEE binary64 scalar. + pub const F64: Self = Self::new_unchecked(DataTypeCode::Float, 64, 1); + /// Eight-bit boolean scalar, matching common DLPack producers. + pub const BOOL: Self = Self::new_unchecked(DataTypeCode::Bool, 8, 1); + + const fn new_unchecked(code: DataTypeCode, bits: u8, lanes: u16) -> Self { + Self { code, bits, lanes } + } + + /// Creates a byte-addressable scalar or vector data type. + pub fn try_new(code: DataTypeCode, bits: u8, lanes: u16) -> Result { + let total_bits = usize::from(bits) + .checked_mul(usize::from(lanes)) + .ok_or(TensorError::InvalidDataType)?; + if bits == 0 || lanes == 0 || total_bits % 8 != 0 { + return Err(TensorError::InvalidDataType); + } + Ok(Self { code, bits, lanes }) + } + + /// Returns the scalar category. + pub const fn code(self) -> DataTypeCode { + self.code + } + + /// Returns bits per lane. + pub const fn bits(self) -> u8 { + self.bits + } + + /// Returns the vector lane count. + pub const fn lanes(self) -> u16 { + self.lanes + } + + /// Returns bytes occupied by one tensor element. + pub fn element_size(self) -> usize { + usize::from(self.bits) * usize::from(self.lanes) / 8 + } +} + +/// Device category represented independently of any execution backend. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum DeviceKind { + /// Ordinary CPU memory. + Cpu, + /// CUDA device memory. + Cuda, + /// CUDA-pinned host memory. + CudaHost, + /// OpenCL device memory. + OpenCl, + /// Vulkan device memory. + Vulkan, + /// Metal device memory. + Metal, + /// Verilog simulator buffer. + Vpi, + /// ROCm device memory. + Rocm, + /// ROCm-pinned host memory. + RocmHost, + /// Backend-specific external memory. + External, + /// CUDA managed memory. + CudaManaged, + /// oneAPI device memory. + OneApi, + /// WebGPU device memory. + WebGpu, + /// Hexagon device memory. + Hexagon, + /// Microsoft MAIA device memory. + Maia, + /// AWS Trainium device memory. + Trainium, + /// Google TPU device memory. + Tpu, + /// Google TPU pinned host memory. + TpuHost, +} + +/// Device category and backend-local ordinal. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct Device { + /// Device category. + pub kind: DeviceKind, + /// Backend-local device ordinal. + pub id: i32, +} + +impl Device { + /// Main CPU device. + pub const CPU: Self = Self { kind: DeviceKind::Cpu, id: 0 }; + + /// Returns whether Rust may safely expose this memory as a host byte slice. + pub const fn is_host_accessible(self) -> bool { + matches!( + self.kind, + DeviceKind::Cpu | DeviceKind::CudaHost | DeviceKind::RocmHost | DeviceKind::TpuHost + ) + } +} + +/// Whether a tensor object owns or borrows its backing allocation. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Ownership { + /// The object owns and drops its storage. + Owned, + /// The object cannot outlive borrowed storage. + Borrowed, +} + +/// Shape, element strides, type, offset, and device for one tensor. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct TensorDescriptor { + dtype: DataType, + shape: Vec, + strides: Option>, + byte_offset: usize, + device: Device, +} + +impl TensorDescriptor { + /// Creates a compact C-order tensor on a device. + pub fn contiguous(dtype: DataType, shape: Vec, device: Device) -> Self { + Self { dtype, shape, strides: None, byte_offset: 0, device } + } + + /// Creates an explicitly strided tensor. Strides are measured in elements. + pub fn try_strided( + dtype: DataType, + shape: Vec, + strides: Vec, + byte_offset: usize, + device: Device, + ) -> Result { + if shape.len() != strides.len() { + return Err(TensorError::RankMismatch { shape: shape.len(), strides: strides.len() }); + } + let descriptor = Self { dtype, shape, strides: Some(strides), byte_offset, device }; + descriptor.required_byte_range()?; + Ok(descriptor) + } + + /// Returns the data type. + pub const fn dtype(&self) -> DataType { + self.dtype + } + + /// Returns dimensions in logical axis order. + pub fn shape(&self) -> &[usize] { + &self.shape + } + + /// Returns element strides, or `None` for compact C order. + pub fn strides(&self) -> Option<&[isize]> { + self.strides.as_deref() + } + + /// Returns the byte offset from the allocation start to logical element zero. + pub const fn byte_offset(&self) -> usize { + self.byte_offset + } + + /// Returns the storage device. + pub const fn device(&self) -> Device { + self.device + } + + /// Returns the number of logical elements, including one for a scalar. + pub fn element_count(&self) -> Result { + self.shape + .iter() + .try_fold(1usize, |count, &dimension| count.checked_mul(dimension)) + .ok_or(TensorError::LayoutOverflow) + } + + /// Returns whether the tensor is compact in C order. + pub fn is_c_contiguous(&self) -> bool { + let Some(strides) = &self.strides else { return true }; + let mut expected = 1usize; + for (&dimension, &stride) in self.shape.iter().zip(strides).rev() { + if dimension > 1 && usize::try_from(stride).ok() != Some(expected) { + return false; + } + let Some(next) = expected.checked_mul(dimension.max(1)) else { return false }; + expected = next; + } + true + } + + /// Computes the smallest allocation byte range reachable by the layout. + pub fn required_byte_range(&self) -> Result, TensorError> { + if self.element_count()? == 0 { + return Ok(self.byte_offset..self.byte_offset); + } + let item_size = + i128::try_from(self.dtype.element_size()).map_err(|_| TensorError::LayoutOverflow)?; + let origin = i128::try_from(self.byte_offset).map_err(|_| TensorError::LayoutOverflow)?; + let mut minimum = origin; + let mut maximum = origin; + if let Some(strides) = &self.strides { + for (&dimension, &stride) in self.shape.iter().zip(strides) { + let steps = i128::try_from(dimension.saturating_sub(1)) + .map_err(|_| TensorError::LayoutOverflow)?; + let stride_bytes = + (stride as i128).checked_mul(item_size).ok_or(TensorError::LayoutOverflow)?; + let delta = steps.checked_mul(stride_bytes).ok_or(TensorError::LayoutOverflow)?; + if delta < 0 { + minimum = minimum.checked_add(delta).ok_or(TensorError::LayoutOverflow)?; + } else { + maximum = maximum.checked_add(delta).ok_or(TensorError::LayoutOverflow)?; + } + } + } else { + let count = + i128::try_from(self.element_count()?).map_err(|_| TensorError::LayoutOverflow)?; + maximum = maximum + .checked_add((count - 1).checked_mul(item_size).ok_or(TensorError::LayoutOverflow)?) + .ok_or(TensorError::LayoutOverflow)?; + } + let end = maximum.checked_add(item_size).ok_or(TensorError::LayoutOverflow)?; + let start = usize::try_from(minimum).map_err(|_| TensorError::StorageOutOfBounds { + start: minimum, + end, + available: 0, + })?; + let end = usize::try_from(end).map_err(|_| TensorError::LayoutOverflow)?; + Ok(start..end) + } + + fn validate_storage(&self, available: usize) -> Result<(), TensorError> { + if !self.device.is_host_accessible() { + return Err(TensorError::DeviceNotHostAccessible(self.device)); + } + let range = self.required_byte_range()?; + if range.start > available || range.end > available { + return Err(TensorError::StorageOutOfBounds { + start: range.start as i128, + end: range.end as i128, + available, + }); + } + Ok(()) + } +} + +/// Lifetime-bound, zero-copy view of host-accessible tensor storage. +#[derive(Clone, Debug)] +pub struct TensorView<'a> { + bytes: &'a [u8], + descriptor: TensorDescriptor, +} + +impl<'a> TensorView<'a> { + /// Validates and borrows a host allocation without copying it. + pub fn try_new(bytes: &'a [u8], descriptor: TensorDescriptor) -> Result { + descriptor.validate_storage(bytes.len())?; + Ok(Self { bytes, descriptor }) + } + + /// Returns the complete borrowed allocation slice. + pub const fn allocation_bytes(&self) -> &'a [u8] { + self.bytes + } + + /// Returns tensor metadata. + pub const fn descriptor(&self) -> &TensorDescriptor { + &self.descriptor + } + + /// Reports borrowed ownership. + pub const fn ownership(&self) -> Ownership { + Ownership::Borrowed + } + + /// Performs an explicit host-to-host copy retaining the same layout. + pub fn to_owned_copy(&self) -> TensorBuffer { + TensorBuffer { + storage: copy_storage(self.bytes, self.descriptor.dtype()), + descriptor: self.descriptor.clone(), + } + } +} + +/// Owned host allocation paired with a validated tensor descriptor. +#[derive(Clone, Debug)] +pub struct TensorBuffer { + storage: TensorStorage, + descriptor: TensorDescriptor, +} + +#[derive(Clone, Debug)] +pub(crate) enum TensorStorage { + Bytes(Arc<[u8]>), + U16(Arc<[u16]>), + I16(Arc<[i16]>), + U32(Arc<[u32]>), + I32(Arc<[i32]>), + I64(Arc<[i64]>), + F32(Arc<[f32]>), + F64(Arc<[f64]>), + External(Arc), +} + +/// Runtime-owned, host-accessible storage retained without a runtime dependency. +/// +/// Backend crates use this boundary to keep an allocator or runtime value alive +/// while exposing its stable CPU allocation. Implementations must keep the +/// returned allocation address and length unchanged for their entire lifetime. +pub trait HostTensorStorage: Any + Debug + Send + Sync { + /// Returns the exact tensor element type stored by this allocation. + fn dtype(&self) -> DataType; + + /// Returns the complete host-accessible allocation. + fn allocation_bytes(&self) -> &[u8]; + + /// Supports backend-specific zero-copy reuse through checked downcasting. + fn as_any(&self) -> &dyn Any; +} + +impl TensorStorage { + pub(crate) fn as_bytes(&self) -> &[u8] { + match self { + Self::Bytes(values) => values, + Self::U16(values) => bytemuck::cast_slice(values), + Self::I16(values) => bytemuck::cast_slice(values), + Self::U32(values) => bytemuck::cast_slice(values), + Self::I32(values) => bytemuck::cast_slice(values), + Self::I64(values) => bytemuck::cast_slice(values), + Self::F32(values) => bytemuck::cast_slice(values), + Self::F64(values) => bytemuck::cast_slice(values), + Self::External(storage) => storage.allocation_bytes(), + } + } + + #[cfg(feature = "dlpack")] + pub(crate) fn is_empty(&self) -> bool { + self.as_bytes().is_empty() + } + + #[cfg(feature = "dlpack")] + pub(crate) fn as_ptr(&self) -> *const u8 { + self.as_bytes().as_ptr() + } + + #[cfg(all(feature = "dlpack", test))] + pub(crate) fn strong_count(&self) -> usize { + match self { + Self::Bytes(values) => Arc::strong_count(values), + Self::U16(values) => Arc::strong_count(values), + Self::I16(values) => Arc::strong_count(values), + Self::U32(values) => Arc::strong_count(values), + Self::I32(values) => Arc::strong_count(values), + Self::I64(values) => Arc::strong_count(values), + Self::F32(values) => Arc::strong_count(values), + Self::F64(values) => Arc::strong_count(values), + Self::External(storage) => Arc::strong_count(storage), + } + } +} + +fn copy_storage(bytes: &[u8], dtype: DataType) -> TensorStorage { + macro_rules! aligned_copy { + ($type:ty, $variant:ident) => { + bytemuck::try_cast_slice::(bytes) + .map(|values| TensorStorage::$variant(Arc::from(values))) + .unwrap_or_else(|_| TensorStorage::Bytes(Arc::from(bytes))) + }; + } + match dtype { + DataType::U16 | DataType::F16 | DataType::BF16 => aligned_copy!(u16, U16), + DataType::I16 => aligned_copy!(i16, I16), + DataType::U32 => aligned_copy!(u32, U32), + DataType::I32 => aligned_copy!(i32, I32), + DataType::I64 => aligned_copy!(i64, I64), + DataType::F32 => aligned_copy!(f32, F32), + DataType::F64 => aligned_copy!(f64, F64), + _ => TensorStorage::Bytes(Arc::from(bytes)), + } +} + +impl PartialEq for TensorBuffer { + fn eq(&self, other: &Self) -> bool { + self.descriptor == other.descriptor && self.allocation_bytes() == other.allocation_bytes() + } +} + +impl Eq for TensorBuffer {} + +impl TensorBuffer { + /// Validates and takes ownership of a host allocation. + pub fn try_new(bytes: Vec, descriptor: TensorDescriptor) -> Result { + descriptor.validate_storage(bytes.len())?; + Ok(Self { storage: TensorStorage::Bytes(Arc::from(bytes)), descriptor }) + } + + /// Validates and owns aligned u16 storage without byte repacking. + pub fn try_from_u16( + values: Vec, + descriptor: TensorDescriptor, + ) -> Result { + Self::try_from_storage(TensorStorage::U16(Arc::from(values)), DataType::U16, descriptor) + } + + /// Validates and owns aligned i16 storage without byte repacking. + pub fn try_from_i16( + values: Vec, + descriptor: TensorDescriptor, + ) -> Result { + Self::try_from_storage(TensorStorage::I16(Arc::from(values)), DataType::I16, descriptor) + } + + /// Validates and owns aligned u32 storage without byte repacking. + pub fn try_from_u32( + values: Vec, + descriptor: TensorDescriptor, + ) -> Result { + Self::try_from_storage(TensorStorage::U32(Arc::from(values)), DataType::U32, descriptor) + } + + /// Validates and owns aligned i32 storage without byte repacking. + pub fn try_from_i32( + values: Vec, + descriptor: TensorDescriptor, + ) -> Result { + Self::try_from_storage(TensorStorage::I32(Arc::from(values)), DataType::I32, descriptor) + } + + /// Validates and owns aligned i64 storage without byte repacking. + pub fn try_from_i64( + values: Vec, + descriptor: TensorDescriptor, + ) -> Result { + Self::try_from_storage(TensorStorage::I64(Arc::from(values)), DataType::I64, descriptor) + } + + /// Validates and owns aligned f32 storage without byte repacking. + pub fn try_from_f32( + values: Vec, + descriptor: TensorDescriptor, + ) -> Result { + Self::try_from_storage(TensorStorage::F32(Arc::from(values)), DataType::F32, descriptor) + } + + /// Validates and owns aligned f64 storage without byte repacking. + pub fn try_from_f64( + values: Vec, + descriptor: TensorDescriptor, + ) -> Result { + Self::try_from_storage(TensorStorage::F64(Arc::from(values)), DataType::F64, descriptor) + } + + /// Validates and retains a runtime-owned host allocation without copying it. + pub fn try_from_host_storage( + storage: Arc, + descriptor: TensorDescriptor, + ) -> Result { + let dtype = storage.dtype(); + Self::try_from_storage(TensorStorage::External(storage), dtype, descriptor) + } + + fn try_from_storage( + storage: TensorStorage, + expected: DataType, + descriptor: TensorDescriptor, + ) -> Result { + if descriptor.dtype() != expected { + return Err(TensorError::DataTypeMismatch { expected, actual: descriptor.dtype() }); + } + descriptor.validate_storage(storage.as_bytes().len())?; + Ok(Self { storage, descriptor }) + } + + /// Returns a zero-copy borrowed view. + pub fn view(&self) -> TensorView<'_> { + TensorView { bytes: self.storage.as_bytes(), descriptor: self.descriptor.clone() } + } + + /// Returns tensor metadata. + pub const fn descriptor(&self) -> &TensorDescriptor { + &self.descriptor + } + + /// Returns the complete owned allocation bytes. + pub fn allocation_bytes(&self) -> &[u8] { + self.storage.as_bytes() + } + + /// Performs an explicit host-to-host copy while preserving typed alignment. + pub fn to_owned_copy(&self) -> Self { + Self { + storage: copy_storage(self.storage.as_bytes(), self.descriptor.dtype()), + descriptor: self.descriptor.clone(), + } + } + + #[cfg(feature = "dlpack")] + pub(crate) fn shared_allocation(&self) -> TensorStorage { + self.storage.clone() + } + + /// Returns shared aligned f32 storage when constructed with [`Self::try_from_f32`]. + pub fn shared_f32(&self) -> Option> { + match &self.storage { + TensorStorage::F32(values) => Some(Arc::clone(values)), + _ => None, + } + } + + /// Returns shared byte storage when constructed with [`Self::try_new`]. + /// + /// This is suitable for zero-copy `u8` and `i8` tensor adapters. Multi-byte + /// element types must use their matching typed constructor and accessor. + pub fn shared_bytes(&self) -> Option> { + match &self.storage { + TensorStorage::Bytes(values) => Some(Arc::clone(values)), + _ => None, + } + } + + /// Returns shared aligned f64 storage when constructed with [`Self::try_from_f64`]. + pub fn shared_f64(&self) -> Option> { + match &self.storage { + TensorStorage::F64(values) => Some(Arc::clone(values)), + _ => None, + } + } + + /// Returns shared aligned i16 storage when constructed with [`Self::try_from_i16`]. + pub fn shared_i16(&self) -> Option> { + match &self.storage { + TensorStorage::I16(values) => Some(Arc::clone(values)), + _ => None, + } + } + + /// Returns shared aligned i32 storage when constructed with [`Self::try_from_i32`]. + pub fn shared_i32(&self) -> Option> { + match &self.storage { + TensorStorage::I32(values) => Some(Arc::clone(values)), + _ => None, + } + } + + /// Returns shared aligned i64 storage when constructed with [`Self::try_from_i64`]. + pub fn shared_i64(&self) -> Option> { + match &self.storage { + TensorStorage::I64(values) => Some(Arc::clone(values)), + _ => None, + } + } + + /// Returns shared aligned u16 storage when constructed with [`Self::try_from_u16`]. + pub fn shared_u16(&self) -> Option> { + match &self.storage { + TensorStorage::U16(values) => Some(Arc::clone(values)), + _ => None, + } + } + + /// Returns shared aligned u32 storage when constructed with [`Self::try_from_u32`]. + pub fn shared_u32(&self) -> Option> { + match &self.storage { + TensorStorage::U32(values) => Some(Arc::clone(values)), + _ => None, + } + } + + /// Returns runtime-owned host storage, when this tensor wraps one. + pub fn host_storage(&self) -> Option<&Arc> { + match &self.storage { + TensorStorage::External(storage) => Some(storage), + _ => None, + } + } + + /// Reports owned storage. + pub const fn ownership(&self) -> Ownership { + Ownership::Owned + } + + /// Explicitly copies the allocation into bytes and returns its metadata. + pub fn into_allocation_bytes_copy(self) -> (Vec, TensorDescriptor) { + (self.storage.as_bytes().to_vec(), self.descriptor) + } +} + +#[cfg(test)] +mod tests { + use super::{ + DataType, Device, DeviceKind, TensorBuffer, TensorDescriptor, TensorError, TensorView, + }; + + #[test] + fn compact_tensor_validates_exact_storage() { + let descriptor = TensorDescriptor::contiguous(DataType::F32, vec![2, 3], Device::CPU); + let tensor = TensorBuffer::try_new(vec![0; 24], descriptor).unwrap(); + assert_eq!(tensor.descriptor().element_count().unwrap(), 6); + assert!(tensor.descriptor().is_c_contiguous()); + assert_eq!(tensor.descriptor().required_byte_range().unwrap(), 0..24); + } + + #[test] + fn strided_roi_and_negative_stride_have_checked_spans() { + let roi = + TensorDescriptor::try_strided(DataType::U8, vec![2, 3], vec![5, 1], 6, Device::CPU) + .unwrap(); + assert_eq!(roi.required_byte_range().unwrap(), 6..14); + TensorView::try_new(&[0; 14], roi).unwrap(); + + let reversed = + TensorDescriptor::try_strided(DataType::U8, vec![4], vec![-1], 3, Device::CPU).unwrap(); + assert_eq!(reversed.required_byte_range().unwrap(), 0..4); + TensorView::try_new(&[0; 4], reversed).unwrap(); + } + + #[test] + fn rejects_short_storage_and_device_memory_as_host_slice() { + let descriptor = TensorDescriptor::contiguous(DataType::U16, vec![4], Device::CPU); + assert!(matches!( + TensorView::try_new(&[0; 7], descriptor), + Err(TensorError::StorageOutOfBounds { .. }) + )); + let cuda = TensorDescriptor::contiguous( + DataType::U8, + vec![1], + Device { kind: DeviceKind::Cuda, id: 0 }, + ); + assert!(matches!( + TensorView::try_new(&[0], cuda), + Err(TensorError::DeviceNotHostAccessible(_)) + )); + } + + #[test] + fn zero_sized_and_scalar_shapes_are_distinct() { + let empty = TensorDescriptor::contiguous(DataType::U8, vec![2, 0, 3], Device::CPU); + assert_eq!(empty.element_count().unwrap(), 0); + assert_eq!(empty.required_byte_range().unwrap(), 0..0); + let scalar = TensorDescriptor::contiguous(DataType::F64, vec![], Device::CPU); + assert_eq!(scalar.element_count().unwrap(), 1); + assert_eq!(scalar.required_byte_range().unwrap(), 0..8); + } +} diff --git a/crates/spatialrust-tensor/src/spatial.rs b/crates/spatialrust-tensor/src/spatial.rs new file mode 100644 index 0000000..cab0383 --- /dev/null +++ b/crates/spatialrust-tensor/src/spatial.rs @@ -0,0 +1,52 @@ +//! Zero-copy bridges from schema-aware point-cloud columns. + +use spatialrust_core::{SpatialError, SpatialTensor}; + +use crate::{DataType, Device, TensorDescriptor, TensorError, TensorView}; + +/// Errors raised while exposing a point-cloud column as a generic tensor. +#[derive(Debug, thiserror::Error)] +pub enum SpatialTensorBridgeError { + /// The requested field is absent or does not have the required dtype. + #[error(transparent)] + Spatial(#[from] SpatialError), + /// The generic tensor descriptor or storage is invalid. + #[error(transparent)] + Tensor(#[from] TensorError), +} + +/// Borrows one `f32` point field as a zero-copy `[point_count]` CPU tensor. +/// +/// Point fields remain separate Schema-SoA columns. Interleaving XYZ or other +/// fields requires a separately named packing operation and is never hidden by +/// this bridge. +pub fn spatial_f32_field_view<'a>( + tensor: &SpatialTensor<'a>, + field_name: &str, +) -> Result, SpatialTensorBridgeError> { + let values = tensor.cloud().field(field_name)?.as_f32()?; + let descriptor = TensorDescriptor::contiguous(DataType::F32, vec![values.len()], Device::CPU); + Ok(TensorView::try_new(bytemuck::cast_slice(values), descriptor)?) +} + +#[cfg(test)] +mod tests { + use super::spatial_f32_field_view; + use spatialrust_core::{PointCloudBuilder, SpatialTensor}; + + #[test] + fn point_field_is_borrowed_without_interleaving() { + let mut builder = PointCloudBuilder::xyz(); + builder.push_point([1.0, 2.0, 3.0]).unwrap(); + builder.push_point([4.0, 5.0, 6.0]).unwrap(); + let cloud = builder.build().unwrap(); + let spatial = SpatialTensor::new(&cloud, 1).unwrap(); + let tensor = spatial_f32_field_view(&spatial, "x").unwrap(); + assert_eq!(tensor.descriptor().shape(), &[2]); + assert_eq!(bytemuck::cast_slice::(tensor.allocation_bytes()), &[1.0, 4.0]); + assert_eq!( + tensor.allocation_bytes().as_ptr(), + cloud.field("x").unwrap().as_f32().unwrap().as_ptr().cast() + ); + } +} diff --git a/crates/spatialrust-tensor/tests/properties.rs b/crates/spatialrust-tensor/tests/properties.rs new file mode 100644 index 0000000..6ff7f3c --- /dev/null +++ b/crates/spatialrust-tensor/tests/properties.rs @@ -0,0 +1,26 @@ +use proptest::prelude::*; +use spatialrust_tensor::{DataType, Device, TensorDescriptor, TensorView}; + +proptest! { + #[test] + fn compact_range_matches_checked_product(shape in prop::collection::vec(0usize..32, 0..6)) { + let descriptor = TensorDescriptor::contiguous(DataType::F32, shape.clone(), Device::CPU); + let count = shape.iter().try_fold(1usize, |value, dimension| value.checked_mul(*dimension)).unwrap(); + let expected = count * 4; + prop_assert_eq!(descriptor.required_byte_range().unwrap(), 0..expected); + prop_assert!(TensorView::try_new(&vec![0; expected], descriptor).is_ok()); + } + + #[test] + fn reversed_vector_span_stays_inside_allocation(length in 1usize..2048) { + let descriptor = TensorDescriptor::try_strided( + DataType::U16, + vec![length], + vec![-1], + (length - 1) * 2, + Device::CPU, + ).unwrap(); + prop_assert_eq!(descriptor.required_byte_range().unwrap(), 0..length * 2); + prop_assert!(TensorView::try_new(&vec![0; length * 2], descriptor).is_ok()); + } +} diff --git a/crates/spatialrust-vision/Cargo.toml b/crates/spatialrust-vision/Cargo.toml index 0769aef..f1d01a3 100644 --- a/crates/spatialrust-vision/Cargo.toml +++ b/crates/spatialrust-vision/Cargo.toml @@ -13,10 +13,16 @@ default = [] resize = [] preprocess = ["resize"] warp = ["resize"] +imgproc-filter = [] +imgproc-morphology = [] +imgproc-analysis = [] +imgproc-canny = ["imgproc-filter"] +feature2d = ["imgproc-filter", "resize"] +geometry = ["dep:spatialrust-camera"] detection = [] dense = ["detection"] spatial = ["dense", "dep:spatialrust-core", "dep:spatialrust-camera"] -full = ["preprocess", "warp", "detection", "dense", "spatial"] +full = ["preprocess", "warp", "imgproc-filter", "imgproc-morphology", "imgproc-analysis", "imgproc-canny", "feature2d", "geometry", "detection", "dense", "spatial"] [dependencies] spatialrust-image.workspace = true @@ -33,3 +39,33 @@ proptest.workspace = true name = "preprocess" harness = false required-features = ["preprocess"] + +[[bench]] +name = "filter" +harness = false +required-features = ["imgproc-filter"] + +[[bench]] +name = "morphology" +harness = false +required-features = ["imgproc-morphology"] + +[[bench]] +name = "analysis" +harness = false +required-features = ["imgproc-analysis"] + +[[bench]] +name = "canny" +harness = false +required-features = ["imgproc-canny"] + +[[bench]] +name = "feature2d" +harness = false +required-features = ["feature2d"] + +[[bench]] +name = "geometry" +harness = false +required-features = ["geometry"] diff --git a/crates/spatialrust-vision/benches/analysis.rs b/crates/spatialrust-vision/benches/analysis.rs new file mode 100644 index 0000000..a345101 --- /dev/null +++ b/crates/spatialrust-vision/benches/analysis.rs @@ -0,0 +1,47 @@ +use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; +use spatialrust_image::Image; +use spatialrust_vision::{clahe, equalize_histogram, integral_image, threshold, ThresholdType}; + +fn benchmark_analysis(c: &mut Criterion) { + for &(name, width, height) in &[("640p", 640, 480), ("1080p", 1920, 1080), ("4k", 3840, 2160)] { + let data = (0..width * height).map(|index| (index & 255) as u8).collect(); + let image = Image::::try_new(width, height, data).unwrap(); + for (operation, mut run) in [ + ( + "threshold", + Box::new(|| { + black_box( + threshold(image.view(), 127.0, 255.0, ThresholdType::Binary).unwrap(), + ); + }) as Box, + ), + ( + "equalize_histogram", + Box::new(|| { + black_box(equalize_histogram(image.view()).unwrap()); + }), + ), + ( + "clahe_8x8", + Box::new(|| { + black_box(clahe(image.view(), 2.0, 8, 8).unwrap()); + }), + ), + ( + "integral", + Box::new(|| { + black_box(integral_image(image.view(), 0).unwrap()); + }), + ), + ] { + let mut group = c.benchmark_group(operation); + group.sample_size(10); + group.throughput(Throughput::Elements((width * height) as u64)); + group.bench_function(BenchmarkId::from_parameter(name), |b| b.iter(&mut run)); + group.finish(); + } + } +} + +criterion_group!(benches, benchmark_analysis); +criterion_main!(benches); diff --git a/crates/spatialrust-vision/benches/canny.rs b/crates/spatialrust-vision/benches/canny.rs new file mode 100644 index 0000000..ec32c6d --- /dev/null +++ b/crates/spatialrust-vision/benches/canny.rs @@ -0,0 +1,24 @@ +use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; +use spatialrust_image::Image; +use spatialrust_vision::{canny, CannyOptions}; + +fn benchmark_canny(c: &mut Criterion) { + let mut group = c.benchmark_group("canny"); + group.sample_size(10); + for &(name, width, height) in &[("640p", 640, 480), ("1080p", 1920, 1080), ("4k", 3840, 2160)] { + let data = (0..width * height) + .map(|index| ((index * 37 + index / width * 17) & 255) as u8) + .collect(); + let image = Image::::try_new(width, height, data).unwrap(); + group.throughput(Throughput::Elements((width * height) as u64)); + group.bench_function(BenchmarkId::from_parameter(name), |b| { + b.iter(|| { + black_box(canny(image.view(), CannyOptions::default()).unwrap()); + }); + }); + } + group.finish(); +} + +criterion_group!(benches, benchmark_canny); +criterion_main!(benches); diff --git a/crates/spatialrust-vision/benches/feature2d.rs b/crates/spatialrust-vision/benches/feature2d.rs new file mode 100644 index 0000000..b3b301c --- /dev/null +++ b/crates/spatialrust-vision/benches/feature2d.rs @@ -0,0 +1,39 @@ +use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; +use spatialrust_image::Image; +use spatialrust_vision::{detect_and_describe_orb, detect_fast, FastOptions, OrbOptions}; + +fn benchmark_feature2d(c: &mut Criterion) { + for &(name, width, height) in &[("640p", 640, 480), ("1080p", 1920, 1080), ("4k", 3840, 2160)] { + let data = (0..width * height) + .map(|index| { + let x = index % width; + let y = index / width; + ((x * 37 + y * 19) ^ (x * y * 3) ^ ((x / 8 + y / 8) * 127)) as u8 + }) + .collect(); + let image = Image::::try_new(width, height, data).unwrap(); + + let mut fast = c.benchmark_group("fast_9_16"); + fast.sample_size(10); + fast.throughput(Throughput::Elements((width * height) as u64)); + fast.bench_function(BenchmarkId::from_parameter(name), |b| { + b.iter(|| { + black_box(detect_fast(image.view(), FastOptions::default()).unwrap()); + }); + }); + fast.finish(); + + let mut orb = c.benchmark_group("orb_500"); + orb.sample_size(10); + orb.throughput(Throughput::Elements((width * height) as u64)); + orb.bench_function(BenchmarkId::from_parameter(name), |b| { + b.iter(|| { + black_box(detect_and_describe_orb(image.view(), OrbOptions::default()).unwrap()); + }); + }); + orb.finish(); + } +} + +criterion_group!(benches, benchmark_feature2d); +criterion_main!(benches); diff --git a/crates/spatialrust-vision/benches/filter.rs b/crates/spatialrust-vision/benches/filter.rs new file mode 100644 index 0000000..fb1c2fa --- /dev/null +++ b/crates/spatialrust-vision/benches/filter.rs @@ -0,0 +1,68 @@ +use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; +use spatialrust_image::Image; +use spatialrust_vision::{ + bilateral_filter, gaussian_blur, median_blur, pyr_down, sobel, BorderMode, +}; + +fn benchmark_gaussian(c: &mut Criterion) { + let mut group = c.benchmark_group("gaussian_blur_rgb8_5x5"); + group.sample_size(10); + for &(name, width, height) in &[("640p", 640, 480), ("1080p", 1920, 1080), ("4k", 3840, 2160)] { + let image = Image::::try_new(width, height, vec![127; width * height * 3]).unwrap(); + group.throughput(Throughput::Elements((width * height) as u64)); + group.bench_with_input(BenchmarkId::from_parameter(name), &image, |b, image| { + b.iter(|| { + gaussian_blur(black_box(image.view()), 5, 5, 1.2, 1.2, BorderMode::Reflect101) + .unwrap() + }); + }); + } + group.finish(); +} + +fn benchmark_advanced_filters(c: &mut Criterion) { + for &(name, width, height) in &[("640p", 640, 480), ("1080p", 1920, 1080), ("4k", 3840, 2160)] { + let rgb = Image::::try_new(width, height, vec![127; width * height * 3]).unwrap(); + let gray = Image::::try_new(width, height, vec![127; width * height]).unwrap(); + let throughput = Throughput::Elements((width * height) as u64); + for (operation, mut run) in [ + ( + "median_3x3", + Box::new(|| { + black_box(median_blur(rgb.view(), 3, BorderMode::Replicate).unwrap()); + }) as Box, + ), + ( + "bilateral_d5", + Box::new(|| { + black_box( + bilateral_filter(rgb.view(), 5, 40.0, 3.0, BorderMode::Reflect101).unwrap(), + ); + }), + ), + ( + "sobel_3x3", + Box::new(|| { + black_box( + sobel(gray.view(), 1, 0, 3, 1.0, 0.0, BorderMode::Reflect101).unwrap(), + ); + }), + ), + ( + "pyr_down", + Box::new(|| { + black_box(pyr_down(rgb.view(), BorderMode::Reflect101).unwrap()); + }), + ), + ] { + let mut group = c.benchmark_group(operation); + group.sample_size(10); + group.throughput(throughput.clone()); + group.bench_function(name, |b| b.iter(&mut run)); + group.finish(); + } + } +} + +criterion_group!(benches, benchmark_gaussian, benchmark_advanced_filters); +criterion_main!(benches); diff --git a/crates/spatialrust-vision/benches/geometry.rs b/crates/spatialrust-vision/benches/geometry.rs new file mode 100644 index 0000000..6c9128f --- /dev/null +++ b/crates/spatialrust-vision/benches/geometry.rs @@ -0,0 +1,130 @@ +use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; +use spatialrust_camera::CameraIntrinsics; +use spatialrust_image::Image; +use spatialrust_math::{Mat3, Vec2, Vec3}; +use spatialrust_vision::{ + estimate_homography_ransac, project_object_point, solve_pnp, stereo_block_match, + track_points_lucas_kanade, AbsolutePose, CameraMatrix3, ObjectImageCorrespondence, + PointCorrespondence2, RobustEstimationOptions, StereoBmOptions, +}; + +fn benchmark_geometry(c: &mut Criterion) { + let camera = CameraMatrix3::from_intrinsics( + CameraIntrinsics::try_new(500.0, 500.0, 320.0, 240.0, 640, 480).unwrap(), + ); + let pose = AbsolutePose::try_new( + Mat3::from_rows([0.98, -0.1, 0.17], [0.12, 0.98, -0.1], [-0.16, 0.12, 0.98]), + Vec3::new(0.1, -0.05, 2.0), + ) + .unwrap(); + + let mut pnp = c.benchmark_group("solve_pnp_correspondences"); + for &count in &[64usize, 256, 1024] { + let pairs = (0..count) + .map(|index| { + let object = Vec3::new( + (index % 16) as f64 * 0.05 - 0.4, + (index / 16) as f64 * 0.05 - 0.4, + (index % 5) as f64 * 0.02, + ); + let image = project_object_point(pose, camera, object).unwrap(); + ObjectImageCorrespondence::try_new(object, image).unwrap() + }) + .collect::>(); + pnp.throughput(Throughput::Elements(count as u64)); + pnp.bench_with_input(BenchmarkId::from_parameter(count), &pairs, |b, pairs| { + b.iter(|| black_box(solve_pnp(pairs, camera).unwrap())); + }); + } + pnp.finish(); + + let mut homography = c.benchmark_group("homography_ransac_correspondences"); + for &count in &[64usize, 256, 1024] { + let pairs = (0..count) + .map(|index| { + let source = Vec2 { + x: (index % 32) as f64 * 10.0, + y: (index / 32) as f64 * 8.0, + }; + PointCorrespondence2::try_new( + source, + Vec2 { x: source.x * 1.05 + 2.0, y: source.y * 0.98 - 1.0 }, + ) + .unwrap() + }) + .collect::>(); + homography.throughput(Throughput::Elements(count as u64)); + homography.bench_with_input(BenchmarkId::from_parameter(count), &pairs, |b, pairs| { + b.iter(|| { + black_box( + estimate_homography_ransac(pairs, RobustEstimationOptions::default()).unwrap(), + ) + }); + }); + } + homography.finish(); + + for &(name, width, height) in &[("640p", 640, 480), ("1080p", 1920, 1080), ("4k", 3840, 2160)] { + let left_data = (0..width * height) + .map(|index| { + let x = index % width; + let y = index / width; + ((x * 17 + y * 29) % 200 + 20) as u8 + }) + .collect::>(); + let mut right_data = vec![0u8; width * height]; + let disparity = 20usize; + for y in 0..height { + for x in disparity..width { + right_data[y * width + (x - disparity)] = left_data[y * width + x]; + } + } + let left = Image::::try_new(width, height, left_data.clone()).unwrap(); + let right = Image::::try_new(width, height, right_data).unwrap(); + + let mut bm = c.benchmark_group("stereo_bm"); + bm.sample_size(10); + bm.throughput(Throughput::Elements((width * height) as u64)); + bm.bench_function(BenchmarkId::from_parameter(name), |b| { + b.iter(|| { + black_box( + stereo_block_match( + left.view(), + right.view(), + StereoBmOptions { + window_size: 15, + min_disparity: 1, + num_disparities: 64, + uniqueness_ratio: 10.0, + }, + ) + .unwrap(), + ) + }); + }); + bm.finish(); + + let points = (0..200) + .map(|index| Vec2 { + x: 40.0 + (index % 20) as f64 * 20.0, + y: 40.0 + (index / 20) as f64 * 20.0, + }) + .collect::>(); + let next = Image::::try_new(width, height, left_data).unwrap(); + let mut lk = c.benchmark_group("lucas_kanade_200"); + lk.sample_size(10); + lk.throughput(Throughput::Elements(points.len() as u64)); + lk.bench_function(BenchmarkId::from_parameter(name), |b| { + b.iter(|| { + black_box( + track_points_lucas_kanade(left.view(), next.view(), &points, Default::default()) + .unwrap(), + ) + }); + }); + lk.finish(); + } +} + +criterion_group!(benches, benchmark_geometry); +criterion_main!(benches); diff --git a/crates/spatialrust-vision/benches/morphology.rs b/crates/spatialrust-vision/benches/morphology.rs new file mode 100644 index 0000000..51e1705 --- /dev/null +++ b/crates/spatialrust-vision/benches/morphology.rs @@ -0,0 +1,31 @@ +use criterion::{black_box, criterion_group, criterion_main, BenchmarkId, Criterion, Throughput}; +use spatialrust_image::Image; +use spatialrust_vision::{ + morphology_ex, BorderMode, MorphologyOperation, MorphologyShape, StructuringElement, +}; + +fn benchmark_morphology(c: &mut Criterion) { + let element = StructuringElement::try_new(MorphologyShape::Ellipse, 5, 5).unwrap(); + let mut group = c.benchmark_group("morphology_open_gray8_5x5"); + group.sample_size(10); + for &(name, width, height) in &[("640p", 640, 480), ("1080p", 1920, 1080), ("4k", 3840, 2160)] { + let image = Image::::try_new(width, height, vec![127; width * height]).unwrap(); + group.throughput(Throughput::Elements((width * height) as u64)); + group.bench_with_input(BenchmarkId::from_parameter(name), &image, |b, image| { + b.iter(|| { + morphology_ex( + black_box(image.view()), + MorphologyOperation::Open, + &element, + 1, + BorderMode::Replicate, + ) + .unwrap() + }); + }); + } + group.finish(); +} + +criterion_group!(benches, benchmark_morphology); +criterion_main!(benches); diff --git a/crates/spatialrust-vision/src/advanced_filter.rs b/crates/spatialrust-vision/src/advanced_filter.rs new file mode 100644 index 0000000..9c3e913 --- /dev/null +++ b/crates/spatialrust-vision/src/advanced_filter.rs @@ -0,0 +1,370 @@ +//! Non-linear filters, image derivatives, and Gaussian pyramids. + +use spatialrust_image::{Image, ImageView}; + +use crate::border::fetch; +use crate::{ + filter2d_f32, separable_filter, separable_filter_f32, BorderMode, Kernel1D, Kernel2D, + PixelComponent, VisionError, VisionResult, +}; + +/// Applies a per-channel median filter with an odd square aperture. +pub fn median_blur( + input: ImageView<'_, T, CHANNELS>, + kernel_size: usize, + border: BorderMode, +) -> VisionResult> { + validate_odd_size(kernel_size, "median")?; + let radius = (kernel_size / 2) as isize; + let area = kernel_size + .checked_mul(kernel_size) + .ok_or_else(|| VisionError::InvalidParameter("median kernel area overflows".into()))?; + let mut samples = (0..CHANNELS).map(|_| Vec::::with_capacity(area)).collect::>(); + let mut output = Vec::with_capacity(input.width() * input.height() * CHANNELS); + for y in 0..input.height() { + for x in 0..input.width() { + for channel in &mut samples { + channel.clear(); + } + for dy in -radius..=radius { + for dx in -radius..=radius { + let pixel = fetch(input, x as isize + dx, y as isize + dy, border); + for channel in 0..CHANNELS { + samples[channel].push(pixel[channel].to_f64()); + } + } + } + for channel in &mut samples { + channel.sort_by(f64::total_cmp); + output.push(T::from_f64(channel[area / 2])); + } + } + } + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +/// Applies an edge-preserving bilateral filter. +/// +/// Color distance is the squared sum of per-channel absolute differences, +/// matching OpenCV's CPU implementation. The operation is out-of-place and +/// never performs a device transfer. +pub fn bilateral_filter( + input: ImageView<'_, T, CHANNELS>, + diameter: usize, + sigma_color: f64, + sigma_space: f64, + border: BorderMode, +) -> VisionResult> { + if diameter == 0 { + return Err(VisionError::InvalidParameter("bilateral diameter must be non-zero".into())); + } + validate_positive_finite(sigma_color, "sigma_color")?; + validate_positive_finite(sigma_space, "sigma_space")?; + let radius = (diameter / 2) as isize; + let color_factor = -0.5 / (sigma_color * sigma_color); + let space_factor = -0.5 / (sigma_space * sigma_space); + let mut output = Vec::with_capacity(input.width() * input.height() * CHANNELS); + for y in 0..input.height() { + for x in 0..input.width() { + let center = fetch(input, x as isize, y as isize, border); + let mut sums = [0.0; CHANNELS]; + let mut total_weight = 0.0; + for dy in -radius..=radius { + for dx in -radius..=radius { + let dx_f64 = dx as f64; + let dy_f64 = dy as f64; + let spatial_distance = dx_f64.mul_add(dx_f64, dy_f64 * dy_f64); + if spatial_distance > (radius as f64) * (radius as f64) { + continue; + } + let pixel = fetch(input, x as isize + dx, y as isize + dy, border); + let color_distance = (0..CHANNELS) + .map(|channel| pixel[channel].to_f64() - center[channel].to_f64()) + .map(f64::abs) + .sum::(); + let weight = ((color_distance * color_distance) + .mul_add(color_factor, spatial_distance * space_factor)) + .exp(); + for channel in 0..CHANNELS { + sums[channel] += pixel[channel].to_f64() * weight; + } + total_weight += weight; + } + } + output.extend(sums.map(|sum| T::from_f64(sum / total_weight))); + } + } + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +/// Computes a Sobel derivative as signed `f32` values. +pub fn sobel( + input: ImageView<'_, T, CHANNELS>, + dx: usize, + dy: usize, + kernel_size: usize, + scale: f64, + delta: f64, + border: BorderMode, +) -> VisionResult> { + validate_derivative(dx, dy, kernel_size, scale, delta)?; + let mut kernel_x = derivative_kernel(kernel_size, dx)?; + let kernel_y = derivative_kernel(kernel_size, dy)?; + kernel_x = + Kernel1D::try_new(kernel_x.coefficients().iter().map(|value| value * scale).collect())?; + separable_filter_f32(input, &kernel_x, &kernel_y, delta, border) +} + +/// Computes a 3×3 Scharr first derivative as signed `f32` values. +pub fn scharr( + input: ImageView<'_, T, CHANNELS>, + dx: usize, + dy: usize, + scale: f64, + delta: f64, + border: BorderMode, +) -> VisionResult> { + if dx + dy != 1 || dx > 1 || dy > 1 { + return Err(VisionError::InvalidParameter( + "Scharr requires derivative order (1, 0) or (0, 1)".into(), + )); + } + if !scale.is_finite() || !delta.is_finite() { + return Err(VisionError::InvalidParameter("scale and delta must be finite".into())); + } + let derivative = Kernel1D::try_new(vec![-scale, 0.0, scale])?; + let smoothing = Kernel1D::try_new(vec![3.0, 10.0, 3.0])?; + if dx == 1 { + separable_filter_f32(input, &derivative, &smoothing, delta, border) + } else { + separable_filter_f32(input, &smoothing, &derivative, delta, border) + } +} + +/// Computes a Laplacian response as signed `f32` values. +pub fn laplacian( + input: ImageView<'_, T, CHANNELS>, + kernel_size: usize, + scale: f64, + delta: f64, + border: BorderMode, +) -> VisionResult> { + if kernel_size == 1 { + if !scale.is_finite() || !delta.is_finite() { + return Err(VisionError::InvalidParameter("scale and delta must be finite".into())); + } + let kernel = Kernel2D::try_new( + 3, + 3, + vec![0.0, scale, 0.0, scale, -4.0 * scale, scale, 0.0, scale, 0.0], + )?; + return filter2d_f32(input, &kernel, delta, border); + } + validate_derivative(2, 0, kernel_size, scale, delta)?; + let dxx = sobel(input, 2, 0, kernel_size, scale, 0.0, border)?; + let dyy = sobel(input, 0, 2, kernel_size, scale, 0.0, border)?; + let output = dxx + .as_slice() + .iter() + .zip(dyy.as_slice()) + .map(|(&x, &y)| (f64::from(x) + f64::from(y) + delta) as f32) + .collect(); + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +/// Reduces each dimension by approximately two using the canonical 5×5 kernel. +pub fn pyr_down( + input: ImageView<'_, T, CHANNELS>, + border: BorderMode, +) -> VisionResult> { + let kernel = + Kernel1D::try_new(vec![1.0 / 16.0, 4.0 / 16.0, 6.0 / 16.0, 4.0 / 16.0, 1.0 / 16.0])?; + let blurred = separable_filter(input, &kernel, &kernel, 0.0, border)?; + let width = input.width().div_ceil(2); + let height = input.height().div_ceil(2); + let mut output = Vec::with_capacity(width * height * CHANNELS); + for y in 0..height { + for x in 0..width { + output.extend_from_slice(blurred.get(x * 2, y * 2).expect("pyramid sample in bounds")); + } + } + Ok(Image::try_new_with_metadata(width, height, output, input.metadata())?) +} + +/// Doubles each dimension using zero insertion and the canonical 5×5 kernel. +pub fn pyr_up( + input: ImageView<'_, T, CHANNELS>, + border: BorderMode, +) -> VisionResult> { + let width = input + .width() + .checked_mul(2) + .ok_or_else(|| VisionError::InvalidDimensions("pyramid width overflow".into()))?; + let height = input + .height() + .checked_mul(2) + .ok_or_else(|| VisionError::InvalidDimensions("pyramid height overflow".into()))?; + let mut expanded = + Image::::from_pixel(width, height, std::array::from_fn(|_| T::from_f64(0.0)))?; + expanded.set_metadata(input.metadata())?; + for y in 0..input.height() { + for x in 0..input.width() { + *expanded.get_mut(x * 2, y * 2).expect("expanded coordinate in bounds") = + *input.get(x, y).expect("source coordinate in bounds"); + } + } + let kernel = Kernel1D::try_new(vec![1.0 / 8.0, 4.0 / 8.0, 6.0 / 8.0, 4.0 / 8.0, 1.0 / 8.0])?; + separable_filter(expanded.view(), &kernel, &kernel, 0.0, border) +} + +/// Builds a packed Gaussian pyramid including the input as level zero. +pub fn build_gaussian_pyramid( + input: ImageView<'_, T, CHANNELS>, + levels: usize, + border: BorderMode, +) -> VisionResult>> { + if levels == 0 { + return Err(VisionError::InvalidParameter("pyramid levels must be non-zero".into())); + } + let mut packed = Vec::with_capacity(input.width() * input.height() * CHANNELS); + for y in 0..input.height() { + for x in 0..input.width() { + packed.extend_from_slice(input.get(x, y).expect("input coordinate in bounds")); + } + } + let first = + Image::try_new_with_metadata(input.width(), input.height(), packed, input.metadata())?; + let mut pyramid = Vec::with_capacity(levels); + pyramid.push(first); + while pyramid.len() < levels { + let next = pyr_down(pyramid.last().expect("level zero exists").view(), border)?; + pyramid.push(next); + } + Ok(pyramid) +} + +fn validate_odd_size(size: usize, name: &str) -> VisionResult<()> { + if size == 0 || size % 2 == 0 { + return Err(VisionError::InvalidParameter(format!( + "{name} kernel size must be positive and odd" + ))); + } + Ok(()) +} + +fn validate_positive_finite(value: f64, name: &str) -> VisionResult<()> { + if !value.is_finite() || value <= 0.0 { + return Err(VisionError::InvalidParameter(format!("{name} must be finite and positive"))); + } + Ok(()) +} + +fn validate_derivative( + dx: usize, + dy: usize, + kernel_size: usize, + scale: f64, + delta: f64, +) -> VisionResult<()> { + if !matches!(kernel_size, 3 | 5 | 7) { + return Err(VisionError::InvalidParameter("Sobel kernel size must be 3, 5, or 7".into())); + } + if dx + dy == 0 || dx > 2 || dy > 2 || dx >= kernel_size || dy >= kernel_size { + return Err(VisionError::InvalidParameter( + "Sobel derivative orders must total at least one and each be at most two".into(), + )); + } + if !scale.is_finite() || !delta.is_finite() { + return Err(VisionError::InvalidParameter("scale and delta must be finite".into())); + } + Ok(()) +} + +fn derivative_kernel(size: usize, order: usize) -> VisionResult { + let mut polynomial = vec![1.0]; + for _ in 0..order { + polynomial = convolve_coefficients(&polynomial, &[-1.0, 1.0]); + } + for _ in 0..(size - 1 - order) { + polynomial = convolve_coefficients(&polynomial, &[1.0, 1.0]); + } + Kernel1D::try_new(polynomial) +} + +fn convolve_coefficients(left: &[f64], right: &[f64]) -> Vec { + let mut output = vec![0.0; left.len() + right.len() - 1]; + for (i, &a) in left.iter().enumerate() { + for (j, &b) in right.iter().enumerate() { + output[i + j] += a * b; + } + } + output +} + +#[cfg(test)] +mod tests { + use super::{ + bilateral_filter, build_gaussian_pyramid, laplacian, median_blur, pyr_down, scharr, sobel, + }; + use crate::BorderMode; + use spatialrust_image::{Image, ImageRegion}; + + #[test] + fn median_removes_impulse_and_accepts_strided_roi() { + let parent = + Image::::try_new(5, 3, vec![0, 0, 0, 0, 0, 0, 10, 255, 10, 0, 0, 10, 10, 10, 0]) + .unwrap(); + let roi = parent.view().subview(ImageRegion::new(1, 0, 3, 3)).unwrap(); + let output = median_blur(roi, 3, BorderMode::Replicate).unwrap(); + assert_eq!(output[(1, 1)][0], 10); + } + + #[test] + fn bilateral_preserves_sharp_constant_regions() { + let image = Image::::try_new(5, 1, vec![0, 0, 0, 255, 255]).unwrap(); + let output = bilateral_filter(image.view(), 3, 5.0, 1.0, BorderMode::Replicate).unwrap(); + assert_eq!(output.as_slice(), image.as_slice()); + } + + #[test] + fn sobel_and_scharr_detect_horizontal_ramp() { + let image = + Image::::try_new(5, 3, (0..3).flat_map(|_| [0, 10, 20, 30, 40]).collect()) + .unwrap(); + let sx = sobel(image.view(), 1, 0, 3, 1.0, 0.0, BorderMode::Reflect101).unwrap(); + let sy = sobel(image.view(), 0, 1, 3, 1.0, 0.0, BorderMode::Reflect101).unwrap(); + let scharr_x = scharr(image.view(), 1, 0, 1.0, 0.0, BorderMode::Reflect101).unwrap(); + assert_eq!(sx[(2, 1)][0], 80.0); + assert_eq!(sy[(2, 1)][0], 0.0); + assert_eq!(scharr_x[(2, 1)][0], 320.0); + } + + #[test] + fn laplacian_of_constant_is_zero() { + let image = Image::::from_pixel(7, 5, [42]).unwrap(); + for size in [1, 3, 5, 7] { + let output = laplacian(image.view(), size, 1.0, 0.0, BorderMode::Reflect101).unwrap(); + assert!(output.as_slice().iter().all(|value| value.abs() < f32::EPSILON)); + } + } + + #[test] + fn pyramid_dimensions_follow_ceil_halving() { + let image = Image::::from_pixel(7, 5, [1.0, 2.0, 3.0]).unwrap(); + let down = pyr_down(image.view(), BorderMode::Reflect101).unwrap(); + assert_eq!((down.width(), down.height()), (4, 3)); + let pyramid = build_gaussian_pyramid(image.view(), 4, BorderMode::Reflect101).unwrap(); + assert_eq!( + pyramid.iter().map(|level| (level.width(), level.height())).collect::>(), + vec![(7, 5), (4, 3), (2, 2), (1, 1)] + ); + } + + #[test] + fn invalid_parameters_are_rejected() { + let image = Image::::try_new(1, 1, vec![0]).unwrap(); + assert!(median_blur(image.view(), 2, BorderMode::Replicate).is_err()); + assert!(bilateral_filter(image.view(), 0, 1.0, 1.0, BorderMode::Replicate).is_err()); + assert!(sobel(image.view(), 0, 0, 3, 1.0, 0.0, BorderMode::Replicate).is_err()); + } +} diff --git a/crates/spatialrust-vision/src/analysis.rs b/crates/spatialrust-vision/src/analysis.rs new file mode 100644 index 0000000..dab4965 --- /dev/null +++ b/crates/spatialrust-vision/src/analysis.rs @@ -0,0 +1,639 @@ +//! Thresholding, histograms, contrast enhancement, and integral images. + +use spatialrust_image::{Image, ImageView}; + +use crate::border::fetch; +use crate::{BorderMode, PixelComponent, VisionError, VisionResult}; + +/// Point-wise threshold transformation. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum ThresholdType { + /// `max_value` above the threshold, zero otherwise. + Binary, + /// Zero above the threshold, `max_value` otherwise. + BinaryInv, + /// Clamp values above the threshold to the threshold. + Truncate, + /// Preserve values above the threshold, zero otherwise. + ToZero, + /// Zero values above the threshold, preserve the others. + ToZeroInv, +} + +/// Adaptive neighborhood statistic. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum AdaptiveThresholdMethod { + /// Uniform neighborhood mean. + Mean, + /// Gaussian-weighted neighborhood mean. + Gaussian, +} + +/// Applies a fixed threshold independently to every component. +pub fn threshold( + input: ImageView<'_, T, CHANNELS>, + threshold: f64, + max_value: f64, + threshold_type: ThresholdType, +) -> VisionResult> { + if !threshold.is_finite() || !max_value.is_finite() { + return Err(VisionError::InvalidParameter("threshold and max_value must be finite".into())); + } + let mut output = Vec::with_capacity(input.width() * input.height() * CHANNELS); + for y in 0..input.height() { + for x in 0..input.width() { + let pixel = input.get(x, y).expect("input coordinate in bounds"); + output.extend(std::array::from_fn::<_, CHANNELS, _>(|channel| { + T::from_f64(apply_threshold( + pixel[channel].to_f64(), + threshold, + max_value, + threshold_type, + )) + })); + } + } + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +/// Selects an Otsu threshold for an 8-bit grayscale image and applies it. +pub fn otsu_threshold_u8( + input: ImageView<'_, u8, 1>, + max_value: u8, + threshold_type: ThresholdType, +) -> VisionResult<(u8, Image)> { + let histogram = histogram_u8(input); + let selected = otsu_from_histogram(&histogram, input.width() * input.height()) as u8; + let output = threshold(input, f64::from(selected), f64::from(max_value), threshold_type)?; + Ok((selected, output)) +} + +/// Selects an Otsu threshold for a 16-bit grayscale image and applies it. +pub fn otsu_threshold_u16( + input: ImageView<'_, u16, 1>, + max_value: u16, + threshold_type: ThresholdType, +) -> VisionResult<(u16, Image)> { + let mut histogram = vec![0_u64; 65_536]; + for y in 0..input.height() { + for x in 0..input.width() { + histogram[input.get(x, y).expect("coordinate in bounds")[0] as usize] += 1; + } + } + let selected = otsu_from_histogram(&histogram, input.width() * input.height()) as u16; + let output = threshold(input, f64::from(selected), f64::from(max_value), threshold_type)?; + Ok((selected, output)) +} + +/// Applies adaptive binary thresholding to an 8-bit grayscale image. +pub fn adaptive_threshold( + input: ImageView<'_, u8, 1>, + max_value: u8, + method: AdaptiveThresholdMethod, + threshold_type: ThresholdType, + block_size: usize, + c: f64, + border: BorderMode, +) -> VisionResult> { + if !matches!(threshold_type, ThresholdType::Binary | ThresholdType::BinaryInv) { + return Err(VisionError::InvalidParameter( + "adaptive threshold supports only binary and binary-inverse output".into(), + )); + } + if block_size <= 1 || block_size % 2 == 0 { + return Err(VisionError::InvalidParameter( + "adaptive block size must be odd and greater than one".into(), + )); + } + if !c.is_finite() { + return Err(VisionError::InvalidParameter("adaptive constant must be finite".into())); + } + let weights = match method { + AdaptiveThresholdMethod::Mean => vec![1.0 / (block_size * block_size) as f64; block_size], + AdaptiveThresholdMethod::Gaussian => gaussian_weights(block_size), + }; + let radius = (block_size / 2) as isize; + let mut output = Vec::with_capacity(input.width() * input.height()); + for y in 0..input.height() { + for x in 0..input.width() { + let mut local = 0.0; + for ky in 0..block_size { + for kx in 0..block_size { + let pixel = fetch( + input, + x as isize + kx as isize - radius, + y as isize + ky as isize - radius, + border, + )[0]; + let weight = match method { + AdaptiveThresholdMethod::Mean => weights[kx], + AdaptiveThresholdMethod::Gaussian => weights[kx] * weights[ky], + }; + local += f64::from(pixel) * weight; + } + } + let source = i64::from(input.get(x, y).expect("coordinate in bounds")[0]); + let local = cv_round(local).clamp(0, 255); + let delta = match threshold_type { + ThresholdType::Binary => c.ceil() as i64, + ThresholdType::BinaryInv => c.floor() as i64, + _ => unreachable!("adaptive threshold type checked above"), + }; + let selected = match threshold_type { + ThresholdType::Binary => source - local > -delta, + ThresholdType::BinaryInv => source - local <= -delta, + _ => unreachable!("adaptive threshold type checked above"), + }; + output.push(if selected { max_value } else { 0 }); + } + } + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +/// Computes a configurable one-channel histogram, optionally through a mask. +pub fn histogram( + input: ImageView<'_, T, CHANNELS>, + channel: usize, + bins: usize, + range: (f64, f64), + mask: Option>, +) -> VisionResult> { + if channel >= CHANNELS { + return Err(VisionError::InvalidParameter(format!( + "histogram channel {channel} is outside {CHANNELS} channels" + ))); + } + if bins == 0 { + return Err(VisionError::InvalidParameter("histogram bins must be non-zero".into())); + } + if !range.0.is_finite() || !range.1.is_finite() || range.1 <= range.0 { + return Err(VisionError::InvalidParameter( + "histogram range must be finite and increasing".into(), + )); + } + if let Some(mask) = mask { + if mask.width() != input.width() || mask.height() != input.height() { + return Err(VisionError::ShapeMismatch( + "histogram mask dimensions must match the image".into(), + )); + } + } + let mut result = vec![0_u64; bins]; + let scale = bins as f64 / (range.1 - range.0); + for y in 0..input.height() { + for x in 0..input.width() { + if mask.is_some_and(|mask| mask.get(x, y).expect("mask coordinate in bounds")[0] == 0) { + continue; + } + let value = input.get(x, y).expect("coordinate in bounds")[channel].to_f64(); + if value >= range.0 && value < range.1 { + let bin = ((value - range.0) * scale).floor() as usize; + result[bin.min(bins - 1)] += 1; + } + } + } + Ok(result) +} + +/// Computes the exact 256-bin histogram of an 8-bit grayscale image. +#[must_use] +pub fn histogram_u8(input: ImageView<'_, u8, 1>) -> Vec { + let mut result = vec![0_u64; 256]; + for y in 0..input.height() { + for x in 0..input.width() { + result[input.get(x, y).expect("coordinate in bounds")[0] as usize] += 1; + } + } + result +} + +/// Equalizes an 8-bit grayscale histogram. +pub fn equalize_histogram(input: ImageView<'_, u8, 1>) -> VisionResult> { + let histogram = histogram_u8(input); + let total = input.width() * input.height(); + if total == 0 { + return Ok(Image::try_new_with_metadata(0, 0, Vec::new(), input.metadata())?); + } + let first = histogram.iter().position(|&count| count != 0).expect("non-empty image"); + if histogram[first] as usize == total { + return Ok(Image::try_new_with_metadata( + input.width(), + input.height(), + vec![first as u8; total], + input.metadata(), + )?); + } + let scale = 255.0 / (total as f64 - histogram[first] as f64); + let mut cumulative = 0_u64; + let mut lookup = [0_u8; 256]; + for (value, &count) in histogram.iter().enumerate() { + cumulative += count; + lookup[value] = if value <= first { + 0 + } else { + cv_round((cumulative - histogram[first]) as f64 * scale).clamp(0, 255) as u8 + }; + } + map_u8(input, &lookup) +} + +/// Applies contrast-limited adaptive histogram equalization to grayscale u8. +pub fn clahe( + input: ImageView<'_, u8, 1>, + clip_limit: f64, + tiles_x: usize, + tiles_y: usize, +) -> VisionResult> { + if !clip_limit.is_finite() || clip_limit < 0.0 { + return Err(VisionError::InvalidParameter( + "CLAHE clip limit must be finite and non-negative".into(), + )); + } + if tiles_x == 0 || tiles_y == 0 { + return Err(VisionError::InvalidParameter( + "CLAHE tile grid dimensions must be non-zero".into(), + )); + } + if input.width() == 0 || input.height() == 0 { + return Ok(Image::try_new_with_metadata(0, 0, Vec::new(), input.metadata())?); + } + let tile_width = input.width().div_ceil(tiles_x); + let tile_height = input.height().div_ceil(tiles_y); + let tile_area = tile_width * tile_height; + let clip_count = if clip_limit > 0.0 { + ((clip_limit * tile_area as f64 / 256.0) as usize).max(1) + } else { + 0 + }; + let mut lookups = vec![[0_u8; 256]; tiles_x * tiles_y]; + for tile_y in 0..tiles_y { + for tile_x in 0..tiles_x { + let mut histogram = [0_usize; 256]; + for local_y in 0..tile_height { + for local_x in 0..tile_width { + let x = tile_x * tile_width + local_x; + let y = tile_y * tile_height + local_y; + let value = fetch(input, x as isize, y as isize, BorderMode::Reflect101)[0]; + histogram[value as usize] += 1; + } + } + if clip_count > 0 { + clip_histogram(&mut histogram, clip_count); + } + let scale = 255.0 / tile_area as f64; + let mut cumulative = 0_usize; + for (value, &count) in histogram.iter().enumerate() { + cumulative += count; + lookups[tile_y * tiles_x + tile_x][value] = + cv_round(cumulative as f64 * scale).clamp(0, 255) as u8; + } + } + } + interpolate_clahe(input, &lookups, tiles_x, tiles_y, tile_width, tile_height) +} + +/// Summed-area table with a zero top row and left column. +#[derive(Clone, Debug, PartialEq)] +pub struct IntegralImage { + source_width: usize, + source_height: usize, + data: Vec, +} + +impl IntegralImage { + /// Source image width, excluding the zero border. + #[must_use] + pub const fn source_width(&self) -> usize { + self.source_width + } + + /// Source image height, excluding the zero border. + #[must_use] + pub const fn source_height(&self) -> usize { + self.source_height + } + + /// Integral table width (`source_width + 1`). + #[must_use] + pub const fn width(&self) -> usize { + self.source_width + 1 + } + + /// Integral table height (`source_height + 1`). + #[must_use] + pub const fn height(&self) -> usize { + self.source_height + 1 + } + + /// Row-major integral values. + #[must_use] + pub fn as_slice(&self) -> &[f64] { + &self.data + } + + /// Returns the sum over half-open source rectangle `[x0,x1) × [y0,y1)`. + pub fn sum_region(&self, x0: usize, y0: usize, x1: usize, y1: usize) -> VisionResult { + if x0 > x1 || y0 > y1 || x1 > self.source_width || y1 > self.source_height { + return Err(VisionError::InvalidParameter( + "integral region is outside source dimensions".into(), + )); + } + let stride = self.width(); + Ok(self.data[y1 * stride + x1] - self.data[y0 * stride + x1] - self.data[y1 * stride + x0] + + self.data[y0 * stride + x0]) + } +} + +/// Computes a summed-area table for one selected channel. +pub fn integral_image( + input: ImageView<'_, T, CHANNELS>, + channel: usize, +) -> VisionResult { + if channel >= CHANNELS { + return Err(VisionError::InvalidParameter(format!( + "integral channel {channel} is outside {CHANNELS} channels" + ))); + } + let width = input + .width() + .checked_add(1) + .ok_or_else(|| VisionError::InvalidDimensions("integral width overflows".into()))?; + let height = input + .height() + .checked_add(1) + .ok_or_else(|| VisionError::InvalidDimensions("integral height overflows".into()))?; + let len = width + .checked_mul(height) + .ok_or_else(|| VisionError::InvalidDimensions("integral allocation overflows".into()))?; + let mut data = vec![0.0; len]; + for y in 0..input.height() { + let mut row_sum = 0.0; + for x in 0..input.width() { + row_sum += input.get(x, y).expect("coordinate in bounds")[channel].to_f64(); + data[(y + 1) * width + x + 1] = data[y * width + x + 1] + row_sum; + } + } + Ok(IntegralImage { source_width: input.width(), source_height: input.height(), data }) +} + +fn apply_threshold( + value: f64, + threshold: f64, + max_value: f64, + threshold_type: ThresholdType, +) -> f64 { + match threshold_type { + ThresholdType::Binary => { + if value > threshold { + max_value + } else { + 0.0 + } + } + ThresholdType::BinaryInv => { + if value > threshold { + 0.0 + } else { + max_value + } + } + ThresholdType::Truncate => value.min(threshold), + ThresholdType::ToZero => { + if value > threshold { + value + } else { + 0.0 + } + } + ThresholdType::ToZeroInv => { + if value > threshold { + 0.0 + } else { + value + } + } + } +} + +fn otsu_from_histogram(histogram: &[u64], total: usize) -> usize { + if total == 0 { + return 0; + } + let global_mean = histogram + .iter() + .enumerate() + .map(|(value, &count)| value as f64 * count as f64) + .sum::(); + let mut background_count = 0_u64; + let mut background_sum = 0.0; + let mut best_variance = -1.0; + let mut best = 0; + for (value, &count) in histogram.iter().enumerate() { + background_count += count; + if background_count == 0 { + continue; + } + let foreground_count = total as u64 - background_count; + if foreground_count == 0 { + break; + } + background_sum += value as f64 * count as f64; + let background_mean = background_sum / background_count as f64; + let foreground_mean = (global_mean - background_sum) / foreground_count as f64; + let difference = background_mean - foreground_mean; + let variance = background_count as f64 * foreground_count as f64 * difference * difference; + if variance > best_variance { + best_variance = variance; + best = value; + } + } + best +} + +fn gaussian_weights(size: usize) -> Vec { + let exact: Option<&[f64]> = match size { + 1 => Some(&[1.0]), + 3 => Some(&[0.25, 0.5, 0.25]), + 5 => Some(&[0.0625, 0.25, 0.375, 0.25, 0.0625]), + 7 => Some(&[0.03125, 0.109375, 0.21875, 0.28125, 0.21875, 0.109375, 0.03125]), + 9 => Some(&[ + 0.015625, 0.05078125, 0.1171875, 0.19921875, 0.234375, 0.19921875, 0.1171875, + 0.05078125, 0.015625, + ]), + _ => None, + }; + if let Some(exact) = exact { + return exact.to_vec(); + } + let sigma = 0.3 * ((size as f64 - 1.0) * 0.5 - 1.0) + 0.8; + let center = (size / 2) as f64; + let mut weights = (0..size) + .map(|index| { + let offset = index as f64 - center; + (-(offset * offset) / (2.0 * sigma * sigma)).exp() + }) + .collect::>(); + let sum = weights.iter().sum::(); + weights.iter_mut().for_each(|weight| *weight /= sum); + weights +} + +fn map_u8(input: ImageView<'_, u8, 1>, lookup: &[u8; 256]) -> VisionResult> { + let mut output = Vec::with_capacity(input.width() * input.height()); + for y in 0..input.height() { + for x in 0..input.width() { + output.push(lookup[input.get(x, y).expect("coordinate in bounds")[0] as usize]); + } + } + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +fn clip_histogram(histogram: &mut [usize; 256], clip_limit: usize) { + let mut clipped = 0_usize; + for count in histogram.iter_mut() { + if *count > clip_limit { + clipped += *count - clip_limit; + *count = clip_limit; + } + } + let batch = clipped / 256; + if batch > 0 { + histogram.iter_mut().for_each(|count| *count += batch); + } + let residual = clipped - batch * 256; + if residual > 0 { + let step = (256 / residual).max(1); + for index in (0..256).step_by(step).take(residual) { + histogram[index] += 1; + } + } +} + +fn interpolate_clahe( + input: ImageView<'_, u8, 1>, + lookups: &[[u8; 256]], + tiles_x: usize, + tiles_y: usize, + tile_width: usize, + tile_height: usize, +) -> VisionResult> { + let mut output = Vec::with_capacity(input.width() * input.height()); + for y in 0..input.height() { + let ty = y as f64 / tile_height as f64 - 0.5; + let y0_raw = ty.floor() as isize; + let ya = ty - ty.floor(); + let y0 = y0_raw.clamp(0, tiles_y as isize - 1) as usize; + let y1 = (y0_raw + 1).clamp(0, tiles_y as isize - 1) as usize; + for x in 0..input.width() { + let tx = x as f64 / tile_width as f64 - 0.5; + let x0_raw = tx.floor() as isize; + let xa = tx - tx.floor(); + let x0 = x0_raw.clamp(0, tiles_x as isize - 1) as usize; + let x1 = (x0_raw + 1).clamp(0, tiles_x as isize - 1) as usize; + let value = input.get(x, y).expect("coordinate in bounds")[0] as usize; + let top = f64::from(lookups[y0 * tiles_x + x0][value]) * (1.0 - xa) + + f64::from(lookups[y0 * tiles_x + x1][value]) * xa; + let bottom = f64::from(lookups[y1 * tiles_x + x0][value]) * (1.0 - xa) + + f64::from(lookups[y1 * tiles_x + x1][value]) * xa; + output.push(cv_round(top * (1.0 - ya) + bottom * ya).clamp(0, 255) as u8); + } + } + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +fn cv_round(value: f64) -> i64 { + let floor = value.floor(); + let fraction = value - floor; + if fraction < 0.5 { + floor as i64 + } else if fraction > 0.5 { + floor as i64 + 1 + } else { + let base = floor as i64; + if base % 2 == 0 { + base + } else { + base + 1 + } + } +} + +#[cfg(test)] +mod tests { + use super::{ + adaptive_threshold, clahe, equalize_histogram, histogram, integral_image, + otsu_threshold_u8, threshold, AdaptiveThresholdMethod, ThresholdType, + }; + use crate::BorderMode; + use spatialrust_image::{Image, ImageRegion}; + + #[test] + fn fixed_threshold_types_match_definitions_on_roi() { + let image = Image::::try_new(5, 1, vec![0, 10, 20, 30, 40]).unwrap(); + let roi = image.view().subview(ImageRegion::new(1, 0, 3, 1)).unwrap(); + assert_eq!( + threshold(roi, 20.0, 100.0, ThresholdType::Binary).unwrap().as_slice(), + &[0, 0, 100] + ); + assert_eq!( + threshold(roi, 20.0, 100.0, ThresholdType::Truncate).unwrap().as_slice(), + &[10, 20, 20] + ); + } + + #[test] + fn otsu_separates_bimodal_values() { + let image = Image::::try_new(6, 1, vec![10, 10, 10, 200, 200, 200]).unwrap(); + let (selected, output) = + otsu_threshold_u8(image.view(), 255, ThresholdType::Binary).unwrap(); + assert_eq!(selected, 10); + assert_eq!(output.as_slice(), &[0, 0, 0, 255, 255, 255]); + } + + #[test] + fn adaptive_threshold_rejects_even_blocks() { + let image = Image::::from_pixel(3, 3, [10]).unwrap(); + assert!(adaptive_threshold( + image.view(), + 255, + AdaptiveThresholdMethod::Mean, + ThresholdType::Binary, + 2, + 0.0, + BorderMode::Replicate + ) + .is_err()); + } + + #[test] + fn histogram_honors_channel_range_and_mask() { + let image = + Image::::try_new(2, 2, vec![0.0, 0.2, 0.5, 0.7, 0.9, 1.1, 1.0, 0.4]).unwrap(); + let mask = Image::::try_new(2, 2, vec![1, 0, 1, 1]).unwrap(); + assert_eq!( + histogram(image.view(), 0, 2, (0.0, 1.0), Some(mask.view())).unwrap(), + vec![1, 1] + ); + } + + #[test] + fn equalization_spreads_low_contrast_values() { + let image = Image::::try_new(4, 1, vec![10, 10, 20, 30]).unwrap(); + assert_eq!(equalize_histogram(image.view()).unwrap().as_slice(), &[0, 0, 128, 255]); + } + + #[test] + fn clahe_preserves_dimensions_and_constant_input() { + let image = Image::::from_pixel(7, 5, [42]).unwrap(); + let output = clahe(image.view(), 2.0, 3, 2).unwrap(); + assert_eq!((output.width(), output.height()), (7, 5)); + } + + #[test] + fn integral_region_matches_direct_sum() { + let image = Image::::try_new(3, 2, vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).unwrap(); + let integral = integral_image(image.view(), 0).unwrap(); + assert_eq!((integral.width(), integral.height()), (4, 3)); + assert_eq!(integral.sum_region(1, 0, 3, 2).unwrap(), 16.0); + } +} diff --git a/crates/spatialrust-vision/src/border.rs b/crates/spatialrust-vision/src/border.rs new file mode 100644 index 0000000..9e334ed --- /dev/null +++ b/crates/spatialrust-vision/src/border.rs @@ -0,0 +1,153 @@ +//! Shared border extrapolation for CPU image kernels. + +#[cfg(any( + feature = "warp", + feature = "imgproc-filter", + feature = "imgproc-morphology", + feature = "imgproc-analysis" +))] +use spatialrust_image::ImageView; + +#[cfg(any( + feature = "warp", + feature = "imgproc-filter", + feature = "imgproc-morphology", + feature = "imgproc-analysis" +))] +use crate::PixelComponent; + +/// Out-of-bounds sampling behavior for CPU image operations. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum BorderMode { + /// Returns a fixed pixel outside the source image. + Constant([T; CHANNELS]), + /// Repeats the closest edge pixel. + Replicate, + /// Reflects including the edge pixel (`fedcba|abcdefgh|hgfedc`). + Reflect, + /// Reflects without repeating the edge (`gfedcb|abcdefgh|gfedcb`). + Reflect101, + /// Periodically wraps source coordinates. + Wrap, +} + +#[cfg(any( + feature = "warp", + feature = "imgproc-filter", + feature = "imgproc-morphology", + feature = "imgproc-analysis" +))] +pub(crate) fn constant_pixel( + border: BorderMode, +) -> [T; CHANNELS] { + match border { + BorderMode::Constant(pixel) => pixel, + BorderMode::Replicate | BorderMode::Reflect | BorderMode::Reflect101 | BorderMode::Wrap => { + std::array::from_fn(|_| T::from_f64(0.0)) + } + } +} + +#[cfg(any( + feature = "warp", + feature = "imgproc-filter", + feature = "imgproc-morphology", + feature = "imgproc-analysis" +))] +pub(crate) fn fetch( + input: ImageView<'_, T, CHANNELS>, + x: isize, + y: isize, + border: BorderMode, +) -> [T; CHANNELS] { + if let (Some(ix), Some(iy)) = + (map_index(x, input.width(), border), map_index(y, input.height(), border)) + { + return *input.get(ix, iy).expect("mapped coordinate is in bounds"); + } + constant_pixel(border) +} + +#[cfg(any( + feature = "warp", + feature = "imgproc-filter", + feature = "imgproc-morphology", + feature = "imgproc-analysis" +))] +pub(crate) fn map_index( + index: isize, + length: usize, + border: BorderMode, +) -> Option { + if length == 0 { + return None; + } + if index >= 0 && index < length as isize { + return Some(index as usize); + } + match border { + BorderMode::Constant(_) => None, + BorderMode::Replicate => Some(index.clamp(0, length as isize - 1) as usize), + BorderMode::Reflect => Some(reflect_index(index, length, false)), + BorderMode::Reflect101 => Some(reflect_index(index, length, true)), + BorderMode::Wrap => Some(index.rem_euclid(length as isize) as usize), + } +} + +#[cfg(any( + feature = "warp", + feature = "imgproc-filter", + feature = "imgproc-morphology", + feature = "imgproc-analysis" +))] +fn reflect_index(mut index: isize, length: usize, reflect101: bool) -> usize { + if length <= 1 { + return 0; + } + let length = length as isize; + while index < 0 || index >= length { + index = if index < 0 { + if reflect101 { + -index + } else { + -index - 1 + } + } else if reflect101 { + 2 * length - index - 2 + } else { + 2 * length - index - 1 + }; + } + index as usize +} + +#[cfg(all( + test, + any( + feature = "warp", + feature = "imgproc-filter", + feature = "imgproc-morphology", + feature = "imgproc-analysis" + ) +))] +mod tests { + use super::{map_index, BorderMode}; + + #[test] + fn maps_single_pixel_without_looping() { + for border in [ + BorderMode::::Replicate, + BorderMode::Reflect, + BorderMode::Reflect101, + BorderMode::Wrap, + ] { + assert_eq!(map_index(-100, 1, border), Some(0)); + assert_eq!(map_index(100, 1, border), Some(0)); + } + } + + #[test] + fn empty_images_always_map_to_constant_space() { + assert_eq!(map_index(0, 0, BorderMode::::Wrap), None); + } +} diff --git a/crates/spatialrust-vision/src/canny.rs b/crates/spatialrust-vision/src/canny.rs new file mode 100644 index 0000000..ce44b77 --- /dev/null +++ b/crates/spatialrust-vision/src/canny.rs @@ -0,0 +1,296 @@ +//! Canny edge detection with inspectable CPU intermediates. + +use std::collections::VecDeque; + +use spatialrust_image::{Image, ImageView}; + +use crate::{sobel, BorderMode, VisionError, VisionResult}; + +/// Validated Canny thresholds and gradient settings. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct CannyOptions { + /// Lower hysteresis threshold. + pub low_threshold: f64, + /// Upper strong-edge threshold. + pub high_threshold: f64, + /// Sobel aperture size: 3, 5, or 7. + pub aperture_size: usize, + /// Use Euclidean gradient magnitude instead of the L1 approximation. + pub l2_gradient: bool, +} + +impl Default for CannyOptions { + fn default() -> Self { + Self { low_threshold: 100.0, high_threshold: 200.0, aperture_size: 3, l2_gradient: false } + } +} + +impl CannyOptions { + fn validate(self) -> VisionResult { + if !self.low_threshold.is_finite() + || !self.high_threshold.is_finite() + || self.low_threshold < 0.0 + || self.high_threshold < 0.0 + { + return Err(VisionError::InvalidParameter( + "Canny thresholds must be finite and non-negative".into(), + )); + } + if !matches!(self.aperture_size, 3 | 5 | 7) { + return Err(VisionError::InvalidParameter( + "Canny aperture size must be 3, 5, or 7".into(), + )); + } + Ok(self) + } +} + +/// Edge map and numerical stages produced by Canny. +#[derive(Clone, Debug, PartialEq)] +pub struct CannyResult { + /// Final binary edge map containing 0 or 255. + pub edges: Image, + /// Horizontal signed Sobel derivative. + pub gradient_x: Image, + /// Vertical signed Sobel derivative. + pub gradient_y: Image, + /// L1 or Euclidean gradient magnitude before suppression. + pub magnitude: Image, + /// Magnitude retained by directional non-maximum suppression. + pub suppressed: Image, +} + +/// Finds edges in a single-channel u8 image. +pub fn canny(input: ImageView<'_, u8, 1>, options: CannyOptions) -> VisionResult> { + Ok(canny_with_intermediates(input, options)?.edges) +} + +/// Runs Canny and retains gradient, magnitude, and suppression stages. +pub fn canny_with_intermediates( + input: ImageView<'_, u8, 1>, + options: CannyOptions, +) -> VisionResult { + let options = options.validate()?; + let scale = if options.aperture_size == 7 { 1.0 / 16.0 } else { 1.0 }; + let gradient_x = sobel(input, 1, 0, options.aperture_size, scale, 0.0, BorderMode::Replicate)?; + let gradient_y = sobel(input, 0, 1, options.aperture_size, scale, 0.0, BorderMode::Replicate)?; + let width = input.width(); + let height = input.height(); + let len = width + .checked_mul(height) + .ok_or_else(|| VisionError::InvalidDimensions("Canny image dimensions overflow".into()))?; + let gx = gradient_x + .as_slice() + .iter() + // OpenCV's CV_16S Sobel path uses cvRound, whose supported CPU paths + // round half-way values to even. This matters for aperture 7's 1/16 scale. + .map(|&value| round_i16_ties_even(value)) + .collect::>(); + let gy = + gradient_y.as_slice().iter().map(|&value| round_i16_ties_even(value)).collect::>(); + let comparison_magnitude = gx + .iter() + .zip(&gy) + .map(|(&x, &y)| { + if options.l2_gradient { + i64::from(x) * i64::from(x) + i64::from(y) * i64::from(y) + } else { + i64::from(x.abs() + y.abs()) + } + }) + .collect::>(); + let magnitude_values = comparison_magnitude + .iter() + .map(|&value| if options.l2_gradient { (value as f64).sqrt() as f32 } else { value as f32 }) + .collect::>(); + + let (mut low, mut high) = (options.low_threshold, options.high_threshold); + if options.aperture_size == 7 { + low /= 16.0; + high /= 16.0; + } + if low > high { + std::mem::swap(&mut low, &mut high); + } + let (low, high) = if options.l2_gradient { + ( + low.min(32767.0).mul_add(low.min(32767.0), 0.0).floor() as i64, + high.min(32767.0).mul_add(high.min(32767.0), 0.0).floor() as i64, + ) + } else { + (low.floor() as i64, high.floor() as i64) + }; + + let mut states = vec![1_u8; len]; + let mut suppressed_values = vec![0.0_f32; len]; + let mut strong = VecDeque::new(); + for y in 0..height { + for x in 0..width { + let index = y * width + x; + let magnitude = comparison_magnitude[index]; + if magnitude <= low + || !is_directional_maximum( + x, + y, + width, + height, + gx[index], + gy[index], + magnitude, + &comparison_magnitude, + ) + { + continue; + } + suppressed_values[index] = magnitude_values[index]; + if magnitude > high { + states[index] = 2; + strong.push_back(index); + } else { + states[index] = 0; + } + } + } + + while let Some(index) = strong.pop_front() { + let x = index % width; + let y = index / width; + for dy in -1_isize..=1 { + for dx in -1_isize..=1 { + if dx == 0 && dy == 0 { + continue; + } + let nx = x as isize + dx; + let ny = y as isize + dy; + if nx < 0 || ny < 0 || nx >= width as isize || ny >= height as isize { + continue; + } + let neighbor = ny as usize * width + nx as usize; + if states[neighbor] == 0 { + states[neighbor] = 2; + strong.push_back(neighbor); + } + } + } + } + let edges = states.into_iter().map(|state| if state == 2 { 255 } else { 0 }).collect(); + let metadata = input.metadata(); + Ok(CannyResult { + edges: Image::try_new_with_metadata(width, height, edges, metadata)?, + gradient_x, + gradient_y, + magnitude: Image::try_new_with_metadata(width, height, magnitude_values, metadata)?, + suppressed: Image::try_new_with_metadata(width, height, suppressed_values, metadata)?, + }) +} + +fn round_i16_ties_even(value: f32) -> i32 { + let value = value.clamp(f32::from(i16::MIN), f32::from(i16::MAX)); + let lower = value.floor(); + let fraction = value - lower; + if fraction < 0.5 { + lower as i32 + } else if fraction > 0.5 { + lower as i32 + 1 + } else { + let lower = lower as i32; + if lower & 1 == 0 { + lower + } else { + lower + 1 + } + } +} + +fn is_directional_maximum( + x: usize, + y: usize, + width: usize, + height: usize, + gradient_x: i32, + gradient_y: i32, + magnitude: i64, + magnitudes: &[i64], +) -> bool { + const TG22: i64 = 13_573; + let abs_x = i64::from(gradient_x.abs()); + let abs_y_scaled = i64::from(gradient_y.abs()) << 15; + let tg22_x = abs_x * TG22; + let get = |offset_x: isize, offset_y: isize| { + let nx = x as isize + offset_x; + let ny = y as isize + offset_y; + if nx < 0 || ny < 0 || nx >= width as isize || ny >= height as isize { + 0 + } else { + magnitudes[ny as usize * width + nx as usize] + } + }; + if abs_y_scaled < tg22_x { + magnitude > get(-1, 0) && magnitude >= get(1, 0) + } else if abs_y_scaled > tg22_x + (abs_x << 16) { + magnitude > get(0, -1) && magnitude >= get(0, 1) + } else { + let sign = if (gradient_x ^ gradient_y) < 0 { -1 } else { 1 }; + magnitude > get(-sign, -1) && magnitude > get(sign, 1) + } +} + +#[cfg(test)] +mod tests { + use super::{canny, canny_with_intermediates, CannyOptions}; + use spatialrust_image::{Image, ImageRegion}; + + #[test] + fn finds_both_sides_of_bright_bar_on_strided_roi() { + let mut data = vec![0_u8; 9 * 7]; + for y in 1..6 { + for x in 3..6 { + data[y * 9 + x] = 255; + } + } + let image = Image::::try_new(9, 7, data).unwrap(); + let roi = image.view().subview(ImageRegion::new(1, 1, 7, 5)).unwrap(); + let edges = canny( + roi, + CannyOptions { low_threshold: 50.0, high_threshold: 100.0, ..Default::default() }, + ) + .unwrap(); + assert!(edges.as_slice().iter().filter(|&&value| value == 255).count() >= 6); + assert!(edges.as_slice().iter().all(|&value| value == 0 || value == 255)); + } + + #[test] + fn intermediates_preserve_shape_and_signed_gradients() { + let image = + Image::::try_new(5, 3, (0..3).flat_map(|_| [0, 10, 20, 30, 40]).collect()) + .unwrap(); + let result = canny_with_intermediates(image.view(), CannyOptions::default()).unwrap(); + assert_eq!((result.edges.width(), result.edges.height()), (5, 3)); + assert!(result.gradient_x.as_slice().iter().any(|&value| value > 0.0)); + assert!(result.gradient_y.as_slice().iter().all(|&value| value == 0.0)); + } + + #[test] + fn flat_and_empty_images_have_no_edges() { + for image in [ + Image::::from_pixel(4, 3, [42]).unwrap(), + Image::::try_new(0, 0, Vec::new()).unwrap(), + ] { + assert!(canny(image.view(), CannyOptions::default()) + .unwrap() + .as_slice() + .iter() + .all(|&value| value == 0)); + } + } + + #[test] + fn validates_thresholds_and_aperture() { + let image = Image::::from_pixel(1, 1, [0]).unwrap(); + assert!( + canny(image.view(), CannyOptions { aperture_size: 4, ..Default::default() }).is_err() + ); + assert!(canny(image.view(), CannyOptions { low_threshold: -1.0, ..Default::default() }) + .is_err()); + } +} diff --git a/crates/spatialrust-vision/src/corners.rs b/crates/spatialrust-vision/src/corners.rs new file mode 100644 index 0000000..6fc090e --- /dev/null +++ b/crates/spatialrust-vision/src/corners.rs @@ -0,0 +1,449 @@ +//! Classical corner and accelerated segment-test detectors. + +use std::cmp::Ordering; + +use spatialrust_image::{Image, ImageView}; + +use crate::border::fetch; +use crate::{sobel, BorderMode, Keypoint2, VisionError, VisionResult}; + +/// Shared response thresholding and spatial suppression options. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct CornerSelectionOptions { + /// Maximum retained corners; zero keeps every accepted corner. + pub max_corners: usize, + /// Fraction of the strongest positive response in `(0, 1]`. + pub quality_level: f32, + /// Minimum Euclidean distance between retained corners in pixels. + pub min_distance: f32, + /// Odd structure-tensor summation window size. + pub block_size: usize, + /// Sobel aperture size: 3, 5, or 7. + pub gradient_size: usize, + /// Source-image border extrapolation used by derivatives. + pub border: BorderMode, +} + +impl Default for CornerSelectionOptions { + fn default() -> Self { + Self { + max_corners: 0, + quality_level: 0.01, + min_distance: 1.0, + block_size: 3, + gradient_size: 3, + border: BorderMode::Reflect101, + } + } +} + +impl CornerSelectionOptions { + fn validate(self) -> VisionResult { + if !self.quality_level.is_finite() || self.quality_level <= 0.0 || self.quality_level > 1.0 + { + return Err(VisionError::InvalidParameter( + "corner quality_level must be finite and in (0, 1]".into(), + )); + } + if !self.min_distance.is_finite() || self.min_distance < 0.0 { + return Err(VisionError::InvalidParameter( + "corner min_distance must be finite and non-negative".into(), + )); + } + if self.block_size == 0 || self.block_size % 2 == 0 { + return Err(VisionError::InvalidParameter( + "corner block_size must be positive and odd".into(), + )); + } + if !matches!(self.gradient_size, 3 | 5 | 7) { + return Err(VisionError::InvalidParameter( + "corner gradient_size must be 3, 5, or 7".into(), + )); + } + Ok(self) + } +} + +/// Harris detector configuration. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct HarrisOptions { + /// Shared candidate selection options. + pub selection: CornerSelectionOptions, + /// Harris trace penalty, normally near `0.04`. + pub k: f32, +} + +impl Default for HarrisOptions { + fn default() -> Self { + Self { selection: CornerSelectionOptions::default(), k: 0.04 } + } +} + +/// Shi–Tomasi minimum-eigenvalue detector configuration. +#[derive(Clone, Copy, Debug, Default, PartialEq)] +pub struct ShiTomasiOptions { + /// Shared candidate selection options. + pub selection: CornerSelectionOptions, +} + +/// Computes an unnormalized Harris structure-tensor response image. +pub fn harris_response( + input: ImageView<'_, u8, 1>, + block_size: usize, + gradient_size: usize, + k: f32, + border: BorderMode, +) -> VisionResult> { + if !k.is_finite() || k < 0.0 { + return Err(VisionError::InvalidParameter( + "Harris k must be finite and non-negative".into(), + )); + } + corner_response(input, block_size, gradient_size, border, |a, b, c| { + a.mul_add(c, -(b * b)) - k * (a + c) * (a + c) + }) +} + +/// Computes an unnormalized Shi–Tomasi minimum-eigenvalue response image. +pub fn shi_tomasi_response( + input: ImageView<'_, u8, 1>, + block_size: usize, + gradient_size: usize, + border: BorderMode, +) -> VisionResult> { + corner_response(input, block_size, gradient_size, border, |a, b, c| { + 0.5 * (a + c - ((a - c) * (a - c) + 4.0 * b * b).sqrt()) + }) +} + +/// Detects Harris corners and returns strongest-first keypoints. +pub fn detect_harris( + input: ImageView<'_, u8, 1>, + options: HarrisOptions, +) -> VisionResult> { + let selection = options.selection.validate()?; + let response = harris_response( + input, + selection.block_size, + selection.gradient_size, + options.k, + selection.border, + )?; + select_corner_responses(response.view(), selection) +} + +/// Detects Shi–Tomasi corners and returns strongest-first keypoints. +pub fn detect_shi_tomasi( + input: ImageView<'_, u8, 1>, + options: ShiTomasiOptions, +) -> VisionResult> { + let selection = options.selection.validate()?; + let response = shi_tomasi_response( + input, + selection.block_size, + selection.gradient_size, + selection.border, + )?; + select_corner_responses(response.view(), selection) +} + +fn corner_response( + input: ImageView<'_, u8, 1>, + block_size: usize, + gradient_size: usize, + border: BorderMode, + score: impl Fn(f32, f32, f32) -> f32, +) -> VisionResult> { + let validation = CornerSelectionOptions { + block_size, + gradient_size, + border, + ..CornerSelectionOptions::default() + } + .validate()?; + let gradient_x = sobel(input, 1, 0, validation.gradient_size, 1.0, 0.0, border)?; + let gradient_y = sobel(input, 0, 1, validation.gradient_size, 1.0, 0.0, border)?; + let products_xx = Image::try_new( + input.width(), + input.height(), + gradient_x.as_slice().iter().map(|value| value * value).collect(), + )?; + let products_xy = Image::try_new( + input.width(), + input.height(), + gradient_x.as_slice().iter().zip(gradient_y.as_slice()).map(|(x, y)| x * y).collect(), + )?; + let products_yy = Image::try_new( + input.width(), + input.height(), + gradient_y.as_slice().iter().map(|value| value * value).collect(), + )?; + let product_border = product_border(border); + let radius = (block_size / 2) as isize; + let mut output = Vec::with_capacity(input.width() * input.height()); + for y in 0..input.height() { + for x in 0..input.width() { + let mut a = 0.0_f32; + let mut b = 0.0_f32; + let mut c = 0.0_f32; + for dy in -radius..=radius { + for dx in -radius..=radius { + a += + fetch(products_xx.view(), x as isize + dx, y as isize + dy, product_border) + [0]; + b += + fetch(products_xy.view(), x as isize + dx, y as isize + dy, product_border) + [0]; + c += + fetch(products_yy.view(), x as isize + dx, y as isize + dy, product_border) + [0]; + } + } + output.push(score(a, b, c)); + } + } + Ok(Image::try_new(input.width(), input.height(), output)?) +} + +fn product_border(border: BorderMode) -> BorderMode { + match border { + BorderMode::Constant(_) => BorderMode::Constant([0.0]), + BorderMode::Replicate => BorderMode::Replicate, + BorderMode::Reflect => BorderMode::Reflect, + BorderMode::Reflect101 => BorderMode::Reflect101, + BorderMode::Wrap => BorderMode::Wrap, + } +} + +fn select_corner_responses( + response: ImageView<'_, f32, 1>, + options: CornerSelectionOptions, +) -> VisionResult> { + let maximum = (0..response.height()) + .flat_map(|y| response.row(y).expect("coordinate in bounds").iter().copied()) + .filter(|value| value.is_finite()) + .fold(f32::NEG_INFINITY, f32::max); + if maximum <= 0.0 || !maximum.is_finite() { + return Ok(Vec::new()); + } + if response.width() < 3 || response.height() < 3 { + return Ok(Vec::new()); + } + let threshold = maximum * options.quality_level; + let mut candidates = Vec::new(); + // OpenCV goodFeaturesToTrack applies 3x3 dilation/NMS only to pixels with + // a complete immediate neighborhood, excluding the outermost image ring. + for y in 1..response.height() - 1 { + for x in 1..response.width() - 1 { + let value = response.get(x, y).expect("coordinate in bounds")[0]; + if value <= threshold || !is_local_maximum(response, x, y, value) { + continue; + } + candidates.push((x, y, value)); + } + } + candidates.sort_by(|left, right| { + right + .2 + .partial_cmp(&left.2) + .unwrap_or(Ordering::Equal) + .then_with(|| left.1.cmp(&right.1)) + .then_with(|| left.0.cmp(&right.0)) + }); + let minimum_squared = options.min_distance * options.min_distance; + let mut selected = Vec::::new(); + for (x, y, value) in candidates { + if options.min_distance > 0.0 + && selected.iter().any(|keypoint| { + let dx = keypoint.x() - x as f32; + let dy = keypoint.y() - y as f32; + dx.mul_add(dx, dy * dy) < minimum_squared + }) + { + continue; + } + selected.push( + Keypoint2::try_new(x as f32, y as f32, value)?.with_size(options.block_size as f32)?, + ); + if options.max_corners != 0 && selected.len() == options.max_corners { + break; + } + } + Ok(selected) +} + +fn is_local_maximum(response: ImageView<'_, f32, 1>, x: usize, y: usize, value: f32) -> bool { + for dy in -1_isize..=1 { + for dx in -1_isize..=1 { + if dx == 0 && dy == 0 { + continue; + } + let nx = x as isize + dx; + let ny = y as isize + dy; + if nx >= 0 + && ny >= 0 + && (nx as usize) < response.width() + && (ny as usize) < response.height() + && response.get(nx as usize, ny as usize).expect("checked coordinate")[0] > value + { + return false; + } + } + } + true +} + +/// FAST detector configuration for the standard 9-of-16 radius-three circle. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct FastOptions { + /// Strict center-to-circle intensity threshold. + pub threshold: u8, + /// Retain only strict 3x3 maxima of the FAST score. + pub nonmax_suppression: bool, +} + +impl Default for FastOptions { + fn default() -> Self { + Self { threshold: 10, nonmax_suppression: true } + } +} + +const FAST_CIRCLE: [(isize, isize); 16] = [ + (0, -3), + (1, -3), + (2, -2), + (3, -1), + (3, 0), + (3, 1), + (2, 2), + (1, 3), + (0, 3), + (-1, 3), + (-2, 2), + (-3, 1), + (-3, 0), + (-3, -1), + (-2, -2), + (-1, -3), +]; + +/// Detects FAST-9/16 corners in scan order, matching OpenCV's radius-three ROI. +pub fn detect_fast( + input: ImageView<'_, u8, 1>, + options: FastOptions, +) -> VisionResult> { + if input.width() < 7 || input.height() < 7 { + return Ok(Vec::new()); + } + let mut scores = vec![0_u8; input.width() * input.height()]; + let mut candidates = Vec::new(); + for y in 3..input.height() - 3 { + for x in 3..input.width() - 3 { + let score = fast_score(input, x, y); + if score >= options.threshold { + scores[y * input.width() + x] = score; + candidates.push((x, y)); + } + } + } + let mut keypoints = Vec::new(); + for (x, y) in candidates { + let score = scores[y * input.width() + x]; + if options.nonmax_suppression { + let strict_maximum = (-1_isize..=1).all(|dy| { + (-1_isize..=1).all(|dx| { + (dx == 0 && dy == 0) + || score + > scores[(y as isize + dy) as usize * input.width() + + (x as isize + dx) as usize] + }) + }); + if !strict_maximum { + continue; + } + } + keypoints.push( + Keypoint2::try_new( + x as f32, + y as f32, + if options.nonmax_suppression { f32::from(score) } else { 0.0 }, + )? + .with_size(7.0)?, + ); + } + Ok(keypoints) +} + +fn fast_score(input: ImageView<'_, u8, 1>, x: usize, y: usize) -> u8 { + let center = i16::from(input.get(x, y).expect("coordinate in bounds")[0]); + let differences = std::array::from_fn::<_, 16, _>(|index| { + let (dx, dy) = FAST_CIRCLE[index]; + i16::from( + input + .get((x as isize + dx) as usize, (y as isize + dy) as usize) + .expect("FAST radius checked")[0], + ) - center + }); + let mut best = 0_i16; + for start in 0..16 { + let mut bright = i16::MAX; + let mut dark = i16::MAX; + for offset in 0..9 { + let difference = differences[(start + offset) % 16]; + bright = bright.min(difference); + dark = dark.min(-difference); + } + best = best.max(bright).max(dark); + } + best.saturating_sub(1).clamp(0, 255) as u8 +} + +#[cfg(test)] +mod tests { + use super::{ + detect_fast, detect_harris, detect_shi_tomasi, FastOptions, HarrisOptions, ShiTomasiOptions, + }; + use spatialrust_image::{Image, ImageRegion}; + + fn square() -> Image { + let mut image = Image::try_new(17, 15, vec![0; 17 * 15]).unwrap(); + for y in 4..11 { + for x in 5..13 { + image.get_mut(x, y).unwrap()[0] = 255; + } + } + image + } + + #[test] + fn harris_and_shi_tomasi_find_square_corners() { + let image = square(); + let harris = detect_harris(image.view(), HarrisOptions::default()).unwrap(); + let shi = detect_shi_tomasi(image.view(), ShiTomasiOptions::default()).unwrap(); + for corners in [&harris, &shi] { + assert!(corners.iter().any(|point| point.x() <= 6.0 && point.y() <= 5.0)); + assert!(corners.iter().any(|point| point.x() >= 11.0 && point.y() >= 9.0)); + } + } + + #[test] + fn corner_detectors_accept_strided_roi() { + let image = square(); + let roi = image.view().subview(ImageRegion::new(2, 2, 13, 11)).unwrap(); + assert!(!detect_harris(roi, HarrisOptions::default()).unwrap().is_empty()); + } + + #[test] + fn fast_detects_contrast_corner_and_handles_tiny_images() { + let mut image = Image::try_new(9, 9, vec![0; 81]).unwrap(); + image.get_mut(4, 4).unwrap()[0] = 255; + let corners = + detect_fast(image.view(), FastOptions { threshold: 20, nonmax_suppression: true }) + .unwrap(); + assert_eq!(corners.len(), 1); + assert_eq!((corners[0].x(), corners[0].y()), (4.0, 4.0)); + assert!(corners.iter().all(|point| point.size() == 7.0)); + let tiny = Image::try_new(6, 6, vec![0; 36]).unwrap(); + assert!(detect_fast(tiny.view(), FastOptions::default()).unwrap().is_empty()); + } +} diff --git a/crates/spatialrust-vision/src/error.rs b/crates/spatialrust-vision/src/error.rs index 38407a8..7801780 100644 --- a/crates/spatialrust-vision/src/error.rs +++ b/crates/spatialrust-vision/src/error.rs @@ -18,6 +18,52 @@ pub enum VisionError { /// Input collections or maps had incompatible shapes. #[error("shape mismatch: {0}")] ShapeMismatch(String), + /// A descriptor buffer does not match its declared row layout. + #[error("descriptor layout {rows}x{width} requires {expected} values, found {actual}")] + DescriptorLayout { + /// Descriptor row count. + rows: usize, + /// Values or bytes per descriptor. + width: usize, + /// Required flat storage length. + expected: usize, + /// Supplied flat storage length. + actual: usize, + }, + /// Keypoint and descriptor row counts differ. + #[error("feature set has {keypoints} keypoints but {descriptors} descriptor rows")] + FeatureCountMismatch { + /// Number of keypoints. + keypoints: usize, + /// Number of descriptors. + descriptors: usize, + }, + /// A feature match references a keypoint outside its collection. + #[error( + "feature match index is out of bounds: query {query}/{queries}, train {train}/{trains}" + )] + MatchIndexOutOfBounds { + /// Query index. + query: usize, + /// Query feature count. + queries: usize, + /// Train index. + train: usize, + /// Train feature count. + trains: usize, + }, + /// Inlier and residual arrays do not match the correspondence count. + #[error( + "geometry result for {correspondences} correspondences has {inliers} inlier flags and {residuals} residuals" + )] + GeometryResultLayout { + /// Number of input correspondences. + correspondences: usize, + /// Number of inlier flags. + inliers: usize, + /// Number of residual values. + residuals: usize, + }, /// A geometric transform could not be inverted. #[error("transform is singular")] SingularTransform, diff --git a/crates/spatialrust-vision/src/feature2d.rs b/crates/spatialrust-vision/src/feature2d.rs new file mode 100644 index 0000000..ae08a2f --- /dev/null +++ b/crates/spatialrust-vision/src/feature2d.rs @@ -0,0 +1,341 @@ +//! Feature2D keypoint, descriptor, and correspondence data contracts. + +use crate::{VisionError, VisionResult}; + +/// One scale-space image keypoint in pixel coordinates. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct Keypoint2 { + x: f32, + y: f32, + size: f32, + angle_degrees: Option, + response: f32, + octave: i32, + class_id: Option, +} + +impl Keypoint2 { + /// Creates an unoriented one-pixel keypoint with a detector response. + pub fn try_new(x: f32, y: f32, response: f32) -> VisionResult { + if !x.is_finite() || !y.is_finite() || !response.is_finite() { + return Err(VisionError::InvalidParameter( + "keypoint coordinates and response must be finite".into(), + )); + } + Ok(Self { x, y, size: 1.0, angle_degrees: None, response, octave: 0, class_id: None }) + } + + /// Sets a positive finite feature diameter in pixels. + pub fn with_size(mut self, size: f32) -> VisionResult { + if !size.is_finite() || size <= 0.0 { + return Err(VisionError::InvalidParameter( + "keypoint size must be positive and finite".into(), + )); + } + self.size = size; + Ok(self) + } + + /// Sets a finite orientation, normalized into `[0, 360)` degrees. + pub fn with_angle_degrees(mut self, angle: f32) -> VisionResult { + if !angle.is_finite() { + return Err(VisionError::InvalidParameter("keypoint angle must be finite".into())); + } + self.angle_degrees = Some(angle.rem_euclid(360.0)); + Ok(self) + } + + /// Sets a scale-pyramid octave. + #[must_use] + pub const fn with_octave(mut self, octave: i32) -> Self { + self.octave = octave; + self + } + + /// Sets an optional detector- or application-defined class identifier. + #[must_use] + pub const fn with_class_id(mut self, class_id: i32) -> Self { + self.class_id = Some(class_id); + self + } + + /// Returns the horizontal pixel coordinate. + pub const fn x(self) -> f32 { + self.x + } + + /// Returns the vertical pixel coordinate. + pub const fn y(self) -> f32 { + self.y + } + + /// Returns the feature diameter in pixels. + pub const fn size(self) -> f32 { + self.size + } + + /// Returns the normalized orientation, or `None` when not estimated. + pub const fn angle_degrees(self) -> Option { + self.angle_degrees + } + + /// Returns the detector response; larger values are more salient. + pub const fn response(self) -> f32 { + self.response + } + + /// Returns the scale-pyramid octave. + pub const fn octave(self) -> i32 { + self.octave + } + + /// Returns the optional application-defined class identifier. + pub const fn class_id(self) -> Option { + self.class_id + } +} + +/// Scalar representation used by a descriptor matrix. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum DescriptorKind { + /// Packed binary descriptor; matching uses Hamming distance. + Binary, + /// Float32 descriptor; matching normally uses L2 distance. + Float32, +} + +#[derive(Clone, Debug, PartialEq)] +enum DescriptorStorage { + Binary(Vec), + Float32(Vec), +} + +/// Row-major descriptor matrix with a fixed width and explicit scalar kind. +#[derive(Clone, Debug, PartialEq)] +pub struct DescriptorBuffer { + rows: usize, + width: usize, + storage: DescriptorStorage, +} + +impl DescriptorBuffer { + /// Creates a checked row-major binary descriptor matrix. + pub fn try_binary(rows: usize, bytes_per_row: usize, data: Vec) -> VisionResult { + Self::validate_layout(rows, bytes_per_row, data.len())?; + Ok(Self { rows, width: bytes_per_row, storage: DescriptorStorage::Binary(data) }) + } + + /// Creates a checked row-major float32 descriptor matrix. + pub fn try_float32(rows: usize, values_per_row: usize, data: Vec) -> VisionResult { + if data.iter().any(|value| !value.is_finite()) { + return Err(VisionError::InvalidParameter( + "float descriptors must contain only finite values".into(), + )); + } + Self::validate_layout(rows, values_per_row, data.len())?; + Ok(Self { rows, width: values_per_row, storage: DescriptorStorage::Float32(data) }) + } + + fn validate_layout(rows: usize, width: usize, actual: usize) -> VisionResult<()> { + if width == 0 { + return Err(VisionError::InvalidParameter("descriptor width must be positive".into())); + } + let expected = rows + .checked_mul(width) + .ok_or_else(|| VisionError::InvalidParameter("descriptor layout overflows".into()))?; + if actual != expected { + return Err(VisionError::DescriptorLayout { rows, width, expected, actual }); + } + Ok(()) + } + + /// Returns the descriptor scalar representation. + pub const fn kind(&self) -> DescriptorKind { + match self.storage { + DescriptorStorage::Binary(_) => DescriptorKind::Binary, + DescriptorStorage::Float32(_) => DescriptorKind::Float32, + } + } + + /// Returns the descriptor row count. + pub const fn len(&self) -> usize { + self.rows + } + + /// Returns whether there are no descriptor rows. + pub const fn is_empty(&self) -> bool { + self.rows == 0 + } + + /// Returns bytes or float values per descriptor. + pub const fn width(&self) -> usize { + self.width + } + + /// Returns one binary descriptor, or `None` for a wrong kind or row index. + pub fn binary_row(&self, index: usize) -> Option<&[u8]> { + let DescriptorStorage::Binary(data) = &self.storage else { + return None; + }; + let start = index.checked_mul(self.width)?; + data.get(start..start + self.width) + } + + /// Returns one float descriptor, or `None` for a wrong kind or row index. + pub fn float32_row(&self, index: usize) -> Option<&[f32]> { + let DescriptorStorage::Float32(data) = &self.storage else { + return None; + }; + let start = index.checked_mul(self.width)?; + data.get(start..start + self.width) + } + + /// Returns packed binary storage when the descriptor kind is binary. + pub fn binary_data(&self) -> Option<&[u8]> { + match &self.storage { + DescriptorStorage::Binary(data) => Some(data), + DescriptorStorage::Float32(_) => None, + } + } + + /// Returns packed float storage when the descriptor kind is float32. + pub fn float32_data(&self) -> Option<&[f32]> { + match &self.storage { + DescriptorStorage::Float32(data) => Some(data), + DescriptorStorage::Binary(_) => None, + } + } +} + +/// Keypoints paired one-to-one with descriptor rows. +#[derive(Clone, Debug, PartialEq)] +pub struct FeatureSet2 { + keypoints: Vec, + descriptors: DescriptorBuffer, +} + +impl FeatureSet2 { + /// Validates and owns a keypoint/descriptor pair. + pub fn try_new(keypoints: Vec, descriptors: DescriptorBuffer) -> VisionResult { + if keypoints.len() != descriptors.len() { + return Err(VisionError::FeatureCountMismatch { + keypoints: keypoints.len(), + descriptors: descriptors.len(), + }); + } + Ok(Self { keypoints, descriptors }) + } + + /// Returns keypoints in descriptor-row order. + pub fn keypoints(&self) -> &[Keypoint2] { + &self.keypoints + } + + /// Returns the associated descriptor matrix. + pub const fn descriptors(&self) -> &DescriptorBuffer { + &self.descriptors + } +} + +/// One descriptor correspondence from a query feature to a train feature. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct FeatureMatch { + query_index: usize, + train_index: usize, + distance: f32, +} + +impl FeatureMatch { + /// Creates a correspondence with a finite non-negative distance. + pub fn try_new(query_index: usize, train_index: usize, distance: f32) -> VisionResult { + if !distance.is_finite() || distance < 0.0 { + return Err(VisionError::InvalidParameter( + "feature-match distance must be finite and non-negative".into(), + )); + } + Ok(Self { query_index, train_index, distance }) + } + + /// Validates both indices against their feature collections. + pub fn validate(self, query_count: usize, train_count: usize) -> VisionResult { + if self.query_index >= query_count || self.train_index >= train_count { + return Err(VisionError::MatchIndexOutOfBounds { + query: self.query_index, + queries: query_count, + train: self.train_index, + trains: train_count, + }); + } + Ok(self) + } + + /// Returns the query descriptor row index. + pub const fn query_index(self) -> usize { + self.query_index + } + + /// Returns the train descriptor row index. + pub const fn train_index(self) -> usize { + self.train_index + } + + /// Returns the descriptor distance; smaller is a better match. + pub const fn distance(self) -> f32 { + self.distance + } +} + +#[cfg(test)] +mod tests { + use super::{DescriptorBuffer, DescriptorKind, FeatureMatch, FeatureSet2, Keypoint2}; + use crate::VisionError; + + #[test] + fn keypoint_normalizes_orientation_and_rejects_non_finite_values() { + let keypoint = Keypoint2::try_new(2.5, 3.5, -0.2) + .unwrap() + .with_size(7.0) + .unwrap() + .with_angle_degrees(-45.0) + .unwrap() + .with_octave(2) + .with_class_id(9); + assert_eq!(keypoint.angle_degrees(), Some(315.0)); + assert_eq!((keypoint.x(), keypoint.y(), keypoint.size()), (2.5, 3.5, 7.0)); + assert_eq!( + (keypoint.response(), keypoint.octave(), keypoint.class_id()), + (-0.2, 2, Some(9)) + ); + assert!(Keypoint2::try_new(f32::NAN, 0.0, 1.0).is_err()); + } + + #[test] + fn descriptor_layout_and_kind_are_checked() { + let binary = DescriptorBuffer::try_binary(2, 4, (0..8).collect()).unwrap(); + assert_eq!(binary.kind(), DescriptorKind::Binary); + assert_eq!(binary.binary_row(1), Some(&[4, 5, 6, 7][..])); + assert!(binary.float32_row(0).is_none()); + assert!(matches!( + DescriptorBuffer::try_float32(2, 3, vec![0.0; 5]), + Err(VisionError::DescriptorLayout { expected: 6, actual: 5, .. }) + )); + assert!(DescriptorBuffer::try_float32(1, 1, vec![f32::NAN]).is_err()); + } + + #[test] + fn feature_rows_and_match_indices_are_checked() { + let keypoint = Keypoint2::try_new(1.0, 2.0, 3.0).unwrap(); + let descriptors = DescriptorBuffer::try_binary(1, 2, vec![0xaa, 0x55]).unwrap(); + let features = FeatureSet2::try_new(vec![keypoint], descriptors).unwrap(); + assert_eq!(features.keypoints(), &[keypoint]); + assert_eq!(features.descriptors().len(), 1); + let correspondence = FeatureMatch::try_new(0, 0, 4.0).unwrap().validate(1, 1).unwrap(); + assert_eq!((correspondence.query_index(), correspondence.train_index()), (0, 0)); + assert_eq!(correspondence.distance(), 4.0); + assert!(FeatureMatch::try_new(0, 0, f32::INFINITY).is_err()); + assert!(matches!( + FeatureMatch::try_new(1, 0, 0.0).unwrap().validate(1, 1), + Err(VisionError::MatchIndexOutOfBounds { .. }) + )); + } +} diff --git a/crates/spatialrust-vision/src/filter.rs b/crates/spatialrust-vision/src/filter.rs new file mode 100644 index 0000000..57415e7 --- /dev/null +++ b/crates/spatialrust-vision/src/filter.rs @@ -0,0 +1,428 @@ +//! Linear CPU image filters with explicit border and kernel contracts. + +use spatialrust_image::{Image, ImageView}; + +use crate::border::{fetch, map_index}; +use crate::{BorderMode, PixelComponent, VisionError, VisionResult}; + +/// Validated two-dimensional correlation kernel. +#[derive(Clone, Debug, PartialEq)] +pub struct Kernel2D { + width: usize, + height: usize, + anchor_x: usize, + anchor_y: usize, + coefficients: Vec, +} + +impl Kernel2D { + /// Creates a kernel with its anchor at `(width / 2, height / 2)`. + pub fn try_new(width: usize, height: usize, coefficients: Vec) -> VisionResult { + Self::try_new_with_anchor(width, height, width / 2, height / 2, coefficients) + } + + /// Creates a kernel with an explicit in-kernel anchor. + pub fn try_new_with_anchor( + width: usize, + height: usize, + anchor_x: usize, + anchor_y: usize, + coefficients: Vec, + ) -> VisionResult { + if width == 0 || height == 0 { + return Err(VisionError::InvalidParameter("kernel dimensions must be non-zero".into())); + } + let expected = width + .checked_mul(height) + .ok_or_else(|| VisionError::InvalidParameter("kernel dimensions overflow".into()))?; + if coefficients.len() != expected { + return Err(VisionError::ShapeMismatch(format!( + "kernel needs {expected} coefficients, found {}", + coefficients.len() + ))); + } + if anchor_x >= width || anchor_y >= height { + return Err(VisionError::InvalidParameter(format!( + "kernel anchor ({anchor_x}, {anchor_y}) is outside {width}x{height}" + ))); + } + if coefficients.iter().any(|value| !value.is_finite()) { + return Err(VisionError::InvalidParameter("kernel coefficients must be finite".into())); + } + Ok(Self { width, height, anchor_x, anchor_y, coefficients }) + } + + /// Kernel width. + #[must_use] + pub const fn width(&self) -> usize { + self.width + } + + /// Kernel height. + #[must_use] + pub const fn height(&self) -> usize { + self.height + } + + /// Kernel anchor `(x, y)`. + #[must_use] + pub const fn anchor(&self) -> (usize, usize) { + (self.anchor_x, self.anchor_y) + } + + /// Row-major coefficients. + #[must_use] + pub fn coefficients(&self) -> &[f64] { + &self.coefficients + } + + /// Returns a kernel reversed in both axes for true convolution. + #[must_use] + pub fn reversed(&self) -> Self { + let mut coefficients = self.coefficients.clone(); + coefficients.reverse(); + Self { + width: self.width, + height: self.height, + anchor_x: self.width - 1 - self.anchor_x, + anchor_y: self.height - 1 - self.anchor_y, + coefficients, + } + } +} + +/// Validated one-dimensional correlation kernel. +#[derive(Clone, Debug, PartialEq)] +pub struct Kernel1D { + anchor: usize, + coefficients: Vec, +} + +impl Kernel1D { + /// Creates a kernel anchored at `coefficients.len() / 2`. + pub fn try_new(coefficients: Vec) -> VisionResult { + let anchor = coefficients.len() / 2; + Self::try_new_with_anchor(coefficients, anchor) + } + + /// Creates a one-dimensional kernel with an explicit anchor. + pub fn try_new_with_anchor(coefficients: Vec, anchor: usize) -> VisionResult { + if coefficients.is_empty() { + return Err(VisionError::InvalidParameter("kernel must not be empty".into())); + } + if anchor >= coefficients.len() { + return Err(VisionError::InvalidParameter(format!( + "kernel anchor {anchor} is outside length {}", + coefficients.len() + ))); + } + if coefficients.iter().any(|value| !value.is_finite()) { + return Err(VisionError::InvalidParameter("kernel coefficients must be finite".into())); + } + Ok(Self { anchor, coefficients }) + } + + /// Kernel length. + #[must_use] + pub fn len(&self) -> usize { + self.coefficients.len() + } + + /// Returns whether the kernel is empty. Valid kernels always return false. + #[must_use] + pub fn is_empty(&self) -> bool { + self.coefficients.is_empty() + } + + /// Anchor index. + #[must_use] + pub const fn anchor(&self) -> usize { + self.anchor + } + + /// Kernel coefficients. + #[must_use] + pub fn coefficients(&self) -> &[f64] { + &self.coefficients + } +} + +/// Applies OpenCV-style correlation and converts the result back to the input dtype. +pub fn filter2d( + input: ImageView<'_, T, CHANNELS>, + kernel: &Kernel2D, + delta: f64, + border: BorderMode, +) -> VisionResult> { + validate_delta(delta)?; + let accumulators = correlate(input, kernel, delta, border); + let output = accumulators.into_iter().map(T::from_f64).collect(); + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +/// Applies OpenCV-style correlation and preserves signed/fractional results as `f32`. +pub fn filter2d_f32( + input: ImageView<'_, T, CHANNELS>, + kernel: &Kernel2D, + delta: f64, + border: BorderMode, +) -> VisionResult> { + validate_delta(delta)?; + let output = + correlate(input, kernel, delta, border).into_iter().map(|value| value as f32).collect(); + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +/// Applies true convolution by reversing the supplied kernel around its anchor. +pub fn convolve2d( + input: ImageView<'_, T, CHANNELS>, + kernel: &Kernel2D, + delta: f64, + border: BorderMode, +) -> VisionResult> { + filter2d(input, &kernel.reversed(), delta, border) +} + +/// Applies horizontal and vertical kernels using an `f64` intermediate buffer. +pub fn separable_filter( + input: ImageView<'_, T, CHANNELS>, + kernel_x: &Kernel1D, + kernel_y: &Kernel1D, + delta: f64, + border: BorderMode, +) -> VisionResult> { + let output = separable_accumulators(input, kernel_x, kernel_y, delta, border)? + .into_iter() + .map(T::from_f64) + .collect(); + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +/// Applies separable kernels and preserves signed/fractional output as `f32`. +pub fn separable_filter_f32( + input: ImageView<'_, T, CHANNELS>, + kernel_x: &Kernel1D, + kernel_y: &Kernel1D, + delta: f64, + border: BorderMode, +) -> VisionResult> { + let output = separable_accumulators(input, kernel_x, kernel_y, delta, border)? + .into_iter() + .map(|value| value as f32) + .collect(); + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +/// Applies a normalized rectangular box filter. +pub fn box_blur( + input: ImageView<'_, T, CHANNELS>, + kernel_width: usize, + kernel_height: usize, + border: BorderMode, +) -> VisionResult> { + if kernel_width == 0 || kernel_height == 0 { + return Err(VisionError::InvalidParameter("box kernel dimensions must be non-zero".into())); + } + let x = Kernel1D::try_new(vec![1.0 / kernel_width as f64; kernel_width])?; + let y = Kernel1D::try_new(vec![1.0 / kernel_height as f64; kernel_height])?; + separable_filter(input, &x, &y, 0.0, border) +} + +/// Applies a separable Gaussian blur with explicit odd sizes and standard deviations. +pub fn gaussian_blur( + input: ImageView<'_, T, CHANNELS>, + kernel_width: usize, + kernel_height: usize, + sigma_x: f64, + sigma_y: f64, + border: BorderMode, +) -> VisionResult> { + let x = gaussian_kernel(kernel_width, sigma_x)?; + let y = gaussian_kernel(kernel_height, sigma_y)?; + separable_filter(input, &x, &y, 0.0, border) +} + +fn gaussian_kernel(size: usize, sigma: f64) -> VisionResult { + if size == 0 || size % 2 == 0 { + return Err(VisionError::InvalidParameter( + "Gaussian kernel size must be positive and odd".into(), + )); + } + if !sigma.is_finite() || sigma <= 0.0 { + return Err(VisionError::InvalidParameter( + "Gaussian sigma must be finite and positive".into(), + )); + } + let center = (size / 2) as f64; + let denominator = 2.0 * sigma * sigma; + let mut coefficients = (0..size) + .map(|index| { + let offset = index as f64 - center; + (-(offset * offset) / denominator).exp() + }) + .collect::>(); + let sum = coefficients.iter().sum::(); + for value in &mut coefficients { + *value /= sum; + } + Kernel1D::try_new(coefficients) +} + +fn validate_delta(delta: f64) -> VisionResult<()> { + if !delta.is_finite() { + return Err(VisionError::InvalidParameter("filter delta must be finite".into())); + } + Ok(()) +} + +fn correlate( + input: ImageView<'_, T, CHANNELS>, + kernel: &Kernel2D, + delta: f64, + border: BorderMode, +) -> Vec { + let mut output = Vec::with_capacity(input.width() * input.height() * CHANNELS); + for y in 0..input.height() { + for x in 0..input.width() { + let mut sums = [delta; CHANNELS]; + for ky in 0..kernel.height { + for kx in 0..kernel.width { + let pixel = fetch( + input, + x as isize + kx as isize - kernel.anchor_x as isize, + y as isize + ky as isize - kernel.anchor_y as isize, + border, + ); + let weight = kernel.coefficients[ky * kernel.width + kx]; + for channel in 0..CHANNELS { + sums[channel] += pixel[channel].to_f64() * weight; + } + } + } + output.extend_from_slice(&sums); + } + } + output +} + +fn separable_accumulators( + input: ImageView<'_, T, CHANNELS>, + kernel_x: &Kernel1D, + kernel_y: &Kernel1D, + delta: f64, + border: BorderMode, +) -> VisionResult> { + validate_delta(delta)?; + let mut horizontal = vec![0.0; input.width() * input.height() * CHANNELS]; + for y in 0..input.height() { + for x in 0..input.width() { + for (kx, &weight) in kernel_x.coefficients.iter().enumerate() { + let pixel = fetch( + input, + x as isize + kx as isize - kernel_x.anchor as isize, + y as isize, + border, + ); + for channel in 0..CHANNELS { + horizontal[(y * input.width() + x) * CHANNELS + channel] += + pixel[channel].to_f64() * weight; + } + } + } + } + + let constant_horizontal = match border { + BorderMode::Constant(pixel) => { + let sum_x = kernel_x.coefficients.iter().sum::(); + std::array::from_fn(|channel| pixel[channel].to_f64() * sum_x) + } + _ => [0.0; CHANNELS], + }; + let mut output = Vec::with_capacity(horizontal.len()); + for y in 0..input.height() { + for x in 0..input.width() { + let mut sums = [delta; CHANNELS]; + for (ky, &weight) in kernel_y.coefficients.iter().enumerate() { + let source_y = y as isize + ky as isize - kernel_y.anchor as isize; + if let Some(mapped_y) = map_index(source_y, input.height(), border) { + let offset = (mapped_y * input.width() + x) * CHANNELS; + for channel in 0..CHANNELS { + sums[channel] += horizontal[offset + channel] * weight; + } + } else { + for channel in 0..CHANNELS { + sums[channel] += constant_horizontal[channel] * weight; + } + } + } + output.extend_from_slice(&sums); + } + } + Ok(output) +} + +#[cfg(test)] +mod tests { + use super::{ + box_blur, convolve2d, filter2d, gaussian_blur, separable_filter, Kernel1D, Kernel2D, + }; + use crate::BorderMode; + use spatialrust_image::{Image, ImageRegion}; + + #[test] + fn filter2d_is_correlation_and_convolution_reverses() { + let image = Image::::try_new(3, 1, vec![1, 2, 4]).unwrap(); + let kernel = Kernel2D::try_new_with_anchor(2, 1, 0, 0, vec![1.0, 10.0]).unwrap(); + let correlation = filter2d(image.view(), &kernel, 0.0, BorderMode::Replicate).unwrap(); + let convolution = convolve2d(image.view(), &kernel, 0.0, BorderMode::Replicate).unwrap(); + assert_eq!(correlation.as_slice(), &[21, 42, 44]); + assert_eq!(convolution.as_slice(), &[11, 12, 24]); + } + + #[test] + fn separable_matches_outer_product_with_constant_border() { + let image = Image::::try_new(2, 2, vec![1, 2, 3, 4]).unwrap(); + let x = Kernel1D::try_new(vec![0.25, 0.5, 0.25]).unwrap(); + let y = Kernel1D::try_new(vec![0.25, 0.5, 0.25]).unwrap(); + let kernel = Kernel2D::try_new( + 3, + 3, + vec![0.0625, 0.125, 0.0625, 0.125, 0.25, 0.125, 0.0625, 0.125, 0.0625], + ) + .unwrap(); + let expected = filter2d(image.view(), &kernel, 0.0, BorderMode::Constant([9])).unwrap(); + let actual = + separable_filter(image.view(), &x, &y, 0.0, BorderMode::Constant([9])).unwrap(); + assert_eq!(actual, expected); + } + + #[test] + fn roi_and_packed_filter_results_match() { + let parent = Image::::try_new(5, 3, (0..15).collect()).unwrap(); + let roi = parent.view().subview(ImageRegion::new(1, 1, 3, 2)).unwrap(); + let packed = Image::::try_new(3, 2, vec![6, 7, 8, 11, 12, 13]).unwrap(); + let kernel = Kernel2D::try_new(3, 1, vec![0.25, 0.5, 0.25]).unwrap(); + assert_eq!( + filter2d(roi, &kernel, 0.0, BorderMode::Reflect101).unwrap(), + filter2d(packed.view(), &kernel, 0.0, BorderMode::Reflect101).unwrap() + ); + } + + #[test] + fn normalized_blurs_preserve_constant_images() { + let image = Image::::from_pixel(4, 3, [2.0, 4.0, 8.0]).unwrap(); + let box_output = box_blur(image.view(), 4, 2, BorderMode::Replicate).unwrap(); + let gaussian = gaussian_blur(image.view(), 5, 3, 1.2, 0.8, BorderMode::Reflect101).unwrap(); + assert_eq!(box_output, image); + for (actual, expected) in gaussian.as_slice().iter().zip(image.as_slice()) { + assert!((actual - expected).abs() < 1e-5); + } + } + + #[test] + fn invalid_kernels_are_rejected() { + assert!(Kernel2D::try_new(0, 1, Vec::new()).is_err()); + assert!(Kernel2D::try_new(2, 2, vec![1.0; 3]).is_err()); + assert!(Kernel1D::try_new(vec![f64::NAN]).is_err()); + } +} diff --git a/crates/spatialrust-vision/src/geometry.rs b/crates/spatialrust-vision/src/geometry.rs new file mode 100644 index 0000000..19747e5 --- /dev/null +++ b/crates/spatialrust-vision/src/geometry.rs @@ -0,0 +1,498 @@ +//! Checked data contracts for calibrated and uncalibrated multiview geometry. + +use spatialrust_camera::CameraIntrinsics; +use spatialrust_math::{Mat3, Vec2, Vec3}; + +use crate::{VisionError, VisionResult}; + +/// One ordered pixel correspondence from a source image to a target image. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct PointCorrespondence2 { + source: Vec2, + target: Vec2, +} + +impl PointCorrespondence2 { + /// Creates a correspondence with finite pixel coordinates. + pub fn try_new(source: Vec2, target: Vec2) -> VisionResult { + if ![source.x, source.y, target.x, target.y].into_iter().all(f64::is_finite) { + return Err(VisionError::InvalidParameter( + "2D correspondence coordinates must be finite".into(), + )); + } + Ok(Self { source, target }) + } + + /// Returns the source-image pixel. + pub const fn source(self) -> Vec2 { + self.source + } + + /// Returns the target-image pixel. + pub const fn target(self) -> Vec2 { + self.target + } +} + +/// Calibrated pinhole intrinsic matrix and its analytic inverse. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct CameraMatrix3 { + matrix: Mat3, + inverse: Mat3, +} + +impl CameraMatrix3 { + /// Builds `K` and `K^-1` from validated camera intrinsics. + #[must_use] + pub fn from_intrinsics(intrinsics: CameraIntrinsics) -> Self { + let matrix = Mat3::from_rows( + [intrinsics.fx, 0.0, intrinsics.cx], + [0.0, intrinsics.fy, intrinsics.cy], + [0.0, 0.0, 1.0], + ); + let inverse = Mat3::from_rows( + [1.0 / intrinsics.fx, 0.0, -intrinsics.cx / intrinsics.fx], + [0.0, 1.0 / intrinsics.fy, -intrinsics.cy / intrinsics.fy], + [0.0, 0.0, 1.0], + ); + Self { matrix, inverse } + } + + /// Builds a checked pinhole `K` with analytic inverse (zero skew). + pub(crate) fn try_from_pinhole( + fx: f64, + fy: f64, + cx: f64, + cy: f64, + ) -> VisionResult { + if ![fx, fy, cx, cy].into_iter().all(f64::is_finite) || fx <= 0.0 || fy <= 0.0 { + return Err(VisionError::InvalidParameter( + "pinhole camera matrix requires finite positive focal lengths".into(), + )); + } + Ok(Self { + matrix: Mat3::from_rows([fx, 0.0, cx], [0.0, fy, cy], [0.0, 0.0, 1.0]), + inverse: Mat3::from_rows( + [1.0 / fx, 0.0, -cx / fx], + [0.0, 1.0 / fy, -cy / fy], + [0.0, 0.0, 1.0], + ), + }) + } + + /// Returns the row-major intrinsic matrix `K`. + pub const fn matrix(self) -> Mat3 { + self.matrix + } + + /// Returns the row-major inverse intrinsic matrix `K^-1`. + pub const fn inverse(self) -> Mat3 { + self.inverse + } + + /// Converts a pixel into the homogeneous normalized camera plane. + #[must_use] + pub fn normalize_pixel(self, pixel: Vec2) -> Vec3 { + self.inverse.mul_vec3(Vec3::new(pixel.x, pixel.y, 1.0)) + } +} + +fn validate_projective_matrix(matrix: Mat3, name: &str) -> VisionResult> { + if matrix.m.iter().flatten().any(|value| !value.is_finite()) { + return Err(VisionError::InvalidParameter(format!( + "{name} matrix elements must be finite" + ))); + } + let norm_squared = matrix.m.iter().flatten().map(|value| value * value).sum::(); + if norm_squared <= f64::EPSILON { + return Err(VisionError::InvalidParameter(format!("{name} matrix must not be all zero"))); + } + Ok(matrix) +} + +macro_rules! projective_model { + ($name:ident, $summary:literal, $label:literal) => { + #[doc = $summary] + #[derive(Clone, Copy, Debug, PartialEq)] + pub struct $name(Mat3); + + impl $name { + /// Creates a checked finite, non-zero projective matrix. + pub fn try_new(matrix: Mat3) -> VisionResult { + Ok(Self(validate_projective_matrix(matrix, $label)?)) + } + + /// Returns the row-major matrix without changing its arbitrary scale. + pub const fn matrix(self) -> Mat3 { + self.0 + } + } + }; +} + +projective_model!(Homography3, "A pixel-to-pixel planar projective transform.", "homography"); +projective_model!( + Fundamental3, + "An uncalibrated two-view epipolar constraint matrix.", + "fundamental" +); +projective_model!(Essential3, "A calibrated two-view epipolar constraint matrix.", "essential"); + +/// Deterministic robust-model sampling and inlier classification settings. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct RobustEstimationOptions { + /// Maximum accepted geometric residual in pixels or normalized units. + pub threshold: f64, + /// Desired probability of sampling an outlier-free minimal set. + pub confidence: f64, + /// Hard iteration limit. + pub max_iterations: usize, + /// Reproducible pseudo-random sampling seed. + pub seed: u64, +} + +impl Default for RobustEstimationOptions { + fn default() -> Self { + Self { threshold: 1.0, confidence: 0.99, max_iterations: 2_000, seed: 0 } + } +} + +impl RobustEstimationOptions { + /// Validates thresholds, confidence, and the iteration budget. + pub fn validate(self) -> VisionResult { + if !self.threshold.is_finite() || self.threshold <= 0.0 { + return Err(VisionError::InvalidParameter( + "geometry threshold must be finite and positive".into(), + )); + } + if !self.confidence.is_finite() || self.confidence <= 0.0 || self.confidence >= 1.0 { + return Err(VisionError::InvalidParameter( + "geometry confidence must be finite and in (0, 1)".into(), + )); + } + if self.max_iterations == 0 { + return Err(VisionError::InvalidParameter( + "geometry max_iterations must be positive".into(), + )); + } + Ok(self) + } +} + +/// A geometric model paired with one inlier decision and residual per input row. +#[derive(Clone, Debug, PartialEq)] +pub struct GeometricEstimate { + model: Model, + inliers: Vec, + residuals: Vec, +} + +/// Target-camera pose expressed in the source-camera coordinate frame. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct RelativePose { + rotation: Mat3, + translation: Vec3, +} + +impl RelativePose { + /// Creates a checked proper rotation and finite non-zero translation. + pub fn try_new(rotation: Mat3, translation: Vec3) -> VisionResult { + validate_rotation(rotation, "relative pose")?; + if ![translation.x, translation.y, translation.z].into_iter().all(f64::is_finite) { + return Err(VisionError::InvalidParameter( + "relative pose translation must be finite".into(), + )); + } + if translation.length() <= f64::EPSILON { + return Err(VisionError::InvalidParameter( + "relative pose translation must be non-zero".into(), + )); + } + Ok(Self { rotation, translation }) + } + + /// Returns the source-to-target rotation. + pub const fn rotation(self) -> Mat3 { + self.rotation + } + + /// Returns the source-to-target translation, whose scale may be arbitrary. + pub const fn translation(self) -> Vec3 { + self.translation + } +} + +/// Object-to-camera pose `X_cam = R X_obj + t` used by absolute orientation (PnP). +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct AbsolutePose { + rotation: Mat3, + translation: Vec3, +} + +impl AbsolutePose { + /// Creates a checked proper rotation and finite translation (may be zero). + pub fn try_new(rotation: Mat3, translation: Vec3) -> VisionResult { + validate_rotation(rotation, "absolute pose")?; + if ![translation.x, translation.y, translation.z].into_iter().all(f64::is_finite) { + return Err(VisionError::InvalidParameter( + "absolute pose translation must be finite".into(), + )); + } + Ok(Self { rotation, translation }) + } + + /// Returns the object-to-camera rotation. + pub const fn rotation(self) -> Mat3 { + self.rotation + } + + /// Returns the object-to-camera translation. + pub const fn translation(self) -> Vec3 { + self.translation + } + + /// Transforms an object-frame point into the camera frame. + #[must_use] + pub fn transform_point(self, point: Vec3) -> Vec3 { + self.rotation.mul_vec3(point) + self.translation + } +} + +/// One ordered object-to-image correspondence for absolute pose estimation. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct ObjectImageCorrespondence { + object: Vec3, + image: Vec2, +} + +impl ObjectImageCorrespondence { + /// Creates a correspondence with finite object and image coordinates. + pub fn try_new(object: Vec3, image: Vec2) -> VisionResult { + if ![object.x, object.y, object.z, image.x, image.y].into_iter().all(f64::is_finite) { + return Err(VisionError::InvalidParameter( + "object-image correspondence coordinates must be finite".into(), + )); + } + Ok(Self { object, image }) + } + + /// Returns the object-frame point. + pub const fn object(self) -> Vec3 { + self.object + } + + /// Returns the image-plane pixel. + pub const fn image(self) -> Vec2 { + self.image + } +} + +fn validate_rotation(rotation: Mat3, name: &str) -> VisionResult<()> { + if rotation.m.iter().flatten().any(|value| !value.is_finite()) { + return Err(VisionError::InvalidParameter(format!("{name} rotation must be finite"))); + } + let orthogonality = rotation.transpose().mul_mat3(rotation); + let identity = Mat3::::identity(); + let maximum_error = orthogonality + .m + .iter() + .flatten() + .zip(identity.m.iter().flatten()) + .map(|(actual, expected)| (actual - expected).abs()) + .fold(0.0_f64, f64::max); + if maximum_error > 1e-6 || determinant(rotation) < 1.0 - 1e-6 { + return Err(VisionError::InvalidParameter(format!( + "{name} rotation must be a proper orthonormal matrix" + ))); + } + Ok(()) +} + +/// One two-view triangulation with cheirality and reprojection diagnostics. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct TriangulatedPoint { + position: Vec3, + source_depth: f64, + target_depth: f64, + reprojection_error: f64, +} + +impl TriangulatedPoint { + pub(crate) fn try_new( + position: Vec3, + source_depth: f64, + target_depth: f64, + reprojection_error: f64, + ) -> VisionResult { + if ![position.x, position.y, position.z, source_depth, target_depth, reprojection_error] + .into_iter() + .all(f64::is_finite) + || reprojection_error < 0.0 + { + return Err(VisionError::InvalidParameter( + "triangulation values must be finite and error non-negative".into(), + )); + } + Ok(Self { position, source_depth, target_depth, reprojection_error }) + } + + /// Returns the point in source-camera coordinates. + pub const fn position(self) -> Vec3 { + self.position + } + + /// Returns its signed source-camera depth. + pub const fn source_depth(self) -> f64 { + self.source_depth + } + + /// Returns its signed target-camera depth. + pub const fn target_depth(self) -> f64 { + self.target_depth + } + + /// Returns the mean normalized-plane reprojection distance. + pub const fn reprojection_error(self) -> f64 { + self.reprojection_error + } + + /// Returns whether the point lies in front of both cameras. + pub fn has_positive_depth(self) -> bool { + self.source_depth > 0.0 && self.target_depth > 0.0 + } +} + +/// Essential-matrix pose disambiguation and per-correspondence triangulation. +#[derive(Clone, Debug, PartialEq)] +pub struct RelativePoseEstimate { + pose: RelativePose, + points: Vec>, + positive_depth_count: usize, +} + +impl RelativePoseEstimate { + pub(crate) fn new(pose: RelativePose, points: Vec>) -> Self { + let positive_depth_count = + points.iter().flatten().filter(|point| point.has_positive_depth()).count(); + Self { pose, points, positive_depth_count } + } + + /// Returns the selected source-to-target pose. + pub const fn pose(&self) -> RelativePose { + self.pose + } + + /// Returns one optional triangulation per input correspondence. + pub fn points(&self) -> &[Option] { + &self.points + } + + /// Returns the number of points in front of both cameras. + pub const fn positive_depth_count(&self) -> usize { + self.positive_depth_count + } +} + +fn determinant(matrix: Mat3) -> f64 { + matrix.m[0][0] * (matrix.m[1][1] * matrix.m[2][2] - matrix.m[1][2] * matrix.m[2][1]) + - matrix.m[0][1] * (matrix.m[1][0] * matrix.m[2][2] - matrix.m[1][2] * matrix.m[2][0]) + + matrix.m[0][2] * (matrix.m[1][0] * matrix.m[2][1] - matrix.m[1][1] * matrix.m[2][0]) +} + +impl GeometricEstimate { + /// Creates a result whose inlier and residual arrays match the input count. + pub fn try_new( + model: Model, + correspondence_count: usize, + inliers: Vec, + residuals: Vec, + ) -> VisionResult { + if inliers.len() != correspondence_count || residuals.len() != correspondence_count { + return Err(VisionError::GeometryResultLayout { + correspondences: correspondence_count, + inliers: inliers.len(), + residuals: residuals.len(), + }); + } + if residuals.iter().any(|value| !value.is_finite() || *value < 0.0) { + return Err(VisionError::InvalidParameter( + "geometry residuals must be finite and non-negative".into(), + )); + } + Ok(Self { model, inliers, residuals }) + } + + /// Returns the estimated model. + pub const fn model(&self) -> &Model { + &self.model + } + + /// Returns one inlier decision per input correspondence. + pub fn inliers(&self) -> &[bool] { + &self.inliers + } + + /// Returns one non-negative geometric residual per input correspondence. + pub fn residuals(&self) -> &[f64] { + &self.residuals + } + + /// Returns the number of accepted correspondences. + pub fn inlier_count(&self) -> usize { + self.inliers.iter().filter(|&&value| value).count() + } +} + +#[cfg(test)] +mod tests { + use super::{ + CameraMatrix3, GeometricEstimate, Homography3, PointCorrespondence2, + RobustEstimationOptions, + }; + use crate::VisionError; + use spatialrust_camera::CameraIntrinsics; + use spatialrust_math::{Mat3, Vec2}; + + #[test] + fn correspondence_and_projective_matrix_validation_is_strict() { + assert!(PointCorrespondence2::try_new(Vec2 { x: 1.0, y: 2.0 }, Vec2 { x: 3.0, y: 4.0 },) + .is_ok()); + assert!(PointCorrespondence2::try_new( + Vec2 { x: f64::NAN, y: 2.0 }, + Vec2 { x: 3.0, y: 4.0 }, + ) + .is_err()); + assert!(Homography3::try_new(Mat3::from_rows( + [0.0, 0.0, 0.0], + [0.0, 0.0, 0.0], + [0.0, 0.0, 0.0], + )) + .is_err()); + } + + #[test] + fn camera_matrix_normalizes_pixels_analytically() { + let intrinsics = CameraIntrinsics::try_new(500.0, 400.0, 320.0, 240.0, 640, 480).unwrap(); + let camera = CameraMatrix3::from_intrinsics(intrinsics); + let normalized = camera.normalize_pixel(Vec2 { x: 420.0, y: 160.0 }); + assert!((normalized.x - 0.2).abs() < 1e-12); + assert!((normalized.y + 0.2).abs() < 1e-12); + assert_eq!(normalized.z, 1.0); + assert_eq!(camera.matrix().mul_mat3(camera.inverse()), Mat3::::identity()); + } + + #[test] + fn estimate_layout_and_robust_options_are_checked() { + let model = Homography3::try_new(Mat3::::identity()).unwrap(); + let estimate = + GeometricEstimate::try_new(model, 2, vec![true, false], vec![0.1, 2.0]).unwrap(); + assert_eq!(estimate.inlier_count(), 1); + assert!(matches!( + GeometricEstimate::try_new(model, 2, vec![true], vec![0.1, 2.0]), + Err(VisionError::GeometryResultLayout { .. }) + )); + assert!(RobustEstimationOptions { confidence: 1.0, ..Default::default() } + .validate() + .is_err()); + } +} diff --git a/crates/spatialrust-vision/src/lib.rs b/crates/spatialrust-vision/src/lib.rs index 0669424..a82e89e 100644 --- a/crates/spatialrust-vision/src/lib.rs +++ b/crates/spatialrust-vision/src/lib.rs @@ -6,34 +6,94 @@ #![deny(unsafe_code)] #![warn(missing_docs)] +mod border; mod error; mod pixel; +#[cfg(feature = "imgproc-filter")] +mod advanced_filter; +#[cfg(feature = "imgproc-analysis")] +mod analysis; +#[cfg(feature = "imgproc-canny")] +mod canny; +#[cfg(feature = "feature2d")] +mod corners; + #[cfg(feature = "dense")] mod dense; #[cfg(feature = "detection")] mod detection; +#[cfg(feature = "feature2d")] +mod feature2d; +#[cfg(feature = "imgproc-filter")] +mod filter; +#[cfg(feature = "geometry")] +mod geometry; +#[cfg(feature = "feature2d")] +mod matcher; +#[cfg(feature = "imgproc-morphology")] +mod morphology; +#[cfg(feature = "geometry")] +mod multiview; +#[cfg(feature = "geometry")] +mod optical_flow; +#[cfg(feature = "geometry")] +mod pnp; +#[cfg(feature = "feature2d")] +mod orb; #[cfg(feature = "preprocess")] mod preprocess; #[cfg(feature = "resize")] mod resize; #[cfg(feature = "spatial")] mod spatial; +#[cfg(feature = "geometry")] +mod stereo; #[cfg(feature = "warp")] mod warp; +pub use border::BorderMode; pub use error::{VisionError, VisionResult}; pub use pixel::PixelComponent; +#[cfg(feature = "imgproc-filter")] +pub use advanced_filter::*; +#[cfg(feature = "imgproc-analysis")] +pub use analysis::*; +#[cfg(feature = "imgproc-canny")] +pub use canny::*; +#[cfg(feature = "feature2d")] +pub use corners::*; + #[cfg(feature = "dense")] pub use dense::*; #[cfg(feature = "detection")] pub use detection::*; +#[cfg(feature = "feature2d")] +pub use feature2d::*; +#[cfg(feature = "imgproc-filter")] +pub use filter::*; +#[cfg(feature = "geometry")] +pub use geometry::*; +#[cfg(feature = "feature2d")] +pub use matcher::*; +#[cfg(feature = "imgproc-morphology")] +pub use morphology::*; +#[cfg(feature = "geometry")] +pub use multiview::*; +#[cfg(feature = "geometry")] +pub use optical_flow::*; +#[cfg(feature = "geometry")] +pub use pnp::*; +#[cfg(feature = "feature2d")] +pub use orb::*; #[cfg(feature = "preprocess")] pub use preprocess::*; #[cfg(feature = "resize")] pub use resize::*; #[cfg(feature = "spatial")] pub use spatial::*; +#[cfg(feature = "geometry")] +pub use stereo::*; #[cfg(feature = "warp")] pub use warp::*; diff --git a/crates/spatialrust-vision/src/matcher.rs b/crates/spatialrust-vision/src/matcher.rs new file mode 100644 index 0000000..f9cdf1d --- /dev/null +++ b/crates/spatialrust-vision/src/matcher.rs @@ -0,0 +1,180 @@ +//! Brute-force descriptor matching with explicit distance semantics. + +use crate::{DescriptorBuffer, DescriptorKind, FeatureMatch, VisionError, VisionResult}; + +/// Filtering applied to brute-force descriptor correspondences. +#[derive(Clone, Copy, Debug, Default, PartialEq)] +pub struct MatchOptions { + /// Keep only pairs whose reverse nearest neighbour is the query row. + pub cross_check: bool, + /// Lowe ratio threshold in `(0, 1)`; requires at least two train rows. + pub ratio: Option, + /// Optional inclusive maximum distance. + pub max_distance: Option, +} + +impl MatchOptions { + fn validate(self) -> VisionResult { + if self.ratio.is_some_and(|ratio| !ratio.is_finite() || ratio <= 0.0 || ratio >= 1.0) { + return Err(VisionError::InvalidParameter( + "descriptor match ratio must be finite and in (0, 1)".into(), + )); + } + if self.max_distance.is_some_and(|distance| !distance.is_finite() || distance < 0.0) { + return Err(VisionError::InvalidParameter( + "descriptor maximum distance must be finite and non-negative".into(), + )); + } + Ok(self) + } +} + +/// Matches each query descriptor to its nearest train descriptor. +/// +/// Binary rows use Hamming distance and float rows use Euclidean L2 distance. +/// Equal distances are resolved by the lowest row index. Returned matches remain +/// in ascending query-row order. +pub fn match_descriptors( + query: &DescriptorBuffer, + train: &DescriptorBuffer, + options: MatchOptions, +) -> VisionResult> { + let options = options.validate()?; + validate_compatibility(query, train)?; + if query.is_empty() || train.is_empty() { + return Ok(Vec::new()); + } + if options.ratio.is_some() && train.len() < 2 { + return Err(VisionError::InvalidParameter( + "descriptor ratio matching requires at least two train rows".into(), + )); + } + + let reverse_best = options + .cross_check + .then(|| (0..train.len()).map(|index| nearest(train, index, query).0).collect::>()); + let mut matches = Vec::with_capacity(query.len()); + for query_index in 0..query.len() { + let (train_index, best, second) = nearest(query, query_index, train); + if options.ratio.is_some_and(|ratio| best >= ratio * second.unwrap_or(f32::INFINITY)) { + continue; + } + if options.max_distance.is_some_and(|maximum| best > maximum) { + continue; + } + if reverse_best.as_ref().is_some_and(|indices| indices[train_index] != query_index) { + continue; + } + matches.push(FeatureMatch::try_new(query_index, train_index, best)?); + } + Ok(matches) +} + +fn validate_compatibility(query: &DescriptorBuffer, train: &DescriptorBuffer) -> VisionResult<()> { + if query.kind() != train.kind() || query.width() != train.width() { + return Err(VisionError::ShapeMismatch(format!( + "descriptor matrices must have equal kind and width (query {:?}/{}, train {:?}/{})", + query.kind(), + query.width(), + train.kind(), + train.width() + ))); + } + Ok(()) +} + +fn nearest( + source: &DescriptorBuffer, + source_index: usize, + target: &DescriptorBuffer, +) -> (usize, f32, Option) { + let mut candidates = (0..target.len()) + .map(|target_index| (target_index, distance(source, source_index, target, target_index))) + .collect::>(); + candidates.sort_by(|left, right| left.1.total_cmp(&right.1).then_with(|| left.0.cmp(&right.0))); + let (best_index, best_distance) = candidates[0]; + (best_index, best_distance, candidates.get(1).map(|candidate| candidate.1)) +} + +fn distance( + left: &DescriptorBuffer, + left_index: usize, + right: &DescriptorBuffer, + right_index: usize, +) -> f32 { + match left.kind() { + DescriptorKind::Binary => left + .binary_row(left_index) + .expect("validated binary row") + .iter() + .zip(right.binary_row(right_index).expect("validated binary row")) + .map(|(a, b)| (a ^ b).count_ones()) + .sum::() as f32, + DescriptorKind::Float32 => left + .float32_row(left_index) + .expect("validated float row") + .iter() + .zip(right.float32_row(right_index).expect("validated float row")) + .map(|(a, b)| { + let delta = a - b; + delta * delta + }) + .sum::() + .sqrt(), + } +} + +#[cfg(test)] +mod tests { + use super::{match_descriptors, MatchOptions}; + use crate::DescriptorBuffer; + + #[test] + fn hamming_matching_is_deterministic_and_filters_distance() { + let query = DescriptorBuffer::try_binary(2, 1, vec![0b0000_0000, 0b1111_0000]).unwrap(); + let train = DescriptorBuffer::try_binary(3, 1, vec![0b0000_0011, 0b0000_1100, 0b1111_1111]) + .unwrap(); + let matches = match_descriptors(&query, &train, MatchOptions::default()).unwrap(); + assert_eq!((matches[0].train_index(), matches[0].distance()), (0, 2.0)); + assert_eq!((matches[1].train_index(), matches[1].distance()), (2, 4.0)); + + let filtered = match_descriptors( + &query, + &train, + MatchOptions { max_distance: Some(2.0), ..MatchOptions::default() }, + ) + .unwrap(); + assert_eq!(filtered.len(), 1); + } + + #[test] + fn l2_ratio_and_cross_check_match_expected_rows() { + let query = DescriptorBuffer::try_float32(2, 2, vec![0.0, 0.0, 10.0, 10.0]).unwrap(); + let train = + DescriptorBuffer::try_float32(3, 2, vec![1.0, 0.0, 3.0, 0.0, 10.0, 9.0]).unwrap(); + let matches = match_descriptors( + &query, + &train, + MatchOptions { cross_check: true, ratio: Some(0.8), max_distance: None }, + ) + .unwrap(); + assert_eq!(matches.len(), 2); + assert_eq!((matches[0].query_index(), matches[0].train_index()), (0, 0)); + assert_eq!((matches[1].query_index(), matches[1].train_index()), (1, 2)); + assert_eq!(matches[0].distance(), 1.0); + assert_eq!(matches[1].distance(), 1.0); + } + + #[test] + fn incompatible_and_invalid_match_options_are_rejected() { + let binary = DescriptorBuffer::try_binary(1, 1, vec![0]).unwrap(); + let float = DescriptorBuffer::try_float32(1, 1, vec![0.0]).unwrap(); + assert!(match_descriptors(&binary, &float, MatchOptions::default()).is_err()); + assert!(match_descriptors( + &binary, + &binary, + MatchOptions { ratio: Some(1.0), ..MatchOptions::default() } + ) + .is_err()); + } +} diff --git a/crates/spatialrust-vision/src/morphology.rs b/crates/spatialrust-vision/src/morphology.rs new file mode 100644 index 0000000..c3c0bd3 --- /dev/null +++ b/crates/spatialrust-vision/src/morphology.rs @@ -0,0 +1,406 @@ +//! CPU mathematical morphology with explicit structuring elements and borders. + +use spatialrust_image::{Image, ImageView}; + +use crate::border::fetch; +use crate::{BorderMode, PixelComponent, VisionError, VisionResult}; + +/// Built-in structuring-element geometry. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum MorphologyShape { + /// Every element in the bounding rectangle is active. + Rect, + /// The anchor row and column are active. + Cross, + /// A filled ellipse inscribed in the bounding rectangle. + Ellipse, + /// A filled Manhattan-distance diamond. + Diamond, +} + +/// A validated binary neighborhood mask and anchor. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct StructuringElement { + width: usize, + height: usize, + anchor_x: usize, + anchor_y: usize, + mask: Vec, +} + +impl StructuringElement { + /// Creates a built-in element anchored at its integer center. + pub fn try_new(shape: MorphologyShape, width: usize, height: usize) -> VisionResult { + Self::try_new_with_anchor(shape, width, height, width / 2, height / 2) + } + + /// Creates a built-in element with an explicit anchor. + pub fn try_new_with_anchor( + shape: MorphologyShape, + width: usize, + height: usize, + anchor_x: usize, + anchor_y: usize, + ) -> VisionResult { + validate_dimensions(width, height, anchor_x, anchor_y)?; + let mut mask = vec![false; width * height]; + match shape { + MorphologyShape::Rect => mask.fill(true), + MorphologyShape::Cross => { + for y in 0..height { + for x in 0..width { + mask[y * width + x] = x == anchor_x || y == anchor_y; + } + } + } + MorphologyShape::Ellipse => fill_ellipse(&mut mask, width, height), + MorphologyShape::Diamond => { + let rx = anchor_x.max(width - 1 - anchor_x).max(1); + let ry = anchor_y.max(height - 1 - anchor_y).max(1); + for y in 0..height { + for x in 0..width { + let dx = x.abs_diff(anchor_x) as f64 / rx as f64; + let dy = y.abs_diff(anchor_y) as f64 / ry as f64; + mask[y * width + x] = dx + dy <= 1.0 + f64::EPSILON; + } + } + } + } + Self::try_from_mask(width, height, anchor_x, anchor_y, mask) + } + + /// Creates an element from a row-major binary mask. + pub fn try_from_mask( + width: usize, + height: usize, + anchor_x: usize, + anchor_y: usize, + mask: Vec, + ) -> VisionResult { + validate_dimensions(width, height, anchor_x, anchor_y)?; + let expected = width + .checked_mul(height) + .ok_or_else(|| VisionError::InvalidParameter("structuring element overflows".into()))?; + if mask.len() != expected { + return Err(VisionError::ShapeMismatch(format!( + "structuring element needs {expected} mask values, found {}", + mask.len() + ))); + } + if !mask.iter().any(|&active| active) { + return Err(VisionError::InvalidParameter( + "structuring element must contain an active sample".into(), + )); + } + Ok(Self { width, height, anchor_x, anchor_y, mask }) + } + + /// Element width. + #[must_use] + pub const fn width(&self) -> usize { + self.width + } + + /// Element height. + #[must_use] + pub const fn height(&self) -> usize { + self.height + } + + /// Anchor coordinate `(x, y)`. + #[must_use] + pub const fn anchor(&self) -> (usize, usize) { + (self.anchor_x, self.anchor_y) + } + + /// Row-major binary mask. + #[must_use] + pub fn mask(&self) -> &[bool] { + &self.mask + } +} + +/// Composite morphology operation. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum MorphologyOperation { + /// Erosion followed by dilation. + Open, + /// Dilation followed by erosion. + Close, + /// Dilation minus erosion. + Gradient, + /// Input minus opening. + TopHat, + /// Closing minus input. + BlackHat, +} + +/// Erodes each channel independently. +pub fn erode( + input: ImageView<'_, T, CHANNELS>, + element: &StructuringElement, + iterations: usize, + border: BorderMode, +) -> VisionResult> { + repeat_extreme(input, element, iterations, border, Extreme::Minimum) +} + +/// Dilates each channel independently. +pub fn dilate( + input: ImageView<'_, T, CHANNELS>, + element: &StructuringElement, + iterations: usize, + border: BorderMode, +) -> VisionResult> { + repeat_extreme(input, element, iterations, border, Extreme::Maximum) +} + +/// Applies a composite morphology operation. +pub fn morphology_ex( + input: ImageView<'_, T, CHANNELS>, + operation: MorphologyOperation, + element: &StructuringElement, + iterations: usize, + border: BorderMode, +) -> VisionResult> { + let original = pack(input)?; + match operation { + MorphologyOperation::Open => { + let eroded = erode(input, element, iterations, border)?; + dilate(eroded.view(), element, iterations, border) + } + MorphologyOperation::Close => { + let dilated = dilate(input, element, iterations, border)?; + erode(dilated.view(), element, iterations, border) + } + MorphologyOperation::Gradient => { + let high = dilate(input, element, iterations, border)?; + let low = erode(input, element, iterations, border)?; + subtract(high.view(), low.view()) + } + MorphologyOperation::TopHat => { + let opened = + morphology_ex(input, MorphologyOperation::Open, element, iterations, border)?; + subtract(original.view(), opened.view()) + } + MorphologyOperation::BlackHat => { + let closed = + morphology_ex(input, MorphologyOperation::Close, element, iterations, border)?; + subtract(closed.view(), original.view()) + } + } +} + +#[derive(Clone, Copy)] +enum Extreme { + Minimum, + Maximum, +} + +fn repeat_extreme( + input: ImageView<'_, T, CHANNELS>, + element: &StructuringElement, + iterations: usize, + border: BorderMode, + extreme: Extreme, +) -> VisionResult> { + let mut output = pack(input)?; + for _ in 0..iterations { + output = extreme_once(output.view(), element, border, extreme)?; + } + Ok(output) +} + +fn extreme_once( + input: ImageView<'_, T, CHANNELS>, + element: &StructuringElement, + border: BorderMode, + extreme: Extreme, +) -> VisionResult> { + let mut output = Vec::with_capacity(input.width() * input.height() * CHANNELS); + for y in 0..input.height() { + for x in 0..input.width() { + let mut values = [0.0; CHANNELS]; + let mut initialized = false; + for ey in 0..element.height { + for ex in 0..element.width { + if !element.mask[ey * element.width + ex] { + continue; + } + let pixel = fetch( + input, + x as isize + ex as isize - element.anchor_x as isize, + y as isize + ey as isize - element.anchor_y as isize, + border, + ); + if !initialized { + values = pixel.map(PixelComponent::to_f64); + initialized = true; + } else { + for channel in 0..CHANNELS { + let candidate = pixel[channel].to_f64(); + let ordering = candidate.total_cmp(&values[channel]); + if matches!(extreme, Extreme::Minimum) && ordering.is_lt() + || matches!(extreme, Extreme::Maximum) && ordering.is_gt() + { + values[channel] = candidate; + } + } + } + } + } + output.extend(values.map(T::from_f64)); + } + } + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +fn subtract( + left: ImageView<'_, T, CHANNELS>, + right: ImageView<'_, T, CHANNELS>, +) -> VisionResult> { + if left.width() != right.width() || left.height() != right.height() { + return Err(VisionError::ShapeMismatch( + "morphology subtraction dimensions must match".into(), + )); + } + let mut output = Vec::with_capacity(left.width() * left.height() * CHANNELS); + for y in 0..left.height() { + for x in 0..left.width() { + let a = left.get(x, y).expect("left coordinate in bounds"); + let b = right.get(x, y).expect("right coordinate in bounds"); + output.extend(std::array::from_fn::<_, CHANNELS, _>(|channel| { + T::from_f64(a[channel].to_f64() - b[channel].to_f64()) + })); + } + } + Ok(Image::try_new_with_metadata(left.width(), left.height(), output, left.metadata())?) +} + +fn pack( + input: ImageView<'_, T, CHANNELS>, +) -> VisionResult> { + let mut output = Vec::with_capacity(input.width() * input.height() * CHANNELS); + for y in 0..input.height() { + for x in 0..input.width() { + output.extend_from_slice(input.get(x, y).expect("input coordinate in bounds")); + } + } + Ok(Image::try_new_with_metadata(input.width(), input.height(), output, input.metadata())?) +} + +fn validate_dimensions( + width: usize, + height: usize, + anchor_x: usize, + anchor_y: usize, +) -> VisionResult<()> { + if width == 0 || height == 0 { + return Err(VisionError::InvalidParameter( + "structuring element dimensions must be non-zero".into(), + )); + } + if anchor_x >= width || anchor_y >= height { + return Err(VisionError::InvalidParameter(format!( + "structuring element anchor ({anchor_x}, {anchor_y}) is outside {width}x{height}" + ))); + } + width + .checked_mul(height) + .ok_or_else(|| VisionError::InvalidParameter("structuring element overflows".into()))?; + Ok(()) +} + +fn fill_ellipse(mask: &mut [bool], width: usize, height: usize) { + let center_x = width / 2; + let center_y = height / 2; + let radius_y = center_y.max(1) as f64; + for y in 0..height { + let dy = y.abs_diff(center_y) as f64; + if dy > radius_y { + continue; + } + let extent = + (center_x as f64 * (1.0 - dy * dy / (radius_y * radius_y)).sqrt()).round() as usize; + let start = center_x.saturating_sub(extent); + let end = (center_x + extent).min(width - 1); + for x in start..=end { + mask[y * width + x] = true; + } + } +} + +#[cfg(test)] +mod tests { + use super::{ + dilate, erode, morphology_ex, MorphologyOperation, MorphologyShape, StructuringElement, + }; + use crate::BorderMode; + use spatialrust_image::{Image, ImageRegion}; + + #[test] + fn built_in_masks_have_expected_3x3_layouts() { + let rect = StructuringElement::try_new(MorphologyShape::Rect, 3, 3).unwrap(); + let cross = StructuringElement::try_new(MorphologyShape::Cross, 3, 3).unwrap(); + let ellipse = StructuringElement::try_new(MorphologyShape::Ellipse, 3, 3).unwrap(); + let diamond = StructuringElement::try_new(MorphologyShape::Diamond, 3, 3).unwrap(); + assert_eq!(rect.mask().iter().filter(|&&v| v).count(), 9); + assert_eq!(cross.mask().iter().filter(|&&v| v).count(), 5); + assert_eq!(ellipse.mask().iter().filter(|&&v| v).count(), 5); + assert_eq!(diamond.mask().iter().filter(|&&v| v).count(), 5); + } + + #[test] + fn erode_and_dilate_match_known_extrema_on_roi() { + let parent = Image::::try_new(5, 3, (0..15).collect()).unwrap(); + let roi = parent.view().subview(ImageRegion::new(1, 0, 3, 3)).unwrap(); + let element = StructuringElement::try_new(MorphologyShape::Rect, 3, 3).unwrap(); + let low = erode(roi, &element, 1, BorderMode::Replicate).unwrap(); + let high = dilate(roi, &element, 1, BorderMode::Replicate).unwrap(); + assert_eq!(low[(1, 1)][0], 1); + assert_eq!(high[(1, 1)][0], 13); + } + + #[test] + fn opening_removes_isolated_impulse() { + let image = Image::::try_new(3, 3, vec![0, 0, 0, 0, 100, 0, 0, 0, 0]).unwrap(); + let element = StructuringElement::try_new(MorphologyShape::Rect, 3, 3).unwrap(); + let output = morphology_ex( + image.view(), + MorphologyOperation::Open, + &element, + 1, + BorderMode::Constant([0]), + ) + .unwrap(); + assert!(output.as_slice().iter().all(|&value| value == 0)); + } + + #[test] + fn gradient_is_dilate_minus_erode_for_float() { + let image = Image::::try_new(3, 1, vec![1.0, 4.0, 9.0]).unwrap(); + let element = StructuringElement::try_new(MorphologyShape::Rect, 3, 1).unwrap(); + let output = morphology_ex( + image.view(), + MorphologyOperation::Gradient, + &element, + 1, + BorderMode::Replicate, + ) + .unwrap(); + assert_eq!(output.as_slice(), &[3.0, 8.0, 5.0]); + } + + #[test] + fn zero_iterations_return_packed_copy() { + let image = Image::::from_pixel(2, 2, [1, 2, 3]).unwrap(); + let element = StructuringElement::try_new(MorphologyShape::Rect, 3, 3).unwrap(); + assert_eq!(erode(image.view(), &element, 0, BorderMode::Replicate).unwrap(), image); + } + + #[test] + fn invalid_masks_are_rejected() { + assert!(StructuringElement::try_from_mask(2, 2, 0, 0, vec![false; 4]).is_err()); + assert!(StructuringElement::try_from_mask(2, 2, 0, 0, vec![true; 3]).is_err()); + } +} diff --git a/crates/spatialrust-vision/src/multiview.rs b/crates/spatialrust-vision/src/multiview.rs new file mode 100644 index 0000000..4b944f3 --- /dev/null +++ b/crates/spatialrust-vision/src/multiview.rs @@ -0,0 +1,779 @@ +//! Linear two-view model estimation and deterministic robust sampling. + +use spatialrust_math::{ + solve_linear_system, symmetric_eigen3, LeastSquaresResult, Mat3, Vec2, Vec3, +}; + +use crate::{ + CameraMatrix3, Essential3, Fundamental3, GeometricEstimate, Homography3, PointCorrespondence2, + RelativePose, RelativePoseEstimate, RobustEstimationOptions, TriangulatedPoint, VisionError, + VisionResult, +}; + +/// Estimates a homography from at least four correspondences using normalized DLT. +pub fn estimate_homography(correspondences: &[PointCorrespondence2]) -> VisionResult { + if correspondences.len() < 4 { + return Err(VisionError::InvalidParameter( + "homography estimation requires at least four correspondences".into(), + )); + } + let source = correspondences.iter().map(|pair| pair.source()).collect::>(); + let target = correspondences.iter().map(|pair| pair.target()).collect::>(); + let source_normalization = Normalization2::from_points(&source)?; + let target_normalization = Normalization2::from_points(&target)?; + let mut normal = vec![vec![0.0; 8]; 8]; + let mut rhs = vec![0.0; 8]; + for pair in correspondences { + let source = source_normalization.apply(pair.source()); + let target = target_normalization.apply(pair.target()); + accumulate_least_squares( + &mut normal, + &mut rhs, + &[source.x, source.y, 1.0, 0.0, 0.0, 0.0, -target.x * source.x, -target.x * source.y], + target.x, + ); + accumulate_least_squares( + &mut normal, + &mut rhs, + &[0.0, 0.0, 0.0, source.x, source.y, 1.0, -target.y * source.x, -target.y * source.y], + target.y, + ); + } + let LeastSquaresResult::Solved(solution) = solve_linear_system(normal, rhs) else { + return Err(VisionError::InvalidParameter( + "homography correspondences are degenerate".into(), + )); + }; + let normalized = Mat3::from_rows( + [solution[0], solution[1], solution[2]], + [solution[3], solution[4], solution[5]], + [solution[6], solution[7], 1.0], + ); + Homography3::try_new( + target_normalization.inverse.mul_mat3(normalized).mul_mat3(source_normalization.matrix), + ) +} + +/// Estimates a fundamental matrix from at least eight correspondences. +/// +/// Points are Hartley-normalized before the linear solve and the result is +/// projected to rank two before denormalization. +pub fn estimate_fundamental( + correspondences: &[PointCorrespondence2], +) -> VisionResult { + if correspondences.len() < 8 { + return Err(VisionError::InvalidParameter( + "fundamental estimation requires at least eight correspondences".into(), + )); + } + let source = correspondences.iter().map(|pair| pair.source()).collect::>(); + let target = correspondences.iter().map(|pair| pair.target()).collect::>(); + let source_normalization = Normalization2::from_points(&source)?; + let target_normalization = Normalization2::from_points(&target)?; + let mut normal = vec![vec![0.0; 9]; 9]; + for pair in correspondences { + let source = source_normalization.apply(pair.source()); + let target = target_normalization.apply(pair.target()); + let row = [ + target.x * source.x, + target.x * source.y, + target.x, + target.y * source.x, + target.y * source.y, + target.y, + source.x, + source.y, + 1.0, + ]; + for row_index in 0..9 { + for column in 0..9 { + normal[row_index][column] += row[row_index] * row[column]; + } + } + } + let vector = smallest_symmetric_eigenvector(normal).ok_or_else(|| { + VisionError::InvalidParameter("fundamental correspondences are degenerate".into()) + })?; + let normalized = enforce_rank_two(Mat3::from_rows( + [vector[0], vector[1], vector[2]], + [vector[3], vector[4], vector[5]], + [vector[6], vector[7], vector[8]], + )); + Fundamental3::try_new( + target_normalization + .matrix + .transpose() + .mul_mat3(normalized) + .mul_mat3(source_normalization.matrix), + ) +} + +/// Robustly estimates a homography using deterministic four-point RANSAC. +pub fn estimate_homography_ransac( + correspondences: &[PointCorrespondence2], + options: RobustEstimationOptions, +) -> VisionResult> { + robust_estimate(correspondences, options, 4, estimate_homography, homography_residual) +} + +/// Robustly estimates a fundamental matrix using deterministic eight-point RANSAC. +pub fn estimate_fundamental_ransac( + correspondences: &[PointCorrespondence2], + options: RobustEstimationOptions, +) -> VisionResult> { + robust_estimate(correspondences, options, 8, estimate_fundamental, fundamental_residual) +} + +/// Estimates an essential matrix after normalizing pixels with both cameras. +pub fn estimate_essential( + correspondences: &[PointCorrespondence2], + source_camera: CameraMatrix3, + target_camera: CameraMatrix3, +) -> VisionResult { + let normalized = normalize_correspondences(correspondences, source_camera, target_camera)?; + estimate_essential_normalized(&normalized) +} + +/// Robustly estimates an essential matrix in normalized-camera coordinates. +/// +/// The RANSAC threshold is therefore expressed on the normalized image plane, +/// not in pixels. +pub fn estimate_essential_ransac( + correspondences: &[PointCorrespondence2], + source_camera: CameraMatrix3, + target_camera: CameraMatrix3, + options: RobustEstimationOptions, +) -> VisionResult> { + let normalized = normalize_correspondences(correspondences, source_camera, target_camera)?; + robust_estimate(&normalized, options, 8, estimate_essential_normalized, essential_residual) +} + +/// Triangulates one calibrated correspondence for a known relative pose. +pub fn triangulate_correspondence( + correspondence: PointCorrespondence2, + source_camera: CameraMatrix3, + target_camera: CameraMatrix3, + pose: RelativePose, +) -> VisionResult { + let source = source_camera.normalize_pixel(correspondence.source()); + let target = target_camera.normalize_pixel(correspondence.target()); + triangulate_normalized( + Vec2 { x: source.x, y: source.y }, + Vec2 { x: target.x, y: target.y }, + pose, + ) + .ok_or_else(|| VisionError::InvalidParameter("triangulation is degenerate".into())) +} + +/// Recovers the essential-matrix pose with the most positive-depth points. +pub fn recover_relative_pose( + essential: Essential3, + correspondences: &[PointCorrespondence2], + source_camera: CameraMatrix3, + target_camera: CameraMatrix3, +) -> VisionResult { + if correspondences.is_empty() { + return Err(VisionError::InvalidParameter( + "pose recovery requires at least one correspondence".into(), + )); + } + let normalized = normalize_correspondences(correspondences, source_camera, target_camera)?; + let (mut left, mut right) = essential_singular_vectors(essential.matrix())?; + if determinant(left) < 0.0 { + negate_column(&mut left, 2); + } + if determinant(right) < 0.0 { + negate_column(&mut right, 2); + } + let w = Mat3::from_rows([0.0, -1.0, 0.0], [1.0, 0.0, 0.0], [0.0, 0.0, 1.0]); + let rotations = [ + proper_rotation(left.mul_mat3(w).mul_mat3(right.transpose())), + proper_rotation(left.mul_mat3(w.transpose()).mul_mat3(right.transpose())), + ]; + let translation = column(left, 2).normalize(); + let mut best: Option<(RelativePose, Vec>, usize, f64)> = None; + for rotation in rotations { + for translation in [translation, Vec3::new(-translation.x, -translation.y, -translation.z)] + { + let pose = RelativePose::try_new(rotation, translation)?; + let points = normalized + .iter() + .map(|pair| triangulate_normalized(pair.source(), pair.target(), pose)) + .collect::>(); + let count = points.iter().flatten().filter(|point| point.has_positive_depth()).count(); + let error = points + .iter() + .flatten() + .filter(|point| point.has_positive_depth()) + .map(|point| point.reprojection_error()) + .sum::(); + if best.as_ref().map_or(true, |candidate| { + count > candidate.2 || (count == candidate.2 && error < candidate.3) + }) { + best = Some((pose, points, count, error)); + } + } + } + let (pose, points, _, _) = best.expect("four essential-pose candidates"); + Ok(RelativePoseEstimate::new(pose, points)) +} + +fn normalize_correspondences( + correspondences: &[PointCorrespondence2], + source_camera: CameraMatrix3, + target_camera: CameraMatrix3, +) -> VisionResult> { + correspondences + .iter() + .map(|pair| { + let source = source_camera.normalize_pixel(pair.source()); + let target = target_camera.normalize_pixel(pair.target()); + PointCorrespondence2::try_new( + Vec2 { x: source.x, y: source.y }, + Vec2 { x: target.x, y: target.y }, + ) + }) + .collect() +} + +fn estimate_essential_normalized( + correspondences: &[PointCorrespondence2], +) -> VisionResult { + let fundamental = estimate_fundamental(correspondences)?; + Essential3::try_new(project_essential(fundamental.matrix())?) +} + +fn robust_estimate( + correspondences: &[PointCorrespondence2], + options: RobustEstimationOptions, + sample_size: usize, + estimate: fn(&[PointCorrespondence2]) -> VisionResult, + residual: fn(Model, PointCorrespondence2) -> f64, +) -> VisionResult> { + let options = options.validate()?; + if correspondences.len() < sample_size { + return Err(VisionError::InvalidParameter(format!( + "robust estimation requires at least {sample_size} correspondences" + ))); + } + let mut rng = XorShift64::new(options.seed); + let mut best: Option<(Model, Vec, Vec, usize, f64)> = None; + let mut iteration_limit = options.max_iterations; + let mut iteration = 0; + while iteration < iteration_limit { + let indices = sample_unique(&mut rng, correspondences.len(), sample_size); + let sample = indices.iter().map(|&index| correspondences[index]).collect::>(); + if let Ok(model) = estimate(&sample) { + let residuals = correspondences + .iter() + .copied() + .map(|pair| residual(model, pair)) + .collect::>(); + let inliers = + residuals.iter().map(|&value| value <= options.threshold).collect::>(); + let count = inliers.iter().filter(|&&value| value).count(); + let error = residuals + .iter() + .zip(&inliers) + .filter_map(|(value, &is_inlier)| is_inlier.then_some(*value)) + .sum::(); + let improves = best.as_ref().map_or(true, |candidate| { + count > candidate.3 || (count == candidate.3 && error < candidate.4) + }); + if improves { + if count >= sample_size { + let inlier_ratio = count as f64 / correspondences.len() as f64; + let success = inlier_ratio.powi(sample_size as i32).clamp(0.0, 1.0); + if success > 0.0 && success < 1.0 { + let required = ((1.0 - options.confidence).ln() / (1.0 - success).ln()) + .ceil() + .max(1.0) as usize; + iteration_limit = iteration_limit.min(required.max(iteration + 1)); + } else if success == 1.0 { + iteration_limit = iteration + 1; + } + } + best = Some((model, inliers, residuals, count, error)); + } + } + iteration += 1; + } + let (_, best_inliers, _, count, _) = best.ok_or_else(|| { + VisionError::InvalidParameter("robust geometry estimation found no valid model".into()) + })?; + if count < sample_size { + return Err(VisionError::InvalidParameter( + "robust geometry estimation found too few inliers".into(), + )); + } + let inlier_pairs = correspondences + .iter() + .zip(&best_inliers) + .filter_map(|(&pair, &is_inlier)| is_inlier.then_some(pair)) + .collect::>(); + let refined = estimate(&inlier_pairs)?; + let residuals = + correspondences.iter().copied().map(|pair| residual(refined, pair)).collect::>(); + let inliers = residuals.iter().map(|&value| value <= options.threshold).collect(); + GeometricEstimate::try_new(refined, correspondences.len(), inliers, residuals) +} + +fn homography_residual(model: Homography3, pair: PointCorrespondence2) -> f64 { + let projected = model.matrix().mul_vec3(Vec3::new(pair.source().x, pair.source().y, 1.0)); + if projected.z.abs() <= f64::EPSILON { + return f64::MAX; + } + let dx = projected.x / projected.z - pair.target().x; + let dy = projected.y / projected.z - pair.target().y; + dx.hypot(dy) +} + +fn fundamental_residual(model: Fundamental3, pair: PointCorrespondence2) -> f64 { + epipolar_residual(model.matrix(), pair) +} + +fn essential_residual(model: Essential3, pair: PointCorrespondence2) -> f64 { + epipolar_residual(model.matrix(), pair) +} + +fn epipolar_residual(matrix: Mat3, pair: PointCorrespondence2) -> f64 { + let source = Vec3::new(pair.source().x, pair.source().y, 1.0); + let target = Vec3::new(pair.target().x, pair.target().y, 1.0); + let line_target = matrix.mul_vec3(source); + let line_source = matrix.transpose().mul_vec3(target); + let numerator = target.dot(line_target).abs(); + let denominator = line_target.x * line_target.x + + line_target.y * line_target.y + + line_source.x * line_source.x + + line_source.y * line_source.y; + if denominator <= f64::EPSILON { + f64::MAX + } else { + numerator / denominator.sqrt() + } +} + +fn project_essential(matrix: Mat3) -> VisionResult> { + let (left, right) = essential_singular_vectors(matrix)?; + let covariance = matrix.transpose().mul_mat3(matrix); + let eigen = symmetric_eigen3(covariance); + let singular = [eigen.eigenvalues[2].max(0.0).sqrt(), eigen.eigenvalues[1].max(0.0).sqrt()]; + let scale = 0.5 * (singular[0] + singular[1]); + let mut result = Mat3::from_rows([0.0; 3], [0.0; 3], [0.0; 3]); + for component in 0..2 { + let left_column = column(left, component); + let right_column = column(right, component); + let left_values = [left_column.x, left_column.y, left_column.z]; + let right_values = [right_column.x, right_column.y, right_column.z]; + for (row, &left_value) in left_values.iter().enumerate() { + for (column, &right_value) in right_values.iter().enumerate() { + result.m[row][column] += scale * left_value * right_value; + } + } + } + Ok(result) +} + +fn essential_singular_vectors(matrix: Mat3) -> VisionResult<(Mat3, Mat3)> { + let eigen = symmetric_eigen3(matrix.transpose().mul_mat3(matrix)); + let right0 = column(eigen.eigenvectors, 2).normalize(); + let right1 = column(eigen.eigenvectors, 1).normalize(); + let right2 = right0.cross(right1).normalize(); + let sigma0 = eigen.eigenvalues[2].max(0.0).sqrt(); + let sigma1 = eigen.eigenvalues[1].max(0.0).sqrt(); + if sigma0 <= 1e-12 || sigma1 <= 1e-12 { + return Err(VisionError::InvalidParameter( + "essential matrix has fewer than two non-zero singular values".into(), + )); + } + let left0 = scale_vec(matrix.mul_vec3(right0), 1.0 / sigma0).normalize(); + let mut left1 = scale_vec(matrix.mul_vec3(right1), 1.0 / sigma1); + left1 = (left1 - scale_vec(left0, left0.dot(left1))).normalize(); + let left2 = left0.cross(left1).normalize(); + Ok((matrix_from_columns(left0, left1, left2), matrix_from_columns(right0, right1, right2))) +} + +fn triangulate_normalized( + source: Vec2, + target: Vec2, + pose: RelativePose, +) -> Option { + let rotation = pose.rotation(); + let translation = pose.translation(); + let rows = [ + vec![-1.0, 0.0, source.x, 0.0], + vec![0.0, -1.0, source.y, 0.0], + vec![ + target.x * rotation.m[2][0] - rotation.m[0][0], + target.x * rotation.m[2][1] - rotation.m[0][1], + target.x * rotation.m[2][2] - rotation.m[0][2], + target.x * translation.z - translation.x, + ], + vec![ + target.y * rotation.m[2][0] - rotation.m[1][0], + target.y * rotation.m[2][1] - rotation.m[1][1], + target.y * rotation.m[2][2] - rotation.m[1][2], + target.y * translation.z - translation.y, + ], + ]; + let mut normal = vec![vec![0.0; 4]; 4]; + for row in rows { + for first in 0..4 { + for second in 0..4 { + normal[first][second] += row[first] * row[second]; + } + } + } + let homogeneous = smallest_symmetric_eigenvector(normal)?; + if homogeneous[3].abs() <= 1e-12 { + return None; + } + let position = Vec3::new( + homogeneous[0] / homogeneous[3], + homogeneous[1] / homogeneous[3], + homogeneous[2] / homogeneous[3], + ); + let target_position = rotation.mul_vec3(position) + translation; + if position.z.abs() <= 1e-12 || target_position.z.abs() <= 1e-12 { + return None; + } + let source_error = + (position.x / position.z - source.x).hypot(position.y / position.z - source.y); + let target_error = (target_position.x / target_position.z - target.x) + .hypot(target_position.y / target_position.z - target.y); + TriangulatedPoint::try_new( + position, + position.z, + target_position.z, + 0.5 * (source_error + target_error), + ) + .ok() +} + +fn matrix_from_columns(first: Vec3, second: Vec3, third: Vec3) -> Mat3 { + Mat3::from_rows( + [first.x, second.x, third.x], + [first.y, second.y, third.y], + [first.z, second.z, third.z], + ) +} + +fn column(matrix: Mat3, index: usize) -> Vec3 { + Vec3::new(matrix.m[0][index], matrix.m[1][index], matrix.m[2][index]) +} + +fn scale_vec(vector: Vec3, scale: f64) -> Vec3 { + Vec3::new(vector.x * scale, vector.y * scale, vector.z * scale) +} + +fn determinant(matrix: Mat3) -> f64 { + matrix.m[0][0] * (matrix.m[1][1] * matrix.m[2][2] - matrix.m[1][2] * matrix.m[2][1]) + - matrix.m[0][1] * (matrix.m[1][0] * matrix.m[2][2] - matrix.m[1][2] * matrix.m[2][0]) + + matrix.m[0][2] * (matrix.m[1][0] * matrix.m[2][1] - matrix.m[1][1] * matrix.m[2][0]) +} + +fn negate_column(matrix: &mut Mat3, column: usize) { + for row in 0..3 { + matrix.m[row][column] = -matrix.m[row][column]; + } +} + +fn proper_rotation(mut matrix: Mat3) -> Mat3 { + if determinant(matrix) < 0.0 { + for row in &mut matrix.m { + for value in row { + *value = -*value; + } + } + } + matrix +} + +fn enforce_rank_two(matrix: Mat3) -> Mat3 { + let covariance = matrix.transpose().mul_mat3(matrix); + let eigen = symmetric_eigen3(covariance); + let vector = Vec3::new( + eigen.eigenvectors.m[0][0], + eigen.eigenvectors.m[1][0], + eigen.eigenvectors.m[2][0], + ); + let image = matrix.mul_vec3(vector); + let mut result = matrix; + for row in 0..3 { + for column in 0..3 { + result.m[row][column] -= + [image.x, image.y, image.z][row] * [vector.x, vector.y, vector.z][column]; + } + } + result +} + +fn accumulate_least_squares(normal: &mut [Vec], rhs: &mut [f64], row: &[f64], value: f64) { + for row_index in 0..row.len() { + rhs[row_index] += row[row_index] * value; + for column in 0..row.len() { + normal[row_index][column] += row[row_index] * row[column]; + } + } +} + +struct Normalization2 { + matrix: Mat3, + inverse: Mat3, +} + +impl Normalization2 { + fn from_points(points: &[Vec2]) -> VisionResult { + let count = points.len() as f64; + let center = Vec2 { + x: points.iter().map(|point| point.x).sum::() / count, + y: points.iter().map(|point| point.y).sum::() / count, + }; + let rms = (points + .iter() + .map(|point| { + let dx = point.x - center.x; + let dy = point.y - center.y; + dx * dx + dy * dy + }) + .sum::() + / count) + .sqrt(); + if !rms.is_finite() || rms <= f64::EPSILON { + return Err(VisionError::InvalidParameter( + "geometry points have zero spatial extent".into(), + )); + } + let scale = 2.0_f64.sqrt() / rms; + Ok(Self { + matrix: Mat3::from_rows( + [scale, 0.0, -scale * center.x], + [0.0, scale, -scale * center.y], + [0.0, 0.0, 1.0], + ), + inverse: Mat3::from_rows( + [1.0 / scale, 0.0, center.x], + [0.0, 1.0 / scale, center.y], + [0.0, 0.0, 1.0], + ), + }) + } + + fn apply(&self, point: Vec2) -> Vec2 { + let normalized = self.matrix.mul_vec3(Vec3::new(point.x, point.y, 1.0)); + Vec2 { x: normalized.x, y: normalized.y } + } +} + +#[allow(clippy::needless_range_loop)] +fn smallest_symmetric_eigenvector(mut matrix: Vec>) -> Option> { + let size = matrix.len(); + if size == 0 || matrix.iter().any(|row| row.len() != size) { + return None; + } + let mut vectors = vec![vec![0.0; size]; size]; + for (index, row) in vectors.iter_mut().enumerate() { + row[index] = 1.0; + } + for _ in 0..size * size * 32 { + let mut pivot = (0, 1); + let mut maximum = 0.0_f64; + for (row, values) in matrix.iter().enumerate() { + for (column, &value) in values.iter().enumerate().skip(row + 1) { + if value.abs() > maximum { + maximum = value.abs(); + pivot = (row, column); + } + } + } + if maximum < 1e-12 { + break; + } + let (p, q) = pivot; + let angle = 0.5 * (2.0 * matrix[p][q]).atan2(matrix[q][q] - matrix[p][p]); + let (sine, cosine) = angle.sin_cos(); + for row in 0..size { + if row != p && row != q { + let rp = matrix[row][p]; + let rq = matrix[row][q]; + matrix[row][p] = cosine * rp - sine * rq; + matrix[p][row] = matrix[row][p]; + matrix[row][q] = sine * rp + cosine * rq; + matrix[q][row] = matrix[row][q]; + } + } + let pp = matrix[p][p]; + let qq = matrix[q][q]; + let pq = matrix[p][q]; + matrix[p][p] = cosine * cosine * pp - 2.0 * sine * cosine * pq + sine * sine * qq; + matrix[q][q] = sine * sine * pp + 2.0 * sine * cosine * pq + cosine * cosine * qq; + matrix[p][q] = 0.0; + matrix[q][p] = 0.0; + for row in &mut vectors { + let rp = row[p]; + let rq = row[q]; + row[p] = cosine * rp - sine * rq; + row[q] = sine * rp + cosine * rq; + } + } + let index = + (0..size).min_by(|&left, &right| matrix[left][left].total_cmp(&matrix[right][right]))?; + let mut vector = vectors.iter().map(|row| row[index]).collect::>(); + let norm = vector.iter().map(|value| value * value).sum::().sqrt(); + if !norm.is_finite() || norm <= f64::EPSILON { + return None; + } + for value in &mut vector { + *value /= norm; + } + Some(vector) +} + +struct XorShift64(u64); + +impl XorShift64 { + fn new(seed: u64) -> Self { + Self(if seed == 0 { 0x9e37_79b9_7f4a_7c15 } else { seed }) + } + + fn next(&mut self) -> u64 { + self.0 ^= self.0 << 13; + self.0 ^= self.0 >> 7; + self.0 ^= self.0 << 17; + self.0 + } +} + +fn sample_unique(rng: &mut XorShift64, population: usize, count: usize) -> Vec { + let mut sample = Vec::with_capacity(count); + while sample.len() < count { + let candidate = (rng.next() % population as u64) as usize; + if !sample.contains(&candidate) { + sample.push(candidate); + } + } + sample +} + +#[cfg(test)] +mod tests { + use super::{ + estimate_essential, estimate_fundamental, estimate_homography, estimate_homography_ransac, + fundamental_residual, homography_residual, recover_relative_pose, + triangulate_correspondence, + }; + use crate::{ + CameraMatrix3, Essential3, PointCorrespondence2, RelativePose, RobustEstimationOptions, + }; + use spatialrust_camera::CameraIntrinsics; + use spatialrust_math::{Mat3, Vec2, Vec3}; + + fn correspondence(source: (f64, f64), target: (f64, f64)) -> PointCorrespondence2 { + PointCorrespondence2::try_new( + Vec2 { x: source.0, y: source.1 }, + Vec2 { x: target.0, y: target.1 }, + ) + .unwrap() + } + + #[test] + fn homography_recovers_known_projective_mapping() { + let pairs = [ + ((0.0, 0.0), (3.0, -2.0)), + ((10.0, 0.0), (23.0, -2.0)), + ((0.0, 5.0), (3.0, 13.0)), + ((10.0, 5.0), (23.0, 13.0)), + ((4.0, 2.0), (11.0, 4.0)), + ] + .map(|(source, target)| correspondence(source, target)); + let model = estimate_homography(&pairs).unwrap(); + assert!(pairs.iter().all(|&pair| homography_residual(model, pair) < 1e-9)); + } + + #[test] + fn homography_ransac_rejects_large_outliers_deterministically() { + let mut pairs = (0..20) + .map(|index| { + let x = f64::from(index % 5) * 8.0; + let y = f64::from(index / 5) * 7.0; + correspondence((x, y), (1.5 * x + 4.0, 0.75 * y - 3.0)) + }) + .collect::>(); + pairs.push(correspondence((5.0, 5.0), (500.0, -300.0))); + pairs.push(correspondence((9.0, 11.0), (-200.0, 400.0))); + let options = RobustEstimationOptions { threshold: 0.1, seed: 7, ..Default::default() }; + let first = estimate_homography_ransac(&pairs, options).unwrap(); + let second = estimate_homography_ransac(&pairs, options).unwrap(); + assert_eq!(first, second); + assert_eq!(first.inlier_count(), 20); + } + + #[test] + fn eight_point_model_satisfies_synthetic_epipolar_constraints() { + let pairs = (0..16) + .map(|index| { + let x = f64::from(index % 4) * 0.3 - 0.4; + let y = f64::from(index / 4) * 0.2 - 0.3; + let depth = 2.0 + f64::from(index) * 0.05; + let source = (x / depth, y / depth); + let target = ((x + 0.2) / depth, y / depth); + correspondence(source, target) + }) + .collect::>(); + let model = estimate_fundamental(&pairs).unwrap(); + assert!(pairs.iter().all(|&pair| fundamental_residual(model, pair) < 1e-7)); + } + + #[test] + fn calibrated_triangulation_and_pose_recovery_choose_positive_depth() { + let intrinsics = CameraIntrinsics::try_new(500.0, 500.0, 320.0, 240.0, 640, 480).unwrap(); + let camera = CameraMatrix3::from_intrinsics(intrinsics); + let known_pose = + RelativePose::try_new(Mat3::::identity(), Vec3::new(0.2, 0.0, 0.0)).unwrap(); + let points = (0..16) + .map(|index| { + Vec3::new( + f64::from(index % 4) * 0.25 - 0.4, + f64::from(index / 4) * 0.2 - 0.3, + 2.0 + f64::from((index * index + 3 * index) % 17) * 0.07, + ) + }) + .collect::>(); + let pairs = points + .iter() + .map(|point| { + let target = *point + known_pose.translation(); + correspondence( + (500.0 * point.x / point.z + 320.0, 500.0 * point.y / point.z + 240.0), + (500.0 * target.x / target.z + 320.0, 500.0 * target.y / target.z + 240.0), + ) + }) + .collect::>(); + let triangulated = + triangulate_correspondence(pairs[3], camera, camera, known_pose).unwrap(); + assert!((triangulated.position().x - points[3].x).abs() < 1e-8); + assert!((triangulated.position().z - points[3].z).abs() < 1e-8); + assert!(triangulated.has_positive_depth()); + + let estimated = estimate_essential(&pairs, camera, camera).unwrap(); + let maximum_residual = pairs + .iter() + .map(|pair| { + let source = camera.normalize_pixel(pair.source()); + let target = camera.normalize_pixel(pair.target()); + let normalized = correspondence((source.x, source.y), (target.x, target.y)); + super::essential_residual(estimated, normalized) + }) + .fold(0.0_f64, f64::max); + assert!(maximum_residual < 1e-7, "maximum residual {maximum_residual}"); + let essential = Essential3::try_new(Mat3::from_rows( + [0.0, 0.0, 0.0], + [0.0, 0.0, -0.2], + [0.0, 0.2, 0.0], + )) + .unwrap(); + let recovered = recover_relative_pose(essential, &pairs, camera, camera).unwrap(); + assert_eq!(recovered.positive_depth_count(), pairs.len()); + assert!(recovered.pose().translation().x.abs() > 0.99); + } +} diff --git a/crates/spatialrust-vision/src/optical_flow.rs b/crates/spatialrust-vision/src/optical_flow.rs new file mode 100644 index 0000000..70cd9ff --- /dev/null +++ b/crates/spatialrust-vision/src/optical_flow.rs @@ -0,0 +1,361 @@ +//! Sparse pyramidal Lucas–Kanade point tracking. + +use spatialrust_image::{Image, ImageView}; +use spatialrust_math::Vec2; + +use crate::{PixelComponent, VisionError, VisionResult}; + +/// Configuration for sparse Lucas–Kanade tracking. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct LucasKanadeOptions { + /// Half-window radius in pixels at each pyramid level. + pub window_radius: usize, + /// Number of pyramid levels including the full-resolution level. + pub pyramid_levels: usize, + /// Maximum Gauss–Newton iterations per pyramid level. + pub max_iterations: usize, + /// Convergence threshold on the update length in pixels. + pub epsilon: f64, + /// Minimum structure-tensor determinant accepted as trackable. + pub min_eigenvalue: f64, +} + +impl Default for LucasKanadeOptions { + fn default() -> Self { + Self { + window_radius: 5, + pyramid_levels: 3, + max_iterations: 30, + epsilon: 1e-3, + min_eigenvalue: 1e-4, + } + } +} + +impl LucasKanadeOptions { + /// Validates window, pyramid, and numerical thresholds. + pub fn validate(self) -> VisionResult { + if self.window_radius == 0 { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade window_radius must be positive".into(), + )); + } + if self.pyramid_levels == 0 { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade pyramid_levels must be positive".into(), + )); + } + if self.max_iterations == 0 { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade max_iterations must be positive".into(), + )); + } + if !self.epsilon.is_finite() || self.epsilon <= 0.0 { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade epsilon must be finite and positive".into(), + )); + } + if !self.min_eigenvalue.is_finite() || self.min_eigenvalue <= 0.0 { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade min_eigenvalue must be finite and positive".into(), + )); + } + Ok(self) + } +} + +/// Result of sparse point tracking between two frames. +#[derive(Clone, Debug, PartialEq)] +pub struct TrackedPoints { + next_points: Vec>, + status: Vec, +} + +impl TrackedPoints { + /// Returns one next-frame coordinate per input point. + pub fn next_points(&self) -> &[Vec2] { + &self.next_points + } + + /// Returns whether each point was successfully tracked. + pub fn status(&self) -> &[bool] { + &self.status + } +} + +/// Tracks sparse points from `previous` to `next` with pyramidal Lucas–Kanade. +pub fn track_points_lucas_kanade( + previous: ImageView<'_, T, 1>, + next: ImageView<'_, T, 1>, + points: &[Vec2], + options: LucasKanadeOptions, +) -> VisionResult { + let options = options.validate()?; + if previous.width() != next.width() || previous.height() != next.height() { + return Err(VisionError::ShapeMismatch( + "Lucas–Kanade frames must share width and height".into(), + )); + } + if previous.width() < 3 || previous.height() < 3 { + return Err(VisionError::InvalidDimensions( + "Lucas–Kanade requires at least 3x3 images".into(), + )); + } + let previous_pyramid = build_pyramid(previous, options.pyramid_levels)?; + let next_pyramid = build_pyramid(next, options.pyramid_levels)?; + let mut next_points = points.to_vec(); + let mut status = vec![true; points.len()]; + let scale = 1.0 / f64::from(1u32 << (options.pyramid_levels.saturating_sub(1) as u32)); + for point in &mut next_points { + point.x *= scale; + point.y *= scale; + } + let mut previous_guess = next_points.clone(); + for level in (0..options.pyramid_levels).rev() { + let level_scale = 1.0 / f64::from(1u32 << (level as u32)); + for (index, previous_point) in points.iter().enumerate() { + if !status[index] { + continue; + } + let mut guess = previous_guess[index]; + let target = Vec2 { + x: previous_point.x * level_scale, + y: previous_point.y * level_scale, + }; + match track_one_level( + previous_pyramid[level].view(), + next_pyramid[level].view(), + target, + guess, + options, + ) { + Ok(updated) => { + guess = updated; + status[index] = true; + } + Err(_) => status[index] = false, + } + previous_guess[index] = guess; + if level > 0 { + previous_guess[index].x *= 2.0; + previous_guess[index].y *= 2.0; + } + } + } + Ok(TrackedPoints { next_points: previous_guess, status }) +} + +fn track_one_level( + previous: ImageView<'_, f32, 1>, + next: ImageView<'_, f32, 1>, + previous_point: Vec2, + mut next_point: Vec2, + options: LucasKanadeOptions, +) -> VisionResult> { + let radius = options.window_radius as f64; + for _ in 0..options.max_iterations { + let mut gxx = 0.0; + let mut gxy = 0.0; + let mut gyy = 0.0; + let mut bx = 0.0; + let mut by = 0.0; + let mut samples = 0usize; + let start = -(options.window_radius as isize); + let end = options.window_radius as isize; + for dy in start..=end { + for dx in start..=end { + let px = previous_point.x + dx as f64; + let py = previous_point.y + dy as f64; + let qx = next_point.x + dx as f64; + let qy = next_point.y + dy as f64; + if !(in_bounds(previous, px, py, radius) && in_bounds(next, qx, qy, radius)) { + continue; + } + let ix = 0.5 + * (sample(previous, px + 1.0, py)? - sample(previous, px - 1.0, py)?); + let iy = 0.5 + * (sample(previous, px, py + 1.0)? - sample(previous, px, py - 1.0)?); + let it = sample(next, qx, qy)? - sample(previous, px, py)?; + gxx += ix * ix; + gxy += ix * iy; + gyy += iy * iy; + bx += ix * it; + by += iy * it; + samples += 1; + } + } + if samples < 4 { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade patch left the image".into(), + )); + } + let det = gxx * gyy - gxy * gxy; + if det.abs() < options.min_eigenvalue { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade structure tensor is singular".into(), + )); + } + let dx = (-gyy * bx + gxy * by) / det; + let dy = (gxy * bx - gxx * by) / det; + next_point.x += dx; + next_point.y += dy; + if dx.hypot(dy) < options.epsilon { + break; + } + } + if !in_bounds(next, next_point.x, next_point.y, 0.0) { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade tracked point left the image".into(), + )); + } + Ok(next_point) +} + +fn build_pyramid( + image: ImageView<'_, T, 1>, + levels: usize, +) -> VisionResult>> { + let mut pyramid = Vec::with_capacity(levels); + pyramid.push(to_f32_image(image)?); + for _ in 1..levels { + let previous = pyramid.last().expect("pyramid starts non-empty"); + pyramid.push(downsample(previous.view())?); + } + Ok(pyramid) +} + +fn to_f32_image(image: ImageView<'_, T, 1>) -> VisionResult> { + let mut data = Vec::with_capacity(image.width() * image.height()); + for y in 0..image.height() { + for x in 0..image.width() { + data.push(image.get(x, y).expect("in-bounds")[0].to_f64() as f32); + } + } + Ok(Image::try_new(image.width(), image.height(), data)?) +} + +fn downsample(image: ImageView<'_, f32, 1>) -> VisionResult> { + let width = (image.width() / 2).max(1); + let height = (image.height() / 2).max(1); + let mut data = Vec::with_capacity(width * height); + for y in 0..height { + for x in 0..width { + let sx = x * 2; + let sy = y * 2; + let mut sum = 0.0_f64; + let mut count = 0.0_f64; + for dy in 0..2 { + for dx in 0..2 { + if let Some(pixel) = image.get(sx + dx, sy + dy) { + sum += f64::from(pixel[0]); + count += 1.0; + } + } + } + data.push((sum / count.max(1.0)) as f32); + } + } + Ok(Image::try_new(width, height, data)?) +} + +fn sample(image: ImageView<'_, f32, 1>, x: f64, y: f64) -> VisionResult { + if !x.is_finite() || !y.is_finite() { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade sample coordinates must be finite".into(), + )); + } + let x0 = x.floor() as isize; + let y0 = y.floor() as isize; + let x1 = x0 + 1; + let y1 = y0 + 1; + if x0 < 0 + || y0 < 0 + || x1 >= image.width() as isize + || y1 >= image.height() as isize + { + return Err(VisionError::InvalidParameter( + "Lucas–Kanade sample is out of bounds".into(), + )); + } + let ax = x - x0 as f64; + let ay = y - y0 as f64; + let i00 = f64::from(image.get(x0 as usize, y0 as usize).expect("in-bounds")[0]); + let i10 = f64::from(image.get(x1 as usize, y0 as usize).expect("in-bounds")[0]); + let i01 = f64::from(image.get(x0 as usize, y1 as usize).expect("in-bounds")[0]); + let i11 = f64::from(image.get(x1 as usize, y1 as usize).expect("in-bounds")[0]); + Ok((1.0 - ax) * (1.0 - ay) * i00 + + ax * (1.0 - ay) * i10 + + (1.0 - ax) * ay * i01 + + ax * ay * i11) +} + +fn in_bounds(image: ImageView<'_, f32, 1>, x: f64, y: f64, margin: f64) -> bool { + x >= margin + && y >= margin + && x < image.width() as f64 - 1.0 - margin + && y < image.height() as f64 - 1.0 - margin +} + +#[cfg(test)] +mod tests { + use super::{track_points_lucas_kanade, LucasKanadeOptions}; + use spatialrust_image::Image; + use spatialrust_math::Vec2; + + #[test] + fn tracks_integer_translation() { + let width = 64; + let height = 48; + let mut previous = vec![0u8; width * height]; + for y in 0..height { + for x in 0..width { + previous[y * width + x] = ((x * 13 + y * 7) % 200 + 20) as u8; + } + } + let shift = 3isize; + let mut next = vec![0u8; width * height]; + for y in 0..height { + for x in 0..width { + let sx = x as isize - shift; + let sy = y as isize; + if (0..width as isize).contains(&sx) && (0..height as isize).contains(&sy) { + next[y * width + x] = previous[sy as usize * width + sx as usize]; + } + } + } + let previous = Image::::try_new(width, height, previous).unwrap(); + let next = Image::::try_new(width, height, next).unwrap(); + let points = [Vec2 { x: 24.0, y: 20.0 }]; + let tracked = track_points_lucas_kanade( + previous.view(), + next.view(), + &points, + LucasKanadeOptions { + window_radius: 4, + pyramid_levels: 1, + max_iterations: 50, + epsilon: 1e-4, + min_eigenvalue: 1e-8, + }, + ) + .unwrap(); + assert!( + tracked.status().iter().all(|&ok| ok), + "status={:?}", + tracked.status() + ); + let actual = tracked.next_points()[0]; + assert!( + (actual.x - (points[0].x + shift as f64)).abs() < 1.5, + "x actual={} expected={}", + actual.x, + points[0].x + shift as f64 + ); + assert!( + (actual.y - points[0].y).abs() < 1.5, + "y actual={} expected={}", + actual.y, + points[0].y + ); + } +} diff --git a/crates/spatialrust-vision/src/orb.rs b/crates/spatialrust-vision/src/orb.rs new file mode 100644 index 0000000..26e44d5 --- /dev/null +++ b/crates/spatialrust-vision/src/orb.rs @@ -0,0 +1,346 @@ +//! Oriented FAST and rotated BRIEF binary features. + +use spatialrust_image::{Image, ImageView}; + +use crate::{ + detect_fast, gaussian_blur, resize, BorderMode, DescriptorBuffer, FastOptions, FeatureSet2, + Interpolation, Keypoint2, VisionError, VisionResult, +}; + +const ORB_DESCRIPTOR_BYTES: usize = 32; + +/// Response used to rank ORB keypoints. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub enum OrbScoreType { + /// Rank FAST candidates by a local Harris response. + #[default] + Harris, + /// Retain the FAST segment-test score. + Fast, +} + +/// Multi-scale ORB detector and descriptor configuration. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct OrbOptions { + /// Maximum number of descriptor rows across every pyramid level. + pub max_features: usize, + /// Scale multiplier between adjacent levels; must be greater than one. + pub scale_factor: f32, + /// Number of pyramid levels. + pub levels: usize, + /// Minimum distance from a level-image edge. + pub edge_threshold: usize, + /// FAST intensity threshold. + pub fast_threshold: u8, + /// Odd intensity-centroid and descriptor patch diameter. + pub patch_size: usize, + /// Candidate ranking response. + pub score_type: OrbScoreType, +} + +impl Default for OrbOptions { + fn default() -> Self { + Self { + max_features: 500, + scale_factor: 1.2, + levels: 8, + edge_threshold: 31, + fast_threshold: 20, + patch_size: 31, + score_type: OrbScoreType::Harris, + } + } +} + +impl OrbOptions { + fn validate(self) -> VisionResult { + if !self.scale_factor.is_finite() || self.scale_factor <= 1.0 { + return Err(VisionError::InvalidParameter( + "ORB scale_factor must be finite and greater than one".into(), + )); + } + if self.levels == 0 { + return Err(VisionError::InvalidParameter("ORB levels must be positive".into())); + } + if self.patch_size < 7 || self.patch_size % 2 == 0 { + return Err(VisionError::InvalidParameter( + "ORB patch_size must be odd and at least seven".into(), + )); + } + Ok(self) + } +} + +#[derive(Clone, Copy)] +struct Candidate { + level: usize, + level_x: usize, + level_y: usize, + scale: f32, + response: f32, +} + +/// Detects oriented multi-scale keypoints and computes 256-bit rotated BRIEF descriptors. +/// +/// The fixed-seed BRIEF sampling pattern is stable across platforms and SpatialRust +/// releases. It intentionally does not promise bit identity with OpenCV's private +/// learned sampling table. +pub fn detect_and_describe_orb( + input: ImageView<'_, u8, 1>, + options: OrbOptions, +) -> VisionResult { + let options = options.validate()?; + if input.width() == 0 || input.height() == 0 || options.max_features == 0 { + return FeatureSet2::try_new( + Vec::new(), + DescriptorBuffer::try_binary(0, ORB_DESCRIPTOR_BYTES, Vec::new())?, + ); + } + + let mut pyramid = Vec::with_capacity(options.levels); + let mut candidates = Vec::new(); + for level in 0..options.levels { + let scale = options.scale_factor.powi(level as i32); + let width = ((input.width() as f32 / scale).round() as usize).max(1); + let height = ((input.height() as f32 / scale).round() as usize).max(1); + let image = if level == 0 { + Image::try_new_with_metadata( + input.width(), + input.height(), + (0..input.height()) + .flat_map(|y| { + (0..input.width()) + .map(move |x| input.get(x, y).expect("coordinate in bounds")[0]) + }) + .collect(), + input.metadata(), + )? + } else { + resize(input, width, height, Interpolation::Bilinear)? + }; + let margin = options.edge_threshold.max(options.patch_size / 2 + 1); + let fast = detect_fast( + image.view(), + FastOptions { threshold: options.fast_threshold, nonmax_suppression: true }, + )?; + for point in fast { + let x = point.x() as usize; + let y = point.y() as usize; + if x < margin + || y < margin + || x.saturating_add(margin) >= image.width() + || y.saturating_add(margin) >= image.height() + { + continue; + } + let response = match options.score_type { + OrbScoreType::Harris => harris_score(image.view(), x, y), + OrbScoreType::Fast => point.response(), + }; + candidates.push(Candidate { level, level_x: x, level_y: y, scale, response }); + } + pyramid.push(image); + } + + candidates.sort_by(|left, right| { + right + .response + .total_cmp(&left.response) + .then_with(|| left.level.cmp(&right.level)) + .then_with(|| left.level_y.cmp(&right.level_y)) + .then_with(|| left.level_x.cmp(&right.level_x)) + }); + candidates.truncate(options.max_features); + + let pattern = brief_pattern(options.patch_size / 2); + let mut blurred = Vec::with_capacity(pyramid.len()); + for image in &pyramid { + blurred.push(gaussian_blur(image.view(), 7, 7, 2.0, 2.0, BorderMode::Reflect101)?); + } + let mut keypoints = Vec::with_capacity(candidates.len()); + let mut descriptors = Vec::with_capacity(candidates.len() * ORB_DESCRIPTOR_BYTES); + for candidate in candidates { + let image = blurred[candidate.level].view(); + let angle = intensity_centroid_angle( + image, + candidate.level_x, + candidate.level_y, + options.patch_size / 2, + ); + keypoints.push( + Keypoint2::try_new( + candidate.level_x as f32 * candidate.scale, + candidate.level_y as f32 * candidate.scale, + candidate.response, + )? + .with_size(options.patch_size as f32 * candidate.scale)? + .with_angle_degrees(angle.to_degrees())? + .with_octave(candidate.level as i32), + ); + descriptors.extend(describe(image, candidate.level_x, candidate.level_y, angle, &pattern)); + } + FeatureSet2::try_new( + keypoints, + DescriptorBuffer::try_binary( + descriptors.len() / ORB_DESCRIPTOR_BYTES, + ORB_DESCRIPTOR_BYTES, + descriptors, + )?, + ) +} + +fn harris_score(image: ImageView<'_, u8, 1>, x: usize, y: usize) -> f32 { + let mut xx = 0.0_f32; + let mut xy = 0.0_f32; + let mut yy = 0.0_f32; + for dy in -3_isize..=3 { + for dx in -3_isize..=3 { + let px = (x as isize + dx) as usize; + let py = (y as isize + dy) as usize; + let gx = f32::from(image.get(px + 1, py).unwrap()[0]) + - f32::from(image.get(px - 1, py).unwrap()[0]); + let gy = f32::from(image.get(px, py + 1).unwrap()[0]) + - f32::from(image.get(px, py - 1).unwrap()[0]); + xx += gx * gx; + xy += gx * gy; + yy += gy * gy; + } + } + xx.mul_add(yy, -(xy * xy)) - 0.04 * (xx + yy) * (xx + yy) +} + +fn intensity_centroid_angle(image: ImageView<'_, u8, 1>, x: usize, y: usize, radius: usize) -> f32 { + let mut m10 = 0_i64; + let mut m01 = 0_i64; + let radius_squared = (radius * radius) as isize; + for dy in -(radius as isize)..=radius as isize { + let extent = ((radius_squared - dy * dy) as f64).sqrt().floor() as isize; + for dx in -extent..=extent { + let intensity = i64::from( + image.get((x as isize + dx) as usize, (y as isize + dy) as usize).unwrap()[0], + ); + m10 += dx as i64 * intensity; + m01 += dy as i64 * intensity; + } + } + (m01 as f32).atan2(m10 as f32) +} + +fn describe( + image: ImageView<'_, u8, 1>, + x: usize, + y: usize, + angle: f32, + pattern: &[((i8, i8), (i8, i8))], +) -> [u8; ORB_DESCRIPTOR_BYTES] { + let (sin, cos) = angle.sin_cos(); + let mut descriptor = [0_u8; ORB_DESCRIPTOR_BYTES]; + for (bit, &(first, second)) in pattern.iter().enumerate() { + let sample = |point: (i8, i8)| { + let rx = (f32::from(point.0) * cos - f32::from(point.1) * sin).round() as isize; + let ry = (f32::from(point.0) * sin + f32::from(point.1) * cos).round() as isize; + image.get((x as isize + rx) as usize, (y as isize + ry) as usize).unwrap()[0] + }; + if sample(first) < sample(second) { + descriptor[bit / 8] |= 1 << (bit % 8); + } + } + descriptor +} + +fn brief_pattern(radius: usize) -> Vec<((i8, i8), (i8, i8))> { + let radius = radius.min(i8::MAX as usize) as i32; + let mut state = 0x6d2b_79f5_u32; + let mut next_point = || loop { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + let span = (radius * 2 + 1) as u32; + let x = (state % span) as i32 - radius; + state = state.rotate_left(11).wrapping_mul(0x9e37_79b1); + let y = (state % span) as i32 - radius; + if x * x + y * y <= radius * radius { + return (x as i8, y as i8); + } + }; + (0..ORB_DESCRIPTOR_BYTES * 8) + .map(|_| { + let first = next_point(); + let mut second = next_point(); + while second == first { + second = next_point(); + } + (first, second) + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::{detect_and_describe_orb, OrbOptions}; + use crate::{match_descriptors, MatchOptions}; + use spatialrust_image::Image; + + fn textured_image() -> Image { + Image::try_new( + 96, + 80, + (0..80) + .flat_map(|y| { + (0..96).map(move |x| { + (((x * 37 + y * 19) ^ (x * y * 3) ^ ((x / 8 + y / 8) * 127)) & 255) as u8 + }) + }) + .collect(), + ) + .unwrap() + } + + #[test] + fn orb_is_deterministic_bounded_and_well_formed() { + let image = textured_image(); + let options = OrbOptions { max_features: 80, edge_threshold: 16, ..OrbOptions::default() }; + let first = detect_and_describe_orb(image.view(), options).unwrap(); + let second = detect_and_describe_orb(image.view(), options).unwrap(); + assert_eq!(first, second); + assert!(!first.keypoints().is_empty()); + assert!(first.keypoints().len() <= 80); + assert_eq!(first.descriptors().width(), 32); + assert!(first.keypoints().iter().all(|point| point.angle_degrees().is_some())); + } + + #[test] + fn orb_descriptors_self_match_at_zero_distance() { + let image = textured_image(); + let features = detect_and_describe_orb( + image.view(), + OrbOptions { max_features: 40, edge_threshold: 16, ..OrbOptions::default() }, + ) + .unwrap(); + let matches = match_descriptors( + features.descriptors(), + features.descriptors(), + MatchOptions { cross_check: true, ..MatchOptions::default() }, + ) + .unwrap(); + assert_eq!(matches.len(), features.keypoints().len()); + assert!(matches.iter().all(|feature_match| feature_match.distance() == 0.0)); + } + + #[test] + fn invalid_or_empty_orb_inputs_are_handled() { + let image = textured_image(); + assert!(detect_and_describe_orb( + image.view(), + OrbOptions { levels: 0, ..OrbOptions::default() } + ) + .is_err()); + let empty = detect_and_describe_orb( + image.view(), + OrbOptions { max_features: 0, ..OrbOptions::default() }, + ) + .unwrap(); + assert!(empty.keypoints().is_empty()); + assert_eq!(empty.descriptors().width(), 32); + } +} diff --git a/crates/spatialrust-vision/src/pnp.rs b/crates/spatialrust-vision/src/pnp.rs new file mode 100644 index 0000000..f3c2397 --- /dev/null +++ b/crates/spatialrust-vision/src/pnp.rs @@ -0,0 +1,591 @@ +//! Absolute pose from 3D–2D correspondences (PnP) with deterministic RANSAC. + +use spatialrust_math::{ + solve_linear_system, symmetric_eigen3, LeastSquaresResult, Mat3, Vec2, Vec3, +}; + +use crate::{ + AbsolutePose, CameraMatrix3, GeometricEstimate, ObjectImageCorrespondence, + RobustEstimationOptions, VisionError, VisionResult, +}; + +/// Estimates an object-to-camera pose from at least four correspondences. +/// +/// Uses a calibrated DLT initialization followed by Gauss–Newton refinement on +/// the SE(3) tangent space. Four points are accepted for the final refine path +/// when an initial pose is recoverable; RANSAC minimal samples use six points. +pub fn solve_pnp( + correspondences: &[ObjectImageCorrespondence], + camera: CameraMatrix3, +) -> VisionResult { + if correspondences.len() < 4 { + return Err(VisionError::InvalidParameter( + "PnP requires at least four object-image correspondences".into(), + )); + } + let initial = estimate_pnp_dlt(correspondences, camera)?; + refine_pnp(correspondences, camera, initial) +} + +/// Robust PnP using deterministic six-point RANSAC and inlier refinement. +pub fn solve_pnp_ransac( + correspondences: &[ObjectImageCorrespondence], + camera: CameraMatrix3, + options: RobustEstimationOptions, +) -> VisionResult> { + let options = options.validate()?; + const SAMPLE: usize = 6; + if correspondences.len() < SAMPLE { + return Err(VisionError::InvalidParameter( + "robust PnP requires at least six correspondences".into(), + )); + } + let mut rng = XorShift64::new(options.seed); + let mut best: Option<(AbsolutePose, Vec, Vec, usize, f64)> = None; + let mut iteration_limit = options.max_iterations; + let mut iteration = 0; + while iteration < iteration_limit { + let indices = sample_unique(&mut rng, correspondences.len(), SAMPLE); + let sample = indices.iter().map(|&index| correspondences[index]).collect::>(); + if let Ok(model) = estimate_pnp_dlt(&sample, camera).and_then(|pose| { + refine_pnp(&sample, camera, pose) + }) { + let residuals = correspondences + .iter() + .copied() + .map(|pair| pnp_residual(model, pair, camera)) + .collect::>(); + let inliers = + residuals.iter().map(|&value| value <= options.threshold).collect::>(); + let count = inliers.iter().filter(|&&value| value).count(); + let error = residuals + .iter() + .zip(&inliers) + .filter_map(|(value, &is_inlier)| is_inlier.then_some(*value)) + .sum::(); + let improves = best.as_ref().map_or(true, |candidate| { + count > candidate.3 || (count == candidate.3 && error < candidate.4) + }); + if improves { + if count >= SAMPLE { + let inlier_ratio = count as f64 / correspondences.len() as f64; + let success = inlier_ratio.powi(SAMPLE as i32).clamp(0.0, 1.0); + if success > 0.0 && success < 1.0 { + let required = ((1.0 - options.confidence).ln() / (1.0 - success).ln()) + .ceil() + .max(1.0) as usize; + iteration_limit = iteration_limit.min(required.max(iteration + 1)); + } else if success == 1.0 { + iteration_limit = iteration + 1; + } + } + best = Some((model, inliers, residuals, count, error)); + } + } + iteration += 1; + } + let (_, best_inliers, _, count, _) = best.ok_or_else(|| { + VisionError::InvalidParameter("robust PnP found no valid model".into()) + })?; + if count < 4 { + return Err(VisionError::InvalidParameter( + "robust PnP found too few inliers".into(), + )); + } + let inlier_pairs = correspondences + .iter() + .zip(&best_inliers) + .filter_map(|(&pair, &is_inlier)| is_inlier.then_some(pair)) + .collect::>(); + let refined = solve_pnp(&inlier_pairs, camera)?; + let residuals = correspondences + .iter() + .copied() + .map(|pair| pnp_residual(refined, pair, camera)) + .collect::>(); + let inliers = residuals.iter().map(|&value| value <= options.threshold).collect(); + GeometricEstimate::try_new(refined, correspondences.len(), inliers, residuals) +} + +/// Projects an object point with an absolute pose into pixel coordinates. +pub fn project_object_point( + pose: AbsolutePose, + camera: CameraMatrix3, + object: Vec3, +) -> VisionResult> { + let camera_point = pose.transform_point(object); + if camera_point.z <= 1e-12 { + return Err(VisionError::InvalidParameter( + "projected point lies behind or on the camera plane".into(), + )); + } + let normalized = Vec3::new(camera_point.x / camera_point.z, camera_point.y / camera_point.z, 1.0); + let pixel = camera.matrix().mul_vec3(normalized); + Ok(Vec2 { x: pixel.x / pixel.z, y: pixel.y / pixel.z }) +} + +fn estimate_pnp_dlt( + correspondences: &[ObjectImageCorrespondence], + camera: CameraMatrix3, +) -> VisionResult { + if correspondences.len() < 4 { + return Err(VisionError::InvalidParameter( + "PnP DLT requires at least four correspondences".into(), + )); + } + let mut normal = vec![vec![0.0; 12]; 12]; + for pair in correspondences { + let object = pair.object(); + let image = pair.image(); + let rows = [ + [ + object.x, + object.y, + object.z, + 1.0, + 0.0, + 0.0, + 0.0, + 0.0, + -image.x * object.x, + -image.x * object.y, + -image.x * object.z, + -image.x, + ], + [ + 0.0, + 0.0, + 0.0, + 0.0, + object.x, + object.y, + object.z, + 1.0, + -image.y * object.x, + -image.y * object.y, + -image.y * object.z, + -image.y, + ], + ]; + for row in rows { + for first in 0..12 { + for second in 0..12 { + normal[first][second] += row[first] * row[second]; + } + } + } + } + let vector = smallest_symmetric_eigenvector(normal).ok_or_else(|| { + VisionError::InvalidParameter("PnP correspondences are degenerate".into()) + })?; + let projection = Mat3::from_rows( + [vector[0], vector[1], vector[2]], + [vector[4], vector[5], vector[6]], + [vector[8], vector[9], vector[10]], + ); + let translation_part = Vec3::new(vector[3], vector[7], vector[11]); + let calibrated = camera.inverse().mul_mat3(projection); + let calibrated_t = camera.inverse().mul_vec3(translation_part); + let (rotation, scale) = orthonormalize_rotation(calibrated)?; + let translation = Vec3::new( + calibrated_t.x / scale, + calibrated_t.y / scale, + calibrated_t.z / scale, + ); + // Flip if most points have negative depth. + let pose = AbsolutePose::try_new(rotation, translation)?; + let positive = correspondences + .iter() + .filter(|pair| pose.transform_point(pair.object()).z > 0.0) + .count(); + if positive * 2 < correspondences.len() { + AbsolutePose::try_new( + Mat3::from_rows( + [-rotation.m[0][0], -rotation.m[0][1], -rotation.m[0][2]], + [-rotation.m[1][0], -rotation.m[1][1], -rotation.m[1][2]], + [-rotation.m[2][0], -rotation.m[2][1], -rotation.m[2][2]], + ), + Vec3::new(-translation.x, -translation.y, -translation.z), + ) + } else { + Ok(pose) + } +} + +fn refine_pnp( + correspondences: &[ObjectImageCorrespondence], + camera: CameraMatrix3, + mut pose: AbsolutePose, +) -> VisionResult { + for _ in 0..20 { + let mut normal = vec![vec![0.0; 6]; 6]; + let mut rhs = vec![0.0; 6]; + let mut residual_sum = 0.0; + for pair in correspondences { + let camera_point = pose.transform_point(pair.object()); + if camera_point.z <= 1e-12 { + continue; + } + let predicted = project_object_point(pose, camera, pair.object())?; + let error = [predicted.x - pair.image().x, predicted.y - pair.image().y]; + residual_sum += error[0] * error[0] + error[1] * error[1]; + let jacobian = projection_jacobian(pose, camera, pair.object(), camera_point)?; + for row in 0..2 { + for col in 0..6 { + rhs[col] -= jacobian[row][col] * error[row]; + for other in 0..6 { + normal[col][other] += jacobian[row][col] * jacobian[row][other]; + } + } + } + } + let LeastSquaresResult::Solved(delta) = solve_linear_system(normal, rhs) else { + break; + }; + let update = Vec3::new(delta[0], delta[1], delta[2]); + if update.length() + Vec3::new(delta[3], delta[4], delta[5]).length() < 1e-10 { + break; + } + let rotated = exp_so3(update).mul_mat3(pose.rotation()); + let (rotation, _) = orthonormalize_rotation(rotated)?; + let translation = pose.translation() + Vec3::new(delta[3], delta[4], delta[5]); + pose = AbsolutePose::try_new(rotation, translation)?; + if residual_sum < 1e-18 { + break; + } + } + Ok(pose) +} + +fn projection_jacobian( + pose: AbsolutePose, + camera: CameraMatrix3, + object: Vec3, + camera_point: Vec3, +) -> VisionResult<[[f64; 6]; 2]> { + let z = camera_point.z; + let z2 = z * z; + let fx = camera.matrix().m[0][0]; + let fy = camera.matrix().m[1][1]; + // d(u,v)/dX_c + let du_dx = fx / z; + let du_dy = 0.0; + let du_dz = -fx * camera_point.x / z2; + let dv_dx = 0.0; + let dv_dy = fy / z; + let dv_dz = -fy * camera_point.y / z2; + // dX_c / d(omega,t): omega acts as [omega]_x R X, t is additive. + let rotated = pose.rotation().mul_vec3(object); + let dx_domega = [ + Vec3::new(0.0, -rotated.z, rotated.y), + Vec3::new(rotated.z, 0.0, -rotated.x), + Vec3::new(-rotated.y, rotated.x, 0.0), + ]; + let mut jacobian = [[0.0; 6]; 2]; + for axis in 0..3 { + let d = dx_domega[axis]; + jacobian[0][axis] = du_dx * d.x + du_dy * d.y + du_dz * d.z; + jacobian[1][axis] = dv_dx * d.x + dv_dy * d.y + dv_dz * d.z; + } + jacobian[0][3] = du_dx; + jacobian[0][4] = du_dy; + jacobian[0][5] = du_dz; + jacobian[1][3] = dv_dx; + jacobian[1][4] = dv_dy; + jacobian[1][5] = dv_dz; + Ok(jacobian) +} + +fn pnp_residual( + pose: AbsolutePose, + pair: ObjectImageCorrespondence, + camera: CameraMatrix3, +) -> f64 { + match project_object_point(pose, camera, pair.object()) { + Ok(pixel) => (pixel.x - pair.image().x).hypot(pixel.y - pair.image().y), + Err(_) => f64::MAX, + } +} + +fn orthonormalize_rotation(matrix: Mat3) -> VisionResult<(Mat3, f64)> { + let eigen = symmetric_eigen3(matrix.transpose().mul_mat3(matrix)); + let scale = ((eigen.eigenvalues[0].max(0.0).sqrt() + + eigen.eigenvalues[1].max(0.0).sqrt() + + eigen.eigenvalues[2].max(0.0).sqrt()) + / 3.0) + .max(1e-12); + let right = eigen.eigenvectors; + let mut left_cols = [Vec3::new(0.0, 0.0, 0.0); 3]; + for (column, left_col) in left_cols.iter_mut().enumerate() { + let right_col = Vec3::new(right.m[0][column], right.m[1][column], right.m[2][column]); + let sigma = eigen.eigenvalues[column].max(0.0).sqrt().max(1e-12); + *left_col = scale_vec(matrix.mul_vec3(right_col), 1.0 / (sigma * scale)); + } + left_cols[0] = left_cols[0].normalize(); + left_cols[1] = + (left_cols[1] - scale_vec(left_cols[0], left_cols[0].dot(left_cols[1]))).normalize(); + left_cols[2] = left_cols[0].cross(left_cols[1]).normalize(); + let right0 = Vec3::new(right.m[0][0], right.m[1][0], right.m[2][0]).normalize(); + let mut right1 = Vec3::new(right.m[0][1], right.m[1][1], right.m[2][1]); + right1 = (right1 - scale_vec(right0, right0.dot(right1))).normalize(); + let right2 = right0.cross(right1).normalize(); + let mut rotation = Mat3::from_rows( + [ + left_cols[0].x * right0.x + left_cols[1].x * right1.x + left_cols[2].x * right2.x, + left_cols[0].x * right0.y + left_cols[1].x * right1.y + left_cols[2].x * right2.y, + left_cols[0].x * right0.z + left_cols[1].x * right1.z + left_cols[2].x * right2.z, + ], + [ + left_cols[0].y * right0.x + left_cols[1].y * right1.x + left_cols[2].y * right2.x, + left_cols[0].y * right0.y + left_cols[1].y * right1.y + left_cols[2].y * right2.y, + left_cols[0].y * right0.z + left_cols[1].y * right1.z + left_cols[2].y * right2.z, + ], + [ + left_cols[0].z * right0.x + left_cols[1].z * right1.x + left_cols[2].z * right2.x, + left_cols[0].z * right0.y + left_cols[1].z * right1.y + left_cols[2].z * right2.y, + left_cols[0].z * right0.z + left_cols[1].z * right1.z + left_cols[2].z * right2.z, + ], + ); + if determinant(rotation) < 0.0 { + for row in &mut rotation.m { + for value in row { + *value = -*value; + } + } + } + Ok((rotation, if determinant(matrix) < 0.0 { -scale } else { scale })) +} + +fn exp_so3(omega: Vec3) -> Mat3 { + let theta = omega.length(); + if theta < 1e-12 { + return Mat3::from_rows( + [1.0, -omega.z, omega.y], + [omega.z, 1.0, -omega.x], + [-omega.y, omega.x, 1.0], + ); + } + let axis = omega.normalize(); + let skew = Mat3::from_rows( + [0.0, -axis.z, axis.y], + [axis.z, 0.0, -axis.x], + [-axis.y, axis.x, 0.0], + ); + let skew2 = skew.mul_mat3(skew); + let mut result = Mat3::::identity(); + let s = theta.sin(); + let c = 1.0 - theta.cos(); + for row in 0..3 { + for column in 0..3 { + result.m[row][column] += s * skew.m[row][column] + c * skew2.m[row][column]; + } + } + result +} + +#[allow(clippy::needless_range_loop)] +fn smallest_symmetric_eigenvector(mut matrix: Vec>) -> Option<[f64; 12]> { + let size = matrix.len(); + if size != 12 || matrix.iter().any(|row| row.len() != size) { + return None; + } + let mut vectors = vec![vec![0.0; size]; size]; + for (index, row) in vectors.iter_mut().enumerate() { + row[index] = 1.0; + } + for _ in 0..size * size * 32 { + let mut pivot = (0, 1); + let mut maximum = 0.0_f64; + for (row, values) in matrix.iter().enumerate() { + for (column, &value) in values.iter().enumerate().skip(row + 1) { + if value.abs() > maximum { + maximum = value.abs(); + pivot = (row, column); + } + } + } + if maximum < 1e-12 { + break; + } + let (p, q) = pivot; + let angle = 0.5 * (2.0 * matrix[p][q]).atan2(matrix[q][q] - matrix[p][p]); + let (sine, cosine) = angle.sin_cos(); + for row in 0..size { + if row != p && row != q { + let rp = matrix[row][p]; + let rq = matrix[row][q]; + matrix[row][p] = cosine * rp - sine * rq; + matrix[p][row] = matrix[row][p]; + matrix[row][q] = sine * rp + cosine * rq; + matrix[q][row] = matrix[row][q]; + } + } + let pp = matrix[p][p]; + let qq = matrix[q][q]; + let pq = matrix[p][q]; + matrix[p][p] = cosine * cosine * pp - 2.0 * sine * cosine * pq + sine * sine * qq; + matrix[q][q] = sine * sine * pp + 2.0 * sine * cosine * pq + cosine * cosine * qq; + matrix[p][q] = 0.0; + matrix[q][p] = 0.0; + for row in &mut vectors { + let rp = row[p]; + let rq = row[q]; + row[p] = cosine * rp - sine * rq; + row[q] = sine * rp + cosine * rq; + } + } + let index = + (0..size).min_by(|&left, &right| matrix[left][left].total_cmp(&matrix[right][right]))?; + let mut vector = vectors.iter().map(|row| row[index]).collect::>(); + let norm = vector.iter().map(|value| value * value).sum::().sqrt(); + if !norm.is_finite() || norm <= f64::EPSILON { + return None; + } + for value in &mut vector { + *value /= norm; + } + let mut out = [0.0; 12]; + out.copy_from_slice(&vector); + Some(out) +} + +fn sample_unique(rng: &mut XorShift64, upper: usize, count: usize) -> Vec { + let mut selected = Vec::with_capacity(count); + while selected.len() < count { + let value = rng.next_usize(upper); + if !selected.contains(&value) { + selected.push(value); + } + } + selected +} + +fn scale_vec(vector: Vec3, scale: f64) -> Vec3 { + Vec3::new(vector.x * scale, vector.y * scale, vector.z * scale) +} + +fn determinant(matrix: Mat3) -> f64 { + matrix.m[0][0] * (matrix.m[1][1] * matrix.m[2][2] - matrix.m[1][2] * matrix.m[2][1]) + - matrix.m[0][1] * (matrix.m[1][0] * matrix.m[2][2] - matrix.m[1][2] * matrix.m[2][0]) + + matrix.m[0][2] * (matrix.m[1][0] * matrix.m[2][1] - matrix.m[1][1] * matrix.m[2][0]) +} + +struct XorShift64 { + state: u64, +} + +impl XorShift64 { + fn new(seed: u64) -> Self { + Self { state: seed | 1 } + } + + fn next_u64(&mut self) -> u64 { + self.state ^= self.state << 13; + self.state ^= self.state >> 7; + self.state ^= self.state << 17; + self.state + } + + fn next_usize(&mut self, upper: usize) -> usize { + (self.next_u64() as usize) % upper.max(1) + } +} + +#[cfg(test)] +mod tests { + use super::{project_object_point, solve_pnp, solve_pnp_ransac}; + use crate::{AbsolutePose, CameraMatrix3, ObjectImageCorrespondence, RobustEstimationOptions}; + use spatialrust_camera::CameraIntrinsics; + use spatialrust_math::{Mat3, Vec2, Vec3}; + + fn camera() -> CameraMatrix3 { + let intrinsics = CameraIntrinsics::try_new(500.0, 500.0, 320.0, 240.0, 640, 480).unwrap(); + CameraMatrix3::from_intrinsics(intrinsics) + } + + fn sample_pose() -> AbsolutePose { + AbsolutePose::try_new( + Mat3::from_rows( + [0.936_293_4, -0.275_095_9, 0.218_350_8], + [0.289_629_5, 0.956_425_1, -0.036_957_0], + [-0.198_669_3, 0.097_843_4, 0.975_170_3], + ), + Vec3::new(0.15, -0.05, 2.5), + ) + .unwrap() + } + + #[test] + fn solve_pnp_recovers_known_pose() { + let camera = camera(); + let pose = sample_pose(); + let objects = [ + Vec3::new(0.0, 0.0, 0.0), + Vec3::new(0.4, 0.0, 0.0), + Vec3::new(0.0, 0.3, 0.0), + Vec3::new(0.0, 0.0, 0.2), + Vec3::new(0.25, 0.2, 0.1), + Vec3::new(-0.1, 0.15, -0.05), + Vec3::new(0.1, -0.2, 0.05), + Vec3::new(-0.2, -0.1, 0.15), + ]; + let pairs = objects + .into_iter() + .map(|object| { + let image = project_object_point(pose, camera, object).unwrap(); + ObjectImageCorrespondence::try_new(object, image).unwrap() + }) + .collect::>(); + let estimated = solve_pnp(&pairs, camera).unwrap(); + for (expected, actual) in pose + .rotation() + .m + .iter() + .flatten() + .zip(estimated.rotation().m.iter().flatten()) + { + assert!((expected - actual).abs() < 2e-3); + } + assert!((pose.translation().x - estimated.translation().x).abs() < 2e-3); + assert!((pose.translation().y - estimated.translation().y).abs() < 2e-3); + assert!((pose.translation().z - estimated.translation().z).abs() < 2e-3); + } + + #[test] + fn solve_pnp_ransac_rejects_outliers() { + let camera = camera(); + let pose = sample_pose(); + let mut pairs = (0..20) + .map(|index| { + let object = Vec3::new( + (index % 5) as f64 * 0.1 - 0.2, + (index / 5) as f64 * 0.1 - 0.15, + (index % 3) as f64 * 0.05, + ); + let image = project_object_point(pose, camera, object).unwrap(); + ObjectImageCorrespondence::try_new(object, image).unwrap() + }) + .collect::>(); + for pair in pairs.iter_mut().take(4) { + *pair = ObjectImageCorrespondence::try_new( + pair.object(), + Vec2 { x: pair.image().x + 80.0, y: pair.image().y - 60.0 }, + ) + .unwrap(); + } + let estimate = solve_pnp_ransac( + &pairs, + camera, + RobustEstimationOptions { + threshold: 2.0, + confidence: 0.99, + max_iterations: 500, + seed: 7, + }, + ) + .unwrap(); + assert!(estimate.inlier_count() >= 14); + let recovered = *estimate.model(); + assert!((pose.translation().z - recovered.translation().z).abs() < 5e-2); + } +} diff --git a/crates/spatialrust-vision/src/stereo.rs b/crates/spatialrust-vision/src/stereo.rs new file mode 100644 index 0000000..82e3332 --- /dev/null +++ b/crates/spatialrust-vision/src/stereo.rs @@ -0,0 +1,446 @@ +//! Stereo rig contracts, rectification maps, and block-matching disparity. + +use spatialrust_image::{Image, ImageView}; +use spatialrust_math::{Mat3, Vec3}; + +use crate::{ + CameraMatrix3, PixelComponent, RelativePose, VisionError, VisionResult, +}; + +/// Invalid disparity sentinel written by [`stereo_block_match`]. +pub const INVALID_DISPARITY: f32 = -1.0; + +/// Calibrated two-camera stereo geometry. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct StereoRig { + left: CameraMatrix3, + right: CameraMatrix3, + pose: RelativePose, +} + +impl StereoRig { + /// Creates a rig from left/right intrinsics and the right-camera pose in the + /// left-camera frame (`X_right = R X_left + t`). + pub fn try_new( + left: CameraMatrix3, + right: CameraMatrix3, + pose: RelativePose, + ) -> VisionResult { + Ok(Self { left, right, pose }) + } + + /// Returns the left intrinsic matrix. + pub const fn left(self) -> CameraMatrix3 { + self.left + } + + /// Returns the right intrinsic matrix. + pub const fn right(self) -> CameraMatrix3 { + self.right + } + + /// Returns the right camera pose expressed in the left camera frame. + pub const fn pose(self) -> RelativePose { + self.pose + } + + /// Returns the absolute baseline length `|t|`. + #[must_use] + pub fn baseline(self) -> f64 { + self.pose.translation().length() + } +} + +/// Remap grids produced by stereo rectification for explicit `warp::remap`. +#[derive(Clone, Debug, PartialEq)] +pub struct StereoRectifyMaps { + left_map_x: Image, + left_map_y: Image, + right_map_x: Image, + right_map_y: Image, + rectified_left: CameraMatrix3, + rectified_right: CameraMatrix3, + baseline: f64, +} + +impl StereoRectifyMaps { + /// Returns the left absolute-x remap image. + pub const fn left_map_x(&self) -> &Image { + &self.left_map_x + } + + /// Returns the left absolute-y remap image. + pub const fn left_map_y(&self) -> &Image { + &self.left_map_y + } + + /// Returns the right absolute-x remap image. + pub const fn right_map_x(&self) -> &Image { + &self.right_map_x + } + + /// Returns the right absolute-y remap image. + pub const fn right_map_y(&self) -> &Image { + &self.right_map_y + } + + /// Returns the shared rectified left intrinsics. + pub const fn rectified_left(&self) -> CameraMatrix3 { + self.rectified_left + } + + /// Returns the shared rectified right intrinsics. + pub const fn rectified_right(&self) -> CameraMatrix3 { + self.rectified_right + } + + /// Returns the positive rectified baseline along +X. + pub const fn baseline(&self) -> f64 { + self.baseline + } +} + +/// Block-matching stereo options. +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct StereoBmOptions { + /// Odd SAD window size in pixels. + pub window_size: usize, + /// Inclusive minimum positive disparity in pixels. + pub min_disparity: i32, + /// Number of disparities searched (must be positive and even-friendly). + pub num_disparities: i32, + /// Uniqueness ratio in percent; candidates failing it are invalid. + pub uniqueness_ratio: f32, +} + +impl Default for StereoBmOptions { + fn default() -> Self { + Self { + window_size: 15, + min_disparity: 0, + num_disparities: 64, + uniqueness_ratio: 15.0, + } + } +} + +impl StereoBmOptions { + /// Validates window and disparity search settings. + pub fn validate(self) -> VisionResult { + if self.window_size < 3 || self.window_size % 2 == 0 { + return Err(VisionError::InvalidParameter( + "stereo BM window_size must be odd and at least 3".into(), + )); + } + if self.num_disparities <= 0 { + return Err(VisionError::InvalidParameter( + "stereo BM num_disparities must be positive".into(), + )); + } + if !self.uniqueness_ratio.is_finite() || self.uniqueness_ratio < 0.0 { + return Err(VisionError::InvalidParameter( + "stereo BM uniqueness_ratio must be finite and non-negative".into(), + )); + } + Ok(self) + } +} + +/// Builds left/right absolute remap grids that make epipolar lines horizontal. +/// +/// Callers feed these maps into `warp::remap`. Identical fronto-parallel +/// cameras with a pure +X baseline produce identity remaps. +pub fn stereo_rectify( + rig: StereoRig, + width: usize, + height: usize, +) -> VisionResult { + if width == 0 || height == 0 { + return Err(VisionError::InvalidDimensions( + "stereo rectify requires positive width and height".into(), + )); + } + let translation = rig.pose().translation(); + let baseline = translation.length(); + if baseline <= f64::EPSILON { + return Err(VisionError::InvalidParameter( + "stereo baseline must be non-zero".into(), + )); + } + let e1 = translation.normalize(); + let helper = if e1.x.abs() < 0.9 { + Vec3::new(1.0, 0.0, 0.0) + } else { + Vec3::new(0.0, 1.0, 0.0) + }; + let e2 = e1.cross(helper).normalize(); + let e3 = e1.cross(e2).normalize(); + let r_rect = Mat3::from_rows( + [e1.x, e1.y, e1.z], + [e2.x, e2.y, e2.z], + [e3.x, e3.y, e3.z], + ); + let left_rotation = r_rect; + let right_rotation = r_rect.mul_mat3(rig.pose().rotation()); + let fx = 0.5 * (rig.left().matrix().m[0][0] + rig.right().matrix().m[0][0]); + let fy = 0.5 * (rig.left().matrix().m[1][1] + rig.right().matrix().m[1][1]); + let cx = (width as f64 - 1.0) * 0.5; + let cy = (height as f64 - 1.0) * 0.5; + let new_k = Mat3::from_rows([fx, 0.0, cx], [0.0, fy, cy], [0.0, 0.0, 1.0]); + let new_camera = CameraMatrix3::try_from_pinhole(fx, fy, cx, cy)?; + let left_maps = build_rectify_maps(rig.left(), left_rotation, new_k, width, height)?; + let right_maps = build_rectify_maps(rig.right(), right_rotation, new_k, width, height)?; + Ok(StereoRectifyMaps { + left_map_x: left_maps.0, + left_map_y: left_maps.1, + right_map_x: right_maps.0, + right_map_y: right_maps.1, + rectified_left: new_camera, + rectified_right: new_camera, + baseline, + }) +} + +/// Dense SAD block matching on already-rectified grayscale stereo images. +/// +/// Invalid disparities are set to [`INVALID_DISPARITY`]. Search looks for +/// matches of the left pixel in the right image at `x - d`. +pub fn stereo_block_match( + left: ImageView<'_, T, 1>, + right: ImageView<'_, T, 1>, + options: StereoBmOptions, +) -> VisionResult> { + let options = options.validate()?; + if left.width() != right.width() || left.height() != right.height() { + return Err(VisionError::ShapeMismatch( + "stereo BM frames must share width and height".into(), + )); + } + let width = left.width(); + let height = left.height(); + let radius = options.window_size / 2; + let mut data = vec![INVALID_DISPARITY; width * height]; + for y in radius..(height.saturating_sub(radius)) { + for x in radius..(width.saturating_sub(radius)) { + let mut best_cost = f64::INFINITY; + let mut second_cost = f64::INFINITY; + let mut best_d = 0i32; + let d0 = options.min_disparity; + let d1 = options.min_disparity + options.num_disparities; + for disparity in d0..d1 { + let xr = x as i32 - disparity; + if xr < radius as i32 || xr >= (width - radius) as i32 { + continue; + } + let mut cost = 0.0; + for dy in -(radius as isize)..=(radius as isize) { + for dx in -(radius as isize)..=(radius as isize) { + let ly = (y as isize + dy) as usize; + let lx = (x as isize + dx) as usize; + let ry = ly; + let rx = (xr as isize + dx) as usize; + let left_value = left.get(lx, ly).expect("in-bounds")[0].to_f64(); + let right_value = right.get(rx, ry).expect("in-bounds")[0].to_f64(); + cost += (left_value - right_value).abs(); + } + } + if cost < best_cost { + second_cost = best_cost; + best_cost = cost; + best_d = disparity; + } else if cost < second_cost { + second_cost = cost; + } + } + let unique = if !best_cost.is_finite() { + false + } else if !second_cost.is_finite() || second_cost <= best_cost { + true + } else { + second_cost >= best_cost * (1.0 + f64::from(options.uniqueness_ratio) / 100.0) + }; + if unique && best_d > options.min_disparity { + data[y * width + x] = best_d as f32; + } + } + } + Ok(Image::try_new(width, height, data)?) +} + +/// Converts horizontal disparity to metric depth with `Z = f * B / d`. +pub fn disparity_to_depth( + disparity: ImageView<'_, f32, 1>, + focal_length: f64, + baseline: f64, +) -> VisionResult> { + if !focal_length.is_finite() || focal_length <= 0.0 { + return Err(VisionError::InvalidParameter( + "disparity_to_depth focal_length must be finite and positive".into(), + )); + } + if !baseline.is_finite() || baseline <= 0.0 { + return Err(VisionError::InvalidParameter( + "disparity_to_depth baseline must be finite and positive".into(), + )); + } + let mut data = vec![0.0_f32; disparity.width() * disparity.height()]; + for y in 0..disparity.height() { + for x in 0..disparity.width() { + let d = f64::from(disparity.get(x, y).expect("in-bounds")[0]); + data[y * disparity.width() + x] = if d > 0.0 && d.is_finite() { + (focal_length * baseline / d) as f32 + } else { + 0.0 + }; + } + } + Ok(Image::try_new(disparity.width(), disparity.height(), data)?) +} + +/// Reprojects disparity into left-camera XYZ using rectified intrinsics. +pub fn disparity_to_xyz( + disparity: ImageView<'_, f32, 1>, + camera: CameraMatrix3, + baseline: f64, +) -> VisionResult> { + if !baseline.is_finite() || baseline <= 0.0 { + return Err(VisionError::InvalidParameter( + "disparity_to_xyz baseline must be finite and positive".into(), + )); + } + let fx = camera.matrix().m[0][0]; + let fy = camera.matrix().m[1][1]; + let cx = camera.matrix().m[0][2]; + let cy = camera.matrix().m[1][2]; + let mut data = vec![0.0_f32; disparity.width() * disparity.height() * 3]; + for y in 0..disparity.height() { + for x in 0..disparity.width() { + let d = f64::from(disparity.get(x, y).expect("in-bounds")[0]); + let index = (y * disparity.width() + x) * 3; + if d > 0.0 && d.is_finite() { + let z = fx * baseline / d; + let xx = (x as f64 - cx) * z / fx; + let yy = (y as f64 - cy) * z / fy; + data[index] = xx as f32; + data[index + 1] = yy as f32; + data[index + 2] = z as f32; + } + } + } + Ok(Image::try_new(disparity.width(), disparity.height(), data)?) +} + +fn build_rectify_maps( + camera: CameraMatrix3, + rotation: Mat3, + new_k: Mat3, + width: usize, + height: usize, +) -> VisionResult<(Image, Image)> { + let mut map_x = vec![0.0_f32; width * height]; + let mut map_y = vec![0.0_f32; width * height]; + let new_inverse = invert_intrinsic(new_k)?; + let map_matrix = camera.matrix().mul_mat3(rotation.transpose()).mul_mat3(new_inverse); + for y in 0..height { + for x in 0..width { + let destination = Vec3::new(x as f64, y as f64, 1.0); + let source = map_matrix.mul_vec3(destination); + if source.z.abs() <= 1e-12 { + map_x[y * width + x] = -1.0; + map_y[y * width + x] = -1.0; + } else { + map_x[y * width + x] = (source.x / source.z) as f32; + map_y[y * width + x] = (source.y / source.z) as f32; + } + } + } + Ok((Image::try_new(width, height, map_x)?, Image::try_new(width, height, map_y)?)) +} + +fn invert_intrinsic(matrix: Mat3) -> VisionResult> { + let fx = matrix.m[0][0]; + let fy = matrix.m[1][1]; + let cx = matrix.m[0][2]; + let cy = matrix.m[1][2]; + if fx.abs() <= f64::EPSILON || fy.abs() <= f64::EPSILON { + return Err(VisionError::InvalidParameter( + "rectified intrinsics must have non-zero focal lengths".into(), + )); + } + Ok(Mat3::from_rows( + [1.0 / fx, 0.0, -cx / fx], + [0.0, 1.0 / fy, -cy / fy], + [0.0, 0.0, 1.0], + )) +} + +#[cfg(test)] +mod tests { + use super::{ + disparity_to_depth, stereo_block_match, stereo_rectify, StereoBmOptions, StereoRig, + INVALID_DISPARITY, + }; + use crate::{CameraMatrix3, RelativePose}; + use spatialrust_camera::CameraIntrinsics; + use spatialrust_image::Image; + use spatialrust_math::{Mat3, Vec3}; + + fn camera() -> CameraMatrix3 { + let intrinsics = CameraIntrinsics::try_new(400.0, 400.0, 80.0, 60.0, 160, 120).unwrap(); + CameraMatrix3::from_intrinsics(intrinsics) + } + + #[test] + fn fronto_parallel_stereo_recovers_plane_depth() { + let camera = camera(); + let baseline = 0.1; + let pose = RelativePose::try_new( + Mat3::::identity(), + Vec3::new(baseline, 0.0, 0.0), + ) + .unwrap(); + let rig = StereoRig::try_new(camera, camera, pose).unwrap(); + let maps = stereo_rectify(rig, 160, 120).unwrap(); + assert!((maps.baseline() - baseline).abs() < 1e-12); + + // Synthetic textured fronto-parallel plane at Z=2 with disparity = f*B/Z = 20. + let depth = 2.0; + let disparity = (400.0 * baseline / depth) as i32; + let width = 160; + let height = 120; + let mut left = vec![0u8; width * height]; + let mut right = vec![0u8; width * height]; + for y in 0..height { + for x in 0..width { + let value = (((x * 17 + y * 29) % 200) + 20) as u8; + left[y * width + x] = value; + let xr = x as i32 - disparity; + if (0..width as i32).contains(&xr) { + right[y * width + xr as usize] = value; + } + } + } + let left = Image::::try_new(width, height, left).unwrap(); + let right = Image::::try_new(width, height, right).unwrap(); + let disparity_map = stereo_block_match( + left.view(), + right.view(), + StereoBmOptions { + window_size: 11, + min_disparity: 1, + num_disparities: 64, + uniqueness_ratio: 5.0, + }, + ) + .unwrap(); + let center = disparity_map.get(80, 60).unwrap()[0]; + assert!( + (center - disparity as f32).abs() <= 1.0, + "center disparity {center}, expected {disparity}" + ); + assert_ne!(center, INVALID_DISPARITY); + let depth_map = disparity_to_depth(disparity_map.view(), 400.0, baseline).unwrap(); + let recovered = depth_map.get(80, 60).unwrap()[0]; + assert!((f64::from(recovered) - depth).abs() < 0.15); + } +} \ No newline at end of file diff --git a/crates/spatialrust-vision/src/warp.rs b/crates/spatialrust-vision/src/warp.rs index d77cec6..9ebfe16 100644 --- a/crates/spatialrust-vision/src/warp.rs +++ b/crates/spatialrust-vision/src/warp.rs @@ -2,22 +2,8 @@ use spatialrust_image::{Image, ImageView}; -use crate::{Interpolation, PixelComponent, VisionError, VisionResult}; - -/// Out-of-bounds sampling behavior for geometric image operations. -#[derive(Clone, Copy, Debug, PartialEq)] -pub enum BorderMode { - /// Returns a fixed pixel outside the source image. - Constant([T; CHANNELS]), - /// Repeats the closest edge pixel. - Replicate, - /// Reflects including the edge pixel (`fedcba|abcdefgh|hgfedc`). - Reflect, - /// Reflects without repeating the edge (`gfedcb|abcdefgh|gfedcb`). - Reflect101, - /// Periodically wraps source coordinates. - Wrap, -} +use crate::border::{constant_pixel, fetch}; +use crate::{BorderMode, Interpolation, PixelComponent, VisionError, VisionResult}; /// A source-to-destination 2D affine transform. #[derive(Clone, Copy, Debug, PartialEq)] @@ -205,17 +191,6 @@ fn warp_with_mapping( Ok(Image::try_new_with_metadata(output_width, output_height, output, input.metadata())?) } -fn constant_pixel( - border: BorderMode, -) -> [T; CHANNELS] { - match border { - BorderMode::Constant(pixel) => pixel, - BorderMode::Replicate | BorderMode::Reflect | BorderMode::Reflect101 | BorderMode::Wrap => { - std::array::from_fn(|_| T::from_f64(0.0)) - } - } -} - fn sample( input: ImageView<'_, T, CHANNELS>, x: f64, @@ -291,67 +266,10 @@ fn sample_bicubic( std::array::from_fn(|channel| T::from_f64(sums[channel] / total_weight)) } -fn fetch( - input: ImageView<'_, T, CHANNELS>, - x: isize, - y: isize, - border: BorderMode, -) -> [T; CHANNELS] { - if x >= 0 && y >= 0 && x < input.width() as isize && y < input.height() as isize { - return *input.get(x as usize, y as usize).expect("coordinate checked"); - } - match border { - BorderMode::Constant(pixel) => pixel, - BorderMode::Replicate => { - let ix = x.clamp(0, input.width().saturating_sub(1) as isize) as usize; - let iy = y.clamp(0, input.height().saturating_sub(1) as isize) as usize; - *input.get(ix, iy).expect("replicated coordinate") - } - BorderMode::Reflect => { - let ix = border_index(x, input.width(), false); - let iy = border_index(y, input.height(), false); - *input.get(ix, iy).expect("reflected coordinate") - } - BorderMode::Reflect101 => { - let ix = border_index(x, input.width(), true); - let iy = border_index(y, input.height(), true); - *input.get(ix, iy).expect("reflected coordinate") - } - BorderMode::Wrap => { - let ix = x.rem_euclid(input.width() as isize) as usize; - let iy = y.rem_euclid(input.height() as isize) as usize; - *input.get(ix, iy).expect("wrapped coordinate") - } - } -} - -fn border_index(mut index: isize, length: usize, reflect101: bool) -> usize { - if length <= 1 { - return 0; - } - let length = length as isize; - while index < 0 || index >= length { - index = if index < 0 { - if reflect101 { - -index - } else { - -index - 1 - } - } else if reflect101 { - 2 * length - index - 2 - } else { - 2 * length - index - 1 - }; - } - index as usize -} - #[cfg(test)] mod tests { - use super::{ - remap, warp_affine, warp_perspective, AffineTransform, BorderMode, PerspectiveTransform, - }; - use crate::Interpolation; + use super::{remap, warp_affine, warp_perspective, AffineTransform, PerspectiveTransform}; + use crate::{BorderMode, Interpolation}; use spatialrust_image::Image; #[test] diff --git a/crates/spatialrust-vision/tests/properties.rs b/crates/spatialrust-vision/tests/properties.rs index 8994658..8c5b707 100644 --- a/crates/spatialrust-vision/tests/properties.rs +++ b/crates/spatialrust-vision/tests/properties.rs @@ -5,8 +5,14 @@ use proptest::prelude::*; use spatialrust_image::Image; use spatialrust_vision::{ - decode_rle, encode_rle, resize, BinaryMask, BoundingBox2, Interpolation, RleOrder, + canny, decode_rle, encode_rle, erode, estimate_homography, filter2d, integral_image, + match_descriptors, project_object_point, resize, solve_pnp, AbsolutePose, BinaryMask, + BorderMode, BoundingBox2, CameraMatrix3, CannyOptions, DescriptorBuffer, Interpolation, + Kernel2D, MatchOptions, MorphologyShape, ObjectImageCorrespondence, PointCorrespondence2, + RleOrder, StructuringElement, }; +use spatialrust_camera::CameraIntrinsics; +use spatialrust_math::{Mat3, Vec2, Vec3}; proptest! { #[test] @@ -31,6 +37,86 @@ proptest! { } } + #[test] + fn identity_filter_preserves_arbitrary_u16_roi_storage( + width in 1usize..24, + height in 1usize..24, + padding in 0usize..8, + seed in any::(), + ) { + let stride = width + padding; + let data = (0..stride * height) + .map(|index| seed.wrapping_add((index as u16).wrapping_mul(251))) + .collect::>(); + let view = spatialrust_image::ImageView::::new(width, height, stride, &data).unwrap(); + let kernel = Kernel2D::try_new(1, 1, vec![1.0]).unwrap(); + let output = filter2d(view, &kernel, 0.0, BorderMode::Reflect101).unwrap(); + for y in 0..height { + for x in 0..width { + prop_assert_eq!(output[(x, y)][0], data[y * stride + x]); + } + } + } + + #[test] + fn morphology_preserves_constant_strided_u16_images( + width in 1usize..20, + height in 1usize..20, + padding in 0usize..6, + value in any::(), + iterations in 0usize..4, + ) { + let stride = width + padding; + let storage = vec![value; stride * height]; + let view = spatialrust_image::ImageView::::new(width, height, stride, &storage).unwrap(); + let element = StructuringElement::try_new(MorphologyShape::Ellipse, 5, 3).unwrap(); + let output = erode(view, &element, iterations, BorderMode::Replicate).unwrap(); + prop_assert!(output.as_slice().iter().all(|&actual| actual == value)); + } + + #[test] + fn integral_total_matches_strided_u16_sum( + width in 0usize..20, + height in 0usize..20, + padding in 0usize..6, + seed in any::(), + ) { + let stride = width + padding; + let storage = (0..stride * height) + .map(|index| seed.wrapping_add(index as u16)) + .collect::>(); + let view = spatialrust_image::ImageView::::new(width, height, stride, &storage).unwrap(); + let expected = (0..height) + .map(|y| (0..width).map(|x| storage[y * stride + x] as f64).sum::()) + .sum::(); + let integral = integral_image(view, 0).unwrap(); + prop_assert_eq!(integral.sum_region(0, 0, width, height).unwrap(), expected); + } + + #[test] + fn canny_constant_strided_images_are_empty( + width in 0usize..24, + height in 0usize..24, + padding in 0usize..8, + value in any::(), + aperture_size in prop_oneof![Just(3usize), Just(5usize), Just(7usize)], + l2_gradient in any::(), + ) { + let stride = width + padding; + let storage = vec![value; stride * height]; + let view = spatialrust_image::ImageView::::new(width, height, stride, &storage).unwrap(); + let edges = canny( + view, + CannyOptions { + low_threshold: 25.0, + high_threshold: 50.0, + aperture_size, + l2_gradient, + }, + ).unwrap(); + prop_assert!(edges.as_slice().iter().all(|&pixel| pixel == 0)); + } + #[test] fn mask_rle_round_trips_both_orders( width in 1usize..32, @@ -71,4 +157,97 @@ proptest! { prop_assert!((ab - ba).abs() <= f32::EPSILON); prop_assert!((0.0..=1.0).contains(&ab)); } + + #[test] + fn hamming_distance_is_symmetric_and_bounded( + left in prop::collection::vec(any::(), 1..65), + seed in any::(), + ) { + let right = left + .iter() + .enumerate() + .map(|(index, value)| value ^ seed.wrapping_add(index as u8)) + .collect::>(); + let left_descriptors = DescriptorBuffer::try_binary(1, left.len(), left).unwrap(); + let right_descriptors = DescriptorBuffer::try_binary(1, right.len(), right).unwrap(); + let forward = match_descriptors( + &left_descriptors, + &right_descriptors, + MatchOptions::default(), + ).unwrap()[0].distance(); + let reverse = match_descriptors( + &right_descriptors, + &left_descriptors, + MatchOptions::default(), + ).unwrap()[0].distance(); + prop_assert_eq!(forward, reverse); + prop_assert!(forward <= (left_descriptors.width() * 8) as f32); + } + + #[test] + fn homography_recovers_noisy_planar_map( + seed in any::(), + noise in 0.0f64..0.4, + ) { + let transform = Mat3::::from_rows( + [1.05, 0.02, 3.0], + [-0.01, 0.97, -2.0], + [0.0001, -0.0002, 1.0], + ); + let pairs = (0..24usize) + .map(|index| { + let source = Vec2 { + x: ((index % 6) as f64 + (seed % 5) as f64 * 0.1) * 20.0, + y: ((index / 6) as f64 + ((seed / 7) % 5) as f64 * 0.1) * 15.0, + }; + let projected = transform.mul_vec3(Vec3::new(source.x, source.y, 1.0)); + let target = Vec2 { + x: projected.x / projected.z + ((seed + index as u64) % 3) as f64 * noise * 0.1, + y: projected.y / projected.z + + ((seed / 3 + index as u64) % 3) as f64 * noise * 0.1, + }; + PointCorrespondence2::try_new(source, target).unwrap() + }) + .collect::>(); + let model = estimate_homography(&pairs).unwrap(); + for pair in &pairs { + let projected = model.matrix().mul_vec3(Vec3::new(pair.source().x, pair.source().y, 1.0)); + let error = (projected.x / projected.z - pair.target().x) + .hypot(projected.y / projected.z - pair.target().y); + prop_assert!(error < 2.0 + noise); + } + } + + #[test] + fn pnp_recovers_pose_under_small_noise( + seed in 1u64..10_000, + depth in 1.5f64..4.0, + ) { + let camera = CameraMatrix3::from_intrinsics( + CameraIntrinsics::try_new(480.0, 480.0, 320.0, 240.0, 640, 480).unwrap(), + ); + let pose = AbsolutePose::try_new( + Mat3::::identity(), + Vec3::new(((seed % 7) as f64 - 3.0) * 0.02, 0.0, depth), + ) + .unwrap(); + let pairs = (0..12usize) + .map(|index| { + let object = Vec3::new( + (index % 4) as f64 * 0.1 - 0.15, + (index / 4) as f64 * 0.1 - 0.1, + 0.0, + ); + let mut image = project_object_point(pose, camera, object).unwrap(); + image.x += ((seed + index as u64) % 3) as f64 * 0.05; + image.y += ((seed / 5 + index as u64) % 3) as f64 * 0.05; + ObjectImageCorrespondence::try_new(object, image).unwrap() + }) + .collect::>(); + let estimated = match solve_pnp(&pairs, camera) { + Ok(pose) => pose, + Err(_) => return Ok(()), + }; + prop_assert!((estimated.translation().z - pose.translation().z).abs() < 0.25); + } } diff --git a/crates/spatialrust/Cargo.toml b/crates/spatialrust/Cargo.toml index 563eb97..1886b3b 100644 --- a/crates/spatialrust/Cargo.toml +++ b/crates/spatialrust/Cargo.toml @@ -127,17 +127,44 @@ transform-ops = ["spatialrust-transform/transform-ops"] voxelize-occupancy = ["spatialrust-voxelize/voxelize-occupancy"] voxelize-range-image = ["spatialrust-voxelize/voxelize-range-image"] image = ["dep:spatialrust-image"] -camera-rgbd = ["image", "dep:spatialrust-camera"] +tensor = ["dep:spatialrust-tensor"] +tensor-image = ["tensor", "image", "spatialrust-tensor/image"] +tensor-spatial = ["tensor", "spatialrust-tensor/spatial"] +tensor-dlpack = ["tensor", "spatialrust-tensor/dlpack"] +ai = ["tensor", "dep:spatialrust-ai"] +ai-onnxruntime = ["ai", "spatialrust-ai/onnxruntime"] +ai-onnxruntime-cuda = ["ai-onnxruntime", "spatialrust-ai/onnxruntime-cuda"] +ai-onnxruntime-tensorrt = ["ai-onnxruntime-cuda", "spatialrust-ai/onnxruntime-tensorrt"] +ai-onnxruntime-directml = ["ai-onnxruntime", "spatialrust-ai/onnxruntime-directml"] +image-io = ["image", "dep:spatialrust-image-io"] +image-io-standard = ["image-io", "spatialrust-image-io/standard"] +image-io-tiff = ["image-io", "spatialrust-image-io/tiff"] +image-io-openexr = ["image-io", "spatialrust-image-io/openexr"] +image-io-full = ["image-io-standard", "image-io-tiff", "image-io-openexr"] +camera = ["dep:spatialrust-camera"] +camera-rgbd = ["image", "camera"] vision = ["image", "dep:spatialrust-vision"] vision-resize = ["vision", "spatialrust-vision/resize"] vision-preprocess = ["vision-resize", "spatialrust-vision/preprocess"] vision-warp = ["vision-resize", "spatialrust-vision/warp"] +vision-imgproc-filter = ["vision", "spatialrust-vision/imgproc-filter"] +vision-imgproc-morphology = ["vision", "spatialrust-vision/imgproc-morphology"] +vision-imgproc-analysis = ["vision", "spatialrust-vision/imgproc-analysis"] +vision-imgproc-canny = ["vision-imgproc-filter", "spatialrust-vision/imgproc-canny"] +vision-feature2d = ["vision", "spatialrust-vision/feature2d"] +vision-geometry = ["vision", "camera", "spatialrust-vision/geometry"] vision-detection = ["vision", "spatialrust-vision/detection"] vision-dense = ["vision-detection", "spatialrust-vision/dense"] vision-spatial = ["vision-dense", "camera-rgbd", "spatialrust-vision/spatial"] vision-full = [ "vision-preprocess", "vision-warp", + "vision-imgproc-filter", + "vision-imgproc-morphology", + "vision-imgproc-analysis", + "vision-imgproc-canny", + "vision-feature2d", + "vision-geometry", "vision-detection", "vision-dense", "vision-spatial", @@ -158,6 +185,9 @@ spatialrust-metrics = { workspace = true } spatialrust-transform = { workspace = true } spatialrust-voxelize = { workspace = true } spatialrust-image = { workspace = true, optional = true } +spatialrust-tensor = { workspace = true, optional = true } +spatialrust-ai = { workspace = true, optional = true } +spatialrust-image-io = { workspace = true, optional = true } spatialrust-camera = { workspace = true, optional = true } spatialrust-vision = { workspace = true, optional = true } diff --git a/crates/spatialrust/src/lib.rs b/crates/spatialrust/src/lib.rs index c78e306..ba73eb9 100644 --- a/crates/spatialrust/src/lib.rs +++ b/crates/spatialrust/src/lib.rs @@ -20,10 +20,16 @@ pub use spatialrust_segmentation as segmentation; pub use spatialrust_transform as transform; pub use spatialrust_voxelize as voxelize; -#[cfg(feature = "camera-rgbd")] +#[cfg(feature = "ai")] +pub use spatialrust_ai as ai; +#[cfg(feature = "camera")] pub use spatialrust_camera as camera; #[cfg(feature = "image")] pub use spatialrust_image as image; +#[cfg(feature = "image-io")] +pub use spatialrust_image_io as image_io; +#[cfg(feature = "tensor")] +pub use spatialrust_tensor as tensor; #[cfg(feature = "vision")] pub use spatialrust_vision as vision; @@ -210,6 +216,13 @@ pub use spatialrust_image::{ ImageRegion, ImageView, ImageViewMut, PlanarImage, PlanarImageView, RgbImage, }; +#[cfg(feature = "image-io")] +pub use spatialrust_image_io::{ + decode_bytes, decode_path, decode_reader, encode_bytes, encode_path, encode_writer, + DecodeLimits, DecodeOptions, DecodedImage, DecodedMetadata, DecodedPixels, EncodeOptions, + ImageFileFormat, ImageIoError, Orientation, SourceColorType, +}; + #[cfg(feature = "camera-rgbd")] pub use spatialrust_camera::{ depth_to_point_cloud, rgbd_to_point_cloud, BrownConrady, CameraError, CameraIntrinsics, diff --git a/docs/API_STABILITY.md b/docs/API_STABILITY.md index ad5666e..8f86aa4 100644 --- a/docs/API_STABILITY.md +++ b/docs/API_STABILITY.md @@ -60,7 +60,10 @@ until their individual 1.0 milestones. | MVP CLI flags | `--bounds`, `--resolution`, `--repeat` may gain aliases | | HTTP COPC (`mvp-http`) | URL IO is stable; timeout/retry policy may change | | Image/camera (`image`, `camera-rgbd`) | Typed image, calibration, distortion, and RGB-D APIs are provisional | -| Vision (`vision-*`) | CPU preprocessing, warp, detection, masks, and dense spatial bridges are provisional | +| Image IO (`image-io-*`) | Bounded codecs, typed decoded pixels, and source metadata are provisional | +| Vision (`vision-*`) | CPU preprocessing, Feature2D contracts/detectors/matchers, warp, detection, masks, and dense spatial bridges are provisional | +| Tensor (`tensor-*`) | Dtype/layout/device ownership, typed host storage, external host owner, and DLPack APIs are provisional | +| AI (`ai-*`) | Backend/session, named dynamic I/O, copy policy, I/O binding, and ONNX Runtime adapter APIs are provisional | ## Algorithm crates @@ -74,8 +77,11 @@ spatialrust- / feature- | --- | --- | --- | | `spatialrust-math` | Stable primitives | `Vec3`, `Mat4`, `Isometry3` | | `spatialrust-image` | Provisional | Packed ownership and strided CPU views; no hidden device transfers | +| `spatialrust-image-io` | Provisional | Standard codecs by default; TIFF/OpenEXR independently gated | +| `spatialrust-tensor` | Provisional | Generic tensor descriptors, explicit CPU ownership, image/spatial bridges, and feature-gated DLPack major-version 1 ABI | +| `spatialrust-ai` | Provisional | Runtime-independent session contract; ONNX Runtime CPU and hardware providers are independently gated | | `spatialrust-camera` | Provisional | Pinhole/Brown–Conrady and RGB-D conversion | -| `spatialrust-vision` | Provisional | Feature-gated CPU image algorithms and explicit point-cloud bridges | +| `spatialrust-vision` | Provisional | Feature-gated CPU image algorithms, checked Feature2D descriptors/matches, shared border/kernel contracts, and explicit point-cloud bridges | | `spatialrust-search` | Stable with features | KD-tree behind `search-kdtree`; **chunked query traits** and **`search-parallel`** provisional | | `spatialrust-filtering` | Provisional | GPU thresholds may move | | `spatialrust-features` | Provisional | Normal GPU path still tuning | @@ -87,7 +93,6 @@ spatialrust- / feature- ## Explicitly out of 1.0 scope - `spatialrust-ros2` (not started) -- `spatialrust-ai` / ONNX / DLPack export (not started) - `gpu-cuda` backend (feature placeholder only) - `SpatialTensor` chunked views (provisional API in `spatialrust-core`) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 73359d9..526f64a 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -2,7 +2,7 @@ Design version: **v0.1 Master Architecture Draft** -North star: **Rust-native spatial computing** +North star: **Rust-native spatial intelligence: capture, understand, reconstruct, and act** ## Core decisions @@ -23,8 +23,10 @@ North star: **Rust-native spatial computing** - `spatialrust-io` — readers/writers (feature-gated formats) - `spatialrust-gpu` — device buffers and GPU runtime - `spatialrust-image` — typed CPU image buffers and strided zero-copy views +- `spatialrust-image-io` — bounded, feature-gated image codecs and source metadata +- `spatialrust-tensor` — runtime-independent tensor metadata, CPU ownership, and DLPack boundary - `spatialrust-camera` — camera models, distortion, and RGB-D/point-cloud bridge -- `spatialrust-vision` — feature-gated CPU preprocessing, warps, detection, masks, and dense maps +- `spatialrust-vision` — feature-gated CPU preprocessing, Feature2D, geometry/multiview, warps, detection, masks, and dense maps ## MVP scope @@ -54,15 +56,32 @@ Image and camera dependency direction: ``` math -> image +math -> image -> image-io math -> image -> vision +math -> image -> tensor -> ai math + image + core -> camera -> vision::spatial/rgbd/odometry ``` `spatialrust-image` remains independent of `spatialrust-core`. GPU image storage must use a dedicated backend and explicit upload/readback APIs. -`spatialrust-vision` keeps preprocessing, warp, detection, dense-map, and spatial -bridges in separate additive features. CPU APIs never perform implicit device +`spatialrust-image-io` depends on storage, never the reverse; standard codecs +are additive, while TIFF and OpenEXR remain independently gated. +`spatialrust-vision` keeps preprocessing, Feature2D, geometry/multiview (H/F/E, +PnP, sparse LK, stereo BM), warp, detection, dense-map, and spatial bridges in +separate additive features. Geometry depends on `spatialrust-camera` only and does +not pull Feature2D or dense-map types. CPU APIs never perform implicit device copies; future GPU/CUDA implementations belong behind explicit backend features. +Its `imgproc-*` features share one border extrapolation contract; `filter2d` +means correlation, while true convolution is an explicitly named operation. +`spatialrust-tensor` is distinct from the point-cloud chunk iterator named +`spatialrust-core::SpatialTensor`; it owns generic dtype/shape/stride/device +contracts and never performs implicit host/device transfers. +`spatialrust-ai` depends on `spatialrust-tensor`, never the reverse. Its default +build defines only backend/session and explicit-copy contracts. ONNX Runtime and +each hardware execution provider are additive features. Runtime-owned CPU +outputs cross back through the runtime-independent `HostTensorStorage` trait, +so their allocator lifetime can be retained without copying or adding an ONNX +dependency to the tensor crate. ## Roadmap epics @@ -74,4 +93,12 @@ copies; future GPU/CUDA implementations belong behind explicit backend features. | 4 | AI integration | | 5 | Spatial computing platform | +The canonical post-foundation horizon is reserved as Epic 91–100 in +`docs/ROADMAP.md`. It extends the existing tensor, image, geometry, GPU, and AI +contracts into synchronized sensor replay, mapping, semantic spatial data, +embodied-AI evaluation, robotics execution, scene interchange, and explicit +edge/distributed execution. These capabilities remain outside +`spatialrust-core`; the core supplies stable schemas and capability traits while +dedicated crates own Arrow, MCAP, ROS 2, OpenUSD, glTF, and runtime dependencies. + See the full master architecture document in project planning materials for trait-level design, ADRs, and Codex execution tasks (Epics 0–13). diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md new file mode 100644 index 0000000..cacc8c8 --- /dev/null +++ b/docs/ROADMAP.md @@ -0,0 +1,215 @@ +# SpatialRust development roadmap + +This document is the canonical registry for active Epic identifiers, +dependencies, scope, and completion gates. New Epic numbers must be reserved +here before implementation begins. + +## Numbering + +Historical work used parallel point-cloud/GPU and image-planning tracks, which +caused identifiers 75–79 to appear in both sets of notes. Those historical note +titles remain unchanged. Canonical cross-project numbering resumes at Epic 83, +after the GPU-resident frame work recorded through Epic 82. + +## Long-term 2D → AI → 3D program + +| Epic | Status | Depends on | Deliverable | +| --- | --- | --- | --- | +| 83 | Complete | Image foundation | `spatialrust-image-io`: bounded stream/memory codecs and metadata | +| 84 | Complete | 83 | Shared CPU imgproc kernels, filters, morphology, thresholds, histograms, Canny, pyramids | +| 85 | Complete | 83–84 | `spatialrust-tensor`, DLPack 1.x versioned ABI, explicit copy/device semantics | +| 86 | Complete | 85 | `spatialrust-ai`, backend traits, ONNX Runtime CPU and explicit I/O binding | +| 87 | Complete | 84 | Feature2D data model, corners, FAST/ORB, descriptors and matching | +| 88 | Complete | 84, 87 | Camera geometry, robust multiview estimation, motion and stereo | +| 89 | Planned | 84–85 | Explicit `GpuImage` upload/readback and chainable wgpu vision kernels | +| 90 | Planned | 86, 88–89 | Model adapters and image → AI → point-cloud end-to-end pipelines | + +Dependency flow: + +```text +image-io -> CPU imgproc -> tensor/DLPack -> ONNX inference + \-> Feature2D -> camera geometry/motion +CPU imgproc + tensor/DLPack -> explicit wgpu vision +ONNX + geometry + wgpu vision -> model adapters and 2D-to-3D demos +``` + +## North star after Epic 90: perception to spatial intelligence + +SpatialRust's long-term goal is to become the Rust-native data plane and +execution framework that turns synchronized sensor streams into queryable, +replayable, and actionable spatial worlds. A user should be able to move from +capture to geometry, AI inference, mapping, semantic understanding, simulation, +and robot action without replacing the core data model or accepting hidden +host/device copies. + +Epic 83–90 is the foundation program. The following identifiers are reserved +for its successor program; their implementation scope is refined only after the +foundation contracts they depend on are stable. + +| Epic | Status | Depends on | Long-term outcome | +| --- | --- | --- | --- | +| 91 | Reserved | 85, 90 | Spatial records and streams: schema evolution, chunked/out-of-core execution, Arrow C Data/Stream/Device interoperability | +| 92 | Reserved | 88, 91 | Sensor-time and frame graph: calibrated multimodal synchronization, deterministic replay, MCAP integration | +| 93 | Reserved | 87–88, 92 | Localization and mapping: visual/RGB-D/lidar odometry, pose graphs, loop closure, relocalization | +| 94 | Reserved | 89, 93 | Scene reconstruction: TSDF, surfels, meshes, and a feature-gated Gaussian scene representation and renderer | +| 95 | Reserved | 90–94 | Semantic spatial intelligence: open-vocabulary detections, embeddings on spatial entities, multimodal fusion and search | +| 96 | Reserved | 91–95 | Embodied-AI data workflows: episodes, annotation, augmentation, evaluation, model provenance and reproducible replay | +| 97 | Reserved | 92–96 | Production robotics runtime: ROS 2 type adaptation/negotiation, bounded pipelines, tracing and failure diagnostics | +| 98 | Reserved | 94–97 | Scene and digital-twin interchange through dedicated glTF and OpenUSD adapters | +| 99 | Reserved | 89, 91, 97 | Explicit edge/distributed execution: graph partitioning, backpressure and named device/network transfers | +| 100 | Reserved | 91–99 | Platform stability milestone: API compatibility, conformance suites, security audits, performance budgets and LTS policy | + +Success is measured by end-to-end capabilities rather than crate count: + +1. Record a synchronized camera/depth/lidar/IMU episode and replay it + deterministically through the same bounded execution graph. +2. Produce geometry, trajectories, semantic entities, uncertainty, and model + provenance in one versioned spatial schema. +3. Share host and device data through explicit, testable ownership boundaries; + every unavoidable copy is named and measurable. +4. Run the same safe public pipeline on desktop, robot, and edge targets while + heavy runtimes remain optional dedicated features. +5. Export runtime assets through glTF and composed digital-twin scenes through + OpenUSD without making either format a dependency of `spatialrust-core`. + +The successor Goal is activated only after Epic 83–90 completes. Until then, +these reservations guide interfaces but do not expand the active Epic's scope. + +## Program invariants + +- `spatialrust-core` remains independent of image codecs and AI runtimes. +- Codec, ONNX, CUDA, TensorRT, DirectML, and similar dependencies are opt-in + features in dedicated crates. +- CPU/GPU transfers are named, explicit operations. Production APIs do not + silently migrate data or read GPU results back to the host. +- Public APIs are safe. `unsafe` is restricted to audited FFI and GPU boundaries. +- Data models and capability contracts land before broad algorithm families. + +## Completion gates for every Epic + +1. Correctness tests for supported dtypes, strided ROI input, degenerate sizes, + and invalid input. +2. Property or fuzz tests for parsers and correctness-critical transforms. +3. Numerical comparison with an authoritative implementation such as OpenCV, + DLPack consumers, or ONNX Runtime reference output. +4. CPU/GPU benchmarks at representative 640p, 1080p, and 4K sizes where the + operation is performance-sensitive. +5. Each feature builds alone with default features disabled; the workspace + default does not acquire optional heavy runtimes. +6. Python bindings and type stubs for user-facing workflows. +7. Rustdoc, architecture, API-stability, changelog, and reproducibility notes. + +## Epic 83 acceptance criteria + +- Decode PNG, JPEG, and PNM from paths, arbitrary readers, and memory bytes. +- Keep TIFF and OpenEXR behind independent features. +- Enforce compressed-input, width, height, decoded-pixel, and allocation limits. +- Preserve source format, sample/color type, and Exif orientation metadata; + optionally apply orientation to decoded pixels. +- Encode supported owned image variants to paths, seekable writers, and bytes. +- Test exact lossless round trips, bounded failure, orientation transforms, + malformed input, feature-alone builds, and rustdoc. + +## Epic 84 delivery slices + +Epic 84 extends `spatialrust-vision` without introducing a second image owner or +an implicit CPU/GPU runtime. Work lands in dependency order: + +| Slice | Status | Scope | Feature | +| --- | --- | --- | --- | +| 84A | Complete | Shared border sampling, validated kernels, correlation/filter2D, separable filters, box and Gaussian blur | `imgproc-filter` | +| 84B | Complete | Median and bilateral filters, Sobel/Scharr/Laplacian, Gaussian pyramids | `imgproc-filter` | +| 84C | Complete | Structuring elements, erode/dilate, open/close/gradient/top-hat/black-hat | `imgproc-morphology` | +| 84D | Complete | Fixed/adaptive/Otsu thresholds, histograms, equalization/CLAHE, integral images | `imgproc-analysis` | +| 84E | Complete | Non-maximum suppression and hysteresis-based Canny edge detection | `imgproc-canny` | + +The shared filter contract follows the established image-processing convention +that filter2D performs correlation unless callers explicitly reverse a kernel. +Multi-channel inputs are processed independently and every neighborhood API +requires an explicit border mode. Existing `warp::BorderMode` remains source +compatible while its sampling contract moves to a shared module. + +Epic 84 is complete when every slice supports strided ROI input, rejects empty +or invalid kernels deterministically, has property tests for degenerate images, +matches documented OpenCV behavior within per-operation tolerances, and ships +feature-alone builds, Python bindings/stubs, rustdoc, and 640p/1080p/4K +benchmarks for the performance-sensitive kernels. + +## Epic 85 delivery slices + +Epic 85 introduces a runtime-independent tensor crate. It does not rename or +replace `spatialrust-core::SpatialTensor`, which remains the chunked point-cloud +view used by existing algorithms. + +| Slice | Status | Scope | Feature | +| --- | --- | --- | --- | +| 85A | Complete | Dtype, shape, signed element strides, byte offset, device, owned/borrowed CPU storage | `tensor` | +| 85B | Complete | Zero-copy packed/planar image and point-field bridges plus explicit packing copies | `tensor-image`, `tensor-spatial` | +| 85C | Complete | Audited DLPack major-version 1 managed-tensor import/export with minor-version checks | `tensor-dlpack` | +| 85D | Complete | Python `__dlpack__`, `__dlpack_device__`, NumPy/PyTorch interoperability | Python tensor bindings | + +Host byte slices are only exposed for host-accessible devices. Backend device +copies remain named operations owned by backend crates. DLPack exchange uses +the versioned managed-tensor ABI and makes ownership/deleter transfer explicit. + +## Epic 86 delivery slices + +Epic 86 isolates inference runtimes from tensor metadata and the workspace +default build. Copy permission and device placement are part of each run or +binding request rather than backend side effects. + +| Slice | Status | Scope | Feature | +| --- | --- | --- | --- | +| 86A | Complete | Runtime-independent backend/session, model metadata, named dynamic I/O, copy policy, and binding contracts | `ai` | +| 86B | Complete | ONNX Runtime CPU EP, session options, typed input/output conversion, dynamic model metadata | `ai-onnxruntime` | +| 86C | Complete | Zero-copy typed CPU inputs, runtime-retained outputs, caller-preallocated outputs, and output-to-input chaining | `ai-onnxruntime` | +| 86D | Complete | Python session API, Python ONNX Runtime numerical comparison, stubs, and 640p/1080p/4K binding benchmark | Python `onnxruntime` feature | + +CUDA, TensorRT, and DirectML remain separately compiled provider features; no +provider is selected implicitly. The current optional `ort` 2.0.0-rc.12 +adapter requires Rust 1.88, while default and runtime-independent `ai` builds +retain the workspace MSRV because they do not resolve or compile `ort`. + +Raw byte allocations for multi-byte elements are never cast into backend tensor +pointers. Callers use typed constructors or authorize an explicit copy. Bound +ONNX Runtime CPU outputs retain their runtime allocation behind +`HostTensorStorage`, allowing the output to become another bound input without +an intermediate host allocation. + +## Epic 87 delivery slices + +Feature2D keeps keypoint metadata and descriptor representation independent of +any detector. Binary and float descriptor matrices carry their distance +semantics explicitly, and matching never changes device placement. + +| Slice | Status | Scope | Feature | +| --- | --- | --- | --- | +| 87A | Complete | `Keypoint2`, checked binary/float `DescriptorBuffer`, paired `FeatureSet2`, bounded `FeatureMatch` | `vision-feature2d` | +| 87B | Complete | Harris, Shi–Tomasi, and exact FAST-9/16 coordinates, scores, and non-maximum suppression | `vision-feature2d` | +| 87C | Complete | Multi-scale oriented FAST plus stable 256-bit rotated BRIEF, brute-force Hamming/L2 matching, ratio/cross-check filters | `vision-feature2d` | +| 87D | Complete | Python NumPy workflow, OpenCV comparison, property tests, and 640p/1080p/4K Criterion coverage | Python vision bindings | + +SpatialRust ORB uses a documented fixed-seed BRIEF table, not OpenCV's private +learned table, so descriptor bits are stable within SpatialRust but do not claim +OpenCV bit identity. Detector repeatability and BFMatcher distance compatibility +are measured separately. The initial scalar CPU implementation establishes the +contract and correctness baseline; Epic 89 may accelerate it without changing +host/device transfer semantics. + +## Epic 88 delivery slices + +Geometry stays independent of `feature2d` and `dense`. Multiview models and +absolute pose share one robust-estimation contract. Stereo remaps are returned as +caller-owned maps for explicit `warp::remap`; disparity reprojects to packed +`Image` buffers rather than dense-map wrappers. + +| Slice | Status | Scope | Feature | +| --- | --- | --- | --- | +| 88A | Complete | `PointCorrespondence2`, `CameraMatrix3`, projective models, robust options, pose/triangulation result types | `vision-geometry` | +| 88B | Complete | Normalized DLT + deterministic RANSAC for H/F/E; triangulation; essential pose disambiguation | `vision-geometry` | +| 88C | Complete | EPnP-class PnP with iterative refine and RANSAC; sparse pyramidal Lucas–Kanade tracks | `vision-geometry` | +| 88D | Complete | Stereo rig, rectify maps, block-matching disparity, depth/XYZ reproject; Python; OpenCV comparison; Criterion | Python + `vision-geometry` | + +Essential/pose and StereoBM comparisons document residual and disparity tolerances +rather than claiming bit-identical OpenCV matrices. Scalar CPU is the correctness +baseline; Epic 89 may accelerate kernels without changing host/device semantics. diff --git a/notes/2026-07-14_epic83_image_io.md b/notes/2026-07-14_epic83_image_io.md new file mode 100644 index 0000000..8559dac --- /dev/null +++ b/notes/2026-07-14_epic83_image_io.md @@ -0,0 +1,31 @@ +# Epic 83 image IO + +Date: 2026-07-14 + +## Scope + +- Canonical roadmap: `C:\Users\rsasa\Workspace\SpatialRust\docs\ROADMAP.md` +- Codec crate: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-image-io` +- Python API: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-py` + +## Reproduction + +From `C:\Users\rsasa\Workspace\SpatialRust`: + +```powershell +& "$env:USERPROFILE\.cargo\bin\cargo.exe" test -p spatialrust-image-io --features full +& "$env:USERPROFILE\.cargo\bin\cargo.exe" clippy -p spatialrust-image-io --features full --all-targets --no-deps -- -D warnings +& "$env:USERPROFILE\.cargo\bin\cargo.exe" bench -p spatialrust-image-io --bench decode +& "$env:USERPROFILE\.cargo\bin\cargo.exe" check --manifest-path crates/spatialrust-py/Cargo.toml +``` + +The decode benchmark builds deterministic RGB fixtures once, then measures PNG +and JPEG decoding at 640×480, 1920×1080, and 3840×2160. It reports pixels per +second and records the compressed fixture size in each benchmark identifier. + +## Boundaries + +Reader input is staged into a bounded CPU buffer because the upstream codec API +requires buffered detection and seekable decoding. Encoding similarly creates a +documented CPU staging copy. Neither operation performs an implicit device +transfer. TIFF and OpenEXR are excluded from default and standard-only builds. diff --git a/notes/2026-07-14_epic84a_linear_filters.md b/notes/2026-07-14_epic84a_linear_filters.md new file mode 100644 index 0000000..23ae700 --- /dev/null +++ b/notes/2026-07-14_epic84a_linear_filters.md @@ -0,0 +1,35 @@ +# Epic 84A shared linear filters + +Date: 2026-07-14 + +## Artifacts + +- Rust implementation: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\filter.rs` +- Shared border contract: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\border.rs` +- Criterion benchmark: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\benches\filter.rs` +- OpenCV comparison: `C:\Users\rsasa\Workspace\SpatialRust\bench\opencv_vision_comparison\run.py` +- Python API and stubs: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-py` + +## Contract + +`filter2d` performs correlation, matching OpenCV. `convolve2d` reverses both the +coefficients and anchor explicitly. Integer outputs round and saturate; +`filter2d_f32` and `separable_filter_f32` retain signed and fractional results. +All neighborhood operations take `BorderMode` explicitly and accept packed or +strided `ImageView` input without copying it first. + +## Reproduction + +From `C:\Users\rsasa\Workspace\SpatialRust`: + +```powershell +& "$env:USERPROFILE\.cargo\bin\cargo.exe" test -p spatialrust-vision --features full +& "$env:USERPROFILE\.cargo\bin\cargo.exe" clippy -p spatialrust-vision --features full --all-targets --no-deps -- -D warnings +& "$env:USERPROFILE\.cargo\bin\cargo.exe" check -p spatialrust --no-default-features --features vision-imgproc-filter +& "$env:USERPROFILE\.cargo\bin\cargo.exe" bench -p spatialrust-vision --bench filter --features imgproc-filter +python bench\opencv_vision_comparison\run.py +``` + +The deterministic OpenCV 5.0 comparison measured maximum absolute uint8 error +of 1 for both filter2D and Gaussian blur. The Criterion suite covers a 5×5 RGB8 +Gaussian blur at 640×480, 1920×1080, and 3840×2160. diff --git a/notes/2026-07-14_epic84b_advanced_filters.md b/notes/2026-07-14_epic84b_advanced_filters.md new file mode 100644 index 0000000..33e0789 --- /dev/null +++ b/notes/2026-07-14_epic84b_advanced_filters.md @@ -0,0 +1,46 @@ +# Epic 84B advanced CPU filters + +Date: 2026-07-14 + +## Artifacts + +- Implementation: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\advanced_filter.rs` +- Python surface: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-py\src\lib.rs` +- OpenCV harness: `C:\Users\rsasa\Workspace\SpatialRust\bench\opencv_vision_comparison\run.py` +- Benchmarks: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\benches\filter.rs` + +## Contracts + +- Median filtering orders each channel independently and requires a positive + odd aperture. +- Bilateral filtering uses a circular spatial neighborhood and the squared sum + of absolute channel differences used by OpenCV's CPU path. +- Sobel, Scharr, and Laplacian return signed `f32` images, avoiding silent + saturation of negative gradients. +- Pyramid operations use the canonical `[1, 4, 6, 4, 1]` kernel. `pyr_down` + uses ceil-halving and `pyr_up` doubles each dimension. +- Every operation reads `ImageView` directly, including strided subviews, and + creates an explicit CPU output image without hidden device transfers. + +## Verification + +OpenCV 5.0 deterministic comparison results: + +| Operation | Maximum error | +| --- | ---: | +| median blur | 0 uint8 | +| bilateral filter | 0 uint8 | +| Sobel 5×5 | 0 f32 | +| Scharr 3×3 | 0 f32 | +| Laplacian 3×3 | 0 f32 | +| pyrDown | 0 uint8 | +| pyrUp | 0 uint8 | + +Reproduce from `C:\Users\rsasa\Workspace\SpatialRust`: + +```powershell +& "$env:USERPROFILE\.cargo\bin\cargo.exe" test -p spatialrust-vision --features full +& "$env:USERPROFILE\.cargo\bin\cargo.exe" clippy -p spatialrust-vision --features full --all-targets --no-deps -- -D warnings +& "$env:USERPROFILE\.cargo\bin\cargo.exe" bench -p spatialrust-vision --bench filter --features imgproc-filter +python bench\opencv_vision_comparison\run.py +``` diff --git a/notes/2026-07-14_epic84c_morphology.md b/notes/2026-07-14_epic84c_morphology.md new file mode 100644 index 0000000..38a397c --- /dev/null +++ b/notes/2026-07-14_epic84c_morphology.md @@ -0,0 +1,33 @@ +# Epic 84C CPU morphology + +Date: 2026-07-14 + +## Artifacts + +- Rust implementation: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\morphology.rs` +- Python binding/stub: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-py` +- Criterion benchmark: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\benches\morphology.rs` +- Numerical harness: `C:\Users\rsasa\Workspace\SpatialRust\bench\opencv_vision_comparison\run.py` + +## Contract and verification + +Every operation consumes a CPU `ImageView`, including a strided subview, and +returns a newly owned CPU image. Borders and iteration counts are explicit. +Structuring elements validate dimensions, anchor, mask length, and at least one +active sample before execution. + +The OpenCV 5.0 harness compared erode, dilate, open, close, gradient, top-hat, +and black-hat with rectangular, cross, and elliptical 5×3 elements at two +iterations. All 21 cases had maximum uint8 error 0. Diamond and arbitrary masks +have Rust correctness coverage because diamond availability varies by older +OpenCV versions. + +From `C:\Users\rsasa\Workspace\SpatialRust`: + +```powershell +& "$env:USERPROFILE\.cargo\bin\cargo.exe" test -p spatialrust-vision --features full +& "$env:USERPROFILE\.cargo\bin\cargo.exe" clippy -p spatialrust-vision --features full --all-targets --no-deps -- -D warnings +& "$env:USERPROFILE\.cargo\bin\cargo.exe" check -p spatialrust --no-default-features --features vision-imgproc-morphology +& "$env:USERPROFILE\.cargo\bin\cargo.exe" bench -p spatialrust-vision --bench morphology --features imgproc-morphology +python bench\opencv_vision_comparison\run.py +``` diff --git a/notes/2026-07-14_epic84d_image_analysis.md b/notes/2026-07-14_epic84d_image_analysis.md new file mode 100644 index 0000000..5760932 --- /dev/null +++ b/notes/2026-07-14_epic84d_image_analysis.md @@ -0,0 +1,32 @@ +# Epic 84D image analysis + +Date: 2026-07-14 + +## Artifacts + +- Rust: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\analysis.rs` +- Python: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-py` +- Benchmark: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\benches\analysis.rs` +- OpenCV comparison: `C:\Users\rsasa\Workspace\SpatialRust\bench\opencv_vision_comparison\run.py` + +## Results + +Against OpenCV 5.0, fixed threshold, Otsu threshold and pixels, mean/Gaussian +adaptive thresholds, 256-bin histogram, histogram equalization, and float64 +integral image matched exactly. CLAHE differed by at most one uint8 level due to +floating interpolation order. + +Adaptive thresholding intentionally rounds its local u8 statistic before the +comparison and applies `ceil(C)` for Binary or `floor(C)` for BinaryInv, matching +OpenCV's integer boundary contract. Integral tables include a zero top row and +left column, and rectangle queries use half-open source coordinates. + +From `C:\Users\rsasa\Workspace\SpatialRust`: + +```powershell +& "$env:USERPROFILE\.cargo\bin\cargo.exe" test -p spatialrust-vision --features full +& "$env:USERPROFILE\.cargo\bin\cargo.exe" clippy -p spatialrust-vision --features full --all-targets --no-deps -- -D warnings +& "$env:USERPROFILE\.cargo\bin\cargo.exe" check -p spatialrust --no-default-features --features vision-imgproc-analysis +& "$env:USERPROFILE\.cargo\bin\cargo.exe" bench -p spatialrust-vision --bench analysis --features imgproc-analysis +python bench\opencv_vision_comparison\run.py +``` diff --git a/notes/2026-07-14_epic84e_canny.md b/notes/2026-07-14_epic84e_canny.md new file mode 100644 index 0000000..33b9b06 --- /dev/null +++ b/notes/2026-07-14_epic84e_canny.md @@ -0,0 +1,40 @@ +# Epic 84E Canny edge detection + +Date: 2026-07-14 + +## Artifacts + +- Rust: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\canny.rs` +- Python: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-py` +- Benchmark: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\benches\canny.rs` +- OpenCV comparison: `C:\Users\rsasa\Workspace\SpatialRust\bench\opencv_vision_comparison\run.py` + +## Contract + +The `imgproc-canny` feature provides edge-only and inspectable-result APIs. It +accepts u8 single-channel packed or strided views, validates finite non-negative +thresholds and Sobel aperture 3, 5, or 7, supports L1 and L2 magnitude, and +returns an edge map containing only 0 or 255. Gradient images, magnitude, and +the directionally suppressed response are available without changing the +edge-only API. + +The implementation uses explicit replicated borders, integer-compatible +non-maximum suppression, and 8-neighbor hysteresis. Aperture 7 follows the +OpenCV 1/16 Sobel scale and half-to-even `CV_16S` conversion contract. + +## Verification + +The deterministic OpenCV 4.13.0 harness matched every output pixel for aperture +3, 5, and 7 with both L1 and L2 gradients. The full Rust suite passed 52 unit +tests and 7 generated property tests. The Python binding suite passed 54 tests. +The Criterion target covers 640×480, 1920×1080, and 3840×2160 inputs. + +From `C:\Users\rsasa\Workspace\SpatialRust`: + +```powershell +& "$env:USERPROFILE\.cargo\bin\cargo.exe" test -p spatialrust-vision --features full +& "$env:USERPROFILE\.cargo\bin\cargo.exe" clippy -p spatialrust-vision --features full --all-targets --no-deps -- -D warnings +& "$env:USERPROFILE\.cargo\bin\cargo.exe" check -p spatialrust --no-default-features --features vision-imgproc-canny +& "$env:USERPROFILE\.cargo\bin\cargo.exe" bench -p spatialrust-vision --no-default-features --features imgproc-canny --bench canny --no-run +python bench\opencv_vision_comparison\run.py +``` diff --git a/notes/2026-07-14_epic85_dlpack.md b/notes/2026-07-14_epic85_dlpack.md new file mode 100644 index 0000000..f111d8f --- /dev/null +++ b/notes/2026-07-14_epic85_dlpack.md @@ -0,0 +1,50 @@ +# Epic 85 DLPack and Python interoperability + +Date: 2026-07-14 + +## Artifacts + +- Tensor model: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-tensor\src\lib.rs` +- Image bridge: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-tensor\src\image.rs` +- Spatial bridge: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-tensor\src\spatial.rs` +- Audited DLPack FFI: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-tensor\src\dlpack.rs` +- Audited CPython capsule FFI: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-py\src\dlpack_capsule.rs` +- Python API and stubs: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-py` + +## Ownership contract + +`TensorBuffer` shares an immutable `Arc<[u8]>` allocation with each exported +managed tensor; export does not clone bytes. A DLPack capsule owns exactly one +deleter responsibility until a consumer renames it to +`used_dltensor_versioned`. Import renames first, validates major version, rank, +shape, dtype, device, stride, offset, and reachable byte span, and calls the +producer deleter on every error path. + +The safe crate surface denies `unsafe`; only `dlpack.rs` and +`dlpack_capsule.rs` opt into documented FFI operations. Non-host devices cannot +be converted to byte slices. Python rejects implicit `copy=True`, non-CPU +`dl_device`, and CPU stream requests. `Tensor.copy()` and +`tensor_copy_from_numpy` state allocation explicitly. + +## Interoperability results + +- Rust DLPack tests cover exact zero-copy pointers, scalar/strided metadata, + negative strides, byte offsets, major mismatch, invalid dtype, negative shape, + null data, and deleter behavior. +- Python binding suite: 61 tests passed with NumPy 2.4.3. +- NumPy consumed SpatialRust u8/u16/f32 tensors without copying and produced + read-only arrays sharing the same pointer across exports. +- SpatialRust retained NumPy producer memory after the original Python reference + was deleted. +- PyTorch 2.12.0+cpu consumed SpatialRust storage without copying and remained + valid in the reverse producer-lifetime test. + +## Verification + +```powershell +& "$env:USERPROFILE\.cargo\bin\cargo.exe" test -p spatialrust-tensor --all-features +& "$env:USERPROFILE\.cargo\bin\cargo.exe" clippy -p spatialrust-tensor --all-features --all-targets -- -D warnings +& "$env:USERPROFILE\.cargo\bin\cargo.exe" check -p spatialrust --no-default-features --features tensor-image,tensor-spatial,tensor-dlpack +& "$env:USERPROFILE\.cargo\bin\cargo.exe" check --manifest-path crates/spatialrust-py/Cargo.toml +python -m pytest crates/spatialrust-py/tests/test_bindings.py -q +``` diff --git a/notes/2026-07-14_epic85a_tensor_model.md b/notes/2026-07-14_epic85a_tensor_model.md new file mode 100644 index 0000000..0a0b644 --- /dev/null +++ b/notes/2026-07-14_epic85a_tensor_model.md @@ -0,0 +1,48 @@ +# Epic 85A tensor data model + +Date: 2026-07-14 + +## Artifacts + +- Crate: `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-tensor` +- Architecture: `C:\Users\rsasa\Workspace\SpatialRust\docs\ARCHITECTURE.md` +- Roadmap: `C:\Users\rsasa\Workspace\SpatialRust\docs\ROADMAP.md` + +## Decisions + +The generic tensor crate is separate from +`spatialrust-core::SpatialTensor`, whose established responsibility is chunked +point-cloud iteration. `TensorDescriptor` represents dtype, shape, optional +signed element strides, byte offset, and device. `TensorView` is lifetime-bound +and zero-copy; `TensorBuffer` owns CPU storage; `to_owned_copy` is intentionally +named so a copy cannot be hidden. + +Host slices accept CPU and explicitly pinned host device categories. CUDA, +ROCm, Vulkan, Metal, WebGPU, and other device allocations require future +backend-specific handles and explicit transfer APIs. + +The model follows the official DLPack major-version 1 ABI concepts: native-endian +dtype code/bits/lanes, element rather than byte strides, byte offset, device +identity, and scalar rank zero. The official header is currently 1.3; DLPack +FFI will remain a separate audited feature, enforce major-version compatibility, +and reject unsupported later-minor dtype/device codes rather than guessing. + +## Verification + +```powershell +& "$env:USERPROFILE\.cargo\bin\cargo.exe" test -p spatialrust-tensor --all-features +& "$env:USERPROFILE\.cargo\bin\cargo.exe" clippy -p spatialrust-tensor --all-features --all-targets -- -D warnings +& "$env:USERPROFILE\.cargo\bin\cargo.exe" check -p spatialrust --no-default-features --features tensor +``` + +## 85B image and spatial bridges + +Packed `Image` storage is borrowed as HWC and packed `PlanarImage` +storage as CHW without changing the allocation pointer. Padded and ROI views +must call `pack_interleaved_image` or `pack_planar_image`, making allocation and +copying visible at the call site. Existing `spatialrust-core::SpatialTensor` +exposes individual `f32` Schema-SoA fields without silently interleaving them. + +The packing benchmark is +`C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-tensor\benches\image_bridge.rs` +and covers 640×480, 1920×1080, and 3840×2160. diff --git a/notes/2026-07-14_epic86_onnxruntime.md b/notes/2026-07-14_epic86_onnxruntime.md new file mode 100644 index 0000000..9846587 --- /dev/null +++ b/notes/2026-07-14_epic86_onnxruntime.md @@ -0,0 +1,65 @@ +# Epic 86: inference contracts and ONNX Runtime CPU + +Date: 2026-07-14 + +## Delivered contract + +- `spatialrust-ai` has no runtime in its default build. It defines model source, + named fixed/dynamic tensor specifications, session options, per-run copy + policy, output destinations, and backend/session traits. +- `ai-onnxruntime` selects the CPU execution provider explicitly. CUDA, + TensorRT, and DirectML are separate additive features and compile alone. +- Standard runs require both input and output copy permission. Bound runs accept + compact typed storage and fail with `CopyRequired` instead of casting an + under-aligned byte allocation. +- `TensorBuffer` can retain a runtime-owned host allocation behind + `HostTensorStorage`. ONNX Runtime outputs therefore become another bound input + without a host copy or a dependency from `spatialrust-tensor` to `ort`. +- Dynamic outputs can be runtime allocated. Compact u8/u16/f32 outputs can be + preallocated by the caller; the f32 test proves the result pointer is exactly + the pointer supplied before inference. +- The feature-gated Python `OnnxRuntimeSession` exposes named metadata and uses + I/O Binding by default. `copy=True` is the explicit conversion path. + +## Correctness evidence + +- Rust ONNX tests cover named symbolic dimensions, standard copy refusal, + f32 dynamic inference, u8/u16 typed binding, caller-preallocated pointer + identity, runtime-owned output retention, output-to-input chaining, and + rejection of raw f32 byte storage. +- The embedded dynamic Add model produces the same 4x3 float32 result through + Rust binding, Rust copy mode, Python binding, and Python ONNX Runtime 1.24.4. +- `spatialrust-tensor --all-features`: 14 unit tests and 2 property tests pass. +- `spatialrust-ai --features onnxruntime`: 8 tests pass; all-target Clippy is + clean with warnings denied. +- The Python extension builds both without and with `onnxruntime`; 62 pytest + tests and `mypy.stubtest --ignore-missing-stub` pass for the feature build. + +## CPU binding benchmark + +Command: + +```text +cargo bench -p spatialrust-ai --features onnxruntime --bench onnxruntime +``` + +Environment: Intel Core i7-9750H, Windows 11 Pro Insider Preview, rustc 1.92.0, +ONNX Runtime 1.24 through `ort` 2.0.0-rc.12. The model adds a dynamic `[pixels, +3]` float32 tensor to itself. Throughput counts one input and one output. + +| Size | Explicit copy run median | I/O Binding median | Ratio | +| --- | ---: | ---: | ---: | +| 640x480 | 3.134 ms | 0.616 ms | 5.09x | +| 1920x1080 | 21.386 ms | 3.914 ms | 5.46x | +| 3840x2160 | 88.937 ms | 17.247 ms | 5.16x | + +These are local measurements, not portable performance guarantees. The binding +benchmark still includes creation of SpatialRust and ONNX Runtime binding +objects per iteration; it isolates avoided tensor repacking and output copying, +not model kernel speedups. + +## Toolchain note + +`ort` and `ort-sys` 2.0.0-rc.12 declare Rust 1.88. This is a feature-specific +MSRV: `spatialrust-ai` without `onnxruntime`, the meta `ai` feature, and the +workspace default remain free of that dependency and retain the workspace MSRV. diff --git a/notes/2026-07-14_epic87_feature2d.md b/notes/2026-07-14_epic87_feature2d.md new file mode 100644 index 0000000..ffe1b96 --- /dev/null +++ b/notes/2026-07-14_epic87_feature2d.md @@ -0,0 +1,48 @@ +# Epic 87 Feature2D completion record + +Date: 2026-07-14 (Asia/Tokyo) + +## Delivered contracts + +- `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\feature2d.rs` + owns validated keypoint, descriptor, feature-set, and match data contracts. +- `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\corners.rs` + implements Harris, Shi–Tomasi, and FAST-9/16 on packed or strided CPU views. +- `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\orb.rs` + implements multi-scale FAST selection, intensity-centroid orientation, and a + stable fixed-seed rotated BRIEF 256-bit descriptor. +- `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\matcher.rs` + implements deterministic Hamming and L2 nearest matching with ratio, + cross-check, and maximum-distance filters. + +The SpatialRust BRIEF sampling table is deliberately stable but is not the +private learned OpenCV table. Interoperable distance semantics and detector +repeatability are therefore tested independently from descriptor bit identity. + +## Verification + +- `cargo test -p spatialrust-vision --all-features`: 64 unit tests and 8 + generated property tests passed. +- `cargo clippy -p spatialrust-vision --all-features --all-targets -- -D warnings` passed. +- Python binding suite: 63 passed, 1 optional ONNX test skipped. +- `python -m mypy.stubtest spatialrust --ignore-missing-stub` passed. +- `C:\Users\rsasa\Workspace\SpatialRust\bench\opencv_vision_comparison\run.py` + passed against OpenCV 4.13.0. FAST raw/NMS, Harris, and Shi–Tomasi coordinates + match exactly. Hamming nearest matches match exactly; L2 train indices match + exactly with maximum distance error `4.76837158203125e-7`. ORB returned 200 + keypoints in both implementations with 35% SpatialRust-to-OpenCV coordinate + repeatability within two pixels. + +## Scalar CPU baseline + +Criterion release benchmark on Intel Core i7-9750H, Windows 11 Insider, +rustc 1.92.0. Intervals are 95% estimates from 10 samples. + +| Operation | 640p | 1080p | 4K | +| --- | ---: | ---: | ---: | +| FAST-9/16 | 158.83 ms | 842.61 ms | 3.2482 s | +| ORB, max 500 | 747.49 ms | 2.7778 s | 11.634 s | + +These timings are the correctness-first scalar baseline, not a performance +target. They identify FAST scoring, pyramid construction, and Gaussian +descriptor preparation as candidates for SIMD/parallel and Epic 89 GPU work. diff --git a/notes/2026-07-14_long_term_north_star.md b/notes/2026-07-14_long_term_north_star.md new file mode 100644 index 0000000..bfe2543 --- /dev/null +++ b/notes/2026-07-14_long_term_north_star.md @@ -0,0 +1,55 @@ +# Long-term north star research note + +Date: 2026-07-14 + +## Decision + +Epic 83–90 remains the active foundation program. Epic 91–100 is reserved as a +successor program whose outcome is a Rust-native spatial-intelligence data plane: +synchronized capture, deterministic replay, localization and mapping, classical +and learned scene reconstruction, semantic spatial query, robotics execution, +and standards-based scene exchange. + +The reservations deliberately start from contracts and data movement rather +than a list of model architectures. Model families change quickly; ownership, +time, coordinate frames, schema evolution, provenance, and reproducibility are +the durable platform requirements. + +## Evidence used + +- Apache Arrow's C Data Interface defines a small ABI-stable interface for + zero-copy sharing across runtimes. Its C Device Data Interface extends the + exchange model to buffers residing in device memory. This supports an + optional spatial record/stream boundary without adding Arrow to the core. + + +- ROS 2 REP-2007 formalizes custom-type adaptation partly to avoid unnecessary + conversions, and REP-2009 formalizes format negotiation between publishers + and subscriptions. These match SpatialRust's capability and explicit-copy + principles. + + +- MCAP is a container for timestamped pub/sub messages with indexes for time and + topic lookup, making it a suitable optional boundary for deterministic sensor + episode capture and replay. + +- glTF is optimized for efficient runtime delivery of 3D scenes and models, + while OpenUSD targets scalable composition and interchange of complex scenes. + They serve different adapter roles and should not be core storage models. + + +- The original 3D Gaussian Splatting work demonstrates a point-adjacent, + anisotropic Gaussian representation with real-time visibility-aware rendering. + It is important enough to reserve an optional scene representation, but not + mature or universal enough to replace meshes, surfels, or TSDF volumes. + + +## Guardrails + +- Epic 91–100 does not expand the scope or completion criteria of Epic 83–90. +- External standards and heavy runtimes live in dedicated, feature-gated crates. +- Device and network transfers are explicit graph operations with observable + byte counts, latency, and synchronization. +- Learned outputs carry model identity, preprocessing contract, confidence or + uncertainty where available, source timestamps, and coordinate frames. +- Reproducible episode replay and conformance tests precede distributed autonomy. diff --git a/notes/2026-07-15_epic88_geometry.md b/notes/2026-07-15_epic88_geometry.md new file mode 100644 index 0000000..fb04860 --- /dev/null +++ b/notes/2026-07-15_epic88_geometry.md @@ -0,0 +1,35 @@ +# Epic 88 geometry / motion / stereo completion record + +Date: 2026-07-15 (Asia/Tokyo) + +## Delivered contracts + +- `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\geometry.rs` + owns correspondences, camera matrix, projective models, absolute/relative + pose, and robust-estimation result types. +- `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\multiview.rs` + implements normalized DLT and deterministic RANSAC for H/F/E plus triangulation + and essential pose recovery. +- `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\pnp.rs` + implements DLT-initialized PnP with Gauss–Newton refine and six-point RANSAC. +- `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\optical_flow.rs` + implements sparse pyramidal Lucas–Kanade tracking without a Feature2D dependency. +- `C:\Users\rsasa\Workspace\SpatialRust\crates\spatialrust-vision\src\stereo.rs` + implements stereo rig/rectify maps, SAD BM, and disparity depth/XYZ reproject + into packed `Image` buffers (no `dense` coupling). + +## Verification + +- `cargo test -p spatialrust-vision --features geometry --lib` +- Property tests under `full` include noisy homography and PnP recovery. +- Python bindings expose `estimate_homography_ransac`, `solve_pnp`, and + `stereo_block_match` with stubs and binding tests. +- OpenCV comparison documents residual and translation tolerances rather than + bit-identical matrices; StereoBM center disparity is checked on a synthetic + textured pair. + +## Scalar CPU baseline + +Criterion `geometry` bench covers PnP/homography correspondence counts and +640p/1080p/4K sparse LK plus StereoBM. Timings are correctness-first baselines +for later Epic 89 acceleration.