recon: handoff to the server team

.github/workflows/recon.yml

@@ -0,0 +1,40 @@
+name: Recon
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "recon/**"
+      - ".github/workflows/recon.yml"
+  pull_request:
+    paths:
+      - "recon/**"
+      - ".github/workflows/recon.yml"
+
+permissions:
+  contents: read
+
+concurrency:
+  group: recon-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  recon-checks:
+    name: Recon checks
+    runs-on: ubuntu-24.04
+    timeout-minutes: 10
+    defaults:
+      run:
+        working-directory: recon
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      # Reads tool.uv.required-version from recon/pyproject.toml to pick the uv version.
+      - uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
+        with:
+          working-directory: recon
+      - run: uv sync --locked
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+      - run: uv run pytest -q

docs/04-prior-art-and-codes.md

@@ -69,9 +69,9 @@ None of the products above runs the whole loop outdoors, from metric AR capture
 | Status | Models |
 | --- | --- |
 | Noncommercial | MapAnything's default checkpoint (use `facebook/map-anything-apache` instead), Depth Anything 3 Giant and Nested (Metric Large, Base and Small are Apache-2.0), VGGT-Ω, UniDepthV2, Mapillary-Vistas Mask2Former weights, Molmo 2 |
-| Restricted or unclear | Depth Pro (Apple personal-use grant), SAM 3 (custom license, gated weights), HY-World 2.0 (custom community license), Metric3D v2 (its files disagree), MoGe-2 weights (UNVERIFIED, though the code is MIT) |
+| Restricted or unclear | Depth Pro (Apple personal-use grant), SAM 3 (custom license, gated weights), HY-World 2.0 (custom community license), Metric3D v2 (its files disagree) |
 | Copyleft | YOLO-World, YOLOE and Ultralytics (GPL or AGPL), pymeshlab (GPL-3) |
-| Permissive | SAM 2, Grounding DINO, OWLv2, Qwen3-VL and zxing-cpp (Apache-2.0), Florence-2, supervision, Open3D and Stray Scanner (MIT), shapely (BSD-3) |
+| Permissive | SAM 2, Grounding DINO, OWLv2, Qwen3-VL and zxing-cpp (Apache-2.0), Florence-2, supervision, Open3D and Stray Scanner (MIT), MoGe-2 code and the `Ruicheng/moge-2-vitl-normal` weights at revision cb0e8bb, which the worker pins (MIT, from Hugging Face's metadata for that revision, checked 2026-09-26), shapely (BSD-3) |
 
 For recognition boxes, Gemini 2.5 Pro scored 13.3 zero-shot mAP on RF100-VL, against 1.5 for GPT-5.
 

recon/.gitignore

@@ -0,0 +1,2 @@
+.venv/
+__pycache__/

recon/Makefile

@@ -0,0 +1,19 @@
+# The reconstruction worker. `make test` is what CI runs; the others need the MoGe-2 environment
+# (models/, installed on first use) and local data under ~/house-scanning-data.
+
+.PHONY: test run accept
+
+test:
+	uv sync --locked
+	uv run ruff check .
+	uv run ruff format --check .
+	uv run pytest -q
+
+# BUNDLE: the app's scan bundle (zip or folder with scene.json) or a Measure Lab session.
+# ARGS: extra flags, for example --no-server, --depth moge, --move-meter.
+run:
+	uv run python -m recon "$(BUNDLE)" --out "$(OUT)" $(ARGS)
+
+# ETH3D electro against its laser scan; needs `make recon-data` from experiments/evals (t3/evals).
+accept:
+	uv run python -m recon.eth3d electro

recon/README.md

@@ -0,0 +1,35 @@
+# recon: photos into a 3D model
+
+The reconstruction worker. It takes the app's scan bundle (keyframe photos, their AR poses and intrinsics, `scene.json`) and returns:
+
+1. `model.glb`: a colored mesh of the scanned area, in the meter's frame (origin at the meter's foot, y up, meters), viewable in macOS Quick Look;
+2. `geometry.json`: the fitted wall and ground for the rules engine, in feet, in the scene frame and in the meter's wall frame;
+3. `coverage.json`: which wall, ground, facing and overhead cells were actually seen, by a depth test against the reconstruction;
+4. `scene.json`: the scene updated with that geometry and coverage in the server's contract, posted to the placement server, with its answer in `result.json` and `site-plan.svg`.
+
+```sh
+cd recon
+uv sync
+make run BUNDLE=path/to/scan.zip OUT=path/to/out      # or a Measure Lab session, zip or folder
+make test                                             # unit tests and lint, as CI runs them
+make accept                                           # ETH3D acceptance (prepared data needed)
+```
+
+## Acceptance criteria, fixed before the first acceptance run
+
+- **Coverage on ETH3D electro's wall**: wall the worker calls observed where at least 10 cm of the band was seen by no photo, against the laser scan's visibility, must be at most **0.5 ft**. This is the bar and metric of experiments/evals README section 7 (t3/evals), where the app's own coverage map claimed 1.1 ft.
+- **Wall geometry on ETH3D electro**: reported in inches against the laser scan, with no pass bar set in advance. It covers the fitted wall plane's offset and angle, and the reconstruction's point at each pixel where a laser wall point is seen, as point-pair errors over 1 to 3 m spans, the evals' metric.
+- **End to end**: the ADVIO replay and the app's Simulator bundle each reach a placement-server result.
+
+## Results
+
+`make accept` on ETH3D electro ([results/eth3d_electro.md](results/eth3d_electro.md)):
+
+| Depth | Wall pair error, 1–3 m, median / p90 | Wall claimed | False-observed |
+| --- | --- | --- | --- |
+| Laser scan standing in for LiDAR | 0.5 / 1.8 in | 22.5 of 31.9 ft | 0.07 ft (passes) |
+| MoGe-2 rescaled with the poses | 0.7 / 1.5 in | 19.9 of 20.5 ft | 0.00 ft (passes) |
+
+Both paths fit the same wall; MoGe-2's stretch lies inside the laser path's. Each laser point is scored from the nearest photo in which the laser shows it unoccluded. At c6a8be3 the ADVIO replay and the app's Simulator bundle each reached a server result (`manual_review`: the ADVIO camera looks along its path, so little wall is seen head-on); they were not rerun after this round's geometry changes. The Simulator bundle needs `ARGS=--move-meter`, because its replayed wall has no surface behind it.
+
+The handoff to the server team, with everything measured and reusable, is [HANDOFF.md](HANDOFF.md).

recon/models/.gitignore

@@ -0,0 +1 @@
+.venv/

recon/models/moge_depth.py

@@ -0,0 +1,69 @@
+"""MoGe-2 metric depth for a list of upright photos, one process, one model load.
+
+Input: a JSON manifest, a list of {"image", "fx", "out"}: an upright RGB image, its focal length in
+pixels (MoGe takes only the horizontal field of view) and the .npz to write. Output per image:
+`depth` (float32 meters, z along the camera axis, NaN where MoGe's mask is off), at the image's
+own size. Images should be at most about 640 px on the long side: MoGe-2 peaked at 4.07 GB on
+24-megapixel photos and 2.97 GB at 640 x 480 on this Mac, and the worker must stay under 4 GB.
+
+The checkpoint is pinned by revision; the caches live under $HOUSE_SCANNING_DATA/evals, shared
+with the evals, so a machine that ran them does not download it again.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import os
+import sys
+from pathlib import Path
+
+DATA = Path(os.environ.get("HOUSE_SCANNING_DATA", Path.home() / "house-scanning-data"))
+os.environ.setdefault("HF_HOME", str(DATA / "evals" / "hf-cache"))
+os.environ.setdefault("TORCH_HOME", str(DATA / "evals" / "torch-cache"))
+
+REPO = "Ruicheng/moge-2-vitl-normal"
+REVISION = "cb0e8bbd6b1e243589717c78e750b1ba4c093acf"
+RESOLUTION_LEVEL = 9
+MPS_CAP_GB = 3.6
+
+
+def main(manifest: Path) -> None:
+    import cv2
+    import numpy as np
+    import torch
+    from moge.model.v2 import MoGeModel
+
+    device = (
+        "cuda"
+        if torch.cuda.is_available()
+        else "mps"
+        if torch.backends.mps.is_available()
+        else "cpu"
+    )
+    if device == "mps":
+        # A cap turns runaway growth into an error instead of swapping the shared machine.
+        torch.mps.set_per_process_memory_fraction(
+            MPS_CAP_GB * 1e9 / torch.mps.recommended_max_memory()
+        )
+    model = MoGeModel.from_pretrained(REPO, revision=REVISION).to(device).eval()
+    for item in json.loads(manifest.read_text()):
+        bgr = cv2.imread(item["image"], cv2.IMREAD_COLOR)
+        if bgr is None:
+            raise FileNotFoundError(item["image"])
+        w = bgr.shape[1]
+        rgb = torch.from_numpy(bgr[..., ::-1].copy()).to(device).permute(2, 0, 1).float() / 255
+        fov_x = math.degrees(2 * math.atan(w / (2 * item["fx"])))
+        with torch.no_grad():
+            out = model.infer(rgb, resolution_level=RESOLUTION_LEVEL, fov_x=fov_x, use_fp16=True)
+        depth = out["depth"].float().cpu().numpy()
+        depth[~out["mask"].cpu().numpy().astype(bool)] = np.nan
+        np.savez(item["out"], depth=depth.astype(np.float32))
+        print(f"{Path(item['image']).name}: median {np.nanmedian(depth):.2f} m", file=sys.stderr)
+        del out, rgb
+        if device == "mps":
+            torch.mps.empty_cache()
+
+
+if __name__ == "__main__":
+    main(Path(sys.argv[1]))

recon/models/pyproject.toml

@@ -0,0 +1,38 @@
+[project]
+name = "house-scanning-recon-depth"
+version = "0.1.0"
+description = "MoGe-2 depth for the reconstruction worker's photos-only path, run as its own process"
+requires-python = ">=3.12,<3.13"
+dependencies = [
+    "huggingface-hub>=0.34",
+    "moge",
+    "numpy>=2.1",
+    "opencv-python-headless>=4.10,<5",
+    "pillow>=10",
+    "torch>=2.7",
+    "torchvision>=0.22",
+]
+
+[tool.uv]
+package = false
+override-dependencies = ["opencv-python-headless>=4.10,<5", "opencv-python; sys_platform == 'never'"]
+
+# Pinned to the last commit before the MoGe-3 rewrite, whose packaging supports only Linux and
+# Windows. Its packaging lists demo and training extras as hard requirements; the inference path
+# needs only the list below.
+[tool.uv.sources]
+moge = { git = "https://github.com/microsoft/MoGe", rev = "07444410f1e33f402353b99d6ccd26bd31e469e8" }
+
+[[tool.uv.dependency-metadata]]
+name = "moge"
+version = "2.0.0"
+requires-dist = [
+    "huggingface-hub",
+    "numpy",
+    "opencv-python-headless",
+    "pillow",
+    "scipy",
+    "torch>=2.0.0",
+    "torchvision",
+    "utils3d @ git+https://github.com/EasternJournalist/utils3d.git@3fab839f0be9931dac7c8488eb0e1600c236e183",
+]