experiments: auto-detect lab for wall objects and the meter brand

experiments/autodetect/.gitignore

@@ -0,0 +1,2 @@
+.build/
+.swiftpm/

experiments/autodetect/.python-version

@@ -0,0 +1 @@
+3.12

experiments/autodetect/PLAN.md

@@ -0,0 +1,102 @@
+# Auto-proposed marks: recommendation, flow and integration plan
+
+Written for the experience design thread, which owns the confirm UI, and for S3 and Aiden, who own
+the app. The measurements are in [README.md](README.md) and `results/`. Line numbers refer to
+`origin/t3/ios-mvf`.
+
+## Recommendation
+
+1. **Ship the meter brand now.** Vision's text recognition already runs on the close-up, and a list
+   of meter makers turns its lines into a brand. It names the right maker on 17 of 19 photos where
+   the brand is printed in plain letters, and on 22 of 42 where it is part of a logo, such as
+   Tatung's round badge or GE's script monogram. It named nothing on the 9 photos without a brand,
+   and a wrong maker once, on a plate carrying two makers' names
+   ([meter_brand/results/brand.md](meter_brand/results/brand.md)). Show the brand pre-filled and
+   editable above the number candidates. It needs no model and no download.
+2. **Don't ship automatic wall-object proposals from public data.** No candidate passed. The best,
+   OWLv2, found 23% of Open Images windows at 55% precision, and it is a 308 MB model that takes
+   about half a second a frame on a Mac GPU. The on-device student (Create ML, 6.8 MB, 50 ms on the
+   Mac) found 7%. The public photos show whole buildings from far away. The app sees one wall from
+   1 to 3 m. Counting only large objects, the student's window AP50 rises from 9% to 38%, which
+   points at the data, not the model size.
+3. **Collect the app's own frames and train on those.** The packet already keeps every keyframe
+   with its pose. Aiden records walls with the app (about 30 to start; nothing yet says how many
+   are enough); OWLv2 pre-labels the keyframes offline (Apache-2.0, and this harness already runs
+   it); a person accepts or fixes each box; the Create ML student retrains on that, and this
+   harness scores it on held-out houses against the same criteria. That is the first run whose
+   result can settle the question, and the same labels give the first test of AC units, gas
+   meters, electrical boxes, garage doors and batteries, which no public set labels.
+4. **Lift boxes from their inner corners.** Lifting a true door box through the laser depth from
+   the midpoints of its sides is off by 0.59 ft p90; from the inner corner of each side, by
+   0.11 ft ([results/extent.md](results/extent.md)). Whatever detector lands, the lifting code
+   should use inner corners.
+5. **Leave the long D-FINE and Grounding DINO runs.** D-FINE would train about 10 hours and
+   Grounding DINO would label for about 5 hours on this shared Mac, both on the same off-view data
+   that limited the student. Neither would change recommendation 3.
+
+Not measured, and worth a device check once the flow exists: ARKit's mesh classes include window
+and door on LiDAR phones only, with no published outdoor accuracy, so they can at most add a cue.
+RoomPlan is documented for indoor rooms. On iOS 27 the Foundation Models framework accepts images,
+but Apple documents no box output, so it could at most verify a crop ("is this a gas meter?").
+
+## The flow once a detector passes
+
+The homeowner answers yes or no to what the phone found, and draws only what it missed.
+
+```
+Walk along the wall (unchanged)
+   │  the detector runs on each kept keyframe, off the main actor
+   │  a proposal appears once the same object lifts to the same wall stretch in 2 keyframes
+   │  it is drawn on the wall as a dashed outline with its kind; no prompt, no haptic,
+   │  so the walk is not interrupted
+   ↓
+"Anything else near your meter?" (MarkFeaturesScreen)
+   │  "We spotted 3 things on this wall", nearest the meter first, at most 6 rows
+   │  each row: crop from the sharpest keyframe with the box drawn · kind · "About 7 ft right"
+   │            [Yes]   [Not a window ▾]   (menu: another kind, or "Nothing there")
+   ├──→ Yes: the row turns solid; a window then asks "Does it open?" as today
+   ├──→ another kind: the kind changes and the row is confirmed
+   ├──→ Nothing there: the row goes away and is recorded as rejected
+   ├──→ tap the crop: the full photo with four edge handles; drag, Done; the box is lifted again
+   ├──→ Add something: today's tap-to-mark flow, unchanged
+   ↓
+"Looks complete": unanswered proposals are exported, never dropped
+```
+
+Why each choice:
+
+- **An unanswered proposal stays.** Dropping an unconfirmed gas meter would turn unsure into clear,
+  which AGENTS.md forbids. It goes to the server as `source: "vlm"` with its `conf`, so the server
+  applies the vision-model error bar.
+- **Adjust on a still photo, not in AR.** Dragging handles on a live view fights hand shake and
+  tracking. The crop is the image the extent was computed from, so an edit maps straight back.
+- **Yes is the big target; rejecting takes two taps.** "Nothing there" sits inside the kind menu.
+- **Dashed means proposed and solid means confirmed**, in AR and in the list, so the state never
+  rests on colour alone.
+- **Motion.** An outline fades in over 200 ms with a strong ease-out from 96% scale, and turns
+  solid over 150 ms when confirmed. With Reduce Motion, opacity only. Nothing bounces.
+- **VoiceOver** reads a row as "Window, about 7 feet right of the meter, proposed", with actions
+  Yes, Change kind and Nothing there. Targets are at least 44 pt.
+
+The meter close-up gains one line above the number candidates: the brand, with "Change". A brand
+not on the list shows "Brand not read" and a picker, and never blocks the step.
+
+## Where it goes in HouseScanKit and the app
+
+PR #52 (`t3/battery-mark`) adds `.battery` and `.elecBox` to `FeatureKind` and should merge first.
+`FeatureKind` still has no garage door, although `PacketMark.Kind` has `garageDoor`.
+
+| Step | Files | Change |
+|---|---|---|
+| 1. Brand | `HouseScan/Runtime/MeterReader/VisionMeterNumberReader.swift:11-70`, new `HouseScanKit/Meter/MeterBrand.swift`, `UI/Screens/MeterCloseUpScreen.swift` | Keep the recognised lines, which are discarded after ranking today; port `meter_brand/brands.py` with strict tests; show the brand. Ships alone. |
+| 2. Detector interface | new `HouseScanKit/Detect/WallObjectDetector.swift`; new `HouseScan/Runtime/Detect/CoreMLWallObjectDetector.swift` | `detect(jpeg:) -> [ImageBox]` in landscape sensor pixels, matching the intrinsics (AGENTS.md). The app target holds the model; the package stays model-free. A file-backed stub lets steps 3-7 ship before a model passes. |
+| 3. Lift to the wall | `HouseScanKit/Geometry/WallFrame.swift:214-257`, `Geometry/Camera.swift:65-107` | Box → rays through the inner corners → keyframe depth when present, else `WallFrame.intersectWall` → span, bottom, top. |
+| 4. Fuse keyframes | new `HouseScanKit/Detect/ProposalTracker.swift` | Match lifted boxes by kind and overlapping span; keep a proposal after 2 views; median edges and their spread. |
+| 5. Run on keyframes | `HouseScan/Runtime/KeyframeStore.swift:66-116`, `Runtime/ScanEngine.swift:524-550` | Detect in the detached task that already writes each keyframe; post results to the engine on the main actor. |
+| 6. State | `HouseScan/Contract/ScanContract.swift:406-449` | `MarkedFeature` gains its origin (tap or proposal with `conf`), a status (unanswered, confirmed, rejected), and the keyframe and box it came from. |
+| 7. Review UI | `UI/Screens/MarkFeaturesScreen.swift`, `UI/Camera/WallMarksOverlay.swift:46-71`, `UI/Copy/ScanCopy.swift:149-178` | The rows, outlines and copy above. |
+| 8. Export | `Runtime/ScanEngine+Export.swift:13-51`, `HouseScanKit/Scene/SceneExport.swift:407-434` | `source: "vlm"` and `conf` for proposals. `SceneDocument.Object` has no `conf` field yet. |
+| 9. Replay test | `Runtime/ReplayPlayer.swift`, `HouseScanKit/Replay/ReplaySession.swift` | Replay a recorded walk through steps 2-5 and check the proposals, without a phone. |
+
+Steps 1, 3 and 4 are pure logic with one correct answer, so they get strict unit tests; step 3 can
+reuse the ETH3D door numbers in `results/extent.md`.

experiments/autodetect/README.md

@@ -0,0 +1,45 @@
+# Autodetect: can the phone propose windows and doors?
+
+Today the homeowner draws each window box and taps its corners. This experiment asks which detector could propose windows and doors for the homeowner to confirm, how fast it runs, and whether a detected box, lifted onto the wall, is accurate enough to replace the taps.
+
+## Pass criteria (fixed before any scored run)
+
+1. **Proposals.** A candidate is good enough to propose window or door when, at its operating threshold, recall is at least 80% and precision at least 60% at IoU 0.5. The threshold is chosen per class on a 150-image Open Images tune set, then applied unchanged to each held-out set: 300 Open Images images and the 378 CMP Facade base images. The verdict is given per set. Open Images scoring follows its own rules for unverified classes and group-of boxes (`autodetect/oieval.py`). Both sets differ from the app's view: they show whole buildings from a distance, and CMP's facades are frontal and rectified. The app sees one wall at an angle from 1 to 3 m.
+2. **Speed class**, from Mac (M4 Pro) timings: live (at most 100 ms a frame), keyframe (at most 1 s), or offline (over 1 s).
+3. **3D extent.** A detected edge lifted to the wall replaces a tap when its p90 error is at most 0.5 ft. Ground truth is each door's or window's edges measured once in the ETH3D electro laser scan. Separately, the spread of one object's lifted edges across photos is reported.
+
+Prompts, thresholds and detector settings are in `autodetect/config.py`.
+
+## Rerun
+
+Needs about 300 MB in `~/house-scanning-data/autodetect/`, plus one model's weights at a time (up to 360 MB) and 230 MB of Create ML checkpoints while the student trains. The 3D step needs the ETH3D electro packet and scan from the evals lane. `results/resources.md` has the time and memory of each step.
+
+```sh
+uv sync && uv run pytest
+uv run python -m autodetect.openimages select && uv run python -m autodetect.openimages download
+uv run python -m autodetect.cmp
+swift build -c release --package-path vision   # copy rects, coremldet, trainod to ~/house-scanning-data/autodetect/bin
+uv run python -m autodetect.run_vision oi_tune oi_eval cmp electro
+uv run python -m autodetect.owl oi_tune oi_eval cmp electro      # needs weights/owlv2/model_fp16.onnx, then delete it
+uv run python -m autodetect.student prepare && uv run python -m autodetect.student train transfer 1000
+uv run python -m autodetect.student crop transfer                # tune set only; scaleFill won
+uv run python -m autodetect.student predict transfer scaleFill
+uv run python -m autodetect.score                                # results/proposals.md
+uv run python -m autodetect.extent_gt save && uv run python -m autodetect.extent   # results/extent.md
+```
+
+Written but not run to completion, for lack of time on the shared Mac: `autodetect.gdino` (Grounding DINO tiny) and `autodetect.dfine` (D-FINE small fine-tune). Each module's docstring says why.
+
+## Result
+
+No candidate passes, so the app should not propose walls' windows and doors from a model trained on public photos. [PLAN.md](PLAN.md) has the recommendation, the confirm flow and the integration plan.
+
+| Candidate | Size, Mac speed | Window P / R, Open Images | Door P / R, Open Images | Door edges lifted, p90 |
+|---|---|---|---|---|
+| OWLv2, zero-shot | 308 MB, 481 ms (GPU) | 55% / 23% | 75% / 58% | 1.62 ft |
+| Create ML student | 6.8 MB, 50 ms | 56% / 7% | 53% / 8% | no door matched |
+| Vision rectangles | in the OS, 12 ms | 6.5% / 15% | – | 1.18 ft |
+
+CMP Facade gives the same verdict (`results/proposals.md`). Lifting a true door box onto the laser-scanned wall is off by 0.11 ft p90 from the box's inner corners and 0.59 ft from its side midpoints, so the lifting step works and the boxes are what fail (`results/extent.md`). The public photos show whole buildings from far away; counting only large objects, the student's window AP50 rises from 9% to 38%.
+
+**Meter brand.** Vision's lines plus a list of meter makers (`meter_brand/`) name the brand on 17 of 19 photos where it is printed in plain letters and 22 of 42 where it is part of a logo, and name nothing on the 9 without one. No pass bar was set before this run. Details and the post-run changes are in `meter_brand/README.md`.

experiments/autodetect/autodetect/cmp.py

@@ -0,0 +1,77 @@
+"""CMP Facade base set (Tylecek and Sara, GCPR 2013; CC BY-SA) as a second, frontal eval set.
+
+Every facade element is annotated, so window and door count as verified in every image: an
+image with no door rectangle is a verified negative for door. There are no group-of boxes.
+
+In the XML, the two <x> values are the rectangle's vertical extent and the two <y> values its
+horizontal extent, both normalized with a top-left origin. Read as labelled, window rectangles
+cover 18% window pixels in the set's own label maps on average; swapped, they cover 87% (the
+rest is blinds and railings drawn over windows), and they sit on the windows by eye.
+
+The images are rectified frontal facades, often several storeys, shot from across a street.
+The app sees one wall from 1 to 3 m at an angle, so this set tests recognition on a different
+view from the product's.
+
+Usage: python -m autodetect.cmp    # DATA/cmp/gt.json and manifests/cmp_base.csv
+"""
+
+from __future__ import annotations
+
+import csv
+import json
+import re
+from pathlib import Path
+
+from PIL import Image
+
+from .paths import CMP, MANIFESTS
+
+URL = "https://cmp.felk.cvut.cz/~tylecr1/facade/CMP_facade_DB_base.zip"
+OBJECT = re.compile(r"<object>(.*?)</object>", re.S)
+
+
+def _field(block: str, tag: str) -> list[str]:
+    return [v.strip() for v in re.findall(rf"<{tag}>(.*?)</{tag}>", block, re.S)]
+
+
+def parse(xml_path: Path) -> list[dict]:
+    boxes = []
+    for block in OBJECT.findall(xml_path.read_text()):
+        (name,) = _field(block, "labelname")
+        if name not in ("window", "door"):
+            continue
+        rows = sorted(float(v) for v in _field(block, "x"))  # vertical, despite the tag
+        cols = sorted(float(v) for v in _field(block, "y"))  # horizontal
+        if len(rows) != 2 or len(cols) != 2:
+            raise ValueError(f"{xml_path.name}: expected 2 <x> and 2 <y> values, got {rows} {cols}")
+        boxes.append({"label": name, "box": [cols[0], rows[0], cols[1], rows[1]], "group": False})
+    return boxes
+
+
+def build() -> None:
+    gt = {}
+    for xml in sorted((CMP / "base").glob("cmp_b*.xml")):
+        image_id = xml.stem
+        boxes = parse(xml)
+        with Image.open(xml.with_suffix(".jpg")) as im:
+            w, h = im.size
+        labels = {b["label"] for b in boxes}
+        gt[image_id] = {
+            "w": w,
+            "h": h,
+            "verified": {c: int(c in labels) for c in ("window", "door")},
+            "boxes": boxes,
+        }
+    (CMP / "gt.json").write_text(json.dumps(gt))
+    with open(MANIFESTS / "cmp_base.csv", "w", newline="") as f:
+        w = csv.writer(f)
+        w.writerow(["image_id", "source_url", "license", "author"])
+        for image_id in gt:
+            w.writerow([image_id, URL, "CC BY-SA 3.0", "R. Tylecek, R. Sara, CMP Prague (collected from various sources)"])
+    n_w = sum(b["label"] == "window" for g in gt.values() for b in g["boxes"])
+    n_d = sum(b["label"] == "door" for g in gt.values() for b in g["boxes"])
+    print(f"{len(gt)} images, {n_w} windows, {n_d} doors, {sum(g['verified']['door'] for g in gt.values())} images with a door")
+
+
+if __name__ == "__main__":
+    build()

experiments/autodetect/autodetect/config.py

@@ -0,0 +1,73 @@
+"""Choices fixed before any scored run. Changing one after scoring means rerunning every candidate.
+
+Committed together with the pass criteria in README.md, before the first scored run.
+"""
+
+# Operating threshold: chosen per model and class on the OI tune set (150 images, disjoint from
+# eval), then applied unchanged to OI eval and CMP. Rule: the lowest score at which precision is
+# at least MIN_PRECISION, which gives the most recall that still meets the precision bar; when no
+# score reaches it, the score with the best F1. See oieval.choose_threshold.
+MIN_PRECISION = 0.60
+MIN_RECALL = 0.80
+IOU = 0.5
+
+# Diagnostic only, not a pass criterion: at 1 to 3 m an iPhone frame holds a window or door at
+# roughly 15% or more of the frame's width. Boxes smaller than this fraction of the image in
+# either dimension are treated as "difficult" (detections on them ignored, not counted as missed)
+# in the "near-sized" tables, and detections that small are dropped there (added after the first
+# scored run; see results/proposals.md).
+NEAR_MIN_SIDE = 0.10
+
+# OWLv2 text queries, one per wall object. Scored classes are window and door; the others are in
+# the query list so a door competes with "garage door" the way it would in the app. Each box
+# takes the query with the highest score.
+OWL_QUERIES = {
+    "window": "a photo of a window",
+    "door": "a photo of a door",
+    "garage_door": "a photo of a garage door",
+    "ac": "a photo of an air conditioner",
+    "gas_meter": "a photo of a gas meter",
+    "elec_box": "a photo of an electrical box",
+    "battery": "a photo of a home battery",
+}
+# Grounding DINO takes one caption with the same objects, each phrase ending in " .".
+GDINO_PHRASES = {
+    "window": "window",
+    "door": "door",
+    "garage_door": "garage door",
+    "ac": "air conditioner",
+    "gas_meter": "gas meter",
+    "elec_box": "electrical box",
+    "battery": "home battery",
+}
+
+# Post-processing for the open-vocabulary models: per-class NMS at this IoU, then the top
+# MAX_DETS boxes above MIN_SCORE per image. The low floor keeps the tail needed for AP. The
+# student gets the same per-image cap after its own NMS (set before it was scored).
+NMS_IOU = 0.5
+MAX_DETS = 100
+MIN_SCORE = 0.01
+
+# Apple Vision VNDetectRectanglesRequest. Defaults would return one rectangle at least 20% of the
+# image; these allow many small ones. Aspect ratio is short side over long side, so 0.2 admits a
+# door (about 0.4). Quadrature tolerance stays at Apple's default, 30 degrees.
+VISION_RECTS = {
+    "maximumObservations": 0,  # 0 = no limit
+    "minimumAspectRatio": 0.2,
+    "maximumAspectRatio": 1.0,
+    "minimumSize": 0.03,
+    "quadratureTolerance": 30.0,
+    "minimumConfidence": 0.0,
+}
+
+# Create ML student: transfer learning on Apple's object feature print, trained on the OI train
+# subset (window and door). Core ML confidence floor for scoring, and the NMS IoU in its pipeline.
+STUDENT_CONFIDENCE_FLOOR = 0.01
+STUDENT_NMS_IOU = 0.45
+
+# Speed classes from Mac timings (M4 Pro); an iPhone is slower, so these are optimistic.
+LIVE_MS = 100
+KEYFRAME_MS = 1000
+
+# 3D extent: an edge lifted to the wall replaces a tap when its p90 error is at most this.
+EXTENT_P90_FT = 0.5