From 465e5aad79f55a7cf301d96df3443ffb666ce113 Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 03:04:57 -0500 Subject: [PATCH 01/16] docs: trim AGENTS.md to what agents need and fix stale decisions AGENTS.md asked every agent to read all of docs/ (about 11,500 words) before any task, left out the HLD in docs/05, and kept an "open conflict" about an Expo app at apps/mobile/ that does not exist on main. It now maps each task to the docs it needs, records native Swift in ios/ as decided (merged in #2), adds "unseen is not clear" as a hard rule, lets rules.yaml label demo placeholders where no public value exists, and replaces the lane-ownership rule with coordinating on files an open PR is changing. docs/00 and docs/05 drop the same resolved Swift/Expo question, docs/00 states that LiDAR is optional and lists docs/05 in its doc map, and the README describes the experiments running now. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 51 +++++++++++++------------------ README.md | 6 ++-- docs/00-overview.md | 6 ++-- docs/05-live-guided-survey-hld.md | 5 +-- 4 files changed, 30 insertions(+), 38 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 941b545ad..2b3065383 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,45 +1,38 @@ -# House scanning: agent guide +# Agent guide -Hackathon project (4-person team, started 2026-09-25). Homeowners walk the outside of their house with an iPhone. -The AR session measures the wall, a server works out where a **Base Power battery** can be installed, and the spot -is shown back in AR. +A homeowner scans the outside wall around their electric meter with an iPhone. The server decides whether a Base Power battery fits there and where, and the app shows the spot in AR. Four-person hackathon, September 25 to 27, 2026. -## Read before doing anything non-trivial +## Where to look -1. `docs/00-overview.md`: goal, the core insight, decisions made, **corrections that supersede earlier drafts**, open questions. -2. `docs/01-feature-map.md`: lanes A–D, features with priorities, the `scene.json` data format, the **rules table (numbers still TBD)**. -3. `docs/02-implementation-plan.md`: how each part gets built, code snippets, schedule, risks. -4. `docs/03-stack-research.md`: which library or open-source project for each part, licenses, gotchas, what to skip. -5. `docs/04-prior-art-and-codes.md`: Base's public rules, competitors, NEC / IRC / Austin Energy citations, pitch angle. -6. `docs/eli5.html`: a visual explainer to open in a browser. Step 4 is an interactive demo of how the solver works. +Read the rows for your task, not all of `docs/`. + +| Task | Read | +| --- | --- | +| Goal, decisions, open questions | `docs/00-overview.md` | +| Capture app (lane A) | `docs/02-implementation-plan.md` "Lane A", `docs/05-live-guided-survey-hld.md` | +| Geometry and detection (lane B) | `docs/02` "Lane B", `docs/03-stack-research.md` | +| Placement solver and rules (lane C) | `docs/02` "Lane C", the rules table in `docs/01-feature-map.md`, citations in `docs/04-prior-art-and-codes.md` | +| Output, testing, pitch (lane D) | `docs/02` "Lane D", `docs/04` | +| Data sent from the app to the server | `scene.json` in `docs/01` | ## Hard rules -- **This repo is PUBLIC.** `private/` holds materials Base gave the team and is gitignored. Never commit, quote or summarize them - in tracked files: prompt text, prompt names, output field names, internal thresholds. Team notes go in `private/internal-notes.md`. -- **The placement decision is plain code, not AI.** Vision models only recognise things (object boxes, panel brand, meter text). - Every clearance number lives in `rules.yaml` with a code citation, never hardcoded. -- **Never run ML on Google Maps/Street View, Mapbox or Esri imagery.** Their terms forbid it. Use StratMap/NAIP. +- **This repository is public.** `private/` holds materials Base gave the team and is git-ignored. Never commit, quote or summarize them in tracked files: prompt text, prompt names, output field names, internal thresholds. Team notes go in `private/internal-notes.md`. +- **Plain code decides placement.** Vision models only recognize things (object boxes, panel brand, meter text). Every clearance number lives in `rules.yaml` with its source: a code citation, Base's public help page, or a labeled demo placeholder. Base's private values load at runtime from `private/`. +- **Unseen is not clear.** A check with no evidence for its area is UNSURE, never PASS. +- **Never run ML on Google Maps, Street View, Mapbox or Esri imagery.** Their terms forbid it. Use StratMap or NAIP. - **Place AR results relative to the meter's anchor**, not in raw world coordinates. Use `.gravity` world alignment, not `.gravityAndHeading`. - The camera image is landscape and the intrinsics match it. Rotate one and you must rotate the other. -## ⚠️ Open conflict: native Swift vs Expo - -The research plan (`docs/00-overview.md`) chose **native Swift + ARKit/RealityKit**. Expo has no first-class ARKit support, -and the capture needs LiDAR mesh export, per-frame position and lens data, and mesh raycasts. -On 2026-09-25 an Expo app was scaffolded at `apps/mobile/` (Expo SDK 57). **The team hasn't decided yet.** -Options: (a) drop Expo for native Swift; (b) Expo shell + a custom native Swift module for the whole AR capture; -(c) Expo for the non-AR screens only. Don't build AR capture in either place until the team picks one. - -`apps/mobile/AGENTS.md` has the Expo-specific rules for that folder. +## Decided -The setup pull request adds a native Swift skeleton at `ios/` as the recommended answer; its description gives the reasons. Remove this section once the team agrees. +The capture app is native Swift with ARKit and RealityKit in `ios/`, because Expo has no ARKit support. LiDAR is used when the phone has it and is never required. ## Working in this repo -- Layout: `ios/` is the capture app (native Swift, project generated by XcodeGen). `server/` is the Python server (uv). `web/` is a browser toolchain for the reviewer view and zero-install capture experiments. `experiments/` holds one folder per experiment. `sites/landing` is the landing page, a submodule; change it in its own repository. -- `make check` runs every suite. `make ios`, `make server` and `make web` run one each. `CONTRIBUTING.md` lists the matching CI checks. +- `ios/` is the capture app (project generated by XcodeGen). `server/` is the Python server (uv). `web/` is a browser toolchain for the reviewer view and zero-install capture experiments. `experiments/` holds one folder per experiment. `sites/landing` is the landing page, a submodule; change it in its own repository. +- `make check` runs every suite; `make ios`, `make server` and `make web` run one each. `CONTRIBUTING.md` lists the matching CI checks. - Branch from `main`, keep one writer per branch, and open a pull request. People merge; agents never push to `main` or merge. -- Teammates own lanes A–D (`docs/01-feature-map.md`). AI agents own setup, experiments and challenger reviews, and don't edit a lane's files without its owner's OK. +- Before editing files that an open pull request also changes, coordinate with its author. - Don't pick a signing team in Xcode's Signing & Capabilities pane: it writes into `project.pbxproj` and fails the CI drift check. Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` instead (see `ios/README.md`). - After editing `ios/project.yml`, run `make ios-project` (needs XcodeGen 2.46.0) and commit the regenerated project. diff --git a/README.md b/README.md index 7e445e764..2532611ce 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # House scanning -We're exploring a simpler way to survey a home for a Base Power battery. +We're exploring a simpler way to survey a home for a Base Power battery. A homeowner walks the outside of their house with a phone app. The app measures the wall around the electric meter, and our code decides whether a Base Power battery can be installed there and, if so, where. The homeowner sees the spot in AR before they put the phone away. @@ -29,8 +29,8 @@ Plain code makes the placement decision. Vision models only recognize things, su The plan targets iPhones, and a LiDAR iPhone makes some measurements easier. Most homeowners don't have one, so the experiments ask how far an ordinary phone gets: - Can AR taps on the ground and the wall measure the wall line, the room overhead, and the gap to a fence without LiDAR? -- Can plain phone video do the job instead? [OOOSplat](https://github.com/ooolabdev/ooosplat) turns a video into a 3D scene. We scale the scene from one known distance and check a second distance against a tape measure. -- How close does each method get? We tape-measure one real house and compare AR, a depth model on plain photos, and a vision model's guesses against the tape. +- Can learned 3D models build accurate geometry from ordinary photos? We measure MapAnything, Depth Anything 3 and MoGe against laser-scanned building walls, with and without one known distance. +- How far does phone AR tracking drift during a walk? We measure it on real outdoor iPhone walks with ground truth, then tape-measure one real house and compare every method against the tape. A convincing 3D view can still miss an obstacle or get a distance wrong, so every result keeps its original photos beside it. diff --git a/docs/00-overview.md b/docs/00-overview.md index b0cc8b4fc..e4a7c3de9 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -27,7 +27,7 @@ vision model. AR adds nothing there beyond better photos. Every check returns PASS / FAIL / UNSURE, with error bars on each measurement. See `02-implementation-plan.md`. - **The vision model only recognises things.** Gemini boxes → SAM 2 → cast onto the wall using where the phone was. Recognition prompts run on the close-ups. -- **Native Swift + ARKit/RealityKit on a LiDAR iPhone.** Not Expo (no first-class ARKit), and not Android for v1. +- **Native Swift + ARKit/RealityKit on iPhone, in `ios/`.** LiDAR is used when present, never required: most homeowners don't have it. Not Expo (no first-class ARKit), and not Android for v1. - **Normal dev app for the hackathon; an App Clip is only the pitch's rollout story** (see the gotchas in `03-stack-research.md`). ## Corrections made after the research (supersede earlier drafts) @@ -51,9 +51,6 @@ vision model. AR adds nothing there beyond better photos. - **Whether the install includes a separate battery disconnect box** beside the meter. Base's public pages mention a wall-mounted transfer switch and a battery disconnect. - **Test these in hours 0–4** (the unknowns everything depends on). See `03-stack-research.md` § Test first. -- **Native Swift vs Expo.** On 2026-09-25 an Expo app (SDK 57) was scaffolded at `apps/mobile/`, which conflicts with the - native-Swift decision above. Options: drop Expo; Expo shell + a native Swift module owning all AR capture; or Expo for the - non-AR screens only. Decide before anyone builds AR capture. ## Doc map @@ -64,4 +61,5 @@ vision model. AR adds nothing there beyond better photos. | `02-implementation-plan.md` | how to build each part, code snippets, 24 h schedule, risks | | `03-stack-research.md` | libraries and open-source projects per part, licenses, gotchas, what to skip | | `04-prior-art-and-codes.md` | what Base does publicly, competitors, electrical-code citations, pitch angle | +| `05-live-guided-survey-hld.md` | live guided survey: automatic capture, coverage display, next-view planning, evidence flow | | `eli5.html` | visual explainer for anyone new; step 4 is an interactive "slide the battery" demo | diff --git a/docs/05-live-guided-survey-hld.md b/docs/05-live-guided-survey-hld.md index c7ae5a3b9..4f940b74c 100644 --- a/docs/05-live-guided-survey-hld.md +++ b/docs/05-live-guided-survey-hld.md @@ -25,7 +25,8 @@ The repository baseline is documented in [the overview](00-overview.md), [featur | **Proposed** | Use native AR for stable spatial guidance, with LiDAR optional and a simpler camera-guidance fallback. | | **Proposed** | Keep immediate tracking and feedback on the phone; run heavier analysis and reconstruction asynchronously. | | **Experiment proposal** | Compare periodic MapAnything and stateful LingBot-Map reconstruction alongside the native AR geometry baseline. | -| **Open** | Resolve the documented native Swift/Expo integration conflict; select representative iPhones, GPU environment, and acceptable latency/cost. | +| **Decided** | Native Swift capture in `ios/`; LiDAR optional. | +| **Open** | Select representative iPhones, GPU environment, and acceptable latency/cost. | | **Open** | Confirm the versioned rules configuration and aggregation contract before expanding automated decisions. | ### Relationship to the existing repository plan @@ -344,7 +345,7 @@ When measurement uncertainty crosses a configured threshold, request a useful ad | [Unity AR Foundation](https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@6.1/manual/index.html) | Shared AR interfaces backed by platform provider plug-ins | Alternative if the team prefers Unity and cross-platform development; does not remove native capability differences. | | [Browser video frame callbacks](https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement/requestVideoFrameCallback) and [OpenCV.js camera processing](https://docs.opencv.org/4.11.0/dd/d00/tutorial_js_video_display.html) | Browser frame analysis and image processing | Useful fallback/prototype for camera coaching. A browser camera feed alone does not provide a persistent AR world map. | -**Platform scope:** preserve the iPhone target and resolve the native Swift/Expo boundary before capture implementation. The proposed direction is native AR capture, with camera-and-motion and optional-depth modes evaluated on actual test devices. Android and browser references are future options. Keep the backend independent of the mobile framework. +**Platform scope:** preserve the iPhone target. Capture is native Swift in `ios/`. The proposed direction is native AR capture, with camera-and-motion and optional-depth modes evaluated on actual test devices. Android and browser references are future options. Keep the backend independent of the mobile framework. ### Perception, reconstruction, and review From 9c9ceb2a608e9762bd002f20f1b979694bb0c92f Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 03:15:02 -0500 Subject: [PATCH 02/16] docs: treat exit code 137 as out of memory Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/AGENTS.md b/AGENTS.md index 2b3065383..7efc947c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,5 +34,6 @@ The capture app is native Swift with ARKit and RealityKit in `ios/`, because Exp - `make check` runs every suite; `make ios`, `make server` and `make web` run one each. `CONTRIBUTING.md` lists the matching CI checks. - Branch from `main`, keep one writer per branch, and open a pull request. People merge; agents never push to `main` or merge. - Before editing files that an open pull request also changes, coordinate with its author. +- Exit code 137 means the system killed the process, usually for memory. Find the allocation before rerunning; macOS swaps before it kills, so one runaway process can freeze the whole machine. - Don't pick a signing team in Xcode's Signing & Capabilities pane: it writes into `project.pbxproj` and fails the CI drift check. Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` instead (see `ios/README.md`). - After editing `ios/project.yml`, run `make ios-project` (needs XcodeGen 2.46.0) and commit the regenerated project. From ab03d7fd735884346f9182e3cb4cbf4aee60994a Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 05:09:39 -0500 Subject: [PATCH 03/16] docs: match rule sources across docs, restore lane ownership, drop the last open Swift/Expo line A fresh review of this PR found that AGENTS.md allowed three kinds of rule source while docs/01, docs/03 and docs/05 still required a code citation; all four now name the same three. It also restores the lane-ownership rule the trim had dropped, removes docs/05's remaining "Swift vs Expo is open" sentence, and names the ETH3D and ADVIO datasets in the README. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 10 +++++----- README.md | 4 ++-- docs/01-feature-map.md | 2 +- docs/03-stack-research.md | 2 +- docs/05-live-guided-survey-hld.md | 6 +++--- 5 files changed, 12 insertions(+), 12 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7efc947c1..feb3b3a81 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # Agent guide -A homeowner scans the outside wall around their electric meter with an iPhone. The server decides whether a Base Power battery fits there and where, and the app shows the spot in AR. Four-person hackathon, September 25 to 27, 2026. +A homeowner scans the outside wall around their electric meter with an iPhone. The server decides whether a Base Power battery fits there and where, and the app shows the spot in AR. Four-person hackathon, started 2026-09-25. ## Where to look @@ -18,7 +18,7 @@ Read the rows for your task, not all of `docs/`. ## Hard rules - **This repository is public.** `private/` holds materials Base gave the team and is git-ignored. Never commit, quote or summarize them in tracked files: prompt text, prompt names, output field names, internal thresholds. Team notes go in `private/internal-notes.md`. -- **Plain code decides placement.** Vision models only recognize things (object boxes, panel brand, meter text). Every clearance number lives in `rules.yaml` with its source: a code citation, Base's public help page, or a labeled demo placeholder. Base's private values load at runtime from `private/`. +- **Plain code decides placement.** Vision models only recognize things (object boxes, panel brand, meter text). Every clearance number lives in `server/rules.yaml` with its source: a code citation, Base's public help page, or a labeled demo placeholder. Values the team can't publish go in the git-ignored `private/rules.yaml`, which the server merges over the public file when it exists. - **Unseen is not clear.** A check with no evidence for its area is UNSURE, never PASS. - **Never run ML on Google Maps, Street View, Mapbox or Esri imagery.** Their terms forbid it. Use StratMap or NAIP. - **Place AR results relative to the meter's anchor**, not in raw world coordinates. Use `.gravity` world alignment, not `.gravityAndHeading`. @@ -26,14 +26,14 @@ Read the rows for your task, not all of `docs/`. ## Decided -The capture app is native Swift with ARKit and RealityKit in `ios/`, because Expo has no ARKit support. LiDAR is used when the phone has it and is never required. +The capture app is native Swift with ARKit and RealityKit in `ios/`, because Expo has no first-class ARKit support. LiDAR is used when the phone has it and is never required. ## Working in this repo - `ios/` is the capture app (project generated by XcodeGen). `server/` is the Python server (uv). `web/` is a browser toolchain for the reviewer view and zero-install capture experiments. `experiments/` holds one folder per experiment. `sites/landing` is the landing page, a submodule; change it in its own repository. - `make check` runs every suite; `make ios`, `make server` and `make web` run one each. `CONTRIBUTING.md` lists the matching CI checks. - Branch from `main`, keep one writer per branch, and open a pull request. People merge; agents never push to `main` or merge. -- Before editing files that an open pull request also changes, coordinate with its author. -- Exit code 137 means the system killed the process, usually for memory. Find the allocation before rerunning; macOS swaps before it kills, so one runaway process can freeze the whole machine. +- Teammates own lanes A–D (`docs/01-feature-map.md`). Agents edit a lane's files only with its owner's OK, and coordinate with the author before editing files an open pull request also changes. +- Several agents share one Mac. Exit code 137 means the system killed the process, usually for memory. Find the allocation before rerunning; macOS swaps before it kills, so one runaway process can freeze the whole machine. - Don't pick a signing team in Xcode's Signing & Capabilities pane: it writes into `project.pbxproj` and fails the CI drift check. Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` instead (see `ios/README.md`). - After editing `ios/project.yml`, run `make ios-project` (needs XcodeGen 2.46.0) and commit the regenerated project. diff --git a/README.md b/README.md index 2532611ce..0bcab1b28 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,8 @@ Plain code makes the placement decision. Vision models only recognize things, su The plan targets iPhones, and a LiDAR iPhone makes some measurements easier. Most homeowners don't have one, so the experiments ask how far an ordinary phone gets: - Can AR taps on the ground and the wall measure the wall line, the room overhead, and the gap to a fence without LiDAR? -- Can learned 3D models build accurate geometry from ordinary photos? We measure MapAnything, Depth Anything 3 and MoGe against laser-scanned building walls, with and without one known distance. -- How far does phone AR tracking drift during a walk? We measure it on real outdoor iPhone walks with ground truth, then tape-measure one real house and compare every method against the tape. +- Can learned 3D models build accurate geometry from ordinary photos? We measure MapAnything, Depth Anything 3 and MoGe against the laser-scanned buildings in the ETH3D dataset, with and without one known distance. +- How far does phone AR tracking drift during a walk? We measure it on the ADVIO dataset's iPhone walks, which have a ground-truth track, then tape-measure one real house and compare every method against the tape. A convincing 3D view can still miss an obstacle or get a distance wrong, so every result keeps its original photos beside it. diff --git a/docs/01-feature-map.md b/docs/01-feature-map.md index 3975bb195..a9b794f82 100644 --- a/docs/01-feature-map.md +++ b/docs/01-feature-map.md @@ -60,7 +60,7 @@ Every lane reads or writes this, so lock it first. Then all four people can work | # | Feature | Pri | Notes | |---|---|---|---| -| C1 | `rules.yaml`: every number, per-utility overrides (ComEd), a code citation per rule | P0 | No hardcoded distances in the code | +| C1 | `rules.yaml`: every number, per-utility overrides (ComEd), a source per rule: code citation, Base's public help page, or a labeled demo placeholder | P0 | No hardcoded distances in the code | | C2 | Sweep the 31″ × 22″ footprint along the unrolled wall, 2″ steps | P0 | | | C3 | Clearance checks: gas, driveway, pool, openings, box above, vent, AC, passage width, headroom, ground | P0 | PASS / FAIL / UNSURE with error bars | | C4 | Cable route along the wall: blocked by door, garage or a stretch with no wall; length ≤ max | P0 | | diff --git a/docs/03-stack-research.md b/docs/03-stack-research.md index 7b8451326..f78cb3f04 100644 --- a/docs/03-stack-research.md +++ b/docs/03-stack-research.md @@ -42,7 +42,7 @@ PHONE: RealityKit box attached to the meter's anchor + cable line | Apple Vision on the phone + zxing | meter number, CL320 | Instant retake; serial-number OCR is the weak spot (63% vs 97%) | | Overture + osmnx | which wall faces the street (flag for homeowner approval of the look) | The compass alone can't tell | | samgeo on StratMap/NAIP | pool, driveway | OSM tags them on < 5% of homes; imagery ML is allowed to run on | -| rules.yaml + citations | all rules | Each "no" cites NEC / IRC / Austin Energy / Texas Gas Service | +| rules.yaml + citations | all rules | Each "no" cites its source: NEC / IRC / Austin Energy / Texas Gas Service, Base's public help page, or a labeled demo placeholder | | Rerun | debugging | Capture + solver reasoning in one timeline | | Depth Anything 3 | pitch | The "AR beats photos" comparison | diff --git a/docs/05-live-guided-survey-hld.md b/docs/05-live-guided-survey-hld.md index 4f940b74c..cb0a73d16 100644 --- a/docs/05-live-guided-survey-hld.md +++ b/docs/05-live-guided-survey-hld.md @@ -22,7 +22,7 @@ The repository baseline is documented in [the overview](00-overview.md), [featur | **Product alignment** | Analyze the live view to understand relevant objects, observed areas, and missing evidence. | | **Product alignment** | Show the customer where to move and where to aim, updating guidance during capture. | | **Product alignment** | Upload selected images during the session and progressively update reconstruction and assessment. | -| **Proposed** | Use native AR for stable spatial guidance, with LiDAR optional and a simpler camera-guidance fallback. | +| **Proposed** | Use native AR for stable spatial guidance, with a simpler camera-guidance fallback. | | **Proposed** | Keep immediate tracking and feedback on the phone; run heavier analysis and reconstruction asynchronously. | | **Experiment proposal** | Compare periodic MapAnything and stateful LingBot-Map reconstruction alongside the native AR geometry baseline. | | **Decided** | Native Swift capture in `ios/`; LiDAR optional. | @@ -31,7 +31,7 @@ The repository baseline is documented in [the overview](00-overview.md), [featur ### Relationship to the existing repository plan -The current plan targets an iPhone demo with native Swift, ARKit/RealityKit, and LiDAR geometry where supported. It uses native AR measurements and a deterministic server-side placement solver, retaining tap-to-mark capture as a dependable fallback. Its Swift-versus-Expo integration question remains open in [AGENTS.md](../AGENTS.md). The existing implementation plan remains the baseline until the team adopts a change. +The current plan targets an iPhone demo with native Swift, ARKit/RealityKit, and LiDAR geometry where supported. It uses native AR measurements and a deterministic server-side placement solver, retaining tap-to-mark capture as a dependable fallback. Capture is native Swift in `ios/`, and LiDAR is optional. The existing implementation plan remains the baseline until the team adopts a change. | Existing baseline | Proposal in this HLD | |---|---| @@ -298,7 +298,7 @@ Perception extracts observable facts. Geometry supplies measured or estimated sp | Responsibility | Contract | |---|---| -| Rule configuration | Keep every clearance and equipment dimension in `rules.yaml`, with the required code citation and applicability metadata; never hardcode them in capture logic. | +| Rule configuration | Keep every clearance and equipment dimension in `rules.yaml`, with its source (a code citation, Base's public help page, or a labeled demo placeholder) and applicability metadata; never hardcode them in capture logic. | | Perception | Associate recognized objects and text with original images and an uncertainty estimate. | | Geometry | Keep coordinate frame, units, scale provenance, error bounds, and evidence references with each measurement. | | Solver | Evaluate the configured rules using the repository's wall-line and polygon approach; expose which observations support each check. | From d79e0e9c97324e6d3adf51b6b8a810d63f1281d3 Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 11:31:13 -0500 Subject: [PATCH 04/16] docs: rebuild onboarding around the client and server team split - Rewrite AGENTS.md around team ownership, a where-to-look table and rules that each state their reason. Lane A-D ownership and the Swift/Expo history are gone. - Make README.md a one-screen repository map and docs/00 the current system overview, including the open 3D path options: traditional reconstruction with LiDAR, learned depth with the phone's poses, and world models, which are not evaluated yet. - Replace docs/eli5.html with a picture-led explainer of the two-team pipeline and the 3D evidence so far. The old slide-the-battery demo is dropped: it showed only pass or fail, with no error bars or unseen areas, which contradicts how the server decides. - Mark docs/01-05 as the day-1 plan and remove two lines that ruled out 3D reconstruction. - Correct the morning briefing's Finding 2, which ruled out 3D modeling, and add a note about the team split. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 56 +- README.md | 59 +- docs/00-overview.md | 150 +-- docs/01-feature-map.md | 4 +- docs/02-implementation-plan.md | 2 + docs/03-stack-research.md | 4 +- docs/04-prior-art-and-codes.md | 2 + docs/05-live-guided-survey-hld.md | 2 + docs/briefings/2026-09-26-morning.html | 3 +- docs/eli5.html | 1286 +++++++++++++----------- 10 files changed, 847 insertions(+), 721 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index feb3b3a81..5313778c6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,39 +1,45 @@ # Agent guide -A homeowner scans the outside wall around their electric meter with an iPhone. The server decides whether a Base Power battery fits there and where, and the app shows the spot in AR. Four-person hackathon, started 2026-09-25. +A homeowner scans the outside of their house around the electric meter with an iPhone. The system works out whether a Base Power battery fits there and where, and the app shows the spot in AR. It is a hackathon project that started on 2026-09-25. The MVP is three problems: a phone that guides the homeowner until everything the model needs is captured, photos and their metadata turned into a 3D model, and that model evaluated against the placement criteria. + +## Who owns what + +- **Client team.** Sam, working with AI agents, owns the iOS app in `ios/`: guided capture, the live on-device 3D map with fog of war, and the capture packet it sends to the server. Aiden films video and gathers sample datasets. +- **Server team.** Hunter, working with his own agents, owns everything after the packet: building a 3D model or point cloud, then evaluating the criteria on it. + +Edit the other team's files only with that team's OK, and talk to the author before editing files that an open pull request also changes. Two writers on one file cost more time than asking does. ## Where to look -Read the rows for your task, not all of `docs/`. +Read the row for your task. Files marked with a pull request exist only on that branch until it merges. | Task | Read | | --- | --- | -| Goal, decisions, open questions | `docs/00-overview.md` | -| Capture app (lane A) | `docs/02-implementation-plan.md` "Lane A", `docs/05-live-guided-survey-hld.md` | -| Geometry and detection (lane B) | `docs/02` "Lane B", `docs/03-stack-research.md` | -| Placement solver and rules (lane C) | `docs/02` "Lane C", the rules table in `docs/01-feature-map.md`, citations in `docs/04-prior-art-and-codes.md` | -| Output, testing, pitch (lane D) | `docs/02` "Lane D", `docs/04` | -| Data sent from the app to the server | `scene.json` in `docs/01` | - -## Hard rules +| What the system is, what's decided, what's open | `docs/00-overview.md` | +| The iOS app | `ios/README.md` | +| The capture packet the app sends | `packet/README.md` (coming on branch `t3/packet`) | +| The rules engine, API and capture contract | `server/README.md` (PR #11), especially "What settles each check" | +| Turning photos into a 3D model | `recon/HANDOFF.md` (PR #20) | +| Accuracy evals on public datasets | `experiments/evals/README.md` (PR #12) | +| Branches, checks and TestFlight | `CONTRIBUTING.md` | +| Rule citations and day-1 reasoning | `docs/01` to `05`; `docs/00` and the component READMEs win where they differ | -- **This repository is public.** `private/` holds materials Base gave the team and is git-ignored. Never commit, quote or summarize them in tracked files: prompt text, prompt names, output field names, internal thresholds. Team notes go in `private/internal-notes.md`. -- **Plain code decides placement.** Vision models only recognize things (object boxes, panel brand, meter text). Every clearance number lives in `server/rules.yaml` with its source: a code citation, Base's public help page, or a labeled demo placeholder. Values the team can't publish go in the git-ignored `private/rules.yaml`, which the server merges over the public file when it exists. -- **Unseen is not clear.** A check with no evidence for its area is UNSURE, never PASS. -- **Never run ML on Google Maps, Street View, Mapbox or Esri imagery.** Their terms forbid it. Use StratMap or NAIP. -- **Place AR results relative to the meter's anchor**, not in raw world coordinates. Use `.gravity` world alignment, not `.gravityAndHeading`. -- The camera image is landscape and the intrinsics match it. Rotate one and you must rotate the other. +## Rules -## Decided +- **Keep Base's materials in `private/`, which git ignores.** The repository is public. Don't commit, quote or summarize those materials in tracked files, including prompt text, prompt names, output field names and internal thresholds. Team notes go in `private/internal-notes.md`. Real captures, photos of homes and dataset images go in `captures/`, `fixtures/real/` or `data/`, which git also ignores. +- **Make placement decisions in deterministic code.** Models build the geometry and recognize things such as the meter's text or an object's outline. Plain code over that geometry makes the pass or fail decision, so every answer traces back to a rule and a measurement. Clearance numbers live in rules files with their sources, never in code: public values with citations in `server/rules.yaml` (PR #11), and Base's values in the git-ignored `private/rules.yaml`, which the server merges over them. +- **Treat unseen as unsure, never as clear.** A gap in the scan could hide a gas meter. When a check depends on an area nobody observed, return UNSURE and name the view that would settle it. +- **Use StratMap or NAIP for aerial imagery.** Google Maps, Street View, Mapbox and Esri forbid running ML on their imagery in their terms. +- **Place AR results relative to the meter's anchor, with `.gravity` world alignment.** ARKit corrects anchors as tracking improves, so a result tied to the meter moves with it while raw world coordinates drift. `.gravityAndHeading` depends on the compass, which is unreliable next to a house. +- **Rotate the intrinsics whenever you rotate a camera image.** Camera images are landscape sensor images and the intrinsics match that orientation. A rotated image with unrotated intrinsics produces wrong 3D geometry without any error. -The capture app is native Swift with ARKit and RealityKit in `ios/`, because Expo has no first-class ARKit support. LiDAR is used when the phone has it and is never required. +Decided: the app is native iOS only (Swift, ARKit, RealityKit), because Expo has no first-class ARKit support. It uses LiDAR and other depth sensors when present and never requires them, since most homeowners' phones lack LiDAR. -## Working in this repo +## Working in the repository -- `ios/` is the capture app (project generated by XcodeGen). `server/` is the Python server (uv). `web/` is a browser toolchain for the reviewer view and zero-install capture experiments. `experiments/` holds one folder per experiment. `sites/landing` is the landing page, a submodule; change it in its own repository. - `make check` runs every suite; `make ios`, `make server` and `make web` run one each. `CONTRIBUTING.md` lists the matching CI checks. -- Branch from `main`, keep one writer per branch, and open a pull request. People merge; agents never push to `main` or merge. -- Teammates own lanes A–D (`docs/01-feature-map.md`). Agents edit a lane's files only with its owner's OK, and coordinate with the author before editing files an open pull request also changes. -- Several agents share one Mac. Exit code 137 means the system killed the process, usually for memory. Find the allocation before rerunning; macOS swaps before it kills, so one runaway process can freeze the whole machine. -- Don't pick a signing team in Xcode's Signing & Capabilities pane: it writes into `project.pbxproj` and fails the CI drift check. Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` instead (see `ios/README.md`). -- After editing `ios/project.yml`, run `make ios-project` (needs XcodeGen 2.46.0) and commit the regenerated project. +- Branch from `main`, keep one writer per branch, and open a pull request. People merge. Agents never push to `main` or merge, so a person sees every change before it lands. +- Several agents share one Mac. Exit code 137 means the system killed the process, usually for memory. Find the large allocation before rerunning, because macOS swaps before it kills and one runaway process can freeze the whole machine. +- Set your signing team in `ios/Config/Local.xcconfig` (copy `Local.xcconfig.example`; see `ios/README.md`), not in Xcode's Signing & Capabilities pane. The pane writes into `project.pbxproj`, and CI fails on that drift. +- After editing `ios/project.yml`, run `make ios-project` (it needs XcodeGen 2.46.0) and commit the regenerated project, because CI regenerates it and fails on any difference. +- `sites/landing` is the landing page, a submodule. Change it in its own repository. diff --git a/README.md b/README.md index 0bcab1b28..10a5c135e 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,30 @@ # House scanning -We're exploring a simpler way to survey a home for a Base Power battery. +A homeowner scans the outside of their house around the electric meter with an iPhone. The system works out whether a Base Power battery fits there and where, and the app shows the spot in AR. It is a hackathon project for Base Power. -A homeowner walks the outside of their house with a phone app. The app measures the wall around the electric meter, and our code decides whether a Base Power battery can be installed there and, if so, where. The homeowner sees the spot in AR before they put the phone away. +The work splits in two. The client team builds the iPhone app, which guides the scan with a live 3D map and sends a capture packet. The server team turns the packet into a 3D model and evaluates the placement rules on it. [docs/00-overview.md](docs/00-overview.md) explains the whole system. -This is a four-person hackathon project for Base Power. Success means one capture session gives Base enough to decide, with no follow-up photos. A person can still review the result. +## Repository map -## Why measure instead of photograph +| Path | What it is | State on 2026-09-26 | +| --- | --- | --- | +| `ios/` | The native iPhone app (Swift, ARKit, RealityKit) | On `main`: an AR session skeleton. Guided capture is in PR #10; the live 3D map is on branch `t3/ios-map3d`, stacked on it | +| `packet/` | The capture packet: photos, poses, intrinsics, motion data, LiDAR depth and mesh, the homeowner's marks | Being specified on branch `t3/packet` | +| `server/` | The deterministic rules engine and API | On `main`: a skeleton. The engine is in PR #11 and runs as a public demo | +| `recon/` | A worker that turns photos and depth into a 3D model and a coverage map | Draft PR #20, handed to the server team | +| `experiments/` | One folder per experiment | Accuracy evals on public datasets in PR #12 | +| `web/` | Browser toolchain | The review page (PR #15) is parked | +| `docs/` | The overview, a visual explainer, the day-1 plan and dated briefings | | +| `sites/landing` | The landing page, a submodule | Change it in its own repository | -Base's site survey starts from a few customer photos, and three things go wrong with them: +## What's live -- Distances are estimated by eye, so borderline clearances go to a reviewer. -- Unordered photos can't show whether two things are on the same wall or how far apart they are. -- Nobody knows what sits just outside the frame, so the homeowner gets asked for more photos. +- The public demo server, with public rules only: https://house-scanning-server.vercel.app. A second deployment loads Base's rules and requires a key. +- A TestFlight workflow, started by hand. See [CONTRIBUTING.md](CONTRIBUTING.md). -An AR session tracks the phone's position as it moves. The app can measure the wall in feet, and it knows which stretches nobody has looked at yet. It asks for those while the homeowner is still standing there. +## Where to start -## How it works +- New to the project: [docs/eli5.html](docs/eli5.html), a picture-first explainer. Open it in a browser. +- Contributors and coding agents: [AGENTS.md](AGENTS.md), then [CONTRIBUTING.md](CONTRIBUTING.md). -1. The homeowner marks the electric meter, walks the wall to its end in each direction, and takes close-ups of the meter and the electrical panel. The app saves frames along with where the camera was for each one. -2. A server lays the walls out as one line, measured in feet from the meter. It slides the battery's footprint along that line and checks each clearance: gas equipment, driveways, doors and windows, room overhead, and the gap to a facing fence. Each check returns pass, fail, or unsure, with the measurement error counted. -3. The app shows the chosen spot, the battery, and the cable route from the meter in AR. - -Plain code makes the placement decision. Vision models only recognize things, such as where a gas meter is or what the meter label says. Several clearance numbers are not final. Base still has to confirm the rules for driveways, pools, the fence gap, and cable length. - -## What we're testing now - -The plan targets iPhones, and a LiDAR iPhone makes some measurements easier. Most homeowners don't have one, so the experiments ask how far an ordinary phone gets: - -- Can AR taps on the ground and the wall measure the wall line, the room overhead, and the gap to a fence without LiDAR? -- Can learned 3D models build accurate geometry from ordinary photos? We measure MapAnything, Depth Anything 3 and MoGe against the laser-scanned buildings in the ETH3D dataset, with and without one known distance. -- How far does phone AR tracking drift during a walk? We measure it on the ADVIO dataset's iPhone walks, which have a ground-truth track, then tape-measure one real house and compare every method against the tape. - -A convincing 3D view can still miss an obstacle or get a distance wrong, so every result keeps its original photos beside it. - -## Read next - -| File | What it covers | -| --- | --- | -| [docs/00-overview.md](docs/00-overview.md) | The goal, decisions made so far, and open questions | -| [docs/01-feature-map.md](docs/01-feature-map.md) | The four work lanes, features by priority, the `scene.json` format, and the rules table | -| [docs/02-implementation-plan.md](docs/02-implementation-plan.md) | How each part gets built, the schedule, and risks | -| [docs/03-stack-research.md](docs/03-stack-research.md) | The open-source projects and libraries we build on, and the ones we skip | -| [docs/04-prior-art-and-codes.md](docs/04-prior-art-and-codes.md) | Base's public rules, similar products, and electrical-code citations | -| [docs/05-live-guided-survey-hld.md](docs/05-live-guided-survey-hld.md) | Proposed live guidance and automatic capture architecture, diagrams, evidence flow, and technical resources | -| [docs/eli5.html](docs/eli5.html) | A visual walkthrough with an interactive demo of the placement search | - -Contributors and coding agents start with [AGENTS.md](AGENTS.md). This repository is public, so materials Base gave the team stay in the git-ignored `private/` folder. +This repository is public. Materials Base gave the team stay in the git-ignored `private/` folder. diff --git a/docs/00-overview.md b/docs/00-overview.md index e4a7c3de9..0d13be4a7 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -1,65 +1,85 @@ -# House scanning: overview and decisions - -Hackathon project, 4-person team, started 2026-09-25. The goal: from a homeowner's phone, work out -**where a Base Power battery can be installed** against the outside of the house, and show the spot -back to them in AR. - -> **This repo is public.** Materials Base gave the team live in `private/`, which is gitignored. Never commit, -> quote or summarize them here. Team-only notes are in `private/internal-notes.md`. - -## The core insight (why AR) - -Today homeowners submit photos and reviewers judge them. Three problems come up that ordinary photos can't solve: -1. **Measuring distance.** A single photo has no real scale, so gaps and clearances get estimated by eye, and the borderline cases go to a person. -2. **Fitting photos together.** A handful of unordered photos can't reliably say whether two things are on the same wall, or how far apart they are. -3. **Coverage.** If part of the wall wasn't photographed, nobody can say what's there, so the homeowner gets asked for more photos. - -AR fixes all three. The recognition half (panel brand, meter text, rust, whether a window opens) stays with a -vision model. AR adds nothing there beyond better photos. - -## Decisions made - -- **AR-guided capture, not "upload a video and process it later."** Offline photogrammetry (COLMAP, splats) fails on - blank siding and stucco, has no real scale, and is slow. ARKit already fuses the camera with the motion sensors on the phone. -- **The phone collects data; the server decides.** Swift app → zip upload → Python FastAPI server → result → AR preview. -- **The placement code is plain code, not AI.** Unroll the house walls into one straight line `s` (feet along the wall - from the meter). Most rules become 1-D interval checks plus 2-D polygon distances. Sweep the battery footprint in small steps. - Every check returns PASS / FAIL / UNSURE, with error bars on each measurement. See `02-implementation-plan.md`. -- **The vision model only recognises things.** Gemini boxes → SAM 2 → cast onto the wall using where the phone was. - Recognition prompts run on the close-ups. -- **Native Swift + ARKit/RealityKit on iPhone, in `ios/`.** LiDAR is used when present, never required: most homeowners don't have it. Not Expo (no first-class ARKit), and not Android for v1. -- **Normal dev app for the hackathon; an App Clip is only the pitch's rollout story** (see the gotchas in `03-stack-research.md`). - -## Corrections made after the research (supersede earlier drafts) - -1. Use `worldAlignment = .gravity`, **not** `.gravityAndHeading`. The compass is unreliable near a house - (developers report errors up to ~176°). Take one compass reading away from the meter as a rough hint. Which wall faces the - street comes from map data: Overture building outline + osmnx nearest road. -2. **Place the battery relative to the meter's anchor**, not in raw world coordinates. ARKit corrects anchors as it goes, - and drift is about 2 cm per second. -3. The battery is **39.5″ tall** (Base Core: 39.5 × 30.68 × 22 in), not 36″. Base's older units are 3 × 3 ft, - so keep all dimensions in the rules config. -4. Claude is **not** the box detector; its docs say its coordinates are approximate. Gemini returns boxes natively as - `[ymin, xmin, ymax, xmax]` on 0–1000. -5. **Never run ML on Google, Mapbox or Esri imagery.** Google's ToS §3.2.3(c) bans it outright. Use StratMap/NAIP (public domain). - -## Open questions (team must decide) - -- **Which clearance numbers to use.** The public baseline is Base's help page: within 20 ft of the meter, within 1 ft of the wall, - and 3 ft from gas meters, fences, AC units and other batteries. Other numbers (driveway, pool, passage width, max cable run) are - still TBD. The team's working notes are in `private/internal-notes.md`. -- **Whether the install includes a separate battery disconnect box** beside the meter. Base's public pages mention a wall-mounted - transfer switch and a battery disconnect. -- **Test these in hours 0–4** (the unknowns everything depends on). See `03-stack-research.md` § Test first. - -## Doc map - -| File | What | -|---|---| -| `00-overview.md` | this file: goal, insight, decisions | -| `01-feature-map.md` | lanes A–D, feature list with priorities, data format, rules table | -| `02-implementation-plan.md` | how to build each part, code snippets, 24 h schedule, risks | -| `03-stack-research.md` | libraries and open-source projects per part, licenses, gotchas, what to skip | -| `04-prior-art-and-codes.md` | what Base does publicly, competitors, electrical-code citations, pitch angle | -| `05-live-guided-survey-hld.md` | live guided survey: automatic capture, coverage display, next-view planning, evidence flow | -| `eli5.html` | visual explainer for anyone new; step 4 is an interactive "slide the battery" demo | +# House scanning: system overview + +Current as of 2026-09-26 midday. Where this file and the day-1 plan (`01` to `05`) disagree, this file wins. Where this file and a component's README disagree, the README wins. + +## The problem + +Base Power installs a home battery on the outside wall, close to the electric meter, because a cable runs from the meter to the battery. The battery is 31 in wide, 22 in deep and 39.5 in tall. Whether it fits depends on clearance rules: distance from the gas meter, from doors and windows, from AC units and fences, clear space in front, headroom above, and cable length from the meter. Today someone judges that from a few customer photos, and three things go wrong: + +- A photo has no scale, so distances are guessed and borderline cases go to a reviewer. +- Loose photos can't show whether two things are on the same wall or how far apart they are. +- Nobody knows what sits just outside the frame, so the homeowner gets asked for more photos. + +## The product + +A homeowner scans the outside of the house around the meter with an iPhone. The system works out whether the battery fits and where, and the app shows the spot in AR on the real wall. One scan should give Base enough to decide without follow-up photos. + +The MVP is three problems: + +1. A phone with a live camera feed that routes the homeowner until everything the model needs is captured. +2. Photos and their metadata turned into a 3D model. +3. The 3D model evaluated against the placement criteria. + +## How it fits together + +``` + client team (iPhone) server team + ┌──────────────────────────┐ capture packet ┌──────────────────────────┐ + │ guided scan │ ─────────────────▶ │ 3D model or point cloud │ + │ live 3D map, fog of war │ │ │ │ + │ │ │ ▼ │ + │ battery spot in AR │ ◀───────────────── │ criteria: PASS/FAIL/ │ + │ (placed from the meter) │ answer │ UNSURE per check │ + └──────────────────────────┘ └──────────────────────────┘ +``` + +**Client team.** Sam, working with AI agents, builds the iOS app. Aiden films video and gathers sample datasets. The app guides capture with a live on-device 3D map, and a haze ("fog of war") covers what the phone hasn't seen yet. Coverage aims at 100% of what the model needs, not the whole house. The app uses LiDAR and other depth sensors first when the phone has them, and the path without LiDAR is built alongside. Guided capture is in PR #10; the live 3D map is on branch `t3/ios-map3d`, stacked on it. + +The app's output is the **capture packet**: photos, camera poses (where the camera was and which way it pointed), intrinsics (the lens's focal length and center), motion-sensor (IMU) readings, LiDAR depth and mesh when present, and the homeowner's marks such as the meter. It is being specified on branch `t3/packet`. + +**Server team.** Hunter, working with his own agents, takes the packet and builds a 3D model or point cloud, then evaluates the criteria on it. They are comparing world models with traditional and newer reconstruction methods: first broadly, then in depth on the most promising paths. Their starting points: + +- `server/` (PR #11): the deterministic rules engine and API. It runs as a public demo at https://house-scanning-server.vercel.app, and a second deployment loads Base's rules behind a key. +- `recon/` (draft PR #20): a working reconstruction worker. It fuses depth into a 3D model and tests coverage against it. +- `experiments/evals/` (PR #12): accuracy evals on public datasets with laser-scanned ground truth. + +## Decided + +- **Native iOS only**, in `ios/`: Swift, ARKit and RealityKit. Expo has no first-class ARKit support. +- **LiDAR first, never required.** Use depth sensors when present; most homeowners' phones lack LiDAR, so every check must also work without it. +- **Deterministic placement decisions.** Models build the geometry and recognize things. The pass or fail decision is plain code over that geometry, with each clearance number and its source in a rules file. +- **Error bars decide.** A check passes only when its margin beats the measurement error, fails only when it misses by more than the error, and is UNSURE otherwise. +- **Unseen is never clear.** A check whose area nobody observed is UNSURE, and the server names the view that would settle it. +- **AR results are placed relative to the meter's anchor**, with `.gravity` world alignment. + +## What the evidence says so far + +All numbers come from public datasets with laser-scanned ground truth. Sources are `experiments/evals/README.md` (PR #12) and `recon/HANDOFF.md` (PR #20). + +- A depth model's own scale is 4 to 12% wrong. +- Rescaled with the phone's poses, learned depth gets walls to about 5 in at p90 (9 in 10 points are closer) at a realistic phone error. Edges come out at 8 in or worse, and clearances are measured from edges. +- The recon worker on the ETH3D building gets walls to p90 2.1 in with a laser scan standing in for LiDAR, and 2.8 in from photos only. +- Coverage tested against depth claims at most about 0.46 ft of wall that no photo saw. +- A current iPhone's tracking stays within the server's error allowance. +- World models have not been evaluated yet. + +So learned depth alone is too rough for edge clearances. 3D modeling stays central, with LiDAR and the phone's poses supplying metric scale. + +## Open + +- **Which 3D path.** Candidates are traditional reconstruction fused with LiDAR, learned depth rescaled with the phone's poses, and world models (large learned models that produce a whole 3D scene from photos or video). The server team decides after evaluating them. Whichever path wins needs metric scale, edges within the error allowance, and an honest account of which parts it observed and which it filled in. +- **Real-phone accuracy.** The numbers above come from public datasets. A field session with a tape survey on a real wall measures our own phone. +- **Base's rules.** The public demo runs on public values plus labeled placeholders. Base's own values load only from the private rules file. + +## Where each component's truth lives + +| Component | Source of truth | +| --- | --- | +| iOS app | `ios/README.md` | +| Capture packet | `packet/README.md` (coming on branch `t3/packet`) | +| Rules engine, API, capture contract | `server/README.md` (PR #11), "What settles each check" | +| Reconstruction worker | `recon/HANDOFF.md` (PR #20) | +| Evals | `experiments/evals/README.md` (PR #12) | +| Agent rules | `AGENTS.md` | + +The day-1 plan in `01` to `05` is kept for its reasoning and citations; the server's rule citations point at the rules table in `01-feature-map.md`. `docs/briefings/` holds dated team briefings. The web review page (PR #15) is parked. diff --git a/docs/01-feature-map.md b/docs/01-feature-map.md index a9b794f82..b11d509fb 100644 --- a/docs/01-feature-map.md +++ b/docs/01-feature-map.md @@ -1,5 +1,7 @@ # Feature map +> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. + **Priority:** **P0** = needed for the demo · **P1** = strong add · **P2** = stretch **Lanes (one teammate each):** **A** capture app · **B** object detection and geometry · **C** rules + placement code · **D** output, testing, pitch @@ -99,4 +101,4 @@ Public sources only. The team's working numbers are in `private/internal-notes.m ## Not doing -Android, Expo, rebuilding a 3D model from uploaded video, full-house or roof scans. +Android, Expo, full-house or roof scans. diff --git a/docs/02-implementation-plan.md b/docs/02-implementation-plan.md index 956faca5d..e6f3d82a1 100644 --- a/docs/02-implementation-plan.md +++ b/docs/02-implementation-plan.md @@ -1,5 +1,7 @@ # Implementation plan +> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. + The main trick is to unroll the house's walls into one straight line, measured in feet from the meter. Almost every rule then becomes a check on a 1-D number line, and the placement code gets simple. diff --git a/docs/03-stack-research.md b/docs/03-stack-research.md index f78cb3f04..4b835ba20 100644 --- a/docs/03-stack-research.md +++ b/docs/03-stack-research.md @@ -1,5 +1,7 @@ # Stack research: libraries, open-source projects, how they fit +> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. + This came from five parallel research agents on 2026-09-25 using live web searches. Anything marked **UNVERIFIED** couldn't be confirmed. Licenses are what each repo or model card said on that date, so re-check them before shipping a product. @@ -32,7 +34,7 @@ PHONE: RealityKit box attached to the meter's anchor + cable line | Piece | Rule(s) it serves | Why this tool, and what we're skipping | |---|---|---| -| ARKit + LiDAR mesh | Passage width, headroom, gaps between openings, cable length | Real measurements (~±6–8 cm on walls, ~5 m range) instead of estimates by eye. Skip photogrammetry. | +| ARKit + LiDAR mesh | Passage width, headroom, gaps between openings, cable length | Real measurements (~±6–8 cm on walls, ~5 m range) instead of estimates by eye. | | Guided walk to both ends | "please send more photos" follow-ups | Makes "unseen space" impossible | | One continuous session | same wall? / panel behind meter | Shared coordinates; nothing pieced together from unordered photos | | Stray Scanner fork | everything downstream | The only maintained MIT recorder of position + lens data + depth. No open-source app also exports the mesh | diff --git a/docs/04-prior-art-and-codes.md b/docs/04-prior-art-and-codes.md index 5c0066287..668a4f354 100644 --- a/docs/04-prior-art-and-codes.md +++ b/docs/04-prior-art-and-codes.md @@ -1,5 +1,7 @@ # Prior art, what Base does publicly, and code citations +> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. + Researched 2026-09-25. UNVERIFIED items are marked. ## What Base does today (public) diff --git a/docs/05-live-guided-survey-hld.md b/docs/05-live-guided-survey-hld.md index cb0a73d16..f32f12ab6 100644 --- a/docs/05-live-guided-survey-hld.md +++ b/docs/05-live-guided-survey-hld.md @@ -1,5 +1,7 @@ # Base Power — Live Guided Site Survey +> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. + **High-Level Design · v0.1 · September 25, 2026** **Status:** Public architecture proposal for review; no implementation or existing lane decisions are changed by this document. diff --git a/docs/briefings/2026-09-26-morning.html b/docs/briefings/2026-09-26-morning.html index 9e8cb28da..24d414450 100644 --- a/docs/briefings/2026-09-26-morning.html +++ b/docs/briefings/2026-09-26-morning.html @@ -237,6 +237,7 @@

House scanning: what we are building, what works, and what is blocked

Team briefing, 26 September 2026, facts as of 09:10. Readers: Sam, Hunter, Shrey, Aiden. About ten minutes.

+

Update 11:15: the team split into client and server teams; see docs/eli5.html.

A homeowner walks the outside wall around their electric meter with an iPhone. The phone measures the wall, a server applies the clearance rules in plain code, and the app shows where the battery goes, in AR, on the real wall.

Today it runs end to end in the iPhone Simulator on a recorded real walk and gets a real answer from the live server. It has never run on a physical iPhone at a real wall. That is the blocker, and the first job this morning.

@@ -785,7 +786,7 @@

Finding 2. Can photos alone give 3D measurements?

Learned depth models (MoGe-2, Depth Anything 3), rescaled with the phone's poses. Flat walls come out at about 5 in at best at a realistic phone error. Window and door edges, which are what clearances are measured from, come out at 8 in or worse in every setting.
-

What it changed: no photo-to-3D measurement pipeline. Every point a check depends on is an AR tap; learned depth only draws the 3D picture. This saved building the wrong thing.

+

What it changed: 3D modeling stays central. Learned depth alone is too rough at edges, so LiDAR and the phone's poses supply the scale. World models are not evaluated yet. (Corrected at midday; the 09:10 version said to drop photo-to-3D measurement.)

diff --git a/docs/eli5.html b/docs/eli5.html index aa86f6ac6..17f09e66f 100644 --- a/docs/eli5.html +++ b/docs/eli5.html @@ -3,607 +3,715 @@ -Where does the battery go? — explained like you're 5 + +How house scanning works -
-
-

Where does the battery go? 🔋🏠

-

Some families get a big battery that sits outside, next to their house. It keeps the lights on when the power goes out. - But it can't go just anywhere. Here's how our app finds the perfect spot.

- -
-
- - -
-
-
- THE PROBLEM -

Photos can't measure 📸🤷

-

Right now, families send in photos. But a photo is flat! It can't tell whether the gap between two windows is as big as a bathtub - or as small as a lunchbox. And if someone forgets to photograph part of the wall, nobody knows what's there.

-

For grown-ups: a single photo has no real scale, so distances get estimated by eye. Borderline cases end up with a human reviewer, - and any wall nobody photographed turns into a "please send more photos" follow-up.

-
-
- - - - - - - - - - - - - ?? ft - "is this 3 feet or 8 feet?" 🤔 - -
-
-
- - -
-
- STEP 1 -

The magic measuring phone 📱✨

-

The family holds up their phone and walks along the wall, like playing "follow the wall." The phone counts how far it went, like - counting footsteps. Fancy phones even shoot out invisible laser dots to feel how far things are, the way a bat uses sound! The phone - says "keep going!" until it has seen the wall all the way to the end, both ways.

-
- - - - - - - - - - - - - - - end ✓ - - end ✓ - - - - - - - 👟 - - -
-
Wall seen
- -
-

For grown-ups: ARKit tracks the phone in real feet using the camera plus the motion sensors. LiDAR iPhones add a 3D mesh - (about ±6–8 cm on walls, out to about 5 m). The capture won't finish until both wall ends are tapped, so "space you haven't seen" can't happen.

-
-
-
- - -
-
- STEP 2 -

A robot plays I Spy 🤖🔍

-

The pictures go to a helper computer. A robot looks at them and plays I Spy: "I spy a window! I spy a door! I spy the gas meter!" - Then it sticks a label on each thing, in exactly the right spot along the wall.

-
- - - - - - - - - - - window! - gas meter! - the meter! - door! - AC! - 🔍 - -

For grown-ups: Gemini draws the boxes. It had the best zero-shot box accuracy in the research, and it returns boxes natively. SAM 2 tightens each box, and supervision merges duplicates across frames. Because the phone knows where it was for every photo, each box - gets cast onto the wall and becomes "window, 2–5 ft along the wall, 3–6.5 ft up."

-
-
-
- - -
-
-
- STEP 3 -

Looking down like a bird 🐦🗺️

-

Some things are easier to see from the sky: the driveway where cars go, the swimming pool, and which side of the house faces the street. - So the helper also checks a map, like a bird looking down.

-

For grown-ups: Census geocoder → Overture building outline → osmnx nearest road → street-facing wall. - Pools and driveways come from samgeo on public-domain StratMap/NAIP aerial photos. OSM tags them on under 5% of homes. Don't run ML on Google or Mapbox imagery; - their terms forbid it.

-
-
- - - - - - - - - - pool - - - - house - garage - - - - - - - - - - keep far fromthe pool - cars drivehere! stay away - faces the street 👀 - the meter → - - 🐦 - -
-
-
- - -
-
- STEP 4 · YOUR TURN -

Slide the box and check the rulebook 🧩

-

Drag the battery along the wall. Every time it moves, the helper checks the rulebook. Green means that rule is happy. - Red means "not here!" Then press Let the computer try every spot to watch it test every single position, one tiny step at a time.

-
-
- - - - - - - - - - - - - driveway - - - - - - - - - - - - - - - - - - - - - - windowgasmeter - doorAC - - - - - - - - BASE - - - - - - - -
- - - -
-
- space workers need at the meter - the wire from the meter - a good spot - not allowed - the stripe under the wall = the computer's test results -
-
-
- -
-
    -

    For grown-ups: this is the whole solver. It slides the 31″ battery along the unrolled wall in small steps and runs every - rule as a plain check (shapely + numpy, numbers in rules.yaml). There's no AI in this step, so every "no" comes with a reason - you can cite.

    -
    -
    -
    -
    - - -
    -
    -
    - STEP 5 -

    Show and tell 👀✨

    -

    The phone draws a pretend battery right on the real wall, like a sticker floating in the air, plus a purple line showing where the wire goes. - The family looks through their phone and says, "Oh! That's where it goes!"

    -

    For grown-ups: RealityKit places a box the battery's size (Base Core is 39.5″ × 30.68″ × 22″), attached to the meter's anchor rather than - raw world coordinates, because ARKit corrects anchors as the phone moves. It's also the moment the family gets to say whether they like how it looks.

    -
    -
    - - - - - - - - - - - - - BASE - - - Here it goes! ✨ - - Looks good 👍 - - - -
    -
    -
    - - -
    -
    - WHY IT MATTERS -

    Why this is better 🏆

    -

    Instead of guessing from flat photos, the phone measures, makes sure it saw everything, and a fair rulebook picks the spot.

    -
    -
    📏

    Real feet, not guesses

    -

    A flat photo can't tell 3 feet from 8 feet. The phone measures to within a few inches.

    -
    🔁

    No "take more photos" loop

    -

    The phone won't finish until it has seen the whole wall, so families don't get asked for more pictures later.

    -
    📖

    Every answer has a reason

    -

    "Too close to the gas meter" comes from a real safety rule, not a robot's feeling. That makes it easy to trust and to double-check.

    -
    -
    -
    - - -
    -
    -
    - 🧑‍🔧 Grown-up corner: what each cartoon really is - - - - - - - - - - - - - - - - - - -
    CartoonReal toolsWhy this one
    📱 Magic measuring phoneSwift + ARKit (gravity alignment, wall and ground detection, LiDAR mesh), frame recorder forked from Stray Scanner (MIT), keyframe picker from arkit-3dgs-scannerMeasures in real feet on the phone. Photogrammetry fails on blank siding, and COLMAP's own docs say so.
    🔍 Robot I SpyGemini boxes → SAM 2 → supervision; each box cast onto the wall using where the phone wasBest zero-shot box accuracy (13.3 vs 1.5 mAP for GPT), with native box output.
    🔢 Reading the meterApple Vision text + barcode reading on the phone; zxing-cpp + Gemini on the serverInstant "retake" on the phone. Serial-number OCR is the weak spot (63% read vs 97% for the dial counters).
    🐦 Bird's-eye mapCensus geocoder, Overture building outlines via DuckDB, osmnx, samgeo on StratMap/NAIPFree, public-domain imagery that ML is allowed to run on. Google's and Mapbox's terms ban it.
    🧩 Slide the boxshapely + numpy sweep, Open3D for passage-width and headroom rays, rules.yamlUnder a second to run, easy to explain, and every rule traces to NEC 110.26, IRC R328, Austin Energy or Texas Gas Service.
    ✨ Show and tellRealityKit box attached to the meter's anchor; drawsvg 2D site plan for reviewersThe demo moment, and where the family sees and approves the spot.
    🐞 DebuggingRerun (its ARKitScenes example logs exactly our data)Camera positions, frames, mesh, detections and solver rays in one timeline.
    📊 The pitch baselineDepth Anything 3 metric (Apache-2.0) on plain photosProves AR beats photos, using a tape measure as the answer key.
    -

    The example wall is made up for illustration: window 2–5 ft, gas meter 8–9 ft, meter at 12 ft, door 24–27 ft, AC 30–33 ft, driveway from 37 ft. - The 3 ft gas, AC, door and window clearances and the 20 ft wire limit come from Base's public placement rules and building codes. The driveway distance is made up for the demo.

    -
    -
    -
    - -
    Made for the Base Power hackathon team · Drag the battery in step 4 · Works offline, one file
    - - + From 8afa331394deda5e15ad9a802361b7082312d638 Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 11:35:20 -0500 Subject: [PATCH 05/16] docs: align the briefing's decision 2 with 3D modeling staying central The morning recommendation read "AR taps, not depth maps", which read as ruling out 3D modeling. It now says to take the points that decide a check from metric depth (LiDAR or AR taps), since learned depth alone is too rough at edges, while the 3D model carries everything else. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/briefings/2026-09-26-morning.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/briefings/2026-09-26-morning.html b/docs/briefings/2026-09-26-morning.html index 24d414450..a25a60c7d 100644 --- a/docs/briefings/2026-09-26-morning.html +++ b/docs/briefings/2026-09-26-morning.html @@ -876,7 +876,7 @@

    Finding 5. Can a first-time homeowner finish a scan?

    7. Decisions for this morning, recommendation first

    1. Run the field test first

      Recommend: about an hour, before anything else.

      A current non-LiDAR iPhone, Measure Lab, a 30 ft tape, a real meter wall. Follow experiments/evals/field/FIELD_SHEET.md. One command turns tape readings into a score and the phone's real scale error.

      2. -

    2. How points get measured

      Recommend: AR taps, not depth maps.

      Finding 2: learned depth gets edges to 8 in or worse.

      3. +

    3. How points get measured

      Recommend: metric depth where precision decides a check.

      Take those points from LiDAR or AR taps; learned depth alone gets edges to 8 in or worse (Finding 2). The 3D model still carries everything else. Updated 11:40 after the team split.

    4. How far to walk

      Recommend: outward from the meter on both sides; stop when the nearest spot that fits is fully seen.

      The server prefers the shortest cable, so walking farther cannot find a better spot, and shorter walks drift less.

    5. Base's real rules

      Recommend: load them into the private rules file now.

      Confirm with Base which rules the demo follows and which number on the meter plate they need (plates carry several). Otherwise the demo can never show a real "fits here".

    6. Merge order

      Recommend: #11 server first, since the others depend on its schemas.

      Then #14 docs, #10 app, #13 verification, #12 evals, then #7, #4, #8, #6, #16, #17, #19.

      7. From b5869de1f1bddc4c5809494a5565b335795016d1 Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 12:35:06 -0500 Subject: [PATCH 06/16] docs: restore day-1 reasoning in the onboarding and keep eli5 as a record - Rebuild docs/00 from the original overview: the core insight, decisions, all five corrections and the open questions return, with the team split, a sixth correction on 3D modeling and the current evidence added where they belong. - Restore README's "why measure", "how it works" and experiment sections, updated with results, alongside the repository map. - Restore AGENTS.md's per-document pointers, rule-source kinds, the reasons capture is native, the repository layout and the DEVELOPMENT_TEAM key. - Move the new explainer to docs/how-it-works.html and restore the day-1 eli5.html with a dated note and three fixes where it claimed unseen space can't happen or omitted UNSURE. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 21 +- README.md | 65 +- docs/00-overview.md | 161 +-- docs/briefings/2026-09-26-morning.html | 4 +- docs/eli5.html | 1292 +++++++++++------------- docs/how-it-works.html | 717 +++++++++++++ 6 files changed, 1482 insertions(+), 778 deletions(-) create mode 100644 docs/how-it-works.html diff --git a/AGENTS.md b/AGENTS.md index 5313778c6..9ea643ceb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # Agent guide -A homeowner scans the outside of their house around the electric meter with an iPhone. The system works out whether a Base Power battery fits there and where, and the app shows the spot in AR. It is a hackathon project that started on 2026-09-25. The MVP is three problems: a phone that guides the homeowner until everything the model needs is captured, photos and their metadata turned into a 3D model, and that model evaluated against the placement criteria. +A homeowner scans the outside of their house around the electric meter with an iPhone. The system works out whether a Base Power battery fits there and where, and the app shows the spot in AR. It is a four-person hackathon project that started on 2026-09-25. The MVP is three problems: a phone that guides the homeowner until everything the model needs is captured, photos and their metadata turned into a 3D model, and that model evaluated against the placement criteria. ## Who owns what @@ -15,31 +15,38 @@ Read the row for your task. Files marked with a pull request exist only on that | Task | Read | | --- | --- | -| What the system is, what's decided, what's open | `docs/00-overview.md` | +| What the system is, what's decided, what's open, and corrections that supersede the day-1 plan | `docs/00-overview.md` | +| A picture-first walkthrough of the current system | `docs/how-it-works.html` | | The iOS app | `ios/README.md` | | The capture packet the app sends | `packet/README.md` (coming on branch `t3/packet`) | | The rules engine, API and capture contract | `server/README.md` (PR #11), especially "What settles each check" | | Turning photos into a 3D model | `recon/HANDOFF.md` (PR #20) | | Accuracy evals on public datasets | `experiments/evals/README.md` (PR #12) | | Branches, checks and TestFlight | `CONTRIBUTING.md` | -| Rule citations and day-1 reasoning | `docs/01` to `05`; `docs/00` and the component READMEs win where they differ | +| The rules table and the `scene.json` data format | `docs/01-feature-map.md` | +| How each part was planned, with code snippets and risks | `docs/02-implementation-plan.md` | +| Libraries and open-source projects per part, licenses, gotchas, what to skip | `docs/03-stack-research.md` | +| Base's public rules, competitors, NEC, IRC and Austin Energy citations | `docs/04-prior-art-and-codes.md` | +| The live guided-survey design: automatic capture, coverage display, next-view planning | `docs/05-live-guided-survey-hld.md` | + +`docs/01` to `05` and `docs/eli5.html` are the day-1 plan and explainer. Where they differ from `docs/00` or a component README, those win. ## Rules - **Keep Base's materials in `private/`, which git ignores.** The repository is public. Don't commit, quote or summarize those materials in tracked files, including prompt text, prompt names, output field names and internal thresholds. Team notes go in `private/internal-notes.md`. Real captures, photos of homes and dataset images go in `captures/`, `fixtures/real/` or `data/`, which git also ignores. -- **Make placement decisions in deterministic code.** Models build the geometry and recognize things such as the meter's text or an object's outline. Plain code over that geometry makes the pass or fail decision, so every answer traces back to a rule and a measurement. Clearance numbers live in rules files with their sources, never in code: public values with citations in `server/rules.yaml` (PR #11), and Base's values in the git-ignored `private/rules.yaml`, which the server merges over them. +- **Make placement decisions in deterministic code.** Models build the geometry and recognize things: object boxes, the panel brand, the meter's text. Plain code over that geometry makes the pass or fail decision, so every answer traces back to a rule and a measurement. Clearance numbers live in rules files with their sources, never in code. `server/rules.yaml` (PR #11) holds the public values, each citing a code, Base's public help page, or marked as a demo placeholder. Base's values go in the git-ignored `private/rules.yaml`, which the server merges over the public file when it exists. - **Treat unseen as unsure, never as clear.** A gap in the scan could hide a gas meter. When a check depends on an area nobody observed, return UNSURE and name the view that would settle it. - **Use StratMap or NAIP for aerial imagery.** Google Maps, Street View, Mapbox and Esri forbid running ML on their imagery in their terms. - **Place AR results relative to the meter's anchor, with `.gravity` world alignment.** ARKit corrects anchors as tracking improves, so a result tied to the meter moves with it while raw world coordinates drift. `.gravityAndHeading` depends on the compass, which is unreliable next to a house. - **Rotate the intrinsics whenever you rotate a camera image.** Camera images are landscape sensor images and the intrinsics match that orientation. A rotated image with unrotated intrinsics produces wrong 3D geometry without any error. -Decided: the app is native iOS only (Swift, ARKit, RealityKit), because Expo has no first-class ARKit support. It uses LiDAR and other depth sensors when present and never requires them, since most homeowners' phones lack LiDAR. +Decided: the app is native iOS only (Swift, ARKit, RealityKit), because Expo has no first-class ARKit support and capture needs LiDAR mesh export, per-frame position and lens data, and mesh raycasts. It uses LiDAR and other depth sensors when present and never requires them, since most homeowners' phones lack LiDAR. ## Working in the repository +- Layout: `ios/` is the capture app, its project generated by XcodeGen. `server/` is the Python server (uv). `web/` is a browser toolchain for the reviewer view and zero-install capture experiments. `experiments/` holds one folder per experiment. `sites/landing` is the landing page, a submodule; change it in its own repository. - `make check` runs every suite; `make ios`, `make server` and `make web` run one each. `CONTRIBUTING.md` lists the matching CI checks. - Branch from `main`, keep one writer per branch, and open a pull request. People merge. Agents never push to `main` or merge, so a person sees every change before it lands. - Several agents share one Mac. Exit code 137 means the system killed the process, usually for memory. Find the large allocation before rerunning, because macOS swaps before it kills and one runaway process can freeze the whole machine. -- Set your signing team in `ios/Config/Local.xcconfig` (copy `Local.xcconfig.example`; see `ios/README.md`), not in Xcode's Signing & Capabilities pane. The pane writes into `project.pbxproj`, and CI fails on that drift. +- Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` (copy `Local.xcconfig.example`; see `ios/README.md`), not in Xcode's Signing & Capabilities pane. The pane writes into `project.pbxproj`, and CI fails on that drift. - After editing `ios/project.yml`, run `make ios-project` (it needs XcodeGen 2.46.0) and commit the regenerated project, because CI regenerates it and fails on any difference. -- `sites/landing` is the landing page, a submodule. Change it in its own repository. diff --git a/README.md b/README.md index 10a5c135e..d17833b12 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,67 @@ # House scanning -A homeowner scans the outside of their house around the electric meter with an iPhone. The system works out whether a Base Power battery fits there and where, and the app shows the spot in AR. It is a hackathon project for Base Power. +We're exploring a simpler way to survey a home for a Base Power battery. -The work splits in two. The client team builds the iPhone app, which guides the scan with a live 3D map and sends a capture packet. The server team turns the packet into a 3D model and evaluates the placement rules on it. [docs/00-overview.md](docs/00-overview.md) explains the whole system. +A homeowner walks the outside of their house with a phone app. The app measures the wall around the electric meter, and our code decides whether a Base Power battery can be installed there and, if so, where. The homeowner sees the spot in AR before they put the phone away. + +This is a four-person hackathon project for Base Power. Success means one capture session gives Base enough to decide, with no follow-up photos. A person can still review the result. + +Since 2026-09-26 the work is split between two teams. The client team builds the iPhone app, which guides the scan with a live 3D map and sends a capture packet. The server team turns the packet into a 3D model and evaluates the placement rules on it. [docs/00-overview.md](docs/00-overview.md) has the details. + +## Why measure instead of photograph + +Base's site survey starts from a few customer photos, and three things go wrong with them: + +- Distances are estimated by eye, so borderline clearances go to a reviewer. +- Unordered photos can't show whether two things are on the same wall or how far apart they are. +- Nobody knows what sits just outside the frame, so the homeowner gets asked for more photos. + +An AR session tracks the phone's position as it moves. The app can measure the wall in feet, and it knows which stretches nobody has looked at yet. It asks for those while the homeowner is still standing there. + +## How it works + +1. The homeowner marks the electric meter, walks the wall to its end in each direction, and takes close-ups of the meter and the electrical panel. A live 3D map on the phone shows what has been seen, and the app guides the homeowner until everything the rules need is covered. The app sends a capture packet: photos, where the camera was for each one, lens data, motion-sensor readings, LiDAR depth when the phone has it, and the homeowner's marks. +2. The server builds a 3D model from the packet and lays the walls out as one line, measured in feet from the meter. It slides the battery's footprint along that line and checks each clearance: gas equipment, driveways, doors and windows, room overhead, and the gap to a facing fence. Each check returns pass, fail, or unsure, with the measurement error counted. Anything the phone never saw counts as unsure. +3. The app shows the chosen spot, the battery, and the cable route from the meter in AR. + +Plain code makes the placement decision. Models build the geometry and recognize things, such as where a gas meter is or what the meter label says. The public demo uses public rule values plus two labeled placeholders (pool 10 ft, driveway 5 ft); Base's own values load from a private file. + +## What we've tested so far + +The plan targets iPhones, and a LiDAR iPhone makes some measurements easier. Most homeowners don't have one, so the experiments ask how far an ordinary phone gets. + +- **Can AR taps on the ground and the wall measure the wall line, the room overhead, and the gap to a fence without LiDAR?** Measure Lab (PR #7) is the test rig. A field session against a tape survey is the remaining measurement. +- **Can learned 3D models build accurate geometry from ordinary photos?** We measured MapAnything, Depth Anything 3 and MoGe against the laser-scanned buildings in the ETH3D dataset (PR #12). MapAnything's scale shifted with input resolution. A model's own scale is 4 to 12% off. Rescaled with the phone's poses, walls come within about 5 in (p90) at a realistic phone error, but edges stay at 8 in or worse. The reconstruction worker (PR #20) gets walls to 2.1 in with a laser scan standing in for LiDAR, and 2.8 in from photos only. World models are not evaluated yet. +- **How far does phone AR tracking drift during a walk?** On ADVIO's 2018 iPhone walks it ran 2 to 3 times over the server's error allowance, partly because of noise in the dataset's own reference. A current iPhone, on the MARViN dataset, stayed inside it, though that phone has LiDAR and the reference may be scaled to its tracking. The field test measures our own phone. + +A convincing 3D view can still miss an obstacle or get a distance wrong, so every result keeps its original photos beside it. ## Repository map | Path | What it is | State on 2026-09-26 | | --- | --- | --- | | `ios/` | The native iPhone app (Swift, ARKit, RealityKit) | On `main`: an AR session skeleton. Guided capture is in PR #10; the live 3D map is on branch `t3/ios-map3d`, stacked on it | -| `packet/` | The capture packet: photos, poses, intrinsics, motion data, LiDAR depth and mesh, the homeowner's marks | Being specified on branch `t3/packet` | +| `packet/` | The capture packet the app sends | Being specified on branch `t3/packet` | | `server/` | The deterministic rules engine and API | On `main`: a skeleton. The engine is in PR #11 and runs as a public demo | | `recon/` | A worker that turns photos and depth into a 3D model and a coverage map | Draft PR #20, handed to the server team | -| `experiments/` | One folder per experiment | Accuracy evals on public datasets in PR #12 | -| `web/` | Browser toolchain | The review page (PR #15) is parked | -| `docs/` | The overview, a visual explainer, the day-1 plan and dated briefings | | +| `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7 | +| `web/` | Browser toolchain for the reviewer view and zero-install capture experiments | The review page (PR #15) is parked | +| `docs/` | The overview, explainers, the day-1 plan and dated briefings | | | `sites/landing` | The landing page, a submodule | Change it in its own repository | -## What's live - -- The public demo server, with public rules only: https://house-scanning-server.vercel.app. A second deployment loads Base's rules and requires a key. -- A TestFlight workflow, started by hand. See [CONTRIBUTING.md](CONTRIBUTING.md). +The public demo server, with public rules only, is at https://house-scanning-server.vercel.app. A second deployment loads Base's rules and requires a key. TestFlight builds are started by hand; see [CONTRIBUTING.md](CONTRIBUTING.md). -## Where to start +## Read next -- New to the project: [docs/eli5.html](docs/eli5.html), a picture-first explainer. Open it in a browser. -- Contributors and coding agents: [AGENTS.md](AGENTS.md), then [CONTRIBUTING.md](CONTRIBUTING.md). +| File | What it covers | +| --- | --- | +| [docs/how-it-works.html](docs/how-it-works.html) | A visual explainer of the current system: the two-team pipeline, error bars, coverage and the 3D path options | +| [docs/00-overview.md](docs/00-overview.md) | The goal, the team split, decisions made so far, evidence and open questions | +| [docs/01-feature-map.md](docs/01-feature-map.md) | Day-1 plan: the four work lanes, features by priority, the `scene.json` format, and the rules table | +| [docs/02-implementation-plan.md](docs/02-implementation-plan.md) | Day-1 plan: how each part gets built, the schedule, and risks | +| [docs/03-stack-research.md](docs/03-stack-research.md) | The open-source projects and libraries we build on, and the ones we skip | +| [docs/04-prior-art-and-codes.md](docs/04-prior-art-and-codes.md) | Base's public rules, similar products, and electrical-code citations | +| [docs/05-live-guided-survey-hld.md](docs/05-live-guided-survey-hld.md) | Proposed live guidance and automatic capture architecture, diagrams, evidence flow, and technical resources | +| [docs/eli5.html](docs/eli5.html) | The day-1 visual walkthrough, with an interactive demo of the placement search | -This repository is public. Materials Base gave the team stay in the git-ignored `private/` folder. +Contributors and coding agents start with [AGENTS.md](AGENTS.md). This repository is public, so materials Base gave the team stay in the git-ignored `private/` folder. diff --git a/docs/00-overview.md b/docs/00-overview.md index 0d13be4a7..c34b1caca 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -1,18 +1,11 @@ -# House scanning: system overview +# House scanning: overview and decisions -Current as of 2026-09-26 midday. Where this file and the day-1 plan (`01` to `05`) disagree, this file wins. Where this file and a component's README disagree, the README wins. +Hackathon project, 4-person team, started 2026-09-25. The goal: from a homeowner's phone, work out +**where a Base Power battery can be installed** against the outside of the house, and show the spot +back to them in AR. Updated 2026-09-26 midday for the team split; where this file and a component's README disagree, the README wins. -## The problem - -Base Power installs a home battery on the outside wall, close to the electric meter, because a cable runs from the meter to the battery. The battery is 31 in wide, 22 in deep and 39.5 in tall. Whether it fits depends on clearance rules: distance from the gas meter, from doors and windows, from AC units and fences, clear space in front, headroom above, and cable length from the meter. Today someone judges that from a few customer photos, and three things go wrong: - -- A photo has no scale, so distances are guessed and borderline cases go to a reviewer. -- Loose photos can't show whether two things are on the same wall or how far apart they are. -- Nobody knows what sits just outside the frame, so the homeowner gets asked for more photos. - -## The product - -A homeowner scans the outside of the house around the meter with an iPhone. The system works out whether the battery fits and where, and the app shows the spot in AR on the real wall. One scan should give Base enough to decide without follow-up photos. +> **This repo is public.** Materials Base gave the team live in `private/`, which is gitignored. Never commit, +> quote or summarize them here. Team-only notes are in `private/internal-notes.md`. The MVP is three problems: @@ -20,7 +13,18 @@ The MVP is three problems: 2. Photos and their metadata turned into a 3D model. 3. The 3D model evaluated against the placement criteria. -## How it fits together +## The core insight (why AR) + +Today homeowners submit photos and reviewers judge them. Three problems come up that ordinary photos can't solve: +1. **Measuring distance.** A single photo has no real scale, so gaps and clearances get estimated by eye, and the borderline cases go to a person. +2. **Fitting photos together.** A handful of unordered photos can't reliably say whether two things are on the same wall, or how far apart they are. +3. **Coverage.** If part of the wall wasn't photographed, nobody can say what's there, so the homeowner gets asked for more photos. + +AR capture fixes all three: the phone's tracking and depth give every photo a known position and a real scale, which a 3D +model can build on, and the app knows what it hasn't seen yet. The recognition half (panel brand, meter text, rust, whether +a window opens) stays with a vision model. AR adds nothing there beyond better photos. + +## Two teams (since 2026-09-26) ``` client team (iPhone) server team @@ -33,53 +37,94 @@ The MVP is three problems: └──────────────────────────┘ └──────────────────────────┘ ``` -**Client team.** Sam, working with AI agents, builds the iOS app. Aiden films video and gathers sample datasets. The app guides capture with a live on-device 3D map, and a haze ("fog of war") covers what the phone hasn't seen yet. Coverage aims at 100% of what the model needs, not the whole house. The app uses LiDAR and other depth sensors first when the phone has them, and the path without LiDAR is built alongside. Guided capture is in PR #10; the live 3D map is on branch `t3/ios-map3d`, stacked on it. - -The app's output is the **capture packet**: photos, camera poses (where the camera was and which way it pointed), intrinsics (the lens's focal length and center), motion-sensor (IMU) readings, LiDAR depth and mesh when present, and the homeowner's marks such as the meter. It is being specified on branch `t3/packet`. - -**Server team.** Hunter, working with his own agents, takes the packet and builds a 3D model or point cloud, then evaluates the criteria on it. They are comparing world models with traditional and newer reconstruction methods: first broadly, then in depth on the most promising paths. Their starting points: - -- `server/` (PR #11): the deterministic rules engine and API. It runs as a public demo at https://house-scanning-server.vercel.app, and a second deployment loads Base's rules behind a key. -- `recon/` (draft PR #20): a working reconstruction worker. It fuses depth into a 3D model and tests coverage against it. -- `experiments/evals/` (PR #12): accuracy evals on public datasets with laser-scanned ground truth. - -## Decided - -- **Native iOS only**, in `ios/`: Swift, ARKit and RealityKit. Expo has no first-class ARKit support. -- **LiDAR first, never required.** Use depth sensors when present; most homeowners' phones lack LiDAR, so every check must also work without it. -- **Deterministic placement decisions.** Models build the geometry and recognize things. The pass or fail decision is plain code over that geometry, with each clearance number and its source in a rules file. -- **Error bars decide.** A check passes only when its margin beats the measurement error, fails only when it misses by more than the error, and is UNSURE otherwise. +- **Client team:** Sam with AI agents; Aiden films video and gathers sample datasets. The iOS app guides capture with a live + on-device 3D map, and a haze ("fog of war") covers what the phone hasn't seen. Coverage aims at 100% of what the model needs, + not the whole house. LiDAR and other depth sensors come first when present; the non-LiDAR path is built alongside. Guided capture + is in PR #10; the live 3D map is on branch `t3/ios-map3d`, stacked on it. The output is the **capture packet**: photos, camera + poses, intrinsics, IMU readings, LiDAR depth and mesh when present, and the homeowner's marks. It is being specified on branch `t3/packet`. +- **Server team:** Hunter with his agents. Packet in, a 3D model or point cloud out, then the criteria evaluated on it. They are + comparing world models with traditional and newer reconstruction methods, broadly first, then in depth on the best paths. + Starting points: `server/` (PR #11), the deterministic rules engine and API, live as a public demo at + https://house-scanning-server.vercel.app plus a private deployment with Base's rules behind a key; `recon/` (draft PR #20), + a working reconstruction worker; and the evals in `experiments/evals/` (PR #12). + +## Decisions made + +- **AR-guided capture, not "upload a video and process it later."** Offline photogrammetry alone (COLMAP, splats) struggles on + blank siding and stucco, has no real scale, and is slow. ARKit already fuses the camera with the motion sensors on the phone, + and the capture packet carries those poses and any LiDAR depth, so the server's 3D model starts at real scale. +- **The phone collects data; the server decides.** Swift app → capture packet → Python FastAPI server (3D model, then rule checks) + → result → AR preview. +- **The placement code is plain code, not AI.** Models build the geometry and recognize things; the decision is ordinary code + over that geometry. Today's rules engine (PR #11) unrolls the house walls into one straight line `s` (feet along the wall + from the meter). Most rules become 1-D interval checks plus 2-D polygon distances. Sweep the battery footprint in small steps. + Every check returns PASS / FAIL / UNSURE, with error bars on each measurement. See `02-implementation-plan.md`. - **Unseen is never clear.** A check whose area nobody observed is UNSURE, and the server names the view that would settle it. -- **AR results are placed relative to the meter's anchor**, with `.gravity` world alignment. - -## What the evidence says so far - -All numbers come from public datasets with laser-scanned ground truth. Sources are `experiments/evals/README.md` (PR #12) and `recon/HANDOFF.md` (PR #20). +- **The vision model only recognises things.** Day-1 plan: Gemini boxes → SAM 2 → cast onto the wall using where the phone was. + Recognition prompts run on the close-ups. +- **Native Swift + ARKit/RealityKit on iPhone, in `ios/`.** iOS only. LiDAR is used when present, never required: most homeowners + don't have it. Not Expo (no first-class ARKit), and not Android. +- **3D modeling stays central.** Learned depth alone is too rough at edges, so LiDAR and the phone's poses supply the scale. See + the evidence below. +- **Normal dev app for the hackathon; an App Clip is only the pitch's rollout story** (see the gotchas in `03-stack-research.md`). + +## Corrections made after the research (supersede earlier drafts) + +1. Use `worldAlignment = .gravity`, **not** `.gravityAndHeading`. The compass is unreliable near a house + (developers report errors up to ~176°). Take one compass reading away from the meter as a rough hint. Which wall faces the + street comes from map data: Overture building outline + osmnx nearest road. +2. **Place the battery relative to the meter's anchor**, not in raw world coordinates. ARKit corrects anchors as it goes, + and drift is about 2 cm per second. +3. The battery is **39.5″ tall** (Base Core: 39.5 × 30.68 × 22 in), not 36″. Base's older units are 3 × 3 ft, + so keep all dimensions in the rules config. +4. Claude is **not** the box detector; its docs say its coordinates are approximate. Gemini returns boxes natively as + `[ymin, xmin, ymax, xmax]` on 0–1000. +5. **Never run ML on Google, Mapbox or Esri imagery.** Google's ToS §3.2.3(c) bans it outright. Use StratMap/NAIP (public domain). +6. **3D modeling is not ruled out** (2026-09-26). The morning briefing's Finding 2 first said "no photo-to-3D measurement pipeline". + The evidence says learned depth *alone* is too rough; with LiDAR or the phone's poses supplying scale, a 3D model is the plan. + +## Evidence so far (2026-09-26) + +From public datasets with laser-scanned ground truth. Sources: `experiments/evals/README.md` (PR #12) and `recon/HANDOFF.md` (PR #20). - A depth model's own scale is 4 to 12% wrong. -- Rescaled with the phone's poses, learned depth gets walls to about 5 in at p90 (9 in 10 points are closer) at a realistic phone error. Edges come out at 8 in or worse, and clearances are measured from edges. +- Rescaled with the phone's poses, learned depth gets walls to about 5 in at p90 at a realistic phone error. Edges come out at + 8 in or worse, and clearances are measured from edges. - The recon worker on the ETH3D building gets walls to p90 2.1 in with a laser scan standing in for LiDAR, and 2.8 in from photos only. - Coverage tested against depth claims at most about 0.46 ft of wall that no photo saw. - A current iPhone's tracking stays within the server's error allowance. -- World models have not been evaluated yet. - -So learned depth alone is too rough for edge clearances. 3D modeling stays central, with LiDAR and the phone's poses supplying metric scale. - -## Open - -- **Which 3D path.** Candidates are traditional reconstruction fused with LiDAR, learned depth rescaled with the phone's poses, and world models (large learned models that produce a whole 3D scene from photos or video). The server team decides after evaluating them. Whichever path wins needs metric scale, edges within the error allowance, and an honest account of which parts it observed and which it filled in. -- **Real-phone accuracy.** The numbers above come from public datasets. A field session with a tape survey on a real wall measures our own phone. -- **Base's rules.** The public demo runs on public values plus labeled placeholders. Base's own values load only from the private rules file. - -## Where each component's truth lives - -| Component | Source of truth | -| --- | --- | -| iOS app | `ios/README.md` | -| Capture packet | `packet/README.md` (coming on branch `t3/packet`) | -| Rules engine, API, capture contract | `server/README.md` (PR #11), "What settles each check" | -| Reconstruction worker | `recon/HANDOFF.md` (PR #20) | -| Evals | `experiments/evals/README.md` (PR #12) | -| Agent rules | `AGENTS.md` | - -The day-1 plan in `01` to `05` is kept for its reasoning and citations; the server's rule citations point at the rules table in `01-feature-map.md`. `docs/briefings/` holds dated team briefings. The web review page (PR #15) is parked. +- World models have not been evaluated yet; the server team will. + +## Open questions (team must decide) + +- **Which clearance numbers to use.** The public baseline is Base's help page: within 20 ft of the meter, within 1 ft of the wall, + and 3 ft from gas meters, fences, AC units and other batteries. The public demo fills the gaps with labeled placeholders + (pool 10 ft, driveway 5 ft) in `server/rules.yaml` (PR #11). Base's own values load from the git-ignored private rules file. + The team's working notes are in `private/internal-notes.md`. +- **Whether the install includes a separate battery disconnect box** beside the meter. Base's public pages mention a wall-mounted + transfer switch and a battery disconnect. +- **Which 3D path.** Traditional reconstruction fused with LiDAR, learned depth rescaled with the phone's poses, or world models + (large learned models that produce a whole 3D scene from photos or video). Whichever wins needs real scale, edges within the + error allowance, and an honest account of which parts it observed and which it filled in. +- **Real-phone accuracy.** The evidence above comes from public datasets. A field session with a tape survey on a real wall + measures our own phone. +- **The day-1 "test first" list** in `03-stack-research.md`: LiDAR on siding in midday sun, tap-to-pixel orientation, + Gemini boxes on real photos, and compass accuracy at the meter. Check which the experiments have answered before relying on one. + +## Doc map + +| File | What | +|---|---| +| `00-overview.md` | this file: goal, insight, team split, decisions, evidence | +| `how-it-works.html` | the current visual explainer: the two-team pipeline, error bars, coverage, the 3D path options | +| `01-feature-map.md` | day-1 plan: lanes A–D, feature list with priorities, data format, rules table | +| `02-implementation-plan.md` | day-1 plan: how to build each part, code snippets, 24 h schedule, risks | +| `03-stack-research.md` | libraries and open-source projects per part, licenses, gotchas, what to skip | +| `04-prior-art-and-codes.md` | what Base does publicly, competitors, electrical-code citations, pitch angle | +| `05-live-guided-survey-hld.md` | live guided survey: automatic capture, coverage display, next-view planning, evidence flow | +| `eli5.html` | the day-1 visual explainer; step 4 is an interactive "slide the battery" demo that predates error bars | +| `briefings/` | dated team briefings | + +Each component's own README is its source of truth: `ios/README.md`; `server/README.md` (PR #11), especially "What settles +each check"; `recon/HANDOFF.md` (PR #20); `experiments/evals/README.md` (PR #12); and `packet/README.md` (coming on branch +`t3/packet`). The web review page (PR #15) is parked. diff --git a/docs/briefings/2026-09-26-morning.html b/docs/briefings/2026-09-26-morning.html index a25a60c7d..b9998acbd 100644 --- a/docs/briefings/2026-09-26-morning.html +++ b/docs/briefings/2026-09-26-morning.html @@ -237,7 +237,7 @@

      House scanning: what we are building, what works, and what is blocked

      Team briefing, 26 September 2026, facts as of 09:10. Readers: Sam, Hunter, Shrey, Aiden. About ten minutes.

      -

      Update 11:15: the team split into client and server teams; see docs/eli5.html.

      +

      Update 11:15: the team split into client and server teams; see docs/how-it-works.html.

      A homeowner walks the outside wall around their electric meter with an iPhone. The phone measures the wall, a server applies the clearance rules in plain code, and the app shows where the battery goes, in AR, on the real wall.

      Today it runs end to end in the iPhone Simulator on a recorded real walk and gets a real answer from the live server. It has never run on a physical iPhone at a real wall. That is the blocker, and the first job this morning.

      @@ -786,7 +786,7 @@

      Finding 2. Can photos alone give 3D measurements?

      Learned depth models (MoGe-2, Depth Anything 3), rescaled with the phone's poses. Flat walls come out at about 5 in at best at a realistic phone error. Window and door edges, which are what clearances are measured from, come out at 8 in or worse in every setting.
      -

      What it changed: 3D modeling stays central. Learned depth alone is too rough at edges, so LiDAR and the phone's poses supply the scale. World models are not evaluated yet. (Corrected at midday; the 09:10 version said to drop photo-to-3D measurement.)

      +

      What it changed: 3D modeling stays central. Learned depth alone is too rough at edges, so the points that decide a check come from metric depth (LiDAR or AR taps), and LiDAR and the phone's poses give the model its scale. World models are not evaluated yet. (Corrected at midday; the 09:10 version said to drop photo-to-3D measurement.)

    diff --git a/docs/eli5.html b/docs/eli5.html index 17f09e66f..abade8549 100644 --- a/docs/eli5.html +++ b/docs/eli5.html @@ -3,715 +3,613 @@ - -How house scanning works +Where does the battery go? Explained like you're 5 (day-1 version) - - - -
    - -

    How house scanning works

    -

    Living explainer for new teammates and agents. Current as of 26 September 2026, midday. About fifteen minutes.

    - -

    A homeowner scans the outside of their house around the electric meter with an iPhone. The phone captures the wall, the server builds a 3D model and checks Base Power's placement rules against it, and the app shows where the battery goes, in AR, on the real wall.

    -

    Picture a home inspector with a tape measure and a checklist. The phone is the inspector's eyes and tape, and the server holds the checklist. The comparison breaks in one place, and that place shapes the design: an inspector can lean around a bush to look, but the server can judge only what the phone actually saw.

    - - - - -

    1. The problem: a battery has to fit between the rules

    - -
    - - An outside wall with the battery and its clearance rules, drawn to scale - Front view of a house wall. From left: an AC unit on the ground, a gas meter, the electric meter, the battery just to its right, a window, a door. Red hatched zones show 3 feet of clearance around the gas meter, AC unit, window and door. A bracket beside the battery shows 6.5 feet of headroom. A blue bracket along the ground shows 20 feet of cable reach either side of the electric meter. A tape along the bottom counts feet from the meter. Below, a view from above shows the battery footprint, 31 by 22 inches, and the 3 feet of clear space needed in front of it. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 20 ft of cable, either side of the meter - - - - AC unit - gas meter - electric meter - window - door - 6.5 ft headroom - battery - - - 3 ft - 3 ft - 3 ft - 3 ft - - - - - - - - - - - 151050 - 510152025 - - feet left and right of the electric meter - - - The same wall from above - - - - - - - - - - - - - - - - - - - - - - - - battery footprint, 31 by 22 in - 3 ft clear in front - AC unit - gas - - -

    Wider than the screen: scroll sideways for the rest of the picture.

    -
    Everything is drawn at one scale: 5 ft between tape marks. The red hatched areas are places the battery may not go. The battery just fits between the electric meter and the window's zone. Rules shown: 3 ft from the gas meter, from AC units, fences and other batteries, and from doors and windows that enter the home; 3 ft of clear space in front; 6.5 ft of headroom; within 20 ft of cable run from the meter; a flat, seen stretch of wall behind it.
    -
    - -
    -

    Base Power puts a home battery on the outside wall, close to the electric meter, because a cable runs from the meter to the battery. The battery is 31 in wide, 22 in deep and 39.5 in tall. Whether it fits, and where, depends on the clearance rules in the picture. Today someone judges that from a few customer photos, which have no scale and can't show what sits just outside the frame.

    -

    The demo's rules come from public sources and live in the server's rules file, each with its citation. Two numbers there are placeholders because no public value exists (pool 10 ft, driveway 5 ft). Base's own thresholds are private, load at runtime from a file that is not in git, and do not appear on this page.

    -
    -
    - The rules the demo uses, with sources -
    - - - - - - - - - - - -
    RuleValueSource
    Distance from the gas meter3 ftBase's public help page; Austin Energy; Texas Gas Service
    Distance from AC units, fences, other batteries3 ftBase's help page
    Distance from doors and windows that enter the home3 ftIRC R328.4
    Clear space in front of the battery3 ftBase's help page
    Cable run from the meterwithin 20 ftBase's help page
    Headroom6.5 ftNEC 110.26, a demo choice
    Wall behind the batteryflat, and seen by the camerademo requirement
    Pool, driveway10 ft, 5 ftplaceholders; no public value exists
    -
    - - -

    2. From scan to answer, one step at a time

    -

    The project splits into three problems: a phone that guides the homeowner until everything needed is captured, photos and their metadata turned into a 3D model, and that model checked against the rules. Two teams share them. Each picture below redraws the last and adds one step. The dashed line down the middle is the boundary between the teams.

    - -
      -
    1. - - Stage 1: the phone guides the scan with a live 3D map - The client team's side holds one box: a phone whose screen shows the wall, with grey haze over the part not yet seen. The label reads: guided scan, live 3D map, fog lifts as it sees. The server team's side is empty. - - -
      The phone guides the scan. The app builds a live 3D map on the phone as the homeowner moves. A haze, which the team calls fog of war, covers what the phone hasn't seen and lifts as it sees it. The app routes the homeowner until every part the model needs is seen.
      -
      2. - -
    2. - - Stage 2: the phone sends a capture packet - The guided scan box from stage 1, with an arrow to a file icon labelled packet, sitting on the line between the teams. - - -
      It sends a capture packet. Photos; where the camera was and which way it pointed for each one (poses); the lens's focal length and center (intrinsics); motion-sensor readings; LiDAR depth and mesh when the phone has LiDAR; and the homeowner's marks, such as the meter. The packet is the contract between the teams.
      -
      3. - -
    3. - - Stage 3: the server builds a 3D model from the packet - Stages 1 and 2, plus an arrow from the packet to a box on the server side showing a dotted 3D wall with a window and the meter, labelled 3D model or point cloud. - - -
      The server builds a 3D model. From the packet, the server team builds a 3D model or point cloud of the wall and the ground in front of it, at real scale. Which method does this best is open; section 4 compares the options.
      -
      4. - -
    4. - - Stage 4: plain code checks the rules against the model - Stages 1 to 3, plus an arrow down from the 3D model to a box labelled rules, plain code. It lists three checks: gas meter PASS, window UNSURE, door FAIL. - - -
      Plain code checks the rules. The rules and their sources live in a file. Code tries the battery's footprint at every spot along the wall, and each check at each spot comes out PASS, FAIL or UNSURE. No model makes this decision, so every answer traces back to a rule and a measurement.
      -
      5. - -
    5. - - Stage 5: the answer returns to the phone and shows in AR - The whole loop. Guided scan, packet, 3D model, rules, and now an arrow labelled answer back across the line to a phone on the client side showing the battery outlined on the wall beside the meter. - - -
      The answer comes back to the phone. The app draws the battery on the real wall in AR. It places the battery relative to the meter's anchor, so the drawing stays on the right spot as the phone's tracking corrects itself. When a check is UNSURE, the answer names the view that would settle it, and the homeowner can capture it on the spot.
      -
      6. -
    - - -

    3. Four ideas that keep the answer honest

    - -

    The error bar decides: PASS, FAIL or UNSURE

    -
    -

    No measurement is exact. The phone tracks its own position by dead reckoning, like counting your steps from a doorway, so its error grows the farther it gets from the meter. The comparison has a limit: a miscounted step only stretches the walk, while the phone's error can also drift sideways.

    -

    The server widens every measurement's error bar by 0.3 ft plus 0.16 ft for each foot from the meter. On public test walks, a current iPhone's tracking stayed inside that allowance. Then each check compares the rule against the whole error bar, not a single number.

    -
    -
    - - Three measurements against the 3 foot gas meter rule - A number line from 0 to 6 feet with a vertical rule at 3 feet. Row one: 4.5 feet plus or minus 0.8, the whole bar sits right of 3, PASS. Row two: 2.0 feet plus or minus 0.8, the whole bar sits left of 3, FAIL. Row three: 3.4 feet plus or minus 0.8, the bar straddles 3, UNSURE. - - - - - 0123456 ft - - - 3 ft rule (gas meter) - too close - clear - - - - - - - 4.5 ft ± 0.8: clear by 1.5, more than 0.8 - - ✓ PASS - - - - - - - 2.0 ft ± 0.8: short by 1.0, more than 0.8 - - ✕ FAIL - - - - - - - 3.4 ft ± 0.8: clear by only 0.4 - - ? UNSURE - -
    Three measurements against the 3 ft gas meter rule, each with an error bar of ± 0.8 ft. A check passes only when its margin beats the error, fails only when it misses by more than the error, and is UNSURE otherwise; a tie is UNSURE. UNSURE means a person looks, or the app asks for a better view. Tighter geometry shrinks the error bars and turns UNSURE answers into PASS or FAIL, which is why the 3D path matters.
    -
    - -

    Unseen is not clear

    -

    The live map's haze works like the fog of war in a strategy game: a dark patch is unexplored, not empty. The comparison stops there. In a game, scouting reveals the truth. Here, even the parts the phone saw carry an error bar.

    -
    - - A check whose ground was never seen stays UNSURE and names the missing view - A candidate battery spot on the wall. The ground beside it is covered by grey haze. Beside it a check reads: ground near the spot, checks need 6 to 13 ft of seen ground, UNSURE, missing evidence: show the ground about 6 ft right of your meter. - - - - - - meter - - - candidate spot - - never seen - - Ground near the spot - checks need 6 to 13 ft of ground - - ? UNSURE - missing evidence: show the - ground 6 ft right of your meter - -
    If the camera never saw part of the area a rule depends on, that check cannot pass. It stays UNSURE, and the server names the view that would settle it. Here the ground beside the spot was never seen, and the checks that use ground need 6 to 13 ft of it. A blank on the map is never treated as empty ground.
    -
    - -

    Coverage means 100% of what the model needs

    -

    The app doesn't need the whole house. It needs every part the rules depend on, all of it. That is a fixed zone around the meter, so the app can say exactly what is left, and the scan can end when the zone is clear.

    -
    - - The zone a scan must cover, seen from above - A plan view of the house with the electric meter on its front wall. A dashed blue rectangle marks the zone the checks need: the wall 20 feet either side of the meter and the ground in front of it out to 13 feet. Everything outside it is grey haze labelled not needed. Inside, a bush stands in front of the wall; the strip of wall behind it is unseen, so checks there are UNSURE. A dotted path shows the phone walking along the zone. - - - - - house, seen from above - - - - - cable reach: 20 ft either side - - - - - not needed - - - - - meter - - - - wall behind the bush: - unseen, so UNSURE - - - - needed: all of it - ground out to 13 ft - -
    Drawn to scale, 1 ft to 8 px. The zone is the wall within 20 ft of cable either side of the meter, the ground in front out to 6 to 13 ft depending on the check, and the space overhead for headroom, which a plan view can't show. The coverage map has to be honest. The app's first map checked only range, angle and framing, and claimed 1.1 ft of wall that no photo saw. Tested against depth, the recon worker claims at most about 0.46 ft.
    -
    - -

    LiDAR first, never required

    -

    Some iPhones have LiDAR, a sensor that measures depth directly by timing reflected light. Most homeowners' phones don't. The app uses LiDAR and other depth sensors whenever the phone has them, and every check must also work without them.

    -
    - - Two ways to get depth at real scale - Left, a phone with LiDAR sends rays to a wall seen from the side, and dots on the wall mark measured depth; its reach is about 5 metres. Right, any iPhone: a dashed line shows a depth model's guess of where the wall is, a little off. The phone at two known positions sees the same point on the real wall, and the two sight lines meet there, which fixes the scale. - - Phone with LiDAR - Any iPhone - - - - - - - - - measured directly - reach about 5 m - - - - - model's guess - - - - - - - known move - guessed, then rescaled - by triangulating with poses - -
    LiDAR works like a laser tape measure: direct distances, but a reach of about 5 m, so capture has to stand within about 4 m of the wall. Without it, a learned model estimates depth in each photo, and the phone's poses fix the scale: the same point seen from two known positions pins down its distance, the way two eyes judge depth. Where that comparison breaks: your eyes sit a fixed distance apart, but the phone's move between photos comes from its tracking and carries that error. Both paths fill the same capture packet.
    -
    - - - -

    4. How to build the 3D model: the options

    -

    Problem 2, photos and metadata into a 3D model, is open. The server team is comparing three families of methods: broadly first, then in depth on the best. None is ruled out. The numbers so far come from public datasets whose buildings were laser-scanned, so the true shape is known to a fraction of an inch.

    - -
    -
    -

    Traditional reconstruction, fused with LiDAR

    -

    Geometry from first principles: find the same points in several photos, work out where they sit in 3D, and merge the depth from every view into one surface. LiDAR depth drops straight into that merge.

    -

    The known weakness is that matching points across photos struggles on blank siding and has no scale of its own. LiDAR and the phone's poses supply the scale.

    -

    Evidence: the recon worker (PR #20) merges depth into one surface. With a laser scan standing in for LiDAR, its walls come out at 2.1 in (p90).

    -
    -
    -

    Learned depth plus the phone's poses

    -

    A neural network estimates depth from a single photo. Its own sense of scale is off by 4 to 12%, so the photos are rescaled with the phone's poses and then merged.

    -

    MapAnything, a learned model that reads several photos at once, was not a stable baseline: its scale shifted with the input resolution.

    -

    Evidence: walls about 5 in (p90) at a realistic phone error, 2.8 in with exact poses. Edges 8 in or worse in every setting.

    -
    -
    -

    World models

    -

    Large learned models that take photos or video and produce a whole 3D scene at once, camera positions included. Think of someone who has seen thousands of houses sketching this one from a few snapshots.

    -

    The sketch comparison also shows the risk: a good sketch fills in what the artist never saw. That is fine for a picture, but a check needs to know which parts were observed.

    -

    Evidence: not evaluated yet. The server team will.

    -
    -
    - -
    -

    Wall error at p90 for each 3D path tested so far, on one laser-scanned building

    - - Wall error at p90, in inches, for each 3D path tested so far - A dot plot from 0 to 20 inches. Depth model alone: about 20 inches. Depth model rescaled with poses at a realistic 2 percent pose error: 5.0 inches, interval 3.8 to 6.8. With exact poses: 2.8 inches, interval 2.0 to 4.4. Edges with learned depth: 8 inches or worse in every setting. Recon worker, photos only: 2.8 inches. Recon worker with a laser scan standing in for LiDAR: 2.1 inches. World models: not evaluated yet. - - - - 05101520 inwall error, p90 (9 in 10 points are closer) - - - - depth model alone - + poses, 2% error - + exact poses - edges, any setting - recon, photos only - recon, laser as LiDAR - world models - - - - - - - - - - - - - - about 20 in - 5.0 in (3.8 to 6.8) - 2.8 in (2.0 to 4.4) - 8 in or worse - 2.8 in - 2.1 in - not evaluated yet - - - Depth model alone, its own scale: p90 about 20 in - Depth rescaled with phone poses at a 2% pose scale error: p90 5.0 in, 95% interval 3.8 to 6.8 - Depth rescaled with exact poses: p90 2.8 in, 95% interval 2.0 to 4.4 - Edges, learned depth: 8 in or worse in every setting - Recon worker, photos only: wall p90 2.8 in - Recon worker, laser scan as LiDAR: wall p90 2.1 in - World models: not evaluated yet - - -
    All rows come from the ETH3D electro building, walls within 6 m. The top four are learned depth (MoGe-2 and Depth Anything 3); the next two are the recon worker; the last has no data yet. Brackets are 95% intervals where they exist. Caveat: the recon worker's two runs chose different walls, so those two numbers are not measured on the same wall. Reading it: learned depth alone is too rough for clearances, which are measured from edges. With LiDAR or the phone's poses supplying scale, walls come within a few inches.
    -
    Data table -
    - - - - - - - - - - -
    PathSettingWall error, p90Source
    Learned depthmodel's own scale (4 to 12% off)about 20 inevals, PR #12
    Learned depthrescaled with poses, 2% pose scale error5.0 in (3.8 to 6.8)evals, PR #12
    Learned depthrescaled with exact poses2.8 in (2.0 to 4.4)evals, PR #12
    Learned depthedges, every setting8 in or worseevals, PR #12
    Recon workerphotos only2.8 inrecon, PR #20
    Recon workerlaser scan standing in for LiDAR2.1 inrecon, PR #20
    World modelsnot evaluatedserver team, next
    -
    -
    +
    +
    +
    Day-1 explainer, written 2026-09-25 and kept as a record. The current one is how-it-works.html. Out of date here: + the team has since split into a client team (the iPhone app and its capture packet) and a server team (a 3D model built from the packet, then the placement checks). + Step 4's demo scores every spot as plain pass or fail; the real checks also answer UNSURE when a measurement's error bar straddles the limit or its area was never seen. + The tool choices in the grown-up corner are the day-1 plan; the server team is now comparing 3D reconstruction methods, including ones that corner rules out.
    +

    Where does the battery go? 🔋🏠

    +

    Some families get a big battery that sits outside, next to their house. It keeps the lights on when the power goes out. + But it can't go just anywhere. Here's how our app finds the perfect spot.

    + +
    +
    + + +
    +
    +
    + THE PROBLEM +

    Photos can't measure 📸🤷

    +

    Right now, families send in photos. But a photo is flat! It can't tell whether the gap between two windows is as big as a bathtub + or as small as a lunchbox. And if someone forgets to photograph part of the wall, nobody knows what's there.

    +

    For grown-ups: a single photo has no real scale, so distances get estimated by eye. Borderline cases end up with a human reviewer, + and any wall nobody photographed turns into a "please send more photos" follow-up.

    +
    +
    + + + + + + + + + + + + + ?? ft + "is this 3 feet or 8 feet?" 🤔 + +
    +
    +
    + + +
    +
    + STEP 1 +

    The magic measuring phone 📱✨

    +

    The family holds up their phone and walks along the wall, like playing "follow the wall." The phone counts how far it went, like + counting footsteps. Fancy phones even shoot out invisible laser dots to feel how far things are, the way a bat uses sound! The phone + says "keep going!" until it has seen the wall all the way to the end, both ways.

    +
    + + + + + + + + + + + + + + + end ✓ + + end ✓ + + + + + + + 👟 + + +
    +
    Wall seen
    + +
    +

    For grown-ups: ARKit tracks the phone in real feet using the camera plus the motion sensors. LiDAR iPhones add a 3D mesh + (about ±6–8 cm on walls, out to about 5 m). The capture won't finish until both wall ends are tapped. Anything the phone still hasn't seen counts as unsure, never as clear.

    +
    +
    +
    + + +
    +
    + STEP 2 +

    A robot plays I Spy 🤖🔍

    +

    The pictures go to a helper computer. A robot looks at them and plays I Spy: "I spy a window! I spy a door! I spy the gas meter!" + Then it sticks a label on each thing, in exactly the right spot along the wall.

    +
    + + + + + + + + + + + window! + gas meter! + the meter! + door! + AC! + 🔍 + +

    For grown-ups: Gemini draws the boxes. It had the best zero-shot box accuracy in the research, and it returns boxes natively. SAM 2 tightens each box, and supervision merges duplicates across frames. Because the phone knows where it was for every photo, each box + gets cast onto the wall and becomes "window, 2–5 ft along the wall, 3–6.5 ft up."

    +
    +
    +
    + + +
    +
    +
    + STEP 3 +

    Looking down like a bird 🐦🗺️

    +

    Some things are easier to see from the sky: the driveway where cars go, the swimming pool, and which side of the house faces the street. + So the helper also checks a map, like a bird looking down.

    +

    For grown-ups: Census geocoder → Overture building outline → osmnx nearest road → street-facing wall. + Pools and driveways come from samgeo on public-domain StratMap/NAIP aerial photos. OSM tags them on under 5% of homes. Don't run ML on Google or Mapbox imagery; + their terms forbid it.

    +
    +
    + + + + + + + + + + pool + + + + house + garage + + + + + + + + + + keep far fromthe pool + cars drivehere! stay away + faces the street 👀 + the meter → + + 🐦 + +
    +
    +
    + + +
    +
    + STEP 4 · YOUR TURN +

    Slide the box and check the rulebook 🧩

    +

    Drag the battery along the wall. Every time it moves, the helper checks the rulebook. Green means that rule is happy. + Red means "not here!" Then press Let the computer try every spot to watch it test every single position, one tiny step at a time.

    +
    +
    + + + + + + + + + + + + + driveway + + + + + + + + + + + + + + + + + + + + + + windowgasmeter + doorAC + + + + + + + + BASE + + + + + + + +
    + + + +
    +
    + space workers need at the meter + the wire from the meter + a good spot + not allowed + the stripe under the wall = the computer's test results +
    +
    +
    + +
    +
      +

      For grown-ups: this is the whole solver. It slides the 31″ battery along the unrolled wall in small steps and runs every + rule as a plain check (shapely + numpy, numbers in rules.yaml). There's no AI in this step, so every "no" comes with a reason + you can cite. This demo predates error bars: the real solver also answers UNSURE when a measurement is too close to call or its area was never seen.

      +
      +
      +
      +
      + + +
      +
      +
      + STEP 5 +

      Show and tell 👀✨

      +

      The phone draws a pretend battery right on the real wall, like a sticker floating in the air, plus a purple line showing where the wire goes. + The family looks through their phone and says, "Oh! That's where it goes!"

      +

      For grown-ups: RealityKit places a box the battery's size (Base Core is 39.5″ × 30.68″ × 22″), attached to the meter's anchor rather than + raw world coordinates, because ARKit corrects anchors as the phone moves. It's also the moment the family gets to say whether they like how it looks.

      +
      +
      + + + + + + + + + + + + + BASE + + + Here it goes! ✨ + + Looks good 👍 + + + +
      +
      +
      + + +
      +
      + WHY IT MATTERS +

      Why this is better 🏆

      +

      Instead of guessing from flat photos, the phone measures, makes sure it saw everything, and a fair rulebook picks the spot.

      +
      +
      📏

      Real feet, not guesses

      +

      A flat photo can't tell 3 feet from 8 feet. The phone measures to within a few inches.

      +
      🔁

      No "take more photos" loop

      +

      The phone won't finish until it has seen every part the rules need, so families don't get asked for more pictures later.

      +
      📖

      Every answer has a reason

      +

      "Too close to the gas meter" comes from a real safety rule, not a robot's feeling. That makes it easy to trust and to double-check.

      +
      +
      +
      + + +
      +
      +
      + 🧑‍🔧 Grown-up corner: what each cartoon really is + + + + + + + + + + + + + + + + + + +
      CartoonReal toolsWhy this one
      📱 Magic measuring phoneSwift + ARKit (gravity alignment, wall and ground detection, LiDAR mesh), frame recorder forked from Stray Scanner (MIT), keyframe picker from arkit-3dgs-scannerMeasures in real feet on the phone. Photogrammetry fails on blank siding, and COLMAP's own docs say so.
      🔍 Robot I SpyGemini boxes → SAM 2 → supervision; each box cast onto the wall using where the phone wasBest zero-shot box accuracy (13.3 vs 1.5 mAP for GPT), with native box output.
      🔢 Reading the meterApple Vision text + barcode reading on the phone; zxing-cpp + Gemini on the serverInstant "retake" on the phone. Serial-number OCR is the weak spot (63% read vs 97% for the dial counters).
      🐦 Bird's-eye mapCensus geocoder, Overture building outlines via DuckDB, osmnx, samgeo on StratMap/NAIPFree, public-domain imagery that ML is allowed to run on. Google's and Mapbox's terms ban it.
      🧩 Slide the boxshapely + numpy sweep, Open3D for passage-width and headroom rays, rules.yamlUnder a second to run, easy to explain, and every rule traces to NEC 110.26, IRC R328, Austin Energy or Texas Gas Service.
      ✨ Show and tellRealityKit box attached to the meter's anchor; drawsvg 2D site plan for reviewersThe demo moment, and where the family sees and approves the spot.
      🐞 DebuggingRerun (its ARKitScenes example logs exactly our data)Camera positions, frames, mesh, detections and solver rays in one timeline.
      📊 The pitch baselineDepth Anything 3 metric (Apache-2.0) on plain photosProves AR beats photos, using a tape measure as the answer key.
      +

      The example wall is made up for illustration: window 2–5 ft, gas meter 8–9 ft, meter at 12 ft, door 24–27 ft, AC 30–33 ft, driveway from 37 ft. + The 3 ft gas, AC, door and window clearances and the 20 ft wire limit come from Base's public placement rules and building codes. The driveway distance is made up for the demo.

      +
      +
      +
      + +
      Made for the Base Power hackathon team · Drag the battery in step 4 · Works offline, one file
      + + diff --git a/docs/how-it-works.html b/docs/how-it-works.html new file mode 100644 index 000000000..c0d432b8d --- /dev/null +++ b/docs/how-it-works.html @@ -0,0 +1,717 @@ + + + + + + +How house scanning works + + + + + + + +
      + +

      How house scanning works

      +

      Living explainer for new teammates and agents. Current as of 26 September 2026, midday. About fifteen minutes.

      + +

      A homeowner scans the outside of their house around the electric meter with an iPhone. The phone captures the wall, the server builds a 3D model and checks Base Power's placement rules against it, and the app shows where the battery goes, in AR, on the real wall.

      +

      Picture a home inspector with a tape measure and a checklist. The phone is the inspector's eyes and tape, and the server holds the checklist. The comparison breaks in one place, and that place shapes the design: an inspector can lean around a bush to look, but the server can judge only what the phone actually saw.

      + + + + +

      1. The problem: a battery has to fit between the rules

      + +
      + + An outside wall with the battery and its clearance rules, drawn to scale + Front view of a house wall. From left: an AC unit on the ground, a gas meter, the electric meter, the battery just to its right, a window, a door. Red hatched zones show 3 feet of clearance around the gas meter, AC unit, window and door. A bracket beside the battery shows 6.5 feet of headroom. A blue bracket along the ground shows 20 feet of cable reach either side of the electric meter. A tape along the bottom counts feet from the meter. Below, a view from above shows the battery footprint, 31 by 22 inches, and the 3 feet of clear space needed in front of it. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 20 ft of cable, either side of the meter + + + + AC unit + gas meter + electric meter + window + door + 6.5 ft headroom + battery + + + 3 ft + 3 ft + 3 ft + 3 ft + + + + + + + + + + + 151050 + 510152025 + + feet left and right of the electric meter + + + The same wall from above + + + + + + + + + + + + + + + + + + + + + + + + battery footprint, 31 by 22 in + 3 ft clear in front + AC unit + gas + + +

      Wider than the screen: scroll sideways for the rest of the picture.

      +
      Everything is drawn at one scale: 5 ft between tape marks. The red hatched areas are places the battery may not go. The battery just fits between the electric meter and the window's zone. Rules shown: 3 ft from the gas meter, from AC units, fences and other batteries, and from doors and windows that enter the home; 3 ft of clear space in front; 6.5 ft of headroom; within 20 ft of cable run from the meter; a flat, seen stretch of wall behind it.
      +
      + +
      +

      Base Power puts a home battery on the outside wall, close to the electric meter, because a cable runs from the meter to the battery. The battery is 31 in wide, 22 in deep and 39.5 in tall. Whether it fits, and where, depends on the clearance rules in the picture. Today someone judges that from a few customer photos, which have no scale and can't show what sits just outside the frame.

      +

      The demo's rules come from public sources and live in the server's rules file, each with its citation. Two numbers there are placeholders because no public value exists (pool 10 ft, driveway 5 ft). Base's own thresholds are private, load at runtime from a file that is not in git, and do not appear on this page.

      +
      +
      + The rules the demo uses, with sources +
      + + + + + + + + + + + +
      RuleValueSource
      Distance from the gas meter3 ftBase's public help page; Austin Energy; Texas Gas Service
      Distance from AC units, fences, other batteries3 ftBase's help page
      Distance from doors and windows that enter the home3 ftIRC R328.4
      Clear space in front of the battery3 ftBase's help page
      Cable run from the meterwithin 20 ftBase's help page
      Headroom6.5 ftNEC 110.26, a demo choice
      Wall behind the batteryflat, and seen by the camerademo requirement
      Pool, driveway10 ft, 5 ftplaceholders; no public value exists
      +
      + + +

      2. From scan to answer, one step at a time

      +

      The project splits into three problems: a phone that guides the homeowner until everything needed is captured, photos and their metadata turned into a 3D model, and that model checked against the rules. Two teams share them. Each picture below redraws the last and adds one step. The dashed line down the middle is the boundary between the teams.

      + +
        +
      1. + + Stage 1: the phone guides the scan with a live 3D map + The client team's side holds one box: a phone whose screen shows the wall, with grey haze over the part not yet seen. The label reads: guided scan, live 3D map, fog lifts as it sees. The server team's side is empty. + + +
        The phone guides the scan. The app builds a live 3D map on the phone as the homeowner moves. A haze, which the team calls fog of war, covers what the phone hasn't seen and lifts as it sees it. The app routes the homeowner until every part the model needs is seen.
        +
        2. + +
      2. + + Stage 2: the phone sends a capture packet + The guided scan box from stage 1, with an arrow to a file icon labelled packet, sitting on the line between the teams. + + +
        It sends a capture packet. Photos; where the camera was and which way it pointed for each one (poses); the lens's focal length and center (intrinsics); motion-sensor readings; LiDAR depth and mesh when the phone has LiDAR; and the homeowner's marks, such as the meter. The packet is the contract between the teams.
        +
        3. + +
      3. + + Stage 3: the server builds a 3D model from the packet + Stages 1 and 2, plus an arrow from the packet to a box on the server side showing a dotted 3D wall with a window and the meter, labelled 3D model or point cloud. + + +
        The server builds a 3D model. From the packet, the server team builds a 3D model or point cloud of the wall and the ground in front of it, at real scale. Which method does this best is open; section 4 compares the options.
        +
        4. + +
      4. + + Stage 4: plain code checks the rules against the model + Stages 1 to 3, plus an arrow down from the 3D model to a box labelled rules, plain code. It lists three checks: gas meter PASS, window UNSURE, door FAIL. + + +
        Plain code checks the rules. The rules and their sources live in a file. Code tries the battery's footprint at every spot along the wall, and each check at each spot comes out PASS, FAIL or UNSURE. No model makes this decision, so every answer traces back to a rule and a measurement.
        +
        5. + +
      5. + + Stage 5: the answer returns to the phone and shows in AR + The whole loop. Guided scan, packet, 3D model, rules, and now an arrow labelled answer back across the line to a phone on the client side showing the battery outlined on the wall beside the meter. + + +
        The answer comes back to the phone. The app draws the battery on the real wall in AR. It places the battery relative to the meter's anchor, so the drawing stays on the right spot as the phone's tracking corrects itself. When a check is UNSURE, the answer names the view that would settle it, and the homeowner can capture it on the spot.
        +
        6. +
      + + +

      3. Four ideas that keep the answer honest

      + +

      The error bar decides: PASS, FAIL or UNSURE

      +
      +

      No measurement is exact. The phone tracks its own position by dead reckoning, like counting your steps from a doorway, so its error grows the farther it gets from the meter. The comparison has a limit: a miscounted step only stretches the walk, while the phone's error can also drift sideways.

      +

      The server widens every measurement's error bar by 0.3 ft plus 0.16 ft for each foot from the meter. On public test walks, a current iPhone's tracking stayed inside that allowance. Then each check compares the rule against the whole error bar, not a single number.

      +
      +
      + + Three measurements against the 3 foot gas meter rule + A number line from 0 to 6 feet with a vertical rule at 3 feet. Row one: 4.5 feet plus or minus 0.8, the whole bar sits right of 3, PASS. Row two: 2.0 feet plus or minus 0.8, the whole bar sits left of 3, FAIL. Row three: 3.4 feet plus or minus 0.8, the bar straddles 3, UNSURE. + + + + + 0123456 ft + + + 3 ft rule (gas meter) + too close + clear + + + + + + + 4.5 ft ± 0.8: clear by 1.5, more than 0.8 + + ✓ PASS + + + + + + + 2.0 ft ± 0.8: short by 1.0, more than 0.8 + + ✕ FAIL + + + + + + + 3.4 ft ± 0.8: clear by only 0.4 + + ? UNSURE + +
      Three measurements against the 3 ft gas meter rule, each with an error bar of ± 0.8 ft. A check passes only when its margin beats the error, fails only when it misses by more than the error, and is UNSURE otherwise; a tie is UNSURE. UNSURE means a person looks, or the app asks for a better view. Tighter geometry shrinks the error bars and turns UNSURE answers into PASS or FAIL, which is why the 3D path matters.
      +
      + +

      Unseen is not clear

      +

      The live map's haze works like the fog of war in a strategy game: a dark patch is unexplored, not empty. The comparison stops there. In a game, scouting reveals the truth. Here, even the parts the phone saw carry an error bar.

      +
      + + A check whose ground was never seen stays UNSURE and names the missing view + A candidate battery spot on the wall. The ground beside it is covered by grey haze. Beside it a check reads: ground near the spot, checks need 6 to 13 ft of seen ground, UNSURE, missing evidence: show the ground about 6 ft right of your meter. + + + + + + meter + + + candidate spot + + never seen + + Ground near the spot + checks need 6 to 13 ft of ground + + ? UNSURE + missing evidence: show the + ground 6 ft right of your meter + +
      If the camera never saw part of the area a rule depends on, that check cannot pass. It stays UNSURE, and the server names the view that would settle it. Here the ground beside the spot was never seen, and the checks that use ground need 6 to 13 ft of it. A blank on the map is never treated as empty ground.
      +
      + +

      Coverage means 100% of what the model needs

      +

      The app doesn't need the whole house. It needs every part the rules depend on, all of it. That is a fixed zone around the meter, so the app can say exactly what is left, and the scan can end when the zone is clear.

      +
      + + The zone a scan must cover, seen from above + A plan view of the house with the electric meter on its front wall. A dashed blue rectangle marks the zone the checks need: the wall 20 feet either side of the meter and the ground in front of it out to 13 feet. Everything outside it is grey haze labelled not needed. Inside, a bush stands in front of the wall; the strip of wall behind it is unseen, so checks there are UNSURE. A dotted path shows the phone walking along the zone. + + + + + house, seen from above + + + + + cable reach: 20 ft either side + + + + + not needed + + + + + meter + + + + wall behind the bush: + unseen, so UNSURE + + + + needed: all of it + ground out to 13 ft + +
      Drawn to scale, 1 ft to 8 px. The zone is the wall within 20 ft of cable either side of the meter, the ground in front out to 6 to 13 ft depending on the check, and the space overhead for headroom, which a plan view can't show. The coverage map has to be honest. The app's first map checked only range, angle and framing, and claimed 1.1 ft of wall that no photo saw. Tested against depth, the recon worker claims at most about 0.46 ft.
      +
      + +

      LiDAR first, never required

      +

      Some iPhones have LiDAR, a sensor that measures depth directly by timing reflected light. Most homeowners' phones don't. The app uses LiDAR and other depth sensors whenever the phone has them, and every check must also work without them.

      +
      + + Two ways to get depth at real scale + Left, a phone with LiDAR sends rays to a wall seen from the side, and dots on the wall mark measured depth; its reach is about 5 metres. Right, any iPhone: a dashed line shows a depth model's guess of where the wall is, a little off. The phone at two known positions sees the same point on the real wall, and the two sight lines meet there, which fixes the scale. + + Phone with LiDAR + Any iPhone + + + + + + + + + measured directly + reach about 5 m + + + + + model's guess + + + + + + + known move + guessed, then rescaled + by triangulating with poses + +
      LiDAR works like a laser tape measure: direct distances, but a reach of about 5 m, so capture has to stand within about 4 m of the wall. Without it, a learned model estimates depth in each photo, and the phone's poses fix the scale: the same point seen from two known positions pins down its distance, the way two eyes judge depth. Where that comparison breaks: your eyes sit a fixed distance apart, but the phone's move between photos comes from its tracking and carries that error. Both paths fill the same capture packet.
      +
      + + + +

      4. How to build the 3D model: the options

      +

      Problem 2, photos and metadata into a 3D model, is open. The server team is comparing three families of methods: broadly first, then in depth on the best. None is ruled out. The numbers so far come from public datasets whose buildings were laser-scanned, so the true shape is known to a fraction of an inch.

      + +
      +
      +

      Traditional reconstruction, fused with LiDAR

      +

      Geometry from first principles: find the same points in several photos, work out where they sit in 3D, and merge the depth from every view into one surface. LiDAR depth drops straight into that merge.

      +

      The known weakness is that matching points across photos struggles on blank siding and has no scale of its own. LiDAR and the phone's poses supply the scale.

      +

      Evidence: the recon worker (PR #20) merges depth into one surface. With a laser scan standing in for LiDAR, its walls come out at 2.1 in (p90).

      +
      +
      +

      Learned depth plus the phone's poses

      +

      A neural network estimates depth from a single photo. Its own sense of scale is off by 4 to 12%, so the photos are rescaled with the phone's poses and then merged.

      +

      MapAnything, a learned model that reads several photos at once, was not a stable baseline: its scale shifted with the input resolution.

      +

      Evidence: walls about 5 in (p90) at a realistic phone error, 2.8 in with exact poses. Edges 8 in or worse in every setting.

      +
      +
      +

      World models

      +

      Large learned models that take photos or video and produce a whole 3D scene at once, camera positions included. Think of someone who has seen thousands of houses sketching this one from a few snapshots.

      +

      The sketch comparison also shows the risk: a good sketch fills in what the artist never saw. That is fine for a picture, but a check needs to know which parts were observed.

      +

      Evidence: not evaluated yet. The server team will.

      +
      +
      + +
      +

      Wall error at p90 for each 3D path tested so far, on one laser-scanned building

      + + Wall error at p90, in inches, for each 3D path tested so far + A dot plot from 0 to 20 inches. Depth model alone: about 20 inches. Depth model rescaled with poses at a realistic 2 percent pose error: 5.0 inches, interval 3.8 to 6.8. With exact poses: 2.8 inches, interval 2.0 to 4.4. Edges with learned depth: 8 inches or worse in every setting. Recon worker, photos only: 2.8 inches. Recon worker with a laser scan standing in for LiDAR: 2.1 inches. World models: not evaluated yet. + + + + 05101520 inwall error, p90 (9 in 10 points are closer) + + + + depth model alone + + poses, 2% error + + exact poses + edges, any setting + recon, photos only + recon, laser as LiDAR + world models + + + + + + + + + + + + + + about 20 in + 5.0 in (3.8 to 6.8) + 2.8 in (2.0 to 4.4) + 8 in or worse + 2.8 in + 2.1 in + not evaluated yet + + + Depth model alone, its own scale: p90 about 20 in + Depth rescaled with phone poses at a 2% pose scale error: p90 5.0 in, 95% interval 3.8 to 6.8 + Depth rescaled with exact poses: p90 2.8 in, 95% interval 2.0 to 4.4 + Edges, learned depth: 8 in or worse in every setting + Recon worker, photos only: wall p90 2.8 in + Recon worker, laser scan as LiDAR: wall p90 2.1 in + World models: not evaluated yet + + +
      All rows come from the ETH3D electro building, walls within 6 m. The top four are learned depth (MoGe-2 and Depth Anything 3); the next two are the recon worker; the last has no data yet. Brackets are 95% intervals where they exist. Caveat: the recon worker's two runs chose different walls, so those two numbers are not measured on the same wall. Reading it: learned depth alone is too rough for clearances, which are measured from edges. With LiDAR or the phone's poses supplying scale, walls come within a few inches.
      +
      Data table +
      + + + + + + + + + + +
      PathSettingWall error, p90Source
      Learned depthmodel's own scale (4 to 12% off)about 20 inevals, PR #12
      Learned depthrescaled with poses, 2% pose scale error5.0 in (3.8 to 6.8)evals, PR #12
      Learned depthrescaled with exact poses2.8 in (2.0 to 4.4)evals, PR #12
      Learned depthedges, every setting8 in or worseevals, PR #12
      Recon workerphotos only2.8 inrecon, PR #20
      Recon workerlaser scan standing in for LiDAR2.1 inrecon, PR #20
      World modelsnot evaluatedserver team, next
      +
      +
      + +

      What any path has to show

      +
        +
      • Real scale. A model's own sense of size is off by 4 to 12%, which is 5 to 14 in across a 10 ft stretch of wall.
        • +
      • Edges within the error allowance. Clearances run from the edge of a door, window or gas meter, and edges are where learned depth is worst.
        • +
      • Which parts were observed. The rules can pass a check only over area the phone saw. A method that fills in plausible wall has to mark what it filled in.
        • +
      • It runs on the packet the phone actually sends. That includes phones without LiDAR.
        • +
      + + +

      5. Where each piece lives

      +

      Files marked with a pull request exist only on that branch until it merges. Each component's README wins over this page.

      +
      + + + + + + + + + + +
      PieceSource of truthState
      iPhone app
      client team      		ios/README.mdGuided capture in PR #10; live 3D map on branch t3/ios-map3d, stacked on it
      Capture packet
      client team      		packet/README.mdBeing specified on branch t3/packet
      Rules engine, API, capture contract
      server team      		server/README.md, "What settles each check"PR #11; live as a public demo with public rules, and privately with Base's rules behind a key
      Reconstruction worker
      server team      		recon/HANDOFF.mdDraft PR #20, handed to the server team
      Accuracy evalsexperiments/evals/README.mdPR #12
      Agent rules, system overviewAGENTS.md, docs/00-overview.mdOn main
      Web review pageParked, PR #15
      +

      The day-1 plan (docs/01 to 05) is kept for its reasoning and rule citations, and the day-1 explainer, with its drag-the-battery demo, is eli5.html. The dated briefings in docs/briefings/ hold the evidence behind several figures here.

      + +
      Base's private thresholds, prompt names and field names do not appear on this page. The drawings are illustrations at a consistent scale, not a real house. Evidence numbers come from public datasets with laser-scanned ground truth; no dataset images are included.
      + +
      + + From a205d4a7a5e9a009da61dcfcc6d3e8b943e63294 Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 13:00:56 -0500 Subject: [PATCH 07/16] docs: keep only the current plan and the facts that still hold - Delete the day-1 feature map, implementation plan, stack research, eli5 explainer and morning briefing. Their lanes, schedules and superseded architecture are gone; what still holds moved first. - docs/00 now holds the plan, decisions with reasons, the measurement conventions the code cites (pixel ray, keyframe spacing, mesh rays, default error bars) and the evidence, including meter reading and coverage numbers from the briefing. - docs/04 gains the public rule table and the model, imagery and dataset license facts, with Esri's terms scoped to what Esri allows. - AGENTS.md and README.md are cut to one job each: rules for agents, and a map of the repository with what is live. - docs/05 loses its Expo question, lane wording and links to deleted files; how-it-works points at docs/00 and 04 and says the app uploads only scene.json today. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 47 +- CONTRIBUTING.md | 12 +- README.md | 78 +-- docs/00-overview.md | 154 +--- docs/01-feature-map.md | 104 --- docs/02-implementation-plan.md | 154 ---- docs/03-stack-research.md | 242 ------- docs/04-prior-art-and-codes.md | 130 ++-- docs/05-live-guided-survey-hld.md | 22 +- docs/briefings/2026-09-26-morning.html | 930 ------------------------- docs/eli5.html | 615 ---------------- docs/how-it-works.html | 10 +- 12 files changed, 167 insertions(+), 2331 deletions(-) delete mode 100644 docs/01-feature-map.md delete mode 100644 docs/02-implementation-plan.md delete mode 100644 docs/03-stack-research.md delete mode 100644 docs/briefings/2026-09-26-morning.html delete mode 100644 docs/eli5.html diff --git a/AGENTS.md b/AGENTS.md index 9ea643ceb..97469e193 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,52 +1,45 @@ # Agent guide -A homeowner scans the outside of their house around the electric meter with an iPhone. The system works out whether a Base Power battery fits there and where, and the app shows the spot in AR. It is a four-person hackathon project that started on 2026-09-25. The MVP is three problems: a phone that guides the homeowner until everything the model needs is captured, photos and their metadata turned into a 3D model, and that model evaluated against the placement criteria. +A homeowner scans the wall around their electric meter with an iPhone; the server decides whether a Base Power battery fits and where, and the app shows the spot in AR. [README.md](README.md) maps the repository; [docs/00-overview.md](docs/00-overview.md) holds the plan, decisions and evidence. ## Who owns what -- **Client team.** Sam, working with AI agents, owns the iOS app in `ios/`: guided capture, the live on-device 3D map with fog of war, and the capture packet it sends to the server. Aiden films video and gathers sample datasets. -- **Server team.** Hunter, working with his own agents, owns everything after the packet: building a 3D model or point cloud, then evaluating the criteria on it. +- **Client team.** Sam, working with AI agents, owns the iOS app in `ios/` and the capture packet it sends. Aiden films video and gathers sample datasets. +- **Server team.** Hunter, working with his own agents, owns everything after the packet: the 3D model and the rule checks, in `server/` and `recon/`. Edit the other team's files only with that team's OK, and talk to the author before editing files that an open pull request also changes. Two writers on one file cost more time than asking does. ## Where to look -Read the row for your task. Files marked with a pull request exist only on that branch until it merges. +A component's README wins over the docs. Files marked with a pull request exist only on that branch until it merges. | Task | Read | | --- | --- | -| What the system is, what's decided, what's open, and corrections that supersede the day-1 plan | `docs/00-overview.md` | -| A picture-first walkthrough of the current system | `docs/how-it-works.html` | -| The iOS app | `ios/README.md` | -| The capture packet the app sends | `packet/README.md` (coming on branch `t3/packet`) | -| The rules engine, API and capture contract | `server/README.md` (PR #11), especially "What settles each check" | -| Turning photos into a 3D model | `recon/HANDOFF.md` (PR #20) | -| Accuracy evals on public datasets | `experiments/evals/README.md` (PR #12) | -| Branches, checks and TestFlight | `CONTRIBUTING.md` | -| The rules table and the `scene.json` data format | `docs/01-feature-map.md` | -| How each part was planned, with code snippets and risks | `docs/02-implementation-plan.md` | -| Libraries and open-source projects per part, licenses, gotchas, what to skip | `docs/03-stack-research.md` | -| Base's public rules, competitors, NEC, IRC and Austin Energy citations | `docs/04-prior-art-and-codes.md` | -| The live guided-survey design: automatic capture, coverage display, next-view planning | `docs/05-live-guided-survey-hld.md` | - -`docs/01` to `05` and `docs/eli5.html` are the day-1 plan and explainer. Where they differ from `docs/00` or a component README, those win. +| The iOS app | `ios/README.md` (PR #10 for guided capture) | +| The capture packet | `packet/README.md` (PR #22) | +| The rules engine, API and scene contract | `server/README.md` (PR #11), especially "What settles each check" | +| Photos to a 3D model | `recon/HANDOFF.md` (PR #20) | +| Accuracy evals and the field test | `experiments/evals/README.md` (PR #12) | +| Measurement conventions the code relies on | `docs/00-overview.md` | +| Public rule values, code citations, model and imagery licenses | `docs/04-prior-art-and-codes.md` | +| The live guided-survey design | `docs/05-live-guided-survey-hld.md` | +| Branches, CI checks and TestFlight | `CONTRIBUTING.md` | ## Rules - **Keep Base's materials in `private/`, which git ignores.** The repository is public. Don't commit, quote or summarize those materials in tracked files, including prompt text, prompt names, output field names and internal thresholds. Team notes go in `private/internal-notes.md`. Real captures, photos of homes and dataset images go in `captures/`, `fixtures/real/` or `data/`, which git also ignores. -- **Make placement decisions in deterministic code.** Models build the geometry and recognize things: object boxes, the panel brand, the meter's text. Plain code over that geometry makes the pass or fail decision, so every answer traces back to a rule and a measurement. Clearance numbers live in rules files with their sources, never in code. `server/rules.yaml` (PR #11) holds the public values, each citing a code, Base's public help page, or marked as a demo placeholder. Base's values go in the git-ignored `private/rules.yaml`, which the server merges over the public file when it exists. +- **Make placement decisions in deterministic code.** Models build geometry and recognize things; plain code passes or fails each check, so every answer traces to a rule and a measurement. Clearance numbers live in rules files with their sources, never in code. - **Treat unseen as unsure, never as clear.** A gap in the scan could hide a gas meter. When a check depends on an area nobody observed, return UNSURE and name the view that would settle it. -- **Use StratMap or NAIP for aerial imagery.** Google Maps, Street View, Mapbox and Esri forbid running ML on their imagery in their terms. +- **Use StratMap or NAIP for aerial imagery.** Google's and Mapbox's terms forbid running ML on their imagery, and Esri allows it only inside ArcGIS for non-commercial use. - **Place AR results relative to the meter's anchor, with `.gravity` world alignment.** ARKit corrects anchors as tracking improves, so a result tied to the meter moves with it while raw world coordinates drift. `.gravityAndHeading` depends on the compass, which is unreliable next to a house. - **Rotate the intrinsics whenever you rotate a camera image.** Camera images are landscape sensor images and the intrinsics match that orientation. A rotated image with unrotated intrinsics produces wrong 3D geometry without any error. - -Decided: the app is native iOS only (Swift, ARKit, RealityKit), because Expo has no first-class ARKit support and capture needs LiDAR mesh export, per-frame position and lens data, and mesh raycasts. It uses LiDAR and other depth sensors when present and never requires them, since most homeowners' phones lack LiDAR. +- **Use LiDAR when present, never require it.** Most homeowners' phones lack LiDAR, so every check must also work from the camera and the phone's poses. ## Working in the repository -- Layout: `ios/` is the capture app, its project generated by XcodeGen. `server/` is the Python server (uv). `web/` is a browser toolchain for the reviewer view and zero-install capture experiments. `experiments/` holds one folder per experiment. `sites/landing` is the landing page, a submodule; change it in its own repository. -- `make check` runs every suite; `make ios`, `make server` and `make web` run one each. `CONTRIBUTING.md` lists the matching CI checks. +- `make check` runs every suite; `make ios`, `make server` and `make web` run one each. - Branch from `main`, keep one writer per branch, and open a pull request. People merge. Agents never push to `main` or merge, so a person sees every change before it lands. -- Several agents share one Mac. Exit code 137 means the system killed the process, usually for memory. Find the large allocation before rerunning, because macOS swaps before it kills and one runaway process can freeze the whole machine. -- Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` (copy `Local.xcconfig.example`; see `ios/README.md`), not in Xcode's Signing & Capabilities pane. The pane writes into `project.pbxproj`, and CI fails on that drift. +- Several agents share one Mac. Exit code 137 means the system killed the process, usually for memory. Find the large allocation before rerunning, because one runaway process can freeze the whole machine. +- Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` (copy `Local.xcconfig.example`), not in Xcode's Signing & Capabilities pane. The pane writes into `project.pbxproj`, and CI fails on that drift. - After editing `ios/project.yml`, run `make ios-project` (it needs XcodeGen 2.46.0) and commit the regenerated project, because CI regenerates it and fails on any difference. +- `sites/landing` is a submodule; change the landing page in its own repository. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d9fd21641..0ce41fefe 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -48,7 +48,7 @@ Keep workflows that run pull-request code away from production credentials and d ### Before the first upload -App Store Connect rejects a build without an app icon. House Scan needs an `AppIcon` set with a 1024×1024 image in an asset catalog inside `ios/HouseScan/`, added by the lane A owner. Measure Lab has one. +App Store Connect rejects a build without an app icon. House Scan needs an `AppIcon` set with a 1024×1024 image in an asset catalog inside `ios/HouseScan/`, added by the client team. Measure Lab has one. Neither `Info.plist` sets `ITSAppUsesNonExemptEncryption`, so each build waits under "Missing Compliance" until someone answers the encryption question on its TestFlight page. Setting the key to `false` removes that step for an app that uses only HTTPS and Apple's system encryption. @@ -67,11 +67,11 @@ Neither `Info.plist` sets `ITSAppUsesNonExemptEncryption`, so each build waits u Set the secrets and variables with the [GitHub CLI](https://cli.github.com), so the key never appears on screen. `gh secret set` without `--body` prompts without echoing. ```sh - base64 -i AuthKey_.p8 | gh secret set ASC_KEY_P8 --env testflight --repo SamGu-NRX/house-scanning - gh secret set ASC_KEY_ID --env testflight --repo SamGu-NRX/house-scanning - gh secret set ASC_ISSUER_ID --env testflight --repo SamGu-NRX/house-scanning - gh variable set APPLE_TEAM_ID --env testflight --repo SamGu-NRX/house-scanning --body - gh variable set BUNDLE_ID_PREFIX --env testflight --repo SamGu-NRX/house-scanning --body + base64 -i AuthKey_.p8 | gh secret set ASC_KEY_P8 --env testflight --repo SamGu-NRX/house-scanning-master + gh secret set ASC_KEY_ID --env testflight --repo SamGu-NRX/house-scanning-master + gh secret set ASC_ISSUER_ID --env testflight --repo SamGu-NRX/house-scanning-master + gh variable set APPLE_TEAM_ID --env testflight --repo SamGu-NRX/house-scanning-master --body + gh variable set BUNDLE_ID_PREFIX --env testflight --repo SamGu-NRX/house-scanning-master --body rm AuthKey_.p8 ``` diff --git a/README.md b/README.md index d17833b12..e52ed7dda 100644 --- a/README.md +++ b/README.md @@ -1,67 +1,39 @@ # House scanning -We're exploring a simpler way to survey a home for a Base Power battery. +A homeowner walks the outside wall around their electric meter with an iPhone. The app guides the scan until it has seen everything the placement rules need, and sends what it captured. The server builds a 3D model, decides in plain code whether a Base Power battery fits and where, and the app shows the spot in AR. -A homeowner walks the outside of their house with a phone app. The app measures the wall around the electric meter, and our code decides whether a Base Power battery can be installed there and, if so, where. The homeowner sees the spot in AR before they put the phone away. +This is a four-person hackathon project for Base Power, started 2026-09-25. Success means one capture session gives Base enough to decide, with no follow-up photos. -This is a four-person hackathon project for Base Power. Success means one capture session gives Base enough to decide, with no follow-up photos. A person can still review the result. +Two teams share the work. The client team (Sam with AI agents; Aiden on video and sample data) builds the iPhone app and the capture packet it sends. The server team (Hunter with his agents) turns the packet into a 3D model and checks the rules on it. -Since 2026-09-26 the work is split between two teams. The client team builds the iPhone app, which guides the scan with a live 3D map and sends a capture packet. The server team turns the packet into a 3D model and evaluates the placement rules on it. [docs/00-overview.md](docs/00-overview.md) has the details. +## Where to start -## Why measure instead of photograph - -Base's site survey starts from a few customer photos, and three things go wrong with them: - -- Distances are estimated by eye, so borderline clearances go to a reviewer. -- Unordered photos can't show whether two things are on the same wall or how far apart they are. -- Nobody knows what sits just outside the frame, so the homeowner gets asked for more photos. - -An AR session tracks the phone's position as it moves. The app can measure the wall in feet, and it knows which stretches nobody has looked at yet. It asks for those while the homeowner is still standing there. - -## How it works - -1. The homeowner marks the electric meter, walks the wall to its end in each direction, and takes close-ups of the meter and the electrical panel. A live 3D map on the phone shows what has been seen, and the app guides the homeowner until everything the rules need is covered. The app sends a capture packet: photos, where the camera was for each one, lens data, motion-sensor readings, LiDAR depth when the phone has it, and the homeowner's marks. -2. The server builds a 3D model from the packet and lays the walls out as one line, measured in feet from the meter. It slides the battery's footprint along that line and checks each clearance: gas equipment, driveways, doors and windows, room overhead, and the gap to a facing fence. Each check returns pass, fail, or unsure, with the measurement error counted. Anything the phone never saw counts as unsure. -3. The app shows the chosen spot, the battery, and the cable route from the meter in AR. - -Plain code makes the placement decision. Models build the geometry and recognize things, such as where a gas meter is or what the meter label says. The public demo uses public rule values plus two labeled placeholders (pool 10 ft, driveway 5 ft); Base's own values load from a private file. - -## What we've tested so far - -The plan targets iPhones, and a LiDAR iPhone makes some measurements easier. Most homeowners don't have one, so the experiments ask how far an ordinary phone gets. - -- **Can AR taps on the ground and the wall measure the wall line, the room overhead, and the gap to a fence without LiDAR?** Measure Lab (PR #7) is the test rig. A field session against a tape survey is the remaining measurement. -- **Can learned 3D models build accurate geometry from ordinary photos?** We measured MapAnything, Depth Anything 3 and MoGe against the laser-scanned buildings in the ETH3D dataset (PR #12). MapAnything's scale shifted with input resolution. A model's own scale is 4 to 12% off. Rescaled with the phone's poses, walls come within about 5 in (p90) at a realistic phone error, but edges stay at 8 in or worse. The reconstruction worker (PR #20) gets walls to 2.1 in with a laser scan standing in for LiDAR, and 2.8 in from photos only. World models are not evaluated yet. -- **How far does phone AR tracking drift during a walk?** On ADVIO's 2018 iPhone walks it ran 2 to 3 times over the server's error allowance, partly because of noise in the dataset's own reference. A current iPhone, on the MARViN dataset, stayed inside it, though that phone has LiDAR and the reference may be scaled to its tracking. The field test measures our own phone. - -A convincing 3D view can still miss an obstacle or get a distance wrong, so every result keeps its original photos beside it. +| Read | For | +| --- | --- | +| [docs/how-it-works.html](docs/how-it-works.html) | A picture-first walkthrough of the system, about fifteen minutes | +| [docs/00-overview.md](docs/00-overview.md) | The plan, the decisions and their reasons, the evidence so far, and open questions | +| [AGENTS.md](AGENTS.md) | The rules for anyone changing this repository, person or agent | +| [CONTRIBUTING.md](CONTRIBUTING.md) | Branches, CI checks and TestFlight | ## Repository map -| Path | What it is | State on 2026-09-26 | +Paths marked with a pull request exist only on that branch until it merges. + +| Path | What it is | State | | --- | --- | --- | -| `ios/` | The native iPhone app (Swift, ARKit, RealityKit) | On `main`: an AR session skeleton. Guided capture is in PR #10; the live 3D map is on branch `t3/ios-map3d`, stacked on it | -| `packet/` | The capture packet the app sends | Being specified on branch `t3/packet` | -| `server/` | The deterministic rules engine and API | On `main`: a skeleton. The engine is in PR #11 and runs as a public demo | -| `recon/` | A worker that turns photos and depth into a 3D model and a coverage map | Draft PR #20, handed to the server team | -| `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7 | -| `web/` | Browser toolchain for the reviewer view and zero-install capture experiments | The review page (PR #15) is parked | -| `docs/` | The overview, explainers, the day-1 plan and dated briefings | | +| `ios/` | The iPhone app (Swift, ARKit, RealityKit) | On `main`, an AR session that shows tracking. Guided capture is in PR #10; the live 3D map is in PR #21 | +| `packet/` | The capture packet's spec, validator and samples | PR #22 | +| `server/` | The rules engine and placement API (Python, uv) | On `main`, a skeleton. The engine is in PR #11 | +| `recon/` | Turns photos and depth into a 3D model and a coverage map | PR #20, handed to the server team | +| `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, meter reading in PR #16 | +| `web/` | Browser toolchain for a reviewer view | The review page (PR #15) is parked | +| `docs/` | The overview, the walkthrough, public rules and code citations, and the live-survey design | | | `sites/landing` | The landing page, a submodule | Change it in its own repository | -The public demo server, with public rules only, is at https://house-scanning-server.vercel.app. A second deployment loads Base's rules and requires a key. TestFlight builds are started by hand; see [CONTRIBUTING.md](CONTRIBUTING.md). - -## Read next +## What's live -| File | What it covers | -| --- | --- | -| [docs/how-it-works.html](docs/how-it-works.html) | A visual explainer of the current system: the two-team pipeline, error bars, coverage and the 3D path options | -| [docs/00-overview.md](docs/00-overview.md) | The goal, the team split, decisions made so far, evidence and open questions | -| [docs/01-feature-map.md](docs/01-feature-map.md) | Day-1 plan: the four work lanes, features by priority, the `scene.json` format, and the rules table | -| [docs/02-implementation-plan.md](docs/02-implementation-plan.md) | Day-1 plan: how each part gets built, the schedule, and risks | -| [docs/03-stack-research.md](docs/03-stack-research.md) | The open-source projects and libraries we build on, and the ones we skip | -| [docs/04-prior-art-and-codes.md](docs/04-prior-art-and-codes.md) | Base's public rules, similar products, and electrical-code citations | -| [docs/05-live-guided-survey-hld.md](docs/05-live-guided-survey-hld.md) | Proposed live guidance and automatic capture architecture, diagrams, evidence flow, and technical resources | -| [docs/eli5.html](docs/eli5.html) | The day-1 visual walkthrough, with an interactive demo of the placement search | +- The demo server at https://house-scanning-server.vercel.app runs the engine from PR #11 with public rules only, and every answer says so. `GET /health` shows which rules are loaded. A second deployment loads Base's rules and requires a key. +- TestFlight builds of the app and Measure Lab start by hand from the Actions tab; see [CONTRIBUTING.md](CONTRIBUTING.md). +- `make check` runs the server, web and iOS suites that CI runs. -Contributors and coding agents start with [AGENTS.md](AGENTS.md). This repository is public, so materials Base gave the team stay in the git-ignored `private/` folder. +This repository is public. Materials Base gave the team stay in the git-ignored `private/` folder, and photos of real homes never enter git. diff --git a/docs/00-overview.md b/docs/00-overview.md index c34b1caca..b47e0bbbc 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -1,130 +1,52 @@ -# House scanning: overview and decisions +# House scanning: plan, decisions and evidence -Hackathon project, 4-person team, started 2026-09-25. The goal: from a homeowner's phone, work out -**where a Base Power battery can be installed** against the outside of the house, and show the spot -back to them in AR. Updated 2026-09-26 midday for the team split; where this file and a component's README disagree, the README wins. +A homeowner scans the outside wall around their electric meter with an iPhone. The server works out whether a Base Power battery fits there and where, and the app shows the spot in AR. Success means one capture session gives Base enough to decide, with no follow-up photos. A person can still review any result. -> **This repo is public.** Materials Base gave the team live in `private/`, which is gitignored. Never commit, -> quote or summarize them here. Team-only notes are in `private/internal-notes.md`. +Base's survey starts today from customer photos, which have no scale, can't show whether two things share a wall, and miss whatever sits just outside the frame. An AR session gives every photo a position and a real scale, and the app knows what it hasn't seen while the homeowner is still there. -The MVP is three problems: +The work is three problems: guide the homeowner until the phone has captured everything the rules need, turn the photos and their metadata into a 3D model, and check that model against the placement rules. Full-house scans, roofs and Android are out of scope. Each component's README is the source of truth for that component and wins where it differs from this file. -1. A phone with a live camera feed that routes the homeowner until everything the model needs is captured. -2. Photos and their metadata turned into a 3D model. -3. The 3D model evaluated against the placement criteria. +## Two teams -## The core insight (why AR) +- **Client team.** Sam with AI agents; Aiden films video and gathers sample datasets. The app in `ios/` builds a live 3D map on the phone, with a haze ("fog of war") over what it hasn't seen, and guides the homeowner until everything the rules need is covered. Guided capture is in PR #10 and the live map in PR #21. The app sends a capture packet: photos, camera poses, intrinsics, motion-sensor readings, LiDAR depth and mesh when the phone has LiDAR, and the homeowner's marks. `packet/README.md` (PR #22) specifies it and lists what the app records today. For now the app uploads only its measurements, `scene.json`, and exports the full packet on request. +- **Server team.** Hunter with his agents. The packet goes in, a 3D model comes out, and the rules are checked on it. Starting points: the rules engine and API in `server/` (PR #11), the reconstruction worker in `recon/` (PR #20), the evals in `experiments/evals/` (PR #12), Hunter's research handoff (PR #18) and his [live-survey design](05-live-guided-survey-hld.md). -Today homeowners submit photos and reviewers judge them. Three problems come up that ordinary photos can't solve: -1. **Measuring distance.** A single photo has no real scale, so gaps and clearances get estimated by eye, and the borderline cases go to a person. -2. **Fitting photos together.** A handful of unordered photos can't reliably say whether two things are on the same wall, or how far apart they are. -3. **Coverage.** If part of the wall wasn't photographed, nobody can say what's there, so the homeowner gets asked for more photos. +## Decisions -AR capture fixes all three: the phone's tracking and depth give every photo a known position and a real scale, which a 3D -model can build on, and the app knows what it hasn't seen yet. The recognition half (panel brand, meter text, rust, whether -a window opens) stays with a vision model. AR adds nothing there beyond better photos. +- **Guided AR capture, not a video processed later.** Photogrammetry alone (COLMAP, splats) has no scale, is slow, and struggles on blank siding; COLMAP's own tutorial says to avoid texture-less walls. The packet carries ARKit's poses and any LiDAR depth, so the server's model starts at real scale. +- **3D modeling stays central,** with LiDAR and the phone's poses supplying scale, because learned depth alone is too rough at edges. +- **Plain code decides placement; models build geometry and recognize things.** The engine unrolls the walls into one line, `s`, in feet from the meter (negative to the left), slides the battery's footprint along it in 2 in steps, and checks each rule as an interval overlap or a polygon distance. +- **Every check answers PASS, FAIL or UNSURE.** A check passes only when its margin beats the measurement's error and fails only when it misses by more than the error. Anything else is UNSURE, including a check on an area nobody observed, and the answer names the view that would settle it. +- **Rule values live in rules files, each with its source.** `server/rules.yaml` (PR #11) holds the public values and two labeled placeholders (pool 10 ft, driveway 5 ft); Base's values load from a git-ignored private file. Battery sizes live there too, because models differ: Base Core is 39.5 in tall on a 30.68 × 22 in footprint, and older units are 3 × 3 ft. +- **Native iOS only.** Expo has no first-class ARKit support, and capture needs mesh export, per-frame pose and lens data, and mesh raycasts. +- **AR results hang off the meter's anchor, with `.gravity` alignment.** ARKit corrects anchors as tracking improves, and drift runs about 2 cm a second. `.gravityAndHeading` depends on the compass, which developers report off by up to about 176° near a house. -## Two teams (since 2026-09-26) +## Conventions the code relies on -``` - client team (iPhone) server team - ┌──────────────────────────┐ capture packet ┌──────────────────────────┐ - │ guided scan │ ─────────────────▶ │ 3D model or point cloud │ - │ live 3D map, fog of war │ │ │ │ - │ │ │ ▼ │ - │ battery spot in AR │ ◀───────────────── │ criteria: PASS/FAIL/ │ - │ (placed from the meter) │ answer │ UNSURE per check │ - └──────────────────────────┘ └──────────────────────────┘ -``` +- `capturedImage` is landscape, as the sensor reads it, and the intrinsics match it. Save it unrotated. Any rotation must also rotate the intrinsics and every image-space annotation. Gemini, the box detector (Claude's docs call its coordinates approximate), returns `[ymin, xmin, ymax, xmax]` on a 0 to 1000 scale, so apply EXIF rotation before inference and before casting. Depth maps are 256 × 192, so scale the intrinsics down to match. +- ARKit camera space is x right, y up, looking down −z, with image v growing downward. The ray through pixel (u, v) has direction ((u − cx)/fx, −(v − cy)/fy, −1), rotated by the camera-to-world transform. +- Keep a frame when the phone has moved 0.5 m or turned 15° since the last one kept. +- Mesh vertices are local to their `ARMeshAnchor`: multiply each by the anchor's transform while copying it out, and never write back into ARKit's buffer. Set `automaticallyConfigureSession = false`, or RealityKit turns mesh classification off. +- On the mesh, the gap to a facing fence is a ray straight out from the wall at 1.5 ft high, and headroom is a ray straight up from 1 ft out. The mesh has holes, so cast a small fan and take a robust minimum. LiDAR reads from about 0.5 m to 5 m, so capture stands 1 to 3 m from the wall, and a miss beyond 5 m is unknown, not open. +- Default error bars are 0.3 ft for an AR tap, 0.5 ft for the LiDAR mesh and 1.5 ft for a position from photo detection. They are untested estimates. The server adds 0.16 ft per foot along the wall from the meter, from measured ARKit drift. -- **Client team:** Sam with AI agents; Aiden films video and gathers sample datasets. The iOS app guides capture with a live - on-device 3D map, and a haze ("fog of war") covers what the phone hasn't seen. Coverage aims at 100% of what the model needs, - not the whole house. LiDAR and other depth sensors come first when present; the non-LiDAR path is built alongside. Guided capture - is in PR #10; the live 3D map is on branch `t3/ios-map3d`, stacked on it. The output is the **capture packet**: photos, camera - poses, intrinsics, IMU readings, LiDAR depth and mesh when present, and the homeowner's marks. It is being specified on branch `t3/packet`. -- **Server team:** Hunter with his agents. Packet in, a 3D model or point cloud out, then the criteria evaluated on it. They are - comparing world models with traditional and newer reconstruction methods, broadly first, then in depth on the best paths. - Starting points: `server/` (PR #11), the deterministic rules engine and API, live as a public demo at - https://house-scanning-server.vercel.app plus a private deployment with Base's rules behind a key; `recon/` (draft PR #20), - a working reconstruction worker; and the evals in `experiments/evals/` (PR #12). +## Evidence so far -## Decisions made +Public datasets with laser-scanned or surveyed ground truth. Sources: `experiments/evals/README.md` (PR #12), `recon/HANDOFF.md` (PR #20), `experiments/meter-closeup/README.md` (PR #16) and `experiments/panel-label/README.md` (PR #17). -- **AR-guided capture, not "upload a video and process it later."** Offline photogrammetry alone (COLMAP, splats) struggles on - blank siding and stucco, has no real scale, and is slow. ARKit already fuses the camera with the motion sensors on the phone, - and the capture packet carries those poses and any LiDAR depth, so the server's 3D model starts at real scale. -- **The phone collects data; the server decides.** Swift app → capture packet → Python FastAPI server (3D model, then rule checks) - → result → AR preview. -- **The placement code is plain code, not AI.** Models build the geometry and recognize things; the decision is ordinary code - over that geometry. Today's rules engine (PR #11) unrolls the house walls into one straight line `s` (feet along the wall - from the meter). Most rules become 1-D interval checks plus 2-D polygon distances. Sweep the battery footprint in small steps. - Every check returns PASS / FAIL / UNSURE, with error bars on each measurement. See `02-implementation-plan.md`. -- **Unseen is never clear.** A check whose area nobody observed is UNSURE, and the server names the view that would settle it. -- **The vision model only recognises things.** Day-1 plan: Gemini boxes → SAM 2 → cast onto the wall using where the phone was. - Recognition prompts run on the close-ups. -- **Native Swift + ARKit/RealityKit on iPhone, in `ios/`.** iOS only. LiDAR is used when present, never required: most homeowners - don't have it. Not Expo (no first-class ARKit), and not Android. -- **3D modeling stays central.** Learned depth alone is too rough at edges, so LiDAR and the phone's poses supply the scale. See - the evidence below. -- **Normal dev app for the hackathon; an App Clip is only the pitch's rollout story** (see the gotchas in `03-stack-research.md`). +- **Tracking.** A current iPhone (14 Pro Max, MARViN dataset) stayed inside the server's allowance: p90 position error 8.6, 13.4 and 18.5 in after 10, 20 and 30 ft, against 19.2, 38.4 and 57.6 in. That phone has LiDAR, and MARViN's reference may be scaled to its tracking. A 2018 iPhone 6s (ADVIO) ran two to three times over, partly from noise in that dataset's reference. +- **Learned depth.** A model's own scale is 4 to 12% off. Rescaled with the phone's poses, walls come within about 5 in at p90 at a realistic 2% pose error, and 2.8 in with exact poses. Edges, where clearances are measured from, stay at 8 in or worse. MapAnything's scale shifted with input resolution. +- **Reconstruction worker.** On the ETH3D building, wall p90 is 2.1 in with a laser scan standing in for LiDAR and 2.8 in from photos only, on different walls. +- **Coverage.** The app's first coverage map claimed 1.1 ft of a 19.3 ft wall that no photo saw. Requiring several view angles fixed that but dropped 6 to 8.5 ft of good wall; a depth test on LiDAR phones fixed it cleanly. The reconstruction worker, checked against depth, claims at most about 0.46 ft. +- **Meter reading.** Apple's on-device text recognition read the full meter number on 71 of 73 real photos but found the right line on only 21 of 75, because nameplates carry several numbers. A barcode held it on 20 of 24 photos that had one, and a list of three candidates held it on 85% of held-out photos. So the app offers three to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small. +- **Panel labels.** Untested: 7 open photos of US panels exist, against the 40 the test needs. +- **World models.** Not evaluated yet. -## Corrections made after the research (supersede earlier drafts) +## Open questions -1. Use `worldAlignment = .gravity`, **not** `.gravityAndHeading`. The compass is unreliable near a house - (developers report errors up to ~176°). Take one compass reading away from the meter as a rough hint. Which wall faces the - street comes from map data: Overture building outline + osmnx nearest road. -2. **Place the battery relative to the meter's anchor**, not in raw world coordinates. ARKit corrects anchors as it goes, - and drift is about 2 cm per second. -3. The battery is **39.5″ tall** (Base Core: 39.5 × 30.68 × 22 in), not 36″. Base's older units are 3 × 3 ft, - so keep all dimensions in the rules config. -4. Claude is **not** the box detector; its docs say its coordinates are approximate. Gemini returns boxes natively as - `[ymin, xmin, ymax, xmax]` on 0–1000. -5. **Never run ML on Google, Mapbox or Esri imagery.** Google's ToS §3.2.3(c) bans it outright. Use StratMap/NAIP (public domain). -6. **3D modeling is not ruled out** (2026-09-26). The morning briefing's Finding 2 first said "no photo-to-3D measurement pipeline". - The evidence says learned depth *alone* is too rough; with LiDAR or the phone's poses supplying scale, a 3D model is the plan. - -## Evidence so far (2026-09-26) - -From public datasets with laser-scanned ground truth. Sources: `experiments/evals/README.md` (PR #12) and `recon/HANDOFF.md` (PR #20). - -- A depth model's own scale is 4 to 12% wrong. -- Rescaled with the phone's poses, learned depth gets walls to about 5 in at p90 at a realistic phone error. Edges come out at - 8 in or worse, and clearances are measured from edges. -- The recon worker on the ETH3D building gets walls to p90 2.1 in with a laser scan standing in for LiDAR, and 2.8 in from photos only. -- Coverage tested against depth claims at most about 0.46 ft of wall that no photo saw. -- A current iPhone's tracking stays within the server's error allowance. -- World models have not been evaluated yet; the server team will. - -## Open questions (team must decide) - -- **Which clearance numbers to use.** The public baseline is Base's help page: within 20 ft of the meter, within 1 ft of the wall, - and 3 ft from gas meters, fences, AC units and other batteries. The public demo fills the gaps with labeled placeholders - (pool 10 ft, driveway 5 ft) in `server/rules.yaml` (PR #11). Base's own values load from the git-ignored private rules file. - The team's working notes are in `private/internal-notes.md`. -- **Whether the install includes a separate battery disconnect box** beside the meter. Base's public pages mention a wall-mounted - transfer switch and a battery disconnect. -- **Which 3D path.** Traditional reconstruction fused with LiDAR, learned depth rescaled with the phone's poses, or world models - (large learned models that produce a whole 3D scene from photos or video). Whichever wins needs real scale, edges within the - error allowance, and an honest account of which parts it observed and which it filled in. -- **Real-phone accuracy.** The evidence above comes from public datasets. A field session with a tape survey on a real wall - measures our own phone. -- **The day-1 "test first" list** in `03-stack-research.md`: LiDAR on siding in midday sun, tap-to-pixel orientation, - Gemini boxes on real photos, and compass accuracy at the meter. Check which the experiments have answered before relying on one. - -## Doc map - -| File | What | -|---|---| -| `00-overview.md` | this file: goal, insight, team split, decisions, evidence | -| `how-it-works.html` | the current visual explainer: the two-team pipeline, error bars, coverage, the 3D path options | -| `01-feature-map.md` | day-1 plan: lanes A–D, feature list with priorities, data format, rules table | -| `02-implementation-plan.md` | day-1 plan: how to build each part, code snippets, 24 h schedule, risks | -| `03-stack-research.md` | libraries and open-source projects per part, licenses, gotchas, what to skip | -| `04-prior-art-and-codes.md` | what Base does publicly, competitors, electrical-code citations, pitch angle | -| `05-live-guided-survey-hld.md` | live guided survey: automatic capture, coverage display, next-view planning, evidence flow | -| `eli5.html` | the day-1 visual explainer; step 4 is an interactive "slide the battery" demo that predates error bars | -| `briefings/` | dated team briefings | - -Each component's own README is its source of truth: `ios/README.md`; `server/README.md` (PR #11), especially "What settles -each check"; `recon/HANDOFF.md` (PR #20); `experiments/evals/README.md` (PR #12); and `packet/README.md` (coming on branch -`t3/packet`). The web review page (PR #15) is parked. +- **Clearance values.** [04-prior-art-and-codes.md](04-prior-art-and-codes.md) has the public values with their citations. Base's own values go in the private rules file, along with which number on the meter plate Base needs. +- **Disconnect box.** Does the install add a battery disconnect beside the meter? Base's public pages mention a wall-mounted transfer switch and a battery disconnect. +- **The 3D path.** Traditional reconstruction fused with LiDAR, learned depth rescaled with the phone's poses, or world models, which produce a whole 3D scene from photos or video. The winner needs real scale, edges within the error allowance, and a record of which parts it observed and which it filled in. +- **Accuracy on our phone.** Nothing above ran on our phone. The field test uses a current iPhone without LiDAR, the Measure Lab build (PR #7), a 30 ft tape and a real meter wall, following `experiments/evals/field/FIELD_SHEET.md` (PR #12); a dry run recovered a planted 1.5% scale error. It also checks the default error bars. LiDAR on siding in midday sun is untested too. +- **Pool and driveway.** Their checks need up to 13 ft of photographed ground. The alternative is one yes-or-no question, recorded as the homeowner's answer. +- **Hidden wall.** Something standing in front of the wall can hide it, and nobody owns that check yet. One proposal: the app shows the homeowner the photo of the chosen spot and asks, and runs the depth test on LiDAR phones. diff --git a/docs/01-feature-map.md b/docs/01-feature-map.md deleted file mode 100644 index b11d509fb..000000000 --- a/docs/01-feature-map.md +++ /dev/null @@ -1,104 +0,0 @@ -# Feature map - -> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. - -**Priority:** **P0** = needed for the demo · **P1** = strong add · **P2** = stretch -**Lanes (one teammate each):** **A** capture app · **B** object detection and geometry · **C** rules + placement code · **D** output, testing, pitch - -## Hour 1: agree on the data format (everyone) - -Every lane reads or writes this, so lock it first. Then all four people can work in parallel against a hand-built -`fixtures/fake-01/scene.json` until real captures exist. - -```json -{ - "meter": {"pos": [x,y,z], "wall_id": "w1"}, - "walls": [{"id": "w1", "baseline": [[x,z], ...], "height_ft": 9}], - "objects": [{"type": "window|door|garage_door|ac|gas_meter|elec_box|vent|downspout|pool", - "wall_id": "w1", "span_ft": [a, b], "bottom_ft": 3.1, "top_ft": 6.2, - "attrs": {"operable": true, "well": false}, "source": "vlm|tap", "conf": 0.8}], - "ground": [{"type": "drive|concrete|gravel|lawn|mulch|deck", "polygon": [[x,z], ...]}], - "overheads": [{"wall_id": "w1", "span_ft": [a, b], "clearance_ft": 3.0}], - "facing": [{"wall_id": "w1", "span_ft": [a, b], "depth_ft": 4.5}], - "keyframes": [{"id": "k1", "pose": [...16 floats, column-major cam_to_world...], "intrinsics": [fx,fy,cx,cy], - "w": 1920, "h": 1440, "img": "k1.jpg"}], - "stills": {"meter_close": "...", "meter_surround": "...", "panel_wall": "...", "panel_open": "..."}, - "gps": {"lat": 0, "lon": 0}, "heading": {"deg": 0, "accuracy_deg": 0} -} -``` - -- `facing` is the gap from the wall straight out to the nearest fence, hedge or wall. -- `span_ft` is the stretch along the wall an item covers, in feet from the meter (negative = left of the meter). -- Keyframe images are saved **unrotated** (landscape, as the sensor reads them), so `intrinsics` apply to them directly. - -## A: Capture app (Swift, ARKit/RealityKit) - -| # | Feature | Pri | Notes | -|---|---|---|---| -| A1 | Tap to mark the meter → anchor | P0 | The starting point; the result is later placed relative to it | -| A2 | Guided walk left/right, tap wall ends + corners | P0 | Can't finish until both ends are tapped, so nothing goes unphotographed | -| A3 | Wall + ground detection → wall line at ground level | P0 | Real feet | -| A4 | Save frames every ~0.5 m / 15° with position + lens data | P0 | Fork Stray Scanner's recorder | -| A5 | Tap-to-mark fallback: gas meter, door, window, AC, garage, driveway edge | **P0** | The demo never depends on detection. Also gives correct answers to test against | -| A6 | Guided close-ups: meter, panel wall, open panel, disconnect | P0 | Input for the recognition checks | -| A7 | Photo checks: sharpness, meter can fully in frame, text readable on the phone (Vision) | P1 | | -| A8 | One session from the meter outside to the panel inside | P1 | Answers "same wall?" and "directly behind the meter?" exactly | -| A9 | Mesh .ply export (anchor transform applied) + zip + foreground upload | P0 | | - -## B: Object detection and geometry (server) - -| # | Feature | Pri | Notes | -|---|---|---|---| -| B1 | Gemini boxes for window, door, garage door, AC, gas meter, electrical box, vent | P0 | Or tap marks from A5 | -| B2 | Cast each box onto the wall using the phone's position → s-range + height | P0 | Test first: tap a point, cast it back into a saved frame, check the pixel | -| B3 | Window details: ground-level from measured height, opens vs fixed, window well | P1 | | -| B4 | Passage width (wall out to the nearest fence/hedge) from mesh rays | P1 | LiDAR only; otherwise UNSURE | -| B5 | Overhead headroom (porch, eave, stairs) from mesh rays | P1 | LiDAR only | -| B6 | Ground surfaces + whether they connect to the driveway | P2 | Hardest. Demo uses a tapped driveway edge | -| B7 | Pool + pool-equipment detection | P2 | Also from the aerial photos | -| B8 | Street-facing wall from Overture outline + osmnx road (flag for homeowner approval of the look) | P2 | | - -## C: Rules + placement code - -| # | Feature | Pri | Notes | -|---|---|---|---| -| C1 | `rules.yaml`: every number, per-utility overrides (ComEd), a source per rule: code citation, Base's public help page, or a labeled demo placeholder | P0 | No hardcoded distances in the code | -| C2 | Sweep the 31″ × 22″ footprint along the unrolled wall, 2″ steps | P0 | | -| C3 | Clearance checks: gas, driveway, pool, openings, box above, vent, AC, passage width, headroom, ground | P0 | PASS / FAIL / UNSURE with error bars | -| C4 | Cable route along the wall: blocked by door, garage or a stretch with no wall; length ≤ max | P0 | | -| C5 | Result: pass / manual_review / reject + reasons + photos still needed | P0 | | -| C6 | Rank the valid spots: shortest run, prefer side walls | P1 | | -| C7 | Check the disconnect box's wall space + conduit route | P2 | Only if the current product still uses it | - -## D: Output, testing, pitch - -| # | Feature | Pri | Notes | -|---|---|---|---| -| D1 | AR preview: RealityKit box attached to the meter's anchor + cable line | P0 | The demo moment | -| D2 | 2D site plan (drawsvg): blue meter area, purple cable, red battery spot | P0 | | -| D3 | Tape-measure one real house: AR vs depth model vs the vision model's guesses | **P0** | The pitch's headline number | -| D4 | Baseline: Depth Anything 3 metric on plain photos | P1 | | -| D5 | Recognition prompts on the close-ups (prompts stay in `private/`) | P1 | | -| D6 | Meter type from the can's real height in inches | P2 | | -| D7 | Reviewer web view / JSON report | P2 | | - -## Rules table (C1): CONFIRM THESE - -Public sources only. The team's working numbers are in `private/internal-notes.md` (gitignored). - -| Rule | Public source | Value | Use | -|---|---|---|---| -| Distance from meter | Base help page | within 20 ft | 20 ft, TBD | -| Distance from wall | Base help page | within 1 ft | 1 ft | -| Gas meter / regulator | Base help page; Austin Energy §1.9; Texas Gas Service | 3 ft | 3 ft | -| AC units, fences, other batteries | Base help page | 3 ft | 3 ft | -| Doors and windows | IRC R328 (batteries); Austin Energy §1.9.2 (meters, ≥ 1 ft) | 3 ft | 3 ft | -| Working space at meter / panel | NEC 110.26 | 30″ wide × 36″ deep × 6.5 ft headroom | as listed | -| Driveway | — | — | **TBD** | -| Pool | — | — | **TBD** | -| Passage width (wall to fence) | — | — | **TBD** | -| Battery footprint | Base Core spec | 39.5″ × 30.68″ × 22″ (older units 3 × 3 ft) | per model | - -## Not doing - -Android, Expo, full-house or roof scans. diff --git a/docs/02-implementation-plan.md b/docs/02-implementation-plan.md deleted file mode 100644 index e6f3d82a1..000000000 --- a/docs/02-implementation-plan.md +++ /dev/null @@ -1,154 +0,0 @@ -# Implementation plan - -> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. - -The main trick is to unroll the house's walls into one straight line, measured in feet from the meter. Almost -every rule then becomes a check on a 1-D number line, and the placement code gets simple. - -## Overall setup - -``` -iPhone (Swift, ARKit) ──upload .zip──▶ Python server (FastAPI) ──result──▶ iPhone AR preview - collects measurements + photos 1. read the upload, build the wall line + 2D site plan - 2. find objects, cast them onto the wall - 3. rule checks → pick a spot - 4. recognition prompts on the close-ups -``` - -The phone only collects data; all the logic runs on the server. Placement code can then be tested against saved -captures without anyone walking a house. - -Repo layout: - -``` -/ios Xcode project -/server ingest.py geometry.py detect.py solver.py rules.yaml plan_svg.py prompts/ -/fixtures saved captures (fake-01 hand-built in hour 1; real ones gitignored) -/eval tape-measure ground truth -/docs these docs -/private materials not for publication (gitignored) -``` - -## Lane A: Capture app - -AR setup: - -```swift -let config = ARWorldTrackingConfiguration() -config.planeDetection = [.horizontal, .vertical] -config.worldAlignment = .gravity // NOT .gravityAndHeading (compass unreliable near a house) -if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) { - config.sceneReconstruction = .meshWithClassification // LiDAR phones only -} -if ARWorldTrackingConfiguration.supportsFrameSemantics(.sceneDepth) { - config.frameSemantics.insert(.sceneDepth) -} -arView.automaticallyConfigureSession = false // otherwise RealityKit turns classification off -arView.session.run(config) -``` - -Capture flow (P0). Use taps rather than automatic wall detection, so the demo can't break: -1. **"Tap the meter."** Raycast from the tap to a vertical surface and add an `ARAnchor`. That's the zero point. -2. **"Walk left until the wall ends, then tap the end."** Then right. Tap each corner. The taps define the wall line. -3. **Save frames while walking** (~every 0.5 m or 15°): `capturedImage` saved unrotated, `camera.transform`, - `camera.intrinsics`, optional `sceneDepth.depthMap` (256×192; the server scales K down to match). - `captureHighResolutionFrame` gives sharper stills and still includes the pose. -4. **Tap-to-mark buttons:** gas meter, door, garage door, window, AC, driveway edge. -5. **Guided close-ups** for the recognition checks. Photo checks run on the phone: sharpness (Laplacian variance), and Apple - Vision `RecognizeTextRequest` + `VNDetectBarcodesRequest` on the meter. -6. **Mesh export:** for each `ARMeshAnchor`, multiply every vertex by `meshAnchor.transform` (vertices are local to the anchor), - **copying them out** of ARKit's `MTLBuffer` rather than writing back into it. Indices are uInt32. Write the PLY by hand - so each face can keep its classification (wall/door/window). -7. Zip `scene.json` + JPEGs + `mesh.ply` and upload with a **foreground** `URLSession`. -8. Take GPS + one compass reading with `CLLocationManager` (trust it only if `headingAccuracy` ≤ 20°). - -## Lane B: Geometry and detection (server) - -**The wall line (`geometry.py`).** Chain the tapped points (left end → corners → meter → corners → right end) into -a line along the ground. Every point on the house gets one coordinate `s`: feet from the meter, negative to the left. -Each segment also stores its outward direction and ground height. - -**Casting a box onto the wall:** - -```python -def pixel_ray(u, v, K, cam_to_world): - # ARKit camera: +x right, +y up, looks down -z; image v grows downward (Rerun calls this RUB) - d = np.array([(u - K[0,2]) / K[0,0], -(v - K[1,2]) / K[1,1], -1.0]) - d = cam_to_world[:3,:3] @ d - return cam_to_world[:3,3], d / np.linalg.norm(d) - -def hit_plane(o, d, p0, n): - denom = d @ n - if abs(denom) < 1e-6: return None - t = ((p0 - o) @ n) / denom - return o + t*d if t > 0 else None -``` - -- **Things on the wall** (windows, doors, boxes): cast rays from the box's left and right edges into the wall plane → the `s` range + height above ground. -- **Things on the ground** (AC unit, gas meter): cast the mask's bottom-center into the ground plane, then snap to the nearest wall point. -- **Merging duplicates:** merge same-type detections whose `s` ranges overlap, and take a majority vote on attributes. -- ⚠️ `capturedImage` is landscape (the way the sensor reads it) and `intrinsics` match it. Rotate one and you must rotate the other. - Gemini returns `[ymin, xmin, ymax, xmax]` on 0–1000, so convert to pixels of the **unrotated** image. -- **Test this in hour 2:** tap a point, cast it back into a saved frame, and check it lands on the right pixel. - -**Mesh measurements (Open3D `RaycastingScene`, float32 rays, a miss = `inf`).** Every 2″ along the wall: -- Passage width: a ray straight out from the wall at 1.5 ft height. -- Headroom: a ray straight up from 1 ft out from the wall. -- The mesh has holes, so cast a small fan of rays and take a robust minimum. A miss beyond ~5 m (the LiDAR limit) means **unknown**, not "open". - -## Lane C: Placement code (`solver.py`) - -Every measurement carries an error estimate: ±0.3 ft for AR taps, ±0.5 ft for the mesh, ±1.5 ft for anything placed from -a model's photo detection. A check answers **PASS** if the margin is larger than the error, **FAIL** if it's clearly -violated, and **UNSURE** otherwise. - -```python -W = rules.battery.width_ft # 31/12 -for s in np.arange(scene.s_min, scene.s_max - W, 1/6): # 2-inch steps - spot = Spot(s, s + W, depth=rules.battery.depth_ft) - checks = [c(scene, spot, rules) for c in CHECKS] # gas, drive, pool, openings, box_above, - route = harness_route(scene, spot, rules) # vent, ac, front, overhead, ground, street_facing -``` - -- **Distance checks** (gas meter, pool, driveway): the spot is a 2D rectangle against the wall; use shapely `distance`. -- **Checks along the wall** (openings, AC, box above, vent): numpy overlap `max(a0,b0) < min(a1,b1)`. The ComEd rule widens each opening by 3 ft on both sides. -- **Cable route:** the path from `s=0` to the spot along the unrolled line. It fails if it crosses a door or garage interval or a stretch with no wall. - Its length is |s| plus a little per corner. Past a configurable "confident reach" it's UNSURE; over the max it's FAIL. -- **Decision:** - - **pass:** some spot passes every check. Pick the shortest run, preferring side walls. - - **manual_review:** some spot has no FAIL but at least one UNSURE. Report those reasons. - - **reject:** both wall ends were reached and every spot FAILs. - - If the wall wasn't walked to its end → manual_review + missing views. -- A brute-force sweep is enough: ~1,200 positions × tens of checks runs in under a second. No OR-Tools or z3. - -## Lane D: Output, prompts, testing - -- **AR preview:** the server returns the spot **relative to the meter's anchor**. The app attaches a `ModelEntity` - (`MeshResource.generateBox`, 31″ × 22″ × 39.5″) to that anchor, plus thin cylinders for the cable. Keep the AR session running - while the server works; relocalizing later with an `ARWorldMap` is P2. -- **2D site plan:** drawsvg; blue meter area, purple cable run, red battery spot and its clearance. -- **Recognition prompts:** load them from `private/` at runtime. `server/prompts/` is gitignored. Never commit prompt text; this repo is public. -- **Testing:** tape-measure 10–15 distances on one real house. Compare AR, Depth Anything 3 on plain photos, and the - vision model's guesses. The error table is the pitch slide. -- **Debugging:** Rerun. Log camera positions (`Pinhole`, `camera_xyz=RUB`), frames, mesh (static), wall line, detections and solver rays. - -## Schedule (~24 h; stretch it for longer) - -| Hours | Everyone | -|---|---| -| 0–2 | Lock `scene.json`. Hand-build `fixtures/fake-01`. Pick the rules numbers | -| 2–10 | Each lane builds against the fake capture: A tap flow + upload, B ray casting + detection, C solver + rules, D AR preview + SVG | -| 10–14 | **First full run on a real wall**, even with tap-only marking | -| 14–20 | Model detection in place of taps where it holds up, mesh measurements, error tuning, prompts on the close-ups | -| 20–24 | Tape-measure test, demo rehearsal on one house, pitch | - -## Risks and fallbacks - -| Risk | Fallback | -|---|---| -| Detection places objects badly | Tap-to-mark is P0 | -| Image orientation / box coordinates wrong | Round-trip test in hour 2 | -| No LiDAR phone | Planes-only mode: passage width and headroom become UNSURE. Borrow a Pro for the demo | -| AR drift | ~2 cm/s; attach results to the meter's anchor; show the tape-measure numbers | -| Driveway rule | Tapped driveway edge; automatic detection is future work | -| Sunlight washes out LiDAR | Scan the real wall at midday in hours 0–4 | diff --git a/docs/03-stack-research.md b/docs/03-stack-research.md deleted file mode 100644 index 4b835ba20..000000000 --- a/docs/03-stack-research.md +++ /dev/null @@ -1,242 +0,0 @@ -# Stack research: libraries, open-source projects, how they fit - -> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. - -This came from five parallel research agents on 2026-09-25 using live web searches. Anything marked -**UNVERIFIED** couldn't be confirmed. Licenses are what each repo or model card said on that date, so re-check them before shipping a product. - -## Data flow, with the tool at each step - -``` -PHONE (Swift) - ARKit: gravity alignment, wall + ground detection, LiDAR mesh, depth ← Apple "Visualizing and interacting with a reconstructed scene" - Frame recorder (position + lens data per frame) ← fork Stray Scanner (MIT) - Which frames to keep ← lift from arkit-3dgs-scanner (Apache-2.0) - Mesh → .ply ← write by hand - Meter close-up checked on the phone ← Apple Vision RecognizeTextRequest + VNDetectBarcodesRequest - GPS + one compass reading away from the meter - │ zip → foreground upload -SERVER (Python, FastAPI) - Read the capture ─────────── Open3D (read .ply, RaycastingScene, segment_plane) + numpy - Map context ──────────────── Census geocoder → Overture building outline (DuckDB) → osmnx nearest road - → street-facing wall; pools/driveways from OSM + samgeo on StratMap/NAIP - Object detection ─────────── Gemini boxes → SAM 2 tightens them → supervision merges them → cast onto the wall - Recognition ──────────────── recognition prompts on Gemini; zxing-cpp double-checks the barcode - Placement code ───────────── shapely + numpy sweep, rules.yaml (each rule cites its code) - Output ───────────────────── JSON + drawsvg 2D site plan - Debugging ────────────────── Rerun (its ARKitScenes example logs exactly our data) - Pitch baseline ───────────── Depth Anything 3 metric (Apache-2.0) on plain photos - │ result, positioned relative to the meter -PHONE: RealityKit box attached to the meter's anchor + cable line -``` - -## Why each piece matters - -| Piece | Rule(s) it serves | Why this tool, and what we're skipping | -|---|---|---| -| ARKit + LiDAR mesh | Passage width, headroom, gaps between openings, cable length | Real measurements (~±6–8 cm on walls, ~5 m range) instead of estimates by eye. | -| Guided walk to both ends | "please send more photos" follow-ups | Makes "unseen space" impossible | -| One continuous session | same wall? / panel behind meter | Shared coordinates; nothing pieced together from unordered photos | -| Stray Scanner fork | everything downstream | The only maintained MIT recorder of position + lens data + depth. No open-source app also exports the mesh | -| Open3D | passage width, headroom, wall fitting | One library for .ply + planes + batch rays | -| Gemini boxes | windows, doors, garage, AC, gas meter, vents | Best zero-shot box accuracy found; native 0–1000 box output | -| SAM 2 | every rule that uses an object's position | Vision-model boxes are loose; a mask's bottom edge is the right point to cast | -| Apple Vision on the phone + zxing | meter number, CL320 | Instant retake; serial-number OCR is the weak spot (63% vs 97%) | -| Overture + osmnx | which wall faces the street (flag for homeowner approval of the look) | The compass alone can't tell | -| samgeo on StratMap/NAIP | pool, driveway | OSM tags them on < 5% of homes; imagery ML is allowed to run on | -| rules.yaml + citations | all rules | Each "no" cites its source: NEC / IRC / Austin Energy / Texas Gas Service, Base's public help page, or a labeled demo placeholder | -| Rerun | debugging | Capture + solver reasoning in one timeline | -| Depth Anything 3 | pitch | The "AR beats photos" comparison | - -## Phone capture (iOS) - -**Use:** -- Apple sample **"Visualizing and interacting with a reconstructed scene"**: RealityKit + `.meshWithClassification`, mesh raycasts, `ARMeshGeometry` extensions. Plane detection flattens the mesh where planes are found, so walls come out clean. -- Apple sample **"Displaying a point cloud using scene depth"**: sceneDepth + confidence + intrinsics → points. -- The battery model is just `MeshResource.generateBox`. Skip Object Capture. - -**Lift code from:** - -| Repo | License | Last commit | Notes | -|---|---|---|---| -| [strayrobots/scanner](https://github.com/strayrobots/scanner) | MIT | 2026-04 | HEVC RGB, 16-bit depth PNG (mm), confidence, `odometry.csv` (position + quaternion + fx fy cx cy), IMU. No mesh. **Fork this** | -| [xiongyiheng/ARKit-Scanner](https://github.com/xiongyiheng/ARKit-Scanner) | Apache-2.0 | 2026-07 | ScanNet++'s capture app; good upload-to-server reference | -| [KuoFengYuan/arkit-3dgs-scanner](https://github.com/KuoFengYuan/arkit-3dgs-scanner) | Apache-2.0 | 2026-09 | Keyframe picker by movement/rotation, COLMAP export | -| [tkuehnl/iPadLIDARScanExport](https://github.com/tkuehnl/iPadLIDARScanExport) | UNVERIFIED | 2020 | ARMeshAnchor → world → OBJ approach; read it, don't copy (it writes back into ARKit's buffer) | -| [jc211/NeRFCapture](https://github.com/jc211/NeRFCapture) | MIT | 2023 | Small, readable, stale | -| [TokyoYoshida/ExampleOfiOSLiDAR](https://github.com/TokyoYoshida/ExampleOfiOSLiDAR) | MIT | 2021 | Snippets | - -**Skip:** -- **ios_logger:** no license, 2020. -- **ARKit-Data-Logger:** poses only. -- **SwiftUI-LiDAR:** no poses. -- **Nerfstudio iOS:** no app of its own; its importers are for closed apps. -- **RoomPlan:** indoor rooms only. - -**Gotchas:** -- `capturedImage` is in the sensor's landscape orientation, and intrinsics are in its pixels. -- Depth is 256×192, so scale K to match. -- LiDAR: stand 1–3 m from the wall (~5 m max, ≥ ~0.5 m min). -- `.gravityAndHeading` needs a location permission, doesn't re-align north after the session starts, and suffers magnetometer interference. -- **App Clips:** - - Size: 15 MB on iOS 16; 100 MB on iOS 17+ if digital-invocation only. - - ARKit isn't on the list of frameworks App Clips can't use. - - **No background URLSession.** - - Location is "When In Use" only, and it resets. - - Whether sceneReconstruction works inside a clip is UNVERIFIED. -- **Android:** only ARKit supports AR Foundation meshing. ARCore Depth is weak on blank walls and has no mesh. Not for v1. - -## Geometry server - -**Use:** - -| Library | Version / license | What for | -|---|---|---| -| [Open3D](https://pypi.org/project/open3d/) | 0.20.0, MIT | `segment_plane`, `detect_planar_patches`, `o3d.t.geometry.RaycastingScene.cast_rays` | -| [shapely](https://pypi.org/project/shapely/) | 2.1.x, BSD-3 | polygon distances; vectorised functions release the GIL | -| numpy | — | 1-D intervals. No interval library needed | -| [Rerun `rerun-sdk`](https://pypi.org/project/rerun-sdk/) | 0.38, MIT/Apache | [ARKitScenes example](https://rerun.io/examples/spatial-computing/arkit_scenes) logs poses + intrinsics + RGB + depth + mesh + 3D boxes | -| [drawsvg](https://pypi.org/project/drawsvg/) | 2.4.x, MIT | site plan; PNG export needs Cairo | - -**Consider:** -- **trimesh 5.1 + embreex 4.4** (MIT/BSD): nicer mesh API, e.g. `section()` for the wall-base line. -- **polyscope:** an interactive viewer. - -**Skip:** - -| Project | Why | -|---|---| -| SpatialLM | CC-BY-NC encoder, indoor, needs CUDA | -| Meta SceneScript | CC BY-NC, Aria input; its FAQ says outdoor results are unpredictable | -| RoomFormer | indoor floorplans | -| pymeshlab | GPL-3 | -| pyransac3d | inactive | -| svgwrite | inactive | -| COLMAP / hloc / nerfstudio / gsplat | COLMAP's own tutorial says to avoid texture-less walls; no scale; slow | - -**Gotchas:** -- The ARKit mesh isn't watertight, so `contains_points` is meaningless. -- Open3D rays must be float32. -- ARKit camera coordinates are RUB (x right, y up, looking down −z), unlike OpenCV (UNVERIFIED from docs; the round-trip test settles it). - -## Object detection and recognition - -**Use:** -1. **Gemini API boxes:** - - Format: `box_2d: [ymin, xmin, ymax, xmax]` on 0–1000; segmentation adds a `mask` polygon. - - Set the thinking level to minimal for segmentation. - - Accuracy: RF100-VL zero-shot mAP is 13.3 for Gemini 2.5 Pro vs 1.5 for GPT-5. Gemini 3 Pro with better prompts reached 26.4 on a subset. - - Docs examples use `gemini-3.8-flash` (model ID UNVERIFIED). -2. **[SAM 2](https://github.com/facebookresearch/sam2)** (Apache-2.0), prompted with each box, to get a tight mask + ground-contact point. [Grounded-SAM-2](https://github.com/IDEA-Research/Grounded-SAM-2) already wires this up. -3. **[Qwen3-VL](https://github.com/QwenLM/Qwen3-VL)** (Apache-2.0; 2B–235B) as a local second opinion. Its coordinate scale is 0–1000 or 0–1 depending on which doc you read, so test it. -4. **[supervision](https://supervision.roboflow.com/)** (MIT) for NMS/NMM, `InferenceSlicer` and drawing. -5. **OCR:** Apple Vision `RecognizeTextRequest` + `VNDetectBarcodesRequest` on the phone. On the server, the vision model reads the text, cross-checked with a `CL\s?(200|320)` regex and [zxing-cpp](https://github.com/zxing-cpp/zxing-cpp) (Apache-2.0). -6. **Driveways:** Gemini segmentation + SAM 2 for the hackathon. Mapillary-Vistas Mask2Former has a driveway class, but its weights are non-commercial. - -**Detector baselines:** - -| Model | License | Notes | -|---|---|---| -| Grounding DINO | Apache-2.0 | | -| OWLv2 | Apache-2.0 | | -| Florence-2 | MIT | 0.77B; also does OCR | -| RF-DETR | Apache-2.0 for Nano to Large | fine-tune target for autodistill | - -My judgement on these: fine on window, door, garage door and fence; weaker on gas meter and AC; poor on window well. - -**Depth from a single photo** (the no-AR baseline): - -| Model | License | Notes | -|---|---|---| -| [Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3) `DA3METRIC-LARGE` | Apache-2.0 (Giant/Nested are NC) | Needs focal length in pixels: `depth = focal*out/300`. ETH3D δ1 0.917 | -| [MoGe-2](https://github.com/microsoft/MoGe) | code MIT; weights UNVERIFIED | δ1 0.908, ~60 ms, also gives normals | -| Depth Pro | Apple personal-use grant | commercial use unclear | -| UniDepthV2 | CC BY-NC | skip | -| Metric3D v2 | license conflicts between files | skip | - -**Datasets:** -- Roboflow Universe gas meters ([402 images](https://universe.roboflow.com/proba-vwwtl/gas-meter-recognition), [1.4k images](https://universe.roboflow.com/gas-meter-zbuni/gas-meter-g7kh6)). -- Electric meters ([wattwise](https://universe.roboflow.com/wattwise/electric-meter-wzeeg-zk5fd), [1,943 images for reading dials](https://universe.roboflow.com/abhinav-kumar-do8z1/utility-meter-reading-dataset-for-automatic-reading-yolo-z0e1h)). -- Small AC sets (whether they show outdoor condensers is UNVERIFIED). -- Open Images V7 has Window and Door. -- Facade datasets (CMP, ECP, eTRIMS) are European and a poor fit. -- **No public data for window wells or breaker panels.** Label your own. [autodistill](https://github.com/autodistill/autodistill) (Apache-2.0) can auto-label. - -**Skip:** - -| Option | Why | -|---|---| -| GPT for boxes | 1.5 mAP | -| Claude for boxes | its docs call the coordinates approximate and ask for absolute pixels | -| Grounding DINO 1.5 / DINO-X | API only | -| YOLO-World / YOLOE / Ultralytics | GPL / AGPL | -| Molmo 2 | non-commercial training data | -| SAM 3 | custom license, CUDA 12.6+ | - -**Gotchas:** -- Gemini returns [y, x]. Apply EXIF rotation before inference **and** before casting. -- Depth Anything 3 needs the focal length in pixels. -- Zinsco breakers can read "Sylvania" or "Magnetrip", and FPE ones "Stab-Lok". Put those strings in the prompt. - -## Maps and aerial photos - -**Use:** -- **[Overture Maps](https://docs.overturemaps.org/getting-data/duckdb/)** via DuckDB: `read_parquet('s3://overturemaps-us-west-2/release//theme=buildings/type=building/*')` + a bbox. Roads carry `subclass=driveway|parking_aisle|alley`. Buildings and transportation are **ODbL**. -- **[osmnx](https://osmnx.readthedocs.io/)**: `ox.graph_from_point(pt, dist=150, network_type="drive")`, then `ox.distance.nearest_edges(..., return_dist=True)` on a projected graph. -- **NAIP** (public domain) via the Planetary Computer STAC `naip`. Pflugerville TX: 2022 at 0.6 m. Naperville IL: 2023 at 0.3 m. At 0.6 m a 3–5 ft driveway buffer is only ~2 pixels. -- **[TxGIO StratMap / CAPCOG](https://geographic.texas.gov/stratmap/index.html)** orthoimagery: 6″ inside Austin, 12″ elsewhere, public domain. The best free imagery ML can run on in Texas. -- **[segment-geospatial (samgeo)](https://samgeo.gishub.org/examples/text_swimming_pools/)**: text-prompted SAM ("swimming pool", "driveway"), no training. -- **Census Geocoder** (`onelineaddress`, `Public_AR_Current`): free, no key. Nominatim fallback (1 request/s, identifying User-Agent, no bulk use). -- **Google Solar API: a maybe.** Its §20.1 allows use "to determine the feasibility of installing energy systems". The dataLayers DSM and mask are 0.1 m. Whether running inference on it conflicts with §3.2.3(c) is UNVERIFIED, so ask before relying on it. - -**Licensing traps:** - -| Source | Trap | -|---|---| -| **Google Maps ToS §3.2.3(c)(vii)** | Bans using Google Maps content "to train, test, validate or fine-tune" ML models, and bans tracing building outlines from satellite imagery | -| **Mapbox §1.5(ii)** | Bans using it to "train, operate or improve" ML ("operate" includes running inference) | -| **Esri World Imagery** | Restricted (UNVERIFIED) | -| **ODbL share-alike** | Applies if you publish a database derived from it; internal use with attribution is fine | -| **Microsoft Global ML Footprints** | CDLA-Permissive-2.0 (the older US repo is ODbL) | -| **Mapillary** | CC BY-SA | - -**Recipe:** -1. Geocode the address. -2. Get the Overture building within ±60 m (the one containing the point, or the nearest). Project to UTM 14N (Texas) or 16N (Illinois). -3. Get the osmnx drive network, leaving out `service=driveway` edges. -4. For each outline edge, score (cosine between its outward direction and the vector to the nearest road) × 1/distance. The best is the street-facing wall; flag two edges on corner lots. -5. Driveways = OSM/Overture driveway lines buffered 1.5 m, `amenity=parking` polygons and samgeo masks. -6. Pools = `leisure=swimming_pool` plus samgeo. - -OSM coverage (Overpass, 2026-09-25): - -| Area | Houses | Driveways | Pools | -|---|---|---|---| -| Pflugerville | 14,675 | 149 | 102 | -| Round Rock | 9,223 | 476 | 170 | -| Austin | 302k buildings | 6,029 | 3,339 | - -**AR ↔ map registration:** a 2D rigid fit. Snap the AR walls to 90°, sweep yaw (±20° around the compass reading if its accuracy is ≤ 20°, otherwise all 4 quadrants), and fit translation by least squares against the outline edges. `ARGeoTrackingConfiguration` coverage in Austin is UNVERIFIED. - -**Skip:** -- Google Static / Tiles / Street View and Mapbox/Esri imagery for ML. -- Nearmap: paid, no hackathon tier. -- EagleView: a 30-day sandbox exists, so maybe one email, but don't block on it. -- TorchGeo training: no time. -- Old pool CNNs: zero-shot SAM beats them. - -## Test first (hours 0–4) - -| Test | If it fails | -|---|---| -| LiDAR mesh of a real siding wall at midday sun | Planes only; passage width and headroom become UNSURE | -| Tap → cast into a saved frame → correct pixel | Fix image orientation before building detection | -| Gemini boxes on ~30 real photos (gas meter, AC, window well) | Tap-to-mark carries the demo | -| Compass accuracy at the meter | Use the building outline + road alone | - -## Fine for the hackathon, check before it becomes a product - -- **Not for commercial use:** Mapillary-Vistas models, Molmo 2 and Depth Anything V2's larger models. -- **Grey areas:** Depth Pro (Apple's personal-use license) and inference on Google Solar imagery. -- **ODbL share-alike:** if you publish a derived database. -- **GPL/AGPL detectors:** YOLO-World, YOLOE, Ultralytics. diff --git a/docs/04-prior-art-and-codes.md b/docs/04-prior-art-and-codes.md index 668a4f354..caa1ef4e2 100644 --- a/docs/04-prior-art-and-codes.md +++ b/docs/04-prior-art-and-codes.md @@ -1,80 +1,76 @@ -# Prior art, what Base does publicly, and code citations - -> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. - -Researched 2026-09-25. UNVERIFIED items are marked. - -## What Base does today (public) - -- **Photo checklist, 9 photos** ([help](https://help.basepowercompany.com/en/articles/10280641)): - - Meter close-up (number legible), meter surroundings from 10+ steps back, area to the left, area to the right, the adjacent wall corner to corner, behind the fence. - - Main breaker box, main disconnect (amperage readable), breaker box surroundings. - - The "Base engineering and installation teams" review them and reach out within 2 days if more is needed. Nothing public says whether AI is involved. -- **Placement rules** ([help](https://help.basepowercompany.com/en/articles/10280705)): - - Footprint 3 × 3 ft, ~36″ tall. Within 20 ft of the meter and within 1 ft of the wall. - - 3 ft from gas meters, fences, AC units and other batteries. Not in front of meters, panels, solar equipment or windows. - - Meter ≤ 6 ft up. Meter and breaker box share a wall; the box can't be in a closet. - - 30 × 36″ working space in front of the meter and panel. - - Transfer switch ~13″ wide with 30″ clearance (spec pages: 3 ft of wall for the transfer switch, then 3 × 3 ft per battery). - - Main breaker 100–200 A (150–200 A in Austin). Dual batteries or solar need a 200 A panel. - - No concrete pads needed (integrated base). -- **Hardware:** - - 25/50 kWh ground units, plus **Base Core**: 39.2 kWh, **39.5 × 30.68 × 22 in**, 11 kW inverter, UL-listed ([pv-magazine](https://pv-magazine-usa.com/2026/08/04/base-power-launches-39-2-kwh-u-s-made-base-core-home-battery-secures-1-billion-in-new-funding/)). - - Wall-mounted transfer switch + battery disconnect. No public mention of a meter collar. -- **Who reviews:** a Deployment Site Designer job post (removed Nov 2025) describes people reviewing customer site photos by hand, using CAD. -- **Operations:** heavy lifting is split from electrical work so crews can do "20 homes in a day". +# Public rules, code citations, prior art and licenses -## How others do it +Researched 2026-09-25 from public sources; Esri's imagery terms were rechecked on 2026-09-26. UNVERIFIED marks a claim nobody could confirm. Licenses change, so recheck one before building a product on it. -- **Tesla Powerwall:** guided self-survey on the phone (~30 min), photo upload, remote design. -- **Qmerit (EV chargers):** photo upload + "Panel Insights" AI (with Schneider Electric) reading breaker spaces and capacity from one panel photo. - The closest precedent for using AI only for recognition ([link](https://qmerit.com/news/qmerit-deploying-ai-for-faster-safer-more-accurate-estimates-for-home-ev-charging-installations/)). -- **Aurora Solar:** Site Surveys API, beta ([docs](https://docs.aurorasolar.com/reference/site-surveys)). -- **Scanifly:** drone photogrammetry. -- **SolarAPP+ (NREL):** instant automated code checks for solar and storage permits. The precedent for rules engines over judgment calls. -- **Heat pumps:** - - Aira, Mitsubishi Ecodan and Alpha Innotec have AR placement, but it's visual only with no clearance checks. - - **Fraunhofer ISE "Heat Pump PlanAR"** (reported 2026-09-23) scans indoor boiler rooms and optimises placement. The nearest analogue, but indoors. +## What Base does today -## Products with APIs +- **Photo checklist, 9 photos** ([help page](https://help.basepowercompany.com/en/articles/10280641)): the meter close up with its number legible, the meter's surroundings from 10 or more steps back, the areas to its left and right, the adjacent wall corner to corner, behind the fence, the main breaker box, the main disconnect with its amperage readable, and the breaker box's surroundings. Base's engineering and installation teams review them and reach out within 2 days if they need more. Nothing public says whether AI is involved. +- **Placement rules** ([help page](https://help.basepowercompany.com/en/articles/10280705)): a 3 × 3 ft footprint about 36 in tall, within 20 ft of the meter and 1 ft of the wall, and 3 ft from gas meters, fences, AC units and other batteries. Not in front of meters, panels, solar equipment or windows. The meter is at most 6 ft up, and the meter and breaker box share a wall; the box can't be in a closet. 30 × 36 in of working space in front of the meter and panel. The transfer switch is about 13 in wide with 30 in of clearance, and the spec pages ask for 3 ft of wall for it, then 3 × 3 ft per battery. The main breaker is 100 to 200 A (150 to 200 A in Austin), and two batteries or solar need a 200 A panel. No concrete pad, because the base is integrated. +- **Hardware.** 25 and 50 kWh ground units, and Base Core: 39.2 kWh, 39.5 × 30.68 × 22 in, an 11 kW inverter, UL-listed ([pv magazine](https://pv-magazine-usa.com/2026/08/04/base-power-launches-39-2-kwh-u-s-made-base-core-home-battery-secures-1-billion-in-new-funding/)). A wall-mounted transfer switch and a battery disconnect; no public mention of a meter collar. +- **Review.** A Deployment Site Designer job post, removed in November 2025, describes people reviewing customer site photos by hand in CAD. Heavy lifting is split from electrical work so crews can do "20 homes in a day". -| Product | What | Notes | -|---|---|---| -| **Hover** | measured exterior 3D model (walls, windows, openings) from phone photos | Real JSON API ([docs](https://developers.hover.to/reference/measurements-and-deliverables)); turnaround likely hours; cross-check later, not the demo | -| EagleView | aerial, plus wall/window/door measurements (2026-03) | Enterprise | -| Polycam | 3D capture | API Enterprise-only | -| Matterport | 3D capture | sandbox is demo models only | -| magicplan | RoomPlan-based | indoor | -| Canvas | 3D capture | says it drifts outside and gives no exterior elevations | +## Public rule values -## Prior art +The demo's `server/rules.yaml` (PR #11) uses these. Base's own values are private and never appear in tracked files. -- iPhone 12 Pro Max LiDAR on facades: ~±6–8 cm against a total station. -- iPad Pro LiDAR range: ~5 m. -- AR layout research (OCTO+, CHI 2023): places virtual content only, no code compliance. -- Meter OCR (YOLOv5 study): counters read 97%, serial numbers only 63%. +| Rule | Value | Source | +| --- | --- | --- | +| Cable run from the meter | within 20 ft | Base's help page | +| Distance from the wall | within 1 ft | Base's help page | +| Gas meter or regulator | 3 ft | Base's help page; Austin Energy §1.9; Texas Gas Service | +| AC units, fences, other batteries | 3 ft | Base's help page | +| Doors and windows entering the home | 3 ft | IRC R328.4 | +| Working space at the meter and panel | 30 in wide, 36 in deep, 6.5 ft headroom | NEC 110.26 | +| Headroom over the battery | 6.5 ft | NEC 110.26, applied to the battery as a demo choice | +| Driveway, pool | none public | demo placeholders, 5 ft and 10 ft | -## Why the rules exist (citations for `rules.yaml`) +## Code citations | Code | What it says | -|---|---| -| **NEC 110.26** ([ICC](https://codes.iccsafe.org/s/ISEP2021P1/national-electrical-code-nec-solar-provisions/ISEP2021P1-NEC-Sec110.26)) | Working space 30″ wide (or the equipment width) × 36″ deep × 6.5 ft headroom; no storage in it. Base's page says "30 in high × 36 in wide", probably a garbled version | -| **Austin Energy Design Criteria (Dec 12, 2023)** ([PDF](https://austinenergy.com/-/media/project/websites/austinenergy/contractors/designcriteriamanual.pdf)) | §1.9.2: 30″ wide / 36″ deep / 6′6″ headroom at meters; socket center 30–72″ above ground; ≥ 1 ft from doors and windows. No meter within a 3 ft radius of gas meters, regulators or relief valves. §1.12: generation enclosures keep the 3 ft radius | -| **Texas Gas Service Meter Setting Requirements** ([PDF](https://www.texasgasservice.com/media/tgs/constructionservices/metersettingrequirements_tgs.pdf)) | Electric meters and outlets 3 ft from the regulator relief vent; 3 ft from operable doors and windows | -| **IRC R328 (2021, from NFPA 855)** ([ICC](https://codes.iccsafe.org/s/IRC2021P2/part-iii-building-planning-and-construction/IRC2021P2-Pt03-Ch03-SecR328)) | UL 9540 listed; outdoors or on exterior walls ≥ 3 ft from doors and windows; 3 ft spacing between batteries unless UL 9540A testing allows closer | +| --- | --- | +| **NEC 110.26** ([ICC](https://codes.iccsafe.org/s/ISEP2021P1/national-electrical-code-nec-solar-provisions/ISEP2021P1-NEC-Sec110.26)) | Working space 30 in wide, or the equipment's width, by 36 in deep with 6.5 ft headroom, and no storage in it. Base's page says "30 in high × 36 in wide", probably a garbled version | +| **Austin Energy Design Criteria, Dec 12 2023** ([PDF](https://austinenergy.com/-/media/project/websites/austinenergy/contractors/designcriteriamanual.pdf)) | §1.9.2: 30 in wide, 36 in deep and 6 ft 6 in headroom at meters; socket center 30 to 72 in above ground; at least 1 ft from doors and windows; no meter within a 3 ft radius of gas meters, regulators or relief valves. §1.12: generation enclosures keep the 3 ft radius | +| **Texas Gas Service Meter Setting Requirements** ([PDF](https://www.texasgasservice.com/media/tgs/constructionservices/metersettingrequirements_tgs.pdf)) | Electric meters and outlets 3 ft from the regulator relief vent, and 3 ft from operable doors and windows | +| **IRC R328, 2021, from NFPA 855** ([ICC](https://codes.iccsafe.org/s/IRC2021P2/part-iii-building-planning-and-construction/IRC2021P2-Pt03-Ch03-SecR328)) | UL 9540 listed; outdoors or on exterior walls at least 3 ft from doors and windows; 3 ft between batteries unless UL 9540A testing allows closer | | **NEC 706.15** | Battery disconnect within sight, and within 10 ft or lockable | -| **NEC 230.85** | Outdoor emergency disconnect (moved into 230.70 in the 2026 NEC) | -| ComEd-specific opening rules | UNVERIFIED. Cite Austin Energy / Texas Gas Service instead | +| **NEC 230.85** | Outdoor emergency disconnect; moved into 230.70 in the 2026 NEC | +| ComEd opening rules | UNVERIFIED; cite Austin Energy or Texas Gas Service instead | + +## How others do it + +- **Tesla Powerwall:** a guided self-survey on the phone, about 30 minutes, then photo upload and remote design. +- **Qmerit (EV chargers):** photo upload plus "Panel Insights", built with Schneider Electric, which reads breaker spaces and capacity from one panel photo ([announcement](https://qmerit.com/news/qmerit-deploying-ai-for-faster-safer-more-accurate-estimates-for-home-ev-charging-installations/)). The closest precedent for using AI only to recognize things. +- **SolarAPP+ (NREL):** instant automated code checks for solar and storage permits, the precedent for a rules engine in place of judgment calls. +- **Aurora Solar** has a Site Surveys API in beta ([docs](https://docs.aurorasolar.com/reference/site-surveys)); **Scanifly** uses drone photogrammetry. +- **Heat pumps:** Aira, Mitsubishi Ecodan and Alpha Innotec offer AR placement that is visual only, with no clearance checks. Fraunhofer ISE's Heat Pump PlanAR, reported 2026-09-23, scans indoor boiler rooms and optimizes placement: the nearest analogue, but indoors. +- **Hover** builds a measured exterior 3D model from phone photos and has a JSON API ([docs](https://developers.hover.to/reference/measurements-and-deliverables)), with turnaround likely in hours; useful as a later cross-check. EagleView sells wall, window and door measurements to enterprises. Polycam's API and Matterport's sandbox are closed to us, magicplan is indoor, and Canvas says it drifts outside. + +Measurements from prior work: iPhone 12 Pro Max LiDAR on facades is within about ±6 to 8 cm of a total station, and iPad Pro LiDAR reaches about 5 m. A YOLOv5 meter-reading study read the counters 97% of the time and serial numbers only 63%. + +What nobody else does: the whole loop outdoors, from metric AR capture through a solver that cites a code for each clearance to an AR preview. Heat-pump AR tools are visual only, and Fraunhofer's solver works indoors. + +## Licenses and terms + +**Aerial imagery.** Use StratMap or NAIP. + +| Source | Terms | +| --- | --- | +| Google Maps, Street View | ToS §3.2.3(c)(vii) bans using the content to "train, test, validate or fine-tune" ML, and tracing building outlines from its imagery. The Solar API's §20.1 allows use "to determine the feasibility of installing energy systems"; whether running inference on it conflicts with §3.2.3(c) is UNVERIFIED | +| Mapbox | §1.5(ii) bans using it to "train, operate or improve" ML; operating includes inference | +| Esri World Imagery | Automated extraction only from an exported tile package, inside ArcGIS, with derived results for non-commercial use ([Esri](https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/imagery/learn-to-use-ai-to-extract-information-from-world-imagery)) | +| TxGIO StratMap orthoimagery ([site](https://geographic.texas.gov/stratmap/index.html)) | Public domain; 6 in pixels inside Austin, 12 in elsewhere | +| NAIP, via Planetary Computer | Public domain; 0.6 m in Pflugerville (2022), 0.3 m in Naperville (2023). At 0.6 m a 3 to 5 ft driveway buffer is about 2 pixels | +| Overture buildings and transportation, OSM | ODbL: share-alike applies only if you publish a derived database. OSM tags driveways and pools on under 5% of homes. On 2026-09-25 Pflugerville had 14,675 houses, 149 driveways and 102 pools; Round Rock 9,223 houses, 476 driveways and 170 pools; Austin 302k buildings, 6,029 driveways and 3,339 pools | + +**Models.** Before relying on one, check its weights' license separately from its code's. -## Pitch angle: what's new +| Status | Models | +| --- | --- | +| Noncommercial | MapAnything's default checkpoint (use `facebook/map-anything-apache`); Depth Anything 3 Giant and Nested (Metric Large, Base and Small are Apache-2.0); VGGT-Ω; UniDepthV2; Mapillary-Vistas Mask2Former weights; Molmo 2 | +| Restricted or unclear | Depth Pro (Apple personal-use grant); SAM 3 (custom license, gated weights); HY-World 2.0 (custom community license); Metric3D v2 (files disagree); MoGe-2 weights (UNVERIFIED; code is MIT) | +| Copyleft | YOLO-World, YOLOE and Ultralytics (GPL or AGPL); pymeshlab (GPL-3) | +| Permissive | SAM 2, Grounding DINO, OWLv2, Qwen3-VL, zxing-cpp (Apache-2.0); Florence-2, supervision, Open3D, Stray Scanner (MIT); shapely (BSD-3) | -1. **Nobody does the whole loop outdoors.** Metric AR capture, a code-cited solver and an AR preview. Heat-pump AR tools are visual only, and Fraunhofer's solver is indoors. -2. **Every clearance becomes a traceable check.** "Fails AE §1.9.2: 3 ft from gas" instead of a vision model's judgment. SolarAPP+ for battery siting. -3. **Enforced coverage fixes Base's stated photo failures:** obstructed or too-zoomed views, unreadable meter numbers. -4. **Recognition stays with AI** (Qmerit's model); **measuring stays deterministic.** +For recognition boxes, Gemini 2.5 Pro scored 13.3 zero-shot mAP on RF100-VL against 1.5 for GPT-5. -Be honest about the limits: -- LiDAR reaches ~5 m and is accurate to ~±6–8 cm, so add margins. -- LiDAR is only on Pro iPhones. -- Unit sizes differ between Base models, so keep them as configuration. -- Some calls stay human: how it looks, and judgment calls a camera can't settle. +**Datasets.** The evals use ADVIO (CC BY-NC 4.0), ETH3D (CC BY-NC-SA 4.0) and MARViN (no license stated) only to measure accuracy, and never redistribute them. Get permission before any commercial use. For training data, Roboflow Universe has gas meter sets ([402 images](https://universe.roboflow.com/proba-vwwtl/gas-meter-recognition), [1.4k images](https://universe.roboflow.com/gas-meter-zbuni/gas-meter-g7kh6)) and electric meter sets ([wattwise](https://universe.roboflow.com/wattwise/electric-meter-wzeeg-zk5fd), [1,943 images](https://universe.roboflow.com/abhinav-kumar-do8z1/utility-meter-reading-dataset-for-automatic-reading-yolo-z0e1h)), and Open Images V7 labels windows and doors. No public data exists for window wells, and open photos of US breaker panels are scarce: PR #17 found 7 against the 40 its test needs. diff --git a/docs/05-live-guided-survey-hld.md b/docs/05-live-guided-survey-hld.md index f32f12ab6..b07dc96c8 100644 --- a/docs/05-live-guided-survey-hld.md +++ b/docs/05-live-guided-survey-hld.md @@ -1,10 +1,8 @@ -# Base Power — Live Guided Site Survey - -> Day-1 plan from 2026-09-25, kept for its reasoning and citations. Where it conflicts with [00-overview.md](00-overview.md) or a component README, those win. +# Base Power: Live Guided Site Survey **High-Level Design · v0.1 · September 25, 2026** -**Status:** Public architecture proposal for review; no implementation or existing lane decisions are changed by this document. +**Status:** Public architecture proposal for review; this document changes no implementation or existing decision. **Audience:** Product, engineering, ML, and Base survey reviewers. @@ -16,7 +14,7 @@ This document describes the intended system, not an implemented or benchmarked p Base customers currently supply site-survey photographs used to assess electrical equipment and possible battery installations. The proposed experience keeps the camera open while the app guides the customer, chooses useful photographs, and develops a spatial understanding of the relevant exterior. -The repository baseline is documented in [the overview](00-overview.md), [feature map](01-feature-map.md), and [implementation plan](02-implementation-plan.md). This HLD records the proposed continuous capture and guidance experience, with public technical resources. Business-rule values and their authority are maintained separately through the approved rules configuration. This public document contains no team-only criteria material. +The repository baseline is documented in [the overview](00-overview.md) and the component READMEs. This HLD records the proposed continuous capture and guidance experience, with public technical resources. Business-rule values and their authority are maintained separately through the approved rules configuration. This public document contains no team-only criteria material. | Status | Decision | |---|---| @@ -33,7 +31,7 @@ The repository baseline is documented in [the overview](00-overview.md), [featur ### Relationship to the existing repository plan -The current plan targets an iPhone demo with native Swift, ARKit/RealityKit, and LiDAR geometry where supported. It uses native AR measurements and a deterministic server-side placement solver, retaining tap-to-mark capture as a dependable fallback. Capture is native Swift in `ios/`, and LiDAR is optional. The existing implementation plan remains the baseline until the team adopts a change. +The current plan targets an iPhone demo with native Swift, ARKit/RealityKit, and LiDAR geometry where supported. It uses native AR measurements and a deterministic server-side placement solver, retaining tap-to-mark capture as a dependable fallback. Capture is native Swift in `ios/`, and LiDAR is optional. The overview remains the baseline until the team adopts a change. | Existing baseline | Proposal in this HLD | |---|---| @@ -44,7 +42,7 @@ The current plan targets an iPhone demo with native Swift, ARKit/RealityKit, and | Gemini detection, SAM 2 segmentation, and native Apple Vision checks in the research plan | Treat newer OCR/segmentation models as benchmark candidates, not silent replacements. | | iPhone demo | Keep Android and browser AR references as future portability research, outside the initial demo commitment. | -The evidence record below extends the existing `scene.json` concept rather than replacing its contract. Proposed metadata additions require coordination with the lane owners. Preserve its existing units at interfaces with explicit conversions, and keep AR results relative to the meter anchor using gravity world alignment. +The evidence record below extends the existing `scene.json` concept rather than replacing its contract. Proposed metadata additions require coordination with the owning team. Preserve its existing units at interfaces with explicit conversions, and keep AR results relative to the meter anchor using gravity world alignment. ### Objectives @@ -308,7 +306,7 @@ Perception extracts observable facts. Geometry supplies measured or estimated sp | Human judgment | Preserve a review path for uncertain measurements, interpretation, or unresolved configuration. | | Aggregation | Use the agreed API contract to combine findings; a successful individual check does not establish overall completion. | -The public-source starting points and unresolved values are recorded in [the feature-map rules table](01-feature-map.md#rules-table-c1-confirm-these) and [public source research](04-prior-art-and-codes.md). This HLD introduces no new rule values and does not resolve disputed requirements. Team-only material stays outside tracked files. +The public-source starting points and unresolved values are recorded in [public rule values](04-prior-art-and-codes.md#public-rule-values) and the overview's [open questions](00-overview.md#open-questions). This HLD introduces no new rule values and does not resolve disputed requirements. Team-only material stays outside tracked files. ### Decision and recapture flow @@ -405,17 +403,17 @@ Authenticate survey access and isolate sessions. Store original photos separatel ## 10. Hackathon implementation plan -### Phase 1 — Prove the closed loop +### Phase 1: Prove the closed loop Use one target device, a meter, and one candidate battery area. Demonstrate automatic capture, one real evidence gap, a targeted corrective instruction, a second capture, and an updated explained finding. Include a visible case where additional evidence changes the initial assessment. Use confirmed public-source rules or clearly labeled synthetic test configuration. Unresolved rule values remain configuration gaps; do not present a synthetic demonstration as installation approval. Provide a basic review screen with original images and the reason behind each result. -### Phase 2 — Add progressive spatial understanding +### Phase 2: Add progressive spatial understanding Integrate periodic reconstruction, coordinate alignment, observed-surface coverage, and a virtual battery footprint. Compare ordinary camera-and-motion capture with optional depth input on the same scene. Preserve image evidence when geometry fails. -### Phase 3 — Evaluate streaming and broader coverage +### Phase 3: Evaluate streaming and broader coverage Test LingBot-Map behind the same reconstruction interface. Expand to adjacent walls, interrupted sessions, panel tasks, and representative phones. Adopt streaming only if its measured benefit exceeds integration, compute, and reliability costs. @@ -445,7 +443,7 @@ Measure incorrect approvals and rejections separately, exact meter-number accura ## 12. Decisions needed before implementation hardens 1. **Policy:** Which publicly justified rules configuration is confirmed, and how does the result contract combine findings? -2. **Device scope:** How will native capture integrate with the app shell, and which iPhones represent actual customers? +2. **Device scope:** Which iPhones represent actual customers? 3. **Measurement standard:** What scale anchor and error tolerance are acceptable near each decision boundary? 4. **Deployment:** What GPU, connectivity, latency, and per-survey cost budget are available? 5. **Review handoff:** What evidence package and result format should Base’s existing review process receive? diff --git a/docs/briefings/2026-09-26-morning.html b/docs/briefings/2026-09-26-morning.html deleted file mode 100644 index b9998acbd..000000000 --- a/docs/briefings/2026-09-26-morning.html +++ /dev/null @@ -1,930 +0,0 @@ - - - - - - -House scanning briefing: what we are building, what works, what is blocked - - - - - - - -
      - -

      House scanning: what we are building, what works, and what is blocked

      -

      Team briefing, 26 September 2026, facts as of 09:10. Readers: Sam, Hunter, Shrey, Aiden. About ten minutes.

      -

      Update 11:15: the team split into client and server teams; see docs/how-it-works.html.

      - -

      A homeowner walks the outside wall around their electric meter with an iPhone. The phone measures the wall, a server applies the clearance rules in plain code, and the app shows where the battery goes, in AR, on the real wall.

      -

      Today it runs end to end in the iPhone Simulator on a recorded real walk and gets a real answer from the live server. It has never run on a physical iPhone at a real wall. That is the blocker, and the first job this morning.

      - - - - -

      1. The problem: a battery has to fit between the rules

      - -
      - - An outside wall with the battery and its clearance rules, drawn to scale - Front view of a house wall. From left: an AC unit on the ground, a gas meter, the electric meter, the battery just to its right, a window, a door. Red hatched zones show 3 feet of clearance around the gas meter, AC unit, window and door. A bracket beside the battery shows 6.5 feet of headroom. A blue bracket along the ground shows 20 feet of cable reach either side of the electric meter. A tape along the bottom counts feet from the meter. Below, a view from above shows the battery footprint, 31 by 22 inches, and the 3 feet of clear space needed in front of it. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 20 ft of cable, either side of the meter - - - - AC unit - gas meter - electric meter - window - door - 6.5 ft headroom - battery - - - 3 ft - 3 ft - 3 ft - 3 ft - - - - - - - - - - - 151050 - 510152025 - - feet left and right of the electric meter - - - The same wall from above - - - - - - - - - - - - - - - - - - - - - - - - battery footprint, 31 by 22 in - 3 ft clear in front - AC unit - gas - - -

      Wider than the screen: scroll sideways for the rest of the picture.

      -
      Everything is drawn at one scale: 5 ft between tape marks. The red hatched areas are places the battery may not go. The battery just fits between the electric meter and the window's zone. Rules shown: 3 ft from the gas meter, from AC units, fences and other batteries, and from doors and windows that enter the home; 3 ft of clear space in front; 6.5 ft of headroom; within 20 ft of cable run from the meter; a flat, seen stretch of wall behind it.
      -
      - -
      -

      Base Power puts a home battery on the outside wall, close to the electric meter, because a cable runs from the meter to the battery. The battery is 31 in wide, 22 in deep and 39.5 in tall. Whether it fits, and where, depends on the clearance rules in the picture. Today a person has to go to the house and check.

      -

      Our product does that check with the homeowner's iPhone, in a couple of minutes. The app measures the wall. A server applies the rules in plain code. The app shows where the battery goes, in AR, on the real wall.

      -

      The demo's rules come from public sources and live in server/rules.yaml, each with its citation. Two numbers there are placeholders because no public value exists (pool 10 ft, driveway 5 ft). Base's own thresholds are private, load at runtime from a file that is not in git, and do not appear on this page.

      -
      -
      - The rules the demo uses, with sources -
      - - - - - - - - - - - -
      RuleValueSource
      Distance from the gas meter3 ftBase's public help page; Austin Energy; Texas Gas Service
      Distance from AC units, fences, other batteries3 ftBase's help page
      Distance from doors and windows that enter the home3 ftIRC R328.4
      Clear space in front of the battery3 ftBase's help page
      Cable run from the meterwithin 20 ftBase's help page
      Headroom6.5 ftNEC 110.26, a demo choice
      Wall behind the batteryflat, and seen by the camerademo requirement
      Pool, driveway10 ft, 5 ftplaceholders; no public value exists
      -
      - - -

      2. How a scan becomes a decision

      -

      Six pictures. Each one is the last one with one thing added. The tape along the bottom is real: the app shows the wall unrolled like a tape measure, with the electric meter at zero.

      - -
        -
      1. - - Step 1: tap the electric meter, which becomes zero - The wall from the problem picture, smaller, with the electric meter in the middle. A blue ring marks a tap on the meter. A dashed line drops from the meter to the zero mark on a tape along the bottom. - - - - - tap the electric meter - it becomes 0 ft; all else is left or right of it - -
        Point at the electric meter and tap. The meter becomes zero. Every position from now on is measured left or right of it, in feet.
        -
        2. - -
      2. - - Step 2: the phone photographs the meter close up, then walks left, then right - Same wall. A dashed frame around the electric meter marks the close-up. A phone stands on the ground below the meter. Blue arrows run left along the wall and then right. - - - close-up, reads the number - - - - walk left - then right - -
        The phone takes a close-up of the meter by itself when the picture is sharp, to read the meter's number. Then you walk along the wall, left and then right. The phone keeps photos automatically.
        -
        3. - -
      3. - - Step 3: a haze covers wall and ground the phone has not seen yet - Same wall. The phone has walked left to about 8 feet and right to about 9 feet. Those parts of the wall and ground are clear. The rest is covered by a grey hatched haze. - - - - - - - - haze: - not seen yet - haze - seen: the haze lifts - -
        A haze covers wall and ground the phone has not seen and lifts as it sees them. The team calls this fog of war. The tape strip grows as you walk.
        -
        4. - -
      4. - - Step 4: mark things as you pass, and answer one request for another view - Same wall, almost all seen. Orange pins mark the gas meter, the window and the AC unit. A small hatched patch of ground about 6 feet right of the meter is still unseen. A message at the top reads: Show the ground about 6 ft right of your meter. - - - - - - gas - window - AC - - - Show the ground about 6 ft right of your meter - -
        Mark things as you pass: gas meter, AC unit, door, window, driveway edge, fence. If something a rule needs is still unseen, the app asks for one more view, in one sentence.
        -
        5. - -
      5. - - Step 5: a small measurement file goes to the server, which tries every spot - Same wall, fully seen. A small file icon labelled scene.json, 17 KB, rises from the phone toward a server box at the top. On the wall, dashed battery outlines slide along in small steps. - - - - - - tries every spot, 2 in apart - - - scene.json, 17 KB - photos stay on the phone - - - server: checks every rule - -
        The phone sends a small file of measurements, about 17 KB for a real walk. The server slides the battery's footprint along the wall in 2-inch steps and checks every rule at every position.
        -
        6. - -
      6. - - Step 6: the answer comes back and the battery is shown on the wall in AR - Same wall. The server box at the top sends an arrow back down to the phone. The battery is drawn on the wall just right of the electric meter with an orange outline, and a cable runs from the meter to it. A label reads: best spot, shortest cable. - - - server answers - - - - - - best spot, - shortest cable - shown in 3D and in AR, placed from the meter - -
        The server answers with the best spot (shortest cable run first), the cable route, and every check's outcome and reason. The phone draws the battery on the wall, placed relative to the meter, in 3D and in AR.
        -
        7. -
      - -
      - - The whole loop: phone to server and back - Two boxes. Left: the phone, which taps, walks, marks and shows. Right: the server, with the rules file and plain code. An arrow from phone to server carries scene.json. An arrow from server to phone carries the spot, the cable route and every check's outcome. - - Phone - tap, walk, mark - show the spot in AR - - Server - rules in a file - plain code, no AI - - scene.json - - spot, cable, checks - -
      The recap. AI is used only to recognise things: it reads the meter's text and finds objects. The placement decision is ordinary code with the rules in a file, so every answer traces back to a rule and a measurement.
      -
      - - -

      3. Three ideas that make the answer honest

      - -

      The phone counts its own steps, so error grows with distance

      -
      - - Error bars widen the farther you walk from the meter - A tape along the top counts feet from the meter at 0, 5, 10, 15 and 20. Below it, four positions carry horizontal error bars that get wider with distance: plus or minus 0.3 feet at the meter, 1.1 feet at 5 feet, 1.9 feet at 10 feet and 3.5 feet at 20 feet. Footprints lead away from the meter. - - - - - - - 05101520 ft - - - meter - - - - - - - - - - - - - - - ± 0.3 ft at the meter - ± 1.1 ft at 5 ft - ± 1.9 ft at 10 ft - ± 3.5 ft at 20 ft - - error bar = 0.3 ft + 0.16 ft for every foot from the meter - -
      AR tracking is dead reckoning: the phone estimates how far it moved from the camera and motion sensors, like counting your own steps from a doorway. Small errors add up, so the farther from the meter, the less sure a position is. The server widens every AR measurement's error bar by 0.3 ft plus 0.16 ft per foot along the wall. Where the analogy breaks: a miscounted step is only a stretch along the wall, while the phone's error also drifts sideways.
      -
      - -

      The error bar decides: PASS, FAIL or UNSURE

      -
      - - Three measurements against the 3 foot gas meter rule - A number line from 0 to 6 feet with a vertical rule at 3 feet. Row one: 4.5 feet plus or minus 0.8, the whole bar sits right of 3, PASS. Row two: 2.0 feet plus or minus 0.8, the whole bar sits left of 3, FAIL. Row three: 3.4 feet plus or minus 0.8, the bar straddles 3, UNSURE. - - - - - 0123456 ft - - - 3 ft rule (gas meter) - too close - clear - - - - - - - 4.5 ft ± 0.8: clear by 1.5, more than 0.8 - - ✓ PASS - - - - - - - 2.0 ft ± 0.8: short by 1.0, more than 0.8 - - ✕ FAIL - - - - - - - 3.4 ft ± 0.8: clear by only 0.4 - - ? UNSURE - -
      Every measurement carries an error bar. A check passes only when the margin beats the error, fails only when it is past the rule by more than the error, and is UNSURE otherwise. Equal is UNSURE. UNSURE means a person checks it, or the app asks for a better view. The three examples are the dossier's, for the 3 ft gas meter rule with ± 0.8 ft.
      -
      - -

      Unseen is not clear

      -
      - - A check whose ground was never seen stays UNSURE and names the missing view - A candidate battery spot on the wall. The ground beside it is covered by grey haze. Beside it a check reads: ground near the spot, checks need 6 to 13 ft of seen ground, UNSURE, missing evidence: show the ground about 6 ft right of your meter. - - - - - - meter - - - candidate spot - - never seen - - Ground near the spot - checks need 6 to 13 ft of ground - - ? UNSURE - missing evidence: show the - ground 6 ft right of your meter - -
      If the camera never saw part of the area a rule depends on, that check cannot pass. It stays UNSURE, and the server names the view that would settle it. Here the ground beside the spot was never seen, and the checks that use ground need 6 to 13 ft of it. A blank on the map is never treated as empty ground.
      -
      - - -

      4. Where it stands this morning

      -
      - ✓ proven in the Simulator on a recorded real walk - ! fixed this morning - ? not yet proven on a real phone -
      -
        -
      1. Tap the meterIt becomes zero.
        ✓Simulator?Real phone
        2. -
      2. Read the meter's numberClose-up; three candidates to tap.
        ✓Simulator?Real phone
        3. -
      3. Walk the wallHaze lifts; tape strip grows.
        ✓Simulator!Now reports ground depth, clear space in front, and a tilt-up view for headroom?Real phone
        4. -
      4. Mark things, one more viewGas, AC, door, window, fence.
        ✓Simulator!Server now publishes exactly what settles each check?Real phone
        5. -
      5. Upload scene.jsonMeasurements only.
        !17 KB, not 16.5 MB of photos; answer in 0.5 s?Real phone
        6. -
      6. Server decidesLive on Vercel.
        ✓252 tests; independent checker 50 of 50
        7. -
      7. Show the spot in ARPlaced from the meter.
        ✓Simulator, 3D and AR?Real phone
        8. -
      -
      ✕
      Never run on a physical iPhone at a real wall.Every stage marked "?" above shares this one gap. Everything that decides accuracy is waiting on that data.
      -
      -
      Placeholder rulesWith the public placeholder rules, automatic approval is off, so every real result says "An installer will take a look."
      -
      Hidden wallThings standing in front of the wall can hide it, and nobody owns checking that yet (decision 7).
      -
      Parked: web review page (#15)It shows the server's answer in a browser. It helped check the server before the app could upload. It is not the product.
      -
      - -
      -
      Does what we built work, and is it part of the app we'd ship?
      In the Simulator on a recorded real walk, yes: meter to result, with a real answer from the live server. The app and the server are the product. The web review page is not, and it is parked.
      -
      Are the overnight tests doing anything?
      Each of the five changed the app or the server (section 6 lists what). Three other pieces of overnight work changed nothing: the panel-label search, finer statistics on the modern-phone data, and the research notes.
      -
      What is done that achieves the goal?
      The server, live with 252 tests. The whole app flow in the Simulator. Uploads fixed to 17 KB. The first-try fixes, merging now while the build is repaired.
      -
      What is the main blocker?
      A real-phone session. See the next section.
      -
      - - -

      5. The main blocker, and what to do this morning

      -
      - - The field test: the phone's reading against a tape on the same wall - A wall with the electric meter. A yellow tape runs along the wall with marks at 5, 10, 15 and 20 feet. A phone stands in front. Blue dots show where the phone thinks the same marks are; the gap between dot and mark grows with distance. A label reads: the gap is the phone's scale error. - - - - - - - - - - - 5101520 ft - - - - - - blue dots: where the phone says the tape marks are - the gap is the scale error - - a current iPhone, no LiDAR - -
      Nothing decides accuracy until this runs. Hunter's research handoff (PR #18) reaches the same conclusion.
      -
      -
      -

      Run the field test first, about an hour. You need a current non-LiDAR iPhone, the Measure Lab build, a 30 ft tape and a real meter wall. Follow experiments/evals/field/FIELD_SHEET.md: type the tape readings into the prepared templates, and one command turns them into a score and the phone's real scale error. A dry run recovered a planted 1.5% error exactly, so the scoring end is ready.

      -

      The overnight tracking numbers (finding 1) come from a 2023 iPhone with LiDAR and a dataset whose reference may itself be scaled to the phone's tracking. Most homeowners' phones lack LiDAR. The field test measures our own phone on our own wall, which is the only direct measurement.

      -
      - - -

      6. What the overnight tests found, and what each changed

      -

      All five ran on public real-world datasets with laser-measured ground truth. Dataset images are never in git. Each finding below ends with the line that answers "did this test do anything".

      - -
      -

      Finding 1. How far off is the phone's own tracking?

      -
      -
      - 2018 iPhone 6s (ADVIO) - 2023 iPhone 14 Pro Max (MARViN) - server's allowance -
      - - p90 position error in inches against feet walked, two phones and the server's allowance - At 3, 10, 20 and 30 feet walked: 2018 iPhone 6s 18.6, 55.7, 94.6, 132.6 inches. 2023 iPhone 14 Pro Max 4.0, 8.6, 13.4, 18.5 inches. Server's allowance 5.8, 19.2, 38.4, 57.6 inches. The 2023 phone stays under the allowance at every distance. The 2018 phone is over it at every distance. - - - - 040 in80 in120 in - 3 ft10 ft20 ft30 ftdistance walked - - - - - - - - - 2018 phone, 132.6 in - allowance, 57.6 in - 2023 phone, 18.5 in - - 3 ft walked: 2018 phone 18.6 in, 2023 phone 4.0 in, allowance 5.8 in - 10 ft walked: 2018 phone 55.7 in, 2023 phone 8.6 in, allowance 19.2 in - 20 ft walked: 2018 phone 94.6 in, 2023 phone 13.4 in, allowance 38.4 in - 30 ft walked: 2018 phone 132.6 in, 2023 phone 18.5 in, allowance 57.6 in - - -
      Position error after walking, p90 (9 in 10 cases are better). A current iPhone's own tracking stays inside the server's allowance at every distance; the 2018 phone is far over it. The allowance line is the 0.16 ft per foot part of the server's error bar. Caveats: the MARViN reference may itself be scaled to the phone's tracking, and that phone has LiDAR, which most homeowners' phones lack.
      -
      Data table -
      - - - - - - - -
      Walked2018 iPhone 6s (ADVIO), in2023 iPhone 14 Pro Max (MARViN), inServer's allowance, in
      3 ft18.64.05.8
      10 ft55.78.619.2
      20 ft94.613.438.4
      30 ft132.618.557.6
      -
      -
      -

      What it changed: the server's error allowance comes from this. The field test measures our own phone.

      -
      - -
      -

      Finding 2. Can photos alone give 3D measurements?

      -
      - - Error of learned depth from photos: flat walls and edges - A horizontal scale from 0 to 10 inches. Flat walls: 5.0 inches at p90, with a 95 percent interval from 3.8 to 6.8. Window and door edges: 8 inches or worse in every setting, drawn as a bar starting at 8 and running off the right edge. - - - - 0246810 inerror, p90 - flat wall - - - - 5.0 in (3.8 to 6.8) - edges - - - 8 in or worse, every setting - -
      Learned depth models (MoGe-2, Depth Anything 3), rescaled with the phone's poses. Flat walls come out at about 5 in at best at a realistic phone error. Window and door edges, which are what clearances are measured from, come out at 8 in or worse in every setting.
      -
      -

      What it changed: 3D modeling stays central. Learned depth alone is too rough at edges, so the points that decide a check come from metric depth (LiDAR or AR taps), and LiDAR and the phone's poses give the model its scale. World models are not evaluated yet. (Corrected at midday; the 09:10 version said to drop photo-to-3D measurement.)

      -
      - -
      -

      Finding 3. Does the app's coverage map claim wall it never saw?

      -
      -
      - wall a photo saw - claimed, never seen - good wall thrown away -
      - - A 19.3 foot wall under three versions of the coverage map - Three bars, each 19.3 feet long. First, as claimed: 1.1 feet marked as seen that no photo saw, hidden behind scaffold footings. Second, after requiring views from different angles: the false 1.1 feet is gone but 6 to 8.5 feet of good wall is also thrown away. Third, after a depth test on LiDAR phones: the false part is removed cleanly and the rest is kept. - - as claimed - - - 1.1 ft never seen - views from different angles - - - - 6 to 8.5 ft of good wall lost too - depth test (LiDAR phones) - - - clean - -
      Run on real photos of a laser-scanned building. The map claimed 1.1 ft of a 19.3 ft wall that no photo saw, hidden behind scaffold footings. Requiring views from different angles removed it only by also discarding 6 to 8.5 ft of good wall. A depth test, available on LiDAR phones, removed it cleanly. The hatched span is drawn at 8.5 ft, the larger end of the range.
      -
      -

      What it changed: decision 7 (who checks what is hidden). A wrong claim in the app's code, "the server rechecks from the images", was corrected: the server never sees images.

      -
      - -
      -

      Finding 4. Can the phone read the meter?

      -
      - - Meter reading success on real meter photos - Four horizontal bars on a 0 to 100 percent scale. Read the full number: 71 of 73, 97 percent. Picked the right line: 21 of 75. Barcode held the number: 20 of 24 photos that had one. A tap list of three candidates contained it: 85 percent of held-out photos. - - - - 0255075100% - - full number readright line pickedbarcode had itin the tap list of 3 - - - - - - 71 of 73, 97% - 21 of 75 - 20 of 24 photos that had one - 85% of held-out photos - - -
      Apple's on-device text recognition read the full meter number on 71 of 73 real meter photos, but could not tell which line on the plate was the number (right on 21 of 75). A barcode held the number on 20 of 24 photos that had one. A tap-to-confirm list of three candidates contained it on 85% of held-out photos.
      -
      -

      What it changed: the app shows three candidates to tap, barcode match first, plus two photo checks that trigger a retake (blurry; number too small, "move closer").

      -
      - -
      -

      Finding 5. Can a first-time homeowner finish a scan?

      -
      - - Ground depth the app reported against what the checks need - A scale from 0 to 14 feet. The app reported ground seen out to 4 feet. The checks need 6 to 13 feet. The two do not overlap, so no scan could finish. - - - - 04812 ft - app reported - - 4 ft of ground seen - checks need - - 6 to 13 ft - -
      The verification agent's friction audit: no, not before this morning's fixes. However well it was walked, the app said it had seen ground only 4 ft out while checks need 6 to 13 ft, never reported the space in front of or above the wall, and turned corners into "walk past this" requests.
      -
      -

      What it changed: the capture contract, and the app fixes in section 4: the server publishes exactly what settles each check, and the app reports real ground depth, the walked path as clear space in front, and a tilt-up view for headroom.

      -
      - -

      Three pieces of overnight work changed nothing in the product: the panel-label search (too few open photos: 7 US panels against 40 needed), finer statistics on the modern-phone data, and the research notes.

      - - -

      7. Decisions for this morning, recommendation first

      -
        -

      1. Run the field test first

        Recommend: about an hour, before anything else.

        A current non-LiDAR iPhone, Measure Lab, a 30 ft tape, a real meter wall. Follow experiments/evals/field/FIELD_SHEET.md. One command turns tape readings into a score and the phone's real scale error.

        2. -

      2. How points get measured

        Recommend: metric depth where precision decides a check.

        Take those points from LiDAR or AR taps; learned depth alone gets edges to 8 in or worse (Finding 2). The 3D model still carries everything else. Updated 11:40 after the team split.

        3. -

      3. How far to walk

        Recommend: outward from the meter on both sides; stop when the nearest spot that fits is fully seen.

        The server prefers the shortest cable, so walking farther cannot find a better spot, and shorter walks drift less.

        4. -

      4. Base's real rules

        Recommend: load them into the private rules file now.

        Confirm with Base which rules the demo follows and which number on the meter plate they need (plates carry several). Otherwise the demo can never show a real "fits here".

        5. -

      5. Merge order

        Recommend: #11 server first, since the others depend on its schemas.

        Then #14 docs, #10 app, #13 verification, #12 evals, then #7, #4, #8, #6, #16, #17, #19.

        6. -

      6. Lane owners

        Recommend: name one owner per lane.

        Capture app, geometry, rules, output. Agents then know whose OK they need.

        7. -

      7. Who checks what is hidden behind things?

        Recommend: the app.

        Before the result, show the homeowner the photo of the chosen spot and ask if anything blocks the wall or ground there. On LiDAR phones, also run a depth test (finding 3).

        8. -

      8. What must the scan see, and what can we just ask?

        Recommend: pool and driveway become one yes/no question, labelled as the homeowner's answer.

        Instead of requiring 8 to 13 ft of photographed ground.

        9. -
      - - -

      8. Asks: only a person can do these

      -
        -
      • Sam: review bots. Only CodeRabbit is installed and it is rate-limited; Greptile, Macroscope, Aikido and Codex review are not installed.
        • -
      • Sam: a one-time App Store Connect key for TestFlight (PR #8).
        • -
      • Sam: web previews on Vercel were turned off because the free plan's 100 deploys a day ran out; reconnect later.
        • -
      • Anyone, 5 minutes: confirm the meter labels on the local review page on Sam's laptop.
        • -
      - - -

      9. Reference: what exists, and how to try it

      -

      Open pull requests. None of these is merged yet. The server (#11) is live.

      -
        -
      • #4 scoring harness
        • -
      • #6 research notes
        • -
      • #7 Measure Lab
        • -
      • #8 TestFlight upload
        • -
      • #9 Hunter's camera-metadata probe
        • -
      • #10 iOS app
        • -
      • #11 placement server (live)
        • -
      • #12 evals
        • -
      • #13 independent verification
        • -
      • #14 docs cleanup
        • -
      • #15 web review page (parked)
        • -
      • #16 meter reading
        • -
      • #17 panel labels (blocked on photos)
        • -
      • #18 Hunter's research handoff
        • -
      • #19 this briefing
        • -
      - -

      How to try it

      -
      -

      Server health: house-scanning-server.vercel.app/health

      -

      App in the Simulator: build ios/HouseScan.xcodeproj and run with

      -
      -replay <folder> -autopilot -serverURL https://house-scanning-server.vercel.app
      -

      On a real iPhone: set your team in ios/Config/Local.xcconfig (see ios/README.md), plug in, run.

      -
      - -
      Facts from the team dossier as of 09:10, 26 September 2026. Base's private thresholds do not appear on this page. The drawings are illustrations at a consistent scale, not a real house.
      - -
      - - diff --git a/docs/eli5.html b/docs/eli5.html deleted file mode 100644 index abade8549..000000000 --- a/docs/eli5.html +++ /dev/null @@ -1,615 +0,0 @@ - - - - - -Where does the battery go? Explained like you're 5 (day-1 version) - - - - -
      -
      -
      Day-1 explainer, written 2026-09-25 and kept as a record. The current one is how-it-works.html. Out of date here: - the team has since split into a client team (the iPhone app and its capture packet) and a server team (a 3D model built from the packet, then the placement checks). - Step 4's demo scores every spot as plain pass or fail; the real checks also answer UNSURE when a measurement's error bar straddles the limit or its area was never seen. - The tool choices in the grown-up corner are the day-1 plan; the server team is now comparing 3D reconstruction methods, including ones that corner rules out.
      -

      Where does the battery go? 🔋🏠

      -

      Some families get a big battery that sits outside, next to their house. It keeps the lights on when the power goes out. - But it can't go just anywhere. Here's how our app finds the perfect spot.

      - -
      -
      - - -
      -
      -
      - THE PROBLEM -

      Photos can't measure 📸🤷

      -

      Right now, families send in photos. But a photo is flat! It can't tell whether the gap between two windows is as big as a bathtub - or as small as a lunchbox. And if someone forgets to photograph part of the wall, nobody knows what's there.

      -

      For grown-ups: a single photo has no real scale, so distances get estimated by eye. Borderline cases end up with a human reviewer, - and any wall nobody photographed turns into a "please send more photos" follow-up.

      -
      -
      - - - - - - - - - - - - - ?? ft - "is this 3 feet or 8 feet?" 🤔 - -
      -
      -
      - - -
      -
      - STEP 1 -

      The magic measuring phone 📱✨

      -

      The family holds up their phone and walks along the wall, like playing "follow the wall." The phone counts how far it went, like - counting footsteps. Fancy phones even shoot out invisible laser dots to feel how far things are, the way a bat uses sound! The phone - says "keep going!" until it has seen the wall all the way to the end, both ways.

      -
      - - - - - - - - - - - - - - - end ✓ - - end ✓ - - - - - - - 👟 - - -
      -
      Wall seen
      - -
      -

      For grown-ups: ARKit tracks the phone in real feet using the camera plus the motion sensors. LiDAR iPhones add a 3D mesh - (about ±6–8 cm on walls, out to about 5 m). The capture won't finish until both wall ends are tapped. Anything the phone still hasn't seen counts as unsure, never as clear.

      -
      -
      -
      - - -
      -
      - STEP 2 -

      A robot plays I Spy 🤖🔍

      -

      The pictures go to a helper computer. A robot looks at them and plays I Spy: "I spy a window! I spy a door! I spy the gas meter!" - Then it sticks a label on each thing, in exactly the right spot along the wall.

      -
      - - - - - - - - - - - window! - gas meter! - the meter! - door! - AC! - 🔍 - -

      For grown-ups: Gemini draws the boxes. It had the best zero-shot box accuracy in the research, and it returns boxes natively. SAM 2 tightens each box, and supervision merges duplicates across frames. Because the phone knows where it was for every photo, each box - gets cast onto the wall and becomes "window, 2–5 ft along the wall, 3–6.5 ft up."

      -
      -
      -
      - - -
      -
      -
      - STEP 3 -

      Looking down like a bird 🐦🗺️

      -

      Some things are easier to see from the sky: the driveway where cars go, the swimming pool, and which side of the house faces the street. - So the helper also checks a map, like a bird looking down.

      -

      For grown-ups: Census geocoder → Overture building outline → osmnx nearest road → street-facing wall. - Pools and driveways come from samgeo on public-domain StratMap/NAIP aerial photos. OSM tags them on under 5% of homes. Don't run ML on Google or Mapbox imagery; - their terms forbid it.

      -
      -
      - - - - - - - - - - pool - - - - house - garage - - - - - - - - - - keep far fromthe pool - cars drivehere! stay away - faces the street 👀 - the meter → - - 🐦 - -
      -
      -
      - - -
      -
      - STEP 4 · YOUR TURN -

      Slide the box and check the rulebook 🧩

      -

      Drag the battery along the wall. Every time it moves, the helper checks the rulebook. Green means that rule is happy. - Red means "not here!" Then press Let the computer try every spot to watch it test every single position, one tiny step at a time.

      -
      -
      - - - - - - - - - - - - - driveway - - - - - - - - - - - - - - - - - - - - - - windowgasmeter - doorAC - - - - - - - - BASE - - - - - - - -
      - - - -
      -
      - space workers need at the meter - the wire from the meter - a good spot - not allowed - the stripe under the wall = the computer's test results -
      -
      -
      - -
      -
        -

        For grown-ups: this is the whole solver. It slides the 31″ battery along the unrolled wall in small steps and runs every - rule as a plain check (shapely + numpy, numbers in rules.yaml). There's no AI in this step, so every "no" comes with a reason - you can cite. This demo predates error bars: the real solver also answers UNSURE when a measurement is too close to call or its area was never seen.

        -
        -
        -
        -
        - - -
        -
        -
        - STEP 5 -

        Show and tell 👀✨

        -

        The phone draws a pretend battery right on the real wall, like a sticker floating in the air, plus a purple line showing where the wire goes. - The family looks through their phone and says, "Oh! That's where it goes!"

        -

        For grown-ups: RealityKit places a box the battery's size (Base Core is 39.5″ × 30.68″ × 22″), attached to the meter's anchor rather than - raw world coordinates, because ARKit corrects anchors as the phone moves. It's also the moment the family gets to say whether they like how it looks.

        -
        -
        - - - - - - - - - - - - - BASE - - - Here it goes! ✨ - - Looks good 👍 - - - -
        -
        -
        - - -
        -
        - WHY IT MATTERS -

        Why this is better 🏆

        -

        Instead of guessing from flat photos, the phone measures, makes sure it saw everything, and a fair rulebook picks the spot.

        -
        -
        📏

        Real feet, not guesses

        -

        A flat photo can't tell 3 feet from 8 feet. The phone measures to within a few inches.

        -
        🔁

        No "take more photos" loop

        -

        The phone won't finish until it has seen every part the rules need, so families don't get asked for more pictures later.

        -
        📖

        Every answer has a reason

        -

        "Too close to the gas meter" comes from a real safety rule, not a robot's feeling. That makes it easy to trust and to double-check.

        -
        -
        -
        - - -
        -
        -
        - 🧑‍🔧 Grown-up corner: what each cartoon really is - - - - - - - - - - - - - - - - - - -
        CartoonReal toolsWhy this one
        📱 Magic measuring phoneSwift + ARKit (gravity alignment, wall and ground detection, LiDAR mesh), frame recorder forked from Stray Scanner (MIT), keyframe picker from arkit-3dgs-scannerMeasures in real feet on the phone. Photogrammetry fails on blank siding, and COLMAP's own docs say so.
        🔍 Robot I SpyGemini boxes → SAM 2 → supervision; each box cast onto the wall using where the phone wasBest zero-shot box accuracy (13.3 vs 1.5 mAP for GPT), with native box output.
        🔢 Reading the meterApple Vision text + barcode reading on the phone; zxing-cpp + Gemini on the serverInstant "retake" on the phone. Serial-number OCR is the weak spot (63% read vs 97% for the dial counters).
        🐦 Bird's-eye mapCensus geocoder, Overture building outlines via DuckDB, osmnx, samgeo on StratMap/NAIPFree, public-domain imagery that ML is allowed to run on. Google's and Mapbox's terms ban it.
        🧩 Slide the boxshapely + numpy sweep, Open3D for passage-width and headroom rays, rules.yamlUnder a second to run, easy to explain, and every rule traces to NEC 110.26, IRC R328, Austin Energy or Texas Gas Service.
        ✨ Show and tellRealityKit box attached to the meter's anchor; drawsvg 2D site plan for reviewersThe demo moment, and where the family sees and approves the spot.
        🐞 DebuggingRerun (its ARKitScenes example logs exactly our data)Camera positions, frames, mesh, detections and solver rays in one timeline.
        📊 The pitch baselineDepth Anything 3 metric (Apache-2.0) on plain photosProves AR beats photos, using a tape measure as the answer key.
        -

        The example wall is made up for illustration: window 2–5 ft, gas meter 8–9 ft, meter at 12 ft, door 24–27 ft, AC 30–33 ft, driveway from 37 ft. - The 3 ft gas, AC, door and window clearances and the 20 ft wire limit come from Base's public placement rules and building codes. The driveway distance is made up for the demo.

        -
        -
        -
        - -
        Made for the Base Power hackathon team · Drag the battery in step 4 · Works offline, one file
        - - - - diff --git a/docs/how-it-works.html b/docs/how-it-works.html index c0d432b8d..eaaf253a2 100644 --- a/docs/how-it-works.html +++ b/docs/how-it-works.html @@ -403,7 +403,7 @@

        2. From scan to answer, one step at a time

        The guided scan box from stage 1, with an arrow to a file icon labelled packet, sitting on the line between the teams. -
        It sends a capture packet. Photos; where the camera was and which way it pointed for each one (poses); the lens's focal length and center (intrinsics); motion-sensor readings; LiDAR depth and mesh when the phone has LiDAR; and the homeowner's marks, such as the meter. The packet is the contract between the teams.
        +
        It sends a capture packet. Today the app uploads only its measurements and exports the full packet on request; the packet's README lists what it records so far. The packet holds photos; where the camera was and which way it pointed for each one (poses); the lens's focal length and center (intrinsics); motion-sensor readings; LiDAR depth and mesh when the phone has LiDAR; and the homeowner's marks, such as the meter. The packet is the contract between the teams.
      • @@ -552,7 +552,7 @@

        Coverage means 100% of what the model needs

        needed: all of it ground out to 13 ft -
        Drawn to scale, 1 ft to 8 px. The zone is the wall within 20 ft of cable either side of the meter, the ground in front out to 6 to 13 ft depending on the check, and the space overhead for headroom, which a plan view can't show. The coverage map has to be honest. The app's first map checked only range, angle and framing, and claimed 1.1 ft of wall that no photo saw. Tested against depth, the recon worker claims at most about 0.46 ft.
        +
        Drawn to scale, 1 ft to 8 px. The zone is the wall within 20 ft of cable either side of the meter, the ground in front out to 6 to 13 ft depending on the check, and the space overhead for headroom, which a plan view can't show. Near each end it reaches past 20 ft by the clearance around a spot at the edge of reach. A pass needs only the chosen spot and its clearances seen, and the server's answer says when the walk has gone far enough. The coverage map has to be honest. The app's first map checked only range, angle and framing, and claimed 1.1 ft of wall that no photo saw. Tested against depth, the recon worker claims at most about 0.46 ft.

        LiDAR first, never required

        @@ -699,8 +699,8 @@

        5. Where each piece lives

        - - + + @@ -708,7 +708,7 @@

        5. Where each piece lives

        PieceSource of truthState
        iPhone app
        client team        		ios/README.mdGuided capture in PR #10; live 3D map on branch t3/ios-map3d, stacked on it
        Capture packet
        client team        		packet/README.mdBeing specified on branch t3/packet
        iPhone app
        client team        		ios/README.mdGuided capture in PR #10; live 3D map in PR #21, stacked on it
        Capture packet
        client team        		packet/README.mdSpec, validator and samples in PR #22
        Rules engine, API, capture contract
        server team        		server/README.md, "What settles each check"PR #11; live as a public demo with public rules, and privately with Base's rules behind a key
        Reconstruction worker
        server team        		recon/HANDOFF.mdDraft PR #20, handed to the server team
        Accuracy evalsexperiments/evals/README.mdPR #12
        Web review pageParked, PR #15
        -

        The day-1 plan (docs/01 to 05) is kept for its reasoning and rule citations, and the day-1 explainer, with its drag-the-battery demo, is eli5.html. The dated briefings in docs/briefings/ hold the evidence behind several figures here.

        +

        The plan, the decisions and the full evidence list are in 00-overview.md. The public rule values, their code citations and the model and imagery licenses are in 04-prior-art-and-codes.md.

        Base's private thresholds, prompt names and field names do not appear on this page. The drawings are illustrations at a consistent scale, not a real house. Evidence numbers come from public datasets with laser-scanned ground truth; no dataset images are included.
        From 3c0f674a1006e20cee592b2f72210d67b680bf3e Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 13:39:57 -0500 Subject: [PATCH 08/16] docs: add the server team's plan and tighten sentences for agents - docs/00 now explains how the pieces fit on both sides, including the server plan from Hunter's research handoff (PR #18): originals go to object storage, analysis runs asynchronously, and revised geometry invalidates dependent coverage. - AGENTS.md tells agents to report to the manager session that holds the plan's history and let it settle plan and product questions. - Semicolons become sentences and long bullets are split across AGENTS.md, README.md, docs/00 and docs/04. Two overstated claims in docs/00 are corrected to match the code: mesh vertices are copied before transforming, and each mesh measurement is the smallest distance over a fan of rays. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 22 ++++++----- README.md | 6 +-- docs/00-overview.md | 71 +++++++++++++++++++-------------- docs/04-prior-art-and-codes.md | 72 +++++++++++++++++----------------- 4 files changed, 94 insertions(+), 77 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 97469e193..a4569113b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,13 +1,15 @@ # Agent guide -A homeowner scans the wall around their electric meter with an iPhone; the server decides whether a Base Power battery fits and where, and the app shows the spot in AR. [README.md](README.md) maps the repository; [docs/00-overview.md](docs/00-overview.md) holds the plan, decisions and evidence. +A homeowner scans the wall around their electric meter with an iPhone. The server decides whether a Base Power battery fits and where, and the app shows the spot in AR. [README.md](README.md) maps the repository, and [docs/00-overview.md](docs/00-overview.md) holds the plan, the decisions and the evidence. ## Who owns what - **Client team.** Sam, working with AI agents, owns the iOS app in `ios/` and the capture packet it sends. Aiden films video and gathers sample datasets. - **Server team.** Hunter, working with his own agents, owns everything after the packet: the 3D model and the rule checks, in `server/` and `recon/`. -Edit the other team's files only with that team's OK, and talk to the author before editing files that an open pull request also changes. Two writers on one file cost more time than asking does. +Most agent work arrives from a manager session that holds the plan's history and the discussions behind it. Report results, blockers and open decisions back to that manager, and let it settle questions about the plan or the product. + +Edit the other team's files only with that team's OK. Talk to the author before editing files that an open pull request also changes, because two writers on one file cost more time than asking does. ## Where to look @@ -27,19 +29,19 @@ A component's README wins over the docs. Files marked with a pull request exist ## Rules -- **Keep Base's materials in `private/`, which git ignores.** The repository is public. Don't commit, quote or summarize those materials in tracked files, including prompt text, prompt names, output field names and internal thresholds. Team notes go in `private/internal-notes.md`. Real captures, photos of homes and dataset images go in `captures/`, `fixtures/real/` or `data/`, which git also ignores. -- **Make placement decisions in deterministic code.** Models build geometry and recognize things; plain code passes or fails each check, so every answer traces to a rule and a measurement. Clearance numbers live in rules files with their sources, never in code. +- **Keep Base's materials in `private/`, which git ignores.** The repository is public, so tracked files carry none of that material: no quotes, summaries, prompt text, prompt names, output field names or internal thresholds. Team notes go in `private/internal-notes.md`. Real captures, photos of homes and dataset images go in `captures/`, `fixtures/real/` or `data/`, which git also ignores. +- **Make placement decisions in deterministic code.** Models build geometry and recognize things, and plain code passes or fails each check, so every answer traces to a rule and a measurement. Clearance numbers live in rules files with their sources, never in code. - **Treat unseen as unsure, never as clear.** A gap in the scan could hide a gas meter. When a check depends on an area nobody observed, return UNSURE and name the view that would settle it. - **Use StratMap or NAIP for aerial imagery.** Google's and Mapbox's terms forbid running ML on their imagery, and Esri allows it only inside ArcGIS for non-commercial use. - **Place AR results relative to the meter's anchor, with `.gravity` world alignment.** ARKit corrects anchors as tracking improves, so a result tied to the meter moves with it while raw world coordinates drift. `.gravityAndHeading` depends on the compass, which is unreliable next to a house. -- **Rotate the intrinsics whenever you rotate a camera image.** Camera images are landscape sensor images and the intrinsics match that orientation. A rotated image with unrotated intrinsics produces wrong 3D geometry without any error. -- **Use LiDAR when present, never require it.** Most homeowners' phones lack LiDAR, so every check must also work from the camera and the phone's poses. +- **Rotate the intrinsics whenever you rotate a camera image.** Camera images are landscape sensor images, and the intrinsics match that orientation. A rotated image with unrotated intrinsics produces wrong 3D geometry without any error. +- **Use LiDAR when present, and never require it.** Most homeowners' phones lack LiDAR, so every check must also work from the camera and the phone's poses. ## Working in the repository -- `make check` runs every suite; `make ios`, `make server` and `make web` run one each. -- Branch from `main`, keep one writer per branch, and open a pull request. People merge. Agents never push to `main` or merge, so a person sees every change before it lands. +- `make check` runs every suite. `make ios`, `make server` and `make web` run one each. +- Branch from `main`, keep one writer per branch, and open a pull request. People merge, and agents never push to `main` or merge, so a person sees every change before it lands. - Several agents share one Mac. Exit code 137 means the system killed the process, usually for memory. Find the large allocation before rerunning, because one runaway process can freeze the whole machine. - Put `DEVELOPMENT_TEAM` in `ios/Config/Local.xcconfig` (copy `Local.xcconfig.example`), not in Xcode's Signing & Capabilities pane. The pane writes into `project.pbxproj`, and CI fails on that drift. -- After editing `ios/project.yml`, run `make ios-project` (it needs XcodeGen 2.46.0) and commit the regenerated project, because CI regenerates it and fails on any difference. -- `sites/landing` is a submodule; change the landing page in its own repository. +- After editing `ios/project.yml`, run `make ios-project` (it needs XcodeGen 2.46.0) and commit the regenerated project. CI regenerates it and fails on any difference. +- `sites/landing` is a submodule. Change the landing page in its own repository. diff --git a/README.md b/README.md index e52ed7dda..d7938a87c 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ A homeowner walks the outside wall around their electric meter with an iPhone. T This is a four-person hackathon project for Base Power, started 2026-09-25. Success means one capture session gives Base enough to decide, with no follow-up photos. -Two teams share the work. The client team (Sam with AI agents; Aiden on video and sample data) builds the iPhone app and the capture packet it sends. The server team (Hunter with his agents) turns the packet into a 3D model and checks the rules on it. +Two teams share the work. The client team, Sam with AI agents and Aiden on video and sample data, builds the iPhone app and the capture packet it sends. The server team, Hunter with his agents, turns the packet into a 3D model and checks the rules on it. ## Where to start @@ -21,7 +21,7 @@ Paths marked with a pull request exist only on that branch until it merges. | Path | What it is | State | | --- | --- | --- | -| `ios/` | The iPhone app (Swift, ARKit, RealityKit) | On `main`, an AR session that shows tracking. Guided capture is in PR #10; the live 3D map is in PR #21 | +| `ios/` | The iPhone app (Swift, ARKit, RealityKit) | On `main`, an AR session that shows tracking. Guided capture is in PR #10, and the live 3D map in PR #21 | | `packet/` | The capture packet's spec, validator and samples | PR #22 | | `server/` | The rules engine and placement API (Python, uv) | On `main`, a skeleton. The engine is in PR #11 | | `recon/` | Turns photos and depth into a 3D model and a coverage map | PR #20, handed to the server team | @@ -33,7 +33,7 @@ Paths marked with a pull request exist only on that branch until it merges. ## What's live - The demo server at https://house-scanning-server.vercel.app runs the engine from PR #11 with public rules only, and every answer says so. `GET /health` shows which rules are loaded. A second deployment loads Base's rules and requires a key. -- TestFlight builds of the app and Measure Lab start by hand from the Actions tab; see [CONTRIBUTING.md](CONTRIBUTING.md). +- TestFlight builds of the app and Measure Lab start by hand from the Actions tab, as [CONTRIBUTING.md](CONTRIBUTING.md) describes. - `make check` runs the server, web and iOS suites that CI runs. This repository is public. Materials Base gave the team stay in the git-ignored `private/` folder, and photos of real homes never enter git. diff --git a/docs/00-overview.md b/docs/00-overview.md index b47e0bbbc..023d440ad 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -1,52 +1,65 @@ # House scanning: plan, decisions and evidence -A homeowner scans the outside wall around their electric meter with an iPhone. The server works out whether a Base Power battery fits there and where, and the app shows the spot in AR. Success means one capture session gives Base enough to decide, with no follow-up photos. A person can still review any result. +A homeowner scans the outside wall around their electric meter with an iPhone. The server works out whether a Base Power battery fits there and where, and the app shows the spot in AR. The goal is one capture session that gives Base enough to decide, with no follow-up photos. A person can still review any result. -Base's survey starts today from customer photos, which have no scale, can't show whether two things share a wall, and miss whatever sits just outside the frame. An AR session gives every photo a position and a real scale, and the app knows what it hasn't seen while the homeowner is still there. +Base's survey starts today from customer photos. A photo has no scale, can't show whether two things share a wall, and misses whatever sits just outside the frame. Borderline cases go to a reviewer, and homeowners get asked for more photos. An AR session gives every photo a position and a real scale, and the app knows what it hasn't seen while the homeowner is still standing there. -The work is three problems: guide the homeowner until the phone has captured everything the rules need, turn the photos and their metadata into a 3D model, and check that model against the placement rules. Full-house scans, roofs and Android are out of scope. Each component's README is the source of truth for that component and wins where it differs from this file. +Full-house scans, roofs and Android are out of scope. Each component's README is the source of truth for that component and wins where it differs from this file. -## Two teams +## How the pieces fit -- **Client team.** Sam with AI agents; Aiden films video and gathers sample datasets. The app in `ios/` builds a live 3D map on the phone, with a haze ("fog of war") over what it hasn't seen, and guides the homeowner until everything the rules need is covered. Guided capture is in PR #10 and the live map in PR #21. The app sends a capture packet: photos, camera poses, intrinsics, motion-sensor readings, LiDAR depth and mesh when the phone has LiDAR, and the homeowner's marks. `packet/README.md` (PR #22) specifies it and lists what the app records today. For now the app uploads only its measurements, `scene.json`, and exports the full packet on request. -- **Server team.** Hunter with his agents. The packet goes in, a 3D model comes out, and the rules are checked on it. Starting points: the rules engine and API in `server/` (PR #11), the reconstruction worker in `recon/` (PR #20), the evals in `experiments/evals/` (PR #12), Hunter's research handoff (PR #18) and his [live-survey design](05-live-guided-survey-hld.md). +The work is three problems. The phone guides the homeowner until it has captured everything the rules need. The photos and their metadata become a 3D model. Plain code checks that model against the placement rules, and the answer comes back to the phone. + +**Client team: Sam with AI agents, and Aiden on video and sample datasets.** The app in `ios/` builds a live 3D map on the phone. A haze, the "fog of war", covers what it hasn't seen and lifts as the homeowner scans. The app guides them until everything the rules need is covered, which is a zone around the meter rather than the whole house. Guided capture is in PR #10 and the live map in PR #21. The app's output is the capture packet: photos, camera poses, intrinsics, motion-sensor readings, LiDAR depth and mesh when the phone has LiDAR, and the homeowner's marks. `packet/README.md` (PR #22) specifies it and lists what the app records today. For now the app uploads only its measurements, `scene.json`, and exports the full packet on request. + +**Server team: Hunter with his agents.** The packet goes in, and a 3D model comes out at real scale with a record of which parts were observed. The rules engine in `server/` (PR #11) then checks every candidate spot. The team is comparing reconstruction methods broadly first, then the best in depth. Hunter's research handoff (PR #18) sets the shape of the server side: +- Original photos travel to object storage, separate from the compact placement request, because a Vercel function accepts at most 4.5 MB. +- Recognition and reconstruction run asynchronously while the phone keeps guiding, and capture, upload, analysis and assessment finish at different times. +- A new reconstruction can revise earlier geometry. Coverage and measurements that depend on it get recomputed, never carried over. +- OCR confidence doesn't settle an exact identifier, so unreadable or conflicting values stay unresolved. + +The reconstruction worker is in `recon/` (PR #20), the accuracy evals in `experiments/evals/` (PR #12), and Hunter's live-survey design in [05-live-guided-survey-hld.md](05-live-guided-survey-hld.md). ## Decisions -- **Guided AR capture, not a video processed later.** Photogrammetry alone (COLMAP, splats) has no scale, is slow, and struggles on blank siding; COLMAP's own tutorial says to avoid texture-less walls. The packet carries ARKit's poses and any LiDAR depth, so the server's model starts at real scale. -- **3D modeling stays central,** with LiDAR and the phone's poses supplying scale, because learned depth alone is too rough at edges. -- **Plain code decides placement; models build geometry and recognize things.** The engine unrolls the walls into one line, `s`, in feet from the meter (negative to the left), slides the battery's footprint along it in 2 in steps, and checks each rule as an interval overlap or a polygon distance. -- **Every check answers PASS, FAIL or UNSURE.** A check passes only when its margin beats the measurement's error and fails only when it misses by more than the error. Anything else is UNSURE, including a check on an area nobody observed, and the answer names the view that would settle it. -- **Rule values live in rules files, each with its source.** `server/rules.yaml` (PR #11) holds the public values and two labeled placeholders (pool 10 ft, driveway 5 ft); Base's values load from a git-ignored private file. Battery sizes live there too, because models differ: Base Core is 39.5 in tall on a 30.68 × 22 in footprint, and older units are 3 × 3 ft. -- **Native iOS only.** Expo has no first-class ARKit support, and capture needs mesh export, per-frame pose and lens data, and mesh raycasts. -- **AR results hang off the meter's anchor, with `.gravity` alignment.** ARKit corrects anchors as tracking improves, and drift runs about 2 cm a second. `.gravityAndHeading` depends on the compass, which developers report off by up to about 176° near a house. +- **Guided AR capture, not a video processed later.** Photogrammetry alone has no scale, is slow, and struggles on blank siding and stucco. COLMAP's own tutorial says to avoid texture-less walls. The packet carries ARKit's poses and any LiDAR depth, so the server's model starts at real scale. +- **3D modeling stays central.** Learned depth alone is too rough at edges, so LiDAR and the phone's poses supply the scale. +- **Models build geometry and recognize things. Plain code decides placement.** The engine unrolls the walls into one line, `s`, in feet from the meter, negative to the left. It slides the battery's footprint along that line in 2 in steps and checks each rule at each position. Every answer then traces to a rule and a measurement. +- **Every check answers PASS, FAIL or UNSURE.** A check passes only when its margin beats the measurement's error, and fails only when it misses by more than the error. Everything else is UNSURE. That includes any check on an area nobody observed, and the answer names the view that would settle it. +- **Rule values live in rules files, each with its source.** `server/rules.yaml` (PR #11) holds the public values and two labeled placeholders: pool 10 ft and driveway 5 ft. Base's values load from a git-ignored private file. Battery sizes live there too, because models differ. Base Core is 39.5 in tall on a 30.68 × 22 in footprint, and older units are 3 × 3 ft. +- **Native iOS only.** Capture needs mesh export, per-frame pose and lens data, and mesh raycasts, and Expo has no first-class ARKit support. +- **AR results hang off the meter's anchor, with `.gravity` alignment.** ARKit corrects anchors as tracking improves, and drift runs about 2 cm a second, so a result tied to the meter moves with the correction. `.gravityAndHeading` depends on the compass, which developers report off by up to about 176° near a house. ## Conventions the code relies on -- `capturedImage` is landscape, as the sensor reads it, and the intrinsics match it. Save it unrotated. Any rotation must also rotate the intrinsics and every image-space annotation. Gemini, the box detector (Claude's docs call its coordinates approximate), returns `[ymin, xmin, ymax, xmax]` on a 0 to 1000 scale, so apply EXIF rotation before inference and before casting. Depth maps are 256 × 192, so scale the intrinsics down to match. -- ARKit camera space is x right, y up, looking down −z, with image v growing downward. The ray through pixel (u, v) has direction ((u − cx)/fx, −(v − cy)/fy, −1), rotated by the camera-to-world transform. -- Keep a frame when the phone has moved 0.5 m or turned 15° since the last one kept. -- Mesh vertices are local to their `ARMeshAnchor`: multiply each by the anchor's transform while copying it out, and never write back into ARKit's buffer. Set `automaticallyConfigureSession = false`, or RealityKit turns mesh classification off. -- On the mesh, the gap to a facing fence is a ray straight out from the wall at 1.5 ft high, and headroom is a ray straight up from 1 ft out. The mesh has holes, so cast a small fan and take a robust minimum. LiDAR reads from about 0.5 m to 5 m, so capture stands 1 to 3 m from the wall, and a miss beyond 5 m is unknown, not open. -- Default error bars are 0.3 ft for an AR tap, 0.5 ft for the LiDAR mesh and 1.5 ft for a position from photo detection. They are untested estimates. The server adds 0.16 ft per foot along the wall from the meter, from measured ARKit drift. +These are the non-obvious facts behind numbers in the code, and where each number came from. + +- `capturedImage` is landscape, as the sensor reads it, and the intrinsics match it. Any rotation must also rotate the intrinsics and every image-space annotation. Gemini is the box detector, because Claude's docs call Claude's coordinates approximate. Gemini returns `[ymin, xmin, ymax, xmax]` on a 0 to 1000 scale, so apply EXIF rotation before inference and before casting a box. +- Depth maps are 256 × 192. Scale the intrinsics down to match. +- ARKit camera space is x right, y up, looking down −z, with image v growing downward. The ray through pixel (u, v) points along ((u − cx)/fx, −(v − cy)/fy, −1) before the camera-to-world rotation. +- Keyframes are kept every 0.5 m or 15° of movement. +- Mesh vertices are local to their `ARMeshAnchor`. Copy each one out of ARKit's buffer, then multiply the copy by the anchor's transform, leaving the buffer untouched. RealityKit turns mesh classification off unless `automaticallyConfigureSession` is `false`. +- On the mesh, the gap to a facing fence is a ray straight out from the wall at 1.5 ft high. Headroom is a ray straight up from 1 ft out. The mesh has holes, so each measurement is the smallest distance over a small fan of rays. +- LiDAR reads from about 0.5 m to 5 m, so capture stands 1 to 3 m from the wall. A ray that meets nothing within 5 m is unknown, not open. +- The default error bars are day-1 estimates, untested: 0.3 ft for an AR tap, 0.5 ft for the LiDAR mesh and 1.5 ft for a position from photo detection. The server adds 0.16 ft per foot along the wall from the meter, from measured ARKit drift. ## Evidence so far -Public datasets with laser-scanned or surveyed ground truth. Sources: `experiments/evals/README.md` (PR #12), `recon/HANDOFF.md` (PR #20), `experiments/meter-closeup/README.md` (PR #16) and `experiments/panel-label/README.md` (PR #17). +These results come from public datasets with laser-scanned or surveyed ground truth. Nothing here was measured on our phone yet. The sources are `experiments/evals/README.md` (PR #12), `recon/HANDOFF.md` (PR #20), `experiments/meter-closeup/README.md` (PR #16) and `experiments/panel-label/README.md` (PR #17). -- **Tracking.** A current iPhone (14 Pro Max, MARViN dataset) stayed inside the server's allowance: p90 position error 8.6, 13.4 and 18.5 in after 10, 20 and 30 ft, against 19.2, 38.4 and 57.6 in. That phone has LiDAR, and MARViN's reference may be scaled to its tracking. A 2018 iPhone 6s (ADVIO) ran two to three times over, partly from noise in that dataset's reference. -- **Learned depth.** A model's own scale is 4 to 12% off. Rescaled with the phone's poses, walls come within about 5 in at p90 at a realistic 2% pose error, and 2.8 in with exact poses. Edges, where clearances are measured from, stay at 8 in or worse. MapAnything's scale shifted with input resolution. -- **Reconstruction worker.** On the ETH3D building, wall p90 is 2.1 in with a laser scan standing in for LiDAR and 2.8 in from photos only, on different walls. -- **Coverage.** The app's first coverage map claimed 1.1 ft of a 19.3 ft wall that no photo saw. Requiring several view angles fixed that but dropped 6 to 8.5 ft of good wall; a depth test on LiDAR phones fixed it cleanly. The reconstruction worker, checked against depth, claims at most about 0.46 ft. -- **Meter reading.** Apple's on-device text recognition read the full meter number on 71 of 73 real photos but found the right line on only 21 of 75, because nameplates carry several numbers. A barcode held it on 20 of 24 photos that had one, and a list of three candidates held it on 85% of held-out photos. So the app offers three to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small. -- **Panel labels.** Untested: 7 open photos of US panels exist, against the 40 the test needs. +- **Tracking.** A current iPhone stayed inside the server's error allowance. On the MARViN dataset, an iPhone 14 Pro Max had p90 position error of 8.6, 13.4 and 18.5 in after 10, 20 and 30 ft, against an allowance of 19.2, 38.4 and 57.6 in. That phone has LiDAR, and MARViN's reference may be scaled to its tracking. A 2018 iPhone 6s on ADVIO ran two to three times over, partly from noise in that dataset's reference. +- **Learned depth.** A model's own scale is 4 to 12% off. Rescaled with the phone's poses, walls come within about 5 in at p90 at a realistic 2% pose error, and within 2.8 in with exact poses. Edges stay at 8 in or worse, and clearances are measured from edges. MapAnything's scale shifted with input resolution. +- **Reconstruction worker.** On the ETH3D building, wall p90 is 2.1 in with a laser scan standing in for LiDAR, and 2.8 in from photos only. The two runs measured different walls. +- **Coverage.** The app's first coverage map claimed 1.1 ft of a 19.3 ft wall that no photo saw. Requiring several view angles removed the false claim but also discarded 6 to 8.5 ft of good wall. A depth test on LiDAR phones removed it cleanly. Checked against depth, the reconstruction worker claims at most about 0.46 ft of unseen wall. +- **Meter reading.** Apple's on-device text recognition read the full meter number on 71 of 73 real photos. It picked the right line on only 21 of 75, because nameplates carry several numbers. A barcode held the number on 20 of the 24 photos that had one. A list of three candidates held it on 85% of held-out photos. So the app offers three candidates to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small. +- **Panel labels.** Not tested. Only 7 open photos of US panels exist, and the test needs 40. - **World models.** Not evaluated yet. ## Open questions - **Clearance values.** [04-prior-art-and-codes.md](04-prior-art-and-codes.md) has the public values with their citations. Base's own values go in the private rules file, along with which number on the meter plate Base needs. - **Disconnect box.** Does the install add a battery disconnect beside the meter? Base's public pages mention a wall-mounted transfer switch and a battery disconnect. -- **The 3D path.** Traditional reconstruction fused with LiDAR, learned depth rescaled with the phone's poses, or world models, which produce a whole 3D scene from photos or video. The winner needs real scale, edges within the error allowance, and a record of which parts it observed and which it filled in. -- **Accuracy on our phone.** Nothing above ran on our phone. The field test uses a current iPhone without LiDAR, the Measure Lab build (PR #7), a 30 ft tape and a real meter wall, following `experiments/evals/field/FIELD_SHEET.md` (PR #12); a dry run recovered a planted 1.5% scale error. It also checks the default error bars. LiDAR on siding in midday sun is untested too. +- **The 3D path.** The candidates are traditional reconstruction fused with LiDAR, learned depth rescaled with the phone's poses, and world models, which produce a whole 3D scene from photos or video. The winner needs real scale, edges within the error allowance, and a record of which parts it observed and which it filled in. +- **Accuracy on our phone.** The field test uses a current iPhone without LiDAR, the Measure Lab build (PR #7), a 30 ft tape and a real meter wall. It follows `experiments/evals/field/FIELD_SHEET.md` (PR #12), and a dry run recovered a planted 1.5% scale error. It also checks the default error bars. LiDAR on siding in midday sun is untested too. - **Pool and driveway.** Their checks need up to 13 ft of photographed ground. The alternative is one yes-or-no question, recorded as the homeowner's answer. -- **Hidden wall.** Something standing in front of the wall can hide it, and nobody owns that check yet. One proposal: the app shows the homeowner the photo of the chosen spot and asks, and runs the depth test on LiDAR phones. +- **Hidden wall.** Something standing in front of the wall can hide it, and nobody owns that check yet. One proposal is for the app to show the homeowner the photo of the chosen spot and ask, and to run the depth test on LiDAR phones. diff --git a/docs/04-prior-art-and-codes.md b/docs/04-prior-art-and-codes.md index caa1ef4e2..d33d6643f 100644 --- a/docs/04-prior-art-and-codes.md +++ b/docs/04-prior-art-and-codes.md @@ -1,53 +1,55 @@ # Public rules, code citations, prior art and licenses -Researched 2026-09-25 from public sources; Esri's imagery terms were rechecked on 2026-09-26. UNVERIFIED marks a claim nobody could confirm. Licenses change, so recheck one before building a product on it. +This was researched on 2026-09-25 from public sources, and Esri's imagery terms were rechecked on 2026-09-26. UNVERIFIED marks a claim nobody could confirm. Licenses change, so recheck one before building a product on it. ## What Base does today -- **Photo checklist, 9 photos** ([help page](https://help.basepowercompany.com/en/articles/10280641)): the meter close up with its number legible, the meter's surroundings from 10 or more steps back, the areas to its left and right, the adjacent wall corner to corner, behind the fence, the main breaker box, the main disconnect with its amperage readable, and the breaker box's surroundings. Base's engineering and installation teams review them and reach out within 2 days if they need more. Nothing public says whether AI is involved. -- **Placement rules** ([help page](https://help.basepowercompany.com/en/articles/10280705)): a 3 × 3 ft footprint about 36 in tall, within 20 ft of the meter and 1 ft of the wall, and 3 ft from gas meters, fences, AC units and other batteries. Not in front of meters, panels, solar equipment or windows. The meter is at most 6 ft up, and the meter and breaker box share a wall; the box can't be in a closet. 30 × 36 in of working space in front of the meter and panel. The transfer switch is about 13 in wide with 30 in of clearance, and the spec pages ask for 3 ft of wall for it, then 3 × 3 ft per battery. The main breaker is 100 to 200 A (150 to 200 A in Austin), and two batteries or solar need a 200 A panel. No concrete pad, because the base is integrated. -- **Hardware.** 25 and 50 kWh ground units, and Base Core: 39.2 kWh, 39.5 × 30.68 × 22 in, an 11 kW inverter, UL-listed ([pv magazine](https://pv-magazine-usa.com/2026/08/04/base-power-launches-39-2-kwh-u-s-made-base-core-home-battery-secures-1-billion-in-new-funding/)). A wall-mounted transfer switch and a battery disconnect; no public mention of a meter collar. +- **Photo checklist.** Base's [help page](https://help.basepowercompany.com/en/articles/10280641) asks for 9 photos. Four cover the meter: a close-up with the number legible, its surroundings from 10 or more steps back, and the areas to its left and right. The others show the adjacent wall corner to corner, the area behind the fence, the main breaker box, the main disconnect with its amperage readable, and the breaker box's surroundings. Base's engineering and installation teams review them and reach out within 2 days if they need more. Nothing public says whether AI is involved. +- **Placement rules.** The [help page](https://help.basepowercompany.com/en/articles/10280705) gives a 3 × 3 ft footprint about 36 in tall. The battery sits within 20 ft of the meter and 1 ft of the wall, and 3 ft from gas meters, fences, AC units and other batteries. It can't stand in front of meters, panels, solar equipment or windows. +- **Meter and panel.** The meter is at most 6 ft up, and the meter and breaker box share a wall. The box can't be in a closet. Both need 30 × 36 in of working space in front. The main breaker is 100 to 200 A, or 150 to 200 A in Austin, and two batteries or solar need a 200 A panel. +- **Transfer switch.** It is about 13 in wide with 30 in of clearance. Base's spec pages ask for 3 ft of wall for the switch, then 3 × 3 ft per battery. +- **Hardware.** Base sells 25 and 50 kWh ground units and Base Core: 39.2 kWh, 39.5 × 30.68 × 22 in, an 11 kW inverter, and a UL listing ([pv magazine](https://pv-magazine-usa.com/2026/08/04/base-power-launches-39-2-kwh-u-s-made-base-core-home-battery-secures-1-billion-in-new-funding/)). The install adds a wall-mounted transfer switch and a battery disconnect. Nothing public mentions a meter collar. The base is integrated, so no concrete pad is needed. - **Review.** A Deployment Site Designer job post, removed in November 2025, describes people reviewing customer site photos by hand in CAD. Heavy lifting is split from electrical work so crews can do "20 homes in a day". ## Public rule values -The demo's `server/rules.yaml` (PR #11) uses these. Base's own values are private and never appear in tracked files. +The demo's `server/rules.yaml` (PR #11) uses these values. Base's own values are private and stay out of tracked files. | Rule | Value | Source | | --- | --- | --- | | Cable run from the meter | within 20 ft | Base's help page | | Distance from the wall | within 1 ft | Base's help page | -| Gas meter or regulator | 3 ft | Base's help page; Austin Energy §1.9; Texas Gas Service | +| Gas meter or regulator | 3 ft | Base's help page, Austin Energy §1.9, Texas Gas Service | | AC units, fences, other batteries | 3 ft | Base's help page | | Doors and windows entering the home | 3 ft | IRC R328.4 | | Working space at the meter and panel | 30 in wide, 36 in deep, 6.5 ft headroom | NEC 110.26 | | Headroom over the battery | 6.5 ft | NEC 110.26, applied to the battery as a demo choice | -| Driveway, pool | none public | demo placeholders, 5 ft and 10 ft | +| Driveway, pool | no public value | demo placeholders of 5 ft and 10 ft | ## Code citations | Code | What it says | | --- | --- | -| **NEC 110.26** ([ICC](https://codes.iccsafe.org/s/ISEP2021P1/national-electrical-code-nec-solar-provisions/ISEP2021P1-NEC-Sec110.26)) | Working space 30 in wide, or the equipment's width, by 36 in deep with 6.5 ft headroom, and no storage in it. Base's page says "30 in high × 36 in wide", probably a garbled version | -| **Austin Energy Design Criteria, Dec 12 2023** ([PDF](https://austinenergy.com/-/media/project/websites/austinenergy/contractors/designcriteriamanual.pdf)) | §1.9.2: 30 in wide, 36 in deep and 6 ft 6 in headroom at meters; socket center 30 to 72 in above ground; at least 1 ft from doors and windows; no meter within a 3 ft radius of gas meters, regulators or relief valves. §1.12: generation enclosures keep the 3 ft radius | -| **Texas Gas Service Meter Setting Requirements** ([PDF](https://www.texasgasservice.com/media/tgs/constructionservices/metersettingrequirements_tgs.pdf)) | Electric meters and outlets 3 ft from the regulator relief vent, and 3 ft from operable doors and windows | -| **IRC R328, 2021, from NFPA 855** ([ICC](https://codes.iccsafe.org/s/IRC2021P2/part-iii-building-planning-and-construction/IRC2021P2-Pt03-Ch03-SecR328)) | UL 9540 listed; outdoors or on exterior walls at least 3 ft from doors and windows; 3 ft between batteries unless UL 9540A testing allows closer | -| **NEC 706.15** | Battery disconnect within sight, and within 10 ft or lockable | -| **NEC 230.85** | Outdoor emergency disconnect; moved into 230.70 in the 2026 NEC | -| ComEd opening rules | UNVERIFIED; cite Austin Energy or Texas Gas Service instead | +| **NEC 110.26** ([ICC](https://codes.iccsafe.org/s/ISEP2021P1/national-electrical-code-nec-solar-provisions/ISEP2021P1-NEC-Sec110.26)) | Working space 30 in wide, or the equipment's width, by 36 in deep, with 6.5 ft of headroom and no storage in it. Base's page says "30 in high × 36 in wide", probably a garbled version. | +| **Austin Energy Design Criteria, Dec 12 2023** ([PDF](https://austinenergy.com/-/media/project/websites/austinenergy/contractors/designcriteriamanual.pdf)) | §1.9.2 asks for 30 in wide, 36 in deep and 6 ft 6 in of headroom at meters, a socket center 30 to 72 in above ground, and at least 1 ft from doors and windows. No meter may sit within a 3 ft radius of gas meters, regulators or relief valves. §1.12 keeps that 3 ft radius for generation enclosures. | +| **Texas Gas Service Meter Setting Requirements** ([PDF](https://www.texasgasservice.com/media/tgs/constructionservices/metersettingrequirements_tgs.pdf)) | Electric meters and outlets stay 3 ft from the regulator relief vent and 3 ft from operable doors and windows. | +| **IRC R328, 2021, from NFPA 855** ([ICC](https://codes.iccsafe.org/s/IRC2021P2/part-iii-building-planning-and-construction/IRC2021P2-Pt03-Ch03-SecR328)) | Batteries are UL 9540 listed and sit outdoors or on exterior walls at least 3 ft from doors and windows. Batteries stay 3 ft apart unless UL 9540A testing allows closer. | +| **NEC 706.15** | The battery disconnect is within sight, and within 10 ft or lockable. | +| **NEC 230.85** | An outdoor emergency disconnect. The 2026 NEC moves it into 230.70. | +| ComEd opening rules | UNVERIFIED. Cite Austin Energy or Texas Gas Service instead. | ## How others do it -- **Tesla Powerwall:** a guided self-survey on the phone, about 30 minutes, then photo upload and remote design. -- **Qmerit (EV chargers):** photo upload plus "Panel Insights", built with Schneider Electric, which reads breaker spaces and capacity from one panel photo ([announcement](https://qmerit.com/news/qmerit-deploying-ai-for-faster-safer-more-accurate-estimates-for-home-ev-charging-installations/)). The closest precedent for using AI only to recognize things. -- **SolarAPP+ (NREL):** instant automated code checks for solar and storage permits, the precedent for a rules engine in place of judgment calls. -- **Aurora Solar** has a Site Surveys API in beta ([docs](https://docs.aurorasolar.com/reference/site-surveys)); **Scanifly** uses drone photogrammetry. -- **Heat pumps:** Aira, Mitsubishi Ecodan and Alpha Innotec offer AR placement that is visual only, with no clearance checks. Fraunhofer ISE's Heat Pump PlanAR, reported 2026-09-23, scans indoor boiler rooms and optimizes placement: the nearest analogue, but indoors. -- **Hover** builds a measured exterior 3D model from phone photos and has a JSON API ([docs](https://developers.hover.to/reference/measurements-and-deliverables)), with turnaround likely in hours; useful as a later cross-check. EagleView sells wall, window and door measurements to enterprises. Polycam's API and Matterport's sandbox are closed to us, magicplan is indoor, and Canvas says it drifts outside. +- **Tesla Powerwall** runs a guided self-survey on the phone of about 30 minutes, then designs remotely from the uploaded photos. +- **Qmerit**, for EV chargers, pairs photo upload with Panel Insights, built with Schneider Electric, which reads breaker spaces and capacity from one panel photo ([announcement](https://qmerit.com/news/qmerit-deploying-ai-for-faster-safer-more-accurate-estimates-for-home-ev-charging-installations/)). It is the closest precedent for using AI only to recognize things. +- **SolarAPP+**, from NREL, runs instant automated code checks for solar and storage permits. It is the precedent for a rules engine in place of judgment calls. +- **Aurora Solar** has a Site Surveys API in beta ([docs](https://docs.aurorasolar.com/reference/site-surveys)). **Scanifly** uses drone photogrammetry. +- **Heat pumps.** Aira, Mitsubishi Ecodan and Alpha Innotec offer AR placement that is visual only, with no clearance checks. Fraunhofer ISE's Heat Pump PlanAR, reported on 2026-09-23, scans indoor boiler rooms and optimizes placement. It is the nearest analogue, but it works indoors. +- **3D capture products.** Hover builds a measured exterior 3D model from phone photos and has a JSON API ([docs](https://developers.hover.to/reference/measurements-and-deliverables)), with a turnaround likely measured in hours, so it suits a later cross-check. EagleView sells wall, window and door measurements to enterprises. Polycam's API and Matterport's sandbox are closed to us, magicplan works indoors, and Canvas says it drifts outside. -Measurements from prior work: iPhone 12 Pro Max LiDAR on facades is within about ±6 to 8 cm of a total station, and iPad Pro LiDAR reaches about 5 m. A YOLOv5 meter-reading study read the counters 97% of the time and serial numbers only 63%. +Prior measurements: iPhone 12 Pro Max LiDAR on facades lands within about ±6 to 8 cm of a total station, and iPad Pro LiDAR reaches about 5 m. A YOLOv5 meter-reading study read the counters 97% of the time and serial numbers only 63%. -What nobody else does: the whole loop outdoors, from metric AR capture through a solver that cites a code for each clearance to an AR preview. Heat-pump AR tools are visual only, and Fraunhofer's solver works indoors. +Nobody else runs the whole loop outdoors, from metric AR capture through a solver that cites a code for each clearance to an AR preview. The heat-pump AR tools are visual only, and Fraunhofer's solver works indoors. ## Licenses and terms @@ -55,22 +57,22 @@ What nobody else does: the whole loop outdoors, from metric AR capture through a | Source | Terms | | --- | --- | -| Google Maps, Street View | ToS §3.2.3(c)(vii) bans using the content to "train, test, validate or fine-tune" ML, and tracing building outlines from its imagery. The Solar API's §20.1 allows use "to determine the feasibility of installing energy systems"; whether running inference on it conflicts with §3.2.3(c) is UNVERIFIED | -| Mapbox | §1.5(ii) bans using it to "train, operate or improve" ML; operating includes inference | -| Esri World Imagery | Automated extraction only from an exported tile package, inside ArcGIS, with derived results for non-commercial use ([Esri](https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/imagery/learn-to-use-ai-to-extract-information-from-world-imagery)) | -| TxGIO StratMap orthoimagery ([site](https://geographic.texas.gov/stratmap/index.html)) | Public domain; 6 in pixels inside Austin, 12 in elsewhere | -| NAIP, via Planetary Computer | Public domain; 0.6 m in Pflugerville (2022), 0.3 m in Naperville (2023). At 0.6 m a 3 to 5 ft driveway buffer is about 2 pixels | -| Overture buildings and transportation, OSM | ODbL: share-alike applies only if you publish a derived database. OSM tags driveways and pools on under 5% of homes. On 2026-09-25 Pflugerville had 14,675 houses, 149 driveways and 102 pools; Round Rock 9,223 houses, 476 driveways and 170 pools; Austin 302k buildings, 6,029 driveways and 3,339 pools | +| Google Maps, Street View | ToS §3.2.3(c)(vii) bans using the content to "train, test, validate or fine-tune" ML, and bans tracing building outlines from its imagery. The Solar API's §20.1 allows use "to determine the feasibility of installing energy systems". Whether running inference on it conflicts with §3.2.3(c) is UNVERIFIED. | +| Mapbox | §1.5(ii) bans using it to "train, operate or improve" ML, and operating includes inference. | +| Esri World Imagery | Automated extraction must start from an exported tile package, stay inside ArcGIS, and keep derived results non-commercial ([Esri](https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/imagery/learn-to-use-ai-to-extract-information-from-world-imagery)). | +| TxGIO StratMap orthoimagery ([site](https://geographic.texas.gov/stratmap/index.html)) | Public domain, with 6 in pixels inside Austin and 12 in elsewhere. | +| NAIP, via Planetary Computer | Public domain, at 0.6 m in Pflugerville (2022) and 0.3 m in Naperville (2023). At 0.6 m a 3 to 5 ft driveway buffer is about 2 pixels. | +| Overture buildings and transportation, OSM | ODbL, whose share-alike applies only if you publish a derived database. OSM tags driveways and pools on under 5% of homes. On 2026-09-25 Pflugerville had 14,675 houses, 149 driveways and 102 pools. Round Rock had 9,223 houses, 476 driveways and 170 pools. Austin had 302k buildings, 6,029 driveways and 3,339 pools. | -**Models.** Before relying on one, check its weights' license separately from its code's. +**Models.** Check a model's weights license separately from its code license. | Status | Models | | --- | --- | -| Noncommercial | MapAnything's default checkpoint (use `facebook/map-anything-apache`); Depth Anything 3 Giant and Nested (Metric Large, Base and Small are Apache-2.0); VGGT-Ω; UniDepthV2; Mapillary-Vistas Mask2Former weights; Molmo 2 | -| Restricted or unclear | Depth Pro (Apple personal-use grant); SAM 3 (custom license, gated weights); HY-World 2.0 (custom community license); Metric3D v2 (files disagree); MoGe-2 weights (UNVERIFIED; code is MIT) | -| Copyleft | YOLO-World, YOLOE and Ultralytics (GPL or AGPL); pymeshlab (GPL-3) | -| Permissive | SAM 2, Grounding DINO, OWLv2, Qwen3-VL, zxing-cpp (Apache-2.0); Florence-2, supervision, Open3D, Stray Scanner (MIT); shapely (BSD-3) | +| Noncommercial | MapAnything's default checkpoint (use `facebook/map-anything-apache` instead), Depth Anything 3 Giant and Nested (Metric Large, Base and Small are Apache-2.0), VGGT-Ω, UniDepthV2, Mapillary-Vistas Mask2Former weights, Molmo 2 | +| Restricted or unclear | Depth Pro (Apple personal-use grant), SAM 3 (custom license, gated weights), HY-World 2.0 (custom community license), Metric3D v2 (its files disagree), MoGe-2 weights (UNVERIFIED, though the code is MIT) | +| Copyleft | YOLO-World, YOLOE and Ultralytics (GPL or AGPL), pymeshlab (GPL-3) | +| Permissive | SAM 2, Grounding DINO, OWLv2, Qwen3-VL and zxing-cpp (Apache-2.0), Florence-2, supervision, Open3D and Stray Scanner (MIT), shapely (BSD-3) | -For recognition boxes, Gemini 2.5 Pro scored 13.3 zero-shot mAP on RF100-VL against 1.5 for GPT-5. +For recognition boxes, Gemini 2.5 Pro scored 13.3 zero-shot mAP on RF100-VL, against 1.5 for GPT-5. -**Datasets.** The evals use ADVIO (CC BY-NC 4.0), ETH3D (CC BY-NC-SA 4.0) and MARViN (no license stated) only to measure accuracy, and never redistribute them. Get permission before any commercial use. For training data, Roboflow Universe has gas meter sets ([402 images](https://universe.roboflow.com/proba-vwwtl/gas-meter-recognition), [1.4k images](https://universe.roboflow.com/gas-meter-zbuni/gas-meter-g7kh6)) and electric meter sets ([wattwise](https://universe.roboflow.com/wattwise/electric-meter-wzeeg-zk5fd), [1,943 images](https://universe.roboflow.com/abhinav-kumar-do8z1/utility-meter-reading-dataset-for-automatic-reading-yolo-z0e1h)), and Open Images V7 labels windows and doors. No public data exists for window wells, and open photos of US breaker panels are scarce: PR #17 found 7 against the 40 its test needs. +**Datasets.** The evals use ADVIO (CC BY-NC 4.0), ETH3D (CC BY-NC-SA 4.0) and MARViN (no license stated) only to measure accuracy, and never redistribute them. Get permission before any commercial use. For training data, Roboflow Universe has gas meter sets of [402](https://universe.roboflow.com/proba-vwwtl/gas-meter-recognition) and [1.4k](https://universe.roboflow.com/gas-meter-zbuni/gas-meter-g7kh6) images and electric meter sets ([wattwise](https://universe.roboflow.com/wattwise/electric-meter-wzeeg-zk5fd), [1,943 images](https://universe.roboflow.com/abhinav-kumar-do8z1/utility-meter-reading-dataset-for-automatic-reading-yolo-z0e1h)). Open Images V7 labels windows and doors. No public data exists for window wells, and open photos of US breaker panels are scarce: PR #17 found 7 against the 40 its test needs. From fcd6405f9930d288652a7d82fdd1aa1210e787fa Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 14:26:40 -0500 Subject: [PATCH 09/16] docs: ship shared writing, planning and review skills for all agents - Add six skills under .agents/skills, where Codex finds them, with per-skill links from .claude/skills for Claude Code: unslop, technical-writing, pr-writing, efficient-implementation-plans and deslop, adapted from the lead's personal set with personal references and cross-links to unshipped skills removed, plus a new prompt-writing skill distilled from Anthropic, OpenAI and Matt Pocock's guidance. - AGENTS.md says which skill applies to which kind of work and that a user's instruction outranks a skill; the README maps the folder. Co-Authored-By: Claude Opus 5.5 (1M context) --- .agents/skills/deslop/SKILL.md | 132 ++++++++++++++++++ .../efficient-implementation-plans/SKILL.md | 63 +++++++++ .agents/skills/pr-writing/SKILL.md | 32 +++++ .../pr-writing/references/visual-evidence.md | 27 ++++ .../pr-writing/references/writing-examples.md | 31 ++++ .agents/skills/prompt-writing/SKILL.md | 47 +++++++ .agents/skills/technical-writing/SKILL.md | 129 +++++++++++++++++ .agents/skills/unslop/SKILL.md | 105 ++++++++++++++ .claude/skills/deslop | 1 + .claude/skills/efficient-implementation-plans | 1 + .claude/skills/pr-writing | 1 + .claude/skills/prompt-writing | 1 + .claude/skills/technical-writing | 1 + .claude/skills/unslop | 1 + AGENTS.md | 12 ++ README.md | 1 + 16 files changed, 585 insertions(+) create mode 100644 .agents/skills/deslop/SKILL.md create mode 100644 .agents/skills/efficient-implementation-plans/SKILL.md create mode 100644 .agents/skills/pr-writing/SKILL.md create mode 100644 .agents/skills/pr-writing/references/visual-evidence.md create mode 100644 .agents/skills/pr-writing/references/writing-examples.md create mode 100644 .agents/skills/prompt-writing/SKILL.md create mode 100644 .agents/skills/technical-writing/SKILL.md create mode 100644 .agents/skills/unslop/SKILL.md create mode 120000 .claude/skills/deslop create mode 120000 .claude/skills/efficient-implementation-plans create mode 120000 .claude/skills/pr-writing create mode 120000 .claude/skills/prompt-writing create mode 120000 .claude/skills/technical-writing create mode 120000 .claude/skills/unslop diff --git a/.agents/skills/deslop/SKILL.md b/.agents/skills/deslop/SKILL.md new file mode 100644 index 000000000..474ede13d --- /dev/null +++ b/.agents/skills/deslop/SKILL.md @@ -0,0 +1,132 @@ +--- +name: deslop +description: Code quality review that reports AI slop, small local design issues, and larger structural refactors with a follow-up prompt. Explicit cleanup requests permit local, behavior-preserving fixes. On TypeScript and JavaScript diffs, also checks low-evidence typing, including casts, widening, and mocks that hide what the author already knew. Use when reviewing a diff, cleaning touched code, removing AI slop, or cleaning a diff before a commit. +--- + +# Cleanup-first code quality review + +Check the diff against `main`. If the user names a different scope, use that +scope instead. A review request reports findings without editing files. An explicit +cleanup request, or a commit workflow that asks for cleanup, permits local, +behavior-preserving edits. In that edit mode, fix the little things first, +then do a brief code-quality pass on the touched area. + +When a commit workflow invokes this skill, the intended staged snapshot is the +scope. Stay inside that boundary so the commit can restage without picking up +unrelated work. + +## Coverage + +- Inventory every changed path in the chosen diff. Inspect every changed hunk; do not rely on a representative sample. +- Account for code, tests, documentation, configuration, migrations, generated files, and lockfiles. Use the right review depth for each file type, but do not silently skip any path. +- Recheck the inventory after cleanup. New or materially changed hunks require another pass. +- Tools such as linters, typecheckers, tests, and searches support the review; they do not replace reading the diff. +- On TypeScript or JavaScript hunks, if the repo already runs anti-slop Oxlint rules, use their output on the touched files as extra evidence; otherwise check the same problems by reading the diff. + +## Direct fixes in edit mode + +- Extra comments that are unnecessary or inconsistent with local style +- Stale, misleading, or obvious comments; preserve comments that explain intent, constraints, ownership, or surprising behavior +- Defensive checks or `try`/`catch` blocks that are abnormal for trusted code paths +- `any`, unsafe casts, or loose types used only to bypass real contracts +- Inline imports in Python that should live with the rest of the imports +- Deeply nested code that should be flattened with early returns or a small helper +- Vague, misleading, or needlessly abbreviated names for variables, functions, types, files, and tests +- Local style inconsistencies with the file and surrounding codebase + +## Quality Signals + +### Good patterns to preserve + +- Clear ownership: one module, component, or service has one main job +- Centralized invariants: shared types, schemas, normalization, and validation live in one place +- Simple control flow: direct branches, early returns, readable state transitions +- Type-safe interfaces: contracts are expressed in types instead of dodge-casts +- Derived state over repaired state: effects bridge external systems instead of bookkeeping local state +- Explicit states and transitions: loading, empty, success, failure, cancellation, retry, and cleanup behavior are coherent where relevant +- Scalable boundaries: split rendering, orchestration, shaping, and side effects when a file is clearly mixing them +- Performance-aware reactive code: avoid duplicate state, repeated heavy work, and unnecessary rerender or effect churn +- Useful comments: explain intent or constraints, not obvious code + +### Bad patterns to report, or fix in edit mode when local and behavior-preserving + +- Patch-on-patch growth: more guards, flags, branches, or effects layered onto already confused logic +- Mixed responsibilities: one file doing orchestration, transformation, rendering, persistence, and error handling at once +- Duplicated business logic: repeated normalization, model mapping, status derivation, or fallback behavior +- Repair logic instead of ownership: `useEffect` or watchers keeping state in sync after the fact +- Missing, duplicated, contradictory, or impossible states; stale state that survives retries, cancellation, navigation, or teardown +- Weak typing: `any`, unsafe casts, loose shapes, or ad hoc object contracts +- Cosmetic abstraction: more wrappers or helpers while the underlying flow gets harder to follow +- Hidden performance costs: effect spam, unnecessary recomputation, repeated expensive work, duplicated subscriptions +- AI-slop artifacts: needless comments, over-defensive code, placeholder prose, awkward helper names, and style drift + +### Design-level patterns to catch + +Use these as review signals, not automatic findings. Tie each claim to the changed code and current goal. + +- Over-engineering the solution. +- Overly defensive programming. +- Hyper-fixating on rare or fictitious edge cases. +- Subsystems with overlapping responsibilities. +- Solving a problem at the wrong architectural layer. +- Duplicating sources of truth, then adding machinery to keep them synchronized. +- Prematurely generalizing a one-off flow. +- Adding timeouts everywhere without a concrete need to bound an operation. +- Adding production code that exists only to satisfy tests. +- Patching a bad premise additively instead of stepping back and deleting or replacing it. + +## Type evidence (TypeScript / JavaScript) + +On changed TypeScript and JavaScript hunks, also treat as slop any code that +discards or fabricates type evidence: the author knew the real type and hid +it from the compiler. Prefer inference, `as const`, `satisfies`, named owner +types, and parsing at the boundary. + +In edit mode, fix directly when the edit is local and behavior-preserving. In review mode, report these findings: + +- Assertion chains that fabricate evidence: `as object as User`, widening a known value to `unknown`, `object`, or an index signature and asserting it back, or annotating a known object as `Record` so its keys disappear. +- `unknown`, `object`, or `{}` standing in for a contract the author could have named, on parameters, returns, aliases, or dictionary values. An explicit error-`cause` convention is fine. +- `Reflect.get` or `Reflect.apply` where ordinary typed access works. + +Route through the structural handoff when the real fix is not local: + +- `vi.mock` / `jest.mock`-style module mocks that stand in for a missing dependency seam. +- Ad hoc `typeof` narrowing of unparsed input that should be parsed once at the boundary. + +A truly necessary assertion stays. If it lacks a `SAFETY:` comment (or the repo's +equivalent) stating the invariant TypeScript cannot express, recommend that +comment in review mode or add it immediately above the assertion in edit mode. A placeholder justification, or a SAFETY comment laundering an +assertion that should not exist, is itself slop. Do not weaken, suppress, or +disable lint rules to pass. + +## Behavior + +1. Direct cleanup pass: report obvious slop in review mode; fix it in edit mode. +2. Small quality pass: identify small, clear, behavior-preserving improvements in the touched area. Apply them only in edit mode. Check names, comments, types, control flow, state transitions, ownership, tests, and local consistency where they apply. On TypeScript and JavaScript hunks, also apply Type evidence. +3. Structural triage: if the issue is larger than a local cleanup, do not do the big rewrite by default. + +Before finishing, re-read changed hunks and confirm that every inventoried path +was handled. + +## Placeholders (minor) + +Real gaps are sometimes written as prose (`// later`, `// for now`, `// temporary`, “should eventually…”, empty stubs) **without** `TODO:`. When the comment clearly means deferred work, recommend one line `TODO: ` so intent is grep-friendly and explicit. Apply that replacement only in edit mode. Skip if the code is done; leave existing `TODO:` or `FIXME:` that already read well. + +## Escalate Instead of Rewrite + +- Report the little things first in review mode; fix them first in edit mode. +- If the structural problem is larger than a modest local refactor, stop short of a broad rewrite. +- Append a compact handoff with: +- `Structural issue: ...` +- `Why it matters: ...` +- `Refactor direction: ...` +- `Follow-up prompt: ...` + +## Guardrails + +- Keep cleanup behavior unchanged. Report clear bugs separately; fixing one requires a separately authorized bug-fix scope, not just cleanup or commit authorization. +- Prefer minimal, focused edits and modest local refactors over broad rewrites. +- In edit mode, use behavior-preserving targeted restructuring when the boundary is obvious and it clearly improves the touched area. In review mode, recommend it without edits. +- Do not claim an exhaustive review when any target path or changed hunk was skipped. State the limitation instead. +- Keep the final summary concise (1-3 sentences) when no escalation is needed. +- Do not install dependencies, rewrite lint config, or expand a commit to vendor tooling. Type-evidence cleanup is in-diff only; plugin setup belongs to a later explicit request. diff --git a/.agents/skills/efficient-implementation-plans/SKILL.md b/.agents/skills/efficient-implementation-plans/SKILL.md new file mode 100644 index 000000000..e4e0a5945 --- /dev/null +++ b/.agents/skills/efficient-implementation-plans/SKILL.md @@ -0,0 +1,63 @@ +--- +name: efficient-implementation-plans +description: Write, review, or revise software implementation plans. Use for planning requests or reviews of planning and testing overhead, not ordinary coding. +--- + +# Efficient implementation plans + +Make the requested outcome reliably achievable and done extremely well with proportionate product, planning, and verification effort. Every additional requirement, abstraction, test, or supporting system needs a concrete benefit for the current goal. + +## Establish the outcome against the code + +Read the current request, agreed constraints, existing plan, and relevant code before proposing replacements. For behavior changes, follow the relevant entry point through the components responsible for the result. Identify what already works and what must change. Investigate a named uncertainty instead of surveying the whole repository. + +In the plan, distinguish verified capabilities from proposed mechanisms. For low-impact, reversible choices, a reasonable assumption or small experiment may be more useful than further investigation. A failed experiment can be useful if its effects are contained and its result informs the next decision. For a consequential uncertain capability, state the assumption and a proportionate check or exploratory implementation that can resolve it before dependent work relies on it. For example, a shared database does not establish that an existing email queue can join its transaction. Verify that capability before promising atomic insertion or budgeting the work as a handler-only change. + +**Define an observable finish first.** State the delivered behavior, scope, relevant dependencies, and sufficient acceptance criteria. Separate binding outcomes and safety boundaries from suggested implementation details. Do not turn "robust" or "production-ready" into an unbounded list of hypothetical requirements. Prefer the simplest design that fully meets the current goal. Discuss the outcome with the user when it is still being shaped or a consequential choice is unsettled, so you are on the same page. Use agreement already established instead of asking for it again. + +Match the requested action: discussion, review, drafting, or editing. Interpret follow-up questions in light of authorization already given. Discussion and review alone do not authorize file changes, implementation, publication, or changes to project rules. Follow existing project contracts; this skill does not authorize silently dropping agreed gates. Propose changes to binding outcomes, scope, or safety requirements for the user's decision. Make routine implementation and verification choices autonomously within the authorized scope. Keep independent work moving while a necessary answer is pending. + +## Make ownership and dependencies concrete + +Name the contracts that matter: inputs and outputs, state that must remain true, and meaningful failure behavior. At an authority boundary, one component owns a decision or state change. State which component owns it and what callers may assume. For example, a client requests approval; the server verifies the current revision and permission before the worker acts. A hidden button alone does not enforce that contract. + +Prefer a small complete behavior through the layers it touches. Each implementation slice needs an outcome, real prerequisites, and sufficient proof. Put the uncertain integration early enough to change the design before surrounding work accumulates. + +System boundaries and work assignments are different. A feature may cross several component owners and still belong in one reviewable PR. Split it when a contract, independent outcome, migration, or permission boundary makes a separate deliverable useful. Scale migration and release safeguards to actual users, retained data, and dependent systems. Where continuity is required, plan usable intermediate states. Otherwise, a direct replacement may be appropriate without compatibility layers or fallback infrastructure. Avoid dividing every feature into disconnected database, API, and UI tasks. + +**Justify expensive tests and new infrastructure. Choose infrastructure for the work it enables.** Reuse existing tools when they fit the goal and simplify the work. Build or replace infrastructure when it better serves the goal. Compare setup and maintenance effort with the benefit to development, verification, or deployment at the project's current stage. A small deployment workflow or repeatable test scene may be the simplest way to get reliable feedback. Make substantial supporting work visible in the plan; do not hide a new testing system under "add tests." + +For multiple PRs, state the base or prerequisite of each and the behavior its own diff adds. For parallel work, the plan names the independent scopes and integration responsibility. Follow the project's existing isolation and dispatch rules; a separate chat is not file isolation. + +## Choose the cheapest sufficient proof + +Development checks, targeted boundary checks, and coherent end-to-end acceptance are available layers, not three mandatory steps for every task. Judge risk by the actual effects of a change; a one-line permission change can affect many actions. Match each consequential acceptance claim to a check that could disprove it. Prefer a focused regression for bug fixes; a costly red baseline for a new feature requires a concrete purpose or explicit contract. Test a strict local invariant directly when it has a precise answer. + +When the claimed outcome is a user journey, specify its normal entry point, destination, and observable result. An internal function call or prepared fixture proves only the boundary exercised. If credentials, hardware, or deployment access are missing, preserve that acceptance gap and continue the work that can be proved locally. + +**Match verification effort to risk.** Ask what plausible incorrect implementation a costly test would catch that cheaper checks would miss. Choose variants for distinct risks and interactions, not an automatic Cartesian product of platforms, providers, and scenarios. Once the relevant checks pass, broaden or repeat them only for new changes, failures, or unresolved concerns. + +**Keep the first working version focused.** For an early prototype, prioritize getting the smallest complete behavior working and evaluating it. Newly introduced hashing, evidence ledgers, or release machinery are a signal to revisit the current goal and implementation stage. In a small prototype they usually mean the steering is off. Leave them out unless a concrete requirement of that stage needs them. Use existing Git and test records for ordinary implementation work. Preserve safeguards for actual effects, and do not invent future risks to justify additional machinery. + +For costly verification, put these conditions in the plan where the implementer needs them: + +- **Rerun according to impact.** A result remains evidence for its tested artifact; assess its applicability to the delivered state using relevant source, dependencies, packaging, build, configuration, and environment changes. A reporter or selector change preserves unrelated evidence only if entry, fixtures, assertions, and success detection remain valid. Renew affected proofs and explain reuse briefly. Investigate uncertain impact and broaden verification where needed. A new revision alone does not require every previous live run. +- **Diagnose before expensive repetition.** Use the failure evidence to choose the next investigation. For example, you can classify failures as product, harness, infrastructure/external service, or access problems at the first failure. Narrow the reproduction when doing so preserves the relevant conditions; keep or broaden the scope when interactions matter. Another expensive run needs to answer a specific question, test a relevant correction, or follow a changed prerequisite. A likely transient failure may justify a bounded retry. Fix the real cause rather than stacking superficial patches, and verify the originally failing behavior -- and oftentimes it is at the smallest reproducible boundary. +- Verify the delivered state collectively. A selector fix cannot make an unreached assertion retrospectively pass. A successful retry establishes that run's outcome; retain original failures and retries when measuring reliability. + +Use unexpected verification effort to reconsider the method, especially when further investigation is unlikely to change the next decision. Improve the approach autonomously within the agreed scope and communicate material deviations. Preserve agreed outcomes and safety requirements. + +## Maintain one current plan + +When plan edits are authorized, update the existing canonical plan in place. Replace superseded decisions and remove obsolete steps when requirements change. Preserve current constraints and the reason behind a non-obvious choice. Carry an old open question forward only if its answer affects the current outcome or next action. Keep execution details in existing task state, PRs, or test reports, with only the links needed to resume or assess the work. Do not prepend updates, preserve old plan versions inside the document, or move all removed prose into a new archive by default. + +For a substantial task, a useful shape is: + +- Goal and scope: the observable finish and the exclusions that prevent likely drift. +- Design: existing components, necessary changes, and consequential contracts or ownership. +- Implementation slices: outcomes in dependency order, with integration and PR boundaries where needed. +- Acceptance and open decisions: sufficient checks, material limitations, and prerequisites still needed. + +Use only the sections the task needs. A small change can be a paragraph. Include verified source locations when they help the implementer start; avoid speculative file inventories and pseudocode that merely restates prose. Keep later stages at outcome level until their dependencies are understood. A plan's length follows its decisions, not a prescribed template or line count. + +**Make reviews subtractive and finishable.** Resolve actionable ambiguities, contradictions, and relevant coverage gaps; remove redundant obligations and obsolete assumptions. A new mandatory requirement needs a current goal or concrete risk. Finish planning when the implementer can identify the next complete outcome, its prerequisites, and sufficient proof. Reopen settled decisions only with new information. Do not launch repeated whole-plan audits or additional reviewers without a specific unresolved question. diff --git a/.agents/skills/pr-writing/SKILL.md b/.agents/skills/pr-writing/SKILL.md new file mode 100644 index 000000000..d8f3b7829 --- /dev/null +++ b/.agents/skills/pr-writing/SKILL.md @@ -0,0 +1,32 @@ +--- +name: pr-writing +description: Write or revise pull request titles, descriptions and visual evidence. Use when preparing a PR, improving its presentation, or updating it after scope changes. Writing alone does not publish the PR. +--- + +# Write a reviewable PR + +Read the repository's template and contributor guidance, the diff against the actual PR base, and the current PR body before drafting. Apply `unslop` and `technical-writing` when installed. Preserve useful human edits, existing evidence links and bot-managed sections. Refresh the body before publishing an edit so another author's changes are not overwritten. + +## Explain the change + +Lead with the concrete problem and resulting behavior. Name the affected screen, operation or interface so someone outside the task can understand it. For a substantial change, explain the old behavior, its consequence, and how the change fixes it. For a new feature, describe the new capability without inventing a broken predecessor. + +Order independent changes by user impact or technical consequence. Headings identify the change; filenames and commit order are not the outline. Explain the mechanism where it helps assess correctness or a tradeoff. Link the relevant diff when available, otherwise the file at the reviewed revision. Include prior research only when it supports a consequential choice, with a direct source and the reason it matters. + +Write connected prose with concrete subjects and verbs. Avoid clipped status phrases, unexplained internal names, claims of thoroughness and narration about the reviewer or your process. Keep useful technical detail; remove sentences that add no decision-relevant information. Read the short [writing examples](references/writing-examples.md) before drafting. Follow their level of explanation, not their sentence pattern or length. + +Scale the body to the change. A small repair needs a short explanation and focused verification. A broad PR can use sections by behavior with evidence beside each claim. Keep long logs, inventories and test matrices in linked artifacts. Do not copy a task transcript, delegate report or commit diary into the description. + +## Present evidence + +For visible changes, read [visual evidence](references/visual-evidence.md) and include matched before/after captures from the running app. Add a short recording when motion, timing or interaction is the change. Show only affected behavior. A new screen can have an after-only capture; say it is new. Missing evidence stays a stated gap. + +Report checks actually run and their results, with commands or durable run links. State material limits next to the claims they qualify: fixtures, simulator-only behavior, a failed or skipped check, or an untested service. Keep reproduction steps separate from completed verification. Green CI does not establish visual quality or product acceptance. + +## Use the repository template + +The repository's template, `.github/pull_request_template.md`, starts with What changed, Why, UI changes and four checklist items. Keep existing repository requirements. Add verification and risk where useful; remove empty optional sections and avoid repeating the same explanation under two headings. Replace inapplicable checklist items with a plain `N/A` note; check only completed work. Large cohesive changes are valid; do not call them small to satisfy a checkbox. + +Write the title around the final change, using the repository's title convention. In stacked work, name the immediate base and necessary dependencies, and describe this PR's own contribution. When scope changes, update the title, body and affected evidence together. Keep still-valid captures with their revision labels instead of implying they show a newer build. + +Use a body file or structured tool argument when publishing. Reopen the saved PR to check its text, links, comparison layout and video playback. A local draft or successful upload alone does not establish that the published body renders correctly. Creating the PR, pushing and handling reviews are separate steps, each within the user's authorization. diff --git a/.agents/skills/pr-writing/references/visual-evidence.md b/.agents/skills/pr-writing/references/visual-evidence.md new file mode 100644 index 000000000..bbb29f8a8 --- /dev/null +++ b/.agents/skills/pr-writing/references/visual-evidence.md @@ -0,0 +1,27 @@ +# Visual evidence + +Choose the changed behavior before collecting media. Capture the actual route or component through the host's supported browser, simulator or desktop tools. Do not redraw an interface to stand in for a screenshot. A design prototype is evidence of that prototype, not of the native or deployed product. + +## Capture a fair comparison + +Record the before and after revisions. Use the same viewport or device, route, data and relevant state. For a bug, reproduce the triggering condition on both versions. For a new feature, show the new flow and identify it as new. If the old build cannot run, label the missing or historical reference rather than fabricating a baseline. + +Use isolated worktrees or existing builds so capturing the base does not disturb the active writer. Use approved test data. Identify fixture-driven behavior where it limits the claim; a visible success state does not prove a live backend write. Keep the capture procedure in the repository when it is reusable, including the route, fixture and build commands that actually worked. + +Place Before and After adjacent at readable, matching sizes. Label them and caption the behavioral difference. Crop to the relevant area consistently without hiding context needed to judge the change. Avoid tall comparison boards with large blank gaps or a gallery of unchanged screens. Preserve original captures when preparing a comparison. + +Record motion at normal speed with the trigger, transition and settled result visible. Trim idle time without hiding failures or changing the timing being evaluated. Use H.264 MP4 when compatible with the host. Watch the encoded recording for dropped frames, legible text and correct orientation before uploading it. + +## Publish and verify + +Use GitHub repository attachments for inline evidence, especially in private repositories. Keep sensitive captures off anonymous public image hosts. A private raw-file URL or a local filesystem path is not a reliable inline attachment. + +Prefer the installed GitHub CLI's documented attachment support when available; check command help before relying on `--attach`. Otherwise use the authenticated GitHub attachment UI or a repository's verified uploader. Do not copy an app-specific authentication workaround into another project. For a local before/after composition tool, verify its output mode before running it so capture files are not uploaded elsewhere. + +With CLI attachment support, reference the local files in the Markdown body and pass each file with `--attach` alongside `--body-file`. The CLI replaces those references with uploaded URLs. If an upload partially fails, inspect the returned PR and its body before retrying: creation can succeed even when the command exits nonzero. Repair the existing PR rather than creating another. + +Use a compact Markdown table for a pair. Give images descriptive alt text. Put an interaction video in its own paragraph so it can render as a player. Preserve the original attachment URLs when editing an existing body. + +Reopen the PR in its actual rendered view. Check that both images load at useful sizes, labels match their revisions, and the video plays. If publication or viewing is unavailable, return the draft and evidence paths with that specific remaining step. + +GitHub documents [attachment access and supported formats](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/attaching-files) and [CLI attachment support](https://docs.github.com/en/github-cli/github-cli/attaching-files-with-github-cli). diff --git a/.agents/skills/pr-writing/references/writing-examples.md b/.agents/skills/pr-writing/references/writing-examples.md new file mode 100644 index 000000000..dde8a413d --- /dev/null +++ b/.agents/skills/pr-writing/references/writing-examples.md @@ -0,0 +1,31 @@ +# Writing examples + +These invented examples illustrate the level of explanation. They are not facts to reuse in a PR. + +## Explain the failure and mechanism + +Weak: "Improved session handling. The stale-response guard is robust and all tests pass." + +Useful: "Switching accounts while the inbox loaded could show the previous account's messages. The response now carries the account that requested it, and the inbox discards it if that account is no longer active. The regression test starts both loads and completes the older request last." + +The useful version names the trigger, consequence and mechanism. A test count alone would not explain why the race is covered. + +## Make a small change small to read + +Title: "fix(export): preserve zero values in CSV downloads" + +"CSV downloads left zero-valued cells empty because the formatter treated every falsy value as missing. It now leaves a cell empty only for null or undefined. `npm run test:csv` passes, including zero, null and undefined cases." + +No architecture narrative or screenshot is needed for this change. Follow required repository sections without padding them. + +## Describe what a comparison establishes + +"The payment error used to replace the form and discard the entered address. It now appears above the submit button, leaving the address available for a retry." + +Place the matched images immediately below that explanation. A caption such as "Same declined-payment fixture at 390 × 844; before a12bc34, after d56ef78" establishes the comparison. If no live charge ran, say so once beside the verification. Do not call the images proof that payments work end to end. + +## Preserve a useful limit + +"The migration was exercised on a copy of the staging schema. Production data volume and lock duration have not been measured." + +This gives the reviewer a concrete remaining concern. "Low risk, thoroughly tested" hides it. A long chronology of failed approaches would bury it. diff --git a/.agents/skills/prompt-writing/SKILL.md b/.agents/skills/prompt-writing/SKILL.md new file mode 100644 index 000000000..c93782df3 --- /dev/null +++ b/.agents/skills/prompt-writing/SKILL.md @@ -0,0 +1,47 @@ +--- +name: prompt-writing +description: Writes and reviews instructions for AI agents, such as AGENTS.md lines, SKILL.md files, and prompts that hand work to another agent or subagent. Use when writing or editing AGENTS.md, CLAUDE.md or a skill, or when briefing another agent. +--- + +# Prompt writing + +The reader is a capable model that can read the repository. Write only what it can't work out from the code, and say each thing once. + +## Give the outcome, the reason and the finish + +- State the outcome, the scope and what done looks like. A clear done condition keeps the agent from stopping early. +- Give the reason behind each constraint. A model that knows why a rule exists applies it to cases the rule didn't name. +- Point exactly: the path, the symbol, the command, the README section. Whether the agent follows a pointer depends on its wording. +- Say what to do rather than what to avoid. +- Hand over what the agent can't discover: decisions already made and why, what was tried and failed, who owns which files, and what the user prefers. + +## Keep instructions calm and consistent + +- Write in a normal voice. Current Claude models follow instructions closely, and capitals, "CRITICAL" or "if in doubt, always" make them overapply a rule. +- Put each instruction in one place. Conflicting guidance across a skill, AGENTS.md and the prompt can make a model stop and block early. Where instructions can conflict, state the priority: the user's instructions win over a skill. +- Write the prompt in the form you want back. A prompt in prose tends to get prose, and one built from headings and bullets tends to get markdown. +- Call each thing by one name throughout. + +## Keep context small + +- Every line of AGENTS.md or CLAUDE.md loads into every session. Put material needed in one task out of ten behind a pointer to the file that holds it. +- Link every reference file directly from the entry file. A file reached through another reference may be read only in part. +- Leave out facts that expire, or put them where the current value lives and point there. + +## Prune by behavior + +- A line is a no-op if the agent acts the same without it. Delete the whole sentence, not a few of its words. +- Don't ask an agent to "shorten" or "streamline" a prompt. It optimizes for length and cuts instructions that mattered. +- Test by running it. Give a fresh agent a real task with only this prompt, watch where it goes wrong, and fix that failure rather than one you imagine. + +## Writing a skill + +- Write the description in the third person. Say what the skill does and when to use it, with the words a user would say. The description is all that loads before the skill is chosen. +- Keep SKILL.md under 500 lines. Move detail that only one branch of the task needs into a file SKILL.md links directly. + +## Sources + +- Anthropic, [Prompting best practices](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices): give context and motivation, say what to do instead of what not to do, match prompt style to output, and dial back aggressive language on current models. +- Anthropic, [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices): assume the model is capable, third-person descriptions, references one level deep, no time-sensitive facts, test with real tasks. +- OpenAI, [Prompting the latest model](https://developers.openai.com/api/docs/guides/latest-model): conflicting skill guidance can block work early, and the user's instructions should outrank a skill. +- Matt Pocock, [Writing skills for agents](https://aihero.dev/skills-writing-for-agents): pointer wording, done conditions, the no-op test, and testing by running the document. diff --git a/.agents/skills/technical-writing/SKILL.md b/.agents/skills/technical-writing/SKILL.md new file mode 100644 index 000000000..ae1208d4b --- /dev/null +++ b/.agents/skills/technical-writing/SKILL.md @@ -0,0 +1,129 @@ +--- +name: technical-writing +description: "Layered technical-writing standard: Diátaxis structure, Google developer style sentences, STE instruction rules, Global English syntax. Use for /technical-writing or when writing or reviewing docs, RFCs, readmes, PR descriptions, or commit messages." +--- + +# Technical writing + +The goal is writing a tired engineer understands on the first read. Four layers get you there, one question each: what kind of document is this, how do sentences address the reader, how much does each sentence carry, and can any sentence be read two ways. Apply all four. + +Three rules sit above the layers: + +- **Cut every word that does no work.** If the sentence survives without a word, the word goes. "In order to" is "to". "It is important to note that" is nothing. +- **Use the short, everyday word.** "Use", not "utilize". "Help", not "facilitate". "Do", not "perform". A long word has to buy its length with precision. +- **When a rule makes a sentence worse, fix the sentence another way or leave it alone.** The rules serve the reader. A sentence that follows every rule and sounds like a machine wrote it has failed. + +The codebase is the word list. Write the real symbol, file, flag, or command name, not a synonym or a description of it. + +Don't invent jargon. Use the words a developer would say out loud: "move", "delete", "a budget that only decreases", not "evacuate", "ratchet", or "endgame". A named pattern is fine when the doc says what it means the first time. Propose new offenders and their replacements for `unslop`'s abstract-metaphor rule in a pull request. + +## Vary the rhythm + +The layers decide what a document says and how much each sentence carries. A doc can obey all of them and still read machine-written: every sentence clipped short, no view anywhere, nothing specific. + +- Mix sentence lengths on purpose. Short sentences land a point. Longer ones that take their time carry a fact with its condition or consequence. +- One thought per sentence does not mean one length per sentence. Split the sentence that carries two thoughts. Keep the long sentence that carries one. +- Have a view where the mode allows it. Explanation weighs trade-offs, so say what you make of them instead of listing pros and cons. Reference stays dry. +- Be specific over sterile. Not "schema changes can cause issues" but "a column rename fails the build". + +## Pick the mode first (Diátaxis) + +One document, one mode. Two questions pick it: does the content inform action (doing) or understanding (thinking), and does it serve learning or work? + +- Action + learning: **tutorial**. +- Action + work: **how-to**. +- Understanding + work: **reference**. +- Understanding + learning: **explanation**. + +Use the compass on a whole document or on one sentence. Reach for it whenever you feel unsure what you are writing. Gut feel is often wrong here. + +**Tutorial: learning by doing.** You are the teacher. The learner's success is your job, not theirs. Open by saying what the learner will build, not what they will "learn". Every step produces a visible result, early and often. Tell them what they should see: the expected output, the prompt change, the log line. Cut explanation to one clause and a link. Teaching pauses break the lesson. Stay concrete. Write as "we", in commands: "First, do x. Now, do y." + +**How-to: steps to a goal.** Solve a problem a person has, not an operation the machine can perform. Assume competence. Skip teaching. Action only: no digressions, no background, no completeness for its own sake. Link those instead. Allow forks and judgment: "If you want x, do y." Name the guide by the task: "How to calibrate the radar array", not "Radar array calibration". + +**Reference: facts for lookup.** Describe. Only describe. No instruction, no persuasion, no opinion. Be dry, complete, and sure: state facts, options, limits, and errors with no hedging. Mirror the structure of the thing described, so code and docs can be navigated together. Put material where readers expect it. Generate from code where possible, so it stays true. + +**Explanation: understanding and why.** One bounded topic, readable away from the product. Each title should tolerate an implicit "About..." in front. Anchor on a real why question. Give context: design decisions, history, constraints, alternatives. Opinion is allowed here and nowhere else. + +Don't mix modes: no reference tables inside a tutorial, no tutorial hand-holding inside reference, no arguing inside a how-to. Split and link instead. + +Source: diataxis.fr, fetched 2026-07-18. + +## Write sentences to the reader (Google developer style) + +- Talk to the reader as "you", in the present tense. "Will" only for things that genuinely happen later. +- Say who does what: "the compiler checks", not "is checked". Passive is fine only when the actor is unknown or beside the point. +- Write instructions as commands: "Click Submit." State facts plainly. Never "should be done". +- Put the condition before the instruction: "To delete the document, click Delete." The reader skips what does not apply. +- Put the common case first. Exceptions after. +- Sound like a knowledgeable friend. No buzzwords, no figurative language, no "please" in instructions, and never "simply", "easy", or "quickly" in a procedure. If it were simple, the reader would not be here. +- Don't pre-announce ("we will soon support...") and don't start consecutive sentences with the same phrase. +- Read the awkward sentence aloud. If it stays awkward, rewrite it. +- Link with words that say where the link goes: the page title or a short description. Never "click here". Prefer a sentence of context on the page over a link off it. +- Headings carry the point, not just the topic ("Pick the mode first", not "Modes"). Sentence case. A task heading is a bare verb phrase ("Create an instance"). A concept heading is a noun phrase. One h1 per page, no skipped levels. +- Numbered lists for sequences, bullets for everything else. Introduce a list with a complete sentence. Keep items parallel. +- Code goes in code font. UI elements go in bold. Use serial commas. Drop "etc." and say up front that a list is partial. + +Source: developers.google.com/style, fetched 2026-07-18. + +## Make statements load one at a time (STE rules) + +- One instruction per sentence. One thought per sentence everywhere else. +- Split instructions longer than about 20 words and other sentences longer than about 25. +- Put the warning or condition before the step it guards: "If hot oil touches your skin, injuries can occur." +- Keep "the" and "a": "Remove backup file" reads two ways. "Remove the backup file" reads one. +- Give each word one meaning and one job, then keep it. If "check" means inspect, don't also use it for restrain. +- Pick one word per action and stick to it: "start", not "start" here and "initiate" there. +- Write procedures as direct commands, never as narration and never in the passive: "Install the component", not "the component must be installed". +- Avoid "-ing" words where you can. They take too many grammatical jobs and breed misreadings. + +Source: asd-ste100.org (Issue 9, 2025), fetched 2026-07-18. The numbered rules and dictionary live in the spec PDF. The principles above are the transferable core. + +## Leave no sentence open to two readings (Global English) + +- Keep words like "only" and "not" next to the word they change: "only fails on growth" and "fails only on growth" say different things. +- Break up long noun strings: "the proto import budget check script" becomes "the script that checks the proto-import budget". +- Make every "it", "they", and "this" point at one obvious thing. Repeat the noun when in doubt. Never use "this" or "which" to point at a whole clause. +- Don't drop verbs: "Phase 1 moves the converters and Phase 2 the runtime" leaves Phase 2 without one. Give it one. +- Keep the small words that show structure. "Ensure that the switch is off" keeps "that" because it makes the sentence parse one way. Never trade clarity for word count. +- Repeat the article in a series when it prevents a misread: "the client and the host", not "the client and host", when they are two things. +- Say which parts "and" or "or" joins when a sentence can group two ways. "Both...and", "either...or", and "if...then" are free disambiguators. +- Use periods, not semicolons. Replace an em dash with a new sentence. +- Make text in parentheses a full grammatical unit or its own sentence. Never form plurals with "(s)". +- No slashes: write "a, b, or both" instead of "a/b" or "and/or". +- Call each thing by one name, everywhere. A doc that says "the gate", "the ratchet", and "the budget check" for one thing teaches three things. Rewording an unchanged sentence between edits costs the same way: don't churn what didn't change. +- Skip idioms, colloquialisms, Latin abbreviations, and metaphors. A non-native reader, a translator, and an agent all parse plain constructions best. + +Source: Kohl, The Global English Style Guide (SAS Press). Guideline text fetched from the Internet Archive and the SAS sample chapter, 2026-07-18. + +## Voice and repo specifics + +- Apply the **unslop** skill to every doc this skill touches. That skill owns the slop-pattern catalog: AI vocabulary, filler, hedging, formatting tells. +- PR descriptions and commit messages are writing too. Every layer except Diátaxis applies to them. +- Product UI strings are not documentation. Use your product's copy guidelines for those. +- Write real paths and real symbols. Make every count or tree claim true at the commit that lands it, and include the command that regenerates it. + +## Worked example + +Before: + +> Configuration of the proto import ratchet budget script parameters is performed via budget.json. Note that it's important to remember that running with --write, which updates the committed budget to reflect the current count, should only be done when lowering it. If exceeded, CI fails. + +After: + +> `budget.mjs` reads the committed budget from `budget.json` and counts the files that import protos. If the count exceeds the budget, CI fails. Run `budget.mjs --write` only to lower the budget. + +The fixes, by layer: "configuration is performed" becomes "`budget.mjs` reads", so someone does something (Google). "Ratchet" goes away. The script's real filename does the naming (jargon rule). The five-noun string breaks up into plain clauses (Global English). The hedge "note that it's important to remember" is deleted (cut every word that does no work). The failure condition moves ahead of the step it explains (STE). The buried "should only be done when lowering" becomes a command with "only" next to its verb (STE). "If exceeded" gets a subject: the count (Global English). + +## Review checklist + +Apply to any prose this skill covers. Item 1 applies only to document sets: + +1. Is each file one Diátaxis mode, with links where modes meet? +2. Is every instruction written as a command, with its condition in front? +3. Does any sentence carry two instructions or two thoughts? Split it. +4. Can any word be cut without losing meaning? Cut it. +5. Is "only" next to the word it changes? Does every "it" point at one thing? Does every clause keep its verb? +6. Does each thing have exactly one name across the docs? +7. Would a developer say these words out loud? Replace invented metaphors and fancy synonyms with the plain word or the real symbol name. +8. Are all symbols, paths, and counts real at this commit, with the commands that regenerate the counts? diff --git a/.agents/skills/unslop/SKILL.md b/.agents/skills/unslop/SKILL.md new file mode 100644 index 000000000..dc2fba687 --- /dev/null +++ b/.agents/skills/unslop/SKILL.md @@ -0,0 +1,105 @@ +--- +name: unslop +description: Removes AI writing tells and adds a human voice. Use when writing or editing any prose, including docs, READMEs, PR descriptions, commit messages, code comments and reports. +--- + +# Unslop + +Edit text to remove AI patterns and add human voice. + +## Judge by density + +Almost nothing on this list is wrong once. One em dash is punctuation. Five in a paragraph is a fingerprint. One or two instances may be coincidental. A passage introducing many of them is among the strongest signals available. + +Where to start looking: + +- Three or more vocabulary-list words inside 100 words. Rewrite the passage, not the words. Swapping synonyms leaves the sentence shape that produced them. +- Two forced triplets in a row, or two "not just X, but Y" constructions in a row. +- Em dashes past about one a paragraph, where the dash is doing structural work a sentence break should do. + +These numbers are judgment, not measurement. No calibration study sits behind them. Adjust once you have read enough of the document to know its own baseline. Em dashes are the deliberate exception. This project sets them at zero, because this tell is cheap to remove and expensive to adjudicate one at a time. + +## Process + +1. Scan for the patterns below. +2. Rewrite. Preserve meaning, match intended tone. +3. Add soul (see next section). +4. Self-audit: "What makes this obviously AI generated?" Fix remaining tells. + +## Adding soul + +Removing patterns is half the job. Sterile, voiceless writing is just as obvious. + +- **Have opinions.** React to facts instead of neutrally listing pros and cons. +- **Vary rhythm.** Short sentences. Then longer ones that take their time. Mix it up. +- **Acknowledge complexity.** "Impressive but also kind of unsettling" beats "impressive." +- **Use "I" when it fits.** First person isn't unprofessional. +- **Let some mess in.** Perfect structure looks machine-made. +- **Be specific.** Not "this is concerning" but "there's something unsettling about agents churning away at 3am." + +## Patterns to detect and fix + +### Content + +1. **Puffery.** "pivotal moment", "testament to", "evolving landscape", "setting the stage for", "indelible mark", "deeply rooted". Cut puffery, state what happened. +2. **Name-dropping.** Listing media outlets without context. Pick one, say what was said. +3. **Superficial -ing phrases.** "highlighting...", "ensuring...", "reflecting...", "showcasing...", "fostering...". Delete or expand with real sources. +4. **Promotional language.** "nestled", "vibrant", "breathtaking", "groundbreaking", "renowned", "stunning", "must-visit". Use neutral descriptions. +5. **Vague attributions.** "Experts believe", "Industry reports suggest", "Some critics argue". Name the source or delete. +6. **Formulaic challenges.** "Despite challenges... continues to thrive." Replace with specific facts. + +### Language + +7. **AI vocabulary.** Additionally, crucial, delve, enduring, enhance, fostering, garner, interplay, intricate, landscape (abstract), pivotal, showcase, tapestry (abstract), testament, underscore, vibrant. Replace with plain words. +8. **Fancy ways to say "is".** "serves as", "stands as", "boasts", "features". Just say "is" or "has". +9. **"Not just X, but Y."** State the point directly instead. +10. **Rule of three.** Forcing ideas into groups of three. Use the natural number. +11. **Synonym cycling.** Protagonist, main character, central figure, hero all in one paragraph. Pick one, repeat it. +12. **False ranges.** "from X to Y" where X and Y aren't on a meaningful scale. List topics directly. + +### Style + +13. **Em dash overuse.** Avoid em dashes entirely. Use periods or commas only (no parentheses, no en dashes, no hyphen-as-dash substitutes). Em dashes are an AI tell, and reaching for parentheses instead just trades one tell for another. If a thought needs separation, end the sentence or use a comma. +14. **Colon overuse.** Colons are fine before a list or example. Not as mid-sentence connectors. "If you're coming from traditional automation: instead of registering event handlers, you describe conditions" adds nothing with the colon. Rewrite to let the point stand on its own without comparison framing. "Describing when the scheduler should fire works best as plain English." Same meaning, no crutch punctuation. +15. **Boldface overuse.** Don't bold every proper noun or acronym. +16. **Inline-header lists.** The tell is a bold label and colon that restates the line: "**Performance:** Performance improved...". Convert those to prose. A bold lead-in that ends in a period, names the item, and is followed by genuinely new detail ("**Schema in TypeScript.** Tables live in one file.") is fine, not a tell. +17. **Title case headings.** Use sentence case. +18. **Decorative emojis.** Remove from headings and bullets. +19. **Curly quotes.** Replace with straight quotes. + +### Communication artifacts + +20. **Chatbot phrases.** "I hope this helps!", "Let me know if...", "Of course!", "Certainly!", "Found the smoking gun!" Remove. +21. **Cutoff disclaimers.** "While specific details are limited..." Find sources or remove. +22. **Sycophantic tone.** "Great question! You're absolutely right!" Respond directly. + +### Filler + +23. **Filler phrases.** "In order to" becomes "To". "Due to the fact that" becomes "Because". "It is important to note that" gets deleted. +24. **Excessive hedging.** "could potentially possibly be argued that it might" becomes "may". +25. **Generic conclusions.** "The future looks bright." State specific plans or facts. + +### Jargon + +26. **Abstract metaphor nouns.** Substrate, wedge, vector, locus, vantage, nexus, primitive (as noun), harness (as metaphor), surface (as in "API surface"), bedrock, scaffolding (as metaphor), modality, paradigm, gold-plating, ratchet (as metaphor), evacuate (for moving code), endgame, north star, flywheel. These read as technical but usually have a plainer concrete word. "Substrate" becomes "base". "Wedge in" becomes "add". "Vector" becomes "way" or "method". "Gold-plating" becomes "more than the job needs". "Ratchet" becomes the mechanism's real name or "a limit that only tightens". "Evacuate" becomes "move out". "Endgame" becomes "the last phase". Pick the concrete word. + +### Plain speech + +27. **Say what it does, not how it feels.** "the database stays close at hand", "SQL you can read", "types that follow your schema" name a feeling. The fix names the mechanism or a number: "`.toSQL()` returns the exact string sent to the database", "a column rename fails the build". Ask what the sentence tells the reader to do or know, then write that. If you can't restate it as a concrete instruction, fact, or number, cut it. One more check: if the sentence could appear unchanged in another project's docs, it says nothing about this one. Cut it. +28. **Shorten or split dense sentences.** If the reader has to backtrack to parse a sentence, break it in two or drop clauses. One idea per sentence. +29. **Active voice.** Prefer it. Catch "is/are/was/were + past participle" and name the actor: "queries are validated" becomes "the compiler validates queries", "the file is parsed by the loader" becomes "the loader parses the file". Passive is fine only when the actor is unknown or genuinely doesn't matter. +30. **Cut adverbs, or use a stronger verb.** "runs quickly" becomes "is fast" or the number. "significantly improves" becomes the measured delta. An adverb propping up a weak verb means the verb is wrong. +31. **Prefer the plain word.** "utilize" becomes "use", "leverage" becomes "use", "facilitate" becomes "help", "numerous" becomes "many", "in the event that" becomes "if". The fancier synonym is rarely clearer. + +## Not evidence + +Do not flag these. Acting on them produces false positives and, worse, a writer who starts avoiding ordinary English. + +- One instance of anything, except a literal machine artifact such as a leftover placeholder or a truncation notice. +- Correct, well-placed punctuation, including a dash a writer clearly chose. +- Any vocabulary-list word in its literal sense. A landscape can be a landscape. +- Clean structure. Organization is not a tell. Evenly weighted structure matching no real shape in the material is. +- Formal register. Some documents are formal. +- Anything written before late 2022. + +Human writing has drifted toward LLM style since 2024, and writers now avoid these patterns on purpose. In controlled studies people identify AI text at close to chance. That is the reason for the density rule. The goal is prose that carries information and sounds like a person wrote it, not prose that beats a detector. Cosmetic cleanup that leaves a vague, unsourced document vague and unsourced has made things worse, because it hid the real problem. diff --git a/.claude/skills/deslop b/.claude/skills/deslop new file mode 120000 index 000000000..efd867d48 --- /dev/null +++ b/.claude/skills/deslop @@ -0,0 +1 @@ +../../.agents/skills/deslop \ No newline at end of file diff --git a/.claude/skills/efficient-implementation-plans b/.claude/skills/efficient-implementation-plans new file mode 120000 index 000000000..fa48659be --- /dev/null +++ b/.claude/skills/efficient-implementation-plans @@ -0,0 +1 @@ +../../.agents/skills/efficient-implementation-plans \ No newline at end of file diff --git a/.claude/skills/pr-writing b/.claude/skills/pr-writing new file mode 120000 index 000000000..8db4f3adb --- /dev/null +++ b/.claude/skills/pr-writing @@ -0,0 +1 @@ +../../.agents/skills/pr-writing \ No newline at end of file diff --git a/.claude/skills/prompt-writing b/.claude/skills/prompt-writing new file mode 120000 index 000000000..05fc91ddb --- /dev/null +++ b/.claude/skills/prompt-writing @@ -0,0 +1 @@ +../../.agents/skills/prompt-writing \ No newline at end of file diff --git a/.claude/skills/technical-writing b/.claude/skills/technical-writing new file mode 120000 index 000000000..bedd96f61 --- /dev/null +++ b/.claude/skills/technical-writing @@ -0,0 +1 @@ +../../.agents/skills/technical-writing \ No newline at end of file diff --git a/.claude/skills/unslop b/.claude/skills/unslop new file mode 120000 index 000000000..158c16b3e --- /dev/null +++ b/.claude/skills/unslop @@ -0,0 +1 @@ +../../.agents/skills/unslop \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index a4569113b..df6a3fc08 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,6 +37,18 @@ A component's README wins over the docs. Files marked with a pull request exist - **Rotate the intrinsics whenever you rotate a camera image.** Camera images are landscape sensor images, and the intrinsics match that orientation. A rotated image with unrotated intrinsics produces wrong 3D geometry without any error. - **Use LiDAR when present, and never require it.** Most homeowners' phones lack LiDAR, so every check must also work from the camera and the phone's poses. +## Writing, planning and review + +The repository ships shared skills in `.agents/skills/`, which Codex reads directly. `.claude/skills/` links to the same folders for Claude Code. Agents without skill support can read each `SKILL.md` as a plain guide. + +- `unslop` for any prose, and `technical-writing` for docs and READMEs. +- `pr-writing` for pull request titles and descriptions. +- `prompt-writing` for AGENTS.md, skills and prompts that brief another agent. +- `efficient-implementation-plans` for implementation plans. +- `deslop` for reviewing or cleaning a diff. + +A user's instruction outranks a skill. + ## Working in the repository - `make check` runs every suite. `make ios`, `make server` and `make web` run one each. diff --git a/README.md b/README.md index d7938a87c..25393f8dc 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ Paths marked with a pull request exist only on that branch until it merges. | `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, meter reading in PR #16 | | `web/` | Browser toolchain for a reviewer view | The review page (PR #15) is parked | | `docs/` | The overview, the walkthrough, public rules and code citations, and the live-survey design | | +| `.agents/skills/` | Shared agent skills for writing, planning and review, linked from `.claude/skills/` | | | `sites/landing` | The landing page, a submodule | Change it in its own repository | ## What's live From e1cdc64daceb4902de2773273080f964611e9fca Mon Sep 17 00:00:00 2001 From: Sam Gu <127461594+SamGu-NRX@users.noreply.github.com> Date: Sat, 26 Sep 2026 14:43:40 -0500 Subject: [PATCH 10/16] Expand project description in README Added a brief overview of the project and its goals. --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 25393f8dc..d64ad1739 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # House scanning +We're exploring a simpler way to survey a home for a Base Power battery. + A homeowner walks the outside wall around their electric meter with an iPhone. The app guides the scan until it has seen everything the placement rules need, and sends what it captured. The server builds a 3D model, decides in plain code whether a Base Power battery fits and where, and the app shows the spot in AR. This is a four-person hackathon project for Base Power, started 2026-09-25. Success means one capture session gives Base enough to decide, with no follow-up photos. From 9c347e182627d7d47082d1c3ae7bc60b9d503d3a Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 14:44:48 -0500 Subject: [PATCH 11/16] docs: keep the latest morning briefing without its correction notes Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 1 + docs/briefings/2026-09-26-morning.html | 930 +++++++++++++++++++++++++ 2 files changed, 931 insertions(+) create mode 100644 docs/briefings/2026-09-26-morning.html diff --git a/README.md b/README.md index d64ad1739..153aea459 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,7 @@ Paths marked with a pull request exist only on that branch until it merges. | `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, meter reading in PR #16 | | `web/` | Browser toolchain for a reviewer view | The review page (PR #15) is parked | | `docs/` | The overview, the walkthrough, public rules and code citations, and the live-survey design | | +| `docs/briefings/` | The 26 September morning briefing, a dated snapshot of findings and decisions | | | `.agents/skills/` | Shared agent skills for writing, planning and review, linked from `.claude/skills/` | | | `sites/landing` | The landing page, a submodule | Change it in its own repository | diff --git a/docs/briefings/2026-09-26-morning.html b/docs/briefings/2026-09-26-morning.html new file mode 100644 index 000000000..a393d63a3 --- /dev/null +++ b/docs/briefings/2026-09-26-morning.html @@ -0,0 +1,930 @@ + + + + + + +House scanning briefing: what we are building, what works, what is blocked + + + + + + + +
        + +

        House scanning: what we are building, what works, and what is blocked

        +

        Team briefing, 26 September 2026, facts as of 09:10. Readers: Sam, Hunter, Shrey, Aiden. About ten minutes.

        +

        Update 11:15: the team split into client and server teams; see docs/how-it-works.html.

        + +

        A homeowner walks the outside wall around their electric meter with an iPhone. The phone measures the wall, a server applies the clearance rules in plain code, and the app shows where the battery goes, in AR, on the real wall.

        +

        Today it runs end to end in the iPhone Simulator on a recorded real walk and gets a real answer from the live server. It has never run on a physical iPhone at a real wall. That is the blocker, and the first job this morning.

        + + + + +

        1. The problem: a battery has to fit between the rules

        + +
        + + An outside wall with the battery and its clearance rules, drawn to scale + Front view of a house wall. From left: an AC unit on the ground, a gas meter, the electric meter, the battery just to its right, a window, a door. Red hatched zones show 3 feet of clearance around the gas meter, AC unit, window and door. A bracket beside the battery shows 6.5 feet of headroom. A blue bracket along the ground shows 20 feet of cable reach either side of the electric meter. A tape along the bottom counts feet from the meter. Below, a view from above shows the battery footprint, 31 by 22 inches, and the 3 feet of clear space needed in front of it. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 20 ft of cable, either side of the meter + + + + AC unit + gas meter + electric meter + window + door + 6.5 ft headroom + battery + + + 3 ft + 3 ft + 3 ft + 3 ft + + + + + + + + + + + 151050 + 510152025 + + feet left and right of the electric meter + + + The same wall from above + + + + + + + + + + + + + + + + + + + + + + + + battery footprint, 31 by 22 in + 3 ft clear in front + AC unit + gas + + +

        Wider than the screen: scroll sideways for the rest of the picture.

        +
        Everything is drawn at one scale: 5 ft between tape marks. The red hatched areas are places the battery may not go. The battery just fits between the electric meter and the window's zone. Rules shown: 3 ft from the gas meter, from AC units, fences and other batteries, and from doors and windows that enter the home; 3 ft of clear space in front; 6.5 ft of headroom; within 20 ft of cable run from the meter; a flat, seen stretch of wall behind it.
        +
        + +
        +

        Base Power puts a home battery on the outside wall, close to the electric meter, because a cable runs from the meter to the battery. The battery is 31 in wide, 22 in deep and 39.5 in tall. Whether it fits, and where, depends on the clearance rules in the picture. Today a person has to go to the house and check.

        +

        Our product does that check with the homeowner's iPhone, in a couple of minutes. The app measures the wall. A server applies the rules in plain code. The app shows where the battery goes, in AR, on the real wall.

        +

        The demo's rules come from public sources and live in server/rules.yaml, each with its citation. Two numbers there are placeholders because no public value exists (pool 10 ft, driveway 5 ft). Base's own thresholds are private, load at runtime from a file that is not in git, and do not appear on this page.

        +
        +
        + The rules the demo uses, with sources +
        + + + + + + + + + + + +
        RuleValueSource
        Distance from the gas meter3 ftBase's public help page; Austin Energy; Texas Gas Service
        Distance from AC units, fences, other batteries3 ftBase's help page
        Distance from doors and windows that enter the home3 ftIRC R328.4
        Clear space in front of the battery3 ftBase's help page
        Cable run from the meterwithin 20 ftBase's help page
        Headroom6.5 ftNEC 110.26, a demo choice
        Wall behind the batteryflat, and seen by the camerademo requirement
        Pool, driveway10 ft, 5 ftplaceholders; no public value exists
        +
        + + +

        2. How a scan becomes a decision

        +

        Six pictures. Each one is the last one with one thing added. The tape along the bottom is real: the app shows the wall unrolled like a tape measure, with the electric meter at zero.

        + +
          +
        1. + + Step 1: tap the electric meter, which becomes zero + The wall from the problem picture, smaller, with the electric meter in the middle. A blue ring marks a tap on the meter. A dashed line drops from the meter to the zero mark on a tape along the bottom. + + + + + tap the electric meter + it becomes 0 ft; all else is left or right of it + +
          Point at the electric meter and tap. The meter becomes zero. Every position from now on is measured left or right of it, in feet.
          +
          2. + +
        2. + + Step 2: the phone photographs the meter close up, then walks left, then right + Same wall. A dashed frame around the electric meter marks the close-up. A phone stands on the ground below the meter. Blue arrows run left along the wall and then right. + + + close-up, reads the number + + + + walk left + then right + +
          The phone takes a close-up of the meter by itself when the picture is sharp, to read the meter's number. Then you walk along the wall, left and then right. The phone keeps photos automatically.
          +
          3. + +
        3. + + Step 3: a haze covers wall and ground the phone has not seen yet + Same wall. The phone has walked left to about 8 feet and right to about 9 feet. Those parts of the wall and ground are clear. The rest is covered by a grey hatched haze. + + + + + + + + haze: + not seen yet + haze + seen: the haze lifts + +
          A haze covers wall and ground the phone has not seen and lifts as it sees them. The team calls this fog of war. The tape strip grows as you walk.
          +
          4. + +
        4. + + Step 4: mark things as you pass, and answer one request for another view + Same wall, almost all seen. Orange pins mark the gas meter, the window and the AC unit. A small hatched patch of ground about 6 feet right of the meter is still unseen. A message at the top reads: Show the ground about 6 ft right of your meter. + + + + + + gas + window + AC + + + Show the ground about 6 ft right of your meter + +
          Mark things as you pass: gas meter, AC unit, door, window, driveway edge, fence. If something a rule needs is still unseen, the app asks for one more view, in one sentence.
          +
          5. + +
        5. + + Step 5: a small measurement file goes to the server, which tries every spot + Same wall, fully seen. A small file icon labelled scene.json, 17 KB, rises from the phone toward a server box at the top. On the wall, dashed battery outlines slide along in small steps. + + + + + + tries every spot, 2 in apart + + + scene.json, 17 KB + photos stay on the phone + + + server: checks every rule + +
          The phone sends a small file of measurements, about 17 KB for a real walk. The server slides the battery's footprint along the wall in 2-inch steps and checks every rule at every position.
          +
          6. + +
        6. + + Step 6: the answer comes back and the battery is shown on the wall in AR + Same wall. The server box at the top sends an arrow back down to the phone. The battery is drawn on the wall just right of the electric meter with an orange outline, and a cable runs from the meter to it. A label reads: best spot, shortest cable. + + + server answers + + + + + + best spot, + shortest cable + shown in 3D and in AR, placed from the meter + +
          The server answers with the best spot (shortest cable run first), the cable route, and every check's outcome and reason. The phone draws the battery on the wall, placed relative to the meter, in 3D and in AR.
          +
          7. +
        + +
        + + The whole loop: phone to server and back + Two boxes. Left: the phone, which taps, walks, marks and shows. Right: the server, with the rules file and plain code. An arrow from phone to server carries scene.json. An arrow from server to phone carries the spot, the cable route and every check's outcome. + + Phone + tap, walk, mark + show the spot in AR + + Server + rules in a file + plain code, no AI + + scene.json + + spot, cable, checks + +
        The recap. AI is used only to recognise things: it reads the meter's text and finds objects. The placement decision is ordinary code with the rules in a file, so every answer traces back to a rule and a measurement.
        +
        + + +

        3. Three ideas that make the answer honest

        + +

        The phone counts its own steps, so error grows with distance

        +
        + + Error bars widen the farther you walk from the meter + A tape along the top counts feet from the meter at 0, 5, 10, 15 and 20. Below it, four positions carry horizontal error bars that get wider with distance: plus or minus 0.3 feet at the meter, 1.1 feet at 5 feet, 1.9 feet at 10 feet and 3.5 feet at 20 feet. Footprints lead away from the meter. + + + + + + + 05101520 ft + + + meter + + + + + + + + + + + + + + + ± 0.3 ft at the meter + ± 1.1 ft at 5 ft + ± 1.9 ft at 10 ft + ± 3.5 ft at 20 ft + + error bar = 0.3 ft + 0.16 ft for every foot from the meter + +
        AR tracking is dead reckoning: the phone estimates how far it moved from the camera and motion sensors, like counting your own steps from a doorway. Small errors add up, so the farther from the meter, the less sure a position is. The server widens every AR measurement's error bar by 0.3 ft plus 0.16 ft per foot along the wall. Where the analogy breaks: a miscounted step is only a stretch along the wall, while the phone's error also drifts sideways.
        +
        + +

        The error bar decides: PASS, FAIL or UNSURE

        +
        + + Three measurements against the 3 foot gas meter rule + A number line from 0 to 6 feet with a vertical rule at 3 feet. Row one: 4.5 feet plus or minus 0.8, the whole bar sits right of 3, PASS. Row two: 2.0 feet plus or minus 0.8, the whole bar sits left of 3, FAIL. Row three: 3.4 feet plus or minus 0.8, the bar straddles 3, UNSURE. + + + + + 0123456 ft + + + 3 ft rule (gas meter) + too close + clear + + + + + + + 4.5 ft ± 0.8: clear by 1.5, more than 0.8 + + ✓ PASS + + + + + + + 2.0 ft ± 0.8: short by 1.0, more than 0.8 + + ✕ FAIL + + + + + + + 3.4 ft ± 0.8: clear by only 0.4 + + ? UNSURE + +
        Every measurement carries an error bar. A check passes only when the margin beats the error, fails only when it is past the rule by more than the error, and is UNSURE otherwise. Equal is UNSURE. UNSURE means a person checks it, or the app asks for a better view. The three examples are the dossier's, for the 3 ft gas meter rule with ± 0.8 ft.
        +
        + +

        Unseen is not clear

        +
        + + A check whose ground was never seen stays UNSURE and names the missing view + A candidate battery spot on the wall. The ground beside it is covered by grey haze. Beside it a check reads: ground near the spot, checks need 6 to 13 ft of seen ground, UNSURE, missing evidence: show the ground about 6 ft right of your meter. + + + + + + meter + + + candidate spot + + never seen + + Ground near the spot + checks need 6 to 13 ft of ground + + ? UNSURE + missing evidence: show the + ground 6 ft right of your meter + +
        If the camera never saw part of the area a rule depends on, that check cannot pass. It stays UNSURE, and the server names the view that would settle it. Here the ground beside the spot was never seen, and the checks that use ground need 6 to 13 ft of it. A blank on the map is never treated as empty ground.
        +
        + + +

        4. Where it stands this morning

        +
        + ✓ proven in the Simulator on a recorded real walk + ! fixed this morning + ? not yet proven on a real phone +
        +
          +
        1. Tap the meterIt becomes zero.
          ✓Simulator?Real phone
          2. +
        2. Read the meter's numberClose-up; three candidates to tap.
          ✓Simulator?Real phone
          3. +
        3. Walk the wallHaze lifts; tape strip grows.
          ✓Simulator!Now reports ground depth, clear space in front, and a tilt-up view for headroom?Real phone
          4. +
        4. Mark things, one more viewGas, AC, door, window, fence.
          ✓Simulator!Server now publishes exactly what settles each check?Real phone
          5. +
        5. Upload scene.jsonMeasurements only.
          !17 KB, not 16.5 MB of photos; answer in 0.5 s?Real phone
          6. +
        6. Server decidesLive on Vercel.
          ✓252 tests; independent checker 50 of 50
          7. +
        7. Show the spot in ARPlaced from the meter.
          ✓Simulator, 3D and AR?Real phone
          8. +
        +
        ✕
        Never run on a physical iPhone at a real wall.Every stage marked "?" above shares this one gap. Everything that decides accuracy is waiting on that data.
        +
        +
        Placeholder rulesWith the public placeholder rules, automatic approval is off, so every real result says "An installer will take a look."
        +
        Hidden wallThings standing in front of the wall can hide it, and nobody owns checking that yet (decision 7).
        +
        Parked: web review page (#15)It shows the server's answer in a browser. It helped check the server before the app could upload. It is not the product.
        +
        + +
        +
        Does what we built work, and is it part of the app we'd ship?
        In the Simulator on a recorded real walk, yes: meter to result, with a real answer from the live server. The app and the server are the product. The web review page is not, and it is parked.
        +
        Are the overnight tests doing anything?
        Each of the five changed the app or the server (section 6 lists what). Three other pieces of overnight work changed nothing: the panel-label search, finer statistics on the modern-phone data, and the research notes.
        +
        What is done that achieves the goal?
        The server, live with 252 tests. The whole app flow in the Simulator. Uploads fixed to 17 KB. The first-try fixes, merging now while the build is repaired.
        +
        What is the main blocker?
        A real-phone session. See the next section.
        +
        + + +

        5. The main blocker, and what to do this morning

        +
        + + The field test: the phone's reading against a tape on the same wall + A wall with the electric meter. A yellow tape runs along the wall with marks at 5, 10, 15 and 20 feet. A phone stands in front. Blue dots show where the phone thinks the same marks are; the gap between dot and mark grows with distance. A label reads: the gap is the phone's scale error. + + + + + + + + + + + 5101520 ft + + + + + + blue dots: where the phone says the tape marks are + the gap is the scale error + + a current iPhone, no LiDAR + +
        Nothing decides accuracy until this runs. Hunter's research handoff (PR #18) reaches the same conclusion.
        +
        +
        +

        Run the field test first, about an hour. You need a current non-LiDAR iPhone, the Measure Lab build, a 30 ft tape and a real meter wall. Follow experiments/evals/field/FIELD_SHEET.md: type the tape readings into the prepared templates, and one command turns them into a score and the phone's real scale error. A dry run recovered a planted 1.5% error exactly, so the scoring end is ready.

        +

        The overnight tracking numbers (finding 1) come from a 2023 iPhone with LiDAR and a dataset whose reference may itself be scaled to the phone's tracking. Most homeowners' phones lack LiDAR. The field test measures our own phone on our own wall, which is the only direct measurement.

        +
        + + +

        6. What the overnight tests found, and what each changed

        +

        All five ran on public real-world datasets with laser-measured ground truth. Dataset images are never in git. Each finding below ends with the line that answers "did this test do anything".

        + +
        +

        Finding 1. How far off is the phone's own tracking?

        +
        +
        + 2018 iPhone 6s (ADVIO) + 2023 iPhone 14 Pro Max (MARViN) + server's allowance +
        + + p90 position error in inches against feet walked, two phones and the server's allowance + At 3, 10, 20 and 30 feet walked: 2018 iPhone 6s 18.6, 55.7, 94.6, 132.6 inches. 2023 iPhone 14 Pro Max 4.0, 8.6, 13.4, 18.5 inches. Server's allowance 5.8, 19.2, 38.4, 57.6 inches. The 2023 phone stays under the allowance at every distance. The 2018 phone is over it at every distance. + + + + 040 in80 in120 in + 3 ft10 ft20 ft30 ftdistance walked + + + + + + + + + 2018 phone, 132.6 in + allowance, 57.6 in + 2023 phone, 18.5 in + + 3 ft walked: 2018 phone 18.6 in, 2023 phone 4.0 in, allowance 5.8 in + 10 ft walked: 2018 phone 55.7 in, 2023 phone 8.6 in, allowance 19.2 in + 20 ft walked: 2018 phone 94.6 in, 2023 phone 13.4 in, allowance 38.4 in + 30 ft walked: 2018 phone 132.6 in, 2023 phone 18.5 in, allowance 57.6 in + + +
        Position error after walking, p90 (9 in 10 cases are better). A current iPhone's own tracking stays inside the server's allowance at every distance; the 2018 phone is far over it. The allowance line is the 0.16 ft per foot part of the server's error bar. Caveats: the MARViN reference may itself be scaled to the phone's tracking, and that phone has LiDAR, which most homeowners' phones lack.
        +
        Data table +
        + + + + + + + +
        Walked2018 iPhone 6s (ADVIO), in2023 iPhone 14 Pro Max (MARViN), inServer's allowance, in
        3 ft18.64.05.8
        10 ft55.78.619.2
        20 ft94.613.438.4
        30 ft132.618.557.6
        +
        +
        +

        What it changed: the server's error allowance comes from this. The field test measures our own phone.

        +
        + +
        +

        Finding 2. Can photos alone give 3D measurements?

        +
        + + Error of learned depth from photos: flat walls and edges + A horizontal scale from 0 to 10 inches. Flat walls: 5.0 inches at p90, with a 95 percent interval from 3.8 to 6.8. Window and door edges: 8 inches or worse in every setting, drawn as a bar starting at 8 and running off the right edge. + + + + 0246810 inerror, p90 + flat wall + + + + 5.0 in (3.8 to 6.8) + edges + + + 8 in or worse, every setting + +
        Learned depth models (MoGe-2, Depth Anything 3), rescaled with the phone's poses. Flat walls come out at about 5 in at best at a realistic phone error. Window and door edges, which are what clearances are measured from, come out at 8 in or worse in every setting.
        +
        +

        What it changed: 3D modeling stays central. Learned depth alone is too rough at edges, so the points that decide a check come from metric depth (LiDAR or AR taps), and LiDAR and the phone's poses give the model its scale. World models are not evaluated yet.

        +
        + +
        +

        Finding 3. Does the app's coverage map claim wall it never saw?

        +
        +
        + wall a photo saw + claimed, never seen + good wall thrown away +
        + + A 19.3 foot wall under three versions of the coverage map + Three bars, each 19.3 feet long. First, as claimed: 1.1 feet marked as seen that no photo saw, hidden behind scaffold footings. Second, after requiring views from different angles: the false 1.1 feet is gone but 6 to 8.5 feet of good wall is also thrown away. Third, after a depth test on LiDAR phones: the false part is removed cleanly and the rest is kept. + + as claimed + + + 1.1 ft never seen + views from different angles + + + + 6 to 8.5 ft of good wall lost too + depth test (LiDAR phones) + + + clean + +
        Run on real photos of a laser-scanned building. The map claimed 1.1 ft of a 19.3 ft wall that no photo saw, hidden behind scaffold footings. Requiring views from different angles removed it only by also discarding 6 to 8.5 ft of good wall. A depth test, available on LiDAR phones, removed it cleanly. The hatched span is drawn at 8.5 ft, the larger end of the range.
        +
        +

        What it changed: decision 7 (who checks what is hidden). A wrong claim in the app's code, "the server rechecks from the images", was corrected: the server never sees images.

        +
        + +
        +

        Finding 4. Can the phone read the meter?

        +
        + + Meter reading success on real meter photos + Four horizontal bars on a 0 to 100 percent scale. Read the full number: 71 of 73, 97 percent. Picked the right line: 21 of 75. Barcode held the number: 20 of 24 photos that had one. A tap list of three candidates contained it: 85 percent of held-out photos. + + + + 0255075100% + + full number readright line pickedbarcode had itin the tap list of 3 + + + + + + 71 of 73, 97% + 21 of 75 + 20 of 24 photos that had one + 85% of held-out photos + + +
        Apple's on-device text recognition read the full meter number on 71 of 73 real meter photos, but could not tell which line on the plate was the number (right on 21 of 75). A barcode held the number on 20 of 24 photos that had one. A tap-to-confirm list of three candidates contained it on 85% of held-out photos.
        +
        +

        What it changed: the app shows three candidates to tap, barcode match first, plus two photo checks that trigger a retake (blurry; number too small, "move closer").

        +
        + +
        +

        Finding 5. Can a first-time homeowner finish a scan?

        +
        + + Ground depth the app reported against what the checks need + A scale from 0 to 14 feet. The app reported ground seen out to 4 feet. The checks need 6 to 13 feet. The two do not overlap, so no scan could finish. + + + + 04812 ft + app reported + + 4 ft of ground seen + checks need + + 6 to 13 ft + +
        The verification agent's friction audit: no, not before this morning's fixes. However well it was walked, the app said it had seen ground only 4 ft out while checks need 6 to 13 ft, never reported the space in front of or above the wall, and turned corners into "walk past this" requests.
        +
        +

        What it changed: the capture contract, and the app fixes in section 4: the server publishes exactly what settles each check, and the app reports real ground depth, the walked path as clear space in front, and a tilt-up view for headroom.

        +
        + +

        Three pieces of overnight work changed nothing in the product: the panel-label search (too few open photos: 7 US panels against 40 needed), finer statistics on the modern-phone data, and the research notes.

        + + +

        7. Decisions for this morning, recommendation first

        +
          +

        1. Run the field test first

          Recommend: about an hour, before anything else.

          A current non-LiDAR iPhone, Measure Lab, a 30 ft tape, a real meter wall. Follow experiments/evals/field/FIELD_SHEET.md. One command turns tape readings into a score and the phone's real scale error.

          2. +

        2. How points get measured

          Recommend: metric depth where precision decides a check.

          Take those points from LiDAR or AR taps; learned depth alone gets edges to 8 in or worse (Finding 2). The 3D model still carries everything else.

          3. +

        3. How far to walk

          Recommend: outward from the meter on both sides; stop when the nearest spot that fits is fully seen.

          The server prefers the shortest cable, so walking farther cannot find a better spot, and shorter walks drift less.

          4. +

        4. Base's real rules

          Recommend: load them into the private rules file now.

          Confirm with Base which rules the demo follows and which number on the meter plate they need (plates carry several). Otherwise the demo can never show a real "fits here".

          5. +

        5. Merge order

          Recommend: #11 server first, since the others depend on its schemas.

          Then #14 docs, #10 app, #13 verification, #12 evals, then #7, #4, #8, #6, #16, #17, #19.

          6. +

        6. Lane owners

          Recommend: name one owner per lane.

          Capture app, geometry, rules, output. Agents then know whose OK they need.

          7. +

        7. Who checks what is hidden behind things?

          Recommend: the app.

          Before the result, show the homeowner the photo of the chosen spot and ask if anything blocks the wall or ground there. On LiDAR phones, also run a depth test (finding 3).

          8. +

        8. What must the scan see, and what can we just ask?

          Recommend: pool and driveway become one yes/no question, labelled as the homeowner's answer.

          Instead of requiring 8 to 13 ft of photographed ground.

          9. +
        + + +

        8. Asks: only a person can do these

        +
          +
        • Sam: review bots. Only CodeRabbit is installed and it is rate-limited; Greptile, Macroscope, Aikido and Codex review are not installed.
          • +
        • Sam: a one-time App Store Connect key for TestFlight (PR #8).
          • +
        • Sam: web previews on Vercel were turned off because the free plan's 100 deploys a day ran out; reconnect later.
          • +
        • Anyone, 5 minutes: confirm the meter labels on the local review page on Sam's laptop.
          • +
        + + +

        9. Reference: what exists, and how to try it

        +

        Open pull requests. None of these is merged yet. The server (#11) is live.

        +
          +
        • #4 scoring harness
          • +
        • #6 research notes
          • +
        • #7 Measure Lab
          • +
        • #8 TestFlight upload
          • +
        • #9 Hunter's camera-metadata probe
          • +
        • #10 iOS app
          • +
        • #11 placement server (live)
          • +
        • #12 evals
          • +
        • #13 independent verification
          • +
        • #14 docs cleanup
          • +
        • #15 web review page (parked)
          • +
        • #16 meter reading
          • +
        • #17 panel labels (blocked on photos)
          • +
        • #18 Hunter's research handoff
          • +
        • #19 this briefing
          • +
        + +

        How to try it

        +
        +

        Server health: house-scanning-server.vercel.app/health

        +

        App in the Simulator: build ios/HouseScan.xcodeproj and run with

        +
        -replay <folder> -autopilot -serverURL https://house-scanning-server.vercel.app
        +

        On a real iPhone: set your team in ios/Config/Local.xcconfig (see ios/README.md), plug in, run.

        +
        + +
        Facts from the team dossier as of 09:10, 26 September 2026. Base's private thresholds do not appear on this page. The drawings are illustrations at a consistent scale, not a real house.
        + +
        + + From f23a776ccfe0f85bdbd2d54938d53f26f21d94aa Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 14:51:31 -0500 Subject: [PATCH 12/16] docs: merge the morning briefing into how-it-works as one page - how-it-works.html keeps the client and server team pipeline and gains the briefing's graphics that still hold: the six pictures of what the homeowner sees, error growing with distance from the meter, and the tracking, coverage and meter-reading charts, plus the field test and how to try the app and server. - Dated parts of the briefing are dropped: the morning status, asks, merge order, lane owners, the fixed ground-depth bug, and the depth chart that section 4 already covers. - Delete the briefing and its README row, so one page holds the current picture. Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 1 - docs/briefings/2026-09-26-morning.html | 930 ------------------------- docs/how-it-works.html | 373 +++++++++- 3 files changed, 365 insertions(+), 939 deletions(-) delete mode 100644 docs/briefings/2026-09-26-morning.html diff --git a/README.md b/README.md index 153aea459..d64ad1739 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,6 @@ Paths marked with a pull request exist only on that branch until it merges. | `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, meter reading in PR #16 | | `web/` | Browser toolchain for a reviewer view | The review page (PR #15) is parked | | `docs/` | The overview, the walkthrough, public rules and code citations, and the live-survey design | | -| `docs/briefings/` | The 26 September morning briefing, a dated snapshot of findings and decisions | | | `.agents/skills/` | Shared agent skills for writing, planning and review, linked from `.claude/skills/` | | | `sites/landing` | The landing page, a submodule | Change it in its own repository | diff --git a/docs/briefings/2026-09-26-morning.html b/docs/briefings/2026-09-26-morning.html deleted file mode 100644 index a393d63a3..000000000 --- a/docs/briefings/2026-09-26-morning.html +++ /dev/null @@ -1,930 +0,0 @@ - - - - - - -House scanning briefing: what we are building, what works, what is blocked - - - - - - - -
        - -

        House scanning: what we are building, what works, and what is blocked

        -

        Team briefing, 26 September 2026, facts as of 09:10. Readers: Sam, Hunter, Shrey, Aiden. About ten minutes.

        -

        Update 11:15: the team split into client and server teams; see docs/how-it-works.html.

        - -

        A homeowner walks the outside wall around their electric meter with an iPhone. The phone measures the wall, a server applies the clearance rules in plain code, and the app shows where the battery goes, in AR, on the real wall.

        -

        Today it runs end to end in the iPhone Simulator on a recorded real walk and gets a real answer from the live server. It has never run on a physical iPhone at a real wall. That is the blocker, and the first job this morning.

        - - - - -

        1. The problem: a battery has to fit between the rules

        - -
        - - An outside wall with the battery and its clearance rules, drawn to scale - Front view of a house wall. From left: an AC unit on the ground, a gas meter, the electric meter, the battery just to its right, a window, a door. Red hatched zones show 3 feet of clearance around the gas meter, AC unit, window and door. A bracket beside the battery shows 6.5 feet of headroom. A blue bracket along the ground shows 20 feet of cable reach either side of the electric meter. A tape along the bottom counts feet from the meter. Below, a view from above shows the battery footprint, 31 by 22 inches, and the 3 feet of clear space needed in front of it. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 20 ft of cable, either side of the meter - - - - AC unit - gas meter - electric meter - window - door - 6.5 ft headroom - battery - - - 3 ft - 3 ft - 3 ft - 3 ft - - - - - - - - - - - 151050 - 510152025 - - feet left and right of the electric meter - - - The same wall from above - - - - - - - - - - - - - - - - - - - - - - - - battery footprint, 31 by 22 in - 3 ft clear in front - AC unit - gas - - -

        Wider than the screen: scroll sideways for the rest of the picture.

        -
        Everything is drawn at one scale: 5 ft between tape marks. The red hatched areas are places the battery may not go. The battery just fits between the electric meter and the window's zone. Rules shown: 3 ft from the gas meter, from AC units, fences and other batteries, and from doors and windows that enter the home; 3 ft of clear space in front; 6.5 ft of headroom; within 20 ft of cable run from the meter; a flat, seen stretch of wall behind it.
        -
        - -
        -

        Base Power puts a home battery on the outside wall, close to the electric meter, because a cable runs from the meter to the battery. The battery is 31 in wide, 22 in deep and 39.5 in tall. Whether it fits, and where, depends on the clearance rules in the picture. Today a person has to go to the house and check.

        -

        Our product does that check with the homeowner's iPhone, in a couple of minutes. The app measures the wall. A server applies the rules in plain code. The app shows where the battery goes, in AR, on the real wall.

        -

        The demo's rules come from public sources and live in server/rules.yaml, each with its citation. Two numbers there are placeholders because no public value exists (pool 10 ft, driveway 5 ft). Base's own thresholds are private, load at runtime from a file that is not in git, and do not appear on this page.

        -
        -
        - The rules the demo uses, with sources -
        - - - - - - - - - - - -
        RuleValueSource
        Distance from the gas meter3 ftBase's public help page; Austin Energy; Texas Gas Service
        Distance from AC units, fences, other batteries3 ftBase's help page
        Distance from doors and windows that enter the home3 ftIRC R328.4
        Clear space in front of the battery3 ftBase's help page
        Cable run from the meterwithin 20 ftBase's help page
        Headroom6.5 ftNEC 110.26, a demo choice
        Wall behind the batteryflat, and seen by the camerademo requirement
        Pool, driveway10 ft, 5 ftplaceholders; no public value exists
        -
        - - -

        2. How a scan becomes a decision

        -

        Six pictures. Each one is the last one with one thing added. The tape along the bottom is real: the app shows the wall unrolled like a tape measure, with the electric meter at zero.

        - -
          -
        1. - - Step 1: tap the electric meter, which becomes zero - The wall from the problem picture, smaller, with the electric meter in the middle. A blue ring marks a tap on the meter. A dashed line drops from the meter to the zero mark on a tape along the bottom. - - - - - tap the electric meter - it becomes 0 ft; all else is left or right of it - -
          Point at the electric meter and tap. The meter becomes zero. Every position from now on is measured left or right of it, in feet.
          -
          2. - -
        2. - - Step 2: the phone photographs the meter close up, then walks left, then right - Same wall. A dashed frame around the electric meter marks the close-up. A phone stands on the ground below the meter. Blue arrows run left along the wall and then right. - - - close-up, reads the number - - - - walk left - then right - -
          The phone takes a close-up of the meter by itself when the picture is sharp, to read the meter's number. Then you walk along the wall, left and then right. The phone keeps photos automatically.
          -
          3. - -
        3. - - Step 3: a haze covers wall and ground the phone has not seen yet - Same wall. The phone has walked left to about 8 feet and right to about 9 feet. Those parts of the wall and ground are clear. The rest is covered by a grey hatched haze. - - - - - - - - haze: - not seen yet - haze - seen: the haze lifts - -
          A haze covers wall and ground the phone has not seen and lifts as it sees them. The team calls this fog of war. The tape strip grows as you walk.
          -
          4. - -
        4. - - Step 4: mark things as you pass, and answer one request for another view - Same wall, almost all seen. Orange pins mark the gas meter, the window and the AC unit. A small hatched patch of ground about 6 feet right of the meter is still unseen. A message at the top reads: Show the ground about 6 ft right of your meter. - - - - - - gas - window - AC - - - Show the ground about 6 ft right of your meter - -
          Mark things as you pass: gas meter, AC unit, door, window, driveway edge, fence. If something a rule needs is still unseen, the app asks for one more view, in one sentence.
          -
          5. - -
        5. - - Step 5: a small measurement file goes to the server, which tries every spot - Same wall, fully seen. A small file icon labelled scene.json, 17 KB, rises from the phone toward a server box at the top. On the wall, dashed battery outlines slide along in small steps. - - - - - - tries every spot, 2 in apart - - - scene.json, 17 KB - photos stay on the phone - - - server: checks every rule - -
          The phone sends a small file of measurements, about 17 KB for a real walk. The server slides the battery's footprint along the wall in 2-inch steps and checks every rule at every position.
          -
          6. - -
        6. - - Step 6: the answer comes back and the battery is shown on the wall in AR - Same wall. The server box at the top sends an arrow back down to the phone. The battery is drawn on the wall just right of the electric meter with an orange outline, and a cable runs from the meter to it. A label reads: best spot, shortest cable. - - - server answers - - - - - - best spot, - shortest cable - shown in 3D and in AR, placed from the meter - -
          The server answers with the best spot (shortest cable run first), the cable route, and every check's outcome and reason. The phone draws the battery on the wall, placed relative to the meter, in 3D and in AR.
          -
          7. -
        - -
        - - The whole loop: phone to server and back - Two boxes. Left: the phone, which taps, walks, marks and shows. Right: the server, with the rules file and plain code. An arrow from phone to server carries scene.json. An arrow from server to phone carries the spot, the cable route and every check's outcome. - - Phone - tap, walk, mark - show the spot in AR - - Server - rules in a file - plain code, no AI - - scene.json - - spot, cable, checks - -
        The recap. AI is used only to recognise things: it reads the meter's text and finds objects. The placement decision is ordinary code with the rules in a file, so every answer traces back to a rule and a measurement.
        -
        - - -

        3. Three ideas that make the answer honest

        - -

        The phone counts its own steps, so error grows with distance

        -
        - - Error bars widen the farther you walk from the meter - A tape along the top counts feet from the meter at 0, 5, 10, 15 and 20. Below it, four positions carry horizontal error bars that get wider with distance: plus or minus 0.3 feet at the meter, 1.1 feet at 5 feet, 1.9 feet at 10 feet and 3.5 feet at 20 feet. Footprints lead away from the meter. - - - - - - - 05101520 ft - - - meter - - - - - - - - - - - - - - - ± 0.3 ft at the meter - ± 1.1 ft at 5 ft - ± 1.9 ft at 10 ft - ± 3.5 ft at 20 ft - - error bar = 0.3 ft + 0.16 ft for every foot from the meter - -
        AR tracking is dead reckoning: the phone estimates how far it moved from the camera and motion sensors, like counting your own steps from a doorway. Small errors add up, so the farther from the meter, the less sure a position is. The server widens every AR measurement's error bar by 0.3 ft plus 0.16 ft per foot along the wall. Where the analogy breaks: a miscounted step is only a stretch along the wall, while the phone's error also drifts sideways.
        -
        - -

        The error bar decides: PASS, FAIL or UNSURE

        -
        - - Three measurements against the 3 foot gas meter rule - A number line from 0 to 6 feet with a vertical rule at 3 feet. Row one: 4.5 feet plus or minus 0.8, the whole bar sits right of 3, PASS. Row two: 2.0 feet plus or minus 0.8, the whole bar sits left of 3, FAIL. Row three: 3.4 feet plus or minus 0.8, the bar straddles 3, UNSURE. - - - - - 0123456 ft - - - 3 ft rule (gas meter) - too close - clear - - - - - - - 4.5 ft ± 0.8: clear by 1.5, more than 0.8 - - ✓ PASS - - - - - - - 2.0 ft ± 0.8: short by 1.0, more than 0.8 - - ✕ FAIL - - - - - - - 3.4 ft ± 0.8: clear by only 0.4 - - ? UNSURE - -
        Every measurement carries an error bar. A check passes only when the margin beats the error, fails only when it is past the rule by more than the error, and is UNSURE otherwise. Equal is UNSURE. UNSURE means a person checks it, or the app asks for a better view. The three examples are the dossier's, for the 3 ft gas meter rule with ± 0.8 ft.
        -
        - -

        Unseen is not clear

        -
        - - A check whose ground was never seen stays UNSURE and names the missing view - A candidate battery spot on the wall. The ground beside it is covered by grey haze. Beside it a check reads: ground near the spot, checks need 6 to 13 ft of seen ground, UNSURE, missing evidence: show the ground about 6 ft right of your meter. - - - - - - meter - - - candidate spot - - never seen - - Ground near the spot - checks need 6 to 13 ft of ground - - ? UNSURE - missing evidence: show the - ground 6 ft right of your meter - -
        If the camera never saw part of the area a rule depends on, that check cannot pass. It stays UNSURE, and the server names the view that would settle it. Here the ground beside the spot was never seen, and the checks that use ground need 6 to 13 ft of it. A blank on the map is never treated as empty ground.
        -
        - - -

        4. Where it stands this morning

        -
        - ✓ proven in the Simulator on a recorded real walk - ! fixed this morning - ? not yet proven on a real phone -
        -
          -
        1. Tap the meterIt becomes zero.
          ✓Simulator?Real phone
          2. -
        2. Read the meter's numberClose-up; three candidates to tap.
          ✓Simulator?Real phone
          3. -
        3. Walk the wallHaze lifts; tape strip grows.
          ✓Simulator!Now reports ground depth, clear space in front, and a tilt-up view for headroom?Real phone
          4. -
        4. Mark things, one more viewGas, AC, door, window, fence.
          ✓Simulator!Server now publishes exactly what settles each check?Real phone
          5. -
        5. Upload scene.jsonMeasurements only.
          !17 KB, not 16.5 MB of photos; answer in 0.5 s?Real phone
          6. -
        6. Server decidesLive on Vercel.
          ✓252 tests; independent checker 50 of 50
          7. -
        7. Show the spot in ARPlaced from the meter.
          ✓Simulator, 3D and AR?Real phone
          8. -
        -
        ✕
        Never run on a physical iPhone at a real wall.Every stage marked "?" above shares this one gap. Everything that decides accuracy is waiting on that data.
        -
        -
        Placeholder rulesWith the public placeholder rules, automatic approval is off, so every real result says "An installer will take a look."
        -
        Hidden wallThings standing in front of the wall can hide it, and nobody owns checking that yet (decision 7).
        -
        Parked: web review page (#15)It shows the server's answer in a browser. It helped check the server before the app could upload. It is not the product.
        -
        - -
        -
        Does what we built work, and is it part of the app we'd ship?
        In the Simulator on a recorded real walk, yes: meter to result, with a real answer from the live server. The app and the server are the product. The web review page is not, and it is parked.
        -
        Are the overnight tests doing anything?
        Each of the five changed the app or the server (section 6 lists what). Three other pieces of overnight work changed nothing: the panel-label search, finer statistics on the modern-phone data, and the research notes.
        -
        What is done that achieves the goal?
        The server, live with 252 tests. The whole app flow in the Simulator. Uploads fixed to 17 KB. The first-try fixes, merging now while the build is repaired.
        -
        What is the main blocker?
        A real-phone session. See the next section.
        -
        - - -

        5. The main blocker, and what to do this morning

        -
        - - The field test: the phone's reading against a tape on the same wall - A wall with the electric meter. A yellow tape runs along the wall with marks at 5, 10, 15 and 20 feet. A phone stands in front. Blue dots show where the phone thinks the same marks are; the gap between dot and mark grows with distance. A label reads: the gap is the phone's scale error. - - - - - - - - - - - 5101520 ft - - - - - - blue dots: where the phone says the tape marks are - the gap is the scale error - - a current iPhone, no LiDAR - -
        Nothing decides accuracy until this runs. Hunter's research handoff (PR #18) reaches the same conclusion.
        -
        -
        -

        Run the field test first, about an hour. You need a current non-LiDAR iPhone, the Measure Lab build, a 30 ft tape and a real meter wall. Follow experiments/evals/field/FIELD_SHEET.md: type the tape readings into the prepared templates, and one command turns them into a score and the phone's real scale error. A dry run recovered a planted 1.5% error exactly, so the scoring end is ready.

        -

        The overnight tracking numbers (finding 1) come from a 2023 iPhone with LiDAR and a dataset whose reference may itself be scaled to the phone's tracking. Most homeowners' phones lack LiDAR. The field test measures our own phone on our own wall, which is the only direct measurement.

        -
        - - -

        6. What the overnight tests found, and what each changed

        -

        All five ran on public real-world datasets with laser-measured ground truth. Dataset images are never in git. Each finding below ends with the line that answers "did this test do anything".

        - -
        -

        Finding 1. How far off is the phone's own tracking?

        -
        -
        - 2018 iPhone 6s (ADVIO) - 2023 iPhone 14 Pro Max (MARViN) - server's allowance -
        - - p90 position error in inches against feet walked, two phones and the server's allowance - At 3, 10, 20 and 30 feet walked: 2018 iPhone 6s 18.6, 55.7, 94.6, 132.6 inches. 2023 iPhone 14 Pro Max 4.0, 8.6, 13.4, 18.5 inches. Server's allowance 5.8, 19.2, 38.4, 57.6 inches. The 2023 phone stays under the allowance at every distance. The 2018 phone is over it at every distance. - - - - 040 in80 in120 in - 3 ft10 ft20 ft30 ftdistance walked - - - - - - - - - 2018 phone, 132.6 in - allowance, 57.6 in - 2023 phone, 18.5 in - - 3 ft walked: 2018 phone 18.6 in, 2023 phone 4.0 in, allowance 5.8 in - 10 ft walked: 2018 phone 55.7 in, 2023 phone 8.6 in, allowance 19.2 in - 20 ft walked: 2018 phone 94.6 in, 2023 phone 13.4 in, allowance 38.4 in - 30 ft walked: 2018 phone 132.6 in, 2023 phone 18.5 in, allowance 57.6 in - - -
        Position error after walking, p90 (9 in 10 cases are better). A current iPhone's own tracking stays inside the server's allowance at every distance; the 2018 phone is far over it. The allowance line is the 0.16 ft per foot part of the server's error bar. Caveats: the MARViN reference may itself be scaled to the phone's tracking, and that phone has LiDAR, which most homeowners' phones lack.
        -
        Data table -
        - - - - - - - -
        Walked2018 iPhone 6s (ADVIO), in2023 iPhone 14 Pro Max (MARViN), inServer's allowance, in
        3 ft18.64.05.8
        10 ft55.78.619.2
        20 ft94.613.438.4
        30 ft132.618.557.6
        -
        -
        -

        What it changed: the server's error allowance comes from this. The field test measures our own phone.

        -
        - -
        -

        Finding 2. Can photos alone give 3D measurements?

        -
        - - Error of learned depth from photos: flat walls and edges - A horizontal scale from 0 to 10 inches. Flat walls: 5.0 inches at p90, with a 95 percent interval from 3.8 to 6.8. Window and door edges: 8 inches or worse in every setting, drawn as a bar starting at 8 and running off the right edge. - - - - 0246810 inerror, p90 - flat wall - - - - 5.0 in (3.8 to 6.8) - edges - - - 8 in or worse, every setting - -
        Learned depth models (MoGe-2, Depth Anything 3), rescaled with the phone's poses. Flat walls come out at about 5 in at best at a realistic phone error. Window and door edges, which are what clearances are measured from, come out at 8 in or worse in every setting.
        -
        -

        What it changed: 3D modeling stays central. Learned depth alone is too rough at edges, so the points that decide a check come from metric depth (LiDAR or AR taps), and LiDAR and the phone's poses give the model its scale. World models are not evaluated yet.

        -
        - -
        -

        Finding 3. Does the app's coverage map claim wall it never saw?

        -
        -
        - wall a photo saw - claimed, never seen - good wall thrown away -
        - - A 19.3 foot wall under three versions of the coverage map - Three bars, each 19.3 feet long. First, as claimed: 1.1 feet marked as seen that no photo saw, hidden behind scaffold footings. Second, after requiring views from different angles: the false 1.1 feet is gone but 6 to 8.5 feet of good wall is also thrown away. Third, after a depth test on LiDAR phones: the false part is removed cleanly and the rest is kept. - - as claimed - - - 1.1 ft never seen - views from different angles - - - - 6 to 8.5 ft of good wall lost too - depth test (LiDAR phones) - - - clean - -
        Run on real photos of a laser-scanned building. The map claimed 1.1 ft of a 19.3 ft wall that no photo saw, hidden behind scaffold footings. Requiring views from different angles removed it only by also discarding 6 to 8.5 ft of good wall. A depth test, available on LiDAR phones, removed it cleanly. The hatched span is drawn at 8.5 ft, the larger end of the range.
        -
        -

        What it changed: decision 7 (who checks what is hidden). A wrong claim in the app's code, "the server rechecks from the images", was corrected: the server never sees images.

        -
        - -
        -

        Finding 4. Can the phone read the meter?

        -
        - - Meter reading success on real meter photos - Four horizontal bars on a 0 to 100 percent scale. Read the full number: 71 of 73, 97 percent. Picked the right line: 21 of 75. Barcode held the number: 20 of 24 photos that had one. A tap list of three candidates contained it: 85 percent of held-out photos. - - - - 0255075100% - - full number readright line pickedbarcode had itin the tap list of 3 - - - - - - 71 of 73, 97% - 21 of 75 - 20 of 24 photos that had one - 85% of held-out photos - - -
        Apple's on-device text recognition read the full meter number on 71 of 73 real meter photos, but could not tell which line on the plate was the number (right on 21 of 75). A barcode held the number on 20 of 24 photos that had one. A tap-to-confirm list of three candidates contained it on 85% of held-out photos.
        -
        -

        What it changed: the app shows three candidates to tap, barcode match first, plus two photo checks that trigger a retake (blurry; number too small, "move closer").

        -
        - -
        -

        Finding 5. Can a first-time homeowner finish a scan?

        -
        - - Ground depth the app reported against what the checks need - A scale from 0 to 14 feet. The app reported ground seen out to 4 feet. The checks need 6 to 13 feet. The two do not overlap, so no scan could finish. - - - - 04812 ft - app reported - - 4 ft of ground seen - checks need - - 6 to 13 ft - -
        The verification agent's friction audit: no, not before this morning's fixes. However well it was walked, the app said it had seen ground only 4 ft out while checks need 6 to 13 ft, never reported the space in front of or above the wall, and turned corners into "walk past this" requests.
        -
        -

        What it changed: the capture contract, and the app fixes in section 4: the server publishes exactly what settles each check, and the app reports real ground depth, the walked path as clear space in front, and a tilt-up view for headroom.

        -
        - -

        Three pieces of overnight work changed nothing in the product: the panel-label search (too few open photos: 7 US panels against 40 needed), finer statistics on the modern-phone data, and the research notes.

        - - -

        7. Decisions for this morning, recommendation first

        -
          -

        1. Run the field test first

          Recommend: about an hour, before anything else.

          A current non-LiDAR iPhone, Measure Lab, a 30 ft tape, a real meter wall. Follow experiments/evals/field/FIELD_SHEET.md. One command turns tape readings into a score and the phone's real scale error.

          2. -

        2. How points get measured

          Recommend: metric depth where precision decides a check.

          Take those points from LiDAR or AR taps; learned depth alone gets edges to 8 in or worse (Finding 2). The 3D model still carries everything else.

          3. -

        3. How far to walk

          Recommend: outward from the meter on both sides; stop when the nearest spot that fits is fully seen.

          The server prefers the shortest cable, so walking farther cannot find a better spot, and shorter walks drift less.

          4. -

        4. Base's real rules

          Recommend: load them into the private rules file now.

          Confirm with Base which rules the demo follows and which number on the meter plate they need (plates carry several). Otherwise the demo can never show a real "fits here".

          5. -

        5. Merge order

          Recommend: #11 server first, since the others depend on its schemas.

          Then #14 docs, #10 app, #13 verification, #12 evals, then #7, #4, #8, #6, #16, #17, #19.

          6. -

        6. Lane owners

          Recommend: name one owner per lane.

          Capture app, geometry, rules, output. Agents then know whose OK they need.

          7. -

        7. Who checks what is hidden behind things?

          Recommend: the app.

          Before the result, show the homeowner the photo of the chosen spot and ask if anything blocks the wall or ground there. On LiDAR phones, also run a depth test (finding 3).

          8. -

        8. What must the scan see, and what can we just ask?

          Recommend: pool and driveway become one yes/no question, labelled as the homeowner's answer.

          Instead of requiring 8 to 13 ft of photographed ground.

          9. -
        - - -

        8. Asks: only a person can do these

        -
          -
        • Sam: review bots. Only CodeRabbit is installed and it is rate-limited; Greptile, Macroscope, Aikido and Codex review are not installed.
          • -
        • Sam: a one-time App Store Connect key for TestFlight (PR #8).
          • -
        • Sam: web previews on Vercel were turned off because the free plan's 100 deploys a day ran out; reconnect later.
          • -
        • Anyone, 5 minutes: confirm the meter labels on the local review page on Sam's laptop.
          • -
        - - -

        9. Reference: what exists, and how to try it

        -

        Open pull requests. None of these is merged yet. The server (#11) is live.

        -
          -
        • #4 scoring harness
          • -
        • #6 research notes
          • -
        • #7 Measure Lab
          • -
        • #8 TestFlight upload
          • -
        • #9 Hunter's camera-metadata probe
          • -
        • #10 iOS app
          • -
        • #11 placement server (live)
          • -
        • #12 evals
          • -
        • #13 independent verification
          • -
        • #14 docs cleanup
          • -
        • #15 web review page (parked)
          • -
        • #16 meter reading
          • -
        • #17 panel labels (blocked on photos)
          • -
        • #18 Hunter's research handoff
          • -
        • #19 this briefing
          • -
        - -

        How to try it

        -
        -

        Server health: house-scanning-server.vercel.app/health

        -

        App in the Simulator: build ios/HouseScan.xcodeproj and run with

        -
        -replay <folder> -autopilot -serverURL https://house-scanning-server.vercel.app
        -

        On a real iPhone: set your team in ios/Config/Local.xcconfig (see ios/README.md), plug in, run.

        -
        - -
        Facts from the team dossier as of 09:10, 26 September 2026. Base's private thresholds do not appear on this page. The drawings are illustrations at a consistent scale, not a real house.
        - -
        - - diff --git a/docs/how-it-works.html b/docs/how-it-works.html index eaaf253a2..9d6fa98a2 100644 --- a/docs/how-it-works.html +++ b/docs/how-it-works.html @@ -57,6 +57,22 @@ details{margin:6px 0 12px} summary{cursor:pointer;font-family:var(--sans);font-weight:600;font-size:15px;color:var(--muted)} + pre{font-family:var(--mono);font-size:14px;line-height:1.5;background:var(--code-bg);padding:10px 12px;border-radius:4px;overflow-x:auto;white-space:pre;margin:8px 0 14px} + + /* Chart legends */ + .legend{display:flex;flex-wrap:wrap;gap:6px 16px;font-family:var(--sans);font-size:14px;color:var(--muted);margin:0 0 8px} + .legend span{display:inline-flex;align-items:center;gap:6px} + .sw{display:inline-block;width:20px;height:0;border-top:3px solid} + .sw.dash{border-top-style:dashed} + .sw.box{width:14px;height:14px;border:0;border-radius:3px} + .sw.hatch{width:14px;height:14px;border:0;background:repeating-linear-gradient(135deg,var(--muted) 0 2px,transparent 2px 6px)} + + /* Test findings */ + .finding{margin-top:34px;padding-top:8px} + .finding h3{margin-top:0} + .changed{font-family:var(--sans);font-size:15px;border:1px solid var(--line);border-left:3px solid var(--phone);border-radius:4px;padding:8px 12px;margin:0 0 6px;background:var(--card)} + .changed b{font-weight:600} + /* Contents */ nav.toc{font-family:var(--sans);font-size:15px;margin:22px 0 0;border:1px solid var(--line);border-radius:6px;padding:12px 16px;background:var(--card)} nav.toc ol{margin:6px 0 0;padding-left:1.4em} @@ -140,6 +156,9 @@ + + + @@ -156,6 +175,37 @@ + + + + + + + + + + + + + + + + + + + + + + + + 151050 + 5101520 + + + + + + @@ -226,7 +276,7 @@

        How house scanning works

        -

        Living explainer for new teammates and agents. Current as of 26 September 2026, midday. About fifteen minutes.

        +

        Living explainer for new teammates and agents. Current as of 26 September 2026. About twenty minutes.

        A homeowner scans the outside of their house around the electric meter with an iPhone. The phone captures the wall, the server builds a 3D model and checks Base Power's placement rules against it, and the app shows where the battery goes, in AR, on the real wall.

        Picture a home inspector with a tape measure and a checklist. The phone is the inspector's eyes and tape, and the server holds the checklist. The comparison breaks in one place, and that place shapes the design: an inspector can lean around a bush to look, but the server can judge only what the phone actually saw.

        @@ -235,10 +285,11 @@

        How house scanning works

        After one read you should be able to explain:
        1. what has to fit where
          2. -
        2. how a scan becomes an answer, and which team owns each step
          3. +
        3. how a scan becomes an answer, which team owns each step, and what the homeowner sees
        4. four ideas that keep the answer honest
          5. -
        5. the options for building the 3D model, and what the evidence says so far
          6. -
        6. where each piece lives
          7. +
        7. the options for building the 3D model
          8. +
        8. what the tests found, and what is still unmeasured
          9. +
        9. where each piece lives, and how to try it
        @@ -403,7 +454,7 @@

        2. From scan to answer, one step at a time

        The guided scan box from stage 1, with an arrow to a file icon labelled packet, sitting on the line between the teams. -
        It sends a capture packet. Today the app uploads only its measurements and exports the full packet on request; the packet's README lists what it records so far. The packet holds photos; where the camera was and which way it pointed for each one (poses); the lens's focal length and center (intrinsics); motion-sensor readings; LiDAR depth and mesh when the phone has LiDAR; and the homeowner's marks, such as the meter. The packet is the contract between the teams.
        +
        It sends a capture packet. Today the app uploads only its measurements and exports the full packet on request. The packet holds photos; where the camera was and which way it pointed for each one (poses); the lens's focal length and center (intrinsics); motion-sensor readings; LiDAR depth and mesh when the phone has LiDAR; and the homeowner's marks, such as the meter. The packet is the contract between the teams.
      • @@ -434,6 +485,119 @@

        2. From scan to answer, one step at a time

        • +

        What the homeowner sees

        +

        The same loop from the phone's side, in six pictures. Each adds one thing to the last. The tape along the bottom is real: the app shows the wall unrolled like a tape measure, with the electric meter at zero.

        + +
          +
        1. + + Step 1: tap the electric meter, which becomes zero + The wall from the problem picture, smaller, with the electric meter in the middle. A blue ring marks a tap on the meter. A dashed line drops from the meter to the zero mark on a tape along the bottom. + + + + + tap the electric meter + it becomes 0 ft; all else is left or right of it + +
          Point at the electric meter and tap. The meter becomes zero. Every position from now on is measured left or right of it, in feet.
          +
          2. + +
        2. + + Step 2: the phone photographs the meter close up, then walks left, then right + Same wall. A dashed frame around the electric meter marks the close-up. A phone stands on the ground below the meter. Blue arrows run left along the wall and then right. + + + close-up, reads the number + + + + walk left + then right + +
          The phone takes a close-up of the meter by itself when the picture is sharp, to read the meter's number. Then you walk along the wall, left and then right. The phone keeps photos automatically.
          +
          3. + +
        3. + + Step 3: a haze covers wall and ground the phone has not seen yet + Same wall. The phone has walked left to about 8 feet and right to about 9 feet. Those parts of the wall and ground are clear. The rest is covered by a grey hatched haze. + + + + + + + + haze: + not seen yet + haze + seen: the haze lifts + +
          A haze covers wall and ground the phone has not seen and lifts as it sees them. The team calls this fog of war. The tape strip grows as you walk.
          +
          4. + +
        4. + + Step 4: mark things as you pass, and answer one request for another view + Same wall, almost all seen. Orange pins mark the gas meter, the window and the AC unit. A small hatched patch of ground about 6 feet right of the meter is still unseen. A message at the top reads: Show the ground about 6 ft right of your meter. + + + + + + gas + window + AC + + + Show the ground about 6 ft right of your meter + +
          Mark things as you pass: gas meter, AC unit, door, window, driveway edge, fence. If something a rule needs is still unseen, the app asks for one more view, in one sentence.
          +
          5. + +
        5. + + Step 5: a small measurement file goes to the server, which tries every spot + Same wall, fully seen. A small file icon labelled scene.json, 17 KB, rises from the phone toward a server box at the top. On the wall, dashed battery outlines slide along in small steps. + + + + + + tries every spot, 2 in apart + + + scene.json, 17 KB + photos stay on the phone + + + server: checks every rule + +
          The phone sends its measurements, scene.json, about 17 KB for a real walk. The server slides the battery's footprint along the wall in 2-inch steps and checks every rule at every position.
          +
          6. + +
        6. + + Step 6: the answer comes back and the battery is shown on the wall in AR + Same wall. The server box at the top sends an arrow back down to the phone. The battery is drawn on the wall just right of the electric meter with an orange outline, and a cable runs from the meter to it. A label reads: best spot, shortest cable. + + + server answers + + + + + + best spot, + shortest cable + shown in 3D and in AR, placed from the meter + +
          The server answers with the best spot (shortest cable run first), the cable route, and every check's outcome and reason. The phone draws the battery on the wall, placed relative to the meter, in 3D and in AR.
          +
          7. +
        +

        3. Four ideas that keep the answer honest

        @@ -442,6 +606,43 @@

        The error bar decides: PASS, FAIL or UNSURE

        No measurement is exact. The phone tracks its own position by dead reckoning, like counting your steps from a doorway, so its error grows the farther it gets from the meter. The comparison has a limit: a miscounted step only stretches the walk, while the phone's error can also drift sideways.

        The server widens every measurement's error bar by 0.3 ft plus 0.16 ft for each foot from the meter. On public test walks, a current iPhone's tracking stayed inside that allowance. Then each check compares the rule against the whole error bar, not a single number.

        +
        + + Error bars widen the farther you walk from the meter + A tape along the top counts feet from the meter at 0, 5, 10, 15 and 20. Below it, four positions carry horizontal error bars that get wider with distance: plus or minus 0.3 feet at the meter, 1.1 feet at 5 feet, 1.9 feet at 10 feet and 3.5 feet at 20 feet. Footprints lead away from the meter. + + + + + + + 05101520 ft + + + meter + + + + + + + + + + + + + + + ± 0.3 ft at the meter + ± 1.1 ft at 5 ft + ± 1.9 ft at 10 ft + ± 3.5 ft at 20 ft + + error bar = 0.3 ft + 0.16 ft for every foot from the meter + +
        The farther from the meter, the less sure a position is. At 20 ft a position is sure only to about ± 3.5 ft.
        +
        Three measurements against the 3 foot gas meter rule @@ -552,7 +753,7 @@

        Coverage means 100% of what the model needs

        needed: all of it ground out to 13 ft -
        Drawn to scale, 1 ft to 8 px. The zone is the wall within 20 ft of cable either side of the meter, the ground in front out to 6 to 13 ft depending on the check, and the space overhead for headroom, which a plan view can't show. Near each end it reaches past 20 ft by the clearance around a spot at the edge of reach. A pass needs only the chosen spot and its clearances seen, and the server's answer says when the walk has gone far enough. The coverage map has to be honest. The app's first map checked only range, angle and framing, and claimed 1.1 ft of wall that no photo saw. Tested against depth, the recon worker claims at most about 0.46 ft.
        +
        Drawn to scale, 1 ft to 8 px. The zone is the wall within 20 ft of cable either side of the meter, the ground in front out to 6 to 13 ft depending on the check, and the space overhead for headroom, which a plan view can't show. Near each end it reaches past 20 ft by the clearance around a spot at the edge of reach. A pass needs only the chosen spot and its clearances seen, and the server's answer says when the walk has gone far enough. Section 5 shows how the coverage map was tested against depth.

        LiDAR first, never required

        @@ -693,8 +894,154 @@

        What any path has to show

      • It runs on the packet the phone actually sends. That includes phones without LiDAR.
        • - -

        5. Where each piece lives

        + +

        5. What the tests found

        +

        These ran on public real-world datasets with laser-measured ground truth. Dataset images never enter git. The 3D path results are in section 4. The panel-label test hasn't run: only 7 open photos of US panels exist, and it needs 40.

        + +
        +

        How far off is the phone's own tracking?

        +
        +
        + 2018 iPhone 6s (ADVIO) + 2023 iPhone 14 Pro Max (MARViN) + server's allowance +
        + + p90 position error in inches against feet walked, two phones and the server's allowance + At 3, 10, 20 and 30 feet walked: 2018 iPhone 6s 18.6, 55.7, 94.6, 132.6 inches. 2023 iPhone 14 Pro Max 4.0, 8.6, 13.4, 18.5 inches. Server's allowance 5.8, 19.2, 38.4, 57.6 inches. The 2023 phone stays under the allowance at every distance. The 2018 phone is over it at every distance. + + + + 040 in80 in120 in + 3 ft10 ft20 ft30 ftdistance walked + + + + + + + + + 2018 phone, 132.6 in + allowance, 57.6 in + 2023 phone, 18.5 in + + 3 ft walked: 2018 phone 18.6 in, 2023 phone 4.0 in, allowance 5.8 in + 10 ft walked: 2018 phone 55.7 in, 2023 phone 8.6 in, allowance 19.2 in + 20 ft walked: 2018 phone 94.6 in, 2023 phone 13.4 in, allowance 38.4 in + 30 ft walked: 2018 phone 132.6 in, 2023 phone 18.5 in, allowance 57.6 in + + +
        Position error after walking, p90 (9 in 10 cases are better). A current iPhone's own tracking stays inside the server's allowance at every distance; the 2018 phone is far over it. The allowance line is the 0.16 ft per foot part of the server's error bar. Caveats: the MARViN reference may itself be scaled to the phone's tracking, and that phone has LiDAR, which most homeowners' phones lack.
        +
        Data table +
        + + + + + + + +
        Walked2018 iPhone 6s (ADVIO), in2023 iPhone 14 Pro Max (MARViN), inServer's allowance, in
        3 ft18.64.05.8
        10 ft55.78.619.2
        20 ft94.613.438.4
        30 ft132.618.557.6
        +
        +
        +

        What it set: the server's allowance of 0.16 ft per foot walked comes from these walks.

        +
        + +
        +

        Does the coverage map claim wall no photo saw?

        +
        +
        + wall a photo saw + claimed, never seen + good wall thrown away +
        + + A 19.3 foot wall under three versions of the coverage map + Three bars, each 19.3 feet long. First, as claimed: 1.1 feet marked as seen that no photo saw, hidden behind scaffold footings. Second, after requiring views from different angles: the false 1.1 feet is gone but 6 to 8.5 feet of good wall is also thrown away. Third, after a depth test on LiDAR phones: the false part is removed cleanly and the rest is kept. + + as claimed + + + 1.1 ft never seen + views from different angles + + + + 6 to 8.5 ft of good wall lost too + depth test (LiDAR phones) + + + clean + +
        Run on real photos of a laser-scanned building. The map claimed 1.1 ft of a 19.3 ft wall that no photo saw, hidden behind scaffold footings. Requiring views from different angles removed it only by also discarding 6 to 8.5 ft of good wall. A depth test, available on LiDAR phones, removed it cleanly. The hatched span is drawn at 8.5 ft, the larger end of the range.
        +
        +

        What it means: coverage has to come from what photos actually saw. LiDAR phones can run the depth test. Tested against depth, the reconstruction worker claims at most about 0.46 ft of unseen wall. Who checks a wall hidden behind something on phones without LiDAR is still open.

        +
        + +
        +

        Can the phone read the meter?

        +
        + + Meter reading success on real meter photos + Four horizontal bars on a 0 to 100 percent scale. Read the full number: 71 of 73, 97 percent. Picked the right line: 21 of 75. Barcode held the number: 20 of 24 photos that had one. A tap list of three candidates contained it: 85 percent of held-out photos. + + + + 0255075100% + + full number readright line pickedbarcode had itin the tap list of 3 + + + + + + 71 of 73, 97% + 21 of 75 + 20 of 24 photos that had one + 85% of held-out photos + + +
        Apple's on-device text recognition read the full meter number on 71 of 73 real meter photos, but could not tell which line on the plate was the number (right on 21 of 75). A barcode held the number on 20 of 24 photos that had one. A tap-to-confirm list of three candidates contained it on 85% of held-out photos.
        +
        +

        What it set: the app offers three candidates to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small.

        +
        + +

        Still unmeasured: our own phone

        +
        + + The field test: the phone's reading against a tape on the same wall + A wall with the electric meter. A yellow tape runs along the wall with marks at 5, 10, 15 and 20 feet. A phone stands in front. Blue dots show where the phone thinks the same marks are; the gap between dot and mark grows with distance. A label reads: the gap is the phone's scale error. + + + + + + + + + + + 5101520 ft + + + + + + blue dots: where the phone says the tape marks are + the gap is the scale error + + a current iPhone, no LiDAR + +
        Every number above comes from public datasets. This test measures our own phone on our own wall.
        +
        +
        +

        The field test needs a current iPhone without LiDAR, the Measure Lab build (PR #7), a 30 ft tape and a real meter wall. Follow experiments/evals/field/FIELD_SHEET.md (PR #12): type the tape readings into the prepared templates, and one command turns them into a score and the phone's real scale error. A dry run recovered a planted 1.5% error exactly, so the scoring end is ready.

        +

        The tracking numbers come from a phone with LiDAR, on a dataset whose reference may be scaled to that phone's own tracking. Most homeowners' phones lack LiDAR, so this test is the only direct measurement.

        +
        + + +

        6. Where each piece lives

        Files marked with a pull request exist only on that branch until it merges. Each component's README wins over this page.

        @@ -710,6 +1057,16 @@

        5. Where each piece lives

        PieceSource of truthState

        The plan, the decisions and the full evidence list are in 00-overview.md. The public rule values, their code citations and the model and imagery licenses are in 04-prior-art-and-codes.md.

        +

        How to try it

        +
        +

        Server health: house-scanning-server.vercel.app/health

        +

        App in the Simulator, with guided capture from PR #10: build ios/HouseScan.xcodeproj and run it with

        +
        -replay <folder>
+-autopilot
+-serverURL https://house-scanning-server.vercel.app
        +

        On a real iPhone: set your team in ios/Config/Local.xcconfig (see ios/README.md), plug in, run.

        +
        +
        Base's private thresholds, prompt names and field names do not appear on this page. The drawings are illustrations at a consistent scale, not a real house. Evidence numbers come from public datasets with laser-scanned ground truth; no dataset images are included.
        From ebc44d45f808e02070e61ea7cfd69950aa662310 Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 15:04:15 -0500 Subject: [PATCH 13/16] docs: make how-it-works concise and skimmable - Shorten every heading to a crisp sentence-case phrase and every caption to one or two sentences. Captions that only repeated a drawing's own labels are gone, and so is the "drawn to scale" note. - Fix facts that shortening had blurred, following an independent review: all results come from public datasets rather than laser truth, the meter reader usually picks the wrong line, and the MARViN reference may be scaled to its own tracking. - The stage 1 drawing says "haze", matching the rest of the page, and visible text no longer uses semicolons. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/how-it-works.html | 184 ++++++++++++++++++++--------------------- 1 file changed, 90 insertions(+), 94 deletions(-) diff --git a/docs/how-it-works.html b/docs/how-it-works.html index 9d6fa98a2..d654f17a1 100644 --- a/docs/how-it-works.html +++ b/docs/how-it-works.html @@ -222,7 +222,7 @@ Guided scan live 3D map - fog lifts as + haze lifts as it sees @@ -276,25 +276,25 @@

        How house scanning works

        -

        Living explainer for new teammates and agents. Current as of 26 September 2026. About twenty minutes.

        +

        For new teammates and agents. 26 September 2026. About ten minutes.

        -

        A homeowner scans the outside of their house around the electric meter with an iPhone. The phone captures the wall, the server builds a 3D model and checks Base Power's placement rules against it, and the app shows where the battery goes, in AR, on the real wall.

        -

        Picture a home inspector with a tape measure and a checklist. The phone is the inspector's eyes and tape, and the server holds the checklist. The comparison breaks in one place, and that place shapes the design: an inspector can lean around a bush to look, but the server can judge only what the phone actually saw.

        +

        A homeowner scans the wall around their electric meter with an iPhone. The server checks Base Power's placement rules on a 3D model of that wall. The app shows where the battery goes, in AR.

        +

        Think of a home inspector with a tape measure and a checklist. The phone is the eyes and the tape, and the server holds the checklist. One difference shapes the design: an inspector can lean around a bush, but the server can judge only what the phone saw.

        -

        1. The problem: a battery has to fit between the rules

        +

        1. Where the battery can go

        @@ -410,42 +410,42 @@

        1. The problem: a battery has to fit between the rules

        Wider than the screen: scroll sideways for the rest of the picture.

        -
        Everything is drawn at one scale: 5 ft between tape marks. The red hatched areas are places the battery may not go. The battery just fits between the electric meter and the window's zone. Rules shown: 3 ft from the gas meter, from AC units, fences and other batteries, and from doors and windows that enter the home; 3 ft of clear space in front; 6.5 ft of headroom; within 20 ft of cable run from the meter; a flat, seen stretch of wall behind it.
        +
        Red hatching marks where the battery can't go. Here it just fits between the meter and the window.
        -

        Base Power puts a home battery on the outside wall, close to the electric meter, because a cable runs from the meter to the battery. The battery is 31 in wide, 22 in deep and 39.5 in tall. Whether it fits, and where, depends on the clearance rules in the picture. Today someone judges that from a few customer photos, which have no scale and can't show what sits just outside the frame.

        -

        The demo's rules come from public sources and live in the server's rules file, each with its citation. Two numbers there are placeholders because no public value exists (pool 10 ft, driveway 5 ft). Base's own thresholds are private, load at runtime from a file that is not in git, and do not appear on this page.

        +

        The battery, 31 × 22 × 39.5 in, sits on the outside wall near the meter, because a cable runs between them. Today a person judges the fit from customer photos, which have no scale and miss what's just out of frame.

        +

        Rule values come from public sources, each cited, in the server's rules file. The pool (10 ft) and driveway (5 ft) values are placeholders. Base's own values stay private.

        - The rules the demo uses, with sources + Rule values and sources
        - + - +
        RuleValueSource
        Distance from the gas meter3 ftBase's public help page; Austin Energy; Texas Gas Service
        Distance from the gas meter3 ftBase's public help page, Austin Energy, Texas Gas Service
        Distance from AC units, fences, other batteries3 ftBase's help page
        Distance from doors and windows that enter the home3 ftIRC R328.4
        Clear space in front of the battery3 ftBase's help page
        Cable run from the meterwithin 20 ftBase's help page
        Headroom6.5 ftNEC 110.26, a demo choice
        Wall behind the batteryflat, and seen by the camerademo requirement
        Pool, driveway10 ft, 5 ftplaceholders; no public value exists
        Pool, driveway10 ft, 5 ftplaceholders, no public value exists
        -

        2. From scan to answer, one step at a time

        -

        The project splits into three problems: a phone that guides the homeowner until everything needed is captured, photos and their metadata turned into a 3D model, and that model checked against the rules. Two teams share them. Each picture below redraws the last and adds one step. The dashed line down the middle is the boundary between the teams.

        +

        2. From scan to answer

        +

        Each picture adds one step. The dashed line splits the client team from the server team.

        1. Stage 1: the phone guides the scan with a live 3D map - The client team's side holds one box: a phone whose screen shows the wall, with grey haze over the part not yet seen. The label reads: guided scan, live 3D map, fog lifts as it sees. The server team's side is empty. + The client team's side holds one box: a phone whose screen shows the wall, with grey haze over the part not yet seen. The label reads: guided scan, live 3D map, haze lifts as it sees. The server team's side is empty. -
          The phone guides the scan. The app builds a live 3D map on the phone as the homeowner moves. A haze, which the team calls fog of war, covers what the phone hasn't seen and lifts as it sees it. The app routes the homeowner until every part the model needs is seen.
          +
          The phone guides the scan until it has seen everything the rules need.
        2. @@ -454,7 +454,7 @@

          2. From scan to answer, one step at a time

          The guided scan box from stage 1, with an arrow to a file icon labelled packet, sitting on the line between the teams. -
          It sends a capture packet. Today the app uploads only its measurements and exports the full packet on request. The packet holds photos; where the camera was and which way it pointed for each one (poses); the lens's focal length and center (intrinsics); motion-sensor readings; LiDAR depth and mesh when the phone has LiDAR; and the homeowner's marks, such as the meter. The packet is the contract between the teams.
          +
          It sends a capture packet of photos, camera poses, lens data, motion data, LiDAR depth when the phone has it, and the homeowner's marks. Today the app uploads only its measurements and exports the full packet on request.
        3. @@ -463,7 +463,7 @@

          2. From scan to answer, one step at a time

          Stages 1 and 2, plus an arrow from the packet to a box on the server side showing a dotted 3D wall with a window and the meter, labelled 3D model or point cloud. -
          The server builds a 3D model. From the packet, the server team builds a 3D model or point cloud of the wall and the ground in front of it, at real scale. Which method does this best is open; section 4 compares the options.
          +
          The server builds a 3D model at real scale. Which method works best is still open (section 4).
        4. @@ -472,7 +472,7 @@

          2. From scan to answer, one step at a time

          Stages 1 to 3, plus an arrow down from the 3D model to a box labelled rules, plain code. It lists three checks: gas meter PASS, window UNSURE, door FAIL. -
          Plain code checks the rules. The rules and their sources live in a file. Code tries the battery's footprint at every spot along the wall, and each check at each spot comes out PASS, FAIL or UNSURE. No model makes this decision, so every answer traces back to a rule and a measurement.
          +
          Plain code checks the rules at every spot along the wall, so every answer traces to a rule and a measurement.
        5. @@ -481,12 +481,12 @@

          2. From scan to answer, one step at a time

          The whole loop. Guided scan, packet, 3D model, rules, and now an arrow labelled answer back across the line to a phone on the client side showing the battery outlined on the wall beside the meter. -
          The answer comes back to the phone. The app draws the battery on the real wall in AR. It places the battery relative to the meter's anchor, so the drawing stays on the right spot as the phone's tracking corrects itself. When a check is UNSURE, the answer names the view that would settle it, and the homeowner can capture it on the spot.
          +
          The answer comes back and the battery appears in AR, pinned to the meter. An UNSURE check names the view that would settle it.
        -

        What the homeowner sees

        -

        The same loop from the phone's side, in six pictures. Each adds one thing to the last. The tape along the bottom is real: the app shows the wall unrolled like a tape measure, with the electric meter at zero.

        +

        The scan, step by step

        +

        The app unrolls the wall like a tape measure, with the meter at zero.

        1. @@ -498,9 +498,9 @@

          What the homeowner sees

          tap the electric meter - it becomes 0 ft; all else is left or right of it + it becomes 0 ft, all else is left or right of it -
          Point at the electric meter and tap. The meter becomes zero. Every position from now on is measured left or right of it, in feet.
          +
        2. @@ -516,7 +516,7 @@

          What the homeowner sees

          walk left then right -
          The phone takes a close-up of the meter by itself when the picture is sharp, to read the meter's number. Then you walk along the wall, left and then right. The phone keeps photos automatically.
          +
          The phone snaps a close-up of the meter. Then you walk left and right while it takes photos on its own.
        3. @@ -535,7 +535,7 @@

          What the homeowner sees

          haze seen: the haze lifts -
          A haze covers wall and ground the phone has not seen and lifts as it sees them. The team calls this fog of war. The tape strip grows as you walk.
          +
        4. @@ -554,7 +554,7 @@

          What the homeowner sees

          Show the ground about 6 ft right of your meter -
          Mark things as you pass: gas meter, AC unit, door, window, driveway edge, fence. If something a rule needs is still unseen, the app asks for one more view, in one sentence.
          +
          Mark the gas meter, AC, doors and windows as you pass. If a rule still needs a view, the app asks for it.
        5. @@ -575,7 +575,7 @@

          What the homeowner sees

          server: checks every rule -
          The phone sends its measurements, scene.json, about 17 KB for a real walk. The server slides the battery's footprint along the wall in 2-inch steps and checks every rule at every position.
          +
        6. @@ -594,17 +594,16 @@

          What the homeowner sees

          shortest cable shown in 3D and in AR, placed from the meter -
          The server answers with the best spot (shortest cable run first), the cable route, and every check's outcome and reason. The phone draws the battery on the wall, placed relative to the meter, in 3D and in AR.
          +
          The answer also lists every check's result and reason.

        3. Four ideas that keep the answer honest

        -

        The error bar decides: PASS, FAIL or UNSURE

        +

        Error bars decide the answer

        -

        No measurement is exact. The phone tracks its own position by dead reckoning, like counting your steps from a doorway, so its error grows the farther it gets from the meter. The comparison has a limit: a miscounted step only stretches the walk, while the phone's error can also drift sideways.

        -

        The server widens every measurement's error bar by 0.3 ft plus 0.16 ft for each foot from the meter. On public test walks, a current iPhone's tracking stayed inside that allowance. Then each check compares the rule against the whole error bar, not a single number.

        +

        The phone tracks itself by dead reckoning, like counting steps, so its error grows with distance.

        @@ -641,7 +640,6 @@

        The error bar decides: PASS, FAIL or UNSURE

        error bar = 0.3 ft + 0.16 ft for every foot from the meter -
        The farther from the meter, the less sure a position is. At 20 ft a position is sure only to about ± 3.5 ft.
        @@ -685,11 +683,11 @@

        The error bar decides: PASS, FAIL or UNSURE

        ? UNSURE -
        Three measurements against the 3 ft gas meter rule, each with an error bar of ± 0.8 ft. A check passes only when its margin beats the error, fails only when it misses by more than the error, and is UNSURE otherwise; a tie is UNSURE. UNSURE means a person looks, or the app asks for a better view. Tighter geometry shrinks the error bars and turns UNSURE answers into PASS or FAIL, which is why the 3D path matters.
        +
        PASS only when the margin beats the error. FAIL only when it misses by more. Anything else is UNSURE, and a person or a better view settles it.
        -

        Unseen is not clear

        -

        The live map's haze works like the fog of war in a strategy game: a dark patch is unexplored, not empty. The comparison stops there. In a game, scouting reveals the truth. Here, even the parts the phone saw carry an error bar.

        +

        Unseen means unsure

        +

        Haze means unexplored, not empty, like fog of war in a strategy game.

        A check whose ground was never seen stays UNSURE and names the missing view @@ -713,15 +711,15 @@

        Unseen is not clear

        missing evidence: show the ground 6 ft right of your meter -
        If the camera never saw part of the area a rule depends on, that check cannot pass. It stays UNSURE, and the server names the view that would settle it. Here the ground beside the spot was never seen, and the checks that use ground need 6 to 13 ft of it. A blank on the map is never treated as empty ground.
        +
        A check on ground nobody saw stays UNSURE, and the app asks for that view.
        -

        Coverage means 100% of what the model needs

        -

        The app doesn't need the whole house. It needs every part the rules depend on, all of it. That is a fixed zone around the meter, so the app can say exactly what is left, and the scan can end when the zone is clear.

        +

        Coverage is a fixed zone

        +

        Not the whole house, just a zone around the meter. The app always knows exactly what's left.

        The zone a scan must cover, seen from above - A plan view of the house with the electric meter on its front wall. A dashed blue rectangle marks the zone the checks need: the wall 20 feet either side of the meter and the ground in front of it out to 13 feet. Everything outside it is grey haze labelled not needed. Inside, a bush stands in front of the wall; the strip of wall behind it is unseen, so checks there are UNSURE. A dotted path shows the phone walking along the zone. + A plan view of the house with the electric meter on its front wall. A dashed blue rectangle marks the zone the checks need: the wall 20 feet either side of the meter and the ground in front of it out to 13 feet. Everything outside it is grey haze labelled not needed. Inside, a bush stands in front of the wall, and the strip of wall behind it is unseen, so checks there are UNSURE. A dotted path shows the phone walking along the zone. @@ -753,15 +751,15 @@

        Coverage means 100% of what the model needs

        needed: all of it ground out to 13 ft -
        Drawn to scale, 1 ft to 8 px. The zone is the wall within 20 ft of cable either side of the meter, the ground in front out to 6 to 13 ft depending on the check, and the space overhead for headroom, which a plan view can't show. Near each end it reaches past 20 ft by the clearance around a spot at the edge of reach. A pass needs only the chosen spot and its clearances seen, and the server's answer says when the walk has gone far enough. Section 5 shows how the coverage map was tested against depth.
        +
        The zone: wall within 20 ft of cable each side, ground out to 6 to 13 ft, and headroom above. It reaches a little past 20 ft for clearances at the edge.
        -

        LiDAR first, never required

        -

        Some iPhones have LiDAR, a sensor that measures depth directly by timing reflected light. Most homeowners' phones don't. The app uses LiDAR and other depth sensors whenever the phone has them, and every check must also work without them.

        +

        LiDAR helps but isn't required

        +

        LiDAR measures depth directly, but most homeowners' phones don't have it. Every check must work without it.

        Two ways to get depth at real scale - Left, a phone with LiDAR sends rays to a wall seen from the side, and dots on the wall mark measured depth; its reach is about 5 metres. Right, any iPhone: a dashed line shows a depth model's guess of where the wall is, a little off. The phone at two known positions sees the same point on the real wall, and the two sight lines meet there, which fixes the scale. + Left, a phone with LiDAR sends rays to a wall seen from the side, and dots on the wall mark measured depth. Its reach is about 5 metres. Right, any iPhone: a dashed line shows a depth model's guess of where the wall is, a little off. The phone at two known positions sees the same point on the real wall, and the two sight lines meet there, which fixes the scale. Phone with LiDAR Any iPhone @@ -790,37 +788,36 @@

        LiDAR first, never required

        guessed, then rescaled by triangulating with poses -
        LiDAR works like a laser tape measure: direct distances, but a reach of about 5 m, so capture has to stand within about 4 m of the wall. Without it, a learned model estimates depth in each photo, and the phone's poses fix the scale: the same point seen from two known positions pins down its distance, the way two eyes judge depth. Where that comparison breaks: your eyes sit a fixed distance apart, but the phone's move between photos comes from its tracking and carries that error. Both paths fill the same capture packet.
        +
        With LiDAR, distances are measured directly, out to about 5 m. Without it, a model guesses depth and the phone's own movement fixes the scale, the way two eyes judge distance.
        -

        4. How to build the 3D model: the options

        -

        Problem 2, photos and metadata into a 3D model, is open. The server team is comparing three families of methods: broadly first, then in depth on the best. None is ruled out. The numbers so far come from public datasets whose buildings were laser-scanned, so the true shape is known to a fraction of an inch.

        +

        4. Building the 3D model

        +

        Still open. The server team is comparing three approaches on laser-scanned buildings, where the true shape is known.

        -

        Traditional reconstruction, fused with LiDAR

        -

        Geometry from first principles: find the same points in several photos, work out where they sit in 3D, and merge the depth from every view into one surface. LiDAR depth drops straight into that merge.

        -

        The known weakness is that matching points across photos struggles on blank siding and has no scale of its own. LiDAR and the phone's poses supply the scale.

        -

        Evidence: the recon worker (PR #20) merges depth into one surface. With a laser scan standing in for LiDAR, its walls come out at 2.1 in (p90).

        +

        Traditional reconstruction + LiDAR

        +

        Match points across photos, place them in 3D, and merge them into one surface. LiDAR depth drops straight in.

        +

        Weak on blank siding, and needs LiDAR or the phone's poses for scale.

        +

        Evidence: walls within 2.1 in (p90), with a laser scan standing in for LiDAR.

        -

        Learned depth plus the phone's poses

        -

        A neural network estimates depth from a single photo. Its own sense of scale is off by 4 to 12%, so the photos are rescaled with the phone's poses and then merged.

        -

        MapAnything, a learned model that reads several photos at once, was not a stable baseline: its scale shifted with the input resolution.

        -

        Evidence: walls about 5 in (p90) at a realistic phone error, 2.8 in with exact poses. Edges 8 in or worse in every setting.

        +

        Learned depth + phone poses

        +

        A neural network guesses depth in each photo. Its scale is 4 to 12% off, so the phone's poses rescale it.

        +

        Evidence: walls about 5 in (p90). Edges 8 in or worse.

        World models

        -

        Large learned models that take photos or video and produce a whole 3D scene at once, camera positions included. Think of someone who has seen thousands of houses sketching this one from a few snapshots.

        -

        The sketch comparison also shows the risk: a good sketch fills in what the artist never saw. That is fine for a picture, but a check needs to know which parts were observed.

        -

        Evidence: not evaluated yet. The server team will.

        +

        Big models that turn photos or video into a whole 3D scene.

        +

        They fill in what nobody saw, and a check must know what was observed.

        +

        Evidence: not tested yet.

        -

        Wall error at p90 for each 3D path tested so far, on one laser-scanned building

        +

        Wall error (p90) on one laser-scanned building

        Wall error at p90, in inches, for each 3D path tested so far A dot plot from 0 to 20 inches. Depth model alone: about 20 inches. Depth model rescaled with poses at a realistic 2 percent pose error: 5.0 inches, interval 3.8 to 6.8. With exact poses: 2.8 inches, interval 2.0 to 4.4. Edges with learned depth: 8 inches or worse in every setting. Recon worker, photos only: 2.8 inches. Recon worker with a laser scan standing in for LiDAR: 2.1 inches. World models: not evaluated yet. @@ -869,7 +866,7 @@

        World models

        World models: not evaluated yet -
        All rows come from the ETH3D electro building, walls within 6 m. The top four are learned depth (MoGe-2 and Depth Anything 3); the next two are the recon worker; the last has no data yet. Brackets are 95% intervals where they exist. Caveat: the recon worker's two runs chose different walls, so those two numbers are not measured on the same wall. Reading it: learned depth alone is too rough for clearances, which are measured from edges. With LiDAR or the phone's poses supplying scale, walls come within a few inches.
        +
        Learned depth alone is too rough for clearances. With LiDAR or the phone's poses for scale, walls land within a few inches. The two reconstruction runs measured different walls.
        Data table
        @@ -886,20 +883,20 @@

        World models

        -

        What any path has to show

        +

        What a path must deliver

          -
        • Real scale. A model's own sense of size is off by 4 to 12%, which is 5 to 14 in across a 10 ft stretch of wall.
          • -
        • Edges within the error allowance. Clearances run from the edge of a door, window or gas meter, and edges are where learned depth is worst.
          • -
        • Which parts were observed. The rules can pass a check only over area the phone saw. A method that fills in plausible wall has to mark what it filled in.
          • -
        • It runs on the packet the phone actually sends. That includes phones without LiDAR.
          • +
        • Real scale. 4 to 12% off is 5 to 14 in over a 10 ft wall.
          • +
        • Sharp edges. Clearances are measured from edges.
          • +
        • A record of what it saw. Filled-in wall must be marked.
          • +
        • It runs on the real packet, including phones without LiDAR.

        5. What the tests found

        -

        These ran on public real-world datasets with laser-measured ground truth. Dataset images never enter git. The 3D path results are in section 4. The panel-label test hasn't run: only 7 open photos of US panels exist, and it needs 40.

        +

        All results come from public datasets, not our phone. Panel-label reading isn't tested, because only 7 open photos of US panels exist and the test needs 40.

        -

        How far off is the phone's own tracking?

        +

        Tracking drift

        2018 iPhone 6s (ADVIO) @@ -932,7 +929,7 @@

        How far off is the phone's own tracking?

        30 ft walked: 2018 phone 132.6 in, 2023 phone 18.5 in, allowance 57.6 in -
        Position error after walking, p90 (9 in 10 cases are better). A current iPhone's own tracking stays inside the server's allowance at every distance; the 2018 phone is far over it. The allowance line is the 0.16 ft per foot part of the server's error bar. Caveats: the MARViN reference may itself be scaled to the phone's tracking, and that phone has LiDAR, which most homeowners' phones lack.
        +
        p90 position error. A current iPhone stays under the server's allowance, and a 2018 iPhone doesn't. That current phone has LiDAR, which most homeowners' phones lack, and its reference may be scaled to its own tracking.
        Data table
        PathSettingWall error, p90Source
        @@ -945,11 +942,11 @@

        How far off is the phone's own tracking?

        Walked2018 iPhone 6s (ADVIO), in2023 iPhone 14 Pro Max (MARViN), inServer's allowance, in
        -

        What it set: the server's allowance of 0.16 ft per foot walked comes from these walks.

        +

        Takeaway: the server's 0.16 ft-per-foot allowance comes from these walks.

        -

        Does the coverage map claim wall no photo saw?

        +

        False coverage

        wall a photo saw @@ -974,13 +971,13 @@

        Does the coverage map claim wall no photo saw?

        clean -
        Run on real photos of a laser-scanned building. The map claimed 1.1 ft of a 19.3 ft wall that no photo saw, hidden behind scaffold footings. Requiring views from different angles removed it only by also discarding 6 to 8.5 ft of good wall. A depth test, available on LiDAR phones, removed it cleanly. The hatched span is drawn at 8.5 ft, the larger end of the range.
        +
        The first map claimed 1.1 ft of wall that no photo saw. Demanding more angles threw away good wall too. A LiDAR depth test fixed it cleanly.
        -

        What it means: coverage has to come from what photos actually saw. LiDAR phones can run the depth test. Tested against depth, the reconstruction worker claims at most about 0.46 ft of unseen wall. Who checks a wall hidden behind something on phones without LiDAR is still open.

        +

        Takeaway: coverage must come from what photos really saw. Who checks hidden wall on phones without LiDAR is still open.

        -

        Can the phone read the meter?

        +

        Meter reading

        Meter reading success on real meter photos @@ -1002,16 +999,16 @@

        Can the phone read the meter?

        85% of held-out photos -
        Apple's on-device text recognition read the full meter number on 71 of 73 real meter photos, but could not tell which line on the plate was the number (right on 21 of 75). A barcode held the number on 20 of 24 photos that had one. A tap-to-confirm list of three candidates contained it on 85% of held-out photos.
        +
        The phone reads the number but usually picks the wrong line, because plates carry several numbers.
        -

        What it set: the app offers three candidates to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small.

        +

        Takeaway: the app offers three candidates to tap, barcode match first.

        -

        Still unmeasured: our own phone

        +

        Field test on our phone

        The field test: the phone's reading against a tape on the same wall - A wall with the electric meter. A yellow tape runs along the wall with marks at 5, 10, 15 and 20 feet. A phone stands in front. Blue dots show where the phone thinks the same marks are; the gap between dot and mark grows with distance. A label reads: the gap is the phone's scale error. + A wall with the electric meter. A yellow tape runs along the wall with marks at 5, 10, 15 and 20 feet. A phone stands in front. Blue dots show where the phone thinks the same marks are, and the gap between dot and mark grows with distance. A label reads: the gap is the phone's scale error. @@ -1033,41 +1030,40 @@

        Still unmeasured: our own phone

        a current iPhone, no LiDAR -
        Every number above comes from public datasets. This test measures our own phone on our own wall.
        +
        Everything above is public data. This test measures our phone on a real wall.
        -

        The field test needs a current iPhone without LiDAR, the Measure Lab build (PR #7), a 30 ft tape and a real meter wall. Follow experiments/evals/field/FIELD_SHEET.md (PR #12): type the tape readings into the prepared templates, and one command turns them into a score and the phone's real scale error. A dry run recovered a planted 1.5% error exactly, so the scoring end is ready.

        -

        The tracking numbers come from a phone with LiDAR, on a dataset whose reference may be scaled to that phone's own tracking. Most homeowners' phones lack LiDAR, so this test is the only direct measurement.

        +

        It needs a current iPhone without LiDAR, the Measure Lab build (PR #7), a 30 ft tape and a meter wall. The steps are in experiments/evals/field/FIELD_SHEET.md (PR #12).

        -

        6. Where each piece lives

        -

        Files marked with a pull request exist only on that branch until it merges. Each component's README wins over this page.

        +

        6. Where things live

        +

        Files marked with a PR live on that branch until it merges. A component's README wins over this page.

        - + - +
        PieceSource of truthState
        iPhone app
        client team        		ios/README.mdGuided capture in PR #10; live 3D map in PR #21, stacked on it
        iPhone app
        client team        		ios/README.mdGuided capture in PR #10, live 3D map in PR #21, stacked on it
        Capture packet
        client team        		packet/README.mdSpec, validator and samples in PR #22
        Rules engine, API, capture contract
        server team        		server/README.md, "What settles each check"PR #11; live as a public demo with public rules, and privately with Base's rules behind a key
        Rules engine, API, capture contract
        server team        		server/README.md, "What settles each check"PR #11, live as a public demo with public rules, and privately with Base's rules behind a key
        Reconstruction worker
        server team        		recon/HANDOFF.mdDraft PR #20, handed to the server team
        Accuracy evalsexperiments/evals/README.mdPR #12
        Agent rules, system overviewAGENTS.md, docs/00-overview.mdOn main
        Web review pageParked, PR #15
        -

        The plan, the decisions and the full evidence list are in 00-overview.md. The public rule values, their code citations and the model and imagery licenses are in 04-prior-art-and-codes.md.

        +

        Plan and evidence: 00-overview.md. Rule sources and licenses: 04-prior-art-and-codes.md.

        -

        How to try it

        +

        Try it

        Server health: house-scanning-server.vercel.app/health

        -

        App in the Simulator, with guided capture from PR #10: build ios/HouseScan.xcodeproj and run it with

        +

        Simulator, with guided capture from PR #10: build ios/HouseScan.xcodeproj and run it with

        -replay <folder>
 -autopilot
 -serverURL https://house-scanning-server.vercel.app

        On a real iPhone: set your team in ios/Config/Local.xcconfig (see ios/README.md), plug in, run.

        -
        Base's private thresholds, prompt names and field names do not appear on this page. The drawings are illustrations at a consistent scale, not a real house. Evidence numbers come from public datasets with laser-scanned ground truth; no dataset images are included.
        +
        The drawings are illustrations, not a real house.
        From beaf7c78fbb01cda3d429ed61978852dec592b50 Mon Sep 17 00:00:00 2001 From: Shrey Suri Date: Sat, 26 Sep 2026 15:09:09 -0500 Subject: [PATCH 14/16] docs: rewrite the writing skills with unslop Apply the unslop skill to technical-writing, pr-writing (with its two reference files) and prompt-writing. The rules stay the same. The prose now uses fewer abstract nouns, triplets and passive sentences, and has no semicolons or slashes. pr-writing now lists the template's Verification and Risk sections, and visual-evidence names `gh pr create --help`. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/pr-writing/SKILL.md | 24 ++-- .../pr-writing/references/visual-evidence.md | 20 +-- .../pr-writing/references/writing-examples.md | 28 ++-- .agents/skills/prompt-writing/SKILL.md | 36 +++--- .agents/skills/technical-writing/SKILL.md | 120 ++++++++++-------- 5 files changed, 118 insertions(+), 110 deletions(-) diff --git a/.agents/skills/pr-writing/SKILL.md b/.agents/skills/pr-writing/SKILL.md index d8f3b7829..222db5b2e 100644 --- a/.agents/skills/pr-writing/SKILL.md +++ b/.agents/skills/pr-writing/SKILL.md @@ -1,32 +1,32 @@ --- name: pr-writing -description: Write or revise pull request titles, descriptions and visual evidence. Use when preparing a PR, improving its presentation, or updating it after scope changes. Writing alone does not publish the PR. +description: Writes or revises pull request titles, descriptions and visual evidence. Use when preparing a PR, improving how it reads, or updating it after its scope changes. Writing the text does not publish the PR. --- # Write a reviewable PR -Read the repository's template and contributor guidance, the diff against the actual PR base, and the current PR body before drafting. Apply `unslop` and `technical-writing` when installed. Preserve useful human edits, existing evidence links and bot-managed sections. Refresh the body before publishing an edit so another author's changes are not overwritten. +Before drafting, read the repository's template and contributor guide, the diff against the PR's real base, and the current PR body. Apply `unslop` and `technical-writing` to everything you write. Keep what others added: human edits, evidence links and sections a bot manages. Fetch the body again right before you publish, so you don't overwrite someone else's change. ## Explain the change -Lead with the concrete problem and resulting behavior. Name the affected screen, operation or interface so someone outside the task can understand it. For a substantial change, explain the old behavior, its consequence, and how the change fixes it. For a new feature, describe the new capability without inventing a broken predecessor. +Open with the problem and what the code does now. Name the screen, operation or interface it touches, so someone who never saw the task can follow. For a large change, say what used to happen, what that cost, and how the change fixes it. A new feature needs no broken predecessor. Describe what it adds. -Order independent changes by user impact or technical consequence. Headings identify the change; filenames and commit order are not the outline. Explain the mechanism where it helps assess correctness or a tradeoff. Link the relevant diff when available, otherwise the file at the reviewed revision. Include prior research only when it supports a consequential choice, with a direct source and the reason it matters. +Order independent changes by how much they matter to users or to the rest of the code. Each heading names a change. Filenames and commit order are not an outline. Explain the mechanism when a reviewer needs it to judge correctness or a trade-off. Link the relevant part of the diff, or the file at the reviewed commit when no diff link exists. Cite earlier research only when a choice rests on it, with the source and why it matters. -Write connected prose with concrete subjects and verbs. Avoid clipped status phrases, unexplained internal names, claims of thoroughness and narration about the reviewer or your process. Keep useful technical detail; remove sentences that add no decision-relevant information. Read the short [writing examples](references/writing-examples.md) before drafting. Follow their level of explanation, not their sentence pattern or length. +Write whole sentences in which someone does something. Leave out clipped status phrases, internal names nobody explained, claims that the work was thorough, and narration about the reviewer or your process. Keep technical detail a reviewer can use, and cut any sentence that changes no decision. Read the short [writing examples](references/writing-examples.md) before drafting. Match how much they explain, not how their sentences are built or how long they are. -Scale the body to the change. A small repair needs a short explanation and focused verification. A broad PR can use sections by behavior with evidence beside each claim. Keep long logs, inventories and test matrices in linked artifacts. Do not copy a task transcript, delegate report or commit diary into the description. +Size the body to the change. A small fix gets a short explanation and the one check that proves it. A broad PR can use one section per behavior, with evidence beside each claim. Put long logs, inventories and test matrices in linked files. Never paste a task transcript, a subagent report or a commit-by-commit diary into the description. -## Present evidence +## Show evidence -For visible changes, read [visual evidence](references/visual-evidence.md) and include matched before/after captures from the running app. Add a short recording when motion, timing or interaction is the change. Show only affected behavior. A new screen can have an after-only capture; say it is new. Missing evidence stays a stated gap. +If the change is visible, read [visual evidence](references/visual-evidence.md) and include matched before-and-after captures from the running app. Add a short recording when the change is motion, timing or interaction. Show only the behavior that changed. A new screen gets an after capture alone, labeled as new. If evidence is missing, say so in the body. -Report checks actually run and their results, with commands or durable run links. State material limits next to the claims they qualify: fixtures, simulator-only behavior, a failed or skipped check, or an untested service. Keep reproduction steps separate from completed verification. Green CI does not establish visual quality or product acceptance. +Report the checks you ran and what they returned, with the command or a link to the run. Put each limit next to the claim it weakens: fixture data, simulator-only behavior, a check that failed or was skipped, a service nobody tested. Keep steps to reproduce apart from checks you finished. Green CI says nothing about how the screen looks or whether the product owner will accept it. ## Use the repository template -The repository's template, `.github/pull_request_template.md`, starts with What changed, Why, UI changes and four checklist items. Keep existing repository requirements. Add verification and risk where useful; remove empty optional sections and avoid repeating the same explanation under two headings. Replace inapplicable checklist items with a plain `N/A` note; check only completed work. Large cohesive changes are valid; do not call them small to satisfy a checkbox. +`.github/pull_request_template.md` has sections for What changed, Why, UI changes, Verification and Risk, then a four-item checklist. Keep every section the repository requires. Delete an optional section that would be empty, and don't explain the same thing under two headings. Where a checklist item doesn't apply, replace it with `N/A` and the reason. Tick only work you finished. A large change that belongs together is fine. Don't call it small to tick a box. -Write the title around the final change, using the repository's title convention. In stacked work, name the immediate base and necessary dependencies, and describe this PR's own contribution. When scope changes, update the title, body and affected evidence together. Keep still-valid captures with their revision labels instead of implying they show a newer build. +Write the title about the final change, in the repository's title format. In stacked work, name the base branch and any PRs this one needs, and describe only what this PR adds. When the scope changes, update the title, the body and the affected evidence in the same edit. Keep captures that still hold, labeled with the commit they show, so nobody mistakes them for the newer build. -Use a body file or structured tool argument when publishing. Reopen the saved PR to check its text, links, comparison layout and video playback. A local draft or successful upload alone does not establish that the published body renders correctly. Creating the PR, pushing and handling reviews are separate steps, each within the user's authorization. +Publish with a body file or a structured tool argument. Then open the saved PR and check that the text, links, side-by-side images and videos render. A local draft or a successful upload doesn't prove the published body looks right. Opening the PR, pushing and answering reviews are separate steps, and each needs the user's go-ahead. diff --git a/.agents/skills/pr-writing/references/visual-evidence.md b/.agents/skills/pr-writing/references/visual-evidence.md index bbb29f8a8..e2597ec60 100644 --- a/.agents/skills/pr-writing/references/visual-evidence.md +++ b/.agents/skills/pr-writing/references/visual-evidence.md @@ -1,27 +1,27 @@ # Visual evidence -Choose the changed behavior before collecting media. Capture the actual route or component through the host's supported browser, simulator or desktop tools. Do not redraw an interface to stand in for a screenshot. A design prototype is evidence of that prototype, not of the native or deployed product. +Decide which behavior changed before you capture anything. Capture the real route or component with the browser, simulator or desktop tools the host supports. Never redraw an interface as a stand-in for a screenshot. A design prototype shows the prototype, not the native app or the deployed product. ## Capture a fair comparison -Record the before and after revisions. Use the same viewport or device, route, data and relevant state. For a bug, reproduce the triggering condition on both versions. For a new feature, show the new flow and identify it as new. If the old build cannot run, label the missing or historical reference rather than fabricating a baseline. +Note the commit for the before and the after. Keep the device or viewport, route, data and state the same in both. For a bug, trigger the same condition on both versions. For a new feature, show the new flow and label it as new. If the old build won't run, label the missing or historical reference. Never fake a baseline. -Use isolated worktrees or existing builds so capturing the base does not disturb the active writer. Use approved test data. Identify fixture-driven behavior where it limits the claim; a visible success state does not prove a live backend write. Keep the capture procedure in the repository when it is reusable, including the route, fixture and build commands that actually worked. +Capture the base in a separate worktree or from an existing build, so you don't disturb whoever is working in the main checkout. Use approved test data. Say when fixture data limits a claim. A success screen doesn't prove the backend saved anything. If the capture procedure will be reused, commit it to the repository with the route, fixture and build commands that worked. -Place Before and After adjacent at readable, matching sizes. Label them and caption the behavioral difference. Crop to the relevant area consistently without hiding context needed to judge the change. Avoid tall comparison boards with large blank gaps or a gallery of unchanged screens. Preserve original captures when preparing a comparison. +Put Before and After side by side at the same readable size. Label each and caption what changed in behavior. Crop both the same way, and keep enough around the change to judge it. Avoid tall comparison boards with blank gaps, and galleries of screens that didn't change. Keep the original captures when you build a comparison. -Record motion at normal speed with the trigger, transition and settled result visible. Trim idle time without hiding failures or changing the timing being evaluated. Use H.264 MP4 when compatible with the host. Watch the encoded recording for dropped frames, legible text and correct orientation before uploading it. +Record motion at normal speed, showing the trigger, the transition and the settled result. Trim idle time, but never cut a failure or change the timing under review. Use H.264 MP4 when the host plays it. Before uploading, watch the encoded file for dropped frames, unreadable text and wrong orientation. ## Publish and verify -Use GitHub repository attachments for inline evidence, especially in private repositories. Keep sensitive captures off anonymous public image hosts. A private raw-file URL or a local filesystem path is not a reliable inline attachment. +Attach inline evidence through GitHub's repository attachments, especially in a private repository. Keep sensitive captures off anonymous public image hosts. A private raw-file URL or a local path won't render inline. -Prefer the installed GitHub CLI's documented attachment support when available; check command help before relying on `--attach`. Otherwise use the authenticated GitHub attachment UI or a repository's verified uploader. Do not copy an app-specific authentication workaround into another project. For a local before/after composition tool, verify its output mode before running it so capture files are not uploaded elsewhere. +Use the GitHub CLI's attachment support if the installed version has it. Check `gh pr create --help` for `--attach` first. Otherwise use GitHub's attachment UI while signed in, or an uploader the repository already trusts. Don't carry one app's authentication workaround into another project. If a local tool composes the before-and-after image, check where it writes its output before you run it, so it doesn't upload captures somewhere else. -With CLI attachment support, reference the local files in the Markdown body and pass each file with `--attach` alongside `--body-file`. The CLI replaces those references with uploaded URLs. If an upload partially fails, inspect the returned PR and its body before retrying: creation can succeed even when the command exits nonzero. Repair the existing PR rather than creating another. +With CLI attachment support, reference the local files in the Markdown body and pass each file with `--attach` next to `--body-file`. The CLI swaps those references for uploaded URLs. If an upload fails partway, look at the returned PR and its body before retrying. The CLI can exit nonzero after it has already created the PR. Fix that PR instead of opening another. -Use a compact Markdown table for a pair. Give images descriptive alt text. Put an interaction video in its own paragraph so it can render as a player. Preserve the original attachment URLs when editing an existing body. +Show a pair in a small Markdown table. Give each image alt text that says what it shows. Put a video in its own paragraph so GitHub renders it as a player. Keep the original attachment URLs when you edit an existing body. -Reopen the PR in its actual rendered view. Check that both images load at useful sizes, labels match their revisions, and the video plays. If publication or viewing is unavailable, return the draft and evidence paths with that specific remaining step. +Open the PR's rendered page. Check that both images load at a useful size, each label matches its commit, and the video plays. If you can't publish or view the PR, hand back the draft and the evidence paths, and name the step still left. GitHub documents [attachment access and supported formats](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/attaching-files) and [CLI attachment support](https://docs.github.com/en/github-cli/github-cli/attaching-files-with-github-cli). diff --git a/.agents/skills/pr-writing/references/writing-examples.md b/.agents/skills/pr-writing/references/writing-examples.md index dde8a413d..4d0feb443 100644 --- a/.agents/skills/pr-writing/references/writing-examples.md +++ b/.agents/skills/pr-writing/references/writing-examples.md @@ -1,31 +1,31 @@ # Writing examples -These invented examples illustrate the level of explanation. They are not facts to reuse in a PR. +These examples are made up. They show how much to explain, and none of their facts belong in a real PR. -## Explain the failure and mechanism +## Explain the failure and the mechanism Weak: "Improved session handling. The stale-response guard is robust and all tests pass." -Useful: "Switching accounts while the inbox loaded could show the previous account's messages. The response now carries the account that requested it, and the inbox discards it if that account is no longer active. The regression test starts both loads and completes the older request last." +Useful: "Switching accounts while the inbox loaded could show the previous account's messages. The response now carries the account that requested it, and the inbox drops it if that account is no longer active. The regression test starts both loads and finishes the older request last." -The useful version names the trigger, consequence and mechanism. A test count alone would not explain why the race is covered. +The useful version names the trigger, what went wrong and how the fix works. A test count alone wouldn't show that the test covers the race. -## Make a small change small to read +## Keep a small change short -Title: "fix(export): preserve zero values in CSV downloads" +Title: "fix(export): keep zero values in CSV downloads" -"CSV downloads left zero-valued cells empty because the formatter treated every falsy value as missing. It now leaves a cell empty only for null or undefined. `npm run test:csv` passes, including zero, null and undefined cases." +"CSV downloads left zero-valued cells empty, because the formatter treated every falsy value as missing. It now leaves a cell empty only for null or undefined. `npm run test:csv` passes, including zero, null and undefined cases." -No architecture narrative or screenshot is needed for this change. Follow required repository sections without padding them. +This change needs no architecture story and no screenshot. Fill in the sections the repository requires, and stop there. -## Describe what a comparison establishes +## Say what a comparison shows -"The payment error used to replace the form and discard the entered address. It now appears above the submit button, leaving the address available for a retry." +"A declined payment used to replace the form and throw away the address the customer typed. The error now appears above the submit button, and the address stays filled in for a retry." -Place the matched images immediately below that explanation. A caption such as "Same declined-payment fixture at 390 × 844; before a12bc34, after d56ef78" establishes the comparison. If no live charge ran, say so once beside the verification. Do not call the images proof that payments work end to end. +Put the matched images right below that paragraph. A caption such as "Same declined-payment fixture at 390 × 844. Before a12bc34, after d56ef78." tells the reviewer the comparison is fair. If no live charge ran, say so once, next to the verification. The images don't prove that payments work end to end, so don't present them that way. -## Preserve a useful limit +## Keep a limit the reviewer needs -"The migration was exercised on a copy of the staging schema. Production data volume and lock duration have not been measured." +"The migration ran on a copy of the staging schema. Nobody has measured it against production data volume or timed how long it holds locks." -This gives the reviewer a concrete remaining concern. "Low risk, thoroughly tested" hides it. A long chronology of failed approaches would bury it. +That hands the reviewer a concrete open risk. "Low risk, thoroughly tested" hides it, and a long history of approaches that failed would bury it. diff --git a/.agents/skills/prompt-writing/SKILL.md b/.agents/skills/prompt-writing/SKILL.md index c93782df3..5cb82b4bc 100644 --- a/.agents/skills/prompt-writing/SKILL.md +++ b/.agents/skills/prompt-writing/SKILL.md @@ -1,6 +1,6 @@ --- name: prompt-writing -description: Writes and reviews instructions for AI agents, such as AGENTS.md lines, SKILL.md files, and prompts that hand work to another agent or subagent. Use when writing or editing AGENTS.md, CLAUDE.md or a skill, or when briefing another agent. +description: Writes and reviews instructions for AI agents, such as AGENTS.md lines, SKILL.md files and prompts that hand work to another agent or subagent. Use when writing or editing AGENTS.md, CLAUDE.md or a skill, or when briefing another agent. --- # Prompt writing @@ -9,39 +9,39 @@ The reader is a capable model that can read the repository. Write only what it c ## Give the outcome, the reason and the finish -- State the outcome, the scope and what done looks like. A clear done condition keeps the agent from stopping early. +- State the outcome, the scope and what done looks like. An agent with a clear done condition doesn't stop early. - Give the reason behind each constraint. A model that knows why a rule exists applies it to cases the rule didn't name. -- Point exactly: the path, the symbol, the command, the README section. Whether the agent follows a pointer depends on its wording. +- Point at the exact path, symbol, command or README section. An agent follows "read `server/README.md`, section 'What settles each check'" and skims past "see the server docs". - Say what to do rather than what to avoid. -- Hand over what the agent can't discover: decisions already made and why, what was tried and failed, who owns which files, and what the user prefers. +- Hand over what the agent can't find in the repository: decisions already made and why, what was tried and failed, who owns which files, and what the user prefers. ## Keep instructions calm and consistent -- Write in a normal voice. Current Claude models follow instructions closely, and capitals, "CRITICAL" or "if in doubt, always" make them overapply a rule. -- Put each instruction in one place. Conflicting guidance across a skill, AGENTS.md and the prompt can make a model stop and block early. Where instructions can conflict, state the priority: the user's instructions win over a skill. -- Write the prompt in the form you want back. A prompt in prose tends to get prose, and one built from headings and bullets tends to get markdown. +- Write in a normal voice. Current Claude models follow instructions closely, and capitals, "CRITICAL" or "if in doubt, always" make them apply a rule too widely. +- Put each instruction in one place. When a skill, AGENTS.md and the prompt disagree, a model can stop and wait for help. Where instructions can conflict, say which wins. The user's instructions outrank a skill. +- Write the prompt in the form you want back. A prompt in prose tends to get prose back, and one built from headings and bullets tends to get markdown. - Call each thing by one name throughout. ## Keep context small -- Every line of AGENTS.md or CLAUDE.md loads into every session. Put material needed in one task out of ten behind a pointer to the file that holds it. -- Link every reference file directly from the entry file. A file reached through another reference may be read only in part. -- Leave out facts that expire, or put them where the current value lives and point there. +- Every line of AGENTS.md or CLAUDE.md loads into every session. If only one task in ten needs some material, move it to its own file and leave a pointer. +- Link every reference file directly from the entry file. An agent may read only part of a file it reached through another reference. +- Leave out facts that expire. If a fact must stay, point to the file that holds its current value. ## Prune by behavior -- A line is a no-op if the agent acts the same without it. Delete the whole sentence, not a few of its words. -- Don't ask an agent to "shorten" or "streamline" a prompt. It optimizes for length and cuts instructions that mattered. -- Test by running it. Give a fresh agent a real task with only this prompt, watch where it goes wrong, and fix that failure rather than one you imagine. +- A line does nothing if the agent acts the same without it. Delete the whole sentence, not a few of its words. +- Don't ask an agent to "shorten" or "streamline" a prompt. It will cut for length and drop instructions that mattered. +- Test the prompt by running it. Give a fresh agent a real task with only this prompt, watch where it goes wrong, and fix that failure rather than one you imagine. -## Writing a skill +## Write a skill -- Write the description in the third person. Say what the skill does and when to use it, with the words a user would say. The description is all that loads before the skill is chosen. -- Keep SKILL.md under 500 lines. Move detail that only one branch of the task needs into a file SKILL.md links directly. +- Write the description in the third person. Say what the skill does and when to use it, in the words a user would type. The description is all the agent sees before it picks the skill. +- Keep SKILL.md under 500 lines. Move detail that only one branch of the task needs into a file that SKILL.md links directly. ## Sources -- Anthropic, [Prompting best practices](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices): give context and motivation, say what to do instead of what not to do, match prompt style to output, and dial back aggressive language on current models. -- Anthropic, [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices): assume the model is capable, third-person descriptions, references one level deep, no time-sensitive facts, test with real tasks. +- Anthropic, [Prompting best practices](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices): give context and motivation, say what to do instead of what not to do, match prompt style to output, and tone down aggressive language on current models. +- Anthropic, [Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices): assume the model is capable, write third-person descriptions, keep references one level deep, leave out time-sensitive facts, and test with real tasks. - OpenAI, [Prompting the latest model](https://developers.openai.com/api/docs/guides/latest-model): conflicting skill guidance can block work early, and the user's instructions should outrank a skill. - Matt Pocock, [Writing skills for agents](https://aihero.dev/skills-writing-for-agents): pointer wording, done conditions, the no-op test, and testing by running the document. diff --git a/.agents/skills/technical-writing/SKILL.md b/.agents/skills/technical-writing/SKILL.md index ae1208d4b..cd08a9e76 100644 --- a/.agents/skills/technical-writing/SKILL.md +++ b/.agents/skills/technical-writing/SKILL.md @@ -1,107 +1,107 @@ --- name: technical-writing -description: "Layered technical-writing standard: Diátaxis structure, Google developer style sentences, STE instruction rules, Global English syntax. Use for /technical-writing or when writing or reviewing docs, RFCs, readmes, PR descriptions, or commit messages." +description: "Checks and writes technical prose against four standards: Diátaxis for document structure, Google developer style for sentences, STE for instructions, and Global English for syntax. Use for /technical-writing or when writing or reviewing docs, RFCs, READMEs, PR descriptions, or commit messages." --- # Technical writing -The goal is writing a tired engineer understands on the first read. Four layers get you there, one question each: what kind of document is this, how do sentences address the reader, how much does each sentence carry, and can any sentence be read two ways. Apply all four. +Write so a tired engineer understands it on the first read. Four layers get you there, and each answers one question. What kind of document is this? How do sentences address the reader? How much does each sentence carry? Can any sentence be read two ways? Apply all four. Three rules sit above the layers: - **Cut every word that does no work.** If the sentence survives without a word, the word goes. "In order to" is "to". "It is important to note that" is nothing. -- **Use the short, everyday word.** "Use", not "utilize". "Help", not "facilitate". "Do", not "perform". A long word has to buy its length with precision. -- **When a rule makes a sentence worse, fix the sentence another way or leave it alone.** The rules serve the reader. A sentence that follows every rule and sounds like a machine wrote it has failed. +- **Use the short, everyday word.** "Use", not "utilize". "Help", not "facilitate". "Do", not "perform". A long word has to earn its length with precision. +- **When a rule makes a sentence worse, fix the sentence another way or leave it alone.** The rules serve the reader. A sentence that follows every rule and still sounds machine-written has failed. The codebase is the word list. Write the real symbol, file, flag, or command name, not a synonym or a description of it. -Don't invent jargon. Use the words a developer would say out loud: "move", "delete", "a budget that only decreases", not "evacuate", "ratchet", or "endgame". A named pattern is fine when the doc says what it means the first time. Propose new offenders and their replacements for `unslop`'s abstract-metaphor rule in a pull request. +Don't invent jargon. Use the words a developer says out loud, such as "move" and "a budget that only decreases" instead of "evacuate" and "ratchet". A named pattern is fine when the doc defines it the first time it appears. To add a word to `unslop`'s abstract-metaphor rule, propose it with its replacement in a pull request. ## Vary the rhythm -The layers decide what a document says and how much each sentence carries. A doc can obey all of them and still read machine-written: every sentence clipped short, no view anywhere, nothing specific. +The layers decide what a document says and how much each sentence carries. A doc can obey all of them and still read as machine-written, with every sentence clipped short, no view anywhere, and nothing specific. -- Mix sentence lengths on purpose. Short sentences land a point. Longer ones that take their time carry a fact with its condition or consequence. -- One thought per sentence does not mean one length per sentence. Split the sentence that carries two thoughts. Keep the long sentence that carries one. -- Have a view where the mode allows it. Explanation weighs trade-offs, so say what you make of them instead of listing pros and cons. Reference stays dry. -- Be specific over sterile. Not "schema changes can cause issues" but "a column rename fails the build". +- Mix sentence lengths on purpose. A short sentence lands a point. A longer one can carry a fact together with its condition or its consequence. +- One thought per sentence does not mean one length per sentence. Split a sentence that carries two thoughts, and keep a long sentence that carries one. +- Take a position where the mode allows it. Explanation weighs trade-offs, so say which one you would pick and why instead of listing pros and cons. Reference stays dry. +- Be specific. "Schema changes can cause issues" tells the reader nothing. "A column rename fails the build" tells them what to watch for. ## Pick the mode first (Diátaxis) -One document, one mode. Two questions pick it: does the content inform action (doing) or understanding (thinking), and does it serve learning or work? +One document, one mode. Two questions pick it. Does the content help the reader act or understand? Does it serve someone learning or someone working? -- Action + learning: **tutorial**. -- Action + work: **how-to**. -- Understanding + work: **reference**. -- Understanding + learning: **explanation**. +- Act while learning: **tutorial**. +- Act while working: **how-to**. +- Understand while working: **reference**. +- Understand while learning: **explanation**. -Use the compass on a whole document or on one sentence. Reach for it whenever you feel unsure what you are writing. Gut feel is often wrong here. +Use these two questions on a whole document or on one sentence, whenever you are unsure what you are writing. Gut feel often picks the wrong mode. -**Tutorial: learning by doing.** You are the teacher. The learner's success is your job, not theirs. Open by saying what the learner will build, not what they will "learn". Every step produces a visible result, early and often. Tell them what they should see: the expected output, the prompt change, the log line. Cut explanation to one clause and a link. Teaching pauses break the lesson. Stay concrete. Write as "we", in commands: "First, do x. Now, do y." +**A tutorial teaches by doing.** You are the teacher, and the learner's success is your job. Open with what the learner will build, not what they will "learn". Every step produces a visible result, early and often. Tell them what they should see, such as the expected output, the prompt change, or the log line. Keep explanation to one clause and a link, because a teaching pause breaks the lesson. Stay concrete, and write as "we", in commands: "First, do x. Now, do y." -**How-to: steps to a goal.** Solve a problem a person has, not an operation the machine can perform. Assume competence. Skip teaching. Action only: no digressions, no background, no completeness for its own sake. Link those instead. Allow forks and judgment: "If you want x, do y." Name the guide by the task: "How to calibrate the radar array", not "Radar array calibration". +**A how-to gets a competent reader to a goal.** It solves a problem a person has, not an operation the machine can perform. Skip teaching and background, and link to them instead. Allow forks and judgment: "If you want x, do y." Name the guide by the task, as in "How to calibrate the radar array", not "Radar array calibration". -**Reference: facts for lookup.** Describe. Only describe. No instruction, no persuasion, no opinion. Be dry, complete, and sure: state facts, options, limits, and errors with no hedging. Mirror the structure of the thing described, so code and docs can be navigated together. Put material where readers expect it. Generate from code where possible, so it stays true. +**Reference holds facts for lookup.** It describes and does nothing else, with no instructions and no opinions. State facts, options, limits, and errors plainly, without hedging, and leave nothing out. Mirror the structure of the thing described, so a reader can move between the code and the docs. Put material where readers expect it. Generate reference from code where you can, so it stays true. -**Explanation: understanding and why.** One bounded topic, readable away from the product. Each title should tolerate an implicit "About..." in front. Anchor on a real why question. Give context: design decisions, history, constraints, alternatives. Opinion is allowed here and nowhere else. +**Explanation builds understanding of why.** It covers one bounded topic and makes sense away from the product. Each title should still read well with "About" in front of it. Anchor on a real why question, and give the context behind it: design decisions, history, constraints, and alternatives. Explanation is the only mode that allows opinion. -Don't mix modes: no reference tables inside a tutorial, no tutorial hand-holding inside reference, no arguing inside a how-to. Split and link instead. +Keep modes apart. A tutorial has no reference tables, reference has no hand-holding, and a how-to has no argument. Split the content into separate documents and link them. Source: diataxis.fr, fetched 2026-07-18. ## Write sentences to the reader (Google developer style) -- Talk to the reader as "you", in the present tense. "Will" only for things that genuinely happen later. +- Address the reader as "you", in the present tense. Use "will" only for things that happen later. - Say who does what: "the compiler checks", not "is checked". Passive is fine only when the actor is unknown or beside the point. -- Write instructions as commands: "Click Submit." State facts plainly. Never "should be done". -- Put the condition before the instruction: "To delete the document, click Delete." The reader skips what does not apply. -- Put the common case first. Exceptions after. -- Sound like a knowledgeable friend. No buzzwords, no figurative language, no "please" in instructions, and never "simply", "easy", or "quickly" in a procedure. If it were simple, the reader would not be here. -- Don't pre-announce ("we will soon support...") and don't start consecutive sentences with the same phrase. -- Read the awkward sentence aloud. If it stays awkward, rewrite it. -- Link with words that say where the link goes: the page title or a short description. Never "click here". Prefer a sentence of context on the page over a link off it. -- Headings carry the point, not just the topic ("Pick the mode first", not "Modes"). Sentence case. A task heading is a bare verb phrase ("Create an instance"). A concept heading is a noun phrase. One h1 per page, no skipped levels. -- Numbered lists for sequences, bullets for everything else. Introduce a list with a complete sentence. Keep items parallel. -- Code goes in code font. UI elements go in bold. Use serial commas. Drop "etc." and say up front that a list is partial. +- Write instructions as commands: "Click Submit." State facts plainly. Never write "should be done". +- Put the condition before the instruction: "To delete the document, click Delete." The reader then skips what doesn't apply. +- Put the common case first and exceptions after it. +- Sound like a friend who knows the system. Leave out buzzwords, figures of speech, and "please" in instructions. Never write "simply", "easy", or "quickly" in a procedure. If it were simple, the reader would not be here. +- Don't announce future plans ("we will soon support..."), and don't start consecutive sentences with the same phrase. +- Read an awkward sentence aloud. If it stays awkward, rewrite it. +- Link text says where the link goes, using the page title or a short description. Never write "click here". A sentence of context on the page beats a link off it. +- A heading states the point, not just the topic: "Pick the mode first", not "Modes". Use sentence case. A task heading is a bare verb phrase ("Create an instance"). A concept heading is a noun phrase. Use one h1 per page and skip no levels. +- Use numbered lists for sequences and bullets for everything else. Introduce a list with a complete sentence, and keep its items parallel. +- Put code in code font and UI elements in bold. Use serial commas. Drop "etc.", and say up front when a list is partial. Source: developers.google.com/style, fetched 2026-07-18. ## Make statements load one at a time (STE rules) -- One instruction per sentence. One thought per sentence everywhere else. -- Split instructions longer than about 20 words and other sentences longer than about 25. +- Give one instruction per sentence, and one thought per sentence everywhere else. +- Split an instruction longer than about 20 words, and any other sentence longer than about 25. - Put the warning or condition before the step it guards: "If hot oil touches your skin, injuries can occur." -- Keep "the" and "a": "Remove backup file" reads two ways. "Remove the backup file" reads one. -- Give each word one meaning and one job, then keep it. If "check" means inspect, don't also use it for restrain. -- Pick one word per action and stick to it: "start", not "start" here and "initiate" there. +- Keep "the" and "a". "Remove backup file" reads two ways. "Remove the backup file" reads one way. +- Give each word one meaning, and keep it. If "check" means inspect, don't also use it to mean restrain. +- Pick one word per action and stick to it. Write "start" every time, not "start" here and "initiate" there. - Write procedures as direct commands, never as narration and never in the passive: "Install the component", not "the component must be installed". -- Avoid "-ing" words where you can. They take too many grammatical jobs and breed misreadings. +- Avoid "-ing" words where you can. They can do several grammatical jobs, so they invite misreadings. -Source: asd-ste100.org (Issue 9, 2025), fetched 2026-07-18. The numbered rules and dictionary live in the spec PDF. The principles above are the transferable core. +Source: asd-ste100.org (Issue 9, 2025), fetched 2026-07-18. The numbered rules and the dictionary are in the spec PDF. The principles above are the parts that carry over to software docs. ## Leave no sentence open to two readings (Global English) -- Keep words like "only" and "not" next to the word they change: "only fails on growth" and "fails only on growth" say different things. -- Break up long noun strings: "the proto import budget check script" becomes "the script that checks the proto-import budget". +- Keep words like "only" and "not" next to the word they change. "Only fails on growth" and "fails only on growth" say different things. +- Break up long noun strings. "The proto import budget check script" becomes "the script that checks the proto-import budget". - Make every "it", "they", and "this" point at one obvious thing. Repeat the noun when in doubt. Never use "this" or "which" to point at a whole clause. -- Don't drop verbs: "Phase 1 moves the converters and Phase 2 the runtime" leaves Phase 2 without one. Give it one. +- Don't drop verbs. "Phase 1 moves the converters and Phase 2 the runtime" leaves Phase 2 without one, so give it one. - Keep the small words that show structure. "Ensure that the switch is off" keeps "that" because it makes the sentence parse one way. Never trade clarity for word count. -- Repeat the article in a series when it prevents a misread: "the client and the host", not "the client and host", when they are two things. -- Say which parts "and" or "or" joins when a sentence can group two ways. "Both...and", "either...or", and "if...then" are free disambiguators. +- Repeat the article in a series when two things could read as one: "the client and the host", not "the client and host". +- When a sentence can group two ways, show which parts "and" or "or" joins. "Both...and", "either...or", and "if...then" cost nothing and settle it. - Use periods, not semicolons. Replace an em dash with a new sentence. -- Make text in parentheses a full grammatical unit or its own sentence. Never form plurals with "(s)". -- No slashes: write "a, b, or both" instead of "a/b" or "and/or". -- Call each thing by one name, everywhere. A doc that says "the gate", "the ratchet", and "the budget check" for one thing teaches three things. Rewording an unchanged sentence between edits costs the same way: don't churn what didn't change. -- Skip idioms, colloquialisms, Latin abbreviations, and metaphors. A non-native reader, a translator, and an agent all parse plain constructions best. +- Text in parentheses must be a full grammatical unit, or its own sentence. Never form plurals with "(s)". +- Don't use slashes. Write "a, b, or both" instead of "a/b" or "and/or". +- Call each thing by one name, everywhere. A doc that calls one thing "the gate", "the ratchet", and "the budget check" teaches three things. Rewording a sentence that didn't change costs readers the same way, so leave unchanged sentences alone. +- Skip idioms, slang, Latin abbreviations, and metaphors. A non-native reader, a translator, and an agent all parse plain sentences best. Source: Kohl, The Global English Style Guide (SAS Press). Guideline text fetched from the Internet Archive and the SAS sample chapter, 2026-07-18. -## Voice and repo specifics +## Apply unslop and the repository rules -- Apply the **unslop** skill to every doc this skill touches. That skill owns the slop-pattern catalog: AI vocabulary, filler, hedging, formatting tells. +- Apply the `unslop` skill to every doc this skill touches. `unslop` owns the catalog of slop patterns, such as AI vocabulary, filler, hedging, and formatting tells. - PR descriptions and commit messages are writing too. Every layer except Diátaxis applies to them. -- Product UI strings are not documentation. Use your product's copy guidelines for those. -- Write real paths and real symbols. Make every count or tree claim true at the commit that lands it, and include the command that regenerates it. +- Product UI strings are not documentation. Follow the product's copy guidelines for those. +- Write real paths and real symbols. Make every count or file-tree claim true at the commit that lands it, and include the command that regenerates it. ## Worked example @@ -113,16 +113,24 @@ After: > `budget.mjs` reads the committed budget from `budget.json` and counts the files that import protos. If the count exceeds the budget, CI fails. Run `budget.mjs --write` only to lower the budget. -The fixes, by layer: "configuration is performed" becomes "`budget.mjs` reads", so someone does something (Google). "Ratchet" goes away. The script's real filename does the naming (jargon rule). The five-noun string breaks up into plain clauses (Global English). The hedge "note that it's important to remember" is deleted (cut every word that does no work). The failure condition moves ahead of the step it explains (STE). The buried "should only be done when lowering" becomes a command with "only" next to its verb (STE). "If exceeded" gets a subject: the count (Global English). +Each fix comes from a layer: + +- "Configuration is performed" becomes "`budget.mjs` reads", so someone does something (Google). +- "Ratchet" goes, and the script's real filename names it instead (the jargon rule). +- The five-noun string breaks up into plain clauses (Global English). +- "Note that it's important to remember" is deleted (cut every word that does no work). +- The failure condition moves ahead of the step it explains (STE). +- "Should only be done when lowering" becomes a command with "only" next to what it limits (STE). +- "If exceeded" gets a subject, the count (Global English). ## Review checklist -Apply to any prose this skill covers. Item 1 applies only to document sets: +Run these checks on any prose this skill covers. Item 1 applies only to a set of documents. 1. Is each file one Diátaxis mode, with links where modes meet? -2. Is every instruction written as a command, with its condition in front? +2. Is every instruction a command, with its condition in front? 3. Does any sentence carry two instructions or two thoughts? Split it. -4. Can any word be cut without losing meaning? Cut it. +4. Can any word go without losing meaning? Cut it. 5. Is "only" next to the word it changes? Does every "it" point at one thing? Does every clause keep its verb? 6. Does each thing have exactly one name across the docs? 7. Would a developer say these words out loud? Replace invented metaphors and fancy synonyms with the plain word or the real symbol name. From 0bda7927ae03dbef4184d403d43b7cc06e12c7d0 Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 15:23:13 -0500 Subject: [PATCH 15/16] docs: mark the upload flow as PR #10 and planned work, and caption every scan step - docs/00 says main's app only shows AR tracking, that PR #10 uploads scene.json, and that the server-side photo flow is planned. - docs/00 notes that classified mesh also needs sceneReconstruction = .meshWithClassification on LiDAR phones. - The three number-only scan captions get short text. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/00-overview.md | 6 +++--- docs/how-it-works.html | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/00-overview.md b/docs/00-overview.md index 023d440ad..f84703e34 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -10,9 +10,9 @@ Full-house scans, roofs and Android are out of scope. Each component's README is The work is three problems. The phone guides the homeowner until it has captured everything the rules need. The photos and their metadata become a 3D model. Plain code checks that model against the placement rules, and the answer comes back to the phone. -**Client team: Sam with AI agents, and Aiden on video and sample datasets.** The app in `ios/` builds a live 3D map on the phone. A haze, the "fog of war", covers what it hasn't seen and lifts as the homeowner scans. The app guides them until everything the rules need is covered, which is a zone around the meter rather than the whole house. Guided capture is in PR #10 and the live map in PR #21. The app's output is the capture packet: photos, camera poses, intrinsics, motion-sensor readings, LiDAR depth and mesh when the phone has LiDAR, and the homeowner's marks. `packet/README.md` (PR #22) specifies it and lists what the app records today. For now the app uploads only its measurements, `scene.json`, and exports the full packet on request. +**Client team: Sam with AI agents, and Aiden on video and sample datasets.** The app in `ios/` builds a live 3D map on the phone. A haze, the "fog of war", covers what it hasn't seen and lifts as the homeowner scans. The app guides them until everything the rules need is covered, which is a zone around the meter rather than the whole house. Guided capture is in PR #10 and the live map in PR #21. The app's output is the capture packet: photos, camera poses, intrinsics, motion-sensor readings, LiDAR depth and mesh when the phone has LiDAR, and the homeowner's marks. `packet/README.md` (PR #22) specifies it and lists what the app records today. On `main` the app only shows AR tracking. With PR #10 it uploads only its measurements, `scene.json`, and exports the full packet on request. -**Server team: Hunter with his agents.** The packet goes in, and a 3D model comes out at real scale with a record of which parts were observed. The rules engine in `server/` (PR #11) then checks every candidate spot. The team is comparing reconstruction methods broadly first, then the best in depth. Hunter's research handoff (PR #18) sets the shape of the server side: +**Server team: Hunter with his agents.** The packet goes in, and a 3D model comes out at real scale with a record of which parts were observed. The rules engine in `server/` (PR #11) then checks every candidate spot. The team is comparing reconstruction methods broadly first, then the best in depth. Hunter's research handoff (PR #18) plans the server side: - Original photos travel to object storage, separate from the compact placement request, because a Vercel function accepts at most 4.5 MB. - Recognition and reconstruction run asynchronously while the phone keeps guiding, and capture, upload, analysis and assessment finish at different times. - A new reconstruction can revise earlier geometry. Coverage and measurements that depend on it get recomputed, never carried over. @@ -38,7 +38,7 @@ These are the non-obvious facts behind numbers in the code, and where each numbe - Depth maps are 256 × 192. Scale the intrinsics down to match. - ARKit camera space is x right, y up, looking down −z, with image v growing downward. The ray through pixel (u, v) points along ((u − cx)/fx, −(v − cy)/fy, −1) before the camera-to-world rotation. - Keyframes are kept every 0.5 m or 15° of movement. -- Mesh vertices are local to their `ARMeshAnchor`. Copy each one out of ARKit's buffer, then multiply the copy by the anchor's transform, leaving the buffer untouched. RealityKit turns mesh classification off unless `automaticallyConfigureSession` is `false`. +- Mesh vertices are local to their `ARMeshAnchor`. Copy each one out of ARKit's buffer, then multiply the copy by the anchor's transform, leaving the buffer untouched. Classified mesh needs `sceneReconstruction = .meshWithClassification` on the world-tracking configuration, on LiDAR phones only. RealityKit also turns classification off unless `automaticallyConfigureSession` is `false`. - On the mesh, the gap to a facing fence is a ray straight out from the wall at 1.5 ft high. Headroom is a ray straight up from 1 ft out. The mesh has holes, so each measurement is the smallest distance over a small fan of rays. - LiDAR reads from about 0.5 m to 5 m, so capture stands 1 to 3 m from the wall. A ray that meets nothing within 5 m is unknown, not open. - The default error bars are day-1 estimates, untested: 0.3 ft for an AR tap, 0.5 ft for the LiDAR mesh and 1.5 ft for a position from photo detection. The server adds 0.16 ft per foot along the wall from the meter, from measured ARKit drift. diff --git a/docs/how-it-works.html b/docs/how-it-works.html index d654f17a1..28927613e 100644 --- a/docs/how-it-works.html +++ b/docs/how-it-works.html @@ -454,7 +454,7 @@

        2. From scan to answer

        The guided scan box from stage 1, with an arrow to a file icon labelled packet, sitting on the line between the teams. -
        It sends a capture packet of photos, camera poses, lens data, motion data, LiDAR depth when the phone has it, and the homeowner's marks. Today the app uploads only its measurements and exports the full packet on request.
        +
        It sends a capture packet of photos, camera poses, lens data, motion data, LiDAR depth when the phone has it, and the homeowner's marks. With PR #10 the app uploads only its measurements and exports the full packet on request.
      • @@ -500,7 +500,7 @@

        The scan, step by step

        tap the electric meter it becomes 0 ft, all else is left or right of it -
        +
        Tap the meter.
      • @@ -516,7 +516,7 @@

        The scan, step by step

        walk left then right -
        The phone snaps a close-up of the meter. Then you walk left and right while it takes photos on its own.
        +
        The phone snaps the meter, then takes photos on its own as you walk the wall.
      • @@ -535,7 +535,7 @@

        The scan, step by step

        haze seen: the haze lifts -
        +
        The haze lifts as the phone sees.
      • @@ -575,7 +575,7 @@

        The scan, step by step

        server: checks every rule -
        +
        The phone uploads its measurements, and the server tries every spot.
      • From 3c0dd56c4e74ec872de7a1904c7d9a9a5b860705 Mon Sep 17 00:00:00 2001 From: SamGu-NRX Date: Sat, 26 Sep 2026 15:35:03 -0500 Subject: [PATCH 16/16] docs: add the first phone run and correct rule and detector facts - docs/00 reports the first TestFlight run on a real wall (PR #23), says rules.yaml marks every value without a public source as a placeholder, and calls Gemini the planned box detector, since no branch uses it yet. - docs/04 notes that the demo has no 1 ft wall-distance rule because the solver places the battery flush. - AGENTS.md and README point to PR #23, PR #17 and the exact docs/00 section, following the rewritten prompt-writing skill. Co-Authored-By: Claude Opus 5.5 (1M context) --- AGENTS.md | 4 ++-- README.md | 2 +- docs/00-overview.md | 9 +++++---- docs/04-prior-art-and-codes.md | 2 +- 4 files changed, 9 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index df6a3fc08..34e3e520d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -21,8 +21,8 @@ A component's README wins over the docs. Files marked with a pull request exist | The capture packet | `packet/README.md` (PR #22) | | The rules engine, API and scene contract | `server/README.md` (PR #11), especially "What settles each check" | | Photos to a 3D model | `recon/HANDOFF.md` (PR #20) | -| Accuracy evals and the field test | `experiments/evals/README.md` (PR #12) | -| Measurement conventions the code relies on | `docs/00-overview.md` | +| Accuracy evals and the field test | `experiments/evals/README.md` (PR #12), and the first phone run in `experiments/device-field-test/README.md` (PR #23) | +| Measurement conventions the code relies on | `docs/00-overview.md`, section "Conventions the code relies on" | | Public rule values, code citations, model and imagery licenses | `docs/04-prior-art-and-codes.md` | | The live guided-survey design | `docs/05-live-guided-survey-hld.md` | | Branches, CI checks and TestFlight | `CONTRIBUTING.md` | diff --git a/README.md b/README.md index d64ad1739..67a521112 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ Paths marked with a pull request exist only on that branch until it merges. | `packet/` | The capture packet's spec, validator and samples | PR #22 | | `server/` | The rules engine and placement API (Python, uv) | On `main`, a skeleton. The engine is in PR #11 | | `recon/` | Turns photos and depth into a 3D model and a coverage map | PR #20, handed to the server team | -| `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, meter reading in PR #16 | +| `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, meter reading in PR #16, panel labels in PR #17, the first device field test in PR #23 | | `web/` | Browser toolchain for a reviewer view | The review page (PR #15) is parked | | `docs/` | The overview, the walkthrough, public rules and code citations, and the live-survey design | | | `.agents/skills/` | Shared agent skills for writing, planning and review, linked from `.claude/skills/` | | diff --git a/docs/00-overview.md b/docs/00-overview.md index f84703e34..20759ef3a 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -23,10 +23,10 @@ The reconstruction worker is in `recon/` (PR #20), the accuracy evals in `experi ## Decisions - **Guided AR capture, not a video processed later.** Photogrammetry alone has no scale, is slow, and struggles on blank siding and stucco. COLMAP's own tutorial says to avoid texture-less walls. The packet carries ARKit's poses and any LiDAR depth, so the server's model starts at real scale. -- **3D modeling stays central.** Learned depth alone is too rough at edges, so LiDAR and the phone's poses supply the scale. +- **The server builds a 3D model at real scale.** Learned depth alone is 4 to 12% off in scale and 8 in or worse at edges, so LiDAR and the phone's poses supply the scale. - **Models build geometry and recognize things. Plain code decides placement.** The engine unrolls the walls into one line, `s`, in feet from the meter, negative to the left. It slides the battery's footprint along that line in 2 in steps and checks each rule at each position. Every answer then traces to a rule and a measurement. - **Every check answers PASS, FAIL or UNSURE.** A check passes only when its margin beats the measurement's error, and fails only when it misses by more than the error. Everything else is UNSURE. That includes any check on an area nobody observed, and the answer names the view that would settle it. -- **Rule values live in rules files, each with its source.** `server/rules.yaml` (PR #11) holds the public values and two labeled placeholders: pool 10 ft and driveway 5 ft. Base's values load from a git-ignored private file. Battery sizes live there too, because models differ. Base Core is 39.5 in tall on a 30.68 × 22 in footprint, and older units are 3 × 3 ft. +- **Rule values live in rules files, each with its source.** `server/rules.yaml` (PR #11) holds the public values and marks each value with no public source `placeholder: true`, such as pool 10 ft and driveway 5 ft. Base's values load from a git-ignored private file. `rules.yaml` also holds the battery's size, because models differ. Base Core is 39.5 in tall on a 30.68 × 22 in footprint, and older units are 3 × 3 ft. - **Native iOS only.** Capture needs mesh export, per-frame pose and lens data, and mesh raycasts, and Expo has no first-class ARKit support. - **AR results hang off the meter's anchor, with `.gravity` alignment.** ARKit corrects anchors as tracking improves, and drift runs about 2 cm a second, so a result tied to the meter moves with the correction. `.gravityAndHeading` depends on the compass, which developers report off by up to about 176° near a house. @@ -34,7 +34,7 @@ The reconstruction worker is in `recon/` (PR #20), the accuracy evals in `experi These are the non-obvious facts behind numbers in the code, and where each number came from. -- `capturedImage` is landscape, as the sensor reads it, and the intrinsics match it. Any rotation must also rotate the intrinsics and every image-space annotation. Gemini is the box detector, because Claude's docs call Claude's coordinates approximate. Gemini returns `[ymin, xmin, ymax, xmax]` on a 0 to 1000 scale, so apply EXIF rotation before inference and before casting a box. +- `capturedImage` is landscape, as the sensor reads it, and the intrinsics match it. Any rotation must also rotate the intrinsics and every image-space annotation. Gemini is the planned box detector, because Claude's docs call Claude's coordinates approximate. Gemini returns `[ymin, xmin, ymax, xmax]` on a 0 to 1000 scale, so apply EXIF rotation before inference and before casting a box. - Depth maps are 256 × 192. Scale the intrinsics down to match. - ARKit camera space is x right, y up, looking down −z, with image v growing downward. The ray through pixel (u, v) points along ((u − cx)/fx, −(v − cy)/fy, −1) before the camera-to-world rotation. - Keyframes are kept every 0.5 m or 15° of movement. @@ -45,7 +45,7 @@ These are the non-obvious facts behind numbers in the code, and where each numbe ## Evidence so far -These results come from public datasets with laser-scanned or surveyed ground truth. Nothing here was measured on our phone yet. The sources are `experiments/evals/README.md` (PR #12), `recon/HANDOFF.md` (PR #20), `experiments/meter-closeup/README.md` (PR #16) and `experiments/panel-label/README.md` (PR #17). +These results come from public datasets with laser-scanned or surveyed ground truth. Only the first phone run below was measured on our phone. The sources are `experiments/evals/README.md` (PR #12), `recon/HANDOFF.md` (PR #20), `experiments/meter-closeup/README.md` (PR #16), `experiments/panel-label/README.md` (PR #17) and `experiments/device-field-test/README.md` (PR #23). - **Tracking.** A current iPhone stayed inside the server's error allowance. On the MARViN dataset, an iPhone 14 Pro Max had p90 position error of 8.6, 13.4 and 18.5 in after 10, 20 and 30 ft, against an allowance of 19.2, 38.4 and 57.6 in. That phone has LiDAR, and MARViN's reference may be scaled to its tracking. A 2018 iPhone 6s on ADVIO ran two to three times over, partly from noise in that dataset's reference. - **Learned depth.** A model's own scale is 4 to 12% off. Rescaled with the phone's poses, walls come within about 5 in at p90 at a realistic 2% pose error, and within 2.8 in with exact poses. Edges stay at 8 in or worse, and clearances are measured from edges. MapAnything's scale shifted with input resolution. @@ -54,6 +54,7 @@ These results come from public datasets with laser-scanned or surveyed ground tr - **Meter reading.** Apple's on-device text recognition read the full meter number on 71 of 73 real photos. It picked the right line on only 21 of 75, because nameplates carry several numbers. A barcode held the number on 20 of the 24 photos that had one. A list of three candidates held it on 85% of held-out photos. So the app offers three candidates to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small. - **Panel labels.** Not tested. Only 7 open photos of US panels exist, and the test needs 40. - **World models.** Not evaluated yet. +- **First phone run.** The TestFlight build's first run on a real wall (PR #23) set both wall ends at the meter, so the server placed no spot. Two of five measured features came within 4 in of the tape. ## Open questions diff --git a/docs/04-prior-art-and-codes.md b/docs/04-prior-art-and-codes.md index d33d6643f..a3aad477f 100644 --- a/docs/04-prior-art-and-codes.md +++ b/docs/04-prior-art-and-codes.md @@ -13,7 +13,7 @@ This was researched on 2026-09-25 from public sources, and Esri's imagery terms ## Public rule values -The demo's `server/rules.yaml` (PR #11) uses these values. Base's own values are private and stay out of tracked files. +The demo's `server/rules.yaml` (PR #11) uses these values, except the 1 ft wall distance, because the solver places the battery flush against the wall. Base's own values are private and stay out of tracked files. | Rule | Value | Source | | --- | --- | --- |