Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ Paths marked with a pull request exist only on that branch until it merges.
| `packet/` | The capture packet's spec, validator and samples | PR #22 |
| `server/` | The rules engine and placement API (Python, uv) | On `main`, a skeleton. The engine is in PR #11 |
| `recon/` | Turns photos and depth into a 3D model and a coverage map | PR #20, handed to the server team |
| `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, meter reading in PR #16, panel labels in PR #17, the first device field test in PR #23 |
| `web/` | Browser toolchain for a reviewer view | The review page (PR #15) is parked |
| `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, scoring in PR #4, meter reading in PR #16, the first device field test in PR #23 |
| `web/` | Browser toolchain for a reviewer view | No page on `main` |
| `docs/` | The overview, the walkthrough, public rules and code citations, and the live-survey design | |
| `.agents/skills/` | Shared agent skills for writing, planning and review, linked from `.claude/skills/` | |
| `sites/landing` | The landing page, a submodule | Change it in its own repository |
Expand Down
13 changes: 7 additions & 6 deletions docs/00-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,32 +27,33 @@ The reconstruction worker is in `recon/` (PR #20), the accuracy evals in `experi
- **Models build geometry and recognize things. Plain code decides placement.** The engine unrolls the walls into one line, `s`, in feet from the meter, negative to the left. It slides the battery's footprint along that line in 2 in steps and checks each rule at each position. Every answer then traces to a rule and a measurement.
- **Every check answers PASS, FAIL or UNSURE.** A check passes only when its margin beats the measurement's error, and fails only when it misses by more than the error. Everything else is UNSURE. That includes any check on an area nobody observed, and the answer names the view that would settle it.
- **Rule values live in rules files, each with its source.** `server/rules.yaml` (PR #11) holds the public values and marks each value with no public source `placeholder: true`, such as pool 10 ft and driveway 5 ft. Base's values load from a git-ignored private file. `rules.yaml` also holds the battery's size, because models differ. Base Core is 39.5 in tall on a 30.68 × 22 in footprint, and older units are 3 × 3 ft.
- **The coverage map shows evidence, not approval.** Gray means nobody saw it, yellow means seen but the evidence isn't enough yet, and green means enough evidence for the current requirement, which is not installation approval. Coverage follows the surfaces the camera saw, not the path the homeowner walked.
- **Native iOS only.** Capture needs mesh export, per-frame pose and lens data, and mesh raycasts, and Expo has no first-class ARKit support.
- **AR results hang off the meter's anchor, with `.gravity` alignment.** ARKit corrects anchors as tracking improves, and drift runs about 2 cm a second, so a result tied to the meter moves with the correction. `.gravityAndHeading` depends on the compass, which developers report off by up to about 176° near a house.

## Conventions the code relies on

These are the non-obvious facts behind numbers in the code, and where each number came from.
These are the facts behind numbers in the code that the code alone doesn't explain.

- `capturedImage` is landscape, as the sensor reads it, and the intrinsics match it. Any rotation must also rotate the intrinsics and every image-space annotation. Gemini is the planned box detector, because Claude's docs call Claude's coordinates approximate. Gemini returns `[ymin, xmin, ymax, xmax]` on a 0 to 1000 scale, so apply EXIF rotation before inference and before casting a box.
- Depth maps are 256 × 192. Scale the intrinsics down to match.
- ARKit's depth maps are 256 × 192, smaller than the camera image. Scale the intrinsics down to match.
- ARKit camera space is x right, y up, looking down −z, with image v growing downward. The ray through pixel (u, v) points along ((u − cx)/fx, −(v − cy)/fy, −1) before the camera-to-world rotation.
- Keyframes are kept every 0.5 m or 15° of movement.
- Keyframes are kept every 0.5 m or 15° of movement, a day-1 choice that Measure Lab kept.
- Mesh vertices are local to their `ARMeshAnchor`. Copy each one out of ARKit's buffer, then multiply the copy by the anchor's transform, leaving the buffer untouched. Classified mesh needs `sceneReconstruction = .meshWithClassification` on the world-tracking configuration, on LiDAR phones only. RealityKit also turns classification off unless `automaticallyConfigureSession` is `false`.
- On the mesh, the gap to a facing fence is a ray straight out from the wall at 1.5 ft high. Headroom is a ray straight up from 1 ft out. The mesh has holes, so each measurement is the smallest distance over a small fan of rays.
- On the mesh, the gap to a facing fence is a ray straight out from the wall at 1.5 ft high, and headroom is a ray straight up from 1 ft out. The 1.5 ft height and the 1 ft offset are day-1 choices, not measured. The mesh has holes, so each measurement is the smallest distance over a small fan of rays.
- LiDAR reads from about 0.5 m to 5 m, so capture stands 1 to 3 m from the wall. A ray that meets nothing within 5 m is unknown, not open.
- The default error bars are day-1 estimates, untested: 0.3 ft for an AR tap, 0.5 ft for the LiDAR mesh and 1.5 ft for a position from photo detection. The server adds 0.16 ft per foot along the wall from the meter, from measured ARKit drift.

## Evidence so far

These results come from public datasets with laser-scanned or surveyed ground truth. Only the first phone run below was measured on our phone. The sources are `experiments/evals/README.md` (PR #12), `recon/HANDOFF.md` (PR #20), `experiments/meter-closeup/README.md` (PR #16), `experiments/panel-label/README.md` (PR #17) and `experiments/device-field-test/README.md` (PR #23).
These results come from public datasets with laser-scanned or surveyed ground truth. Only the first phone run below was measured on our phone. The sources are `experiments/evals/README.md` (PR #12), `recon/HANDOFF.md` (PR #20), `experiments/meter-closeup/README.md` (PR #16) and `experiments/device-field-test/README.md` (PR #23).

- **Tracking.** A current iPhone stayed inside the server's error allowance. On the MARViN dataset, an iPhone 14 Pro Max had p90 position error of 8.6, 13.4 and 18.5 in after 10, 20 and 30 ft, against an allowance of 19.2, 38.4 and 57.6 in. That phone has LiDAR, and MARViN's reference may be scaled to its tracking. A 2018 iPhone 6s on ADVIO ran two to three times over, partly from noise in that dataset's reference.
- **Learned depth.** A model's own scale is 4 to 12% off. Rescaled with the phone's poses, walls come within about 5 in at p90 at a realistic 2% pose error, and within 2.8 in with exact poses. Edges stay at 8 in or worse, and clearances are measured from edges. MapAnything's scale shifted with input resolution.
- **Reconstruction worker.** On the ETH3D building, wall p90 is 2.1 in with a laser scan standing in for LiDAR, and 2.8 in from photos only. The two runs measured different walls.
- **Coverage.** The app's first coverage map claimed 1.1 ft of a 19.3 ft wall that no photo saw. Requiring several view angles removed the false claim but also discarded 6 to 8.5 ft of good wall. A depth test on LiDAR phones removed it cleanly. Checked against depth, the reconstruction worker claims at most about 0.46 ft of unseen wall.
- **Meter reading.** Apple's on-device text recognition read the full meter number on 71 of 73 real photos. It picked the right line on only 21 of 75, because nameplates carry several numbers. A barcode held the number on 20 of the 24 photos that had one. A list of three candidates held it on 85% of held-out photos. So the app offers three candidates to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small.
- **Panel labels.** Not tested. Only 7 open photos of US panels exist, and the test needs 40.
- **Panel labels.** Not tested. A search found 7 usable open photos of US panels, and the test needs 40.
- **World models.** Not evaluated yet.
- **First phone run.** The TestFlight build's first run on a real wall (PR #23) set both wall ends at the meter, so the server placed no spot. Two of five measured features came within 4 in of the tape.

Expand Down
2 changes: 1 addition & 1 deletion docs/04-prior-art-and-codes.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The demo's `server/rules.yaml` (PR #11) uses these values, except the 1 ft wall

Prior measurements: iPhone 12 Pro Max LiDAR on facades lands within about ±6 to 8 cm of a total station, and iPad Pro LiDAR reaches about 5 m. A YOLOv5 meter-reading study read the counters 97% of the time and serial numbers only 63%.

Nobody else runs the whole loop outdoors, from metric AR capture through a solver that cites a code for each clearance to an AR preview. The heat-pump AR tools are visual only, and Fraunhofer's solver works indoors.
None of the products above runs the whole loop outdoors, from metric AR capture through a solver that cites a code for each clearance to an AR preview.

## Licenses and terms

Expand Down
17 changes: 2 additions & 15 deletions docs/how-it-works.html
Original file line number Diff line number Diff line change
Expand Up @@ -893,7 +893,7 @@ <h3>What a path must deliver</h3>

<!-- ================= 5. EVIDENCE ================= -->
<h2 id="evidence">5. What the tests found</h2>
<p class="prose">All results come from public datasets, not our phone. Panel-label reading isn't tested, because only 7 open photos of US panels exist and the test needs 40.</p>
<p class="prose">The charts below come from public datasets. The app's first run on a real phone (PR #23) set both wall ends at the meter, so the server placed no spot. Panel-label reading isn't tested, because a search found only 7 usable open photos of US panels and the test needs 40.</p>

<div class="finding" id="finding-1">
<h3>Tracking drift</h3>
Expand Down Expand Up @@ -1038,20 +1038,7 @@ <h3>Field test on our phone</h3>

<!-- ================= 6. WHERE ================= -->
<h2 id="where">6. Where things live</h2>
<p class="prose">Files marked with a PR live on that branch until it merges. A component's README wins over this page.</p>
<div class="scroll-x"><table>
<thead><tr><th>Piece</th><th>Source of truth</th><th>State</th></tr></thead>
<tbody>
<tr><td>iPhone app<br><span class="meta">client team</span></td><td><code>ios/README.md</code></td><td>Guided capture in <a href="https://github.com/SamGu-NRX/house-scanning-master/pull/10" target="_blank" rel="noopener noreferrer">PR #10</a>, live 3D map in <a href="https://github.com/SamGu-NRX/house-scanning-master/pull/21" target="_blank" rel="noopener noreferrer">PR #21</a>, stacked on it</td></tr>
<tr><td>Capture packet<br><span class="meta">client team</span></td><td><code>packet/README.md</code></td><td>Spec, validator and samples in <a href="https://github.com/SamGu-NRX/house-scanning-master/pull/22" target="_blank" rel="noopener noreferrer">PR #22</a></td></tr>
<tr><td>Rules engine, API, capture contract<br><span class="meta">server team</span></td><td><code>server/README.md</code>, "What settles each check"</td><td><a href="https://github.com/SamGu-NRX/house-scanning-master/pull/11" target="_blank" rel="noopener noreferrer">PR #11</a>, live as a <a href="https://house-scanning-server.vercel.app/health" target="_blank" rel="noopener noreferrer">public demo</a> with public rules, and privately with Base's rules behind a key</td></tr>
<tr><td>Reconstruction worker<br><span class="meta">server team</span></td><td><code>recon/HANDOFF.md</code></td><td>Draft <a href="https://github.com/SamGu-NRX/house-scanning-master/pull/20" target="_blank" rel="noopener noreferrer">PR #20</a>, handed to the server team</td></tr>
<tr><td>Accuracy evals</td><td><code>experiments/evals/README.md</code></td><td><a href="https://github.com/SamGu-NRX/house-scanning-master/pull/12" target="_blank" rel="noopener noreferrer">PR #12</a></td></tr>
<tr><td>Agent rules, system overview</td><td><code>AGENTS.md</code>, <code>docs/00-overview.md</code></td><td>On <code>main</code></td></tr>
<tr><td>Web review page</td><td></td><td>Parked, <a href="https://github.com/SamGu-NRX/house-scanning-master/pull/15" target="_blank" rel="noopener noreferrer">PR #15</a></td></tr>
</tbody>
</table></div>
<p class="prose">Plan and evidence: <a href="00-overview.md">00-overview.md</a>. Rule sources and licenses: <a href="04-prior-art-and-codes.md">04-prior-art-and-codes.md</a>.</p>
<p class="prose"><a href="../README.md">README.md</a> maps the repository, and each component's README is the source of truth for its part. Plan and evidence: <a href="00-overview.md">00-overview.md</a>. Rule sources and licenses: <a href="04-prior-art-and-codes.md">04-prior-art-and-codes.md</a>.</p>

<h3>Try it</h3>
<div class="prose">
Expand Down
15 changes: 9 additions & 6 deletions experiments/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
# Experiments

An experiment answers one question the team needs settled before building on an assumption. For example: can an iPhone without LiDAR measure the gap between a wall and a fence to within a few inches?
An experiment settles one question before the team builds on the answer. For example: can an iPhone without LiDAR measure the gap between a wall and a fence to within a few inches?

- Give each experiment its own folder, `experiments/<short-name>/`.
- Its README states the question, the method and the pass criteria before the run, then the result and what it changes after. Keep the result when it fails; a failure tells us which part to change next.
- Keep dependencies inside the folder, in its own uv, pnpm or Xcode project. Don't add them to `server/`, `web/` or `ios/`.
- Write outputs to `data/`, which git ignores. Never commit photos, video or measurements of a real home, or Base's materials.
- Read and write the team's `scene.json` once it exists, so every method's numbers compare on the same terms.
The record of an experiment is its code and the files it produced. Its README is the front door, not the report.

- One folder per experiment, `experiments/<short-name>/`, with its own uv, pnpm or Xcode project. Its dependencies stay out of `server/`, `ios/` and `web/`.
- The README stays under about 200 words: the question and its pass criteria, written before the run, the command that runs it and what that needs, the result in a sentence or two with its numbers, and what it changed in the product or the plan. Keep a failed result, because it says which part to change next.
- Generated tables and plots from public data go in `results/` inside the folder, with the command that produced them. Anything from a real home, and anything Base gave us, goes in git-ignored `data/`.
- When the product or the plan relies on the result, `docs/00-overview.md` gets one line under "Evidence so far" that points here.
- Delete an experiment when a later one supersedes it. First point its line in `docs/00` at the experiment that replaced it, so the finding stays and its source still exists. Git keeps the old code.
- Read and write `scene.json`, the app's measurement file, so every method's numbers compare on the same terms.