diff --git a/docs/research/README.md b/docs/research/README.md new file mode 100644 index 000000000..096487bf5 --- /dev/null +++ b/docs/research/README.md @@ -0,0 +1,13 @@ +# Research notes + +Notes from the agents working beside the four lanes. They build on the team plan in [docs/00](../00-overview.md) through [docs/05](../05-live-guided-survey-hld.md). Each answers one question for one reader. They propose; the lane owners decide. + +| Note | Question it answers | For | +| --- | --- | --- | +| [t3-capture-surfaces.md](t3-capture-surfaces.md) | Which phone surface can measure what, and how does Android get covered later? | Everyone | +| [t3-no-lidar-capture.md](t3-no-lidar-capture.md) | How can an iPhone without LiDAR measure what the placement checks need, and how do we test that before relying on it? | Lanes A and B | +| [t3-first-try-capture.md](t3-first-try-capture.md) | How does the capture know it has enough before the homeowner leaves, and what does the homeowner see? | Lane A | +| [t3-lane-c-review.md](t3-lane-c-review.md) | What must the solver settle before it can approve or reject a site, and which test cases prove it? | Lane C | +| [t3-scoring-protocol.md](t3-scoring-protocol.md) | How do we compare every capture method fairly on one to three houses, and what can we claim? | Lane D | + +Base's own criteria never appear here: thresholds without a public source are named `rules.yaml` parameters. diff --git a/docs/research/t3-capture-surfaces.md b/docs/research/t3-capture-surfaces.md new file mode 100644 index 000000000..35c5be7c4 --- /dev/null +++ b/docs/research/t3-capture-surfaces.md @@ -0,0 +1,49 @@ +# Capture surfaces: what each phone surface can measure + +Most homeowners don't have a LiDAR iPhone, and they reach Base through a link. This note compares the places a capture can run, so the team can decide which surface serves which phone. The references below were collected on 2026-09-25. Nobody has tested these capabilities on a phone for this project yet. + +## Capabilities by surface + +| Surface | AR tracking in real units | Planes and raycasts | Depth or mesh | Camera frames with pose | Entry friction | +| --- | --- | --- | --- | --- | --- | +| iOS app, no LiDAR | Yes (ARKit world tracking) | Yes | No | Yes | App Store or TestFlight install | +| iOS app, LiDAR (Pro models) | Yes | Yes | Yes (scene depth, mesh) | Yes | Same | +| App Clip on iOS 17+ | Yes, ARKit and camera allowed | Yes | Unverified inside a clip; would need a LiDAR phone | Yes | No install; opens from a link, a Messages bubble, a QR code or NFC | +| iPhone Safari | No: Safari 27 still has no WebXR AR | No | No | Video without pose | None | +| Android Chrome | Yes (WebXR on ARCore phones) | Yes, plane detection since Chrome 147 | Depth from motion | Yes (raw camera access) | None | +| Android app | Yes (ARCore) | Yes | Depth from motion, best at 0.5 to 5 m | Yes | Play Store install; Play Instant ended in December 2025 | +| Photos only (Base today) | No | No | Model estimates only | No pose | A link with 7 to 9 photos | + +CI builds and unit-tests the iOS app and the web toolchain, and agents can drive web pages in a browser. AR tracking runs only on a real phone outdoors, so every end-to-end check of the AR path needs a person holding the phone. + +## What the table means for the product + +- **On iPhone, AR needs native code.** The App Clip gets it without a full install. A clip opened from a QR code or NFC tag must stay under 15 MB; a clip opened only from links can reach 100 MB on iOS 17 and later. Uploads must finish in the foreground, because clips can't use background transfers. +- **Android Chrome is a no-install capture candidate.** Before treating it as an alternative to native ARCore, verify saved camera images with matching poses and calibration, negotiated depth export, and interrupted-session recovery on supported devices. Until then, native Kotlin remains the proposed Android capture path. +- **Everyone else falls back to photos or video processed on the server.** The best learned models report roughly 13 to 15 percent scale error on research benchmarks. Nobody has measured them on house walls, so the photo baseline gets scored against the same tape-measured spots before anyone trusts an error bound. +- **No published test covers ARKit accuracy without LiDAR on an exterior walk like this one.** We have to measure it ourselves with a tape before the rules' margins can rely on it. +- **Generative world models are for pictures, not measurements.** World Labs Marble and Atlas and Runway's GWM fill in regions the camera never saw. A filled-in corner is not evidence that the corner is clear. + +## Reaching Android later + +Keep Swift capture for the hackathon. Android work starts after one iPhone capture runs end to end, and it should not change the server. + +- **Add a Kotlin ARCore capture adapter.** ARCore gives metric poses, planes, hit tests and camera intrinsics without a depth sensor. It has no ARKit-style scene mesh, and [raw depth](https://developers.google.com/ar/develop/java/depth/raw-depth) usually has zero confidence on textureless walls. Facing gaps and headroom stay UNSURE unless another source measures them. The adapter can run in a native Android app or behind an Expo shell. +- **Keep the upload format independent of ARKit and LiDAR.** A minimal, versioned manifest records the capture provider, device, session ID, units, and each frame's timestamp, pose, intrinsics and image size. Depth and mesh are optional. A missing mesh means unknown, never clear space. +- **Treat Android WebXR export as a short experiment, not parity.** The table shows what Chrome exposes. The proposed WebXR capture path has not demonstrated saved camera images with matching poses and calibration. Test that first: [raw camera access](https://immersive-web.github.io/raw-camera-access/), [depth sensing](https://www.w3.org/TR/webxr-depth-sensing-1/) on both its CPU and GPU paths, reprojection of saved frames, and an interrupted session. If export fails, the phone routes to photos before the walk starts. +- **Guided photos stay the broadest fallback**, including phones without ARCore. The app should check the photos for the deciding views before the homeowner leaves. + +## Closest prior art + +- Base's current flow sends a link that asks for 7 to 9 photos: the meter, both sides of the installation area, the adjacent wall, the main breaker and the disconnect. Engineering follows up within two days when it needs more. +- Hover asks for at least eight guided exterior photos and returns a measured model, claiming about 5 percent accuracy after 2 to 4 hours of processing. +- SiteCapture and Qmerit send homeowners a browser link for photo surveys with no app, which is the friction bar an App Clip has to match. + +## Sources + +- ARKit without LiDAR: [world tracking](https://developer.apple.com/documentation/arkit/understanding-world-tracking), [plane detection](https://developer.apple.com/documentation/arkit/arworldtrackingconfiguration/planedetection), [raycasting](https://developer.apple.com/documentation/arkit/raycasting), [scene reconstruction needs LiDAR](https://developer.apple.com/documentation/arkit/arworldtrackingconfiguration/supportsscenereconstruction%28_%3A%29) +- App Clips: [size limits](https://developer.apple.com/help/app-store-connect/reference/app-uploads/maximum-build-file-sizes), [allowed functionality](https://developer.apple.com/documentation/appclip/choosing-the-right-functionality-for-your-app-clip), [testing](https://developer.apple.com/documentation/appclip/testing-the-launch-experience-of-your-app-clip) +- iPhone Safari: [Apple forum answer on immersive-ar](https://developer.apple.com/forums/thread/756850?answerId=790772022#790772022), [Safari 27.0 features](https://webkit.org/blog/18325/webkit-features-for-safari-27-0/) +- Android: [WebXR on ARCore](https://developers.google.com/ar/develop/webxr/arcore-comparison), [Chrome 147 release notes](https://developer.chrome.com/release-notes/147), [ARCore Depth API](https://developers.google.com/ar/develop/depth), [Play Instant notice](https://developer.android.com/topic/google-play-instant) +- World models: [Marble API FAQ](https://docs.worldlabs.ai/api/faq), [Atlas announcement](https://www.worldlabs.ai/blog/atlas), [Runway GWM-1](https://runway.com/research/introducing-runway-gwm-1) +- Prior art: [Base photo review](https://help.basepowercompany.com/en/articles/10280641), [Hover exterior scans](https://help.hover.to/en/articles/9185612-exterior-scans), [Hover accuracy](https://hover.to/architects/), [SiteCapture self-surveys](https://sitecapture.com/self-surveys-inspections/) diff --git a/docs/research/t3-first-try-capture.md b/docs/research/t3-first-try-capture.md new file mode 100644 index 000000000..9e956dbc8 --- /dev/null +++ b/docs/research/t3-first-try-capture.md @@ -0,0 +1,105 @@ +# First-try capture: enough evidence before the homeowner leaves + +[docs/05-live-guided-survey-hld.md](../05-live-guided-survey-hld.md) describes the live guided survey: automatic capture, the coverage display, next-view planning and recovery. This note fills in what the capture needs before the homeowner leaves: the evidence per check, the finish rule, a proposal to stop early, the close-up gates and the first trial. + +The capture has enough when one candidate spot has a result for every placement check backed by saved keyframes, the meter and panel close-ups have passed their photo checks, and the server has acknowledged every upload. Anything the phone cannot settle stays UNSURE and goes to installer review with its reason. It never becomes a later photo request. This is a direction for lane A, not a hackathon promise. + +## What counts as evidence, by check + +Each check returns PASS, FAIL or UNSURE, as in [02-implementation-plan.md](../02-implementation-plan.md). PASS needs the deciding area in retained keyframes and a margin larger than the error. The error bars are team estimates, not measurements: ±0.3 ft for AR taps, ±0.5 ft for the LiDAR mesh, ±1.5 ft for photo detections. Freeze one set of `rules.yaml` values before capture. + +| Check | Evidence that settles it | Threshold | +|---|---|---| +| Wall and footprint | Wall behind the whole 31 x 22 in footprint, seen from two positions | Base: within 1 ft of the wall | +| Ground | Surface and slope of the ground under and around the footprint | Chosen policy; Base's page gives a 3 by 3 ft area | +| Gates and paths | Gate swings and paths past the spot, with clear widths | Chosen policy | +| Nearby equipment | Footprint and service side of each nearby AC unit, battery or other equipment | Base: 3 ft from AC units and other batteries | +| Above the spot | Vents and wall boxes on the wall above the footprint | Chosen policy | +| Gas | Gas meter and pipes located; ground within the clearance of every footprint edge seen | `gas_clearance_ft`; Base's page says 3 ft | +| Driveway | Nearest boundary measured, plus retained views showing connected vehicle access, including around corners; unseen connections remain UNSURE | `drive_clearance_ft`, set by Base | +| Pool | Pool edge seen, or the area within the clearance seen empty | `pool_clearance_ft`, set by Base | +| Doors and windows | Position measured; whether it opens comes from the homeowner | `opening_clearance_ft`; IRC R328 says 3 ft | +| Facing gap | Nearest fence, hedge or wall across the full footprint width | `facing_gap_ft`, set by Base | +| Headroom | Lowest overhead object across the installation and its working space | `headroom_ft`, set by Base | +| Cable route | Continuous wall from meter to spot, no door or garage crossing | `max_route_ft`; Base: within 20 ft of the meter | +| Working space | Footprint outside the space in front of meter and panel | NEC 110.26: 30 in x 36 in x 6.5 ft | +| Disconnect, if the install uses one | Free wall space for the disconnect box | NEC 706.15: within sight, and within 10 ft or lockable | + +This table summarizes what to capture. It is not the acceptance checklist: before capture, list every check the chosen policy applies. A check without evidence stays unsure and cannot count toward a pass. + +A non-LiDAR phone has no mesh, so facing gap and headroom stay UNSURE. Hypothesis to test: taps on the wall and the facing fence at both footprint edges, raycast onto detected planes, bound the gap within ±0.3 ft. + +Ask the homeowner only for facts a camera cannot establish: whether a window opens, whether they would move the bins, whether a street-facing spot is acceptable. Answers about hazards steer capture but never pass a check. + +Proposed `scene.json` addition for lanes A to C: each check result lists its supporting keyframe IDs. + +## Coverage tracker and finish rule + +The tracker extends the coverage display in [docs/05 section 2](../05-live-guided-survey-hld.md#2-customer-journey) with a failed state and a strip showing the unrolled wall with the meter at zero, plus ground and overhead bands. Cells are 6 in wide, a display guess to try. Each cell is unseen, seen but not enough, enough, or failed. Tapping one shows the single view it needs. In AR, a fog on the wall clears where the camera has looked. + +The fog is guidance, never proof that space is clear. A cell clears only from retained keyframes with normal tracking, from two positions. A non-LiDAR phone still cannot tell that a bin hid the ground behind it; the server's checks decide. + +**Finish** unlocks when all four hold: + +1. One candidate passes every placement check with linked keyframes, or the homeowner has reached both walkable limits and each unknown is recorded for installer review. +2. Both close-ups passed their gates, or the homeowner chose "Can't get a clear shot". +3. The server acknowledged every upload. Saved offline is not finished. +4. For the hackathon, a teammate acting as Base's reviewer has checked the packet while the homeowner is still outside. + +## Early stop + +This is a proposal the team has not approved. A2 in [01-feature-map.md](../01-feature-map.md) requires walking to both wall ends, and [docs/05 section 7](../05-live-guided-survey-hld.md#7-deterministic-criteria-evaluation) keeps that gate until the team approves a tested completion rule. The proposal: one fully evidenced candidate ends the walk. Two limits apply: + +- Evidence can extend past the candidate. A gas meter 2 ft beyond the footprint still needs its surrounding ground seen. +- Proving the shortest route is optional. If the other side of the meter is unseen out to the same distance, the result says "A closer spot may exist on the left". + +With no candidate on the meter wall, continue to reachable adjacent walls when more placement evidence is needed. Unseen alternatives remain unknown. This capture checklist does not define installation eligibility or authorize a site-level rejection. + +## Close-up gates + +- **Meter label.** Run Vision's `RecognizeTextRequest` with `recognitionLevel` accurate and `usesLanguageCorrection` off. Require the text inside the reticle, all label edges in frame, and matching reads from two frames. OCR confidence cannot say which of the meter's numbers is the meter number; the server decides. +- **Blur.** Laplacian variance on the label crop, not the whole frame. +- **Glare.** Reject when near-white pixels cover any recognized character's box. +- **Panel.** Three shots: panel and surrounding wall, breakers with the door open, main breaker rating up close. + +All cutoffs are hypotheses to tune. After two failed tries, "Can't get a clear shot" sends that check to installer review instead of looping. + +## What the homeowner sees + +1. A text with a link. The hackathon build opens the development app. +2. Camera permission, then "Go to your electric meter" and **I'm here**. +3. The meter close-up. The shutter fires itself when the gates pass. +4. "Tap the meter." A raycast onto a detected wall plane sets the anchor. With no plane, it asks them to step closer; it never guesses a depth. +5. "If there's room, step back until the whole meter box and the ground below it are in view." +6. The walk with strip and fog. One question at a time, two large buttons. +7. The early stop, then the panel while the wall uploads. +8. Corrections on saved frames. The homeowner marks a missed object in a saved keyframe. Its pixel and stored pose define a ray, placed on a validated wall plane only for an object on that wall. Otherwise its position stays UNSURE. +9. The reveal at the meter: a box on the meter's anchor and a cable line, with clearances as numbers on tap. + +Opening the panel: + +> Open the panel's hinged door. Don't unscrew or remove the metal cover behind it, and don't touch the breakers. If the door is locked or stuck, tap **Skip**. + +Borderline reveal: + +> This spot is 3 ft 2 in from the gas meter. The rule is 3 ft and our measurement can be off by about 4 in, so an installer will check it. The panel still needs an electrician's review. + +## Recovery + +These add to the operational cases in [docs/05 section 9](../05-live-guided-survey-hld.md#9-reliability-latency-and-data-handling). + +- **Tracking drops.** Coverage credit stops and the prompt changes, with no modal: "Slow down" for `excessiveMotion`, "Aim at a corner or somewhere with more texture" for `insufficientFeatures`. +- **Interruptions.** Keyframes and close-ups go to disk after each step. On return, `sessionShouldAttemptRelocalization(_:)` returns true and the app shows a saved meter view: "Point at the meter like this." +- **Failed relocalization.** Start a new coordinate frame: re-tap the meter and re-walk. One matching point does not fix heading, and a 5 degree heading error moves a point 20 ft away by about 1.7 ft. +- **Uploads.** Foreground only, resumed on return, never restarted. + +## First trial + +Run one manually guided end-to-end capture on a non-LiDAR iPhone, with the operator keeping the evidence checklist by hand. First, a second person lists every hazard and tape-measures each deciding clearance. Stage four cases: + +1. A clear candidate. Expected: without validated measurements, facing gap and headroom stay UNSURE and go to installer review, so the spot cannot fully pass. +2. A candidate whose ground is hidden behind a bin. Expected: a view request during the walk, or UNSURE. +3. A gap within the error bar of its threshold. Expected: UNSURE, installer review. +4. An electrical label made unreadable by tape or glare. Expected: the gate refuses it, then installer review after two tries. + +Fail the trial if any PASS contradicts the independent measurements or hazard inventory, or if a packet is marked evidence-complete while a required view, readable label or measurement is missing. Record evidence-complete, review-pending and installer-assessment outcomes separately, along with unresolved capture needs. Also record walk time, retakes, questions asked and wall length walked. diff --git a/docs/research/t3-lane-c-review.md b/docs/research/t3-lane-c-review.md new file mode 100644 index 000000000..2ecbb1d3f --- /dev/null +++ b/docs/research/t3-lane-c-review.md @@ -0,0 +1,65 @@ +# Lane C: what the solver spec must settle before it approves or rejects + +The [docs/02 Lane C solver][c] can't yet approve or reject a site safely. Twelve gaps block it, ranked by how many decisions each can flip. The 14 golden tests below check that each gap is closed. Thresholds appear as `rules.yaml` parameters. + +## What the spec must settle + +1. **Select one complete policy.** The public rules table leaves installation requirements unresolved. Require an approved, versioned configuration before enabling automatic approval. [Lane C][c] has one decision function. Record the policy ID and version in the [rules table][r]. Until a policy is selected, the solver must not approve automatically. +2. **Define error propagation.** [Lane C][c] has a strict PASS margin but no FAIL or equality rule, and it doesn't separate a missing measurement from a measured zero. Two ±0.3 ft taps give a gap with ±0.6 ft error. Proposed, pending Base: for a minimum clearance T and a measured gap d ± e, PASS if d − e > T, FAIL if d + e < T, else UNSURE. Reverse both inequalities for maximum lengths. The docs/02 error sizes are guesses until the D3 tape-measure test checks them. +3. **Define what a reject must have seen.** [Lane C][c] rejects once both wall ends are tapped, but a tapped corner or a closed gate doesn't end the search. Record the observed wall, ground and route intervals, and the type of each end. Reject only when coverage is complete and every candidate region has a clear FAIL. +4. **Store wall geometry.** The [`scene.json` schema][s] doesn't store segment order, outward normals, ground elevations or the meter-anchor transform, although docs/02 describes some of them in prose. The [Lane C][c] sweep also lets a footprint cross a corner. Require each rigid footprint to sit flush on one straight wall segment. +5. **Keep ground-object shapes.** Lane B snaps ground objects to the nearest wall point. [Lane C][c] then can't see that two objects far apart in s sit close together around an inside corner. Keep a ground polygon for every part of an object, such as a gas meter and its regulator, and measure from the whole footprint. +6. **Measure the routed length.** [Lane C][c] uses |s| plus a corner allowance. For a spot left of the meter, s is the far edge of the footprint, and the formula ignores vertical runs and detours. Sum the supported route polyline and carry its error. +7. **Complete the route rules.** The [Lane C][c] route blocks only doors, garages and stretches with no wall, and the pseudocode never feeds `route` into the decision. Store route support, blockers and visibility, apply the selected policy's crossing rules, and treat a hidden stretch that decides the result as UNSURE. +8. **Model openings and access as shapes.** [Lane C][c] checks openings as overlap between wall intervals. That can't express a door or gate swing area, or the working space in front of equipment (NEC 110.26). Keep the attributes each opening rule needs, such as `operable` and `well`. An unknown attribute is UNSURE, never false. +9. **Check facing gap and headroom everywhere.** Lane B samples passage width at 1.5 ft height and headroom 1 ft out from the wall every 2 inches, using a small fan of rays. That sampling does not establish clearance over the whole footprint and working area. Take the conservative minimum over the whole checked region and record which parts were observed. The rules must also say whether the facing threshold counts from the wall or from the battery's front face. +10. **Give ground its meaning.** The ground `type` in the [schema][s] records surface material only. It can't express whether a vehicle can reach the surface (feature B6), the slope, the height above grade or the separation from a pool. An empty pool list doesn't prove there is no pool. If the capture didn't cover the area, the pool check is UNSURE. +11. **Don't reject from samples.** The [Lane C][c] loop stops before its last start, `s_max - W`, and a legal start interval narrower than 2 in can fall between samples. Add segment ends and constraint boundaries as candidates, and confirm infeasibility over continuous intervals before rejecting. +12. **Specify the result record.** C5 in the [feature map][f] lists a verdict, reasons and photos still needed. An audit also needs the policy and version, the footprint relative to the meter anchor, each check's measured value, error and verdict, the route polyline and its length and error, and how scale was set. A reject also needs both wall ends and the reason each region fails. Keep the photo request separate from the verdict. UNSURE has two causes: a measured gap inside its error band, and an area nobody observed. More photos fix only the second. + +## Golden tests + +Every expected result below follows from geometry, logic, measurement error or the public sources. Further policy-specific cases live in the team's private notes. + +| Parameter | Value | Source | +| --- | --- | --- | +| battery footprint | 31/12 × 11/6 ft; test 12 uses a 2 ft wide test battery | Base Core 30.68 × 22 in, rounded to 31 in as in C2 | +| `gas_clearance_ft` | 3 ft | Base help page | +| `opening_clearance_ft` | 3 ft, added to both sides of an opening as in docs/02 | IRC R328 | +| `headroom_ft` | TBD with Base | NEC 110.26 gives 6.5 ft for working space at the meter and panel | +| `confident_reach_ft`, `max_route_ft` | TBD with Base; tests need `max_route_ft` > `confident_reach_ft` + 1 | Base help page: within 20 ft of the meter | + +**Shared fixture.** These are test settings, not Base policy. Units are feet, and geometry is exact unless a test gives an error. Wall w1 is straight, with s = x, the meter at 0 and outward +z. The only usable pad is level ground at s = [6, 9]. Every other location fails a fixed ground check. Both wall ends are observed, the facing gap and headroom are 9, and the supported route to the nearest battery edge is 6. Every check that a test doesn't name passes. Footprint containment is inclusive. Tests 09, 10 and 12 rely on the margin rule in item 2. + +**01. A fully observed spot passes without unrelated views.** Setup: the wall beyond a garage opening on the far side of the meter is unseen. Expected: pass at s = [6, 103/12], z = [0, 11/6], route 6, no photo request. + +**02. No selected policy, no automatic approval.** Setup: shared fixture with no policy selected. Expected: manual review naming policy selection as the reason, never pass. + +**03. Negative s uses the near edge.** Setup: pads [−9, −6] and [7, 10]. Expected: pass at [−103/12, −6] with route 6, chosen over [7, 115/12] with route 7. Using |−103/12| as the left route length is wrong. + +**04. A corner allows an adjacent wall, not a bent battery.** Setup: w1 runs from [−12, 0] to [4, 0] with outward normal [0, −1] and the meter at [0, 0]. w2 runs from [4, 0] to [4, 12] with outward normal [1, 0], and s = 4 + z. Only the w2 pad at z = [2, 5] works, with route 6. Candidate s = [3, 67/12] crosses the corner at s = 4. Expected: pass on w2 at z = [2, 55/12], x = 4, with a footprint extending toward +x to 35/6. The corner candidate must FAIL backing and footprint support, and its failure must not invalidate the valid w2 spot. + +**05. Inside-corner gas distance is Euclidean.** Setup: w1 runs along z = 0 from x = −6 to 4, outward +z, with the meter at x = −6. A return wall runs along x = 4, outward −x. The only pad is x = [0, 3]. A gas meter sits at [4, 4]. Expected: reject. From the footprint at x = [0, 31/12], the distance is √((17/12)² + (13/6)²) ≈ 2.589, below `gas_clearance_ft`, and later starts are closer. The unrolled distance of 65/12 ≈ 5.417 must not produce a pass. + +**06. Every part of an object counts.** Setup: the gas meter body is 10 from the wall, but its regulator reaches x = 7, z = 23/6. Every start on the pad covers x = 7. Expected: reject. The regulator is 2 from the footprint's front edge. Measuring from the wall or from the meter body's center is wrong. + +**07. Headroom covers the whole footprint.** Setup: the ray 1 ft out from the wall reads 9. A landing spans the whole pad over z = [1.5, 2.5], so it covers the footprint's front strip, with its underside at `headroom_ft` − 1 ± 0.5. Variants: underside at `headroom_ft` + 0.7 ± 0.5, at `headroom_ft` + 0.3 ± 0.5, and unobserved. Expected: reject; pass; manual review; manual review. + +**08. The route decides the result.** Setup: the only pad is [10, 13], and every other check passes there. A garage opening spans s = [3, 7]. In a second variant, a stretch with no wall spans s = [3, 4] instead. Expected: reject in both variants, because the route FAILs for every candidate. No route may interpolate across the missing wall. + +**09. Reach uses routed length.** Setup: the only pad starts L along a straight supported wall, so the route is L. Variants: L = `confident_reach_ft` − 1, + 0 and + 1; L = `max_route_ft` + 0 and + 1; L = `max_route_ft` + 0.2 ± 0.3 and + 0.4 ± 0.3. Then a pad `confident_reach_ft` − 4 along the wall whose route needs 5 of vertical run. Expected: pass, manual review, manual review; manual review, reject; manual review, reject; manual review, because the routed length is `confident_reach_ft` + 1. + +**10. Margins cover equality and compound error.** Setup: T = `gas_clearance_ft`, and every start has the same gas gap d. For each e in {0.3, 0.5, 1.5}, test d = T + e + 0.01, T + e, T − e and T − e − 0.01. Then derive d = T + 0.5 from two positions that each carry ±0.3. Expected: pass; manual review; manual review; reject. The derived gap is manual review, because its error is ±0.6. A recognition confidence score is not a distance error. + +**11. A reject needs the real wall ends and no hidden alternative.** Setup: every candidate on w1 fails gas clearance by a clear margin. The left end is observed, and at s = 12 w1 turns onto an adjacent wall. Variants: (a) only the corner is tapped and the adjacent wall is unseen; (b) the adjacent wall is observed to its real end and every candidate there fails; (c) as b, but a closed gate hides part of the ground; (d) as c, plus a fully observed valid pad and route before the gate. Expected: (a) manual review requesting the adjacent wall past the corner; (b) reject, recording both wall ends and no footprint; (c) manual review requesting the area behind the gate; (d) pass at the valid pad with no photo request. + +**12. Sub-grid intervals and the last start count.** Setup: 2 ft test battery, `opening_clearance_ft` = 3, windows at s = [3, 4] and [12.1, 13.1]. Replace the shared ground fixture: the space between these windows is usable ground, and all other ground is unusable. All non-opening checks pass there. Starts in (7, 7.1) are the only legal ones, and the 2 in grid from s = 0 samples 7 and 43/6 but nothing strictly between 7 and 7.1. A second fixture, independent of both the shared fixture and the one above, has one support interval [6, 8] with a 2 ft battery and every other check set to PASS. Expected: pass with a start in (7, 7.1), such as [7.05, 9.05]; a grid-only manual review is wrong. In the second fixture, start 6 is enumerated, not skipped. + +**13. An UNSURE spot never outranks a pass.** Setup: pad F has route 6 and a gas gap of `gas_clearance_ft` + 0.2 ± 0.3. Pad S has route 10 and passes every check. Then F alone. Expected: pass at S with route 10. With F alone: manual review with no photo request, because F is fully observed and only its margin is uncertain. + +**14. The result records its measurements.** Setup: shared fixture. Expected: pass, recording the policy and version, the footprint relative to the meter anchor, route 6 with its error, and each check's measured value, error and verdict, including facing gap 9 and headroom 9. The photo request is empty. + +[c]: ../02-implementation-plan.md#lane-c-placement-code-solverpy +[r]: ../01-feature-map.md#rules-table-c1-confirm-these +[s]: ../01-feature-map.md#hour-1-agree-on-the-data-format-everyone +[f]: ../01-feature-map.md#c-rules--placement-code diff --git a/docs/research/t3-no-lidar-capture.md b/docs/research/t3-no-lidar-capture.md new file mode 100644 index 000000000..2d354fc0c --- /dev/null +++ b/docs/research/t3-no-lidar-capture.md @@ -0,0 +1,113 @@ +# Measuring clearances on an iPhone without LiDAR + +ARKit world tracking, ground raycasts, and two-view triangulation are candidates for measuring visible boundaries on an iPhone without LiDAR. No device accuracy has been established. This proposal tests one straight wall and a visible rigid facing gap, with overhead triangulation only if that path works. Keep the B4 and B5 fallback in [01-feature-map.md](../01-feature-map.md) outside a tape-checked demonstration. Passing the one-hour experiment does not validate error bounds for automatic approval. + +## Build in this order + +1. A recorder. Per tap, save the frame ID, unrotated image, `camera.transform`, `camera.intrinsics`, tracking state, and tap pixel. `ARView.makeRaycastQuery(from:allowing:alignment:)` uses the live camera, so a tap on a frozen frame needs a ray rebuilt from that frame's pose. +2. One straight wall: its plane from ground contacts, with the meter and openings on it. +3. A visible rigid facing gap, such as a fence. +4. One overhead triangulation, only after step 2 works on a physical non-LiDAR phone. + +Slope, hedges, and hidden boundaries stay in review. + +## Method and fallback per quantity + +| Quantity | Rule input | Method | Fallback | +| --- | --- | --- | --- | +| Wall line, corners | All checks | Ground-contact taps, one plane per straight facade | Shrub: triangulate two facade points | +| Meter | Within 20 ft (Base help page) | Tap the enclosure edges on the wall plane | Tape | +| Gas meter and piping | `gas_clearance_ft` | Triangulate the outer extents | Tape or review | +| Doors, windows | `opening_clearance_ft` | Tap edges, sill, and top | Window wells: review | +| Facing fence or wall | `facing_gap_ft` | Ground taps along the rigid base | Hedge or leaning fence: review or tape | +| Eaves, decks, stairs | `headroom_ft` | Two-view triangulation | Third view, then tape or review | +| Driveway, pool | `drive_clearance_ft`, `pool_clearance_ft` | Nearest visible edge as a ground polygon | Hidden edge: review | +| Ground slope | Set by Base | Review | Spirit-level rise over run | +| Cable route | `review_route_ft`, `max_route_ft` | Wall lengths plus vertical detours | Drift on return: review | + +Measure obstacle separation from the battery boundary where the rule requires it. Record wall-to-facing-obstruction gap separately from free space beyond the battery, and match each rule to its specified measurement origin. Measure heights from local ground. + +## Build the wall from the ground, so blank walls don't matter + +ARKit's vertical plane detection needs wall texture, and blank stucco may never produce a plane. The wall's base usually has texture, and gravity supplies vertical. + +1. With `worldAlignment = .gravity`, up is `g = (0, 1, 0)`. +2. Raycast two wall-ground contacts at least 6.5 ft apart. Prefer `.existingPlaneGeometry` on confirmed ground over `.estimatedPlane`. +3. Normalize the horizontal part of `p2 − p1` to get wall direction `u`. +4. Set `n = u × g`. The sign follows tap order, so flip `n` when `n · (o − p1) < 0`, with `o` the camera position. +5. Tap a third contact. `|n · (p3 − p1)|` validates the plane. + +Each later tap is a ray `o + t·d` from its saved frame (`pixel_ray` in [02-implementation-plan.md](../02-implementation-plan.md)). It meets the wall at: + +``` +t = n · (p1 − o) / (n · d) point = o + t·d +``` + +`hit_plane` in docs/02 does this. Reject `t ≤ 0`, grazing rays, and vertically stacked contacts. Wall texture never enters the math. Tracking still needs texture, so the prompt says "Aim lower to include the ground." + +## Triangulate overhead points from two views + +The user taps the same eave corner in two frozen frames taken 2.5 to 5 ft apart sideways. Take the rays' closest points: + +``` +w = o1 − o2 a = d1·d1 b = d1·d2 c = d2·d2 d = d1·w e = d2·w +t1 = (b·e − c·d) / (a·c − b²) t2 = (a·e − b·d) / (a·c − b²) +point = midpoint of (o1 + t1·d1) and (o2 + t2·d2) +``` + +Reject a negative `t`, an intersection angle under 15°, rays passing more than 2 in apart, or a third view off by more than 0.5°. Depth error grows as about `Z²·δθ / b` for distance `Z`, angular error `δθ`, and baseline `b`: about 3 in at 10 ft with a 3.3 ft baseline and 0.5° error, and 9 in at 16 ft. Taps on two different physical points can still nearly intersect, and no residual detects shared scale error. + +## Accuracy hypotheses + +At 3 to 10 ft, expect 2 to 4 in for accepted ground and wall taps and 4 to 8 in for triangulated points. The ±0.3 ft tap error in docs/02 fits that range but is unmeasured without LiDAR. + +From 4.6 ft, with 0.8 in height error and 0.5° angle error, a ground tap's error is about 1.7 in at 45° look-down, 3.3 in at 30°, and 10 in at 15°. + +Over 30 ft, a 2% scale error gives 7 in. The meter anchor doesn't bound it. Across the 22 in battery depth, 0.8 in errors at each end give about 2.9° of slope uncertainty, so slope stays in review. + +## Acceptance gates + +Accept a measurement only when every gate, each a hypothesis, holds. + +- `ARCamera.TrackingState` is `.normal` and stable for 1 s. This alone proves no accuracy. +- Ground taps look down 30° or more. +- Wall rays are within 60° of the normal: `|n · d| ≥ 0.5`. +- Repeat taps, and the third wall contact, agree within 2 in. +- A missing hit or sparse `rawFeaturePoints` never counts as empty space. + +## Decision rule + +`error` is a bound covering both boundaries and shared bias, not a one-sigma value. For a minimum clearance `T`: + +- PASS when `distance − error > T`. +- FAIL when `distance + error < T`. +- UNSURE otherwise. + +Reverse the inequalities for maximums such as `max_route_ft`. An unobserved area returns UNSURE whatever the margin, because missing coverage is never a pass. Confirm the applicable installation requirements with Base before showing compliance results. + +## Fallbacks + +After one guided retry, the app offers: + +- **Tape entry.** A typed value, stored with source `tape` and an error lane C sets. +- **Needs installer measurement.** The capture ends complete with a named review reason. That is not a pass. + +## One-hour outdoor test + +The test tries to break the method, not calibrate it. Bring a non-LiDAR iPhone running the step 1 recorder, a tape, a spirit level, and two people. One operator never saw the tape values and gets no coaching. Avoid gas fittings and climbing. + +1. Minutes 0 to 5. Confirm `supportsSceneReconstruction(.mesh)` returns false. Check a tapped corner reprojects to the right pixel. +2. Minutes 5 to 15. Tape a 30 ft span, an opening, the meter height, a facing gap, the lowest overhead, and footprint rise over run. +3. Minutes 15 to 40. Make three captures, one by the uncoached operator. Measure a reference before and after a 30 ft walk and back. +4. Minutes 40 to 50. Try blank stucco, bright sun, a shrub-hidden contact, and low parallax. Tap different points in the two views. Build one wall from a planter's base. +5. Minutes 50 to 60. Record maximum and median error per method, abstentions, and capture time. Test decisions on both sides of a threshold. + +The method passes when: + +- Wall, opening, and rigid-ground distances within 4 in of the tape. Facing gap and overhead within 6 in. +- 30 ft span within 8 in. Return-to-reference gap within 4 in. +- Every accepted interval contains the taped value. No false PASS. +- Hidden, low-parallax, mismatched-tap, and wrong-plane cases abstain. +- The uncoached operator finishes in 8 minutes, with at most one corrective prompt per measurement. + +On a failure, narrow the method's claim or send that quantity to review. Never widen error bars to pass. One house can admit a method to the demo, not prove it across homes. diff --git a/docs/research/t3-scoring-protocol.md b/docs/research/t3-scoring-protocol.md new file mode 100644 index 000000000..6b9e7491a --- /dev/null +++ b/docs/research/t3-scoring-protocol.md @@ -0,0 +1,87 @@ +# Scoring protocol for capture pipelines + +For lane D and the reconstruction team. + +A comparison is fair when every pipeline processes the same recording, at the same three marked spots, under one frozen `rules.yaml`, scored against a tape survey no pipeline sees. The honest claim is a case series: at these spots, each pipeline missed these distances by this much and made this many unsafe calls. It is not an accuracy rate or a safety result. + +## Fix the inputs first + +- **One recording per house**, from a non-LiDAR iPhone running [Measure Lab](https://github.com/SamGu-NRX/house-scanning/pull/7) (`experiments/measure-lab`) until the lane A app in the [feature map](../01-feature-map.md) records sessions. LiDAR is a separate labeled condition. +- **Three candidate spots per house.** Mark each 31 in x 22 in footprint with chalk or painter's tape before recording. Pick a clear-looking spot, one near a rule boundary, and one between. Pipelines see the marks in images, never distances. Score spot placement error separately. +- **One policy.** Every pipeline is scored under one complete set of `rules.yaml` parameters that does not change during the study. Unpublished values read "set by Base". +- **One frame list** for every row, plus the close-ups. About 50 frames is a compute guess, not a tested optimum. + +Freeze the frame list, model versions, extraction code, timeouts and the `rules.yaml` hash before anyone opens the survey. + +## Rows to run + +The required rows are AR taps, where a failed raycast counts as missing, and plain photos: Depth Anything 3 metric with intrinsics but no poses, taps or tape ([stack research](../03-stack-research.md)). + +Reconstruction rows such as MapAnything or OOOSplat join only if their predictions arrive before a deadline the team sets before fieldwork. The demo never depends on them. Each declares one scale source: native metric output, the scale distance S below, or AR camera poses. Each variant is its own row. + +## Gates + +- **Fieldwork waits for a working exporter and one verified projection.** Measure Lab saves each keyframe's timestamp, pose, intrinsics and image size from the same `ARFrame`, and the capture-contract probe in [#9](https://github.com/SamGu-NRX/house-scanning/pull/9) checks that metadata. Before fieldwork, still run the [implementation plan](../02-implementation-plan.md) test: project a tapped mark into three saved frames and record the pixel error. +- **Decision scoring waits for a complete policy evaluator** that returns UNSURE when evidence is missing. Distance errors do not wait. +- **House 1 is for debugging.** Freeze all code before houses 2 and 3. + +## Distances to survey + +Survey after recording, so no tape shows in frames. Locate spots by tape offsets from permanent house corners, never AR coordinates. Two people measure each distance independently and re-measure together if they differ by more than 1 cm, an untested starting value. Withhold endpoint photos from every pipeline. + +Say whether each distance starts at the wall or the footprint edge. Mixing them subtracts battery depth twice. Use a tape unless noted. + +- **S. Scale distance.** Two marked points on one wall, 4 to 6 ft apart. Given only to rows that declare it. Never scored. +- **1. Wall length.** Exterior corner to corner. Laser to a target card. +- **2. Usable span.** Nearest limiting edges on each side of the spot. +- **3. Cable route.** Supported cable route from the declared meter connection point to the battery connection point, measured along every segment, including vertical legs and detours around obstacles. Record both connection endpoints and the route polyline. Base's 20 ft from the meter, `review_route_ft`, `max_route_ft`. +- **4. Gas.** Footprint to the nearest gas meter, regulator or pipe, without touching. `gas_clearance_ft`, 3 ft per Base. +- **5. AC unit.** 3 ft per Base. +- **6. Drivable surface.** Footprint to the nearest surface a car can reach. `drive_clearance_ft`. +- **7. Pool.** Footprint to the permanent pool's edge. `pool_clearance_ft`. +- **8. Door.** Footprint side to the nearest jamb, noting the swing. `opening_clearance_ft`, 3 ft in IRC R328. +- **9. Window.** Footprint side to the nearest operable window. `opening_clearance_ft`. +- **10. Window height.** Ground to the lowest operable window opening. +- **11. Facing gap.** Wall to the nearest facing fence, hedge or wall, minimum across the spot's width. Laser at three points. `facing_gap_ft`. +- **12. Headroom.** Ground to the lowest overhead underside above the spot. Vertical laser. `headroom_ft`. +- **13. Ground slope.** Height difference across the footprint depth. Level and ruler. + +Record a missing feature as "absent" and an unreachable one as "not measured", never a guess. Every distance applies to every spot, so the denominator is fixed. Also record which windows open, door swings, drivable surfaces and ground type. + +## Survey and results files + +The formats are in the harness README, `experiments/scoring/README.md` ([#4](https://github.com/SamGu-NRX/house-scanning/pull/4)): a rules file, one survey file per house and one results file per pipeline run. All three use feet and the survey IDs above. Keep real surveys and results beside the capture in `captures/` or `data/`, never in git. Use the recording's sha256 as its capture ID; every results file carries the rules file's sha256. The harness matches those IDs and the rules hash, but cannot independently verify which recording a pipeline processed. + +To score the AR row, write a short map after the walk from each Measure Lab measurement to its survey ID. `score import-measure-lab` turns the session and the map into a results file. + +The harness derives each check's survey outcome from the survey value, its uncertainty and the frozen rules, with the strict rule in its README: a value exactly on a threshold is borderline. A teammate who built no pipeline records the facts a tape cannot settle, such as which windows open and which surfaces a car can reach, before seeing any output. This reference is not an installer. + +## Metrics + +Report every row at every spot. Never average away missing outputs. + +- **Absolute error.** |predicted - true| in inches, with the signed error. Median and maximum per house. +- **Error relative to the deciding threshold.** With margin m = |true - threshold| and survey uncertainty u, report error / max(m, u). At 1 or above, the error could put the run on a boundary or flip the check. If both are zero, report "at threshold". +- **Missing outputs.** Against the fixed denominator, split into unsupported, failed, and not surveyed. +- **Unsafe passes.** A pipeline PASS where the label is FAIL. This matters most. Count a PASS where the label is borderline or UNSURE separately as a missed review, and a pipeline UNSURE or FAIL where the label is PASS as over-caution. Report false rejections beside them, and count a pass or fail made without its measurement separately. +- **Abstentions.** UNSURE rate, split into justified, when the label is borderline or evidence is missing, and avoidable. +- **Capture time.** Walk time, taps and retakes of the shared recording. Photo rows cannot claim a shorter capture from it. +- **Latency.** Upload finished to result shown, cold and warm, with crashes and timeouts. + +## What one to three houses can and cannot show + +They can show: + +- Which distances each pipeline returns, and any unsafe pass at these spots. +- Paired errors on identical input, such as each row's facing-gap error at spot c2. +- Whether errors fit the plan's error bars: plus or minus 0.3 ft for taps, 0.5 ft for the mesh, 1.5 ft for photo detection. Those bars are hypotheses to test. + +They cannot show: + +- An accuracy rate for homes in general. Frames and distances from one house are not independent samples. +- Validated error bars. Tuning them on a house and scoring that house proves nothing. +- Safety. Zero unsafe passes at nine spots is a sample result. +- Homeowner effort. One shared recording cannot measure a first-time user's effort with photos. +- Installation eligibility. Electrical checks are out of scope, and three failed spots do not make a house unsuitable. + +An honest pitch sentence: "On [N] houses, AR taps measured [k] of [n] distances, median error [x] in, [u] unsafe passes."