Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/research/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Research notes

Notes from the agents working beside the four lanes. They build on the team plan in [docs/00](../00-overview.md) through [docs/05](../05-live-guided-survey-hld.md). Each answers one question for one reader. They propose; the lane owners decide.

| Note | Question it answers | For |
| --- | --- | --- |
| [t3-capture-surfaces.md](t3-capture-surfaces.md) | Which phone surface can measure what, and how does Android get covered later? | Everyone |
| [t3-no-lidar-capture.md](t3-no-lidar-capture.md) | How can an iPhone without LiDAR measure what the placement checks need, and how do we test that before relying on it? | Lanes A and B |
| [t3-first-try-capture.md](t3-first-try-capture.md) | How does the capture know it has enough before the homeowner leaves, and what does the homeowner see? | Lane A |
| [t3-lane-c-review.md](t3-lane-c-review.md) | What must the solver settle before it can approve or reject a site, and which test cases prove it? | Lane C |
| [t3-scoring-protocol.md](t3-scoring-protocol.md) | How do we compare every capture method fairly on one to three houses, and what can we claim? | Lane D |

Base's own criteria never appear here: thresholds without a public source are named `rules.yaml` parameters.
49 changes: 49 additions & 0 deletions docs/research/t3-capture-surfaces.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Capture surfaces: what each phone surface can measure

Most homeowners don't have a LiDAR iPhone, and they reach Base through a link. This note compares the places a capture can run, so the team can decide which surface serves which phone. The references below were collected on 2026-09-25. Nobody has tested these capabilities on a phone for this project yet.

## Capabilities by surface

| Surface | AR tracking in real units | Planes and raycasts | Depth or mesh | Camera frames with pose | Entry friction |
| --- | --- | --- | --- | --- | --- |
| iOS app, no LiDAR | Yes (ARKit world tracking) | Yes | No | Yes | App Store or TestFlight install |
| iOS app, LiDAR (Pro models) | Yes | Yes | Yes (scene depth, mesh) | Yes | Same |
| App Clip on iOS 17+ | Yes, ARKit and camera allowed | Yes | Unverified inside a clip; would need a LiDAR phone | Yes | No install; opens from a link, a Messages bubble, a QR code or NFC |
| iPhone Safari | No: Safari 27 still has no WebXR AR | No | No | Video without pose | None |
| Android Chrome | Yes (WebXR on ARCore phones) | Yes, plane detection since Chrome 147 | Depth from motion | Yes (raw camera access) | None |
| Android app | Yes (ARCore) | Yes | Depth from motion, best at 0.5 to 5 m | Yes | Play Store install; Play Instant ended in December 2025 |
| Photos only (Base today) | No | No | Model estimates only | No pose | A link with 7 to 9 photos |

CI builds and unit-tests the iOS app and the web toolchain, and agents can drive web pages in a browser. AR tracking runs only on a real phone outdoors, so every end-to-end check of the AR path needs a person holding the phone.

## What the table means for the product

- **On iPhone, AR needs native code.** The App Clip gets it without a full install. A clip opened from a QR code or NFC tag must stay under 15 MB; a clip opened only from links can reach 100 MB on iOS 17 and later. Uploads must finish in the foreground, because clips can't use background transfers.
- **Android Chrome is a no-install capture candidate.** Before treating it as an alternative to native ARCore, verify saved camera images with matching poses and calibration, negotiated depth export, and interrupted-session recovery on supported devices. Until then, native Kotlin remains the proposed Android capture path.
- **Everyone else falls back to photos or video processed on the server.** The best learned models report roughly 13 to 15 percent scale error on research benchmarks. Nobody has measured them on house walls, so the photo baseline gets scored against the same tape-measured spots before anyone trusts an error bound.
- **No published test covers ARKit accuracy without LiDAR on an exterior walk like this one.** We have to measure it ourselves with a tape before the rules' margins can rely on it.
- **Generative world models are for pictures, not measurements.** World Labs Marble and Atlas and Runway's GWM fill in regions the camera never saw. A filled-in corner is not evidence that the corner is clear.

## Reaching Android later

Keep Swift capture for the hackathon. Android work starts after one iPhone capture runs end to end, and it should not change the server.

- **Add a Kotlin ARCore capture adapter.** ARCore gives metric poses, planes, hit tests and camera intrinsics without a depth sensor. It has no ARKit-style scene mesh, and [raw depth](https://developers.google.com/ar/develop/java/depth/raw-depth) usually has zero confidence on textureless walls. Facing gaps and headroom stay UNSURE unless another source measures them. The adapter can run in a native Android app or behind an Expo shell.
- **Keep the upload format independent of ARKit and LiDAR.** A minimal, versioned manifest records the capture provider, device, session ID, units, and each frame's timestamp, pose, intrinsics and image size. Depth and mesh are optional. A missing mesh means unknown, never clear space.
- **Treat Android WebXR export as a short experiment, not parity.** The table shows what Chrome exposes. The proposed WebXR capture path has not demonstrated saved camera images with matching poses and calibration. Test that first: [raw camera access](https://immersive-web.github.io/raw-camera-access/), [depth sensing](https://www.w3.org/TR/webxr-depth-sensing-1/) on both its CPU and GPU paths, reprojection of saved frames, and an interrupted session. If export fails, the phone routes to photos before the walk starts.
- **Guided photos stay the broadest fallback**, including phones without ARCore. The app should check the photos for the deciding views before the homeowner leaves.

## Closest prior art

- Base's current flow sends a link that asks for 7 to 9 photos: the meter, both sides of the installation area, the adjacent wall, the main breaker and the disconnect. Engineering follows up within two days when it needs more.
- Hover asks for at least eight guided exterior photos and returns a measured model, claiming about 5 percent accuracy after 2 to 4 hours of processing.
- SiteCapture and Qmerit send homeowners a browser link for photo surveys with no app, which is the friction bar an App Clip has to match.

## Sources

- ARKit without LiDAR: [world tracking](https://developer.apple.com/documentation/arkit/understanding-world-tracking), [plane detection](https://developer.apple.com/documentation/arkit/arworldtrackingconfiguration/planedetection), [raycasting](https://developer.apple.com/documentation/arkit/raycasting), [scene reconstruction needs LiDAR](https://developer.apple.com/documentation/arkit/arworldtrackingconfiguration/supportsscenereconstruction%28_%3A%29)
- App Clips: [size limits](https://developer.apple.com/help/app-store-connect/reference/app-uploads/maximum-build-file-sizes), [allowed functionality](https://developer.apple.com/documentation/appclip/choosing-the-right-functionality-for-your-app-clip), [testing](https://developer.apple.com/documentation/appclip/testing-the-launch-experience-of-your-app-clip)
- iPhone Safari: [Apple forum answer on immersive-ar](https://developer.apple.com/forums/thread/756850?answerId=790772022#790772022), [Safari 27.0 features](https://webkit.org/blog/18325/webkit-features-for-safari-27-0/)
- Android: [WebXR on ARCore](https://developers.google.com/ar/develop/webxr/arcore-comparison), [Chrome 147 release notes](https://developer.chrome.com/release-notes/147), [ARCore Depth API](https://developers.google.com/ar/develop/depth), [Play Instant notice](https://developer.android.com/topic/google-play-instant)
- World models: [Marble API FAQ](https://docs.worldlabs.ai/api/faq), [Atlas announcement](https://www.worldlabs.ai/blog/atlas), [Runway GWM-1](https://runway.com/research/introducing-runway-gwm-1)
- Prior art: [Base photo review](https://help.basepowercompany.com/en/articles/10280641), [Hover exterior scans](https://help.hover.to/en/articles/9185612-exterior-scans), [Hover accuracy](https://hover.to/architects/), [SiteCapture self-surveys](https://sitecapture.com/self-surveys-inspections/)
105 changes: 105 additions & 0 deletions docs/research/t3-first-try-capture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# First-try capture: enough evidence before the homeowner leaves

[docs/05-live-guided-survey-hld.md](../05-live-guided-survey-hld.md) describes the live guided survey: automatic capture, the coverage display, next-view planning and recovery. This note fills in what the capture needs before the homeowner leaves: the evidence per check, the finish rule, a proposal to stop early, the close-up gates and the first trial.

The capture has enough when one candidate spot has a result for every placement check backed by saved keyframes, the meter and panel close-ups have passed their photo checks, and the server has acknowledged every upload. Anything the phone cannot settle stays UNSURE and goes to installer review with its reason. It never becomes a later photo request. This is a direction for lane A, not a hackathon promise.

## What counts as evidence, by check

Each check returns PASS, FAIL or UNSURE, as in [02-implementation-plan.md](../02-implementation-plan.md). PASS needs the deciding area in retained keyframes and a margin larger than the error. The error bars are team estimates, not measurements: 卤0.3 ft for AR taps, 卤0.5 ft for the LiDAR mesh, 卤1.5 ft for photo detections. Freeze one set of `rules.yaml` values before capture.

| Check | Evidence that settles it | Threshold |
|---|---|---|
| Wall and footprint | Wall behind the whole 31 x 22 in footprint, seen from two positions | Base: within 1 ft of the wall |
| Ground | Surface and slope of the ground under and around the footprint | Chosen policy; Base's page gives a 3 by 3 ft area |
| Gates and paths | Gate swings and paths past the spot, with clear widths | Chosen policy |
| Nearby equipment | Footprint and service side of each nearby AC unit, battery or other equipment | Base: 3 ft from AC units and other batteries |
| Above the spot | Vents and wall boxes on the wall above the footprint | Chosen policy |
| Gas | Gas meter and pipes located; ground within the clearance of every footprint edge seen | `gas_clearance_ft`; Base's page says 3 ft |
| Driveway | Nearest boundary measured, plus retained views showing connected vehicle access, including around corners; unseen connections remain UNSURE | `drive_clearance_ft`, set by Base |
| Pool | Pool edge seen, or the area within the clearance seen empty | `pool_clearance_ft`, set by Base |
| Doors and windows | Position measured; whether it opens comes from the homeowner | `opening_clearance_ft`; IRC R328 says 3 ft |
| Facing gap | Nearest fence, hedge or wall across the full footprint width | `facing_gap_ft`, set by Base |
| Headroom | Lowest overhead object across the installation and its working space | `headroom_ft`, set by Base |
| Cable route | Continuous wall from meter to spot, no door or garage crossing | `max_route_ft`; Base: within 20 ft of the meter |
| Working space | Footprint outside the space in front of meter and panel | NEC 110.26: 30 in x 36 in x 6.5 ft |
| Disconnect, if the install uses one | Free wall space for the disconnect box | NEC 706.15: within sight, and within 10 ft or lockable |

This table summarizes what to capture. It is not the acceptance checklist: before capture, list every check the chosen policy applies. A check without evidence stays unsure and cannot count toward a pass.

A non-LiDAR phone has no mesh, so facing gap and headroom stay UNSURE. Hypothesis to test: taps on the wall and the facing fence at both footprint edges, raycast onto detected planes, bound the gap within 卤0.3 ft.

Ask the homeowner only for facts a camera cannot establish: whether a window opens, whether they would move the bins, whether a street-facing spot is acceptable. Answers about hazards steer capture but never pass a check.

Proposed `scene.json` addition for lanes A to C: each check result lists its supporting keyframe IDs.

## Coverage tracker and finish rule

The tracker extends the coverage display in [docs/05 section 2](../05-live-guided-survey-hld.md#2-customer-journey) with a failed state and a strip showing the unrolled wall with the meter at zero, plus ground and overhead bands. Cells are 6 in wide, a display guess to try. Each cell is unseen, seen but not enough, enough, or failed. Tapping one shows the single view it needs. In AR, a fog on the wall clears where the camera has looked.

The fog is guidance, never proof that space is clear. A cell clears only from retained keyframes with normal tracking, from two positions. A non-LiDAR phone still cannot tell that a bin hid the ground behind it; the server's checks decide.

**Finish** unlocks when all four hold:

1. One candidate passes every placement check with linked keyframes, or the homeowner has reached both walkable limits and each unknown is recorded for installer review.
2. Both close-ups passed their gates, or the homeowner chose "Can't get a clear shot".
3. The server acknowledged every upload. Saved offline is not finished.
4. For the hackathon, a teammate acting as Base's reviewer has checked the packet while the homeowner is still outside.

## Early stop

This is a proposal the team has not approved. A2 in [01-feature-map.md](../01-feature-map.md) requires walking to both wall ends, and [docs/05 section 7](../05-live-guided-survey-hld.md#7-deterministic-criteria-evaluation) keeps that gate until the team approves a tested completion rule. The proposal: one fully evidenced candidate ends the walk. Two limits apply:

- Evidence can extend past the candidate. A gas meter 2 ft beyond the footprint still needs its surrounding ground seen.
- Proving the shortest route is optional. If the other side of the meter is unseen out to the same distance, the result says "A closer spot may exist on the left".

With no candidate on the meter wall, continue to reachable adjacent walls when more placement evidence is needed. Unseen alternatives remain unknown. This capture checklist does not define installation eligibility or authorize a site-level rejection.

## Close-up gates

- **Meter label.** Run Vision's `RecognizeTextRequest` with `recognitionLevel` accurate and `usesLanguageCorrection` off. Require the text inside the reticle, all label edges in frame, and matching reads from two frames. OCR confidence cannot say which of the meter's numbers is the meter number; the server decides.
- **Blur.** Laplacian variance on the label crop, not the whole frame.
- **Glare.** Reject when near-white pixels cover any recognized character's box.
- **Panel.** Three shots: panel and surrounding wall, breakers with the door open, main breaker rating up close.

All cutoffs are hypotheses to tune. After two failed tries, "Can't get a clear shot" sends that check to installer review instead of looping.

## What the homeowner sees

1. A text with a link. The hackathon build opens the development app.
2. Camera permission, then "Go to your electric meter" and **I'm here**.
3. The meter close-up. The shutter fires itself when the gates pass.
4. "Tap the meter." A raycast onto a detected wall plane sets the anchor. With no plane, it asks them to step closer; it never guesses a depth.
5. "If there's room, step back until the whole meter box and the ground below it are in view."
6. The walk with strip and fog. One question at a time, two large buttons.
7. The early stop, then the panel while the wall uploads.
8. Corrections on saved frames. The homeowner marks a missed object in a saved keyframe. Its pixel and stored pose define a ray, placed on a validated wall plane only for an object on that wall. Otherwise its position stays UNSURE.
9. The reveal at the meter: a box on the meter's anchor and a cable line, with clearances as numbers on tap.

Opening the panel:

> Open the panel's hinged door. Don't unscrew or remove the metal cover behind it, and don't touch the breakers. If the door is locked or stuck, tap **Skip**.

Borderline reveal:

> This spot is 3 ft 2 in from the gas meter. The rule is 3 ft and our measurement can be off by about 4 in, so an installer will check it. The panel still needs an electrician's review.

## Recovery

These add to the operational cases in [docs/05 section 9](../05-live-guided-survey-hld.md#9-reliability-latency-and-data-handling).

- **Tracking drops.** Coverage credit stops and the prompt changes, with no modal: "Slow down" for `excessiveMotion`, "Aim at a corner or somewhere with more texture" for `insufficientFeatures`.
- **Interruptions.** Keyframes and close-ups go to disk after each step. On return, `sessionShouldAttemptRelocalization(_:)` returns true and the app shows a saved meter view: "Point at the meter like this."
- **Failed relocalization.** Start a new coordinate frame: re-tap the meter and re-walk. One matching point does not fix heading, and a 5 degree heading error moves a point 20 ft away by about 1.7 ft.
- **Uploads.** Foreground only, resumed on return, never restarted.

## First trial

Run one manually guided end-to-end capture on a non-LiDAR iPhone, with the operator keeping the evidence checklist by hand. First, a second person lists every hazard and tape-measures each deciding clearance. Stage four cases:

1. A clear candidate. Expected: without validated measurements, facing gap and headroom stay UNSURE and go to installer review, so the spot cannot fully pass.
2. A candidate whose ground is hidden behind a bin. Expected: a view request during the walk, or UNSURE.
3. A gap within the error bar of its threshold. Expected: UNSURE, installer review.
4. An electrical label made unreadable by tape or glare. Expected: the gate refuses it, then installer review after two tries.

Fail the trial if any PASS contradicts the independent measurements or hazard inventory, or if a packet is marked evidence-complete while a required view, readable label or measurement is missing. Record evidence-complete, review-pending and installer-assessment outcomes separately, along with unresolved capture needs. Also record walk time, retakes, questions asked and wall length walked.
Loading