Skip to content

experiments: device field test runbook and run 1 of the TestFlight build - #23

Merged
AidenJohnston merged 4 commits into
mainfrom
t3/device-field-test
Sep 27, 2026
Merged

AidenJohnston merged 4 commits into
mainfrom
t3/device-field-test

Conversation

@AidenJohnston

Copy link
Copy Markdown
Collaborator

What changed

The first run of the House Scan TestFlight build (1.1, t3/ios-mvf 657ab28) on a real iPhone uploaded a scan the server couldn't use. The phone walked about 19 ft of wall and kept 44 photos, but both wall ends were saved within 4 in of the meter. So the scan reported no observed wall or ground, and the server answered manual_review with no spot.

experiments/device-field-test/ adds:

  • README.md: the run 1 result, how the walk was lost (with code links), and the app's measurements against a tape.
  • RUNBOOK.md: every device-only claim from the t3/verification product description, with test IDs to say aloud while screen recording.
  • coverage_replay.py: replays a scene.json's photos through the app's coverage rules (uv run coverage_replay.py scene.json).
  • setup-plan.html/.png: how the mock wall was prepared, measured and walked.

Why the walk was lost

"Can't get there" sets that side's end at GuidancePlanner.reach (ScanEngine+Actions.swift L315). That's the unbroken stretch from the meter where the wall and ground are both covered. The screen recording shows both taps: the left one before any ground was seen, and the right one at the corner, 16 ft away, while the ground row next to the meter was still amber (seen once). The tilt prompts never asked for the ground in front of the meter, so reach stayed at 0 ft and both ends landed at the meter. After that, coverage past the ends is dropped, even though the walk kept taking photos and accepting marks up to 19 ft away without saying so. The logic is unchanged at the t3/ios-mvf head. The README lists fixes for the lane A owner to choose from; no app code changes here.

Measurements, app minus tape: window right edge −0.5 in, sill +1.9 in, left edge +5.4 in, gas stand-in 4.8 in nearer the meter.

Verification

  • I sent the exported scene.json to the deployed server again: manual_review, ends at ±0.328 ft.
  • I checked coverage_replay.py against Camera.pixel(of:) and CoverageMap, and ran it on the run 1 scan. Its output is in the README.
  • The tester's local session reviewed the 1:54 screen recording frame by frame and confirmed the two taps and the strip's state; the README scores the device-only claims it shows.
  • Not done yet: runs 2 and 3, and haptics (not narrated).
  • The recording, photos and raw tape readings stay out of git.

Checklist

  • The scope is focused; related changes belong together
  • I explained what changed and why
  • N/A before/after images: no app code changes
  • N/A video: the recording shows a real site and stays out of the repo

🤖 Generated with Claude Code

AidenJohnston and others added 3 commits September 26, 2026 13:31
Adds a runbook of every device-only claim from the product description on
t3/verification, the setup plan for the first site, and a script that
replays a scene.json's photos through the app's coverage rules.

Run 1 on a mock wall uploaded a scan whose ends sat within 4 in of the
meter: "Can't get there" places an end at GuidancePlanner.reach, which
stayed at 0 ft until photo 43 of 44 because the ground in front of the
meter was only photographed on the way back. The server answered
manual_review with no spot.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The recording shows the two "Can't get there" taps: the left one before
any ground was seen, the right one at the corner while the ground row by
the meter was still amber. Scores the device-only claims it shows and
notes the tilt prompts never asked for the ground in front of the meter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 0d5fdb97-8f26-4582-9057-92ffa883a6c0


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the size:XL 500-999 effective changed lines. label Sep 26, 2026
@codemetrics-ai

Copy link
Copy Markdown

Note

Review skipped

Codemetrics is installed on this repository, but it hasn't been added to your team yet.

To enable AI code reviews for this repo, go to your dashboard and add house-scanning-master to your team's repositories.

Spell out the positions and photos the coverage needs before a "Can't get
there", add the see-behind check for builds with LiDAR occlusion, run C-4
before C-3, and group run 3 into short recordings with O-4 last.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SamGu-NRX added a commit that referenced this pull request Sep 26, 2026
- docs/00 reports the first TestFlight run on a real wall (PR #23),
  says rules.yaml marks every value without a public source as a
  placeholder, and calls Gemini the planned box detector, since no
  branch uses it yet.
- docs/04 notes that the demo has no 1 ft wall-distance rule because
  the solver places the battery flush.
- AGENTS.md and README point to PR #23, PR #17 and the exact docs/00
  section, following the rewritten prompt-writing skill.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@SamGu-NRX

Copy link
Copy Markdown
Owner

Before this merges, please cut its write-up to the experiment format in #33 (experiments/README.md). The README should stay under about 200 words: the question and pass criteria, the command, the result with its numbers, and what it changed. Put generated tables in results/ with the command that made them. docs/00 already carries the one-line finding.

SamGu-NRX added a commit that referenced this pull request Sep 26, 2026
The evidence section said every result came from public datasets,
but PR #23 has since run the app on a real phone.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@SamGu-NRX

Copy link
Copy Markdown
Owner

@AidenJohnston Sam’s experiment-format request is still open at 321ea7e. Please shorten the README to about 200 words using experiments/README.md, and keep any run evidence from the real site in git-ignored data/. I am reviewing this PR against main, including the replay and setup-plan artifacts. I will post any correctness or privacy findings separately. I will not edit your branch or #46.

@SamGu-NRX SamGu-NRX left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main-track review findings at the pinned head. Fixes belong to the branch writer; no source changes made by this caretaker.

Comment thread experiments/device-field-test/coverage_replay.py
@SamGu-NRX

Copy link
Copy Markdown
Owner

@greptileai

@greptile-apps

greptile-apps Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 4/5

[Low risk] Adds experiment documentation and test tooling.

The PR appears safe to merge, with non-blocking corrections needed to make the replay and runbook useful across the intended tests.

Fix All in Claude CodeFindings

  1. P2 Left reach is capped ▶
  2. P2 Checks follow screen transition ▶
Fix with agent prompt
### Issue 1
experiments/device-field-test/coverage_replay.py:92
The replay tracks only three cells left of the meter, so a scan with more than 1.5 ft of covered wall there still reports at most 1.5 ft of left reach. Unlike the right-side limit, this cannot be changed with `--cells`, making the diagnostic misleading for walls whose meter is farther from the left end.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
experiments/device-field-test/RUNBOOK.md:41-43
F-1's successful meter tap advances to the close-up, but F-3 and F-4 come afterward in this full-flow run. A tester following the listed order can no longer perform those Find meter checks. The onboarding-only checks after O-1 have the same problem. Put the checks before the actions that leave their screens, or specify separate attempts.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Summary

This PR records the first iPhone field test, adds a device-test runbook and site plan, and introduces a script for replaying exported photo coverage. The first run documents a scan whose wall ends landed near the meter despite a longer photographed walk.

Reviews (1) · Last reviewed commit: "experiments: how to fill the strip at th..."

Comment thread experiments/device-field-test/coverage_replay.py
Comment thread experiments/device-field-test/RUNBOOK.md
@AidenJohnston
AidenJohnston merged commit c2698f1 into main Sep 27, 2026
3 checks passed
SamGu-NRX pushed a commit that referenced this pull request Oct 15, 2026
* Sync a10-sparse-photo-flow breadth work from the Project store (no result file yet)

* Sync a10-sparse-photo-flow breadth work from the Project store (no result file yet)

* Sync a10-sparse-photo-flow breadth work from the Project store (no result file yet)

* Sync a10-sparse-photo-flow breadth work from the Project store (no result file yet)

* Sync a10-sparse-photo-flow breadth work from the Project store (no result file yet)

* Sync a10-sparse-photo-flow breadth work from the Project store (no result file yet)

* Sync a10-sparse-photo-flow breadth work from the Project store (no result file yet)

* Sync a10-sparse-photo-flow breadth work from the Project store (no result file yet)

* Sync a10-sparse-photo-flow breadth work from the Project store (no result file yet)

* Sync a10-sparse-photo-flow breadth work from the Project store (result file A10-sparse-photo-flow.md)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL 500-999 effective changed lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants