Device field test: runs 2 and 3 #32

Activity

1. AidenJohnston

   added

   help wantedExtra attention is neededExtra attention is needed

   on Sep 26, 2026
2. SamGu-NRX commented on Sep 27, 2026

   

   SamGu-NRX

   on Sep 27, 2026

   Owner

   More actions

   @AidenJohnston, a coordination note from Sam's agents about beta. Sam's Codex beta QA manager is assembling the next beta and asks for a single writer: before either of you pushes beta again (it's at 45017ba now), agree who assembles it, so two people don't both merge branches into it. Codex's handoff notes are with Sam.
3. SamGu-NRX commented on Sep 27, 2026

   

   SamGu-NRX

   on Sep 27, 2026

   Owner

   More actions

   @AidenJohnston, here are some extra steps for run 3 from the signals lab. They need no app change, only how you stand and what you tape. The evidence is in #59. The main finding: the angle you tap an edge from sets most of its error. Seen within 15° of face-on, a window edge lands within about 1 in; from more than 45° off, it can be off by 24 in.

   Before scanning

   1. From the meter's centre line, tape the window's left and right edges, the sill height, and the gas meter's left and right edges (edges, not just its centre). Measures: the truth that every step below is scored against, and whether Window mark off by up to 5.4 in and gas meter by 4.8 in against the tape #30's left-edge miss happens again.
   2. Put two painter's-tape marks on the ground, each 6 ft out from the wall: one straight out from the middle of the window (the face-on spot) and one straight out from the meter (the meter spot). Tape how far each is from the meter along the wall. Measures: the view angle of every tap, worked out afterwards from these two positions, so you don't have to estimate it.
   3. Write down the phone model. A phone without LiDAR is the more useful one if you have it.

   During the scan
   4. Mark the window twice. Once while standing on the face-on spot, once while standing on the meter spot. Say "face-on" or "from the meter" aloud before each one. Do the same for the gas meter. Measures: the same edge seen from two angles. We expect the face-on marks to land within about 2 in of the tape and the ones from the meter to be off by several inches.
   5. One step-back span photo per window. Stand about 10 ft from the wall, halfway between the meter and the window. Turn the phone sideways so the meter and the whole window are both in frame, and hold still for 3 s so the app keeps a photo. Then take one Camera-app photo from the same spot and tape your distance to the wall. Measures: a meter-to-window distance taken inside one photo, which leaves out the drift that builds up as you walk (#59, drift-anatomy).
   6. ID card at the meter. Hold a bank card or driver's licence flat on the wall right next to the meter while you tap the meter and take its close-up. Measures: ARKit's scale on this phone near the meter, since the card is 85.60 mm wide. Nobody has measured that scale on a phone without LiDAR.

   After the run
   7. Export with Share scan. Keep scan.zip, the screen recording, the Camera photo and the tape readings off git, as in run 1 (experiments/device-field-test). Send them to Sam through the team drive, and they'll go into ~/house-scanning-data/field/, which git ignores.

   One more thing: run 1's export. Could you also send run 1's scan.zip (27.3 MB) through Sam the same way, never git? #59 suggests #30 comes from the wall plane's direction. The app places every tap on a plane whose direction comes from the one raycast at the meter, and a few degrees of error there, seen at an angle, moves a window edge by inches. Run 1's photos can confirm or rule that out: I'd refit the wall from them and compare its direction with the one the app exported.
4. AidenJohnston commented on Sep 27, 2026

   

   AidenJohnston

   on Sep 27, 2026

   CollaboratorAuthor

   More actions

   Build 4.1 field test (TestFlight 4.1, beta 45017ba, iPhone 14 without LiDAR, three runs on 2026-09-26): the runs are done.

   • Run 1 (20:00): the indoor mock wall. Analysis.
   • Run 2 (20:15): the same mock wall. Analysis.
   • Run 3 (20:55): a real house, outdoors at night. Analysis. It was cut short after 73 s on the first task.

   All three runs ended in manual_review. "See it on your wall" drew nothing in runs 1 and 2 (run 3 didn't open it). None of the runs took tape readings, so M-1 to M-3 and the steps from #59 are still open.

   The analyses are in PR #60. The clips are in the pre-release https://github.com/SamGu-NRX/house-scanning-master/releases/tag/field-test-4.1-media and on the new issues. The existing issues #24–#29, #31, #39, #41 and #45 each have a build 4.1 comment.

   New issues

   P0:

   • "See it on your wall" draws nothing on phones without LiDAR (regression from #41) #67 "See it on your wall" draws nothing on phones without LiDAR (regression from AR view draws the battery over people and objects in front of it (R-4) #41)
   • Ground height taken from a table plane, so every height is about 3 ft too high #62 Ground height taken from a table plane, so every height is about 3 ft too high
   • Meter tap on an estimated plane put the wall 1.1 m behind the real wall and 46° off, and nothing corrects it #69 Meter tap on an estimated plane put the wall 1.1 m behind the real wall and 46° off, and nothing corrects it
   • An unwalked, tapped wall still gets a spot (here inside the house) with wall backing and cable route passing #76 An unwalked, tapped wall still gets a spot (here inside the house) with wall backing and cable route passing

   P1:

   • The first "Tilt down… at your meter" task blocked the start in all three runs: no timeout, and the two-spots rule is never explained #77 The first "Tilt down… at your meter" task blocked the start in all three runs: no timeout, and the two-spots rule is never explained
   • The target ring and arrows show no progress, aren't explained, and the arrow can contradict the card #81 The target ring and arrows show no progress, aren't explained, and the arrow can contradict the card
   • Corner flow: no "the wall just ends" answer, the refusals loop, and a surface behind the end post was accepted as the next wall #70 Corner flow: no "the wall just ends" answer, the refusals loop, and a surface behind the end post was accepted as the next wall
   • Wall ends still land in the wrong place: at the meter on a short side, or 13 ft from where the phone went #71 Wall ends still land in the wrong place: at the meter on a short side, or 13 ft from where the phone went
   • An AC is one tap, modelled as a 1 ft square, so the battery was placed under the AC stand-in #72 An AC is one tap, modelled as a 1 ft square, so the battery was placed under the AC stand-in
   • AR overlays drift off their objects after walking away (window outline drawn over the panel) #73 AR overlays drift off their objects after walking away (window outline drawn over the panel)
   • Result text shows internal ids ("objects[1] ac"), says "a AC unit", and shows the demo-rules notice to homeowners #74 Result text shows internal ids ("objects[1] ac"), says "a AC unit", and shows the demo-rules notice to homeowners
   • Ground requests driven by placeholder rules ask for 14–16 ft of ground nobody can show, and the copy misdescribes the span #75 Ground requests driven by placeholder rules ask for 14–16 ft of ground nobody can show, and the copy misdescribes the span
   • 0.16 ft/ft drift, added per feature, makes any spot more than a few feet from the meter unresolvable #79 0.16 ft/ft drift, added per feature, makes any spot more than a few feet from the meter unresolvable
   • "It's too dark to see the wall" flickers, hides the task and its button, and asks for a light that is already on #80 "It's too dark to see the wall" flickers, hides the task and its button, and asks for a light that is already on
   • No way to finish a scan early: stopping took 7 rapid "Can't get there" taps, each silently setting ends #82 No way to finish a scan early: stopping took 7 rapid "Can't get there" taps, each silently setting ends

   P2:

   • "Still needed" asks for wall past a limit end (wall to 26 ft 2 in; the wall ends at 19 ft 9 in) #78 "Still needed" asks for wall past a limit end (wall to 26 ft 2 in; the wall ends at 19 ft 9 in)
   • "A closer spot may exist on the left" shows after the left end was reached, or when the spot is already on the left #83 "A closer spot may exist on the left" shows after the left end was reached, or when the spot is already on the left
   • The walk card changes every ~3 s, and target distances jump backwards #84 The walk card changes every ~3 s, and target distances jump backwards
   • "Can't get there" reads wrong on tilt tasks; testers look for a "that's the ground" answer instead #63 "Can't get there" reads wrong on tilt tasks; testers look for a "that's the ground" answer instead
   • The overhead task is asked from 19 ft away, and its card's button overlaps the text #64 The overhead task is asked from 19 ft away, and its card's button overlaps the text

   P3:

   • "Looks complete" sends with the window's open/shut answer and the ground type unanswered #65 "Looks complete" sends with the window's open/shut answer and the ground type unanswered
   • "Ending the wall here leaves out N ft you walked" shows during tilt tasks, unprompted #66 "Ending the wall here leaves out N ft you walked" shows during tilt tasks, unprompted
   • The shared scan zip is 90–120 MB #68 The shared scan zip is 90–120 MB

   Test process, for the next runs

   • Turn on a Focus mode. Notification banners covered the task card in runs 1 and 2, and a Low Battery alert covered it in run 2.
   • Charge the phone first. It went from 22% to 13% in run 2 and from 12% to 2% in run 3, where the recording stopped during the upload.
   • Keep run 3's full recording, photos and scan private. They show identifying details of a real house. The only public run 3 clips are trimmed to a stretch where none are visible. Use the same care with any future real-house run.
5. SamGu-NRX commented on Sep 27, 2026

   

   SamGu-NRX

   on Sep 27, 2026

   Owner

   More actions

   @AidenJohnston, beta QA acceptance update after the build 4.1 reports in #60. The next candidate needs evidence for #62 ground selection, #69 contradictory wall anchoring, #76 unsupported placement/PASS claims and #67 blank non-LiDAR AR. The existing Astra caretakers linked overlapping guidance and AR findings; there is no new review clearance yet.

   The new iOS integration head 4b71161d3bcc0b5d8ffd8fe438899f9fbe0fd672 contains ground/anchor/source fixes, with CI still running at inspection. Its cleanup repair is narrower than the requested field-export retention: KeyframeStore.swift:49 still deletes prior scans, and ScanFolderCleanup.swift:15 snapshots every folder except the current one. That protects a newer scan from a late cleanup but does not preserve the prior export across launch/start-over. Please keep the separate retention requirement with S3 before the next measured field build. No second implementation lane is needed.

   The completed 20:55 run remains useful field evidence; it did not perform the tape/edge/scale-reference steps from #59. Keep those acceptance checks open for the next measured run, and keep the full real-house media private. The earlier single-writer beta assembly agreement remains pending; this QA chat has not pushed beta or uploaded a build.
6. SamGu-NRX commented on Sep 27, 2026

   

   SamGu-NRX

   on Sep 27, 2026

   Owner

   More actions

   @AidenJohnston, a coordination note from Sam's agents: Sam asked Codex's beta QA manager to assemble a testable beta candidate now. It's the single beta writer for this round. It builds on beta 45017ba plus S3's latest, and layers your runtime fixes in. Please don't push beta until it hands back the tested candidate SHA. If you have fixes you want included, point it to them here or on your PRs.
7. SamGu-NRX

   added a commit that references this issue on Oct 15, 2026

   Add the COLMAP pose backbone with ARKit priors (#32)

   ...

   74d6fb4