Repository navigation
ios tests: audit the walk strictly once held and in motion to report; triage of the UI test failures - #165
ios tests: audit the walk strictly once held and in motion to report; triage of the UI test failures#165AidenJohnston wants to merge 5 commits into
Conversation
The full-flow tests audited the walk the moment it appeared, while the autopilot was still marking features, setting the ends and tilting up over a playing replay. Each audit pass caught a different crossfade; a copy fading out has no identifier, so transient issues from two passes matched on a label or on the "no element" key and failed as one lasting issue (runs 36304552823, 36305050676, 36307476187). The app now leaves <phase>.held in the gate folder while it waits for the test to finish with a screen, and the walk is audited once that file appears, on the settled "Done with this wall" state. Every state the walk passes through is still audited frozen by ScreenStatesUITests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (4)
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe app now writes a held-phase marker while waiting for a phase gate file. UI tests use the marker to coordinate report-only audits of moving screens with the existing settled-screen audits. ChangesHeld-screen audits
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~12 minutes Change: Other Sequence Diagram(s)sequenceDiagram
participant FullFlowUITests
participant ScanEngine
participant GateFolder
participant audit
ScanEngine->>GateFolder: write phase.held while waiting for phase gate file
FullFlowUITests->>audit: audit moving screen in report-only mode
FullFlowUITests->>GateFolder: wait for phase.held
GateFolder-->>FullFlowUITests: phase.held exists
FullFlowUITests->>audit: audit settled screen
Suggested reviewers: Merge Risk: ⚪ Minimal · up to The wall-walk screen is audited in motion and then strictly after the app signals it is held. No concrete merge-blocking issue remains. Security Architecture ReviewSecurity architecture risk: ⚪ Minimal · up to The coordination change remains explicitly enabled through launch options. The supported UI tests isolate each run in a fresh directory and retain a strict held-screen audit. No material security risk introduced or worsened by this PR was identified. Retained concerns Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Note Review skipped Codemetrics is installed on this repository, but it hasn't been added to your team yet. To enable AI code reviews for this repo, go to your dashboard and add house-scanning-master to your team's repositories. |
|
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The walk's audit in motion caught real bugs that only show mid-transition: the map legend's entries drawn over each other (1faa79b), the two reply titles half-drawn over each other (b1a2444) and the translucent reply fill (397b6d6). Auditing only the held walk would have hidden them. The walk is now audited on its first appearance as well, report-only: findings are attached as audit-wallWalk-moving and printed, as with HOUSESCAN_AUDIT_REPORT_ONLY, and the strict audit stays on the held walk. Also: LaunchOptions documents <phase>.held, and the LiDAR test's comment no longer times the walk. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… a held screen A test audits a held screen only once <phase>.held appears, and the walk's map check after it can wait 90 s; the two-minute gate could expire first on a slow runner and let the app leave mid-audit (Greptile on #169). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> (cherry picked from commit c3aeabe)
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
@greptileai review Please review the current head |
| moving.lifetime = .keepAlways | ||
| add(moving) | ||
| do { | ||
| try audit(app, screen: "\(phase)-moving", reportOnly: true) |
There was a problem hiding this comment.
Single-pass findings disappear The new in-motion audit reports only findings seen in at least two passes, which are six seconds apart. A defect visible only during a brief walk transition can be gone by the second pass, leaving no
AUDIT line or attachment for anyone to review. Record first-pass findings in this report-only audit so those transitions remain visible.
Prompt To Fix With AI
This is a comment left during a code review.
Path: ios/HouseScanUITests/FullFlowUITests.swift
Line: 327
Comment:
**Single-pass findings disappear** The new in-motion audit reports only findings seen in at least two passes, which are six seconds apart. A defect visible only during a brief walk transition can be gone by the second pass, leaving no `AUDIT` line or attachment for anyone to review. Record first-pass findings in this report-only audit so those transitions remain visible.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Caretaker check at 4925f37: confirmed, but the filtering predates this PR and only diagnostic evidence is lost. The new moving audit runs the existing collector with reportOnly (FullFlowUITests.swift:327), and AccessibilityAudit.swift:118-131 drops findings seen fewer than twice. A defect that clears before the second pass, six seconds later, leaves no AUDIT line. Keeping two sightings as the failure rule is deliberate (lines 17-28). Report-only mode could still attach findings seen once. Non-blocking for this PR.
There was a problem hiding this comment.
That distinction makes sense. Keeping the two-sighting threshold for failures is deliberate, while retaining single-sighting findings only in reportOnly would improve diagnostic visibility without changing pass/fail behavior. Since the filtering predates this PR and the impact is limited to optional evidence, I agree this is non-blocking and can be tracked as follow-up hardening.
The iOS workflow on
t3/ios-mvfwas red on intermittent UI test failures, so pull requests into it inherited a red check. Commits already on the base have fixed every one of them; the evidence is below. The current base no longer fails the tests this touches: 20 of 20 Mac mini runs passed without any change from here, and base push run 36318952517 on 6be087d passed with the full audit. So this PR is optional hardening of the full-flow audit, plus a triage record. It doesn't fix a live failure.What it changes
FullFlowUITests):audit-wallWalk-movingand printed asAUDITlines, as withHOUSESCAN_AUDIT_REPORT_ONLY. They don't fail the job.<phase>.heldin the gate folder while it waits on a gate, and the test waits for that file first. At that point both ends are set and "Done with this wall" is on offer..heldappears, so two minutes was too tight for a slow runner. Greptile raised this on ios: bring the t3/ios-mvf UI test flake fixes to beta #169.LaunchOptionsand the README describe.held; the LiDAR test's comment no longer times the walk.The failures and what fixed them
1.
screen.wallWalkcontrast / Dynamic Type in the full-flow tests: real bugs, fixed by Sam. My first diagnosis here was wrong. I blamed the audit's timing alone, but the in-motion audit was catching real mid-transition bugs:wallWalk-fullLegendstate.The runs with an empty id or label (36305050676 'Mark something', 36304552823) and the Dynamic Type one (36302958452) came from the same moving walk. I didn't find a separate bug behind them.
2.
testCantGetThereEndsWhereThePhoneIs: screen.gapRequest never appeared: fixed by 4461f13. Runs 36307478637, 36305052363, 36302961646, 36301217657 and 36296803483. The app went from the review straight to the upload, because the held-back gap wasn't one this flow asks for. 4461f13 plans the window against the ends this flow sets.3.
Failed to get matching snapshot: fixed by bc5a1ac. Six runs, all on commits without it.4.
Failed to terminate dev.housescanning.housescan: Simulator infrastructure, fixed by 20bf369. Three runs, all with a workflow that didn't boot the Simulator first. None since.5.
feature.acnot tappable: fixed by 3040033. The d4e008c runs.6.
testWholeFlowThroughTheButtons: missing meter.candidate.0: fixed by 1faa79b. The Simulator stalled the app's main thread for 14.6 s. 1faa79b raises the picker wait from 15 s to 30 s.7.
testWholeFlowThroughTheButtons: missing result.details: seen once on the Mac mini, not fixed. In 1 of 7 runs, the AR view's Done tap registered in XCTest but the app stayed on the AR view. I haven't found why, and it hasn't appeared in CI.Testing
On the Mac mini (Xcode 26.6, iPhone 17 on iOS 26.5, the workflow's
xcodebuildflags):testCantGetThereEndsWhereThePhoneIsx10,testFullFlowFromReplayx10 (-test-iterations 10, every iteration run)FullFlowUITestsx3testCantGetThereEndsWhereThePhoneIsx10,FullFlowUITestsx4,testWholeFlowThroughTheButtonsx7Base push run 36318952517 (6be087d, with the every-state audit): passed. CI on this pull request's first head: run 36319754126 passed. 4925f37 changes only the gate's timeout and wasn't in the Mac mini builds.
My first A/B used base 29ebe84, which had neither b1a2444 nor 1faa79b. Its one failure there was the
action.cannotAccessoverlap that b1a2444 fixed, so it proved nothing about the gate.🤖 Generated with Claude Code
Summary by CodeRabbit
Tests
Documentation