ios tests: audit the walk strictly once held and in motion to report; triage of the UI test failures

ios/HouseScan/Runtime/LaunchOptions.swift

@@ -14,8 +14,10 @@ import OSLog
 ///   UI tests raise it so each screen stays long enough to screenshot and audit.
 /// - `-autopilotGate <folder>`: before the flow leaves a screen, wait until a file named after
 ///   that phase exists in the folder. UI tests write it once they have screenshotted and audited
-///   the screen, so a slow audit can never miss a screen. After the result shows, the autopilot
-///   also writes the scan's scene.json there, for the test to check.
+///   the screen, so a slow audit can never miss a screen. While it waits, the app leaves
+///   `<phase>.held` there, so a test can tell when the autopilot has finished with a screen and
+///   holds it still. After the result shows, the autopilot also writes the scan's scene.json
+///   there, for the test to check.
 /// - `-autopilotCantGetThere`: the autopilot ends the walk with "Can't get there" instead of
 ///   marking the ends (`Autopilot.endWalkByCantGetThere`).
 /// - `-autopilotSomethingThere`: the autopilot answers the first spot check "Something's there"

ios/HouseScan/Runtime/ScanEngine.swift

@@ -293,11 +293,19 @@ final class ScanEngine {
     private var autoAdvanceDelay: Double { options.autopilot ? max(1.2, options.autopilotHold) : 1.2 }
 
     /// With `-autopilotGate`, waits until the UI test has finished with `phase` (its file exists),
-    /// for at most two minutes so a lost gate file can't hang the app.
+    /// for at most four minutes so a lost gate file can't hang the app. While it waits, the file
+    /// `<phase>.held` says the autopilot has finished with the screen and holds it still, for a UI
+    /// test that must audit it settled rather than while the autopilot drives it. A test starts
+    /// that audit only once `.held` appears, then reads the screen (the walk's map check can wait
+    /// up to 90 s), so the wait must outlast all of that; at two minutes a slow runner could have
+    /// let the app leave mid-audit.
     func waitForGate(_ phase: ScanPhase) async {
         guard options.autopilot, let gate = options.autopilotGate else { return }
         let file = gate.appending(path: phase.rawValue)
-        let deadline = ContinuousClock.now + .seconds(120)
+        if !FileManager.default.fileExists(atPath: file.path) {
+            try? Data().write(to: gate.appending(path: "\(phase.rawValue).held"))
+        }
+        let deadline = ContinuousClock.now + .seconds(240)
         while ContinuousClock.now < deadline, !FileManager.default.fileExists(atPath: file.path) {
             try? await Task.sleep(for: .milliseconds(100))
         }

ios/HouseScanUITests/FullFlowUITests.swift

@@ -14,6 +14,20 @@ final class FullFlowUITests: XCTestCase {
     /// Screens in the order the flow must show them.
     static let flow = ["onboarding", "findMeter", "meterCloseUp", "wallWalk", "markFeatures", "gapRequest", "uploading", "spotConfirm", "result", "resultAR"]
 
+    /// Screens audited strictly only once the autopilot has finished with them and holds them
+    /// still (`<phase>.held` in the gate folder). On the walk the autopilot marks features, sets
+    /// the ends and tilts up while the replay plays, so the screen changes several times a second
+    /// and two audit passes read two different moments. A finding that isn't the same element in
+    /// both passes can still match: a view fading out has no identifier, so it is keyed by its
+    /// label, or by "no element". The walk is therefore also audited on its first appearance, in
+    /// motion, in report-only mode (`audit-wallWalk-moving` attachments, and `AUDIT` lines in the
+    /// log), because that audit is what caught real bugs that only show mid-transition: the map's
+    /// legend entries drawn over each other (1faa79b, run 36307476187), "Can't get there" and
+    /// "Can't see past it" half-drawn over each other (b1a2444), and the reply's translucent fill
+    /// (397b6d6). Read those attachments when the walk's chrome or its animations change. The
+    /// strict audit then checks the held walk: both ends set and "Done with this wall" on offer.
+    static let auditedOnceHeld: Set<String> = ["wallWalk"]
+
     override func setUp() {
         continueAfterFailure = false
     }
@@ -91,8 +105,9 @@ final class FullFlowUITests: XCTestCase {
         try runFlow(replay: Self.lidarFixture) { app, phase in
             guard !showedHidden else { return }
             switch phase {
-            // At the autopilot's 3x the walk plays in about 6 s. No other frame sees the wall
-            // behind the bin, so its cells stay hidden once the bin has been in view.
+            // Read once the autopilot holds the walk, after the whole replay has played. No
+            // other frame sees the wall behind the bin, so its cells stay hidden once the bin has
+            // been in view.
             case "wallWalk": showedHidden = Self.wallTapeShowsHidden(app, timeout: 60)
             case "gapRequest": showedHidden = Self.wallTapeShowsHidden(app, timeout: 30)
             default: break
@@ -237,14 +252,17 @@ final class FullFlowUITests: XCTestCase {
     /// Audits the screen twice, six seconds apart, and fails on the issues found both times
     /// (`AccessibilityAudit.run`).
     ///
-    /// The walk keeps taking photos while it is audited, so the photo count's number is often
-    /// mid-roll, and the result's text is still fading in when the screen appears. A single audit
-    /// reports those passing frames as contrast failures; a real contrast or clipping problem is
-    /// still there later. The walk also rebuilds parts of the screen while it is audited, so an
-    /// issue's element can be gone before it is read: that one is attached, not failed on. Every
+    /// Replay screens can still be taking photos while they are audited, so the photo count's
+    /// number is often mid-roll, and the result's text is still fading in when the screen appears.
+    /// A single audit reports those passing frames as contrast failures; a real contrast or
+    /// clipping problem is still there later. The walk is audited strictly once the autopilot
+    /// holds it, and in motion only to report (`auditedOnceHeld`). With `reportOnly`, or
+    /// HOUSESCAN_AUDIT_REPORT_ONLY=1, the issues found twice are attached as `audit-<screen>`
+    /// and printed, not failed on. A screen that rebuilds parts of itself while it is audited can
+    /// lose an issue's element before it is read: that one is attached, not failed on. Every
     /// screen state is also audited frozen by ScreenStatesUITests.
     @MainActor
-    private func audit(_ app: XCUIApplication, screen: String) throws {
+    private func audit(_ app: XCUIApplication, screen: String, reportOnly: Bool = false) throws {
         // Six seconds: long enough for a system notification banner to leave. On CI one slid in
         // over the find-meter screen and failed its contrast check twice a second apart (run
         // 36264789032), as ScreenStatesUITests' audit also allows for.
@@ -257,7 +275,7 @@ final class FullFlowUITests: XCTestCase {
         }
         for (_, finding) in outcome.persistent {
             let text = "screen.\(screen): \(finding.message)"
-            if Self.environment["HOUSESCAN_AUDIT_REPORT_ONLY"] == "1" {
+            if reportOnly || Self.environment["HOUSESCAN_AUDIT_REPORT_ONLY"] == "1" {
                 let note = XCTAttachment(string: text)
                 note.name = "audit-\(screen)"
                 note.lifetime = .keepAlways
@@ -298,6 +316,26 @@ final class FullFlowUITests: XCTestCase {
             // The walk replays the whole recording; everything else takes seconds.
             let timeout: TimeInterval = phase == "markFeatures" || phase == "result" ? 150 : 60
             XCTAssertTrue(screen.waitForExistence(timeout: timeout), "screen.\(phase) never appeared")
+            if Self.auditedOnceHeld.contains(phase) {
+                // In motion, report-only: see `auditedOnceHeld`.
+                Thread.sleep(forTimeInterval: 1.0)
+                let moving = XCTAttachment(screenshot: app.screenshot())
+                moving.name = "\(phase)-moving"
+                moving.lifetime = .keepAlways
+                add(moving)
+                do {
+                    try audit(app, screen: "\(phase)-moving", reportOnly: true)
+                } catch {
+                    let note = XCTAttachment(string: "the audit couldn't read the screen in motion: \(error)")
+                    note.name = "audit-\(phase)-moving-unread"
+                    note.lifetime = .keepAlways
+                    add(note)
+                }
+                let held = gate.appending(path: "\(phase).held")
+                let deadline = Date().addingTimeInterval(150)
+                while !FileManager.default.fileExists(atPath: held.path), Date() < deadline { Thread.sleep(forTimeInterval: 0.2) }
+                XCTAssertTrue(FileManager.default.fileExists(atPath: held.path), "the autopilot never finished with screen.\(phase)")
+            }
             // Let the entrance animation finish so the screenshot and audit see the settled screen.
             Thread.sleep(forTimeInterval: 1.0)
             let shot = XCTAttachment(screenshot: app.screenshot())

ios/README.md

@@ -19,7 +19,7 @@ Native iPhone app for the AR capture walk. The homeowner marks the electric mete
 | `-autopilotHold <s>` | How long the autopilot leaves each screen up (default 1.2 s). |
 | `-autopilotCantGetThere` | With `-autopilot`, ends the walk the way device run 1 did. It answers "Can't get there" to the walk's first request, the ground in front of the meter, before the replay has shown it. Then, instead of marking the ends, it plays again the frame where the walk went farthest on each side and answers "Can't get there" when the walk asks to go on that way, so the end lands where the phone stands. `FullFlowUITests.testCantGetThereEndsWhereThePhoneIs` runs it. |
 | `-autopilotSomethingThere` | With `-autopilot`, answers the first spot check "Something's in the way" instead of "It's clear". The scan stops claiming that area, is uploaded again, and later checks get "It's clear". The stretch taken out (by this answer or by Continue) goes to the `-autopilotGate` folder as `spot-refusal.json` (`withdrawn_span_ft`, feet along the wall as in scene.json). `FullFlowUITests.testSomethingThereChecksTheWallAgain` runs it. |
-| `-autopilotGate <folder>` | Used by the UI tests. Before leaving a screen, the app waits for a file named after that phase in this folder. Once the result shows, the autopilot writes the scan's scene.json there. |
+| `-autopilotGate <folder>` | Used by the UI tests. Before leaving a screen, the app waits for a file named after that phase in this folder, and while it waits it leaves `<phase>.held` there, so a test can audit the screen once the autopilot has finished with it. Once the result shows, the autopilot writes the scan's scene.json there. |
 | `-uiDemo` | Runs the screens on a scripted fake engine instead of the capture engine, for design work and for auditing states a replay can't reach. The arguments it takes are listed in `HouseScan/UI/Preview/UIDemo.swift`; for example `-uiDemoPhase spotConfirm -uiDemoSpotStep ground` opens the spot check's ground question, `-uiDemoPhase meterCloseUp -uiDemoMeterChoose` shows the meter number picker, and `-uiDemoPhase result -uiDemoCorner` shows the result model of a wall that turns a corner. |
 
 **Spot check.** Before the result (`STATE=spotConfirm`), the app shows the kept photo that best sees the answer's spot and its front clearance, outlined, and asks at most two questions on it. First: is anything in the marked area? Photos without depth and the walked path claim wall, ground and clear space they never saw, and the server keeps clearances only from marked things. "It's clear" backs those claims for that spot; "Something's in the way" takes them back over the area, so scene.json reports it unseen; "A gas meter, AC, window or door" opens the camera to mark it. Then, once it is clear: what is the ground where the battery would stand? The answer goes to the server as a ground patch over the footprint and its error margin, the only ground patch the app sends. Every answer but "It's clear" with "Not sure" sends the scan again. When no kept photo shows the whole area, nothing is asked and the area is left out. The rules are in `Runtime/ScanEngine+Confirm.swift` and HouseScanKit's `Confirm/SpotConfirmation.swift`.