Repository navigation
Draw the prototype's fog and live dots over the walk #158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
SamGu-NRX
wants to merge
6
commits into
t3/ios-mvf
Choose a base branch
from
t3/live-fog
base: t3/ios-mvf
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+2,072
−36
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
51184fe
Draw the prototype's fog and live dots over the walk
SamGu-NRX 94c2e2a
Keep hidden and requested tints in the band, and rag the fog's edges
SamGu-NRX 48d024a
Say in the iOS README where the live fog's rule and dots live
SamGu-NRX 9defff1
Keep narrow unseen cells fogged, move dots exactly on corrections
SamGu-NRX c666f69
Merge remote-tracking branch 'origin/t3/ios-mvf' into t3/live-fog
SamGu-NRX f57310c
Tune the fog, the hidden tint and the dots against the prototype videos
SamGu-NRX File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| import Foundation | ||
| import HouseScanKit | ||
| import os | ||
|
|
||
| /// Runs the LiDAR dot field (`SurfaceDots`) off the main actor and publishes its dots for the | ||
| /// camera overlay (`ScanViewState.liveDots`). | ||
| /// | ||
| /// The engine hands it each kept keyframe that has depth, the same frames coverage observes, so | ||
| /// the dots and the fog move together. Fusing a 256 x 192 map takes a few milliseconds (3.4 ms | ||
| /// median on an M4 Pro in a Release build; not measured on a phone), so it runs on its own | ||
| /// serial queue. A keyframe arriving while two are still waiting is dropped: the next one covers | ||
| /// the same wall. An anchor correction is a matrix product (`SurfaceDots.apply`) and republishes | ||
| /// only when no keyframe is waiting to publish anyway, so neither kind of work can pile up. | ||
| final class LiveDotsFeed: @unchecked Sendable { | ||
| // `@unchecked Sendable`: everything below `queue` is read and written only on it; `pending` | ||
| // is a lock. | ||
| private let queue = DispatchQueue(label: "dev.housescanning.housescan.live-dots", qos: .utility) | ||
| private var field = SurfaceDots() | ||
| private var revision = 0 | ||
| private var last: (eye: SIMD3<Float>, wall: WallFrame)? | ||
| private let pending = OSAllocatedUnfairLock(initialState: 0) | ||
| private let publish: @MainActor @Sendable (LiveDots, _ generation: Int) -> Void | ||
|
|
||
| /// Keyframes waiting on the queue past which new ones are dropped. | ||
| static let maxPending = 2 | ||
|
|
||
| /// `publish` runs on the main actor with the scan generation the work was queued under, so | ||
| /// the engine can ignore dots from before a Start over. | ||
| init(publish: @escaping @MainActor @Sendable (LiveDots, _ generation: Int) -> Void) { | ||
| self.publish = publish | ||
| } | ||
|
|
||
| func integrate(camera: CameraFrame, depth: DepthImage, wall: WallFrame, generation: Int) { | ||
| let accepted = pending.withLock { count -> Bool in | ||
| guard count < Self.maxPending else { return false } | ||
| count += 1 | ||
| return true | ||
| } | ||
| guard accepted else { | ||
| RuntimeLog.capture.info("live dots: dropped a keyframe, two still waiting") | ||
| return | ||
| } | ||
| queue.async { [self] in | ||
| field.integrate(camera: camera, depth: depth) | ||
| last = (camera.position, wall) | ||
| pending.withLock { $0 -= 1 } | ||
| send(generation) | ||
| } | ||
| } | ||
|
|
||
| /// ARKit's correction to the meter's anchor, applied as coverage applies it. | ||
| func apply(_ correction: YawCorrection, wall: WallFrame, generation: Int) { | ||
| queue.async { [self] in | ||
| field.apply(correction) | ||
| if let eye = last?.eye { last = (correction.point(eye), wall) } | ||
| if pending.withLock({ $0 }) == 0 { send(generation) } | ||
| } | ||
| } | ||
|
|
||
| func reset() { | ||
| queue.async { [self] in | ||
| field = SurfaceDots() | ||
| last = nil | ||
| } | ||
| } | ||
|
|
||
| /// On `queue`. | ||
| private func send(_ generation: Int) { | ||
| guard let last else { return } | ||
| revision += 1 | ||
| let dots = LiveDots( | ||
| dots: field.dots(near: last.eye, wall: last.wall).map { | ||
| LiveDots.Dot(id: $0.id, position: $0.position, isEdge: $0.isEdge, onOccluder: $0.onOccluder, opacity: $0.opacity) | ||
| }, | ||
| revision: revision) | ||
| let publish = publish | ||
| DispatchQueue.main.async { | ||
| MainActor.assumeIsolated { publish(dots, generation) } | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| import SwiftUI | ||
| import simd | ||
|
|
||
| /// The crisp lines over the live fog, which a soft mask can't draw: a dashed violet outline | ||
| /// round each stretch hidden behind something, a slate hatch where the homeowner said "I can't | ||
| /// get there", and an amber outline round the stretch a gap request asks for. The same marks | ||
| /// and dash the tape map uses, so the camera and the map read alike. | ||
| struct FogMarks: View { | ||
| var coverage: CoverageStrip | ||
| var wall: WallGeometry | ||
| var projection: CameraProjection | ||
| var highlight: GapRequest? | ||
|
|
||
| var body: some View { | ||
| Canvas(rendersAsynchronously: false) { context, size in | ||
| let geometry = WallProjection(projection: projection, wall: wall, size: size) | ||
| let runs = runs() | ||
| // The map's dash, lengthened for the larger shapes on the camera. | ||
| let dash = StrokeStyle(lineWidth: 2, lineCap: .round, lineJoin: .round, dash: Palette.hiddenDash.map { $0 * 1.6 }) | ||
| for run in runs.hidden { | ||
| guard let outline = outline(run.band, run.edges, geometry) else { continue } | ||
| // A dark halo under the dash keeps it legible on a sunlit wall. | ||
| context.stroke(outline, with: .color(.black.opacity(0.35)), style: StrokeStyle(lineWidth: 4, lineJoin: .round)) | ||
| context.stroke(outline, with: .color(Palette.hidden), style: dash) | ||
| } | ||
| for run in runs.skipped { | ||
| guard let quad = outline(run.band, run.edges, geometry) else { continue } | ||
| context.fill(quad, with: .color(Palette.skipped.opacity(0.3))) | ||
| var hatched = context | ||
| hatched.clip(to: quad) | ||
| let bounds = quad.boundingRect | ||
| var lines = Path() | ||
| var x = bounds.minX - bounds.height | ||
| while x < bounds.maxX { | ||
| lines.move(to: CGPoint(x: x, y: bounds.maxY)) | ||
| lines.addLine(to: CGPoint(x: x + bounds.height, y: bounds.minY)) | ||
| x += 10 | ||
| } | ||
| hatched.stroke(lines, with: .color(.white.opacity(0.55)), lineWidth: 1.5) | ||
| } | ||
| for run in runs.requested { | ||
| guard let outline = outline(run.band, run.edges, geometry) else { continue } | ||
| context.stroke(outline, with: .color(.black.opacity(0.3)), style: StrokeStyle(lineWidth: 4, lineJoin: .round)) | ||
| context.stroke(outline, with: .color(Palette.caution), style: StrokeStyle(lineWidth: 2, lineJoin: .round)) | ||
| } | ||
| } | ||
| .allowsHitTesting(false) | ||
| .accessibilityHidden(true) | ||
| } | ||
|
|
||
| private enum Kind { case hidden, skipped, requested } | ||
|
|
||
| private struct Run { | ||
| var band: CoverageBand | ||
| /// The s of every cell edge along the run, clipped to the visible range. | ||
| var edges: [Float] | ||
| } | ||
|
|
||
| /// Runs of hidden, of skipped, and of requested-but-not-covered cells in each band. Coverage | ||
| /// filling the middle of a request splits it, and each piece keeps its outline. | ||
| private func runs() -> (hidden: [Run], skipped: [Run], requested: [Run]) { | ||
| var hidden: [Run] = [], skipped: [Run] = [], requested: [Run] = [] | ||
| let visible = coverage.visibleRange | ||
| guard visible.upperBound > visible.lowerBound else { return ([], [], []) } | ||
| for band in CoverageBand.allCases { | ||
| let states = coverage.cells(band) | ||
| var current: (kind: Kind, run: Run)? | ||
| func close() { | ||
| if let current { | ||
| switch current.kind { | ||
| case .hidden: hidden.append(current.run) | ||
| case .skipped: skipped.append(current.run) | ||
| case .requested: requested.append(current.run) | ||
| } | ||
| } | ||
| current = nil | ||
| } | ||
| for index in states.indices { | ||
| let range = coverage.cellRange(index) | ||
| guard range.upperBound > visible.lowerBound, range.lowerBound < visible.upperBound else { close(); continue } | ||
| let s0 = max(range.lowerBound, visible.lowerBound), s1 = min(range.upperBound, visible.upperBound) | ||
| let state = states[index] | ||
| let isRequested = highlight.map { $0.band == band && $0.span.lowerBound < range.upperBound && range.lowerBound < $0.span.upperBound } ?? false | ||
| let kind: Kind? = if isRequested, state != .covered { .requested } | ||
| else if state == .hidden { .hidden } | ||
| else if state == .skipped { .skipped } | ||
| else { nil } | ||
| guard let kind else { close(); continue } | ||
| if var open = current, open.kind == kind { | ||
| open.run.edges.append(s1) | ||
| current = open | ||
| } else { | ||
| close() | ||
| current = (kind, Run(band: band, edges: [s0, s1])) | ||
| } | ||
| } | ||
| close() | ||
| } | ||
| return (hidden, skipped, requested) | ||
| } | ||
|
|
||
| /// The outline of a run through every cell edge, so it follows the wall round a corner. Nil | ||
| /// when any point is behind the camera. | ||
| private func outline(_ band: CoverageBand, _ edges: [Float], _ geometry: WallProjection) -> Path? { | ||
| let corners: [SIMD3<Float>] = switch band { | ||
| case .wall: | ||
| edges.map { wall.world(s: $0, height: coverage.wallBandHeight) } + edges.reversed().map { wall.world(s: $0, height: 0) } | ||
| case .ground: | ||
| edges.map { wall.world(s: $0, height: 0, out: coverage.groundBandDepth) } + edges.reversed().map { wall.world(s: $0, height: 0) } | ||
| } | ||
| return geometry.polygon(corners) | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
resetSpatialStatediscards the old wall and coverage, but unlike this reset path, it does not reset the dot field or clearstate.liveDots. When the homeowner starts again from the meter, new keyframes fuse depth into dots from the old world frame, so dots can appear in the wrong places. Reset the feed and published dots during spatial reset too.Prompt To Fix With AI
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Caretaker check at f57310c: confirmed, and new in this PR. A relocalization timeout calls
resetSpatialState(ScanEngine.swift:1113-1169), which restarts ARKit with.resetTrackingbut keepsLiveDotsFeed'sSurfaceDotsfield andstate.liveDots. Depth from the new world frame then fuses into old-frame dots.resetAllalready clears both (lines 1598-1599). The effect is on the overlay only; capture, coverage and export never read the dots. This is a reset bug, separate from the deferred dots/veil redesign.The three-line fix (the same two calls in
resetSpatialState) is prepared on a caretaker branch for the owner to pick up; it isn't pushed here. The app has no unit-test target, so the fix has only been checked against the source: the serial queue and the generation guard makereset()drop queued old-frame publishes, as they do for Start over.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Follow-up: the prepared fix, 010f322 on the caretaker branch
codex/greptile-158-dot-reset(local, not pushed), builds. A Debug iphonesimulator xcodebuild of the HouseScan scheme succeeded through the shared QA queue (jobgreptile-158-build-010f322-r2, 20:07–20:09 UTC). That shows it compiles, not that the reset behaves correctly on a device.