Skip to content

[obvious-fleet] Backend: deterministic app-shaped synthetic capture generator - #299

Draft
SamGu-NRX wants to merge 2 commits into
alphafrom
obv/burst14-bs-synthetic-app-capture
Draft

SamGu-NRX wants to merge 2 commits into
alphafrom
obv/burst14-bs-synthetic-app-capture

Conversation

@SamGu-NRX

Copy link
Copy Markdown
Owner

fleet-packet: burst14-bs-synthetic-app-capture

Branch obv/burst14-bs-synthetic-app-capture @ ffeb9340 (base alpha @ 969d4e93), two commits: feat(backend): deterministic app-shaped synthetic capture generator and docs(backend): synthetic capture validator reports and ground truth. This packet is the PR description; the maintainer opens the PR from it.

Load-bearing assumption

The premise: end-to-end validation and reviewer training on BaseScanning need app-format capture fixtures whose truth is known exactly — because real captures carry no ground truth, and the committed mini-arkit fixture is a single fixed scene with no hazards. Evidence that confirms this premise: a geometry change in the server's packet conventions (tap replay, pose handling, depth scaling) failing the new tests. Evidence that would break it: a real device capture with independently measured geometry contradicting what the generator claims (the generator's scenes are simple by design — one flat wall, meters as axis-aligned boxes — so a real-capture mismatch would show up as convention drift, not fixture error).

Problem

The repo validates against exactly one committed app-shaped capture: the six-frame mini-arkit fixture. It has no hazards, no close-up stills workflow, no tap-vs-hazard interaction, and no ground-truth file — so validation and placement runs on it can never check what should have been found, only that the pipeline does not crash. Building more real fixtures by hand cannot state exact geometry (metres, planes, clearances), cannot be regenerated when the packet format changes, and cannot vary scene content (hazard left vs right vs absent) without a new capture session.

Solution

backend/scripts/gen_app_capture.py (~1,100 lines, pure NumPy/PIL) renders a complete app-format packet from a small JSON scene spec, with every quantity derived from exact geometry:

  • Spec → scene: a flat wall at z = −2 m, one electric meter box (0.45 × 0.6 × 0.18 m, front face 0.18 m proud), optional hazards (e.g. a gas meter at a spec'd centre offset), and a 30-keyframe 30 Hz walk path (1.45 m height, ±0.45 m lateral sway, fixed forward pose). Camera heading aims each frame at the meter's front-face centre.
  • Rendered per keyframe: 640×480 RGB (procedurally textured wall with shading and slat variation, meter face with grille and border), 192×256 float32 LiDAR depth, uint8 confidence, and 0–1 feature points — all from the same analytic raycast (cast_scene: two wall patches + meter box + hazard box, nearest-hit t/normal/surface). Depth rays share the keyframe FOV with proportionally scaled intrinsics (fx·w/640, cx = w/2), matching ARKit's scaled-plane convention.
  • App-shaped packet (formatVersion 0.4): 30 keyframes (poses column-major, orthonormalized, arkit_fixed reference rows), gzipped motion/arkitPoses CSV streams (gravity projected onto the up axis, heading unwrapped), a confirmed meter tap whose pixel/hit.position/hit.distance are packed from the same raycast the depth uses, two close-up stills (head-on + 30° left oblique with viewAngleDeg/arkitDistanceM in range), planes, epochs, and a full sha256 file manifest. gzip.GzipFile(mtime=0) and fixed PIL encode settings make every byte — JPEGs included — reproducible for a given spec + seed.
  • Ground truth alongside: ground_truth.json with the wall plane, meter AABB and front-face z, hazard list, and measured clearances (centre-to-centre and edge-to-edge on the wall plane) — the numbers the placement/criteria stages are supposed to reproduce. No rule thresholds are encoded; ground truth states geometry, policy stays in the rules.
  • Scene specs (committed): clear-wall.json (no hazards) and gas-left.json (gas meter 0.6 m centre-to-centre left, edges 0.2 m apart) under backend/tests/fixtures/app_capture_synth/.

Verification

  • Focused suite: 11 passed (tests/test_app_capture_synth.py), covering: layout vs the fixture contract, the real server.packet validator + normalizer accepting both scenes (only the expected synthetic_source warning, tap replay error < 1e-6), byte-identical re-rendering, ground truth matching the spec (including 0.6/0.2 m clearances), tap replay landing on the meter front face through the packet's own conventions, depth pixels agreeing with the packed tap distance, the 30 Hz keyframe grid, and still angle/distance bounds.
  • Validator CLI on both rendered packets: ok: true, 30 keyframes, 2 stills, 1 tap, 127 files hashed, max tap replay error 3.46e-10.
  • Full backend suite (make test): 4 failed / 1517 passed / 19 skipped / 14 errors in 720 s — every non-pass is the documented pristine-alpha baseline: the four known SD-96 gate failures in server/eval/tests/test_gates.py (committed-fixtures scoring, offline-table engine skip, route-over-the-bar expiry, pin refresh) plus the 14 known test_capture_cold_restart.py collection errors. No new failures or errors from this branch.
  • CPU poses stage on the committed mini-arkit fixture (2 cores): exit 0, status: fallback_arkit_fixed, mode: arkit_fixed — free SfM produced no 3D points on the tiny fixture (documented behavior), all 4 keyframes registered via ARKit carries (registered_fraction: 1.0, per-camera pose_source: arkit), and the meter was located from the committed tap (source: tap, frame k00002).
  • ruff check + ruff format --check + lint-imports: clean.
  • evidence/ in this packet (images only, per packet rules): the mid-walk keyframe k00015 and both close-up stills for each scene — clear-wall-keyframe.jpg, clear-wall-still-close.jpg, clear-wall-still-oblique.jpg, and the matching gas-left-* set. The validator's full report and each scene's ground_truth.json are committed on the work branch at backend/docs/app-capture-synth-evidence/. The hss-validation-report/v0 lines (timings omitted), re-run live at packet-publication time, quoted here:
    • clear-wall: {"format": "hss-validation-report/v0", "packetId": "609e3067892652719af2ae64cd3a7c09", "formatVersion": "0.4", "tier": "arkit_lidar", "stage": "full", "ok": true, "errors": [], "warnings": [{"code": "synthetic_source", "level": "warning", "pointer": "/source/kind", "message": "synthetic packet", "value": null}], "keyframes": 30, "stills": 2, "taps": 1, "epochs": 1, "files": 127, "filesHashed": 127, "maxTapReplayError": 3.4585401209596967e-10, "imagesUsable": true, "imagesDecodable": 32}
    • gas-left: {"format": "hss-validation-report/v0", "packetId": "1115853d975a5129a2202ee27d758b1d", "formatVersion": "0.4", "tier": "arkit_lidar", "stage": "full", "ok": true, "errors": [], "warnings": [{"code": "synthetic_source", "level": "warning", "pointer": "/source/kind", "message": "synthetic packet", "value": null}], "keyframes": 30, "stills": 2, "taps": 1, "epochs": 1, "files": 127, "filesHashed": 127, "maxTapReplayError": 3.4585401209596967e-10, "imagesUsable": true, "imagesDecodable": 32}
      Packets are regenerable byte-identically: uv run python backend/scripts/gen_app_capture.py backend/tests/fixtures/app_capture_synth/clear-wall.json /tmp/synth-clear-wall (same for gas-left.json).

Draft note

draft: true per fleet precedent: the full suite is not green in absolute terms — the 4 SD-96 gate failures and 14 cold-restart collection errors are the documented alpha baseline, reproduced identically on base 969d4e93. Everything this branch adds passes; flip out of draft when the repo-wide baseline is cleared.

Assumptions

  • The generator pins the current packet contract (formatVersion 0.4, arkit_fixed tier, tap/geometry conventions). If the contract changes, the fixtures regenerate rather than the tests loosening — the specs are the source of truth, the committed bytes are derived.
  • Rendering stays deliberately simple (flat wall, axis-aligned boxes, procedural textures). It exercises geometry and conventions, not photometric realism; anything needing real textures needs a real capture.

Human author: Sam Gu (sgu07966@gmail.com)

🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact


Opened by the maintainer from the Obvious thread's own description, so every review bot reviews it (CodeRabbit skips PRs authored by bots). The work and its commits are by Obvious.

Co-authored-by: obvious-autobuild[bot] <262744130+obvious-autobuild[bot]@users.noreply.github.com>

clear-wall-keyframe

clear-wall-still-close

clear-wall-still-oblique

gas-left-keyframe

gas-left-still-close

gas-left-still-oblique

ObviousApp and others added 2 commits October 16, 2026 18:14
Render a 30-keyframe walk, LiDAR depth/confidence, CSV streams, meter
stills, a confirmed tap and ground truth from a scene spec, with pure
NumPy/PIL. The packed tap replays through the packet's own conventions
onto the meter box, depth rays share the keyframe FOV with scaled
intrinsics, and gzip/JPEG/ids are byte-stable for a given spec and seed.
The real server.packet validator and normalizer accept both scenes.

Scene specs: clear-wall (no hazards) and gas-left (gas meter 0.6 m
centre-to-centre left of the meter). No rule thresholds are encoded.

Co-authored-by: Sam Gu <sgu07966@gmail.com>
@SamGu-NRX

Copy link
Copy Markdown
Owner Author

@codex review

@coderabbitai

coderabbitai Bot commented Oct 16, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the size:XXL 1,000+ effective changed lines. label Oct 16, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T23:33:04.284210Z ffeb934 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ffeb93404d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1135 to +1137
meter_left_edge = scene.meter["x"] - scene.meter["size"][0] / 2.0
gas_right_edge = scene.gas["x"] + scene.gas["size"][0] / 2.0
clearances["meterEdgeToGasEdgeM"] = round(meter_left_edge - gas_right_edge, 6)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Compute gas clearance independently of hazard side

When a scene spec places the gas meter to the right of the electric meter, this always subtracts the gas meter's right edge from the electric meter's left edge, producing a negative value even when the boxes are separated. Because the generator accepts an arbitrary gasMeter.x offset, the emitted ground truth misstates the edge-to-edge clearance for right-side hazards and can corrupt downstream placement evaluation; compute the gap from the nearer pair of edges (or otherwise handle both sides).

Useful? React with 👍 / 👎.

Comment on lines +654 to +660
# --- arkitPoses (30 Hz, keyframe times sit exactly on this grid) ---
lines = ["t,epoch,tracking,tx,ty,tz,qx,qy,qz,qw,fx,fy,cx,cy"]
n_poses = int(round((WALK_SECONDS + 3.5) * 30))
for k in range(n_poses + 1):
t = k / 30.0
pose = _pose_interp(scene, t)
quat = rotmat_to_quat_xyzw(_device_from_world(pose))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop the synthetic trajectory when the walk ends

After WALK_SECONDS the location stream reports zero speed, but the motion and ARKit pose streams continue evaluating pose_at(scene, t), whose walk_x keeps advancing at 1 m/s through the extra 3.5 seconds. Consumers that use the pose stream for reconstruction therefore see the camera moving while the accompanying sensor stream says it is stationary, adding an artificial trajectory segment and inconsistent motion data; hold the final pose (and zero angular/linear motion) during the still-capture tail.

Useful? React with 👍 / 👎.

Comment on lines +1175 to +1179
"capture": {
"keyframes": len(packet["keyframes"]),
"keyframeResolution": [KEYFRAME_W, KEYFRAME_H],
"depthResolution": [DEPTH_W, DEPTH_H],
"walkDistanceToWallM": [1.5, 2.5],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Derive the reported walk-to-wall range from the scene

For any spec whose wall.distance differs from the default 2.0 m, this hard-coded ground-truth range is wrong: the camera path is still generated relative to the wall at -scene.wall_z, so the actual distances are approximately wall.distance ± 0.45. The resulting ground_truth.json can therefore contradict the rendered geometry and mislead evaluators; compute this range from the scene distance and the path extrema.

Useful? React with 👍 / 👎.

Comment on lines +539 to +542
d_origin, d_dir = ray_grid(pose, DEPTH_W, DEPTH_H)
dt, _normal, _surface = cast_scene(scene, d_origin, d_dir)
depth = dt.astype("<f4")
confidence = np.where(np.isfinite(depth), np.where(depth <= 6.0, 3, 2).astype(np.uint8), np.uint8(0))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Emit only valid ARConfidenceLevel values

The packet contract defines confidence bytes as 0 for low, 1 for medium, and 2 for high, but every finite depth sample within 6 m is written as 3. This makes the generated confidence maps invalid and causes consumers that aggregate the documented 0/1/2 levels to treat all measured samples as an unknown or absent confidence level; map the high-confidence case to 2.

Useful? React with 👍 / 👎.

Comment on lines +564 to +570
points.append(
(
float(cc[r, c] + 0.5),
float(rr[r, c] + 0.5),
float(response[r, c]),
float(rng.uniform(2.0, 5.0)),
0.0,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Serialize feature points using the packet's raw-point layout

The packet contract's 20-byte feature record is three float32 world-space coordinates followed by a uint64 identifier, but these rows pack five float32 values for image coordinates, response, scale, and angle. Any reconstruction stage reading features.bin as the declared raw ARKit point cloud will therefore interpret the response/scale bytes as geometry and split identifiers across unrelated rows, producing corrupt tracks and geometry; emit the documented {x, y, z, identifier} records or change the packet representation.

Useful? React with 👍 / 👎.

Comment on lines +539 to +542
d_origin, d_dir = ray_grid(pose, DEPTH_W, DEPTH_H)
dt, _normal, _surface = cast_scene(scene, d_origin, d_dir)
depth = dt.astype("<f4")
confidence = np.where(np.isfinite(depth), np.where(depth <= 6.0, 3, 2).astype(np.uint8), np.uint8(0))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Encode missing depth samples as zero

Misses from cast_scene remain inf in dt and are written directly to the float32 depth files. The packet contract requires finite non-negative depth values with zero representing no measurement, so scenes whose rays miss the finite wall/ground patches produce invalid depth maps that can be rejected or poison depth statistics and reconstruction; replace non-finite samples with zero before serialization.

Useful? React with 👍 / 👎.

@SamGu-NRX

Copy link
Copy Markdown
Owner Author

@greptileai review this at plus

@greptile-apps

greptile-apps Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 2/5 Tier: plus

[Low impact] Adds test fixtures and a synthetic data generator script.

Fix the depth convention, close-up keyframe links and right-side clearance calculation before merging.

Fix All in CodexFindings

  1. P1 Depth stretches reconstructed surfaces ▶
  2. P1 Close-up views are skipped ▶
  3. P1 Right-side clearances are wrong ▶
  4. P2 Feature records use wrong fields ▶
  5. P2 Stream orientations disagree ▶
  6. P2 Plane anchors use wrong axes ▶
  7. P2 Confidence exceeds the app range ▶
  8. P2 User acceleration includes gravity ▶
  9. P2 Final rotation rates are false ▶
  10. P2 Oblique view has wrong side ▶
Fix with agent prompt
### Issue 1
backend/scripts/gen_app_capture.py:539-541
`render_frame` saves distance along a unit ray into `depth.f32`, but the server reads depth along the camera’s forward axis. Off-centre pixels reconstruct too far away-about 15% at the horizontal edge-so both default scenes disagree with `ground_truth.json`.

Convert the hit distance to camera-axis depth before saving it. Add an off-centre check through the server’s `unproject` function; the current near-centre tap check misses this difference.

### Issue 2
backend/scripts/gen_app_capture.py:905
Both stills link to the same walking keyframe, `k00012`, despite being rendered from different close-up positions. In the guided path, `packet_io` skips still images with existing keyframe links and marks the linked frame as the close-up. The second still overwrites the first frame’s purpose, so `meter_close` disappears and neither close-up JPEG enters reconstruction.

Emit a matching keyframe for each still and link to that frame. Otherwise these fixtures do not exercise the two-view close-up workflow.

### Issue 3
backend/scripts/gen_app_capture.py:1135-1137
`meterEdgeToGasEdgeM` assumes the gas meter is always left of the electric meter. Moving it to `x = 2.6` with the existing sizes gives `-1.0` m instead of the true `0.2` m gap. A correct placement result would then appear wrong against this ground truth.

Compute the horizontal gap from the ordered boxes, or use absolute centre separation minus their half-widths. Add a right-side scene test.

### Issue 4
backend/scripts/gen_app_capture.py:547-548
`features.bin` contains five floats for image coordinates, response, scale and angle. The app format requires three floats for a world-space point followed by a `uint64` identifier. Both records occupy 20 bytes, so the validator accepts this unrelated layout.

Write actual scene points and identifiers, and test a record decoded using the app format. Otherwise the fixtures cannot check how app feature points are read.

### Issue 5
backend/scripts/gen_app_capture.py:659-660
`arkitPoses` converts each camera rotation through `DEVICE_FROM_CAMERA` before writing its quaternion. This adds a fixed 90° rotation that the real app does not write, so stream orientations disagree with the matching keyframe poses. The fixtures therefore preserve the wrong convention even while validation passes.

Use `pose[:3, :3]` directly here and test the stream quaternion against the keyframe rotation.

### Issue 6
backend/scripts/gen_app_capture.py:1089-1096
The app format puts a plane in local `x-z`, with local `+y` as its normal. Here the wall’s normal points up and the floor’s normal points sideways. Both `center` values are also world coordinates rather than coordinates local to the anchor.

Build the transforms using the app convention and keep the centres local. These fixtures otherwise cannot check plane geometry faithfully; current wall completion ignores the affected fields and hides the mismatch.

### Issue 7
backend/scripts/gen_app_capture.py:542
Near-depth pixels get confidence `3`, but the app format only permits `0`, `1` and `2`, with `2` meaning high confidence. The fixtures therefore contain values no app capture produces.

Emit the app’s values and assert their range in the tests. The current validator only checks for all-low confidence, so its passing report does not catch this mismatch.

### Issue 8
backend/scripts/gen_app_capture.py:625
`motion.accX..Z` writes `grav + acc_noise`, but those fields represent `CMDeviceMotion.userAcceleration`, which excludes gravity. Only `imuRaw.ax..az` includes it. The motion stream therefore reports roughly 1 g of user acceleration without that movement.

Keep gravity out of the motion stream’s acceleration fields and test the difference between the two streams.

### Issue 9
backend/scripts/gen_app_capture.py:622-624
After `WALK_SECONDS`, `yaw_next` stays frozen at 10 seconds while `yaw_prev` keeps advancing through the 13.5-second stream. Dividing that growing angle difference by `0.01` creates false angular speeds that disagree with the saved camera movement.

Sample both ends around the same time, or stop the pose and both samples together. Apply the same fix to `imuRaw` so the fixtures can check motion against poses.

### Issue 10
backend/scripts/gen_app_capture.py:794
The oblique camera sits at `meter.x + 0.38`, which is the homeowner’s right under this scene’s coordinates, but its metadata and test call it `left`. Reviewers and tests receive a right-side image labelled as a left-side view.

Move the camera left or label the view `right`. Assert the side from the camera position rather than repeating a constant.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds a deterministic scene-to-packet generator, two scene specifications, ground-truth files, validator reports and focused tests.

  • Scene specs now produce repeatable app-shaped captures.
  • Each generated capture comes with its scene's exact geometry.

SamGu-NRX reports four existing gate-test failures and fourteen cold-restart errors on the base branch, with cleanup deferred. Simple rendering is intentional. No runtime tests were run during this review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Spec["Scene JSON + seed"] --> Render["Ray cast scene"]
  Spec --> Streams["Build sensor streams"]
  Spec --> Truth["ground_truth.json"]
  Render --> Images["Keyframes and stills"]
  Render --> Depth["Depth and confidence"]
  Images --> Packet["App-shaped packet"]
  Depth --> Packet
  Streams --> Packet
  Packet --> Validate["Packet validator"]
  Packet --> Poses["Pose reader and reconstruction"]
  Poses --> Scale["Scale and placement"]
  Truth --> Compare["Compare recovered geometry"]
  Scale --> Compare
Loading

Reviews (1) · Last reviewed commit: "docs(backend): synthetic capture validat..." · Reviewed by Greptile

Comment on lines +539 to +541
d_origin, d_dir = ray_grid(pose, DEPTH_W, DEPTH_H)
dt, _normal, _surface = cast_scene(scene, d_origin, d_dir)
depth = dt.astype("<f4")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Depth stretches reconstructed surfaces

render_frame saves distance along a unit ray into depth.f32, but the server reads depth along the camera’s forward axis. Off-centre pixels reconstruct too far away—about 15% at the horizontal edge—so both default scenes disagree with ground_truth.json.

Convert the hit distance to camera-axis depth before saving it. Add an off-centre check through the server’s unproject function; the current near-centre tap check misses this difference.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 539-541

Comment:
**Depth stretches reconstructed surfaces**

`render_frame` saves distance along a unit ray into `depth.f32`, but the server reads depth along the camera’s forward axis. Off-centre pixels reconstruct too far away—about 15% at the horizontal edge—so both default scenes disagree with `ground_truth.json`.

Convert the hit distance to camera-axis depth before saving it. Add an off-centre check through the server’s `unproject` function; the current near-centre tap check misses this difference.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

img = out_dir / "stills" / f"{view['id']}.jpg"
Image.fromarray(rgb).save(img, format="JPEG", quality=JPEG_QUALITY, subsampling=0)
files.append(_file_entry(img, out_dir, "still"))
link = frames[min(range(len(frames)), key=lambda i: abs(walk_x(keyframe_times()[i]) - scene.meter["x"]))]["id"]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Close-up views are skipped

Both stills link to the same walking keyframe, k00012, despite being rendered from different close-up positions. In the guided path, packet_io skips still images with existing keyframe links and marks the linked frame as the close-up. The second still overwrites the first frame’s purpose, so meter_close disappears and neither close-up JPEG enters reconstruction.

Emit a matching keyframe for each still and link to that frame. Otherwise these fixtures do not exercise the two-view close-up workflow.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 905

Comment:
**Close-up views are skipped**

Both stills link to the same walking keyframe, `k00012`, despite being rendered from different close-up positions. In the guided path, `packet_io` skips still images with existing keyframe links and marks the linked frame as the close-up. The second still overwrites the first frame’s purpose, so `meter_close` disappears and neither close-up JPEG enters reconstruction.

Emit a matching keyframe for each still and link to that frame. Otherwise these fixtures do not exercise the two-view close-up workflow.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Comment on lines +1135 to +1137
meter_left_edge = scene.meter["x"] - scene.meter["size"][0] / 2.0
gas_right_edge = scene.gas["x"] + scene.gas["size"][0] / 2.0
clearances["meterEdgeToGasEdgeM"] = round(meter_left_edge - gas_right_edge, 6)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Right-side clearances are wrong

meterEdgeToGasEdgeM assumes the gas meter is always left of the electric meter. Moving it to x = 2.6 with the existing sizes gives -1.0 m instead of the true 0.2 m gap. A correct placement result would then appear wrong against this ground truth.

Compute the horizontal gap from the ordered boxes, or use absolute centre separation minus their half-widths. Add a right-side scene test.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 1135-1137

Comment:
**Right-side clearances are wrong**

`meterEdgeToGasEdgeM` assumes the gas meter is always left of the electric meter. Moving it to `x = 2.6` with the existing sizes gives `-1.0` m instead of the true `0.2` m gap. A correct placement result would then appear wrong against this ground truth.

Compute the horizontal gap from the ordered boxes, or use absolute centre separation minus their half-widths. Add a right-side scene test.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Comment on lines +547 to +548
def _features(rgb, t, frame_index, seed) -> list[tuple[float, float, float, float, float]]:
"""Deterministic 20-byte feature rows: u, v, response, scale, angle (float32 LE)."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Feature records use wrong fields

features.bin contains five floats for image coordinates, response, scale and angle. The app format requires three floats for a world-space point followed by a uint64 identifier. Both records occupy 20 bytes, so the validator accepts this unrelated layout.

Write actual scene points and identifiers, and test a record decoded using the app format. Otherwise the fixtures cannot check how app feature points are read.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 547-548

Comment:
**Feature records use wrong fields**

`features.bin` contains five floats for image coordinates, response, scale and angle. The app format requires three floats for a world-space point followed by a `uint64` identifier. Both records occupy 20 bytes, so the validator accepts this unrelated layout.

Write actual scene points and identifiers, and test a record decoded using the app format. Otherwise the fixtures cannot check how app feature points are read.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Comment on lines +659 to +660
pose = _pose_interp(scene, t)
quat = rotmat_to_quat_xyzw(_device_from_world(pose))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Stream orientations disagree

arkitPoses converts each camera rotation through DEVICE_FROM_CAMERA before writing its quaternion. This adds a fixed 90° rotation that the real app does not write, so stream orientations disagree with the matching keyframe poses. The fixtures therefore preserve the wrong convention even while validation passes.

Use pose[:3, :3] directly here and test the stream quaternion against the keyframe rotation.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 659-660

Comment:
**Stream orientations disagree**

`arkitPoses` converts each camera rotation through `DEVICE_FROM_CAMERA` before writing its quaternion. This adds a fixed 90° rotation that the real app does not write, so stream orientations disagree with the matching keyframe poses. The fixtures therefore preserve the wrong convention even while validation passes.

Use `pose[:3, :3]` directly here and test the stream quaternion against the keyframe rotation.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Comment on lines +1089 to +1096
[1.0, 0.0, 0.0, cx],
[0.0, 1.0, 0.0, cy],
[0.0, 0.0, 1.0, scene.wall_z],
[0.0, 0.0, 0.0, 1.0],
]
)
),
"center": [round(cx, 4), round(cy, 4), round(scene.wall_z, 4)],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Plane anchors use wrong axes

The app format puts a plane in local x-z, with local +y as its normal. Here the wall’s normal points up and the floor’s normal points sideways. Both center values are also world coordinates rather than coordinates local to the anchor.

Build the transforms using the app convention and keep the centres local. These fixtures otherwise cannot check plane geometry faithfully; current wall completion ignores the affected fields and hides the mismatch.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 1089-1096

Comment:
**Plane anchors use wrong axes**

The app format puts a plane in local `x-z`, with local `+y` as its normal. Here the wall’s normal points up and the floor’s normal points sideways. Both `center` values are also world coordinates rather than coordinates local to the anchor.

Build the transforms using the app convention and keep the centres local. These fixtures otherwise cannot check plane geometry faithfully; current wall completion ignores the affected fields and hides the mismatch.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

d_origin, d_dir = ray_grid(pose, DEPTH_W, DEPTH_H)
dt, _normal, _surface = cast_scene(scene, d_origin, d_dir)
depth = dt.astype("<f4")
confidence = np.where(np.isfinite(depth), np.where(depth <= 6.0, 3, 2).astype(np.uint8), np.uint8(0))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Confidence exceeds the app range

Near-depth pixels get confidence 3, but the app format only permits 0, 1 and 2, with 2 meaning high confidence. The fixtures therefore contain values no app capture produces.

Emit the app’s values and assert their range in the tests. The current validator only checks for all-low confidence, so its passing report does not catch this mismatch.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 542

Comment:
**Confidence exceeds the app range**

Near-depth pixels get confidence `3`, but the app format only permits `0`, `1` and `2`, with `2` meaning high confidence. The fixtures therefore contain values no app capture produces.

Emit the app’s values and assert their range in the tests. The current validator only checks for all-low confidence, so its passing report does not catch this mismatch.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

yaw_prev = _yaw(_pose_interp(scene, max(t - 0.005, 0.0)))
yaw_next = _yaw(_pose_interp(scene, min(t + 0.005, WALK_SECONDS)))
rot = [0.004, (yaw_next - yaw_prev) / 0.01, 0.002]
acc = grav + acc_noise[min(k, steps - 1)]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 User acceleration includes gravity

motion.accX..Z writes grav + acc_noise, but those fields represent CMDeviceMotion.userAcceleration, which excludes gravity. Only imuRaw.ax..az includes it. The motion stream therefore reports roughly 1 g of user acceleration without that movement.

Keep gravity out of the motion stream’s acceleration fields and test the difference between the two streams.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 625

Comment:
**User acceleration includes gravity**

`motion.accX..Z` writes `grav + acc_noise`, but those fields represent `CMDeviceMotion.userAcceleration`, which excludes gravity. Only `imuRaw.ax..az` includes it. The motion stream therefore reports roughly 1 g of user acceleration without that movement.

Keep gravity out of the motion stream’s acceleration fields and test the difference between the two streams.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

Comment on lines +622 to +624
yaw_prev = _yaw(_pose_interp(scene, max(t - 0.005, 0.0)))
yaw_next = _yaw(_pose_interp(scene, min(t + 0.005, WALK_SECONDS)))
rot = [0.004, (yaw_next - yaw_prev) / 0.01, 0.002]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Final rotation rates are false

After WALK_SECONDS, yaw_next stays frozen at 10 seconds while yaw_prev keeps advancing through the 13.5-second stream. Dividing that growing angle difference by 0.01 creates false angular speeds that disagree with the saved camera movement.

Sample both ends around the same time, or stop the pose and both samples together. Apply the same fix to imuRaw so the fixtures can check motion against poses.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 622-624

Comment:
**Final rotation rates are false**

After `WALK_SECONDS`, `yaw_next` stays frozen at 10 seconds while `yaw_prev` keeps advancing through the 13.5-second stream. Dividing that growing angle difference by `0.01` creates false angular speeds that disagree with the saved camera movement.

Sample both ends around the same time, or stop the pose and both samples together. Apply the same fix to `imuRaw` so the fixtures can check motion against poses.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

"""The meter close-up (head-on) and the oblique still the scale reference needs."""
meter_c = np.array([scene.meter["x"], scene.meter["centre_y"], scene.wall_z + scene.meter["size"][2] / 2.0])
close_pos = np.array([scene.meter["x"], scene.meter["centre_y"], scene.wall_z + 0.55])
oblique_pos = np.array([scene.meter["x"] + 0.38, scene.meter["centre_y"] + 0.12, scene.wall_z + 0.52])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Oblique view has wrong side

The oblique camera sits at meter.x + 0.38, which is the homeowner’s right under this scene’s coordinates, but its metadata and test call it left. Reviewers and tests receive a right-side image labelled as a left-side view.

Move the camera left or label the view right. Assert the side from the camera position rather than repeating a constant.

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/scripts/gen_app_capture.py
Line: 794

Comment:
**Oblique view has wrong side**

The oblique camera sits at `meter.x + 0.38`, which is the homeowner’s right under this scene’s coordinates, but its metadata and test call it `left`. Reviewers and tests receive a right-side image labelled as a left-side view.

Move the camera left or label the view `right`. Assert the side from the camera position rather than repeating a constant.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ effective changed lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants