Skip to content

experiments: signals lab round 1, which extra phone signals sharpen clearances - #59

Merged
AidenJohnston merged 3 commits into
mainfrom
t3/signals-lab
Sep 27, 2026
Merged

AidenJohnston merged 3 commits into
mainfrom
t3/signals-lab

Conversation

@SamGu-NRX

@SamGu-NRX SamGu-NRX commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Four experiments asking which extra things the phone could capture to make clearance measurements more accurate. Pass criteria were committed in 397507f before any run. Everything runs on public data with laser or surveyed truth, or is closed-form physics. No app or server code changes.

Experiment Question Result
edge-geometry How does view angle affect edge error along the wall, and can an edge be placed without reading depth at it? P1 passes: in-plane edges are 0.9 in p90 within 15° of face-on and 24 in past 45°. The tap, snap and median method and two-photo matching fail. A plane fitted near the meter, as the app does, has a yaw error of 2.2° median and 6.7° p90.
drift-anatomy Which part of ARKit's drift is worth attacking? All four fail. At 20 ft, along-travel p90 is 9.6 of 10.4 in (MARViN). A return-to-meter residual bounds 54% of loops. Scale wanders 1.8% within a walk. UWB helps only at σ 5 cm.
plane-consistency Can photo agreement through the wall plane catch occluders without LiDAR? Fails: 1.1 ft still falsely claimed and 9.2 ft missed, against 0.5 and 3.3.
sensor-budget What error does physics allow for signals we can't test on data? Dropped: dual-camera disparity, focus distance, the barometer, and a meter cover of unknown make. Worth a device test: an ID card or Letter sheet at the meter, one photo holding both ends of a span, and a contact probe.

What this suggests, pending the manager's call:

Each folder runs with uv run python run.py (or budget.py), and uv run pytest -q passes in each. The three data experiments import the evals harness read-only from t3/evals at 190ed33 (#12).

Limits:

  • ETH3D is two DSLR-photographed institutional buildings. The edge results use an oracle depth scale.
  • MARViN's phone has LiDAR, and its reference scale is unverified.
  • Each README is a short front door that keeps the pre-registered pass criteria word for word. Method detail and full results are in each folder's METHODS.md.

🤖 Generated with Claude Code

SamGu-NRX and others added 2 commits September 26, 2026 18:57
Four experiments for which extra phone signals would sharpen geometry:
edge geometry, plane consistency (occlusion without LiDAR), drift
anatomy and a sensor error budget. Pass criteria are fixed here, before
any run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- edge-geometry: along-wall error at edges is set by view angle. Within
  15 degrees of face-on it is 0.9 in at p90, and past 30 degrees 20 in.
  One wall normal fitted near the meter, as the app does, adds
  s*psi*tan(angle) with psi at a median 2.2 degrees.
- drift-anatomy: 92% of ARKit's error runs along the direction of travel.
  A return-to-meter residual can't bound it, and scale wanders 1 to 2%
  within a walk.
- plane-consistency: photo agreement through the wall plane can't
  replace a depth test on phones without LiDAR (1.1 ft false, 9.2 ft
  missed).
- sensor-budget: closed-form budgets drop dual-camera disparity, focus
  distance, the barometer and a meter cover of unknown make.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: c29b404a-0ac8-459b-bf33-8684ec28046c

📥 Commits

Reviewing files that changed from the base of the PR and between f852207 and 0cde05e.

⛔ Files ignored due to path filters (4)
  • experiments/drift-anatomy/uv.lock is excluded by !**/*.lock
  • experiments/edge-geometry/uv.lock is excluded by !**/*.lock
  • experiments/plane-consistency/uv.lock is excluded by !**/*.lock
  • experiments/sensor-budget/uv.lock is excluded by !**/*.lock
📒 Files selected for processing (35)
  • experiments/drift-anatomy/METHODS.md
  • experiments/drift-anatomy/README.md
  • experiments/drift-anatomy/metrics.py
  • experiments/drift-anatomy/pyproject.toml
  • experiments/drift-anatomy/results/drift_anatomy.json
  • experiments/drift-anatomy/results/drift_anatomy.md
  • experiments/drift-anatomy/run.py
  • experiments/drift-anatomy/tests/test_metrics.py
  • experiments/drift-anatomy/walks.py
  • experiments/edge-geometry/METHODS.md
  • experiments/edge-geometry/README.md
  • experiments/edge-geometry/geom.py
  • experiments/edge-geometry/imagematch.py
  • experiments/edge-geometry/pyproject.toml
  • experiments/edge-geometry/results/edge_geometry.json
  • experiments/edge-geometry/results/edge_geometry.md
  • experiments/edge-geometry/run.py
  • experiments/edge-geometry/stats.py
  • experiments/edge-geometry/tests/test_geom.py
  • experiments/edge-geometry/tests/test_imagematch.py
  • experiments/plane-consistency/METHODS.md
  • experiments/plane-consistency/README.md
  • experiments/plane-consistency/plane_filter.py
  • experiments/plane-consistency/pyproject.toml
  • experiments/plane-consistency/results/plane_consistency.json
  • experiments/plane-consistency/results/plane_consistency.md
  • experiments/plane-consistency/run.py
  • experiments/plane-consistency/tests/test_plane_filter.py
  • experiments/sensor-budget/METHODS.md
  • experiments/sensor-budget/README.md
  • experiments/sensor-budget/budget.py
  • experiments/sensor-budget/pyproject.toml
  • experiments/sensor-budget/results/budget.json
  • experiments/sensor-budget/results/budget.md
  • experiments/sensor-budget/tests/test_budget.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the size:XXL 1,000+ effective changed lines. label Sep 27, 2026
…hort front door

The pass criteria stay in each README word for word, as pre-registered
in 397507f.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@SamGu-NRX

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-27T02:38:15.612103Z 0cde05e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

SamGu-NRX added a commit that referenced this pull request Sep 27, 2026
…out the server's paths

From #59: the wall plane's source (single raycast yaw 2.2° median, 6.7°
p90), span photos, scale references in view, the meter anchor's pose over
time, and an optional UWB distance stream. The server's repository is
private, so validator-checks.md no longer names its internal files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0cde05e30a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"p90_3d_in": p90(np.linalg.norm(err, axis=1)) / INCH,
"along_share_of_total_p90": p90(np.abs(along)) / p90(horiz),
}
true_range = np.linalg.norm(ref, axis=1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Compute UWB ranges from the fixed meter anchor

For every sliding window that does not start at the physical meter, ref is an end-minus-window-start displacement, so this range and the subsequent radial correction simulate moving the ranging phone to each window's start. This does not test A4's stated scenario of a second phone left at the meter and can materially change the reported 20/30 ft residuals; use positions relative to one fixed meter origin, or restrict the analysis to windows that start there.

AGENTS.md reference: AGENTS.md:L36-L36

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at experiments/drift-anatomy/run.py:271. A4 ranges use window displacement, so anchor moves. Design must use endpoint positions relative to one fixed physical meter and rotate/apply estimator corrections in that same frame. Existing UWB residuals do not support fixed-meter claims.

Comment thread experiments/sensor-budget/budget.py
Comment thread experiments/edge-geometry/run.py
Comment thread experiments/drift-anatomy/run.py
Comment on lines +21 to +22
[tool.pytest.ini_options]
testpaths = ["tests"]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wire the new experiment suites into make check

This commit adds four independent pytest configurations and test directories, but make check still runs only the server, web, and iOS targets, and no workflow invokes these experiment suites. Consequently regressions in the geometry and grading code can merge without any of the added tests running; add experiment test targets to the repository check/CI path.

AGENTS.md reference: AGENTS.md:L54-L54

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at Makefile:26. Four suites absent from make check and CI. CI edits forbidden and root Makefile outside assigned paths; Sam must authorize check/CI wiring.


**Run:** `uv run python run.py`: 2 min, 0.9 GB. Needs the evals checkout at 190ed33 and its ETH3D and MoGe-2 caches.

**What it changes:** ask for each edge from within about 20° of face-on, record the tap's view angle, and fit the wall plane across the walk.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Link the plan-changing results from the evidence overview

These experiments all state concrete product or plan changes, but none is linked from docs/00-overview.md. The repository's experiment guide requires a line under “Evidence so far” whenever the product or plan relies on a result, so the authoritative overview remains stale and future work cannot trace its new capture and plane decisions back to these measurements. Add overview entries for the results that the plan now relies on.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still open at experiments/edge-geometry/README.md:28. Plan-changing results still absent from overview. docs/00-overview.md outside worker scope; lead/Sam must add evidence entries after correcting invalid drift and scoring claims.

Comment thread experiments/plane-consistency/run.py
SamGu-NRX added a commit that referenced this pull request Sep 27, 2026
…the checks' limits

From #22's review and Codex review: the eleven items are proposals, not
contract; #59's numbers name their proxy, dataset and simulation; the
retired sharpness and consent fields are not claimed to be covered; window
operability is evidence for the rules; the boundary frame goes in a new
field. In the proposed checks: positions only (not rotation), shape not
alignment for depth, non-negative indices and finite vertices for the mesh,
finiteness of every parsed number. The pointer no longer lists the intake
routes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SamGu-NRX added a commit that referenced this pull request Sep 27, 2026
- Hazards: a huge JSON integer parses as an int, and math.isfinite raises on it; check integers
  against their field's range and report failures as problems.
- Proposal 10: each anchor-pose sample needs its timestamp and epoch.
- Proposal 11: the #59 result assumes a still second phone at a known place; the proposal
  requires that, a time per range and the range's uncertainty.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@SamGu-NRX

Copy link
Copy Markdown
Owner Author

@greptileai

@greptile-apps

greptile-apps Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 3/5

[Low risk] Adds four isolated experiment directories with analysis code.

The PR does not appear safe to merge until the drift experiment measures the wall-plane and fixed-meter UWB scenarios it claims to evaluate.

Fix All in CodexFindings

  1. P1 Range anchor moves ▶
  2. P1 Wrong axis for wall plane ▶
  3. P2 Match rate excludes unpaired points ▶
  4. P2 Pair selection favors narrow angles ▶
Fix with agent prompt
### Issue 1
experiments/drift-anatomy/run.py:271-275
A4 is meant to test a second phone left at the meter. Here, each simulated range runs from a walk window’s start to its end, so the anchor moves with every window. The reported UWB errors and failure verdict therefore do not measure the proposed fixed-meter setup.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
experiments/drift-anatomy/run.py:262-266
A3 judges whether a wall plane would help by splitting error along and across the camera’s direction of travel. A wall plane instead constrains error along the wall’s fixed normal. When the camera does not travel parallel to the wall, these directions differ, so the reported split and pass criterion cannot establish the proposed wall-plane benefit.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 3
experiments/edge-geometry/run.py:830-832
P4 requires at least 70% of points to match, but this denominator includes only points for which a second photo was selected. On electro, only 2,489 of 3,000 sampled points have such a pair. The rest do not count as unmatched, so the reported match rate overstates coverage of the sampled points, although P4 still fails.

### Issue 4
experiments/edge-geometry/run.py:454-461
M3 selects the smallest ray angle above its 5° minimum. A pair near that minimum provides weaker triangulation than a wider-angle pair, even when another visible photo is available. This choice can inflate the reported two-photo position error; evaluate or justify the pairing policy before treating that error as representative.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Summary

The PR adds four reproducible experiments on edge placement, AR drift, photo-based wall consistency, and candidate sensor error budgets.

  • The drift experiment’s wall-plane and fixed-meter UWB conclusions rely on measurements of different geometries than those stated.
  • Two earlier edge-geometry findings remain open; the earlier wall-origin thread is resolved.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  W["Reference walk windows"] --> E["ARKit displacement error"]
  E --> A3["A3: split by travel direction"]
  E --> A4["A4: range from window start"]
  A3 -. "Intended comparison" .-> P["Fixed wall-plane normal"]
  A4 -. "Intended anchor" .-> M["Fixed phone at meter"]
Loading

Reviews (2) · Last reviewed commit: "experiments: move method detail into MET..."

Comment thread experiments/plane-consistency/run.py
Comment thread experiments/edge-geometry/run.py
Comment thread experiments/edge-geometry/run.py
@AidenJohnston
AidenJohnston marked this pull request as ready for review September 27, 2026 15:27
@AidenJohnston
AidenJohnston merged commit 0bdca46 into main Sep 27, 2026
3 checks passed
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@codemetrics-ai

Copy link
Copy Markdown

Note

Review skipped

Codemetrics is installed on this repository, but it hasn't been added to your team yet.

To enable AI code reviews for this repo, go to your dashboard and add house-scanning-master to your team's repositories.

Comment on lines +271 to +275
true_range = np.linalg.norm(ref, axis=1)
a4[dataset][ft] = {"before_p90_in": p90(horiz) / INCH}
for sigma in RANGE_SIGMAS_M:
noise = rng.normal(0.0, sigma, size=(RANGE_DRAWS, len(ref)))
measured = np.maximum(true_range[None] + noise, 0.0)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Range anchor moves A4 is meant to test a second phone left at the meter. Here, each simulated range runs from a walk window’s start to its end, so the anchor moves with every window. The reported UWB errors and failure verdict therefore do not measure the proposed fixed-meter setup.

Prompt To Fix With AI
This is a comment left during a code review.
Path: experiments/drift-anatomy/run.py
Line: 271-275

Comment:
**Range anchor moves** A4 is meant to test a second phone left at the meter. Here, each simulated range runs from a walk window’s start to its end, so the anchor moves with every window. The reported UWB errors and failure verdict therefore do not measure the proposed fixed-meter setup.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Codex Fix in Claude Code

Comment thread experiments/drift-anatomy/run.py
SamGu-NRX pushed a commit that referenced this pull request Oct 15, 2026
The dev deploy defaults to none when that variable is unset. Setting it to static or jwt turns auth back on without a code change.

Co-authored-by: hss-cloud-agent <hss-cloud-agent@users.noreply.github.com>
@SamGu-NRX
SamGu-NRX deleted the t3/signals-lab branch October 15, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ effective changed lines.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants