diff --git a/AGENTS.md b/AGENTS.md index af4a8b6f5..abb4beb23 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,15 +13,15 @@ Edit the other team's files only with that team's OK. Talk to the author before ## Where to look -A component's README wins over the docs. Files marked with a pull request exist only on that branch until it merges. +A component's README wins over the docs. The pull requests named in the table are still open. | Task | Read | | --- | --- | -| The iOS app | `ios/README.md` (PR #10 for guided capture) | -| The capture packet | `packet/README.md` (PR #22) | -| The rules engine, API and scene contract | `server/README.md` (PR #11), especially "What settles each check" | -| Photos to a 3D model | `recon/HANDOFF.md` (PR #20) | -| Accuracy evals and the field test | `experiments/evals/README.md` (PR #12), and the first phone run in `experiments/device-field-test/README.md` (PR #23) | +| The iOS app | `ios/README.md`, with app work continuing in open PR #10 | +| The capture packet | `packet/README.md` | +| The rules engine, API and scene contract | `server/README.md` (newest revision in open PR #11), especially "What settles each check" | +| Photos to a 3D model | `recon/HANDOFF.md` | +| Accuracy evals and the field test | `experiments/evals/README.md`, and the first phone run in `experiments/device-field-test/README.md` | | Measurement conventions the code relies on | `docs/00-overview.md`, section "Conventions the code relies on" | | Public rule values, code citations, model and imagery licenses | `docs/04-prior-art-and-codes.md` | | The live guided-survey design | `docs/05-live-guided-survey-hld.md` | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c0b53fc1a..faa5adc3e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -53,9 +53,9 @@ Keep workflows that run pull-request code away from production credentials and d ### Before the first upload -App Store Connect rejects a build without an app icon. House Scan needs an `AppIcon` set with a 1024×1024 image in an asset catalog inside `ios/HouseScan/`, added by the client team. Measure Lab has one. +App Store Connect rejects a build without an app icon. House Scan and Measure Lab each have one: an `AppIcon` set with a 1024×1024 image, in `ios/HouseScan/Assets.xcassets/` and `experiments/measure-lab/MeasureLab/Assets.xcassets/`. -Neither `Info.plist` sets `ITSAppUsesNonExemptEncryption`, so each build waits under "Missing Compliance" until someone answers the encryption question on its TestFlight page. Setting the key to `false` removes that step for an app that uses only HTTPS and Apple's system encryption. +House Scan's `ios/Config/Info.plist` sets `ITSAppUsesNonExemptEncryption` to `false`. Measure Lab's `experiments/measure-lab/Config/Info.plist` does not set the key, so its builds wait under "Missing Compliance" until someone answers the encryption question on its TestFlight page. Setting the key to `false` removes that step for an app that uses only HTTPS and Apple's system encryption. ### One-time setup @@ -72,11 +72,11 @@ Neither `Info.plist` sets `ITSAppUsesNonExemptEncryption`, so each build waits u Set the secrets and variables with the [GitHub CLI](https://cli.github.com), so the key never appears on screen. `gh secret set` without `--body` prompts without echoing. ```sh - base64 -i AuthKey_.p8 | gh secret set ASC_KEY_P8 --env testflight --repo SamGu-NRX/house-scanning-master - gh secret set ASC_KEY_ID --env testflight --repo SamGu-NRX/house-scanning-master - gh secret set ASC_ISSUER_ID --env testflight --repo SamGu-NRX/house-scanning-master - gh variable set APPLE_TEAM_ID --env testflight --repo SamGu-NRX/house-scanning-master --body - gh variable set BUNDLE_ID_PREFIX --env testflight --repo SamGu-NRX/house-scanning-master --body + base64 -i AuthKey_.p8 | gh secret set ASC_KEY_P8 --env testflight --repo SamGu-NRX/BaseScanning + gh secret set ASC_KEY_ID --env testflight --repo SamGu-NRX/BaseScanning + gh secret set ASC_ISSUER_ID --env testflight --repo SamGu-NRX/BaseScanning + gh variable set APPLE_TEAM_ID --env testflight --repo SamGu-NRX/BaseScanning --body + gh variable set BUNDLE_ID_PREFIX --env testflight --repo SamGu-NRX/BaseScanning --body rm AuthKey_.p8 ``` diff --git a/README.md b/README.md index 4de84535d..4d9683f82 100644 --- a/README.md +++ b/README.md @@ -30,8 +30,8 @@ It replies with `"status":"ok"` and a note about which rules it's using. The dem To work on the code, you'll need [uv](https://docs.astral.sh/uv/), Node 24 with pnpm, and Xcode 26 or newer. Clone the repository with its submodules and run the tests: ```bash -git clone --recurse-submodules https://github.com/SamGu-NRX/house-scanning-master.git -cd house-scanning-master +git clone --recurse-submodules https://github.com/SamGu-NRX/BaseScanning.git +cd BaseScanning make check ``` @@ -62,7 +62,7 @@ Each part has its own folder: | --- | --- | --- | | iPhone app | Swift 6, SwiftUI, ARKit, RealityKit, Vision, Metal, XcodeGen | `ios/` | | Rules engine and API | Python 3.12, FastAPI, shapely, jsonschema, uv, pytest, deployed on Vercel | `server/` | -| 3D reconstruction | Python 3.12, MoGe-2 on PyTorch, OpenCV, NumPy, SciPy, scikit-image | `recon/` (PR #20) | +| 3D reconstruction | Python 3.12, MoGe-2 on PyTorch, OpenCV, NumPy, SciPy, scikit-image | `recon/` | | Reviewer view | TypeScript, Vite, Vitest, Biome | `web/` | | Landing page | Static site on Vercel | `sites/landing` | @@ -114,13 +114,13 @@ The battery measures 31 × 22 × 39.5 in, a bit bigger than a dishwasher. It sta | Driveway | At least 5 ft away | Placeholder, no public value | | Pool | At least 10 ft away | Placeholder, no public value | -Each check comes back PASS, FAIL or UNSURE, along with what the server measured, how far off that measurement could be and a reason in plain English. NEC is the National Electrical Code and IRC is the International Residential Code, the two building codes behind several of these rules. The numbers themselves live in `server/rules.yaml` on PR #11's branch, each one next to its source, and [docs/04](docs/04-prior-art-and-codes.md) has the full citations. +Each check comes back PASS, FAIL or UNSURE, along with what the server measured, how far off that measurement could be and a reason in plain English. NEC is the National Electrical Code and IRC is the International Residential Code, the two building codes behind several of these rules. The numbers themselves live in `server/rules.yaml`, each one next to its source, and [docs/04](docs/04-prior-art-and-codes.md) has the full citations. The server then gives one of three answers for the whole scan. It says `pass` when a spot passes every check. It says `reject` only when every spot within cable reach fails and the app knows where the wall ends on both sides. Anything in between is `manual_review`, and it goes to a person. ### One request, start to finish -Here's what that looks like in practice. We took the example scene from the server's tests in PR #11 and sent it to the demo server. The scene is made up. We wrote it by hand, with a gas meter, a window and an AC unit spread over two walls. The reply is real, though, trimmed to the parts worth reading. +Here's what that looks like in practice. We took the example scene from `server/tests/fixtures/example-scene.json` and sent it to the demo server. The scene is made up. We wrote it by hand, with a gas meter, a window and an AC unit spread over two walls. The reply is real, though, trimmed to the parts worth reading. The demo server keeps taking PR #11's updates, so a reply you get today won't match these numbers. The server stopped short of placing the battery itself. Its best spot was 9 ft 11 in left of the meter. From there, the AC unit around the corner measured 4 ft 1 in away, against a 3 ft rule. That sounds like a pass until you see the margin of error, which is give or take 3 ft 10 in. It's that wide because the AC unit sits far along the wall from the meter, where tracking error piles up the most. With an error that wide, the result is too close to call. So the answer is a manual review, plus a request to keep walking past the left end of the scan, because a spot within cable reach might be there. @@ -189,12 +189,11 @@ The server stopped short of placing the battery itself. Its best spot was 9 ft 1 ## Run the demo yourself -Before you start, know that most of the working system still lives in open pull requests. On `main` you'll find an AR session that shows tracking and a bare server skeleton, and not much else. Each step below names the PR it needs, and the [GitHub CLI](https://cli.github.com) checks one out for you with `gh pr checkout`. +Everything below runs from `main`, which now holds the whole system: the app, the placement server, the reconstruction worker and the experiments. Four pull requests are still open: #7 (Measure Lab), #10 (the guided wall scan), #11 (the server's newest revision, which the hosted demo runs) and #21 (a new 3D coverage model). -**Ask the demo server for a placement.** The demo server runs the engine from PR #11 with public rules only, and every answer says so. You still want PR #11 checked out, because that's where the example scene lives: +**Ask the demo server for a placement.** The demo server runs the engine from PR #11 with public rules only, and every answer says so. The example scene ships with the repository: ```bash -gh pr checkout 11 curl -s https://house-scanning-server.vercel.app/v1/placements \ -H 'Content-Type: application/json' \ --data-binary @server/tests/fixtures/example-scene.json @@ -202,10 +201,9 @@ curl -s https://house-scanning-server.vercel.app/v1/placements \ For a drawing of the wall with the chosen spot, post the same scene to `/v1/placements/site-plan.svg` instead. -**Run the server on your own machine.** From PR #11's branch, install the locked dependencies and start the API on port 8000: +**Run the server on your own machine.** From the repository root, install the locked dependencies and start the API on port 8000: ```bash -gh pr checkout 11 cd server uv sync --locked uv run uvicorn api:app --host 0.0.0.0 --port 8000 @@ -213,13 +211,7 @@ uv run uvicorn api:app --host 0.0.0.0 --port 8000 Then send the same `curl` request to `http://localhost:8000` instead of the demo server. -**Run the app.** Guided capture lives in PR #10, which is a different branch from the server's, so switch to it first: - -```bash -gh pr checkout 10 -``` - -You don't need a phone to try it. Build `ios/HouseScan.xcodeproj` for the Simulator and pass the launch arguments `-replay -autopilot -serverURL https://house-scanning-server.vercel.app`. `-replay` plays a recorded capture in place of the camera, and `-autopilot` steps through every screen for you. PR #10 comes with two made-up captures you can replay, in `ios/HouseScanUITests/Fixtures/`. +**Run the app.** `main` has the whole guided walk: the meter close-up, the walk along the wall, the upload and the AR result. You don't need a phone to try it. Build `ios/HouseScan.xcodeproj` for the Simulator and pass the launch arguments `-replay -autopilot -serverURL https://house-scanning-server.vercel.app`. `-replay` plays a recorded capture in place of the camera, and `-autopilot` steps through every screen for you. Two made-up captures ship in `ios/HouseScanUITests/Fixtures/`: `synthetic-wall` and `synthetic-wall-lidar`. To run it on a real iPhone, set up signing first: @@ -246,7 +238,7 @@ You don't need any API keys, and the server runs the public rules with nothing s # HOUSESCAN_API_KEY= ``` -Two more belong to other parts of the project. The reconstruction worker in PR #20 looks for public datasets and cached models in `HOUSE_SCANNING_DATA`, which defaults to `~/house-scanning-data`, and posts its results to the server named in `HOUSESCAN_SERVER`. TestFlight uploads use repository secrets, and [CONTRIBUTING.md](CONTRIBUTING.md) describes them. +Two more belong to other parts of the project. The reconstruction worker in `recon/` looks for public datasets and cached models in `HOUSE_SCANNING_DATA`, which defaults to `~/house-scanning-data`, and posts its results to the server named in `HOUSESCAN_SERVER`. TestFlight uploads use repository secrets, and [CONTRIBUTING.md](CONTRIBUTING.md) describes them. ## Where our data came from @@ -273,21 +265,21 @@ A few terms first, in case they're new to you. Tracking drift is how far the pho | What | Result | Source | | --- | --- | --- | -| Tracking drift, recent iPhone | 8.6, 13.4 and 18.5 in (p90) after 10, 20 and 30 ft, inside the server's allowance of 19.2, 38.4 and 57.6 in | MARViN, iPhone 14 Pro Max, PR #12 | -| Tracking drift, older iPhone | Two to three times over that allowance | ADVIO, iPhone 6s, PR #12 | -| Learned depth on its own | Scale 4 to 12% off, which puts walls about 20 in out (p90) | ETH3D, PR #12 | -| Learned depth, rescaled with the phone's poses | Walls within about 5 in (p90), or 2.8 in with exact poses. Edges stay at 8 in or worse | ETH3D, PR #12 | -| Reconstruction worker | Walls within 2.1 in (p90) with a laser scan standing in for LiDAR, and 2.8 in from photos only | ETH3D, PR #20 | -| Reading the meter number | Read in full on 71 of 73 photos, but the right line on only 21 of 75. A list of three candidates held it on 27 of 34 held-out photos | Photos of real meters, PR #16 | -| First run on our phone | Both wall ends landed at the meter, so the server placed no spot. Two of five features came within 4 in of the tape | TestFlight build, PR #23 | +| Tracking drift, recent iPhone | 8.6, 13.4 and 18.5 in (p90) after 10, 20 and 30 ft, inside the server's allowance of 19.2, 38.4 and 57.6 in | MARViN, iPhone 14 Pro Max, `experiments/evals/results/modern_arkit.md` | +| Tracking drift, older iPhone | Two to three times over that allowance | ADVIO, iPhone 6s, `experiments/evals/results/advio_drift.md` | +| Learned depth on its own | Scale 4 to 12% off, which puts walls about 20 in out (p90) | ETH3D, `experiments/evals/results/eth3d_recon.md` | +| Learned depth, rescaled with the phone's poses | Walls within about 5 in (p90), or 2.8 in with exact poses. Edges stay at 8 in or worse | ETH3D, `experiments/evals/results/pose_priors.md` | +| Reconstruction worker | Walls within 1.8 in (p90) with a laser scan standing in for LiDAR, and 1.5 in from photos only | ETH3D, `recon/results/eth3d_electro.md` | +| Reading the meter number | Read in full on 71 of 73 photos, but the right line on only 21 of 75. A list of three candidates held it on 27 of 34 held-out photos | Photos of real meters, `experiments/meter-closeup/results/clean.md` and `locate.md` | +| First run on our phone | Both wall ends landed at the meter, so the server placed no spot. Two of five features came within 4 in of the tape | TestFlight build, `experiments/device-field-test/README.md` | ## What doesn't work yet Here's what we know is either missing or broken: -- **Most of it isn't merged.** Guided capture is in PR #10, the rules engine in PR #11, reconstruction in PR #20 and the packet spec in PR #22. All four still live on their own branches. +- **Four pull requests are still open.** #10 and #21 carry app work (the guided wall scan, and a new 3D coverage model), #7 carries Measure Lab, and #11 is the server's newest revision, which the hosted demo runs. Reconstruction, the packet notes, the evals, meter reading and the first field test have all merged to `main`. - **Our first real phone run placed nothing.** The app put both ends of the wall right at the meter, which left the server a wall with no length to search (PR #23). -- **We haven't settled on how to build the 3D model.** A depth model on its own puts walls about 20 in off. Correcting its scale with the phone's poses brings that down to about 5 in, but edges stay 8 in off or worse, and the clearance rules measure from edges. With a laser scan standing in for LiDAR, walls land within 2.1 in. +- **We haven't settled on how to build the 3D model.** A depth model on its own puts walls about 20 in off. Correcting its scale with the phone's poses brings that down to about 5 in, but edges stay 8 in off or worse, and the clearance rules measure from edges. With a laser scan standing in for LiDAR, walls land within 1.8 in. - **Our best tracking result came from a phone with LiDAR.** An iPhone 14 Pro Max stayed inside the error allowance, but it has LiDAR, and most homeowners' phones don't. An older iPhone 6s ran two to three times over. - **Nobody has taken on hidden walls yet.** A bush or a trash can in front of the wall can hide what's behind it, and on phones without LiDAR, nothing checks for that. - **Two rules use placeholder numbers.** We couldn't find a public value for how far a battery should sit from a pool or a driveway, so for now they're 10 ft and 5 ft. @@ -298,7 +290,7 @@ Here's what we know is either missing or broken: If we keep going, this is where we'd start: -1. Take a current iPhone without LiDAR to a real wall and run the field test in `experiments/evals/field/FIELD_SHEET.md` (PR #12). The same trip checks our default error bars against a tape measure. +1. Take a current iPhone without LiDAR to a real wall and run the field test in `experiments/evals/field/FIELD_SHEET.md`. The same trip checks our default error bars against a tape measure. 2. Pick a way to build the 3D model. We'd also like to try world models, which generate a whole 3D scene from photos or video. Nobody here has tested one yet. 3. Decide who owns the hidden-wall check. One idea is to show the homeowner the photo of the chosen spot and ask them. 4. Swap the placeholder values for Base's real ones on the private deployment. @@ -314,16 +306,17 @@ If we keep going, this is where we'd start: ## Where everything lives -Anything marked with a pull request exists only on that PR's branch until it merges. +Everything in the table is on `main`. The four open pull requests are #7, #10, #11 and #21. | Path | What it is | State | | --- | --- | --- | -| `ios/` | The iPhone app | On `main`, an AR session that shows tracking. Guided capture is in PR #10, and the live 3D map in PR #21 | -| `packet/` | The capture packet's spec, validator and samples | PR #22 | -| `server/` | The rules engine and placement API | On `main`, a skeleton. The engine is in PR #11 | -| `recon/` | Turns photos and depth into a 3D model and a coverage map | PR #20, handed to the server team | -| `experiments/` | One folder per experiment | Accuracy evals in PR #12, Measure Lab in PR #7, scoring in PR #4, meter reading in PR #16, the first device field test in PR #23 | -| `web/` | Browser toolchain for a reviewer view | No page on `main` | +| `ios/` | The iPhone app | On `main`, the whole guided walk through the AR result. App work continues in open PRs #10 and #21 | +| `packet/` | The app side's notes on the server team's capture packet | On `main` | +| `server/` | The rules engine and placement API | On `main`. The hosted demo runs the newest revision, in open PR #11 | +| `recon/` | Turns photos and depth into a 3D model and a coverage map | On `main`, with benchmark results in `recon/results/` | +| `experiments/` | One folder per experiment | On `main`: accuracy evals, Measure Lab, scoring, meter reading, the first device field test, and more | +| `verification/` | Checks that the app and server do what the plan says, with a scoreboard | On `main` | +| `web/` | Browser app for capture experiments | A placeholder page and a units library; no reviewer view yet | | `docs/` | The overview, the walkthrough, public rules and code citations, and the live-survey design | | | `.agents/skills/` | Shared agent skills for writing, planning and review, linked from `.claude/skills/` | | | `sites/landing` | The landing page, a submodule | Change it in its own repository |