experiments: meter close-up reading, tap-to-confirm and two retake checks

.github/workflows/meter-closeup.yml

@@ -1,11 +1,16 @@
 name: Meter close-up experiment
 
+# The leak check needs METER_HMAC_KEY. Pull-request code must never receive it, because the key
+# lets anyone reverse the short meter numbers behind the committed digests. So pull requests get
+# lint and tests only, and the keyed job runs only on main: after merge, or dispatched from main.
+# A dispatch from any other branch would run that branch's code with the key, so it is skipped.
+# Before asking for review, run `make leakcheck` locally.
 on:
+  workflow_dispatch:
+  # Every push to main, not only experiment changes: the keyed leak check scans the whole
+  # repository, and a copied meter number can land in any file.
   push:
     branches: [main]
-    paths:
-      - "experiments/meter-closeup/**"
-      - ".github/workflows/meter-closeup.yml"
   pull_request:
     paths:
       - "experiments/meter-closeup/**"
@@ -38,8 +43,24 @@ jobs:
       - run: uv run ruff check .
       - run: uv run ruff format --check .
       - run: uv run pytest -q
-      # Looks for any transcribed meter identifier in the tracked files. The key stays in a
-      # repository secret; without it the step fails with an explanation.
+
+  leakcheck:
+    name: Meter close-up leak check
+    if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
+    runs-on: ubuntu-24.04
+    timeout-minutes: 10
+    defaults:
+      run:
+        working-directory: experiments/meter-closeup
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0
+        with:
+          working-directory: experiments/meter-closeup
+      - run: uv sync --locked
+      # Looks for any transcribed meter identifier in the tracked files.
       - run: uv run python -m meter_eval.leakcheck
         env:
           METER_HMAC_KEY: ${{ secrets.METER_HMAC_KEY }}

docs/00-overview.md

@@ -52,7 +52,7 @@ These results come from public datasets with laser-scanned or surveyed ground tr
 - **Learned depth.** A model's own scale is 4 to 12% off. Rescaled with the phone's poses, walls come within about 5 in at p90 at a realistic 2% pose error, and within 2.8 in with exact poses. Edges stay at 8 in or worse, and clearances are measured from edges. MapAnything's scale shifted with input resolution.
 - **Reconstruction worker.** On the ETH3D building, wall p90 is 2.1 in with a laser scan standing in for LiDAR, and 2.8 in from photos only. The two runs measured different walls.
 - **Coverage.** The app's first coverage map claimed 1.1 ft of a 19.3 ft wall that no photo saw. Requiring several view angles removed the false claim but also discarded 6 to 8.5 ft of good wall. A depth test on LiDAR phones removed it cleanly. Checked against depth, the reconstruction worker claims at most about 0.46 ft of unseen wall.
-- **Meter reading.** Apple's on-device text recognition read the full meter number on 71 of 73 real photos. It picked the right line on only 21 of 75, because nameplates carry several numbers. A barcode held the number on 20 of the 24 photos that had one. A list of three candidates held it on 85% of held-out photos. So the app offers three candidates to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small.
+- **Meter reading.** Apple's on-device text recognition read the full meter number on 71 of 73 real photos. It picked the right line on only 21 of 75, because nameplates carry several numbers. A barcode held the number on 20 of the 24 photos that had one. A list of three candidates held it on about 4 in 5 held-out photos (27 of 34). So the app offers three candidates to tap, barcode match first, and asks for a retake when the photo is blurry or the number too small.
 - **Panel labels.** Not tested. A search found 7 usable open photos of US panels, and the test needs 40.
 - **World models.** Not evaluated yet.
 - **First phone run.** The TestFlight build's first run on a real wall (PR #23) set both wall ends at the meter, so the server placed no spot. Two of five measured features came within 4 in of the tape.

docs/how-it-works.html

@@ -981,7 +981,7 @@ <h3>Meter reading</h3>
   <figure class="narrow">
     <svg viewBox="0 0 480 230" role="img" aria-labelledby="c4t c4d">
       <title id="c4t">Meter reading success on real meter photos</title>
-      <desc id="c4d">Four horizontal bars on a 0 to 100 percent scale. Read the full number: 71 of 73, 97 percent. Picked the right line: 21 of 75. Barcode held the number: 20 of 24 photos that had one. A tap list of three candidates contained it: 85 percent of held-out photos.</desc>
+      <desc id="c4d">Four horizontal bars on a 0 to 100 percent scale. Read the full number: 71 of 73, 97 percent. Picked the right line: 21 of 75. Barcode held the number: 20 of 24 photos that had one. A tap list of three candidates contained it: about 4 in 5 held-out photos, 27 of 34, 79 percent.</desc>
       <!-- x = 190 + pct*2.6 -->
       <g class="grid"><line x1="190" y1="20" x2="190" y2="188"/><line x1="255" y1="20" x2="255" y2="188"/><line x1="320" y1="20" x2="320" y2="188"/><line x1="385" y1="20" x2="385" y2="188"/><line x1="450" y1="20" x2="450" y2="188"/></g>
       <line x1="190" y1="188" x2="450" y2="188" class="axis"/>
@@ -990,13 +990,13 @@ <h3>Meter reading</h3>
         <text x="180" y="44">full number read</text><text x="180" y="86">right line picked</text><text x="180" y="128">barcode had it</text><text x="180" y="170">in the tap list of 3</text>
       </g>
       <g fill="var(--phone)">
-        <rect x="190" y="34" width="252.9" height="14" rx="4"/><rect x="190" y="76" width="72.8" height="14" rx="4"/><rect x="190" y="118" width="216.7" height="14" rx="4"/><rect x="190" y="160" width="221" height="14" rx="4"/>
+        <rect x="190" y="34" width="252.9" height="14" rx="4"/><rect x="190" y="76" width="72.8" height="14" rx="4"/><rect x="190" y="118" width="216.7" height="14" rx="4"/><rect x="190" y="160" width="206.5" height="14" rx="4"/>
       </g>
       <g class="lab s">
         <text x="192" y="29">71 of 73, 97%</text>
         <text x="192" y="71">21 of 75</text>
         <text x="192" y="113">20 of 24 photos that had one</text>
-        <text x="192" y="155">85% of held-out photos</text>
+        <text x="192" y="155">about 4 in 5 held-out photos (27 of 34)</text>
       </g>
     </svg>
     <figcaption>The phone reads the number but usually picks the wrong line, because plates carry several numbers.</figcaption>

experiments/meter-closeup/FIELD_TEST.md

@@ -0,0 +1,21 @@
+# How to check the retake thresholds on an iPhone
+
+The thresholds in [PORTING.md](PORTING.md) come from processed Commons JPEGs, not from the app's camera. Sharpening and noise reduction change sharpness values, so the focus threshold may move. Test both thresholds on one real meter before relying on them.
+
+1. Take the photos below with the app's capture path, or the iPhone camera if the app is not ready.
+2. Copy them into one folder. JPEG and HEIC both work. The command measures each photo's own pixels and never re-encodes them, because a JPEG round trip can lift sharpness across the threshold.
+3. From this folder, run:
+
+	```sh
+	uv run python -m meter_eval.fieldtest PHOTO_DIR --number "<number as printed>"
+	```
+
+The command prints each photo's read result, the candidate's rank and the check values. It then counts retakes asked for photos that read, and photos accepted that did not. Like the app, its checks use the ranking's top candidate; `--number` only scores the outcome and is never written to disk.
+
+| Threshold | Photos to take | The threshold holds if |
+|---|---|---|
+| Out of focus | three with focus locked on a distant background, one from 5 cm, three sharp | blurred photos score 6.68 or less and fail to read, and sharp photos score far above |
+| Number too small | square to the meter from 15, 30, 50, 80 and 120 cm | photos whose top candidate is taller than 33.9 px read, with the number in the top three |
+| Glare (no check) | a second phone's flashlight beside the camera at 0°, 20° and 45°, or direct sun on the number | it records how often glare alone breaks reading |
+| Cut off (no check) | the frame edge cutting a quarter digit, half a digit and a whole digit | it records whether a cut number ever becomes a candidate |
+| Hand shake (no check) | three in shade while sweeping the phone sideways | it records how often shake alone breaks reading; if it does, gate capture on gyroscope motion |

experiments/meter-closeup/METHODS.md

@@ -0,0 +1,92 @@
+# How the meter close-up numbers were made
+
+This file says how each number in [README.md](README.md) was produced and how far to trust it. The tables themselves are in `results/`, each with the command that generated it.
+
+## Photos
+
+83 photos from Wikimedia Commons, all CC0, public domain, CC BY or CC BY-SA. They come from the `Electricity meters (kWh)` category and meter-brand searches. `manifest.csv` records each photo's URL, license and author. `make images` downloads them to `~/house-scanning-data/meter`; they are never committed.
+
+Three photos are excluded: two show several meters, and one label shows a customer's name and address. Of the 80 usable photos, 22 carry a US-style class label (CL200, 200 CL, CL20) and 57 an IEC current rating such as `10(60)A`. Most meters are Taiwanese, European or Canadian; US residential meters are a minority.
+
+## Labels
+
+Two AI models, not people, read every photo, so every number is only as good as their labels. Reader 1 (Claude Opus, the experiment's author) zoomed into each original. Reader 2 (a separate Claude Sonnet agent) read each photo without seeing reader 1's labels. The label is always reader 1's number, so reader 1 chose which printed identifier is the meter number: a utility plate over a maker serial.
+
+`labels.py` records two agreement rules:
+
+- **Loose** (73 photos, used for the headline): reader 1's number equals any number reader 2 listed, or one contains the other and the shorter has at least 6 characters, and neither reader doubted a character.
+- **Strict** (62 photos): both readers' main numbers are identical and both were sure. Reading and ranking results are reported under both rules.
+
+To confirm the labels by hand, run `make review` and open `~/house-scanning-data/meter-closeup/review.html`. Each photo shows whole, beside the agreed number. Press K to keep the number, or F to type a correction. Then run `uv run python -m meter_eval.review ingest <downloaded csv>` and rebuild with `uv run python -m meter_eval.labels`. A kept number counts as agreed under both rules; a corrected one does not until someone keeps it. Ingest merges new answers into earlier ones, and a rebuilt page shows each earlier correction in place of the readers' number, so Keep confirms the correction even in a new browser.
+
+## Keeping meter numbers out of the repository
+
+`CONTRIBUTING.md` forbids committing meter numbers. `manifest.csv` stores each one as an HMAC-SHA256 digest. An unkeyed hash of a short number can be reversed by trying every number of that length. The keyed digests cannot be reversed without the key, which lives in `~/house-scanning-data/meter-closeup/hmac.key` and in the `METER_HMAC_KEY` repository secret. To start over without the key, run `uv run python -m meter_eval.labels --new-key` and rebuild the manifest from the plaintext labels.
+
+`make leakcheck` digests every run of 4 or more digits in every tracked file of the repository, not only this folder, and fails on any identifier a reader transcribed; the shortest is a four-digit utility plate number. Only cells in the measurement columns listed in `leakcheck.py`, which this code fills with measured numbers, are exempt. Any other cell is scanned whatever its shape, and other files are scanned whole with every separator dropped, as `normalize()` drops them, so an identifier written with dots, slashes, colons or line breaks is still caught. Before the labels' free-text notes reach `manifest.csv`, every digit the leak check could count toward an identifier becomes `#`: any digit within 24 letters and digits of three others, once separators are dropped.
+
+Pull-request code never receives the key, because the key would let it reverse the digests. CI runs the keyed check only on `main`, after merge or when dispatched from `main`; run `make leakcheck` locally before asking for review.
+
+## Reading (question 1)
+
+`meterocr/` is a Swift tool that calls Vision's `VNRecognizeTextRequest` (revision 3, `en-US`, `minimumTextHeight` 0) on macOS, the API the iPhone app would call. The headline uses the `accurate` level without language correction. A number counts as read when the whole labelled number appears in one recognized line, or in lines sharing a row. `results/clean.md` has every configuration. The `fast` level misses a quarter of numbers, and language correction adds nothing.
+
+The class-label matcher was loosened after the first run to ignore separators and read `×` as `x`, because every first-run miss was one of those. The meter-number matcher was never changed.
+
+## Retake checks (question 2)
+
+The 71 photos whose agreed number read correctly were degraded in five controlled ways: Gaussian blur, horizontal motion blur, downscaling, a white glare patch over the number, and cropping the frame into the number. These are synthetic changes to real photos. Each level is scaled to the height of the number's line.
+
+Each degraded image is saved as a JPEG, and Vision reads that JPEG. Every check is computed on the same decoded JPEG. An earlier run computed the checks on the pixels before encoding. Moving to the decoded JPEG changed no read result and moved the focus threshold from 6.63 to 6.68, 0.8%.
+
+Each check is measured three ways. The whole photo needs no locator. The top candidate of the number-finding ranking is what the phone can compute. The true number's box, from the undegraded read, is unknown to the phone, so those rows only show what a perfect locator would allow.
+
+For each photo and degradation, the break is the first level from which the photo never reads again. A check's retake threshold is the value at or below which 95% of photos have broken. Every threshold meets the pass criterion by construction, so two tests decide which checks work: the AUC over all degraded reads, and how many of the 75 real photos that read correctly each threshold would reject. `results/sweep.md` has the curves, thresholds and rejection counts.
+
+Only two checks work on the phone. Glare and framing checks work on the true box but not on the top candidate, because a washed-out or cut number usually stops being the top candidate: it was the number on 1,366 of 3,309 degraded reads. "Rejects 0 of 75" is weak evidence, because no real photo is near a threshold. The smallest label is 22 px, and the lowest whole-photo sharpness is 58.7. Even above every break, 0.7–2.6% of degraded reads failed anyway.
+
+Each photo's rows are written to one file under `~/house-scanning-data/meter/sweep/rows/` by an atomic rename, tagged with the digest of the label they were scored against and a digest of the measurement inputs: the photo's bytes, its number box and the degradation levels. A changed box changes every degraded image even when the same levels run, so it forces a new sweep. `degrade.expected_levels` gives the exact levels each photo takes, from its size and number box; a photo skips only the downscaling and edge levels it cannot reach. Rows also carry a digest of the measurement code: the Swift reader and the degradation, check, matching and ranking modules. `make sweep` skips a photo only when its file holds exactly those levels under the current label, inputs and code. If a shard fails, `make sweep` fails too. `make q2` refuses to run, naming the photos, when any file falls short.
+
+## Second pass (question 3)
+
+Every failed degraded read was retried on a crop around the top candidate, and on that crop upscaled 2x. Either retry recovered 2–5% of failures, and 10% for numbers made too small. A retake beats a second pass.
+
+## Barcodes and finding the number (questions 4 and 5)
+
+`meterocr` also runs `VNDetectBarcodesRequest` (revision 4). Candidates are the digit-bearing tokens of every recognized line. A candidate is right when it equals the labelled number after dropping separators and any leading letters. `locate.py` scores candidates by barcode confirmation, a `No.`, `Nr.` or `#:` label, standing alone on the line, and length, and penalizes specification lines, rotated text and runs of zeros.
+
+The rules and weights were written on the odd-numbered photos only and committed before the even-numbered photos were scored (`results/locate_dev.md`, then `results/locate.md`).
+
+The held-out set is split by physical meter, not by photo number, so no meter the rules saw is scored as held out. A meter is identified by its labelled number's keyed digest. A meter with any odd-numbered photo goes wholly to the design side (`locate.meter_splits`). One meter had three photos across both halves; moving its even-numbered photo to the design side changed barcode precision from 6/8 to 5/7.
+
+The held-out headline counts every agreed photo: the top three held the number on 27 of 34 (79%), and the first candidate on 20 of 34 (59%). Two of the 34 are photos Vision did not read at all, which cannot offer the number, so they count as misses. Over only the 32 photos Vision read, the rates are 27/32 (84%) and 20/32 (62%); `results/locate.md` reports both. The earlier write-up reported 28/33 (85%), which both excluded the unread photos and let one meter sit in both halves. Both criteria fail under every count. Two photos have no labelled number, so their meter is unknown; neither is held out. A cross-check against every number either reader transcribed found one more shared string, a model code printed on two different meters whose labelled numbers differ, so those stay separate.
+
+The number-finding misses split evenly:
+
+- **Several real identifiers on one plate.** The rule picked a maker serial, a second barcode or a `SERIAL#` line. Base needs to say which identifier it uses.
+- **Vision split the number.** A prefix or digits landed in a separate observation, so the exact number never became a candidate.
+
+The US-only barcode result (2 of 2 held out) was chosen after scoring, so it is exploratory.
+
+## Limits
+
+- Commons photos are mostly deliberate, well-lit close-ups, so homeowners' photos will read less often than 97%.
+- The degradations cover one glare model and horizontal motion only, with no low light, noise or perspective. Class labels were not swept.
+- Sharpness depends on the camera's processing. [FIELD_TEST.md](FIELD_TEST.md) checks both thresholds on an iPhone.
+- macOS Vision stands in for iOS Vision, with the same API and revision, and is not verified on a phone.
+- The rules were written after the author had read every photo, so the held-out half limits fitting but does not remove it. Its 33 photos give wide intervals.
+- This branch's history was rewritten on 2026-09-26 to remove example meter numbers and unkeyed digests. Older commits may stay reachable on GitHub by their IDs.
+
+## Reproduce
+
+These commands need macOS with Xcode 26, uv and the HMAC key. Run them from this folder:
+
+```sh
+make images   # download the photos (about 200 MB)
+make q1       # results/clean.md, about a minute
+make sweep    # the degraded reads, two processes of up to 2 GB, about 25 minutes
+make q2       # results/sweep.md
+make q45      # results/locate.md
+make check    # lint and unit tests
+make leakcheck
+```

experiments/meter-closeup/Makefile

@@ -28,9 +28,11 @@ q1: $(OCR)
 # The degraded reads behind results/sweep.md, in two processes of up to 2 GB each
 # (about 25 minutes). Needs results/clean_per_image.csv from q1.
 sweep: $(OCR)
-	uv run python -m meter_eval.sweep --shard 0 --shards 2 & \
-	uv run python -m meter_eval.sweep --shard 1 --shards 2 & \
-	wait
+	uv run python -m meter_eval.sweep --shard 0 --shards 2 & first=$$!; \
+	uv run python -m meter_eval.sweep --shard 1 --shards 2 & second=$$!; \
+	status=0; wait $$first || status=$$?; wait $$second || status=$$?; \
+	if [ $$status -ne 0 ]; then echo "a sweep shard failed (exit $$status); rerun make sweep" >&2; fi; \
+	exit $$status
 
 # results/sweep.md and results/sweep_rows.csv from the sweep rows.
 q2: