You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
evals: fit-cache key settings, coverage.json written before its check, untested evaluation helpers #173
Follow-up from the final Opus review of PR #12 at 4f52036 (findings 2, 3 and 6). None of them changes a committed number: the reviewer regenerated six results byte for byte, and recomputed four triangulation fits that match the cache.
1. The fit cache key leaves out the triangulation settings.fit_inputs in experiments/evals/evals/pose_priors.py (lines 131 to 143 at 4f52036) hashes the poses, intrinsics, pixel threshold and image and depth bytes. It leaves out triangulate.RATIO, MAX_FEATURES, min_points and min_angle_deg (evals/triangulate.py lines 22 to 23 and 170 to 172), so changing any of them silently reuses old fits.
Fix: add those four values to the spec dict, and add a test that changing each one alone gives a new key.
2. coverage.json is written before its check refuses.main in experiments/evals/evals/coverage.py (lines 868 to 874) writes results/coverage.json before if bad: raise. A failed run therefore overwrites the tracked JSON, leaves coverage.md stale, and feeds map3d.py (line 384), which reads that JSON. The module docstring (line 15) says main refuses to report.
Fix: check the replica mismatches before writing either file, and write both only on success. Test it with a mocked mismatch.
3. Evaluation helpers with no tests. None of these, and no caller of them, is exercised by a test in tests/ or models/tests/:
eth3d.visible_scan_points, which decides the ETH3D evaluation points;
coverage.evaluate_band, the coverage false-observed result;
coverage.level_world, the vertical cohort's up direction;
map3d.seen_empty and map3d.space_claims, the Map3D facing verdict;
drift.reference_scale_check, the GPS rescale.
Each has one right answer on a small synthetic scene, so each needs a strict hand-computed test.
Activity
evals: correct three statements the final review found