Skip to content

evals: fit-cache key settings, coverage.json written before its check, untested evaluation helpers #173

Open

Description

@SamGu-NRX

Follow-up from the final Opus review of PR #12 at 4f52036 (findings 2, 3 and 6). None of them changes a committed number: the reviewer regenerated six results byte for byte, and recomputed four triangulation fits that match the cache.

1. The fit cache key leaves out the triangulation settings. fit_inputs in experiments/evals/evals/pose_priors.py (lines 131 to 143 at 4f52036) hashes the poses, intrinsics, pixel threshold and image and depth bytes. It leaves out triangulate.RATIO, MAX_FEATURES, min_points and min_angle_deg (evals/triangulate.py lines 22 to 23 and 170 to 172), so changing any of them silently reuses old fits.

  • Fix: add those four values to the spec dict, and add a test that changing each one alone gives a new key.

2. coverage.json is written before its check refuses. main in experiments/evals/evals/coverage.py (lines 868 to 874) writes results/coverage.json before if bad: raise. A failed run therefore overwrites the tracked JSON, leaves coverage.md stale, and feeds map3d.py (line 384), which reads that JSON. The module docstring (line 15) says main refuses to report.

  • Fix: check the replica mismatches before writing either file, and write both only on success. Test it with a mocked mismatch.

3. Evaluation helpers with no tests. None of these, and no caller of them, is exercised by a test in tests/ or models/tests/:

  • eth3d.visible_scan_points, which decides the ETH3D evaluation points;
  • coverage.evaluate_band, the coverage false-observed result;
  • coverage.level_world, the vertical cohort's up direction;
  • map3d.seen_empty and map3d.space_claims, the Map3D facing verdict;
  • drift.reference_scale_check, the GPS rescale.

Each has one right answer on a small synthetic scene, so each needs a strict hand-computed test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions