Skip to content

Make quality diagnostics inspectable and add video integrity checks - #42

Merged
arpitg1304 merged 1 commit into
mainfrom
feat/proprio-quality-diagnostics
Oct 17, 2026
Merged

arpitg1304 merged 1 commit into
mainfrom
feat/proprio-quality-diagnostics

Conversation

@arpitg1304

@arpitg1304 arpitg1304 commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

Quality checks could hide missing observations, score stationary feedback from moving commands, and misread episode timing. This PR makes those cases explicit and adds evidence that users can inspect during playback.

  • Add held-command, motion-pause, direction-reversal, and opt-in tracking-stall events.
  • Add per-camera coverage, decode failures, timestamp-gap estimates, and localized video events.
  • Add forge quality --visualize and improve episode search, camera controls, playback, and action/state plots.
  • Preserve LeRobot metadata FPS and declared cameras. Raise on missing requested video frames instead of substituting images.
  • Update guides, examples, changelog, benchmark documentation. Add quality CI on Python 3.10–3.12.

Compatibility

  • Scorers are q-v2-proprio and video-v2-integrity. Legacy JSON remains readable.
  • Gripper channels and physical action limits require explicit configuration. Unsupported motion can return a null score.
  • video_min_sharpness now uses the minimum sampled value. --video-max-frames caps decode attempts per camera. Existing thresholds may need recalibration.
  • Scores and events are heuristics. Timestamp gaps estimate missing observations; they do not prove camera capture drops.

Validation

  • 264 quality, video, viewer backend, filter, catalog, and LeRobot compatibility tests passed.
  • 7 streaming tests and the Chrome integration test passed.
  • Ruff passed on the quality implementation, viewer, reader, filter engine, new tests, and the benchmark script. Whitespace checks passed.
  • Broader CLI suite: 16 passed; two existing Zarr fixture setup errors remain (create_dataset requires shape).

The CPU benchmark measures about 321 FPS for 720p decode plus pixel/motion checks at default analysis resolutions on an Apple M5 Pro. These measurements do not establish detection accuracy or training gains.

See validation and reproduction notes. Generated results and historical survey material are kept locally and excluded from this PR.

@arpitg1304
arpitg1304 force-pushed the feat/proprio-quality-diagnostics branch from 41bef1b to beb7ef3 Compare October 17, 2026 05:44
@arpitg1304
arpitg1304 merged commit 12a3167 into main Oct 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant