Skip to content

Add model comparison experiment and README (provisional, synthetic data) - #105

Open
AdityasagarR123 wants to merge 1 commit into
ruxailab:mainfrom
AdityasagarR123:feat/experiments-model-comparison
Open

AdityasagarR123 wants to merge 1 commit into
ruxailab:mainfrom
AdityasagarR123:feat/experiments-model-comparison

Conversation

@AdityasagarR123

@AdityasagarR123 AdityasagarR123 commented Oct 23, 2026 •

Copy link
Copy Markdown

Summary

Adds a benchmark in experiments/ that compares 10 regression models for gaze estimation

What's included

  • run_experiment.py: stratified 80/20 split and leave-one-point-out (LOPO) evaluation
  • synthetic_data.py: 9-point calibration data generator with configurable noise and head shift
  • sanity_check.py: diagnostics for data, learnability, units, leakage, fold integrity, and memorization
  • README.md: setup, commands, arguments, metric definitions, interpretation
  • results/: outputs and diagnostic reports from the synthetic benchmark

Verification

  • Noiseless LOPO error: 0.00 px (evaluation pipeline is correct)
  • Noiseless linear fit R² = 1.0

Status and limitations

  • Results use synthetic data. Model rankings are provisional and should not be used to select a production model until real calibration recordings are benchmarked.
  • Random Forest's low LOPO error is an artifact of the 3×3 grid (each axis is trained separately) and is excluded from the verdict.
  • No changes to app/ or tests/ in this PR.

Open items (not in this PR)

  • Real calibration dataset (gitignored; requested from maintainers)
  • Production centering in app/services/gaze_tracker.py, lines 580–587 (pending decision)
  • Broken tests in tests/test_gaze_tracker.py and tests/test_routes.py (pending decision)

Reviewer notes

Start with experiments/README.md, then run python experiments/run_experiment.py to reproduce the default results.

- Benchmark 10 regression models on stratified 80/20 and leave-one-point-out splits

- Add synthetic calibration data generator and sanity checks

- Document setup, CLI arguments, metrics, and interpretation

- Results are on synthetic data; rankings are provisional pending real calibration recordings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant