[Gaudy Blanco (gaudyb)]

Summary

• Add calibrated absolute answer scoring with a reusable, frozen five-level quality scale.
• Introduce absolute-calibrate to rank overlapping answer batches, calculate Elo ratings, map results to levels 1–5, and select percentile exemplars for each criterion.
• Introduce absolute-score to classify unseen answers against frozen exemplars using either one pass or three-pass voting, with confidence, evidence, rationale, and insufficient-evidence reporting.
• Add concurrency-safe caching for calibration rankings and scoring judgments, including credential-safe configuration fingerprints and lease handling.
• Add configuration scaffolding, local/blob storage support, CSV/JSON outputs, model-usage reporting, and a baseline notebook comparing calibrated scoring with repeated direct judgments.

Reliability and experiment updates

• Bound calibration and scoring cache/API work to concurrent_requests to prevent SQLite connection storms and excessive provider concurrency.
• Preserve deterministic result ordering and cancel remaining workers when a request fails.
• Increase the default calibration coverage from three to five appearances.
• Generate calibrated calibration and scoring settings with concurrent_requests: 2.
• Recover provider responses that omit a rationale, bound pathological evidence/rationale lengths, and preserve strict validation for cached final scores.
• Add notebook-safe progress callbacks for pairwise and assertion scoring, including cache hits.
• Add the global retrieval-budget proxy experiment comparing b50 with b200 across calibrated absolute, pairwise, and assertion scoring.
• Document the proxy experiment and ignore its local notebook cache.

Validation

• uv run pytest tests/autoe/calibrated_scoring_test.py tests/autoe/assertion/cache_test.py tests/autoe/pairwise/cache_test.py tests/test_init_config_cli.py tests/test_load_config.py — 88 passed
• Ruff checks pass for all modified Python files.
• Pyright reports 0 errors for all modified Python and test files.
• All notebook code cells parse successfully, and the committed notebook contains no error outputs.

Notes

The retrieval-budget experiment treats b200 > b50 as weak directional ground truth for global questions, not as human-adjudicated ground truth.