test(metrics): pin the MMLU/GSM8K extractor invariants exhaustively (closes #32)

[dchaudhari7177]

Closes #32.

Ten invariant tests over extract_mmlu_answer / extract_gsm8k_answer, covering the documented input space rather than sampled points. Test-only; no production code touched.

Why not hypothesis

The issue offered either hypothesis or hand-rolled parametric cases. I went with parametric, for two reasons:

1. The MMLU space is small enough to enumerate exhaustively — 10 letters × 6 wrappings × 10 choice-counts — which is strictly stronger than randomly sampling it. Every valid (letter, wrapping, num_choices) combination is actually checked.
2. Adding hypothesis to the dev group requires relocking, and uv lock on this tree regenerates uv.lock into a 2,893-line diff that pulls torch, cuda-* and datasets into the lockfile (the committed lock is currently minimal). That is a large, risky change to carry on a test-only PR — happy to add it in a separate PR if you'd rather have it.

Where a value is genuinely unbounded (a GSM8K number), a seeded random.Random(20260812) supplies boundary values (0, 1, 999, 1000, 1_000_000) plus a spread, so runs are deterministic and failures reproducible — meeting the "no random seed drift" criterion.

Invariants

MMLU — every wrapping of a valid letter agrees; a letter beyond num_choices is never returned; the result is always None or a valid index; the last standalone letter wins; an explicit answer is/: beats a standalone letter even when the standalone one appears later (precedence is by kind, not position).

GSM8K — #### n always beats a trailing number; $, thousands commas and a trailing period normalise alike (checked both inline and behind the marker); the last number wins absent a marker; digit-free text is None; the result is always None or a float()-parseable canonical string.

The last MMLU and GSM8K invariants matter because score_mmlu indexes gold answers with the result and score_gsm8k calls float() on it — a bad value would surface as a silent mis-score or a crash in the scorer, not here.

Verified by mutation, not by passing

Tests that only ever pass prove little, so I confirmed each is load-bearing:

Mutation	Result
first standalone letter wins instead of last	test_mmlu_last_standalone_letter_wins fails
drop the num_choices validity filter	test_mmlu_result_is_always_none_or_a_valid_index fails
remove the #### marker preference	test_gsm8k_marker_always_beats_a_trailing_number fails

Two behaviours worth your call

Both are pinned as they are, not changed — flagging rather than deciding:

• The standalone pattern is case-sensitive, so a bare lowercase "a" extracts nothing, while "answer: a" does (that pattern is IGNORECASE). The docstring lists "lowercase" among tolerated formats without drawing this distinction.
• num_choices=0 clamps to 1 rather than rejecting, so extract_mmlu_answer("A", 0) returns 0. The range invariant is asserted over num_choices >= 1 accordingly.

Local results

pytest -q                 403 passed
ruff check .              All checks passed!
ruff format --check .     20 files already formatted

mypy src reports one error on this machine — numpy/__init__.pyi:737: Type statement is only supported in Python 3.12 and greater — which reproduces identically on a pristine tree (git stash), so it is a local venv/numpy-stub artefact rather than anything from this change. mypy src does not cover tests/ in any case.

🤖 Generated with Claude Code