fix(datasets): anchored grade extraction + surface judge failures in generic llmjudge

[YuhaoLin2005]

Motivation

_generic_llmjudge_postprocess scans the judge's free-text reply for the first A/B anywhere in it. Judges routinely justify their verdict before stating it, so a reasoning sentence that merely contains a letter silently overrides the real grade ("As a judge, I considered the evidence carefully. The final grade is B." parses as A). Separately, when a judge fails to emit a usable grade (API error, truncation, no letter in the reply), the result degrades to 'unknown', which get_final_results folds into the accuracy denominator — indistinguishable from a genuine wrong answer. Related upstream reports: #1232, #2522, #2392.

The fix (two parts)

1. Anchored grade extraction (no regression)

Only a grade that is explicitly anchored is trusted: a cue word (grade, verdict, final answer) immediately followed by a connector (is / was / be / of / : / = / :=) and the letter. Everything else keeps the existing loose scan, so every judge reply that previously parsed still parses (zero recall regression, no silently-dropped samples).

Deliberate exclusions, each with a regression test:

• bare answer is not a cue — "the answer is A" names the option, not the grade;
• a connector is required — "grade A of this study" is a quality qualifier, not an assignment.

2. Judge failures are surfaced, not absorbed

get_final_results now counts judge failures separately (judge_error_count, plus a judge_error flag on each failing detail), while accuracy, accuracy_given_attempted, not_attempted_count, and friends keep their exact historical semantics (unknown still counts as not-attempted). A judge failure can no longer be silently read as a wrong answer.

Tests

tests/datasets/test_generic_llmjudge_postprocess.py — 13 cases covering anchored extraction, the anti-false-anchor contract, and the aggregation surface. Follows tests/TESTING_GUIDE.md conventions.

Impact on existing results

This fix changes how judge replies that contain an anchored grade are parsed:
previously such replies could be mis-parsed as the first letter appearing in the
judge's prose (often wrong); now the anchored grade wins. Replies with a single
letter, or with no conflicting prose letter, are parsed exactly as before, so
only previously-wrong results change — in the correct direction. accuracy,
accuracy_given_attempted, and not_attempted_count keep their exact
historical semantics; judge_error_count is a new, purely additive field.

Out of scope

• Connector-less verb forms ("I grade A") are deliberately not covered and
  continue to follow the legacy scan — adding them would reintroduce the
  qualifier false-positive the connector requirement exists to prevent.
• The same loose-scan logic is copy-pasted into MedXpertQA.py,
  medmcqa.py, supergpqa/supergpqa.py, and atlas/evaluation.py (each with
  its own _generic_llmjudge_postprocess / get_final_results). This PR keeps
  the diff focused on the canonical generic.py; a follow-up could apply the
  same anchored-extraction pattern to the copies.

[YuhaoLin2005]

Anchored grade extraction: 8 defects found and fixed before review

I ran an adversarial review panel over the anchored-grade regex in this branch and reproduced every finding against real CPython re, not by inspection. The idea is sound; the implementation had defects that silently changed reported scores. This commit repairs all of them.

Confirmed regressions (each reproduced, with the wrong value it produced)

Input	This branch returned	Truth / upstream
Grade: ambiguous, but leaning strongly B.	A — scored correct	B
Verdict: acceptable only after correction, ultimately B	A	B
The final answer was A, but it is wrong. Grade: B.	A	B
Verdict: a clear miss on the key claim.	A — a judge failure scored correct	unknown
Grade: A. However the justification is fatally wrong. Verdict: B.	A	B
GRADE: C	A — the scan read the A inside the word GRADE	unknown
Grade: C (with true_tag='A+')	A — the correct sample was scored not attempted	A+

Root causes, in order of severity:

1. re.IGNORECASE applied to the whole pattern, including the ([AB]) capture class. The indefinite article in "Grade: ambiguous…" was captured as the grade. The pattern comment claimed "no judge reply that used to parse now becomes an error"; the opposite was true — replies that safely parsed as unknown became confident, wrong grades.
2. final answer was a grade cue. In a judge reply it names the candidate's answer, so the branch's most common phrasing recorded wrong answers as correct.
3. .search() returns the first anchor, so a draft grade or a grade quoted from the student beat the judge's actual final verdict.
4. No word boundary after the letter, so the first letter of the next word became the grade: "Verdict: Based on…" → B.
5. An unterminated verdict fell back to the loose scan and invented a grade.
6. The anchor hardcoded [AB], ignoring the caller's true_tag/false_tag even though both entry points forward them.
7. detail['judge_error'] was set only on failure rows, giving a ragged details table.
8. The comments claimed judge failures are "never silently absorbed into accuracy." accuracy = is_correct_count / count and count includes them — a 3/10 judge outage still costs 30 accuracy points.

What changed

• The anchor is built from the caller's tags (_anchored_grade_re(true_tag, false_tag)), with the alternation sorted longest-first so a prefix tag cannot shadow a longer one.
• The cue sits in a scoped (?i:…) island, so the letter is matched case-sensitively, with a (?![A-Za-z0-9]) right boundary.
• The last anchor wins — a judge that revises itself is read correctly.
• The fallback (_bare_grade_re) requires a standalone token, which is what removes "Verdict: Based on" → B and "GRADE: C" → A while still finding a real trailing verdict such as the B in "leaning strongly B.".
• detail['judge_error'] is present on every row. Note generic_llmjudge_postprocess overwrites results['details'] with the raw judge output, so the flag is visible to direct get_final_results callers, not through that entry point — the comment now says so.
• Comments state the real denominator behaviour: accuracy still counts judge failures; <metric>_given_attempted is the figure that excludes them.
• get_final_results folds the elif into else, so any grade that is neither tag counts as a judge error rather than only the literal 'unknown'.

Also fixed: a quadratic backtracking path

\s*[(\[]?\s* let two independent \s* split the same whitespace run, giving O(n²) — "grade:" + 32000 spaces took 3.96 s where upstream took 0.08 ms. Now \s*(?:[(\[]\s*)?, which is mutually exclusive: same 64 000-space input is 3.9 ms (~1000× faster, linear).

Verification

• flake8 5.0.4, yapf 0.32.0, isort 5.11.5, codespell 2.2.1 — all the versions pinned in .pre-commit-config.yaml — clean. (lint was failing on this branch: yapf wanted the re.IGNORECASE argument joined to the previous line.)
• 32 tests, all passing.
• Mutation-tested: reverting each of the 9 repairs individually makes the suite fail, so no repair can silently regress. The previous 13-test version had 8 of 13 still green with the entire feature deleted; this suite has 4 of 32.
• Differential check against the upstream parser over ~6 000 inputs: divergence falls into exactly two families, both intended — a tag embedded inside a longer word is no longer accepted, and the last anchored verdict wins. Zero inputs where the new parser scores worse than upstream.

One design note for reviewers

accuracy still divides by the full sample count, so judge failures depress it; judge_error_count is currently always equal to not_attempted_count. If the maintainers would rather the headline metric exclude judge failures outright, that is a separate decision from this PR and I would rather not make it unilaterally — flagging it rather than bundling it.