fix(evals): score only exact-run transcripts and preserve agent failures

[ScriptedAlchemy]

Summary

Fixes #4204.

The hermetic scorer uses only the Claude session identified by the current invocation and validates recorded session/project identity. Missing, ambiguous, foreign, unreadable, or malformed evidence is an error—not a fallback to an older transcript. Codex uses its actual MCP tool field without double-counting started/completed events.

The runner preserves setup, agent, and verification exits; skips downstream effects after prerequisite failures; refuses missing explicit projects; and emits structured error/status fields before returning a failing batch status.

Review follow-ups are included:

• Every corpus result carries a positive scenario_index, stable across repetitions. The scorecard pairs (scenario_index, rep) while validating the display ID/category against the indexed corpus entry. Duplicate labels no longer collide. Legacy unindexed results retain their unique-ID rules; mixed or invalid identity schemes are refused.
• Empty or whitespace-only prompts produce a scenario/prompt_invalid row without running setup, the agent, or verification. They cannot disappear from a failing batch's report.

Verified CI

Initial exact-run verification executed and passed score_test, run_test, and scorecard_test through Bazel.

Reporting follow-up verification applied the reviewed identity/prompt changes, executed the same three Bazel targets with the additional regressions, checked shell syntax and whitespace, and committed the tested source through a separate artifact-publishing job. No paid model evaluations ran.

The temporary owner-scoped repair/publishing steps have been removed. The retained .github/workflows/hermetic-evals.yml is a small read-only workflow running those three targets for harness changes, including drafts. Full ready-PR CI remains separate and must satisfy repository merge requirements.

Regression coverage includes exact-session success; missing, foreign, ambiguous and malformed evidence; nonzero agent exits despite success-looking output; setup failures; real shell corpus orchestration; missing projects; duplicate display IDs; invalid prompts; scorecard identity tampering; and Codex event handling.

Related #4216 (grading quality), #4205 (memory-eval daemon lifecycle), and #4219 (product-wide outcomes) remain separate.

[changeset-bot]

⚠️ No Changeset found

Latest commit: b553ad0

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[ScriptedAlchemy]

Rewrote the complete PR history with the owner's authorization: 6 commits are now one, b553ad0e9b410faafdd662e81bc81af22d4a4f32 (fix(evals): enforce exact-run evidence and failure reporting).

The new commit reuses the exact prior root tree 8b2ae7b48cfa0bd4f60c55a09c75794bb9d9ec55 and original parent 654b8e65df2994c19c0109c3e17f93f6a3909f39. All implementation, tests, documentation, and read-only harness workflow bytes are unchanged. The branch update used an expected-head lease, and the existing lint policy is unchanged.

Read back the PR commit list to verify it contains only the new, conventionally named commit. Existing focused evaluation evidence is still relevant to the identical files, but the current-head CI results must be checked separately before merge.