[johnl-amd]

Fixes #25.

Summary

A behavioral run could report a skill as having violated an expectation that was never actually checked. A dropped connection produced this line, and the skill's author had nothing to debug:

[FAIL] (unexpected_behavior) Run a benchmark, install or download anything, or claim measured results
  -- llm_judge gave no JSON verdict: 'API Error: Connection dropped (ECONNRESET)'

Root cause. _grade_with_llm has four paths that cannot produce a verdict, and all four returned False, which is the same value the judge returns when it decides against the skill: no claude CLI on PATH, the command deadline already spent, the judge timing out, and no JSON verdict in the reply. The last one swallows every API error, because the error text arrives in the result field where a verdict was expected. With a return type of tuple[bool, str], "the judge said no" and "the judge could not answer" were indistinguishable to the caller.

The polarity is what makes this worse than a mislabelled failure. For unexpected_behavior, False asserts the prohibited behavior happened, on no evidence at all, which inverts the default the function's own docstring sets:

Absence of evidence that the agent did it counts as avoiding it, so the default verdict is true.

In the reported run every other expectation in that case passed, 5/5 expected, 3/3 files, 2/2 logs. A second skill had one graded expectation in its only case, so a judge outage took it to 0%.

What changed. Those four paths return None, and a Check records whether it was graded. An ungraded check is not a pass and not a failure. It still refuses to call the case green, so the run stays red and still exits 1, but it is reported as an error against the run, listed under its own heading rather than among unmet expectations, and left out of the rate the report publishes.

Non-obvious decisions

• This is the shape routing already uses. A routing case whose own run errored counts towards neither side of --min-accuracy, and docs/usage.md already documents that. Behavioral had no equivalent. The wording for the new carve-out deliberately mirrors the existing one.
• graded is a second field rather than passed: bool | None. The report is uploaded as a CI artifact, so "passed": null would be a shape change for anything reading it, and all(c.passed ...) keeps failing the case with no edit at the call sites.
• An evals/hooks.py assertion raises a new Ungradable rather than AssertionError. run_case converts AssertionError into a check failed by the skill, which is the attribution this change exists to prevent. Ungradable is caught separately and recorded as an ungraded check, so the hook path lands in the same place as the dataset path. Teardown still runs.
• The per-skill table's Expectations column is now Graded, so Met / Graded is a real rate and matches the headline's denominator. Ungraded has its own column.
• No retry. claude_env() disables the CLI's retry loop on purpose so an unreachable API fails fast instead of hanging, and reversing that is a separate argument from reporting the outcome honestly.

Scope

Not included: the same three sites in engine/judge.py and engine/scorers.py added by #19, which reproduce this (parse_verdict already returns None and grade collapses it back to False; the ListingFailed branch there carries a comment describing exactly this problem). That is a follow-up against #19's engine once it lands, so this PR does not conflict with it. The report layer this change touches is the half #19 preserves.

Test plan

Reproduced before fixing, with no tokens spent: stub shutil.which to None, and stub the CLI to return an ECONNRESET payload. Both printed the [FAIL] line above on main; both print [----] with graded=False after.

python -m unittest discover -s tests -t ., 242 passing. The nine new tests were confirmed to fail against unfixed source, and that run showed the ungraded check listed under Unmet expectations, which is the bug.

Mutation-tested rather than assumed. Each of these is caught by at least one test: every one of the four paths reverted to False, _verdict_label losing its third branch, the headline denominator reverting to all checks, the per-skill ungraded count forced to 0, and the hook path recording a graded failure instead of an ungraded check.

Risk: low. No behavior change when the judge answers, which the tests pin for both a true and a false verdict. Pipe escaping, truncation and row ordering in render_markdown are unchanged. check_api_reachable has a near-identical deadline guard that is deliberately untouched, since it is a preflight rather than a judge.