[aroff]

Why

The trials=1 preflight (run 33914096408) took 76 minutes wall clock. At trials=5 that projects to ~6.3 h sequentially — past GitHub's hard 6-hour per-job ceiling, which is a platform limit timeout-minutes cannot raise. fastskill eval run 0.9.212 exposes no concurrency flag (only --trials), so trials run one after another.

The three suites share nothing until the scorecard, so they can run at the same time. Longest job becomes consultation at ~4.4 h, comfortably under the ceiling.

What changed

evals/v2/run.sh takes optional suite names: run.sh <agent> <out-dir> [trials] [suite...]. Names are validated before mkdir and before the vacuity guard, so a typo costs nothing and leaves nothing behind.

A subset run deliberately does not score itself. Every metric in metrics.toml is scoped to a case prefix, and a metric matching no case is a hard error (EVAL_SCORECARD_EMPTY_METRIC) — so scoring restraint alone would fail on the two judge_score metrics rather than tell you anything. A subset run says so and points at the fold.

evals/v2/scorecard.sh (new) is that fold, extracted so it can run against a directory it did not produce — in CI, the reassembled artifacts of three jobs. It refuses a root missing a suite and names the missing suite, instead of letting the scorecard blame a metric for a gap in the inputs.

.github/workflows/skill-evals.yml: live-eval becomes a fail-fast: false matrix over the three suites with per-suite timeout-minutes (330 / 180 / 90, sized from the measured trials=1 times) and per-suite artifacts. A new scorecard job (needs: live-eval, !cancelled()) downloads them with merge-multiple: true and applies the gates. That job needs no secrets and no Tailnet.

live-eval stays dispatch-only. No schedule:, no cron: — verified by parsing the workflow. A sweep that runs for hours with nobody waiting on it is a sweep nobody reads.

Docs

Corrected two stale claims while in there:

• The sweep was documented as "~$6, ~1 h". That was a token-cost estimate at Anthropic rates, which is fiction against the self-hosted gateway this actually runs on; and the hour was wrong by 6x. Replaced with the measured 76 min at trials=1 and the units that actually meter (agent runs, judge calls).
• evals/README.md still said "No CI job runs a v2 sweep" and described live-eval as v1-only. Both stopped being true when the sweep landed.

Verification

• bash -n and shellcheck clean on both scripts.
• run.sh exercised with a stubbed fastskill on PATH across all four argument shapes (no suite / trials only / one suite / two suites) plus a bad suite name → exit 2 with no directory created. restraint alone correctly does not demand AIKIT_LLM_URL / JUDGE_API_KEY.
• scorecard.sh against a partial root → exit 1 naming restraint; complete root → exit 0; nonexistent root → exit 2.
• guard_vacuity.py passes; build.py produces no diff, so no suite content moved.
• Workflow parses; jobs are validate, validate-v2, score-fixtures, live-eval, scorecard; triggers are push, pull_request, merge_group, workflow_dispatch — no schedule.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DobwJ8YTnfowwqwhspU1kQ