CI: benchmark PRs through lightcone-bench

[dkn16]

Adds three small workflows that hand a PR, or a merge to main, to lightcone-bench, where this repo's own eval (evals/prompt.md + evals/tasks/snae) runs through Harbor as the task lightcone-cli/snae-build. Nothing runs in this repo; each file is one API call.

What they do

• benchmark-on-pr.yml — every non-draft PR is benchmarked automatically against its head commit: on open, on each push, on reopen, and when a draft is marked ready. Fork PRs are skipped (they don't receive this repo's secrets).
• benchmark-on-comment.yml — a maintainer (OWNER / MEMBER / COLLABORATOR) comments /benchmark on a PR for the same run on demand: for a draft PR, or a re-run. Reacts 🚀 as the ack.
• benchmark-baseline.yml — every push to main dispatches the same run with save_baseline, so the bench's stored "current main" numbers track this repo.

All three pass use_skills, so the agent runs with the skills from agent-skills main installed. The skills axis is held fixed here; varying it is the agent-skills repo's own job.

What a run does

In lightcone-bench: the task's image is built with this repo pinned to the requested sha, the oracle gate runs the scripted reference solution and must score 1.0 (no LLM, so a PR that breaks the CLI contract fails here, visibly), then each agent leg runs the task and the report shows per-task rewards, the delta against the stored main baseline, and a trace analysis. Today the opencode leg runs; claude-code and codex join once their API keys are set in lightcone-bench (they skip with a notice until then). Results live in the dispatched run's job summary under lightcone-bench's Actions tab; posting them back onto the PR is a planned follow-up.

Setup (one secret)

BENCH_DISPATCH_TOKEN in this repo's Actions secrets: a fine-grained PAT (or App token) with repository access to LightconeResearch/lightcone-bench and Actions: read and write. The built-in GITHUB_TOKEN cannot dispatch into another repository.

Relation to eval.yml

eval.yml runs Claude Code on every PR inside this repo; this is the same task run by the bench, across agents, with an oracle gate and stored baselines. They don't interact. Whether the bench replaces eval.yml is a separate decision.

Testing

benchmark-on-pr.yml runs from the PR's own branch, so once the secret exists, this PR benchmarks itself on its next push. The /benchmark comment and the baseline trigger only take effect after merge: GitHub serves issue_comment and push workflows from main. The bench side was validated by dispatching it directly with this repo's current main sha, which is exactly what the senders send: oracle 1.0, opencode leg ran, report produced. The first runs will show NEW — no baseline for the skills variant until the first merge to main refreshes it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VcKmGNUek7oWEPv8zLqo9Q

[github-actions]

❌ Eval

Metric	Value
Outputs check	skipped
Agent run	failure
Turns	1
Tool calls	0
Cost	$0.00
Agent wall time	0m00s
Model	claude-sonnet-5-5

lc status

unavailable

Confusion & pain points (Claude analysis)

Trace analysis unavailable.

Full trace: agent-trace artifact on this run.

[github-actions]

lightcone-bench — benchmark of 837d607 ✅ passed: run 38194200456 — the per-agent tables, baseline deltas and trace report are in its job summary.

All agents pooled (reward averaged over scored trials, the rest over all trials):

trials	scored	mean reward	steps	tool calls	shell calls	out tokens (k)	cached %	cost $	agent min	total cost $
9	9	1.00	68	89	40	50.9	99	0.060	9.9	0.537