Repository navigation
feat(eval): kirocrew eval run for task cases across harnesses - #38187
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change is additive, Notes
[DESIGN-REVIEWED] b35defb Description read[DESCRIPTION-READ] 192e9a88376be5aadffc4e11b398b41f26bbbca9f7a423b2d530147c723d5862 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The base RFC Not checked: a real model on any arm (the author says so too), and the Crew arms on a host with a sandbox backend. Notes
What this change shipsInventory (10 items) — 9 justifiedIntent: run the same coding task unattended on Crew chat, Crew conductor mode and Claude Code, and compare scored results per harness and tier. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] b35defb Description read[DESCRIPTION-READ] 869e19e173ebd5aea9786f02ca89a687f2c614b85a9e3e913dd855e0497bd963 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ human override acceptedHuman judgment by @iamwhatever overrides the GPT 6.1 finding for [GPT-OVERRIDE] b35defb This comment is updated in place on each push. The model was not re-run because an authorized human decision supersedes it. False positive or not applicable? A repository writer can comment: |
91dc939 to
72b6708
Compare
|
span=5c53eeb474af
|
|
span=ba3dcad07ed8
|
|
span=5c5ae19bb800
|
|
span=57ad4ba643cc
|
|
span=82e7335dc4c7
|
|
span=f5494d6316d5
|
|
span=f33ebde45a53
|
|
span=7bf6e3122239
|
|
span=7bf6e3122239
|
72b6708 to
cc8f399
Compare
|
span=6ae74ebf2ab4
|
|
span=7bf6e3122239
|
|
span=7bf6e3122239
|
|
span=c5c426992518
|
cc8f399 to
5734a59
Compare
|
span=7bf6e3122239
|
|
span=c5c426992518
|
06523ea to
273426d
Compare
|
span=f5494d6316d5
|
|
span=7bf6e3122239
|
|
span=7bf6e3122239
|
273426d to
45bcebd
Compare
|
span=c5c426992518
|
45bcebd to
9f9ae5c
Compare
|
span=7bf6e3122239
|
kirocrew eval run --suite tasks runs real coding cases unattended on Crew chat, Crew conductor mode and Claude Code, and scores each run PASSED, FAILED or VOID by rfc-builtin-eval.md section 5: any ask of a person fails it, so do the wall and spend caps and the hidden check. Each run writes one JSON result (asks, wall, cost, idle after done, the check's parts, the resolved conductor agent); the batch writes one row per harness and tier through the existing eval artifact writer. Each run gets its own HOME, KIROCREW_HOME, KIRO_HOME and TMPDIR, an env with only the arm's eval sign-in, and the strict sandbox tier. Preflight refuses when the running user holds credential directories or can list another login user's home. A Crew arm's gateway starts in the run's data home and stops only after the final check, so a server the agent started is alive while it is judged. kiro-cli V3 and Codex are VOID until their adapters are built. Two cases from merged PRs ship under evals/tasks (one simple, one medium). --check proves each fails on its base and passes on its reference with no model. Task repos use a trunk branch, because Kiro Crew refuses any push to a branch named main. The RFC's new section 14.1 records where v1 does less than the design.
9f9ae5c to
b35defb
Compare
|
span=f5494d6316d5
|
|
span=7bf6e3122239
|
|
span=7bf6e3122239
|
|
span=7bf6e3122239
|
|
/ai-review override gpt b35defb not-applicable: GPT provider refused this diff for its content (security test fixtures); Opus, Design, First Principles, UX pass on this head. |
Human judgment recorded@iamwhatever recorded a not-applicable release (class
This decision applies only to this commit. A new push requires a new judgment. |
Problem / Motivation
Goal: Run the same real coding task unattended on Crew chat, Crew conductor mode and Claude Code, and get one scored result per run plus a per-harness, per-tier summary.
Today
kirocrew evalruns only short Crew-only scenarios, andkirocrew chat --headless(#37768) runs one task without scoring it. Nobody can compare the harnesses on the same task, or check a prompt change before and after, with numbers. This is the first code slice of the accepted RFCdocs/request-for-change/rfc-builtin-eval.md(#37669).Why it matters
Without a shared runner, every comparison is a one-off script against the operator's real home, with real credentials reachable. The RFC's owner asked for an unattended, isolated, repeatable measure where any ask of a person fails the run.
Not a goal
kirocrew eval report,harvest,admit, the weekly cron,--baseline/--junit/--detach/--budget-weekfor tasks.kirocrew chat --headlessitself.What changed (motivation → approach → change)
The RFC's single runner (section 3.3), cut to what one PR can prove.
kirocrew eval run --suite tasksbuilds each case's repo with a local origin. It runs the arm on the seed and samples the hidden check every 60 s. When the arm ends it checks once more, then scores the run PASSED, FAILED or VOID by section 5: any ask, the wall cap, the spend cap and the check each fail a run. Each run writesruns/<id>/result.json; the batch writes one row per harness and tier with the existingwrite_eval_artifacts.HOME,KIROCREW_HOME,KIRO_HOMEandTMPDIR, only the arm's eval key, and the strict sandbox. Preflight refuses (exit 3) when the user's home holds any credential path the sandbox hides (~/.aws,~/.ssh,~/.netrc,~/.git-credentialsand the rest ofsensitive_home_dirs), can list another person's home, has no sandbox backend for a Crew arm, or lacks an arm's key.kirocrew chat --headless --approval-mode trust(conductor:--mode conductor --until-done) on a loopback gateway the runner starts in the run's home. The gateway, and anything the arm left running, stops only after the final check, so a server the agent started is still up when it is judged. Hidden checks keep their own limits, so a submission cannot pass by raising a constant. The result records the resolved conductor agent.claude -p --permission-mode dontAsk --allowedTools <case tools> --strict-mcp-config --max-budget-usd.--checkproves with no model that each fails on its base and passes on its reference. Task repos use the branchtrunk: Crew refuses a push tomain, even to a local origin.--repeatnow defaults to unset: one run for scenarios as before, the tier's k foreval run.evals/tasks/README.mddocument the command.test/spawn_audit_helpers.pywith their reasons.run_checkis marked as deliberately not routed through the sandbox: the hidden check imports the agent's pushed code by design, the dedicated eval user contains it, and a sandboxed check is the follow-up recorded in RFC section 14.1.lib/file is VOID and the batch stops, so no run can rewrite the grader for later runs.src/kiro_crew/eval/tasks.py, the armssrc/kiro_crew/eval/arms.py(documented insrc/kiro_crew/eval/README.md). The flags are insrc/kiro_crew/cli_parser_admin.py.src/kiro_crew/cli.pydispatcheseval runto_run_eval_tasksinsrc/kiro_crew/cli_commands_eval.py, whichsrc/kiro_crew/cli_commands.pylists for type checkers.Backwards compatibility
Compatible:
kirocrew evalwithoutrunbehaves as before;--repeatunset still means one run (_run_evalreadsNoneas 1).runbecomes a reserved first name, as RFC section 3.4 says; none of the bundled scenarios uses it.Removes nothing: the change only adds a subcommand, flags, modules and case files.
Tests
test/test_eval_arms.py: each arm's exact argv (trust on its own chat, conductor mode without--agent,dontAskwith only the case tools and no MCP, no bypass flags); Crew--jsonand Claude stream-json parsing (asks, denials, credits or USD, budget stop, VOID statuses); the two unbuilt arms report "not available".test/test_eval_tasks.py: shipped cases load and pass--check; a case whose base already passes is named; badcase.jsonfields are refused. Preflight cases: credential dirs, a readable person's home (a macOS uid 501 included), a service account skipped, no sandbox backend, Windows, a missing key. A run's env keeps only its homes and its arm's key. Scoring follows section 5 in order. The summary counts pass^k per case only when every requested run passed (a VOID run leaves the case incomplete) and leaves VOID out of the medians. The audit matches the suite path only as a whole path component. End to end with a fakeclaudeon PATH: a pushed fix passes, and idle-after-done is sampled (the fake stays up until a spy on the check sees it pass, so no timing bet). An ask fails, no change fails the check, output naming the suite is VOID, and an arm past its cap is killed. A child the agent left running is gone after the run, an arm interrupted mid-run is stopped before the interrupt propagates, the kill deadline fires even while a check sample is running, a slow sample does not hold the arm, and a process the checked code starts dies with the check. A submission that raises its own length limit still fails S1. A Crew run is judged before its gateway stops. Also covered: the batch's default k, harness names, an eval home around or inside the suite (refused), and the CLI's exit codes 0/1/2/3.test/test_cli_eval_user_scenarios.py: the--repeatdefault is nowNone.test_eval_tasks.py+test_eval_arms.pypassed 20/20 at-n0, 10/10 at-n 2besidetest_cli_eval_user_scenarios.pyandtest_eval_baseline_gate.py, and in a shuffled order (seed 4242, 99 ids).-n 2, not-n 4: this host's shared-load rule caps workers at 2.Manual verification
kirocrew eval run --suite tasks --checkfrom the worktree:2/2 cases bite and are solvable.Refused:line per reason (credential dirs, readable homes, missing key).start_gatewayin a scratch run home:ss -ltnshowed it on127.0.0.1and[::1]only, andstop_processstopped it.Related Issues
Closes #38161
Part of #37669 (the RFC)
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)