Repository navigation
docs(rfc): built-in eval tool across harnesses and tiers - #37669
Conversation
fb8f695 to
8b6ac3b
Compare
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS This is a draft RFC and changes no code. The new document and its index row take nothing away and add no user-visible surface, so no check applies. [DESIGN-REVIEWED] ea10aac Description read[DESCRIPTION-READ] 5c27e41ad24c410e419fcb96459f625f64b563b78d76992c2eac65ad62f4ac31 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
8b6ac3b to
b5794f2
Compare
|
Addressed the Opus 5.5 note on 8b6ac3b in b5794f2: it was right that |
b5794f2 to
9cbc7f7
Compare
9cbc7f7 to
372d4c0
Compare
|
Opus 5.5 notes on 9cbc7f7, answered in 372d4c0:
|
Code change approver recommendation: A person should decideWhy: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 713 lines, more than the 400 it may approve on its own. This is a docs-only PR. It adds a draft RFC for a built-in eval tool and one index row in I checked the RFC's trust-boundary claims against the code, and they match:
The proposed eval home closes the boundaries it touches:
An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
This adds a new concept, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. The review found no blocking issues on this head. The diff is docs only: one new draft RFC and one index row in docs/request-for-change/README.md. Every relative link resolves at the head. The code the RFC cites exists, including One thing done well: section 4.1 explains, with the code cited, why swapping Optional:
|
Draft RFC (rfc-builtin-eval.md): kirocrew eval run/report/harvest/admit on the existing runner core; simple/medium/complex tiers; five arms where any ask of a person fails the run; a Crew headless CLI on kirocrew chat used by both Crew arms, with conductor mode resolved through agent.crew_mode_agent; the isolated eval home; single-change before/after; standing cases harvested from merged PRs with admission gates; three-layer code grading; docs cases; a weekly cron, report, trend and alerts. Docs only.
372d4c0 to
ea10aac
Compare
|
RFC about eval mechanism. Since it does not influence the actual production usage, merge it first to enable implementation. The actual eval can be updated based on the team discussion later. |
Problem / Motivation
Goal: Record, as a draft RFC, a built-in eval tool that runs every week across five harnesses and three task tiers, with a Crew headless CLI and a steady way to turn merged PRs into standing cases, where any ask of a person fails the run.
Today nothing in the repository runs a task across harnesses:
evals/holds one skill A/B,src/kiro_crew/eval/runs short Crew-only scenarios, andkirocrew chat -manswers one turn and waits on the terminal for tool prompts. The repository owner wants to compare Crew chat, Crew conductor mode, kiro-cli V3, Claude Code and Codex fully unattended; judge prompt and mechanism changes before and after; publish cases others can check; compare the code when several harnesses pass; add docs tasks later; and run it weekly over simple, medium and complex tasks. Running such tasks today needs a person to grant Trust after every gateway restart, and the out-of-tree bench ran Crew arms with the operator's real HOME. (Scope widened by the owner on 2026-11-15 from long tasks only.)Why it matters
Without a written design, each eval effort picks its own isolation, failure rule, runner and run count, and results are not comparable or safe: a run with the real HOME exposes
~/.awsand~/.sshto shells under the default sandbox tier, and an eval that answers asks for the agent hides exactly what the owner wants measured. A Crew arm driven through the gateway API while Claude Code and Codex run through their CLIs is not a like-for-like comparison. And without a harvest step the case set stays small and stale.Not a goal
kirocrew chatwithout--headless.What changed (motivation → approach → change)
kirocrew eval run | report | harvest | admit, headless flags onkirocrew chat, an eval suite underevals/tasks/), fixes what an unattended run may be granted in an eval home (a trust boundary), and publishes cases in a public repository.kirocrew evalrunner core with pass^k,--baseline/--max-drop,--junitand its artifact writer; the skill evals'--check/--runsplit;kirocrew chat -m;kirocrew cron --approval-modeand--command) and add only what is missing. The sections:tasks-v2,tasksandtasks-realsets); five arm adapters; ONE runner for every suite and tier; thekirocrew evalCLI entry and its exit codes.[sent by session.agent.crew_mode_agentpinned.evals/tasks/cases/<tier>/; §7.3 and §7.4 unchanged.--commandjobs), report crossed by harness and tier (pass^k, asks, wall, cost, idle after done, code scores, harvest counts), trend file, alerts.kirocrew chat(kirocrew runis the existingTASK.mdrunner) with--headless,--on-ask,--mode chat|conductor(agent resolved bycrew_mode.crew_mode_agent(), never hard-coded),--approval-mode auto,--until-done,--cwd,--timeout,--budget,--resume,--json(asks, wall, credits, completed, transcript, resolved agent and its source) and exit codes; both Crew arms use it.Part of #Nor RFCimplementation-prs; seed from the pre-work issue or RFC text (never PR diffs or bodies); base from the first PR; hidden checks from the PRs' own tests plus behaviour checks; a stand-in timed from their real CI and review; admission gates; training-data risk and its mitigations; draft → held-out → public → retired lifecycle with a per-tier mix; ledger or chat mining optional and off by default.docs/request-for-change/rfc-long-task-eval.mdrenamed torfc-builtin-eval.md(statusdraft, new title) and its index row indocs/request-for-change/README.mdupdated.Backwards compatibility
Compatible: docs only; no code, config or default changes.
Removes nothing: the diff adds one document (renamed within this PR) and one index row; no other file links the old name.
Tests
Proof: not provable -- docs or config only
python3 scripts/docs_lint.pypasses locally on this head.Manual verification
Every code reference in the RFC was read at
4bc26f6ecb(crew_mode.crew_mode_agent,DEFAULT_CREW_MODE_AGENT,set_unusable,stamp_for_switch,runs_crew_mode,register_chat_parser,register_run_parser,cron add --approval-mode,merge_repeats,write_eval_artifacts,ConductorRecord,WorkItem). CLI flags were read from the installedkiro-cli 3.3.0, Claude Code2.1.325andcodex-cli 0.169.0--helpoutput. No harness was run.Related Issues
Depends on #37609 (a turn nobody typed skips the conductor's plan gate).
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.