Skip to content

docs(rfc): built-in eval tool across harnesses and tiers - #37669

Merged
iamwhatever merged 1 commit into
mainfrom
docs/rfc-long-task-eval
Nov 16, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
docs/rfc-long-task-eval

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Record, as a draft RFC, a built-in eval tool that runs every week across five harnesses and three task tiers, with a Crew headless CLI and a steady way to turn merged PRs into standing cases, where any ask of a person fails the run.

Today nothing in the repository runs a task across harnesses: evals/ holds one skill A/B, src/kiro_crew/eval/ runs short Crew-only scenarios, and kirocrew chat -m answers one turn and waits on the terminal for tool prompts. The repository owner wants to compare Crew chat, Crew conductor mode, kiro-cli V3, Claude Code and Codex fully unattended; judge prompt and mechanism changes before and after; publish cases others can check; compare the code when several harnesses pass; add docs tasks later; and run it weekly over simple, medium and complex tasks. Running such tasks today needs a person to grant Trust after every gateway restart, and the out-of-tree bench ran Crew arms with the operator's real HOME. (Scope widened by the owner on 2026-11-15 from long tasks only.)

Why it matters

Without a written design, each eval effort picks its own isolation, failure rule, runner and run count, and results are not comparable or safe: a run with the real HOME exposes ~/.aws and ~/.ssh to shells under the default sandbox tier, and an eval that answers asks for the agent hides exactly what the owner wants measured. A Crew arm driven through the gateway API while Claude Code and Codex run through their CLIs is not a like-for-like comparison. And without a harvest step the case set stays small and stale.

Not a goal

  • Building any of it: no runner, CLI flag, adapter, case or grading script lands here.
  • Changing any default of the real home (sandbox tier, Trust, approvals) or of kirocrew chat without --headless.
  • Publishing cases; this only fixes the format, location, harvest and publishing rules.
  • Deciding feat(prompts): one reroute-first ask policy for every crew kind #37609; this RFC depends on it for the conductor's plan gate.

What changed (motivation → approach → change)

  • Goal → a design document before any code, because it adds new first-class surfaces (kirocrew eval run | report | harvest | admit, headless flags on kirocrew chat, an eval suite under evals/tasks/), fixes what an unattended run may be granted in an eval home (a trust boundary), and publishes cases in a public repository.
  • Approach: reuse what exists (the kirocrew eval runner core with pass^k, --baseline / --max-drop, --junit and its artifact writer; the skill evals' --check / --run split; kirocrew chat -m; kirocrew cron --approval-mode and --command) and add only what is missing. The sections:
    • §3 Three tiers with their own timeout, spend cap and k (candidates from the out-of-tree tasks-v2, tasks and tasks-real sets); five arm adapters; ONE runner for every suite and tier; the kirocrew eval CLI entry and its exit codes.
    • §4 Isolated eval home: unchanged from the previous head.
    • §5 Failure rules per tier (ask, wall, spend, hidden check, VOID); the conductor's plan gate counts as an ask, seeds open with [sent by session.
    • §6 Before/after: one change, interleaved, 5 runs per case per side from the measured log-wall spread; agent.crew_mode_agent pinned.
    • §7 Case layout under evals/tasks/cases/<tier>/; §7.3 and §7.4 unchanged.
    • §10 Weekly cron (two zero-token --command jobs), report crossed by harness and tier (pass^k, asks, wall, cost, idle after done, code scores, harvest counts), trend file, alerts.
    • §12 Crew headless CLI: extend kirocrew chat (kirocrew run is the existing TASK.md runner) with --headless, --on-ask, --mode chat|conductor (agent resolved by crew_mode.crew_mode_agent(), never hard-coded), --approval-mode auto, --until-done, --cwd, --timeout, --budget, --resume, --json (asks, wall, credits, completed, transcript, resolved agent and its source) and exit codes; both Crew arms use it.
    • §13 Case collection: merged PRs grouped by tracking issue, Part of #N or RFC implementation-prs; seed from the pre-work issue or RFC text (never PR diffs or bodies); base from the first PR; hidden checks from the PRs' own tests plus behaviour checks; a stand-in timed from their real CI and review; admission gates; training-data risk and its mitigations; draft → held-out → public → retired lifecycle with a per-tier mix; ledger or chat mining optional and off by default.
  • Change: docs/request-for-change/rfc-long-task-eval.md renamed to rfc-builtin-eval.md (status draft, new title) and its index row in docs/request-for-change/README.md updated.

Backwards compatibility

Compatible: docs only; no code, config or default changes.

Removes nothing: the diff adds one document (renamed within this PR) and one index row; no other file links the old name.

Tests

Proof: not provable -- docs or config only

python3 scripts/docs_lint.py passes locally on this head.

Manual verification

Every code reference in the RFC was read at 4bc26f6ecb (crew_mode.crew_mode_agent, DEFAULT_CREW_MODE_AGENT, set_unusable, stamp_for_switch, runs_crew_mode, register_chat_parser, register_run_parser, cron add --approval-mode, merge_repeats, write_eval_artifacts, ConductorRecord, WorkItem). CLI flags were read from the installed kiro-cli 3.3.0, Claude Code 2.1.325 and codex-cli 0.169.0 --help output. No harness was run.

Related Issues

Depends on #37609 (a turn nobody typed skips the conductor's plan gate).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 15, 2026 19:13
@iamwhatever
iamwhatever force-pushed the docs/rfc-long-task-eval branch from fb8f695 to 8b6ac3b Compare November 15, 2026 19:13
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ea10aac61ffc661e395419f27ceb6e3608ee3771 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This is a draft RFC and changes no code. The new document and its index row take nothing away and add no user-visible surface, so no check applies.

[DESIGN-REVIEWED] ea10aac

Description read

[DESCRIPTION-READ] 5c27e41ad24c410e419fcb96459f625f64b563b78d76992c2eac65ad62f4ac31

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ea10aac61ffc661e395419f27ceb6e3608ee3771 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ea10aac

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ea10aac61ffc661e395419f27ceb6e3608ee3771 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision ea10aac61ffc661e395419f27ceb6e3608ee3771 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ea10aac61ffc661e395419f27ceb6e3608ee3771 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ea10aac61ffc661e395419f27ceb6e3608ee3771 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ea10aac

Verdict parsed from the review's SHA-scoped output markers for commit ea10aac61ffc661e395419f27ceb6e3608ee3771.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ea10aac61ffc661e395419f27ceb6e3608ee3771 false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the docs/rfc-long-task-eval branch from 8b6ac3b to b5794f2 Compare November 15, 2026 19:32
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Addressed the Opus 5.5 note on 8b6ac3b in b5794f2: it was right that sandbox_tiers.py builds its masks from Path.home(), so swapping HOME leaves the operator's real ~/.aws / ~/.ssh reachable by absolute path. §4.1 now requires every process of a run to run as a dedicated OS user who cannot read the operator's home (fallback: a mount namespace hiding it; the runner refuses when neither holds), §4.4 point 1 no longer relies on the strict tier for that, and §11 lists the eval user as not yet verified. Also fixed a stale §10 -> §11 cross-reference.

@iamwhatever
iamwhatever force-pushed the docs/rfc-long-task-eval branch from b5794f2 to 9cbc7f7 Compare November 15, 2026 20:21
@iamwhatever iamwhatever changed the title docs(rfc): built-in long-task evaluation across harnesses docs(rfc): built-in eval tool across harnesses and tiers Nov 15, 2026
@iamwhatever
iamwhatever force-pushed the docs/rfc-long-task-eval branch from 9cbc7f7 to 372d4c0 Compare November 15, 2026 21:19
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Opus 5.5 notes on 9cbc7f7, answered in 372d4c0:

  • §6 spread wording: fixed ("the widest of the non-wait tasks").
  • §10 harvest network: fixed. Harvest now runs in its own harvest home (read-only public GitHub data, no model credentials, drafts to a private dir); the eval home keeps its loopback + model-endpoint network rule, and a draft reaches it only through kirocrew eval admit.
  • creator fence: right, the fence is cron_creator_admission and the mode rule is cron_mode_refusal (both in dashboard/handlers/_shared_route_guards.py). §12.3 now names both. §4.2 is kept word for word at the repository owner's instruction for the safety sections, so its one-name citation is left as is there; the design it describes is unchanged.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 713 lines, more than the 400 it may approve on its own.

This is a docs-only PR. It adds a draft RFC for a built-in eval tool and one index row in docs/request-for-change/README.md. No code, config or defaults change. Nothing else in repo/ links the old rfc-long-task-eval name, so the rename breaks no links.

I checked the RFC's trust-boundary claims against the code, and they match:

  • Cron trust allowlist: cron_mode_refusal in _shared_route_guards.py allows a cron to set only trust or trust_reads, and only on a slot it names. Every refusal is audited.
  • Creator fence: cron_creator_admission limits a cron to slots it created itself.
  • Cron trust path: ScriptContext.set_session_mode in cron_script_context.py is the cron's route to set Trust without a person.
  • Worker inheritance: in session_create.py, a worker inherits _trust and _trust_reads from its parent. Per-command grants are not inherited.
  • Conductor agent: crew_mode_agent, set_unusable, stamp_for_switch and runs_crew_mode all exist in crew_mode.py.

The proposed eval home closes the boundaries it touches:

  • It runs as a separate OS user.
  • It uses the strict sandbox tier and never runs unsandboxed.
  • Trust is limited to one

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 372d4c0e84 · 713 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This adds a new concept, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here.

The review found no blocking issues on this head. The diff is docs only: one new draft RFC and one index row in docs/request-for-change/README.md. Every relative link resolves at the head. The code the RFC cites exists, including cron_mode_refusal, cron_creator_admission, crew_mode_agent, merge_repeats, write_eval_artifacts, the cron add --approval-mode choices and the sandbox tier docstring. None of the five bundled scenarios uses a proposed reserved word. PR Readiness is green, and every review lane is PASS or skipped on this commit.

One thing done well: section 4.1 explains, with the code cited, why swapping HOME alone is not a fence. It requires a separate OS user instead, and each arm gets only one slot's Trust through the audited cron route, never yolo.

Optional:

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 16, 2026
Draft RFC (rfc-builtin-eval.md): kirocrew eval run/report/harvest/admit on the existing runner core; simple/medium/complex tiers; five arms where any ask of a person fails the run; a Crew headless CLI on kirocrew chat used by both Crew arms, with conductor mode resolved through agent.crew_mode_agent; the isolated eval home; single-change before/after; standing cases harvested from merged PRs with admission gates; three-layer code grading; docs cases; a weekly cron, report, trend and alerts. Docs only.
@iamwhatever
iamwhatever force-pushed the docs/rfc-long-task-eval branch from 372d4c0 to ea10aac Compare November 16, 2026 01:38
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 16, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

RFC about eval mechanism. Since it does not influence the actual production usage, merge it first to enable implementation. The actual eval can be updated based on the team discussion later.

@iamwhatever
iamwhatever merged commit cb2f78b into main Nov 16, 2026
41 of 42 checks passed
@iamwhatever
iamwhatever deleted the docs/rfc-long-task-eval branch November 16, 2026 01:41
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants