Skip to content

feat(eval): kirocrew eval run for task cases across harnesses - #38187

Merged
iamwhatever merged 1 commit into
mainfrom
feat/eval-run-v1
Nov 17, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
feat/eval-run-v1

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Run the same real coding task unattended on Crew chat, Crew conductor mode and Claude Code, and get one scored result per run plus a per-harness, per-tier summary.

Today kirocrew eval runs only short Crew-only scenarios, and kirocrew chat --headless (#37768) runs one task without scoring it. Nobody can compare the harnesses on the same task, or check a prompt change before and after, with numbers. This is the first code slice of the accepted RFC docs/request-for-change/rfc-builtin-eval.md (#37669).

Why it matters

Without a shared runner, every comparison is a one-off script against the operator's real home, with real credentials reachable. The RFC's owner asked for an unattended, isolated, repeatable measure where any ask of a person fails the run.

Not a goal

  • kirocrew eval report, harvest, admit, the weekly cron, --baseline / --junit / --detach / --budget-week for tasks.
  • The kiro-cli V3 and Codex adapters (registered; every run of them is VOID "not available").
  • Blinded code grading (RFC section 8), the PR/CI stand-in, docs cases.
  • Changing kirocrew chat --headless itself.

What changed (motivation → approach → change)

The RFC's single runner (section 3.3), cut to what one PR can prove.

  • kirocrew eval run --suite tasks builds each case's repo with a local origin. It runs the arm on the seed and samples the hidden check every 60 s. When the arm ends it checks once more, then scores the run PASSED, FAILED or VOID by section 5: any ask, the wall cap, the spend cap and the check each fail a run. Each run writes runs/<id>/result.json; the batch writes one row per harness and tier with the existing write_eval_artifacts.
  • Each run gets its own HOME, KIROCREW_HOME, KIRO_HOME and TMPDIR, only the arm's eval key, and the strict sandbox. Preflight refuses (exit 3) when the user's home holds any credential path the sandbox hides (~/.aws, ~/.ssh, ~/.netrc, ~/.git-credentials and the rest of sensitive_home_dirs), can list another person's home, has no sandbox backend for a Crew arm, or lacks an arm's key.
  • Crew arms run kirocrew chat --headless --approval-mode trust (conductor: --mode conductor --until-done) on a loopback gateway the runner starts in the run's home. The gateway, and anything the arm left running, stops only after the final check, so a server the agent started is still up when it is judged. Hidden checks keep their own limits, so a submission cannot pass by raising a constant. The result records the resolved conductor agent.
  • Claude Code runs claude -p --permission-mode dontAsk --allowedTools <case tools> --strict-mcp-config --max-budget-usd.
  • Two cases from merged public PRs: S1 (simple, fix(chat): keep the image extension when a stored chat-image name is cut #36699) and M1 (medium, issues bug(meetings): cancelled calendar events still show as upcoming and can be pre-created #28572 and bug(meetings): a large calendar feed can hide the nearest meetings depending on event order #28652). --check proves with no model that each fails on its base and passes on its reference. Task repos use the branch trunk: Crew refuses a push to main, even to a local origin.
  • --repeat now defaults to unset: one run for scenarios as before, the tier's k for eval run.
  • RFC section 14.1 lists where v1 does less than the design. The CLI spec, the eval README and evals/tasks/README.md document the command.
  • The runner's five subprocess sites are listed in test/spawn_audit_helpers.py with their reasons. run_check is marked as deliberately not routed through the sandbox: the hidden check imports the agent's pushed code by design, the dedicated eval user contains it, and a sandboxed check is the follow-up recorded in RFC section 14.1.
  • The suite is fingerprinted at batch start and after every run; a run that changed any case, check or lib/ file is VOID and the batch stops, so no run can rewrite the grader for later runs.
  • Code: the runner is src/kiro_crew/eval/tasks.py, the arms src/kiro_crew/eval/arms.py (documented in src/kiro_crew/eval/README.md). The flags are in src/kiro_crew/cli_parser_admin.py. src/kiro_crew/cli.py dispatches eval run to _run_eval_tasks in src/kiro_crew/cli_commands_eval.py, which src/kiro_crew/cli_commands.py lists for type checkers.

Backwards compatibility

Compatible: kirocrew eval without run behaves as before; --repeat unset still means one run (_run_eval reads None as 1). run becomes a reserved first name, as RFC section 3.4 says; none of the bundled scenarios uses it.

Removes nothing: the change only adds a subcommand, flags, modules and case files.

Tests

  • test/test_eval_arms.py: each arm's exact argv (trust on its own chat, conductor mode without --agent, dontAsk with only the case tools and no MCP, no bypass flags); Crew --json and Claude stream-json parsing (asks, denials, credits or USD, budget stop, VOID statuses); the two unbuilt arms report "not available".
  • test/test_eval_tasks.py: shipped cases load and pass --check; a case whose base already passes is named; bad case.json fields are refused. Preflight cases: credential dirs, a readable person's home (a macOS uid 501 included), a service account skipped, no sandbox backend, Windows, a missing key. A run's env keeps only its homes and its arm's key. Scoring follows section 5 in order. The summary counts pass^k per case only when every requested run passed (a VOID run leaves the case incomplete) and leaves VOID out of the medians. The audit matches the suite path only as a whole path component. End to end with a fake claude on PATH: a pushed fix passes, and idle-after-done is sampled (the fake stays up until a spy on the check sees it pass, so no timing bet). An ask fails, no change fails the check, output naming the suite is VOID, and an arm past its cap is killed. A child the agent left running is gone after the run, an arm interrupted mid-run is stopped before the interrupt propagates, the kill deadline fires even while a check sample is running, a slow sample does not hold the arm, and a process the checked code starts dies with the check. A submission that raises its own length limit still fails S1. A Crew run is judged before its gateway stops. Also covered: the batch's default k, harness names, an eval home around or inside the suite (refused), and the CLI's exit codes 0/1/2/3.
  • test/test_cli_eval_user_scenarios.py: the --repeat default is now None.
  • Determinism: test_eval_tasks.py + test_eval_arms.py passed 20/20 at -n0, 10/10 at -n 2 beside test_cli_eval_user_scenarios.py and test_eval_baseline_gate.py, and in a shuffled order (seed 4242, 99 ids). -n 2, not -n 4: this host's shared-load rule caps workers at 2.

Manual verification

  • kirocrew eval run --suite tasks --check from the worktree: 2/2 cases bite and are solvable.
  • Run as an operator user with credentials: exit 3 with one Refused: line per reason (credential dirs, readable homes, missing key).
  • Started an eval gateway with start_gateway in a scratch run home: ss -ltn showed it on 127.0.0.1 and [::1] only, and stop_process stopped it.
  • Not run: a real model on any arm. This host has no eval-only key, and its kernel refuses user namespaces, so preflight would refuse a Crew arm here.

Related Issues

Closes #38161
Part of #37669 (the RFC)

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 17, 2026 00:31
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b35defb35ca3d633432476147b530d3a5b7fb4fb touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b35defb35ca3d633432476147b530d3a5b7fb4fb — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change is additive, --repeat unset still reads as one run in _run_eval, it adds no new dashboard UI, and nothing outside the patch reads what it changes.

Notes

  • run_check imports the agent's pushed code outside the sandbox, so containment rests only on running under a dedicated eval user, and preflight cannot enforce that. Until the sandboxed check in RFC §14.1 lands, the CLI spec and README should say plainly that this mode is for a dedicated user only.

[DESIGN-REVIEWED] b35defb

Description read

[DESCRIPTION-READ] 192e9a88376be5aadffc4e11b398b41f26bbbca9f7a423b2d530147c723d5862

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b35defb35ca3d633432476147b530d3a5b7fb4fb — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The base RFC rfc-builtin-eval.md is accepted and covers this command. It is a new subcommand you have to ask for, and the default kirocrew eval behaves as before.

Not checked: a real model on any arm (the author says so too), and the Crew arms on a host with a sandbox backend.

Notes

  • Item 9 — oversized (minor): stop_process and kill_group in eval/tasks.py are a third spelling of process-group teardown, beside platform_compat_kill.kill_popen_tree (20 references across src/kiro_crew) and kill_process_group. Subtraction: replace kill_group's signal step with platform_compat.kill_process_group called directly, and drop the _signal_group wrapper. Only the TERM-wait-KILL loop is new.

What this change ships

Inventory (10 items) — 9 justified

Intent: run the same coding task unattended on Crew chat, Crew conductor mode and Claude Code, and compare scored results per harness and tier. This is an ADDITION.

  1. New kirocrew eval run --suite tasks scores each run PASSED, FAILED or VOID — justified
  2. Each run gets its own home dirs and only the arm's key, under the strict sandbox — justified
  3. Preflight refuses (exit 3) on credential dirs, a readable home of another user, no sandbox backend, or a missing key — justified
  4. The runner starts a loopback gateway for Crew arms and stops it after the final check — justified
  5. A Claude Code arm with dontAsk, only the case's tools, and no MCP — justified
  6. Two task cases (S1, M1) and the shared lib/ helpers — justified
  7. --check proves with no model that each case fails on its base and passes on its reference — justified
  8. The --repeat default changes to unset: still one run for scenarios, the tier's k for eval run — justified
  9. Process-group stop and kill helpers in eval/tasks.py — oversized (overlaps platform_compat_kill)
  10. A suite fingerprint taken after each run makes a run that changed the grader VOID and stops the batch — justified

[FIRST-PRINCIPLES-REVIEWED] b35defb

Description read

[DESCRIPTION-READ] 869e19e173ebd5aea9786f02ca89a687f2c614b85a9e3e913dd855e0497bd963

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b35defb35ca3d633432476147b530d3a5b7fb4fb — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b35defb

Verdict parsed from the review's SHA-scoped output markers for commit b35defb35ca3d633432476147b530d3a5b7fb4fb.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b35defb35ca3d633432476147b530d3a5b7fb4fb false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ human override accepted

Human judgment by @iamwhatever overrides the GPT 6.1 finding for b35defb35ca3d633432476147b530d3a5b7fb4fb; the recorded reason is authoritative for this commit.

[GPT-OVERRIDE] b35defb

This comment is updated in place on each push.

The model was not re-run because an authorized human decision supersedes it.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b35defb35ca3d633432476147b530d3a5b7fb4fb false-positive: <reason, 20 to 500 characters>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=5c53eeb474af
fixed in 72b6708

stop_process and kill_group in src/kiro_crew/eval/tasks.py now go through one _signal_group, which calls platform_compat.kill_process_group (its broadcast/self-group guard and macOS all-zombie EPERM handling included) and reads ProcessLookupError / PermissionError / ValueError as "gone".
src/kiro_crew/eval/tasks.py has no direct os.killpg call left.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=ba3dcad07ed8
fixed in 72b6708

stop_process and kill_group in src/kiro_crew/eval/tasks.py now go through one _signal_group, which calls platform_compat.kill_process_group (its broadcast/self-group guard and macOS all-zombie EPERM handling included) and reads ProcessLookupError / PermissionError / ValueError as "gone".
src/kiro_crew/eval/tasks.py has no direct os.killpg call left.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=5c5ae19bb800
fixed in 72b6708

stop_process and kill_group in src/kiro_crew/eval/tasks.py now go through one _signal_group, which calls platform_compat.kill_process_group (its broadcast/self-group guard and macOS all-zombie EPERM handling included) and reads ProcessLookupError / PermissionError / ValueError as "gone".
src/kiro_crew/eval/tasks.py has no direct os.killpg call left.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=57ad4ba643cc
fixed in 72b6708

stop_process and kill_group in src/kiro_crew/eval/tasks.py now go through one _signal_group, which calls platform_compat.kill_process_group (its broadcast/self-group guard and macOS all-zombie EPERM handling included) and reads ProcessLookupError / PermissionError / ValueError as "gone".
src/kiro_crew/eval/tasks.py has no direct os.killpg call left.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=82e7335dc4c7
fixed in 72b6708

stop_process and kill_group in src/kiro_crew/eval/tasks.py now go through one _signal_group, which calls platform_compat.kill_process_group (its broadcast/self-group guard and macOS all-zombie EPERM handling included) and reads ProcessLookupError / PermissionError / ValueError as "gone".
src/kiro_crew/eval/tasks.py has no direct os.killpg call left.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=f5494d6316d5
fixed in 72b6708

audit_run now matches the suite path only as a whole path component (re.escape(path) + r"(?![\w.-])"), so an eval home such as evals/tasks-runs next to the suite is no longer a hit, while evals/tasks/cases/... still is.
Test: test_the_audit_matches_the_suite_path_only_as_a_whole_component.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=f33ebde45a53
rebutted: not a defect in a safety boundary; a scoring-integrity limit v1 records.

The finding: submitted code imported by evals/tasks/lib/hidden.py can print a fake verdict.
A hidden check is the execution of submitted code (RFC §5, §7.1); any in-process channel the check reports through, the submitted code can write too.
The only effect is on that run's own score. No credential, other user's data or operator file is reachable, because §4.4's boundary is the dedicated eval OS user with no credentials, which preflight enforces.
RFC §14.1 now records this limit, and a sandboxed, separate-process check as the follow-up.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Incomplete repetitions are reported as pass^k

fixed in 72b6708: summarize (src/kiro_crew/eval/tasks.py) counts a case toward pass^k only when every requested run of it is PASSED; a VOID run leaves the case incomplete.
Test: test_summary_counts_pass_k_per_case_and_leaves_void_runs_out_of_the_medians.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Hidden checks execute submitted code outside the strict sandbox

rebutted, not a defect in a safety boundary. A hidden check is the execution of submitted code (RFC §5, §7.1). It runs as the dedicated eval OS user, which preflight requires to hold no credential directory and to be unable to list another login user's home (RFC §4.1, §4.4).
Code that edits its own run directory (that run's crew/ config included) during the check reaches nothing outside that run, and the directory is deleted after copy-out, so the only effect is on that run's own score.
RFC §14.1 records this scoring-integrity limit and a sandboxed check as the follow-up.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=6ae74ebf2ab4
fixed in cc8f399

max_drop is now in _SCENARIO_ONLY (src/kiro_crew/cli_commands_eval.py), so kirocrew eval run --max-drop 0.1 exits 2 with "does not take --max-drop".
Test: the --max-drop row of test_the_two_eval_forms_do_not_mix.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Final spend above the cap still passes

fixed in cc8f399: score takes the run's caps and fails a run whose final reported cost is above the cap for its unit (credits or USD), not only one the arm itself stopped at the cap.
Test: three new rows of test_score_follows_the_rfc_rules_in_order (31 credits and 3.5 USD fail, exactly 30 credits passes).

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Blocking checks inflate arm duration and cause false timeouts

fixed in cc8f399: drive_arm timestamps the arm's exit in a waiter thread, so a hidden-check sample still running at exit does not count toward wall time or the timeout.
Test: test_a_slow_check_sample_does_not_count_toward_the_arm_wall_time (the sample waits until the arm is gone; wall time ends before the sample returns).

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=c5c426992518
fixed in cc8f399

The preflight tests give the synthetic eval user a fixed uid (4000), distinct from every other fixture user (501, 4242, 4343), so the test no longer depends on the uid it runs as.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Interrupted launches leave eval processes running

fixed in 5734a59: drive_arm and start_gateway (src/kiro_crew/eval/tasks.py) now stop the process group they started when they unwind with any exception (KeyboardInterrupt included), then re-raise; they start it in its own session, so no Ctrl-C reaches it otherwise.
Test: test_an_interrupted_arm_is_stopped_before_the_interrupt_propagates.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=c5c426992518

  • Subprocess deadlines exceed the test timeout

fixed in 06523ea: an autouse fixture in test/test_eval_tasks.py sets the build and hidden-check deadlines (BUILD_TIMEOUT_SECS, CHECK_TIMEOUT_SECS, now a module constant each) to default_timeout() and the kill grace to 0; arm caps in the tests are default_timeout() / 2, so a stalled subprocess fails as a bounded result inside the test timeout. Deliberate zero-deadline cases are kept.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=f5494d6316d5

fixed in 273426d: _copy_out copies the run's output with symlinks=True (a link is copied as a link, never followed) and keeps what it could copy on shutil.Error / OSError; the run directory is removed in a nested finally, and run_one also records ValueError / TypeError from a result parse as a VOID harness error instead of aborting the batch.
Test: test_a_link_the_agent_leaves_in_the_run_output_does_not_break_the_batch (a dangling link in the run output).

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Relative package-search paths break source-only Crew launches

fixed in 273426d: run_env resolves every PYTHONPATH entry against the runner's own working directory before any child starts elsewhere, so a source checkout's relative src still imports.
Test: test_a_relative_package_path_is_resolved_before_children_change_directory.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • An aborted batch counts incomplete cases as pass^k

fixed in 273426d: summarize takes each case's requested run count (requested_runs, shared with run_batch) and counts a case toward pass^k only when that many runs happened and all PASSED, so a batch that stopped early leaves its cases incomplete.
Test: the last assertion of test_summary_counts_pass_k_per_case_and_leaves_void_runs_out_of_the_medians, and test_requested_runs_are_the_repeat_or_the_tier_k.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=c5c426992518

  • Hidden-check subprocesses leave bytecode in the checkout

fixed in 45bcebd: both hidden-check launches run with -B (run_check in src/kiro_crew/eval/tasks.py and the isolated child in evals/tasks/lib/hidden.py), so no __pycache__ is written into the suite or any checkout; this covers every test site that runs the live suite's checks.
Test: test_running_the_checks_writes_no_bytecode_into_the_suite.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Copied artifact symlinks redirect the result writer

fixed in 9f9ae5c: _write_record publishes result.json by writing a uniquely named temp file and os.replace-ing it into place, so a link already at that name is replaced as a link and never followed (both the normal and the suite-tampering VOID path use it); _copy_out also skips any result.json in the run output.
Tests: test_a_result_json_link_the_agent_plants_writes_no_other_file, test_a_record_replaces_a_link_at_its_name_instead_of_following_it.

kirocrew eval run --suite tasks runs real coding cases unattended on
Crew chat, Crew conductor mode and Claude Code, and scores each run
PASSED, FAILED or VOID by rfc-builtin-eval.md section 5: any ask of a
person fails it, so do the wall and spend caps and the hidden check.
Each run writes one JSON result (asks, wall, cost, idle after done,
the check's parts, the resolved conductor agent); the batch writes one
row per harness and tier through the existing eval artifact writer.

Each run gets its own HOME, KIROCREW_HOME, KIRO_HOME and TMPDIR, an env
with only the arm's eval sign-in, and the strict sandbox tier. Preflight
refuses when the running user holds credential directories or can list
another login user's home. A Crew arm's gateway starts in the run's
data home and stops only after the final check, so a server the agent
started is alive while it is judged. kiro-cli V3 and Codex are VOID
until their adapters are built.

Two cases from merged PRs ship under evals/tasks (one simple, one
medium). --check proves each fails on its base and passes on its
reference with no model. Task repos use a trunk branch, because Kiro
Crew refuses any push to a branch named main. The RFC's new section
14.1 records where v1 does less than the design.
@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=f5494d6316d5

fixed in b35defb: both hidden-check interpreters now run with -X pycache_prefix=<fresh private dir> as well as -B, so a bytecode cache inside the suite (or the checked tree) is never read, and suite_digest now fingerprints bytecode caches too, so a planted .pyc VOIDs the run and stops the batch.
Tests: test_a_cache_in_the_suite_is_never_read_by_the_check (an unchecked-hash pyc forging main is ignored), test_a_planted_bytecode_cache_changes_the_suite_fingerprint.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Excluded bytecode can replace the grader for subsequent runs

fixed in b35defb: the checks read bytecode only from a fresh private pycache_prefix (never the suite's __pycache__), and suite_digest now includes bytecode caches, so a planted cache both goes unread and VOIDs the run.
Tests: test_a_cache_in_the_suite_is_never_read_by_the_check, test_a_planted_bytecode_cache_changes_the_suite_fingerprint.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Transcript copying follows agent-planted destination links

fixed in b35defb: _copy_transcript copies to a fresh uniquely named file in the run output and os.replace-s it into transcript.jsonl, so a link already there is replaced, never followed.
Test: test_a_transcript_link_the_agent_plants_writes_no_other_file.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=7bf6e3122239

  • Preflight skips readable administrator homes

fixed in b35defb: _home_refusals now applies the listing check to uid 0 as well (only uids 1-499 and the eval user itself are skipped), so a root home the eval user can list refuses the batch.
Test: test_an_administrator_home_the_eval_user_can_list_is_refused.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt b35defb not-applicable: GPT provider refused this diff for its content (security test fixtures); Opus, Design, First Principles, UX pass on this head.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@iamwhatever recorded a not-applicable release (class not-applicable) for the gpt AI finding on b35defb35ca3d633432476147b530d3a5b7fb4fb.

GPT provider refused this diff for its content (security test fixtures); Opus, Design, First Principles, UX pass on this head.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 17, 2026
@iamwhatever
iamwhatever merged commit 226c9d6 into main Nov 17, 2026
156 of 159 checks passed
@iamwhatever
iamwhatever deleted the feat/eval-run-v1 branch November 17, 2026 17:00
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(eval): kirocrew eval run for task cases across harnesses

3 participants