Repository navigation
feat(eval): --use-operator-credentials runs every arm with your own sign-ins - #39317
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The flag is opt-in, the owner's decision is recorded, the risk is printed and written to every run record, and the default path still gives each run its own fresh home. Notes
[DESIGN-REVIEWED] b0f2803 Description read[DESCRIPTION-READ] cd9ff6692b55621d5426491ce022b4a30ac45235760b9c2130415b9880f75e2d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the flag is off by default, the owner's decision is recorded, and each item is needed for the one-command run the Goal asks for. Notes
What this change shipsInventory (9 items) — 8 justifiedIntent: let the owner run the first
[FIRST-PRINCIPLES-REVIEWED] b0f2803 Description read[DESCRIPTION-READ] 9af456bf95108a92670a8477f700727b558ce33b148ef0ab26c3d6a43243a499 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
5dd8480 to
fd78149
Compare
|
Evidence on the Opus 5.5 and GPT 6.1 security blocks (login-route kiro-cli store readable by same-user unsandboxed code: the Claude Code arm and the hidden check). Head The finding is accurate about reachability. What it changes is the credential class, not who can reach a credential:
Containing the hidden check and the Claude Code arm in a sandbox is the RFC §14.1 follow-up and is out of this PR's goal. The owner accepted using their own account for the eval sign-in to keep setup to one command; the override decision is theirs. |
fd78149 to
0c3beca
Compare
|
Head |
0c3beca to
89684af
Compare
|
Head On the Opus 5.5 and GPT 6.1 security blocks of |
89684af to
f43f323
Compare
|
Head
|
f43f323 to
d084874
Compare
|
Head
|
d084874 to
514929e
Compare
|
Head
|
514929e to
39bb2bc
Compare
|
Head |
f4588f3 to
b07695e
Compare
b07695e to
38ce5bd
Compare
|
Head
Rebased on |
38ce5bd to
64141e8
Compare
|
Head |
…ign-ins An opt-in flag for a machine where kiro-cli and Claude Code are already signed in: every arm runs as the operator, with their env and real HOME, minus their KIROCREW_* values (a relative PYTHONPATH made absolute), while each run keeps its own data home, KIRO_HOME, TMPDIR and work directory. Preflight skips the eval-user checks and instead requires `kiro-cli whoami` (the binary a gateway session would run, with a fresh KIRO_HOME) and `claude -p "say ok"` to exit 0; not-built arms stay VOID per run. The CLI prints a risk line, the batch JSON carries it, and every run record carries `credentials`. Without the flag, preflight is main's. RFC §4.3/§14.1, docs/system-specs/modules/cli.md and evals/tasks/README.md describe it.
64141e8 to
b0f2803
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Maintainer decision (Bolin): approved on b0f2803441. Running evals with the operator's own sign-ins under the opt-in --use-operator-credentials flag is accepted, with the trade-off the Captain named: under the flag the Claude Code arm and the hidden check run as the operator and can reach those sign-ins, and the eval-user fence of RFC §4.4/§14.1 does not hold. That is stated in the RFC, the decision record and every run record, and without the flag behaviour is main's. The flag is for the operator's own machine only, not CI or shared hosts.
AI-assisted, sent by Bolin Review bot
Problem / Motivation
Goal: the first real
kirocrew eval runtakes the owner one command, with no new OS user, no sudo and no minted keys: an opt-in--use-operator-credentialsruns every arm with the operator's existing kiro-cli and Claude Code sign-ins (owner decision, 2026-11-19).kirocrew eval run --suite taskstoday only runs as a dedicated eval OS user with eval-only keys (KIRO_API_KEY, an Anthropic or Bedrock key). The owner found that setup too much work: a new user, sudo installs, a separate Kiro key and a Bedrock key, for a host where Kiro and Claude Code are already signed in.Why it matters
The first eval run (Crew chat vs Crew conductor vs Claude Code) has been blocked on that setup. The owner accepts the risk of running the arms with their own sign-ins on their own machine.
Not a goal
What changed (motivation → approach → change)
New opt-in flag
--use-operator-credentials(src/kiro_crew/cli_parser_admin.py, wired insrc/kiro_crew/cli_commands_eval.py). The CLI copies the resolved arms (never the sharedARMSinstances) and setsoperator_credentialson each._operator_refusalsineval/tasks.py): with the flag the eval-user checks of RFC §4.1 are skipped, since the operator's credentials are the point. Instead each arm's own sign-in must answer:kiro-cli whoamifor the Crew arms (once for both; the binaryresolve_kiro_cli()picks, as a gateway session would, run with a freshKIRO_HOMElike each run gets) andclaude -p "say ok"for Claude Code (operator_check/_probeineval/arms.py). Either failing refuses with exit 3 before any case is built. Each probe runs in its own process group (viaplatform_compat), killed whole on every exit, with a 120 s limit and a bounded reap that closes its pipes. Not-built arms (kiro-cli-v3,codex, included by--harness all) are skipped there and stay VOID per run, as without the flag. A Crew arm still needs the strict sandbox tier.run_env): the operator's own env and realHOME, so their sign-ins work, minus theirKIROCREW_*values; a relativePYTHONPATHis made absolute as on the default path;KIROCREW_KIRO_BINis pinned to the binary the preflightwhoamiresolved from the same env (operator_kiro_cli, absolute), so preflight and the runs use one kiro-cli; each run still gets its own Crew data home,KIRO_HOME,TMPDIRand work directory.credentials(eval-onlyby default,operator's own (--use-operator-credentials)with the flag).HOMEaclaudePATH shim works, so it is kept as found (made absolute) instead of resolved to its real path.~/.claude), because a sign-in kept there (apiKeyHelper) is what the flag reuses. Its allow rules and hooks then merge with--allowedTools, so RFC §3.2's same-tools rule does not fully hold for that arm; the RFC row and the decision record say so.~/.kiro/settings/mcp.jsonfollows the realHOME); the RFC §4.3 and §14.1 rows say so. The run's Crew config (write_crew_config) now setskiro_hooks_autoimport: falsein every mode, which stops the~/.kiro/hooks/*.shscript import; the operator's standalone~/.kiro/hooks/*.jsonhook files still run under the flag, and the RFC says so.Docs:
docs/request-for-change/rfc-builtin-eval.md§4.3 gains the row and §4.4 says plainly that with the flag items 1, 4 and 6 do not hold; §14.1 lists the flag and says the eval-user fence does not hold under it. The owner's decision is recorded indocs/decisions/2026-11-19-eval-runs-may-use-the-operators-own-sign-ins.md(indexed indocs/decisions/README.md); its evidence links the owner's decision comment on this PR (#39317 (comment)). The usage error without--eval-homeno longer claims arms never run in your own home.docs/system-specs/modules/cli.mdandevals/tasks/README.mddescribe the flag. The probe spawn is in the spawn audit allowlist (test/spawn_audit_helpers.py).The remaining exposure is the owner's explicit decision.
Backwards compatibility
Compatible: without
--use-operator-credentialsevery check, env and argv is main's; run records gain one field,credentials, and the batch JSON one key.Removes nothing: no flag, field or check is taken away.
Tests
test_operator_credentials_skip_the_eval_user_checks_and_ask_each_sign_in: with the flag a home holding.sshand a readable other home passes when both sign-ins answer, with the not-built arms in the batch; both Crew arms share onekiro-cli whoami; a failingsay okrefuses; without the flag the same home is refused exactly as on main.test_operator_credentials_keep_the_runners_env_and_home_but_a_run_data_home: env is the operator's minusKIROCREW_*,PYTHONPATHmade absolute, with the run's own data home,KIRO_HOME,TMPDIR.test_operator_credentials_from_the_command_line_print_the_risk_and_swap_copies: flags parsed, copies flagged, sharedARMSuntouched, risk line printed.test_operator_checks_run_whoami_and_say_ok: exit 0 passes;whoamisees a freshKIRO_HOME; otherwise the command, exit code and last stderr line; a missingkiro-cliis named.test_under_operator_credentials_a_claude_shim_is_kept_as_found.test_a_probe_that_hangs_is_killed_with_everything_it_started: a wrapper that forks a child and hangs is reported as timed out, and its child is gone.credentials == "eval-only".Local:
test/test_eval_arms.py test/test_eval_tasks.py test/test_spawn_audit.pypass (106).Manual verification
Not run end to end: the batch spends Kiro credits and Claude Code dollars and needs the owner's sign-ins. The owner's one command does it; a failing sign-in stops at preflight with nothing spent.
Real-machine verification
N/A -- the real run needs the owner's own kiro-cli and Claude Code sign-ins and spends their credits; unit tests cover preflight, env and records.
Related Issues
N/A -- follow-up to the eval run v1 (#38187) for the first real run.
Checklist