Skip to content

feat(eval): --use-operator-credentials runs every arm with your own sign-ins - #39317

Merged
bolichen97 merged 1 commit into
mainfrom
feat/eval-own-kiro-login
Nov 20, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
feat/eval-own-kiro-login

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: the first real kirocrew eval run takes the owner one command, with no new OS user, no sudo and no minted keys: an opt-in --use-operator-credentials runs every arm with the operator's existing kiro-cli and Claude Code sign-ins (owner decision, 2026-11-19).

kirocrew eval run --suite tasks today only runs as a dedicated eval OS user with eval-only keys (KIRO_API_KEY, an Anthropic or Bedrock key). The owner found that setup too much work: a new user, sudo installs, a separate Kiro key and a Bedrock key, for a host where Kiro and Claude Code are already signed in.

Why it matters

The first eval run (Crew chat vs Crew conductor vs Claude Code) has been blocked on that setup. The owner accepts the risk of running the arms with their own sign-ins on their own machine.

Not a goal

  • No change to preflight without the flag: it is exactly main's.
  • No sandbox for the Claude Code arm or the hidden check (RFC §14.1 follow-up).
  • No batch-wide spend cap flag; per-case caps stay as they are.
  • No kiro-cli-v3 or Codex arm.

What changed (motivation → approach → change)

New opt-in flag --use-operator-credentials (src/kiro_crew/cli_parser_admin.py, wired in src/kiro_crew/cli_commands_eval.py). The CLI copies the resolved arms (never the shared ARMS instances) and sets operator_credentials on each.

  1. Preflight (_operator_refusals in eval/tasks.py): with the flag the eval-user checks of RFC §4.1 are skipped, since the operator's credentials are the point. Instead each arm's own sign-in must answer: kiro-cli whoami for the Crew arms (once for both; the binary resolve_kiro_cli() picks, as a gateway session would, run with a fresh KIRO_HOME like each run gets) and claude -p "say ok" for Claude Code (operator_check / _probe in eval/arms.py). Either failing refuses with exit 3 before any case is built. Each probe runs in its own process group (via platform_compat), killed whole on every exit, with a 120 s limit and a bounded reap that closes its pipes. Not-built arms (kiro-cli-v3, codex, included by --harness all) are skipped there and stay VOID per run, as without the flag. A Crew arm still needs the strict sandbox tier.
  2. Run env (run_env): the operator's own env and real HOME, so their sign-ins work, minus their KIROCREW_* values; a relative PYTHONPATH is made absolute as on the default path; KIROCREW_KIRO_BIN is pinned to the binary the preflight whoami resolved from the same env (operator_kiro_cli, absolute), so preflight and the runs use one kiro-cli; each run still gets its own Crew data home, KIRO_HOME, TMPDIR and work directory.
  3. Said out loud: the CLI prints a risk line before the batch, the batch JSON carries it, and every run record carries credentials (eval-only by default, operator's own (--use-operator-credentials) with the flag).
  4. Under the operator's HOME a claude PATH shim works, so it is kept as found (made absolute) instead of resolved to its real path.
  5. Claude Code keeps reading the operator's user settings (~/.claude), because a sign-in kept there (apiKeyHelper) is what the flag reuses. Its allow rules and hooks then merge with --allowedTools, so RFC §3.2's same-tools rule does not fully hold for that arm; the RFC row and the decision record say so.
  6. A Crew arm under the flag also loads the operator's user-level kiro-cli MCP servers (~/.kiro/settings/mcp.json follows the real HOME); the RFC §4.3 and §14.1 rows say so. The run's Crew config (write_crew_config) now sets kiro_hooks_autoimport: false in every mode, which stops the ~/.kiro/hooks/*.sh script import; the operator's standalone ~/.kiro/hooks/*.json hook files still run under the flag, and the RFC says so.

Docs: docs/request-for-change/rfc-builtin-eval.md §4.3 gains the row and §4.4 says plainly that with the flag items 1, 4 and 6 do not hold; §14.1 lists the flag and says the eval-user fence does not hold under it. The owner's decision is recorded in docs/decisions/2026-11-19-eval-runs-may-use-the-operators-own-sign-ins.md (indexed in docs/decisions/README.md); its evidence links the owner's decision comment on this PR (#39317 (comment)). The usage error without --eval-home no longer claims arms never run in your own home. docs/system-specs/modules/cli.md and evals/tasks/README.md describe the flag. The probe spawn is in the spawn audit allowlist (test/spawn_audit_helpers.py).

The remaining exposure is the owner's explicit decision.

Backwards compatibility

Compatible: without --use-operator-credentials every check, env and argv is main's; run records gain one field, credentials, and the batch JSON one key.
Removes nothing: no flag, field or check is taken away.

Tests

  • test_operator_credentials_skip_the_eval_user_checks_and_ask_each_sign_in: with the flag a home holding .ssh and a readable other home passes when both sign-ins answer, with the not-built arms in the batch; both Crew arms share one kiro-cli whoami; a failing say ok refuses; without the flag the same home is refused exactly as on main.
  • test_operator_credentials_keep_the_runners_env_and_home_but_a_run_data_home: env is the operator's minus KIROCREW_*, PYTHONPATH made absolute, with the run's own data home, KIRO_HOME, TMPDIR.
  • test_operator_credentials_from_the_command_line_print_the_risk_and_swap_copies: flags parsed, copies flagged, shared ARMS untouched, risk line printed.
  • test_operator_checks_run_whoami_and_say_ok: exit 0 passes; whoami sees a fresh KIRO_HOME; otherwise the command, exit code and last stderr line; a missing kiro-cli is named.
  • test_under_operator_credentials_a_claude_shim_is_kept_as_found.
  • test_a_probe_that_hangs_is_killed_with_everything_it_started: a wrapper that forks a child and hangs is reported as timed out, and its child is gone.
  • The Claude Code run test now also checks credentials == "eval-only".

Local: test/test_eval_arms.py test/test_eval_tasks.py test/test_spawn_audit.py pass (106).

Manual verification

Not run end to end: the batch spends Kiro credits and Claude Code dollars and needs the owner's sign-ins. The owner's one command does it; a failing sign-in stops at preflight with nothing spent.

Real-machine verification

N/A -- the real run needs the owner's own kiro-cli and Claude Code sign-ins and spends their credits; unit tests cover preflight, env and records.

Related Issues

N/A -- follow-up to the eval run v1 (#38187) for the first real run.

Checklist

  • A Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (RFC §4.3, §14.1; docs/system-specs/modules/cli.md; evals/tasks/README.md)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 18, 2026 23:46
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b0f2803441b118f4342cd8c7eee8bd1663688cc5 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b0f2803441b118f4342cd8c7eee8bd1663688cc5 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The flag is opt-in, the owner's decision is recorded, the risk is printed and written to every run record, and the default path still gives each run its own fresh home.

Notes

  • write_crew_config now sets kiro_hooks_autoimport: false in every mode, not just under the flag. That conflicts with "every check, env and argv is main's". It has no effect today because the default path's run HOME is fresh, but the compatibility line should mention it.

[DESIGN-REVIEWED] b0f2803

Description read

[DESCRIPTION-READ] cd9ff6692b55621d5426491ce022b4a30ac45235760b9c2130415b9880f75e2d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b0f2803441b118f4342cd8c7eee8bd1663688cc5 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the flag is off by default, the owner's decision is recorded, and each item is needed for the one-command run the Goal asks for.

Notes

  • Item 2 — duplicate of src/kiro_crew/eval/tasks.py::kill_group/_signal_group: the inline try: platform_compat.kill_process_group(proc.pid, SIGKILL) except (OSError, ValueError): pass in _probe is a second spelling of the eval package's one group-kill helper (1 existing site; arms.py cannot import tasks.py without an import cycle). Subtraction: move _probe into eval/tasks.py beside _run_check and call kill_group(proc.pid) there, deleting the inline copy.

What this change ships

Inventory (9 items) — 8 justified

Intent: let the owner run the first kirocrew eval run with their existing kiro-cli and Claude Code sign-ins, with no eval OS user or new keys. This is an ADDITION.

  1. New opt-in kirocrew eval run --use-operator-credentials flag, off by default — justified
  2. With the flag, preflight skips the eval-user checks and runs kiro-cli whoami / claude -p "say ok", refusing with exit 3 if either fails — duplicate of eval/tasks.py::kill_group (group-kill only)
  3. With the flag, runs use the operator's env and real HOME, minus KIROCREW_*, with their own data home, KIRO_HOME and TMPDIR — justified
  4. Runs are pinned to the kiro-cli binary the preflight checked (KIROCREW_KIRO_BIN) — justified
  5. A risk line is printed before the batch, and the batch JSON gains a credentials key — justified
  6. Every run record carries credentials (eval-only by default) — justified
  7. A claude PATH shim is kept as found under the flag — justified
  8. The run config turns off ~/.kiro/hooks/*.sh auto-import in every mode — rides along; a no-op without the flag (fresh HOME)
  9. The usage error without --eval-home is reworded — justified

[FIRST-PRINCIPLES-REVIEWED] b0f2803

Description read

[DESCRIPTION-READ] 9af456bf95108a92670a8477f700727b558ce33b148ef0ab26c3d6a43243a499

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b0f2803441b118f4342cd8c7eee8bd1663688cc5 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b0f2803

Verdict parsed from the review's SHA-scoped output markers for commit b0f2803441b118f4342cd8c7eee8bd1663688cc5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b0f2803441b118f4342cd8c7eee8bd1663688cc5 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b0f2803441b118f4342cd8c7eee8bd1663688cc5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b0f2803

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b0f2803441b118f4342cd8c7eee8bd1663688cc5 false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch 3 times, most recently from 5dd8480 to fd78149 Compare November 19, 2026 00:12
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Evidence on the Opus 5.5 and GPT 6.1 security blocks (login-route kiro-cli store readable by same-user unsandboxed code: the Claude Code arm and the hidden check). Head fd78149ef9.

The finding is accurate about reachability. What it changes is the credential class, not who can reach a credential:

  1. Every eval credential is already same-user readable. The runner holds KIRO_API_KEY and AWS_BEARER_TOKEN_BEDROCK in its own environment, and any process of the same user can read /proc/<runner pid>/environ. Measured on the eval host (kernel.yama.ptrace_scope = 0): a same-uid sibling process read another process's environment variable verbatim. So on main the Claude Code arm and the hidden check could already read the Crew key and the Bedrock key; "KIRO_API_KEY reached only the Crew arms" holds for env_keys, not for a same-uid reader.
  2. The token is a separate, revocable sign-in. kceval runs its own kiro-cli login (device flow) in its own home. Nothing of the operator's is read or copied. It can be revoked by signing out, like a key can be rotated.
  3. It lives on disk only for the run. The owner script signs kceval out with kiro-cli logout on every exit (done, failed or Ctrl-C, via trap ... EXIT), so no login stays on disk between batches.
  4. The operator fence is unchanged. Preflight still refuses any other credential path in the eval user's home (.aws, .ssh, amazon-q store, ...), still refuses a readable home of any other login user, and the store is admitted only on this route (a Crew arm with no KIRO_API_KEY). Tests: test_preflight_admits_the_eval_users_kiro_cli_store_only_when_it_is_the_sign_in.

Containing the hidden check and the Claude Code arm in a sandbox is the RFC §14.1 follow-up and is out of this PR's goal. The owner accepted using their own account for the eval sign-in to keep setup to one command; the override decision is theirs.

@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from fd78149 to 0c3beca Compare November 19, 2026 00:29
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 0c3becaf2a adds --cc-own-login (owner decision, 2026-11-19 00:20Z: assume claude already works on the machine and do not manage its key). It is opt-in, claude-code only and alone in its batch; the Crew arms keep the eval-user fence and its own kiro-cli login. With the flag the Claude Code arm runs as the operator with the operator's credentials reachable, which RFC §4.3 and §14.1 now state plainly; preflight runs claude -p "say ok" first and every run record carries credentials. The evidence above on the kiro-cli login route still stands for the Crew arms.

@iamwhatever iamwhatever changed the title feat(eval): Crew arms also accept the eval user's own kiro-cli login feat(eval): --use-operator-credentials runs every arm with your own sign-ins Nov 19, 2026
@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from 0c3beca to 89684af Compare November 19, 2026 00:44
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 89684afbd1 replaces the earlier routes (the eval user's own kiro-cli login and --cc-own-login) with one opt-in flag, --use-operator-credentials, per the owner's decision at 2026-11-19 00:30Z: the host is already signed in to Kiro and Claude Code, use those sign-ins, the owner accepts the risk.

On the Opus 5.5 and GPT 6.1 security blocks of fe4e83793e: the kiro-cli store exemption and per-run copy they flagged are removed. Without the flag, preflight is main's exactly. With the flag, every arm runs as the operator and can reach the operator's credentials; that is the owner's explicit trade, stated in RFC §4.3 (where §4.4 items 1 and 4 are marked as not holding), printed by the CLI before the batch, and recorded as credentials in every run record. Preflight requires kiro-cli whoami and claude -p "say ok" to succeed first. The same-user /proc/<pid>/environ evidence above still applies to the eval-user route on main.

@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from 89684af to f43f323 Compare November 19, 2026 01:10
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head f43f323735 addresses every finding on 89684afbd1 in one push:

  • Opus 5.5 / GPT 6.1 F2, relative PYTHONPATH (run_env): fixed. Both branches now share the absolute-path step; test_operator_credentials_keep_the_runners_env_and_home_but_a_run_data_home pins PYTHONPATH=src becoming absolute under the flag.
  • GPT 6.1 F1, --harness all refused by not-built arms (_operator_refusals): fixed. NotBuiltArm placeholders are skipped in the operator preflight and stay VOID per run, as without the flag; the preflight test now includes kiro-cli-v3 and codex.
  • Design note, probe vs run KIRO_HOME: fixed. The kiro-cli whoami probe runs with a fresh temporary KIRO_HOME, as each run gets; the arm test asserts the probe saw neither the real nor an empty value.
  • First Principles item 2, shutil.which duplicate: fixed. The probe uses resolve_kiro_cli(), the binary a gateway session would run.
  • First Principles item 5, --claude-bin beyond the Goal: removed (flag, binary_path, its error branch and docs); the claude on PATH is used.
  • CI: Backend Lint (black on test/test_eval_tasks.py) and Backend Ratchets (subprocess encoding in _probe, now **UTF8_TEXT) fixed; Coverage Gate failed only because lint failed first.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from f43f323 to d084874 Compare November 19, 2026 01:40
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head d084874943 addresses the findings on f43f323735:

  • Opus 5.5 / GPT 6.1 F1, preflight and runs resolving different kiro-cli binaries: fixed. Operator-mode run_env pins KIROCREW_KIRO_BIN to resolve_kiro_cli(environ=source), the same call and env the preflight whoami used, so an operator override (KIROCREW_KIRO_BIN) or the bundled directory reaches the run's gateway as the one checked binary. Pinned by test_operator_credentials_keep_the_runners_env_and_home_but_a_run_data_home.
  • GPT 6.1 F2, a timed-out probe leaving a forked child: fixed. _probe starts the probe in its own process group and kills the whole group on every exit (success, failure, timeout, interrupt). test_a_probe_that_hangs_is_killed_with_everything_it_started runs a wrapper that forks sleep and hangs, and checks the child is gone.
  • Rate-limited Analyze (actions), Dependency License Gate and Publish readiness signal reds on the previous head were infra only; this push runs them again.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from d084874 to 514929e Compare November 19, 2026 01:55
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 514929e729 addresses the findings on d084874943:

  • GPT 6.1 F1, relative claude path in operator mode: fixed. ClaudeCodeArm.binary() returns os.path.abspath(found) under the flag (shim kept, symlinks not resolved); the shim test now also covers a relative PATH entry.
  • GPT 6.1 F2, unbounded drain after a timeout: fixed. Cleanup kills the group, reaps with wait(timeout=5) and closes the captured pipes instead of communicate().
  • GPT 6.1 F3, zombie counted as alive in the test: fixed. The probe test treats a zombie as gone, the same rule as _alive() in test/test_eval_tasks.py.
  • PR Policy, Cross-Platform Portability (os.killpg / signal.SIGKILL): fixed. _probe uses platform_compat.kill_process_group(..., platform_compat.SIGKILL), as tasks.kill_group does; the gate's patterns find no hit in the added src/ lines.

@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from 514929e to 39bb2bc Compare November 19, 2026 02:27
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 39bb2bce62 addresses the GPT 6.1 finding on 514929e729 (relative kiro-cli path): fixed. One helper, operator_kiro_cli() in eval/arms.py, resolves the binary with resolve_kiro_cli and returns it absolute (symlinks kept); the preflight whoami and the runs' pinned KIROCREW_KIRO_BIN both use it. test_operator_credentials_keep_the_runners_env_and_home_but_a_run_data_home now pins a relative override becoming absolute. Opus 5.5 had no findings on 514929e729. Analyze (javascript-typescript) was rate-limited only; this push runs it again.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 19, 2026
@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from f4588f3 to b07695e Compare November 19, 2026 23:53
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 19, 2026
@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from b07695e to 38ce5bd Compare November 20, 2026 00:11
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 38ce5bda61 answers the Opus 5.5 finding on b07695e14b (operator ~/.kiro MCP servers and hooks reach the Crew arms under --use-operator-credentials):

  • Hooks: fixed. write_crew_config now writes "kiro_hooks_autoimport": false, so no ~/.kiro/hooks script runs on an eval turn in either mode. The run-config test pins the key.
  • User-level MCP servers: documented, not changed. ~/.kiro/settings/mcp.json follows the real HOME, not KIRO_HOME, and Crew has no switch that skips it; adding one is a new mechanism beyond this PR's goal. Under the flag a Crew arm therefore loads them, the same kind of operator-settings merge the owner accepted for Claude Code. RFC §4.3 and the §14.1 network/MCP row now say so; without the flag nothing changes.

Rebased on origin/main at the same time. Local: test/test_eval_arms.py, test/test_eval_tasks.py pass (108); check_decisions_history.py passes.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 20, 2026
@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from 38ce5bd to 64141e8 Compare November 20, 2026 01:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 20, 2026
@iamwhatever

iamwhatever commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Head b0f2803441 answers the Opus 5.5 finding on 64141e8025 (kiro_hooks_autoimport stops only the *.sh import; standalone ~/.kiro/hooks/*.json hook files still run on KAS turns under the flag): documented, not changed. The finding is right, and no existing setting turns those files off: resolve_kiro_hooks_dir only accepts a directory under the real HOME, so a new eval-scoped key would be a new mechanism beyond this PR's goal. The write_crew_config docstring, the RFC §4.3 row and the §14.1 network/MCP row now say the *.json hook files still run under --use-operator-credentials, next to the operator MCP servers. Without the flag nothing changes. Rebased on origin/main; eval tests pass (108).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 20, 2026
…ign-ins

An opt-in flag for a machine where kiro-cli and Claude Code are already
signed in: every arm runs as the operator, with their env and real HOME,
minus their KIROCREW_* values (a relative PYTHONPATH made absolute), while
each run keeps its own data home, KIRO_HOME, TMPDIR and work directory.
Preflight skips the eval-user checks and instead requires `kiro-cli whoami`
(the binary a gateway session would run, with a fresh KIRO_HOME) and
`claude -p "say ok"` to exit 0; not-built arms stay VOID per run. The CLI
prints a risk line, the batch JSON carries it, and every run record carries
`credentials`. Without the flag, preflight is main's.

RFC §4.3/§14.1, docs/system-specs/modules/cli.md and evals/tasks/README.md
describe it.
@iamwhatever
iamwhatever force-pushed the feat/eval-own-kiro-login branch from 64141e8 to b0f2803 Compare November 20, 2026 01:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 20, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer decision (Bolin): approved on b0f2803441. Running evals with the operator's own sign-ins under the opt-in --use-operator-credentials flag is accepted, with the trade-off the Captain named: under the flag the Claude Code arm and the hidden check run as the operator and can reach those sign-ins, and the eval-user fence of RFC §4.4/§14.1 does not hold. That is stated in the RFC, the decision record and every run record, and without the flag behaviour is main's. The flag is for the operator's own machine only, not CI or shared hosts.

AI-assisted, sent by Bolin Review bot

@bolichen97
bolichen97 merged commit e4949a5 into main Nov 20, 2026
101 checks passed
@bolichen97
bolichen97 deleted the feat/eval-own-kiro-login branch November 20, 2026 07:34
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants