Skip to content

fix(kas): renew a refused crew sign-in instead of failing every start - #38978

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/kas-token-refresh-38912
Nov 18, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/kas-token-refresh-38912

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When Kiro refuses the token Kiro Crew gave its own engine, the next start gets a new token without a gateway restart, and a sign-in Kiro will not renew fails at once as a sign-in error.

A macOS user on the KAS engine saw KAS auth callback: refresh failed (ClientConnectorDNSError). After that every new KAS runtime got The bearer token included in the request is invalid on management calls, governance fail-closed, MCP was off, and each start ended as AcpRequestTimeout: KAS managed MCP readiness timed out after 90s. Only a gateway restart fixed it.

Why it matters

Every bot check-in and cron on that gateway failed for hours, each after a 90-second wait, and the error pointed at MCP governance instead of the sign-in. Any user on the Crew-owned KAS route can hit it after one refused token.

Not a goal

What changed (motivation → approach → change)

Why the token stuck: the engine caches the token it was served until that token's own expiry, and it never asks again when Kiro refuses it. Kiro Crew's callback renews only a token that is close to expiry, so a token Kiro refused early went to every new runtime until it expired.

Case Before After
Renewal fails on the network (DNS, connect, timeout) callback fails at once tried 3 times (0.5s, 1.5s), no try starts past 10s
Start fail-closed on governance, Kiro Crew owns the token retryable governance error, same token next time the token this runtime was served is renewed, so the retry gets a new one
...and the renewal is refused, nobody is signed in, or it cannot be attempted for an expired token retryable governance error, forever AcpAuthRequired with the KAS sign-in message, no retry
...and the renewal fails on the network retryable governance error unchanged

The callback retry and the renewal live in acp/kas_host_auth.py; the fail-closed start calls it from acp/runtime_session_load.py. The runtime (runtime_host_requests.py, runtime_class.py) keeps only a sha256 digest of the token it served, never the token. The renewal uses ensure_fresh's existing rejected_access_token and its lock, so many starts at once make one refresh. The sign-in error is the existing KAS signed-out message and AcpAuthRequired path. The KAS auth spec (docs/system-specs/modules/kas-auth.md) says what the callback and the fail-closed start do, and names the known limit: sessions that share one engine process keep its cached token until it expires.

Backwards compatibility

Compatible: a start that worked still works. The new check runs only after a Crew-owned start already failed on a governance read.

Removes nothing: no field, route, key or behavior is taken away; one error case becomes the sign-in error it really is.

Tests

test/test_kas_token_refresh_recovery.py, against a real TokenStore; only the HTTP refresh is replaced.

  • DNS error then success, and timeout then success: the callback serves the new token.
  • A renewal that never succeeds stops after 3 tries; a refused one is tried once; no retry past the window.
  • Root cause pin: a live stored token is served again and again with no refresh.
  • Renewing the served token gives the next callback a new one; a refused renewal is a sign-in and records the marker; a network failure is not; a renewal blocked locally is a sign-in only for an expired token; a token a peer already replaced is not renewed again; an empty vault is a sign-in.
  • 5 runtimes at once make one refresh.
  • The start: refused renewal raises AcpAuthRequired (not transient, sign-in text, no governance text); renewed keeps the retryable KasGovernanceOffError; a cli-owned runtime is not checked.

Determinism: the file ran 3 times in a row green, and with three neighbour files under -n 4.

Proof: PROVEN test/test_kas_token_refresh_recovery.py::TestAFailClosedStart::test_a_renewed_sign_in_keeps_the_retryable_error (head 571e028)

Manual verification

The engine behavior was read from the pinned KAS engine 0.66.117 (AcpCallbackAuthProvider.ensureToken returns the cached token until expiresAt; nothing clears it on a refused request). Not run against a live Kiro refusal: that needs a token Kiro rejects before its expiry.

Related Issues

Closes #38912

Sibling sweep

Swept: git grep -n "rejected_access_token|answer_get_access_token" -- src/
Sibling: src/kiro_crew/dashboard/handlers/relay_viewer.py -- not affected -- it already renews a token the service answered 401
Sibling: src/kiro_crew/relay_client/lease_signin.py -- not affected -- it already passes the refused bearer to the provider
Sibling: src/kiro_crew/dashboard/sign_in_refresher.py -- not affected -- it already retries a network failure with backoff

Pattern harvest

Not generalizable: the other token consumers already feed a refusal back; only the engine callback could not, because the engine never reports it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 571e028a146ac77a921f850e1f6c38eb37616f7d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 571e028a146ac77a921f850e1f6c38eb37616f7d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The renewal and the sign-in error act only on a start that has already failed fail-closed, and they reuse ensure_fresh's rejected_access_token and its lock. One rider sits outside the Goal.

Not checked: whether the pinned KAS engine 0.66.117 keeps a refused token until expiresAt and never asks for a new one.

Notes

  • Item 1 — rides along, oversized: the Goal is "When Kiro refuses the token … the next start gets a new token". The callback retry (REFRESH_RETRY_DELAYS_SECS, REFRESH_RETRY_WINDOW_SECS, _is_transient_refresh_error) handles a network blip, which is a different defect. Items 3–4 remove the stuck refused token without it. The background loop in dashboard/sign_in_refresher.py already retries network failures. Subtraction: delete the retry loop in answer_get_access_token and keep the single return await handle_get_access_token(provider).

What this change ships

Inventory (5 items) — 4 justified

Intent: a KAS start recovers from a token Kiro refused without restarting the gateway, and a dead sign-in says it is a sign-in problem. This is a FIX.

  1. The engine's token callback retries a refresh that failed on the network up to 3 times within 10s — rides along, outside the Goal
  2. Each runtime remembers a sha256 digest of the last token it served — justified
  3. A start that fails on governance renews the token the engine holds, so the retry gets a new one — justified
  4. A refused renewal or an empty vault now fails at once as AcpAuthRequired with the sign-in message, instead of a retryable governance error — justified
  5. The KAS auth spec describes the callback retry, the renewal and the shared-process limit — justified

[FIRST-PRINCIPLES-REVIEWED] 571e028

Description read

[DESCRIPTION-READ] 990b0eeb62ed36d866741dd26e186185d1bd2485174f81698f5b70f0c97054b6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 571e028a146ac77a921f850e1f6c38eb37616f7d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 571e028

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 571e028a146ac77a921f850e1f6c38eb37616f7d false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 571e028a146ac77a921f850e1f6c38eb37616f7d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix reuses ensure_fresh's existing rejected_access_token path and lock, is limited to Crew-owned fail-closed starts, and removes nothing.

Notes

  • KasGovernanceOffError is treated as a possible auth refusal whatever its cause. If the governance endpoint fails for a non-auth reason, every retried start forces a token refresh. Narrowing the check to a frame that names an invalid bearer, if the engine reports one, would avoid those wasted refreshes.
  • The known limit for shared engine processes (chat_runtime_sharing, subagents) leaves those sessions stuck until the token expires. Track it as a follow-up issue instead of leaving it only in the spec.

[DESIGN-REVIEWED] 571e028

Description read

[DESCRIPTION-READ] 1595752686bda251d8c0b15a59e13a38bcfb5399b4976bd61b3fce466c9afb60

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 571e028a146ac77a921f850e1f6c38eb37616f7d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 571e028

Verdict parsed from the review's SHA-scoped output markers for commit 571e028a146ac77a921f850e1f6c38eb37616f7d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 571e028a146ac77a921f850e1f6c38eb37616f7d false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/kas-token-refresh-38912 branch from 098d67b to 726a1a2 Compare November 18, 2026 04:53
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Dispositions for head 098d67b, fixed in 726a1a2:

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/kas-token-refresh-38912 branch from 726a1a2 to 077dca2 Compare November 18, 2026 05:54
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Disposition for head 726a1a2, fixed in 077dca2:

  • GPT 6.1 BLOCKING src/kiro_crew/acp/kas_host_auth.py:169 (a locally blocked renewal read as a dead sign-in): fixed. RefreshBlocked is checked locally and asks no issuer, so sign_in_refused now returns True for it only when the stored token is already past its margin. A live token that cannot be renewed (for example an external IdP entry with no refresh token) keeps the retryable KasGovernanceOffError. New tests: test_a_live_token_that_cannot_be_renewed_is_not_a_sign_in, test_an_expired_token_that_cannot_be_renewed_is_a_sign_in. The spec line in kas-auth.md says the same.
  • The other lanes on 726a1a2 (First Principles, Design, Opus 5.5) passed after one rerun past the GitHub installation rate limit.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 18, 2026
@iamwhatever
iamwhatever force-pushed the fix/kas-token-refresh-38912 branch from 077dca2 to 571e028 Compare November 18, 2026 06:52
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Dispositions for head 077dca2, fixed in 571e028:

  • GPT 6.1 BLOCKING src/kiro_crew/acp/kas_host_auth.py:112 (a retry could start after the window, past its backoff): fixed. The window check now adds the backoff, so no try starts past REFRESH_RETRY_WINDOW_SECS. New test: test_no_retry_whose_backoff_ends_past_the_window.
  • Backend Tests (3.12, 1) and (Windows) (8): test_runtime_bindings.py::test_each_owner_names_the_fields_it_writes_and_reads[kiro_crew.acp.runtime_session_load] was red from this PR: runtime_session_load reads _kas_host_auth and _kas_served_token_digest without naming them. Fixed: both are in its module docstring's read list. Coverage Gate failed closed on that same backend failure.
  • CodeQL (actions, javascript-typescript), Publish readiness signal: GitHub installation API rate limit, not this diff; they rerun on the new head.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 18, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own.

When a Crew-owned KAS start fails on a governance read, the PR renews the token the engine was served, matched by its sha256 digest. The renewal reuses the existing ensure_fresh(rejected_access_token) path, so it already has the lock, the re-read and the refused-renewal marker. A refused renewal, an empty vault, or a locally blocked renewal of an expired token now fails at once as AcpAuthRequired. A network failure keeps the retryable governance error, and cli-owned runtimes are skipped. Both callers of _wait_managed_mcp pass the new error up the same way as the old one. The callback's network-error retry is limited by a 10s window and only covers aiohttp connection errors and timeouts; the error mapping is unchanged. Tests against a real TokenStore cover each branch, including concurrent starts, and CI is green. Nothing regresses compared with main.

Tier: 2, held for a person: the automated review did not clear it.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 571e028a14 · 482 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 18, 2026 20:14

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of head 571e028. We read the full diff and followed both callers of _wait_managed_mcp (create and load). The renewal reuses ensure_fresh(rejected_access_token) under its existing lock, and 5 concurrent starts make one refresh. Only an issuer refusal, an empty vault, or an expired and locally blocked token becomes AcpAuthRequired. A network failure keeps the retryable governance error. The retry loop retries only connection and timeout errors, and no try starts past its window. All 18 tests in the new test file pass on this head. The two fail-closed start cases fail on main.

Security: the runtime keeps only a sha256 digest of the served token, never the token, and nothing logs it. The check runs only on a Crew-owned route, through a positive self._kas_host_auth test. Governance still fails closed. When the renewal check is false, the original fail-closed error is raised again, so no path fails open.

Done well: the tests use a real TokenStore and replace only the HTTP refresh, so the lock, the refusal marker and single-flight run as real code.

Optional:

  • No test checks that the runtime records the digest of the token it served. The fixture sets it directly. If the write at runtime_host_requests.py:416 stored None, the renewal would stop working and every test would stay green. A test that serves getAccessToken through _answer_ownerless_request and then asserts the digest would close this gap.
  • Any governance api_failure forces a renewal, so an outage that has nothing to do with the token rotates it once per retried start. A disposition for the Design note, or one line in kas-auth.md, would record this.
  • Coordinate with #39128: whichever PR merges second should pass route=self.kas_spawn_route to the new signed_out_message call in runtime_session_load.py.

@KiroCrewCaptain
KiroCrewCaptain merged commit 0ca13b0 into main Nov 18, 2026
158 of 171 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/kas-token-refresh-38912 branch November 18, 2026 20:14
buluoray pushed a commit that referenced this pull request Nov 18, 2026
#39262)

#38978 added a direct AcpSessionHandle.__new__ in
test_kas_token_refresh_recovery.py after #38996's guard landed, so
test_bare_session_handle_guard fails on main.

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-10-71.ec2.internal>
buluoray added a commit that referenced this pull request Nov 19, 2026
…#38978)

Original author: Joe Guo. Conflict resolved on release/0.13.0.

(cherry picked from commit 0ca13b0)
buluoray added a commit that referenced this pull request Nov 19, 2026
…#38978)

Original author: Joe Guo. Conflict resolved on release/0.13.0.

(cherry picked from commit 0ca13b0)
buluoray pushed a commit that referenced this pull request Nov 19, 2026
…#38978)

Co-authored-by: zejiangg <zejiangg@amazon.com>
(cherry picked from commit 0ca13b0)
buluoray pushed a commit that referenced this pull request Nov 19, 2026
…#38978)

Co-authored-by: zejiangg <zejiangg@amazon.com>
(cherry picked from commit 0ca13b0)
bolichen97 added a commit that referenced this pull request Nov 19, 2026
… main (#39387)

Backports to release/0.13.0 the fixes from #39342, squashed into one
commit and brought up to date with release/0.13.0 after the Kiro Memory
preview backport (#39360). Picks, in order; each [pick sha] is the
commit on the #39342 branch, whose body has main's cherry-picked-from line:

- fix(monitor): wake the session once when its watched PR merges or closes (#37776) [pick 84da419]
- fix(security): match deny rules against Windows program suffixes and cmd/PowerShell in-word escapes (#33569) [pick e4941b7]
- fix(mimocode): refuse a chat that would answer from Xiaomi's hosted model unsigned (#37629) [pick 1bfdce3]
- fix(acp): end a quiet agent-started cycle on the shared runtime's stale gate (#37932) [pick fa870a2]
- fix(conductor): re-read goal-conductor after a compaction (#37917) [pick ccdd67c]
- fix(security): refuse a command nested past 32 substitutions before the floors walk it (#37875) [pick 2f9bd67]
- fix(mates): carry a moved Bot's shell toolsSettings entry as execute_bash instead of holding it (#37973) [pick 16022f4]
- fix: hold an app to its own sessions, jobs and runs (#37804) [pick 3dff0c0]
- fix(acp): keep a KAS chat's stored record when its resume fails after the load (#37955) [pick 6ceb7b5]
- fix(gateway): ignore a repeat SIGTERM during graceful shutdown (#38031) [pick b48287d]
- fix(security): keep a quote in a $( ) from hiding a later ssh (#37558) [pick b9fed61]
- fix(agent): rebuild takes a server's disabledTools from the kiro global (#38147) [pick c50bbcc]
- fix(mcp_gateway): keep the backend when a stub leaves an open listen (#38121) [pick 3e9c7f9]
- fix: refresh plan and model list after a kiro-cli sign-in to another account (#38053) [pick 5ad3d36]
- fix(slack): keep a picture sent in a linked thread for the dashboard turn (#38076) [pick 7f8903e]
- fix: keep a resolved tool policy when the gateway is too slow to answer (#36669) [pick 9be5691]
- fix: open owner-only files and prune overlay folders without following links (#36994) [pick 6a1acc9]
- fix: refuse the local token mint to app backends (#38113) [pick d166dab]
- fix(subagent): hold a spawn's first prompt for its MCP servers (#38127) [pick e3e0298]
- fix(security): refuse a headless git --output write with no approver (#36326) [pick 1abee40]
- fix(restore): require an explicit mode, never default to replace (#38094) [pick ca86c06]
- fix: make the deny gate's shell walk linear in command length (#36437) [pick 22c100b]
- fix(chat): let a cancelled background start finish its session teardown (#36664) [pick b1dd34f]
- fix(acp): refuse a start when two differing specs declare the agent's name (#38411) [pick 80ccfff]
- fix: probe a silent turn while its co-tenant keeps the process busy (#38662) [pick 6354490]
- fix(kas): let a reopened chat switch to models its load select omits (#38376) [pick 1640822]
- fix(cloud): install a patched node 22 on run on aws instances (#38732) [pick 059cb90]
- fix: write member record files without following links (#37303) [pick aac9642]
- fix: fail a kas start fast once the host has turned its mcp off (#38677) [pick e147238]
- fix(decisions): fall back to HTTP/1.1 when h2 is missing (#38759) [pick fc16482]
- fix(mcp_gateway): free a spawn permit when its backend exits before initialize (#38793) [pick a084ba2]
- fix(kas): start an app chat whose tool policy hides every kirocrew-cron tool (#38761) [pick ccea5a9]
- fix(acp): give the tool-stall watchdog a margin over native waits (#34117) [pick b5d9e89]
- fix(mcp): bind the working-folder answer to its launch; observe launch changes behind name grants (#38983) [pick a1069be]
- fix(mcp): hide the gateway secret files from the npm install and third-party probe spawns (#39000) [pick 14d8571]
- fix(sandbox): fold env name case on Windows in the agent env scrub (#33997) [pick 9bdce76]
- fix(apps): let app turns follow the Auto approval mode (#38830) [pick e8336df]
- fix: end a dashboard turn stopped while it is still being prepared (#36478) [pick 37fff93]
- fix(acp): cleanup_session deletes the record on the session's own host (#39046) [pick 31b5ec4]
- fix(acp): scan only the new part of a streamed tool result (#38852) [pick e2870db]
- fix(sandbox): make an unscoped session host the OOM victim, not the gateway (#39189) [pick 2be6dde]
- fix(security): read aws s3 downloads piped into tail, with 2>&1, or with an AWS_PROFILE prefix (#38271) [pick bf65d97]
- fix: redact four display and log paths and use a fixed 502 message for session switch refusals (#38982) [pick 55d9ec7]
- fix(sandbox): pin a credential-masking app's folder after the masks on the namespace backend (#34339) [pick afd2b14]
- fix(acp): wait for systemd scope admission as its own startup stage (#37008) [pick 3e29c86]
- fix(acp): wait for systemd scope admission on the AcpClient spawn path (#38303) [pick 85f0a25]
- fix(acp): stamp the spawn instance on the AcpClient child (#38810) [pick f36eb5a]
- fix: end the turn when the agent never confirms a stop (#38512) [pick 2665c9d]
- fix(kas): renew a refused crew sign-in instead of failing every start (#38978) [pick 2f48cac]
- fix(security): keep self-host detection when the ssh verb is a glob (#34056) [pick ab53f10]
- fix(security): judge each substitution body for ssh-to-self (#37616) [pick d5b7ac3]
- test: check the ssh floor's nesting bound below the gate's depth cap (#37964) [pick 6c9ae1d]
- fix(mate): apply held Bot limits on shared claude sessions and refuse unnarrowed KAS children (#39212) [pick 721bbbe]
- test(integration): wait out and reset the hidden-chat refresh between boots (#37933) [pick e77586d]
- fix(browser): fall back to Kiro's Chromium under Automatic when Chrome's remote debugging is off (#39265) [pick 31912eb]

Left out of #39342's set (not needed for 0.13.0): the provider_labels
refactor, #34045, #36668, #39120 and #37627.

Merge with release/0.13.0: kept both imports in runtime_session_load,
runtime_spawn_launch and session_provider; session_allocation lists the
memory_policy owner; the keep-record resume path passes kas_memory_mode
as main does; dropped the ACP_BACKENDS_SESSION_EVICTION import that
moved to session_provider_new_conversation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KAS: a failed token refresh never recovers, and every bot then fails with a 90s MCP readiness timeout instead of a sign-in error

2 participants