Repository navigation
fix(acp): recycle a claude backend that reports cloud_credential_error - #19293
akushonkamen wants to merge 1 commit into
Conversation
A long-lived Claude-on-Bedrock backend whose AWS credentials went stale answered every later turn with a frame whose data names errorKind cloud_credential_error. _raise_acp_error read only prose, so the frame became a plain AcpError(transient=False) with no tag, and no surface replaced the stuck process: each turn re-asked the same one and drew the same answer, and only a conversation reset recovered it (a chat started fresh in the same window worked, so the helper was healthy). _raise_acp_error now reads the kind STRUCTURALLY from a mapping data field (_error_kind_from_frame) and tags AcpError.credential_reload, pinning the verdict non-transient because the prose ends in 'try again' and must never point a retry ladder back at the process that cannot reload. Deliberately not auth_required: that tag offers the Kiro sign-in card, which signs in the wrong identity for a harness that loads its own credentials (harness parity H6). llm_helpers.acp_error_is_session_not_found now also reads the tag, so every surface that resets and retries once for a lost session -- dashboard chat, subagent runs, channel turns, the Slack gateway and handler -- recycles this backend the same way, with the same one-retry-per-turn budget and no-replay-after-tool-activity rule. Fixes kirodotdev#19262 Signed-off-by: akushonkamen <akushonkamen@163.com>
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The non-transient pin does nothing today: on base this frame is already Not justified as shipped
What this change shipsInventory (4 items) — 2 justifiedIntent: a Claude-on-Bedrock chat whose process holds stale AWS credentials recovers on its own instead of failing every turn. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] df48724 Description read[DESCRIPTION-READ] da733ec688171b9381f9329ee5aabdb0187ff2cfbd1ce932c9e939c509578ae4 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Opus 5.5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS When credentials are truly expired, a recycled Claude backend now tells the user its "session" was lost, and the provider's "refresh your AWS credentials" advice is dropped. Watch
Suggestions
[DESIGN-REVIEWED] df48724 Description read[DESCRIPTION-READ] 2fac0101f39e9c1d46255ba11b872e86e4257858b0471db52570b9676d06448c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING — a stale-credential error now hides the provider's "refresh your AWS credentials" message FINDING — src/kiro_crew/llm_helpers.py:157 — [OPUS-REVIEWED] df48724 |
|
Thank you @akushonkamen for this fix and the thorough tests. Your branch had fallen far behind main, and its remaining CI reds were main-side, so we carried it into a same-repo PR: #20973. It keeps your commit and authorship unchanged; the only edits are a docs conflict resolution in |
Problem / Motivation
Goal: A long-lived Claude backend whose cached AWS credentials go stale is recycled into a fresh process on its next turn, the same way a lost session already is.
With
agent.acp_backendset toclaude(Claude Code on Bedrock, credentials loaded by a credential helper), a long-lived dashboard session started failing every turn with:Nothing replaced the process: every turn re-asked the same backend and drew the same answer, and only a manual conversation reset recovered it (a chat opened fresh in the same window worked on the first try, so the credential helper itself was healthy).
Why it matters
The backend process is alive and answers, so nothing marks it dead and the dead-provider eviction never fires — but its credential cache is stale, so every retry on that one process draws the identical failure while a fresh process loads fresh credentials. Until a user resets the conversation by hand, an interactive chat fails every turn, and a monitor loop burns all five of its stand-down cycles for nothing.
Not a goal
auth_requiredpath: the Kiro sign-in card remains whatauth_requiredoffers; this error deliberately does not take it (it would sign in the wrong identity for a harness that loads its own credentials).What changed (motivation → approach → change)
AcpError(transient=False)namingcloud_credential_error; no surface recycles the process._raise_acp_error(src/kiro_crew/acp/transport_errors.py) classified the frame from its message prose only and never read the structurederrorKindin the frame'sdata—cloud_credential_errorhad zero references in the tree. The formatter's unknown-shape branch pastes the dict's repr onto the message (that is the: {'errorKind': ...}tail in the transcript), but nothing read it back._error_kind_from_frame()readserrorKindONLY from a mappingdatafield; when it namescloud_credential_error,_raise_acp_errorsets the newAcpError.credential_reloadtag and pinstransient = False— the prose ends in "try again", so a future rewording that trips a retry-hint pattern must never point a retry ladder back at the one process that cannot reload. Deliberately NOTauth_required: that tag offers the Kiro sign-in card, which signs in the wrong identity for a harness that loads its own credentials (harness parity H6).llm_helpers.acp_error_is_session_not_found(src/kiro_crew/llm_helpers.py) now also reads the tag, so every surface that resets and retries once for a lost session — dashboard chat, subagent runs, channel turns, the Slack gateway and handler — recycles this backend through the replace-and-run-again seam it already owns, under the same one-reload-per-turn budget and the same no-replay-after-tool-activity rule. Nothing in those seams reads which of the two answers routed in.docs/system-specs/common/error-handling.mdanddocs/system-specs/modules/session.md.Diff: 2 src files (+ tests, + spec pages).
Alternatives considered and rejected:
auth_required: its remedy is the Kiro sign-in card; for Claude-on-Bedrock the credentials come from the harness's own helper, so the card signs in the wrong identity.Backwards compatibility
Compatible:
credential_reloaddefaults toFalseon every existingAcpError, and a frame whosedatacarries noerrorKind, a stringifieddata, an unknown kind, or a non-dict error tags nothing and behaves exactly as before — the tests lock each of those shapes.Removes nothing: the change only adds a tag on one frame shape and routes it down an existing path.
Tests
test/test_acp_error_surface.py(TestCloudCredentialErrorNamesAFreshProcess, 16 cases) locks: the frame is taggedcredential_reloadthrough both raise paths (wait + dispatch); the verdict is pinnedtransient=False;auth_requiredstays unset; no other tag lands; the provider's remedy prose survives formatting; anunknownkind, a kind name in prose or stringifieddata, a missingdatafield and a non-dict error all stay untagged.test/test_session_not_found_surfaces.py(2 cases) locks the consumption seam end to end through the real Slack handler: a stale-credential answer drops the binding once and replays once on a fresh process (the replay's answer lands), and a second credential failure ends the turn on the give-up text instead of looping.Red → green, re-run after rebasing onto current main (origin/main @ 3a23b67, branch head df48724;
.venv-scoutinterpreter,-q -o addopts="" --timeout=60 -n0):git checkout origin/main -- src/), the same 18 new tests:timeout 120 .venv-scout/bin/python -m pytest <18 new tests> -q→ 12 failed / 6 passed (thecredential_reloadattribute does not exist).test_acp_error_surface/test_session_not_found_surfaces/test_chat_session_not_found_recovery/test_chat_runner_composition_contract/test_acp_credential_propagation_retry/test_connection_recovery_window/test_acp_provider_retry_hint/test_recovery_l1_chat_runner/test_channel_prompt_busy/test_cron_acp_retry/test_model_unentitled_meta) → 488 passed, 26.83s.test_acp_client.py+test_run_bg_oneliner.py+test_subagent_turn_resilience.py+test_dashboard_chat.py -k 'not slow'→ 1702 passed / 2 deselected, 81.82s.timeout 300 pytest src/kiro_crew/acp -q -o addopts="" --timeout=60collects 0 tests — there are notest_*files undersrc/kiro_crew/acp, so that path is not a test set in this repo; the two batches above are the regression evidence.scripts/check_harness_parity.py,check_comment_history.py(forced scope,COMMENT_HISTORY_BASE_REF=origin/main),check_subprocess_encoding.py,check_sync_io_in_async.py,check_brand_name.py,check_black_formatting.py(repo-pinned black 26.3.1)..github/coverage-baselines/backend.txt, and every added src line is exercised by the tests above.Manual verification
N/A — unit coverage drives
_raise_acp_errorand the real Slack handler end to end, and the fix adds no UI. (Reproducing the issue verbatim needs a long-lived Bedrock-backed process whose credentials rotate underneath it; the tests simulate it through the real raise path.)Related Issues
Fixes #19262
Pattern harvest
Rule candidate: review-prompt
Pattern: an ACP frame's structured
data.errorKindis the provider's stable verdict — error classification that reads only the message prose misses machine-readable kinds, and a prose match can contradict them (a "try again" tail on a non-retryable answer).Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license. Prepared with a coding agent; every line reviewed and all checks above re-run by the author (also disclosed in the claim comment on #19262).