Skip to content

fix(acp): recycle a claude backend that reports cloud_credential_error - #19293

Closed
akushonkamen wants to merge 1 commit into
kirodotdev:mainfrom
akushonkamen:fix/19262-credential-reload
Closed

akushonkamen wants to merge 1 commit into
kirodotdev:mainfrom
akushonkamen:fix/19262-credential-reload

Conversation

@akushonkamen

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A long-lived Claude backend whose cached AWS credentials go stale is recycled into a fresh process on its next turn, the same way a lost session already is.

With agent.acp_backend set to claude (Claude Code on Bedrock, credentials loaded by a credential helper), a long-lived dashboard session started failing every turn with:

[AcpError] Internal error: API Error: Could not load AWS credentials · Could not load credentials from any providers. Check or refresh your AWS credentials and try again.: {'errorKind': 'cloud_credential_error'}

Nothing replaced the process: every turn re-asked the same backend and drew the same answer, and only a manual conversation reset recovered it (a chat opened fresh in the same window worked on the first try, so the credential helper itself was healthy).

Why it matters

The backend process is alive and answers, so nothing marks it dead and the dead-provider eviction never fires — but its credential cache is stale, so every retry on that one process draws the identical failure while a fresh process loads fresh credentials. Until a user resets the conversation by hand, an interactive chat fails every turn, and a monitor loop burns all five of its stand-down cycles for nothing.

Not a goal

  • A general retry ladder for transient provider errors — the verdict is pinned non-transient and the existing retry-hint classifier is untouched.
  • Changing the auth_required path: the Kiro sign-in card remains what auth_required offers; this error deliberately does not take it (it would sign in the wrong identity for a harness that loads its own credentials).
  • Any change to the one-retry-per-turn budget or the no-replay-after-tool-activity rule — this error rides them unchanged.

What changed (motivation → approach → change)

  • Symptom: every turn of the stuck session fails with an untagged AcpError(transient=False) naming cloud_credential_error; no surface recycles the process.
  • Root cause: _raise_acp_error (src/kiro_crew/acp/transport_errors.py) classified the frame from its message prose only and never read the structured errorKind in the frame's data — cloud_credential_error had zero references in the tree. The formatter's unknown-shape branch pastes the dict's repr onto the message (that is the : {'errorKind': ...} tail in the transcript), but nothing read it back.
  • Change 1 — tag the frame structurally: new _error_kind_from_frame() reads errorKind ONLY from a mapping data field; when it names cloud_credential_error, _raise_acp_error sets the new AcpError.credential_reload tag and pins transient = False — the prose ends in "try again", so a future rewording that trips a retry-hint pattern must never point a retry ladder back at the one process that cannot reload. Deliberately NOT auth_required: that tag offers the Kiro sign-in card, which signs in the wrong identity for a harness that loads its own credentials (harness parity H6).
  • Change 2 — one consumption seam: llm_helpers.acp_error_is_session_not_found (src/kiro_crew/llm_helpers.py) now also reads the tag, so every surface that resets and retries once for a lost session — dashboard chat, subagent runs, channel turns, the Slack gateway and handler — recycles this backend through the replace-and-run-again seam it already owns, under the same one-reload-per-turn budget and the same no-replay-after-tool-activity rule. Nothing in those seams reads which of the two answers routed in.
  • The specs record the new answer on the existing pages: docs/system-specs/common/error-handling.md and docs/system-specs/modules/session.md.

Diff: 2 src files (+ tests, + spec pages).

Alternatives considered and rejected:

  • Reuse auth_required: its remedy is the Kiro sign-in card; for Claude-on-Bedrock the credentials come from the harness's own helper, so the card signs in the wrong identity.
  • Match the prose ("Could not load credentials"): it is the provider's wording and can drift, and it ends in "try again", which the retry-hint classifier reads — a prose match would flip between retry and reload as wording changes. The structured field is the provider's own stable answer.

Backwards compatibility

Compatible: credential_reload defaults to False on every existing AcpError, and a frame whose data carries no errorKind, a stringified data, an unknown kind, or a non-dict error tags nothing and behaves exactly as before — the tests lock each of those shapes.

Removes nothing: the change only adds a tag on one frame shape and routes it down an existing path.

Tests

test/test_acp_error_surface.py (TestCloudCredentialErrorNamesAFreshProcess, 16 cases) locks: the frame is tagged credential_reload through both raise paths (wait + dispatch); the verdict is pinned transient=False; auth_required stays unset; no other tag lands; the provider's remedy prose survives formatting; an unknown kind, a kind name in prose or stringified data, a missing data field and a non-dict error all stay untagged.

test/test_session_not_found_surfaces.py (2 cases) locks the consumption seam end to end through the real Slack handler: a stale-credential answer drops the binding once and replays once on a fresh process (the replay's answer lands), and a second credential failure ends the turn on the give-up text instead of looping.

Red → green, re-run after rebasing onto current main (origin/main @ 3a23b67, branch head df48724; .venv-scout interpreter, -q -o addopts="" --timeout=60 -n0):

  • Fix reverted (git checkout origin/main -- src/), the same 18 new tests: timeout 120 .venv-scout/bin/python -m pytest <18 new tests> -q → 12 failed / 6 passed (the credential_reload attribute does not exist).
  • Fix applied, same command → 18 passed in 0.90s.
  • Regression batch A — the 11 related files (test_acp_error_surface / test_session_not_found_surfaces / test_chat_session_not_found_recovery / test_chat_runner_composition_contract / test_acp_credential_propagation_retry / test_connection_recovery_window / test_acp_provider_retry_hint / test_recovery_l1_chat_runner / test_channel_prompt_busy / test_cron_acp_retry / test_model_unentitled_meta) → 488 passed, 26.83s.
  • Regression batch B — test_acp_client.py + test_run_bg_oneliner.py + test_subagent_turn_resilience.py + test_dashboard_chat.py -k 'not slow' → 1702 passed / 2 deselected, 81.82s.
  • The issue's literal command timeout 300 pytest src/kiro_crew/acp -q -o addopts="" --timeout=60 collects 0 tests — there are no test_* files under src/kiro_crew/acp, so that path is not a test set in this repo; the two batches above are the regression evidence.
  • Local gates all exit 0: scripts/check_harness_parity.py, check_comment_history.py (forced scope, COMMENT_HISTORY_BASE_REF=origin/main), check_subprocess_encoding.py, check_sync_io_in_async.py, check_brand_name.py, check_black_formatting.py (repo-pinned black 26.3.1).
  • Not run: the CI coverage-baselines full backend lane (not reproducible locally). Neither changed src file is listed in .github/coverage-baselines/backend.txt, and every added src line is exercised by the tests above.

Manual verification

N/A — unit coverage drives _raise_acp_error and the real Slack handler end to end, and the fix adds no UI. (Reproducing the issue verbatim needs a long-lived Bedrock-backed process whose credentials rotate underneath it; the tests simulate it through the real raise path.)

Related Issues

Fixes #19262

Pattern harvest

Rule candidate: review-prompt
Pattern: an ACP frame's structured data.errorKind is the provider's stable verdict — error classification that reads only the message prose misses machine-readable kinds, and a prose match can contradict them (a "try again" tail on a non-retryable answer).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license. Prepared with a coding agent; every line reviewed and all checks above re-run by the author (also disclosed in the claim comment on #19262).

A long-lived Claude-on-Bedrock backend whose AWS credentials went stale
answered every later turn with a frame whose data names
errorKind cloud_credential_error. _raise_acp_error read only prose, so
the frame became a plain AcpError(transient=False) with no tag, and no
surface replaced the stuck process: each turn re-asked the same one and
drew the same answer, and only a conversation reset recovered it (a chat
started fresh in the same window worked, so the helper was healthy).

_raise_acp_error now reads the kind STRUCTURALLY from a mapping data
field (_error_kind_from_frame) and tags AcpError.credential_reload,
pinning the verdict non-transient because the prose ends in 'try again'
and must never point a retry ladder back at the process that cannot
reload. Deliberately not auth_required: that tag offers the Kiro
sign-in card, which signs in the wrong identity for a harness that
loads its own credentials (harness parity H6).

llm_helpers.acp_error_is_session_not_found now also reads the tag, so
every surface that resets and retries once for a lost session --
dashboard chat, subagent runs, channel turns, the Slack gateway and
handler -- recycles this backend the same way, with the same
one-retry-per-turn budget and no-replay-after-tool-activity rule.

Fixes kirodotdev#19262

Signed-off-by: akushonkamen <akushonkamen@163.com>
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision df487242aa2112f075de1df12b4ae158fdd89072 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of df487242aa2112f075de1df12b4ae158fdd89072 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The non-transient pin does nothing today: on base this frame is already transient=False. Its only stated reason is a rewording that might happen later.

Not justified as shipped

  • Item 4 — inherited: the stated reason is a guess about the future, "a future rewording that trips a retry-hint pattern must never point a retry ladder back". The new tests show the verdict is already False on base: the 6 that pass on base include both test_credential_frame_verdict_is_terminal cases.
    Subtraction: delete err.transient = False in _raise_acp_error, plus the "pinned non-transient" sentences in its comment and in error-handling.md.
    Clears when: the line is deleted, or a test that fails on base shows this frame classified transient.
  • Item 2 — undeclared: the description never says a credential failure now shows SESSION_NOT_FOUND_RETRY_NOTICE ("The agent lost its session — reconnecting…") and the matching give-up text.

What this change ships

Inventory (4 items) — 2 justified

Intent: a Claude-on-Bedrock chat whose process holds stale AWS credentials recovers on its own instead of failing every turn. This is a FIX.

  1. A backend answering cloud_credential_error is replaced by a fresh process and the turn replays once, on all 10 acp_error_is_session_not_found call sites — justified
  2. That failure now shows the "lost its session — reconnecting" notice, or the give-up text — undeclared
  3. New AcpError.credential_reload tag, read by exactly one consumer (llm_helpers.py:325) — justified
  4. This frame's retry verdict is forced non-transient — inherited, has no effect today

[FIRST-PRINCIPLES-REVIEWED] df48724

Description read

[DESCRIPTION-READ] da733ec688171b9381f9329ee5aabdb0187ff2cfbd1ce932c9e939c509578ae4

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed df487242aa2112f075de1df12b4ae158fdd89072 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] df48724

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of df487242aa2112f075de1df12b4ae158fdd89072 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

When credentials are truly expired, a recycled Claude backend now tells the user its "session" was lost, and the provider's "refresh your AWS credentials" advice is dropped.

Watch

  • The second failure hides the real fix. The patch makes acp_error_is_session_not_found return True whenever getattr(exc, "credential_reload", False) is True. That sends this error down the lost-session path, which writes its own messages:

    • the retry notice is "The agent lost its session — reconnecting…";
    • if the credential helper itself is broken (for example, an expired SSO login), the fresh process fails too, and the turn ends on SESSION_NOT_FOUND_GIVE_UP_TEXT;
    • in slack/handler.py:2468 and dashboard/chat_runner.py:18303, that give-up text replaces the ❌ {e} that would otherwise have shown "Could not load AWS credentials… Check or refresh your AWS credentials".

    So the one case a reset cannot fix now shows "Could not reconnect the agent's session", and the user gets no hint that their credentials need refreshing. test_slack_stale_credential_error_second_failure_is_not_a_loop locks that wording in.
    Clears when: the give-up and retry text on the credential_reload branch keeps the provider's message (or names credentials), with a test asserting "AWS credentials" survives the second failure.

  • The frame shape is inferred, not captured. The tag only fires when data is a dict. The PR infers that from the formatted transcript tail : {'errorKind': ...}, not from a captured raw frame. If claude-agent-acp actually sends data as a string, the fix silently does nothing. test_kind_name_in_prose_is_not_read deliberately leaves that string case untagged.
    Clears when: a raw JSON-RPC frame from claude-agent-acp (a log line or adapter source) shows data as an object carrying errorKind.

Suggestions

  • Give the recycle path a reason-specific message, or keep a separate predicate that shares the same seams. Either way, the two different causes get their own text without duplicating the recovery machinery.

[DESIGN-REVIEWED] df48724

Description read

[DESCRIPTION-READ] 2fac0101f39e9c1d46255ba11b872e86e4257858b0471db52570b9676d06448c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed df487242aa2112f075de1df12b4ae158fdd89072 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING — a stale-credential error now hides the provider's "refresh your AWS credentials" message

FINDING — src/kiro_crew/llm_helpers.py:157 — return getattr(exc, "credential_reload", False) is True sends a credentials error that is really expired (an SSO session that ran out, so the fresh process fails the same way) to the session-loss give-up text. Users then see SESSION_NOT_FOUND_GIVE_UP_TEXT ("Could not reconnect this chat's session… start a new chat") in dashboard/chat_runner.py:18303 and slack/handler.py:2468 instead of the provider's "Check or refresh your AWS credentials" that ❌ {e} used to show, so the advice points at the wrong cause and the wrong fix → Fix: on the give-up and not-replayed branches, when exc.credential_reload is set, add str(exc) to the message, as slack/gateway.py and subagent_manager/run.py already do with f"{SESSION_NOT_FOUND_GIVE_UP_TEXT} {exc}". This edits files the PR does not touch, so it is advisory.

[OPUS-REVIEWED] df48724

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 13, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you @akushonkamen for this fix and the thorough tests. Your branch had fallen far behind main, and its remaining CI reds were main-side, so we carried it into a same-repo PR: #20973. It keeps your commit and authorship unchanged; the only edits are a docs conflict resolution in error-handling.md and a comment trim to keep transport_errors.py under the 2000-line cap. #20973 is green and awaiting review. Closing this one in favor of it; your branch is left in place. All credit for the fix is yours.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

acp: a Claude backend that can no longer load AWS credentials keeps failing every turn, because cloud_credential_error never recycles the process

2 participants