Skip to content

fix(acp): recycle a claude backend that reports cloud_credential_error - #20973

Merged
iamwhatever merged 2 commits into
mainfrom
replace/19293
Oct 17, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
replace/19293

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A long-lived Claude backend whose cached AWS credentials go stale is recycled into a fresh process on its next turn, the same way a lost session already is.

With agent.acp_backend set to claude (Claude Code on Bedrock, credentials loaded by a credential helper), a long-lived dashboard session started failing every turn with:

[AcpError] Internal error: API Error: Could not load AWS credentials · Could not load credentials from any providers. Check or refresh your AWS credentials and try again.: {'errorKind': 'cloud_credential_error'}

Nothing replaced the process: every turn re-asked the same backend and drew the same answer, and only a manual conversation reset recovered it (a chat opened fresh in the same window worked on the first try, so the credential helper itself was healthy).

Why it matters

The backend process is alive and answers, so nothing marks it dead and the dead-provider eviction never fires — but its credential cache is stale, so every retry on that one process draws the identical failure while a fresh process loads fresh credentials. Until a user resets the conversation by hand, an interactive chat fails every turn, and a monitor loop burns all five of its stand-down cycles for nothing.

Not a goal

  • A general retry ladder for transient provider errors — the verdict is pinned non-transient and the existing retry-hint classifier is untouched.
  • Changing the auth_required path: the Kiro sign-in card remains what auth_required offers; this error deliberately does not take it (it would sign in the wrong identity for a harness that loads its own credentials).
  • Any change to the one-retry-per-turn budget or the no-replay-after-tool-activity rule — this error rides them unchanged.

What changed (motivation → approach → change)

  • Symptom: every turn of the stuck session fails with an untagged AcpError(transient=False) naming cloud_credential_error; no surface recycles the process.
  • Root cause: _raise_acp_error (src/kiro_crew/acp/transport_errors.py) classified the frame from its message prose only and never read the structured errorKind in the frame's data — cloud_credential_error had zero references in the tree. The formatter's unknown-shape branch pastes the dict's repr onto the message (that is the : {'errorKind': ...} tail in the transcript), but nothing read it back.
  • Change 1 — tag the frame structurally: new _error_kind_from_frame() reads errorKind ONLY from a mapping data field; when it names cloud_credential_error, _raise_acp_error sets the new AcpError.credential_reload tag and pins transient = False — the prose ends in "try again", so a future rewording that trips a retry-hint pattern must never point a retry ladder back at the one process that cannot reload. Deliberately NOT auth_required: that tag offers the Kiro sign-in card, which signs in the wrong identity for a harness that loads its own credentials (harness parity H6).
  • Change 2 — one consumption seam: llm_helpers.acp_error_is_session_not_found (src/kiro_crew/llm_helpers.py) now also reads the tag, so every surface that resets and retries once for a lost session — dashboard chat, subagent runs, channel turns, the Slack gateway and handler — recycles this backend through the replace-and-run-again seam it already owns, under the same one-reload-per-turn budget and the same no-replay-after-tool-activity rule. Nothing in those seams reads which of the two answers routed in.
  • The specs record the new answer on the existing pages: docs/system-specs/common/error-handling.md and docs/system-specs/modules/session.md.

Diff: 5 src files (+ tests, + spec pages). The review follow-up adds dashboard/chat_runner.py, channel.py and slack/handler.py (wording and the provider's remedy on give-up).

Alternatives considered and rejected:

  • Reuse auth_required: its remedy is the Kiro sign-in card; for Claude-on-Bedrock the credentials come from the harness's own helper, so the card signs in the wrong identity.
  • Match the prose ("Could not load credentials"): it is the provider's wording and can drift, and it ends in "try again", which the retry-hint classifier reads — a prose match would flip between retry and reload as wording changes. The structured field is the provider's own stable answer.

Backwards compatibility

Compatible: credential_reload defaults to False on every existing AcpError, and a frame whose data carries no errorKind, a stringified data, an unknown kind, or a non-dict error tags nothing and behaves exactly as before — the tests lock each of those shapes.

Removes nothing: the change only adds a tag on one frame shape and routes it down an existing path.

Tests

test/test_acp_error_surface.py (TestCloudCredentialErrorNamesAFreshProcess, 16 cases) locks: the frame is tagged credential_reload through both raise paths (wait + dispatch); the verdict is pinned transient=False; auth_required stays unset; no other tag lands; the provider's remedy prose survives formatting; an unknown kind, a kind name in prose or stringified data, a missing data field and a non-dict error all stay untagged.

test/test_session_not_found_surfaces.py (2 cases) locks the consumption seam end to end through the real Slack handler: a stale-credential answer drops the binding once and replays once on a fresh process (the replay's answer lands), and a second credential failure ends the turn on the give-up text instead of looping.

Red → green, re-run after rebasing onto current main (origin/main @ 3a23b67, branch head df48724; .venv-scout interpreter, -q -o addopts="" --timeout=60 -n0):

  • Fix reverted (git checkout origin/main -- src/), the same 18 new tests: timeout 120 .venv-scout/bin/python -m pytest <18 new tests> -q → 12 failed / 6 passed (the credential_reload attribute does not exist).
  • Fix applied, same command → 18 passed in 0.90s.
  • Regression batch A — the 11 related files (test_acp_error_surface / test_session_not_found_surfaces / test_chat_session_not_found_recovery / test_chat_runner_composition_contract / test_acp_credential_propagation_retry / test_connection_recovery_window / test_acp_provider_retry_hint / test_recovery_l1_chat_runner / test_channel_prompt_busy / test_cron_acp_retry / test_model_unentitled_meta) → 488 passed, 26.83s.
  • Regression batch B — test_acp_client.py + test_run_bg_oneliner.py + test_subagent_turn_resilience.py + test_dashboard_chat.py -k 'not slow' → 1702 passed / 2 deselected, 81.82s.
  • The issue's literal command timeout 300 pytest src/kiro_crew/acp -q -o addopts="" --timeout=60 collects 0 tests — there are no test_* files under src/kiro_crew/acp, so that path is not a test set in this repo; the two batches above are the regression evidence.
  • Local gates all exit 0: scripts/check_harness_parity.py, check_comment_history.py (forced scope, COMMENT_HISTORY_BASE_REF=origin/main), check_subprocess_encoding.py, check_sync_io_in_async.py, check_brand_name.py, check_black_formatting.py (repo-pinned black 26.3.1).
  • Not run: the CI coverage-baselines full backend lane (not reproducible locally). Neither changed src file is listed in .github/coverage-baselines/backend.txt, and every added src line is exercised by the tests above.

Replacement branch, rebased on main 0f98bd9: every test/test_*.py that references transport_errors, llm_helpers, acp_error_is_session_not_found or the two spec pages, plus the campaign facade inventory tests (133 files) -> 10100 passed, 1 skipped. black/isort/flake8 clean on the touched files; feature-map gate passes.

Manual verification

N/A — unit coverage drives _raise_acp_error and the real Slack handler end to end, and the fix adds no UI. (Reproducing the issue verbatim needs a long-lived Bedrock-backed process whose credentials rotate underneath it; the tests simulate it through the real raise path.)

Related Issues

Fixes #19262
Replaces #19293

This is a same-repo carry of @akushonkamen's #19293. Fork CI could not go green on its own: the reds were main-side (test_app_cron_timeout_manifest, fixed on main by #19280) plus the Coverage cascade, and the branch had fallen far behind main. The commit is cherry-picked with @akushonkamen's authorship; all credit for the fix is theirs. A second commit answers the maintainer review (see Review history below) and also condenses the two new comments in acp/transport_errors.py so the file stays under the 2000-line cap (1999 lines). The other edit is the conflict resolution in docs/system-specs/common/error-handling.md: main's text is kept and the fork's paragraph is added to the Lost backend session bullet. The touched code still lives in acp/transport_errors.py and llm_helpers.py on current main, so nothing was re-homed and no facade/member inventory changed. Closed #19469 (@neilieva) fixed the same issue and suggested credential-specific retry/give-up wording; the review below asked for the same, and the second commit adds it.

Review history

@buluoray's review at 00e2910ac (CHANGES_REQUESTED). Answered in d0cd7861c, rebased on main 32dddfc85:

Point Answer
Blocking: a broken helper (expired SSO, revoked role) loses the provider's remedy on dashboard and channel llm_helpers.credential_give_up_text (llm_helpers.py:374) carries the provider message. Dashboard give-up uses it (chat_runner.py:16986); channel give-up uses it through _lost_give_up_text (channel.py:1190, :1110), fed by _stream_task (channel.py:1856). Slack gateway already appended the exception, unchanged.
Blocking (ideally): credential-specific retry wording CREDENTIAL_RELOAD_RETRY_NOTICE (llm_helpers.py:401) on dashboard and channel; dashboard log line and persisted partial reply name the credential cause (chat_runner.py:16940).
Blocking: test that a replay failure surfaces the AWS remedy test_broken_credential_helper_ends_on_the_providers_remedy (test/test_chat_session_not_found_recovery.py:179), test_channel_broken_credential_helper_ends_on_the_providers_remedy (test/test_session_not_found_surfaces.py:278). Both red with src/ reverted, green with the fix.
Follow-up 1: transient = False pin untested Kept the pin; test_credential_frame_stays_terminal_when_the_prose_reads_as_a_retry (test/test_acp_error_surface.py:842) uses "Please try again" prose, which trips _RE_5XX_HINT.
Follow-up 2: dashboard path untested Both dashboard tests above drive _run_chat end to end, plus test_stale_credentials_recycle_once_then_land (:213) for the healthy-helper case.
Follow-up 3: predicate name mislabels the cause Added the sibling cause predicate acp_error_needs_fresh_credentials (llm_helpers.py:364), used for wording on dashboard and channel. The routing predicate keeps its name to avoid touching its nine call sites in this PR.
Not traced in the review: Slack handler give-up It had the same gap (a constant line, no exception), so its give-up now uses credential_give_up_text too (slack/handler.py); test_slack_stale_credential_error_second_failure_is_not_a_loop asserts the AWS remedy lands (red without the handler change).
Scope of the wording (deferred on purpose) The other readers of the predicate (slack/gateway.py, slack/gateway_runtime/cron_turn.py, slack/gateway_runtime/cron_failure.py, subagent_manager/run.py) keep the lost-session lead-in but already append the exception, so the provider's remedy still shows there. Moving every reader to one llm_helpers chooser is a follow-up.
Follow-up 4: claude-code-provider.md silent New "Stale cloud credentials" section (docs/system-specs/modules/claude-code-provider.md:87), naming errorKind as the trigger and that any backend emitting it is recycled.

Tests at d0cd7861c (TMPDIR=/tmp/r20973 PYTHONPATH=src pytest -o addopts=""): the four touched/related files -> 156 passed; the 13-file regression batch (error surface, SNF surfaces, dashboard SNF recovery, composition contract, credential propagation, connection recovery, retry hint, L1 chat runner, channel prompt busy, cron ACP retry, turn error arms, llm_helpers) -> 781 passed; facade inventories + test_dashboard_chat.py -k 'not slow' -> 2163 passed. black/isort/flake8 clean; line-cap, harness-parity, feature-map, comment-history, sync-io, brand-name and docs-lint gates exit 0.

Pattern harvest

Rule candidate: review-prompt
Pattern: an ACP frame's structured data.errorKind is the provider's stable verdict — error classification that reads only the message prose misses machine-readable kinds, and a prose match can contradict them (a "try again" tail on a non-retryable answer).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license. Prepared with a coding agent; every line reviewed and all checks above re-run by the author (also disclosed in the claim comment on #19262).

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d0cd7861cbb5160f8613bfbb25d60078f7a1ef59 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of d0cd7861cbb5160f8613bfbb25d60078f7a1ef59 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The recycle fix is enough to clear the reported defect. The credential-specific wording added on three surfaces goes beyond the frozen Goal and adds a new ChannelAgent state field.

Not justified as shipped

  • Item 3 — rides along, oversized: the Goal is "recycled into a fresh process on its next turn". The description says the reported chat recovered on a fresh process ("a chat opened fresh in the same window worked"). So the reported user never reaches the new give-up branch, credential_give_up_text ("Fix them first. The provider said: …"). The "expired SSO login, a revoked role" case it handles is asserted by the description, not reported by anyone.
    Subtraction: delete credential_give_up_text and its three call sites (chat_runner.py elif _creds:, channel.py _lost_give_up_text, the slack/handler.py elif branch), and delete ChannelAgent._credential_error. The give-up path then falls back to SESSION_NOT_FOUND_GIVE_UP_TEXT.
    Clears when: the mechanism is removed, or a human amends the Goal.
  • Item 4 — rides along: the CREDENTIAL_RELOAD_RETRY_NOTICE retry notice and the credential wording in the log line and partial reply only relabel a path that already exists. Slack keeps SESSION_NOT_FOUND_RETRY_NOTICE.

What this change ships

Inventory (4 items) — 2 justified

Intent: a Claude-on-Bedrock chat whose cached credentials went stale recovers by itself instead of failing every turn until the user resets it by hand. This is a FIX.

  1. An errorKind: cloud_credential_error frame is tagged credential_reload and pinned non-transient — justified
  2. Every surface that already recovers a lost session (reset once and replay) now does the same for that tag — justified
  3. When the fresh process fails too, the give-up message quotes the provider's remedy instead of "could not reconnect" (dashboard, channel, Slack) — rides along, oversized: beyond Goal, unreported case
  4. Credential wording for the retry notice, log line and partial reply (dashboard, channel) — rides along

[FIRST-PRINCIPLES-REVIEWED] d0cd786

Description read

[DESCRIPTION-READ] 4ec38f4e5aaa01add7b7eef6ba8a13f52cc92b18181df7dabab6b526c13bee8c

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d0cd7861cbb5160f8613bfbb25d60078f7a1ef59 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check one live claude-agent-acp stale-credential frame, captured raw, to confirm errorKind sits under a data mapping. The new tag is keyed on exactly that shape.

[DESIGN-REVIEWED] d0cd786

Description read

[DESCRIPTION-READ] b5cc28ba2311ede5799e02d966ce80d89d405973e530e81862866ecbe2181d28

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d0cd7861cbb5160f8613bfbb25d60078f7a1ef59 — this comment is updated in place on each push.

Review details

No findings block this merge.

FINDING — src/kiro_crew/dashboard/chat_runner.py:16986 — a nested turn (_run_chat(..., _prompt_depth=1) from the /prompts get expansion) that hits cloud_credential_error for the first time skips the retry branch (_prompt_depth == 0) and falls into elif _creds:. The user then reads "still could not load its cloud credentials after a restart. Fix them first", though no fresh process ever tried and the finally-block reset means a plain resend would work → Fix: make it elif _creds and slot._session_not_found_retry_used: so the credential give-up line only shows after the one replay was actually used.

[OPUS-REVIEWED] d0cd786

Verdict parsed from the review's SHA-scoped output markers for commit d0cd7861cbb5160f8613bfbb25d60078f7a1ef59.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d0cd7861cbb5160f8613bfbb25d60078f7a1ef59: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d0cd7861cbb5160f8613bfbb25d60078f7a1ef59 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d0cd786

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d0cd7861cbb5160f8613bfbb25d60078f7a1ef59: <one-sentence reason>

@iamwhatever
iamwhatever marked this pull request as ready for review October 15, 2026 23:32
@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 15, 2026 23:32
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 15, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 00e2910ac0f9f3c582c74c239f57aa05ba3159a2.

The detection and routing are sound. The structured data.errorKind read is narrow, the reset is bounded, and the PR adds no new kill path. What blocks is the case the PR doesn't cover: a credential helper that is actually broken (expired SSO, revoked role). On the dashboard and channel surfaces, that user stops seeing the provider's only actionable instruction.

What I verified

  • Detection is structural and narrow. _error_kind_from_frame reads errorKind only when data is a mapping and the value is a string (src/kiro_crew/acp/transport_errors.py:1862-1880). The tag is set once, at the end of _raise_acp_error, after every other tag (transport_errors.py:1996-1998). Some frames take the early raise for AcpPromptBusy or the binding mismatch, so they stay untagged. For those the code fails closed: the behaviour is the same as on main.
  • The consumer is the one existing predicate. acp_error_is_session_not_found stays scoped to AcpError and adds only the tag check (src/kiro_crew/llm_helpers.py:356-361). All nine call sites I found (dashboard chat_runner.py:16797, channel.py:1834, slack/gateway.py ×5, slack/handler.py:2419, subagent_manager/run.py:1168) are existing reset-and-retry-once seams. Each keeps its own one-shot flag (for example slot._session_not_found_retry_used, info._session_not_found_retry_used) and its tool-activity gate, so a persistent failure can't loop. Per user turn the cost is at most one extra process recycle.
  • No new kill path. The recycle goes through SessionManager.reset (src/kiro_crew/session.py:2739) and the existing teardown and kill funnel. Nothing in the diff signals a process directly.
  • Harness scoping. The PR doesn't check harness identity. The trigger is the provider's own errorKind, and in-tree docs name claude-agent-acp as the only adapter that emits errorKind (transport_errors.py:695-699, docs/system-specs/modules/acp-client.md:2260). If another backend ever emitted the same kind, it would get the same bounded reset. I'm treating that as acceptable rather than as a gap, but see the follow-ups.

Mutations, one at a time, each restored with a clean git diff afterwards, run against test/test_acp_error_surface.py and test/test_session_not_found_surfaces.py:

Mutation Result
Drop err.credential_reload = True 6 red: test_credential_frame_is_tagged_for_a_process_reload ×2, test_credential_frame_does_not_offer_the_kiro_sign_in_card ×2, both test_slack_stale_credential_error_*
Make acp_error_is_session_not_found ignore the tag 2 red: both test_slack_stale_credential_error_*
Also match the kind from stringified data or message prose 2 red: test_kind_name_in_prose_is_not_read ×2
Drop the err.transient = False pin all green (see follow-ups)

Suites at head: test_acp_error_surface.py, test_session_not_found_surfaces.py, test_chat_session_not_found_recovery.py and test_acp_provider_retry_hint.py give 134 passed.

CI at this head (latest run per check name): 84 success, 15 skipped. Three earlier runs of Automated Rule Check, GPT 6.1 Review and Sensitive Change Review show as cancelled. Each was superseded by a later run of the same name that succeeded. There are no prior human reviews.

Blocking

A genuinely broken credential helper now loses the provider's remedy on the dashboard and channel surfaces. The PR's premise is a healthy helper and a stale per-process cache, and for that case the fix works. For a broken helper, the fresh process fails too:

  • On the dashboard, the first failure shows SESSION_NOT_FOUND_RETRY_TEXT ("The agent lost this chat's session — reconnecting…", src/kiro_crew/dashboard/chat_runner.py:718). It also persists "error: session not found" as the partial reply (chat_runner.py:16810).
  • The replay fails, and the user gets SESSION_NOT_FOUND_GIVE_UP_TEXT (chat_runner.py:720-723, appended at chat_runner.py:16848). That text says to start a new chat, and a new chat fails the same way.
  • The provider's text, "Check or refresh your AWS credentials", never reaches the user. On main, this frame fell through to the generic error branch, which showed it.
  • channel.py:1104 has the same shape: a constant give-up line with no exception text.
  • The Slack gateway is fine: it appends the exception to its give-up text (slack/gateway.py:3236).

Smallest fix: when getattr(exc, "credential_reload", False) is set, have the dashboard and channel give-up lines carry the provider message, the way the Slack gateway already does. Ideally also use credential-specific retry wording instead of "lost this chat's session". Add a test that a credential failure on the replay surfaces the AWS remedy text. The Design Review lane flagged the same thing.

Follow-ups (non-blocking)

  1. The transient = False pin is untested. Removing it leaves every test green, because the current prose ("…and try again.") doesn't match _RE_5XX_HINT, so the base verdict is already non-transient. test_credential_frame_verdict_is_terminal (test/test_acp_error_surface.py:831) passes either way. If the pin is meant to guard against a future rewording, the test needs a frame whose prose does trip the hint, for example "please try again". Otherwise, drop the pin.
  2. The dashboard path for this error has no test. Only the Slack handler is driven end to end (test/test_session_not_found_surfaces.py:154, :183). The dashboard is the surface from the original report.
  3. The predicate's name no longer matches what it answers. acp_error_is_session_not_found, plus the "Backend lost the session" warning and the "error: session not found" transcript record, mislabel the credential cause in logs and history. Consider a sibling predicate, such as "needs a fresh process", with a cause check for wording.
  4. docs/system-specs/modules/claude-code-provider.md doesn't mention this recovery, though the PR title scopes it to the Claude backend. Add a line there, including that the trigger is the provider's errorKind, not a harness check, so a reader knows another backend emitting the same kind would be recycled too.

Not verified

  • I didn't run the full suite or any coverage lane.
  • I didn't reproduce against a live Bedrock-backed claude process.
  • I didn't trace the Slack handler (slack/handler.py) give-up wording beyond the gateway's.
  • I didn't verify that kiro-cli never emits errorKind: cloud_credential_error. The claim above rests on the in-tree docs only.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 16, 2026
A long-lived Claude-on-Bedrock backend whose AWS credentials went stale
answered every later turn with a frame whose data names
errorKind cloud_credential_error. _raise_acp_error read only prose, so
the frame became a plain AcpError(transient=False) with no tag, and no
surface replaced the stuck process: each turn re-asked the same one and
drew the same answer, and only a conversation reset recovered it (a chat
started fresh in the same window worked, so the helper was healthy).

_raise_acp_error now reads the kind STRUCTURALLY from a mapping data
field (_error_kind_from_frame) and tags AcpError.credential_reload,
pinning the verdict non-transient because the prose ends in 'try again'
and must never point a retry ladder back at the process that cannot
reload. Deliberately not auth_required: that tag offers the Kiro
sign-in card, which signs in the wrong identity for a harness that
loads its own credentials (harness parity H6).

llm_helpers.acp_error_is_session_not_found now also reads the tag, so
every surface that resets and retries once for a lost session --
dashboard chat, subagent runs, channel turns, the Slack gateway and
handler -- recycles this backend the same way, with the same
one-retry-per-turn budget and no-replay-after-tool-activity rule.

Fixes #19262

Signed-off-by: akushonkamen <akushonkamen@163.com>
iamwhatever pushed a commit that referenced this pull request Oct 16, 2026
Review follow-up for #20973. A credential helper that is really broken
(an expired SSO login, a revoked role) fails the fresh process as well.
The dashboard and channel then ended on the lost-session give-up line
("start a new chat"), which fails the same way, and the provider's
"Check or refresh your AWS credentials" instruction never reached the
user.

- llm_helpers: acp_error_needs_fresh_credentials (the cause half of
  acp_error_is_session_not_found), CREDENTIAL_RELOAD_RETRY_NOTICE and
  credential_give_up_text, which carries the provider's message.
- dashboard chat_runner: the retry notice, log line and persisted
  partial reply name the credential cause; the give-up line carries the
  provider's remedy instead of SESSION_NOT_FOUND_GIVE_UP_TEXT.
- channel: _stream_task records the provider text on the agent; the
  retry notice and the give-up post use it.
- Tests: broken-helper and stale-then-lands paths on the dashboard, the
  broken-helper path through run_channel_agent, and a reworded frame
  ("Please try again") that only the transient=False pin keeps terminal.
- Specs: error-handling, session and claude-code-provider pages.

Also condenses the two comments on the credential_reload tag so
transport_errors.py stays at 1999 lines.

Co-authored-by: akushonkamen <akushonkamen@163.com>
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 16, 2026
@iamwhatever

iamwhatever commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @buluoray for the careful review and the mutation table. Rebased on main and answered in d0cd7861c (one new commit on top of @akushonkamen's, which is unchanged).

Blocking: a broken helper loses the provider's remedy. Fixed on both surfaces you named.

  • New llm_helpers.credential_give_up_text(exc) carries the provider message verbatim.
  • Dashboard: when the replay fails on credentials too, the give-up line is now that text plus "Then send your message again", not "start a new chat" (chat_runner.py:16986).
  • Channel: _stream_task records the provider text on the agent (channel.py:1856), and the give-up post uses it (_lost_give_up_text, channel.py:1190).
  • Slack handler (you noted you had not traced it): it had the same gap, a constant give-up with no exception text. It now uses the same credential_give_up_text. The gateway, cron and subagent paths already append the exception, so they are unchanged.
  • Retry wording: CREDENTIAL_RELOAD_RETRY_NOTICE ("could not load its cloud credentials — restarting it…") on both surfaces. The dashboard log line and persisted partial reply also name the credential cause.
  • Tests: test_broken_credential_helper_ends_on_the_providers_remedy (dashboard, through _run_chat) and test_channel_broken_credential_helper_ends_on_the_providers_remedy (through run_channel_agent) assert the AWS remedy text lands. Both are red with src/ reverted and green with the fix.

Follow-up 1 (pin untested). Kept the pin and added test_credential_frame_stays_terminal_when_the_prose_reads_as_a_retry, whose prose ("Please try again") trips _RE_5XX_HINT. It fails if the pin is dropped.

Follow-up 2 (dashboard untested). The two dashboard tests above, plus test_stale_credentials_recycle_once_then_land for the healthy-helper case.

Follow-up 3 (predicate name). Added the sibling cause predicate acp_error_needs_fresh_credentials, which the wording reads. I left acp_error_is_session_not_found named as is, to keep its nine call sites out of this PR. Happy to rename it in a follow-up if you prefer.

Follow-up 4 (provider spec). New "Stale cloud credentials" section in docs/system-specs/modules/claude-code-provider.md. It says the trigger is the provider's errorKind, not a harness check, so any backend emitting it is recycled too.

Local runs and the full point-by-point map are in the PR body under "Review history".

@iamwhatever
iamwhatever requested a review from buluoray October 16, 2026 16:25
iamwhatever pushed a commit that referenced this pull request Oct 16, 2026
Review follow-up for #20973. A credential helper that is really broken
(an expired SSO login, a revoked role) fails the fresh process as well.
The dashboard and channel then ended on the lost-session give-up line
("start a new chat"), which fails the same way, and the provider's
"Check or refresh your AWS credentials" instruction never reached the
user.

- llm_helpers: acp_error_needs_fresh_credentials (the cause half of
  acp_error_is_session_not_found), CREDENTIAL_RELOAD_RETRY_NOTICE and
  credential_give_up_text, which carries the provider's message.
- dashboard chat_runner: the retry notice, log line and persisted
  partial reply name the credential cause; the give-up line carries the
  provider's remedy instead of SESSION_NOT_FOUND_GIVE_UP_TEXT.
- channel: _stream_task records the provider text on the agent; the
  retry notice and the give-up post use it.
- Tests: broken-helper and stale-then-lands paths on the dashboard, the
  broken-helper path through run_channel_agent, and a reworded frame
  ("Please try again") that only the transient=False pin keeps terminal.
- Specs: error-handling, session and claude-code-provider pages.

Also condenses the two comments on the credential_reload tag so
transport_errors.py stays at 1999 lines.

Co-authored-by: akushonkamen <akushonkamen@163.com>
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 16, 2026
iamwhatever pushed a commit that referenced this pull request Oct 16, 2026
Review follow-up for #20973. A credential helper that is really broken
(an expired SSO login, a revoked role) fails the fresh process as well.
The dashboard and channel then ended on the lost-session give-up line
("start a new chat"), which fails the same way, and the provider's
"Check or refresh your AWS credentials" instruction never reached the
user.

- llm_helpers: acp_error_needs_fresh_credentials (the cause half of
  acp_error_is_session_not_found), CREDENTIAL_RELOAD_RETRY_NOTICE and
  credential_give_up_text, which carries the provider's message.
- dashboard chat_runner: the retry notice, log line and persisted
  partial reply name the credential cause; the give-up line carries the
  provider's remedy instead of SESSION_NOT_FOUND_GIVE_UP_TEXT.
- channel: _stream_task records the provider text on the agent; the
  retry notice and the give-up post use it.
- Tests: broken-helper and stale-then-lands paths on the dashboard, the
  broken-helper path through run_channel_agent, and a reworded frame
  ("Please try again") that only the transient=False pin keeps terminal.
- Specs: error-handling, session and claude-code-provider pages.

Also condenses the two comments on the credential_reload tag so
transport_errors.py stays at 1999 lines.

Co-authored-by: akushonkamen <akushonkamen@163.com>
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 16, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 16, 2026
Review follow-up for #20973. A credential helper that is really broken
(an expired SSO login, a revoked role) fails the fresh process as well.
The dashboard and channel then ended on the lost-session give-up line
("start a new chat"), which fails the same way, and the provider's
"Check or refresh your AWS credentials" instruction never reached the
user.

- llm_helpers: acp_error_needs_fresh_credentials (the cause half of
  acp_error_is_session_not_found), CREDENTIAL_RELOAD_RETRY_NOTICE and
  credential_give_up_text, which carries the provider's message.
- dashboard chat_runner: the retry notice, log line and persisted
  partial reply name the credential cause; the give-up line carries the
  provider's remedy instead of SESSION_NOT_FOUND_GIVE_UP_TEXT.
- channel: _stream_task records the provider text on the agent; the
  retry notice and the give-up post use it.
- Tests: broken-helper and stale-then-lands paths on the dashboard, the
  broken-helper path through run_channel_agent, and a reworded frame
  ("Please try again") that only the transient=False pin keeps terminal.
- Specs: error-handling, session and claude-code-provider pages.

Also condenses the two comments on the credential_reload tag so
transport_errors.py stays at 1999 lines.

Co-authored-by: akushonkamen <akushonkamen@163.com>
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 16, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) October 17, 2026 05:26

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at d0cd786. This supersedes my earlier CHANGES_REQUESTED at 00e2910: the blocker is fixed.

Verified:

  • Dashboard: chat_runner.py:16986-16994 now ends a credential failure on the provider remedy instead of SESSION_NOT_FOUND_GIVE_UP_TEXT; the retry notice, log line and partial reply name the cause.
  • Channel: channel.py:1856 records the provider text, and _lost_give_up_text (:1190-1193) and the retry notice (:1285-1289) use it. Slack carries the remedy too (slack/handler.py:2469-2474).
  • Still bounded: one recycle per turn on every surface, through the existing SessionManager.reset seam.
  • Mutations: disabling the dashboard branch reddens test_broken_credential_helper_ends_on_the_providers_remedy; forcing the channel give-up to the generic text reddens test_channel_broken_credential_helper_ends_on_the_providers_remedy; disabling the Slack branch reddens test_slack_stale_credential_error_second_failure_is_not_a_loop.

Follow-up, tracked in #21491 (non-blocking by owner decision): credential_give_up_text (llm_helpers.py:374-385) puts the provider text into the dashboard row (chat_runner.py:16992) and the channel post (channel.py:1193) without the redact_exfiltration_urls / redact_credentials pass every other provider-error row gets (chat_runner.py:17985-17986), and that main applied to this error. Slack is unaffected. Low likelihood, but please pick it up soon: redact inside credential_give_up_text and add a test.

Non-blocking: Slack still posts SESSION_NOT_FOUND_RETRY_NOTICE on a credential recycle, while the dashboard and channel use CREDENTIAL_RELOAD_RETRY_NOTICE.

On the First Principles CONCERNS: keeping the provider remedy is intended; main showed it for this error, and it is the only actionable line when the helper itself is broken.

CI at d0cd786: 84 passed, 14 skipped, 0 failed; PR Readiness success. Auto-merge is on, so this approval may merge it.

@iamwhatever
iamwhatever merged commit 1f0a19a into main Oct 17, 2026
104 of 112 checks passed
@iamwhatever
iamwhatever deleted the replace/19293 branch October 17, 2026 05:48
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 17, 2026
@Pearcekieser Pearcekieser mentioned this pull request Oct 18, 2026
5 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

acp: a Claude backend that can no longer load AWS credentials keeps failing every turn, because cloud_credential_error never recycles the process

4 participants