Skip to content

fix(acp): recycle the backend once when it can no longer load cloud credentials - #19469

Closed
neilieva wants to merge 1 commit into
kirodotdev:mainfrom
neilieva:fix/acp-cloud-credential-recycle
Closed

neilieva wants to merge 1 commit into
kirodotdev:mainfrom
neilieva:fix/acp-cloud-credential-recycle

Conversation

@neilieva

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Claude backend that can no longer load its cloud credentials gets one fresh process instead of failing every turn until someone resets the session by hand.

A long-lived Claude Code backend on Bedrock can end up unable to load AWS credentials while a newly started process loads them fine. Every prompt then fails with errorKind: cloud_credential_error, and every surface sends the next turn to the same process. In the case in #19262, a monitor loop failed 20 times over about 5.5 hours, while a new chat opened in the same window worked first time.

Why it matters

Anyone running the Claude backend with a credential helper hits this when the in-process credentials go stale. Interactive chats keep failing until the user resets them. Loops burn every cycle until they stand down after 5 failures. Subagents, cron jobs and Slack threads fail the same way.

Not a goal

  • Detecting stale credentials before a turn fails, or refreshing them inside a running process.
  • Changing the one-retry budget, or the rule that a turn is not replayed after a tool ran.
  • Changing how any other error kind is classified.

What changed (motivation → approach → change)

The frame already says what went wrong: its data carries errorKind: "cloud_credential_error". On main, _raise_acp_error in acp/transport_errors.py ignored it, so the error was a plain non-retryable AcpError that nothing treated as a reason to replace the backend. The remedy is the same as for "session not found": the process answered, but only a fresh one can recover.

_raise_acp_error now sets credential_reload from that structured field only. llm_helpers.py adds acp_error_is_credential_reload (the tag, with a text fallback for an error that was re-wrapped) and acp_error_needs_fresh_backend (either cause). Every surface that recovers a lost session decides through the new predicate:

  • dashboard chat (dashboard/chat_runner.py)
  • subagent runs (subagent_manager/run.py, with the binding in subagent.py)
  • channel turns (channel.py)
  • cron, cron-sequence steps, the cron failure alert and subagent completion injection (slack/gateway.py)
  • Slack threads (slack/handler.py)

Dashboard chat and Slack show credential-specific retry and give-up text. The give-up tells the user to refresh their credentials, because a second failure means a restart did not help. The other surfaces already append the backend's own error, which names the credential failure. docs/system-specs/common/error-handling.md describes the rule.

Backwards compatibility

Compatible: session-not-found handling is unchanged. The new predicate matches everything the old one matched, and every surface keeps its retry budget and side-effect gates.

Removes nothing: a frame without the structured errorKind is classified exactly as before.

Tests

  • test_acp_cloud_credential_recycle.py:
    • The structured errorKind sets the tag. A different kind, no data, the kind only inside a string data, or only in the message does not.
    • acp_error_needs_fresh_backend matches a credential error, a re-wrapped credential error and a lost session, and nothing else.
    • Dashboard chat recycles the backend once and lands. On a second failure it ends on the refresh-your-credentials text.
  • test_session_not_found_surfaces.py runs every surface test twice, once with a lost session and once with the real credential frame. The Slack tests expect the wording for the cause under test. On main, 19 of the 23 credential variants fail; the other 4 check outcomes main already gets right, such as no replay after a tool ran.

Manual verification

N/A. The tests drive each surface with the adapter's real frame shape. The live failure needs credentials to go stale inside a running process, which I can't reproduce on demand.

Related Issues

Fixes #19262

Pattern harvest

Rule candidate: review-prompt
Pattern: an adapter frame's structured data field (such as errorKind) ignored in favour of matching message text.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@neilieva
neilieva requested review from a team and KiroCrewCaptain as code owners October 12, 2026 16:49
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision e7cb20cf4885ef0927ceccdba2638e733d976db9 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of e7cb20cf4885ef0927ceccdba2638e733d976db9 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that sessions.reset really kills the Claude adapter process and doesn't just drop the session. The fix only works if a fresh process reloads credentials, and every test mocks reset.

[DESIGN-REVIEWED] e7cb20c

Description read

[DESCRIPTION-READ] 71aca3778ee88a7669e78ebcc34227b3a7ea25a2eef8c156176ab3f36e24e1ba

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of e7cb20cf4885ef0927ceccdba2638e733d976db9 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The "cloud_credential_error" in str(exc) text fallback has no re-wrap site that needs it. The structured tag already covers the reported defect.

Not justified as shipped

  • Item 3 — inherited: the stated reason is "with a text fallback for an error that was re-wrapped". On base, raise AcpError\(\s*f?"[^"]*\{(exc|e|err)\} matches 2 re-wrap sites, both at slack/gateway.py:9837-9841. Both prefix SESSION_NOT_FOUND_NOT_REPLAYED_TEXT/SESSION_NOT_FOUND_GIVE_UP_TEXT, and the only check that reads the re-wrapped error (gateway.py:6169) excludes those texts anyway. So the fallback changes no outcome. It also contradicts the spec's "from that structured field only, never from message text".
    Subtraction: drop or "cloud_credential_error" in str(exc) from acp_error_is_credential_reload so it reads exc.credential_reload is True, and drop the re-wrapped test case.
    Clears when: a named re-wrap path that loses the tag and reaches acp_error_needs_fresh_backend with a different outcome, or the fallback is deleted.

What this change ships

Inventory (5 items) — 4 justified

Intent: a Claude/Bedrock backend whose credentials went stale gets one fresh process instead of failing every turn. This is a FIX (#19262).

  1. A frame carrying errorKind: cloud_credential_error now triggers the one fresh-backend recycle that session-not-found already gets — justified
  2. All 8 base acp_error_is_session_not_found recovery call sites now route through the combined predicate (I grepped all of them; no unfixed sibling) — justified
  3. The credential classification also matches on message text — rides along; the re-wrap it is meant for doesn't need it
  4. Dashboard chat and Slack show credential-specific retry text, and a give-up that says "refresh your credentials" — justified
  5. Log and persisted-partial labels name the credential cause — justified

[FIRST-PRINCIPLES-REVIEWED] e7cb20c

Description read

[DESCRIPTION-READ] 4dbd6d16310cd07988d1e68eeff3df614c62bfe50e79558fb9d5c28525c680cf

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed e7cb20cf4885ef0927ceccdba2638e733d976db9 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e7cb20c

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed e7cb20cf4885ef0927ceccdba2638e733d976db9 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e7cb20c

@neilieva
neilieva force-pushed the fix/acp-cloud-credential-recycle branch from ea1139c to a207501 Compare October 12, 2026 17:28
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 12, 2026
@neilieva

Copy link
Copy Markdown
Contributor Author
  • Item 3: duplicate CREDENTIAL_RELOAD_* constants in dashboard/chat_runner.py

fixed in a207501
dashboard/chat_runner.py no longer defines any CREDENTIAL_RELOAD_* constant. It imports CREDENTIAL_RELOAD_RETRY_NOTICE and CREDENTIAL_RELOAD_GIVE_UP_TEXT from llm_helpers, and the dashboard give-up adds its ❌ prefix inline.
The dashboard tests import the same constants from llm_helpers.

@github-actions github-actions Bot added readiness: checking Automated validation is still running fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@neilieva
neilieva force-pushed the fix/acp-cloud-credential-recycle branch from a207501 to 6c6c18e Compare October 12, 2026 21:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@neilieva
neilieva force-pushed the fix/acp-cloud-credential-recycle branch from 6c6c18e to 881e559 Compare October 13, 2026 00:32
@neilieva

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in channel.py, dashboard/chat_runner.py, llm_helpers.py, slack/gateway.py, subagent.py and acp/transport_errors.py since this head went green (ebe8a18 -> 315affc); CI re-running.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 13, 2026
…redentials

A Claude backend that could no longer load its AWS credentials answered every turn with errorKind cloud_credential_error, and every surface sent the next turn to the same process, so the session kept failing until the user reset it. The error is now tagged credential_reload from the frame's structured errorKind, and acp_error_needs_fresh_backend treats it like a lost session: dashboard chat, subagent runs, channel turns, cron and cron-sequence steps, subagent completion injection and Slack threads reset the session and retry once on a fresh runtime, with the existing one-retry budget and no replay after a tool ran. Dashboard chat and Slack say the credentials could not be loaded and, when the retry also fails, tell the user to refresh them. The error-handling spec describes the rule.

Fixes kirodotdev#19262
@neilieva
neilieva force-pushed the fix/acp-cloud-credential-recycle branch from 881e559 to e7cb20c Compare October 13, 2026 05:29
@neilieva

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/chat_runner.py and 11 import-overlapping files since this head went green (315affc -> de2d143); CI re-running.

@neilieva

Copy link
Copy Markdown
Contributor Author

Closing this in favor of #19293, which fixes #19262 the same way and was opened first. I missed it before opening this one.

One difference in case it's useful for #19293: this PR gave dashboard chat and Slack credential-specific text instead of the session-not-found wording. The retry notice said the agent could not load its cloud credentials, and the give-up told the user to refresh their credentials, since a restart can't fix an expired credential.

@neilieva neilieva closed this Oct 13, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

acp: a Claude backend that can no longer load AWS credentials keeps failing every turn, because cloud_credential_error never recycles the process

1 participant