Skip to content

fix(chat): re-load a chat whose live backend lost its session (release/0.8.0) - #16412

Merged
bolichen97 merged 2 commits into
release/0.8.0from
fix/bp16221-session-not-found-0.8
Oct 3, 2026
Merged

bolichen97 merged 2 commits into
release/0.8.0from
fix/bp16221-session-not-found-0.8

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: On release/0.8.0, a chat whose backend still runs but has lost the chat's session recovers on its own, instead of failing every prompt until the gateway restarts.

A live backend process can answer every session/prompt with Session not found for the backend session id the chat is bound to. The process is up, so the claim-time dead-provider eviction never runs, and the error is not retry-eligible, so the turn ends on a bare error card. Every later prompt, Continue included, hits the same binding. Only a gateway restart re-attaches the chat.

Why it matters

The chat looks healthy but is unusable, and the only remedy users found was restarting the whole gateway, which interrupts every other chat, cron and subagent on it. The reports are from the 0.8 line.

Not a goal

What changed

One commit, a git cherry-pick -x of #16221 (cf32ce7d27). It applied cleanly: every helper it calls (_session_stop_generation_for, _has_user_queued_followup, _should_suppress_requeue, _queue_recovery, build_recovery_requeue, ResetCause.CONNECTION_LOST) exists on release/0.8.0 with the same shape, so no adaptation was needed.

  • src/kiro_crew/llm_helpers.py: acp_error_is_session_not_found(exc), scoped to AcpError.
  • src/kiro_crew/dashboard/chat_runner.py: a new AcpError arm sets needs_session_reset, so the turn's finally resets the session. That drops the live binding and keeps the session-map entry, so the next claim spawns a fresh runtime that session/loads the same backend id. The turn is retried once behind "⟳ The agent lost this chat's session — reconnecting…". The queued replay is dropped at the drain and at the consume seam when a Stop, a follow-up or steer, or a rebind landed meanwhile.
  • src/kiro_crew/dashboard/state.py: the one-shot guard _session_not_found_retry_used, re-armed only by a landed turn, plus the replay's queue id, Stop counters and binding. A repeat loss ends on a clear error, never a loop.
  • Second commit 290725c (cherry-pick -x of main fix(chat): let a resolved Stop veto the lost-session reconnect replay #16548, Fixes Lost-session reconnect replays a prompt whose Stop already resolved #16547): the lost-session requeue gate is _should_suppress_requeue(slot) or _stop_pressed(), so a Stop that already resolved to idle vetoes the replay. New test test_stop_resolved_before_the_error_never_queues_a_replay fails without it (1 failed / 8 passed) and passes with it (9/9).

Backwards compatibility

Compatible. Only an AcpError whose text says the session was not found takes the new path; every other error is handled as before.

Tests

  • test/test_chat_session_not_found_recovery.py (from fix(chat): re-load a chat whose live backend lost its session #16221): 8 passed.
  • Revert check: disabling the new AcpError arm fails 3 of the 8 (retry-once-then-lands, twice-ends-without-looping, Stop-during-reset purges the replay). The other 5 are the 4 predicate cases and the consume-seam test, which seeds the replay directly.
  • Neighbouring suites (test_acp_error_surface, test_acp_provider_retry_hint, test_dashboard_chat and the other files that cover the sibling one-shot recoveries): 976 passed.

Manual verification

N/A — unit coverage drives the turn through a fake provider that answers Session not found.

Related Issues

Backport of #16221 to release/0.8.0.

Pattern harvest

Rule candidate: review-prompt
Pattern: a recovery that requeues a turn at the queue head across an awaited teardown must snapshot its queue id, Stop counters and session binding, and re-check them at the drain and the consume seam.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

A backend process that is still alive can answer every prompt with
"Session not found" for the backend session id the chat is bound to. The
process is up, so nothing marks it dead and the claim-time dead-provider
eviction never runs; the error was not retry-eligible either, so the turn
ended on a bare error card and every later prompt (Continue included) hit the
same binding. Only a gateway restart re-attached the chat.

Treat that answer as a lost session: reset the slot's session (which keeps
the session-map entry, so the next claim spawns a fresh runtime that
session/loads the same backend id, falling back to a new session with history
replay only when the load fails), show a short reconnect notice, and retry
the turn once. A one-shot guard, re-armed only by a landed turn, ends a
repeat loss on a clear error instead of looping.

Co-authored-by: Joe Guo <zejiangg@amazon.com>
(cherry picked from commit cf32ce7)
@iamwhatever
iamwhatever requested a review from a team as a code owner October 3, 2026 07:31
@iamwhatever
iamwhatever requested review from bolichen97 and removed request for a team October 3, 2026 07:31
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 290725c4840ff5bae11a5976f45d2397e5a1555d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm no further follow-up fixes to #16221 landed on main after c692a07 — the backport should carry the complete set.

[DESIGN-REVIEWED] 290725c

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 290725c4840ff5bae11a5976f45d2397e5a1555d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify that the ~4,500 diff lines outside the four declared files are base commits #16385/#16284/#16396 swept in by the merge diff, not undeclared branch work.

Not justified as shipped

What this change ships

Inventory (6 items) — 5 justified

Intent: FIX — a chat bound to a backend session the live process no longer holds recovers on its own instead of erroring until a gateway restart (backport of #16221 + #16548 to release/0.8.0).

  1. A lost-session chat reconnects on a fresh runtime re-loading the same backend id and retries the turn once, behind a visible notice — justified
  2. A repeat loss ends on a clear give-up error, never a retry loop (one-shot _session_not_found_retry_used, re-armed only by a landed turn) — justified
  3. A Stop, follow-up/steer, or session rebind landing during the awaited reset cancels the queued replay at the drain and consume seams, with a cancelled notice — justified
  4. A Stop that already resolved to idle also vetoes the replay (second commit, Fixes Lost-session reconnect replays a prompt whose Stop already resolved #16547, test fails without it) — justified
  5. New predicate acp_error_is_session_not_found in llm_helpers.py, beside its sibling acp_error_is_transient; one consumer, singular form — justified
  6. Default-agent / onboarding-import / skills hunks across ~20 files — rides along: base commits fix(agents): let an installed agent become the default agent (release/0.8.0) #16385/fix(onboarding): let the import chapter be left when its flag will not save (release/0.8.0) #16284/fix(skills): a project always-skill or a misencoded SKILL.md no longer fails the session (release/0.8.0) #16396 via the merge diff, not this PR's work

[FIRST-PRINCIPLES-REVIEWED] 290725c

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ human override accepted

Human judgment by @bolichen97 overrides the GPT 5.6 finding for 290725c4840ff5bae11a5976f45d2397e5a1555d; the recorded reason is authoritative for this commit.

[GPT-OVERRIDE] 290725c

This comment is updated in place on each push.

The model was not re-run because an authorized human decision supersedes it.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 290725c4840ff5bae11a5976f45d2397e5a1555d: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 290725c4840ff5bae11a5976f45d2397e5a1555d — this comment is updated in place on each push.

Review details

Nothing blocks — one advisory: the new lost-session give-up arm is a no-requeue exit that skips the per-turn retry-budget refresh every other terminal arm performs.

FINDING — src/kiro_crew/dashboard/chat_runner.py:19275 — the else: slot.append("error", SESSION_NOT_FOUND_GIVE_UP_TEXT, ...) arm (and the _should_suppress_requeue(slot) or _stop_pressed(): pass arm above it) ends the turn with nothing re-queued, but unlike the terminal else at 20303–20315 and the two sibling drop arms at 19453–19456 / 19558–19561 it leaves _transient_5xx_retries, _infra_retries, _fallback_candidate_idx and _fallback_walked at their partially-spent values — and nothing refreshes them at turn entry (state.py:3362 resets only at slot construction; the only other reset is the landed block at 18591) — so a cycle that already spent a transient retry or an L1 capacity wait and then loses the session twice leaves the next genuine user turn with a short 5xx ladder, an inflated backoff seed, a restore probe suppressed by the stale walk index and the slot reading "recovering" on the health panel while idle → Fix: in both non-enqueueing arms of the lost-session branch add slot._transient_5xx_retries = 0, slot._infra_retries = 0, slot._fallback_candidate_idx = 0, slot._fallback_walked = [], as the sibling no-requeue exits do.

[OPUS-REVIEWED] 290725c

Verdict parsed from the review's SHA-scoped output markers for commit 290725c4840ff5bae11a5976f45d2397e5a1555d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 290725c4840ff5bae11a5976f45d2397e5a1555d: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 3, 2026
The session-not-found arm queued its one reconnect replay unless
_should_suppress_requeue(slot) was true, which only sees a Stop still in
flight. A Stop that already resolved back to idle moved only the Stop
counters, and the replay snapshots are taken after it, so the drain-side
veto read 'not stopped' and the cancelled prompt ran again. Pair the
gate with _stop_pressed(), as every sibling requeue gate does.

Fixes #16547

(cherry picked from commit c692a07)
@bolichen97

Copy link
Copy Markdown
Contributor

mechanism: requeue gate now pairs _should_suppress_requeue with _stop_pressed (new regression test)

  • A completed Stop can still replay the cancelled prompt span=db37e66c9e96

Outcome: fixed in 290725c (cherry-pick -x of main #16548, Fixes #16547).
The session-not-found arm's gate is now _should_suppress_requeue(slot) or _stop_pressed(), the same pairing as the sibling requeue gates, so a Stop that already resolved to idle (only the Stop generations moved) vetoes the replay before it is queued.
Regression: new test_stop_resolved_before_the_error_never_queues_a_replay fails without the fix (the cancelled prompt reaches the provider twice; 1 failed / 8 passed) and passes with it (9/9).

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ✅ PASS

UX-level review of 290725c4840ff5bae11a5976f45d2397e5a1555d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

Verify "ℹ️ Session reconnect cancelled — nothing was run." doesn't read as the chat staying broken — the reset already happened and the next prompt reconnects.

Suggestions

  • SESSION_NOT_FOUND_CANCELLED_TEXT in chat_runner.py: replace "Session reconnect cancelled — nothing was run." with "ℹ️ Your message was not re-sent — the chat will reconnect on your next message.", since only the replay was cancelled, not the recovery.

[UX-REVIEWED] 290725c

@bolichen97

Copy link
Copy Markdown
Contributor

/ai-review override gpt 290725c: The only blocking finding is at src/kiro_crew/config/loader.py:6466 (the default-agent healer from #16385), a file this PR does not change (its diff is chat_runner.py, state.py, llm_helpers.py and one test), so it is out of scope here and is tracked in #16555 against the code that owns it.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 290725c4840ff5bae11a5976f45d2397e5a1555d.

The only blocking finding is at src/kiro_crew/config/loader.py:6466 (the default-agent healer from #16385), a file this PR does not change (its diff is chat_runner.py, state.py, llm_helpers.py and one test), so it is out of scope here and is tracked in #16555 against the code that owns it.

This decision applies only to this commit. A new push requires a new judgment.

@bolichen97
bolichen97 merged commit facd28b into release/0.8.0 Oct 3, 2026
24 of 32 checks passed
@bolichen97
bolichen97 deleted the fix/bp16221-session-not-found-0.8 branch October 3, 2026 16:58
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants