Skip to content

fix(chat): let a resolved Stop veto the runtime-death and busy requeues - #16767

Closed
leonlaiyc wants to merge 2 commits into
kirodotdev:mainfrom
leonlaiyc:fix/requeue-gates-see-resolved-stop
Closed

leonlaiyc wants to merge 2 commits into
kirodotdev:mainfrom
leonlaiyc:fix/requeue-gates-see-resolved-stop

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A prompt the user stopped is not replayed on a fresh runtime when the runtime dies or answers busy after that Stop has already resolved.

#16548 (for #16547) fixed this on _run_chat's lost-session arm. Both the Design and the First Principles reviews of that PR named the same gap on the sibling arms of the same except chain. Two of those arms are still open on main @ 68782006c:

  • the runtime-death arm (except AcpProcessDied, the shared/own-death requeue), and
  • the busy / pipe-death arm (except AcpError when _retry_eligible: AcpPromptBusy, "already in progress", "process exited", "not running").

Each one gates its replay on _should_suppress_requeue(slot) alone, and that check sees only a Stop that is still in flight. A Stop that pressed and snapped back to idle before the error reached the handler leaves _stopping False and moves only the Stop counters. The arm therefore posts "⟳ Connection lost — retrying…" or "⟳ Session busy — retrying…" and queues the stopped prompt (for example "delete the old branches") for verbatim replay.

Why it matters

Stop is how a user withdraws a request. Here the withdrawn request runs anyway, on a fresh runtime, with no further prompt. That covers any side effect the stopped prompt asked for. The trigger needs nothing unusual: a kiro-cli death or a busy answer that lands just after a Stop, which is when a stopped process is most likely to be torn down.

Not a goal

What changed (motivation → approach → change)

Symptom: a stopped prompt is replayed after a runtime death or busy answer. Root cause: these three gates read only the in-flight Stop. The replay goes through _queue_recovery, which takes no Stop snapshot of its own, so nothing downstream can see a Stop that has already resolved. _stop_pressed() reads the slot's and the session's Stop generations live against the turn's entry snapshot, which is exactly that signal.

Change, src/kiro_crew/dashboard/chat_runner.py:

  • The runtime-death gate is now _should_suppress_requeue(slot) or _stop_pressed(), with a comment saying why only this gate can see a resolved Stop.
  • The busy / pipe-death gate gets the same pairing.

Change, src/kiro_crew/dashboard/chat_turn/recovery.py (commit 2, from the First Principles review of f984716):

  • _requeue_after_prompt_busy, the gate behind the except PromptBusyExhaustedError arm, takes _run_chat's _stop_pressed callable and applies the same pairing. It is passed the callable, not a value, so the Stop is read live at the gate, as _stop_pressed requires.

All three arms' pass branch is the existing behaviour for a Stop still in flight, so a resolved Stop now ends the turn the same way an in-flight one already did.

Each reviewer-named sibling, graded on its own:

Arm Replay of a resolved-Stop prompt on main Here
lost session (acp_error_is_session_not_found) fixed by #16548 unchanged
runtime death (AcpProcessDied) yes: test red on main fixed
busy / pipe death (_retry_eligible AcpError) yes: red for both the pipe-death and the AcpPromptBusy shape fixed
busy retries exhausted (PromptBusyExhaustedError → _requeue_after_prompt_busy) yes: red for the PromptBusyExhaustedError shape fixed (commit 2)
transient 5xx (acp_error_is_transient, not _turn_emitted) no: after _recovery_delay it re-reads _should_suppress_requeue(slot) or _stop_pressed() and drops the replay with TRANSIENT_GIVE_UP_TEXT unchanged

The 5xx arm's bare pre-backoff check only means a resolved Stop shows a "retrying" card, waits out one backoff, and then reports the give-up. That arm also resets its per-turn budgets on the drop path, so changing its gate is a separate change rather than one more line.

Backwards compatibility

Compatible: the only behaviour change is that these three arms no longer requeue after a Stop the user already pressed; with no Stop, each arm requeues exactly as before (pinned by the positive-control tests below).

Removes nothing: no route, key, field or function is removed or renamed; three existing gate conditions gain a second disjunct (_requeue_after_prompt_busy gains the keyword it reads it from; its only caller is _run_chat).

Tests

New test/test_chat_requeue_resolved_stop.py drives the real _run_chat through a stubbed ACP client, in the same harness shape as #16548's test_chat_session_not_found_recovery.py. There are four error shapes: AcpProcessDied, a pipe-death AcpError("… process exited …"), AcpPromptBusy and PromptBusyExhaustedError. Each runs twice:

  • test_the_arm_requeues_the_prompt_when_nothing_was_stopped[*]: the positive control. With no Stop, each arm posts its "retrying" card and replays the prompt once (calls == [msg, msg]). This proves the harness reaches the arm under test. It passes on main and with the fix.
  • test_a_stop_resolved_before_the_error_never_queues_a_replay[*]: the Stop generation moves and _stopping returns to False inside the stream before it raises. The test asserts one call, an empty queue and no "retrying" card. Red on main for all four shapes (calls holds the replay), green with the fix. The PromptBusyExhaustedError case was re-checked at commit 2: with only the two src/ hunks reverted, it fails and the other seven pass. With them applied, all 8 pass.

Adjacent suites, run locally with the fix (-n 6): test_dashboard_chat.py, test_gateway_pipe_death.py, test_session_idle_busy_guard.py, test_runtime_death_is_a_process_event.py, test_recovery_l1_chat_runner.py, test_chat_runner_composition_contract.py, test_chat_threads.py, test_refusal_inband_notice.py, test_chat_session_not_found_recovery.py and the new file: 1275 passed, 3 skipped, 4 failed. The 4 failures are test_chat_threads.py symlink tests that fail identically with this change reverted (OSError: [WinError 1314], because creating a symlink needs elevation on the local Windows box); they never reach _run_chat. black --check and flake8 are clean on the changed files.

At commit 2 (346f8ad), all of these suites were re-run: test_chat_requeue_resolved_stop.py, test_chat_runner_composition_contract.py, test_chat_runner_coverage.py, test_dashboard_chat.py, test_gateway_pipe_death.py, test_llm_helpers.py, test_recovery_l1_chat_runner.py and test_runtime_death_is_a_process_event.py. Result: 1630 passed, 0 failed. flake8, black --check, isort --check-only and mypy on chat_runner.py + chat_turn/recovery.py are clean, and so is the comment-history gate.

Manual verification

N/A — unit coverage sufficient: the defect is a gate condition inside _run_chat's except chain, and the new tests drive that exact chain end to end with the Stop counters moved the way the dashboard Stop handler moves them, plus a positive control per arm.

Related Issues

Follow-up to #16548 (for #16547): the sibling arms both of that PR's AI reviews named and the PR left out of scope. No separate issue was filed.

Pattern harvest

Rule candidate: review-prompt
Pattern: a requeue/replay gate that reads only the in-flight Stop state (_should_suppress_requeue) without the Stop generation (_stop_pressed) misses a Stop that already resolved to idle; every replay arm that has no post-wait re-read needs both.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

@leonlaiyc
leonlaiyc requested a review from a team as a code owner October 4, 2026 12:07
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 4, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/requeue-gates-see-resolved-stop branch from 88249b4 to f984716 Compare October 4, 2026 12:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 4, 2026
@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of c23f4e2cf9befd507c8f5d9030b5f4d83b1d7aef via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] c23f4e2

Description read

[DESCRIPTION-READ] aa0116294a666a63b6e9ee1237b812276730eedac3eb21fc702c770ee61206e2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of c23f4e2cf9befd507c8f5d9030b5f4d83b1d7aef via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

This patches three gates, but the root cause the description names, that "_queue_recovery takes no Stop snapshot of its own", stays in place, and other bare gates still replay a stopped prompt.

Not justified as shipped

  • Item 1 — symptom-level: the description says "nothing downstream can see a Stop that has already resolved" and blames _queue_recovery. The diff then adds or _stop_pressed() to the gate of each arm. In the same except chain on base, I grepped for _should_suppress_requeue(slot) replay gates that have no _stop_pressed(). Three remain that the PR neither fixes nor grades: chat_runner.py:17341 (image-history replay), :17551 (fallback-model swap) and :18097 (poison canary). The image arm's stop snapshot (:17403) is taken after the error, so a Stop that resolved before the error is invisible to it.
    Subtraction: replace the per-arm or _stop_pressed() disjuncts with one turn-entry Stop check inside _queue_recovery. That is one site, and the _stop_pressed parameter on _requeue_after_prompt_busy goes away.
    Clears when: either _queue_recovery vetoes a resolved Stop itself, or the description grades :17341, :17551 and :18097 as not replaying a resolved-Stop prompt.

What this change ships

Inventory (4 items) — 3 justified

Intent: a prompt the user stopped must not re-run on a fresh runtime after a runtime death or busy error. FIX.

  1. The runtime-death, pipe-death and busy arms no longer replay a prompt whose Stop has already resolved — symptom-level: the cause is in _queue_recovery, and 3 sibling gates are unfixed
  2. The busy-retries-exhausted arm no longer replays a prompt whose Stop has already resolved — justified
  3. These arms no longer show a "retrying" card after a resolved Stop — justified
  4. New regression suite with a positive control for each arm — justified

[FIRST-PRINCIPLES-REVIEWED] c23f4e2

Description read

[DESCRIPTION-READ] af305f564ace1370c02ce991cd8dbc670a0dbf927640b85b6e942711d1b03117

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed c23f4e2cf9befd507c8f5d9030b5f4d83b1d7aef via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] c23f4e2

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed c23f4e2cf9befd507c8f5d9030b5f4d83b1d7aef via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c23f4e2

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 4, 2026
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

CI note on head 346f8ad83:

No push for these. A re-run after #16801 lands should clear shard 7.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 4, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/requeue-gates-see-resolved-stop branch from 346f8ad to da6213f Compare October 6, 2026 08:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 6, 2026
leonlaiyc and others added 2 commits October 6, 2026 20:52
kirodotdev#16548 paired _stop_pressed() with _should_suppress_requeue() on _run_chat's
lost-session arm. Two sibling arms of the same except chain still gated
their replay on the suppress check alone, which sees only a Stop still in
flight: the AcpProcessDied requeue and the busy / pipe-death AcpError
requeue. A Stop that pressed and resolved back to idle before the error
reached the handler moved only the Stop counters, so the stopped prompt
was queued for verbatim replay on a fresh runtime. _queue_recovery takes
no Stop snapshot of its own, so the gate is the only place that can see
it.

The transient-5xx arm, named alongside these by the kirodotdev#16548 reviews, already
re-reads _stop_pressed() after its backoff and drops the replay, so it is
left as is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The First Principles review of f984716 counted one more gate of the same
class: `_requeue_after_prompt_busy`, behind the `PromptBusyExhaustedError`
arm, still checked only for a Stop in flight. It now takes `_run_chat`'s
live `_stop_pressed` read and pairs it with `_should_suppress_requeue`,
like the other three arms. The regression test gains that error shape:
red with only the source hunks reverted, green with them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leonlaiyc
leonlaiyc force-pushed the fix/requeue-gates-see-resolved-stop branch from da6213f to c23f4e2 Compare October 6, 2026 12:54
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 6, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks @leonlaiyc for catching that the two sibling requeue arms still replay a resolved Stop — this is a real gap. Because this touches core-repo files, I've carried your change forward as a main-repo PR so it can run the full check + review suite: #18401 (Fixes #18400). It ports your fix verbatim (both chat_runner arms + _requeue_after_prompt_busy threaded with _stop_pressed) plus your test, rebased onto current main, and credits you as co-author. Closing this one in favor of #18401. Appreciate the follow-up to #16548!

@iamwhatever iamwhatever closed this Oct 9, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 9, 2026
iamwhatever pushed a commit that referenced this pull request Oct 10, 2026
…e arms

Follow-up to merged #16548, which gave only the lost-session arm the
`_stop_pressed()` read. The runtime-death arm, the prompt-busy / pipe-death
arm, and the post-busy-exhaustion requeue helper still gate on
`_should_suppress_requeue(slot)` alone, so a Stop that pressed and resolved
back to idle before the error reached the handler (invisible to the suppress
check, and `_queue_recovery` takes no Stop snapshot of its own) still queues
the stopped prompt for replay on a fresh runtime.

Pair each remaining arm's gate with `_stop_pressed()`, the same pairing #16548
used, and thread `_stop_pressed` into `_requeue_after_prompt_busy` so the
recovery helper can see it too. Adds a parametrised test covering all four
arms with a positive control (no Stop replays once) and the stopped case
(resolved Stop queues nothing).

Replaces #16767 by @leonlaiyc.

Co-authored-by: Leon <276034861+leonlaiyc@users.noreply.github.com>
iamwhatever pushed a commit that referenced this pull request Oct 10, 2026
…e arms

Follow-up to merged #16548, which gave only the lost-session arm the
`_stop_pressed()` read. The runtime-death arm, the prompt-busy / pipe-death
arm, and the post-busy-exhaustion requeue helper still gate on
`_should_suppress_requeue(slot)` alone, so a Stop that pressed and resolved
back to idle before the error reached the handler (invisible to the suppress
check, and `_queue_recovery` takes no Stop snapshot of its own) still queues
the stopped prompt for replay on a fresh runtime.

Pair each remaining arm's gate with `_stop_pressed()`, the same pairing #16548
used, and thread `_stop_pressed` into `_requeue_after_prompt_busy` so the
recovery helper can see it too. Adds a parametrised test covering all four
arms with a positive control (no Stop replays once) and the stopped case
(resolved Stop queues nothing).

Replaces #16767 by @leonlaiyc.

Co-authored-by: Leon <276034861+leonlaiyc@users.noreply.github.com>
iamwhatever pushed a commit that referenced this pull request Oct 11, 2026
…e arms

Follow-up to merged #16548, which gave only the lost-session arm the
`_stop_pressed()` read. The runtime-death arm, the prompt-busy / pipe-death
arm, and the post-busy-exhaustion requeue helper still gate on
`_should_suppress_requeue(slot)` alone, so a Stop that pressed and resolved
back to idle before the error reached the handler (invisible to the suppress
check, and `_queue_recovery` takes no Stop snapshot of its own) still queues
the stopped prompt for replay on a fresh runtime.

Pair each remaining arm's gate with `_stop_pressed()`, the same pairing #16548
used, and thread `_stop_pressed` into `_requeue_after_prompt_busy` so the
recovery helper can see it too. Adds a parametrised test covering all four
arms with a positive control (no Stop replays once) and the stopped case
(resolved Stop queues nothing).

Replaces #16767 by @leonlaiyc.

Co-authored-by: Leon <276034861+leonlaiyc@users.noreply.github.com>
chenmingwei23 pushed a commit that referenced this pull request Oct 11, 2026
…e arms (#18401)

* fix(chat): let a resolved Stop veto the runtime-death and busy requeue arms

Follow-up to merged #16548, which gave only the lost-session arm the
`_stop_pressed()` read. The runtime-death arm, the prompt-busy / pipe-death
arm, and the post-busy-exhaustion requeue helper still gate on
`_should_suppress_requeue(slot)` alone, so a Stop that pressed and resolved
back to idle before the error reached the handler (invisible to the suppress
check, and `_queue_recovery` takes no Stop snapshot of its own) still queues
the stopped prompt for replay on a fresh runtime.

Pair each remaining arm's gate with `_stop_pressed()`, the same pairing #16548
used, and thread `_stop_pressed` into `_requeue_after_prompt_busy` so the
recovery helper can see it too. Adds a parametrised test covering all four
arms with a positive control (no Stop replays once) and the stopped case
(resolved Stop queues nothing).

Replaces #16767 by @leonlaiyc.

Co-authored-by: Leon <276034861+leonlaiyc@users.noreply.github.com>

* fix(chat): let a resolved Stop veto the image-history and poisoned-reset replays

Self-review follow-up. The image-history recovery and the poisoned-conversation
reset arm still gated their replay on _should_suppress_requeue(slot) alone, and
each takes its Stop snapshot (_image_stop_gen, _stop_gen_before_canary) after a
resolved Stop already moved the counter, so the stopped prompt was replayed on
a fresh session. Pair both gates with _stop_pressed() (the image arm's
owed-completion half is unchanged). Also give the transient-5xx arm's first
gate its own resolved-Stop branch, so a resolved Stop no longer flashes a
"retrying" card the post-backoff re-read withdraws. That branch ends the turn
with no requeue, so it resets the same four per-turn budgets the post-backoff
drop arm resets (_transient_5xx_retries, _infra_retries,
_fallback_candidate_idx, _fallback_walked); otherwise the attempt it just
counted shortens the next turn's 5xx ladder.

Tests: image-history, transient-5xx and poisoned-reset cases in
test/test_chat_requeue_resolved_stop.py; each stopped case fails on base. A
budget case checks a resolved Stop on a transient 5xx leaves all four fields at
zero for the next turn. The file patches the runner's _recovery_delay seam
instead of a process-wide asyncio.sleep (Determinism contract / flake-pattern
ratchet).

---------

Co-authored-by: cixuuz <26782336+cixuuz@users.noreply.github.com>
Co-authored-by: Leon <276034861+leonlaiyc@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants