Skip to content

fix(chat): let a resolved Stop veto the runtime-death and busy requeue arms - #18401

Merged
chenmingwei23 merged 2 commits into
mainfrom
fix/requeue-see-resolved-stop-r23
Oct 11, 2026
Merged

chenmingwei23 merged 2 commits into
mainfrom
fix/requeue-see-resolved-stop-r23

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Stop the user already pressed can no longer be replayed on a fresh runtime after a mid-turn runtime failure.

_run_chat re-queues a turn when the provider dies or a prompt-busy retry runs out. Each arm must veto the replay when the user pressed Stop. Merged #16548 fixed only the lost-session arm. Six arms still gate on _should_suppress_requeue(slot) alone, so a Stop that resolved to idle before the error reached the handler (invisible to the suppress check) still queues the stopped prompt for replay.

Why it matters

Without this, a user who Stops a prompt and then hits an unrelated runtime death during the same teardown has the cancelled prompt silently replayed on the fresh runtime — the agent runs work the user explicitly stopped.

Not a goal

What changed (motivation → approach → change)

  • Symptom: a resolved Stop is replayed on runtime death or busy-exhaustion.
  • Root cause: _should_suppress_requeue reads only an in-flight Stop (_stopping True). A Stop that snapped back to idle leaves _stopping False and moves only the Stop counters, which _stop_pressed() reads. The replay path _queue_recovery takes no Stop snapshot of its own, so each arm's own gate is the only place a resolved Stop can be seen.
  • Change: pair each remaining arm's gate with _stop_pressed() — the runtime-death arm and the prompt-busy / pipe-death arm in chat_runner.py — and thread _stop_pressed into _requeue_after_prompt_busy (chat_turn/recovery.py) so the post-busy-exhaustion helper can read it too. The image-history recovery (image_format_unsupported, which discards the conversation and replays the prompt on a fresh session) gets the same pairing on its non-owed replay condition; its owed-completion half is unchanged. The poisoned-conversation reset arm (canary probe, conversation discard, verbatim replay) gets the same pairing: its _stop_gen_before_canary snapshot, like _image_stop_gen, is taken after a resolved Stop already moved the counter. The transient-5xx arm's first gate gets its own resolved-Stop branch, so a resolved Stop no longer flashes a "retrying" card the post-backoff re-read then withdraws. That branch ends the turn with no requeue, so it resets the same four per-turn budgets the post-backoff drop arm resets (_transient_5xx_retries, _infra_retries, _fallback_candidate_idx, _fallback_walked); otherwise the attempt it just counted would shorten the next turn's 5xx ladder. Every other arm uses the identical one-line pairing to the lost-session arm.

Why per arm and not inside _queue_recovery: _queue_recovery also queues an owed sub-agent completion (_requeue_owed_completion, _replay_owes_delivery), which by contract no Stop cancels. A blanket Stop veto there would drop a result the parent is still owed.

Deferred: the post-token transient helper (chat_turn/acp_recovery.py _will_recover) can still show a "resuming…" card that its own post-backoff re-read withdraws. It replays nothing, so it is a display-only flash left for a follow-up.

Backwards compatibility

Compatible: adds a disjunct to existing requeue gates and one keyword argument to an internal helper; nothing that works today stops working. A resolved Stop that previously replayed now (correctly) does not.

Tests

test/test_chat_requeue_resolved_stop.py (new): parametrised over six arms (runtime death, pipe death, prompt-busy, post-busy-exhaustion, image-history, transient-5xx), plus a two-cycle poisoned-reset pair (positive control discards and replays once; a Stop resolved on the last attempt discards nothing). The file patches the runner's _recovery_delay seam, not a process-wide asyncio.sleep. Each case runs twice — a positive control proving the arm replays once when nothing was stopped, and the stopped case proving a Stop resolved before the error queues no replay and shows no "retrying" card. A budget case checks that a resolved Stop on a transient 5xx leaves all four retry/fallback fields at zero for the next turn. 15/15 pass locally. All seven stopped cases (process-died, pipe-death, prompt-busy, prompt-busy-exhausted, image-history, transient-5xx, poisoned-reset) fail on the base commit (current main), and the budget case fails on the previous head.

Manual verification

N/A — the behaviour is a race between a Stop and a runtime failure during turn teardown, reproduced deterministically by the new unit tests rather than by hand.

Screenshots / video

N/A — no user-visible surface changes; the fix is a backend requeue gate.

Related Issues

Fixes #18400
Replaces #16767 by @leonlaiyc

Pattern harvest

Rule candidate: review-prompt
Pattern: a turn-teardown requeue arm gates on _should_suppress_requeue(slot) alone — any arm that can replay a prompt must also read _stop_pressed(), because a Stop resolved back to idle is invisible to the suppress check.

Checklist

  • Tests added/updated and passing locally
  • PR body matches the current diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 9, 2026 08:27
@iamwhatever
iamwhatever requested a review from bolichen97 October 9, 2026 08:27
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 52c5b19c3c1c5fa67a46dfdcbb681c68cd228800 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 52c5b19c3c1c5fa67a46dfdcbb681c68cd228800 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 52c5b19

Description read

[DESCRIPTION-READ] 14926f132422c4b516bf834004426ec52a8abec9c696e435329d9b0b6252213e

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 52c5b19c3c1c5fa67a46dfdcbb681c68cd228800 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check the throttle-fallback arm at chat_runner.py:18446. Its entry gate is still _should_suppress_requeue alone, so a resolved Stop still swaps the model and posts the "throttled" notice. Only the replay is dropped.

What this change ships

Inventory (7 items) — 7 justified

Intent: when a user stops a prompt and the runtime then fails during teardown, the stopped prompt should not run again on a fresh runtime. FIX (#18400; the PR's tests fail on base).

  1. When the runtime dies, a prompt whose Stop already resolved is no longer replayed — justified
  2. On pipe death or a prompt-busy answer, a stopped prompt is no longer re-queued — justified
  3. After the busy retries run out, the provider-reset re-queue respects a resolved Stop (one internal kwarg, one caller) — justified
  4. Image-history recovery no longer discards the conversation and replays a stopped prompt — justified
  5. A transient 5xx after a resolved Stop ends the turn with no "retrying" card shown and then withdrawn — justified
  6. That early end resets the four per-turn retry/fallback budgets, as the existing drop arm does — justified
  7. Poisoned-conversation reset no longer runs the canary, discard and replay after a resolved Stop — justified

[FIRST-PRINCIPLES-REVIEWED] 52c5b19

Description read

[DESCRIPTION-READ] 52a416f9ed709bd2c2f1e649fe925823e2c3544aa629941bc90a01457a1e7e4d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 52c5b19c3c1c5fa67a46dfdcbb681c68cd228800 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 52c5b19

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 52c5b19c3c1c5fa67a46dfdcbb681c68cd228800: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 52c5b19c3c1c5fa67a46dfdcbb681c68cd228800 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 52c5b19

Verdict parsed from the review's SHA-scoped output markers for commit 52c5b19c3c1c5fa67a46dfdcbb681c68cd228800.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 52c5b19c3c1c5fa67a46dfdcbb681c68cd228800: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 9, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Self-review (crew-pr-reviewer, read-only): CHANGES NEEDED

This PR stops a Stop that already went back to idle from being replayed after a runtime death, a pipe death, a prompt-busy answer or busy exhaustion, by pairing each of those gates with _stop_pressed(). For those four arms the root cause is fixed: I read the diff and the head's chat_runner.py, and _stop_pressed() reads the live stop counters, so the new disjunct catches exactly the case _should_suppress_requeue misses. The tests have a positive control and a stopped case for every arm, and all five AI lanes PASS with no open findings. But the stated goal ("can no longer be replayed on a fresh runtime after a mid-turn runtime failure") is not met for one sibling arm. The image-history recovery at src/kiro_crew/dashboard/chat_runner.py:~17977 (the image_format_unsupported branch) still gates its non-owed replay on not _should_suppress_requeue(slot) alone. It discards the conversation and re-queues the user's message verbatim on a fresh session, and its _image_stop_gen snapshot (~18034) is taken after the Stop already moved the counter, so a resolved Stop is invisible there too. Suggested fix: add and not _stop_pressed() next to not _should_suppress_requeue(slot) in that condition (leave the _image_replay_owed half alone), and add an image-history case to test/test_chat_requeue_resolved_stop.py. Advisory, not blocking: the Design lane's note is right that the transient-5xx arm (~18093) still shows a "retrying" card before its post-backoff re-read drops the replay, and pairing that first gate too would remove the flash. — crew-pr-reviewer

@iamwhatever
iamwhatever marked this pull request as draft October 9, 2026 16:34
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Moved to draft by crew-pr-reviewer: the self-review above is CHANGES NEEDED. GitHub refuses a self-request-changes and the readiness check ignores human reviews, so draft keeps this PR from merging (it also turns auto-merge off). Once the points above are fixed, mark it ready with gh pr ready 18401 and it will be re-reviewed on the new head. — crew-pr-reviewer

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 9, 2026
@iamwhatever
iamwhatever force-pushed the fix/requeue-see-resolved-stop-r23 branch from c7caa2f to 54f5b80 Compare October 10, 2026 20:23
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Fixed on head 54f5b80b2 (rebased once onto current main; the original commit by @cixuuz is kept, with its Co-authored-by for @leonlaiyc, plus one follow-up commit).

  1. Image-history recovery ignored a resolved Stop (CHANGES NEEDED). chat_runner.py — the image_format_unsupported branch's non-owed replay condition now reads not _should_suppress_requeue(slot) and not _stop_pressed() and .... The _image_replay_owed half is unchanged. Test: an image-history case in test/test_chat_requeue_resolved_stop.py (positive control replays once; resolved Stop queues nothing and shows no recovery card). The stopped case fails on the base commit.
  2. Transient-5xx "retrying" flash (advisory). The arm's first gate is now _should_suppress_requeue(slot) or _stop_pressed(), so a resolved Stop shows no "retrying" card the post-backoff re-read would withdraw. Test: a transient-5xx case in the same file; its stopped case also fails on base.

Local: the new file 12/12; 36 related test files (image/transient/stop recovery, test_dashboard_chat.py, composition contract) 4540 passed. PR body updated to cover the two extra arms. No new parsing or regex.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@iamwhatever
iamwhatever force-pushed the fix/requeue-see-resolved-stop-r23 branch from 54f5b80 to 77ac9c3 Compare October 10, 2026 20:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 77ac9c3bf addresses the review lanes on 54f5b80b2:

  • Opus 5.5 BLOCK (process-wide asyncio.sleep patch in the new test): fixed. test/test_chat_requeue_resolved_stop.py now patches chat_runner._recovery_delay, the runner's own backoff seam. test/test_flake_pattern_ratchet.py passes locally; it failed before.
  • First Principles CONCERNS (poisoned-reset sibling arm): fixed. The poisoned-conversation reset gate now also reads _stop_pressed(). New two-cycle test pair; the stopped case fails on base. The display-only acp_recovery._will_recover "resuming…" flash is named as deferred in the PR body.
  • Backend Tests shard 4 (Linux + Windows) / Coverage Gate: test/test_approval_decide_owner_bound.py fails 3/8 on main itself (ec2b33724), reproduced locally. Not caused by this diff.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@iamwhatever
iamwhatever force-pushed the fix/requeue-see-resolved-stop-r23 branch from 77ac9c3 to 6461b59 Compare October 10, 2026 22:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@iamwhatever
iamwhatever marked this pull request as ready for review October 10, 2026 23:00
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Self-review (crew-pr-reviewer, read-only): CHANGES NEEDED

This PR stops a Stop that already went back to idle from being replayed after a runtime failure. It pairs each requeue arm's _should_suppress_requeue(slot) with the live _stop_pressed() read, and the root cause is real: _queue_recovery (chat_runner.py:9143) has no Stop check of its own. My earlier points are addressed: the image-history arm now reads not _stop_pressed() and the transient-5xx "retrying" flash is gone, with a test for each. But the head (6461b59) brings a real side effect that nobody has answered. The Opus lane flagged it and I confirmed it in the code. At src/kiro_crew/dashboard/chat_runner.py:18261 the new pre-backoff if _should_suppress_requeue(slot) or _stop_pressed(): pass now catches the resolved-Stop case. Before, that case reached the post-backoff drop arm (~18346), which resets _transient_5xx_retries, _infra_retries, _fallback_candidate_idx and _fallback_walked. Now the turn ends with _transient_5xx_retries already raised at 18235, so the user's next turn gets a shorter 5xx ladder, a bigger backoff seed and an earlier fallback swap. Suggested fix: split that gate, keep pass for the suppress case, and in the _stop_pressed() branch reset the same four fields, with a test that the next turn starts at zero. The current head's Design CONCERNS (do the veto once inside _queue_recovery) and First Principles CONCERNS also have no disposition. The PR body itself says the stopped cases for process-died, pipe-death, prompt-busy and prompt-busy-exhausted pass on base, so the tests do not prove the four arms in the title; show a base failure for them or answer why the gate is still needed. — crew-pr-reviewer

@iamwhatever
iamwhatever marked this pull request as draft October 11, 2026 00:23
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Converted to draft by crew-pr-reviewer: the self-review above is CHANGES NEEDED. Mark ready again (gh pr ready 18401) once the points are addressed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 11, 2026
…e arms

Follow-up to merged #16548, which gave only the lost-session arm the
`_stop_pressed()` read. The runtime-death arm, the prompt-busy / pipe-death
arm, and the post-busy-exhaustion requeue helper still gate on
`_should_suppress_requeue(slot)` alone, so a Stop that pressed and resolved
back to idle before the error reached the handler (invisible to the suppress
check, and `_queue_recovery` takes no Stop snapshot of its own) still queues
the stopped prompt for replay on a fresh runtime.

Pair each remaining arm's gate with `_stop_pressed()`, the same pairing #16548
used, and thread `_stop_pressed` into `_requeue_after_prompt_busy` so the
recovery helper can see it too. Adds a parametrised test covering all four
arms with a positive control (no Stop replays once) and the stopped case
(resolved Stop queues nothing).

Replaces #16767 by @leonlaiyc.

Co-authored-by: Leon <276034861+leonlaiyc@users.noreply.github.com>
@iamwhatever
iamwhatever force-pushed the fix/requeue-see-resolved-stop-r23 branch from 6461b59 to 30f0588 Compare October 11, 2026 07:55
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Re the CHANGES NEEDED self-review of 2026-10-11 00:23Z. Fixed on head 30f05882e (rebased once onto current main 6548d1dc6; still two commits, authored by @cixuuz, Co-authored-by for @leonlaiyc kept).

  1. Transient-5xx resolved-Stop gate left the retry budgets raised (Opus finding). Fixed. chat_runner.py: the pre-backoff gate is split. _should_suppress_requeue(slot) keeps pass. A new elif _stop_pressed(): branch resets _transient_5xx_retries, _infra_retries, _fallback_candidate_idx and _fallback_walked, the same four fields the post-backoff drop arm resets. Test: test_a_resolved_stop_on_a_transient_5xx_leaves_the_next_turn_a_full_ladder in test/test_chat_requeue_resolved_stop.py. It fails on the previous head (assert 1 == 0) and passes now.
  2. Design CONCERNS (do the veto once inside _queue_recovery). Not done, on purpose. _queue_recovery also queues an owed sub-agent completion (_requeue_owed_completion, gated by _replay_owes_delivery). Its docstring says no Stop cancels it, because it is a result the parent is still owed. A blanket _stop_pressed() veto inside _queue_recovery would drop that result. A per-call-site lint test would be new machinery outside this fix. The reason is now in the PR body.
  3. First Principles CONCERNS / "the four title arms have no base failure". The PR body was stale. I re-ran the new test file against current main source (chat_runner.py and chat_turn/recovery.py checked out from origin/main). All seven stopped cases fail on base: process-died, pipe-death, prompt-busy, prompt-busy-exhausted, image-history, transient-5xx and poisoned-reset. Each of the four title arms queues the stopped prompt again (calls == [msg, msg]). So every gate in the diff has its own reproduction, and the _stop_pressed kwarg on _requeue_after_prompt_busy stays. PR body corrected.

Local: the test file passes 15/15. 939 passed across the related recovery/transient/prompt-busy/steer/composition-contract files, and 903 passed in test_dashboard_chat.py. The comment-history and brand gates are clean.

…set replays

Self-review follow-up. The image-history recovery and the poisoned-conversation
reset arm still gated their replay on _should_suppress_requeue(slot) alone, and
each takes its Stop snapshot (_image_stop_gen, _stop_gen_before_canary) after a
resolved Stop already moved the counter, so the stopped prompt was replayed on
a fresh session. Pair both gates with _stop_pressed() (the image arm's
owed-completion half is unchanged). Also give the transient-5xx arm's first
gate its own resolved-Stop branch, so a resolved Stop no longer flashes a
"retrying" card the post-backoff re-read withdraws. That branch ends the turn
with no requeue, so it resets the same four per-turn budgets the post-backoff
drop arm resets (_transient_5xx_retries, _infra_retries,
_fallback_candidate_idx, _fallback_walked); otherwise the attempt it just
counted shortens the next turn's 5xx ladder.

Tests: image-history, transient-5xx and poisoned-reset cases in
test/test_chat_requeue_resolved_stop.py; each stopped case fails on base. A
budget case checks a resolved Stop on a transient 5xx leaves all four fields at
zero for the next turn. The file patches the runner's _recovery_delay seam
instead of a process-wide asyncio.sleep (Determinism contract / flake-pattern
ratchet).
@iamwhatever
iamwhatever force-pushed the fix/requeue-see-resolved-stop-r23 branch from 30f0588 to 52c5b19 Compare October 11, 2026 08:17
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head is now 52c5b19c3: the same change as 30f05882e, plus a black reflow of one line in test/test_chat_requeue_resolved_stop.py that Backend Lint flagged. No logic change.

@iamwhatever
iamwhatever marked this pull request as ready for review October 11, 2026 08:42
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@chenmingwei23
chenmingwei23 merged commit 4ef0faa into main Oct 11, 2026
103 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/requeue-see-resolved-stop-r23 branch October 11, 2026 08:56
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A resolved Stop can still replay the prompt on the runtime-death and busy-requeue arms

3 participants