Skip to content

fix(dashboard): await the chat_done payload when a refusal replay is aborted at consume - #16403

Merged
bolichen97 merged 1 commit into
mainfrom
fix/chat-turn-refusal-abort-chat-done
Oct 3, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/chat-turn-refusal-abort-chat-done

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When a content-filter retry is cancelled before it runs, the dashboard gets a real end-of-turn frame, so the chat stops looking busy and the completion hint is correct.

A content-filter retry is queued after a refusal and re-checked when its turn starts. If a Stop, a session rebind or a newer message landed in between, the check cancels the retry: it posts "ℹ️ Content-filter retry cancelled — …" and ends the turn without calling the model. That turn's chat_done frame is then the only signal that the turn is over. The cancel path passed chat_done_payload(state, slot) to broadcast_ws without await, so the payload was a coroutine object instead of the {slot, continuing, needs_input} dict. With a browser connected, the WebSocket hub's json.dumps raises on it, the gate's except logs it at debug level, and no chat_done frame is sent; the coroutine is never awaited, and Python warns about it (#16230).

Why it matters

The cancelled-retry path is the one turn end with no provider output, so a missing chat_done frame there means the browser never hears that the turn ended: the end-of-turn hint and the completion sound do not fire for it. Every other turn end already awaits the payload. This path fails silently, apart from a never-awaited-coroutine warning on each cancellation.

Not a goal

  • No change to when a retry is cancelled, the notice text, the spent allowance or the model swap unwind.
  • No change to the drain-time purge or the other consume checks (image-history and model-access recovery already await the payload).
  • No change to chat_done_payload itself or to the frame's shape.
  • No typing change to broadcast_ws (its data: object parameter is why mypy did not catch this).

What changed (motivation → approach → change)

Cancel point Site Fix
Consume check at turn entry (Stop, rebind or newer message) _refusal_replay_vetoed_at_consume in src/kiro_crew/dashboard/chat_turn/recovery.py await chat_done_payload(state, slot)
Dispatch check (newer message queued while the session was acquired) _run_chat in src/kiro_crew/dashboard/chat_runner.py await chat_done_payload(state, slot)

The bug is one missing await at each of the two points where a content-filter retry is cancelled before it runs. Both now await the payload, as every other turn end does; git grep finds no other un-awaited chat_done_payload call in src. The consume phase's only caller in _run_chat already awaits it.

Each new await runs after its check and after every slot change (retry text and queue id cleared, notice appended), right before the turn returns, so each check-then-change stays atomic on the event loop. That is why the consume phase moves from the composition contract's _AWAIT_FREE_PHASES table in test/test_chat_runner_composition_contract.py to a new _DECIDE_THEN_SUSPEND_PHASES table: it now suspends once, at the broadcast, after the gate has decided. The new table keeps the guard the old one gave. Its bytecode sweep finds the phase's first suspension point and fails if any instruction after it reads slot or writes an attribute, a subscript or a global, so an await placed between the check and the change still goes red.

Backwards compatibility

Compatible: the cancel path now sends the same chat_done dict every other turn end sends; nothing that works today stops working.

Tests

  • test/test_refusal_replay_consume_abort_chat_done.py (new) drives both cancel points through chat_runner and asserts one chat_done frame whose payload is a dict, not a coroutine, with the {slot, continuing, needs_input} keys and the same value a fresh chat_done_payload returns. The consume check is covered for each cancel reason (stopped, rebound, superseded), plus a retry allowed to run (no frame) and a stopped retry end to end through _run_chat(..., _refusal_replay=True). The dispatch check is covered by a newer message queued inside session acquisition. The notice text and the cleared retry record are checked too, and no case reaches the model. Each site's tests fail with only that site's await removed (consume: 4 failed, plus the contract's suspension check; dispatch: 1 failed; every regression failure is on "chat_done carried an un-awaited coroutine") and all 6 pass with the fix, including under -W error::RuntimeWarning.
  • test/test_chat_runner_composition_contract.py: the consume phase leaves _AWAIT_FREE_PHASES and is listed in _DECIDE_THEN_SUSPEND_PHASES. test_a_phase_suspends_only_after_its_decision asserts the phase is a coroutine defined in the owner directory, that it still suspends, and that nothing after its first suspension touches the slot or writes state. test_the_decide_then_suspend_scan_sees_state_after_an_await is the scan's self-test: a write before an await passes, and a write or a slot read after one is flagged. Adding a slot write after the broadcast makes the sweep fail (1 failed); the existing await-free sweep and its self-test stay green.
  • Ran on the committed tree, rebased onto current main: the new test under -W error::RuntimeWarning (6 passed); the new test, the contract test, test_chat_runner_coverage.py and test_refusal_inband_notice.py (610 passed); those plus every test file naming the refusal replay or fallback, 13 files (2,634 passed, 1 skipped); TestRunChatRefusalFallback under -W error::RuntimeWarning (42 passed). The static floor is green: black, isort, flake8, mypy, comment history, brand, subprocess encoding and docs-lint. Semgrep 1.78 in baseline mode reports 0 new findings, and its rule tests pass.

Manual verification

N/A — unit coverage sufficient: the frame is built and sent inside the gate, and the tests read the exact broadcast_ws call.

Related Issues

Closes #13824

The same bug was reported again as #16230, which is closed as a duplicate of #13824.

Pattern harvest

Rule candidate: semgrep
Pattern: an async def helper called without await as an argument to a sink typed object (here broadcast_ws(..., chat_done_payload(...))), which mypy cannot flag

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 3, 2026 07:03
@bolichen97
bolichen97 requested a review from iamwhatever October 3, 2026 07:03
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 03105c7eb919c237775ffbbd4478249bd176fd13 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 03105c7

Description read

[DESCRIPTION-READ] 78aff34fbf3c36013a80d6fce2dc53445b812c974d16c6565659d6af258b17a2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 03105c7eb919c237775ffbbd4478249bd176fd13 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (3 items) — 3 justified

Intent: when a content-filter retry is cancelled before it runs, the browser should still learn that the turn ended. This is a FIX (#13824 / #16230).

  1. A retry cancelled at turn entry (Stop, rebind or a newer message) now sends a real chat_done frame, so the hint and the completion sound fire. — justified
  2. A retry cancelled at the dispatch check (a newer message queued while the session was being acquired) now sends a real chat_done frame. — justified
  3. The consume phase leaves the await-free contract and gets a "decide, then suspend once" bytecode check. That check keeps the old pin's stated reason ("an await added to one would let another task change the slot between the check and the mutation"). — justified

[FIRST-PRINCIPLES-REVIEWED] 03105c7

Description read

[DESCRIPTION-READ] 71c7af8cad54707be6586bdc84d6c9db75e0230be8a667cac1f913661a92ad2a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 03105c7eb919c237775ffbbd4478249bd176fd13 — this comment is updated in place on each push.

Review details

Not blocking. One advisory finding: the owning spec wasn't updated to match the test change.

FINDING — docs/system-specs/modules/learn-cron-dashboard.md:1397 — the spec still says "A phase cut from a block that never awaited stays await-free … the contract test pins each one (_AWAIT_FREE_PHASES)". That is no longer true: this diff takes _refusal_replay_vetoed_at_consume out of that list, adds an await chat_done_payload(...) inside it, and moves it to a new _DECIDE_THEN_SUSPEND_PHASES list that the spec doesn't mention. A contributor following the spec would treat the consume gate as await-free and wouldn't know the rule that now applies to it. → Fix: in the same paragraph, add a sentence on the decide-then-suspend category. It covers a phase that ends the turn with one awaited chat_done broadcast. Such a phase may suspend only after its check and every slot change, and _DECIDE_THEN_SUSPEND_PHASES pins it.

[OPUS-REVIEWED] 03105c7

Verdict parsed from the review's SHA-scoped output markers for commit 03105c7eb919c237775ffbbd4478249bd176fd13.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 03105c7eb919c237775ffbbd4478249bd176fd13: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 03105c7eb919c237775ffbbd4478249bd176fd13 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 03105c7

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 03105c7eb919c237775ffbbd4478249bd176fd13: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the fix/chat-turn-refusal-abort-chat-done branch from f104a15 to 2c4fc7f Compare October 3, 2026 07:38
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=acb2783d1738 Item 1: the superseded-before-dispatch chat_done_payload in _run_chat is still un-awaited

Outcome: fixed in 2c4fc7f.
_run_chat (src/kiro_crew/dashboard/chat_runner.py) now awaits chat_done_payload(state, slot) on the "your newer message runs instead" dispatch-check path, after the check and every slot mutation, right before the return.
git grep finds no other un-awaited chat_done_payload call in src.
New test test_a_replay_superseded_while_preparing_ends_with_the_awaited_done_payload queues a newer message inside session acquisition and asserts one dict chat_done frame. It fails with only that await removed and passes with it.

@bolichen97

Copy link
Copy Markdown
Contributor Author

New evidence since the PR opened: the dispatch-check cancel path in _run_chat ("your newer message runs instead", reached when a newer message is queued while the session is being acquired) had the same un-awaited chat_done_payload. First Principles found it on f104a15. 2c4fc7f fixes both sites in the one commit, and git grep finds no other un-awaited call in src. The frozen "Every other turn end already awaits the payload" sentence was written before this second site was found; with 2c4fc7f it holds.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 3, 2026
A content-filter retry replay is cancelled at two points before it
runs: at its consume seam (a Stop, a session rebind or a newer message
landed after the drain queued it) and at the dispatch check (a newer
message was queued while the session was being acquired). Either way
the turn ends without a provider call, and its chat_done frame is the
only signal that the turn is over. Both paths broadcast
chat_done_payload(state, slot) without awaiting it, so the payload was
a coroutine instead of the {slot, continuing, needs_input} dict: the
WebSocket hub's json.dumps raises on it, the best-effort except
swallows the error, no chat_done frame is sent, and the coroutine is
never awaited. Await it at both sites, as every other turn end does.

Each await comes after the check and every slot mutation, so each
check-then-mutate stays atomic on the loop. The consume phase moves
from the composition contract's await-free table to a
decide-then-suspend table, whose bytecode sweep pins that nothing
after its first suspension point reads the slot or writes state.

Closes #16230
@bolichen97
bolichen97 force-pushed the fix/chat-turn-refusal-abort-chat-done branch from 2c4fc7f to 03105c7 Compare October 3, 2026 16:33
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=9ea72b4acea7 The await-free guard on the consume gate is removed, not kept

Outcome: fixed in 03105c7, through the review's second clearing path: a sweep now pins that the gate's only suspension point comes after its last slot change.
test/test_chat_runner_composition_contract.py lists _refusal_replay_vetoed_at_consume in a new _DECIDE_THEN_SUSPEND_PHASES table. test_a_phase_suspends_only_after_its_decision reads the phase's bytecode, finds its first suspension point, and fails if any instruction after it loads slot or writes an attribute, a subscript or a global. It also asserts the phase is still a coroutine defined in the owner directory and still suspends, so it cannot quietly drop out of both tables.
test_the_decide_then_suspend_scan_sees_state_after_an_await is the scan's self-test: a write before an await passes, and a write or a slot read after one is flagged.
Mutation proof on the pushed tree: adding slot._refusal_replay_queue_id = "" after the broadcast fails the sweep (1 failed), and removing the broadcast's await fails it too ("no longer suspends"). With the fix, the contract file and the regression file pass.
The broadcast stays in the gate rather than moving to _run_chat: the gate owns the cancel notice and the cleared retry record, and the sweep now guards the same check-then-change the await-free table did.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Merged-tree probe (no push): head 03105c7eb919c237775ffbbd4478249bd176fd13 cherry-picked onto current main ff007b88b8b3e589d56d50a37b35cedc72252815, which is 21 commits ahead of this PR's base.

  • Two of those commits touch a file this PR changes: fix(chat): let a resolved Stop veto the lost-session reconnect replay #16548 (105630f06) and fix(skill-view): record a view's source agent before its sidecar is deleted #16073 (cc57eeb34), both in src/kiro_crew/dashboard/chat_runner.py. Their hunks are at L79, L653, L8939 and L16438. This PR's one-line change is at L10592, so none of them land near it. git merge-tree --write-tree is clean, and the cherry-pick applies with no conflict.
  • On the probe tree, which imports kiro_crew from the probe's own src:
    • test/test_refusal_replay_consume_abort_chat_done.py under -W error::RuntimeWarning: 6 passed.
    • These 10 files together: 1,793 passed, 0 failed:
      • this PR's files: test_chat_runner_composition_contract.py, test_chat_runner_coverage.py, test_refusal_inband_notice.py and test_refusal_replay_consume_abort_chat_done.py;
      • the upstream commits' tests: test_chat_session_not_found_recovery.py, test_native_skill_projection.py, test_resume_retired_agent_names.py and test_skill_view_mode_recovery.py;
      • test_dashboard_chat.py and test_chat_slot_switch_atomicity.py.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE

Reviewed at head 03105c7eb919c237775ffbbd4478249bd176fd13 (base 8b26fc718b). The fix adds the missing await at both points where a content-filter retry is cancelled before it runs: src/kiro_crew/dashboard/chat_turn/recovery.py:520 (consume check) and src/kiro_crew/dashboard/chat_runner.py:10592 (dispatch check). Both now match the existing model-access recovery cancel path (recovery.py:458) and every other turn end. git grep finds no other un-awaited chat_done_payload( in src; all 8 call sites await it.

Blocking

None.

Non-blocking follow-ups

  1. Stale spec text. docs/system-specs/modules/learn-cron-dashboard.md:1396-1399 still says a consume gate's check-then-mutate "runs without a suspension point; the contract test pins each one (_AWAIT_FREE_PHASES)". _refusal_replay_vetoed_at_consume has moved out of that table into _DECIDE_THEN_SUSPEND_PHASES, and it now suspends once, after its decision. The paragraph should name the new table and its rule (no slot read and no state write after the first suspension). The Opus bot raised the same advisory.
  2. The new bytecode scan has a blind spot. _touches_state_after_suspending (test/test_chat_runner_composition_contract.py) flags LOAD_FAST slot and STORE_*/DELETE_* after the first suspension. It does not flag a mutating method call on another local, such as state.something.append(...) after the await. Nothing does that today, and slot.append(...) is caught through the slot load. A note in the table's comment would keep a future reader from treating the scan as complete.
  3. Cancellation during the new await. CancelledError is a BaseException, so it passes through the except Exception at both sites and propagates out of _run_chat. This fails closed. Every slot change (retry text and queue id cleared, notice appended) has already run before the await, so cancellation leaves the slot consistent, and the only loss is the chat_done frame, which was also missing before this PR. This is the same behaviour as every other awaited turn end. Nothing to change; recorded so the disposition is explicit.

Unbounded awaits

  • Production. The only suspension inside chat_done_payload (chat_utils.py:1310) is subagents_attached_async → _queued_depth_off_loop → queued_count_for_async, a task-store read on the writer thread. It has no timeout of its own. That is unchanged: the same await already runs at every other turn end, and this PR adds no new wait primitive. Error paths fail closed: a probe exception becomes queued = 1, and an outer exception becomes continuing = True, so the frame is still built.
  • Tests. No test can block forever. In the new test file, state.subagents comes from _make_state, the MagicMock double has no coroutine queued_count_for_async, and so the sync branch is taken. sessions.get_or_create is an AsyncMock, and its _acquire side effect returns immediately. Nothing awaits an event, queue or future that no one sets. Wall time for both files was about 17 s.

Exactly-once delivery

  • All 5 cancel paths assert exactly one chat_done frame (_assert_one_done_frame, len(payloads) == 1): consume × {stopped, rebound, superseded}, a stopped retry through _run_chat, and a newer message queued during session acquisition. The allowed-to-run case asserts zero frames.
  • The two _run_chat cases prove the runner's caller does not add a second frame after return. Line 8138 returns straight after a True veto, and the dispatch-site return comes before any later broadcast.
  • I added a double-broadcast mutation at the consume site (M4 below). It reddens all 4 consume-path tests and the contract sweep.

What I verified (mutations; only the two related test files, -W error::RuntimeWarning)

Mutation Result
Baseline 219 passed
M1: drop await at recovery.py:520 5 failed: test_a_cancelled_replay_broadcasts_the_awaited_done_payload[stopped,rebound,superseded], test_run_chat_ends_a_stopped_replay_with_the_awaited_done_payload, test_a_phase_suspends_only_after_its_decision[_refusal_replay_vetoed_at_consume]
M2: drop await at chat_runner.py:10592 1 failed: test_a_replay_superseded_while_preparing_ends_with_the_awaited_done_payload
M3: slot write after the consume broadcast 1 failed: test_a_phase_suspends_only_after_its_decision[_refusal_replay_vetoed_at_consume]
M4: duplicate chat_done broadcast at consume 5 failed (the M1 set: 4 frame-count assertions plus the contract sweep)

Every mutation was restored, and git status --short was clean afterwards. I did not run the full suite.

PR body claims checked against the source: the two sites and the one-line fix at each; no other un-awaited call; the consume phase moved between contract tables; the scan's self-test covers write-before-await (passes), write-after (flagged) and read-after (flagged); per-site red counts (consume 4 + contract, dispatch 1). All match.

Bots: Design PASS, First Principles PASS, Opus no blocking findings (advisory: the spec, follow-up 1), GPT no findings. The earlier First Principles finding (the dispatch site was also un-awaited) is fixed at this head.

CI at 03105c7eb9: 95 check runs: 83 success, 12 skipped, 0 failure, 0 pending. Auto-merge is not enabled. No human review yet.

@bolichen97
bolichen97 enabled auto-merge (squash) October 3, 2026 20:44
@bolichen97
bolichen97 merged commit 4588f6c into main Oct 3, 2026
142 of 144 checks passed
@bolichen97
bolichen97 deleted the fix/chat-turn-refusal-abort-chat-done branch October 3, 2026 20:44
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(dashboard): two refusal-replay aborts broadcast an un-awaited chat_done_payload coroutine

2 participants