Repository navigation
fix(dashboard): await the chat_done payload when a refusal replay is aborted at consume - #16403
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 03105c7 Description read[DESCRIPTION-READ] 78aff34fbf3c36013a80d6fce2dc53445b812c974d16c6565659d6af258b17a2 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (3 items) — 3 justifiedIntent: when a content-filter retry is cancelled before it runs, the browser should still learn that the turn ended. This is a FIX (#13824 / #16230).
[FIRST-PRINCIPLES-REVIEWED] 03105c7 Description read[DESCRIPTION-READ] 71c7af8cad54707be6586bdc84d6c9db75e0230be8a667cac1f913661a92ad2a That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNot blocking. One advisory finding: the owning spec wasn't updated to match the test change. FINDING — docs/system-specs/modules/learn-cron-dashboard.md:1397 — the spec still says "A phase cut from a block that never awaited stays await-free … the contract test pins each one ( [OPUS-REVIEWED] 03105c7 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
f104a15 to
2c4fc7f
Compare
|
|
New evidence since the PR opened: the dispatch-check cancel path in |
A content-filter retry replay is cancelled at two points before it
runs: at its consume seam (a Stop, a session rebind or a newer message
landed after the drain queued it) and at the dispatch check (a newer
message was queued while the session was being acquired). Either way
the turn ends without a provider call, and its chat_done frame is the
only signal that the turn is over. Both paths broadcast
chat_done_payload(state, slot) without awaiting it, so the payload was
a coroutine instead of the {slot, continuing, needs_input} dict: the
WebSocket hub's json.dumps raises on it, the best-effort except
swallows the error, no chat_done frame is sent, and the coroutine is
never awaited. Await it at both sites, as every other turn end does.
Each await comes after the check and every slot mutation, so each
check-then-mutate stays atomic on the loop. The consume phase moves
from the composition contract's await-free table to a
decide-then-suspend table, whose bytecode sweep pins that nothing
after its first suspension point reads the slot or writes state.
Closes #16230
2c4fc7f to
03105c7
Compare
|
|
Merged-tree probe (no push): head
|
buluoray
left a comment
There was a problem hiding this comment.
Verdict: APPROVE
Reviewed at head 03105c7eb919c237775ffbbd4478249bd176fd13 (base 8b26fc718b). The fix adds the missing await at both points where a content-filter retry is cancelled before it runs: src/kiro_crew/dashboard/chat_turn/recovery.py:520 (consume check) and src/kiro_crew/dashboard/chat_runner.py:10592 (dispatch check). Both now match the existing model-access recovery cancel path (recovery.py:458) and every other turn end. git grep finds no other un-awaited chat_done_payload( in src; all 8 call sites await it.
Blocking
None.
Non-blocking follow-ups
- Stale spec text.
docs/system-specs/modules/learn-cron-dashboard.md:1396-1399still says a consume gate's check-then-mutate "runs without a suspension point; the contract test pins each one (_AWAIT_FREE_PHASES)"._refusal_replay_vetoed_at_consumehas moved out of that table into_DECIDE_THEN_SUSPEND_PHASES, and it now suspends once, after its decision. The paragraph should name the new table and its rule (no slot read and no state write after the first suspension). The Opus bot raised the same advisory. - The new bytecode scan has a blind spot.
_touches_state_after_suspending(test/test_chat_runner_composition_contract.py) flagsLOAD_FAST slotandSTORE_*/DELETE_*after the first suspension. It does not flag a mutating method call on another local, such asstate.something.append(...)after the await. Nothing does that today, andslot.append(...)is caught through theslotload. A note in the table's comment would keep a future reader from treating the scan as complete. - Cancellation during the new await.
CancelledErroris aBaseException, so it passes through theexcept Exceptionat both sites and propagates out of_run_chat. This fails closed. Every slot change (retry text and queue id cleared, notice appended) has already run before the await, so cancellation leaves the slot consistent, and the only loss is thechat_doneframe, which was also missing before this PR. This is the same behaviour as every other awaited turn end. Nothing to change; recorded so the disposition is explicit.
Unbounded awaits
- Production. The only suspension inside
chat_done_payload(chat_utils.py:1310) issubagents_attached_async→_queued_depth_off_loop→queued_count_for_async, a task-store read on the writer thread. It has no timeout of its own. That is unchanged: the same await already runs at every other turn end, and this PR adds no new wait primitive. Error paths fail closed: a probe exception becomesqueued = 1, and an outer exception becomescontinuing = True, so the frame is still built. - Tests. No test can block forever. In the new test file,
state.subagentscomes from_make_state, theMagicMockdouble has no coroutinequeued_count_for_async, and so the sync branch is taken.sessions.get_or_createis anAsyncMock, and its_acquireside effect returns immediately. Nothing awaits an event, queue or future that no one sets. Wall time for both files was about 17 s.
Exactly-once delivery
- All 5 cancel paths assert exactly one
chat_doneframe (_assert_one_done_frame,len(payloads) == 1): consume × {stopped, rebound, superseded}, a stopped retry through_run_chat, and a newer message queued during session acquisition. The allowed-to-run case asserts zero frames. - The two
_run_chatcases prove the runner's caller does not add a second frame afterreturn. Line 8138 returns straight after a True veto, and the dispatch-sitereturncomes before any later broadcast. - I added a double-broadcast mutation at the consume site (M4 below). It reddens all 4 consume-path tests and the contract sweep.
What I verified (mutations; only the two related test files, -W error::RuntimeWarning)
| Mutation | Result |
|---|---|
| Baseline | 219 passed |
M1: drop await at recovery.py:520 |
5 failed: test_a_cancelled_replay_broadcasts_the_awaited_done_payload[stopped,rebound,superseded], test_run_chat_ends_a_stopped_replay_with_the_awaited_done_payload, test_a_phase_suspends_only_after_its_decision[_refusal_replay_vetoed_at_consume] |
M2: drop await at chat_runner.py:10592 |
1 failed: test_a_replay_superseded_while_preparing_ends_with_the_awaited_done_payload |
| M3: slot write after the consume broadcast | 1 failed: test_a_phase_suspends_only_after_its_decision[_refusal_replay_vetoed_at_consume] |
M4: duplicate chat_done broadcast at consume |
5 failed (the M1 set: 4 frame-count assertions plus the contract sweep) |
Every mutation was restored, and git status --short was clean afterwards. I did not run the full suite.
PR body claims checked against the source: the two sites and the one-line fix at each; no other un-awaited call; the consume phase moved between contract tables; the scan's self-test covers write-before-await (passes), write-after (flagged) and read-after (flagged); per-site red counts (consume 4 + contract, dispatch 1). All match.
Bots: Design PASS, First Principles PASS, Opus no blocking findings (advisory: the spec, follow-up 1), GPT no findings. The earlier First Principles finding (the dispatch site was also un-awaited) is fixed at this head.
CI at 03105c7eb9: 95 check runs: 83 success, 12 skipped, 0 failure, 0 pending. Auto-merge is not enabled. No human review yet.
Problem / Motivation
Goal: When a content-filter retry is cancelled before it runs, the dashboard gets a real end-of-turn frame, so the chat stops looking busy and the completion hint is correct.
A content-filter retry is queued after a refusal and re-checked when its turn starts. If a Stop, a session rebind or a newer message landed in between, the check cancels the retry: it posts "ℹ️ Content-filter retry cancelled — …" and ends the turn without calling the model. That turn's
chat_doneframe is then the only signal that the turn is over. The cancel path passedchat_done_payload(state, slot)tobroadcast_wswithoutawait, so the payload was a coroutine object instead of the{slot, continuing, needs_input}dict. With a browser connected, the WebSocket hub'sjson.dumpsraises on it, the gate'sexceptlogs it at debug level, and nochat_doneframe is sent; the coroutine is never awaited, and Python warns about it (#16230).Why it matters
The cancelled-retry path is the one turn end with no provider output, so a missing
chat_doneframe there means the browser never hears that the turn ended: the end-of-turn hint and the completion sound do not fire for it. Every other turn end already awaits the payload. This path fails silently, apart from a never-awaited-coroutine warning on each cancellation.Not a goal
chat_done_payloaditself or to the frame's shape.broadcast_ws(itsdata: objectparameter is why mypy did not catch this).What changed (motivation → approach → change)
_refusal_replay_vetoed_at_consumeinsrc/kiro_crew/dashboard/chat_turn/recovery.pyawait chat_done_payload(state, slot)_run_chatinsrc/kiro_crew/dashboard/chat_runner.pyawait chat_done_payload(state, slot)The bug is one missing
awaitat each of the two points where a content-filter retry is cancelled before it runs. Both now await the payload, as every other turn end does;git grepfinds no other un-awaitedchat_done_payloadcall insrc. The consume phase's only caller in_run_chatalready awaits it.Each new
awaitruns after its check and after every slot change (retry text and queue id cleared, notice appended), right before the turn returns, so each check-then-change stays atomic on the event loop. That is why the consume phase moves from the composition contract's_AWAIT_FREE_PHASEStable intest/test_chat_runner_composition_contract.pyto a new_DECIDE_THEN_SUSPEND_PHASEStable: it now suspends once, at the broadcast, after the gate has decided. The new table keeps the guard the old one gave. Its bytecode sweep finds the phase's first suspension point and fails if any instruction after it readsslotor writes an attribute, a subscript or a global, so anawaitplaced between the check and the change still goes red.Backwards compatibility
Compatible: the cancel path now sends the same
chat_donedict every other turn end sends; nothing that works today stops working.Tests
test/test_refusal_replay_consume_abort_chat_done.py(new) drives both cancel points throughchat_runnerand asserts onechat_doneframe whose payload is a dict, not a coroutine, with the{slot, continuing, needs_input}keys and the same value a freshchat_done_payloadreturns. The consume check is covered for each cancel reason (stopped, rebound, superseded), plus a retry allowed to run (no frame) and a stopped retry end to end through_run_chat(..., _refusal_replay=True). The dispatch check is covered by a newer message queued inside session acquisition. The notice text and the cleared retry record are checked too, and no case reaches the model. Each site's tests fail with only that site'sawaitremoved (consume: 4 failed, plus the contract's suspension check; dispatch: 1 failed; every regression failure is on "chat_done carried an un-awaited coroutine") and all 6 pass with the fix, including under-W error::RuntimeWarning.test/test_chat_runner_composition_contract.py: the consume phase leaves_AWAIT_FREE_PHASESand is listed in_DECIDE_THEN_SUSPEND_PHASES.test_a_phase_suspends_only_after_its_decisionasserts the phase is a coroutine defined in the owner directory, that it still suspends, and that nothing after its first suspension touches the slot or writes state.test_the_decide_then_suspend_scan_sees_state_after_an_awaitis the scan's self-test: a write before anawaitpasses, and a write or a slot read after one is flagged. Adding a slot write after the broadcast makes the sweep fail (1 failed); the existing await-free sweep and its self-test stay green.main: the new test under-W error::RuntimeWarning(6 passed); the new test, the contract test,test_chat_runner_coverage.pyandtest_refusal_inband_notice.py(610 passed); those plus every test file naming the refusal replay or fallback, 13 files (2,634 passed, 1 skipped);TestRunChatRefusalFallbackunder-W error::RuntimeWarning(42 passed). The static floor is green: black, isort, flake8, mypy, comment history, brand, subprocess encoding and docs-lint. Semgrep 1.78 in baseline mode reports 0 new findings, and its rule tests pass.Manual verification
N/A — unit coverage sufficient: the frame is built and sent inside the gate, and the tests read the exact
broadcast_wscall.Related Issues
Closes #13824
The same bug was reported again as #16230, which is closed as a duplicate of #13824.
Pattern harvest
Rule candidate: semgrep
Pattern: an
async defhelper called withoutawaitas an argument to a sink typedobject(herebroadcast_ws(..., chat_done_payload(...))), which mypy cannot flagChecklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)