Repository navigation
fix(chat): hand off the queue on every _run_chat exit - #16279
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Before merge, confirm one behaviour change: after this PR, a soft Stop no longer cancels a replayed sub-agent completion that the model never consumed. [DESIGN-REVIEWED] f34b7b5 Description read[DESCRIPTION-READ] d5a4ae0b0b0121807771591a143e6587b1bebef5ac9ec7531d7db1e1ebc94db4 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The owed-completion reclassification rides along with a queue hand-off fix: it changes what a soft Stop cancels and lets a replay run ahead of newer user messages. Not justified as shipped
What this change shipsInventory (10 items) — 8 justifiedIntent: make every way a chat turn can end run the next queued message, send
[FIRST-PRINCIPLES-REVIEWED] f34b7b5 Description read[DESCRIPTION-READ] 45799b727e8629bc60c7df45b189d33400262e86e8048d0b2338d63c2e046b18 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
73feb14 to
9fcc3a4
Compare
9fcc3a4 to
46f373b
Compare
46f373b to
2ad8af9
Compare
buluoray
left a comment
There was a problem hiding this comment.
Review of 424f33ba90 — queue hand-off on every _run_chat exit
Verdict: approve. Each exit path I traced hands the queue off exactly once. Every guard I removed made a named test fail, with one exception, listed as follow-up 1. #16020's start-priority plumbing is still in place.
Exactly-once hand-off, by exit path
- Normal end, or an exception or cancel inside the main
try. Thefinallyrecordsturn_exit.outcomesecond, after a synchronous lock release and before its first await (chat_runner.py:18617). It ends withawait run_to_completion(_end_turn_tail(...))(:18915)._end_turn_tailsetshanded_off = Trueas its first statement (:7699). The guard then skips its own tail becausehanded_offis set (:7845). - A
finallycut short by a second cancel.outcomeis already recorded, so the guard runs the tail with the turn's own policy rather than the early-exit policy. - Early returns above the
try(a replay cancelled at its consume seam, a local or blocked slash command). The guard'sfinallyruns the tail withoutcome=None. With no outcome, the tail consumes no deferred reset, leaves the wait state alone, and gets the memory verdict from_memory_preparation_settled(:7600). - An exception above the
try. The guard appends an error row taggedturn_failed. It holds the queue (setup_failed) unless the failure came from a local command's own handler (:7836-7841). - Nested
/prompts get. The outer frame setshanded_off = Truebefore it awaits the nested_run_chat(:9459). The nested call has its own guard and runs the only tail. - Remote-slot refusal. This exit opts out on purpose (
:8203), because a remote slot's queue drains on its peer. GeneratorExit. No tail runs. The PR lists this as a non-goal.- Leftover
chat_doneemitters. Only two remain outside_finish_queue_cycle: the remote refusal and the mid-turn/compactcontinuing=Trueframe (:15642). Neither ends a cycle, so the client still gets one terminal frame per cycle. - Cancel during the tail. Both call sites wrap the tail in
run_to_completion(shield loop,chat_utils.py:248). A cancel is deferred until the hand-off has run, then re-raised. The coroutine is scheduled withensure_future, sohanded_offis set before the guard reads it, even when a cancel lands before the tail's first step. - Race with a send during the payload read.
_finish_queue_cyclereads the payload while the finishing turn still holdsslot.task, and loops until the queue stops moving (:7468-7483). A send that arrives during the read queues behind the turn and then gets the floor. The release has no await betweenslot.task = Noneand the broadcast.
#16020 interaction
_turn_start_priority and all of its _turn_priority call sites are still present. The deferred-reset respawn moved into the tail as schedule_eager_spawn(..., start_priority=outcome.start_priority) (:7717). Dropping that kwarg fails test_start_priority.py::test_a_claimer_passes_its_priority[...-_end_turn_tail-schedule_eager_spawn-outcome.start_priority].
Mutation probes (each reverted; worktree left clean)
| Guard removed | Test that fails |
|---|---|
handed_off = True at tail entry (double hand-off) |
many, e.g. test_a_local_command_runs_the_message_queued_behind_it[/goal status], TestRunChatRefusalFallback::* |
guard finally runs no tail (lost hand-off) |
test_a_local_command_runs_the_message_queued_behind_it[/workflow|/quit|/prompts get ...], test_superseded_replay_abort_runs_the_queued_followup[...], test_aborted_replay_starts_a_pending_synthesis |
outcome recorded just before the tail instead of at the top of finally |
test_a_cancel_cutting_the_finally_short_keeps_the_turns_own_policy |
finally's run_to_completion → plain await |
test_a_cancel_inside_the_tail_still_hands_off_the_queue |
/prompts get outer opt-out |
test_a_nested_prompts_get_turn_hands_off_exactly_once |
| remote refusal opt-out | test_the_remote_slot_refusal_does_not_drain_locally |
setup_failed forced False |
test_a_setup_error_holds_the_queue, test_a_setup_error_shows_an_error_and_a_finished_frame |
| payload re-read loop | test_a_send_during_the_payload_read_queues_and_runs, test_a_send_during_the_read_of_a_held_queue_runs[setup|sign-in] |
GeneratorExit → tail |
test_a_closed_coroutine_runs_no_tail |
guard's run_to_completion → plain await |
no test fails (follow-up 1) |
All 15 changed test files pass at this head (2223 passed, 1 skipped). CI at 424f33ba90: 90 pass, 8 skipped, 0 failing.
Fail-open / fail-closed
- Setup-error hold: fail-closed. The slot is released,
chat_donegoes out, and the queue is kept until the user's next send (the same remedy as the sign-in hold). - Early exit while memory preparation is unsettled: fail-closed. The queue is held and
_arm_parked_queue_drainstarts it once preparation finishes. - Uncancellable tail: fail-open on cancel latency only. The delay is bounded, because the teardown inside uses
skip_if_busy=Trueand the ACP shutdown runs under_TERMINATE_TIMEOUT. The cancel is re-raised afterwards. Base already ran the same consume in the samefinally. I accept the author's rebuttal on the design-review thread. - A tail that raises before
_hand_off_queue: fail-open, back to the stranded state. The guard only logs it. The one step that can realistically raise (_consume_pending_reset) is wrapped intry, so I consider this acceptable.
Non-blocking follow-ups
- The guard's
run_to_completion(:7850) has no test. Turning it into a plain await keeps the suite green. Add a test for a cancel that lands during a tail the guard runs: for example, a local command with a queued entry, cancelled insidechat_done_payload. Without the shield, that exit would leaveslot.taskpointing at a finished task. - The nested
/prompts getturn runs the only tail at depth 1, so_turn_channel_narrowedis not cleared there (:7727). Base behaves the same way (its outer frame returned before anyfinally), and the next depth-0 entry clears it (:8211). This is informational only.
I did not check the frontend or mypy.
_run_chat's try/finally covers a turn only from session admission on. The exits above it returned past the queue drain: a content-filter, model-access or image-history replay cancelled at its consume seam, a blocked or local slash command (/prompts, /goal, /workflow, an unsupported /compact), or an exception in that setup. A message queued behind them never ran, no terminal chat_done went out, and slot.task kept pointing at a finished task, so the composer stayed blocked. _end_turn_tail is now the one ordered list of per-exit duties, ending in the queue hand-off, and it runs once per turn: at the end of the finally, or from the _hands_off_queue_on_exit guard for any exit the finally did not finish. Both sites await it through run_to_completion, so a cancel landing inside the tail is deferred until the hand-off ran. The guard passes the turn a _TurnExit record, skips a closing coroutine, and takes the tail's policy from what the turn recorded: the finally records its outcome (sign-in result, barrier verdict, start priority, AutoNudge facts) before its first await; an exit that never reached the try consumes no deferred reset, retires no wait state, keeps the slot's recorded sign-in hold and asks the memory barrier's own rule; an exception there shows an error row, tagged so the OpenAI-compatible endpoint answers server_error with a code, and holds the queue unless a local command's own handler raised it. Local commands no longer append their own done row. _finish_queue_cycle reads the chat_done payload while the finishing turn still holds the slot and hands a send that queued during the read the floor; a held queue does not count as continuing work in that frame, and titling runs only for a cycle a provider served. _send_chat_done is the runner's one chat_done emitter. The three replay families revoke by one rule (_replay_revocation in chat_turn/recovery.py) at the drain and at the consume seam, and the consume gates only decide: the tail sends the frame, also for the lost-session replay's consume abort. The model-access replay is identified only by its queue id, so the _model_access_recovery_pending latch is gone. An unconsumed sub-agent completion is classified once at its requeue and queued as the completion it is, so only a hard kill discards it. The websocket fan-out payload is typed as WsPayload (a Mapping), so mypy rejects an un-awaited payload coroutine at the call site. Fixes #13824
424f33b to
f34b7b5
Compare
|
|
|
Reviewed, not approving from my side: ~2.5k added / ~800 removed lines across 31 files changing how the chat queue is handed off on every _run_chat exit. That is the core turn loop, where a lost or doubled queued message is easy to miss, and at this size it needs a maintainer's own review. Auto-merge is on, so please make sure a maintainer approves explicitly. |
buluoray
left a comment
There was a problem hiding this comment.
Re-approving at head f34b7b5a2f after the rebase. git range-diff against the head I approved (424f33ba90) shows the commit unchanged except error-code-baseline.json, regenerated against the new main; every source and test file in the PR diff is identical, so my earlier review stands. CI at this head: 95 checks, none red.
Problem / Motivation
Goal: Every way a dashboard chat turn can end hands the floor back: the next queued message runs, the terminal
chat_doneframe goes out, and the composer unblocks._run_chat'stry/finallyonly covers a turn from session admission on. Some exits happen above it: a content-filter, model-access or image-history replay cancelled at its consume seam, a blocked or local slash command (/prompts,/goal,/workflow, an unsupported/compact), and an exception in that setup. Those exits returned without running the queue drain. A message queued behind them never ran, no terminalchat_donewas sent, andslot.taskstill pointed at a finished task, so the composer stayed blocked. The content-filter abort also passed the asyncchat_done_payloadtobroadcast_wswithout awaiting it, so its frame failed to serialize (#13824, #16230).Why it matters
Any user who queues a message behind a local command, or behind a recovery replay that gets cancelled, ends up with a chat that looks busy forever and a queued message that never runs. The only way out is a reload or a Stop.
Not a goal
run_background_turntimeout, a cancel before start)./goalstop-sentinel fix, which go in a separate PR.What changed (motivation → approach → change)
Symptom: early exits stranded the queue. Root cause: the queue hand-off only ran at the end of the main
finally, so it lived where only some exits reach. Fix: every exit runs one ordered tail._end_turn_tail_hands_off_queue_on_exitfinallydid not finish_send_chat_donechat_doneemitter; always awaits the payload_replay_revocationrun_to_completion, so a cancel landing inside the tail is deferred until the hand-off ran, then re-raised._TurnExitrecord. Thefinallyrecords the turn's outcome before its first await, so a second cancel cannot fall back to the early-exit policy. An exit above thetryconsumes no deferred reset or discard and retires no wait state. It keeps the slot's recorded sign-in hold and asks the memory barrier's own rule. An exception there shows an error row (tagged, so the OpenAI-compatible endpoint answersserver_errorwith acode) and holds the queue, unless the failing code was a local command's own handler. A closing coroutine (GeneratorExit) runs no tail._finish_queue_cyclereads thechat_donepayload while the finishing turn still holdsslot.task, then releases it with no await in between. A send that arrives during the read gets the floor. A held queue is not reported as continuing. Titling and the history refresh run only for a cycle a provider served.chat_turn/recovery.pyonly decide; none of them broadcasts. The lost-session replay's consume abort leaves itschat_doneto the tail too. The model-access replay is identified by its queue id, so the_model_access_recovery_pendinglatch is gone.SUBAGENT_COMPLETION_KIND, so a soft Stop, a rebind or a newer message no longer cancels it. A hard kill still discards it.WsPayload(Mapping[str, Any]) on all eight signatures inwebsocket_hub.py,state.pyandeventlog/service.py, so mypy rejects an unawaited payload coroutine at the call site.Specs updated:
learn-cron-dashboard.md(new section "Every turn exit runs the turn's tail", WsPayload note),model-fallback.md,error-handling.md,turn-complete-chime.md,session-summary.md.memory_startup.pyexposesmemory_prepared()for the tail's barrier verdict.chat_utils.chat_done_payloadtakesqueue_held, so a held queue does not read as continuing.error-code-baseline.jsonis re-snapshotted:openai_compat.pyhas one fewer uncoded error.Backwards compatibility
Compatible: no API, config or persisted shape changes. Local slash commands no longer append their own
donerow, because the cycle end now writes it, so the client still sees exactly one.Reader: src/kiro_crew/dashboard/chat_turn/recovery.py:_drop_superseded_model_access_replay -- chat -- the removed
_model_access_recovery_pendinglatch is replaced by the already-recorded_model_access_recovery_queue_id; test_the_drain_names_the_model_access_replay_by_its_queue_idReader: src/kiro_crew/dashboard/chat_turn/recovery.py:_refresh_genuine_turn_allowances -- chat -- reads the
_model_access_replaykwarg the drain passes instead of the latch; test_a_stale_model_access_record_does_not_cancel_the_next_turnReader: src/kiro_crew/dashboard/state.py:_ChatSlot.slots -- chat -- the slot drops the attribute; it was process-local and never persisted
Reader: test/test_model_access_fallback_interactive.py -- chat -- its latch assertions are rewritten against the queue id; test_recovery_record_cleared_when_the_admission_sweep_removes_the_entry
Reader: per-command
donerows in/prompts,/goal,/workflow,/compact-- chat -- the cycle end now writes the onedonerow; test_a_local_command_runs_the_message_queued_behind_itThe
.github/black-baseline.txtentry fortest_goal_command.pyis pruned because that file is now formatted.Tests
test_dashboard_chat.py::TestRunChatEarlyExitHandOff: a local command runs the message queued behind it; a lone local command ends the cycle without titling; a sign-in or memory-preparation hold survives an early exit; a refused command leaves a deferred reset armed; a setup error holds the queue and shows an error plus a finished frame; a failing local command does not hold the queue; a cancel that cuts thefinallyshort keeps the turn's policy; a closed coroutine runs no tail; a nested/prompts gethands off once; the remote-slot refusal does not drain locally; a send during the payload read runs (both drained and held queues) and stays queued while sub-agents run; an early exit leaves a sub-agent's wait state alone; a setup error is aserver_erroron the OpenAI endpoint (blocking and streaming).test_dashboard_chat.py,test_model_access_fallback_interactive.py): a superseded replay abort runs the queued follow-up; the drain names the model-access replay by its queue id; a stale record does not cancel the next turn; an owed completion is replayed as a completion despite a Stop; synthesis turns and runner continuations stay cancellable; a hard kill discards an owed completion; the verbatim mark travels through the queue.test_refusal_replay_consume_abort_chat_done.py: the consume gate decides without a broadcast;_run_chatends a stopped or rebound replay with one awaited frame; a replay superseded while preparing hands the floor to the newer message.test_dashboard_chat.py::TestRunChatEarlyExitHandOff::test_a_cancel_inside_the_tail_still_hands_off_the_queue: a cancel during the tail's deferred-reset consume still runs the queued message.test_start_priority.py,test_active_turn_session_key.py,test_chat_runner_coverage.py: pins follow the tail (the eager respawn's priority comes from the recorded outcome).test_chat_runner_composition_contract.py: the consume gates are await-free phases; the entry signature names the guard'sturn_exitand the two new drain flags.test_wake_judge_feedback.py,test_prompts.py,test_steer_requeue.py,test_goal_command.py,test_workflow_slash_command.py,test_subagent_delivery_ttl_anchor.py,test_chat_completion_sound.py: updated for the single tail and the singledonerow.Each core fix was mutation-checked: at least one test fails when the fix is reverted.
Manual verification
N/A: unit coverage drives
_run_chatend to end through each exit with a fake provider. The change has no UI surface beyond the existingchat_doneframe.Related Issues
Fixes #13824
Also reported as #16230 (closed as a duplicate of #13824).
Pattern harvest
Rule candidate: semgrep
Pattern: an
async defresult passed to a synchronous call withoutawait(herebroadcast_ws(..., chat_done_payload(...))). TheWsPayloadtyping closes this class for the fan-out by typing the payload as aMapping, so mypy rejects a coroutine.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement