Repository navigation
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the residual gap the description admits is tracked: an armed sub-agent synthesis turn can still start during shutdown and replace the cut reply's interruption row. [DESIGN-REVIEWED] c730725 Description read[DESCRIPTION-READ] 6941dfd66c94a2237629e6cc6027b8159a3356147a7600c0dbd68642e0ab4055 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check before merge: the armed sub-agent synthesis turn ( What this change shipsInventory (5 items) — 5 justifiedIntent: keep the part of a dashboard reply the user watched stream when a graceful shutdown or in-process restart cuts the turn off, so the next start shows it with the interruption row and the Resume button. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] c730725 Description read[DESCRIPTION-READ] 991dc1a9fbc184c05e3dc41a334148a1c23ff2f8e94ec649b72d52fe916f26d7 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/dashboard/chat_persistence.py:475 -- Function-local |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNot blocking: one advisory finding. FINDING — src/kiro_crew/dashboard/chat_runner.py:7034 — the new [OPUS-REVIEWED] c730725 |
334efaf to
6addcb7
Compare
65f54d6 to
d654fb3
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This PR keeps a cut dashboard reply's streamed text across a SIGTERM shutdown. It saves the slot in _persist_partial_reply while _gateway_shutdown_requested, waits up to DASHBOARD_TURN_UNWIND_SECS (2 s, never cancelling) at the end of _shutdown, and stops _start_next_queued_turn from starting a recovery turn during shutdown. For the signal path the fix is correct and well proven: tests are red on main and green here, there are controls, and the hung-turn test shows the wait is bounded. The fix is incomplete, though, and the description claims more than it fixes. "Why it matters" names "an update" as a fixed case. But _restart_after_update_claimed (slack/gateway.py:12053) saves the slots, runs close_all() (:12105), stops the broker, drains callbacks and execs, and never waits for dashboard turn tasks. The new partial save does run there (final_drain_started makes the predicate true), but the exec can beat it, and the turn's text is lost exactly as before. The First Principles lane raised this on head d654fb3 and it has no disposition. Fix: add the same bounded asyncio.wait over unfinished dashboard_state._slots[*].task right after close_all() in _restart_after_update_claimed, ideally through one shared helper with _shutdown, plus a test. Or name that path as deferred in Not a goal and drop "an update" from Why it matters. — crew-pr-reviewer (deep re-check)
d654fb3 to
fba47ea
Compare
|
All three points are fixed in code, not deferred.
Tests: The branch is rebased onto main 0a47021 with no conflicts. Change, Tests and Related Issues in the description are updated; #9052 meets this branch in one place, where both add a last step to |
fba47ea to
f318862
Compare
|
Amended the PR's second commit, so the PR keeps its two commits (PR Hygiene allows at most two): the 10 s and 20 s ceilings in test/test_local_turn_restart_marker.py, test/test_restart_waits_for_cut_turns.py and test/test_slack_gateway.py are now 30 s and the two 10 ms poll loops wait with async_wait_until, because the repo's wait-discipline ratchet (test/test_wait_discipline_ratchet.py) rejects both. |
…s it off A graceful shutdown saves every chat slot first, then closes the ACP sessions. A dashboard turn that was mid-reply only appends the text its user watched stream when its session dies, after that save, and its teardown skips its own save while the process is going away so the in-flight marker survives. Nothing else saved the row, so the next start showed the user's question with no answer and the streamed text was gone. _persist_partial_reply now saves the slot right after appending the partial reply when the gateway is going away, keeping the in-flight marker, so the next start shows the text with its interruption row and the Resume button. The shutdown also waits, bounded by DASHBOARD_TURN_UNWIND_SECS and without cancelling, for dashboard turns that are unwinding after the sessions close, so the hard exit does not beat that save. The update restart, the dashboard restart and the Slack `/kirocrew restart` close the sessions in the same order and then exec or exit, so the bounded wait is one helper, `wait_for_unwinding_turns` in `dashboard/chat_persistence.py`, and all four paths await it right after `close_all()`.
The death arm that saves a cut reply also queues the turn's automatic recovery, and the queue drain at the turn's end started it even though the process was going away. That recovery turn writes its own in-flight marker before `get_or_create`'s closing gate refuses it, and its teardown keeps that marker during a shutdown. When the write landed before the exit, the next start reconciled the recovery's inject row instead of the cut reply's interruption row, so the restart offered no Resume for the text the user had watched stream. `_start_next_queued_turn` now returns before any dequeue while `_gateway_shutdown_requested(state)`, as the parked drain already does (`_parked_queue_drain_admitted`), and the queue stays as it is, so the history save keeps it for the next start. Every queue drain goes through this function, so no queued entry starts a turn during a shutdown. Tests: `test_a_shutdown_starts_no_recovery_turn_over_the_cut_reply` holds the successor at the session gate after its marker write, the way a registration parks behind the shutdown snapshot, and checks the restart shows the cut reply's interruption row and that no second turn started. The shutdown test helper now waits for every turn the drain started to settle before it restores, which makes the existing test fail deterministically without the gate.
f318862 to
c730725
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This keeps the text a dashboard reply had streamed when a graceful shutdown or restart cuts the turn off: _persist_partial_reply saves the slot (marker kept) while _gateway_shutdown_requested, and every path that saves slots, runs close_all() and then exits or execs waits for that save through one bounded, never-cancelling helper, which fixes the root cause (the save ran before the work it described had ended). Head c730725 meets my change request point by point: wait_for_unwinding_turns (dashboard/chat_persistence.py) is the single shared helper, and it is awaited right after close_all() in _restart_after_update_claimed (slack/gateway.py), in _restart_gateway (dashboard/handlers/updates.py) and in the Slack _handle_restart (slack/events.py), plus as the last step of _shutdown; the description now corrects its wrong Not a goal bullet instead of claiming more than it fixes. I checked that the save actually fires on the restart paths: _gateway_shutdown_requested is shutdown_event.is_set() or sessions.final_drain_started, so it is true after close_all() on all four. I tried to break it: a hung turn is neither cancelled nor waited past DASHBOARD_TURN_UNWIND_SECS (2 s of the 10 s budget), and the restart still reaches its exec (control per path); the new _start_next_queued_turn early return only defers a queue the history save already kept, so no queued work is dropped. The per-restart tests make the cut turn save only once the restart waits on its task, so a restart without the wait fails them, and the author reports them red on the previous head. All AI lanes PASS, checks are green and the PR is MERGEABLE on main. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: Keep the part of a dashboard reply the user watched stream when a graceful shutdown cuts the turn off, so the next start shows it with the existing interruption row and Resume button.
A graceful shutdown (
Gateway._shutdown) saves every chat slot first (save_all_slots_to_history), then closes the ACP sessions (sessions.close_all()). A dashboard turn that is mid-reply holds its streamed text only as chunk rows at that point. When its session dies, the runner's_persist_partial_replyappends the text as an assistant row, but only in memory. The turn's teardown then skips its own save on purpose while the process is going away, so the in-flight marker survives for the next start (chat_runner.py, the_gateway_shutdown_requestedbranch before_end_turn_tail). Nothing else saves the slot beforeos._exit.The result: after a restart, the transcript holds only the user's question. The next start offers Resume, but the text the user watched stream is gone.
_persist_partial_reply's own docstring says "a shutdown that interrupts that await must not also drop the body the user watched stream". That holds for memory and for the crew log (drained before the exit), not for the transcript.Why it matters
A planned restart (an update, a dashboard restart,
systemctl restart) loses whatever a long reply had streamed so far. The user has to ask again and cannot see what the agent had already said. No test pinned the transcript copy.Not a goal
_restart_gateway) is not changed. It drains withclose_all()before it saves, so it already keeps the text (pinned bytest/test_inapp_restart_keeps_turn_marker.py).What changed (motivation → approach → change)
In short: a dashboard turn that
close_all()cuts saves its streamed text as it unwinds. The four paths that save the chat slots, close the sessions and then end the process (the graceful shutdown, the update restart, the dashboard restart and the Slack/kirocrew restart) now wait up to 2 s for that save before they exit or exec. Red on main0a47021090with only this branch's three test files added: 6 failed, 16 passed, in each of three runs. Green: those files give 453 passed in each of three runs, and the 499 test files that import a module this branch changes give 31,975 passed, 10 skipped.dashboard/chat_runner.py,_persist_partial_reply: when_gateway_shutdown_requested(state)(dashboard/chat_runner.py:9119), after appending the partial reply, callsave_slot_off_loop(state, slot, force=True, expected_history_key=..., expected_slot_name=...). That is the same save_clear_local_turn_markerruns, without retiring the marker. It runs before the usage write and the recovery rows, and a failure is logged at debug and never raised. The next start's_reconcile_local_turn_markersees a finished-looking tail and appends its "This turn was interrupted when the app restarted. Review the partial output above, then resume to continue." row after the text.dashboard/chat_persistence.py,wait_for_unwinding_turns(dashboard/chat_persistence.py:453), besidesave_all_slots_to_history(:429): oneasyncio.wait(:477), bounded byDASHBOARD_TURN_UNWIND_SECS, over the dashboard turn tasks that are still running. It never cancels, and a turn still running at the deadline is left to the exit or exec.slack/gateway.py,_shutdown: awaits the helper as its last step (slack/gateway.py:11782), after the cleanup gather that runsclose_all()(:11734),subagent_mgr.closeand_stop_memory_startup. Nothing waited for those turns before. Placing it last keeps the earlier steps' share of the deadline._shutdownstays under_shutdown_and_exit'swait_for(..., GRACEFUL_SHUTDOWN_SECS)(:14453).close_all(), then exec or exit, the same order, and none of them waited for the turns either. Each awaits the helper right afterclose_all():_restart_after_update_claimed: the save atslack/gateway.py:12138,close_all()at:12180,wait_for_unwinding_turnsat:12184, and the exec,reexec_python_module, at:12226. The wait comes before_stop_mcp_broker()and the final callback drain, so nothing is awaited between that drain and the exec._restart_gateway: the save atdashboard/handlers/updates.py:1762,close_all()at:1773,wait_for_unwinding_turnsat:1779, andreexec_python_moduleat:1826. The Not a goal bullet about this path is wrong: it saves before it closes the sessions, like the others.test/test_inapp_restart_keeps_turn_marker.pypassed because its stand-inclose_all()awaits the turn itself (test/test_inapp_restart_keeps_turn_marker.py:66). The real one cancels each native turn and waits for that turn's ack (session_lifecycle.py:3064), not for the dashboard turn's task./kirocrew restart,_handle_restart: the save atslack/events.py:960,close_all()at:980,wait_for_unwinding_turnsat:989, andhard_exitat:1014.gateway_shutdown_budget.py:DASHBOARD_TURN_UNWIND_SECS = GRACEFUL_SHUTDOWN_SECS * 0.2(2 s) (gateway_shutdown_budget.py:35), read through the module at the point of use._start_next_queued_turn,dashboard/chat_runner.py:7005) started it even though the process was going away: only the parked drain checked for a shutdown (_parked_queue_drain_admitted,dashboard/chat_runner.py:6871). That recovery turn writes its own in-flight marker (_begin_local_turn_marker,dashboard/chat_runner.py:10358) beforeget_or_create's closing gate refuses it (dashboard/chat_runner.py:10852, theSessionClosingErrorarm at:19620), and its teardown keeps that marker during a shutdown. When that write landed before the exit, the next start reconciled the recovery's inject row, so the restart showed no interruption row after the cut reply._start_next_queued_turnnow returns before any dequeue while_gateway_shutdown_requested(state)(dashboard/chat_runner.py:7034), and the queue stays as it is, so the history save keeps it for the next start, as the parked drain already does. Every queue drain, including the turn tail, the parked drain, the dashboard routes and rewind, goes through this function, so no queued entry starts a turn during a shutdown. The armed sub-agent synthesis turn has its own starter in the turn tail (_finish_queue_cycle,dashboard/chat_runner.py:7827) and is not changed here. When a sub-agent batch armed it before the cut, it can still start during a shutdown, and when its marker write lands before the exit, the restart shows the synthesis prompt as the interrupted turn instead of an interruption row after the cut reply. The cut reply's text is saved either way. Gating that starter would drop the synthesis, because its arm is held only in memory (_pending_synthesis,dashboard/state.py:2991) and a completion that was already delivered is not delivered again after a restart. Chosen over skipping the recovery requeue in the death arm alone, which would have left the other queued entries starting.test/test_local_turn_restart_marker.py::test_a_shutdown_keeps_the_text_the_user_watched_stream: the real_run_chat,shutdown_eventset, the graceful save, then the session dies withAcpProcessDied. The restored slot holds the text, reads as interrupted, and has one restart row after the text.test_a_turn_that_finishes_keeps_its_reply_and_clears_the_marker(control): a finished turn restores with its reply, not interrupted, and no restart row.test_a_shutdown_starts_no_recovery_turn_over_the_cut_reply(second commit): the same shutdown, withget_or_createrecording each call and holding a second caller at the session gate until the cut turn has ended, then refusing it withSessionClosingError, the way a registration parked behind the shutdown snapshot is refused. The restart shows no[Connection lost — automatic recovery]row, one restart row, andget_or_createwas called once. The helper now waits for every turn the drain started to settle before it restores.test/test_slack_gateway.py::TestShutdown::test_shutdown_waits_for_a_dashboard_turn_that_is_unwinding: a turn cut byclose_all()and parked mid-unwind has saved before_shutdownreturns.test_shutdown_never_cancels_or_outwaits_a_hung_dashboard_turn(control): a hung turn is neither cancelled nor waited on past the budget.test/test_restart_waits_for_cut_turns.py::test_a_restart_waits_for_the_turn_it_cut_before_it_ends, once per restart (update, dashboard, Slack): the cut turn's task records when the restart starts waiting on it (a done callback registered on it, which any wait on a future does) and saves only then. The order isclose_all, the wait, the save, then the exec or exit.test_a_restart_never_cancels_or_outwaits_a_hung_turn(control, per restart): with a 0.2 s budget, a turn that never ends is not cancelled, and the restart still reaches its exec or exit.Backwards compatibility
Compatible: a turn that finishes, or a shutdown or restart with no turn in flight, behaves as before. A shutdown with a turn mid-reply writes one more save and waits at most 2 s of its existing 10 s budget. An update, dashboard or Slack restart with a turn mid-reply waits at most 2 s more before its exec or exit.
Removes nothing: no function, field, config key or test is removed.
Tests
Red, this branch's three test files against the current
main(0a47021) with only those files added (-n 0,-k "shutdown or cut_turns", three serial runs, the same each time): 6 failed, 16 passed.On
mainthe restart holds neither the streamed text nor an interruption row: the cut turn's teardown skipped its save, and the recovery turn the drain started wrote its own marker and inject row. The last three are the update, dashboard and Slack restarts: each reached its exec or exit before the turn it cut had saved. Every control passes on both sides. The shutdown test parks the cut turn on an event and releases it only once shutdown has collected the dashboard turns afterclose_all(), or has already returned; it asserts the ordersaved, thenshutdown returned. The restart tests make the cut turn's task a future that records when the restart registers a done callback on it, and the turn saves only then, so a restart that reads the slots without waiting on them still fails. No sleep and no yield count.This round's file against the previous head of this branch (d654fb3, three runs): 3 failed, 3 passed, the same three restart failures as above. Removing the wait from one restart fails only that restart's test; making the helper cancel the turns still running at the deadline fails only the three restart controls; dropping its bound fails the three restart controls and the shutdown one.
Red for the second commit, this branch's
test/test_local_turn_restart_marker.pyagainst the first commit's source (-n 0,-k shutdown, five serial runs, the same each time): 2 failed, 2 passed.The first failure is the one CI saw on the first commit (about 1 run in 10 on the merge with current main): with the restore now waiting for the drain to settle, it fails every time, because the recovery turn's marker always lands first. Green on this branch, rebased onto that
main(-n 0, three runs): the three test files give 453 passed each time. With their four neighbours (test/test_inapp_restart_keeps_turn_marker.py,test/test_abnormal_turn_usage_row_15013.py,test/test_crew_log_order.py,test/test_update_installer_graceful_stop.py) the seven files pass 10 of 10 runs under-n 4(499 passed each). The two shutdown tests oftest_local_turn_restart_marker.pyand the six restart tests pass 30 of 30 serial runs. Every test file that imports a module this branch changes (499 files) gives 31,975 passed, 10 skipped under-n 16.Static gates on the head: mypy, flake8, isort, black (
gateway_shutdown_budget.pyandslack/events.pyare in the black baseline and are not reformatted),scripts/check_black_formatting.py,COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py,./scripts/docs-lint.sh,scripts/check_subprocess_encoding.py,scripts/check_sync_io_in_async.py,scripts/check_memory_store_seam.pyandscripts/check_harness_parity.pyall exit 0.Manual verification
The reproducer for this report runs the shutdown order on a fake client and reads the transcript the next boot reads. On main: the body is in memory after the unwind and not on disk; the restored chat holds only the user's row. On this branch: the restored chat holds the text, then the restart row, with Resume offered.
Related Issues
Overlap: four open PRs touch these files (#7163 in
chat_runner.py; #11678 inchat_runner.pyandchat_persistence.py; #9052 inslack/gateway.pyandgateway_shutdown_budget.py; #11794 in those two anddashboard/handlers/updates.py).git merge-treeof each head against this branch's head and against its base (0a47021) gives the same conflicted paths for #7163 (34), #11678 (34) and #11794 (none). #9052 conflicts in one place, where both append a last step to_shutdown(slack/gateway.py:11782); keeping both, with the wait first, leaves its service published while the cut turns unwind.Related: #13112 and its fix #13113 added the turn-in-flight marker, for a partial reply that was already a row on disk. #15128 kept the marker across the in-app restart. Neither covers text that exists only as chunks when the graceful save runs. No open issue or PR reports this (searched
_persist_partial_reply,is_turn_interrupted,local turn marker).Pattern harvest
Rule candidate: review-prompt
Pattern: a shutdown that saves state first and then ends the work that state describes, so whatever the work records while ending is never saved.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)