Skip to content

fix(dashboard): keep the streamed part of a reply when a shutdown cuts it off - #18789

Merged
javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/shutdown-keeps-streamed-partial-reply
Oct 13, 2026
Merged

javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/shutdown-keeps-streamed-partial-reply

Conversation

@javenciu

@javenciu javenciu commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Keep the part of a dashboard reply the user watched stream when a graceful shutdown cuts the turn off, so the next start shows it with the existing interruption row and Resume button.

A graceful shutdown (Gateway._shutdown) saves every chat slot first (save_all_slots_to_history), then closes the ACP sessions (sessions.close_all()). A dashboard turn that is mid-reply holds its streamed text only as chunk rows at that point. When its session dies, the runner's _persist_partial_reply appends the text as an assistant row, but only in memory. The turn's teardown then skips its own save on purpose while the process is going away, so the in-flight marker survives for the next start (chat_runner.py, the _gateway_shutdown_requested branch before _end_turn_tail). Nothing else saves the slot before os._exit.

The result: after a restart, the transcript holds only the user's question. The next start offers Resume, but the text the user watched stream is gone. _persist_partial_reply's own docstring says "a shutdown that interrupts that await must not also drop the body the user watched stream". That holds for memory and for the crew log (drained before the exit), not for the transcript.

Why it matters

A planned restart (an update, a dashboard restart, systemctl restart) loses whatever a long reply had streamed so far. The user has to ask again and cannot see what the agent had already said. No test pinned the transcript copy.

Not a goal

  • The in-app restart (_restart_gateway) is not changed. It drains with close_all() before it saves, so it already keeps the text (pinned by test/test_inapp_restart_keeps_turn_marker.py).
  • The connection-lost recovery rows the runner adds after a session dies ("⟳ Connection lost — retrying…" and the queued recovery prompt) are not saved by this change. They are written after the new save, so they still exist only in memory, as before. Saving them would show a retry notice at the next start that no retry follows. The save here runs before them on purpose.
  • A turn cut by cancelling its task while the gateway keeps running (closing its tab) is a separate defect with its own change.

What changed (motivation → approach → change)

In short: a dashboard turn that close_all() cuts saves its streamed text as it unwinds. The four paths that save the chat slots, close the sessions and then end the process (the graceful shutdown, the update restart, the dashboard restart and the Slack /kirocrew restart) now wait up to 2 s for that save before they exit or exec. Red on main 0a47021090 with only this branch's three test files added: 6 failed, 16 passed, in each of three runs. Green: those files give 453 passed in each of three runs, and the 499 test files that import a module this branch changes give 31,975 passed, 10 skipped.

  • Motivation: the only save after the partial row is appended is one the shutdown path skips, and the exit or exec that ends the process can beat the turn's own save.
  • Approach: save the slot at the point the partial row exists, keeping the marker, and make every path that closes the sessions and then ends the process wait for that save.
  • Change:
    • dashboard/chat_runner.py, _persist_partial_reply: when _gateway_shutdown_requested(state) (dashboard/chat_runner.py:9119), after appending the partial reply, call save_slot_off_loop(state, slot, force=True, expected_history_key=..., expected_slot_name=...). That is the same save _clear_local_turn_marker runs, without retiring the marker. It runs before the usage write and the recovery rows, and a failure is logged at debug and never raised. The next start's _reconcile_local_turn_marker sees a finished-looking tail and appends its "This turn was interrupted when the app restarted. Review the partial output above, then resume to continue." row after the text.
    • dashboard/chat_persistence.py, wait_for_unwinding_turns (dashboard/chat_persistence.py:453), beside save_all_slots_to_history (:429): one asyncio.wait (:477), bounded by DASHBOARD_TURN_UNWIND_SECS, over the dashboard turn tasks that are still running. It never cancels, and a turn still running at the deadline is left to the exit or exec.
    • slack/gateway.py, _shutdown: awaits the helper as its last step (slack/gateway.py:11782), after the cleanup gather that runs close_all() (:11734), subagent_mgr.close and _stop_memory_startup. Nothing waited for those turns before. Placing it last keeps the earlier steps' share of the deadline. _shutdown stays under _shutdown_and_exit's wait_for(..., GRACEFUL_SHUTDOWN_SECS) (:14453).
    • The three in-process restarts save the slots, then close_all(), then exec or exit, the same order, and none of them waited for the turns either. Each awaits the helper right after close_all():
      • the update restart, _restart_after_update_claimed: the save at slack/gateway.py:12138, close_all() at :12180, wait_for_unwinding_turns at :12184, and the exec, reexec_python_module, at :12226. The wait comes before _stop_mcp_broker() and the final callback drain, so nothing is awaited between that drain and the exec.
      • the dashboard restart, _restart_gateway: the save at dashboard/handlers/updates.py:1762, close_all() at :1773, wait_for_unwinding_turns at :1779, and reexec_python_module at :1826. The Not a goal bullet about this path is wrong: it saves before it closes the sessions, like the others. test/test_inapp_restart_keeps_turn_marker.py passed because its stand-in close_all() awaits the turn itself (test/test_inapp_restart_keeps_turn_marker.py:66). The real one cancels each native turn and waits for that turn's ack (session_lifecycle.py:3064), not for the dashboard turn's task.
      • the Slack /kirocrew restart, _handle_restart: the save at slack/events.py:960, close_all() at :980, wait_for_unwinding_turns at :989, and hard_exit at :1014.
    • gateway_shutdown_budget.py: DASHBOARD_TURN_UNWIND_SECS = GRACEFUL_SHUTDOWN_SECS * 0.2 (2 s) (gateway_shutdown_budget.py:35), read through the module at the point of use.
  • Second commit, the drain during a shutdown. The death arm that saves the cut reply also queues the turn's automatic recovery, and the queue drain at the turn's end (_start_next_queued_turn, dashboard/chat_runner.py:7005) started it even though the process was going away: only the parked drain checked for a shutdown (_parked_queue_drain_admitted, dashboard/chat_runner.py:6871). That recovery turn writes its own in-flight marker (_begin_local_turn_marker, dashboard/chat_runner.py:10358) before get_or_create's closing gate refuses it (dashboard/chat_runner.py:10852, the SessionClosingError arm at :19620), and its teardown keeps that marker during a shutdown. When that write landed before the exit, the next start reconciled the recovery's inject row, so the restart showed no interruption row after the cut reply. _start_next_queued_turn now returns before any dequeue while _gateway_shutdown_requested(state) (dashboard/chat_runner.py:7034), and the queue stays as it is, so the history save keeps it for the next start, as the parked drain already does. Every queue drain, including the turn tail, the parked drain, the dashboard routes and rewind, goes through this function, so no queued entry starts a turn during a shutdown. The armed sub-agent synthesis turn has its own starter in the turn tail (_finish_queue_cycle, dashboard/chat_runner.py:7827) and is not changed here. When a sub-agent batch armed it before the cut, it can still start during a shutdown, and when its marker write lands before the exit, the restart shows the synthesis prompt as the interrupted turn instead of an interruption row after the cut reply. The cut reply's text is saved either way. Gating that starter would drop the synthesis, because its arm is held only in memory (_pending_synthesis, dashboard/state.py:2991) and a completion that was already delivered is not delivered again after a restart. Chosen over skipping the recovery requeue in the death arm alone, which would have left the other queued entries starting.
  • Tests:
    • test/test_local_turn_restart_marker.py::test_a_shutdown_keeps_the_text_the_user_watched_stream: the real _run_chat, shutdown_event set, the graceful save, then the session dies with AcpProcessDied. The restored slot holds the text, reads as interrupted, and has one restart row after the text.
    • test_a_turn_that_finishes_keeps_its_reply_and_clears_the_marker (control): a finished turn restores with its reply, not interrupted, and no restart row.
    • test_a_shutdown_starts_no_recovery_turn_over_the_cut_reply (second commit): the same shutdown, with get_or_create recording each call and holding a second caller at the session gate until the cut turn has ended, then refusing it with SessionClosingError, the way a registration parked behind the shutdown snapshot is refused. The restart shows no [Connection lost — automatic recovery] row, one restart row, and get_or_create was called once. The helper now waits for every turn the drain started to settle before it restores.
    • test/test_slack_gateway.py::TestShutdown::test_shutdown_waits_for_a_dashboard_turn_that_is_unwinding: a turn cut by close_all() and parked mid-unwind has saved before _shutdown returns.
    • test_shutdown_never_cancels_or_outwaits_a_hung_dashboard_turn (control): a hung turn is neither cancelled nor waited on past the budget.
    • test/test_restart_waits_for_cut_turns.py::test_a_restart_waits_for_the_turn_it_cut_before_it_ends, once per restart (update, dashboard, Slack): the cut turn's task records when the restart starts waiting on it (a done callback registered on it, which any wait on a future does) and saves only then. The order is close_all, the wait, the save, then the exec or exit.
    • test_a_restart_never_cancels_or_outwaits_a_hung_turn (control, per restart): with a 0.2 s budget, a turn that never ends is not cancelled, and the restart still reaches its exec or exit.

Backwards compatibility

Compatible: a turn that finishes, or a shutdown or restart with no turn in flight, behaves as before. A shutdown with a turn mid-reply writes one more save and waits at most 2 s of its existing 10 s budget. An update, dashboard or Slack restart with a turn mid-reply waits at most 2 s more before its exec or exit.
Removes nothing: no function, field, config key or test is removed.

Tests

Red, this branch's three test files against the current main (0a47021) with only those files added (-n 0, -k "shutdown or cut_turns", three serial runs, the same each time): 6 failed, 16 passed.

E       AssertionError: the text the user watched stream is gone after the restart: ['explain the plan', '[Connection lost — automatic recovery]\n...']
E       AssertionError: the restart shows the recovery turn's row instead of the cut reply's: ['explain the plan', '[Connection lost — automatic recovery]\n...']
E           AssertionError: shutdown returned before the cut turn saved: ['shutdown returned', 'saved']
E           AssertionError: the restart did not wait for the turn it cut: ['close_all', 'exec', 'saved']
E           AssertionError: the restart did not wait for the turn it cut: ['close_all', 'exec', 'saved']
E           AssertionError: the restart did not wait for the turn it cut: ['close_all', 'exit', 'saved']

On main the restart holds neither the streamed text nor an interruption row: the cut turn's teardown skipped its save, and the recovery turn the drain started wrote its own marker and inject row. The last three are the update, dashboard and Slack restarts: each reached its exec or exit before the turn it cut had saved. Every control passes on both sides. The shutdown test parks the cut turn on an event and releases it only once shutdown has collected the dashboard turns after close_all(), or has already returned; it asserts the order saved, then shutdown returned. The restart tests make the cut turn's task a future that records when the restart registers a done callback on it, and the turn saves only then, so a restart that reads the slots without waiting on them still fails. No sleep and no yield count.

This round's file against the previous head of this branch (d654fb3, three runs): 3 failed, 3 passed, the same three restart failures as above. Removing the wait from one restart fails only that restart's test; making the helper cancel the turns still running at the deadline fails only the three restart controls; dropping its bound fails the three restart controls and the shutdown one.

Red for the second commit, this branch's test/test_local_turn_restart_marker.py against the first commit's source (-n 0, -k shutdown, five serial runs, the same each time): 2 failed, 2 passed.

FAILED test_a_shutdown_keeps_the_text_the_user_watched_stream - assert 0 == 1
E        +  where 0 = len([])
FAILED test_a_shutdown_starts_no_recovery_turn_over_the_cut_reply
E       AssertionError: the restart shows the recovery turn's row instead of the cut reply's: ['explain the plan', 'Part one of the plan is to move the queue first', '[Connection lost — automatic recovery]\n...']

The first failure is the one CI saw on the first commit (about 1 run in 10 on the merge with current main): with the restore now waiting for the drain to settle, it fails every time, because the recovery turn's marker always lands first. Green on this branch, rebased onto that main (-n 0, three runs): the three test files give 453 passed each time. With their four neighbours (test/test_inapp_restart_keeps_turn_marker.py, test/test_abnormal_turn_usage_row_15013.py, test/test_crew_log_order.py, test/test_update_installer_graceful_stop.py) the seven files pass 10 of 10 runs under -n 4 (499 passed each). The two shutdown tests of test_local_turn_restart_marker.py and the six restart tests pass 30 of 30 serial runs. Every test file that imports a module this branch changes (499 files) gives 31,975 passed, 10 skipped under -n 16.

Static gates on the head: mypy, flake8, isort, black (gateway_shutdown_budget.py and slack/events.py are in the black baseline and are not reformatted), scripts/check_black_formatting.py, COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py, ./scripts/docs-lint.sh, scripts/check_subprocess_encoding.py, scripts/check_sync_io_in_async.py, scripts/check_memory_store_seam.py and scripts/check_harness_parity.py all exit 0.

Manual verification

The reproducer for this report runs the shutdown order on a fake client and reads the transcript the next boot reads. On main: the body is in memory after the unwind and not on disk; the restored chat holds only the user's row. On this branch: the restored chat holds the text, then the restart row, with Resume offered.

Related Issues

Overlap: four open PRs touch these files (#7163 in chat_runner.py; #11678 in chat_runner.py and chat_persistence.py; #9052 in slack/gateway.py and gateway_shutdown_budget.py; #11794 in those two and dashboard/handlers/updates.py). git merge-tree of each head against this branch's head and against its base (0a47021) gives the same conflicted paths for #7163 (34), #11678 (34) and #11794 (none). #9052 conflicts in one place, where both append a last step to _shutdown (slack/gateway.py:11782); keeping both, with the wait first, leaves its service published while the cut turns unwind.

Related: #13112 and its fix #13113 added the turn-in-flight marker, for a partial reply that was already a row on disk. #15128 kept the marker across the in-app restart. Neither covers text that exists only as chunks when the graceful save runs. No open issue or PR reports this (searched _persist_partial_reply, is_turn_interrupted, local turn marker).

Pattern harvest

Rule candidate: review-prompt
Pattern: a shutdown that saves state first and then ends the work that state describes, so whatever the work records while ending is never saved.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): not applicable
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 10, 2026 08:37
@javenciu
javenciu requested a review from buluoray October 10, 2026 08:37
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision c730725dd2d90ddee35dfa4e6a64f015cbe0ae08 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of c730725dd2d90ddee35dfa4e6a64f015cbe0ae08 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the residual gap the description admits is tracked: an armed sub-agent synthesis turn can still start during shutdown and replace the cut reply's interruption row.

[DESIGN-REVIEWED] c730725

Description read

[DESCRIPTION-READ] 6941dfd66c94a2237629e6cc6027b8159a3356147a7600c0dbd68642e0ab4055

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of c730725dd2d90ddee35dfa4e6a64f015cbe0ae08 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check before merge: the armed sub-agent synthesis turn (_finish_queue_cycle) can still start during a shutdown. The description says it leaves this unfixed.

What this change ships

Inventory (5 items) — 5 justified

Intent: keep the part of a dashboard reply the user watched stream when a graceful shutdown or in-process restart cuts the turn off, so the next start shows it with the interruption row and the Resume button. This is a FIX.

  1. Text from a reply cut by shutdown is saved to the transcript as the turn unwinds, with the in-flight marker kept — justified
  2. Graceful shutdown waits up to 2 s for cut dashboard turns to finish unwinding before exit — justified
  3. The update restart, the dashboard restart and the Slack /kirocrew restart wait the same way before exec or exit. These are all the close_all() callers that hold dashboard state; the fourth, cli_server.py:3155, has none — justified
  4. New budget constant DASHBOARD_TURN_UNWIND_SECS, set to 20% of the graceful-shutdown budget (2 s) — justified
  5. Queued turns, including the automatic recovery, no longer start while the process is shutting down, and the queue is kept for the next start — justified

[FIRST-PRINCIPLES-REVIEWED] c730725

Description read

[DESCRIPTION-READ] 991dc1a9fbc184c05e3dc41a334148a1c23ff2f8e94ec649b72d52fe916f26d7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed c730725dd2d90ddee35dfa4e6a64f015cbe0ae08 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/dashboard/chat_persistence.py:475 -- Function-local "from kiro_crew import gateway_shutdown_budget" hides a mandatory, non-cyclic dependency, violating top-level-imports -> Fix: move this import to the top-level import block while retaining the call-time constant lookup.
[GPT-REVIEWED] c730725

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed c730725dd2d90ddee35dfa4e6a64f015cbe0ae08 via the fork AI-review pipeline; updated in place on each push.

Review details

Not blocking: one advisory finding.

FINDING — src/kiro_crew/dashboard/chat_runner.py:7034 — the new if _gateway_shutdown_requested(state): return False only gates the queue drain. When a turn has synthesis armed and is cut by close_all(), it still reaches _hand_off_queue → _finish_queue_cycle (will_synthesize on a CLEAR verdict) → _launch_synthesis → _run_pending_synthesis. That path calls slot.append("inject", SUBAGENT_SYNTHESIS_PROMPT, ...), then a _run_chat that writes its own in-flight marker before the session gate refuses it. The restart then shows the synthesis row instead of the cut reply's interruption row and Resume, which is the outcome this PR's new test forbids for the recovery successor. → Fix: also check _gateway_shutdown_requested(state) before _launch_synthesis in _finish_queue_cycle, or at the top of _run_pending_synthesis. Both are outside this PR's changed lines.

[OPUS-REVIEWED] c730725

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 10, 2026
@javenciu
javenciu force-pushed the fix/shutdown-keeps-streamed-partial-reply branch from 334efaf to 6addcb7 Compare October 11, 2026 08:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@javenciu
javenciu force-pushed the fix/shutdown-keeps-streamed-partial-reply branch from 65f54d6 to d654fb3 Compare October 11, 2026 12:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR keeps a cut dashboard reply's streamed text across a SIGTERM shutdown. It saves the slot in _persist_partial_reply while _gateway_shutdown_requested, waits up to DASHBOARD_TURN_UNWIND_SECS (2 s, never cancelling) at the end of _shutdown, and stops _start_next_queued_turn from starting a recovery turn during shutdown. For the signal path the fix is correct and well proven: tests are red on main and green here, there are controls, and the hung-turn test shows the wait is bounded. The fix is incomplete, though, and the description claims more than it fixes. "Why it matters" names "an update" as a fixed case. But _restart_after_update_claimed (slack/gateway.py:12053) saves the slots, runs close_all() (:12105), stops the broker, drains callbacks and execs, and never waits for dashboard turn tasks. The new partial save does run there (final_drain_started makes the predicate true), but the exec can beat it, and the turn's text is lost exactly as before. The First Principles lane raised this on head d654fb3 and it has no disposition. Fix: add the same bounded asyncio.wait over unfinished dashboard_state._slots[*].task right after close_all() in _restart_after_update_claimed, ideally through one shared helper with _shutdown, plus a test. Or name that path as deferred in Not a goal and drop "an update" from Why it matters. — crew-pr-reviewer (deep re-check)

@javenciu
javenciu force-pushed the fix/shutdown-keeps-streamed-partial-reply branch from d654fb3 to fba47ea Compare October 12, 2026 09:34
@javenciu

Copy link
Copy Markdown
Contributor Author

All three points are fixed in code, not deferred.

  • The update restart (_restart_after_update_claimed). The bounded wait is now one helper, wait_for_unwinding_turns (dashboard/chat_persistence.py:453), next to save_all_slots_to_history. _shutdown awaits it as its last step (slack/gateway.py:11756). The update restart awaits it right after close_all() (slack/gateway.py:12158), before the broker stop and the final callback drain, so nothing is awaited between that drain and the exec.
  • The same order on two more paths. _restart_gateway and the Slack /kirocrew restart (_handle_restart) also save, then close_all(), then exec or exit, with no wait. The Not a goal bullet saying _restart_gateway drains before it saves was wrong: it saves at dashboard/handlers/updates.py:1761 and closes at :1772. test/test_inapp_restart_keeps_turn_marker.py passed only because its stand-in close_all() awaits the turn itself (test/test_inapp_restart_keeps_turn_marker.py:66). The real one's drain_active_turns waits for the native turn's ack (session_lifecycle.py:3064), not for the dashboard turn's task. Both paths now await wait_for_unwinding_turns right after closing the sessions (dashboard/handlers/updates.py:1778, slack/events.py:989).
  • "An update" in Why it matters. With the update restart waiting, that sentence holds, as does "a dashboard restart", so the section is unchanged.
  • The First Principles concern. Resolved by the fix above rather than by a deferral: the update path is covered and tested, and so is the path it counted as already draining first.

Tests: test/test_restart_waits_for_cut_turns.py runs each of the three restarts with a cut turn whose task records when the restart starts waiting on it, and saves only then. It asserts close_all, the wait, the save, then the exec or exit. A control per restart checks that a hung turn is not cancelled and that the restart still ends at the budget. Red on the previous head d654fb3 and on main 0a47021, three runs each: 3 failed, 3 passed (the three restart tests fail, the controls pass). Removing the wait from one restart fails only that restart's test. Green: the three test files give 453 passed in each of three runs, and the 499 test files that import a changed module give 31,975 passed.

The branch is rebased onto main 0a47021 with no conflicts. Change, Tests and Related Issues in the description are updated; #9052 meets this branch in one place, where both add a last step to _shutdown.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu
javenciu force-pushed the fix/shutdown-keeps-streamed-partial-reply branch from fba47ea to f318862 Compare October 12, 2026 14:49
@javenciu

Copy link
Copy Markdown
Contributor Author

Amended the PR's second commit, so the PR keeps its two commits (PR Hygiene allows at most two): the 10 s and 20 s ceilings in test/test_local_turn_restart_marker.py, test/test_restart_waits_for_cut_turns.py and test/test_slack_gateway.py are now 30 s and the two 10 ms poll loops wait with async_wait_until, because the repo's wait-discipline ratchet (test/test_wait_discipline_ratchet.py) rejects both.
The three files counted 9, 4 and 16 short fixed waits and count 2, 0 and 9 again, their baselines; the ratchet passes on the head merged with main (c4c51aa), and the three files pass 3 of 3 runs (453 tests each).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
…s it off

A graceful shutdown saves every chat slot first, then closes the ACP
sessions. A dashboard turn that was mid-reply only appends the text its
user watched stream when its session dies, after that save, and its
teardown skips its own save while the process is going away so the
in-flight marker survives. Nothing else saved the row, so the next start
showed the user's question with no answer and the streamed text was gone.

_persist_partial_reply now saves the slot right after appending the
partial reply when the gateway is going away, keeping the in-flight
marker, so the next start shows the text with its interruption row and
the Resume button. The shutdown also waits, bounded by
DASHBOARD_TURN_UNWIND_SECS and without cancelling, for dashboard turns
that are unwinding after the sessions close, so the hard exit does not
beat that save.

The update restart, the dashboard restart and the Slack `/kirocrew
restart` close the sessions in the same order and then exec or exit, so
the bounded wait is one helper, `wait_for_unwinding_turns` in
`dashboard/chat_persistence.py`, and all four paths await it right after
`close_all()`.
The death arm that saves a cut reply also queues the turn's automatic
recovery, and the queue drain at the turn's end started it even though the
process was going away. That recovery turn writes its own in-flight marker
before `get_or_create`'s closing gate refuses it, and its teardown keeps
that marker during a shutdown. When the write landed before the exit, the
next start reconciled the recovery's inject row instead of the cut reply's
interruption row, so the restart offered no Resume for the text the user
had watched stream.

`_start_next_queued_turn` now returns before any dequeue while
`_gateway_shutdown_requested(state)`, as the parked drain already does
(`_parked_queue_drain_admitted`), and the queue stays as it is, so the
history save keeps it for the next start. Every queue drain goes through
this function, so no queued entry starts a turn during a shutdown.

Tests: `test_a_shutdown_starts_no_recovery_turn_over_the_cut_reply` holds
the successor at the session gate after its marker write, the way a
registration parks behind the shutdown snapshot, and checks the restart
shows the cut reply's interruption row and that no second turn started.
The shutdown test helper now waits for every turn the drain started to
settle before it restores, which makes the existing test fail
deterministically without the gate.
@javenciu
javenciu force-pushed the fix/shutdown-keeps-streamed-partial-reply branch from f318862 to c730725 Compare October 12, 2026 20:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu
javenciu enabled auto-merge (squash) October 12, 2026 22:49
@javenciu
javenciu requested a review from iamwhatever October 12, 2026 22:49

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This keeps the text a dashboard reply had streamed when a graceful shutdown or restart cuts the turn off: _persist_partial_reply saves the slot (marker kept) while _gateway_shutdown_requested, and every path that saves slots, runs close_all() and then exits or execs waits for that save through one bounded, never-cancelling helper, which fixes the root cause (the save ran before the work it described had ended). Head c730725 meets my change request point by point: wait_for_unwinding_turns (dashboard/chat_persistence.py) is the single shared helper, and it is awaited right after close_all() in _restart_after_update_claimed (slack/gateway.py), in _restart_gateway (dashboard/handlers/updates.py) and in the Slack _handle_restart (slack/events.py), plus as the last step of _shutdown; the description now corrects its wrong Not a goal bullet instead of claiming more than it fixes. I checked that the save actually fires on the restart paths: _gateway_shutdown_requested is shutdown_event.is_set() or sessions.final_drain_started, so it is true after close_all() on all four. I tried to break it: a hung turn is neither cancelled nor waited past DASHBOARD_TURN_UNWIND_SECS (2 s of the 10 s budget), and the restart still reaches its exec (control per path); the new _start_next_queued_turn early return only defers a queue the history save already kept, so no queued work is dropped. The per-restart tests make the cut turn save only once the restart waits on its task, so a restart without the wait fails them, and the author reports them red on the previous head. All AI lanes PASS, checks are green and the PR is MERGEABLE on main. — crew-pr-reviewer (deep re-check)

@javenciu
javenciu merged commit 4031ee2 into kirodotdev:main Oct 13, 2026
106 of 114 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants