Repository navigation
fix(dashboard): persist a turn-in-flight marker so a restart mid-turn shows Resume - #13113
Conversation
|
Intent: Make a gateway restart that lands mid-turn show the existing Turn interrupted · Resume treatment even when the dead turn had already streamed a partial reply and completed tool rows, by recording turn admission durably instead of inferring liveness from the transcript. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Description's [DESIGN-REVIEWED] 94a8111 |
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
Also actionable, not an inventory item: the What this change shipsInventory (9 items) — 8 justifiedIntent: make a gateway restart that kills an in-flight local turn show the existing "Turn interrupted · Resume" treatment even when the dead turn already streamed output — a FIX (live repro #13112, RED-on-main tests).
[FIRST-PRINCIPLES-REVIEWED] 94a8111 |
UX Review (Fable 5, fork) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS No first-time reader has seen the new interruption row or the cron/nudge-triggered Resume states; every element is screenshot-covered, so this is the lane's cap, not a gap. Watch
[UX-REVIEWED] 94a8111 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/history.py:221 -- |
Opus 5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo blocking defects; one advisory finding. FINDING — src/kiro_crew/dashboard/chat_persistence.py:~316 — the recovered opener is appended as [OPUS-REVIEWED] 94a8111 |
0c97124 to
8b1bfec
Compare
|
|
|
8b1bfec to
6acfa22
Compare
|
self-added: yes
|
|
6ef5978 to
783f813
Compare
|
Head F1 (GPT, F2 (GPT / Opus / Design / FP, On the Design / FP subtraction (route through Local on this head: 1178 passed across the six touched test files; 86 in the marker and interruption suites; tsc, eslint and the 105-test ChatSlice file pass; black gate, flake8, isort, mypy and the comment-history gate clean. Re-sync budget unchanged at 2 of 3. |
783f813 to
67e06bd
Compare
|
Head GPT F1 (fenced, Writing that test exposed a sibling gap: when the loaded window holds only tool rows plus the restart's error row, GPT F2 and Opus BLOCKING ( Opus FINDING ( Backend shard 5 ( Local on this head: 1182 passed across the six touched test files; 90 in the marker and classifier suites; tsc, eslint and the 106-test ChatSlice file pass; black gate, flake8, isort, mypy and the comment-history gate clean. Re-sync budget unchanged at 2 of 3. |
67e06bd to
c0c37cf
Compare
|
Head |
Security Scope Review (fork) — 🔴 BLOCK (blocking)Which legitimate operations does no leg reported, and this was not the nothing-to-adjudicate short circuit No |
|
Head GPT F1 (fenced, GPT F2 ( Round-12 shard reds -- none this round; shard 5 and Windows shard 7 both passed on Local on this head: 1184 passed across the six touched test files; 92 in the marker and classifier suites; tsc, eslint and the ChatSlice file pass; black gate, flake8, isort, mypy and the comment-history gate clean. Re-sync budget 3 of 3 used (round 12); this push is same-base. |
|
Head GPT F1 ( GPT F2 (fenced, Opus FINDING ( Round-13 shard reds. Backend 3.12 shard 3 and Windows shard 3 fail Local on this head: 1186 passed across the six touched test files; 94 in the marker and classifier suites; black gate, flake8, isort, mypy and the comment-history gate clean. Re-sync budget stays at 3 of 3 (round 12); this push is same-base. |
|
Head Three reds on
|
|
Head Round 15 on Local on this head: 1206 passed across the touched suites plus |
|
Head GPT F1 (fenced, Windows shard 3 on Local on this head: 1188 passed across the six touched test files; 96 in the marker and classifier suites; 107 in the ChatSlice file; tsc, eslint, black gate, flake8, isort, mypy and the comment-history gate clean. |
… shows Resume A gateway force exit during a turn runs no finally, so the transcript ends in user -> assistant(partial) -> tool rows with no error row. That tail is shape-identical to a finished answer, and the restore heuristic (is_turn_interrupted / selectTurnInterrupted) reads it as finished: the session comes back with no "Turn interrupted · Resume" treatment and the user has to notice and type "continue". _run_chat now writes the turn's generation to the slot-owned metadata key turn_in_flight_generation before provider dispatch and omits it at teardown. Absence is the clear. Admission is a metadata-only merge (the rows stay with the periodic flush, so what the prompt builder reads off disk is unchanged); only a transcript that does not exist yet takes the full forced save that creates it. Both are best-effort, like the queued-prompt copy: a failing disk never refuses the turn. The generation is bound before the save is awaited so a cancellation inside it still reaches the finally with something to retire. A landed turn retires the marker before its existing end-of-turn save, so the normal path pays no extra write; other exits pay one forced save. The marker is preserved when the process is shutting down and the turn has not landed, which covers both the hard CancelledError and the provider-side cancel a graceful shutdown produces. The marker also carries a copy of the row that opened the turn (turn_in_flight_prompt: role, content, cls, ts, and a meta reduced to mid, files, dirs, injectKind), the same way queued_prompts carries the queue. The row itself waits for the periodic flush, so a death inside that window loses it and a restore judging the tail would see the previous turn's ending instead. Every restore path appends the copy first when no row with its identity is in the window; the unanswered opener is then the interruption evidence and Resume replays the right prompt. The copy is validated field by field on the way back in and cleared with the generation. Nothing is written for a restricted session (memory_mode other than persistent): the transcript save honours that retention mode, so a marker there could never be cleared and the prompt copy would outlive the session. Admission skips the marker in memory and on disk, and such a session restores exactly as before. Every restore path reconciles a leftover marker after the window is built: startup rehydration, recent-session restore, the shared resume/import materialiser, and channel surfacing. If the transcript already proves the interruption nothing is added; a deliberate trailing Stop card wins; otherwise one error row (meta.kind=gateway_restart_interruption) is appended past the window boundary so the classifier, composer and sidebar agree. Malformed values (booleans, strings, non-positive) are ignored, and the decision is idempotent across repeated restarts. The resume materialiser now carries a row's persisted cls like the startup restores do, so a resumed Stop card keeps its discriminator. Backend and frontend classifiers both treat an inject row stamped with a dispatching meta.injectKind (cron, recovery, user_replay, synthesis) as a turn opener, so an older Stop cannot mask a newer interrupted queued/recovery/synthesis turn; untagged inject rows (notes, Stop-hook halt cards, refusal notices) dispatched nothing and are looked through.
|
Head Two shard reds on |
Problem / Motivation
Goal: A gateway restart that lands mid-turn shows the existing "Turn interrupted · Resume" treatment even when the dead turn had already streamed a partial reply and completed tool rows.
A gateway restart that lands mid-turn shows no "Turn interrupted · Resume" when the turn had already streamed a partial reply and run tools. The session comes back looking finished and the user has to type "continue" by hand. Live repro in #13112:
user -> assistant(partial) -> 11 tool rows -> force exit, rehydrated with no Resume label.Why it matters
Every unattended session (goal loops, monitors, long tool runs) that hits a restart silently stalls. The sidebar shows it idle, the composer offers Send, and nothing says the last turn died. The one existing recovery UX (#8542) only fires when the transcript happens to prove the interruption.
Not a goal
history_projection, which the queued-prompt writes already pay.What changed (motivation → approach → change)
turn_in_flight_generation(slot-owned metadata key,history.py)_run_chatbefore provider dispatch, omitted at teardown. Absence is the clear._begin_local_turn_marker/_retire_local_turn_marker/_clear_local_turn_marker(chat_runner.py)_settled), so a tab close mid-admission still reaches thefinallywith something to retire and the clear is ordered after the write. A landed turn retires the marker before its existing end-of-turn save, so the normal path pays no extra write. Other exits pay one forced save in thefinally, after the session permit release and before the queue drain.turn_in_flight_prompt(slot-owned metadata key,history.py;_local_turn_opening_rowinchat_runner.py,_local_turn_promptinchat_persistence.py)mid,files,dirs,injectKind), written in the same metadata merge as the generation, the wayqueued_promptscarries the queue. The row itself waits for the periodic flush; a death inside that window loses it, and a restore that judged the tail without it would read the previous turn's ending (a Stop card, a finished answer) as the state of this one. Every restore path appends the copy first when no row with its identity (mid, elsets+role) is in the window; the unanswered opener is then the interruption evidence and Resume replays the right prompt. Validated field by field on the way back in (a non-dict, a non-opener role, empty content or a mixed-type list is dropped; no directive or send flags are carried). Cleared with the generation:Noneon the merge save, omitted on the full save.shutdown_eventis set and the turn has not landed: that covers the hardCancelledErrorand the provider-side cancel a graceful shutdown produces._reconcile_local_turn_marker(chat_persistence.py)chat_handlers.py), channel surfacing (channel_slots.py). Appends oneerrorrow ("This turn was interrupted when the app restarted. Review the partial output above, then resume to continue.",meta.kind=gateway_restart_interruption, the same wording family as the existing relay-interruption row) unless the transcript already proves the interruption or the newest turn ends in the user's own Stop card. Malformed values are ignored; the decision is idempotent across restarts.state.py,chatSlice.ts)injectrow carrying a dispatchingmeta.injectKind(cron,recovery,user_replay,synthesis) opens a turn like a user row, so an older Stop card cannot mask a newer interrupted queued/recovery turn. Untagged inject rows (notes, Stop-hook halt cards, refusal notices) dispatched nothing and are looked through.chat_runner.py)close_slot's 2 s grace, so the routing pin alone cannot tell the cancelled predecessor from its replacement. The metadata merge's guard checksstate._slots.get(key) is slotunder the owner lock, and both forced saves passexpected_history_keyandexpected_slot_name, the fence_save_slot_to_historyalready enforces at the locked commit boundary; a stale runner writes nothing over the replacement's line or rows..github/black-baseline.txt:test/test_knowledge.pyprunedcheck_black_formatting.pyfails on a graduated entry (baseline - unformatted), and main already formats that file (unchanged here,black --checkclean). No source change rides along.cls(chat_handlers.py)clslike the startup restores do, so a resumed Stop card keeps its discriminator.total/next_beforecount the hydrated window so paging stays consistent when a row was appended.The transcript cannot record a process death: a force exit runs no
finally, so the tail is shape-identical to a finished answer. The remote-executor path already solves this withrelay_in_flight; local turns had nothing. #11677 and #10521 persist queued prompts and startup admission, not the dispatched turn, so this adds the missing marker in the same slot-owned, absence-is-clear shape rather than a parallel store.Backwards compatibility
Compatible: a new optional metadata key, ignored when absent or malformed; transcripts written before this change restore exactly as today. The
clscarry on resume only affects rows that already persisted one (system rows). The opening-row copy adds one metadata field per in-flight turn, the size of the prompt, in the same file the row itself lands in.Tests
test/test_local_turn_restart_marker.py(new): marker lifecycle, generation-checked retire, metadata-only admission (rows on disk unchanged) with the newborn-transcript fallback, best-effort on a failing merge, cancel inside the admission save still clears, a cancelled admission write lands before the cancel propagates, merge-writer clearable field, reconciliation on rehydrate / recent restore, idempotency across restarts, Stop precedence, malformed values, local+relay markers -> one row,_run_chatwiring (landed clears; cancel-while-alive clears; shutdown preserves an unlanded turn; shutdown+landed clears; marker on disk before the provider streams).test/test_local_turn_restart_marker.py, opening-row copy: admission writes it and the clear drops it; retire clears it with the generation; the copy picks the newest unanswered opener (user or tagged inject) and keeps only identity/attachments; a lost opener behind a Stop tail is restored, flagged, saved and survives a second restart; a copy whose row is on disk is not restored twice; a copy without an id matches on ts+role; malformed copies are ignored while the generation still counts; a planted flag on the copy does not come back on the row.test/test_dashboard_chat.py:TestHistoryResumeInterruptedTurnMarker(resume reconciles, preserves Stop, ignores boolean marker, paging totals);TestEmptyResponseRetrypins the two marker saves as the only saves on the re-queue path.test/test_channel_slots.py: channel surfacing reconciles the marker.test/test_is_interrupted.py,test/test_to_dict_board_fields.py: dispatching inject kinds open turns, untagged injects (halt card, note) do not, older Stop no longer masks a newer inject turn.website/src/test/chatSlice.continuable.test.ts,ChatSliceCoverage.test.tsx: the same contract on the frontend mirror.The two core restore tests, the two inject-opener tests and the lost-opener restore test fail on
origin/mainand pass here.Manual verification
N/A — unit coverage sufficient: the restore paths and the runner's dispatch/teardown seams are driven end to end in tests (
_run_chatwith a mocked provider, the resume endpoint overTestClient), and the shutdown case is exercised by settingshutdown_event.Screenshots / video
Captured on head 4d49e26 with the repo's e2e harness (
spawn_feature_gateway, fixtureminimal, transcripts seeded before start, slots listed inopen_slots.jsonso the real restart restore path runs) and Playwright at 1280x900./api/chat/slotsreportedinterrupted: truefor each session before the frame was taken. The images are committed undertemp-screenshots/restart-interruption/on the PR head (the fork committed-media path); the merging maintainer drops them in the usual follow-up.Restored session after a mid-turn restart: the appended error row with its inline Resume, the composer's Resume control, and the sidebar's "Turn interrupted · Resume" state. Finished tool rows render collapsed.
A trailing unanswered
injectrow withinjectKind: crontreated as the interrupted turn (the_is_turn_inject/TURN_INJECT_KINDSchange):The lost-opener case: on disk the tail was the previous turn's Stop card and the new prompt was still in the flush window; the restore appended the opener from
turn_in_flight_promptand Resume is offered for it.Related Issues
Fixes #13112
Pattern harvest
Rule candidate: review-prompt
Pattern: "liveness inferred from transcript shape" — any recovery UX decided from the last rows of a log misses a process death that wrote no terminal row; the durable answer is a marker written before the work starts and cleared after it ends.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)