Skip to content

fix(dashboard): persist a turn-in-flight marker so a restart mid-turn shows Resume - #13113

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/durable-turn-in-flight-marker
Sep 27, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/durable-turn-in-flight-marker

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway restart that lands mid-turn shows the existing "Turn interrupted · Resume" treatment even when the dead turn had already streamed a partial reply and completed tool rows.

A gateway restart that lands mid-turn shows no "Turn interrupted · Resume" when the turn had already streamed a partial reply and run tools. The session comes back looking finished and the user has to type "continue" by hand. Live repro in #13112: user -> assistant(partial) -> 11 tool rows -> force exit, rehydrated with no Resume label.

Why it matters

Every unattended session (goal loops, monitors, long tool runs) that hits a restart silently stalls. The sidebar shows it idle, the composer offers Send, and nothing says the last turn died. The one existing recovery UX (#8542) only fires when the transcript happens to prove the interruption.

Not a goal

  • A new recovery UI; the interruption row, composer state and sidebar label are the existing fix(chat): surface interrupted sessions in the sidebar #8542 treatment, only their trigger changes.
  • Automatic re-dispatch of the interrupted turn; Resume stays a user action.
  • Changing what a deliberate Stop means; a turn whose newest row is the user's Stop card is still read as ended on purpose.
  • Bounding or rejecting attachments on the send path; the marker copy is dropped whole when it exceeds its own bounds.
  • Reducing the per-turn cost of the metadata-line rewrite in history_projection, which the queued-prompt writes already pay.

What changed (motivation → approach → change)

Piece What it does
turn_in_flight_generation (slot-owned metadata key, history.py) Written by _run_chat before provider dispatch, omitted at teardown. Absence is the clear.
_begin_local_turn_marker / _retire_local_turn_marker / _clear_local_turn_marker (chat_runner.py) Admission is a metadata-only merge (rows stay with the periodic flush, so the prompt builder reads the same disk it did before); a transcript that does not exist yet takes the full forced save that creates it. Best-effort: a failing disk never refuses the turn (same rule as the queued-prompt copy). The generation is bound before the save is awaited, and the write is awaited to completion even under cancellation (_settled), so a tab close mid-admission still reaches the finally with something to retire and the clear is ordered after the write. A landed turn retires the marker before its existing end-of-turn save, so the normal path pays no extra write. Other exits pay one forced save in the finally, after the session permit release and before the queue drain.
turn_in_flight_prompt (slot-owned metadata key, history.py; _local_turn_opening_row in chat_runner.py, _local_turn_prompt in chat_persistence.py) A copy of the row that opened the turn (role, content, cls, ts, and a meta reduced to mid, files, dirs, injectKind), written in the same metadata merge as the generation, the way queued_prompts carries the queue. The row itself waits for the periodic flush; a death inside that window loses it, and a restore that judged the tail without it would read the previous turn's ending (a Stop card, a finished answer) as the state of this one. Every restore path appends the copy first when no row with its identity (mid, else ts+role) is in the window; the unanswered opener is then the interruption evidence and Resume replays the right prompt. Validated field by field on the way back in (a non-dict, a non-opener role, empty content or a mixed-type list is dropped; no directive or send flags are carried). Cleared with the generation: None on the merge save, omitted on the full save.
Shutdown rule The marker is kept when shutdown_event is set and the turn has not landed: that covers the hard CancelledError and the provider-side cancel a graceful shutdown produces.
_reconcile_local_turn_marker (chat_persistence.py) Runs on every restore path after the window is built: startup rehydrate, recent-session restore, the shared resume/import materialiser (chat_handlers.py), channel surfacing (channel_slots.py). Appends one error row ("This turn was interrupted when the app restarted. Review the partial output above, then resume to continue.", meta.kind=gateway_restart_interruption, the same wording family as the existing relay-interruption row) unless the transcript already proves the interruption or the newest turn ends in the user's own Stop card. Malformed values are ignored; the decision is idempotent across restarts.
Inject rows as turn openers (state.py, chatSlice.ts) An inject row carrying a dispatching meta.injectKind (cron, recovery, user_replay, synthesis) opens a turn like a user row, so an older Stop card cannot mask a newer interrupted queued/recovery turn. Untagged inject rows (notes, Stop-hook halt cards, refusal notices) dispatched nothing and are looked through.
Incarnation fence on the marker writes (chat_runner.py) A tab closed mid-turn and reopened under the same key resumes the same transcript inside close_slot's 2 s grace, so the routing pin alone cannot tell the cancelled predecessor from its replacement. The metadata merge's guard checks state._slots.get(key) is slot under the owner lock, and both forced saves pass expected_history_key and expected_slot_name, the fence _save_slot_to_history already enforces at the locked commit boundary; a stale runner writes nothing over the replacement's line or rows.
.github/black-baseline.txt: test/test_knowledge.py pruned Gate-mandated after the rebase: check_black_formatting.py fails on a graduated entry (baseline - unformatted), and main already formats that file (unchanged here, black --check clean). No source change rides along.
Resume carries cls (chat_handlers.py) The materialiser keeps a row's persisted cls like the startup restores do, so a resumed Stop card keeps its discriminator. total/next_before count the hydrated window so paging stays consistent when a row was appended.

The transcript cannot record a process death: a force exit runs no finally, so the tail is shape-identical to a finished answer. The remote-executor path already solves this with relay_in_flight; local turns had nothing. #11677 and #10521 persist queued prompts and startup admission, not the dispatched turn, so this adds the missing marker in the same slot-owned, absence-is-clear shape rather than a parallel store.

Backwards compatibility

Compatible: a new optional metadata key, ignored when absent or malformed; transcripts written before this change restore exactly as today. The cls carry on resume only affects rows that already persisted one (system rows). The opening-row copy adds one metadata field per in-flight turn, the size of the prompt, in the same file the row itself lands in.

Tests

  • test/test_local_turn_restart_marker.py (new): marker lifecycle, generation-checked retire, metadata-only admission (rows on disk unchanged) with the newborn-transcript fallback, best-effort on a failing merge, cancel inside the admission save still clears, a cancelled admission write lands before the cancel propagates, merge-writer clearable field, reconciliation on rehydrate / recent restore, idempotency across restarts, Stop precedence, malformed values, local+relay markers -> one row, _run_chat wiring (landed clears; cancel-while-alive clears; shutdown preserves an unlanded turn; shutdown+landed clears; marker on disk before the provider streams).
  • test/test_local_turn_restart_marker.py, opening-row copy: admission writes it and the clear drops it; retire clears it with the generation; the copy picks the newest unanswered opener (user or tagged inject) and keeps only identity/attachments; a lost opener behind a Stop tail is restored, flagged, saved and survives a second restart; a copy whose row is on disk is not restored twice; a copy without an id matches on ts+role; malformed copies are ignored while the generation still counts; a planted flag on the copy does not come back on the row.
  • test/test_dashboard_chat.py: TestHistoryResumeInterruptedTurnMarker (resume reconciles, preserves Stop, ignores boolean marker, paging totals); TestEmptyResponseRetry pins the two marker saves as the only saves on the re-queue path.
  • test/test_channel_slots.py: channel surfacing reconciles the marker.
  • test/test_is_interrupted.py, test/test_to_dict_board_fields.py: dispatching inject kinds open turns, untagged injects (halt card, note) do not, older Stop no longer masks a newer inject turn.
  • website/src/test/chatSlice.continuable.test.ts, ChatSliceCoverage.test.tsx: the same contract on the frontend mirror.

The two core restore tests, the two inject-opener tests and the lost-opener restore test fail on origin/main and pass here.

Manual verification

N/A — unit coverage sufficient: the restore paths and the runner's dispatch/teardown seams are driven end to end in tests (_run_chat with a mocked provider, the resume endpoint over TestClient), and the shutdown case is exercised by setting shutdown_event.

Screenshots / video

Captured on head 4d49e26 with the repo's e2e harness (spawn_feature_gateway, fixture minimal, transcripts seeded before start, slots listed in open_slots.json so the real restart restore path runs) and Playwright at 1280x900. /api/chat/slots reported interrupted: true for each session before the frame was taken. The images are committed under temp-screenshots/restart-interruption/ on the PR head (the fork committed-media path); the merging maintainer drops them in the usual follow-up.

Restored session after a mid-turn restart: the appended error row with its inline Resume, the composer's Resume control, and the sidebar's "Turn interrupted · Resume" state. Finished tool rows render collapsed.

Restored session with the restart-interruption row and Resume

A trailing unanswered inject row with injectKind: cron treated as the interrupted turn (the _is_turn_inject / TURN_INJECT_KINDS change):

Interrupted cron-dispatched turn offering Resume

The lost-opener case: on disk the tail was the previous turn's Stop card and the new prompt was still in the flush window; the restore appended the opener from turn_in_flight_prompt and Resume is offered for it.

Lost opening row restored from the marker copy

Related Issues

Fixes #13112

Pattern harvest

Rule candidate: review-prompt
Pattern: "liveness inferred from transcript shape" — any recovery UX decided from the last rows of a log misses a process death that wrote no terminal row; the durable answer is a marker written before the work starts and cleared after it ends.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and pepmach September 23, 2026 15:49
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 23, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Intent: Make a gateway restart that lands mid-turn show the existing Turn interrupted · Resume treatment even when the dead turn had already streamed a partial reply and completed tool rows, by recording turn admission durably instead of inferring liveness from the transcript.
Not a goal: A new recovery UI, automatic re-dispatch of the interrupted turn, or changing what a deliberate Stop means.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 94a81112c231155319fc554e3937a66c866cf560 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Description's .github/black-baseline.txt prune is not in this diff — reconcile that claim (or the missing hunk) before merge.

[DESIGN-REVIEWED] 94a8111

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 94a81112c231155319fc554e3937a66c866cf560 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

_LOCAL_TURN_PROMPT_META_KEYS ships twice and the chat_persistence.py copy has zero consumers; the description's black-baseline table row names a hunk this diff no longer carries.

Not justified as shipped

  • Item 9 — zero consumers: grep of the diff finds _LOCAL_TURN_PROMPT_META_KEYS 3 times — a definition in chat_persistence.py, a definition in chat_runner.py, and one use (_local_turn_opening_row, which reads only the chat_runner copy); _local_turn_prompt and local_turn_prompt_within_bounds spell the keys as literals, so the chat_persistence copy is dead and the twin definitions will diverge silently.
    Subtraction: delete _LOCAL_TURN_PROMPT_META_KEYS from chat_persistence.py.
    Clears when: the unused definition is deleted (one shared definition remaining).

Also actionable, not an inventory item: the ## What changed row ".github/black-baseline.txt: test/test_knowledge.py pruned" matches no hunk in the diff, and the base baseline holds no such entry — drop the stale row.

What this change ships

Inventory (9 items) — 8 justified

Intent: make a gateway restart that kills an in-flight local turn show the existing "Turn interrupted · Resume" treatment even when the dead turn already streamed output — a FIX (live repro #13112, RED-on-main tests).

  1. A restart mid-turn now shows Resume even after partial output and finished tool rows (durable marker, reconciled on every restore path) — justified
  2. A prompt still in the flush window at process death comes back on the next start — justified
  3. A trailing dispatching inject or nudge row now reads as an unanswered turn on both surfaces, so an older Stop cannot mask it — justified
  4. A loaded window of only tool rows with a trailing error now reads as interrupted — justified
  5. A resumed session keeps each row's persisted cls, so a Stop card keeps its discriminator — justified
  6. Resume paging counts the hydrated window, so a recovered row cannot repeat a page — justified
  7. Channel-surfaced slots reconcile the marker like the dashboard restores — justified
  8. Three review screenshots committed under temp-screenshots/ — justified
  9. _LOCAL_TURN_PROMPT_META_KEYS duplicated into chat_persistence.py — zero consumers: only the chat_runner.py twin is read

[FIRST-PRINCIPLES-REVIEWED] 94a8111

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5, fork) — 🟡 CONCERNS

UX-level review of 94a81112c231155319fc554e3937a66c866cf560 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

No first-time reader has seen the new interruption row or the cron/nudge-triggered Resume states; every element is screenshot-covered, so this is the lane's cap, not a gap.

Watch

  • In attachment-01 an interrupted cron turn shows the red "Turn interrupted · Resume" label with no row explaining why (the unanswered opener is the classifier's evidence, not a human's); a user finds a stalled scheduled session with no stated cause. Matches the existing fix(chat): surface interrupted sessions in the sidebar #8542 treatment for unanswered user rows, so low severity — but a human should confirm the label-without-explanation reads acceptably for automation-opened turns, which the user never typed.

[UX-REVIEWED] 94a8111

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 94a81112c231155319fc554e3937a66c866cf560 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/history.py:221 -- "role, content, cls, ts, meta" contradicts marker serialization, which omits cls -> Fix: remove cls from the field list.
[GPT-REVIEWED] 94a8111

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 94a81112c231155319fc554e3937a66c866cf560 via the fork AI-review pipeline; updated in place on each push.

Review details

No blocking defects; one advisory finding.

FINDING — src/kiro_crew/dashboard/chat_persistence.py:~316 — the recovered opener is appended as str(prompt["content"]) with no redaction, while all four loaders that materialise the surrounding window apply redact_display_content / _redact_assistant for role != "user", so a nudge or inject opener (a cron script result is injected raw — slack/gateway.py:4260) re-enters the window with raw credential/exfil bytes and is then persisted back into it by the slot._dirty = True flush → Fix: redact at the append, redact_display_content(str(prompt["content"])) (_redact_assistant for the channel_slots caller), or once inside _local_turn_prompt since every non-user admitted role needs it.

[OPUS-REVIEWED] 94a8111

@Pearcekieser
Pearcekieser force-pushed the fix/durable-turn-in-flight-marker branch from 0c97124 to 8b1bfec Compare September 23, 2026 16:28
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • fixed span=db37e66c9e96 — cancellation during marker persistence leaves a false in-flight marker

Legitimate: the generation local was bound from the awaited save's return value, so a cancel inside that await left it 0 and _retire_local_turn_marker no-oped in the finally while the save had already committed the value.
Fixed in 8b1bfec: _local_turn_generation_for(slot) binds the generation BEFORE the await and _begin_local_turn_marker takes it as an argument; regression test test_a_cancel_inside_the_admission_save_still_clears_the_marker drives _run_chat with a cancel raised inside the admission save and asserts the on-disk key is gone afterwards.
Same head also makes admission a metadata-only merge (update_metadata_if(require_existing=True), full save only for a transcript that does not exist yet), so admitting a turn no longer writes window rows ahead of the periodic flush.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • fixed span=b3e60c886656 — same event, two names ("gateway" vs "app")

Legitimate: the sibling relay-interruption row says "when the app restarted"; the new row said "gateway", the process's internal name.
Fixed in 8b1bfec: _RESTART_INTERRUPTION_MSG now reads "This turn was interrupted when the app restarted. Review the partial output above, then resume to continue." with a comment tying it to the sibling row.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • needs-a-decision — evidence gap: screenshot of the restored interruption row and the composer's Resume state

The lane asks for a screenshot of a restored session showing the new error row and the Resume composer. The rendered surface is the EXISTING interruption treatment (same msg msg-err renderer, same composer gate, same sidebar interrupted field); this PR changes only WHEN it appears, and the row's copy is the existing relay-interruption sentence with the same wording ("interrupted when the app restarted").
This is a fork PR prepared from an agent sandbox: gh pr edit --attach needs push access to kirodotdev/KiroCrew, and no pod/browser surface is available here, so no attachment can be produced from this side.
Question for the maintainer / PR owner: accept the no-visual-delta justification in the body (or apply the no-screenshots label), or should @Pearcekieser drag a screenshot of a restored session into the description from the web UI before the next push?

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/durable-turn-in-flight-marker branch from 8b1bfec to 6acfa22 Compare September 23, 2026 17:17
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: _settled — shield + await-to-completion wrapper around the two admission writes

  • fixed span=db37e66c9e96 — cancelled admission write can outlive its clear

Legitimate: await asyncio.to_thread(update_metadata_if, ...) abandons the worker on cancellation, so a tab close during admission could let the metadata merge land after the finally's clear save and resurrect the marker.
Fixed in 6acfa22: both admission writes go through _settled, which shields the job and, on cancellation, waits for it to finish before re-raising, so every later write of the same file is ordered after it. Regression test test_a_cancelled_admission_write_lands_before_the_cancel_propagates gates the merge on a threading.Event, cancels the awaiting task mid-write, and asserts the cancel is delivered only after the value is on disk and that a subsequent clear removes it.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • rebutted span=b3e60c886656 — second sentences name different recovery verbs ("resume to continue" vs "send again to pick the conversation back up")

The two rows describe different recoveries on purpose. The local row lands on a slot whose turn died on THIS machine, where the composer offers the Resume control and the transcript above it is the complete record, so "resume to continue" is the exact action available. The relay row (_rehydrate_slot_from_history) lands on a slot whose turn ran on a peer crew: the peer may have finished it and the local tail is a truncated mirror, so the honest instruction is to send again and let the next turn re-synchronise the visible history. Aligning the verbs would tell one of the two users to do the wrong thing.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/durable-turn-in-flight-marker branch from 6ef5978 to 783f813 Compare September 27, 2026 03:48
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Head 783f8139f (round 10, same base as round 9, no rebase). Both round-9 GPT blocks fixed; Opus raised the same two as non-blocking, and Design / First Principles raised the second.

F1 (GPT, chat_runner.py _begin_local_turn_marker, restricted sessions skip restart recovery) -- fixed. The if slot.is_restricted: return skip predates #13760 on main, which now writes every retention mode's transcript and records the strictest memory_mode on the line. On this base the skip only removed recovery from incognito and temporary sessions. Removed. The old test_a_restricted_session_never_gets_a_marker is replaced by two tests in test/test_local_turn_restart_marker.py: test_a_restricted_session_gets_and_clears_the_marker (marker and prompt copy written at admission, both gone after _clear_local_turn_marker, memory_mode on the line unchanged) and test_a_restricted_session_killed_mid_turn_restores_resume (prompt row in the unflushed window, slot dropped, _rehydrate_slot_from_history restores the row, memory_mode preserved, interrupted true). Both parametrised over incognito and temporary; both fail on 6ef5978db.

F2 (GPT / Opus / Design / FP, state.py _TURN_INJECT_KINDS and chatSlice.ts TURN_INJECT_KINDS, mcp_app omitted) -- fixed. mcp_app added to both. The frontend set is now derived from TURN_INJECT_DISPATCHED: Readonly<Record<InjectKind, boolean>>, so a kind added to InjectKind does not compile until it is classified, the same guard INJECT_KIND_OPENS_TURN carries. Tests: test_every_dispatching_kind_is_an_opener in test/test_is_interrupted.py now includes mcp_app; test_opening_row_copy_picks_the_newest_unanswered_opener copies an mcp_app opener; the ChatSlice treats a trailing dispatching inject as a new, unanswered turn floor test iterates all five kinds.

On the Design / FP subtraction (route through _TURN_OPENING_INJECT_KINDS / injectOpensTurn instead of a second set): not taken, and the reason is now in the docstring beside each set. _TURN_OPENING_INJECT_KINDS = {cron, synthesis} answers "does this row start a turn the session-start failure streak should count", and deliberately walks past recovery and user_replay because they resume the same turn. This PR asks "did a dispatch happen that got no reply", and a recovery or replay dispatch that died is exactly such a turn (test_older_stop_does_not_mask_a_newer_inject_turn). Reusing the narrower set would reintroduce the masking bug for those two kinds.

Local on this head: 1178 passed across the six touched test files; 86 in the marker and interruption suites; tsc, eslint and the 105-test ChatSlice file pass; black gate, flake8, isort, mypy and the comment-history gate clean. Re-sync budget unchanged at 2 of 3.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/durable-turn-in-flight-marker branch from 783f813 to 67e06bd Compare September 27, 2026 04:32
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Head 67e06bd71 (round 11, same base as rounds 9 and 10, no rebase). Round-10 findings, each with its disposition.

GPT F1 (fenced, chat_persistence.py _reconcile_local_turn_marker, opener in the frozen prefix duplicated) -- fixed. The identity check ran over the loaded window only. The function now takes persisted, the full on-disk row list, and every one of its four callers (_rehydrate_slot_from_history, restore_recent_sessions, the resume handler in chat_handlers.py, the channel restore in channel_slots.py) already held that list and passes it. Test test_an_opening_row_in_the_frozen_prefix_is_not_restored_twice: 520 tool rows after the opener, _disk_older_count > 0, restore adds no user row, a second restart from the saved bytes still holds exactly one opener and one interruption row across the whole file. Fails on 783f8139f, passes here.

Writing that test exposed a sibling gap: when the loaded window holds only tool rows plus the restart's error row, is_turn_interrupted and selectTurnInterrupted both ran off the start of the walk and returned False. Both now return the trailing-error evidence, the rule the assistant branch already applies. Tests: test_window_of_only_tool_rows_reads_its_trailing_error (test_is_interrupted.py) and reads a trailing error when the loaded window holds only tool rows (ChatSliceCoverage.test.tsx).

GPT F2 and Opus BLOCKING (_local_turn_opening_row / _local_turn_prompt, prompt copy unbounded) -- fixed, with one deliberate deviation from the suggested fix. New local_turn_prompt_within_bounds in chat_persistence.py: 128 KiB serialized total (MAX_PROMPT_BYTES is 100 KB, plus identity and paths), 64 attachments per list, 4096 chars per cls, ts, mid, injectKind and per attachment path. Applied at retention (_local_turn_opening_row, so an oversize copy is never written) and at restore (_local_turn_prompt, because the metadata line is a writable file). An out-of-bounds copy is DROPPED, not truncated: the restore re-appends the copy as the user's own transcript row, so a truncated copy would put a shortened version of their words into the transcript as if they had sent it, which is the adverse effect the round-10 adjudicator named. Dropping the copy falls back to the round-8 behaviour, where the generation alone flags the turn. Tests: test_an_oversize_opener_is_not_copied_and_never_truncated (each bound, plus the at-bound case carried whole) and test_an_oversize_stored_copy_is_dropped_at_restore.

Opus FINDING (_begin_local_turn_marker, update_metadata_if rewrites the whole file per turn) -- not taken in this PR. The cost is real and pre-existing: #11677 already performs the same metadata-only merge on the same line for every queued prompt, through the same _update_metadata_locked. This PR adds one more merge per admitted turn on that path. An in-place line-0 write or a sidecar changes history_projection, which the finding itself marks as untouched code, and would apply equally to the queued-prompt writes; it belongs in its own change rather than inside a fix for a restart-recovery gap.

Backend shard 5 (test_decisions_end_to_end.py) and Windows shard 7 (test_runtime_cleanup_windows.py) -- modules this PR does not touch; shard 5's test passes locally on this head; Coverage Gate was red only because shard 5 was. Both rerun with this push.

Local on this head: 1182 passed across the six touched test files; 90 in the marker and classifier suites; tsc, eslint and the 106-test ChatSlice file pass; black gate, flake8, isort, mypy and the comment-history gate clean. Re-sync budget unchanged at 2 of 3.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@Pearcekieser
Pearcekieser force-pushed the fix/durable-turn-in-flight-marker branch from 67e06bd to c0c37cf Compare September 27, 2026 04:43
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 27, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Head c0c37cfa8 (round 12): same code as 67e06bd71, rebased onto current main (re-sync 3 of 3), plus the three review screenshots committed under temp-screenshots/restart-interruption/ so the UX lane's committed-evidence reader admits them (gh --attach is unavailable to a fork contributor and fork media-ref URLs are not read). The description now links those paths. Merging maintainer: please drop the directory in the usual chore(evidence) follow-up.

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Security Scope Review (fork) — 🔴 BLOCK (blocking)

Which legitimate operations does 67e06bd71496053751add23832854307e118f387 newly refuse? Model proposes, scripts/deny_diff.py decides — updated in place on each push. A script-confirmed regression blocks PR readiness.

no leg reported, and this was not the nothing-to-adjudicate short circuit

No [SCOPE-REVIEWED] marker for this head was produced, so the model's text is withheld: it cannot be shown to describe this revision. See the job logs.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Head eca609ddf (round 13, same base as round 12, no rebase). Round-12 findings and their disposition. Opus stamped c0c37cfa8 with no blocking findings; UX moved to CONCERNS (advisory) once the committed media under temp-screenshots/restart-interruption/ was read.

GPT F1 (fenced, chat_runner.py _begin_local_turn_marker, marker merge escapes the close/replacement fence) -- fixed. The merge ran through asyncio.to_thread with _still_this_incarnation as a worker-thread read of event-loop state, was not registered in slot._guarded_history_writes, and did not read is_closing. Now: when slot.is_closing the merge is refused and the slot marked dirty (the same fence save_slot_off_loop._dispatch_to_worker reads, with no suspension between the read and the registration); otherwise the merge is dispatched with run_in_executor, its future is registered through register_guarded_history_write before the first await, and awaited through run_to_completion. The retraction therefore waits on this write like every truncating save. Tests in test/test_local_turn_restart_marker.py: test_admission_merge_refuses_once_the_slot_is_closing (fence up via begin_close(), metadata line unchanged, _dirty set) and test_admission_merge_is_a_guarded_history_write (the worker is held on an event; _pending_guarded_history_writes(slot) holds one future while it runs and none after; the marker lands).

GPT F2 (_LOCAL_TURN_PROMPT_MAX_ATTACHMENTS = 64 narrower than the accept path) -- not changed; the adjudicator downgraded it as a disproportionate remedy and that reasoning is adopted here. The outcome needs four conditions at once: more than 64 attachments accepted, the opener row still unflushed at the kill, a force exit, and an on-disk tail ending in a Stop card. Any other tail still gets the interruption row (is_turn_interrupted). When all four hold, the marker is zeroed and the slot marked dirty, and the transcript shows the pre-copy behaviour of round 8: the message is absent and the user re-sends. The two remedies both reach outside this PR: a send-path rejection over N attachments changes accepted behaviour on the hot path, and sizing the metadata total for an unbounded list (64 x 4096 already exceeds the 128 KiB total) defeats bounding a line that is re-read on every rehydrate. A bound that drops the copy whole was the round-11 answer to the same rule, and the docstring on _LOCAL_TURN_PROMPT_MAX_BYTES states the trade.

Round-12 shard reds -- none this round; shard 5 and Windows shard 7 both passed on c0c37cfa8.

Local on this head: 1184 passed across the six touched test files; 92 in the marker and classifier suites; tsc, eslint and the ChatSlice file pass; black gate, flake8, isort, mypy and the comment-history gate clean. Re-sync budget 3 of 3 used (round 12); this push is same-base.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Head a2260a98d (round 14, same base as rounds 12 and 13, no rebase). Round-13 findings and their disposition. Opus stamped eca609ddf with no blocking findings and one advisory finding, taken below.

GPT F1 (chat_persistence.py, _LOCAL_TURN_PROMPT_MAX_ATTACHMENTS = 64 narrower than the send path) -- fixed. The bounds are now the send path's own: _LOCAL_TURN_PROMPT_MAX_ATTACHMENTS = chat_delivery.ATTACHMENT_LIST_MAX_ITEMS (256) and _LOCAL_TURN_PROMPT_MAX_FIELD_CHARS = ATTACHMENT_PATH_MAX_LEN (4096), with the serialized total sized for MAX_PROMPT_BYTES plus two full lists (128 KiB + 2 * 256 * (4096 + 4)), so every row the send path accepts fits the copy and the total only rejects a line no writer of this gateway produced. The existing bound tests read the constants, so they exercise the new values.

GPT F2 (fenced, chat_runner.py _begin_local_turn_marker, an omitted copy preserves a stale prompt) -- fixed. The admission merge now writes both turn_in_flight_generation and turn_in_flight_prompt every time, with None as the cleared value the restore already reads as "no copy" (the same rule the merge save in _save_slot_to_history uses). Test test_admission_without_a_copy_clears_a_stale_copy_on_disk: a stale copy planted on the line, an oversize next opener, the merge leaves turn_in_flight_prompt: null, and the restart restores no "stale prompt" row. Fails on eca609ddf, passes here.

Opus FINDING (_local_turn_opening_row copies a cron inject's JSON cls) -- taken. The copy no longer carries cls at all: the transcript never persists one for a user or inject row, and a cron inject's in-memory cls is json.dumps({"cronLabel": ...}), which the emit path (_prepare_messages -> parse_cls_meta) would parse into meta over the row's mid and injectKind, after which selectTurnInterrupted skips the row and the frontend disagrees with the backend. _local_turn_prompt ignores a stored cls, and the reconcile assigns the loader's own default (msg msg-u / msg msg-a) on re-append. Test test_a_cron_openers_json_cls_does_not_round_trip: the restored inject row keeps mid and injectKind, parse_cls_meta(row["cls"]) is None, and interrupted is true. Fails on eca609ddf, passes here.

Round-13 shard reds. Backend 3.12 shard 3 and Windows shard 3 fail test_review_repair_routing_skill::test_slim_skills_do_not_regrow_duplicate_guidance_or_wire_model_ids[prepare-pr-68000] with 68015 <= 68000: prepare-pr/SKILL.md on current origin/main is 68015 bytes (this PR's base has it at 67960, and the PR does not touch it); CI runs on the merge commit, so this is main-red and will clear when main does. Coverage Gate follows those shards. Frontend Tests shard 1 is red but the fork job log is not readable through the API; the two files this PR touches pass locally (161 tests) and a full local vitest run is in progress.

Local on this head: 1186 passed across the six touched test files; 94 in the marker and classifier suites; black gate, flake8, isort, mypy and the comment-history gate clean. Re-sync budget stays at 3 of 3 (round 12); this push is same-base.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Head dc2842d86 (round 15): same tree as a2260a98d, timestamp-only amend, same base, no code change. Retrigger for the round-14 CI-app rate-limit wave at 06:31-06:33Z (GPT, Opus and Internal Content Scan "could not run", Security Scope "no leg reported", Resolve desktop matrix "API rate limit exceeded for installation"), pushed after ten minutes without a fresh 403.

Three reds on a2260a98d are inherited and will show again on this head until main moves:

  • Frontend Tests (1): CrewWebviewContainment.test.tsx > "dates the retained document by the version on screen" asserts not.toContain("3d ago") against a fixture dated 2026-09-04; from 2026-09-27 the rendered stamp is "23d ago", which contains that substring. Calendar-dependent, in a file this PR does not touch; fails identically on main today. Frontend Coverage Merge follows it.
  • Backend Tests (3.12, 3) / (Windows) (3): test_review_repair_routing_skill size cap, prepare-pr/SKILL.md is 68015 bytes on current main against a 68000 cap (this PR's base has 67960 and does not touch it). Coverage Gate follows it.
  • Backend Tests (3.12, 2): test_ai_review_workflows::TestReviewLanesPublishOnlyTheReview::test_the_trim_and_count_execute_as_written, untouched by this PR and passing locally on this head.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Head f5fead093 (round 16): same code as dc2842d86, rebased onto current main at f2c732ecf, which is #14398 "unbreak the two base-side gates red on every open PR". That commit is exactly the fix for the two inherited reds this PR has been carrying since round 13 (prepare-pr/SKILL.md size cap, now 67854 bytes; the CrewWebviewContainment "23d ago" fixture), so this rebase is the retrigger that clears them rather than a same-tree amend. No conflict.

Round 15 on dc2842d86: GPT green for the first time since round 8; Opus "review incomplete" (lane-side, no verdict produced), rerun by this push; Design, First Principles and UX unchanged; 83 lanes green.

Local on this head: 1206 passed across the touched suites plus test_review_repair_routing_skill; CrewWebviewContainment and ChatSliceCoverage pass (150); black gate and comment-history gate clean.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Head e2e1ee381 (round 17, same base as round 16 at f2c732ecf, no rebase). Opus stamped f5fead093 with no blocking findings; the base-side reds from rounds 13-15 are gone on this base.

GPT F1 (fenced, chat_runner.py _local_turn_opening_row, nudge openers lost after restart) -- fixed. A monitor loop's cycle is appended as a nudge row (slack/gateway.py, turn_slot.append("nudge", ...)) and dispatches a turn, but the copier accepted only user and dispatching inject rows, and neither is_turn_interrupted nor selectTurnInterrupted classified nudge (the frontend skipped it through CONTINUE_SCAN_SKIP). Now _LOCAL_TURN_OPENER_ROLES = {user, nudge} drives the copier, nudge is in _LOCAL_TURN_PROMPT_ROLES for the restore, and both classifiers return interrupted for an unanswered nudge row before the skip set, mirroring each other. Tests: test_nudge_row_is_an_opener (test_is_interrupted.py; unanswered nudge, nudge behind an older Stop card, answered nudge), the nudge case in test_opening_row_copy_picks_the_newest_unanswered_opener, test_a_lost_nudge_opener_behind_a_stop_card_is_restored (production Stop-card shape on disk, nudge row unflushed, restore re-appends it with its mid and reports interrupted), and treats a trailing nudge row as a new, unanswered turn floor (ChatSliceCoverage.test.tsx). The backend and frontend nudge tests both fail on f5fead093. The restored row carries identity only; NudgeCard renders from the [auto-nudge cycle N] content tag when meta.nudge is absent, as it already does for history-restored rows.

Windows shard 3 on f5fead093 -- test_wakatime_heartbeats::test_enabled_and_opted_in_sends_one_heartbeat, a module this PR does not touch; reruns with this push.

Local on this head: 1188 passed across the six touched test files; 96 in the marker and classifier suites; 107 in the ChatSlice file; tsc, eslint, black gate, flake8, isort, mypy and the comment-history gate clean.

… shows Resume

A gateway force exit during a turn runs no finally, so the transcript ends
in user -> assistant(partial) -> tool rows with no error row. That tail is
shape-identical to a finished answer, and the restore heuristic
(is_turn_interrupted / selectTurnInterrupted) reads it as finished: the
session comes back with no "Turn interrupted · Resume" treatment and the
user has to notice and type "continue".

_run_chat now writes the turn's generation to the slot-owned metadata key
turn_in_flight_generation before provider dispatch and omits it at
teardown. Absence is the clear. Admission is a metadata-only merge (the
rows stay with the periodic flush, so what the prompt builder reads off
disk is unchanged); only a transcript that does not exist yet takes the
full forced save that creates it. Both are best-effort, like the
queued-prompt copy: a failing disk never refuses the turn. The generation
is bound before the save is awaited so a cancellation inside it still
reaches the finally with something to retire. A landed turn retires the
marker before its existing end-of-turn save, so the normal path pays no
extra write; other exits pay one forced save. The marker is preserved when
the process is shutting down and the turn has not landed, which covers
both the hard CancelledError and the provider-side cancel a graceful
shutdown produces.

The marker also carries a copy of the row that opened the turn
(turn_in_flight_prompt: role, content, cls, ts, and a meta reduced to mid,
files, dirs, injectKind), the same way queued_prompts carries the queue.
The row itself waits for the periodic flush, so a death inside that window
loses it and a restore judging the tail would see the previous turn's
ending instead. Every restore path appends the copy first when no row with
its identity is in the window; the unanswered opener is then the
interruption evidence and Resume replays the right prompt. The copy is
validated field by field on the way back in and cleared with the
generation.

Nothing is written for a restricted session (memory_mode other than
persistent): the transcript save honours that retention mode, so a marker
there could never be cleared and the prompt copy would outlive the
session. Admission skips the marker in memory and on disk, and such a
session restores exactly as before.

Every restore path reconciles a leftover marker after the window is built:
startup rehydration, recent-session restore, the shared resume/import
materialiser, and channel surfacing. If the transcript already proves the
interruption nothing is added; a deliberate trailing Stop card wins;
otherwise one error row (meta.kind=gateway_restart_interruption) is
appended past the window boundary so the classifier, composer and sidebar
agree. Malformed values (booleans, strings, non-positive) are ignored, and
the decision is idempotent across repeated restarts.

The resume materialiser now carries a row's persisted cls like the startup
restores do, so a resumed Stop card keeps its discriminator. Backend and
frontend classifiers both treat an inject row stamped with a dispatching
meta.injectKind (cron, recovery, user_replay, synthesis) as a turn opener,
so an older Stop cannot mask a newer interrupted queued/recovery/synthesis
turn; untagged inject rows (notes, Stop-hook halt cards, refusal notices)
dispatched nothing and are looked through.
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Head 94a81112c (round 18): same tree as e2e1ee381, timestamp-only amend, same base, no code change. Retrigger for the second CI-app rate-limit wave of the day at 08:31-08:33Z on e2e1ee381 (GPT, Opus and Internal Content Scan "could not run", Security Scope "no leg reported", desktop matrix "API rate limit exceeded for installation", Fork PR Description "could not list check-runs"), pushed after ten minutes without a fresh 403.

Two shard reds on e2e1ee381 are in modules this PR does not touch: Backend Tests (3.12, 1) on test_pdf_extract::TestWindowsCeiling (the xdist worker crash seen in round 3 lives in the same file) and macOS (4) on test_chat_threads::test_a_failed_sidecar_staging_leaks_no_descriptor (assert 92 == 95, a descriptor-count assertion). Both rerun on this head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Restart mid-turn after partial reply + tool rows shows no Resume (transcript heuristic cannot see a force exit)

2 participants