Skip to content

fix(chat): hold a restored queue instead of showing it as work in flight - #23971

Closed
Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/restart-queued-lifecycle
Closed

Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:fix/restart-queued-lifecycle

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: After a restart, a chat whose turn was cut while messages were queued shows no work in flight, offers Resume, and never sends the queued messages without the person.

A turn is running and a message is queued behind it. The app restarts. The chat comes back with "This turn was interrupted when the app restarted. Review the partial output above, then resume to continue." and the queued message as a card. Nothing runs (running: false), yet:

  • the sidebar row spins "In progress", the board files the chat under Working and the command center marks it running;
  • the queue card's hourglass keeps animating;
  • Resume is hidden, and POST /continue answers 409 slot_queue_pending on every press;
  • the only way out is a new message, which runs first, and then the restored card sends itself behind it.

Why it matters

Every chat that has a queued message when the app restarts (an update, a crash, a watchdog exit) is affected. The UI says work is in progress when nothing is, hides the one control the notice tells the person to use, and later sends words the person typed to a process that no longer exists, out of order.

Not a goal

  • Changing what happens to a live queue behind a running turn.
  • Auto-resuming interrupted turns (feat(chat): resume interrupted turns on their own, within a budget #20750) or a bulk Resume (feat(chat): resume every interrupted session from the sidebar #21031).
  • Reopening a chat whose tab was created less than about 5 s before a SIGKILL (open_slots.json cadence). Separate cause, listed under Manual verification.
  • Making a force Stop's queue clear durable before the next 5 s flush. Separate cause; with this change a resurrected entry comes back held rather than auto-run.
  • A SIGKILL in the seconds after a queued message starts its turn brings that message back twice, as the interrupted opener row and as a card. With this change the card comes back held, so it never sends itself. That includes a send whose row was written when it was sent: its held copy also keeps its row out of Resume until the person sends or cancels the card. Writing every dequeue atomically with the turn marker, which retires that copy, is a follow-up PR.
  • Running a held card's Send now later, once memory preparation ends. While it runs, the gateway refuses that Send now with 409 memory_preparing and the card stays held, so the person sends it again when memory is ready.

What changed (motivation → approach → change)

Cause: a queue restored from disk has no owner. Nothing drains it on boot, by design (session.md: restored entries are handed back as cards, not dispatched), except accepted parked sends (drain_restored_parked_sends). But every reader still asked queue_depth whether work was coming, and the drain picked restored entries up after any later turn.

Restored entries are now held (is_held_queue_entry). The drain passes them by, and only the person's Send now on the card starts one. Send now starts a held card directly, on an idle slot. Behind a running turn the gateway refuses it with 409 held_turn_running and the card stays held, because the drain at that turn's end passes held cards by; the card's Send now is disabled while a turn runs, with the tooltip "Available when the current turn ends." If a restart hands back held a card whose ordinary Send now was accepted, the client (website/src/hooks/useQueuedMessageActions.ts) drops its Send now latch, so the card's controls come back.

"Is work coming?" readers now count queue_ready_depth: Continue's refusal, the completion sound, session status and the members-thread reset. The slot frame carries queue_held_depth and each held entry meta.held (types/index.chat.ts). The store carries held through wire.ts and queue.ts (including a card a first pane hydrate keeps), and selectContinuable (selectors.ts) looks past a held card. The sidebar, board, command center and card (website/src/components/QueueStack.tsx) stop showing a held card as working, and the card reads "Held" (new strings in every website/src/i18n/locales/*.json catalog). Held cards no longer count as user intervention for recovery gates. A Resume skips the written rows of held sends, so it continues the cut request. Continue after a Stop skips those rows too (build_cancelled_turn_preamble(skip_row=...)). Both recovery paths apply it: the native-resume preamble and the cold-start replay (build_session_replay(skip_row=...) in src/kiro_crew/context.py), which drops the rows from the disk transcript and the live window alike. Agents see the split too: the session-control status and read payloads carry queue_held_depth, and session_status and session_read_message in src/kiro_crew/mcp_dashboard_session_runs.py count only ready entries as queued (_queue_phrase, forwarded on the src/kiro_crew/mcp_dashboard.py facade).

The turn-completion chime (website/src/hooks/websocket/turnCompletion.ts) counts only ready entries too. meta.held overrides every content classifier, so a held prompt that happens to read like a system delivery is still an actionable card on every surface. A crewmate chat (ChatPane with crewmate) draws held entries as Held cards with Send now instead of sent bubbles, with no typing bubble for them. A held entry whose row is already written (meta.rowWritten) has no Edit and no Cancel: the gateway still refuses it with 409 queue_item_already_sent, as on main, so Send now is its way out. Its hint (held_hint_sent) says so.

Backwards compatibility

Compatible: new fields are additive (queue_held_depth, meta.held), and a client that ignores them behaves as before. A gateway without them reads as zero held.

Behaviour change: restored cards no longer send themselves after the next turn; they wait for Send now, an edit or a delete. This is the documented session.md contract, now enforced. Session status reports idle instead of queued for a slot whose only queued entries are held.

Removes nothing: no field, route or capability is removed. The restored-entry drain path is still reachable, through the card's Send now. Accepted parked sends still resume on boot (_held_for_drain).

Tests

  • test/test_restart_held_queue.py: held vs ready counts and projection; restore holds every entry; drain skips held entries, keeps their order, runs a named card, refuses Send now on a held card behind a running turn (409 held_turn_running, nothing promoted or stopped), never falls through to a held head after a purge, does not park held cards on memory preparation and refuses Send now on a held card there with 409 memory_preparing, keeping it held, is not reached by the sub-agent hold's system scan (a restored steer), and does not flush a deferred note into a cron turn behind a held head; Continue runs ahead of held cards, refuses a duplicate press, still refuses on a ready entry; held cards are not user intervention; repeated restarts keep one notice and the same cards.
  • test/test_chat_mode_security_send.py: the four restored-entry security tests now reach restored text through Send now, the only path that runs it.
  • website/src/test/heldQueueAfterRestart.test.tsx: wire and hydrate carry held (including a crewmate row kept in place); Resume is offered past a held card and withheld behind a ready one; lanes and command center treat a held queue as idle; the card shows "Held" with no hourglass.
  • test/test_chat_slot_facade_contract.py pins the new queue_held_depth key.
  • ChatPane.queueActions.test.tsx, useQueuedMessageActions.test.tsx, crewmateBubbles.test.ts: a crewmate's held entry is a Held card with Cancel and no typing bubble; an accepted Send now stays latched until a restart re-hydrates the card held. heldQueueAfterRestart.test.tsx: a held card's Send now is disabled while a turn runs, a ready card's is not.
  • TestResumeNeverContinuesAHeldSend: the interrupted-turn preamble and the cancelled-turn preamble skip the row of a held send; the cold-start replay skips it whether it comes from disk or the live window. A written send cut before its turn marker comes back as a held card. Send now on a held card behind a running turn is refused without claiming a stop, and a press whose turn changed during the body read stands down. test/test_dashboard_chat_handlers_coverage.py checks a held send whose row is written is still refused with 409 and stays held.
  • TestAgentsAreToldHeldIsNotQueued: the agent-facing session tools leave held cards out of the queued count, and an older gateway's payload still reads as queued.

Proof: not provable -- other: the new tests import the held-queue API, so prove.py's revert errors at collection. Run by hand on main c24c7bc with that API shimmed, before the turn-marker and parked Send now changes moved out of this PR: 14 of the 20 backend tests then in the file failed and the other 6 passed, pinning behaviour main already has.

Manual verification

Reproduced on a disposable gateway (fake ACP backend, loopback, throwaway home), never the live install, with a driver script that restarts at each boundary:

Scenario main c24c7bc this PR
SIGTERM mid-turn, 1 queued running:false, queue_depth:1, sidebar "In progress", hourglass, no Resume, /continue 409 sidebar "Turn interrupted · Resume", static Held card, Resume on notice and composer
Same, 2 queued both restored, stuck both held, in order
Resume, then SIGTERM again 409 every press 200, recovery runs, card stays held, Resume offered again after the second restart
New send after restart new message runs, then the restored card runs itself new message runs, card stays held
Browser: Resume, then Send now on the card n/a recovery ran, then the card ran once, queue empty

Not fixed, separate causes: a tab created under 5 s before SIGKILL is not reopened (its transcript, marker and queue are on disk); a force Stop's queue clear rides the 5 s flush.

Screenshots / video

Before (main): "In progress" spinner, animated hourglass, no Resume.

before

After: "Turn interrupted · Resume", Held card, Resume on the notice and the composer.

after

Related Issues

Fixes #23970
Closes #16780

Supersedes #20029. Its meta.restored projection and Resume gate are covered here, and its open review blocker (restored cards drain on their own after the recovery turn) is answered by keeping them held.

Sibling sweep

Swept: git grep -nE "queue_depth|len\(slot\._queue\)|if slot\._queue|not slot\._queue" -- src/ and git grep -n "queue_depth" -- website/src
Sibling: src/kiro_crew/dashboard/chat_api/continue_turn.py -- changed here
Sibling: src/kiro_crew/dashboard/chat_utils_completion.py -- changed here
Sibling: src/kiro_crew/dashboard/handlers/members_thread_turns.py -- changed here
Sibling: src/kiro_crew/dashboard/session_control_parts/session_status.py -- changed here
Sibling: src/kiro_crew/mcp_dashboard_session_runs.py -- changed here (agent-facing queued count)
Sibling: src/kiro_crew/builtin_skills/goal-conductor/SKILL.md -- not affected -- it reads queue_depth through session_read_message, which now counts only ready entries
Sibling: website/src/pages/chat/sessionLane.ts -- changed here
Sibling: website/src/pages/chat/command-center/model.ts -- changed here (node state); the options-suppression read keeps raw queue_depth, since a held card may still be the answer
Sibling: website/src/lib/closeBusyGate.ts -- not affected -- closing still discards unsent words, so a held card keeps the confirm
Sibling: src/kiro_crew/dashboard/chat_turn/queue_cycle.py -- not affected -- its queue reads feed _start_next_queued_turn, which now skips held entries

Pattern harvest

Rule candidate: review-prompt
Pattern: a predicate that asks "will this run on its own?" reads raw queue/ownership state that a restart can leave without a live owner

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (docs/system-specs/modules/session.md)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and KiroCrewCaptain as code owners October 25, 2026 07:45
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 25, 2026
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 5e03b0bb8c279cbfcf9e1407699eeb93c986a3ac via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The hold restores the session.md rule that a restored prompt is never auto-resent. The one note: the refusal of Send now behind a running turn is extra machinery that an existing stamp makes unnecessary.

Notes

  • Item 5 — oversized: the description says "Behind a running turn the gateway refuses it with 409 held_turn_running … because the drain at that turn's end passes held cards by". The refusal only works around this PR's own hold. It costs the early-body-read path in api_chat_slot_interrupt (its stand-down and stale-press guards), the channel-turn refusal, the turnRunning prop, and held_send_now_after_turn in 15 locales. Subtraction: delete all of those. When Send now is accepted, stamp the named card with the existing BOOT_ACCEPTED_KEY (slot_queue_repository.py), so is_held_queue_entry is false and the existing promote-then-stop interrupt path runs it.

What this change ships

Inventory (10 items) — 9 justified

Intent: after a restart, a chat cut while messages were queued shows idle, offers Resume, and never sends the queued words on its own. FIX.

  1. Restored queued messages no longer send themselves after a later turn; only Send now runs them — justified
  2. An interrupted chat with only held cards offers Resume; Continue no longer answers 409 — justified
  3. Sidebar, board, command center and completion chime stop treating held cards as work in flight — justified
  4. Queue card reads "Held" with a hint and no hourglass; new strings in every locale — justified
  5. A held card's Send now is disabled behind a running turn; the gateway refuses it with 409 held_turn_running — oversized: a stamp on accept would let the existing interrupt path run it
  6. Send now on a held card while memory is preparing is refused with 409 memory_preparing — justified
  7. A crewmate chat draws held entries as cards; a card whose row is already written loses Edit and Cancel — justified
  8. Resume and Continue-after-Stop skip the rows of held sends (skip_row) — justified
  9. Held cards no longer cancel recovery as user intervention — justified
  10. Agent session tools report queue_held_depth and show "idle" instead of "queued" for a held-only queue — justified

[FIRST-PRINCIPLES-REVIEWED] 5e03b0b

Description read

[DESCRIPTION-READ] ad4e37d583f4892f93bc27fa87cd2943eb38ffda96d9fbf5109fee30b5be3ce2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 347cfa725c5c4f1696489a28adddc269db061b93 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 5e03b0bb8c279cbfcf9e1407699eeb93c986a3ac via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The hold follows the restored-entry contract in session.md, which says restored entries are never auto-sent. The fields it adds are additive, the behaviour change is declared, and screenshots are attached.

Notes

  • Restored entries keep their source_channels, so a message that arrived on a channel and was queued before a restart now waits for a dashboard Send now. On main it ran after the next turn. A Slack or Discord sender has no card to press, so their message stays unsent until someone opens the dashboard. Say in session.md whether that is intended.
  • The interrupt handler now reads the request body before the stop claim, but only when the queue has a held card. That gives api_chat_slot_interrupt two orderings to keep in sync. A simpler fix is a separate held-card refusal path that runs before the claim and does not touch the claim ordering.

[DESIGN-REVIEWED] 5e03b0b

Description read

[DESCRIPTION-READ] 86cf53507b25f2da71fcdc1979807d2cd62567104c4c67cd73e16f413548dbe5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 347cfa725c5c4f1696489a28adddc269db061b93 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 347cfa7

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 347cfa725c5c4f1696489a28adddc269db061b93 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 347cfa7

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 25, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: sub-agent hold system scan reached held entries

  • span=d0cfad1260e9 issue (blocking): the sub-agent dequeue still dispatches held restored steers

fixed in 9fed98063: _dequeue_next_system_message now skips held entries (releases_past_subagent_hold(item) and not is_held_queue_entry(item)), so a restored steer runs only on its own Send now.
Regression: test_a_held_steer_is_not_released_by_the_sub_agent_hold fails on b90f6dc and passes on 9fed98063.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: deferred-note flush read the raw queue head

  • span=156d59329fce issue (blocking): a deferred note can flush into a cron or sub-agent turn behind a held card

fixed in 9fed98063: the flush guard reads _next_dispatchable_entry (the named card, else the first entry not held), so a cron entry behind a held head withholds the note as before.
Regression: test_a_note_is_not_flushed_into_a_cron_turn_behind_a_held_card fails on b90f6dc and passes on 9fed98063; the source-shape pin in test_gateway_appkit_endpoints_note.py names the new guard.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • praise: one owner for restored entries, one count for "is work coming?"

Acknowledged. The round-1 fixes keep that shape: the sub-agent scan and the note flush now read the same held predicate (is_held_queue_entry) instead of the raw queue head.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • note: the Held card and the "Turn interrupted · Resume" row are only in the after screenshot

Acknowledged. Both were driven in a real browser against a disposable gateway: Resume ran the cut turn with the card still Held, then Send now ran the card once and emptied the queue. No UI changed in round 1.

@Pearcekieser
Pearcekieser force-pushed the fix/restart-queued-lifecycle branch from b90f6dc to f674220 Compare October 25, 2026 08:45
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Pushed f674220. It carries the two round-1 fixes (dispositions above) and one locale fix for the reds on b90f6dc:

  • Frontend Tests (1), (2), (3) and Frontend Coverage Merge failed on the register rules in src/i18n/style (bn, fr, hi, ru). The new held_hint used the formal address. It now uses তুমি / tu / तुम / ты forms; all 12 locale style tests and i18n:check pass locally.
  • Backend Tests (3.12, 4) failed on test_governance_policy_viewer.py::TestNoPolicyDefault::test_enterprise_unnamed_scope_reports_the_baseline (PlatformCompositionError). The same test fails locally on current main 813e25c, whose own CI run is red, so it is not from this diff.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: turn-marker merge carries the post-dequeue queue

  • span=c146d09d67c5 issue (blocking): a racing pre-dequeue save preserves the consumed prompt

fixed in f9c1219: the opener copy and the shortened queue now land together or not at all. On a stale witness the merge drops queued_prompts AND writes turn_in_flight_prompt: None, so the earlier writer's paired save holds the entry exactly once (as a card if it snapshotted before the dequeue, as its row if after).
Regression: test_a_stale_queue_snapshot_drops_the_opener_copy_too (one copy after restore) and the updated test_a_newer_committed_queue_is_not_written_over; both fail on f674220.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: Send now release before a parked dispatch

  • span=5a68e05795fe issue (blocking): Send now releases a held card despite returning a refusal

fixed in f9c1219: the idle /interrupt path now answers 200 {"ok": true, "outcome": "parked"} when the drain parked on memory preparation and the card is still queued. The client treats any success as accepted and keeps its pending latch until the card retires (useQueuedMessageActions.onInterrupt), so the controls do not come back for a card that is about to run.
Regression: test_a_parked_send_now_is_answered_as_accepted fails on f674220 (409).

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: turn-marker merge carries the post-dequeue queue

  • span=096d6517711e issue (blocking): a prompt with no retained opener copy is dropped by the shortened queue

fixed in f9c1219: queued_prompts joins the marker merge only when slot._turn_in_flight_prompt is set. With no copy (an oversize prompt) the queue key stays with the paired row-and-queue save, so after a death the entry comes back as a held card.
Regression: test_without_an_opener_copy_the_queue_key_is_left_alone fails on f674220.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • suggestion: the session-status queue_held_depth key has no code reader

fixed in f9c1219: removed from the session-status row. Its status field still reports idle rather than queued for a slot whose only queued entries are held, which is the reader-facing part. The slot frame keeps queue_held_depth, read by the sidebar, board and command center.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • praise: held at the drain, one count for "will this drain?"

Acknowledged. Round 2 keeps that shape; its changes are in the turn-marker write (opener copy and shortened queue land together) and the parked Send now answer.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • note: no first-time reader has seen the Held card or the sidebar row

Acknowledged. The branch lives on a fork, so the blind read is not available to this PR; the screenshots and the browser run (Resume, then Send now) are the evidence. No UI changed in this round.

@Pearcekieser
Pearcekieser force-pushed the fix/restart-queued-lifecycle branch from f674220 to f9c1219 Compare October 25, 2026 09:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 25, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • thought (non-blocking): Commit the dequeue before recovery treats its row as unsent span=d0cfad1260e9

Rebutted: not changed; downgraded on adjudication. The trigger needs the gateway to die inside one queue save that starts as the drain dispatches, after the turn marker lands. That is the residual named on the previous disposition. An awaited write of the dequeue together with the marker is the atomic-dequeue change this PR moved out; it will be its own PR from main after this merges.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): waiting on #25706. Main fails mypy since #25480 (two imports of _frame_parents from session_create after #25669 moved it to crew_filing), so this PR is not pushed until that fix merges. It also conflicts with main again; the rebase follows the fix.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • thought (non-blocking): Preserve the interrupted opener when its dequeue has not persisted span=9dd41fe0962a

Rebutted: not changed; downgraded on adjudication. It is the residual named on the earlier dispositions: the gateway has to die inside the one queue save the drain starts as it dispatches a send whose row was written at send time. Writing the dequeue in the turn marker's own merge closes it, and that is the follow-up PR from main after this one merges.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the cancelled-turn preamble takes the same held-send filter as Resume

  • issue (blocking): Continue replays held sends after a cooperative Stop span=afb76f4f83e9

fixed in 2965ebe: build_cancelled_turn_preamble takes skip_row, and _turn_build_prompt passes _unsent_send_filter(slot). The rows of sends still held on their cards are dropped before the backward walk, so Continue restores the stopped request and never the held one.

Regression: test_the_cancelled_preamble_skips_a_held_sends_written_row fails with the filter removed.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: turn admission awaits the dispatched send's queue removal before the turn marker

  • span=156d59329fce issue (blocking): the dequeue of a written send is fired off, the turn marker is awaited

fixed in 2965ebe: the un-awaited start_queue_persist call in the drain is gone. _persist_dispatched_send runs in turn admission right before _begin_local_turn_marker and awaits a forced save when the dispatched row carries queuedEntryId and the queue differs from disk, so the entry is off the queue on disk before the marker lands.
Regression: test_a_dispatched_written_send_is_off_the_queue_before_the_marker fails with the save skipped.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the held-send filter only matches a string queue entry id

  • issue (blocking): Validate row metadata before set membership span=9dd41fe0962a

fixed in 3354c4a: _unsent_send_filter checks isinstance(entry_id, str) before entry_id in skipped. A row whose queuedEntryId is a list, dict, number or null is kept, and the lookup no longer raises.

Regression: test_a_row_with_a_non_string_entry_id_is_kept_and_does_not_raise fails with TypeError for the list and dict cases on the previous code.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • thought (non-blocking): Confirm queue removal before admitting a written send span=c146d09d67c5

Rebutted: not changed; downgraded on adjudication. The trigger needs the forced save to lose the history lock, the marker merge to win it right after, and the gateway to die before the periodic flush retries the save, which best_effort re-arms by marking the slot dirty. Refusing the turn on a failed queue save would turn a lock timeout into a lost send. Writing the dequeue in the marker's own merge closes this, and that is the follow-up PR from main after this one merges.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a Send now that releases a held card starts it when no turn end will

  • span=85e61fc1149b issue (blocking): Send now on a held card in the turn's end tail leaves it ready with no drain

fixed in 19595f2: api_chat_slot_interrupt records whether this press released a held card. When stop_turn answers idle, or the claim was superseded benignly, it calls _run_released_card, which starts that card under slot._lock with required_queue_id=queue_id after the idle Send now path's checks. A real cut still leaves the start to that turn's end.
Regression: test_a_released_card_runs_when_no_turn_is_left_to_stop[idle] fails without the call.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a Send now that releases a held card starts it when no turn end will

  • issue (blocking): Send now strands a held card when the turn ends during the body read span=5a68e05795fe

fixed in 19595f2: when the stop answers idle, or the press's claim stands down because the turn already ended, api_chat_slot_interrupt calls _run_released_card. Under slot._lock it starts the released card with required_queue_id, after the same checks as the idle Send now path (same slot, not running or stopping, this press's stop generation, card still queued, memory ready, caller authorized). A turn that was really cut still drains the card at its own end.

Regression: test_a_released_card_runs_when_no_turn_is_left_to_stop[idle] fails without the call.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • thought (non-blocking): Confirm queue removal before admitting a written send span=c146d09d67c5

Rebutted: not changed; downgraded on adjudication. Same trigger as on 2965ebe: the forced save has to lose the history lock, the marker merge has to win it right after, and the gateway has to die before the periodic flush retries the save. Refusing the turn on a failed queue save would turn a lock timeout into a lost send. Writing the dequeue in the marker's own merge closes it, in the follow-up PR from main after this one merges.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
resolution: removed

  • issue (blocking): Send now can strand a card during turn completion span=5a68e05795fe

fixed in 7af470e: the deferred path is gone. Send now on a held card behind a running turn now answers 409 held_turn_running before anything is promoted, released or stopped, so the card stays held. release_held_queue_entry, HOLD_RELEASED_KEY and _run_released_card are deleted, and _finish_queue_cycle is back to main. A held card now runs only through the idle Send now path, which starts it directly under slot._lock. The UI disables that card's Send now while a turn runs.

Regression: test_send_now_on_a_held_card_is_refused_while_a_turn_runs fails without the refusal.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

self-added: yes
resolution: removed

  • span=85e61fc1149b issue (blocking): Send now on a held card can leave the card stuck while the turn is finishing

fixed in 7af470e: Send now on a held card behind a running turn is refused with 409 held_turn_running before any claim, promote or release, so no card is ever released for a turn-end drain to miss. _run_released_card, release_held_queue_entry and HOLD_RELEASED_KEY are deleted. A held card runs only on the idle Send now path, which dispatches it by required_queue_id under slot._lock.
Regression: test_send_now_on_a_held_card_is_refused_while_a_turn_runs fails without the refusal.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): Held hint promises an Edit button that crewmate chats don't have span=ac46a35fbd05

fixed in 7af470e: a stack with no onEdit, which is the crewmate chat's held stack, now uses components.queueStack.held_hint_no_edit: "Restored after a restart. It won't send on its own. Send it now, or cancel it." The key is in every catalog, with en-XA regenerated. held_hint now says "cancel" instead of "delete", matching the X button. ChatPane.queueActions.test.tsx checks the crewmate Held card's title offers no edit.

Not changed in this push: the evidence-gap screenshots.

@Pearcekieser

Pearcekieser commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): a dispatched held send can be restored held after a crash span=4803882cd425

fixed in 8b18324: the separate awaited queue save in _persist_dispatched_send is gone, since that save was the window. The turn marker now carries the row's queuedEntryId (set through a TURN_DISPATCH_ROW ContextVar around _begin_local_turn_marker, kept and bounded in _local_turn_prompt), and _reconcile_local_turn_marker drops that entry's stale queue copy on restore. The marker and the dequeue land in one write. A periodic flush that saves the queue before the marker is a gap main also has; the atomic-dequeue follow-up closes it.

Regression: test_a_written_send_cut_after_its_marker_is_not_restored_held fails without the marker field.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): a refused held Send now still claims a stop span=5a68e05795fe

fixed in 8b18324: when the queue holds a held card, api_chat_slot_interrupt reads the body before the stop claim, refuses there, rechecks _stop_state, and only then claims. A refused Send now no longer moves _stop_generation, so a queued recovery continuation keyed on it is not purged.

Regression: test_send_now_on_a_held_card_is_refused_while_a_turn_runs asserts _stop_generation is unchanged and fails without the early read.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): The tooltip on a disabled Send now promises an automatic send that never happens span=6d61c6c4afeb

fixed in 5850170: components.queueStack.held_send_now_after_turn now reads "Available when the current turn ends." in en, and says the same in every other catalog, with en-XA regenerated. No catalog promises the card will send itself.

Not changed in this push: the evidence-gap screenshots.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • span=85e61fc1149b issue (blocking): A Send now press can interrupt the turn it just started

fixed in 5850170: before the early body read, the handler records the executing turn as (slot.task, slot._turn_generation, slot.turn_running). If a turn was executing and that identity changed during the read, the press stands down: it promotes the named card, claims no stop, and calls neither _unblock_pending_waits nor stop_turn, as the claim-first path does when its claim is superseded. A slot that was only reserved falls through to the claim.
Regression: test_send_now_stands_down_when_the_turn_changes_during_the_body_read fails without the identity check.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • span=096d6517711e issue (blocking): After a crash, a merged crewmate turn comes back partly held

fixed in 5850170: the drain now sets TURN_DISPATCH_ENTRY_IDS (dispatched_entries(consumed)) around the turn task's creation, so the turn inherits every consumed queue id. When the marker's opener copy comes from a written send's row, it carries the other merged ids as queuedEntryIds. _local_turn_prompt keeps and bounds that list, and _reconcile_local_turn_marker retires every named entry from the restored queue, then strips the key before re-appending the row.
Regression: test_a_merged_run_of_written_sends_is_retired_whole fails without the reconcile change.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): Preserve every send consumed by a merged crewmate turn span=c146d09d67c5

fixed in 5850170: the drain sets TURN_DISPATCH_ENTRY_IDS (dispatched_entries(consumed)) around the turn task's creation, so admission sees every consumed queue id. The marker's opener copy from a written send's row carries the other merged ids as queuedEntryIds, _local_turn_prompt keeps and bounds that list, and _reconcile_local_turn_marker retires every named entry. No merged entry comes back held, so _unsent_send_filter drops none of their rows from the Resume replay. The merged request's other rows are already in the transcript, written at send, so the marker needs to carry only their ids.

Regression: test_a_merged_run_of_written_sends_is_retired_whole fails without the reconcile change.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): Base's resume path for restored parked sends stops working, and the description doesn't say so span=3a6f25249f72

fixed in eb4a449: the hold no longer covers the sends drain_restored_parked_sends accepts. When a durable parked work row shows a restored send was accepted for dispatch, the boot drain stamps the entry with the process-local BOOT_ACCEPTED_KEY, and is_held_queue_entry returns False for it. The drain, queue_held_depth, the slot frame and the slot-detail snapshot then all treat it as ready, including while the drain waits on memory preparation. The stamp is not in _DURABLE_QUEUE_KEYS, so a later restart hands the entry back held and the boot drain decides again. The description now says accepted parked sends still resume on boot.

Regression: test_shared_crew_dispatch_boundaries.py (calls == ["accepted 0", "accepted 1"], unchanged from main) passes again, and a new check there that only the unaccepted cards count as held fails without the stamp. test_a_boot_accepted_restored_send_is_not_shown_held in test_slot_detail_offload.py covers the slot-detail snapshot.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): Preserve the opener identity after live-window eviction span=c146d09d67c5

Rebutted, no code change (adjudication downgraded it). The trigger needs four things at once: a queued crewmate send whose row was written at send, the 10,000-row window cap evicting that row during one streaming reply, the drain then reusing it through the durable-position check, and a crash between the marker write and the next periodic queue flush. When all of that happens, the entry comes back as a held card, which is the safe outcome: nothing is resent without the person, and Send now or cancel is one click. Carrying the opener identity for a row outside the live window would need the marker to read the on-disk transcript at admission. The atomic-dequeue follow-up, which writes the dequeue in the same save as the marker, closes this window and the periodic-flush gap together.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): A stale interrupt overwrites a later queue selection span=5a68e05795fe

fixed in 6a56017: the held-queue early body read now captures _stop_generation with the turn identity, and the changed-turn stand-down promotes the press's card only while that generation is unchanged. A later stop that landed during the read keeps the selection its own press made; the older press stands down without promoting or stopping anything.

Regression: test_a_stale_press_does_not_overwrite_a_later_stops_selection fails without the generation check.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): Retiring merged cards leaves native Resume without the complete request span=0755d1f84a9c

Rebutted, no code change (adjudication downgraded it). Retiring the merged cards is correct: the marker proves that turn started, and leaving them as held cards would let Send now run them a second time. The preamble gap is separate and not new. build_interrupted_turn_preamble names only the newest user row of any merged run, and main does the same for a merged run cut by a restart. Every merged send's row is in the transcript, written at send, so the cold-start replay and the person both still see the whole request. Making the native-resume preamble name a whole merged run belongs with the atomic-dequeue follow-up, which reworks what the marker records.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

resolution: removed -- SEAM LIMIT retrospective: the marker-identity retirement chain is out of this PR, and the atomic-dequeue follow-up owns the class
mechanism: turn marker names the queue entry it ran, and the restore retires that entry

  • issue (blocking): Preserve dispatch identity after the written row leaves the window span=c146d09d67c5

removed in 5a1d003: this span was answered in 10 rounds, so the retrospective removed the whole mechanism instead of a third witness for it. _local_turn_opening_row, TURN_DISPATCH_ROW, TURN_DISPATCH_ENTRY_IDS, queuedEntryId / queuedEntryIds in the marker copy and the reconcile retirement are gone; chat_turn/turn_marker.py, chat_turn/turn_admission.py and slot_persistence/turn_marker.py now match main. The class that remains is wider than the eviction case: any written crewmate send cut between its turn marker and the next queue save (5 s or less) comes back as a held card, and its row stays out of Resume until the person sends or cancels the card. Nothing is resent without the person, and the words stay on the card and in the transcript. Writing the dequeue in the same save as the turn marker fixes the class at its source; that is the atomic-dequeue follow-up PR named in the description. test_a_written_send_cut_after_its_marker_is_not_restored_held[True] pins the residual as xfail(strict=True), so it fails the moment that follow-up lands.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • issue (blocking): Preserve the opener of a queued turn that actually started span=9dd41fe0962a

Rebutted, no code change (adjudication downgraded it as disproportionate). This is the residual the SEAM LIMIT retrospective accepted when it removed the marker-identity chain: a written crewmate send cut between its turn marker and the next queue save (5 s or less) comes back as a held card, and _unsent_send_filter keeps its row out of Resume while the card is held. Nothing is resent without the person, and the words stay on the card and in the transcript; Send now on the card runs the request. The PR description names it, test_a_written_send_cut_after_its_marker_is_not_restored_held[True] pins it as xfail(strict=True), and the atomic-dequeue follow-up, which writes the dequeue in the same save as the marker, removes the stale card at its source. Retiring the card from the marker instead is the mechanism this PR just removed after 10 rounds on span c146d09d67c5.

A queue restored after a restart has no owner: nothing drains it on boot, by
design. Every reader still treated a non-empty queue as work about to run. The
sidebar spun "In progress", the board filed the chat under Working, the queue
card's hourglass kept flipping, and Continue refused with slot_queue_pending,
so the restart notice's "resume to continue" had no working control. A later
send then ran first and the restored card ran on its own behind it.

Restored entries are now held (is_held_queue_entry). The drain passes them by
and starts one only when the person names it (Send now). Continue, the
completion sound and session status count queue_ready_depth. The slot frame
carries queue_held_depth and each held entry meta.held, so the sidebar, board,
command center and queue card stop showing a held card as working.

The turn marker's metadata merge also writes the post-dequeue queue in the same
atomic write as the opener copy. Written apart, a SIGKILL between them restored
the consumed entry twice: as the interrupted turn's row and as a queue card.
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

resolution: removed -- the xfail pin is gone; the residual it pinned stays stated in the description with the atomic-dequeue follow-up

  • issue (blocking): Do not waive recovery of an already-started queued send span=7cc36cefcf64

removed in 347cfa7: the xfail(strict=True) case is deleted, so no expected-failure exemption is left in test/test_restart_held_queue.py. The test that remains, test_a_written_send_cut_before_its_marker_is_restored_held, asserts the case this PR handles: a written send cut before its turn marker comes back as one held card. The file is new in this PR, so no ratchet on main is loosened. The residual the pin pointed at is unchanged and stated in the description: a written crewmate send cut between its turn marker and the next queue save comes back held, and its row stays out of Resume until the person sends or cancels the card; nothing is resent without the person. Fixing it means writing the dequeue in the same save as the turn marker, which is the atomic-dequeue follow-up PR from main, and that PR adds the test for the after-marker case.

@bolichen97

Copy link
Copy Markdown
Contributor

[operator: bolichen97] Continuing this in #33214; thanks for the start.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks for this fix, @Pearcekieser. Closing it as superseded: the restored-queue behaviour it targets landed on main in commit 0c7b3bd ("fix: stop showing a restored-only queue as work coming", #33214), which closed #23970, and the Resume part of #16780 landed in #20029. If something here is still missing on latest main, please reopen with the case and we will take another look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

3 participants