Skip to content

fix: order truncating rewrites against same-key slot takeover - #12276

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
dimwael:fix/rewrite-commit-slot-incarnation
Nov 3, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
dimwael:fix/rewrite-commit-slot-incarnation

Conversation

@dimwael

@dimwael dimwael commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A conversation reopened while one of its edits is still saving never ends up TORN — a tab showing the pre-edit window over a transcript the edit truncated, with the removed turns re-serialized behind the truncation on the next turn and the edit silently undone. The outcome is exactly one consistent history: the edit applied and the reopen reads it (the save committed first; test_takeover_after_commit_reads_the_rewritten_transcript), or the edit refused — reported to its caller — and the reopen reads the pre-edit transcript intact (the reopen's read came first; test_resume_beginning_mid_write_makes_the_truncating_save_refuse). Never both.

Base reproduction: test_a_reopen_racing_an_edit_never_hydrates_the_truncation_with_the_removed_tail_behind_it (test_slot_close_recreation_race_takeover_ordering.py) fails on main (verified on 5494e55c6 with the test file dropped in alone): a tab owing a truncating rewrite is closed, the user reopens it while the archive is queued for its lock, the reopen reads the four-row transcript, the archive then lands two rows plus closed, and the reopen publishes its four-row window over them — close 200, tab shows four turns, disk holds two and is marked closed. The next turn's save re-serializes the four rows plus the new one (the edit is gone, with no error), and the marker the tab never saw makes it vanish at the next restart. Neither consistent history shows the user any of that; "the edit applied, then a reopen reads two rows" is the one the lane accepts, and this test accepts it too — it fails only on the torn state.

Why the smaller alternative does not fix it. The Design lane proposes that the close teardown await any in-flight truncating save for the key before writing its tombstone. The torn case is a READ-side race, not a tombstone-side one: in the reproduction the close's archive is the truncating save, and the reopen's read is already done when it lands — awaiting the save on the close path cannot order a read that began between the pop and the commit, and the tombstone only decides whether a later delivery rehydrate is a reopen (it never gates the explicit resume endpoint, which exists to reopen closed tabs). The original #12090 report is the same shape with a rewind in place of the close: a resume reading while the rewind's save is queued, no close involved. What orders the read is the arbitration this PR adds — a takeover note that the save's commit gate consults, and a read that waits for a pending truncating save it did not overtake — which is also why test_resume_beginning_mid_write_makes_the_truncating_save_refuse pins the refused-edit history rather than the edit-applied one: a reopen that already read the pre-edit transcript must not have a truncation land under it.

A truncating history rewrite (rewind, edit-resend, regenerate, switch-variant) can commit its truncated window onto a transcript that a same-name replacement slot has taken over. The user-visible symptom (the base reproduction above): close a tab while its rewrite is saving, reopen the same conversation, and the reopened tab shows the pre-edit window over a transcript the rewrite truncated and closed — the next turn writes the removed turns back and the edit is lost without a word; a restart loses the tab.

Why it matters

The replacement resumes the same conversation the user just had open, and ends up holding a window the transcript no longer matches: a rewrite that semantically belonged to the torn-down tab lands under it, and the replacement's next save overwrites the rewrite. The commit-boundary incarnation check (expected_slot_name) cannot cover this: the save holds the per-session lock across its whole read-modify-write, so a replacement that begins mid-write is simply blocked from reading until the save releases — and then reads the truncated content. Shrinking the check-to-write gap changes nothing, because the harm is the read landing after the replace, not the publication landing before the check.

Not a goal

  • Serializing slot publication against the per-session lock, or taking any lock on state._slots.
  • An atomic_write pre-replace hook (313 call sites untouched) or holding a mutex across the commit instant.
  • Deferring publication until an in-flight save settles — publication never waits; the contract decides a winner instead.
  • The RENAMED-replacement axis (expected_history_key), which main already covers, or the slot-create transaction of fix(chat): build and file nothing for a create that is refused #15592 (a rebase neighbour, not a duplicate).
  • Serializing the explicit user resume against the close: when both are in flight the arbitration decides (the resume that began first wins, see fix(dashboard): an unpublished resume that begins mid-teardown replays rows the user's rewind removed #18898 option (A), implemented here); a resume that began AFTER the close is a reopen of a closed tab and is refused by the closed flag as any such reopen is.

What changed (motivation → approach → change)

Symptom → root cause: slot publication (event loop) and the persistence commit (worker thread) have no ordering contract. The cleanup pops state._slots[name] and the facade republishes through SlotRegistry.put_slot; neither takes the per-session lock, and no pre-commit map re-read can observe a replacement that does not exist yet.

Approach: the issue sketches a claim that put_slot consults, with same-name publication deferred until the claim clears and the write refusing when a publication is pending. This PR lands the same contract with a strictly simpler arbitration: instead of deferring publication, a single mutex-guarded compare-and-set decides the winner, so publication never waits and there is no deferred-publication machinery to keep alive.

Change:

  • TruncationClaim (slot_registry.py): every truncating save registers a claim on its map key for the whole write (opened before the per-session lock, so the lock wait is covered). Claims for one key are held as a list, because a popped slot's hand-over drain and its replacement's own rewrite can be in flight for one key at once.
  • Every same-key takeover marks the claim via SlotRegistry.note_same_key_takeover: called by put_slot itself (publication) and at the start of a resume's transcript read (rehydrate_slot_from_history_async, chat_api/resume.py) — before the read is dispatched, which is what makes a mid-write takeover visible to the save at all. Fire-and-forget; callers never branch on it.
  • The note is tombstone-gated (_slot_closed_before, chat_persistence.py, reading channel_slots._RECENT_CLOSES): a noter whose start postdates the key's close tombstone is a reopen of a closed tab, not a takeover — it notes nothing, waits for the ordering to settle, and is then refused by the closed flag the close wrote. A delivery rehydrate therefore never wins a close (one begun before the close is refused by its own slot_closed_since check); only an explicit user resume that began before the close does.
  • A close — the single-tab close and the bulk cleanup alike — that handed its key over while the taker was still unpublished keeps its dismissal OWED (DeferredDismissal, SlotRegistry.hand_basis_to_dismissal): the teardown watch outlives the close (non-truncating, so the taker's read is not held). It settles only where that is FINAL: a non-noting publication settles it at its insert; a noter's publication settles it at the noter's success exit (settle_dismissal_on_publish — both noters insert before their last refusal point, so the insert itself is not final); the last standing taker's retraction returns it to be written (takeover_ordering.record_deferred_dismissal, under the transcript lock, guarded at write time on the key still being unpublished so a reopen that landed meanwhile is never stamped closed, only onto an unclosed line of a still-existing session) and it is cleared from the watch only once that write is confirmed (settle_written) — a failed write leaves it owed and the next exit on the key (owed_dismissal) retries it. Both paths share that protocol: with nobody standing the arm writes the marker itself against the same registered record, and because no later exit is guaranteed to retry it, a write that does not land makes the close report history_save_failed (the cleanup files the key under failed) rather than a 200 the recent-session restore would contradict; the dismissal stays owed on its watch so any later taker's exit still writes it. So a taker that gives up after the close answered 200 — before or after registering its slot — leaves the tab dismissed rather than resurrected by the recent-session restore, and a dismissal is never lost to one failed write.
  • A noter that noted and then published nothing — refused after its note, or handed the key to nobody — retracts the note it opened (SlotRegistry.open_same_key_takeover_note returns a frozen TakeoverNote(watch_id, seq) — the admitting watch's process-unique id, never reused, plus the note's sequence on it; retract_same_key_takeover withdraws exactly that note from TakeoverWatch.standing and only while the key's watch is still the issuing one, so a stale handle from a retired watch is a no-op against the fresh watch's same-numbered note; both noters retract in a finally when no slot was published). Accounting is per note and basis-relative, never a net counter: basis_moved and the claim registration both ask "does a note newer than this basis still stand?", so one noter's refusal — before or after the basis opened — can neither hide nor fabricate another taker's note. Without it a close whose archive lost to a noter later refused reported 200 with no closed written and no session removed, leaving the key to nobody.
  • A takeover firing while a save is still in dispatch has no claim to mark, so dispatchers open a refcounted takeover watch (TakeoverBasis) at their last synchronous instant — save_slot_off_loop, the periodic flush pass, the shutdown pass, and the rewind handler's dispatch — and begin_truncation_claim compares the watch's count against the snapshot: a moved count means the key changed hands in the gap and the claim is born already overtaken. save_slot_off_loop opens a basis for every save its caller did not open one for — the worker reads _pending_rewrite itself, so an append-shaped dispatch can become a truncating rewrite in that gap, and only a basis opened before it can observe a takeover that landed there. The vouch itself is a dispatch-time occupancy verify: guarded callers (expected_slot_name, wired into rewind and edit-resend as well) and the flush/shutdown passes re-read the live occupant and refuse or skip when the key is already someone else's. The watch table retains a key only while a basis is open for it, so it is bounded by in-flight saves.
  • The close/cleanup teardowns (chat_api/slot_lifecycle.py) write POPPED slots by design, so they open one watch before their pop and carry it through every await and every save of the teardown; the hand-over drain they call (_persist_handover_tail, chat_api/handover_drain.py) takes it as takeover_basis, handed over only where the key names the drain's own file (_teardown_drain_basis, same module). A refusal is not a commit: the teardowns check the save's result, and basis_moved routes a takeover refusal to the hand-over arms (drain the tail, hand over the key-scoped markers, report the close honestly). A delete-won refusal is verified positively (session_deletion_confirmed, slot_persistence/write_guards.py: definitive evidence only, an unreadable probe is not a deletion) before being ignored; any other refusal restores the tab when the key is still free or drains its tail, and reports the close failed rather than discarding acknowledged rows. A hand-over drain carrying a _pending_rewrite retry whose key was taken over does not rebuild: it persists the window through the history layer's id-deduped appends (append_if_absent under atomic_appends), so the unsaved tail lands and the lost truncation drops out.
  • The commit gate (commit_truncation_claim) decides committed vs overtaken in one atomic transition, placed before the rewrite's first mutation — ahead of the payload composition whose dropped-line archive is that mutation, and ahead of the file replace, both under the per-session lock the save holds throughout. Losing refuses the whole save (False, nothing written). For the hand-over shape the refusal lands the popped slot's unsaved rows append-safely under that same lock, ahead of the takeover's queued read.
  • Declared failure modes, both intended. (1) A reopen whose takeover note overtook a save still in flight waits for that save to settle (bounded by the save's own lock deadline plus headroom, ~14 s); on expiry it refuses with the retryable 503 takeover_ordering_unsettled at POST /api/chat/slots/{slot}/resume (HistoryLockTimeout for a delivery rehydrate) rather than hydrate a window the refused save appends behind — the user sees the resume's error toast and reopens again. (2) A close whose archive lost the ordering does not report a success it did not commit: it hands the key over (drains the original's tail append-safely, reports 200) when a taker holds or is reading the key, and otherwise restores the tab and reports 500 history_save_failed — the user sees the tab stay open with its unsaved tail dirty and closes it again. Both are recorded (SEL history_save_refused; the close's own log line) because a refusal that is folded into a success is exactly the a-refusal-is-not-a-commit shape.
  • The read side honours the ordering: a resume whose takeover note overtook an in-flight save waits, off the loop and bounded by the save's own lock deadline, for the overtaken claims and the key's open watch to retire. After the wait the endpoint re-reads the close metadata (get_metadata_status; unreadable refuses) and derives defer_clear / the reserved publish / cleared_closed_at from that read — the save it waited for can be the user's own close committing closed=True — while the pre-wait meta stays the identity witness the later checks compare against. The wait fails closed — HistoryLockTimeout on the facade, 503 takeover_ordering_unsettled at POST /api/chat/slots/{slot}/resume — rather than hydrating a window whose missing tail a later rewrite deletes.
  • ConversationLog.append/append_if_absent (bodies in history_append.py, the owner module history.py binds them from) accept a fully built entry, so the append-safe fallbacks persist the same ts, provenance, variants and meta the committed payload would have.
  • Both append-safe fallbacks settle the transcript header before their first row, under the lock the appends take (tighten_line_for_appends, slot_persistence/metadata_line.py): append mints a mode-less header for an absent file and leaves a present one untouched, so a restricted slot's rows would otherwise land under a line every learning reader takes as persistent. The header is created with the slot's retained memory_mode or tightened to it, the carried execution record folded to match, the store a restricted line must not name cleared, and the replacement's own fields kept — the same ratchet the rows-only save applies. The settled mode is CONFIRMED by re-reading the header under the lock: a header that cannot be read (the merge defers on a transient read without running its guard), or that still reads looser than the retained mode, refuses the append outright (HeaderUnconfirmed, write_guards.py) — the drain reports the rows and the close fails, the commit-gate fallback keeps the window owed.
  • The commit gate's claim refusal calls _keep_owed_after_refusal, like the sibling retryable refusals, so a periodic flush that lost its claim leaves the slot dirty; the dispatch-verify refusal in save_slot_off_loop does the same.
  • Both append-safe replays go through one helper, write_guards.append_window_rows: header settled first, each row built by _build_message_entry, appended id-deduped — there is no second row pipeline.
  • Both replays judge "already persisted" against the WHOLE durable history: ConversationLog.append_if_absent gains include_rotated, which folds the size-rotation archives (read_rotated_messages, the projection's cached corpus, resolved by the PHYSICAL file stem rotation names its segments after — a pre-migration Slack transcript addressed as slack:<ts> lives in the bare <ts> file and its archives are <ts>__*) into its absence check under the same lock, so a window reaching past the live file's rotation cut lands only the rows the transcript lacks instead of re-landing every archived row under a duplicate identity. Single-row writers (cron/workflow results) leave it off. An archive that cannot be enumerated raises into the fallbacks' refusal arms rather than reading as "absent".
  • The drain prepares its rows the way the full save does: the window snapshot goes through drop_notes_authorized_elsewhere (a note row stamped for a session the slot was rebound away from is dropped and audited as note_save_drop, as every full save does) before _build_message_entry builds each entry — an append that skipped the filter would land such a row in the other session's transcript. Every transcript-identity comparison this change makes is by stem (transcript_stems, the identity replacement_shares_transcript compares by): note_same_key_takeover records a declared read target as its stems, takeover_noted_transcript and _teardown_drain_basis match by stem, so a resume declaring slack_<ts> is seen by a teardown writing slack:<ts>.
  • A hand-over whose drain does not land (both append-safe writes failed, say on a full disk) does not drop the popped original: the replacement that moved the watch may itself have left the key (a recreate closed again), so the key can still be free, and the original comes back under it with the compensation a raised archive save already gets (_restore_original_after_failed_close, chat_api/slot_lifecycle.py: restore while _slot_still_ours, the retired nudge loop and the app's dismissal taken back with the tab, the restricted marker re-derived); the close reports history_save_failed, the tab stays dirty for the flush, and the waiting resume dedups to the restored tab. A replacement already published keeps the key and the failure is reported, as main's published-replacement exit does. The idle sweep's takeover arm does the same.
  • A teardown watch retains at most _MAX_NOTED_TRANSCRIPTS_PER_WATCH (64) declared read targets (slot_registry.py): the name comes from the resume request body, so the bound is enforced at admission — note_same_key_takeover returns False and records nothing (no count, no claim mark) for an overflowing note, and the resume behind it is refused (503 takeover_note_refused at the endpoint, HistoryLockTimeout on the facade) before its read runs. Publications and already-held names are always admitted.
  • The drain's append runs as a shielded executor future and hands its takeover basis to the future's settlement on cancellation (_transfer_basis_to_worker, the hand-off save_slot_off_loop already makes), so a shutdown cancelling a close mid-drain neither drops the acknowledged tail's writer nor retires the watch under the running thread; the rewind handler makes the same hand-off for a rewrite that did not settle within its cancellation drain.
  • Watch ownership survives cancellation: every executor save in save_slot_off_loop is shielded, the worker closes its own basis when it settles, and a caller-opened basis transfers to the worker on cancellation (_transfer_basis_to_worker).
  • Switch-variant captures every field adopt_variant_text touches (redaction records included) and restores them when its durable write is refused — before any answer goes out, the app-token path's anti-enumeration 404 included, so a takeover that pops the slot mid-save cannot have the drain persist a variant the caller was told failed.
  • Claims and takeover notes carry the transcript they write / declare (stems, _transcripts_meet): a publication notes slot_history_key(slot), a resume the key it reads, and only a matching (or unknown) pair marks a claim overtaken or leaves one born so — a same-name publication resuming a different transcript no longer fails a linked tab's closing archive.
  • takeover_ordering_pending counts a watch only through its TRUNCATING holders (dispatchers state the shape at open_takeover_basis(truncating=); a worker raises an append that turned truncating in its dispatch gap via mark_basis_truncating before its claim); the read-side wait is loop-side (SlotRegistry.await_takeover_ordering_settled: a future registered under the claim mutex, resolved through its loop when the last pending basis or claim retires), so a burst of resumes holds no executor thread the saves need.
  • The two delete witnesses (session_was_deleted, protective; session_deletion_confirmed, destructive) map one shared verdict, write_guards._session_delete_evidence → unobserved | deleted | present | unverifiable.
  • The takeover arbitration's sync rules live in a new owner, slot_persistence/takeover_ordering.py (_slot_closed_before, admit_rehydrate_takeover_note, _truncation_claim_scope, open_dispatch_basis, _transfer_basis_to_worker, refuse_overtaken_rewrite), bound on the chat_persistence facade like write_guards. The save transaction, its on-loop entry point and the rehydrate drivers now live where upstream's chat_persistence split put them — chat_persistence_parts/save_transaction.py (save_slot_off_loop with its basis plumbing, _save_slot_to_history with the claim scope and commit gate, the shutdown pass's watch) and chat_persistence_parts/slot_builders.py (the rehydrate_slot_from_history_async wrapper, _rehydrate_slot_from_history_noted, and the loop-side _await_takeover_ordering_settled) — composed onto the facade as before, so every chat_persistence.<name> patch still reaches them. That split also took chat_persistence.py to ~850 lines, under its line-cap baseline row.
  • record_deferred_dismissal reports written | superseded | failed: superseded is a publication that took the key and settled the dismissal while the write was queued (a plain same-name create between the hand-off and the write) — nothing was owed and the tab is legitimately live, so both close arms treat it as a committed close; only failed (the write raised, or the line is unreadable) is reported as history_save_failed / under failed.
  • The owed marker is ordered against explicit resume reads (the marker side yields): the resume endpoint registers its read on the key's watch before its close-metadata snapshot (SlotRegistry.begin_resume_read / end_resume_read) and the write-time guard answers False while one is in flight, so a marker never commits between a reopen's header snapshot and its publish — the deferred holder is deliberately non-truncating (readers are never held behind a dismissal), so such a reopen records no note, waits on nothing and would otherwise publish against an open header with the marker on its line. The resume's own exit then settles the dismissal (publish) or writes it (refusal, or a cancelled resume that was the last reader); the pre-close-resume-wins / delivery-rehydrate-loses contract is unchanged. "Published" is an occupancy test — the slot the resume answered with must now hold the key the wrapper noted — because _live_slot_for_resume dedups by session identity across slots and can answer a resume of key A with a live slot under key B, which publishes nothing on A; that answer takes the owed-dismissal path like a refusal. The rehydrate wrapper needs no such test: its body only ever returns state._slots[slot_name], the key it was asked for. The registration is a registry transition and get_metadata a cached file read, while the marker's guard runs under the transcript lock, so the two can interleave (guard sees no reader → resume registers and snapshots an open header → rename); to close that window the resume takes its close-metadata snapshot UNDER the transcript lock whenever the key carries a watch (takeover_ordering.close_metadata_under_lock, off-loop, no await or registry mutex held inside): a marker that landed before the read is seen and defer_clear clears it before publishing; one that has not landed cannot land afterwards, because the guard now finds the read. The publish side is ordered too: a plain same-name create takes no lock, so its put_slot can settle the dismissal between the guard's "still owed" and the rename; the record is the witness through the write, and after the rename commits — still under the transcript lock (update_metadata_if's after_commit_under_lock) — the ownership is read again — the hook only records the verdict, honouring update_metadata_if's I/O-free contract — and right after the call returns a marker a publication has since won against is un-stamped as a compare-and-clear under the transcript's own lock (unstamp_superseded_marker: clear_closed(only_if_closed_id=our closed_id), keyed on the close identity — every close write stamps a closed_id nonce beside closed_at (metadata_line.close_identity for the slot saves, DeferredDismissal.closed_id for the owed marker; a slot-owned key, dropped with the stamp on reopen) and the clear compares it under the transcript lock, so exactly our stamp comes off: a legitimate close of the new tab stamped on the same clock tick carries an equal closed_at (Windows ticks ~15.6 ms) and its own id and is kept, and a stamp carrying no id — an older build's line — is never ours to clear), the outcome superseded. A clear that fails is parked as an owed un-stamp on the key's watch (OwedUnstamp, park_owed_unstamp) — the dismissal itself is settled, so it cannot ride the owed-dismissal retry — and every noter exit on the key retries it (retry_owed_unstamp) until it lands or the line provably carries a different stamp; a settled dismissal is never reported as a failed close.
  • record_deferred_dismissal settles a skipped write only on evidence — closed on the line, or the session gone — never on the line merely reading back: a transient read failure inside update_metadata_if answers False without writing, and the open line it leaves is failed (still owed, retried by the next exit), not written.
  • The owed marker carries the INCARNATION it belongs to: DeferredDismissal.created_at is the closed transcript's created_at as the slot last observed it (_disk_meta_created_at, the same witness the delete probes use), and the write guard — inside the update_metadata_if lock — requires the line's created_at to match. A session permanently deleted and the same name-derived key created again while the dismissal is owed is a different conversation: the marker is dropped and the dismissal settled as satisfied (superseded), never retried against the new line. A closed slot that recorded no identity (a legacy line) owes a marker nobody can place: it is not written and the dismissal settles rather than retrying forever — stamping a line one cannot identify is the harm the witness exists to prevent. Both close arms pass the witness.
  • Rewind dispatch-verify audit (reported in triage): the rewind handler's save_task is awaited only on the branch that dispatched it, so a key that changed hands before dispatch settles saved = False directly and audits as history_save_refused rather than tripping an UnboundLocalError into history_save_exception.

Rebased onto current main (8c7417ea8): the contract sits inside the slot_persistence/ and chat_api/ owner splits (the commit gate precedes transcript_merge.compose_payload; every new name an owner function reads is bound on the chat_handlers facade), composes with #13381's guarded-write registry and is_closing fence (the fence read, the dispatch verify and the registration share one synchronous block), and the two composition-contract pins and the #13381 ordering pins were updated to the current shapes. slot_lifecycle.py crossed the 2,000-line cap on the rebase (main alone is 1,651); the two repair/reset endpoint handlers api_chat_slot_repair_images and api_chat_slot_reset_conversation move to chat_api/slot_repair.py unchanged, per the responsibility-split rule (new slot_repair owner row and _BASE_SURFACE row in test_chat_handlers_composition_contract.py; the path pins in test_json_object_body_guard.py and test_slot_running_readers.py and the owner list in learn-cron-dashboard.md follow it).

Backwards compatibility

Compatible: the claim only ever refuses a truncating save when a same-key takeover began during that save's own write — a schedule in which landing the write was the defect. Saves with no concurrent takeover decide committed and behave exactly as before; put_slot keeps its signature and return; takeover_basis is an optional keyword on _save_slot_to_history / save_slot_off_loop / flush_slot_now; entry is an optional keyword on append / append_if_absent and include_rotated (default off) on append_if_absent; owners without a claim table (partial test doubles) fall back to the pre-contract shape via the same getattr idiom as _slots_under_construction.

Every existing call shape, return value and error code is kept; the diff adds three refusal paths (503 takeover_ordering_unsettled and 503 takeover_note_refused on resume, and 500 history_save_failed on a refused close archive) and widens no validator. The one thing that leaves a module is the responsibility split: api_chat_slot_repair_images and api_chat_slot_reset_conversation move from chat_api/slot_lifecycle.py to chat_api/slot_repair.py with their bodies unchanged, and every reader still resolves because none of them imported the owner module directly -- they all read the chat_handlers facade, which composes the new owner.

Reader: src/kiro_crew/dashboard/routes/chat.py:api_chat_slot_reset_conversation -- chat -- route registration reads chat.api_chat_slot_reset_conversation, bound by dashboard/chat.py from the facade; test_chat_slot_reset_conversation.py, test_teardown_routes_sibling_alias_busy.py
Reader: src/kiro_crew/dashboard/routes/chat.py:api_chat_slot_repair_images -- chat -- route registration reads chat.api_chat_slot_repair_images from the same facade binding; test_chat_slot_repair_images.py (12 tests)
Reader: src/kiro_crew/dashboard/chat.py:api_chat_slot_repair_images -- chat -- re-export from chat_handlers, which now imports both names from chat_api.slot_repair (lines 77-78 unchanged); test_dashboard_structure_lock.py::test_reset_conversation_keeps_the_lock_and_keeps_the_unlock
Reader: src/kiro_crew/dashboard/chat_handlers.py:api_chat_slot_reset_conversation -- chat -- the facade composes _owner_slot_repair and imports the two names from it, so patches of chat_handlers.<name> still reach the moved bodies; test_chat_handlers_composition_contract.py (slot_repair owner row and _BASE_SURFACE row), path pins in test_json_object_body_guard.py and test_slot_running_readers.py
Reader: docs/system-specs/modules/learn-cron-dashboard.md:slot_repair -- chat -- owner list updated in this diff; docs/request-for-change/rfc-transcript-section-markers.md and docs/feature-map/README.md name slot_repair.py as the handlers' file (docs only, nothing executes them)

Tests

test/test_rewrite_publication_ordering.py (28 tests; test_append_safe_fallback_never_lands_a_row_that_sorts_before_the_one_it_follows (a prebuilt entry whose own ts predates the rows on disk is stamped one microsecond past the last of them by history_append.append, the rule a clock-stamped append already follows, so a reader ordering by ts never places the fallback's row ahead of the row it followed; fails on the previous head), test_a_resume_behind_a_saturated_executor_still_settles proves the loop-side wait on a two-thread default executor with a probe job, test_an_append_shaped_save_does_not_make_the_ordering_pending, test_a_dispatch_that_turns_truncating_raises_its_basis_before_its_claim), staged with the sibling close-race suite's fixtures and its bounded-seam parking idiom (the seam is the metadata-line build, which runs inside the lock after the map recheck and before the commit gate):

  • test_resume_beginning_mid_write_makes_the_truncating_save_refuse / test_publication_mid_write_makes_the_truncating_save_refuse — the acceptance race from the issue, via a resume and via put_slot: the save refuses, disk keeps all rows, no dropped-line archive is written.
  • test_truncating_save_commits_when_no_takeover_interleaves / test_takeover_after_commit_reads_the_rewritten_transcript — the two sequential histories.
  • test_takeover_during_dispatch_makes_an_unguarded_truncating_save_refuse, test_a_save_that_becomes_truncating_after_dispatch_still_loses_to_a_takeover (an append-shaped dispatch whose worker adopts _pending_rewrite in the gap, with a takeover landing in the same gap: the save refuses, the transcript is untouched), test_a_stale_basis_leaves_the_claim_born_overtaken, test_takeovers_of_unwatched_keys_leave_no_record, test_flush_pass_skips_a_slot_whose_key_changed_owner_mid_pass, test_shutdown_saves_the_occupant_that_won_a_refused_key — watch semantics and the dispatch-time vouch.
  • test_cancelled_dispatch_keeps_the_watch_open_for_its_running_worker, test_cancelled_caller_leaves_its_watch_open_for_the_running_worker — watch ownership across cancellation.
  • test_refused_handover_save_lands_the_tail_ahead_of_the_takeover_read, test_drain_reports_a_mid_save_takeover_refusal_as_the_append_it_becomes, test_append_safe_fallback_persists_the_complete_row, test_commit_gate_fallback_lands_private_rows_under_a_restricted_line, test_a_refused_claim_keeps_the_window_owed_through_the_flush, test_commit_gate_fallback_does_not_duplicate_rows_rotated_into_archives (a real flush_slot_now whose save loses its claim and whose fallback cannot confirm the header: nothing appended, _dirty survives the pass) — the append-safe arm, the entry passthrough, and the header ratchet at the commit gate (an incognito slot's refused hand-over save over a persistent replacement line: mode tightened, store cleared, carried execution folded, the replacement's title kept, the derivation seam withholds the file).
  • test_resume_waits_for_a_save_still_in_dispatch_under_an_open_watch, test_resume_refuses_rather_than_reading_unsettled_history_on_timeout — the read-side wait and its fail-closed expiry.
  • test_archive_dedup_resolves_the_physical_stem_of_a_legacy_transcript (a bare <ts>.jsonl with <ts>__* archives, addressed as slack:<ts>: an archived identity is not appended again), test_takeover_note_marks_every_undecided_claim_and_only_those, test_a_watch_admits_a_bounded_number_of_declared_transcripts (the 65th distinct name is refused with nothing recorded; a held name and a publication stay admitted; the refused resume raises and publishes nothing), test_claim_helpers_tolerate_owners_without_a_claim_table.

test/test_slot_close_recreation_race_takeover_ordering.py (new, 21 tests: the arbitration, the drain's arms, the base reproduction) and test/test_slot_close_recreation_race_takeover_retraction.py (new, 27 tests: the tombstone-gated note, TakeoverNote retraction and basis-relative accounting, the owed dismissal, the clock-frozen pre/post-close variants) — both split out of test_slot_close_recreation_race.py to respect the 2,000-line cap the way upstream's _handover_metadata / _privacy_ratchet parts were, sharing slot_close_recreation_race_helpers.py; the 4 pre-existing close-race tests whose save doubles now return a value stay in test_slot_close_recreation_race.py: test_a_resume_that_waited_out_a_committing_close_does_not_publish_over_its_marker (a reopen parked in its ordering wait while the close's archive commits closed=True: the reopen decisions come from the post-wait metadata read, so the resume refuses or clears the marker, never publishing over it), test_close_archive_yields_to_a_user_resume_begun_before_the_close (end-to-end through the real close handler and the real resume endpoint: the resume is observed in its ordering wait before the close's cancel completes, the close hands over with 200, the resume publishes the untruncated transcript), test_close_archive_commits_over_a_delivery_that_begins_mid_teardown (a rehydrate begun after the pop notes nothing: the archive commits closed, the session is removed, the rehydrate is refused, disk holds the rewound 2 rows), test_a_rehydrate_begun_after_the_close_does_not_make_it_yield, test_a_noter_refused_after_noting_does_not_make_the_close_yield (the regression: a noter refused after its note, the close takes its retry arm and restores the original instead of reporting a success it never committed), test_a_retracted_note_no_longer_moves_the_basis, test_a_retraction_before_the_basis_opened_does_not_hide_a_later_takeover / test_a_pre_basis_note_retracted_after_the_basis_does_not_hide_a_later_takeover (per-note accounting: a refused noter's retraction, before or after the basis opened, leaves a later real takeover visible to both basis_moved and the claim registration), test_a_stale_retraction_across_watch_recreation_does_not_withdraw_a_newer_note (the handle is bound to the issuing watch: a noter refused after its watch retired does not withdraw the fresh watch's first note), test_a_divergent_publication_does_not_overtake_a_linked_tabs_rewrite_archive (a same-name publication resuming another transcript leaves a linked tab's closing rewrite archive to commit closed), test_a_reopen_racing_an_edit_never_hydrates_the_truncation_with_the_removed_tail_behind_it (the base reproduction, see Goal), test_a_taker_that_gives_up_after_the_close_returned_still_leaves_the_tab_dismissed (an adopt_closed rehydrate cancelled after the hand-over close returned: closed lands, no slot, watch retired), test_a_taker_refused_after_registering_its_slot_still_leaves_the_tab_dismissed[refused|cancelled] (the taker's slot is registered and then its window load fails: the publication was not final, the retraction writes the marker), test_a_publication_that_noted_nothing_settles_the_dismissal_at_once, test_cleanup_keeps_its_dismissal_owed_when_the_taker_gives_up_later (the bulk cleanup arm), test_an_owed_marker_is_not_written_over_a_reopen_that_published_meanwhile (a reopen landing while the owed write is queued keeps its tab), test_an_owed_marker_whose_write_fails_stays_owed_and_lands_on_the_next_exit, test_a_taker_that_gives_up_during_the_drain_leaves_the_close_to_write_its_own_marker[single|cleanup] (the immediate path: the taker retracts during the drain, the arm finds nobody standing and writes the registered record itself), test_a_close_whose_own_marker_write_fails_does_not_report_success[single|cleanup] (the immediate write fails → 500 history_save_failed / failed, dismissal stays owed, a later taker's exit writes it), test_a_plain_create_between_the_hand_off_and_the_queued_write_is_a_committed_close[single|cleanup] (a publication lands while the close's own write is queued → superseded → 200, no marker, the new tab live), test_an_owed_marker_yields_to_a_user_resume_that_read_the_header_before_it (the chain: close hands over → the adopt_closed taker backs out while a user resume is between its header snapshot and its publish → no marker lands, the resume publishes, the live tab's line stays open), test_an_owed_marker_lands_when_the_resume_it_yielded_to_refuses, test_two_resumes_in_flight_and_a_cancelled_one_still_settle_the_owed_marker, test_a_resume_admitted_between_the_markers_guard_and_its_rename_reopens_the_tab (the owed write parked after its guard, before its rename; a resume admitted in that window reads the marker under the lock, clears it and publishes open), test_a_skipped_marker_write_whose_line_reads_open_stays_owed (a transient read failure inside the guarded update → failed, still owed, written by the next exit). test/test_slot_close_recreation_race_takeover_dismissal.py (new, 15 tests; the owed marker's incarnation witness, whose publication settles it, the un-stamp a late publication owes -- keyed on the close identity: test_an_unstamp_keeps_a_newer_close_stamped_on_the_same_clock_tick freezes channel_slots.time, lands the owed marker, closes a reopened tab on the same tick (equal closed_at, its own closed_id) and asserts the un-stamp leaves that close in place; fails with the instant-keyed clear restored -- and the wrappers' post-exit ownership recheck: test_a_rehydrate_whose_slot_is_deleted_during_its_exit_work_returns_none and test_a_resume_whose_slot_is_deleted_during_its_exit_work_is_refused park the exit work's last worker hop (retry_owed_unstamp) with the slot published, retire it as a permanent deletion does, and assert the wrapper answers None / resume_session_deleted rather than the retired slot; both fail with the recheck removed): test_an_owed_marker_is_not_written_onto_a_new_incarnation_of_the_key (close defers → transcript deleted and the key recreated → the pending reopen's retraction drops the old dismissal; the new conversation carries no closed), test_an_owed_marker_lands_on_the_same_incarnation (the happy path still writes), test_an_owed_marker_without_an_identity_is_never_written, test_a_retried_owed_marker_also_checks_the_incarnation, test_the_dismissal_record_carries_the_closed_incarnation, test_a_resume_answered_by_another_keys_live_slot_does_not_settle_this_keys_dismissal (resume A by B's session key with B live → dedup to B → A's dismissal stays owed and lands when A's taker gives up; B untouched), test_a_resume_that_dedups_to_this_keys_own_live_slot_counts_as_published, test_a_plain_create_between_the_markers_guard_and_its_rename_is_unstamped (the write parked after its guard, before its rename; a same-name create lands; release → the marker is cleared again under the lock, superseded, the new tab's line open), test_an_owed_marker_with_no_publication_in_the_window_is_written, test_an_unstamp_that_fails_once_stays_owed_and_the_next_exit_clears_it (every clear of our stamp fails while the disk is "broken" → superseded still, un-stamp owed, the next exit clears it once the disk recovers), test_an_unstamp_never_removes_a_newer_close_of_the_created_tab (compare-and-clear; idempotent), test_the_after_commit_hook_does_no_io (the hook's only call is dismissal_still_owed, pinned by AST); test_a_handed_over_dismissal_settles_on_publish_or_on_the_last_retraction, test_both_delete_witnesses_map_the_one_evidence_verdict, test_close_reports_failure_for_a_retryable_archive_refusal, test_deletion_is_only_confirmed_on_definitive_evidence, test_teardown_drain_basis_follows_the_transcript_not_the_key, test_handover_drain_saves_the_tail_a_lost_truncation_rode_with, test_divergent_takeover_does_not_relabel_an_unrelated_archive_refusal, test_a_delete_won_refusal_beside_a_takeover_does_not_resurrect_the_transcript, test_handover_append_arm_does_not_resurrect_a_deleted_transcript, test_handover_append_creates_the_header_under_the_restricted_mode (incognito original, no committed header, pending rewrite, takeover: the file is born with memory_mode, transcript_withholds_derivation is true and derive_messages raises TranscriptWithheld), test_handover_append_tightens_a_persistent_replacement_line, test_handover_append_drops_notes_authorized_for_another_session (a note stamped for the slot's former session rides the window into a hand-over onto its channel transcript: not written, note_save_drop audited), test_a_declared_transcript_matches_every_spelling_of_its_file (slack_123.456 declared, slack:123.456 written: the watch is matched and the drain keeps its basis), test_a_failed_handover_to_an_unpublished_takeover_restores_the_original (end to end through the close handler with every append-safe write raising ENOSPC: 500, the original back under its key and dirty, the waiting resume returns it), test_cleanup_restores_the_original_when_its_handover_to_a_takeover_fails, test_handover_append_does_not_duplicate_rows_rotated_into_archives (window past the rotation cut: the replay lands only the owed tail, identities stay unique, existing indices do not move), test_handover_append_refuses_when_the_header_cannot_be_confirmed (the tightening merge defers once on an incognito hand-over with no header: no row written, the hand-over reports failure, the tail stays owed), test_a_cancelled_handover_keeps_its_append_and_its_watch (the drain cancelled while its append is parked: the basis is transferred, the watch survives the caller's close and retires at the append's settlement, the rows land).

Every parked test double in the ordering suite raises on an unreleased park, and the two tests that await a resume while a save or a close is parked observe the resume inside its ordering wait before releasing, so no test passes by waiting out a timeout.

test/test_dashboard_chat_rewind.py (+1): test_rewind_dispatch_verify_refusal_audits_as_a_refused_save — mints a same-name replacement during the sid flush (the last await before the verify) and asserts the SEL record reads history_save_refused with no worker dispatched. Fails on the pre-fix shape with history_save_exception.

test/test_chat_regenerate_cov80.py (+2): test_switch_variant_restores_the_row_when_the_save_is_refused, test_switch_variant_restores_the_row_before_an_app_is_answered_not_found.

Updated pins: test_chat_persistence_composition_contract.py (_SIGNATURES for the new keyword; the takeover_ordering owner), test_chat_persistence_parts_composition_contract.py (the two names added to slot_builders; the shutdown-save double accepts takeover_basis), test_history_bindings.py (signature digests for append / append_if_absent and the new _holds_identical_row member of history_append), test_chat_regenerate_cov80_edit_resend_boundary.py (the replaced-slot edit-resend restaging followed its test out of test_chat_regenerate_cov80.py), test_chat_handlers_composition_contract.py (the handover_drain owner surface: _persist_handover_tail gains takeover_basis, _teardown_drain_basis is added; and test_conductor_wake.py (save/drain doubles return a value and accept the basis), test_truncating_save_recreate_order.py (zero awaits between the fence read and the registration; the periodic save carries takeover_basis=None when called directly), plus the save_slot_off_loop doubles in test_slot_close_nudge_race.py, test_dashboard_chat_session_authority.py and test_resume_publishes_hydrated_slot.py.

Manual verification

N/A — unit coverage sufficient: the race is staged deterministically in unit tests with real ConversationLog files and there is no UI surface. Full local run on Linux/py3.12 of the 483 test files importing the touched modules, against a clean main worktree in the same environment: the failure-id sets are identical (main's local failures are all ModuleNotFoundError: snowballstemmer / websockets), with 26 more tests passing on this branch. black, check_comment_history, docs-lint, isort, flake8 and mypy src/kiro_crew/ pass locally.

Related Issues

Fixes #12090

Pattern harvest

Rule candidate: review-prompt
Pattern: a commit-boundary identity re-check guards a read-modify-write against actors that never take the writer's lock — any such check has an uncoverable check-to-commit gap and needs a two-sided arbitration token, not a later or duplicated check.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@dimwael
dimwael requested review from a team and bolichen97 September 20, 2026 11:18
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of eac18f2501bd93b071c87f32c4ce06d8944e1cd0 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item traces back to the reproduction test that fails on base. The size comes from the hand-over arm, which makes a lot of permanent surface but doesn't trigger a block.

Notes

  • Item 5 — oversized: the description's own cause is "publication and the persistence commit have no ordering contract". Yet 18 of the 47 new symbols (DeferredDismissal, OwedUnstamp, settle_*, begin/end_resume_read, record_deferred_dismissal, unstamp_superseded_marker, retry_owed_unstamp, close_metadata_under_lock) only exist to clean up after a close that "handed its key over while the taker was still unpublished". That arm creates the owed-marker, re-stamp and un-stamp races, and the persisted closed_id and its retry ledger exist to settle them.
    Subtraction: a close that loses the arbitration takes the existing retry arm (restore the original, tail owed) instead of handing over. DeferredDismissal, OwedUnstamp, closed_id, the only_if_closed_id= parameter and the ~100-line spec paragraph for them all go away.
    Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (9 items) — 8 justified

Intent: a tab reopened while its own edit is still saving never ends up with a mismatched (torn) history; either the edit applies or it is refused. FIX.

  1. A truncating edit racing a same-key reopen either commits (the reopen sees it) or is refused (the reopen sees the pre-edit transcript) — justified
  2. A resume waits for an in-flight truncating save; when the wait times out it fails with the new 503 takeover_ordering_unsettled — justified
  3. A reopen that starts after the close's tombstone notes nothing and is refused by closed — justified
  4. An explicit resume that began before a close wins it; the close's dismissal stays owed — justified
  5. Closes stamp a new persisted closed_id; a superseded marker is cleared by compare-and-clear, with parked retries — oversized
  6. A refused hand-over drain lands the unsaved tail through id-deduped appends (append_if_absent gains entry= and include_rotated=) — justified
  7. A request-controlled set of declared read targets is capped per watch; an overflow is refused as lock contention — justified
  8. The image-repair and reset-conversation handlers move to slot_repair.py (base slot_lifecycle.py is 1651 lines, over the line cap) — justified
  9. A switch-variant whose save is refused restores the row — justified

[FIRST-PRINCIPLES-REVIEWED] eac18f2

Description read

[DESCRIPTION-READ] 674bc38478c971c12e8f14fa9982f2788a128d85bbb2fb4ca8b9cf0c324f99e9

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of eac18f2501bd93b071c87f32c4ce06d8944e1cd0 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix targets the read-side race the base reproduction shows, every persisted change only adds something, and no website surface changes.

Notes

  • The resume path already waits on the loop, bounded, for pending truncating saves (await_takeover_ordering_settled), so the "publication never waits" premise only partly holds. A per-key loop-side serializer over truncating dispatch and resume read would pay the same wait without the note/retract/deferred-dismissal/owed-unstamp machinery. Weigh it before more arms accrete.
  • There are two new fail-closed refusals on core user paths: resume answers 503 takeover_ordering_unsettled, and close now reports history_save_failed on a falsy save or an unlanded marker. Check that the frontend handles both as retryable rather than as dead tabs.
  • Take-away check: the flush_slot_now and clear_closed changes only add parameters, and closed_id is an optional new field, so existing readers are unaffected. The Backwards compatibility section sits past the description truncation and was not judged.

[DESIGN-REVIEWED] eac18f2

Description read

[DESCRIPTION-READ] d007663ab559712b289db53822f28f8d5a341607d1beef97236ff310dc58b87d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed eac18f2501bd93b071c87f32c4ce06d8944e1cd0 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] eac18f2

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings (all downgraded on adjudication)

Reviewed eac18f2501bd93b071c87f32c4ce06d8944e1cd0 via the fork AI-review pipeline; updated in place on each push.

Opus 5.5 adjudication downgraded every blocking finding. Downgraded: the remedy each one needs is disproportionate to the harm it prevents. Adjudication downgraded all 1 blocking finding(s): the remedy each one requires is disproportionate to the harm it prevents.

Review details

BLOCKING -- src/kiro_crew/dashboard/slot_persistence/takeover_ordering.py:94 -- Post-close deliveries bypass an outstanding deferred dismissal

return None
Trigger: A close hands its dismissal to an unpublished member-thread reopen (api_member_thread → rehydrate_slot_from_history_async(adopt_closed=True)). A subsequent cron send_message(session="origin") reaches this branch through the default rehydrator: it skips noting, the deferred holder is non-truncating, and the transcript still reads open. slot_closed_since passes because this delivery began after the close.
Outcome: The cron delivery republishes the dismissed thread and admits work into it. If the deliberate reopen then fails or cancels, its retraction finds the delivery occupying the key and cannot write the owed dismissal; the successful close is overridden by an unsolicited delivery.
Anchor: defect
Fix: Refuse non-adopting rehydrates while a post-close deferred dismissal remains outstanding, rather than reading its temporarily open transcript.

[BLOCK-MERGE-DOWNGRADED] eac18f2
[GPT-REVIEWED] eac18f2

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 is a narrow race with a visible outcome the user can undo, so I'm downgrading it.

F1 — LOW harm. The failure needs four things at once. First, a close that defers its dismissal to an unpublished adopt_closed reopen. Second, a cron delivery that starts during that reopen's pending read; it skips the note because of the not adopt_closed branch at takeover_ordering.py:88-94 (new file, read from the patch) and is admitted through slot_builders.py:775. Third, the cron delivery publishes, which settles the dismissal at slot_builders.py:668. Fourth, the deliberate reopen then fails or is cancelled, and its retraction at slot_builders.py:670-680 finds nothing owed. The result is that the tab the user asked to reopen stays open, holding a cron message that was really sent to that session. If the reopen succeeds, the tab would have ended up open anyway. Nothing is lost or silently corrupted, and the user can close the tab again through the normal close path.
Cost of the real fix: one more coupling between owed dismissals and rehydrate admission inside an already large ordering mechanism. It also changes behaviour, because refusing the delivery could drop or delay a real cron message. That cost is higher than a visible, recoverable extra open tab.

[ADJUDICATION] eac18f2501bd93b071c87f32c4ce06d8944e1cd0 total=1 uphold=0 downgrade=1
DOWNGRADE F1 src/kiro_crew/dashboard/slot_persistence/takeover_ordering.py:94 reason=disproportionate-remedy
[GPT-ADJUDICATED] eac18f2501bd93b071c87f32c4ce06d8944e1cd0

@dimwael
dimwael force-pushed the fix/rewrite-commit-slot-incarnation branch from 838275c to 1eefe98 Compare September 20, 2026 11:49
@dimwael

dimwael commented Sep 20, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the GPT 5.6 blocking finding (takeover before claim registration unordered) in 1eefe98b.

The gap was real: a claim registered on the worker cannot be marked by a takeover that fires while the save is still in dispatch, and the unguarded pending-rewrite flush retry has no map recheck to fall back on. The fix follows the finding's suggestion — a per-key takeover generation:

  • note_same_key_takeover bumps the key's monotonic generation in the same mutex-guarded transition that marks live claims.
  • Callers read the generation at their authorization instant, while they can still vouch their slot is the live occupant: save_slot_off_loop entry (all loop-side callers), the periodic flush pass's slot-table snapshot in _flush_dirty_slots (threaded through flush_slot_now as takeover_basis), and the shutdown pass's per-slot turn.
  • begin_truncation_claim compares the basis under the claim mutex; a moved generation leaves the claim born overtaken, and the commit gate refuses exactly as for a mid-write takeover.

New tests: test_takeover_during_dispatch_makes_an_unguarded_truncating_save_refuse stages the finding's exact arc (flush-shaped save with no expected_slot_name, parked at claim registration after its basis was read; a full resume completes in the gap; the save must refuse and the resume must read the untruncated transcript) and fails without the generation; test_a_stale_basis_leaves_the_claim_born_overtaken pins the basis semantics in both directions.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@dimwael
dimwael force-pushed the fix/rewrite-commit-slot-incarnation branch from 1eefe98 to f52fee3 Compare September 20, 2026 12:28
@dimwael
dimwael requested a review from a team September 20, 2026 12:28
@dimwael

dimwael commented Sep 20, 2026

Copy link
Copy Markdown
Contributor Author

Both GPT 5.6 blocking findings on 1eefe98b addressed in f52fee37:

Finding 1 (post-takeover basis authorizes a stale slot snapshot): correct — a basis read at the stale slot's turn is captured after the takeover and observes nothing. Fixed as prescribed: _flush_dirty_slots re-reads the live occupant after capturing the basis and skips the save when the map has stopped backing the snapshot entry (a takeover before the capture changes the occupant, one after it moves the generation, so no arrival instant goes unobserved). save_all_slots_to_history gets the same capture-then-verify, and additionally re-fetches the occupant per key so shutdown persists the replacement's state rather than neither. Pinned by test_flush_pass_skips_a_slot_whose_key_changed_owner_mid_pass.

Finding 2 (claim refusal occurs after archive mutation): correct — a refusal after _archive_dropped_lines records still-live rows as dropped. The commit gate now sits ahead of the rewrite's first mutation (before the archive, still under the per-session lock through archival and replace), so a refusal leaves no trace. Both mid-write race tests now assert the refusal wrote no dropped-line archive; the park seam moved to _frozen_prefix_and_foreign_appends, which still lies between the map recheck and the gate.

@dimwael
dimwael force-pushed the fix/rewrite-commit-slot-incarnation branch from f52fee3 to 3116f7f Compare September 20, 2026 13:28
@dimwael

dimwael commented Sep 20, 2026

Copy link
Copy Markdown
Contributor Author

All three GPT 5.6 blocking findings on 1eefe98b/f52fee37 addressed in 3116f7fc:

Post-takeover basis authorizes a stale slot snapshot: fixed as prescribed. The flush pass re-reads the live occupant after opening its watch and skips a snapshot entry whose key changed owner (the close flow that popped it owns its tail); the shutdown pass re-fetches the occupant per key and applies the same capture-then-verify, so it persists the replacement's state rather than a stale entry's. The same dispatch-time occupancy verify now guards save_slot_off_loop callers that pin expected_slot_name — wired into rewind and edit-resend, whose awaits between authorization and dispatch are exactly where a takeover completes with only its pop left to observe.

Claim refusal occurs after archive mutation: fixed as prescribed. The commit gate moved ahead of _archive_dropped_lines (the rewrite's first mutation), with the per-session lock held through archival and replace; both mid-write race tests now assert a refusal writes no dropped-line archive.

Takeover generations retain every slot key forever: fixed as prescribed — the per-key counter is gone. Dispatchers open a refcounted takeover watch at their last synchronous instant and close it once the save's outcome is in hand; a takeover note bumps only an open watch (no save in flight means nobody is left to observe it), and the last holder out retires the entry, so the table is bounded by in-flight saves. test_takeovers_of_unwatched_keys_leave_no_record pins the bound, and every race test asserts the watch table drains.

Note on FollowController.ts: that finding was against an accidental state in f52fee37 — a squash against a moved base briefly reverted two unrelated upstream commits (#12272, #12222). 3116f7fc is rebased cleanly; the diff touches only the dashboard persistence/registry files and their tests, and website/ is untouched.

@dimwael
dimwael force-pushed the fix/rewrite-commit-slot-incarnation branch from 3116f7f to 157dc65 Compare September 20, 2026 14:41
@dimwael

dimwael commented Sep 20, 2026

Copy link
Copy Markdown
Contributor Author

All three GPT 5.6 blocking findings on 3116f7fc addressed in 157dc653:

Archival callers treat takeover refusal as success: fixed as prescribed. The close and cleanup archive saves now check the result; a refusal whose teardown watch moved (SlotRegistry.basis_moved — the holder's post-hoc discriminator) routes to the existing hand-over arms (drain the original's tail under the same watch, hand over key-scoped markers, report history_save_failed / the failed column), while a delete-won refusal stays ignored, since the delete already disposed of the archive's subject.

Cancellation retires the watch before its worker finishes: fixed. The worker's _do closes its own basis when the save settles (close is idempotent per basis), and save_slot_off_loop shields the executor future — the asyncio wrapper reports cancelled() even over a running worker, so shielding is what actually keeps both the watch and the authorized durable write alive through a caller's cancellation. Pinned by test_cancelled_dispatch_keeps_the_watch_open_for_its_running_worker.

Takeovers before archival dispatch leave no ordering record: fixed as prescribed. Close and cleanup open one takeover watch before their pop and carry it through every teardown await and every save (archive, hand-over drains), so a resume beginning during the task-cancel wait — publication still pending behind its transcript read — is observed and the rewrite-shaped archive is born overtaken. Pinned end-to-end through the real close handler by test_close_archive_loses_to_a_resume_that_begins_mid_teardown.

Also note for the earlier shard red: Backend Tests (3.12, 6) on 3116f7fc failed on an xdist worker crash in test_argv_floor_inline_and_brace_scope.py (unrelated to this diff; the file passes locally on this branch). The new push re-runs it.

@dimwael
dimwael force-pushed the fix/rewrite-commit-slot-incarnation branch from 157dc65 to a4f1011 Compare September 20, 2026 15:22
@dimwael

dimwael commented Sep 20, 2026

Copy link
Copy Markdown
Contributor Author

GPT 5.6 blocking finding on 157dc653 (retryable save refusals misclassified as deletion) addressed in a4f10116, as prescribed: the close and cleanup archive arms now verify the deletion positively (session_was_deleted, off-loop) before ignoring a refusal. A refusal that is neither a takeover (basis_moved) nor a verified deletion — the stale-queue guard's, for one — restores the tab when the key is still free so the periodic flush retries the write (with the nudge-loop and app-dismissal rollbacks the exception arm already performs), otherwise drains the tail, and reports history_save_failed / the failed column instead of 200. Pinned by test_close_reports_failure_for_a_retryable_archive_refusal.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@dimwael

dimwael commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

a04f41c7f — @bolichen97, the GPT finding on the 57e7bdda9 roll, two lines plus tests: the resume wrapper took "no refusal" as "published", but _live_slot_for_resume dedups by session identity across slots, so POST /slots/A/resume {"key": "dashboard:B"} with B live is answered by B — nothing published under A — and the wrapper settled A's owed dismissal and released its holder; A's taker then retracted into nothing and A came back at restart. "Published" is now an occupancy test: the slot the resume answered with must hold the key the wrapper noted (state._slots.get(noted_key) is outcome.slot); otherwise the exit takes the owed-dismissal path as for a refusal. The rehydrate wrapper needs no such test — its body only ever returns state._slots[slot_name].

Tests (test_slot_close_recreation_race_takeover_dismissal.py): test_a_resume_answered_by_another_keys_live_slot_does_not_settle_this_keys_dismissal (A owed, B live, resume A by B's key → dedup to B → A's dismissal still owed → A's taker gives up → A's closed lands, B open and untouched; with the old test restored it fails with A's watch gone) and test_a_resume_that_dedups_to_this_keys_own_live_slot_counts_as_published (same key, already live → settles). slot_lifecycle.py, slot_registry.py and takeover_ordering.py are byte-identical to the previous head. 38 files, 1 commit, rebased on current main; 1,513 passed across the touched suites, both ratchets green; mypy, lint, cap/black/comment-history/docs gates and semgrep clean.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental review of a04f41c against the previously reviewed 57e7bdd.

The new head is a rebase onto current main plus one change on top of what was already judged ready: resume_slot_from_history now counts a resume as published only when the key it noted actually holds the slot it answered with (state._slots.get(noted_key) is outcome.slot). Before this, a resume answered by a live slot under a different key (the session identity dedup in _live_slot_for_resume) settled this key's owed dismissal even though nothing was published here, so the taker later retracted into nothing and the closed tab could come back on restart. The new condition routes that case through the retraction path, and the incarnation witness added in the previous round keeps the deferred marker off the other key's live transcript.

Both new tests pin the right edges: a dedup to another key leaves the dismissal owed and the marker lands only on A while B stays open; a dedup to this key's own live slot still settles. The range diff shows no other change to the PR's own patch beyond the rebase.

No new findings. The held fork CI runs on this head are now approved. Approval waits on the readiness check finishing green on this head.

@dimwael

dimwael commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Status on a04f41c7f: the Fast Gate is red on Harness Parity Gate (x-capture rule) and skipped every heavy job, so PR Readiness can't evaluate. That gate landed on main today in #22102 and is currently failing on main itself (Fast Gate on 6257f698f and 602ba3c1d → Harness Parity Gate failure). This PR touches none of the agent_sdk/backends* files the gate scans (38 files, none under agent_sdk/).

Not pushing — nothing in the diff can change this; it needs the gate fixed or relaxed on main, after which a re-run here should go green. @bolichen97 FYI since you said approval waits on readiness.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked on head c866641. This head is a clean merge of main 3e39e7a into the branch. The PR's own patch is byte-identical to a04f41c (same 38 files, same stable patch id), so my earlier read still holds: good to ship on the incremental change.

Not approving yet. What is left:

  • The fork CI runs for this head (20 workflows, including Fast Gate, CI and the AI review lanes) are held in action_required and need maintainer approval before they start.
  • The readiness label is readiness: action required and the PR Readiness check is red on this head, so readiness has not passed.

Once the runs are approved, go green and readiness reads passed on this head, I will approve and enable auto-merge. No new push is needed from you.

@dimwael

dimwael commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

24272996b — @bolichen97, the GPT finding on c866641cb1: the publish-side twin of the resume-side gap closed two heads ago. The owed marker's guard decides under the transcript lock, but a plain same-name create (POST /api/chat/slots → get_or_create_slot → put_slot) takes no lock and registers no read, so it could settle the dismissal between the guard's "still owed" and the rename, and the rename then stamped closed onto the line of a tab that was live — hidden at the next restart.

Fix. The record stays the witness through the write: update_metadata_if's after_commit_under_lock hook re-reads the ownership after the rename, still under the transcript lock (a brief registry-mutex read, the same order the guard already uses), and if a publication has settled the dismissal meanwhile it un-stamps the marker on the same line under the same lock (clear_closed; the lock is reentrant for the key) and the outcome is superseded. No lock is taken inside put_slot; the registry mutex still never waits on I/O. The create path writes no metadata of its own, so the un-stamp clobbers nothing.

Tests (test_slot_close_recreation_race_takeover_dismissal.py): test_a_plain_create_between_the_markers_guard_and_its_rename_is_unstamped — the writer parked after its guard, before its rename; a same-name create lands; release → superseded, no closed on the new tab's line, watch retired (with the re-check removed it fails with written over the live tab); test_an_owed_marker_with_no_publication_in_the_window_is_written (the unopposed write is unchanged).

Push shape. Your merge commit c866641cb1 carried our patch unchanged (I verified the 38 files byte-identical across it). The repo's review gate caps a PR at two commits and asks for a rebase rather than a merge, so a third commit on top of the merge would have failed it; I rebased our single commit plus this fix onto current main (33ad75e01, which includes everything your merge brought in) and pushed with the lease pinned to c866641cb1 — same one-commit shape as every previous push, your merge folded into the rebase. 38 files; 1,515 passed across the touched suites, both ratchets green; mypy, lint, cap/black/comment-history/docs gates and semgrep clean.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental review of 2427299 against the previously reviewed c866641.

This head is more than a merge of main. The branch is rebased onto current main as one commit, and on top of the patch already reviewed it adds a post-commit re-check in record_deferred_dismissal: a same-name create takes no transcript lock, so its put_slot could land between the guard's "still owed" and the rename and the rename would then stamp closed onto the line of a live tab. The new _recheck_after_commit reads ownership again after the rename, still under the transcript lock, and clears the stamp when a publication has won, reporting superseded. The spec paragraph in history.md is updated to match, and two tests pin both edges (create inside the window is un-stamped; no create leaves written with the marker on disk).

The logic is sound. ConversationLog._locked is reentrant (per-key RLock plus held-flock reuse), so calling clear_closed from inside the update_metadata_if callback does not deadlock, and any create that lands after the re-check is ordered after a committed marker, which is the case the existing reopen path already handles.

One 🟡 inline below: the callback now does file I/O, which update_metadata_if documents as forbidden for that hook, and a failure there leaves the stamp on a live tab with the dismissal already settled.

Not approving yet: the fork CI runs on this head are still held, so readiness is not green on this commit.

if SlotRegistry.dismissal_still_owed(state, dismissal):
return
superseded_after_commit[0] = True
log.clear_closed(dismissal.transcript)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The un-stamp runs file I/O inside a hook documented as I/O-free, and its failure leaves a live tab stamped closed.

update_metadata_if says its after_commit_under_lock callback "must not perform I/O or mutate event-loop-affine state". _recheck_after_commit now calls log.clear_closed(...), which reads the file and does an atomic_write. It works because the lock is reentrant, but the contract no longer describes its callers. The failure path is the real gap: if clear_closed raises (disk full, permission, a transient read), the exception escapes update_metadata_if, the except returns "failed", and the next exit's retry sees dismissal_still_owed False (the publication already settled it), so nothing in this path clears the stamp. Unless a later save happens to rewrite that line, the newly created tab's line keeps closed: True and is hidden at the next restart, which is the exact outcome this change prevents.

    def _recheck_after_commit() -> None:
        if SlotRegistry.dismissal_still_owed(state, dismissal):
            return
        superseded_after_commit[0] = True
        log.clear_closed(dismissal.transcript)

Suggestion: either widen the update_metadata_if docstring to allow a same-key metadata rewrite in the hook, or set only the flag in the callback and do the un-stamp right after update_metadata_if returns (a compare-and-clear under its own lock, for example clear_closed(..., only_if_closed_before=...) keyed on dismissal.closed_at), logging and retrying on failure rather than reporting failed for a dismissal the registry has already settled. A test that makes clear_closed raise once would pin it.

Rule: a lock-held callback must honor the contract its host documents, and a recovery write must not fail into a state no retry revisits.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 96a23e120, your second option. The after_commit_under_lock callback now only records the verdict (_recheck_after_commit sets the flag; test_the_after_commit_hook_does_no_io pins by AST that its only call is dismissal_still_owed), and the un-stamp runs right after update_metadata_if returns, as a compare-and-clear under the transcript's own lock: clear_closed(..., only_if_closed_before=math.nextafter(our_closed_at, inf)), so exactly the stamp we wrote comes off and a later legitimate close of the created tab is never removed (test_an_unstamp_never_removes_a_newer_close_of_the_created_tab).

The failure path you traced is closed: the dismissal is settled on the registry regardless, so the outcome is superseded either way — never failed — and a clear that does not land is parked as an owed un-stamp on the key's watch (OwedUnstamp, SlotRegistry.park_owed_unstamp; it holds the watch open) and retried by every noter exit on the key (retry_owed_unstamp, called from both the resume and rehydrate wrappers) until it lands or the line provably carries a different stamp. Test: test_an_unstamp_that_fails_once_stays_owed_and_the_next_exit_clears_it — every clear of our stamp raises while the disk is "broken" (the superseded write's own attempt and its exit's immediate retry both fail), the stamp stays on and the clear stays owed, the disk recovers, the next exit on the key clears it. With the parking removed it fails on "the clear must stay owed".

@dimwael

dimwael commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

96a23e120 — @bolichen97, your inline on 24272996b, fixed as you suggested (details on the thread): the after_commit_under_lock hook is flag-only again, the un-stamp is a compare-and-clear keyed on our own closed_at under its own lock, and a clear that fails is parked as an owed un-stamp on the key's watch and retried by every exit on the key — a settled dismissal is never reported as a failed close. Three tests added (…fails_once_stays_owed…, …never_removes_a_newer_close…, test_the_after_commit_hook_does_no_io); the two push-28 tests are unchanged and green. slot_lifecycle.py is byte-identical. 38 files, 1 commit, rebased on current main; 1,518 passed across the touched suites, both ratchets green; mypy, lint, cap/black/comment-history/docs gates and semgrep clean.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental review of 96a23e1 against the previously reviewed 2427299 (range diff of the PR's own commit across the rebase onto 512f13c).

The 🟡 from my last review is fixed. Outside the rebase, the only change is that one finding:

  • _recheck_after_commit now only records the verdict. Its sole call is SlotRegistry.dismissal_still_owed, so the update_metadata_if hook is I/O-free again, and test_the_after_commit_hook_does_no_io pins that with an AST check.
  • The un-stamp runs after update_metadata_if returns, in unstamp_superseded_marker. It is a compare-and-clear: clear_closed(..., only_if_closed_before=nextafter(closed_at)) under the transcript's own lock, so a later legitimate close of the new tab is never removed (test_an_unstamp_never_removes_a_newer_close_of_the_created_tab).
  • A failed clear no longer reports failed for a settled dismissal. It is parked as an OwedUnstamp on the key's watch and retried by the next resume or rehydrate exit (test_an_unstamp_that_fails_once_stays_owed_and_the_next_exit_clears_it). The parked record takes a plain holder, not a truncating one, so takeover_ordering_pending does not make readers wait on it.
  • The history.md paragraph is updated to match.

No new findings on this head. A small residual, not blocking: the owed un-stamp lives in memory, so if the clear fails and the gateway restarts before another exit on that key, the stamp stays. That needs a write failure in a narrow window, and it is the same durability every other in-memory owed record here has.

Not approving yet: readiness on this head is action required. Fork CI is still held for approval, most required checks are pending, and Sensitive Change Review and Dependency License Gate report failures. I will look again once readiness is green on this commit.

@dimwael

dimwael commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Status on 96a23e120 (while CI finishes): the three reds that appeared on this head are all infrastructure, none from the diff —

  • Dependency License Gate: Dependency review is not supported on this repository. Please ensure that Dependency graph is enabled — a repo setting (settings/security_analysis), and this PR touches no dependency manifest. It never ran on any of the previous 29 heads of this PR; it passed on an in-repo PR a minute earlier, so it may be fork-token-specific. Not something a push can change.
  • Sensitive Change Review: Could not read the PR from the API (CalledProcessError); re-run this job — transient; it is in the readiness required set, so it needs a re-run (maintainer).
  • Inclusive Language: The operation was canceled (superseded run); should settle on its own or with a re-run.

Not pushing — the diff is final. @bolichen97 when you look again: Sensitive Change Review needs the re-run click; Dependency License Gate needs either the repo's dependency graph enabled or the license-override label (no dependencies change in this PR).

@dimwael

dimwael commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Status on 96a23e120: GPT 6.1 ✅ and Opus 5.5 ✅ (GPT's one remaining item was downgraded to advisory by adjudication; Opus green for the 15th consecutive head). 74 checks pass, MERGEABLE.

What stands between this head and readiness: passed, none of it from the diff:

  • Backend Tests (3.12, 5): one failure, test_tool_call_classification_provenance_bound.py::…::test_a_payload_the_encoder_could_not_render_is_refused_not_retained_unmeasured (ExceptionGroup of unraisable warnings) — not in this PR's 38 files; shard passed 29,485 others. Re-run should clear it; Coverage Gate follows.
  • Sensitive Change Review: "Could not read the PR from the API … re-run this job" — re-run.
  • Dependency License Gate: dependency graph not enabled on the repo (never ran on prior heads; no manifest changes here) — license-override label or the repo setting.
  • Inclusive Language: cancelled superseded run — settles on re-run.

@bolichen97 that's the set; no push needed from my side.

@dimwael

dimwael commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Update on 96a23e120 after the re-runs — thank you: shard 5, Coverage Gate and Sensitive Change Review are green; 78 checks pass, GPT 6.1 ✅, Opus 5.5 ✅, MERGEABLE. PR Readiness now reports one blocking item. The two reds left are Dependency License Gate (dependency graph not enabled on the repo; no manifest in this diff) and Inclusive Language (still the cancelled superseded run from 23:03Z, never re-run). Whichever of the two readiness counts, both need a maintainer click (re-run / license-override or the repo setting), not a push.

@bolichen97

Copy link
Copy Markdown
Contributor

This branch conflicts with main, which has moved a lot in the dashboard persistence and chat API files this PR touches. Could you rebase onto current main when you have time? The fix for the linked issue is still wanted, and the Dependency License Gate red on this head should get a fresh run after the rebase.

@dimwael

dimwael commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97 — rebased as you asked. Head is now 50b88d331e, one commit on current main (484d7378a8), 42 files (38 before the rebase; the four new ones are all from the split below). The eleven conflicts were adjacency unions: save_transaction.py keeps your retired_rows / swap-won guard inside our claim scope's with (...) header, slot_lifecycle.py takes main's file-change flush, notice token, warm-brief and remote-dispatch hunks inside our try/finally teardown blocks, and the rest are import and pin unions. One non-mechanical line: main's new rebind_manager parameter landed on resume_slot_from_history, which this PR had turned into a noting wrapper, so the wrapper now forwards it into _resume_slot_from_history_noted (same default).

The rebase pushed chat_api/slot_lifecycle.py over the 2,000-line cap: main alone is now 1,651 and this PR adds 363 (the teardown watch and the hand-over arms), for 2,014. Per the responsibility-split rule I moved the two endpoint handlers that do not touch the close path, api_chat_slot_repair_images and api_chat_slot_reset_conversation, into a new owner chat_api/slot_repair.py, bodies byte-identical to main's, with only the TYPE_CHECKING names they read. slot_lifecycle.py is 1,592 lines; chat_handlers.py composes the new owner and imports the two names from it (no re-export left on slot_lifecycle); the slot_repair owner row and _BASE_SURFACE row in test_chat_handlers_composition_contract.py, the path pins in test_json_object_body_guard.py and test_slot_running_readers.py, and the owner list in learn-cron-dashboard.md follow it. If you'd rather see a different cut, say which and I'll redo it.

Verified on 50b88d331e: 1,631 passed across the touched race/ordering/contract/repair suites plus the suites covering the merged upstream regions; both ratchets green; mypy, black 26.10.1, isort, flake8, the cap, comment-history, brand, subprocess-encoding, harness-parity and docs gates clean; semgrep test-determinism clean on added lines. Could you approve a fresh run of the Dependency License Gate and Inclusive Language lanes on this head? They need a maintainer click on a fork branch.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the rebase. Full review at 50b88d3.

One thing blocks the merge, and it is in the description, not the code. PR Hygiene fails with: "This diff takes something away: list each reader as 'Reader: : -- -- ' under '## Backwards compatibility'." PR Readiness is red with it. Could you add that list? For example, api_chat_slot_repair_images and api_chat_slot_reset_conversation moved out of chat_api/slot_lifecycle.py, and save_slot_off_loop now refuses some saves it used to commit.

Code review: no blocking findings. Coverage of the 42 changed files:

  • All 19 src/ files and both docs files were read hunk by hunk.
  • The code moved from chat_api/slot_lifecycle.py into the new chat_api/slot_repair.py was checked mechanically: every line of slot_repair.py except 17 header and import lines comes from the removed block.
  • The 17 edited test files were read in full. The 4 new test files (about 4,850 lines) were read line by line in a second pass.

Non-blocking notes:

  • history_append.append with a prebuilt entry keeps the row's own ts and skips monotonic_transcript_ts, so a fallback append can land rows after newer ones with older timestamps. Worth a line if any reader orders by ts.
  • An owed closed marker whose write fails keeps its watch open until the next resume or rehydrate on that key. If none comes, that watch stays, one per key at most.
  • The new-incarnation check compares created_at (an ISO timestamp). A coarse clock could give a delete-and-recreate the same value, and _recreate_the_key_under_a_new_incarnation in the dismissal tests asserts they differ.
  • A few tests check less than their names say: test_a_publication_that_noted_nothing_settles_the_dismissal_at_once never publishes in its second half, and test_cleanup_restores_the_original_when_its_handover_to_a_takeover_fails does not check _takeover_watches == {} the way its single-tab sibling does. test_slot_close_recreation_race_takeover_retraction.py also has an assert chat_persistence is not None that checks nothing.

CI at this head: apart from PR Hygiene and PR Readiness, the remaining reds are cancelled duplicate runs, and E2E, Integration and the desktop builds have passed. Backend test shards were still running. Once the description is updated and CI is green, I expect to approve.

@dimwael

dimwael commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

@bolichen97 thanks for the full read. The description was updated at 15:19Z with five Reader: lines under ## Backwards compatibility (routes/chat.py ×2, dashboard/chat.py, chat_handlers.py, and the learn-cron-dashboard owner list); the re-run of PR Hygiene on 50b88d3 is green now that the fork workflows are approved. The save_slot_off_loop refusal is described in the Compatible paragraph and the two declared failure modes (503 takeover_ordering_unsettled on resume, 500 history_save_failed on a refused close archive).

On the four non-blocking notes: all fair, and the three test ones are cheap (the dead assert chat_persistence is not None, the second half of test_a_publication_that_noted_nothing_settles_the_dismissal_at_once, the missing _takeover_watches == {} check in the cleanup sibling). The ts note is right and has a real reader: context_assembly/replay.py sorts rows by transcript_sort_key(row["ts"]), so a fallback-appended row with an older ts would replay out of file order there. Correction over my first draft of this comment, which said no reader orders by ts. The fix would be to run the prebuilt entry through monotonic_transcript_ts against the previous row in history_append.append, plus a test. I'd rather not re-push this head while GPT/Design/FP are green on it; say the word and I'll either fold the four items in here as one more commit-squash, or open a small follow-up PR after this merges. Your call.

@bolichen97

Copy link
Copy Markdown
Contributor

main had a break in the conductor tests (test_mate_conductor_agent.py, test_agent_refactor_spec_bytes.py), fixed by #28301 (1dd2be3). A re-run of a failed job reuses the old merge commit and will not pick that fix up. Please update this branch onto main. The PR Hygiene point from my earlier review (the Reader: list under ## Backwards compatibility) still stands. I'll review the new head on its own CI.

@dimwael

dimwael commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

@bolichen97 — updated onto main as asked: head f94c1d86b1, one commit on b55ccea475, which includes #28301 (1dd2be3), so the new CI runs against the conductor fix.

Since the head had to move anyway, I folded in the four non-blocking notes from your review rather than re-pushing a bare rebase:

  • history_append.append with a prebuilt entry now stamps the row by the same rule as a clock-stamped append — monotonic_transcript_ts against the last row on disk, so the row's own ts is kept when it already sorts after that row and moved one microsecond past it when it does not. test_append_safe_fallback_never_lands_a_row_that_sorts_before_the_one_it_follows (ordering suite) fails on the previous head; one sentence in history.md.
  • The dead assert chat_persistence is not None is gone (with the import it kept alive).
  • test_a_publication_that_noted_nothing_settles_the_dismissal_at_once now publishes in its second half (get_or_create_slot → put_slot settles at the insert; the close's late write then yields).
  • test_cleanup_restores_the_original_when_its_handover_to_a_takeover_fails checks _takeover_watches == {} and that nothing reached the transcript, as its single-tab sibling does.

Also the two doc owner-name fixes for the slot_repair.py move (rfc-transcript-section-markers.md, feature-map/README.md), and your coarse-clock point: the dismissal tests no longer rely on the clock separating two created_at stamps (the recreate's stamp is set a second past the old one). No production change there — a same-tick delete-and-recreate would read as one incarnation, a one-tick exposure whose failure mode is the pre-PR one (the marker lands on the recreate).

The Reader: list has been under ## Backwards compatibility since 15:19Z and PR Hygiene passed on it at 16:08Z; it is unchanged in the new body. 44 files, 1 commit; 1,078 passed across the race/ordering/contract/repair suites on this head (the full recipe ran on the pre-rebase sha: ~4,560 passed, both ratchets, mypy, cap, black 26.10.1, lint, comment-history, semgrep and docs gates clean). Could you approve the fork workflows on the new head?

@dimwael

dimwael commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 44bca0ab10: the Backend shard-2 red was ours — the new dismissal test patched metadata_now_iso on history_append (the owner); the facade-mirror guard requires the patch on the history facade. One test line, nothing else changed; rebased on current main (865d8d2b74). The E2E Gateway Boot (Windows) failures were a runner sign-in problem, not this branch. @bolichen97 could you approve the fork workflows on the new head?

@dimwael

dimwael commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

b222c192a3 — @bolichen97, the GPT finding on 44bca0a fixed: rehydrate_slot_from_history_async returned from inside its try and then, in its finally, awaited the owed-dismissal record and the un-stamp retry on worker threads, so a permanent deletion landing in that window could retire the slot it had already published and the wrapper still handed it back (the outbox's file-card delivery rehydrates on a miss and appends to what it gets). The wrapper now returns after that exit work and re-reads the key: if the slot it published is not the one under its key any more it answers None; nothing is retracted, since the publication happened and settled what the key owed. resume_slot_from_history has the same shape and gets the same recheck, answering resume_session_deleted (409) for its hydrate arm only — the dedup arm's slot predates the resume and may live under another key.

Tests (test_slot_close_recreation_race_takeover_dismissal.py, now 14): test_a_rehydrate_whose_slot_is_deleted_during_its_exit_work_returns_none and test_a_resume_whose_slot_is_deleted_during_its_exit_work_is_refused park retry_owed_unstamp with the slot published, pop it and delete the transcript as a permanent deletion does, release, and assert the wrapper's answer; both fail with the recheck removed (the retired slot comes back). One clause in history.md. Everything else is unchanged; rebased on current main (2519d06919). Backend shard 2, Opus and Security Scope were green on 44bca0a. Could you approve the fork workflows on the new head?

@bolichen97

Copy link
Copy Markdown
Contributor

E2E Gateway Boot (windows-latest) was red here because of a main break that hit every fork run: KAS took the bundle route with "kiro-cli is not signed in". That is fixed by #28792 (ed12dcd). A re-run reuses the old merge commit, so please update this branch onto main. I'll review the new head on its own CI.

@dimwael

dimwael commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main 35cb020bb9 (includes #28792); no code changes vs b222c19. @bolichen97 could you approve the fork workflows on the new head, c19d15df3d?

@dimwael

dimwael commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main f45380e92a across #28994 (import-order union in history_append.py, no code changes vs c19d15d). The reds on c19d15d were a GitHub installation rate limit at ~10:05Z across Sensitive Change / License Gate / GPT / Opus / Content Scan / Security Scope, not findings. @bolichen97 could you approve the fork workflows on the new head, 3642a4c721?

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes at 3642a4c for one finding. This head is a rebase of c19d15d: the range-diff shows only the import logging line from main in history_append.py, and I read every other hunk in earlier rounds (44 files).

Blocking (the GPT lane's finding, which I checked in the code): a tied close time lets the unstamp clear a newer close.
unstamp_superseded_marker (takeover_ordering.py:497) calls clear_closed(record.transcript, only_if_closed_before=math.nextafter(record.closed_at, math.inf)). clear_closed keeps the stamp only when close_time >= only_if_closed_before (history_projection.py:615), so any stamp with closed_at equal to ours is cleared. The close time comes from time.time() (channel_slots.py:904). On Windows that clock can return the same value for two closes a few ms apart. So: a publication supersedes our deferred dismissal, the user then closes the new tab, and the unstamp runs after that, on the same tick. The new tab's legitimate closed marker is removed, and with Restore Sessions on, the conversation the user closed comes back after a restart.

A fix that keeps closed_at for activity comparisons: write a per-key close identity that is strictly increasing (or unique, like a nonce) next to closed_at, and have the unstamp clear only when the stored identity equals the one we wrote, checked under the transcript lock. A test that stamps two closes with the same closed_at and asserts the newer one survives the unstamp would pin it.

Other CI: 79 checks green. E2E (stub) and the Linux backend shards are still running.

A truncating history rewrite (rewind, edit-resend, regenerate,
switch-variant) commits on a worker thread while a same-name
close-and-recreate pops and republishes the slot on the event loop
with no lock between the layers. The recreate-won map recheck at the
commit boundary only sees replacements already published, and the
replacement's own transcript read blocks behind the per-session lock
until the save releases -- while a reopen whose read came FIRST, with
the save still queued for the lock, publishes the pre-edit window over
the truncated file: the tab shows turns the edit removed, the next
turn's save writes them back, and the edit is silently undone.

Introduce a publication-ordering claim (TruncationClaim): every
truncating save registers one for the whole write, every same-key
takeover marks it (SlotRegistry.put_slot and the start of a resume's
transcript read), and a single mutex-guarded transition decides the
winner before the rewrite's first mutation -- ahead of the
dropped-line archive the payload composer performs, so a refusal
leaves no sidecar recording still-live rows as dropped. A takeover
that begins first makes the save refuse with nothing written; a save
that commits first leaves the claim committed, and a later takeover
knowingly resumes the post-rewrite transcript.

A takeover firing while a save is still in dispatch has no claim to
mark, so dispatchers open a refcounted takeover watch (TakeoverBasis)
at their last synchronous instant and the claim registration compares
its count, leaving the claim born overtaken when the key changed hands
in the gap. The vouch is a dispatch-time occupancy verify for
live-slot writers (save_slot_off_loop callers pinning
expected_slot_name, plus the flush and shutdown passes, which re-read
the live occupant and skip stale snapshot entries). The close/cleanup
teardowns write popped slots by design, so they open one watch before
their pop and carry it through every teardown save (the hand-over
drain lives in chat_api/handover_drain.py and takes the basis as
takeover_basis, handed to it only where the key names the drain's
file, _teardown_drain_basis). Their archive
result is read, not assumed: a refusal whose watch moved routes to the
hand-over arms, a verified deletion (definitive evidence only; an
unreadable probe is not a deletion) stays ignored, and any other
refusal restores the tab or drains its tail and reports the close
failed rather than discarding acknowledged rows.

The takeover note is tombstone-gated: a noter whose start postdates the
key's close tombstone is a reopen of a closed tab, not a takeover. It
notes nothing, waits for the ordering to settle, and is then refused by
the closed flag the close wrote, so the close commits the user's edit
instead of yielding the key to a reopen that cannot survive it. A
delivery rehydrate therefore never wins a close; only an explicit user
resume that began before the close does. A noter that noted and then
published nothing withdraws exactly the note it opened, through a
handle bound to the watch that issued it (a retired watch's handle is
a no-op against the fresh watch that replaced it). Accounting is per
note and basis-relative, never a net counter: the claim
registration and basis_moved both ask whether a note newer than the
basis still stands, so one noter's refusal can neither hide nor
fabricate another taker's note, and a close never yields the key to a
taker that gave up: it commits, or takes the retry arm and restores
the original with its tail owed; and a close that handed its key to a
still-unpublished taker keeps its dismissal owed until that taker
publishes or, as the last one standing, retracts and writes the closed
marker the refused archive left unwritten. Claims and notes carry the
transcript they write or declare, so a same-name publication resuming
a different file leaves a linked tab's archive to commit. The
arbitration's sync rules live in slot_persistence/takeover_ordering.py,
bound on the chat_persistence facade.

The read side honours the ordering: a resume whose takeover note
overtook an in-flight save waits, on the loop and bounded by the
save's own lock deadline, for the overtaken claims and the key's
truncating watch holders to retire (a plain append holds the watch
without holding readers; the waiter holds no executor thread), because a refused hand-over save lands the popped
slot's unsaved rows append-safely under its lock before releasing and
the transcript projection's cached, best-effort-locked reads cannot be
ordered by the lock alone. The wait fails closed: on expiry the resume
refuses with the retryable lock-contention error (HistoryLockTimeout;
503 takeover_ordering_unsettled at the endpoint) rather than hydrating
a window whose missing tail a later rewrite deletes.

The append-safe fallbacks persist the complete serialized rows: the
history layer accepts a fully built entry alongside its dedup identity,
so an appended tail carries the same ts, provenance, variants and meta
the committed payload would have. Watch ownership survives
cancellation: every executor save is shielded, the worker closes its
own basis when it settles, and a caller-opened basis transfers to the
worker on cancellation so the watch outlives every takeover the running
write must still observe. Switch-variant restores the pre-switch row
fields (redaction records included) when its durable write is refused.

The append-safe fallbacks settle the transcript header before their
first row, under the lock the appends take: ConversationLog.append mints
a header with no memory_mode for an absent file and leaves a present one
untouched, so a restricted slot's rows would otherwise land under a line
every learning reader takes as persistent. The header is created with
the slot's retained mode or tightened to it, the carried execution
record folded to match and the store a restricted line must not name
cleared, with the replacement's own fields kept -- the same ratchet the
rows-only save applies. The settled mode is confirmed by re-reading the
header under the lock; a header that cannot be read, or that still reads
looser than the retained mode, refuses the append outright (the drain
reports the rows and the close fails; the gate keeps the window owed).
The gate's claim refusal keeps the window owed like every other
retryable refusal, so a flush that lost its claim stays dirty; the
drain's append runs as a shielded executor future whose basis transfers
to its settlement on cancellation, and the rewind handler hands its
watch to an unsettled worker the same way. Both replays judge "already
persisted" against the whole durable history: append_if_absent gains
include_rotated, which folds the size-rotation archives into its
absence check, so a window reaching past the live file's rotation cut
lands only the rows the transcript lacks rather than re-landing every
archived one. A hand-over whose drain does not land restores the popped
original while its key is still free, with the compensation a raised
archive save already gets, so the tail's only holder is never dropped;
and a teardown watch retains a bounded number of declared read targets,
refusing an overflowing takeover note at admission. The hand-over drain
prepares its rows as the full save does -- the window filtered through
drop_notes_authorized_elsewhere, each row built by _build_message_entry
-- and declared read targets are recorded and matched by transcript
stem, so an alias spelling of one file is the same file. Both replays
share one helper, and the archive lookup resolves the physical file
stem rotation names its segments after, so a legacy bare-stem
transcript's archived rows are not appended again. save_slot_off_loop
opens a watch for every save its caller did not open one for: the
worker reads _pending_rewrite itself, so an append-shaped dispatch can
become a truncating rewrite in the gap, and a takeover landing there
must still leave the claim born overtaken.

The rewind handler's dispatch-verify refusal settles without a worker:
the save task is awaited only on the branch that dispatched it, so a
key that changed hands before dispatch audits as history_save_refused
rather than as a raised save.

Every interleaving resolves to one of two sequential histories. The
mutex guards in-memory transitions only and is never held across the
file replace, so the event loop never waits on the commit.

slot_lifecycle.py crossed the 2,000-line cap on the rebase (main alone
is 1,651); the repair-images and reset-conversation handlers move to
chat_api/slot_repair.py unchanged, per the responsibility-split rule.

Fixes kirodotdev#12090
@dimwael

dimwael commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

eac18f2501 — @bolichen97, the close-identity fix as you described it. Every close write now stamps a closed_id beside closed_at: a secrets.token_hex(8) nonce, minted per write — metadata_line.close_identity for the two slot-save forms (the codec gained the row, in both key orders, and SLOT_OWNED_META_KEYS, so a reopen or a later close drops a stale one with the stamp), and the owed marker's own DeferredDismissal.closed_id, which record_deferred_dismissal writes and OwedUnstamp carries. The un-stamp clears through clear_closed(only_if_closed_id=<ours>), compared inside the transcript lock against the line as it is then; the nextafter(closed_at) bound is gone, since the identity decides. A stamp carrying a different id stays, and a stamp carrying no id — an older build's line — is never ours to clear (fail closed). The test is the one you asked for: test_an_unstamp_keeps_a_newer_close_stamped_on_the_same_clock_tick freezes channel_slots.time, lands the owed marker, reopens and closes the new tab on the same tick (equal closed_at, its own id), runs the un-stamp and asserts the newer close is still there; with the instant-keyed clear restored it fails with "the newer close was erased by the un-stamp".

Separately, the Backend shard-1/8 reds on 3642a4c were a regression from my 44bca0a fix and are fixed here. The rehydrate wrapper's post-exit ownership recheck answered None whenever the slot it got back was not in state._slots — which is also what a body replaced by a test double looks like (test_rehydrate_async.py and test_mcp_app_absence_marked.py build a slot without registering it). The recheck now rests on evidence of retirement: the wrapper reads the key's occupancy as the body returns, before its exit work's first await, and refuses only a slot that held the key then and does not afterwards. Same change on the resume twin. The six upstream tests pass; the two tests from 44bca0a still fail with the recheck removed; and I ran every test file that calls either wrapper (38 files, 1,869 passed).

49 files, 1 commit, rebased on current main (0a44bf1265); 4,363 passed across the touched suites, both ratchets green; mypy, cap (resume.py 1,998), black 26.10.1, lint, comment-history, semgrep and docs gates clean. Could you approve the fork workflows on the new head?

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at eac18f2. This replaces my change request at 3642a4c.

My change request is fixed. Every close write now stamps a closed_id nonce beside closed_at: metadata_line.close_identity for slot saves, and the DeferredDismissal's own id for the owed marker. unstamp_superseded_marker calls clear_closed(only_if_closed_id=...), which compares the id under the transcript lock and keeps every other stamp. That includes a close of the new tab on the same clock tick, with an equal closed_at and its own id, and a stamp written without an id. test_an_unstamp_keeps_a_newer_close_stamped_on_the_same_clock_tick pins the equal-closed_at case, and the existing unstamp test now also checks the no-id stamp is kept. closed_id is slot-owned in history_schema, so a reopen drops it with the stamp.

The rest of the delta from 3642a4c: the two noter wrappers now read whether their slot held the key as the body returned, before the first exit await. They answer None / resume_session_deleted only when a slot that was under the key is gone afterwards, so a body that never registered its slot is handed back as built. The golden metadata line and the codec tables now carry closed_id.

Coverage: 44 files. I read every hunk of this PR across the earlier rounds, and each later head was a rebase or a delta I read in full.

CI: 86 checks green, including both AI review lanes, E2E (stub and hosted), the Windows E2E Gateway Boot, Integration and Native Redaction. The 8 cancelled entries in the rollup are older duplicate runs, and each has a newer passing run on this head. Only four Backend Tests (3.12) shards and PR Readiness are still running, and auto-merge waits for them.

@dimwael

dimwael commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @bolichen97 — holding this head as is; nothing further from my side until auto-merge lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Truncating history rewrites can commit onto a same-name replacement's transcript: slot publication is not ordered against the persistence commit

5 participants