Skip to content

fix(dashboard): preserve previous reply when regenerate ends empty - #15417

Closed
RohanK6 wants to merge 1 commit into
kirodotdev:mainfrom
RohanK6:fix/14632-regenerate-preserve-reply
Closed

RohanK6 wants to merge 1 commit into
kirodotdev:mainfrom
RohanK6:fix/14632-regenerate-preserve-reply

Conversation

@RohanK6

@RohanK6 RohanK6 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Regenerate that produces no new reply leaves the previous reply in place instead of destroying it.

Bug fix: pressing Regenerate and then having the new turn end before it produces any content permanently deletes the previous assistant reply. The old reply the user was trying to improve on is gone from the transcript and from disk, with no way to recover it. One concrete trigger is a Regenerate pressed while gateway memory preparation is still running — the turn is refused before it produces a reply — but any early exit from the turn before its first message hits the same path.

Why it matters

This is silent, user-visible data loss on a common action. A user reaches for Regenerate precisely because they value the current answer and want a variation; the failure mode throws that answer away exactly when the retry fails to replace it. There is no undo and no copy left anywhere, so the content cannot be recovered.

Not a goal

  • Not changing normal Regenerate behavior: once the new reply exists it still replaces the old one, which is retained as a variant.
  • Not adding a new recovery UI, undo affordance, or variant-management surface.
  • Not touching the streaming / segment-flush hot path or introducing a parallel persistence mechanism.

What changed (motivation → approach → change)

Symptom. Press Regenerate. If the new turn ends with no reply (refused, stopped before any text, failed), the old reply is gone from the screen and from disk.

Cause. The endpoint deleted the old reply and rewrote the transcript before the new turn ran. The only copy was destroyed before a replacement existed.

Change. Nothing is deleted until a new reply exists.

regenerate endpoint: keep the old reply; arm a RegenerateTarget
                     (old rows + transcript key + slot name + turn generation);
                     dispatch the turn. No write.
turn runs as usual; each reply row it writes is recorded on the target
turn end (_end_turn_tail, before the queue hands off):
  ├─ a reply was recorded → one pinned rewrite: old rows out, old reply kept as a variant
  └─ no reply             → drop the target; old reply untouched
  • chat_turn/regenerate_swap.py (new owner): RegenerateTarget, settle_regenerate, note_regenerate_reply, rows_before_regenerate_target.
  • chat_regenerate.py: the up-front del + truncating save + regenerate_save_refused 409 are gone; the endpoint arms the target. The done-callback drops a target no turn end settled.
  • queue_cycle._end_turn_tail: calls settle_regenerate at depth 0, before _hand_off_queue, so a queued successor never sees the target.
  • file_changes._note_reply_row (model segments, partial replies) and segment_flush._append_command_reply (slash-command replies) record the reply on the target.
  • segment_flush: the variant-stash loop moved verbatim into _adopt_pending_variants, so the settle step adopts variants for a reply no flush wrote.
  • save_transaction.save_slot_off_loop: forwards expected_disk_older_count to _save_slot_to_history, so the replace write pairs its snapshot with the frozen-prefix boundary read beside it.
  • docs/system-specs/modules/learn-cron-dashboard.md: a row for the new owner in the chat-runner composition table.
  • chat_runner: the crew-log ordinal skips the rows being replaced, so a regenerate is still a retry of the same turn.
  • turnActions.ts: the optimistic truncateAfterIndex and its rollback are removed. The old reply stays on screen until the server replaces it; the settle step broadcasts chat_variant_switch, which makes tabs refetch.

The replace write passes expected_history_key and expected_slot_name captured at the endpoint and the _disk_older_count read with its snapshot, with best_effort=False. The target records reply row ids, capped at _MAX_SLOT_MESSAGES. A rebind or a same-name recreate during the turn refuses it; a refused or failed write changes nothing in memory, so both replies stay and nothing is lost. Settling at the turn's end keeps the turn-stats and file-change boundaries on the window they were captured for, so turn_stats.py is untouched and the black-baseline edit from the earlier revision is not needed.

Backwards compatibility

Compatible: a Regenerate that produces a reply ends with the same window and variants as before; an empty turn keeps the reply.

Visible during a Regenerate: the previous reply stays on screen and the new reply appears below it until the turn ends, when the two collapse into one reply with the variant switcher.

Accepted outcome: if the gateway stops mid-turn after the turn's own save, the transcript keeps user, old, new with no variant link. Nothing is lost, and both rows render as ordinary assistant rows.

Reader: website/src/pages/chat/page/turnActions.ts:handleRegenerate -- chat -- the 409 regenerate_save_refused code had no reader (git grep on origin/main finds only the endpoint and its test); other refusals still reach showRefusedPress.
Reader: src/kiro_crew/dashboard/chat_turn/segment_flush.py:_flush_segment -- chat -- still adopts _pending_variants on the first segment (test_pending_variants_are_attached_and_broadcast).

Tests

test/test_regenerate_swap.py (new):

  • test_a_refused_regenerate_turn_keeps_the_previous_reply: real endpoint + real _run_chat, provider refuses → old reply in window and on disk. This is the Regenerate permanently deletes the previous reply when the new turn ends before its first message #14632 symptom.
  • test_a_regenerate_turn_with_a_reply_replaces_the_previous_one: real endpoint + _run_chat, provider streams text → window and disk hold only the new reply, variants [old, new].
  • Settle unit tests: a front-trim during the write refuses it, the reply record is bounded ids, empty turn, partial reply (Stop mid-stream), multi-segment reply keeps the flush's variants on the first segment, pins reach the save, a same-name recreate refuses the write and leaves the replacement's transcript alone, a failed write leaves both replies, a successor turn (other generation) neither records nor settles, the ordinal excludes the replaced rows, and _end_turn_tail settles before hand-off at depth 0 only.

test/test_dashboard_chat_regenerate.py: test_regenerate_empty_turn_keeps_the_previous_reply and test_regenerate_replaces_the_reply_once_a_new_one_exists replace the old truncate-and-persist test. Non-vacuous: re-adding the up-front del slot.messages[u_idx + 1 :] makes the empty-turn test fail.

test/test_chat_regenerate_cov80.py: the tests pinning the up-front write now assert the endpoint writes nothing and arms a pinned target. test_slot_ownership_checkpoint_post_await.py drops the regenerate case: the endpoint holds no await after its ownership check now. ChatPageMoreCoverage.test.tsx: the reply stays on screen after Regenerate.

Nearby inputs tested: trailing system notice after the reply, a newer /compact user row (still refused), slash-command reply, multi-segment reply, Stop mid-stream, recreate, write failure, queued successor.

Determinism: test_regenerate_swap.py + test_dashboard_chat_regenerate.py + test_chat_regenerate_cov80.py passed 3 repeats (96/96 each, -n 4), and the first two passed in a seeded shuffled order (42/42, -n 0). black / isort / flake8 / mypy (3304 files) / eslint / tsc -p tsconfig.app.json clean.

Manual verification

N/A — unit coverage sufficient: the two end-to-end tests drive the real endpoint and _run_chat with a fake provider and read the transcript back from disk.

Screenshots / video

Why no screenshot: no component, layout or style changes; the existing rows and variant switcher render as before, and only the moment the previous reply leaves changes (see Backwards compatibility).

Related Issues

Fixes #14632

Continues the work of @RohanK6 (co-author on the commit), ported onto current main.

Pattern harvest

Rule candidate: review-prompt
Pattern: a replace operation deletes or truncates live state before its replacement exists.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@RohanK6
RohanK6 requested a review from a team as a code owner September 30, 2026 13:28
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@RohanK6 RohanK6 changed the title [KiroCrew] Preserve previous reply when regenerate ends before its first message fix(dashboard): preserve previous reply when regenerate ends empty Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 34c036e19315010dbc505c5abcd8330b7b93e2ec via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check the accepted crash outcome: a gateway stop mid-turn leaves user, old, new on disk with no variant link and nothing ever merges them.

What this change ships

Inventory (9 items) — 9 justified

Intent: stop a Regenerate whose new turn ends empty (refused, stopped, failed) from permanently deleting the previous reply (#14632). FIX.

  1. A Regenerate that ends with no reply now keeps the previous reply on screen and on disk — justified
  2. During a Regenerate the old reply stays visible and the new one appears below it until the turn ends — justified
  3. At the turn's end the old reply is replaced and kept as a variant, in one write tied to the original transcript and slot — justified
  4. The endpoint's 409 regenerate_save_refused response is removed — justified
  5. The dashboard no longer hides the old reply as soon as Regenerate is pressed, and no longer restores it on failure — justified
  6. The settle step broadcasts chat_variant_switch so open tabs refetch — justified
  7. The crew-log turn number ignores the reply being replaced, so a Regenerate is still logged as a retry — justified
  8. save_slot_off_loop passes the existing expected_disk_older_count through to _save_slot_to_history — justified
  9. The variant-stash loop moves into _adopt_pending_variants, which partial replies and command replies now use too — justified

[FIRST-PRINCIPLES-REVIEWED] 34c036e

Description read

[DESCRIPTION-READ] dd200c265b79dc4dbc2434931fc403b9fd378d6f8c55c76281b202316ff8d016

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 34c036e19315010dbc505c5abcd8330b7b93e2ec via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check the new in-between screen by hand: the old reply stays visible with the new one streaming under it, then refreshSlot merges them. No screenshot covers this.

[DESIGN-REVIEWED] 34c036e

Description read

[DESCRIPTION-READ] 143a1c83de1352133893f79bfeb987665db2d7530400202d05e7ff029aedf80a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 34c036e19315010dbc505c5abcd8330b7b93e2ec via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 34c036e

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 34c036e19315010dbc505c5abcd8330b7b93e2ec via the fork AI-review pipeline; updated in place on each push.

Review details

Advisory only: the history spec still describes the regenerate truncation that this PR removed.

FINDING — docs/system-specs/modules/history.md:1450 — the spec still says "_pending_rewrite is set by rewind/regenerate after they truncate the window", and line 1680 still says "a regenerate keeps _pending_rewrite and the next save retries". chat_regenerate.py no longer truncates or sets _pending_rewrite: the regenerate write now happens at turn end in settle_regenerate, and if that write is refused it keeps both replies. The spec now contradicts the code. → Fix: in the same commit, update history.md lines 1445–1450 and 1680, plus the regenerate entry in session.md:2253, to describe the pinned turn-end replace write.

[OPUS-REVIEWED] 34c036e

@RohanK6
RohanK6 force-pushed the fix/14632-regenerate-preserve-reply branch from 59e37bf to 53270d3 Compare September 30, 2026 14:16
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 30, 2026
@RohanK6

RohanK6 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the Opus 5 review on 59e37bfc2b at root in 53270d3ea7:

  • BLOCKING — refusal-not-read (chat_regenerate.py): the restore's re-persist now reads save_slot_off_loop's return. A refused guarded write (slot closing, rebound, or replaced) leaves slot._dirty/slot._pending_rewrite armed for the periodic flush and logs a warning naming the slot — no success line, no unconditional publish. A refusal is not a commit.
  • FINDING — system-notice tail guard: the "did a reply land after the user row" check now uses is_system_notice("assistant", meta), so a turn that emitted only an auto-compaction notice still restores the previous reply instead of taking the cannot restore branch — the one row class the endpoint's own reply scan already skips.
  • FINDING — silent restore: each restored row is now broadcast as a chat_message frame carrying its meta.mid (append's own delivery door), so a viewer's optimistic truncation is reversed instead of showing the reply as deleted until reload.

Tests added: restore survives a compaction-notice-only turn; the restored reply is broadcast; a refused re-persist leaves the slot dirty and makes no success claim.

Backend Tests (Windows) (7) — test/test_runtime_cleanup_windows.py::test_windows_failed_owner_drain_recovers_after_owner_drop (assert None is not None): unrelated to this change. That module is skipif(sys.platform != "win32") and exercises AcpSessionProvider.shutdown() native Windows process-tree cleanup (platform_compat/session_pid/acp.runtime) under a 15s timeout — a different subsystem this diff (dashboard chat regenerate only) does not touch or import. Treating it as a Windows-runner timing flake; the new head re-runs CI.

@RohanK6
RohanK6 force-pushed the fix/14632-regenerate-preserve-reply branch from 53270d3 to 86ba435 Compare September 30, 2026 14:50
@RohanK6

RohanK6 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the GPT 5.6 review on 53270d3ea7 at root in 86ba43539f — all four findings fixed with a persist-first redesign of the restore:

  • F1 (chat_regenerate.py:342) — restored metadata bypassed display redaction: the restored rows are deep-copied from the live window, so their content and meta are now run through redact_display_content / _redact_meta_for_role before broadcast, matching the live broadcast door. No raw meta reaches clients.
  • F2 (:380) — restore could follow a concurrently rebound transcript: the restore now authorizes against the expected_history_key captured before the endpoint's awaits (not one re-resolved inside the coroutine), and revalidates slot identity (map object + history key) before both the save and the install. A mid-turn cron/workflow rebind cannot land the old rows on another transcript.
  • F3 (:398) — restoration published before durable commit: restructured to persist-first. The restored window is written to disk first through the guarded rewrite; only a committed write installs it on the live slot and broadcasts it. A refused or crashed write leaves the truncated transcript rather than a published-but-unsaved restore.
  • F4 (test_chat_regenerate_cov80.py:234 / test_dashboard_chat.py) — loosened poll budgets: the restore runs as a tracked task (slot._regenerate_restore_task); tests now await that task and the turn instead of raising yield counts, so a missing synchronization fails rather than being absorbed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@RohanK6
RohanK6 force-pushed the fix/14632-regenerate-preserve-reply branch from 86ba435 to b7b2eb7 Compare September 30, 2026 15:20
@RohanK6

RohanK6 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the GPT 5.6 review on 86ba43539f at root in b7b2eb7219:

  • BLOCKING (chat_regenerate.py:329) — restore rewrite erased concurrent durable appends: the persist-first save takes the rewrite path (collect_foreign off), so a row a sub-agent/cron/CLI appended to the shared transcript during the save's executor wait would be dropped from the canonical transcript. Fixed: on commit the restore rebuilds the live window as head + restored rows + tail where tail includes any row that arrived during the save (identified against the pre-save row-id set), and it now clears _pending_rewrite instead of re-arming it, so the next flush merges on the ordinary collect_foreign path rather than a rewrite. Added test_regenerate_restore_preserves_a_concurrent_foreign_append.
  • FINDING (:389) — function-local import: moved redact_display_content to the module-level chat_utils import.

1004 tests pass across the touched + adjacent modules; comment-history and black clean.

@RohanK6
RohanK6 force-pushed the fix/14632-regenerate-preserve-reply branch from b7b2eb7 to 565dc6a Compare September 30, 2026 15:24
@RohanK6

RohanK6 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Head 565dc6a739:

  • Backend Lint & Type Check — the failure was an isort ordering error in this diff: the added from kiro_crew.dashboard.chat_utils import _redact_meta_for_role as redact_meta_for_role was placed after the multi-name chat_utils import instead of before it. Fixed by running isort (the aliased single-name import sorts onto its own line ahead of the grouped import). flake8, mypy, isort, and black now all pass locally on the touched files. Not the black-baseline-shrink gate — no baselined file changed.
  • GPT 5.6 BLOCK (restore rewrite erased concurrent durable appends) — fixed at root on the prior head and carried here: the restore now rebuilds the live window as head + restored + tail where tail includes any row that landed during the persist-first save (identified against the pre-save row-id set), and clears _pending_rewrite so the next flush merges on the ordinary collect_foreign path rather than a rewrite dropping the append to the archive. Covered by test_regenerate_restore_preserves_a_concurrent_foreign_append.

1004+ tests pass across the touched + adjacent modules; comment-history clean.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 30, 2026
@RohanK6
RohanK6 force-pushed the fix/14632-regenerate-preserve-reply branch from 565dc6a to 9477319 Compare September 30, 2026 16:02
@RohanK6

RohanK6 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Head 9477319817 — addressed the GPT 5.6 review on 565dc6a739:

  • BLOCKING F1 (chat_regenerate.py:339) — restore could publish an unsaved or prefix-misaligned snapshot: the restore's persist-first write went through save_slot_off_loop (default best_effort=True), which swallows a lock timeout / I/O error and returns True, so a "committed" restore could be published with nothing on disk; and it had no way to pair the snapshot with the frozen-prefix boundary, so a concurrent cap-trim could duplicate the trimmed rows. Fixed by persisting through _save_slot_to_history directly (the same path the sibling rewind uses): it raises on failure — the except turns that into committed=False, so nothing is installed or broadcast — and it is passed expected_disk_older_count (the disk-older count captured before the save) so a moved window boundary refuses the write. register_guarded_history_write holds it against a slot-closing retraction, as rewind's write is.
  • F2 (:388) — window-cap bypass: downgraded to advisory by the Opus adjudicator (a self-erasing +1 overshoot that the next ordinary append trims back, and the real fix would replicate frozen-prefix counter bookkeeping and add a corruption surface). Left as-is per the adjudication.

flake8 / isort / mypy / black clean on the touched files; 973 module tests pass; comment-history clean; disposition here; PR body in lockstep.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 30, 2026
@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the variant inline-image fidelity finding at history.py recovery: an agent-produced inline image carried in a variant (an alternate reply the user can switch back to) was redacted but not copied into the durable attachments store on the rebound-slot recovery path — so it kept a reference to the agent-scratch dir, whose bytes are reclaimed when the agent process dies, with no live window to repair it later.

The recovery variant loop now persists each variant's inline-image bytes into the transcript's durable attachments store before redaction, mirroring the primary-content call directly above it and the slot save's own variant loop. Added test_recovery_persists_a_variant_inline_image (verified non-vacuous: without the variant persist step the recovered variant keeps its scratch-dir reference and the test fails).

Gates green: regenerate/history/dashboard suites pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, rebased onto current origin/main.

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the recovery-registration finding at chat_regenerate.py: the FIRST recovery write to the original transcript was awaited while registered nowhere — the drain registry entry was created only after that first write failed, inside the retry path. A gateway stop landing during that first await (rows already held out of the slot window, the coroutine suspended on disk I/O) would find nothing registered, so the shutdown drain could not finish it and the reply was lost.

Fix: _recover_to_original_transcript now registers the write in the drain registry BEFORE awaiting it, removing the entry only on a confirmed commit and retaining it on failure — so every in-flight recovery write (first attempt or retry) is visible to the shutdown drain. The registry is keyed by the original transcript (one in-flight recovery per transcript), so a retry replaces rather than stacks, and the per-attempt registration in the retry path (now redundant) is removed.

Added test_regenerate_first_recovery_write_is_registered_before_it_is_awaited (verified non-vacuous: without register-before-await the entry is absent when the first write runs and the test fails).

Gates green: 1354 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, 0-behind current origin/main.

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the residual-truncation window at chat_regenerate.py: the eager truncation persists the original transcript short of its reply BEFORE the turn is dispatched, but the removed reply was only registered as a pending recovery at turn-END (in the done-callback). For the entire turn the reply lived only in an in-memory closure — invisible to the shutdown drain — so a gateway stop during the turn (both ordinary) left the original durably truncated with the recovery registered nowhere.

Fix: the removed reply is now registered as a pending recovery SYNCHRONOUSLY at the truncation (keyed by the original transcript), before the turn runs, so a stop anywhere during the turn is drain-visible. The pre-turn entry is cleared the moment the reply is durable again by any route — a new reply landed and the slot save persisted it, the empty-turn restore spliced it back and committed, or the recovery wrote it to the original transcript. A deleted/absent transcript is still honored (not resurrected).

Added test_regenerate_shutdown_during_the_turn_recovers_the_truncated_reply — drains while the turn is blocked mid-flight and asserts the reply reaches the original transcript (verified non-vacuous: without the pre-turn registration nothing is registered during the turn and the test fails).

Gates green: 1355 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, 0-behind current origin/main.

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the stale-pending-entry finding at chat_regenerate.py: when a regenerate turn ends abnormally with a partial reply (persisted outside _flush_segment, so _pending_variants stays set) and the variant-attach save does NOT commit on the same transcript, the old code left the slot dirty for the periodic flush AND left the pre-turn _PENDING_RECOVERIES entry in place. Two durability routes then collided: the periodic flush persists the old reply as a nested VARIANT of the partial reply, while the drain entry writes it as a TOP-LEVEL row — and the top-level write is not deduped against a nested variant (dedup scans top-level rows by mid), so a later shutdown drain would DUPLICATE the reply. There is also no general dirty-slot flush at shutdown, so the nested-variant route is not even a guaranteed durable copy.

Fix: on the non-committed same-transcript attach, revert the dirty variant attach to its pre-attach snapshot (so the periodic flush persists no second copy) and recover the old reply through the SAME single drain-registry path (a top-level write, cleared on commit, retried on failure) — mirroring the sibling rebind branch minus the rebind. The drain now owns exactly one copy and cannot duplicate the reply, and no stale entry lingers.

Added test_regenerate_partial_reply_attach_failure_clears_the_pending_entry (verified non-vacuous: with the old held-dirty behavior the entry lingers and the test fails).

Gates green: 1356 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, 0-behind current origin/main.

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the reply-landed success-path finding at chat_regenerate.py: the done-callback dropped the pre-turn recovery entry SYNCHRONOUSLY on the reply-landed branch, before confirming the replacement was durable. _flush_segment attaches the old reply as a variant of the new reply in the LIVE window only (it has no durable write of its own, and slot.append is in-memory), so a transient replacement-save failure plus a correlated gateway stop would lose BOTH replies from an already-truncated transcript — the drain only covers entries still registered, and this one was already popped.

Fix: on the reply-landed path the entry is popped only AFTER a confirmed replacement save (an awaited save_slot_off_loop with best_effort=False, pinned to the slot's transcript), mirroring the drain's commit-confirmed-then-pop discipline. A non-committing save (or a mid-await rebind to a different transcript) RETAINS the entry so the drain still has the old reply.

Added test_regenerate_reply_landed_retains_entry_until_replacement_save_commits (verified non-vacuous: an unconditional pop drops the entry on a failed save and the test fails).

Gates green: 1357 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, rebased onto current origin/main (0-behind).

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

This push carries three recovery-machinery fixes:

  1. Survivor-save pop guard (chat_regenerate.py): the empty-turn restore's survivor-save pop cleared the drain entry by key unconditionally. When bound enforcement evicts rows and their recovery write does not commit, a pending entry (holding the evicted rows) is owed under the SAME key — a committed survivor save would then drop it, losing the evicted rows. The pop is now guarded so it does not clear a still-pending evicted-rows recovery.

  2. Variant-chain dedup (history.py append_full_message_if_absent): a row already present as a nested variants entry of an on-disk row was not deduped against a top-level write, so the shutdown drain could duplicate a reply already persisted as a variant (a regenerate attaches the previous reply as a variant of the new reply). The dedup now also scans the variant chain and treats such a row as already persisted.

  3. Bounded recovery retry (chat_regenerate.py _retry_recovery): a durable write that can never commit (full / read-only data home) re-armed the retry forever at 4Hz, pinning _regenerate_restore_pending and wedging the slot into a permanent slot_restoring 409 on every regenerate/edit-resend/switch-variant (and making _close_slot raise forever). The retry is now bounded (reusing the drain's attempt cap): on exhaustion it stops re-arming, LEAVES the _PENDING_RECOVERIES entry for the shutdown drain, and RELEASES the slot fences.

Three non-vacuous tests added, all verified. Gates green: 1360 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, 0-behind current origin/main.

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the second-regenerate overwrite finding at chat_regenerate.py: the pre-turn recovery registration keyed _PENDING_RECOVERIES by the transcript key, so a SECOND regenerate on the same transcript — admitted once the first's fence releases (which the new bounded-retry does on exhaustion, leaving the entry pending) — unconditionally OVERWROTE the first's still-pending entry. A stop then flushed only the second reply; the first was silently lost from the already-truncated transcript.

Fix: the registry is now keyed PER-REGENERATE (a unique id per regenerate), so two regenerates on the same transcript keep distinct entries and the second cannot clobber the first's still-pending recovery. The entry's stored transcript_key is unchanged, so the drain still writes each to the correct transcript; a retry WITHIN one regenerate reuses its own id (replace, not stack); and _clear_pending_recovery / the commit-pop target the regenerate's own id.

Added test_second_regenerate_does_not_clobber_the_first_pending_recovery (verified non-vacuous: with a shared transcript key the second overwrites the first and the test fails).

Gates green: 1361 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, rebased onto current origin/main (0-behind).

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the absent-transcript finding at history.py: recovery treated an absent transcript as a legitimately-deleted session unconditionally (settle, do not recreate). Disk state alone cannot tell a DELETE from a NEVER-PERSISTED truncation (both leave no file), so a never-persisted turn's reply was silently discarded as if the user had deleted the session.

Fix: the recovery record now carries an existed_at_truncation bit — captured at the eager truncation from whether the transcript file existed then — and restore_full_rows_off_loop honors absence (settle, no recreate) ONLY when the transcript existed at truncation (a genuine delete). A transcript that never existed (existed_at_truncation=False) and is absent now was never deleted — its truncation simply never committed — so the recovered reply is WRITTEN to recreate the transcript (its rightful durable home) rather than discarded. The bit is threaded through the drain registry entry too, so the shutdown drain applies the same rule.

Added test_recovery_absence_honored_only_when_transcript_existed_at_truncation (verified non-vacuous: with the distinction removed, the never-persisted case fails to recreate and the test fails). Both branches covered: never-existed → recreated with the reply; existed-then-deleted → honored.

Gates green: 1367 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, rebased onto current origin/main (0-behind).

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Two fixes in this push — one a regression from the previous existed-at-truncation fix, one a plain retention bound:

F2 (regression) — history.py: don't resurrect a user-deleted session. The existed_at_truncation bit is captured once at truncation and can go stale: a user deleting the session AFTER truncation (on a not-yet-flushed conversation the bit even reads existed=False while the truncation save created the file and the user then deleted it) would hit the never-persisted branch and RECREATE the deleted session. Fix: delete_session now records the key in a bounded recently-deleted tombstone, and recovery consults it FIRST — a recorded intervening delete is honored (settle, do not recreate) regardless of the stale bit. So: never-existed + no delete → recreate the reply; any recorded delete → honor it (no resurrection).

F3 — chat_regenerate.py: bound the recovery registry. _PENDING_RECOVERIES had no retention bound. Added _register_pending_recovery with a FIFO cap (_PENDING_RECOVERIES_MAX): a new entry past the cap evicts the oldest (with a warning), so a pathological run of never-committing recoveries can't grow the registry without limit. A retry within one regenerate refreshes its own key in place without growing.

Two non-vacuous tests added (intervening-delete honored despite stale bit; registry bounded), both verified fail-without-fix. The tombstone is bounded (FIFO, 256) and the registry cap is 512.

Gates green: 1369 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, rebased onto current origin/main (0-behind).

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the unbounded-payload finding at chat_regenerate.py: the recovery registry had only an entry-count cap, which does not bound MEMORY — each entry holds a full message-row payload (a long reply, its variant chain, inline-image refs), so N entries of unbounded size is still unbounded. Added an aggregate payload-byte cap alongside the entry cap: _register_pending_recovery tracks each entry's approximate serialized size, keeps a running total, and evicts the oldest (FIFO, with a warning) until BOTH the entry count AND the byte total are under their caps. A single oversize entry is kept (the reply must survive to be drained); the cap bounds accumulation across entries. All pop paths (commit, clear, drain) route through a single _forget_pending_recovery so the byte total never drifts.

On the cross-process-tombstone finding (F2): the in-memory recently-deleted tombstone correctly honors an intervening delete within THIS process (the common case), and a session that existed at truncation is already honored as a delete by the existed_at_truncation bit WITHOUT any tombstone — so the only residual gap is a second gateway on the same data home deleting a never-flushed conversation mid-regenerate. Making that cross-process-correct needs a durable on-disk deletion marker with a lifecycle that outlives any in-flight recovery in ANY process — i.e. cross-process coordination machinery, the same hard-exit/multi-gateway class deemed disproportionate for this PR. Flagged for the maintainer rather than fixed in scope.

Non-vacuous tests: the registry stays bounded by both entry count and byte budget under repeated recoveries (verified fail-without-fix). Gates green: 1369 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, rebased onto current origin/main (0-behind).

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Three fixes this push:

F3+F4 (chat_regenerate.py) — no recovery path bypasses the memory ceiling. The byte cap kept a single oversized entry (the eviction loop stopped at the just-added key), so one huge payload could sit in memory unbounded at either registration site. Now after evicting older entries, a lone entry still over the byte ceiling is dropped (logged) — its in-flight write already ran, so this only declines to RETAIN an over-ceiling payload for the drain, the same bounded loss eviction makes for older entries. Both registration sites route through the one _register_pending_recovery, so neither can bypass it.

F2 (history.py) — tombstone normalized across alias spellings. The recently-deleted tombstone keyed on the raw string, so it missed the alias forms _path collapses (slack:/slack_/bare <ts>): a delete under one spelling wouldn't match a recovery checking under another, resurrecting the deleted session. The tombstone now records AND checks under transcript_lock_stems(key) — the same symmetric alias collapsing — so any spelling matches.

F1 (history.py:800) verdict: the existed_at_truncation=False-by-respelling vector is now covered by the F2 normalization — a delete under any spelling is honored regardless of the stale bit. The only residual F1:800 case is a genuine cross-process delete (a second gateway on the same data home), which a process-local tombstone cannot see — the same cross-process/shutdown-during-save class flagged with F1:86 as out of scope for this PR.

Three non-vacuous tests added/extended (single-oversize ceiling; byte-cap eviction; alias-spelling tombstone match), all verified fail-without-fix. Gates green: 1370 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, 0-behind current origin/main.

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Made the recovery-registry eviction LOSSLESS at the ceiling (F1:143). The prior byte/entry cap dropped an over-ceiling or evicted pending recovery with only a warning — trading unbounded memory for data loss, since an entry at the ceiling owes an already-truncated reply.

Now _register_pending_recovery is async and, before removing any entry at the ceiling, FLUSHES it to its transcript via the same confirmed restore_full_rows_off_loop the shutdown drain uses, removing it only once the write COMMITS. Fail-closed: a flush that does not commit KEEPS the entry (bounded memory growth beats silent data loss) and the loop skips it so one stuck transcript cannot wedge eviction. This applies to all three eviction paths — oldest-at-count-cap, byte-cap, and a single over-ceiling payload. The ceiling now bounds in-memory retention WITHOUT losing an owed reply.

Test (test_pending_recovery_registry_is_bounded_and_eviction_is_lossless) fills to each ceiling, triggers eviction, and asserts the evicted reply is PRESENT on its transcript (verified non-vacuous: a lossy drop fails it).

The only residual F1 case is the genuine cross-process delete (second gateway on the same data home) + shutdown-during-save (F1:86) — out-of-scope cross-process coordination, flagged for the maintainer.

Gates green: 877 dashboard + 493 regenerate/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, rebased onto current origin/main (0-behind).

@RohanK6

RohanK6 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Simplified the recovery-registry bound to a plain entry-count cap, removing the byte-cap + lossless-flush-on-eviction machinery (per the adjudicator's subtraction remedy). The flush-on-eviction was circular: the only thing that fills the registry is a durable write that keeps failing, and the lossless eviction ran that SAME write — so at the ceiling the flush also failed and nothing was evicted, defeating the bound.

Removed: _PENDING_RECOVERIES_MAX_BYTES, _PENDING_RECOVERY_SIZES, _PENDING_RECOVERY_TOTAL_BYTES, _recovery_payload_bytes, the _flush_then_forget eviction loop, and the json import they needed. _register_pending_recovery is sync again and keeps only the count cap (_PENDING_RECOVERIES_MAX=512): on overflow it drops the oldest entry (FIFO), which bounds the registry deterministically even when every write is failing — no dependency on the durable write.

Test simplified to test_pending_recovery_registry_is_bounded: fills 100 entries with a cap of 4 and every write failing, asserts the registry stays bounded and the oldest were dropped (verified non-vacuous).

The only remaining fenced finding is the cross-process-delete / tombstone-ordering class (second gateway on the same data home) — out-of-scope cross-process coordination, flagged for the maintainer.

Gates green: 1370 regenerate/dashboard/history tests pass; black/isort/flake8/mypy + comment-history + black-baseline clean. One commit, rebased onto current origin/main (0-behind).

@RohanK6

RohanK6 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the three bounded findings from the latest review (head `2752668126`, one commit, rebased on fresh `origin/main`):

  • F2 — variant-index corruption (chat_regenerate.py:923): the partial-reply variant attach now ALWAYS appends the partial before applying the cap, matching `_flush_segment`. The prior content-dedup could skip the append while still setting `variant_idx = len(variants) - 1`, so the active index pointed at an older matching variant instead of the partial. New test `test_restore_variant_idx_names_the_partial_when_content_matches_an_old_variant` (non-vacuous against the dedup-skip).
  • F1 — drain-before-attach duplicate on shutdown: the shutdown drain now settles each regenerate's turn + attach/confirm/restore task before writing its fallback recovery entry, then re-reads the registry so a settled task that already persisted-and-dropped the entry is not written a second time. The gate never blocks on a turn still in flight (a shutdown mid-turn, which the drain must flush). New test `test_shutdown_drain_settles_attach_before_writing_so_no_duplicate_reply` (non-vacuous: a blind drain makes a redundant write).
  • :99 — memory-bound at admission: a named serialized-payload byte bound (`_MAX_RECOVERY_PAYLOAD_BYTES`) and a reserve-capacity-before-truncate check are enforced at ADMISSION, before the destructive truncation. A payload over the bound or a full registry refuses the regenerate (409) with the previous reply intact — not the circular eviction-flush bound. New tests `test_regenerate_refuses_when_recovery_payload_exceeds_bound_reply_intact`, `test_regenerate_refuses_when_recovery_registry_is_full_reply_intact`.

Gates green on the rebased tree: regenerate/history/dashboard suites, black (check script), isort/flake8/mypy, comment-history enforced vs origin/main. MERGEABLE, one commit. PR description updated in lockstep.

@RohanK6

RohanK6 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the Opus 5.5 blocking finding (close mid-regenerate: restore saves refused by the close fence → out-of-order fallback write + _PENDING_RECOVERIES leak) at root on head 45dbe8afb1.

Root cause: when a tab is closed mid-regenerate, _close_slot sets is_closing, cancels the turn, and awaits the scheduled restore before archiving. The restore's guarded save_slot_off_loop calls were refused by the is_closing dispatch fence (they did not pass issued_by_the_retraction). That drove the partial branch to revert the variant attach and write the old reply as a stray TOP-LEVEL row, which the close's archival merge then interleaved by ts into user → OLD reply → NEW partial reply; the empty-restore and replacement-confirm saves were likewise refused and never popped their recovery entry, leaking one per close toward the 512 cap.

Fix (the reviewer's named remedy): these restore saves ARE the retraction's own drain write — sequenced by the close, which awaits them, and the last chance the reply has to reach disk in order — so they now pass issued_by_the_retraction=getattr(slot, "is_closing", False). The fence lets them through; the restore commits in order (old reply kept only as the partial's variant) and the recovery entry is popped. Applied to all three guarded restore saves (partial-attach, empty-restore merge, replacement-confirm).

Test (red-then-green, non-vacuous): test_close_mid_regenerate_keeps_order_and_pops_recovery_with_real_save drives a close mid-regenerate through the REAL save_slot_off_loop (prior close tests mocked it, bypassing the fence — which is why this slipped through). It asserts the archived order is user → partial (old reply only as the partial's variant, no stray top-level row) and that _PENDING_RECOVERIES is popped. Confirmed it fails pre-fix with user → assistant → assistant.

Full regenerate + dashboard suites green; black / isort / flake8 / mypy / comment-history clean. One commit, rebased on current main.

@RohanK6

RohanK6 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Addressed both GPT 6.1 blocking findings at root on head 951407fc90.

F1 (chat_regenerate.py partial-branch rebound recovery — cross-transcript variant leak): on the partial-reply branch, when the variant-attach save finds the slot REBOUND to a different conversation, the recovery to the original transcript was awaited while the partial row still carried the old conversation's attached variants — so a periodic flush landing during that await could persist them into the newly-bound transcript. Fixed by reverting the attached variants/index/content BEFORE the recovery await (mirroring the identity-rollback the commit/refuse branches already do); the rows survive in removed_rows, so the revert loses nothing recoverable.

F2 (admission reservation — concurrent-regenerate TOCTOU): _recovery_admission_refusal read len(_PENDING_RECOVERIES) for headroom, but the admitted entry is not registered until many awaits later, so two concurrent regenerates at the MAX-1 boundary could both pass admission and both later register, overflowing the cap and FIFO-dropping an unpaid older recovery. Fixed with a synchronous reservation counter (_RESERVED_RECOVERIES): incremented at admission with no await between the headroom check and the increment, counted alongside live entries, and released exactly once — at the pre-turn registration (where it becomes a real entry) or on each aborted-truncation return path before registration. The minimal form the review named — one reserve at admission, release on each abort/exception branch; it never evicts, only refuses a second over-committing admission.

Tests (both red-then-green, confirmed non-vacuous):

  • test_partial_attach_rebind_does_not_leak_old_variants_to_new_transcript — observes the partial row's attached-variant state at the recovery await and asserts the old reply is absent there; fails when the revert is ordered after the await.
  • test_concurrent_regenerate_second_refused_at_admission_no_unpaid_eviction — the first regenerate holds its reservation across its truncating-save await, the second is refused at admission (409 recovery_capacity_exhausted) and no owed entry is evicted; fails when the reservation is not counted.

Full regenerate + dashboard suites green; black / isort / flake8 / mypy / comment-history clean. One commit, rebased on current main.

@RohanK6

RohanK6 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Addressed both GPT 6.1 blocking findings at root on 4c319d6583 (one commit, rebased on fresh main, MERGEABLE):

F1 (history.py — delete+recreate reinsert). A failed-regenerate recovery was pending; deleting the Slack history and posting to the same thread recreated the same-name transcript, so the existence check saw the file present and the stale reply was reinserted into the recreated session. Added a monotonic per-stem deletion generation on ConversationLog, bumped on every delete and captured at truncation. restore_full_rows_off_loop now rechecks it under atomic_appends regardless of whether the file exists, so any delete in the window — recreated or not — is a generation bump and the delete is honored (write nothing, settle). Threaded through the drain entry too.

F2 (chat_regenerate.py — shutdown-before-first-segment duplicate). _settle_gate returned immediately for an in-flight turn, so a shutdown mid-stream let the drain write the old reply standalone while the live turn's own replacement save also persisted it (standalone + variant). It now quiesces the in-flight turn — cancels it and awaits its done-callback's settle task — then re-reads the registry and writes nothing of its own, so exactly one writer touches the row.

Tests (red-then-green, each confirmed non-vacuous):

  • test_drain_does_not_reinsert_into_a_deleted_then_recreated_transcript
  • test_shutdown_before_first_segment_does_not_duplicate_the_old_reply

Gates: 115 regenerate + 878 dashboard + 403 history/conversation tests pass; black / isort / flake8 / mypy / comment-history (enforced vs main) clean. PR body updated in lockstep.

@RohanK6

RohanK6 commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

@iamwhatever Thanks for the detailed review — the snapshot-soundness read is right, and I've kept the current design (no inversion). Rebased onto fresh main (was 248 behind; main's extraction of the slot-lifecycle machinery into chat_api/slot_lifecycle.py was the only non-trivial collision — the close-path changes below now live there). Head is now a single commit, MERGEABLE.

The four asks are addressed:

Ask 1 — _destructive_history_busy must treat a pending restore as busy.
It now returns the 409 slot_restoring refusal when either the _regenerate_restore_pending marker is set OR _regenerate_restore_task is present and not done. So a second Regenerate/Edit/Clear landing while the restore has spliced the old reply back and is still awaiting its disk save (the window you flagged at :411→:439) is refused rather than racing the re-broadcast.

Ask 2a — the vacuous test now exercises the real field.
test_regenerate_partial_reply_survives_successor_clearing_turn_mids now clears/asserts _turn_reply_mids_all (the list the restore actually reads), not _turn_reply_mids, so it genuinely drives the snapshot path instead of passing around it.

Ask 2b — new second-op-during-save test.
test_second_regenerate_during_restore_save_is_refused holds the restore's save in flight and asserts the second regenerate gets 409 slot_restoring (ask-1's guard working), then that _regenerate_restore_task resets to None once the restore settles.

Minor 3 — splice/stat-attach alignment.
A running follow-up's rows are anchored by turn_start_mid identity (_turn_rows), not by the pre-splice numeric boundary, so the restore's splice inserting rows before that anchor no longer shifts _attach_turn_stats onto the wrong row.

Minor 4 — reset the restore task.
_regenerate_restore_task is set back to None when the restore completes (done-callback and the settle paths), so a stale handle can't keep the slot read as busy.

Full local sweep green on the rebased head: black (py310) / isort / flake8 / mypy, the black-baseline and comment-history gates, and the regenerate + dashboard-chat + composition-contract + unattended-guardrails + history suites. Ready for re-review.

@RohanK6

RohanK6 commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Addressed both bounded recovery-reservation/entry leaks in the admission machinery at root (head 311a22b9d5, one commit, rebased on current main, MERGEABLE).

BLOCKING — chat_regenerate.py app-token slot-replaced return leaked an admission reservation. An app-token regenerate passes admission and _reserve_recovery() runs; the slot is then closed and recreated under the same name while the truncating save awaits, and the slot_not_found() return exited before registration without releasing. Admission counts len(_PENDING_RECOVERIES) + _RESERVED_RECOVERIES, so each occurrence permanently consumed one of the 512 slots. Fix: release the reservation before that return, exactly as the two save-refused branches already do — restoring "release the reservation on every refusal".

FINDING — chat_regenerate.py / chat_api/slot_lifecycle.py close-mid-stream leaked a recovery entry. When a tab is closed after the regenerated reply has streamed through _flush_segment (which clears the restore marker and _pending_variants, so _await_regenerate_restore returns True at once), _close_slot pops the slot and the turn's _confirm_replacement_then_clear save is refused on the popped slot, so its own _clear_pending_recovery never runs and the pre-turn entry lingers toward the registry cap (and the shutdown drain would re-write rows the archive already holds). Fix: the closed=True archival save is the durable record of the conversation as closed, so once it commits the close drops any recovery entry owed to that transcript via a transcript-scoped forget_recoveries_for_transcript(history_key).

Tests (both red-then-green, each confirmed non-vacuous):

  • test_app_token_regenerate_slot_replaced_during_save_releases_reservation — the reservation returns to its prior value after the aborted app-token regenerate; fails pre-fix with the counter stuck.
  • test_close_after_regenerated_reply_streams_drops_recovery_entry — closing a tab after the reply fully streamed leaves no recovery entry owed to the transcript; fails pre-fix (the entry leaks).

The existing test_close_mid_regenerate_keeps_order_and_pops_recovery_with_real_save was hardened to clear the process-global registry before its exact-equality baseline, since the new transcript-scoped drop would otherwise remove a sibling test's residual entry.

Gates green on the rebased tree: regenerate (116) + dashboard/contract/guardrails (1280) + history (608) suites pass; black (py310) / isort / flake8 / mypy + comment-history (enforced) + black-baseline clean. One commit, MERGEABLE.

@RohanK6

RohanK6 commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Addressed all four blocking findings at root on 8b1067bb0b. These are correctness fixes needed whichever way the architecture question resolves — if the recovery machinery stays, it must be correct.

F1 — reply identity cleared by a successor before the snapshot (chat_regenerate.py). The restore no longer reads the shared _turn_reply_mids_all after the fact. A turn-OWNED sink (slot._reply_mid_capture) is installed before dispatch and fed by _note_reply_row as each reply row is produced, inside the turn's own frame — so a queued follow-up's turn-start clear of _turn_reply_mids_all (reachable when _run_chat awaits the queue hand-off in its own finally) can no longer empty the identity the restore reads. The installer owns the sink's lifecycle and detaches it after its turn.

F2 — retry-exhaustion close deleting an unpaid recovery (slot_lifecycle.py → chat_regenerate.py). forget_recoveries_for_transcript now drops an entry ONLY when its rows are CONFIRMED PRESENT in the committed transcript (ConversationLog.rows_all_present, which reuses the exact idempotence matching of append_full_message_if_absent via a factored-out _full_row_present_on_disk). If partial-reply persistence and its retries failed and the archive holds only the partial, the entry is RETAINED for the shutdown drain rather than deleted — the previous reply is never lost. A read failure fails safe to retention.

F3 — rebind during eviction recovery authorizing the wrong transcript (chat_regenerate.py). After the eviction-recovery await (the rebind window), the binding is revalidated BEFORE the survivor save. A move to a DIFFERENT transcript strips the spliced rows off the rebound slot and recovers the previous reply to its ORIGINAL transcript, authorizing no write against the rebound one; a same-file respelling is carried through by pinning the save to the revalidated authorized identity. The post-save re-verification remains as a backstop for a rebind landing during the save-await itself.

F4 — unbounded deletion-generation retention (history.py). _deletion_generations is now FIFO-ordered on bump and bounded by _DELETION_GENERATIONS_MAX (1024, > the pending-recovery registry so every stem a live recovery could recheck survives). A stem still in the tombstone window is never pruned, so the fence a recovery rechecks is intact; the long tail of distinct deleted stems is pruned.

Tests (each red-then-green, verified non-vacuous by reverting the fix):

  • F1 test_regenerate_partial_reply_identity_survives_turn_mids_all_clear
  • F2 test_forget_recoveries_for_transcript_retains_unconfirmed_rows
  • F3 test_regenerate_rebind_during_eviction_recovery_pins_the_original_transcript
  • F4 test_deletion_generations_are_bounded_and_keep_tombstoned_stems

The turn-owned accumulator (F1) and the pre-save pin-to-authorized-transcript (F3) are designed together — both anchor a turn's restored rows to the identity captured in the turn's own frame rather than a shared, mutable slot field.

Gates: black/isort/flake8/mypy, black-baseline and comment-history (enforced) all green; regenerate, history, dashboard, composition-contract and guardrails suites pass (1670). One commit, rebased on current main, MERGEABLE.

@RohanK6

RohanK6 commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Fixed at root in 668af0311580b475f886d14841721e43f275def8.

Finding (correct): the F2 "retain unless rows confirmed present" change made _close_slot (a coroutine) call forget_recoveries_for_transcript → ConversationLog.rows_all_present, which takes the conversation lock and reads the full file via _read_messages, entering _locked on the event loop. Under strict/dev that raises OnLoopPersistError (swallowed by the except → entry silently never dropped); in prod it blocks the loop on the flock + full read, or raises HistoryLockTimeout under contention so the entry is kept. That violates no-sync-store-call-from-a-coroutine / no-blocking-call-on-event-loop.

Fix: new forget_recoveries_for_transcript_off_loop(history_key) — it computes the owed registry keys on the loop (cheap), then for each entry runs the presence read via await asyncio.to_thread(conversation_log.rows_all_present, key, rows) so _locked executes in a worker thread (_check_on_loop_persist_discipline sees no running loop → no-op), and does only the cheap registry pop back on the loop (re-reading the entry under the current tick so a concurrent confirm-clear can't double-pop). _close_slot now awaits this variant.

The F2 invariant is unchanged — an entry is dropped only once its rows are confirmed present in the committed transcript. A read that cannot confirm presence errs on the SAFE side: a raised exception or a not-yet-on-disk row retains the entry (present = False), so the previous reply is never dropped on an unreadable/partial transcript and the shutdown drain can still restore it. (The on-loop discipline error can no longer fire now that the read is off-loop; the except only ever sees genuine read failures, and those retain.)

Test (red→green, non-vacuous): test_forget_recoveries_off_loop_runs_presence_check_off_the_loop runs under KIROCREW_STRICT_ON_LOOP_PERSIST=1. It asserts the inline on-loop read rows_all_present(...) raises OnLoopPersistError (the pre-fix path), then that the off-loop variant does not raise and still drops the confirmed-present entry while retaining the absent one. Reverting the asyncio.to_thread to an inline call fails the test (dropped == 0 — the error is swallowed and nothing drops).

Rebased onto current main (resolved one _close_slot conflict with the Autopilot stage-boundary retirement). Gates: black/isort/flake8/mypy + black-baseline + comment-history; suites regenerate + dashboard + history + composition-contract + guardrails + session-control = all green. One commit, MERGEABLE.

@RohanK6

RohanK6 commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

@iamwhatever — I inverted the regenerate flow as the Design Review and First Principles reviews both asked, and removed the recovery machinery.

The change (head bb43d3bbcd):

  • The previous reply now stays live in the window and on disk while the new turn runs. The regenerate endpoint no longer deletes it or persists a truncated transcript up front — it just stashes the reply's variant chain and marks the old-reply rows.
  • When the new turn produces its first segment, _flush_segment swaps the old reply out in place (via replace_regenerate_target) and folds it onto the new reply as a variant — the same point variants already attach. A turn that ends before any content never removed the old reply, so it is a clean no-op. That is #14632 fixed at its source.
  • A regenerate whose re-sent message routes to a local (slash) command applies the same replace-on-first-content through _append_local_command_reply, so the old reply is never left beside a command reply.

Removed with the cause-level fix (every race they handled is now impossible, because the reply is never removed before its replacement is committed): the _PENDING_RECOVERIES registry, admission reservations and settle gates; the shutdown drain; the rebind write-back path; the close-time _await_regenerate_restore wait and the regenerate_restore_pending close refusal; the recovery_payload_too_large / recovery_capacity_exhausted 409s; and the new history.py surface (append_full_message_if_absent, deletion-generation / tombstone tracking) — history.py is back to its mainline state.

Footprint: 4 source files + 2 test files, a large net deletion (~+150 / −1700 in source). Tests that pinned the old truncate-up-front behavior were updated to the deferred-swap behavior, and new red-then-green tests cover the empty-turn no-op (test_regenerate_empty_turn_preserves_the_previous_reply), the first-segment replace (test_flush_segment_replaces_the_live_old_reply), and the local-command replace (test_local_command_reply_replaces_the_regenerate_target). The PR description is rewritten to match.

@RohanK6

RohanK6 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the GPT 6.1 review on f94194db9e at root on 0c65e33d0e (one commit, rebased on fresh main, MERGEABLE).

F1 (UPHOLD-FENCED) — chat_regenerate.py:201: a queued successor consumes the regenerate target. FIXED.

The hole was real: _run_chat dispatches a queued successor from its shielded end-of-turn tail (_start_next_queued_turn) before the regenerate wrapper's finally clears the markers. An empty regenerate with a queued message therefore let the successor's _flush_segment / _append_local_command_reply see the stale _pending_variants + replacing-markers and graft the old reply onto an unrelated reply.

Root fix — scope the replacement to the owning turn by generation:

  • The regenerate endpoint records slot._regenerate_replacing_generation = slot._turn_generation right after slot.task = task (the assignment that bumps the generation to the regenerate turn's own value).
  • Both consumption sites now require slot._regenerate_replacing_generation == slot._turn_generation before consuming the stash/markers. A queued successor is dispatched via slot.task = <successor>, which bumps the generation, so its flush finds a mismatch and leaves the old reply untouched — the regenerate wrapper's finally then clears the markers as the no-op it already was.
  • The finally also resets the generation to -1.

Regression test test_queued_successor_does_not_consume_the_regenerate_target: arms the markers at generation G, bumps the generation (the successor dispatch), then runs _append_local_command_reply and asserts the old reply is NOT removed, the successor reply carries no grafted variants, and the markers survive for the finally. Non-vacuous: without the generation guard the successor removes the old reply and grafts the variant chain.

F2 (FLAG, not upheld) — file_changes.py:500: a target row evicted into the on-disk frozen prefix is not removed by the live-window replacement. RESIDUAL, not fixed, with reason below.

The adjudication FLAGs this rather than upholding it, and I agree it is a defensible residual:

  • It requires ~_MAX_SLOT_MESSAGES (10,000) rows to be appended between the target reply and the regenerate's first _flush_segment before the reply is trimmed out of the live window (state.py message-bound trim). A single regenerate turn producing 10k streamed/tool rows before its first segment is not a path real usage reaches.
  • The outcome is a visible duplicate of the old answer sitting on disk next to the new one. No text is lost, nothing is corrupted, no credentials or permissions are involved — the user can see and switch to either.

Fixing it would mean deleting a specific row from the already-persisted on-disk frozen prefix by id and fixing up its counters/cache — i.e. re-introducing exactly the durable-history-surgery machinery (restore_full_rows_off_loop, deletion-generation tombstones, rows_all_present) that this inversion deleted at the Design and First Principles reviews' request. That machinery was the source of the whole edge-case class those lanes asked to remove. Trading a no-content-loss ~10k-row duplicate for the return of that class is the wrong call, so I am recording this as an accepted residual rather than reconstructing the apparatus.

If a maintainer disagrees, the surgical option is to extend the first-flush rewrite to drop the target id from the archived prefix; I'll implement it on request. The fence keeps F2 mechanically blocking until a repository writer records an override, which I cannot issue as the fork author.

Gates: black / isort / flake8 / mypy clean on the touched files; comment-history + black-baseline pass; regenerate suite (70) + dashboard_chat suite (895, incl. the new F1 test) green. Zero host/internal in code and this comment.

@RohanK6

RohanK6 commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Addressed GPT 6.1 F1 (BLOCKING, UPHOLD-FENCED) on head 657d5fc836.

Finding: on a single-segment regenerate the inverted flow removes the old reply row from the live window on first content, so the window ends one row shorter than the index boundary captured at turn start. _attach_turn_stats scanned slot.messages[boundary:], which is then empty, so the new reply's turn_stats (elapsed, credits, ttft_ms) was silently never persisted. A common path, not an extreme one.

Fix (root, as prescribed — select the destination by the turn's recorded reply ids, not the pre-replacement window index): _attach_turn_stats now scopes by the pre-turn tail row's id (turn_start_mid) via the existing _turn_rows helper and filters to this runner's recorded reply ids (_turn_reply_mids), exactly mirroring _flush_file_changes. The id walks to the correct boundary when the window index points past the window's end; the window index remains the fallback for pre-id transcripts. turn_start_mid is threaded from the same call site that already passes it to _flush_file_changes.

Added a non-vacuous red-then-green test (test_single_segment_regenerate_keeps_stats_on_new_reply): it reproduces the shrink (window length drops one short of the boundary, asserted) and confirms the new reply keeps its full turn_stats including ttft_ms. Gates green (black/isort/flake8/mypy, comment-history), regenerate + dashboard_chat + turn_stats suites pass.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — 🔴 BLOCK

UX-level review of 34c036e19315010dbc505c5abcd8330b7b93e2ec via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: BLOCK

cannot evaluate: missing a recording of Regenerate. The previous reply now stays on screen while the new one streams below it, then disappears in a hard swap when the turn ends.

Blockers

The old reply disappears without a transition when the turn ends.

  • Cause: turnActions.ts drops dispatch(truncateAfterIndex(uIdx + 1)), so the old reply stays mounted while the new reply streams in under it.
  • Mechanism: at the end of the turn the server sends chat_variant_switch. useWebSocket.ts:783 handles it with dispatch(refreshSlot(data.slot)), which re-reads the whole transcript. The old row unmounts and the new reply moves up into its place with a variant switcher. Nothing animates.
  • What the user sees: while the turn runs, two assistant answers sit one under the other, which looks like the agent answered twice. Then the reply they were reading disappears and the text below jumps up. The description admits this ("the two collapse into one reply with the variant switcher") but gives no reason for doing it without a transition.
  • Severity: happens on every Regenerate that produces a reply (frequent) × confusing, though no work is lost (friction) × every time.
  • Fix: add a "Regenerating — replaces the reply above" marker on the new row while it streams, and make the collapse visibly one element (dim the old reply, then animate it into the variant switcher). Or argue in the description why a hard swap is acceptable here.

Evidence gaps

  • During Regenerate (old reply plus new reply streaming below it): no image or recording; both evidence lists are empty. A screenshot of the mid-turn state closes this.
  • End-of-turn collapse into one reply with the variant switcher: this is a lens-13 state change and needs a recording (.webm/.gif attached, or committed under temp-screenshots/). The <!-- no-visual-delta --> claim doesn't match the visible change the description itself lists under Backwards compatibility.
  • Regenerate that ends empty (old reply kept, nothing collapses): needs a screenshot showing what the user sees when no new reply arrives.

[UX-REVIEWED] 34c036e

Description read

[DESCRIPTION-READ] 757e157dcfe478b53354461af45f42fd2e3c53d4a80e2304a1fa1945af13b241

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@iamwhatever

Copy link
Copy Markdown
Contributor

Head befe38f0b: rebuilt on current main (one commit, co-authored by @RohanK6). The replace step moved from the first segment flush to the turn's end. Dispositions for the open findings on a0a3bc2:

  • fixed GPT F1 (deferred rewrite loses ownership/close guards)

The replace write is now one save_slot_off_loop in regenerate_swap.settle_regenerate with expected_history_key + expected_slot_name captured at the endpoint and best_effort=False. A guarded write registers its future and honours the close fence; a refusal or error changes nothing in memory. No _pending_rewrite is armed, so no unpinned rewrite exists. Test: test_the_pinned_write_refuses_a_recreated_slot.

  • fixed GPT F2 (durable turn identities)

The ordinal is durable_row_count(rows_before_regenerate_target(...)): the rows being replaced are not counted, so the re-run draws the ordinal of the turn it repeats. Test: test_the_ordinal_rows_exclude_the_reply_being_replaced.

  • fixed Design Watch: crew-log turn numbering — same fix as F2; the stale comment is rewritten.
  • fixed Design Watch + First Principles item 1: dashboard truncation — the optimistic truncateAfterIndex and its rollback are removed from turnActions.ts; the settle step broadcasts chat_variant_switch, which refetches the window.
  • fixed First Principles item 5 / Design suggestion: turn_stats.py duplicate — settling at turn end keeps the window unchanged while stats attach, so turn_stats.py is untouched.
  • fixed First Principles item 7: _resumed_count reset now runs in the settle step after the window shrinks.
  • fixed Opus advisory: Stop mid-stream — _note_reply_row records partial replies, and the settle step adopts the stashed variants for them. Test: test_a_partial_reply_replaces_the_previous_one_and_keeps_it_as_a_variant.
  • fixed @iamwhatever review (restore task busy-check, snapshot test) — the restore machinery it reviewed no longer exists; nothing is restored because nothing is removed early.

@iamwhatever

Copy link
Copy Markdown
Contributor

Dispositions for b2deb2c findings, fixed on 840544ed9:

  • fixed GPT F1 regenerate_swap.py:126 (frozen-prefix boundary)

settle_regenerate reads slot._disk_older_count in the same synchronous stretch as the snapshot and passes it as expected_disk_older_count; save_slot_off_loop now forwards it to _save_slot_to_history, which refuses on drift. Test: test_a_front_trim_during_the_write_refuses_it.

  • fixed GPT F2 regenerate_swap.py:68 (unbounded reply record)

RegenerateTarget.reply_mids holds minted row ids, capped at _MAX_SLOT_MESSAGES; the first reply is resolved from the live window. Test: test_the_reply_record_holds_bounded_ids.

  • fixed (description) target=design Watch 1 (transcript during a Regenerate)

The Backwards compatibility section now states what shows during a Regenerate. Media is pending: this host has no browser that can render the capture fixture.

  • accepted target=design Watch 2 (crash mid-turn keeps two replies)

Named in Backwards compatibility as an accepted outcome: nothing is lost and both rows render as ordinary assistant rows.

  • rebutted Opus advisory chat_regenerate.py:80 (cold-session replay includes the old reply)

A warm session already carries the old reply in the provider's own history on main, so the cold replay now matches the warm path; the regenerate hint tells the model not to reference it.

Regenerate deleted the previous reply and rewrote the transcript before the
new turn ran, so a turn that ended with no reply (refused, cancelled before
output, failed) lost the reply for good.

The endpoint now writes nothing. It arms a RegenerateTarget naming the old
reply rows, pinned to the transcript key and slot it authorized. At the
turn's end (_end_turn_tail, before the queue hands off) settle_regenerate
removes those rows and keeps the old reply as a variant only when the turn
recorded a reply, through one pinned rewrite; an empty turn leaves the
reply in place. The crew-log ordinal excludes the rows being replaced, so a
regenerate stays a retry of the same turn. The dashboard no longer hides
the reply optimistically.

Fixes kirodotdev#14632

Co-authored-by: Rohan Khanderia <51836404+RohanK6@users.noreply.github.com>
@RohanK6

RohanK6 commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #23769, which re-applies this fix as one commit on current main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Regenerate permanently deletes the previous reply when the new turn ends before its first message

2 participants