Skip to content

fix(history): sync rotation and compaction replacements, tidy failed archives - #20060

Closed
jeeshofone wants to merge 2 commits into
kirodotdev:mainfrom
jeeshofone:fix/19362-archive-durability-followup
Closed

jeeshofone wants to merge 2 commits into
kirodotdev:mainfrom
jeeshofone:fix/19362-archive-durability-followup

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A power loss during any rewrite that replaces a whole transcript cannot lose the rows that file keeps, the channel transcript migration cannot delete the only other copy of a conversation before its merge is on disk, and a lasting archive fault no longer leaves one archive copy per retry. The hot metadata writes pay for that sync without holding the session lock through it.

From a post-merge review of #19362.

  • Rotation and compaction replaced the transcript unsynced. On XFS or btrfs a power loss can bring it back empty.
  • The metadata-only rewrites change only line 0 but rename the WHOLE file unsynced, so the risk is the conversation, not the marker.
  • The channel transcript migration wrote its merge unsynced, then delete_session'd the orphan, which held the only other copy.
  • _archive_lines left its file behind when its sync failed, one copy per retry.
  • _frozen_prefix_and_foreign_appends accepted archive_dedup=None, dropping dedup lines unarchived.

Why it matters

These run unattended, and mark_consolidated runs on every consolidation pass, so its lock hold matters as much as its durability.

Not a goal

  • No change to the dashboard save, which already syncs.
  • No F_FULLFSYNC on macOS: repo-wide atomic_write policy.
  • Rotation, compaction, clear_closed and the _update_metadata_locked callers keep the sync under the lock: their output, guard or after-commit step needs it, and all are far rarer than the three hot paths.

What changed (motivation → approach → change)

  • history_rewrite.rewrite_transcript(path, content): atomic_write(..., fsync=True) with no fsync parameter. Rotation, compaction, clear_closed, _update_metadata_locked, the selection rollback and the migration merge use it.
  • history_rewrite.rewrite_metadata_line(log, key, build), for mark_consolidated, the sub-offset write and update_metadata: build runs on an unlocked read; the result is staged and synced (atomic_write.stage_replacement) before the lock. Under the lock it is renamed over only if the path for the key and the file's bytes are exactly what build read (no stat shortcut: these writers restore the old mtime). Any difference discards the staged file and rebuilds under the lock, so nothing that landed in between is lost. On that fallback build runs twice and the sync runs under the lock, and mark_consolidated and the sub-offset write log their messages twice when an append lands in between (accepted). The directory is synced after release, best effort. An interrupt after published is set and before publish_replacement enters its own try is caught (except BaseException), which discards the staged copy and re-raises.
  • atomic_write._stage_temp is the one staging sequence for atomic_write and stage_replacement. Its close runs inside the protected region after the descriptor's ownership is cleared, and a failing cleanup close is swallowed, so a failed write, fsync or close removes the temp file. The by-name publish (publish_replacement) runs inside try/except BaseException with _unlink_temp, so an interrupt before its own try removes it too.
  • After a first-line rewrite, rewrite_metadata_line and _update_metadata_locked call carry_index_over_head_rewrite (fix(history): persist the transcript page index and carry it across first-line rewrites #24295), so the next history page does not frame the whole transcript again. The marker and cleared-close writes pass bytes, so a Windows write keeps LF line ends.
  • update_metadata's merge rules move to _metadata_replacement, shared with _update_metadata_locked; it still writes bytes.
  • channel_transcript_migration._migrate_one: fsync_dir runs right before every orphan delete, including an unchanged retry. A failed sync keeps the orphan.
  • _archive_lines: a failed write or sync unlinks the file it created before re-raising.
  • transcript_merge.py: archive_dedup required.
  • Files: atomic_write.py, history_files.py (_archive_lines), history_marker.py (the two marker writes), history_projection.py, session_agent_selection.py, history.md, plus the modules named above.

Lock hold on XFS, median of 15, mark_consolidated / update_metadata: 100 KB 3.3 / 3.4 ms → 0.2 / 0.2 ms, 1 MB 8.7 / 8.5 → 0.4 / 0.4 ms, 10 MB 46.1 / 44.9 → 2.5 / 2.4 ms (sync under the lock, fbbfbef → sync before it, 67ff65a; this locked path is 67ff65a's).

Backwards compatibility

Compatible: no file format or return value changes.

Reader: src/kiro_crew/dashboard/slot_persistence/transcript_merge.py:compose_payload -- chat -- the only production caller already passes archive_dedup; test_archive_fails_closed.py::TestForeignDedupFailsClosed::test_the_save_archives_dedup_drops_through_the_same_path
Reader: src/kiro_crew/dashboard/session_control_parts/session_create.py:create_session -- chat -- update_metadata's hand-rolled unsynced temp-and-rename is replaced by rewrite_metadata_line with the same upsert and byte output; test_history_locking_remediation.py::TestReducedLockHold::test_update_metadata_still_upserts_a_missing_transcript
Reader: src/kiro_crew/history_rewrite.py:HistoryRewriteCoordinator -- chat -- rotation and compaction lose the unsynced default, not their output; test_archive_fails_closed.py::TestTheTranscriptReplaceIsSynced
Reader: src/kiro_crew/channel_transcript_migration.py:_migrate_one -- release -- _write_merged no longer syncs the directory itself; _migrate_one syncs it before every delete; TestTheChannelMigrationSyncsBeforeItDeletesTheOrphan::test_an_unchanged_retry_after_a_failed_directory_sync_keeps_the_orphan
Reader: test/test_history_locking_remediation.py:TestReducedLockHold -- chat -- the pin is restored (fbbfbef had replaced it), now asserting no file fsync under the session lock

Tests

  • TestReducedLockHold::test_no_file_fsync_while_the_session_lock_is_held[3 writes]: one file fsync, at lock depth 0, and one best-effort directory sync after release. Red on fbbfbef.
  • TestReducedLockHold::test_a_write_between_the_sync_and_the_rename_is_kept[3 writes × append|edit]: the injected write survives and the staged file is removed. Red on fbbfbef.
  • test_an_unchanged_retry_after_a_failed_directory_sync_keeps_the_orphan: red on fbbfbef.
  • test_atomic_write_staging_cleanup.py (3 writers × close|fsync|write): no .tmp remains and the target is untouched or edited under the lock. 8 of 9 red on c969570.
  • The staging cleanup cases assert one close of the staged descriptor. test_a_failed_pinned_rename_removes_its_temp_and_keeps_the_original pins the pinned rename cleanup.
  • TestReducedLockHold: test_a_same_length_edit_that_restores_the_mtime_is_kept (only the bytes differ), test_a_key_that_moves_before_the_lock_is_rewritten_at_its_new_path (a hard link), test_a_crlf_transcript_keeps_one_line_ending, each × 3 writes. test_an_interrupt_before_the_publish_starts_removes_the_temp_file is red on 67ff65a.
  • TestReducedLockHold::test_an_interrupt_as_the_publish_starts_removes_the_staged_copy[3 writes]: a KeyboardInterrupt as publish_replacement is entered leaves no staged copy and the transcript unchanged. Red on 5a8f238. test_an_interrupt_after_the_rename_keeps_the_published_edit[3 writes]: the edit and both rows survive an interrupt after the rename.

Mutations KILLED: no byte re-check (10), always the locked path (3), migration dir sync only after a write (1), close after the protected region (3), unguarded cleanup close (6), fd ownership kept across the close (2), no cleanup on a failed pinned rename (1), no path check (3), no CRLF normalisation (3), publish outside the cleanup try (1), no directory sync after release (3), no discard on an interrupted hand-off to publish_replacement (3).

Run locally on this head (rebased onto 3ecd50f): the scoped backend suite for this diff (16,762 passed, 1 failed: test_snapshot_merge_core_files_pinned.py::test_the_rewrite_keeps_the_live_stores_access_control_acl, where this host's setfacl rejects u:nobody:--- on any file); the history page-index, locked-section, archive, atomic-write, migration and persistence files (197 passed; removing the page-index carry turns 3 test_history_page_index_persist.py cases red); black gate, isort, flake8, mypy src/kiro_crew/, comment-history, subprocess-encoding, docs lint, feature map, brand name, harness parity.

Run locally on an earlier head (base 6362ff1): 68 history, migration, archive, atomic-write and caller files (5174 passed, 1 skipped); 59 ratchet and facade files (5915 passed); flake-pattern ratchet (58 passed); black gate, isort, flake8, mypy src/kiro_crew/, comment-history, subprocess-encoding, docs lint, feature map; Semgrep 1.78.0 with semgrep/ (0 on our lines). macOS Tests is path-skipped here.

Manual verification

N/A — unit coverage sufficient: the fault paths are reachable only by injection.

Related Issues

Follow-up to #19362.

Part of #19361

Pattern harvest

Rule candidate: review-prompt
Pattern: a whole-file replace fsyncs even for a one-line edit, before any shared lock, renaming under it only if the file is unchanged. Deleting the other copy syncs the directory first. A temp file's close stays inside its cleanup region.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 14, 2026 01:59
@jeeshofone
jeeshofone requested a review from mbriones98 October 14, 2026 01:59
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision a831312d0425992d3174c7ba93e41527fb9d2b1a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of a831312d0425992d3174c7ba93e41527fb9d2b1a via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix is sound and proportionate. Staging and syncing outside the lock, then publishing only if the bytes match, keeps the old lock-hold pin while adding durability.

Notes

  • The description says "After a first-line rewrite, rewrite_metadata_line and _update_metadata_locked call carry_index_over_head_rewrite (fix(history): persist the transcript page index and carry it across first-line rewrites #24295)", but no such call exists in the diff or the base tree. Add the call, or drop the claim.
  • The match check reads the whole transcript twice on the hot path (once unlocked, once under the lock). At 10 MB that read under the lock is now the main part of the 2.5 ms hold. This is acceptable as it stands, but it is the next thing to cut if mark_consolidated frequency grows.

[DESIGN-REVIEWED] a831312

Description read

[DESCRIPTION-READ] f77b3d3315b610d13bb365560d026ac653d314dc70f40a4ac09f5094a4aeba86

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of a831312d0425992d3174c7ba93e41527fb9d2b1a via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. One gap: the description claims a carry_index_over_head_rewrite call that the diff does not make.

Notes

What this change ships

Inventory (9 items) — 8 justified

Intent: a power loss or a lasting disk fault can no longer lose or duplicate a conversation's rows. FIX.

  1. Rotation and compaction replacements are synced before the rename — justified
  2. Metadata-only rewrites (consolidation marker, sub-offset, update_metadata, clear_closed, selection rollback) are synced, because each one replaces the whole file — justified
  3. The three hot metadata writes stage and sync before taking the lock, rename only if the bytes are unchanged, and otherwise rebuild under the lock — justified
  4. The channel migration syncs the directory before every orphan delete, retries included — justified
  5. A failed archive write or sync removes its file instead of leaving one copy per retry — justified
  6. archive_dedup is now a required argument, so dedup lines can no longer be dropped unarchived — justified
  7. atomic_write staging is split into _stage_temp plus stage_replacement, publish_replacement and discard_replacement (1, 2 and 1 consumers) — justified
  8. The old "metadata rewrite must not fsync" pins are replaced by "no file fsync while the lock is held" — justified
  9. A first-line rewrite carries the history index over — undeclared gap: claimed but not in the diff

[FIRST-PRINCIPLES-REVIEWED] a831312

Description read

[DESCRIPTION-READ] f5c946fa1d7c439f5de29580e46880bc876f8c567f6b904df606d2ccb7018898

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed a831312d0425992d3174c7ba93e41527fb9d2b1a via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a831312

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 95312683ef927fb547404c1ba181dd625a0874cb via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9531268

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 14, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Dispositions for head fbbfbef (review of 5b6adeb plus the First Principles note on 5b6adeb).

SHOULD-FIX A (metadata-only rewrites replace the whole transcript unsynced): fixed. New history_rewrite.rewrite_transcript(path, content) always writes with fsync=True and has no fsync parameter. mark_consolidated, the consolidation sub-offset, update_metadata, clear_closed and the agent-selection rollback all go through it, as do compaction and rotation. The sweep covered every atomic_write(, atomic_write_at(, replace_with_retry( and os.replace( under src/kiro_crew. The other transcript writers (dashboard save, import prefix, image repair) already sync. Pinned by TestEveryWholeTranscriptReplaceIsSynced (one recording test per site, red on 5b6adeb) and a source pin that no transcript module contains fsync=False. The old TestReducedLockHold pinned the opposite trade and is replaced by TestMetadataRewritesAreSynced.

SHOULD-FIX B (channel migration deletes the orphan after an unsynced merge): fixed. _write_merged now writes through rewrite_transcript and then calls fsync_dir(path.parent), both before delete_session. A failed sync raises and the orphan stays. Pinned by TestTheChannelMigrationSyncsBeforeItDeletesTheOrphan, which records write, dir sync and delete in that order, and checks that a failed dir sync keeps the orphan.

NIT C (except BaseException unpinned): fixed. test_an_interruption_inside_the_write_leaves_nothing[KeyboardInterrupt|CancelledError|SystemExit]. Narrowing to except Exception is now KILLED.

NIT D (macOS F_FULLFSYNC): out of scope. One line under Not a goal and one in the history spec: this is repo-wide atomic_write policy.

NIT E (best-effort removal): fixed. The history spec says the removal is best effort, a failed removal is logged, and the sync error is still raised.

First Principles note (rotation's fsync=True runs inside append's lock): accepted, with numbers. This head widens the same trade to the metadata writes, so it applies there too. The lock is the per-session lock, so only writers of that one session wait. Measured on XFS, the filesystem without auto_da_alloc that this change protects, a whole-file replace costs:

  • 100 KB: 0.4 ms median unsynced, 3.1 ms synced
  • 1 MB: 1.6 ms unsynced, 6.2 ms synced
  • 10 MB: 12.5 ms unsynced, 19.7 ms synced

Rotation itself already reads and rewrites up to 10 MB under that lock. The alternative is a power loss returning an empty transcript, so the conversation is lost rather than a marker. A slow disk raises both columns, but it does not change which side of the trade is right. A sidecar for the marker would avoid the whole-file rewrite, but it changes the format and every last_consolidated reader, so it is noted in the body as the route if that latency ever matters.

Mutations, each KILLED on fbbfbef:

  • dropping the sync inside the helper
  • an unsynced write at each of the five metadata sites
  • dropping the migration's fsync_dir
  • except Exception in the archive cleanup

The 5b6adeb mutations stay killed. Backend Tests (3.12, 4) on 5b6adeb was cancelled at the job cap while still passing tests (its progress artifact shows no failure). This head reruns it.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Correction to the first-pass line in the comment above, for head fbbfbef. Two R8 gaps remain:

  • The body is 10,995 bytes, over the 8,000 bytes the review lanes read. Everything the lanes judge on comes first and is inside the cut: Goal, the problem list, Not a goal, What changed, the "Why not" lines, the compatibility note with the latency numbers, and the new tests. Past the cut are the older 5b6adeb test bullets, the full mutation list (repeated in the comment above), the local run list and the Pattern harvest. The template check (pr-description-check.sh) passes on the full body. I have not trimmed the body, because a body edit after this push would cancel the lanes on this head.
  • No per-rule list of the review rulebook was made for this round. The diff is sync calls, one helper, tests and docs. It adds no new state, no await points and no caps.

Everything else in R1-R8 holds:

  • every hunk maps to the Goal;
  • the helper replaces the hand-rolled temp-and-rename in update_metadata and does not sit beside it;
  • no sensitive path or audit id is touched;
  • local gates are green on this exact head;
  • main does not touch these files.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 14, 2026
@jeeshofone
jeeshofone force-pushed the fix/19362-archive-durability-followup branch from fbbfbef to fff1062 Compare October 14, 2026 05:45
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Dispositions for head fff1062 (lanes on fbbfbef).

GPT 6.1 F1 (channel_transcript_migration.py:349, unchanged retry skips the directory sync): fixed. fsync_dir(channel_path.parent) moved out of _write_merged into _migrate_one, right before delete_session, so it runs on every deleting pass, including a retry whose merge is byte-identical and skips the write. A failed sync raises and the orphan stays. Pinned by test_an_unchanged_retry_after_a_failed_directory_sync_keeps_the_orphan: directory-sync EIO after the rename, then a same-boot removing pass with no write recorded keeps the orphan; a pass with a working sync then removes it. Red on fbbfbef (the second pass deleted the orphan). Mutation "sync only after a write" is KILLED.

Design CONCERNS (fsync under the session lock on every metadata write, reversed TestReducedLockHold): fixed by taking the lane's suggestion. mark_consolidated, the consolidation sub-offset and update_metadata now go through history_rewrite.rewrite_metadata_line. It builds the edit on an unlocked read, then stages and syncs the replacement (atomic_write.stage_replacement) before taking the lock. Under the lock it renames only if the file still holds exactly the bytes the edit was built from, and the key still resolves to the same path. On any difference it discards the staged file and rebuilds under the lock. The check compares bytes, not a size+mtime stamp, because these writers restore the previous mtime, so a same-length line-0 edit by another writer would pass a stamp. The directory is synced after the lock is released, best effort, because the rename has already happened.

TestReducedLockHold is restored with its original intent. It records the lock depth at every regular-file fsync: exactly one per write, at depth 0, red on fbbfbef (depth 1). It also injects an append or another metadata edit between the staged sync and the lock: the late row or field survives and the stale staged file is removed. Lock hold measured on XFS, median of 15, for 10 MB: 46.1 ms on fbbfbef, 2.5 ms on this head (mark_consolidated). At 100 KB: 3.3 ms vs 0.2 ms. Table in the body. The flush is still paid, by the caller, outside the lock. That covers the Windows and slow-disk worry, because a slow FlushFileBuffers no longer extends anyone else's wait.

Rotation, compaction, clear_closed and the guarded _update_metadata_locked callers keep the sync under the lock. Rotation and compaction archive first, and their output depends on the whole file under the lock. The others run a guard or an after-commit step in the same critical section. The body's Not a goal says so. Mutations KILLED: renaming without the byte re-check (6 red), always taking the under-lock path (3 red).

PR Hygiene (take-away Reader lines, body over 8,000 B): fixed. ## Backwards compatibility now carries five Reader: lines: compose_payload, the update_metadata caller in session_control.py, HistoryRewriteCoordinator, _migrate_one, and the restored TestReducedLockHold. The body is 7,993 B. take_away_check.py and pr-description-check.sh pass locally. The body was edited before this push.

E2E (stub ACP backend, offline) on fbbfbef: not ours. test_dashboard_playwright_suite "Playwright ran 1020s, past the 1020s bound". This PR touches no website file. Main-side, tracked as #18185 and #18632. This head is the retry.

Local on fff1062:

  • 42 history, migration, consolidation, archive, persistence, transcript and atomic-write test files: 1968 passed, 1 skipped.
  • 70 other test files calling the three writes: 5891 passed, 5 xfailed.
  • Ratchets plus the facade test: 283 passed.
  • black on changed hunks, isort --check-only, flake8, mypy on changed sources: clean.
  • check_comment_history, check_feature_map, check_subprocess_encoding, docs_lint: pass.
  • Semgrep 1.78.0 with semgrep/ on the changed files: 3 findings, all on lines main already has, 0 on ours.

First-pass check: R1-R8 OK. R3.1 (other writers): this round's new state is the staged file and the snapshot. The lock-held byte comparison checks them against every other writer and so plays the compare-and-swap role. The interleaving test pins it.

@jeeshofone
jeeshofone force-pushed the fix/19362-archive-durability-followup branch from fff1062 to c969570 Compare October 14, 2026 11:54
@jeeshofone

jeeshofone commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author

New head c969570, rebased onto current main (d06b611). Content change is wording only.

  • Comment History Gate: the docstring in test/test_history_locking_remediation.py that said the staged copy "no longer matches" now states the present behaviour ("does not match"). One more narration in history_projection.py (Bytes, as before,) is reworded the same way. The gate passes against both the merge-base and main's tip, and a sweep of the PR's added comments and docstrings for the gate's other markers found nothing further.
  • Await Fast Gate, Dependency License Gate and Sensitive Change Review failed on the previous head from the comment-history failure and a GitHub rate limit, so none of the heavy jobs ran there. This head runs them.
  • Local results on this head: the history, atomic_write, channel migration, history_projection and history_rewrite test files pass (1,285 tests across the history family, including TestReducedLockHold); isort, flake8, mypy, the baselined black gate, the subprocess-encoding gate and the harness-parity gate pass; Semgrep 1.78.0 with the repo rules reports nothing on lines this PR adds.

First-pass check: R1-R8 OK. The PR body is unchanged, since no claim in it changed.

@jeeshofone
jeeshofone force-pushed the fix/19362-archive-durability-followup branch from c969570 to 67ff65a Compare October 14, 2026 12:59
@jeeshofone

jeeshofone commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Dispositions for head 67ff65a (lanes on c969570).

GPT 6.1 F1 (atomic_write.py:1384, a failing success-path close leaves the staged copy): fixed at class level. The staging sequence now exists once, as atomic_write._stage_temp, and both atomic_write (by-name and pinned routes) and stage_replacement call it. Inside it, ownership of the descriptor is cleared before the close, and the close runs inside the protected region, so a close that reports an error removes the temp file before raising. The cleanup close for an orphaned descriptor is now guarded too: before, a write or fsync failure whose cleanup os.close also failed raised out of the except and skipped the unlink, in stage_replacement and in atomic_write alike. publish_replacement (now also atomic_write's by-name publish) and the pinned rename unlink the temp file on a failed rename, as before. rewrite_metadata_line relies on stage_replacement cleaning up when it raises, and now it always does.

Pinned by test/test_atomic_write_staging_cleanup.py, 9 cases: stage_replacement, synced atomic_write and update_metadata, each with an injected EIO on close, fsync or write, where the write and fsync cases also fail the cleanup close. Each asserts that no .tmp file remains and the target is untouched. For update_metadata, the edit lands through the under-lock rewrite with both rows intact. On c969570, 8 of 9 are red. atomic_write[close] already passed there, because atomic_write cleared the descriptor before closing; only the copy in stage_replacement did not. Mutations KILLED: moving the close after the protected region (3 red), an unguarded cleanup close (6 red).

First Principles item 3 (stage_replacement duplicates atomic_write's by-name body) and the Design suggestion: fixed by the same change. The duplicate body is gone. atomic_write's by-name fsync=True route is now _stage_temp + publish_replacement, and stage_replacement is _stage_temp with that route's defaults. Mode, data-home directories, newline handling and cleanup now live in one place.

SAST (Semgrep) on c969570: infra. The job hit its 15-minute execution limit, and this head re-runs it. Locally, Semgrep 1.78.0 with semgrep/ on every changed file finds 3 results, all on lines main already has (test_channel_transcript_migration.py:198, test_history_locking_remediation.py:1695 and :2597), and 0 on ours.

Earlier rulings for this PR are unchanged and still in: the whole-transcript writes go through rewrite_transcript/rewrite_metadata_line; the migration syncs the directory before every orphan delete, including the unchanged retry; TestReducedLockHold is restored; the body has the take-away Reader: lines and is under 8,000 B (7,995 B, edited before this push).

Local on 67ff65a:

  • 51 history, migration, consolidation, archive, transcript and atomic-write test files, including test_history_locking_remediation.py::TestReducedLockHold: 2471 passed, 1 skipped.
  • 261 other test files that use atomic_write: 30629 passed, 25 failed. The failing 25 (test_design_tweak_backend.py, test_cli.py, a setfacl test, one skill-projection test) fail the same way with c969570's atomic_write.py swapped back in, so they come from this host's environment.
  • black, isort --check-only, flake8, mypy on the changed sources; the black and subprocess-encoding gates; check_comment_history against the merge base; pr-description-check.sh and take_away_check.py on the body: all pass.
  • No rebase: main moved 2 commits, none of them touching this PR's files, and the merge is clean.

First-pass check: R1-R8 OK. R0.3: both new invariant lines (the in-region close, the guarded cleanup close) are mutation-pinned above. R1: the staging change subtracts a duplicate mechanism (prefer-subtracting-a-duplicate-mechanism).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 14, 2026
@jeeshofone
jeeshofone force-pushed the fix/19362-archive-durability-followup branch from 67ff65a to 9502aa6 Compare October 14, 2026 15:12
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 06789c5.

The fix for @bolichen97's blocking finding looks right to me. Once published is set, an outer except BaseException discards the staged copy and re-raises (history_rewrite.py#L126-L135). Discarding a name the rename already moved is a no-op, so a published edit is never removed. The interrupt tests cover all three hot metadata writers (L335, L366). All lanes are clean on this head and CI is green.

Blockers:

  • @bolichen97's change request (review) is still active. It needs their re-review.
  • The PR changes data-deleting paths, which are reserved for a human: the failed-archive unlink in _archive_lines, the orphan delete in the channel migration, and the staged-file cleanup in atomic_write.py.

What clears it: @bolichen97 re-reviews the fix and approves or dismisses the request, and a maintainer approves the deletion paths.

— crew-pr-reviewer (r123)

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 26, 2026
@jeeshofone
jeeshofone force-pushed the fix/19362-archive-durability-followup branch from 06789c5 to 4f7b520 Compare October 28, 2026 03:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 28, 2026
@jeeshofone
jeeshofone force-pushed the fix/19362-archive-durability-followup branch from 4f7b520 to 089a815 Compare October 28, 2026 04:02
@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • First Principles item 3: history_rewrite._stat_or_none copies history_projection._stat_or_none

fixed in 089a815: history_projection imports the one helper from history_rewrite and its copy is deleted (def _stat_or_none now has 1 definition).
The second copy came from the rebase onto #24295. That change added the helper to history_projection while this PR moved the first-line writers into rewrite_metadata_line, which needs the same stat for the page-index carry.
Pinned by the existing test_history_page_index_persist.py cases, which stay green with the carry in place and go red (3) when the carry is removed.

@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • Design note: every hot metadata write reads the whole transcript twice

accepted, as in the earlier rounds: the unlocked read is what lets the sync run off the lock, and the locked read is the byte compare that keeps a racing append from being lost. Main read the whole file under the lock once, so the locked work is unchanged and the added read is outside it.
The lock-hold numbers in the body include both reads. Pinned by TestReducedLockHold::test_a_write_between_the_sync_and_the_rename_is_kept.

@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • Design note: under steady appends the fallback rebuilds and syncs under the lock, so the lock-hold savings shrink

accepted, and the body says so: the fallback runs only when a write lands between the unlocked read and the lock. It then runs build twice and syncs under the lock, which is what the earlier sync-under-the-lock design paid on every call, and it loses no row or edit.
Pinned by TestReducedLockHold::test_a_write_between_the_sync_and_the_rename_is_kept (append and metadata edit, x3 writes). The staged copy is freed before the locked rewrite (test_the_fallback_frees_the_stale_staged_copy_before_writing).

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 28, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 089a815.

I found no defect on this head. The fix for @bolichen97's finding is still in place: once published is set, the outer except BaseException discards only an unpublished staged copy (history_rewrite.py). The new First Principles and Design notes are answered, the duplicate _stat_or_none is removed, CI is green, and every AI lane passes or skipped.

Blockers:

  • @bolichen97's change request at 5a8f238 is still active and needs their re-review.
  • The PR changes data-deleting paths, which stay with a human reviewer: the failed-archive unlink in _archive_lines, the ordering before the orphan delete, and the staged-file cleanup in atomic_write.py.

What clears it: @bolichen97 approves or dismisses the request, and a maintainer approves the deletion paths.

— crew-pr-reviewer (r131)

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 29, 2026
@iamwhatever
iamwhatever force-pushed the fix/19362-archive-durability-followup branch from 089a815 to 9531268 Compare October 29, 2026 17:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 29, 2026
jeeshofone and others added 2 commits November 1, 2026 21:46
…cements

Rotation, compaction, clear_closed, _update_metadata_locked, the selection
rollback and the channel transcript migration merge now replace the whole
transcript synced (history_rewrite.rewrite_transcript). The three hot
metadata writes (mark_consolidated, the sub-offset write, update_metadata)
stage and sync the replacement before taking the session lock and rename
under it only if the file is unchanged (rewrite_metadata_line). A failed
archive write or sync removes its file, and archive_dedup is required.

Rebased onto main after kirodotdev#21014 split history.py: the _archive_lines change
lands in history_files.py and the marker writes in history_marker.py; the
TestReducedLockHold pins move to
test_history_locking_remediation_locked_section.py after kirodotdev#21783.

Co-authored-by: iamwhatever <iamwhatever@users.noreply.github.com>
…ebuild

GPT 6.1 on 185dd9b: when a write lands between staging and the lock,
rewrite_metadata_line kept the stale staged file until its finally while
rewrite_transcript wrote a second whole copy, so a disk with room for one
replacement failed the edit with ENOSPC. Discard it first. Pinned by
test_the_fallback_frees_the_stale_staged_copy_before_writing (3 writes;
red without the fix).

Also keep atomic_write and replace_with_retry on the history_projection
facade: test_campaign_facades_base_public_names pins every public name the
base facade served.
@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • Design note: count how often the fallback rewrite runs. No change in this PR. The fallback is the correctness path when a write lands between staging and the lock, and its cost is one extra flush on that edit only. A counter would add a metric with no reader today; if a production signal is wanted, it belongs in a follow-up that adds it alongside the other history write timings.

The next push is a rebase onto main only. Main removed the page-index carry after first-line rewrites (the index is now checked by content), so the carry calls this PR had added in history_rewrite.py and history_projection.py are dropped with it. No behaviour of this PR changes.

@jeeshofone

jeeshofone commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=14be12c8abdc F1 (downgraded): leased consolidation still syncs under the transcript lock. Agreed with the adjudication, no change here. On the leased path the outer lease fence already holds the session lock when the marker writer runs, so staging cannot happen outside it on that path. The write stays correct; only the lock-hold saving does not apply to leased consolidation. Moving the stage ahead of the fence needs a lease check threaded into the locked compare-and-publish across three modules, which is a larger contract than this fix should carry.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Moved to #27514: same commits, now on a branch of this repo so CI uses the CodeBuild fleet. Review history stays here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants