Skip to content

fix: dedupe overlapping archive segments in the rotated corpus - #8488

Closed
buluoray wants to merge 1 commit into
mainfrom
fix/archive-segment-overlap
Closed

buluoray wants to merge 1 commit into
mainfrom
fix/archive-segment-overlap

Conversation

@buluoray

@buluoray buluoray commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Closes #8484. Split out of #7916 under a human override: the finding was real, but low-probability, so it did not hold that PR.

The seam

read_rotated_messages concatenated archive segments with a bare rows.append(row) and no cross-segment identity check. drop_persisted_tail_prefix guards the archive↔live seam — it cannot reach this one, because the duplicate is already inside rotated before that comparison runs.

How two archive segments come to overlap

HistoryRewriter._maybe_rotate writes the archive before rewriting the live file, deliberately: until the archive lands, the live file is the only copy of those rows, so archiving is a precondition of the rewrite rather than a best-effort side effect.

A hard crash between those two steps leaves dropped in both places:

  1. archive segment N is written;
  2. the process dies before the live-file rewrite;
  3. the live file still begins with those same rows;
  4. the next rotation computes dropped' = message_lines[:-keep_count], which starts with that same prefix, and archives it again as segment N+1;
  5. segments N and N+1 overlap, and the reader concatenates both.

The header-damage branch a few lines above already named this hazard — "a retry rotation can archive the same rows successfully, and then a partial read of this segment duplicates them" — but nothing guarded it.

Why a duplicate is worse than showing a row twice

read_rotated_messages is the pagination index space: before / next_before cursors and the fork index path both resolve a rendered row's position against it. A duplicated row shifts every index above it, silently. A reader pages positions that no longer mean what they meant, and an index-addressed fork copies a different cutoff than the one on screen. Nothing raises.

The fix

Accumulate each segment's rows, then apply drop_persisted_tail_prefix against the corpus so far before extending. Same rule, same helper, one level down — no new identity logic. Both producers persist a prefix, never an interior slice, so the duplication is always a segment's head against what the corpus already ends with, which is exactly the shape that helper was written for.

Test power — verified in both directions

test/test_archive_segment_overlap.py, 6 cases. Three pin the fix and three pin its limits, so the suite fails if the guard is missing or too greedy:

mutation reddens
drop the dedup (rows.extend(seg_rows)) wholly_duplicated → ['1','2','1','2','3'], partially_duplicated → ['1','2','2','3'], overlap_matched_by_stable_id
widen the identity rule to (ts, role) only distinct_rows_sharing_ts_and_role_are_both_kept

The three upper-bound cases stay green under the first mutation by design — they assert the guard does not eat rows — which is why the second mutation is what proves they are not vacuous.

non_overlapping_segments_are_all_served keeps a healthy archive intact; an_interior_repeat_is_not_treated_as_overlap pins that a row reappearing mid-segment is the reader's own repeated message, not a re-archived prefix.

Cost

The helper's all(...) short-circuits on the first mismatch, so the healthy no-overlap path costs one comparison per candidate offset, and the whole read is cached on the segment files' stat signature — it runs once per archive change, not once per page.

Checks run locally

pytest on the new file plus the five adjacent archive/fork/pagination suites (51 passed), isort, flake8, mypy --platform linux, and the black gate. The full suite is left to CI.

Why no screenshot: backend-only change to a transcript read projection. There is no user-visible surface in this diff — the observable effect is the absence of duplicated rows in a corpus that only appears after a crash-interrupted rotation, which is not reproducible in a screenshot.

Rotation writes the archive segment before rewriting the live file, and
that order is deliberate: until the archive lands, the live file is the
only copy of those rows. A crash between the two steps therefore leaves
the archived prefix in the live file as well, so the next rotation
computes its own dropped set from a live file that still begins with
those rows and archives them again as the following segment.

read_rotated_messages concatenated segments with no cross-segment
identity check, so those rows were served twice. That is not cosmetic:
this corpus is the index space before/next_before cursors and the fork
index path resolve a rendered row's position against, so one duplicated
row shifts every index above it, silently -- a reader pages positions
that no longer mean what they meant, and an index-addressed fork copies
a different cutoff than the one on screen.

drop_persisted_tail_prefix already owns this identity rule for the
archive/live seam; apply it one level down at the segment boundary. Both
producers persist a prefix, never an interior slice, so the duplication
is always a segment's head against what the corpus already ends with.
@buluoray
buluoray requested review from a team and CrysisDeu September 4, 2026 16:12
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 4, 2026
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 4056e2ba5691ca8bab8f329fecdd9169a5d1a223 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Right layer, right rule: the one function assembling the rotated corpus reuses the existing seam helper, and reader-side dedup also heals corpora already damaged on disk.

The reader-side placement is the correct root-cause choice, not a symptom patch: a writer-side guard in _maybe_rotate could only prevent future overlaps, while this heals segments already duplicated by past crashes, and it mirrors how the archive↔live seam is already handled (live = drop_persisted_tail_prefix(rotated, live) at history_projection.py:537). The failure mode when the prefix shape doesn't hold is append-everything — no worse than pre-fix — and the incomplete-read path still raises before caching, so the guard never composes with a partially-read archive. Tests pin both the fix and its upper bound (interior repeats and same-second distinct rows survive), which is what keeps the deleting-matcher honest.

[DESIGN-REVIEWED] 4056e2b

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 4056e2ba5691ca8bab8f329fecdd9169a5d1a223 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4056e2b

Verdict parsed from the review's SHA-scoped output markers for commit 4056e2ba5691ca8bab8f329fecdd9169a5d1a223.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4056e2ba5691ca8bab8f329fecdd9169a5d1a223: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 4, 2026
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 4056e2ba5691ca8bab8f329fecdd9169a5d1a223 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All claims verified: the writer ordering in history_rewrite.py:182-224 (archive first, rewrite second), the existing helper drop_persisted_tail_prefix at history_projection.py:58 designed for exactly this shape, and the fact that read_rotated_messages is the single shared segment-concatenation point (both read_messages_chained_full and read_rotated_messages_chained route through it). One sibling consumer of overlapping segments exists (session_digest.py), which only counts turns. Final review:

First-Principles-Verdict: PASS

A reported crash-window defect fixed at the one shared reader with the existing helper — no new surface, no new identity rule, tests pin both directions.

What this change ships

Intent: after a crash-interrupted rotation, serve each archived row once so pagination cursors and fork cutoffs land on the row on screen — a FIX.

  1. Rows re-archived by a retry rotation now appear once in the transcript corpus — justified (reported defect, split from perf(chat): reachable archived history and stable phone scrolling #7916).
  2. A segment head matching the corpus tail is silently dropped, bounded by the existing meta.mid / (ts, role, content) rule — justified, declared.
  3. Six-test file pinning the guard and its over-greed limits — rides with the fix, declared.

No new config key, flag, or public symbol; the guard reuses drop_persisted_tail_prefix (history_projection.py:58), the same rule already applied at the archive↔live seam (history_projection.py:537), so lens 4's answer is "it uses the existing one." The fix sits at the shared mechanism: both chained readers route through read_rotated_messages, so this is one fix, not a point patch — and the true cause (two file writes cannot be atomic; archiving deliberately precedes the rewrite to fail closed, history_rewrite.py:189-207) is a physical constraint, with reader-side healing the only fix that also repairs corpora already on disk.

Watch

One counted sibling of the root cause: grepped ARCHIVE_SEGMENT_DELIMITER consumers — session_digest.py:65-88 concatenates segments plus the live file with no dedupe at either seam, so an overlapping corpus double-counts turns. A cosmetic count, not an index space; accepted-and-deferred, not a demand.

[FIRST-PRINCIPLES-REVIEWED] 4056e2b

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging 4056e2ba5691ca8bab8f329fecdd9169a5d1a223. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/history_projection.py:471 -- Reused message IDs delete distinct archived rows

rows.extend(drop_persisted_tail_prefix(rows, seg_rows))
Distinct /api/chat messages sharing caller-supplied meta.mid -> rotation places them across segment boundaries -> the latter row disappears from pagination and fork indices.
Anchor: residual/crash-data-loss-corruption
Fix: Corroborate matching IDs with equal body or timestamp before dropping the prefix.
[BLOCK-MERGE] 4056e2b
[GPT-REVIEWED] 4056e2b
False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4056e2ba5691ca8bab8f329fecdd9169a5d1a223: <one-sentence reason>

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 5, 2026
@dwu96 dwu96 added the needs-pr-triage PR scanner: awaiting automated triage label Sep 8, 2026
@chenmingwei23 chenmingwei23 added drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 8, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor

Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days with failing CI. I have assessed the blockers and they appear resolvable -- I will push fixes directly to this branch as a co-author.

Assessment: Three blockers, all mechanical. (1) Branch is behind main (mergeable_state=dirty) -- needs a rebase. (2) GPT 5.6 blocking finding at history_projection.py:471: drop_persisted_tail_prefix drops the tail prefix by matching meta.mid alone, so two distinct /api/chat messages that share a caller-supplied meta.mid across a segment boundary would delete the later row -- the fix is to corroborate a matching id with equal body or timestamp before dropping. (3) The two red test lanes are pre-existing flakes unrelated to this diff: Backend Tests (Windows) shard 2 is the test_denied_commands_security.py:4496 wall-clock ratio assertion, and E2E is a Playwright toBeVisible 5000ms timeout -- both clear on re-run. Design / Opus / First Principles all PASS.

If you would prefer I do not touch this PR, add the pr-no-autofix label.

@bolichen97

Copy link
Copy Markdown
Contributor

Closing as superseded by merged #8590 (2201757ab).

Same goal, same seam, same helper in src/kiro_crew/history_projection.py: serve each archived row once when a crash between the archive write and the live-file rewrite makes rotation N+1 re-archive segment N's prefix. main's version is a strict superset -- it dedupes the same cross-segment overlap, adds a warning log per drop, uses a safer identity rule (require_mid, which is exactly the answer to the still-open blocking review finding on this PR), and also guards the archive->live seam in read_messages_chained_full and the fork flat-prepend branch, with tests.

This branch is ~465 commits behind with a merge conflict, and nothing in it is missing from main. Thanks for spotting the overlap bug.

@bolichen97 bolichen97 closed this Sep 8, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 8, 2026
@dwu96 dwu96 removed the drive-to-green PR claimed by drive-to-green pipeline label Sep 8, 2026
@bolichen97
bolichen97 deleted the fix/archive-segment-overlap branch September 20, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: dedupe overlapping archive segments in the rotated-transcript corpus

4 participants