Skip to content

fix(memory): never consolidate a remote or incognito transcript into local memory - #10855

Closed
cixuuz wants to merge 1 commit into
mainfrom
fix/no-consolidate-remote-transcripts
Closed

cixuuz wants to merge 1 commit into
mainfrom
fix/no-consolidate-remote-transcripts

Conversation

@cixuuz

@cixuuz cixuuz commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Two kinds of transcript sit on this machine's disk that must never be distilled into local memory: the local mirror of a session bound to a remote peer crew, and a session running in incognito or temporary memory mode. When either expires, or when someone runs kirocrew consolidate --all, the consolidator summarises it exactly as if it were an ordinary persistent local conversation.

Why it matters

A mirrored transcript is another machine's conversation, possibly another person's. An incognito transcript is one the user was promised leaves no memory (history.py defines those modes as never summarizable, and the dashboard turn path honours that). Once consolidated, either summary is re-injected into unrelated local sessions, travels inside kirocrew snapshot, and turns up in local full-text search, with no way to tell afterwards which memories came from where. Nothing in history_consolidation.py looks at executor, memory_mode, or any privacy marker.

What changed (motivation → approach → change)

Peer content reaches the local transcript because the relay mirrors a remote turn's frames through ordinary local slot.append calls; incognito rows are on disk for the live session.

The guard lives inside HistoryConsolidator._consolidate, beside the retry-eligibility gate: every entry point funnels through it, so a caller with no check of its own (kirocrew consolidate --all globs every transcript; session expiry has no restricted-session check) cannot leak. It sits inside the existing try so the finally releases self._running on every outcome, and the metadata read runs under asyncio.to_thread like the sibling I/O.

_transcript_privacy(key) makes one get_metadata_status read and returns one of three outcomes. executor == "remote" or is_incognito_transcript(memory_mode) (the shared predicate in history.py) is private: ConversationLog.fence_consolidation(key) reads the message count and rotation generation under one re-entrant _locked hold and advances last_consolidated to that count via mark_consolidated, handing no row to the consolidator. Behind the marker the rows never consolidate even after the marker clears, the idle sweep stops re-classifying the key, a repeat pass rewrites nothing, and the pass returns the completed-pass value. last_consolidated therefore means "every row below this index is settled": distilled, or fenced on purpose. An absent or unrecognised value is local. An unreadable header, or a read that raises, is unknown: the pass returns _CONSOLIDATION_REFUSED, which every done-callback already reads as "no pass ran", so offsets and the failure budget stay untouched, consolidate_now returns False, a later pass retries, and src/kiro_crew/cli.py prints a retryable skip. Both headers are in SLOT_OWNED_META_KEYS.

The fence only holds if nothing moves the offset back below the fenced rows, and rotation did: src/kiro_crew/history_rewrite.py wrote last_consolidated = 0 when a transcript exceeded its size cap, although it knows how many rows it dropped, so a private session that rotated after its marker cleared re-entered local memory. Rotation now rebases the offset to max(0, old - len(dropped)); the rotation_generation bump still rejects an in-flight consolidator's stale write. mark_consolidated's two stale-write fallbacks (generation mismatch, offset beyond the message count) preserve the bounded on-disk value instead of writing 0, and release the retry budget only when the caller's offset was applied (an explicit offset_applied flag, not value equality). Two named consequences for local transcripts: the retained tail is no longer re-distilled after every rotation, and a dashboard edit racing a consolidation no longer re-extracts the replaced tail through the stale completion (the edit owner carries the offset, as the non-racing rewrite path already does). Rotated rows go to the archive directory, which no consolidation path reads. src/kiro_crew/channel_transcript_migration.py's docstring, which cited the reset as precedent for the merge path's invalidation, now cites the rebase; the merge still invalidates to zero because it renumbers every row.

src/kiro_crew/docs/memory-and-learning.md states the three outcomes, the fence, the slot-owned markers, the post-transition rows that consolidate by design, and that rotation rebases rather than resets. docs/system-specs/modules/history.md records the boundary ownership: rotation and rewrite own the offset, a stale consolidator never lowers it, archives are outside the path, and the dashboard-edit trade is named.

header state before after
executor: remote 🟨 consolidated into local memory 🟩 fenced behind last_consolidated, nothing distilled, completed pass
memory_mode: incognito / temporary 🟨 consolidated into local memory 🟩 fenced behind last_consolidated, nothing distilled, completed pass
private rows after the slot unbinds or memory mode returns to normal 🟨 consolidated on the next pass 🟩 stay behind the offset; only rows appended after the transition consolidate
idle sweep on a private transcript 🟨 re-classifies the key every tick 🟩 skips it (nothing unconsolidated)
private rows retained across a transcript rotation 🟨 offset reset to 0, rows consolidated on the next pass 🟩 offset rebased by the dropped-row count, rows stay fenced
local transcript after a rotation 🟨 retained tail re-distilled every rotation 🟨 retained tail stays settled; only new rows consolidate
stale mark_consolidated after a rotation or edit 🟨 offset reset to 0 🟨 on-disk offset preserved (bounded by the row count); budget released only when the offset was applied
absent or other values (ordinary local session) 🟦 consolidated 🟦 consolidated
header present but unreadable 🟨 consolidated 🟩 refused, _running released, bookkeeping untouched, retried later
metadata read raises 🟨 consolidated 🟩 refused, _running released, bookkeeping untouched, retried later
CLI consolidate on a refused key 🟨 done ✓ 🟩 skipped (retryable; no consolidation pass ran)
memory spec, history module spec 🟨 say only when consolidation runs; describe rotation as a reset 🟩 state which transcripts it may touch, the fence, and that rotation/rewrite own the boundary

🟩 added · 🟨 changed · 🟦 unchanged

Tests

test/test_consolidation_remote_guard.py (new, 33 tests including parametrised cases):

  • Classifier: remote executor is private; absent markers are local; other executor values are local; remote comparison is case-insensitive; incognito/temporary are private case-insensitively; other memory modes are local; an unreadable header is unknown; a raising get_metadata_status is unknown.
  • Choke point: the guard is evaluated off the event loop; a remote transcript is refused before the transcript body is read; an incognito/temporary transcript is refused before the snapshot (both leave unconsolidated_count == 0, last_consolidated == 3, no LLM call); rows fenced while private stay out after the marker clears (executor dropped from the header / memory_mode back to persistent, two local rows appended, the LLM prompt carries only the local rows and the offset lands at 5); a second private pass rewrites nothing; the idle sweep never reaches _consolidate for a fenced transcript; a failed fence write propagates and releases the _running claim with the rows still unconsolidated and retry_eligible true; an unreadable header releases the _running claim, leaves offsets and mark_consolidated untouched, and makes consolidate_now return False; an absent header still consolidates; a local transcript reaches the snapshot; the CLI path reports done for a confirmed private skip.
  • Rotation: rows fenced while private, then the marker clears, two local rows are appended and a real rotation (size cap forced down) retains the tail: the LLM prompt carries only the local rows and no private row, for both the executor and the memory-mode transition. Fails on base with the offset reset to 0.

test/test_history_consolidation_guard.py (new, 5 tests): rotation rebases the offset by the dropped-row count (subtract, floor at zero, fully fenced tail stays fenced); a generation-mismatched and an oversized stale mark_consolidated each keep the rotation's rebased offset instead of writing 0.

Re-pinned in test/test_history.py, test/test_history_locking_remediation.py and test/test_history_consolidation_retry.py: the reset-to-zero contract those files asserted after a rotation or stale write is renamed and asserted as preserve-the-boundary (test_rotation_rebases_consolidated records the local-transcript trade in its docstring), and a stale write whose rejected offset happens to equal the preserved boundary still keeps a capped retry budget. The == 0 pins left in place govern different contracts (an interleaving merge invalidates the offset on purpose; a fresh transcript starts at zero).

The consolidation and rotation suites pass on this head (827 tests across the guard, arithmetic, history, locking-remediation, retry, coverage, migration, CLI and placeholder files). Mutation checks: moving the guard back above the try fails the release/pending test; returning the completed-pass value for an unknown header fails the same test; neutering the memory_mode branch fails six tests including both before-snapshot end-to-end cases; removing the fence call fails nine tests including both lifecycle cases; fencing one row short (total - 1) fails eight, both lifecycle cases among them; rebasing the rotation offset one row short fails both rotation lifecycle cases, and one row long fails eight (both lifecycle cases and the renamed local pin, so the arithmetic is pinned on the completeness side too); setting either stale-write fallback back to 0 fails the matching arithmetic test; removing the generation bump fails the locking-remediation generation test; releasing the retry budget on value equality instead of offset_applied fails the equal-offset cap test.

Manual verification

N/A — unit coverage sufficient. Every entry point (kirocrew consolidate --all, threshold, idle sweep, dashboard trigger, session expiry) calls _consolidate, and the end-to-end tests drive that function with real transcript files on disk.

Related Issues

no linked issue: the leak was found while reviewing the adopt path landed in #10288 (which first produced remote-mirrored transcripts on disk) and was never filed separately.

Pattern harvest

Rule candidate: review-prompt
Pattern: privacy or authorization check placed at one caller instead of the single function every caller funnels through, so an entry point added later (or a CLI that globs everything) bypasses it

Rule candidate: review-prompt
Pattern: an early-return guard added above a function's try/finally skips the finally that releases a claim the callers took before dispatching, and returns the success value for a "could not determine" case so callers advance bookkeeping on nothing

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@cixuuz
cixuuz requested review from a team and hoang-phan98 September 14, 2026 20:14
@cixuuz

cixuuz commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

Intent: A transcript mirrored from a remote peer crew is never distilled into local semantic memory, no matter which entry point (session expiry, kirocrew consolidate --all) triggers consolidation, and an unreadable executor header fails closed.
Not a goal: Keying on memory_mode or touching the open-slots persistence filter; classifying a headerless or hand-edited local transcript as remote; syncing or reconciling local and peer transcripts.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 14, 2026
@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 011a410fbbb2c4607da972773b664430a437eccd — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The fence is written only when a consolidation pass runs while the privacy marker is still on the header — a transition before the first pass leaks.

Watch

  • Lazy fencing leaves an hours-wide window. executor/memory_mode are slot-owned ("a slot save clears a stale value by omitting it"), and the idle sweep waits history_idle_secs (default 3h) before a pass touches an idle key. A remote transcript that unbinds/is adopted, or an incognito slot whose mode returns to normal, before any pass ran carries un-fenced private rows that the next pass classifies LOCAL and consolidates — the exact leak the PR's table claims closed ("private rows after the slot unbinds … stay behind the offset"), which holds only for rows fenced beforehand. No caller outside the consolidator invokes fence_consolidation, so nothing fences at the transition itself.
    Clears when: the path that clears a privacy marker (slot save/unbind/adopt) fences first, or a test/analysis shows every marker-clearing transition is preceded by a pass.
  • The reset-to-0 fallback in mark_consolidated was self-healing: a rewrite path that mismanaged the offset caused harmless re-extraction. Preserving the on-disk value flips that failure direction to silent under-extraction for any future structural-rewrite path that doesn't correctly maintain last_consolidated — an invariant now distributed across rewrite owners and enforced only by the spec.
    Clears when: history.md's "rotation and rewrite own the offset" section explicitly names this flipped failure mode so a new rewrite path is written against it.

[DESIGN-REVIEWED] 011a410

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — 🔴 changes requested (blocking)

GPT 5.6 found at least one blocking issue that must be resolved before merging 011a410fbbb2c4607da972773b664430a437eccd. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/history_consolidation.py:824 -- resuming an unfenced remote transcript loses its privacy boundary (origin: validation)
if str(meta.get("executor") or "").lower() == "remote": … return _TranscriptPrivacy.LOCAL
Remote transcript under 30 rows -> History resume intentionally restores it locally -> next save removes executor -> consolidation snapshots mirrored rows into local memory.
Anchor: residual/security
Fix: Durably fence private rows before any slot save can clear their privacy marker.
[BLOCK-MERGE] 011a410
[GPT-REVIEWED] 011a410

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

I have enough to rule. Key findings from the code I opened:

  • The durable privacy fence (fence_consolidation, advancing last_consolidated) is only applied when _consolidate actually runs on the transcript (history_consolidation.py:900-909). Both privacy markers (executor, memory_mode) are slot-owned (history.py:167-192), so a save that omits executor clears it and a later _transcript_privacy read returns LOCAL (history_consolidation.py:824-828).
  • Consolidation is triggered for small transcripts only via the idle sweep (check_idle_sessions:684-704, fires on unconsolidated >= 1), not the per-turn maybe_consolidate path (gated by _CONSOLIDATION_THRESHOLD at :653). So whether a small remote transcript is fenced before a local resume clears executor is a genuine timing race, not a contradiction.

The scenario — a remote session resumed locally (an intentional, supported operation per the code comment) before its small transcript was fenced — is an ordinary path, not one requiring mutually contradicting timing or inputs the writer cannot produce. The consequence is silent, unrecoverable cross-context leakage of peer-authored content into local semantic memory. I cannot complete a rarity record that a human would accept this residual risk, so the FLAG bar is not met.

Harm rung: UNBOUNDED (privacy boundary / cross-context leak, silent and unrecoverable).
Conditions confirmed: slot-owned executor cleared on save (history.py:192); privacy re-reads LOCAL after clear (history_consolidation.py:824); fence only applied inside _consolidate (history_consolidation.py:900).
Recovery path: none — content already distilled, re-injected, snapshotted, and searchable.

[ADJUDICATION] 011a410 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 011a410

[ADJUDICATION-FENCED] 011a410 fenced=1 flagged=0
UPHOLD-FENCED F1 src/kiro_crew/history_consolidation.py:824 -- Resuming a not-yet-fenced small remote transcript locally is an ordinary supported path that races the idle-sweep fence, yielding silent unrecoverable leakage of peer content into local memory; the extreme-rarity bar for FLAG is not met.
[GPT-ADJUDICATED-FENCED] 011a410

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 011a410fbbb2c4607da972773b664430a437eccd: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 011a410fbbb2c4607da972773b664430a437eccd — this comment is updated in place on each push.

Review details

I've reviewed both candidates against the code.

Candidate 1 (private rows leak after the slot-owned marker clears): The fencing feature is entirely new in this PR — before it, remote/incognito transcripts leaked unconditionally on the consolidate --all and session-expiry paths, so this diff strictly reduces leakage. The residual gap the candidate describes (rows appended-while-private-but-after-the-last-fence, or a never-fenced short session) leaks only if a memory-mode/binding transition beats the next fencing pass (maybe_consolidate threshold at history_consolidation.py:646, the idle sweep at :684, or session-end fence). Every concrete instantiation reduces to that timing "if the transition happens before a fence runs." The discovery pass itself could not pin the cadence, and I cannot re-derive input (a) as deterministic rather than a race. The minimal fix (fence at the transition/append point) also lives in untouched transition-handler code. Dropped.

Candidate 2 (max(0, old_offset - len(dropped)) uses raw line count): Confirmed last_consolidated is a parsed-message index (history_projection.py:805-817 skips blank/non-JSON/non-dict/metadata lines) while dropped = message_lines[:-keep_count] (history_rewrite.py:181) is raw lines. But the divergence only exists when a blank or corrupt line sits inside the dropped region; the append path writes one JSON message per line, so in practice len(dropped) equals the parsed count and the rebase is exact (verified against test_rotation_rebases_consolidated and the rebase arithmetic). Establishing the defect requires assuming a corrupted transcript — not an input that occurs in practice at the required bar. Dropped.

No Step 2 additions: the mark_consolidated preserve-on-disk change and the rotation rebase are internally consistent and the stale-write / generation-mismatch / offset-overflow paths never mark unconsolidated content as done (the on-disk boundary is only ever rebased down by rotation or carried verbatim by rewrite, then bounded by the live count).

No findings.

[OPUS-REVIEWED] 011a410

Verdict parsed from the review's SHA-scoped output markers for commit 011a410fbbb2c4607da972773b664430a437eccd.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 011a410fbbb2c4607da972773b664430a437eccd: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 011a410fbbb2c4607da972773b664430a437eccd — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify that a non-racing dashboard edit already leaves a replaced tail unextracted — preserving the stale offset in mark_consolidated is safe only then.

What this change ships

Inventory (9 items) — 9 justified

Intent: stop remote-mirror and incognito/temporary transcripts from being distilled into local memory — a FIX (documented invariant at history.py:770 plus the external-content boundary).

  1. Remote-mirror transcripts are fenced, never distilled into local memory — justified
  2. Incognito/temporary transcripts are fenced, restoring history.py's "never summarizable" invariant — justified
  3. Rows fenced while private stay excluded after the slot-owned marker clears — justified
  4. An unreadable metadata header refuses the pass and retries later, instead of consolidating — justified
  5. Rotation rebases last_consolidated by dropped rows; the retained local tail is no longer re-distilled — justified
  6. A stale consolidator write preserves the on-disk boundary instead of resetting it to 0 — justified
  7. Retry budget is released only when the caller's offset was actually applied — justified
  8. CLI skip message now reads "retryable; no consolidation pass ran" — justified
  9. Memory doc and history spec record the fence and boundary ownership in the same commit — justified

[FIRST-PRINCIPLES-REVIEWED] 011a410

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 14, 2026
@cixuuz cixuuz changed the title fix(memory): never consolidate a remote peer's transcript into local memory fix(memory): never consolidate a remote or incognito transcript into local memory Sep 15, 2026
@cixuuz
cixuuz force-pushed the fix/no-consolidate-remote-transcripts branch from e4104d6 to 24038b3 Compare September 15, 2026 05:33
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 15, 2026
@cixuuz

cixuuz commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author
  • span=e4634eec0079 fixed — confirmed: the early return None sat above the try, so the finally at the end of _consolidate never released self._running for a key whose header read failed, and maybe_consolidate / the idle sweep / the dashboard trigger then skipped it for the life of the process. The privacy classification now runs inside the existing try (first statement, still before snapshot_for_consolidation), so every outcome reaches the release. test_an_unreadable_header_releases_the_claim_and_stays_pending pre-claims the key, drives an unreadable header, and asserts the key is released, offsets and mark_consolidated are untouched, and consolidate_now returns False; moving the guard back above the try fails it. Fix SHA 24038b3.

@cixuuz

cixuuz commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author
  • span=73a12e5b3d79 fixed — agreed: a confirmed remote header and an unreadable one both returned the completed-pass value, so an unknown set the done-throttle and the CLI printed done. _is_remote_transcript is replaced by _transcript_privacy, returning LOCAL / PRIVATE / UNKNOWN from one metadata read. PRIVATE (confirmed remote, or incognito/temporary) is a deliberate permanent skip and keeps returning None; UNKNOWN returns _CONSOLIDATION_REFUSED, which every caller's done-callback already treats as "no pass ran", so bookkeeping stays pending, the failure budget is not charged, and consolidate_now returns False. The CLI skip line now names the general condition. Returning None for UNKNOWN fails the release/pending test. Fix SHA 24038b3.

@cixuuz

cixuuz commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author
  • span=d31e59532b94 fixed — agreed: the description named memory_mode as a marker nothing read, and the diff gated only executor, while kirocrew consolidate --all reaches incognito transcripts with no upstream filter. The same single get_metadata_status read at the choke point now also classifies is_incognito_transcript(meta.get("memory_mode")) as PRIVATE, reusing the shared predicate in history.py rather than a second spelling of the set; absent or unrecognized modes stay local per that predicate's documented convention. Pinned by test_restricted_memory_modes_are_private_case_insensitively and test_a_restricted_memory_transcript_is_refused_before_snapshot (both modes); neutering the branch fails six tests. Title and body now describe both markers. Fix SHA 24038b3.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 15, 2026
@cixuuz
cixuuz force-pushed the fix/no-consolidate-remote-transcripts branch from 24038b3 to 5d88eea Compare September 15, 2026 16:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 15, 2026
@cixuuz

cixuuz commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author
  • span=383d93b4b1b8 accepted-deferred — agreed, and out of this guard's reach. executor is slot-owned, so a slot that unbinds after an adopt or relay backfill rewrites its header without the marker, and the peer rows it mirrored earlier become consolidatable on the next pass. Nothing at the consolidation chokepoint can recover that: the header is the only slot-level fact the consolidator reads, and the rows themselves carry no per-row origin. Closing it needs a fence on the pre-unbind rows, owned by the unbind path — which is precisely what feat(adopt): record one peer-row watermark per adopted slot #10856's slot-owned peer_row_watermark would anchor (its first consumer). The residual is now stated in the spec: src/kiro_crew/docs/memory-and-learning.md § Consolidation names the unbind case as the known gap and points at the watermark. Fix SHA 5d88eea.

@cixuuz

cixuuz commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author
  • span=34be52e3fbc1 fixed — agreed: the three outcomes lived only in code comments, so the owning spec described the pre-guard behaviour. src/kiro_crew/docs/memory-and-learning.md § Consolidation now states the contract as the code implements it: one metadata read of the slot-owned executor and memory_mode headers before the transcript snapshot; LOCAL consolidates; PRIVATE (executor == remote, or incognito/temporary memory_mode) is a permanent skip returning the completed-pass value; UNKNOWN (unreadable header) returns _CONSOLIDATION_REFUSED, so the claim is released, bookkeeping stays pending and the failure budget is not charged. The paragraph also names the unbind residual from span 383d93b4b1b8. Docs-only change; comment-history and body-accounting gates green. Fix SHA 5d88eea.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 15, 2026
@cixuuz
cixuuz force-pushed the fix/no-consolidate-remote-transcripts branch from 5d88eea to 31c4496 Compare September 15, 2026 21:28
@cixuuz

cixuuz commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author
  • span=4f398b0da803 fixed — agreed, and the maintainer ruled to fence in this PR rather than defer: read-time refusal is not durable exclusion when both markers are slot-owned. A confirmed PRIVATE skip now records itself in the same bookkeeping the success path writes. ConversationLog.fence_consolidation(key) (src/kiro_crew/history.py, after snapshot_for_consolidation) reads the message count and rotation generation and advances last_consolidated to that count via mark_consolidated, all under one _locked hold (RLock plus refcounted flock, re-entrant), without handing any row to the consolidator. _consolidate calls it via asyncio.to_thread inside the existing try, so a failed marker write still releases _running through the finally; the return stays the completed-pass value and a repeat pass rewrites nothing. Pinned by test_rows_fenced_while_private_stay_out_after_the_marker_clears[executor|memory_mode] (3 private rows fenced, marker cleared — executor dropped from the header the way the slot save omits it, or memory_mode back to persistent — 2 local rows appended, the LLM prompt carries only the local rows and the offset lands at 5), plus a second-pass no-rewrite test and a failed-write test that asserts the claim is released and the rows stay unconsolidated. Two independent mutations bite: removing the fence call fails 9 tests, fencing one row short (total - 1) fails 8, both lifecycle cases among them. Disclosed cost: last_consolidated also means deliberately skipped; mark_consolidated's docstring and the spec paragraph say so. Residual named in the spec rather than closed: a transcript rotation resets the offset to 0 and re-consolidates the retained tail, so a rotation that follows the marker clearing and retains pre-transition rows lets them consolidate — rotation's existing reset-and-reconsolidate design, unchanged here. Fix SHA 31c4496.

@cixuuz

cixuuz commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author
  • span=6c4b0a4678fb fixed — implemented as suggested, with one shape difference stated for the record: rather than reusing the LOCAL path's snapshot_for_consolidation (which returns the unconsolidated rows, and two existing tests pin that a private transcript's rows are never surfaced to the consolidator), the fence is a dedicated ConversationLog.fence_consolidation(key) that reads only the count and rotation generation and calls mark_consolidated(key, total, generation) nested inside the same _locked hold, so no append or rotation can pair a stale count with a fresh generation. The side effect the suggestion predicted is pinned: test_the_idle_sweep_skips_a_fenced_private_transcript runs the real check_idle_sessions pre-check (unconsolidated < 1) after a fence and asserts _consolidate is never reached and _running/_tasks stay empty. The existing before-snapshot tests now assert unconsolidated_count == 0 and last_consolidated == 3 after a private pass. Docstrings on mark_consolidated and the guard comment state that the offset also marks a deliberate skip; the spec's "Private" bullet describes the fence and the old "only the current header is consulted" paragraph is replaced with the post-transition rule. Fix SHA 31c4496.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 15, 2026
…local memory

Two kinds of transcript on this disk must never be distilled into local
memory: the local mirror of a session bound to a remote peer (the relay
mirrors a remote turn's frames through ordinary local slot.append calls,
which is what makes the mirror faithful and also lands peer-authored
conversation in local history), and a session running incognito/temporary
(history.py defines those modes as never summarizable). The consolidator
checks neither: history_consolidation carries no executor, memory_mode or
privacy gate of any kind, so a distilled summary of another machine's
conversation -- or of a session the user was promised leaves no memory --
can be re-injected into unrelated local sessions, carried into `kirocrew
snapshot`, and surfaced in local full-text search.

Gate it inside _consolidate, for the same reason the retry-eligibility gate
beside it lives there: every entry point funnels through that function, so a
caller carrying no check of its own still cannot leak. That is not
hypothetical -- `kirocrew consolidate --all` globs every transcript on disk
with no filter, and the session-expiry consolidator has no restricted-session
check at all. The guard sits inside the function's existing try so the
finally releases self._running on every outcome; callers claim that key
before dispatching and rely on the release.

One get_metadata_status read yields three outcomes. executor == "remote" or
is_incognito_transcript(memory_mode) -- the shared predicate in history.py --
is PRIVATE: the pass fences the transcript instead of distilling it.
ConversationLog.fence_consolidation reads the message count and rotation
generation and advances last_consolidated to that count through
mark_consolidated, all under one re-entrant _locked hold and without handing
a row to the consolidator; the write is offloaded like the sibling I/O and
sits inside the try so a failure releases the claim. Read-time refusal alone
is not durable exclusion: both markers are slot-owned, so a slot that unbinds
from its peer or a session whose memory mode returns to normal clears them,
and the next pass would read the same rows as local. Behind the marker they
stay out of memory across that transition, only rows appended afterwards
consolidate, and the idle sweep stops re-classifying the key every tick
because nothing is left unconsolidated. The pass returns the completed-pass
value: the marker write is the pass for a private transcript, and a repeat
pass rewrites nothing. last_consolidated therefore means every row below it
is settled, distilled or fenced on purpose, and its docstring says so.
An absent or unrecognized value is LOCAL, matching is_incognito_transcript's
documented convention. An unreadable header, or a read that raises, is
UNKNOWN: the pass returns _CONSOLIDATION_REFUSED, which every caller's
done-callback already reads as "no pass ran", so offsets and
mark_consolidated stay untouched, the failure budget is not charged,
consolidate_now returns False, and a later pass retries. Both headers are
slot-owned, so a slot save clears a stale value by omitting it. The CLI skip
line names the general condition since the sentinel now has two causes.

The fence only holds if nothing later moves the offset back below the
fenced rows. A transcript rotation (history_rewrite, when the file exceeds
its size cap) wrote last_consolidated = 0 even though it knows exactly how
many rows it dropped, so a private session long enough to rotate after its
marker cleared would re-enter local memory on the next pass. Rotation now
rebases the offset to max(0, old - len(dropped)): a fully fenced transcript
stays fully fenced, local rows appended after the transition keep the fence
at the same boundary, and the rotation_generation bump still rejects an
in-flight consolidator's stale write. mark_consolidated's two stale-write
fallbacks (generation mismatch, offset beyond the message count) preserve
the bounded on-disk value instead of writing 0, so a racing consolidator
cannot lower the fence either, and the retry budget is released only when
the caller's offset was actually applied (an explicit offset_applied flag,
not value equality). Two behaviour changes for ordinary local transcripts
follow: the retained tail is no longer re-distilled after every rotation,
and a dashboard edit racing a consolidation no longer re-extracts the
replaced tail via the stale completion; the edit owner carries the offset,
as the non-racing rewrite path already does. Rotated rows go to the archive
directory, which no consolidation path reads.

The memory spec (docs/memory-and-learning.md) states the three outcomes
under Consolidation as the contract every entry point inherits: the fence,
the slot-owned markers, the post-transition rows that consolidate by
design, and that rotation rebases rather than resets. The history module
spec records the boundary ownership: rotation and rewrite own the offset in
the new numbering, a stale consolidator never lowers it, archives are
outside the consolidation path, and the dashboard-edit trade is named.
@cixuuz
cixuuz force-pushed the fix/no-consolidate-remote-transcripts branch from 31c4496 to 011a410 Compare September 16, 2026 07:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 16, 2026
@cixuuz

cixuuz commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author
  • span=78ebcb5cb0e3 fixed — agreed, and the "pre-existing design" framing in the previous disposition was wrong: the fence is what turns rotation's reset into a privacy hole, so it is this PR's residual to close. _maybe_rotate in src/kiro_crew/history_rewrite.py knows exactly how many rows it dropped and now rebases the persisted boundary to max(0, old - len(dropped)) instead of writing 0: a fully fenced transcript stays fully fenced through any number of rotations, a transcript with local rows appended after the transition keeps the fence at the same boundary, and the rotation_generation bump still rejects an in-flight consolidator's stale write so mark_consolidated's safety argument is unchanged. The two stale-write fallbacks in mark_consolidated (src/kiro_crew/history.py: generation mismatch, offset beyond the message count) also used to reset to 0 — a consolidator racing the rotation would have lowered the fence the same way — and now preserve the bounded on-disk value; the retry budget is released only on an explicit offset_applied flag rather than value equality, so a preserved value equal to the stale caller's number does not hand a capped span a fresh budget. Pinned end-to-end by test_rotation_keeps_pre_transition_rows_behind_the_fence[executor|memory_mode] in test/test_consolidation_remote_guard.py (3 private rows fenced, marker cleared, 2 local rows appended, a real size-cap rotation retains the tail, the LLM prompt carries only the local rows and no private row — fails on the reset-to-0 code with both arms), by the arithmetic and stale-write tests in the new test/test_history_consolidation_guard.py, and by the re-pinned rotation/stale-write assertions in test/test_history.py, test/test_history_locking_remediation.py and test/test_history_consolidation_retry.py (the reset-to-zero contract is renamed and asserted as preserve-the-boundary; the == 0 pins left in place govern the interleaving-merge invalidation and fresh-transcript defaults, which are different contracts). Mutations that bite: rebase one row short fails both lifecycle arms; one row long fails 8 including the renamed local pin; either fallback back to 0 fails its arithmetic test; removing the generation bump fails the locking-remediation generation test; releasing the budget on value equality fails the equal-offset cap test. Archives were checked: rotation writes dropped rows to the archive directory, consolidate --all globs only the live directory, and history_consolidation.py never reads archives. Named behaviour change for local transcripts, recorded in the body, mark_consolidated's docstring and both specs: the retained tail is no longer re-distilled after every rotation, and a dashboard edit racing a consolidation no longer re-extracts the replaced tail through the stale completion (the edit owner carries the offset, as the non-racing rewrite path already does). Fix SHA 011a410.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Sep 16, 2026
@NicholasRBowers NicholasRBowers added needs-pr-triage PR scanner: awaiting automated triage needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 22, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

  • main has landed a competing version of the incognito half of this fix, after your base, and the two disagree on policy. Your merge-base is 85847f00b (2026-09-16) and had no privacy check in history_consolidation.py — verified: zero hits for is_incognito_transcript or read_session_execution in that file at your base. Commit bc0d2b0cf ("refactor(memory): simplify member memory to one SQLite store", 2026-09-18) then added one at the same choke point:

    execution = await asyncio.to_thread(read_session_execution, key)
    if execution is not None and execution.memory_mode != "persistent":
        return _CONSOLIDATION_REFUSED
    metadata = await asyncio.to_thread(self._log.get_metadata, key)
    if isinstance(metadata, dict) and is_incognito_transcript(metadata.get("memory_mode")):
        return _CONSOLIDATION_REFUSED

    That is the single conflicting hunk in history_consolidation.py, and it is not a textual union. Two things have to be decided, not merged: (a) main refuses the pass and leaves the bookkeeping pending so a later pass retries, while your _transcript_privacy path fences and completes (fence_consolidation, then return None, with the marker write treated as the pass) — those are different behaviours on the same privacy boundary and main shipped one of them deliberately; and (b) main's read_session_execution execution-context check catches a slot whose live memory mode is non-persistent even when the transcript header does not say so, which your metadata-only classifier does not. Does the unified classifier replace main's two checks, wrap them, or take main's refusal policy for incognito and keep the fence for remote only?

  • The GPT 5.6 finding is upheld, security-fenced, and its remedy moves where the fence lives. history_consolidation.py:824, [BLOCK-MERGE], and Opus recorded UPHOLD-FENCED rather than FLAG: because both privacy markers are slot-owned, a save that omits executor clears it, _transcript_privacy then reads LOCAL, and small transcripts are only consolidated by the idle sweep — so a remote session resumed locally before its transcript was fenced is an ordinary supported path with no recovery once the rows are distilled. GPT's fix is "durably fence private rows before any slot save can clear their privacy marker", and Design's Watch span=e599ca531143 independently asks for the same thing: "the path that clears a privacy marker (slot save/unbind/adopt) fences first". That inverts the fence from consolidation-time to marker-clear-time and puts it in history.py's save/unbind/adopt paths — a new cross-module invariant outside this PR's current 12 files, and it lands on the same seam as the policy question above. A security-class fence clears only by that code change or by a repository-writer /ai-review override, never by a disposition, and I won't self-authorize one.

  • Design's Watch span=0837f3d31258 is a documentation decision about a failure direction you flipped. The reset-to-0 fallback in mark_consolidated used to be self-healing — a rewrite path that mismanaged the offset caused harmless re-extraction. Preserving the on-disk value flips that to silent under-extraction for any future structural-rewrite path. It clears when history.md's "rotation and rewrite own the offset" section names the flipped failure mode explicitly, so a new rewrite path is written against it.

First Principles PASSes, Opus 4.8 has no blocking findings, and everything else is green — the two test-file conflict hunks in test_history_consolidation_retry.py are mechanical once the policy above is settled.

When you've addressed these, the pipeline will re-assess on its next cycle.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you @cixuuz for this — never distilling a remote or incognito transcript into local memory is important, and your intent was spot on. Main has since covered the same hole through a more complete mechanism: #13760 (keep incognito and temporary transcripts, learn nothing), backed by the read_session_execution persistent-mode guard in src/kiro_crew/history_consolidation.py, which now refuses consolidation whenever memory_mode isn't 'persistent' (remote executor, incognito, or temporary), plus the is_incognito_transcript check and a withhold_restricted re-read under lock. So I'm closing this as superseded. Thanks again for the careful memory-privacy work — if something still feels uncovered, you're very welcome to open a small follow-up.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 30, 2026
@bolichen97
bolichen97 deleted the fix/no-consolidate-remote-transcripts branch November 6, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants