Repository navigation
feat(ledger): expire closed session ledgers on the archive switch - #11193
Conversation
Design Review (Fable 5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Conversation-clearing resets unmap the sid but record non-terminal WatchThe compaction-escalation path calls Suggestions
[DESIGN-REVIEWED] db993b1 |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify one coverage call: the RFC records the delete half (FR-7, #10705) but lists "Session ledger retention" as an open question — confirm riding What this change shipsInventory (10 items) — 10 justifiedIntent: make deleting a conversation actually delete its ledger and stop opted-in ledgers growing without bound — an ADDITION (the retention path
[FIRST-PRINCIPLES-REVIEWED] db993b1 |
GPT 5.6 Review (fork) — 🔴 changes requested (blocking)Reviewed 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands. BLOCKING -- src/kiro_crew/ledger/store.py:733 -- queued revival can lose a live session ledger (origin: validation) Adjudication (Opus 4.8) — is blocking on each finding proportionate?The adjudicable block is empty (0 findings). One fenced finding, F1. F1 — Conditions I confirmed from code opened this run:
I cannot complete a record showing the condition combination is contradictory or that the trusted writer alone cannot produce it; the async-write window the finding names is real and the loss is irreversible. When torn on an unbounded, unrecoverable class, the contract requires UPHOLD-FENCED. [ADJUDICATION] db993b1 total=0 uphold=0 downgrade=0 [ADJUDICATION-FENCED] db993b1 fenced=1 flagged=0 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
fbc1e3c to
91129df
Compare
91129df to
05b7cbc
Compare
05b7cbc to
7c131a5
Compare
7c131a5 to
e915f69
Compare
…switch Session message bodies live in the append-only ledger now, but nothing expired it: `session.archive_retention_days` collected the transcript archive while the ledger it points into grew forever, and permanently deleting a session removed only the transcript. One removal primitive, `store.remove_unit`, is what both callers use -- two callers deleting one tree two ways is two chances to get the order wrong, and the order is the whole correctness argument. It claims the unit's `.lease` non-blocking and SOLE, removes the segments, then the per-append lock, then the lease LAST and only as its holder, counts every failure, and writes nothing into a ledger it is about to delete. `sole` is load bearing rather than decorative: the lease is refcounted per process, so a plain claim against a unit this process is already writing succeeds by joining that count and proves nothing -- which would let a sweep inside the gateway unlink the segments the emitter's cached handle is appending to. A sole holder also blocks every later claim until it is released, so a writer cannot join the remover's own lock from the other side. `store.sweep_expired` runs inside `history._cleanup_old_archives`, on the same setting and the same hourly throttle, with no new config key and a negative value disabling both halves. It walks `ledgers/sessions` alone, ages a unit from the last `session/closed` entry's own time (mtime only as a fallback, since a restore resets it), and skips an open unit, a torn tail, and a directory no id addresses. The permanent-delete funnel removes the deleted session's ledger by the ACP session id its claim proved, after the teardown released the lease. That is the opposite of the rule for the WORK ledger, which the same funnel preserves, and the difference is mechanical: a work ledger is keyed by the recycled slot key, so a successor legitimately inherits it, while a session ledger is keyed by the ACP session id and holds a lease.
e915f69 to
db993b1
Compare
Owner disposition: not fixing "queued revival can lose a live session ledger"Recording an owner decision on the GPT lane's fenced finding at The mechanism is real, and verified
Why it is not fixed hereThe fix requires the emitter to register a pending open on the SYNCHRONOUS path -- before the write is
Adding the registration and the query means changing Residual risk, stated plainlyReaching this window requires all of: a Two properties bound the damage rather than removing it. The loss is RECORDED, not silent: the queued What would close itA pending-open registration on |
…switch (kirodotdev#11193) Session message bodies live in the append-only ledger now, but nothing expired it: `session.archive_retention_days` collected the transcript archive while the ledger it points into grew forever, and permanently deleting a session removed only the transcript. One removal primitive, `store.remove_unit`, is what both callers use -- two callers deleting one tree two ways is two chances to get the order wrong, and the order is the whole correctness argument. It claims the unit's `.lease` non-blocking and SOLE, removes the segments, then the per-append lock, then the lease LAST and only as its holder, counts every failure, and writes nothing into a ledger it is about to delete. `sole` is load bearing rather than decorative: the lease is refcounted per process, so a plain claim against a unit this process is already writing succeeds by joining that count and proves nothing -- which would let a sweep inside the gateway unlink the segments the emitter's cached handle is appending to. A sole holder also blocks every later claim until it is released, so a writer cannot join the remover's own lock from the other side. `store.sweep_expired` runs inside `history._cleanup_old_archives`, on the same setting and the same hourly throttle, with no new config key and a negative value disabling both halves. It walks `ledgers/sessions` alone, ages a unit from the last `session/closed` entry's own time (mtime only as a fallback, since a restore resets it), and skips an open unit, a torn tail, and a directory no id addresses. The permanent-delete funnel removes the deleted session's ledger by the ACP session id its claim proved, after the teardown released the lease. That is the opposite of the rule for the WORK ledger, which the same funnel preserves, and the difference is mechanical: a work ledger is keyed by the recycled slot key, so a successor legitimately inherits it, while a session ledger is keyed by the ACP session id and holds a lease.
Problem / Motivation
The append-only session ledger had no retention writer. Transcript archives expire on
session.archive_retention_days, but the ledger beside them grew without bound, and permanentlydeleting a session removed only its transcript. A Design-lane advisory on #10091 named it: bodies now
live in the ledger, so the smaller half of a conversation's history was being expired while the larger
half stayed on disk forever.
Two further defects were found while building the fix, and both were invisible from the outside.
session_lifecycle.destroywrote nosession/closed-- only the reset route did -- and that singlemissing entry defeated retention from both sides at once. The emitter keeps a per-session handle that
carries the unit's write lease, so a removal claiming that lease
soleansweredownedfor everysession the gateway had actually written for; and a unit whose newest lifecycle entry is not a close
reads as OPEN, so the sweep skipped it whatever its age. A destroyed session's ledger was immortal.
Why it matters
Someone who deletes a conversation expects it gone. They were getting the transcript row removed while
the ledger kept the message bodies, the tool calls and the model configuration of that conversation on
disk indefinitely -- and with no second setting to find, because none existed. On a long-lived gateway
the same gap grows the data directory without limit.
The two defects made it worse than a missing feature: a delete that reported success removed nothing
of the ledger, and no later pass would ever collect it either.
What changed
Chained from the symptom to the root cause:
store.remove_unitis the one spelling of deletion in the package -- lease claimed non-blocking andsole, a REQUIREDguardre-decision inside the hold, segments first,.leaseunlinked last andonly by its holder -- and both the sweep and the delete funnel call it.
store.sweep_expiredrunsfrom
history._cleanup_old_archives, sosession.archive_retention_daysgoverns ledgers too: oneswitch, the existing hourly throttle, no new config key, and a negative value disables both halves.
owned. Cause:destroynever recorded itsteardown, so the emitter never released the handle holding the lease.
destroynow writessession/closed {destroyed}beside the reset route's entry, which is also what releases the handle;the funnel flushes and then claims the lease. A flush that times out is not a failure -- the entry
stays owed, so the sweep collects the unit once it lands.
as open. Fixed by the same change, which is why one entry closes both.
destroyedclose authorizes collection, and that word asserts REVOCATION COMPLETED.destroyremoves exactly one map key, so before claiming it the teardown checks whether any OTHER keystill maps that ACP id; when one does it records
destroyed_sid_retainedinstead and the log stays.Two keys on one sid is a state the system itself produces -- importing a transferred session twice
allocates a new slot key each time and deliberately leaves the source intact
(
dashboard/session_transfer.py) -- so the other holder is not revoked to make the record tidy, andthe shared log is kept while it can resume. An unreadable map withholds the claim for the same reason,
and the permanent-delete funnel applies the same veto because it deletes without consulting the reason.
Reading the map to WITHHOLD a claim is safe in the way reading it to grant one is not: a forged or
emptied map can only make the gateway assert less than the truth, and the sweep itself still reads
nothing but the ledger.
resetis excluded even though it cold-starts its successor on a new id,because its own
clear_sidis guarded byif clear_conversation and session is not None: a reset thatkeeps the conversation writes
session/closed {reset}while leaving the old id mapped and resumable.An earlier revision asked
session_map.jsonwhich sessions were still mapped and KEPT those; that wasremoved, because a VALID EMPTY map is not a failed read -- it reads as "nothing is revivable" and hands
the trusted sweep a positive answer that authorizes deleting a fenced unit. Absence of protection must
never be authorization.
ledger it already had, so
... closed ... opened ...is a live conversation; reading the newestclose alone would have deleted its log.
partial removal is history already gone with something else left standing, and the log line says
which of the two happened.
Open units, torn tails, crew ledgers, linked unit directories, a linked kind root, and a header whose
id does not fold back to its own directory are all refused regardless of age. Nothing is ever written
into a ledger that is about to be removed; a citation into a removed unit is already answered by
Ref.resolvereportinggone.Alternatives considered and rejected: holding the write lease from a session's open through its append
(it contradicts ownership ending between turns by design, which is what lets an idle session be
collected at all); gating removal on the ledger being READABLE (that would make a damaged unit
immortal -- the one unit nothing can use would be the one retention could never reclaim); and matching
the delete funnel's target by slot key (slots are recycled, so a successor tab's ledger would be
collateral, which is exactly why the work ledger is preserved by this same funnel).
Tests
1,275 tests pass across
test_ledger_retention.py,test_remove_slot_for_history_key.py,test_ledger_core.py,test_ledger_lease.py,test_session_ledger_emit.py,test_history.pyandtest_session.py.What the new tests lock in: an expired closed unit is removed and an open one never is, however old;
a torn tail is left to the resume repair; a revived session is not expired by its older close; crew
ledgers are never scanned; a negative setting disables both halves; a unit a writer still owns is
skipped while its neighbour is collected; the lease file is removed last; a failure is counted and
never reported as a removal;
destroywrites its teardown entry; a destroyed unit is then collectableby the sweep; the funnel removes the matched ACP unit and no sibling; an unresolvable id removes
nothing; a refused or raising teardown keeps the ledger; both halves of the partial-removal report; a
mapping that names another slot's session removes nothing; a header with no slot is left to retention;
a
destroyedclose is collectable while reset, discarded, shutdown, crash, eviction, unknown, absentand non-string reasons are not; a VALID EMPTY session map cannot authorize collecting a reset-closed
expired unit; and the sweep opens no file outside the ledger tree.
Every new test is mutation-verified -- the behaviour is inverted and the test must go red. 20 of 20
reddened on the final commit, including
destroywriting no close, the funnel skipping its flush, theterminal-reason check removed and then made to reject everything,
resetre-admitted as terminal, thereason no longer type-checked, a
session_map.jsonread re-introduced into the sweep, the header-slotcheck removed, a slotless header accepted, and the header reader skipping its id fold-back.
Two tests were strengthened after a mutation SURVIVED, rather than the mutation being weakened. The
no-file-created test now refuses the removal, because a ledger created and then removed leaves exactly
the state of one never created -- the end-state assertion could not see the property at all. And the
funnel test asserts
ownedBEFORE running the funnel, so a fixture that does not reproduce theemitter-held lease fails loudly instead of passing quietly. That second one is how the defect above
hid: every earlier removal test built its ledger with
Ledger.create, which leaves nothing holdingthe unit, so they were green while the gateway could not collect a real one.
Manual verification
The emitter-held-lease behaviour was measured directly against the real emitter in a throwaway data
home, because it is the claim the whole delete-funnel half rests on: a ledger opened through the
emitter answers
ownedto a removal, andremovedonly once its close has landed.The retention pass was also driven through the real config-resolved entry point
(
history._cleanup_old_archiveswith no explicit retention) in an isolated home holding a real archivefile plus open, revived, closed and crew ledgers. It collected both halves in one pass and left the
open, revived and crew units alone.
Pod verification was attempted and is BLOCKED, not skipped:
systemctl --usercannot reach the sessionbus from this session's process, so
kirocrew pod upexits 0 while starting nothing. The runtime checkabove was substituted rather than falling back to the live gateway.
Related Issues
No linked issue.
Implements the retention section of merged RFC #10090 (
docs/request-for-change/rfc-append-only-ledger.mdon main). Follows #10091, merged as
69aeb2803. Sibling #11185 wires the remaining emit sites; it doesnot touch
session_lifecycle.py, so the teardown entry here does not collide with it.Pattern harvest
Rule candidate: review-prompt
Pattern: a test fixture that builds the object under test directly, rather than through the producer
production uses, can leave the feature dead while the suite is green. Here every removal test created
its ledger with
Ledger.create, which leaves nothing holding the unit's write lease -- so thesole-lease refusal that made the delete funnel a no-op in the gateway was never exercised. Thegeneral shape: when a behaviour depends on OWNERSHIP or on state a long-lived process caches, the
fixture has to reproduce that ownership, and the test should assert the refusal it expects BEFORE
asserting the outcome, so a fixture that does not reproduce it fails instead of passing quietly.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Other suggestions on the work
OWNER DISPOSITION -- "queued revival can lose a live session ledger" is NOT fixed here, by owner
decision. The GPT lane's fenced finding is real and its mechanism is verified:
on_session_openeddoes its work inside a job handed to the writer thread, so between a resume starting and its
session/openedlanding, the sweep's under-lease re-read still sees the old close and can win therace. Closing it needs a pending-open registration on that function's SYNCHRONOUS path plus a public
per-session query --
live_turnreports only a PINNED turn, andbuffered_writes()is process-global-- which means changing
session_ledger_emit.py's queue structure and public surface. That file andits test belong to sibling PR feat(ledger): resolve slot keys and wire eight session emitters #11185, which is open and editing both, and this change is deliberately
log-side only. Residual risk and the bound on it are recorded in a comment on this PR; the follow-up
is
has_pending_or_live(session_id)in the emitter with the guard extended to_still_expired(...) and not has_pending_or_live(unit_id).All three security-class findings from the review rounds are FIXED in this commit, not deferred.
The forged-mapping path is closed by requiring the unit's own header to name a proven slot. The
unreadable-map path and then the valid-empty-map path are closed by removing
session_map.jsonfromthe decision entirely: a unit is now collectable only on positive proof from inside the fenced tree,
its own terminal close reason. My first two passes got this wrong in the same direction twice -- first
documenting the window as an accepted residual, then guarding only the UNREADABLE map while a valid
empty one still read as "nothing is revivable". Absence of protection is not authorization.
session_map.jsonbeing agent-writable is still worth its own issue. Nothing in this PR trustsit any more, but the same file authorizes session resume and subagent continuation across the
gateway, so a forged entry still grants read and write of a victim's live conversation. Sealing it
means adding the file to
security/paths.py, a trust root, which does not belong in a retentionchange.
A close reason this build does not know keeps the unit forever. That is the safe direction, but it
means a future teardown path that emits a new reason will silently stop collecting its sessions until
the reason is added to
_TERMINAL_CLOSE_REASONS. The alternative -- treating unknown reasons asterminal -- would delete history on a guess, so the tradeoff is deliberate; a doctor check counting
uncollectable units would be a reasonable follow-up.