Repository navigation
feat(crew-log): resume a session fold from a savepoint on disk - #12030
Conversation
|
Intent: Stop a session's side panel costing more the longer the session has run, by writing each crew-log fold's resumable state beside the log and resuming from it, so a read pays for the entries that arrived rather than for the whole history. Not a goal: any new fold, any change to what a fold computes, any UI, and any change to the crew log's own format. The savepoints stay disposable: a read with no usable savepoint must behave exactly as it does today. |
|
Mutation verification for the savepoint guards. Each row removes ONE guard this PR adds and re-runs Two guards were REMOVED during this check rather than given a test, because no single-line mutation of either changed an observable outcome: establishing the log's identity already stats the newest segment, so a separate pre-lease segment check was unreachable, and the post-write discard already enforces the outcome a second under-lease recheck was guarding. |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the digest trade on real logs: an idle resume spends nearly all its 44.8 ms hashing, so the shipped win is 1.7–4x, not the RFC's "never replays". What this change shipsInventory (10 items) — 10 justifiedIntent: stop session side-panel reads from costing the session's whole history on every refresh, restart and cache eviction — an ADDITION, decided by
[FIRST-PRINCIPLES-REVIEWED] 85baec8 |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the win survives real session lengths: every resume rehashes the whole prefix, so the 1.7–4x measured at 10k entries decays as logs grow. Suggestions
[DESIGN-REVIEWED] 85baec8 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
8e72c00 to
14c5438
Compare
14c5438 to
8c93d44
Compare
|
self-added: yes
|
|
self-added: yes
|
8c93d44 to
297e3bb
Compare
|
self-added: yes
|
|
self-added: yes
|
|
Fixed: I measured the mechanism before fixing it, and the first test I wrote was too weak. On a log with one blank interior line: the blank IS a record to the raw framing, the savepoint's seq span is 273, the file holds 274 raw records, and the digest hashes 273 of them, so record 273 sits outside it. Then:
So the exposure is real, and it is value-only damage to a trailing consumed record that reaches it. The fix makes both sides count the same thing. The boundary is no longer derived as That walk reads a seq, so it decodes a record per line. It is therefore called on the WRITE path only: a savepoint is written rarely, gated by The load-side cross-check moves from Two tests, and each was mutation-verified in the direction it owns.
Spec corrected. The earlier revision justified the seq-arithmetic boundary by saying the log is append-only with contiguous seqs, so a count names the same boundary and a damaged line still occupies its slot. That was wrong, and the section now says so and says why: a substituted line does occupy its slot, but a skipped one is still a record to the raw walk. Verified on this head: 480 tests across the savepoint, projection, core, retention and routes suites, plus 701 including emit. The repo black gate, isort, flake8, comment-history and changelog-history against |
75af582 to
8f1a02a
Compare
|
Classification of the 33 red lanes on The 32 are one import failure. Reproduced with none of this PR's code present. A detached worktree at pristine Introduced by Two things follow. A job rerun cannot clear it, because a The one PR-owned red is
|
8f1a02a to
760b7ec
Compare
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/crew_log/projection.py:572 -- verified prefix can change before resumed state is served (origin: validation) |
760b7ec to
9650f26
Compare
The prefix digest is now checked after the pass as well as before it
Reproduced first, and the reproduction is the load-bearing part. The fix adds no new mechanism. Mutation-verified: neutering the new branch reddens exactly that one test out of 123. 530 tests pass across the checkpoint, projection, core, retention, routes and tree suites. Rebased onto One thing to note about that verdict rather than about the code: its adjudication stage reported FAILED TO RUN, so it stood unreviewed. It was fixed on its merits, not overridden. |
The prefix digest is read before the pass, so a write cannot certify bytes no fold saw
What changed. The expensive half of the old write path moved rather than grew: Evidence. Two tests, and a mutation restoring the write-time re-read reddens exactly those two of 41 in the file. Also on this head. The Comment History Gate was red on three added phrases that state a condition but read as history narration; they are reworded, and the gate passes against the merge base. The description now declares the new public surface with its measured consumers, and corrects the test count for that file to 41. Verified before pushing: 532 tests across the savepoint, projection, core, retention, routes and tree suites, and eight gates at exit 0, being the repo black gate, isort, flake8, mypy, comment-history, changelog-history, docs-lint and the brand check. One note on the verdict rather than the code: its adjudication recorded |
|
|
Round record: savepoint parse guard misses
|
The projection route folds a session's crew log from seq 1 on every read, so showing a session's status costs what its whole history costs. The push avoids that with an in-memory bundle, but that cache dies with the process and holds a bounded number of sessions, so a restart and an eviction each pay for the file again. Each fold's state is now written beside the log it came from, one file per fold under the unit's own directory, and resumed on the next read. Measured on a 10,001-entry log: 99.5 ms to fold cold, 3.2 ms to resume. The files are disposable: every failure falls back to folding from seq 1. Three checks retire a savepoint that would disagree with a cold fold -- the log's creation identity, the oldest surviving segment's first seq, and a log shorter than the savepoint -- and the identity is read again after the pass, so a unit recreated mid-fold is folded afresh rather than spliced. The write goes through the unit's lease so a removal cannot run underneath it, and anything written for a unit that turns out to be gone is deleted again, its directory included.
Fix record for head
|
| read | cost |
|---|---|
| cold fold | 184.4 ms |
| resume, nothing new to fold | 44.8 ms |
| resume, short tail | 105.5 ms |
| resume that also earns a write | 197.1 ms |
| cached read with a short tail | 64.2 ms |
Attribution: hashing the whole prefix is 20.6 ms, and a resume does it twice, once
on the way in and once after the pass, which is essentially the whole of the 44.8
ms. A write-earning resume also resolves the record count and hashes again
(prefix_witness 66.6 ms) plus the recheck, which is why it lands at about what
folding cold costs.
So the win on the reads that dominate is roughly 1.7x to 4x, not an order of
magnitude. The description previously said 99.5 ms cold in one place and 343.6 ms
in another for the same log, and the spec said 3.2 ms to resume; all three are
replaced by the one run above. Read them as ratios on one host, not as portable
constants.
The description's payload example also listed seven keys where save writes nine.
Corrected, the same omission that was already fixed in the spec.
Verified before pushing
736 tests across the seven owned files, and eight gates at exit 0: the repo black
gate, isort, flake8, mypy, comment-history, changelog-history, docs-lint and the
brand check. Rebased onto current main first, so the tested tree is the pushed tree.
One thing left alone
state_is_serializable has the same blind spot on its own json.dumps that was
fixed at checkpoint.py:388. It pre-exists on the merge base, so it is not this
branch's to fold in.
Disposition of the Design Review advisory on
|
| log | cold | resume, nothing new | ratio |
|---|---|---|---|
| 10,001 entries, 1.48 MB | 187.1 ms | 45.9 ms | 4.08x |
| 30,001 entries, 4.47 MB | 542.7 ms | 125.4 ms | 4.33x |
The ratio holds, and if anything it improves. Both sides are linear in the prefix:
the fold decodes and folds every record, the guard hashes every record. Tripling the
log scales both by about the same factor, so there is no term that decays. The win
is scale-invariant in entry COUNT; what it does depend on is the mix of reads, which
the table in the description gives.
Suggestion 1 is sound, and is deferred rather than refuted
Skipping resumed_prefix_still_verifies when the pass folded zero entries is a real
saving, and the reasoning is right: load is the last thing that touched the file,
so rechecking its digest immediately afterwards covers microseconds in which nothing
was read.
Worth recording why that is NOT true of the identity recheck on the same path, since
the two look symmetric and are not. origin is sampled at the top of the pass,
BEFORE load runs, so rechecking it afterwards covers a window in which a real
recreation can land -- and it is pinned by
test_an_unchanged_tail_still_rechecks_the_log_identity, which shows the recheck is
what buys the cold retry. The digest has no such gap, because the reading and the
recheck sit next to each other.
Deferred on cost, not on merit. It changes no answer, only the time to reach one,
and the head it would need re-rolls GPT and Opus, which have both just returned
clean.
Suggestion 2 is out of scope
Memoizing a verified prefix across calls would remove the dominant per-read cost,
and it is the right next step. It is also a new mechanism rather than one more term
in an existing guard: a cross-call cache needs its own invalidation story, and that
story has to answer the same question this PR just settled -- whether a digest read
at one moment is evidence about the bytes a fold consumed at another. A cache keyed
on (origin, prefix_records, prefix_sha) is keyed on the digest it is meant to
avoid recomputing, so the key has to come from somewhere that is itself trusted.
That belongs in its own change, argued on its own.
Final verification on
|
| check | result |
|---|---|
| lanes, newest attempt each | 90 -- 79 pass, 11 skipped |
| failing | 0 |
| cancelled with no later success | 0 |
| running | 0 |
| combined commit status | success |
| PR Readiness | success |
mergeable |
MERGEABLE |
| unresolved review threads | 0 of 0 |
active BLOCK-MERGE marker on this head |
none |
| incomplete-verdict notices | 0 |
green_age.py |
exit 30, STALE |
The raw check-run count equals the deduplicated lane count, 90 either way, so no lane
has a superseded attempt hiding under it. All four reviewed-marker trailers name this
exact head. The two BLOCK-MERGE markers in the thread name 065556f5 and
760b7eca, both superseded.
The one failing item is a heuristic artifact, not a collision
green_age.py reports STALE. Its whole finding is this:
base +1 commits (f8d77e49 -> 10b0fb81)
overlap: test/test_memory_multistore_stress.py (import)
The commit is 10b0fb81b4 (#12309), test(memory): say what the mixed-load budget caught when it expires. It is +49/-5 in ONE file, and that file is a test. It changes
no source at all.
The import class fires on the bare top-level package. The script's own extractor
records kiro_crew as an imported target, deliberately, because from kiro_crew import context may bind a submodule rather than an attribute. Its matcher then
accepts a hit when changed.startswith(target + "."), and every module in this diff
is under kiro_crew.. So the pair that matched is kiro_crew against
kiro_crew.crew_log.store -- which means any PR touching any file under
src/kiro_crew/ overlaps any moved file that imports the package by name. That is the
script erring toward reporting, which is the right default for a merger's advisory.
Checked substantively rather than argued:
- the moved file never names
crew_log, at all; - none of the six modules it imports (
context,embeddings,executors,
vector_memory,config.loader,slack.gateway) importscrew_logeither, so it
cannot reach this diff transitively; - the commit touches no source, so it cannot change behaviour these tests rest on.
No rebase is proposed on this. The script's own docstring calls it information for
the merger and never a merge gate, and a new head would re-roll four non-deterministic
review lanes that have just returned clean on this one.
The description's figures stand on this head
Re-verified against the live body: the re-measured timings are present, the payload
example lists the nine keys save writes, and the test count reads 44, which is what
test_crew_log_checkpoint.py collects. No stale figure remains outside the sentence
that retracts it by name.
iamwhatever
left a comment
There was a problem hiding this comment.
Approved on the current head: CI green, readiness passed, no security-sensitive path in the diff.
Problem / Motivation
Open a long-running session's side panel and the gateway reads that session's
whole crew log to answer it. Every projection (
status,usage,timeline,tools,approvals) is a fold from entry 1, so a session with a long historycosts more to look at than a fresh one, and it costs that again on every refresh.
The live push already avoids this for sessions it is watching, by keeping the
fold state in memory. That memory dies with the process and holds only a bounded
number of sessions, so a gateway restart and a cache eviction each go back to
reading the whole file.
Why it matters
The panel gets slower the longer you use a session, which is backwards: the
sessions worth looking at are the ones that have been running. A 10,001-entry log
takes 184.4 ms to fold, and that is one session, one refresh, on a warm page
cache. The projection route holds a worker thread for the whole read, so a few
long sessions refreshing together spend real gateway capacity re-deriving numbers
that did not change.
What changed (motivation -> approach -> change)
Folding is cheap per entry and unbounded in total, so the fix is not to fold
faster. It is to stop folding from the start. The fold surface was already built
for this: a
Checkpointis(name, seq, state), JSON-serializable, and themodule's own docstring called that shape "what a later checkpoint file would
carry". It just had nowhere to live.
Now it does.
src/kiro_crew/crew_log/checkpoint.pywrites each fold's statebeside the log it came from, one file per fold, at the layout the RFC's section 3
names:
src/kiro_crew/crew_log/projection.pyresumes from it instead of from entry 1,and
docs/system-specs/modules/crew-log-projection.mdrecords the layout, theretirement rules and what is deliberately left out. The same 10,001-entry log now
answers from 24 KB of files instead of refolding, and the read verifies the bytes it is skipping. The honest figures, re-measured on one host after the digest guard landed, best of five: 184.4 ms to fold cold, 44.8 ms to resume with nothing new to fold, 105.5 ms with a short tail, and 197.1 ms on the read that also earns a new savepoint. So the win on the reads that dominate is roughly 1.7x to 4x, not an order of magnitude, and a write-earning resume costs about what folding cold costs. An earlier version of this sentence quoted 19.6 ms of hashing against a 343.6 ms cold fold and called the guard 6% of the work it protects; that compared the guard against the read it does NOT run on, and the 343.6 ms also disagreed with the 99.5 ms this description used elsewhere for the same log. Both are replaced by the one run above. State the limit plainly: a savepoint saves the FOLD work, not the parse.
iter_fromdrops entries below the requested seq after constructing them, so a resumed read that has anything to fold still decodes the prefix it skips folding, and the digest adds hashing passes over those bytes -- one on the way in, one after the pass, and two more when a write is earned. A read is not independent of log size. One file per fold rather than one for allfive, so a payload this build cannot read costs that fold its savepoint instead
of costing all of them.
loadandsavenever raise.prefix_shaandprefix_records, and a resume recomputes the digest before trusting the file. Without it the module's promise rests on an assumption STRONGER than the store's own posture:_iter_entriesskips a damaged interior line on purpose, so a cold fold omits that entry while a savepoint written before the damage keeps the value it folded, and the savepoint is the one that looks clean. The digest is over raw record bytes and decodes no JSON, which is why it is affordable; it counts records rather than reading seqs, because reading a seq means decoding its line. New public helperCrewLog.raw_prefix_digestinstore.py, one consumer:crew_log/checkpoint.py.since=bundle records no digest --SessionProjectionshas no such field -- and the bytes below its seq were consumed by a call that already returned, so a digest taken now is honest about the file and wrong about the state beside it. Those two would then agree with each other, every later resume would recompute the same bytes, match, and serve that state for the life of the unit. So an incremental read takes no witness andsavewrites nothing without one. The cost is that the dashboard push loop's cached sessions bring their savepoint forward on the next read that folds the prefix rather than on every publish, which is the lag the row above already accepts.SessionProjections.originalready compares, soprojection.py's private_log_originis RENAMED to publiclog_originforcheckpoint.pyto import. That is the whole of the rename, and it is one new public name with one consumer: two spellings of "same log" could disagree, and the lenient one would fold a retired file's state onto a live file's bytes.CHECKPOINT_VERSION, and a test digests each fold's stored state so forgetting the bump fails CI rather than serving pre-fix numbers for the life of a session. One global number over a per-fold one is deliberate: it over-retires, and over-retiring costs a refold while under-retiring serves a wrong number.New public surface, declared with its consumers because every name here is one that
two spellings could disagree about. In
store.py:CrewLog.raw_prefix_digest,CrewLog.raw_records_throughandunit_header_created_at. Incheckpoint.py:resumed_prefix_still_verifies,prefix_witness,prefix_unchanged,write_is_earned, and thePrefixWitnessthose last three carry. One importingmodule each, measured rather than assumed: the store's three are imported only by
crew_log/checkpoint.pyandcrew_log/projection.py, and the savepoint module's fiveonly by
crew_log/projection.py.savealso gains a keyword-onlyprefixwith NOdefault, deliberately: a caller that omits it would write no savepoint, which is the
safe answer, but a caller should have to say which digest it means rather than inherit
one.
No flush per write. A savepoint a crash leaves unwritten is an older savepoint or
no savepoint, and folding further answers both, so the rename's atomicity is all
this needs. The payload escapes every non-ASCII character, because a crew log's
own JSON admits a lone surrogate and a serializer that passes one through makes
the encode fail.
The savepoint directory sits inside the crew log tree, which is already hidden
from a sandboxed process and refused to the agent's file tools, so it needs no
protection of its own. The store reads its segments by name and ignores every
other neighbour, and deleting a unit deletes its whole directory.
Backwards compatibility
Compatible: the savepoints are new files that nothing reads yet, and a read with
none behaves exactly as it does today.
fold_sessionkeeps its existing signatureand gains no parameter; resuming is unconditional, and a caller that wants the
from-scratch answer already has
foldandadvance, which are that answer.Tests
test/test_crew_log_checkpoint.py, 44 tests. The load-bearing one is that aresumed fold equals a cold fold, asserted on all five projections and on the
sequence the resumed read starts from, so a test cannot pass by quietly re-reading
the file.
The rest are reasons NOT to resume, each asserted by the fold still landing on the
cold answer: a savepoint past the log's end, one whose front moved, one from a
recreated log, one naming another fold or another unit, one from a newer build,
one that is truncated, one over the size cap, and one whose state is not shaped
like the fold it claims. That last check is NEW here, not pre-existing: this PR
adds
_state_matches_fold, because a savepoint is the first thing that ever fedfold state in from outside the process, and without it a wrong-shape state resumed
and failed inside the fold's own step. It lives on
Checkpoint.from_dict, so theone other caller that parses stored state gets it too, and the savepoint loader is
its only production caller today.
Then the write rules: a short session leaves no file, a small
growth rewrites nothing, one requested fold writes one file, a lone surrogate
round-trips, a reader holding no shared claim on the unit writes nothing, a unit
whose history is gone is not touched at all, and a savepoint written as a unit
disappears is removed again with its directory.
Two cover the incremental read specifically. One asserts that a read reusing a
since=bundle leaves the savepoint on disk exactly where it was, even with a taillong enough that a write would otherwise be owed. The other shows why: with a
consumed record altered below the cached bundle's seq, a later read still agrees
with a cold fold, instead of resuming from a savepoint whose digest and whose state
describe different bytes. The second one folds
usagerather thanstatusbecausethat is the fold which READS the altered value -- against
statusit passes withthe fix reverted, which is a mistake an earlier test in this file made.
One test guards the guards rather than the code: it digests what each fold stores over a fixed script with the clock frozen, so a fold whose meaning changes while its keys do not reddens CI and the failure names the version bump that retires the savepoints written before it.
Every guard was mutation-verified: removing it makes a named test fail. The run is
in a PR comment.
Manual verification
N/A: unit coverage sufficient. The read path has no UI in this change, and the
timing figures quoted above come from folding a generated 10,001-entry log both
ways in one process.
Related Issues
no linked issue: this is the fold-checkpoint work item from the crew log plan, not
a reported defect.