Repository navigation
perf(crew-log): replay each unit's log from a watermark on boot - #34882
Conversation
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the watermark goes after the cause in #34846 (a boot's cost tracked log size, not what changed) and deletes the two scan caches it replaces. Not checked: real-home M1 on this head. The description says it was not re-measured. What this change shipsInventory (8 items) — 8 justifiedIntent: a gateway restart rebuilds the session tree by reading only the session-log bytes written since the last boot. A FIX (performance, #34846).
[FIRST-PRINCIPLES-REVIEWED] 23aea82 Description read[DESCRIPTION-READ] 043947717344eddb58a1643b268b5a070d80452eb251de9cfcf0e8e7d024b61c That sha256 names the PR description this verdict read. Recompute it with |
2597b64 to
843c890
Compare
|
Re the GPT 6.1 review of 2597b64 (#34882 (comment)): both BLOCKING findings are fixed in head 843c890.
The store facade now binds and declares the new names ( |
843c890 to
ec3530c
Compare
|
Re the GPT 6.1 review of 843c890 (#34882 (comment)): the BLOCKING finding at A final record that is whole but has no trailing newline is now folded, as the tail reader folds it, and the watermark stays before it. The next read takes it again and folds it to the same answer. A final record that does not parse is still left for the next read, and nothing after it moves the watermark. Tests: |
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 20% chance that this change introduces a defect, above the 12% limit for approving on its own. Each unit now gets a saved mark recording how far the last boot read its log, and the next boot reads only the bytes after it. Every case where the mark might be wrong falls back to main's full read: the cut counter is missing, odd or has moved; the oldest or an older segment changed; the newest file has a new inode or is shorter than the mark; the read stopped short of end of file (checked with An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the per-unit watermark is the right shape, any doubt falls back to main's full read, and bumping Notes
[DESIGN-REVIEWED] 23aea82 Description read[DESCRIPTION-READ] 483039377bcccb7d0265b3dbb0ee2c10d4e90f2fb6fd0cd5d4ba8352f602cacf That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
ec3530c to
78af7aa
Compare
|
Re the Opus 5.5 review of ec3530c (#34882 (comment)): the BLOCKING finding at A unit with no The same head also makes a unit whose spawned entries would pass |
78af7aa to
fc4b6e9
Compare
|
Re the GPT 6.1 review of 78af7aa (#34882 (comment)): the BLOCKING finding at When a checkpoint decision or spawn row does not decode, the loader now drops the watermark of the unit the row names, so that unit is read in full and the decision comes back from the log. A row that names no unit discards the whole checkpoint (cold rebuild). Test: The same head also fixes the two CI reds on 78af7aa:
Real-home M1 2.6008 -> 0.9042 s, M4 identical tree; numbers are in the PR body. |
fc4b6e9 to
2697e58
Compare
ca206b5 to
f65d8f5
Compare
|
Re the GPT 6.1 review of ca206b5 (#34882 (comment)): both triggers at Every way
Trigger 1 also needed the full read's decision to replace the held one outright: the held adoption and the release share a seq, so Intact logs stay on the delta path: synthetic M1 0.0261 / 0.0287 / 0.031 s (main 0.0329 / 0.1854 / 1.388 s), M4 identical. Real-home M1 is still not re-measured (busy-host ssh refused again at 02:04Z). |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Thanks for the fixes. I checked both items from my earlier review on this head:
- Root-listing failure kept marks for dropped spawns: fixed.
crew_log/session_tree_projection_seed.py:447now filters the loaded marks to held records before the listing, and thes-gonecase is back intest_spawns_are_kept_and_reported_when_the_root_cannot_be_resolved. - Marks outlived record eviction: fixed. Marks are filtered to held records at install (
seed.py:373), eviction pops the mark, and_MARKS_CAPis gone.
One thing still needs a change before merge:
test/test_session_tree_projection_watermark.py:441and:178: two new tests run close to the per-test timeout. The CI duration tables on this head showtest_the_tree_after_an_append_equals_a_full_read_at_every_size[16000]at 97 s of the 120 s cap on Linux and 166 s of the 180 s cap on Windows.test_a_boot_after_a_small_append_reads_only_the_append_at_every_log_sizetakes 64 s on Linux and 109 s on Windows. They are the slowest tests in both shards. On Windows the suite runs with--max-worker-restart=0, so a slower runner pushes the 16000 case past the cap. pytest-timeout then kills the worker and the whole shard aborts, which turns CI red for unrelated PRs. Almost all of the time is setup:_unit(line 133) writes each filler line through its ownCrewLog.append. The 16000 equivalence case makes about 32k appends, and the small-append test makes 21,250. Smallest fix: write the filler with one raw bulk write per unit, as_raw_lineand your bench synth mode already do, and keepCrewLog.appendonly for the records the tests assert on. Or drop the 16000 size:DELTA_DECODEDdoes not depend on size, so 250, 1000 and 4000 already show that the read stays flat. Please check the slowest-durations table on both OSes after the change.
Optional:
- The seed now goes through
CrewLog.open, so for each changed unit it takes the append lock and can truncate a torn tail. Before, it only read. One sentence in the PR body would make that visible to reviewers. - The new checkpoint-loader branches in
session_tree_projection_checkpoint.pyaround line 501 have no test. Two cases would pin them: anedgesrow with nosid(the loader returns None), and a damagedspawnedrow (that unit's mark is dropped). - The current CI reds in
test_update_boot_apply.pycome from main and were fixed there after your merge-base, so a rebase should clear them. The body line "the same file passes on main" is true only for the equivalence cases, because the file reads_marks.
f65d8f5 to
7b1a8f6
Compare
|
Thanks @KiroCrewCaptain for the review (5868782641 on f65d8f5; 7b1a8f6 is the same patch rebased onto main). Both earlier code findings are confirmed fixed:
The new blocker is real: two tests in
Smallest fix we would take: write the filler lines with one raw bulk write per unit (as The PR is parked as draft. This fix waits for Mingwei's GO; nothing more will be pushed until then. |
The session tree checkpoint now stores, per unit, where the last boot's read of that unit's log stopped (oldest and newest segment, newest segment inode, byte offset). A boot reads only the bytes after it, in one pass for decisions and spawns. A unit with no mark, or whose segments changed under it (new or removed segment, replaced or shorter file, offset not on a record boundary), is read in full. The scan caches (scans, spawn_scans) leave the checkpoint; CHECKPOINT_VERSION 5 -> 6. Refs #34846
7b1a8f6 to
23aea82
Compare
|
Re the KiroCrewCaptain review of f65d8f5 (#34882 (review)): the test-duration blocker is fixed in head 23aea82. CI on that head: 98 ok, 0 fail. Both earlier code fixes stand unchanged ( The test fix is test-only: Slowest-durations tables, New-test stability jobs, slowest of the passes:
Could you take another look? |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
This review checked head 23aea82. It read the whole diff. It ran the watermark, seed, checkpoint, spawned, adopt and facade tests on this head (all passed), and ran the same crew-log tests with the diff applied onto current main (all passed). It ran the equivalence cases against main's code, where they also hold. It confirmed that no reader of the removed names (tree_edge_scan_identity, scans, spawn_scans) is left, and that git merge-tree against main is clean.
Earlier review items:
- Root-listing failure kept marks for dropped spawns: fixed on the previous head and still fixed here (
seed.py:447,s-gonecase restored). - Marks outlived record eviction: fixed on the previous head and still fixed here (filtered at install, popped on eviction,
_MARKS_CAPremoved). - Watermark tests near the per-test timeout: fixed.
_unitnow writes the filler in one raw write. The 16000 case ran in under 1 s locally, and no watermark test appears in the slowest-durations table of any backend shard on this head.
Done well: every case the watermark cannot vouch for falls back to main's own full read, and the equivalence suite proves this against a cold SessionTree().reading at four sizes and ten change-while-down cases.
Optional:
- The mark drop on decision eviction at
session_tree_projection.py:672still has no test that fails without it. A test withTREE_UNIT_CAPpatched small that evicts a held unit's decision and boots twice would pin it. - The busy-home row (2.5605 s to 0.5203 s) was measured on an earlier head. Re-running it on 23aea82 and posting the result would back the headline number for homes over the unit cap.
test_a_boot_after_a_small_append_reads_only_the_append_at_every_log_sizeasserts decoded records, not bytes read. A name like "decodes only the append" would match what it checks.
Problem / Motivation
Goal: A gateway restart reads only the session-log bytes written since the last boot to rebuild the session tree, not every unit's whole log.
On a real home, any unit that got even one new line since the last boot had its whole log read again by the session tree seed, twice (once for adopt/release decisions, once for spawns), with every record JSON-parsed. The checkpoint held the tree's state but not how far into each log that state reached, so the seed could not tell what it had missed.
Why it matters
Startup session restore spends tens of seconds in this seed on a busy home (#34846: 14.3 s in
find_last_tree_edgeplus 5.0 s infind_spawned_entries). The cost grows with the size of the logs, not with what changed, so it gets worse the longer workers run.Not a goal
_window_can_wait, item 4 of Startup session restore re-reads whole crew-log units and worker transcripts on every boot #34846) andrestore_inputs.py:_seed_session_treeare not touched.TREE_UNIT_CAPon an over-cap store are unchanged (same path as main).What changed (motivation -> approach -> change)
The checkpoint now keeps, per unit, a watermark: the
ResumePointits last boot'sCrewLog.iter_fromwalk reached (segment, record offset, seq, cut count), thesession/spawnedentries counted so far, and the identity of its oldest segment and newest file (inode, size, mtime). A unit whose files are unchanged is not opened. Any other unit is read throughiter_from(resume=..., cursor=...), which verifies the point against the cut count and the record there, and the decisions and spawns it yields are folded with the rules the full read uses.One unit is read in full when it has no watermark, its oldest segment changed (retention), its cut count moved or proves nothing (a rolled-back append is a cut), its newest file was replaced or is shorter than the mark, or its spawned entries would pass
SPAWNED_PER_UNIT_CAP(so the cap drops the same entries main drops). A full read replaces that unit's held answer, so a decision retained out of the log is still dropped. A checkpoint decision or spawn row that does not decode drops that unit's watermark; a row that names no unit discards the checkpoint. If the newest decision past the mark is one the full read refuses, the held decision is dropped as the full read drops it. Ifiter_fromskips a record in the walked range (damaged or oversized, the end of the log included: the walk's last record must end where the file ends) or refuses the segments, that unit is read the way main reads it, so an oversized record keeps the held evidence, reports the unit incomplete and leaves the mark before it.A watermark is held only for a unit whose record the tree holds: when
TREE_UNIT_CAPevicts the record, its watermark goes too, and a unit that comes back pays one full read, as on main. The negative-verdict scan caches (scans,spawn_scans) andtree_edge_scan_identityare removed;CHECKPOINT_VERSIONgoes from 5 to 6, so the first boot after upgrade rebuilds cold. No setting is added, and the tree is the same as main's (M4 below). The seed now opens each changed unit throughCrewLog.open, so it takes that unit's append lock and can truncate a torn tail, where before it only read.GIL time held by one boot seed on a busy-home copy, measured before the watermark stopped outliving cap eviction: 2.5605 s -> 0.5203 s. Real-home M1 was not re-measured on the current head: the busy-host ssh was refused at 00:05Z. Evicted units now pay one full read on return, as on main, so that number may rise; re-measure pending.
Backwards compatibility
Compatible: the only reader of the checkpoint file is this module; an older file fails the version check and is rebuilt by the existing cold scan, which is the documented recovery.
Reader: src/kiro_crew/crew_log/session_tree_projection_checkpoint.py:_load_checkpoint -- release -- reads
marksinstead ofscans/spawn_scans; a v5 file is discarded (test_a_checkpoint_it_cannot_prove_loads_as_none[older-version])Reader: src/kiro_crew/crew_log/session_tree_projection_seed.py:_seed_inner -- crew page -- the only caller of the removed
_replay_spawned, now one_replay_tailpass (test_session_tree_projection_watermark.py)Reader: src/kiro_crew/crew_log/session_tree_projection.py:SessionTreeProjection -- crew page --
_scans/_spawn_scansreplaced by_marksat every site (test_crew_log_session_tree_spawned.py)Tests
test/test_session_tree_projection_watermark.py(new):.cuts, a re-spawn past the per-unit cap, and a creator past the unit cap behave as main;SessionTree().reading); the equivalence cases also pass on main.Benchmark (M1-M4 of the perf acceptance doc), same machine, before = main, after = this PR. M1 is
time.thread_time()around the seed in its own thread, in a separate process, median of 5. M2 is a probe thread sleeping 5 ms while 4 seeds run at once (lateness in ms). M3 varies only the total log size (100 units with 200, 2000 and 20000 lines each, and a read-only copy of a busy home of 2.5 GB and 5218 units) with a fixed small change: one line on 10 units, onesession/spawned, onesession/adopted(the busy home took its own real appends over about 10 s). Each code boots cold, then replays (3 more boots on the busy home), then the change lands, then the measured boot runs from that code's own checkpoint. M4 compares a hash of the whole tree.Checkpoint size on the busy home: 1901384 bytes on main, 1630577 bytes here. Each changed unit pays a fixed read cost in
CrewLog.openanditer_from(the 72 KiB tail window and header/first-record reads of up to 1 MiB each), so bytes read at the small sizes are above main's while M1 stays flat across the synthetic sizes (0.0262 / 0.0289 / 0.0315 s, main 0.0342 / 0.1815 / 1.4011 s). The busy-home row is a different store (5218 units, over the 4096 cap, with real appends), not the same fixed change.bench_tree.py
bench_driver.py
Raw output (summary lines)
Manual verification
The benchmarks above ran on read-only copies; no live home was written.
Follow-up
PR C, not built here. After this PR and the guard PR (direct crew-log segment opens limited to an allowlist) both merge, and Audit B's list is fixed or allowlisted with reasons, PR C makes an over-budget crew-log read on the gateway process raise an error by default;
offstays the only escape hatch. Direct segment opens are routed through the counted CrewLog entry points or covered by the guard test. PR C tests: a delta read passes, a from-the-top read on the gateway raises, a child process is unaffected, and every remaining boot and runtime caller stays under budget on a real-home copy.Also a follow-up: trim the header and first-record reads in
CrewLog.open/iter_from(up to 1 MiB each,jsonl_util.py_FRAME_CHUNK) so a resumed read of a changed unit costs only its delta.Related Issues
Refs #34846
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)