Skip to content

perf(crew-log): cut the fixed reads of a warm session fold pass - #38220

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix-38210
Nov 19, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix-38210

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A warm session fold pass with nothing new to fold opens the crew log fewer times and resolves the data home once.

A warm fold_session_warm pass with nothing new opened log.jsonl 5 times and ran _checked_crew_log_root 10 times, each resolving the data home again. The recap route ran three such passes per read.

Why it matters

Every eager wake, publisher backstop and panel read pays this fixed cost per pass. On a 15 s capture one 40 KB log was opened 230 times.

Not a goal

  • Making folds resume in O(delta) from record 1: that is crew log: every fold cell resumes in O(delta); no gateway read path scans from the first record #33827, and this PR does not touch the resume path, iter_from or the stored resume point.
  • Holding one file handle across the whole pass so every step reads through it. That could bring the pass to one open, but it changes how the store reads a unit and is a possible follow-up, not this fix.
  • Skipping the kind-root check for the rest of a pass, or dropping the header re-read after the pass. Both stay (see below).

What changed (motivation → approach → change)

Traced on main, nothing-new pass. Openers of log.jsonl:

  1. CrewLog.open > _settle_tail > _scan_tail (tail scan)
  2. CrewLog.open > _read_header_line (header)
  3. _fold_attempt identity before the pass: _unit_log_identity > unit_header_created_at > _proved_header (header again)
  4. held_still > log_origin after the pass (header a third time)
  5. extent_holds in _fold_attempt

Callers of _checked_crew_log_root (10): segment_paths from CrewLog.exists, CrewLog.open, extent_holds and both identity reads; _lock_path in _settle_tail; _unit_header_object directly and through crew_log_dir, once per identity read each.

Changes:

  • fresh_log_identity in projection_session_log.py: when _fold_attempt opened the handle itself, the identity before the pass takes created_at from the header CrewLog.open just read, and still stats the file for device and inode. A handle passed in by the caller is older and still goes through _log_identity. A recreation between the open's header read and the stat mixes the old stamp with the new inode; the after-pass read then returns the new stamp, the pass fails held_still, and nothing is kept.
  • resolved_home_for_pass in store_paths.py, entered by fold_session: data_home().resolve() runs once per pass. The kind root's lstat, exists, resolve() and comparison still run on every call. A data home that moves mid-pass makes the root resolve away from the cached path, and the check fails closed. The memo is a context variable, so it never crosses threads or passes.
  • _session_folds in src/kiro_crew/dashboard/handlers/crew_worker_recap.py reads timeline, approvals and tools from one fold_session_warm call while all three are eager, so the three values come from the same moment.

The 4 opens left on the nothing-new path:

Open Step it serves
tail scan in CrewLog.open settles a torn tail under the lock and reads the last seq
header in CrewLog.open parses and checks the header id, and now supplies the identity's created_at
header in held_still after the pass catches a log recreated onto a recycled inode mid-pass
extent_holds proves the prefix files held still, for custody of the cached state

With new entries the pass goes from 10 opens to 9, all from iter_from and the savepoint's extent read on top of the four above.

Latent race

The savepoint write refuses a file that moved during the pass by comparing a stat-only (size, mtime) fingerprint. A same-size rewrite that lands inside one mtime tick leaves both equal, so a savepoint was written over records that had changed. Main carries this: a probe that puts the pre-edit mtime back fails 30/30 on main. The extent's fixed tail digest (PrefixExtent.tail_sha) is not enough, because its own docstring says it cannot see an edit further back than its tail, and the edit in test_a_same_size_rewrite_during_a_pass_writes_no_savepoint is about 190 KB back. This PR's faster pass made that test hit the race more often (alone, 30 runs: main 2/30 failing, this branch before the detector 11/30).

ConsumedSpan in store_segment_reads.py hashes every record the walk frames, with its offsets, as the walk reads it. Just before the savepoint write, checkpoint.save frames that same range from disk again and writes nothing on a mismatch (one debug line; the next pass re-folds from the older savepoint). The fingerprint stays the first check. Hashing on the way in is in memory over bytes the walk already read. The re-read runs under the savepoint lease, after every other refusal, so only a write that is about to happen pays it; it costs the bytes the pass consumed and is counted in the read budget. A pass whose resumed fold failed and fell back to a cold walk writes no savepoint, since its span does not describe that walk. After: the original test and the probe each 0/30 failing on this branch.

Backwards compatibility

Compatible: what gets folded, the stored resume point and iter_from are unchanged; a savepoint is now skipped when the records its pass folded changed before the write, and the next pass writes it.

Removes nothing: no field, key, route or function is removed; the root check keeps both of its conditions.

Tests

  • test/test_crew_log_warm_pass_reads.py counts opens of log.jsonl with an audit hook and resolutions of the data home: exactly 4 opens and 1 resolution on a nothing-new pass (5 and 10 on main), and 9 opens and 1 resolution when new entries are present (10 and 14 on main). A third test checks the pass still folds the new entry.
  • test/test_crew_log_consumed_span.py: a same-size rewrite with the old mtime put back is refused for the FIRST folded record (about 190 KB below the end) and for the LAST; a unit that only grew still gets its savepoint; a byte outside the framed range does not refuse and the re-read covers exactly the framed bytes. Skipping the re-hash by hand makes the first-record test fail 30/30 (the mutation proof; see the Proof line).
  • test/test_crew_log_direct_segment_open_ratchet.py lists _holds as a direct segment open that counts itself in the read budget, as raw_prefix_digest does.
  • test/test_crew_worker_recap.py pins that _session_folds runs one warm pass for its three folds.

Proof: not provable -- other: prove.py was inconclusive because reverting every production hunk fails test collection on the new ConsumedSpan import; the mutation was done by hand: with the savepoint re-hash skipped, the same-tick probe fails 30/30

Manual verification

N/A — the counts are pinned by the tests above; macOS fs_usage was not re-run.

Related Issues

Fixes #38210

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners November 17, 2026 02:07
@jeeshofone
jeeshofone requested a review from buluoray November 17, 2026 02:07
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 802ae5bfd124ad87ebdb0cf6099609690e52d5f7 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 802ae5bfd124ad87ebdb0cf6099609690e52d5f7 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The open-count and home-resolution cuts are pinned by tests, the same-tick savepoint race fix is accounted for in the description, and the patch removes nothing.

Notes

  • The span re-read costs as much as the pass read. On a cold or fallback-resumed pass that writes a savepoint, the ConsumedSpan (created as cursor = ReadCursor(span=ConsumedSpan()) on every pass) makes checkpoint.save read the whole log a second time. For a large unit, that doubles cold-fold I/O in a PR titled perf. Measure that path, or say in the description that cold folds now pay it.
  • The ConsumedSpan race detector is a correctness fix that stands on its own, bundled into a perf PR. If a revert is ever needed, it would be cleaner as its own commit.

[DESIGN-REVIEWED] 802ae5b

Description read

[DESCRIPTION-READ] 5afeb4caa40d442b8daf17299ab23a892b856faeeda97d4dc500603466c65350

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 802ae5bfd124ad87ebdb0cf6099609690e52d5f7 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The open and resolve cuts sit at mechanism level, with exact counts pinned in tests. The savepoint-race detector is outside the Goal but rests on a race reproduced on main.

Notes

  • Item 4 — rides along: the Goal is "opens the crew log fewer times and resolves the data home once". ConsumedSpan, the new span= parameter on save and the ReadCursor.span field instead add a re-read of the consumed bytes on every pass that writes a savepoint. Its provenance is a race that fails 30/30 on main, and this PR's faster pass makes it more frequent. Subtraction: ship ConsumedSpan and its checkpoint.save hunk as a separate fix PR that merges before this one, so this PR carries only the read cuts.
  • Item 5 — rides along: cursor.span.spoil() in _drive_session means a pass that falls back to a cold walk now writes no savepoint, so the next pass also folds cold. Subtraction: none smaller without adding; it is the cost of item 4 and goes with it if item 4 splits.

What this change ships

Inventory (7 items) — 4 justified

Intent: make the warm session fold pass (paid by every eager wake and panel read) cheaper in file opens and data-home resolutions. This is a FIX for #38210.

  1. A warm pass with nothing new opens log.jsonl 4 times instead of 5, by reusing the header the open just read — justified
  2. The data home resolves once per fold pass, and the kind-root checks still run on every call — justified
  3. The recap panel reads timeline, approvals and tools from one warm pass instead of three — justified
  4. A savepoint is refused when records already folded were rewritten at the same size within one mtime tick — rides along
  5. A pass that fell back to a cold walk writes no savepoint — rides along
  6. A refused savepoint's bundle drops its origin and cuts, so the memo does not keep it — justified
  7. The span re-read is counted under a new "span" read-budget category — justified

[FIRST-PRINCIPLES-REVIEWED] 802ae5b

Description read

[DESCRIPTION-READ] 568d221f260d8c753a4581e22fbb7c26df20ab7095f7fc1e3dff7127cb432732

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 802ae5bfd124ad87ebdb0cf6099609690e52d5f7 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 802ae5b

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 802ae5bfd124ad87ebdb0cf6099609690e52d5f7 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 802ae5b

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 17, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Update on head f0e4e77.

Facade red on a054167. test/test_projection_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkers failed because resolved_home_for_pass was served across the projection facade without a TYPE_CHECKING declaration. It is now declared beside the other kiro_crew.crew_log.store names, and so is the new ConsumedSpan. The test is unchanged.

Savepoint race. test_a_same_size_rewrite_during_a_pass_writes_no_savepoint was flaky: run alone 30 times, main failed 2/30 and the previous head failed 11/30. The savepoint write compared only a stat (size, mtime) fingerprint, so a same-size rewrite inside one mtime tick got a savepoint written over it; a probe that restores the pre-edit mtime fails 30/30 on main. This head adds a digest of the records the pass framed, re-checked from disk under the savepoint lease just before the write. The original test (unchanged) and the probe now fail 0/30. The details are in the body's Latent race section.

Local lane dispositions (r4 and r5, GPT and Opus). No blocking findings in either round. Taken: the re-check moved under the lease after every other refusal, so only a write about to happen pays the re-read; a resumed fold that falls back to a cold walk spoils the span, so it writes no savepoint; the new tests patch through monkeypatch; a spoiled-span test was added; the body states that hashing on the way in is in memory and only the re-read is limited to writing passes.

Declined, with reasons:

  • Counting the re-read in the read budget as records == delta. These are the pass's own delta records read a second time, so the below-delta count stays unchanged; the only effect is that a cold pass of exactly 256 records reaches the over-budget counter one record early.
  • Releasing the lease in a finally around the span re-check. holds already returns False on OSError, and the framer reports over-cap records instead of raising, so no exception can leave the lease held today.
  • An end-to-end test of the cold-walk fallback calling spoil(). That wiring is one line; test_a_spoiled_span_refuses_however_the_bytes_read covers the refusal it causes.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 17, 2026
@neilieva

Copy link
Copy Markdown
Contributor

Thanks, the three read cuts match what I asked for in #38210. The handle now supplies the before-pass header (crew_log/projection_session.py:267). The data-home resolve is shared across the pass (crew_log/store_paths.py:88-121). The recap folds once (dashboard/handlers/crew_worker_recap.py:201-204).

On the ConsumedSpan check:

  • In crew_log/checkpoint.py:928, crew_log_dir(...) runs after _hold but before the try whose finally releases the lease (:979-980). crew_log_dir can raise CrewLogError (crew_log/store_paths.py:212). If it does, save raises even though its docstring says it never raises, and the lease stays held. Moving the check inside the try releases the lease. Returning the bundle on a CrewLogError, as the refused-span path already does (:932-938), would also keep the no-raise promise (:863).
  • The span check addresses the same-size rewrite race. Your description explains why it's here: the faster pass took that test from 2/30 to 11/30 failures. It's still a correctness change on top of the read cuts, so if it splits out easily, a separate PR would be simpler to review and revert. If not, keeping it here is fine with me.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Update on head a4300e4.

GPT F1, fixed (checkpoint.py, refused savepoint left eligible for later persistence). The trigger is outside tampering: a same-size, same-mtime-tick rewrite of a consumed record, which the append-only writer never makes. Base code persisted the same state unconditionally. The span refusal was partial, because the refused bundle kept its origin and cut custody, so the warm memo could serve it and a later pass could save it. A refused write now returns the bundle with origin=None and cuts=None, so the memo drops it and the next pass folds from the older validated savepoint. That makes the refusal complete. Pin: test_a_refused_savepoint_leaves_no_bundle_a_later_pass_could_persist fails on f0e4e77.

GPT F2, fixed (checkpoint.py, path validation escaping with the lease held). The span check now runs inside the try whose finally releases the lease, through _span_holds, which turns a CrewLogError from crew_log_dir into a skipped write with a debug log. save keeps its no-raise promise. Pin: test_a_span_check_that_cannot_place_its_path_releases_the_lease fails on f0e4e77. @neilieva, this is the same point as your first bullet; thank you for raising it.

Backend Tests (3.12, 7) on f0e4e77 was ours: test_facade_mirror_guards.py::test_every_mirrored_facade_manifest_names_what_it_forwards found fresh_log_identity forwarded by the projection facade without a declaration. It is now declared beside _unit_log_identity. Coverage Gate failed closed on that shard.

Local Opus review, round 6. Its one gap is taken: the lease test now asserts the span check actually ran before it checks the lease was released.

Savepoint race, root cause. The savepoint write compared only a stat (size, mtime) fingerprint, so a same-size rewrite inside one mtime tick got a savepoint written over it. test_a_same_size_rewrite_during_a_pass_writes_no_savepoint, run alone 30 times, failed 2/30 on main and 11/30 on the earlier head of this PR, whose faster pass hit the race more often. With the consumed-span digest it fails 0/30. The body's Latent race section has the details.

@neilieva, on splitting the span check out: I've kept it here because this PR's read cuts are what made the existing race show up in CI, and the check, its pins and the test it fixes belong together. If a maintainer prefers a split, it separates cleanly as ConsumedSpan plus its wiring in checkpoint.save and _fold_attempt.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

@neilieva, on your comment above, both points as of a4300e4:

  • The span check moved inside the try at src/kiro_crew/crew_log/checkpoint.py:932 (check at :933), whose finally releases the lease at :976.
  • A CrewLogError from the path is caught in _span_holds (:1539) and treated as a refusal, so save returns the bundle at :939 instead of raising, and the no-raise promise holds.

The span check stays in this PR because this PR's faster pass is what turns the latent race from 2/30 into 11/30 failures, so the detector fix belongs with the change that exposes it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 18, 2026
The identity read before a pass takes created_at from the handle the same
call just opened, the data home is resolved once per fold pass while the
kind root is still lstat'd and resolved on every call, and the worker recap
reads its three session folds from one warm pass.

The savepoint write also re-checks a digest of the records the pass framed,
so a same-size rewrite inside one mtime tick no longer gets a savepoint
written over it.

Fixes kirodotdev#38210
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Rebased onto current main with no content change, so the base includes #39262, which fixes the main-side test_bare_session_handle_guard red tracked in #39261. That red is what failed Backend Tests (3.12, 8) and the Coverage Gate behind it on the previous head. Added and removed lines are identical to ad5f99d.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 18, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 19, 2026 02:16

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked on this head: the cached data-home resolve still runs the kind-root lstat, exists and resolve checks on every call, and it refuses if the home moves mid-pass. fresh_log_identity reuses the header that the open just validated, and a recreation between the open and the stat makes the pass retry. Every way the savepoint span check can fail only skips a write. The recap handler gets the same three values from one pass. Reverting the read cuts brings main's counts back and turns test_crew_log_warm_pass_reads red. Turning off the span check turns 4 tests in test_crew_log_consumed_span red on 3 of 3 runs. The crew-log tests pass at head.
Done well: the same-tick rewrite tests restore the exact pre-edit mtime, so a race that failed now and then on main becomes a deterministic red.
Optional:

  • Create ConsumedSpan only when prefix_seen is set. A pass that cannot write a savepoint then skips the sha256 work.
  • Add a span row to the read-budget table and a short note on the savepoint refusal in docs/system-specs/modules/crew-log-projection.md. Also say in the body that a cold pass that writes a savepoint reads the log twice.
  • In test_crew_log_warm_pass_reads.py, install the audit hook lazily and count only opens under tmp_path or on the measuring thread. Consider a test for the fallback spoil() call too, because deleting it reds no test today.

@KiroCrewCaptain
KiroCrewCaptain merged commit 27921e9 into kirodotdev:main Nov 19, 2026
95 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A warm session fold with nothing new opens log.jsonl 5 times and checks the crew-log root 10 times

3 participants