Repository navigation
test(crew-log): ratchet direct opens of crew-log segment files - #34869
Conversation
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change only adds a test, a shrink-only ratchet built like the existing read-budget ratchets. Notes
[DESIGN-REVIEWED] 7284f07 Description read[DESCRIPTION-READ] 1fdb45ba0d9bf9b8694bfcb5f4d65e7e491d8578c03ce0b3fbb84355a685a08e That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
The review checked head 7284f07. The diff adds one test file and changes nothing under src/. The new test file passes on its own (5 passed) and under xdist next to the crew-log read-budget tests. Linux and Windows backend shards are green on this head. Both pinned tables match the code on main, and the static scan is clean on main. The review also repeated the hand checks from the body: removing an allowlist entry makes both checks fail, and a hand-built segment path fails the runtime check.
Done well: there are two layers, and the runtime spy catches paths the static scan cannot see. It credits each open to the first source frame and uses a thread-local depth counter, so an open is never counted twice.
- Optional: #34882, #34264 and #33462 each add a direct open that this ratchet will flag. Whichever of them merges after this PR needs a SEGMENT_OPENERS or NOT_A_SEGMENT row. A one-line note in the body saying who adds it would help.
- Optional: add a small synthetic test for
_diff. If_diffis changed to always return({}, {}), all 5 tests still pass. - Optional: the two store_retention reasons say the session-tree watermark PR moves these reads to iter_from. On #34882's head, both functions still open segments directly.
Refs #34846.
Problem / Motivation
Goal: A new piece of code that reads a crew-log file behind the read counter's back fails a test, so every such read is either counted or argued for in review.
The gateway counts how much of the crew log each read decodes (
crew_log/read_budget.py). It only sees reads that go throughCrewLog.iter_from,get,pageand the two raw prefix walks. Code that opens a log segment file itself is invisible to it. The audit for #34846 found 23 such places on main; the biggest ones re-read whole logs on every boot.Why it matters
The fixes for #34846 move slow startup reads onto counted, delta-sized reads. Without a guard, a new direct read can land next to them and nobody sees it until a slow boot shows up again.
Not a goal
What changed (motivation -> approach -> change)
One new test file,
test/test_crew_log_direct_segment_open_ratchet.py, in the same shape as the existing ratchets intest/test_crew_log_read_budget.py(pinned table, "new" and "stale" both fail, so the table only shrinks).It does two checks:
crew_logpackage, any module importing a segment-path helper from it, and any module holding both a"crew-log"and a"log.jsonl"string. Every raw file open there (open,os.open,io.open,Path.open,read_bytes,read_text,mmap) must be listed by file and function, either as a segment opener or as "not a segment". A second table pins who calls the uncounted record framer.open,io.openandos.open, and drivesappend,iter_from,get,page, the raw prefix walks,open(repair=True), a session and a slot fold, the session tree read, and the two whole-unit tree scans. Every segment open must come from a listed function.The allowlist, with the reason recorded for each entry:
Segment openers:
store_segment_reads.read_head: header and first entry only.store_segment_reads._scan_tail: last 72 KiB, for the writer's seq recovery.store_segment_reads._iter_entries_located: the framer underiter_from,get,page.store_segment_reads._read_header_line: one header line.store_segment_reads._read_first_entry_seq: first entry, to check a segment name.store_segment_writes._write_then_sync: the append itself.store_segment_writes._undo_residue,_rollback_append,_truncate_now: undo a failed append.store_handle.raw_prefix_digest,raw_records_through: counted by the budget already.prefix_extent.extent_at,extent_holds: one record at a stored offset.store_tail_repair._orphan_chunk_offset,_open_tail: NOT counted, whole newest segment onopen(repair=True).store_retention.find_spawned_entries,_scan_whole_for_types: NOT counted, whole unit; the session-tree watermark PR moves these.holders._read_from: resumes from a cached offset; no product code builds its scanner today.read._unit_id_in: first line only.subagent_replay_scope._tail: 64 bytes.subagent_replay_scope._contains_needle: byte search, no decode; whole unit on a first scan.Callers of the uncounted framer:
store_handle._iter_segments_located,_resume_at: insideiter_from, counted there.store_segment_reads._iter_entries: the plain wrapper.store_segment_reads._anchor_exists: NOT counted, walks to an old thread anchor; rare.store_segment_reads._full_scan_last_seq: NOT counted, tail window with no parseable line; rare.Not a segment: the lease file, pending-open markers, the session-tree checkpoint, the
.cutscounter (four functions), the.lockfile, the trash hold file, a directory fd ineventlog/service.py, a session-ledger key file.Order: this PR and PR A (the session-tree boot watermark) go first; PR C (the read budget's gateway error mode) comes after both merge.
Backwards compatibility
Compatible: test only, nothing in
src/changes.Removes nothing: the diff only adds a test file.
Tests
test_raw_opens_near_crew_log_segments_only_go_down: a new direct open in scope fails; a listed function that no longer opens a file fails.test_callers_of_the_uncounted_framer_only_go_down: same rule for the framer's callers.test_the_store_and_tree_readers_open_segments_only_through_listed_functions: the runtime check.test_the_scan_sees_an_open_it_was_not_told_about,test_every_reason_is_one_line: the scanner and the table.Proof: not provable -- test-only change
Shown by hand in a throwaway copy of this branch:
crew_log/projection_readers.pythat openscrew_log_path(...): the static test failed namingcrew_log/projection_readers.py: _throwaway_direct_read."log" + ".jsonl", called fromCrewLog.get: the static tests still passed and the runtime test failed namingthrowaway_peek.py: peek.Manual verification
N/A -- the change is a test; the three throwaway edits above are the check that it catches what it should.
Related Issues
Refs #34846
Checklist
Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.