Repository navigation
perf(crew-log): skip the JSON parse for lines that are not a tree edge - #35721
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. Skipping lines before the parse is the same approach [DESIGN-REVIEWED] c90904c Description read[DESCRIPTION-READ] 8e7e0c305d30388032c54f62905aa71ec418aa73984e1a0645d99bb5f3b27e27 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS This is a speed fix aimed at the real cost (one JSON parse per line), and #34846 measured that cost. It adds one private helper, and the facade's bind-every-owner-name rule is what requires re-exporting it. What this change shipsInventory (3 items) — 3 justifiedIntent: a gateway restart finds who owns each fully re-read session log faster. This is a FIX (performance).
[FIRST-PRINCIPLES-REVIEWED] c90904c Description read[DESCRIPTION-READ] 099e8d4fe60cc6ddd0b589f9aaa50e7d7132ec11b53872bcae8f6de31ad8009e That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. This adds a byte check before the JSON parse in _scan_whole_for_types: a line that does not contain a wanted type as a quoted JSON string is skipped without being parsed. That check is safe because serialize writes with ensure_ascii=True and compact separators, and json does not escape '/', so every adoption or release record contains the needle. The other JSON-line writer, pending_open.py:83, uses the same json.dumps settings. A line that does contain a needle is still parsed and its type re-checked, so the answer cannot change. Damaged or oversized records raise inside strict_raw_records before the new check, so the conversion to ValueError still works and the existing test for it still passes. The new tests count how many lines reach the parser, so they would fail without the change; one test pins the bytes serialize writes for each edge type. The CodeQL failure was a GitHub API rate limit. The Coverage Gate failure names no files; I don't think this PR caused it, since the new code runs under the new tests, but I couldn't confirm which files it means. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
`find_last_tree_edge` reads a unit's whole log when the tail window holds no adoption or release. `_scan_whole_for_types` parsed every line of that log to compare its type, and nearly every line of a busy log is something else. A record of a type holds that type as a JSON string, so a line that holds none of the wanted types is now skipped before the parse. This is the same skip that `find_spawned_entries` already makes for `session/spawned`. The type is still checked after the parse, so a line that only holds the string costs a parse and is not returned. A test pins that the store's own serialization of both edge types holds the bytes the skip looks for. Measured on a copy of real session logs, 6 interleaved rounds, host load 12 to 22, main / main again / this change, median time for the whole read: the 67 units a restart re-reads (204.6 MB) 3,716 ms / 3,131 ms / 1,190 ms every unit past the tail window (858, 438.8 MB) 9,756 ms / 10,059 ms / 4,534 ms The answer for every unit was identical in every run, and this change was slower than main in none of the 6 rounds. The tests count parses through the `kiro_crew.crew_log.store` facade, because the facade guard refuses a patch on the owner module.
70edeff to
c90904c
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed the whole diff at c90904c, tests included.
The whole-log read for a tree edge no longer parses every line as JSON. Before the parse, a line is skipped unless it contains the quoted type of one of the wanted entry types as plain bytes (a substring check with in, no regex). The type is still checked after the parse, so a line that only mentions a type text costs one parse and gives the same answer as before. find_spawned_entries already skips lines the same way.
What I checked:
- The skip is safe only if every record of those types holds its type spelled exactly as the needle. The serializer writes ASCII and does not escape the slash, and the type names are ASCII. A parametrized test writes one record of each wanted type through the real serializer and asserts the needle is in the line.
- The newest-wins result is unchanged. The tests cover a buried decision found with only that line parsed, a log with no decision parsing nothing, the newer of two decisions winning, and a turn whose text equals a decision type being parsed and then rejected by the type check.
- A type quoted inside a longer text is escaped on disk, so it is skipped without a parse. The test shows that too.
- No regex, no new refusal, and nothing outside the read path changes. The facade only gains the new helper name.
All checks are green and nobody holds auto-merge, so I am arming it.
Problem / Motivation
Goal: A gateway restart spends less time finding who owns each session log that it has to read in full.
find_last_tree_edgereads a unit's whole log when the tail window holds no adoption or release._scan_whole_for_typesparsed every line of that log as JSON to compare its type, and nearly every line of a busy log is some other type.Measured on a copy of real session logs (6 interleaved rounds, host load 12 to 22), time for the whole read:
Why it matters
This read runs during startup session restore. #34846 measured 14.3 s of a restart in it on a busy home, with every record parsed to find two entry types.
Saves memory? No. This is a speed change, and memory was not measured.
Not a goal
find_spawned_entries, which already skips by bytes.What changed (motivation → approach → change)
A record of a type holds that type as a JSON string. So a line that holds none of the wanted types cannot be one, and it is skipped before the parse.
find_spawned_entriesalready makes the same skip forsession/spawned.crew_log/store_retention.py_type_needlecrew_log/store_retention.py_scan_whole_for_typescrew_log/store.py_type_needlethrough the facade, as the other names of that module are.No spec changes:
docs/system-specs/modules/crew-log-core.mddescribes whatfind_last_tree_edgereads (every segment), which is unchanged.Backwards compatibility
Compatible: the same entry is returned for every unit. The answers were identical on main and on this branch in every benchmark run (both reads in the table above), and a test pins that the store's own serialization of both edge types holds the bytes the skip looks for.
Removes nothing: one private helper is added.
Tests
test/test_crew_log_tree_edge_scan.py(7 cases):session/adoptedand a writtensession/releasedeach hold the bytes the skip looks for.The tests count parses through the
kiro_crew.crew_log.storefacade, because the facade guard refuses a patch on the owner module.Proof: PROVEN test/test_crew_log_tree_edge_scan.py::test_a_buried_decision_is_found_and_the_turns_after_it_are_not_parsed, test/test_crew_log_tree_edge_scan.py::test_a_log_with_no_decision_parses_nothing, test/test_crew_log_tree_edge_scan.py::test_a_turn_whose_text_is_a_decision_type_is_not_one, test/test_crew_log_tree_edge_scan.py::test_a_decision_type_quoted_inside_a_text_is_not_even_parsed (head c90904c)
Determinism proof: 20/20 runs at
-n0; 10/10 runs at-n 4withtest/test_crew_log_core_segments.pyandtest/test_crew_log_cut_counter.py; shuffled order seed=9935 (7 ids) passed.Related suites (108 test files): 4,444 passed, 51 skipped. Lint (black, isort, flake8, mypy), the file line cap, the comment-history gate, semgrep against the base and
test/test_flake_pattern_ratchet.pypass.Manual verification
On a copy of real session logs under a scratch home, never the live gateway. Main, main again and this branch ran interleaved, order rotated, 6 rounds. Numbers are in the first section.
The numbers were taken on
2df02ab46ab.origin/mainhas not changed any source file this PR touches since then.Not measured: a whole gateway restart with this change, this change on top of #34882, and memory.
Related Issues
Part of #34846
This is item 2 of #34846 only. #34882 is open against the same issue and replaces most of these whole-log reads with a watermark; it still calls
find_last_tree_edgefor a unit that it reads in full, which is the read this PR makes cheaper. The diff of #34882 applies on top of this branch without a conflict (git apply --check).Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.