Repository navigation
test: keep the whole-tree scans inside the per-test timeout - #20459
Conversation
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (10 items) — 10 justifiedIntent: stop whole-tree scan tests from timing out on slow CI shards and turning unrelated PRs red. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 33f4a56 Description read[DESCRIPTION-READ] dece5fd33af4e52129dc62d56910000d836c71f338adbddda8bfa7abf0d01606 That sha256 names the PR description this verdict read. Recompute it with |
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this changes only tests, can be reverted cleanly, and the literal narrowing is restricted to scans of names, which the source text has to spell. Notes
[DESIGN-REVIEWED] 33f4a56 Description read[DESCRIPTION-READ] 48fc9d80d644db3bee8b7d866656b50c870b0c030c0e044ca94897838175d937 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
b384b6e to
01656d5
Compare
01656d5 to
adfe075
Compare
|
Disposition for the independent review of 01656d5 (NOT READY, 1 blocking / 5 should-fix / 4 nit). New head: adfe075, rebased on main 740bf62. One commit, test-only. B1 (guard red on the merge with main): fixed by subtraction. The hand-typed SF1 (whole-tree scans outside any group): fixed. SF2 + SF4 (stale table, 4x floor vs 8x convention): fixed, Design's simpler option taken. SF3 (docs): fixed. SF5 (missed shapes): fixed for the cheap shapes. The guard now detects N1 (stale body counts): fixed. The body is rewritten for this head: 195 scan tests, 2 explicit marks, and 118 tests in the forced-timeout run. Proofs on adfe075:
First-pass check: R1-R8 OK. The AUTOSDE rules that match: |
adfe075 to
928d12f
Compare
|
Disposition for the independent delta review of adfe075 (APPROVE-WITH-SHOULD-FIX, 0 blocking / 3 should-fix / 5 nit). New head: 928d12f, rebased on main 213339f. All eight items are fixed; nothing is deferred.
Checks on adfe075: Backend Tests 3.12 (6) and Windows (6) were main's codeql Tests on 928d12f: the guard 12 passed (3x, and in reverse order). Touched modules and neighbours at First-pass check: R1-R8 OK. |
706dbd3 to
45b3113
Compare
|
First Principles BLOCK on
|
45b3113 to
054ab5d
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Blocking finding: the raw-source prefilter in test/test_memory.py narrows the prior AST scan. Adjacent literals like '1' '4 full days' and escaped digits still form a forbidden AST string, but the new require_all=("14", "full", "days") filter skips the file.
Required fix: preserve every Python string form the old scan detected. Add regression tests through the real prefilter for adjacent literals and escaped digits.
054ab5d to
673bc4d
Compare
|
@bolichen97 Fixed in 673bc4d, answering your review. The decay gate no longer filters on The regression tests go through
With the filter set back to 054ab5d's, all 17 spelling tests fail. With two modules planted in the package tree, one holding the adjacent-literal copy and one the escaped-digit copy, 054ab5d's gate passes and this head's fails naming both. On this head the filter keeps 20 of 3,589 modules, the same 20 the plain words keep, so the gate costs 0.7 s of CPU against 14 s for parsing every module. The other narrowed scans match names (identifiers, attributes, keywords, |
673bc4d to
79fcc48
Compare
|
@bolichen97 One more revision, 79fcc48. I probed the filter from my last comment with adversarial input and found that its join backtracked. After a quote, a run of backslash-CRLF line joins could be read two ways per unit, so a search that found nothing doubled in cost with each unit: 0.7 s of CPU at 21 units.
The 17 spelling tests still fail with 054ab5d's filter. The two planted modules still pass 054ab5d's gate and fail this one. The rebase onto b0f93f7 also takes in #23010's new |
The slow whole-tree scans timed out together on a slow shard of PRs that touched none of the code they guard, because each parsed every module though a hit can only be in a file whose text spells the names it matches. Narrowed through source_corpus.parsed_candidates with the literal a hit must spell: the vouching-binder, pathres and histogram scans, and one slow scan in each of 13 more modules, which drop the tree_scan group an earlier revision gave them. The icacls rationale guard skips a file with no icacls mention before matching its lines. The history decay gate in test_memory.py matches str values, which an escape, a backslash-newline or adjacent literals spell without the words, so no literal can narrow it: it parses every module, in a function-level tree_scan group. test/conftest.py gives every test in a tree_scan group that has no timeout value of its own a group default of 300 s, eight times 37.5 s. The two scans measured above 37.5 s of CPU carry their own larger timeout. Modules whose scans no literal can narrow keep their group, with a comment saying why. test_tree_scan_budget.py pins that hook. testing-conventions.md says to narrow first, group second, and budget third, and that a scan of str values cannot narrow on the words it looks for. Closes kirodotdev#19907
79fcc48 to
33f4a56
Compare
|
@bolichen97 The pre-filter is gone in 33f4a56, answering your review, so the decay gate parses every module again and sees every string form the old AST scan saw. The full scan takes 15.2 s of CPU on a fast host (3 runs, 3,602 modules), and it runs in a function-level
|
|
HOLD (active change request) at 33f4a56. bolichen97's change request on 054ab5d is still active. On this head the decay gate's text pre-filter is gone: The rest is test-only. The narrowed scans filter on identifiers a hit must spell, and the Clears when: bolichen97 re-reviews 33f4a56 and dismisses or approves. — crew-pr-reviewer (r124) |
bolichen97
left a comment
There was a problem hiding this comment.
[operator: bolichen97] My earlier change request is addressed on 33f4a56. Approving to clear it; other open gates on this PR still apply.
Problem / Motivation
Goal: every whole-tree scan stays inside its per-test timeout on a slow CI runner and catches everything it caught before, by parsing only the files that can hold a hit, with a
tree_scangroup default for the scans no literal can narrow.The slow scans take 20-45 s of CPU on a fast host and about three times that on a hosted runner, so they time out together on one backend shard of PRs that touch none of the code they guard. #19907 has the first sightings. Shard 8 of #20292 then hit the facade and binder scans, and shard 2 of #20313 hit four more (
test_host_service_guardx2,test_wait_discipline_ratchet,test_link_screen_hold_pin).Why it matters
A shard that times out turns Coverage Gate and PR Readiness red for an unrelated PR, and only a fresh head clears it.
Not a goal
--timeout=120.What changed (motivation → approach → change)
Root cause: each slow scan parses and walks every module, though a hit can only be in a file whose text spells the names the scan matches.
source_corpus.parsed_candidates(require_all=/require_any=), each with a comment naming the literal a hit must spell: the vouching-binder scan, the pathres call-site scans, both histogram scans, and one slow scan in each of 13 modules that an earlier revision grouped instead:test_external_logout_detection_retirement(_session_map),test_dir_fd_create_race_ratchet(dir_fd+O_CREAT),test_transport_permission_floorx2 (approve_tool),test_runtime_death_is_a_process_event(AcpProcessDied; it reads the dashboard runner's phases from everychat_turnmodule, because a phase owner need not name the exception),test_model_scope_wire_paths(the scope entry points),test_instance_credential_pairing,test_overload_integration_glue,test_telegram_parity,test_cloud_config,test_agent_sdk_host_auth,test_messaging_link,test_hooks_edit_gate_diff_path,test_transcript_derivation_seam. Each of those literals is a name the hit spells (an identifier, attribute, keyword orexceptname), which the source text must hold. The 14th,test_icacls_rationale_guard, reads Markdown too, so it skips a file whose text has no case-insensitiveicaclsbefore matching its lines. Those 14 modules drop their group line.test/conftest.pygives every test in atree_scangroup with no timeout value of its ownTREE_SCAN_TIMEOUT_SECS = 300. A valueless mark (timeout(method="thread")) takes the closest explicit value at any level (a moduletimeout(600)stays 600), else the default; its method is kept.test_dashboard_files_composition_contract(its tree-route check runs the link-screen gate, whose resolver names come from a call graph over every module) andtest_source_corpus(the walk ratchet already filters onwalk(and.parent, its narrowest necessary text). Four whole-tree scanners join the group in this PR. One test joins as a function-level group:test_memory's history decay gate matchesstrvalues, which an escape (\x31), a backslash-newline or adjacent literals ('1' '4 full days') spell without the words, so no literal can narrow it and it parses every module.timeout(420)) andtest_posix_scope_is_unfiltered(40 s,timeout(360)).test/test_tree_scan_budget.py, 160 lines. pytester pins for the hook: the default for a module-level and a function-leveltree_scangroup, an explicit value kept, a valueless mark taking the closest explicit value, and the default staying above the suite--timeout. There is no static mark check: the hook reads the closestxdist_group, so a function-level group already gets 300 s. The function-level group on the narrowedtest_the_history_append_has_exactly_one_call_siteis dropped; the one on main'stest_every_name_the_tests_patch_on_the_facade_reaches_every_owner_readerstays. Known gap, documented in its docstring andtesting-conventions.md: a slow scan with no group runs at 120 s and nothing flags it; the fix is to narrow it.testing-conventions.mdandbuiltin_skills/kirocrew-dev/writing-tests/SKILL.md: narrow first, group second, explicit budget third. A literal narrows a scan of names, never a scan ofstrvalues.Why 300 s: the slowest scans that take the default cost 30-37 s of CPU alone, and a hosted runner is about 3.1x slower. 300 s is eight times 37.5 s.
Trade-off: the default is scoped to the group. 4,343 tests in 52
tree_scanmodules (5,639 in 65 before this revision) took it at eda7400, so a hang in one of those surfaces after 300 s instead of 120 s.Why not group every slow module: it serialises each module on one worker and lifts its unit tests to 300 s. Why not a reachability guard over the suite: it cannot see an unmarked module either, and reading every module costs about 175 s of CPU. Why no static mark check: the hook already covers a group at any level, so the check would only forbid per-test groups that are a deliberate choice. Why not an explicit
timeout(300)per grouped module: a module that gains the group without the budget runs at 120 s, which is #19907 itself; the hook ties the budget to the group mark.Backwards compatibility
Compatible: test-only; no assertion on main is removed and every scan's expected inventory is unchanged.
Tests
Rebased on b0f93f7 (2026-10-23). This revision answers bolichen97's review of the decay gate's filter by removing the filter, as First Principles proposed on 673bc4d:
test_every_history_decay_description_matches_the_tiersruns in a function-leveltree_scangroup. The full scan takes 15.2 s of CPU on a fast host in each of 3 runs over 3,602 modules, about 46 s at three times that on a hosted runner, inside the group's 300 s. Run alone with--timeout=1it passes in 16.0 s, so the group's 300 s reaches it.test_the_scan_sees_copies_the_text_does_not_spellplants'1' '4 full days, ...'and'\x31\x34 full days, ...'and runs them through_source_decay_copies, the gate's own scan. With 054ab5d's filter (the three words in NFKC text) put back in front of it, both fail.'1' '4 full days, then summaries to day 90'and'\x31\x34 full days, then summaries to day 90', 054ab5d's gate passes and this head's fails naming both.chat_turnowners, and 36 of those 41 modules do not nameAcpProcessDied. With phases read from everychat_turnmodule, the handlers, the 128 phases, the swept phases and the unguarded charges equal main's full scan.test_memory.pyandtest_tree_scan_budget.pyat-n 8, 73 passed; the pre-push gate (black, isort, flake8, mypy, comment-history, docs lint, line cap and the rest) clean. The other touched test modules are unchanged since 79fcc48, where the 27 at-n 8passed 1,467.On 16d8b9d main split three touched modules and the edits moved with the code (see the PR comment). Older runs, rebased on 4b135be (the CPU, forced-timeout and planted lines are from eda7400):
-n 8(call CPU, before -> after): external_logout 30.7 -> 2.2 s, dir_fd 28.8 -> 4.9, transport_floor 55.8 -> 6.7, runtime_death 35.8 -> 11.1, model_scope 26.1 -> 1.6, icacls 22.5 -> 3.6, memory 23.0 -> 5.0 (with the filter this revision removes), instance_credential 22.1 -> 2.4, overload_glue 22.2 -> 3.5, telegram 21.1 -> 2.7, cloud_config 18.8 -> 1.2, sdk_host_auth 18.3 -> 1.1, messaging_link 18.2 -> 13.3, hooks_diff_path 17.7 -> 2.2, transcript_seam 57.4 -> 25.3 (three tests, 8 s each). Grouped and unchanged: dashboard_files 62, source_corpus 33. Slowest ungrouped test now: 13.3 s.-n 8 --timeout=30over the ruling's four, the six formerly ungrouped scans and the 17 modules plustest_agent_panel_routes.py: 1,584 passed, 0 timeouts.-n 8, 2,445 passed; black gate, isort, flake8, comment-history, docs lint, line cap clean; no mypy error in the guard. Semgrep 1.78.0 (semgrep/,p/python,p/security-audit): 0 findings on changed lines, run on eda7400.Manual verification
N/A: test-only change, covered by the runs above.
Related Issues
Closes #19907
Pattern harvest
Rule candidate: review-prompt
Pattern: a test that parses every module to find calls or attributes spelled with a fixed name. Read the tree through
source_corpus.parsed_candidateswith that name asrequire_any/require_all, and keep atree_scangroup only for scans no literal can narrow. A scan ofstrvalues cannot take that filter: an escape or adjacent literals spell a value its text does not hold.Checklist