Skip to content

test: keep the whole-tree scans inside the per-test timeout - #20459

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/19907-tree-scan-timeouts
Oct 25, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/19907-tree-scan-timeouts

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: every whole-tree scan stays inside its per-test timeout on a slow CI runner and catches everything it caught before, by parsing only the files that can hold a hit, with a tree_scan group default for the scans no literal can narrow.

The slow scans take 20-45 s of CPU on a fast host and about three times that on a hosted runner, so they time out together on one backend shard of PRs that touch none of the code they guard. #19907 has the first sightings. Shard 8 of #20292 then hit the facade and binder scans, and shard 2 of #20313 hit four more (test_host_service_guard x2, test_wait_discipline_ratchet, test_link_screen_hold_pin).

Why it matters

A shard that times out turns Coverage Gate and PR Readiness red for an unrelated PR, and only a fresh head clears it.

Not a goal

  • Changing what any scan reports. Every expected inventory and allowlist is unchanged.
  • Changing the suite-wide --timeout=120.
  • Measuring cost at run time. The guard checks marks, not CPU.

What changed (motivation → approach → change)

Root cause: each slow scan parses and walks every module, though a hit can only be in a file whose text spells the names the scan matches.

  • Narrowed through source_corpus.parsed_candidates(require_all=/require_any=), each with a comment naming the literal a hit must spell: the vouching-binder scan, the pathres call-site scans, both histogram scans, and one slow scan in each of 13 modules that an earlier revision grouped instead: test_external_logout_detection_retirement (_session_map), test_dir_fd_create_race_ratchet (dir_fd + O_CREAT), test_transport_permission_floor x2 (approve_tool), test_runtime_death_is_a_process_event (AcpProcessDied; it reads the dashboard runner's phases from every chat_turn module, because a phase owner need not name the exception), test_model_scope_wire_paths (the scope entry points), test_instance_credential_pairing, test_overload_integration_glue, test_telegram_parity, test_cloud_config, test_agent_sdk_host_auth, test_messaging_link, test_hooks_edit_gate_diff_path, test_transcript_derivation_seam. Each of those literals is a name the hit spells (an identifier, attribute, keyword or except name), which the source text must hold. The 14th, test_icacls_rationale_guard, reads Markdown too, so it skips a file whose text has no case-insensitive icacls before matching its lines. Those 14 modules drop their group line.
  • One group default. test/conftest.py gives every test in a tree_scan group with no timeout value of its own TREE_SCAN_TIMEOUT_SECS = 300. A valueless mark (timeout(method="thread")) takes the closest explicit value at any level (a module timeout(600) stays 600), else the default; its method is kept.
  • Kept in a group, one comment each saying why: test_dashboard_files_composition_contract (its tree-route check runs the link-screen gate, whose resolver names come from a call graph over every module) and test_source_corpus (the walk ratchet already filters on walk( and .parent, its narrowest necessary text). Four whole-tree scanners join the group in this PR. One test joins as a function-level group: test_memory's history decay gate matches str values, which an escape (\x31), a backslash-newline or adjacent literals ('1' '4 full days') spell without the words, so no literal can narrow it and it parses every module.
  • Two explicit budgets, for the scans measured over 37.5 s alone: the backend forwarded-name patch scan (42 s, timeout(420)) and test_posix_scope_is_unfiltered (40 s, timeout(360)).
  • Hook pins test/test_tree_scan_budget.py, 160 lines. pytester pins for the hook: the default for a module-level and a function-level tree_scan group, an explicit value kept, a valueless mark taking the closest explicit value, and the default staying above the suite --timeout. There is no static mark check: the hook reads the closest xdist_group, so a function-level group already gets 300 s. The function-level group on the narrowed test_the_history_append_has_exactly_one_call_site is dropped; the one on main's test_every_name_the_tests_patch_on_the_facade_reaches_every_owner_reader stays. Known gap, documented in its docstring and testing-conventions.md: a slow scan with no group runs at 120 s and nothing flags it; the fix is to narrow it.
  • testing-conventions.md and builtin_skills/kirocrew-dev/writing-tests/SKILL.md: narrow first, group second, explicit budget third. A literal narrows a scan of names, never a scan of str values.

Why 300 s: the slowest scans that take the default cost 30-37 s of CPU alone, and a hosted runner is about 3.1x slower. 300 s is eight times 37.5 s.

Trade-off: the default is scoped to the group. 4,343 tests in 52 tree_scan modules (5,639 in 65 before this revision) took it at eda7400, so a hang in one of those surfaces after 300 s instead of 120 s.

Why not group every slow module: it serialises each module on one worker and lifts its unit tests to 300 s. Why not a reachability guard over the suite: it cannot see an unmarked module either, and reading every module costs about 175 s of CPU. Why no static mark check: the hook already covers a group at any level, so the check would only forbid per-test groups that are a deliberate choice. Why not an explicit timeout(300) per grouped module: a module that gains the group without the budget runs at 120 s, which is #19907 itself; the hook ties the budget to the group mark.

Backwards compatibility

Compatible: test-only; no assertion on main is removed and every scan's expected inventory is unchanged.

Tests

Rebased on b0f93f7 (2026-10-23). This revision answers bolichen97's review of the decay gate's filter by removing the filter, as First Principles proposed on 673bc4d:

  • The decay gate parses every module again. The filter, its pattern helpers and its three pin tests are gone, and test_every_history_decay_description_matches_the_tiers runs in a function-level tree_scan group. The full scan takes 15.2 s of CPU on a fast host in each of 3 runs over 3,602 modules, about 46 s at three times that on a hosted runner, inside the group's 300 s. Run alone with --timeout=1 it passes in 16.0 s, so the group's 300 s reaches it.
  • The forms bolichen97 named are flagged: test_the_scan_sees_copies_the_text_does_not_spell plants '1' '4 full days, ...' and '\x31\x34 full days, ...' and runs them through _source_decay_copies, the gate's own scan. With 054ab5d's filter (the three words in NFKC text) put back in front of it, both fail.
  • End to end: with two modules planted in the package tree, '1' '4 full days, then summaries to day 90' and '\x31\x34 full days, then summaries to day 90', 054ab5d's gate passes and this head's fails naming both.
  • Runtime-death scan after the rebase: main's handler sweep now follows the dashboard runner's phases into its chat_turn owners, and 36 of those 41 modules do not name AcpProcessDied. With phases read from every chat_turn module, the handlers, the 128 phases, the swept phases and the unguarded charges equal main's full scan.
  • Gates on 33f4a56: test_memory.py and test_tree_scan_budget.py at -n 8, 73 passed; the pre-push gate (black, isort, flake8, mypy, comment-history, docs lint, line cap and the rest) clean. The other touched test modules are unchanged since 79fcc48, where the 27 at -n 8 passed 1,467.

On 16d8b9d main split three touched modules and the edits moved with the code (see the PR comment). Older runs, rebased on 4b135be (the CPU, forced-timeout and planted lines are from eda7400):

  • CPU, the 17 modules at -n 8 (call CPU, before -> after): external_logout 30.7 -> 2.2 s, dir_fd 28.8 -> 4.9, transport_floor 55.8 -> 6.7, runtime_death 35.8 -> 11.1, model_scope 26.1 -> 1.6, icacls 22.5 -> 3.6, memory 23.0 -> 5.0 (with the filter this revision removes), instance_credential 22.1 -> 2.4, overload_glue 22.2 -> 3.5, telegram 21.1 -> 2.7, cloud_config 18.8 -> 1.2, sdk_host_auth 18.3 -> 1.1, messaging_link 18.2 -> 13.3, hooks_diff_path 17.7 -> 2.2, transcript_seam 57.4 -> 25.3 (three tests, 8 s each). Grouped and unchanged: dashboard_files 62, source_corpus 33. Slowest ungrouped test now: 13.3 s.
  • Forced timeout: -n 8 --timeout=30 over the ruling's four, the six formerly ungrouped scans and the 17 modules plus test_agent_panel_routes.py: 1,584 passed, 0 timeouts.
  • Narrowed scans still catch: a planted module with one violation per gate turned all 16 narrowed tests red, each naming the plant.
  • The hook pins: on the rebased head they pass, including the new function-level pin (grouped test 300 s, ungrouped sibling unchanged). The hook: removing its "explicit value" skip turns three hook pins red (60, module 600, function 400 over module 60), and dropping the closest-explicit rule turns the valueless-under-module-600 pin red (300 != 600).
  • Gates, on 4b135be: the 29 touched test modules at -n 8, 2,445 passed; black gate, isort, flake8, comment-history, docs lint, line cap clean; no mypy error in the guard. Semgrep 1.78.0 (semgrep/, p/python, p/security-audit): 0 findings on changed lines, run on eda7400.

Manual verification

N/A: test-only change, covered by the runs above.

Related Issues

Closes #19907

Pattern harvest

Rule candidate: review-prompt
Pattern: a test that parses every module to find calls or attributes spelled with a fixed name. Read the tree through source_corpus.parsed_candidates with that name as require_any/require_all, and keep a tree_scan group only for scans no literal can narrow. A scan of str values cannot take that filter: an escape or adjacent literals spell a value its text does not hold.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 14, 2026 19:50
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 33f4a5648a74b7027b6a627d7ac4ce3b0b373fe6 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (10 items) — 10 justified

Intent: stop whole-tree scan tests from timing out on slow CI shards and turning unrelated PRs red. This is a FIX.

  1. Seventeen slow scans in 15 modules now parse only the files whose text names what they look for, using the existing source_corpus.parsed_candidates — justified
  2. The icacls guard skips any file that never mentions icacls before it checks line by line — justified
  3. Every test in a tree_scan group with no timeout of its own now gets 300 s instead of 120 s — justified
  4. Four whole-tree scanners join their own tree_scan group: refactor create guard, refactor facade, kill chokepoint, persistence writer inventory — justified
  5. Two scans measured above 37.5 s get explicit budgets of 420 s and 360 s — justified
  6. Fourteen narrowed modules leave their group, and the history-append test loses its function-level group, so they run at 120 s again — justified
  7. The history decay gate still parses every module and gets a function-level group — justified
  8. New pins check that the decay gate still finds drifted copies written as adjacent literals or escaped digits — justified
  9. A new test_tree_scan_budget.py pins how the hook resolves timeouts — justified
  10. The testing conventions and the writing-tests skill now say: narrow first, then group, then set an explicit budget — justified

[FIRST-PRINCIPLES-REVIEWED] 33f4a56

Description read

[DESCRIPTION-READ] dece5fd33af4e52129dc62d56910000d836c71f338adbddda8bfa7abf0d01606

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 33f4a5648a74b7027b6a627d7ac4ce3b0b373fe6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 33f4a5648a74b7027b6a627d7ac4ce3b0b373fe6 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this changes only tests, can be reverted cleanly, and the literal narrowing is restricted to scans of names, which the source text has to spell.

Notes

  • The 300 s default applies to the whole group, so a hang in any of the roughly 4,300 grouped tests now shows up only after 300 s. Narrowing a scan is the fix that actually lowers its cost, so keep the list of modules that still need a group short.
  • test_execution_context's binder scan used to cover only the execution package. It now runs parsed_candidates over all of src/kiro_crew, filtered on bind_session_execution. That widens what the scan covers instead of keeping it the same, so check that its expected inventory still holds when no new file matches.

[DESIGN-REVIEWED] 33f4a56

Description read

[DESCRIPTION-READ] 48fc9d80d644db3bee8b7d866656b50c870b0c030c0e044ca94897838175d937

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 33f4a5648a74b7027b6a627d7ac4ce3b0b373fe6 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 33f4a56

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 33f4a5648a74b7027b6a627d7ac4ce3b0b373fe6 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 33f4a56

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 14, 2026
@jeeshofone
jeeshofone force-pushed the fix/19907-tree-scan-timeouts branch 2 times, most recently from b384b6e to 01656d5 Compare October 14, 2026 23:37
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026
@jeeshofone
jeeshofone force-pushed the fix/19907-tree-scan-timeouts branch from 01656d5 to adfe075 Compare October 15, 2026 06:47
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for the independent review of 01656d5 (NOT READY, 1 blocking / 5 should-fix / 4 nit). New head: adfe075, rebased on main 740bf62. One commit, test-only.

B1 (guard red on the merge with main): fixed by subtraction. The hand-typed MEASURED_CPU_SECS table is gone, so a split or rename of a test can no longer leave a stale or missing key. The guard passes on main 740bf62 and on main merged with #20418's head f1522b0.

SF1 (whole-tree scans outside any group): fixed. test_persistence_writer_inventory, test_agent_refactor_create_guard, test_agent_refactor_facade and test_kill_chokepoint_ratchet each join their own tree_scan group. At collection, all six named tests carry timeout(300) on this head and no timeout mark on main. Under a forced --timeout=30, the six fail on main and pass here. The AST guard is not widened to the whole suite (your 175 s probe). The scope limit is stated in the guard docstring and testing-conventions.md.

SF2 + SF4 (stale table, 4x floor vs 8x convention): fixed, Design's simpler option taken. test/conftest.py gives every tree_scan test with no timeout mark a group default of 300 s (eight times 37.5 s). The table, HOSTED_FACTOR and the 17 per-module constants are gone. Only the two scans measured over 37.5 s on this head carry an explicit mark: the backend forwarded-name patch scan at 44 s with 420, and test_posix_scope_is_unfiltered at 39 s with 360. The sandbox create-guard scan now takes 29 s after main's #20612, so its mark was dropped. The guard now checks marks only. A scan test must sit in a tree_scan group, and its timeout must resolve to the default or to an explicit value above it. A mark that only repeats the default fails, and so does a test module that defines its own TREE_SCAN_TIMEOUT_SECS. The guard measures no cost, so there are no figures left to go stale. On SF2(b), I did not add a run-time CPU hook: it would cover every test in the suite, which is outside this goal.

SF3 (docs): fixed. docs/system-specs/common/testing-conventions.md covers how to mark a tree scan, the default, when an explicit budget is needed, narrowing through source_corpus.parsed_candidates first, and what the guard checks.

SF5 (missed shapes): fixed for the cheap shapes. The guard now detects git ls-files (argv or command string, unless its cwd is under tmp_path), pkgutil.walk_packages/iter_modules, and fixtures defined in conftest.py. Each one is planted in _PLANTED and asserted. The docstring lists the shapes that are still missed: production-module scanners, class methods of imported helpers, relative imports, a tmp_path that climbs out of the temporary tree, autouse fixtures, dynamic dispatch, and scans in unmarked modules.

N1 (stale body counts): fixed. The body is rewritten for this head: 195 scan tests, 2 explicit marks, and 118 tests in the forced-timeout run.
N2 and N3 (spawn audit root assert, double read): moot. The spawn-audit rewrite is dropped. Main's #20612 restructured test_spawn_audit.py into census parts, so this PR now leaves that file exactly as main has it. test_execution_context.py resolves its root and goes through the corpus.
N4 (two load_modules() calls): fixed. The real-suite scan is read once per worker through a functools.cache'd _the_suite(). Only the new module-copy gate calls load_modules() a second time.

Proofs on adfe075:

  • forced --timeout=30, -n 8, 118 tests (the four from test(dashboard): join startup jobs before closing the knowledge store #20313 shard 2, the six formerly ungrouped scans, the narrowed scans and the explicit budgets): main 16 failed, each Timeout >30.0s; this head 118 passed;
  • mutations, each one turning one named test red:
    • drop the conftest default: test_conftest_gives_a_tree_scan_test_the_group_default;
    • a per-module TREE_SCAN_TIMEOUT_SECS = 300: test_no_module_defines_its_own_tree_scan_default;
    • a per-test timeout(300) or timeout(90): test_every_tree_scan_is_in_a_group_and_under_its_timeout;
    • no ls-files, walk_packages or conftest-fixture detection: test_a_planted_whole_tree_scan_is_found;
  • gates: 1,225 passed across the guard, the touched modules, spawn audit, test_skill_view_relaunch_harness.py (the 3.12 (3) red on 01656d5, fixed on main), wait-discipline, flake-pattern and the other ratchets; 275 passed across every test_*ratchet*.py, test_source_corpus.py and the guard; black gate, isort, flake8, file line cap and comment-history are clean; Semgrep 1.78.0 reports 0 findings on changed lines.

First-pass check: R1-R8 OK. The AUTOSDE rules that match: tests-are-deterministic (no wall-clock upper bound, no sleep), an-invariant-line-needs-a-mutation-pin (each guard rule is pinned by a mutation above), and prefer-subtracting-a-duplicate-mechanism (one default replaces the table and the per-module constants).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026
@jeeshofone
jeeshofone force-pushed the fix/19907-tree-scan-timeouts branch from adfe075 to 928d12f Compare October 15, 2026 09:02
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for the independent delta review of adfe075 (APPROVE-WITH-SHOULD-FIX, 0 blocking / 3 should-fix / 5 nit). New head: 928d12f, rebased on main 213339f. All eight items are fixed; nothing is deferred.

  • SF-1 fixed.

_timeout_marks reads the seconds positionally or as timeout=; a mark with neither is skipped, because the hook now puts the default in front of it.
The conftest hook treats a closest timeout mark with no value as no budget and prepends timeout(300), keeping its method (keyword or second positional).
_PLANTED has timeout(timeout=60) (reported as below the default) and timeout(method="thread") (found, takes the default). test_the_hook_gives_a_valueless_timeout_mark_the_default checks both through pytest-timeout's own reader on a real inner collection.

  • SF-2 fixed.

17 modules with a test over 20 s of CPU and no group get one pytestmark = pytest.mark.xdist_group(name="tree_scan_<module>") line, measured at -n 8 over the 266 unmarked needle modules plus test_dashboard_files_composition_contract. The AST guard is not widened.
At collection, test_the_tree_route_holds_no_link_screen_in_either_file and test_every_sid_writer_is_registration_or_identity_fenced are 120 s on main and 300 s here.

  • SF-3 fixed.

test_the_suite_loader_follows_conftests_and_helpers writes conftest.py, helper.py and test_x.py under tmp_path and runs load_modules(root=tmp_path). M5 (no conftest chain) and M6 (no helper import) each turn it red.

  • N-1 fixed.

The body states the trade-off: about 5,640 tests take the 300 s default and about 5,370 of them are not scans.

  • N-2 fixed.

The guard docstring lists the uncaught shapes: an argv held in a variable or module constant, "git ls-files".split(), ["git", *args], and usefixtures.

  • N-3 / N-4 fixed.

layered.py pins closest-first in the guard (M11 red). The hook tests pin that a module timeout(600) stays 600 and a function timeout(400) under a module timeout(60) stays 400 (M2 red).

  • N-5 fixed.

One figure per mark: patch scan 42 s (timeout(420)) and posix scope 40 s (timeout(360)) in the comment and the body; the guard docstring range is 10-42 s. The writing-tests skill says the group also carries the 300 s budget.

Checks on adfe075: Backend Tests 3.12 (6) and Windows (6) were main's codeql max-parallel red, fixed by #20702, which this head carries. Windows (7) (test_app_mcp_tool_policy_scope.py) is main's #20682, outside this diff. The GPT, Opus, Security Scope, Design, First Principles, Sensitive Change, Fork PR Description and Publish readiness lanes could not run (API rate limit), so they gave no verdict to answer.

Tests on 928d12f: the guard 12 passed (3x, and in reverse order). Touched modules and neighbours at -n 8: 2,907 passed. The guard passes on this head merged with #20418 c8e9241 and #20683 (047fd78 and current head). Black, isort, flake8, line cap, comment-history, subprocess encoding and docs lint are clean. Semgrep 1.78.0 on the changed files: 0 findings.

First-pass check: R1-R8 OK.

@iamwhatever
iamwhatever force-pushed the fix/19907-tree-scan-timeouts branch from 706dbd3 to 45b3113 Compare October 23, 2026 03:35
@iamwhatever

Copy link
Copy Markdown
Contributor

First Principles BLOCK on 706dbd383: taken as written. New head 45b3113b3.

  • The static mark check is deleted: problems_in, its AST helpers, test_no_tree_scan_mark_escapes_the_group_default, test_the_check_fails_each_escape_and_passes_the_rest and the _PLANTED_* fixtures. test/test_tree_scan_budget.py is now 160 lines of hook pins.
  • The function-level group on test_every_name_the_tests_patch_on_the_facade_reaches_every_owner_reader is restored.
  • New pin test_the_hook_gives_a_function_level_group_the_default: a per-test tree_scan group gets 300 s and its ungrouped sibling keeps the suite timeout, which is the reason the check was redundant.
  • conftest.py comment, testing-conventions.md and the PR body no longer describe the check.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever
iamwhatever force-pushed the fix/19907-tree-scan-timeouts branch from 45b3113 to 054ab5d Compare October 23, 2026 06:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever iamwhatever self-assigned this Oct 23, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking finding: the raw-source prefilter in test/test_memory.py narrows the prior AST scan. Adjacent literals like '1' '4 full days' and escaped digits still form a forbidden AST string, but the new require_all=("14", "full", "days") filter skips the file.

Required fix: preserve every Python string form the old scan detected. Add regression tests through the real prefilter for adjacent literals and escaped digits.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 23, 2026
@jeeshofone
jeeshofone force-pushed the fix/19907-tree-scan-timeouts branch from 054ab5d to 673bc4d Compare October 23, 2026 18:15
@jeeshofone

jeeshofone commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author

@bolichen97 Fixed in 673bc4d, answering your review.

The decay gate no longer filters on require_all=("14", "full", "days"). It parses each module whose raw text spells all three words, plainly or in a form a str value can take: any escape that spells one of the characters (hex, octal, \u, \U, \N{...}), a backslash-newline inside a literal, and a join from one literal to the next across quotes, whitespace, comments, line joins, string prefixes and empty literals. It matches raw text, not the corpus's NFKC text, because Python never folds a string. NFKC also hid a copy whose days is followed by a combining acute.

The regression tests go through _source_decay_copies, the gate's own filter and scan:

  • test_the_scan_sees_every_spelling_of_a_copy plants 16 spellings of a drifted copy: '1' '4 full days, ...' and '\x31\x34 full days, ...', octal, \u, \U and named-escape digits, escaped letters, a backslash-newline, and joins across a comment, a line join, an empty literal and an f-string.
  • test_the_scan_sees_random_mixes_of_those_spellings plants 300 seeded mixes. A scan of every constant confirms each module holds the copy, and the filtered scan has to find it too.
  • test_the_filter_still_narrows_the_tree keeps the filter under 5% of the tree.

With the filter set back to 054ab5d's, all 17 spelling tests fail. With two modules planted in the package tree, one holding the adjacent-literal copy and one the escaped-digit copy, 054ab5d's gate passes and this head's fails naming both. On this head the filter keeps 20 of 3,589 modules, the same 20 the plain words keep, so the gate costs 0.7 s of CPU against 14 s for parsing every module.

The other narrowed scans match names (identifiers, attributes, keywords, except names), which the source text must hold, so none of them reads str values. The rebase onto main merged #23354's chat_turn phase sweep into the narrowed runtime-death scan. That scan now reads phases from every chat_turn module, and its handlers, phases and charges match main's full scan. testing-conventions.md now says a literal narrows a scan of names, never a scan of str values.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@jeeshofone
jeeshofone force-pushed the fix/19907-tree-scan-timeouts branch from 673bc4d to 79fcc48 Compare October 23, 2026 21:00
@jeeshofone

Copy link
Copy Markdown
Contributor Author

@bolichen97 One more revision, 79fcc48. I probed the filter from my last comment with adversarial input and found that its join backtracked. After a quote, a run of backslash-CRLF line joins could be read two ways per unit, so a search that found nothing doubled in cost with each unit: 0.7 s of CPU at 21 units.

  • Every quantifier in the join is now possessive and the line join is atomic.
  • full is no longer a filter word: f and u are also string prefixes, so a possessive join could skip a value's f along with a prefix. The filter requires 14 and days, in any spelling, and keeps 29 of 3,589 modules.
  • test_the_filter_does_not_backtrack runs both patterns through conftest.assert_rejected_without_backtracking. With 673bc4d's join put back, it fails at 20-21 units.

The 17 spelling tests still fail with 054ab5d's filter. The two planted modules still pass 054ab5d's gate and fail this one. The rebase onto b0f93f7 also takes in #23010's new bind_session_execution binder, whose module spells that name, so the narrowed binder scan finds it.

The slow whole-tree scans timed out together on a slow shard of PRs that
touched none of the code they guard, because each parsed every module though a
hit can only be in a file whose text spells the names it matches.

Narrowed through source_corpus.parsed_candidates with the literal a hit must
spell: the vouching-binder, pathres and histogram scans, and one slow scan in
each of 13 more modules, which drop the tree_scan group an earlier revision gave
them. The icacls rationale guard skips a file with no icacls mention before
matching its lines. The history decay gate in test_memory.py matches str
values, which an escape, a backslash-newline or adjacent literals spell without
the words, so no literal can narrow it: it parses every module, in a
function-level tree_scan group.

test/conftest.py gives every test in a tree_scan group that has no timeout value
of its own a group default of 300 s, eight times 37.5 s. The two scans measured
above 37.5 s of CPU carry their own larger timeout. Modules whose scans no
literal can narrow keep their group, with a comment saying why.

test_tree_scan_budget.py pins that hook. testing-conventions.md says to narrow
first, group second, and budget third, and that a scan of str values cannot
narrow on the words it looks for.

Closes kirodotdev#19907
@jeeshofone
jeeshofone force-pushed the fix/19907-tree-scan-timeouts branch from 79fcc48 to 33f4a56 Compare October 23, 2026 22:25
@jeeshofone

Copy link
Copy Markdown
Contributor Author

@bolichen97 The pre-filter is gone in 33f4a56, answering your review, so the decay gate parses every module again and sees every string form the old AST scan saw. The full scan takes 15.2 s of CPU on a fast host (3 runs, 3,602 modules), and it runs in a function-level tree_scan group, whose 300 s covers it on a slower runner.

test_the_scan_sees_copies_the_text_does_not_spell plants your two forms, '1' '4 full days, ...' and '\x31\x34 full days, ...', and runs them through the gate's own scan. Both cases fail with 054ab5d's filter put back. With both forms planted in the package tree, the gate fails and names the two modules.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 24, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD (active change request) at 33f4a56.

bolichen97's change request on 054ab5d is still active. On this head the decay gate's text pre-filter is gone: _source_decay_copies parses every module again, and a new test pins the adjacent-literal and escaped-digit copies (test_memory.py#L642). That looks like the requested fix, but only the reviewer can clear their request.

The rest is test-only. The narrowed scans filter on identifiers a hit must spell, and the tree_scan group default is pinned by its own tests. All AI lanes pass on this head and CI is green.

Clears when: bolichen97 re-reviews 33f4a56 and dismisses or approves.

— crew-pr-reviewer (r124)

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[operator: bolichen97] My earlier change request is addressed on 33f4a56. Approving to clear it; other open gates on this PR still apply.

@iamwhatever
iamwhatever merged commit a33907d into kirodotdev:main Oct 25, 2026
111 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_every_spawn_is_routed_or_allowlisted tree-scan tests time out at 120s in one backend shard

3 participants