Skip to content

test: untrace corpus scans and stub readiness backoff sleep - #39202

Merged
iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/linux-next-scans
Nov 18, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/linux-next-scans

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Cut test-only time from the slowest Linux shards by keeping coverage's tracer off whole-tree test scans and skipping a real retry backoff wait, with no change to what any test checks or covers.

On the instrumented Linux shards (5, 6 and 2 are the slowest after #39025), much of the time in the facade, census, GNU-form and ratchet gates goes to coverage's tracer. The tracer follows every Python call these gates make while they walk and parse the test or source tree. That scan is test-side code that coverage does not report. Three more costs repeat:

  • test_gnu_only_shell_usage's module-scoped all_hits walk ran once on each worker its two tests landed on. In CI that was 29.2 s on gw4 and 28.6 s on gw2.
  • test_subagent_spawn_host_pin's cached test/ scan ran once on each of three workers, about 12 s each.
  • In test_pr_readiness_publish, seven read-failure cases waited out the real gh_retry backoff (sleep 2 + sleep 4). CI shows each at about 6.0 to 6.5 s.

Why it matters

These 17 files cost about 800 CPU-seconds of setup and call time on Linux shards 5, 6 and 2. That is the median of three main runs after #39025: 620 s for the first 13 files and 176 s for the four ratchets. Local runs under --cov with the change take about 119 s less across the 17 files. In CI the tracer costs more, and the two xdist groups also remove repeated tree walks. Shorter shards return CI results sooner on every PR.

Not a goal

  • No change under src/ and no change to test/conftest.py.
  • No assertion is removed or loosened, and no test is added or deleted.
  • No tracer pause around any call that runs product code. Imports, importlib.import_module, and executed patch checks stay traced.
  • No slow file whose cost is in product code is changed here (for example chat_runner_redact, local_gate, knowledge_dedup).
  • The Windows-only ~2 s loopback dials in the instances tests are not changed here.

What changed (motivation → approach → change)

Symptom: tracer CPU spent on test-side tree scans, repeated per-worker walks, and a real backoff sleep. Approach: reuse the repo's existing source_corpus.untraced() (already used by 32 gates) as a decorator, only on functions that read text or ASTs. Changes:

  • Shared scan helpers.
    • facade_mirror_scan.patch_events and text_patched_names are wrapped. They are pure AST and regex readers.
    • owner_import_scan.owner_importers is wrapped. It parses sources and walks the tree.
    • source_corpus.repo_files is wrapped. It runs git ls-files and filters paths. Its cache is cleared at every module's teardown, so each corpus module paid for it again under the tracer.
  • Per-file corpus scans, wrapped.
    • _corpus_patched_names in the exfil, security_split_patch_reach, sessions_parts, members_handler, _shared_handlers, memory_handlers, events and runner facade tests.
    • _corpus in test_cron_service_bindings.
    • _owner_importers in test_runner_facade.
  • check_shared_bindings census. census_parts and census_part in test_sandbox_refactor_facade.py are wrapped. test_platform_compat_refactor_facade shares them.
  • test_gnu_only_shell_usage. The module gets xdist_group(name="tree_scan_gnu_only_shell_usage"), so all_hits is built once. The walk inside the fixture runs under untraced().
  • Four more ratchets.
    • _create_patches in test_sandbox_refactor_create_guard.py is wrapped.
    • The two tree-walking tests in test_slot_running_readers.py are wrapped. Their bodies parse src/kiro_crew/dashboard and call no product code.
    • _tree_sites in test_bounded_read_ratchet.py is wrapped.
    • _spawning_modules and _pinned_modules_that_undo in test_subagent_spawn_host_pin.py are wrapped, and the module gets xdist_group(name="tree_scan_subagent_spawn_host_pin").
  • test_pr_readiness_publish. It now has a sleep stub that records the requested seconds. The stub sits on the harness PATH beside the gh stub, the same pattern test_pr_readiness_evaluate.py already uses. A new Result.backoff property reads the record back. Each of the seven read-failure tests now asserts result.backoff == [2, 4], which is one exhausted read with three attempts. Before this change, nothing checked the backoff schedule.

Backwards compatibility

Compatible: only test files change. Every test keeps its assertions, and the covered src lines and branches are identical.

Removes nothing: no assertion, test, fixture or helper is removed. The sleep stub replaces a wall-clock wait that no assertion read, and the schedule is now asserted.

Tests

Proof: not provable -- test-only change

Base is 3ccb7aaf09. The change was built in two steps, and each step was measured against the one before it.

  • Shared helpers, facades, census, gnu_only and pr_readiness, over the 113 files that use the touched helpers:
    • Test count: 5,780 before and after. Per-file counts are identical for the 13 heavy files.
    • Coverage under --cov=kiro_crew --cov=sage_lib -n 4: 146,984 executed src lines and 2,666 executed branches across 4,724 files, both before and after. No file differs.
  • The four ratchets, over the 6 files that hold or import them:
    • Test count: 1,372 before and after.
    • Coverage: 96,778 executed src lines and 1,368 executed branches, both before and after. No file differs.
  • Time, each file alone, -n0 --cov (s):
File Before After
test_exfil_facade 29.9 18.6
test_security_split_patch_reach 25.6 19.3
test_sessions_parts_facade 22.2 16.7
test_runner_facade 20.4 17.6
test_members_handler_facade 17.6 14.7
test__shared_handlers_facade 18.2 14.3
test_memory_handlers_facade 20.7 16.8
test_events_facade 26.8 25.0
test_cron_service_bindings 20.9 16.9
test_sandbox_refactor_facade 29.7 25.4
test_platform_compat_refactor_facade 29.2 24.7
test_gnu_only_shell_usage 17.8 14.3
test_pr_readiness_publish 77.3 35.4
test_sandbox_refactor_create_guard 26.4 19.2
test_slot_running_readers 17.7 11.0
test_bounded_read_ratchet 23.3 17.4
test_subagent_spawn_host_pin 8.8 6.7
Total 432.3 313.7
  • Planted errors are still caught. Each plant was removed after its run.
    • A src module importing a sessions_parts owner fails test_only_the_family_imports_an_owner (shared owner_importers).
    • A test patching kiro_crew.security._CARVEOUT_VERBS fails test_every_package_routed_name_lands_on_this_facade (shared patch_events + corpus).
    • A test patching sandbox.os fails test_a_patched_name_bound_by_the_facade_and_an_owner_is_a_listed_one (census).
    • A scripts/*.sh file with grep -P fails test_no_gnu_only_form_reaches_a_macos_host_unprobed.
    • A test with mock.patch.object(sandbox, "CANONICAL_TEMP_KEYS", (), create=True) fails test_no_test_patches_a_forwarded_name_that_it_may_create.
    • A dashboard module reading slot.task under if slot.running: fails test_running_guarded_task_loads_null_check_task.
    • A src fh.read(cap + 1) fails test_every_capped_read_of_a_file_goes_through_the_helper.
    • An unpinned test module driving SubagentManager.spawn fails test_every_spawning_module_is_pinned_or_excluded.
    • Changing the workflow backoff to attempt * 3 fails all seven read-failure tests ([3, 6] == [2, 4]). Before this change the same plant passes 59/59, so the schedule check is new.
  • Order: all 16 changed test files pass at -n 4 --dist loadgroup (1,151 passed) and at -n0 in reverse collection order (1,151 passed).
  • Gates: black, isort and flake8 on the changed files; check_comment_history; check_brand_name; check_file_line_cap; test_flake_pattern_ratchet (75 passed).

Manual verification

N/A — unit coverage sufficient: the change is test-only, and the before/after runs above are the verification.

Real-machine verification

N/A -- test-only change. The measurements above ran on a Linux x86_64 dev host (Python from the repo venv), with base at 3ccb7aaf09 and head on this branch.

Related Issues

Follows the per-file cost scan of Linux shards 5/6/2 after #39025.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 18, 2026 19:11
@iamwhatever
iamwhatever enabled auto-merge (squash) November 18, 2026 19:11
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 550b2fc6c44d835c6428e13ce35d9e6d8d19debf ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 550b2fc6c44d835c6428e13ce35d9e6d8d19debf touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 550b2fc6c44d835c6428e13ce35d9e6d8d19debf — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Only test files change, and they reuse existing seams (source_corpus.untraced(), the PATH stub pattern, xdist groups). The coverage parity they report has no design-level gap.

Notes

  • untraced() is now applied to the shared helpers repo_files, owner_importers and patch_events. If one of them ever starts importing or running product code, its coverage will disappear without any warning. A short docstring on each helper saying "pure text/AST reader" would warn whoever edits it next.

[DESIGN-REVIEWED] 550b2fc

Description read

[DESCRIPTION-READ] 36eb14490986f482d380b0366ef59ac69607938541451228f513d143cafaa64d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 550b2fc6c44d835c6428e13ce35d9e6d8d19debf — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 550b2fc

Verdict parsed from the review's SHA-scoped output markers for commit 550b2fc6c44d835c6428e13ce35d9e6d8d19debf.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 550b2fc6c44d835c6428e13ce35d9e6d8d19debf false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 550b2fc6c44d835c6428e13ce35d9e6d8d19debf and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 550b2fc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 550b2fc6c44d835c6428e13ce35d9e6d8d19debf false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the crew/testoptimizer/linux-next-scans branch from 78ae56d to 622311a Compare November 18, 2026 20:08
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_sandbox_refactor_facade.py since this head went green (3ccb7aa -> 5762722); CI re-running. Related tests (sandbox_refactor_facade, platform_compat_refactor_facade, sandbox_refactor_create_guard): 1361 passed locally; flake ratchet green.

@iamwhatever
iamwhatever force-pushed the crew/testoptimizer/linux-next-scans branch from 622311a to 25c746f Compare November 18, 2026 21:06
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_sandbox_refactor_facade.py, test/test_slot_running_readers.py since this head went green (5762722 -> 640338a); CI re-running. Related tests (sandbox_refactor_facade, platform_compat_refactor_facade, slot_running_readers): 1300 passed locally; flake ratchet green.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 8) red is main's, not this PR's: test/test_bare_session_handle_guard.py::test_no_test_builds_a_bare_session_handle_directly also fails on origin/main 640338a (flags test_kas_token_refresh_recovery.py:245 from #38978). Tracked in #39261; fixes open in #39262 and #39274. This PR rebases once one lands.

@iamwhatever
iamwhatever disabled auto-merge November 18, 2026 22:03
The facade, census, GNU-form, create-guard, slot-reader, bounded-read
and spawn-host-pin gates walk and parse the whole test or source tree
under coverage's tracer on the Linux shards, though none of that scan
is product code. Wrap only the scan functions in
source_corpus.untraced(); every src call stays traced, and covered lines
and branches are unchanged. gnu_only's module walk and the spawn-host-pin
scan ran once per worker; an xdist group makes each run once.

pr_readiness_publish's seven read-failure cases waited out the real
2+4 s retry backoff. A recording sleep stub on the harness PATH skips
the wait and the cases now assert the [2, 4] schedule.
@iamwhatever
iamwhatever force-pushed the crew/testoptimizer/linux-next-scans branch from 25c746f to 550b2fc Compare November 18, 2026 22:04
@iamwhatever
iamwhatever enabled auto-merge (squash) November 18, 2026 22:04
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 18, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier 2: reviewer-finding: Design Review.

Test-only change. It applies the existing source_corpus.untraced() context manager as a decorator (valid: it is built with contextlib.contextmanager, so it doubles as a decorator) to test-side scan helpers that only read files and ASTs. It adds per-file tree_scan_* xdist groups, which is the documented pattern in testing-conventions.md, and stubs sleep in the pr-readiness harness, with a new assertion that pins the [2, 4] backoff from gh_retry's only sleep. lru_cache stacking keeps working with conftest's memo release (wrapper type and module are preserved). No wrapped function runs product code, coverage only measures src packages, and CI including Coverage Gate is green. No blocking issues found.

Tier: 2, held for a person: reviewer-finding: Design Review.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 550b2fc6c4 · 74 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head 550b2fc. Every untraced() sits on a test-side reader that does text, regex, AST or git ls-files work. Each one sits below its lru_cache, so the tree_scan memo release still clears those caches. The only sleep in pr-readiness.yml is the one in gh_retry, so the new PATH stub cannot hide any other wait. Every changed file is under test/, nothing is loosened or removed, and PR Readiness passes on this head.
Well done: the read-failure tests no longer sit through a real 6 s backoff. They now assert the retry schedule [2, 4], which no test checked before.
Optional:

  • test/test_pr_readiness_publish.py:961: add "sleeps" to _hold's stale-file list, so both harness entry points reset the same files.
  • test/test_exfil_facade.py:216: _family_defined_names() calls _facade_mirror._defined_names, so this scan is not purely test-side. Other tests still cover that function. A one-line note on the decorated helpers would keep them pure readers.
  • The body says module teardown clears the repo_files cache. No hook clears it, so the saving is about one call per worker, not one per module.

@iamwhatever
iamwhatever merged commit 0e67154 into main Nov 18, 2026
101 of 104 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/linux-next-scans branch November 18, 2026 23:59
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants