Repository navigation
test: untrace corpus scans and stub readiness backoff sleep - #39202
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Only test files change, and they reuse existing seams ( Notes
[DESIGN-REVIEWED] 550b2fc Description read[DESCRIPTION-READ] 36eb14490986f482d380b0366ef59ac69607938541451228f513d143cafaa64d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
78ae56d to
622311a
Compare
622311a to
25c746f
Compare
|
Backend Tests (3.12, 8) red is main's, not this PR's: test/test_bare_session_handle_guard.py::test_no_test_builds_a_bare_session_handle_directly also fails on origin/main 640338a (flags test_kas_token_refresh_recovery.py:245 from #38978). Tracked in #39261; fixes open in #39262 and #39274. This PR rebases once one lands. |
The facade, census, GNU-form, create-guard, slot-reader, bounded-read and spawn-host-pin gates walk and parse the whole test or source tree under coverage's tracer on the Linux shards, though none of that scan is product code. Wrap only the scan functions in source_corpus.untraced(); every src call stays traced, and covered lines and branches are unchanged. gnu_only's module walk and the spawn-host-pin scan ran once per worker; an xdist group makes each run once. pr_readiness_publish's seven read-failure cases waited out the real 2+4 s retry backoff. A recording sleep stub on the harness PATH skips the wait and the cases now assert the [2, 4] schedule.
25c746f to
550b2fc
Compare
Code change approver recommendation: A person should decideWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier 2: reviewer-finding: Design Review. Test-only change. It applies the existing Tier: 2, held for a person: reviewer-finding: Design Review. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
I reviewed head 550b2fc. Every untraced() sits on a test-side reader that does text, regex, AST or git ls-files work. Each one sits below its lru_cache, so the tree_scan memo release still clears those caches. The only sleep in pr-readiness.yml is the one in gh_retry, so the new PATH stub cannot hide any other wait. Every changed file is under test/, nothing is loosened or removed, and PR Readiness passes on this head.
Well done: the read-failure tests no longer sit through a real 6 s backoff. They now assert the retry schedule [2, 4], which no test checked before.
Optional:
- test/test_pr_readiness_publish.py:961: add "sleeps" to
_hold's stale-file list, so both harness entry points reset the same files. - test/test_exfil_facade.py:216:
_family_defined_names()calls_facade_mirror._defined_names, so this scan is not purely test-side. Other tests still cover that function. A one-line note on the decorated helpers would keep them pure readers. - The body says module teardown clears the
repo_filescache. No hook clears it, so the saving is about one call per worker, not one per module.
Problem / Motivation
Goal: Cut test-only time from the slowest Linux shards by keeping coverage's tracer off whole-tree test scans and skipping a real retry backoff wait, with no change to what any test checks or covers.
On the instrumented Linux shards (5, 6 and 2 are the slowest after #39025), much of the time in the facade, census, GNU-form and ratchet gates goes to coverage's tracer. The tracer follows every Python call these gates make while they walk and parse the test or source tree. That scan is test-side code that coverage does not report. Three more costs repeat:
test_gnu_only_shell_usage's module-scopedall_hitswalk ran once on each worker its two tests landed on. In CI that was 29.2 s on gw4 and 28.6 s on gw2.test_subagent_spawn_host_pin's cachedtest/scan ran once on each of three workers, about 12 s each.test_pr_readiness_publish, seven read-failure cases waited out the realgh_retrybackoff (sleep 2+sleep 4). CI shows each at about 6.0 to 6.5 s.Why it matters
These 17 files cost about 800 CPU-seconds of setup and call time on Linux shards 5, 6 and 2. That is the median of three main runs after #39025: 620 s for the first 13 files and 176 s for the four ratchets. Local runs under
--covwith the change take about 119 s less across the 17 files. In CI the tracer costs more, and the two xdist groups also remove repeated tree walks. Shorter shards return CI results sooner on every PR.Not a goal
src/and no change totest/conftest.py.importlib.import_module, and executed patch checks stay traced.What changed (motivation → approach → change)
Symptom: tracer CPU spent on test-side tree scans, repeated per-worker walks, and a real backoff sleep. Approach: reuse the repo's existing
source_corpus.untraced()(already used by 32 gates) as a decorator, only on functions that read text or ASTs. Changes:facade_mirror_scan.patch_eventsandtext_patched_namesare wrapped. They are pure AST and regex readers.owner_import_scan.owner_importersis wrapped. It parses sources and walks the tree.source_corpus.repo_filesis wrapped. It runsgit ls-filesand filters paths. Its cache is cleared at every module's teardown, so each corpus module paid for it again under the tracer._corpus_patched_namesin the exfil, security_split_patch_reach, sessions_parts, members_handler, _shared_handlers, memory_handlers, events and runner facade tests._corpusintest_cron_service_bindings._owner_importersintest_runner_facade.check_shared_bindingscensus.census_partsandcensus_partintest_sandbox_refactor_facade.pyare wrapped.test_platform_compat_refactor_facadeshares them.test_gnu_only_shell_usage. The module getsxdist_group(name="tree_scan_gnu_only_shell_usage"), soall_hitsis built once. The walk inside the fixture runs underuntraced()._create_patchesintest_sandbox_refactor_create_guard.pyis wrapped.test_slot_running_readers.pyare wrapped. Their bodies parsesrc/kiro_crew/dashboardand call no product code._tree_sitesintest_bounded_read_ratchet.pyis wrapped._spawning_modulesand_pinned_modules_that_undointest_subagent_spawn_host_pin.pyare wrapped, and the module getsxdist_group(name="tree_scan_subagent_spawn_host_pin").test_pr_readiness_publish. It now has asleepstub that records the requested seconds. The stub sits on the harness PATH beside theghstub, the same patterntest_pr_readiness_evaluate.pyalready uses. A newResult.backoffproperty reads the record back. Each of the seven read-failure tests now assertsresult.backoff == [2, 4], which is one exhausted read with three attempts. Before this change, nothing checked the backoff schedule.Backwards compatibility
Compatible: only test files change. Every test keeps its assertions, and the covered src lines and branches are identical.
Removes nothing: no assertion, test, fixture or helper is removed. The
sleepstub replaces a wall-clock wait that no assertion read, and the schedule is now asserted.Tests
Proof: not provable -- test-only change
Base is
3ccb7aaf09. The change was built in two steps, and each step was measured against the one before it.--cov=kiro_crew --cov=sage_lib -n 4: 146,984 executed src lines and 2,666 executed branches across 4,724 files, both before and after. No file differs.-n0 --cov(s):test_only_the_family_imports_an_owner(sharedowner_importers).kiro_crew.security._CARVEOUT_VERBSfailstest_every_package_routed_name_lands_on_this_facade(sharedpatch_events+ corpus).sandbox.osfailstest_a_patched_name_bound_by_the_facade_and_an_owner_is_a_listed_one(census).scripts/*.shfile withgrep -Pfailstest_no_gnu_only_form_reaches_a_macos_host_unprobed.mock.patch.object(sandbox, "CANONICAL_TEMP_KEYS", (), create=True)failstest_no_test_patches_a_forwarded_name_that_it_may_create.slot.taskunderif slot.running:failstest_running_guarded_task_loads_null_check_task.fh.read(cap + 1)failstest_every_capped_read_of_a_file_goes_through_the_helper.SubagentManager.spawnfailstest_every_spawning_module_is_pinned_or_excluded.attempt * 3fails all seven read-failure tests ([3, 6] == [2, 4]). Before this change the same plant passes 59/59, so the schedule check is new.-n 4 --dist loadgroup(1,151 passed) and at-n0in reverse collection order (1,151 passed).Manual verification
N/A — unit coverage sufficient: the change is test-only, and the before/after runs above are the verification.
Real-machine verification
N/A -- test-only change. The measurements above ran on a Linux x86_64 dev host (Python from the repo venv), with base at
3ccb7aaf09and head on this branch.Related Issues
Follows the per-file cost scan of Linux shards 5/6/2 after #39025.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.