Repository navigation
test: run the shared facade checks' tree scans with the tracer paused - #38674
Conversation
The shared facade checks call each module's _corpus_patched_names(), which reads the whole test tree. Under CI's --cov the tracer ran on every call of that test-side read. The four call sites now go through one helper that pauses the tracer, except for test_projection_facade, whose scan imports and calls production code to resolve a receiver and so stays traced.
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a test-only change, reversible, and it reuses the Notes
[DESIGN-REVIEWED] e864eca Description read[DESCRIPTION-READ] e564d135f109d81da10175a3f4a1c881f68e9c92bebf04abf35a34ad18e6a098 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier pending: being reviewed. Test-only change that routes four shared facade checks' whole-tree patch scans through source_corpus.untraced(), the existing per-thread settrace pause used by ~20 other gates, including the sibling test_bindings_shared_checks. What the checks assert is unchanged, since pausing the tracer only affects coverage. The one scan that imports production modules and calls their helpers (test_projection_facade, via _returns_the_facade) is opted out and stays traced. The other opted-in scans read and parse text through facade_mirror_scan; the import_module calls in those files are in test bodies, not in the scan. The reported 43-line drop in _facade_mirror is said to be covered by test_facade_mirror.py, and the Coverage Gate passed. Because the one traced module is excluded, the order in which the cached scan result is first filled doesn't change what coverage reports. CI is green on Linux and Windows. Tier: pending: being reviewed. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved. On this head the review checked that the four module._corpus_patched_names() call sites now go through one helper that runs the scan inside the existing source_corpus.untraced(). That context manager puts the tracer back in a finally. The assertions sit outside it, and no assertion, count or allowlist changes. We checked the 64 opted-in scans. Only test_projection_facade imports and calls production modules, so it is the one scan that must stay traced. _load imports by stem, so the module.__name__ match holds. The PR touches no src/, security, sandbox or auth path.
Done well: you measured the coverage cost before you changed anything. You found the 7,300-line loss in the projection scan, kept that scan traced, and traced the remaining 43 lines to _facade_mirror.py and the test that already covers them.
Optional:
- The comment at
test/test_facade_shared_checks.py:166and the docstring at :173 say scans that run production code stay traced. But the argv_floor, denied_rules, exfil, paths, redaction and session_ledger scans call_facade_mirrorhelpers with the tracer off. A more exact rule: a scan stays traced when it runs production code that no other test covers. - A one-line pin such as
assert _SCANS_THAT_RUN_PRODUCT_CODE <= set(_OPTED_IN)would catch a renamed or mistyped stem. Today that mistake would quietly run the projection scan untraced. - A short disposition reply to the Design lane note about the hand-kept list would close it. The Known gap section in the body already covers it.
Problem / Motivation
Goal: The shared facade checks run each module's whole-test-tree patch scan with the coverage tracer paused, so the scan costs much less under CI's
--cov, with the same names found.test_facade_shared_checks.pycalls each opted-in module's_corpus_patched_names(), a read and parse of the whole test tree. On the Linux shards the file takes about 817 s under coverage against 261 s on Windows, which runs without it; most of the difference is the tracer running on every call of that test-side read.Why it matters
The backend suite runs on every PR, and CI time has grown (#30257). The scan is test-side code that coverage does not report, so tracing it buys nothing.
Not a goal
src/, to any module's own scan, or totest/conftest.py.What changed (motivation → approach → change)
The four call sites go through one helper,
_corpus_patched_names(module), which runs the module's scan insidesource_corpus.untraced(), the mechanism the other tree-scan gates use.One module is listed in
_SCANS_THAT_RUN_PRODUCT_CODEand stays traced:test_projection_facade's scan imports production modules and calls their lazy helpers to resolve a receiver. Run untraced, those modules' module-level lines and helpers would never be reported (measured: 7,300 lines lost before the opt-out).Known gap: nothing checks that the list is complete. A future module whose scan imports or calls production code would lose that code's coverage silently until someone adds it to
_SCANS_THAT_RUN_PRODUCT_CODE.Backwards compatibility
Compatible: test file only.
Removes nothing: every check, module and assertion stays.
Tests
The four affected tests (
test_the_patch_scan_finds_every_known_seam,..._routed_to_every_owner_holding_it,..._reaches_every_module_reading_it; 78 cases), Linux,-n0, CI's--covflags, same host:The 43-line difference is all in
src/kiro_crew/_facade_mirror.py(_module_level_statements,_defined_names,_walrus_and_capture_names), which some scans call as a helper;test/test_facade_mirror.pycovers every one of those 43 lines on its own (checked under--cov), so the combined run loses none. Memory is unchanged: nothing new is held.Planted seams, one per scan shape, under
--cov: a test file patchingZZ_PLANTED_SEAMonkiro_crew.acp._dispatch(set-shapedfacade_mirror_scanscan),kiro_crew.security.argv_floor(by-target scan) andkiro_crew.crew_log.projection(the module's own traced scan). All three are found through the helper, none is found once the file is removed, and the tracer is back in place after each call.Whole file: 305 passed under
-n 4 --dist loadgroup, and again in a shuffled order. 5 repeats of the affected tests under-n 4(390 passed). Local gate: 1,207 related backend tests pass.test_flake_pattern_ratchet.pypasses.Proof: not provable -- test-only change
Manual verification
N/A — unit coverage sufficient: the change is test-only and the plants above show each scan shape still finds a planted patch.
Related Issues
Part of #30257
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement