Skip to content

test: run the shared facade checks' tree scans with the tracer paused - #38674

Merged
iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/facade-seam-shared-parse
Nov 17, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
crew/testoptimizer/facade-seam-shared-parse

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The shared facade checks run each module's whole-test-tree patch scan with the coverage tracer paused, so the scan costs much less under CI's --cov, with the same names found.

test_facade_shared_checks.py calls each opted-in module's _corpus_patched_names(), a read and parse of the whole test tree. On the Linux shards the file takes about 817 s under coverage against 261 s on Windows, which runs without it; most of the difference is the tracer running on every call of that test-side read.

Why it matters

The backend suite runs on every PR, and CI time has grown (#30257). The scan is test-side code that coverage does not report, so tracing it buys nothing.

Not a goal

What changed (motivation → approach → change)

The four call sites go through one helper, _corpus_patched_names(module), which runs the module's scan inside source_corpus.untraced(), the mechanism the other tree-scan gates use.

One module is listed in _SCANS_THAT_RUN_PRODUCT_CODE and stays traced: test_projection_facade's scan imports production modules and calls their lazy helpers to resolve a receiver. Run untraced, those modules' module-level lines and helpers would never be reported (measured: 7,300 lines lost before the opt-out).

Known gap: nothing checks that the list is complete. A future module whose scan imports or calls production code would lose that code's coverage silently until someone adds it to _SCANS_THAT_RUN_PRODUCT_CODE.

Backwards compatibility

Compatible: test file only.

Removes nothing: every check, module and assertion stays.

Tests

The four affected tests (test_the_patch_scan_finds_every_known_seam, ..._routed_to_every_owner_holding_it, ..._reaches_every_module_reading_it; 78 cases), Linux, -n0, CI's --cov flags, same host:

main branch
summed call time (s) 344.4 243.8
wall (s) 483.7 373.7
peak RSS (VmHWM, MB) 943 945
covered lines 128,300 128,257

The 43-line difference is all in src/kiro_crew/_facade_mirror.py (_module_level_statements, _defined_names, _walrus_and_capture_names), which some scans call as a helper; test/test_facade_mirror.py covers every one of those 43 lines on its own (checked under --cov), so the combined run loses none. Memory is unchanged: nothing new is held.

Planted seams, one per scan shape, under --cov: a test file patching ZZ_PLANTED_SEAM on kiro_crew.acp._dispatch (set-shaped facade_mirror_scan scan), kiro_crew.security.argv_floor (by-target scan) and kiro_crew.crew_log.projection (the module's own traced scan). All three are found through the helper, none is found once the file is removed, and the tracer is back in place after each call.

Whole file: 305 passed under -n 4 --dist loadgroup, and again in a shuffled order. 5 repeats of the affected tests under -n 4 (390 passed). Local gate: 1,207 related backend tests pass. test_flake_pattern_ratchet.py passes.

Proof: not provable -- test-only change

Manual verification

N/A — unit coverage sufficient: the change is test-only and the plants above show each scan shape still finds a planted patch.

Related Issues

Part of #30257

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

The shared facade checks call each module's _corpus_patched_names(),
which reads the whole test tree. Under CI's --cov the tracer ran on
every call of that test-side read. The four call sites now go through
one helper that pauses the tracer, except for test_projection_facade,
whose scan imports and calls production code to resolve a receiver and
so stays traced.
@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 17, 2026 17:41
@iamwhatever
iamwhatever enabled auto-merge (squash) November 17, 2026 17:41
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision e864eca22379d97f1f762da5de42efbcb431295b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision e864eca22379d97f1f762da5de42efbcb431295b ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of e864eca22379d97f1f762da5de42efbcb431295b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only change, reversible, and it reuses the source_corpus.untraced() mechanism that already exists.

Notes

  • _SCANS_THAT_RUN_PRODUCT_CODE is a hand-kept deny-list. A new opted-in module whose scan imports production code would quietly lose coverage lines until someone adds its stem. A cheap fix is an allow-list, or asserting that the covered-line delta stays at zero.

[DESIGN-REVIEWED] e864eca

Description read

[DESCRIPTION-READ] e564d135f109d81da10175a3f4a1c881f68e9c92bebf04abf35a34ad18e6a098

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of e864eca22379d97f1f762da5de42efbcb431295b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e864eca

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e864eca22379d97f1f762da5de42efbcb431295b false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e864eca22379d97f1f762da5de42efbcb431295b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e864eca

Verdict parsed from the review's SHA-scoped output markers for commit e864eca22379d97f1f762da5de42efbcb431295b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e864eca22379d97f1f762da5de42efbcb431295b false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 17, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, but it is held at tier pending: being reviewed.

Test-only change that routes four shared facade checks' whole-tree patch scans through source_corpus.untraced(), the existing per-thread settrace pause used by ~20 other gates, including the sibling test_bindings_shared_checks. What the checks assert is unchanged, since pausing the tracer only affects coverage. The one scan that imports production modules and calls their helpers (test_projection_facade, via _returns_the_facade) is opted out and stays traced. The other opted-in scans read and parse text through facade_mirror_scan; the import_module calls in those files are in test bodies, not in the scan. The reported 43-line drop in _facade_mirror is said to be covered by test_facade_mirror.py, and the Coverage Gate passed. Because the one traced module is excluded, the order in which the cached scan result is first filled doesn't change what coverage reports. CI is green on Linux and Windows.

Tier: pending: being reviewed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit e864eca223 · 27 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On this head the review checked that the four module._corpus_patched_names() call sites now go through one helper that runs the scan inside the existing source_corpus.untraced(). That context manager puts the tracer back in a finally. The assertions sit outside it, and no assertion, count or allowlist changes. We checked the 64 opted-in scans. Only test_projection_facade imports and calls production modules, so it is the one scan that must stay traced. _load imports by stem, so the module.__name__ match holds. The PR touches no src/, security, sandbox or auth path.

Done well: you measured the coverage cost before you changed anything. You found the 7,300-line loss in the projection scan, kept that scan traced, and traced the remaining 43 lines to _facade_mirror.py and the test that already covers them.

Optional:

  • The comment at test/test_facade_shared_checks.py:166 and the docstring at :173 say scans that run production code stay traced. But the argv_floor, denied_rules, exfil, paths, redaction and session_ledger scans call _facade_mirror helpers with the tracer off. A more exact rule: a scan stays traced when it runs production code that no other test covers.
  • A one-line pin such as assert _SCANS_THAT_RUN_PRODUCT_CODE <= set(_OPTED_IN) would catch a renamed or mistyped stem. Today that mistake would quietly run the projection scan untraced.
  • A short disposition reply to the Design lane note about the hand-kept list would close it. The Known gap section in the body already covers it.

@iamwhatever
iamwhatever merged commit e364a09 into main Nov 17, 2026
104 of 105 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/facade-seam-shared-parse branch November 17, 2026 19:23
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants