Skip to content

test: pause the coverage tracer while timing the self-restart scan - #37532

Closed
chenmingwei23 wants to merge 1 commit into
mainfrom
fix/37515-scan-budget-coverage
Closed

chenmingwei23 wants to merge 1 commit into
mainfrom
fix/37515-scan-budget-coverage

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_the_scan_is_linear_not_quadratic measures the self-restart scan, not coverage's line tracer, so it stops going red on Linux Backend Tests (3.12, 7) for PRs that never touch the scan.

The test's 0.5 s CPU backstop is read while coverage traces every line. The CI shards run with --cov, and the tracer multiplies the scan's CPU about five-fold, so a linear scan reads 0.6 to 0.7 s on a slower runner. Main itself is red on it (2k tokens took 0.677s), and the minimum of five readings does not help, because every traced reading is that slow.

Why it matters

Every PR whose run lands on shard 7 goes red on a test it did not touch, and a rerun usually fails again. Authors either wait or rerun, and the red hides real failures on that shard.

Not a goal

  • Raising the 0.5 s bound or skipping the test.
  • The other thread_time budgets in this file: they pass with headroom under coverage today.

What changed (motivation → approach → change)

The backstop exists to catch a quadratic scan, so it should time the scan alone. Each timed reading now pauses the Python trace function (sys.settrace(None)) and restores it afterwards. The scan's frames start after the pause, so none of them is traced. The bound, the five readings and the structural call-count checks are unchanged.

Backwards compatibility

Test-only change. No product code changes.

Tests

Measured on one Linux host under coverage (CTracer), 2,000 tokens, CPU seconds:

reading result
tracer on (before) 0.257 to 0.266
tracer paused (this change) 0.046 to 0.048
planted quadratic, tracer paused 3.37 to 3.42, still far over 0.5

The single test passes with and without --cov=kiro_crew locally.

Manual verification

Not applicable: test-only change.

Screenshots / video

Not applicable: no UI change.

Related Issues

Fixes #37515

Related: #37446

Sibling sweep

The same file has four other time.thread_time() budgets (the deny-rule backtracking checks). They pass under coverage with headroom and are left as they are.

Pattern harvest

  • Pattern: a CPU budget read while coverage traces the code under test measures the tracer. Where a test times code against an absolute bound, pause the tracer for the timed window.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 2d790db82103e658bcbe5933934f3f5a6f2f919d ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 2d790db82103e658bcbe5933934f3f5a6f2f919d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 2d790db82103e658bcbe5933934f3f5a6f2f919d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This is a test-only change that can be reverted cleanly, and the quadratic-scan backstop still catches a planted quadratic by a wide margin.

Notes

  • sys.settrace(None) only pauses a settrace-based tracer such as CTracer. If coverage runs on its sysmon core (COVERAGE_CORE=sysmon, or the default on newer Pythons), this pause does nothing and the flake comes back. Pausing through coverage's own API (coverage.Coverage.current(), then .stop() and .start()) would hold on either core.

[DESIGN-REVIEWED] 2d790db

Description read

[DESCRIPTION-READ] 44ffc76f4eb465f3018db5067b48dc6ed185cbdc12fd1acc0f6b1e1ef9a2b06d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 2d790db82103e658bcbe5933934f3f5a6f2f919d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2d790db

Verdict parsed from the review's SHA-scoped output markers for commit 2d790db82103e658bcbe5933934f3f5a6f2f919d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2d790db82103e658bcbe5933934f3f5a6f2f919d false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 2d790db82103e658bcbe5933934f3f5a6f2f919d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2d790db

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2d790db82103e658bcbe5933934f3f5a6f2f919d false-positive: <reason, 20 to 500 characters>

@chenmingwei23
chenmingwei23 force-pushed the fix/37515-scan-budget-coverage branch from 6b483fd to 2d790db Compare November 15, 2026 06:28
@chenmingwei23
chenmingwei23 enabled auto-merge (squash) November 15, 2026 06:37
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

This PR changes one test and nothing else. Each timed reading of the self-restart scan now pauses the trace function with sys.settrace(None) and puts back the saved tracer in a finally. The 0.5 s bound, the five-reading minimum and the structural call-count checks are unchanged. The quadratic regression the test exists to catch still reads about 3.4 s, well over 0.5 s, so the backstop still works.

Restoring the tracer object returned by sys.gettrace() is safe. It is the same save-and-restore pattern several other tests in the repo already use. Coverage's CTracer also supports being called this way and puts its C hook back on the next call.

Design Review worried about coverage's sysmon core, where this pause would do nothing. That doesn't apply on this CI. Coverage is pinned to 7.10.6 and the shards run Python 3.12, and on 3.12 that version uses the settrace-based CTracer by default. setup.cfg also sets branch = true, which the sysmon core does not support on 3.12. So the pause takes effect.

Coverage of the security module is not lost: the call-count part of the same test runs without the pause and still goes through those lines. Backend Tests (3.12, 7), the shard this

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 2d790db821 · 17 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This PR and #37530 both close #37515. Both change only TestSelfModuleIndexIsLinear::test_the_scan_is_linear_not_quadratic in test/test_denied_commands_security_cost.py, and both use the same fix: save sys.gettrace(), call sys.settrace(None) for the five thread_time readings, and put the tracer back in a finally. Please keep one of #37530 and #37532 and close the other, carrying over anything only this one has.

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Closing as a duplicate: #37530 merged the same fix (the scan budget is read with the tracer paused) and closed #37515.

auto-merge was automatically disabled November 15, 2026 07:27

Pull request was closed

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_the_scan_is_linear_not_quadratic bills coverage tracing to the scan's CPU budget

2 participants