Skip to content

test(security): read the self-module scan budget with the tracer paused - #37530

Merged
bolichen97 merged 1 commit into
mainfrom
test/scan-budget-untraced
Nov 15, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
test/scan-budget-untraced

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_the_scan_is_linear_not_quadratic passes for the linear self-module scan on the Linux full-suite shards, and still fails when the scan goes quadratic.

Its backstop holds a 2,000-token scan to 0.5 s of thread CPU. Those shards run under coverage, and the line tracer multiplies the scan's CPU about fivefold, every time. So on the runners every reading of the linear scan is 0.6 to 0.7 s, and Backend Tests (3.12, 7) fails on PRs that never touch the scan. Reruns fail again: #36967 read 0.678 s, then 0.660 s on its rerun. #37515 lists four more PRs.

Why it matters

Every PR whose run puts this test under coverage is red until someone looks, and a rerun does not help. #36967 is held by it now. A bound that measures the instrumentation teaches people to stop reading it.

Not a goal

  • Any src/ change.
  • The structural checks in the same test (one precompute per frame, call counts in an arithmetic progression). They are unchanged.
  • The other thread-CPU ceilings in this file. Those have 1 s and 5 s budgets, or time a match that runs inside re, which the line tracer does not trace.

What changed (motivation → approach → change)

#37451 (merged) takes the smallest of five readings. That handles a one-off collection or frequency dip, but not this cost, which is the same on every reading. The readings need to measure the scan, not coverage. sys.settrace is the hook coverage's tracer uses on 3.12, so the test removes this thread's trace function for the five readings and puts it back in a finally. Several suites here already swap the trace function this way: test/redaction_structured_cases.py, test_acp_mimocode_config.py, test_per_tool_mcp_deny_single_binary.py.

  • test/test_denied_commands_security_cost.py, TestSelfModuleIndexIsLinear::test_the_scan_is_linear_not_quadratic: the five 2,000-token readings are timed with the tracer removed. The constant (0.5 s), the input, the warm-up call and the minimum of five are the same.
  • Coverage still records the scan's lines: the instrumented calls later in the test run the same path with the tracer back on.

Measured on one host (Python 3.12.13, coverage 7.10.6, CTracer), 2,000 tokens, smallest of three readings:

reading traced untraced
under coverage run 0.125 s 0.024 s
no coverage 0.026 s 0.027 s

Backwards compatibility

Compatible: test-only.

Removes nothing: no assertion, constant, input or reading count changes.

Tests

Proof: not provable -- test-only change

Planted bug in src/kiro_crew/security/argv_floor_self.py _self_module_name_index Result
re-normalize the token tail on every call (the historic quadratic), no coverage fails: 2k tokens took 0.977s
the same, under --cov=kiro_crew fails: 2k tokens took 0.991s
Run Result
the test 20 times under --cov=kiro_crew 20 passed
test_denied_commands_security_cost.py, -n 2, under --cov=kiro_crew 111 passed
test_flake_pattern_ratchet.py 68 passed
flake8, black clean

Manual verification

N/A: test-only change. The planted-bug runs and the repeat runs above are the proof.

Related Issues

Closes #37515

Related: #37446, #37451

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 06:16
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision dce2dcc17082c8061a2240b5a6cc3de1144237c1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision dce2dcc17082c8061a2240b5a6cc3de1144237c1 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of dce2dcc17082c8061a2240b5a6cc3de1144237c1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only change, and the untraced readings keep the quadratic check, with a planted quadratic still reading about 1 s.

Notes

  • If the shards ever run coverage with COVERAGE_CORE=sysmon, sys.settrace(None) won't remove the tracer and the inflated readings come back. A one-line comment saying the fix assumes CTracer would save the next person from rediscovering it.

[DESIGN-REVIEWED] dce2dcc

Description read

[DESCRIPTION-READ] c62f39a5270de70284cf9101baa6b5a43a85194f5bc781b8a3d1b409ebbfd2ef

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed dce2dcc17082c8061a2240b5a6cc3de1144237c1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] dce2dcc

Verdict parsed from the review's SHA-scoped output markers for commit dce2dcc17082c8061a2240b5a6cc3de1144237c1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable dce2dcc17082c8061a2240b5a6cc3de1144237c1 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of dce2dcc17082c8061a2240b5a6cc3de1144237c1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] dce2dcc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt dce2dcc17082c8061a2240b5a6cc3de1144237c1 false-positive: <reason, 20 to 500 characters>

test_the_scan_is_linear_not_quadratic holds a 2,000-token scan to 0.5 s
of thread CPU, now the smallest of five readings (#37451). The
full-suite Linux shards run under coverage, and its line tracer
multiplies the scan's CPU about fivefold on every reading, so each
reading of the linear scan is 0.6-0.7 s there and no minimum gets under
the bound. The test fails on PRs that never touch the scan, and reruns
fail again (#37515).

Take the five readings with this thread's trace function removed,
restoring it in a finally. Coverage still records the scan from the
instrumented calls later in the test, which take the same path. The
constant, the input, the reading count and the structural call-count
checks are unchanged.

Closes #37515

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on dce2dcc. This test-only change makes the self-module scan budget measure the scan instead of the coverage tracer. It removes this thread's trace function only around the five timed readings that #37451 added and restores it in a finally block; the 0.5 s constant, the input and the minimum of five are unchanged. A planted quadratic scan still fails the budget at about 2.2 s with and without coverage, while the linear scan reads about 0.04 s untraced under coverage. The readings are thread CPU time, not wall clock. No product code or denied-rule data changed. Decisive lines: test/test_denied_commands_security_cost.py:606-613 (tracer paused only around the five readings, restored in finally)

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@bolichen97
bolichen97 merged commit e9d6223 into main Nov 15, 2026
98 checks passed
@bolichen97
bolichen97 deleted the test/scan-budget-untraced branch November 15, 2026 07:22
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
chenmingwei23 pushed a commit that referenced this pull request Nov 15, 2026
…#37539

Two backend tests on release/0.13.0 were red for reasons main has
already fixed in test-only commits:

- test/tracing/test_tracing_otlp.py expected four spans, but #37005
  (already on the branch) adds kirocrew.step. Cherry-picked #37171
  unchanged.
- TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s
  budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of
  test/test_denied_commands_security_cost.py and
  test/denied_commands_security_helpers.py, which carries #37451 (min of
  five readings), #37530 (tracer paused) and #37539 (count work, not CPU
  time), plus the #37539 hunk of
  docs/system-specs/common/testing-conventions.md. The picks conflict
  one by one because the file also carries #28293, a test-only change.

No src/ changes.

(cherry picked from commit 6de8528)
buluoray added a commit that referenced this pull request Nov 16, 2026
…#37539

Two backend tests on release/0.13.0 were red for reasons main has
already fixed in test-only commits:

- test/tracing/test_tracing_otlp.py expected four spans, but #37005
  (already on the branch) adds kirocrew.step. Cherry-picked #37171
  unchanged.
- TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s
  budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of
  test/test_denied_commands_security_cost.py and
  test/denied_commands_security_helpers.py, which carries #37451 (min of
  five readings), #37530 (tracer paused) and #37539 (count work, not CPU
  time), plus the #37539 hunk of
  docs/system-specs/common/testing-conventions.md. The picks conflict
  one by one because the file also carries #28293, a test-only change.

No src/ changes.

(cherry picked from commit 6de8528)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_the_scan_is_linear_not_quadratic bills coverage tracing to the scan's CPU budget

2 participants