Repository navigation
test(security): read the self-module scan budget with the tracer paused - #37530
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a test-only change, and the untraced readings keep the quadratic check, with a planted quadratic still reading about 1 s. Notes
[DESIGN-REVIEWED] dce2dcc Description read[DESCRIPTION-READ] c62f39a5270de70284cf9101baa6b5a43a85194f5bc781b8a3d1b409ebbfd2ef That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
test_the_scan_is_linear_not_quadratic holds a 2,000-token scan to 0.5 s of thread CPU, now the smallest of five readings (#37451). The full-suite Linux shards run under coverage, and its line tracer multiplies the scan's CPU about fivefold on every reading, so each reading of the linear scan is 0.6-0.7 s there and no minimum gets under the bound. The test fails on PRs that never touch the scan, and reruns fail again (#37515). Take the five readings with this thread's trace function removed, restoring it in a finally. Coverage still records the scan from the instrumented calls later in the test, which take the same path. The constant, the input, the reading count and the structural call-count checks are unchanged. Closes #37515
1bb3dc2 to
dce2dcc
Compare
buluoray
left a comment
There was a problem hiding this comment.
Approve on dce2dcc. This test-only change makes the self-module scan budget measure the scan instead of the coverage tracer. It removes this thread's trace function only around the five timed readings that #37451 added and restores it in a finally block; the 0.5 s constant, the input and the minimum of five are unchanged. A planted quadratic scan still fails the budget at about 2.2 s with and without coverage, while the linear scan reads about 0.04 s untraced under coverage. The readings are thread CPU time, not wall clock. No product code or denied-rule data changed. Decisive lines: test/test_denied_commands_security_cost.py:606-613 (tracer paused only around the five readings, restored in finally)
…#37539 Two backend tests on release/0.13.0 were red for reasons main has already fixed in test-only commits: - test/tracing/test_tracing_otlp.py expected four spans, but #37005 (already on the branch) adds kirocrew.step. Cherry-picked #37171 unchanged. - TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of test/test_denied_commands_security_cost.py and test/denied_commands_security_helpers.py, which carries #37451 (min of five readings), #37530 (tracer paused) and #37539 (count work, not CPU time), plus the #37539 hunk of docs/system-specs/common/testing-conventions.md. The picks conflict one by one because the file also carries #28293, a test-only change. No src/ changes. (cherry picked from commit 6de8528)
…#37539 Two backend tests on release/0.13.0 were red for reasons main has already fixed in test-only commits: - test/tracing/test_tracing_otlp.py expected four spans, but #37005 (already on the branch) adds kirocrew.step. Cherry-picked #37171 unchanged. - TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of test/test_denied_commands_security_cost.py and test/denied_commands_security_helpers.py, which carries #37451 (min of five readings), #37530 (tracer paused) and #37539 (count work, not CPU time), plus the #37539 hunk of docs/system-specs/common/testing-conventions.md. The picks conflict one by one because the file also carries #28293, a test-only change. No src/ changes. (cherry picked from commit 6de8528)
Problem / Motivation
Goal:
test_the_scan_is_linear_not_quadraticpasses for the linear self-module scan on the Linux full-suite shards, and still fails when the scan goes quadratic.Its backstop holds a 2,000-token scan to 0.5 s of thread CPU. Those shards run under coverage, and the line tracer multiplies the scan's CPU about fivefold, every time. So on the runners every reading of the linear scan is 0.6 to 0.7 s, and
Backend Tests (3.12, 7)fails on PRs that never touch the scan. Reruns fail again: #36967 read 0.678 s, then 0.660 s on its rerun. #37515 lists four more PRs.Why it matters
Every PR whose run puts this test under coverage is red until someone looks, and a rerun does not help. #36967 is held by it now. A bound that measures the instrumentation teaches people to stop reading it.
Not a goal
src/change.re, which the line tracer does not trace.What changed (motivation → approach → change)
#37451 (merged) takes the smallest of five readings. That handles a one-off collection or frequency dip, but not this cost, which is the same on every reading. The readings need to measure the scan, not coverage.
sys.settraceis the hook coverage's tracer uses on 3.12, so the test removes this thread's trace function for the five readings and puts it back in afinally. Several suites here already swap the trace function this way:test/redaction_structured_cases.py,test_acp_mimocode_config.py,test_per_tool_mcp_deny_single_binary.py.test/test_denied_commands_security_cost.py,TestSelfModuleIndexIsLinear::test_the_scan_is_linear_not_quadratic: the five 2,000-token readings are timed with the tracer removed. The constant (0.5 s), the input, the warm-up call and the minimum of five are the same.Measured on one host (Python 3.12.13, coverage 7.10.6,
CTracer), 2,000 tokens, smallest of three readings:coverage runBackwards compatibility
Compatible: test-only.
Removes nothing: no assertion, constant, input or reading count changes.
Tests
Proof: not provable -- test-only change
src/kiro_crew/security/argv_floor_self.py_self_module_name_index2k tokens took 0.977s--cov=kiro_crew2k tokens took 0.991s--cov=kiro_crewtest_denied_commands_security_cost.py,-n 2, under--cov=kiro_crewtest_flake_pattern_ratchet.pyManual verification
N/A: test-only change. The planted-bug runs and the repeat runs above are the proof.
Related Issues
Closes #37515
Related: #37446, #37451
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)