Repository navigation
test: count work, not CPU seconds, in the deny-gate ReDoS and redirect guards - #37600
Conversation
…t guards Three deny-gate test files held their complexity guards to a thread-CPU budget (0.2 s, 1 s, 5 s). Coverage's tracer multiplies the CPU of every traced line, so each budget read the shard, not the code. - test_denied_commands_security_stdin.py: the glued-redirect guard counts interpreter instructions over three sizes and bounds what one operator costs. - test_denied_commands_security_quoting.py: the quoting regex's ambiguity is read off its parse: no character starts two alternatives of a repeated body. - test_denied_commands_security.py: TestIsDeniedReDoSResistance asserts that no full-input pattern carries the exponential aws flag run or a top-level .* gap, then that instructions grow linearly. The clock helper and the two tests that only proved it are removed. - testing-conventions.md names both shapes.
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a test-only change. Each CPU-seconds guard becomes a structural check on what the regex engine is handed plus a linear instruction-count check, and both still fail when the guarded regression returns. Notes
[DESIGN-REVIEWED] 414a8a3 Description read[DESCRIPTION-READ] f853e48e124edab2ef136e37bc580067bb86cc5fc5eae31900577240fc5e085c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Test-only change that replaces CPU-seconds budgets in the deny-gate ReDoS, quoting and glued-redirect guards with checks that don't read a clock: a structural check on what the regex engine is handed, run before any hostile input, plus linear interpreter-instruction growth. It reuses the An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Reviewed head 414a8a3, a change to tests and docs only. It replaces the CPU-seconds budgets in the deny-gate ReDoS, quoting and glued-redirect guards with two checks: a structural check on what the regex engine is handed, and a check that interpreter-instruction counts grow linearly. The changed tests pass both normally and under coverage instrumentation. To confirm they still catch what they guard, I planted three regressions: an ambiguous alternative in the ANSI-C quote regex, the aws flag-run linearization switched off, and a quadratic Python loop in the deny matcher. Each one turned the new checks red. No product code changes, and I found no defect that should block this.
Problem / Motivation
Goal: The deny-gate ReDoS, quoting and glued-redirect tests stop failing on slow or instrumented Linux shards, and each one still fails when the regression it guards comes back.
Three deny-gate test files still hold their complexity guards to a CPU-seconds budget: 0.2 s in
test_a_chain_of_glued_redirects_is_linear, 1 s intest_the_quoting_regex_is_not_redos_prone, and 5 s acrossTestIsDeniedReDoSResistance. Coverage's tracer multiplies the CPU of every traced line about fivefold, so each budget reads the shard, not the code. #37539 removed the same shape fromtest_denied_commands_security_cost.py.Why it matters
A red security test on a pull request that never touched the gate blocks it and teaches people to stop reading that test. The 0.2 s budget is the tightest absolute CPU budget left in the suite.
Not a goal
src/.What changed (motivation → approach → change)
A clock cannot tell "the code got slower" from "the runner got slower". So each test now asserts the two things that make the code fast, and neither moves with the host:
test_a_chain_of_glued_redirects_is_linear(stdin)test_the_quoting_regex_is_not_redos_prone(quoting)re._parser: inside every repeated alternation, no character can start two alternatives, so a run has one tilingTestIsDeniedReDoSResistance(security)_assert_scan_is_linear: no full-input pattern carries the exponential aws flag run or a top-level.*gap, then instructions grow linearlyEach structural check runs BEFORE the hostile input, so a regression fails at that line instead of hanging the worker. The two tests that only proved the clock helper (
test_elapsed_routes_through_the_cpu_cost_chokepoint,test_cpu_cost_is_immune_to_other_threads_where_process_time_is_not) go with the helper.testing-conventions.mdnames both new shapes beside the ones #37539 added.Backwards compatibility
Compatible: test files and docs only; no product behaviour changes.
Reader: test/test_denied_commands_security.py:TestIsDeniedReDoSResistance._cpu_cost -- release -- its only callers were this class's own budget tests, all converted here
Reader: test/test_denied_commands_security.py:test_cpu_cost_is_immune_to_other_threads_where_process_time_is_not -- release -- it tested only
_cpu_cost, which no test uses nowTests
Proof: not provable -- test-only change
Planted regressions, each red on the new tests (all green without them):
src/shell_normalizer.py)\\alternative in_ANSI_C_QUOTE_RE(the spelling check still passes)'\\' can start 2 alternativesdenied_rules.py)denied_rules.py)_DenyMatcher.matchForced-condition repeat runs on an 8-core DevSpace, main (
dc4692c96f) against this branch (414a8a36a5), the 14 to 16 converted tests,-n0. Main and branch ran at the same time, so the busy rows had both suites plus 32 busy loops on 8 cores.pytest-timeout120 s, host overload)pytest-timeout120 s, host overload)test_mid_dotstar_chain_spam_stays_linearand 3 more over budget)-n 8, coverage + busy + clock 6x, 3 runsThe timeouts are the 120 s per-test cap on a host running at 4x its cores; main shows the same class, so they are not from the instruction count.
python3 scripts/local-gate.py: backend related set 2677 passed. Repository semgrep rules on the diff: no finding.test_flake_pattern_ratchet.py: 68 passed.Manual verification
N/A — test-only change; the forced-condition runs above are the check.
Related Issues
Closes #37599
Sibling sweep
Swept: the K8 detector from #37576 over every test file, plus
git grep -nE "time\.(thread_time|process_time)\(\)" -- testSibling: test/test_denied_commands_security_stdin.py -- changed here
Sibling: test/test_denied_commands_security_quoting.py -- changed here
Sibling: test/test_denied_commands_security.py -- changed here
Sibling: test/test_mcp_gateway_stub_admission_fallback.py -- tracked in #37599 -- 0.4 s wall bound, next tightest
Sibling: test/test_context_marker_neutralization.py -- tracked in #37599 -- 1 s regex backtracking budget
Sibling: test/test_preview_text.py -- tracked in #37599 -- 1 s regex backtracking budget
Sibling: test/test_security_regex_linearity.py -- tracked in #37599 -- 5 s budget
Sibling: test/test_acp_prompt_blocks.py -- tracked in #37599 -- 5 s scan budget
Sibling: test/test_image_refs_replay_strip.py -- tracked in #37599 -- 5 s scan budget
Sibling: test/test_jsonl_util.py -- tracked in #37599 -- 5 s process_time backstop
Pattern harvest
Rule candidate: lint
Pattern: a duration held in a name the K8 detector does not read (
large,cost,self._elapsed(...)) escapes the ratchet; two of the three budgets fixed here were invisible to itChecklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)