Skip to content

test: count work, not CPU seconds, in the deny-gate ReDoS and redirect guards - #37600

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/deny-gate-timing-budgets
Nov 15, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/deny-gate-timing-budgets

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The deny-gate ReDoS, quoting and glued-redirect tests stop failing on slow or instrumented Linux shards, and each one still fails when the regression it guards comes back.

Three deny-gate test files still hold their complexity guards to a CPU-seconds budget: 0.2 s in test_a_chain_of_glued_redirects_is_linear, 1 s in test_the_quoting_regex_is_not_redos_prone, and 5 s across TestIsDeniedReDoSResistance. Coverage's tracer multiplies the CPU of every traced line about fivefold, so each budget reads the shard, not the code. #37539 removed the same shape from test_denied_commands_security_cost.py.

Why it matters

A red security test on a pull request that never touched the gate blocks it and teaches people to stop reading that test. The 0.2 s budget is the tightest absolute CPU budget left in the suite.

Not a goal

  • Any change under src/.
  • The other timing asserts the K8 ratchet lists in other files.
  • The call-count and engine-trace checks these tests already carry. They stay as they are.

What changed (motivation → approach → change)

A clock cannot tell "the code got slower" from "the runner got slower". So each test now asserts the two things that make the code fast, and neither moves with the host:

Test Was Now
test_a_chain_of_glued_redirects_is_linear (stdin) 1,600 operators in < 0.2 s thread CPU interpreter instructions grow linearly over 400/800/1,200 operators, and each operator costs under a fixed ceiling
test_the_quoting_regex_is_not_redos_prone (quoting) 2,000 backslashes decode in < 1 s parsed with re._parser: inside every repeated alternation, no character can start two alternatives, so a run has one tiling
TestIsDeniedReDoSResistance (security) 9 tests, each < 5 s thread CPU _assert_scan_is_linear: no full-input pattern carries the exponential aws flag run or a top-level .* gap, then instructions grow linearly

Each structural check runs BEFORE the hostile input, so a regression fails at that line instead of hanging the worker. The two tests that only proved the clock helper (test_elapsed_routes_through_the_cpu_cost_chokepoint, test_cpu_cost_is_immune_to_other_threads_where_process_time_is_not) go with the helper. testing-conventions.md names both new shapes beside the ones #37539 added.

Backwards compatibility

Compatible: test files and docs only; no product behaviour changes.

Reader: test/test_denied_commands_security.py:TestIsDeniedReDoSResistance._cpu_cost -- release -- its only callers were this class's own budget tests, all converted here
Reader: test/test_denied_commands_security.py:test_cpu_cost_is_immune_to_other_threads_where_process_time_is_not -- release -- it tested only _cpu_cost, which no test uses now

Tests

Proof: not provable -- test-only change

Planted regressions, each red on the new tests (all green without them):

Planted in src/ New test
per-operator Python walk over the rest of the word (shell_normalizer.py) red: 1.16M, 4.57M, 10.2M instructions at 400/800/1,200
an extra \\ alternative in _ANSI_C_QUOTE_RE (the spelling check still passes) red: '\\' can start 2 alternatives
aws flag run not linearized (denied_rules.py) 8 of 12 red at the engine check, before any hostile input runs
fragment split removed (denied_rules.py) 8 of 12 red
quadratic Python loop in _DenyMatcher.match 7 of 12 red

Forced-condition repeat runs on an 8-core DevSpace, main (dc4692c96f) against this branch (414a8a36a5), the 14 to 16 converted tests, -n0. Main and branch ran at the same time, so the busy rows had both suites plus 32 busy loops on 8 cores.

Condition main this branch
plain, 10 runs 10/10 10/10
coverage, 10 runs 10/10 10/10
coverage + 16 busy loops, 10 runs 8/10 (2 pytest-timeout 120 s, host overload) 9/10 (1 pytest-timeout 120 s, host overload)
every clock 6x fast, 10 runs 10/10 10/10
coverage + busy + clock 6x, 10 runs 0/10 (test_mid_dotstar_chain_spam_stays_linear and 3 more over budget) 10/10
whole files, -n 8, coverage + busy + clock 6x, 3 runs 0/3 3/3
shuffled order (seeds 37576, 1, 2), coverage 3/3 3/3

The timeouts are the 120 s per-test cap on a host running at 4x its cores; main shows the same class, so they are not from the instruction count.

python3 scripts/local-gate.py: backend related set 2677 passed. Repository semgrep rules on the diff: no finding. test_flake_pattern_ratchet.py: 68 passed.

Manual verification

N/A — test-only change; the forced-condition runs above are the check.

Related Issues

Closes #37599

Sibling sweep

Swept: the K8 detector from #37576 over every test file, plus git grep -nE "time\.(thread_time|process_time)\(\)" -- test
Sibling: test/test_denied_commands_security_stdin.py -- changed here
Sibling: test/test_denied_commands_security_quoting.py -- changed here
Sibling: test/test_denied_commands_security.py -- changed here
Sibling: test/test_mcp_gateway_stub_admission_fallback.py -- tracked in #37599 -- 0.4 s wall bound, next tightest
Sibling: test/test_context_marker_neutralization.py -- tracked in #37599 -- 1 s regex backtracking budget
Sibling: test/test_preview_text.py -- tracked in #37599 -- 1 s regex backtracking budget
Sibling: test/test_security_regex_linearity.py -- tracked in #37599 -- 5 s budget
Sibling: test/test_acp_prompt_blocks.py -- tracked in #37599 -- 5 s scan budget
Sibling: test/test_image_refs_replay_strip.py -- tracked in #37599 -- 5 s scan budget
Sibling: test/test_jsonl_util.py -- tracked in #37599 -- 5 s process_time backstop

Pattern harvest

Rule candidate: lint
Pattern: a duration held in a name the K8 detector does not read (large, cost, self._elapsed(...)) escapes the ratchet; two of the three budgets fixed here were invisible to it

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…t guards

Three deny-gate test files held their complexity guards to a thread-CPU
budget (0.2 s, 1 s, 5 s). Coverage's tracer multiplies the CPU of every
traced line, so each budget read the shard, not the code.

- test_denied_commands_security_stdin.py: the glued-redirect guard
  counts interpreter instructions over three sizes and bounds what one
  operator costs.
- test_denied_commands_security_quoting.py: the quoting regex's
  ambiguity is read off its parse: no character starts two alternatives
  of a repeated body.
- test_denied_commands_security.py: TestIsDeniedReDoSResistance asserts
  that no full-input pattern carries the exponential aws flag run or a
  top-level .* gap, then that instructions grow linearly. The clock
  helper and the two tests that only proved it are removed.
- testing-conventions.md names both shapes.
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 11:45
@bolichen97
bolichen97 requested a review from CrysisDeu November 15, 2026 11:45
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 414a8a36a58bc8fe4f4f870e96e975ca4480e3fa touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 414a8a36a58bc8fe4f4f870e96e975ca4480e3fa ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 414a8a36a58bc8fe4f4f870e96e975ca4480e3fa — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only change. Each CPU-seconds guard becomes a structural check on what the regex engine is handed plus a linear instruction-count check, and both still fail when the guarded regression returns.

Notes

  • _assert_scan_is_linear builds its pattern list from the private names it imports from kiro_crew.security (_GIT_PUBLISH_RULE_PATTERNS, _deny_matcher, _split_deny_frags). If a matcher gets renamed or restructured, this test breaks loudly; it won't pass silently.

[DESIGN-REVIEWED] 414a8a3

Description read

[DESCRIPTION-READ] f853e48e124edab2ef136e37bc580067bb86cc5fc5eae31900577240fc5e085c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 414a8a36a58bc8fe4f4f870e96e975ca4480e3fa — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 414a8a3

Verdict parsed from the review's SHA-scoped output markers for commit 414a8a36a58bc8fe4f4f870e96e975ca4480e3fa.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 414a8a36a58bc8fe4f4f870e96e975ca4480e3fa false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 414a8a36a58bc8fe4f4f870e96e975ca4480e3fa and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 414a8a3

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 414a8a36a58bc8fe4f4f870e96e975ca4480e3fa false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only change that replaces CPU-seconds budgets in the deny-gate ReDoS, quoting and glued-redirect guards with checks that don't read a clock: a structural check on what the regex engine is handed, run before any hostile input, plus linear interpreter-instruction growth. It reuses the _interpreter_work and _linear_work helpers that #37539 already introduced. Every private name the new checks read exists in denied_rules.py. The re._parser check on _ANSI_C_QUOTE_RE finds exactly two repeated alternations, with no character able to start two alternatives. The removed helpers have no remaining callers. No src/ changes, so product behaviour can't regress.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 414a8a36a5 · 373 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 414a8a3, a change to tests and docs only. It replaces the CPU-seconds budgets in the deny-gate ReDoS, quoting and glued-redirect guards with two checks: a structural check on what the regex engine is handed, and a check that interpreter-instruction counts grow linearly. The changed tests pass both normally and under coverage instrumentation. To confirm they still catch what they guard, I planted three regressions: an ambiguous alternative in the ANSI-C quote regex, the aws flag-run linearization switched off, and a quadratic Python loop in the deny matcher. Each one turned the new checks red. No product code changes, and I found no defect that should block this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: deny-gate ReDoS and redirect guards still assert CPU seconds on instrumented shards

2 participants