Skip to content

test: count work, not CPU time, in the deny-gate cost tests - #37539

Merged
bolichen97 merged 1 commit into
mainfrom
test/deny-gate-cost-no-clock
Nov 15, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
test/deny-gate-cost-no-clock

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The deny-gate cost tests stop failing on slow or instrumented CI shards, and each one still fails when the regression it guards comes back.

test_the_scan_is_linear_not_quadratic holds a 2,000-token scan to 0.5 s of thread CPU. Under coverage the unchanged scan read 0.51 to 0.92 s on the Linux shards and failed pull requests that never touched it. #37451 and #37530 merged first and kept the clock reading. #37530 stopped those reds: the readings are taken with the trace function removed. The file still reads a clock in five places. A clock reads the host. And under a regression the regex budgets do not fail: the test never returns.

Why it matters

Each false red blocks an unrelated pull request and turns Coverage Gate red with it. A rerun does not help, because every reading on that shard is over the line. A security test that is red for no reason teaches people to stop reading it.

Not a goal

  • Any change under src/.
  • The other test files that assert a time ceiling. They are listed under Related Issues.
  • The call-count checks in the same test. They stay as they are.

What changed (motivation → approach → change)

No test in test/test_denied_commands_security_cost.py reads a clock any more. Each clock is replaced by a count that is the same on every host.

Test Before After
test_the_scan_is_linear_not_quadratic the smallest of five untraced readings of 2,000 tokens under 0.5 s of thread CPU interpreter instructions at 250, 500 and 750 tokens grow in equal steps, and one more token costs under 20,000 instructions (4,265 today)
test_all_builtins_matchable_without_hanging flag spam under 5 s no compiled built-in holds the exponential flag run, and instructions at 250, 500 and 750 flags grow in equal steps
test_catastrophic_user_regex_does_not_freeze_is_denied, test_catastrophic_pattern_among_builtins_stays_fast_and_covers under 5 s the stored pattern's matcher is disabled and compiled nothing
test_the_flagged_pattern_is_handed_only_the_capped_window (was ..._evaluates_fast_on_a_long_input) under 1 s the regex engine is handed the 2,000-character window of a 20,001-character command, once

The count is _interpreter_work in test/denied_commands_security_helpers.py: the interpreter instructions the call executes on this thread, read through sys.monitoring. "Equal steps" means the second step costs at most a twentieth more than the first (_linear_work). Today the steps are exactly equal.

One call, three hosts:

Host state Instructions at 250, 500, 750 tokens Traced thread CPU at 2,000 tokens
quiet 1077207, 2143457, 3209707 0.068 s
under coverage 1077207, 2143457, 3209707 0.343 s
64 busy loops on 32 cores 1077207, 2143457, 3209707 0.068 s

The count sees every step taken in Python, however the loop is spelled. It cannot see work inside one C call (a slice, in on a list, str.join, a regex search), and the docstring says so. The 0.5 s clock did not catch that shape either (P4 below). The regex tests have all their work inside one call, so they assert what the engine is handed, before the hostile input runs.

The trace-function pause from #37530 leaves with the reading it served. Nothing else of the test changes: the call-count checks are as they were.

docs/system-specs/common/testing-conventions.md (D7 and class 5) said an existing guard keeps its thread-CPU budget. It now also says how a budget may leave: a count takes over, and each regression the budget caught is planted and shown red.

Backwards compatibility

Compatible: tests and docs only; no file under src/ changes.

Removes nothing: no product path, record or key is taken away. The clock assertions leave the tests, and the table under Tests shows what each one guarded failing its replacement.

Tests

Proof: not provable -- test-only change

Each regression is planted alone in a scratch copy of src/kiro_crew/security/ as it is on main 55b2e4a5e0. "main's test" is the file on that commit, with the five untraced readings. "No verdict" means the test was still running after 45 s and was killed.

Planted regression main's test This PR's test
P1 _self_module_name_index re-walks and re-normalizes the tail (the old quadratic) fails: 2k tokens took 3.324s fails: [4617961, 16224961, 34831961] instructions at (250, 500, 750) tokens
P2 it walks scan.norm with inline comparisons, calling no counted function passes fails: [1395461, 3404961, 6039461]
P3 the same walk on one line, for _t in scan.norm[i + 1 :]: pass passes fails: [1174336, 2525211, 4063586]
P4 the same walk inside one C call, "-m" in scan.norm[i + 1 :] passes passes (the stated limit)
P5 _DenyMatcher runs a pattern is_safe_user_regex refuses no verdict fails at once: _disabled is False for (a+)+$
P6 _linearize_deny_pattern returns its input unchanged no verdict fails at once: credential-exfil-s3-cp compiled the flag run
P7 the bounded match drops its 2,000-character slice no verdict fails at once: the engine was handed 20001 characters
P8 a+a+$ takes the full-input path no verdict fails at once: the engine was handed 20001 characters
P9 _next_stop_indexes walks the tail once per token (the flag-spam path) passes fails: [1068609, 2854859, 5391109] instructions at (250, 500, 750) flags
P10 the same extra loop for every token (a uniform slowdown) fails: 2k tokens took 0.614s fails: each token costs 52270 instructions

Every result in the last column is the same bare and under --cov=kiro_crew.

Run Result
the five changed tests, 20 times at -n0 20 of 20 pass
the file with test_deny_gate_call_sharing.py and test_denied_commands_security_stdin.py, 10 times at -n 4 --dist loadgroup --max-worker-restart=2 10 of 10 pass, 383 tests each
the file in a shuffled order 111 passed
a host ten times slower (a throwaway plugin scales time.thread_time), main's test red: 2k tokens took 0.673s
the same plugin, this PR 5 passed
test_flake_pattern_ratchet.py 68 passed
the repository's semgrep rules on the diff no finding

The two tests that count instructions take 1.6 s and 1.0 s on a quiet host.

Manual verification

N/A — test-only change; the planted runs above are the check.

Related Issues

Closes #37446

History: #37515 and #28212 hold the earlier reports of the same red.

Follow-up, not changed here: these test files still assert an upper bound on a clock reading. test_denied_commands_security.py, test_denied_commands_security_stdin.py (0.2 s), test_denied_commands_security_quoting.py, test_security_gate_liveness.py, test_security_regex_linearity.py, test_argv_floor_inline_and_brace_scope.py, test_acp_prompt_blocks.py, test_context_marker_neutralization.py, test_dashboard_chat_slot.py, test_image_refs_replay_strip.py, test_jsonl_util.py, test_leaked_toolcall_notice.py, test_llm_helpers_tool_input_offload.py, test_options_marker_glue_reflow.py, test_preview_text.py, test_push_branch_gate.py, test_teams_attachments.py, test_voice_reply.py, and every caller of assert_rejected_without_backtracking in test/conftest.py.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 06:50
@bolichen97
bolichen97 requested a review from CrysisDeu November 15, 2026 06:50
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8e2a08f44d3e6f7566ec32ed73199303ca314f58 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 8e2a08f44d3e6f7566ec32ed73199303ca314f58 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8e2a08f44d3e6f7566ec32ed73199303ca314f58 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: each clock budget is replaced by a work count that is the same on every host, and the planted regressions show each count still fails.

Notes

  • The PR loosens the "a ratchet may only tighten" rule in testing-conventions D7 and class 5. Planting each regression the budget caught is the right condition for removing one. The follow-up list of other clock-bound files should reuse _interpreter_work and _linear_work, not write their own copies.
  • The P4 gap (work hidden inside one C call) is now covered by neither the clock nor the count. This is stated, but the scan guard has no protection against that shape. Think about adding a characters-handed assertion like the one the regex tests use, if the scan ever builds slices per token.

[DESIGN-REVIEWED] 8e2a08f

Description read

[DESCRIPTION-READ] 2cd74dc1b15c6e70da34e33e15d4fbf982e5209307bae0ee825e5f7bb09623d9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8e2a08f44d3e6f7566ec32ed73199303ca314f58 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8e2a08f

Verdict parsed from the review's SHA-scoped output markers for commit 8e2a08f44d3e6f7566ec32ed73199303ca314f58.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8e2a08f44d3e6f7566ec32ed73199303ca314f58 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8e2a08f44d3e6f7566ec32ed73199303ca314f58 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8e2a08f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8e2a08f44d3e6f7566ec32ed73199303ca314f58 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

This PR only touches tests and docs. It replaces the clock budgets in the deny-gate cost tests with deterministic checks: a count of interpreter instructions read through sys.monitoring, checked for equal steps across three sizes plus a per-token ceiling, and, for the regex tests, assertions on what the engine is handed (a disabled matcher with nothing compiled, a built-in with no exponential flag run compiled, a single search over the 2,000-character window). I checked each against the source. _DenyMatcher's slots can be reassigned, the bounded match makes one search on text[:2000], _deny_matcher is the cache is_denied uses, and the project requires Python >=3.12. The helper uses unnamed tool slots 3 and 4 and cleans up in finally. The 20k flag-spam still runs end to end in test_denied_commands_security.py. The code matches the description. CI is green; the earlier failures were flakes in unrelated files that passed on rerun.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 8e2a08f44d · 317 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 15, 2026
test_the_scan_is_linear_not_quadratic holds a 2,000-token scan to 0.5 s of
thread CPU: the smallest of five readings, taken with the trace function
removed, since coverage's tracer put every traced reading over the line.
That is still a clock, and a clock reads the host.

The budget had one job its call counts did not do: cost added outside the
counted functions. That job moves to a count of the interpreter
instructions the call executes on this thread (sys.monitoring), bounded as
a ratio between three sizes, with a wide ceiling on what one more token
costs. The count is the same bare, under coverage and on a loaded host,
so the trace-function pause leaves with the reading. It fails the old
quadratic re-walk and also an inlined per-token walk the 0.5 s budget
passed; it cannot see work inside one C call, and the docstring says so.

The file's four other clock readings guard the regex engine, where a
regression does not finish instead of failing a budget. They now assert
what the engine is handed (nothing for a refused pattern, never the
exponential flag run, at most the capped window) before the hostile input
runs, and the flag-spam walk is held to the same instruction ratio.

testing-conventions records when a kept budget may be replaced.
@bolichen97
bolichen97 force-pushed the test/deny-gate-cost-no-clock branch from 10fbc12 to 8e2a08f Compare November 15, 2026 08:33
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 15, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 15, 2026 10:06

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff (3 files, tests and docs only; nothing under src/).

Sensitive change review (deny-gate cost guards)

  • What changed: the five thread-CPU clock assertions in test_denied_commands_security_cost.py become deterministic checks: _interpreter_work (instructions counted through sys.monitoring, this thread only, GC paused) with an equal-step ratio and a per-token ceiling for the Python walks, and handoff assertions for the regex cases (a stored catastrophic pattern compiles nothing; no compiled built-in holds _DANGEROUS_AWS_FLAG_RUN; the bounded engine is handed exactly the capped window, once).
  • Before -> after: clock budgets that read the host (0.51-0.92 s false reds under coverage) -> counts that read the same bare, traced and loaded.
  • Worst case if wrong: a quadratic or ReDoS regression in the deny gate passes these tests. Runtime behaviour does not change either way.
  • Evidence checked: the planted regressions in the PR body (P1 fails both; P2/P3 passed main's clock test and fail this one); requires-python = ">=3.12", so sys.monitoring is always present; the call-count checks are kept; CodeQL finding from the earlier head addressed on this head; PR Readiness green on this head.
  • No regression: test and docs only.
  • Undo: revert the squash commit.

@bolichen97
bolichen97 merged commit 9bb1b40 into main Nov 15, 2026
156 of 158 checks passed
@bolichen97
bolichen97 deleted the test/deny-gate-cost-no-clock branch November 15, 2026 10:37
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
chenmingwei23 pushed a commit that referenced this pull request Nov 15, 2026
…#37539

Two backend tests on release/0.13.0 were red for reasons main has
already fixed in test-only commits:

- test/tracing/test_tracing_otlp.py expected four spans, but #37005
  (already on the branch) adds kirocrew.step. Cherry-picked #37171
  unchanged.
- TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s
  budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of
  test/test_denied_commands_security_cost.py and
  test/denied_commands_security_helpers.py, which carries #37451 (min of
  five readings), #37530 (tracer paused) and #37539 (count work, not CPU
  time), plus the #37539 hunk of
  docs/system-specs/common/testing-conventions.md. The picks conflict
  one by one because the file also carries #28293, a test-only change.

No src/ changes.

(cherry picked from commit 6de8528)
buluoray added a commit that referenced this pull request Nov 16, 2026
…#37539

Two backend tests on release/0.13.0 were red for reasons main has
already fixed in test-only commits:

- test/tracing/test_tracing_otlp.py expected four spans, but #37005
  (already on the branch) adds kirocrew.step. Cherry-picked #37171
  unchanged.
- TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s
  budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of
  test/test_denied_commands_security_cost.py and
  test/denied_commands_security_helpers.py, which carries #37451 (min of
  five readings), #37530 (tracer paused) and #37539 (count work, not CPU
  time), plus the #37539 hunk of
  docs/system-specs/common/testing-conventions.md. The picks conflict
  one by one because the file also carries #28293, a test-only change.

No src/ changes.

(cherry picked from commit 6de8528)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_the_scan_is_linear_not_quadratic trips its wall-clock bound on a loaded Linux shard again

3 participants