Repository navigation
test: count work, not CPU time, in the deny-gate cost tests - #37539
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns: each clock budget is replaced by a work count that is the same on every host, and the planted regressions show each count still fails. Notes
[DESIGN-REVIEWED] 8e2a08f Description read[DESCRIPTION-READ] 2cd74dc1b15c6e70da34e33e15d4fbf982e5209307bae0ee825e5f7bb09623d9 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. This PR only touches tests and docs. It replaces the clock budgets in the deny-gate cost tests with deterministic checks: a count of interpreter instructions read through sys.monitoring, checked for equal steps across three sizes plus a per-token ceiling, and, for the regex tests, assertions on what the engine is handed (a disabled matcher with nothing compiled, a built-in with no exponential flag run compiled, a single search over the 2,000-character window). I checked each against the source. _DenyMatcher's slots can be reassigned, the bounded match makes one search on text[:2000], _deny_matcher is the cache is_denied uses, and the project requires Python >=3.12. The helper uses unnamed tool slots 3 and 4 and cleans up in finally. The 20k flag-spam still runs end to end in test_denied_commands_security.py. The code matches the description. CI is green; the earlier failures were flakes in unrelated files that passed on rerun. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
test_the_scan_is_linear_not_quadratic holds a 2,000-token scan to 0.5 s of thread CPU: the smallest of five readings, taken with the trace function removed, since coverage's tracer put every traced reading over the line. That is still a clock, and a clock reads the host. The budget had one job its call counts did not do: cost added outside the counted functions. That job moves to a count of the interpreter instructions the call executes on this thread (sys.monitoring), bounded as a ratio between three sizes, with a wide ceiling on what one more token costs. The count is the same bare, under coverage and on a loaded host, so the trace-function pause leaves with the reading. It fails the old quadratic re-walk and also an inlined per-token walk the 0.5 s budget passed; it cannot see work inside one C call, and the docstring says so. The file's four other clock readings guard the regex engine, where a regression does not finish instead of failing a budget. They now assert what the engine is handed (nothing for a refused pattern, never the exponential flag run, at most the capped window) before the hostile input runs, and the flag-spam walk is held to the same instruction ratio. testing-conventions records when a kept budget may be replaced.
10fbc12 to
8e2a08f
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Reviewed the full diff (3 files, tests and docs only; nothing under src/).
Sensitive change review (deny-gate cost guards)
- What changed: the five thread-CPU clock assertions in
test_denied_commands_security_cost.pybecome deterministic checks:_interpreter_work(instructions counted throughsys.monitoring, this thread only, GC paused) with an equal-step ratio and a per-token ceiling for the Python walks, and handoff assertions for the regex cases (a stored catastrophic pattern compiles nothing; no compiled built-in holds_DANGEROUS_AWS_FLAG_RUN; the bounded engine is handed exactly the capped window, once). - Before -> after: clock budgets that read the host (0.51-0.92 s false reds under coverage) -> counts that read the same bare, traced and loaded.
- Worst case if wrong: a quadratic or ReDoS regression in the deny gate passes these tests. Runtime behaviour does not change either way.
- Evidence checked: the planted regressions in the PR body (P1 fails both; P2/P3 passed main's clock test and fail this one);
requires-python = ">=3.12", sosys.monitoringis always present; the call-count checks are kept; CodeQL finding from the earlier head addressed on this head; PR Readiness green on this head. - No regression: test and docs only.
- Undo: revert the squash commit.
…#37539 Two backend tests on release/0.13.0 were red for reasons main has already fixed in test-only commits: - test/tracing/test_tracing_otlp.py expected four spans, but #37005 (already on the branch) adds kirocrew.step. Cherry-picked #37171 unchanged. - TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of test/test_denied_commands_security_cost.py and test/denied_commands_security_helpers.py, which carries #37451 (min of five readings), #37530 (tracer paused) and #37539 (count work, not CPU time), plus the #37539 hunk of docs/system-specs/common/testing-conventions.md. The picks conflict one by one because the file also carries #28293, a test-only change. No src/ changes. (cherry picked from commit 6de8528)
…#37539 Two backend tests on release/0.13.0 were red for reasons main has already fixed in test-only commits: - test/tracing/test_tracing_otlp.py expected four spans, but #37005 (already on the branch) adds kirocrew.step. Cherry-picked #37171 unchanged. - TestSelfModuleIndexIsLinear failed three runs in a row on the 0.5 s budget (0.522 s, 0.511 s, 0.509 s). Took main's end state of test/test_denied_commands_security_cost.py and test/denied_commands_security_helpers.py, which carries #37451 (min of five readings), #37530 (tracer paused) and #37539 (count work, not CPU time), plus the #37539 hunk of docs/system-specs/common/testing-conventions.md. The picks conflict one by one because the file also carries #28293, a test-only change. No src/ changes. (cherry picked from commit 6de8528)
Problem / Motivation
Goal: The deny-gate cost tests stop failing on slow or instrumented CI shards, and each one still fails when the regression it guards comes back.
test_the_scan_is_linear_not_quadraticholds a 2,000-token scan to 0.5 s of thread CPU. Under coverage the unchanged scan read 0.51 to 0.92 s on the Linux shards and failed pull requests that never touched it. #37451 and #37530 merged first and kept the clock reading. #37530 stopped those reds: the readings are taken with the trace function removed. The file still reads a clock in five places. A clock reads the host. And under a regression the regex budgets do not fail: the test never returns.Why it matters
Each false red blocks an unrelated pull request and turns
Coverage Gatered with it. A rerun does not help, because every reading on that shard is over the line. A security test that is red for no reason teaches people to stop reading it.Not a goal
src/.What changed (motivation → approach → change)
No test in
test/test_denied_commands_security_cost.pyreads a clock any more. Each clock is replaced by a count that is the same on every host.test_the_scan_is_linear_not_quadratictest_all_builtins_matchable_without_hangingtest_catastrophic_user_regex_does_not_freeze_is_denied,test_catastrophic_pattern_among_builtins_stays_fast_and_coverstest_the_flagged_pattern_is_handed_only_the_capped_window(was..._evaluates_fast_on_a_long_input)The count is
_interpreter_workintest/denied_commands_security_helpers.py: the interpreter instructions the call executes on this thread, read throughsys.monitoring. "Equal steps" means the second step costs at most a twentieth more than the first (_linear_work). Today the steps are exactly equal.One call, three hosts:
The count sees every step taken in Python, however the loop is spelled. It cannot see work inside one C call (a slice,
inon a list,str.join, a regex search), and the docstring says so. The 0.5 s clock did not catch that shape either (P4 below). The regex tests have all their work inside one call, so they assert what the engine is handed, before the hostile input runs.The trace-function pause from #37530 leaves with the reading it served. Nothing else of the test changes: the call-count checks are as they were.
docs/system-specs/common/testing-conventions.md(D7 and class 5) said an existing guard keeps its thread-CPU budget. It now also says how a budget may leave: a count takes over, and each regression the budget caught is planted and shown red.Backwards compatibility
Compatible: tests and docs only; no file under
src/changes.Removes nothing: no product path, record or key is taken away. The clock assertions leave the tests, and the table under Tests shows what each one guarded failing its replacement.
Tests
Proof: not provable -- test-only change
Each regression is planted alone in a scratch copy of
src/kiro_crew/security/as it is on main55b2e4a5e0. "main's test" is the file on that commit, with the five untraced readings. "No verdict" means the test was still running after 45 s and was killed._self_module_name_indexre-walks and re-normalizes the tail (the old quadratic)2k tokens took 3.324s[4617961, 16224961, 34831961] instructions at (250, 500, 750) tokensscan.normwith inline comparisons, calling no counted function[1395461, 3404961, 6039461]for _t in scan.norm[i + 1 :]: pass[1174336, 2525211, 4063586]"-m" in scan.norm[i + 1 :]_DenyMatcherruns a patternis_safe_user_regexrefuses_disabledisFalsefor(a+)+$_linearize_deny_patternreturns its input unchangedcredential-exfil-s3-cpcompiled the flag runmatchdrops its 2,000-character slicethe engine was handed 20001 charactersa+a+$takes the full-input paththe engine was handed 20001 characters_next_stop_indexeswalks the tail once per token (the flag-spam path)[1068609, 2854859, 5391109] instructions at (250, 500, 750) flags2k tokens took 0.614seach token costs 52270 instructionsEvery result in the last column is the same bare and under
--cov=kiro_crew.-n0test_deny_gate_call_sharing.pyandtest_denied_commands_security_stdin.py, 10 times at-n 4 --dist loadgroup --max-worker-restart=2time.thread_time), main's test2k tokens took 0.673stest_flake_pattern_ratchet.pyThe two tests that count instructions take 1.6 s and 1.0 s on a quiet host.
Manual verification
N/A — test-only change; the planted runs above are the check.
Related Issues
Closes #37446
History: #37515 and #28212 hold the earlier reports of the same red.
Follow-up, not changed here: these test files still assert an upper bound on a clock reading.
test_denied_commands_security.py,test_denied_commands_security_stdin.py(0.2 s),test_denied_commands_security_quoting.py,test_security_gate_liveness.py,test_security_regex_linearity.py,test_argv_floor_inline_and_brace_scope.py,test_acp_prompt_blocks.py,test_context_marker_neutralization.py,test_dashboard_chat_slot.py,test_image_refs_replay_strip.py,test_jsonl_util.py,test_leaked_toolcall_notice.py,test_llm_helpers_tool_input_offload.py,test_options_marker_glue_reflow.py,test_preview_text.py,test_push_branch_gate.py,test_teams_attachments.py,test_voice_reply.py, and every caller ofassert_rejected_without_backtrackingintest/conftest.py.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.