Repository navigation
test: refuse CPU-clock budgets in the flake ratchet and review rules - #37576
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The change only widens a ratchet detector and adds review and doc text. The ratchet compares each file against its merge-base, so a file nobody touched cannot turn red. [DESIGN-REVIEWED] f3824f8 Description read[DESCRIPTION-READ] 962732b298a7d22c85cb87997e9df0cf501842bd4ac6fbedb59d4523c3b4d972 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Each item stops a CPU-clock budget from coming back, and the ratchet has exactly one consumer ( What this change shipsInventory (4 items) — 4 justifiedIntent: Stop the deny-gate cost tests failing for no reason on CI shards that run under coverage, and keep the seconds-budget shape out of future tests. This is a FIX (it guards against regression).
[FIRST-PRINCIPLES-REVIEWED] f3824f8 Description read[DESCRIPTION-READ] 34bc6e5dcbd32a5af66911c7904eae19d4b3c5a88458a585bf14a745fd394a87 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. The PR adds thread_time, process_time and their _ns forms to the clock calls the K8 duration detector counts. It adds a planted case to the detector table and aligns the AUTOSDE rule, D7 in testing-conventions and the writing-tests skill text. change_risen scores both the merge-base and head copies of each changed file with the current detectors, so files the PR doesn't touch can't turn red. _CLOCK_CALLS has one consumer (K8). The new planted case would fail without the change. The doc and rule text agree with each other, and the helpers they cite exist. No trust boundary is touched. The Flake Pattern Ratchet and Source Gates checks passed. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
2b41721 to
9c59ced
Compare
|
A note on how this PR relates to two other open PRs that touch
Which way to go is your call; this is only so the overlap is visible in one place. |
The deny-gate scan budget was read on thread CPU, so the flake-pattern ratchet's duration detector never counted it: it knew only wall clocks. Coverage's tracer multiplies traced CPU about five times, so that budget turned 159 pull requests red in 14 days, and two fixes in a row reshaped its reading (minimum of five, tracer paused) instead of removing it. The K8 detector now counts thread_time and process_time deltas like any other clock; counts are per file against the merge-base, so no untouched file turns red. AUTOSDE's tests-are-deterministic item 5 says a CPU clock is no exception and flags reshaping a kept budget. testing-conventions D7 and the writing-tests skill say the same and point at _interpreter_work as the count to use.
9c59ced to
f3824f8
Compare
buluoray
left a comment
There was a problem hiding this comment.
Approve on f3824f8. This PR keeps CPU-clock budgets from coming back into the test suite now that #37539 has removed the one in the deny-gate cost test. The flake ratchet's duration check now treats thread_time and process_time (and their _ns forms) as clocks. The ratchet compares each changed file with its own count at the merge-base, so a file that a PR does not touch cannot turn red, and no file in the tree has the newly counted inline shape today. When we removed the four CPU clock names from the detector, the ratchet's planted thread_time case failed, so the new case guards the change. The review rule stays scoped to added or changed test lines, and it still lets an existing complexity guard keep its untouched budget, which matches the testing conventions. One follow-up: the shared ReDoS helper in test/conftest.py reads thread CPU against an absolute budget and takes the minimum of two or three readings, and the testing conventions still tell authors to convert a regex guard to it. The new review text flags "the minimum of N readings" without naming that helper as an exception, so the review bot could flag a conversion that the conventions prescribe. A one-line carve-out in the rule and the skill would remove the conflict. Decisive lines: test/test_flake_pattern_ratchet.py:122-136, test/test_flake_pattern_ratchet.py:372-382, test/test_flake_pattern_ratchet.py:1151-1191, test/test_flake_pattern_ratchet.py:1280-1281, AUTOSDE.yaml:370-371, AUTOSDE.yaml:390, AUTOSDE.yaml:416-431, docs/system-specs/common/testing-conventions.md:97-103
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Verified on this head: the only code change adds the four CPU clock names to _CLOCK_CALLS, and K8 is its only reader. The planted time.thread_time() case counts 1 at head and 0 with main's set, so reverting the hunk turns the detector table test red. The ratchet counts each file against its own merge-base, so files that already hold a CPU clock budget stay green. AUTOSDE.yaml parses with no rule id changed, and the cited _interpreter_work / _linear_work helpers exist. PR Readiness is green.
Done well: the PR was rebased onto #37539 and now carries only the prevention work, as a small data change with a real mutation pin.
Optional:
- The new D7 sentence suggests K8 catches a reshaped budget. It does not: K8 counts only an inline clock delta, or a name like
elapsed, compared against a numeric literal. Solarge = min(...)followed byassert large < 0.5stays uncounted. Consider narrowing the sentence to what the detector counts. - AUTOSDE.yaml item 5 and the new SKILL.md paragraph flag "the minimum of N readings", but
conftest.assert_rejected_without_backtracking, which the same docs prescribe for regex guards, does exactly that. A carve-out for that helper would avoid a false BLOCK later. - "fails there on every rerun" and "about fivefold" do not match the body's numbers (0.07 s x 5 is under 0.5 s, and 146 of 159 went green on a rerun). Something like "can land on either side of the budget on an instrumented shard" would be accurate. A
process_timepositive case would also pin the second clock family.
Problem / Motivation
Goal: The deny-gate cost tests stop failing on slow or instrumented CI shards, and each one still fails when the regression it guards comes back.
test_the_scan_is_linear_not_quadraticholds a 2,000-token scan to 0.5 s of thread CPU. The scan costs about 0.07 s bare. Coverage's tracer multiplies that about five times, so on the instrumented Linux shards the unchanged scan read 0.51 to 0.92 s. In the last 14 days it turned 159 pull requests red, 146 of them green on a rerun, and 13 open pull requests were still held by it. #37451 took the smallest of five readings; #37530 pausessys.settracearound them. Both still assert seconds.Why it matters
Each false red blocks an unrelated pull request and turns
Coverage Gatered with it. A security test that is red for no reason teaches people to stop reading it.Not a goal
src/other than thewriting-testsskill text.What changed (motivation → approach → change)
The test change itself landed in #37539 (count instructions, not CPU seconds). This PR is what keeps the shape from coming back; rebased onto main after #37539 merged, it carries only that.
The budget escaped review twice because nothing recognised a CPU clock as a clock: the K8 detector in the flake ratchet knew only wall clocks, and two fixes in a row reshaped the reading (#37451 minimum of five, #37530 tracer paused) instead of removing it.
test/test_flake_pattern_ratchet.py: the K8 duration detector countsthread_timeandprocess_timedeltas like any other clock, with a planted case. Counts are compared per file against the merge-base, so no untouched file turns red.AUTOSDE.yamltests-are-deterministicitem 5: a CPU clock is no exception, and reshaping a kept budget (minimum of N, tracer paused, bigger constant) is flagged.docs/system-specs/common/testing-conventions.md(D7) andsrc/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md: reshaping a kept budget is not a fix; count the work and bound its growth (_interpreter_work/_linear_work).Backwards compatibility
Compatible: a test ratchet, a review rule and docs only; no product behaviour changes.
Removes nothing: the ratchet only stops a file's count of the shape from rising.
Tests
Proof: not provable -- test-only change
test_flake_pattern_ratchet.py(its detector-table test now includes athread_timebudget that K8 must count)test_denied_commands_security_cost.py,-n 4The deny-gate test repeat runs (60/60 serial and 10/10 at
-n 8under coverage on a saturated 8-core host; main's 0.5 s budget 0/10 under a 10x clock) were run on this branch before #37539 merged and are recorded in #37539's thread.Manual verification
N/A — test-only change; the ratchet's own detector table is the check.
Related Issues
Part of #37446
Sibling sweep
Swept: the K8 detector over every test file, and
git grep -nE "time\.(thread_time|process_time)\(\)" -- test src/kiro_crew/appsSibling: test/test_denied_commands_security_stdin.py -- not affected -- not changed here, 0.2 s thread-CPU budget, the next to fix
Sibling: test/test_denied_commands_security.py -- not affected -- not changed here, 5 s thread-CPU budgets on flag spam
Sibling: test/test_denied_commands_security_quoting.py -- not affected -- not changed here, 1 s thread-CPU decode budget
Sibling: test/test_jsonl_util.py -- not affected -- not changed here, 5 s process_time backstop
Sibling: test/test_image_refs_replay_strip.py -- not affected -- not changed here, 2 s and 5 s budgets
Sibling: test/test_acp_prompt_blocks.py -- not affected -- not changed here, 5 s thread-CPU scan budget
Sibling: test/test_security_regex_linearity.py -- not affected -- not changed here, 5 s budget
Sibling: test/test_argv_floor_inline_and_brace_scope.py -- not affected -- not changed here, thread-CPU timed helper
Sibling: test/test_options_cap_contract.py -- not affected -- not changed here, a process_time ratio, not an absolute budget
Pattern harvest
Rule candidate: review-prompt
Pattern: an upper bound in seconds on a CPU or wall clock reading used as a complexity guard; a tracer or a slow runner moves it, so count the work and bound its growth between sizes
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)