Skip to content

test: refuse CPU-clock budgets in the flake ratchet and review rules - #37576

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/flaky-scan-linear-not-quadratic
Nov 15, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/flaky-scan-linear-not-quadratic

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The deny-gate cost tests stop failing on slow or instrumented CI shards, and each one still fails when the regression it guards comes back.

test_the_scan_is_linear_not_quadratic holds a 2,000-token scan to 0.5 s of thread CPU. The scan costs about 0.07 s bare. Coverage's tracer multiplies that about five times, so on the instrumented Linux shards the unchanged scan read 0.51 to 0.92 s. In the last 14 days it turned 159 pull requests red, 146 of them green on a rerun, and 13 open pull requests were still held by it. #37451 took the smallest of five readings; #37530 pauses sys.settrace around them. Both still assert seconds.

Why it matters

Each false red blocks an unrelated pull request and turns Coverage Gate red with it. A security test that is red for no reason teaches people to stop reading it.

Not a goal

  • Any change under src/ other than the writing-tests skill text.
  • Fixing the other test files that assert a time ceiling. They are listed under Sibling sweep.
  • The call-count checks in the same test. They stay as they are.

What changed (motivation → approach → change)

The test change itself landed in #37539 (count instructions, not CPU seconds). This PR is what keeps the shape from coming back; rebased onto main after #37539 merged, it carries only that.

The budget escaped review twice because nothing recognised a CPU clock as a clock: the K8 detector in the flake ratchet knew only wall clocks, and two fixes in a row reshaped the reading (#37451 minimum of five, #37530 tracer paused) instead of removing it.

  • test/test_flake_pattern_ratchet.py: the K8 duration detector counts thread_time and process_time deltas like any other clock, with a planted case. Counts are compared per file against the merge-base, so no untouched file turns red.
  • AUTOSDE.yaml tests-are-deterministic item 5: a CPU clock is no exception, and reshaping a kept budget (minimum of N, tracer paused, bigger constant) is flagged.
  • docs/system-specs/common/testing-conventions.md (D7) and src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md: reshaping a kept budget is not a fix; count the work and bound its growth (_interpreter_work / _linear_work).

Backwards compatibility

Compatible: a test ratchet, a review rule and docs only; no product behaviour changes.

Removes nothing: the ratchet only stops a file's count of the shape from rising.

Tests

Proof: not provable -- test-only change

Run Result
test_flake_pattern_ratchet.py (its detector-table test now includes a thread_time budget that K8 must count) 68 passed
with test_denied_commands_security_cost.py, -n 4 179 passed
the repository's semgrep rules on the diff no finding

The deny-gate test repeat runs (60/60 serial and 10/10 at -n 8 under coverage on a saturated 8-core host; main's 0.5 s budget 0/10 under a 10x clock) were run on this branch before #37539 merged and are recorded in #37539's thread.

Manual verification

N/A — test-only change; the ratchet's own detector table is the check.

Related Issues

Part of #37446

Sibling sweep

Swept: the K8 detector over every test file, and git grep -nE "time\.(thread_time|process_time)\(\)" -- test src/kiro_crew/apps
Sibling: test/test_denied_commands_security_stdin.py -- not affected -- not changed here, 0.2 s thread-CPU budget, the next to fix
Sibling: test/test_denied_commands_security.py -- not affected -- not changed here, 5 s thread-CPU budgets on flag spam
Sibling: test/test_denied_commands_security_quoting.py -- not affected -- not changed here, 1 s thread-CPU decode budget
Sibling: test/test_jsonl_util.py -- not affected -- not changed here, 5 s process_time backstop
Sibling: test/test_image_refs_replay_strip.py -- not affected -- not changed here, 2 s and 5 s budgets
Sibling: test/test_acp_prompt_blocks.py -- not affected -- not changed here, 5 s thread-CPU scan budget
Sibling: test/test_security_regex_linearity.py -- not affected -- not changed here, 5 s budget
Sibling: test/test_argv_floor_inline_and_brace_scope.py -- not affected -- not changed here, thread-CPU timed helper
Sibling: test/test_options_cap_contract.py -- not affected -- not changed here, a process_time ratio, not an absolute budget

Pattern harvest

Rule candidate: review-prompt
Pattern: an upper bound in seconds on a CPU or wall clock reading used as a complexity guard; a tracer or a slow runner moves it, so count the work and bound its growth between sizes

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 09:24
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f3824f8f5135af2b7a858ffacb065c55cef02e0e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f3824f8f5135af2b7a858ffacb065c55cef02e0e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The change only widens a ratchet detector and adds review and doc text. The ratchet compares each file against its merge-base, so a file nobody touched cannot turn red.

[DESIGN-REVIEWED] f3824f8

Description read

[DESCRIPTION-READ] 962732b298a7d22c85cb87997e9df0cf501842bd4ac6fbedb59d4523c3b4d972

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of f3824f8f5135af2b7a858ffacb065c55cef02e0e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Each item stops a CPU-clock budget from coming back, and the ratchet has exactly one consumer (_CLOCK_CALLS, used only by K8 at test/test_flake_pattern_ratchet.py:376).

What this change ships

Inventory (4 items) — 4 justified

Intent: Stop the deny-gate cost tests failing for no reason on CI shards that run under coverage, and keep the seconds-budget shape out of future tests. This is a FIX (it guards against regression).

  1. The flake ratchet's K8 duration detector now counts thread_time/process_time deltas (and their _ns forms) as clock readings, with a planted case. — justified
  2. The tests-are-deterministic review rule now flags CPU-clock budgets, and flags reshaping a kept budget (minimum of N, tracer paused, bigger constant). — justified
  3. testing-conventions D7 now says reshaping a kept budget's reading is not a fix. — justified
  4. The writing-tests skill gains a paragraph: count the work and bound its growth instead of asserting seconds. — justified

[FIRST-PRINCIPLES-REVIEWED] f3824f8

Description read

[DESCRIPTION-READ] 34bc6e5dcbd32a5af66911c7904eae19d4b3c5a88458a585bf14a745fd394a87

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f3824f8f5135af2b7a858ffacb065c55cef02e0e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f3824f8

Verdict parsed from the review's SHA-scoped output markers for commit f3824f8f5135af2b7a858ffacb065c55cef02e0e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f3824f8f5135af2b7a858ffacb065c55cef02e0e false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f3824f8f5135af2b7a858ffacb065c55cef02e0e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f3824f8

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f3824f8f5135af2b7a858ffacb065c55cef02e0e false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

The PR adds thread_time, process_time and their _ns forms to the clock calls the K8 duration detector counts. It adds a planted case to the detector table and aligns the AUTOSDE rule, D7 in testing-conventions and the writing-tests skill text. change_risen scores both the merge-base and head copies of each changed file with the current detectors, so files the PR doesn't touch can't turn red. _CLOCK_CALLS has one consumer (K8). The new planted case would fail without the change. The doc and rule text agree with each other, and the helpers they cite exist. No trust boundary is touched. The Flake Pattern Ratchet and Source Gates checks passed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit f3824f8f51 · 49 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/flaky-scan-linear-not-quadratic branch from 2b41721 to 9c59ced Compare November 15, 2026 09:54
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 15, 2026
@buluoray

Copy link
Copy Markdown
Contributor

A note on how this PR relates to two other open PRs that touch test/test_denied_commands_security_cost.py:

Which way to go is your call; this is only so the overlap is visible in one place.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 15, 2026
The deny-gate scan budget was read on thread CPU, so the flake-pattern
ratchet's duration detector never counted it: it knew only wall clocks.
Coverage's tracer multiplies traced CPU about five times, so that budget
turned 159 pull requests red in 14 days, and two fixes in a row reshaped
its reading (minimum of five, tracer paused) instead of removing it.

The K8 detector now counts thread_time and process_time deltas like any
other clock; counts are per file against the merge-base, so no untouched
file turns red. AUTOSDE's tests-are-deterministic item 5 says a CPU clock
is no exception and flags reshaping a kept budget. testing-conventions D7
and the writing-tests skill say the same and point at _interpreter_work
as the count to use.
@bolichen97 bolichen97 changed the title test: count work, not CPU time, in the deny-gate cost tests test: refuse CPU-clock budgets in the flake ratchet and review rules Nov 15, 2026
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/flaky-scan-linear-not-quadratic branch from 9c59ced to f3824f8 Compare November 15, 2026 10:45
@buluoray

Copy link
Copy Markdown
Contributor

Correction to my note above: #37539 has merged, and this PR is now one commit on top of it (the ratchet, review-rule and doc parts only), so the #37539 point no longer applies. Whether this PR and #37531 still conflict is part of the re-review of f3824f8.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 15, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on f3824f8. This PR keeps CPU-clock budgets from coming back into the test suite now that #37539 has removed the one in the deny-gate cost test. The flake ratchet's duration check now treats thread_time and process_time (and their _ns forms) as clocks. The ratchet compares each changed file with its own count at the merge-base, so a file that a PR does not touch cannot turn red, and no file in the tree has the newly counted inline shape today. When we removed the four CPU clock names from the detector, the ratchet's planted thread_time case failed, so the new case guards the change. The review rule stays scoped to added or changed test lines, and it still lets an existing complexity guard keep its untouched budget, which matches the testing conventions. One follow-up: the shared ReDoS helper in test/conftest.py reads thread CPU against an absolute budget and takes the minimum of two or three readings, and the testing conventions still tell authors to convert a regex guard to it. The new review text flags "the minimum of N readings" without naming that helper as an exception, so the review bot could flag a conversion that the conventions prescribe. A one-line carve-out in the rule and the skill would remove the conflict. Decisive lines: test/test_flake_pattern_ratchet.py:122-136, test/test_flake_pattern_ratchet.py:372-382, test/test_flake_pattern_ratchet.py:1151-1191, test/test_flake_pattern_ratchet.py:1280-1281, AUTOSDE.yaml:370-371, AUTOSDE.yaml:390, AUTOSDE.yaml:416-431, docs/system-specs/common/testing-conventions.md:97-103

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@KiroCrewCaptain
KiroCrewCaptain merged commit b72efd2 into main Nov 15, 2026
99 checks passed

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on this head: the only code change adds the four CPU clock names to _CLOCK_CALLS, and K8 is its only reader. The planted time.thread_time() case counts 1 at head and 0 with main's set, so reverting the hunk turns the detector table test red. The ratchet counts each file against its own merge-base, so files that already hold a CPU clock budget stay green. AUTOSDE.yaml parses with no rule id changed, and the cited _interpreter_work / _linear_work helpers exist. PR Readiness is green.

Done well: the PR was rebased onto #37539 and now carries only the prevention work, as a small data change with a real mutation pin.

Optional:

  • The new D7 sentence suggests K8 catches a reshaped budget. It does not: K8 counts only an inline clock delta, or a name like elapsed, compared against a numeric literal. So large = min(...) followed by assert large < 0.5 stays uncounted. Consider narrowing the sentence to what the detector counts.
  • AUTOSDE.yaml item 5 and the new SKILL.md paragraph flag "the minimum of N readings", but conftest.assert_rejected_without_backtracking, which the same docs prescribe for regex guards, does exactly that. A carve-out for that helper would avoid a false BLOCK later.
  • "fails there on every rerun" and "about fivefold" do not match the body's numbers (0.07 s x 5 is under 0.5 s, and 146 of 159 went green on a rerun). Something like "can land on either side of the budget on an instrumented shard" would be accurate. A process_time positive case would also pin the second clock family.

@KiroCrewCaptain
KiroCrewCaptain deleted the crew/LinuxFlackyGuard/flaky-scan-linear-not-quadratic branch November 15, 2026 12:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants