Skip to content

test: count the self-restart floor path's work, not only its time - #37531

Closed
bolichen97 wants to merge 1 commit into
mainfrom
fix/37446-count-self-module-scan
Closed

bolichen97 wants to merge 1 commit into
mainfrom
fix/37446-count-self-module-scan

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A quadratic anywhere on the self-restart floor path turns TestSelfModuleIndexIsLinear red on every runner, proven by counting the work instead of trusting a clock; this replaces the min-of-N sampling from #37451.

test_the_scan_is_linear_not_quadratic counts two primitives (_normalize_operand, _is_self_module_flag). Everything else on the path is guarded only by its 0.5 s thread-CPU budget, which reds Backend Tests (3.12, 7) on unrelated PRs with the path intact (#37446). #37451 answered that by taking the minimum of five readings, which is a retry under tests-are-deterministic. Run bare, the budget also misses an inline per-token walk.

Why it matters

The budget is the only guard for most of the path, it misses a real quadratic on an uninstrumented run, and its min-of-five form lets an intermittent slowdown through. A deterministic count of the whole path proves the property on every runner and is what any later decision about the budget needs.

Not a goal

  • Removing or raising the 0.5 s budget. testing-conventions D7 and class 5 say an existing complexity guard keeps its thread-CPU budget at its original constant, and no spec says a deterministic pin replaces it. So the single reading this PR restores can still red a slow shard 7, and this PR does not by itself end the shard-7 reds tracked in Flaky: test_the_scan_is_linear_not_quadratic trips its wall-clock bound on a loaded Linux shard again #37446; that needs a maintainer ruling (see the PR comment).
  • Making the floor path cheaper.
  • Changing product code.

What changed (motivation → approach → change)

The shard-7 reds are real CPU, not noise. #36903 made the timed path about 1.8x heavier (_only_single_quoted_openers and _is_message_flag_command in the payload walk): 2k tokens went from 0.031 s to 0.055 s bare and from 0.17 s to 0.30 s under branch coverage on a developer host. A slower instrumented runner then read 0.51 to 0.92 s. Eight of the nine failing heads quoted in #37446 contain #36903. #37530 has since taken coverage's tracer out of the reading.

A new test, test_the_floor_path_executes_linear_work, counts the work instead, in the unit class 5 asks for. A per-thread line tracer counts every source line the security package executes for one _is_self_restart call at 1,000 and 2,000 tokens. It requires at most 2.5x per doubling and at most 2,000 lines a token (measured: 2.00x, 672). The count is the same on every runner and under every coverage core.

That count replaces the min-of-five sampling from #37451: the budget is read once again, at its original constant and input, still with the trace function removed as #37530 does. Its docstring now says what it still covers alone: work hidden inside one C call.

Backwards compatibility

Compatible: test-only; no product code changes.

Removes nothing: the min-of-five sampling goes back to a single reading (still untraced), which is the stricter reading, and every other assertion keeps its constant and input.

Tests

test/test_denied_commands_security_cost.py::TestSelfModuleIndexIsLinear::test_the_floor_path_executes_linear_work locks in linear work across the whole floor path.

Mutation check (throwaway worktree, --no-cov):

Mutant New test Existing counts 0.5 s budget, bare
pre-fix re-normalizing walk red, 3.89x, 24,706 lines a token red red (2.80 s)
inline forward walk over scan.stops red, 3.19x, 2,676 green green
_argv_programs once per token red, 3.83x, 16,737 green red (1.07 s)
a slice per token (C work) green, 2.00x green green

Line counts are identical under the ctrace, sysmon and pytrace coverage cores. Python 3.14 reads 687 lines a token, 2.00x.

Forced condition (a throwaway -p plugin burning C-level CPU on every _program_basename call, a slower core): the new test stays green on this head. The budget reds on the parent, min of five untraced included (0.517 s), and on this head (0.514 s), because a slow core slows every reading. Under plain coverage the untraced single reading is 0.055 s on a developer host. That red is the reason for the Not a goal line above.

Determinism (D12): the new test passed 20 of 20 at -n0; the whole file passed 10 of 10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test_security_publish_floor.py and test_denied_commands_security_ssh_self.py, and 112 of 112 in a shuffled order (seed 8906).

Proof: not provable -- test-only change

Manual verification

N/A: unit coverage sufficient; the change is one test file.

Related Issues

Part of #37446

Replaces the min-of-N sampling from #37451.

Pattern harvest

Rule candidate: review-prompt
Pattern: a complexity guard that counts some primitives and leaves the rest of the path to a wall or CPU budget; count executed lines over the whole path instead.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Needs a decision (maintainer): with the min-of-five sampling from #37451 replaced, the 0.5 s budget is read once again, so on its own this PR does not stop the shard-7 reds in #37446. This PR's own Backend Tests (3.12, 7) can red on that reading too.

needs-a-decision: the single 0.5 s reading in test_the_scan_is_linear_not_quadratic is kept at its original constant because testing-conventions D7 and class 5 say an existing complexity guard keeps it, and no spec says a deterministic pin replaces it.
The red is real CPU: #36903 made the timed path about 1.8x heavier (0.031 s to 0.055 s bare, 0.17 s to 0.30 s under branch coverage on a developer host), so a slower instrumented runner reads 0.51 to 0.92 s.
Under a forced slow core the budget reds on the parent, min of five included, and on this head; the new line count stays green.
Among the mutants tried, the budget catches nothing the new count misses: it misses the inline walk bare and the per-token slice everywhere. Only C-level work hidden in one call, such as a backtracking regex, is left to a clock.

Options:

  1. Rule here that the whole-path line count supersedes the 0.5 s budget for this test. The reading is then dropped in this PR, with a matching line in testing-conventions class 5, and the body switches to Closes #37446.
  2. Keep the single reading and bring the floor path's cost back under it (a product change, separate PR).
  3. Keep both as they are and accept a rerun when shard 7 reds.

Until one is chosen this PR is a test-only strengthening, Part of #37446.

@bolichen97
bolichen97 enabled auto-merge (squash) November 15, 2026 06:17
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 2e7368793316632c1972a8b729590c8f6d29b34f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2e73687

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2e7368793316632c1972a8b729590c8f6d29b34f false-positive: <reason, 20 to 500 characters>

@bolichen97

Copy link
Copy Markdown
Contributor Author

Body note: one wording fix in the Not a goal bullet, made before any review ran. Its sentence read "does not by itself close #37446", which GitHub parses as a closing keyword and would have closed #37446 on merge. It now says "does not by itself end the shard-7 reds tracked in #37446". The goal and scope are unchanged.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 2e7368793316632c1972a8b729590c8f6d29b34f ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 2e7368793316632c1972a8b729590c8f6d29b34f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 2e7368793316632c1972a8b729590c8f6d29b34f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change is test-only and replaces timing with a deterministic count of executed lines across the whole floor path.

Notes

[DESIGN-REVIEWED] 2e73687

Description read

[DESCRIPTION-READ] 5f964af6f2c831cfc6850925da75d88ab736661e2646c8d64f49e01db6b35042

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 2e7368793316632c1972a8b729590c8f6d29b34f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2e73687

Verdict parsed from the review's SHA-scoped output markers for commit 2e7368793316632c1972a8b729590c8f6d29b34f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2e7368793316632c1972a8b729590c8f6d29b34f false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only change. It adds a deterministic count of executed lines across the security package for the self-restart floor path: at most 2.5x per doubling and at most 2,000 lines a token, as testing-conventions D7 and class 5 ask for in a new complexity guard. It also turns the 0.5 s thread-CPU budget from a minimum of five readings back into one untraced reading at its original constant, consistent with AUTOSDE's ruling that a retry is never the fix. In the code: the tracer filters by the security package directory (security is a package), a warm-up call keeps first-call imports out of the count, the trace function is restored in a finally block, the verdict is asserted so the count is real, and the path keeps no memo between calls. No regressions or contract violations found.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 2e73687933 · 108 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
test_the_scan_is_linear_not_quadratic proves the module-flag scan
linear by counting two primitives, and leaves everything else on the
floor path to a 0.5 s thread-CPU budget. That budget does not see an
inline per-token walk at all, and as a clock it also reads the runner.

Add test_the_floor_path_executes_linear_work: a per-thread line tracer
counts every source line the security package executes for one
_is_self_restart call at 1,000 and 2,000 tokens, and requires at most
2.5x per doubling and at most 2,000 lines a token. The count is the
same on every runner and under each coverage core.

The count replaces the minimum-of-five sampling of that budget. The
budget is read once again, still with the trace function removed, at
its original constant, as testing-conventions D7 and class 5 require
of an existing complexity guard: the cheapest of several readings is a
retry.
@bolichen97
bolichen97 force-pushed the fix/37446-count-self-module-scan branch from bda6cfc to 2e73687 Compare November 15, 2026 07:32
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto #37530 (merged after this PR opened), which reads the 0.5 s budget with this thread's trace function removed. This head keeps that and replaces the min-of-five sampling with one reading, so the budget is now a single untraced reading at its original constant, next to the new line count.

New evidence for the decision comment above: under plain branch coverage the untraced single reading costs 0.055 s on a developer host, about 9x under the bound, so the coverage multiplier behind most shard-7 reds is gone from it. What is left of the question is only whether the line count lets the clock be retired for this test; nothing here waits on it. A forced slow core still reds the budget on the parent (0.517 s) and on this head (0.514 s) while the count stays green, and the mutation results in the body hold on this head.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 15, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (6) is red on three test/metrics/test_kiro_export_destination.py::TestOffOffer tests. That is the main red tracked in #37546 (#37460 and #37463 crossed), with the fix in #37545; this PR changes only test/test_denied_commands_security_cost.py. Not rerunning: main code fails the same way. This branch rebases once #37545 lands.

@buluoray

Copy link
Copy Markdown
Contributor

Review passed on this head. Please rebase onto the latest main: the only red tests on this head are main's test/metrics/test_kiro_export_destination.py::TestOffOffer failures (#37546), which #37545 fixed after this run started. The approval will follow once the rebased head is green.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Superseded by #37539, which merged on main: test_the_scan_is_linear_not_quadratic now counts the self-restart scan's executed instructions at three sizes and holds them to a linear progression, which is what this PR added, and #37446 is closed. Closing this one rather than rebasing a second copy of the same check.

auto-merge was automatically disabled November 15, 2026 18:18

Pull request was closed

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants