Repository navigation
test: count the self-restart floor path's work, not only its time - #37531
bolichen97 wants to merge 1 commit into
Conversation
|
Needs a decision (maintainer): with the min-of-five sampling from #37451 replaced, the 0.5 s budget is read once again, so on its own this PR does not stop the shard-7 reds in #37446. This PR's own
Options:
Until one is chosen this PR is a test-only strengthening, |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Body note: one wording fix in the |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change is test-only and replaces timing with a deterministic count of executed lines across the whole floor path. Notes
[DESIGN-REVIEWED] 2e73687 Description read[DESCRIPTION-READ] 5f964af6f2c831cfc6850925da75d88ab736661e2646c8d64f49e01db6b35042 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Test-only change. It adds a deterministic count of executed lines across the security package for the self-restart floor path: at most 2.5x per doubling and at most 2,000 lines a token, as testing-conventions D7 and class 5 ask for in a new complexity guard. It also turns the 0.5 s thread-CPU budget from a minimum of five readings back into one untraced reading at its original constant, consistent with AUTOSDE's ruling that a retry is never the fix. In the code: the tracer filters by the security package directory (security is a package), a warm-up call keeps first-call imports out of the count, the trace function is restored in a finally block, the verdict is asserted so the count is real, and the path keeps no memo between calls. No regressions or contract violations found. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
test_the_scan_is_linear_not_quadratic proves the module-flag scan linear by counting two primitives, and leaves everything else on the floor path to a 0.5 s thread-CPU budget. That budget does not see an inline per-token walk at all, and as a clock it also reads the runner. Add test_the_floor_path_executes_linear_work: a per-thread line tracer counts every source line the security package executes for one _is_self_restart call at 1,000 and 2,000 tokens, and requires at most 2.5x per doubling and at most 2,000 lines a token. The count is the same on every runner and under each coverage core. The count replaces the minimum-of-five sampling of that budget. The budget is read once again, still with the trace function removed, at its original constant, as testing-conventions D7 and class 5 require of an existing complexity guard: the cheapest of several readings is a retry.
bda6cfc to
2e73687
Compare
|
Rebased onto #37530 (merged after this PR opened), which reads the 0.5 s budget with this thread's trace function removed. This head keeps that and replaces the min-of-five sampling with one reading, so the budget is now a single untraced reading at its original constant, next to the new line count. New evidence for the decision comment above: under plain branch coverage the untraced single reading costs 0.055 s on a developer host, about 9x under the bound, so the coverage multiplier behind most shard-7 reds is gone from it. What is left of the question is only whether the line count lets the clock be retired for this test; nothing here waits on it. A forced slow core still reds the budget on the parent (0.517 s) and on this head (0.514 s) while the count stays green, and the mutation results in the body hold on this head. |
|
|
|
Superseded by #37539, which merged on main: |
Pull request was closed
Problem / Motivation
Goal: A quadratic anywhere on the self-restart floor path turns
TestSelfModuleIndexIsLinearred on every runner, proven by counting the work instead of trusting a clock; this replaces the min-of-N sampling from #37451.test_the_scan_is_linear_not_quadraticcounts two primitives (_normalize_operand,_is_self_module_flag). Everything else on the path is guarded only by its 0.5 s thread-CPU budget, which redsBackend Tests (3.12, 7)on unrelated PRs with the path intact (#37446). #37451 answered that by taking the minimum of five readings, which is a retry undertests-are-deterministic. Run bare, the budget also misses an inline per-token walk.Why it matters
The budget is the only guard for most of the path, it misses a real quadratic on an uninstrumented run, and its min-of-five form lets an intermittent slowdown through. A deterministic count of the whole path proves the property on every runner and is what any later decision about the budget needs.
Not a goal
What changed (motivation → approach → change)
The shard-7 reds are real CPU, not noise. #36903 made the timed path about 1.8x heavier (
_only_single_quoted_openersand_is_message_flag_commandin the payload walk): 2k tokens went from 0.031 s to 0.055 s bare and from 0.17 s to 0.30 s under branch coverage on a developer host. A slower instrumented runner then read 0.51 to 0.92 s. Eight of the nine failing heads quoted in #37446 contain #36903. #37530 has since taken coverage's tracer out of the reading.A new test,
test_the_floor_path_executes_linear_work, counts the work instead, in the unit class 5 asks for. A per-thread line tracer counts every source line the security package executes for one_is_self_restartcall at 1,000 and 2,000 tokens. It requires at most 2.5x per doubling and at most 2,000 lines a token (measured: 2.00x, 672). The count is the same on every runner and under every coverage core.That count replaces the min-of-five sampling from #37451: the budget is read once again, at its original constant and input, still with the trace function removed as #37530 does. Its docstring now says what it still covers alone: work hidden inside one C call.
Backwards compatibility
Compatible: test-only; no product code changes.
Removes nothing: the min-of-five sampling goes back to a single reading (still untraced), which is the stricter reading, and every other assertion keeps its constant and input.
Tests
test/test_denied_commands_security_cost.py::TestSelfModuleIndexIsLinear::test_the_floor_path_executes_linear_worklocks in linear work across the whole floor path.Mutation check (throwaway worktree,
--no-cov):scan.stops_argv_programsonce per tokenLine counts are identical under the
ctrace,sysmonandpytracecoverage cores. Python 3.14 reads 687 lines a token, 2.00x.Forced condition (a throwaway
-pplugin burning C-level CPU on every_program_basenamecall, a slower core): the new test stays green on this head. The budget reds on the parent, min of five untraced included (0.517 s), and on this head (0.514 s), because a slow core slows every reading. Under plain coverage the untraced single reading is 0.055 s on a developer host. That red is the reason for theNot a goalline above.Determinism (D12): the new test passed 20 of 20 at
-n0; the whole file passed 10 of 10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest_security_publish_floor.pyandtest_denied_commands_security_ssh_self.py, and 112 of 112 in a shuffled order (seed 8906).Proof: not provable -- test-only change
Manual verification
N/A: unit coverage sufficient; the change is one test file.
Related Issues
Part of #37446
Replaces the min-of-N sampling from #37451.
Pattern harvest
Rule candidate: review-prompt
Pattern: a complexity guard that counts some primitives and leaves the rest of the path to a wall or CPU budget; count executed lines over the whole path instead.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.