Repository navigation
test: assert the work or the wait, not seconds, in twenty timing bounds - #37675
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this changes tests only, each one checks the same claim it did before, and any part of it can be reverted cleanly. Notes
[DESIGN-REVIEWED] f13b011 Description read[DESCRIPTION-READ] ebd95d9765e9c418b5ac63e058316aeb64cac255cc52bc7559e96ed6149c5e0b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] f13b011 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
6e19340 to
e1cd746
Compare
Nineteen tests held their claim to a one-second wall clock and one to five seconds of it. Each now asserts what the bound stood for: instructions that grow linearly (frame suffixing, the read-only classifier), a regex's parse (no two run-consuming repeats, one fixed-width repeat body), a budget the call is granted, or that the wait was never taken (a lock that refuses a blocking acquire, a sleep or condition wait that raises, a timer that must not fire, a frozen clock). The regex readers move into test/regex_structure_helpers.py, and testing-conventions.md names them.
e1cd746 to
f13b011
Compare
Code change approver recommendation: A person should decideWhy: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 741 lines, more than the 400 it may approve on its own. Test-only change that swaps 20 wall-clock bounds for checks on the wait or the work. I verified that each spy patches the binding the source actually reads: cron store uses platform_compat module attributes, the subprocess pool uses executor.time and _free.get, wait_until sleeps through _sleep, and the history facade passes the _time write on to history_locks (so the Opus lane's note is wrong). The stalled-anchor grant matches paths_resolver's formula under the fixture's 0.2 s budgets. close_all with an immediately-acking provider arms no sleep timer. The regex parse helper fails closed on the planted shapes. CI is green across Linux and Windows backend shards and the flake ratchet. No src change, no regression path found. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Reviewed on head f13b011. The diff changes only tests: 18 test files, one new test helper and a short note in testing-conventions.md. Nothing under src/ or website/ changes. The review checked each new spy against the binding the code under test actually reads. It also ran all 18 changed test files plus the flake-pattern ratchet and facade-mirror guards on this head (1073 passed), probed the new regex structure helper against regressed patterns, and spot-checked the history-lock planted regression. No assertion was loosened, and no new wall-clock bound or sleep was added.
Done well: each test now asserts the signal its old bound stood for, such as the exact deadline granted, an armed sleep timer, or the regex parse. The body's planted-regression table shows several regressions that slipped past the old one-second bounds and now fail by name.
Optional:
- test/test_session_drain.py:372 records only asyncio.sleep timers. A planted
asyncio.wait(..., timeout=2.0)in close_all passes the new test but failed the old one. Recording every timer that fires during close_all, except the 3600 s drain wait, would close that gap. The docstring still says "small drain_timeout". - test/regex_structure_helpers.py:87 reports
x?\s*ENDand(?<=\s)\s*ENDas linear, but both are quadratic. No current caller is affected. Skipping leading empty-width items, or reporting them as unproven, would keep the helper safe for future guards. - test/test_history_locking_remediation.py:166 patches
_LoopTimeon the history facade. Consider patchinglocks_mod._timedirectly, so the refusal reaches history_locks' own binding in every case.
Problem / Motivation
Goal: Twenty tests that hold their claim to a 1 s (one of them 5 s) wall clock stop reading the host's speed, and each still fails when the regression it guards comes back.
After #37600 and #37630 these are the next-tightest duration bounds the K8 ratchet lists: nineteen at
< 1.0s andtest_many_keys_collapsing_to_one_spelling_cost_linear_workat< 5.0s, which reads about 10 s on main when every clock runs 6x fast. A clock cannot tell a slow or coverage-instrumented shard from slow code, so these bounds red PRs that never touched the code, while a short wait or a small regression slips under them.Why it matters
A red that follows the runner blocks unrelated PRs and teaches people to rerun. Several of the planted regressions below passed the old one-second bounds.
Not a goal
src/.What changed (motivation → approach → change)
Each bound is replaced by what it stood for, chosen by the claim the test makes: a complexity claim counts work or reads the regex's parse, a "does not wait" claim proves the wait was never taken.
_linear_workon interpreter instructions at three sizesrun_cost_unproven,unbounded_repeat_bodies)^assert_rejected_without_backtracking, the original 1.0 s constantsel()) is refused on the test's threadlast_usedEvery spy records and refuses on the test's own thread only and passes other threads through, patches the defining module's binding, and is scoped with
monkeypatch.context(). The regex parse readers live in the newtest/regex_structure_helpers.py;testing-conventions.mdnames them.False positives skipped: none in this set.
test/test_bench_retrieval_stats.py:762(two deltas, not a duration) was listed in #37630 and stays.Backwards compatibility
Compatible: test files, one test helper and docs only; no product behaviour changes.
Removes nothing: every test keeps its name and its claim; only how the claim is checked changes.
Tests
Proof: not provable -- test-only change
Planted regressions in a scratch copy of
src/(run with-o pythonpath=<copy>), one per test. Each new test fails by name; the main column is main's original test on the same plant.#2[2629233, 10295433, 23001633]CONTEXT\s*\s*\]^\s*>\s?MULTILINE${scan via\$\{[^=]*=budget=12.0[16.0]_B64_REbody{4}|{2}_hand_oversleeps after submitlist_jobstakes the store lockopen_lock_filewait_untilsleeps 0.5 s before its first check[0.5]max(2.0, timeout)[2.0]close_allsleeps 0.5 sclose_all slept: [0.5]release()stops stampinglast_usedlast_usedsel()on the caller's threadForced-condition repeat runs on an 8-core DevSpace, the twenty converted tests,
-n0, main (25860b22c6) against this branch. "Busy" is 8 busy loops per tree; both trees ran at the same time. Branch rows are on641fbe1999, except the two marked rows, re-run on the head after the fix below.Main's reds:
test_a_stalled_root_anchor_refuses_within_the_budgetin every forced run, plustest_long_command_is_classified_in_linear_time,test_blank_line_runs_do_not_backtrackandtest_many_keys_collapsing_to_one_spelling_cost_linear_workunder load. On641fbe1999, clock 6x + busy was 7/10: the stalled-anchor test's warm-up was a real resolution under the fixture's 0.2 s budget, so a slow host turned the setup itself into a stall. The head gives that warm-up the lost-run bound; the stall under test keeps the 0.2 s budget.The eighteen whole files at
-n 8under coverage + busy + clock 6x are red on both trees (0/3 each), on tests this PR does not change (for exampletest_shutdown_is_not_reported_drained_while_a_batch_is_still_writing,test_a_stalled_resolution_is_refused_within_the_budget,test_communicate_timeout_reports_the_child_instead_of_raising_bare): those are the looser K8 hits left for later. Of the twenty converted tests, main redstest_many_keys_collapsing_to_one_spelling_cost_linear_work,test_long_command_is_classified_in_linear_timeandtest_a_stalled_root_anchor_refuses_within_the_budgetthere (3/3 each);641fbe1999reds only the stalled-anchor test, once, the warm-up fixed on the head (whole-file runs were not repeated on the head).local-gatebackend related set and the 18 changed files (984 passed), isort, flake8, black gate,test_flake_pattern_ratchet.py(68 passed) and the repository semgrep rules: clean.Manual verification
N/A — test-only change; the planted regressions and forced-condition runs above are the check.
Related Issues
Part of #37599
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)