Repository navigation
test: count and remove bounds on a real span against any limit - #37946
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this changes only tests and docs, and each converted bound keeps its claim in a form that doesn't depend on host speed. Notes
[DESIGN-REVIEWED] 5636d6b Description read[DESCRIPTION-READ] e3b01805eae2f332940a1d137d153764b9f6a9240671682a46c03f5c152fbb59 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
093d5a9 to
26b2cac
Compare
26b2cac to
4ebd855
Compare
4ea83f0 to
3757817
Compare
K8 counted only `assert <span> < <number literal>`, so a measured real-clock span against a name or an expression (`elapsed < budget`), the reversed and age forms (`budget > elapsed`, `stamp > time.monotonic() - 5`), unittest assertLess and assertGreater, and spans held in names like `waited` or `cpu` all landed unseen. The detector, now in test/flake_pattern_k8.py, counts them on real clocks (the stdlib clocks, their aliases, and a running loop's time) and exempts clocks the test drives, remaining-time checks, cutoffs taken before the call and lower bounds. The ratchet stays per file against the merge base, so concurrent merges cannot race it. The real host-speed bounds it finds are converted to assert the wait, the timeout handed to it, or a clock only the code's own waits advance. A kept complexity budget and two bounds that read no host clock carry flake-ok with a reason. test_governance_profiles.py splits a section into a sibling to stay under the line cap. Two follow-ups for tests #37715 and #37906 converted: the graceful terminate_pgid test accepts the EPERM a macOS zombie group answers, and the failed-compaction turn test drives the dispatch loop on a manual clock, so a fast or busy host can no longer let the turn deadline win.
3757817 to
5636d6b
Compare
Problem / Motivation
Goal: A test can no longer bound a real wall-clock span from above in any spelling without the flake ratchet counting it, and the bounds of that kind still on main stop reading the host's speed.
The K8 ratchet counted only
assert <span> < <number literal>. A real-clock span against a name or an expression (elapsed < budget,time.monotonic() - started < _CEILING), the reversed and age forms (budget > elapsed,stamp > time.monotonic() - 5), unittestassertLess/assertGreater, and a span held in a name such aswaitedorcpuall landed unseen. A scan of main found 30 such sites besides the literal ones #37906 converts; the extended detector found 15 more.Why it matters
These are the same flakes the earlier batches removed (#37600 through #37906): they red PRs on a slow or instrumented shard, and the regressions they guard often slip under them or hang. Without a detector that sees them, the next one lands the same way.
Not a goal
src/.What changed (motivation → approach → change)
Detector.
_k8_duration_boundmoves intotest/flake_pattern_k8.py(the ratchet file stays under the line cap) and counts an upper bound on a span read off a real clock: the stdlib clocks and their aliases (followed through assignments, cycle-guarded, scoped to the function), a running loop'stime(), named spans whose value is opaque (elapsed = run_timed()), non-literal limits, the reversed forms, andassertLess/assertLessEqual/assertGreater/assertGreaterEqual. It exempts clocks the test drives (aManualClock, a fake or virtual clock), remaining-time checks (deadline - time.monotonic() <= cap, which only shrink on a slow host), cutoffs taken before the call (t0 = time.monotonic() - 5.0; stamp > t0), lower bounds andflake-oklines. Each shape has a flagged or clean fixture. The ratchet stays per file against the merge base, counting the base with the same detector, so concurrent merges cannot race it.testing-conventions.md(rule D7) says what K8 counts.Sites. Of the 45, 27 were real host-speed bounds and are converted:
asyncio.timeoutarmed and when it expireswait_forbudgets of the test's own task:[0.3]flock, sleeps on a pinned clock that sum to the ceiling, kernel-blockingflockrefused on the test threadcall_laterdelay armed<= ceiling + 1e-6)last_usedequals the module's frozen clockManualClockcommunicatetimeouts handed, after the child reports it printedwaits == [0.4]on aManualClock, second call[]The other 18 are not host-speed bounds. 17 are left unchanged: test-driven clocks, remaining-time checks, string indices, constants, before/after brackets and stamp-ordering checks; the extended detector does not count them. The image-ref scrub's thread-CPU budget, kept at its original constant (D7), carries
flake-okwith a reason, as do two literal-limit false positives the old detector counted: the benchmark's fixed metric deltas and the turn harness's virtual clock (the third,test_mcp_quarantine.py:757's string offsets, the new detector no longer counts).test_governance_profiles.pywould have crossed the 2,000-line cap, so five unchanged tests move to a sibling file.Over the current tree the detector counts 22 unexcused sites, all of them the literal bounds #37906 converts.
Backwards compatibility
Compatible: test files, test helpers and docs only; no product behaviour changes.
Removes nothing: every test keeps its claim; five tests move unchanged to a sibling file.
Tests
Proof: not provable -- test-only change
Planted regressions (in-memory monkeypatches or in-memory recompiles of the product function;
src/untouched), the same plant on main's original test and this branch's test:wait_forper phaseLOCK_NB/ loop polls / +40 s overshootSlow-host repros (real pauses injected in memory, 1-3 s, before a spawn, a ping write, a poll or the first wait) pass on the branch. On main several of them go red with no bug: the decisions tests after 125 s, the embedding child killed before it prints, the stub bridge at 4.5 s, the subagent write at 1.2 s, the 1.5 s
flockpause.Forced-condition repeat runs on an 8-core DevSpace,
-n0, main (cada1cf842) against this branch. "Busy" is 8 busy loops; the trees ran one after the other. The 28 converted tests (28 ids) ran on4ebd855bbe(the later heads change only the detector andtest_harness.py); the two follow-up tests,test_failed_compaction_then_no_response_ends_the_turnandtest_terminate_pgid_graceful_exit_returns_fast, ran again on the head5636d6becd.Main's reds with the clock 6x fast, 10/10 each: the stub bridge liveness, path-resolution grace, gateway claim, installer tree reap, embedding communicate and SSE cut tests, plus the push guard with the CPU busy. On the follow-ups main reds
test_failed_compaction_then_no_response_ends_the_turn(its 5 s queue poll waits real seconds while the turn deadline reads the scaled clock).Review fixes since the PR opened: the detector follows stamp aliases with a cycle guard (no recursion on
run = run.__wrapped__), countselapsed = stop - startand reversed named spans, scopes clock aliases to their function, does not count a floor or ratio between two spans (elapsed >= min_duration), and counts a real loop'stime()again; the installer recorder watches only the child under test (a Windowstaskkillpasses through); the gracefulterminate_pgidtest accepts macOS's EPERM for an unreaped zombie group only on macOS, with theFalsethat path returns.Changed test files:
-n 4, 1173 passed; the 4 failures areTestBackendLifecycleintest_app_backend.py, which fail the same way on main on this host (no user namespaces). isort, flake8, black gate, file line cap, comment-history and brand gates, docs lint,test_flake_pattern_ratchet.py,test_facade_mirror_guards.pyandsemgrep --config semgrep/(no finding on an added line): clean.Manual verification
N/A — test-only change; the planted regressions, slow-host repros and forced-condition runs above are the check.
Related Issues
Follows #37906 (no issue).
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.