Repository navigation
test: assert the wait or the work, not seconds, in 28 timing bounds - #37715
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Test-only change with no product code touched; each wall-clock bound becomes an assertion on the wait or the work it stood for, and the description backs each one with a planted regression. Notes
[DESIGN-REVIEWED] 8625ef8 Description read[DESCRIPTION-READ] 7c6f89263c7e8eaa4e38ccdc11ada12a6d297f543ea630cdda98608da23cb0d1 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Twenty-eight tests bounded a measured duration between 1.5 and 3 seconds. Under coverage, a busy CPU or a fast clock they go red with no bug. Each now asserts the claim the bound stood for: the wait the call never takes, the timeout it hands a wait, the signals a teardown sends, or interpreter work that grows linearly. Regex guards with no work counter keep the shared ramped thread-CPU helper at the original 2 s.
a575d2d to
8625ef8
Compare
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 1113 lines, more than the 400 it may approve on its own. Test-only PR that replaces 28 wall-clock bounds (1.5-3 s) with assertions on the wait or work they stood for. I checked each spy/patch seam against product code: history/_time and security.paths patches propagate via _facade_mirror; frame-record guard runs on the same loop thread as the runtime reader task; cli_logging joins listener._thread with the caller timeout; SEL critical path skips flush on loop; subprocess pool _note_spawn_errors/_failed_queued match; harness terminate_pgid only uses time.monotonic/sleep; breadcrumb uses os.open/os.read directly; chat_attachments and session_browse import the patched names at module scope. Moved Windows tests have no fixture dependencies. CI fully green including new-test stability lanes. Non-blocking: proof table for repeat/forced-condition runs is still pending in the description; ReDoS pumps are smaller than the original inputs but backed by ratio checks. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (20 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change that replaces wall-clock timing bounds with assertions on the wait or work requested; no product code changes. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/testing-conventions.md.
|
This merged while my review of 8625ef8 was still running, so here is its one finding as a follow-up item. It is a latent flake on main now, not a red yet.
Smallest fix: record only this executor's events, as the sibling test at line 1041 already does: append in |
K8 counted only `assert <span> < <number literal>`, so a measured real-clock span against a name or an expression (`elapsed < budget`), the reversed and age forms (`budget > elapsed`, `stamp > time.monotonic() - 5`), unittest assertLess and assertGreater, and spans held in names like `waited` or `cpu` all landed unseen. The detector, now in test/flake_pattern_k8.py, counts them on real clocks (the stdlib clocks, their aliases, and a running loop's time) and exempts clocks the test drives, remaining-time checks, cutoffs taken before the call and lower bounds. The ratchet stays per file against the merge base, so concurrent merges cannot race it. The real host-speed bounds it finds are converted to assert the wait, the timeout handed to it, or a clock only the code's own waits advance. A kept complexity budget and two bounds that read no host clock carry flake-ok with a reason. test_governance_profiles.py splits a section into a sibling to stay under the line cap. Two follow-ups for tests #37715 and #37906 converted: the graceful terminate_pgid test accepts the EPERM a macOS zombie group answers, and the failed-compaction turn test drives the dispatch loop on a manual clock, so a fast or busy host can no longer let the turn deadline win.
K8 counted only `assert <span> < <number literal>`, so a measured real-clock span against a name or an expression (`elapsed < budget`), the reversed and age forms (`budget > elapsed`, `stamp > time.monotonic() - 5`), unittest assertLess and assertGreater, and spans held in names like `waited` or `cpu` all landed unseen. The detector, now in test/flake_pattern_k8.py, counts them on real clocks (the stdlib clocks, their aliases, and a running loop's time) and exempts clocks the test drives, remaining-time checks, cutoffs taken before the call and lower bounds. The ratchet stays per file against the merge base, so concurrent merges cannot race it. The real host-speed bounds it finds are converted to assert the wait, the timeout handed to it, or a clock only the code's own waits advance. A kept complexity budget and two bounds that read no host clock carry flake-ok with a reason. test_governance_profiles.py splits a section into a sibling to stay under the line cap. Two follow-ups for tests #37715 and #37906 converted: the graceful terminate_pgid test accepts the EPERM a macOS zombie group answers, and the failed-compaction turn test drives the dispatch loop on a manual clock, so a fast or busy host can no longer let the turn deadline win.
K8 counted only `assert <span> < <number literal>`, so a measured real-clock span against a name or an expression (`elapsed < budget`), the reversed and age forms (`budget > elapsed`, `stamp > time.monotonic() - 5`), unittest assertLess and assertGreater, and spans held in names like `waited` or `cpu` all landed unseen. The detector, now in test/flake_pattern_k8.py, counts them on real clocks (the stdlib clocks, their aliases, and a running loop's time) and exempts clocks the test drives, remaining-time checks, cutoffs taken before the call and lower bounds. The ratchet stays per file against the merge base, so concurrent merges cannot race it. The real host-speed bounds it finds are converted to assert the wait, the timeout handed to it, or a clock only the code's own waits advance. A kept complexity budget and two bounds that read no host clock carry flake-ok with a reason. test_governance_profiles.py splits a section into a sibling to stay under the line cap. Two follow-ups for tests #37715 and #37906 converted: the graceful terminate_pgid test accepts the EPERM a macOS zombie group answers, and the failed-compaction turn test drives the dispatch loop on a manual clock, so a fast or busy host can no longer let the turn deadline win.
K8 counted only `assert <span> < <number literal>`, so a measured real-clock span against a name or an expression (`elapsed < budget`), the reversed and age forms (`budget > elapsed`, `stamp > time.monotonic() - 5`), unittest assertLess and assertGreater, and spans held in names like `waited` or `cpu` all landed unseen. The detector, now in test/flake_pattern_k8.py, counts them on real clocks (the stdlib clocks, their aliases, and a running loop's time) and exempts clocks the test drives, remaining-time checks, cutoffs taken before the call and lower bounds. The ratchet stays per file against the merge base, so concurrent merges cannot race it. The real host-speed bounds it finds are converted to assert the wait, the timeout handed to it, or a clock only the code's own waits advance. A kept complexity budget and two bounds that read no host clock carry flake-ok with a reason. test_governance_profiles.py splits a section into a sibling to stay under the line cap. Two follow-ups for tests #37715 and #37906 converted: the graceful terminate_pgid test accepts the EPERM a macOS zombie group answers, and the failed-compaction turn test drives the dispatch loop on a manual clock, so a fast or busy host can no longer let the turn deadline win.
K8 counted only `assert <span> < <number literal>`, so a measured real-clock span against a name or an expression (`elapsed < budget`), the reversed and age forms (`budget > elapsed`, `stamp > time.monotonic() - 5`), unittest assertLess and assertGreater, and spans held in names like `waited` or `cpu` all landed unseen. The detector, now in test/flake_pattern_k8.py, counts them on real clocks (the stdlib clocks, their aliases, and a running loop's time) and exempts clocks the test drives, remaining-time checks, cutoffs taken before the call and lower bounds. The ratchet stays per file against the merge base, so concurrent merges cannot race it. The real host-speed bounds it finds are converted to assert the wait, the timeout handed to it, or a clock only the code's own waits advance. A kept complexity budget and two bounds that read no host clock carry flake-ok with a reason. test_governance_profiles.py splits a section into a sibling to stay under the line cap. Two follow-ups for tests #37715 and #37906 converted: the graceful terminate_pgid test accepts the EPERM a macOS zombie group answers, and the failed-compaction turn test drives the dispatch loop on a manual clock, so a fast or busy host can no longer let the turn deadline win.
K8 counted only `assert <span> < <number literal>`, so a measured real-clock span against a name or an expression (`elapsed < budget`), the reversed and age forms (`budget > elapsed`, `stamp > time.monotonic() - 5`), unittest assertLess and assertGreater, and spans held in names like `waited` or `cpu` all landed unseen. The detector, now in test/flake_pattern_k8.py, counts them on real clocks (the stdlib clocks, their aliases, and a running loop's time) and exempts clocks the test drives, remaining-time checks, cutoffs taken before the call and lower bounds. The ratchet stays per file against the merge base, so concurrent merges cannot race it. The real host-speed bounds it finds are converted to assert the wait, the timeout handed to it, or a clock only the code's own waits advance. A kept complexity budget and two bounds that read no host clock carry flake-ok with a reason. test_governance_profiles.py splits a section into a sibling to stay under the line cap. Two follow-ups for tests #37715 and #37906 converted: the graceful terminate_pgid test accepts the EPERM a macOS zombie group answers, and the failed-compaction turn test drives the dispatch loop on a manual clock, so a fast or busy host can no longer let the turn deadline win.
Problem / Motivation
Goal: Twenty-eight tests that hold their claim to a 1.5-3 s wall clock stop reading the host's speed, and each still fails when the regression it guards comes back.
After #37600, #37630 and #37675 these are the next-tightest duration bounds the K8 ratchet lists: three at
< 1.5s, eighteen at< 2.0/< 2s, three at< 2.5s and four at< 3.0s, in eighteen files. A clock cannot tell a slow or coverage-instrumented shard from slow code, so these bounds red PRs that never touched the code, while a short wait or a small regression slips under them.Why it matters
A red that follows the runner blocks unrelated PRs and teaches people to rerun. Some of the planted regressions below passed the old bounds (for example a spawn verdict held back two reaper ticks).
Not a goal
src/.What changed (motivation → approach → change)
Each bound is replaced by what it stood for, chosen by the claim the test makes:
_linear_workon interpreter instructions at three sizes; for the browse diffs, onegit diff --no-indexcall builds the hunks and the work around it grows at most 2.5x per doublingsaylocale grouptest/conftest.py::assert_rejected_without_backtrackingat its default 2.0 s budget, the constant each bound carried (no work counter sees regex or C cost)os.openwithoutO_NONBLOCK, a pull past the marker, an unbounded reader wait) is refused or recorded on the thread under test(True, 0.1),[0.2], the granted allowance), or a clock that moves only by the poll's own sleepsterminate_pgidgraceful exit, wait hook, default graceFileNotFoundError, notTimeoutError, and the marker is queued on the first note after the failed spawniter_local_refsis never calledA spy that refuses or replaces a blocking primitive is installed for the thread under test only; a spy that only records arguments passes the call through. Globals are patched through
monkeypatchormonkeypatch.context(). Three tests whose names promised speed are renamed to their claim (..._get_their_hunks_from_git×2,..._left_alone_unscanned).test/test_harness.pywould have crossed the 2,000-line cap, so its two self-contained Windows-teardown routing tests move unchanged totest/test_harness_windows_teardown.py.testing-conventions.mdgains the three value-assertion shapes this batch adds.False positives skipped: none in this set.
Backwards compatibility
Compatible: test files and docs only; no product behaviour changes.
Removes nothing: every test keeps its claim; three are renamed to it and two move to a sibling file unchanged.
Tests
Proof: not provable -- test-only change
Planted regressions (in-memory monkeypatches of the product function, or a scratch
src/edit reverted afterwards), one or more per site. Every new assertion fails by name:None/ resolves in-processresult(timeout=1.0)/ timeout + 1.0terminate_pgidgraceful / hook / defaultleader_exited = False/ grace hard-coded 5.0write_frame/ blockinglock.acquire()_on_event_loop→ False / reversed lock orderO_NONBLOCK/S_ISREG→ True_in_open_fenceNone\s*\n\s*regexsaylocale[-_]\w+Forced-condition repeat runs on an 8-core DevSpace, the 28 converted tests (55 ids with parameters),
-n0, main (686d00fc11) against this branch. "Busy" is 8 busy loops; the two trees ran one after the other.Branch rows are on the head,
8625ef8ec0.Main's reds:
test_a_stalled_resolution_is_refused_within_the_budgetin every forced run,test_a_bounded_read_that_misses_its_deadline_retires_the_slot_without_waiting(10/10 with the clock 6x fast, fewer under load), andtest_reordered_repeated_lines_get_their_hunks_fastunder load (6/10 with clock 6x + busy, 10/10 with all three).After the first push, CI found two defects in this PR's own tests, both fixed on the head: the Windows reader-arm wait read
0.3000000000001819against<= 0.3(an absolute deadline minus a coarse clock read; the bound now allows float error), and the history flock poll patchedkiro_crew.history_locks._timeon one owner instead of through thekiro_crew.historyfacade (test_facade_mirror_guards.py).local-gaterelated set: the 18 changed test files and the new sibling (1398 passed, 3 skipped), isort, flake8, black gate, file line cap, comment-history and brand gates, docs lint,test_flake_pattern_ratchet.py(68 passed): clean.Manual verification
N/A — test-only change; the planted regressions and forced-condition runs above are the check.
Related Issues
Follows #37600, #37630 and #37675 (no issue).
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.