Repository navigation
test: ratchet fixed short waits; add await_until and LoopHop helpers - #19248
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check the 19 s whole-tree AST scan on the slowest CI runner (Windows): it must stay well under the 120 s per-test timeout. [DESIGN-REVIEWED] f739119 Description read[DESCRIPTION-READ] a4c489f2500e54aceeb391a6139fbab1071e703bbe5a8577a17bf205638f7e75 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS For any wait under 2 s, the new ratchet counts the same Not justified as shipped
What this change shipsInventory (8 items) — 6 justifiedIntent: stop new fixed short waits from entering
[FIRST-PRINCIPLES-REVIEWED] f739119 Description read[DESCRIPTION-READ] 7f40070ab84d2fdb5556974e4da8badd882a00f49b3c9240ea7285ebe5c29099 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
57d5728 to
c882292
Compare
c882292 to
f739119
Compare
|
Disposition of the advisory findings on
|
iamwhatever
left a comment
There was a problem hiding this comment.
This PR adds a per-file ratchet (test/test_wait_discipline_ratchet.py plus a committed baseline) that stops new test files gaining fixed short waits: wait_for, asyncio.timeout and .result under 30 s, and sleeps under 5 s inside loops. It also adds await_until and LoopHop to kiro_crew.testing.wait and converts eleven sites to use them. That attacks the root cause the description names (tests betting on a short clock), which caused 7 of the last 18 flaky-test fixes, and does it at the tree level CI can read on a depth-1 checkout. I read wait.py in full. await_until never cancels the caller's future, cancels only its own Event waiter, and raises a named AssertionError. LoopHop's except TimeoutError is correct because requires-python is >=3.12, where concurrent.futures.TimeoutError is the builtin. I also read the AST detector: it handles import aliases and module constants, exempts pytest.raises(TimeoutError), sleep(0) and flake-ok. A non-literal timeout gets past it, which is acceptable for a ratchet and stated as scope. All checks are green, GPT/Opus/Design pass, and the First Principles K9-overlap item has an answer on the PR. The --move subtraction is advisory and can stay, since it is what answers Design's rename concern. — crew-pr-reviewer (deep re-check)
…ng, compaction_stop, taskq_admission Four backend test files bet on fixed short waits for things the code signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate every counted site to a wait on the real signal, using the kiro_crew.testing.wait helpers from #19248. Site counts (test/wait_discipline_baseline.txt): all four lines removed. test_queue_depth_reconcile.py 43 -> 0 test_codex_steering_request.py 42 -> 0 test_compaction_stop_declined.py 35 -> 0 test_taskq_admission_integration.py 20 -> 0 No test's intent changed; no assertion was weakened; no site moved to another file. Part of the wait-discipline burn-down behind #19248.
…ances, test_lifecycle_hooks Migrate every short fixed wait in the three session-area test files to the signal the code under test emits, using the kiro_crew.testing.wait helpers from #19248 (await_until, async_wait_until). Lowers the three wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22). Every test's intent is preserved; no timeout was raised to 30, no assertion weakened, no site moved or skipped. Closes #19526 Part of #19246
…ng, compaction_stop, taskq_admission Four backend test files bet on fixed short waits for things the code signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate every counted site to a wait on the real signal, using the kiro_crew.testing.wait helpers from #19248. Site counts (test/wait_discipline_baseline.txt): all four lines removed. test_queue_depth_reconcile.py 43 -> 0 test_codex_steering_request.py 42 -> 0 test_compaction_stop_declined.py 35 -> 0 test_taskq_admission_integration.py 20 -> 0 No test's intent changed; no assertion was weakened; no site moved to another file. Part of the wait-discipline burn-down behind #19248.
…ances, test_lifecycle_hooks Migrate every short fixed wait in the three session-area test files to the signal the code under test emits, using the kiro_crew.testing.wait helpers from #19248 (await_until, async_wait_until). Lowers the three wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22). Every test's intent is preserved; no timeout was raised to 30, no assertion weakened, no site moved or skipped. Closes #19526 Part of #19246
…ng, compaction_stop, taskq_admission Four backend test files bet on fixed short waits for things the code signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate every counted site to a wait on the real signal, using the kiro_crew.testing.wait helpers from #19248. Site counts (test/wait_discipline_baseline.txt): all four lines removed. test_queue_depth_reconcile.py 43 -> 0 test_codex_steering_request.py 42 -> 0 test_compaction_stop_declined.py 35 -> 0 test_taskq_admission_integration.py 20 -> 0 No test's intent changed; no assertion was weakened; no site moved to another file. Part of the wait-discipline burn-down behind #19248.
…ances, test_lifecycle_hooks Migrate every short fixed wait in the three session-area test files to the signal the code under test emits, using the kiro_crew.testing.wait helpers from #19248 (await_until, async_wait_until). Lowers the three wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22). Every test's intent is preserved; no timeout was raised to 30, no assertion weakened, no site moved or skipped. Closes #19526 Part of #19246
…ances, test_lifecycle_hooks (#19527) Migrate every short fixed wait in the three session-area test files to the signal the code under test emits, using the kiro_crew.testing.wait helpers from #19248 (await_until, async_wait_until). Lowers the three wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22). Every test's intent is preserved; no timeout was raised to 30, no assertion weakened, no site moved or skipped. Closes #19526 Part of #19246
…ng, compaction_stop, taskq_admission Four backend test files bet on fixed short waits for things the code signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate every counted site to a wait on the real signal, using the kiro_crew.testing.wait helpers from #19248. Site counts (test/wait_discipline_baseline.txt): all four lines removed. test_queue_depth_reconcile.py 43 -> 0 test_codex_steering_request.py 42 -> 0 test_compaction_stop_declined.py 35 -> 0 test_taskq_admission_integration.py 20 -> 0 No test's intent changed; no assertion was weakened; no site moved to another file. Part of the wait-discipline burn-down behind #19248.
…ng, compaction_stop, taskq_admission (#19496) Four backend test files bet on fixed short waits for things the code signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate every counted site to a wait on the real signal, using the kiro_crew.testing.wait helpers from #19248. Site counts (test/wait_discipline_baseline.txt): all four lines removed. test_queue_depth_reconcile.py 43 -> 0 test_codex_steering_request.py 42 -> 0 test_compaction_stop_declined.py 35 -> 0 test_taskq_admission_integration.py 20 -> 0 No test's intent changed; no assertion was weakened; no site moved to another file. Part of the wait-discipline burn-down behind #19248.
Problem / Motivation
Goal: A test that waits a fixed short time for something the code signals can no longer be added to
test/; the ones already there are counted, file by file, and that count only goes down.Of the last 18 flaky backend tests fixed here, 7 had one cause: the test bet on a short clock -- a 5 s
asyncio.wait_for(#19093), a.result(timeout=5)(#19144), a poll of 200 five-millisecond turns (#19177), a 50 ms module constant (#19205) -- and a CI runner was slower than the dev box.test/holds 2,912 more such bets in 528 files. Nothing stops the next PR adding one.Why it matters
Every one of these reds a pull request that did not touch it, and each fix costs a day of someone's time to find the race, size a ceiling and prove it. Stopping the number growing is cheaper than fixing them one at a time after they fire.
Not a goal
test/test_flake_pattern_ratchet.pyorsemgrep/test-determinism.yaml. They judge a diff; this judges the tree. The# flake-ok: <reason>marker is shared so no site needs two.kiro_crew.testing.waitis the test helper module that ships in the wheel; nothing outside tests reads it.src/kiro_crew/apps/builtins/*/tests.What changed (motivation → approach → change)
test/test_wait_discipline_ratchet.pytest/**/*.py. Countsasyncio.wait_for,asyncio.timeoutand.result(timeout=)under 30 s, andtime.sleep/asyncio.sleepunder 5 s inside a loop. A literal, or a module-level constant bound to one.sleep(0), a sleep outside a loop, a site underpytest.raises(TimeoutError)and a# flake-ok: <reason>line do not count.test/wait_discipline_baseline.txt<count> <file>line per file that has any, measured onmainatd6341a8b99: 2,912 sites in 528 files, minus the 11 this PR converts. A count may only go down; a file not listed starts at zero.--write-baselinelowers lines and drops gone files, never raises or adds one;--move OLD NEWcarries a renamed file's line to its new path, count unchanged, and the failure messages say so.await_untilinsrc/kiro_crew/testing/wait.pyasyncio.wait_for(signal, <short number>): anasyncio.Eventor an asyncio future (a task, agather). Default bound is half the test's timeout; expiry raises anAssertionErrornamingwhatand the elapsed time. The test's own future is never cancelled. A state with neither behind it stays withasync_wait_until.LoopHopinsrc/kiro_crew/testing/wait.py_LoopHopfromtest/test_slot_ownership_checkpoint.py, moved here with the same behaviour and the same 60 s bound (half the test timeout); that test imports it.test_subagent_manager_boundaries.pyandtest_workflows_async_persistence.pyawait their Events throughawait_untilinstead of a 2 s / 4 swait_for;test_members_dm_thread.pyhops its winner's publish throughLoopHopinstead of.result(timeout=10). Their baseline lines drop accordingly.testing-conventions.mdWhy a committed per-file baseline when
test_flake_pattern_ratchet.pyalready counts short waits: that gate compares against the merge-base, which CI's depth-1 checkout cannot read, so it is a local pre-push gate, and its "short" is under 2 s. A baseline file is readable on any checkout, and a rise names the exact file and line.Why one floor at 30 s: D1 says a ceiling of 30 s or more only names a hang.
timeout=5is the commonest value in the tree (1,013wait_forsites) and the one that lost on CI, so a floor that let 5 through would make raising 2 to 5 the cheapest way past a red gate; the off-loop hop onto a loop doing durable work (a SQLite commit, an fsync) exceeds 5 s on the Windows runner. The helpers default to 60 s, half the suite's--timeout=120.Why this gate beside K9 in
test_flake_pattern_ratchet.py: K9 counts the same call shapes under 2 s, against a merge-base, and so cannot run on CI's depth-1 checkout. Trimming K9 is that gate's own change, not this one's; the# flake-okmarker is shared so a site never needs two.Backwards compatibility
Compatible: two names are added to
kiro_crew.testing.wait.__all__; nothing it exported changes.test_slot_ownership_checkpoint.pykeeps the same hop behaviour through the shared class.Removes nothing: the
_LoopHopclass and its 60 s constant move from one test file into the shared helper module; the test that used them imports the shared class.Tests
test_wait_discipline_ratchet.py: the gate (test_no_file_gains_a_short_fixed_wait), staleness (test_the_baseline_is_not_stale), canonical form of the baseline, 14 positive detector cases and 15 near misses (floor values, non-literals,pytest.raises(TimeoutError),sleep(0), a sleep in a nested def, a name in a docstring, the helper with a short bound), a planted-file end-to-end on a scratch tree, the baseline reader refusing a malformed line, andlowered_baselinenever raising or adding.test_testing_determinism_helpers.py:TestAwaitUntil(event set / already set / never set; future result / exception / left pending at the bound; agather;TypeErrorfor a callable or a bare coroutine; the default bound is half the test timeout) andTestLoopHop(runs on the loop from a thread, the coroutine's own exception, an expired hop recorded forcheck(), the default ceiling).test_slot_ownership_checkpoint.pyandtest_members_dm_thread.py: unchanged assertions with the sharedLoopHop.test_subagent_manager_boundaries.pyandtest_workflows_async_persistence.py: unchanged assertions throughawait_until.LoopHopexpiry test forces expiry only after the coroutine has entered, through a stand-in future whoseresult()waits for the entry mark and then raisesTimeoutError; a real short bound would race the loop's scheduling. The twoawait_untilnegatives that take real time use a 0.05 s bound on a signal that never fires, so nothing can arrive late.test/test_zz_planted_probe.pywith onewait_for(..., timeout=1)failstest_no_file_gains_a_short_fixed_waitastest/test_zz_planted_probe.py: 0 -> 1/:5 wait_for(1s)plus the helper hint; removing it is green again.Manual verification
python test/test_wait_discipline_ratchet.pyprints the burn-down (on this branch 2,901 in 526 files: 2,230wait_for, 598sleep-in-loop, 67result, 6asyncio.timeout); the scan takes about 19 s on a dev box. The new helper tests pass 10 of 10 repeats at-n0and beside their neighbours at-n 4. Black, isort, flake8, mypy, comment-history, docs-lint and the rest of the prepare-pr static floor pass locally.Related Issues
Fixes #19246
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)