Repository navigation
test(queue): wait on signals, not clocks -- queue_depth, codex_steering, compaction_stop, taskq_admission - #19496
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the reworked [DESIGN-REVIEWED] 0e9699a Description read[DESCRIPTION-READ] f2a4bf3c0f1a5cf64374bce5ce4a5c1c081177c65196182ed81553fe645330eb That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
addc922 to
c14a768
Compare
buluoray
left a comment
There was a problem hiding this comment.
Most of this migration is a clean win: every wait_for(x, N) now goes through await_until / _finished under the lost-run bound, the soft polls became asserting waits, and the baseline only drops the four touched files. The ratchet passes (39) and the four files pass (364) on this head.
One blocking issue in test_a_steer_proven_mid_turn_is_not_reported_consumed_if_the_turn_is_cut (test_codex_steering_request.py ~688-692). The new wait for "text_chunk" in kinds and handle._queue.empty() only becomes true when the turn hits its own prompt(timeout=5.0): with timing added, the condition turns true at 4.98 s with kinds already ['text_chunk', 'complete']. So the reject / cancel the test is about lands after the turn has ended and is never exercised, and the final asserts pass only because a timeout stop is never reported as consumed. The test's runtime also moved from about 2 s to about 5 s. Concretely, removing self._turn_steering_denied from the guard in _release_proven_steers (acp/session_handle.py ~3523) turns the [denied] case red on main but leaves it green on this head (only test_after_a_denied_approval_nothing_in_the_turn_is_reported_consumed still catches it). Please wait on a signal that holds while the turn is still live (for example the steer proven and the chunk taken off the queue, not yet yielded to the consumer), and assert complete is not in kinds before cutting the turn.
Non-blocking:
test_compaction_stop_declined.pystill usesasyncio.sleep(0.05)as a barrier in a few places (around:200,:303,:368,:372,:1502, e.g. "inside _await_cotenants now") and twoasyncio.wait({task}, timeout=5)calls (:2411,:2441). The ratchet does not count those forms, so they are worth migrating (until_parkedfits) or naming as out of scope._finished()is copied into three files; it could live inkiro_crew.testing.wait.
The cancelled checks on this head are superseded duplicate runs; every workflow has a later successful run on the same SHA.
c14a768 to
d5246d3
Compare
|
Thanks for the careful read, and for pinning the exact failure mode. Blocking —
Verification: the Non-blocking 1 — remaining
Non-blocking 2 — hoist
|
|
Reviewed, not approving yet: @buluoray's CHANGES_REQUESTED review is still open. Please re-review (or dismiss it) once you agree it is met; I would approve after that. My read: This test-only PR replaces 140 short fixed waits in four backend test files with waits on the events, futures and state the code emits ( |
d5246d3 to
919869f
Compare
|
NO-REPOST: This test-only PR swaps 140 short fixed waits in four backend test files for waits on the events, futures and state the code under test emits (await_until, async_wait_until, until_parked), and drops those four files' lines from the wait-discipline baseline. The head moved from d5246d3 to 919869f, but that is a rebase onto main. I diffed all four files between the two SHAs. Three are byte-identical. The only change in test_compaction_stop_declined.py is tests main added in the meantime, and those are not in this PR's diff. The steer-cut fix I verified last time is unchanged: the test waits on _steers_proven with the queue empty, then asserts neither complete nor text_chunk is in kinds before cutting. Checks are green and base is main. The blocker is still procedural: buluoray's CHANGES_REQUESTED review from c14a768 is open and has not been re-reviewed or dismissed, though by my read its blocking point is fully met. @buluoray, please re-review or dismiss it, and I would approve after that. — crew-pr-reviewer (deep re-check) |
buluoray
left a comment
There was a problem hiding this comment.
Verdict: APPROVE at 919869f6db
What it fixes
This change moves four queue and steering test files off fixed sleeps and onto signal waits under the lost-run bound, and drops those four files from the wait-discipline baseline. Since my review on c14a768e1a, the PR has been rebased, and the only change to its own commit is the denied/cancelled steer test.
What I verified
- The blocking point from my last review is fixed. In
test/test_codex_steering_request.py:695-699,test_a_steer_proven_mid_turn_is_not_reported_consumed_if_the_turn_is_cutnow waits onhandle._steers_provenbeing non-empty andhandle._queue.empty(). That condition holds while the turn is still live, because codex buffers the chunk until the terminal frame. Line 700 then asserts that neithercompletenortext_chunkhas reached the consumer before the reject or cancel lands. The cut now really happens mid-turn. - The call phase now takes 0.03 s. On the previous head it took about 5 s, because the test was waiting for the prompt's own
timeout=5.0. - Mutation: I removed
or self._turn_steering_deniedfrom_release_proven_steers(src/kiro_crew/acp/session_handle.py:3523). This turnstest_a_steer_proven_mid_turn_is_not_reported_consumed_if_the_turn_is_cut[denied]red (['text_chunk', 'steer_consumed', 'complete']), along withtest_after_a_denied_approval_nothing_in_the_turn_is_reported_consumed. Onc14a768e1a, only the second test went red. - The four touched files pass (370), and
test_wait_discipline_ratchet.pypasses (39).
Non-blocking follow-ups
-
The
[cancelled]case of the same test does not pin theself._cancelledguard on its own. With that guard neutralised (line 3522), onlytest_a_cancelled_turn_reports_no_steer_consumedgoes red. The cancelled case stays green becausestopReason: "cancelled"is already caught by thereason != end_turnallowlist. The behaviour is still covered elsewhere; this note is only about which test pins which guard. -
The items from my last review are still open:
test/test_compaction_stop_declined.pystill usesasyncio.sleep(0.05)as a barrier at:202,:483,:548,:552and:1682, and still hasasyncio.wait({task}, timeout=5)at:2591and:2621._finished()is still copied into three files.
Either migrate these (
until_parkedfits the barrier cases) or name them as out of scope.
Not verified
- I did not re-read CI on this head.
- I did not re-run the full suite. I only ran the four touched files and the ratchet.
iamwhatever
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES at 919869f. @buluoray's approval clears their earlier blocking finding, but the current-head Design concern remains unanswered. The steer-cut test deliberately changes its wait condition and adds an assertion. Please describe that coverage fix and correct the claim that no assertions changed. Also fix or track the two follow-ups in buluoray's review: remaining sleep/wait barriers in test_compaction_stop_declined.py, and the duplicated _finished helper. If deferred, open one issue per item and link both in the PR description. The description's 'Remaining short fixed waits ... none' must distinguish ratchet-counted waits from the waits still present. I verified the review, description, and source evidence; tests were not run locally. — crew-pr-reviewer (r99)
…ng, compaction_stop, taskq_admission Four backend test files bet on fixed short waits for things the code signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate every counted site to a wait on the real signal, using the kiro_crew.testing.wait helpers from #19248. Site counts (test/wait_discipline_baseline.txt): all four lines removed. test_queue_depth_reconcile.py 43 -> 0 test_codex_steering_request.py 42 -> 0 test_compaction_stop_declined.py 35 -> 0 test_taskq_admission_integration.py 20 -> 0 No test's intent changed; no assertion was weakened; no site moved to another file. Part of the wait-discipline burn-down behind #19248.
919869f to
0e9699a
Compare
|
Addressed all four items on head (1) Design concern — the test's wait+assertion change is now named as a deliberate coverage fix. The body's "Not a goal" now carves out this one exception, and a new "Deliberate coverage fix in test_codex_steering_request.py" subsection explains it: the old 2 s poll of (2) "Remaining ... none" corrected. The body now says zero ratchet-counted sites, and adds a "Waits still present after this PR" subsection listing the non-counted waits that remain. (3) buluoray's two non-blocking follow-ups → tracked in #20209.
(4) Re-requesting both reviewers. |
iamwhatever
left a comment
There was a problem hiding this comment.
APPROVE at 0e9699a.
All r99 items are now addressed. The description names the steer-cut change as a deliberate coverage fix, and the test waits on _steers_proven with the queue empty, then asserts the turn is still live before cutting (test_codex_steering_request.py#L710-L714). "Zero" now means zero ratchet-counted waits, and the uncounted ones are listed. Both deferred items, the remaining test_compaction_stop_declined.py holds/barriers and the copied _finished() helper, are tracked with their own sections in the assigned issue #20209, which the description links. Since 919869f the only change is a rebase adaptation to main's STEERING_ANSWER_WAIT_SECS rename. @buluoray's approval supersedes their earlier change request. The Design punchline is answered, and all checks on this head are green. No tests were run locally this round.
— crew-pr-reviewer (r100)
Problem / Motivation
Goal: Four queue/steering backend test files wait on the signals their code emits, never on a guessed number of seconds.
These four files held 140 short fixed waits --
asyncio.wait_for(..., 5/10/20),.result(timeout=...),asyncio.timeout(5), and 5 ms poll loops. Each one bets that a CI runner finishes a store cancel, a dispatch, a compaction, or a steer inside a number tuned on a fast machine; a slow shard loses the bet and reds a pull request that never touched this code.Why it matters
A lost clock bet in one of these files reds an unrelated PR, and the failure names a timeout, not a cause. Zero ratchet-counted fixed waits here means each file is far less likely to fail for a reason it cannot name.
Not a goal
test_a_steer_proven_mid_turn_is_not_reported_consumed_if_the_turn_is_cuthad its wait condition changed and an assertion added, because the straight migration silently broke that test's coverage (see below). This is a coverage fix, not a behavior change to the code under test.test/wait_discipline_baseline.txt.kiro_crew.testing.wait(the test: ratchet fixed short waits; add await_until and LoopHop helpers #19248 helpers are used as they are).What changed (motivation → approach → change)
Every counted site now awaits the event, future or state the code under test emits, under the lost-run bound the helpers default to (half the test's own timeout), instead of a short number.
Per-file ratchet-counted site count in
test/wait_discipline_baseline.txt, all four lines removed:Each kind of wait, and the signal it now rides on:
wait_for(event.wait(), 2/5/10)on anasyncio.Eventawait_until(event, what=...)-- returns once the event the code sets is setwait_for(task/future/gather, 5/10/20)the test holdsawait_until(signal, what=...)-- the test owns the task; the bound never cancels itwait_for(<coroutine the test calls>, N)--mgr.cancel(...),_dispatch_async_impl(...),reads.get(),real_post(...)_finished(coro, what)-- the coroutine becomes a task awaited withawait_untilunder the bound, cancelled on every early exit aswait_fordidfor _ in range(200): if cond: break; sleep(0.01)andasync with asyncio.timeout(5): while not cond: sleep(0)async_wait_until(lambda: cond, describe=...)on the state the poll watchedwhile not session.semaphore._waiters: sleep(0)(wait for a parked claimant)until_parked(session.semaphore)-- reads the primitive's own parked-waiter countasyncio.timeout(5)guarding astartedNewTurn/notice loopasync_wait_untilon the notice listDeliberate coverage fix in
test_codex_steering_request.pyOne site is not a mechanical swap.
test_a_steer_proven_mid_turn_is_not_reported_consumed_if_the_turn_is_cutcuts the turn (reject / cancel) while it is still live, then checks the proven steer is not reported consumed. The original guard was a 2 s bounded poll of"text_chunk" in kinds and handle._queue.empty()that, in practice, never became true (codex buffers the chunk to the terminal, sokindsonly fills at the end) -- the loop just ran out after 2 s and the cut landed mid-turn by luck of timing. A straight migration toasync_wait_until(that condition)only becomes true at the prompt's own 5 s timeout, by which point the turn is alreadycompleteand the reject/cancel path is never exercised: the test would pass without testing anything, and its runtime jumped 2 s → 5 s.The fix waits on the real mid-turn signal --
handle._steers_provennon-empty ANDhandle._queue.empty()(steer proven, chunk taken off the queue, not yet yielded) -- and then asserts"complete" not in kinds and "text_chunk" not in kindsbefore cutting, so the test now fails by name if the cut ever lands after the turn ended. Verified: removing theself._turn_steering_deniedguard from_release_proven_steersinacp/session_handle.pyturns the[denied]case red on this head (it stayed green before the fix); runtime is back to 0.03 s. This is a change to what the test covers, made on purpose.The baseline diff (the only change to
test/wait_discipline_baseline.txt): the four lines above are removed, no other line moves.Waits still present after this PR
"Zero" above means zero ratchet-counted sites (the ratchet counts
wait_for/asyncio.timeout/.result(timeout)under 30 s andsleep(0<N<5)inside a loop). Two shapes the ratchet does not count still live intest_compaction_stop_declined.pyand are left as tracked debt, not silently claimed gone:asyncio.sleep(0.05)permit-holds and failure-path barriers (:202,:483,:548,:552,:1682) -- a parked claimant deliberately holds the turn permit for a span, or a failure path advances into_await_cotenants; neither is a signal barrieruntil_parkedfits.done, _ = await asyncio.wait({task}, timeout=5); assert done(:2591,:2621) -- already a bounded lost-run wait on the task finishing.Also, the
_finished()wrapper is copied into three of the four files rather than living inkiro_crew.testing.wait(changing that module is out of this PR's goal, and three in-flight sibling PRs build on it).Both are tracked in #20209 and are out of this PR's frozen goal (the 140 counted sites). No
# flake-okmarker was added.Backwards compatibility
Compatible: test-only change; no product code or public symbol moves, and every file-local helper keeps its name and signature.
Removes nothing: the only removed lines are these four files' own entries in a test-only ratchet baseline, which the gate reads as a count of zero.
Tests
test/test_wait_discipline_ratchet.py: green on this branch; this PR's four lines are gone and no file rose because of this PR.Manual verification
N/A -- unit coverage sufficient: the change is to how these tests wait, and the tests themselves are the evidence (the coverage fix above is verified by the guard-removal check).
Related Issues
Part of #19248
Follow-up debt: #20209