Repository navigation
test(chat): wait on signals, not clocks -- test_chat_agent_selection.py - #19325
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] ef1a774 Description read[DESCRIPTION-READ] f8085f5f6d29d70a22274670df63d21f18d25ec2ab5feb07f16289af0bccbe1c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
0908cb6 to
eb94dc3
Compare
Every one of the 137 short fixed waits in test/test_chat_agent_selection.py (asyncio.wait_for with a 10, 11, 15 or 20 s ceiling) now rides on the signal it was guessing at, through kiro_crew.testing.wait: - a coroutine the test itself calls (a chat turn, a request, the background drain, the template chat setup) runs under await_until via a local _finished wrapper that owns the task and cancels it on every early exit, as wait_for did; the two waits inside _template_chat take half the caller's bound so the step that hung is the one named; - an asyncio.Event, a task, a future or a gather is awaited with await_until; - a threading.Event a worker thread sets is read on the loop with async_wait_until(event.is_set) via a local _signalled wrapper, instead of a to_thread(event.wait, 10) under an 11 s wait_for; - the in-thread release gates (release.wait(15)) use default_timeout() so the thread fails by name on the same lost-run bound the loop side uses. The file's line in test/wait_discipline_baseline.txt (137) is removed: its count is zero. No assertion changed; every test keeps its intent.
eb94dc3 to
ef1a774
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This test-only PR replaces all 137 fixed waits in test/test_chat_agent_selection.py with signal waits and drops that file's baseline line. Coroutines the test calls run through _finished (an owned task, cancelled if the bound expires, as wait_for did). Worker-thread events go through _signalled (async_wait_until(event.is_set)). Tasks and gathers go through await_until, and in-thread release.wait(15) gates become default_timeout(). I read the whole diff and filtered out the mechanical swaps. No assertion changed, the shield() removals are safe because await_until does not cancel what it waits on, and the nested _template_chat setup gives its inner waits half the bound so the step that hung is the one the failure names. No product code changes, the lane is PASS, and checks are green. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal:
test/test_chat_agent_selection.pywaits on the signals its code emits, never on a guessed number of seconds.The file held 137 short fixed waits --
asyncio.wait_for(..., 10),(..., 11),(..., 15),(..., 20)-- the most of any file intest/wait_discipline_baseline.txt. Each one bets that a CI runner finishes a chat turn, a request or a worker-thread write inside a number tuned on a fast machine; a slow shard loses the bet and reds an unrelated PR.Why it matters
A lost clock bet in this file reds a pull request that never touched chat agent selection, and the failure names a timeout, not a cause. Zero fixed waits here means the file can only fail for a reason it can name.
Not a goal
test/wait_discipline_baseline.txt.kiro_crew.testing.wait(the helpers from test: ratchet fixed short waits; add await_until and LoopHop helpers #19248 are used as they are).What changed (motivation → approach → change)
Site count:
test/test_chat_agent_selection.py137 → 0. Its line intest/wait_discipline_baseline.txtis gone (--write-baselinelowered it; no other line moved).Each kind of wait, and the signal it now rides on:
wait_for(<coroutine the test calls>, 10/20)--_run_chat,drain_background_tasks,_template_chat,client.post,_eager_spawn,api_chat_slot_agent,rehydrate_slot_from_history_async,release_prewarmed_session,to_thread(fn)_finished(coro, what): the coroutine becomes a task the test owns, awaited withawait_untilunder the lost-run bound (half the test timeout); the task is cancelled only if that bound expires, aswait_fordidwait_for(asyncio.to_thread(event.wait, 10), 11)on athreading.Eventa worker thread setsto_thread(event.wait, 10))_signalled(event, what):async_wait_until(event.is_set), a plain thread-safe flag read on the loop, failing by name with the event's rolewait_for(asyncio_event.wait(), 10),wait_for(checkpoint.wait(), 10)await_until(event, what=...)wait_for(asyncio.gather(...), 15)await_until(gather, what="the tasks settling")wait_for(task / future / shield(task), 10)await_until(task, what=...)--await_untilnever cancels the future it waits on, so theshieldis no longer neededrelease.wait(15)/.wait(10)gates (not counted by the ratchet, same bet)release.wait(default_timeout()): the thread fails by name on the same lost-run bound the loop side uses_finishedcancels its task on every early exit (the bound expiring, or an enclosing wait cancelling it), so a hung step never outlives the test; the two waits inside_template_chattake half the caller's bound so the step that hung is the one named, as the old 20 s outer / 10 s inner ceilings did.Why two small local wrappers instead of bare helper calls:
await_untiltakes anasyncio.Eventor a future the test already holds, and refuses a coroutine or athreading.Eventby design (the caller must own the task; a thread flag is not an asyncio signal)._finishedand_signalledhold exactly that ownership and give every site awhatthe failure message names. Neither sleeps.Remaining short fixed waits in this file: none.
Backwards compatibility
Compatible: test-only change; no product code or public symbol moves.
Removes nothing: the only removed line is this file's own entry in a test-only ratchet baseline, which the gate reads as a count of zero.
Tests
test/test_chat_agent_selection.py: 81 tests, every one keeps its assertions; passed 4× at-n 4and 1× at-n0locally.test/test_wait_discipline_ratchet.py --write-baselineremoved only this file's line; the ratchet's own tests are green on this branch (rebased past the test: re-baseline the wait-discipline ratchet to current main (#19248 merge drift) #19329 re-baseline).test/test_flake_pattern_ratchet.pypasses against the merge-base (no site rose).check_sync_io_in_async.py,check_comment_history.py: green.Manual verification
N/A — unit coverage sufficient: the change is to how tests wait, and the tests themselves are the evidence.
Related Issues
Part of the wait-discipline burn-down behind #19248.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)