Repository navigation
test(voice): wait on signals, not clocks -- terminal_handler, stt_stream, piper_runtime, stt_engine - #19339
Conversation
…eam, piper_runtime, stt_engine Every fixed short wait in the four voice/terminal test files now rides on the signal it was waiting for, through kiro_crew.testing.wait: an Event or a task goes through await_until, a coroutine is wrapped in asyncio.ensure_future and awaited the same way, and a poll of N short sleeps becomes async_wait_until on the state it watched. The lost-run bound is the helpers' default (half the test's timeout), so a hang fails by name instead of racing the runner. Baseline: 58 + 42 + 22 + 17 = 139 sites -> 0; the four lines leave test/wait_discipline_baseline.txt. The retired _AUDIT_WAIT_TIMEOUT_SECS constant in test_stt_stream.py goes with its last caller. Part of #19246
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 77d8414 Description read[DESCRIPTION-READ] 938e2016b051ef71c877e336e07f49c190dafb417d737c7dcd8cceb49aea0a84 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
iamwhatever
left a comment
There was a problem hiding this comment.
This is test-only. It replaces 139 short fixed waits in four voice/terminal test files with await_until / async_wait_until on the signal each test was really waiting for, and drops those four files' lines from the wait-discipline baseline. I read the diff and the helper in kiro_crew/testing/wait.py. await_until returns the future's own result or exception and never cancels it, so the pytest.raises(CancelledError) blocks keep their meaning. Bare coroutines are wrapped in ensure_future as the helper's contract requires. Every absence assertion is still anchored on its own event or task. One test monkeypatches asyncio.wait_for globally (terminal.asyncio is the asyncio module), and the new helper waits with asyncio.wait, which that patch does not touch. That is safer than before. The one behaviour change, awaiting the SIGKILLed child's reap in a finally instead of suppressing a timeout, can only turn a hidden hang into a named failure. The only red check is macOS Tests (on demand) shard 2, and it fails in test_provider_spawn_off_loop_18172.py::test_cancellation_during_the_spawn_still_reaps. This PR does not touch that test, and the failure is already tracked as main/nightly flakes #19310 and #19451. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: The four voice/terminal test files no longer bet on the runner's speed: every fixed short wait in them now waits on the signal the code emits.
test/test_terminal_handler.py,test/test_stt_stream.py,test/test_piper_runtime.pyandtest/test_stt_engine.pyheld 139 short fixed waits between them --asyncio.wait_for(..., timeout=1..20),.wait()on an Event under a 5 s cap, and polls of 50 to 250 short sleeps. Each one passes only while the CI runner is as fast as the machine the test was written on. #19248 landed the helpers (kiro_crew.testing.wait) and the per-file ratchet; this is the burn-down for batch D (voice / terminal).Why it matters
A 5 s
wait_forthat loses on a slow Windows or macOS runner reds an unrelated pull request, and #19093, #19144, #19177 and #19205 were each one of these.test_terminal_handler.pyalone was the fourth-largest holder of such waits intest/.Not a goal
src/kiro_crew/testing/wait.py.test/wait_discipline_baseline.txt(the main-drift red on that gate belongs to test: re-baseline the wait-discipline ratchet to current main (#19248 merge drift) #19329).sleep(0.2)outside a loop before an absence assertion; a product-side_MAX_STREAM_DURATION_SECS = 0.05that is the thing under test). Those belong tosemgrep/test-determinism.yamlandtest_flake_pattern_ratchet.py.What changed (motivation → approach → change)
The ratchet counts a site when the wait's ceiling decides pass versus fail. The fix for each site is to wait on what the test was actually waiting for and let the helper's lost-run bound (half the test's own timeout) only name a hang.
Per file, counted sites before → after:
test/test_terminal_handler.pytest/test_stt_stream.pytest/test_piper_runtime.pytest/test_stt_engine.pyEach kind of wait, and the signal it now rides on:
asyncio.wait_for(event.wait(), N)→await await_until(event, what=...): the Event itself (a fake's "entered" / "feeding" / "close started" / "sent" / "reaping" marker, or the test's own release gate).asyncio.wait_for(task_or_future, N)→await await_until(task, what=...): the task or future the test created (gather(..., return_exceptions=True),run_in_executor,asyncio.shield(engine._idle)), with its own result or exception propagating as before;pytest.raises(CancelledError)blocks are unchanged around it.asyncio.wait_for(coroutine(), N)→await await_until(asyncio.ensure_future(coroutine()), what=...): the helper takes an Event or a future by contract (it refuses a bare coroutine), so the coroutine the product exposes --_kill_session,_replace_terminal_ws,_run_local_session,_relay_download_progress,collect(...),engine.close(),eng.decode(...),ws.receive_json()-- is wrapped in a task first. 33 sites.for _ in range(N): if cond: break; await sleep(0.0x)) →await async_wait_until(predicate, describe=...)on the state the loop watched:read_future.done,entered.is_set/in_decode.is_set(threading Events read from the loop),not eng.loaded,"env" in captured,sess.ws is takeover_ws and sess.last_ws_disconnect is None, and the end-auditoutcomeslist. 9 sites.wait_foralias insideexpire_after_probe_starts(test_terminal_handler) and theasyncio.wait({task}, timeout=...)joins before an absence assertion (test_receive_until_takes_its_pending_read_down_with_an_outer_cancellationin test_stt_stream; the setup-close-deadline test in test_terminal_handler), nowawait_until(gather(task, return_exceptions=True))-- the helper never cancels the future it waits on, which is the property those comments asked for.Helpers local to
test_stt_stream.py:_recv_json/_recvwrapws.receive_json()/ws.receive()under the bound,_wait_for_operationkeeps its name and docstring but waits withasync_wait_untilon the audit list, and_await_end_audit(outcomes)replaces the seven copies of thefor _ in range(int(_AUDIT_WAIT_TIMEOUT_SECS / 0.02))poll._AUDIT_WAIT_TIMEOUT_SECSleaves with its last caller; the retiredasyncio.wait_for-vs-asyncio.timeout3.10 comment goes with thewait_forit explained.One cleanup-path change of behaviour:
test_close_completes_with_a_reader_parked_on_a_real_ptyused tosuppress(TimeoutError)around a 10 sproc.wait()in itsfinally. It now awaits the reap under the lost-run bound, so a SIGKILLed child that never reaps fails the test by name rather than being swallowed on the way out.Remaining counted sites in the four files: none. No
# flake-okwas added.Baseline diff: the four lines
58 test_terminal_handler.py,42 test_stt_stream.py,22 test_piper_runtime.py,17 test_stt_engine.pyare deleted fromtest/wait_discipline_baseline.txt(--write-baseline; no other line moves).Main drift on the ratchet gate:
test_wait_discipline_ratchet.py::test_no_file_gains_a_short_fixed_waitwas red on bareorigin/main(six files merged after the #19248 baseline; #19317). The re-baseline #19329 has merged and is under this head's base (0a470210), so the gate runs fully green here: 39 passed locally, and--write-baselineon this head changes no further line.Backwards compatibility
Compatible: test-only change; no product code, public function or configuration is touched.
Removes nothing: the only deletions are test-internal (
_AUDIT_WAIT_TIMEOUT_SECS, the four baseline lines), each with no reader left in the tree.Tests
Changed tests (all four files, 139 sites) keep every assertion; what each one locks in is unchanged. Determinism proof on this host (pytest-asyncio 0.20.3 as CI pins it):
-n0: 20/20 runs, 708 passed, 31 skipped.-n 4: 10/10 runs, 708 passed, 31 skipped.python test/test_wait_discipline_ratchet.pyreports 0 for the four files;--write-baselinetouches only their four lines.test/test_flake_pattern_ratchet.py(merge-base comparison): 53 passed.Not a flaky-test FIX in the D12 sense (no forced-condition red is claimed): these waits had not yet lost on CI; the change removes the bet before they do.
Manual verification
N/A — unit coverage sufficient: the diff is the tests themselves.
Related Issues
Part of #19246
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)