Skip to content

test(voice): wait on signals, not clocks -- terminal_handler, stt_stream, piper_runtime, stt_engine - #19339

Merged
iamwhatever merged 1 commit into
mainfrom
unflaky/wait-migrate-voice
Oct 12, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
unflaky/wait-migrate-voice

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The four voice/terminal test files no longer bet on the runner's speed: every fixed short wait in them now waits on the signal the code emits.

test/test_terminal_handler.py, test/test_stt_stream.py, test/test_piper_runtime.py and test/test_stt_engine.py held 139 short fixed waits between them -- asyncio.wait_for(..., timeout=1..20), .wait() on an Event under a 5 s cap, and polls of 50 to 250 short sleeps. Each one passes only while the CI runner is as fast as the machine the test was written on. #19248 landed the helpers (kiro_crew.testing.wait) and the per-file ratchet; this is the burn-down for batch D (voice / terminal).

Why it matters

A 5 s wait_for that loses on a slow Windows or macOS runner reds an unrelated pull request, and #19093, #19144, #19177 and #19205 were each one of these. test_terminal_handler.py alone was the fourth-largest holder of such waits in test/.

Not a goal

  • Changing any product code or src/kiro_crew/testing/wait.py.
  • Touching any test file outside the four named above, or any other line of test/wait_discipline_baseline.txt (the main-drift red on that gate belongs to test: re-baseline the wait-discipline ratchet to current main (#19248 merge drift) #19329).
  • Removing the fixed sleeps the ratchet does not count (a sleep(0.2) outside a loop before an absence assertion; a product-side _MAX_STREAM_DURATION_SECS = 0.05 that is the thing under test). Those belong to semgrep/test-determinism.yaml and test_flake_pattern_ratchet.py.

What changed (motivation → approach → change)

The ratchet counts a site when the wait's ceiling decides pass versus fail. The fix for each site is to wait on what the test was actually waiting for and let the helper's lost-run bound (half the test's own timeout) only name a hang.

Per file, counted sites before → after:

File Before After
test/test_terminal_handler.py 58 0
test/test_stt_stream.py 42 0
test/test_piper_runtime.py 22 0
test/test_stt_engine.py 17 0

Each kind of wait, and the signal it now rides on:

  • asyncio.wait_for(event.wait(), N) → await await_until(event, what=...): the Event itself (a fake's "entered" / "feeding" / "close started" / "sent" / "reaping" marker, or the test's own release gate).
  • asyncio.wait_for(task_or_future, N) → await await_until(task, what=...): the task or future the test created (gather(..., return_exceptions=True), run_in_executor, asyncio.shield(engine._idle)), with its own result or exception propagating as before; pytest.raises(CancelledError) blocks are unchanged around it.
  • asyncio.wait_for(coroutine(), N) → await await_until(asyncio.ensure_future(coroutine()), what=...): the helper takes an Event or a future by contract (it refuses a bare coroutine), so the coroutine the product exposes -- _kill_session, _replace_terminal_ws, _run_local_session, _relay_download_progress, collect(...), engine.close(), eng.decode(...), ws.receive_json() -- is wrapped in a task first. 33 sites.
  • A poll (for _ in range(N): if cond: break; await sleep(0.0x)) → await async_wait_until(predicate, describe=...) on the state the loop watched: read_future.done, entered.is_set / in_decode.is_set (threading Events read from the loop), not eng.loaded, "env" in captured, sess.ws is takeover_ws and sess.last_ws_disconnect is None, and the end-audit outcomes list. 9 sites.
  • Two sites the ratchet did not count but that were the same bet are converted alongside: the wait_for alias inside expire_after_probe_starts (test_terminal_handler) and the asyncio.wait({task}, timeout=...) joins before an absence assertion (test_receive_until_takes_its_pending_read_down_with_an_outer_cancellation in test_stt_stream; the setup-close-deadline test in test_terminal_handler), now await_until(gather(task, return_exceptions=True)) -- the helper never cancels the future it waits on, which is the property those comments asked for.

Helpers local to test_stt_stream.py: _recv_json / _recv wrap ws.receive_json() / ws.receive() under the bound, _wait_for_operation keeps its name and docstring but waits with async_wait_until on the audit list, and _await_end_audit(outcomes) replaces the seven copies of the for _ in range(int(_AUDIT_WAIT_TIMEOUT_SECS / 0.02)) poll. _AUDIT_WAIT_TIMEOUT_SECS leaves with its last caller; the retired asyncio.wait_for-vs-asyncio.timeout 3.10 comment goes with the wait_for it explained.

One cleanup-path change of behaviour: test_close_completes_with_a_reader_parked_on_a_real_pty used to suppress(TimeoutError) around a 10 s proc.wait() in its finally. It now awaits the reap under the lost-run bound, so a SIGKILLed child that never reaps fails the test by name rather than being swallowed on the way out.

Remaining counted sites in the four files: none. No # flake-ok was added.

Baseline diff: the four lines 58 test_terminal_handler.py, 42 test_stt_stream.py, 22 test_piper_runtime.py, 17 test_stt_engine.py are deleted from test/wait_discipline_baseline.txt (--write-baseline; no other line moves).

Main drift on the ratchet gate: test_wait_discipline_ratchet.py::test_no_file_gains_a_short_fixed_wait was red on bare origin/main (six files merged after the #19248 baseline; #19317). The re-baseline #19329 has merged and is under this head's base (0a470210), so the gate runs fully green here: 39 passed locally, and --write-baseline on this head changes no further line.

Backwards compatibility

Compatible: test-only change; no product code, public function or configuration is touched.

Removes nothing: the only deletions are test-internal (_AUDIT_WAIT_TIMEOUT_SECS, the four baseline lines), each with no reader left in the tree.

Tests

Changed tests (all four files, 139 sites) keep every assertion; what each one locks in is unchanged. Determinism proof on this host (pytest-asyncio 0.20.3 as CI pins it):

  • -n0: 20/20 runs, 708 passed, 31 skipped.
  • -n 4: 10/10 runs, 708 passed, 31 skipped.
  • Shuffled node ids (seeds 735958 and 897335, 739 ids): 708 passed each.
  • python test/test_wait_discipline_ratchet.py reports 0 for the four files; --write-baseline touches only their four lines.
  • test/test_flake_pattern_ratchet.py (merge-base comparison): 53 passed.
  • Static gates: black (baselined), isort, flake8, sync-io-in-async, subprocess-encoding all exit 0.

Not a flaky-test FIX in the D12 sense (no forced-condition red is claimed): these waits had not yet lost on CI; the change removes the bet before they do.

Manual verification

N/A — unit coverage sufficient: the diff is the tests themselves.

Related Issues

Part of #19246

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…eam, piper_runtime, stt_engine

Every fixed short wait in the four voice/terminal test files now rides on the
signal it was waiting for, through kiro_crew.testing.wait: an Event or a task
goes through await_until, a coroutine is wrapped in asyncio.ensure_future and
awaited the same way, and a poll of N short sleeps becomes async_wait_until on
the state it watched. The lost-run bound is the helpers' default (half the
test's timeout), so a hang fails by name instead of racing the runner.

Baseline: 58 + 42 + 22 + 17 = 139 sites -> 0; the four lines leave
test/wait_discipline_baseline.txt. The retired _AUDIT_WAIT_TIMEOUT_SECS
constant in test_stt_stream.py goes with its last caller.

Part of #19246
@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 07:52
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 77d8414aada8b42cd11caf2ceb14c89f822b2517 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 77d8414aada8b42cd11caf2ceb14c89f822b2517 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 77d8414aada8b42cd11caf2ceb14c89f822b2517 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 77d8414

Description read

[DESCRIPTION-READ] 938e2016b051ef71c877e336e07f49c190dafb417d737c7dcd8cceb49aea0a84

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 77d8414aada8b42cd11caf2ceb14c89f822b2517 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 77d8414

Verdict parsed from the review's SHA-scoped output markers for commit 77d8414aada8b42cd11caf2ceb14c89f822b2517.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 77d8414aada8b42cd11caf2ceb14c89f822b2517: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 77d8414aada8b42cd11caf2ceb14c89f822b2517 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 77d8414

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 77d8414aada8b42cd11caf2ceb14c89f822b2517: <one-sentence reason>

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is test-only. It replaces 139 short fixed waits in four voice/terminal test files with await_until / async_wait_until on the signal each test was really waiting for, and drops those four files' lines from the wait-discipline baseline. I read the diff and the helper in kiro_crew/testing/wait.py. await_until returns the future's own result or exception and never cancels it, so the pytest.raises(CancelledError) blocks keep their meaning. Bare coroutines are wrapped in ensure_future as the helper's contract requires. Every absence assertion is still anchored on its own event or task. One test monkeypatches asyncio.wait_for globally (terminal.asyncio is the asyncio module), and the new helper waits with asyncio.wait, which that patch does not touch. That is safer than before. The one behaviour change, awaiting the SIGKILLed child's reap in a finally instead of suppressing a timeout, can only turn a hidden hang into a named failure. The only red check is macOS Tests (on demand) shard 2, and it fails in test_provider_spawn_off_loop_18172.py::test_cancellation_during_the_spawn_still_reaps. This PR does not touch that test, and the failure is already tracked as main/nightly flakes #19310 and #19451. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 12, 2026 17:22
@iamwhatever
iamwhatever merged commit f0fc8db into main Oct 12, 2026
116 of 119 checks passed
@iamwhatever
iamwhatever deleted the unflaky/wait-migrate-voice branch October 12, 2026 17:23
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants