Repository navigation
test(subagent): wait on signals, not clocks -- 4 subagent test files - #19334
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the two [DESIGN-REVIEWED] f6b997f Description read[DESCRIPTION-READ] dc1285efbd57867a347d1be234049757b3aa1abcbae09cf92887da2281f91cee That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Migrate the 147 short fixed waits in test_subagent_continuable.py, test_subagent_force_stop_audit.py, test_subagent_stop_reason_consistency.py and test_subagent_turn_resilience.py to the signal each one was waiting for (kiro_crew.testing.wait: await_until on the task or Event, async_wait_until on a state), and drop their four lines from test/wait_discipline_baseline.txt. Three sites keep a short wait with a flake-ok reason: the two _GIVE_UP_BOUND ceilings that assert a patched 0.4 s give-up took effect, and the 10 ms tick that is the loop-liveness probe itself. Part of #19333
eaae94a to
f6b997f
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This test-only PR removes 147 fixed waits from four subagent test files. Each wait_for(..., N), sleep-poll and asyncio.wait({run}, timeout=15) becomes await_until or async_wait_until on the task, event or state the test actually waits for, under the shared lost-run bound, and the four baseline lines are deleted. That removes the root cause of these flakes, because a slow runner can no longer lose a clock bet. I read the full diff and filtered out the mechanical swaps to find anything that changes test meaning. The 1200x50 ms poll becomes async_wait_until(lambda: seen) followed by seen[0], so the old None fallback is gone and a missing report now fails by name. The negative stop-join check now catches AssertionError from await_until instead of a 1 s TimeoutError, so it still fails, only later. The two _GIVE_UP_BOUND upper-bound waits and the loop-liveness tick keep a short bound with a flake-ok reason, which is right because those bounds are the assertion. No product code changes, the PASS lanes have no findings, and checks are green. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: The four subagent test files wait on the signal the code emits, never on a clock, so a slow runner can no longer red them.
Between them,
test_subagent_continuable.py(64),test_subagent_force_stop_audit.py(38),test_subagent_stop_reason_consistency.py(25) andtest_subagent_turn_resilience.py(20) hold 147 short fixed waits:asyncio.wait_for(..., 5|10|15|20)on a task or anasyncio.Eventthe code finishes on its own, hand-rolled polls of 10-50 ms sleeps, and module constants tuned to one runner (_RESPAWN_TIMEOUTwas raised from 5 s to 20 s after a Windows shard red). Each is a bet that CI is as fast as the machine the test was written on. #19248 added the helpers and froze the counts; this is the first burn-down of them.Why it matters
A lost bet reds a pull request that did not touch these files, and the fix costs a day of someone's time to find the race and size a ceiling. These four files are the largest block of such bets in one area of the tree.
Not a goal
test/wait_discipline_baseline.txt. Sibling batches take them.test/changes.asyncio.sleepwindows intest_subagent_turn_resilience.pythat prove a respawn has NOT fired yet. There the duration is the test, and they are not counted.main(main red: wait-discipline ratchet fails on six files merged after it #19317); test: re-baseline the wait-discipline ratchet to current main (#19248 merge drift) #19329 re-baselined them and is under this head. This PR only deletes its own four lines.What changed (motivation → approach → change)
Every wait now rides on the thing it was waiting for, under the helpers' lost-run bound (half the test's timeout; expiry fails by name, quoting the signal and the last state read).
gather, a task the test created (stop,reap,recovery,respawn)wait_for(task, 5..20)await_until(task, what=...)-- the task's own exception propagates as itselfasyncio.Eventthe code sets (entered,popped,started,reset_started, ...)wait_for(event.wait(), 2..20)await_until(event, what=...)close_all,cancel,cancel_all,get_or_create,_force_reap)wait_for(coro, 5..15)await_until(asyncio.create_task(coro), what=...)threading.Eventread from the loop, the parent's completion list)sleep(0.01..0.05)async_wait_until(predicate, describe=...)_tasks(4 tests)_respawned_task(), oneasync_wait_untilthat returns the task and quotesinfo.erroron expirysleep(0.01)_report_tasksthe writes join, awaited until the set is empty_wait_untilhelper (stop_reason_consistency, 25 callers)async_wait_until, keeping itswhatSame shape, not counted by the ratchet, converted in the same files so one idiom remains: 8
asyncio.wait({run}, timeout=15)+assert run in done, 12wait_for(started.wait(), _START_TIMEOUT)at 30 s, 2to_thread(event.wait, N)and 2wait_for(..., 60)._START_TIMEOUTand_RESPAWN_TIMEOUTare gone with the header comment that argued their numbers;_GIVE_UP_BOUNDand its comment stay.Two cancelled-run waits in
test_subagent_turn_resilience.py(..._survives_a_forward_wall_clock_step,..._gives_up_at_its_deadline) areasyncio.wait({task1}, timeout=default_timeout())+assert task1 in donerather thanawait_until: the test must fill the pool before the recovery's first capacity check, and the helper's waiter on agatheradds a loop turn that let the recovery respawn first (both tests failed that way on the first local run). The bound is the helpers' own lost-run bound, and the comment at each site says so.Per file:
continuable64 -> 0,force_stop_audit38 -> 0,stop_reason_consistency25 -> 0,turn_resilience20 -> 0; the four baseline lines are deleted. Three sites keep a short wait with a# flake-okreason, so they are not counted:turn_resiliencex2:wait_for(gather(rec), timeout=_GIVE_UP_BOUND)-- an UPPER bound that asserts the patched 0.4 s production give-up took effect (a 60 s bound would pass with the patch silently inert); the file's comment on_GIVE_UP_BOUNDrecords the reasoning.force_stop_auditx1: the 10 msasyncio.sleeptick in_ticks_during-- the tick IS the loop-liveness instrument; the assertion is a lower bound on ticks served while a kill waited.Backwards compatibility
Compatible: test-only change; no product code, no public test helper's contract.
_wait_untilintest_subagent_stop_reason_consistency.pydrops its unusedtimeoutparameter (no caller passed one).Removes nothing: two module constants private to one test file go with the waits that read them.
Tests
The four files themselves: 415 tests, unchanged assertions, run 3x at
-n 4and once at-n0, all green.test/test_wait_discipline_ratchet.pyis fully green on this head (39 passed) and reports the four files at 0.test_flake_pattern_ratchet.pyagainstorigin/main: green. Black, isort, flake8 on the touched files: green.Manual verification
N/A -- unit coverage sufficient: the change is to the tests' own waits, and the tests prove them.
Related Issues
Fixes #19333
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)