Repository navigation
test(session): wait on signals, not clocks -- test_session, test_instances, test_lifecycle_hooks - #19527
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Fix the description: it says the startup wait "became a plain [DESIGN-REVIEWED] 04a516c Description read[DESCRIPTION-READ] 9ddb19cf8a82829336e6c8c2e8a37f487da6c7be54e75370990a3b173fa551e3 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
buluoray
left a comment
There was a problem hiding this comment.
Most of this migration is a straight win: event and task waits now fail by name instead of TimeoutError, the hand-rolled poll loops in test_session.py collapse into async_wait_until, and the baseline only drops the three touched files. The ratchet passes (39) and the three files pass (968) on this head. Two spots need changes, because in both the old short bound was the assertion itself:
-
TestGatewayShutdownBackendSweep::test_the_sweep_shares_one_deadline_instead_of_stacking_waitsand::test_budget_overrun_does_not_cancel_queued_stops(test_lifecycle_hooks.py~1242 and ~1336). The wedged stop parks for 5 s, so the oldwait_for(..., timeout=2.0)is what failed a sweep that ignored_BACKEND_STOP_BUDGET_SECS. Withawait_until's default (~60 s) that regression now passes. Checked by removing thewait_for(..., timeout=_BACKEND_STOP_BUDGET_SECS)inon_gateway_shutdown(apps/hooks_integration.py~1021): both tests stay green on this head (5.00 s each) and fail on main. Please keep an explicit bound well below the wedge (for exampletimeout=2.0) or assert elapsed < budget. -
TestStartupRevive::test_instances_startup_schedules_revive_in_background(test_instances.py~6798) now does a bareawait startup_handler(app). The reason for droppingwait_foris right (on 3.11 it wraps the coroutine in a task and gives the loop a turn), but if_instances_startupregresses to awaiting the revive, the test hangs until pytest-timeout instead of failing by name. Checked by making_instances_startupawait_retake_hops_then_revive: the test only fails via--timeout. A bounded check that yields no loop turn:coro = startup_handler(app)thenwith pytest.raises(StopIteration): coro.send(None).
Non-blocking: a few other short bounds that encoded "returns promptly" (test_lifecycle_hooks.py 0.2-0.3 s waits around dispatch_shutdown / _invoke near 368, 409, 424 and 810, and the 3 s cold-start waits in test_session.py near 429 and 481) now go red only after the default bound. They still catch a hang, just slowly.
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
85d7192 to
ce4cd5e
Compare
|
|
ce4cd5e to
f9a34c4
Compare
|
@buluoray thanks — both blocking spots are addressed on 1.
2. startup_coro = startup_handler(app)
try:
with pytest.raises(StopIteration):
startup_coro.send(None)
finally:
startup_coro.close()Driving the coroutine one step yields no loop turn (so the
Non-blocking items (left as-is, out of scope for this batch): the short Full suites green locally (session 408, instances 483, lifecycle 38; 929 together), ratchet green, rebased on current |
|
Reviewed, not approving yet: @buluoray's CHANGES_REQUESTED review is still open. Please re-review (or dismiss it) once you agree it is met; I would approve after that. My read: This test-only PR replaces the 85 short fixed waits in test_session.py, test_instances.py and test_lifecycle_hooks.py with waits on the event, task or predicate the code really signals ( |
buluoray
left a comment
There was a problem hiding this comment.
Verdict: APPROVE (head f9a34c4)
What it fixes
The tests in test_session.py, test_instances.py and test_lifecycle_hooks.py used to wait on short wall-clock timeouts. They now wait on events, tasks and predicates through kiro_crew.testing.wait, so a hang fails with a named reason, and the three files drop out of the wait-discipline baseline. Nothing in the product changes.
What I verified
Both blocking points from my review on 85d7192 are fixed.
- Sweep budget (
test/test_lifecycle_hooks.py:1247and:1345). Both tests keep an explicitasyncio.wait_for(on_gateway_shutdown(), timeout=2.0)against a 0.05 s patched budget, markedflake-okwith a comment saying the bound is the property under test. The wedged stop now blocks onrelease.wait()with no timeout (:1231-1233), so a sweep that waits for the stop can never pass. Mutation: I replaced thewait_for(asyncio.shield(gathered), timeout=_BACKEND_STOP_BUDGET_SECS)insrc/kiro_crew/apps/hooks_integration.py:1021-1024withawait gathered.test_the_sweep_shares_one_deadline_instead_of_stacking_waitsandtest_budget_overrun_does_not_cancel_queued_stopsboth fail withTimeoutErrorat about 2 s each. - Startup returns without awaiting the revive (
test/test_instances.py:6798-6803). The test now drives the handler withstartup_coro.send(None)insidepytest.raises(StopIteration). That gives the event loop no turn, which keeps thearmed == []ordering check below it meaningful. Mutation: I addedawait revive_taskafter thecreate_taskinsrc/kiro_crew/dashboard/server.py:1547-1548.test_instances_startup_schedules_revive_in_backgroundfails at once withDID NOT RAISE <class 'StopIteration'>, not through pytest-timeout.
I ran each mutation alone and confirmed git status was clean between them. At this head the three files pass (929 tests) and the wait-discipline ratchet passes (39). No other reviewer has an open CHANGES_REQUESTED.
Blocking
None.
Non-blocking follow-ups
-
These short waits from my earlier note still go red only at the default bound, not quickly:
- the 0.2-0.3 s waits around
dispatch_shutdown/_invokeintest_lifecycle_hooks.py - the 3 s cold-start waits in
test_session.py
They still catch a hang, just slowly. If any of them is meant to assert "returns promptly", give it the same
flake-okshort bound as the two sweep tests. - the 0.2-0.3 s waits around
Not verified
- The full test suite: I ran only the three touched files and the ratchet.
- CI results at this head: I did not read them.
…ances, test_lifecycle_hooks Migrate every short fixed wait in the three session-area test files to the signal the code under test emits, using the kiro_crew.testing.wait helpers from #19248 (await_until, async_wait_until). Lowers the three wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22). Every test's intent is preserved; no timeout was raised to 30, no assertion weakened, no site moved or skipped. Closes #19526 Part of #19246
f9a34c4 to
04a516c
Compare
Problem / Motivation
Goal: Stop the session-area backend tests from flaking on slow CI by making every short fixed wait wait on the signal the code emits instead of on a clock.
test/test_session.py,test/test_instances.pyandtest/test_lifecycle_hooks.pyheld 85 short fixed waits between them intest/wait_discipline_baseline.txt-- almost allasyncio.wait_for(..., timeout=1|2|3|5)on a task, agather, or anasyncio.Eventthe code under test completes on its own, plus a handful of hand-rolled polls of 5-50 ms sleeps. D1 of the Determinism contract names this the single commonest cause of a flaky backend test in this repo: the wait bets the CI runner is as fast as the machine the test was written on, and loses often enough to red unrelated PRs.Why it matters
Every one of these is a latent red that lands on someone else's unrelated PR. #19248 added the helpers (
await_until,async_wait_until,LoopHop) and froze the per-file counts so the number can only fall; this is the session-area burn-down.Not a goal
What changed (motivation -> approach -> change)
Each site now waits on its real signal via
kiro_crew.testing.wait, whose default ceiling is half the test timeout and only names a hang. Per-file counts:test/test_session.pytest/test_instances.pytest/test_lifecycle_hooks.pySignal each kind of wait now rides on:
asyncio.wait_for(event.wait(), timeout=N)->await await_until(event, what=...)(theEventitself).asyncio.wait_for(task_or_gather, timeout=N)->await await_until(task, what=...)(the task/future's own result; destructuring preserved).asyncio.wait_for(coro(), timeout=N)on a coroutine ->await await_until(asyncio.ensure_future(coro()), what=...).for _ in range(N): if cond: return/break; await asyncio.sleep(S)) ->await async_wait_until(predicate, describe=...); the port-free poll intest_instances.pyusesoff_loop=Truebecause it binds a socket.Three conversions are not purely mechanical:
test_instances.py::test_instances_startup_schedules_revive_in_background: the outer wait became a plainawait startup_handler(app)._instances_startuphas no await points by design -- it schedules the revive as a tracked background task and returns -- so awaiting it inline hands control straight back before the loop can give that background task a turn, which the followingarmed == []ordering assertion depends on. A bounded wrapper (await_until/wait_for) adds that loop turn and runs the off-boot-path hop re-take before the assertion can see it has not run. Explained in a code comment at the site.test_session.pycleanup-loop tests (test_cleanup_loop_continues_after_expire_idle_crash,test_cleanup_loop_logs_expire_idle_exception): the outerwait_for(mgr._cleanup_loop(), timeout=5)becameawait await_until(asyncio.ensure_future(mgr._cleanup_loop()), ...). These tests patchasyncio.wait_forto speed only the loop's own inter-sweep wait (timeout >= 60);await_untilusesasyncio.wait, so it is unaffected by that patch. The stale comment referencing the outerwait_forwas updated.test_session.pyfence-release absence loop:for _ in range(200): await sleep(0.01); assert not waiter.done(); if not reserved: breakbecameawait async_wait_until(lambda: not reserved, ...)followed by the sameassert not waiter.done()-- a wrongly-handed session leaves the waiter task done, which the post-wait assert still catches.Baseline diff: the three lines (
39 test_session.py,24 test_instances.py,22 test_lifecycle_hooks.py) are deleted; nothing else changes. Produced bypython test/test_wait_discipline_ratchet.py --write-baseline.Remaining short-wait sites: none. No
# flake-okmarker was added in any of the three files.Backwards compatibility
Removes nothing: test-only change; no production code, public symbol, or data shape is touched. The two edited test helpers (
_until,_until_the_shutdown_hangs) are private static methods on their own test classes and keep their signatures.Merge-order note
None. This change removes no module-level constant or helper and renames nothing, so no open PR can depend on a name it drops.
Tests
The three files are the tests. All pass locally:
test_session.py408,test_instances.py483,test_lifecycle_hooks.py38. The suite was run 3x at-n0and once at-n4(929 passed each time) to check for flakiness introduced by the migration;test/test_wait_discipline_ratchet.pyandtest/test_flake_pattern_ratchet.pyboth pass, andblack --checkis clean on the two non-baselined files.Manual verification
N/A -- unit coverage sufficient; this is a test-only refactor verified by the suites above.
Related Issues
Closes #19526
Part of #19246
Checklist
test: ...)blackclean on gated files)