Skip to content

test(session): wait on signals, not clocks -- test_session, test_instances, test_lifecycle_hooks - #19527

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
unflaky/wait-migrate-session
Oct 14, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
unflaky/wait-migrate-session

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop the session-area backend tests from flaking on slow CI by making every short fixed wait wait on the signal the code emits instead of on a clock.

test/test_session.py, test/test_instances.py and test/test_lifecycle_hooks.py held 85 short fixed waits between them in test/wait_discipline_baseline.txt -- almost all asyncio.wait_for(..., timeout=1|2|3|5) on a task, a gather, or an asyncio.Event the code under test completes on its own, plus a handful of hand-rolled polls of 5-50 ms sleeps. D1 of the Determinism contract names this the single commonest cause of a flaky backend test in this repo: the wait bets the CI runner is as fast as the machine the test was written on, and loses often enough to red unrelated PRs.

Why it matters

Every one of these is a latent red that lands on someone else's unrelated PR. #19248 added the helpers (await_until, async_wait_until, LoopHop) and froze the per-file counts so the number can only fall; this is the session-area burn-down.

Not a goal

  • No production code change -- test files and the baseline only.
  • No change to what any test asserts; intent is preserved site for site.
  • Not touching sites in any other file's baseline line.

What changed (motivation -> approach -> change)

Each site now waits on its real signal via kiro_crew.testing.wait, whose default ceiling is half the test timeout and only names a hang. Per-file counts:

File Before After
test/test_session.py 39 0
test/test_instances.py 24 0
test/test_lifecycle_hooks.py 22 0

Signal each kind of wait now rides on:

  • asyncio.wait_for(event.wait(), timeout=N) -> await await_until(event, what=...) (the Event itself).
  • asyncio.wait_for(task_or_gather, timeout=N) -> await await_until(task, what=...) (the task/future's own result; destructuring preserved).
  • asyncio.wait_for(coro(), timeout=N) on a coroutine -> await await_until(asyncio.ensure_future(coro()), what=...).
  • Hand-rolled poll loops (for _ in range(N): if cond: return/break; await asyncio.sleep(S)) -> await async_wait_until(predicate, describe=...); the port-free poll in test_instances.py uses off_loop=True because it binds a socket.

Three conversions are not purely mechanical:

  • test_instances.py::test_instances_startup_schedules_revive_in_background: the outer wait became a plain await startup_handler(app). _instances_startup has no await points by design -- it schedules the revive as a tracked background task and returns -- so awaiting it inline hands control straight back before the loop can give that background task a turn, which the following armed == [] ordering assertion depends on. A bounded wrapper (await_until/wait_for) adds that loop turn and runs the off-boot-path hop re-take before the assertion can see it has not run. Explained in a code comment at the site.
  • test_session.py cleanup-loop tests (test_cleanup_loop_continues_after_expire_idle_crash, test_cleanup_loop_logs_expire_idle_exception): the outer wait_for(mgr._cleanup_loop(), timeout=5) became await await_until(asyncio.ensure_future(mgr._cleanup_loop()), ...). These tests patch asyncio.wait_for to speed only the loop's own inter-sweep wait (timeout >= 60); await_until uses asyncio.wait, so it is unaffected by that patch. The stale comment referencing the outer wait_for was updated.
  • test_session.py fence-release absence loop: for _ in range(200): await sleep(0.01); assert not waiter.done(); if not reserved: break became await async_wait_until(lambda: not reserved, ...) followed by the same assert not waiter.done() -- a wrongly-handed session leaves the waiter task done, which the post-wait assert still catches.

Baseline diff: the three lines (39 test_session.py, 24 test_instances.py, 22 test_lifecycle_hooks.py) are deleted; nothing else changes. Produced by python test/test_wait_discipline_ratchet.py --write-baseline.

Remaining short-wait sites: none. No # flake-ok marker was added in any of the three files.

Backwards compatibility

Removes nothing: test-only change; no production code, public symbol, or data shape is touched. The two edited test helpers (_until, _until_the_shutdown_hangs) are private static methods on their own test classes and keep their signatures.

Merge-order note

None. This change removes no module-level constant or helper and renames nothing, so no open PR can depend on a name it drops.

Tests

The three files are the tests. All pass locally: test_session.py 408, test_instances.py 483, test_lifecycle_hooks.py 38. The suite was run 3x at -n0 and once at -n4 (929 passed each time) to check for flakiness introduced by the migration; test/test_wait_discipline_ratchet.py and test/test_flake_pattern_ratchet.py both pass, and black --check is clean on the two non-baselined files.

Manual verification

N/A -- unit coverage sufficient; this is a test-only refactor verified by the suites above.

Related Issues

Closes #19526
Part of #19246

Checklist

  • At most two commits (one here), Conventional Commits title (test: ...)
  • Existing tests pass; no new behavior to add tests for (migration preserves intent)
  • Self-review completed; follows project style (black clean on gated files)
  • Documentation updated (if applicable) -- N/A
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 20:21
@CrysisDeu
CrysisDeu requested a review from buluoray October 12, 2026 20:21
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 04a516c4e7ffe1229f05b5c460abd712252e7a5d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Fix the description: it says the startup wait "became a plain await startup_handler(app)", but the diff drives it with send(None) and expects StopIteration.

[DESIGN-REVIEWED] 04a516c

Description read

[DESCRIPTION-READ] 9ddb19cf8a82829336e6c8c2e8a37f487da6c7be54e75370990a3b173fa551e3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 04a516c4e7ffe1229f05b5c460abd712252e7a5d ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 04a516c4e7ffe1229f05b5c460abd712252e7a5d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 04a516c4e7ffe1229f05b5c460abd712252e7a5d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 04a516c

Verdict parsed from the review's SHA-scoped output markers for commit 04a516c4e7ffe1229f05b5c460abd712252e7a5d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 04a516c4e7ffe1229f05b5c460abd712252e7a5d: <one-sentence reason>

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of this migration is a straight win: event and task waits now fail by name instead of TimeoutError, the hand-rolled poll loops in test_session.py collapse into async_wait_until, and the baseline only drops the three touched files. The ratchet passes (39) and the three files pass (968) on this head. Two spots need changes, because in both the old short bound was the assertion itself:

  1. TestGatewayShutdownBackendSweep::test_the_sweep_shares_one_deadline_instead_of_stacking_waits and ::test_budget_overrun_does_not_cancel_queued_stops (test_lifecycle_hooks.py ~1242 and ~1336). The wedged stop parks for 5 s, so the old wait_for(..., timeout=2.0) is what failed a sweep that ignored _BACKEND_STOP_BUDGET_SECS. With await_until's default (~60 s) that regression now passes. Checked by removing the wait_for(..., timeout=_BACKEND_STOP_BUDGET_SECS) in on_gateway_shutdown (apps/hooks_integration.py ~1021): both tests stay green on this head (5.00 s each) and fail on main. Please keep an explicit bound well below the wedge (for example timeout=2.0) or assert elapsed < budget.

  2. TestStartupRevive::test_instances_startup_schedules_revive_in_background (test_instances.py ~6798) now does a bare await startup_handler(app). The reason for dropping wait_for is right (on 3.11 it wraps the coroutine in a task and gives the loop a turn), but if _instances_startup regresses to awaiting the revive, the test hangs until pytest-timeout instead of failing by name. Checked by making _instances_startup await _retake_hops_then_revive: the test only fails via --timeout. A bounded check that yields no loop turn: coro = startup_handler(app) then with pytest.raises(StopIteration): coro.send(None).

Non-blocking: a few other short bounds that encoded "returns promptly" (test_lifecycle_hooks.py 0.2-0.3 s waits around dispatch_shutdown / _invoke near 368, 409, 424 and 810, and the 3 s cold-start waits in test_session.py near 429 and 481) now go red only after the default bound. They still catch a hang, just slowly.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 04a516c4e7ffe1229f05b5c460abd712252e7a5d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 04a516c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 04a516c4e7ffe1229f05b5c460abd712252e7a5d: <one-sentence reason>

@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-session branch from 85d7192 to ce4cd5e Compare October 12, 2026 21:04
@CrysisDeu

CrysisDeu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=75267e7243dc fixed: the sweep budget is asserted again, by a bound below the wedge

These two sweep tests are about a time BUDGET -- the sweep must return within _BACKEND_STOP_BUDGET_SECS even while a backend stop is wedged -- so the bound IS the property under test (D7's "the wait is the thing under test"). Restored a short asyncio.wait_for(integration.on_gateway_shutdown(), timeout=2.0) carrying # flake-ok: backend-stop budget is the thing under test; the wedged-stop stubs now release.wait() with no self-release, so a budget-ignoring sweep blocks on them. Proof: stripping the budget wait_for in on_gateway_shutdown makes both tests fail by name at ~2s (TimeoutError), and they pass at ~0.05s unmutated. The ratchet still counts the file at 0 (flake-ok excused). Fixed in f9a34c4.

@CrysisDeu

CrysisDeu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=e8a71d947c7a fixed: the revive handler now fails by name instead of hanging

Replaced await startup_handler(app) with driving the coroutine one step and no loop turn: with pytest.raises(StopIteration): startup_coro.send(None), then startup_coro.close(). A completed handler raises StopIteration so the background revive task still gets no loop turn (the armed == [] ordering holds); a regression that awaits the sequencer inline yields a value here and fails at once instead of hanging until pytest-timeout kills the xdist worker (D8). Proof: making _instances_startup await the revive inline makes the test fail by name at ~0.2s ("DID NOT RAISE StopIteration"), no hang. Fixed in f9a34c4.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 12, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-session branch from ce4cd5e to f9a34c4 Compare October 12, 2026 21:21
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

@buluoray thanks — both blocking spots are addressed on f9a34c49f461f52f23b07608a50a8a583b7f297f, and I verified each with your mutation (test red on the mutation, green unmutated).

1. test_the_sweep_shares_one_deadline_instead_of_stacking_waits / test_budget_overrun_does_not_cancel_queued_stops — these are genuinely about the budget, so the bound is the thing under test. I kept a short bound below the wedge rather than await_until's lost-run ceiling: restored await asyncio.wait_for(integration.on_gateway_shutdown(), timeout=2.0) with # flake-ok: backend-stop budget is the thing under test, and the wedged-stop stubs now release.wait() (no self-release) so a budget-ignoring sweep blocks on them.

  • Mutation (strip asyncio.wait_for(asyncio.shield(gathered), timeout=_BACKEND_STOP_BUDGET_SECS) in apps/hooks_integration.py): both tests fail by name at ~2s (TimeoutError); unmutated they pass at ~0.05s. The wait-discipline ratchet still counts the file at 0 (flake-ok excused).

2. test_instances_startup_schedules_revive_in_background — took your suggested shape exactly:

startup_coro = startup_handler(app)
try:
    with pytest.raises(StopIteration):
        startup_coro.send(None)
finally:
    startup_coro.close()

Driving the coroutine one step yields no loop turn (so the armed == [] ordering still holds), and a regression that awaits the revive inline yields a value instead of raising StopIteration.

  • Mutation (make _instances_startup await _retake_hops_then_revive(...) inline): test fails by name at ~0.2s (DID NOT RAISE StopIteration), no hang.

Non-blocking items (left as-is, out of scope for this batch): the short dispatch_shutdown/stop_detached_startup_hooks waits in test_lifecycle_hooks.py and the 3s cold-start waits in test_session.py assert a return value (== [], == ["b-hook"], the provider identity), which already distinguishes correct from regressed behaviour — unlike the sweep case, the elapsed time there was never the discriminator, so a lost-run ceiling is the right bound and no short-time property is lost.

Full suites green locally (session 408, instances 483, lifecycle 38; 929 together), ratchet green, rebased on current main. Re-requesting your review.

@CrysisDeu
CrysisDeu requested a review from buluoray October 12, 2026 21:22
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 12, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Reviewed, not approving yet: @buluoray's CHANGES_REQUESTED review is still open. Please re-review (or dismiss it) once you agree it is met; I would approve after that. My read: This test-only PR replaces the 85 short fixed waits in test_session.py, test_instances.py and test_lifecycle_hooks.py with waits on the event, task or predicate the code really signals (await_until / async_wait_until), and drops those three baseline lines. That removes the clock bet that flakes slow CI, which is the root cause. I read the whole diff at f9a34c4 and checked both points in buluoray's CHANGES_REQUESTED review on 85d7192 against this head. The two backend-stop budget tests (test_lifecycle_hooks.py ~1239 and ~1337) are covered again: they keep a 2 s wait_for marked flake-ok, and the wedged stubs now block on a bare release.wait(), so a sweep that ignores _BACKEND_STOP_BUDGET_SECS fails by name. The revive-startup test (test_instances.py ~6799) is covered too: it uses the suggested startup_coro.send(None) under pytest.raises(StopIteration), which gives the loop no turn and fails at once if the handler awaits the revive inline. I also checked the rewrites that are not mechanical. The fence-release loop still catches a wrongly handed waiter, because a waiter that is done stays done for the assert after the wait. The cleanup-loop tests are not affected by their patched asyncio.wait_for, since await_until goes through asyncio.wait. No assertion was weakened. Base is main, it is mergeable, and checks and AI lanes are green. Two small notes: buluoray's review still shows CHANGES_REQUESTED and should be re-reviewed or dismissed, and the description still says "plain await startup_handler(app)" where the code now uses the send(None) form. — crew-pr-reviewer (deep re-check)

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE (head f9a34c4)

What it fixes

The tests in test_session.py, test_instances.py and test_lifecycle_hooks.py used to wait on short wall-clock timeouts. They now wait on events, tasks and predicates through kiro_crew.testing.wait, so a hang fails with a named reason, and the three files drop out of the wait-discipline baseline. Nothing in the product changes.

What I verified

Both blocking points from my review on 85d7192 are fixed.

  1. Sweep budget (test/test_lifecycle_hooks.py:1247 and :1345). Both tests keep an explicit asyncio.wait_for(on_gateway_shutdown(), timeout=2.0) against a 0.05 s patched budget, marked flake-ok with a comment saying the bound is the property under test. The wedged stop now blocks on release.wait() with no timeout (:1231-1233), so a sweep that waits for the stop can never pass. Mutation: I replaced the wait_for(asyncio.shield(gathered), timeout=_BACKEND_STOP_BUDGET_SECS) in src/kiro_crew/apps/hooks_integration.py:1021-1024 with await gathered. test_the_sweep_shares_one_deadline_instead_of_stacking_waits and test_budget_overrun_does_not_cancel_queued_stops both fail with TimeoutError at about 2 s each.
  2. Startup returns without awaiting the revive (test/test_instances.py:6798-6803). The test now drives the handler with startup_coro.send(None) inside pytest.raises(StopIteration). That gives the event loop no turn, which keeps the armed == [] ordering check below it meaningful. Mutation: I added await revive_task after the create_task in src/kiro_crew/dashboard/server.py:1547-1548. test_instances_startup_schedules_revive_in_background fails at once with DID NOT RAISE <class 'StopIteration'>, not through pytest-timeout.

I ran each mutation alone and confirmed git status was clean between them. At this head the three files pass (929 tests) and the wait-discipline ratchet passes (39). No other reviewer has an open CHANGES_REQUESTED.

Blocking

None.

Non-blocking follow-ups

  • These short waits from my earlier note still go red only at the default bound, not quickly:

    • the 0.2-0.3 s waits around dispatch_shutdown / _invoke in test_lifecycle_hooks.py
    • the 3 s cold-start waits in test_session.py

    They still catch a hang, just slowly. If any of them is meant to assert "returns promptly", give it the same flake-ok short bound as the two sweep tests.

Not verified

  • The full test suite: I ran only the three touched files and the ratchet.
  • CI results at this head: I did not read them.

…ances, test_lifecycle_hooks

Migrate every short fixed wait in the three session-area test files to the
signal the code under test emits, using the kiro_crew.testing.wait helpers
from #19248 (await_until, async_wait_until). Lowers the three
wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22).

Every test's intent is preserved; no timeout was raised to 30, no assertion
weakened, no site moved or skipped.

Closes #19526
Part of #19246
@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-session branch from f9a34c4 to 04a516c Compare October 14, 2026 00:29
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 14, 2026
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 14, 2026
This was referenced Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: session suites bet on short fixed waits (85 sites in 3 files) -- wait on signals

4 participants