Skip to content

test(subagent): wait on signals, not clocks -- 4 subagent test files - #19334

Merged
iamwhatever merged 1 commit into
mainfrom
unflaky/wait-migrate-subagent
Oct 12, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
unflaky/wait-migrate-subagent

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The four subagent test files wait on the signal the code emits, never on a clock, so a slow runner can no longer red them.

Between them, test_subagent_continuable.py (64), test_subagent_force_stop_audit.py (38), test_subagent_stop_reason_consistency.py (25) and test_subagent_turn_resilience.py (20) hold 147 short fixed waits: asyncio.wait_for(..., 5|10|15|20) on a task or an asyncio.Event the code finishes on its own, hand-rolled polls of 10-50 ms sleeps, and module constants tuned to one runner (_RESPAWN_TIMEOUT was raised from 5 s to 20 s after a Windows shard red). Each is a bet that CI is as fast as the machine the test was written on. #19248 added the helpers and froze the counts; this is the first burn-down of them.

Why it matters

A lost bet reds a pull request that did not touch these files, and the fix costs a day of someone's time to find the race and size a ceiling. These four files are the largest block of such bets in one area of the tree.

Not a goal

What changed (motivation → approach → change)

Every wait now rides on the thing it was waiting for, under the helpers' lost-run bound (half the test's timeout; expiry fails by name, quoting the signal and the last state read).

What the test waited for Was Now
a run's task, a gather, a task the test created (stop, reap, recovery, respawn) wait_for(task, 5..20) await_until(task, what=...) -- the task's own exception propagates as itself
an asyncio.Event the code sets (entered, popped, started, reset_started, ...) wait_for(event.wait(), 2..20) await_until(event, what=...)
a coroutine the code finishes (close_all, cancel, cancel_all, get_or_create, _force_reap) wait_for(coro, 5..15) await_until(asyncio.create_task(coro), what=...)
a state with no event behind it (a task registered after a queue drain, an abandoned writer gone, a threading.Event read from the loop, the parent's completion list) poll with sleep(0.01..0.05) async_wait_until(predicate, describe=...)
the respawn appearing in _tasks (4 tests) a 20 s deadline loop of 50 ms sleeps _respawned_task(), one async_wait_until that returns the task and quotes info.error on expiry
the posted store writes landing before an on-loop-call absence assertion (2 pins) 40 x sleep(0.01) the _report_tasks the writes join, awaited until the set is empty
the file's own _wait_until helper (stop_reason_consistency, 25 callers) a 15 s deadline loop of 20 ms sleeps delegates to async_wait_until, keeping its what

Same shape, not counted by the ratchet, converted in the same files so one idiom remains: 8 asyncio.wait({run}, timeout=15) + assert run in done, 12 wait_for(started.wait(), _START_TIMEOUT) at 30 s, 2 to_thread(event.wait, N) and 2 wait_for(..., 60). _START_TIMEOUT and _RESPAWN_TIMEOUT are gone with the header comment that argued their numbers; _GIVE_UP_BOUND and its comment stay.

Two cancelled-run waits in test_subagent_turn_resilience.py (..._survives_a_forward_wall_clock_step, ..._gives_up_at_its_deadline) are asyncio.wait({task1}, timeout=default_timeout()) + assert task1 in done rather than await_until: the test must fill the pool before the recovery's first capacity check, and the helper's waiter on a gather adds a loop turn that let the recovery respawn first (both tests failed that way on the first local run). The bound is the helpers' own lost-run bound, and the comment at each site says so.

Per file: continuable 64 -> 0, force_stop_audit 38 -> 0, stop_reason_consistency 25 -> 0, turn_resilience 20 -> 0; the four baseline lines are deleted. Three sites keep a short wait with a # flake-ok reason, so they are not counted:

  • turn_resilience x2: wait_for(gather(rec), timeout=_GIVE_UP_BOUND) -- an UPPER bound that asserts the patched 0.4 s production give-up took effect (a 60 s bound would pass with the patch silently inert); the file's comment on _GIVE_UP_BOUND records the reasoning.
  • force_stop_audit x1: the 10 ms asyncio.sleep tick in _ticks_during -- the tick IS the loop-liveness instrument; the assertion is a lower bound on ticks served while a kill waited.

Backwards compatibility

Compatible: test-only change; no product code, no public test helper's contract. _wait_until in test_subagent_stop_reason_consistency.py drops its unused timeout parameter (no caller passed one).

Removes nothing: two module constants private to one test file go with the waits that read them.

Tests

The four files themselves: 415 tests, unchanged assertions, run 3x at -n 4 and once at -n0, all green. test/test_wait_discipline_ratchet.py is fully green on this head (39 passed) and reports the four files at 0. test_flake_pattern_ratchet.py against origin/main: green. Black, isort, flake8 on the touched files: green.

Manual verification

N/A -- unit coverage sufficient: the change is to the tests' own waits, and the tests prove them.

Related Issues

Fixes #19333

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 07:36
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f6b997f66485228010a55ee0927e5addddcc6b3f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision f6b997f66485228010a55ee0927e5addddcc6b3f ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f6b997f66485228010a55ee0927e5addddcc6b3f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the two asyncio.wait({task1}, timeout=default_timeout()) sites in turn_resilience do not need the extra loop turn await_until adds anywhere else in these files.

[DESIGN-REVIEWED] f6b997f

Description read

[DESCRIPTION-READ] dc1285efbd57867a347d1be234049757b3aa1abcbae09cf92887da2281f91cee

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f6b997f66485228010a55ee0927e5addddcc6b3f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f6b997f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f6b997f66485228010a55ee0927e5addddcc6b3f: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f6b997f66485228010a55ee0927e5addddcc6b3f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f6b997f

Verdict parsed from the review's SHA-scoped output markers for commit f6b997f66485228010a55ee0927e5addddcc6b3f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f6b997f66485228010a55ee0927e5addddcc6b3f: <one-sentence reason>

Migrate the 147 short fixed waits in test_subagent_continuable.py,
test_subagent_force_stop_audit.py, test_subagent_stop_reason_consistency.py
and test_subagent_turn_resilience.py to the signal each one was waiting for
(kiro_crew.testing.wait: await_until on the task or Event, async_wait_until
on a state), and drop their four lines from test/wait_discipline_baseline.txt.

Three sites keep a short wait with a flake-ok reason: the two
_GIVE_UP_BOUND ceilings that assert a patched 0.4 s give-up took effect, and
the 10 ms tick that is the loop-liveness probe itself.

Part of #19333
@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-subagent branch from eaae94a to f6b997f Compare October 12, 2026 07:55
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 12, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test-only PR removes 147 fixed waits from four subagent test files. Each wait_for(..., N), sleep-poll and asyncio.wait({run}, timeout=15) becomes await_until or async_wait_until on the task, event or state the test actually waits for, under the shared lost-run bound, and the four baseline lines are deleted. That removes the root cause of these flakes, because a slow runner can no longer lose a clock bet. I read the full diff and filtered out the mechanical swaps to find anything that changes test meaning. The 1200x50 ms poll becomes async_wait_until(lambda: seen) followed by seen[0], so the old None fallback is gone and a missing report now fails by name. The negative stop-join check now catches AssertionError from await_until instead of a 1 s TimeoutError, so it still fails, only later. The two _GIVE_UP_BOUND upper-bound waits and the loop-liveness tick keep a short bound with a flake-ok reason, which is right because those bounds are the assertion. No product code changes, the PASS lanes have no findings, and checks are green. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever merged commit 6fba29c into main Oct 12, 2026
116 of 123 checks passed
@iamwhatever
iamwhatever deleted the unflaky/wait-migrate-subagent branch October 12, 2026 17:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: subagent suites bet on short fixed waits (147 sites in 4 files) -- wait on signals

2 participants