Skip to content

test(chat): wait on signals, not clocks -- test_chat_agent_selection.py - #19325

Merged
iamwhatever merged 1 commit into
mainfrom
unflaky/wait-migrate-chat-agent-selection
Oct 12, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
unflaky/wait-migrate-chat-agent-selection

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test/test_chat_agent_selection.py waits on the signals its code emits, never on a guessed number of seconds.

The file held 137 short fixed waits -- asyncio.wait_for(..., 10), (..., 11), (..., 15), (..., 20) -- the most of any file in test/wait_discipline_baseline.txt. Each one bets that a CI runner finishes a chat turn, a request or a worker-thread write inside a number tuned on a fast machine; a slow shard loses the bet and reds an unrelated PR.

Why it matters

A lost clock bet in this file reds a pull request that never touched chat agent selection, and the failure names a timeout, not a cause. Zero fixed waits here means the file can only fail for a reason it can name.

Not a goal

What changed (motivation → approach → change)

Site count: test/test_chat_agent_selection.py 137 → 0. Its line in test/wait_discipline_baseline.txt is gone (--write-baseline lowered it; no other line moved).

Each kind of wait, and the signal it now rides on:

Was Count Now rides on
wait_for(<coroutine the test calls>, 10/20) -- _run_chat, drain_background_tasks, _template_chat, client.post, _eager_spawn, api_chat_slot_agent, rehydrate_slot_from_history_async, release_prewarmed_session, to_thread(fn) 92 _finished(coro, what): the coroutine becomes a task the test owns, awaited with await_until under the lost-run bound (half the test timeout); the task is cancelled only if that bound expires, as wait_for did
wait_for(asyncio.to_thread(event.wait, 10), 11) on a threading.Event a worker thread sets 21 (+2 bare to_thread(event.wait, 10)) _signalled(event, what): async_wait_until(event.is_set), a plain thread-safe flag read on the loop, failing by name with the event's role
wait_for(asyncio_event.wait(), 10), wait_for(checkpoint.wait(), 10) 6 await_until(event, what=...)
wait_for(asyncio.gather(...), 15) 12 await_until(gather, what="the tasks settling")
wait_for(task / future / shield(task), 10) 6 await_until(task, what=...) -- await_until never cancels the future it waits on, so the shield is no longer needed
in-thread release.wait(15) / .wait(10) gates (not counted by the ratchet, same bet) 9 release.wait(default_timeout()): the thread fails by name on the same lost-run bound the loop side uses

_finished cancels its task on every early exit (the bound expiring, or an enclosing wait cancelling it), so a hung step never outlives the test; the two waits inside _template_chat take half the caller's bound so the step that hung is the one named, as the old 20 s outer / 10 s inner ceilings did.

Why two small local wrappers instead of bare helper calls: await_until takes an asyncio.Event or a future the test already holds, and refuses a coroutine or a threading.Event by design (the caller must own the task; a thread flag is not an asyncio signal). _finished and _signalled hold exactly that ownership and give every site a what the failure message names. Neither sleeps.

Remaining short fixed waits in this file: none.

Backwards compatibility

Compatible: test-only change; no product code or public symbol moves.

Removes nothing: the only removed line is this file's own entry in a test-only ratchet baseline, which the gate reads as a count of zero.

Tests

  • test/test_chat_agent_selection.py: 81 tests, every one keeps its assertions; passed 4× at -n 4 and 1× at -n0 locally.
  • test/test_wait_discipline_ratchet.py --write-baseline removed only this file's line; the ratchet's own tests are green on this branch (rebased past the test: re-baseline the wait-discipline ratchet to current main (#19248 merge drift) #19329 re-baseline).
  • test/test_flake_pattern_ratchet.py passes against the merge-base (no site rose).
  • black, isort, flake8, check_sync_io_in_async.py, check_comment_history.py: green.

Manual verification

N/A — unit coverage sufficient: the change is to how tests wait, and the tests themselves are the evidence.

Related Issues

Part of the wait-discipline burn-down behind #19248.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 07:22
@CrysisDeu
CrysisDeu requested a review from buluoray October 12, 2026 07:22
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision ef1a774dcba2360148db58a64f5d4e062dedfe42 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ef1a774dcba2360148db58a64f5d4e062dedfe42 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] ef1a774

Description read

[DESCRIPTION-READ] f8085f5f6d29d70a22274670df63d21f18d25ec2ab5feb07f16289af0bccbe1c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ef1a774dcba2360148db58a64f5d4e062dedfe42 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ef1a774

Verdict parsed from the review's SHA-scoped output markers for commit ef1a774dcba2360148db58a64f5d4e062dedfe42.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ef1a774dcba2360148db58a64f5d4e062dedfe42: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ef1a774dcba2360148db58a64f5d4e062dedfe42 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ef1a774dcba2360148db58a64f5d4e062dedfe42 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ef1a774

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ef1a774dcba2360148db58a64f5d4e062dedfe42: <one-sentence reason>

@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-chat-agent-selection branch from 0908cb6 to eb94dc3 Compare October 12, 2026 07:40
Every one of the 137 short fixed waits in test/test_chat_agent_selection.py
(asyncio.wait_for with a 10, 11, 15 or 20 s ceiling) now rides on the signal it
was guessing at, through kiro_crew.testing.wait:

- a coroutine the test itself calls (a chat turn, a request, the background
  drain, the template chat setup) runs under await_until via a local _finished
  wrapper that owns the task and cancels it on every early exit, as wait_for
  did; the two waits inside _template_chat take half the caller's bound so the
  step that hung is the one named;
- an asyncio.Event, a task, a future or a gather is awaited with await_until;
- a threading.Event a worker thread sets is read on the loop with
  async_wait_until(event.is_set) via a local _signalled wrapper, instead of a
  to_thread(event.wait, 10) under an 11 s wait_for;
- the in-thread release gates (release.wait(15)) use default_timeout() so the
  thread fails by name on the same lost-run bound the loop side uses.

The file's line in test/wait_discipline_baseline.txt (137) is removed: its
count is zero. No assertion changed; every test keeps its intent.
@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-migrate-chat-agent-selection branch from eb94dc3 to ef1a774 Compare October 12, 2026 07:58
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 12, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test-only PR replaces all 137 fixed waits in test/test_chat_agent_selection.py with signal waits and drops that file's baseline line. Coroutines the test calls run through _finished (an owned task, cancelled if the bound expires, as wait_for did). Worker-thread events go through _signalled (async_wait_until(event.is_set)). Tasks and gathers go through await_until, and in-thread release.wait(15) gates become default_timeout(). I read the whole diff and filtered out the mechanical swaps. No assertion changed, the shield() removals are safe because await_until does not cancel what it waits on, and the nested _template_chat setup gives its inner waits half the bound so the step that hung is the one the failure names. No product code changes, the lane is PASS, and checks are green. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 12, 2026 17:06
@iamwhatever
iamwhatever merged commit db42120 into main Oct 12, 2026
135 of 163 checks passed
@iamwhatever
iamwhatever deleted the unflaky/wait-migrate-chat-agent-selection branch October 12, 2026 17:07
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants