Skip to content

test(crewmate-feed): let the drain helper yield to the loop - #31386

Merged
bolichen97 merged 1 commit into
mainfrom
fix/24356-drain-yields
Nov 6, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/24356-drain-yields

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_a_retired_question_refreshes_the_waiting_list_with_no_log_growth stops hanging until the CI timeout on a slow runner.

The test times out now and then in CI: Failed: Timeout >180.0s on the macOS backend shard (#24225's run) and Failed: Timeout >120.0s on a Linux shard (#14783's run). It passes in a few seconds whenever it does not hang. The CI stack dump shows the event loop thread busy inside the test's _drain helper, in asyncio.gather, with the executor thread idle, so the loop was spinning, not waiting.

Why it matters

A hang costs a whole shard (three minutes on macOS) and a rerun on PRs that never touch the crewmate feed, and it trains authors to rerun reds. The earlier _isolated clean-up (#23644) was already in the base of the failing macOS run, so it did not stop this.

Not a goal

  • Any change under src/. Production code never drains crewmate_feed._keep; it only holds strong references there.
  • A timeout, retry, skip or looser assertion in the helper or the test.
  • Drain loops over any set other than crewmate_feed._keep: the 17 other while <set>: await asyncio.gather(*<set>) loops in test/, and the drain_promotions, drain_relaunches and channel_restarts_settled helpers under src/kiro_crew/, which no production path calls (drain_relaunches has no caller; tests call the other two). None of them has a flake report, and they are left for a follow-up.

What changed (motivation → approach → change)

The test's _drain helper loops while crewmate_feed._keep: await asyncio.gather(*list(crewmate_feed._keep)). A task leaves _keep through its done callback (_keep.discard), and the event loop runs that callback on its next pass. On Python 3.12, asyncio.gather over tasks that are all done returns a future that is already done, and awaiting a done future does not yield. So when _drain starts in the same loop pass in which the last task (the goal read) finished, that task is still in _keep, gather returns at once, the discard never gets a chance to run, and the while spins on the loop thread until pytest-timeout fires. It is a timing race, which is why a loaded runner shows it.

The helper now yields once per pass with await asyncio.sleep(0) after each gather, so the queued discard callbacks run before _keep is checked again. test_crewmate_feed_cold_store.py's _settle has the same loop and gets the same line. This is a test-only change.

Backwards compatibility

Compatible: only two test helpers change; no product code, config or API is touched.

Removes nothing: the diff adds one yield per helper loop and one test.

Tests

  • test_drain_returns_when_a_finished_task_still_waits_for_its_discard (new, in test/test_crewmate_feed.py) builds the exact state: a finished task still in _keep with its discard queued. It counts gather calls instead of hanging. With the yield reverted it fails at once, on its 50th gather call, with AssertionError: _drain spun without yielding to the loop; with the yield it passes.

Proof: not provable -- test-only change

Manual verification

On macOS 26.7.2 (arm64, Apple silicon), Python 3.12.13. The host was shared and already loaded (load average 28 to 60 on 18 cores), so no extra CPU hogs were added; the forced condition below is the decisive evidence, not host load.

Run Parent (unchanged helper) This change
Named test, serial -n 0, repeated 19 passed, then run 20 hung: Timeout >60.0s in _drain -> gather, the child a finished _record_current_goal task, executor thread idle 20 of 20 passed (named test plus the new test)
Both files, -n 2 --dist loadgroup --max-worker-restart=2, 20 runs 20 of 20 passed (test_crewmate_feed.py only) 20 of 20 passed, 43 tests each
Forced condition (throwaway plugin below) 3 of 3 hung in _drain 20 of 20 passed
Shuffled order, both files, -n 0, seeds 4242, 17, 30893 not run 43 passed for each seed

The forced condition is a throwaway -p plugin, not committed. It swaps the module's async_wait_until for one that polls on every loop pass and returns in the first pass in which every _keep task is done, so the discard of the last one is still queued. Every forced run printed _keep = ['done'] at that point.

The parent's hang on this Mac (excerpt of the pytest-timeout dump: cut lines marked ..., caret lines dropped, interpreter paths shortened)
...
test/test_crewmate_feed.py:900: in go
    await _drain()
test/test_crewmate_feed.py:89: in _drain
    await asyncio.gather(*list(crewmate_feed._keep))
<python>/lib/python3.12/asyncio/tasks.py:854: in gather
    outer = _GatheringFuture(children, loop=loop)
...
children = [<Task finished name='Task-3' coro=<_record_current_goal() done, defined at src/kiro_crew/dashboard/crewmate_feed.py:422> result=None>]
...
E   Failed: Timeout >60.0s
...
~~~~~~~~~~~~~~~~~~~~~~~ Stack of asyncio_0 ~~~~~~~~~~~~~~~~~~~~~~~
...
  File "<python>/lib/python3.12/concurrent/futures/thread.py", line 90, in _worker
    work_item = work_queue.get(block=True)
Forced-condition plugin and command
import asyncio

import pytest


@pytest.fixture(autouse=True)
def _force_drain_race(request, monkeypatch):
    if request.module.__name__.rsplit(".", 1)[-1] != "test_crewmate_feed":
        yield
        return
    from kiro_crew.dashboard import crewmate_feed

    async def forced(predicate, **_kw):
        while not predicate():
            await asyncio.sleep(0)
        while any(not t.done() for t in crewmate_feed._keep):
            await asyncio.sleep(0)
        print("FORCED: returning with _keep =",
              [("done" if t.done() else "pending") for t in crewmate_feed._keep])
        return predicate()

    monkeypatch.setattr(request.module, "async_wait_until", forced)
    yield
PYTHONPATH=<plugin dir> python -m pytest -p force_drain_race -n 0 -q -s --timeout 20 \
  "test/test_crewmate_feed.py::test_a_retired_question_refreshes_the_waiting_list_with_no_log_growth"

Related Issues

Closes #24356

Related: #23674 (the same test timing out on Linux), #23644 (the earlier _isolated clean-up)

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

The crewmate feed tests drain crewmate_feed._keep with
`while _keep: await asyncio.gather(*_keep)`. A task leaves _keep through
its done callback, which the loop runs on its next pass. On Python 3.12
a gather over tasks that are all done completes eagerly and awaiting it
does not yield, so when the helper starts in the same pass in which the
last task finished, the discard never runs and the loop spins on the
event loop thread until the test times out.

Yield once per pass in test_crewmate_feed.py's _drain and in the same
loop in test_crewmate_feed_cold_store.py's _settle, and pin the helper
with a test that counts gather calls instead of hanging. Test-only;
nothing under src/ changes.
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 6, 2026 11:36
@bolichen97 bolichen97 added the ci:macos Run the on-demand macOS suite against this PR head label Nov 6, 2026
@bolichen97
bolichen97 requested a review from Zedmor November 6, 2026 11:36
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 208ff12f5d1bf172d38e0f6586bc81c2d44535c2 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 208ff12f5d1bf172d38e0f6586bc81c2d44535c2 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 208ff12f5d1bf172d38e0f6586bc81c2d44535c2 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only fix that targets the real cause, a drain loop that never yields to the event loop, and the new test pins the regression.

Notes

  • The same spin can happen in the other 17 while <set>: await asyncio.gather(...) loops in test/, as the description says. One shared drain helper in a test util would fix them all in that follow-up, instead of copying the line 17 more times.

[DESIGN-REVIEWED] 208ff12

Description read

[DESCRIPTION-READ] 65ce2e88ee3c32c6d192bf61683f0467f5af7a350c1f5f139ff5b90146d2c409

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 208ff12f5d1bf172d38e0f6586bc81c2d44535c2 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 208ff12

Verdict parsed from the review's SHA-scoped output markers for commit 208ff12f5d1bf172d38e0f6586bc81c2d44535c2.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 208ff12f5d1bf172d38e0f6586bc81c2d44535c2 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 208ff12f5d1bf172d38e0f6586bc81c2d44535c2 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 208ff12

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 208ff12f5d1bf172d38e0f6586bc81c2d44535c2 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 6, 2026 11:43
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] macOS Tests (on demand) shard 2 is red on 208ff12f5d1b with test/test_resume_retired_agent_names.py::test_a_view_the_real_boot_drain_retired_still_resumes[alias-rewritten] (assert (not True) at line 275). That is the known flake #29696: the same node id is red on several other heads that do not touch skill projection (macOS, Windows and Linux, listed there), and it passes locally on macOS at this head (all 3 params, -n 0). This PR changes only test/test_crewmate_feed.py and test/test_crewmate_feed_cold_store.py. Recorded on #29696; the failed job gets one rerun once the run finishes.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the test drain helper yields to the loop so finished tasks leave the keep set.

@bolichen97
bolichen97 merged commit e2b2822 into main Nov 6, 2026
114 of 123 checks passed
@bolichen97
bolichen97 deleted the fix/24356-drain-yields branch November 6, 2026 13:24
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run the on-demand macOS suite against this PR head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_retired_question_refreshes_the_waiting_list_with_no_log_growth times out on macOS

2 participants