Repository navigation
test(crewmate-feed): let the drain helper yield to the loop - #31386
Conversation
The crewmate feed tests drain crewmate_feed._keep with `while _keep: await asyncio.gather(*_keep)`. A task leaves _keep through its done callback, which the loop runs on its next pass. On Python 3.12 a gather over tasks that are all done completes eagerly and awaiting it does not yield, so when the helper starts in the same pass in which the last task finished, the discard never runs and the loop spins on the event loop thread until the test times out. Yield once per pass in test_crewmate_feed.py's _drain and in the same loop in test_crewmate_feed_cold_store.py's _settle, and pin the helper with a test that counts gather calls instead of hanging. Test-only; nothing under src/ changes.
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a test-only fix that targets the real cause, a drain loop that never yields to the event loop, and the new test pins the regression. Notes
[DESIGN-REVIEWED] 208ff12 Description read[DESCRIPTION-READ] 65ce2e88ee3c32c6d192bf61683f0467f5af7a350c1f5f139ff5b90146d2c409 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
[operator: bolichen97] |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the test drain helper yields to the loop so finished tasks leave the keep set.
Problem / Motivation
Goal:
test_a_retired_question_refreshes_the_waiting_list_with_no_log_growthstops hanging until the CI timeout on a slow runner.The test times out now and then in CI:
Failed: Timeout >180.0son the macOS backend shard (#24225's run) andFailed: Timeout >120.0son a Linux shard (#14783's run). It passes in a few seconds whenever it does not hang. The CI stack dump shows the event loop thread busy inside the test's_drainhelper, inasyncio.gather, with the executor thread idle, so the loop was spinning, not waiting.Why it matters
A hang costs a whole shard (three minutes on macOS) and a rerun on PRs that never touch the crewmate feed, and it trains authors to rerun reds. The earlier
_isolatedclean-up (#23644) was already in the base of the failing macOS run, so it did not stop this.Not a goal
src/. Production code never drainscrewmate_feed._keep; it only holds strong references there.crewmate_feed._keep: the 17 otherwhile <set>: await asyncio.gather(*<set>)loops intest/, and thedrain_promotions,drain_relaunchesandchannel_restarts_settledhelpers undersrc/kiro_crew/, which no production path calls (drain_relauncheshas no caller; tests call the other two). None of them has a flake report, and they are left for a follow-up.What changed (motivation → approach → change)
The test's
_drainhelper loopswhile crewmate_feed._keep: await asyncio.gather(*list(crewmate_feed._keep)). A task leaves_keepthrough its done callback (_keep.discard), and the event loop runs that callback on its next pass. On Python 3.12,asyncio.gatherover tasks that are all done returns a future that is already done, and awaiting a done future does not yield. So when_drainstarts in the same loop pass in which the last task (the goal read) finished, that task is still in_keep,gatherreturns at once, the discard never gets a chance to run, and thewhilespins on the loop thread until pytest-timeout fires. It is a timing race, which is why a loaded runner shows it.The helper now yields once per pass with
await asyncio.sleep(0)after eachgather, so the queued discard callbacks run before_keepis checked again.test_crewmate_feed_cold_store.py's_settlehas the same loop and gets the same line. This is a test-only change.Backwards compatibility
Compatible: only two test helpers change; no product code, config or API is touched.
Removes nothing: the diff adds one yield per helper loop and one test.
Tests
test_drain_returns_when_a_finished_task_still_waits_for_its_discard(new, intest/test_crewmate_feed.py) builds the exact state: a finished task still in_keepwith its discard queued. It countsgathercalls instead of hanging. With the yield reverted it fails at once, on its 50thgathercall, withAssertionError: _drain spun without yielding to the loop; with the yield it passes.Proof: not provable -- test-only change
Manual verification
On macOS 26.7.2 (arm64, Apple silicon), Python 3.12.13. The host was shared and already loaded (load average 28 to 60 on 18 cores), so no extra CPU hogs were added; the forced condition below is the decisive evidence, not host load.
-n 0, repeatedTimeout >60.0sin_drain->gather, the child a finished_record_current_goaltask, executor thread idle-n 2 --dist loadgroup --max-worker-restart=2, 20 runstest_crewmate_feed.pyonly)_drain-n 0, seeds 4242, 17, 30893The forced condition is a throwaway
-pplugin, not committed. It swaps the module'sasync_wait_untilfor one that polls on every loop pass and returns in the first pass in which every_keeptask is done, so the discard of the last one is still queued. Every forced run printed_keep = ['done']at that point.The parent's hang on this Mac (excerpt of the pytest-timeout dump: cut lines marked ..., caret lines dropped, interpreter paths shortened)
Forced-condition plugin and command
Related Issues
Closes #24356
Related: #23674 (the same test timing out on Linux), #23644 (the earlier
_isolatedclean-up)Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.