Repository navigation
test(history): let the consolidator task drains yield to the loop - #37302
Conversation
On Python 3.12 a gather over tasks that are all done does not yield, so a drain loop that starts in the loop pass in which its last task finished never lets that task's _tasks.discard callback run, and the while spins until pytest-timeout fires. Yield once per pass after each gather in every consolidator _tasks drain, and pin the env-release helper with a test that counts gather calls instead of hanging. Fixes #37162
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Test-only fix for the real root cause: each drain loop now yields once so the queued Notes
[DESIGN-REVIEWED] c979da8 Description read[DESCRIPTION-READ] 658fe80bcb9533d6a4139e6d0b9420e14d6fc7a7dc5cfe0357e3d0a3e0673e98 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. This change touches tests only. After each The root cause holds up. On Python 3.12, The new regression test builds exactly that state. It creates a task, adds the discard callback, and calls An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
The review checked this head. All six while c._tasks: drain loops over HistoryConsolidator._tasks now get await asyncio.sleep(0) after the gather, and nothing under src/ changes. The root cause was reproduced on Python 3.12: a gather over tasks that are already done returns without yielding, so a queued discard callback never runs and the loop spins. On 3.11 the same loop exits. The new regression test fails on the old _drain shape and passes with the fix. The four touched files pass locally, and the backend, Windows, SAST and Flake Pattern Ratchet checks are green on this head.
Well done: the regression test counts gather calls instead of waiting on a timeout. A hang becomes a quick, named failure, and the test cannot become the next 180 s timeout itself.
- Optional: only the yield in env_release
_drainhas a test that fails when it is removed. A shared drain helper used by all six sites would let that test cover all of them. - Optional: in a follow-up, the three browser LiveHub drain helpers (test/test_browser_live_hub.py:73, test/test_browser_live_loop.py:52, test/test_browser_handback_note.py:432) have the same loop shape and could get the same one-line yield.
Problem / Motivation
Goal:
test_sessions_refused_while_the_host_was_broken_become_eligibleand the other history-consolidation tests that drainHistoryConsolidator._tasksstop hanging until the CI timeout on a loaded runner.The test hit
Failed: Timeout >180.0son Backend Tests (macOS) (6) in run 43573488656, on a PR that does not touch history consolidation. It passes in about 0.05 s whenever it does not hang.Why it matters
A hang costs a whole macOS shard (three minutes of timeout plus a rerun) on PRs that never touch consolidation, and it trains authors to rerun reds.
Not a goal
src/. Production code never drains_tasks; it only keeps strong references there.HistoryConsolidator._tasks(for example the browser hub tests). They have no flake report here.What changed (motivation → approach → change)
Symptom: the CI failure points at
test/test_history_consolidation_env_release.py:49, thewhile c._tasks:line of the_drainhelper. The test coroutine is the frame running when pytest-timeout's signal lands. Every other thread in the dump is idle (theasyncio_Nexecutor threads sit inwork_queue.get). So the event loop thread was spinning in_drain. It was not parked inselect(), which would showselectors.selectas the last frame. I checked that locally: a never-done future in the same loop shape fails inselectors.py, and a done one fails in_drain/gather.Root cause: this is the same race #31386 fixed for the crewmate feed. A task leaves
_tasksthrough its done callback (_tasks.discard), and the loop runs that callback on its next pass. On Python 3.12, agatherover tasks that are all done does not yield. So when the drain starts in the same pass in which the last task (here the_release_environment_backofftask) finished, that task is still in_tasks. Thegatherthen returns at once, the discard never runs, and thewhilespins until the timeout.Change: each consolidator
_tasksdrain now callsawait asyncio.sleep(0)after itsgather, so the queued discard callbacks run before the set is checked again. Test-only.Backwards compatibility
Compatible: only test helpers change. No product code, config or API is touched.
Removes nothing: the diff only adds one yield per drain loop and one test.
Tests
test_drain_returns_when_a_finished_task_still_waits_for_its_discard(new, intest/test_history_consolidation_env_release.py) builds the exact state: a finished task still in a real consolidator's_tasks, with its discard queued. It countsgathercalls instead of hanging. Onorigin/main's helper (4163203) it fails at once withAssertionError: _drain spun without yielding to the loop. With the yield it passes.Proof: not provable -- test-only change
Manual verification
pytest -n 2over the four touched files: 215 passed.test/test_flake_pattern_ratchet.py, black, isort, flake8,check_comment_history.pyandcheck_brand_name.pyon the touched files: all clean.Related Issues
Fixes #37162
Same root cause as #31386 (crewmate feed
_drain).Sibling sweep
Swept: git grep -n "while c._tasks:" -- test
Sibling: test/test_history_consolidation_env_release.py:49 -- changed here
Sibling: test/test_history_consolidation_bounds.py:373 -- changed here
Sibling: test/test_history_consolidation_bounds.py:384 -- changed here
Sibling: test/test_history_consolidation_bounds.py:481 -- changed here
Sibling: test/test_history_consolidation_retry_accounting.py:198 -- changed here
Sibling: test/test_history_consolidation_threshold_off_loop.py:63 -- changed here
Pattern harvest
Rule candidate: semgrep
Pattern:
while <set>: await asyncio.gather(*<set>)with no yield to the loop after the gather spins on Python 3.12 once every task in the set is done but its discard callback is still queued.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.