Skip to content

test(history): let the consolidator task drains yield to the loop - #37302

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/37162-history-consolidation-timeout
Nov 14, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/37162-history-consolidation-timeout

Conversation

@JLargent42

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_sessions_refused_while_the_host_was_broken_become_eligible and the other history-consolidation tests that drain HistoryConsolidator._tasks stop hanging until the CI timeout on a loaded runner.

The test hit Failed: Timeout >180.0s on Backend Tests (macOS) (6) in run 43573488656, on a PR that does not touch history consolidation. It passes in about 0.05 s whenever it does not hang.

Why it matters

A hang costs a whole macOS shard (three minutes of timeout plus a rerun) on PRs that never touch consolidation, and it trains authors to rerun reds.

Not a goal

  • Any change under src/. Production code never drains _tasks; it only keeps strong references there.
  • A timeout, retry, skip or looser assertion in any helper or test.
  • Drain loops over sets other than HistoryConsolidator._tasks (for example the browser hub tests). They have no flake report here.

What changed (motivation → approach → change)

Symptom: the CI failure points at test/test_history_consolidation_env_release.py:49, the while c._tasks: line of the _drain helper. The test coroutine is the frame running when pytest-timeout's signal lands. Every other thread in the dump is idle (the asyncio_N executor threads sit in work_queue.get). So the event loop thread was spinning in _drain. It was not parked in select(), which would show selectors.select as the last frame. I checked that locally: a never-done future in the same loop shape fails in selectors.py, and a done one fails in _drain/gather.

Root cause: this is the same race #31386 fixed for the crewmate feed. A task leaves _tasks through its done callback (_tasks.discard), and the loop runs that callback on its next pass. On Python 3.12, a gather over tasks that are all done does not yield. So when the drain starts in the same pass in which the last task (here the _release_environment_backoff task) finished, that task is still in _tasks. The gather then returns at once, the discard never runs, and the while spins until the timeout.

Change: each consolidator _tasks drain now calls await asyncio.sleep(0) after its gather, so the queued discard callbacks run before the set is checked again. Test-only.

Backwards compatibility

Compatible: only test helpers change. No product code, config or API is touched.

Removes nothing: the diff only adds one yield per drain loop and one test.

Tests

  • test_drain_returns_when_a_finished_task_still_waits_for_its_discard (new, in test/test_history_consolidation_env_release.py) builds the exact state: a finished task still in a real consolidator's _tasks, with its discard queued. It counts gather calls instead of hanging. On origin/main's helper (4163203) it fails at once with AssertionError: _drain spun without yielding to the loop. With the yield it passes.

Proof: not provable -- test-only change

Manual verification

  • pytest -n 2 over the four touched files: 215 passed.
  • test/test_flake_pattern_ratchet.py, black, isort, flake8, check_comment_history.py and check_brand_name.py on the touched files: all clean.
  • Linux only. No macOS host was available. The fix does not depend on the platform: the race only needs a loaded runner.

Related Issues

Fixes #37162

Same root cause as #31386 (crewmate feed _drain).

Sibling sweep

Swept: git grep -n "while c._tasks:" -- test
Sibling: test/test_history_consolidation_env_release.py:49 -- changed here
Sibling: test/test_history_consolidation_bounds.py:373 -- changed here
Sibling: test/test_history_consolidation_bounds.py:384 -- changed here
Sibling: test/test_history_consolidation_bounds.py:481 -- changed here
Sibling: test/test_history_consolidation_retry_accounting.py:198 -- changed here
Sibling: test/test_history_consolidation_threshold_off_loop.py:63 -- changed here

Pattern harvest

Rule candidate: semgrep
Pattern: while <set>: await asyncio.gather(*<set>) with no yield to the loop after the gather spins on Python 3.12 once every task in the set is done but its discard callback is still queued.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

On Python 3.12 a gather over tasks that are all done does not yield, so
a drain loop that starts in the loop pass in which its last task
finished never lets that task's _tasks.discard callback run, and the
while spins until pytest-timeout fires. Yield once per pass after each
gather in every consolidator _tasks drain, and pin the env-release
helper with a test that counts gather calls instead of hanging.

Fixes #37162
@JLargent42
JLargent42 requested review from a team and KiroCrewCaptain as code owners November 14, 2026 18:39
@JLargent42 JLargent42 self-assigned this Nov 14, 2026
@JLargent42
JLargent42 requested a review from CrysisDeu November 14, 2026 18:40
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision c979da8816b85f8ca2f11ca6513d0c46cece610a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision c979da8816b85f8ca2f11ca6513d0c46cece610a ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of c979da8816b85f8ca2f11ca6513d0c46cece610a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only fix for the real root cause: each drain loop now yields once so the queued _tasks.discard callbacks run. The sibling sweep covers every while c._tasks: site.

Notes

  • Each of the five drain sites now carries its own copy of the same loop and comment. One shared drain_consolidator_tasks(c) test helper would mean the next drain added can't leave out the yield.

[DESIGN-REVIEWED] c979da8

Description read

[DESCRIPTION-READ] 658fe80bcb9533d6a4139e6d0b9420e14d6fc7a7dc5cfe0357e3d0a3e0673e98

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed c979da8816b85f8ca2f11ca6513d0c46cece610a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] c979da8

Verdict parsed from the review's SHA-scoped output markers for commit c979da8816b85f8ca2f11ca6513d0c46cece610a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c979da8816b85f8ca2f11ca6513d0c46cece610a false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of c979da8816b85f8ca2f11ca6513d0c46cece610a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c979da8

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c979da8816b85f8ca2f11ca6513d0c46cece610a false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

This change touches tests only. After each gather, every while c._tasks: drain loop over the HistoryConsolidator now calls await asyncio.sleep(0). That covers all five sites: env_release, bounds ×3 (wait, that's 3 in bounds) — env_release ×1, bounds ×3, retry_accounting ×1 and threshold_off_loop ×1. I searched the repo and found no other spinning drain of consolidator _tasks. The rest are single gather calls, which cannot spin.

The root cause holds up. On Python 3.12, gather handles children that are already done eagerly, so await on an all-done gather never yields to the loop. The _tasks.discard done-callback is queued with call_soon, so it never runs while the loop is spinning, and the while loop never ends. sleep(0) puts the test task back in the ready queue after the discard callbacks that are already queued, so the set is empty before it is checked again.

The new regression test builds exactly that state. It creates a task, adds the discard callback, and calls sleep(0) once, so the task finishes with its discard still queued. It then patches asyncio.gather to count calls, which turns the hang into a failing assertion. Without the yield it would fail a

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit c979da8816 · 37 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 14, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 14, 2026 19:59

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The review checked this head. All six while c._tasks: drain loops over HistoryConsolidator._tasks now get await asyncio.sleep(0) after the gather, and nothing under src/ changes. The root cause was reproduced on Python 3.12: a gather over tasks that are already done returns without yielding, so a queued discard callback never runs and the loop spins. On 3.11 the same loop exits. The new regression test fails on the old _drain shape and passes with the fix. The four touched files pass locally, and the backend, Windows, SAST and Flake Pattern Ratchet checks are green on this head.
Well done: the regression test counts gather calls instead of waiting on a timeout. A hang becomes a quick, named failure, and the test cannot become the next 180 s timeout itself.

  • Optional: only the yield in env_release _drain has a test that fails when it is removed. A shared drain helper used by all six sites would let that test cover all of them.
  • Optional: in a follow-up, the three browser LiveHub drain helpers (test/test_browser_live_hub.py:73, test/test_browser_live_loop.py:52, test/test_browser_handback_note.py:432) have the same loop shape and could get the same one-line yield.

@KiroCrewCaptain
KiroCrewCaptain merged commit 9517f90 into main Nov 14, 2026
97 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/37162-history-consolidation-timeout branch November 14, 2026 19:59
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_sessions_refused_while_the_host_was_broken_become_eligible times out at 180 s on a macOS shard

2 participants