Skip to content

test: bound every task-store open wait above Windows runner stalls - #16615

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/16612-taskq-open-ceiling
Oct 4, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/16612-taskq-open-ceiling

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The subagent queue tests that wait for the task-store open stop failing on Windows runners when the open is merely slow.

test/test_queue_depth_reconcile.py fails on Backend Tests (Windows) (2) with TimeoutError on PRs that do not touch it (#15596 head 6d0f391, #14870 head ad75048). Both failures come from _manager's await asyncio.wait_for(mgr.wait_taskq_ready(), 5) in a [writer-thread] case, before the test body runs. Evidence and timings are in #16612.

Why it matters

A red Windows shard on an unrelated PR blocks readiness and costs a fresh head to retry. The same 5 s bound sits at eight sites across the subagent queue tests, so any of them can hit it on a stalled runner.

Not a goal

  • No change to SubagentManager, the task store or how the open runs. The open is already event-driven and completes. Only the tests' guard around it changes.
  • No change to the polled _until helper's design (a 10 ms poll for timer-driven re-reads). It keeps its shape and only shares the file's ceiling.
  • No change to other wait_taskq_ready() call sites, which are unbounded and rely on the test timeout.

What changed (motivation → approach → change)

Before After
Eight sites wrap the off-loop store open in wait_for(..., 5) All eight call overload_fakes.wait_taskq_open(mgr)
A 5 s stall fails as a bare TimeoutError A wedged open fails as AssertionError: the task store did not open within 20.0s (state: ...)
_until and the emit drain in _settle carry private 5 s bounds Both wait under the file's _SETTLE_SECS

The symptom is a timeout out of the store-open wait. The cause is the bound, not a race. wait_taskq_ready awaits the open's own task, so the wait is a signal. The 5 s wait_for around it is a lost-run guard, but it is shorter than stalls the hosted Windows runner produces. The progress artifacts show 5.5 to 6.9 s calls in this file, against a median of about 0.3 s there and about 15 ms on an idle Linux host. Only [writer-thread] cases failed because only that mode bounds the open. The [on-loop] mode opens inside the constructor with no bound.

The change keeps one guard in one place. wait_taskq_open in test/overload_fakes.py awaits the open and, past STORE_OPEN_CEILING_SECS (20 s, above the measured stalls and below the timeout(30) these tests carry), fails by name with the manager's _taskq_unavailable state. The eight sites are in test_queue_depth_reconcile.py, test_adaptive_startup_memory.py (two), test_queue_depth_after_deferred_start.py, test_subagent_queued_visibility.py, test_session_cleanup.py, test_admission_gate.py (two) and test_spawn_floor_defaults.py.

Backwards compatibility

Compatible: test-only change. No product code, config or public function changes.

Removes nothing: the change only replaces a timeout guard in tests with a shared helper.

Tests

  • test_a_store_open_held_up_by_a_host_freeze_still_attaches holds the manager's _open_taskq on its worker thread for 6 s, then requires the store to attach and a stage cancel to publish {"queued": 0}. On the parent commit it fails with the same TimeoutError out of _manager as CI.
  • test_a_store_open_that_never_returns_fails_by_name holds the open and requires wait_taskq_open(mgr, ceiling=0.1) to raise the named AssertionError, so the guard still catches a wedged open.

Run on this head:

  • pytest test/test_queue_depth_reconcile.py test/test_adaptive_startup_memory.py test/test_queue_depth_after_deferred_start.py test/test_session_cleanup.py test/test_subagent_queued_visibility.py test/test_admission_gate.py test/test_spawn_floor_defaults.py -n 8: 375 passed.
  • Stress: test/test_queue_depth_reconcile.py run as two concurrent -n 8 runs per round pinned to 4 cores (16 xdist workers on 4 cores). 200 runs on the change before its rebase onto main and 100 runs on this head: every test passed in all 300.
  • scripts/check_black_formatting.py, isort --check-only, flake8 on the changed files: clean. mypy on the changed files: same output as the parent commit. COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py: passed.

Manual verification

N/A — test-only change. The Linux stress loop cannot reproduce a Windows runner stall, so the injected 6 s hold in the regression stands in for it.

Related Issues

Closes #16612

Pattern harvest

Rule candidate: review-prompt
Pattern: a fixed short wait_for around an event-driven wait in a test, where the bound is meant as a lost-run guard but is shorter than CI runner stalls.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested a review from a team as a code owner October 3, 2026 21:37
@jeeshofone
jeeshofone requested a review from CrysisDeu October 3, 2026 21:37
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 501e6f80e9e4c44ab94b50713e4aa00ae8f0080c via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 501e6f8

Description read

[DESCRIPTION-READ] 0fdc1a5aa3b16bfd1d23f30cc0b63bf4f5d1ae4756453389bdd15b0e38d03480

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 501e6f80e9e4c44ab94b50713e4aa00ae8f0080c via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 501e6f8

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 501e6f80e9e4c44ab94b50713e4aa00ae8f0080c via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 501e6f8

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 3, 2026
The writer-thread cases in test_queue_depth_reconcile.py awaited the
manager's off-loop task-store open under a fixed 5 s wait_for. On a
hosted Windows runner the process stalls for 5 to 7 s at a time, so the
wait timed out on an open that would have attached.

All eight sites that bounded that open at 5 s now call one helper,
overload_fakes.wait_taskq_open, which awaits the open and fails by name
with the manager's refusal state under a 20 s ceiling that stays below
the tests' timeout(30). A regression holds the open for 6 s and requires
it to still attach; a second test shows a wedged open still fails by
name. The file's _until and emit-drain waits use the same _SETTLE_SECS
ceiling.

Closes kirodotdev#16612
@jeeshofone
jeeshofone force-pushed the fix/16612-taskq-open-ceiling branch from d55f4bd to 501e6f8 Compare October 3, 2026 23:05
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@bolichen97
bolichen97 merged commit e828bb8 into kirodotdev:main Oct 4, 2026
101 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flake: test_queue_depth_reconcile [writer-thread] times out on Windows behind a 5 s bound on the task-store open

2 participants