Repository navigation
test: bound every task-store open wait above Windows runner stalls - #16615
Conversation
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 501e6f8 Description read[DESCRIPTION-READ] 0fdc1a5aa3b16bfd1d23f30cc0b63bf4f5d1ae4756453389bdd15b0e38d03480 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
The writer-thread cases in test_queue_depth_reconcile.py awaited the manager's off-loop task-store open under a fixed 5 s wait_for. On a hosted Windows runner the process stalls for 5 to 7 s at a time, so the wait timed out on an open that would have attached. All eight sites that bounded that open at 5 s now call one helper, overload_fakes.wait_taskq_open, which awaits the open and fails by name with the manager's refusal state under a 20 s ceiling that stays below the tests' timeout(30). A regression holds the open for 6 s and requires it to still attach; a second test shows a wedged open still fails by name. The file's _until and emit-drain waits use the same _SETTLE_SECS ceiling. Closes kirodotdev#16612
d55f4bd to
501e6f8
Compare
Problem / Motivation
Goal: The subagent queue tests that wait for the task-store open stop failing on Windows runners when the open is merely slow.
test/test_queue_depth_reconcile.pyfails onBackend Tests (Windows) (2)withTimeoutErroron PRs that do not touch it (#15596 head 6d0f391, #14870 head ad75048). Both failures come from_manager'sawait asyncio.wait_for(mgr.wait_taskq_ready(), 5)in a[writer-thread]case, before the test body runs. Evidence and timings are in #16612.Why it matters
A red Windows shard on an unrelated PR blocks readiness and costs a fresh head to retry. The same 5 s bound sits at eight sites across the subagent queue tests, so any of them can hit it on a stalled runner.
Not a goal
SubagentManager, the task store or how the open runs. The open is already event-driven and completes. Only the tests' guard around it changes._untilhelper's design (a 10 ms poll for timer-driven re-reads). It keeps its shape and only shares the file's ceiling.wait_taskq_ready()call sites, which are unbounded and rely on the test timeout.What changed (motivation → approach → change)
wait_for(..., 5)overload_fakes.wait_taskq_open(mgr)TimeoutErrorAssertionError: the task store did not open within 20.0s (state: ...)_untiland the emit drain in_settlecarry private 5 s bounds_SETTLE_SECSThe symptom is a timeout out of the store-open wait. The cause is the bound, not a race.
wait_taskq_readyawaits the open's own task, so the wait is a signal. The 5 swait_foraround it is a lost-run guard, but it is shorter than stalls the hosted Windows runner produces. The progress artifacts show 5.5 to 6.9 s calls in this file, against a median of about 0.3 s there and about 15 ms on an idle Linux host. Only[writer-thread]cases failed because only that mode bounds the open. The[on-loop]mode opens inside the constructor with no bound.The change keeps one guard in one place.
wait_taskq_openintest/overload_fakes.pyawaits the open and, pastSTORE_OPEN_CEILING_SECS(20 s, above the measured stalls and below thetimeout(30)these tests carry), fails by name with the manager's_taskq_unavailablestate. The eight sites are intest_queue_depth_reconcile.py,test_adaptive_startup_memory.py(two),test_queue_depth_after_deferred_start.py,test_subagent_queued_visibility.py,test_session_cleanup.py,test_admission_gate.py(two) andtest_spawn_floor_defaults.py.Backwards compatibility
Compatible: test-only change. No product code, config or public function changes.
Removes nothing: the change only replaces a timeout guard in tests with a shared helper.
Tests
test_a_store_open_held_up_by_a_host_freeze_still_attachesholds the manager's_open_taskqon its worker thread for 6 s, then requires the store to attach and a stage cancel to publish{"queued": 0}. On the parent commit it fails with the sameTimeoutErrorout of_manageras CI.test_a_store_open_that_never_returns_fails_by_nameholds the open and requireswait_taskq_open(mgr, ceiling=0.1)to raise the namedAssertionError, so the guard still catches a wedged open.Run on this head:
pytest test/test_queue_depth_reconcile.py test/test_adaptive_startup_memory.py test/test_queue_depth_after_deferred_start.py test/test_session_cleanup.py test/test_subagent_queued_visibility.py test/test_admission_gate.py test/test_spawn_floor_defaults.py -n 8: 375 passed.test/test_queue_depth_reconcile.pyrun as two concurrent-n 8runs per round pinned to 4 cores (16 xdist workers on 4 cores). 200 runs on the change before its rebase onto main and 100 runs on this head: every test passed in all 300.scripts/check_black_formatting.py,isort --check-only,flake8on the changed files: clean.mypyon the changed files: same output as the parent commit.COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py: passed.Manual verification
N/A — test-only change. The Linux stress loop cannot reproduce a Windows runner stall, so the injected 6 s hold in the regression stands in for it.
Related Issues
Closes #16612
Pattern harvest
Rule candidate: review-prompt
Pattern: a fixed short
wait_foraround an event-driven wait in a test, where the bound is meant as a lost-run guard but is shorter than CI runner stalls.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)