Repository navigation
fix: wait on the zombie watchdog's stop signal in the sharing-violation test (#9478) - #10287
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Diagnosis verified against the code: Design-Verdict: PASS Correct root-cause diagnosis of a silent [DESIGN-REVIEWED] 3bdda5d |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only fix to test/test_mcp_gatewayd_coverage.py -- replaces asyncio.wait_for(coro, timeout=5) with a bounded asyncio.wait on the watchdog task plus its stop event, because the watchdog swallows CancelledError so on Python 3.11+ wait_for could never raise TimeoutError and the assertions failed misleadingly instead; adds a finally-block drain so the watchdog cannot leak past monkeypatch teardown. No production code touched.
chenmingwei23
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only coverage for mcp gatewayd, test/ file only.
dwu96
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change to test/test_mcp_gatewayd_coverage.py -- waits on the zombie watchdog's own stop signal instead of a wait_for that cannot time out once the watchdog swallows CancelledError, with a bounded 30s wait and a cancel-and-drain teardown so the watchdog never leaks past monkeypatch teardown.
bolichen97
left a comment
There was a problem hiding this comment.
Maintainer review (agent-assisted), pinned to 3bdda5d. Approving this test-only correction. The sharing-violation simulation and diagnostic-tag/stop assertions are preserved. Waiting on both the watchdog task and its stop signal surfaces early task failures and distinguishes a real timeout from swallowed cancellation; finally cancels and drains both tasks before fixture teardown. The actual watchdog sets stop only after its write completes. A standalone cancellation probe confirmed the mechanism, and all five TestZombieDiagnostic tests passed locally in an isolated worktree. Current-head PR Readiness success rechecked. Accepted: explicit wait budget increases to 30s under the test timeout; production behavior is unchanged.
Problem / Motivation
test_zombie_dump_survives_a_windows_sharing_violationfails non-deterministically on theBackend Tests (Windows)shard, in two confusing forms: aFileNotFoundErrorreadingdiag.jsonl, orassert stop.is_set()failing. It has redded unrelated PRs (#9213, #9223).Why it matters
A flaky required shard blocks merges that have nothing to do with the gateway watchdog. The two failure forms also point away from the real cause, so each occurrence costs a fresh investigation.
What changed (motivation → approach → change)
The symptom: on a slow Windows runner the test's 5-second budget expires mid-run. The test drove the watchdog with
asyncio.wait_for(gw._zombie_diagnostic(...), timeout=5).The root cause: the watchdog ends with
except asyncio.CancelledError: pass. Whenwait_fortimes out it cancels the coroutine, the coroutine swallows the cancel and returns None, and on Python 3.11+wait_foronly raisesTimeoutErrorwhen the cancellation actually propagates. So the timeout is silent. The test falls through to a downstream assertion, and which one fails depends on where the cancel landed: before the dump write gives form 1, between the write andstop_event.set()gives form 2. The dump itself is synchronous with respect to the coroutine (await asyncio.to_thread(...)thenstop_event.set()then return), so this is a test-harness defect, not a Windows write defect.The change: the test now creates the watchdog as a task and waits on the watchdog's own completion signal with
asyncio.wait({task, stop_wait}, timeout=30, return_when=FIRST_COMPLETED). Waiting on both means a watchdog that raises before settingstopsurfaces its exception immediately instead of hiding behind the 30s budget. Afinallycancels and drains both tasks so nothing leaks past monkeypatch teardown. A genuinely slow runner now fails legibly at the bounded wait with a named assertion instead of a misleading downstream error. Production code ingatewayd.pyis unchanged; whether the watchdog should re-raiseCancelledErroris a separate design question.The sibling tests were audited:
test_prefired_stop_event_writes_nothingreturns immediately and keeps itswait_forshape;test_cancellation_is_swallowedtests the swallow itself and is untouched; the other tests already use task-based shapes.Tests
This is a test-only change. It keeps the sharing-violation coverage intact (same
PermissionError-on-second-open simulation, same assertions ontags == ["probe", "zombie_detected"]andstop.is_set()), and changes only how the test bounds and observes watchdog completion.Manual verification
The flake is Windows-only and non-deterministic, so a local green run proves little. The change removes the two misleading failure forms structurally and turns residual runner slowness into an explicit failure at the bounded wait under a 30s budget. CI (including the Windows shard) is the verification. Local isort/flake8/mypy pass.
Related Issues
Closes #9478
Pattern harvest
Rule candidate: review-prompt
Pattern:
asyncio.wait_foraround a coroutine that swallowsCancelledErrorcannot report a timeout on Python 3.11+ — it returns None and downstream assertions fail misleadingly; wait on the code's own completion signal instead.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)