Skip to content

fix: wait on the zombie watchdog's stop signal in the sharing-violation test (#9478) - #10287

Merged
iamwhatever merged 1 commit into
mainfrom
fix/zombie-diagnostic-test-await-stop-9478
Sep 12, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/zombie-diagnostic-test-await-stop-9478

Conversation

@CrysisDeu

Copy link
Copy Markdown
Contributor

Problem / Motivation

test_zombie_dump_survives_a_windows_sharing_violation fails non-deterministically on the Backend Tests (Windows) shard, in two confusing forms: a FileNotFoundError reading diag.jsonl, or assert stop.is_set() failing. It has redded unrelated PRs (#9213, #9223).

Why it matters

A flaky required shard blocks merges that have nothing to do with the gateway watchdog. The two failure forms also point away from the real cause, so each occurrence costs a fresh investigation.

What changed (motivation → approach → change)

The symptom: on a slow Windows runner the test's 5-second budget expires mid-run. The test drove the watchdog with asyncio.wait_for(gw._zombie_diagnostic(...), timeout=5).

The root cause: the watchdog ends with except asyncio.CancelledError: pass. When wait_for times out it cancels the coroutine, the coroutine swallows the cancel and returns None, and on Python 3.11+ wait_for only raises TimeoutError when the cancellation actually propagates. So the timeout is silent. The test falls through to a downstream assertion, and which one fails depends on where the cancel landed: before the dump write gives form 1, between the write and stop_event.set() gives form 2. The dump itself is synchronous with respect to the coroutine (await asyncio.to_thread(...) then stop_event.set() then return), so this is a test-harness defect, not a Windows write defect.

The change: the test now creates the watchdog as a task and waits on the watchdog's own completion signal with asyncio.wait({task, stop_wait}, timeout=30, return_when=FIRST_COMPLETED). Waiting on both means a watchdog that raises before setting stop surfaces its exception immediately instead of hiding behind the 30s budget. A finally cancels and drains both tasks so nothing leaks past monkeypatch teardown. A genuinely slow runner now fails legibly at the bounded wait with a named assertion instead of a misleading downstream error. Production code in gatewayd.py is unchanged; whether the watchdog should re-raise CancelledError is a separate design question.

The sibling tests were audited: test_prefired_stop_event_writes_nothing returns immediately and keeps its wait_for shape; test_cancellation_is_swallowed tests the swallow itself and is untouched; the other tests already use task-based shapes.

Tests

This is a test-only change. It keeps the sharing-violation coverage intact (same PermissionError-on-second-open simulation, same assertions on tags == ["probe", "zombie_detected"] and stop.is_set()), and changes only how the test bounds and observes watchdog completion.

Manual verification

The flake is Windows-only and non-deterministic, so a local green run proves little. The change removes the two misleading failure forms structurally and turns residual runner slowness into an explicit failure at the bounded wait under a 30s budget. CI (including the Windows shard) is the verification. Local isort/flake8/mypy pass.

Related Issues

Closes #9478

Pattern harvest

Rule candidate: review-prompt
Pattern: asyncio.wait_for around a coroutine that swallows CancelledError cannot report a timeout on Python 3.11+ — it returns None and downstream assertions fail misleadingly; wait on the code's own completion signal instead.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 3bdda5d0ae1fb2d1970a609670318c7f6101066c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Diagnosis verified against the code: gatewayd.py:4330-4331 does end the watchdog with except asyncio.CancelledError: pass, the sibling test_cancellation_is_swallowed pins that swallow as deliberate, and the success path (_write_diagnostic → stop_event.set() → return) makes waiting on the stop signal the correct observation point. The change is test-only, the production question is explicitly deferred, and the new shape drains both tasks before monkeypatch teardown. No design-level issues found.

Design-Verdict: PASS

Correct root-cause diagnosis of a silent wait_for timeout, fixed at the right layer (the test), with production semantics deliberately left untouched.

[DESIGN-REVIEWED] 3bdda5d

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 3bdda5d0ae1fb2d1970a609670318c7f6101066c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3bdda5d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3bdda5d0ae1fb2d1970a609670318c7f6101066c: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 3bdda5d0ae1fb2d1970a609670318c7f6101066c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3bdda5d

Verdict parsed from the review's SHA-scoped output markers for commit 3bdda5d0ae1fb2d1970a609670318c7f6101066c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3bdda5d0ae1fb2d1970a609670318c7f6101066c: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 12, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 12, 2026 05:04

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only fix to test/test_mcp_gatewayd_coverage.py -- replaces asyncio.wait_for(coro, timeout=5) with a bounded asyncio.wait on the watchdog task plus its stop event, because the watchdog swallows CancelledError so on Python 3.11+ wait_for could never raise TimeoutError and the assertions failed misleadingly instead; adds a finally-block drain so the watchdog cannot leak past monkeypatch teardown. No production code touched.

@iamwhatever
iamwhatever merged commit cbfd3a6 into main Sep 12, 2026
70 checks passed
@iamwhatever
iamwhatever deleted the fix/zombie-diagnostic-test-await-stop-9478 branch September 12, 2026 05:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 12, 2026

@chenmingwei23 chenmingwei23 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only coverage for mcp gatewayd, test/ file only.

@dwu96 dwu96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: test-only change to test/test_mcp_gatewayd_coverage.py -- waits on the zombie watchdog's own stop signal instead of a wait_for that cannot time out once the watchdog swallows CancelledError, with a bounded 30s wait and a cancel-and-drain teardown so the watchdog never leaks past monkeypatch teardown.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer review (agent-assisted), pinned to 3bdda5d. Approving this test-only correction. The sharing-violation simulation and diagnostic-tag/stop assertions are preserved. Waiting on both the watchdog task and its stop signal surfaces early task failures and distinguishes a real timeout from swallowed cancellation; finally cancels and drains both tasks before fixture teardown. The actual watchdog sets stop only after its write completes. A standalone cancellation probe confirmed the mechanism, and all five TestZombieDiagnostic tests passed locally in an isolated worktree. Current-head PR Readiness success rechecked. Accepted: explicit wait budget increases to 30s under the test timeout; production behavior is unchanged.

amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestZombieDiagnostic windows-sharing-violation test is flaky two different ways, reds Windows shards on unrelated PRs

5 participants