Skip to content

test(windows): drain the loop before calling the owner graph retained - #12348

Merged
bolichen97 merged 1 commit into
mainfrom
test/reap-weakref-drain
Sep 20, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
test/reap-weakref-drain

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A backend test shard reds on code the PR under review never touched. The message is always the same:

FAILED test/test_windows_tree_reap.py::test_failed_owner_cleanup_survives_gc_and_retries_from_maintenance
  AssertionError: pending cleanup retained the process owner graph

Seen on #11827 (Windows shard 1, attempt 1 and the automatic attempt 2) and on #11807 (Linux shard 1). Neither diff touches Windows process control, so it is not Windows-only and it is not the author's change.

Why it matters

The shard is a required check, and a fork contributor cannot re-run a single failed job. Every hit costs that author a force-push and a fresh CI round of roughly 40 minutes on a PR that is already green.

What changed (motivation → approach → change)

The test provokes a refusal, drops its last strong reference to a fake process, and asserts a weakref to it is dead. It samples that weakref one event-loop tick and one gc.collect() after the drop.

What holds the referent at that moment is a closure cell on the frames of the refusal itself. Those frames hang off the exception's traceback, so the release waits on the cyclic collector seeing a traceback that the tick happens to drop. That tick is the entire margin, measured: on an idle host the weakref clears on exactly that pass and no earlier. A shard that needs one more pass reds.

The except block clears the caught exception's __traceback__. The frames go with it, so a single gc.collect() frees the referent with no tick at all, and nothing about the outcome depends on when CPython finalizes anything. A retention the product itself holds, brief or permanent, still reds.

__import__("gc") and __import__("weakref") become ordinary top-level imports, because the collection is now named in two places.

Backwards compatibility

Compatible: test-only change. No product code, no public contract, and the assertion covers strictly more than before.

Tests

test/test_windows_tree_reap.py is the only file touched. Its 13 tests pass, the target test runs in 0.01s, and 40 consecutive runs under random ordering pass.

Three controls, built locally and discarded:

Control Expected Result
Permanent retention: the duplicate-handle stub stashes the process in a module-level list fails fails, same message
Brief retention: a call_later releases the process 30ms later fails fails, same message
Causality: the __traceback__ clear removed, the single collect kept fails fails, same message

The first two show the assertion still catches a retention the product holds, whichever kind. The third shows the clear is what frees the referent, so the pass is not luck.

Manual verification

N/A — unit coverage sufficient. The controls above reproduce the CI failure and then close it directly.

Related Issues

Fixes #12336

Pattern harvest

Rule candidate: review-prompt
Pattern: a test asserts a weakref referent is already collected after a fixed number of event-loop ticks, which makes CPython finalization timing a contract.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and buluoray September 20, 2026 19:44
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: make the owner-release assertion in test_windows_tree_reap.py depend on the product actually releasing the process graph, not on which event-loop tick CPython finalizes the refusal traceback on, so an unrelated PR stops going red on a loaded shard.
Not a goal: touching any product code, changing what the assertion proves, or fixing other timing-dependent tests in the suite.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed f59c53d8dcb4c837bc0f880fc6cd9d43e78817ab — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f59c53d

Verdict parsed from the review's SHA-scoped output markers for commit f59c53d8dcb4c837bc0f880fc6cd9d43e78817ab.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f59c53d8dcb4c837bc0f880fc6cd9d43e78817ab: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of f59c53d8dcb4c837bc0f880fc6cd9d43e78817ab — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

The change removes the actual root of the flake (a finalization-timing dependency, not the retention assertion), the product code confirms no exception is stashed that the __traceback__ clear could mask — _PENDING_WINDOWS_TREE_CLEANUPS holds pids/handles, not the process object — and the documented retention/causality controls establish the assertion still reds on both permanent and brief product-side retention. Test-only, fully reversible.

[DESIGN-REVIEWED] f59c53d

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of f59c53d8dcb4c837bc0f880fc6cd9d43e78817ab and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f59c53d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f59c53d8dcb4c837bc0f880fc6cd9d43e78817ab: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
…r graph

The owner-release assertion in test_failed_owner_cleanup_survives_gc_and_retries_from_maintenance samples a weakref one event-loop tick and one gc.collect() after dropping the last strong reference. What holds the referent at that point is a closure cell on the frames of the refusal the test just provoked, so the release waits on the cyclic collector seeing a traceback the tick happens to drop. That tick is the whole margin, and a loaded CI shard that needs another one reds on unrelated code.

Clear the caught exception's __traceback__ inside the except block instead. The frames go with it, so one gc.collect() frees the referent with no tick at all, and a retention the product itself holds -- transient or permanent -- still reds.

Fixes #12336
@bolichen97
bolichen97 force-pushed the test/reap-weakref-drain branch from dc2a912 to f59c53d Compare September 20, 2026 20:01
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: none added; the poll budget and its module constant are removed, leaving one deterministic collect.

  • Raised retry budget weakens the owner-retention ratchet (span=beba01d7219d): fixed in f59c53d.

Legitimate, and the local review missed it. A poll budget cannot tell "CPython needed another tick" from "the product held the process 200ms and let go", so a brief retention regression lands inside the designed-for window. The prescribed fix is what shipped: exc.__traceback__ = None in the except block drops the frames holding the closure cell, one gc.collect() then frees the referent with no event-loop tick, and _OWNER_RELEASE_PASSES with its loop is gone.
Verified with three controls, each a throwaway copy of the test, all failing with the original message: permanent retention (stub stashes the process in a module list), brief retention (call_later releases it after 30ms), and causality (the __traceback__ clear removed, the single collect kept). The brief-retention control is the property this finding names; it reds now. The causality control shows the clear is what frees the referent.
13 tests pass, 40 consecutive runs under random ordering pass, target test 0.01s. isort/flake8/black/comment-history/docs-lint/sync-io/testpaths/loop-bound-locks all exit 0.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on f59c53d8dcb4c837bc0f880fc6cd9d43e78817ab after source-level verification.

Verified, not taken on trust:

  • The assertion is still falsifiable after the __traceback__ clear. Injected a product-side strong reference to the process object into state.terminally_scanned (reachable from the module-global _PENDING_WINDOWS_TREE_CLEANUPS) and re-ran the target test: it reddened with exactly pending cleanup retained the process owner graph at test/test_windows_tree_reap.py:185. Restoring the product code turns it green again.
  • Clearing the traceback masks nothing. The exception is already asserted at test/test_windows_tree_reap.py:174 before the clear at :177, the no-exception path is guarded by pytest.fail at :179, and nothing re-raises. The frame held process only as the parameter of terminate_windows_asyncio_tree — a harness artifact, not the product's retention path: _PendingWindowsTreeCleanup.__slots__ stores handles/pins only (src/kiro_crew/platform_compat.py:3862-3888), and the product reference is one-directional (process._windows_cleanup_state = state, :3985).
  • git grep _OWNER_RELEASE_PASSES is empty at both this head and origin/main — no dangling reference.

Non-blocking: if one gc.collect() were ever insufficient in place of the old sleep(0) + collect(), the failure direction is a flaky RED (weakref still set), never a false GREEN — the safe side.

@bolichen97
bolichen97 merged commit 56c4e3c into main Sep 20, 2026
95 checks passed
@bolichen97
bolichen97 deleted the test/reap-weakref-drain branch September 20, 2026 22:44
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 20, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(windows): test_failed_owner_cleanup_survives_gc_and_retries_from_maintenance is flaky (weakref collection timing)

3 participants