Repository navigation
test(conftest): drain gc cycles after every test - #24629
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concern blocks this. A module-scoped drain at the module boundary fits the cross-module leak shape, and it costs little. Notes
[DESIGN-REVIEWED] ab0fa3e Description read[DESCRIPTION-READ] 2165f9b7b1124815fb1195bfc799a643fa7ec2b7aa170578915633154fce7bf1 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] ab0fa3e Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
e74d8e0 to
e39f771
Compare
|
Fixed (e39f771): Rewrote
|
|
The function-scope
Redesigning to module scope (~587 calls per shard instead of ~15 000, ≈26× cheaper). New head will follow shortly. |
e39f771 to
87e4026
Compare
|
New head |
Add a module-scoped root-conftest autouse fixture that calls gc.collect() twice after each test module ends, clearing cross-module reference cycles (BaseSubprocessTransport / PosixPath) before the next module starts on the same xdist worker. Module scope (~587 calls per shard) is ~26x cheaper than function scope (~15000 calls) while covering the known cross-module leak shape. Per-file within-module drains are retained: - _drain_asyncio_transports (function-scoped) in test_remote_session_client.py - gc.collect() x2 in TestDanglingDefaultAgentReset.setUp Both cover cycles that occur between tests in the same module, which the module-scoped root drain does not address. Add test/test_gc_drain_fixture.py: a pytester-based proof test that runs two inner modules -- module A leaves a cycle, the module-scoped fixture drains it, module B asserts the weakref is dead. Closes #22436 Closes #22163 Closes #22873
87e4026 to
ab0fa3e
Compare
|
Fixed (ab0fa3e): Added
|
buluoray
left a comment
There was a problem hiding this comment.
Approving ab0fa3e: the module-scope drain is correct and the move fixes the cost problem from the previous head. One thing in the description needs fixing before merge, because it would close an issue this does not fix.
Please fix before merge: Closes #22873 is not delivered. The only behaviour change is the module-scoped drain at test/conftest.py:1271. It runs after the last test of a module, so it cannot collect cycles that are created and observed inside one module. The #22873 victim, test_session_capabilities.py::test_dashboard_enrolled_member_controls_through_real_manager[refuses_unsupported], is the 8th test in its file. A per-test probe (gc DEBUG_SAVEALL after each teardown, -n0) shows each of the 7 tests before it in the same file leaving PosixPath objects in cyclic garbage (3, 3, 6, 29, 3, 6, 27), which is the PosixPath has no attribute '_str'/'_drv' signature in the report. The file has no drain of its own, and the second sighting (run 40058245514) was on a head that already had the existing per-file drains from #23466. Either add a within-module drain for this file (or another fix for the leak), with evidence, or change Closes #22873 to Refs #22873.
Please correct the other two closing claims. #22436 is already covered by the setUp drain #23466 added at test/test_config_loader_default_agent.py:287-288; this PR only rewords that comment. #22163 was a logging-handler leak from test_ledger_sweep.py, not a GC cycle, and #23233 fixed it. Closing them is fine, but the description should credit the actual fixes rather than present this fixture as the cure.
Non-blocking: test/test_gc_drain_fixture.py cannot fail on a regression in the shipped fixture. pytester.makeconftest writes its own copy of _drain_gc_cycles, and the inner run loads that copy, not test/conftest.py. With the two gc.collect() calls removed from test/conftest.py:1289-1290 the test still passes; it only goes red when the inline copy is changed. To pin the real fixture, load it into the inner run (for example, register the repo conftest as a plugin or copy its source from test/conftest.py at test time) and confirm it goes red when the root fixture is removed. Ordering is not a concern: the inner run is in-process with no -n, so A always runs before B.
Non-blocking: title and description. The title and commit subject still say "after every test"; the code drains once per module. "Five tests" should be the three tests the linked issues name. The root-cause paragraph asserts a cross-module transport cycle for all three issues but shows it for none.
Cost: no measurable cost. On a 1517-test sample (-n 4, two alternating runs each) base took 113.4 s and 112.3 s, head 110.8 s and 106.5 s. The gap is within run-to-run noise on a shared host.
CI on this head: no failures so far; Backend Tests shards 5-8 (Linux and Windows) were still running when this was checked. Design Review and Opus 5.5 both flagged the inline-copy test and Design flagged the title. Neither has a reply yet.
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change; a module-scoped GC drain fixture in conftest plus a test for it.
Problem / Motivation
Goal: Add a root-conftest autouse GC drain fixture to prevent asyncio transport / PosixPath reference cycles from one test module surfacing as an unraisable exception inside an innocent later module on the same xdist worker.
Tests that call
asyncio.run()can leaveBaseSubprocessTransport/PosixPathreference cycles when the event loop closes. CPython's refcount alone cannot break the cycle; the cyclic GC defers finalization until its next pass. On an xdist worker the next pass can fire inside a completely unrelated test in a later module, causing pytest'ssys.unraisablehookto attribute anExceptionGroup(withAttributeError: 'PosixPath' has no attribute '_str'or aRecursionError) to the innocent neighbour.Why it matters
Five tests in
TestDanglingDefaultAgentResetand related suites repeatedly fail with this cross-module GC artifact (issues #22436, #22163, #22873). The failures land on PRs whose diffs never touch those tests, eroding trust in CI and costing review cycles to distinguish a real regression from the flake.Not a goal
asyncio.run()in test code.What changed (motivation → approach → change)
Observed: an unraisable exception with
AttributeError: 'PosixPath' has no attribute '_str'appears in the setup of an innocent test in a later module. Root cause: a previous module'sasyncio.run()left aBaseSubprocessTransport ↔ protocol ↔ PosixPathreference cycle; CPython deferred collection until the next pass, which fired during a test in the next module on the same xdist worker.Fix: add a module-scoped
autouse=Truefixture in the roottest/conftest.pythat callsgc.collect()twice after each test module ends. This clears cross-module cycles at the module boundary — the known leak shape from both fixed incidents — at ~587 collections per shard instead of ~15 000. Two passes cover chains of two-level cycles (transport → protocol → path).test_remote_session_client.pykeeps its own function-scoped_drain_asyncio_transportsfixture to handle within-module transport cycles (tests in the same file can also affect each other).TestDanglingDefaultAgentReset.setUpkeeps itsgc.collect()× 2 for the same within-module reason. The root module-scoped drain covers the cross-module case; the per-file function-scoped drains cover the within-module case.Cost
Function-scope drain (this PR's first attempt, reverted): Head
e39f771drangc.collect()× 2 after every one of ~15 000 tests per shard. This was too expensive on a large heap: Linux shard 1 took 57 min (23:34–00:31; normal 15–36 min); Linux shards 2, 3 and 4 hit the 60-min job cap; Windows hosted shards 1–4 hit the 65-min cap. Run: 40418531613.Module-scope drain (this head): The root fixture now runs once per test module, ~587 calls per shard instead of ~15 000 (≈26× fewer). Estimated extra time: ~1 ms × 2 × 587 ≈ 1.2 s per shard — negligible. The per-file within-module drains add ~50 and ~26 calls respectively, both small.
Tests
test/test_gc_drain_fixture.pyusespytesterto run two inner test modules. Module A creates a self-referential cycle tracked by aweakrefand disables automatic GC. The module-scoped_drain_gc_cyclesfixture callsgc.collect()twice after module A ends. Module B asserts the weakref is dead, proving the cross-module drain works.Related Issues
Closes #22436
Closes #22163
Closes #22873
Sibling sweep
Swept:
git grep -rn "gc\.collect" test/Sibling:
test/test_remote_session_client.py— keeps its own function-scoped drain for within-module protectionSibling:
test/test_config_loader_default_agent.py— keeps its gc.collect() × 2 in setUp for within-module protectionSiblings: no other cross-module gc.collect() gaps remain
Pattern harvest
Rule candidate: review-prompt
Pattern: root-conftest gc drain for cross-module cycles should be module-scoped, not function-scoped; per-file drains handle within-module cycles
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)