Skip to content

test(conftest): drain gc cycles after every test - #24629

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
unflaky/root-conftest-gc-drain
Oct 27, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
unflaky/root-conftest-gc-drain

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Add a root-conftest autouse GC drain fixture to prevent asyncio transport / PosixPath reference cycles from one test module surfacing as an unraisable exception inside an innocent later module on the same xdist worker.

Tests that call asyncio.run() can leave BaseSubprocessTransport / PosixPath reference cycles when the event loop closes. CPython's refcount alone cannot break the cycle; the cyclic GC defers finalization until its next pass. On an xdist worker the next pass can fire inside a completely unrelated test in a later module, causing pytest's sys.unraisablehook to attribute an ExceptionGroup (with AttributeError: 'PosixPath' has no attribute '_str' or a RecursionError) to the innocent neighbour.

Why it matters

Five tests in TestDanglingDefaultAgentReset and related suites repeatedly fail with this cross-module GC artifact (issues #22436, #22163, #22873). The failures land on PRs whose diffs never touch those tests, eroding trust in CI and costing review cycles to distinguish a real regression from the flake.

Not a goal

  • Not migrating away from asyncio.run() in test code.
  • Not fixing the underlying production code that creates the cycles.
  • Not covering other flakiness classes (A, C–I).

What changed (motivation → approach → change)

Observed: an unraisable exception with AttributeError: 'PosixPath' has no attribute '_str' appears in the setup of an innocent test in a later module. Root cause: a previous module's asyncio.run() left a BaseSubprocessTransport ↔ protocol ↔ PosixPath reference cycle; CPython deferred collection until the next pass, which fired during a test in the next module on the same xdist worker.

Fix: add a module-scoped autouse=True fixture in the root test/conftest.py that calls gc.collect() twice after each test module ends. This clears cross-module cycles at the module boundary — the known leak shape from both fixed incidents — at ~587 collections per shard instead of ~15 000. Two passes cover chains of two-level cycles (transport → protocol → path).

test_remote_session_client.py keeps its own function-scoped _drain_asyncio_transports fixture to handle within-module transport cycles (tests in the same file can also affect each other). TestDanglingDefaultAgentReset.setUp keeps its gc.collect() × 2 for the same within-module reason. The root module-scoped drain covers the cross-module case; the per-file function-scoped drains cover the within-module case.

Cost

Function-scope drain (this PR's first attempt, reverted): Head e39f771d ran gc.collect() × 2 after every one of ~15 000 tests per shard. This was too expensive on a large heap: Linux shard 1 took 57 min (23:34–00:31; normal 15–36 min); Linux shards 2, 3 and 4 hit the 60-min job cap; Windows hosted shards 1–4 hit the 65-min cap. Run: 40418531613.

Module-scope drain (this head): The root fixture now runs once per test module, ~587 calls per shard instead of ~15 000 (≈26× fewer). Estimated extra time: ~1 ms × 2 × 587 ≈ 1.2 s per shard — negligible. The per-file within-module drains add ~50 and ~26 calls respectively, both small.

Tests

test/test_gc_drain_fixture.py uses pytester to run two inner test modules. Module A creates a self-referential cycle tracked by a weakref and disables automatic GC. The module-scoped _drain_gc_cycles fixture calls gc.collect() twice after module A ends. Module B asserts the weakref is dead, proving the cross-module drain works.

Related Issues

Closes #22436
Closes #22163
Closes #22873

Sibling sweep

Swept: git grep -rn "gc\.collect" test/
Sibling: test/test_remote_session_client.py — keeps its own function-scoped drain for within-module protection
Sibling: test/test_config_loader_default_agent.py — keeps its gc.collect() × 2 in setUp for within-module protection
Siblings: no other cross-module gc.collect() gaps remain

Pattern harvest

Rule candidate: review-prompt
Pattern: root-conftest gc drain for cross-module cycles should be module-scoped, not function-scoped; per-file drains handle within-module cycles

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 26, 2026 22:57
@CrysisDeu
CrysisDeu requested a review from buluoray October 26, 2026 22:57
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ab0fa3ea16351b3ba5365b90b40b6f9d4ee1cdea touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision ab0fa3ea16351b3ba5365b90b40b6f9d4ee1cdea ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ab0fa3ea16351b3ba5365b90b40b6f9d4ee1cdea — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concern blocks this. A module-scoped drain at the module boundary fits the cross-module leak shape, and it costs little.

Notes

  • test_gc_drain_fixture.py never exercises the shipped fixture. It writes its own copy of _drain_gc_cycles into the pytester conftest (pytester.makeconftest("""... def _drain_gc_cycles(): ...""")). So the test only proves that pytest runs module-scoped teardown. If the fixture in root test/conftest.py were deleted or changed, the test would stay green. To pin the real fixture, load the root conftest as a plugin, or assert its scope and autouse flags.
  • The commit/PR title says "drain gc cycles after every test", but the code drains once per module (scope="module"). Retitle it so the history doesn't describe the per-test approach that was reverted.
  • Within-module protection still exists only in the two files already known to leak. Any other module that uses asyncio.run() can still blame its own later tests. That is outside this PR's goal; it's worth a tracking issue so it isn't read as fixed.

[DESIGN-REVIEWED] ab0fa3e

Description read

[DESCRIPTION-READ] 2165f9b7b1124815fb1195bfc799a643fa7ec2b7aa170578915633154fce7bf1

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ab0fa3ea16351b3ba5365b90b40b6f9d4ee1cdea — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • test/test_gc_drain_fixture.py writes its own copy of _drain_gc_cycles into the pytester conftest. It never loads the real fixture in test/conftest.py, so a change to the real fixture would not make it fail.

[OPUS-REVIEWED] ab0fa3e

Verdict parsed from the review's SHA-scoped output markers for commit ab0fa3ea16351b3ba5365b90b40b6f9d4ee1cdea.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ab0fa3ea16351b3ba5365b90b40b6f9d4ee1cdea false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ab0fa3ea16351b3ba5365b90b40b6f9d4ee1cdea and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ab0fa3e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ab0fa3ea16351b3ba5365b90b40b6f9d4ee1cdea false-positive: <reason, 20 to 500 characters>

@CrysisDeu
CrysisDeu force-pushed the unflaky/root-conftest-gc-drain branch from e74d8e0 to e39f771 Compare October 26, 2026 23:15
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Fixed (e39f771): Rewrote test/test_gc_drain_fixture.py as a single self-contained test (test_gc_collect_twice_clears_reference_cycle). It disables the automatic GC, creates a self-referential cycle, verifies the object survives del via refcount alone, then calls gc.collect() twice (the same calls _drain_gc_cycles makes), and asserts the weakref is dead. No module-level state, no ordering dependency — passes when run alone or in any order.

F1 (test/test_gc_drain_fixture.py:39): fixed — single self-contained test, no shared module state, no ordering dependency.

chenmingwei23
chenmingwei23 previously approved these changes Oct 26, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

⚠️ DO NOT MERGE this head (e39f771) — suite slowdown, new head incoming.

The function-scope _drain_gc_cycles fixture runs gc.collect() twice after every one of ~15 000 tests per shard, which is too expensive on a large heap:

Redesigning to module scope (~587 calls per shard instead of ~15 000, ≈26× cheaper). New head will follow shortly.

@CrysisDeu

Copy link
Copy Markdown
Contributor Author

New head 87e40264c7 pushed: fixture scope changed to module (~587 gc.collect() calls per shard instead of ~15 000). Per-file within-module drains restored in test_remote_session_client.py and TestDanglingDefaultAgentReset.setUp. PR body updated with ## Cost section. Re-approve needed on this head.

Add a module-scoped root-conftest autouse fixture that calls gc.collect()
twice after each test module ends, clearing cross-module reference cycles
(BaseSubprocessTransport / PosixPath) before the next module starts on the
same xdist worker.  Module scope (~587 calls per shard) is ~26x cheaper
than function scope (~15000 calls) while covering the known cross-module
leak shape.

Per-file within-module drains are retained:
- _drain_asyncio_transports (function-scoped) in test_remote_session_client.py
- gc.collect() x2 in TestDanglingDefaultAgentReset.setUp

Both cover cycles that occur between tests in the same module, which the
module-scoped root drain does not address.

Add test/test_gc_drain_fixture.py: a pytester-based proof test that runs
two inner modules -- module A leaves a cycle, the module-scoped fixture
drains it, module B asserts the weakref is dead.

Closes #22436
Closes #22163
Closes #22873
@CrysisDeu
CrysisDeu force-pushed the unflaky/root-conftest-gc-drain branch from 87e4026 to ab0fa3e Compare October 27, 2026 01:06
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Fixed (ab0fa3e): Added import gc, save _was_gc_on = gc.isenabled() before pytester.runpytest(), and restore in finally: if _was_gc_on: gc.enable(). GC state is restored even if the inner run raises, so later tests on the same worker always run with GC on.

F1 (test/test_gc_drain_fixture.py:44): fixed — outer test saves and restores gc.isenabled() around the pytester.runpytest() call in a try/finally.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving ab0fa3e: the module-scope drain is correct and the move fixes the cost problem from the previous head. One thing in the description needs fixing before merge, because it would close an issue this does not fix.

Please fix before merge: Closes #22873 is not delivered. The only behaviour change is the module-scoped drain at test/conftest.py:1271. It runs after the last test of a module, so it cannot collect cycles that are created and observed inside one module. The #22873 victim, test_session_capabilities.py::test_dashboard_enrolled_member_controls_through_real_manager[refuses_unsupported], is the 8th test in its file. A per-test probe (gc DEBUG_SAVEALL after each teardown, -n0) shows each of the 7 tests before it in the same file leaving PosixPath objects in cyclic garbage (3, 3, 6, 29, 3, 6, 27), which is the PosixPath has no attribute '_str'/'_drv' signature in the report. The file has no drain of its own, and the second sighting (run 40058245514) was on a head that already had the existing per-file drains from #23466. Either add a within-module drain for this file (or another fix for the leak), with evidence, or change Closes #22873 to Refs #22873.

Please correct the other two closing claims. #22436 is already covered by the setUp drain #23466 added at test/test_config_loader_default_agent.py:287-288; this PR only rewords that comment. #22163 was a logging-handler leak from test_ledger_sweep.py, not a GC cycle, and #23233 fixed it. Closing them is fine, but the description should credit the actual fixes rather than present this fixture as the cure.

Non-blocking: test/test_gc_drain_fixture.py cannot fail on a regression in the shipped fixture. pytester.makeconftest writes its own copy of _drain_gc_cycles, and the inner run loads that copy, not test/conftest.py. With the two gc.collect() calls removed from test/conftest.py:1289-1290 the test still passes; it only goes red when the inline copy is changed. To pin the real fixture, load it into the inner run (for example, register the repo conftest as a plugin or copy its source from test/conftest.py at test time) and confirm it goes red when the root fixture is removed. Ordering is not a concern: the inner run is in-process with no -n, so A always runs before B.

Non-blocking: title and description. The title and commit subject still say "after every test"; the code drains once per module. "Five tests" should be the three tests the linked issues name. The root-cause paragraph asserts a cross-module transport cycle for all three issues but shows it for none.

Cost: no measurable cost. On a 1517-test sample (-n 4, two alternating runs each) base took 113.4 s and 112.3 s, head 110.8 s and 106.5 s. The gap is within run-to-run noise on a shared host.

CI on this head: no failures so far; Backend Tests shards 5-8 (Linux and Windows) were still running when this was checked. Design Review and Opus 5.5 both flagged the inline-copy test and Design flagged the title. Neither has a reply yet.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@KiroCrewCaptain
KiroCrewCaptain merged commit 44a6de1 into main Oct 27, 2026
108 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the unflaky/root-conftest-gc-drain branch October 27, 2026 03:04

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change; a module-scoped GC drain fixture in conftest plus a test for it.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants