Skip to content

test(unraisable-leak): drain asyncio transport and PosixPath GC leaks - #23466

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
unflaky/posixpath-unraisable-leak
Oct 23, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
unflaky/posixpath-unraisable-leak

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Fix flaky ExceptionGroup: multiple unraisable exception warnings on shard 7 Linux Backend Tests (3.12).

Two related leaks caused spurious failures attributed to TestDanglingDefaultAgentReset::*, appearing after #19833 merged.

Leak 1 — unclosed BaseSubprocessTransport (test_remote_session_client.py)

test_link_through_a_worker_attach_dial_delivers_and_finishes creates an attach subprocess via command_dial. link.aclose() calls _ProcessWriter.close() which sends SIGTERM but does not await proc.wait(). The asyncio event loop then closes (via asyncio.run()), leaving the _UnixSubprocessTransport ↔ SubprocessStreamProtocol reference cycle unresolved until the next GC pass — which fires during an innocent later test. CPython's BaseSubprocessTransport.__del__ then tries self._loop.call_soon(…) on the closed loop → RuntimeError: Event loop is closed.

Evidence: PytestUnraisableExceptionWarning: Exception ignored in: <function BaseSubprocessTransport.__del__> appeared in all three failing CI runs (jobs 120301033368, 120352662382, 120325593903).

Leak 2 — PosixPath with cleared slots (TestDanglingDefaultAgentReset)

The workspace-aware tests at lines 766+ (first to invoke _workspace_declares) call refresh_materialized_agents(heal_default=True). The GC cycle created during that call (exception tracebacks / LogRecords holding a PosixPath in args) is not immediately collected. When CPython's cyclic collector fires during the next test in the class, tp_clear clears the PosixPath's __slots__ (_raw_paths, _drv, _str). Any finalizer that then calls str(path) gets AttributeError: 'PosixPath' object has no attribute '_str' → cascading AttributeErrors → RecursionError in traceback.format_exception → RuntimeError: Failed to process unraisable exception.

Why it matters

Both leaks cause false test failures on otherwise-green PRs — the failing tests have nothing to do with the code under review. Shard 7 was the victim on two separate branches (unflaky/wait-w2-t, fix/capped-tree-says-nothing-is-broken) after #19833 merged, costing CI reruns and masking real issues.

Not a goal

  • Fixing any pre-existing _workspace_declares GC cycle at its root source. The gc.collect() drain in setUp is sufficient: it ensures the cycle pays its cost between tests instead of during a victim.
  • Changing the production link.py or client.py code. The fix lives entirely in the test file.

What changed (motivation → approach → change)

test/test_remote_session_client.py

Root cause: _ProcessWriter.close() terminates the subprocess but does not await its exit, so asyncio's child-watcher callback (transport._process_exited()) never fires before asyncio.run() closes the loop.

Fix: wrap command_dial in a tracked_dial closure that captures the asyncio.subprocess.Process. After link.aclose() terminates the subprocess, await proc.wait() (capped at 30 s — the wait-discipline ceiling) lets the child-watcher fire and close the transport cleanly before the event loop shuts down.

Regression guard: the new autouse _drain_asyncio_transports fixture calls gc.collect() twice after every test in this module, so any future transport leak is attributed to the correct test immediately.

test/test_config_loader_default_agent.py

Root cause: the workspace-aware setUp-class tests (lines 766+) create a GC cycle involving a PosixPath; the cycle is collected at a random allocation during the next test's body rather than at a deterministic boundary.

Fix: TestDanglingDefaultAgentReset.setUp (run before and after each test via tearDown = setUp) now calls gc.collect() twice. This drains the previous test's cyclic garbage before the new test starts — any remaining unraisable fires here, correctly blamed on the leaker, not a victim.

Backwards compatibility

Compatible: test-only change; no production behaviour changed.

Tests

  • The autouse _drain_asyncio_transports fixture acts as its own regression test: any future asyncio transport leak that goes unclosed is attributed to the correct test in this module instead of a later victim.
  • The gc.collect() in TestDanglingDefaultAgentReset.setUp ensures that if the PosixPath cycle ever resurfaces, it fails at setUp time, identifying the exact preceding leaker test.

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 23, 2026 12:53
@CrysisDeu
CrysisDeu requested a review from bolichen97 October 23, 2026 12:53
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 1e13ff68884d2be1bba6ccabe46173ffcb26dc99 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 1e13ff68884d2be1bba6ccabe46173ffcb26dc99 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 1e13ff68884d2be1bba6ccabe46173ffcb26dc99 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Test-only change. The drain points are deterministic and nothing is taken away.

Notes

  • The root cause is in production: _ProcessWriter.close() in src/kiro_crew/remote_session/client.py:39 sends SIGTERM but never awaits the child. Any production caller that closes its loop after aclose() will leak the same transport. The test works around this rather than fixing it, so it belongs in a follow-up issue.
  • tracked_dial uses hasattr(writer, "_process"), which reaches into a private attribute. If that attribute is renamed, the fix stops doing anything and no error is raised. An assert attach_proc would make that failure loud.

[DESIGN-REVIEWED] 1e13ff6

Description read

[DESCRIPTION-READ] 95c9ee5c32a2d44b6edebfc45156ed41c494a0e93e38320214a44d5d53a3ddc9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 1e13ff68884d2be1bba6ccabe46173ffcb26dc99 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1e13ff6

Verdict parsed from the review's SHA-scoped output markers for commit 1e13ff68884d2be1bba6ccabe46173ffcb26dc99.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1e13ff68884d2be1bba6ccabe46173ffcb26dc99: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 1e13ff68884d2be1bba6ccabe46173ffcb26dc99 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1e13ff6

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1e13ff68884d2be1bba6ccabe46173ffcb26dc99: <one-sentence reason>

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for chasing these two flakes. I reviewed head 332414e.

What I verified locally on Python 3.12:

  1. Leak 1 is real and this change fixes it in the test. On main, running test_link_through_a_worker_attach_dial_delivers_and_finishes under python -X dev with ResourceWarning enabled reports an unclosed _UnixSubprocessTransport, "subprocess is still running", and a PytestUnraisableExceptionWarning from BaseSubprocessTransport.__del__ with RuntimeError: Event loop is closed on every run (18 of 18 under parallel load). With this PR applied the same command reports none of these in 5 of 5 runs, and both touched modules pass with unraisable warnings promoted to errors (54 passed).
  2. Nothing is swallowed. The new gc.collect() calls only move collection to a test boundary; any unraisable still goes through the pytest hook and still fails the run. The change is limited to two test files.
  3. Leak 2 is mitigated, not fixed. Details inline. I could not reproduce the shard 7 failure in isolation (the class passes and KiroCrewConfig.load never nests when the class runs alone), so I cannot confirm the new collections stop it.

On the automated design notes: the point that the description claims a tearDown drain the diff lacks does not hold, because the class already has tearDown = setUp (line 284), so the drain also runs after each test. That means a finalizer error is now reported in the teardown of the test that left the garbage, which is the attribution you wanted. Its point that the production close path terminates the child without awaiting it is accurate; that is a fair follow up, not something this test fix needs.

Two non blocking suggestions inline. Merge is still waiting on the readiness signal and the backend test shards for this head.

# only collected during *this* test's body the unraisable exception is
# blamed on the innocent victim here rather than on the actual leaker.
# Two passes cover chains of cycles.
gc.collect()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The drain lowers the odds of the shard 7 flake but does not remove its cause

The comment says the PosixPath errors come from a reference cycle that the workspace tests leave behind. The recorded CI signature points elsewhere: both sub exceptions are RecursionError raised inside the pytest unraisable hook while it formats a traceback, and the PosixPath that fails is half initialised (no _str, then no _drv). That matches the open report that KiroCrewConfig.load can re enter itself through current_context() until the recursion limit, with the RecursionError swallowed by safe_context_call (#21089). Any finalizer that runs near that depth has no stack left to format its error.

Collecting in setUp and tearDown runs pending finalizers at a shallow depth, so fewer of them land at the bottom of that recursion. A collection triggered by allocation inside the deep recursion during a test body can still hit it, so the flake can still recur.

Suggestion: reword the comment to say this is a mitigation and point at #21089, and keep the flake tracking issue open until the re entry is fixed.

Rule: a comment should describe the mechanism the evidence shows, so the next reader does not treat the flake as solved.


async def tracked_dial():
reader, writer = await base_dial()
if hasattr(writer, "_process"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 If the private attribute is renamed, this fix turns itself off without a failure

if hasattr(writer, "_process"):
    attach_proc.append(writer._process)

The wait in the finally block only runs when this captured a process. If _ProcessWriter ever renames _process, hasattr returns False, the list stays empty, the test still passes, and the unclosed transport leak returns to whichever later test the collector lands in.

Suggestion: drop the hasattr guard so a rename fails here:

attach_proc.append(writer._process)

A longer term option is to have _ProcessWriter.close or RemoteLink.aclose wait for the child, which would let this test drop the private access entirely.

Rule: a test guard should fail loudly when the thing it guards disappears.

@CrysisDeu
CrysisDeu force-pushed the unflaky/posixpath-unraisable-leak branch from 332414e to a1f728f Compare October 23, 2026 14:12

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at head a1f728f. Compared with 332414e, the two files this PR changes are byte for byte identical; the new head is a rebase onto main and picks up no change of its own.

So the two points from my earlier review still apply, and both inline comments are already attached to this head:

  1. The setUp drain in test/test_config_loader_default_agent.py mitigates the shard 7 flake rather than fixing its cause. Please reword the comment to say so, point at #21089, and keep the flake tracking issue open. The Design and First Principles reviews on this head reach the same conclusion.
  2. The hasattr(writer, "_process") guard in test/test_remote_session_client.py turns the fix off without any failure if the private attribute is renamed. Please drop the guard so a rename fails in this test. The Design review on this head flags the same line.

One CI note: Backend Tests (Windows) shard 4 is red on this head because test/test_session_memory_prefetch.py::test_the_join_ends_at_the_build_embed_deadline missed a 0.05 second timing bound. That test is outside this PR and the remote session module is skipped on Windows, so a rerun should clear it. Linux shards 6 to 8 and the readiness signal are still pending.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/posixpath-unraisable-leak branch from a1f728f to 1e13ff6 Compare October 23, 2026 16:22
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at head 1e13ff6. Compared with a1f728f, the two files this PR changes are byte for byte identical. The new head is a rebase onto main. It now includes the fix for the Windows timing test that made shard 4 red on the previous head.

The two points from my earlier review still apply. Both inline comments are still attached to this head:

  1. The setUp drain in test/test_config_loader_default_agent.py mitigates the shard 7 flake but does not fix its cause. Please reword the comment to say so, point at #21089, and keep the flake tracking issue open.
  2. The hasattr(writer, "_process") guard in test/test_remote_session_client.py turns the fix off without a failure if the private attribute is renamed. Please drop the guard, or assert that attach_proc is not empty, so a rename fails in this test. The Design review on this head flags the same line.

CI note: Linux shards and readiness are still pending on this head.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 23, 2026 19:03

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change that awaits the attach subprocess and collects garbage between tests, so an unraisable asyncio transport or PosixPath leak is reported on the test that caused it.

@KiroCrewCaptain
KiroCrewCaptain merged commit 680291f into main Oct 23, 2026
153 of 158 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the unflaky/posixpath-unraisable-leak branch October 23, 2026 19:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 23, 2026
CrysisDeu added a commit that referenced this pull request Oct 24, 2026
Two tests call asyncio.run() with a DashboardState whose ConversationLog
stores tmp_path as a PosixPath.  When the loop closes, pending asyncio
objects that reference the state form reference cycles with the PosixPath.
CPython's cyclic collector defers those cycles until a later GC pass that
can land inside an unrelated test; tp_clear blanks the path's slots
(_drv, _str) before __del__ fires, raising AttributeError inside
sys.unraisablehook and blaming the innocent victim
(test_config_loader_default_agent.py::TestDanglingDefaultAgentReset).

Fix mirrors the pattern from test_remote_session_client.py (PR #23466):
add an autouse gc.collect() × 2 fixture that runs AFTER each test in this
module, paying the GC cost in the right place.
KiroCrewCaptain pushed a commit that referenced this pull request Oct 24, 2026
…23608)

Two tests call asyncio.run() with a DashboardState whose ConversationLog
stores tmp_path as a PosixPath.  When the loop closes, pending asyncio
objects that reference the state form reference cycles with the PosixPath.
CPython's cyclic collector defers those cycles until a later GC pass that
can land inside an unrelated test; tp_clear blanks the path's slots
(_drv, _str) before __del__ fires, raising AttributeError inside
sys.unraisablehook and blaming the innocent victim
(test_config_loader_default_agent.py::TestDanglingDefaultAgentReset).

Fix mirrors the pattern from test_remote_session_client.py (PR #23466):
add an autouse gc.collect() × 2 fixture that runs AFTER each test in this
module, paying the GC cost in the right place.

Co-authored-by: CrysisDeu <crysisDeu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants