Repository navigation
test(unraisable-leak): drain asyncio transport and PosixPath GC leaks - #23466
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Test-only change. The drain points are deterministic and nothing is taken away. Notes
[DESIGN-REVIEWED] 1e13ff6 Description read[DESCRIPTION-READ] 95c9ee5c32a2d44b6edebfc45156ed41c494a0e93e38320214a44d5d53a3ddc9 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for chasing these two flakes. I reviewed head 332414e.
What I verified locally on Python 3.12:
- Leak 1 is real and this change fixes it in the test. On main, running
test_link_through_a_worker_attach_dial_delivers_and_finishesunderpython -X devwith ResourceWarning enabled reports an unclosed_UnixSubprocessTransport, "subprocess is still running", and aPytestUnraisableExceptionWarningfromBaseSubprocessTransport.__del__withRuntimeError: Event loop is closedon every run (18 of 18 under parallel load). With this PR applied the same command reports none of these in 5 of 5 runs, and both touched modules pass with unraisable warnings promoted to errors (54 passed). - Nothing is swallowed. The new
gc.collect()calls only move collection to a test boundary; any unraisable still goes through the pytest hook and still fails the run. The change is limited to two test files. - Leak 2 is mitigated, not fixed. Details inline. I could not reproduce the shard 7 failure in isolation (the class passes and
KiroCrewConfig.loadnever nests when the class runs alone), so I cannot confirm the new collections stop it.
On the automated design notes: the point that the description claims a tearDown drain the diff lacks does not hold, because the class already has tearDown = setUp (line 284), so the drain also runs after each test. That means a finalizer error is now reported in the teardown of the test that left the garbage, which is the attribution you wanted. Its point that the production close path terminates the child without awaiting it is accurate; that is a fair follow up, not something this test fix needs.
Two non blocking suggestions inline. Merge is still waiting on the readiness signal and the backend test shards for this head.
| # only collected during *this* test's body the unraisable exception is | ||
| # blamed on the innocent victim here rather than on the actual leaker. | ||
| # Two passes cover chains of cycles. | ||
| gc.collect() |
There was a problem hiding this comment.
🟡 The drain lowers the odds of the shard 7 flake but does not remove its cause
The comment says the PosixPath errors come from a reference cycle that the workspace tests leave behind. The recorded CI signature points elsewhere: both sub exceptions are RecursionError raised inside the pytest unraisable hook while it formats a traceback, and the PosixPath that fails is half initialised (no _str, then no _drv). That matches the open report that KiroCrewConfig.load can re enter itself through current_context() until the recursion limit, with the RecursionError swallowed by safe_context_call (#21089). Any finalizer that runs near that depth has no stack left to format its error.
Collecting in setUp and tearDown runs pending finalizers at a shallow depth, so fewer of them land at the bottom of that recursion. A collection triggered by allocation inside the deep recursion during a test body can still hit it, so the flake can still recur.
Suggestion: reword the comment to say this is a mitigation and point at #21089, and keep the flake tracking issue open until the re entry is fixed.
Rule: a comment should describe the mechanism the evidence shows, so the next reader does not treat the flake as solved.
|
|
||
| async def tracked_dial(): | ||
| reader, writer = await base_dial() | ||
| if hasattr(writer, "_process"): |
There was a problem hiding this comment.
🟡 If the private attribute is renamed, this fix turns itself off without a failure
if hasattr(writer, "_process"):
attach_proc.append(writer._process)The wait in the finally block only runs when this captured a process. If _ProcessWriter ever renames _process, hasattr returns False, the list stays empty, the test still passes, and the unclosed transport leak returns to whichever later test the collector lands in.
Suggestion: drop the hasattr guard so a rename fails here:
attach_proc.append(writer._process)A longer term option is to have _ProcessWriter.close or RemoteLink.aclose wait for the child, which would let this test drop the private access entirely.
Rule: a test guard should fail loudly when the thing it guards disappears.
332414e to
a1f728f
Compare
bolichen97
left a comment
There was a problem hiding this comment.
I looked at head a1f728f. Compared with 332414e, the two files this PR changes are byte for byte identical; the new head is a rebase onto main and picks up no change of its own.
So the two points from my earlier review still apply, and both inline comments are already attached to this head:
- The
setUpdrain intest/test_config_loader_default_agent.pymitigates the shard 7 flake rather than fixing its cause. Please reword the comment to say so, point at #21089, and keep the flake tracking issue open. The Design and First Principles reviews on this head reach the same conclusion. - The
hasattr(writer, "_process")guard intest/test_remote_session_client.pyturns the fix off without any failure if the private attribute is renamed. Please drop the guard so a rename fails in this test. The Design review on this head flags the same line.
One CI note: Backend Tests (Windows) shard 4 is red on this head because test/test_session_memory_prefetch.py::test_the_join_ends_at_the_build_embed_deadline missed a 0.05 second timing bound. That test is outside this PR and the remote session module is skipped on Windows, so a rerun should clear it. Linux shards 6 to 8 and the readiness signal are still pending.
a1f728f to
1e13ff6
Compare
bolichen97
left a comment
There was a problem hiding this comment.
I looked at head 1e13ff6. Compared with a1f728f, the two files this PR changes are byte for byte identical. The new head is a rebase onto main. It now includes the fix for the Windows timing test that made shard 4 red on the previous head.
The two points from my earlier review still apply. Both inline comments are still attached to this head:
- The
setUpdrain intest/test_config_loader_default_agent.pymitigates the shard 7 flake but does not fix its cause. Please reword the comment to say so, point at #21089, and keep the flake tracking issue open. - The
hasattr(writer, "_process")guard intest/test_remote_session_client.pyturns the fix off without a failure if the private attribute is renamed. Please drop the guard, or assert thatattach_procis not empty, so a rename fails in this test. The Design review on this head flags the same line.
CI note: Linux shards and readiness are still pending on this head.
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: test-only change that awaits the attach subprocess and collects garbage between tests, so an unraisable asyncio transport or PosixPath leak is reported on the test that caused it.
Two tests call asyncio.run() with a DashboardState whose ConversationLog stores tmp_path as a PosixPath. When the loop closes, pending asyncio objects that reference the state form reference cycles with the PosixPath. CPython's cyclic collector defers those cycles until a later GC pass that can land inside an unrelated test; tp_clear blanks the path's slots (_drv, _str) before __del__ fires, raising AttributeError inside sys.unraisablehook and blaming the innocent victim (test_config_loader_default_agent.py::TestDanglingDefaultAgentReset). Fix mirrors the pattern from test_remote_session_client.py (PR #23466): add an autouse gc.collect() × 2 fixture that runs AFTER each test in this module, paying the GC cost in the right place.
…23608) Two tests call asyncio.run() with a DashboardState whose ConversationLog stores tmp_path as a PosixPath. When the loop closes, pending asyncio objects that reference the state form reference cycles with the PosixPath. CPython's cyclic collector defers those cycles until a later GC pass that can land inside an unrelated test; tp_clear blanks the path's slots (_drv, _str) before __del__ fires, raising AttributeError inside sys.unraisablehook and blaming the innocent victim (test_config_loader_default_agent.py::TestDanglingDefaultAgentReset). Fix mirrors the pattern from test_remote_session_client.py (PR #23466): add an autouse gc.collect() × 2 fixture that runs AFTER each test in this module, paying the GC cost in the right place. Co-authored-by: CrysisDeu <crysisDeu@users.noreply.github.com>
Problem / Motivation
Goal: Fix flaky
ExceptionGroup: multiple unraisable exception warningson shard 7 Linux Backend Tests (3.12).Two related leaks caused spurious failures attributed to
TestDanglingDefaultAgentReset::*, appearing after #19833 merged.Leak 1 — unclosed
BaseSubprocessTransport(test_remote_session_client.py)test_link_through_a_worker_attach_dial_delivers_and_finishescreates an attach subprocess viacommand_dial.link.aclose()calls_ProcessWriter.close()which sends SIGTERM but does notawait proc.wait(). The asyncio event loop then closes (viaasyncio.run()), leaving the_UnixSubprocessTransport ↔ SubprocessStreamProtocolreference cycle unresolved until the next GC pass — which fires during an innocent later test. CPython'sBaseSubprocessTransport.__del__then triesself._loop.call_soon(…)on the closed loop →RuntimeError: Event loop is closed.Evidence:
PytestUnraisableExceptionWarning: Exception ignored in: <function BaseSubprocessTransport.__del__>appeared in all three failing CI runs (jobs 120301033368, 120352662382, 120325593903).Leak 2 — PosixPath with cleared slots (
TestDanglingDefaultAgentReset)The workspace-aware tests at lines 766+ (first to invoke
_workspace_declares) callrefresh_materialized_agents(heal_default=True). The GC cycle created during that call (exception tracebacks / LogRecords holding a PosixPath inargs) is not immediately collected. When CPython's cyclic collector fires during the next test in the class,tp_clearclears the PosixPath's__slots__(_raw_paths,_drv,_str). Any finalizer that then callsstr(path)getsAttributeError: 'PosixPath' object has no attribute '_str'→ cascading AttributeErrors →RecursionErrorintraceback.format_exception→RuntimeError: Failed to process unraisable exception.Why it matters
Both leaks cause false test failures on otherwise-green PRs — the failing tests have nothing to do with the code under review. Shard 7 was the victim on two separate branches (unflaky/wait-w2-t, fix/capped-tree-says-nothing-is-broken) after #19833 merged, costing CI reruns and masking real issues.
Not a goal
_workspace_declaresGC cycle at its root source. Thegc.collect()drain insetUpis sufficient: it ensures the cycle pays its cost between tests instead of during a victim.link.pyorclient.pycode. The fix lives entirely in the test file.What changed (motivation → approach → change)
test/test_remote_session_client.py
Root cause:
_ProcessWriter.close()terminates the subprocess but does not await its exit, so asyncio's child-watcher callback (transport._process_exited()) never fires beforeasyncio.run()closes the loop.Fix: wrap
command_dialin atracked_dialclosure that captures theasyncio.subprocess.Process. Afterlink.aclose()terminates the subprocess,await proc.wait()(capped at 30 s — the wait-discipline ceiling) lets the child-watcher fire and close the transport cleanly before the event loop shuts down.Regression guard: the new autouse
_drain_asyncio_transportsfixture callsgc.collect()twice after every test in this module, so any future transport leak is attributed to the correct test immediately.test/test_config_loader_default_agent.py
Root cause: the workspace-aware
setUp-class tests (lines 766+) create a GC cycle involving a PosixPath; the cycle is collected at a random allocation during the next test's body rather than at a deterministic boundary.Fix:
TestDanglingDefaultAgentReset.setUp(run before and after each test viatearDown = setUp) now callsgc.collect()twice. This drains the previous test's cyclic garbage before the new test starts — any remaining unraisable fires here, correctly blamed on the leaker, not a victim.Backwards compatibility
Compatible: test-only change; no production behaviour changed.
Tests
_drain_asyncio_transportsfixture acts as its own regression test: any future asyncio transport leak that goes unclosed is attributed to the correct test in this module instead of a later victim.gc.collect()inTestDanglingDefaultAgentReset.setUpensures that if the PosixPath cycle ever resurfaces, it fails at setUp time, identifying the exact preceding leaker test.