Repository navigation
test: fail a test that leaves a patched module global behind - #19964
Conversation
The teardown tripwire in test/conftest.py restores socket state a test left rebound and fails that test by name. It also checks the LEAK_SENTINELS table: asyncio.to_thread, time.monotonic, time.sleep and asyncio.sleep, module globals that later tests read at call time. The shape that leaks one: _a_shared_monkeypatch_first builds the shared monkeypatch before every fixture a test file declares, so a fixture holding an attribute through `with patch(...)` unwinds first and puts the original back, and a monkeypatch.setattr the test made over that attribute is undone second and re-installs the fixture's stand-in for every later test on the worker. With asyncio.to_thread that runs every later off-loop hop on the event loop: tests in other files that assert work runs off the loop fail, while a run of the leaking file alone passes. The check reads the state after every fixture has torn down, costs a getattr per entry when nothing leaks, and puts the original back before failing the test that left it. The fixture's docstring states the teardown order, testing-conventions.md gains the rule, and test_socket_tripwire.py pins the layered asyncio.to_thread leak.
|
Closing and reopening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683). |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that a full CI run is green with the new check: the author's local sweep skipped test files that read the home directory. [DESIGN-REVIEWED] 94cb4c4 Description read[DESCRIPTION-READ] a73da96d9a3c349f84fe69a31e50fdc37e7cbc36277496441f941d58e4f80bdb That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (3 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: extends the test teardown tripwire to restore and report rebound asyncio.to_thread, time.monotonic, time.sleep and asyncio.sleep, with a pytester test; test files plus the testing-conventions note. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/testing-conventions.md.
Problem / Motivation
Goal: A test that leaves
asyncio.to_thread,time.monotonic,time.sleeporasyncio.sleeprebound fails by name at its own teardown, instead of making unrelated tests fail later on the same worker.A test can leave a stand-in for
asyncio.to_threadinstalled after it ends, and nothing in the suite notices. The tests that fail are the ones that run later on the same xdist worker and assert that work runs off the event loop, for example intest/test_taskrunner_atomic_persistence.py,test/test_taskrunner_execution_capture_async.py,test/test_unattended_slot_guardrails.pyandtest/test_wheel_apply.py. The test that left the stand-in passes, and so does a run of its file alone.The shape that does it looks ordinary. A fixture in the test file holds the attribute through
with patch(...), and the test patches the same attribute again withmonkeypatch.setattr._a_shared_monkeypatch_first(test/conftest.py:666) builds the sharedmonkeypatchbefore every function-scoped fixture a test file declares. At teardown the fixture'swith patch(...)therefore exits first and puts the original back. The test'smonkeypatchundo runs second and puts back the value it saw atsetattrtime, which is the fixture's stand-in.test/test_tag_session.py:953-955scopes its inner patch to one call for this reason.Why it matters
A leak like this shows up as a flake in files the change never touched. Under
--dist loadgroup, the set of tests sharing a worker with the leaking one changes from run to run ("Order dependence and shared state" indocs/system-specs/common/testing-conventions.md). The red lands on different tests each time and points away from its cause, while a run of the leaking file or of its neighbours passes.D11 of the determinism contract (
docs/system-specs/common/testing-conventions.md:103) has process globals change only throughmonkeypatchand gives a new process global a floor entry. Every later test reads these four at call time. The rootdir conftest already captures three of them at session start forkiro_crew.testing.wait(_WAIT_CLOCKS,conftest.py:5117), but no floor checks the stdlib attributes after a test.Not a goal
asyncio.create_subprocess_exec,os.getpid,threading.Thread). Each becomes a table entry once a sweep shows that no current test leaves it rebound. This change adds the four that tests read on every off-loop hop, sleep and timer._floor_monkeypatch, or the sharedmonkeypatchre-declaration in the rootdirconftest.py.src/kiro_crew/apps/builtins. They do not loadtest/conftest.py, so neither the existing tripwire nor this extension runs there.What changed (motivation → approach → change)
asyncio.to_threadinstalled.monkeypatchis undone after every function-scoped fixture a test file declares (_a_shared_monkeypatch_first,test/conftest.py:666). Amonkeypatch.setattrlayered over a fixture'swith patch(...)therefore restores the fixture's stand-in, and nothing checks the attribute after the test.test/conftest.pychecks a second table.LEAK_SENTINELS(test/conftest.py:72) listsasyncio.to_thread,time.monotonic,time.sleepandasyncio.sleep, each with the reason later tests depend on it._CHECKED_STATE(test/conftest.py:83) is the socket table plusLEAK_SENTINELS.pytest_runtest_setup(test/conftest.py:88) snapshots it. After every fixture has torn down,pytest_runtest_teardown(test/conftest.py:93) compares each entry by identity, puts back what changed, and fails the test withtest left asyncio.to_thread rebound (restored)(test/conftest.py:112)._a_shared_monkeypatch_firstdocstring (test/conftest.py:677) states the teardown order and the fix: scope the inner patch to the one call with its ownwith patch(...).docs/system-specs/common/testing-conventions.md:4825adds the rule to "Order dependence and shared state", and the tripwire bullet at line 2308 names the new table.getattrand one identity comparison per entry per test. The restore and the failure run only on a mismatch.test/conftest.pyare torn down before the sharedmonkeypatchis undone, so such a fixture would read each test's ownmonkeypatch.setattras a leak. The hook runs after all fixtures, which is why the socket check is already a hook (its docstring,test/conftest.py:93)._kiro_crew_package_attr_floordoes in the rootdirconftest.py. The restore alone stops the cascade, but the leaking test stays green and nothing prompts a fix.Backwards compatibility
Compatible: the change is test-only. A test that passes on main still passes here unless it leaves one of the four attributes rebound, and no test in the sweep under Tests does.
Removes nothing: the change adds a table, its check and documentation. No fixture, hook, test or key is removed or renamed.
Tests
In short: a new pytester case reproduces the layered patch; it is red with main's
test/conftest.pyand green here. A sweep of the test files that patch these attributes did not trip the new check.test/test_socket_tripwire.py::test_a_monkeypatch_over_a_fixture_patch_of_to_thread_fails_that_test_and_is_restored(test/test_socket_tripwire.py:58) runs an inner session with this suite's two hooks. In that session, a fixture holdsasyncio.to_threadthroughwith patch(...), the test patches it again withmonkeypatch.setattr, and a later test asserts thatasyncio.to_threadruns its function off the calling thread. The case expects the leaking test to error at teardown namingasyncio.to_thread, the later test to pass, andasyncio.to_threadto be the original object afterwards.Red with
test/conftest.pyfrom main, 3 of 3 runs:Green here, 3 of 3 runs:
3 passedfor the file. The D12 repeats are also green: 20 of 20 runs at-n0, 10 of 10 at-n 4beside the four off-loop files above, and one run in reversed order.test_a_monkeypatched_socket_attribute_is_not_a_leakin the same file still passes. It is the control: a patch thatmonkeypatchundoes is not reported.-n0, withtest/test_socket_tripwire.pyfirst and then the four off-loop files andtest/test_tag_session.py:165 passed.-n 16, the check fired for none of thetest/files that nameto_thread,monotonicorsleepin a patch target (files that read the home directory were left out of this local run). The only reds were tests that fail the same way on an unchanged checkout of main on the same machine.-n 4, the four off-loop files,test/test_tag_session.pyand 20test/files picked by a seeded shuffle: green.-n0:test_wait_discipline_ratchet.py,test_flake_pattern_ratchet.py,test_pid_reader_ratchet.py,test_spawn_audit.py,test_kill_chokepoint_ratchet.pyandtest_host_isolation_floor.pyare green.pytest --collect-only testreports no errors.scripts/check_black_formatting.py,scripts/check_subprocess_encoding.py, andscripts/check_comment_history.pyandscripts/check_brand_name.pyagainst the merge-base, plusscripts/docs-lint.sh. mypy ontest/conftest.pyreports the same errors as on main and no new one.Manual verification
This run puts the layered leak first and then the four off-loop files, in one process at
-n0, on an unchanged checkout of main and on this branch. The leaking file (not committed):On main, the leaking test passes and the run ends with
30 failed, 83 passedin 247.76s. The 83 include the leaking test. The failures are 12 intest_taskrunner_execution_capture_async.py, 9 intest_unattended_slot_guardrails.py, 8 intest_wheel_apply.pyand 1 intest_taskrunner_atomic_persistence.py.On this branch the run ends with
113 passed, 1 errorin 8.30s. The error is the leaking test:Failed: test left asyncio.to_thread rebound (restored).A second scratch file repeats the layered shape once for each of the four attributes. Each run errors at teardown naming that attribute, and a test after them finds all four are their original objects again (3 of 3 runs).
Screenshots / video
N/A: the diff touches no user-visible surface. It changes test infrastructure and a spec document.
Related Issues
No linked issue. A search of open and closed issues and pull requests for the symptom and for the mechanism found none that covers this check.
Pattern harvest
Rule candidate: review-prompt
Pattern: a test
monkeypatch.setattrs an attribute that a fixture it requests already holds throughwith patch(...). At runtime the tripwire catches this only for theLEAK_SENTINELSattributes.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.