Skip to content

test: fail a test that leaves a patched module global behind - #19964

Merged
javenciu merged 1 commit into
mainfrom
test/leak-sentinel-ratchet
Oct 14, 2026
Merged

javenciu merged 1 commit into
mainfrom
test/leak-sentinel-ratchet

Conversation

@javenciu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A test that leaves asyncio.to_thread, time.monotonic, time.sleep or asyncio.sleep rebound fails by name at its own teardown, instead of making unrelated tests fail later on the same worker.

A test can leave a stand-in for asyncio.to_thread installed after it ends, and nothing in the suite notices. The tests that fail are the ones that run later on the same xdist worker and assert that work runs off the event loop, for example in test/test_taskrunner_atomic_persistence.py, test/test_taskrunner_execution_capture_async.py, test/test_unattended_slot_guardrails.py and test/test_wheel_apply.py. The test that left the stand-in passes, and so does a run of its file alone.

The shape that does it looks ordinary. A fixture in the test file holds the attribute through with patch(...), and the test patches the same attribute again with monkeypatch.setattr. _a_shared_monkeypatch_first (test/conftest.py:666) builds the shared monkeypatch before every function-scoped fixture a test file declares. At teardown the fixture's with patch(...) therefore exits first and puts the original back. The test's monkeypatch undo runs second and puts back the value it saw at setattr time, which is the fixture's stand-in. test/test_tag_session.py:953-955 scopes its inner patch to one call for this reason.

Why it matters

A leak like this shows up as a flake in files the change never touched. Under --dist loadgroup, the set of tests sharing a worker with the leaking one changes from run to run ("Order dependence and shared state" in docs/system-specs/common/testing-conventions.md). The red lands on different tests each time and points away from its cause, while a run of the leaking file or of its neighbours passes.

D11 of the determinism contract (docs/system-specs/common/testing-conventions.md:103) has process globals change only through monkeypatch and gives a new process global a floor entry. Every later test reads these four at call time. The rootdir conftest already captures three of them at session start for kiro_crew.testing.wait (_WAIT_CLOCKS, conftest.py:5117), but no floor checks the stdlib attributes after a test.

Not a goal

  • Other module globals the suite patches on the stdlib module (asyncio.create_subprocess_exec, os.getpid, threading.Thread). Each becomes a table entry once a sweep shows that no current test leaves it rebound. This change adds the four that tests read on every off-loop hop, sleep and timer.
  • Changing the fixture order, _floor_monkeypatch, or the shared monkeypatch re-declaration in the rootdir conftest.py.
  • The in-package app suites under src/kiro_crew/apps/builtins. They do not load test/conftest.py, so neither the existing tripwire nor this extension runs there.
  • A static rule for the shape. It spans a fixture and a test body, which the existing per-line semgrep rules do not model.

What changed (motivation → approach → change)

  • Symptom: tests that assert work runs off the event loop fail on a worker after a test that left an inline asyncio.to_thread installed.
  • Root cause: the shared monkeypatch is undone after every function-scoped fixture a test file declares (_a_shared_monkeypatch_first, test/conftest.py:666). A monkeypatch.setattr layered over a fixture's with patch(...) therefore restores the fixture's stand-in, and nothing checks the attribute after the test.
  • Change: the existing teardown tripwire in test/conftest.py checks a second table.
    • LEAK_SENTINELS (test/conftest.py:72) lists asyncio.to_thread, time.monotonic, time.sleep and asyncio.sleep, each with the reason later tests depend on it.
    • _CHECKED_STATE (test/conftest.py:83) is the socket table plus LEAK_SENTINELS. pytest_runtest_setup (test/conftest.py:88) snapshots it. After every fixture has torn down, pytest_runtest_teardown (test/conftest.py:93) compares each entry by identity, puts back what changed, and fails the test with test left asyncio.to_thread rebound (restored) (test/conftest.py:112).
    • The _a_shared_monkeypatch_first docstring (test/conftest.py:677) states the teardown order and the fix: scope the inner patch to the one call with its own with patch(...).
    • docs/system-specs/common/testing-conventions.md:4825 adds the rule to "Order dependence and shared state", and the tripwire bullet at line 2308 names the new table.
  • Cost: one getattr and one identity comparison per entry per test. The restore and the failure run only on a mismatch.
  • Rejected alternatives:
    • An autouse fixture that checks at its own teardown. Autouse fixtures in test/conftest.py are torn down before the shared monkeypatch is undone, so such a fixture would read each test's own monkeypatch.setattr as a leak. The hook runs after all fixtures, which is why the socket check is already a hook (its docstring, test/conftest.py:93).
    • Restore-and-report, as _kiro_crew_package_attr_floor does in the rootdir conftest.py. The restore alone stops the cascade, but the leaking test stays green and nothing prompts a fix.
    • One check at session end. It names no test, and by then every later test on the worker has already failed.

Backwards compatibility

Compatible: the change is test-only. A test that passes on main still passes here unless it leaves one of the four attributes rebound, and no test in the sweep under Tests does.

Removes nothing: the change adds a table, its check and documentation. No fixture, hook, test or key is removed or renamed.

Tests

In short: a new pytester case reproduces the layered patch; it is red with main's test/conftest.py and green here. A sweep of the test files that patch these attributes did not trip the new check.

test/test_socket_tripwire.py::test_a_monkeypatch_over_a_fixture_patch_of_to_thread_fails_that_test_and_is_restored (test/test_socket_tripwire.py:58) runs an inner session with this suite's two hooks. In that session, a fixture holds asyncio.to_thread through with patch(...), the test patches it again with monkeypatch.setattr, and a later test asserts that asyncio.to_thread runs its function off the calling thread. The case expects the leaking test to error at teardown naming asyncio.to_thread, the later test to pass, and asyncio.to_thread to be the original object afterwards.

Red with test/conftest.py from main, 3 of 3 runs:

E         {'passed': 1} != {'passed': 2}
E         {'errors': 0} != {'errors': 1}
E         {'failed': 1} != {'failed': 0}

Green here, 3 of 3 runs: 3 passed for the file. The D12 repeats are also green: 20 of 20 runs at -n0, 10 of 10 at -n 4 beside the four off-loop files above, and one run in reversed order.

  • test_a_monkeypatched_socket_attribute_is_not_a_leak in the same file still passes. It is the control: a patch that monkeypatch undoes is not reported.
  • Leak-after, one process at -n0, with test/test_socket_tripwire.py first and then the four off-loop files and test/test_tag_session.py: 165 passed.
  • At -n 16, the check fired for none of the test/ files that name to_thread, monotonic or sleep in a patch target (files that read the home directory were left out of this local run). The only reds were tests that fail the same way on an unchanged checkout of main on the same machine.
  • At -n 4, the four off-loop files, test/test_tag_session.py and 20 test/ files picked by a seeded shuffle: green.
  • At -n0: test_wait_discipline_ratchet.py, test_flake_pattern_ratchet.py, test_pid_reader_ratchet.py, test_spawn_audit.py, test_kill_chokepoint_ratchet.py and test_host_isolation_floor.py are green. pytest --collect-only test reports no errors.
  • These pass: isort, flake8, scripts/check_black_formatting.py, scripts/check_subprocess_encoding.py, and scripts/check_comment_history.py and scripts/check_brand_name.py against the merge-base, plus scripts/docs-lint.sh. mypy on test/conftest.py reports the same errors as on main and no new one.

Manual verification

This run puts the layered leak first and then the four off-loop files, in one process at -n0, on an unchanged checkout of main and on this branch. The leaking file (not committed):

import asyncio
from unittest.mock import patch

import pytest


@pytest.fixture
def inline_to_thread():
    async def _inline(fn, *args, **kwargs):
        return fn(*args, **kwargs)

    with patch("asyncio.to_thread", side_effect=_inline):
        yield


def test_layers_a_monkeypatch_over_the_fixture(inline_to_thread, monkeypatch):
    async def _recording(fn, *args, **kwargs):
        return fn(*args, **kwargs)

    monkeypatch.setattr(asyncio, "to_thread", _recording)

On main, the leaking test passes and the run ends with 30 failed, 83 passed in 247.76s. The 83 include the leaking test. The failures are 12 in test_taskrunner_execution_capture_async.py, 9 in test_unattended_slot_guardrails.py, 8 in test_wheel_apply.py and 1 in test_taskrunner_atomic_persistence.py.

On this branch the run ends with 113 passed, 1 error in 8.30s. The error is the leaking test: Failed: test left asyncio.to_thread rebound (restored).

A second scratch file repeats the layered shape once for each of the four attributes. Each run errors at teardown naming that attribute, and a test after them finds all four are their original objects again (3 of 3 runs).

Screenshots / video

N/A: the diff touches no user-visible surface. It changes test infrastructure and a spec document.

Related Issues

No linked issue. A search of open and closed issues and pull requests for the symptom and for the mechanism found none that covers this check.

Pattern harvest

Rule candidate: review-prompt
Pattern: a test monkeypatch.setattrs an attribute that a fixture it requests already holds through with patch(...). At runtime the tripwire catches this only for the LEAK_SENTINELS attributes.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

The teardown tripwire in test/conftest.py restores socket state a test
left rebound and fails that test by name. It also checks the
LEAK_SENTINELS table: asyncio.to_thread, time.monotonic, time.sleep
and asyncio.sleep, module globals that later tests read at call time.

The shape that leaks one: _a_shared_monkeypatch_first builds the
shared monkeypatch before every fixture a test file declares, so a
fixture holding an attribute through `with patch(...)` unwinds first
and puts the original back, and a monkeypatch.setattr the test made
over that attribute is undone second and re-installs the fixture's
stand-in for every later test on the worker. With asyncio.to_thread
that runs every later off-loop hop on the event loop: tests in other
files that assert work runs off the loop fail, while a run of the
leaking file alone passes.

The check reads the state after every fixture has torn down, costs a
getattr per entry when nothing leaks, and puts the original back
before failing the test that left it. The fixture's docstring states
the teardown order, testing-conventions.md gains the rule, and
test_socket_tripwire.py pins the layered asyncio.to_thread leak.
@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 13, 2026 21:44
@javenciu
javenciu requested a review from mbriones98 October 13, 2026 21:44
@javenciu

Copy link
Copy Markdown
Contributor Author

Closing and reopening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683).

@javenciu javenciu closed this Oct 13, 2026
@javenciu javenciu reopened this Oct 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 94cb4c4e53ff4698bd2a89d270b5cd255008cb0a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 94cb4c4e53ff4698bd2a89d270b5cd255008cb0a ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 94cb4c4e53ff4698bd2a89d270b5cd255008cb0a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that a full CI run is green with the new check: the author's local sweep skipped test files that read the home directory.

[DESIGN-REVIEWED] 94cb4c4

Description read

[DESCRIPTION-READ] a73da96d9a3c349f84fe69a31e50fdc37e7cbc36277496441f941d58e4f80bdb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 94cb4c4e53ff4698bd2a89d270b5cd255008cb0a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 94cb4c4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 94cb4c4e53ff4698bd2a89d270b5cd255008cb0a: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 94cb4c4e53ff4698bd2a89d270b5cd255008cb0a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 94cb4c4

Verdict parsed from the review's SHA-scoped output markers for commit 94cb4c4e53ff4698bd2a89d270b5cd255008cb0a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 94cb4c4e53ff4698bd2a89d270b5cd255008cb0a: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@javenciu
javenciu enabled auto-merge (squash) October 14, 2026 00:19

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (3 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: extends the test teardown tripwire to restore and report rebound asyncio.to_thread, time.monotonic, time.sleep and asyncio.sleep, with a pytester test; test files plus the testing-conventions note. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/testing-conventions.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants