Skip to content

test(acp): give the root-tracking hop tests their own subprocess pool - #35627

Merged
bolichen97 merged 1 commit into
mainfrom
fix/35142-spawn-hop-wait
Nov 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/35142-spawn-hop-wait

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: the root-tracking hop tests in TestRuntimeRootTrackingOffLoop pass whatever an earlier test on the same xdist worker left running on the shared subprocess pool.

test_a_second_cancel_during_the_cleanup_still_reaps_exactly_once failed on CI (Backend Tests (3.12, 6)) in a PR that touches nothing it reads: AssertionError: the event had not happened after 60.06s (lost-run bound 60.00s); it is still clear. It failed at its first wait, so spawn() never reached the tracking hop the test parks.

Why it matters

A red Backend Tests shard on an unrelated PR costs a rerun and a round of triage, and the sibling test in the same class has the same shape.

Not a goal

  • No change to spawn(), its reap guard, or the 60 s lost-run bound.
  • Not hunting the earlier test whose leftover pool work this class waited behind; the conftest shared-pool floor already reports those by name.
  • Not isolating other tests that spawn through the shared pool; none of them waits on a hop event under a lost-run bound.

What changed (motivation → approach → change)

The three tests in TestRuntimeRootTrackingOffLoop no longer wait behind work that other tests left on the shared subprocess pool.

The tests wait for spawn() to reach its root-tracking hop. Before that hop, spawn() runs its resume step on subprocess_executor(), the process-wide eight-worker pool that every test on an xdist worker shares. When earlier tests still hold all eight workers, the resume step queues, the tracking hop is never reached, and the wait fails after 60 s with nothing to say why.

Each test in TestRuntimeRootTrackingOffLoop now runs spawn() against its own one-worker pool, through an autouse fixture that patches subprocess_executor for the class. It is the same seam test_failed_spawn_unwinds_live_tree_and_tracking in this file already uses. The tests' assertions are unchanged.

Backwards compatibility

Compatible: test-only change; no product code is touched.

Removes nothing: the diff adds one test fixture.

Tests

test/test_acp_spawn_offload.py::TestRuntimeRootTrackingOffLoop: all three tests run spawn() on a private pool; what each one pins is unchanged.

Determinism proof (testing-conventions, Proving a determinism fix):

Run Parent 758e63ca5b97 This head a1edd914c73b
Forced condition: a throwaway -p plugin holds all eight subprocess_executor workers for each test of the class (-n0 --timeout=20, so the lost-run bound is 10 s) 3 failed; both hop tests with the event had not happened after 10.01s (lost-run bound 10.00s); it is still clear 3 passed
Repeats, -n0 20/20 green
Repeats, -n 4 --dist loadgroup --max-worker-restart=2 with test_acp_offload.py and test_acp_spawn_label.py as co-tenants 10/10 green
Shuffled order, test_acp_spawn_offload.py + test_acp_spawn_label.py (seed 29799, 71 ids) 71 passed
test/test_flake_pattern_ratchet.py 68 passed

Proof: not provable -- test-only change

Manual verification

N/A — test-only change; the forced-condition runs above are the verification.

Screenshots / video

N/A — no user-facing surface changes.

Related Issues

Closes #35142

Sibling sweep

Swept: git grep -nE "runtime.spawn()|client._spawn()|await_until(" -- test/test_acp_spawn_offload.py
Sibling: test/test_acp_spawn_offload.py:1638 -- changed here
Sibling: test/test_acp_spawn_offload.py:1696 -- changed here
Sibling: test/test_acp_spawn_offload.py:1375 -- not affected -- already runs spawn() on its own one-worker pool
Sibling: test/test_acp_spawn_offload.py:557 -- tracked in #35771
Sibling: test/test_acp_spawn_offload.py:1489 -- not affected -- awaits spawn() itself with no hop event under a lost-run bound
Sibling: test/test_acp_spawn_offload.py:1532 -- not affected -- awaits spawn() itself with no hop event under a lost-run bound
Sibling: test/test_acp_spawn_offload.py:358 -- not affected -- awaits spawn() itself with no hop event under a lost-run bound

Pattern harvest

Not generalizable: the class is already watched; the conftest shared-pool floor names every test that leaves work on a process-wide pool, and this PR removes one waiting test's dependence on that pool.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Forced-condition evidence for the determinism proof in the description: the throwaway plugin (never committed) and the commands.

"""Throwaway forced condition for #35142 (never committed).

Before each TestRuntimeRootTrackingOffLoop test runs, hold every worker of the
process-wide ``subprocess_executor`` -- the state an earlier test on the same xdist
worker can leave behind when its own pool jobs outlive it. Released at teardown.
"""

import threading

import pytest

_held = {}


@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
    import os

    if os.environ.get("FORCE_SCOPE", "TestRuntimeRootTrackingOffLoop") not in item.nodeid:
        yield
        return
    from kiro_crew import executors

    pool = executors.subprocess_executor()
    release = threading.Event()
    started = threading.Barrier(pool._max_workers + 1)

    def hold():
        started.wait(30)
        release.wait(300)

    futures = [pool.submit(hold) for _ in range(pool._max_workers)]
    started.wait(30)
    print(f"\n[forced] holding all {pool._max_workers} subprocess_executor workers")
    try:
        yield
    finally:
        release.set()
        for f in futures:
            f.result(timeout=30)
# on the parent ee7a026dd093 and on this head 2723e9a1566b
PYTHONPATH=<dir holding force_pool_full.py> python -m pytest -n0 -p no:cacheprovider -p force_pool_full --timeout=20 -q -rA "test/test_acp_spawn_offload.py::TestRuntimeRootTrackingOffLoop"

Parent: 3 failed, both hop tests with AssertionError: the event had not happened after 10.01s (lost-run bound 10.00s); it is still clear. This head: 3 passed.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a1edd914c73b1654b710e9b91c465b76f30b7ec4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision a1edd914c73b1654b710e9b91c465b76f30b7ec4 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a1edd914c73b1654b710e9b91c465b76f30b7ec4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this test-only fixture removes the class's dependence on the shared pool, using the seam the file's sibling test already relies on, and the facade mirror carries the patch through to runtime_spawn_launch.

[DESIGN-REVIEWED] a1edd91

Description read

[DESCRIPTION-READ] 9f87c446ea24ad97fff90b599a00f5f75e451e34128345f046fc2c8c99377835

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a1edd914c73b1654b710e9b91c465b76f30b7ec4 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a1edd91

Verdict parsed from the review's SHA-scoped output markers for commit a1edd914c73b1654b710e9b91c465b76f30b7ec4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a1edd914c73b1654b710e9b91c465b76f30b7ec4 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a1edd914c73b1654b710e9b91c465b76f30b7ec4 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a1edd91

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a1edd914c73b1654b710e9b91c465b76f30b7ec4 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 11, 2026 22:48
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 11, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on 2723e9a. This PR gives the three root-tracking hop tests in TestRuntimeRootTrackingOffLoop their own one-worker thread pool. Before, spawn() used the subprocess pool that every test on the same xdist worker shares, so an earlier test that still held those workers could keep spawn from ever reaching the tracking hop. The change is nine lines in one test file and touches no product code. It removes no assertion, adds no sleep or timeout, and uses the same pool seam that the sibling test at test/test_acp_spawn_offload.py:1325 already uses. All 56 tests in the file pass, both serially and under xdist. Breaking the product guard (running _track_root_pids on the event loop thread) turns all three tests red. Making the new private pool raise turns test_runtime_tracks_both_files_off_the_loop_thread red, which proves that spawn really uses the private pool. Decisive lines: test/test_acp_spawn_offload.py:1502-1509, src/kiro_crew/acp/runtime_spawn_launch.py:1022, src/kiro_crew/acp/runtime.py:526-553

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The red Backend Tests shards (3.12, 5), (3.12, 6), (Windows) (2), (Windows) (4) and the Coverage Gate on 2723e9a are main's, not this PR's: the same two tests fail on main's own push run at d80dd05 (run 43117490567), and this diff touches neither file (it changes only test/test_acp_spawn_offload.py).

  • test/test_wait_discipline_ratchet.py::test_the_scan_matches_known_sites (test/test_update_boot_apply.py:335 wait_for(10s))
  • test/test_updates_handlers_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkers

Tracked in #35660; fixes in flight: #35625 and #35614. This PR waits for them, then re-syncs onto the main that carries the fix.

spawn() hops onto the process-wide subprocess_executor to resume the
child before it reaches the root-tracking hop these tests wait for.
That pool has eight workers shared by every test on an xdist worker, so
when an earlier test still holds all of them the spawn never reaches the
hop and the wait fails at its 60 s lost-run bound.

Each test in TestRuntimeRootTrackingOffLoop now runs spawn() against a
private one-worker pool, the same seam
test_failed_spawn_unwinds_live_tree_and_tracking in this file already
uses.

Closes #35142
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_acp_spawn_offload.py since this head went green (ee7a026 -> 758e63c); CI re-running.

The new base carries #35625, which fixes the two main test failures the previous head inherited (#35660). The diff is unchanged: the same nine-line fixture, now on a1edd91. A test main added to this file in the meantime has the same shared-pool exposure and is outside this PR's goal; it is tracked in #35771.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 12, 2026
@bolichen97
bolichen97 merged commit b3f274d into main Nov 12, 2026
99 of 100 checks passed
@bolichen97
bolichen97 deleted the fix/35142-spawn-hop-wait branch November 12, 2026 13:55
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_second_cancel_during_the_cleanup_still_reaps_exactly_once (test_acp_spawn_offload.py) times out at the 60 s lost-run bound

2 participants