Repository navigation
test(acp): give the root-tracking hop tests their own subprocess pool - #35627
Conversation
|
[operator: bolichen97] Forced-condition evidence for the determinism proof in the description: the throwaway plugin (never committed) and the commands. """Throwaway forced condition for #35142 (never committed).
Before each TestRuntimeRootTrackingOffLoop test runs, hold every worker of the
process-wide ``subprocess_executor`` -- the state an earlier test on the same xdist
worker can leave behind when its own pool jobs outlive it. Released at teardown.
"""
import threading
import pytest
_held = {}
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
import os
if os.environ.get("FORCE_SCOPE", "TestRuntimeRootTrackingOffLoop") not in item.nodeid:
yield
return
from kiro_crew import executors
pool = executors.subprocess_executor()
release = threading.Event()
started = threading.Barrier(pool._max_workers + 1)
def hold():
started.wait(30)
release.wait(300)
futures = [pool.submit(hold) for _ in range(pool._max_workers)]
started.wait(30)
print(f"\n[forced] holding all {pool._max_workers} subprocess_executor workers")
try:
yield
finally:
release.set()
for f in futures:
f.result(timeout=30)# on the parent ee7a026dd093 and on this head 2723e9a1566b
PYTHONPATH=<dir holding force_pool_full.py> python -m pytest -n0 -p no:cacheprovider -p force_pool_full --timeout=20 -q -rA "test/test_acp_spawn_offload.py::TestRuntimeRootTrackingOffLoop"Parent: |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this test-only fixture removes the class's dependence on the shared pool, using the seam the file's sibling test already relies on, and the facade mirror carries the patch through to [DESIGN-REVIEWED] a1edd91 Description read[DESCRIPTION-READ] 9f87c446ea24ad97fff90b599a00f5f75e451e34128345f046fc2c8c99377835 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
buluoray
left a comment
There was a problem hiding this comment.
Approve on 2723e9a. This PR gives the three root-tracking hop tests in TestRuntimeRootTrackingOffLoop their own one-worker thread pool. Before, spawn() used the subprocess pool that every test on the same xdist worker shares, so an earlier test that still held those workers could keep spawn from ever reaching the tracking hop. The change is nine lines in one test file and touches no product code. It removes no assertion, adds no sleep or timeout, and uses the same pool seam that the sibling test at test/test_acp_spawn_offload.py:1325 already uses. All 56 tests in the file pass, both serially and under xdist. Breaking the product guard (running _track_root_pids on the event loop thread) turns all three tests red. Making the new private pool raise turns test_runtime_tracks_both_files_off_the_loop_thread red, which proves that spawn really uses the private pool. Decisive lines: test/test_acp_spawn_offload.py:1502-1509, src/kiro_crew/acp/runtime_spawn_launch.py:1022, src/kiro_crew/acp/runtime.py:526-553
|
[operator: bolichen97] The red Backend Tests shards (3.12, 5), (3.12, 6), (Windows) (2), (Windows) (4) and the Coverage Gate on 2723e9a are
Tracked in #35660; fixes in flight: #35625 and #35614. This PR waits for them, then re-syncs onto the |
spawn() hops onto the process-wide subprocess_executor to resume the child before it reaches the root-tracking hop these tests wait for. That pool has eight workers shared by every test on an xdist worker, so when an earlier test still holds all of them the spawn never reaches the hop and the wait fails at its 60 s lost-run bound. Each test in TestRuntimeRootTrackingOffLoop now runs spawn() against a private one-worker pool, the same seam test_failed_spawn_unwinds_live_tree_and_tracking in this file already uses. Closes #35142
2723e9a to
a1edd91
Compare
|
Rebased: main moved in test/test_acp_spawn_offload.py since this head went green (ee7a026 -> 758e63c); CI re-running. The new base carries #35625, which fixes the two [operator: bolichen97] |
Problem / Motivation
Goal: the root-tracking hop tests in
TestRuntimeRootTrackingOffLooppass whatever an earlier test on the same xdist worker left running on the shared subprocess pool.test_a_second_cancel_during_the_cleanup_still_reaps_exactly_oncefailed on CI (Backend Tests (3.12, 6)) in a PR that touches nothing it reads:AssertionError: the event had not happened after 60.06s (lost-run bound 60.00s); it is still clear. It failed at its first wait, sospawn()never reached the tracking hop the test parks.Why it matters
A red Backend Tests shard on an unrelated PR costs a rerun and a round of triage, and the sibling test in the same class has the same shape.
Not a goal
spawn(), its reap guard, or the 60 s lost-run bound.What changed (motivation → approach → change)
The three tests in
TestRuntimeRootTrackingOffLoopno longer wait behind work that other tests left on the shared subprocess pool.The tests wait for
spawn()to reach its root-tracking hop. Before that hop,spawn()runs its resume step onsubprocess_executor(), the process-wide eight-worker pool that every test on an xdist worker shares. When earlier tests still hold all eight workers, the resume step queues, the tracking hop is never reached, and the wait fails after 60 s with nothing to say why.Each test in
TestRuntimeRootTrackingOffLoopnow runsspawn()against its own one-worker pool, through an autouse fixture that patchessubprocess_executorfor the class. It is the same seamtest_failed_spawn_unwinds_live_tree_and_trackingin this file already uses. The tests' assertions are unchanged.Backwards compatibility
Compatible: test-only change; no product code is touched.
Removes nothing: the diff adds one test fixture.
Tests
test/test_acp_spawn_offload.py::TestRuntimeRootTrackingOffLoop: all three tests runspawn()on a private pool; what each one pins is unchanged.Determinism proof (testing-conventions, Proving a determinism fix):
758e63ca5b97a1edd914c73b-pplugin holds all eightsubprocess_executorworkers for each test of the class (-n0 --timeout=20, so the lost-run bound is 10 s)the event had not happened after 10.01s (lost-run bound 10.00s); it is still clear-n0-n 4 --dist loadgroup --max-worker-restart=2withtest_acp_offload.pyandtest_acp_spawn_label.pyas co-tenantstest_acp_spawn_offload.py+test_acp_spawn_label.py(seed 29799, 71 ids)test/test_flake_pattern_ratchet.pyProof: not provable -- test-only change
Manual verification
N/A — test-only change; the forced-condition runs above are the verification.
Screenshots / video
N/A — no user-facing surface changes.
Related Issues
Closes #35142
Sibling sweep
Swept: git grep -nE "runtime.spawn()|client._spawn()|await_until(" -- test/test_acp_spawn_offload.py
Sibling: test/test_acp_spawn_offload.py:1638 -- changed here
Sibling: test/test_acp_spawn_offload.py:1696 -- changed here
Sibling: test/test_acp_spawn_offload.py:1375 -- not affected -- already runs spawn() on its own one-worker pool
Sibling: test/test_acp_spawn_offload.py:557 -- tracked in #35771
Sibling: test/test_acp_spawn_offload.py:1489 -- not affected -- awaits spawn() itself with no hop event under a lost-run bound
Sibling: test/test_acp_spawn_offload.py:1532 -- not affected -- awaits spawn() itself with no hop event under a lost-run bound
Sibling: test/test_acp_spawn_offload.py:358 -- not affected -- awaits spawn() itself with no hop event under a lost-run bound
Pattern harvest
Not generalizable: the class is already watched; the conftest shared-pool floor names every test that leaves work on a process-wide pool, and this PR removes one waiting test's dependence on that pool.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)