Repository navigation
test(apps): bound the spawn-ownership races by a lost-run ceiling - #19011
Conversation
First Principles Review (Opus 5.5, fork) — ⏭️ skippedRevision |
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 20afcd9 Description read[DESCRIPTION-READ] e7266b0d435c46369e6dd9e58e039ea2bd328d7b7a73868f038fe433421fee8b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
11bebc6 to
a47d07a
Compare
|
Rebased onto main 57df464 (new head a47d07a;
Review lanes on 11bebc6: GPT and Opus had no findings, Design passed, and First Principles and UX were skipped (test-only). None of them raised anything to dispose of. First-pass check: R1-R8 OK (test-only; no sensitive path, no new state; the D7 upper bound is replaced by a named lost-run ceiling under |
The threaded TestSpawnPublicationOwnership tests joined their workers with a 2 s bound and then asserted the workers had exited. Once every gate is released the workers depend on nothing the test holds, but their remaining work is real file I/O (the fsync'd pidfile write, the backend log), and a contended Windows runner held it past 2 s. Both restart tests went red together on one shard while every worker on that runner was slow. Replace each 2 s coordination wait and join with one named lost-run ceiling, sized from Windows CI measurements, and quote the time spent joining when a worker is still alive. Under a forced 2.5 s stall at the pidfile write the parent fails both tests the way CI did, and this version passes; a stall past the ceiling fails by name at 30 s. Fixes kirodotdev#18989
a47d07a to
20afcd9
Compare
|
New head 20afcd9, rebased onto main d8185f1.
Gates re-run on the new head: whole-tree isort and flake8, the black gate, |
iamwhatever
left a comment
There was a problem hiding this comment.
This PR replaces the 2 s join/wait bounds in TestSpawnPublicationOwnership with one named lost-run ceiling, _SPAWN_RACE_CEILING_SECS = 30.0. _join_started now shares one deadline across its threads and returns the elapsed time for the failure message. That fixes the slow-Windows assert not True reds (#18989) at their real cause: a wall-clock upper bound on real fsync'd file work after the gates have opened, not an ordering bug. The description shows the forced 2.5 s pidfile stall failing on the parent and passing here, with the ceiling-back-to-2.0 mutant failing. I checked the worst case against --timeout=120. A coordination wait that times out (30 s), plus a second one (30 s), plus the shared join (30 s), stays at 90 s, so a real hang still fails by name instead of being killed by the suite timeout. Each finally sets the release event before joining, so no worker is left blocked on a test-held event. It is test-only with no product change, every check is green and every AI lane passes, so it is safe to merge. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: The
TestSpawnPublicationOwnershiprestart tests give the same verdict on a slow Windows runner as on a fast one.On Backend Tests (Windows) (3),
test_restart_joins_a_public_start_already_in_flight(line 1175) andtest_restart_spawn_retired_by_stop_start_never_publishes_or_leaks(line 1097) both failed withassert not Trueonis_alive(), on a PR head (#18910) that touches nothing underapps/. Main's own push CI on Windows (3) was green at faa0b5b.Why it matters
Each red blocks Coverage Gate and PR Readiness on whatever unrelated PR draws that shard, and fork contributors cannot rerun the job.
Not a goal
src/kiro_crew/apps/. The start/restart ordering these tests cover is unchanged and still pinned by them.TestSpawnPublicationOwnership, which are not threaded handoffs.What changed (motivation → approach → change)
Symptom: the worker thread is still alive when the test checks it, about 2 s after the last gate opens.
Root cause, test-side: each test releases its gates, joins the workers with
join(2)and then asserts they exited. That is a wall-clock upper bound (tests-are-deterministicD7). Once the gates are open the workers wait on nothing the test holds, so they always finish. Their remaining work is real file I/O, though: the fsync'd pidfile write in_record_app_pidand, for the second test, a whole spawn body. The runner'swindows-pytest-progress-3artifact shows both tests starting at the same instant on two workers (gw0, gw3) and failing at 2.07 s and 2.17 s, while gw1 ran two tests of about 2 s each across the same window, so the whole runner was slow, not one test. Passing Windows runs of the same tests take 0.16-0.17 s.Production code was checked for a way to keep the thread alive: after
allow_publicationis set, the public start's path (publish under_lock,_record_app_pid, flock release) takes no lock the restart thread holds, and_await_inflight_spawnholds_lockonly per poll. No ordering bug was found, so the code is not changed.Change: one named lost-run ceiling,
_SPAWN_RACE_CEILING_SECS = 30.0, replaces the 2 s bound on every coordination wait and join inTestSpawnPublicationOwnership. It is sized astesting-conventions.mddefines: more than 10x the measured work, under half of--timeout=120, named, with the measurement cited beside it._join_startednow shares one deadline across its threads and returns the time spent joining, which the still-alive assertions quote.Why not a longer join alone: a per-thread join would let two stuck workers spend two ceilings. Why not an event set at the end of each worker: waiting on it is the same bound as
join, andjoinalso guarantees the thread is gone before the monkeypatches are undone.Backwards compatibility
Compatible: test-only change.
Removes nothing: the diff only widens a test's wait bound and adds a failure message. No product path, record or key is touched.
Tests
Changed:
test/test_apps_backend_coverage.py::TestSpawnPublicationOwnership(the two failing tests plustest_retired_cleanup_precedes_successor_reservation, which uses the same helper and waits).Proof, run locally on Linux (Python 3.12):
-pplugin, not committed, that delayspidfile._write_pidfileby 2.5 s, standing in for the stalled fsync): the parent faa0b5b fails both tests withassert not Trueonis_alive(), as CI did. This head passes 6/6.2.0, the same plugin fails both tests withAssertionError: alive after 2.0 s of joining.test_restart_joins_a_public_start_already_in_flightby name atalive after 30.0 s of joining(32.6 s total), under the suite timeout.-n0; 10/10 at-n 4 --dist loadgroup --max-worker-restart=2withtest_app_backend.pyandtest_app_backend_stale_reap.pyas co-tenants; the whole file shuffled (seed 20483), 236 passed.Gates on this head:
check_black_formatting.py, isort, flake8,mypy src/kiro_crew/(0 errors in 2258 files), everyscripts/check_*.py(comment-history and changelog-history against origin/main, harness parity from the merge-base, subprocess-encoding),docs_lint.py, alltest/test_*ratchet*.py,test_error_code_contract.py,test_agent_refactor_facade.py,test_app_bridges.py(names this file) and this file: 780 passed, 2 skipped.Manual verification
N/A — test-only change; the forced-condition, mutation and negative-control runs above stand in for it.
Related Issues
Fixes #18989
Pattern harvest
Rule candidate: review-prompt
Pattern: a test joins worker threads with a short literal
join(N)and then assertsnot thread.is_alive(), which turns runner speed into the verdict; a named lost-run ceiling belongs there.Checklist
Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.