Skip to content

test(apps): bound the spawn-ownership races by a lost-run ceiling - #19011

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/spawn-publication-windows-flake
Oct 11, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/spawn-publication-windows-flake

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The TestSpawnPublicationOwnership restart tests give the same verdict on a slow Windows runner as on a fast one.

On Backend Tests (Windows) (3), test_restart_joins_a_public_start_already_in_flight (line 1175) and test_restart_spawn_retired_by_stop_start_never_publishes_or_leaks (line 1097) both failed with assert not True on is_alive(), on a PR head (#18910) that touches nothing under apps/. Main's own push CI on Windows (3) was green at faa0b5b.

Why it matters

Each red blocks Coverage Gate and PR Readiness on whatever unrelated PR draws that shard, and fork contributors cannot rerun the job.

Not a goal

  • No change to src/kiro_crew/apps/. The start/restart ordering these tests cover is unchanged and still pinned by them.
  • Other 2 s waits in this file outside TestSpawnPublicationOwnership, which are not threaded handoffs.

What changed (motivation → approach → change)

Symptom: the worker thread is still alive when the test checks it, about 2 s after the last gate opens.

Root cause, test-side: each test releases its gates, joins the workers with join(2) and then asserts they exited. That is a wall-clock upper bound (tests-are-deterministic D7). Once the gates are open the workers wait on nothing the test holds, so they always finish. Their remaining work is real file I/O, though: the fsync'd pidfile write in _record_app_pid and, for the second test, a whole spawn body. The runner's windows-pytest-progress-3 artifact shows both tests starting at the same instant on two workers (gw0, gw3) and failing at 2.07 s and 2.17 s, while gw1 ran two tests of about 2 s each across the same window, so the whole runner was slow, not one test. Passing Windows runs of the same tests take 0.16-0.17 s.

Production code was checked for a way to keep the thread alive: after allow_publication is set, the public start's path (publish under _lock, _record_app_pid, flock release) takes no lock the restart thread holds, and _await_inflight_spawn holds _lock only per poll. No ordering bug was found, so the code is not changed.

Change: one named lost-run ceiling, _SPAWN_RACE_CEILING_SECS = 30.0, replaces the 2 s bound on every coordination wait and join in TestSpawnPublicationOwnership. It is sized as testing-conventions.md defines: more than 10x the measured work, under half of --timeout=120, named, with the measurement cited beside it. _join_started now shares one deadline across its threads and returns the time spent joining, which the still-alive assertions quote.

Why not a longer join alone: a per-thread join would let two stuck workers spend two ceilings. Why not an event set at the end of each worker: waiting on it is the same bound as join, and join also guarantees the thread is gone before the monkeypatches are undone.

Backwards compatibility

Compatible: test-only change.

Removes nothing: the diff only widens a test's wait bound and adds a failure message. No product path, record or key is touched.

Tests

Changed: test/test_apps_backend_coverage.py::TestSpawnPublicationOwnership (the two failing tests plus test_retired_cleanup_precedes_successor_reservation, which uses the same helper and waits).

Proof, run locally on Linux (Python 3.12):

  • Forced condition (a throwaway -p plugin, not committed, that delays pidfile._write_pidfile by 2.5 s, standing in for the stalled fsync): the parent faa0b5b fails both tests with assert not True on is_alive(), as CI did. This head passes 6/6.
  • Mutation: with the ceiling set back to 2.0, the same plugin fails both tests with AssertionError: alive after 2.0 s of joining.
  • Negative control: a 45 s stall fails test_restart_joins_a_public_start_already_in_flight by name at alive after 30.0 s of joining (32.6 s total), under the suite timeout.
  • Repeats: 20/20 at -n0; 10/10 at -n 4 --dist loadgroup --max-worker-restart=2 with test_app_backend.py and test_app_backend_stale_reap.py as co-tenants; the whole file shuffled (seed 20483), 236 passed.

Gates on this head: check_black_formatting.py, isort, flake8, mypy src/kiro_crew/ (0 errors in 2258 files), every scripts/check_*.py (comment-history and changelog-history against origin/main, harness parity from the merge-base, subprocess-encoding), docs_lint.py, all test/test_*ratchet*.py, test_error_code_contract.py, test_agent_refactor_facade.py, test_app_bridges.py (names this file) and this file: 780 passed, 2 skipped.

Manual verification

N/A — test-only change; the forced-condition, mutation and negative-control runs above stand in for it.

Related Issues

Fixes #18989

Pattern harvest

Rule candidate: review-prompt
Pattern: a test joins worker threads with a short literal join(N) and then asserts not thread.is_alive(), which turns runner speed into the verdict; a named lost-run ceiling belongs there.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — N/A
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 11, 2026 02:14
@jeeshofone
jeeshofone requested a review from CrysisDeu October 11, 2026 02:14
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision 20afcd92e12dd60900e07809b048d4544558d87b ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 20afcd92e12dd60900e07809b048d4544558d87b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 20afcd92e12dd60900e07809b048d4544558d87b via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 20afcd9

Description read

[DESCRIPTION-READ] e7266b0d435c46369e6dd9e58e039ea2bd328d7b7a73868f038fe433421fee8b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 20afcd92e12dd60900e07809b048d4544558d87b via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 20afcd9

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 20afcd92e12dd60900e07809b048d4544558d87b via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 20afcd9

@jeeshofone
jeeshofone force-pushed the fix/spawn-publication-windows-flake branch from 11bebc6 to a47d07a Compare October 11, 2026 03:09
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Rebased onto main 57df464 (new head a47d07a; git range-diff shows the commit unchanged).

Backend Lint & Type Check (3.12) on 11bebc6 failed at Check imports (isort) on test/test_crew_log_types.py. That file is not in this diff. The error was inherited from main after #18984, and main's own push CI hit it too at 2ada192. #18999 fixed it, and the rebase picks that fix up. On the new head, local whole-tree isort, flake8, the black gate, mypy src/kiro_crew/ (0 errors), comment-history, changelog-history, subprocess-encoding and this file plus test_app_bridges.py (459 passed) all pass.

Review lanes on 11bebc6: GPT and Opus had no findings, Design passed, and First Principles and UX were skipped (test-only). None of them raised anything to dispose of.

First-pass check: R1-R8 OK (test-only; no sensitive path, no new state; the D7 upper bound is replaced by a named lost-run ceiling under tests-are-deterministic).

The threaded TestSpawnPublicationOwnership tests joined their workers with
a 2 s bound and then asserted the workers had exited. Once every gate is
released the workers depend on nothing the test holds, but their remaining
work is real file I/O (the fsync'd pidfile write, the backend log), and a
contended Windows runner held it past 2 s. Both restart tests went red
together on one shard while every worker on that runner was slow.

Replace each 2 s coordination wait and join with one named lost-run
ceiling, sized from Windows CI measurements, and quote the time spent
joining when a worker is still alive. Under a forced 2.5 s stall at the
pidfile write the parent fails both tests the way CI did, and this
version passes; a stall past the ceiling fails by name at 30 s.

Fixes kirodotdev#18989
@jeeshofone
jeeshofone force-pushed the fix/spawn-publication-windows-flake branch from a47d07a to 20afcd9 Compare October 11, 2026 04:01
@jeeshofone

Copy link
Copy Markdown
Contributor Author

New head 20afcd9, rebased onto main d8185f1. git range-diff shows the commit unchanged.

Backend Tests (Windows) (4) on a47d07a failed in test/test_crew_log_emit.py::test_a_repair_dropped_with_the_batch_it_waited_in_comes_back at assert emit.drain_for_shutdown(timeout=3.0). That file is outside this diff, and this PR changes no product code. It is the same shape as #18989: a 3 s wall-clock bound around real store writes on a slow Windows runner. It is now tracked as #19029. This push is the one fresh-head retry. It changes nothing in the commit, and I won't push again just to rerun it.

Gates re-run on the new head: whole-tree isort and flake8, the black gate, mypy src/kiro_crew/ (0 errors in 2260 files), comment-history, changelog-history, this file plus test_app_bridges.py (459 passed), and the forced 2.5 s pidfile stall (6 passed).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR replaces the 2 s join/wait bounds in TestSpawnPublicationOwnership with one named lost-run ceiling, _SPAWN_RACE_CEILING_SECS = 30.0. _join_started now shares one deadline across its threads and returns the elapsed time for the failure message. That fixes the slow-Windows assert not True reds (#18989) at their real cause: a wall-clock upper bound on real fsync'd file work after the gates have opened, not an ordering bug. The description shows the forced 2.5 s pidfile stall failing on the parent and passing here, with the ceiling-back-to-2.0 mutant failing. I checked the worst case against --timeout=120. A coordination wait that times out (30 s), plus a second one (30 s), plus the shared join (30 s), stays at 90 s, so a real hang still fails by name instead of being killed by the suite timeout. Each finally sets the release event before joining, so no worker is left blocked on a test-held event. It is test-only with no product change, every check is green and every AI lane passes, so it is safe to merge. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 11, 2026 06:06
@iamwhatever
iamwhatever merged commit 0eb5cc5 into kirodotdev:main Oct 11, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flaky: TestSpawnPublicationOwnership restart tests leave the public start thread alive on Windows

2 participants