Repository navigation
Conversation
`--max-worker-restart=0` was set on `backend-test-windows` only (kirodotdev#5043, kirodotdev#5267), because the trigger understood at the time is Windows-only: `pytest-timeout` has no SIGALRM there, so every per-test timeout kills a whole worker. The destination is not platform-specific. Whatever kills a worker, setup.cfg's `=2` permits the replacement, and xdist's node-replacement path leaves a node in `assigned_work` but absent from `registered_collections` (kirodotdev#2803) -- which either dies with INTERNALERROR or never completes the session. `Backend Tests (3.12, 4)` of run 33788776072 (2026-09-03) reached it on ubuntu-latest, with `timeout method: signal` in the header so this was not the Windows trigger: 19:12:56 [gw2] node down: Not properly terminated 19:12:56 F 19:12:56 replacing crashed worker gw2 19:16:12 last progress output -- 99.8% of 21077 items reported 19:28:32 ##[error]The operation was canceled. <- the 40-minute cap 12m20s of zero output, no summary, so the one recorded failure -- the crashed item itself -- was never named. The three sibling shards passed in 28 to 32 minutes on the same commit, so wall-time capacity (kirodotdev#7516) was not the cause. Applies the flag to all four POSIX xdist invocations and generalises the existing guard to every xdist invocation regardless of platform. `-n0` runs in-process with no workers, so `backend-test-macos` stays excluded, and that exclusion is now itself pinned rather than left implicit. Refs kirodotdev#2803, kirodotdev#5043, kirodotdev#5044, kirodotdev#5267, kirodotdev#7516
|
Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable -- I'll push fixes directly to this branch as a co-author. Assessment: Branch is dirty (merge conflict with main); the only other red is fork workflows awaiting maintainer approval, not a code failure. This is a small ci.yml change (add --max-worker-restart=0 to the POSIX xdist jobs). Fix plan: rebase onto current main and resolve the ci.yml conflict guided by the PR's stated intent. If you'd prefer I don't touch this PR, add the |
|
Thank you for this fix, @JiaDe-Wu — the diagnosis (a POSIX shard reaching xdist's node-replacement path and burning the job cap without ever naming the victim) was exactly right, and the measurements you included made it easy to confirm. Since this PR was opened, The contract test ( Because the goal is already met on |
Refs #4227 (recurrence evidence in this comment).
Problem / Motivation
--max-worker-restart=0is set onbackend-test-windowsonly. Enumerating every xdist invocation inci.yml, 4 of 5 lack it, and all four are the POSIX job.That scoping was reasonable, because the trigger understood at the time is Windows-only:
pytest-timeouthas no SIGALRM there, so it falls back to its thread method, which cannot fail a single test — it terminates the whole worker, and every per-test timeout surfaces asnode down: Not properly terminated.The destination is not platform-specific. Whatever kills a worker,
setup.cfg's=2permits the replacement, and xdist's node-replacement path leaves a node inassigned_workbut absent fromregistered_collections(#2803) — which either dies with INTERNALERROR or never completes the session.A POSIX shard reached it.
Backend Tests (3.12, 4)of run33788776072(2026-09-03, ubuntu-latest), withtimeout method: signalin its header, so this was not the Windows trigger:12m20s of zero output, then
Terminate orphan process: pid (2405) (pytest)plus four(python)in post-job cleanup.Why it matters
The victim's name is destroyed by the hang. The run recorded exactly one
F— the crashed item — and the session never reached its summary, so nothing names it. That is #4227's problem 2, and it is what makes this class of red uninterpretable: a maintainer sees a cancellation on a diff that cannot account for it.It is not a capacity problem, and the same run proves that. The three sibling shards passed in 28m44s, 30m49s and 31m56s — inside the band #7516 measured across 44 main-head shards (median 28.30 min, peak 30.62 min). Same commit, same runner pool, same coverage settings. Only shard 4 reached the wall, and it reached it producing nothing for the last 12 minutes. #7516's timeout half stays closed; this shard did not run out of budget doing work.
The cost is measurable. The crash was already known at 19:12:56, 24m39s into the job. @chenmingwei23's local measurement on these exact pins puts
=0at 12.4s to exit with the test named, so the run would have ended around the 25-minute mark instead of being cancelled at 40m18s.What this evidence does not establish: why
gw2died. The log records no OOM,MemoryErroror kill message. This change is deliberately root-cause-independent — it is #4227's "one root-cause-independent, cheap fix", applied to the job that still lacks it.What changed (motivation → approach → change)
The flag goes on all four POSIX invocations, not just the sharded one. The guard's own reasoning applies: one invocation losing the flag is enough, because a shard hangs on the first worker it happens to lose. The leaf and reduced-scope paths run under
-n autotoo.The
ci.ymlcomment states the trigger/destination split so the next reader does not have to rediscover why a Windows rationale ended up on a POSIX job, and cites the measured run.The guard test generalises rather than being duplicated.
test_xdist_worker_restart_contract.pyalready anticipated this exact change:The direction is now decided, so the platform filter is gone and the discovery covers every xdist invocation in
ci.yml.Worker-count detection is explicit about
-n0. The old scan keyed on\s-n\s, which excluded-n0only by accident of spacing and would have missed-n4or--numprocesses=auto. It now parses the value and skips0: an in-process run has no worker to lose, so requiring the flag onbackend-test-macoswould red a job for a hazard it cannot have.Tests
test_both_platforms_are_coveredis the new guard against the gap this PR closes reopening quietly — a platform filter reintroduced in the discovery would otherwise leave every other assertion in the file passing.test_a_serial_invocation_is_not_required_to_carry_the_flagpins the-n0exclusion, which is a boundary no call site makes visible.5 passed. Mutation-tested on the EC2 dev box, five mutations:
test_every_xdist_invocation_refuses_worker_replacementnamesbackend-test--max-worker-restart=1"windows" not in runs-onfiltertest_both_platforms_are_covered-n0macOS invocation-n0as xdistThe fourth is disclosed rather than hidden, because 0 reds normally reads as a coverage gap. Passing the flag to a
-n0run is an inert no-op, and a test rejecting it would red a harmless change for no gain — the same trap the original docstring warned about in the other direction. What is actually load-bearing there is the discovery boundary, and the fifth mutation confirms that boundary bites.Manual verification
python -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))"parses.scripts/scrub-lint.shreports nothing against either file. No line exceeds the 100-column limit.ruffis not installed on that box, so formatting was checked by hand againstline-length = 100rather than by the tool — worth a reviewer's eye.I cannot demonstrate the fix end to end, because that needs a worker to die in CI, which is exactly the unpredictable event being guarded. What is demonstrable is the pre-state:
=2is what this job runs today, and the run above is what happened under it.Related Issues
Refs #2803, #4227, #5043, #5044, #5267, #7516.