Skip to content

fix(ci): refuse xdist worker replacement on the POSIX test job too - #8691

Closed
JiaDe-Wu wants to merge 1 commit into
kirodotdev:mainfrom
JiaDe-Wu:fix/posix-xdist-fail-fast
Closed

JiaDe-Wu wants to merge 1 commit into
kirodotdev:mainfrom
JiaDe-Wu:fix/posix-xdist-fail-fast

Conversation

@JiaDe-Wu

@JiaDe-Wu JiaDe-Wu commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Refs #4227 (recurrence evidence in this comment).

Problem / Motivation

--max-worker-restart=0 is set on backend-test-windows only. Enumerating every xdist invocation in ci.yml, 4 of 5 lack it, and all four are the POSIX job.

That scoping was reasonable, because the trigger understood at the time is Windows-only: pytest-timeout has no SIGALRM there, so it falls back to its thread method, which cannot fail a single test — it terminates the whole worker, and every per-test timeout surfaces as node down: Not properly terminated.

The destination is not platform-specific. Whatever kills a worker, setup.cfg's =2 permits the replacement, and xdist's node-replacement path leaves a node in assigned_work but absent from registered_collections (#2803) — which either dies with INTERNALERROR or never completes the session.

A POSIX shard reached it. Backend Tests (3.12, 4) of run 33788776072 (2026-09-03, ubuntu-latest), with timeout method: signal in its header, so this was not the Windows trigger:

19:12:56  [gw2] node down: Not properly terminated
19:12:56  F
19:12:56  replacing crashed worker gw2
19:16:12  last progress output -- 99.8% of 21077 items reported
19:28:32  ##[error]The operation was canceled.        <- the 40-minute cap

12m20s of zero output, then Terminate orphan process: pid (2405) (pytest) plus four (python) in post-job cleanup.

Why it matters

The victim's name is destroyed by the hang. The run recorded exactly one F — the crashed item — and the session never reached its summary, so nothing names it. That is #4227's problem 2, and it is what makes this class of red uninterpretable: a maintainer sees a cancellation on a diff that cannot account for it.

It is not a capacity problem, and the same run proves that. The three sibling shards passed in 28m44s, 30m49s and 31m56s — inside the band #7516 measured across 44 main-head shards (median 28.30 min, peak 30.62 min). Same commit, same runner pool, same coverage settings. Only shard 4 reached the wall, and it reached it producing nothing for the last 12 minutes. #7516's timeout half stays closed; this shard did not run out of budget doing work.

The cost is measurable. The crash was already known at 19:12:56, 24m39s into the job. @chenmingwei23's local measurement on these exact pins puts =0 at 12.4s to exit with the test named, so the run would have ended around the 25-minute mark instead of being cancelled at 40m18s.

What this evidence does not establish: why gw2 died. The log records no OOM, MemoryError or kill message. This change is deliberately root-cause-independent — it is #4227's "one root-cause-independent, cheap fix", applied to the job that still lacks it.

What changed (motivation → approach → change)

The flag goes on all four POSIX invocations, not just the sharded one. The guard's own reasoning applies: one invocation losing the flag is enough, because a shard hangs on the first worker it happens to lose. The leaf and reduced-scope paths run under -n auto too.

The ci.yml comment states the trigger/destination split so the next reader does not have to rediscover why a Windows rationale ended up on a POSIX job, and cites the measured run.

The guard test generalises rather than being duplicated. test_xdist_worker_restart_contract.py already anticipated this exact change:

Deliberately one-directional. It does NOT assert that the POSIX job lacks the flag: adopting it there may well be right (a worker can die on Linux too, e.g. under OOM), and a test forbidding that would turn a good change red.

The direction is now decided, so the platform filter is gone and the discovery covers every xdist invocation in ci.yml.

Worker-count detection is explicit about -n0. The old scan keyed on \s-n\s, which excluded -n0 only by accident of spacing and would have missed -n4 or --numprocesses=auto. It now parses the value and skips 0: an in-process run has no worker to lose, so requiring the flag on backend-test-macos would red a job for a hazard it cannot have.

Tests

test_both_platforms_are_covered is the new guard against the gap this PR closes reopening quietly — a platform filter reintroduced in the discovery would otherwise leave every other assertion in the file passing. test_a_serial_invocation_is_not_required_to_carry_the_flag pins the -n0 exclusion, which is a boundary no call site makes visible.

5 passed. Mutation-tested on the EC2 dev box, five mutations:

mutation reds
drop the flag from one POSIX invocation 1 — test_every_xdist_invocation_refuses_worker_replacement names backend-test
weaken one to --max-worker-restart=1 2
reintroduce the "windows" not in runs-on filter 1 — test_both_platforms_are_covered
add the flag to the -n0 macOS invocation 0, by design
let the discovery treat -n0 as xdist 2

The fourth is disclosed rather than hidden, because 0 reds normally reads as a coverage gap. Passing the flag to a -n0 run is an inert no-op, and a test rejecting it would red a harmless change for no gain — the same trap the original docstring warned about in the other direction. What is actually load-bearing there is the discovery boundary, and the fifth mutation confirms that boundary bites.

Manual verification

python -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" parses. scripts/scrub-lint.sh reports nothing against either file. No line exceeds the 100-column limit. ruff is not installed on that box, so formatting was checked by hand against line-length = 100 rather than by the tool — worth a reviewer's eye.

I cannot demonstrate the fix end to end, because that needs a worker to die in CI, which is exactly the unpredictable event being guarded. What is demonstrable is the pre-state: =2 is what this job runs today, and the run above is what happened under it.

Related Issues

Refs #2803, #4227, #5043, #5044, #5267, #7516.

`--max-worker-restart=0` was set on `backend-test-windows` only (kirodotdev#5043,
kirodotdev#5267), because the trigger understood at the time is Windows-only:
`pytest-timeout` has no SIGALRM there, so every per-test timeout kills a whole
worker. The destination is not platform-specific. Whatever kills a worker,
setup.cfg's `=2` permits the replacement, and xdist's node-replacement path
leaves a node in `assigned_work` but absent from `registered_collections`
(kirodotdev#2803) -- which either dies with INTERNALERROR or never completes the session.

`Backend Tests (3.12, 4)` of run 33788776072 (2026-09-03) reached it on
ubuntu-latest, with `timeout method: signal` in the header so this was not the
Windows trigger:

    19:12:56  [gw2] node down: Not properly terminated
    19:12:56  F
    19:12:56  replacing crashed worker gw2
    19:16:12  last progress output -- 99.8% of 21077 items reported
    19:28:32  ##[error]The operation was canceled.   <- the 40-minute cap

12m20s of zero output, no summary, so the one recorded failure -- the crashed
item itself -- was never named. The three sibling shards passed in 28 to 32
minutes on the same commit, so wall-time capacity (kirodotdev#7516) was not the cause.

Applies the flag to all four POSIX xdist invocations and generalises the
existing guard to every xdist invocation regardless of platform. `-n0` runs
in-process with no workers, so `backend-test-macos` stays excluded, and that
exclusion is now itself pinned rather than left implicit.

Refs kirodotdev#2803, kirodotdev#5043, kirodotdev#5044, kirodotdev#5267, kirodotdev#7516
@JiaDe-Wu
JiaDe-Wu requested review from a team and cixuuz September 5, 2026 07:38
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 5, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 5, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 10, 2026
@dwu96 dwu96 added the needs-pr-triage PR scanner: awaiting automated triage label Sep 14, 2026
@chenmingwei23 chenmingwei23 added drive-to-green PR claimed by drive-to-green pipeline and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 14, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor

Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable -- I'll push fixes directly to this branch as a co-author.

Assessment: Branch is dirty (merge conflict with main); the only other red is fork workflows awaiting maintainer approval, not a code failure. This is a small ci.yml change (add --max-worker-restart=0 to the POSIX xdist jobs). Fix plan: rebase onto current main and resolve the ci.yml conflict guided by the PR's stated intent.

If you'd prefer I don't touch this PR, add the pr-no-autofix label.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you for this fix, @JiaDe-Wu — the diagnosis (a POSIX shard reaching xdist's node-replacement path and burning the job cap without ever naming the victim) was exactly right, and the measurements you included made it easy to confirm.

Since this PR was opened, main has independently grown the same protection. The POSIX backend-test job now applies --max-worker-restart=0 to every xdist pytest invocation through a shared PROGRESS=(...) array — see .github/workflows/ci.yml line 746 (landed in #11880). The inline pytest -q -n auto … lines this PR edits no longer exist in that form, so the functional intent here is already enforced on main.

The contract test (test/test_xdist_worker_restart_contract.py) was also reworked separately in #10915: it now scans both ci.yml and platform-tests.yml across Windows and macOS, and is deliberately one-directional — its docstring states it must not assert that the POSIX job lacks the flag, because adopting it there is considered correct. Making this PR green would therefore require re-authoring it onto main's current design and reversing that deliberate test choice, rather than a rebase.

Because the goal is already met on main, I'm closing this as superseded by #11880. Your authorship and detailed root-cause analysis stand in the history — thank you again for catching and documenting this.

@iamwhatever

Copy link
Copy Markdown
Contributor

Closing as superseded by #11880 (see prior comment). Thanks again, @JiaDe-Wu.

@iamwhatever iamwhatever closed this Oct 3, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 3, 2026
@mbriones98 mbriones98 removed the drive-to-green PR claimed by drive-to-green pipeline label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants