Skip to content

ci: run the Windows backend shards with 6 xdist workers - #39666

Merged
Zedmor merged 1 commit into
mainfrom
ci/windows-six-workers-retry
Nov 20, 2026
Merged

Zedmor merged 1 commit into
mainfrom
ci/windows-six-workers-retry

Conversation

@Zedmor

@Zedmor Zedmor commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Windows backend shards finish sooner by using idle cores, without making wall-clock tests fail.

The 8 backend-test-windows shards are the slowest jobs of a full CI run, and they leave half the runner idle: at 4 xdist workers a shard kept 3.6-3.7 of the fleet runner's 8 vCPU busy and peaked at 8,603-8,901 of 16,111 MiB.

This is the second attempt. The first, #39390, was reverted by #39516 because tests that depend on wall-clock time failed at 6 workers on Windows shards 5, 7 and 8 (and one suite on shard 1 ran slow). Measured on 2026-11-19: the bar-budget test failed in 31 of 69 shard-8 jobs and the watcher tests in 29 of 71 shard-5 jobs. The worker count was not the defect, the tests were, and they have since been fixed on main:

Why it matters

Every full-scope pull request and every push to main waits on these shards. The median Run tests step at 4 workers is 20:42 (runs 44489163860, 44478694150, 44472309891), and 19:35 for the three 4-worker runs that already had the new pins and prewarm. Two more workers on cores that are idle today shorten that wait for every contributor at no runner cost.

Not a goal

  • Fixing or quarantining load-sensitive tests in this PR. If one fails during the measurement, this PR is not shipped and the test is fixed first, in its own PR.
  • Changing the Linux backend-test worker count, the shard count, or the runner size.
  • Changing per-test timeouts.

What changed (motivation → approach → change)

Goal: use the idle cores. Approach: keep the explicit worker formula and change only its memory divisor, so the count stays bounded by both memory and cores. Change:

  • .github/workflows/ci.yml, job backend-test-windows: by_mem=$(( (mem_mib - 2048) / 2048 )) instead of / 3072. One worker per 2 GiB after 2 GiB for the host, capped at the core count: 6 on the 16 GiB fleet runner, 4 on a 4-core hosted runner. The comment above it states the measurement (peak 9,858-11,234 of 16,111 MiB at 6 workers; runs 44434126975 and 44440752272).
  • test/test_ci_pytest_progress.py: the wiring test asserts the divisor line, the core-count cap and the floor of 1.
  • docs/ci/ci-and-reviews.md: the backend-test-windows row states the new rule.

Compared with #39390, the comment and the docs row no longer claim a speed-up percentage: that is what the measurement below has to show.

Measurement plan (draft until done)

This PR stays a draft until the plan below is complete. One green run is not evidence: the first attempt was green on its own PR.

Runs: this PR's own full-suite run plus at least four re-runs of the 8 Windows shards (at least 40 shard jobs).

Recorded per run and shard: the pytest workers line, peak memory, the Run tests step time, and every failing test.

Ship criteria, all required:

  1. pytest workers: 6 on every shard.
  2. Peak memory under 85% of runner memory on every shard.
  3. Zero failures from timeouts or wall-clock waits across all runs.
  4. The median shard's Run tests step time is not slower than the 4-worker median: 20:42 (runs 44489163860, 44478694150, 44472309891); 19:35 for the three 4-worker runs that already had the new pins and prewarm.

If any load-sensitive test fails, this PR is not shipped and the test is fixed first. The results are posted as a PR comment.

Backwards compatibility

Compatible: only the xdist worker count of one CI job changes; no product code, API or configuration that users rely on is touched.

Removes nothing: the diff changes one divisor in a CI step, adds three assertions to a test and rewords one docs row.

Tests

test/test_ci_pytest_progress.py::TestCIProgress::test_windows_wiring_keeps_acceptance_and_other_jobs_unchanged now asserts that the Windows step computes one worker per 2 GiB after 2 GiB for the host, caps it at nproc and never goes below 1.

Proof: not provable -- docs or config only

Manual verification

N/A — the change runs only on CI runners; it is verified by the measurement plan above.

Real-machine verification

N/A -- CI workflow only; measured on CI runners

Related Issues

Closes #39365

The closing keyword is added only when the measurement passes.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

AI-assisted, opened by Claude Code

The Windows backend shards are CPU-bound, not memory-bound: at 4 xdist
workers a shard kept 3.6-3.7 of the fleet runner's 8 vCPU busy and
peaked at 8,603-8,901 of 16,111 MiB. Grant one worker per 2 GiB after
2 GiB for the host instead of one per 3 GiB, still capped at the core
count: 6 on the 16 GiB fleet runner, 4 on a 4-core hosted runner. At 6
workers a shard peaked at 9,858-11,234 MiB (runs 44434126975 and
44440752272).

The first attempt (#39390) was reverted (#39516) because tests that
depend on wall-clock time failed at 6 workers on Windows shards 5, 7
and 8. #39505, #39545 and #39618 removed those dependencies from the
tests.

This change is measured over several full Windows runs before review:
one run is not evidence that load-sensitive tests stay green.

Related: #39365
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 51f640864e8947fbb9e9993e5eb83f43d6f26e08 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 51f640864e8947fbb9e9993e5eb83f43d6f26e08 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This only changes the worker count of one CI job, which a revert undoes cleanly; the draft measurement plan is what decides whether it ships.

Notes

[DESIGN-REVIEWED] 51f6408

Description read

[DESCRIPTION-READ] fadfd7eec414c4b546af531088f46cdef793eed5f79d165fa0e2e1162f36a75b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 51f640864e8947fbb9e9993e5eb83f43d6f26e08 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The PR changes one memory divisor in a CI step, and the measured headroom supports it: 6 workers peaked at 11,234 of 16,111 MiB. The wall-clock flakes that caused the earlier revert have their own fixes on main (#39505, #39545, #39618).

What this change ships

Inventory (4 items) — 4 justified

Intent: make the Windows backend CI shards finish sooner by using CPU cores they leave idle today. This is an ADDITION (CI tuning) and changes no product default.

  1. On the 16 GiB fleet runner, Windows backend shards now run 6 xdist workers instead of 4. A 4-core hosted runner still gets 4. — justified
  2. The wiring test now pins the per-worker memory divisor, the core-count cap and the floor of 1. — justified
  3. The backend-test-windows row in docs/ci/ci-and-reviews.md states the new rule and the measured peak. — justified
  4. The workflow comment now records the CPU and memory measurement behind the divisor. — justified

[FIRST-PRINCIPLES-REVIEWED] 51f6408

Description read

[DESCRIPTION-READ] 63ce7d24cf0e37be22748cc22daf59806b752f33ab18d9b3bb0543fa36b89ea8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 51f640864e8947fbb9e9993e5eb83f43d6f26e08 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 51f6408

Verdict parsed from the review's SHA-scoped output markers for commit 51f640864e8947fbb9e9993e5eb83f43d6f26e08.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 51f640864e8947fbb9e9993e5eb83f43d6f26e08 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 51f640864e8947fbb9e9993e5eb83f43d6f26e08 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 51f6408

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 51f640864e8947fbb9e9993e5eb83f43d6f26e08 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This PR edits .github/workflows/ci.yml, and #39327 also edits that file and is still open. Whichever lands second will need a rebase, so please rebase this one after #39327 merges and re-check the Windows shard settings against its version.

AI-assisted, sent by Kiro Crew Captain

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

A data point before more Windows worker work: when 6 workers were on main (#39390, reverted by #39516), the Windows test step was about 17 seconds shorter than comparable 4-worker runs, which is inside run-to-run spread. In the same window Windows shard 8 went red in 31 of 69 jobs and shard 5 in 29 of 71 (45% and 41%, from the #39516 description).

Could this draft show, on about 30 main-equivalent full runs, a test-step gain larger than that spread and no newly red Windows test, before it leaves draft or more Windows worker changes build on it?

AI-assisted, sent by Kiro Crew Captain

@Zedmor

Zedmor commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Five full re-run sets of the Windows backend shards on this head (51f6408) are clean. Run 44633873832, attempts 1, 3, 4, 5 and 6; each is a full run, so every shard is a new sample. Attempt 2 re-ran shard 1 only and adds one sample.

Set Attempt Shards passed Workers Peak memory in use (worst shard) Run tests summed Slowest shard
1 1 8 of 8 6 10,681 of 16,095 MiB 8,284 s 1,219 s
2 3 8 of 8 6 10,679 of 16,111 MiB 8,281 s 1,212 s
3 4 8 of 8 6 10,494 of 16,095 MiB 8,597 s 1,289 s
4 5 8 of 8 6 10,514 of 16,095 MiB 8,304 s 1,244 s
5 6 8 of 8 6 10,542 of 16,111 MiB 8,615 s 1,256 s
  • 41 Windows shard samples, 41 passed, no failing test in any of them.
  • Every shard ran with 6 workers; peak memory in use never went above 66% of the runner's total.
  • The first attempt of this change (ci: run the Windows backend shards with 6 xdist workers #39390) passed one run and then failed 41-45% of Windows shard jobs, which is why this one was held for five sets. These five sets did not reproduce that failure; they are five runs, not a guarantee.

Per-shard rows (job ids, peak memory, seconds) are kept with the run; any job id above can be opened from the run page.

AI-assisted, posted by Claude Code

@Zedmor
Zedmor marked this pull request as ready for review November 20, 2026 03:39
@Zedmor
Zedmor requested review from a team and KiroCrewCaptain as code owners November 20, 2026 03:39
@Zedmor
Zedmor requested a review from dodgeblaster November 20, 2026 03:39
@Zedmor
Zedmor enabled auto-merge (squash) November 20, 2026 03:39
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 20, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 20% chance that this change introduces a defect, above the 12% limit for approving on its own.

This one-line CI change lowers the backend-test-windows memory divisor from 3072 to 2048. That gives 6 xdist workers on the 16,111 MiB fleet runner and keeps 4 on a 4-core hosted runner. The core cap and the floor of 1 are unchanged, and the formula now matches the Linux backend-test and test-map jobs. The wiring test pins the divisor, cap and floor, and it reads only the Windows job's step. The test fixes behind the earlier revert are on main: the hook-scan stubs in the bar-budget, board, workspace and surface suites, and the test_pr_watchers wait fix documented in testing-conventions.md. All 8 Windows shards passed on this head. Timeouts are unchanged, and a revert fully undoes the change.

Tier: 2, held for a person: reviewer-lanes-untrusted.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 51f640864e · 16 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review checked head 51f6408. The only change in behaviour is the Windows memory divisor going from /3072 to /2048. On the 8 vCPU fleet runner that gives 6 workers. The nproc cap still holds a 4-core hosted runner at 4, and the floor of 1 is unchanged. Every pytest call in the step uses -n "$WORKERS", and the per-test timeout, --max-worker-restart=0 and the job timeout are unchanged. The three wall-clock test fixes that caused the earlier revert (#39505, #39545, #39618) are ancestors of this head, and the PR merges cleanly onto main. The CI run data agrees with your comment: 41 of 41 full-scope Windows shard samples ran with 6 workers and passed. Peak memory stayed at or below 66%. The median shard Run tests time dropped from about 20:40 to about 17:20.

Done well: the new assertion pins the exact divisor line inside the Windows job's own step, so it goes red on a revert, and the Linux job's identical formula cannot satisfy it.

Optional:

  • Post the per-set median and slowest shard next to the 4-worker baseline medians. That answers ship criterion 4 and the run-to-run spread question in the thread.
  • The body still says the PR stays a draft and that the closing keyword waits for the measurement. Both steps are done, so those sentences could point to the results comment.
  • #39738 edits the adjacent docs table row, so whichever PR lands second may need a small rebase.

@Zedmor
Zedmor merged commit 283ee7a into main Nov 20, 2026
403 checks passed
@Zedmor
Zedmor deleted the ci/windows-six-workers-retry branch November 20, 2026 05:00
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 20, 2026
Zedmor pushed a commit that referenced this pull request Nov 20, 2026
One conflict, in the job table of docs/ci/ci-and-reviews.md: main changed
the backend-test-windows row (#39666) next to the backend-test row this
branch changes. Main's rows are kept and this branch's backend-test row
is put back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: run the Windows backend shards with 6 xdist workers instead of 4 (measured 8-14% faster execution)

2 participants