Repository navigation
ci: run the Windows backend shards with 6 xdist workers - #39666
Conversation
The Windows backend shards are CPU-bound, not memory-bound: at 4 xdist workers a shard kept 3.6-3.7 of the fleet runner's 8 vCPU busy and peaked at 8,603-8,901 of 16,111 MiB. Grant one worker per 2 GiB after 2 GiB for the host instead of one per 3 GiB, still capped at the core count: 6 on the 16 GiB fleet runner, 4 on a 4-core hosted runner. At 6 workers a shard peaked at 9,858-11,234 MiB (runs 44434126975 and 44440752272). The first attempt (#39390) was reverted (#39516) because tests that depend on wall-clock time failed at 6 workers on Windows shards 5, 7 and 8. #39505, #39545 and #39618 removed those dependencies from the tests. This change is measured over several full Windows runs before review: one run is not evidence that load-sensitive tests stay green. Related: #39365
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS This only changes the worker count of one CI job, which a revert undoes cleanly; the draft measurement plan is what decides whether it ships. Notes
[DESIGN-REVIEWED] 51f6408 Description read[DESCRIPTION-READ] fadfd7eec414c4b546af531088f46cdef793eed5f79d165fa0e2e1162f36a75b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The PR changes one memory divisor in a CI step, and the measured headroom supports it: 6 workers peaked at 11,234 of 16,111 MiB. The wall-clock flakes that caused the earlier revert have their own fixes on main (#39505, #39545, #39618). What this change shipsInventory (4 items) — 4 justifiedIntent: make the Windows backend CI shards finish sooner by using CPU cores they leave idle today. This is an ADDITION (CI tuning) and changes no product default.
[FIRST-PRINCIPLES-REVIEWED] 51f6408 Description read[DESCRIPTION-READ] 63ce7d24cf0e37be22748cc22daf59806b752f33ab18d9b3bb0543fa36b89ea8 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
A data point before more Windows worker work: when 6 workers were on main (#39390, reverted by #39516), the Windows test step was about 17 seconds shorter than comparable 4-worker runs, which is inside run-to-run spread. In the same window Windows shard 8 went red in 31 of 69 jobs and shard 5 in 29 of 71 (45% and 41%, from the #39516 description). Could this draft show, on about 30 main-equivalent full runs, a test-step gain larger than that spread and no newly red Windows test, before it leaves draft or more Windows worker changes build on it? AI-assisted, sent by Kiro Crew Captain |
|
Five full re-run sets of the Windows backend shards on this head (51f6408) are clean. Run 44633873832, attempts 1, 3, 4, 5 and 6; each is a full run, so every shard is a new sample. Attempt 2 re-ran shard 1 only and adds one sample.
Per-shard rows (job ids, peak memory, seconds) are kept with the run; any job id above can be opened from the run page. AI-assisted, posted by Claude Code |
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 20% chance that this change introduces a defect, above the 12% limit for approving on its own. This one-line CI change lowers the backend-test-windows memory divisor from 3072 to 2048. That gives 6 xdist workers on the 16,111 MiB fleet runner and keeps 4 on a 4-core hosted runner. The core cap and the floor of 1 are unchanged, and the formula now matches the Linux backend-test and test-map jobs. The wiring test pins the divisor, cap and floor, and it reads only the Windows job's step. The test fixes behind the earlier revert are on main: the hook-scan stubs in the bar-budget, board, workspace and surface suites, and the test_pr_watchers wait fix documented in testing-conventions.md. All 8 Windows shards passed on this head. Timeouts are unchanged, and a revert fully undoes the change. Tier: 2, held for a person: reviewer-lanes-untrusted. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
This review checked head 51f6408. The only change in behaviour is the Windows memory divisor going from /3072 to /2048. On the 8 vCPU fleet runner that gives 6 workers. The nproc cap still holds a 4-core hosted runner at 4, and the floor of 1 is unchanged. Every pytest call in the step uses -n "$WORKERS", and the per-test timeout, --max-worker-restart=0 and the job timeout are unchanged. The three wall-clock test fixes that caused the earlier revert (#39505, #39545, #39618) are ancestors of this head, and the PR merges cleanly onto main. The CI run data agrees with your comment: 41 of 41 full-scope Windows shard samples ran with 6 workers and passed. Peak memory stayed at or below 66%. The median shard Run tests time dropped from about 20:40 to about 17:20.
Done well: the new assertion pins the exact divisor line inside the Windows job's own step, so it goes red on a revert, and the Linux job's identical formula cannot satisfy it.
Optional:
- Post the per-set median and slowest shard next to the 4-worker baseline medians. That answers ship criterion 4 and the run-to-run spread question in the thread.
- The body still says the PR stays a draft and that the closing keyword waits for the measurement. Both steps are done, so those sentences could point to the results comment.
- #39738 edits the adjacent docs table row, so whichever PR lands second may need a small rebase.
One conflict, in the job table of docs/ci/ci-and-reviews.md: main changed the backend-test-windows row (#39666) next to the backend-test row this branch changes. Main's rows are kept and this branch's backend-test row is put back.
Problem / Motivation
Goal: Windows backend shards finish sooner by using idle cores, without making wall-clock tests fail.
The 8
backend-test-windowsshards are the slowest jobs of a full CI run, and they leave half the runner idle: at 4 xdist workers a shard kept 3.6-3.7 of the fleet runner's 8 vCPU busy and peaked at 8,603-8,901 of 16,111 MiB.This is the second attempt. The first, #39390, was reverted by #39516 because tests that depend on wall-clock time failed at 6 workers on Windows shards 5, 7 and 8 (and one suite on shard 1 ran slow). Measured on 2026-11-19: the bar-budget test failed in 31 of 69 shard-8 jobs and the watcher tests in 29 of 71 shard-5 jobs. The worker count was not the defect, the tests were, and they have since been fixed on main:
test_pr_watcherswait on the lost-run bound.Why it matters
Every full-scope pull request and every push to
mainwaits on these shards. The medianRun testsstep at 4 workers is 20:42 (runs 44489163860, 44478694150, 44472309891), and 19:35 for the three 4-worker runs that already had the new pins and prewarm. Two more workers on cores that are idle today shorten that wait for every contributor at no runner cost.Not a goal
backend-testworker count, the shard count, or the runner size.What changed (motivation → approach → change)
Goal: use the idle cores. Approach: keep the explicit worker formula and change only its memory divisor, so the count stays bounded by both memory and cores. Change:
.github/workflows/ci.yml, jobbackend-test-windows:by_mem=$(( (mem_mib - 2048) / 2048 ))instead of/ 3072. One worker per 2 GiB after 2 GiB for the host, capped at the core count: 6 on the 16 GiB fleet runner, 4 on a 4-core hosted runner. The comment above it states the measurement (peak 9,858-11,234 of 16,111 MiB at 6 workers; runs 44434126975 and 44440752272).test/test_ci_pytest_progress.py: the wiring test asserts the divisor line, the core-count cap and the floor of 1.docs/ci/ci-and-reviews.md: thebackend-test-windowsrow states the new rule.Compared with #39390, the comment and the docs row no longer claim a speed-up percentage: that is what the measurement below has to show.
Measurement plan (draft until done)
This PR stays a draft until the plan below is complete. One green run is not evidence: the first attempt was green on its own PR.
Runs: this PR's own full-suite run plus at least four re-runs of the 8 Windows shards (at least 40 shard jobs).
Recorded per run and shard: the
pytest workersline, peak memory, theRun testsstep time, and every failing test.Ship criteria, all required:
pytest workers: 6on every shard.Run testsstep time is not slower than the 4-worker median: 20:42 (runs 44489163860, 44478694150, 44472309891); 19:35 for the three 4-worker runs that already had the new pins and prewarm.If any load-sensitive test fails, this PR is not shipped and the test is fixed first. The results are posted as a PR comment.
Backwards compatibility
Compatible: only the xdist worker count of one CI job changes; no product code, API or configuration that users rely on is touched.
Removes nothing: the diff changes one divisor in a CI step, adds three assertions to a test and rewords one docs row.
Tests
test/test_ci_pytest_progress.py::TestCIProgress::test_windows_wiring_keeps_acceptance_and_other_jobs_unchangednow asserts that the Windows step computes one worker per 2 GiB after 2 GiB for the host, caps it atnprocand never goes below 1.Proof: not provable -- docs or config only
Manual verification
N/A — the change runs only on CI runners; it is verified by the measurement plan above.
Real-machine verification
N/A -- CI workflow only; measured on CI runners
Related Issues
Closes #39365
The closing keyword is added only when the measurement passes.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commitsContribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.
AI-assisted, opened by Claude Code