Repository navigation
ci(platform-tests): run the macOS shards on two xdist workers - #14162
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the first [DESIGN-REVIEWED] 7ec4b8a |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Before merge, confirm the re-measure dispatch on this head is green: run 36228922099 showed 8 failures at two workers that passed at one. What this change shipsInventory (3 items) — 3 justifiedIntent: keep the macOS nightly shards under the 40-minute cap by running two xdist workers instead of the one the memory budget grants — a FIX (#13720, cancelled shards skipped every publish job).
The reversed comment is a prior pin, but the diff quotes the budget's own log line ( [FIRST-PRINCIPLES-REVIEWED] 7ec4b8a |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
`-n auto` on the macos-15 runner has resolved to ONE worker on every nightly from at least 09-16 to 09-26: the budget plugin (xdist_budget.py) sizes workers by free memory, the runner has ~3 GiB free of 7 after the OS and toolchain, and the plugin's per-worker reservation (3 GiB then, 16 GiB for macOS since #10840) never fits twice. The log line is xdist worker budget: 1 of 3 workers (3.0 GiB free, 7 GiB installed) so each shard ran its tests serially on a 3-core machine at about 23 tests a second. Measured with two workers on the previous 3-shard shape (dispatch run 36228922099): 21-27 minutes a shard against 32-39 the same day at one worker. The plugin protects a laptop it does not know; this runner is a fixed shape the lane can state outright, and an explicit count bypasses the budget (it hooks pytest_xdist_auto_num_workers only). Two workers, not three: each worker imports and holds its shard's files, and the 16 GiB figure was measured on a 36 GiB laptop, not here. The gate test pins the explicit count so a revert to `auto` is visible. Sits on top of #14173 (four file-shards), which owns the shard count and the cap; this changes only the workers per shard. Refs #13720.
64d9100 to
7ec4b8a
Compare
|
Measurement on this head (
Against main's shape at one worker (09-26 nightly, 3 shards: 32-39m; 4 file-shards at |
Summary
Every macOS shard has been running on one xdist worker since at least 09-16. This PR runs them on two.
Rebased onto #14173 (four file-shards, cap stays 40), which owns the shard count and the cap; the earlier cap-raise commit is dropped. This changes only the workers per shard.
Why the shards are slow
Every shard log from 09-16 to 09-26 carries:
-n autogoes throughxdist_budget.py, which sizes workers by free memory. The macos-15 runner has 3 cores and 7 GiB, about 3 GiB free after the OS and toolchain; the plugin's per-worker reservation (3 GiB then, 16 GiB for macOS since #10840) fits once. So each shard ran serially on a 3-core machine at about 23 tests a second, and outgrew the 40-minute cap as the suite grew (two nightlies cancelled a still-passing shard at 40m, which skipped every publish job and kept #13720 open).Measured
Dispatch run 36228922099 on the previous 3-shard shape with
-n 2:created: 2/2 workerson every shard; no memory kill. Shard 1 had 8 failures intest_cron_resilience/test_cron_reaper/test_chat_send_agent_model_default/test_crew_log_previous_latch(timeouts and "awaited 0 times") that passed the same day at one worker -- being re-measured on the four-file-shard shape by a dispatch on this head.Change
-n auto→-n 2on the shard step. An explicit count bypasses the budget (it hookspytest_xdist_auto_num_workersonly). Two rather than three: each worker imports and holds its shard's files, and the 16 GiB/worker figure the plugin carries for macOS was measured on a 36 GiB laptop, not here. The gate test pins the explicit count so a revert toautois visible. Swap pressure or a jetsam kill means back to 1; two workers holding well under the cap makes 3 the next measurement.No product change. No test behaviour change beyond the pinned flag.
Refs #13720 -- not
Fixes: the watcher's issue should close on the next green scheduled run, not on merge.Verification
Static gates on the changed test file (black, isort, flake8); YAML parse of the workflow. Tests run in CI. Worker-count effect measured by
platform-tests.ymldispatch on this branch.