Skip to content

ci(platform-tests): run the macOS shards on two xdist workers - #14162

Merged
iamwhatever merged 1 commit into
mainfrom
ci/macos-shard-cap
Sep 27, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
ci/macos-shard-cap

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Every macOS shard has been running on one xdist worker since at least 09-16. This PR runs them on two.

Rebased onto #14173 (four file-shards, cap stays 40), which owns the shard count and the cap; the earlier cap-raise commit is dropped. This changes only the workers per shard.

Why the shards are slow

Every shard log from 09-16 to 09-26 carries:

xdist_budget.py: UserWarning: xdist worker budget: 1 of 3 workers (3.0 GiB free, 7 GiB installed)
created: 1/1 worker
1 worker [45712 items]

-n auto goes through xdist_budget.py, which sizes workers by free memory. The macos-15 runner has 3 cores and 7 GiB, about 3 GiB free after the OS and toolchain; the plugin's per-worker reservation (3 GiB then, 16 GiB for macOS since #10840) fits once. So each shard ran serially on a 3-core machine at about 23 tests a second, and outgrew the 40-minute cap as the suite grew (two nightlies cancelled a still-passing shard at 40m, which skipped every publish job and kept #13720 open).

Measured

Dispatch run 36228922099 on the previous 3-shard shape with -n 2:

1 worker (09-26 nightly) 2 workers (dispatch)
shard 1 33m 23m
shard 2 cancelled @40m 21m
shard 3 32m 27m

created: 2/2 workers on every shard; no memory kill. Shard 1 had 8 failures in test_cron_resilience / test_cron_reaper / test_chat_send_agent_model_default / test_crew_log_previous_latch (timeouts and "awaited 0 times") that passed the same day at one worker -- being re-measured on the four-file-shard shape by a dispatch on this head.

Change

-n auto → -n 2 on the shard step. An explicit count bypasses the budget (it hooks pytest_xdist_auto_num_workers only). Two rather than three: each worker imports and holds its shard's files, and the 16 GiB/worker figure the plugin carries for macOS was measured on a 36 GiB laptop, not here. The gate test pins the explicit count so a revert to auto is visible. Swap pressure or a jetsam kill means back to 1; two workers holding well under the cap makes 3 the next measurement.

No product change. No test behaviour change beyond the pinned flag.

Refs #13720 -- not Fixes: the watcher's issue should close on the next green scheduled run, not on merge.

Verification

Static gates on the changed test file (black, isort, flake8); YAML parse of the workflow. Tests run in CI. Worker-count effect measured by platform-tests.yml dispatch on this branch.

@CrysisDeu
CrysisDeu requested review from a team and pepmach September 26, 2026 07:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 7ec4b8a790c8d6275bab16b54aa21414d7b4110d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7ec4b8a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7ec4b8a790c8d6275bab16b54aa21414d7b4110d: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 7ec4b8a790c8d6275bab16b54aa21414d7b4110d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the first -n 2 nightlies show no swap/jetsam events — the two-worker measurement was taken on the 3-shard shape, not the current four.

[DESIGN-REVIEWED] 7ec4b8a

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 7ec4b8a790c8d6275bab16b54aa21414d7b4110d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Before merge, confirm the re-measure dispatch on this head is green: run 36228922099 showed 8 failures at two workers that passed at one.

What this change ships

Inventory (3 items) — 3 justified

Intent: keep the macOS nightly shards under the 40-minute cap by running two xdist workers instead of the one the memory budget grants — a FIX (#13720, cancelled shards skipped every publish job).

  1. macOS test shards run on two workers instead of one, finishing 21–27m instead of 32m–cancelled — justified
  2. The gate test now pins -n 2 and rejects a revert to -n auto — justified
  3. The workflow comment's "-n auto already saturates the runner" rationale is replaced by the measured budget explanation — justified

The reversed comment is a prior pin, but the diff quotes the budget's own log line (1 of 3 workers) and a measured dispatch proving it wrong, which is the evidence lens 8 requires. -n <N> is xdist_budget.py's own documented bypass (its clamp warning names it), so no duplicate mechanism; the Windows and ubuntu -n auto jobs sit on the 3 GiB non-macOS reservation and are not siblings of the 16 GiB macOS root cause.

[FIRST-PRINCIPLES-REVIEWED] 7ec4b8a

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 7ec4b8a790c8d6275bab16b54aa21414d7b4110d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7ec4b8a

Verdict parsed from the review's SHA-scoped output markers for commit 7ec4b8a790c8d6275bab16b54aa21414d7b4110d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7ec4b8a790c8d6275bab16b54aa21414d7b4110d: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 26, 2026
`-n auto` on the macos-15 runner has resolved to ONE worker on every
nightly from at least 09-16 to 09-26: the budget plugin
(xdist_budget.py) sizes workers by free memory, the runner has ~3 GiB
free of 7 after the OS and toolchain, and the plugin's per-worker
reservation (3 GiB then, 16 GiB for macOS since #10840) never fits
twice. The log line is

  xdist worker budget: 1 of 3 workers (3.0 GiB free, 7 GiB installed)

so each shard ran its tests serially on a 3-core machine at about 23
tests a second. Measured with two workers on the previous 3-shard shape
(dispatch run 36228922099): 21-27 minutes a shard against 32-39 the same
day at one worker.

The plugin protects a laptop it does not know; this runner is a fixed
shape the lane can state outright, and an explicit count bypasses the
budget (it hooks pytest_xdist_auto_num_workers only). Two workers, not
three: each worker imports and holds its shard's files, and the 16 GiB
figure was measured on a 36 GiB laptop, not here. The gate test pins the
explicit count so a revert to `auto` is visible.

Sits on top of #14173 (four file-shards), which owns the shard count and
the cap; this changes only the workers per shard.

Refs #13720.
@CrysisDeu CrysisDeu changed the title ci(platform-tests): raise the macOS shard cap to 60 minutes ci(platform-tests): run the macOS shards on two xdist workers Sep 26, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Measurement on this head (7ec4b8a790, four file-shards, -n 2): run 36280872205 -- all four shards green, created: 2/2 workers on each, no memory kill, 0 failures (the 8 shard-1 failures from the 3-shard measurement did not recur).

shard items pytest wall job total
1 32293 14:00 13m
2 37660 13:39 14m
3 33368 13:17 14m
4 35287 12:54 14m

Against main's shape at one worker (09-26 nightly, 3 shards: 32-39m; 4 file-shards at auto would be ~25m by item count), this is roughly half. Slowest shard sits at 14 of the 40-minute cap.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@iamwhatever
iamwhatever merged commit b1de40b into main Sep 27, 2026
100 checks passed
@iamwhatever
iamwhatever deleted the ci/macos-shard-cap branch September 27, 2026 03:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants