Repository navigation
ci(macos): give the 40-minute cap headroom back with four file-shards - #14173
Conversation
01a0037 to
e79edfd
Compare
|
Pushed The shard step invoked bare Reproduced and fixed under measurement rather than by inspection: bare The other reds on the previous head were classified, not assumed:
|
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the first live 4-shard run: hash sharding is not duration-balanced, and the per-shard minutes are — as the description admits — still unproven. [DESIGN-REVIEWED] c450527 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the per-shard minutes on the next macOS nightly — the description proves collection savings but concedes the wall-clock effect is not yet measured. Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: FIX — stop the 40-minute cap cancelling a nearly finished macOS shard and skipping nightly publication (runs 36223077241, 36200630682).
[FIRST-PRINCIPLES-REVIEWED] c450527 |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsThe marker-filter sentence in the file-sharding spec now names Linux and macOS and silently drops the Windows shard, which filters exactly like Linux — advisory only, nothing blocks. FINDING — docs/ci/ci-and-reviews.md:414 — the rewritten clause [OPUS-REVIEWED] c450527 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
e79edfd to
5c07373
Compare
|
disposition=fixed span=925f96ac92f0
|
|
disposition=fixed span=06dd12527d74
|
|
disposition=fixed span=90dfb3da3a34
|
|
disposition=accepted-and-deferred span=db2b04455c68
|
|
disposition=fixed span=ffacf959459e
|
|
disposition=rebutted span=bc7f13e953d3
|
5c07373 to
e4fe86a
Compare
|
disposition=fixed span=67a7bd4acab4
|
|
disposition=fixed span=a238d30975ca
|
|
disposition=accepted-and-deferred span=68a045beba8e
|
|
Re-synced onto The reason is the base, not a red of mine: the four green_age flagged seven overlapping files at import level ( All four review lanes were fresh with zero findings on the previous head; they will restamp on this one. |
|
disposition=fixed span=564cfcda1953
|
|
disposition=accepted-and-deferred span=16dc68ebc817
|
`-n auto` on the macos-15 runner has resolved to ONE worker on every nightly from at least 09-16 to 09-26: the budget plugin (xdist_budget.py) sizes workers by free memory, the runner has ~3 GiB free of 7 after the OS and toolchain, and the plugin's per-worker reservation (3 GiB then, 16 GiB for macOS since #10840) never fits twice. The log line is xdist worker budget: 1 of 3 workers (3.0 GiB free, 7 GiB installed) so each shard ran its tests serially on a 3-core machine at about 23 tests a second. Measured with two workers on the previous 3-shard shape (dispatch run 36228922099): 21-27 minutes a shard against 32-39 the same day at one worker. The plugin protects a laptop it does not know; this runner is a fixed shape the lane can state outright, and an explicit count bypasses the budget (it hooks pytest_xdist_auto_num_workers only). Two workers, not three: each worker imports and holds its shard's files, and the 16 GiB figure was measured on a 36 GiB laptop, not here. The gate test pins the explicit count so a revert to `auto` is visible. Sits on top of #14173 (four file-shards), which owns the shard count and the cap; this changes only the workers per shard. Refs #13720.
|
Closing the measurement this PR could not make before merging -- the Design WATCH (span 085f621144ab) asked for the four shards' real minutes, and its bar was "well inside 40, which the 1.5x rule puts at about 27 or less". Nightly run 36280765452, on At 3 shards with pytest-split the same lane read 33.0 / 40.3 (cancelled AT the cap, 98% through) / 32.2. So the slowest shard went 40.3 -> 25.0 minutes, which is 1.60x headroom against the 40-minute cap. The bar is met, and the WATCH's concern -- that hash assignment balances COUNT, not minutes -- is answered by the spread it produced: 19.6 to 25.0, max/min 1.28 in minutes against the 1.18 measured in test count. One number worth recording because it settles the cost question the Opus lane raised twice: total macOS test time went from 105.5 minutes across 3 shards to 91.8 across 4. More shards, less total work -- because what this change removed was not distributed at all. Every shard had been importing the whole tree to discard three quarters of it. The run finished 28/28 success, 0 skipped, 0 failed, and the publish chain RAN: CLI wheel, Docker image, Windows installer, six Linux packages, and sign + notarize + staple + distribution. The 09-25 and 09-26 nightlies skipped every one of those. |
Problem / Motivation
The nightly's macOS gate stopped being a hang guard and started cancelling publication.
On run 36223077241 the pytest step took 33.0 and 32.2 minutes on shards 1 and 3, and
shard 2 was 98% through its progress line when the 40-minute
timeout-minutescapcancelled it at 40.0. Run 36200630682 lost the same shard the same way. Neither was a
hang and neither was a defect: the suite outgrew what 3 shards fit inside the cap.
Why it matters
platform-tests.ymlholds the nightly publish jobs. A cancelled shard skips everyone of them, so two consecutive nightlies produced no Docker image, no signed macOS
build, no CLI and no Linux/Windows packages. Left alone it recurs at random, because the
slowest shard now sits within a minute of the cap.
What changed (motivation → approach → change)
Symptom: one macOS shard cancelled at the cap while nearly finished. Root cause: two
things, both measured.
First, the cap is a SPEND guard as much as a hang guard -- a macOS shard bills about ten
times a Linux one -- so raising it buys a longer runaway rather than headroom. The job's
own comment already named the right knob ("the shard count is the billing knob"), so the
matrix goes 3 -> 4. The fixed per-shard cost is 39-43 s measured as job wall minus the
pytest step, with a 3-5 s
uvinstall inside it.Second, this job was the ONLY one still on pytest-split, which collects the whole suite
in every shard and deselects the rest;
backend-testandbackend-test-windowsalreadyuse
scripts.ci_file_shards, where only the owning shard imports a file. It now does too,invoked through
python -m pytestso the plugin is importable (barepytestdoes not putthe working directory on
sys.path).Consequential hunks, not drive-bys:
macos-on-demand.yml:LANE_MAX_LIVE_RUNS6 -> 4 andLANE_OCCUPANCY_MAX_READS12 -> 8. That ceiling counts RUNS but its subject is the hosted macOS pool, and a run
holds one job per shard: 6 x 3 = 18 jobs was the calibration, so leaving it at 6 would
have made it 24 on the pool whose starvation it exists to prevent. The probe budget is
justified beside itself as twice the ceiling.
throughput figure appeared as literals, and
conftest.pycarried a near-identical twinof one of those sentences.
Backwards compatibility
Compatible. Nothing outside CI reads any of these values, the four shards' union is the
same test population as the three shards' was, and
pytest-splitstays installed -- therecording workflow and three tests still load it. No production code changes.
Tests
test/test_macos_platform_tests_gate.py: the existing pins move to four shards, thefile-shard flags and
--splitsasserted absent. One case is NEW --test_the_on_demand_ceiling_is_a_job_ceiling_not_a_run_ceiling-- which readsSHARD_COUNTandLANE_MAX_LIVE_RUNSout of the two workflows and asserts the PRODUCT isat most
CALIBRATED_PEAK_MACOS_JOBS(18) concurrentmacos-15jobs, plusLANE_OCCUPANCY_MAX_READS == 2 xthe ceiling. It locks in the coupling a comment cannotenforce: verified by mutation, restoring
LANE_MAX_LIVE_RUNS: "6"fails it with4 shards x 6 live runs = 24 concurrent macos-15 jobs, over the 18 ....Manual verification
--splits 3 --group 2spends 94.8 s collecting 137,829 tests to run 45,943 of them;--file-shards 3 --file-shard 2spends 29.8 s collecting only the 46,744 it owns(warm, Linux).
overlap: 35,086 + 37,696 + 32,080 + 32,967, max/min 1.18.
bare
pytest -p scripts.ci_file_shardsfails locally withModuleNotFoundError: No module named 'scripts'exactly as the runner did;python -m pytest ... --file-shards 4 --file-shard 1collects 35,120 tests, exit 0.black --check,flake8, docs lint, the comment-history gate and a YAML parse of bothworkflows clean.
platform-tests.ymldoes not runon a pull request, so the four shards' real minutes only exist on the next nightly or
workflow_dispatchrun. The collection measurement above is proven; the minutes are theopen Design WATCH (span 085f621144ab) and are being read on run 36280765452.
Screenshots / video
N/A -- no user-visible frontend surface is touched. The diff is two workflows, one gate
test, three docs and one docstring line.
Related Issues
No linked issue: nightly runs 36223077241 and 36200630682 are the record for the
cancellations. Deferred follow-up tracked in #14187 (
deferred-finding, assigned,Due: 2026-10-17):
test-durations.ymland the.test_durationsit records now balancenothing on any platform, and removing them touches three test pins plus four doc passages
that use the workflow as a worked example -- a separate surface from this CI-capacity fix.
Checklist
ci(macos): ...)