Skip to content

ci(macos): give the 40-minute cap headroom back with four file-shards - #14173

Merged
bolichen97 merged 1 commit into
mainfrom
fix/macos-shard-cap-headroom
Sep 26, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/macos-shard-cap-headroom

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

The nightly's macOS gate stopped being a hang guard and started cancelling publication.
On run 36223077241 the pytest step took 33.0 and 32.2 minutes on shards 1 and 3, and
shard 2 was 98% through its progress line when the 40-minute timeout-minutes cap
cancelled it at 40.0. Run 36200630682 lost the same shard the same way. Neither was a
hang and neither was a defect: the suite outgrew what 3 shards fit inside the cap.

Why it matters

platform-tests.yml holds the nightly publish jobs. A cancelled shard skips every
one of them, so two consecutive nightlies produced no Docker image, no signed macOS
build, no CLI and no Linux/Windows packages. Left alone it recurs at random, because the
slowest shard now sits within a minute of the cap.

What changed (motivation → approach → change)

Symptom: one macOS shard cancelled at the cap while nearly finished. Root cause: two
things, both measured.

First, the cap is a SPEND guard as much as a hang guard -- a macOS shard bills about ten
times a Linux one -- so raising it buys a longer runaway rather than headroom. The job's
own comment already named the right knob ("the shard count is the billing knob"), so the
matrix goes 3 -> 4. The fixed per-shard cost is 39-43 s measured as job wall minus the
pytest step, with a 3-5 s uv install inside it.

Second, this job was the ONLY one still on pytest-split, which collects the whole suite
in every shard and deselects the rest; backend-test and backend-test-windows already
use scripts.ci_file_shards, where only the owning shard imports a file. It now does too,
invoked through python -m pytest so the plugin is importable (bare pytest does not put
the working directory on sys.path).

Consequential hunks, not drive-bys:

  • macos-on-demand.yml: LANE_MAX_LIVE_RUNS 6 -> 4 and LANE_OCCUPANCY_MAX_READS
    12 -> 8. That ceiling counts RUNS but its subject is the hosted macOS pool, and a run
    holds one job per shard: 6 x 3 = 18 jobs was the calibration, so leaving it at 6 would
    have made it 24 on the pool whose starvation it exists to prevent. The probe budget is
    justified beside itself as twice the ceiling.
  • Docs: five passages still said macOS used pytest-split, the ceiling's value and
    throughput figure appeared as literals, and conftest.py carried a near-identical twin
    of one of those sentences.

Backwards compatibility

Compatible. Nothing outside CI reads any of these values, the four shards' union is the
same test population as the three shards' was, and pytest-split stays installed -- the
recording workflow and three tests still load it. No production code changes.

Tests

test/test_macos_platform_tests_gate.py: the existing pins move to four shards, the
file-shard flags and --splits asserted absent. One case is NEW --
test_the_on_demand_ceiling_is_a_job_ceiling_not_a_run_ceiling -- which reads
SHARD_COUNT and LANE_MAX_LIVE_RUNS out of the two workflows and asserts the PRODUCT is
at most CALIBRATED_PEAK_MACOS_JOBS (18) concurrent macos-15 jobs, plus
LANE_OCCUPANCY_MAX_READS == 2 x the ceiling. It locks in the coupling a comment cannot
enforce: verified by mutation, restoring LANE_MAX_LIVE_RUNS: "6" fails it with
4 shards x 6 live runs = 24 concurrent macos-15 jobs, over the 18 ....

Manual verification

  • --splits 3 --group 2 spends 94.8 s collecting 137,829 tests to run 45,943 of them;
    --file-shards 3 --file-shard 2 spends 29.8 s collecting only the 46,744 it owns
    (warm, Linux).
  • The four file-shards sum to exactly 137,829 tests -- the full suite -- with no
    overlap: 35,086 + 37,696 + 32,080 + 32,967, max/min 1.18.
  • The plugin-import failure an earlier head hit is reproduced and fixed under measurement:
    bare pytest -p scripts.ci_file_shards fails locally with ModuleNotFoundError: No module named 'scripts' exactly as the runner did; python -m pytest ... --file-shards 4 --file-shard 1 collects 35,120 tests, exit 0.
  • Gate pins plus the overlapping suites green (767 passed / 1 skipped on the final base);
    black --check, flake8, docs lint, the comment-history gate and a YAML parse of both
    workflows clean.
  • NOT verified here, by nature: the per-shard wall clock. platform-tests.yml does not run
    on a pull request, so the four shards' real minutes only exist on the next nightly or
    workflow_dispatch run. The collection measurement above is proven; the minutes are the
    open Design WATCH (span 085f621144ab) and are being read on run 36280765452.

Screenshots / video

N/A -- no user-visible frontend surface is touched. The diff is two workflows, one gate
test, three docs and one docstring line.

Related Issues

No linked issue: nightly runs 36223077241 and 36200630682 are the record for the
cancellations. Deferred follow-up tracked in #14187 (deferred-finding, assigned,
Due: 2026-10-17): test-durations.yml and the .test_durations it records now balance
nothing on any platform, and removing them touches three test pins plus four doc passages
that use the workflow as a worked example -- a separate surface from this CI-capacity fix.

Checklist

  • One commit, Conventional Commits title (ci(macos): ...)
  • Existing tests pass; a new test added for the ceiling/shard-count coupling
  • Self-review completed; style gates clean
  • Documentation updated (five passages across three docs plus one docstring)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and cixuuz September 26, 2026 08:20
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Sep 26, 2026
@bolichen97
bolichen97 force-pushed the fix/macos-shard-cap-headroom branch from 01a0037 to e79edfd Compare September 26, 2026 08:52
@bolichen97

Copy link
Copy Markdown
Contributor Author

Pushed e79edfd07d: the previous head redded all four macOS shards and the cause was mine, not the sharding.

The shard step invoked bare pytest, which does not put the working directory on sys.path, so -p scripts.ci_file_shards could not be imported: ModuleNotFoundError: No module named 'scripts', the step died in seconds, and every shard failed identically. backend-test and backend-test-windows both invoke python -m pytest, which is exactly what makes the plugin resolvable there. Now this job does too, with the reason in a comment beside it so the next person does not re-discover it.

Reproduced and fixed under measurement rather than by inspection: bare pytest -p scripts.ci_file_shards fails locally with the same ModuleNotFoundError, and python -m pytest -p scripts.ci_file_shards --file-shards 4 --file-shard 1 collects 35,120 tests, exit 0. (My earlier local验证 happened to use python -m, which is why it passed here and not on the runner -- that gap is what this head closes.)

The other reds on the previous head were classified, not assumed:

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of c450527df14cc50dccc80c61b96dbbe74497c24f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c450527

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c450527df14cc50dccc80c61b96dbbe74497c24f: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of c450527df14cc50dccc80c61b96dbbe74497c24f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the first live 4-shard run: hash sharding is not duration-balanced, and the per-shard minutes are — as the description admits — still unproven.

[DESIGN-REVIEWED] c450527

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of c450527df14cc50dccc80c61b96dbbe74497c24f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the per-shard minutes on the next macOS nightly — the description proves collection savings but concedes the wall-clock effect is not yet measured.

Not justified as shipped

  • Item 7 — undeclared: the description says "Six files, config/infra and docs only" but the diff also edits conftest.py's _restore_log_record_factory docstring; comment-only, harm-free rider.

What this change ships

Inventory (7 items) — 6 justified

Intent: FIX — stop the 40-minute cap cancelling a nearly finished macOS shard and skipping nightly publication (runs 36223077241, 36200630682).

  1. macOS backend job runs 4 shards instead of 3 — justified
  2. macOS shards now use the existing scripts.ci_file_shards plugin instead of pytest-split's collect-then-deselect — justified
  3. On-demand macOS lane admits 4 live runs instead of 6, keeping 4×4=16 jobs under the calibrated 18 — justified
  4. Occupancy probe budget drops 12→8, preserving the documented 2×-ceiling rule — justified
  5. New test pins shards × ceiling ≤ 18 jobs and probe = 2× ceiling — justified
  6. Docs and lane table updated in the same commit; ceiling literal replaced by the variable name — justified
  7. conftest.py docstring now says every platform file-shards — undeclared: not among the description's "Six files"

[FIRST-PRINCIPLES-REVIEWED] c450527

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed c450527df14cc50dccc80c61b96dbbe74497c24f — this comment is updated in place on each push.

Review details

The marker-filter sentence in the file-sharding spec now names Linux and macOS and silently drops the Windows shard, which filters exactly like Linux — advisory only, nothing blocks.

FINDING — docs/ci/ci-and-reviews.md:414 — the rewritten clause ordinary Linux shard excludes only \ipv6_required` ... the macOS command passes no marker filter at allomitsbackend-test-windows, whose shard command carries the same -m "not ipv6_required" (ci.yml:930) and whose items are likewise covered by the hosted IPv6 lane, so a section that enumerates all three matrices four lines above (The Linux, Windows and macOS matrices) now leaves a reader to infer Windows behaves like macOS → Fix: "the ordinary Linux and Windows shards exclude only ipv6_required` and the dedicated hosted lane runs those items; the macOS command passes no marker filter at all".

[OPUS-REVIEWED] c450527

Verdict parsed from the review's SHA-scoped output markers for commit c450527df14cc50dccc80c61b96dbbe74497c24f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c450527df14cc50dccc80c61b96dbbe74497c24f: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@bolichen97
bolichen97 force-pushed the fix/macos-shard-cap-headroom branch from e79edfd to 5c07373 Compare September 26, 2026 09:16
@bolichen97

Copy link
Copy Markdown
Contributor Author

disposition=fixed span=925f96ac92f0

The best finding on this PR and the one I missed: the on-demand lane's ceiling is expressed in live RUNS with the multiplier only in prose ("each holds three shards"), so a 3 -> 4 matrix silently lifted its peak concurrent macos-15 jobs from 18 to 24 with nothing going red -- on the very pool whose starvation that ceiling exists to prevent.
Fixed by moving the ceiling with the count: LANE_MAX_LIVE_RUNS 6 -> 4, so 4 runs x 4 shards = 16 jobs, the same order as the 18 it was calibrated at and deliberately not more. The comment now says the quantity the pool actually feels is JOBS and that this number has to move whenever the shard count does, which is the drift that let one change the other invisibly. docs/ci/ci-and-reviews.md's table carries the new value and the same caveat.
test_macos_pool_ceiling_posix.py passes unchanged: it parameterises the ceiling rather than pinning 6.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 26, 2026
@bolichen97

bolichen97 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

disposition=fixed span=06dd12527d74

Legitimate: the rationale block above the job still derived the knob the other way ("3 shards, not Windows' 4 ... 3 is the smallest count that keeps the slowest shard inside a budget"), so the file carried two measured-sounding derivations that disagreed -- worse than either alone, because a reader cannot tell which one the code follows.
Rewritten to state the rule rather than a number: the count is the smallest that keeps the slowest shard inside the cap with room to spare, the cap's own derivation below carries the measurement, and raising the COUNT is what absorbs suite growth where raising the cap would only buy a longer runaway on a lane billed at ten times Linux. "duration-balanced shards" in the same block is now "file-assigned shards", since duration balancing left with pytest-split.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 26, 2026
@bolichen97

bolichen97 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

disposition=fixed span=90dfb3da3a34

Legitimate and all five passages are updated on this head. testing-conventions.md: the "Narrow by FILE" rule no longer says pytest-split is "retained for macOS" and now says CI uses scripts.ci_file_shards on every platform; the leak-locality passage says every platform assigns whole files; the macOS paragraph states four file-assigned shards, why the COUNT is the knob rather than the cap, and that test-durations.yml and .test_durations balance nothing now. docs/ci/ci-and-reviews.md: pytest-split is described as installed but invoked by no job, and the rollback note counts four matrix invocations instead of three.
Docs lint passes.

@bolichen97

Copy link
Copy Markdown
Contributor Author

disposition=accepted-and-deferred span=db2b04455c68

Correct on both halves, and they separate. The SPEC half is fixed on this head: no passage asserts macOS uses pytest-split any more (five updated across testing-conventions.md and docs/ci/ci-and-reviews.md).
The ORPHAN half is deferred to #14187 (deferred-finding, assigned, Due: 2026-10-17), and deliberately not folded in: test-durations.yml is referenced by test_ci_failure_annotations.py, test_workflow_checkout_credentials.py and test_coverage_omit_contract.py, so deleting it means moving those three pins in the same diff, and dropping a declared dependency deserves its own review rather than riding along inside a CI-capacity fix. Worth recording that test-durations.yml was ALREADY orphaned before this PR -- its header describes its purpose in terms of ci.yml, which moved to file sharding earlier -- so this change exposes that debt rather than creating it.

@bolichen97

Copy link
Copy Markdown
Contributor Author

disposition=fixed span=ffacf959459e

Already scheduled, before this suggestion landed: #14187 carries the deferred-finding label, an assignee and Due: 2026-10-17, and names the whole scope -- delete test-durations.yml, drop pytest-split==0.11.0, move the three test pins that reference the workflow, and remove the remaining doc reference at docs/ci/ci-and-reviews.md:135.
Agreed on the reason the suggestion gives: a dead workflow invites re-adoption. Worth recording that it was already dead before this PR -- its header describes its purpose in terms of ci.yml, which moved to file sharding earlier -- so the tracked work is pre-existing debt this change exposed.

@bolichen97

Copy link
Copy Markdown
Contributor Author

disposition=rebutted span=bc7f13e953d3

Declined, and the two lanes disagree here rather than me disagreeing with review: the Opus lane raised the OPPOSITE finding on the previous head (span 925f96ac92f0) -- that leaving the ceiling at 6 lifts this lane's peak concurrent macos-15 jobs from 18 to 24 with nothing going red. Keeping 6 is what that finding calls the defect.
The substance is that the ceiling's unit is runs but its SUBJECT is the hosted macOS pool, and what the pool feels is jobs: a run holds one per shard. Its own comment says why the number exists -- this lane once held 53 of 56 in-progress macOS jobs and a shard waited 14 hours for a runner while the signing paths queued behind it -- so a matrix change that raises peak jobs by a third on that pool is inside this PR's scope, not outside it. 4 runs x 4 shards = 16 jobs, under the 18 the ceiling was calibrated at; 5 would be 20, over it.
On the "one-job scope" point: the job whose shard count this PR changes is the one this ceiling multiplies, so the ceiling is a consequence of the change, not an unrelated edit. The comment now states that dependency explicitly so the next shard-count change cannot move the job ceiling silently again.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@bolichen97
bolichen97 force-pushed the fix/macos-shard-cap-headroom branch from 5c07373 to e4fe86a Compare September 26, 2026 09:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

disposition=fixed span=67a7bd4acab4

Legitimate and mine: the description still said "one workflow job and its pin test. No production code, no other job" after the diff had grown a second workflow and two docs. An undeclared ceiling change is exactly the shape a reader cannot audit.
Rewritten to enumerate all four files, and the ceiling hunk now carries its job-parity arithmetic in the description as well as in the comment: the ceiling counts RUNS but its subject is the hosted macOS pool, a run holds one job per shard, 6 x 3 = 18 was the calibrated number, leaving 6 would have made it 24, and 4 x 4 = 16 stays under it where 5 x 4 = 20 would not. It also says why the hunk belongs in this commit rather than its own: it is a consequence of the shard count this PR changes, and shipping the two apart would leave one head where the ceiling is 18 jobs and the next where it is 24.

@bolichen97

bolichen97 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor Author

disposition=fixed span=a238d30975ca

Accepted: "raising the count buys headroom" asserted a cap consequence I have not measured, and this PR's own description says the per-shard minutes are not proven yet. A spec sentence is the wrong place for the part that is still an expectation.
The causal claim is gone. The passage now states only the mechanism -- a higher cap bounds a runaway more loosely, a higher count divides the work -- and says the count is the knob to reach for rather than that it buys a specific outcome. Docs lint clean.

@bolichen97

Copy link
Copy Markdown
Contributor Author

disposition=accepted-and-deferred span=68a045beba8e

The count is right and I do not dispute it: zero --splits invocations remain, so pytest-split==0.11.0 and test-durations.yml have zero consumers. Tracked in #14187 -- deferred-finding, assigned, Due: 2026-10-17 -- with the full scope named: delete the workflow, drop the dependency, move the test pins, remove the doc references.
Why it is not folded in, counted rather than asserted: test-durations.yml is referenced by three test files, including a dedicated case in test_ci_failure_annotations.py that parses the workflow and a named entry in its annotation-matcher list, and it is used as a WORKED EXAMPLE in four passages of docs/ci/ci-and-reviews.md ("already pays for a full suite on main", "opens a rolling PR rather than committing directly, like test-durations.yml", "carry the same guard"). Removing it means rewriting prose that teaches a pattern through it, which is judgement, not deletion.
The debt also predates this change: the workflow's own header states its purpose in terms of ci.yml, which moved to file sharding before this PR existed, so it has been balancing nothing on any platform already. This change makes that visible; it did not create it.
Recording the round count: this is the second time this lane has raised the orphan, under a new span. If it returns a third time I will stop answering it and put the fold-in-or-defer choice to bolichen rather than restating the same reasoning.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Re-synced onto 0721b4a1a4 and pushed c450527df1. No content change -- the same single commit, rebased.

The reason is the base, not a red of mine: the four Frontend Tests (2)/(3) + Frontend Coverage Merge + Coverage Gate failures on the previous head were the main-level red that #14200 ("regenerate settings registry") fixed, and that merged minutes ago. This head is the first one built on a base that contains it.

green_age flagged seven overlapping files at import level (test_native_skill_projection.py, test_prompts.py, the Slack suites, test_telegram.py). Verified rather than assumed: those files at main's version, run together with this change and the gate pins, give 767 passed / 1 skipped.

All four review lanes were fresh with zero findings on the previous head; they will restamp on this one.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 26, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 26, 2026 23:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 26, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

disposition=fixed span=564cfcda1953

Correct and answered without a push, because the gap was in the description rather than in the tree: conftest.py is in this head and was missing from the file list. It is now item 7, named as ONE docstring line in _restore_log_record_factory -- the byte-near-identical twin of a sentence in testing-conventions.md, so leaving it would have made the two copies a test author consults disagree about which lane observes a log-factory leak. No executable line in that file changes.
"Six files" is now "Seven files", so the header and the list agree.

@bolichen97

Copy link
Copy Markdown
Contributor Author

disposition=accepted-and-deferred span=16dc68ebc817

The finding is right: I split that clause to keep macOS out of a Linux statement and dropped backend-test-windows, whose shard command carries the same -m "not ipv6_required" (ci.yml:930). The accurate clause is "the ordinary Linux and Windows shards exclude only ipv6_required", and I have that one-line edit ready.
Deferring the PUSH, not the fix, and saying plainly why: this head has just reached readiness: passed with 96/96 checks settled and zero failing, after eleven heads. A push discards that verdict and starts a fresh wave on a base that moved 54 commits in the last hour, so the cost of landing a prose correction now is a real chance of not getting back to a stable green today -- against a reader-facing inaccuracy in one clause of a CI doc.
Two ways to close it, bolichen's call: land it as a one-line follow-up PR after this merges, or say the word and I push it here and drive the next wave. If this PR needs a new head for any other reason, the edit rides along with it.

@bolichen97
bolichen97 merged commit 4d64908 into main Sep 26, 2026
147 of 150 checks passed
@bolichen97
bolichen97 deleted the fix/macos-shard-cap-headroom branch September 26, 2026 23:51
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 26, 2026
CrysisDeu added a commit that referenced this pull request Sep 26, 2026
`-n auto` on the macos-15 runner has resolved to ONE worker on every
nightly from at least 09-16 to 09-26: the budget plugin
(xdist_budget.py) sizes workers by free memory, the runner has ~3 GiB
free of 7 after the OS and toolchain, and the plugin's per-worker
reservation (3 GiB then, 16 GiB for macOS since #10840) never fits
twice. The log line is

  xdist worker budget: 1 of 3 workers (3.0 GiB free, 7 GiB installed)

so each shard ran its tests serially on a 3-core machine at about 23
tests a second. Measured with two workers on the previous 3-shard shape
(dispatch run 36228922099): 21-27 minutes a shard against 32-39 the same
day at one worker.

The plugin protects a laptop it does not know; this runner is a fixed
shape the lane can state outright, and an explicit count bypasses the
budget (it hooks pytest_xdist_auto_num_workers only). Two workers, not
three: each worker imports and holds its shard's files, and the 16 GiB
figure was measured on a 36 GiB laptop, not here. The gate test pins the
explicit count so a revert to `auto` is visible.

Sits on top of #14173 (four file-shards), which owns the shard count and
the cap; this changes only the workers per shard.

Refs #13720.
@bolichen97

Copy link
Copy Markdown
Contributor Author

Closing the measurement this PR could not make before merging -- the Design WATCH (span 085f621144ab) asked for the four shards' real minutes, and its bar was "well inside 40, which the 1.5x rule puts at about 27 or less".

Nightly run 36280765452, on 4d6490860c (this PR's merge commit), the first nightly with four file-shards:

shard            pytest step    job wall
macOS (1)         19.6 min      20.4 min   success
macOS (2)         25.0 min      25.6 min   success   <- slowest
macOS (3)         24.3 min      24.9 min   success
macOS (4)         22.9 min      23.5 min   success

At 3 shards with pytest-split the same lane read 33.0 / 40.3 (cancelled AT the cap, 98% through) / 32.2. So the slowest shard went 40.3 -> 25.0 minutes, which is 1.60x headroom against the 40-minute cap. The bar is met, and the WATCH's concern -- that hash assignment balances COUNT, not minutes -- is answered by the spread it produced: 19.6 to 25.0, max/min 1.28 in minutes against the 1.18 measured in test count.

One number worth recording because it settles the cost question the Opus lane raised twice: total macOS test time went from 105.5 minutes across 3 shards to 91.8 across 4. More shards, less total work -- because what this change removed was not distributed at all. Every shard had been importing the whole tree to discard three quarters of it.

The run finished 28/28 success, 0 skipped, 0 failed, and the publish chain RAN: CLI wheel, Docker image, Windows installer, six Linux packages, and sign + notarize + staple + distribution. The 09-25 and 09-26 nightlies skipped every one of those.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants