Repository navigation
ci: raise Windows stability and macOS shard time limits - #36198
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Both limits are raised for jobs that were cancelled with no failing test, on linked runs. The macOS pin that said "this number stays put" is replaced openly, with evidence attached. Notes
What this change shipsInventory (4 items) — 3 justifiedIntent: let a large test PR pass the Windows New-test stability leg, and let a macOS shard on a slow runner finish, instead of being cancelled at the time limit. FIX.
[FIRST-PRINCIPLES-REVIEWED] 15e7c79 Description read[DESCRIPTION-READ] 511f319651adae33e76006466c03144fb1a2b05b18f5fed8b076c59213f52b3b That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: two CI time limits go up on one platform each, both can be reverted cleanly, and the PR openly supersedes the "40 stays put" rule instead of treating it as a gap. Notes
[DESIGN-REVIEWED] 15e7c79 Description read[DESCRIPTION-READ] cdfc66c414b152ddad3d9f945215d89ffa071a411e7891d770cf8b2a3911f1b8 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
76589f8 to
f12b84b
Compare
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. CI-only change: the Windows New-test stability leg goes to 90 minutes through a matrix-keyed timeout-minutes (a valid context; Ubuntu falls back to 60), and the macOS backend shard limit goes from 40 to 60, with the gate test, the spec and the comment updated to match. new-test-stability runs only on pull_request, so the merge-queue budget is unaffected, and the budget test reads the An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
f12b84b to
02d7b2b
Compare
|
Backend Tests (3.12, 8) on 02d7b2b fails |
New-test stability's Windows leg gets 90 minutes through a per-leg matrix value; the Ubuntu leg stays at 60. The macOS backend shards in platform-tests.yml go from 40 to 60 minutes. Both jobs were being cancelled at their limits with no failing test. Closes #36183
02d7b2b to
15e7c79
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Read the full diff. Fixes #36183, CI time limits only. The Windows leg of the new-test stability job now gets 90 minutes through a per-leg timeout_minutes matrix value, with 60 as the fallback, so Ubuntu is unchanged. That fits three passes at the 25-27 minutes one pass takes there. The macOS shard cap goes from 40 to 60 minutes for slow or capacity-limited arm64 runners. The shard count stays the lever for suite growth, as the testing-conventions text and the pinned test now say. The comments, the docs and test_macos_platform_tests_gate.py all move to 60 together. No product code, no new refusal. LGTM.
Problem / Motivation
Goal: A large test PR can pass the Windows New-test stability leg, and a macOS backend shard on a slow runner can finish, instead of either being cancelled at its time limit.
Two CI jobs are cancelled at their
timeout-minuteswith no failing test:platform-tests.yml, called bymacos-on-demand.ymlandnightly.yml), limit 40. Shard 5 was cancelled at 40 minutes with no failing test onmacOS Tests (on demand)runs 43237025822 (both attempts), 43281324254 and 43280020343, the last two on unrelated PRs. On another PR shards 6 and 7 were cancelled at the limit alongside a macOS arm64 runner capacity notice.Why it matters
PR Readiness counts the cancelled New-test stability leg as blocking, so a PR whose tests all pass still can't become ready, and re-running only repeats the same cancel. A cancelled macOS shard returns no verdict on the on-demand lane, and the same 40-minute limit applies to the nightly that gates publishing.
Not a goal
LEAF_REPEAT= 3), per-test timeouts, runners or the macOS shard count.What changed (motivation → approach → change)
Both jobs need more wall time on one platform only, so only that platform's limit rises.
.github/workflows/ci.yml,new-test-stability:timeout-minutesis now${{ matrix.timeout_minutes || 60 }}, and thewindows-latestmatrix entry setstimeout_minutes: 90. Ubuntu keeps 60. Job-leveltimeout-minutescan read thematrixcontext (therunnercontext is not available there), andbuild-desktop.ymlalready uses this exact form for its macOS leg..github/workflows/platform-tests.yml,backend-test-macos:timeout-minutes40 → 60, with a comment saying why. This supersedes the recorded decision that the 40-minute cap stays put while suite growth goes into the shard count. That rule still holds for growth: shards usually take 20-30 minutes. The cancels above are a different case: a slow or capacity-limited arm64 runner pushes a shard that has no failing test past 40 minutes. A shard that truly hangs is still cancelled, just 20 minutes later.test/test_macos_platform_tests_gate.pypins the new value. The spec that owns the cap (docs/system-specs/common/testing-conventions.md) and the comment in.github/workflows/macos-on-demand.ymlnow say the same.Cost: a stalled Windows stability leg can now hold a runner 30 minutes longer, and a stalled macOS shard 20 minutes longer. Runs that finish today finish the same, because the limit is only a ceiling.
Backwards compatibility
Compatible: only two job time limits rise; every run that finished under the old limits runs the same way.
Removes nothing: the diff only raises two limits and updates the test that pins one of them.
Tests
test/test_macos_platform_tests_gate.pynow asserts the macOS shard limit is 60.test/test_ci_fleet_routing_expression_parity.py::test_merge_queue_readiness_budget_covers_ci_critical_pathstill passes. It reads the|| 60fallback of the new expression. Counting the Windows leg at 90, New-test stability's longestneedschain is 135 minutes, under the 220-minute merge-queue budget, and the critical path (210, throughcoverage-gate) does not change.ci.ymlorplatform-tests.yml, all 30 that readplatform-tests.yml,macos-on-demand.ymlortesting-conventions.md, andscripts/local-gate.py --base origin/main(4,384 related backend tests and 14,676 related frontend tests): all passed. flake8, black and isort are clean on the changed test file.Proof: not provable -- docs or config only
Manual verification
N/A — CI config only; this PR's own CI runs both changed jobs (New-test stability because the diff modifies a test file, the macOS lane because
macos-on-demand.ymltreats aplatform-tests.ymlchange as a darwin path).Related Issues
Closes #36183
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)