Skip to content

ci: raise Windows stability and macOS shard time limits - #36198

Merged
mbriones98 merged 1 commit into
mainfrom
ci/windows-macos-time-limits
Nov 13, 2026
Merged

mbriones98 merged 1 commit into
mainfrom
ci/windows-macos-time-limits

Conversation

@mbriones98

@mbriones98 mbriones98 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A large test PR can pass the Windows New-test stability leg, and a macOS backend shard on a slow runner can finish, instead of either being cancelled at its time limit.

Two CI jobs are cancelled at their timeout-minutes with no failing test:

  • New-test stability (windows-latest), limit 60. On run 43274625818 (a PR modifying about 500 test files), the Ubuntu leg passed in 52 minutes. The Windows leg was cancelled at the limit on attempt 2 and again on attempt 3. One Windows pass over those files takes about 25-27 minutes, and the job runs three passes.
  • macOS Tests / Backend Tests (macOS) (platform-tests.yml, called by macos-on-demand.yml and nightly.yml), limit 40. Shard 5 was cancelled at 40 minutes with no failing test on macOS Tests (on demand) runs 43237025822 (both attempts), 43281324254 and 43280020343, the last two on unrelated PRs. On another PR shards 6 and 7 were cancelled at the limit alongside a macOS arm64 runner capacity notice.

Why it matters

PR Readiness counts the cancelled New-test stability leg as blocking, so a PR whose tests all pass still can't become ready, and re-running only repeats the same cancel. A cancelled macOS shard returns no verdict on the on-demand lane, and the same 40-minute limit applies to the nightly that gates publishing.

Not a goal

  • Changing what either job runs, how many times New-test stability repeats (LEAF_REPEAT = 3), per-test timeouts, runners or the macOS shard count.
  • Changing the Ubuntu New-test stability leg's limit (it stays at 60).
  • Finding which macOS test stalls (the macOS log prints only progress dots).

What changed (motivation → approach → change)

Both jobs need more wall time on one platform only, so only that platform's limit rises.

  • .github/workflows/ci.yml, new-test-stability: timeout-minutes is now ${{ matrix.timeout_minutes || 60 }}, and the windows-latest matrix entry sets timeout_minutes: 90. Ubuntu keeps 60. Job-level timeout-minutes can read the matrix context (the runner context is not available there), and build-desktop.yml already uses this exact form for its macOS leg.
  • .github/workflows/platform-tests.yml, backend-test-macos: timeout-minutes 40 → 60, with a comment saying why. This supersedes the recorded decision that the 40-minute cap stays put while suite growth goes into the shard count. That rule still holds for growth: shards usually take 20-30 minutes. The cancels above are a different case: a slow or capacity-limited arm64 runner pushes a shard that has no failing test past 40 minutes. A shard that truly hangs is still cancelled, just 20 minutes later. test/test_macos_platform_tests_gate.py pins the new value. The spec that owns the cap (docs/system-specs/common/testing-conventions.md) and the comment in .github/workflows/macos-on-demand.yml now say the same.

Cost: a stalled Windows stability leg can now hold a runner 30 minutes longer, and a stalled macOS shard 20 minutes longer. Runs that finish today finish the same, because the limit is only a ceiling.

Backwards compatibility

Compatible: only two job time limits rise; every run that finished under the old limits runs the same way.

Removes nothing: the diff only raises two limits and updates the test that pins one of them.

Tests

  • test/test_macos_platform_tests_gate.py now asserts the macOS shard limit is 60.
  • test/test_ci_fleet_routing_expression_parity.py::test_merge_queue_readiness_budget_covers_ci_critical_path still passes. It reads the || 60 fallback of the new expression. Counting the Windows leg at 90, New-test stability's longest needs chain is 135 minutes, under the 220-minute merge-queue budget, and the critical path (210, through coverage-gate) does not change.
  • Ran all 78 test files that read ci.yml or platform-tests.yml, all 30 that read platform-tests.yml, macos-on-demand.yml or testing-conventions.md, and scripts/local-gate.py --base origin/main (4,384 related backend tests and 14,676 related frontend tests): all passed. flake8, black and isort are clean on the changed test file.

Proof: not provable -- docs or config only

Manual verification

N/A — CI config only; this PR's own CI runs both changed jobs (New-test stability because the diff modifies a test file, the macOS lane because macos-on-demand.yml treats a platform-tests.yml change as a darwin path).

Related Issues

Closes #36183

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@mbriones98
mbriones98 requested review from a team and KiroCrewCaptain as code owners November 12, 2026 23:33
@mbriones98 mbriones98 self-assigned this Nov 12, 2026
@mbriones98
mbriones98 requested a review from CrysisDeu November 12, 2026 23:33
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 15e7c7951f47e39f3772b1bc5bd1873da0a3bc87 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 15e7c7951f47e39f3772b1bc5bd1873da0a3bc87 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Both limits are raised for jobs that were cancelled with no failing test, on linked runs. The macOS pin that said "this number stays put" is replaced openly, with evidence attached.

Notes

  • Item 2 — symptom-level: "a slow or capacity-limited arm64 runner pushes a shard … past 40 minutes". The cause, a test that stalls, is excluded by "Finding which macOS test stalls" under Not a goal. Subtraction: put the cap back to 40 once the stalling test is found and fixed.

What this change ships

Inventory (4 items) — 3 justified

Intent: let a large test PR pass the Windows New-test stability leg, and let a macOS shard on a slow runner finish, instead of being cancelled at the time limit. FIX.

  1. The Windows New-test stability leg may now run 90 minutes instead of 60; Ubuntu stays at 60 — justified
  2. Each macOS backend shard may now run 60 minutes instead of 40, in both the on-demand lane and the nightly — symptom-level: the stalling test is unidentified, raising the cap only tolerates it
  3. The test that pins the macOS cap now expects 60 and its comment is rewritten — justified
  4. The testing-conventions spec and the comment in macos-on-demand.yml now give the new cap — justified

[FIRST-PRINCIPLES-REVIEWED] 15e7c79

Description read

[DESCRIPTION-READ] 511f319651adae33e76006466c03144fb1a2b05b18f5fed8b076c59213f52b3b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 15e7c7951f47e39f3772b1bc5bd1873da0a3bc87 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: two CI time limits go up on one platform each, both can be reverted cleanly, and the PR openly supersedes the "40 stays put" rule instead of treating it as a gap.

Notes

  • The 60-minute macOS cap only gives more room to a shard that stalls without failing. Nothing yet finds the stall, so file an issue with a recurrence threshold before the cap gets raised again.
  • Shard 5 was the one cancelled on three separate runs. That looks like one file assigned to that shard, more than general runner slowness. The uploaded log of the cancelled shard would show which test, and that test is the next thing to fix.

[DESIGN-REVIEWED] 15e7c79

Description read

[DESCRIPTION-READ] cdfc66c414b152ddad3d9f945215d89ffa071a411e7891d770cf8b2a3911f1b8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 15e7c7951f47e39f3772b1bc5bd1873da0a3bc87 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 15e7c79

Verdict parsed from the review's SHA-scoped output markers for commit 15e7c7951f47e39f3772b1bc5bd1873da0a3bc87.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 15e7c7951f47e39f3772b1bc5bd1873da0a3bc87 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 15e7c7951f47e39f3772b1bc5bd1873da0a3bc87 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 15e7c79

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 15e7c7951f47e39f3772b1bc5bd1873da0a3bc87 false-positive: <reason, 20 to 500 characters>

@mbriones98
mbriones98 force-pushed the ci/windows-macos-time-limits branch from 76589f8 to f12b84b Compare November 12, 2026 23:56
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

CI-only change: the Windows New-test stability leg goes to 90 minutes through a matrix-keyed timeout-minutes (a valid context; Ubuntu falls back to 60), and the macOS backend shard limit goes from 40 to 60, with the gate test, the spec and the comment updated to match. new-test-stability runs only on pull_request, so the merge-queue budget is unaffected, and the budget test reads the || 60 fallback correctly. test_workflow_bounded_checkout still sees ≥20. The macOS on-demand occupancy ceiling counts live jobs and has no time assumption. Nightly publish jobs only use needs. All CI is green, including both changed jobs. Only problems found: a stale comment and a missing pin test for the Windows value.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 15e7c7951f · 33 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@mbriones98

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/common/model-selection.md, test/test_workflows_runner.py since this head went green (2503fd9 -> a02e90c); CI re-running.

@mbriones98
mbriones98 force-pushed the ci/windows-macos-time-limits branch from f12b84b to 02d7b2b Compare November 13, 2026 01:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 13, 2026
@mbriones98

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 8) on 02d7b2b fails test/test_reply_format_once.py::TestEveryMessageWhereTheBlockCannotBeTakenOut::test_the_table_names_every_selectable_harness (missing mimocode). It fails the same way on clean main 9b32941, from 1be8d2c (#24732); this PR does not touch that test. Waiting on #36290, then updating this branch.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
New-test stability's Windows leg gets 90 minutes through a per-leg
matrix value; the Ubuntu leg stays at 60. The macOS backend shards
in platform-tests.yml go from 40 to 60 minutes. Both jobs were being
cancelled at their limits with no failing test.

Closes #36183
@mbriones98
mbriones98 force-pushed the ci/windows-macos-time-limits branch from 02d7b2b to 15e7c79 Compare November 13, 2026 03:22
@mbriones98

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_reply_format_once.py (the red-main fix #36290, eee70be) since this head was tested (a02e90c -> eee70be); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@mbriones98
mbriones98 enabled auto-merge (squash) November 13, 2026 05:35

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the full diff. Fixes #36183, CI time limits only. The Windows leg of the new-test stability job now gets 90 minutes through a per-leg timeout_minutes matrix value, with 60 as the fallback, so Ubuntu is unchanged. That fits three passes at the 25-27 minutes one pass takes there. The macOS shard cap goes from 40 to 60 minutes for slow or capacity-limited arm64 runners. The shard count stays the lever for suite growth, as the testing-conventions text and the pinned test now say. The comments, the docs and test_macos_platform_tests_gate.py all move to 60 together. No product code, no new refusal. LGTM.

@mbriones98
mbriones98 merged commit 5d9ba40 into main Nov 13, 2026
110 of 114 checks passed
@mbriones98
mbriones98 deleted the ci/windows-macos-time-limits branch November 13, 2026 06:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: Windows New-test stability and macOS backend shards are cancelled at their time limits

3 participants