Skip to content

ci: give New-test stability 150 min on Windows, 90 on Ubuntu - #36724

Merged
bolichen97 merged 1 commit into
mainfrom
ci/stability-limits-120-75
Nov 13, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
ci/stability-limits-120-75

Conversation

@mbriones98

@mbriones98 mbriones98 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A large test PR can pass New-test stability on both legs instead of being cancelled at the job's time limit.

New-test stability is still cancelled at its timeout-minutes on #31863 (head 7c763a5, 403 modified test files), with no failing test:

  • Windows (limit 90): passes took 32:24 and 32:17, about 18 minutes of each spent collecting. Three passes do not fit, and the leg was cancelled at 90 three times with 0 failures.
  • Ubuntu (limit 60): one run passed in 42.5 minutes; two were cancelled at 60 on slow runners. One of them collected for about 15 minutes, took 24:09 and 22:40 for its first two passes and was cut off in the third, so it needed about 72 minutes.

Why it matters

PR Readiness counts a cancelled leg as blocking, so a large test PR whose tests all pass cannot become ready, and a re-run repeats the same cancel.

Not a goal

  • Changing what the job selects, how many times it repeats (LEAF_REPEAT = 3), its per-test timeouts or its runners.
  • Speeding up Windows collection.

What changed (motivation → approach → change)

Three passes need about 100 minutes on Windows and about 72 on a slow Ubuntu runner, so each leg gets a limit with room above that. In .github/workflows/ci.yml, job new-test-stability, the Windows matrix entry's timeout_minutes goes from 90 to 150 and the Ubuntu entry gets an explicit timeout_minutes: 90 (it used the || 60 default). The comment above the limit states the measured pass times; the run they come from is the one cited above.

The merge-queue budget test in test/test_ci_fleet_routing_expression_parity.py counted only the || 60 fallback for this job, so it could not see either leg's own limit. It now counts the largest of the fallback and every leg's value, and a new test pins both legs at 90 and 150.

Cost: a stalled leg can now hold its runner 60 minutes longer on Windows and 30 on Ubuntu. A run that finishes today finishes the same, because the limit is only a ceiling.

Backwards compatibility

Compatible: only two job time limits rise; every run that finished under the old limits runs the same way.

Removes nothing: the diff raises two limits, rewrites one comment and widens what one test counts.

Tests

  • test/test_ci_fleet_routing_expression_parity.py::test_new_test_stability_sets_each_legs_own_limit (new) pins the Ubuntu leg at 90 and the Windows leg at 150.
  • test_merge_queue_readiness_budget_covers_ci_critical_path now counts each leg's own timeout_minutes, not only the || 60 fallback. With the Windows leg temporarily set to 300 it fails; before this change it passed. At 150 this job's longest needs chain is about 195 minutes, under the 220-minute merge-queue budget, and ci.yml's critical path (210, through coverage-gate) does not change.
  • Determinism: the edited test file passed 3 repeats and a shuffled order (22 tests each), and test/test_flake_pattern_ratchet.py passes.
  • No doc pins this job's limits (docs/ci/ci-and-reviews.md describes the job without one).
  • Ran all 78 test files that read ci.yml (4,808 passed); flake8, black and isort are clean on the changed test file.

Proof: not provable -- docs or config only

Manual verification

N/A — CI config and a test only. This PR modifies a test file, so its own New-test stability run exercises the new limits on a small scope; the large case takes effect on the next large test PR.

Related Issues

Closes #36722

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

🤖 Opened by the Crew Tasks Bot for mdbrio.

@mbriones98
mbriones98 requested review from a team and KiroCrewCaptain as code owners November 13, 2026 16:16
@mbriones98 mbriones98 self-assigned this Nov 13, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 831c750d254c3aa49d0da3254d2d77486fab9f22 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 831c750d254c3aa49d0da3254d2d77486fab9f22 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Both limit raises cite measured pass times from #31863 and are tied to #36722. The test change fixes a real blind spot: the budget check could not see the per-leg limits.

Notes

  • Item 5 — inherited: the new pin test hard-codes {"ubuntu-latest": 90, "windows-latest": 150}. The widened budget test already catches the harmful direction, a limit too high. The pin only blocks lowering a limit, and the run being cancelled already shows that. Subtraction: delete test_new_test_stability_sets_each_legs_own_limit and rely on the widened cap_minutes in test_merge_queue_readiness_budget_covers_ci_critical_path.

What this change ships

Inventory (5 items) — 4 justified

Intent: let a large test PR finish New-test stability's three passes instead of being cancelled at the time limit. This is a FIX.

  1. Windows New-test stability leg may now run up to 150 minutes instead of 90 — justified
  2. Ubuntu leg gets its own 90-minute limit instead of falling back to the 60-minute default — justified
  3. The workflow comment states the new measured pass and collection times — justified
  4. The merge-queue budget test now counts each leg's own limit, not just the || 60 fallback — justified
  5. A new test pins both legs at exactly 90 and 150 — rides along, inherited: the widened budget test already bounds the harmful direction

[FIRST-PRINCIPLES-REVIEWED] 831c750

Description read

[DESCRIPTION-READ] a7c4577de08e762d792ffc033223cb66380f0b2aaba7ac2887a0382ec2c8dbf7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 831c750d254c3aa49d0da3254d2d77486fab9f22 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The new limits come from measured pass times, and the merge-queue budget test now counts each leg's own limit, so a future increase that overruns the 220-minute budget will fail the test.

Notes

  • With 150 minutes on Windows, this job's longest needs chain is about 195 minutes, leaving about 25 minutes under the 220-minute merge-queue poll budget. Making Windows collection faster is a better next step than raising the limit again.

[DESIGN-REVIEWED] 831c750

Description read

[DESCRIPTION-READ] 55962e6c79790c31cb4c38a4fd8472019be8467c143718eb7bd1506df3689063

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 831c750d254c3aa49d0da3254d2d77486fab9f22 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 831c750

Verdict parsed from the review's SHA-scoped output markers for commit 831c750d254c3aa49d0da3254d2d77486fab9f22.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 831c750d254c3aa49d0da3254d2d77486fab9f22 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 831c750d254c3aa49d0da3254d2d77486fab9f22 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 831c750

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 831c750d254c3aa49d0da3254d2d77486fab9f22 false-positive: <reason, 20 to 500 characters>

@mbriones98 mbriones98 changed the title ci: give New-test stability 120 min on Windows, 75 on Ubuntu ci: give New-test stability 120 min on Windows, 90 on Ubuntu Nov 13, 2026
@mbriones98
mbriones98 force-pushed the ci/stability-limits-120-75 branch from 1835106 to c704701 Compare November 13, 2026 16:29
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

The PR raises the New-test stability time limits: Windows from 90 to 150 minutes, and Ubuntu gets its own 90 instead of the || 60 default. This fits the measured runtimes in the description and leaves the selection logic, the 3 repeats and the per-test timeouts unchanged. The merge-queue budget test now reads each leg's own limit. The job's longest dependency chain becomes about 195 minutes (changes → backend-lint → lint-gate → 150), under the 220-minute budget, and the job only runs on pull requests, so it never runs in the merge queue. No other ci.yml timeout uses a per-leg matrix.<key> value, so the new parsing changes no other job's count. No other code, watchdog or doc depends on these limits. A new test pins both legs' values.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit ea382cce5a · 34 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@mbriones98

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in .github/workflows/codex-review.yml (same directory as ci.yml) since this head went green (6a35758 -> 0e5b203); CI re-running. The patch is unchanged (range-diff: identical).

@mbriones98
mbriones98 force-pushed the ci/stability-limits-120-75 branch from c704701 to c0925f4 Compare November 13, 2026 17:35
@mbriones98 mbriones98 changed the title ci: give New-test stability 120 min on Windows, 90 on Ubuntu ci: give New-test stability 150 min on Windows, 90 on Ubuntu Nov 13, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@mbriones98
mbriones98 force-pushed the ci/stability-limits-120-75 branch from c0925f4 to ea382cc Compare November 13, 2026 17:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 13, 2026
A large test PR still ran past the job's limits with no failing test:
one Windows pass over about 400 files takes about 32 minutes, so three
do not fit in 90, and a slow Ubuntu runner needs about 72 minutes, past
60. The Windows leg's matrix limit goes to 150 and the Ubuntu leg gets
an explicit 90.

The merge-queue budget test now counts each leg's own limit rather than
only the shared fallback, and a new test pins both legs.

Closes #36722
@mbriones98
mbriones98 force-pushed the ci/stability-limits-120-75 branch from ea382cc to 831c750 Compare November 13, 2026 20:35
@mbriones98

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in nothing this PR touches; rebased onto the red-main fix #36804 (af8d570) so Coverage Gate measures it (c60b886 -> af8d570). Patch unchanged (range-diff ea382cc = 831c750); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 20:36
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97
bolichen97 merged commit 648d4c0 into main Nov 13, 2026
101 of 104 checks passed
@bolichen97
bolichen97 deleted the ci/stability-limits-120-75 branch November 13, 2026 21:53
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: New-test stability still cancelled at its limits on large test PRs (Windows 90, Ubuntu 60)

3 participants