Repository navigation
ci(matrix): lift the fan-out cap on CodeBuild-routed shards - #24738
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the cap now lifts only on CodeBuild-routed runs, which addresses the idle-container timeout at its cause, and hosted and fork runs keep the cap of 4. Notes
[DESIGN-REVIEWED] cc304f4 Description read[DESCRIPTION-READ] 13bb93b5526e94a8fc10364a4101666c9f9594ece1a5301e567ebb12f4d5c668 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the cap is lifted only where CodeBuild was already starting all 8 containers, and the hosted fallback stays at 4. What this change shipsInventory (6 items) — 4 justifiedIntent: stop CodeBuild-routed backend shards 5–8 from timing out while they wait behind
[FIRST-PRINCIPLES-REVIEWED] cc304f4 Description read[DESCRIPTION-READ] de6b12bd856795ffe9a779673226db98856a15a778a26833cd96b5e1c358d277 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
4d60ea6 to
58fc52e
Compare
Dispositions for head
|
58fc52e to
02c8f85
Compare
|
Backend Lint is red on this head with |
02c8f85 to
99ebc37
Compare
|
Backend Tests shard 2 (Linux + Windows) failed with XPASS(strict) on |
99ebc37 to
08ce036
Compare
Dispositions for head
|
dc96c4b to
2643082
Compare
buluoray
left a comment
There was a problem hiding this comment.
Requesting changes on 2643082.
The code change is correct, and I'd approve it as-is apart from one doc-sync item.
What I checked:
- Only CodeBuild runs are uncapped. The 8-wide cap is selected only when the job's own
runs-onoutput carries thecodebuild-label.changesemits that label only for this repository, an admitted actor, and push, merge_group, or a same-repositoryopened/synchronizepull_request. Fork PRs, other PR activities and non-admitted actors resolve toubuntu-latest/windows-latestand keepmax-parallel: 4. The workflow usespull_request, notpull_request_target. - On this head's CI run, all 8 Linux shards and all 8 Windows shards started within seconds of each other on the CodeBuild labels. The hosted-fallback branch was not exercised by this run; I checked it from source only.
- The diff adds no
run:line, no secret and no PR-controlled string. The expression reads only thechangesjob's own output. - The new
test_workflow_matrix_max_parallel.pyrules hold the shape tightly. Three mutations each turn a named test red: a hosted fallback above 4, a predicate testing a different output thanruns-on, and a predicate prefix other thancodebuild-.
Requested change:
docs/ci/ci-and-reviews.md(around line 136) still says "backend-testcounts twice:max-parallel: 4runs its 8 shards in two rounds". After this change that holds only on the hosted fallback. Please update it in this PR so it says CodeBuild-routed runs (including admitted merge_group runs) run all 8 shards in one round, the budget stays sized for the hosted two-round worst case, and the 4-cap exists to protect the shared hosted pool. A line in the CodeBuild section would also help: on CodeBuild the only remaining fan-out bound is the CodeBuild account concurrency, andci-runner-watchdog.ymlalready treats saturating it as a known state. The repo's rule is that the owning spec changes in the same commit as the behaviour it documents.
Non-blocking:
test/test_ci_fleet_routing_expression_parity.pyimportsreinside the function body; a module-level import would match the rest of the file.
CI on this head: 53 success, 9 skipped, 0 failures, and PR Readiness passed. Design and First Principles PASS. Opus and GPT report no findings. Every earlier-head BLOCK was answered with a fix commit.
|
HOLD on 2643082. The workflow change looks correct. The 8-wide cap applies only when the job's own runner output carries the Also non-blocking: the description says three test files changed, but — crew-pr-reviewer (r129) |
2643082 to
3ee6d4d
Compare
|
@buluoray — thanks for the review. Two changes at
|
buluoray
left a comment
There was a problem hiding this comment.
Approving 3ee6d4d. This supersedes my request-changes review on 2643082. docs/ci/ci-and-reviews.md now says the two-round max-parallel: 4 schedule applies only on the hosted fallback, that CodeBuild (including admitted merge_group runs) starts all 8 shards in one round, and that the remaining fan-out bound there is the CodeBuild account concurrency, which ci-runner-watchdog already treats as a known state. The rest of the interdiff only hoists the repeated local import re in the test to one module import. The workflow logic is unchanged from the version I reviewed.
|
Rebased: main moved in |
3ee6d4d to
cc304f4
Compare
Problem / Motivation
Goal: Make the matrix fan-out cap conditional on runner type so CodeBuild-routed shards all start in the first wave.
The
strategy.max-parallel: 4cap protects GitHub's shared hosted runner pool. For jobs routed to the team's AWS CodeBuild fleet, this cap causes a different failure: CodeBuild starts a build container for every shard in the matrix at the first webhook, but GitHub's scheduler respectsmax-paralleland holds shards 5-8 in a queue until shards 1-4 complete. Those waiting runners sit idle burning against CodeBuild's 60-minute build-start timeout (counted from container creation, not from when a job is handed to the runner). When the timeout fires, CodeBuild kills the container and GitHub logs "The self-hosted runner lost communication with the server" with no failed step — the shard appears cancelled, not failed.Measured on run 40399332194 (2026-10-26): CodeBuild starts 8 runner containers at 21:04; GitHub hands out jobs 5-8 only at ~21:33–21:40 (after shards 1–4 finish); CodeBuild's timeout kills shards 5-8 at ~22:15. This is the primary source of lost-shard reruns in the repo (~200/day).
Why it matters
Each stranded shard requires a human to notice, open the Actions tab, and trigger a full rerun. The CodeBuild account's 300-job quota is consumed by idle containers that never run a test.
Not a goal
release.yml,platform-tests.yml, or any other workflow file.What changed (motivation → approach → change)
.github/workflows/ci.ymltwo jobs have conditionalmax-parallelreplacing the literal 4. Thechangesjob resolver writesubuntu-latest(orwindows-latest) on the hosted fallback, never an empty string. UsingstartsWith(..., 'codebuild-')tests the label prefix directly: it is true only when the resolver assigned a CodeBuild label, and false on all hosted or fork runs.This evaluates to 8 on CodeBuild (all 8 shards start at once) and to 4 on every hosted-runner or fork run (preserving the existing shared-pool protection).
runs-onbackend-testlinux_runner_large || ubuntu-latest, 8 shardsstartsWith(linux_runner_large, 'codebuild-') && 8 || 4backend-test-windowswindows_runner || windows-latest, 8 shardsstartsWith(windows_runner, 'codebuild-') && 8 || 4backend-lintlinux_runner_large || ubuntu-latest, 1 shardfrontend-testlinux_runner_large || ubuntu-latest, 4 shardswheel-closureubuntu-latestonlybackend-test-ipv6matrix.os= ubuntu/windows hostede2e-boot-matrixThree test files are updated:
test_ci_nonroot_boundary.pyandtest_ci_windows_canary.pyupdate the exact strategy assertion, andtest_workflow_matrix_max_parallel.pyreplaces the earlier approach with a single parametrized invariant derived from_CODEBUILD_EXPR_RE: the hosted fallback cap (group 3) must be <= 4 (shared GitHub-runner pool limit), the CodeBuild cap (group 2) must be >= the hosted cap and <= the matrix leg count (no artificial throttle), and the output name (group 1) must match theneeds.changes.outputs.*referenced in the job's ownruns-onso the predicate tracks the runner it guards.Backwards compatibility
Compatible: on every hosted-runner or fork run the expression evaluates to 4, identical to the previous literal. The CodeBuild fleet receives cap 8, which is the full 8-shard matrix size.
Removes nothing: no API, config, data, or user-visible capability is removed.
Tests
Run 40466343040 on head
99ebc37c— all 8 Linux backend-test shards started within 6 seconds of each other, confirming the cap lifts on CodeBuild:Spread: 6 seconds (07:31:02–07:31:08). With the old cap of 4, shards 5–8 would queue until shards 1–4 finish (~60 min later) and then time out on CodeBuild.
Manual verification
N/A — effect is observable in a CodeBuild run: all 8 backend-test shards should start within the first wave instead of 30+ min after shards 1–4 finish.
Screenshots / video
N/A — no user-visible UI change.
Related Issues
Closes #24731