Skip to content

ci(matrix): lift the fan-out cap on CodeBuild-routed shards - #24738

Merged
CrysisDeu merged 1 commit into
mainfrom
unflaky/codebuild-matrix-fanout
Oct 27, 2026
Merged

CrysisDeu merged 1 commit into
mainfrom
unflaky/codebuild-matrix-fanout

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make the matrix fan-out cap conditional on runner type so CodeBuild-routed shards all start in the first wave.

The strategy.max-parallel: 4 cap protects GitHub's shared hosted runner pool. For jobs routed to the team's AWS CodeBuild fleet, this cap causes a different failure: CodeBuild starts a build container for every shard in the matrix at the first webhook, but GitHub's scheduler respects max-parallel and holds shards 5-8 in a queue until shards 1-4 complete. Those waiting runners sit idle burning against CodeBuild's 60-minute build-start timeout (counted from container creation, not from when a job is handed to the runner). When the timeout fires, CodeBuild kills the container and GitHub logs "The self-hosted runner lost communication with the server" with no failed step — the shard appears cancelled, not failed.

Measured on run 40399332194 (2026-10-26): CodeBuild starts 8 runner containers at 21:04; GitHub hands out jobs 5-8 only at ~21:33–21:40 (after shards 1–4 finish); CodeBuild's timeout kills shards 5-8 at ~22:15. This is the primary source of lost-shard reruns in the repo (~200/day).

Why it matters

Each stranded shard requires a human to notice, open the Actions tab, and trigger a full rerun. The CodeBuild account's 300-job quota is consumed by idle containers that never run a test.

Not a goal

  • Changing the hosted-runner cap (shared pool protection still applies).
  • Touching release.yml, platform-tests.yml, or any other workflow file.
  • Lifting the cap on jobs that are hosted-only or whose matrix never exceeds the current cap.

What changed (motivation → approach → change)

.github/workflows/ci.yml two jobs have conditional max-parallel replacing the literal 4. The changes job resolver writes ubuntu-latest (or windows-latest) on the hosted fallback, never an empty string. Using startsWith(..., 'codebuild-') tests the label prefix directly: it is true only when the resolver assigned a CodeBuild label, and false on all hosted or fork runs.

max-parallel: ${{ startsWith(needs.changes.outputs.linux_runner_large, 'codebuild-') && 8 || 4 }}

This evaluates to 8 on CodeBuild (all 8 shards start at once) and to 4 on every hosted-runner or fork run (preserving the existing shared-pool protection).

Job runs-on Decision
backend-test linux_runner_large || ubuntu-latest, 8 shards startsWith(linux_runner_large, 'codebuild-') && 8 || 4
backend-test-windows windows_runner || windows-latest, 8 shards startsWith(windows_runner, 'codebuild-') && 8 || 4
backend-lint linux_runner_large || ubuntu-latest, 1 shard Keep 4 (matrix has 1 leg; cap never activates)
frontend-test linux_runner_large || ubuntu-latest, 4 shards Keep 4 (cap already equals matrix size)
wheel-closure ubuntu-latest only Keep 4 (hosted; no CodeBuild path)
backend-test-ipv6 matrix.os = ubuntu/windows hosted Keep 4 (no CodeBuild path)
e2e-boot-matrix complex, max 3 legs Keep 4 (cap exceeds max matrix size; never throttles)

Three test files are updated: test_ci_nonroot_boundary.py and test_ci_windows_canary.py update the exact strategy assertion, and test_workflow_matrix_max_parallel.py replaces the earlier approach with a single parametrized invariant derived from _CODEBUILD_EXPR_RE: the hosted fallback cap (group 3) must be <= 4 (shared GitHub-runner pool limit), the CodeBuild cap (group 2) must be >= the hosted cap and <= the matrix leg count (no artificial throttle), and the output name (group 1) must match the needs.changes.outputs.* referenced in the job's own runs-on so the predicate tracks the runner it guards.

Backwards compatibility

Compatible: on every hosted-runner or fork run the expression evaluates to 4, identical to the previous literal. The CodeBuild fleet receives cap 8, which is the full 8-shard matrix size.

Removes nothing: no API, config, data, or user-visible capability is removed.

Tests

Run 40466343040 on head 99ebc37c — all 8 Linux backend-test shards started within 6 seconds of each other, confirming the cap lifts on CodeBuild:

Shard Started (UTC)
Backend Tests (3.12, 1) 07:31:05
Backend Tests (3.12, 2) 07:31:07
Backend Tests (3.12, 3) 07:31:02
Backend Tests (3.12, 4) 07:31:02
Backend Tests (3.12, 5) 07:31:06
Backend Tests (3.12, 6) 07:31:06
Backend Tests (3.12, 7) 07:31:07
Backend Tests (3.12, 8) 07:31:08

Spread: 6 seconds (07:31:02–07:31:08). With the old cap of 4, shards 5–8 would queue until shards 1–4 finish (~60 min later) and then time out on CodeBuild.

Manual verification

N/A — effect is observable in a CodeBuild run: all 8 backend-test shards should start within the first wave instead of 30+ min after shards 1–4 finish.

Screenshots / video

N/A — no user-visible UI change.

Related Issues

Closes #24731

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 27, 2026 05:35
@CrysisDeu
CrysisDeu requested a review from mbriones98 October 27, 2026 05:35
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision cc304f42eec82238b5fde50b04cddf672f974d45 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of cc304f42eec82238b5fde50b04cddf672f974d45 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the cap now lifts only on CodeBuild-routed runs, which addresses the idle-container timeout at its cause, and hosted and fork runs keep the cap of 4.

Notes

  • test_ci_fleet_routing_expression_parity.py reads the hosted fallback with a loose \|\|\s*(\d+)\s*\}\} search. If a future max-parallel expression doesn't match it, the cap is quietly treated as None and the job counts as one round. The two budget tests could share _CODEBUILD_EXPR_RE, so a new shape fails loudly instead of under-counting the budget.

[DESIGN-REVIEWED] cc304f4

Description read

[DESCRIPTION-READ] 13bb93b5526e94a8fc10364a4101666c9f9594ece1a5301e567ebb12f4d5c668

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of cc304f42eec82238b5fde50b04cddf672f974d45 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the cap is lifted only where CodeBuild was already starting all 8 containers, and the hosted fallback stays at 4.

What this change ships

Inventory (6 items) — 4 justified

Intent: stop CodeBuild-routed backend shards 5–8 from timing out while they wait behind max-parallel: 4. This is a FIX (#24731, measured run 40399332194).

  1. On CodeBuild, all 8 Linux backend-test shards now start in one wave. — justified
  2. On CodeBuild, all 8 Windows backend-test-windows shards now start in one wave. — justified
  3. The repo-wide max-parallel invariant test now accepts one pinned CodeBuild expression form, with hosted cap ≤4, CodeBuild cap ≤ leg count, and the output matching runs-on. — justified
  4. The readiness-budget test reads the || N hosted fallback for its worst-case round count. — justified
  5. Function-local import re moved to module scope in test_ci_fleet_routing_expression_parity.py. — rides along
  6. ci-and-reviews.md now says CodeBuild runs one round and gains a sentence on the 300-build account bound. — undeclared

[FIRST-PRINCIPLES-REVIEWED] cc304f4

Description read

[DESCRIPTION-READ] de6b12bd856795ffe9a779673226db98856a15a778a26833cd96b5e1c358d277

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed cc304f42eec82238b5fde50b04cddf672f974d45 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] cc304f4

Verdict parsed from the review's SHA-scoped output markers for commit cc304f42eec82238b5fde50b04cddf672f974d45.

False positive or not applicable? A repository writer can comment:
/ai-review override fable cc304f42eec82238b5fde50b04cddf672f974d45 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of cc304f42eec82238b5fde50b04cddf672f974d45 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] cc304f4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt cc304f42eec82238b5fde50b04cddf672f974d45 false-positive: <reason, 20 to 500 characters>

@CrysisDeu
CrysisDeu force-pushed the unflaky/codebuild-matrix-fanout branch from 4d60ea6 to 58fc52e Compare October 27, 2026 05:45
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Dispositions for head 58fc52ef573886714c5a74e6c3bb0ec751e5263f

First Principles, Opus, GPT — all three raised the same root defect. fixed at 58fc52e.

BLOCK — != '' is always true; hosted fallback resolves to ubuntu-latest / windows-latest, not an empty string.

The predicate is now startsWith(needs.changes.outputs.linux_runner_large, 'codebuild-') (and windows_runner equivalent). The resolver's else-branch writes a plain hosted label, which does not start with codebuild-, so this expression is false on every hosted and fork run.

First Principles additional items (fixed):

  • backend-lint reverted to literal max-parallel: 4 (1-leg matrix, cap never activates, change was unjustified).
  • _CODEBUILD_EXPR_RE regex replaced with an exact frozenset of the two expression strings; a new parametrized shape test asserts each entry uses startsWith(..., 'codebuild-') and not != ''.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/codebuild-matrix-fanout branch from 58fc52e to 02c8f85 Compare October 27, 2026 06:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Backend Lint is red on this head with F821 undefined name 'lists_checked_by_caller' in src/kiro_crew/publish_governance.py:299 — a file not in this diff. This was introduced by commit 559f264 on main. Opened #24764 to track it. This PR is waiting for that fix to merge before rebasing and re-running.

@CrysisDeu
CrysisDeu force-pushed the unflaky/codebuild-matrix-fanout branch from 02c8f85 to 99ebc37 Compare October 27, 2026 07:18
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Backend Tests shard 2 (Linux + Windows) failed with XPASS(strict) on test_a_warm_session_read_that_earns_a_savepoint_reads_only_the_gap — a test not in this diff that has an xfail marker now invalidated by a recently merged savepoint fix. PR #24774 removes the stale xfail. Waiting for it to merge before rebasing.

@CrysisDeu
CrysisDeu force-pushed the unflaky/codebuild-matrix-fanout branch from 99ebc37 to 08ce036 Compare October 27, 2026 08:29
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Dispositions for head 08ce036778c78b3edb0a28f6a4439bf90c1474fb

Opus BLOCK (head 99ebc37) — test loosened to fit the change — fixed at 08ce036778c78b3edb0a28f6a4439bf90c1474fb.

fixed in 08ce036: test_matrix_job_sets_max_parallel now uses a regex (_CODEBUILD_EXPR_RE) that captures and asserts both numeric bounds: CodeBuild cap <= MAX_PARALLEL_CODEBUILD (8) and hosted fallback <= MAX_PARALLEL (4). test_codebuild_expressions_are_in_bound collects all string caps from the workflows and asserts each matches the pattern with valid bounds. A bare frozenset membership check is replaced by verified bound assertions.

Known CI failure on this head: Backend Tests shard 2 (Linux + Windows) fail with XPASS(strict) on test_a_warm_session_read_that_earns_a_savepoint_reads_only_the_gap. This is a pre-existing main-level issue caused by a savepoint fix that landed after this branch was cut; the xfail marker was not removed at the same time. PR #24774 removes the stale xfail and is currently open. This failure is NOT caused by this PR's diff.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/codebuild-matrix-fanout branch from dc96c4b to 2643082 Compare October 27, 2026 14:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 27, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on 2643082.

The code change is correct, and I'd approve it as-is apart from one doc-sync item.

What I checked:

  • Only CodeBuild runs are uncapped. The 8-wide cap is selected only when the job's own runs-on output carries the codebuild- label. changes emits that label only for this repository, an admitted actor, and push, merge_group, or a same-repository opened/synchronize pull_request. Fork PRs, other PR activities and non-admitted actors resolve to ubuntu-latest/windows-latest and keep max-parallel: 4. The workflow uses pull_request, not pull_request_target.
  • On this head's CI run, all 8 Linux shards and all 8 Windows shards started within seconds of each other on the CodeBuild labels. The hosted-fallback branch was not exercised by this run; I checked it from source only.
  • The diff adds no run: line, no secret and no PR-controlled string. The expression reads only the changes job's own output.
  • The new test_workflow_matrix_max_parallel.py rules hold the shape tightly. Three mutations each turn a named test red: a hosted fallback above 4, a predicate testing a different output than runs-on, and a predicate prefix other than codebuild-.

Requested change:

  • docs/ci/ci-and-reviews.md (around line 136) still says "backend-test counts twice: max-parallel: 4 runs its 8 shards in two rounds". After this change that holds only on the hosted fallback. Please update it in this PR so it says CodeBuild-routed runs (including admitted merge_group runs) run all 8 shards in one round, the budget stays sized for the hosted two-round worst case, and the 4-cap exists to protect the shared hosted pool. A line in the CodeBuild section would also help: on CodeBuild the only remaining fan-out bound is the CodeBuild account concurrency, and ci-runner-watchdog.yml already treats saturating it as a known state. The repo's rule is that the owning spec changes in the same commit as the behaviour it documents.

Non-blocking:

  • test/test_ci_fleet_routing_expression_parity.py imports re inside the function body; a module-level import would match the rest of the file.

CI on this head: 53 success, 9 skipped, 0 failures, and PR Readiness passed. Design and First Principles PASS. Opus and GPT report no findings. Every earlier-head BLOCK was answered with a fix commit.

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD on 2643082.

The workflow change looks correct. The 8-wide cap applies only when the job's own runner output carries the codebuild- label, the hosted fallback keeps max-parallel: 4 (ci.yml), and no permission or secret changes. One thing blocks it: there is an active change request from @buluoray on this head. docs/ci/ci-and-reviews.md still says max-parallel: 4 runs the 8 shards in two rounds. I confirmed that sentence. Updating it in this PR and having @buluoray re-review clears the hold.

Also non-blocking: the description says three test files changed, but test_ci_fleet_routing_expression_parity.py is a fourth.

— crew-pr-reviewer (r129)

@CrysisDeu
CrysisDeu force-pushed the unflaky/codebuild-matrix-fanout branch from 2643082 to 3ee6d4d Compare October 27, 2026 19:27
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

@buluoray — thanks for the review. Two changes at 3ee6d4d6801f8243d5bb7b91f0f8dbaf260e0a6a:

  1. docs/ci/ci-and-reviews.md (~line 136): rewrote the budget parenthetical to say the two-round count applies only on the hosted fallback; on CodeBuild (including admitted merge_group runs) all 8 shards start in one round, the budget stays sized for the hosted worst case, and the cap of 4 exists to protect the shared hosted pool. Added one sentence in the CodeBuild section: on CodeBuild the only remaining fan-out bound is the account concurrency (300 builds), and ci-runner-watchdog.yml already treats saturating it as a known state.

  2. test/test_ci_fleet_routing_expression_parity.py: moved the three inline import re occurrences to module level.

@CrysisDeu
CrysisDeu requested a review from buluoray October 27, 2026 19:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 27, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 3ee6d4d. This supersedes my request-changes review on 2643082. docs/ci/ci-and-reviews.md now says the two-round max-parallel: 4 schedule applies only on the hosted fallback, that CodeBuild (including admitted merge_group runs) starts all 8 shards in one round, and that the remaining fan-out bound there is the CodeBuild account concurrency, which ci-runner-watchdog already treats as a known state. The rest of the interdiff only hoists the repeated local import re in the test to one module import. The workflow logic is unchanged from the version I reviewed.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in .github/workflows/ci.yml and docs/ci/ci-and-reviews.md since this head went green; CI re-running on cc304f42eec82238b5fde50b04cddf672f974d45.

@CrysisDeu
CrysisDeu force-pushed the unflaky/codebuild-matrix-fanout branch from 3ee6d4d to cc304f4 Compare October 27, 2026 20:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 27, 2026
@CrysisDeu
CrysisDeu merged commit e5de771 into main Oct 27, 2026
109 of 110 checks passed
@CrysisDeu
CrysisDeu deleted the unflaky/codebuild-matrix-fanout branch October 27, 2026 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: matrix fan-out cap throttles CodeBuild shards, causing idle-runner timeouts

3 participants