Skip to content

ci: run CI, Fast Gate and Build on merge groups and report PR Readiness there - #15200

Merged
bolichen97 merged 1 commit into
mainfrom
ci/merge-queue
Oct 2, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
ci/merge-queue

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Every commit that lands on main has passed CI on the exact tree it lands on, so a green PR cannot break main.

Today a PR's CI runs against the main of the moment its run started. A run takes ~25 minutes on the fleet and ~13 PRs merge in that window, so the tree a PR was tested on and the tree it lands on are never the same. Two PRs that are each green can be red together, and nothing tests the pair. main's own post-merge CI cannot catch it either: over the last 17 hours 81 of its 100 runs were evicted by the next merge and it went 7 hours without a single verdict.

Why it matters

main is red several times a day with no PR at fault, and the team finds out from a broken local test run rather than from CI. Every red main costs the next PR a false failure and someone an hour of triage.

Not a goal

  • Turning the merge queue on. That is one ruleset click by an admin after this lands; this PR only makes the workflows able to serve it.
  • Running the AI review lanes on merge groups. The PR head already carried them; the queue re-tests, it does not re-review.
  • Speeding up CI or changing which tests run on a pull request.
  • Changing anything about fork PRs.
  • Running the macOS legs inside the queue. They run once per landed commit on main instead, once MERGE_QUEUE_ENABLED is set.

What changed (motivation → approach → change)

Piece What it does
ci.yml, fast-gate.yml, build.yml, main-ratchet-audit.yml, client-py.yml Also run on merge_group (fleet-routed where they already are), with the diff-scoped gates diffing against the group's own base commit and the ratchet audit judging the whole integrated tree
merge-queue-readiness.yml (new) On a merge group, waits for every test workflow a push to main runs (CI, Fast Gate, Build, Internal Content Scan Gate, Python Client Tests, Main Ratchet Audit) on the group's commit and reports their joint verdict as the PR Readiness check, within a 150-minute budget; a failed run has its failed jobs rerun up to twice before it fails the group; one runs listing per tick (60 s, then 180 s once all six are seen) and a rate-limit backoff keep it cheap on the shared installation token; an API error is retried, never read as a missing run or counted as a rerun
Push to main Governed by repository variable MERGE_QUEUE_ENABLED: unset, exactly today -- full matrix, per-ref concurrency, Fast Gate runs; set (with the ruleset flip), only the two macOS legs (ci.yml boot matrix, build.yml desktop build + DMG smoke) in one concurrency group per commit, never evicted, and Fast Gate's push run skips every job
internal-content-scan-gate.yml Comment only: it already ran on merge_group; the note says the queue waits on it
test/ The fleet-routing parity test pins the new expression and the trigger set; the macOS gate test accepts the != 'push' shape
docs/ci, docs/system-specs Name the merge-group path beside the push and pull-request paths they already describe, including why a merge group is routed to the fleet like a push and not fenced like a fork PR

GitHub's merge queue tests a PR on a temporary commit that is main plus everything queued ahead of it, and lands the group only when the required check passes on that commit. Our required check is PR Readiness. pr-readiness.yml cannot report it there: it is keyed on a PR number and folds in the AI lanes. So a new, merge_group-only workflow with one job named PR Readiness polls every test workflow a push to main would run -- CI, Fast Gate, Build, Internal Content Scan Gate, Python Client Tests, Main Ratchet Audit -- for the group's commit and passes only when all six did. Its 150-minute budget covers ci.yml's 100-minute longest chain of job caps plus a rerun, so a slow-but-green group is not dequeued. A failed run has its failed jobs rerun (gh run rerun --failed, hence actions: write) up to twice and fails the check only if the third attempt fails: ~15% of approved heads fail their first try, mostly Windows shards and lost runners, and each ejection rebuilds every group behind it. CI is rerun only after Fast Gate succeeds, since its barrier fails at once on a failed gate. A failed API call is retried and never spends a rerun; a rate-limited one sleeps until the reset. One listing per tick serves all six workflows: ~800 calls an hour at 40 groups, not ~4,800. It lives in its own file on purpose: a same-named job inside ci.yml behind an if: would still create a skipped check run on every PR head, and GitHub counts a skipped required check as passing.

The three test workflows gain the merge_group trigger. Their runner expression admits the event so the group's jobs reach the fleet; a merge_group run's actor is the person who queued the PR, so the existing actor list already admits it. The eleven diff-scoped gates take their base from merge_group.base_sha, the way they take pull_request.base.sha on a PR. The two macOS legs (ci.yml's boot matrix, build.yml's desktop build) are the one exception: a hosted mac runner has waited 176-213 minutes at this repository's merge rate; inside the queue that wait would outlive the check's budget, dequeue green PRs and hold every group behind them. They run on the push to main instead. Repository variable MERGE_QUEUE_ENABLED decides that path. Unset: exactly today, full matrix, per-ref concurrency, Fast Gate runs. Set (in the same operation as enabling the queue): only the macOS legs, in a concurrency group keyed on the commit SHA so the next merge cannot evict the run (81 of the last 100 main runs were evicted) and each landed commit gets its verdict; Fast Gate's push run skips every job. Unsetting both restores today in one step. A merge group's leaf-test scope reads merge_group.base_sha, so a test-only PR pays the narrow run in the queue, not the full matrix.

Nothing changes for push or pull_request runs: the expressions add a branch, they do not alter the existing ones, and the parity tests pin every copy.

Backwards compatibility

Compatible: no trigger, job name, check name or runner label that works today changes; merge_group events do not exist until the ruleset enables the queue, and the push path keeps its full matrix until an admin sets MERGE_QUEUE_ENABLED. Nothing on the push path changes until MERGE_QUEUE_ENABLED is set.

Tests

  • test/test_ci_fleet_routing_expression_parity.py: canonical routing expressions for the merge_group-triggered workflows, a test that the set of workflows declaring merge_group equals the set routed for it plus the hosted ones, a test that the readiness poll's WORKFLOWS list equals that trigger set minus itself, a test that the poll's TOTAL_BUDGET covers ci.yml's longest needs chain of timeout-minutes (and the job cap exceeds the budget), pins of the rollout values (budget 9000 s, rerun cap 2, job cap 155 min), a test that the job holds actions: write and only contents: read besides and reruns with --failed, and three tests that run the real poll step under bash against a fake gh: a failed first attempt is rerun rather than ejected, the group fails only after the third attempt, a refused rerun request is retried without counting, CI is not rerun until Fast Gate has recovered, one listing per tick serves every workflow, and a rate-limited listing backs off to the reset without a verdict, plus pins that the poll makes no per-workflow listing and of its 60 s / 180 s ticks, and the Fast Gate / resolver assertions moved to the new expression. Locks in that a workflow cannot gain the trigger without the fleet route or the poll, or the route without the trigger.
  • test/test_macos_platform_tests_gate.py: the boot matrix's permitted shape is github.event_name != 'push', so macOS stays off both pull requests and merge groups.
  • test/test_fast_gate_barrier.py: both workflows' trigger blocks are pinned as exact dictionaries, and the push path is pinned job by job: exactly four jobs skip on a push only while MERGE_QUEUE_ENABLED is set, every other job except e2e-boot-matrix needs one of them and carries no always()/!cancelled() override, and the boot matrix has three exact legs: ["ubuntu-latest","windows-latest"] off push, ["macos-15"] on push with the variable set, all three on push without it.
  • test/test_ci_kernel_owner_routing.py, test/test_ci_ipv6_routing.py: coverage-gate stays always() on every event but a push, where the matrix it judges does not run.
  • test/test_publish_feed_contract.py: the desktop-matrix compute step emits exactly three platform sets -- macos-15 alone under the push-with-variable guard, all three under the push-or-packaging-changed-PR guard, both Linux arches under else -- pinned as an exact set with the guard that selects each.
  • Scoped backend run on this diff: related test files, 5179 passed. Two failures are inherited on this host (pytest_split not installed; identical on origin/main).

Manual verification

  • All four workflow files parse; the on: blocks are push, pull_request, merge_group (and merge_group alone for the new file).
  • Brand-name and comment-history gates run clean against origin/main; black, isort and flake8 clean on the touched tests.
  • Confirmed on public repositories using the queue (oxc-project/oxc, zed-industries/zed) that a merge_group run's actor is the human who queued the PR, not a bot, so CODEBUILD_ACTOR_IDS needs no new entry.
  • Not verifiable before the ruleset flips: a real merge_group event. Rollout after merge: admin enables Require merge queue on ruleset protected-branches with squash merges, max_entries_to_build 40, max_entries_to_merge 1 (one PR per main commit, so a red mac leg names one author), status-check timeout 180 minutes (the 150-minute poll budget plus hosted runner pickup), and in the same operation sets repository variable MERGE_QUEUE_ENABLED=true. Staged check before anyone else merges: queue one trivial same-repo PR and confirm three things on its group -- the changes job gets a codebuild-* runner within minutes, PR Readiness reports all six workflows found and green, and the group lands on main. If any of the three fails, untick Require merge queue and unset MERGE_QUEUE_ENABLED together; that restores today's behaviour instantly.

Why these values: a queue simulation over last week's real merges found that with no retry the queue backs up badly (p50 ~1.9 h, ~280 flaky ejections a week). One rerun of the failed jobs removes ~90% of flaky ejections and a second removes almost all the rest; a third made the queue slower, because a real failure takes ~85 minutes to eject and invalidates everything queued behind it. A build depth beyond ~40 did not help.

Related Issues

no linked issue: raised from today's investigation of why green PRs break main; the measurement is in the team channel thread.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner September 30, 2026 00:11
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 8f9071bb87652ae746bae56c4bc858ea4efe5e68 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Two independent switches — the MERGE_QUEUE_ENABLED variable and the ruleset's queue toggle — encode one state by convention; drift silently un-tests main.

Watch

  • With the variable set but the queue not (or no longer) enabled — or on any ruleset-bypass push — a commit lands with only the macOS legs: "the merge group proved everything else on that exact tree" ("the merge group built this wheel… on this exact tree") holds only when a group actually ran, and nothing checks that it did. Linux/Windows coverage on main disappears silently until someone notices.
    Clears when: the queue-on push path verifies a merge_group run exists for github.sha before trimming (falling back to the full matrix), or an audit/doc'd operational check pins the variable↔ruleset pairing.
  • The readiness poll's rerun path depends on CodeBuild honoring the stale attempt-suffix label, which the repo's own docs call undocumented; the PR's evidence is one observed run ("It did on CI run 36831273812"). If CodeBuild stops honoring it, every rerun hangs to the 150-minute budget and ejects the group — queue throughput collapses with a misleading "still pending" error.
    Clears when: the behavior is confirmed with CodeBuild (docs or support), or the poll's budget-expiry error names a hung fleet rerun as a suspect so triage is minutes, not hours.

Suggestions

  • Fold the Watch-1 guard into desktop-matrix/changes (one API read of merge_group runs for the SHA, full matrix on miss) rather than trusting the admin's two-knob choreography.

[DESIGN-REVIEWED] 8f9071b

Description read

[DESCRIPTION-READ] 5e6b7d277274b282f3ae01bd5d657c07cd4046e0c9b5d94ff96a9a507e77eab7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 8f9071bb87652ae746bae56c4bc858ea4efe5e68 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The rerun lane rests on CodeBuild honoring a stale -1 runner label — observed on one run, against the repo's own recorded caution.

Not justified as shipped

  • Item 4 — symptom-level with a premise risk: the poll's gh run rerun --failed relies on "(It did on CI run 36831273812…)" — a single observed run — while the standing doc it amends says "CodeBuild's documentation does not say whether it honours that," which is why ci-runner-watchdog.yml's heal path still reruns ALL jobs rather than only failed ones. If the stale attempt-suffix label stops being honored, a rerun's fleet jobs never get a runner, the 150-minute budget burns, and the group is ejected anyway — the exact outcome the two reruns exist to prevent.
    Subtraction: replace --failed in merge-queue-readiness.yml (and the all("--failed" in line) pin in test_ci_fleet_routing_expression_parity.py) with the full gh run rerun spelling the watchdog already trusts.
    Clears when: the stale-label behavior is confirmed beyond run 36831273812 (vendor documentation or repeated observed reruns on the routed Linux and Windows labels), or the poll adopts the full-rerun spelling.

What this change ships

Inventory (10 items) — 9 justified

Intent: make every commit land on main only after CI passed on the exact tree it lands on, by making the test workflows serve GitHub's merge queue — an ADDITION.

  1. CI, Fast Gate and Build now also run on merge-queue groups, fleet-routed, diff gates scoped to the group's own base — justified
  2. Main Ratchet Audit and Python Client Tests also run on merge groups — justified
  3. New PR Readiness check on merge groups: a dedicated workflow waits on all six runs within 150 minutes (not pr-readiness.yml, which is PR-number-keyed and never fires on merge_group — verified) — justified
  4. A failed run on a group has its failed jobs rerun up to twice before ejecting — symptom-level with a premise risk: --failed label behavior observed once (entry above)
  5. New repo variable MERGE_QUEUE_ENABLED, default unset (today unchanged): set, a push to main runs only the two macOS legs, per-commit, never evicted — justified
  6. Fast Gate's "every gate is unconditional" pin rewritten to permit exactly the queue-push skip clause — justified
  7. macOS legs never run inside the queue; they stay on the push to main — justified
  8. The poll reads one runs listing per tick with rate-limit backoff — justified
  9. CI docs, e2e-gate and oss-fork-boundaries describe the merge-group path — justified
  10. Every merge group runs the full client-py suite regardless of touched paths — justified

[FIRST-PRINCIPLES-REVIEWED] 8f9071b

Description read

[DESCRIPTION-READ] cd3ee487c8df2a2cca86c7f938cac53628db00df47a6cd4d9ebeaef54d876cbb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 8f9071bb87652ae746bae56c4bc858ea4efe5e68 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8f9071b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8f9071bb87652ae746bae56c4bc858ea4efe5e68: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 8f9071bb87652ae746bae56c4bc858ea4efe5e68 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8f9071b

Verdict parsed from the review's SHA-scoped output markers for commit 8f9071bb87652ae746bae56c4bc858ea4efe5e68.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8f9071bb87652ae746bae56c4bc858ea4efe5e68: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 30, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed -- span=96672f5cb92a -- readiness poll budget below CI's critical path

Fixed in 1af2399: TOTAL_BUDGET is 7200 s (120 min) and the job cap 125 min.
ci.yml's critical path of job caps is changes 15 -> await-fast-gate 15 -> backend-test 60 -> coverage-combine 15 -> coverage-gate 10 = 115 min; the budget now sits above it.
The file and docs/ci/ci-and-reviews.md state the arithmetic and require the ruleset's merge-queue status-check timeout to be 150 min or more.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed -- span=0bea0379bfe0 -- API transport failure read as an absent run

Fixed in 1af2399: latest_run() returns non-zero on a gh api or jq failure instead of printing null; the caller classifies that tick as errored, logs a ::warning:: with the error head, and retries.
A run id is remembered once listed (seen_run), so an empty listing for a workflow already seen is also errored, never missing.
Only a completed non-success run, or a never-seen run after the 300 s appear window, fails the check; errored can only reach the TOTAL_BUDGET backstop.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed -- span=a6e42d56b053 -- poll cannot tell an API error from an absent run

Fixed in 1af2399: the poll distinguishes the two. A failed or empty-after-seen read is errored and retried until the 120-min budget; a never-seen run after 300 s is missing and fails; a completed non-success conclusion fails at once.
Clears-when condition met: the loop remembers each workflow's run id once seen and fails only on a confirmed non-success conclusion or a confirmed absence.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed -- span=f24e18638ea4 -- lengthen the sleep once every run is observed

Fixed in 1af2399: the tick stays 30 s while any run is still unseen (the appear race) and becomes 120 s once all four workflows have a listed run, cutting the per-group API budget about four-fold.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 30, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) September 30, 2026 01:55
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • rebutted -- span=ef0d4a5cb33d -- merge_group fleet routing carries no same-repository conjunct

Not a defect. The fork fence guards code NOBODY has approved: a pull_request event can carry any commit a contributor pushes. A merge_group cannot: the queue admits a PR only once it satisfies every protected-branches rule (1 approving review, code-owner review, last-push approval, PR Readiness), the queued head cannot be amended (a new push dequeues it), and the group's tree is the exact tree that lands on main, where the push run executes it on the fleet minutes later.
So the trust boundary a merge group sits on is "approved for main", which is the boundary the fleet already accepts for every push to main. Routing the group hosted would not change what code the fleet runs; it would only run the same approved tree twice, once slow.
The only actors who can place a fork's code into a group are the ruleset's bypass roles (Maintainer, Admin) and approving reviewers, who can equally push that code to main directly; the fleet already trusts exactly them.
docs/ci/ci-and-reviews.md now states this beside the "a fork PR stays hosted" invariant (ae06045). The webhook's actor filter is unchanged and still the independent fence.
Routing merge groups hosted was measured against the queue's purpose: hosted CI is ~90 min with 12-21 min per-job runner waits, which at 172 merges/day makes the queue the bottleneck it exists to remove.
Maintainer override, if this reasoning is accepted: /ai-review override fable ae06045522eb4ea0ef3c22b7725cb296bf849fb7: a merge group holds only a maintainer-approved head that lands on main and runs on the fleet regardless; the fork fence targets unapproved code, which a merge group cannot carry

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed -- span=bf553d00556b -- no assertion ties the poll list to the merge_group trigger set

Fixed in ae06045: test_merge_queue_readiness_polls_every_merge_group_workflow reads the readiness job's WORKFLOWS env and asserts it equals the set of workflows declaring merge_group, minus merge-queue-readiness.yml itself. It also pins the job name to the ruleset's required check.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • accepted-and-deferred -- span=1df039f5fec9 -- the poll script has no in-repo test

Deferred to #15243 (label deferred-finding, assignee bolichen97, Due 2026-10-14): extract the poll into .github/scripts/merge-queue-readiness.sh and test it against a stubbed gh over the classification cases.
The logic was exercised during this PR with an offline stub harness (all green -> 0; one failure -> 1 at once; listed-then-missing -> errored and recovered; API failing from tick 0 -> 1 only at the total budget), and the enabling ruleset change is a supervised first-group rollout; the harness is not in the repository, which is what the issue closes.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • accepted-and-deferred -- span=cd6f89e8584d -- extracting the poll lets the bash-3.2 test cover it directly

Same extraction as #15243 (deferred-finding, bolichen97, Due 2026-10-14); test_workflow_array_expansion_bash32.py already covers the inline run: block by sweeping every workflow's scripts, and will cover the extracted file directly once it exists.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed -- span=16dc68ebc817 -- merge-queue bullet filed under "never gate a PR"; ratchet-audit prose contradicts its new trigger

Fixed in 5c58cdb: the merge-queue bullet moved into the gate list directly under the "real merge gate" bullet, which now names the queue; the ratchet-audit bullet says the lane runs on every push to main AND every merge group, where the queue's required check waits on it, and its "full serialization or a merge queue is a runner-budget call" sentence is replaced; main-ratchet-audit.yml's header parenthetical says the same.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • accepted-and-deferred -- span=637f10acc42c -- a red macOS leg on main has no pager once the queue carries everything else

Deferred to #15257 (label deferred-finding, assignee bolichen97, Due 2026-10-14): a post-merge alert for the two macOS legs, in the shape of main-ratchet-audit's tracking issue or a scheduled read of the latest macos-15 jobs on main.
Out of this PR's goal (the workflows that let the queue be enabled); the alert is a new surface and lands on its own.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • rebutted -- span=93fb6e0bcb97 -- merge groups can run fork-controlled code on the private fleet

Not a defect: the code a merge group runs is code a maintainer has already approved for main. The queue admits a pull request only after it satisfies every protected-branches rule (one approving review, code-owner review, approval of the LAST push, PR Readiness); a queued head cannot change (a new push dequeues it); and the group's tree is the tree that lands on main minutes later, where the push run executes it on the same fleet with the same host identity.
The fork fence exists for code nobody has approved, which a pull_request event can carry and a merge_group cannot. "Fork-controlled" stops being true at approval: after review the head is maintainer-accepted code, not contributor-controlled code.
The actors who can put a fork's code into a group are approving reviewers and the ruleset's bypass roles, who can equally push that code to main; the fleet already trusts exactly them. Routing groups hosted would run the same approved tree once slowly and once on the fleet, changing what the fleet executes by nothing.
Recorded in docs/ci/ci-and-reviews.md beside the fork-fence invariant. Opus 5 raised the same class on 5a801e7 and withdrew it on ae06045 after that rationale.
Security-class findings do not converge by disposition; a maintainer's /ai-review override gpt <head> is the ruling path if this reasoning is accepted.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=dec1f1d275e7 Gate the push trim on a successful merge_group run

deferred as a follow-up: checking for a successful merge_group run on the pushed SHA would make the trim self-verifying, and it changes every push-to-main path, beyond this PR's queue wiring.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=3ea2e42fc531 Spawned bash writes outside test isolation

fixed in bd4ad50: the harness runs the poll with TMPDIR and HOME inside tmp_path, so the step's mktemp error file and anything gh would write stay in the test's own directory.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=78188b23fd92 The poll keys on an unverified head_branch spelling

fixed in bd4ad50 by subtraction, as suggested: the .head_branch == $BRANCH match is deleted. A merge group's commit exists for that group alone, so head_sha plus event=merge_group identifies its runs. The fake gh in the tests now spells head_branch differently from BRANCH and the poll still clears the group.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=0f3c2f413125 The pinned paths-filter's merge_group case is unconfirmed

confirmed: src/main.ts at the pinned dorny/paths-filter@ceb8a2b8 has case 'merge_group': at line 126, taking base = merge_group.base_sha and ref = merge_group.head_sha.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=343eed390958 Drop the head_branch match

fixed in bd4ad50: the match is gone, as suggested; see the first-principles disposition.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=6d9864f62990 Two premises unexercised before the flip

one premise is removed (the poll no longer depends on head_branch). The other, the fleet webhook admitting merge_group jobs, is what the rollout's staged check confirms first: one trivial queued PR whose changes job must get a codebuild-* runner, with the one-step rollback written beside it.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=51eda6d77e86 The variable can be set while the queue is off

acknowledged, deferred as before (c157f7c8d4e8): both are admin-only controls set together in one operation, and the rollout states the joint rollback. A self-verifying push trim is a follow-up that touches every push-to-main path.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 1, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed -- span=efcb6379d331 -- the poll's child process runs in the repo root

Fixed in 8f9071b: _run_readiness passes cwd=tmp_path, beside the TMPDIR and HOME it already set there, so a relative write a later poll edit adds lands in the test's own directory.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • rebutted -- span=0bea0379bfe0 -- gh run rerun --failed reuses a stale runner label

Measured on this PR, not assumed: CI run 36831273812 was rerun with --failed twice. On attempts 2 and 3, every routed job kept the codebuild-kirocrew-gha-{linux,windows}-36831273812-1 label, and each got a fresh runner within two minutes (attempt 2 started 08:07-08:11, attempt 3 08:32-08:34). CodeBuild honours the stale suffix, so --failed stays; it reruns one failed shard instead of the whole matrix. The watchdog paragraph in ci-and-reviews.md now records the observation.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • rebutted -- span=35261a5c3ee1 -- the queue-on push path drops the bound on concurrent hosted macOS jobs

The same finding as spans 35261a5c3ee1, ff43b6f1b705 and 278cb944d161 on earlier heads, with the same answer: it is the design the maintainer chose (one macOS verdict per landed commit, never evicted), and demand was measured. Over 24 h there were 104 macos-15 jobs with a peak of 28 concurrent. A commit holds at most two mac jobs at once, because the smoke waits on the build. At the 35/h peak that is ~18 concurrent; at the 7/h average, ~4. The PR also removes the mac legs from every pull request. If the pool proves smaller, unsetting MERGE_QUEUE_ENABLED restores the per-ref group in one step.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. Reviewed at 8f9071bb87652ae746bae56c4bc858ea4efe5e68.

What it does. Wires CI, Fast Gate, Build, Main Ratchet Audit and Python Client Tests to merge_group (diff-scoped gates take merge_group.base_sha), and adds merge-queue-readiness.yml, a merge_group-only workflow whose single job PR Readiness polls those six workflows on the group's commit, reruns failed jobs up to twice, and reports the joint verdict under the ruleset's required check name. The push-to-main trim to macOS-only legs is gated on repo variable MERGE_QUEUE_ENABLED; with it unset, push and pull_request behave as today. Nothing activates until an admin enables the queue.

Privilege / trust boundary (checked, no blocking issue)

  • No pull_request_target is added or touched. merge-queue-readiness.yml checks out no code. It holds actions: write + contents: read only (job-level, merge-queue-readiness.yml:70-77), and github.sha/ref_name reach the script through env:, not ${{ }} interpolation inside run:.
  • Fleet routing for merge_group (ci.yml:70, fast-gate.yml:80, build.yml:37,272, main-ratchet-audit.yml:94,218,296,344) still needs github.repository == 'kirodotdev/KiroCrew' plus an admitted actor_id. The docs argue that a merge group holds only maintainer-approved heads (docs/ci/ci-and-reviews.md, the fleet section), and the live protected-branches ruleset backs that up: it has dismiss_stale_reviews_on_push: true, require_last_push_approval: true and 1 required approval. So a merge group runs code a maintainer approved at that exact head, which is the same tree the existing push run already executes on the fleet minutes later. That keeps fork code out of an unapproved privileged context.
  • The workflow_run consumers of CI/Fast Gate do not pick up merge-group runs. Every fork-*.yml filters workflow_run.event == 'pull_request', and pr-readiness.yml:204-207 admits only pull_request/dynamic. main-ratchet-audit.yml:348 keeps the tracking-issue writer on push only.

Required-check names

  • On pull_request: PR Readiness still comes from the unchanged pr-readiness.yml. The commit status at this head is success.
  • On merge_group: the job is named PR Readiness (merge-queue-readiness.yml:67), and the workflow fires on nothing else, so PR heads get no skipped same-named check run (rationale at :22-28). The ruleset requires exactly {"context":"PR Readiness"}.
  • ci.yml await-fast-gate matches Fast Gate runs by head_branch == github.ref_name on a merge group. I confirmed on zed-industries/zed that merge-group runs carry head_branch: gh-readonly-queue/main/pr-N-<sha>, the same spelling as ref_name, with head_repository = the base repo.

Applies to current main. git merge-tree --write-tree origin/main HEAD is clean (82 commits behind). On a probe commit of that merged tree, the related suites (below plus test_ci_surface_tests.py) gave 335 passed. On the merged tree, the push-to-main workflow set is still exactly the six polled ones plus non-test pages/publish-installer/pr-merge-conflict-label.

Tests run (worktree source confirmed loaded): test_ci_fleet_routing_expression_parity, test_fast_gate_barrier, test_macos_platform_tests_gate, test_publish_feed_contract, test_ci_kernel_owner_routing, test_ci_ipv6_routing, test_builtin_skill_scope, test_vendor_manifest, test_workflow_array_expansion_bash32: 264 passed.

Mutations (each restored, tree clean afterwards):

  1. Dropped fast-gate.yml from the poll's WORKFLOWS. Reddened test_merge_queue_readiness_polls_every_merge_group_workflow.
  2. Disabled the rerun cap (if false). Reddened test_the_group_fails_only_after_the_third_attempt.
  3. Removed merge_group from one routing copy in build.yml. Reddened test_every_copy_of_the_routing_expression_matches_the_canonical_one.
  4. Reverted the boot matrix to == 'pull_request'. Reddened TestTheGatesLiveInTheGateWorkflow::test_a_push_to_main_reaches_exactly_the_macos_boot_leg and test_ci_declares_no_macos_runner_reachable_from_a_pull_request.

Fail-open / fail-closed

  • Fail-closed: a run still missing after 300 s, the attempt cap, and the 9000 s budget all exit 1. A failed API read or rerun request is retried and never cleared. A rerun that never gets a runner burns the budget and ejects the group, which is no worse than having no retry.
  • The one fail-open path is operational rather than code: setting MERGE_QUEUE_ENABLED=true without the queue, or a ruleset-bypass push, lands a commit with only the macOS legs.

Bot concerns, disposition

  • Design / variable↔ruleset drift (CONCERNS): checked. It is real but does not gate: both knobs are admin-only, the PR body and docs document the joint set/unset, and with the variable unset nothing changes. Follow-up below.
  • First-principles + design / --failed relies on CodeBuild honouring the stale -1 label (CONCERNS): checked. It fails closed: in the worst case a rerun hangs, the budget expires and the group is ejected, which is the pre-retry outcome. The author's measured run 36831273812 is cited in docs. Follow-up below.
  • GPT and Opus: no findings at this head. Earlier-head dispositions (head_branch match removed, paths-filter merge_group case, test cwd isolation) are present in source at this head.

Commit message. Single commit. The title accurately describes the change.

Non-blocking follow-ups

  1. Make the queue-on push trim self-verifying: before skipping the Linux/Windows matrix, check that a successful merge_group run exists for github.sha, and fall back to the full matrix if not. That closes the variable↔ruleset drift and the bypass-push gap.
  2. During the staged rollout canary, force one --failed rerun on a routed Linux and a routed Windows job to confirm the stale-label behaviour beyond a single run. Otherwise switch the poll to a full gh run rerun like the watchdog.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants