Repository navigation
ci: run CI, Fast Gate and Build on merge groups and report PR Readiness there - #15200
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Two independent switches — the Watch
Suggestions
[DESIGN-REVIEWED] 8f9071b Description read[DESCRIPTION-READ] 5e6b7d277274b282f3ae01bd5d657c07cd4046e0c9b5d94ff96a9a507e77eab7 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The rerun lane rests on CodeBuild honoring a stale Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: make every commit land on main only after CI passed on the exact tree it lands on, by making the test workflows serve GitHub's merge queue — an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 8f9071b Description read[DESCRIPTION-READ] cd3ee487c8df2a2cca86c7f938cac53628db00df47a6cd4d9ebeaef54d876cbb That sha256 names the PR description this verdict read. Recompute it with |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
f44d5a8 to
1af2399
Compare
|
|
|
|
1af2399 to
7b3d67a
Compare
7b3d67a to
5a801e7
Compare
5a801e7 to
ae06045
Compare
|
|
|
|
ae06045 to
5c58cdb
Compare
|
|
|
|
d420556 to
ae723a0
Compare
ae723a0 to
bd4ad50
Compare
|
|
|
|
|
|
|
|
|
buluoray
left a comment
There was a problem hiding this comment.
Approve. Reviewed at 8f9071bb87652ae746bae56c4bc858ea4efe5e68.
What it does. Wires CI, Fast Gate, Build, Main Ratchet Audit and Python Client Tests to merge_group (diff-scoped gates take merge_group.base_sha), and adds merge-queue-readiness.yml, a merge_group-only workflow whose single job PR Readiness polls those six workflows on the group's commit, reruns failed jobs up to twice, and reports the joint verdict under the ruleset's required check name. The push-to-main trim to macOS-only legs is gated on repo variable MERGE_QUEUE_ENABLED; with it unset, push and pull_request behave as today. Nothing activates until an admin enables the queue.
Privilege / trust boundary (checked, no blocking issue)
- No
pull_request_targetis added or touched.merge-queue-readiness.ymlchecks out no code. It holdsactions: write+contents: readonly (job-level,merge-queue-readiness.yml:70-77), andgithub.sha/ref_namereach the script throughenv:, not${{ }}interpolation insiderun:. - Fleet routing for
merge_group(ci.yml:70,fast-gate.yml:80,build.yml:37,272,main-ratchet-audit.yml:94,218,296,344) still needsgithub.repository == 'kirodotdev/KiroCrew'plus an admittedactor_id. The docs argue that a merge group holds only maintainer-approved heads (docs/ci/ci-and-reviews.md, the fleet section), and the liveprotected-branchesruleset backs that up: it hasdismiss_stale_reviews_on_push: true,require_last_push_approval: trueand 1 required approval. So a merge group runs code a maintainer approved at that exact head, which is the same tree the existing push run already executes on the fleet minutes later. That keeps fork code out of an unapproved privileged context. - The
workflow_runconsumers of CI/Fast Gate do not pick up merge-group runs. Everyfork-*.ymlfiltersworkflow_run.event == 'pull_request', andpr-readiness.yml:204-207admits onlypull_request/dynamic.main-ratchet-audit.yml:348keeps the tracking-issue writer onpushonly.
Required-check names
- On
pull_request:PR Readinessstill comes from the unchangedpr-readiness.yml. The commit status at this head issuccess. - On
merge_group: the job is namedPR Readiness(merge-queue-readiness.yml:67), and the workflow fires on nothing else, so PR heads get no skipped same-named check run (rationale at:22-28). The ruleset requires exactly{"context":"PR Readiness"}. ci.ymlawait-fast-gatematches Fast Gate runs byhead_branch == github.ref_nameon a merge group. I confirmed on zed-industries/zed that merge-group runs carryhead_branch: gh-readonly-queue/main/pr-N-<sha>, the same spelling asref_name, withhead_repository= the base repo.
Applies to current main. git merge-tree --write-tree origin/main HEAD is clean (82 commits behind). On a probe commit of that merged tree, the related suites (below plus test_ci_surface_tests.py) gave 335 passed. On the merged tree, the push-to-main workflow set is still exactly the six polled ones plus non-test pages/publish-installer/pr-merge-conflict-label.
Tests run (worktree source confirmed loaded): test_ci_fleet_routing_expression_parity, test_fast_gate_barrier, test_macos_platform_tests_gate, test_publish_feed_contract, test_ci_kernel_owner_routing, test_ci_ipv6_routing, test_builtin_skill_scope, test_vendor_manifest, test_workflow_array_expansion_bash32: 264 passed.
Mutations (each restored, tree clean afterwards):
- Dropped
fast-gate.ymlfrom the poll'sWORKFLOWS. Reddenedtest_merge_queue_readiness_polls_every_merge_group_workflow. - Disabled the rerun cap (
if false). Reddenedtest_the_group_fails_only_after_the_third_attempt. - Removed
merge_groupfrom one routing copy inbuild.yml. Reddenedtest_every_copy_of_the_routing_expression_matches_the_canonical_one. - Reverted the boot matrix to
== 'pull_request'. ReddenedTestTheGatesLiveInTheGateWorkflow::test_a_push_to_main_reaches_exactly_the_macos_boot_legandtest_ci_declares_no_macos_runner_reachable_from_a_pull_request.
Fail-open / fail-closed
- Fail-closed: a run still missing after 300 s, the attempt cap, and the 9000 s budget all
exit 1. A failed API read or rerun request is retried and never cleared. A rerun that never gets a runner burns the budget and ejects the group, which is no worse than having no retry. - The one fail-open path is operational rather than code: setting
MERGE_QUEUE_ENABLED=truewithout the queue, or a ruleset-bypass push, lands a commit with only the macOS legs.
Bot concerns, disposition
- Design / variable↔ruleset drift (CONCERNS): checked. It is real but does not gate: both knobs are admin-only, the PR body and docs document the joint set/unset, and with the variable unset nothing changes. Follow-up below.
- First-principles + design /
--failedrelies on CodeBuild honouring the stale-1label (CONCERNS): checked. It fails closed: in the worst case a rerun hangs, the budget expires and the group is ejected, which is the pre-retry outcome. The author's measured run 36831273812 is cited in docs. Follow-up below. - GPT and Opus: no findings at this head. Earlier-head dispositions (head_branch match removed, paths-filter
merge_groupcase, test cwd isolation) are present in source at this head.
Commit message. Single commit. The title accurately describes the change.
Non-blocking follow-ups
- Make the queue-on push trim self-verifying: before skipping the Linux/Windows matrix, check that a successful
merge_grouprun exists forgithub.sha, and fall back to the full matrix if not. That closes the variable↔ruleset drift and the bypass-push gap. - During the staged rollout canary, force one
--failedrerun on a routed Linux and a routed Windows job to confirm the stale-label behaviour beyond a single run. Otherwise switch the poll to a fullgh run rerunlike the watchdog.
Problem / Motivation
Goal: Every commit that lands on
mainhas passed CI on the exact tree it lands on, so a green PR cannot breakmain.Today a PR's CI runs against the
mainof the moment its run started. A run takes ~25 minutes on the fleet and ~13 PRs merge in that window, so the tree a PR was tested on and the tree it lands on are never the same. Two PRs that are each green can be red together, and nothing tests the pair.main's own post-merge CI cannot catch it either: over the last 17 hours 81 of its 100 runs were evicted by the next merge and it went 7 hours without a single verdict.Why it matters
mainis red several times a day with no PR at fault, and the team finds out from a broken local test run rather than from CI. Every redmaincosts the next PR a false failure and someone an hour of triage.Not a goal
maininstead, onceMERGE_QUEUE_ENABLEDis set.What changed (motivation → approach → change)
ci.yml,fast-gate.yml,build.yml,main-ratchet-audit.yml,client-py.ymlmerge_group(fleet-routed where they already are), with the diff-scoped gates diffing against the group's own base commit and the ratchet audit judging the whole integrated treemerge-queue-readiness.yml(new)PR Readinesscheck, within a 150-minute budget; a failed run has its failed jobs rerun up to twice before it fails the group; one runs listing per tick (60 s, then 180 s once all six are seen) and a rate-limit backoff keep it cheap on the shared installation token; an API error is retried, never read as a missing run or counted as a rerunmainMERGE_QUEUE_ENABLED: unset, exactly today -- full matrix, per-ref concurrency, Fast Gate runs; set (with the ruleset flip), only the two macOS legs (ci.ymlboot matrix,build.ymldesktop build + DMG smoke) in one concurrency group per commit, never evicted, and Fast Gate's push run skips every jobinternal-content-scan-gate.ymlmerge_group; the note says the queue waits on ittest/!= 'push'shapedocs/ci,docs/system-specsGitHub's merge queue tests a PR on a temporary commit that is
mainplus everything queued ahead of it, and lands the group only when the required check passes on that commit. Our required check isPR Readiness.pr-readiness.ymlcannot report it there: it is keyed on a PR number and folds in the AI lanes. So a new,merge_group-only workflow with one job namedPR Readinesspolls every test workflow a push to main would run -- CI, Fast Gate, Build, Internal Content Scan Gate, Python Client Tests, Main Ratchet Audit -- for the group's commit and passes only when all six did. Its 150-minute budget coversci.yml's 100-minute longest chain of job caps plus a rerun, so a slow-but-green group is not dequeued. A failed run has its failed jobs rerun (gh run rerun --failed, henceactions: write) up to twice and fails the check only if the third attempt fails: ~15% of approved heads fail their first try, mostly Windows shards and lost runners, and each ejection rebuilds every group behind it. CI is rerun only after Fast Gate succeeds, since its barrier fails at once on a failed gate. A failed API call is retried and never spends a rerun; a rate-limited one sleeps until the reset. One listing per tick serves all six workflows: ~800 calls an hour at 40 groups, not ~4,800. It lives in its own file on purpose: a same-named job insideci.ymlbehind anif:would still create askippedcheck run on every PR head, and GitHub counts a skipped required check as passing.The three test workflows gain the
merge_grouptrigger. Their runner expression admits the event so the group's jobs reach the fleet; amerge_grouprun's actor is the person who queued the PR, so the existing actor list already admits it. The eleven diff-scoped gates take their base frommerge_group.base_sha, the way they takepull_request.base.shaon a PR. The two macOS legs (ci.yml's boot matrix,build.yml's desktop build) are the one exception: a hosted mac runner has waited 176-213 minutes at this repository's merge rate; inside the queue that wait would outlive the check's budget, dequeue green PRs and hold every group behind them. They run on the push tomaininstead. Repository variableMERGE_QUEUE_ENABLEDdecides that path. Unset: exactly today, full matrix, per-ref concurrency, Fast Gate runs. Set (in the same operation as enabling the queue): only the macOS legs, in a concurrency group keyed on the commit SHA so the next merge cannot evict the run (81 of the last 100 main runs were evicted) and each landed commit gets its verdict; Fast Gate's push run skips every job. Unsetting both restores today in one step. A merge group's leaf-test scope readsmerge_group.base_sha, so a test-only PR pays the narrow run in the queue, not the full matrix.Nothing changes for
pushorpull_requestruns: the expressions add a branch, they do not alter the existing ones, and the parity tests pin every copy.Backwards compatibility
Compatible: no trigger, job name, check name or runner label that works today changes;
merge_groupevents do not exist until the ruleset enables the queue, and the push path keeps its full matrix until an admin setsMERGE_QUEUE_ENABLED. Nothing on the push path changes untilMERGE_QUEUE_ENABLEDis set.Tests
test/test_ci_fleet_routing_expression_parity.py: canonical routing expressions for themerge_group-triggered workflows, a test that the set of workflows declaringmerge_groupequals the set routed for it plus the hosted ones, a test that the readiness poll'sWORKFLOWSlist equals that trigger set minus itself, a test that the poll'sTOTAL_BUDGETcoversci.yml's longestneedschain oftimeout-minutes(and the job cap exceeds the budget), pins of the rollout values (budget 9000 s, rerun cap 2, job cap 155 min), a test that the job holdsactions: writeand onlycontents: readbesides and reruns with--failed, and three tests that run the real poll step under bash against a fakegh: a failed first attempt is rerun rather than ejected, the group fails only after the third attempt, a refused rerun request is retried without counting, CI is not rerun until Fast Gate has recovered, one listing per tick serves every workflow, and a rate-limited listing backs off to the reset without a verdict, plus pins that the poll makes no per-workflow listing and of its 60 s / 180 s ticks, and the Fast Gate / resolver assertions moved to the new expression. Locks in that a workflow cannot gain the trigger without the fleet route or the poll, or the route without the trigger.test/test_macos_platform_tests_gate.py: the boot matrix's permitted shape isgithub.event_name != 'push', so macOS stays off both pull requests and merge groups.test/test_fast_gate_barrier.py: both workflows' trigger blocks are pinned as exact dictionaries, and the push path is pinned job by job: exactly four jobs skip on a push only whileMERGE_QUEUE_ENABLEDis set, every other job excepte2e-boot-matrixneeds one of them and carries noalways()/!cancelled()override, and the boot matrix has three exact legs:["ubuntu-latest","windows-latest"]off push,["macos-15"]on push with the variable set, all three on push without it.test/test_ci_kernel_owner_routing.py,test/test_ci_ipv6_routing.py:coverage-gatestaysalways()on every event but a push, where the matrix it judges does not run.test/test_publish_feed_contract.py: the desktop-matrix compute step emits exactly three platform sets --macos-15alone under the push-with-variable guard, all three under the push-or-packaging-changed-PR guard, both Linux arches underelse-- pinned as an exact set with the guard that selects each.pytest_splitnot installed; identical onorigin/main).Manual verification
on:blocks arepush,pull_request,merge_group(andmerge_groupalone for the new file).origin/main; black, isort and flake8 clean on the touched tests.merge_grouprun'sactoris the human who queued the PR, not a bot, soCODEBUILD_ACTOR_IDSneeds no new entry.merge_groupevent. Rollout after merge: admin enables Require merge queue on rulesetprotected-brancheswith squash merges,max_entries_to_build40,max_entries_to_merge1 (one PR per main commit, so a red mac leg names one author), status-check timeout 180 minutes (the 150-minute poll budget plus hosted runner pickup), and in the same operation sets repository variableMERGE_QUEUE_ENABLED=true. Staged check before anyone else merges: queue one trivial same-repo PR and confirm three things on its group -- thechangesjob gets acodebuild-*runner within minutes,PR Readinessreports all six workflows found and green, and the group lands on main. If any of the three fails, untick Require merge queue and unsetMERGE_QUEUE_ENABLEDtogether; that restores today's behaviour instantly.Why these values: a queue simulation over last week's real merges found that with no retry the queue backs up badly (p50 ~1.9 h, ~280 flaky ejections a week). One rerun of the failed jobs removes ~90% of flaky ejections and a second removes almost all the rest; a third made the queue slower, because a real failure takes ~85 minutes to eject and invalidates everything queued behind it. A build depth beyond ~40 did not help.
Related Issues
no linked issue: raised from today's investigation of why green PRs break
main; the measurement is in the team channel thread.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)