From 4868ba834351f4ca55447784e856fde24e4b13ae Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sun, 1 Nov 2026 06:51:33 -0800 Subject: [PATCH] docs(runbooks): ci-heavy-runner reflects Latchkey on push:main, not merge_group Follows the routing move in #3237. Latchkey heavy runners now fire only on a push to main that touched a heavy path (the changes filter), not on merge_group; the merge-queue required check runs on free GitHub-hosted runners. Updates the when-it-routes, held-job/backstop, variable table, and trust-boundary sections accordingly. The trust boundary is now stricter: only merged code (past review and the queue) reaches the privileged runners. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_015AKgpNr8tFMMj8aFvFPUdA --- docs/runbooks/ci-heavy-runner.md | 52 ++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 23 deletions(-) diff --git a/docs/runbooks/ci-heavy-runner.md b/docs/runbooks/ci-heavy-runner.md index 68d2f8e974..9aefddbb01 100644 --- a/docs/runbooks/ci-heavy-runner.md +++ b/docs/runbooks/ci-heavy-runner.md @@ -5,19 +5,24 @@ frontend tests and the two Tier 1 service shards) can run on Latchkey runners, which don't count against the org's 20-concurrent-job cap on GitHub-hosted runners. That cap is what made the merge queue wait. -They do so only on **merge-queue runs**, and only while the repository -variable `LATCHKEY_HEAVY` is `on`. Every other run, PRs included, stays on -GitHub-hosted runners, which are free on this public repo. Backend runs on -`latchkey-medium` and frontend and service on `latchkey-small` (sizes below). +They do so only on a **push to `main`** that touched a heavy path (the +`changes` path filter in the workflow), and only while the repository variable +`LATCHKEY_HEAVY` is `on`. Every other run — PRs, and the `merge_group` required +check — stays on GitHub-hosted runners, which are free on this public repo, so a +docs-only merge spends no Latchkey minutes. Backend runs on `latchkey-medium` +and frontend and service on `latchkey-small` (sizes below). (Before 2026-11-01 +this routed on `merge_group`; it moved to push:main + the path filter to stay +under the free-minute cap, because the queue ran Latchkey on every entry.) ## How a run is routed A held Latchkey job (free minutes spent, the spend limit reached, or an -outage) does not fail. It sits "queued" on GitHub with no error, and a -required check that waits on it leaves the merge queue stalled with nothing -red anywhere. So nothing is sent to Latchkey blind. The logic is -`scripts/latchkey-route.mjs`, with tests in -`scripts/__tests__/latchkey-route.test.mjs`. +outage) does not fail. It sits "queued" on GitHub with no error. Latchkey now +runs only on the post-merge push to `main`, so a held job no longer stalls a +required check (the merge queue runs its heavy jobs on GitHub-hosted runners); +the backstop below still cancels the held post-merge run so it does not hang. +Nothing is sent to Latchkey blind. The logic is `scripts/latchkey-route.mjs`, +with tests in `scripts/__tests__/latchkey-route.test.mjs`. 1. **Latchkey probe** asks for a `latchkey-small` runner and only echoes. It is not in the gate's needs, so a held probe never holds the required check. @@ -32,9 +37,10 @@ red anywhere. So nothing is sent to Latchkey blind. The logic is GitHub-hosted. 3. **Latchkey backstop** (GitHub-hosted) watches the heavy jobs once they are routed. If one is still queued after 300 s, it records a hold and cancels - the run. The queue entry then fails loudly instead of waiting, and later - runs stay on GitHub until the hold expires. The press re-queues the PR as - for any dequeue. + the (post-merge) run instead of letting it hang, and later push-to-main runs + stay on GitHub until the hold expires. Since this is post-merge there is no + queue entry to re-run; the change is already on `main`, and the `merge_group` + check that gated it ran on GitHub-hosted runners. The budget is an estimate: Latchkey runs since `LATCHKEY_BUDGET_START` times `LATCHKEY_EQ_PER_RUN` (default 48 small-equivalent minutes, measured on the 10-25 rehearsal), stopped at 80% @@ -46,7 +52,7 @@ figure below, and raise `LATCHKEY_EQ_PER_RUN` if the measured figure is higher. | variable | meaning | default | |---|---|---| -| `LATCHKEY_HEAVY` | `on` routes merge-queue runs through Latchkey; anything else is off | off | +| `LATCHKEY_HEAVY` | `on` routes push-to-`main` runs that touched a heavy path through Latchkey; anything else is off | off | | `LATCHKEY_BUDGET_START` | ISO start of the free-minute period; unset keeps everything on GitHub | none | | `LATCHKEY_BUDGET_EQMIN` | free small-equivalent minutes in the period | 10000 | | `LATCHKEY_EQ_PER_RUN` | estimated small-equivalent minutes per routed run | 48 | @@ -89,22 +95,22 @@ a few percent on the 10-24 trial. Compare it to the budget estimate above. - Every `pull_request` run uses `ubuntu-latest`, fork or not. The routing jobs are skipped on PR runs. -- A `merge_group` run can use Latchkey, including a group that holds a - reviewed fork change. The route does not inspect where queued PRs came - from. -- So queue admission is the trust decision: a maintainer puts a PR in the - queue only after its gate passed. A green test run, or this expression, does - not establish that trust. GitHub doesn't enforce the seat review either; the - ruleset requires only a green merge queue, so the review gate is operating - process. +- `merge_group` runs also use `ubuntu-latest` — the routing jobs are skipped + there too, so the required check never touches Latchkey. +- Latchkey runs only on a push to `main`, i.e. after a change has already + merged. So only code that cleared review and the merge queue ever reaches the + privileged runners — a stricter boundary than the old `merge_group` routing, + where the group held PRs that had passed their gate but not yet merged. The + merge is the trust decision; a green test run, or this expression, does not + establish it. - That is acceptable here because the test jobs get only `contents: read`, the workflow has no `pull_request_target` trigger, and the jobs see test fixtures, not production secrets. Two routing jobs get more, and both run on GitHub-hosted runners: the route gets `actions: read`, to read the run's jobs and the hold and budget artifacts, and the backstop gets `actions: write`, to cancel its own run. A queued change can rewrite those - jobs like any other part of the workflow, which is why queue admission, not - this file, is the trust decision. Latchkey documents single-job VMs + jobs like any other part of the workflow, which is why the merge to `main`, + not this file, is the trust decision. Latchkey documents single-job VMs destroyed after each job (vendor-published, not independently attested). ## Why these sizes, and the Mongo ulimit