Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 29 additions & 23 deletions docs/runbooks/ci-heavy-runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,24 @@ frontend tests and the two Tier 1 service shards) can run on Latchkey runners,
which don't count against the org's 20-concurrent-job cap on GitHub-hosted
runners. That cap is what made the merge queue wait.

They do so only on **merge-queue runs**, and only while the repository
variable `LATCHKEY_HEAVY` is `on`. Every other run, PRs included, stays on
GitHub-hosted runners, which are free on this public repo. Backend runs on
`latchkey-medium` and frontend and service on `latchkey-small` (sizes below).
They do so only on a **push to `main`** that touched a heavy path (the
`changes` path filter in the workflow), and only while the repository variable
`LATCHKEY_HEAVY` is `on`. Every other run — PRs, and the `merge_group` required
check — stays on GitHub-hosted runners, which are free on this public repo, so a
docs-only merge spends no Latchkey minutes. Backend runs on `latchkey-medium`
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
and frontend and service on `latchkey-small` (sizes below). (Before 2026-11-01
this routed on `merge_group`; it moved to push:main + the path filter to stay
under the free-minute cap, because the queue ran Latchkey on every entry.)

## How a run is routed

A held Latchkey job (free minutes spent, the spend limit reached, or an
outage) does not fail. It sits "queued" on GitHub with no error, and a
required check that waits on it leaves the merge queue stalled with nothing
red anywhere. So nothing is sent to Latchkey blind. The logic is
`scripts/latchkey-route.mjs`, with tests in
`scripts/__tests__/latchkey-route.test.mjs`.
outage) does not fail. It sits "queued" on GitHub with no error. Latchkey now
runs only on the post-merge push to `main`, so a held job no longer stalls a
required check (the merge queue runs its heavy jobs on GitHub-hosted runners);
the backstop below still cancels the held post-merge run so it does not hang.
Nothing is sent to Latchkey blind. The logic is `scripts/latchkey-route.mjs`,
with tests in `scripts/__tests__/latchkey-route.test.mjs`.

1. **Latchkey probe** asks for a `latchkey-small` runner and only echoes. It
is not in the gate's needs, so a held probe never holds the required check.
Expand All @@ -32,9 +37,10 @@ red anywhere. So nothing is sent to Latchkey blind. The logic is
GitHub-hosted.
3. **Latchkey backstop** (GitHub-hosted) watches the heavy jobs once they are
routed. If one is still queued after 300 s, it records a hold and cancels
the run. The queue entry then fails loudly instead of waiting, and later
runs stay on GitHub until the hold expires. The press re-queues the PR as
for any dequeue.
the (post-merge) run instead of letting it hang, and later push-to-main runs
stay on GitHub until the hold expires. Since this is post-merge there is no
queue entry to re-run; the change is already on `main`, and the `merge_group`
check that gated it ran on GitHub-hosted runners.

The budget is an estimate: Latchkey runs since `LATCHKEY_BUDGET_START` times
`LATCHKEY_EQ_PER_RUN` (default 48 small-equivalent minutes, measured on the 10-25 rehearsal), stopped at 80%
Expand All @@ -46,7 +52,7 @@ figure below, and raise `LATCHKEY_EQ_PER_RUN` if the measured figure is higher.

| variable | meaning | default |
|---|---|---|
| `LATCHKEY_HEAVY` | `on` routes merge-queue runs through Latchkey; anything else is off | off |
| `LATCHKEY_HEAVY` | `on` routes push-to-`main` runs that touched a heavy path through Latchkey; anything else is off | off |
| `LATCHKEY_BUDGET_START` | ISO start of the free-minute period; unset keeps everything on GitHub | none |
| `LATCHKEY_BUDGET_EQMIN` | free small-equivalent minutes in the period | 10000 |
| `LATCHKEY_EQ_PER_RUN` | estimated small-equivalent minutes per routed run | 48 |
Expand Down Expand Up @@ -89,22 +95,22 @@ a few percent on the 10-24 trial. Compare it to the budget estimate above.

- Every `pull_request` run uses `ubuntu-latest`, fork or not. The routing
jobs are skipped on PR runs.
- A `merge_group` run can use Latchkey, including a group that holds a
reviewed fork change. The route does not inspect where queued PRs came
from.
- So queue admission is the trust decision: a maintainer puts a PR in the
queue only after its gate passed. A green test run, or this expression, does
not establish that trust. GitHub doesn't enforce the seat review either; the
ruleset requires only a green merge queue, so the review gate is operating
process.
- `merge_group` runs also use `ubuntu-latest` — the routing jobs are skipped
there too, so the required check never touches Latchkey.
- Latchkey runs only on a push to `main`, i.e. after a change has already
merged. So only code that cleared review and the merge queue ever reaches the
privileged runners — a stricter boundary than the old `merge_group` routing,
where the group held PRs that had passed their gate but not yet merged. The
merge is the trust decision; a green test run, or this expression, does not
establish it.
- That is acceptable here because the test jobs get only `contents: read`,
the workflow has no `pull_request_target` trigger, and the jobs see test
fixtures, not production secrets. Two routing jobs get more, and both run
on GitHub-hosted runners: the route gets `actions: read`, to read the run's
jobs and the hold and budget artifacts, and the backstop gets
`actions: write`, to cancel its own run. A queued change can rewrite those
jobs like any other part of the workflow, which is why queue admission, not
this file, is the trust decision. Latchkey documents single-job VMs
jobs like any other part of the workflow, which is why the merge to `main`,
not this file, is the trust decision. Latchkey documents single-job VMs
destroyed after each job (vendor-published, not independently attested).

## Why these sizes, and the Mongo ulimit
Expand Down
Loading