ci: route merge-queue heavy jobs to Latchkey behind a fallback guard

.github/workflows/tests.yml

@@ -22,6 +22,19 @@ on:
   schedule:
     - cron: '0 8 * * *'
   workflow_dispatch:
+    inputs:
+      latchkey:
+        description: Route this run's heavy jobs through the Latchkey path (rehearsal; the merge queue uses vars.LATCHKEY_HEAVY)
+        type: boolean
+        default: false
+      latchkey_probe_label:
+        description: Runner label for the probe; a label with no runners rehearses a held job
+        type: string
+        default: ''
+      latchkey_budget_eqmin:
+        description: Override the free-minute budget; 0 rehearses spent minutes
+        type: string
+        default: ''
 
 permissions:
   contents: read
@@ -48,21 +61,123 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  # Heavy jobs (backend, frontend, service tests) run on Latchkey runners on
+  # merge-queue runs when vars.LATCHKEY_HEAVY is 'on': those runners don't
+  # count against the org's 20-concurrent-job cap that made the queue wait
+  # (D37). Every other run, PRs included, stays on GitHub-hosted runners, which
+  # are free on this public repo. Unsetting LATCHKEY_HEAVY turns it all off.
+  #
+  # A Latchkey job that is held (free minutes spent, or an outage) sits
+  # "queued" on GitHub with no error, so the route never trusts Latchkey
+  # blind: it falls back to GitHub-hosted on a recent hold or a spent budget,
+  # and otherwise waits for this probe to be picked up first. The probe is
+  # deliberately outside the gate's needs: a held probe must not hold the
+  # required check. See docs/runbooks/ci-heavy-runner.md.
+  latchkey-probe:
+    name: Latchkey probe
+    if: ${{ (github.event_name == 'merge_group' && vars.LATCHKEY_HEAVY == 'on') || (github.event_name == 'workflow_dispatch' && inputs.latchkey) }}
+    runs-on: ${{ inputs.latchkey_probe_label || vars.LATCHKEY_LIGHT_LABEL || 'latchkey-small' }}
+    timeout-minutes: 2
+    steps:
+      - run: echo "picked up by $RUNNER_NAME"
+
+  latchkey-route:
+    name: Latchkey route
+    if: ${{ (github.event_name == 'merge_group' && vars.LATCHKEY_HEAVY == 'on') || (github.event_name == 'workflow_dispatch' && inputs.latchkey) }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 6
+    permissions:
+      actions: read
+      contents: read
+    outputs:
+      backend: ${{ steps.route.outputs.backend }}
+      frontend: ${{ steps.route.outputs.frontend }}
+      service: ${{ steps.route.outputs.service }}
+      use: ${{ steps.route.outputs.use }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          sparse-checkout: scripts/latchkey-route.mjs
+          sparse-checkout-cone-mode: false
+      - id: route
+        run: node scripts/latchkey-route.mjs route
+        env:
+          GH_TOKEN: ${{ github.token }}
+          BUDGET_START: ${{ vars.LATCHKEY_BUDGET_START }}
+          BUDGET_EQMIN: ${{ inputs.latchkey_budget_eqmin || vars.LATCHKEY_BUDGET_EQMIN }}
+          EQ_PER_RUN: ${{ vars.LATCHKEY_EQ_PER_RUN }}
+          PICKUP_SEC: ${{ vars.LATCHKEY_PICKUP_SEC }}
+          BACKEND_LABEL: ${{ vars.LATCHKEY_BACKEND_LABEL }}
+          LIGHT_LABEL: ${{ vars.LATCHKEY_LIGHT_LABEL }}
+      - if: ${{ steps.route.outputs.hold == 'true' }}
+        run: echo "probe not picked up in run ${{ github.run_id }}" > latchkey-hold.txt
+      - if: ${{ steps.route.outputs.hold == 'true' }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: latchkey-hold
+          path: latchkey-hold.txt
+          retention-days: 1
+          overwrite: true
+      # One artifact per Latchkey run is the budget's counter.
+      - if: ${{ steps.route.outputs.use == 'true' }}
+        run: echo "${{ github.run_id }}" > latchkey-run.txt
+      - if: ${{ steps.route.outputs.use == 'true' }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: latchkey-run
+          path: latchkey-run.txt
+          retention-days: 35
+          overwrite: true
+
+  # Once the heavy jobs are routed to Latchkey, a held one would leave the
+  # required check pending. This cancels the run instead, so the queue entry
+  # fails loudly, and records a hold so the next runs stay on GitHub.
+  latchkey-backstop:
+    name: Latchkey backstop
+    needs: latchkey-route
+    if: ${{ needs.latchkey-route.outputs.use == 'true' }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 12
+    permissions:
+      actions: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          sparse-checkout: scripts/latchkey-route.mjs
+          sparse-checkout-cone-mode: false
+      - id: watch
+        run: node scripts/latchkey-route.mjs backstop
+        env:
+          GH_TOKEN: ${{ github.token }}
+          # Backend shards + frontend + service shards. Keep in step with the
+          # matrices below, or the backstop passes before every heavy job is listed.
+          EXPECTED_HEAVY: '6'
+      - if: ${{ steps.watch.outputs.tripped == 'true' }}
+        run: echo "heavy job held in run ${{ github.run_id }}" > latchkey-hold.txt
+      - if: ${{ steps.watch.outputs.tripped == 'true' }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: latchkey-hold
+          path: latchkey-hold.txt
+          retention-days: 1
+          overwrite: true
+      - if: ${{ steps.watch.outputs.tripped == 'true' }}
+        run: gh run cancel ${{ github.run_id }} --repo ${{ github.repository }}
+        env:
+          GH_TOKEN: ${{ github.token }}
+
   backend-tests:
     name: Backend tests (${{ matrix.shard }}/3)
     strategy:
       fail-fast: false
       matrix:
         shard: [1, 2, 3]
-    # Heavy jobs run where the repo variable HEAVY_RUNNER points (D37: Latchkey
-    # runners, which don't count against the org's 20-concurrent-job cap).
-    # Unset means GitHub-hosted, so rolling back is deleting one variable. The
-    # job names, and so the required check, don't change either way. A fork
-    # PR's own pull_request run stays on GitHub. A merge_group run uses
-    # HEAVY_RUNNER even when the group holds a reviewed fork change: queue
-    # admission (a maintainer queues a gated PR) is the trust decision, not
-    # this expression. See docs/runbooks/ci-heavy-runner.md.
-    runs-on: ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && vars.HEAVY_RUNNER || 'ubuntu-latest' }}
+    # Routed by latchkey-route (backend on medium, the rest on small; see
+    # the route job). Skipped routing leaves the output empty: GitHub-hosted.
+    needs: latchkey-route
+    if: ${{ !cancelled() }}
+    runs-on: ${{ needs.latchkey-route.outputs.backend || 'ubuntu-latest' }}
     env:
       JWT_SECRET: test-jwt-secret
       SENDGRID_API_KEY: fake-api-key
@@ -109,15 +224,10 @@ jobs:
 
   frontend-tests:
     name: Frontend tests
-    # Heavy jobs run where the repo variable HEAVY_RUNNER points (D37: Latchkey
-    # runners, which don't count against the org's 20-concurrent-job cap).
-    # Unset means GitHub-hosted, so rolling back is deleting one variable. The
-    # job names, and so the required check, don't change either way. A fork
-    # PR's own pull_request run stays on GitHub. A merge_group run uses
-    # HEAVY_RUNNER even when the group holds a reviewed fork change: queue
-    # admission (a maintainer queues a gated PR) is the trust decision, not
-    # this expression. See docs/runbooks/ci-heavy-runner.md.
-    runs-on: ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && vars.HEAVY_RUNNER || 'ubuntu-latest' }}
+    # Routed by latchkey-route, as backend-tests.
+    needs: latchkey-route
+    if: ${{ !cancelled() }}
+    runs-on: ${{ needs.latchkey-route.outputs.frontend || 'ubuntu-latest' }}
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-node@v4
@@ -174,6 +284,7 @@ jobs:
       - run: npm run verify:moltbot-tools
       - run: npm run verify:litellm-patch-runner
       - run: npm run verify:readme-invariants
+      - run: node --test scripts/__tests__/latchkey-route.test.mjs
       # These node:test files lived only in the retired all-in-one job.
       - name: Run agent service tests
         run: |
@@ -190,7 +301,8 @@ jobs:
   test-gate:
     name: Test & Coverage
     if: always()
-    needs: [backend-tests, backend-checks, frontend-tests, tool-tests, service-test, chart-lint]
+    # latchkey-probe is left out on purpose: see the comment above it.
+    needs: [latchkey-route, latchkey-backstop, backend-tests, backend-checks, frontend-tests, tool-tests, service-test, chart-lint]
     runs-on: ubuntu-latest
     env:
       NEEDS_JSON: ${{ toJSON(needs) }}
@@ -199,14 +311,16 @@ jobs:
       - name: Verify gate covers every suite
         run: |
           diff -u \
-            <(awk '/^jobs:$/ { jobs=1; next } jobs && /^  [[:alnum:]_-]+:$/ { id=$1; sub(/:$/, "", id); if (id != "test-gate") print id }' .github/workflows/tests.yml | sort) \
+            <(awk '/^jobs:$/ { jobs=1; next } jobs && /^  [[:alnum:]_-]+:$/ { id=$1; sub(/:$/, "", id); if (id != "test-gate" && id != "latchkey-probe") print id }' .github/workflows/tests.yml | sort) \
             <(printf '%s\n' "$NEEDS_JSON" | jq -r 'keys[]' | sort)
       - name: Verify every check passed
         run: |
           # Suite jobs run unconditionally; a skipped result is a failed gate.
+          # Only the Latchkey routing jobs may be skipped (off on PR runs).
           # Revisit this predicate if a later PR adds path-based job skipping.
           echo "$NEEDS_JSON" | jq -e '
-            [.[].result] | all(. == "success")
+            to_entries | all(.value.result == "success"
+              or ((.key | startswith("latchkey-")) and .value.result == "skipped"))
           '
 
   service-test:
@@ -215,15 +329,10 @@ jobs:
       fail-fast: false
       matrix:
         shard: [1, 2]
-    # Heavy jobs run where the repo variable HEAVY_RUNNER points (D37: Latchkey
-    # runners, which don't count against the org's 20-concurrent-job cap).
-    # Unset means GitHub-hosted, so rolling back is deleting one variable. The
-    # job names, and so the required check, don't change either way. A fork
-    # PR's own pull_request run stays on GitHub. A merge_group run uses
-    # HEAVY_RUNNER even when the group holds a reviewed fork change: queue
-    # admission (a maintainer queues a gated PR) is the trust decision, not
-    # this expression. See docs/runbooks/ci-heavy-runner.md.
-    runs-on: ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && vars.HEAVY_RUNNER || 'ubuntu-latest' }}
+    # Routed by latchkey-route, as backend-tests.
+    needs: latchkey-route
+    if: ${{ !cancelled() }}
+    runs-on: ${{ needs.latchkey-route.outputs.service || 'ubuntu-latest' }}
 
     services:
       mongo:

docs/runbooks/ci-heavy-runner.md

@@ -1,48 +1,113 @@
-# CI heavy jobs on Latchkey runners (HEAVY_RUNNER)
+# CI heavy jobs on Latchkey runners
 
 The heavy jobs in `.github/workflows/tests.yml` (the three backend shards,
-frontend tests and the Tier 1 service tests) run on whatever runner label the
-repository variable `HEAVY_RUNNER` names, and on `ubuntu-latest` when it is
-unset. The plan (D54, Sam 2026-10-24) is to set it to `latchkey-medium` once
-the PR adding it merges: Latchkey runners don't count against the org's
-20-concurrent-job cap on GitHub-hosted runners, which was what made the merge
-queue wait.
+frontend tests and the two Tier 1 service shards) can run on Latchkey runners,
+which don't count against the org's 20-concurrent-job cap on GitHub-hosted
+runners. That cap is what made the merge queue wait (D37).
 
-Everything else, including tool-tests, the guards, backend checks and chart
-lint, stays on GitHub. Job names don't change either way, so the required check
-("Test & Coverage") and merge-queue runs are unaffected.
+They do so only on **merge-queue runs**, and only while the repository
+variable `LATCHKEY_HEAVY` is `on`. Every other run, PRs included, stays on
+GitHub-hosted runners, which are free on this public repo. Backend runs on
+`latchkey-medium` and frontend and service on `latchkey-small` (sizes below).
+
+## How a run is routed
+
+A held Latchkey job (free minutes spent, the spend limit reached, or an
+outage) does not fail. It sits "queued" on GitHub with no error, and a
+required check that waits on it leaves the merge queue stalled with nothing
+red anywhere. So nothing is sent to Latchkey blind. The logic is
+`scripts/latchkey-route.mjs`, with tests in
+`scripts/__tests__/latchkey-route.test.mjs`.
+
+1. **Latchkey probe** asks for a `latchkey-small` runner and only echoes. It
+   is not in the gate's needs, so a held probe never holds the required check.
+2. **Latchkey route** (GitHub-hosted) picks GitHub-hosted for the heavy jobs
+   when:
+   - a `latchkey-hold` artifact was recorded in the last 60 minutes;
+   - the budget is spent, or `LATCHKEY_BUDGET_START` is unset; or
+   - the probe isn't picked up within 180 s. This case also records a hold.
+
+   Otherwise it sends the heavy jobs to Latchkey and records a `latchkey-run`
+   artifact, which is the budget's counter. Any API error falls back to
+   GitHub-hosted.
+3. **Latchkey backstop** (GitHub-hosted) watches the heavy jobs once they are
+   routed. If one is still queued after 300 s, it records a hold and cancels
+   the run. The queue entry then fails loudly instead of waiting, and later
+   runs stay on GitHub until the hold expires. The press re-queues the PR as
+   for any dequeue.
+
+The budget is an estimate: Latchkey runs since `LATCHKEY_BUDGET_START` times
+`LATCHKEY_EQ_PER_RUN` (default 48 small-equivalent minutes, measured on the 10-25 rehearsal), stopped at 80%
+of `LATCHKEY_BUDGET_EQMIN` (default 10000). The 20% left is margin for the
+estimate and for runs already in flight. Check it against the measured
+figure below, and raise `LATCHKEY_EQ_PER_RUN` if the measured figure is higher.
+
+## Variables
+
+| variable | meaning | default |
+|---|---|---|
+| `LATCHKEY_HEAVY` | `on` routes merge-queue runs through Latchkey; anything else is off | off |
+| `LATCHKEY_BUDGET_START` | ISO start of the free-minute period; unset keeps everything on GitHub | none |
+| `LATCHKEY_BUDGET_EQMIN` | free small-equivalent minutes in the period | 10000 |
+| `LATCHKEY_EQ_PER_RUN` | estimated small-equivalent minutes per routed run | 48 |
+| `LATCHKEY_PICKUP_SEC` | how long the route waits for the probe | 180 |
+| `LATCHKEY_BACKEND_LABEL` / `LATCHKEY_LIGHT_LABEL` | runner sizes | `latchkey-medium` / `latchkey-small` |
+
+## Turn it on, and off
+
+```bash
+gh variable set LATCHKEY_BUDGET_START --body 2026-10-26T00:00:00Z --repo Team-Commonly/commonly
+gh variable set LATCHKEY_HEAVY --body on --repo Team-Commonly/commonly
+# off: runs that start afterwards are entirely GitHub-hosted
+gh variable delete LATCHKEY_HEAVY --repo Team-Commonly/commonly
+```
+
+Runs already routed keep their labels. If one is held, the backstop cancels it.
+
+## Rehearse the fallback
+
+Dispatch Tests with `latchkey: true`. This works whether or not
+`LATCHKEY_HEAVY` is set, because dispatch is maintainer-only.
+
+- **A held job:** set `latchkey_probe_label` to a label no runner serves. The
+  route waits out the pickup time, records a hold and runs the heavy jobs on
+  GitHub. Cancel the run afterwards, since the probe stays queued. Then delete
+  the hold artifact, or wait an hour for it to expire.
+- **Spent minutes:** set `latchkey_budget_eqmin` to `0`. The route falls back
+  without waiting.
+- **The real path:** no overrides. This spends about 30 small-equivalent
+  minutes.
+
+## Measure the burn
+
+Latchkey bills per job, rounded up to the minute, at 1× for small and 2× for
+medium. Sum `ceil(job minutes) × size` over the Latchkey-labelled jobs from
+the Actions jobs API, per day. The figure matched Latchkey's own meter within
+a few percent on the 10-24 trial. Compare it to the budget estimate above.
 
 ## Trust boundary
 
-- A fork PR's own `pull_request` run always uses `ubuntu-latest`. Unreviewed
-  fork code does not get a Latchkey runner from opening a PR.
-- A `merge_group` run uses `HEAVY_RUNNER`, including a group that holds a
-  reviewed fork change. The expression does not inspect where queued PRs came
+- Every `pull_request` run uses `ubuntu-latest`, fork or not. The routing
+  jobs are skipped on PR runs.
+- A `merge_group` run can use Latchkey, including a group that holds a
+  reviewed fork change. The route does not inspect where queued PRs came
   from.
 - So queue admission is the trust decision: a maintainer puts a PR in the
   queue only after its gate passed. A green test run, or this expression, does
   not establish that trust. GitHub doesn't enforce the seat review either; the
   ruleset requires only a green merge queue, so the review gate is operating
   process.
-- That is acceptable here because the Tests workflow grants only
-  `contents: read`, has no `pull_request_target` trigger, and exposes test
-  fixtures, not production secrets. Latchkey documents single-job VMs
+- That is acceptable here because the test jobs get only `contents: read`,
+  the workflow has no `pull_request_target` trigger, and the jobs see test
+  fixtures, not production secrets. Two routing jobs get more, and both run
+  on GitHub-hosted runners: the route gets `actions: read`, to read the run's
+  jobs and the hold and budget artifacts, and the backstop gets
+  `actions: write`, to cancel its own run. A queued change can rewrite those
+  jobs like any other part of the workflow, which is why queue admission, not
+  this file, is the trust decision. Latchkey documents single-job VMs
   destroyed after each job (vendor-published, not independently attested).
 
-## Flip back to GitHub-hosted
-
-Do this if Latchkey is down, jobs sit queued with no runner, or the free
-minutes run out:
-
-```bash
-gh variable delete HEAVY_RUNNER --repo Team-Commonly/commonly
-```
-
-Runs that start after the delete go to `ubuntu-latest`. Runs already queued
-keep the label they were given, so cancel and re-run those. To flip back to
-Latchkey: `gh variable set HEAVY_RUNNER --body latchkey-medium --repo Team-Commonly/commonly`.
-
-## Why medium, and the Mongo ulimit
+## Why these sizes, and the Mongo ulimit
 
 Measured 2026-10-24 on dispatch-only trial runs:
 
@@ -60,6 +125,10 @@ Measured 2026-10-24 on dispatch-only trial runs:
 tool-tests stays on GitHub: its CLI PATH test fails on Latchkey, and its Jest
 process doesn't exit.
 
-Medium jobs count double against the free budget (14k minutes a month, plus a
-10k bonus to 2026-11-23). Check usage on the Latchkey dashboard before the
-bonus ends.
+Frontend took 2.8 minutes on small against 1.9 on medium, and the service
+tests 5.5 minutes on small, unsharded. Medium counts double against the free
+minutes and is rounded up per job, so a bigger size pays off only when it at
+least halves a job's time: it does for the backend shards (about 18
+small-equivalent minutes a run on medium against 36 on small) and does not for
+frontend or service. Each figure is one trial run, so re-measure before
+changing sizes.