Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## Goal

The work in flight now: a passing Nightly, fixes for the `release` line, the
quest merge driver, and CI cost and latency.
quest merge driver, CI cost and latency, and test flakes.

## Plan

Expand All @@ -22,4 +22,5 @@ edge, which embeds moq-relay; it moved there on 2026-09-28.

- [Nightly passes on main](/quest/m0/nightly/README.md) - the scheduled Nightly passes end to end, so red means a new break
- [Release shutdown backport](/quest/m0/ffi-shutdown-backport.md) - `release` publishes a non-breaking moq-ffi with an async `drain()` and qmux sessions on the QUIC stream limit
- [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in the merge queue as one `just ci` pass, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last
- [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in a batched merge gate as one `just ci` pass per batch, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last
- [More tests under load](/quest/m0/test-flakes-2/README.md) - load-only and queue failures, one quest per flake, fixed at the cause, since each one fails a whole batch
22 changes: 18 additions & 4 deletions quest/m0/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
## Goal

Pull requests are validated locally with `just ci` and get their cloud verdict
from the merge queue without waiting behind the hosted runner limits, and
every job spends its minutes on the work its diff selected rather
than on setup, cold caches, or repeated builds. Concurrent local agents share
from the [batched gate](/quest/m0/ci/land.md) without waiting behind the
hosted runner limits, and every job spends its minutes on the work its diff
selected rather than on setup, cold caches, or repeated builds. Concurrent local agents share
expensive-command capacity, and retained tests earn their cost through useful
regression signal. Valuable checks stay gated or move to an appropriate queue
or nightly run; measured low-value redundancy may be retired.
Expand Down Expand Up @@ -66,6 +66,18 @@ Decisions (maintainer, 2026-10-13):
not needed for isolation. The `nix/runner` VM module is not the deployment
target.

Decided 2026-10-26 (maintainer), from 414 queue merges over 2026-10-23 to
10-26: the queue's latency is serial waiting (98%), not GitHub's overhead
(about 1 minute per merge), with a median of 18 entries ahead and 12.7% of
runs failing. GitHub's merge queue cannot test several PRs in one run, so a
batched gate replaces it on the one self-hosted runner: `quest land`, shared
through kixelated/quest, lands a tested chain of squash commits by
fast-forward ([land](/quest/m0/ci/land.md)). This supersedes concurrent queue
entries on a second runner. Batching gains most when failures are rare, so the
[flake epic](/quest/m0/test-flakes-2/README.md) moved to m0 and
`MIGRATE.md` Unreleased sections, a top source of conflicts, stop being
written per PR.

Decided 2026-10-23 (maintainer), to speed up the serial queue: structural
changes first (concurrent entries, parallel modules, one test build), then a
measured nightly-compiler experiment.
Expand Down Expand Up @@ -114,7 +126,9 @@ The historical samples locate candidates; they are not controlled benchmarks.

## Required

- [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one
- [Land main through quest land](/quest/m0/ci/land.md) - one self-hosted runner tests every queued PR as a squash chain and fast-forwards `main` to the longest passing prefix
- [Bump quest to quest land](/quest/m0/ci/land-bump.md) - condition: kixelated/quest ships `quest land`; then bump the flake input
- [Feature PRs stop editing MIGRATE.md](/quest/m0/ci/migrate-unreleased.md) - Unreleased sections stop conflicting, and the cut writes each version's section
- [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks
- [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands
- [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm CI and releases use `nix.moq.dev` and retire Cachix
Expand Down
15 changes: 15 additions & 0 deletions quest/m0/ci/land-bump.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# [XS] Bump quest to quest land

## Goal

Condition: a kixelated/quest commit on `main` ships `quest land`
([kixelated/quest's plan](https://github.com/kixelated/quest/blob/main/quest/m0/helpers/land.md)).
Check with `git ls-remote https://github.com/kixelated/quest main` and
`quest land --help` from that revision. Then bump the `quest` flake input in
`flake.nix` to that revision, and delete this quest.

## Plan

Decided 2026-10-26 (maintainer): the batched merge gate's engine lives in
kixelated/quest, shared with kixelated/quest itself, moq.pro, and
kixelated/rust2, so moq picks it up through its pinned flake input.
105 changes: 105 additions & 0 deletions quest/m0/ci/land.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# [M] Land main through quest land

## Goal

`main` changes only through a batched gate. `quest land` takes every PR whose
landing was requested and builds them, in enqueue order, into a chain of
squash commits on top of `main`. It runs `just ci` once on the chain's tip on
the self-hosted queue runner, then fast-forwards `main` to the longest
passing prefix of the chain. GitHub's merge queue is gone. Merged PRs per
hour on a busy day rise measurably over the baseline below, while `main`
only ever holds a tree the gate tested.

## Plan

Baseline, measured 2026-10-26 over 2026-10-23 to 10-26 (414 queue merges,
76.7 h):

- Throughput: 5.5 merged PRs per hour overall, and a median of 6.0 in the
hours the queue never emptied. One PR at a time caps it at about 11.
- Latency: from the final enqueue to merged, a median of 87 minutes and a
p90 of 162. Waiting behind other entries is 98% of that. GitHub adds about
1 minute per merge, and a job is a median of 4.7 minutes (p90 15.1).
- Depth: a new entry found a median of 18 entries ahead of it.
- Failures: 60 of about 470 runs failed (12.7%):
- 25 semantic conflicts with PRs that landed ahead. A PR's base was a
median of 22 commits behind the queue's.
- 10 flaky tests, since fixed or tracked in the
[flake epic](/quest/m0/test-flakes-2/README.md).
- 8 lint failures and 9 failures from the PR's own tests or compile
errors. A local `just ci` would have caught about 14 of these 17.
- 8 infra failures, from mbx cache eviction before
[moq-dev/moq#6709](https://github.com/moq-dev/moq/pull/6709).
- Conflicts: 246 merge-conflict removals. 71% touched `quest/` and 41
touched a `MIGRATE.md` Unreleased section.

A simulation of one-runner batching with bisection gave 1.25 merged PRs per
CI run at a 12.7% failure rate (best batch near 4), against 0.88 serial. At
5% it gave about 2.4. The failure rate is the lever, which is why the flake
epic moved to m0 and [MIGRATE.md stops conflicting](/quest/m0/ci/migrate-unreleased.md).

Decided 2026-10-26 (maintainer):

- GitHub's merge queue cannot do this. Its "merge limits do not combine
`merge_group` builds": every entry still runs its own CI. HEADGREEN with
more concurrent builds needs more runners. Mergify (free for open source,
with batching and bisection) was rejected: it merges through the API, so
it can neither guarantee the tested tree nor resolve quest-file conflicts.
- One self-hosted runner stays the gate. This replaces the planned second
queue runner (`queue-concurrency.md`, deleted).
- The engine is `quest land` in kixelated/quest, shared with kixelated/quest,
moq.pro, and kixelated/rust2. This repository adds a thin workflow, as
moq.pro's `gate.yml` and `queue.yml` do today.
- The gate lands by fast-forward. For each PR it writes the squash commit
onto the chain (subject is the PR title, body keeps the co-author
trailers, as CONTRIBUTING describes), and resolves quest-file conflicts
with quest's merge driver. It force-pushes each PR's branch, with a lease
on the head it tested, to that PR's commit, then fast-forwards `main` to
the tip in one push. GitHub marks a PR merged once its head is on the base,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
so the PRs show as merged, `main` stays linear, and the landed tree is
exactly the tested one. A conflict outside `quest/` ejects that PR. The
engine quest decides whether a fork branch that blocks maintainer edits is
refused at enqueue or lands another way.
- Batches take every queued PR, with no size constant. On failure the gate
tests the first half. A passing prefix lands at once and the rest
continues. A PR that fails alone is ejected with a comment linking the
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
run. There are no retries.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
- The rule that quest, test, and CI PRs jump the queue is dropped. Order is
enqueue order.
- The push credential is moq-bot, added to the main ruleset's bypass list.
Its key (`APP_ID`/`APP_CLIENT_ID`, `APP_PRIVATE_KEY`) moves from repository
secrets into an Environment limited to `main`, `release`, and release tags.
That way a workflow on a same-repo PR branch can no longer read a key that
can push to `main`. The release workflows (some run on pushes to
`release`), `dependabot.yml`, and `smoke-toolchain.yml` that use it
reference the Environment. Only the hosted land job reads it, never the job
that runs PR code.

Work in this repository, once [the bump](/quest/m0/ci/land-bump.md) lands:

- The workflow: queue events (landing requested through `quest merge
--land`, a push dequeuing), one concurrency group, inputs resolved on a
hosted job, the chain built and tested on `vars.MOQ_QUEUE_RUNNER` with a
read-only token, and the land job on a hosted runner.
- The main ruleset drops its `merge_queue` rule. It requires a check that PR
heads never carry, so GitHub's merge button and auto-merge cannot land
around the gate. The admin role keeps its emergency bypass. Ruleset writes
Comment on lines +84 to +86

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Migrate Dependabot's enqueue action to the new gate

When the planned ruleset disables GitHub auto-merge, Dependabot updates lose their landing path: .github/workflows/dependabot.yml ends with gh pr merge --auto --squash, while this plan only updates that workflow's credential Environment. The inspected gh pr merge --help defines --auto as “Automatically merge only after necessary requirements are met,” so it cannot satisfy the deliberately unavailable check or request entry into quest land. Include replacing that workflow's auto-merge step with the new enqueue mechanism, preserving its existing coupled-smoke-update guard, so dependency updates continue landing automatically.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Fixed in 83f69e2: the work list now has dependabot.yml request landing through the gate instead of gh pr merge --auto, keeping its coupled smoke update guard.

(Written by Claude Opus 5.5)

are the maintainer's; agents only read.
- `check.yml` and `alert.yml` stop listing `merge_group`, and Check runs
inside the gate. `.github/actions/rust-cache`, `.github/actionlint.yaml`,
and `sh/gh/queue.test.ts` follow, and the test asserts the new event policy.
- `dependabot.yml` requests landing through the gate instead of
`gh pr merge --auto`, which the new ruleset leaves with no path to `main`,
and keeps its coupled smoke update guard.
- Docs that describe the queue: `CONTRIBUTING.md` (its merge queue section
and priority rule), `README.md`, `doc/setup/dev.md`, `infra/README.md`,
`justfile`, `sh/kt/check.sh`, and `test/drill/README.md`.
- `release` keeps its own ruleset and PR checks. It has no queue.
- Measure merged PRs per hour and enqueue-to-merged latency for a busy day
against the baseline, and record both in the PR.

Public API: none. Wire: none.

## Required

- [Bump quest to `quest land`](/quest/m0/ci/land-bump.md) - condition: kixelated/quest ships `quest land`; then bump the flake input
34 changes: 34 additions & 0 deletions quest/m0/ci/migrate-unreleased.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# [S] Feature PRs stop editing MIGRATE.md

## Goal

No PR to `main` adds an "Unreleased" section to a package's `MIGRATE.md`, so
breaking PRs stop conflicting with each other there. Each version's migration
section is written only by the release cut, as
[doc/setup/upgrade.md](/doc/setup/upgrade.md) already says, from the `!` PRs'
Impact bullets and the API diff.

## Plan

Measured 2026-10-26: from 2026-10-23 to 10-26, 41 of the merge queue's 246
merge-conflict removals touched a `MIGRATE.md` Unreleased section
(`rs/moq-net` alone 21, the binding guides about 8 to 11 each). About 40
`MIGRATE.md` files carry one today, though `doc/setup/upgrade.md` and
`doc/lib/index.md` say a section "is written when that release is cut, not as
changes land on `main`".

Decided 2026-10-26 (maintainer): drop the Unreleased sections rather than
switch to per-change fragment files. The cut already rebuilds each section
from the PRs' Impact bullets (CONTRIBUTING's release steps), so the per-PR
edits only duplicated it.

- Find where the habit comes from (an `AGENTS.md`, a skill, or copying), and
correct that source in the same PR. Any rule change is reviewed like one.
- Keep the content of the existing Unreleased sections until the next
breaking cut consumes it. The cut's [migration
audit](/quest/m2/release/migrate-audit.md) starts each version section from
those entries, so no curated note is lost.
- A check that refuses an Unreleased heading on `main` is optional, only if
the source cannot be corrected.

Public API: none. Wire: none.
2 changes: 1 addition & 1 deletion quest/m0/ci/nightly-levers.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,4 +33,4 @@ Public API: none. Wire: none.

## Required

- [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the structural baseline
- [Land main through quest land](/quest/m0/ci/land.md) - the structural baseline, replacing concurrent queue entries
62 changes: 0 additions & 62 deletions quest/m0/ci/queue-concurrency.md

This file was deleted.

2 changes: 1 addition & 1 deletion quest/m0/nightly/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ This README's own work, after the children:
there: the audio near-zero p95 row belongs to
[audio jitter target](/quest/m1/audio-jitter-target/README.md), the
macOS `receive_credit` timeout to
[test flakes](/quest/m1/test-flakes-2/README.md), and the paired bench
[test flakes](/quest/m0/test-flakes-2/README.md), and the paired bench
threshold to [bench CI](/quest/m1/bench-ci.md).

Public API: none. Wire: none.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,21 @@ fixed by [moq-dev/moq#6113](https://github.com/moq-dev/moq/pull/6113).
Added 2026-10-21 and 2026-10-22 (maintainer): failures from the m0/m1
merge runs, one child each.

Decided 2026-10-26 (maintainer): the epic moved from m1 to m0, since every
flake now fails a whole batch of the [batched merge gate](/quest/m0/ci/land.md)
rather than one PR. The queue's recurring flakes from 2026-10-23 to 10-26
were already fixed (the moq-rtsp stall tests, the moq-gst pinned-instance and
sibling-restart tests, the moq-mux missing-sequence test,
`js/common/worklet.test.ts`, and the IETF datagram range test in
[moq-dev/moq#6593](https://github.com/moq-dev/moq/pull/6593)).

Public API: none. Wire: none.

## Required

- [moq-cli relay tests under load](/quest/m1/test-flakes-2/cli-relay-load.md) - the fetch and announced tests wait on events, not timing
- [FFI publisher audio gaps](/quest/m1/test-flakes-2/ffi-audio-gap.md) - interop tone checks stop finding gaps from FFI publishers
- [moq-uring allocations under load](/quest/m1/test-flakes-2/uring-alloc-load.md) - the steady-state send test and MEMLOCK users pass in a parallel check
- [IETF22 fallback](/quest/m1/test-flakes-2/ietf22-fallback.md) - an IETF22 browser viewer follows a fallback within the media lane's bound, keeping its session and its audio
- [Stuck-window eviction test on macOS](/quest/m1/test-flakes-2/stuck-window-evicts.md) - the evicted writer reliably sees its stop on macOS, fixed at the race
- [Swapped Rust to JS Opus](/quest/m1/test-flakes-2/swapped-js-opus.md) - the `rust-swapped -> js` interop cell stops timing out under load
- [moq-cli relay tests under load](/quest/m0/test-flakes-2/cli-relay-load.md) - the fetch and announced tests wait on events, not timing
- [FFI publisher audio gaps](/quest/m0/test-flakes-2/ffi-audio-gap.md) - interop tone checks stop finding gaps from FFI publishers
- [moq-uring allocations under load](/quest/m0/test-flakes-2/uring-alloc-load.md) - the steady-state send test and MEMLOCK users pass in a parallel check
- [IETF22 fallback](/quest/m0/test-flakes-2/ietf22-fallback.md) - an IETF22 browser viewer follows a fallback within the media lane's bound, keeping its session and its audio
- [Stuck-window eviction test on macOS](/quest/m0/test-flakes-2/stuck-window-evicts.md) - the evicted writer reliably sees its stop on macOS, fixed at the race
- [Swapped Rust to JS Opus](/quest/m0/test-flakes-2/swapped-js-opus.md) - the `rust-swapped -> js` interop cell stops timing out under load
Loading
Loading