diff --git a/quest/m0/README.md b/quest/m0/README.md index 5ecad752d36..55f21f35783 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -3,7 +3,7 @@ ## Goal The work in flight now: a passing Nightly, fixes for the `release` line, the -quest merge driver, and CI cost and latency. +quest merge driver, CI cost and latency, and test flakes. ## Plan @@ -22,4 +22,5 @@ edge, which embeds moq-relay; it moved there on 2026-09-28. - [Nightly passes on main](/quest/m0/nightly/README.md) - the scheduled Nightly passes end to end, so red means a new break - [Release shutdown backport](/quest/m0/ffi-shutdown-backport.md) - `release` publishes a non-breaking moq-ffi with an async `drain()` and qmux sessions on the QUIC stream limit -- [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in the merge queue as one `just ci` pass, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last +- [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in a batched merge gate as one `just ci` pass per batch, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last +- [More tests under load](/quest/m0/test-flakes-2/README.md) - load-only and queue failures, one quest per flake, fixed at the cause, since each one fails a whole batch diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 467214ab847..93c3a268214 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -3,9 +3,9 @@ ## Goal Pull requests are validated locally with `just ci` and get their cloud verdict -from the merge queue without waiting behind the hosted runner limits, and -every job spends its minutes on the work its diff selected rather -than on setup, cold caches, or repeated builds. Concurrent local agents share +from the [batched gate](/quest/m0/ci/land.md) without waiting behind the +hosted runner limits, and every job spends its minutes on the work its diff +selected rather than on setup, cold caches, or repeated builds. Concurrent local agents share expensive-command capacity, and retained tests earn their cost through useful regression signal. Valuable checks stay gated or move to an appropriate queue or nightly run; measured low-value redundancy may be retired. @@ -66,6 +66,18 @@ Decisions (maintainer, 2026-10-13): not needed for isolation. The `nix/runner` VM module is not the deployment target. +Decided 2026-10-26 (maintainer), from 414 queue merges over 2026-10-23 to +10-26: the queue's latency is serial waiting (98%), not GitHub's overhead +(about 1 minute per merge), with a median of 18 entries ahead and 12.7% of +runs failing. GitHub's merge queue cannot test several PRs in one run, so a +batched gate replaces it on the one self-hosted runner: `quest land`, shared +through kixelated/quest, lands a tested chain of squash commits by +fast-forward ([land](/quest/m0/ci/land.md)). This supersedes concurrent queue +entries on a second runner. Batching gains most when failures are rare, so the +[flake epic](/quest/m0/test-flakes-2/README.md) moved to m0 and +`MIGRATE.md` Unreleased sections, a top source of conflicts, stop being +written per PR. + Decided 2026-10-23 (maintainer), to speed up the serial queue: structural changes first (concurrent entries, parallel modules, one test build), then a measured nightly-compiler experiment. @@ -114,7 +126,9 @@ The historical samples locate candidates; they are not controlled benchmarks. ## Required -- [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one +- [Land main through quest land](/quest/m0/ci/land.md) - one self-hosted runner tests every queued PR as a squash chain and fast-forwards `main` to the longest passing prefix +- [Bump quest to quest land](/quest/m0/ci/land-bump.md) - condition: kixelated/quest ships `quest land`; then bump the flake input +- [Feature PRs stop editing MIGRATE.md](/quest/m0/ci/migrate-unreleased.md) - Unreleased sections stop conflicting, and the cut writes each version's section - [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands - [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm CI and releases use `nix.moq.dev` and retire Cachix diff --git a/quest/m0/ci/land-bump.md b/quest/m0/ci/land-bump.md new file mode 100644 index 00000000000..28b36d986ea --- /dev/null +++ b/quest/m0/ci/land-bump.md @@ -0,0 +1,15 @@ +# [XS] Bump quest to quest land + +## Goal + +Condition: a kixelated/quest commit on `main` ships `quest land` +([kixelated/quest's plan](https://github.com/kixelated/quest/blob/main/quest/m0/helpers/land.md)). +Check with `git ls-remote https://github.com/kixelated/quest main` and +`quest land --help` from that revision. Then bump the `quest` flake input in +`flake.nix` to that revision, and delete this quest. + +## Plan + +Decided 2026-10-26 (maintainer): the batched merge gate's engine lives in +kixelated/quest, shared with kixelated/quest itself, moq.pro, and +kixelated/rust2, so moq picks it up through its pinned flake input. diff --git a/quest/m0/ci/land.md b/quest/m0/ci/land.md new file mode 100644 index 00000000000..270c8125390 --- /dev/null +++ b/quest/m0/ci/land.md @@ -0,0 +1,105 @@ +# [M] Land main through quest land + +## Goal + +`main` changes only through a batched gate. `quest land` takes every PR whose +landing was requested and builds them, in enqueue order, into a chain of +squash commits on top of `main`. It runs `just ci` once on the chain's tip on +the self-hosted queue runner, then fast-forwards `main` to the longest +passing prefix of the chain. GitHub's merge queue is gone. Merged PRs per +hour on a busy day rise measurably over the baseline below, while `main` +only ever holds a tree the gate tested. + +## Plan + +Baseline, measured 2026-10-26 over 2026-10-23 to 10-26 (414 queue merges, +76.7 h): + +- Throughput: 5.5 merged PRs per hour overall, and a median of 6.0 in the + hours the queue never emptied. One PR at a time caps it at about 11. +- Latency: from the final enqueue to merged, a median of 87 minutes and a + p90 of 162. Waiting behind other entries is 98% of that. GitHub adds about + 1 minute per merge, and a job is a median of 4.7 minutes (p90 15.1). +- Depth: a new entry found a median of 18 entries ahead of it. +- Failures: 60 of about 470 runs failed (12.7%): + - 25 semantic conflicts with PRs that landed ahead. A PR's base was a + median of 22 commits behind the queue's. + - 10 flaky tests, since fixed or tracked in the + [flake epic](/quest/m0/test-flakes-2/README.md). + - 8 lint failures and 9 failures from the PR's own tests or compile + errors. A local `just ci` would have caught about 14 of these 17. + - 8 infra failures, from mbx cache eviction before + [moq-dev/moq#6709](https://github.com/moq-dev/moq/pull/6709). +- Conflicts: 246 merge-conflict removals. 71% touched `quest/` and 41 + touched a `MIGRATE.md` Unreleased section. + +A simulation of one-runner batching with bisection gave 1.25 merged PRs per +CI run at a 12.7% failure rate (best batch near 4), against 0.88 serial. At +5% it gave about 2.4. The failure rate is the lever, which is why the flake +epic moved to m0 and [MIGRATE.md stops conflicting](/quest/m0/ci/migrate-unreleased.md). + +Decided 2026-10-26 (maintainer): + +- GitHub's merge queue cannot do this. Its "merge limits do not combine + `merge_group` builds": every entry still runs its own CI. HEADGREEN with + more concurrent builds needs more runners. Mergify (free for open source, + with batching and bisection) was rejected: it merges through the API, so + it can neither guarantee the tested tree nor resolve quest-file conflicts. +- One self-hosted runner stays the gate. This replaces the planned second + queue runner (`queue-concurrency.md`, deleted). +- The engine is `quest land` in kixelated/quest, shared with kixelated/quest, + moq.pro, and kixelated/rust2. This repository adds a thin workflow, as + moq.pro's `gate.yml` and `queue.yml` do today. +- The gate lands by fast-forward. For each PR it writes the squash commit + onto the chain (subject is the PR title, body keeps the co-author + trailers, as CONTRIBUTING describes), and resolves quest-file conflicts + with quest's merge driver. It force-pushes each PR's branch, with a lease + on the head it tested, to that PR's commit, then fast-forwards `main` to + the tip in one push. GitHub marks a PR merged once its head is on the base, + so the PRs show as merged, `main` stays linear, and the landed tree is + exactly the tested one. A conflict outside `quest/` ejects that PR. The + engine quest decides whether a fork branch that blocks maintainer edits is + refused at enqueue or lands another way. +- Batches take every queued PR, with no size constant. On failure the gate + tests the first half. A passing prefix lands at once and the rest + continues. A PR that fails alone is ejected with a comment linking the + run. There are no retries. +- The rule that quest, test, and CI PRs jump the queue is dropped. Order is + enqueue order. +- The push credential is moq-bot, added to the main ruleset's bypass list. + Its key (`APP_ID`/`APP_CLIENT_ID`, `APP_PRIVATE_KEY`) moves from repository + secrets into an Environment limited to `main`, `release`, and release tags. + That way a workflow on a same-repo PR branch can no longer read a key that + can push to `main`. The release workflows (some run on pushes to + `release`), `dependabot.yml`, and `smoke-toolchain.yml` that use it + reference the Environment. Only the hosted land job reads it, never the job + that runs PR code. + +Work in this repository, once [the bump](/quest/m0/ci/land-bump.md) lands: + +- The workflow: queue events (landing requested through `quest merge + --land`, a push dequeuing), one concurrency group, inputs resolved on a + hosted job, the chain built and tested on `vars.MOQ_QUEUE_RUNNER` with a + read-only token, and the land job on a hosted runner. +- The main ruleset drops its `merge_queue` rule. It requires a check that PR + heads never carry, so GitHub's merge button and auto-merge cannot land + around the gate. The admin role keeps its emergency bypass. Ruleset writes + are the maintainer's; agents only read. +- `check.yml` and `alert.yml` stop listing `merge_group`, and Check runs + inside the gate. `.github/actions/rust-cache`, `.github/actionlint.yaml`, + and `sh/gh/queue.test.ts` follow, and the test asserts the new event policy. +- `dependabot.yml` requests landing through the gate instead of + `gh pr merge --auto`, which the new ruleset leaves with no path to `main`, + and keeps its coupled smoke update guard. +- Docs that describe the queue: `CONTRIBUTING.md` (its merge queue section + and priority rule), `README.md`, `doc/setup/dev.md`, `infra/README.md`, + `justfile`, `sh/kt/check.sh`, and `test/drill/README.md`. +- `release` keeps its own ruleset and PR checks. It has no queue. +- Measure merged PRs per hour and enqueue-to-merged latency for a busy day + against the baseline, and record both in the PR. + +Public API: none. Wire: none. + +## Required + +- [Bump quest to `quest land`](/quest/m0/ci/land-bump.md) - condition: kixelated/quest ships `quest land`; then bump the flake input diff --git a/quest/m0/ci/migrate-unreleased.md b/quest/m0/ci/migrate-unreleased.md new file mode 100644 index 00000000000..b7c0f2de6b6 --- /dev/null +++ b/quest/m0/ci/migrate-unreleased.md @@ -0,0 +1,34 @@ +# [S] Feature PRs stop editing MIGRATE.md + +## Goal + +No PR to `main` adds an "Unreleased" section to a package's `MIGRATE.md`, so +breaking PRs stop conflicting with each other there. Each version's migration +section is written only by the release cut, as +[doc/setup/upgrade.md](/doc/setup/upgrade.md) already says, from the `!` PRs' +Impact bullets and the API diff. + +## Plan + +Measured 2026-10-26: from 2026-10-23 to 10-26, 41 of the merge queue's 246 +merge-conflict removals touched a `MIGRATE.md` Unreleased section +(`rs/moq-net` alone 21, the binding guides about 8 to 11 each). About 40 +`MIGRATE.md` files carry one today, though `doc/setup/upgrade.md` and +`doc/lib/index.md` say a section "is written when that release is cut, not as +changes land on `main`". + +Decided 2026-10-26 (maintainer): drop the Unreleased sections rather than +switch to per-change fragment files. The cut already rebuilds each section +from the PRs' Impact bullets (CONTRIBUTING's release steps), so the per-PR +edits only duplicated it. + +- Find where the habit comes from (an `AGENTS.md`, a skill, or copying), and + correct that source in the same PR. Any rule change is reviewed like one. +- Keep the content of the existing Unreleased sections until the next + breaking cut consumes it. The cut's [migration + audit](/quest/m2/release/migrate-audit.md) starts each version section from + those entries, so no curated note is lost. +- A check that refuses an Unreleased heading on `main` is optional, only if + the source cannot be corrected. + +Public API: none. Wire: none. diff --git a/quest/m0/ci/nightly-levers.md b/quest/m0/ci/nightly-levers.md index 46c8a7dced0..14261285a45 100644 --- a/quest/m0/ci/nightly-levers.md +++ b/quest/m0/ci/nightly-levers.md @@ -33,4 +33,4 @@ Public API: none. Wire: none. ## Required -- [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the structural baseline +- [Land main through quest land](/quest/m0/ci/land.md) - the structural baseline, replacing concurrent queue entries diff --git a/quest/m0/ci/queue-concurrency.md b/quest/m0/ci/queue-concurrency.md deleted file mode 100644 index 4d926161ec0..00000000000 --- a/quest/m0/ci/queue-concurrency.md +++ /dev/null @@ -1,62 +0,0 @@ -# [S] Build queue entries concurrently - -## Goal - -The merge queue builds two or more entries at once on self-hosted capacity, so -merged PRs per hour rise measurably over the 2026-10-23 baseline while every -entry still runs the same gate. - -## Plan - -Decided with the maintainer on 2026-10-23: structural queue changes come before -faster toolchains. - -Measured 2026-10-23 over 32 merge-queue runs: `max_entries_to_build: 1` with one -`moq-queue` runner makes the queue fully serial, about 7 merged PRs per hour, -with a median entry of 135 s (p90 516 s) and 3 to 12 s of runner wait. - -Decided 2026-10-25 (maintainer): no VM. The second runner goes into the -deployed `container@moq-ci` systemd-nspawn container, sandboxed by kernel -primitives (see the [CI epic](/quest/m0/ci/README.md)), not into the -`nix/runner` VM module. The container is defined in the host's own flake, not -in this repository, so registering the runner is a host change. - -Each runner keeps its own work directory, since the nixpkgs runner service -empties it on start. The self-hosted cache step keeps its mbx version stamp per -workspace, so a runner sharing a home with another still drops its own -workspace's state after an mbx version change. - -Decided 2026-10-25 (maintainer): ✅ one home under one static user for both -runners. Concurrent entries share mbx's store, wait on a compilation the other -entry is running instead of repeating it, and draw on one machine-wide compile -budget, which mbx documents as supported -([how it works](https://github.com/jdx/mr-boxington/blob/v1.25.0/docs/how-it-works.md#machine-wide-scheduling)). -nixpkgs gives each runner its own `DynamicUser` by default, so both need the -same `user` and `HOME`. Rejected: a home per runner (two cold caches, double -the disk, two mbx schedulers each budgeting the whole container's CPUs) and a -second container (those costs plus another container). A NixOS container -builds through the host's Nix daemon by default, so the Nix store is shared -either way. - -Remaining (maintainer, on the host): - -- Register a second runner in `container@moq-ci` with its own name and work - directory, under the same static user and `HOME` as the first, with a fresh - registration token. -- Size the container's share of `ci.slice`, which it shares with the `moq-pro` - runner and a desktop, so concurrent entries do not slow each other past the - gain. Queue entries skip `quest gate` (`sh/local.sh` execs directly under - GitHub Actions), so the cgroup quota, not admission, keeps them apart. -- Keep `NEXTEST_TEST_THREADS=24` per entry (48 workers across two entries; - mbx budgets compiles, not tests) and watch the first day for timing flakes - rather than guess a lower number. -- Make the pre-build mbx sweep safe for a concurrent entry first. With - [moq-dev/moq#6709](https://github.com/moq-dev/moq/pull/6709), each job runs - `mbx gc` before it builds (`.github/actions/rust-cache`), and in a shared - home that sweep can evict generated source trees another entry's next - build reads. -- Raise the main ruleset's `max_entries_to_build` from 1 to 2. -- Measure merged PRs per hour and per-entry wall time for a day against the - baseline before going past 2. - -Public API: none. Wire: none. diff --git a/quest/m0/nightly/README.md b/quest/m0/nightly/README.md index c4e59de3204..b1ae1052197 100644 --- a/quest/m0/nightly/README.md +++ b/quest/m0/nightly/README.md @@ -24,7 +24,7 @@ This README's own work, after the children: there: the audio near-zero p95 row belongs to [audio jitter target](/quest/m1/audio-jitter-target/README.md), the macOS `receive_credit` timeout to - [test flakes](/quest/m1/test-flakes-2/README.md), and the paired bench + [test flakes](/quest/m0/test-flakes-2/README.md), and the paired bench threshold to [bench CI](/quest/m1/bench-ci.md). Public API: none. Wire: none. diff --git a/quest/m1/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md similarity index 71% rename from quest/m1/test-flakes-2/README.md rename to quest/m0/test-flakes-2/README.md index 280e9355cee..fd53b12013c 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -36,13 +36,21 @@ fixed by [moq-dev/moq#6113](https://github.com/moq-dev/moq/pull/6113). Added 2026-10-21 and 2026-10-22 (maintainer): failures from the m0/m1 merge runs, one child each. +Decided 2026-10-26 (maintainer): the epic moved from m1 to m0, since every +flake now fails a whole batch of the [batched merge gate](/quest/m0/ci/land.md) +rather than one PR. The queue's recurring flakes from 2026-10-23 to 10-26 +were already fixed (the moq-rtsp stall tests, the moq-gst pinned-instance and +sibling-restart tests, the moq-mux missing-sequence test, +`js/common/worklet.test.ts`, and the IETF datagram range test in +[moq-dev/moq#6593](https://github.com/moq-dev/moq/pull/6593)). + Public API: none. Wire: none. ## Required -- [moq-cli relay tests under load](/quest/m1/test-flakes-2/cli-relay-load.md) - the fetch and announced tests wait on events, not timing -- [FFI publisher audio gaps](/quest/m1/test-flakes-2/ffi-audio-gap.md) - interop tone checks stop finding gaps from FFI publishers -- [moq-uring allocations under load](/quest/m1/test-flakes-2/uring-alloc-load.md) - the steady-state send test and MEMLOCK users pass in a parallel check -- [IETF22 fallback](/quest/m1/test-flakes-2/ietf22-fallback.md) - an IETF22 browser viewer follows a fallback within the media lane's bound, keeping its session and its audio -- [Stuck-window eviction test on macOS](/quest/m1/test-flakes-2/stuck-window-evicts.md) - the evicted writer reliably sees its stop on macOS, fixed at the race -- [Swapped Rust to JS Opus](/quest/m1/test-flakes-2/swapped-js-opus.md) - the `rust-swapped -> js` interop cell stops timing out under load +- [moq-cli relay tests under load](/quest/m0/test-flakes-2/cli-relay-load.md) - the fetch and announced tests wait on events, not timing +- [FFI publisher audio gaps](/quest/m0/test-flakes-2/ffi-audio-gap.md) - interop tone checks stop finding gaps from FFI publishers +- [moq-uring allocations under load](/quest/m0/test-flakes-2/uring-alloc-load.md) - the steady-state send test and MEMLOCK users pass in a parallel check +- [IETF22 fallback](/quest/m0/test-flakes-2/ietf22-fallback.md) - an IETF22 browser viewer follows a fallback within the media lane's bound, keeping its session and its audio +- [Stuck-window eviction test on macOS](/quest/m0/test-flakes-2/stuck-window-evicts.md) - the evicted writer reliably sees its stop on macOS, fixed at the race +- [Swapped Rust to JS Opus](/quest/m0/test-flakes-2/swapped-js-opus.md) - the `rust-swapped -> js` interop cell stops timing out under load diff --git a/quest/m1/test-flakes-2/cli-relay-load.md b/quest/m0/test-flakes-2/cli-relay-load.md similarity index 100% rename from quest/m1/test-flakes-2/cli-relay-load.md rename to quest/m0/test-flakes-2/cli-relay-load.md diff --git a/quest/m1/test-flakes-2/ffi-audio-gap.md b/quest/m0/test-flakes-2/ffi-audio-gap.md similarity index 100% rename from quest/m1/test-flakes-2/ffi-audio-gap.md rename to quest/m0/test-flakes-2/ffi-audio-gap.md diff --git a/quest/m1/test-flakes-2/ietf22-fallback.md b/quest/m0/test-flakes-2/ietf22-fallback.md similarity index 100% rename from quest/m1/test-flakes-2/ietf22-fallback.md rename to quest/m0/test-flakes-2/ietf22-fallback.md diff --git a/quest/m1/test-flakes-2/stuck-window-evicts.md b/quest/m0/test-flakes-2/stuck-window-evicts.md similarity index 100% rename from quest/m1/test-flakes-2/stuck-window-evicts.md rename to quest/m0/test-flakes-2/stuck-window-evicts.md diff --git a/quest/m1/test-flakes-2/swapped-js-opus.md b/quest/m0/test-flakes-2/swapped-js-opus.md similarity index 100% rename from quest/m1/test-flakes-2/swapped-js-opus.md rename to quest/m0/test-flakes-2/swapped-js-opus.md diff --git a/quest/m1/test-flakes-2/uring-alloc-load.md b/quest/m0/test-flakes-2/uring-alloc-load.md similarity index 100% rename from quest/m1/test-flakes-2/uring-alloc-load.md rename to quest/m0/test-flakes-2/uring-alloc-load.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 9c70ce9de3d..06b36db1c8c 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -134,7 +134,6 @@ surface and zero-copy texture import, land before the cut. - [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it - [Moxygen minimal interop](/quest/m1/smoke-moxygen.md) - waiting on Moxygen to send a registered filter type, then the smoke lane stops hiding its failure - [Moxygen pin bump](/quest/m1/moxygen-pin-bump.md) - waiting on moxygen#264, then the TRACK_STATUS patch is dropped -- [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause - [Broadcast lifecycle](/quest/m1/broadcast-lifecycle/README.md) - broadcasts are online or offline with no finished state in moq-net, finished-broadcast retention is a fixed moq-mux policy, and each linger timer has one meaning - [A smaller, safer API than release](/quest/m1/api-audit/README.md) - every surface added since release has a consumer or is gone, overlapping operations are collapsed, names mirror across Rust, JS, and the bindings, and every break says how to migrate - [moq-transport 18 and 22 only](/quest/m1/ietf-drop/README.md) - drafts 14-17 and 19-21 are deleted from Rust, JS, tests, interop lanes, and docs, removing the legacy control-stream adapter and its request limit