From 065f1ba579df5e70fb7eabcb6a4537cde86fece1 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 22:49:30 -0700 Subject: [PATCH 1/4] quest: plan a batched merge gate, MIGRATE conflicts, and m0 flakes Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 7 +- quest/m0/ci/README.md | 16 ++- quest/m0/ci/land-bump.md | 15 +++ quest/m0/ci/land.md | 99 +++++++++++++++++++ quest/m0/ci/migrate-unreleased.md | 33 +++++++ quest/m0/ci/nightly-levers.md | 2 +- quest/m0/ci/queue-concurrency.md | 62 ------------ quest/m0/nightly/README.md | 2 +- quest/m0/test-flakes-2/README.md | 58 +++++++++++ .../test-flakes-2/cli-relay-load.md | 0 .../test-flakes-2/export-echo-time.md | 0 .../{m1 => m0}/test-flakes-2/ffi-audio-gap.md | 0 quest/m0/test-flakes-2/ietf-datagram-range.md | 21 ++++ .../test-flakes-2/ietf22-fallback.md | 0 .../test-flakes-2/stuck-window-evicts.md | 0 .../test-flakes-2/uring-alloc-load.md | 0 quest/m1/README.md | 2 +- quest/m1/test-flakes-2/README.md | 47 ++------- 18 files changed, 256 insertions(+), 108 deletions(-) create mode 100644 quest/m0/ci/land-bump.md create mode 100644 quest/m0/ci/land.md create mode 100644 quest/m0/ci/migrate-unreleased.md delete mode 100644 quest/m0/ci/queue-concurrency.md create mode 100644 quest/m0/test-flakes-2/README.md rename quest/{m1 => m0}/test-flakes-2/cli-relay-load.md (100%) rename quest/{m1 => m0}/test-flakes-2/export-echo-time.md (100%) rename quest/{m1 => m0}/test-flakes-2/ffi-audio-gap.md (100%) create mode 100644 quest/m0/test-flakes-2/ietf-datagram-range.md rename quest/{m1 => m0}/test-flakes-2/ietf22-fallback.md (100%) rename quest/{m1 => m0}/test-flakes-2/stuck-window-evicts.md (100%) rename quest/{m1 => m0}/test-flakes-2/uring-alloc-load.md (100%) diff --git a/quest/m0/README.md b/quest/m0/README.md index 000707c559a..4e736a71427 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -2,8 +2,8 @@ ## Goal -The work in flight now: fixes for the `release` line, CI cost, and the -recurring audits. +The work in flight now: fixes for the `release` line, CI cost and test +flakes, and the recurring audits. ## Plan @@ -27,4 +27,5 @@ Each lands its fixes in its own PR and plans only what needs a decision. - [Nightly passes on main](/quest/m0/nightly/README.md) - the scheduled Nightly passes end to end, so red means a new break - [Release shutdown backport](/quest/m0/ffi-shutdown-backport.md) - `release` publishes a non-breaking moq-ffi with an async `drain()` and qmux sessions on the QUIC stream limit -- [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in the merge queue as one `just ci` pass, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last +- [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in a batched merge gate as one `just ci` pass per batch, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last +- [More tests under load](/quest/m0/test-flakes-2/README.md) - load-only and queue failures, one quest per flake, fixed at the cause, since each one fails a whole batch diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 881b908829a..e711fd3de07 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -66,6 +66,18 @@ Decisions (maintainer, 2026-10-13): not needed for isolation. The `nix/runner` VM module is not the deployment target. +Decided 2026-10-26 (maintainer), from 414 queue merges over 2026-10-23 to +10-26: the queue's latency is serial waiting (98%), not GitHub's overhead +(about 1 minute per merge), with a median of 18 entries ahead and 12.7% of +runs failing. GitHub's merge queue cannot test several PRs in one run, so a +batched gate replaces it on the one self-hosted runner: `quest land`, shared +through kixelated/quest, lands a tested chain of squash commits by +fast-forward ([land](/quest/m0/ci/land.md)). This supersedes concurrent queue +entries on a second runner. Batching gains most when failures are rare, so the +[flake epic](/quest/m0/test-flakes-2/README.md) moved to m0 and +`MIGRATE.md` Unreleased sections, a top source of conflicts, stop being +written per PR. + Decided 2026-10-23 (maintainer), to speed up the serial queue: structural changes first (concurrent entries, parallel modules, one test build), then a measured nightly-compiler experiment. @@ -113,7 +125,9 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Require the release gate](/quest/m0/ci/release-gates.md) - condition: the next cut carries `release-gate.yml`; then release PRs require it - [Seed release with API reports](/quest/m0/ci/release-api-seed.md) - the first cut's API diff has a baseline on `release` -- [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one +- [Land main through quest land](/quest/m0/ci/land.md) - one self-hosted runner tests every queued PR as a squash chain and fast-forwards `main` to the longest passing prefix +- [Bump quest to quest land](/quest/m0/ci/land-bump.md) - condition: kixelated/quest ships `quest land`; then bump the flake input +- [Feature PRs stop editing MIGRATE.md](/quest/m0/ci/migrate-unreleased.md) - Unreleased sections stop conflicting, and the cut writes each version's section - [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks - [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the Rust cache's pin - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands diff --git a/quest/m0/ci/land-bump.md b/quest/m0/ci/land-bump.md new file mode 100644 index 00000000000..28b36d986ea --- /dev/null +++ b/quest/m0/ci/land-bump.md @@ -0,0 +1,15 @@ +# [XS] Bump quest to quest land + +## Goal + +Condition: a kixelated/quest commit on `main` ships `quest land` +([kixelated/quest's plan](https://github.com/kixelated/quest/blob/main/quest/m0/helpers/land.md)). +Check with `git ls-remote https://github.com/kixelated/quest main` and +`quest land --help` from that revision. Then bump the `quest` flake input in +`flake.nix` to that revision, and delete this quest. + +## Plan + +Decided 2026-10-26 (maintainer): the batched merge gate's engine lives in +kixelated/quest, shared with kixelated/quest itself, moq.pro, and +kixelated/rust2, so moq picks it up through its pinned flake input. diff --git a/quest/m0/ci/land.md b/quest/m0/ci/land.md new file mode 100644 index 00000000000..ec86b4258fd --- /dev/null +++ b/quest/m0/ci/land.md @@ -0,0 +1,99 @@ +# [M] Land main through quest land + +## Goal + +`main` changes only through a batched gate. `quest land` takes every PR whose +landing was requested and builds them, in enqueue order, into a chain of +squash commits on top of `main`. It runs `just ci` once on the chain's tip on +the self-hosted queue runner, then fast-forwards `main` to the longest +passing prefix of the chain. GitHub's merge queue is gone. Merged PRs per +hour on a busy day rise measurably over the baseline below, while `main` +only ever holds a tree the gate tested. + +## Plan + +Baseline, measured 2026-10-26 over 2026-10-23 to 10-26 (414 queue merges, +76.7 h): + +- Throughput: 5.5 merged PRs per hour overall, and a median of 6.0 in the + hours the queue never emptied. One PR at a time caps it at about 11. +- Latency: from the final enqueue to merged, a median of 87 minutes and a + p90 of 162. Waiting behind other entries is 98% of that. GitHub adds about + 1 minute per merge, and a job is a median of 4.7 minutes (p90 15.1). +- Depth: a new entry found a median of 18 entries ahead of it. +- Failures: 60 of about 470 runs failed (12.7%): + - 25 semantic conflicts with PRs that landed ahead. A PR's base was a + median of 22 commits behind the queue's. + - 10 flaky tests, since fixed or tracked in the + [flake epic](/quest/m0/test-flakes-2/README.md). + - 8 lint failures and 9 failures from the PR's own tests or compile + errors. A local `just ci` would have caught about 14 of these 17. + - 8 infra failures, from mbx cache eviction before + [moq-dev/moq#6709](https://github.com/moq-dev/moq/pull/6709). +- Conflicts: 246 merge-conflict removals. 71% touched `quest/` and 41 + touched a `MIGRATE.md` Unreleased section. + +A simulation of one-runner batching with bisection gave 1.25 merged PRs per +CI run at a 12.7% failure rate (best batch near 4), against 0.88 serial. At +5% it gave about 2.4. The failure rate is the lever, which is why the flake +epic moved to m0 and [MIGRATE.md stops conflicting](/quest/m0/ci/migrate-unreleased.md). + +Decided 2026-10-26 (maintainer): + +- GitHub's merge queue cannot do this. Its "merge limits do not combine + `merge_group` builds": every entry still runs its own CI. HEADGREEN with + more concurrent builds needs more runners. Mergify (free for open source, + with batching and bisection) was rejected: it merges through the API, so + it can neither guarantee the tested tree nor resolve quest-file conflicts. +- One self-hosted runner stays the gate. This replaces the planned second + queue runner (`queue-concurrency.md`, deleted). +- The engine is `quest land` in kixelated/quest, shared with kixelated/quest, + moq.pro, and kixelated/rust2. This repository adds a thin workflow, as + moq.pro's `gate.yml` and `queue.yml` do today. +- The gate lands by fast-forward. For each PR it writes the squash commit + onto the chain (subject is the PR title, body keeps the co-author + trailers, as CONTRIBUTING describes), and resolves quest-file conflicts + with quest's merge driver. It force-pushes each PR's branch, with a lease + on the head it tested, to that PR's commit, then fast-forwards `main` to + the tip in one push. GitHub marks a PR merged once its head is on the base, + so the PRs show as merged, `main` stays linear, and the landed tree is + exactly the tested one. A conflict outside `quest/` ejects that PR. +- Batches take every queued PR, with no size constant. On failure the gate + tests the first half. A passing prefix lands at once and the rest + continues. A PR that fails alone is ejected with a comment linking the + run. There are no retries. +- The rule that quest, test, and CI PRs jump the queue is dropped. Order is + enqueue order. +- The push credential is moq-bot, added to the main ruleset's bypass list. + Its key (`APP_ID`/`APP_CLIENT_ID`, `APP_PRIVATE_KEY`) moves from repository + secrets into an Environment limited to `main` and release tags. That way a + workflow on a same-repo PR branch can no longer read a key that can push to + `main`. The release workflows and `dependabot.yml` that use it reference + the Environment. Only the hosted land job reads it, never the job that runs + PR code. + +Work in this repository, once [the bump](/quest/m0/ci/land-bump.md) lands: + +- The workflow: queue events (landing requested through `quest merge + --land`, a push dequeuing), one concurrency group, inputs resolved on a + hosted job, the chain built and tested on `vars.MOQ_QUEUE_RUNNER` with a + read-only token, and the land job on a hosted runner. +- The main ruleset drops its `merge_queue` rule. It requires a check that PR + heads never carry, so GitHub's merge button and auto-merge cannot land + around the gate. The admin role keeps its emergency bypass. Ruleset writes + are the maintainer's; agents only read. +- `check.yml` and `alert.yml` stop listing `merge_group`, and Check runs + inside the gate. `.github/actions/rust-cache`, `.github/actionlint.yaml`, + and `sh/gh/queue.test.ts` follow, and the test asserts the new event policy. +- Docs that describe the queue: `CONTRIBUTING.md` (its merge queue section + and priority rule), `README.md`, `doc/setup/dev.md`, `infra/README.md`, + `justfile`, `sh/kt/check.sh`, and `test/drill/README.md`. +- `release` keeps its own ruleset and PR checks. It has no queue. +- Measure merged PRs per hour and enqueue-to-merged latency for a busy day + against the baseline, and record both in the PR. + +Public API: none. Wire: none. + +## Required + +- [Bump quest to `quest land`](/quest/m0/ci/land-bump.md) - condition: kixelated/quest ships `quest land`; then bump the flake input diff --git a/quest/m0/ci/migrate-unreleased.md b/quest/m0/ci/migrate-unreleased.md new file mode 100644 index 00000000000..889e67eb440 --- /dev/null +++ b/quest/m0/ci/migrate-unreleased.md @@ -0,0 +1,33 @@ +# [S] Feature PRs stop editing MIGRATE.md + +## Goal + +No PR to `main` adds an "Unreleased" section to a package's `MIGRATE.md`, so +breaking PRs stop conflicting with each other there. Each version's migration +section is written only by the release cut, as +[doc/setup/upgrade.md](/doc/setup/upgrade.md) already says, from the `!` PRs' +Impact bullets and the API diff. + +## Plan + +Measured 2026-10-26: from 2026-10-23 to 10-26, 41 of the merge queue's 246 +merge-conflict removals touched a `MIGRATE.md` Unreleased section +(`rs/moq-net` alone 21, the binding guides about 8 to 11 each). About 40 +`MIGRATE.md` files carry one today, though `doc/setup/upgrade.md` and +`doc/lib/index.md` say a section "is written when that release is cut, not as +changes land on `main`". + +Decided 2026-10-26 (maintainer): drop the Unreleased sections rather than +switch to per-change fragment files. The cut already rebuilds each section +from the PRs' Impact bullets (CONTRIBUTING's release steps), so the per-PR +edits only duplicated it. + +- Find where the habit comes from (an `AGENTS.md`, a skill, or copying), and + correct that source in the same PR. Any rule change is reviewed like one. +- Keep the content of the existing Unreleased sections until the next + breaking cut consumes it. The [release cut](/quest/m2/release/cut.md) folds + each into its version section and deletes it, so no curated note is lost. +- A check that refuses an Unreleased heading on `main` is optional, only if + the source cannot be corrected. + +Public API: none. Wire: none. diff --git a/quest/m0/ci/nightly-levers.md b/quest/m0/ci/nightly-levers.md index 46c8a7dced0..14261285a45 100644 --- a/quest/m0/ci/nightly-levers.md +++ b/quest/m0/ci/nightly-levers.md @@ -33,4 +33,4 @@ Public API: none. Wire: none. ## Required -- [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the structural baseline +- [Land main through quest land](/quest/m0/ci/land.md) - the structural baseline, replacing concurrent queue entries diff --git a/quest/m0/ci/queue-concurrency.md b/quest/m0/ci/queue-concurrency.md deleted file mode 100644 index 4d926161ec0..00000000000 --- a/quest/m0/ci/queue-concurrency.md +++ /dev/null @@ -1,62 +0,0 @@ -# [S] Build queue entries concurrently - -## Goal - -The merge queue builds two or more entries at once on self-hosted capacity, so -merged PRs per hour rise measurably over the 2026-10-23 baseline while every -entry still runs the same gate. - -## Plan - -Decided with the maintainer on 2026-10-23: structural queue changes come before -faster toolchains. - -Measured 2026-10-23 over 32 merge-queue runs: `max_entries_to_build: 1` with one -`moq-queue` runner makes the queue fully serial, about 7 merged PRs per hour, -with a median entry of 135 s (p90 516 s) and 3 to 12 s of runner wait. - -Decided 2026-10-25 (maintainer): no VM. The second runner goes into the -deployed `container@moq-ci` systemd-nspawn container, sandboxed by kernel -primitives (see the [CI epic](/quest/m0/ci/README.md)), not into the -`nix/runner` VM module. The container is defined in the host's own flake, not -in this repository, so registering the runner is a host change. - -Each runner keeps its own work directory, since the nixpkgs runner service -empties it on start. The self-hosted cache step keeps its mbx version stamp per -workspace, so a runner sharing a home with another still drops its own -workspace's state after an mbx version change. - -Decided 2026-10-25 (maintainer): ✅ one home under one static user for both -runners. Concurrent entries share mbx's store, wait on a compilation the other -entry is running instead of repeating it, and draw on one machine-wide compile -budget, which mbx documents as supported -([how it works](https://github.com/jdx/mr-boxington/blob/v1.25.0/docs/how-it-works.md#machine-wide-scheduling)). -nixpkgs gives each runner its own `DynamicUser` by default, so both need the -same `user` and `HOME`. Rejected: a home per runner (two cold caches, double -the disk, two mbx schedulers each budgeting the whole container's CPUs) and a -second container (those costs plus another container). A NixOS container -builds through the host's Nix daemon by default, so the Nix store is shared -either way. - -Remaining (maintainer, on the host): - -- Register a second runner in `container@moq-ci` with its own name and work - directory, under the same static user and `HOME` as the first, with a fresh - registration token. -- Size the container's share of `ci.slice`, which it shares with the `moq-pro` - runner and a desktop, so concurrent entries do not slow each other past the - gain. Queue entries skip `quest gate` (`sh/local.sh` execs directly under - GitHub Actions), so the cgroup quota, not admission, keeps them apart. -- Keep `NEXTEST_TEST_THREADS=24` per entry (48 workers across two entries; - mbx budgets compiles, not tests) and watch the first day for timing flakes - rather than guess a lower number. -- Make the pre-build mbx sweep safe for a concurrent entry first. With - [moq-dev/moq#6709](https://github.com/moq-dev/moq/pull/6709), each job runs - `mbx gc` before it builds (`.github/actions/rust-cache`), and in a shared - home that sweep can evict generated source trees another entry's next - build reads. -- Raise the main ruleset's `max_entries_to_build` from 1 to 2. -- Measure merged PRs per hour and per-entry wall time for a day against the - baseline before going past 2. - -Public API: none. Wire: none. diff --git a/quest/m0/nightly/README.md b/quest/m0/nightly/README.md index 01e0d4c9971..30eef61a961 100644 --- a/quest/m0/nightly/README.md +++ b/quest/m0/nightly/README.md @@ -24,7 +24,7 @@ This README's own work, after the children: there: the audio near-zero p95 row belongs to [audio jitter target](/quest/m1/audio-jitter-target/README.md), the macOS `receive_credit` timeout to - [test flakes](/quest/m1/test-flakes-2/README.md), and the paired bench + [test flakes](/quest/m0/test-flakes-2/README.md), and the paired bench threshold to [bench CI](/quest/m1/bench-ci.md). The `cli-export` bufferbloat failure is expected to clear with [moq-dev/moq#6656](https://github.com/moq-dev/moq/pull/6656). diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md new file mode 100644 index 00000000000..09df02de555 --- /dev/null +++ b/quest/m0/test-flakes-2/README.md @@ -0,0 +1,58 @@ +# More tests hold up under load + +## Goal + +A second round after [#4286](https://github.com/moq-dev/moq/pull/4286): +tests that pass alone but have failed under a loaded `just check`, or +intermittently in CI or interop, pass reliably, each fixed at its cause, +never by raising a timeout or adding a retry. + +## Plan + +Decided in the 2026-09-30 audit: the round held independent flakes in one +quest, so it split into one child per flake, grouping only those that share +a fixture. Each child lands on its own. + +Rules every child keeps: + +- Prefer a paused clock over wall time (`moq-cli`'s subscribe tests already + use `#[tokio::test(start_paused = true)]`), or assert on an event instead + of a deadline. If a test is slow under load because the code under test is + slow, fix that. +- A paused clock auto-advances while the runtime idles, so it can fire a + timer before a real socket delivers. Where real sockets fight the paused + clock, drive the timers with a `moq_clock::Manual` clock or the sim (every + crate's timers run on `moq-clock`), or move the test off real sockets. +- A mocked clock still flakes when the code jitters its timers with `rand` + (the draft-16 cancel race was a retry sweep jittered past a fixed settle + budget). Wait out the jitter's whole window, not a count of small steps. + +This README's own work, after the children: run `just check --all` several +times on a loaded machine, as the first round did. + +The redundant failover drift moved to m0 on 2026-10-20 (maintainer) and is +fixed by [moq-dev/moq#6113](https://github.com/moq-dev/moq/pull/6113). + +Added 2026-10-21 and 2026-10-22 (maintainer): failures from the m0/m1 +merge runs, one child each. + +Decided 2026-10-26 (maintainer): the epic moved from m1 to m0, since every +flake now fails a whole batch of the [batched merge gate](/quest/m0/ci/land.md) +rather than one PR. The claimed media resource child stays in +[m1](/quest/m1/test-flakes-2/README.md) until its PR's claim ends. The queue's +recurring flakes from 2026-10-23 to 10-26 were already fixed (the moq-rtsp +stall tests, the moq-gst pinned-instance and sibling-restart tests, the +moq-mux missing-sequence test, and `js/common/worklet.test.ts`); the one left +is the IETF datagram range test. + +Public API: none. Wire: none. + +## Required + +- [IETF datagram range test](/quest/m0/test-flakes-2/ietf-datagram-range.md) - the `@moq/net` datagram range test stops seeing no objects under load +- [moq-cli relay tests under load](/quest/m0/test-flakes-2/cli-relay-load.md) - the fetch and announced tests wait on events, not timing +- [FFI publisher audio gaps](/quest/m0/test-flakes-2/ffi-audio-gap.md) - interop tone checks stop finding gaps from FFI publishers +- [moq-uring allocations under load](/quest/m0/test-flakes-2/uring-alloc-load.md) - the steady-state send test and MEMLOCK users pass in a parallel check +- [IETF22 fallback](/quest/m0/test-flakes-2/ietf22-fallback.md) - an IETF22 browser viewer follows a fallback within the media lane's bound, keeping its session and its audio +- [Echo report test on mocked time](/quest/m0/test-flakes-2/export-echo-time.md) - the moq-cli echo report test stops taking 35 s of wall time +- [Stuck-window eviction test on macOS](/quest/m0/test-flakes-2/stuck-window-evicts.md) - the evicted writer reliably sees its stop on macOS, fixed at the race diff --git a/quest/m1/test-flakes-2/cli-relay-load.md b/quest/m0/test-flakes-2/cli-relay-load.md similarity index 100% rename from quest/m1/test-flakes-2/cli-relay-load.md rename to quest/m0/test-flakes-2/cli-relay-load.md diff --git a/quest/m1/test-flakes-2/export-echo-time.md b/quest/m0/test-flakes-2/export-echo-time.md similarity index 100% rename from quest/m1/test-flakes-2/export-echo-time.md rename to quest/m0/test-flakes-2/export-echo-time.md diff --git a/quest/m1/test-flakes-2/ffi-audio-gap.md b/quest/m0/test-flakes-2/ffi-audio-gap.md similarity index 100% rename from quest/m1/test-flakes-2/ffi-audio-gap.md rename to quest/m0/test-flakes-2/ffi-audio-gap.md diff --git a/quest/m0/test-flakes-2/ietf-datagram-range.md b/quest/m0/test-flakes-2/ietf-datagram-range.md new file mode 100644 index 00000000000..50059af018f --- /dev/null +++ b/quest/m0/test-flakes-2/ietf-datagram-range.md @@ -0,0 +1,21 @@ +# [S] IETF datagram range test + +## Goal + +`@moq/net`'s "IETF %s: datagrams honor the absolute object range" +(`js/net/src/ietf/publisher.test.ts`) passes reliably under a loaded +`just check`, fixed at its cause. + +## Plan + +It failed once in the merge queue, in +[run 40003622936](https://github.com/moq-dev/moq/actions/runs/40003622936) +for [moq-dev/moq#6528](https://github.com/moq-dev/moq/pull/6528) on +2026-10-23: the received objects were `[]` where `[1]` was expected, after +4 ms, so an assertion ran before the datagram arrived rather than a timeout +firing. It was the only queue failure of this test in about 470 runs from +2026-10-23 to 10-26. Reproduce it under load first; if it cannot be +reproduced, find the ordering the test assumes by reading it, and wait on the +event instead of the assumption. + +Public API: none. Wire: none. diff --git a/quest/m1/test-flakes-2/ietf22-fallback.md b/quest/m0/test-flakes-2/ietf22-fallback.md similarity index 100% rename from quest/m1/test-flakes-2/ietf22-fallback.md rename to quest/m0/test-flakes-2/ietf22-fallback.md diff --git a/quest/m1/test-flakes-2/stuck-window-evicts.md b/quest/m0/test-flakes-2/stuck-window-evicts.md similarity index 100% rename from quest/m1/test-flakes-2/stuck-window-evicts.md rename to quest/m0/test-flakes-2/stuck-window-evicts.md diff --git a/quest/m1/test-flakes-2/uring-alloc-load.md b/quest/m0/test-flakes-2/uring-alloc-load.md similarity index 100% rename from quest/m1/test-flakes-2/uring-alloc-load.md rename to quest/m0/test-flakes-2/uring-alloc-load.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 4f272c6af85..e18c553fa6f 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -130,7 +130,7 @@ surface and zero-copy texture import, land before the cut. - [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it - [Moxygen minimal interop](/quest/m1/smoke-moxygen.md) - waiting on Moxygen to send a registered filter type, then the smoke lane stops hiding its failure - [Moxygen pin bump](/quest/m1/moxygen-pin-bump.md) - waiting on moxygen#264, then the TRACK_STATUS patch is dropped -- [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause +- [Claimed flake awaiting m0](/quest/m1/test-flakes-2/README.md) - the media resource flake joins the m0 flake epic once its PR's claim ends - [Broadcast lifecycle](/quest/m1/broadcast-lifecycle/README.md) - broadcasts are online or offline with no finished state in moq-net, finished-broadcast retention is a fixed moq-mux policy, and each linger timer has one meaning - [A smaller, safer API than release](/quest/m1/api-audit/README.md) - every surface added since release has a consumer or is gone, overlapping operations are collapsed, names mirror across Rust, JS, and the bindings, and every break says how to migrate - [Studio on the GPU](/quest/m1/studio-gpu/README.md) - moq-video encoders take a GPU surface and its renderer reports zero-copy import, proven through moq-studio on NVENC, VideoToolbox, Media Foundation, DMA-BUF, and Metal diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index 54ca836172f..6d1aaa2661d 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -1,49 +1,18 @@ -# More tests hold up under load +# Claimed flake awaiting its move to m0 ## Goal -A second round after [#4286](https://github.com/moq-dev/moq/pull/4286): -tests that pass alone but have failed under a loaded `just check`, or -intermittently in CI or interop, pass reliably, each fixed at its cause, -never by raising a timeout or adding a retry. +The claimed media resource flake joins [More tests under load](/quest/m0/test-flakes-2/README.md) +in m0 once its PR, [moq-dev/moq#6439](https://github.com/moq-dev/moq/pull/6439), +merges or releases its claim. ## Plan -Decided in the 2026-09-30 audit: the round held independent flakes in one -quest, so it split into one child per flake, grouping only those that share -a fixture. Each child lands on its own. - -Rules every child keeps: - -- Prefer a paused clock over wall time (`moq-cli`'s subscribe tests already - use `#[tokio::test(start_paused = true)]`), or assert on an event instead - of a deadline. If a test is slow under load because the code under test is - slow, fix that. -- A paused clock auto-advances while the runtime idles, so it can fire a - timer before a real socket delivers. Where real sockets fight the paused - clock, drive the timers with a `moq_clock::Manual` clock or the sim (every - crate's timers run on `moq-clock`), or move the test off real sockets. -- A mocked clock still flakes when the code jitters its timers with `rand` - (the draft-16 cancel race was a retry sweep jittered past a fixed settle - budget). Wait out the jitter's whole window, not a count of small steps. - -This README's own work, after the children: run `just check --all` several -times on a loaded machine, as the first round did. - -The redundant failover drift moved to m0 on 2026-10-20 (maintainer) and is -fixed by [moq-dev/moq#6113](https://github.com/moq-dev/moq/pull/6113). - -Added 2026-10-21 and 2026-10-22 (maintainer): failures from the m0/m1 -merge runs, one child each. - -Public API: none. Wire: none. +Decided 2026-10-26 (maintainer): the flake epic moved to m0. `quest mv` +refuses to move a claimed quest, since its branch would no longer name it, so +this child stays here until then. If it is still open when its claim ends, +move it with `quest mv` into `/quest/m0/test-flakes-2/` and delete this README. ## Required - [Media resource instrumentation](/quest/m1/test-flakes-2/resource-instrumentation.md) - the media test stops seeing a playing player report nothing to release -- [moq-cli relay tests under load](/quest/m1/test-flakes-2/cli-relay-load.md) - the fetch and announced tests wait on events, not timing -- [FFI publisher audio gaps](/quest/m1/test-flakes-2/ffi-audio-gap.md) - interop tone checks stop finding gaps from FFI publishers -- [moq-uring allocations under load](/quest/m1/test-flakes-2/uring-alloc-load.md) - the steady-state send test and MEMLOCK users pass in a parallel check -- [IETF22 fallback](/quest/m1/test-flakes-2/ietf22-fallback.md) - an IETF22 browser viewer follows a fallback within the media lane's bound, keeping its session and its audio -- [Echo report test on mocked time](/quest/m1/test-flakes-2/export-echo-time.md) - the moq-cli echo report test stops taking 35 s of wall time -- [Stuck-window eviction test on macOS](/quest/m1/test-flakes-2/stuck-window-evicts.md) - the evicted writer reliably sees its stop on macOS, fixed at the race From 02b440f3d92e42316a5873c16ce51f951f458dde Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 22:53:02 -0700 Subject: [PATCH 2/4] quest: let the release branch read the land credential Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/land.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/quest/m0/ci/land.md b/quest/m0/ci/land.md index ec86b4258fd..678ed8dd9fd 100644 --- a/quest/m0/ci/land.md +++ b/quest/m0/ci/land.md @@ -66,11 +66,12 @@ Decided 2026-10-26 (maintainer): enqueue order. - The push credential is moq-bot, added to the main ruleset's bypass list. Its key (`APP_ID`/`APP_CLIENT_ID`, `APP_PRIVATE_KEY`) moves from repository - secrets into an Environment limited to `main` and release tags. That way a - workflow on a same-repo PR branch can no longer read a key that can push to - `main`. The release workflows and `dependabot.yml` that use it reference - the Environment. Only the hosted land job reads it, never the job that runs - PR code. + secrets into an Environment limited to `main`, `release`, and release tags. + That way a workflow on a same-repo PR branch can no longer read a key that + can push to `main`. The release workflows (some run on pushes to + `release`), `dependabot.yml`, and `smoke-toolchain.yml` that use it + reference the Environment. Only the hosted land job reads it, never the job + that runs PR code. Work in this repository, once [the bump](/quest/m0/ci/land-bump.md) lands: From 484bb4987232f8978fa4c2d480d762f607dffcc3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 25 Oct 2026 22:59:49 -0700 Subject: [PATCH 3/4] quest: address review on the batched gate plan Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/README.md | 6 +++--- quest/m0/ci/land.md | 4 +++- quest/m0/ci/migrate-unreleased.md | 5 +++-- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index e711fd3de07..ed84cbbe798 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -3,9 +3,9 @@ ## Goal Pull requests are validated locally with `just ci` and get their cloud verdict -from the merge queue without waiting behind the hosted runner limits, and -every job spends its minutes on the work its diff selected rather -than on setup, cold caches, or repeated builds. Concurrent local agents share +from the [batched gate](/quest/m0/ci/land.md) without waiting behind the +hosted runner limits, and every job spends its minutes on the work its diff +selected rather than on setup, cold caches, or repeated builds. Concurrent local agents share expensive-command capacity, and retained tests earn their cost through useful regression signal. Valuable checks stay gated or move to an appropriate queue or nightly run; measured low-value redundancy may be retired. diff --git a/quest/m0/ci/land.md b/quest/m0/ci/land.md index 678ed8dd9fd..aaa7eb7aac1 100644 --- a/quest/m0/ci/land.md +++ b/quest/m0/ci/land.md @@ -57,7 +57,9 @@ Decided 2026-10-26 (maintainer): on the head it tested, to that PR's commit, then fast-forwards `main` to the tip in one push. GitHub marks a PR merged once its head is on the base, so the PRs show as merged, `main` stays linear, and the landed tree is - exactly the tested one. A conflict outside `quest/` ejects that PR. + exactly the tested one. A conflict outside `quest/` ejects that PR. The + engine quest decides whether a fork branch that blocks maintainer edits is + refused at enqueue or lands another way. - Batches take every queued PR, with no size constant. On failure the gate tests the first half. A passing prefix lands at once and the rest continues. A PR that fails alone is ejected with a comment linking the diff --git a/quest/m0/ci/migrate-unreleased.md b/quest/m0/ci/migrate-unreleased.md index 889e67eb440..b7c0f2de6b6 100644 --- a/quest/m0/ci/migrate-unreleased.md +++ b/quest/m0/ci/migrate-unreleased.md @@ -25,8 +25,9 @@ edits only duplicated it. - Find where the habit comes from (an `AGENTS.md`, a skill, or copying), and correct that source in the same PR. Any rule change is reviewed like one. - Keep the content of the existing Unreleased sections until the next - breaking cut consumes it. The [release cut](/quest/m2/release/cut.md) folds - each into its version section and deletes it, so no curated note is lost. + breaking cut consumes it. The cut's [migration + audit](/quest/m2/release/migrate-audit.md) starts each version section from + those entries, so no curated note is lost. - A check that refuses an Unreleased heading on `main` is optional, only if the source cannot be corrected. From 83f69e281daee2e8b70a58aec25b284deec6483f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 26 Oct 2026 21:50:05 -0700 Subject: [PATCH 4/4] quest(ci): move Dependabot's auto-merge onto the batched gate Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/land.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/quest/m0/ci/land.md b/quest/m0/ci/land.md index aaa7eb7aac1..270c8125390 100644 --- a/quest/m0/ci/land.md +++ b/quest/m0/ci/land.md @@ -88,6 +88,9 @@ Work in this repository, once [the bump](/quest/m0/ci/land-bump.md) lands: - `check.yml` and `alert.yml` stop listing `merge_group`, and Check runs inside the gate. `.github/actions/rust-cache`, `.github/actionlint.yaml`, and `sh/gh/queue.test.ts` follow, and the test asserts the new event policy. +- `dependabot.yml` requests landing through the gate instead of + `gh pr merge --auto`, which the new ruleset leaves with no path to `main`, + and keeps its coupled smoke update guard. - Docs that describe the queue: `CONTRIBUTING.md` (its merge queue section and priority rule), `README.md`, `doc/setup/dev.md`, `infra/README.md`, `justfile`, `sh/kt/check.sh`, and `test/drill/README.md`.