From f1b01d0193213492bf1fdb3e3c4c293bd9f83572 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 18:43:29 -0700 Subject: [PATCH 1/3] quest(m1): plan the QUIC-wins race flake and a PR-time drill mutation gate Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/drill-mutation-gate.md | 48 +++++++++++++++++++++ quest/m1/test-flakes-2/README.md | 1 + quest/m1/test-flakes-2/quic-winning-race.md | 29 +++++++++++++ 4 files changed, 79 insertions(+) create mode 100644 quest/m1/drill-mutation-gate.md create mode 100644 quest/m1/test-flakes-2/quic-winning-race.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 43ac7d19607..6f8079b8ce4 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -72,6 +72,7 @@ session outcomes, and embedder refusals inside session outcomes. - [Live TS clock validation](/quest/m1/ts-live-clock.md) - resolve the local PCR-rate failure with reproducible timing evidence and CI controls - [Same-instance TS resume](/quest/m1/ts-same-instance-resume.md) - prove valid continuation and fix the unexpected final exporter failure without weakening identity - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause +- [Drill mutation gate](/quest/m1/drill-mutation-gate.md) - a PR touching a drill mutation's target runs that mutation's sensitivity check as a required job - [END_OF_TRACK placement](/quest/m1/ietf-end-of-track-placement.md) - END_OF_TRACK rides the upstream's Location, and Rust's header stops claiming END_OF_GROUP - [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone - [Pipelined requests](/quest/m1/pipeline-requests/README.md) - SUBSCRIBE and the first FETCH go out with the track-info request at every hop, so first data arrives a round trip sooner per hop diff --git a/quest/m1/drill-mutation-gate.md b/quest/m1/drill-mutation-gate.md new file mode 100644 index 00000000000..1df26d1175a --- /dev/null +++ b/quest/m1/drill-mutation-gate.md @@ -0,0 +1,48 @@ +# [M] Drill mutations prove sensitivity on the PR that could break them + +## Goal + +A PR that stops a drill mutation from proving sensitivity fails a required CI +check before merge, rather than turning the nightly +`Tests (test drill-sensitivity)` job red after it lands. + +## Plan + +Why: #5054 (merged 2026-10-08) moved the sources' release out of what +`test/drill/mutations/subscriber-leaks-broadcasts.patch` leaked. The patch +still applied, so the `--apply-only` gate in `just check` (`check:drill` in +`sh/dispatch.sh`) stayed green. The mutation no longer caused a leak, and the +nightly stayed red for 3 days until #5338. `--apply-only` catches a patch +that stops applying, not one that applies but no longer reintroduces the bug. + +Decided 2026-10-11: + +- **Mechanism:** PR CI runs `test/drill/sensitivity.sh` for just the + mutations the PR could break. Running it in local `just check` was + rejected because it costs every developer a cold snapshot build. Alerting + on a red nightly instead was rejected because it doesn't stop the merge. +- **Trigger:** a mutation runs when the PR changes a file its patch modifies + (the patch's `+++ b/` paths), the patch itself, or + `rs/moq-relay/tests/drills.rs`. A change to `test/drill/sensitivity.sh` + runs every mutation. +- **Required, not advisory:** a stale mutation is a real coverage gap. The + drills run on mocked time, so the job should be deterministic. +- **Cost, measured 2026-10-11:** 62 of 973 commits on `main` in the prior two + weeks touched a target file, 42 of them `rs/moq-net/src/lite/subscriber.rs`. + The 2026-10-11 nightly ran five mutations in about 18 minutes. + +Implementation guidance: + +- A required check has to report on every PR, so the job always runs and + passes quickly when nothing is selected. Branch protection only accepts a + check that reports. +- Making the check required is a repository setting. Ask the maintainer to + add it once the job is green on `main`. +- Keep the nightly full run: it still catches drift from changes the + trigger misses, such as a refactor in a file the patch doesn't name. + +Public API: none. Wire: none. + +## Related + +- [More tests under load](/quest/m1/test-flakes-2/README.md) - the other test-reliability work in this milestone diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index 190a5307845..2bfe2d0cb29 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -34,6 +34,7 @@ Public API: none. Wire: none. - [Interop close code](/quest/m1/test-flakes-2/interop-close-code.md) - the browser close-code test reliably sees `unauthorized` for a refused session - [TS passthrough jump](/quest/m1/test-flakes-2/ts-passthrough-jump.md) - moq-cli's flagged-jump TS passthrough test passes under load without wall-clock pacing +- [QUIC-wins race](/quest/m1/test-flakes-2/quic-winning-race.md) - moq-tokio's QUIC-wins test sets up QUIC's win instead of racing the WebSocket head start ## Related diff --git a/quest/m1/test-flakes-2/quic-winning-race.md b/quest/m1/test-flakes-2/quic-winning-race.md new file mode 100644 index 00000000000..d819f048b97 --- /dev/null +++ b/quest/m1/test-flakes-2/quic-winning-race.md @@ -0,0 +1,29 @@ +# [S] QUIC-wins race test holds under load + +## Goal + +`connection::tests::quic_winning_opens_one_session` (`rs/moq-tokio`) passes +under a loaded `just check`, fixed at its cause. It still proves that when +QUIC wins, the WebSocket fallback is dropped and never opens a session. + +## Plan + +Seen 2026-10-11 by two agents independently, at load averages of 50 to 130. +WebSocket won the race, so the test's assertion that the transport is +`WebTransport` failed. It passes alone. + +Cause: the test races a real QUIC handshake against the 200 ms WebSocket head +start (`DEFAULT_DELAY` in `rs/moq-tokio/src/websocket.rs`). A loaded machine +can take longer than that to finish the handshake, and then WebSocket winning +is the product working as designed. The test asserts the outcome of a +wall-clock race. + +Decided 2026-10-11: make QUIC's win part of the test's setup rather than the +result of a race. For example, the fixture's WebSocket side doesn't answer +before the QUIC session is accepted. Pick the mechanism that keeps the +"dropped before it opened a session" assertion meaningful. Don't block on +[Controlled time](/quest/m1/time/README.md): real sockets fight a paused +clock, so the fixture needs this shape either way. Never widen the head start +or a wait, per this questline's rules. + +Public API: none. Wire: none. From a9af73b72ee7004f0e790899acec929c7a438a64 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 20:18:54 -0700 Subject: [PATCH 2/3] quest(m1): keep the drill mutation gate advisory until controlled time The drills use real sockets and wall-clock timing, so a mutated run can fail for the wrong reason under load. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 +- quest/m1/drill-mutation-gate.md | 28 +++++++++++++++++++--------- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index 6f8079b8ce4..80ea3c954f0 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -72,7 +72,7 @@ session outcomes, and embedder refusals inside session outcomes. - [Live TS clock validation](/quest/m1/ts-live-clock.md) - resolve the local PCR-rate failure with reproducible timing evidence and CI controls - [Same-instance TS resume](/quest/m1/ts-same-instance-resume.md) - prove valid continuation and fix the unexpected final exporter failure without weakening identity - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause -- [Drill mutation gate](/quest/m1/drill-mutation-gate.md) - a PR touching a drill mutation's target runs that mutation's sensitivity check as a required job +- [Drill mutation gate](/quest/m1/drill-mutation-gate.md) - a PR touching a drill mutation's target runs that mutation's sensitivity check, advisory until the drills run on controlled time - [END_OF_TRACK placement](/quest/m1/ietf-end-of-track-placement.md) - END_OF_TRACK rides the upstream's Location, and Rust's header stops claiming END_OF_GROUP - [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone - [Pipelined requests](/quest/m1/pipeline-requests/README.md) - SUBSCRIBE and the first FETCH go out with the track-info request at every hop, so first data arrives a round trip sooner per hop diff --git a/quest/m1/drill-mutation-gate.md b/quest/m1/drill-mutation-gate.md index 1df26d1175a..00c777cf1b7 100644 --- a/quest/m1/drill-mutation-gate.md +++ b/quest/m1/drill-mutation-gate.md @@ -2,9 +2,10 @@ ## Goal -A PR that stops a drill mutation from proving sensitivity fails a required CI -check before merge, rather than turning the nightly -`Tests (test drill-sensitivity)` job red after it lands. +A PR that stops a drill mutation from proving sensitivity turns a CI check +red before merge, rather than turning the nightly +`Tests (test drill-sensitivity)` job red after it lands. The check is +advisory until the drills run on a controlled clock, then required. ## Plan @@ -25,19 +26,27 @@ Decided 2026-10-11: (the patch's `+++ b/` paths), the patch itself, or `rs/moq-relay/tests/drills.rs`. A change to `test/drill/sensitivity.sh` runs every mutation. -- **Required, not advisory:** a stale mutation is a real coverage gap. The - drills run on mocked time, so the job should be deterministic. +- **Advisory until controlled time:** the drills are not deterministic yet. + `rs/moq-relay/tests/drills.rs` uses real sockets and wall-clock sleeps and + timeouts, and the impaired lane (which `ingress-burst-never-collected` + runs) draws a random `MOQ_SHAPER_SEED` unless one is set. A mutated run + can fail for the wrong reason under load. Pin the seed in the gate anyway, + so a red run is reproducible. Flip the check to required once the drills + move onto [Controlled time](/quest/m1/time/README.md)'s clock. Blocking on + that was rejected so the signal exists sooner. Reversed on review: the + first pass chose required on the false premise that the drills already + ran on mocked time. - **Cost, measured 2026-10-11:** 62 of 973 commits on `main` in the prior two weeks touched a target file, 42 of them `rs/moq-net/src/lite/subscriber.rs`. The 2026-10-11 nightly ran five mutations in about 18 minutes. Implementation guidance: -- A required check has to report on every PR, so the job always runs and - passes quickly when nothing is selected. Branch protection only accepts a +- Shape the job so it can later become required: it always runs and passes + quickly when nothing is selected, because branch protection only accepts a check that reports. -- Making the check required is a repository setting. Ask the maintainer to - add it once the job is green on `main`. +- Making the check required is a repository setting. Once the drills are on + a controlled clock, ask the maintainer to add it. - Keep the nightly full run: it still catches drift from changes the trigger misses, such as a refactor in a file the patch doesn't name. @@ -46,3 +55,4 @@ Public API: none. Wire: none. ## Related - [More tests under load](/quest/m1/test-flakes-2/README.md) - the other test-reliability work in this milestone +- [Controlled time](/quest/m1/time/README.md) - puts the drills on a controlled clock, which lets this check become required From 59817ea8a71890ff831cbb69682ab1997bfd9681 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 11 Oct 2026 20:23:03 -0700 Subject: [PATCH 3/3] quest(m1): narrow the seed claim and add review guardrails Co-Authored-By: Claude Opus 5.5 --- quest/m1/drill-mutation-gate.md | 7 ++++++- quest/m1/test-flakes-2/quic-winning-race.md | 5 +++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/quest/m1/drill-mutation-gate.md b/quest/m1/drill-mutation-gate.md index 00c777cf1b7..2dd13bcaea3 100644 --- a/quest/m1/drill-mutation-gate.md +++ b/quest/m1/drill-mutation-gate.md @@ -31,7 +31,8 @@ Decided 2026-10-11: timeouts, and the impaired lane (which `ingress-burst-never-collected` runs) draws a random `MOQ_SHAPER_SEED` unless one is set. A mutated run can fail for the wrong reason under load. Pin the seed in the gate anyway, - so a red run is reproducible. Flip the check to required once the drills + so the impaired lane's shaper decisions replay; socket timing and load + still won't. Flip the check to required once the drills move onto [Controlled time](/quest/m1/time/README.md)'s clock. Blocking on that was rejected so the signal exists sooner. Reversed on review: the first pass chose required on the false premise that the drills already @@ -45,6 +46,10 @@ Implementation guidance: - Shape the job so it can later become required: it always runs and passes quickly when nothing is selected, because branch protection only accepts a check that reports. +- Run on `merge_group` too, selecting against its base SHA like + `.github/workflows/check.yml`, so merge-queue commits get the result a + required check needs. Reuse that workflow and `sensitivity.sh`'s + mutation-name arguments rather than a parallel CI framework. - Making the check required is a repository setting. Once the drills are on a controlled clock, ask the maintainer to add it. - Keep the nightly full run: it still catches drift from changes the diff --git a/quest/m1/test-flakes-2/quic-winning-race.md b/quest/m1/test-flakes-2/quic-winning-race.md index d819f048b97..272bc36b9af 100644 --- a/quest/m1/test-flakes-2/quic-winning-race.md +++ b/quest/m1/test-flakes-2/quic-winning-race.md @@ -26,4 +26,9 @@ before the QUIC session is accepted. Pick the mechanism that keeps the clock, so the fixture needs this shape either way. Never widen the head start or a wait, per this questline's rules. +Observe the fallback's cancellation directly, for example by a drop or +connection-close signal. Today `accepted.try_recv()` only proves the queue +is empty at one instant, and a held WebSocket side could hide a fallback +that was never dropped. + Public API: none. Wire: none.