diff --git a/quest/m1/README.md b/quest/m1/README.md index 43ac7d19607..80ea3c954f0 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -72,6 +72,7 @@ session outcomes, and embedder refusals inside session outcomes. - [Live TS clock validation](/quest/m1/ts-live-clock.md) - resolve the local PCR-rate failure with reproducible timing evidence and CI controls - [Same-instance TS resume](/quest/m1/ts-same-instance-resume.md) - prove valid continuation and fix the unexpected final exporter failure without weakening identity - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause +- [Drill mutation gate](/quest/m1/drill-mutation-gate.md) - a PR touching a drill mutation's target runs that mutation's sensitivity check, advisory until the drills run on controlled time - [END_OF_TRACK placement](/quest/m1/ietf-end-of-track-placement.md) - END_OF_TRACK rides the upstream's Location, and Rust's header stops claiming END_OF_GROUP - [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone - [Pipelined requests](/quest/m1/pipeline-requests/README.md) - SUBSCRIBE and the first FETCH go out with the track-info request at every hop, so first data arrives a round trip sooner per hop diff --git a/quest/m1/drill-mutation-gate.md b/quest/m1/drill-mutation-gate.md new file mode 100644 index 00000000000..2dd13bcaea3 --- /dev/null +++ b/quest/m1/drill-mutation-gate.md @@ -0,0 +1,63 @@ +# [M] Drill mutations prove sensitivity on the PR that could break them + +## Goal + +A PR that stops a drill mutation from proving sensitivity turns a CI check +red before merge, rather than turning the nightly +`Tests (test drill-sensitivity)` job red after it lands. The check is +advisory until the drills run on a controlled clock, then required. + +## Plan + +Why: #5054 (merged 2026-10-08) moved the sources' release out of what +`test/drill/mutations/subscriber-leaks-broadcasts.patch` leaked. The patch +still applied, so the `--apply-only` gate in `just check` (`check:drill` in +`sh/dispatch.sh`) stayed green. The mutation no longer caused a leak, and the +nightly stayed red for 3 days until #5338. `--apply-only` catches a patch +that stops applying, not one that applies but no longer reintroduces the bug. + +Decided 2026-10-11: + +- **Mechanism:** PR CI runs `test/drill/sensitivity.sh` for just the + mutations the PR could break. Running it in local `just check` was + rejected because it costs every developer a cold snapshot build. Alerting + on a red nightly instead was rejected because it doesn't stop the merge. +- **Trigger:** a mutation runs when the PR changes a file its patch modifies + (the patch's `+++ b/` paths), the patch itself, or + `rs/moq-relay/tests/drills.rs`. A change to `test/drill/sensitivity.sh` + runs every mutation. +- **Advisory until controlled time:** the drills are not deterministic yet. + `rs/moq-relay/tests/drills.rs` uses real sockets and wall-clock sleeps and + timeouts, and the impaired lane (which `ingress-burst-never-collected` + runs) draws a random `MOQ_SHAPER_SEED` unless one is set. A mutated run + can fail for the wrong reason under load. Pin the seed in the gate anyway, + so the impaired lane's shaper decisions replay; socket timing and load + still won't. Flip the check to required once the drills + move onto [Controlled time](/quest/m1/time/README.md)'s clock. Blocking on + that was rejected so the signal exists sooner. Reversed on review: the + first pass chose required on the false premise that the drills already + ran on mocked time. +- **Cost, measured 2026-10-11:** 62 of 973 commits on `main` in the prior two + weeks touched a target file, 42 of them `rs/moq-net/src/lite/subscriber.rs`. + The 2026-10-11 nightly ran five mutations in about 18 minutes. + +Implementation guidance: + +- Shape the job so it can later become required: it always runs and passes + quickly when nothing is selected, because branch protection only accepts a + check that reports. +- Run on `merge_group` too, selecting against its base SHA like + `.github/workflows/check.yml`, so merge-queue commits get the result a + required check needs. Reuse that workflow and `sensitivity.sh`'s + mutation-name arguments rather than a parallel CI framework. +- Making the check required is a repository setting. Once the drills are on + a controlled clock, ask the maintainer to add it. +- Keep the nightly full run: it still catches drift from changes the + trigger misses, such as a refactor in a file the patch doesn't name. + +Public API: none. Wire: none. + +## Related + +- [More tests under load](/quest/m1/test-flakes-2/README.md) - the other test-reliability work in this milestone +- [Controlled time](/quest/m1/time/README.md) - puts the drills on a controlled clock, which lets this check become required diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index 190a5307845..2bfe2d0cb29 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -34,6 +34,7 @@ Public API: none. Wire: none. - [Interop close code](/quest/m1/test-flakes-2/interop-close-code.md) - the browser close-code test reliably sees `unauthorized` for a refused session - [TS passthrough jump](/quest/m1/test-flakes-2/ts-passthrough-jump.md) - moq-cli's flagged-jump TS passthrough test passes under load without wall-clock pacing +- [QUIC-wins race](/quest/m1/test-flakes-2/quic-winning-race.md) - moq-tokio's QUIC-wins test sets up QUIC's win instead of racing the WebSocket head start ## Related diff --git a/quest/m1/test-flakes-2/quic-winning-race.md b/quest/m1/test-flakes-2/quic-winning-race.md new file mode 100644 index 00000000000..272bc36b9af --- /dev/null +++ b/quest/m1/test-flakes-2/quic-winning-race.md @@ -0,0 +1,34 @@ +# [S] QUIC-wins race test holds under load + +## Goal + +`connection::tests::quic_winning_opens_one_session` (`rs/moq-tokio`) passes +under a loaded `just check`, fixed at its cause. It still proves that when +QUIC wins, the WebSocket fallback is dropped and never opens a session. + +## Plan + +Seen 2026-10-11 by two agents independently, at load averages of 50 to 130. +WebSocket won the race, so the test's assertion that the transport is +`WebTransport` failed. It passes alone. + +Cause: the test races a real QUIC handshake against the 200 ms WebSocket head +start (`DEFAULT_DELAY` in `rs/moq-tokio/src/websocket.rs`). A loaded machine +can take longer than that to finish the handshake, and then WebSocket winning +is the product working as designed. The test asserts the outcome of a +wall-clock race. + +Decided 2026-10-11: make QUIC's win part of the test's setup rather than the +result of a race. For example, the fixture's WebSocket side doesn't answer +before the QUIC session is accepted. Pick the mechanism that keeps the +"dropped before it opened a session" assertion meaningful. Don't block on +[Controlled time](/quest/m1/time/README.md): real sockets fight a paused +clock, so the fixture needs this shape either way. Never widen the head start +or a wait, per this questline's rules. + +Observe the fallback's cancellation directly, for example by a drop or +connection-close signal. Today `accepted.try_recv()` only proves the queue +is empty at one instant, and a held WebSocket side could hide a fallback +that was never dropped. + +Public API: none. Wire: none.