Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ session outcomes, and embedder refusals inside session outcomes.
- [Live TS clock validation](/quest/m1/ts-live-clock.md) - resolve the local PCR-rate failure with reproducible timing evidence and CI controls
- [Same-instance TS resume](/quest/m1/ts-same-instance-resume.md) - prove valid continuation and fix the unexpected final exporter failure without weakening identity
- [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause
- [Drill mutation gate](/quest/m1/drill-mutation-gate.md) - a PR touching a drill mutation's target runs that mutation's sensitivity check, advisory until the drills run on controlled time
- [END_OF_TRACK placement](/quest/m1/ietf-end-of-track-placement.md) - END_OF_TRACK rides the upstream's Location, and Rust's header stops claiming END_OF_GROUP
- [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone
- [Pipelined requests](/quest/m1/pipeline-requests/README.md) - SUBSCRIBE and the first FETCH go out with the track-info request at every hop, so first data arrives a round trip sooner per hop
Expand Down
63 changes: 63 additions & 0 deletions quest/m1/drill-mutation-gate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# [M] Drill mutations prove sensitivity on the PR that could break them

## Goal

A PR that stops a drill mutation from proving sensitivity turns a CI check
red before merge, rather than turning the nightly
`Tests (test drill-sensitivity)` job red after it lands. The check is
advisory until the drills run on a controlled clock, then required.

## Plan

Why: #5054 (merged 2026-10-08) moved the sources' release out of what
`test/drill/mutations/subscriber-leaks-broadcasts.patch` leaked. The patch
still applied, so the `--apply-only` gate in `just check` (`check:drill` in
`sh/dispatch.sh`) stayed green. The mutation no longer caused a leak, and the
nightly stayed red for 3 days until #5338. `--apply-only` catches a patch
that stops applying, not one that applies but no longer reintroduces the bug.

Decided 2026-10-11:

- **Mechanism:** PR CI runs `test/drill/sensitivity.sh` for just the
mutations the PR could break. Running it in local `just check` was
rejected because it costs every developer a cold snapshot build. Alerting
on a red nightly instead was rejected because it doesn't stop the merge.
- **Trigger:** a mutation runs when the PR changes a file its patch modifies
(the patch's `+++ b/` paths), the patch itself, or
`rs/moq-relay/tests/drills.rs`. A change to `test/drill/sensitivity.sh`
runs every mutation.
- **Advisory until controlled time:** the drills are not deterministic yet.
`rs/moq-relay/tests/drills.rs` uses real sockets and wall-clock sleeps and
timeouts, and the impaired lane (which `ingress-burst-never-collected`
runs) draws a random `MOQ_SHAPER_SEED` unless one is set. A mutated run
can fail for the wrong reason under load. Pin the seed in the gate anyway,
so the impaired lane's shaper decisions replay; socket timing and load
still won't. Flip the check to required once the drills
move onto [Controlled time](/quest/m1/time/README.md)'s clock. Blocking on
that was rejected so the signal exists sooner. Reversed on review: the
first pass chose required on the false premise that the drills already
ran on mocked time.
- **Cost, measured 2026-10-11:** 62 of 973 commits on `main` in the prior two
weeks touched a target file, 42 of them `rs/moq-net/src/lite/subscriber.rs`.
The 2026-10-11 nightly ran five mutations in about 18 minutes.

Implementation guidance:

- Shape the job so it can later become required: it always runs and passes
quickly when nothing is selected, because branch protection only accepts a
check that reports.
- Run on `merge_group` too, selecting against its base SHA like
`.github/workflows/check.yml`, so merge-queue commits get the result a
required check needs. Reuse that workflow and `sensitivity.sh`'s
mutation-name arguments rather than a parallel CI framework.
- Making the check required is a repository setting. Once the drills are on
a controlled clock, ask the maintainer to add it.
- Keep the nightly full run: it still catches drift from changes the
trigger misses, such as a refactor in a file the patch doesn't name.

Public API: none. Wire: none.

## Related

- [More tests under load](/quest/m1/test-flakes-2/README.md) - the other test-reliability work in this milestone
- [Controlled time](/quest/m1/time/README.md) - puts the drills on a controlled clock, which lets this check become required
1 change: 1 addition & 0 deletions quest/m1/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ Public API: none. Wire: none.

- [Interop close code](/quest/m1/test-flakes-2/interop-close-code.md) - the browser close-code test reliably sees `unauthorized` for a refused session
- [TS passthrough jump](/quest/m1/test-flakes-2/ts-passthrough-jump.md) - moq-cli's flagged-jump TS passthrough test passes under load without wall-clock pacing
- [QUIC-wins race](/quest/m1/test-flakes-2/quic-winning-race.md) - moq-tokio's QUIC-wins test sets up QUIC's win instead of racing the WebSocket head start

## Related

Expand Down
34 changes: 34 additions & 0 deletions quest/m1/test-flakes-2/quic-winning-race.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# [S] QUIC-wins race test holds under load

## Goal

`connection::tests::quic_winning_opens_one_session` (`rs/moq-tokio`) passes
under a loaded `just check`, fixed at its cause. It still proves that when
QUIC wins, the WebSocket fallback is dropped and never opens a session.

## Plan

Seen 2026-10-11 by two agents independently, at load averages of 50 to 130.
WebSocket won the race, so the test's assertion that the transport is
`WebTransport` failed. It passes alone.

Cause: the test races a real QUIC handshake against the 200 ms WebSocket head
start (`DEFAULT_DELAY` in `rs/moq-tokio/src/websocket.rs`). A loaded machine
can take longer than that to finish the handshake, and then WebSocket winning
is the product working as designed. The test asserts the outcome of a
wall-clock race.

Decided 2026-10-11: make QUIC's win part of the test's setup rather than the
result of a race. For example, the fixture's WebSocket side doesn't answer
before the QUIC session is accepted. Pick the mechanism that keeps the
"dropped before it opened a session" assertion meaningful. Don't block on
[Controlled time](/quest/m1/time/README.md): real sockets fight a paused
clock, so the fixture needs this shape either way. Never widen the head start
or a wait, per this questline's rules.

Observe the fallback's cancellation directly, for example by a drop or
connection-close signal. Today `accepted.try_recv()` only proves the queue
is empty at one instant, and a held WebSocket side could hide a fallback
that was never dropped.

Public API: none. Wire: none.
Loading