From 4ffee2881ea6ffc7db9d58742aaff3db71cb096a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 15 Oct 2026 12:58:44 -0700 Subject: [PATCH 1/2] quest: plan the io_uring connection log, three flakes, and Biome on doc/demo Co-Authored-By: Claude Opus 5.5 --- quest/m1/test-flakes-2/README.md | 12 +++-- .../m1/test-flakes-2/cli-announced-connect.md | 31 +++++++++++ quest/m1/test-flakes-2/js-go-rewind.md | 35 +++++++++++++ quest/m1/test-flakes-2/wasm-ietf-read.md | 34 ++++++++++++ quest/m2/README.md | 2 + quest/m2/biome-doc-demo.md | 31 +++++++++++ quest/m2/uring-connection-log.md | 52 +++++++++++++++++++ 7 files changed, 194 insertions(+), 3 deletions(-) create mode 100644 quest/m1/test-flakes-2/cli-announced-connect.md create mode 100644 quest/m1/test-flakes-2/js-go-rewind.md create mode 100644 quest/m1/test-flakes-2/wasm-ietf-read.md create mode 100644 quest/m2/biome-doc-demo.md create mode 100644 quest/m2/uring-connection-log.md diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index 504a9699393..17ce3e6602a 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -3,9 +3,9 @@ ## Goal A second round after [#4286](https://github.com/moq-dev/moq/pull/4286): -tests that pass alone but have failed under a loaded `just check` pass -reliably, each fixed at its cause, never by raising a timeout or adding a -retry. +tests that pass alone but have failed under a loaded `just check`, or +intermittently in CI or interop, pass reliably, each fixed at its cause, +never by raising a timeout or adding a retry. ## Plan @@ -23,6 +23,9 @@ Rules every child keeps: timer before a real socket delivers. Where real sockets fight the paused clock, make the timers mockable or move the test off real sockets. +Added 2026-10-15 (maintainer): three flakes seen while landing that day's +merge batch, ranked by how likely each hides a product bug. + This README's own work, after the children: run `just check --all` several times on a loaded machine, as the first round did. moq-tokio's `a_subscription_cut_by_the_publisher_disconnecting_does_not_end_clean` is a @@ -42,6 +45,9 @@ Public API: none. Wire: none. - [Auth serve signals](/quest/m1/test-flakes-2/auth-serve-signals.md) - moq-cli's auth serve signal tests hold under load - [Export exit tests](/quest/m1/test-flakes-2/export-failure-exit.md) - moq-cli's failed-export and interrupted-publisher exit tests hold under load - [noq client close](/quest/m1/test-flakes-2/noq-client-close.md) - moq-tokio's client close test orders its close after the server's accept instead of racing it +- [js -> go rewind](/quest/m1/test-flakes-2/js-go-rewind.md) - the `js -> go` interop cell never reads a frame below the previous group's start, fixed on the side that produced it +- [WASM IETF read](/quest/m1/test-flakes-2/wasm-ietf-read.md) - the WASM suite's IETF read case reads whole groups on a loaded runner, fixed at the burst or the assertion as the evidence says +- [announced fixture connect](/quest/m1/test-flakes-2/cli-announced-connect.md) - moq-cli's `announced` fixture publisher connects to its ready relay under load - [BBR marking seed](/quest/m1/test-flakes-2/bbr-marking-seed.md) - the BBR marking test seeds its probe schedule, replays a failing seed, and its rare drop is fixed at the cause ## Related diff --git a/quest/m1/test-flakes-2/cli-announced-connect.md b/quest/m1/test-flakes-2/cli-announced-connect.md new file mode 100644 index 00000000000..cb392bad356 --- /dev/null +++ b/quest/m1/test-flakes-2/cli-announced-connect.md @@ -0,0 +1,31 @@ +# [S] moq-cli's announced fixture connects under load + +## Goal + +The `announced` tests in `rs/moq-cli/src/announced.rs`, starting with +`fails_when_the_session_ends`, pass under a loaded `just check`, fixed at +their cause. Their fixture's publisher connects to a relay that has already +reported ready. + +## Plan + +Seen once, locally under `cargo test`, while landing #5575 (2026-10-15). +`fails_when_the_session_ends` failed before reaching its assertion: the +fixture's publisher connect (`Fixture::new`) returned `TransportRace`, +meaning both the QUIC dial and the WebSocket fallback failed +(`race_transport_connect` in `rs/moq-tokio/src/client.rs`). A rerun passed. +Not found in CI. The fixture shares `moq_relay::test_relay` with the other +tests in the file, so all three are exposed. + +A relay that reported ready refusing both transports is not expected under +load, so start by reproducing it (the test in a loop beside a CPU hog, or +`--stress-count` while `just check` runs) and reading both causes the +`TransportRace` error carries. Fix the dial or the relay if either gives up +too early; fix the fixture only if it starts dialing before the relay can +accept. No retry, no wider timeout. + +Public API: none expected. Wire: none. + +## Related + +- [Auth serve signals](/quest/m1/test-flakes-2/auth-serve-signals.md) - another moq-cli test that fails only under load diff --git a/quest/m1/test-flakes-2/js-go-rewind.md b/quest/m1/test-flakes-2/js-go-rewind.md new file mode 100644 index 00000000000..48309a434bc --- /dev/null +++ b/quest/m1/test-flakes-2/js-go-rewind.md @@ -0,0 +1,35 @@ +# [S] js -> go interop never reads a rewound frame + +## Goal + +The `js -> go` interop cell never fails with moq-mux's "frame timestamp ... is +below the previous group's start", fixed at whichever side produces it. If the +browser publisher emitted a group that starts before its predecessor, that is +a publisher bug: rewinds are unsupported, so the publisher must never write +one. If the frames were in order and the Go subscriber's consumer saw them in +the wrong order, that is a consumer bug. + +## Plan + +Seen once, in a local `just test interop --all` on #5546 (2026-10-15): one +`js -> go` cell failed with that error and the rerun passed every cell. Not +found in CI: the 13 failed Interop runs on 2026-10-15 and the nightlies of +2026-10-13 to 2026-10-15 do not contain it. The message comes from moq-mux's +container consumer (`rs/moq-mux/src/container/mod.rs`), which the Go client +reads through moq-ffi. The JS publisher is a real browser encoder +(`test/interop/clients/js`), and js/hang's producers already refuse a frame +below the previous group's start when it is written. So a rewind reaching the +consumer suggests either a JS write path that skips that check, or groups +reordered or skipped on the subscribe side. + +Reproduce first: loop the `js -> go` cell under CPU load, and keep the +timestamps and group sequences both sides saw. Fix the cause, never by +relaxing the consumer's check or retrying the cell. Land a regression at the +layer that was wrong. + +Public API: none expected. Wire: none. + +## Related + +- [Refuse rewinds](/quest/m1/refuse-rewinds.md) - makes every hop refuse a rewound group, which would move this failure to the publisher +- [Resume reorder](/quest/m1/resume-reorder.md) - a late group header on the subscribe side, one candidate cause diff --git a/quest/m1/test-flakes-2/wasm-ietf-read.md b/quest/m1/test-flakes-2/wasm-ietf-read.md new file mode 100644 index 00000000000..b22942eeb21 --- /dev/null +++ b/quest/m1/test-flakes-2/wasm-ietf-read.md @@ -0,0 +1,34 @@ +# [S] The WASM IETF read case holds on CI + +## Goal + +The WASM suite's `ietf: reads a published broadcast` case (`test/wasm/src/main.ts`) +passes on loaded CI runners, fixed at its cause. It still proves that a +browser session over moq-transport reads whole groups, in order, from a +published broadcast. + +## Plan + +Seen twice on the WASM job, both as `read 0 whole groups in 8 attempts, want 2`: +run 38778431765 on #5577 (2026-10-15) and run 38607947479 on +`quest/m0/ci/js-select` (2026-10-14). The lite and SETUP flavours of the same +case passed in both. In the first log, group 0 arrives, then about 500 ms +pass with nothing, then groups 1 to 5 arrive within 6 ms and 6 to 8 soon +after, and the reader hits its attempt bound having read none whole. +`expectGroups` treats a group the live edge passed as a legal skip (`isOld`, +from the default `Latency::REAL_TIME`), so every one of those groups was +retired before it was read. + +Find out why the IETF path stalls and then delivers in a burst while lite +does not, and whether retiring a group that arrived whole within the budget, +only because a newer header landed in the same burst, is correct real-time +behaviour. A product bug is fixed in the product. If the behaviour is right +and the case asks for something real-time delivery does not promise, change +what the case asserts, not its attempt bound or timing. Reproduce under CPU +load before deciding. + +Earlier WASM failures of the same case (2026-10-02 to 2026-10-12) failed with +`unsupported`, a different signature; confirm it is gone rather than folding +it in. + +Public API: none expected. Wire: none. diff --git a/quest/m2/README.md b/quest/m2/README.md index d74752286f7..8e7d0b14e13 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -27,6 +27,7 @@ ranked just below deterministic TS legs, which it requires. - [Binding audio delay](/quest/m2/binding-surface.md) - moq-ffi and every wrapper configure and observe audio playout delay - [Drill mutation gate](/quest/m2/drill-mutation-gate.md) - a PR touching a drill mutation's target runs that mutation's sensitivity check, advisory until the drills run on controlled time +- [Biome on doc/ and demo/](/quest/m2/biome-doc-demo.md) - `just check` and CI lint and format `doc/` and `demo/` with Biome, and both pass - [Window properties](/quest/m2/json-window-properties.md) - random push/pop sequences through the JS and Rust Window codecs decode to the producer's window, for readers that keep up or join late - [Pipelined requests](/quest/m2/pipeline-requests/README.md) - SUBSCRIBE and the first FETCH go out with the track-info request at every hop, so first data arrives a round trip sooner per hop - [OBS native codecs](/quest/m2/obs-moq-video/README.md) - replace FFmpeg video and audio decoding with moq-video and moq-audio, deliver GPU frames, and use native audio/video encoders @@ -129,6 +130,7 @@ ranked just below deterministic TS legs, which it requires. - [Range-addressed HLS playlists](/quest/m2/hls-ranges.md) - `moq-hls` lists a start-to-end range of a recording as its own playlist, for moq.pro's managed HLS - [IETF on the ring](/quest/m2/uring-ietf.md) - the io_uring workers serve moq-transport sessions too, so a uring relay drops no client protocol - [Dropped uring session closes](/quest/m2/uring-drop-close.md) - a moq-uring session dropped without close() closes its connection +- [One connection lifecycle](/quest/m2/uring-connection-log.md) - every accept path, io_uring included, logs a connection's span and close from one owner, so Interop's accounting works on the ring - [Relay io_uring packages](/quest/m2/relay-io-uring-package.md) - Linux relay packages ship io_uring once the ring is on par with tokio - [Opus implementation](/quest/m2/audio-opus-backend.md) - compare Opus codec quality, CPU, build cost, and the loss recovery each backend offers - [Latency ledger](/quest/m2/latency-ledger.md) - a viewer reports its share of playback delay stage by stage: jitter buffer, decode, render, and device diff --git a/quest/m2/biome-doc-demo.md b/quest/m2/biome-doc-demo.md new file mode 100644 index 00000000000..64663531716 --- /dev/null +++ b/quest/m2/biome-doc-demo.md @@ -0,0 +1,31 @@ +# [XS] Biome checks doc/ and demo/ + +## Goal + +`just check`, `just fix`, and CI run Biome over the TypeScript, Vue, and +config files under `doc/` and `demo/`, as #5589 did for `test/`, and both +trees pass. + +## Plan + +#5589 widened the Biome paths in `sh/js/workspace.py` and `js/justfile` +to `test/`. `doc/` and `demo/` are outside them, although the dispatcher's +`js` scope (`sh/dispatch.sh`) already selects JS checks for changes there. + +Measured on main at `de6c27302` with `bun biome check doc demo`: + +- One format error in `doc/.vitepress/drafts.ts`, which `just fix` resolves. +- One `noUnusedVariables` warning on `compact` in + `doc/.vitepress/theme/Community.vue`. It is a false positive: the template + uses it, and Biome does not read Vue templates. Fix it in `biome.jsonc` for + `.vue` files (enable template support if the pinned Biome has it, else + scope the rule off for `.vue`), never by renaming the variable. + +`demo/pub` is a Worker with its own lockfile (the dispatcher's `workers` +scope). Biome reads only the root config, so cover it too unless that +breaks; record the reason if it stays out. + +Keep the selection test in `sh/js/workspace.test.py` covering the new +paths, and update `sh/js/README.md`. + +Public API: none. Wire: none. diff --git a/quest/m2/uring-connection-log.md b/quest/m2/uring-connection-log.md new file mode 100644 index 00000000000..a5cf427d711 --- /dev/null +++ b/quest/m2/uring-connection-log.md @@ -0,0 +1,52 @@ +# [S] One owner logs every relay connection's lifecycle + +## Goal + +Every relay accept path, the io_uring workers included, logs a connection's +`negotiated` and `connection closed` lines under the `moq_relay::connection` +target inside its `conn{id=N}` span, from one place. A connection that is +refused, fails its handshake, or panics still logs its close. The Interop +harness's accounting (`test/interop/relay-log.sh`), which filters to that +target and fails a round whose opened connection never closes, then works +unchanged on a relay started with `--runtime-io-uring`. + +## Plan + +From #5602, which moved the close log into `moq_relay::Connection::run` +(`rs/moq-relay/src/connection.rs`) so QUIC and WebSocket sessions pair their +span with their close under one filter. Checked on main at `de6c27302`: the +io_uring accept path (`serve_connection` in `rs/moq-relay/src/uring.rs`) +does not go through `Connection::run`. It logs `negotiated` and +`connection closed` under `moq_relay::uring`, with the id as a field and no +`conn` span. A panic in its supervise task is isolated and dropped without a +close line. Interop does not run the ring today, so nothing fails yet. + +Root problem: the ring path re-implements the connection lifecycle that +`Connection` owns: admission (the LAN credential branch, auth on the shared +runtime, refusal recording), limits, `lease.preparing`, supervision, and the +logs around them. The logs are only the visible drift. The copy exists +because the handshake and session driver must stay on the `!Send` worker +while admission and supervision run on the shared runtime. + +Recommendations: + +- Make one owner of the lifecycle that both accept paths call, so the span, + the id, the `negotiated` line, and the close (including a panic) cannot + drift again. Prefer sharing admission and supervision with `Connection` + over a logging helper that only hides the duplication; if part of the + admission cannot be shared across the worker boundary, record why. +- The ring connection spans two runtimes, so the `conn` span has to follow + both its worker future and its shared-runtime task. +- Regression: an io_uring session, logged with only the + `moq_relay::connection` target as the harness filters, shows its `conn` + span and its close, like `relay_websocket_logs_its_close_under_a_cluster_id` + in `rs/moq-relay/tests/smoke.rs`. It belongs in `tests/runtime_uring.rs`, + which skips loudly below the kernel floor. + +Public API: none expected. Wire: none. + +## Related + +- [IETF on the ring](/quest/m2/uring-ietf.md) - also touches the ring's accept path +- [Relay io_uring packages](/quest/m2/relay-io-uring-package.md) - ships the ring, whose operators read these logs +- [Session outcomes](/quest/m1/session-outcomes.md) - counts refusals and ends on both paths, from the same duplicated admission From 903200bf932ca364c94a2a6e0731de27f5fb0cc4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 15 Oct 2026 13:02:54 -0700 Subject: [PATCH 2/2] quest: require a panic regression for the ring's connection close Co-Authored-By: Claude Opus 5.5 --- quest/m2/uring-connection-log.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/quest/m2/uring-connection-log.md b/quest/m2/uring-connection-log.md index a5cf427d711..775ff2106cf 100644 --- a/quest/m2/uring-connection-log.md +++ b/quest/m2/uring-connection-log.md @@ -37,11 +37,16 @@ Recommendations: admission cannot be shared across the worker boundary, record why. - The ring connection spans two runtimes, so the `conn` span has to follow both its worker future and its shared-runtime task. -- Regression: an io_uring session, logged with only the - `moq_relay::connection` target as the harness filters, shows its `conn` - span and its close, like `relay_websocket_logs_its_close_under_a_cluster_id` - in `rs/moq-relay/tests/smoke.rs`. It belongs in `tests/runtime_uring.rs`, - which skips loudly below the kernel floor. +- Regressions, logged with only the `moq_relay::connection` target as the + harness filters: + - An io_uring session shows its `conn` span and its close, like + `relay_websocket_logs_its_close_under_a_cluster_id` in + `rs/moq-relay/tests/smoke.rs`. It belongs in `tests/runtime_uring.rs`, + which skips loudly below the kernel floor. + - A panic inside the lifecycle owner, forced on the ring's + shared-runtime supervision, still logs that connection's close. This is + the concrete missing close today: `serve_connection` isolates the + supervise task's panic and drops the result. It must fail without the fix. Public API: none expected. Wire: none.