Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions quest/m1/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
## Goal

A second round after [#4286](https://github.com/moq-dev/moq/pull/4286):
tests that pass alone but have failed under a loaded `just check` pass
reliably, each fixed at its cause, never by raising a timeout or adding a
retry.
tests that pass alone but have failed under a loaded `just check`, or
intermittently in CI or interop, pass reliably, each fixed at its cause,
never by raising a timeout or adding a retry.

## Plan

Expand All @@ -23,6 +23,9 @@ Rules every child keeps:
timer before a real socket delivers. Where real sockets fight the paused
clock, make the timers mockable or move the test off real sockets.

Added 2026-10-15 (maintainer): three flakes seen while landing that day's
merge batch, ranked by how likely each hides a product bug.

This README's own work, after the children: run `just check --all` several
times on a loaded machine, as the first round did. moq-tokio's
`a_subscription_cut_by_the_publisher_disconnecting_does_not_end_clean` is a
Expand All @@ -42,6 +45,9 @@ Public API: none. Wire: none.
- [Auth serve signals](/quest/m1/test-flakes-2/auth-serve-signals.md) - moq-cli's auth serve signal tests hold under load
- [Export exit tests](/quest/m1/test-flakes-2/export-failure-exit.md) - moq-cli's failed-export and interrupted-publisher exit tests hold under load
- [noq client close](/quest/m1/test-flakes-2/noq-client-close.md) - moq-tokio's client close test orders its close after the server's accept instead of racing it
- [js -> go rewind](/quest/m1/test-flakes-2/js-go-rewind.md) - the `js -> go` interop cell never reads a frame below the previous group's start, fixed on the side that produced it
- [WASM IETF read](/quest/m1/test-flakes-2/wasm-ietf-read.md) - the WASM suite's IETF read case reads whole groups on a loaded runner, fixed at the burst or the assertion as the evidence says
- [announced fixture connect](/quest/m1/test-flakes-2/cli-announced-connect.md) - moq-cli's `announced` fixture publisher connects to its ready relay under load
- [BBR marking seed](/quest/m1/test-flakes-2/bbr-marking-seed.md) - the BBR marking test seeds its probe schedule, replays a failing seed, and its rare drop is fixed at the cause

## Related
Expand Down
31 changes: 31 additions & 0 deletions quest/m1/test-flakes-2/cli-announced-connect.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# [S] moq-cli's announced fixture connects under load

## Goal

The `announced` tests in `rs/moq-cli/src/announced.rs`, starting with
`fails_when_the_session_ends`, pass under a loaded `just check`, fixed at
their cause. Their fixture's publisher connects to a relay that has already
reported ready.

## Plan

Seen once, locally under `cargo test`, while landing #5575 (2026-10-15).
`fails_when_the_session_ends` failed before reaching its assertion: the
fixture's publisher connect (`Fixture::new`) returned `TransportRace`,
meaning both the QUIC dial and the WebSocket fallback failed
(`race_transport_connect` in `rs/moq-tokio/src/client.rs`). A rerun passed.
Not found in CI. The fixture shares `moq_relay::test_relay` with the other
tests in the file, so all three are exposed.

A relay that reported ready refusing both transports is not expected under
load, so start by reproducing it (the test in a loop beside a CPU hog, or
`--stress-count` while `just check` runs) and reading both causes the
`TransportRace` error carries. Fix the dial or the relay if either gives up
too early; fix the fixture only if it starts dialing before the relay can
accept. No retry, no wider timeout.

Public API: none expected. Wire: none.

## Related

- [Auth serve signals](/quest/m1/test-flakes-2/auth-serve-signals.md) - another moq-cli test that fails only under load
35 changes: 35 additions & 0 deletions quest/m1/test-flakes-2/js-go-rewind.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# [S] js -> go interop never reads a rewound frame

## Goal

The `js -> go` interop cell never fails with moq-mux's "frame timestamp ... is
below the previous group's start", fixed at whichever side produces it. If the
browser publisher emitted a group that starts before its predecessor, that is
a publisher bug: rewinds are unsupported, so the publisher must never write
one. If the frames were in order and the Go subscriber's consumer saw them in
the wrong order, that is a consumer bug.

## Plan

Seen once, in a local `just test interop --all` on #5546 (2026-10-15): one
`js -> go` cell failed with that error and the rerun passed every cell. Not
found in CI: the 13 failed Interop runs on 2026-10-15 and the nightlies of
2026-10-13 to 2026-10-15 do not contain it. The message comes from moq-mux's
container consumer (`rs/moq-mux/src/container/mod.rs`), which the Go client
reads through moq-ffi. The JS publisher is a real browser encoder
(`test/interop/clients/js`), and js/hang's producers already refuse a frame
below the previous group's start when it is written. So a rewind reaching the
consumer suggests either a JS write path that skips that check, or groups
reordered or skipped on the subscribe side.

Reproduce first: loop the `js -> go` cell under CPU load, and keep the
timestamps and group sequences both sides saw. Fix the cause, never by
relaxing the consumer's check or retrying the cell. Land a regression at the
layer that was wrong.

Public API: none expected. Wire: none.

## Related

- [Refuse rewinds](/quest/m1/refuse-rewinds.md) - makes every hop refuse a rewound group, which would move this failure to the publisher
- [Resume reorder](/quest/m1/resume-reorder.md) - a late group header on the subscribe side, one candidate cause
34 changes: 34 additions & 0 deletions quest/m1/test-flakes-2/wasm-ietf-read.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# [S] The WASM IETF read case holds on CI

## Goal

The WASM suite's `ietf: reads a published broadcast` case (`test/wasm/src/main.ts`)
passes on loaded CI runners, fixed at its cause. It still proves that a
browser session over moq-transport reads whole groups, in order, from a
published broadcast.

## Plan

Seen twice on the WASM job, both as `read 0 whole groups in 8 attempts, want 2`:
run 38778431765 on #5577 (2026-10-15) and run 38607947479 on
`quest/m0/ci/js-select` (2026-10-14). The lite and SETUP flavours of the same
case passed in both. In the first log, group 0 arrives, then about 500 ms
pass with nothing, then groups 1 to 5 arrive within 6 ms and 6 to 8 soon
after, and the reader hits its attempt bound having read none whole.
`expectGroups` treats a group the live edge passed as a legal skip (`isOld`,
from the default `Latency::REAL_TIME`), so every one of those groups was
retired before it was read.

Find out why the IETF path stalls and then delivers in a burst while lite
does not, and whether retiring a group that arrived whole within the budget,
only because a newer header landed in the same burst, is correct real-time
behaviour. A product bug is fixed in the product. If the behaviour is right
and the case asks for something real-time delivery does not promise, change
what the case asserts, not its attempt bound or timing. Reproduce under CPU
load before deciding.

Earlier WASM failures of the same case (2026-10-02 to 2026-10-12) failed with
`unsupported`, a different signature; confirm it is gone rather than folding
it in.

Public API: none expected. Wire: none.
2 changes: 2 additions & 0 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ ranked just below deterministic TS legs, which it requires.

- [Binding audio delay](/quest/m2/binding-surface.md) - moq-ffi and every wrapper configure and observe audio playout delay
- [Drill mutation gate](/quest/m2/drill-mutation-gate.md) - a PR touching a drill mutation's target runs that mutation's sensitivity check, advisory until the drills run on controlled time
- [Biome on doc/ and demo/](/quest/m2/biome-doc-demo.md) - `just check` and CI lint and format `doc/` and `demo/` with Biome, and both pass
- [Window properties](/quest/m2/json-window-properties.md) - random push/pop sequences through the JS and Rust Window codecs decode to the producer's window, for readers that keep up or join late
- [Pipelined requests](/quest/m2/pipeline-requests/README.md) - SUBSCRIBE and the first FETCH go out with the track-info request at every hop, so first data arrives a round trip sooner per hop
- [OBS native codecs](/quest/m2/obs-moq-video/README.md) - replace FFmpeg video and audio decoding with moq-video and moq-audio, deliver GPU frames, and use native audio/video encoders
Expand Down Expand Up @@ -129,6 +130,7 @@ ranked just below deterministic TS legs, which it requires.
- [Range-addressed HLS playlists](/quest/m2/hls-ranges.md) - `moq-hls` lists a start-to-end range of a recording as its own playlist, for moq.pro's managed HLS
- [IETF on the ring](/quest/m2/uring-ietf.md) - the io_uring workers serve moq-transport sessions too, so a uring relay drops no client protocol
- [Dropped uring session closes](/quest/m2/uring-drop-close.md) - a moq-uring session dropped without close() closes its connection
- [One connection lifecycle](/quest/m2/uring-connection-log.md) - every accept path, io_uring included, logs a connection's span and close from one owner, so Interop's accounting works on the ring
- [Relay io_uring packages](/quest/m2/relay-io-uring-package.md) - Linux relay packages ship io_uring once the ring is on par with tokio
- [Opus implementation](/quest/m2/audio-opus-backend.md) - compare Opus codec quality, CPU, build cost, and the loss recovery each backend offers
- [Latency ledger](/quest/m2/latency-ledger.md) - a viewer reports its share of playback delay stage by stage: jitter buffer, decode, render, and device
Expand Down
31 changes: 31 additions & 0 deletions quest/m2/biome-doc-demo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# [XS] Biome checks doc/ and demo/

## Goal

`just check`, `just fix`, and CI run Biome over the TypeScript, Vue, and
config files under `doc/` and `demo/`, as #5589 did for `test/`, and both
trees pass.

## Plan

#5589 widened the Biome paths in `sh/js/workspace.py` and `js/justfile`
to `test/`. `doc/` and `demo/` are outside them, although the dispatcher's
`js` scope (`sh/dispatch.sh`) already selects JS checks for changes there.

Measured on main at `de6c27302` with `bun biome check doc demo`:

- One format error in `doc/.vitepress/drafts.ts`, which `just fix` resolves.
- One `noUnusedVariables` warning on `compact` in
`doc/.vitepress/theme/Community.vue`. It is a false positive: the template
uses it, and Biome does not read Vue templates. Fix it in `biome.jsonc` for
`.vue` files (enable template support if the pinned Biome has it, else
scope the rule off for `.vue`), never by renaming the variable.

`demo/pub` is a Worker with its own lockfile (the dispatcher's `workers`
scope). Biome reads only the root config, so cover it too unless that
breaks; record the reason if it stays out.

Keep the selection test in `sh/js/workspace.test.py` covering the new
paths, and update `sh/js/README.md`.

Public API: none. Wire: none.
57 changes: 57 additions & 0 deletions quest/m2/uring-connection-log.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# [S] One owner logs every relay connection's lifecycle

## Goal

Every relay accept path, the io_uring workers included, logs a connection's
`negotiated` and `connection closed` lines under the `moq_relay::connection`
target inside its `conn{id=N}` span, from one place. A connection that is
refused, fails its handshake, or panics still logs its close. The Interop
harness's accounting (`test/interop/relay-log.sh`), which filters to that
target and fails a round whose opened connection never closes, then works
unchanged on a relay started with `--runtime-io-uring`.

## Plan

From #5602, which moved the close log into `moq_relay::Connection::run`
(`rs/moq-relay/src/connection.rs`) so QUIC and WebSocket sessions pair their
span with their close under one filter. Checked on main at `de6c27302`: the
io_uring accept path (`serve_connection` in `rs/moq-relay/src/uring.rs`)
does not go through `Connection::run`. It logs `negotiated` and
`connection closed` under `moq_relay::uring`, with the id as a field and no
`conn` span. A panic in its supervise task is isolated and dropped without a
close line. Interop does not run the ring today, so nothing fails yet.

Root problem: the ring path re-implements the connection lifecycle that
`Connection` owns: admission (the LAN credential branch, auth on the shared
runtime, refusal recording), limits, `lease.preparing`, supervision, and the
logs around them. The logs are only the visible drift. The copy exists
because the handshake and session driver must stay on the `!Send` worker
while admission and supervision run on the shared runtime.

Recommendations:

- Make one owner of the lifecycle that both accept paths call, so the span,
the id, the `negotiated` line, and the close (including a panic) cannot
drift again. Prefer sharing admission and supervision with `Connection`
over a logging helper that only hides the duplication; if part of the
admission cannot be shared across the worker boundary, record why.
- The ring connection spans two runtimes, so the `conn` span has to follow
both its worker future and its shared-runtime task.
- Regressions, logged with only the `moq_relay::connection` target as the
harness filters:
- An io_uring session shows its `conn` span and its close, like
`relay_websocket_logs_its_close_under_a_cluster_id` in
`rs/moq-relay/tests/smoke.rs`. It belongs in `tests/runtime_uring.rs`,
which skips loudly below the kernel floor.
- A panic inside the lifecycle owner, forced on the ring's
shared-runtime supervision, still logs that connection's close. This is
the concrete missing close today: `serve_connection` isolates the
supervise task's panic and drops the result. It must fail without the fix.

Public API: none expected. Wire: none.

## Related

- [IETF on the ring](/quest/m2/uring-ietf.md) - also touches the ring's accept path
- [Relay io_uring packages](/quest/m2/relay-io-uring-package.md) - ships the ring, whose operators read these logs
- [Session outcomes](/quest/m1/session-outcomes.md) - counts refusals and ends on both paths, from the same duplicated admission
Loading