diff --git a/quest/m0/README.md b/quest/m0/README.md index 9428970e7af..04cc561a80d 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -25,6 +25,8 @@ view (`origin::Consumer::local()`) to localhost workers belongs to moq.pro's edge, which embeds moq-relay; it moved there on 2026-09-28. Pools of claim workers (transcoders) also need the relay to forget a front nobody reads (found 2026-10-07). +A demand poll that loses a reader's wake keeps an unread front and its +upstream subscription alive (found 2026-10-08). Interop: Fastly's moq-relay-interop report (run of 2026-09-23, build 7ee2b02) was triaged against `main` on 2026-10-07. Its SETUP, UNSUBSCRIBE @@ -47,7 +49,10 @@ publishers mint epochs. Backport patches cut from `release` don't carry Liveness: a serve loop with work always ready never yields, which starved an FFI publisher's QUIC driver and fails hosted Interop's go lanes (found 2026-10-08 landing #4225). The [serve budget](/quest/m0/serve-budget.md) -bounds every kio task's loop. +bounds every kio task's loop. The Go and Python interop cells it fails +moved here from m1 the same day, because the stall masks interop on every +wire PR and hides as slow passes; the harness now fails a cell whose +connection idles out. Audio playout: the jitter target is default in both languages (#4162); the [line](/quest/m1/audio-jitter-target/README.md) moved to m1 in the @@ -61,6 +66,8 @@ remain and no release waits on them. - [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.4, before Seattle - [Capped stream END_OF_GROUP](/quest/m0/ietf-end-of-track-location.md) - a stream capped by the subscription's end Location never claims END_OF_GROUP; moving End of Track's Location is deferred - [End of Group status](/quest/m0/ietf-end-of-group-status.md) - an End of Group status on a stream whose header already marks the group's end is accepted, so imquic's last object per group arrives +- [Demand lost wake](/quest/m0/demand-lost-wake.md) - a reader that comes and goes between a demand poll and its re-read never leaves a front or `broadcast::Demand` without a wake +- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - every Go and Python publisher cell passes reliably once the serve budget lands, and a cell fails when a connection idles out - [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to - [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines - [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription diff --git a/quest/m0/demand-lost-wake.md b/quest/m0/demand-lost-wake.md new file mode 100644 index 00000000000..b1be3e24146 --- /dev/null +++ b/quest/m0/demand-lost-wake.md @@ -0,0 +1,86 @@ +# [S] Demand polls never lose a wake + +## Goal + +A reader that arrives or leaves between a demand poll and its re-read never +leaves the waiter without a wake. Today a relay front's driver can miss a +track's `Unused` edge: the copy keeps its upstream subscription for nobody, +and the front never retires, with nothing to bound it on a quiet +single-track front. In the other direction a reader stalls on a parked track +until the linger expires. `broadcast::Demand::poll_demand` has the same shape +and is fixed with it, and a closed broadcast never spins the front's holder +edge. + +## Plan + +Facts from `main` at 7c6b6afc1 (line numbers at time of writing): + +- `serve_front`'s wait closure (`rs/moq-net/src/model/origin.rs:2728-2735`) + calls `io.weak.poll_unused` or `poll_used`, then steps only if + `io.weak.is_used() != io.used`. `TrackWeak::poll_used` and `poll_unused` + (`track.rs:2550-2558`) discard the result, and kio + (`rs/kio/src/weak.rs:168-188`) returns `Ready` without registering when + the edge is already met. With `used` true: no consumers, so `Ready` without + a registration; a reader arrives, so `is_used()` is true and the closure + returns `Pending`; the reader leaves and nothing wakes the driver. +- `Step::Demand` (`origin.rs:2841-2857`) re-reads `is_used()` and can emit a + duplicate `Used` or `Unused`. +- `broadcast::Demand::poll_demand` (`broadcast.rs:136-147`, `887-904`): + `register_demand` drops each track's `Ready`, then recomputes `is_used()`. +- On #5054's head 3b6d36f67, `poll_held` and `poll_unheld` + (`broadcast.rs:386-393`) map kio's `Ready(Err(Closed))`, returned before + registering, to `Ready`, and `Step::Holders` `continue`s while + `is_held() == held`. A closed broadcast would livelock the driver in + `kio::wait`, which has no budget. Not reachable today: the only closers of + the front's broadcast, `Action::End` and `Retire` through `End`, end the + driver in the same pass, and `FrontTask` owns the producer. + +Decided 2026-10-08: + +- m0: it undermines the idle-front retire (#5054) and leaks an upstream + subscription. +- Covers `broadcast::Demand` too: same pattern, and publishers rely on its + edges being clean. +- Reuse the handler shape Idle fronts uses for the holder edge (`poll_held`, + `poll_unheld`): the closure steps on `Ready`, and the handler re-reads and + polls again (registering) when nothing changed. Not its wrappers, which + map a closed channel to `Ready` on every pass. +- Fold the holder-edge hardening in, since this rewrites the same closure: a + closed broadcast ends the front (or skips the holder edge) instead of + polling it again, so safety is local rather than resting on who owns the + producer. +- Distinct from kio's level-only demand, which + [Front deadline index](/quest/m1/front-deadline-index.md) scopes out: there + a reader that comes and goes between polls skips restarting the linger, but + no wake is lost. + +Pitfalls: + +- A track's `Ready(Err(Closed))` is an edge only while the track is recorded + used. Closing flips `is_used()` to false (an upstream refusing an active + track ends it this way), so it owes one final `Unused`, or it stays used + and its front never retires. After that it is no edge, or an ended unused + track spins. +- `broadcast::Demand` has no handler to poll again: publishers await + `poll_demand` directly, so the fix lives inside the poll. Prefer deciding + from the per-track poll results over re-reading `is_used()`: every track + `Ready` meets `unused`, any `Ready(Ok)` meets `used`, and otherwise the + tracks in the way hold registrations. Never retry just because a track was + `Ready`: with one track unused and another still read, that spins until + the last viewer leaves. + +Verification: a loom model beside Idle fronts' +`a_request_never_joins_a_retiring_front` in `rs/moq-net/tests/loom.rs`, with +a reader thread taking and dropping a track around the driver's poll, failing +without the fix; the same for `Demand`. Also cover a track that closes while +used (one `Unused`) and one that closes already unused (no spin), and for +`Demand` two tracks: `Pending` while only one is read, done once its last +reader leaves. A front whose broadcast closes while unheld must not spin; +close it from a test, since no production path does yet. The simulated +network can't hit a window inside a single poll. + +Public API: none. Wire: none. + +## Related + +- [Front deadline index](/quest/m1/front-deadline-index.md) - later replaces this closure with per-track wakes, which must keep the fix diff --git a/quest/m0/ffi-publisher-stall.md b/quest/m0/ffi-publisher-stall.md new file mode 100644 index 00000000000..1dd2616da1d --- /dev/null +++ b/quest/m0/ffi-publisher-stall.md @@ -0,0 +1,62 @@ +# [S] FFI publishers stay connected through interop + +## Goal + +Every cell of `just test interop --all` with a Go or Python publisher passes +reliably on `main` and in nightly CI. Today a moq-ffi publisher goes silent, +not even sending QUIC keep-alives, until the relay's 10 s idle timeout closes +its connection and it reconnects. The browser cells (`go -> js`, +`python -> js`) can't absorb that within their 30 s limit and fail at 32 to +33 s; `-> rust` and `-> gst` pass slowly at 10 to 11 s, and in some runs most +cells of one publisher fail at 11 s. The harness fails any cell whose +connection idles out, so a drop and reconnect can no longer pass as a slow +cell. + +## Plan + +The cause is known: a serve loop with work always ready never yields and +starves moq-ffi's single runtime thread until the relay times the publisher +out (found 2026-10-08 landing #4225). [Serve budget](/quest/m0/serve-budget.md) +fixes it, so this quest starts once it lands and no bisect is needed. + +Decided 2026-10-08: + +- In m0, moved from m1 and widened from the browser cells: the stall masks + interop on every wire PR and may disconnect real FFI publishers. +- Re-check every Go and Python publisher cell once the serve budget lands. + The #4225 root-cause comment saw a 64-iteration yield still time the + publisher out, so if a cell still stalls, find the rest of the cause. + Never raise a timeout or add a retry. +- The harness fails a cell when any connection in it idles out, so this + class of regression fails every run instead of passing slowly. +- The harness shuts down every client it starts cleanly, so every idle-out + counts. Today `gst-launch` dies on SIGPIPE and `moq-cli` can be killed + under `timeout -k`, which may leave a QUIC connection to idle out. No + exception list for killed clients: it would hide a real idle-out behind a + harness kill. + +Harness facts (2026-10-08): + +- The one-byte subscribers exit on their first byte, so they pass even when + the publisher stalls after it; `-> rust` and `-> gst` exit only on their + next write after `head` closes, and the browser cells wait for rendered + video, then audio, pause, and resume. Judge a fix by those. +- The js cell (`test/interop/interop.sh` js subscriber, `driver.ts`) runs + each step within the cell's timeout, and `harness.ts` `waitForWatch` has + its own hard-coded 30 s. +- One relay serves the whole run, and its `connection closed err=... timed + out` warning (`rs/moq-relay/src/relay.rs`) names no connection and lands + about 10 s after the peer went quiet, often in a later cell. Tie each + idle-out to its connection first, by logging the connection there or + having the client report its own drop. + +Public API: none expected. Wire: none. + +## Required + +- [Serve budget](/quest/m0/serve-budget.md) - fixes the stall's cause; this quest verifies the cells and hardens the harness + +## Related + +- [FFI runtime](/quest/m1/ffi-runtime.md) - FFI apps run on more than one thread, which also hides the stall +- [Audio group duration](/quest/m1/audio-group-duration.md) - the go client's 2.5 ms Opus frames stop minting a group each diff --git a/quest/m0/serve-budget.md b/quest/m0/serve-budget.md index 97365a7c1b3..f76ec2bfd3f 100644 --- a/quest/m0/serve-budget.md +++ b/quest/m0/serve-budget.md @@ -94,5 +94,5 @@ Public API: `kio::coop::budget` (new). Wire: none. - [FFI runtime](/quest/m1/ffi-runtime.md) - FFI apps run on more than one thread - [Audio group duration](/quest/m1/audio-group-duration.md) - small audio frames stop minting a group each -- [Browser interop cells](/quest/m1/interop-browser-timeouts.md) - the `go -> js` and `python -> js` timeouts, likely the same stall +- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - verifies the Go and Python interop cells once this lands, and fails a cell whose connection idles out - [Run to quiescence](/quest/m1/perf/uring-quiescence.md) - its pass count bounds passes per turn; this budget bounds one task's loop, so they compose diff --git a/quest/m1/README.md b/quest/m1/README.md index 3ace48e2444..b199a99e691 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -67,7 +67,6 @@ were done or not worth their cost. - [moq-uring tests under load](/quest/m1/uring-tests-under-load.md) - uring tests pass while parallel checks share locked memory - [FFI runtime](/quest/m1/ffi-runtime.md) - moq-ffi and moq-c drive moq on a multi-thread runtime instead of one thread - [Audio group duration](/quest/m1/audio-group-duration.md) - audio groups span at least 20 ms by default, so small frames don't mint a group each -- [Browser interop cells](/quest/m1/interop-browser-timeouts.md) - `go -> js` and `python -> js` pass, fixed at the cause rather than by a longer timeout - [Wire compatibility](/quest/m1/wire-compat.md) - a nightly run tests this checkout against the last published release for tokens, session wire, and catalog/container - [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev` - [Homebrew moq-gst](/quest/m1/gst-brew-path.md) - `brew install moq-gst` installs the plugin from the tarball's `lib/gstreamer-1.0/` diff --git a/quest/m1/interop-browser-timeouts.md b/quest/m1/interop-browser-timeouts.md deleted file mode 100644 index 5b7c4aa26d6..00000000000 --- a/quest/m1/interop-browser-timeouts.md +++ /dev/null @@ -1,32 +0,0 @@ -# [M] Browser interop cells pass with FFI publishers - -## Goal - -The `go -> js` and `python -> js` cells of `just test interop --all` pass -reliably on `main` and in nightly CI, with the cause found and fixed. They -time out today at 32 to 33 s against a 30 s limit, on `main` too (run -37511624075), and on the auth line the browser's tone subscription is reset -with code 0. - -## Plan - -Decided 2026-10-07: investigate now rather than wait for nightly results -after #5005 (expiry-wakes, since landed), which may have been the cause: an -FFI publisher's 2 s audio max age parked hundreds of group serves. - -- Reproduce locally with `just test interop --all`, under load and alone. - The js cell (`test/interop/interop.sh` js subscriber, `driver.ts`) waits for - rendered video, then audio, pause, and resume, each within the timeout; - `harness.ts` `waitForWatch` has its own hard-coded 30 s. -- Find which step stalls and why, from the player trace and the publisher - side. Check whether #5005 alone fixed it before changing anything else. -- Never raise the timeout or add a retry. -- Found 2026-10-08 while landing #4225: an FFI publisher's serve loop can - starve moq-ffi's single runtime thread until the relay times it out. - Re-check these cells once [Serve budget](/quest/m0/serve-budget.md) lands. - -Public API: none expected. Wire: none. - -## Related - -- [Serve budget](/quest/m0/serve-budget.md) - likely the same stall