Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ view (`origin::Consumer::local()`) to localhost workers belongs to moq.pro's
edge, which embeds moq-relay; it moved there on 2026-09-28.
Pools of claim workers (transcoders) also need the relay to forget a front
nobody reads (found 2026-10-07).
A demand poll that loses a reader's wake keeps an unread front and its
upstream subscription alive (found 2026-10-08).

Interop: Fastly's moq-relay-interop report (run of 2026-09-23, build
7ee2b02) was triaged against `main` on 2026-10-07. Its SETUP, UNSUBSCRIBE
Expand All @@ -47,7 +49,10 @@ publishers mint epochs. Backport patches cut from `release` don't carry
Liveness: a serve loop with work always ready never yields, which starved
an FFI publisher's QUIC driver and fails hosted Interop's go lanes (found
2026-10-08 landing #4225). The [serve budget](/quest/m0/serve-budget.md)
bounds every kio task's loop.
bounds every kio task's loop. The Go and Python interop cells it fails
moved here from m1 the same day, because the stall masks interop on every
wire PR and hides as slow passes; the harness now fails a cell whose
connection idles out.

Audio playout: the jitter target is default in both languages (#4162); the
[line](/quest/m1/audio-jitter-target/README.md) moved to m1 in the
Expand All @@ -61,6 +66,8 @@ remain and no release waits on them.
- [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.4, before Seattle
- [Capped stream END_OF_GROUP](/quest/m0/ietf-end-of-track-location.md) - a stream capped by the subscription's end Location never claims END_OF_GROUP; moving End of Track's Location is deferred
- [End of Group status](/quest/m0/ietf-end-of-group-status.md) - an End of Group status on a stream whose header already marks the group's end is accepted, so imquic's last object per group arrives
- [Demand lost wake](/quest/m0/demand-lost-wake.md) - a reader that comes and goes between a demand poll and its re-read never leaves a front or `broadcast::Demand` without a wake
- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - every Go and Python publisher cell passes reliably once the serve budget lands, and a cell fails when a connection idles out
- [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to
- [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines
- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription
Expand Down
86 changes: 86 additions & 0 deletions quest/m0/demand-lost-wake.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# [S] Demand polls never lose a wake

## Goal

A reader that arrives or leaves between a demand poll and its re-read never
leaves the waiter without a wake. Today a relay front's driver can miss a
track's `Unused` edge: the copy keeps its upstream subscription for nobody,
and the front never retires, with nothing to bound it on a quiet
single-track front. In the other direction a reader stalls on a parked track
until the linger expires. `broadcast::Demand::poll_demand` has the same shape
and is fixed with it, and a closed broadcast never spins the front's holder
edge.

## Plan

Facts from `main` at 7c6b6afc1 (line numbers at time of writing):

- `serve_front`'s wait closure (`rs/moq-net/src/model/origin.rs:2728-2735`)
calls `io.weak.poll_unused` or `poll_used`, then steps only if
`io.weak.is_used() != io.used`. `TrackWeak::poll_used` and `poll_unused`
(`track.rs:2550-2558`) discard the result, and kio
(`rs/kio/src/weak.rs:168-188`) returns `Ready` without registering when
the edge is already met. With `used` true: no consumers, so `Ready` without
a registration; a reader arrives, so `is_used()` is true and the closure
returns `Pending`; the reader leaves and nothing wakes the driver.
- `Step::Demand` (`origin.rs:2841-2857`) re-reads `is_used()` and can emit a
duplicate `Used` or `Unused`.
- `broadcast::Demand::poll_demand` (`broadcast.rs:136-147`, `887-904`):
`register_demand` drops each track's `Ready`, then recomputes `is_used()`.
- On #5054's head 3b6d36f67, `poll_held` and `poll_unheld`
(`broadcast.rs:386-393`) map kio's `Ready(Err(Closed))`, returned before
registering, to `Ready`, and `Step::Holders` `continue`s while
`is_held() == held`. A closed broadcast would livelock the driver in
`kio::wait`, which has no budget. Not reachable today: the only closers of
the front's broadcast, `Action::End` and `Retire` through `End`, end the
driver in the same pass, and `FrontTask` owns the producer.

Decided 2026-10-08:

- m0: it undermines the idle-front retire (#5054) and leaks an upstream
subscription.
- Covers `broadcast::Demand` too: same pattern, and publishers rely on its
edges being clean.
- Reuse the handler shape Idle fronts uses for the holder edge (`poll_held`,
`poll_unheld`): the closure steps on `Ready`, and the handler re-reads and
polls again (registering) when nothing changed. Not its wrappers, which
map a closed channel to `Ready` on every pass.
- Fold the holder-edge hardening in, since this rewrites the same closure: a
closed broadcast ends the front (or skips the holder edge) instead of
polling it again, so safety is local rather than resting on who owns the
producer.
- Distinct from kio's level-only demand, which
[Front deadline index](/quest/m1/front-deadline-index.md) scopes out: there
a reader that comes and goes between polls skips restarting the linger, but
no wake is lost.

Pitfalls:

- A track's `Ready(Err(Closed))` is an edge only while the track is recorded
used. Closing flips `is_used()` to false (an upstream refusing an active
track ends it this way), so it owes one final `Unused`, or it stays used
and its front never retires. After that it is no edge, or an ended unused
track spins.
- `broadcast::Demand` has no handler to poll again: publishers await
`poll_demand` directly, so the fix lives inside the poll. Prefer deciding
from the per-track poll results over re-reading `is_used()`: every track
`Ready` meets `unused`, any `Ready(Ok)` meets `used`, and otherwise the
tracks in the way hold registrations. Never retry just because a track was
`Ready`: with one track unused and another still read, that spins until
the last viewer leaves.

Verification: a loom model beside Idle fronts'
`a_request_never_joins_a_retiring_front` in `rs/moq-net/tests/loom.rs`, with
a reader thread taking and dropping a track around the driver's poll, failing
without the fix; the same for `Demand`. Also cover a track that closes while
used (one `Unused`) and one that closes already unused (no spin), and for
`Demand` two tracks: `Pending` while only one is read, done once its last
reader leaves. A front whose broadcast closes while unheld must not spin;
close it from a test, since no production path does yet. The simulated
network can't hit a window inside a single poll.

Public API: none. Wire: none.

## Related

- [Front deadline index](/quest/m1/front-deadline-index.md) - later replaces this closure with per-track wakes, which must keep the fix
62 changes: 62 additions & 0 deletions quest/m0/ffi-publisher-stall.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# [S] FFI publishers stay connected through interop

## Goal

Every cell of `just test interop --all` with a Go or Python publisher passes
reliably on `main` and in nightly CI. Today a moq-ffi publisher goes silent,
not even sending QUIC keep-alives, until the relay's 10 s idle timeout closes
its connection and it reconnects. The browser cells (`go -> js`,
`python -> js`) can't absorb that within their 30 s limit and fail at 32 to
33 s; `-> rust` and `-> gst` pass slowly at 10 to 11 s, and in some runs most
cells of one publisher fail at 11 s. The harness fails any cell whose
connection idles out, so a drop and reconnect can no longer pass as a slow
cell.

## Plan

The cause is known: a serve loop with work always ready never yields and
starves moq-ffi's single runtime thread until the relay times the publisher
out (found 2026-10-08 landing #4225). [Serve budget](/quest/m0/serve-budget.md)
fixes it, so this quest starts once it lands and no bisect is needed.

Decided 2026-10-08:

- In m0, moved from m1 and widened from the browser cells: the stall masks
interop on every wire PR and may disconnect real FFI publishers.
- Re-check every Go and Python publisher cell once the serve budget lands.
The #4225 root-cause comment saw a 64-iteration yield still time the
publisher out, so if a cell still stalls, find the rest of the cause.
Never raise a timeout or add a retry.
- The harness fails a cell when any connection in it idles out, so this
class of regression fails every run instead of passing slowly.
- The harness shuts down every client it starts cleanly, so every idle-out
counts. Today `gst-launch` dies on SIGPIPE and `moq-cli` can be killed
under `timeout -k`, which may leave a QUIC connection to idle out. No
exception list for killed clients: it would hide a real idle-out behind a
harness kill.

Harness facts (2026-10-08):

- The one-byte subscribers exit on their first byte, so they pass even when
the publisher stalls after it; `-> rust` and `-> gst` exit only on their
next write after `head` closes, and the browser cells wait for rendered
video, then audio, pause, and resume. Judge a fix by those.
- The js cell (`test/interop/interop.sh` js subscriber, `driver.ts`) runs
each step within the cell's timeout, and `harness.ts` `waitForWatch` has
its own hard-coded 30 s.
- One relay serves the whole run, and its `connection closed err=... timed
out` warning (`rs/moq-relay/src/relay.rs`) names no connection and lands
about 10 s after the peer went quiet, often in a later cell. Tie each
idle-out to its connection first, by logging the connection there or
having the client report its own drop.

Public API: none expected. Wire: none.

## Required

- [Serve budget](/quest/m0/serve-budget.md) - fixes the stall's cause; this quest verifies the cells and hardens the harness

## Related

- [FFI runtime](/quest/m1/ffi-runtime.md) - FFI apps run on more than one thread, which also hides the stall
- [Audio group duration](/quest/m1/audio-group-duration.md) - the go client's 2.5 ms Opus frames stop minting a group each
2 changes: 1 addition & 1 deletion quest/m0/serve-budget.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,5 +94,5 @@ Public API: `kio::coop::budget` (new). Wire: none.

- [FFI runtime](/quest/m1/ffi-runtime.md) - FFI apps run on more than one thread
- [Audio group duration](/quest/m1/audio-group-duration.md) - small audio frames stop minting a group each
- [Browser interop cells](/quest/m1/interop-browser-timeouts.md) - the `go -> js` and `python -> js` timeouts, likely the same stall
- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - verifies the Go and Python interop cells once this lands, and fails a cell whose connection idles out
- [Run to quiescence](/quest/m1/perf/uring-quiescence.md) - its pass count bounds passes per turn; this budget bounds one task's loop, so they compose
1 change: 0 additions & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,6 @@ were done or not worth their cost.
- [moq-uring tests under load](/quest/m1/uring-tests-under-load.md) - uring tests pass while parallel checks share locked memory
- [FFI runtime](/quest/m1/ffi-runtime.md) - moq-ffi and moq-c drive moq on a multi-thread runtime instead of one thread
- [Audio group duration](/quest/m1/audio-group-duration.md) - audio groups span at least 20 ms by default, so small frames don't mint a group each
- [Browser interop cells](/quest/m1/interop-browser-timeouts.md) - `go -> js` and `python -> js` pass, fixed at the cause rather than by a longer timeout
- [Wire compatibility](/quest/m1/wire-compat.md) - a nightly run tests this checkout against the last published release for tokens, session wire, and catalog/container
- [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev`
- [Homebrew moq-gst](/quest/m1/gst-brew-path.md) - `brew install moq-gst` installs the plugin from the tarball's `lib/gstreamer-1.0/`
Expand Down
32 changes: 0 additions & 32 deletions quest/m1/interop-browser-timeouts.md

This file was deleted.

Loading