Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
f69b066
chore(quest): claim quest/m1/unbounded-groups/fastest-reader.md
kixelated Nov 2, 2026
5c264b9
Merge remote-tracking branch 'origin/main' into quest/m1/unbounded-gr…
kixelated Nov 2, 2026
e3f71fb
wip: sized readiness, one-segment floor, smaller windows, qmux native…
kixelated Nov 2, 2026
3c64765
wip: qmux sessions take the [quic] windows and widen for peers; JS se…
kixelated Nov 2, 2026
fe9f81b
wip: absorb the qmux receive-lease quest, record the 2026-11-02 decis…
kixelated Nov 2, 2026
b421af2
wip: format, fix bench and sender types
kixelated Nov 2, 2026
fec637a
fix(qmux): free a stream whose reset a lease sees; accept an unlimite…
kixelated Nov 2, 2026
fb31c89
style(qmux): format the set_limits test
kixelated Nov 2, 2026
9bf847c
test(relay): expect the new default windows
kixelated Nov 2, 2026
cc360bf
Merge remote-tracking branch 'origin/main' into quest/m1/unbounded-gr…
kixelated Nov 2, 2026
7ee8289
Merge remote-tracking branch 'origin/main' into t3/ee31b2ea
kixelated Nov 2, 2026
bf8b840
feat: park TRACK_INFO waits at the lowest receive priority, apply [qu…
kixelated Nov 2, 2026
959c4bd
docs: record the settled #7624 decisions, rewrite the quest to what r…
kixelated Nov 2, 2026
868992a
fix(moq-net): withdraw a parked GROUP stream that fails before TRACK_…
kixelated Nov 2, 2026
968d07b
chore: satisfy clippy and markdown formatting
kixelated Nov 2, 2026
fda2c9b
fix(moq-tokio): fail the web reset wait with ReadError, not a panic, …
kixelated Nov 2, 2026
7b1b262
Merge remote-tracking branch 'origin/main' into t3/ee31b2ea
kixelated Nov 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 31 additions & 23 deletions doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,9 +122,9 @@ idle_timeout = "10s" # Drop a connection after this long with no
# keep_alive = "3s" # Ping interval; unset or "0s" derives it from the idle timeout. iroh keeps a separate 5s path heartbeat.
gso = true # UDP segmentation offload. iroh cannot turn it off.
mtu_discovery = false # Path MTU discovery. Default.
receive_window = 134217728 # Whole-connection window, in bytes. Default (128 MiB).
stream_receive_window = 4194304 # Per-stream window, in bytes. Default (4 MiB).
send_window = 16777216 # Unacknowledged outgoing data, in bytes. Default (16 MiB).
receive_window = 4194304 # Whole-connection window, in bytes. Default (4 MiB).
stream_receive_window = 1048576 # Per-stream window, in bytes. Default (1 MiB).
send_window = 4194304 # Unacknowledged outgoing data, in bytes. Default (4 MiB).
qlog = "/var/log/moq/qlog" # Existing directory. Needs the `qlog` build feature.
# keylog = "/private/keys.log" # Explicit secret log path; unset is off.
```
Expand Down Expand Up @@ -152,17 +152,23 @@ the idle timeout. A value no shorter than the negotiated idle timeout could
never keep the connection up, so it is capped at the derived interval. iroh
cannot see the peer's timeout, so it derives from the local `idle_timeout`.

Raise the receive windows when a fat, long path idles below the link rate. The
128 MiB `receive_window` default carries several Gbps at a 100 ms RTT and bounds
how much unread data and incomplete messages one peer can make the relay
buffer. It has to fit the largest message, just over 64 MiB, so a smaller
value, here or in `[limits.peer.quic]`, is refused at startup. When incomplete
messages fill the window, the relay cancels the least urgent stream: the
lowest subscription priority, then the group its subscription would deliver
last. Over iroh the window bounds only
unread data, and over WebSocket or TCP nothing bounds incomplete messages.
`stream_receive_window` only paces one stream: its credit returns as bytes are
read, whether or not their message is complete.
Raise the receive windows when a fat, long path idles below the link rate: a
connection carries at most its window per round trip. The 4 MiB
`receive_window` default carries about 270 Mbps at a 100 ms RTT, plenty for one
viewer or publisher, and bounds how much unread data and incomplete messages
one peer can make the relay buffer. It has to fit the largest unit a session
assembles, one 1 MiB segment, so a smaller value, here or in
`[limits.peer.quic]`, is refused at startup. When incomplete messages fill the
window, the relay cancels the least urgent stream: the lowest subscription
priority, then the group its subscription would deliver last. A group still
waiting for its track's TRACK\_INFO goes first, so that control message always
gets through. WebSocket and TCP sessions cancel the same way; the relay accepts
them at these windows too and widens peers to `[limits.peer.quic]`, and links
it dials take these settings.
Over iroh the window bounds only unread data. `stream_receive_window` only paces
one stream: its credit returns as bytes are read, whether or not their message
is complete, so the 1 MiB default caps one group stream at about 67 Mbps at a
100 ms RTT.

## \[limits.client] and \[limits.peer]

Expand All @@ -180,17 +186,18 @@ announces = 100000

[limits.peer.quic]
max_streams = 100000 # Concurrent QUIC streams per direction.
receive_window = 671088640 # Connection-wide receive window (640 MiB).
send_window = 671088640 # Unacknowledged outgoing data (640 MiB).
receive_window = 16777216 # Connection-wide receive window (16 MiB).
send_window = 16777216 # Unacknowledged outgoing data (16 MiB).
```

`[limits.peer.quic]` takes only `max_streams`, `receive_window`, and `send_window`;
any other `[quic]` key is refused. They replace the `[quic]` values for a
peer's QUIC connection once its grant is known, before the SETUP reply. Links
the relay dials get them from the handshake on. A link carrying a whole cluster is then
not throttled by `[quic]` limits sized for one viewer, for example
`--limits-peer-quic-max-streams=200000`. Peers over WebSocket, TCP, or iroh keep
their transport's own limits.
peer's connection once its grant is known, before the SETUP reply, over QUIC,
WebSocket, or TCP alike. Links the relay dials get them from the handshake on.
A link carrying a whole cluster is then not throttled by `[quic]` limits sized
for one viewer, for example `--limits-peer-quic-max-streams=200000`. Peers over
iroh keep iroh's own limits. The 16 MiB windows carry about 450 Mbps at a 250 ms
RTT and about 1 Gbps at 100 ms; raise both for a busier link.

These are the defaults. Each subscription and FETCH holds a stream, as does each
PUBLISH a peer's track listing gets, so `[quic]` or `[limits.peer.quic]`
Expand Down Expand Up @@ -397,8 +404,9 @@ flow control slows the upstream hop and the publisher. A slower reader is never
waited on: one that trails the fastest by more than the rest of the 32 MB
reaches a shed frame. A group nobody reads keeps arriving and shedding.
A stopped stream keeps its receive credit, so the relay stops reading at most as
many streams per connection as fit in half of `quic.receive_window`; past that,
groups keep arriving and shedding as if unpaced.
many streams per connection as fit in half of its receive window, a stream
window each; past that, groups keep arriving and shedding as if unpaced. At the
defaults that is two streams per client session and eight per peer link.

## \[stats]

Expand Down
105 changes: 89 additions & 16 deletions quest/m1/unbounded-groups/fastest-reader.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,14 @@ otherwise refreshes or ends with `Error::Evicted`. A byte stream over MoQ then
loses nothing to a reader that keeps up, and each hop keeps its own congestion
control and retransmissions.

The paced hops include qmux sessions (WebSocket, `tcp://`, `tls://`,
`unix://`, and WebRTC): they lease receive credit natively, so incomplete
messages are bounded by the session's receive window and the lowest-priority
holder is evicted when it is stuck, as on native QUIC. The per-session memory
a peer can make us hold is one deliberate, documented number across native
QUIC and qmux. The browser keeps reading unpaced until WebTransport offers a
real receive-credit contract.

## Plan

Decided 2026-10-25 (maintainer, mesh planning interview), reopening the
Expand Down Expand Up @@ -112,28 +120,93 @@ in `rs/moq-net/src/transport/boxed.rs`; moq-quic reports its windows through
moq-tokio and moq-uring, and a transport that reports none never pauses) and
the public readiness future.

Decided 2026-11-02 (maintainer, /quest-grill interview on
[moq-dev/moq#7624](https://github.com/moq-dev/moq/pull/7624)):

- Native only: finish the moq-quic path (tokio, io_uring, and qmux on its
mux). Browser parity waits on a real receive-credit contract for
WebTransport, since `Reader.closed` watches resets but nothing says how much
credit a paused stream holds; guessed window constants would move memory
outside flow control. Rejected: a browser read-ahead cap, and dropping the
browser from scope.
- qmux is handled here, absorbing the m2 qmux receive-lease quest: qmux's
`RecvStream` leases natively on moq-quic's mux instead of through
`poll::Buffered`, evicts by priority, and so closes the 2026-10-27 audit's
NET-3 gap (`Buffered` bounded each stream's incomplete message by the largest
message, and nothing bounded them in aggregate). Rejected: deferring qmux to
that quest, and adding hooks here while keeping leases in m2.
- The lease floor is one 1 MiB segment for every transport and version: the
lite-01/02 `ANNOUNCE_INIT` cap drops from 64 MiB to a segment less its 16
bytes of prefixes, so `moq_net::transport::MIN_RECEIVE_WINDOW` is 1 MiB. A
legacy origin whose initial set exceeds it fails its announce loudly, on
both ends. Rejected: a 4 MiB cap, and keeping 64 MiB with a per-version
floor.
- Window defaults are one per-session number across native QUIC and qmux:
clients get a 4 MiB connection and a 1 MiB stream receive window, and qmux
takes the same client/peer split as native QUIC. The cluster peer windows
(`[limits.peer.quic]`, 640 MiB) and the client `send_window` (16 MiB) are
re-derived from measured bandwidth-delay needs, with the maintainer
suggesting about 16 MiB for peers. Rejected for clients: 16, 32, or 128 MiB
connection windows, and a 4 MiB stream window.
- Local readiness takes the next frame's `frame::Info`
(`group::Producer::poll_ready(waiter, &info)` and `ready(&info)`), reusing
ingest's admission semantics, including its whole-frame deadlock exception.
Rejected: a bare size on the same methods.
- Whole-batch fills move to [flights](/quest/m1/unbounded-groups/flights.md),
since the failure crosses groups while this mechanism paces one group copy.

The adapter inventory this plan once listed is stale: moq-tokio has no Quinn
backend, and moq-ffi drives sessions through moq-tokio rather than a
transport of its own, so neither needs an adapter. Iroh, which returns credit
as it reads, leaves with its removal
([moq-dev/moq#7345](https://github.com/moq-dev/moq/pull/7345)); until then its
ingest never pauses.

Decided 2026-11-02 (maintainer, settling the open items of
[moq-dev/moq#7624](https://github.com/moq-dev/moq/pull/7624)):

- Peer windows stay at 16 MiB receive and send in `[limits.peer.quic]`, with
the 1 MiB stream window from `[quic]`. Rejected: 32 MiB, and a separate
per-stream peer window.
- The client `send_window` is 4 MiB, matching its receive window. Rejected:
keeping 16 MiB.
- A GROUP stream waiting unread for its TRACK_INFO reserves nothing from the
paused budget: it waits at the lowest receive priority, so a full window
evicts it before control streams, and TRACK_INFO still gets credit. This
replaces the 2026-11-01 wording that the paused cap covers such streams.
Rejected: stopping it once the budget is full.
- Every qmux session a server accepts takes the configured `[quic]` windows,
including the relay's own WebSocket listener. Rejected: leaving them at the
defaults until admission.
- Split: this PR lands the windows, the floor, qmux leases, sized readiness,
TRACK_INFO priority, and the accept paths, and the rest stays here.
Rejected: finishing everything in one PR.

Landed in [moq-dev/moq#7624](https://github.com/moq-dev/moq/pull/7624):
sized local readiness; the one-segment floor; 4 MiB, 1 MiB, and 4 MiB client
receive, stream, and send windows in moq-tokio, moq-uring, and qmux; 16 MiB
cluster peer windows; native qmux leases with eviction, window reporting, and
`poll_reset`; qmux sessions starting at the server's `[quic]` windows on every
accept path and widening to `[limits.peer.quic]` for peers; and a Bun sender
exhausting a qmux receiver over WebSocket (`just test receive-credit`). At the
client defaults a session pauses two group streams (half the window, a stream
window each), and a peer link eight; past that, groups read and shed as
before pacing. The same PR ranks streams waiting on TRACK_INFO (lite GROUP
streams and FETCH answers, and ietf data streams waiting on their alias)
below every other stream, and they watch `poll_reset`, which now reports an
eviction, so an evicted stream is stopped and eviction resumes.

What remains:

- Receive-window reporting for the transports that report none yet (quinn,
iroh, qmux, WebTransport in the browser, and the FFI adapters), whose ingest
never pauses until they do. Each must also answer
`poll::RecvStream::poll_reset`, or a reset goes unseen while paused.
- A sized readiness check for local writers: `group::Producer::ready()`
waits for room but not for the next frame's size, so a local frame larger
than the room left still sheds what the egress has yet to send.
- JS parity.
- A two-relay chain test with memory measured near the window.
- Per-session retained bytes and throughput across many qmux sessions at the
chosen window, in the
[transport x version benches](/quest/m2/bench-transport-sweep.md).
- A quiet-host comparison against the base of `group_read_frames`,
`group_contended`, and `group_relay_frames`, plus `group_paced` (a writer
parked at the window, over 1, 8, and 64 readers), which the first slice
added. The cursor's report is one relaxed load per frame or batch, and a
`fetch_max` only for the fastest reader. On the busy host that landed it,
runs swung by up to 50% either way between repeats.
- Fills that write a whole batch of groups at once evict their own earlier
groups before the reader takes them (the follow-up from
[moq-dev/moq#7339](https://github.com/moq-dev/moq/pull/7339)). Pacing within
a group does not cover this; decide whether it belongs here or with
[flights](/quest/m1/unbounded-groups/flights.md).

qmux transports carry the same rule through their own stream credit
([qmux receive lease](/quest/m2/qmux-receive-lease.md)).
- JS parity, once the browser offers a receive-credit contract. The JS qmux
receiver could pace today but follows the browser.
7 changes: 7 additions & 0 deletions quest/m1/unbounded-groups/flights.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,13 @@ never does.
flight without slowing the fast one; no flight fetches a frame twice; a
track with no handler refreshes.
- Benchmarks: a case swept over readers and how far apart they join.
- Fills that write a whole batch of groups at once evict their own earlier
groups before the reader takes them (the follow-up from
[moq-dev/moq#7339](https://github.com/moq-dev/moq/pull/7339)). Moved here
from [fastest-reader pacing](/quest/m1/unbounded-groups/fastest-reader.md)
on 2026-11-02 (maintainer): the failure crosses groups, while pacing holds
back one group copy, so a flight's fill paces the batch by its riders
instead.

Public API: none. Wire: none, beyond the existing frame-offset fills.

Expand Down
1 change: 0 additions & 1 deletion quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,6 @@ lite-07; bitrate enforcement is not part of this release.
- [A parked read of a resumed group reads its budget without re-syncing](/quest/m2/recover-budget.md) - measure, then drop the per-frame reader sync
- [Transcode rungs scale off the runtime and keep bounded state](/quest/m2/transcode-rungs.md) - resize on encoder threads, memoize rungs by shape, one writer per rung track
- [One port](/quest/m2/one-port/README.md) - one UDP port carries QUIC, SRT, and WebRTC media through an embedder hook, and one TCP port carries HTTP, RTMP, and RTMPS
- [qmux receive leases](/quest/m2/qmux-receive-lease.md) - WebSocket, TCP, and WebRTC sessions bound incomplete messages by their receive window, at a per-session cost the maintainer picks
- [WebSocket caps](/quest/m2/rate-websocket.md) - paced reads and writes over bounded socket buffers hold WebSocket to the same caps
- [Publishers learn their cap](/quest/m2/rate-grant.md) - the AUTH grant carries the caps and publishers clamp their encoder to them
- [moq-rtc pacing](/quest/m2/rtc-pacing.md) - a capped WHEP viewer never receives more than its cap plus burst, NACK resends included
Expand Down
73 changes: 0 additions & 73 deletions quest/m2/qmux-receive-lease.md

This file was deleted.

Loading
Loading