From 1cbf61c8b7a356c31408e052b4cd9e3c984110ab Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 11:01:18 -0700 Subject: [PATCH 1/4] quest(m1): bound what a stalled subscriber makes a publisher hold Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 + quest/m1/egress-held-state.md | 110 ++++++++++++++++++++++++++++++++++ 2 files changed, 111 insertions(+) create mode 100644 quest/m1/egress-held-state.md diff --git a/quest/m1/README.md b/quest/m1/README.md index a0085108875..96f9899174b 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -102,6 +102,7 @@ lands before the cut. - [Decode errors end the reader](/quest/m1/decode-errors.md) - a container decode error fails the reader in Rust and JS instead of reading as loss - [Transcode finishes its group](/quest/m1/transcode-open-group.md) - `moq-transcode` never aborts a group a fetch's live loop opened - [Rust request credit](/quest/m1/rs-request-credit.md) - a moq-net request fails at once without stream credit instead of waiting +- [A stalled subscriber holds bounded state](/quest/m1/egress-held-state.md) - group machines and bytes a credit-withholding downstream makes a publisher hold plateau at what its credit and the latency budget allow, on lite and moq-transport - [qmux no-wait opens](/quest/m1/qmux-no-wait.md) - @moq/qmux rejects an over-limit create when waitUntilAvailable is false, like Chrome - [Bufferbloat harness](/quest/m1/bufferbloat.md) - acceptance and performance checks impair paths with a rate limit and a deep queue, not random loss - [Random-loss audit](/quest/m1/random-loss-audit.md) - every decision random-loss gates drove is kept, reworked, or reverted on shaped paths diff --git a/quest/m1/egress-held-state.md b/quest/m1/egress-held-state.md new file mode 100644 index 00000000000..fd363e2b83c --- /dev/null +++ b/quest/m1/egress-held-state.md @@ -0,0 +1,110 @@ +# [L] A stalled subscriber holds bounded publisher state + +## Goal + +What a downstream peer can make a publisher hold is bounded by the protocol's +own shape, not by how long it stalls. A subscriber that withholds stream +credit, or drains slower than the media, across as many tracks as its session +limits admit, makes a relay (or any moq-net publisher) hold group state and +bytes that plateau at what the peer's credit and the latency budget allow, +on both moq-lite and moq-transport. A healthy viewer behind a congested link +still gets the freshest groups. + +Non-goals: capping sessions or connections (see Plan), ingress memory +([Bound incomplete messages](/quest/m1/quic/receive-memory.md) owns it), and +new limit constants that only refuse input. + +## Plan + +Planned 2026-10-19 after the egress audit in #5958, under the rule that +network input is untrusted and its cost is bounded by design (#5961). Facts +as of 83805955a; re-verify, they move. + +Where it stands: + +- Each group a subscription reaches becomes its own serve machine at once, + and waits in `Open` for stream credit with no in-flight bound: lite + `TrackRun::serve` (`rs/moq-net/src/lite/publisher.rs`, around 3722), IETF + `TrackServe` ("one machine per group, with unlimited concurrency", + `rs/moq-net/src/ietf/publisher.rs`, around 4002). +- A parked machine leaves only when its group expires. An aborted group never + does: `expiry_pending` is false once aborted (`model/group.rs`, around 1924 + and 2174), and the `Open` arms check only expiry. The pool's idle window + (30 s by default) evicts the unread group's bytes, so the machine outlives + its content and parks forever. When credit finally arrives, it may be spent + on a group that can no longer be served. +- The serving budget is effectively unbounded. A peer whose wire carries no + max delay (lite-01/02, every moq-transport draft) is served at about 2^53 ms + (`MAX_SAFE_DELAY_MS` in both publishers), and a lite-03+ peer names any + value it likes. Locally only the origin's `cache_duration` (`Duration::MAX` + unless the relay sets `--cache-duration`) or the publisher's `max_age` + clamps it. +- Session limits cap subscriptions and fetches (`rs/moq-relay/src/limits.rs`: + 1000 subscriptions and 10000 fetches per client session), never groups in + flight per subscription. + +Abuse scenarios to cover: + +1. A subscriber at its subscription cap never grants uni-stream credit. Held + machines grow with every group published, forever, even after the pool + frees their bytes. +2. The same peer with an inert pool (no capacity, no idle window, as a library + user may configure): held bytes grow too. +3. A peer grants stream credit but withholds byte credit, so opened streams + pin their groups past the point they are useful. +4. A lite-03+ peer asks for an enormous max delay so nothing ever expires. +5. The benign twin: a real viewer on a bufferbloated link. The fix must not + cost it the live edge. + +Recommended directions (the implementer settles the shape): + +- Make a group's serve state follow credit rather than arrival. One option is + to open a stream only when the session can grant one and then pick the best + group still worth sending at that moment, so a parked backlog lives in the + cache (already billed and evicted) rather than in per-subscriber machines. + Whatever the shape, cross-subscription priority and per-broadcast fairness + ([Scope track priority](/quest/m1/track-priority-scope.md)) must still + decide who gets the next stream. +- A parked group that can no longer be served (aborted, evicted, expired) + leaves at once and is named to the subscriber where the version can, through + [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md). +- The serving budget never exceeds what this hop would retain anyway: clamp + the requested or defaulted max delay by the local retention (the + `cache_duration` ceiling, the track's `max_age`, and possibly the pool's idle + window). Consider whether the relay's default `cache_duration` should follow + the pool's window instead of `Duration::MAX`; that changes a user-facing + default, so update `doc/bin/relay/` with it. +- Prefer bounds the protocol already carries (stream credit, max delay, + `max_age`, the cache pool) over a new per-subscription constant. If a number + is unavoidable, it must come from one of those. + +Sessions: capping them is out of scope. A session costs a handshake, auth, and +the per-session limits above, and a count cap would be a constant that only +throws. Admission control (per-token or per-address rate, or a cap derived +from memory) is a separate concern; plan it as its own quest if a drill shows +sessions are the cheapest remaining abuse. + +Reproduce first. A mock-time sim test (or a `just` drill against a real relay) +with a subscriber that withholds stream credit, or drains below the media rate +behind a deep finite queue, while the publisher writes groups across many +tracks. Measure held serve machines (the publishers already count pending +opens) and pool-billed bytes over time: on main they grow with time; done, +they plateau. Sweep tracks by groups per track, per the fan-out benchmark +rule, so a cost that grows with the table shows up as a slope. Cover lite and +moq-transport, and scenario 5 still receives the latest group. + +Done looks like: those measurements plateau on both protocols and every +scenario above, the regression tests fail on main and run in CI, the JS +publisher in `js/net` is checked for the same shape and fixed or its gap +named in the PR, and the PR reports public API and wire impact (expected: +none on the wire; possibly a relay default). + +## Related + +- [Bound incomplete messages](/quest/m1/quic/receive-memory.md) - the ingress counterpart, bounded by QUIC connection credit +- [Rust request credit](/quest/m1/rs-request-credit.md) - requests fail fast without stream credit; publisher group streams are left to this quest +- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - how a publisher names a group it gave up on +- [Scope track priority](/quest/m1/track-priority-scope.md) - who gets the next stream across subscriptions and tenants +- [Public serving](/quest/m1/public-model/serve.md) - makes group expiry while serving public; coordinate the publisher serve loops with it +- [Treat network input as untrusted](https://github.com/moq-dev/moq/pull/5961) - the rule this quest applies +- [Resumed group head](https://github.com/moq-dev/moq/pull/5958) - the audit that found this, and bounded held heads by the cache pool From 68c1fb8aff4fdd588121846151e4f79c24bf3e61 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 11:18:45 -0700 Subject: [PATCH 2/4] quest(m1): admission stays in the auth server; test a benign withholder Co-Authored-By: Claude Opus 5.5 --- quest/m1/egress-held-state.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/quest/m1/egress-held-state.md b/quest/m1/egress-held-state.md index fd363e2b83c..a7703706ed5 100644 --- a/quest/m1/egress-held-state.md +++ b/quest/m1/egress-held-state.md @@ -80,9 +80,9 @@ Recommended directions (the implementer settles the shape): Sessions: capping them is out of scope. A session costs a handshake, auth, and the per-session limits above, and a count cap would be a constant that only -throws. Admission control (per-token or per-address rate, or a cap derived -from memory) is a separate concern; plan it as its own quest if a drill shows -sessions are the cheapest remaining abuse. +throws. Admission control (per-token or per-address session caps) belongs to +the deployment's auth server, which already refuses a connect with a 401 or +403 ([relay auth](/doc/bin/relay/auth.md)); this repo needs nothing more. Reproduce first. A mock-time sim test (or a `just` drill against a real relay) with a subscriber that withholds stream credit, or drains below the media rate @@ -92,6 +92,10 @@ opens) and pool-billed bytes over time: on main they grow with time; done, they plateau. Sweep tracks by groups per track, per the fan-out benchmark rule, so a cost that grows with the table shows up as a slope. Cover lite and moq-transport, and scenario 5 still receives the latest group. +Include a benign withholder: a peer bounded by +[Bound incomplete messages](/quest/m1/quic/receive-memory.md) withholds credit +by design under memory pressure, so two relays running both quests must still +plateau and keep the live edge. Done looks like: those measurements plateau on both protocols and every scenario above, the regression tests fail on main and run in CI, the JS @@ -101,7 +105,7 @@ none on the wire; possibly a relay default). ## Related -- [Bound incomplete messages](/quest/m1/quic/receive-memory.md) - the ingress counterpart, bounded by QUIC connection credit +- [Bound incomplete messages](/quest/m1/quic/receive-memory.md) - the ingress counterpart: a receiver withholds QUIC credit to bound what it holds, and this quest bounds what the sender holds while it does - [Rust request credit](/quest/m1/rs-request-credit.md) - requests fail fast without stream credit; publisher group streams are left to this quest - [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - how a publisher names a group it gave up on - [Scope track priority](/quest/m1/track-priority-scope.md) - who gets the next stream across subscriptions and tenants From 87f98ec4c61d4219423f06144c81e68785b6cd89 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 11:27:20 -0700 Subject: [PATCH 3/4] quest(m1): measure live egress state against a no-subscriber baseline Co-Authored-By: Claude Opus 5.5 --- quest/m1/egress-held-state.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/quest/m1/egress-held-state.md b/quest/m1/egress-held-state.md index a7703706ed5..828d17b6f9c 100644 --- a/quest/m1/egress-held-state.md +++ b/quest/m1/egress-held-state.md @@ -49,7 +49,8 @@ Abuse scenarios to cover: machines grow with every group published, forever, even after the pool frees their bytes. 2. The same peer with an inert pool (no capacity, no idle window, as a library - user may configure): held bytes grow too. + user may configure). The pool keeps every group by policy, so judge only + what the subscriber adds: held state beyond a no-subscriber baseline. 3. A peer grants stream credit but withholds byte credit, so opened streams pin their groups past the point they are useful. 4. A lite-03+ peer asks for an enormous max delay so nothing ever expires. @@ -87,9 +88,12 @@ the deployment's auth server, which already refuses a connect with a 401 or Reproduce first. A mock-time sim test (or a `just` drill against a real relay) with a subscriber that withholds stream credit, or drains below the media rate behind a deep finite queue, while the publisher writes groups across many -tracks. Measure held serve machines (the publishers already count pending -opens) and pool-billed bytes over time: on main they grow with time; done, -they plateau. Sweep tracks by groups per track, per the fan-out benchmark +tracks. Measure, over time, live serve machines (pending and opened, as a +gauge) and the bytes they retain outside the pool (batches, chunks, and +unsent transport data), beside pool-billed bytes compared with a +no-subscriber baseline. Only lite counts pending opens today; IETF +`TrackServe` counts streams only after they open, so add the gauge there. On +main they grow with time; done, they plateau. Sweep tracks by groups per track, per the fan-out benchmark rule, so a cost that grows with the table shows up as a slope. Cover lite and moq-transport, and scenario 5 still receives the latest group. Include a benign withholder: a peer bounded by From 211cdeea08664d521093d5399ca23bafdf2ccdfb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 19 Oct 2026 12:53:13 -0700 Subject: [PATCH 4/4] quest(m1): sweep subscribers by tracks in the egress benchmark Co-Authored-By: Claude Opus 5.5 --- quest/m1/egress-held-state.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/quest/m1/egress-held-state.md b/quest/m1/egress-held-state.md index 828d17b6f9c..d77f3b60928 100644 --- a/quest/m1/egress-held-state.md +++ b/quest/m1/egress-held-state.md @@ -93,8 +93,10 @@ gauge) and the bytes they retain outside the pool (batches, chunks, and unsent transport data), beside pool-billed bytes compared with a no-subscriber baseline. Only lite counts pending opens today; IETF `TrackServe` counts streams only after they open, so add the gauge there. On -main they grow with time; done, they plateau. Sweep tracks by groups per track, per the fan-out benchmark -rule, so a cost that grows with the table shows up as a slope. Cover lite and +main they grow with time; done, they plateau. Per the fan-out benchmark rule, +sweep tracks by groups per track, and subscribers by tracks, timing each +stream grant too, so a cost that grows with the table (such as scanning every +subscription when one stream frees up) shows up as a slope. Cover lite and moq-transport, and scenario 5 still receives the latest group. Include a benign withholder: a peer bounded by [Bound incomplete messages](/quest/m1/quic/receive-memory.md) withholds credit