diff --git a/quest/m2/README.md b/quest/m2/README.md index b865ee7f5e8..4e1e3659a9c 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -151,9 +151,8 @@ lite-07; bitrate enforcement is not part of this release. - [poll_acked on moq-net's send stream](/quest/m2/quic-ack-hook.md) - the backend-neutral hook on moq-net's own transport trait that awaits an acknowledged stream offset, implemented by the moq-tokio and moq-uring adapters -- [Starvation at frame granularity](/quest/m2/starvation-frames.md) - the - acknowledged frontier moves at every frame boundary through `poll_acked`, - with a delivery-delay histogram for jitter +- [Delivery delay](/quest/m2/delivery-delay.md) - a per-frame + delivery-delay histogram from `poll_acked`, so viewer jitter is visible - [#3199](/quest/m2/3199-moq-uring-remove-sq-indirection-and-per-enter-ring-fd.md) - moq-uring: remove SQ indirection and per-enter ring fd lookup, if a profile shows the win - [#3200](/quest/m2/3200-moq-uring-batch-completion-wakeups-with-min-timeout.md) - moq-uring: batch completion wakeups with MIN_TIMEOUT - [Browser media report](/quest/m2/rs2ts-browser-report.md) - compare Rust-backed and handwritten @moq/net through full media workloads, supplementing the pre-adoption comparisons diff --git a/quest/m2/delivery-delay.md b/quest/m2/delivery-delay.md new file mode 100644 index 00000000000..c7323c60595 --- /dev/null +++ b/quest/m2/delivery-delay.md @@ -0,0 +1,37 @@ +# [S] Per-frame delivery delay histogram + +## Goal + +The relay's `moq-stats` egress side reports, per broadcast, a byte-weighted +cumulative histogram of per-frame delivery delay (from when a frame is +written to when the peer acknowledges it), so viewer jitter is visible beside +the `lag` histogram. Subscriptions on a backend that cannot see +acknowledgements leave it absent, never zero. + +## Plan + +Decided 2026-10-26 (maintainer, on +[moq-dev/moq#6791](https://github.com/moq-dev/moq/pull/6791)): this quest's +frontier half, moving the acknowledged frontier at every frame end and +counting `dropped` per frame, moved into the +[viewer lag histogram](/quest/m2/qos/lag-histogram.md), which no longer +lands at group granularity. That quest already awaits each frame's +acknowledgement through `poll_acked`; this one adds the delay it observes. + +Record `acked_at - written_at` per frame, where `acked_at` is when the serve +task is woken by `poll_acked`. Its low edge approaches the path RTT and its +spread is the jitter. Use the `lag` histogram's type and bucket contract. +Document that the value still includes the return one-way delay and the +peer's delayed-ACK timer, since the hook reports no corrected instant +(decided 2026-10-17 on #5817). + +Tests: delivery delay staying inside one bucket of the true RTT plus the +peer's `max_ack_delay`, and the histogram absent over a backend without the +hook. + +Public API: an additive `Traffic` field. Wire: an additive stats JSON field. + +## Required + +- [Viewer lag histogram](/quest/m2/qos/lag-histogram.md) - the per-frame + acknowledgement tracking and histogram type this reuses diff --git a/quest/m2/qos-lag-dashboard.md b/quest/m2/qos-lag-dashboard.md index 39979adf29e..bbaa331ee1b 100644 --- a/quest/m2/qos-lag-dashboard.md +++ b/quest/m2/qos-lag-dashboard.md @@ -4,7 +4,7 @@ The demo stats dashboard (`demo/web/src/stats.ts`) shows the relay's viewer lag and dropped media: percentiles of the egress `lag` histogram over the -chart window, and `dropped` duration, bytes, and groups as rates, cluster-wide +chart window, and `dropped` bytes, frames, and groups as rates, cluster-wide and per node like the existing counters. ## Plan @@ -15,8 +15,11 @@ and per node like the existing counters. lands them on the egress side of its per-group totals and prefix tracks (decided 2026-10-08). `lag` is a cumulative byte count per bucket keyed by its upper edge (`"50ms"` to `"5s"`, then `"inf"`, empty buckets - omitted); `dropped` is `{ duration, bytes, groups }` with the duration in - fractional milliseconds. `doc/concept/stats.md` documents them. Diff two + omitted); `dropped` has the same four fields as `stale` (`bytes`, `frames`, + `groups`, `datagrams`), written only when non-empty. `lag` is absent where + no viewer's backend reports acknowledgements (qmux, the browser + transport), so show no data there rather than zero lag. + `doc/concept/stats.md` documents them. Diff two samples within one subscription for an interval's distribution, as the dashboard already does to turn cumulative bytes into rates, and sum nodes bucket by bucket. diff --git a/quest/m2/qos/README.md b/quest/m2/qos/README.md index ef385e5d8ac..c9982455d8b 100644 --- a/quest/m2/qos/README.md +++ b/quest/m2/qos/README.md @@ -45,8 +45,10 @@ histogram and dropped counters across. Decided (2026-10-08): the egress histogram is its own quest, [viewer lag histogram](/quest/m2/qos/lag-histogram.md), since every other -child reads it. The line's PR, #4133, closed without landing (2026-10-09); -its code is kept on `wip/4133-lag-histogram`. +child reads it. The line's PR, #4133, closed without landing (2026-10-09). +Its port, #6791, closed unmerged on 2026-10-26 to wait for the ACK hook, +since `lag` and `dropped` are measured per acknowledged frame; its branch is +the histogram quest's starting point. Decided (2026-09-28): the line's moq-stats changes break the published crate. Client stats left the line (2026-09-29) diff --git a/quest/m2/qos/lag-histogram.md b/quest/m2/qos/lag-histogram.md index 596e617f6cb..ac25bbbaba1 100644 --- a/quest/m2/qos/lag-histogram.md +++ b/quest/m2/qos/lag-histogram.md @@ -1,58 +1,104 @@ -# [M] Viewer lag histogram +# [L] Viewer lag histogram ## Goal The relay's `moq-stats` egress side reports, per broadcast, how far behind -its viewers are according to what the network has acknowledged: a -byte-weighted cumulative `lag` histogram and `dropped` counters (duration, -bytes, and groups dropped before they were acknowledged), on the stats totals -and prefix tracks (`/prefix.json`). A spliced subscription -stays exact across a route switch, and a closing subscription records its -last partial interval. +its viewers are according to the frames the network has acknowledged: a +byte-weighted cumulative `lag` histogram and `dropped` counters (the frames +written after the newest acknowledged one when a group stream ends without +delivering them), on the stats totals, a group's totals, and the prefix tracks +(`/prefix.json`). A spliced subscription stays exact across a route +switch, and a closing subscription records its last partial interval. +Subscriptions on a backend that cannot see acknowledgements report neither +field: absent, never zero. ## Plan -Decided 2026-10-08: split out of the QoS line so the dashboard and publisher -timeliness can require a histogram that exists on `main`. This quest lands it -on `main` now that prefix tracks have landed -([#5811](https://github.com/moq-dev/moq/pull/5811)), moved from the -group tracks' per-path `paths` rows it writes today -onto the totals and prefix tracks. - -Decided 2026-10-09: the QoS line -([#4133](https://github.com/moq-dev/moq/pull/4133)) closed without landing, -since every code change it had over `main` is this histogram. Its code is kept -on `wip/4133-lag-histogram` as this quest's starting point: the sampler -(#4298, `rs/moq-net/src/stats.rs`), the splice fixes (#5009), and the final -sample (#4451). Porting it is not mechanical: - -- `main` moved to caller-supplied time (#4437) and removed the frozen test - clock (`model::clock::advance`). The sampler reads wall time in - `Registry::report`, when a `Frontier` opens, on `Delivery` acks, and in the - final sample in `Drop for FrontierInner`, which has no caller to supply a - time. Redesign those to take time from their callers, and give the final - sample an explicit close path. -- Untimed tracks (#4822): record `Production` only for timed frames. -- `max_age` became `max_delay` on subscriptions (#4917). - -Open review findings on #4133 to fix while porting: - -- A timestamp regression hides a stalled viewer - ([3](https://github.com/moq-dev/moq/pull/4133#issuecomment-6053929743)). -- A copy served across a no-route gap stays watched - ([B](https://github.com/moq-dev/moq/pull/4133#issuecomment-6053929743)). -- A partial write is charged the frame's full declared size - ([P2](https://github.com/moq-dev/moq/pull/4133#pullrequestreview-5452586850)). -- `Dropped::add` should saturate instead of overflowing - ([comment](https://github.com/moq-dev/moq/pull/4133#discussion_r4215691174)). -- Bound the frontier list when `Registry::report` is not called - ([comment](https://github.com/moq-dev/moq/pull/4133#discussion_r4215691184)). +Starting point: the branch `quest/m2/qos/lag-histogram` on +[t0ms's fork](https://github.com/t0ms/moq/tree/quest/m2/qos/lag-histogram), +head `9ed8b831f93f39565333a92d5dd4a5f24d2c3fc5`, from +[moq-dev/moq#6791](https://github.com/moq-dev/moq/pull/6791) (closed +unmerged 2026-10-26). It ports the #4133 sampler onto `main` with +caller-supplied time (`Registry::report(&self, report, now)`), an explicit +close path, route-copy feeds, untimed tracks recording nothing, the five open +#4133 findings fixed, docs, and tests. Its frontier moves only on an +acknowledged group FIN, which this quest replaces. The PR body has the +details of what it settled; bring it up to date with `main` first. + +Decided 2026-10-26 (maintainer, on #6791), each with its reason: + +- **Frame granularity, through the ACK hook.** Group granularity is rejected + as a definition: a healthy zero-RTT viewer on 2 s groups reads 0.5 s to 5 s + behind, and every unsubscribe, rendition switch, or session close charges + the open group's whole content to `dropped`, so a healthy audience with + churn reads a dropped baseline. The frontier instead moves at every + acknowledged frame end through [poll_acked](/quest/m2/quic-ack-hook.md), so + the sampler reads a frontier at most one frame stale. `dropped` counts only + the frames written after the newest acknowledged one. This folds in the + frontier half of the former starvation-frames quest; the delivery-delay + histogram stays in [its own quest](/quest/m2/delivery-delay.md). Its sketch, + to adjust as the code dictates: the group stream's writer counts the bytes + it writes, each frame end records its offset and timestamp, and one waiter + per stream polls the oldest pending offset, interleaved with later writes so + a lagging ACK never stalls sending, retiring every frame at or below what is + acknowledged. +- **No hook, no fields.** A subscription whose backend reports `poll_acked` + unsupported (qmux, the browser transport) keeps no frontier and adds + nothing to `lag` or `dropped`. A node serving only such viewers leaves both + absent. Document this in `doc/concept/stats.md`. +- **The lag edge is the newest group's.** On the branch, `Produced.newest` is + a lifetime `fetch_max`, so one far-future timestamp pins the edge and every + later sample reads `inf`. Take the edge from the newest group instead (still + a max within it, since B-frames present out of order), so a jump clears at + the next group. Only the publisher's own broadcast is affected, so no + refusal or clamp. + +Decided on the branch, kept: + +- The cache track's field is `produced: Produced`, mirroring `Delivered` and + `group::Producer::produced()`. +- `dropped` is a `Content` (`bytes`, `frames`, `groups`, `datagrams`), the + same type as `stale`, and a subset of the top-level payload counters. + `datagrams` is always zero, since a datagram has no acknowledgement to + miss. Like `lag`, it is written only when non-empty, so ingress, idle, and a + group's per-path `paths` rows stay unchanged on the wire. + +Revisit under frame granularity: the branch counts a stall in whole report +intervals and charges no further stall in the final sample. Both were +compromises for a frontier that moved once per group. + +Write-path cost, measured on the branch (`perf stat` user-space instructions +per frame, the 300-minus-100-group difference so setup cancels, medians of +three; criterion wall time was unusable on a loaded host): + +| Workload | `main` | branch, `Duration` stamp | branch head | +| --- | ---: | ---: | ---: | +| `write_frame`, 64 B | 994 | 1,064 (+7.0%) | 1,027 (+3.3%) | +| `write_frames` x32, 64 B | 417 | 523 (+25%) | 471 (+13%) | +| 16 KiB frame, 16 chunks | 12,957 | 14,304 (+10%) | 13,690 (+5.7%) | + +What remains per timed write is one `fetch_max`, one release `fetch_add`, and +a u128 rescale. Store the raw track-scale value and rescale in the sampler +instead, and re-measure; the per-frame offset bookkeeping above adds to this +path too, so measure it with the same method. Keep the histogram contract the dashboard reads: buckets keyed by upper edge (`"50ms"` to `"5s"`, then `"inf"`, empty buckets omitted), cumulative and -monotonic so readers diff two samples and sum nodes bucket by bucket. Update -`doc/concept/stats.md` and the stats section of `doc/bin/relay/config.md`. +monotonic so readers diff two samples and sum nodes bucket by bucket. + +Tests cover what group granularity hid: groups written across report ticks +rather than whole between them, a peer acknowledging in bursts, a reset +mid-group counting only the unacknowledged frames as `dropped`, an +unsupported backend leaving both fields absent, and a far-future timestamp +clearing at the next group. + +Public API: breaking in `moq-net` (`stats::Registry::report` takes the +drain's instant); additive `Traffic` fields and a new `stats::Histogram`, +re-exported by `moq-stats`. Wire: the stats JSON gains the egress fields. +Update `doc/concept/stats.md` and the stats section of +`doc/bin/relay/config.md`. -Public API: breaking in `moq-stats`, as the line already decided. Wire: the -stats JSON gains the egress fields. +## Required +- [poll_acked on moq-net's send stream](/quest/m2/quic-ack-hook.md) - the + per-frame acknowledgement the frontier moves on diff --git a/quest/m2/quic-ack-hook.md b/quest/m2/quic-ack-hook.md index 1aa1cb6fbf6..d545e52f5f2 100644 --- a/quest/m2/quic-ack-hook.md +++ b/quest/m2/quic-ack-hook.md @@ -70,8 +70,8 @@ lives: (which truncates to 0), shrinks the acknowledged prefix. The waiters fail anyway; keep the prefix a `poll_acked` reports monotonic. - The offset counts bytes on the QUIC stream. The trait method takes a byte - offset, like `moq-quic` (decided 2026-10-17 on #5817), and the starvation - quest's writer counts the bytes it writes, so neither driver needs a public + offset, like `moq-quic` (decided 2026-10-17 on #5817), and the lag + histogram's writer counts the bytes it writes, so neither driver needs a public `offset()`. A layer that writes its own prefix adds its length: `quic::web` already tracks its WebTransport header for `reset_at`. @@ -79,8 +79,11 @@ The moq-tokio and moq-uring adapters implement the trait method over the driver's. Leave the browser transport on the default; the browser's `WebTransportSendStream.getStats()` is unimplemented in shipping Chrome and its `bytesAcknowledged` is at risk in the W3C draft. Decided 2026-10-22 (maintainer): qmux's `poll_acked` reports -unsupported too, since writing into TCP is not delivery; over qmux the -histogram stays at group granularity. +unsupported too, since writing into TCP is not delivery. Decided 2026-10-26 +(maintainer, on [moq-dev/moq#6791](https://github.com/moq-dev/moq/pull/6791)): +over qmux and the browser transport, the lag histogram, `dropped`, and the +delivery-delay histogram are absent, never zero; group granularity is +rejected as a fallback. Acceptance includes the first two traps, through each driver: a finished stream whose bytes were all acknowledged resolves `poll_acked` as delivered, @@ -93,8 +96,9 @@ offset order, reset by sender, STOP_SENDING by the receiver, and session close. -Decided in the 2026-09-30 audit: moved to m2 with its only consumer, -[frame-granularity starvation](/quest/m2/starvation-frames.md). +Decided in the 2026-09-30 audit: moved to m2 with its consumers, the +[viewer lag histogram](/quest/m2/qos/lag-histogram.md) and the +[delivery-delay histogram](/quest/m2/delivery-delay.md). ## Required @@ -102,8 +106,3 @@ Decided in the 2026-09-30 audit: moved to m2 with its only consumer, driver shape `poll_acked` takes - [moq-net polls on kio waiters](/quest/m1/quic-api/net-poll.md) - the trait shape the moq-net method takes - -## Related - -- [Starvation at frame granularity](/quest/m2/starvation-frames.md) - the - moq-net consumer diff --git a/quest/m2/starvation-frames.md b/quest/m2/starvation-frames.md deleted file mode 100644 index 753a13974e0..00000000000 --- a/quest/m2/starvation-frames.md +++ /dev/null @@ -1,66 +0,0 @@ -# [S] Starvation at frame granularity - -## Goal - -Each subscription's acknowledged frontier moves at every frame boundary -instead of once per group, so the starvation histogram and dropped-media -counters from the lag histogram read a frontier at most one frame stale, and a -second histogram reports per-frame delivery delay so viewer jitter is visible -per broadcast. The wire shape from the lag histogram is unchanged apart from -the new histogram. - -## Plan - -A frame is atomic to the viewer: it is useful only once fully delivered. The -sampler from the lag histogram is unchanged; what changes is how often the -frontier it reads moves. `Writer` in `rs/moq-net/src/coding/writer.rs` gains -a monotonic written-offset counter, since every write funnels through it, so -at each frame end the per-group serve task knows the stream offset and the -frame timestamp. Record `(offset, timestamp, bytes, written_at)` and await -`poll_acked(offset)` on moq-net's own `transport::poll::SendStream`, -interleaved with the writes of later frames so a lagging ACK never stalls -sending. One waiter per stream suffices: offsets are -acknowledged in order for the purpose of this metric, so poll the oldest -pending frame and retire everything at or below the acknowledged offset. - -Each acknowledged frame moves the subscription's frontier to that frame's -timestamp, so the interval samples from the lag histogram read a frontier that -is at most one frame stale instead of one group. When a stream -is reset before a frame is acknowledged, `dropped_duration` now grows by the -span from the newest acknowledged frame to the newest written one, which is -the exact media the viewer lost. - -Add a second byte-weighted cumulative histogram of delivery delay: -`acked_at - written_at` per frame, where `acked_at` is when the serve task is -woken by `poll_acked`. Its low edge approaches the path RTT and its spread is -the jitter. Document that the value still includes the return one-way delay -and the peer's delayed-ACK timer, since the hook reports no corrected instant -(decided 2026-10-17 on #5817), and that a backend without the hook leaves the -histogram absent, never zero. - -Backends that return unsupported from `poll_acked` keep the group-granularity -sampling from the lag histogram, so the histogram never disappears over qmux or -a browser transport; document which resolution a node offers. - -Deferred to m2 with the parent QoS sampler. Group-granularity lag lands -first after the next release; frame-granularity sampling then adds the ACK -hook without making either resolution a release gate. - -Decided 2026-10-08: the group-granularity sampler is the -[viewer lag histogram](/quest/m2/qos/lag-histogram.md), not yet on main (in -progress as [moq-dev/moq#6791](https://github.com/moq-dev/moq/pull/6791)), so -that quest is Required. - -Tests: the frontier tracking frame ends under a peer that acknowledges in -bursts, with interval samples landing one bucket lower than group-granularity -tracking of the same run; a reset mid-group attributing only the -unacknowledged span to `dropped_duration`; delivery delay staying inside one bucket of the true RTT plus the peer's -`max_ack_delay`; and fallback to group-granularity frontier tracking when the hook is -unsupported. - -## Required - -- [poll_acked on moq-net's send stream](/quest/m2/quic-ack-hook.md) - the - hook this samples through -- [Egress lag histogram](/quest/m2/qos/lag-histogram.md) - the - group-granularity sampler and starvation histogram this refines