Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,8 @@ lite-07; bitrate enforcement is not part of this release.
- [poll_acked on moq-net's send stream](/quest/m2/quic-ack-hook.md) - the
backend-neutral hook on moq-net's own transport trait that awaits an
acknowledged stream offset, implemented by the moq-tokio and moq-uring adapters
- [Starvation at frame granularity](/quest/m2/starvation-frames.md) - the
acknowledged frontier moves at every frame boundary through `poll_acked`,
with a delivery-delay histogram for jitter
- [Delivery delay](/quest/m2/delivery-delay.md) - a per-frame
delivery-delay histogram from `poll_acked`, so viewer jitter is visible
- [#3199](/quest/m2/3199-moq-uring-remove-sq-indirection-and-per-enter-ring-fd.md) - moq-uring: remove SQ indirection and per-enter ring fd lookup, if a profile shows the win
- [#3200](/quest/m2/3200-moq-uring-batch-completion-wakeups-with-min-timeout.md) - moq-uring: batch completion wakeups with MIN_TIMEOUT
- [Browser media report](/quest/m2/rs2ts-browser-report.md) - compare Rust-backed and handwritten @moq/net through full media workloads, supplementing the pre-adoption comparisons
Expand Down
37 changes: 37 additions & 0 deletions quest/m2/delivery-delay.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# [S] Per-frame delivery delay histogram

## Goal

The relay's `moq-stats` egress side reports, per broadcast, a byte-weighted
cumulative histogram of per-frame delivery delay (from when a frame is
written to when the peer acknowledges it), so viewer jitter is visible beside
the `lag` histogram. Subscriptions on a backend that cannot see
acknowledgements leave it absent, never zero.

## Plan

Decided 2026-10-26 (maintainer, on
[moq-dev/moq#6791](https://github.com/moq-dev/moq/pull/6791)): this quest's
frontier half, moving the acknowledged frontier at every frame end and
counting `dropped` per frame, moved into the
[viewer lag histogram](/quest/m2/qos/lag-histogram.md), which no longer
lands at group granularity. That quest already awaits each frame's
acknowledgement through `poll_acked`; this one adds the delay it observes.

Record `acked_at - written_at` per frame, where `acked_at` is when the serve
task is woken by `poll_acked`. Its low edge approaches the path RTT and its
spread is the jitter. Use the `lag` histogram's type and bucket contract.
Document that the value still includes the return one-way delay and the
peer's delayed-ACK timer, since the hook reports no corrected instant
(decided 2026-10-17 on #5817).

Tests: delivery delay staying inside one bucket of the true RTT plus the
peer's `max_ack_delay`, and the histogram absent over a backend without the
hook.

Public API: an additive `Traffic` field. Wire: an additive stats JSON field.

## Required

- [Viewer lag histogram](/quest/m2/qos/lag-histogram.md) - the per-frame
acknowledgement tracking and histogram type this reuses
9 changes: 6 additions & 3 deletions quest/m2/qos-lag-dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

The demo stats dashboard (`demo/web/src/stats.ts`) shows the relay's viewer
lag and dropped media: percentiles of the egress `lag` histogram over the
chart window, and `dropped` duration, bytes, and groups as rates, cluster-wide
chart window, and `dropped` bytes, frames, and groups as rates, cluster-wide
and per node like the existing counters.

## Plan
Expand All @@ -15,8 +15,11 @@ and per node like the existing counters.
lands them on the egress side of its per-group totals and prefix tracks
(decided 2026-10-08). `lag` is a cumulative byte count per bucket keyed
by its upper edge (`"50ms"` to `"5s"`, then `"inf"`, empty buckets
omitted); `dropped` is `{ duration, bytes, groups }` with the duration in
fractional milliseconds. `doc/concept/stats.md` documents them. Diff two
omitted); `dropped` has the same four fields as `stale` (`bytes`, `frames`,
`groups`, `datagrams`), written only when non-empty. `lag` is absent where
no viewer's backend reports acknowledgements (qmux, the browser
transport), so show no data there rather than zero lag.
`doc/concept/stats.md` documents them. Diff two
samples within one subscription for an interval's distribution, as the
dashboard already does to turn cumulative bytes into rates, and sum nodes
bucket by bucket.
Expand Down
6 changes: 4 additions & 2 deletions quest/m2/qos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,10 @@ histogram and dropped counters across.

Decided (2026-10-08): the egress histogram is its own quest,
[viewer lag histogram](/quest/m2/qos/lag-histogram.md), since every other
child reads it. The line's PR, #4133, closed without landing (2026-10-09);
its code is kept on `wip/4133-lag-histogram`.
child reads it. The line's PR, #4133, closed without landing (2026-10-09).
Its port, #6791, closed unmerged on 2026-10-26 to wait for the ACK hook,
since `lag` and `dropped` are measured per acknowledged frame; its branch is
the histogram quest's starting point.

Decided (2026-09-28): the line's moq-stats changes break the published
crate. Client stats left the line (2026-09-29)
Expand Down
138 changes: 92 additions & 46 deletions quest/m2/qos/lag-histogram.md
Original file line number Diff line number Diff line change
@@ -1,58 +1,104 @@
# [M] Viewer lag histogram
# [L] Viewer lag histogram

## Goal

The relay's `moq-stats` egress side reports, per broadcast, how far behind
its viewers are according to what the network has acknowledged: a
byte-weighted cumulative `lag` histogram and `dropped` counters (duration,
bytes, and groups dropped before they were acknowledged), on the stats totals
and prefix tracks (`<path>/prefix.json`). A spliced subscription
stays exact across a route switch, and a closing subscription records its
last partial interval.
its viewers are according to the frames the network has acknowledged: a
byte-weighted cumulative `lag` histogram and `dropped` counters (the frames
written after the newest acknowledged one when a group stream ends without
delivering them), on the stats totals, a group's totals, and the prefix tracks
(`<path>/prefix.json`). A spliced subscription stays exact across a route
switch, and a closing subscription records its last partial interval.
Subscriptions on a backend that cannot see acknowledgements report neither
field: absent, never zero.

## Plan

Decided 2026-10-08: split out of the QoS line so the dashboard and publisher
timeliness can require a histogram that exists on `main`. This quest lands it
on `main` now that prefix tracks have landed
([#5811](https://github.com/moq-dev/moq/pull/5811)), moved from the
group tracks' per-path `paths` rows it writes today
onto the totals and prefix tracks.

Decided 2026-10-09: the QoS line
([#4133](https://github.com/moq-dev/moq/pull/4133)) closed without landing,
since every code change it had over `main` is this histogram. Its code is kept
on `wip/4133-lag-histogram` as this quest's starting point: the sampler
(#4298, `rs/moq-net/src/stats.rs`), the splice fixes (#5009), and the final
sample (#4451). Porting it is not mechanical:

- `main` moved to caller-supplied time (#4437) and removed the frozen test
clock (`model::clock::advance`). The sampler reads wall time in
`Registry::report`, when a `Frontier` opens, on `Delivery` acks, and in the
final sample in `Drop for FrontierInner`, which has no caller to supply a
time. Redesign those to take time from their callers, and give the final
sample an explicit close path.
- Untimed tracks (#4822): record `Production` only for timed frames.
- `max_age` became `max_delay` on subscriptions (#4917).

Open review findings on #4133 to fix while porting:

- A timestamp regression hides a stalled viewer
([3](https://github.com/moq-dev/moq/pull/4133#issuecomment-6053929743)).
- A copy served across a no-route gap stays watched
([B](https://github.com/moq-dev/moq/pull/4133#issuecomment-6053929743)).
- A partial write is charged the frame's full declared size
([P2](https://github.com/moq-dev/moq/pull/4133#pullrequestreview-5452586850)).
- `Dropped::add` should saturate instead of overflowing
([comment](https://github.com/moq-dev/moq/pull/4133#discussion_r4215691174)).
- Bound the frontier list when `Registry::report` is not called
([comment](https://github.com/moq-dev/moq/pull/4133#discussion_r4215691184)).
Starting point: the branch `quest/m2/qos/lag-histogram` on
[t0ms's fork](https://github.com/t0ms/moq/tree/quest/m2/qos/lag-histogram),
head `9ed8b831f93f39565333a92d5dd4a5f24d2c3fc5`, from
[moq-dev/moq#6791](https://github.com/moq-dev/moq/pull/6791) (closed
unmerged 2026-10-26). It ports the #4133 sampler onto `main` with
caller-supplied time (`Registry::report(&self, report, now)`), an explicit
close path, route-copy feeds, untimed tracks recording nothing, the five open
#4133 findings fixed, docs, and tests. Its frontier moves only on an
acknowledged group FIN, which this quest replaces. The PR body has the
details of what it settled; bring it up to date with `main` first.

Decided 2026-10-26 (maintainer, on #6791), each with its reason:

- **Frame granularity, through the ACK hook.** Group granularity is rejected
as a definition: a healthy zero-RTT viewer on 2 s groups reads 0.5 s to 5 s
behind, and every unsubscribe, rendition switch, or session close charges
the open group's whole content to `dropped`, so a healthy audience with
churn reads a dropped baseline. The frontier instead moves at every
acknowledged frame end through [poll_acked](/quest/m2/quic-ack-hook.md), so
the sampler reads a frontier at most one frame stale. `dropped` counts only
the frames written after the newest acknowledged one. This folds in the
frontier half of the former starvation-frames quest; the delivery-delay
histogram stays in [its own quest](/quest/m2/delivery-delay.md). Its sketch,
to adjust as the code dictates: the group stream's writer counts the bytes
it writes, each frame end records its offset and timestamp, and one waiter
per stream polls the oldest pending offset, interleaved with later writes so
a lagging ACK never stalls sending, retiring every frame at or below what is
acknowledged.
- **No hook, no fields.** A subscription whose backend reports `poll_acked`
unsupported (qmux, the browser transport) keeps no frontier and adds
nothing to `lag` or `dropped`. A node serving only such viewers leaves both
absent. Document this in `doc/concept/stats.md`.
- **The lag edge is the newest group's.** On the branch, `Produced.newest` is
a lifetime `fetch_max`, so one far-future timestamp pins the edge and every
later sample reads `inf`. Take the edge from the newest group instead (still
a max within it, since B-frames present out of order), so a jump clears at
the next group. Only the publisher's own broadcast is affected, so no
refusal or clamp.

Decided on the branch, kept:

- The cache track's field is `produced: Produced`, mirroring `Delivered` and
`group::Producer::produced()`.
- `dropped` is a `Content` (`bytes`, `frames`, `groups`, `datagrams`), the
same type as `stale`, and a subset of the top-level payload counters.
`datagrams` is always zero, since a datagram has no acknowledgement to
miss. Like `lag`, it is written only when non-empty, so ingress, idle, and a
group's per-path `paths` rows stay unchanged on the wire.

Revisit under frame granularity: the branch counts a stall in whole report
intervals and charges no further stall in the final sample. Both were
compromises for a frontier that moved once per group.

Write-path cost, measured on the branch (`perf stat` user-space instructions
per frame, the 300-minus-100-group difference so setup cancels, medians of
three; criterion wall time was unusable on a loaded host):

| Workload | `main` | branch, `Duration` stamp | branch head |
| --- | ---: | ---: | ---: |
| `write_frame`, 64 B | 994 | 1,064 (+7.0%) | 1,027 (+3.3%) |
| `write_frames` x32, 64 B | 417 | 523 (+25%) | 471 (+13%) |
| 16 KiB frame, 16 chunks | 12,957 | 14,304 (+10%) | 13,690 (+5.7%) |

What remains per timed write is one `fetch_max`, one release `fetch_add`, and
a u128 rescale. Store the raw track-scale value and rescale in the sampler
instead, and re-measure; the per-frame offset bookkeeping above adds to this
path too, so measure it with the same method.

Keep the histogram contract the dashboard reads: buckets keyed by upper edge
(`"50ms"` to `"5s"`, then `"inf"`, empty buckets omitted), cumulative and
monotonic so readers diff two samples and sum nodes bucket by bucket. Update
`doc/concept/stats.md` and the stats section of `doc/bin/relay/config.md`.
monotonic so readers diff two samples and sum nodes bucket by bucket.

Tests cover what group granularity hid: groups written across report ticks
rather than whole between them, a peer acknowledging in bursts, a reset
mid-group counting only the unacknowledged frames as `dropped`, an
unsupported backend leaving both fields absent, and a far-future timestamp
clearing at the next group.

Public API: breaking in `moq-net` (`stats::Registry::report` takes the
drain's instant); additive `Traffic` fields and a new `stats::Histogram`,
re-exported by `moq-stats`. Wire: the stats JSON gains the egress fields.
Update `doc/concept/stats.md` and the stats section of
`doc/bin/relay/config.md`.

Public API: breaking in `moq-stats`, as the line already decided. Wire: the
stats JSON gains the egress fields.
## Required

- [poll_acked on moq-net's send stream](/quest/m2/quic-ack-hook.md) - the
per-frame acknowledgement the frontier moves on
21 changes: 10 additions & 11 deletions quest/m2/quic-ack-hook.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,17 +70,20 @@ lives:
(which truncates to 0), shrinks the acknowledged prefix. The waiters fail
anyway; keep the prefix a `poll_acked` reports monotonic.
- The offset counts bytes on the QUIC stream. The trait method takes a byte
offset, like `moq-quic` (decided 2026-10-17 on #5817), and the starvation
quest's writer counts the bytes it writes, so neither driver needs a public
offset, like `moq-quic` (decided 2026-10-17 on #5817), and the lag
histogram's writer counts the bytes it writes, so neither driver needs a public
`offset()`. A layer that writes its own prefix adds its length: `quic::web`
already tracks its WebTransport header for `reset_at`.

The moq-tokio and moq-uring adapters implement the trait method over the
driver's. Leave the browser transport on the default; the browser's `WebTransportSendStream.getStats()` is
unimplemented in shipping Chrome and its `bytesAcknowledged` is at risk in the
W3C draft. Decided 2026-10-22 (maintainer): qmux's `poll_acked` reports
unsupported too, since writing into TCP is not delivery; over qmux the
histogram stays at group granularity.
unsupported too, since writing into TCP is not delivery. Decided 2026-10-26
(maintainer, on [moq-dev/moq#6791](https://github.com/moq-dev/moq/pull/6791)):
over qmux and the browser transport, the lag histogram, `dropped`, and the
delivery-delay histogram are absent, never zero; group granularity is
rejected as a fallback.

Acceptance includes the first two traps, through each driver: a finished
stream whose bytes were all acknowledged resolves `poll_acked` as delivered,
Expand All @@ -93,17 +96,13 @@ offset order, reset by sender, STOP_SENDING by the receiver, and session
close.


Decided in the 2026-09-30 audit: moved to m2 with its only consumer,
[frame-granularity starvation](/quest/m2/starvation-frames.md).
Decided in the 2026-09-30 audit: moved to m2 with its consumers, the
[viewer lag histogram](/quest/m2/qos/lag-histogram.md) and the
[delivery-delay histogram](/quest/m2/delivery-delay.md).

## Required

- [QUIC driver poll waits](/quest/m1/quic-api/poll.md) - the `&self, &kio::Waiter`
driver shape `poll_acked` takes
- [moq-net polls on kio waiters](/quest/m1/quic-api/net-poll.md) - the trait
shape the moq-net method takes

## Related

- [Starvation at frame granularity](/quest/m2/starvation-frames.md) - the
moq-net consumer
66 changes: 0 additions & 66 deletions quest/m2/starvation-frames.md

This file was deleted.

Loading