From 2a6899080b41cb9dd856c28b1910d4a6aa3889be Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 16 Oct 2026 17:04:48 -0700 Subject: [PATCH 1/3] quest: claim quest/m2/stats-prefix-tracks From cbff7dd698abf807c55412e188cbacdc651e101f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 16 Oct 2026 17:27:23 -0700 Subject: [PATCH 2/3] feat(stats): requested prefix tracks with rollup, self, and children A reader can request `/traffic.json` for any broadcast prefix in a stats group and gets, per tier, the prefix's rollup, the broadcast at exactly the prefix (`self`), and a one-level map of each direct child, produced only while held. Sums are drain-time like presence: each row walks its ancestors in the held set, so nothing touches the per-frame path. An oversized prefix frame ends that flavor with FrameTooLarge. The aggregator merges prefix tracks with a per-node baseline, so the rollup never regresses; children sum the live nodes only. Co-Authored-By: Claude Opus 5.5 --- doc/bin/inspect.md | 3 +- doc/concept/stats.md | 41 +- quest/m0/broadcast-epoch/README.md | 2 - quest/m2/README.md | 1 - quest/m2/qos-lag-dashboard.md | 6 +- quest/m2/qos-publisher-timeliness.md | 6 +- quest/m2/qos/README.md | 8 +- quest/m2/qos/lag-histogram.md | 5 +- quest/m2/stats-prefix-tracks.md | 100 ----- rs/moq-stats/benches/producer_tick.rs | 76 +++- rs/moq-stats/src/aggregate.rs | 212 +++++++++- rs/moq-stats/src/consume.rs | 46 +- rs/moq-stats/src/lib.rs | 146 ++++++- rs/moq-stats/src/produce.rs | 581 ++++++++++++++++++++++++-- rs/moq-stats/src/produce_bench.rs | 14 + 15 files changed, 1051 insertions(+), 196 deletions(-) delete mode 100644 quest/m2/stats-prefix-tracks.md diff --git a/doc/bin/inspect.md b/doc/bin/inspect.md index cd18220d1bc..f46c49ecc85 100644 --- a/doc/bin/inspect.md +++ b/doc/bin/inspect.md @@ -77,6 +77,7 @@ The node name (`local/host` here) is `stats.node`. Each fetch returns the newest snapshot of cumulative counters. `totals.json` sums each tier's traffic and sessions, `publisher.json` is egress per broadcast path, `subscriber.json` is ingress, and `sessions.json` counts sessions per auth -root. Fetch twice and divide by the interval for a +root. A subscriber may also ask for `/traffic.json`, one broadcast +prefix rolled up with each child, produced only while held. Fetch twice and divide by the interval for a rate. The `.json.z` twins carry compressed patches, which these tools print as raw bytes. [Stats](/concept/stats) describes every field. diff --git a/doc/concept/stats.md b/doc/concept/stats.md index 58e3cbef7fe..4a298381238 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -67,6 +67,7 @@ comes in two encodings: | `[/]subscriber.json` | broadcast path | [Traffic](#traffic) this node received (ingress) | | `[/]sessions.json` | auth root | [Sessions](#sessions) connected now | | `[/]presence.json` | tier | [Sessions](#sessions) under one auth root | +| `[/]traffic.json` | tier | [Prefix](#prefix) traffic under one broadcast prefix | The default tier's per-tier tracks are unprefixed. A named tier prefixes each name with its label and a slash: tier `region/sjc` publishes @@ -83,7 +84,9 @@ A presence track exists only while subscribed. Ask for `/presence.json` for any auth root in the group (`presence.json` for the empty root). The relay answers with the root's current sessions and counts from there, holding a root with no sessions open at `{}`, and stops producing it once the last subscriber -leaves. A root outside the group, or any other name, is refused. +leaves. A prefix track works the same way for any broadcast prefix in the +group: ask for `/traffic.json` (`traffic.json` for the empty prefix). A +root or prefix outside the group, or any other name, is refused. ## Frames @@ -133,6 +136,36 @@ A presence track's entries are cumulative from when it was first held: a session that ends stays counted, and so does every one after it, while anyone holds the track. A track held again starts over. +### Prefix + +```json +{ + "": { + "publisher": { ... }, "subscriber": { ... }, + "self": { "publisher": { ... }, "subscriber": { ... } }, + "children": { "live": { "publisher": { ... }, "subscriber": { ... } } } + } +} +``` + +Each tier the prefix recorded on, with three views of its [Traffic](#traffic): +the top-level `publisher` and `subscriber` roll up every broadcast at or under +the prefix, so a [totals](#totals) reader parses them unchanged; `self` is the +broadcast at exactly the prefix, apart from any nested under it; and +`children` rolls up each direct child, keyed by its next path segment. + +Like presence, the rollup and `self` are cumulative from when the track was +first held: each frame sums the live broadcasts' counters plus the final +counters of those that ended since. A broadcast that was live before the hold +carries its history from before it, so absolute values are not authoritative: +take differences within one subscription, and read the [totals](#totals) for +real sums. Started minus ended stays exact. A child appears while a broadcast +under it has counters, and leaves after the frame carrying its last one; its +tail stays in the rollup, and a child that returns starts over. Children are +never capped, so roughly ten thousand across four tiers outgrow the 32 MB frame +limit: that encoding's track then ends with `FRAME_TOO_LARGE` instead of going +stale. + ## Entries ### Traffic @@ -206,9 +239,9 @@ count is started minus ended. A frame never shows ended above started. A new epoch starts every counter from zero, so a reader summing a node over time adds each epoch's counters rather than diffing across them. Within one epoch the totals never go down. A per-tier counter going **down** means the -entry was dropped and re-created, and a presence counter going down means the -track was held again. Treat it as the start of a fresh segment rather than a -negative rate. On a route without an epoch (an older relay, or a session older +entry was dropped and re-created, and a presence or prefix counter going down +means the track was held again. Treat it as the start of a fresh segment +rather than a negative rate. On a route without an epoch (an older relay, or a session older than moq-lite 07), a counter going down also means the node restarted. A reader ignores unknown fields, so a newer relay can add counters, and diff --git a/quest/m0/broadcast-epoch/README.md b/quest/m0/broadcast-epoch/README.md index 3a0dee83cd3..93883bdba61 100644 --- a/quest/m0/broadcast-epoch/README.md +++ b/quest/m0/broadcast-epoch/README.md @@ -128,5 +128,3 @@ End+Start. The child quests own their focused regressions. ## Related - [JS consume identity](https://github.com/moq-dev/moq/pull/5254) - requests carry the serving prefix's epoch and new winners cannot reuse an old cached broadcast - -- [Stats prefix tracks](/quest/m2/stats-prefix-tracks.md) - deferred with QoS to m2; neither gates epoch correctness or the next cut diff --git a/quest/m2/README.md b/quest/m2/README.md index 813b339b87a..106e67c62c7 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -37,7 +37,6 @@ ranked just below deterministic TS legs, which it requires. - [Kotlin calls](/quest/m2/kt-ffi-calls.md) - Kotlin calls into moq-ffi cost about what Python's and Go's do, sync and async, once the fork's Kotlin generator drops JNA's per-call overhead - [Poll-native async spike](/quest/m2/ffi-poll.md) - measure kio poll-native exports and foreign poll against `ffi::Task`, then pick the shape - [Rendition preference](/quest/m2/rendition-preference.md) - automatic selection by ``, `Video::ranked`, and WHEP keeps the highest `preference` that decodes, so a compatibility transcode is only picked when nothing preferred decodes -- [Stats prefix tracks](/quest/m2/stats-prefix-tracks.md) - additive tracks required by the breaking lag API; existing maps remain supported - [QoS](/quest/m2/qos/README.md) - broadcast health: relay starvation and timeliness histograms - [Plan: watch worker](/quest/m2/plan-watch-worker.md) - prototype an invisible page worker against app-spawned workers, and land the jank harness that decides - [Watch worker](/quest/m2/watch-worker.md) - watch playback runs in a worker onto an OffscreenCanvas, so main-thread jank never stalls video or audio diff --git a/quest/m2/qos-lag-dashboard.md b/quest/m2/qos-lag-dashboard.md index 78af074f40d..421ea67eaa0 100644 --- a/quest/m2/qos-lag-dashboard.md +++ b/quest/m2/qos-lag-dashboard.md @@ -9,9 +9,8 @@ and per node like the existing counters. ## Plan -- Read both from the stats totals and [prefix - tracks](/quest/m2/stats-prefix-tracks.md), which land first, - rather than the `publisher.json` rows they sit on today (decided in the +- Read both from the stats totals and prefix tracks + (`/traffic.json`) rather than the `publisher.json` rows they sit on today (decided in the 2026-10-06 audit; the rows stay for existing readers, 2026-10-12); [viewer lag histogram](/quest/m2/qos/lag-histogram.md) lands them on the egress side of its per-group totals and prefix tracks (decided 2026-10-08). `lag` is a cumulative byte count per bucket keyed @@ -35,4 +34,3 @@ Public API: none. Wire: none. ## Required - [Viewer lag histogram](/quest/m2/qos/lag-histogram.md) - the `lag` histogram and `dropped` counters the dashboard reads -- [Stats prefix tracks](/quest/m2/stats-prefix-tracks.md) - the prefix tracks the dashboard reads lag from, beside the totals diff --git a/quest/m2/qos-publisher-timeliness.md b/quest/m2/qos-publisher-timeliness.md index eda75132bd7..f3b956d1ea2 100644 --- a/quest/m2/qos-publisher-timeliness.md +++ b/quest/m2/qos-publisher-timeliness.md @@ -28,9 +28,8 @@ milliseconds, and the re-anchor happens on its own as old minima expire. Document the window length beside the bucket edges. Aggregate as a byte-weighted cumulative histogram of drift on the ingress -(`Role::Subscriber`) side of the stats totals and [prefix -tracks](/quest/m2/stats-prefix-tracks.md), which land first, -rather than the per-path rows (decided in the 2026-10-06 audit), with the same +(`Role::Subscriber`) side of the stats totals and prefix tracks +(`/traffic.json`) rather than the per-path rows (decided in the 2026-10-06 audit), with the same bucket edges and the same monotonic contract as the egress `lag` histogram. Beside it keep two cumulative counters: `timestamp_regressions`, frames whose timestamp is below the previous group's newest timestamp, and `stalls`, gaps between complete frames @@ -56,7 +55,6 @@ excluded. ## Required - [Viewer lag histogram](/quest/m2/qos/lag-histogram.md) - the egress histogram whose type and edges this reuses -- [Stats prefix tracks](/quest/m2/stats-prefix-tracks.md) - the prefix tracks the drift histogram lands on, beside the totals ## Related diff --git a/quest/m2/qos/README.md b/quest/m2/qos/README.md index 4e41e175634..46498189b18 100644 --- a/quest/m2/qos/README.md +++ b/quest/m2/qos/README.md @@ -33,9 +33,8 @@ The per-broadcast verdict combining client reports, the relay's starvation, and publisher timeliness lives downstream. This line only reports the counters. -Decided (2026-10-06 audit): [stats prefix -tracks](/quest/m2/stats-prefix-tracks.md) lands first, since it is -a prerequisite of the lag API; both are deferred to m2 by the maintainer +Decided (2026-10-06 audit): stats prefix tracks (`/traffic.json`) +landed first, since they are a prerequisite of the lag API; both are deferred to m2 by the maintainer on 2026-10-15; the lag histogram lands on the stats totals and prefix tracks rather than the per-path `publisher.json` and `subscriber.json` map rows, which stay unchanged for existing readers @@ -61,9 +60,6 @@ on `main`. ## Related -- [Stats prefix tracks](/quest/m2/stats-prefix-tracks.md) - - lands first beside the map rows; this line's histograms land on the - totals and prefix tracks - [Media stats](/quest/m2/stats/README.md) - publishers and viewers report their own media, transport, and playback health, the media half of a verdict - [Loss delay](/quest/m3/cut-through/loss-delay.md) - an ingress counter of diff --git a/quest/m2/qos/lag-histogram.md b/quest/m2/qos/lag-histogram.md index 86a8510f4e8..fc07250d37a 100644 --- a/quest/m2/qos/lag-histogram.md +++ b/quest/m2/qos/lag-histogram.md @@ -6,7 +6,7 @@ The relay's `moq-stats` egress side reports, per broadcast, how far behind its viewers are according to what the network has acknowledged: a byte-weighted cumulative `lag` histogram and `dropped` counters (duration, bytes, and groups dropped before they were acknowledged), on the stats totals -and [prefix tracks](/quest/m2/stats-prefix-tracks.md). A spliced subscription +and prefix tracks (`/traffic.json`). A spliced subscription stays exact across a route switch, and a closing subscription records its last partial interval. @@ -55,6 +55,3 @@ monotonic so readers diff two samples and sum nodes bucket by bucket. Update Public API: breaking in `moq-stats`, as the line already decided. Wire: the stats JSON gains the egress fields. -## Required - -- [Stats prefix tracks](/quest/m2/stats-prefix-tracks.md) - the prefix tracks the histogram lands on, beside the totals diff --git a/quest/m2/stats-prefix-tracks.md b/quest/m2/stats-prefix-tracks.md deleted file mode 100644 index c02cfc6ef33..00000000000 --- a/quest/m2/stats-prefix-tracks.md +++ /dev/null @@ -1,100 +0,0 @@ -# [L] Stats prefix tracks - -## Goal - -A reader of a node's stats broadcast can request a stats track for any prefix -within the group and gets that prefix's rollup, its self counters, and a -one-level map of its direct children, produced only while requested. The -per-path map tracks (`[/]publisher.json`, `[/]subscriber.json`, and -their `.json.z` siblings) stay beside them, unchanged, for existing readers. - -## Plan - -Split from stats-split (decided 2026-10-08), which landed the per-group -`totals.json` and the requested per-root `/presence.json` beside the -existing tracks. Reuse what it built: the `/.json` naming (the last -segment picks the kind, everything before it is the key), the requested-track -hold and reclaim in `rs/moq-stats/src/produce.rs`, and the aggregator's -per-node reset baseline in `rs/moq-stats/src/aggregate.rs`. - -Decided 2026-10-05 and 2026-10-07: - -- **Prefix tracks.** Routing is by prefix, so a reader requests any prefix of - a member path (depth bounded by `Path::MAX_PARTS` less the group's own - segments) and gets one track: the prefix's rollup, its self counters, and a - one-level map of each direct child's rollup (a channel's broadcasts). - Nothing is produced for a prefix no one holds. The relay's `--stats-depth` - only places the group broadcasts; requests are not capped by it. Track names - must stay injective: tiers and paths both contain `/`. -- Counters follow presence (revised 2026-10-16, #5719): each drain sums the - live members' readouts plus the closing readouts of members that ended - while the prefix was held. Live members carry their history from before - the hold, so absolute values are not authoritative and readers take - differences within one subscription; the group totals carry the real sums. - Every `active` (started less ended) is exact for a prefix held mid-stream. - Test a prefix held while 300 subscriptions are live, then watch them leave. -- A request for a prefix with no members is held open at zero and reclaimed - when its last consumer leaves. -- A child leaves the one-level map after the frame carrying its closing - readout; its tail stays in the parent's rollup. -- **Self counters.** A prefix track also carries cumulative counters for - members at exactly the prefix, so a reader can show a broadcast that also has - nested broadcasts. A reader can't derive them as the rollup less the current - children: a pruned child's tail stays in the rollup, so the difference jumps - when the child leaves the map. They follow the rollup's rules. Test a nested - broadcast ending and leaving the map while the parent prefix stays held: the - parent's self counters stay flat and its rollup keeps the child's tail. -- The aggregator merges a prefix track across nodes with a baseline per - upstream (node, epoch) hold and treats a decrease as that upstream's reset, - so a merged prefix track never regresses within one downstream - subscription. Test a join, a leave, and a restart under a held prefix. -- **Drain-time sums** (revised 2026-10-16, #5719). Sum held prefixes in - `moq-stats` per drain with `Cumulative`, like presence: walk each report - row's ancestors and look each up in the held set, so the cost scales with - rows times path depth, not held prefixes, and nothing touches the per-frame - path. No `moq-net` API change: registry-level prefix accumulators would - push request lifecycle into the collection layer and put shared atomics on - every frame bump under a held prefix. Extend `producer_tick` to sweep rows, - held prefixes, and depth. -- Held prefixes have no cap: a hold is reclaimed when its last reader leaves, - like presence tracks, whose cap was dropped for that reason (decided - 2026-10-12 in #5084). -- Update the aggregator, `doc/concept/stats.md`, and `doc/bin/inspect.md`. - -Decided 2026-10-12 (#5084): the map tracks are kept, not retired. Prefix -tracks are added beside them so existing readers keep working, the same -backwards-compatible shape stats-split took when it kept `sessions.json`. -This supersedes the 2026-10-05 plan to retire the maps in the same release. - -Decided 2026-10-16 (#5719): - -- **Name.** `/traffic.json`, where `` is the full broadcast - prefix, as presence takes the full root, and a prefix outside the group is - refused. The last segment picks the kind and everything before it is the - key, like `/presence.json`, so the name is injective without a - namespace segment. Tiers ride the payload, never the name, so one - subscription returns the whole prefix and no escaping rule is needed. -- **Payload.** Keyed by tier like `totals.json`. Each tier entry carries the - rollup's `publisher` and `subscriber` at its top level, so a totals reader - reads the rollup unchanged, plus `self: {publisher, subscriber}` and - `children: {"": {publisher, subscriber}}`, keyed by the child's - next path segment. -- **Encodings.** Plain and `.json.z` siblings, like every other stats track. -- **No cap or paging on children.** A cap would make the rollup disagree with - its visible children, and paging adds ordering and consistency rules no - measurement asks for yet. A child entry carries both roles per tier, so the - frame reaches the 32 MB `MAX_CACHE_BYTES` limit sooner than `publisher.json` - does (roughly ten thousand children across four tiers), far past the - hundreds a channel holds. An oversized frame must fail loud: end the prefix - track with `FrameTooLarge` instead of the existing tracks' debug log, which - leaves readers on a stale snapshot. Test a prefix whose frame overflows. - -Public API: `moq-stats` producer and consumer types. Wire: stats track names -and payloads. - -## Related - -- [Viewer lag histogram](/quest/m2/qos/lag-histogram.md) - lands on the totals - and these prefix tracks -- [Publisher timeliness at relay ingest](/quest/m2/qos-publisher-timeliness.md) - the drift - histogram lands on the same tracks diff --git a/rs/moq-stats/benches/producer_tick.rs b/rs/moq-stats/benches/producer_tick.rs index 3fb6951f19c..58087ea0df2 100644 --- a/rs/moq-stats/benches/producer_tick.rs +++ b/rs/moq-stats/benches/producer_tick.rs @@ -13,6 +13,10 @@ //! `stats/presence` sweeps the same tick over connected auth roots and the //! presence tracks held for them. //! +//! `stats/prefix` sweeps it over held paths, path depth, and held prefix +//! tracks: each row walks its ancestors once a prefix is held, so the cost +//! grows with rows times depth, plus each held track's encode. +//! //! Run `cargo bench -p moq-stats --features bench --bench producer_tick`. use std::alloc::{GlobalAlloc, Layout, System}; @@ -83,15 +87,18 @@ struct Relay { _publishers: Vec, _cursors: Vec, _sessions: Vec, + _held: Vec, } impl Relay { - fn build(paths: usize, tiers: usize, changed_percent: usize) -> Self { + /// `depth` is each path's segment count, at least 2. + fn build(paths: usize, tiers: usize, changed_percent: usize, depth: usize) -> Self { let registry = stats::Registry::new( stats::Config::new().with_exclude(moq_net::Pattern::subtree(".stats").expect("literal prefix")), ); let (origin, origin_driver) = origin::Producer::new(origin::Config::default()); - let path_names: Vec = (0..paths).map(|index| format!("room/{index:06}")).collect(); + let nested = "/s".repeat(depth - 2); + let path_names: Vec = (0..paths).map(|index| format!("room/{index:06}{nested}")).collect(); let publishers: Vec<_> = path_names .iter() .map(|path| { @@ -130,6 +137,15 @@ impl Relay { _publishers: publishers, _cursors: cursors, _sessions: sessions, + _held: Vec::new(), + } + } + + /// Hold the first `held` paths' `room/` prefix tracks. + fn hold(&mut self, held: usize) { + for index in 0..held { + let sub = self.driver.hold_prefix(&format!("room/{index:06}")); + self._held.push(sub); } } @@ -213,7 +229,7 @@ fn bench(c: &mut Criterion) { let mut relay = None; group.bench_function(id, |b| { let relay = relay.get_or_insert_with(|| { - let mut relay = Relay::build(paths, tiers, changed); + let mut relay = Relay::build(paths, tiers, changed, 2); let row = sample(&mut relay); let micros = row.elapsed.as_secs_f64() * 1e6; println!( @@ -327,5 +343,57 @@ fn presence(c: &mut Criterion) { group.finish(); } -criterion_group!(benches, bench, presence); +const PREFIX_PATHS: [usize; 2] = [1_000, 10_000]; + +/// Segments per path. +const PREFIX_DEPTH: [usize; 2] = [2, 8]; + +/// Prefix tracks a reader holds. +const PREFIX_HELD: [usize; 3] = [0, 10, 100]; + +/// What holding prefix tracks costs a tick, with a tenth of the paths moving. +fn prefix(c: &mut Criterion) { + println!("paths depth held allocs/tick us/tick"); + + let mut group = c.benchmark_group("stats/prefix"); + group.sample_size(10); + group.warm_up_time(Duration::from_millis(200)); + group.measurement_time(Duration::from_secs(1)); + for &paths in &PREFIX_PATHS { + for &depth in &PREFIX_DEPTH { + for &held in &PREFIX_HELD { + let id = BenchmarkId::from_parameter(format!("{paths}p_{depth}d_{held}h")); + group.throughput(Throughput::Elements(paths as u64)); + let mut relay = None; + group.bench_function(id, |b| { + let relay = relay.get_or_insert_with(|| { + let mut relay = Relay::build(paths, 1, 10, depth); + black_box(relay.driver.tick()); + relay.hold(held); + let row = sample(&mut relay); + let micros = row.elapsed.as_secs_f64() * 1e6; + println!( + "{paths:>5} {depth:>5} {held:>4} {allocs:>11} {micros:>8.1}", + allocs = row.allocs + ); + relay + }); + b.iter_custom(|iters| { + let mut total = Duration::ZERO; + for _ in 0..iters { + relay.touch(); + let start = Instant::now(); + black_box(relay.driver.tick()); + total += start.elapsed(); + } + total + }); + }); + } + } + } + group.finish(); +} + +criterion_group!(benches, bench, presence, prefix); criterion_main!(benches); diff --git a/rs/moq-stats/src/aggregate.rs b/rs/moq-stats/src/aggregate.rs index 01585b950c3..da27beb2000 100644 --- a/rs/moq-stats/src/aggregate.rs +++ b/rs/moq-stats/src/aggregate.rs @@ -18,8 +18,8 @@ use moq_net::{AsPath, Epoch, PathOwned, origin}; use web_async::time::Instant; use crate::{ - PresenceFrame, Result, SessionsFrame, Totals, TotalsFrame, TrafficFrame, parse_node_path, presence_track, - sessions_track, totals_track, traffic_track, + Prefix, PrefixFrame, PresenceFrame, Result, SessionsFrame, Totals, TotalsFrame, TrafficFrame, Usage, + parse_node_path, prefix_track, presence_track, sessions_track, totals_track, traffic_track, }; /// Configuration for an [`Consumer`]. Construct with [`Config::new`] and chain @@ -99,7 +99,7 @@ impl Default for Config { /// /// Scope an [`origin::Consumer`] to a single group (e.g. `.stats/`) and /// hand it here; each [`Self::totals`] / [`Self::traffic`] / [`Self::presence`] -/// call opens its own announce cursor and subscribes to that track on every +/// / [`Self::prefix`] call opens its own announce cursor and subscribes to that track on every /// node broadcast in the group, summing the cumulative counters per key, so /// the merged track has the same format as one node's. /// @@ -121,10 +121,13 @@ impl Default for Config { /// A sessions track is the exception: it is a gauge, so a departed node stops /// counting its sessions immediately. /// -/// The totals and presence tracks never regress: a node's counter going down -/// is that node resetting (a restart on a route without an epoch, or a -/// presence track held again from zero), so its previous readout is kept as a -/// baseline the new one adds to. A node that resumes after its departure +/// The totals, presence, and prefix tracks never regress: a node's counter +/// going down is that node resetting (a restart on a route without an epoch, +/// or a presence or prefix track held again from zero), so its previous +/// readout is kept as a baseline the new one adds to. A prefix's children are +/// the exception: they sum the live nodes' current children, so a child leaves +/// the merged map with the last node carrying it while its tail stays in the +/// rollup. A node that resumes after its departure /// closed its open pairs reopens them as new starts rather than lowering their /// ends. Per-path traffic also reopens pairs when a node resumes without /// regressing, but keeps the reset contract of a single node: a route without @@ -177,6 +180,15 @@ impl Consumer { inner: Merged::new(self.origin.clone(), &self.config, name), } } + + /// A merged reader over broadcast prefix `path`'s track, holding it on + /// every node broadcast in the group; see [`Self::totals`]. + pub fn prefix(&self, path: impl AsPath) -> PrefixConsumer { + let name = prefix_track(path, self.config.compression); + PrefixConsumer { + inner: Merged::new(self.origin.clone(), &self.config, name), + } + } } /// A merged reader over the totals track across every node in the group. Yields @@ -236,8 +248,21 @@ impl PresenceConsumer { } } +/// A merged reader over one broadcast prefix's track across every node in the +/// group; see [`TotalsConsumer`]. +pub struct PrefixConsumer { + inner: Merged, +} + +impl PrefixConsumer { + /// The next merged frame, or `None` once the announce stream ends. + pub async fn next(&mut self) -> Result> { + kio::wait(|waiter| self.inner.poll_next(waiter)).await + } +} + /// A per-key cumulative counter that folds across nodes. -trait Mergeable: serde::de::DeserializeOwned + Default + Copy + PartialEq + 'static { +trait Mergeable: serde::de::DeserializeOwned + Default + Clone + PartialEq + 'static { /// Whether a node's last contribution survives its departure, with its /// open pairs closed. Only the sessions gauge drops it instead. const STICKY: bool = true; @@ -426,6 +451,71 @@ impl Mergeable for Totals { } } +impl Mergeable for Usage { + const RESETS: bool = true; + + fn merge(acc: &mut Self, other: Self) { + acc.add(other); + } + + fn retire(self) -> Self { + Self { + publisher: self.publisher.retire(), + subscriber: self.subscriber.retire(), + } + } + + fn reopen(self, next: &Self) -> Self { + Self { + publisher: self.publisher.reopen(&next.publisher), + subscriber: self.subscriber.reopen(&next.subscriber), + } + } + + fn regressed(&self, prev: &Self) -> bool { + self.publisher.regressed(&prev.publisher) || self.subscriber.regressed(&prev.subscriber) + } +} + +impl Mergeable for Prefix { + const RESETS: bool = true; + + fn merge(acc: &mut Self, other: Self) { + acc.publisher.add(other.publisher); + acc.subscriber.add(other.subscriber); + acc.own.add(other.own); + for (segment, child) in other.children { + acc.children.entry(segment).or_default().add(child); + } + } + + /// A departed node's children leave the map; their tails stay in the rollup. + fn retire(self) -> Self { + Self { + publisher: self.publisher.retire(), + subscriber: self.subscriber.retire(), + own: self.own.retire(), + children: BTreeMap::new(), + } + } + + fn reopen(self, next: &Self) -> Self { + Self { + publisher: self.publisher.reopen(&next.publisher), + subscriber: self.subscriber.reopen(&next.subscriber), + own: self.own.reopen(&next.own), + children: BTreeMap::new(), + } + } + + /// Children are left out: one leaving or returning lower is not a reset. + fn regressed(&self, prev: &Self) -> bool { + self.publisher.regressed(&prev.publisher) + || self.subscriber.regressed(&prev.subscriber) + || self.own.regressed(&prev.own) + } +} + /// One node's subscription to the merged track. enum Reader { /// Resolving the announced path into a broadcast. `queued` records whether @@ -487,7 +577,10 @@ impl Node { return false; } self.stale = true; - self.last.iter().flatten().any(|(_, value)| value.retire() != *value) + self.last + .iter() + .flatten() + .any(|(_, value)| value.clone().retire() != *value) } /// Take a fresh frame. A resume reopens the closed pairs still open as new @@ -522,10 +615,13 @@ impl Node { /// Add this node's contribution to `acc`. fn contribute(&self, acc: &mut BTreeMap) { for (key, value) in &self.base { - V::merge(acc.entry(key.clone()).or_default(), *value); + V::merge(acc.entry(key.clone()).or_default(), value.clone()); } for (key, value) in self.last.iter().flatten() { - let value = if self.stale { value.retire() } else { *value }; + let value = match self.stale { + true => value.clone().retire(), + false => value.clone(), + }; V::merge(acc.entry(key.clone()).or_default(), value); } } @@ -1707,7 +1803,7 @@ mod tests { .expect("frame"); if let Some(prev) = &prev { for (key, before) in prev { - let after = frame.get(key).copied().unwrap_or_default(); + let after = frame.get(key).cloned().unwrap_or_default(); assert!(monotonic(before, &after), "{key} regressed: {before:?} -> {after:?}"); } } @@ -2110,4 +2206,96 @@ mod tests { let frame = read_until(&mut presence.inner, grew, |frame| frame[""].active() == 2).await; assert_eq!(frame[""].sessions_started, 3); } + + #[tokio::test(start_paused = true)] + async fn prefix_survives_join_leave_and_restart() { + // One prefix held across nodes: a node joining adds its traffic and its + // children, one leaving closes its subscriptions and drops its children + // but keeps its rollup, and a restarted node is a new epoch that adds to + // it. The merged rollup never goes down. + let origin = produce_origin(); + let node_a = node_producer(&origin, "a"); + let _fa = feed(&node_a, Tier::default(), "acme", "acme/room", 8).await; + drive_tick().await; + + let agg = Consumer::new(origin.consume(), Config::new().with_depth(1)); + let mut prefix = agg.prefix("acme"); + let grew = |before: &Prefix, after: &Prefix| !after.regressed(before); + let live = |count| { + move |frame: &PrefixFrame| { + frame + .get("") + .is_some_and(|p| p.publisher.active_subscriptions() == count) + } + }; + let frame = read_until(&mut prefix.inner, grew, live(1)).await; + assert_eq!(frame[""].children["room"].publisher.bytes, 8); + + // Join. + let node_b = node_producer(&origin, "b"); + let fb = feed(&node_b, Tier::default(), "acme", "acme/other", 5).await; + let frame = read_until(&mut prefix.inner, grew, live(2)).await; + assert_eq!(frame[""].publisher.bytes, 13); + assert_eq!(frame[""].children.len(), 2); + + // Leave: node b's child leaves the map, its tail stays in the rollup. + drop(fb); + drop(node_b); + let frame = read_until(&mut prefix.inner, grew, live(1)).await; + assert_eq!(frame[""].publisher.bytes, 13); + assert_eq!(frame[""].publisher.subscriptions_started, 2); + assert!(!frame[""].children.contains_key("other"), "{frame:?}"); + + // Restart: a new epoch at node b's path. + let node_b = node_producer(&origin, "b"); + let _fb = feed(&node_b, Tier::default(), "acme", "acme/other", 3).await; + let frame = read_until(&mut prefix.inner, grew, |frame| { + live(2)(frame) && frame[""].publisher.bytes == 16 + }) + .await; + assert_eq!(frame[""].publisher.subscriptions_started, 3); + assert_eq!(frame[""].children["other"].publisher.bytes, 3, "the new epoch's child"); + } + + #[test] + fn a_prefix_reset_keeps_the_rollup_and_drops_the_children() { + let (mut merged, key) = hand_merged::(); + let prefix = |bytes, child: Option| { + let mut prefix = Prefix::default(); + prefix.publisher.bytes = bytes; + prefix.own.publisher.bytes = bytes; + if let Some(child) = child { + let mut usage = Usage::default(); + usage.publisher.bytes = child; + prefix.children.insert("cam".into(), usage); + } + prefix + }; + let node = merged.nodes.get_mut(&key).expect("node"); + node.observe(BTreeMap::from([(String::new(), prefix(10, Some(4)))])); + assert_eq!(merged.merged()[""], prefix(10, Some(4))); + + // A child leaving the map is not a reset. + let node = merged.nodes.get_mut(&key).expect("node"); + node.observe(BTreeMap::from([(String::new(), prefix(10, None))])); + assert_eq!(merged.merged()[""], prefix(10, None)); + + // Held again from zero: the old rollup is a baseline, the children are + // only the live ones. + let node = merged.nodes.get_mut(&key).expect("node"); + node.observe(BTreeMap::from([(String::new(), prefix(3, Some(1)))])); + assert_eq!(merged.merged()[""], { + let mut expected = prefix(13, Some(1)); + expected.own.publisher.bytes = 13; + expected + }); + + // A departed node keeps its rollup and drops its children. + merged.nodes.get_mut(&key).expect("node").depart(); + assert_eq!(merged.merged()[""], { + let mut expected = prefix(13, None); + expected.own.publisher.bytes = 13; + expected + }); + } } diff --git a/rs/moq-stats/src/consume.rs b/rs/moq-stats/src/consume.rs index 514360b2cf8..275c5be0c1f 100644 --- a/rs/moq-stats/src/consume.rs +++ b/rs/moq-stats/src/consume.rs @@ -4,8 +4,8 @@ use moq_net::stats::{Role, Tier}; use moq_net::{AsPath, broadcast}; use crate::{ - PresenceFrame, Result, SessionsFrame, TotalsFrame, TrafficFrame, presence_track, sessions_track, totals_track, - traffic_track, + PrefixFrame, PresenceFrame, Result, SessionsFrame, TotalsFrame, TrafficFrame, prefix_track, presence_track, + sessions_track, totals_track, traffic_track, }; /// Configuration for a [`Consumer`]. Construct with [`Config::new`] @@ -36,9 +36,10 @@ impl Config { /// yielding typed frames per track. /// /// Subscribe to the tracks you care about with [`Self::totals`], -/// [`Self::traffic`], [`Self::sessions`], and [`Self::presence`]. A named -/// tier's tracks are held open with an empty frame until the tier records, and -/// a root's presence track is produced only while subscribed. +/// [`Self::traffic`], [`Self::sessions`], [`Self::presence`], and +/// [`Self::prefix`]. A named tier's tracks are held open with an empty frame +/// until the tier records, and a root's presence track and a prefix's track +/// are produced only while subscribed. pub struct Consumer { broadcast: broadcast::Consumer, config: Config, @@ -87,6 +88,16 @@ impl Consumer { }) } + /// Subscribe to the prefix track for broadcast prefix `path`, awaiting the + /// subscription handshake. Fails with + /// [`NotFound`](moq_net::Error::NotFound) for a prefix outside the group. + pub async fn prefix(&self, path: impl AsPath) -> Result { + let name = prefix_track(path, self.config.compression); + Ok(Prefix { + inner: self.subscribe(&name).await?, + }) + } + async fn subscribe(&self, name: &str) -> Result> { let track = self.broadcast.track(name)?.subscribe(None).await?; let mut config = moq_json::snapshot::consumer::Config::default(); @@ -148,6 +159,21 @@ impl Presence { } } +/// A typed reader over one broadcast prefix's track; see [`Traffic`]. +pub struct Prefix { + inner: moq_json::snapshot::Consumer, +} + +impl Prefix { + /// The next frame, or `None` once the track ends (the producer went away + /// or its group unannounced). Fails with + /// [`FrameTooLarge`](moq_net::Error::FrameTooLarge) once the frame outgrows + /// the cache. + pub async fn next(&mut self) -> Result> { + Ok(self.inner.next().await?) + } +} + #[cfg(test)] mod tests { /// A millisecond track: the frames written here carry timestamps. @@ -313,5 +339,15 @@ mod tests { let mut presence = presence.await.expect("subscribe presence"); let frame = presence.next().await.expect("read").expect("frame"); assert_eq!(frame.get("").expect("default tier").active(), 1); + + // So is a prefix track. + let mut prefix = std::pin::pin!(compressed.prefix("foo")); + assert!(futures::poll!(prefix.as_mut()).is_pending()); + drive_tick().await; + let mut prefix = prefix.await.expect("subscribe prefix"); + let frame = prefix.next().await.expect("read").expect("frame"); + let tier = frame.get("").expect("default tier"); + assert_eq!(tier.publisher.bytes, 50); + assert_eq!(tier.children["bar"].publisher.bytes, 50); } } diff --git a/rs/moq-stats/src/lib.rs b/rs/moq-stats/src/lib.rs index ab7dacdce5a..90eef920c76 100644 --- a/rs/moq-stats/src/lib.rs +++ b/rs/moq-stats/src/lib.rs @@ -24,10 +24,11 @@ //! `totals.json` track of each [`Tier`]'s [`Totals`] for the epoch, each tier //! carries `publisher.json` and `subscriber.json` tracks of per-path //! [`Traffic`] and a `sessions.json` track of per-root [`Presence`], and a -//! reader may request any auth root's `/presence.json` track. Every +//! reader may request any auth root's `/presence.json` track and any +//! broadcast prefix's `/traffic.json` track of [`Prefix`] counters. Every //! track has a `.z` sibling encoded with [`moq_json::snapshot`]; compute names -//! with [`totals_track`], [`traffic_track`], [`sessions_track`], and -//! [`presence_track`]. The full +//! with [`totals_track`], [`traffic_track`], [`sessions_track`], +//! [`presence_track`], and [`prefix_track`]. The full //! contract (paths, tracks, both encodings, and counter semantics) is at //! . @@ -60,6 +61,10 @@ pub type TotalsFrame = BTreeMap; /// (`""` for the default tier). pub type PresenceFrame = BTreeMap; +/// One frame off a prefix track: one broadcast prefix's [`Prefix`] counters, +/// keyed by tier label (`""` for the default tier). +pub type PrefixFrame = BTreeMap; + /// One tier's cumulative counters for a whole group: everything its entries /// sent, received, and connected since the group's epoch began, including /// entries that have since ended. @@ -84,6 +89,43 @@ impl Totals { } } +/// Egress and ingress traffic for one part of a prefix. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(default)] +#[non_exhaustive] +pub struct Usage { + /// Egress: what this node sent. + pub publisher: Traffic, + /// Ingress: what this node received. + pub subscriber: Traffic, +} + +impl Usage { + pub(crate) fn add(&mut self, other: Usage) { + self.publisher.add(other.publisher); + self.subscriber.add(other.subscriber); + } +} + +/// One tier's counters on a prefix track: the rollup of every broadcast path +/// at or under the prefix, the paths at exactly the prefix, and the rollup of +/// each direct child. The rollup sits at the top level, so a [`Totals`] reader +/// parses it unchanged. +#[derive(Debug, Default, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(default)] +#[non_exhaustive] +pub struct Prefix { + /// Egress under the prefix. + pub publisher: Traffic, + /// Ingress under the prefix. + pub subscriber: Traffic, + /// The broadcast at exactly the prefix, apart from any nested under it. + #[serde(rename = "self")] + pub own: Usage, + /// Each direct child's rollup, keyed by its next path segment. + pub children: BTreeMap, +} + /// Suffix appended to a plain track name for its compressed sibling. pub const COMPRESSED_SUFFIX: &str = ".z"; @@ -93,6 +135,9 @@ pub(crate) const TOTALS: &str = "totals.json"; /// The last segment of a presence track's plain name. const PRESENCE: &str = "presence.json"; +/// The last segment of a prefix track's plain name. +const PREFIX: &str = "traffic.json"; + /// The traffic track name for a tier and role: `.json` at the prefix root /// on the default tier (`publisher.json` / `subscriber.json`), `/.json` /// on a named one, plus [`COMPRESSED_SUFFIX`] when `compressed`. @@ -129,10 +174,21 @@ pub fn totals_track(compressed: bool) -> String { /// `presence.json` for the empty root, plus [`COMPRESSED_SUFFIX`] when /// `compressed`. Served only while requested. pub fn presence_track(root: impl AsPath, compressed: bool) -> String { - let root = root.as_path(); - let mut name = match root.is_empty() { - true => PRESENCE.to_string(), - false => format!("{}/{PRESENCE}", root.as_str()), + keyed_track(root.as_path(), PRESENCE, compressed) +} + +/// The prefix track name for a broadcast prefix: `/traffic.json`, or +/// `traffic.json` for the empty prefix, plus [`COMPRESSED_SUFFIX`] when +/// `compressed`. Served only while requested. +pub fn prefix_track(path: impl AsPath, compressed: bool) -> String { + keyed_track(path.as_path(), PREFIX, compressed) +} + +/// `/`, or `` for the empty key, plus the compressed suffix. +fn keyed_track(key: Path, kind: &str, compressed: bool) -> String { + let mut name = match key.is_empty() { + true => kind.to_string(), + false => format!("{}/{kind}", key.as_str()), }; if compressed { name.push_str(COMPRESSED_SUFFIX); @@ -144,17 +200,30 @@ pub fn presence_track(root: impl AsPath, compressed: bool) -> String { /// compressed flavor; `None` for any other name. The inverse of /// [`presence_track`]. pub(crate) fn parse_presence_track(name: &str) -> Option<(PathOwned, bool)> { + parse_keyed_track(name, PRESENCE) +} + +/// The broadcast prefix a prefix track name requests, and whether it is the +/// compressed flavor; `None` for any other name. The inverse of +/// [`prefix_track`]. +pub(crate) fn parse_prefix_track(name: &str) -> Option<(PathOwned, bool)> { + parse_keyed_track(name, PREFIX) +} + +/// The key a `/` name requests: the last segment picks the kind and +/// everything before it is the key, so names stay injective across kinds. +fn parse_keyed_track(name: &str, kind: &str) -> Option<(PathOwned, bool)> { let (plain, compressed) = match name.strip_suffix(COMPRESSED_SUFFIX) { Some(plain) => (plain, true), None => (name, false), }; - let root = match plain.strip_suffix(PRESENCE)? { + let key = match plain.strip_suffix(kind)? { "" => "", - rest => rest.strip_suffix('/').filter(|root| !root.is_empty())?, + rest => rest.strip_suffix('/').filter(|key| !key.is_empty())?, }; - // Only a normalized root round-trips, so each root has exactly one name. - let path = Path::new(root); - if path.as_str() != root { + // Only a normalized key round-trips, so each key has exactly one name. + let path = Path::new(key); + if path.as_str() != key { return None; } Some((path.to_owned(), compressed)) @@ -291,6 +360,8 @@ mod tests { assert_eq!(totals_track(true), "totals.json.z"); assert_eq!(presence_track("acme/live", false), "acme/live/presence.json"); assert_eq!(presence_track("", true), "presence.json.z"); + assert_eq!(prefix_track("acme/live", false), "acme/live/traffic.json"); + assert_eq!(prefix_track("", true), "traffic.json.z"); } #[test] @@ -324,4 +395,55 @@ mod tests { assert_eq!(parse_presence_track(name), None, "{name}"); } } + + #[test] + fn prefix_track_names_round_trip() { + for path in [ + "", + "acme", + "acme/live", + "a.json", + "x/traffic.json", + "x/presence.json", + "publisher", + ] { + for compressed in [false, true] { + let name = prefix_track(path, compressed); + assert_eq!( + parse_prefix_track(&name), + Some((PathOwned::from(path), compressed)), + "{name}" + ); + assert_eq!(parse_presence_track(&name), None, "{name} is not a presence track"); + } + } + + for name in [ + "publisher.json", + "acme/publisher.json", + "presence.json", + "acme/presence.json", + "totals.json", + "xtraffic.json", + "/traffic.json", + "acme//traffic.json", + "traffic.json.z.z", + ] { + assert_eq!(parse_prefix_track(name), None, "{name}"); + } + } + + #[test] + fn prefix_entry_reads_as_totals() { + let mut prefix = Prefix::default(); + prefix.publisher.bytes = 7; + prefix.own.publisher.bytes = 3; + prefix.children.insert("cam".into(), Usage::default()); + let json = serde_json::to_value(&prefix).unwrap(); + assert_eq!(json["self"]["publisher"]["bytes"], 3); + assert!(json["children"]["cam"]["subscriber"].is_object()); + let totals: Totals = serde_json::from_value(json.clone()).unwrap(); + assert_eq!(totals.publisher.bytes, 7, "a totals reader reads the rollup"); + assert_eq!(serde_json::from_value::(json).unwrap(), prefix); + } } diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs index ad348957a2c..c97faebfb26 100644 --- a/rs/moq-stats/src/produce.rs +++ b/rs/moq-stats/src/produce.rs @@ -7,14 +7,15 @@ use std::time::Duration; use std::task::Poll; -use moq_net::stats::{Presence, Registry, Report, Role, Tier, Traffic}; +use moq_net::stats::{Presence, Registry, Report, Role, Tier, Traffic, TrafficEntry}; use moq_net::{Epoch, Path, PathOwned, broadcast, kio, origin, track}; use serde::Serialize; use web_async::spawn; use web_async::time::Instant; use crate::{ - COMPRESSED_SUFFIX, Totals, parse_presence_track, presence_track, sessions_track, totals_track, traffic_track, + COMPRESSED_SUFFIX, Prefix, Totals, Usage, parse_prefix_track, parse_presence_track, prefix_track, presence_track, + sessions_track, totals_track, traffic_track, }; /// Settings for a [`Producer`]. Construct with [`Config::new`] and chain @@ -381,7 +382,7 @@ impl Drain { // Serve consumer requests for tracks no drain has created yet: a // tier's tracks appear lazily on its first traffic, and a root's // presence track only while requested. See `serve_requests`. - group.serve_requests(&self.report, depth); + group.serve_requests(&self.report, depth, self.tick); } } @@ -503,6 +504,27 @@ impl Snapshot { Ok(Some(len)) } + /// [`Self::update`], logging a failure. With `abort_oversized`, a frame + /// over the cache limit ends the track with `FrameTooLarge` instead, so its + /// readers learn it stopped rather than sit on a stale snapshot. + fn publish(&mut self, value: &Frame, name: &str, abort_oversized: bool) -> Option { + if self.track.is_closed() { + return None; + } + match self.update(value) { + Ok(len) => len, + Err(moq_json::Error::Net(moq_net::Error::FrameTooLarge)) if abort_oversized => { + tracing::warn!(name, "stats: frame too large, ending the track"); + let _ = self.track.clone().abort(moq_net::Error::FrameTooLarge); + None + } + Err(err) => { + tracing::debug!(?err, name, "stats: failed to write frame"); + None + } + } + } + fn finish(&mut self) -> moq_json::Result<()> { if let Some(group) = self.group.take() { group.finish()?; @@ -520,6 +542,8 @@ struct TrackPair { compressed: Snapshot, /// This drain's entries, published and cleared by [`Self::publish`]. frame: Frame, + /// End a flavor whose frame outgrows the cache instead of logging it. + abort_oversized: bool, /// Last plain payload written, kept so a later unchanged tick can still /// report the snapshot size. Bench builds only. #[cfg(feature = "bench")] @@ -553,6 +577,7 @@ impl TrackPair { plain: Snapshot::new(plain_track, plain_config, sequence.clone()), compressed: Snapshot::new(compressed_track, compressed_config, sequence), frame: Frame::default(), + abort_oversized: false, #[cfg(feature = "bench")] held: 0, #[cfg(feature = "bench")] @@ -571,20 +596,10 @@ impl TrackPair { /// and clear them for the next drain; moq-json skips unchanged values. fn publish(&mut self, name: &str) { self.frame.entries.sort_unstable_by(|a, b| a.0.cmp(&b.0)); - let plain = match self.plain.update(&self.frame) { - Ok(len) => len, - Err(err) => { - tracing::debug!(?err, name, "stats: failed to write frame"); - None - } - }; - let compressed = match self.compressed.update(&self.frame) { - Ok(len) => len, - Err(err) => { - tracing::debug!(?err, name, "stats: failed to write compressed frame"); - None - } - }; + // Each flavor fails alone: DEFLATE can shrink a plain frame that + // overflows, or slightly grow one that fits. + let plain = self.plain.publish(&self.frame, name, self.abort_oversized); + let compressed = self.compressed.publish(&self.frame, name, self.abort_oversized); self.record_bench(plain, compressed); self.frame.entries.clear(); } @@ -857,6 +872,9 @@ struct GroupPublisher { totals: TotalsTrack, /// Held presence tracks, keyed by auth root. presence: HashMap, + /// Held prefix tracks, keyed by broadcast prefix. A `String` key, so a + /// drain looks each row's ancestors up by `&str` without allocating. + prefixes: HashMap, /// This drain's entries for the group, as indices into the report. traffic_rows: Vec, session_rows: Vec, @@ -897,6 +915,22 @@ struct Sum { current: V, } +impl Sum { + /// Start a drain: the value is its folded part until the drain's rows are + /// added. + fn reset(&mut self) { + self.current = self.folded; + } + + /// Add one readout, folding it for good when it is `closed`. + fn add(&mut self, closed: bool, add: impl Fn(&mut V)) { + add(&mut self.current); + if closed { + add(&mut self.folded); + } + } +} + impl Cumulative { fn new() -> Self { Self { tiers: HashMap::new() } @@ -906,21 +940,13 @@ impl Cumulative { /// rows are added. fn reset(&mut self) { for sum in self.tiers.values_mut() { - sum.current = sum.folded; + sum.reset(); } } /// Add one readout to `tier`, folding it for good when it is `closed`. fn add(&mut self, tier: &Tier, closed: bool, add: impl Fn(&mut V)) { - // Look up before inserting, so a steady drain never clones a tier. - if !self.tiers.contains_key(tier) { - self.tiers.insert(tier.clone(), Sum::default()); - } - let sum = self.tiers.get_mut(tier).expect("just ensured"); - add(&mut sum.current); - if closed { - add(&mut sum.folded); - } + tier_entry(&mut self.tiers, tier).add(closed, add); } /// Fill `frame` with each tier's current value. @@ -931,6 +957,15 @@ impl Cumulative { } } +/// `tier`'s value in `tiers`, inserted on first sight. Looks up before +/// inserting, so a steady drain never clones a tier. +fn tier_entry<'a, V: Default>(tiers: &'a mut HashMap, tier: &Tier) -> &'a mut V { + if !tiers.contains_key(tier) { + tiers.insert(tier.clone(), V::default()); + } + tiers.get_mut(tier).expect("just ensured") +} + /// The always-present `totals.json` pair and its per-tier sums. Never pruned /// while the group broadcast lives, so it is cumulative within the epoch. struct TotalsTrack { @@ -947,6 +982,107 @@ struct PresenceTrack { sums: Cumulative, } +/// One broadcast prefix's requested `traffic.json` pair and its per-tier sums, +/// cumulative from when it was first held. Reclaimed once nobody reads it. +struct PrefixTrack { + /// The plain track name, for logs. + name: String, + pair: TrackPair, + tiers: HashMap, +} + +/// One tier's sums on a [`PrefixTrack`]. +#[derive(Default)] +struct PrefixSums { + /// Every path at or under the prefix. + rollup: Sum, + /// The path at exactly the prefix. + own: Sum, + /// Keyed by the child's next path segment. A child is dropped on the first + /// drain with no row under it, so its folded tail stays only in `rollup`. + children: HashMap, +} + +#[derive(Default)] +struct ChildSum { + sum: Sum, + /// The last drain that carried a row under this child. + seen: u64, +} + +impl PrefixTrack { + fn reset(&mut self) { + for sums in self.tiers.values_mut() { + sums.rollup.reset(); + sums.own.reset(); + for child in sums.children.values_mut() { + child.sum.reset(); + } + } + } + + /// Add one row at or under the prefix: at it when `child` is `None`, else + /// under that direct child. + fn add(&mut self, entry: &TrafficEntry, child: Option<&str>, tick: u64) { + let usage = Usage { + publisher: entry.publisher, + subscriber: entry.subscriber, + }; + let add = |sum: &mut Usage| sum.add(usage); + let sums = tier_entry(&mut self.tiers, &entry.tier); + sums.rollup.add(entry.closed, add); + let Some(child) = child else { + sums.own.add(entry.closed, add); + return; + }; + if !sums.children.contains_key(child) { + sums.children.insert(child.to_string(), ChildSum::default()); + } + let state = sums.children.get_mut(child).expect("just ensured"); + state.seen = tick; + state.sum.add(entry.closed, add); + } + + /// Publish each tier's sums, dropping children this drain carried no row + /// for. + fn publish(&mut self, tick: u64) { + for (tier, sums) in self.tiers.iter_mut() { + sums.children.retain(|_, child| child.seen == tick); + self.pair.frame.entries.push(( + tier.label().clone(), + Prefix { + publisher: sums.rollup.current.publisher, + subscriber: sums.rollup.current.subscriber, + own: sums.own.current, + children: sums + .children + .iter() + .map(|(segment, child)| (segment.clone(), child.sum.current)) + .collect(), + }, + )); + } + self.pair.publish(&self.name); + } +} + +/// Each prefix of `path` from the root down to `path` itself, paired with the +/// next segment below it (`None` at `path`). +fn ancestors(path: &str) -> impl Iterator)> { + let mut at = Some(0); + std::iter::from_fn(move || { + let end = at?; + if end == path.len() { + at = None; + return Some((path, None)); + } + let start = if end == 0 { 0 } else { end + 1 }; + let next = path[start..].find('/').map_or(path.len(), |i| start + i); + at = Some(next); + Some((&path[..end], Some(&path[start..next]))) + }) +} + impl GroupPublisher { fn create(task: &Task, group: &Path) -> Option { let advertised = advertised_path(&task.prefix, group, task.node()); @@ -1015,6 +1151,7 @@ impl GroupPublisher { names: HashMap::new(), totals, presence: HashMap::new(), + prefixes: HashMap::new(), traffic_rows: Vec::new(), session_rows: Vec::new(), empty_since: None, @@ -1033,7 +1170,7 @@ impl GroupPublisher { } /// Run this drain's rows through change detection into the pending frames, - /// and add them to the totals and every held presence track. + /// and add them to the totals and every held presence and prefix track. fn collect(&mut self, report: &Report, tick: u64) { let Self { broadcast, @@ -1044,11 +1181,28 @@ impl GroupPublisher { names, totals, presence, + prefixes, traffic_rows, session_rows, .. } = self; + // Each row looks its ancestors up in the held set, so the cost is rows + // times path depth, and nothing when no prefix is held. + if !prefixes.is_empty() { + for track in prefixes.values_mut() { + track.reset(); + } + for &i in traffic_rows.iter() { + let entry = &report.traffic[i]; + for (prefix, child) in ancestors(entry.path.as_str()) { + if let Some(track) = prefixes.get_mut(prefix) { + track.add(entry, child, tick); + } + } + } + } + totals.sums.reset(); for &i in traffic_rows.iter() { let entry = &report.traffic[i]; @@ -1119,6 +1273,9 @@ impl GroupPublisher { track.sums.fill(&mut track.pair.frame); track.pair.publish(&track.name); } + for track in self.prefixes.values_mut() { + track.publish(tick); + } self.local.retain(|_, tiers| { tiers.retain(|_, slots| slots.seen == tick); @@ -1140,8 +1297,9 @@ impl GroupPublisher { /// A presence track is held only while requested, for a root in this group, /// and answered at once with the root's current counters from `report`. /// It is reclaimed once its last consumer leaves, so the held set follows - /// live readers. Any other name is rejected. - fn serve_requests(&mut self, report: &Report, depth: usize) { + /// live readers. A prefix track follows the same rules for a broadcast + /// prefix in this group. Any other name is rejected. + fn serve_requests(&mut self, report: &Report, depth: usize, tick: u64) { // Reclaim before parking and adopting, so a freed slot is usable by // this very drain. self.traffic.reclaim(&mut self.requested); @@ -1153,19 +1311,34 @@ impl GroupPublisher { track.pair.finish(); false }); + self.prefixes.retain(|_, track| { + if track.pair.is_used() { + return true; + } + track.pair.finish(); + false + }); // Pop everything queued, grouping the two flavors of one plain name so // a pair is built from the actual requests where present. Only tier // names past the parked bound are rejected. let mut roots: HashMap = HashMap::new(); + let mut paths: HashMap = HashMap::new(); let noop = kio::Waiter::noop(); while let Poll::Ready(Ok(request)) = self.dynamic.poll_requested_track(&noop) { - if let Some((root, compressed)) = parse_presence_track(request.name()) { - if group_key(root.as_str(), depth) != self.key.as_str() { + let keyed = match parse_presence_track(request.name()) { + Some((root, compressed)) => Some((&mut roots, root, compressed)), + None => parse_prefix_track(request.name()) + // No broadcast path is deeper than `MAX_PARTS`. + .filter(|(path, _)| path.parts().count() <= Path::MAX_PARTS) + .map(|(path, compressed)| (&mut paths, path, compressed)), + }; + if let Some((held, key, compressed)) = keyed { + if group_key(key.as_str(), depth) != self.key.as_str() { request.reject(moq_net::Error::NotFound); continue; } - let pending = roots.entry(root).or_default(); + let pending = held.entry(key).or_default(); let slot = match compressed { true => &mut pending.compressed, false => &mut pending.plain, @@ -1192,6 +1365,9 @@ impl GroupPublisher { for (root, pending) in roots { self.adopt_presence(report, root, pending); } + for (path, pending) in paths { + self.adopt_prefix(report, path, pending, tick); + } } /// Hold a requested root's presence track, publishing its current counters @@ -1227,6 +1403,40 @@ impl GroupPublisher { self.presence.insert(root, track); } + /// Hold a requested prefix track, publishing its current counters so the + /// subscription resolves at once. Cumulative from here on. + fn adopt_prefix(&mut self, report: &Report, path: PathOwned, pending: PendingPair, tick: u64) { + // Defensive only, as in `adopt_presence`. + if self.prefixes.contains_key(path.as_str()) { + pending.reject(moq_net::Error::NotFound); + return; + } + let name = prefix_track(&path, false); + let mut pair = match pending.adopt(&self.broadcast, &name, self.traffic.sequence.clone()) { + Ok(pair) => pair, + Err(err) => { + tracing::warn!(?err, name, "stats: failed to adopt prefix track"); + return; + } + }; + pair.abort_oversized = true; + let mut track = PrefixTrack { + name, + pair, + tiers: HashMap::new(), + }; + for &i in &self.traffic_rows { + let entry = &report.traffic[i]; + let Some(rest) = entry.path.strip_prefix(&path) else { + continue; + }; + let child = rest.as_str().split('/').next().filter(|child| !child.is_empty()); + track.add(entry, child, tick); + } + track.publish(tick); + self.prefixes.insert(path.as_str().to_string(), track); + } + /// Deliberately end the broadcast: finish every pair, then close the broadcast. fn finish(mut self) { self.traffic.finish(); @@ -1235,6 +1445,9 @@ impl GroupPublisher { for track in self.presence.values_mut() { track.pair.finish(); } + for track in self.prefixes.values_mut() { + track.pair.finish(); + } self.broadcast.close(); } } @@ -1426,7 +1639,7 @@ mod tests { use moq_net::{Timestamp, announce, broadcast, track}; use super::*; - use crate::{PresenceFrame, TotalsFrame}; + use crate::{PrefixFrame, PresenceFrame, TotalsFrame}; fn test_producer(node: Option<&str>) -> (Producer, origin::Producer) { let origin = produce_origin(); @@ -2292,6 +2505,280 @@ mod tests { subscribing.await.expect("the compressed sibling of a held root"); } + /// Subscribe to `path`'s prefix track on `broadcast`, driving a tick so the + /// producer adopts the request. + async fn hold_prefix(broadcast: &moq_net::broadcast::Consumer, path: &str) -> track::Ordered { + let subscribing = broadcast + .track(&prefix_track(path, false)) + .expect("track") + .subscribe(None); + drive_tick().await; + subscribing.await.expect("held").ordered() + } + + /// The latest buffered prefix frame, `None` when nothing is buffered. + fn last_prefix(track: &mut track::Ordered) -> Option { + let mut last = None; + while let Some(frame) = try_next_frame(track) { + last = Some(serde_json::from_slice(&frame.payload).expect("json parse")); + } + last + } + + #[test] + fn ancestors_walk_from_the_root() { + let walk = |path| ancestors(path).collect::>(); + assert_eq!(walk(""), [("", None)]); + assert_eq!(walk("a"), [("", Some("a")), ("a", None)]); + assert_eq!( + walk("a/bc/d"), + [ + ("", Some("a")), + ("a", Some("bc")), + ("a/bc", Some("d")), + ("a/bc/d", None) + ] + ); + } + + /// A held prefix carries its rollup, the broadcast at exactly the prefix, + /// and one level of children, each summing everything beneath it. + #[tokio::test(start_paused = true)] + async fn prefix_track_rolls_up_self_and_children() { + let (producer, origin, _events) = grouped_producer(DEFAULT_LINGER); + let registry = producer.registry(); + let _own = feed(registry, Tier::default(), "acme", true, 1, 1).await; + let _live = feed(registry, Tier::default(), "acme/live", true, 1, 10).await; + let _cam = feed(registry, Tier::default(), "acme/live/cam", true, 1, 100).await; + let _rtmp = feed(registry, Tier::new("rtmp"), "acme/vod", true, 1, 1000).await; + let _other = feed(registry, Tier::default(), "beta/live", true, 1, 5).await; + drive_tick().await; + let acme = origin.consume().request_broadcast(ACME, None).await.expect("resolve"); + + let mut track = hold_prefix(&acme, "acme").await; + let frame = last_prefix(&mut track).expect("answered at once"); + let default = &frame[""]; + assert_eq!(default.publisher.bytes, 111); + assert_eq!(default.own.publisher.bytes, 1); + assert_eq!(default.children.len(), 1); + assert_eq!( + default.children["live"].publisher.bytes, 110, + "a child rolls up its subtree" + ); + assert_eq!(frame["rtmp"].publisher.bytes, 1000, "every tier the prefix records on"); + assert_eq!(frame["rtmp"].children["vod"].publisher.bytes, 1000); + assert_eq!(frame.len(), 2); + + let mut track = hold_prefix(&acme, "acme/live/cam").await; + let frame = last_prefix(&mut track).expect("answered at once"); + assert_eq!(frame[""].own.publisher.bytes, 100); + assert!(frame[""].children.is_empty()); + + // A prefix outside the group, or deeper than any path, is refused. + for path in [ + "beta".to_string(), + "".to_string(), + ["acme"; Path::MAX_PARTS + 1].join("/"), + ] { + let subscribing = acme + .track(&prefix_track(path.as_str(), false)) + .expect("track") + .subscribe(None); + drive_tick().await; + assert!( + matches!(subscribing.await, Err(moq_net::Error::NotFound)), + "{path} is refused" + ); + } + } + + /// Counters follow presence: a prefix held while 300 subscriptions are + /// live counts all 300 as active, and every one leaving reads as ended. + /// The child leaves the map after its closing readout; its tail stays in + /// the rollup. + #[tokio::test(start_paused = true)] + async fn prefix_held_mid_stream_counts_active_exactly() { + let (producer, origin) = test_producer(Some("sjc")); + let registry = producer.registry(); + let data = produce_origin(); + let source = data + .clone() + .with_stats(registry.tier(Tier::default()).session("publisher")) + .publish("acme/live", origin::Route::default()) + .expect("publish"); + let video = source.create_track("video", timed()).expect("create_track"); + let egress = data + .consume() + .with_stats(registry.tier(Tier::default()).session("viewer")); + let broadcast = egress.request_broadcast("acme/live", None).await.expect("resolve"); + let mut subs = Vec::new(); + for _ in 0..300 { + let track = broadcast.track("video").expect("track"); + subs.push(track.subscribe(None).await.expect("subscribe")); + } + { + let mut group = video.append_group().expect("group"); + group.write_frame(Timestamp::ZERO, vec![0u8; 10]).expect("write"); + group.finish().expect("finish"); + } + drive_tick().await; + let (_, stats) = announced(&origin).await; + + let mut track = hold_prefix(&stats, "acme").await; + let frame = last_prefix(&mut track).expect("answered at once"); + assert_eq!(frame[""].publisher.active_subscriptions(), 300); + assert_eq!(frame[""].children["live"].publisher.active_subscriptions(), 300); + + drop(subs); + drive_tick().await; + let frame = last_prefix(&mut track).expect("the leave"); + assert_eq!(frame[""].publisher.subscriptions_started, 300); + assert_eq!(frame[""].publisher.active_subscriptions(), 0); + + // The broadcast ends: once its counters are dropped the child leaves the + // map, and the rollup keeps everything. + let before = frame[""].publisher; + drop((video, source, broadcast)); + for _ in 0..3 { + drive_tick().await; + } + let frame = last_prefix(&mut track).expect("the child leaves"); + assert!(frame[""].children.is_empty(), "{frame:?}"); + assert_eq!(frame[""].publisher.subscriptions_started, before.subscriptions_started); + assert_eq!(frame[""].publisher.bytes, before.bytes); + assert_eq!(frame[""].publisher.active_subscriptions(), 0); + } + + /// A nested broadcast ending leaves the parent's own counters flat, while + /// its rollup keeps the nested broadcast's tail. + #[tokio::test(start_paused = true)] + async fn nested_broadcast_ending_keeps_the_parent_self_flat() { + let (producer, origin) = test_producer(Some("sjc")); + let registry = producer.registry(); + let _parent = feed(registry, Tier::default(), "acme/live", true, 1, 10).await; + let nested = feed(registry, Tier::default(), "acme/live/cam", true, 1, 100).await; + drive_tick().await; + let (_, stats) = announced(&origin).await; + + let mut track = hold_prefix(&stats, "acme/live").await; + let frame = last_prefix(&mut track).expect("answered at once"); + let own = frame[""].own; + assert_eq!(own.publisher.bytes, 10); + assert_eq!(frame[""].publisher.bytes, 110); + + drop(nested); + for _ in 0..3 { + drive_tick().await; + } + let frame = last_prefix(&mut track).expect("the nested broadcast ended"); + assert!(frame[""].children.is_empty(), "{frame:?}"); + assert_eq!(frame[""].own, own, "the parent's own counters stay flat"); + assert_eq!(frame[""].publisher.bytes, 110, "the rollup keeps the nested tail"); + assert_eq!(frame[""].publisher.active_subscriptions(), 1, "the parent's viewer"); + } + + /// A prefix with no members is held open at zero, carries its first + /// member on the same subscription, and is reclaimed once its last + /// consumer leaves. + #[tokio::test(start_paused = true)] + async fn prefix_track_for_an_idle_prefix_is_held_then_reclaimed() { + let registry = Registry::new(moq_net::stats::Config::new()); + let origin = produce_origin(); + let mut drain = Drain::new(Task { + registry: registry.clone(), + origin: origin.clone(), + prefix: PathOwned::from(".stats"), + node: None, + depth: 0, + linger: DEFAULT_LINGER, + interval: Duration::from_secs(1), + }) + .expect("drain"); + let mut tick = || { + drain.collect(); + drain.publish(Instant::now()); + drain.groups[""].prefixes.len() + }; + let broadcast = origin + .consume() + .request_broadcast(".stats/node", None) + .await + .expect("resolve"); + + let subscribing = broadcast + .track(&prefix_track("acme", false)) + .expect("track") + .subscribe(None); + for _ in 0..8 { + tokio::task::yield_now().await; + } + assert_eq!(tick(), 1, "held"); + let mut track = subscribing.await.expect("held").ordered(); + assert_eq!(last_prefix(&mut track), Some(PrefixFrame::new()), "held at zero"); + + let _feed = feed(®istry, Tier::default(), "acme/live", true, 1, 7).await; + tick(); + let frame = last_prefix(&mut track).expect("recorded"); + assert_eq!(frame[""].children["live"].publisher.bytes, 7); + + drop(track); + for _ in 0..4 { + tokio::task::yield_now().await; + } + tokio::time::advance(Duration::from_secs(31)).await; + for _ in 0..4 { + tokio::task::yield_now().await; + } + assert_eq!(tick(), 0, "reclaimed"); + } + + /// A prefix frame past the cache limit ends that flavor with + /// `FrameTooLarge` instead of leaving its readers on a stale snapshot, + /// while the sibling that still fits keeps publishing. + #[tokio::test(start_paused = true)] + async fn oversized_prefix_frame_ends_the_flavor() { + let broadcast = broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let mut pair = TrackPair::::create(&broadcast, "traffic.json", Arc::new(AtomicU64::new(0))).unwrap(); + pair.abort_oversized = true; + let mut plain = subscribe(&consumer, "traffic.json").await; + let mut compressed = consumer + .track("traffic.json.z") + .unwrap() + .subscribe(None) + .await + .unwrap() + .ordered(); + + // One child whose name alone outgrows the cache: DEFLATE shrinks it to + // almost nothing, so only the plain flavor overflows. + let huge = "a".repeat(moq_net::group::MAX_CACHE_BYTES as usize + 1); + let mut prefix = Prefix::default(); + prefix.children.insert(huge, Usage::default()); + pair.frame.entries.push((PathOwned::from(""), prefix.clone())); + pair.publish("traffic.json"); + + let err = loop { + match plain.next_group().await { + Ok(Some(_)) => continue, + Ok(None) => panic!("ended cleanly"), + Err(err) => break err, + } + }; + assert!(matches!(err, moq_net::Error::FrameTooLarge), "{err:?}"); + let mut group = compressed.next_group().await.expect("ok").expect("group"); + group + .read_frame() + .await + .expect("ok") + .expect("the compressed flavor fits"); + + prefix.publisher.bytes = 1; + pair.frame.entries.push((PathOwned::from(""), prefix)); + pair.publish("traffic.json"); + group.read_frame().await.expect("ok").expect("and keeps updating"); + } + #[tokio::test(start_paused = true)] async fn unused_slots_dont_surface() { // A broadcast that only sees default-tier publisher traffic must NOT @@ -2652,8 +3139,9 @@ mod tests { async fn steady_drain_collects_without_allocating() { // Once every group, track, and buffer exists, draining the registry // into the pending frames allocates nothing, however many broadcasts - // and tiers there are. Encoding the frames (moq-json) is out of scope. - for depth in [0, 1] { + // and tiers there are, or whether a prefix is held. Encoding the frames + // (moq-json) is out of scope. + for (depth, hold) in [(0, false), (1, false), (0, true), (1, true)] { for (broadcasts, tiers) in [(1, 1), (16, 1), (1, 4), (16, 4)] { let registry = Registry::new(moq_net::stats::Config::new()); let mut feeds = Vec::new(); @@ -2683,6 +3171,15 @@ mod tests { drain.collect(); drain.publish(Instant::now()); } + // Hold a prefix with a child per tier, as a reader would. + let _held = hold.then(|| { + let tick = drain.tick; + let group = drain.groups.get_mut(group_key("room0/cam", depth)).expect("group"); + group.adopt_prefix(&drain.report, PathOwned::from("room0"), PendingPair::default(), tick); + group.prefixes["room0"].pair.plain.track.subscribe(None) + }); + drain.collect(); + drain.publish(Instant::now()); let before = counting::allocs(); drain.collect(); @@ -2695,7 +3192,17 @@ mod tests { .map(|group| group.traffic_rows.len() + group.session_rows.len()) .sum(); assert!(pending > 0, "the drain carried entries"); - assert_eq!(allocs, 0, "depth {depth}, {broadcasts} broadcasts x {tiers} tiers"); + assert_eq!( + allocs, 0, + "depth {depth}, {broadcasts} broadcasts x {tiers} tiers, held prefix {hold}" + ); + if hold { + assert_eq!( + drain.groups[group_key("room0/cam", depth)].prefixes.len(), + 1, + "still held" + ); + } } } } diff --git a/rs/moq-stats/src/produce_bench.rs b/rs/moq-stats/src/produce_bench.rs index 0a9b992b820..b02327c0cce 100644 --- a/rs/moq-stats/src/produce_bench.rs +++ b/rs/moq-stats/src/produce_bench.rs @@ -64,4 +64,18 @@ impl Driver { group.adopt_presence(&self.drain.report, root.clone(), PendingPair::default()); group.presence[&root].pair.plain.track.subscribe(None) } + + /// Hold broadcast prefix `path`'s track as a reader would; see + /// [`Self::hold_presence`]. + pub fn hold_prefix(&mut self, path: &str) -> track::Subscriber { + let tick = self.drain.tick; + let group = self.drain.groups.get_mut("").expect("the depth 0 broadcast"); + group.adopt_prefix( + &self.drain.report, + PathOwned::from(path.to_string()), + PendingPair::default(), + tick, + ); + group.prefixes[path].pair.plain.track.subscribe(None) + } } From e3e8c4428113c1a69a9f71d65993f020667275e4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 16 Oct 2026 17:49:59 -0700 Subject: [PATCH 3/3] quest: point the oversized-frame quest at the landed prefix tracks Co-Authored-By: Claude Opus 5.5 --- quest/m1/stats-frame-too-large.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/quest/m1/stats-frame-too-large.md b/quest/m1/stats-frame-too-large.md index 72437489ef2..cb1d9add508 100644 --- a/quest/m1/stats-frame-too-large.md +++ b/quest/m1/stats-frame-too-large.md @@ -15,6 +15,6 @@ reader learns the track stopped tracking the relay. - Abort only the flavor whose `update` failed (log at warn) and keep its sibling publishing: DEFLATE can shrink an oversized plain frame or slightly grow a fitting one. Test that the overflowing flavor ends with - `FrameTooLarge` while the other keeps updating. Prefix tracks get the same - rule from [Stats prefix tracks](/quest/m2/stats-prefix-tracks.md). + `FrameTooLarge` while the other keeps updating. Prefix tracks already do + this (#5811): set `TrackPair::abort_oversized` on the remaining tracks. - Found in #5719.