diff --git a/doc/bin/cli.md b/doc/bin/cli.md index fab9bdecf50..7195b9feda1 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -557,8 +557,9 @@ of `--follow`. `import --report` names a track in the catalog's `stats` section and reports what each rendition has written, about once a second while someone subscribes, -with the connection's RTT, estimated send rate, and loss. It covers the stdin imports; -capture, the gateways, `archive`, and `ts --program all` refuse it. +with the connection's RTT, estimated send rate, and loss. It covers the stdin imports +and `capture`, whose renditions also report the encoder's target bitrate and the +video frames it fell behind on; the gateways, `archive`, and `ts --program all` refuse it. `import --echo-path ` solicits feedback from the same imports: it sets the catalog's `echo` path, resolved against the broadcast like a relative URL, diff --git a/doc/concept/stats.md b/doc/concept/stats.md index 8f9dbe55733..011e2256c80 100644 --- a/doc/concept/stats.md +++ b/doc/concept/stats.md @@ -146,8 +146,9 @@ A viewer's loss counters describe its own uplink, so they are reported but not graded. Loss and headroom divide the connection's transport counters by the report's own media, so they hold when the broadcast is the only one its connection publishes. The transport limits were measured on one paced TS feed -through a shaped link over five-second intervals; no CLI reporter emits -`skipped`, `late`, `errors`, or `stalled`, so their limits are not measured. +through a shaped link over five-second intervals. Only `import capture` +emits `skipped`, and no CLI reporter emits `late`, `errors`, or `stalled`, so +those limits are not measured. A publisher whose link falls below its media rate sends no report at all, since its report track sits below every media track. Its health goes stale diff --git a/quest/m2/stats/README.md b/quest/m2/stats/README.md index c6e5f84b87c..1a64be617c6 100644 --- a/quest/m2/stats/README.md +++ b/quest/m2/stats/README.md @@ -111,8 +111,8 @@ connection view and the dashboard flow. ## Required -- [Rust reporters](/quest/m2/stats/rust.md) - the player, encoders, and - gateways publish stats and feedback, as the CLI remuxes do +- [Rust reporters](/quest/m2/stats/rust.md) - the player and gateways + publish stats and feedback, as the CLI remuxes and capture do - [Browser reporters](/quest/m2/stats/js.md) - `` publishes stats and `` publishes feedback - [Preflight](/quest/m2/stats/preflight.md) - a bounded test run over a diff --git a/quest/m2/stats/rust.md b/quest/m2/stats/rust.md index a6105d56dc7..b24203cd5dd 100644 --- a/quest/m2/stats/rust.md +++ b/quest/m2/stats/rust.md @@ -1,41 +1,53 @@ -# [M] The Rust player, encoders, and gateways report stats and feedback +# [M] The Rust player and gateways report stats and feedback ## Goal -- `moq import capture --report` announces a stats track in the catalog and - fills it with what its encoders send, `skipped` and `bitrate` included. - `moq play`, given `--echo `, publishes an `.echo` broadcast under each watched catalog's echo path and fills its feedback track with what it received, decoded, and played. - The gateway imports (`hls`, `rtmp`, `srt`, `rtsp`, `rtc`) and `archive` take - `--report`. -- Capture, the gateways, and `archive` take `--echo-path`, which sets the - catalog's `echo` section as it does for the stdin imports. + `--report` and `--echo-path`, which set the catalog's `stats` and `echo` + sections as they do for the stdin imports and capture. +- `import capture --report` counts the audio frames a microphone capture + dropped in `skipped`, as it already does for video. ## Plan -- `moq import --report` for the stdin imports, `moq export --echo`, and the - `export report` and `export echo` sinks already report what the `moq-mux` - importers write and exporters read (`rs/moq-mux/src/stats.rs`, - `rs/moq-cli/src/report.rs`), and `import --echo-path` solicits feedback for - the stdin imports. This quest adds the counters they lack and reuses the - rest; `import --report` and `--echo-path` refuse capture and the gateways - until it lands. +- `import --report` and `--echo-path` already cover the stdin imports and + capture, and `export --echo` and the `export report` and `export echo` sinks + already read and print the reports (`rs/moq-mux/src/stats.rs`, + `rs/moq-cli/src/report.rs`). The catalog's `Sent` registry counts what every + track minted under it writes, and an encoder reports its skips and target + through `container::Producer::report_skipped` and `report_bitrate`. This quest + adds the counters still missing and reuses the rest; `--report` and + `--echo-path` refuse the gateways until it lands. - Counters come from the source, per track, as cumulative `stats()` snapshots: no callback, no per-frame lock. - - Publisher: the `moq-video` and `moq-audio` encode producers create their - tracks directly on the broadcast, so they count frames, bytes, keyframes, - and drops themselves, and expose the target bitrate. - Viewer: the decode and render paths count received, decoded, late, stalled, underruns, and errors. `underruns` in - `rs/moq-audio/src/playback` is already counted privately. -- The gateways build their catalogs in their own crates, so each needs a way - to name the stats track and the echo path before its first catalog. - Capture publishes its catalog as its encoders start, so both sections reach - it before they do. + `rs/moq-audio/src/playback` is already counted privately. The player reads + video through `moq_mux::container::Consumer` directly, so it bypasses the + `Source`'s `Received` registry today. + - Audio capture: `capture::Samples::gap` only flags that buffers were + dropped. Counting `skipped` needs the drop size, converted to codec frames. +- The gateways build their catalogs in their own crates (`moq-rtmp`, + `moq-srt`, `moq-rtc`, `moq-archive`; `hls` and `rtsp` build theirs in + `rs/moq-cli`), so each needs a way to name the stats track and the echo path + before its first catalog, and a reporter to run beside each session's + catalog. The report writer lives in `rs/moq-cli/src/report.rs` and reads the + transport from a `moq_tokio` connection monitor, which the gateway crates do + not see. +- Decided 2026-10-31 (maintainer): move the report writer into `moq-mux`, with + the transport passed in as data rather than read from a `moq_tokio` monitor, + and give each gateway builder `with_report` and `with_echo`. A library user + embedding a gateway gets reports too; the cost, accepted, is that `Stats` + becomes a library API. Rejected: handing each session's catalog back to the + CLI through a returned handle, which keeps the writer in `moq-cli` but leaves + library users without reports. - `moq play --echo` reconciles each watched catalog's `echo` section as `export --echo` does, and refuses `--catalog-format msf` the same way: the - MSF catalog does not carry an `echo` solicitation. + MSF catalog does not carry an `echo` solicitation. `play` attaches only the + consume direction today, so it also needs the publish direction. - The media test captures with `--report` and plays with `--echo` against a publisher that solicits feedback. It asserts the encoder's frame count and that the player's newest arrival advances. @@ -43,7 +55,6 @@ broadcast yet, so [Encrypted native CLI](/quest/m2/e2ee/cli.md) refuses `--report` and `--echo` when it adds one: they would publish rendition IDs and per-track counters in plaintext beside it. - - Decided 2026-10-22 (maintainer): the publisher's solicit option moved here from [encoder feedback](/quest/m3/stats-encoder-feedback.md), which keeps only the adaptation loop and reuses `--echo-path`. diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index 4dd085fcd93..4c539e0cde2 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -194,6 +194,8 @@ impl Reserved { /// Spend the reservation on a registered encoder, publishing through the track. pub(crate) fn encode(self, registered: Registered) -> Producer { let retention = self.track.track().info().max_age.unwrap_or(hang::container::MAX_AGE); + let bitrate = registered.encoder.bitrate(); + self.track.report_bitrate(bitrate); Producer { backlog: Backlog::new( @@ -203,7 +205,7 @@ impl Reserved { ), active: true, reset_pending: false, - bitrate: registered.encoder.bitrate(), + bitrate, encoder: Some(registered.encoder), input: registered.input, remix: registered.remix, @@ -319,6 +321,7 @@ impl Producer { return Err(err.into()); } self.bitrate = encoder.bitrate(); + self.track.report_bitrate(self.bitrate); Ok(()) } @@ -680,6 +683,27 @@ mod tests { } } + /// The catalog's stats follow the encoder's target from the first rendition on. + #[tokio::test] + async fn stats_follow_the_target_bitrate() { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let options = Options { + track: Some("audio".into()), + ..Options::default() + }; + let mut producer = Producer::new(&mut broadcast, catalog.clone(), Input::default(), &options).unwrap(); + let reported = || catalog.stats().remove("audio").and_then(|stats| stats.bitrate); + assert_eq!(reported(), Some(producer.bitrate().as_bps())); + + let initial = producer.bitrate(); + producer + .set_bitrate(moq_net::bandwidth::Rate::from_bps(initial.as_bps() / 2)) + .unwrap(); + assert_ne!(producer.bitrate(), initial); + assert_eq!(reported(), Some(producer.bitrate().as_bps())); + } + #[tokio::test] #[ignore = "measures real codec CPU; run with just rs bench-playback"] async fn benchmark_raw_audio() { diff --git a/rs/moq-cli/src/args.rs b/rs/moq-cli/src/args.rs index 539143e1c68..e2b915d3de4 100644 --- a/rs/moq-cli/src/args.rs +++ b/rs/moq-cli/src/args.rs @@ -333,7 +333,7 @@ impl Invocation { } if let Command::Import(import) = command && import.report - && let Some(reason) = import.source.report_refusal() + && let Some(reason) = import.source.catalog_refusal() { anyhow::bail!("--report names a track in the catalog the import builds, but {reason}"); } @@ -826,13 +826,14 @@ pub struct Import { latency_max: Option, /// Name a media report track in the catalog's `stats` section and fill it with what each - /// track has written, about once a second while subscribed. The stdin imports only. + /// track has written, about once a second while subscribed. The stdin imports and capture + /// only. #[usage(long)] pub report: bool, /// Solicit viewer feedback in the catalog: `export --echo ` publishes `.echo` /// under this prefix, resolved against the broadcast like a relative URL (`viewers` from - /// `room/event` is `room/viewers`). The stdin imports only. + /// `room/event` is `room/viewers`). The stdin imports and capture only. #[usage(long, value_name = "PATH")] pub echo_path: Option, @@ -908,7 +909,8 @@ impl ImportSource { } /// Why the CLI can't add a section to this source's first catalog, or `None` when it can: - /// a stdin import's one catalog waits for its importer's tracks. + /// a stdin import's one catalog waits for its importer's tracks, and a capture's for its + /// encoders. fn catalog_refusal(&self) -> Option<&'static str> { match self { Self::Ts(args) if args.program == Some(TsProgram::All) => { @@ -918,21 +920,11 @@ impl ImportSource { Some("the gateway and archive imports build their own catalogs") } #[cfg(feature = "capture")] - Self::Capture(_) => Some("`import capture` publishes its catalog as its encoders start"), + Self::Capture(_) => None, Self::Avc3 | Self::Fmp4(_) | Self::Ts(_) | Self::Flv => None, } } - /// Why `--report` can't report this source, or `None` when it can: the CLI builds the one - /// catalog of a stdin import, whose tracks `moq-mux` counts. - fn report_refusal(&self) -> Option<&'static str> { - #[cfg(feature = "capture")] - if matches!(self, Self::Capture(_)) { - return Some("`import capture` encodes, and its encoders do not count yet"); - } - self.catalog_refusal() - } - /// Why `--linger` can't apply to this source, or `None` when it can: only a source with a /// clean end has a finished broadcast to keep up. fn linger_refusal(&self) -> Option<&'static str> { @@ -1846,6 +1838,8 @@ mod tests { ] { import(flags, source).validate().unwrap(); } + #[cfg(feature = "capture")] + import(flags, &["capture"]).validate().unwrap(); for source in [ ["ts", "--program", "all"].as_slice(), &["hls", "http://origin/master.m3u8"], diff --git a/rs/moq-cli/src/publish.rs b/rs/moq-cli/src/publish.rs index 836e4656480..687dba88103 100644 --- a/rs/moq-cli/src/publish.rs +++ b/rs/moq-cli/src/publish.rs @@ -332,6 +332,9 @@ enum Source { catalog: moq_mux::catalog::Producer, video: Option, audio: Option, + /// Withholds the catalog until [`Publish::run`] starts the encoders, so the `stats` and + /// `echo` sections set before then ride the first catalog rather than an empty one. + gate: moq_mux::catalog::Reserved, }, } @@ -554,6 +557,7 @@ impl Publish { Ok(Self { source: Source::Capture { broadcast: broadcast.clone(), + gate: catalog.reserve(), catalog, video, audio, @@ -573,16 +577,20 @@ impl Publish { /// /// Call it before the importer declares its tracks, so the section rides the first catalog. pub fn with_report(mut self, connection: Option) -> anyhow::Result { - let (Some(broadcast), Source::Stream { catalog, .. }) = (self.broadcast.as_mut(), &self.source) else { - anyhow::bail!("--report reports a stdin import's one catalog"); + let Some(broadcast) = self.broadcast.as_mut() else { + anyhow::bail!("`import ts --program all` builds a catalog per program"); }; if let Some(republish) = &mut self.republish { republish.report = Some(connection.clone()); } - let report = match catalog { - PublishCatalog::Media(catalog) => crate::report::Stats::new(broadcast, catalog, connection)?, - PublishCatalog::Ts(catalog) => crate::report::Stats::new(broadcast, catalog, connection)?, - PublishCatalog::TsPrograms => anyhow::bail!("`import ts --program all` builds a catalog per program"), + let report = match &self.source { + Source::Stream { catalog, .. } => match catalog { + PublishCatalog::Media(catalog) => crate::report::Stats::new(broadcast, catalog, connection)?, + PublishCatalog::Ts(catalog) => crate::report::Stats::new(broadcast, catalog, connection)?, + PublishCatalog::TsPrograms => anyhow::bail!("`import ts --program all` builds a catalog per program"), + }, + #[cfg(feature = "capture")] + Source::Capture { catalog, .. } => crate::report::Stats::new(broadcast, catalog, connection)?, }; self.report = Some(report); Ok(self) @@ -595,7 +603,10 @@ impl Publish { let catalog = match &mut self.source { Source::Stream { catalog, .. } => catalog, #[cfg(feature = "capture")] - Source::Capture { .. } => anyhow::bail!("--echo-path solicits feedback in a stdin import's one catalog"), + Source::Capture { catalog, .. } => { + catalog.mutate(|catalog| catalog.echo = Some(echo))?; + return Ok(self); + } }; if let Some(republish) = &mut self.republish { republish.echo = Some(echo.clone()); @@ -654,7 +665,10 @@ impl Publish { catalog, video, audio, + gate, } => { + drop(gate); + let mut report = self.report; // Each enabled medium publishes its own track onto the shared // broadcast + catalog. Both stamp frames on the catalog's // advertised clock, so HLS/DASH wall times match the mapping on @@ -683,8 +697,21 @@ impl Publish { } }; - tokio::try_join!(video_fut, audio_fut)?; - Ok(()) + let capture = async { tokio::try_join!(video_fut, audio_fut).map(|_| ()) }; + // Written beside the encoders, so a capture waiting on its first viewer still reports. + let result = match report.as_mut() { + Some(report) => tokio::select! { + result = capture => result, + never = report.run() => match never {}, + }, + None => capture.await, + }; + match (&result, report) { + (Ok(()), Some(mut report)) => report.finish(), + (Err(err), Some(report)) => report.abort(moq_net::Error::Transport(err.to_string())), + (_, None) => {} + } + result } } } diff --git a/rs/moq-mux/src/codec/h264/import.rs b/rs/moq-mux/src/codec/h264/import.rs index d5afd3ed75e..82c26367dcd 100644 --- a/rs/moq-mux/src/codec/h264/import.rs +++ b/rs/moq-mux/src/codec/h264/import.rs @@ -173,6 +173,18 @@ impl Import { Ok(()) } + /// Report frames the encoder feeding this track skipped. See + /// [`Producer::report_skipped`](crate::container::Producer::report_skipped). + pub fn report_skipped(&self, frames: u64) { + self.track.report_skipped(frames); + } + + /// Report the target bitrate of the encoder feeding this track. See + /// [`Producer::report_bitrate`](crate::container::Producer::report_bitrate). + pub fn report_bitrate(&self, bitrate: moq_net::bandwidth::Rate) { + self.track.report_bitrate(bitrate); + } + /// Close the current group and open the next one at `sequence`. pub fn seek(&mut self, sequence: u64) -> Result<()> { self.track.seek(sequence)?; diff --git a/rs/moq-mux/src/codec/h265/import.rs b/rs/moq-mux/src/codec/h265/import.rs index 121f157fa6a..1b280b1e14d 100644 --- a/rs/moq-mux/src/codec/h265/import.rs +++ b/rs/moq-mux/src/codec/h265/import.rs @@ -176,6 +176,18 @@ impl Import { Ok(()) } + /// Report frames the encoder feeding this track skipped. See + /// [`Producer::report_skipped`](crate::container::Producer::report_skipped). + pub fn report_skipped(&self, frames: u64) { + self.track.report_skipped(frames); + } + + /// Report the target bitrate of the encoder feeding this track. See + /// [`Producer::report_bitrate`](crate::container::Producer::report_bitrate). + pub fn report_bitrate(&self, bitrate: moq_net::bandwidth::Rate) { + self.track.report_bitrate(bitrate); + } + /// Close the current group and open the next one at `sequence`. pub fn seek(&mut self, sequence: u64) -> Result<()> { self.track.seek(sequence)?; diff --git a/rs/moq-mux/src/container/producer.rs b/rs/moq-mux/src/container/producer.rs index c3f11190b59..1187decb8a1 100644 --- a/rs/moq-mux/src/container/producer.rs +++ b/rs/moq-mux/src/container/producer.rs @@ -545,6 +545,22 @@ where self } + /// Report `frames` frames the source produced but never handed to this track, such as ones an + /// encoder fell behind on, in its catalog's [`stats`](crate::catalog::Producer::stats). + pub fn report_skipped(&self, frames: u64) { + if let Some(sent) = &self.sent { + sent.skip(frames); + } + } + + /// Report the encoder's latest target bitrate in its catalog's + /// [`stats`](crate::catalog::Producer::stats). The catalog's `bitrate` stays the maximum. + pub fn report_bitrate(&self, bitrate: moq_net::bandwidth::Rate) { + if let Some(sent) = &self.sent { + sent.target(bitrate); + } + } + /// Count a written frame. Only video keyframes count as keyframes: elsewhere the bit only /// marks where a group starts. fn count(&self, bytes: usize, keyframe: bool) { diff --git a/rs/moq-mux/src/stats.rs b/rs/moq-mux/src/stats.rs index c9d057a6bd7..4bc39b73ae7 100644 --- a/rs/moq-mux/src/stats.rs +++ b/rs/moq-mux/src/stats.rs @@ -17,6 +17,9 @@ pub(crate) struct Sent { frames: AtomicU64, bytes: AtomicU64, keyframes: AtomicU64, + skipped: AtomicU64, + /// The encoder's latest target, in bits per second, or zero before one is reported. + bitrate: AtomicU64, } impl Sent { @@ -27,11 +30,23 @@ impl Sent { self.keyframes.fetch_add(keyframes, Relaxed); } + /// Count `frames` frames the source produced but never sent. + pub(crate) fn skip(&self, frames: u64) { + self.skipped.fetch_add(frames, Relaxed); + } + + /// Record the encoder's latest target bitrate. + pub(crate) fn target(&self, bitrate: moq_net::bandwidth::Rate) { + self.bitrate.store(bitrate.as_bps(), Relaxed); + } + fn report(&self) -> hang::stats::Rendition { let mut report = hang::stats::Rendition::default(); report.frames = self.frames.load(Relaxed); report.bytes = self.bytes.load(Relaxed); report.keyframes = self.keyframes.load(Relaxed); + report.skipped = self.skipped.load(Relaxed); + report.bitrate = Some(self.bitrate.load(Relaxed)).filter(|&bitrate| bitrate != 0); report } } @@ -177,6 +192,22 @@ mod tests { } } + /// An encoder's skips accumulate, and its target is the latest one set: none until then. + #[test] + fn an_encoder_reports_skips_and_its_latest_target() { + let registry = Registry::::default(); + let track = registry.track("video"); + assert_eq!(registry.sent()["video"].bitrate, None); + + track.skip(2); + track.skip(1); + track.target(moq_net::bandwidth::Rate::from_bps(2_000_000)); + track.target(moq_net::bandwidth::Rate::from_bps(1_500_000)); + let report = ®istry.sent()["video"]; + assert_eq!(report.skipped, 3); + assert_eq!(report.bitrate, Some(1_500_000)); + } + #[test] fn a_returning_track_continues_its_counts() { let registry = Registry::::default(); diff --git a/rs/moq-video/src/capture/channel.rs b/rs/moq-video/src/capture/channel.rs index 707bd080932..f387d0390bd 100644 --- a/rs/moq-video/src/capture/channel.rs +++ b/rs/moq-video/src/capture/channel.rs @@ -30,6 +30,17 @@ struct State { native: Option, closed: bool, error: Option, + /// Frames replaced before the consumer reached them. + skipped: u64, +} + +impl State { + /// Hold `frame` as the latest, counting the one it replaces as skipped. + fn replace(&mut self, frame: Frame) { + if self.frame.replace(frame).is_some() { + self.skipped += 1; + } + } } /// How a device's own timeline maps onto this stream's. @@ -54,6 +65,7 @@ impl FrameChannel { native: None, closed: false, error: None, + skipped: 0, }), notify: Notify::new(), epoch: Instant::now(), @@ -122,7 +134,7 @@ impl FrameChannel { mapped: timestamp, ..anchor }); - state.frame = Some(Frame::new(surface, timestamp)); + state.replace(Frame::new(surface, timestamp)); drop(state); self.notify.notify_one(); } @@ -132,11 +144,16 @@ impl FrameChannel { if state.closed { return; } - state.frame = Some(frame); + state.replace(frame); drop(state); self.notify.notify_one(); } + /// How many frames a newer one replaced before [`recv`](Self::recv) reached them. + pub(super) fn skipped(&self) -> u64 { + self.state.lock().unwrap().skipped + } + /// Mark the source ended, so a parked [`recv`](Self::recv) returns `None`. pub(super) fn close(&self) { let mut state = self.state.lock().unwrap(); @@ -235,6 +252,7 @@ mod tests { chan.push(frame(id)); } assert_eq!(chan.recv().await.unwrap().unwrap().surface.width(), 6); + assert_eq!(chan.skipped(), 5); } #[tokio::test] diff --git a/rs/moq-video/src/capture/mod.rs b/rs/moq-video/src/capture/mod.rs index 4ce7ce838fe..b7092e5277d 100644 --- a/rs/moq-video/src/capture/mod.rs +++ b/rs/moq-video/src/capture/mod.rs @@ -357,6 +357,12 @@ impl Stream { Ok(Some(Frame::new(surface, frame.timestamp))) } + /// How many captured frames no [`read`](Self::read) returned, because a newer one replaced + /// them while the reader was busy. + pub(crate) fn skipped(&self) -> u64 { + self.chan.skipped() + } + /// Width of the captured frames in pixels. pub fn width(&self) -> u32 { self.width diff --git a/rs/moq-video/src/encode/producer.rs b/rs/moq-video/src/encode/producer.rs index ec80446f92a..5d7a0ea958d 100644 --- a/rs/moq-video/src/encode/producer.rs +++ b/rs/moq-video/src/encode/producer.rs @@ -234,6 +234,24 @@ impl Producer { Ok(()) } + /// Report capture frames the encoder never saw, in the catalog's stats. + #[cfg(feature = "capture")] + fn report_skipped(&self, frames: u64) { + match &self.codecs { + Codecs::H264 { import, .. } => import.report_skipped(frames), + Codecs::H265 { import, .. } => import.report_skipped(frames), + } + } + + /// Report the encoder's target bitrate, in the catalog's stats. + #[cfg(feature = "capture")] + fn report_bitrate(&self, bitrate: moq_net::bandwidth::Rate) { + match &self.codecs { + Codecs::H264 { import, .. } => import.report_bitrate(bitrate), + Codecs::H265 { import, .. } => import.report_bitrate(bitrate), + } + } + /// Finalize the track. /// /// Borrows rather than consumes, so a later [`abort`](Self::abort) can still @@ -512,37 +530,46 @@ async fn next_target(rate: &mut RateControl) -> Result) { - let Some(follower) = rate.as_mut() else { return }; +async fn apply_target( + encoder: &mut Sink, + rate: &mut RateControl, + target: Result, +) -> Option { + let follower = rate.as_mut()?; let Ok(target) = target else { tracing::debug!("bandwidth estimate ended; holding the current encoder bitrate"); *rate = None; - return; + return None; }; let rate::Target::Enabled(bitrate) = target else { unreachable!("a capture follows its grant with no lower rendition, so it never disables") }; match encoder.set_bitrate(bitrate).await { - Ok(()) => tracing::debug!(bitrate = bitrate.as_bps(), "adjusted encoder bitrate"), + Ok(()) => { + tracing::debug!(bitrate = bitrate.as_bps(), "adjusted encoder bitrate"); + Some(bitrate) + } // The encoder can't retune, so keep encoding at the rate it opened with // and stop asking. Dropping the source also stops the estimate arm, which // would otherwise wake this loop for nothing on every change. Err(Error::BitrateUnsupported(name)) => { tracing::warn!(encoder = name, "encoder cannot follow the bandwidth estimate"); *rate = None; + None } // A transient failure: the follower takes the target back and retries it, // so a quiet grant still recovers. Err(err) => { tracing::warn!(error = %err, bitrate = bitrate.as_bps(), "failed to adjust encoder bitrate"); follower.failed(); + None } } } @@ -607,26 +634,32 @@ async fn wait_capture( /// Return an encoder nobody watches to its ceiling, and start its rate control over from there /// for the next viewer. Its groups wait in the cache rather than being paced to a subscriber, -/// and a past viewer's grant is stale by the time the next one arrives. +/// and a past viewer's grant is stale by the time the next one arrives. Returns the ceiling once +/// the encoder takes it. #[cfg(feature = "capture")] async fn rest( encoder: &mut Sink, rate: &mut RateControl, claim: &moq_net::bandwidth::Reservation, ceiling: moq_net::bandwidth::Rate, -) { +) -> Option { // Retired: the encoder cannot retune, or the grant is gone and it holds its rate. - if rate.is_none() { - return; - } + rate.as_ref()?; match encoder.set_bitrate(ceiling).await { - Ok(()) => *rate = Some(rate::Follower::new(claim.consume(), rate::Policy::new(ceiling))), + Ok(()) => { + *rate = Some(rate::Follower::new(claim.consume(), rate::Policy::new(ceiling))); + Some(ceiling) + } Err(Error::BitrateUnsupported(name)) => { tracing::warn!(encoder = name, "encoder cannot follow the bandwidth estimate"); *rate = None; + None } // The follower still matches the rate the encoder kept, so the next viewer follows on from it. - Err(err) => tracing::warn!(error = %err, bitrate = ceiling.as_bps(), "failed to rest the encoder bitrate"), + Err(err) => { + tracing::warn!(error = %err, bitrate = ceiling.as_bps(), "failed to rest the encoder bitrate"); + None + } } } @@ -710,12 +743,18 @@ async fn capture_loop( // reopened camera starts optimistic again rather than inheriting the // backed-off rate from whatever the link was doing last time. let mut rate = Some(rate::Follower::new(claim.consume(), rate::Policy::new(ceiling))); + // The encoder's current target, reported again on any track the capture moves to. + let mut bitrate = ceiling; + producer.report_bitrate(bitrate); // Per encoder, since a fresh one opens with a keyframe that serves every earlier ask. let mut forced = Cuts::default(); // Rate control follows the grant only while someone watches. The first poll of `watch` // reports the current demand. let mut watch = demand.clone(); let mut watched = false; + // The camera's skips already accounted for, from its first read on: frames it replaced + // while this encoder was still opening are startup, not frames the encoder fell behind on. + let mut reported = None; loop { // Race the next frame against demand, so an on-demand capture releases the @@ -734,8 +773,11 @@ async fn capture_loop( if on_demand && !wanted { break; // no viewers: release the camera, then wait for one } - if watched && !wanted { - rest(&mut encoder, &mut rate, claim, ceiling).await; + if watched + && !wanted && let Some(rested) = rest(&mut encoder, &mut rate, claim, ceiling).await + { + bitrate = rested; + producer.report_bitrate(bitrate); } watched = wanted; continue; @@ -743,7 +785,10 @@ async fn capture_loop( // Retune between frames rather than mid-encode, and only when // the policy says the target actually moved. target = next_target(&mut rate), if watched => { - apply_target(&mut encoder, &mut rate, target).await; + if let Some(applied) = apply_target(&mut encoder, &mut rate, target).await { + bitrate = applied; + producer.report_bitrate(bitrate); + } continue; } // A read error is terminal for this selection (the source is gone @@ -752,6 +797,12 @@ async fn capture_loop( }; let Some(mut frame) = frame else { break }; + // Frames the camera replaced while this loop was encoding or retuning. + let skipped = camera.skipped(); + if let Some(reported) = reported { + producer.report_skipped(skipped - reported); + } + reported = Some(skipped); frame.timestamp = map_capture_timestamp(capture_epoch, frame.timestamp)?; // A backend that cannot cut leaves every group to its cadence. let wanted = cuts.then(|| producer.wanted()).flatten(); @@ -775,10 +826,19 @@ async fn capture_loop( } watch = demand.clone(); watched = false; - rest(&mut encoder, &mut rate, claim, ceiling).await; + // The new track's stats start empty, so they learn the target whether or not it rests. + if let Some(rested) = rest(&mut encoder, &mut rate, claim, ceiling).await { + bitrate = rested; + } + producer.report_bitrate(bitrate); } } + // Frames the camera replaced after the last read, before demand or the stream ended + // the loop: no read follows to report them. + if let Some(reported) = reported { + producer.report_skipped(camera.skipped() - reported); + } // Drop the camera (LED off) and encoder before opening another. drop(camera); drop(encoder); @@ -1310,6 +1370,61 @@ mod tests { .await } + /// The catalog's stats carry the encoder's target and every frame the camera replaced + /// before the loop read it, once the loop has read its first. + #[tokio::test] + async fn stats_report_the_target_and_skipped_frames() { + tokio::task::LocalSet::new() + .run_until(async { + let fixture = Fixture::start(Duration::from_secs(1), kio::clock::wall()).await; + let mut track = fixture.subscribe().await; + let camera = fixture.camera(); + + // Replaced before the first read, while the encoder was still opening: not a skip. + camera.push_at(surface(), Instant::now()); + camera.push_at(surface(), Instant::now()); + read(&mut track).await; + // Back to back once encoding, so the loop only ever sees the second. + camera.push_at(surface(), Instant::now()); + camera.push_at(surface(), Instant::now()); + read(&mut track).await; + + let stats = fixture.catalog.stats().remove("video").expect("the video rendition"); + assert_eq!(stats.skipped, 1); + let ceiling = Config::new(320, 240, crate::FrameRate::new(30, 1).unwrap()).resolved_bitrate(); + assert_eq!(stats.bitrate, Some(ceiling.as_bps())); + fixture.finish().await; + }) + .await + } + + /// Frames the camera replaced after the last read still count when the last viewer + /// leaves before another read. + #[tokio::test] + async fn stats_report_skips_left_when_the_viewer_leaves() { + tokio::task::LocalSet::new() + .run_until(async { + let fixture = Fixture::start(Duration::from_secs(1), kio::clock::wall()).await; + let mut track = fixture.subscribe().await; + let camera = fixture.camera(); + camera.push_at(surface(), Instant::now()); + read(&mut track).await; + + // The viewer leaves, then the camera replaces a frame before the loop runs again. + drop(track); + camera.push_at(surface(), Instant::now()); + camera.push_at(surface(), Instant::now()); + while !camera.released() { + tokio::task::yield_now().await; + } + + let stats = fixture.catalog.stats().remove("video").expect("the video rendition"); + assert_eq!(stats.skipped, 1); + fixture.finish().await; + }) + .await + } + /// A device clock that restarts at zero, mid-stream or across a reopen, continues the /// broadcast forward with the device's spacing instead of rewinding it. #[tokio::test]