From e409c2eae407f6a4dceed3b32538183433041122 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 28 Oct 2026 15:43:13 -0700 Subject: [PATCH 1/2] chore(quest): claim quest/m1/eager-encode/video.md Quest-Claim: 9f733dab939e9864 From 1b3971780982fa4073886aaee0e52f4a75c8ff80 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 28 Oct 2026 16:07:52 -0700 Subject: [PATCH 2/2] feat(moq-video)!: capture encodes while enabled, on demand as an opt-in `publish_capture` keeps the camera open and encodes whether or not anyone watches, so a late subscriber or fetch finds the recent past in the track's cache. It captures from the probe's open instead of reopening the source, sparing a second permission prompt or portal picker. `Capture::on_demand` restores the close-between-viewers behavior. While nobody watches, the encoder returns to its configured bitrate and ignores the grant; the next viewer's rate control starts over from there. A rendition move keeps capturing onto the new track instead of reopening the camera, unless on demand. Co-Authored-By: Claude Opus 5.5 --- doc/bin/cli.md | 2 +- doc/lib/rs/moq-video.md | 9 +- quest/m1/eager-encode/README.md | 1 - quest/m1/eager-encode/cli.md | 8 +- quest/m1/eager-encode/video.md | 24 -- rs/moq-cli/src/publish.rs | 7 +- rs/moq-net/src/model/bandwidth.rs | 2 +- rs/moq-transcode/src/lib.rs | 2 - rs/moq-transcode/src/pipeline.rs | 4 +- rs/moq-video/README.md | 5 +- rs/moq-video/src/capture/mod.rs | 10 +- rs/moq-video/src/capture/pipewire.rs | 18 +- rs/moq-video/src/encode/mod.rs | 4 +- rs/moq-video/src/encode/producer.rs | 561 ++++++++++++++++++++------- rs/moq-video/src/lib.rs | 8 +- 15 files changed, 457 insertions(+), 208 deletions(-) delete mode 100644 quest/m1/eager-encode/video.md diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 00aefd0d5f8..92a40f904d7 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -250,7 +250,7 @@ moq ... import capture --camera 0 --width 1280 --height 720 --fps 30 --bitrate 3 Video goes through the platform hardware encoder, with an H.264 software fallback when built with `openh264`, and audio is Opus; [moq-video](/lib/rs/moq-video) lists the -backends. The camera opens only while someone is watching, and `--bitrate` is a +backends. The camera stays open while publishing, and `--bitrate` is a ceiling that backends with live bitrate control lower to fit the connection. `moq devices` prints every source id. On Windows, `display:N` is an enumeration index rather than a stable monitor id, so re-run `moq devices` if a diff --git a/doc/lib/rs/moq-video.md b/doc/lib/rs/moq-video.md index 6a898ce49bc..0faed2db33e 100644 --- a/doc/lib/rs/moq-video.md +++ b/doc/lib/rs/moq-video.md @@ -26,15 +26,18 @@ feature (vendored C++); the published bindings and binaries enable it. H.265 is hardware-only, AV1 decodes through NVDEC and MediaCodec, and VP8 and VP9 decode in software through the opt-in `vpx` feature. -Capture is pull-driven: `encode::publish_capture` advertises the track up -front and opens the camera only while someone subscribes. A viewer who joins +`encode::publish_capture` advertises the track up front, then keeps the camera +open and encoding whether or not anyone subscribes, so a late viewer can fetch +the recent past. Set `encode::Capture::on_demand` to open the camera only while +someone subscribes, at the cost of that past. A viewer who joins once the current group's deltas outweigh twice its keyframe gets a fresh keyframe, so the GOP is only the longest a group runs. Through a moq-lite relay, only the join that opens the relay's upstream subscription counts. A build with the `v4l2` feature keeps every `Auto` or `Hardware` selection to its GOP, whichever encoder it lands on, since a V4L2 driver may refuse the request. Backends that can -change bitrate live follow the connection's send estimate. Where the platform +change bitrate live follow the connection's send estimate while someone +watches, and return to the configured bitrate while nobody does. Where the platform allows it, frames stay on the GPU between capture, decode, encode, and render. Opt-in features: `capture`, `vaapi` (Intel and AMD on Linux; the build needs diff --git a/quest/m1/eager-encode/README.md b/quest/m1/eager-encode/README.md index db1025c011c..27ad178e972 100644 --- a/quest/m1/eager-encode/README.md +++ b/quest/m1/eager-encode/README.md @@ -51,7 +51,6 @@ The epic's own work, once the children land: ## Required - [@moq/publish encodes while enabled](/quest/m1/eager-encode/js.md) - audio and video encoders drop the demand gate, with an on-demand opt-in -- [moq-video captures while enabled](/quest/m1/eager-encode/video.md) - the camera stays open and encodes while enabled, with an on-demand opt-in - [moq-audio captures while enabled](/quest/m1/eager-encode/audio.md) - the microphone stays open and encodes while enabled, with an on-demand opt-in - [moq-cli on-demand flag](/quest/m1/eager-encode/cli.md) - `moq publish` capture encodes always, unless `--on-demand` - [Raw audio on demand](/quest/m1/eager-encode/raw-audio.md) - audio encodes on demand again, from raw frames kept up to `maxAge` diff --git a/quest/m1/eager-encode/cli.md b/quest/m1/eager-encode/cli.md index 3553e303f48..e4642d4f595 100644 --- a/quest/m1/eager-encode/cli.md +++ b/quest/m1/eager-encode/cli.md @@ -10,9 +10,10 @@ as today. Decided 2026-10-28 (maintainer); see the [epic](/quest/m1/eager-encode/README.md). `rs/moq-cli/src/publish.rs:592` calls both `publish_capture`s, so the default -follows the libraries. The flag selects their on-demand option. Update the -comment there and `doc/bin/cli.md:250`. Name the flag under the repository's -CLI rule. +follows the libraries. The flag selects their on-demand option +(`moq_video::encode::Capture::on_demand` for video). The comment there and +`doc/bin/cli.md:250` describe the always-on camera; add the flag and the +microphone. Name the flag under the repository's CLI rule. Tests: the CLI's capture args parse `--on-demand`, and the default publishes without a subscriber. @@ -21,5 +22,4 @@ Public API: CLI default change, plus a new flag. Wire: none. ## Required -- [moq-video captures while enabled](/quest/m1/eager-encode/video.md) - the video option this selects - [moq-audio captures while enabled](/quest/m1/eager-encode/audio.md) - the audio option this selects diff --git a/quest/m1/eager-encode/video.md b/quest/m1/eager-encode/video.md deleted file mode 100644 index c0a9f713a3b..00000000000 --- a/quest/m1/eager-encode/video.md +++ /dev/null @@ -1,24 +0,0 @@ -# [M] moq-video captures and encodes while enabled - -## Goal - -`moq-video`'s capture publisher keeps the camera open and encodes while -enabled, whatever the demand. An on-demand option on the capture config -restores today's close-between-viewers behavior. - -## Plan - -Decided 2026-10-28 (maintainer); see the [epic](/quest/m1/eager-encode/README.md). - -- The gate: `rs/moq-video/src/encode/producer.rs:3-13` (the module doc's - "strictly on demand"), `:525` (`publish_capture`) and `:660-700` - (`until_wanted`/`wait_capture`). -- The demand-driven PipeWire reopen (`capture/pipewire.rs:14,157,304,866`) - serves only the opt-in. -- Update the notes at `capture/mod.rs:116`, `lib.rs:29` and `encode/mod.rs:19`, - and `doc/lib/rs/moq-video.md:30`. - -Tests: with no subscriber, the capture publisher writes groups. The opt-in -keeps the camera closed until demand arrives. - -Public API: default behavior change, plus an additive option. Wire: none. diff --git a/rs/moq-cli/src/publish.rs b/rs/moq-cli/src/publish.rs index 068c293f6d1..7f9915fc947 100644 --- a/rs/moq-cli/src/publish.rs +++ b/rs/moq-cli/src/publish.rs @@ -589,10 +589,9 @@ impl Publish { // Each enabled medium publishes its own track onto the shared // broadcast + catalog. Both stamp frames on the catalog's // advertised clock, so HLS/DASH wall times match the mapping on - // the wire. Video encodes on demand (camera opens only while - // subscribed). Both run on this task rather than a spawn: on - // macOS the audio future holds ObjC handles across an await, - // so it is `!Send`. + // the wire. Video captures for the whole run. Both run on this + // task rather than a spawn: on macOS the audio future holds ObjC + // handles across an await, so it is `!Send`. let video_fut = { let broadcast = broadcast.clone(); let catalog = catalog.clone(); diff --git a/rs/moq-net/src/model/bandwidth.rs b/rs/moq-net/src/model/bandwidth.rs index ecafcde5693..3c266084cf9 100644 --- a/rs/moq-net/src/model/bandwidth.rs +++ b/rs/moq-net/src/model/bandwidth.rs @@ -922,7 +922,7 @@ mod tests { assert_eq!(second_share.rate(), Some(bps(1_000_000))); } - /// An unwatched track releases its reservation to the others, since `publish_capture` + /// An unwatched track releases its reservation to the others, since an on-demand sender /// stops encoding entirely while nothing is subscribed, but still reads the slice it /// would get back, which is what a stopped rendition its viewers left resumes on. #[test] diff --git a/rs/moq-transcode/src/lib.rs b/rs/moq-transcode/src/lib.rs index 1b1323dc394..72c0922fb77 100644 --- a/rs/moq-transcode/src/lib.rs +++ b/rs/moq-transcode/src/lib.rs @@ -1029,8 +1029,6 @@ mod tests { /// A source that resizes mid-stream takes the ladder with it. /// - /// `moq_video::encode::publish_capture` opens its source twice by design (once - /// to probe the mode, once when the first subscriber arrives), and a window /// A source that changes aspect ratio keeps every rung height while moving /// every rung width, so a rung retires and its replacement serves the same /// height. That replacement must not reuse the retired track's name: it is a diff --git a/rs/moq-transcode/src/pipeline.rs b/rs/moq-transcode/src/pipeline.rs index 7a174fcd1da..39113d97223 100644 --- a/rs/moq-transcode/src/pipeline.rs +++ b/rs/moq-transcode/src/pipeline.rs @@ -1,8 +1,8 @@ //! The ladder a transcoder publishes for one source, and the rungs serving it. //! //! The ladder is sized against the source picture, so it has to follow it. -//! `moq_video::encode::publish_capture` opens its source twice by design (once -//! to probe the mode, once when the first subscriber arrives), and a window +//! An on-demand `moq_video::encode::publish_capture` opens its source twice +//! (once to probe the mode, once when the first subscriber arrives), and a window //! capture derives its geometry from the window on every open, so the picture a //! ladder was resolved against is routinely not the one the source ends up //! carrying. A reconnecting publisher and a renegotiated screen share do the diff --git a/rs/moq-video/README.md b/rs/moq-video/README.md index a5cff96caac..3f16c3b9111 100644 --- a/rs/moq-video/README.md +++ b/rs/moq-video/README.md @@ -127,8 +127,9 @@ way, awaited. Public entry points: -- `encode::publish_capture(...)` captures a webcam, encodes it, and publishes on - demand: the track and catalog are advertised up front, but the camera opens +- `encode::publish_capture(...)` captures a webcam, encodes it, and publishes it: + the track and catalog are advertised up front, and the camera stays open + whether or not anyone is watching. With `Capture::on_demand`, the camera opens only while a subscriber is watching and is released when the last one leaves. - `encode::Producer` publishes frames you encoded yourself (`publish(&[Encoded])`), handling the catalog and framing. Each is published at its own timestamp. diff --git a/rs/moq-video/src/capture/mod.rs b/rs/moq-video/src/capture/mod.rs index 5f017b2ad1b..23b1cb229fc 100644 --- a/rs/moq-video/src/capture/mod.rs +++ b/rs/moq-video/src/capture/mod.rs @@ -113,8 +113,9 @@ pub enum Source { /// A screen or window chosen through the Linux system picker. /// - /// Clone the selection for demand-driven reopens; create a new one to ask - /// for a different source. Restoration depends on compositor support. + /// Clone the selection to reopen the same source, as an on-demand capture + /// does for each viewer; create a new one to ask for a different source. + /// Restoration depends on compositor support. #[cfg(all(target_os = "linux", feature = "pipewire"))] Portal(portal::Selection), } @@ -425,6 +426,11 @@ impl Synthetic { pub(crate) fn close(&self) { self.chan.close(); } + + /// Whether the stream reading this device was dropped, as a closed camera is. + pub(crate) fn released(&self) -> bool { + Arc::strong_count(&self.chan) == 1 + } } /// Open the capture source described by `config`. diff --git a/rs/moq-video/src/capture/pipewire.rs b/rs/moq-video/src/capture/pipewire.rs index f67d804b16a..cbceded2b5c 100644 --- a/rs/moq-video/src/capture/pipewire.rs +++ b/rs/moq-video/src/capture/pipewire.rs @@ -10,11 +10,11 @@ //! pull-style pump). //! //! Two screen quirks worth knowing: -//! - `publish_capture` releases the capture while unwatched and reopens it on -//! demand. A fresh portal session would re-prompt the picker every time, so the -//! portal's restore token is kept with the selection and consumed by the -//! next [`open`], which restores the same grant without a dialog (on -//! compositors that support persistence). The token is forgotten when the +//! - An on-demand `publish_capture` releases the capture while unwatched and +//! reopens it for the next viewer. A fresh portal session would re-prompt the +//! picker every time, so the portal's restore token is kept with the selection +//! and consumed by the next [`open`], which restores the same grant without a +//! dialog (on compositors that support persistence). The token is forgotten when the //! compositor ends the stream (the user hit "stop sharing"), so a revoked //! grant is asked for again rather than silently resumed. //! - Compositors only deliver frames on damage, so a static screen would starve @@ -154,7 +154,7 @@ async fn start(config: &Config, capture: Capture, session: Option) return_tx, }) { // Surface a setup failure through the awaiting `open`; a mid-stream - // failure just ends the stream (the encode loop reopens on demand). + // failure just ends the stream (the encode loop reopens it). match state.borrow_mut().geo_tx.take() { Some(tx) => drop(tx.send(Err(e))), None => chan.fail(e), @@ -301,8 +301,8 @@ async fn portal_negotiate(cursor: bool, selection: &portal::Selection) -> Result } /// Closes the portal session when dropped, so the compositor's "screen is being -/// shared" indicator turns off and sessions don't pile up across demand-driven -/// reopens. The close call is async and `Drop` is not, so a task spawned here +/// shared" indicator turns off and sessions don't pile up across an on-demand +/// capture's reopens. The close call is async and `Drop` is not, so a task spawned here /// waits for the guard to drop. Closing does not invalidate the restore token. struct SessionGuard { _close: tokio::sync::oneshot::Sender<()>, @@ -863,7 +863,7 @@ fn run_loop(args: CaptureLoop) -> Result<(), Error> { if done { // The producer ended the stream while our loop was live (the // user hit "stop sharing", or the output or camera went away). - // Forget the screen restore token so the demand-driven reopen + // Forget the screen restore token so the next reopen // asks again instead of silently resuming a grant the user just // revoked. Our own teardown quits the loop before anything // disconnects, so it never reaches this path. diff --git a/rs/moq-video/src/encode/mod.rs b/rs/moq-video/src/encode/mod.rs index 446a8f4de9d..3b797388c7c 100644 --- a/rs/moq-video/src/encode/mod.rs +++ b/rs/moq-video/src/encode/mod.rs @@ -16,8 +16,8 @@ //! //! A [`Producer`] advertises its catalog rendition as soon as the track exists, //! resolved from the encoder itself via [`Config::probe`], so a subscriber can -//! discover a track nothing has encoded yet. That's what makes on-demand -//! encoding possible at all. +//! discover a track nothing has encoded yet. That's what lets an on-demand +//! capture wait for its first viewer at all. //! //! `Capture` (with `capture`) / [`Kind`] / [`Config`] configure them. //! [`Config::preset`] picks a [`Preset`], how the encoder trades latency for diff --git a/rs/moq-video/src/encode/producer.rs b/rs/moq-video/src/encode/producer.rs index 82d344fcdf3..2f647ccbdbf 100644 --- a/rs/moq-video/src/encode/producer.rs +++ b/rs/moq-video/src/encode/producer.rs @@ -1,12 +1,11 @@ //! Publish encoded video frames as a moq video track, with optional capture. //! -//! Encoding is strictly on demand: the track and its catalog rendition are -//! advertised immediately (the rendition is probed from the encoder, since -//! nothing has been encoded yet), and the encoder itself only runs while a -//! subscriber is watching. Capture opens its camera once at startup to learn -//! the mode it negotiates, then keeps it closed between viewers. This mirrors -//! `moq-boy`, which pauses its emulator while -//! [`Demand::wanted`](moq_net::track::Demand::wanted) is `None`. +//! The track and its catalog rendition are advertised immediately (the rendition is probed from +//! the encoder, since nothing has been encoded yet). Capture opens its camera at startup to learn +//! the mode it negotiates, then keeps it open and encodes whether or not anyone is watching, so a +//! late subscriber can reach the recent past within its `max_delay`. With +//! `Capture::on_demand` it instead closes the camera between viewers, encoding only while +//! [`Demand::wanted`](moq_net::track::Demand::wanted) is `Some`. //! //! Capture advertises [`track::Info::refresh`](moq_net::track::Info::refresh) when its encoder //! is sure to honor a cut, and forces a keyframe whenever a joining subscriber asks for a group @@ -309,6 +308,13 @@ pub struct Capture { /// want when the estimate isn't meaningful (a local file, a test harness) or /// unavailable (a publisher that only accepts inbound sessions). pub bandwidth: moq_net::bandwidth::Allocator, + /// Open the source and encode only while a subscriber wants the track, closing it between + /// viewers. + /// + /// Off by default: capture runs for the whole publish, so a late subscriber or fetch finds + /// the recent past in the track's cache. On demand saves the capture and encode while nobody + /// watches, and leaves nothing to fetch from before the first viewer arrived. + pub on_demand: bool, } /// The keyframe requests shared between the [`Control`]s and the [`Driver`]. @@ -375,8 +381,8 @@ impl Control { /// is the longest a group runs, and a joining subscriber already opens one through /// [`track::Info::refresh`](moq_net::track::Info::refresh). Requests between two frames /// coalesce into one keyframe on the next, with no spacing: a caller in a loop pins the - /// encoder at all-IDR. A request while nothing is watching is served by the keyframe every - /// fresh encoder opens with. + /// encoder at all-IDR. A request while an [`on_demand`](Capture::on_demand) capture has + /// nothing watching is served by the keyframe every fresh encoder opens with. /// /// # Errors /// @@ -426,10 +432,11 @@ impl std::fmt::Debug for Driver { impl Driver { /// Run capture until the final control handle drops or the MoQ track ends. /// - /// The source is opened once at startup to probe the mode it negotiates, then released until a - /// subscriber arrives and reopened for as long as one is watching. That one open is what lets - /// the catalog rendition be exact before a single frame is published, so a consumer can size - /// itself against it without waiting for an encoder that may never run. + /// The source is opened at startup to probe the mode it negotiates, then captured from for + /// the whole run, or [`on_demand`](Capture::on_demand), released until a subscriber arrives + /// and reopened for as long as one is watching. That open is what lets the catalog rendition + /// be exact before a single frame is published, so a consumer can size itself against it + /// without waiting for an encoder that may not run yet. /// /// Dropping the final [`Control`] ends it promptly wherever it is waiting, including a camera /// open or permission prompt still in flight, which is abandoned rather than finished. @@ -447,11 +454,10 @@ impl Driver { let clock = catalog.clock(); let controls = requests.consume(); - // Open the camera once to find out what it actually negotiated, since a requested size is - // only a hint (macOS ignores it outright) and the encoder is built from the mode, not the - // request. It closes again immediately: this costs one camera open at startup and buys a - // rendition that says exactly what the stream will carry, rather than one every consumer has - // to treat as provisional. + // Open the camera to find out what it actually negotiated, since a requested size is only a + // hint (macOS ignores it outright) and the encoder is built from the mode, not the request. + // This buys a rendition that says exactly what the stream will carry, rather than one every + // consumer has to treat as provisional. let rendition = async { let camera = source.open(&options.capture).await?; let mut probe_config = encoder::Config::new( @@ -471,7 +477,7 @@ impl Driver { // Learned here so a caller finds out before the first viewer, not on its first cut. learn(&requests, refusal(&mut sink).await?); // The encoder is built from the same defaults, so the probe's GOP is the one it emits. - Ok::<_, Error>((rendition, probe_config.gop.duration(probe_config.framerate))) + Ok::<_, Error>((rendition, probe_config.gop.duration(probe_config.framerate), camera)) }; // Every wait below races the controls closing, so an abandoned publish never holds a camera // or a permission prompt open for an owner that is gone. @@ -484,7 +490,7 @@ impl Driver { } rendition = rendition => rendition, }; - let (rendition, gop) = match rendition { + let (rendition, gop, camera) = match rendition { Ok(probed) => probed, Err(err) => { // A track that already ended has nobody left to tell. @@ -494,6 +500,12 @@ impl Driver { }; let mut producer = Producer::with_track(track, catalog.reserve().with_hls_hint(gop), rendition)?; + // Capturing from the start, the loop keeps the probe's camera rather than reopening it, + // which could mean a second permission prompt or picker. On demand it closes until a viewer. + let mut source = Opened { + stream: (!options.on_demand).then_some(camera), + source, + }; let result = tokio::select! { biased; @@ -518,11 +530,13 @@ impl Driver { } } -/// Capture a webcam and publish it as an on-demand video track. +/// Capture a webcam and publish it as a video track. /// -/// The source is opened once at startup to probe the mode it negotiates, so the -/// catalog rendition is exact before a frame is published, then reopened only -/// while a subscriber is watching. Drop the future to stop it. +/// The source is opened at startup to probe the mode it negotiates, so the +/// catalog rendition is exact before a frame is published, then captured and +/// encoded whether or not anyone is watching. Set +/// [`Capture::on_demand`] to capture only while a subscriber is watching +/// instead. Drop the future to stop it. /// /// Returns when the broadcast is dropped (the track stops being announced) or /// the capture fails. Frames are stamped on the catalog's clock, so a @@ -564,6 +578,23 @@ trait CaptureSource { #[cfg(feature = "capture")] struct DeviceSource; +/// Hands out a stream that is already open before opening `source` again. +#[cfg(feature = "capture")] +struct Opened { + stream: Option, + source: S, +} + +#[cfg(feature = "capture")] +impl CaptureSource for Opened { + async fn open(&mut self, config: &capture::Config) -> Result { + match self.stream.take() { + Some(stream) => Ok(stream), + None => self.source.open(config).await, + } + } +} + #[cfg(feature = "capture")] impl CaptureSource for DeviceSource { async fn open(&mut self, config: &capture::Config) -> Result { @@ -665,34 +696,63 @@ fn capture_stopped(producer: &mut Producer) -> Result<(), Error> { producer.discontinuity() } -/// Wait until some viewer wants the track, or none does. +/// Wait until capture should run (`run`) or release its source (`!run`). On demand it runs +/// while some viewer wants the track; otherwise it runs, and never releases, while the track +/// is open. Errors once the track closes. #[cfg(feature = "capture")] -async fn until_wanted(demand: &moq_net::track::Demand, wanted: bool) -> Result<(), moq_net::Error> { +async fn until_capturing(demand: &moq_net::track::Demand, on_demand: bool, run: bool) -> Result<(), moq_net::Error> { let mut demand = demand.clone(); - while demand.wanted_changed().await?.is_some() != wanted {} - Ok(()) + loop { + let wanted = demand.wanted_changed().await?.is_some(); + if (wanted || !on_demand) == run { + return Ok(()); + } + } } -// Race source setup or an encode against the last viewer leaving, so an unwatched track -// stops waiting promptly. +// Race source setup or an encode against capture releasing its source, so an on-demand track +// its last viewer left, or a track that ended, stops waiting promptly. `None` when it released; +// the next `until_capturing` reports a track that ended. #[cfg(feature = "capture")] async fn wait_capture( demand: &moq_net::track::Demand, + on_demand: bool, work: impl std::future::Future>, ) -> Result, Error> { tokio::select! { biased; - res = until_wanted(demand, false) => { - if let Err(err) = res { - log_track_ended(err); - } - Ok(None) - } + _ = until_capturing(demand, on_demand, false) => Ok(None), res = work => res.map(Some), } } -/// Async capture/encode loop. Opens the camera while at least one viewer is +/// Return an encoder nobody watches to its ceiling, and start its rate control over from there +/// for the next viewer. Its groups wait in the cache rather than being paced to a subscriber, +/// and a past viewer's grant is stale by the time the next one arrives. +#[cfg(feature = "capture")] +async fn rest( + encoder: &mut Sink, + rate: &mut RateControl, + claim: &moq_net::bandwidth::Reservation, + ceiling: moq_net::bandwidth::Rate, +) { + // Retired: the encoder cannot retune, or the grant is gone and it holds its rate. + if rate.is_none() { + return; + } + match encoder.set_bitrate(ceiling).await { + Ok(()) => *rate = Some(rate::Follower::new(claim.consume(), rate::Policy::new(ceiling))), + Err(Error::BitrateUnsupported(name)) => { + tracing::warn!(encoder = name, "encoder cannot follow the bandwidth estimate"); + *rate = None; + } + // The follower still matches the rate the encoder kept, so the next viewer follows on from it. + Err(err) => tracing::warn!(error = %err, bitrate = ceiling.as_bps(), "failed to rest the encoder bitrate"), + } +} + +/// Async capture/encode loop. Opens the camera and keeps it open, or +/// [`on_demand`](Capture::on_demand), opens it while at least one viewer is /// watching and releases it when the last one leaves. It never returns on its /// own when the controls close; [`Driver::run`] races it against that instead. /// @@ -716,21 +776,21 @@ async fn capture_loop( let mut reservation: Option = None; let counted = requests.consume(); let capture = &options.capture; + let on_demand = options.on_demand; // The published track's demand, followed onto the next track when a frame moves the rendition. let mut demand = producer.demand(); - let demand = &mut demand; loop { - // Idle until a viewer subscribes; the track ending is a clean exit. The - // catalog rendition was published when the track was created, so a - // subscriber can get here without a frame ever having been encoded. - if let Err(err) = until_wanted(demand, true).await { + // On demand, idle until a viewer subscribes. The catalog rendition was published when + // the track was created, so a subscriber can get here without a frame ever having been + // encoded. Either way the track ending is a clean exit. + if let Err(err) = until_capturing(&demand, on_demand, true).await { log_track_ended(err); return Ok(()); } // Open the camera and an encoder sized to its negotiated mode. - let Some(mut camera) = wait_capture(demand, source.open(capture)).await? else { + let Some(mut camera) = wait_capture(&demand, on_demand, source.open(capture)).await? else { continue; }; // Capture timestamps use a private monotonic timeline. Sample both clocks @@ -753,7 +813,7 @@ async fn capture_loop( // No cut on reopen: a fresh encoder opens with a keyframe on every backend, // so the viewer whose subscription reopened the camera can decode from the // first frame regardless, and a backend that cannot cut still captures. - let Some(mut encoder) = wait_capture(demand, Sink::open(&encoder_config)).await? else { + let Some(mut encoder) = wait_capture(&demand, on_demand, Sink::open(&encoder_config)).await? else { continue; }; // A reopen can land on another backend than the probe did, so the controls @@ -767,7 +827,7 @@ async fn capture_loop( // larger mode below what it can send, or keep claiming room a smaller one no // longer needs. let ceiling = encoder_config.resolved_bitrate(); - let claim = reservation.get_or_insert_with(|| options.bandwidth.reserve(demand, ceiling)); + let claim = reservation.get_or_insert_with(|| options.bandwidth.reserve(&demand, ceiling)); claim.update(ceiling); // Rate control is per encoder: this one opened at the configured bitrate, @@ -777,25 +837,37 @@ async fn capture_loop( let mut rate = Some(rate::Follower::new(claim.consume(), rate::Policy::new(ceiling))); // Per encoder, since a fresh one opens with a keyframe that serves every earlier ask. let mut forced = Cuts::new(served); - // Set when a frame moves the rendition to a newly named track. - let mut moved = false; + // Rate control follows the grant only while someone watches. The first poll of `watch` + // reports the current demand. + let mut watch = demand.clone(); + let mut watched = false; loop { - // Race the next frame against the last viewer leaving so we release the - // camera promptly when demand drops. `biased` checks demand first so an - // unwatched track stops before reading another frame. + // Race the next frame against demand, so an on-demand capture releases the + // camera promptly when the last viewer leaves. `biased` checks demand first + // so an unwatched on-demand track stops before reading another frame. let frame = tokio::select! { biased; - res = until_wanted(demand, false) => { - if let Err(err) = res { - log_track_ended(err); - return Ok(()); + wanted = watch.wanted_changed() => { + let wanted = match wanted { + Ok(wanted) => wanted.is_some(), + Err(err) => { + log_track_ended(err); + return Ok(()); + } + }; + if on_demand && !wanted { + break; // no viewers: release the camera, then wait for one + } + if watched && !wanted { + rest(&mut encoder, &mut rate, claim, ceiling).await; } - break; // no viewers: release the camera, then wait for one + watched = wanted; + continue; } // Retune between frames rather than mid-encode, and only when // the policy says the target actually moved. - target = next_target(&mut rate) => { + target = next_target(&mut rate), if watched => { apply_target(&mut encoder, &mut rate, target).await; continue; } @@ -813,30 +885,32 @@ async fn capture_loop( // Never refused: `Control::cut` stops counting once this encoder is known to. encoder.cut().await?; } - let Some(encoded) = wait_capture(demand, encoder.encode(frame)).await? else { + let Some(encoded) = wait_capture(&demand, on_demand, encoder.encode(frame)).await? else { break; }; producer.publish(&encoded)?; // A frame whose decoder identity differs from the probed one (a reopen onto another - // mode or backend) publishes under a new name and ends this track. Its viewers find - // the new name in the catalog, so idle until one subscribes there. + // mode or backend) publishes under a new name and ends this track. Capture follows it + // there with a claim against the new track, which nobody watches yet: its viewers find + // it in the catalog, so on demand the camera closes until one subscribes there. if producer.track().name() != demand.name() { - moved = true; - break; + demand = producer.demand(); + tracing::info!(track = %demand.name(), "capture moved to a new track"); + *claim = options.bandwidth.reserve(&demand, ceiling); + if on_demand { + break; + } + watch = demand.clone(); + watched = false; + rest(&mut encoder, &mut rate, claim, ceiling).await; } } - // Drop the camera (LED off) and encoder before waiting for the next viewer. + // Drop the camera (LED off) and encoder before opening another. drop(camera); drop(encoder); capture_stopped(producer)?; tracing::info!("capture stopped; released source"); - if moved { - *demand = producer.demand(); - tracing::info!(track = %demand.name(), "capture moved to a new track"); - // The claim watches the old track's demand, so take a new one against the new track. - reservation = None; - } } } @@ -1232,8 +1306,10 @@ mod tests { // Local: a capture stream is `!Send` on macOS, so each test runs on a `LocalSet`. let task = tokio::task::spawn_local(async move { let mut source = Opens(rx); + // Each subscription opens the camera, so a fixture can idle and restart it. let options = Capture { kind: encoder::Kind::Software, + on_demand: true, ..Capture::default() }; // Held, so the loop has a channel to report each encoder's cut support into. @@ -1677,92 +1753,81 @@ mod tests { async fn capture_groups(steps: Vec) -> (moq_net::track::Info, Vec>) { use moq_mux::container::Container as _; - // Off Apple, encoding waits on an OS thread. Keep Tokio from auto-advancing - // the paused clock to a timer deadline while that thread answers. - let (hold, released) = tokio::sync::oneshot::channel::<()>(); - let clock = tokio::task::spawn_blocking(move || { - let _ = released.blocking_recv(); - }); let started = tokio::time::Instant::now(); - // Capture streams are !Send on macOS. - let result = tokio::task::LocalSet::new() - .run_until(async { - let mut broadcast = moq_net::broadcast::Info::new().produce(); - let consumer = broadcast.consume(); - let catalog = - moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); - let options = Capture { - kind: encoder::Kind::Software, - // Room for noise, which rate control would otherwise skip frames of. - bitrate: Some(moq_net::bandwidth::Rate::from_mbps(4)), - ..Capture::default() - }; - let (control, driver) = Control::new(broadcast.clone(), catalog, options).unwrap(); - - // Ten nominal fps makes the default two-second GOP twenty frames, which every - // script below stays inside. - let rate = crate::FrameRate::new(10, 1).unwrap(); - let (_probe, probe) = Synthetic::open(crate::Size::new(320, 240), rate); - let (camera, stream) = Synthetic::open(crate::Size::new(320, 240), rate); - let run = tokio::task::spawn_local(driver.run_with(Streams::new([probe, stream]))); - let mut catalog = - moq_mux::catalog::Consumer::<()>::new(&consumer, moq_mux::catalog::CatalogFormat::Hang) - .await - .unwrap(); - let (name, mut subscriber, format) = loop { - let snapshot = catalog.next().await.unwrap().expect("a catalog"); - if let Some((name, rendition)) = snapshot.video.renditions.iter().next() { - break ( - name.clone(), - consumer.track(name).unwrap().subscribe(None).await.unwrap(), - moq_mux::catalog::hang::Container::try_from(rendition).unwrap(), - ); - } - }; - let info = subscriber.info().clone(); - - let (published, mut received) = tokio::sync::mpsc::unbounded_channel(); - let collect = tokio::task::spawn_local(async move { - let mut groups = Vec::new(); - while let Some(mut group) = subscriber.recv_group().await.unwrap() { - let mut frames = Vec::new(); - while let Some(wire) = group.read_frame().await.unwrap() { - for frame in format.decode(wire).unwrap() { - // A duration marker closes the group but is not an encoded picture. - if format.end(&frame).is_none() { - frames.push(frame.payload.len()); - published.send(()).unwrap(); - } + let result = local(async { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = + moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let options = Capture { + kind: encoder::Kind::Software, + // Room for noise, which rate control would otherwise skip frames of. + bitrate: Some(moq_net::bandwidth::Rate::from_mbps(4)), + ..Capture::default() + }; + let (control, driver) = Control::new(broadcast.clone(), catalog, options).unwrap(); + + // Ten nominal fps makes the default two-second GOP twenty frames, which every + // script below stays inside. The capture keeps the probe's camera. + let rate = crate::FrameRate::new(10, 1).unwrap(); + let (camera, stream) = Synthetic::open(crate::Size::new(320, 240), rate); + let run = tokio::task::spawn_local(driver.run_with(Streams::new([stream]))); + let mut catalog = + moq_mux::catalog::Consumer::<()>::new(&consumer, moq_mux::catalog::CatalogFormat::Hang) + .await + .unwrap(); + let (name, mut subscriber, format) = loop { + let snapshot = catalog.next().await.unwrap().expect("a catalog"); + if let Some((name, rendition)) = snapshot.video.renditions.iter().next() { + break ( + name.clone(), + consumer.track(name).unwrap().subscribe(None).await.unwrap(), + moq_mux::catalog::hang::Container::try_from(rendition).unwrap(), + ); + } + }; + let info = subscriber.info().clone(); + + let (published, mut received) = tokio::sync::mpsc::unbounded_channel(); + let collect = tokio::task::spawn_local(async move { + let mut groups = Vec::new(); + while let Some(mut group) = subscriber.recv_group().await.unwrap() { + let mut frames = Vec::new(); + while let Some(wire) = group.read_frame().await.unwrap() { + for frame in format.decode(wire).unwrap() { + // A duration marker closes the group but is not an encoded picture. + if format.end(&frame).is_none() { + frames.push(frame.payload.len()); + published.send(()).unwrap(); } } - groups.push(frames); } - groups - }); - - let mut joined = Vec::new(); - let mut millis = 0; - for step in steps { - match step { - Step::Frame(surface) => { - camera.push_native(surface, us(millis * 1000)); - millis += 10; - // Wait for delivery before the next step, which also keeps the - // capture's latest slot from being replaced. - received.recv().await.expect("the frame reached the subscriber"); - } - Step::Cut => control.cut().unwrap(), - Step::Join => joined.push(consumer.track(&name).unwrap().subscribe(None).await.unwrap()), + groups.push(frames); + } + groups + }); + + let mut joined = Vec::new(); + let mut millis = 0; + for step in steps { + match step { + Step::Frame(surface) => { + camera.push_native(surface, us(millis * 1000)); + millis += 10; + // Wait for delivery before the next step, which also keeps the + // capture's latest slot from being replaced. + received.recv().await.expect("the frame reached the subscriber"); } + Step::Cut => control.cut().unwrap(), + Step::Join => joined.push(consumer.track(&name).unwrap().subscribe(None).await.unwrap()), } - drop(control); - run.await.unwrap().unwrap(); - (info, collect.await.unwrap()) - }) - .await; + } + drop(control); + run.await.unwrap().unwrap(); + (info, collect.await.unwrap()) + }) + .await; assert_eq!(started.elapsed(), Duration::ZERO); - drop(hold); - clock.await.unwrap(); result } @@ -1835,6 +1900,207 @@ mod tests { } } + /// Run `test` on a `LocalSet`, since capture streams are `!Send` on macOS. + /// + /// Off Apple, encoding waits on an OS thread, so a blocking task keeps Tokio from + /// auto-advancing the paused clock to a timer deadline while that thread answers. + async fn local(test: impl std::future::Future) -> T { + let (hold, released) = tokio::sync::oneshot::channel::<()>(); + let clock = tokio::task::spawn_blocking(move || { + let _ = released.blocking_recv(); + }); + let result = tokio::task::LocalSet::new().run_until(test).await; + drop(hold); + clock.await.unwrap(); + result + } + + /// The name of the first video rendition `consumer`'s catalog publishes. + async fn rendition_name(consumer: &moq_net::broadcast::Consumer) -> String { + let mut catalog = moq_mux::catalog::Consumer::<()>::new(consumer, moq_mux::catalog::CatalogFormat::Hang) + .await + .unwrap(); + loop { + let snapshot = catalog.next().await.unwrap().expect("a catalog"); + if let Some(name) = snapshot.video.renditions.keys().next() { + return name.clone(); + } + } + } + + /// Nobody watching, the capture still encodes, from the probe's camera rather than a + /// second open, and a subscriber arriving later reads what it wrote meanwhile. + #[tokio::test(start_paused = true)] + async fn an_unwatched_capture_keeps_its_past() { + local(async { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = + moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let options = Capture { + kind: encoder::Kind::Software, + ..Capture::default() + }; + let (control, driver) = Control::new(broadcast.clone(), catalog, options).unwrap(); + let (camera, stream) = camera(); + let source = Streams::new([stream]); + let opens = source.opens.clone(); + let run = tokio::task::spawn_local(driver.run_with(source)); + + let name = rendition_name(&consumer).await; + let mut track = consumer.track(&name).unwrap(); + for group in 0..2 { + if group > 0 { + control.cut().unwrap(); + } + camera.push_native(surface(), us(group * 100_000)); + while track.largest().await.expect("the track is live").group < group {} + } + assert_eq!( + opens.load(Ordering::SeqCst), + 1, + "the capture reopened the probe's camera" + ); + assert!(!camera.released(), "the capture released its camera"); + + let mut subscriber = consumer + .track(&name) + .unwrap() + .subscribe(moq_net::track::Subscription::default().with_max_delay(Duration::from_secs(5))) + .await + .unwrap(); + for sequence in 0..2 { + let group = subscriber.recv_group().await.unwrap().expect("a cached group"); + assert_eq!(group.sequence, sequence); + } + + drop(control); + run.await.unwrap().unwrap(); + }) + .await + } + + /// On demand, the probe's camera closes at once, and only a viewer opens it again. + #[tokio::test(start_paused = true)] + async fn on_demand_opens_the_camera_for_a_viewer() { + local(async { + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let consumer = broadcast.consume(); + let catalog = + moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let options = Capture { + kind: encoder::Kind::Software, + on_demand: true, + ..Capture::default() + }; + let (control, driver) = Control::new(broadcast.clone(), catalog, options).unwrap(); + let (probed, probe) = camera(); + let (camera, stream) = camera(); + let source = Streams::new([probe, stream]); + let opens = source.opens.clone(); + let run = tokio::task::spawn_local(driver.run_with(source)); + + let name = rendition_name(&consumer).await; + until(|| probed.released()).await; + assert_eq!(opens.load(Ordering::SeqCst), 1, "only the probe opened the camera"); + + let mut subscriber = consumer.track(&name).unwrap().subscribe(None).await.unwrap(); + camera.push_native(surface(), us(0)); + subscriber.recv_group().await.unwrap().expect("a group"); + assert_eq!(opens.load(Ordering::SeqCst), 2, "the viewer opened the camera"); + + drop(subscriber); + until(|| camera.released()).await; + drop(control); + run.await.unwrap().unwrap(); + }) + .await + } + + /// Drive `run` until the bitrates the probe encoder accepted satisfy `done`, returning them. + async fn settle( + mut run: std::pin::Pin<&mut impl std::future::Future>>, + done: impl Fn(&[u64]) -> bool, + ) -> Vec { + let mut accepted = Vec::new(); + loop { + accepted.extend(probe::bitrates()); + if done(&accepted) { + return accepted; + } + tokio::select! { + res = &mut run => panic!("the capture ended: {res:?}"), + () = tokio::task::yield_now() => {} + } + } + } + + /// A watched capture follows its grant. Once nobody watches, it returns to its ceiling + /// and ignores the grant, and the next viewer's grant applies afresh. + #[tokio::test(start_paused = true)] + #[expect( + clippy::await_holding_lock, + reason = "the guard only keeps other tests out of the process-wide probe log, and this runtime has one thread, so holding it across an await blocks nothing" + )] + async fn an_unwatched_capture_rests_at_its_ceiling() { + use moq_net::bandwidth::Rate; + + let _probe = probe::exclusive(); + let ceiling = Rate::from_mbps(4); + let estimate = moq_net::bandwidth::Producer::new(); + estimate.set(Some(Rate::from_mbps(1))).unwrap(); + + let mut broadcast = moq_net::broadcast::Info::new().produce(); + let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); + let track = broadcast + .create_track("video", catalog.track_info(hang::catalog::PRIORITY.video)) + .unwrap(); + let watched = track.clone(); + let mut rendition = Config::new(320, 240, crate::FrameRate::new(30, 1).unwrap()); + rendition.kind = encoder::Kind::Software; + let mut producer = Producer::with_track( + track, + catalog + .reserve() + .with_hls_hint(rendition.gop.duration(rendition.framerate)), + rendition.probe().await.unwrap(), + ) + .unwrap(); + + let (_camera, stream) = camera(); + let mut source = Streams::new([stream]); + let options = Capture { + kind: encoder::Kind::Named(probe::NAME.into()), + bitrate: Some(ceiling), + bandwidth: moq_net::bandwidth::Allocator::new(estimate.consume()), + ..Capture::default() + }; + let controls = kio::Producer::new(Requests::default()); + let requests = controls.weak(); + let clock = catalog.clock(); + let mut run = std::pin::pin!(capture_loop(&mut producer, &mut source, &options, &clock, &requests)); + + let viewer = watched.subscribe(None); + let followed = settle(run.as_mut(), |set| !set.is_empty()).await; + assert!(followed.iter().all(|&bps| bps < ceiling.as_bps()), "{followed:?}"); + + drop(viewer); + let rested = settle(run.as_mut(), |set| set.contains(&ceiling.as_bps())).await; + assert_eq!(rested, [ceiling.as_bps()]); + + // Unwatched, a grant change retunes nothing, however long it holds. + estimate.set(Some(Rate::from_kbps(500))).unwrap(); + tokio::select! { + res = &mut run => panic!("the capture ended: {res:?}"), + () = tokio::time::sleep(Duration::from_secs(60)) => {} + } + assert_eq!(probe::bitrates(), Vec::::new()); + + let _viewer = watched.subscribe(None); + let resumed = settle(run.as_mut(), |set| !set.is_empty()).await; + assert!(resumed.iter().all(|&bps| bps <= 500_000), "{resumed:?}"); + } + /// Drop the controls while the driver sits in its `opens`th open, which never finishes. async fn abandon( control: Control, @@ -1876,6 +2142,7 @@ mod tests { let catalog = moq_mux::catalog::Producer::new(&mut broadcast, moq_mux::catalog::Config::default()).unwrap(); let options = Capture { kind: encoder::Kind::Software, + on_demand: true, ..Capture::default() }; let (control, driver) = Control::new(broadcast.clone(), catalog.clone(), options).unwrap(); diff --git a/rs/moq-video/src/lib.rs b/rs/moq-video/src/lib.rs index 695ec908d23..41691fa467a 100644 --- a/rs/moq-video/src/lib.rs +++ b/rs/moq-video/src/lib.rs @@ -26,10 +26,10 @@ //! VideoToolbox / Media Foundation / NVENC / VAAPI / V4L2) or H.265 //! (VideoToolbox / Media Foundation / NVENC). Entry points: //! - `encode::publish_capture` captures a webcam and publishes it (turnkey). -//! It encodes strictly on demand: the track and catalog are advertised up -//! front (the camera opens once at startup so they can be exact), and the -//! encoder runs only while a subscriber is watching. Requires the `capture` -//! feature. +//! The track and catalog are advertised up front (the camera opens at +//! startup so they can be exact), and the camera stays open and encodes +//! whether or not anyone is watching, unless `encode::Capture::on_demand` +//! limits it to while a subscriber is. Requires the `capture` feature. //! - [`encode::Encoder`] encodes [`Frame`]s you supply (from capture, a //! decoder, or your own pixels via [`Surface::rgba`]) and //! [`encode::Producer`] publishes the results.