From 34db31ed1c7ef02ed7a97d989e4114ffa30ca1cc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 19:49:23 -0800 Subject: [PATCH 1/3] chore(quest): claim quest/m1/precut-fixes/capture-audio-reopen.md Quest-Claim: 536b637b9d76caf1 From d32ff034019b24dd3a96e794d48f1c9b58780e1e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 21:16:13 -0800 Subject: [PATCH 2/3] fix(cli): moq import capture reopens a failed microphone and keeps the video The audio future returned the capture error and try_join! failed the whole command, video included. The CLI now reopens the capture and a new audio track with release's 0.5 s to 4 s backoff, gives up after 10 s of failures with no healthy capture between them, and then ends only the audio track while the video keeps publishing. A first open that fails still ends the import at once. A failed track drains its retained audio in the background so the reopen is not held up. Co-Authored-By: Claude Opus 5.5 --- doc/bin/cli.md | 7 +- quest/m1/precut-fixes/capture-audio-reopen.md | 30 --- quest/m2/export-replaced-rendition.md | 30 +++ rs/moq-cli/src/publish.rs | 206 ++++++++++++++++-- 4 files changed, 222 insertions(+), 51 deletions(-) delete mode 100644 quest/m1/precut-fixes/capture-audio-reopen.md create mode 100644 quest/m2/export-replaced-rendition.md diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 2a3bead4c16..9bc61b1445a 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -246,8 +246,11 @@ moq ... import capture --audio on # encode audio continuo Video goes through the platform hardware encoder, with an H.264 software fallback when built with `openh264`, and audio is Opus; [moq-video](/lib/rs/moq-video) lists the -backends. An audio device that fails or stops ends the import with its source -error; restart the command after the device is available again. Audio appears +backends. An audio device that fails mid-session, such as a USB microphone +re-enumerating, is reopened every 0.5 to 4 seconds as a new audio track. One +still failing after 10 seconds ends only the audio track: video keeps +publishing, and the import exits with the audio error once it stops. A device +that fails to open at startup ends the import at once. Audio appears in the catalog once its device opens; a microphone permission prompt leaves other media available. The camera and microphone stay open while publishing, so a late diff --git a/quest/m1/precut-fixes/capture-audio-reopen.md b/quest/m1/precut-fixes/capture-audio-reopen.md deleted file mode 100644 index b1ed2d90fd6..00000000000 --- a/quest/m1/precut-fixes/capture-audio-reopen.md +++ /dev/null @@ -1,30 +0,0 @@ -# [S] moq import capture keeps publishing when its microphone fails - -## Goal - -`moq import capture` survives a microphone that fails mid-session (a USB mic -re-enumerating, another app taking the default input, a sample-rate change): -the audio track reopens the device with backoff, as `release` did, and the -video track never stops because audio failed. - -## Plan - -Found by the 2026-11-17 pre-cut correctness audit; confirmed from source. In -`rs/moq-cli/src/publish.rs` the audio future returns the capture error and -`tokio::try_join!` fails the whole command, video included. -[moq-dev/moq#7706](https://github.com/moq-dev/moq/pull/7706) removed the -library's automatic reopen on purpose (`rs/moq-audio/MIGRATE.md`: "no -automatic device reopen"), but nothing moved the reopen into the CLI. -`release` recovered within 0.5 to 4 s. - -Decided 2026-11-17 (maintainer, [moq-dev/moq#9010](https://github.com/moq-dev/moq/pull/9010#issuecomment-6959731587)): ✅ the CLI reopens the capture and its publication in a loop -with `release`'s backoff, keeping #7706's decision that the library does not -(rejected: putting the reopen back in moq-audio, which reverses that -decision; ending only the audio track with no reopen). A failure that persists past the backoff ends only the audio -track, with its error, and the video keeps running. - -Public API: none (CLI behavior). Wire: none. - -Caused by, for the fix PR's `## Caused by`: - -- [moq-dev/moq#7706](https://github.com/moq-dev/moq/pull/7706) - removed moq-audio's device reopen without moving it to the CLI diff --git a/quest/m2/export-replaced-rendition.md b/quest/m2/export-replaced-rendition.md new file mode 100644 index 00000000000..1f3bdb51eba --- /dev/null +++ b/quest/m2/export-replaced-rendition.md @@ -0,0 +1,30 @@ +# [M] Container exports outlive a rendition that fails and is replaced + +## Goal + +`moq export ts`, `fmp4`, `mkv`, and `flv` keep writing when one rendition's +track ends with an error and the catalog lists a replacement, as `moq import +capture` does when it reopens a failed microphone as a new audio track. The +export follows the replacement, or keeps the other media and says so, instead +of ending with the failed track's error. + +## Plan + +Found while validating the CLI's microphone reopen: with +`moq import capture -- export ts`, the reopen kept the video publishing, yet +the export ended with `audio capture: The requested audio device is not +available` once the old audio track failed +(`rs/moq-mux/src/container/consumer.rs`, "track failed; ending the reader"). +An `export h264` of the same broadcast kept writing, since it reads only video. + +Before that quest, the failed microphone ended the whole publish, so no +export outlived it; the reopen is what exposes this. + +Recommended: a rendition whose track fails while the catalog still lists the +broadcast ends only that rendition's reader, and the export picks up a new +rendition of the same medium the way it handles a catalog change; the export +fails only when no rendition of a selected medium remains. Check what the +catalog-identity republish ([moq-dev/moq#7110](https://github.com/moq-dev/moq/pull/7110)) +already does for a cleanly finished track and reuse it. + +Public API: none expected. Wire: none. diff --git a/rs/moq-cli/src/publish.rs b/rs/moq-cli/src/publish.rs index a8cf183ff0b..7927030da1b 100644 --- a/rs/moq-cli/src/publish.rs +++ b/rs/moq-cli/src/publish.rs @@ -679,6 +679,7 @@ impl Publish { } => { drop(gate); let mut report = self.report; + let has_video = video.is_some(); // Each enabled medium publishes its own track onto the shared // broadcast + catalog. Both stamp frames on the catalog's // advertised clock, so HLS/DASH wall times match the mapping on @@ -702,29 +703,31 @@ impl Publish { }; let audio_fut = async move { match audio { - Some((config, encode, retain)) => { - let capture = moq_audio::capture::Producer::open(config).await?; - let mut options = moq_audio::encode::Capture::new(capture.consume()); - options.encode = encode; - options.retain = retain; - let (_publication, driver) = - moq_audio::encode::publish_capture(broadcast, catalog, options)?; - let mut publish = std::pin::pin!(driver.run()); - tokio::select! { - result = capture.run() => { - // Let the encoder serve its retained audio, then end its - // track with the source error, before its future is dropped. - let published = publish.await; - result.and(published).map_err(anyhow::Error::from) - } - result = &mut publish => result.map_err(anyhow::Error::from), + Some(audio) => publish_audio(broadcast, catalog, audio).await, + None => Ok(Ok(())), + } + }; + + // A video failure ends the publish. An audio failure that outlasted its reopens + // ends only the audio track: the video keeps publishing, and the command reports + // the audio failure once the video ends. + let capture = async { + let mut video_fut = std::pin::pin!(video_fut); + let mut audio_fut = std::pin::pin!(audio_fut); + tokio::select! { + result = &mut video_fut => { + result?; + audio_fut.await? + } + result = &mut audio_fut => { + let ended = result?; + if let Err(err) = &ended && has_video { + tracing::error!(error = %format_args!("{err:#}"), "audio capture ended; video keeps publishing"); } + video_fut.await.and(ended) } - None => Ok(()), } }; - - let capture = async { tokio::try_join!(video_fut, audio_fut).map(|_| ()) }; // Written beside the encoders, so a capture waiting on its first viewer still reports. let result = match report.as_mut() { Some(report) => tokio::select! { @@ -744,6 +747,119 @@ impl Publish { } } +/// Publish the captured audio, reopening the device and its track when the capture fails +/// mid-session, as a USB microphone re-enumerating or a sample-rate change does. +/// +/// moq-audio never reopens a failed device, so this loop does. Each reopen publishes a new track, +/// since the failed one ends with its error. Returns once the track closes, or with the last +/// failure once failures outlast [`Reopen::BUDGET`]. +/// +/// A first open or publication that fails is a setup problem, such as a wrong device id or a +/// denied permission, so it fails the whole publish at once: the outer error. The inner one ends +/// only the audio. +#[cfg(feature = "capture")] +async fn publish_audio( + broadcast: moq_net::broadcast::Producer, + catalog: moq_mux::catalog::Producer, + (config, encode, retain): (moq_audio::capture::Config, moq_audio::encode::Options, bool), +) -> anyhow::Result> { + // A failed track keeps serving its retained audio until it is served or expires, up to + // `--max-age`, without holding up the reopen. Dropping the set cancels them. + let mut draining = tokio::task::JoinSet::new(); + let mut reopen = Reopen::default(); + let mut published = false; + let result = loop { + let mut live = None; + let attempt = async { + let capture = moq_audio::capture::Producer::open(config.clone()).await?; + live = Some(kio::clock::Instant::now()); + let mut options = moq_audio::encode::Capture::new(capture.consume()); + options.encode = encode.clone(); + options.retain = retain; + let (_publication, driver) = + moq_audio::encode::publish_capture(broadcast.clone(), catalog.clone(), options)?; + published = true; + let mut publish = Box::pin(driver.run()); + tokio::select! { + result = capture.run() => { + draining.spawn(publish); + result + } + result = &mut publish => result, + } + }; + let err = match attempt.await { + Ok(()) => break Ok(()), + Err(err) => anyhow::Error::from(err), + }; + if !published { + return Err(err); + } + while draining.try_join_next().is_some() {} + match reopen.failed(live, kio::clock::Instant::now()) { + Some(wait) => { + tracing::warn!(error = %format_args!("{err:#}"), ?wait, "audio capture failed; reopening"); + kio::clock::sleep(wait).await; + } + None => break Err(err.context(format!("audio capture kept failing for {:?}", Reopen::BUDGET))), + } + }; + while let Some(joined) = draining.join_next().await { + if let Err(err) = joined + && err.is_panic() + { + std::panic::resume_unwind(err.into_panic()); + } + } + Ok(result) +} + +/// When to reopen a failed audio capture, and when to give up on it. +#[cfg(feature = "capture")] +struct Reopen { + /// The wait before the next reopen. + delay: std::time::Duration, + /// When the current run of failures began, with no healthy capture since. + failing: Option, +} + +#[cfg(feature = "capture")] +impl Default for Reopen { + fn default() -> Self { + Self { + delay: Self::MIN, + failing: None, + } + } +} + +#[cfg(feature = "capture")] +impl Reopen { + /// The first wait. It covers a USB device re-enumerating. + const MIN: std::time::Duration = std::time::Duration::from_millis(500); + /// The longest wait, so a returning device is picked up within seconds. + const MAX: std::time::Duration = std::time::Duration::from_secs(4); + /// A capture that ran this long recovered, so its failure starts a fresh run of retries. + const HEALTHY: std::time::Duration = std::time::Duration::from_secs(1); + /// How long failures may persist before the audio track ends. + const BUDGET: std::time::Duration = std::time::Duration::from_secs(10); + + /// Record a failure at `now` of a capture live since `live` (`None` when it never opened), + /// returning the wait before the next reopen, or `None` to give up. + fn failed(&mut self, live: Option, now: kio::clock::Instant) -> Option { + if live.is_some_and(|live| now.saturating_duration_since(live) >= Self::HEALTHY) { + *self = Self::default(); + } + let failing = *self.failing.get_or_insert(now); + if now.saturating_duration_since(failing) >= Self::BUDGET { + return None; + } + let wait = self.delay; + self.delay = (self.delay * 2).min(Self::MAX); + Some(wait) + } +} + /// What a stdin import does around its media: announce, report, and retain the end. #[derive(Default)] struct Publication { @@ -1220,6 +1336,58 @@ pub(crate) mod tests { use super::*; + /// Each wait in a run of reopens that never open, from the first failure until giving up. + #[cfg(feature = "capture")] + fn reopen_waits(reopen: &mut Reopen, mut now: kio::clock::Instant) -> Vec { + let mut waits = Vec::new(); + while let Some(wait) = reopen.failed(None, now) { + waits.push(wait); + now = now.saturating_add(wait); + } + waits + } + + #[cfg(feature = "capture")] + #[test] + fn a_failing_audio_capture_backs_off_then_gives_up() { + let ms = Duration::from_millis; + let waits = reopen_waits(&mut Reopen::default(), kio::clock::Instant::ZERO); + assert_eq!(waits, [ms(500), ms(1000), ms(2000), ms(4000), ms(4000)]); + assert!(waits.iter().sum::() >= Reopen::BUDGET); + } + + #[cfg(feature = "capture")] + #[test] + fn a_healthy_audio_capture_restarts_its_reopens() { + let start = kio::clock::Instant::ZERO; + let mut reopen = Reopen::default(); + reopen_waits(&mut reopen, start); + + // A capture that ran past healthy recovered, however long the failures before it. + let live = start.saturating_add(Duration::from_secs(60)); + let failed = live.saturating_add(Reopen::HEALTHY); + assert_eq!(reopen.failed(Some(live), failed), Some(Reopen::MIN)); + assert_eq!( + reopen.failed(None, failed.saturating_add(Reopen::MIN)), + Some(Reopen::MIN * 2) + ); + } + + #[cfg(feature = "capture")] + #[test] + fn a_flapping_audio_capture_still_gives_up() { + // Opens that fail sooner than healthy are no recovery, so the budget still runs out. + let mut reopen = Reopen::default(); + let mut now = kio::clock::Instant::ZERO; + let mut attempts = 0; + while let Some(wait) = reopen.failed(Some(now), now.saturating_add(Reopen::HEALTHY / 2)) { + now = now.saturating_add(Reopen::HEALTHY / 2 + wait); + attempts += 1; + assert!(attempts < 100, "a flapping capture never gave up"); + } + assert_eq!(attempts, 5); + } + #[cfg(feature = "capture")] fn capture_args(flags: &[&str]) -> CaptureArgs { let mut argv = vec!["moq", "--connect", "http://localhost", "import", "capture"]; From 5a6c8b46d01ff493ffd613802a8b7b87196d5241 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 22:25:00 -0800 Subject: [PATCH 3/3] fix(cli): report the audio give-up without waiting on drains Co-Authored-By: Claude Opus 5.5 --- rs/moq-cli/src/publish.rs | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/rs/moq-cli/src/publish.rs b/rs/moq-cli/src/publish.rs index 7927030da1b..240cac02219 100644 --- a/rs/moq-cli/src/publish.rs +++ b/rs/moq-cli/src/publish.rs @@ -764,7 +764,7 @@ async fn publish_audio( (config, encode, retain): (moq_audio::capture::Config, moq_audio::encode::Options, bool), ) -> anyhow::Result> { // A failed track keeps serving its retained audio until it is served or expires, up to - // `--max-age`, without holding up the reopen. Dropping the set cancels them. + // `--max-age`, without holding up the reopen or the report. let mut draining = tokio::task::JoinSet::new(); let mut reopen = Reopen::default(); let mut published = false; @@ -804,13 +804,9 @@ async fn publish_audio( None => break Err(err.context(format!("audio capture kept failing for {:?}", Reopen::BUDGET))), } }; - while let Some(joined) = draining.join_next().await { - if let Err(err) = joined - && err.is_panic() - { - std::panic::resume_unwind(err.into_panic()); - } - } + // Report at once: the failed tracks keep draining in the background while the video runs, + // each ending with its error, and are dropped when the command exits. + draining.detach_all(); Ok(result) }