Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions doc/bin/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,8 +246,11 @@ moq ... import capture --audio on # encode audio continuo

Video goes through the platform hardware encoder, with an H.264 software
fallback when built with `openh264`, and audio is Opus; [moq-video](/lib/rs/moq-video) lists the
backends. An audio device that fails or stops ends the import with its source
error; restart the command after the device is available again. Audio appears
backends. An audio device that fails mid-session, such as a USB microphone
re-enumerating, is reopened every 0.5 to 4 seconds as a new audio track. One
still failing after 10 seconds ends only the audio track: video keeps
publishing, and the import exits with the audio error once it stops. A device
that fails to open at startup ends the import at once. Audio appears
in the catalog once its device opens; a microphone permission prompt leaves
other media available. The camera and
microphone stay open while publishing, so a late
Expand Down
30 changes: 0 additions & 30 deletions quest/m1/precut-fixes/capture-audio-reopen.md

This file was deleted.

30 changes: 30 additions & 0 deletions quest/m2/export-replaced-rendition.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# [M] Container exports outlive a rendition that fails and is replaced

## Goal

`moq export ts`, `fmp4`, `mkv`, and `flv` keep writing when one rendition's
track ends with an error and the catalog lists a replacement, as `moq import
capture` does when it reopens a failed microphone as a new audio track. The
export follows the replacement, or keeps the other media and says so, instead
of ending with the failed track's error.

## Plan

Found while validating the CLI's microphone reopen: with
`moq import capture -- export ts`, the reopen kept the video publishing, yet
the export ended with `audio capture: The requested audio device is not
available` once the old audio track failed
(`rs/moq-mux/src/container/consumer.rs`, "track failed; ending the reader").
An `export h264` of the same broadcast kept writing, since it reads only video.

Before that quest, the failed microphone ended the whole publish, so no
export outlived it; the reopen is what exposes this.

Recommended: a rendition whose track fails while the catalog still lists the
broadcast ends only that rendition's reader, and the export picks up a new
rendition of the same medium the way it handles a catalog change; the export
fails only when no rendition of a selected medium remains. Check what the
catalog-identity republish ([moq-dev/moq#7110](https://github.com/moq-dev/moq/pull/7110))
already does for a cleanly finished track and reuse it.

Public API: none expected. Wire: none.
202 changes: 183 additions & 19 deletions rs/moq-cli/src/publish.rs
Original file line number Diff line number Diff line change
Expand Up @@ -679,6 +679,7 @@ impl Publish {
} => {
drop(gate);
let mut report = self.report;
let has_video = video.is_some();
// Each enabled medium publishes its own track onto the shared
// broadcast + catalog. Both stamp frames on the catalog's
// advertised clock, so HLS/DASH wall times match the mapping on
Expand All @@ -702,29 +703,31 @@ impl Publish {
};
let audio_fut = async move {
match audio {
Some((config, encode, retain)) => {
let capture = moq_audio::capture::Producer::open(config).await?;
let mut options = moq_audio::encode::Capture::new(capture.consume());
options.encode = encode;
options.retain = retain;
let (_publication, driver) =
moq_audio::encode::publish_capture(broadcast, catalog, options)?;
let mut publish = std::pin::pin!(driver.run());
tokio::select! {
result = capture.run() => {
// Let the encoder serve its retained audio, then end its
// track with the source error, before its future is dropped.
let published = publish.await;
result.and(published).map_err(anyhow::Error::from)
}
result = &mut publish => result.map_err(anyhow::Error::from),
Some(audio) => publish_audio(broadcast, catalog, audio).await,
None => Ok(Ok(())),
}
};

// A video failure ends the publish. An audio failure that outlasted its reopens
// ends only the audio track: the video keeps publishing, and the command reports
// the audio failure once the video ends.
let capture = async {
let mut video_fut = std::pin::pin!(video_fut);
let mut audio_fut = std::pin::pin!(audio_fut);
tokio::select! {
result = &mut video_fut => {
result?;
audio_fut.await?
}
result = &mut audio_fut => {
let ended = result?;
if let Err(err) = &ended && has_video {
tracing::error!(error = %format_args!("{err:#}"), "audio capture ended; video keeps publishing");
}
video_fut.await.and(ended)
}
None => Ok(()),
}
};

let capture = async { tokio::try_join!(video_fut, audio_fut).map(|_| ()) };
// Written beside the encoders, so a capture waiting on its first viewer still reports.
let result = match report.as_mut() {
Some(report) => tokio::select! {
Expand All @@ -744,6 +747,115 @@ impl Publish {
}
}

/// Publish the captured audio, reopening the device and its track when the capture fails
/// mid-session, as a USB microphone re-enumerating or a sample-rate change does.
///
/// moq-audio never reopens a failed device, so this loop does. Each reopen publishes a new track,
/// since the failed one ends with its error. Returns once the track closes, or with the last
/// failure once failures outlast [`Reopen::BUDGET`].
///
/// A first open or publication that fails is a setup problem, such as a wrong device id or a
/// denied permission, so it fails the whole publish at once: the outer error. The inner one ends
/// only the audio.
#[cfg(feature = "capture")]
async fn publish_audio(
broadcast: moq_net::broadcast::Producer,
catalog: moq_mux::catalog::Producer,
(config, encode, retain): (moq_audio::capture::Config, moq_audio::encode::Options, bool),
) -> anyhow::Result<anyhow::Result<()>> {
// A failed track keeps serving its retained audio until it is served or expires, up to
// `--max-age`, without holding up the reopen or the report.
let mut draining = tokio::task::JoinSet::new();
let mut reopen = Reopen::default();
let mut published = false;
let result = loop {
let mut live = None;
let attempt = async {
let capture = moq_audio::capture::Producer::open(config.clone()).await?;
live = Some(kio::clock::Instant::now());
let mut options = moq_audio::encode::Capture::new(capture.consume());
options.encode = encode.clone();
options.retain = retain;
let (_publication, driver) =
moq_audio::encode::publish_capture(broadcast.clone(), catalog.clone(), options)?;
published = true;
let mut publish = Box::pin(driver.run());
tokio::select! {
result = capture.run() => {
draining.spawn(publish);
result
}
result = &mut publish => result,
}
};
let err = match attempt.await {
Ok(()) => break Ok(()),
Err(err) => anyhow::Error::from(err),
};
if !published {
return Err(err);
}
while draining.try_join_next().is_some() {}
match reopen.failed(live, kio::clock::Instant::now()) {
Some(wait) => {
tracing::warn!(error = %format_args!("{err:#}"), ?wait, "audio capture failed; reopening");
kio::clock::sleep(wait).await;
}
None => break Err(err.context(format!("audio capture kept failing for {:?}", Reopen::BUDGET))),
}
};
// Report at once: the failed tracks keep draining in the background while the video runs,
// each ending with its error, and are dropped when the command exits.
draining.detach_all();
Ok(result)
}

/// When to reopen a failed audio capture, and when to give up on it.
#[cfg(feature = "capture")]
struct Reopen {
/// The wait before the next reopen.
delay: std::time::Duration,
/// When the current run of failures began, with no healthy capture since.
failing: Option<kio::clock::Instant>,
}

#[cfg(feature = "capture")]
impl Default for Reopen {
fn default() -> Self {
Self {
delay: Self::MIN,
failing: None,
}
}
}

#[cfg(feature = "capture")]
impl Reopen {
/// The first wait. It covers a USB device re-enumerating.
const MIN: std::time::Duration = std::time::Duration::from_millis(500);
/// The longest wait, so a returning device is picked up within seconds.
const MAX: std::time::Duration = std::time::Duration::from_secs(4);
/// A capture that ran this long recovered, so its failure starts a fresh run of retries.
const HEALTHY: std::time::Duration = std::time::Duration::from_secs(1);
/// How long failures may persist before the audio track ends.
const BUDGET: std::time::Duration = std::time::Duration::from_secs(10);

/// Record a failure at `now` of a capture live since `live` (`None` when it never opened),
/// returning the wait before the next reopen, or `None` to give up.
fn failed(&mut self, live: Option<kio::clock::Instant>, now: kio::clock::Instant) -> Option<std::time::Duration> {
if live.is_some_and(|live| now.saturating_duration_since(live) >= Self::HEALTHY) {
*self = Self::default();
}
let failing = *self.failing.get_or_insert(now);
if now.saturating_duration_since(failing) >= Self::BUDGET {
return None;
}
let wait = self.delay;
self.delay = (self.delay * 2).min(Self::MAX);
Some(wait)
}
}

/// What a stdin import does around its media: announce, report, and retain the end.
#[derive(Default)]
struct Publication {
Expand Down Expand Up @@ -1220,6 +1332,58 @@ pub(crate) mod tests {

use super::*;

/// Each wait in a run of reopens that never open, from the first failure until giving up.
#[cfg(feature = "capture")]
fn reopen_waits(reopen: &mut Reopen, mut now: kio::clock::Instant) -> Vec<Duration> {
let mut waits = Vec::new();
while let Some(wait) = reopen.failed(None, now) {
waits.push(wait);
now = now.saturating_add(wait);
}
waits
}

#[cfg(feature = "capture")]
#[test]
fn a_failing_audio_capture_backs_off_then_gives_up() {
let ms = Duration::from_millis;
let waits = reopen_waits(&mut Reopen::default(), kio::clock::Instant::ZERO);
assert_eq!(waits, [ms(500), ms(1000), ms(2000), ms(4000), ms(4000)]);
assert!(waits.iter().sum::<Duration>() >= Reopen::BUDGET);
}

#[cfg(feature = "capture")]
#[test]
fn a_healthy_audio_capture_restarts_its_reopens() {
let start = kio::clock::Instant::ZERO;
let mut reopen = Reopen::default();
reopen_waits(&mut reopen, start);

// A capture that ran past healthy recovered, however long the failures before it.
let live = start.saturating_add(Duration::from_secs(60));
let failed = live.saturating_add(Reopen::HEALTHY);
assert_eq!(reopen.failed(Some(live), failed), Some(Reopen::MIN));
assert_eq!(
reopen.failed(None, failed.saturating_add(Reopen::MIN)),
Some(Reopen::MIN * 2)
);
}

#[cfg(feature = "capture")]
#[test]
fn a_flapping_audio_capture_still_gives_up() {
// Opens that fail sooner than healthy are no recovery, so the budget still runs out.
let mut reopen = Reopen::default();
let mut now = kio::clock::Instant::ZERO;
let mut attempts = 0;
while let Some(wait) = reopen.failed(Some(now), now.saturating_add(Reopen::HEALTHY / 2)) {
now = now.saturating_add(Reopen::HEALTHY / 2 + wait);
attempts += 1;
assert!(attempts < 100, "a flapping capture never gave up");
}
assert_eq!(attempts, 5);
}

#[cfg(feature = "capture")]
fn capture_args(flags: &[&str]) -> CaptureArgs {
let mut argv = vec!["moq", "--connect", "http://localhost", "import", "capture"];
Expand Down