diff --git a/quest/m1/README.md b/quest/m1/README.md index 8698a16b258..43b4d5310f6 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -59,7 +59,6 @@ moves what remains of them once their claims merge. - [Relay settings parse once into a validated configuration](/quest/m1/relay-config-checks.md) - settings that cannot work together are refused at load instead of accepted and ignored - [Track drop runs are forgotten below the floor](/quest/m1/drop-runs-prune.md) - a track's dropped-group record stays proportional to its window in Rust and JS - [A macOS camera delivers frames the renderer imports](/quest/m1/camera-nv12-macos.md) - AVFoundation capture asks for NV12, so a camera whose native format is `2vuy` reaches the renderer -- [Encoders keyframe by time, not frame count](/quest/m1/keyframe-by-time.md) - groups stay within the declared join interval when a source's frame rate drifts from the configured one ## Related diff --git a/quest/m1/keyframe-by-time.md b/quest/m1/keyframe-by-time.md deleted file mode 100644 index 0f919badb21..00000000000 --- a/quest/m1/keyframe-by-time.md +++ /dev/null @@ -1,31 +0,0 @@ -# [M] Encoders keyframe by time, not frame count - -## Goal - -A video encoder's groups stay within its declared join interval -(`encode::Config::join`) when the source runs slower or faster than the -configured `framerate`, such as a camera dropping to 20 fps in low light. Today -a 2 s join at 30 fps keyframes every 60 frames, which a 20 fps source -stretches to 3 s, so moq-mux warns that the track's groups are longer than the -declared interval and HLS lists them as overruns. - -## Plan - -Found by moq-dev/studio's first Mac capture run -([moq-dev/studio#70](https://github.com/moq-dev/studio/pull/70)). Decided -2026-11-07 (maintainer, moq-dev/studio's /quest-unblock): ✅ keyframe by -elapsed time (rejected: have AVFoundation report the device's frame rate, -which stays 30 when auto-exposure slows the camera; drop it); ✅ m1 (rejected: -m3), since it changes no public API and fixes every variable-rate source. - -`Config::keyframe_interval()` (`rs/moq-video/src/encode/encoder.rs`, -crate-private) converts `join` to frames at the configured rate, and each -backend sets that as its GOP length. Instead, force a keyframe on the first -frame whose timestamp reaches the last keyframe's plus `join`, through the -forced-keyframe path that `Cuts` (`rs/moq-video/src/encode/cuts.rs`) already -drives on every backend. VideoToolbox can also take -`kVTCompressionPropertyKey_MaxKeyFrameIntervalDuration`. Decide in the PR -whether a backend's own GOP stays as a ceiling beyond `join` or goes. - -Test with a mocked source whose frame rate varies: no group exceeds `join` by -more than one frame. diff --git a/rs/moq-video/README.md b/rs/moq-video/README.md index 043c5403aed..f80c425db84 100644 --- a/rs/moq-video/README.md +++ b/rs/moq-video/README.md @@ -115,14 +115,16 @@ later one goes in, and for the tail `finish()` drains. Bring your own pixels wit Group boundaries are automatic: `Config::join` is the time between join points (a keyframe every two seconds by default, rounded to whole frames, and a join of -zero is refused at open), so an application never has to think about them. +zero is refused at open), so an application never has to think about them. It +is measured by the frames' timestamps, so a source running slower than +`Config::framerate` still keys every join. `Encoder::cut()` opens a group at the next frame when something outside the encoder needs a decodable starting point there: a source group boundary, a scene change, a source switch. The request is held until a frame arrives, so it is safe to call before you have one. It fails with `Error::CutUnsupported` on a backend that cannot force a boundary (a V4L2 driver without the force-keyframe control), and queues nothing then: groups -keep falling where `Config::join` puts them. `encode::Sink` answers the same +keep falling every `Config::join`'s worth of frames at `Config::framerate`. `encode::Sink` answers the same way, awaited. Public entry points: diff --git a/rs/moq-video/src/encode/backend/mod.rs b/rs/moq-video/src/encode/backend/mod.rs index 22eaa8c3762..541d46033ca 100644 --- a/rs/moq-video/src/encode/backend/mod.rs +++ b/rs/moq-video/src/encode/backend/mod.rs @@ -51,10 +51,13 @@ mod vaapi; /// whether forced by `cut` or placed by the backend's own GOP cadence. pub(crate) trait Backend { /// Encode one frame, opening a group at it (an IDR) when `cut` is set. - /// Backends place group boundaries on their own per [`Config::join`], so this - /// is only the caller's extra request, arriving via - /// [`Encoder::cut`](super::Encoder::cut), and only on a backend whose - /// [`can_cut`](Self::can_cut) said yes. + /// Backends place group boundaries on their own every + /// [`Config::keyframe_interval`] frames, so this is only the encoder's extra + /// request: a caller's [`Encoder::cut`](super::Encoder::cut), or a + /// [`Config::join`] that a source slower than its frame rate reached first. It + /// arrives only on a backend whose [`can_cut`](Self::can_cut) said yes. A + /// backend that drops the frame carries the cut to the next one it encodes, + /// or the group waits a whole join for its keyframe. fn encode(&mut self, frame: &Frame, cut: bool) -> Result, Error>; /// Return every access unit the codec is still holding, leaving the encoder diff --git a/rs/moq-video/src/encode/backend/videotoolbox.rs b/rs/moq-video/src/encode/backend/videotoolbox.rs index 333bef6cc14..2642110e9a8 100644 --- a/rs/moq-video/src/encode/backend/videotoolbox.rs +++ b/rs/moq-video/src/encode/backend/videotoolbox.rs @@ -68,6 +68,9 @@ pub(crate) struct VideoToolbox { framerate_numerator: i32, framerate_denominator: i64, frame_index: i64, + /// A cut whose frame VideoToolbox dropped, carried to the next frame so the + /// group it asked for still opens. + keyframe_pending: bool, applied: Applied, } @@ -206,6 +209,7 @@ impl VideoToolbox { framerate_numerator, framerate_denominator: i64::from(config.framerate.denominator()), frame_index: 0, + keyframe_pending: false, applied, }) } @@ -234,6 +238,7 @@ impl Backend for VideoToolbox { }; self.frame_index += 1; + let cut = cut || self.keyframe_pending; let frame_properties = cut.then_some(&*self.force_keyframe); let status = unsafe { @@ -265,6 +270,8 @@ impl Backend for VideoToolbox { "VideoToolbox encode callback failed: {status}" ))); } + // A dropped frame comes back with no packet and no error. + self.keyframe_pending = cut && self.sink.packets.is_empty(); // `complete_frames` above forces this frame out before returning, so every // packet collected here came from it and carries its timestamp. Ok(std::mem::take(&mut self.sink.packets) diff --git a/rs/moq-video/src/encode/encoder.rs b/rs/moq-video/src/encode/encoder.rs index c092a502b46..1f21a891d0d 100644 --- a/rs/moq-video/src/encode/encoder.rs +++ b/rs/moq-video/src/encode/encoder.rs @@ -154,8 +154,10 @@ pub struct Config { /// in band) opens a group this often, so a subscriber joining mid-stream waits /// at most this long for a picture it can decode. [`Encoder::cut`] opens one /// sooner. Rounded to whole frames at `framerate`, never below one; see - /// [`effective_join`](Self::effective_join). [`Encoder::new`] refuses zero. - /// Defaults to two seconds. + /// [`effective_join`](Self::effective_join). Measured by the frames' + /// timestamps on a backend that can cut, so a source running slower than + /// `framerate` still opens a group this often, overshooting by less than one of + /// its frames. [`Encoder::new`] refuses zero. Defaults to two seconds. pub join: std::time::Duration, /// Output codec. Defaults to [`Codec::H264`]. pub codec: Codec, @@ -231,9 +233,10 @@ impl Config { std::time::Duration::from_nanos(nanos.try_into().unwrap_or(u64::MAX)) } - /// Frames between keyframes: [`join`](Self::join) rounded to the nearest - /// frame, and never below one, since a positive join shorter than a frame - /// still opens a group per frame. + /// Frames between keyframes at `framerate`: [`join`](Self::join) rounded to + /// the nearest frame, and never below one, since a positive join shorter than + /// a frame still opens a group per frame. Each backend keys on this count, and + /// the encoder keys sooner when the timestamps reach it first. pub(crate) fn keyframe_interval(&self) -> u32 { self.framerate.frames(self.join).max(1) } @@ -347,6 +350,8 @@ pub struct Encoder { /// rather than applied immediately because the caller decides a group /// boundary before it has the frame that opens it. pending_cut: bool, + /// Where [`Config::join`] cuts a source slower than its frame rate. + join: Join, /// How to scale a frame at another size, if at all; see [`Config::resize`]. resize: Option, #[cfg(target_os = "linux")] @@ -376,6 +381,7 @@ impl Encoder { bitrate: config.resolved_bitrate(), color: config.resolved_color(), pending_cut: false, + join: Join::new(config), resize: config.resize, #[cfg(target_os = "linux")] input: config.input, @@ -456,9 +462,9 @@ impl Encoder { /// /// Returns [`Error::CutUnsupported`] when this backend cannot force a group /// boundary (a V4L2 driver without the force-keyframe control), in which case - /// nothing is queued: groups keep falling where [`Config::join`] puts them, and - /// the caller decides whether that layout is acceptable rather than finding - /// out from the stream. + /// nothing is queued: groups keep falling every [`Config::join`]'s worth of + /// frames at [`Config::framerate`], and the caller decides whether that layout + /// is acceptable rather than finding out from the stream. pub fn cut(&mut self) -> Result<(), Error> { self.check_cut()?; self.pending_cut = true; @@ -536,11 +542,15 @@ impl Encoder { encoder: self.color, }); } - let encoded = self.backend.encode(frame, self.pending_cut)?; + let timestamp = std::time::Duration::from(frame.timestamp); + let cut = self.pending_cut || (self.backend.can_cut() && self.join.due(timestamp)); + let encoded = self.backend.encode(frame, cut)?; // Cleared only once the frame is through: a failed encode produced no // picture, so the request still belongs to whatever comes next rather than // being swallowed. The size check above returns early for the same reason. self.pending_cut = false; + self.join.encoded(timestamp, cut); + self.join.observe(&encoded); Ok(encoded) } @@ -559,7 +569,9 @@ impl Encoder { /// encode and the next frame's submission, so flush at boundaries rather than /// per frame. pub fn flush(&mut self) -> Result, Error> { - self.backend.flush() + let encoded = self.backend.flush()?; + self.join.observe(&encoded); + Ok(encoded) } /// Flush the encoder, returning any buffered frames. Each keeps the timestamp @@ -573,6 +585,57 @@ impl Encoder { } } +/// The keyframes [`Config::join`] adds for a source slower than [`Config::framerate`]. +/// +/// Each backend keys every [`Config::keyframe_interval`] frames, which a slower +/// source (a camera dropping to 20 fps in low light) stretches past the join the +/// catalog declared. This keys the first frame whose distance from the last +/// keyframe, counted in frames at the configured rate and rounded to the nearest, +/// reaches that interval: a source at the configured rate keys on the same frame +/// as the backend whatever its jitter, and a slower one overshoots the join by +/// less than one of its frames. A faster source reaches the backend's count first. +struct Join { + framerate: FrameRate, + interval: u32, + /// The last keyframe's timestamp, `None` before the first frame, which every + /// backend keys. + last: Option, +} + +impl Join { + fn new(config: &Config) -> Self { + Self { + framerate: config.framerate, + interval: config.keyframe_interval(), + last: None, + } + } + + /// Whether the frame at `timestamp` is due a keyframe. + fn due(&self, timestamp: std::time::Duration) -> bool { + self.last + .and_then(|last| timestamp.checked_sub(last)) + .is_some_and(|elapsed| self.framerate.frames(elapsed) >= self.interval) + } + + /// The frame at `timestamp` went in, keyed when `cut`. + fn encoded(&mut self, timestamp: std::time::Duration, cut: bool) { + // A timestamp behind the last keyframe starts a new timeline, measured from + // here rather than held until it catches up with the old one. + if cut || self.last.is_none_or(|last| timestamp < last) { + self.last = Some(timestamp); + } + } + + /// Count the keyframes among `encoded`, including those the backend placed on + /// its own count, so the next is measured from the latest. + fn observe(&mut self, encoded: &[Encoded]) { + for keyframe in encoded.iter().filter(|encoded| encoded.keyframe) { + self.last = self.last.max(Some(keyframe.timestamp.into())); + } + } +} + #[cfg(test)] mod tests { #![cfg_attr( @@ -1168,6 +1231,7 @@ mod tests { bitrate: config.resolved_bitrate(), color: config.resolved_color(), pending_cut: false, + join: Join::new(config), resize: config.resize, #[cfg(target_os = "linux")] input: None, @@ -1293,6 +1357,208 @@ mod tests { assert_eq!(*log.lock().unwrap(), vec![false]); } + /// A 320x240 gray frame at `micros`, for a source whose rate varies. + fn gray_at(micros: u64) -> Frame { + let surface = Surface::rgba(&gray_rgba(320, 240), Size::new(320, 240)).unwrap(); + Frame::new(surface, moq_net::Timestamp::from_micros(micros).unwrap()) + } + + /// A source slower than the configured rate (a camera dropping frames in low + /// light) still opens a group every join, by its timestamps: no group holds a + /// frame a join past its keyframe, so it overshoots by less than one of its + /// frames, and none closes more than half a configured frame early. A backend + /// that cannot cut is never asked to. + #[test] + fn a_slow_source_keys_by_time() { + let config = Config { + join: std::time::Duration::from_secs(1), + ..Config::new(320, 240, crate::FrameRate::new(30, 1).unwrap()) + }; + // 30 fps, then 20, then 12.5, then 24, the rates a camera drifts through as + // the light changes. + let mut times = Vec::new(); + let mut micros = 0u64; + for (interval, count) in [(33_333, 45), (50_000, 50), (80_000, 30), (41_667, 60)] { + for _ in 0..count { + times.push(micros); + micros += interval; + } + } + + for cuts in [true, false] { + let log = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let backend = Recorder { log: log.clone(), cuts }; + let mut encoder = encoder_with(Box::new(backend), &config); + for µs in × { + encoder.encode(&gray_at(micros)).unwrap(); + } + let log = log.lock().unwrap(); + if !cuts { + assert!(log.iter().all(|cut| !cut), "a backend that cannot cut was asked to"); + continue; + } + + let mut key = times[0]; + for (µs, &cut) in times.iter().zip(log.iter()).skip(1) { + if cut { + assert!( + micros - key + 16_667 >= 1_000_000, + "the group at {key}us closed at {micros}us" + ); + key = micros; + } else { + assert!( + micros - key < 1_000_000, + "the group at {key}us holds a frame at {micros}us" + ); + } + } + } + } + + /// A source at the configured rate keys on the same frames as the backend's + /// own count, however its timestamps jitter, so measuring by time adds none. + #[test] + fn a_jittery_source_at_the_configured_rate_keys_on_the_count() { + let config = Config { + join: std::time::Duration::from_secs(1), + ..Config::new(320, 240, crate::FrameRate::new(30, 1).unwrap()) + }; + let log = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let backend = Recorder { + log: log.clone(), + cuts: true, + }; + let mut encoder = encoder_with(Box::new(backend), &config); + + // Up to a quarter frame either way, so neither end of a group can pull it + // past half a frame. + let jitter = [0, 8_000, -8_000, 5_000, -5_000]; + for i in 0..120u64 { + let micros = (i * 33_333) + .checked_add_signed(jitter[i as usize % jitter.len()]) + .unwrap(); + encoder.encode(&gray_at(micros)).unwrap(); + } + + let log = log.lock().unwrap(); + let cuts: Vec = (0..log.len()).filter(|&i| log[i]).collect(); + assert_eq!(cuts, [30, 60, 90]); + } + + /// Keys on its own every `interval` frames, restarting the count at a cut the + /// way a real codec's GOP does, and records the cut flag each frame came with. + struct Counting { + log: std::sync::Arc>>, + interval: u32, + count: u32, + } + + impl Backend for Counting { + fn encode(&mut self, frame: &Frame, cut: bool) -> Result, Error> { + self.log.lock().unwrap().push(cut); + let keyframe = cut || self.count.is_multiple_of(self.interval); + self.count = if keyframe { 1 } else { self.count + 1 }; + Ok(vec![Encoded::new(bytes::Bytes::new(), frame.timestamp, keyframe)]) + } + + fn flush(&mut self) -> Result, Error> { + Ok(Vec::new()) + } + + fn finish(&mut self) -> Result, Error> { + Ok(Vec::new()) + } + + fn set_bitrate(&mut self, _bitrate: u64) -> Result<(), Error> { + Ok(()) + } + + fn can_cut(&self) -> bool { + true + } + + fn name(&self) -> &'static str { + "counting" + } + } + + /// A source faster than the configured rate reaches the backend's count before + /// the join, and each keyframe the backend places restarts the clock, so the + /// encoder forces none between them. + #[test] + fn a_fast_source_keys_on_the_backend_count() { + let config = Config { + join: std::time::Duration::from_secs(1), + ..Config::new(320, 240, crate::FrameRate::new(30, 1).unwrap()) + }; + let log = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let backend = Counting { + log: log.clone(), + interval: config.keyframe_interval(), + count: 0, + }; + let mut encoder = encoder_with(Box::new(backend), &config); + + // Three seconds at 60 fps: the backend keys every half second. + for i in 0..180 { + encoder.encode(&gray_at(i * 16_667)).unwrap(); + } + assert!(log.lock().unwrap().iter().all(|cut| !cut)); + } + + /// A timestamp behind the last keyframe starts a new timeline: the next + /// keyframe comes a join after the jump, not once the source catches up with + /// where it was. + #[test] + fn a_jump_back_restarts_the_join() { + let config = Config { + join: std::time::Duration::from_secs(1), + ..Config::new(320, 240, crate::FrameRate::new(30, 1).unwrap()) + }; + let log = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let backend = Recorder { + log: log.clone(), + cuts: true, + }; + let mut encoder = encoder_with(Box::new(backend), &config); + + for i in 0..20 { + encoder.encode(&gray_at(10_000_000 + i * 33_333)).unwrap(); + } + for i in 0..40 { + encoder.encode(&gray_at(i * 33_333)).unwrap(); + } + + let log = log.lock().unwrap(); + let cuts: Vec = (0..log.len()).filter(|&i| log[i]).collect(); + assert_eq!(cuts, [50], "a join after the jump at frame 20"); + } + + /// openh264 keys where the join asks and restarts its own count there, so a + /// 20 fps source on a 30 fps config keys every 20 frames, with no stray IDR + /// on the 30-frame count in between. + #[test] + #[cfg(feature = "openh264")] + fn openh264_keys_a_slow_source_by_time() { + let config = Config { + kind: Kind::Software, + join: std::time::Duration::from_secs(1), + ..Config::new(320, 240, crate::FrameRate::new(30, 1).unwrap()) + }; + let mut encoder = Encoder::new(&config).unwrap(); + + let mut keys = Vec::new(); + for i in 0..70 { + for encoded in encoder.encode(&gray_at(i * 50_000)).unwrap() { + if encoded.keyframe { + keys.push(encoded.timestamp.as_micros() / 50_000); + } + } + } + assert_eq!(keys, [0, 20, 40, 60]); + } + /// A join of zero opens no group, so it is refused at open rather than /// handed to a backend, each of which would do something different with it /// (openh264 keys every frame, V4L2 rejects the control, VideoToolbox reads