Skip to content

Catalog delay counts B-frame reorder: Order compares presentation timestamps written in decode order #7579

Closed

Description

@michaelstuurman

Order (rs/moq-mux/src/catalog/estimate.rs) measures a frame's delay against the highest timestamp another track has already written. Every caller passes the presentation timestamp, and frames are written in decode order. A track with B-frames writes each reference picture before the B-frames that present before it, so its highest timestamp runs ahead of the frames it is writing, by two frame durations (40 ms) at 50 fps with two B-frames. Any track written alongside it then reads as trailing by that much: another rendition's B-frames, and the audio of the same decode slot.

The comment on Order says "a B-frame trailing its own reference picture is already reorder jitter". Measured against another rendition's reference picture, the same B-frame counts as delay.

delay exists for a constant offset between tracks, which cancels out of the receiver's measured (doc/concept/audio-jitter.md, "Across renditions"). This reorder isn't constant: reference pictures don't trail, B-frames do. So for the B-frame tracks measured already contains it, and adding it as delay counts it twice. The same doc gives that reason for jitter being a floor, not an addend. For the audio the delay is simply spurious: nothing in it trails.

Unit test on main at 80e29de (estimate.rs is unchanged up to f00dce6). Two renditions of one 50 fps source with two B-frames, plus audio, written in lockstep in decode order, one frame of each per 20 ms:

  • No reorder declared, as from moq-c: all three tracks get delay 40 ms, and none gets jitter.
  • Reorder declared, as the MPEG-TS and fMP4 importers do (observe_reorder, Producer::reorder): both renditions get jitter 60 ms, and all three tracks still get delay 40 ms. So this is not specific to moq-c.
Test (in estimate.rs mod tests)
	/// Two renditions of one 50 fps source with two B-frames (I0 P60 B20 B40 P120 B80 B100 …) and
	/// audio, written in lockstep in decode order: one frame of each track per 20 ms.
	fn lockstep(declare_reorder: bool) -> [Estimate; 3] {
		let [mut hevc, mut avc, mut audio] = tracks();
		if declare_reorder {
			// What the MPEG-TS and fMP4 importers report for this GOP: its largest PTS - DTS, with
			// decoding starting one frame before the first PTS.
			hevc.reorder(micros(60_000));
			avc.reorder(micros(60_000));
		}
		let decode_order = [0, 60, 20, 40].into_iter().chain((1..10u64).flat_map(|gop| {
			let base = gop * 60;
			[base + 60, base + 20, base + 40]
		}));
		for (tick, pts) in decode_order.enumerate() {
			hevc.write(micros(pts * 1_000), 1);
			avc.write(micros(pts * 1_000), 1);
			audio.write(micros(tick as u64 * 20_000), 1);
		}
		[hevc.estimate(), avc.estimate(), audio.estimate()]
	}

	#[test]
	fn bframe_reorder_becomes_delay() {
		let [hevc, avc, audio] = lockstep(false);
		assert_eq!((hevc.jitter, hevc.delay), (None, Some(ms(40))));
		assert_eq!((avc.jitter, avc.delay), (None, Some(ms(40))));
		assert_eq!((audio.jitter, audio.delay), (None, Some(ms(40))));

		// With the reorder declared, the video gets it as jitter, and every track still gets it as delay.
		let [hevc, avc, audio] = lockstep(true);
		assert_eq!((hevc.jitter, hevc.delay), (Some(ms(60)), Some(ms(40))));
		assert_eq!((avc.jitter, avc.delay), (Some(ms(60)), Some(ms(40))));
		assert_eq!((audio.jitter, audio.delay), (None, Some(ms(40))));
	}

On a real stream, moq-c at 80e29de.

  • Setup: an encoder publishes an HEVC and an H.264 rendition of the same 50 fps source, plus Opus, with moq_publish_media_frame.
  • It paces all tracks on the capture cadence. A picture is written once a captured frame is a fixed delay past its DTS plus its rendition's largest PTS − DTS. An Opus packet is written once a captured frame is that delay past its PTS.
  • So reference pictures leave with the audio of the same PTS, and each B-frame leaves 60 ms after it: its forward reference presents two frames later and decodes one slot earlier.
  • We applied a local patch for moq-c: a JSON write without a timestamp is ordered on the broadcast clock, which C media can't read, so every media track advertises ~8 s delay #7573, so the broadcast's JSON tracks take no part in the order.
HEVC jitter / delay H.264 jitter / delay Opus jitter / delay @moq/watch playout (HEVC + Opus)
No B-frames – / – – / – – / – 30–50 ms
Two B-frames – / 50 ms – / 50 ms – / 10 ms 130 ms
Two B-frames, libmoq 0.6.17 64 / 2 ms 65 / 42 ms 16 / 12 ms 82 ms

"–" means the field is absent. The 0.6.17 row is a different run from the one in #7573, with B-frames on.

Without B-frames no track has a delay. With them both renditions advertise delay 50 ms and no jitter, although the catalog declares reorder 1 (HEVC) and 2 (H.264) from the SPS. The B-frames' lateness is real: 0.6.17, which measured wall-clock flush lateness, put it into jitter (64 and 65 ms). Main counts it as delay. We haven't pinned down why it is 50 ms rather than 60, or where Opus's 10 ms comes from. The H.264 delay of 42 ms on 0.6.17 was a real hold in our encoder at the time.

The player computes playout = max over subscribed renditions of (delay + target), with target = max(measured, jitter), plus one frame for audio. The measured arrival lateness already includes the B-frames, so the delay adds them a second time. @moq/watch buffered 130 ms, against 82 ms on 0.6.17, where the same lateness was only a jitter floor.

A C publisher can't declare the reorder:

  • moq_publish_media_frame(media, payload, payload_size, timestamp_us) takes one timestamp, used as the PTS, and no DTS. moq-c never calls reorder.
  • The catalog reorder count from the SPS doesn't reach the estimator.

Packagings differ here. MPEG-TS and fMP4 carry each frame's decode time, and their importers turn PTS − DTS into the reorder. LOCMAF (draft-einarsson-moq-locmaf-01, an individual draft) keeps the CMAF decode timeline: the base media decode time, sample durations and per-sample composition time offsets. A receiver can therefore reconstruct each frame's DTS, as in fMP4. LOC (draft-ietf-moq-loc-04) carries one Timestamp per object (§2.3.1.1) and puts objects in decode order (§4.2), as moq-c does. Either way, Order sees only the PTS.

Possible fix: measure delay at decode edges, a track's high less its declared reorder. Discount a frame's trail by at most its own track's declared reorder, and compare with each other track's high less that track's declared reorder. A track without a declared reorder is measured as today, so a genuinely late track, such as a stale JSON capture, keeps its delay.

We prototyped this:

  • On the test above with the reorder declared, no track gets a delay, and the renditions keep their 60 ms jitter as a floor.
  • Without a declared reorder, nothing changes.
  • All other moq-mux tests pass. The prototype scans every seat instead of keeping the top two, and resets a seat's reorder when the seat is freed.
Prototype diff on 80e29de
--- a/rs/moq-mux/src/catalog/estimate.rs
+++ b/rs/moq-mux/src/catalog/estimate.rs
@@ -183,6 +183,7 @@
 	/// time.
 	pub fn reorder(&mut self, delay: Timestamp) {
 		self.burst(Duration::from(delay));
+		self.seat.reorder(Duration::from(delay).as_nanos());
 	}
 
 	/// Record the media duration emitted together by a container importer.
@@ -445,6 +446,8 @@
 	first: Option<(usize, u128)>,
 	/// The largest high on any other seat.
 	second: Option<(usize, u128)>,
+	/// Each seat's declared reorder: how far its high runs ahead of its decode edge.
+	reorders: Vec<u128>,
 }
 
 impl Highs {
@@ -459,10 +462,17 @@
 
 	/// Raise `index`'s high to `timestamp` and return how far `timestamp` trails the others' highest.
 	fn write(&mut self, index: usize, timestamp: u128) -> Option<Duration> {
-		let others = match self.first {
-			Some((first, _)) if first == index => self.second,
-			first => first,
-		};
+		// Compare decode edges: this frame's timestamp plus its own track's reorder, against
+		// each other track's high less that track's reorder. A scan, for the prototype.
+		let reorder = |seat: usize| self.reorders.get(seat).copied().unwrap_or(0);
+		let others = self
+			.seats
+			.iter()
+			.enumerate()
+			.filter(|(seat, _)| *seat != index)
+			.filter_map(|(seat, high)| Some(high.flatten()?.saturating_sub(reorder(seat))))
+			.max();
+		let edge = timestamp + reorder(index);
 
 		let high = self.seats[index]
 			.as_mut()
@@ -484,13 +494,15 @@
 			_ => {}
 		}
 
-		let (_, others) = others?;
-		elapsed(timestamp, others)
+		elapsed(edge, others?)
 	}
 
 	/// Forget `index`'s high, freeing its seat when `leave` is set.
 	fn clear(&mut self, index: usize, leave: bool) {
 		self.seats[index] = if leave { None } else { Some(None) };
+		if leave && let Some(reorder) = self.reorders.get_mut(index) {
+			*reorder = 0;
+		}
 		if [self.first, self.second]
 			.iter()
 			.flatten()
@@ -538,6 +550,14 @@
 		self.lock().write(self.index, timestamp)
 	}
 
+	fn reorder(&self, reorder: u128) {
+		let mut highs = self.lock();
+		if highs.reorders.len() <= self.index {
+			highs.reorders.resize(self.index + 1, 0);
+		}
+		highs.reorders[self.index] = highs.reorders[self.index].max(reorder);
+	}
+
 	/// A break in the timeline may resume anywhere, so the old high must not hold others back.
 	fn discontinuity(&self) {
 		self.lock().clear(self.index, false);

Where the reorder comes from. The TS and fMP4 importers have each frame's PTS − DTS. A C publisher has none, but its catalog already carries the depth from the SPS, and moq-mux already turns a depth into decode times. DeclaredClock (rs/moq-mux/src/container/declared.rs) decodes a group's n-th frame at its (n − depth)-th presentation slot, for the TS, FLV and MKV exporters, and for fMP4 per quest/m2/fmp4-declared-clock.md. For this GOP, depth 1, as the HEVC SPS declares, gives the 60 ms the test declares. The estimator could use the same rule, so moq-c would need no new API.

Two shortcuts fall short:

  • Count × frame duration gives 20 ms for depth 1.
  • The trail observed behind a track's own high arrives after the audio has already measured its delay, which never falls. decode_order_pts_gap_never_becomes_provisional_jitter asserts that an observed decode-order gap alone never becomes jitter, so using it would reverse that choice.

Related: #7573 (JSON writes in the same order).

(Written by Claude Opus 5.5)

Activity

  1. added
    questBeing tracked/planned in a quest. See `quest/`
    on Nov 3, 2026
  2. kixelated commented on Nov 3, 2026

    @kixelated
    Collaborator

    Thanks for the test and the prototype. It's planned in #7665 as quest/m1/catalog-delay-min.md.

    Decision: a track's delay becomes the smallest amount its frames trailed another track, not the largest.

    • That's the constant offset the field exists for. Reference pictures, and the audio of the same decode slot, trail by zero, so the reorder drops out.
    • A genuinely late track, every frame of which trails, keeps its offset.
    • Variable lateness is left to jitter and the receiver's measurement.

    The decode-edge prototype, with the reorder taken from the SPS depth, wasn't taken: the maintainer would rather not derive it from SPS depth. Your lockstep test becomes the acceptance test, asserting no delay with or without a declared reorder.

    To pick it up, run /quest-start quest/m1/catalog-delay-min. Anyone who wants to donate tokens is welcome to.

    (Written by Claude Opus 5.5)

  3. added a commit that references this issue on Nov 3, 2026
  4. added a commit that references this issue on Nov 3, 2026
    699879a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions