From f75da6b65129c49f7c65824b7ca7f0a1d2016545 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 11:31:24 -0800 Subject: [PATCH 1/9] chore(quest): claim quest/m0/interop-green/replacement-audio.md Quest-Claim: c4d90fed25da2519 From 6c7db1cea7c243ca4c05dc094b15d4b9b1c86acf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 12:45:58 -0800 Subject: [PATCH 2/9] quest: root-cause the Firefox replacement fallback miss as replayed on-demand audio The incumbent's on-demand audio encoder replays its retained backlog to a returning viewer, cut only by the subscription's max delay (or the whole retention for a moq-transport viewer). Firefox's in-order WebSocket path delivers that replay ahead of the live edge and the player stalls on it. Records the evidence and the fix options for the maintainer. Co-Authored-By: Claude Opus 5.5 --- quest/m0/interop-green/replacement-audio.md | 57 +++++++++++++++++++-- 1 file changed, 52 insertions(+), 5 deletions(-) diff --git a/quest/m0/interop-green/replacement-audio.md b/quest/m0/interop-green/replacement-audio.md index 71e2a81ecbb..cab9ec8dd6d 100644 --- a/quest/m0/interop-green/replacement-audio.md +++ b/quest/m0/interop-green/replacement-audio.md @@ -1,4 +1,4 @@ -# [M] Restore the replacement media case on Firefox +# [L] Restore the replacement media case on Firefox ## Goal @@ -20,9 +20,56 @@ Decided 2026-11-16 (maintainer, on [moq-dev/moq#8980](https://github.com/moq-dev - ✅ Leave Firefox to this quest (rejected: fix it in that PR). Its fallback misses the bound as often before the gap fixes as after, so it is a separate cause. -On Firefox, the fallback (`live incumbent fallback plays`) misses its 1039 ms bound about one run in -five on an unloaded host, as often before those fixes as after: one failing run had neither a -picture nor started audio, another a picture with the audio stalled. Root-cause it on Firefox first, -reading the player's state across the fallback. WebKit was never run; check it once Firefox passes. +### Root cause of the Firefox fallback miss + +On Firefox the fallback misses its 1039 ms bound in about one run in twenty on this host (3 of about +60), and its fallback time swings from 207 ms to over 1000 ms on the passing runs. The picture comes +back on time; the audio does not. + +The incumbent's audio is replayed history. `@moq/publish` encodes audio on demand +(`js/publish/src/audio/encoder.ts`): once the viewer moves to the replacement, the relay cancels its +idle upstream subscription, encoding stops, and capture keeps filling the retained `Backlog`. When +the viewer falls back, demand returns and the encoder replays the retained PCM as fresh groups, cut +only by the subscription's max delay: + +- moq-lite-07 and -06 viewers ask audio with a 2.13 s max delay, so about 2.1 s of audio arrives + behind the live picture. One failing run played the incumbent's video at 17.26 s while its audio + sat stalled at 15.09 s with the replayed groups already received. +- A moq-transport-22 viewer carries no max delay, so the relay asks upstream with none, and the whole + retention is replayed. The publisher answers before encoding the replay, so its Largest (and the + viewer's join) is where the earlier subscription stopped, about 8 s back. Two failing runs joined + there, dropped hundreds of groups as slow, and froze the picture with the audio stuck seconds + behind it. + +Chromium hides this because WebTransport delivers newest first, so the live edge lands ahead of the +replay. Firefox plays over the WebSocket fallback, which delivers in order, so the player gets the +replay first and stalls on it instead of skipping onto the shared clock. + +Ruled out: the relay. Re-opening its idle copy from the newest group it cached asks for less backlog +than a fresh copy's oldest-group ask, and the replay arrives as new sequence numbers either way: +asking from the live edge instead failed the same way. + +Fix options, pending the maintainer under Waiting: + +1. js/watch drops audio that lands behind the shared clock before it decodes, so any backlog within + the max delay (this replay, or a relay cache on a cold join) costs the player nothing and it + rejoins the clock at once. Recommended: the max delay is the player's own ask, so it must handle + what that ask returns from every publisher and relay; Firefox's in-order transport shows it does + not today. +2. `@moq/publish` replays retained audio to a returning subscriber only up to its min delay (the + jitter target) rather than its max. Narrower: it fixes this publisher only, while a relay cache + still serves the same backlog. +3. The moq-transport path carries the viewer's budget upstream (draft-22's delivery timeout), so an + IETF viewer stops asking for the whole retention. Worth it only alongside 1, since moq-transport + is the low-effort compatibility layer. + +WebKit was never run; check it once Firefox passes. Public API: none. Wire: none. + +## Waiting + +- Maintainer: pick the fix for the replayed audio a returning viewer gets (Plan options 1 to 3). + Recommended: option 1 in js/watch, since the player's own max delay invites this backlog from any + source; option 3 only if moq-transport-22 must pass on its own. Done once this item is replaced by + the decision in the Plan. From 037e0a72afe819316adb1c3f8dc104b3e1e3e441 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 14:39:06 -0800 Subject: [PATCH 3/9] fix(watch,publish): a returning viewer rejoins the live edge instead of the replayed audio backlog The incumbent replays its retained audio from the subscription's min delay (playout delay plus warmup), falling back to its max delay where none is stated. The player decodes audio that trails the clock past its bound only to warm the codec, and never writes it to the ring. Co-Authored-By: Claude Opus 5.5 --- doc/concept/audio-jitter.md | 5 +- doc/lib/js/publish.md | 4 +- doc/lib/rs/moq-audio.md | 3 +- js/publish/src/audio/encoder.test.ts | 100 +++++++++++++++------------ js/publish/src/audio/encoder.ts | 21 ++++-- js/watch/src/audio/buffer.test.ts | 23 ++++++ js/watch/src/audio/buffer.ts | 25 +++++++ js/watch/src/audio/decoder.test.ts | 35 +++++++++- js/watch/src/audio/decoder.ts | 7 ++ rs/moq-audio/src/encode/capture.rs | 9 +-- rs/moq-audio/src/encode/producer.rs | 60 +++++++++++----- test/interop/README.md | 15 ++-- test/interop/clients/js/media.ts | 16 +++-- 13 files changed, 235 insertions(+), 88 deletions(-) diff --git a/doc/concept/audio-jitter.md b/doc/concept/audio-jitter.md index 1e74a898e4e..75831e7537b 100644 --- a/doc/concept/audio-jitter.md +++ b/doc/concept/audio-jitter.md @@ -843,7 +843,10 @@ content; past the threshold it snaps onto the clock, as video does. When the audio received ahead of the playhead is about to run out, playback slows down to bridge the gap, never trailing the clock past the threshold. A congestion stall within the threshold costs a slowdown, a gap, and a catch-up; one past it -costs a skip. +costs a skip. Audio that arrives trailing the clock past the threshold, such as a +publisher's replayed history ahead of the live edge, is decoded only to warm the +codec and never reaches the playout buffer, so playback starts at the live edge +instead of anchoring on the backlog. The splice is waveform-similarity overlap-add (WSOLA), after Verhelst and Roelands, "An overlap-add technique based on waveform similarity (WSOLA) for diff --git a/doc/lib/js/publish.md b/doc/lib/js/publish.md index bbb2a3c93fd..fd05808003f 100644 --- a/doc/lib/js/publish.md +++ b/doc/lib/js/publish.md @@ -76,7 +76,9 @@ Video encodes while enabled, so its recent groups wait in the track's cache for a late subscription's `delay.max` or a fetch. The cache keeps an idle group for 30 seconds whatever `maxAge` says; `maxAge` only bounds how far back a reader may reach into it. Audio keeps raw PCM while nobody listens, then encodes the part of -that history within the subscribers' max delay ahead of live audio when one arrives. Raw audio keeps up to +that history within the subscribers' min delay (their playout delay plus the codec's +warmup) ahead of live audio when one arrives. A subscription that states no min delay, +as over moq-transport, gets the history within its max delay instead. Raw audio keeps up to `maxAge`, which for media defaults to 30 seconds (about 11.5 MB for 48 kHz stereo f32). With `maxAge: null`, which sets no publisher limit on the media, raw audio still keeps 30 seconds, since raw history needs a bound. The catalog and diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 162d6bc7585..08dc2c9c74d 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -39,7 +39,8 @@ glides onto its clock, instead of writing silence or failing at each step. Without a subscriber the encoder keeps raw PCM up to the track's `max_age`, then encodes the part of that history within the -subscribers' max delay before continuing live when someone subscribes. With an unlimited `max_age` it keeps at most 30 seconds +subscribers' min delay (their playout delay plus the codec's warmup), or their max delay +when they state no min, before continuing live when someone subscribes. With an unlimited `max_age` it keeps at most 30 seconds (about 11.5 MB for 48 kHz stereo f32). Set `encode::Capture::retain` to false to encode continuously instead. Dropping an encoder leaves the shared capture running. A device failure keeps the retained history on the track until it is subscribed or expires, then the diff --git a/js/publish/src/audio/encoder.test.ts b/js/publish/src/audio/encoder.test.ts index 0f2b8a495b2..ea0a0474a7f 100644 --- a/js/publish/src/audio/encoder.test.ts +++ b/js/publish/src/audio/encoder.test.ts @@ -1331,7 +1331,7 @@ test.each([undefined, "off", "all"] as const)( await push(4); expect(encoder.out.stats.peek().frames).toBe(onDemand === "off" ? 4 : 0); - late = net.track("audio").subscribe({ delay: { max: Duration.millis(10_000) } }); + late = net.track("audio").subscribe({ delay: { min: Duration.millis(10_000), max: Duration.millis(10_000) } }); await settle(); expect(encoder.out.active.peek()).toBe(true); await push(3); @@ -1410,7 +1410,7 @@ test("a backlog encode refusal aborts its rendition when demand arrives", async const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); // A live-edge budget would leave the retained frame unencoded. - subscriber = net.track("audio").subscribe({ delay: { max: null } }); + subscriber = net.track("audio").subscribe({ delay: { min: null, max: null } }); await settle(); expect(encoder.out.error.peek()).toBe(failure); expect(subscriber.closed.peek()).toBe(failure); @@ -1448,7 +1448,7 @@ test("default audio replays the last second after three seconds without a subscr expect(encoder.out.active.peek()).toBe(false); const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); await settle(); expect(encoder.out.stats.peek().frames).toBe(50); for (let index = 150; index < 153; index++) { @@ -1474,43 +1474,51 @@ test("default audio replays the last second after three seconds without a subscr }); // Replayed groups reach the track as fresh arrivals, so only the encoder can tell they are old: a -// live viewer rejoining after seconds idle would otherwise wade through all of them first. -test("demand replays only the retained audio its max delay still wants", async () => { - using _webcodecs = installFakeWebCodecs(); - const clock = new Manual(); - const installed = clock.install(); - const broadcast = new Broadcast({ origin: new Moq.Origin.Producer(), name: Moq.Path.from("rejoin.hang") }); - const feed = new Feed(); - const configuring = configured(); - const encoder = new Encoder("audio", { broadcast, capture: fakeCapture(feed) as never }); - let late: Moq.Track.Subscriber | undefined; - try { - await configuring; - await settle(); - for (let index = 0; index < 150; index++) { - await feed.push({ timestamp: Duration.micros(index * 20_000), channels: [new Float32Array(960)] }); +// live viewer rejoining after seconds idle would otherwise wade through all of them first. It states +// how far back it plays as its min delay, which bounds the replay however far its max delay reaches; +// one that states none (moq-transport, moq-lite before 07) is bounded by its max delay alone. +for (const [name, delay] of [ + ["its min delay", { min: Duration.millis(130), max: Duration.millis(10_000) }], + ["its max delay when it states no min", { min: null, max: Duration.millis(130) }], + ["its max delay when that is shorter than its min", { min: Duration.millis(10_000), max: Duration.millis(130) }], +] as const) { + test(`demand replays only the retained audio ${name} still wants`, async () => { + using _webcodecs = installFakeWebCodecs(); + const clock = new Manual(); + const installed = clock.install(); + const broadcast = new Broadcast({ origin: new Moq.Origin.Producer(), name: Moq.Path.from("rejoin.hang") }); + const feed = new Feed(); + const configuring = configured(); + const encoder = new Encoder("audio", { broadcast, capture: fakeCapture(feed) as never }); + let late: Moq.Track.Subscriber | undefined; + try { + await configuring; + await settle(); + for (let index = 0; index < 150; index++) { + await feed.push({ timestamp: Duration.micros(index * 20_000), channels: [new Float32Array(960)] }); + await feed.drain(); + await clock.advance(Duration.millis(20)); + } + const net = broadcast.net.peek(); + if (!net) throw new Error("expected network broadcast"); + // At 3000 ms, a 130 ms budget still wants the frames ending after 2870 ms: 143 onward. + late = net.track("audio").subscribe({ delay }); + await settle(); + expect(encoder.out.stats.peek().frames).toBe(7 - FakeAudioEncoder.LAG); + await feed.push({ timestamp: Duration.micros(150 * 20_000), channels: [new Float32Array(960)] }); await feed.drain(); - await clock.advance(Duration.millis(20)); + const group = await late.ordered().nextGroup(); + const frame = await group?.readFrame(); + if (!frame) throw new Error("expected a replayed frame"); + expect(Moq.Varint.decode(frame.payload)[0]).toBe(143 * 20_000); + } finally { + late?.close(); + encoder.close(); + broadcast.close(); + installed.close(); } - const net = broadcast.net.peek(); - if (!net) throw new Error("expected network broadcast"); - // At 3000 ms, a 130 ms budget still wants the frames ending after 2870 ms: 143 onward. - late = net.track("audio").subscribe({ delay: { max: Duration.millis(130) } }); - await settle(); - expect(encoder.out.stats.peek().frames).toBe(7 - FakeAudioEncoder.LAG); - await feed.push({ timestamp: Duration.micros(150 * 20_000), channels: [new Float32Array(960)] }); - await feed.drain(); - const group = await late.ordered().nextGroup(); - const frame = await group?.readFrame(); - if (!frame) throw new Error("expected a replayed frame"); - expect(Moq.Varint.decode(frame.payload)[0]).toBe(143 * 20_000); - } finally { - late?.close(); - encoder.close(); - broadcast.close(); - installed.close(); - } -}); + }); +} // A fetch's own budget judges only the group it asks for, so it must not narrow what the encoder // replays for the live subscribers that come later. @@ -1529,7 +1537,7 @@ test("a fetch alone replays the whole retention", async () => { const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); // A first viewer leaves group 0 in the cache for the fetch to find. - const first = net.track("audio").subscribe({ delay: { max: null } }); + const first = net.track("audio").subscribe({ delay: { min: null, max: null } }); await settle(); await feed.push({ timestamp: Duration.micros(0), channels: [new Float32Array(960)] }); await feed.drain(); @@ -1581,7 +1589,7 @@ test("replayed raw audio raises no catalog delay", async () => { new Estimator().write(150 * 20_000, broadcast.order, true); const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); await settle(); for (let index = 150; index < 153; index++) { await feed.push({ timestamp: Duration.micros(index * 20_000), channels: [new Float32Array(960)] }); @@ -1706,7 +1714,7 @@ test.each([false, true])("audio retained before a disable is encoded on demand ( const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); await settle(); expect(encoder.out.catalog.peek()?.enabled).toBe(false); expect(encoder.out.active.peek()).toBe(false); @@ -1773,7 +1781,7 @@ test("audio retained across a disable and re-enable is served before the break", const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); await settle(20); // The four retained frames, then the marker, then the three resumed. expect(encoder.out.stats.peek().frames).toBe(7); @@ -1840,7 +1848,7 @@ test("a capture gap inside replayed audio keeps every frame on both sides of it" const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); await settle(20); // Every frame on both sides of the gap. expect(encoder.out.stats.peek().frames).toBe(8); @@ -1965,7 +1973,7 @@ test("every disable retained before demand gets its own break", async () => { const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); await settle(20); expect(encoder.out.stats.peek().frames).toBe(7); expect(discontinuity.mock.calls.map(([end]) => end)).toEqual([ @@ -2020,7 +2028,7 @@ test("a break after expired history keeps the resumed audio flowing", async () = const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); await settle(20); expect(encoder.out.stats.peek().frames).toBe(3); await push(2_060_000, 2); @@ -2071,7 +2079,7 @@ test("demand between a re-enable and the resumed audio still breaks after the re const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); await settle(20); expect(encoder.out.stats.peek().frames).toBe(4); expect(discontinuity.mock.calls.map(([end]) => end)).toEqual([Duration.micros(80_000)]); diff --git a/js/publish/src/audio/encoder.ts b/js/publish/src/audio/encoder.ts index 55c247c52f8..943c9579546 100644 --- a/js/publish/src/audio/encoder.ts +++ b/js/publish/src/audio/encoder.ts @@ -220,9 +220,11 @@ export class Encoder { // discontinuity and re-anchors on. #pipeline: Pipeline | undefined; #encoding = false; - // The demand's max delay, past which retained audio is stale to every subscriber, or undefined - // for the retention alone. Replayed groups land as fresh arrivals, so only this cut keeps a - // rejoining live viewer from receiving all the retained audio before the live edge. + // How far back retained audio is replayed: the demand's min delay, which a player states as its + // playout delay plus the codec's warmup, so a joining viewer gets the audio its buffer and decoder + // need and no older. A subscription stating none (moq-transport, moq-lite before 07) falls back to + // its max delay, and fetches alone to the retention. Replayed groups land as fresh arrivals, so only + // this cut keeps a rejoining live viewer from receiving all the retained audio before the live edge. #budget: Duration | undefined; // The rendition's track and producer, kept across subscriber changes. @@ -489,8 +491,7 @@ export class Encoder { const onDemand = effect.get(broadcast.in.onDemand) !== "off"; const demanded = !!track && effect.get(track.demand().wanted) !== undefined; // Demand from fetches alone carries no subscription, so no budget beyond the retention. - const max = track && effect.get(track.subscription)?.delay?.max; - this.#budget = max === undefined || max === null ? undefined : max; + this.#budget = replayBudget(track && effect.get(track.subscription)); // Not gated on `enabled` or `muted`: a paused rendition takes no capture, but still encodes what it retained. this.#encoding = !!format && !!track && !fatal && (!onDemand || demanded); effect.set(this.#out.active, enabled && this.#encoding, false); @@ -1030,6 +1031,16 @@ export function resolve(captured: Format, selected: Codec): Resolved { }; } +// How far back a subscription wants retained audio replayed: its stated min delay, within its max, +// or its max when it states no min. Undefined with no subscription, leaving the retention alone. +function replayBudget(subscription: Moq.Track.Subscription | undefined): Duration | undefined { + const delay = subscription?.delay; + if (!delay) return undefined; + const max = delay.max ?? undefined; + if (delay.min === null || delay.min === undefined) return max; + return max === undefined ? delay.min : delay.min.min(max); +} + /** * How far apart audio groups start: a frame opens one once it is at least `join` past the * open group's start, so groups are a whole number of frames apart, and at least one. diff --git a/js/watch/src/audio/buffer.test.ts b/js/watch/src/audio/buffer.test.ts index f777dd1ecd8..a047e2acc4f 100644 --- a/js/watch/src/audio/buffer.test.ts +++ b/js/watch/src/audio/buffer.test.ts @@ -668,6 +668,29 @@ describe("AudioBuffer steering", () => { } }); + // A rejoin's replayed backlog trails the clock by seconds, and the ring skips it unheard. Held to the + // ring's pace, the live edge behind it would arrive only once the ring had drained it all. + it("releases at once, and reports stale, what trails the clock past its bound", () => { + const clock = started(); + const { buffer, state } = fake(100, { + bound: 200, + clock: { position: () => 3000, out: { reference: new Signal(0) } }, + }); + const signal = new AbortController().signal; + try { + buffer.insert(ms(2900), [new Float32Array(160)]); + state(ms(2900), false, 0); + // Within the bound the ring plays it, so the gate holds what lies past its floor and window. + expect(buffer.stale(ms(2850))).toBe(false); + expect(Bun.peek.status(buffer.wait(ms(3300), signal))).toBe("pending"); + expect(buffer.stale(ms(2700))).toBe(true); + expect(Bun.peek.status(buffer.wait(ms(1000), signal))).toBe("fulfilled"); + } finally { + buffer.close(); + clock.close(); + } + }); + // Before media zero's turn, as while an opening waits for its output to sound it, the clock pairs // at its negative position rather than the zero it reads as. it("pairs a clock short of media zero at its negative position", () => { diff --git a/js/watch/src/audio/buffer.ts b/js/watch/src/audio/buffer.ts index 98bf4983e74..0177b4a1b0c 100644 --- a/js/watch/src/audio/buffer.ts +++ b/js/watch/src/audio/buffer.ts @@ -187,6 +187,15 @@ export class Backpressure { }); } + /** + * Release a frame at once whose output the ring would skip unheard, so a backlog behind the clock + * decodes at the codec's pace, only to warm it, rather than at the pace the ring drains. It counts + * as decoding until its output answers it. + */ + pass(timestamp: number): void { + this.#decoding.push(timestamp); + } + /** The ring was flushed for a new timeline: its playhead is unknown until the next frame anchors it. */ reset(): void { this.#playhead = undefined; @@ -614,10 +623,26 @@ export class AudioBuffer { * whose whole target fits the floor. Resolves early, releasing the wait, once `signal` aborts. */ wait(timestamp: number, signal: AbortSignal): Promise { + if (this.stale(timestamp)) { + this.#backpressure.pass(timestamp); + return Promise.resolve(); + } // The worklet-reported playhead lags by a state-message interval, which only holds frames longer. return this.#backpressure.wait(timestamp, signal); } + /** + * Whether media at `timestamp`, in microseconds, trails the shared clock past the bound, so the ring + * would skip it unheard, as a publisher's replayed backlog does on a rejoin. The decoder still + * decodes it to warm the codec, but writes none of it: a ring anchored on it would play catch-up + * instead of the live edge. + */ + stale(timestamp: number): boolean { + const position = this.#sync?.position(); + if (position === undefined) return false; + return position * 1000 - timestamp > Math.max(this.#bound.asMicros(), WINDOW); + } + /** Stop observing the worklet and release every pending wait. */ close(): void { this.#backpressure.close(); // never leave a decode loop awaiting a closed buffer diff --git a/js/watch/src/audio/decoder.test.ts b/js/watch/src/audio/decoder.test.ts index a7663194b3b..74f92a69bb8 100644 --- a/js/watch/src/audio/decoder.test.ts +++ b/js/watch/src/audio/decoder.test.ts @@ -1031,7 +1031,9 @@ describe("Decoder across a rendition's absence", () => { expect(playback.stalled()).toBe(false); expect(playback.ended()).toBe(false); - await clock.advance(Duration.millis(1000)); + // Past the end, but not so far that the replacement continuing the timeline trails the clock + // past the stretch bound, which a live replacement never does. + await clock.advance(Duration.millis(300)); await microtasks(); expect(playback.ended()).toBe(true); expect(playback.stats()).toMatchObject({ underruns: 0, stalls: 0 }); @@ -1164,6 +1166,37 @@ describe("Decoder across a rendition's absence", () => { } }); + // A publisher replays the audio it captured while nobody listened, so a viewer falling back to it + // receives seconds of it behind the picture, in order over a WebSocket. Written to the ring, it + // anchored playback there, and the ring skipped and re-stalled on each refill until it ran out. + it("decodes a backlog behind the clock to warm the codec but plays from the live edge", async () => { + const playback = await play(Duration.millis(50)); + const video = playback.sync.track({ target: new Signal(undefined) }); + try { + // The picture anchors the clock at the live edge, 3 s in. + video.received(Duration.secs(3), now()); + const stale = []; + for (let ms = 1000; ms < 1400; ms += 20) { + frameTimestamp = (ms - 20) * 1000; + playback.push({ ...frame(), group: BigInt(ms / 20) }); + stale.push(ms * 1000); + await microtasks(); + } + for (let ms = 3000; ms < 3200; ms += 20) { + frameTimestamp = (ms - 20) * 1000; + playback.push({ ...frame(), group: BigInt(ms / 20) }); + await microtasks(); + } + expect(audioInputs.slice(0, stale.length)).toEqual(stale); + const inserted = playback.inserted(); + expect(inserted.length).toBeGreaterThan(0); + expect(Math.min(...inserted)).toBeGreaterThanOrEqual(3_000_000); + } finally { + video.close(); + playback.close(); + } + }); + for (const [name, patch] of Object.entries({ container: { container: { kind: "loc" } }, codec: { codec: "mp4a.40.2" }, diff --git a/js/watch/src/audio/decoder.ts b/js/watch/src/audio/decoder.ts index 0f9b299970d..29b66380e03 100644 --- a/js/watch/src/audio/decoder.ts +++ b/js/watch/src/audio/decoder.ts @@ -1099,6 +1099,13 @@ export class Decoder { const start = Duration.micros(timestamp); const end = Duration.micros(timestamp + durationMicro); + // Audio the ring would skip anyway only warmed the codec. Written, it would anchor the ring behind + // the clock, which then skips and re-stalls on each refill until the backlog runs out. + if (ring.stale(end.asMicros())) { + sample.close(); + return; + } + // A new subscription has taken over the timeline: drop the previous one's write-ahead tail // rather than letting it play out after this frame. See #runDecoder. const takeover = this.#handover.takeover(); diff --git a/rs/moq-audio/src/encode/capture.rs b/rs/moq-audio/src/encode/capture.rs index a1ee203e5fe..a2394bdb947 100644 --- a/rs/moq-audio/src/encode/capture.rs +++ b/rs/moq-audio/src/encode/capture.rs @@ -547,10 +547,11 @@ mod tests { } } - /// Subscribe with a `max` delay, which bounds how much retained history the encoder replays. - async fn subscribe(broadcast: &moq_net::broadcast::Consumer, max: Duration) -> Track { - let subscription = - moq_net::track::Subscription::default().with_delay(moq_net::track::Delay::default().with_max(max)); + /// Subscribe playing `delay` behind the live edge, which bounds how much retained history the + /// encoder replays. + async fn subscribe(broadcast: &moq_net::broadcast::Consumer, delay: Duration) -> Track { + let subscription = moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_min(delay).with_max(delay)); let subscriber = broadcast.track("audio").unwrap().subscribe(subscription).await.unwrap(); Track::new(subscriber, Wire(Kind::Audio)) } diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index 48b811a3f89..e170009a64a 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -539,7 +539,7 @@ impl Producer { self.backlog.expire(now); // Replayed groups land as fresh arrivals, so only this cut keeps a rejoining live // subscriber from receiving all the retained audio before the live edge. - if active && let Some(budget) = self.track.track().subscription().and_then(|sub| sub.delay.max) { + if active && let Some(budget) = self.track.track().subscription().and_then(|sub| replay_budget(sub.delay)) { self.backlog.expire_within(now, budget); } // Live input completes the partial frame within one codec frame, so only idle refreshes expire it. @@ -883,6 +883,17 @@ impl Finished { } } +/// How far back a subscription wants retained audio replayed: its stated min delay, which a player +/// states as its playout delay plus the codec's warmup, within its max. One stating no min +/// (moq-transport, moq-lite before 07) is bounded by its max alone, and `None` leaves the retention. +fn replay_budget(delay: moq_net::track::Delay) -> Option { + match (delay.min, delay.max) { + (Some(min), Some(max)) => Some(min.min(max)), + (Some(min), None) => Some(min), + (None, max) => max, + } +} + #[cfg(test)] mod tests { async fn decoded_frame(consumer: &mut crate::decode::Consumer) -> Result, Error> { @@ -1030,7 +1041,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -1068,9 +1079,23 @@ mod tests { } /// Replayed groups reach the track as fresh arrivals, so only the encoder can tell they are old: - /// a live viewer rejoining after seconds idle would otherwise receive all of them first. + /// a live viewer rejoining after seconds idle would otherwise receive all of them first. It states + /// how far back it plays as its min delay, which bounds the replay however far its max delay + /// reaches; one that states none (moq-transport, moq-lite before 07) is bounded by its max alone. #[tokio::test(start_paused = true)] - async fn demand_replays_only_the_history_its_max_delay_wants() { + async fn demand_replays_only_the_history_its_delay_wants() { + let budget = Duration::from_millis(130); + let long = Duration::from_secs(10); + for delay in [ + moq_net::track::Delay::default().with_min(budget).with_max(long), + moq_net::track::Delay::default().with_min(None).with_max(budget), + moq_net::track::Delay::default().with_min(long).with_max(budget), + ] { + replays_from(delay).await; + } + } + + async fn replays_from(delay: moq_net::track::Delay) { let input = Input::default(); let options = Options { track: Some("audio".into()), @@ -1095,8 +1120,7 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_millis(130))), + moq_net::track::Subscription::default().with_delay(delay), ) .await .unwrap(); @@ -1176,7 +1200,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -1222,7 +1246,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -1455,7 +1479,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -1529,7 +1553,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), ) .await .unwrap(), @@ -1740,7 +1764,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -1874,7 +1898,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -2034,7 +2058,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2079,7 +2103,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2190,7 +2214,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2245,7 +2269,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2300,7 +2324,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2361,7 +2385,7 @@ mod tests { .unwrap() .subscribe( moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), + .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), ) .await .unwrap(); diff --git a/test/interop/README.md b/test/interop/README.md index c54040e5236..a3bcfcba31c 100644 --- a/test/interop/README.md +++ b/test/interop/README.md @@ -550,11 +550,16 @@ at QUIC's 1 ms timer granularity, the 500 ms GOP, one frame to decode, the 50 ms playback delay, and one page sample plus one driver poll. Publisher preparation is not part of it. -The replacement case is quarantined from the default run off Chromium: on Firefox its -fallback intermittently misses the switch bound, as often with the audio fixes that -restored it on Chromium as without them. Run it explicitly with -`--media-cases replacement --media-engines firefox`; its content, sync, switch, and -fallback assertions remain active. +Off Chromium, the replacement case skips its IETF draft 22 viewer and reports it as +quarantined. The fallback rejoins an incumbent that encodes its audio on demand, so the +incumbent replays the audio it captured while nobody listened, from where the returning +subscription says it plays. moq-transport states no delay, so the incumbent can only cut +that replay at the retention, and a WebSocket delivers all of it ahead of the live edge; +Chromium's WebTransport delivers newest first. This is a known moq-transport limitation. +A lite-07 viewer states its delay, so the replay starts there; a lite-06 viewer states +only its max delay, which bounds it to about two seconds. Either way the player decodes +audio that trails the clock past its bound only to warm the codec, so the moq-lite +viewers rejoin at the live edge on every engine. The CLI integration suite's `instances` cases exercise the real QUIC relay with lite-07, lite-06, and IETF draft 22. Held catalog snapshots and media diff --git a/test/interop/clients/js/media.ts b/test/interop/clients/js/media.ts index 9efef98118f..18d0cc8ef09 100644 --- a/test/interop/clients/js/media.ts +++ b/test/interop/clients/js/media.ts @@ -129,13 +129,17 @@ const native = values.publisher === "native"; const isEngine = (name: string): name is Engine => ENGINES.some((engine) => engine === name); const requestedPublisher = values["publisher-engine"] ?? "chromium"; const requestedPlayer = values["player-engine"] ?? "chromium"; -// Restored on Chromium only: on Firefox its fallback intermittently misses the switch bound. -const quarantined = !native && values.cases === undefined && requestedPlayer !== "chromium"; +// moq-transport states no delay, so the incumbent replays its whole retention to a returning +// viewer, and an in-order transport delivers all of it ahead of the live edge. Chromium's +// WebTransport delivers newest first and hides it; the other engines play over WebSocket. +const replacementVersions = ["moq-lite-07-wip", "moq-lite-06", "moq-transport-22"].filter( + (version) => requestedPlayer === "chromium" || version !== "moq-transport-22", +); const selected = new Set( values.cases === undefined ? native ? NATIVE_CASES - : CASES.filter((name) => !quarantined || name !== "replacement") + : CASES : values.cases === "none" ? [] : values.cases.split(","), @@ -163,9 +167,9 @@ if ( const publisherEngine = requestedPublisher as Engine; const playerEngine = requestedPlayer as Engine; const wants = (name: Case) => selected.has(name); -if (quarantined) { +if (!native && selected.has("replacement") && !replacementVersions.includes("moq-transport-22")) { console.error( - "QUARANTINE\treplacement\tintermittent fallback past the switch bound off Chromium; see quest/m0/interop-green/replacement-audio.md", + "QUARANTINE\treplacement moq-transport-22\tmoq-transport states no delay, so the incumbent replays its whole retention ahead of the live edge over an in-order transport; see test/interop/README.md", ); } @@ -1292,7 +1296,7 @@ try { // can be explained by the incumbent timing out or closing its session. if (wants("replacement")) { const sourceBrowser = await browserFor("publisher"); - for (const version of ["moq-lite-07-wip", "moq-lite-06", "moq-transport-22"]) { + for (const version of replacementVersions) { console.error(`=== live replacement and fallback: ${version} ===`); const name = `replacement-${version}-${process.pid}.hang`; const [old, oldErrors] = await open( From adadd40b2e8fc948f50369358a4a93d1265a2b57 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 15:10:40 -0800 Subject: [PATCH 4/9] quest: finish replacement-audio; format and gate replay_budget Co-Authored-By: Claude Opus 5.5 --- js/publish/src/audio/encoder.test.ts | 4 +- quest/m0/interop-green/README.md | 11 ++- quest/m0/interop-green/replacement-audio.md | 75 -------------- rs/moq-audio/src/encode/producer.rs | 104 ++++++++++++++------ 4 files changed, 84 insertions(+), 110 deletions(-) delete mode 100644 quest/m0/interop-green/replacement-audio.md diff --git a/js/publish/src/audio/encoder.test.ts b/js/publish/src/audio/encoder.test.ts index ea0a0474a7f..9cea701289f 100644 --- a/js/publish/src/audio/encoder.test.ts +++ b/js/publish/src/audio/encoder.test.ts @@ -1331,7 +1331,9 @@ test.each([undefined, "off", "all"] as const)( await push(4); expect(encoder.out.stats.peek().frames).toBe(onDemand === "off" ? 4 : 0); - late = net.track("audio").subscribe({ delay: { min: Duration.millis(10_000), max: Duration.millis(10_000) } }); + late = net + .track("audio") + .subscribe({ delay: { min: Duration.millis(10_000), max: Duration.millis(10_000) } }); await settle(); expect(encoder.out.active.peek()).toBe(true); await push(3); diff --git a/quest/m0/interop-green/README.md b/quest/m0/interop-green/README.md index ccc4f91dcdf..512d8d288a7 100644 --- a/quest/m0/interop-green/README.md +++ b/quest/m0/interop-green/README.md @@ -224,10 +224,13 @@ Scheduled runs stalls past the 1039 ms bound after the route falls back. Reproduced at 1 of 8 and 2 of 12 loaded local runs (the audio ring still stalled 600 ms after the first audio group of the returned incumbent). Decided 2026-11-15 (AFK, recommendation, for review): - `replacement` is quarantined by default on every engine again, and the child - [replacement audio](/quest/m0/interop-green/replacement-audio.md) root-causes and restores it. - Its first PR found the stall: the decode gate released the backlog behind the clock one window - per worklet report. Its second restored the case on Chromium; Firefox's quarantine is in the child. + `replacement` is quarantined by default on every engine again, and a child quest root-caused + and restored it. Its first PR found the stall: the decode gate released the backlog behind the + clock one window per worklet report. Its second restored the case on Chromium, and + [moq-dev/moq#9023](https://github.com/moq-dev/moq/pull/9023) on Firefox: the incumbent replays + audio only from a returning viewer's min delay, and the player decodes what trails the clock + only to warm the codec. Off Chromium the moq-transport-22 row stays quarantined, since that + wire states no delay to cut the replay at. Public API: none. Wire: none. diff --git a/quest/m0/interop-green/replacement-audio.md b/quest/m0/interop-green/replacement-audio.md deleted file mode 100644 index cab9ec8dd6d..00000000000 --- a/quest/m0/interop-green/replacement-audio.md +++ /dev/null @@ -1,75 +0,0 @@ -# [L] Restore the replacement media case on Firefox - -## Goal - -`just test media --media-cases replacement --media-engines firefox` passes every run, and the -default media run on Firefox runs it again. Remove the quarantine off Chromium in -`test/interop/clients/js/media.ts` and `test/interop/README.md`. - -## Plan - -Chromium runs the case again by default. Its audio no longer goes silent across the groups a -starved viewer skips: the restarted decoder is re-primed on the frames before each gap, and a gap -lands what the decoder still holds instead of resetting it away. - -Decided 2026-11-16 (maintainer, on [moq-dev/moq#8980](https://github.com/moq-dev/moq/pull/8980#issuecomment-6943194650)): - -- ✅ Lift the quarantine on Chromium even though loaded runs still fail about one in six (rejected: - keep it until the underruns are fixed). What still fails there is the steady-state starvation the - sibling cases share, not the gaps. -- ✅ Leave Firefox to this quest (rejected: fix it in that PR). Its fallback misses the bound as - often before the gap fixes as after, so it is a separate cause. - -### Root cause of the Firefox fallback miss - -On Firefox the fallback misses its 1039 ms bound in about one run in twenty on this host (3 of about -60), and its fallback time swings from 207 ms to over 1000 ms on the passing runs. The picture comes -back on time; the audio does not. - -The incumbent's audio is replayed history. `@moq/publish` encodes audio on demand -(`js/publish/src/audio/encoder.ts`): once the viewer moves to the replacement, the relay cancels its -idle upstream subscription, encoding stops, and capture keeps filling the retained `Backlog`. When -the viewer falls back, demand returns and the encoder replays the retained PCM as fresh groups, cut -only by the subscription's max delay: - -- moq-lite-07 and -06 viewers ask audio with a 2.13 s max delay, so about 2.1 s of audio arrives - behind the live picture. One failing run played the incumbent's video at 17.26 s while its audio - sat stalled at 15.09 s with the replayed groups already received. -- A moq-transport-22 viewer carries no max delay, so the relay asks upstream with none, and the whole - retention is replayed. The publisher answers before encoding the replay, so its Largest (and the - viewer's join) is where the earlier subscription stopped, about 8 s back. Two failing runs joined - there, dropped hundreds of groups as slow, and froze the picture with the audio stuck seconds - behind it. - -Chromium hides this because WebTransport delivers newest first, so the live edge lands ahead of the -replay. Firefox plays over the WebSocket fallback, which delivers in order, so the player gets the -replay first and stalls on it instead of skipping onto the shared clock. - -Ruled out: the relay. Re-opening its idle copy from the newest group it cached asks for less backlog -than a fresh copy's oldest-group ask, and the replay arrives as new sequence numbers either way: -asking from the live edge instead failed the same way. - -Fix options, pending the maintainer under Waiting: - -1. js/watch drops audio that lands behind the shared clock before it decodes, so any backlog within - the max delay (this replay, or a relay cache on a cold join) costs the player nothing and it - rejoins the clock at once. Recommended: the max delay is the player's own ask, so it must handle - what that ask returns from every publisher and relay; Firefox's in-order transport shows it does - not today. -2. `@moq/publish` replays retained audio to a returning subscriber only up to its min delay (the - jitter target) rather than its max. Narrower: it fixes this publisher only, while a relay cache - still serves the same backlog. -3. The moq-transport path carries the viewer's budget upstream (draft-22's delivery timeout), so an - IETF viewer stops asking for the whole retention. Worth it only alongside 1, since moq-transport - is the low-effort compatibility layer. - -WebKit was never run; check it once Firefox passes. - -Public API: none. Wire: none. - -## Waiting - -- Maintainer: pick the fix for the replayed audio a returning viewer gets (Plan options 1 to 3). - Recommended: option 1 in js/watch, since the player's own max delay invites this backlog from any - source; option 3 only if moq-transport-22 must pass on its own. Done once this item is replaced by - the decision in the Plan. diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index e170009a64a..f426a76a6ff 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -539,7 +539,13 @@ impl Producer { self.backlog.expire(now); // Replayed groups land as fresh arrivals, so only this cut keeps a rejoining live // subscriber from receiving all the retained audio before the live edge. - if active && let Some(budget) = self.track.track().subscription().and_then(|sub| replay_budget(sub.delay)) { + if active + && let Some(budget) = self + .track + .track() + .subscription() + .and_then(|sub| replay_budget(sub.delay)) + { self.backlog.expire_within(now, budget); } // Live input completes the partial frame within one codec frame, so only idle refreshes expire it. @@ -886,6 +892,7 @@ impl Finished { /// How far back a subscription wants retained audio replayed: its stated min delay, which a player /// states as its playout delay plus the codec's warmup, within its max. One stating no min /// (moq-transport, moq-lite before 07) is bounded by its max alone, and `None` leaves the retention. +#[cfg(any(feature = "capture", test))] fn replay_budget(delay: moq_net::track::Delay) -> Option { match (delay.min, delay.max) { (Some(min), Some(max)) => Some(min.min(max)), @@ -1040,8 +1047,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(1)) + .with_max(Duration::from_secs(1)), + ), ) .await .unwrap(); @@ -1119,9 +1129,7 @@ mod tests { let track = consumer .track("audio") .unwrap() - .subscribe( - moq_net::track::Subscription::default().with_delay(delay), - ) + .subscribe(moq_net::track::Subscription::default().with_delay(delay)) .await .unwrap(); let mut reader = @@ -1199,8 +1207,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(60)) + .with_max(Duration::from_secs(60)), + ), ) .await .unwrap(); @@ -1245,8 +1256,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(60)) + .with_max(Duration::from_secs(60)), + ), ) .await .unwrap(); @@ -1478,8 +1492,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(60)) + .with_max(Duration::from_secs(60)), + ), ) .await .unwrap(); @@ -1552,8 +1569,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(1)) + .with_max(Duration::from_secs(1)), + ), ) .await .unwrap(), @@ -1763,8 +1783,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(60)) + .with_max(Duration::from_secs(60)), + ), ) .await .unwrap(); @@ -1897,8 +1920,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(60)).with_max(Duration::from_secs(60))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(60)) + .with_max(Duration::from_secs(60)), + ), ) .await .unwrap(); @@ -2057,8 +2083,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(1)) + .with_max(Duration::from_secs(1)), + ), ) .await .unwrap(); @@ -2102,8 +2131,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(1)) + .with_max(Duration::from_secs(1)), + ), ) .await .unwrap(); @@ -2213,8 +2245,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(1)) + .with_max(Duration::from_secs(1)), + ), ) .await .unwrap(); @@ -2268,8 +2303,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(1)) + .with_max(Duration::from_secs(1)), + ), ) .await .unwrap(); @@ -2323,8 +2361,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(1)) + .with_max(Duration::from_secs(1)), + ), ) .await .unwrap(); @@ -2384,8 +2425,11 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(Duration::from_secs(1)).with_max(Duration::from_secs(1))), + moq_net::track::Subscription::default().with_delay( + moq_net::track::Delay::default() + .with_min(Duration::from_secs(1)) + .with_max(Duration::from_secs(1)), + ), ) .await .unwrap(); From 12f1ee8588ce92cb048d9ddb8f37622b19b80d5c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 15:20:11 -0800 Subject: [PATCH 5/9] fix(publish,audio): only a live join's min delay bounds the replay; keep finite tails A history range (an archive's oldest start) ignores min delay on the wire, so it keeps the max-delay cut. Audio before a declared end is never stale, since the ring plays a finite tail out. Co-Authored-By: Claude Opus 5.5 --- doc/lib/js/publish.md | 5 +- doc/lib/rs/moq-audio.md | 4 +- js/publish/src/audio/encoder.test.ts | 49 +++++---- js/publish/src/audio/encoder.ts | 18 ++-- js/watch/src/audio/buffer.test.ts | 11 +- js/watch/src/audio/buffer.ts | 10 +- quest/m0/interop-green/README.md | 5 +- rs/moq-audio/src/encode/capture.rs | 9 +- rs/moq-audio/src/encode/producer.rs | 148 +++++++++++---------------- test/interop/README.md | 15 ++- 10 files changed, 135 insertions(+), 139 deletions(-) diff --git a/doc/lib/js/publish.md b/doc/lib/js/publish.md index fd05808003f..52615980189 100644 --- a/doc/lib/js/publish.md +++ b/doc/lib/js/publish.md @@ -76,9 +76,8 @@ Video encodes while enabled, so its recent groups wait in the track's cache for a late subscription's `delay.max` or a fetch. The cache keeps an idle group for 30 seconds whatever `maxAge` says; `maxAge` only bounds how far back a reader may reach into it. Audio keeps raw PCM while nobody listens, then encodes the part of -that history within the subscribers' min delay (their playout delay plus the codec's -warmup) ahead of live audio when one arrives. A subscription that states no min delay, -as over moq-transport, gets the history within its max delay instead. Raw audio keeps up to +that history a subscriber wants ahead of live audio when one arrives: from a live join's +min delay (its playout delay plus the codec's warmup), or else within its max delay. Raw audio keeps up to `maxAge`, which for media defaults to 30 seconds (about 11.5 MB for 48 kHz stereo f32). With `maxAge: null`, which sets no publisher limit on the media, raw audio still keeps 30 seconds, since raw history needs a bound. The catalog and diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md index 08dc2c9c74d..3d9a135bd15 100644 --- a/doc/lib/rs/moq-audio.md +++ b/doc/lib/rs/moq-audio.md @@ -39,8 +39,8 @@ glides onto its clock, instead of writing silence or failing at each step. Without a subscriber the encoder keeps raw PCM up to the track's `max_age`, then encodes the part of that history within the -subscribers' min delay (their playout delay plus the codec's warmup), or their max delay -when they state no min, before continuing live when someone subscribes. With an unlimited `max_age` it keeps at most 30 seconds +subscribers want, from a live join's min delay (its playout delay plus the codec's warmup) +or else within their max delay, before continuing live when someone subscribes. With an unlimited `max_age` it keeps at most 30 seconds (about 11.5 MB for 48 kHz stereo f32). Set `encode::Capture::retain` to false to encode continuously instead. Dropping an encoder leaves the shared capture running. A device failure keeps the retained history on the track until it is subscribed or expires, then the diff --git a/js/publish/src/audio/encoder.test.ts b/js/publish/src/audio/encoder.test.ts index 9cea701289f..656be83857f 100644 --- a/js/publish/src/audio/encoder.test.ts +++ b/js/publish/src/audio/encoder.test.ts @@ -1331,9 +1331,7 @@ test.each([undefined, "off", "all"] as const)( await push(4); expect(encoder.out.stats.peek().frames).toBe(onDemand === "off" ? 4 : 0); - late = net - .track("audio") - .subscribe({ delay: { min: Duration.millis(10_000), max: Duration.millis(10_000) } }); + late = net.track("audio").subscribe({ delay: { max: Duration.millis(10_000) } }); await settle(); expect(encoder.out.active.peek()).toBe(true); await push(3); @@ -1412,7 +1410,7 @@ test("a backlog encode refusal aborts its rendition when demand arrives", async const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); // A live-edge budget would leave the retained frame unencoded. - subscriber = net.track("audio").subscribe({ delay: { min: null, max: null } }); + subscriber = net.track("audio").subscribe({ delay: { max: null } }); await settle(); expect(encoder.out.error.peek()).toBe(failure); expect(subscriber.closed.peek()).toBe(failure); @@ -1450,7 +1448,7 @@ test("default audio replays the last second after three seconds without a subscr expect(encoder.out.active.peek()).toBe(false); const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); await settle(); expect(encoder.out.stats.peek().frames).toBe(50); for (let index = 150; index < 153; index++) { @@ -1476,13 +1474,22 @@ test("default audio replays the last second after three seconds without a subscr }); // Replayed groups reach the track as fresh arrivals, so only the encoder can tell they are old: a -// live viewer rejoining after seconds idle would otherwise wade through all of them first. It states -// how far back it plays as its min delay, which bounds the replay however far its max delay reaches; -// one that states none (moq-transport, moq-lite before 07) is bounded by its max delay alone. -for (const [name, delay] of [ - ["its min delay", { min: Duration.millis(130), max: Duration.millis(10_000) }], - ["its max delay when it states no min", { min: null, max: Duration.millis(130) }], - ["its max delay when that is shorter than its min", { min: Duration.millis(10_000), max: Duration.millis(130) }], +// live viewer rejoining after seconds idle would otherwise wade through all of them first. A live +// join states how far back it plays as its min delay, which bounds the replay however far its max +// delay reaches; one that states none (moq-transport, moq-lite before 07) is bounded by its max +// alone, as is a reader asking for history. +const LIVE = [{ start: { relative: 0n } }]; +for (const [name, subscription] of [ + ["a live join's min delay", { delay: { min: Duration.millis(130), max: Duration.millis(10_000) }, ranges: LIVE }], + [ + "a live join's max delay when it states no min", + { delay: { min: null, max: Duration.millis(130) }, ranges: LIVE }, + ], + [ + "a live join's max delay when that is shorter than its min", + { delay: { min: Duration.millis(10_000), max: Duration.millis(130) }, ranges: LIVE }, + ], + ["a history reader's max delay", { delay: { max: Duration.millis(130) } }], ] as const) { test(`demand replays only the retained audio ${name} still wants`, async () => { using _webcodecs = installFakeWebCodecs(); @@ -1504,7 +1511,7 @@ for (const [name, delay] of [ const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); // At 3000 ms, a 130 ms budget still wants the frames ending after 2870 ms: 143 onward. - late = net.track("audio").subscribe({ delay }); + late = net.track("audio").subscribe(subscription); await settle(); expect(encoder.out.stats.peek().frames).toBe(7 - FakeAudioEncoder.LAG); await feed.push({ timestamp: Duration.micros(150 * 20_000), channels: [new Float32Array(960)] }); @@ -1539,7 +1546,7 @@ test("a fetch alone replays the whole retention", async () => { const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); // A first viewer leaves group 0 in the cache for the fetch to find. - const first = net.track("audio").subscribe({ delay: { min: null, max: null } }); + const first = net.track("audio").subscribe({ delay: { max: null } }); await settle(); await feed.push({ timestamp: Duration.micros(0), channels: [new Float32Array(960)] }); await feed.drain(); @@ -1591,7 +1598,7 @@ test("replayed raw audio raises no catalog delay", async () => { new Estimator().write(150 * 20_000, broadcast.order, true); const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); await settle(); for (let index = 150; index < 153; index++) { await feed.push({ timestamp: Duration.micros(index * 20_000), channels: [new Float32Array(960)] }); @@ -1716,7 +1723,7 @@ test.each([false, true])("audio retained before a disable is encoded on demand ( const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); await settle(); expect(encoder.out.catalog.peek()?.enabled).toBe(false); expect(encoder.out.active.peek()).toBe(false); @@ -1783,7 +1790,7 @@ test("audio retained across a disable and re-enable is served before the break", const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); await settle(20); // The four retained frames, then the marker, then the three resumed. expect(encoder.out.stats.peek().frames).toBe(7); @@ -1850,7 +1857,7 @@ test("a capture gap inside replayed audio keeps every frame on both sides of it" const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); await settle(20); // Every frame on both sides of the gap. expect(encoder.out.stats.peek().frames).toBe(8); @@ -1975,7 +1982,7 @@ test("every disable retained before demand gets its own break", async () => { const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); await settle(20); expect(encoder.out.stats.peek().frames).toBe(7); expect(discontinuity.mock.calls.map(([end]) => end)).toEqual([ @@ -2030,7 +2037,7 @@ test("a break after expired history keeps the resumed audio flowing", async () = const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); await settle(20); expect(encoder.out.stats.peek().frames).toBe(3); await push(2_060_000, 2); @@ -2081,7 +2088,7 @@ test("demand between a re-enable and the resumed audio still breaks after the re const net = broadcast.net.peek(); if (!net) throw new Error("expected network broadcast"); - late = net.track("audio").subscribe({ delay: { min: Duration.millis(1000), max: Duration.millis(1000) } }); + late = net.track("audio").subscribe({ delay: { max: Duration.millis(1000) } }); await settle(20); expect(encoder.out.stats.peek().frames).toBe(4); expect(discontinuity.mock.calls.map(([end]) => end)).toEqual([Duration.micros(80_000)]); diff --git a/js/publish/src/audio/encoder.ts b/js/publish/src/audio/encoder.ts index 943c9579546..59853c1d82e 100644 --- a/js/publish/src/audio/encoder.ts +++ b/js/publish/src/audio/encoder.ts @@ -220,11 +220,9 @@ export class Encoder { // discontinuity and re-anchors on. #pipeline: Pipeline | undefined; #encoding = false; - // How far back retained audio is replayed: the demand's min delay, which a player states as its - // playout delay plus the codec's warmup, so a joining viewer gets the audio its buffer and decoder - // need and no older. A subscription stating none (moq-transport, moq-lite before 07) falls back to - // its max delay, and fetches alone to the retention. Replayed groups land as fresh arrivals, so only - // this cut keeps a rejoining live viewer from receiving all the retained audio before the live edge. + // How far back retained audio is replayed (see replayBudget): a live join's min delay, else the max + // delay, and for fetches alone the retention. Replayed groups land as fresh arrivals, so only this + // cut keeps a rejoining live viewer from receiving all the retained audio before the live edge. #budget: Duration | undefined; // The rendition's track and producer, kept across subscriber changes. @@ -1031,13 +1029,17 @@ export function resolve(captured: Format, selected: Codec): Resolved { }; } -// How far back a subscription wants retained audio replayed: its stated min delay, within its max, -// or its max when it states no min. Undefined with no subscription, leaving the retention alone. +// How far back a subscription wants retained audio replayed. A live join starts its min delay behind +// the live edge, which a player states as its playout delay plus the codec's warmup, so that bounds +// it within the max delay. Any other range, or a live join stating no min (moq-transport, moq-lite +// before 07), wants all its max delay keeps. Undefined with no subscription, leaving the retention. function replayBudget(subscription: Moq.Track.Subscription | undefined): Duration | undefined { const delay = subscription?.delay; if (!delay) return undefined; const max = delay.max ?? undefined; - if (delay.min === null || delay.min === undefined) return max; + const ranges = subscription.ranges ?? [{}]; + const live = ranges.length > 0 && ranges.every((range) => range.start?.relative === 0n && range.end === undefined); + if (!live || delay.min === null || delay.min === undefined) return max; return max === undefined ? delay.min : delay.min.min(max); } diff --git a/js/watch/src/audio/buffer.test.ts b/js/watch/src/audio/buffer.test.ts index a047e2acc4f..d1dca1aa3f9 100644 --- a/js/watch/src/audio/buffer.test.ts +++ b/js/watch/src/audio/buffer.test.ts @@ -670,7 +670,7 @@ describe("AudioBuffer steering", () => { // A rejoin's replayed backlog trails the clock by seconds, and the ring skips it unheard. Held to the // ring's pace, the live edge behind it would arrive only once the ring had drained it all. - it("releases at once, and reports stale, what trails the clock past its bound", () => { + it("releases at once, and reports stale, what trails the clock past its bound short of a declared end", () => { const clock = started(); const { buffer, state } = fake(100, { bound: 200, @@ -685,6 +685,15 @@ describe("AudioBuffer steering", () => { expect(Bun.peek.status(buffer.wait(ms(3300), signal))).toBe("pending"); expect(buffer.stale(ms(2700))).toBe(true); expect(Bun.peek.status(buffer.wait(ms(1000), signal))).toBe("fulfilled"); + + // A finite tail plays out however far it trails the clock, through the next utterance too. + buffer.end(ms(2800)); + expect(buffer.stale(ms(2700))).toBe(false); + buffer.next(); + expect(buffer.stale(ms(2700))).toBe(false); + expect(buffer.stale(ms(2790))).toBe(false); + buffer.reset(); + expect(buffer.stale(ms(2700))).toBe(true); } finally { buffer.close(); clock.close(); diff --git a/js/watch/src/audio/buffer.ts b/js/watch/src/audio/buffer.ts index 0177b4a1b0c..c5a0391e0b5 100644 --- a/js/watch/src/audio/buffer.ts +++ b/js/watch/src/audio/buffer.ts @@ -321,6 +321,9 @@ export class AudioBuffer { #backpressure: Backpressure; // Whether an end was declared since the last reset, which the next audio follows. #declared = false; + // Where the latest declared end stands, until a reset that is not the next utterance. Audio before + // it is a finite tail, which the ring plays out however far it trails the clock. + #tail?: number; // Bumped per reset. A state message posted before the worklet saw it reports the ring on the old // timeline, which would open the gate. #epoch = 0; @@ -572,6 +575,7 @@ export class AudioBuffer { */ truncate(timestamp: number): void { this.#backpressure.truncated(timestamp); + if (this.#tail !== undefined) this.#tail = Math.min(this.#tail, timestamp); const msg: Truncate = { type: "truncate", timestamp }; this.#worklet.port.postMessage(msg); if (this.#declared) this.next(); @@ -584,6 +588,7 @@ export class AudioBuffer { */ end(timestamp: number): void { this.#declared = true; + this.#tail = timestamp; const msg: End = { type: "end", timestamp }; this.#worklet.port.postMessage(msg); } @@ -604,6 +609,7 @@ export class AudioBuffer { #reset(next: boolean): void { this.#declared = false; + if (!next) this.#tail = undefined; this.#epoch++; const msg: Reset = { type: "reset", @@ -635,9 +641,11 @@ export class AudioBuffer { * Whether media at `timestamp`, in microseconds, trails the shared clock past the bound, so the ring * would skip it unheard, as a publisher's replayed backlog does on a rejoin. The decoder still * decodes it to warm the codec, but writes none of it: a ring anchored on it would play catch-up - * instead of the live edge. + * instead of the live edge. Audio before a declared end is never stale, since the ring plays a + * finite tail out rather than skip it. */ stale(timestamp: number): boolean { + if (this.#tail !== undefined && timestamp < this.#tail) return false; const position = this.#sync?.position(); if (position === undefined) return false; return position * 1000 - timestamp > Math.max(this.#bound.asMicros(), WINDOW); diff --git a/quest/m0/interop-green/README.md b/quest/m0/interop-green/README.md index 512d8d288a7..557659d2250 100644 --- a/quest/m0/interop-green/README.md +++ b/quest/m0/interop-green/README.md @@ -227,9 +227,8 @@ Scheduled runs `replacement` is quarantined by default on every engine again, and a child quest root-caused and restored it. Its first PR found the stall: the decode gate released the backlog behind the clock one window per worklet report. Its second restored the case on Chromium, and - [moq-dev/moq#9023](https://github.com/moq-dev/moq/pull/9023) on Firefox: the incumbent replays - audio only from a returning viewer's min delay, and the player decodes what trails the clock - only to warm the codec. Off Chromium the moq-transport-22 row stays quarantined, since that + [moq-dev/moq#9023](https://github.com/moq-dev/moq/pull/9023) on Firefox: the player decodes the + audio an incumbent replays behind the clock only to warm the codec. Off Chromium the moq-transport-22 row stays quarantined, since that wire states no delay to cut the replay at. Public API: none. Wire: none. diff --git a/rs/moq-audio/src/encode/capture.rs b/rs/moq-audio/src/encode/capture.rs index a2394bdb947..a1ee203e5fe 100644 --- a/rs/moq-audio/src/encode/capture.rs +++ b/rs/moq-audio/src/encode/capture.rs @@ -547,11 +547,10 @@ mod tests { } } - /// Subscribe playing `delay` behind the live edge, which bounds how much retained history the - /// encoder replays. - async fn subscribe(broadcast: &moq_net::broadcast::Consumer, delay: Duration) -> Track { - let subscription = moq_net::track::Subscription::default() - .with_delay(moq_net::track::Delay::default().with_min(delay).with_max(delay)); + /// Subscribe with a `max` delay, which bounds how much retained history the encoder replays. + async fn subscribe(broadcast: &moq_net::broadcast::Consumer, max: Duration) -> Track { + let subscription = + moq_net::track::Subscription::default().with_delay(moq_net::track::Delay::default().with_max(max)); let subscriber = broadcast.track("audio").unwrap().subscribe(subscription).await.unwrap(); Track::new(subscriber, Wire(Kind::Audio)) } diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index f426a76a6ff..cea6577215c 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -539,13 +539,7 @@ impl Producer { self.backlog.expire(now); // Replayed groups land as fresh arrivals, so only this cut keeps a rejoining live // subscriber from receiving all the retained audio before the live edge. - if active - && let Some(budget) = self - .track - .track() - .subscription() - .and_then(|sub| replay_budget(sub.delay)) - { + if active && let Some(budget) = self.track.track().subscription().and_then(|sub| replay_budget(&sub)) { self.backlog.expire_within(now, budget); } // Live input completes the partial frame within one codec frame, so only idle refreshes expire it. @@ -889,12 +883,19 @@ impl Finished { } } -/// How far back a subscription wants retained audio replayed: its stated min delay, which a player -/// states as its playout delay plus the codec's warmup, within its max. One stating no min -/// (moq-transport, moq-lite before 07) is bounded by its max alone, and `None` leaves the retention. +/// How far back a subscription wants retained audio replayed. A live join starts its min delay +/// behind the live edge, which a player states as its playout delay plus the codec's warmup, so +/// that bounds it within the max delay. Any other range, or a live join stating no min +/// (moq-transport, moq-lite before 07), wants all its max delay keeps; `None` leaves the retention. #[cfg(any(feature = "capture", test))] -fn replay_budget(delay: moq_net::track::Delay) -> Option { - match (delay.min, delay.max) { +fn replay_budget(subscription: &moq_net::track::Subscription) -> Option { + let live = !subscription.ranges.is_empty() + && subscription + .ranges + .iter() + .all(|range| *range == moq_net::track::Range::live()); + let max = subscription.delay.max; + match (subscription.delay.min.filter(|_| live), max) { (Some(min), Some(max)) => Some(min.min(max)), (Some(min), None) => Some(min), (None, max) => max, @@ -1047,11 +1048,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(1)) - .with_max(Duration::from_secs(1)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -1089,23 +1087,35 @@ mod tests { } /// Replayed groups reach the track as fresh arrivals, so only the encoder can tell they are old: - /// a live viewer rejoining after seconds idle would otherwise receive all of them first. It states - /// how far back it plays as its min delay, which bounds the replay however far its max delay - /// reaches; one that states none (moq-transport, moq-lite before 07) is bounded by its max alone. + /// a live viewer rejoining after seconds idle would otherwise receive all of them first. A live + /// join states how far back it plays as its min delay, which bounds the replay however far its + /// max delay reaches; one that states none (moq-transport, moq-lite before 07) is bounded by its + /// max alone, as is a reader asking for history. #[tokio::test(start_paused = true)] async fn demand_replays_only_the_history_its_delay_wants() { let budget = Duration::from_millis(130); let long = Duration::from_secs(10); - for delay in [ - moq_net::track::Delay::default().with_min(budget).with_max(long), - moq_net::track::Delay::default().with_min(None).with_max(budget), - moq_net::track::Delay::default().with_min(long).with_max(budget), + let live = || vec![moq_net::track::Range::live()]; + for (delay, ranges) in [ + (moq_net::track::Delay::default().with_min(budget).with_max(long), live()), + (moq_net::track::Delay::default().with_min(None).with_max(budget), live()), + (moq_net::track::Delay::default().with_min(long).with_max(budget), live()), + // A reader reaching back for history, such as an archive, wants all its max delay keeps. + ( + moq_net::track::Delay::default().with_max(budget), + vec![moq_net::track::Range::oldest()], + ), ] { - replays_from(delay).await; + replays_from( + moq_net::track::Subscription::default() + .with_delay(delay) + .with_ranges(ranges), + ) + .await; } } - async fn replays_from(delay: moq_net::track::Delay) { + async fn replays_from(subscription: moq_net::track::Subscription) { let input = Input::default(); let options = Options { track: Some("audio".into()), @@ -1129,7 +1139,7 @@ mod tests { let track = consumer .track("audio") .unwrap() - .subscribe(moq_net::track::Subscription::default().with_delay(delay)) + .subscribe(subscription) .await .unwrap(); let mut reader = @@ -1207,11 +1217,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(60)) - .with_max(Duration::from_secs(60)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -1256,11 +1263,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(60)) - .with_max(Duration::from_secs(60)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -1492,11 +1496,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(60)) - .with_max(Duration::from_secs(60)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -1569,11 +1570,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(1)) - .with_max(Duration::from_secs(1)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), ) .await .unwrap(), @@ -1783,11 +1781,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(60)) - .with_max(Duration::from_secs(60)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -1920,11 +1915,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(60)) - .with_max(Duration::from_secs(60)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(60))), ) .await .unwrap(); @@ -2083,11 +2075,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(1)) - .with_max(Duration::from_secs(1)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2131,11 +2120,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(1)) - .with_max(Duration::from_secs(1)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2245,11 +2231,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(1)) - .with_max(Duration::from_secs(1)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2303,11 +2286,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(1)) - .with_max(Duration::from_secs(1)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2361,11 +2341,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(1)) - .with_max(Duration::from_secs(1)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), ) .await .unwrap(); @@ -2425,11 +2402,8 @@ mod tests { .track("audio") .unwrap() .subscribe( - moq_net::track::Subscription::default().with_delay( - moq_net::track::Delay::default() - .with_min(Duration::from_secs(1)) - .with_max(Duration::from_secs(1)), - ), + moq_net::track::Subscription::default() + .with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(1))), ) .await .unwrap(); diff --git a/test/interop/README.md b/test/interop/README.md index a3bcfcba31c..b7a36323cd0 100644 --- a/test/interop/README.md +++ b/test/interop/README.md @@ -552,14 +552,13 @@ is not part of it. Off Chromium, the replacement case skips its IETF draft 22 viewer and reports it as quarantined. The fallback rejoins an incumbent that encodes its audio on demand, so the -incumbent replays the audio it captured while nobody listened, from where the returning -subscription says it plays. moq-transport states no delay, so the incumbent can only cut -that replay at the retention, and a WebSocket delivers all of it ahead of the live edge; -Chromium's WebTransport delivers newest first. This is a known moq-transport limitation. -A lite-07 viewer states its delay, so the replay starts there; a lite-06 viewer states -only its max delay, which bounds it to about two seconds. Either way the player decodes -audio that trails the clock past its bound only to warm the codec, so the moq-lite -viewers rejoin at the live edge on every engine. +incumbent replays the audio it captured while nobody listened, as far back as the +returning subscription asks. A moq-lite viewer's max delay bounds that to about two +seconds, and the player decodes audio that trails the clock past its bound only to warm +the codec, so it rejoins at the live edge on every engine. moq-transport states no +delay, so the incumbent can only cut the replay at the retention, and a WebSocket +delivers all of it ahead of the live edge; Chromium's WebTransport delivers newest +first. This is a known moq-transport limitation. The CLI integration suite's `instances` cases exercise the real QUIC relay with lite-07, lite-06, and IETF draft 22. Held catalog snapshots and media From a2601b5829e85feaa5e22aa8014e90b1b7fee7f8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 15:23:03 -0800 Subject: [PATCH 6/9] style: cargo fmt Co-Authored-By: Claude Opus 5.5 --- rs/moq-audio/src/encode/producer.rs | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/rs/moq-audio/src/encode/producer.rs b/rs/moq-audio/src/encode/producer.rs index cea6577215c..f34ce300ed3 100644 --- a/rs/moq-audio/src/encode/producer.rs +++ b/rs/moq-audio/src/encode/producer.rs @@ -1136,12 +1136,7 @@ mod tests { )) .unwrap(); } - let track = consumer - .track("audio") - .unwrap() - .subscribe(subscription) - .await - .unwrap(); + let track = consumer.track("audio").unwrap().subscribe(subscription).await.unwrap(); let mut reader = moq_mux::container::Consumer::new(track, moq_mux::container::legacy::Wire(moq_mux::container::Kind::Audio)); // One replay turn, read at once as a network subscriber is served: the group it wrote is From 7b423c437d4438bb0c440093435b617ca528c076 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 16:23:52 -0800 Subject: [PATCH 7/9] fix(watch): the last packet of a finite tail ends at the declared end, so it is not stale Co-Authored-By: Claude Opus 5.5 --- js/watch/src/audio/buffer.test.ts | 2 ++ js/watch/src/audio/buffer.ts | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/js/watch/src/audio/buffer.test.ts b/js/watch/src/audio/buffer.test.ts index d1dca1aa3f9..bc0af33a947 100644 --- a/js/watch/src/audio/buffer.test.ts +++ b/js/watch/src/audio/buffer.test.ts @@ -689,6 +689,8 @@ describe("AudioBuffer steering", () => { // A finite tail plays out however far it trails the clock, through the next utterance too. buffer.end(ms(2800)); expect(buffer.stale(ms(2700))).toBe(false); + // The decoder asks with where its output ends, which for the last packet is the end itself. + expect(buffer.stale(ms(2800))).toBe(false); buffer.next(); expect(buffer.stale(ms(2700))).toBe(false); expect(buffer.stale(ms(2790))).toBe(false); diff --git a/js/watch/src/audio/buffer.ts b/js/watch/src/audio/buffer.ts index c5a0391e0b5..f372d476d60 100644 --- a/js/watch/src/audio/buffer.ts +++ b/js/watch/src/audio/buffer.ts @@ -641,11 +641,11 @@ export class AudioBuffer { * Whether media at `timestamp`, in microseconds, trails the shared clock past the bound, so the ring * would skip it unheard, as a publisher's replayed backlog does on a rejoin. The decoder still * decodes it to warm the codec, but writes none of it: a ring anchored on it would play catch-up - * instead of the live edge. Audio before a declared end is never stale, since the ring plays a + * instead of the live edge. Audio up to a declared end is never stale, since the ring plays a * finite tail out rather than skip it. */ stale(timestamp: number): boolean { - if (this.#tail !== undefined && timestamp < this.#tail) return false; + if (this.#tail !== undefined && timestamp <= this.#tail) return false; const position = this.#sync?.position(); if (position === undefined) return false; return position * 1000 - timestamp > Math.max(this.#bound.asMicros(), WINDOW); From b3ac112dd92e9880d439b4ef0bc40698ed05c4f5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 16:59:55 -0800 Subject: [PATCH 8/9] fix(watch): an aborted stale wait counts no frame, and the jitter doc names the finite-tail exception Co-Authored-By: Claude Opus 5.5 --- doc/concept/audio-jitter.md | 3 ++- js/watch/src/audio/buffer.ts | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/concept/audio-jitter.md b/doc/concept/audio-jitter.md index 75831e7537b..35bee863911 100644 --- a/doc/concept/audio-jitter.md +++ b/doc/concept/audio-jitter.md @@ -846,7 +846,8 @@ stall within the threshold costs a slowdown, a gap, and a catch-up; one past it costs a skip. Audio that arrives trailing the clock past the threshold, such as a publisher's replayed history ahead of the live edge, is decoded only to warm the codec and never reaches the playout buffer, so playback starts at the live edge -instead of anchoring on the backlog. +instead of anchoring on the backlog. The exception is audio up to a finished +stream's declared end, which plays out rather than being skipped. The splice is waveform-similarity overlap-add (WSOLA), after Verhelst and Roelands, "An overlap-add technique based on waveform similarity (WSOLA) for diff --git a/js/watch/src/audio/buffer.ts b/js/watch/src/audio/buffer.ts index f372d476d60..0addbd35c1b 100644 --- a/js/watch/src/audio/buffer.ts +++ b/js/watch/src/audio/buffer.ts @@ -629,6 +629,8 @@ export class AudioBuffer { * whose whole target fits the floor. Resolves early, releasing the wait, once `signal` aborts. */ wait(timestamp: number, signal: AbortSignal): Promise { + // An aborted loop already dropped what it was decoding, so it must not count another frame. + if (signal.aborted) return Promise.resolve(); if (this.stale(timestamp)) { this.#backpressure.pass(timestamp); return Promise.resolve(); From dda0079c4f82cadc1ed699c1c0d2663425a0dab6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 17 Nov 2026 17:55:46 -0800 Subject: [PATCH 9/9] doc(audio-jitter): the finite-tail exception covers only ends declared before the audio decodes Co-Authored-By: Claude Opus 5.5 --- doc/concept/audio-jitter.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/concept/audio-jitter.md b/doc/concept/audio-jitter.md index 35bee863911..fbfd1c48c8b 100644 --- a/doc/concept/audio-jitter.md +++ b/doc/concept/audio-jitter.md @@ -846,8 +846,10 @@ stall within the threshold costs a slowdown, a gap, and a catch-up; one past it costs a skip. Audio that arrives trailing the clock past the threshold, such as a publisher's replayed history ahead of the live edge, is decoded only to warm the codec and never reaches the playout buffer, so playback starts at the live edge -instead of anchoring on the backlog. The exception is audio up to a finished -stream's declared end, which plays out rather than being skipped. +instead of anchoring on the backlog. The exception is audio up to an end the +stream declares before that audio decodes, such as a mute, which plays out +rather than being skipped; a track's finish declares its end only after its +last audio decodes, so that audio is skipped like the rest. The splice is waveform-similarity overlap-add (WSOLA), after Verhelst and Roelands, "An overlap-add technique based on waveform similarity (WSOLA) for