Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion doc/concept/audio-jitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -843,7 +843,13 @@ content; past the threshold it snaps onto the clock, as video does. When the
audio received ahead of the playhead is about to run out, playback slows down
to bridge the gap, never trailing the clock past the threshold. A congestion
stall within the threshold costs a slowdown, a gap, and a catch-up; one past it
costs a skip.
costs a skip. Audio that arrives trailing the clock past the threshold, such as a
publisher's replayed history ahead of the live edge, is decoded only to warm the
codec and never reaches the playout buffer, so playback starts at the live edge
instead of anchoring on the backlog. The exception is audio up to an end the
stream declares before that audio decodes, such as a mute, which plays out
rather than being skipped; a track's finish declares its end only after its
last audio decodes, so that audio is skipped like the rest.

The splice is waveform-similarity overlap-add (WSOLA), after Verhelst and
Roelands, "An overlap-add technique based on waveform similarity (WSOLA) for
Expand Down
3 changes: 2 additions & 1 deletion doc/lib/js/publish.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,8 @@ Video encodes while enabled, so its recent groups wait in the track's cache for
a late subscription's `delay.max` or a fetch. The cache keeps an idle group for
30 seconds whatever `maxAge` says; `maxAge` only bounds how far back a reader
may reach into it. Audio keeps raw PCM while nobody listens, then encodes the part of
that history within the subscribers' max delay ahead of live audio when one arrives. Raw audio keeps up to
that history a subscriber wants ahead of live audio when one arrives: from a live join's
min delay (its playout delay plus the codec's warmup), or else within its max delay. Raw audio keeps up to
`maxAge`, which for media defaults to 30 seconds (about 11.5 MB for 48 kHz
stereo f32). With `maxAge: null`, which sets no publisher limit on the media, raw
audio still keeps 30 seconds, since raw history needs a bound. The catalog and
Expand Down
3 changes: 2 additions & 1 deletion doc/lib/rs/moq-audio.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@ glides onto its clock, instead of writing silence or failing at each step.

Without a subscriber the encoder keeps raw PCM
up to the track's `max_age`, then encodes the part of that history within the
subscribers' max delay before continuing live when someone subscribes. With an unlimited `max_age` it keeps at most 30 seconds
subscribers want, from a live join's min delay (its playout delay plus the codec's warmup)
or else within their max delay, before continuing live when someone subscribes. With an unlimited `max_age` it keeps at most 30 seconds
(about 11.5 MB for 48 kHz stereo f32). Set `encode::Capture::retain` to false
to encode continuously instead. Dropping an encoder leaves the shared capture running. A device
failure keeps the retained history on the track until it is subscribed or expires, then the
Expand Down
87 changes: 52 additions & 35 deletions js/publish/src/audio/encoder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1474,43 +1474,60 @@ test("default audio replays the last second after three seconds without a subscr
});

// Replayed groups reach the track as fresh arrivals, so only the encoder can tell they are old: a
// live viewer rejoining after seconds idle would otherwise wade through all of them first.
test("demand replays only the retained audio its max delay still wants", async () => {
using _webcodecs = installFakeWebCodecs();
const clock = new Manual();
const installed = clock.install();
const broadcast = new Broadcast({ origin: new Moq.Origin.Producer(), name: Moq.Path.from("rejoin.hang") });
const feed = new Feed();
const configuring = configured();
const encoder = new Encoder("audio", { broadcast, capture: fakeCapture(feed) as never });
let late: Moq.Track.Subscriber | undefined;
try {
await configuring;
await settle();
for (let index = 0; index < 150; index++) {
await feed.push({ timestamp: Duration.micros(index * 20_000), channels: [new Float32Array(960)] });
// live viewer rejoining after seconds idle would otherwise wade through all of them first. A live
// join states how far back it plays as its min delay, which bounds the replay however far its max
// delay reaches; one that states none (moq-transport, moq-lite before 07) is bounded by its max
// alone, as is a reader asking for history.
const LIVE = [{ start: { relative: 0n } }];
for (const [name, subscription] of [
["a live join's min delay", { delay: { min: Duration.millis(130), max: Duration.millis(10_000) }, ranges: LIVE }],
[
"a live join's max delay when it states no min",
{ delay: { min: null, max: Duration.millis(130) }, ranges: LIVE },
],
[
"a live join's max delay when that is shorter than its min",
{ delay: { min: Duration.millis(10_000), max: Duration.millis(130) }, ranges: LIVE },
],
["a history reader's max delay", { delay: { max: Duration.millis(130) } }],
] as const) {
test(`demand replays only the retained audio ${name} still wants`, async () => {
using _webcodecs = installFakeWebCodecs();
const clock = new Manual();
const installed = clock.install();
const broadcast = new Broadcast({ origin: new Moq.Origin.Producer(), name: Moq.Path.from("rejoin.hang") });
const feed = new Feed();
const configuring = configured();
const encoder = new Encoder("audio", { broadcast, capture: fakeCapture(feed) as never });
let late: Moq.Track.Subscriber | undefined;
try {
await configuring;
await settle();
for (let index = 0; index < 150; index++) {
await feed.push({ timestamp: Duration.micros(index * 20_000), channels: [new Float32Array(960)] });
await feed.drain();
await clock.advance(Duration.millis(20));
}
const net = broadcast.net.peek();
if (!net) throw new Error("expected network broadcast");
// At 3000 ms, a 130 ms budget still wants the frames ending after 2870 ms: 143 onward.
late = net.track("audio").subscribe(subscription);
await settle();
expect(encoder.out.stats.peek().frames).toBe(7 - FakeAudioEncoder.LAG);
await feed.push({ timestamp: Duration.micros(150 * 20_000), channels: [new Float32Array(960)] });
await feed.drain();
await clock.advance(Duration.millis(20));
const group = await late.ordered().nextGroup();
const frame = await group?.readFrame();
if (!frame) throw new Error("expected a replayed frame");
expect(Moq.Varint.decode(frame.payload)[0]).toBe(143 * 20_000);
} finally {
late?.close();
encoder.close();
broadcast.close();
installed.close();
}
const net = broadcast.net.peek();
if (!net) throw new Error("expected network broadcast");
// At 3000 ms, a 130 ms budget still wants the frames ending after 2870 ms: 143 onward.
late = net.track("audio").subscribe({ delay: { max: Duration.millis(130) } });
await settle();
expect(encoder.out.stats.peek().frames).toBe(7 - FakeAudioEncoder.LAG);
await feed.push({ timestamp: Duration.micros(150 * 20_000), channels: [new Float32Array(960)] });
await feed.drain();
const group = await late.ordered().nextGroup();
const frame = await group?.readFrame();
if (!frame) throw new Error("expected a replayed frame");
expect(Moq.Varint.decode(frame.payload)[0]).toBe(143 * 20_000);
} finally {
late?.close();
encoder.close();
broadcast.close();
installed.close();
}
});
});
}

// A fetch's own budget judges only the group it asks for, so it must not narrow what the encoder
// replays for the live subscribers that come later.
Expand Down
23 changes: 18 additions & 5 deletions js/publish/src/audio/encoder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -220,9 +220,9 @@ export class Encoder {
// discontinuity and re-anchors on.
#pipeline: Pipeline | undefined;
#encoding = false;
// The demand's max delay, past which retained audio is stale to every subscriber, or undefined
// for the retention alone. Replayed groups land as fresh arrivals, so only this cut keeps a
// rejoining live viewer from receiving all the retained audio before the live edge.
// How far back retained audio is replayed (see replayBudget): a live join's min delay, else the max
// delay, and for fetches alone the retention. Replayed groups land as fresh arrivals, so only this
// cut keeps a rejoining live viewer from receiving all the retained audio before the live edge.
#budget: Duration | undefined;

// The rendition's track and producer, kept across subscriber changes.
Expand Down Expand Up @@ -489,8 +489,7 @@ export class Encoder {
const onDemand = effect.get(broadcast.in.onDemand) !== "off";
const demanded = !!track && effect.get(track.demand().wanted) !== undefined;
// Demand from fetches alone carries no subscription, so no budget beyond the retention.
const max = track && effect.get(track.subscription)?.delay?.max;
this.#budget = max === undefined || max === null ? undefined : max;
this.#budget = replayBudget(track && effect.get(track.subscription));
// Not gated on `enabled` or `muted`: a paused rendition takes no capture, but still encodes what it retained.
this.#encoding = !!format && !!track && !fatal && (!onDemand || demanded);
effect.set(this.#out.active, enabled && this.#encoding, false);
Expand Down Expand Up @@ -1030,6 +1029,20 @@ export function resolve(captured: Format, selected: Codec): Resolved {
};
}

// How far back a subscription wants retained audio replayed. A live join starts its min delay behind
// the live edge, which a player states as its playout delay plus the codec's warmup, so that bounds
// it within the max delay. Any other range, or a live join stating no min (moq-transport, moq-lite
// before 07), wants all its max delay keeps. Undefined with no subscription, leaving the retention.
function replayBudget(subscription: Moq.Track.Subscription | undefined): Duration | undefined {
const delay = subscription?.delay;
if (!delay) return undefined;
const max = delay.max ?? undefined;
const ranges = subscription.ranges ?? [{}];
const live = ranges.length > 0 && ranges.every((range) => range.start?.relative === 0n && range.end === undefined);
if (!live || delay.min === null || delay.min === undefined) return max;
return max === undefined ? delay.min : delay.min.min(max);
}

/**
* How far apart audio groups start: a frame opens one once it is at least `join` past the
* open group's start, so groups are a whole number of frames apart, and at least one.
Expand Down
34 changes: 34 additions & 0 deletions js/watch/src/audio/buffer.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -668,6 +668,40 @@ describe("AudioBuffer steering", () => {
}
});

// A rejoin's replayed backlog trails the clock by seconds, and the ring skips it unheard. Held to the
// ring's pace, the live edge behind it would arrive only once the ring had drained it all.
it("releases at once, and reports stale, what trails the clock past its bound short of a declared end", () => {
const clock = started();
const { buffer, state } = fake(100, {
bound: 200,
clock: { position: () => 3000, out: { reference: new Signal<number | undefined>(0) } },
});
const signal = new AbortController().signal;
try {
buffer.insert(ms(2900), [new Float32Array(160)]);
state(ms(2900), false, 0);
// Within the bound the ring plays it, so the gate holds what lies past its floor and window.
expect(buffer.stale(ms(2850))).toBe(false);
expect(Bun.peek.status(buffer.wait(ms(3300), signal))).toBe("pending");
expect(buffer.stale(ms(2700))).toBe(true);
expect(Bun.peek.status(buffer.wait(ms(1000), signal))).toBe("fulfilled");

// A finite tail plays out however far it trails the clock, through the next utterance too.
buffer.end(ms(2800));
expect(buffer.stale(ms(2700))).toBe(false);
// The decoder asks with where its output ends, which for the last packet is the end itself.
expect(buffer.stale(ms(2800))).toBe(false);
buffer.next();
expect(buffer.stale(ms(2700))).toBe(false);
expect(buffer.stale(ms(2790))).toBe(false);
buffer.reset();
expect(buffer.stale(ms(2700))).toBe(true);
} finally {
buffer.close();
clock.close();
}
});

// Before media zero's turn, as while an opening waits for its output to sound it, the clock pairs
// at its negative position rather than the zero it reads as.
it("pairs a clock short of media zero at its negative position", () => {
Expand Down
35 changes: 35 additions & 0 deletions js/watch/src/audio/buffer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,15 @@ export class Backpressure {
});
}

/**
* Release a frame at once whose output the ring would skip unheard, so a backlog behind the clock
* decodes at the codec's pace, only to warm it, rather than at the pace the ring drains. It counts
* as decoding until its output answers it.
*/
pass(timestamp: number): void {
this.#decoding.push(timestamp);
}

/** The ring was flushed for a new timeline: its playhead is unknown until the next frame anchors it. */
reset(): void {
this.#playhead = undefined;
Expand Down Expand Up @@ -312,6 +321,9 @@ export class AudioBuffer {
#backpressure: Backpressure;
// Whether an end was declared since the last reset, which the next audio follows.
#declared = false;
// Where the latest declared end stands, until a reset that is not the next utterance. Audio before
// it is a finite tail, which the ring plays out however far it trails the clock.
#tail?: number;
// Bumped per reset. A state message posted before the worklet saw it reports the ring on the old
// timeline, which would open the gate.
#epoch = 0;
Expand Down Expand Up @@ -563,6 +575,7 @@ export class AudioBuffer {
*/
truncate(timestamp: number): void {
this.#backpressure.truncated(timestamp);
if (this.#tail !== undefined) this.#tail = Math.min(this.#tail, timestamp);
const msg: Truncate = { type: "truncate", timestamp };
this.#worklet.port.postMessage(msg);
if (this.#declared) this.next();
Expand All @@ -575,6 +588,7 @@ export class AudioBuffer {
*/
end(timestamp: number): void {
this.#declared = true;
this.#tail = timestamp;
const msg: End = { type: "end", timestamp };
this.#worklet.port.postMessage(msg);
}
Expand All @@ -595,6 +609,7 @@ export class AudioBuffer {

#reset(next: boolean): void {
this.#declared = false;
if (!next) this.#tail = undefined;
this.#epoch++;
const msg: Reset = {
type: "reset",
Expand All @@ -614,10 +629,30 @@ export class AudioBuffer {
* whose whole target fits the floor. Resolves early, releasing the wait, once `signal` aborts.
*/
wait(timestamp: number, signal: AbortSignal): Promise<void> {
// An aborted loop already dropped what it was decoding, so it must not count another frame.
if (signal.aborted) return Promise.resolve();
if (this.stale(timestamp)) {
this.#backpressure.pass(timestamp);
return Promise.resolve();
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
}
// The worklet-reported playhead lags by a state-message interval, which only holds frames longer.
return this.#backpressure.wait(timestamp, signal);
}

/**
* Whether media at `timestamp`, in microseconds, trails the shared clock past the bound, so the ring
* would skip it unheard, as a publisher's replayed backlog does on a rejoin. The decoder still
* decodes it to warm the codec, but writes none of it: a ring anchored on it would play catch-up
* instead of the live edge. Audio up to a declared end is never stale, since the ring plays a
* finite tail out rather than skip it.
*/
stale(timestamp: number): boolean {
if (this.#tail !== undefined && timestamp <= this.#tail) return false;
const position = this.#sync?.position();
if (position === undefined) return false;
return position * 1000 - timestamp > Math.max(this.#bound.asMicros(), WINDOW);
Comment thread
kixelated marked this conversation as resolved.
Resolved
}

/** Stop observing the worklet and release every pending wait. */
close(): void {
this.#backpressure.close(); // never leave a decode loop awaiting a closed buffer
Expand Down
35 changes: 34 additions & 1 deletion js/watch/src/audio/decoder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1031,7 +1031,9 @@ describe("Decoder across a rendition's absence", () => {
expect(playback.stalled()).toBe(false);
expect(playback.ended()).toBe(false);

await clock.advance(Duration.millis(1000));
// Past the end, but not so far that the replacement continuing the timeline trails the clock
// past the stretch bound, which a live replacement never does.
await clock.advance(Duration.millis(300));
await microtasks();
expect(playback.ended()).toBe(true);
expect(playback.stats()).toMatchObject({ underruns: 0, stalls: 0 });
Expand Down Expand Up @@ -1164,6 +1166,37 @@ describe("Decoder across a rendition's absence", () => {
}
});

// A publisher replays the audio it captured while nobody listened, so a viewer falling back to it
// receives seconds of it behind the picture, in order over a WebSocket. Written to the ring, it
// anchored playback there, and the ring skipped and re-stalled on each refill until it ran out.
it("decodes a backlog behind the clock to warm the codec but plays from the live edge", async () => {
const playback = await play(Duration.millis(50));
const video = playback.sync.track({ target: new Signal<Duration | undefined>(undefined) });
try {
// The picture anchors the clock at the live edge, 3 s in.
video.received(Duration.secs(3), now());
const stale = [];
for (let ms = 1000; ms < 1400; ms += 20) {
frameTimestamp = (ms - 20) * 1000;
playback.push({ ...frame(), group: BigInt(ms / 20) });
stale.push(ms * 1000);
await microtasks();
}
for (let ms = 3000; ms < 3200; ms += 20) {
frameTimestamp = (ms - 20) * 1000;
playback.push({ ...frame(), group: BigInt(ms / 20) });
await microtasks();
}
expect(audioInputs.slice(0, stale.length)).toEqual(stale);
const inserted = playback.inserted();
expect(inserted.length).toBeGreaterThan(0);
expect(Math.min(...inserted)).toBeGreaterThanOrEqual(3_000_000);
} finally {
video.close();
playback.close();
}
});

for (const [name, patch] of Object.entries({
container: { container: { kind: "loc" } },
codec: { codec: "mp4a.40.2" },
Expand Down
7 changes: 7 additions & 0 deletions js/watch/src/audio/decoder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1099,6 +1099,13 @@ export class Decoder {
const start = Duration.micros(timestamp);
const end = Duration.micros(timestamp + durationMicro);

// Audio the ring would skip anyway only warmed the codec. Written, it would anchor the ring behind
// the clock, which then skips and re-stalls on each refill until the backlog runs out.
if (ring.stale(end.asMicros())) {
Comment thread
kixelated marked this conversation as resolved.
Resolved
sample.close();
return;
}

// A new subscription has taken over the timeline: drop the previous one's write-ahead tail
// rather than letting it play out after this frame. See #runDecoder.
const takeover = this.#handover.takeover();
Expand Down
10 changes: 6 additions & 4 deletions quest/m0/interop-green/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,10 +224,12 @@ Scheduled runs
stalls past the 1039 ms bound after the route falls back. Reproduced
at 1 of 8 and 2 of 12 loaded local runs (the audio ring still stalled 600 ms after the first
audio group of the returned incumbent). Decided 2026-11-15 (AFK, recommendation, for review):
`replacement` is quarantined by default on every engine again, and the child
[replacement audio](/quest/m0/interop-green/replacement-audio.md) root-causes and restores it.
Its first PR found the stall: the decode gate released the backlog behind the clock one window
per worklet report. Its second restored the case on Chromium; Firefox's quarantine is in the child.
`replacement` is quarantined by default on every engine again, and a child quest root-caused
and restored it. Its first PR found the stall: the decode gate released the backlog behind the
clock one window per worklet report. Its second restored the case on Chromium, and
[moq-dev/moq#9023](https://github.com/moq-dev/moq/pull/9023) on Firefox: the player decodes the
audio an incumbent replays behind the clock only to warm the codec. Off Chromium the moq-transport-22 row stays quarantined, since that
wire states no delay to cut the replay at.

Public API: none. Wire: none.

Loading