Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions doc/bin/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ MPEG-TS export is covered in the [MPEG-TS guide](/bin/ts#export).
```bash
moq --connect https://relay.example.com/anon --broadcast my-stream.hang play
moq ... play --delay 500ms # fix the delay instead of measuring it
moq ... play --stretch 200ms # let audio trail the clock by at most 200 ms
moq ... play --stretch 500ms # let audio trail the clock by up to 500 ms before it skips
moq ... play --video off # audio only, without a window
moq ... play --video off --metrics play.ndjson # log what the speaker played
```
Expand All @@ -192,9 +192,9 @@ default, `auto`, sizes the buffer to how unevenly audio arrives, with the same
delay instead. Audio and video play from one clock and stay in sync while both
arrive on time. After a stall, video shows each late frame as it arrives, while
audio slows down and falls behind the clock rather than dropping what arrives
late, then speeds up until it is back on it. `--stretch` (default 2 s) bounds
how far audio may trail before it skips ahead, and a stalled group is waited on
for the delay plus the stretch before it is skipped. `--stretch 0` plays
late, then speeds up until it is back on it. Audio that trails by more than
`--stretch` (default 200 ms) skips onto the clock, as video does, and audio
that leads it by as much waits in silence. `--stretch 0` plays
received audio unaltered: late audio is skipped, an underrun is silence, and
drift against the publisher's clock is corrected by a small skip or pad.
Each role follows the catalog, switching rendition when the publisher retires
Expand Down
40 changes: 20 additions & 20 deletions doc/concept/audio-jitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ no frame, is waited for only the playout delay plus any lookahead, since past th
it would play late: a reordered backlog's holes waited out to the whole budget
would delay the frames behind them, which the estimator would measure as
jitter. A caller that estimates keeps the budget past the target, which is why
`moq play` waits on a stalled group for the delay plus its `--stretch`, 2 s by
default. The observation is also only as honest as
`moq play` waits on a stalled audio group for the delay plus the estimate's 2 s
ceiling. The observation is also only as honest as
the caller is prompt: `read()` is timed when it resolves, so a player drains it
as packets come and lets the playback buffer hold the target, rather than
pacing reads to the speaker.
Expand All @@ -59,15 +59,12 @@ before any container sees them, at the relay and in the receiver's own transport
subscriber, so a subscription cut to the target hides every frame later than
the target: the estimate creeps up one bucket at a time while the frames it
should have measured go unplayed. Both implementations subscribe with the delay
plus the player's `stretch` setting: `moq play` through its age budget, and the
browser through the audio subscription alone, since its container consumer
observes each frame before applying the local budget. The 2 s default reaches
the estimate's own ceiling, so any lateness the estimator can measure is
observed at once. A smaller stretch hides frames later than the target plus the
stretch, so a jump in lateness larger than the stretch is learned in steps of
about the stretch; a zero stretch cuts a live subscription to the target and
accepts the one-bucket creep. A buffered browser player keeps its subscription
at least the delay plus its lookahead, whatever the stretch.
plus the estimate's own 2 s ceiling, whatever the player's `stretch` setting:
`moq play` through its age budget, and the browser through the audio
subscription alone, since its container consumer observes each frame before
applying the local budget. So any lateness the estimator can measure is
observed at once. A buffered browser player keeps its subscription at least the
delay plus its lookahead.
Browser video keeps the shared budget, because a longer one fetches whole stale
groups, so a broadcast with no audio measures only the lateness that budget
lets through.
Expand Down Expand Up @@ -681,10 +678,10 @@ At each window boundary a player decides on five inputs:
clock from speeding into it.
- `lag`: how far the playhead trails the shared clock, negative when it plays
ahead of it.
- `bound`: how far audio may trail the clock, which is the subscription's max
delay less the playout delay: the player's `stretch` setting (2 s by
default), or for a buffered player the larger of it and the lookahead. A zero
stretch turns the stretch off; see [below](#skips-underruns-and-silence).
- `bound`: how far audio may trail the clock: the player's `stretch` setting
(200 ms by default), or for a buffered player the larger of it and the
lookahead. It sizes nothing else. A zero stretch turns the stretch off; see
[below](#skips-underruns-and-silence).

`level` passes through NetEq's `BufferLevelFilter` once per window: three of its
10 ms steps at the factor the target selects, 251, 252, 253, or 254 in 256ths
Expand Down Expand Up @@ -724,18 +721,21 @@ playhead ahead of the clock back onto it. These are NetEq's thresholds and
rates, untuned. A moved target moves the clock, so a rise leaves audio ahead of
it and a fall leaves audio behind, and both become ordinary slowing down and
speeding up: nothing parks in silence on a rise, and nothing skips buffered
audio on a fall shorter than the bound and one window. A fall past that skips
to the bound, as any lag past it does.
audio on a fall, within the bound. Past it, a fall skips onto the clock and a
rise pads silence until the clock arrives, as any lag past it does.

Without a clock reading the target is where the playhead belongs, so
`lag = level - target`. The browser then never skips; native playback bounds
the lag at three quarters of its ring's capacity above the target.

### Skips, underruns, and silence

A playhead that trails the clock by more than `bound` and one window skips
ahead to `bound` behind it, dropping the staged window and the audio it passes.
Less than that is the stretch's to catch up. A join is the exception: until
A playhead that trails the clock by more than `bound`, or one window when that
is larger, skips onto the clock, dropping the staged window and the audio it
passes, as video skips the frames a stall made late. One that leads the clock
by as much, as a target rise past the bound leaves it, plays silence until the
clock reaches it, dropping nothing. Less than that either way is the stretch's
to catch up. A join is the exception: until
the player has read once on the clock, its bound is zero, so the backlog a
subscription opens on is skipped rather than played out at the stretch's pace,
and playback starts on the clock. A shared clock whose reference moves earlier,
Expand Down
11 changes: 5 additions & 6 deletions doc/lib/js/watch.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ in sync at the latency you ask for.
| `paused`, `muted`, `volume` | The usual player controls. |
| `delay` | How far playback trails the live edge: `"auto"` (the default, sized from how late frames arrive, at most 10 s; see [audio jitter](/concept/audio-jitter)), a duration like `"300ms"`, or `"instant"` to paint video as it decodes, with no audio. |
| `lookahead` | How far ahead of the live edge media may run; see [buffered playback](#buffered-playback). |
| `stretch` | How far audio may trail the clock before it skips ahead: a duration like `"2s"` (the default), or `"0"` to turn the time stretch off; see [catching up](#catching-up). |
| `stretch` | How far audio may trail the clock before it skips onto it: a duration like `"200ms"` (the default), or `"0"` to turn the time stretch off; see [catching up](#catching-up). |
| `captions` | The caption track to show. |
| `visible` | Only download video while the element is on screen (or near it). |
| `backend` | How the canvas is drawn; see [@moq/video](/lib/js/video). `auto` (default), `webgpu`, or `2d`. |
Expand Down Expand Up @@ -224,11 +224,10 @@ leave the connection's estimate unclaimed. Without a claim, video selects the be
After a stall, video shows each late frame as it arrives and is live again at
once. Audio cannot skip without losing what arrives late, so it slows down
slightly when the audio received ahead of it runs low, falls behind the clock,
and speeds back up once it recovers until it is back on it. `stretch` bounds
how far it may trail: only audio later than that is skipped, and the audio
subscription waits on a stalled group for `delay + stretch`, so the late audio
still arrives. A voice app might ask for `stretch="200ms"`, a sports stream for
the default 2 s, whether `delay` is `auto` or fixed.
and speeds back up once it recovers until it is back on it. Audio that trails
by more than `stretch`, 200 ms by default, skips onto the clock as video does,
so a long stall costs one skip instead of seconds of lag, and audio that leads
it by as much, as when the delay rises, waits for it in silence.

`stretch="0"` turns the time stretch off, for content where a pitch-period
splice is audible, such as music: received audio plays unaltered, late audio
Expand Down
2 changes: 1 addition & 1 deletion js/watch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ The simplest way to watch a stream:
| `follow` | boolean | true | Follow the broadcast's announcements: wait for one before subscribing, and move to each restart. Ignored when the relay does not support broadcast discovery. |
| `delay` | `auto`, duration, `instant` | `auto` | Distance from the live edge. `instant` paints frames as they decode and disables audio. |
| `lookahead` | duration | `0ms` | How far future-dated media may run ahead of the live edge. Not a jitter buffer: live streams tune `delay`. |
| `stretch` | duration | `2000ms` | How far audio may trail the clock after a stall before it skips ahead. `0` turns the time stretch off, so received audio plays unaltered. |
| `stretch` | duration | `200ms` | How far audio may trail the clock after a stall before it skips onto it. `0` turns the time stretch off, so received audio plays unaltered. |
| `captions` | string | off | Text rendition to render. |
| `catalog-format` | hang, msf, manual | auto-detected | The catalog format; detected from the name suffix unless set. |
| `echo` | path segment | - | The viewer's feedback name: once the catalog solicits feedback, publish `<echo path>/<name>.echo` reporting what each rendition received and played. |
Expand Down
36 changes: 30 additions & 6 deletions js/watch/src/audio/decoder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1774,21 +1774,45 @@ it.each([false, true])("releases shared startup across a hole in queued audio (C
});

// Audio may trail the clock by the stretch, or a buffered player's whole lookahead when that is longer,
// and a zero stretch turns the time stretch off in either mode.
// and a zero stretch turns the time stretch off in either mode. The stretch sizes nothing else: the
// subscription reaches the estimator's 2 s ceiling past the delay whatever it is.
for (const buffered of [false, true]) {
it(`bounds ${buffered ? "buffered" : "live"} audio's lag by the stretch`, async () => {
it(`bounds ${buffered ? "buffered" : "live"} audio's lag by the stretch alone`, async () => {
// The max delay each subscribe and update asked the publisher for, in order.
const asked: (Duration | null | undefined)[] = [];
const subscribe = Moq.Track.Consumer.prototype.subscribe;
const update = Moq.Track.Subscriber.prototype.update;
const subscribes = spyOn(Moq.Track.Consumer.prototype, "subscribe").mockImplementation(function (
this: Moq.Track.Consumer,
...args: Parameters<typeof subscribe>
) {
asked.push(args[0]?.delay?.max);
return subscribe.apply(this, args);
});
const updates = spyOn(Moq.Track.Subscriber.prototype, "update").mockImplementation(function (
this: Moq.Track.Subscriber,
...args: Parameters<typeof update>
) {
asked.push(args[0]?.delay?.max);
return update.apply(this, args);
});
const playback = await play(Duration.millis(100), buffered);
try {
await playback.play(12);
const bound = () => worklets.at(-1)?.terms?.bound;
expect(bound()).toBe(2000);
playback.stretch.set(Duration.millis(300));
expect(bound()).toBe(buffered ? 1000 : 200);
const reach = Duration.millis(2100);
expect(asked.at(-1)).toEqual(reach);
playback.stretch.set(Duration.millis(1500));
await microtasks();
expect(bound()).toBe(buffered ? 1000 : 300);
expect(bound()).toBe(1500);
playback.stretch.set(Duration.ZERO);
await microtasks();
expect(bound()).toBe(0);
expect(asked.at(-1)).toEqual(reach);
} finally {
subscribes.mockRestore();
updates.mockRestore();
playback.close();
await microtasks();
}
Expand Down Expand Up @@ -1816,7 +1840,7 @@ for (const cmaf of [false, true]) {

// Either ring keeps what it receives and plays onto the clock, which audio may trail by the
// larger of the lookahead and the default stretch, even at a fixed delay.
expect(worklets.at(-1)?.terms).toEqual({ latency: 100, target: 100, bound: 2000 });
expect(worklets.at(-1)?.terms).toEqual({ latency: 100, target: 100, bound: buffered ? 1000 : 200 });
} finally {
playback.close();
await microtasks();
Expand Down
12 changes: 6 additions & 6 deletions js/watch/src/audio/decoder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ export class Decoder {
// The container consumer's arrival estimate, unset while nothing is subscribed.
#measured = new Signal<Duration | undefined>(undefined);

// The subscription's max delay: the shared budget, raised to the delay plus the stretch.
// The subscription's max delay: the shared budget, raised to the delay plus the estimator's ceiling.
#subscribeMaxDelay = new Signal<Duration>(Duration.ZERO);
// The container consumer's delay: the playout delay to start at and the shared budget. Its warmup
// reaches the start further back, to the media the decoder primes on.
Expand Down Expand Up @@ -321,9 +321,9 @@ export class Decoder {
});
}

// The subscription reaches the stretch past the delay, so audio the ring could still catch up on
// arrives. A group the relay expires is never observed, so a zero stretch, which cuts the
// subscription to the target, leaves the estimate rising one bucket at a time. The container
// The subscription reaches the estimator's 2 s ceiling past the delay, whatever the stretch, since a
// group the relay expires is never observed: a subscription cut to the target would leave the
// estimate rising one bucket at a time. The container
// consumer drops aged media at the same max delay, since it observes each frame before applying
// it, and carves out the warmup the subscription adds once it has warmed, as Rust's does. It waits
// on a missing group only for the delay plus the lookahead: past that the group would play late,
Expand All @@ -338,7 +338,7 @@ export class Decoder {
this.#subscribeMaxDelay.set(max.add(warmup));
}

// The terms of the subscription's max delay: the shared delay and max delay, and the stretch.
// The shared delay and max delay, and the stretch threshold the ring skips past.
#budget(effect: Effect): { delay: Duration; maxDelay: Duration; stretch: Duration } {
return {
delay: effect.get(this.sync.out.delay),
Expand Down Expand Up @@ -511,7 +511,7 @@ export class Decoder {
if (!ring) return;

// The ring plays onto the shared clock, slowing down onto a rise and speeding up onto a fall,
// and skips only audio trailing it past what the subscription still delivers.
// and skips onto it only audio trailing it past the stretch threshold.
const budget = this.#budget(effect);
ring.setLatency(budget.delay, bound(budget));
}
Expand Down
20 changes: 13 additions & 7 deletions js/watch/src/audio/latency.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,20 +26,26 @@ export function target(props: Target): Duration {
return floored.add(props.frame ?? Duration.ZERO).add(props.delay ?? Duration.ZERO);
}

// The arrival estimate's ceiling, `BUCKET * BUCKETS` in @moq/hang's jitter estimator: no lateness past
// it moves the target, so a subscription reaching that far past the delay observes all it can measure.
const REACH = Duration.secs(2);

/**
* The audio subscription's max delay, less warmup: the shared `maxDelay`, raised to `delay + stretch`
* so audio trailing the clock by up to the stretch still arrives for the ring to catch up on.
* The audio subscription's max delay, less warmup: the shared `maxDelay`, raised to the delay plus the
* estimator's 2 s ceiling, so every frame late enough to move the target is still observed. The
* `stretch` setting does not size it.
*/
export function maxDelay(props: { delay: Duration; maxDelay: Duration; stretch: Duration }): Duration {
return props.maxDelay.max(props.delay.add(props.stretch));
export function maxDelay(props: { delay: Duration; maxDelay: Duration }): Duration {
return props.maxDelay.max(props.delay.add(REACH));
}

/**
* How far the ring lets audio trail the clock before it skips ahead: the audio max delay less the
* playout delay, or zero, which turns the time stretch off, when `stretch` is zero.
* How far the ring lets audio trail the clock before it skips onto it: the `stretch` threshold, or
* for a buffered player the larger of it and the lookahead (`maxDelay` less the delay). Zero, which
* turns the time stretch off, when `stretch` is zero.
*/
export function bound(props: { delay: Duration; maxDelay: Duration; stretch: Duration }): Duration {
return props.stretch.isZero() ? Duration.ZERO : maxDelay(props).saturatingSub(props.delay);
return props.stretch.isZero() ? Duration.ZERO : props.stretch.max(props.maxDelay.saturatingSub(props.delay));
}

// An AudioWorkletProcessor renders in fixed 128-sample quanta, so a ring shallower than one can
Expand Down
8 changes: 4 additions & 4 deletions js/watch/src/audio/playout.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -127,12 +127,12 @@ async function measure(t: Arrival[], budget: (target: Duration) => Duration): Pr
return result;
}

/** The subscription "auto" asks for at the default stretch: the target plus the stretch. */
const stretched = (target: Duration) => maxDelay({ delay: target, maxDelay: target, stretch: STRETCH });
/** The subscription "auto" asks for: the target plus the estimator's 2 s ceiling. */
const reached = (target: Duration) => maxDelay({ delay: target, maxDelay: target });

/** The target the estimator settles on by the end of `t`, given the subscription "auto" asks for. */
async function settled(t: Arrival[]): Promise<number> {
const { targets } = await measure(t, stretched);
const { targets } = await measure(t, reached);
return targets[targets.length - 1];
}

Expand Down Expand Up @@ -244,7 +244,7 @@ describe("subscription budget", () => {
}

it("reaches a flush that starts after the target settled", async () => {
const { targets, delivered } = await measure(t, stretched);
const { targets, delivered } = await measure(t, reached);
expect(targets[PACED - 1]).toBe(2 * CHUNK_MS);
expect(delivered).toBe(t.length);
// Every flush lands whole, so its 100ms reaches the 95th percentile within a few intervals.
Expand Down
2 changes: 1 addition & 1 deletion js/watch/src/audio/replay.ts
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
const track = new Track.Producer("audio");
// The subscription's max delay, as the decoder sets it, which the consumer drops aged media at too.
const maxDelay = new Derived([sync.out.delay, sync.out.maxDelay], (delay, max) =>
audioMaxDelay({ delay, maxDelay: max, stretch }),
audioMaxDelay({ delay, maxDelay: max }),
);
const consumer = new Container.Consumer(track.subscribe({ delay: { max: WIRE_MAX_DELAY } }), {
format: new Container.Legacy.Format("audio"),
Expand Down
Loading