Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion doc/concept/audio-jitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,11 @@ age budget.
Natively that holds only as far as the caller lets it. The container consumer
reads groups in sequence order and applies its age budget inside `read()`, so a
group skipped for age is never observed, and a frame queued behind a stalled
group is observed when the stall clears. A caller that estimates keeps the budget
group is observed when the stall clears. A missing group, one that has delivered
no frame, is waited for only the playout delay plus any lookahead, since past that
it would play late: a reordered backlog's holes waited out to the whole budget
would delay the frames behind them, which the estimator would measure as
jitter. A caller that estimates keeps the budget
at least as long as the estimate's ceiling, which is why `moq play` waits 2 s on
a stalled group under `--delay auto`. The observation is also only as honest as
the caller is prompt: `read()` is timed when it resolves, so a player drains it
Expand Down
1 change: 1 addition & 0 deletions js/hang/api/container.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ export interface ConsumerProps {
delay?: Signals.GetterInit<Net.Track.Delay>;
format: Format;
predecessor?: Consumer;
wait?: Signals.GetterInit<Net.Clock.Duration | undefined>;
warmup?: Net.Clock.Duration;
}

Expand Down
64 changes: 64 additions & 0 deletions js/hang/src/container/consumer.holes.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -394,3 +394,67 @@ test("a newest-first flush past a hole delivers the groups landing behind its ne
}
}
});

// A player waits on a hole for its playout delay, while groups that did arrive play until the max.
test("a hole is waited for the wait while an arrived group plays up to the max delay", async () => {
const clock = new Manual();
const installed = clock.install();
const track = new Track.Producer("audio");
const consumer = new Consumer(track.subscribe({ delay: { max: Duration.millis(30_000) } }), {
format: new Format("audio"),
delay: { max: Duration.millis(1000) },
wait: Duration.millis(100),
});
const results: Read[] = [];
const reading = (async () => {
for (;;) {
const read = await consumer.next();
if (!read) return;
results.push(read);
}
})();
const write = (sequence: bigint, ms: number, finish = true) => {
const group = track.createGroup(sequence);
group.writeFrame({
payload: encodeFrame(new Uint8Array([1]), Duration.millis(ms)),
timestamp: Time.Timestamp.fromMicros(ms * 1000),
});
if (finish) group.finish();
return group;
};
const reads = () => results.map((read) => (read.kind === "frame" ? read.frame.timestamp.asMillis() : read.kind));
try {
// A stalled group 500 ms behind the newest content is still within the max delay.
const stalled = write(0n, 0, false);
write(1n, 500);
await flush();
await clock.advance(Duration.millis(500));
expect(reads()).toEqual([0]);
stalled.finish();
await flush();
expect(reads()).toEqual([0, 500]);

// A missing group expires the wait after the first group past it arrived.
write(3n, 520);
await flush();
await clock.advance(Duration.millis(99));
expect(reads()).toEqual([0, 500]);
await clock.advance(Duration.millis(1));
expect(reads()).toEqual([0, 500, "gap", 520]);

// Or at once, when the newest content is already the wait past where the group after it begins.
write(5n, 700);
write(6n, 800);
await flush();
expect(reads()).toEqual([0, 500, "gap", 520, "gap", 700, 800]);
} finally {
consumer.close();
track.finish();
track.close();
try {
await reading;
} finally {
installed.close();
}
}
});
36 changes: 27 additions & 9 deletions js/hang/src/container/consumer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@ export interface ConsumerProps {
* for it when the newest group arrives first; an unstated (`null`) `min` reads as zero. A `min`
* past `max` starts at `max`, since the publisher serves nothing older. `max` is measured as
* the span from the oldest buffered frame to the newest, so it bounds how long a late group is
* waited for. A missing sequence also expires this much monotonic time after the first group
* past it arrived, even when the next group is a marker on a paused track. It is the local
* waited for, and a missing one unless {@link ConsumerProps.wait} is shorter. It is the local
* half of the subscription's `delay.max`; both measure the same budget, one on the wire and one
* as frames are read. It bounds a join's lateness too: a backlog that landed before the first
* read starts at the first group whose first frame is within it, plus any warmup, of the
Expand All @@ -26,6 +25,16 @@ export interface ConsumerProps {
*/
// Read-only: a Getter (e.g. another component's output) is accepted directly.
delay?: GetterInit<Moq.Track.Delay>;
/**
* How long a missing group is waited for before the consumer skips past it, at most `delay.max`
* (default: `delay.max`). It expires once where the first group past it begins falls this far
* behind the newest content, or this much monotonic time after that group arrived, even when
* it is a marker on a paused track. Groups that did arrive still play until they age past
* `delay.max`, so a player can wait on holes for its playout delay alone while still receiving
* the late media it catches up on.
*/
// Read-only: a Getter (e.g. another component's output) is accepted directly.
wait?: GetterInit<Duration | undefined>;
/**
* Cap the bytes of decoded frames held ahead of the reader, counting the whole wire payload they
* share and each frame's and group's own overhead (default: one group at the transport's size
Expand Down Expand Up @@ -173,6 +182,7 @@ export class Consumer {
#track: Moq.Track.Subscriber;
#format: Format;
#delay: Getter<Moq.Track.Delay>;
#wait: Getter<Duration | undefined>;
#maxBuffer: number;
// How much further than the min delay the start reaches, and the age skip tolerates while warming.
#warmup: Duration;
Expand Down Expand Up @@ -247,6 +257,7 @@ export class Consumer {
this.#track = track;
this.#format = props.format;
this.#delay = getter(props.delay ?? {});
this.#wait = getter(props.wait);
this.#maxBuffer = props.maxBuffer ?? DEFAULT_MAX_BUFFER;
this.#warmup = props.warmup ?? Duration.ZERO;
this.#jitter = props.predecessor ? props.predecessor.#jitter.fork() : new Jitter();
Expand Down Expand Up @@ -838,8 +849,8 @@ export class Consumer {
// Otherwise promote #active when the first buffered group continues the timeline we left off at,
// when a completed empty group can be walked (empty groups mean nothing), or when the hole
// is proven: everything a missing group could still present ends where the first group past
// it begins, so once that start falls a whole max delay behind the newest content, all of it
// would arrive too old to play. Mirrors Rust's `aged`. Where presentation left off proves only
// it begins, so once that start falls the whole wait behind the newest content, all of it
// would arrive too late to play. Mirrors Rust's `aged`. Where presentation left off proves only
// the oldest of it stale, and groups go out newest first, so a flush's newest group lands just
// ahead of the groups behind it, which would then arrive below the cursor and never play.
// Proving it here matters: #checkMaxDelay would instead drop the head, the very group to play
Expand All @@ -857,11 +868,11 @@ export class Consumer {
const startup = this.#start === undefined && !this.#floored;
const contiguous = ptsContiguous(this.#presentedEnd, head.frames.at(0)?.timestamp.asMicros());
const empty = head.empty && head.consumer.done;
const maxDelay = maxMicros(this.#delay.peek());
const wait = this.#holeWait(this.#delay.peek(), this.#wait.peek());
const bound = this.#groups.find((group) => group.start !== undefined)?.start;
const edge = this.#liveEdge();
const aged = bound !== undefined && edge !== undefined && edge - bound >= maxDelay;
const elapsed = now().since(this.#hole).asMicros() >= maxDelay;
const aged = bound !== undefined && edge !== undefined && edge - bound >= wait;
const elapsed = now().since(this.#hole).asMicros() >= wait;
// The ranges ask for nothing in between, so nothing there is owed.
const next = this.#track.nextInRange(this.#active);
const excluded = next === undefined || next >= head.consumer.sequence;
Expand Down Expand Up @@ -977,8 +988,9 @@ export class Consumer {
timer = new Effect();
timer.run((effect) => {
const delay = effect.get(this.#delay);
// The start waits out its reach, and a hole the max delay, which an unlimited one never does.
const budget = starting === undefined ? maxMicros(delay) : this.#reach(delay);
// The start waits out its reach, and a hole the wait, which an unlimited one never does.
const budget =
starting === undefined ? this.#holeWait(delay, effect.get(this.#wait)) : this.#reach(delay);
if (!Number.isFinite(budget)) return;
effect.timer(() => this.#notify?.(), Duration.micros(budget).saturatingSub(now().since(since)));
});
Expand All @@ -999,6 +1011,12 @@ export class Consumer {
}
}

// How long a missing group is waited for, in microseconds: the wait, at most the max delay.
#holeWait(delay: Moq.Track.Delay, wait: Duration | undefined): number {
const max = maxMicros(delay);
return wait === undefined ? max : Math.min(wait.asMicros(), max);
}

// How far behind the live edge the start reaches, in microseconds: the min delay plus the warmup.
// The publisher serves nothing older than the max delay, so a min delay past it reaches no further.
#reach(delay: Moq.Track.Delay): number {
Expand Down
9 changes: 6 additions & 3 deletions js/watch/src/audio/decoder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -322,9 +322,11 @@ export class Decoder {
}

// A group the relay expires is never observed, so a subscription cut to the target would cap the
// estimate at the target it already holds. The container consumer skips at the same max delay,
// since it observes each frame before applying it, so audio the ring could still catch up on is
// delivered, and carves out the warmup the subscription adds once it has warmed, as Rust's does.
// estimate at the target it already holds. The container consumer drops aged media at the same max
// delay, since it observes each frame before applying it, so audio the ring could still catch up on
// is delivered, and carves out the warmup the subscription adds once it has warmed, as Rust's does.
// It waits on a missing group only for the delay plus the lookahead: past that the group would play
// late, and a reordered backlog's holes waited out to the max would read as jitter to the estimator.
// See doc/concept/audio-jitter.md.
#runSubscribeMaxDelay(effect: Effect): void {
const delay = effect.get(this.sync.out.delay);
Expand Down Expand Up @@ -852,6 +854,7 @@ export class Decoder {
const consumer = new Container.Consumer(sub, {
format,
delay: this.#delay,
wait: this.sync.out.maxDelay,
warmup: Duration.millis(config.warmup ?? 0),
predecessor: continues ? outgoing.consumer : undefined,
});
Expand Down
7 changes: 4 additions & 3 deletions js/watch/src/audio/replay.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
* Replay a recorded arrival trace through the player's consumer and ring on a simulated clock.
*
* Every frame is written into its group at the instant the trace says it arrived, and the player's
* real `Container.Consumer` decides what to deliver, wait for, and skip, at the max delay the delay
* sets. Delivered frames go into the real ring, which carries on through a gap the way the decoder's
* real `Container.Consumer` decides what to deliver, wait for, and skip, at the hole wait and max delay
* the decoder sets. Delivered frames go into the real ring, which carries on through a gap the way the decoder's
* does: a gap restarts only the codec. One render quantum is read every quantum's worth of that same
* clock, which is what the AudioWorklet does. A real {@link Sync} resolves the delay from the playout target the decoder
* registers, measured by that same consumer from the recorded catalog config, and the ring and the
Expand Down Expand Up @@ -145,13 +145,14 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
const buffered = !lookahead.isZero();
const sync = new Sync({ delay: options.delay, lookahead });
const track = new Track.Producer("audio");
// The subscription's max delay, as the decoder sets it, which the consumer skips at too.
// The subscription's max delay, as the decoder sets it, which the consumer drops aged media at too.
const maxDelay = new Derived([sync.out.maxDelay], (max) =>
options.delay === "auto" ? max.max(AUTO_MAX_DELAY) : max,
);
const consumer = new Container.Consumer(track.subscribe({ delay: { max: WIRE_MAX_DELAY } }), {
format: new Container.Legacy.Format("audio"),
delay: new Derived([sync.out.delay, maxDelay], (min, max) => ({ min, max })),
wait: sync.out.maxDelay,
});

// Match the decoder: a codec constant until a delivered packet states its own duration.
Expand Down
29 changes: 0 additions & 29 deletions quest/m1/audio-jitter-target/hole-wait.md

This file was deleted.

5 changes: 5 additions & 0 deletions rs/moq-audio/src/decode/consumer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -572,6 +572,11 @@ impl Consumer {
/// that arrives meanwhile, or `None` once the track ends.
async fn next(&mut self) -> Result<Option<Encoded>, Error> {
loop {
// A missing group is waited for only the playout delay, since past it the group
// would play late, and a reordered backlog's holes waited out to the max delay
// would read as jitter. Late groups that did arrive still play up to the max.
// Refreshed before every poll, since each arrival and each decoded frame can move it.
self.track.set_wait(self.delay());
// Read before judging the head, so any update after that wakes the wait.
let seen = self.gate.as_ref().map_or(0, |gate| gate.read().version);
let deadline = match self.pending.front() {
Expand Down
Loading