Skip to content
Merged
36 changes: 30 additions & 6 deletions doc/concept/audio-jitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,9 @@ frame:

Both sit below the decoder and above nothing: no reordering, no gap handling, no
age budget. Except one skip: both consumers close a group below where delivery
stands unread, so a join's backlog they skip is never observed, and a join
delayed only by it is measured as lateness.
stands unread, so its frames are never observed. When it is
[backlog](#the-joins-backlog), they report its arrival instead, which ends the
join as late as its observed backlog would.

Natively that holds only as far as the caller lets it. The container consumer
reads groups in sequence order and applies its age budget inside `read()`, so a
Expand Down Expand Up @@ -125,8 +126,8 @@ observe(arrival, media):

if oldest + fastest < joined: # the join, not the path: folds nothing
pending += 1
else:
close(bucket of peak)
else: # open starts true
open = false; close(bucket of peak)
if resumed: # a declared silence folds nothing
settle(); origin = arrival; index = 0
else: # held until this interval closes
Expand All @@ -153,6 +154,10 @@ settle(): # a stall is one observation
if a stall is open:
fold(stall.bucket); fold(nothing) min(stall.span, IDLE_MAX) times; close it

backlog(arrival): # a backlog group skipped unread arrived
if not first observation and open:
joined = max(joined, arrival)

end(): # a declared end precedes the next observation
if not first observation: ended = true

Expand Down Expand Up @@ -324,6 +329,22 @@ is measured from its first flush past the backlog. A first frame slower than the
path after it is itself backlog, so most subscriptions skip their first interval,
which leaves the prior 500 ms later.

A group the consumer skips unread carries no timestamp to test, so the consumer
tells backlog by its sequence instead: one at or below the publisher's largest
group when the subscription began, the largest the subscriber knows once the
publisher answered (moq-lite's subscribe answer carries it, and without one the
groups that landed stand in), was produced before it. It is read as the answer
arrives, not when the first group does: any earlier, a cached group racing the
answer would set it, and any later, live groups landing meanwhile would. Only such a group's arrival is reported,
with `backlog`, so a path that drops late groups from the start, all produced
since, holds no join open. A report extends the join only while the join is
**open**, before any interval has measured the path: after that, a skipped group
is an outage's, whose frames queued behind it belong to the stall rule. On
`relay-mic-rtt40` the browser closes 267 groups 5 to 6 s old, arriving 1.0 to
1.3 s in, and the live frames queued behind them are up to 300 ms late; observed
or reported, they fold nothing, where unreported they hold the target at 320 ms
for a 20 s half-life.

### The histogram

An empirical distribution of `BUCKETS` buckets `BUCKET` wide, holding total
Expand Down Expand Up @@ -764,8 +785,10 @@ directly, so neither is graded against the other.
`arrival` and `media` are parallel arrays in **delivery order**, the order the
receiver observes them, so `media` is deliberately out of order in the
`reorder` and `backlog` cases. `advertised` and `delay` are present only where the case uses
them, as is `ends`: the indices, ascending, of the frames observed right after a
[declared end](#declared-silence), each of which calls `end()` before it is observed. `target` lists `[frame index, target]` at every frame where the target
them, as are `ends`: the indices, ascending, of the frames observed right after a
[declared end](#declared-silence), each of which calls `end()` before it is observed,
and `skipped`: the indices, ascending, of the frames a consumer closed unread, each
reported with `backlog(arrival)` instead of observed. `target` lists `[frame index, target]` at every frame where the target
differs from the previous frame's, starting at frame 0, so an implementation
folds the trace one frame at a time and compares its target at each listed
index. Tolerance is 1e-6 ms, which the `f64` rules above make generous.
Expand All @@ -779,6 +802,7 @@ index. Tolerance is 1e-6 ms, which the `f64` rules above make generous.
| `reorder` | A run delivered after a newer one is measured against the frames before it, so the target covers waiting for it |
| `backlog` | A backlog delivered newest first behind the live edge is not measured, so its media span never reaches the target |
| `join` | A recorded join whose backlog arrives interleaved with the live edge, and delays the live frames queued behind it, folds nothing, so the target leaves the prior for the path's own |
| `skipped` | The `join` recording with the stale backlog a consumer closes below its cursor reported rather than observed still folds nothing for the join, so the target follows `join`'s |
| `idle` | A 30 s gap relaxes the histogram toward the prior |
| `silence` | A declared 60 s silence and a declared pause leave the target where it was, and a later undeclared gap still relaxes it |
| `tunein` | A stale frame then the live edge 14.5 s later leaves the target near the frame duration. This is the #3517 regression |
Expand Down
48 changes: 42 additions & 6 deletions doc/concept/audio-jitter/corpus.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ export interface Case {
media: number[];
/** Indices of the frames observed right after a declared end, in ascending order. */
ends?: number[];
/** Indices of the frames skipped unread, reported by arrival instead of observed, in ascending order. */
skipped?: number[];
/** `[frame index, target]` wherever the target differs from the previous frame's. */
target: [number, number][];
}
Expand Down Expand Up @@ -107,7 +109,7 @@ function paced(frames: number, frame: number, base: number): Trace {

/** Replay a trace through the reference and keep only the frames where the target moved. */
function run(
trace: Trace & { ends?: number[] },
trace: Trace & { ends?: number[]; skipped?: number[] },
config: { frame: number; advertised?: number; delay?: number },
): [number, number][] {
if (config.delay !== undefined) return [[0, config.delay]];
Expand All @@ -117,9 +119,11 @@ function run(
let previous: number | undefined;

const ends = new Set(trace.ends);
const skipped = new Set(trace.skipped);
for (let i = 0; i < trace.arrival.length; i++) {
if (ends.has(i)) estimator.end();
estimator.observe({ arrival: trace.arrival[i], media: trace.media[i] });
if (skipped.has(i)) estimator.backlog(trace.arrival[i]);
else estimator.observe({ arrival: trace.arrival[i], media: trace.media[i] });
const value = estimator.target;
if (value !== previous) target.push([i, value]);
previous = value;
Expand All @@ -130,15 +134,41 @@ function run(

const FRAME = 20;

/** The first `until` ms of a recorded replay trace under `test/audio-quality/traces`. */
function recorded(name: string, until: number): Trace {
/** A recorded replay trace under `test/audio-quality/traces`: each frame's arrival, media, and group. */
function arrivals(name: string): [number, number, number][] {
const here = dirname(fileURLToPath(import.meta.url));
const path = join(here, "../../../test/audio-quality/traces", `${name}.json`);
const { arrivals } = JSON.parse(readFileSync(path, "utf8")) as { arrivals: [number, number][] };
const kept = arrivals.filter(([arrival]) => arrival < until);
return (JSON.parse(readFileSync(path, "utf8")) as { arrivals: [number, number, number][] }).arrivals;
}

/** The first `until` ms of a recorded replay trace. */
function recorded(name: string, until: number): Trace {
const kept = arrivals(name).filter(([arrival]) => arrival < until);
return { arrival: kept.map(([arrival]) => arrival), media: kept.map(([, media]) => media) };
}

/**
* The first `until` ms of a recorded replay trace, with the frames of every group that arrives two or
* more below the newest one within `[from, to)` ms skipped, as a container consumer closes the groups
* that arrive below its cursor.
*/
function skipping(name: string, until: number, from: number, to: number): Trace & { skipped: number[] } {
const kept = arrivals(name).filter(([arrival]) => arrival < until);
const skipped: number[] = [];
const closed = new Set<number>();
const opened = new Set<number>();
let newest = Number.NEGATIVE_INFINITY;
kept.forEach(([arrival, , group], i) => {
if (!opened.has(group)) {
opened.add(group);
if (arrival >= from && arrival < to && group < newest - 1) closed.add(group);
}
if (closed.has(group)) skipped.push(i);
newest = Math.max(newest, group);
});
return { arrival: kept.map(([arrival]) => arrival), media: kept.map(([, media]) => media), skipped };
}

const traces: Omit<Case, "target">[] = [
{
name: "paced",
Expand Down Expand Up @@ -204,6 +234,12 @@ const traces: Omit<Case, "target">[] = [
frame: FRAME,
...recorded("relay-mic-rtt40", 30_000),
},
{
name: "skipped",
about: "The `join` recording as a container consumer reads it: from 0.9 s on, it closes every group arriving two or more below the newest unread, the join's stale backlog among them, and reports each one's arrival instead of observing it. The estimator never sees those frames are backlog, but the join still lasts until they arrived, so the target leaves the prior for the path's own as in `join`, instead of folding the live frames queued behind the backlog as a 300 ms stall. A group skipped once an interval has measured the path is an outage's, and leaves the join where it ended.",
frame: FRAME,
...skipping("relay-mic-rtt40", 30_000, 900, 30_000),
},
{
name: "idle",
about: "A 30 s gap in both clocks, as a suspended tab produces. The idle intervals relax the histogram back toward the prior rather than freezing it.",
Expand Down
14 changes: 13 additions & 1 deletion doc/concept/audio-jitter/reference.ts
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@ export class Estimator {
#start?: number;
/** The latest a frame produced before the subscription arrived, starting at `#start`. */
#joined = 0;
/** Every interval closed so far measured the join, so a skipped backlog can still extend it. */
#open = true;
/** A declared end precedes the next measured observation, so the silence before it is not idle. */
#ended = false;
/** The earliest a frame stalled in this interval was due. */
Expand Down Expand Up @@ -167,7 +169,10 @@ export class Estimator {
// join, not the path: those frames queued behind the backlog, and the player skipped onto
// the clock past all of it. It folds nothing, like an interval with no arrival.
if (this.#oldest + this.#fastest() < this.#joined) this.#pending += 1;
else this.#close(Math.min(BUCKETS - 1, Math.floor(this.#peak / BUCKET)));
else {
this.#open = false;
this.#close(Math.min(BUCKETS - 1, Math.floor(this.#peak / BUCKET)));
}

if (resumed) {
// The publisher declared the silence, so it is not evidence of anything: the
Expand Down Expand Up @@ -244,6 +249,13 @@ export class Estimator {
this.#stall = undefined;
}

/** A frame produced before the subscription arrived at `arrival` but was skipped unread. */
backlog(arrival: number): void {
// Before any observation the join has not started, and starts later than this.
// Once an interval measured the path the join is over, and a skip is an outage's.
if (this.#start !== undefined && this.#open) this.#joined = Math.max(this.#joined, arrival);
}

/** A declared end lies between the previous observation and the next. */
end(): void {
// Before any observation there is no interval for the silence to stretch.
Expand Down
1 change: 1 addition & 0 deletions doc/concept/audio-jitter/skipped.json

Large diffs are not rendered by default.

Loading