diff --git a/doc/concept/audio-jitter.md b/doc/concept/audio-jitter.md
index a7d55b60b45..443e0877bc0 100644
--- a/doc/concept/audio-jitter.md
+++ b/doc/concept/audio-jitter.md
@@ -400,8 +400,7 @@ such as the new name a codec change publishes under, continues the outgoing
track's histogram, prior weight included, instead of starting over. The path's
arrivals did not change with the name; a track on another path starts from the prior.
Restarting would drop the target to the cold-start 100 ms at the switch, then
-raise it as the lateness is relearned, and the playout follows both moves:
-buffered audio is skipped on the drop and playback parks on each rise. Only the
+raise it as the lateness is relearned, and the playout follows both moves. Only the
reference window restarts, so the new track's frames are measured against its
own fastest. Its opening backlog would read as seconds late against the old
track's arrivals.
@@ -614,15 +613,18 @@ the `bun doc/concept/audio-jitter/corpus.ts` command above. It runs in
## Received-audio time stretch
-Both players adapt received PCM to the target with the seams of WebRTC
-NetEq's time stretch: native playback in `rs/moq-audio` with its accelerate and
-preemptive expand, and the browser in the AudioWorklet's ring in `js/watch`
-with its accelerate alone. Neither generates replacement audio for a missing
-packet, so a hole in the media plays as a hole. The estimator above is
-unchanged. Both share the pitch search, admission, seams, and
-[PCM corpus](audio-jitter/stretch/corpus.json) below; they differ only in
-[what they decide on](#browser-decision). The browser stretches only live
-playback: buffered playback plays every sample it holds, as it always has.
+Every player adapts received PCM to the shared clock with the seams of WebRTC
+NetEq's time stretch: native playback in `rs/moq-audio`, and the browser in the
+AudioWorklet's ring in `js/watch`, live or buffered. Neither generates
+replacement audio for a missing packet, so a hole in the media plays as a hole.
+The estimator above is unchanged. Both share the decision, level filter, pitch
+search, admission, seams, and [corpus](#stretch-corpus) below.
+
+So in steady state audio plays on the shared clock at the playout delay. When
+the audio received ahead of the playhead runs low, playback slows down and
+falls behind the clock; once that buffer is back above the target, playback
+speeds up until audio is on the clock again. A congestion stall costs a gap and
+a catch-up, not the audio that arrived late.
The source is WebRTC's
[`TimeStretch`](https://webrtc.googlesource.com/src/+/refs/heads/main/modules/audio_coding/neteq/time_stretch.cc),
@@ -646,45 +648,116 @@ does the tail before a [declared end](#declared-silence): running dry after it
is silence rather than an underrun, so nothing expands toward a target that no
arrival will refill.
-`buffered` counts unread frames in the ring plus staged output. `target` is
-the current playout target. The state is `cooldown`, initially 60 ms
-(NetEq's six startup ticks), reduced
-by each frame rendered, saturating at zero. At a window boundary, native
-playback chooses in this order, with `high = target + 20 ms`:
+The state is `cooldown`, initially 60 ms (NetEq's six startup ticks), reduced
+by each frame rendered, saturating at zero. A successful stretch sets it to
+50 ms (NetEq's five 10 ms ticks); a rejected seam leaves it unchanged. There is
+no concealment state.
+
+### Decision
+
+At each window boundary a player decides on five inputs:
+
+- `target`: the playout delay the shared clock plays at.
+- `level`: how far the playhead trailed the newest received audio when it
+ arrived, decoded or not, less what the stretch and skips removed since.
+ Between arrivals the depth drains because the next one has not landed, not
+ because audio is ahead of the clock, so deciding on the depth would slow down
+ into every gap and settle a burst behind. `level` is the playout delay NetEq's
+ `DecisionLogic` compares, with the publisher's cadence taken out: a batched
+ publisher such as AAC in TS, which delivers a whole PES at once, settles
+ where an evenly paced one does.
+- `raw`: the received audio ahead of the playhead, decoded or not. A stall
+ delivers no arrival to lower `level`, so `raw` stops a playhead behind the
+ clock from speeding into it.
+- `lag`: how far the playhead trails the shared clock, negative when it plays
+ ahead of it.
+- `bound`: how far audio may trail the clock, which is the subscription's max
+ delay less the playout delay. It is zero at a fixed live delay, the lookahead
+ for a buffered player, and at `auto` the shared max delay raised to the
+ estimator's 2 s ceiling, less the delay.
+
+`level` passes through NetEq's `BufferLevelFilter` once per window: three of its
+10 ms steps at the factor the target selects, 251, 252, 253, or 254 in 256ths
+for a target up to 20, 60, or 140 ms, or longer. With `w = (factor / 256)^3`
+and `stretched` the frames the stretch removed since the previous update
+(negative when it added them):
+
+```
+filtered = max(0, w * filtered + (1 - w) * level - stretched)
+```
+
+The first update after a start, an underrun, or a new timeline takes `level`
+as it is.
+
+With NetEq's limits, its 85 ms deceleration offset and 20 ms granularity,
+
+```
+low = max(floor(3 * target / 4), target - 85 ms)
+high = max(target, low + 20 ms)
+buffer = min(filtered, raw)
+```
+
+choose in this order:
-1. `buffered >= 4 * high`: fast accelerate, even during cooldown.
+1. `lag > 0` and `buffer >= 4 * high`: fast accelerate, up to `lag`, even
+ during cooldown.
2. `cooldown > 0`: hold.
-3. `buffered >= high`: accelerate.
-4. `buffered < target`: preemptive expand.
-5. Otherwise: hold.
-
-A successful stretch sets cooldown to 50 ms (NetEq's five 10 ms ticks).
-A rejected seam leaves it unchanged. This preserves NetEq's 20 ms adjustment
-granularity, 4x fast threshold, and minimum interval. NetEq's extra
-packet-history maximum-delay term is not added again: this controller sees the
-already decoded PCM queue and its jitter target rather than an RTP playout
-timestamp and packet-arrival history. There is no concealment state.
-
-### Browser decision
-
-The browser's ring starts on its target and plays at the shared clock video
-plays from, so slowing it down only ever settles audio behind that clock. It
-never expands. It decides on `peak`, the ring's length right after the latest
-write less what the stretch has removed since: how far the playhead trails the
-newest arrival, which is the playout delay NetEq's `DecisionLogic` compares.
-`buffered` drains between arrivals because the next one has not landed, not
-because audio is ahead of the clock, so deciding on it expands into every gap
-and settles a burst behind. With `excess = peak - target`, choose in this order:
-
-1. `excess >= 20 ms`: fast accelerate, even during cooldown.
-2. `cooldown > 0` or `excess <= 0`: hold.
-3. Otherwise: accelerate, admitting only a lag `L <= excess`.
-
-So the ring rests within one pitch period of its target right after each
-arrival, where dropping back to the target used to leave it, and a fast
-accelerate after a refill catches up about as fast as that drop did. Arrivals that
-outrun the target underrun instead: that is the cost of a target short of the
-spread, and the estimator's to fix, not the stretch's.
+3. `lag > 0` and `buffer >= high`: accelerate, up to `lag`.
+4. `filtered < low` and `lag < bound`: preemptive expand, up to `bound - lag`.
+5. `lag < 0`: preemptive expand, up to `-lag`.
+6. Otherwise: hold.
+
+The "up to" is the decision's allowance: a seam never changes the output by
+more. So speeding up only ever catches a playhead behind the clock up to it,
+and slowing down lets audio fall behind by at most `bound`, or brings a
+playhead ahead of the clock back onto it. These are NetEq's thresholds and
+rates, untuned. A moved target moves the clock, so a rise leaves audio ahead of
+it and a fall leaves audio behind, and both become ordinary slowing down and
+speeding up: nothing parks in silence on a rise, and in `auto` nothing skips
+buffered audio on a fall, since the bound, the max delay less the target,
+always covers a fall of the target. A fixed delay lowered at runtime by more
+than its bound and one window skips to the bound, as any lag past it does.
+
+Without a clock reading the target is where the playhead belongs, so
+`lag = level - target`. The browser then never skips; native playback bounds
+the lag at three quarters of its ring's capacity above the target.
+
+### Skips, underruns, and silence
+
+A playhead that trails the clock by more than `bound` and one window skips
+ahead to `bound` behind it, dropping the staged window and the audio it passes.
+Less than that is the stretch's to catch up. A join is the exception: until
+the player has read once on the clock, its bound is zero, so the backlog a
+subscription opens on is skipped rather than played out at the stretch's pace,
+and playback starts on the clock. A shared clock whose reference moves earlier,
+as it does through a tune-in backlog that arrives faster than real time, jumps
+ahead by as much, and audio skips those jumps as video skips the frames they
+pass, as far as the playhead trails the clock and the player holds audio past
+its target, which the early media that moved the reference leaves it. Jumps
+skip once they add up to more than one window, so a publisher's clock drifting
+fast, which moves the reference by a little at a time, and a reference moving
+while the buffer runs low, are the stretch's to catch up. Only those
+skips and a ring overflowing its fixed capacity drop received audio; each
+counts as a skip in
+the player's stats. A finite tail before a declared end plays as received, and
+a playhead behind the clock skips a declared silence by its lag rather than
+stretching across it, which drops no audio.
+
+After an underrun the player resumes on a cushion of one window and one read,
+the device period natively and the render quantum in the browser, and never
+more than it starts on. It resets the filter, so the low level it resumes on
+slows playback down until the buffer is back at the target. Refilling to the
+whole target instead would add up to one target of silence and leave audio
+that much further behind. A declared end releases the wait, so a short tail
+drains.
+
+`level` and `raw` count everything received ahead of the playhead, decoded or
+still encoded. The browser holds the part of the target above its ring's floor
+encoded, and a buffered player its whole lookahead, while native playback
+decodes everything at once. The browser's decode gate releases a playing ring's
+next frame while what it was given, landed or still decoding, reaches less than
+its floor and one window past the playhead, so the ring always holds a full
+window to stretch, and a hole in the media plays as silence.
### Pitch search and admission
@@ -727,7 +800,9 @@ still needs the listening proof required by the quest.
### Seam and boundaries
-In fast mode replace `L` by `floor(M/L)*L` after admission. For a crossfade of
+Fast mode takes every whole period that fits both before the middle and in the
+allowance, `min(floor(M/L), floor(allowance/L)) * L`, after admission; the
+others take `L` when `L <= allowance` and hold otherwise. For a crossfade of
`L` frames, frame `i` uses `w=(i+1)/(L+1)` and
`f32(a*(1-w)+b*w)` independently for every channel.
@@ -743,29 +818,24 @@ setup; callback processing has no allocation, lock, or host-clock read.
`buffered()` includes staged output, and finishing accounts for it before
completing the sink.
-After a real underrun, emergency recovery emits silence directly until the
-ring and staged output hold the target again. Padding never enters the pitch
-search. Finishing bypasses that wait so a short tail can drain. Trimming is
-emergency recovery at the ring's capacity watermark. The
-CLI's incoming-media trim is reserved for a distance at the fast threshold,
-not ordinary 20 ms drift. The browser's ring receives each decoded frame as a
-write, so it re-anchors on its target there, at the lower of `4 * high` and
-three quarters of its headroom above the target, and counts each re-anchor as
-a skip in its stats. These bounds do not promise concealment of missing
-media or convergence of a signal whose seams continually fail admission.
+These bounds do not promise concealment of missing media or convergence of a
+signal whose seams continually fail admission.
### Stretch corpus
`audio-jitter/stretch/corpus.json` holds literal interleaved input and expected
-output PCM, device rate and channels, buffered and target frame counts,
-incoming cooldown, requested decision, and admitted lag (zero means hold).
-Compare output within 1e-6 absolute sample amplitude. It includes hold,
-cooldown, fast-mode bypass, linked stereo, passive noise, rejected active
-noise, and 8/44.1/48 kHz seams. Both implementations consume these exact
-vectors; generating expected output from the implementation under test is
-not conformance. The vectors decide as native playback does, so the browser
-grades its accelerate and fast seams against them, at an excess inside and
-past the 20 ms band, and holds wherever native expands.
+output PCM, device rate and channels, the decision and allowance to stage, and
+the admitted lag (zero means hold). Compare output within 1e-6 absolute sample
+amplitude. It includes hold, accelerate and fast seams within and past their
+allowance, linked stereo, passive noise, rejected active noise, the normal and
+fast thresholds, and 8/44.1/48 kHz seams.
+
+`audio-jitter/stretch/decisions.json` holds the rule's cases, each a target,
+filtered level, raw depth, lag, bound, and cooldown with the decision and
+allowance they produce, and the filter's, each a target and a series of levels
+and stretched frames with the filtered level after each update. Both
+implementations consume these exact vectors; generating expected output from
+the implementation under test is not conformance.
## Comfort noise
diff --git a/doc/concept/audio-jitter/stretch/corpus.json b/doc/concept/audio-jitter/stretch/corpus.json
index e864b8552e1..189c37fa97f 100644
--- a/doc/concept/audio-jitter/stretch/corpus.json
+++ b/doc/concept/audio-jitter/stretch/corpus.json
@@ -53,10 +53,8 @@
-0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
-0.11349762231111526, -0.07725425064563751, -0.03910861536860466
],
- "buffered": 320,
- "target": 640,
- "cooldown": 0,
"decision": "Expand",
+ "allowance": 8000,
"lag": 40,
"output": [
0.0, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
@@ -171,10 +169,8 @@
-0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
-0.11349762231111526, -0.07725425064563751, -0.03910861536860466
],
- "buffered": 800,
- "target": 640,
- "cooldown": 0,
"decision": "Accelerate",
+ "allowance": 8000,
"lag": 40,
"output": [
0.0, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
@@ -220,7 +216,7 @@
]
},
{
- "name": "fast-voiced",
+ "name": "accelerate-past-allowance",
"rate": 8000,
"channels": 1,
"input": [
@@ -273,40 +269,62 @@
-0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
-0.11349762231111526, -0.07725425064563751, -0.03910861536860466
],
- "buffered": 3200,
- "target": 640,
- "cooldown": 160,
- "decision": "Fast",
- "lag": 120,
+ "decision": "Accelerate",
+ "allowance": 39,
+ "lag": 0,
"output": [
- -1.518157148455191e-18, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928,
- 0.1767766922712326, 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25,
- 0.2469220906496048, 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326,
- 0.14694631099700928, 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -1.8340316039463714e-16,
- -0.03910861536860466, -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326,
- -0.2022542506456375, -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25,
- -0.2469220906496048, -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326,
- -0.14694631099700928, -0.11349762231111526, -0.07725425064563751, -0.03910861536860466,
- 3.2428256691958903e-16, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928,
- 0.1767766922712326, 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25,
- 0.2469220906496048, 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326,
- 0.14694631099700928, 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, 1.8445610175741304e-16,
- -0.03910861536860466, -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326,
- -0.2022542506456375, -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25,
- -0.2469220906496048, -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326,
- -0.14694631099700928, -0.11349762231111526, -0.07725425064563751, -0.03910861536860466,
- -9.862906562621747e-17, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928,
- 0.1767766922712326, 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25,
- 0.2469220906496048, 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326,
- 0.14694631099700928, 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -4.34957347645714e-16,
- -0.03910861536860466, -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326,
- -0.2022542506456375, -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25,
- -0.2469220906496048, -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326,
- -0.14694631099700928, -0.11349762231111526, -0.07725425064563751, -0.03910861536860466
+ 0.0, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
+ 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25, 0.2469220906496048,
+ 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326, 0.14694631099700928,
+ 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -8.040613114788646e-17, -0.03910861536860466,
+ -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326, -0.2022542506456375,
+ -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25, -0.2469220906496048,
+ -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
+ -0.11349762231111526, -0.07725425064563751, -0.03910861536860466, 1.6081226229577292e-16,
+ 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
+ 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25, 0.2469220906496048,
+ 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326, 0.14694631099700928,
+ 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, 9.184850732644269e-17, -0.03910861536860466,
+ -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326, -0.2022542506456375,
+ -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25, -0.2469220906496048,
+ -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
+ -0.11349762231111526, -0.07725425064563751, -0.03910861536860466, 3.2162452459154583e-16,
+ 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
+ 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25, 0.2469220906496048,
+ 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326, 0.14694631099700928,
+ 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, 1.5308084995570108e-16, -0.03910861536860466,
+ -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326, -0.2022542506456375,
+ -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25, -0.2469220906496048,
+ -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
+ -0.11349762231111526, -0.07725425064563751, -0.03910861536860466, -1.8369701465288538e-16,
+ 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
+ 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25, 0.2469220906496048,
+ 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326, 0.14694631099700928,
+ 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -6.73865213880276e-16, -0.03910861536860466,
+ -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326, -0.2022542506456375,
+ -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25, -0.2469220906496048,
+ -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
+ -0.11349762231111526, -0.07725425064563751, -0.03910861536860466, 6.432490491830917e-16,
+ 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
+ 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25, 0.2469220906496048,
+ 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326, 0.14694631099700928,
+ 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, 2.7554553521421787e-16, -0.03910861536860466,
+ -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326, -0.2022542506456375,
+ -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25, -0.2469220906496048,
+ -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
+ -0.11349762231111526, -0.07725425064563751, -0.03910861536860466, -3.0616169991140216e-16,
+ 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
+ 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25, 0.2469220906496048,
+ 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326, 0.14694631099700928,
+ 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -5.514005550915388e-16, -0.03910861536860466,
+ -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326, -0.2022542506456375,
+ -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25, -0.2469220906496048,
+ -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
+ -0.11349762231111526, -0.07725425064563751, -0.03910861536860466
]
},
{
- "name": "hold-in-band",
+ "name": "fast-voiced",
"rate": 8000,
"channels": 1,
"input": [
@@ -359,12 +377,41 @@
-0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
-0.11349762231111526, -0.07725425064563751, -0.03910861536860466
],
- "buffered": 720,
- "target": 640,
- "cooldown": 0,
- "decision": "Hold",
- "lag": 0,
+ "decision": "Fast",
+ "allowance": 8000,
+ "lag": 120,
"output": [
+ -1.518157148455191e-18, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928,
+ 0.1767766922712326, 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25,
+ 0.2469220906496048, 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326,
+ 0.14694631099700928, 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -1.8340316039463714e-16,
+ -0.03910861536860466, -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326,
+ -0.2022542506456375, -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25,
+ -0.2469220906496048, -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326,
+ -0.14694631099700928, -0.11349762231111526, -0.07725425064563751, -0.03910861536860466,
+ 3.2428256691958903e-16, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928,
+ 0.1767766922712326, 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25,
+ 0.2469220906496048, 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326,
+ 0.14694631099700928, 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, 1.8445610175741304e-16,
+ -0.03910861536860466, -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326,
+ -0.2022542506456375, -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25,
+ -0.2469220906496048, -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326,
+ -0.14694631099700928, -0.11349762231111526, -0.07725425064563751, -0.03910861536860466,
+ -9.862906562621747e-17, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928,
+ 0.1767766922712326, 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25,
+ 0.2469220906496048, 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326,
+ 0.14694631099700928, 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -4.34957347645714e-16,
+ -0.03910861536860466, -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326,
+ -0.2022542506456375, -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25,
+ -0.2469220906496048, -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326,
+ -0.14694631099700928, -0.11349762231111526, -0.07725425064563751, -0.03910861536860466
+ ]
+ },
+ {
+ "name": "fast-within-allowance",
+ "rate": 8000,
+ "channels": 1,
+ "input": [
0.0, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25, 0.2469220906496048,
0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326, 0.14694631099700928,
@@ -413,10 +460,47 @@
-0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25, -0.2469220906496048,
-0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
-0.11349762231111526, -0.07725425064563751, -0.03910861536860466
+ ],
+ "decision": "Fast",
+ "allowance": 119,
+ "lag": 80,
+ "output": [
+ 0.0, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
+ 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25, 0.2469220906496048,
+ 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326, 0.14694631099700928,
+ 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -8.040613114788646e-17, -0.03910861536860466,
+ -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326, -0.2022542506456375,
+ -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25, -0.2469220906496048,
+ -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
+ -0.11349762231111526, -0.07725425064563751, -0.03910861536860466, 1.5655905840470286e-16,
+ 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
+ 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25, 0.2469220906496048,
+ 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326, 0.14694631099700928,
+ 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -1.0666986336771883e-16,
+ -0.03910861536860466, -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326,
+ -0.2022542506456375, -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25,
+ -0.2469220906496048, -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326,
+ -0.14694631099700928, -0.11349762231111526, -0.07725425064563751, -0.03910861536860466,
+ 4.844221130016746e-16, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928,
+ 0.1767766922712326, 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25,
+ 0.2469220906496048, 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326,
+ 0.14694631099700928, 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, 2.453073324999662e-16,
+ -0.03910861536860466, -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326,
+ -0.2022542506456375, -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25,
+ -0.2469220906496048, -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326,
+ -0.14694631099700928, -0.11349762231111526, -0.07725425064563751, -0.03910861536860466,
+ -3.0616169991140216e-16, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928,
+ 0.1767766922712326, 0.2022542506456375, 0.22275163233280182, 0.2377641350030899, 0.2469220906496048, 0.25,
+ 0.2469220906496048, 0.2377641350030899, 0.22275163233280182, 0.2022542506456375, 0.1767766922712326,
+ 0.14694631099700928, 0.11349762231111526, 0.07725425064563751, 0.03910861536860466, -5.514005550915388e-16,
+ -0.03910861536860466, -0.07725425064563751, -0.11349762231111526, -0.14694631099700928, -0.1767766922712326,
+ -0.2022542506456375, -0.22275163233280182, -0.2377641350030899, -0.2469220906496048, -0.25,
+ -0.2469220906496048, -0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326,
+ -0.14694631099700928, -0.11349762231111526, -0.07725425064563751, -0.03910861536860466
]
},
{
- "name": "hold-cooldown",
+ "name": "hold",
"rate": 8000,
"channels": 1,
"input": [
@@ -469,10 +553,8 @@
-0.2377641350030899, -0.22275163233280182, -0.2022542506456375, -0.1767766922712326, -0.14694631099700928,
-0.11349762231111526, -0.07725425064563751, -0.03910861536860466
],
- "buffered": 320,
- "target": 640,
- "cooldown": 1,
"decision": "Hold",
+ "allowance": 0,
"lag": 0,
"output": [
0.0, 0.03910861536860466, 0.07725425064563751, 0.11349762231111526, 0.14694631099700928, 0.1767766922712326,
@@ -580,10 +662,8 @@
-0.6991210579872131, -0.7436450123786926, 0.511226236820221, 0.30140137672424316, -0.3033992648124695,
0.7745504975318909, -0.3500271439552307
],
- "buffered": 320,
- "target": 640,
- "cooldown": 0,
"decision": "Expand",
+ "allowance": 8000,
"lag": 0,
"output": [
0.2834013104438782, 0.45585817098617554, 0.0327458530664444, 0.0183867234736681, -0.1703445464372635,
@@ -703,10 +783,8 @@
-0.0009706376004032791, 0.0006069837254472077, 0.00024918612325564027, 0.0006460933363996446,
-0.0007419105968438089, -0.00017247976211365312, 0.00025107196415774524, 0.0007953110034577549
],
- "buffered": 320,
- "target": 640,
- "cooldown": 0,
"decision": "Expand",
+ "allowance": 8000,
"lag": 62,
"output": [
-0.0007030649576336145, 0.0009325314313173294, -0.0008886777213774621, -0.00027434012736193836,
@@ -889,10 +967,8 @@
-0.1767766922712326, 0.1767766922712326, -0.14694631099700928, 0.14694631099700928, -0.11349762231111526,
0.11349762231111526, -0.07725425064563751, 0.07725425064563751, -0.03910861536860466, 0.03910861536860466
],
- "buffered": 320,
- "target": 640,
- "cooldown": 0,
"decision": "Expand",
+ "allowance": 8000,
"lag": 40,
"output": [
0.0, -0.0, 0.03910861536860466, -0.03910861536860466, 0.07725425064563751, -0.07725425064563751,
@@ -1294,10 +1370,8 @@
-0.04953653737902641, -0.04253482446074486, -0.03549857810139656, -0.028433511033654213,
-0.021345358341932297, -0.014239874668419361, -0.007122828625142574
],
- "buffered": 1764,
- "target": 3528,
- "cooldown": 0,
"decision": "Expand",
+ "allowance": 44100,
"lag": 441,
"output": [
0.0, 0.007122828625142574, 0.014239874668419361, 0.021345358341932297, 0.028433511033654213,
@@ -1976,10 +2050,8 @@
-0.045558881014585495, -0.03910861536860466, -0.03263154625892639, -0.026132116094231606,
-0.019614774733781815, -0.013083988800644875, -0.006544237025082111
],
- "buffered": 4800,
- "target": 3840,
- "cooldown": 0,
"decision": "Accelerate",
+ "allowance": 48000,
"lag": 240,
"output": [
0.0, 0.006544237025082111, 0.013083988800644875, 0.019614774733781815, 0.026132116094231606,
@@ -2286,10 +2358,8 @@
-0.603949248790741, -0.10583518445491791, -0.32648566365242004, -0.4554884731769562, -0.27422934770584106,
-0.09860859811306
],
- "buffered": 800,
- "target": 640,
- "cooldown": 0,
"decision": "Accelerate",
+ "allowance": 8000,
"lag": 0,
"output": [
0.20665311813354492, 0.2251681536436081, 0.09140059351921082, 0.05964577570557594, 0.27126818895339966,
@@ -2398,10 +2468,8 @@
-0.603949248790741, -0.10583518445491791, -0.32648566365242004, -0.4554884731769562, -0.27422934770584106,
-0.09860859811306
],
- "buffered": 3200,
- "target": 640,
- "cooldown": 0,
"decision": "Fast",
+ "allowance": 8000,
"lag": 81,
"output": [
0.20665311813354492, 0.2251681536436081, 0.09140059351921082, 0.05964577570557594, 0.27126818895339966,
diff --git a/doc/concept/audio-jitter/stretch/decisions.json b/doc/concept/audio-jitter/stretch/decisions.json
new file mode 100644
index 00000000000..4efea705077
--- /dev/null
+++ b/doc/concept/audio-jitter/stretch/decisions.json
@@ -0,0 +1,360 @@
+{
+ "decisions": [
+ {
+ "name": "on-the-clock-holds",
+ "rate": 8000,
+ "target": 800,
+ "level": 900,
+ "raw": 900,
+ "lag": 0,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "behind-with-a-full-buffer-accelerates",
+ "rate": 8000,
+ "target": 800,
+ "level": 900,
+ "raw": 900,
+ "lag": 100,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Accelerate",
+ "allowance": 100
+ },
+ {
+ "name": "behind-in-the-band-holds",
+ "rate": 8000,
+ "target": 800,
+ "level": 700,
+ "raw": 900,
+ "lag": 100,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "behind-into-a-stall-holds",
+ "rate": 8000,
+ "target": 800,
+ "level": 900,
+ "raw": 500,
+ "lag": 100,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "a-low-level-expands-to-the-bound",
+ "rate": 8000,
+ "target": 800,
+ "level": 500,
+ "raw": 500,
+ "lag": 100,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Expand",
+ "allowance": 14300
+ },
+ {
+ "name": "a-low-level-at-the-bound-holds",
+ "rate": 8000,
+ "target": 800,
+ "level": 500,
+ "raw": 500,
+ "lag": 14400,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "a-low-level-without-a-bound-holds",
+ "rate": 8000,
+ "target": 800,
+ "level": 500,
+ "raw": 500,
+ "lag": 0,
+ "bound": 0,
+ "cooldown": 0,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "a-low-level-ahead-expands-to-the-bound",
+ "rate": 8000,
+ "target": 800,
+ "level": 500,
+ "raw": 500,
+ "lag": -100,
+ "bound": 0,
+ "cooldown": 0,
+ "decision": "Expand",
+ "allowance": 100
+ },
+ {
+ "name": "ahead-expands-onto-the-clock",
+ "rate": 8000,
+ "target": 800,
+ "level": 900,
+ "raw": 900,
+ "lag": -100,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Expand",
+ "allowance": 100
+ },
+ {
+ "name": "far-behind-fast-accelerates",
+ "rate": 8000,
+ "target": 800,
+ "level": 3300,
+ "raw": 3300,
+ "lag": 2000,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Fast",
+ "allowance": 2000
+ },
+ {
+ "name": "far-behind-fast-accelerates-in-cooldown",
+ "rate": 8000,
+ "target": 800,
+ "level": 3300,
+ "raw": 3300,
+ "lag": 2000,
+ "bound": 14400,
+ "cooldown": 1,
+ "decision": "Fast",
+ "allowance": 2000
+ },
+ {
+ "name": "behind-in-cooldown-holds",
+ "rate": 8000,
+ "target": 800,
+ "level": 900,
+ "raw": 900,
+ "lag": 100,
+ "bound": 14400,
+ "cooldown": 1,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "ahead-in-cooldown-holds",
+ "rate": 8000,
+ "target": 800,
+ "level": 900,
+ "raw": 900,
+ "lag": -100,
+ "bound": 14400,
+ "cooldown": 1,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "fast-needs-the-raw-depth-too",
+ "rate": 8000,
+ "target": 800,
+ "level": 3300,
+ "raw": 900,
+ "lag": 2000,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Accelerate",
+ "allowance": 2000
+ },
+ {
+ "name": "a-deep-buffer-on-the-clock-holds",
+ "rate": 8000,
+ "target": 800,
+ "level": 3300,
+ "raw": 3300,
+ "lag": 0,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "a-filtered-level-just-short-of-high-holds",
+ "rate": 8000,
+ "target": 800,
+ "level": 799.5,
+ "raw": 900,
+ "lag": 10,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "a-filtered-level-at-high-accelerates",
+ "rate": 8000,
+ "target": 800,
+ "level": 800.0,
+ "raw": 900,
+ "lag": 10,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Accelerate",
+ "allowance": 10
+ },
+ {
+ "name": "a-short-target-holds-above-it",
+ "rate": 8000,
+ "target": 160,
+ "level": 200,
+ "raw": 200,
+ "lag": 50,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "a-short-target-accelerates-past-the-band",
+ "rate": 8000,
+ "target": 160,
+ "level": 300,
+ "raw": 300,
+ "lag": 50,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Accelerate",
+ "allowance": 50
+ },
+ {
+ "name": "a-long-target-holds-within-85-ms",
+ "rate": 8000,
+ "target": 4000,
+ "level": 3400,
+ "raw": 3400,
+ "lag": 0,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Hold",
+ "allowance": 0
+ },
+ {
+ "name": "a-long-target-expands-past-85-ms",
+ "rate": 8000,
+ "target": 4000,
+ "level": 3300,
+ "raw": 3300,
+ "lag": 0,
+ "bound": 14400,
+ "cooldown": 0,
+ "decision": "Expand",
+ "allowance": 14400
+ },
+ {
+ "name": "rates-scale-the-limits",
+ "rate": 48000,
+ "target": 5760,
+ "level": 5760,
+ "raw": 5760,
+ "lag": 1,
+ "bound": 86400,
+ "cooldown": 0,
+ "decision": "Accelerate",
+ "allowance": 1
+ },
+ {
+ "name": "rates-scale-the-low-limit",
+ "rate": 48000,
+ "target": 5760,
+ "level": 4319,
+ "raw": 5760,
+ "lag": 0,
+ "bound": 86400,
+ "cooldown": 0,
+ "decision": "Expand",
+ "allowance": 86400
+ }
+ ],
+ "filter": [
+ {
+ "name": "starts-at-the-first-level",
+ "rate": 8000,
+ "target": 800,
+ "steps": [[640, 0]],
+ "filtered": [640.0]
+ },
+ {
+ "name": "smooths-a-step",
+ "rate": 8000,
+ "target": 800,
+ "steps": [
+ [800, 0],
+ [1600, 0],
+ [1600, 0],
+ [1600, 0],
+ [1600, 0],
+ [1600, 0],
+ [1600, 0],
+ [1600, 0],
+ [1600, 0],
+ [1600, 0],
+ [1600, 0]
+ ],
+ "filtered": [
+ 800.0, 827.7966976165771, 854.62757473517, 880.5261896311965, 905.5249345685318, 929.6550763135957,
+ 952.9467952417438, 975.4292230848718, 997.1304793674474, 1018.0777065765398, 1038.297104109836
+ ]
+ },
+ {
+ "name": "factor-at-15-ms",
+ "rate": 8000,
+ "target": 120,
+ "steps": [[1000, 0], [0, 0]],
+ "filtered": [1000.0, 942.5432085990906]
+ },
+ {
+ "name": "factor-at-50-ms",
+ "rate": 8000,
+ "target": 400,
+ "steps": [[1000, 0], [0, 0]],
+ "filtered": [1000.0, 953.8536071777344]
+ },
+ {
+ "name": "factor-at-100-ms",
+ "rate": 8000,
+ "target": 800,
+ "steps": [[1000, 0], [0, 0]],
+ "filtered": [1000.0, 965.2541279792786]
+ },
+ {
+ "name": "factor-at-200-ms",
+ "rate": 8000,
+ "target": 1600,
+ "steps": [[1000, 0], [0, 0]],
+ "filtered": [1000.0, 976.7451286315918]
+ },
+ {
+ "name": "takes-out-what-the-stretch-removed",
+ "rate": 8000,
+ "target": 800,
+ "steps": [[800, 0], [760, 40], [760, 0]],
+ "filtered": [800.0, 758.6101651191711, 758.6584561440703]
+ },
+ {
+ "name": "adds-back-what-the-stretch-added",
+ "rate": 8000,
+ "target": 800,
+ "steps": [[800, 0], [840, -40], [840, 0]],
+ "filtered": [800.0, 841.3898348808289, 841.3415438559297]
+ },
+ {
+ "name": "never-falls-below-zero",
+ "rate": 8000,
+ "target": 800,
+ "steps": [[100, 0], [0, 500]],
+ "filtered": [100.0, 0.0]
+ }
+ ]
+}
diff --git a/doc/lib/js/watch.md b/doc/lib/js/watch.md
index 5b3625766cd..cfb5510b6f0 100644
--- a/doc/lib/js/watch.md
+++ b/doc/lib/js/watch.md
@@ -220,11 +220,14 @@ leave the connection's estimate unclaimed. Without a claim, video selects the be
## Buffered playback
-By default the player minimizes latency: it time-stretches received audio
-that falls behind the delay back onto it, playing it slightly faster, and skips
-ahead only when media piles up far past it. Content produced faster than real time, such as a TTS
+By default the player minimizes latency: audio plays at the delay, slows down
+slightly when the audio received ahead of it runs low, speeds back up once it
+recovers, and skips ahead only audio trailing by more than the subscription
+still delivers. Content produced faster than real time, such as a TTS
response emitted in one burst with future timestamps, wants the opposite. Set
-`lookahead` to how far ahead it may run and it plays through at the encoded pace:
+`lookahead` to how far ahead it may run and it plays through at the encoded pace.
+If a stall leaves audio behind, it catches up by playing faster, as live
+playback does, and skips only what trails by more than the lookahead:
```html
diff --git a/doc/lib/rs/moq-audio.md b/doc/lib/rs/moq-audio.md
index 747c3ee3e40..3a111f7cf1e 100644
--- a/doc/lib/rs/moq-audio.md
+++ b/doc/lib/rs/moq-audio.md
@@ -18,7 +18,7 @@ capture or playback is enabled.
| `capture` | Microphones via CoreAudio, WASAPI, ALSA (and PipeWire/PulseAudio hosts), plus macOS system audio |
| `encode` | PCM to Opus (mono or stereo, with DTX and voice activity) or raw PCM |
| `decode` | Opus (up to 7.1), PCM, and AAC-LC (mono or stereo) back to PCM, resampled to the rate you want |
-| `playback` | One output device mixing every track in a call, with click-free volume ramps |
+| `playback` | One output device mixing every track in a call, with click-free volume ramps, each track's received audio stretched onto the caller's clock (`Sink::due`) |
| `aec` | Acoustic echo cancellation (a port of WebRTC's) for caller-supplied PCM or the built-in devices |
| `bus` | Routes sources into several mixes, each with its own meter and mix-minus, advanced on the caller's clock |
diff --git a/js/watch/src/audio/buffer.test.ts b/js/watch/src/audio/buffer.test.ts
index ecce393df95..17ae04d469e 100644
--- a/js/watch/src/audio/buffer.test.ts
+++ b/js/watch/src/audio/buffer.test.ts
@@ -1,5 +1,6 @@
import { describe, expect, it } from "bun:test";
import { Duration } from "@moq/clock";
+import { type Getter, Signal } from "@moq/signals";
import { AudioBuffer, Backpressure } from "./buffer";
import type { Init, Message, State } from "./render";
import { AudioRingBuffer } from "./ring-buffer";
@@ -79,13 +80,13 @@ describe("AudioBuffer", () => {
buffer.reset();
await received;
expect(messages).toEqual([
- { type: "init", channels: 2, rate: 48_000, latency: 100, buffered: true },
+ { type: "init", channels: 2, rate: 48_000, latency: 100, target: 100, bound: 0 },
{
type: "data",
timestamp: 500_000,
data: [new Float32Array([1, 2, 3]), new Float32Array([4, 5, 6])],
},
- { type: "latency", latency: 200 },
+ { type: "latency", latency: 200, target: 200, bound: 0 },
{ type: "truncate", timestamp: 502_000 },
{ type: "reset", epoch: 1 },
]);
@@ -114,6 +115,7 @@ describe("AudioBuffer", () => {
const { buffer, state, close } = open();
try {
await state(100_000, false, 0);
+ buffer.insert(100_000, [new Float32Array(9600)]);
const held = buffer.wait(10_000_000, new AbortController().signal);
expect(Bun.peek.status(held)).toBe("pending");
buffer.close();
@@ -132,6 +134,7 @@ describe("AudioBuffer", () => {
const { buffer, state, close } = open();
try {
await state(0, false, 0);
+ buffer.insert(0, [new Float32Array(9600)]);
const controller = new AbortController();
const wait = buffer.wait(10_000_000, controller.signal);
await Promise.resolve();
@@ -146,21 +149,21 @@ describe("AudioBuffer", () => {
});
// Media timestamps are absolute microseconds, so they pass 2^31 after ~36 minutes of broadcast
- // and 2^32 after ~72. The gate has to hold a frame ahead of the playhead at any of them.
- it("holds a frame one second ahead at any timestamp", async () => {
+ // and 2^32 after ~72. The gate has to hold a playing ring to its floor and a window at any of them.
+ it("holds a playing ring to its floor and a stretch window at any timestamp", async () => {
for (const base of [0, 2 ** 31, 2 ** 32, 3_600_000_000]) {
const { buffer, state, close } = open();
try {
await state(base, false, 0);
- const wait = buffer.wait(base + 1_000_000, new AbortController().signal);
+ buffer.insert(base, [new Float32Array(9600)]);
+ const wait = buffer.wait(base + 200_000, new AbortController().signal);
expect(Bun.peek.status(wait)).toBe("pending");
- // Still more than the floor behind the frame.
- await state(base + 800_000, false, 0);
+ // The ring still holds the 100 ms floor and a 30 ms window.
+ await state(base + 70_000, false, 0);
expect(Bun.peek.status(wait)).toBe("pending");
- // Within the floor of the frame.
- await state(base + 900_000, false, 0);
+ await state(base + 80_000, false, 0);
expect(Bun.peek.status(wait)).toBe("fulfilled");
} finally {
close();
@@ -168,21 +171,21 @@ describe("AudioBuffer", () => {
}
});
- // The decoder's output reaches the ring after the decode loop has moved on, so a stalled ring
- // still looks stalled while the loop releases frame after frame. Releasing all of them pushes
- // the whole lookahead into a floor-sized ring, which drops the oldest unplayed audio.
- it("holds the lookahead while a stalled ring refills", async () => {
+ // The decoder's output reaches the ring after the decode loop has moved on, so the frames still
+ // decoding count toward what the ring holds. Counting only the PCM that landed would push the
+ // whole lookahead into a floor-sized ring, which drops the oldest unplayed audio.
+ it("counts the frames still decoding toward the floor and window", async () => {
const { buffer, state, close } = open();
const signal = new AbortController().signal;
try {
- expect(Bun.peek.status(buffer.wait(0, signal))).toBe("fulfilled");
- expect(Bun.peek.status(buffer.wait(100_000, signal))).toBe("fulfilled");
- const ahead = buffer.wait(120_000, signal);
+ await state(0, false, 0);
+ buffer.insert(0, [new Float32Array(960)]);
+ for (let t = 20_000; t <= 120_000; t += 20_000)
+ expect(Bun.peek.status(buffer.wait(t, signal))).toBe("fulfilled");
+ const ahead = buffer.wait(140_000, signal);
expect(Bun.peek.status(ahead)).toBe("pending");
- // The floor lands and plays past the frame's headroom.
- await state(60_000, false, 0);
- expect(Bun.peek.status(buffer.wait(170_000, signal))).toBe("pending");
+ await state(20_000, false, 0);
expect(Bun.peek.status(ahead)).toBe("fulfilled");
} finally {
close();
@@ -222,11 +225,13 @@ describe("AudioBuffer", () => {
const next = buffer.wait(10_020_000, signal);
expect(Bun.peek.status(next)).toBe("pending");
- // The frame past the gap un-stalls the ring, which plays the gap out up to it.
+ // The frame past the gap un-stalls the ring, which plays the gap out up to it and refills to
+ // its floor and a window.
await state(10_000_000, false, 0);
expect(Bun.peek.status(next)).toBe("fulfilled");
- expect(Bun.peek.status(buffer.wait(10_090_000, signal))).toBe("fulfilled");
- expect(Bun.peek.status(buffer.wait(10_110_000, signal))).toBe("pending");
+ for (let t = 10_040_000; t <= 10_120_000; t += 20_000)
+ expect(Bun.peek.status(buffer.wait(t, signal))).toBe("fulfilled");
+ expect(Bun.peek.status(buffer.wait(10_140_000, signal))).toBe("pending");
} finally {
close();
}
@@ -271,26 +276,36 @@ describe("AudioBuffer", () => {
});
describe("Backpressure", () => {
- it("releases every live frame when the whole target fits the floor", () => {
- const gate = new Backpressure(1000, false, Duration.millis(100));
- gate.observe(ms(0), false);
- expect(Bun.peek.status(gate.wait(ms(10_000), new AbortController().signal))).toBe("fulfilled");
- });
+ // Two seconds is a 500 ms floor and 1.5 s held encoded. Once playing, whatever is queued, the ring
+ // holds its floor and a 30 ms window: the rest waits upstream.
+ for (const buffered of [false, true]) {
+ it(`holds a playing ${buffered ? "buffered" : "live"} ring to its floor and a window`, () => {
+ const gate = new Backpressure(1000, buffered, Duration.millis(2000));
+ gate.queued(ms(60_000));
+ gate.observe(ms(0), false);
+ for (let t = 0; t < 1000; t += 20) gate.inserted(ms(t), ms(t + 20));
+ const wait = gate.wait(ms(1000), new AbortController().signal);
+ gate.observe(ms(470), false);
+ expect(Bun.peek.status(wait)).toBe("pending");
+ gate.observe(ms(471), false);
+ expect(Bun.peek.status(wait)).toBe("fulfilled");
+ expect(Bun.peek.status(gate.wait(ms(1020), new AbortController().signal))).toBe("pending");
+ });
+ }
- it("holds buffered playback to the floor ahead of the playhead", () => {
- const gate = new Backpressure(1000, true, Duration.millis(200));
+ // The next frame past a hole lies far ahead of the playhead, which never nears it while the ring
+ // drains, so it is released by what the ring holds and the hole plays as silence.
+ it("releases the frame past a hole once the ring runs short", () => {
+ const gate = new Backpressure(1000, false, Duration.millis(200));
gate.observe(ms(0), false);
- const wait = gate.wait(ms(1000), new AbortController().signal);
- gate.observe(ms(799), false);
- expect(Bun.peek.status(wait)).toBe("pending");
- gate.observe(ms(800), false);
- expect(Bun.peek.status(wait)).toBe("fulfilled");
+ gate.inserted(ms(0), ms(100));
+ const signal = new AbortController().signal;
+ expect(Bun.peek.status(gate.wait(ms(10_000), signal))).toBe("fulfilled");
+ expect(Bun.peek.status(gate.wait(ms(10_100), signal))).toBe("pending");
});
- // Two seconds is a 500 ms floor and 1.5 s held encoded.
- it("releases a live frame once the queued media covers the hold behind it", () => {
+ it("starts a live ring once the queued media covers the hold behind its first frame", () => {
const gate = new Backpressure(1000, false, Duration.millis(2000));
- gate.observe(ms(0), false);
const wait = gate.wait(ms(1000), new AbortController().signal);
gate.queued(ms(2499));
expect(Bun.peek.status(wait)).toBe("pending");
@@ -298,16 +313,6 @@ describe("Backpressure", () => {
expect(Bun.peek.status(wait)).toBe("fulfilled");
});
- it("plays buffered media through instead of releasing it ahead of the floor", () => {
- const gate = new Backpressure(1000, true, Duration.millis(2000));
- gate.observe(ms(0), false);
- const wait = gate.wait(ms(1000), new AbortController().signal);
- gate.queued(ms(60_000));
- expect(Bun.peek.status(wait)).toBe("pending");
- gate.observe(ms(500), false);
- expect(Bun.peek.status(wait)).toBe("fulfilled");
- });
-
it("starts a stalled ring only once the queued media covers the hold", () => {
const gate = new Backpressure(1000, true, Duration.millis(2000));
const wait = gate.wait(ms(0), new AbortController().signal);
@@ -328,31 +333,6 @@ describe("Backpressure", () => {
expect(Bun.peek.status(gate.wait(ms(520), signal))).toBe("pending");
});
- // A refill that released its floor before the ring reported playing holds the next frame back, so
- // no write could end the park. The queued media covering the hold is what ends it, once.
- it("resumes a park once the queued media covers the hold", () => {
- const gate = new Backpressure(1000, true, Duration.millis(1000));
- expect(gate.queued(ms(1000))).toBe(false);
- const signal = new AbortController().signal;
- for (let t = 0; t <= 500; t += 20) expect(Bun.peek.status(gate.wait(ms(t), signal))).toBe("fulfilled");
- gate.observe(ms(0), true);
- expect(gate.setLatency(Duration.millis(2000))).toBe(true);
- expect(gate.queued(ms(1999))).toBe(false);
- expect(gate.queued(ms(2000))).toBe(true);
- expect(gate.queued(ms(3000))).toBe(false);
- });
-
- // The ring starts playing a report interval before the gate learns its playhead, so a rise in that
- // window parks from the frame the ring anchors on.
- it("parks a ring that started before reporting its playhead", () => {
- const gate = new Backpressure(1000, false, Duration.millis(1000));
- gate.queued(ms(1000));
- const signal = new AbortController().signal;
- for (let t = 0; t <= 500; t += 20) expect(Bun.peek.status(gate.wait(ms(t), signal))).toBe("fulfilled");
- expect(gate.setLatency(Duration.millis(2000))).toBe(true);
- expect(gate.queued(ms(2000))).toBe(true);
- });
-
// A warmup longer than the floor discards everything the refill releases toward it, so the ring
// would never anchor and report the playhead the gate waits on.
it("refills past output discarded before the ring anchors", () => {
@@ -385,18 +365,18 @@ describe("Backpressure", () => {
expect(Bun.peek.status(gate.wait(ms(2040), signal))).toBe("pending");
});
- // Opus pre-skip stamps the PCM behind the input it was decoded from, so a playing ring compares a
- // frame to its playhead where its output lands, or a pre-skip past the floor would drain it.
- it("releases a playing frame by where its output lands", () => {
+ // Opus pre-skip stamps the PCM behind the input it was decoded from, so a playing ring counts a
+ // frame still decoding where its output lands, or a pre-skip past the floor would drain it.
+ it("counts a frame still decoding where its output lands", () => {
const gate = new Backpressure(1000, true, Duration.millis(2000));
- gate.queued(ms(60_000));
gate.observe(ms(0), false);
+ gate.decoded(ms(520), ms(500));
+ gate.inserted(ms(480), ms(500));
const signal = new AbortController().signal;
- const wait = gate.wait(ms(1500), signal);
- expect(Bun.peek.status(wait)).toBe("pending");
- gate.decoded(ms(1020), ms(20));
- expect(Bun.peek.status(wait)).toBe("fulfilled");
- expect(Bun.peek.status(gate.wait(ms(1520), signal))).toBe("pending");
+ expect(Bun.peek.status(gate.wait(ms(520), signal))).toBe("fulfilled");
+ // Its output lands at 520 ms, short of the 530 ms floor and window, not at 540 ms.
+ expect(Bun.peek.status(gate.wait(ms(540), signal))).toBe("fulfilled");
+ expect(Bun.peek.status(gate.wait(ms(560), signal))).toBe("pending");
});
// A replacement rendition keeps the ring, and the decoder it replaced never answers its frames.
@@ -423,17 +403,27 @@ describe("Backpressure", () => {
});
// A worklet whose port records what the buffer posts and delivers state reports synchronously.
-function fake(delay: number, buffered = false) {
+function fake(
+ delay: number,
+ props: {
+ bound?: number;
+ clock?: { now(): Duration | undefined; out: { reference: Getter } };
+ } = {},
+) {
const sent: Message[] = [];
const port = Object.assign(new EventTarget(), {
postMessage: (msg: Message) => sent.push(msg),
start() {},
});
- const buffer = new AudioBuffer({ port } as unknown as AudioWorkletNode, {
+ // The context's time in seconds, which the test advances.
+ const context = { currentTime: 0 };
+ const buffer = new AudioBuffer({ port, context } as unknown as AudioWorkletNode, {
channels: 1,
rate: 1000,
delay: Duration.millis(delay),
- buffered,
+ buffered: false,
+ ...(props.bound !== undefined && { bound: Duration.millis(props.bound) }),
+ clock: props.clock,
});
const state = (timestamp: number, stalled: boolean, epoch: number) => {
const data: State = {
@@ -450,7 +440,7 @@ function fake(delay: number, buffered = false) {
};
port.dispatchEvent(new MessageEvent("message", { data }));
};
- return { buffer, sent, state };
+ return { buffer, sent, state, context };
}
describe("AudioBuffer past its decoded floor", () => {
@@ -489,63 +479,79 @@ describe("AudioBuffer past its decoded floor", () => {
buffer.close();
}
});
+});
- // One second is a 500 ms floor and 500 ms held encoded. Playing 100 ms in, a rise to 1.5 s needs
- // queued media through 1600 ms to keep playing. A report of a stall may be stale, so it parks
- // regardless: parking a ring that is still stalled changes nothing.
- for (const stalled of [false, true]) {
- for (const [queued, parked] of [
- [1599, true],
- [1600, false],
- ] as const) {
- it(`${parked ? "parks" : "keeps playing"} on a rise with ${queued} ms queued, reported ${stalled ? "stalled" : "playing"}`, () => {
- const { buffer, sent, state } = fake(1000);
- try {
- state(ms(100), stalled, 0);
- buffer.queued(ms(queued));
- buffer.setLatency(Duration.millis(1500));
- expect(sent.slice(1)).toEqual([{ type: "latency", latency: 500, ...(parked ? { park: 1 } : {}) }]);
- } finally {
- buffer.close();
- }
- });
+describe("AudioBuffer steering", () => {
+ // A moved delay moves the clock, which the ring plays onto by stretching, so every move only sends
+ // the new terms: nothing parks on a rise or skips on a fall.
+ it("sends the terms on every move of the delay", () => {
+ const { buffer, sent, state } = fake(1000, { bound: 1000 });
+ try {
+ state(ms(100), false, 0);
+ buffer.setLatency(Duration.millis(1500));
+ buffer.setLatency(Duration.millis(200), Duration.millis(1800));
+ expect(sent).toEqual([
+ { type: "init", channels: 1, rate: 1000, latency: 500, target: 1000, bound: 1000 },
+ { type: "latency", latency: 500, target: 1500, bound: 1000 },
+ { type: "latency", latency: 200, target: 200, bound: 1800 },
+ ]);
+ } finally {
+ buffer.close();
}
- }
+ });
- // The ring resumes on any floor update without a park, so a fall that leaves the remaining hold
- // uncovered parks again, and the queued media covering it resumes the ring.
- it("keeps a park across a fall until the queued media covers the hold", () => {
- const { buffer, sent, state } = fake(1000);
+ // Pre-skip lands output behind the frame it was decoded from, so what arrived is placed where it
+ // will land in the ring.
+ it("places received media on the ring's timeline", () => {
+ const { buffer, sent } = fake(100);
try {
- state(ms(0), false, 0);
- buffer.queued(ms(1000));
- buffer.setLatency(Duration.millis(2000));
- buffer.setLatency(Duration.millis(1800));
- buffer.queued(ms(1800));
- buffer.setLatency(Duration.millis(1600));
+ buffer.received(Duration.millis(1000));
+ buffer.decoded(ms(1020), ms(1014));
+ buffer.received(Duration.millis(1040));
expect(sent.slice(1)).toEqual([
- { type: "latency", latency: 500, park: 1 },
- { type: "latency", latency: 500, park: 2 },
- { type: "latency", latency: 500 },
- { type: "latency", latency: 500 },
+ { type: "received", end: 1_000_000 },
+ { type: "received", end: 1_034_000 },
]);
} finally {
buffer.close();
}
});
- // A report the worklet posted before it applied the park still says the ring is playing.
- it("ignores state from before a park", () => {
- const { buffer, state } = fake(1000);
+ // The context's time advances a render quantum at a time, so a reading of the clock lands up to a
+ // quantum late against it. The offset keeps the earliest, rises only as slowly as drift needs, and
+ // jumps when the delay moves the clock for real.
+ it("sends the clock at its earliest offset from the context's time", () => {
+ let now: Duration | undefined = Duration.millis(1000);
+ const reference = new Signal(0);
+ const { buffer, sent, state, context } = fake(100, { clock: { now: () => now, out: { reference } } });
+ const report = (time: number, clock: number | undefined) => {
+ context.currentTime = time;
+ now = clock === undefined ? undefined : Duration.millis(clock);
+ state(ms(0), false, 0);
+ };
try {
- state(ms(200), false, 0);
- buffer.queued(ms(1599));
- buffer.setLatency(Duration.millis(1500));
- const wait = buffer.wait(ms(700), new AbortController().signal);
- state(ms(200), false, 0);
- expect(Bun.peek.status(wait)).toBe("pending");
- state(ms(200), false, 1);
- expect(Bun.peek.status(wait)).toBe("fulfilled");
+ report(1, 1000);
+ // Read 2 ms late: the offset rises by only 100 us.
+ report(1.25, 1252);
+ report(1.5, 1500);
+ // A delay change moved the clock 50 ms.
+ report(1.75, 1800);
+ // The reference moved 5 ms earlier: a jump the offset takes whole, under the filter's.
+ reference.set(-5);
+ report(2, 2055);
+ // A later reference is no jump.
+ reference.set(0);
+ report(2.25, 2300);
+ report(2.5, undefined);
+ expect(sent.filter((msg) => msg.type === "clock")).toEqual([
+ { type: "clock", timestamp: 1_000_000, time: 1 },
+ { type: "clock", timestamp: 1_250_100, time: 1.25 },
+ { type: "clock", timestamp: 1_500_000, time: 1.5 },
+ { type: "clock", timestamp: 1_800_000, time: 1.75 },
+ { type: "clock", timestamp: 2_055_000, time: 2, jumped: 5000 },
+ { type: "clock", timestamp: 2_300_000, time: 2.25 },
+ { type: "clock", time: 2.5 },
+ ]);
} finally {
buffer.close();
}
diff --git a/js/watch/src/audio/buffer.ts b/js/watch/src/audio/buffer.ts
index f100a1ca109..be3d2ae17fc 100644
--- a/js/watch/src/audio/buffer.ts
+++ b/js/watch/src/audio/buffer.ts
@@ -1,7 +1,19 @@
import { Duration } from "@moq/clock";
import { Effect, type Getter, Signal } from "@moq/signals";
import { ringHold, ringSamples } from "./latency";
-import type { Data, End, Init, Latency, Playing, Reset, State, Truncate } from "./render";
+import type { Clock, Data, End, Init, Latency, Playing, Received, Reset, State, Truncate } from "./render";
+
+// The stretch's 30 ms processing window, in microseconds.
+const WINDOW = 30_000;
+
+// How far the clock's offset from the context's time may rise per state report: far past any drift
+// between the two clocks, and well short of the render callback a reading can land within.
+const RISE = 100;
+
+// An offset that moves further than this moved for real, as a delay or a reference change moves it,
+// rather than by where a reading landed within a render callback. Readings after a longer callback
+// all land just after it, so they agree without the filter.
+const JUMP = 10_000;
/**
* Decides when the decode loop may decode a frame, which is how the target delay is split between
@@ -11,19 +23,18 @@ import type { Data, End, Init, Latency, Playing, Reset, State, Truncate } from "
* hold) and, in buffered mode, the lookahead stay encoded in the container consumer, so memory follows
* the bytes actually received rather than whatever a catalog advertised. A frame at `timestamp` is due:
*
- * - while the ring is stalled, once the media queued behind it covers the hold, so playback starts
- * or refills with the whole target ahead of the playhead, decoded or not. A buffered refill also
- * stops once the PCM in the ring plus the frames still decoding reach the floor, since the ring
- * plays through and cannot shed what overflows it;
- * - while playing, once the playhead is within the floor of where its output lands, keeping the
- * ring on its floor. Live, that is where the output ends: the ring's time stretch measures its
- * depth right after each write, and reads a tail past the floor as audio behind the clock;
- * - while playing live, also once the media queued behind it covers the hold, so media past the
- * target reaches the ring, whose skip band catches up to the live edge. Buffered mode plays
- * through instead.
+ * - before playback starts on a timeline, once the media queued behind it covers the hold, so
+ * playback starts with the whole target ahead of the playhead, decoded or not. A buffered start
+ * also stops once the PCM in the ring plus the frames still decoding reach the floor, since
+ * nothing drains the ring until it starts;
+ * - once it has started, while what the ring was given, landed or still decoding, reaches less
+ * than the floor and one stretch window past the playhead, so the ring holds its floor and a
+ * full window to stretch. The frame's own timestamp does not matter: across a hole the next
+ * frame lies far ahead, and the ring plays the hole as silence. A stall keeps releasing by the
+ * playhead: the ring resumes on a cushion and slows down to rebuild, and media past the floor
+ * stays encoded, counted toward the level the ring decides on.
*
- * With no hold, live playback releases every frame at once and the ring bounds itself. The playhead
- * comes from the worklet's state messages.
+ * The playhead comes from the worklet's state messages.
*
* @internal Exported for the replay harness.
*/
@@ -42,14 +53,8 @@ export class Backpressure {
// The ring as last observed. The playhead is undefined until one is known on the current timeline.
#playhead?: number;
- #stalled = true;
- // The first frame released on the current timeline. Until PCM lands and a report carries a
- // playhead, it stands in for where the ring anchors, since the ring may already be playing.
- #anchor?: number;
- // Whether the ring is parked for a hold the queued media does not cover yet. The worklet keeps a
- // park until the next floor update, so only this decides when it ends: a write of output decoded
- // before the park cannot.
- #parked = false;
+ // Whether the ring has played on the current timeline.
+ #started = false;
// A buffered refill counts what reaches the ring, not what it released: output the decoder
// discards (warmup, Opus pre-skip, a terminal trim) never lands, so counting released frames would
@@ -61,12 +66,10 @@ export class Backpressure {
#decoding: number[] = [];
#inserted?: { start: number; end: number };
// How far output lands in the ring behind the frame it was decoded from (Opus pre-skip). A playing
- // ring compares a frame to its playhead where the output lands, or an offset past the floor would
+ // ring counts a frame still decoding where its output lands, or an offset past the floor would
// drain it.
#offset = 0;
- // The duration of the latest output, which the next frame's is taken to match. A live frame released
- // by where its output starts would land with its tail past the floor, and the stretch accelerating
- // that away would only draw the next frame out sooner, playing through the hold.
+ // The duration of the latest output, which a frame still decoding is taken to match.
#frame = 0;
#waiters: Array<{ timestamp: number; resolve: () => void }> = [];
@@ -82,47 +85,30 @@ export class Backpressure {
return this.#floor;
}
- /**
- * Split `delay` between the floor and the hold. Returns true when the ring must park with the new
- * floor: a rise in the hold that the queued media does not cover yet holds playback until it does,
- * the way a deeper floor parks the ring until it refills, since video holds the extra delay on its
- * own. A floor update ends a park, so a ring already parked stays parked while the remaining hold
- * is uncovered.
- */
- setLatency(delay: Duration): boolean {
+ /** How far output lands in the ring behind the frame it was decoded from, in microseconds. */
+ get offset(): number {
+ return this.#offset;
+ }
+
+ /** Split `delay` between the floor and the hold. */
+ setLatency(delay: Duration): void {
this.#floor = ringSamples(this.#rate, delay);
this.#headroom = (this.#floor / this.#rate) * 1_000_000;
- const hold = ringHold(delay).asMicros();
- const rise = hold > this.#hold;
- this.#hold = hold;
-
- // Parks whatever the ring was last reported doing: a report of a stall may be stale, and
- // parking a ring that is still stalled changes nothing.
- const playhead = this.#start();
- this.#parked = (rise || this.#parked) && playhead !== undefined && !this.#covered(playhead + this.#headroom);
- if (this.#parked) this.#stalled = true;
+ this.#hold = ringHold(delay).asMicros();
this.#release();
- return this.#parked;
}
/** Note the ring's playhead and whether it is stalled, and release every frame now due. */
observe(playhead: number | undefined, stalled: boolean): void {
this.#playhead = playhead;
- this.#stalled = stalled;
+ if (playhead !== undefined && !stalled) this.#started = true;
this.#release();
}
- /**
- * Note the newest media timestamp queued ahead of the decoder, or undefined once none is. Returns
- * true when that covers the hold of a parked ring, which a floor update then resumes.
- */
- queued(newest: number | undefined): boolean {
+ /** Note the newest media timestamp queued ahead of the decoder, or undefined once none is. */
+ queued(newest: number | undefined): void {
this.#queued = newest;
- const playhead = this.#start();
- const resume = this.#parked && playhead !== undefined && this.#covered(playhead + this.#headroom);
- if (resume) this.#parked = false;
this.#release();
- return resume;
}
/**
@@ -183,9 +169,7 @@ export class Backpressure {
/** The ring was flushed for a new timeline: its playhead is unknown until the next frame anchors it. */
reset(): void {
this.#playhead = undefined;
- this.#anchor = undefined;
- this.#stalled = true;
- this.#parked = false;
+ this.#started = false;
this.#decoding = [];
this.#inserted = undefined;
this.#offset = 0;
@@ -215,38 +199,23 @@ export class Backpressure {
return this.#queued !== undefined && this.#queued >= timestamp + this.#hold;
}
- // The frame the ring plays from: its reported playhead, else where it anchored or will anchor,
- // mapped back to the frame its output came from, which is what the queued media covers.
- #start(): number | undefined {
- const start = this.#playhead ?? this.#inserted?.start;
- if (start !== undefined) return start + this.#offset;
- return this.#anchor;
- }
-
#admit(timestamp: number): boolean {
const due = this.#due(timestamp);
- if (due) {
- this.#anchor ??= timestamp;
- if (this.#buffered) this.#decoding.push(timestamp);
- }
+ if (due) this.#decoding.push(timestamp);
return due;
}
#due(timestamp: number): boolean {
const playhead = this.#playhead;
- if (!this.#stalled && playhead !== undefined) {
- const tail = this.#buffered ? 0 : this.#frame;
- if (timestamp + tail - this.#offset - this.#headroom <= playhead) return true;
- return !this.#buffered && this.#covered(timestamp);
- }
+ if (this.#started && playhead !== undefined) return this.#reach(playhead) - playhead < this.#headroom + WINDOW;
if (!this.#covered(timestamp)) return false;
- // A live ring sheds what overflows its floor, so it refills from everything that is due.
+ // A live ring is bounded only by its capacity until it starts, so it fills from everything due.
if (!this.#buffered) return true;
- // Refill while the PCM in the ring past its playhead, plus the span of frames still decoding, is
- // short of the floor. The ring un-stalls once it holds the floor, and output the decoder
- // discards never counts, so the refill cannot stop with the ring short of it.
+ // Fill while the PCM in the ring past its playhead, plus the span of frames still decoding, is
+ // short of the floor. The ring starts once it holds the floor, and output the decoder discards
+ // never counts, so the fill cannot stop with the ring short of it.
const inserted = this.#inserted;
const start = playhead ?? inserted?.start;
const buffered = inserted && start !== undefined ? Math.max(0, inserted.end - start) : 0;
@@ -254,6 +223,14 @@ export class Backpressure {
const pending = decoding.length > 0 ? decoding[decoding.length - 1] - decoding[0] : 0;
return buffered + pending < this.#headroom;
}
+
+ // Where what the ring was given ends, on its timeline: the newest frame still decoding, where its
+ // output lands, or else the PCM it holds.
+ #reach(playhead: number): number {
+ const landed = this.#inserted?.end ?? playhead;
+ const decoding = this.#decoding.at(-1);
+ return decoding === undefined ? landed : Math.max(landed, decoding + this.#frame - this.#offset);
+ }
}
/**
@@ -295,10 +272,20 @@ export class AudioBuffer {
// Backpressure runs off the playhead the worklet reports in its state messages.
#backpressure: Backpressure;
- // Bumped per reset or park. A state message posted before the worklet saw it reports the ring
- // playing on the old terms, which would open the gate.
+ // Bumped per reset. A state message posted before the worklet saw it reports the ring on the old
+ // timeline, which would open the gate.
#epoch = 0;
#playing?: Getter;
+ // How far audio may trail the shared clock before the ring skips ahead.
+ #bound: Duration;
+ // The shared clock less the context's time, in microseconds, as last sent. The context's time
+ // advances a render quantum at a time, so a reading lands up to a quantum late: the least offset is
+ // the reading taken nearest a quantum's start. It rises only slowly, as drift needs, unless it moves
+ // past any quantum, as a delay or reference change moves it.
+ #offset?: number;
+ // The shared clock's reference as last sent, in milliseconds. One that moved earlier jumped the
+ // clock ahead, which the ring skips rather than speeds through, as video skips the frames it passed.
+ #reference?: number;
#signals = new Effect();
@@ -309,17 +296,22 @@ export class AudioBuffer {
rate: number;
/** The target delay. The ring holds up to a fixed floor of it decoded; the rest stays encoded. */
delay: Duration;
- /** Anchor to the first frame and play through, holding the lookahead upstream. */
+ /** How far audio may trail the shared clock before the ring skips ahead. */
+ bound?: Duration;
+ /** Anchor to the first frame and hold the lookahead upstream until playback starts. */
buffered: boolean;
+ /** The shared clock the ring plays onto, and the reference it plays from. */
+ clock?: { now(): Duration | undefined; out: { reference: Getter } };
/** Hold consumption until the shared startup floor is ready. */
playing?: Getter;
},
) {
- const { channels, rate, buffered, playing } = props;
+ const { channels, rate, buffered, playing, clock } = props;
this.#playing = props.playing;
this.#worklet = worklet;
this.channels = channels;
this.rate = rate;
+ this.#bound = props.bound ?? Duration.ZERO;
this.#backpressure = new Backpressure(rate, buffered, props.delay);
@@ -327,8 +319,7 @@ export class AudioBuffer {
type: "init",
channels,
rate,
- latency: this.#latency(),
- buffered,
+ ...this.#terms(props.delay),
...(playing && { playing: playing.peek() }),
};
worklet.port.postMessage(msg);
@@ -351,6 +342,7 @@ export class AudioBuffer {
this.#silent.set(data.silent);
if (data.epoch === this.#epoch) {
this.#backpressure.observe(data.anchored ? data.timestamp : undefined, data.stalled);
+ if (clock) this.#clock(clock.now(), clock.out.reference.peek());
}
}
});
@@ -358,9 +350,51 @@ export class AudioBuffer {
worklet.port.start();
}
- // The floor as the worklet takes it.
- #latency(): number {
- return (this.#backpressure.floor / this.rate) * 1000;
+ // The terms as the worklet takes them.
+ #terms(delay: Duration): Omit {
+ return {
+ latency: (this.#backpressure.floor / this.rate) * 1000,
+ target: delay.asMillis(),
+ bound: this.#bound.asMillis(),
+ };
+ }
+
+ // Send where the shared clock stands, in the ring's coordinates, read together with the context's
+ // time so a late message misplaces neither. Each reading follows a state report, which lands just
+ // after the context renders a callback, when its time runs up to that callback ahead of the wall
+ // clock: the ring then trails the clock by up to one callback, the audio a callback pulls at once.
+ #clock(now: Duration | undefined, reference: number | undefined): void {
+ const time = this.#worklet.context.currentTime;
+ const last = this.#reference;
+ this.#reference = reference;
+ if (now === undefined) {
+ this.#offset = undefined;
+ const msg: Clock = { type: "clock", time };
+ this.#worklet.port.postMessage(msg);
+ return;
+ }
+ // A jump is a step of the clock, not where a reading landed, so the offset takes it whole.
+ const jumped = last !== undefined && reference !== undefined ? Math.max(0, last - reference) * 1000 : 0;
+ const offset = now.asMicros() - this.#backpressure.offset - time * 1_000_000;
+ const previous = this.#offset === undefined ? undefined : this.#offset + jumped;
+ this.#offset =
+ previous !== undefined && Math.abs(offset - previous) < JUMP ? Math.min(offset, previous + RISE) : offset;
+ const msg: Clock = {
+ type: "clock",
+ timestamp: this.#offset + time * 1_000_000,
+ time,
+ ...(jumped > 0 && { jumped }),
+ };
+ this.#worklet.port.postMessage(msg);
+ }
+
+ /**
+ * Note that a frame arrived whose media ends at `end`, on the decoder's timeline. How far the
+ * playhead trails it is the level the ring's stretch decides on, decoded or not.
+ */
+ received(end: Duration): void {
+ const msg: Received = { type: "received", end: end.asMicros() - this.#backpressure.offset };
+ this.#worklet.port.postMessage(msg);
}
/**
@@ -392,25 +426,26 @@ export class AudioBuffer {
}
/**
- * Update the target delay. The ring holds up to a fixed floor of it decoded, and the rest stays
- * encoded upstream until the media queued behind a frame covers it.
+ * Update the target delay, and how far audio may trail the shared clock before the ring skips
+ * ahead. The ring holds up to a fixed floor of the delay decoded, and the rest stays encoded
+ * upstream. A moved delay moves the clock, which the ring plays onto by stretching: slower onto a
+ * rise, faster onto a fall.
*/
- setLatency(delay: Duration): void {
- const park = this.#backpressure.setLatency(delay);
- const msg: Latency = { type: "latency", latency: this.#latency() };
- // One message, so the worklet never plays a quantum between the new floor and the park.
- if (park) msg.park = ++this.#epoch;
+ setLatency(delay: Duration, bound: Duration = this.#bound): void {
+ this.#bound = bound;
+ this.#backpressure.setLatency(delay);
+ // The clock moves with the delay, so its offset starts over with the next reading.
+ this.#offset = undefined;
+ const msg: Latency = { type: "latency", ...this.#terms(delay) };
this.#worklet.port.postMessage(msg);
}
/**
* Note the newest media timestamp queued ahead of the decoder, or undefined once none is. The
- * queue is what covers the part of the target held encoded, so it is what resumes a parked ring.
+ * queue is what covers the part of the target held encoded before playback starts.
*/
queued(newest: number | undefined): void {
- if (!this.#backpressure.queued(newest)) return;
- const msg: Latency = { type: "latency", latency: this.#latency() };
- this.#worklet.port.postMessage(msg);
+ this.#backpressure.queued(newest);
}
/**
@@ -460,6 +495,7 @@ export class AudioBuffer {
};
this.#worklet.port.postMessage(msg);
this.#backpressure.reset();
+ this.#offset = undefined;
}
/**
diff --git a/js/watch/src/audio/decoder.test.ts b/js/watch/src/audio/decoder.test.ts
index 064f0b65e4d..3392338a718 100644
--- a/js/watch/src/audio/decoder.test.ts
+++ b/js/watch/src/audio/decoder.test.ts
@@ -11,7 +11,7 @@ import { type Delay, Sync } from "../sync";
import { Decoder as VideoDecoder } from "../video/decoder";
import { Source as VideoSource } from "../video/source";
import { decoderConfigKey } from "./config";
-import type { Message, State } from "./render";
+import type { Message, State, Terms } from "./render";
import { AudioRingBuffer } from "./ring-buffer";
import { Source } from "./source";
@@ -30,6 +30,7 @@ let moduleLoaded: Promise = Promise.resolve();
class FakeContext extends EventTarget {
state: AudioContextState = "running";
+ readonly currentTime = 0;
readonly sampleRate: number;
readonly audioWorklet = { addModule: () => moduleLoaded };
readonly calls = { suspend: 0, resume: 0, close: 0 };
@@ -60,24 +61,26 @@ class FakeContext extends EventTarget {
let worklets: FakeWorklet[] = [];
// The render worklet's side of the port, synchronous: each message applies to a real ring, and
-// every quantum read reports the ring's state back, the way `render-worklet.ts` does.
+// every quantum read reports the ring's state back, the way `render-worklet.ts` does. It takes no
+// clock readings, so the ring decides on its buffer alone.
class FakeWorklet {
ring?: AudioRingBuffer;
+ // The terms the ring was last given, in ms.
+ terms?: Terms;
+ readonly context: FakeContext;
#epoch = 0;
#playing = true;
readonly port = Object.assign(new EventTarget(), {
postMessage: (msg: Message) => {
+ if (msg.type === "init" || msg.type === "latency")
+ this.terms = { latency: msg.latency, target: msg.target, bound: msg.bound };
if (msg.type === "init") {
this.ring = new AudioRingBuffer(msg);
this.#playing = msg.playing ?? true;
} else if (msg.type === "data") this.ring?.write(msg.timestamp, msg.data);
- else if (msg.type === "latency") {
- this.ring?.setLatency(msg.latency);
- if (msg.park !== undefined) {
- this.ring?.park();
- this.#epoch = msg.park;
- }
- } else if (msg.type === "truncate") this.ring?.truncate(msg.timestamp);
+ else if (msg.type === "received") this.ring?.received(msg.end);
+ else if (msg.type === "latency") this.ring?.setLatency(msg.latency, msg);
+ else if (msg.type === "truncate") this.ring?.truncate(msg.timestamp);
else if (msg.type === "end") this.ring?.end(msg.timestamp);
else if (msg.type === "reset") {
if (msg.next) this.ring?.next();
@@ -88,7 +91,8 @@ class FakeWorklet {
},
start() {},
});
- constructor() {
+ constructor(context: FakeContext) {
+ this.context = context;
worklets.push(this);
}
get stalled(): boolean {
@@ -997,6 +1001,28 @@ for (const cmaf of [false, true]) {
});
}
+// The gate keeps what the ring does not need yet encoded in the consumer, and that audio has arrived
+// all the same, so the level the stretch steers on reaches past it.
+it("counts the audio the consumer holds encoded toward the level", async () => {
+ const received = spyOn(AudioRingBuffer.prototype, "received");
+ const playback = await play(Duration.millis(1000));
+ try {
+ await playback.play(12);
+ const reached = () => Math.max(...received.mock.calls.map(([end]) => end as number));
+ // Past everything pushed, so only the consumer's ranges can reach this far.
+ const held = frameTimestamp + 100_000;
+ playback.queued([{ start: Duration.micros(frameTimestamp), end: Duration.micros(held) }]);
+ await microtasks();
+ const before = reached();
+ playback.queued([{ start: Duration.micros(frameTimestamp), end: Duration.micros(held + 500_000) }]);
+ await microtasks();
+ expect(reached() - before).toBe(500_000);
+ } finally {
+ playback.close();
+ await microtasks();
+ }
+});
+
describe("Decoder stalls", () => {
// Play a quantum, so the rendition is playing, then end it the given way and run the ring dry.
async function stopAfterPlaying(stop: (playback: Awaited>) => void) {
@@ -1318,7 +1344,9 @@ it("a missing live group re-primes without clearing decoded PCM or resetting Syn
const buffered = ring?.length;
await playback.hole();
expect(ring?.length).toBe(buffered);
- await playback.run(2000);
+ // A missing group is waited on for the subscription's max delay, as late audio could still
+ // arrive inside it, and skipped once the frames past it span that much.
+ await playback.run(4000);
expect(prime).toHaveBeenCalledTimes(1);
expect(reset).not.toHaveBeenCalled();
expect(playback.ring()).toBe(ring);
@@ -1702,16 +1730,9 @@ for (const cmaf of [false, true]) {
(buffered ? [0, 20, 40, 60, 80] : [140, 160]).map((ms) => start + ms * 1000),
);
- // A running live ring sheds old audio and lands on its floor once it reaches the fast
- // accelerate threshold, four times a 100 ms floor plus its 20 ms band; a buffered ring
- // plays it all.
- const ring = worklets.at(-1)?.ring;
- if (!ring) throw new Error("no running audio ring");
- ring.reset();
- const samples = [new Float32Array(4800), new Float32Array(4800)];
- for (let timestamp = 0; timestamp < 500_000; timestamp += 100_000) ring.write(timestamp, samples);
- expect(ring.length).toBe(samples[0].length * (buffered ? 5 : 1));
- expect(ring.skips).toBe(buffered ? 0 : 1);
+ // Either ring keeps what it receives and plays onto the clock. Buffered audio may trail it
+ // by the lookahead; live audio at a fixed delay by nothing the subscription still delivers.
+ expect(worklets.at(-1)?.terms).toEqual({ latency: 100, target: 100, bound: buffered ? 1000 : 0 });
} finally {
playback.close();
await microtasks();
diff --git a/js/watch/src/audio/decoder.ts b/js/watch/src/audio/decoder.ts
index 76da95dbf83..1b9b6c276be 100644
--- a/js/watch/src/audio/decoder.ts
+++ b/js/watch/src/audio/decoder.ts
@@ -286,6 +286,8 @@ export class Decoder {
// Count a received frame and whether it arrived too late for playout.
#receive(key: Key, frame: Container.ReceivedFrame, timestamp: Duration, late: boolean): void {
+ // How far the playhead trails what arrived, decoded or not, is the level the ring steers on.
+ this.#ring.peek()?.received(timestamp.add(this.#frame.peek() ?? Duration.ZERO));
const bytes = frame.payload.byteLength;
const arrived = arrivedAt(frame.arrival);
this.#tally.arrive(key, timestamp, arrived);
@@ -320,16 +322,24 @@ export class Decoder {
}
// A group the relay expires is never observed, so a subscription cut to the target would cap the
- // estimate at the target it already holds. The container consumer keeps the shared max delay as its
- // local skip, since it observes each frame before applying it, and carves out the warmup the
- // subscription adds once it has warmed, as Rust's does. See doc/concept/audio-jitter.md.
+ // estimate at the target it already holds. The container consumer skips at the same max delay,
+ // since it observes each frame before applying it, so audio the ring could still catch up on is
+ // delivered, and carves out the warmup the subscription adds once it has warmed, as Rust's does.
+ // See doc/concept/audio-jitter.md.
#runSubscribeMaxDelay(effect: Effect): void {
const delay = effect.get(this.sync.out.delay);
+ const max = this.#maxDelay(effect);
+ const warmup = Duration.millis(effect.get(this.source.out.config)?.warmup ?? 0);
+ this.#delay.set({ min: delay, max });
+ this.#subscribeMaxDelay.set(max.add(warmup));
+ }
+
+ // The subscription's max delay, less warmup: the shared max delay, at least the estimator's ceiling
+ // for "auto".
+ #maxDelay(effect: Effect): Duration {
const maxDelay = effect.get(this.sync.out.maxDelay);
const auto = effect.get(this.sync.in.delay) === "auto";
- const warmup = Duration.millis(effect.get(this.source.out.config)?.warmup ?? 0);
- this.#delay.set({ min: delay, max: maxDelay });
- this.#subscribeMaxDelay.set((auto ? maxDelay.max(AUTO_MAX_DELAY) : maxDelay).add(warmup));
+ return auto ? maxDelay.max(AUTO_MAX_DELAY) : maxDelay;
}
#runShape(effect: Effect): void {
@@ -402,6 +412,7 @@ export class Decoder {
rate: sampleRate,
delay: this.sync.out.delay.peek(),
buffered,
+ clock: this.sync,
playing: this.sync.playing,
});
effect.cleanup(() => ring.close());
@@ -494,10 +505,10 @@ export class Decoder {
const ring = effect.get(this.#ring);
if (!ring) return;
- // A rise parks playback until the queued media covers the new target, which is what keeps
- // audio in step with video. The measured target moves a bucket at a time, and a rise the
- // buffered slack already covers costs no silence at all.
- ring.setLatency(effect.get(this.sync.out.delay));
+ // The ring plays onto the shared clock, slowing down onto a rise and speeding up onto a fall,
+ // and skips only audio trailing it past what the subscription still delivers.
+ const delay = effect.get(this.sync.out.delay);
+ ring.setLatency(delay, this.#maxDelay(effect).saturatingSub(delay));
}
#runDecoder(effect: Effect): void {
@@ -870,12 +881,17 @@ export class Decoder {
const pending = inner.get(read);
const network = inner.get(consumer.buffered);
let newest = pending.newest;
+ // What the consumer holds has arrived too, so its newest end is how far the received audio
+ // reaches, though the gate keeps it encoded.
+ let arrived: Duration | undefined;
for (const { end } of network) {
const micro = end.asMicros();
if (newest === undefined || micro > newest) newest = micro;
+ if (arrived === undefined || end.gt(arrived)) arrived = end;
}
inner.set(this.#queued, Container.mergeBufferedRanges(network, pending.buffered));
ring.queued(newest);
+ if (arrived !== undefined) ring.received(arrived);
});
// Feed the arrival estimate into the playout target. It outlives this consumer, so the target
diff --git a/js/watch/src/audio/render-worklet.test.ts b/js/watch/src/audio/render-worklet.test.ts
index 2c7af5a6bec..e9fbbab923b 100644
--- a/js/watch/src/audio/render-worklet.test.ts
+++ b/js/watch/src/audio/render-worklet.test.ts
@@ -36,10 +36,11 @@ try {
}
}
-function renderer() {
+// A 100 ms floor at 1 kHz, with no clock until a test sends one.
+function renderer(target = 100) {
const render = new Render();
const send = (data: Message) => render.port.onmessage?.({ data } as MessageEvent);
- send({ type: "init", channels: 1, rate: 1000, latency: 100, buffered: true });
+ send({ type: "init", channels: 1, rate: 1000, latency: 100, target, bound: 0 });
return {
send,
sent: render.sent,
@@ -53,11 +54,14 @@ function renderer() {
describe("render dispatch", () => {
it("truncates the old track's tail without flushing the due prefix", () => {
- const { send, read } = renderer();
+ // A target that holds what arrives, so the stretch leaves every sample in place.
+ const { send, read } = renderer(200);
send({ type: "data", timestamp: 0, data: [new Float32Array(200).fill(1)] });
expect([...read(50)]).toEqual(Array(50).fill(1));
send({ type: "truncate", timestamp: 100_000 });
send({ type: "data", timestamp: 100_000, data: [new Float32Array(20).fill(2)] });
+ // The retired tail no longer counts toward the depth, so a target at what is left holds it.
+ send({ type: "latency", latency: 100, target: 70, bound: 0 });
const output = read(100);
// A short quantum fades its final 64 samples. The full prefix remains untouched and the
// retired tail is silent, so ignoring either message changes the output.
@@ -92,47 +96,43 @@ describe("render dispatch", () => {
]);
});
- // A covered ring parks and resumes only on the main thread's word: data decoded before the park
- // may still land. The echoed epoch is what tells the main thread that state reports no longer
- // predate the park.
- it("parks until the next floor update and echoes the park's epoch", () => {
- const { send, read, sent } = renderer();
- send({ type: "data", timestamp: 0, data: [new Float32Array(200).fill(1)] });
- expect([...read(20)]).toEqual(Array(20).fill(1));
- send({ type: "latency", latency: 100, park: 3 });
- expect([...read(20)]).toEqual(Array(20).fill(0));
- send({ type: "data", timestamp: 200_000, data: [new Float32Array(20).fill(2)] });
- expect([...read(20)]).toEqual(Array(20).fill(0));
- send({ type: "latency", latency: 100 });
- // It fades back in on the parked audio, not the new data.
- expect(read(20)[19]).toBe(1);
- read(20);
- expect(sent).toEqual([
- {
- type: "state",
- epoch: 3,
- timestamp: 60_000,
- stalled: false,
- anchored: true,
- underruns: 0,
- skips: 0,
- trimmed: 0,
- accelerated: 0,
- silent: false,
- },
- ]);
- });
-
- it("parks on a latency rise and preserves unread PCM", () => {
+ it("keeps playing through a latency rise, keeping unread PCM", () => {
const { send, read } = renderer();
send({ type: "data", timestamp: 0, data: [new Float32Array(100).fill(1)] });
expect([...read(20)]).toEqual(Array(20).fill(1));
- send({ type: "latency", latency: 200 });
- expect([...read(20)]).toEqual(Array(20).fill(0));
+ send({ type: "latency", latency: 200, target: 200, bound: 0 });
+ expect([...read(20)]).toEqual(Array(20).fill(1));
send({ type: "data", timestamp: 100_000, data: [new Float32Array(120).fill(2)] });
const output = read(100);
- expect(output[79]).toBe(1);
- expect([...output.slice(80)]).toEqual(Array(20).fill(2));
+ expect(output.slice(0, 60).every((value) => value === 1)).toBe(true);
+ expect(output.slice(-20).every((value) => value === 2)).toBe(true);
+ });
+
+ it("plays onto the clock it is sent, skipping only past the bound", () => {
+ const { send, read, sent } = renderer();
+ const time = Object.getOwnPropertyDescriptor(globalThis, "currentTime");
+ // The context's time, advanced a quantum per read as the clock is.
+ let now = 1;
+ const tick = () => {
+ read(20);
+ now += 0.02;
+ };
+ try {
+ Object.defineProperty(globalThis, "currentTime", { configurable: true, get: () => now });
+ send({ type: "data", timestamp: 0, data: [new Float32Array(1000).fill(1)] });
+ // Half a second behind a clock that allows none, so the next quantum lands on the clock.
+ send({ type: "clock", timestamp: 500_000, time: now });
+ for (let i = 0; i < 5; i++) tick();
+ expect(sent.at(-1)).toMatchObject({ timestamp: 600_000, skips: 1, trimmed: 500 });
+ // A latency update moves the clock, so the old reading no longer applies.
+ send({ type: "latency", latency: 100, target: 100, bound: 0 });
+ now += 1;
+ for (let i = 0; i < 5; i++) tick();
+ expect(sent.at(-1)).toMatchObject({ skips: 1, trimmed: 500 });
+ } finally {
+ if (time) Object.defineProperty(globalThis, "currentTime", time);
+ else Reflect.deleteProperty(globalThis, "currentTime");
+ }
});
});
diff --git a/js/watch/src/audio/render-worklet.ts b/js/watch/src/audio/render-worklet.ts
index ffcb5421f10..9b9d25982e9 100644
--- a/js/watch/src/audio/render-worklet.ts
+++ b/js/watch/src/audio/render-worklet.ts
@@ -1,5 +1,5 @@
import { ComfortNoise } from "./comfort";
-import type { Message, State } from "./render";
+import type { Clock, Message, State } from "./render";
import { AudioRingBuffer } from "./ring-buffer";
// Samples to fade over when playback stops or resumes short of a full quantum. A shortfall
@@ -16,6 +16,8 @@ class Render extends AudioWorkletProcessor {
#playing = true;
// Whether the previous quantum ended short, so the next one fades back in.
#short = false;
+ // The shared clock's latest reading, which the ring plays onto.
+ #clock?: Clock;
constructor() {
super();
@@ -28,14 +30,18 @@ class Render extends AudioWorkletProcessor {
this.#playing = msg.playing ?? true;
this.#underflow = 0;
this.#short = false;
+ this.#clock = undefined;
} else if (msg.type === "data") {
this.#ring?.write(msg.timestamp, msg.data);
+ } else if (msg.type === "received") {
+ this.#ring?.received(msg.end);
+ } else if (msg.type === "clock") {
+ this.#clock = msg;
+ if (msg.jumped) this.#ring?.jumped(msg.jumped);
} else if (msg.type === "latency") {
- this.#ring?.setLatency(msg.latency);
- if (msg.park !== undefined) {
- this.#ring?.park();
- this.#epoch = msg.park;
- }
+ this.#ring?.setLatency(msg.latency, msg);
+ // The clock moves with the delay, so a reading from before it would misplace audio.
+ this.#clock = undefined;
} else if (msg.type === "end") {
this.#ring?.end(msg.timestamp);
} else if (msg.type === "truncate") {
@@ -48,6 +54,8 @@ class Render extends AudioWorkletProcessor {
}
this.#epoch = msg.epoch;
this.#playing = msg.playing ?? true;
+ // A reset ends the timeline the clock was read on.
+ this.#clock = undefined;
} else if (msg.type === "playing" && msg.epoch === this.#epoch) {
this.#playing = msg.playing;
}
@@ -57,6 +65,10 @@ class Render extends AudioWorkletProcessor {
process(_inputs: Float32Array[][], outputs: Float32Array[][], _parameters: Record) {
const output = outputs[0];
const ring = this.#ring;
+ const clock = this.#clock;
+ ring?.clock(
+ clock?.timestamp !== undefined ? clock.timestamp + (currentTime - clock.time) * 1_000_000 : undefined,
+ );
const samplesRead = this.#playing ? (ring?.read(output) ?? 0) : 0;
// A declared silence between two buffered utterances plays as comfort noise, faded like any other.
const quiet = this.#playing ? ring?.quiet : undefined;
diff --git a/js/watch/src/audio/render.ts b/js/watch/src/audio/render.ts
index 66769614107..688d0bd9c1d 100644
--- a/js/watch/src/audio/render.ts
+++ b/js/watch/src/audio/render.ts
@@ -2,20 +2,30 @@
* Everything the main thread sends the render worklet over its port. Structured clone carries no
* clock classes, so latencies are plain milliseconds and timestamps plain microseconds.
*/
-export type Message = Init | Data | Latency | Reset | Truncate | Playing | End;
+export type Message = Init | Data | Latency | Reset | Truncate | Playing | End | Clock | Received;
/** Create the worklet's ring. */
-export interface Init {
+export interface Init extends Terms {
type: "init";
channels: number;
rate: number;
- latency: number;
- // Buffered mode: anchor to the first frame and play through. The ring holds the floor; the
- // lookahead above it is held back upstream (the main thread applies the backpressure).
- buffered: boolean;
playing?: boolean;
}
+/**
+ * What the ring holds and steers toward. Only the floor is decoded PCM: the rest of the target, and in
+ * buffered mode the lookahead, stays encoded upstream (the main thread applies the backpressure).
+ */
+export interface Terms {
+ // The floor held as decoded PCM, which playback starts on.
+ latency: number;
+ // The playout delay the shared clock plays at: the depth the stretch keeps, decoded or not.
+ target: number;
+ // How far audio may trail the shared clock before the ring skips ahead: the subscription's max
+ // delay less the playout delay.
+ bound: number;
+}
+
/** Flush the buffer and re-stall. */
export interface Reset {
type: "reset";
@@ -45,13 +55,30 @@ export interface Data {
timestamp: number;
}
-/** Latency update: the floor held as decoded PCM. */
-export interface Latency {
+/** Latency update: new terms, which also move the shared clock. */
+export interface Latency extends Terms {
type: "latency";
- latency: number;
- // When set, also hold playback, keeping what is buffered, until the next latency message without
- // it. Echoed like a reset's epoch, so state from before the park cannot report the ring playing.
- park?: number;
+}
+
+/**
+ * Where the shared clock stands, for the ring to time-stretch onto. The clock runs at real time, so one
+ * reading maps it onto the worklet's own time until the next.
+ */
+export interface Clock {
+ type: "clock";
+ // The clock's media time at context time `time`, in the ring's coordinates, unset while it has none.
+ timestamp?: number;
+ // The AudioContext's `currentTime` when the clock was read, in seconds.
+ time: number;
+ // How far the clock's reference moved earlier since the last reading, in microseconds, which the
+ // ring skips rather than speeds through.
+ jumped?: number;
+}
+
+/** Media arrived ending at `end`, in the ring's coordinates, decoded or not. */
+export interface Received {
+ type: "received";
+ end: number;
}
/** State update from the worklet back to the main thread. */
@@ -71,7 +98,7 @@ export interface State {
accelerated: number;
// Whether playback is at a declared end, reached or announced before any audio, and nothing has played since.
silent: boolean;
- // The epoch of the last reset or park the worklet applied.
+ // The epoch of the last reset the worklet applied.
epoch: number;
}
diff --git a/js/watch/src/audio/replay.test.ts b/js/watch/src/audio/replay.test.ts
index ead7c0aa033..f6b0ea660bd 100644
--- a/js/watch/src/audio/replay.test.ts
+++ b/js/watch/src/audio/replay.test.ts
@@ -1,7 +1,7 @@
import { afterEach, beforeEach, describe, expect, it, type Mock, spyOn } from "bun:test";
import { Duration } from "@moq/clock";
import type * as Catalog from "@moq/hang/catalog";
-import { type Arrival, type Options, QUANTUM, replay } from "./replay";
+import { type Arrival, type Options, QUANTUM, type Quantum, replay } from "./replay";
const RATE = 48000;
const CONFIG = { codec: "opus", sampleRate: RATE, numberOfChannels: 1 } as Catalog.AudioConfig;
@@ -68,7 +68,8 @@ describe("replay", () => {
});
it("counts each run-dry the way the player's stats do", async () => {
- // One frame 200 ms late against a 100 ms target: the ring runs dry once and re-stalls to refill.
+ // One frame 200 ms late against a 100 ms target: the ring runs dry once and re-stalls to refill,
+ // which counts the short quantum it ran dry in and the stall after it.
const trace = paced(10, (i) => (i === 250 ? 230 : 30)).sort((a, b) => a.at - b.at);
const counted = async (arrivals: Arrival[]) => {
let last = 0;
@@ -82,7 +83,8 @@ describe("replay", () => {
return last;
};
expect(await counted(paced(10, () => 30))).toBe(0);
- expect(await counted(trace)).toBe(1);
+ expect(await counted(trace)).toBeGreaterThan(0);
+ expect(await counted(trace)).toBeLessThanOrEqual(2);
});
it("holds newer groups behind a missing one until the max delay gives up on it", async () => {
@@ -153,10 +155,13 @@ async function lag(trace: Arrival[], options: Options, settle: number) {
return { lags, median: lags[lags.length >> 1], underruns };
}
-// Audio and video keep separate clocks that align once the delay covers the arrivals. Settling audio
-// later than the clock would trade lip sync for fewer underruns, so these hold the line: at steady
-// state the playhead rests on or just ahead of the clock, and a target short of the spread underruns
-// rather than drifting behind it.
+/** The shortest pitch period a seam moves by: 2.5 ms. A lag shorter than that cannot be corrected. */
+const PERIOD = 2.5;
+
+// Audio and video keep separate clocks that align once the delay covers the arrivals. The ring plays
+// onto the shared clock: in steady state the playhead rests within a pitch period of it, either way.
+// A buffer that runs low slows playback and leaves it behind the clock, only as far as the
+// subscription's max delay still delivers, which a fixed delay leaves none of.
describe("replay against the shared clock", () => {
let warn: Mock;
beforeEach(() => {
@@ -164,23 +169,23 @@ describe("replay against the shared clock", () => {
});
afterEach(() => warn.mockRestore());
- // The ring rests one frame ahead of the newest arrival's playout, as the trim that preceded the
- // time stretch left it, so audio leads the clock by up to that frame and never trails it.
for (const [name, lateness] of [
["evenly paced", () => 30],
["flushed five frames at a time", (i: number) => 30 + (4 - (i % 5)) * 20],
] as const) {
- it(`plays ${name} frames on the clock at auto, never behind it`, async () => {
+ it(`plays ${name} frames on the clock at auto`, async () => {
const options: Options = { rate: RATE, config: CONFIG, delay: "auto", duration: observed(60) };
- const { lags } = await lag(paced(60, lateness), options, 30_000);
+ const { lags, underruns } = await lag(paced(60, lateness), options, 30_000);
expect(lags.length).toBeGreaterThan(0);
- expect(lags[0]).toBeGreaterThanOrEqual(-25);
- expect(lags.at(-1)).toBeLessThanOrEqual(0);
+ expect(lags[0]).toBeGreaterThanOrEqual(-PERIOD);
+ expect(lags.at(-1)).toBeLessThanOrEqual(PERIOD);
+ expect(underruns).toBe(0);
});
}
- it("underruns a target short of the spread rather than settling behind the clock", async () => {
- // 80 ms of spread against a 40 ms target: every flush runs the ring dry.
+ it("underruns a fixed target short of the spread rather than trailing the clock", async () => {
+ // 80 ms of spread against a fixed 40 ms target: every flush runs the ring dry, and a fixed delay
+ // has no max delay past it to fall behind into.
const trace = paced(60, (i) => 30 + (4 - (i % 5)) * 20);
const options: Options = { rate: RATE, config: CONFIG, delay: Duration.millis(40), duration: observed(60) };
const { lags, median, underruns } = await lag(trace, options, 30_000);
@@ -295,7 +300,7 @@ it("plays an advertised flush span past the decoded floor at its delay without a
const filled = quantum.output.findLastIndex((v) => v !== 0) + 1;
if (start === undefined && filled > 0) start = quantum.at;
if (start !== undefined && filled < QUANTUM) short++;
- lag = quantum.at - quantum.timestamp;
+ lag = (quantum.clock ?? 0) - quantum.timestamp;
delay = quantum.delay;
}
expect(delay).toBe(5020);
@@ -303,6 +308,98 @@ it("plays an advertised flush span past the decoded floor at its delay without a
expect(start).toBeGreaterThanOrEqual(5020);
expect(start).toBeLessThan(5040);
expect(short).toBe(0);
- expect(lag).toBeGreaterThanOrEqual(5020);
- expect(lag).toBeLessThan(5040);
+ // Then onto the shared clock, which counts that lateness too.
+ expect(Math.abs(lag)).toBeLessThanOrEqual(PERIOD);
+});
+
+// The same rule plays a buffered player onto the clock: its lookahead keeps the level high, so it plays
+// faster whenever it trails the clock, and the container keeps every group, since audio on the clock
+// stays within the max delay of the newest arrival.
+describe("buffered replay against the shared clock", () => {
+ let warn: Mock;
+ beforeEach(() => {
+ warn = spyOn(console, "warn").mockImplementation(() => {});
+ });
+ afterEach(() => warn.mockRestore());
+
+ /** Plays `trace`, returning how far audio trails the clock once settled, and what was dropped. */
+ async function play(trace: Arrival[], options: Options, settled: (at: number) => boolean) {
+ const lags: number[] = [];
+ let heard = 0;
+ let last: Quantum | undefined;
+ for await (const quantum of replay(trace, options)) {
+ heard += quantum.output.filter((v) => v !== 0).length;
+ if (settled(quantum.at) && !quantum.stalled && quantum.clock !== undefined) {
+ lags.push(quantum.clock - quantum.timestamp);
+ }
+ last = quantum;
+ }
+ if (!last) throw new Error("nothing played");
+ return {
+ min: Math.min(...lags),
+ max: Math.max(...lags),
+ skips: last.skips,
+ trimmed: last.trimmed,
+ accelerated: last.accelerated,
+ // Frames the consumer passed over rather than delivered.
+ lost: trace.length - last.delivered,
+ heard: (heard * 1000) / RATE,
+ };
+ }
+
+ it("catches up after a stall", async () => {
+ // Five seconds in, 400 ms of frames are held and land at once.
+ const trace = paced(30, (i) => 30 + (i >= 250 && i < 270 ? (270 - i) * 20 : 0)).sort((a, b) => a.at - b.at);
+ const options: Options = {
+ rate: RATE,
+ config: CONFIG,
+ delay: Duration.millis(100),
+ lookahead: 500,
+ duration: observed(30),
+ };
+ // NetEq's accelerate takes one pitch period per stretch, so this 2.5 ms tone gains about 30 ms a
+ // second: the stall's lag is gone well before 20 s.
+ const result = await play(trace, options, (at) => (at >= 2000 && at < 5000) || at >= 20_000);
+ expect(result.min).toBeGreaterThanOrEqual(-PERIOD);
+ expect(result.max).toBeLessThanOrEqual(PERIOD);
+ expect(result).toMatchObject({ skips: 0, trimmed: 0, lost: 0 });
+ expect(warn).not.toHaveBeenCalled();
+ });
+
+ it("follows a publisher clock running 1% fast at auto", async () => {
+ const trace = paced(60, (i) => 30 - (i * 20) / 101);
+ const options: Options = {
+ rate: RATE,
+ config: CONFIG,
+ delay: "auto",
+ lookahead: 500,
+ duration: observed(60),
+ };
+ const result = await play(trace, options, (at) => at >= 5000);
+ expect(result.min).toBeGreaterThanOrEqual(-PERIOD);
+ expect(result.max).toBeLessThan(23);
+ expect(result).toMatchObject({ skips: 0, trimmed: 0, lost: 0 });
+ expect(warn).not.toHaveBeenCalled();
+ });
+
+ it("plays every word of an utterance that stalls mid-burst, catching up after", async () => {
+ // Twenty seconds of speech generated four times faster than real time, held for 3.5 s after its
+ // first two seconds, so playback runs dry and resumes about two seconds behind the clock.
+ const trace = paced(20, (i) => (i < 100 ? 30 : 3500) - i * 15);
+ const options: Options = {
+ rate: RATE,
+ config: CONFIG,
+ delay: Duration.millis(100),
+ lookahead: 30_000,
+ duration: 22_000,
+ };
+ const result = await play(trace, options, (at) => at >= 15_000);
+ expect(result.min).toBeGreaterThanOrEqual(-PERIOD);
+ expect(result.max).toBeLessThanOrEqual(PERIOD);
+ expect(result).toMatchObject({ skips: 0, trimmed: 0, lost: 0 });
+ // Every frame played: what was heard and what the stretch shortened cover the utterance. A
+ // slow-down only adds to what is heard.
+ expect(Math.round(result.heard + result.accelerated)).toBeGreaterThanOrEqual(20_000);
+ expect(warn).not.toHaveBeenCalled();
+ });
});
diff --git a/js/watch/src/audio/replay.ts b/js/watch/src/audio/replay.ts
index 5ad24a3f5f7..3ff4f73f19a 100644
--- a/js/watch/src/audio/replay.ts
+++ b/js/watch/src/audio/replay.ts
@@ -24,7 +24,7 @@ import { Derived, Effect, Signal } from "@moq/signals";
import { type Delay, Sync } from "../sync";
import { Backpressure } from "./buffer";
import { frameDuration } from "./config";
-import { target } from "./latency";
+import { AUTO_MAX_DELAY, ringHold, target } from "./latency";
import { AudioRingBuffer } from "./ring-buffer";
/** An AudioWorklet render quantum, in frames. */
@@ -60,6 +60,8 @@ export interface Options {
rate: number;
/** The configured delay, as the element takes it. "instant" plays no audio, so it has no replay. */
delay: Exclude;
+ /** The configured lookahead, in ms: nonzero plays in buffered mode, as the decoder does. */
+ lookahead?: number;
/** The rendition as the catalog described it: its advertised jitter, delay, and codec. */
config: Catalog.AudioConfig;
/**
@@ -71,6 +73,8 @@ export interface Options {
/** One render quantum, as the worklet produced it. */
export interface Quantum {
+ /** Frames the decoder received so far. */
+ delivered: number;
/** When it finished rendering, on the trace's clock, in ms. */
at: number;
/** The quantum: what the ring supplied at the front, zeros after it. Reused between yields. */
@@ -137,11 +141,17 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
ends.set(arrival.group, i);
});
- const sync = new Sync({ delay: options.delay });
+ const lookahead = Duration.millis(options.lookahead ?? 0);
+ const buffered = !lookahead.isZero();
+ const sync = new Sync({ delay: options.delay, lookahead });
const track = new Track.Producer("audio");
+ // The subscription's max delay, as the decoder sets it, which the consumer skips at too.
+ const maxDelay = new Derived([sync.out.maxDelay], (max) =>
+ options.delay === "auto" ? max.max(AUTO_MAX_DELAY) : max,
+ );
const consumer = new Container.Consumer(track.subscribe({ delay: { max: WIRE_MAX_DELAY } }), {
format: new Container.Legacy.Format("audio"),
- delay: new Derived([sync.out.delay, sync.out.maxDelay], (min, max) => ({ min, max })),
+ delay: new Derived([sync.out.delay, maxDelay], (min, max) => ({ min, max })),
});
// Match the decoder: a codec constant until a delivered packet states its own duration.
@@ -155,28 +165,50 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
delay: config.delay !== undefined ? Duration.millis(config.delay) : undefined,
}),
);
- const audio = sync.track({ target: playout });
+ // Buffered startup holds playback until the decoded span, plus whatever of the hold is queued
+ // encoded past it, covers the delay, the way the decoder reports its startup span.
+ const decoded = new Signal<{ start: Duration; end: Duration } | null>(null);
+ const newest = new Signal(undefined);
+ const startup = buffered
+ ? signals.computed((effect) => {
+ const span = effect.get(decoded);
+ if (!span) return null;
+ const queued = Duration.micros(effect.get(newest) ?? 0).saturatingSub(span.end);
+ return { start: span.start, end: span.end.add(queued.min(ringHold(effect.get(sync.out.delay)))) };
+ })
+ : undefined;
+ const audio = sync.track({ target: playout, startup });
signals.cleanup(() => audio.close());
await settle();
let delay = sync.out.delay.peek();
+ const terms = () => ({ target: delay.asMillis(), bound: maxDelay.peek().saturatingSub(delay).asMillis() });
// The decode gate, and the worklet's ring on the floor it sets, the way `AudioBuffer` drives both.
- const gate = new Backpressure(rate, false, delay);
- const ring = new AudioRingBuffer({ rate, channels: 1, latency: millis(rate, gate.floor) });
+ const gate = new Backpressure(rate, buffered, delay);
+ const ring = new AudioRingBuffer({ rate, channels: 1, latency: millis(rate, gate.floor), ...terms() });
const observe = () => gate.observe(ring.anchored ? ring.timestamp : undefined, ring.stalled);
// Arrived but not yet decoded media, read or still in the consumer, is what covers the part of the
- // target held encoded, and what resumes a parked ring.
+ // target held encoded before playback starts.
// In microseconds, as the gate takes them.
const read = new Signal(undefined);
+ // What the consumer holds has arrived too, so its newest end is how far the received audio reaches.
signals.run((effect) => {
- let newest = effect.get(read);
- for (const { end } of effect.get(consumer.buffered)) newest = Math.max(newest ?? 0, Math.round(end.asMicros()));
- if (gate.queued(newest)) ring.setLatency(millis(rate, gate.floor));
+ let queued = effect.get(read);
+ let arrived: number | undefined;
+ for (const { end } of effect.get(consumer.buffered)) {
+ const micro = Math.round(end.asMicros());
+ queued = Math.max(queued ?? 0, micro);
+ arrived = Math.max(arrived ?? 0, micro);
+ }
+ newest.set(queued);
+ gate.queued(queued);
+ if (arrived !== undefined) ring.received(arrived);
});
// The decoder's read loop, with the codec taken out. Traces record no declared ends, so a gap is
// the only codec epoch a replay meets. A failure is rethrown at the next quantum.
+ let delivered = 0;
const delivery = Promise.withResolvers();
if (options.decodeAfter === undefined) delivery.resolve();
let failure: { error: unknown } | undefined;
@@ -187,6 +219,7 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
const next = await consumer.next();
if (!next) return;
if (next.kind !== "frame") continue;
+ delivered++;
const { timestamp: media, arrival } = next.frame;
const timestamp = media.asMicros();
@@ -195,6 +228,8 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
const entry = recorded.get(timestamp);
if (!entry) throw new Error(`the consumer delivered an unrecorded frame at ${timestamp} us`);
frame.set(Duration.millis(entry.duration));
+ // The decoder tells the ring what arrived before it waits to decode it.
+ ring.received(timestamp + micros(entry.duration));
await gate.wait(timestamp, abort.signal);
if (abort.signal.aborted) return;
const pcm = new Float32Array(Math.round((entry.duration * rate) / 1000)).fill(0.5);
@@ -202,6 +237,7 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
// The decoder reports each output to the gate the way `AudioBuffer` and `Decoder.#emit` do.
const end = timestamp + micros((pcm.length / rate) * 1000);
gate.inserted(timestamp, end);
+ decoded.update((span) => ({ start: span?.start ?? media, end: Duration.micros(end) }));
gate.decoded(end, end);
}
})().catch((error: unknown) => {
@@ -212,6 +248,9 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
const step = (QUANTUM / rate) * 1000;
const output = new Float32Array(QUANTUM);
let next = 0;
+ // The shared clock's reference as the worklet last read it. One that moved earlier jumped the clock
+ // ahead, which the ring skips, the way `AudioBuffer` reports it.
+ let reference: number | undefined;
// The consumer stamps each arrival on the monotonic clock, which the estimator measures, so a
// manual clock reads the trace's time instead until the replay finishes.
@@ -267,15 +306,23 @@ export async function* replay(trace: Arrival[], options: Options): AsyncGenerato
if (!resolved.eq(delay)) {
delay = resolved;
observe();
- const park = gate.setLatency(delay);
- ring.setLatency(millis(rate, gate.floor));
- if (park) ring.park();
+ gate.setLatency(delay);
+ ring.setLatency(millis(rate, gate.floor), terms());
}
+ // The worklet reads nothing until startup releases it, then the shared clock as of the
+ // quantum's start, mapped onto its own time.
+ const shared = sync.now();
+ const moved = sync.out.reference.peek();
+ if (reference !== undefined && moved !== undefined && moved < reference)
+ ring.jumped((reference - moved) * 1000);
+ reference = moved;
+ ring.clock(shared !== undefined ? (shared.asMillis() - step) * 1000 : undefined);
output.fill(0);
- ring.read([output]);
+ if (sync.playing.peek()) ring.read([output]);
observe();
yield {
+ delivered,
at: now,
output,
stalled: ring.stalled,
diff --git a/js/watch/src/audio/ring-buffer.test.ts b/js/watch/src/audio/ring-buffer.test.ts
index 70e8b22b1ef..3e1a9baa04f 100644
--- a/js/watch/src/audio/ring-buffer.test.ts
+++ b/js/watch/src/audio/ring-buffer.test.ts
@@ -1,11 +1,21 @@
import { describe, expect, it } from "bun:test";
import { AudioRingBuffer } from "./ring-buffer";
+// Rings that play exactly what they hold: on a shared clock pinned to the playhead with no depth to
+// keep, the stretch has nothing to steer toward, so every window holds.
+const exacts = new WeakSet();
+function exact(props: { rate: number; channels: number; latency: number }): AudioRingBuffer {
+ const buffer = new AudioRingBuffer({ ...props, target: 0 });
+ exacts.add(buffer);
+ return buffer;
+}
+
function read(buffer: AudioRingBuffer, samples: number, channelCount = 2): Float32Array[] {
const output: Float32Array[] = [];
for (let i = 0; i < channelCount; i++) {
output.push(new Float32Array(samples));
}
+ if (exacts.has(buffer)) buffer.clock(buffer.timestamp);
const samplesRead = buffer.read(output);
// Return the output arrays with only the read samples
if (samplesRead < samples) {
@@ -103,10 +113,10 @@ describe("writing data", () => {
}
});
- // These drain the ring below its floor without refilling it, which a live ring stretches, so they
- // run buffered, which plays exactly what the ring holds.
+ // These drain the ring below its floor without refilling it, which the stretch slows down for, so
+ // they run exact.
it("should handle late-arriving samples (out-of-order writes)", () => {
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100, buffered: true });
+ const buffer = exact({ rate: 1000, channels: 1, latency: 100 });
// Fill buffer to exit stalled mode
write(buffer, 0, 100, { channels: 1, value: 0.0 });
@@ -143,7 +153,7 @@ describe("writing data", () => {
});
it("should discard samples that are too old", () => {
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100, buffered: true });
+ const buffer = exact({ rate: 1000, channels: 2, latency: 100 });
// Exit stalled mode by filling buffer
write(buffer, 0, 100, { channels: 2, value: 0.0 });
@@ -168,10 +178,10 @@ describe("writing data", () => {
});
});
-// Buffered, for the same reason as the writes above.
+// Exact, for the same reason as the writes above.
describe("reading data", () => {
it("should read available data", () => {
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100, buffered: true });
+ const buffer = exact({ rate: 1000, channels: 2, latency: 100 });
// Exit stalled mode by filling the buffer
write(buffer, 0, 100, { channels: 2, value: 0.0 });
@@ -209,7 +219,7 @@ describe("reading data", () => {
});
it("should handle partial reads", () => {
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100, buffered: true });
+ const buffer = exact({ rate: 1000, channels: 2, latency: 100 });
// Exit stalled mode by filling the buffer
write(buffer, 0, 100, { channels: 2, value: 0.0 });
@@ -267,8 +277,8 @@ describe("stall behavior", () => {
describe("ring buffer wrapping", () => {
it("should wrap around when buffer is full", () => {
- // A 1000-sample ring, buffered so it plays through everything it holds.
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100, buffered: true });
+ // A 1000-sample ring, exact so it plays through everything it holds.
+ const buffer = exact({ rate: 1000, channels: 1, latency: 100 });
write(buffer, 0, 600, { channels: 1, value: 1.0 });
expect(read(buffer, 500, 1)[0].length).toBe(500);
@@ -283,7 +293,7 @@ describe("ring buffer wrapping", () => {
describe("multi-channel handling", () => {
it("should handle stereo data correctly", () => {
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 2, latency: 100, buffered: true });
+ const buffer = exact({ rate: 1000, channels: 2, latency: 100 });
// Exit stalled mode by filling buffer
write(buffer, 0, 100, { channels: 2, value: 0.5 });
@@ -363,14 +373,13 @@ describe("setLatency", () => {
expect(buffer.stalled).toBe(true);
});
- // Buffered playback promises to play through everything queued, so a lower floor must not drop
- // the oldest unread audio the way a live ring does.
- it("keeps every unread sample when a buffered ring's floor drops", () => {
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100, buffered: true });
+ // A lower floor moves the clock, which the ring speeds up onto, so it drops no unread audio.
+ it("keeps every unread sample when the floor drops", () => {
+ const buffer = exact({ rate: 1000, channels: 1, latency: 100 });
write(buffer, 0, 100, { channels: 1, value: 1.0 });
write(buffer, 100, 80, { channels: 1, value: 2.0 });
- buffer.setLatency(30);
+ buffer.setLatency(30, { target: 0, bound: 0 });
write(buffer, 180, 20, { channels: 1, value: 3.0 });
expect(buffer.length).toBe(200);
expect(buffer.timestamp / 1000).toBe(0);
@@ -386,27 +395,6 @@ describe("setLatency", () => {
});
});
-describe("park", () => {
- it("holds a covered ring until the next floor update, keeping what it buffered", () => {
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 50 });
- write(buffer, 0, 100, { channels: 1, value: 1.0 });
- expect(read(buffer, 10, 1)[0].length).toBe(10);
-
- buffer.park();
- expect(buffer.stalled).toBe(true);
- expect(read(buffer, 10, 1)[0].length).toBe(0);
-
- // Output decoded before the park can still land, so a write does not resume it.
- write(buffer, 100, 10, { channels: 1, value: 2.0 });
- expect(buffer.stalled).toBe(true);
-
- // The main thread knows when the media it holds encoded covers the target.
- buffer.setLatency(50);
- expect(buffer.stalled).toBe(false);
- expect(read(buffer, 10, 1)[0]).toEqual(new Float32Array(10).fill(1.0));
- });
-});
-
describe("decoded rate must match the ring (#2352)", () => {
// The ring maps a frame's timestamp to a sample index using its own rate, so the ring must run at
// the rate the decoder actually output. A 20ms Opus frame decodes to 960 samples (48kHz); if the
@@ -451,32 +439,35 @@ describe("live anchoring", () => {
});
describe("re-buffer", () => {
- it("re-stalls and refills to the target after running dry", () => {
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 40 });
+ it("re-stalls after running dry and resumes on a cushion, slowing down to rebuild", () => {
+ const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100 });
- write(buffer, 0, 20, { channels: 1, value: 1.0 });
- write(buffer, 20, 20, { channels: 1, value: 1.0 });
+ write(buffer, 0, 100, { channels: 1, value: 1.0 });
expect(buffer.stalled).toBe(false);
// Drain it.
- expect(read(buffer, 40, 1)[0].length).toBe(40);
+ expect(read(buffer, 100, 1)[0].length).toBe(100);
expect(buffer.length).toBe(0);
- // The next read finds nothing and parks playback rather than handing the worklet a quantum
+ // The next read finds nothing and stalls playback rather than handing the worklet a quantum
// of silence and trying again on the next one.
expect(read(buffer, 20, 1)[0].length).toBe(0);
expect(buffer.stalled).toBe(true);
expect(buffer.underruns).toBe(1);
- // A single chunk is not the target, so playback stays parked.
- write(buffer, 40, 20, { channels: 1, value: 2.0 });
+ // The cushion is one 30 ms window and a 20 ms read, not the 100 ms floor.
+ write(buffer, 100, 40, { channels: 1, value: 2.0 });
expect(buffer.stalled).toBe(true);
expect(read(buffer, 20, 1)[0].length).toBe(0);
-
- // Reaching the target resumes it, and nothing buffered in the meantime was lost.
- write(buffer, 60, 20, { channels: 1, value: 2.0 });
+ write(buffer, 140, 20, { channels: 1, value: 2.0 });
expect(buffer.stalled).toBe(false);
- expect(read(buffer, 40, 1)[0].length).toBe(40);
+
+ // It resumed 60 ms short of its target, so it slows down: 60 ms of audio outlasts 60 ms of reads,
+ // and nothing buffered in the meantime was lost.
+ expect(read(buffer, 60, 1)[0].length).toBe(60);
+ expect(buffer.length).toBeGreaterThan(0);
+ expect(buffer.timestamp / 1000).toBeLessThan(160);
+ expect(buffer.underruns).toBe(1);
});
it("counts a quantum it could only partly fill as an underrun", () => {
@@ -499,7 +490,7 @@ describe("re-buffer", () => {
expect(buffer.underruns).toBe(1);
});
- it("does not count an underrun while parked", () => {
+ it("does not count an underrun before it starts", () => {
const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 40 });
read(buffer, 20, 1);
read(buffer, 20, 1);
@@ -599,21 +590,26 @@ describe("declared end", () => {
expect(buffer.underruns).toBe(1);
});
- // The declared silence between them must not count toward the live band and push the unplayed
- // tail out; once the tail has played, the band catches up by skipping the silence instead.
- it("keeps the end's tail when the next utterance starts after a gap", () => {
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 40 });
+ // A playhead 30 ms behind the clock plays the end's tail as received; only then does it catch up,
+ // by skipping that much of the declared silence, which drops no audio.
+ it("keeps the end's tail and skips the declared silence after it by the lag", () => {
+ const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 40, bound: 1000 });
write(buffer, 0, 40, { channels: 1, value: 1 });
+ buffer.clock(30_000);
expect(read(buffer, 20, 1)[0].length).toBe(20);
buffer.end(40_000);
buffer.next();
write(buffer, 100, 20, { channels: 1, value: 2 });
+ buffer.clock(50_000);
expect([...read(buffer, 20, 1)[0]]).toEqual(Array(20).fill(1));
write(buffer, 120, 20, { channels: 1, value: 2 });
- expect([...read(buffer, 40, 1)[0]]).toEqual(Array(40).fill(2));
+ buffer.clock(70_000);
+ expect([...read(buffer, 40, 1)[0]]).toEqual([...Array(30).fill(0), ...Array(10).fill(2)]);
expect(buffer.underruns).toBe(0);
+ expect(buffer.skips).toBe(0);
+ expect(buffer.trimmed).toBe(0);
});
it("reports a declared silence across reads until the next utterance plays", () => {
@@ -676,9 +672,10 @@ describe("declared end", () => {
});
});
-describe("buffered mode", () => {
+// Backpressure keeps a ring near its floor, but nothing in the ring depends on it.
+describe("holding more than its floor", () => {
function createBuffered(latency: number) {
- return new AudioRingBuffer({ rate: 1000, channels: 1, latency: latency, buffered: true });
+ return exact({ rate: 1000, channels: 1, latency });
}
it("anchors to the first frame and un-stalls at the latency target", () => {
@@ -690,8 +687,7 @@ describe("buffered mode", () => {
});
it("plays frames in order within the floor-sized ring without skipping ahead", () => {
- // 600ms floor at 1000Hz -> 1200-sample ring, comfortably holding the 1000-sample utterance
- // the backpressure loop would feed it. A non-buffered ring would have skipped to write-latency.
+ // A 1000-sample utterance in a ring that holds it.
const buffer = createBuffered(600);
for (let i = 0; i < 10; i++) {
write(buffer, 2000 + i * 100, 100, { channels: 1, value: (i + 1) / 10 });
@@ -763,11 +759,10 @@ describe("buffered mode", () => {
});
describe("latency increase", () => {
- // Video holds a deeper floor on its own, so audio has to park for the difference or it runs
- // ahead. Flushing instead cost the whole floor in silence for every rise, however small.
- it("parks a playing ring until the deeper floor refills, keeping what it buffered", () => {
- // Buffered, so it plays its floor down without the expand a live ring stretches it with.
- const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100, buffered: true });
+ // A deeper target moves the clock later, which the ring slows down onto rather than stopping for:
+ // parking cost a gap of silence for every rise, however small.
+ it("keeps playing through a deeper floor, keeping what it buffered", () => {
+ const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100 });
// Fill to the floor so playback starts, then play half of it.
write(buffer, 0, 100, { channels: 1, value: 1.0 });
@@ -775,16 +770,8 @@ describe("latency increase", () => {
read(buffer, 50, 1);
buffer.setLatency(110);
- expect(buffer.stalled).toBe(true);
- expect(buffer.length).toBe(50);
-
- // Refilling to the 110-sample floor takes 60 more samples, not the whole floor.
- write(buffer, 100, 59, { channels: 1, value: 2.0 });
- expect(buffer.stalled).toBe(true);
- write(buffer, 159, 1, { channels: 1, value: 2.0 });
expect(buffer.stalled).toBe(false);
-
- // Playback resumes where it parked rather than at the refill.
+ expect(buffer.length).toBe(50);
expect(buffer.timestamp / 1000).toBe(50);
expect(read(buffer, 1, 1)[0][0]).toBe(1.0);
});
@@ -798,15 +785,12 @@ describe("latency increase", () => {
expect(buffer.stalled).toBe(false);
});
- it("resumes a refill once a shallower floor is already covered", () => {
+ it("starts once a lowered floor is already covered", () => {
const buffer = new AudioRingBuffer({ rate: 1000, channels: 1, latency: 100 });
- write(buffer, 0, 100, { channels: 1, value: 1.0 });
- read(buffer, 50, 1);
-
- buffer.setLatency(110);
+ write(buffer, 0, 60, { channels: 1, value: 1.0 });
expect(buffer.stalled).toBe(true);
- // No further frame arrives, as when the source stops, so only the lower floor can resume it.
+ // No further frame arrives, as when the source stops, so only the lower floor can start it.
buffer.setLatency(50);
expect(buffer.stalled).toBe(false);
expect(read(buffer, 1, 1)[0][0]).toBe(1.0);
@@ -835,22 +819,25 @@ describe("time stretch", () => {
// period, 2.5 ms or 120 frames.
const RATE = 48000;
const QUANTUM = 128;
- const live = (latency: number) => new AudioRingBuffer({ rate: RATE, channels: 1, latency: latency });
+ const live = (latency: number, terms: { bound?: number } = {}) =>
+ new AudioRingBuffer({ rate: RATE, channels: 1, latency, ...terms });
const frames = (ms: number) => (RATE * ms) / 1000;
- // Slowing down only ever settles audio behind the shared clock, so a ring short of its floor plays
- // what it holds at its own pace and runs dry: the underrun is the cost of arrivals running late.
- it("never expands a ring short of its floor, so it plays only what it holds before running dry", () => {
+ // Without a clock, the target is where the playhead belongs, so audio landing 40 ms short of it
+ // plays slower: the ring stretches what it holds rather than run dry on time.
+ it("slows down once arrivals land short of its target", () => {
const buffer = live(100);
write(buffer, 0, frames(100), { value: 0.5 });
+ read(buffer, frames(60), 1);
+ write(buffer, 100, frames(20), { value: 0.5 });
let played = 0;
for (;;) {
const [output] = read(buffer, QUANTUM, 1);
played += output.length;
if (output.length < QUANTUM) break;
}
- expect(played).toBe(frames(100));
- expect(buffer.underruns).toBe(1);
+ expect(played).toBeGreaterThan(frames(60));
+ expect(buffer.accelerated).toBe(0);
});
it("drains the tail before a declared end as received, since running dry there is silence", () => {
@@ -929,9 +916,9 @@ describe("time stretch", () => {
expect(buffer.underruns).toBe(0);
});
- // A floor short of the arrival spread underruns, and that is the cost of the floor: the ring must
- // not grow its depth to stop underrunning, which only plays everything later.
- it("underruns every burst that outruns its floor rather than settling later", async () => {
+ // A floor short of the arrival spread underruns, and that is the cost of the floor: each burst
+ // lands above the target, which leaves nothing to slow down for.
+ it("underruns every burst that outruns its target", async () => {
const buffer = live(60);
write(buffer, 0, frames(60), { value: 0.5 });
const { peaks } = await converge(buffer, frames(60), 8, 100);
@@ -940,28 +927,29 @@ describe("time stretch", () => {
expect(buffer.underruns).toBeGreaterThanOrEqual(70);
});
- it("fast-accelerates a ring four times past the band, even straight after a stretch", async () => {
+ it("fast-accelerates a ring four times past its high limit, even straight after a stretch", async () => {
const buffer = live(100);
write(buffer, 0, frames(600), { value: 0.5 });
await converge(buffer, frames(600), 1);
- // Fast accelerate removes up to 15 ms of every 30 ms window until the ring is under 480 ms.
- expect(buffer.length).toBeLessThan(frames(480));
+ // Fast accelerate removes up to 15 ms of every 30 ms window until the ring is under 400 ms.
+ expect(buffer.length).toBeLessThan(frames(400 + 20));
});
- it("drops the staged window with the audio a re-anchor skips", () => {
+ // A playhead trailing the clock past its bound and a window skips back to the bound, dropping the
+ // staged window with the audio it passes.
+ it("drops the staged window with the audio a skip passes", () => {
const buffer = live(100);
write(buffer, 0, frames(100), { value: 0.25 });
read(buffer, QUANTUM, 1);
- // 400 ms more crosses the 480 ms fast threshold with a 30 ms window still staged.
write(buffer, 100, frames(400), { value: 0.5 });
+ buffer.clock(400_000);
+ const [output] = read(buffer, QUANTUM, 1);
expect(buffer.skips).toBe(1);
- expect(buffer.length).toBe(frames(100));
- expect(buffer.timestamp / 1000).toBeCloseTo(400, 6);
+ expect(buffer.timestamp / 1000).toBeCloseTo(400 + (QUANTUM * 1000) / RATE, 6);
// Everything before the new playhead but the quantum that played.
expect(buffer.trimmed).toBeCloseTo(400 - (QUANTUM * 1000) / RATE, 6);
expect(buffer.accelerated).toBe(0);
- const [output] = read(buffer, QUANTUM, 1);
expect(output.length).toBe(QUANTUM);
expect(output.every((sample) => sample === 0.5)).toBe(true);
});
@@ -970,11 +958,13 @@ describe("time stretch", () => {
const buffer = live(100);
write(buffer, 0, frames(100), { value: 0.25 });
read(buffer, QUANTUM, 1);
- // A 100 ms gap, then enough to cross the 480 ms fast threshold and re-anchor at 410 ms.
+ // A 100 ms gap, then audio past it, with the clock at 410 ms.
write(buffer, 200, frames(10), { value: 0.5 });
write(buffer, 210, frames(300), { value: 0.5 });
+ buffer.clock(410_000);
+ read(buffer, QUANTUM, 1);
expect(buffer.skips).toBe(1);
- expect(buffer.timestamp / 1000).toBeCloseTo(410, 6);
+ expect(buffer.timestamp / 1000).toBeCloseTo(410 + (QUANTUM * 1000) / RATE, 6);
expect(buffer.trimmed).toBeCloseTo(410 - (QUANTUM * 1000) / RATE, 6);
});
@@ -1011,68 +1001,187 @@ describe("time stretch", () => {
expect(buffer.trimmed).toBe(0);
});
- it("counts a re-anchor that drops only the staged window as a skip", () => {
- const buffer = live(100);
- // Four whole 30 ms windows, so once the playhead passes 90 ms the ring is empty and only the
- // last window is left, staged.
+ // A playhead behind the clock skips a declared silence rather than stretching across it, and only
+ // once the staged window before it has played.
+ it("skips a declared silence by the lag only after the staged window plays", () => {
+ const buffer = live(100, { bound: 1000 });
write(buffer, 0, frames(120), { value: 0.25 });
- while (buffer.timestamp / 1000 <= 90) read(buffer, QUANTUM, 1);
- expect(buffer.length).toBeGreaterThan(0);
- expect(buffer.length).toBeLessThan(frames(30));
-
- write(buffer, 120, frames(500), { value: 0.5 });
- expect(buffer.skips).toBe(1);
- expect(buffer.length).toBe(frames(100));
- });
-
- // The ring past the read index holds less than the floor, but the staged window lifts the total a
- // chunk above it, so the catch-up over the declared silence must count the staged output too.
- it("catches up over a declared silence without rewinding behind a staged window", () => {
- const buffer = live(100);
- write(buffer, 0, frames(120), { value: 0.25 });
- while (buffer.timestamp / 1000 <= 90) read(buffer, QUANTUM, 1);
+ while (buffer.timestamp / 1000 <= 90) {
+ buffer.clock(buffer.timestamp);
+ read(buffer, QUANTUM, 1);
+ }
const staged = buffer.length;
- expect(staged).toBeGreaterThan(frames(20));
+ expect(staged).toBeGreaterThan(0);
+ expect(staged).toBeLessThan(frames(30));
- // The next utterance lands 80 ms after the last, 10 ms long: 90 ms of ring behind the window.
+ // The next utterance lands 80 ms after the last, 10 ms long, with the playhead 40 ms behind.
buffer.next();
- const playhead = buffer.timestamp;
write(buffer, 200, frames(10), { value: 0.5 });
- expect(buffer.timestamp).toBeGreaterThanOrEqual(playhead);
- expect(buffer.length).toBe(frames(100));
+ buffer.clock(buffer.timestamp + 40_000);
+ const [output] = read(buffer, staged + frames(50), 1);
+
+ // The staged window, then the 80 ms silence less the 40 ms lag, then the utterance.
+ expect(output.length).toBe(staged + frames(50));
+ expect(output.slice(0, staged).every((sample) => sample === 0.25)).toBe(true);
+ expect(output.slice(staged, staged + frames(40)).every((sample) => sample === 0)).toBe(true);
+ expect(output.slice(staged + frames(40)).every((sample) => sample === 0.5)).toBe(true);
expect(buffer.skips).toBe(0);
+ expect(buffer.trimmed).toBe(0);
+ });
+
+ // A jump counts only as far as the ring holds audio past its target, so one that left no audio to
+ // skip, as a reference moving on jitter while the buffer runs low, is the stretch's.
+ it("never skips a pending jump below its target", () => {
+ const buffer = live(20, { bound: 2000 });
+ write(buffer, 0, frames(100), { value: 0.5 });
+ buffer.clock(0);
+ read(buffer, QUANTUM, 1);
- // The staged window plays out once, then the rest of the silence, then the utterance.
- const [output] = read(buffer, frames(100), 1);
- expect(output.length).toBe(frames(100));
- const silence = output.findIndex((sample) => sample === 0);
- expect(silence).toBe(staged);
- expect(output.slice(silence).every((sample) => sample === 0 || sample === 0.5)).toBe(true);
- expect(output.at(-1)).toBe(0.5);
+ // 25 ms pending, short of a window, then 15 ms more once only 35 ms is held: one 20 ms target and
+ // 15 ms past it.
+ buffer.jumped(25_000);
+ buffer.clock(buffer.timestamp + 25_000);
+ read(buffer, QUANTUM, 1);
+ expect(buffer.skips).toBe(0);
+ while (buffer.length > frames(35) + QUANTUM) {
+ buffer.clock(buffer.timestamp + 25_000);
+ read(buffer, QUANTUM, 1);
+ }
+ buffer.jumped(15_000);
+ buffer.clock(buffer.timestamp + 40_000);
+ read(buffer, QUANTUM, 1);
+ expect(buffer.skips).toBe(0);
+ expect(buffer.stalled).toBe(false);
});
- // Past NetEq's fast threshold, four times the floor plus its band, or three quarters of the
- // headroom above the floor, whichever is lower, stretching back would take seconds, so the ring
- // re-anchors on its floor as moq-audio does. With 1000 ms of capacity, a 100 ms floor reaches the
- // first at 480 ms, and a 300 ms floor the second at 825 ms.
- for (const [floor, threshold] of [
- [100, 480],
- [300, 825],
- ]) {
- it(`re-anchors a ${floor} ms floor only once the ring holds ${threshold} ms`, () => {
- const buffer = live(floor);
- write(buffer, 0, frames(floor), { value: 0.5 });
+ it("ignores a jump while it holds no audio past its target", () => {
+ const buffer = live(100, { bound: 2000 });
+ write(buffer, 0, frames(100), { value: 0.5 });
+ buffer.clock(0);
+ read(buffer, QUANTUM, 1);
+
+ buffer.jumped(50_000);
+ buffer.clock(buffer.timestamp + 50_000);
+ read(buffer, QUANTUM, 1);
+ expect(buffer.skips).toBe(0);
+ });
+
+ // A clock whose reference moved earlier skips as much, as far as the playhead trails it, where the
+ // same lag from a fallen target is caught up by speeding up.
+ for (const jumped of [false, true]) {
+ it(`${jumped ? "skips" : "catches up on"} a clock moved ahead by ${jumped ? "its reference" : "a fall in its target"}`, () => {
+ const buffer = live(200, { bound: 2000 });
+ write(buffer, 0, frames(1000), { value: 0.5 });
+ // Joined on the clock.
+ buffer.clock(0);
read(buffer, QUANTUM, 1);
- const playhead = buffer.timestamp;
- write(buffer, floor, frames(threshold - 10 - floor), { value: 0.5 });
- expect(buffer.timestamp).toBe(playhead);
+ // The clock moves 100 ms ahead.
+ if (jumped) buffer.jumped(150_000);
+ else buffer.setLatency(100, { target: 100, bound: 2000 });
+ const quantum = (QUANTUM * 1_000_000) / RATE;
+ let clock = buffer.timestamp + 100_000;
+ buffer.clock(clock);
+ read(buffer, QUANTUM, 1);
+ if (jumped) {
+ // Never past the clock, however far the reference moved.
+ expect(buffer.skips).toBe(1);
+ expect(Math.abs(buffer.timestamp - clock - quantum)).toBeLessThan(50);
+ return;
+ }
+ for (let i = 0; i < 40; i++) {
+ clock += quantum;
+ buffer.clock(clock);
+ read(buffer, QUANTUM, 1);
+ }
expect(buffer.skips).toBe(0);
+ expect(buffer.accelerated).toBeGreaterThan(0);
+ });
+ }
+
+ // A successor's arrivals count from where it cut the predecessor, so audio the predecessor had
+ // received past the cut never reads as depth to speed through.
+ it("counts a successor's arrivals from the cut", () => {
+ const buffer = live(100, { bound: 2000 });
+ write(buffer, 0, frames(200), { value: 0.5 });
+ buffer.received(10_000_000);
+ buffer.clock(0);
+ read(buffer, QUANTUM, 1);
- write(buffer, threshold - 10, frames(20), { value: 0.5 });
- expect(buffer.timestamp).toBeGreaterThan(playhead);
- expect(buffer.length).toBeLessThan(frames(floor) + frames(30));
+ buffer.truncate(100_000);
+ write(buffer, 100, frames(60), { value: 0.9 });
+ // 20 ms behind the clock, on 60 ms of the successor's audio: too little to speed through.
+ for (let i = 0; i < 20; i++) {
+ buffer.clock(buffer.timestamp + 20_000);
+ read(buffer, QUANTUM, 1);
+ }
+ expect(buffer.accelerated).toBe(0);
+ });
+
+ // A jump past what is decoded skips the rest once it lands, whether or not the lag past the bound
+ // skipped first.
+ for (const bound of [2000, 200]) {
+ it(`skips the rest of a jump past what is decoded once it lands, at a ${bound} ms bound`, () => {
+ const buffer = live(100, { bound });
+ write(buffer, 0, frames(100), { value: 0.5 });
+ buffer.clock(0);
+ read(buffer, QUANTUM, 1);
+
+ // The backlog that moved the reference arrived, still encoded past what is decoded.
+ buffer.received(900_000);
+ buffer.jumped(500_000);
+ const clock = buffer.timestamp + 500_000;
+ buffer.clock(clock);
+ read(buffer, QUANTUM, 1);
+ expect(buffer.skips).toBe(1);
+
+ write(buffer, 100, frames(800), { value: 0.9 });
+ buffer.clock(clock);
+ read(buffer, QUANTUM, 1);
+ expect(Math.abs(buffer.timestamp - clock - (QUANTUM * 1_000_000) / RATE)).toBeLessThan(50);
+ });
+ }
+
+ // Nothing trailing the clock was heard before the first read on it, so a join lands on the clock
+ // past a backlog the bound would otherwise keep.
+ it("joins on the clock past a backlog inside its bound", () => {
+ const buffer = live(100, { bound: 1000 });
+ write(buffer, 0, frames(800), { value: 0.5 });
+ buffer.clock(500_000);
+ read(buffer, QUANTUM, 1);
+ expect(buffer.skips).toBe(1);
+ expect(buffer.timestamp / 1000).toBeCloseTo(500 + (QUANTUM * 1000) / RATE, 6);
+
+ // A reset starts a new timeline, which joins again.
+ buffer.reset();
+ write(buffer, 2000, frames(800), { value: 0.5 });
+ buffer.clock(2_300_000);
+ read(buffer, QUANTUM, 1);
+ expect(buffer.skips).toBe(2);
+ });
+
+ // Audio the stretch could still catch up on plays; only what trails the clock past the bound and
+ // one window skips, back to the bound.
+ for (const bound of [0, 200]) {
+ it(`skips only audio trailing the clock past a ${bound} ms bound and a window`, () => {
+ const buffer = live(100, { bound });
+ write(buffer, 0, frames(800), { value: 0.5 });
+ // Joined on the clock.
+ buffer.clock(0);
+ read(buffer, QUANTUM, 1);
+
+ buffer.clock(buffer.timestamp + (bound + 30) * 1000);
+ read(buffer, QUANTUM, 1);
+ expect(buffer.skips).toBe(0);
+
+ const clock = buffer.timestamp + (bound + 31) * 1000;
+ buffer.clock(clock);
+ read(buffer, QUANTUM, 1);
expect(buffer.skips).toBe(1);
+ // Back to the bound, then a quantum on, which a stretch may have sped up.
+ const quantum = (QUANTUM * 1_000_000) / RATE;
+ expect(buffer.timestamp).toBeGreaterThanOrEqual(clock - bound * 1000 + quantum);
+ expect(buffer.timestamp).toBeLessThan(clock - bound * 1000 + 2 * quantum + 1);
});
}
});
diff --git a/js/watch/src/audio/ring-buffer.ts b/js/watch/src/audio/ring-buffer.ts
index c3672abb7ed..5680df17a71 100644
--- a/js/watch/src/audio/ring-buffer.ts
+++ b/js/watch/src/audio/ring-buffer.ts
@@ -1,7 +1,19 @@
import { ringCapacity } from "./latency";
-import { Stretch } from "./stretch";
-
-/** The worklet's PCM ring. Latencies are in milliseconds and timestamps in microseconds, as the port carries them. */
+import type { Terms } from "./render";
+import { type Inputs, Stretch } from "./stretch";
+
+// An AudioWorklet render quantum, the read size until the first read says otherwise.
+const QUANTUM = 128;
+
+/**
+ * The worklet's PCM ring. Latencies are in milliseconds and timestamps in microseconds, as the port
+ * carries them.
+ *
+ * Playback follows the decision rule in doc/concept/audio-jitter.md: each window of received PCM may
+ * play faster to catch up with the shared clock once the buffer holds more than the target, or slower
+ * while the buffer runs low or the playhead runs ahead of the clock. Only audio trailing the clock past
+ * the subscription's max delay is skipped.
+ */
export class AudioRingBuffer {
#buffer: Float32Array[];
#writeIndex = 0;
@@ -10,10 +22,21 @@ export class AudioRingBuffer {
readonly rate: number;
readonly channels: number;
#stalled = true;
- // Held stalled for the part of the target above the floor, until the next floor update.
- #parked = false;
+ // Whether playback started since creation or the last reset, so running dry refills one stretch
+ // window and a quantum rather than the floor.
+ #started = false;
#underruns = 0;
#skips = 0;
+ // Whether the last read skipped, so a run of reads skipping late audio counts once.
+ #skipping = false;
+ // Whether the ring has read on the clock since the last reset. Until it has, nothing trailing the
+ // clock was heard, so a join lands on it rather than at the bound.
+ #joined = false;
+ // Samples the shared clock jumped ahead by moving its reference earlier since the last read on it,
+ // and those not yet skipped, as far as the ring holds audio past its target and the playhead still
+ // trails the clock.
+ #incoming = 0;
+ #jumped = 0;
// Samples of media a skip dropped, and that the stretch removed by accelerating, gap zeros
// included and declared silence excluded, as moq-audio counts them.
#trimmed = 0;
@@ -26,9 +49,6 @@ export class AudioRingBuffer {
#previous?: number;
// Set by `next` until that first write, which marks where the previous utterance's audio stops.
#next = false;
- // Where the audio before the latest declared silence stops. The silence is declared, so it must not
- // count toward the live band and push the unplayed tail out.
- #tail?: number;
// Declared silences still ahead of the playhead, oldest first, as [start, end) sample indices. The
// ring holds them as zeros; the reader fills them with comfort noise. Each one follows a distinct
// write, and every move of the playhead drops those behind it, so the ring's capacity bounds them.
@@ -39,37 +59,33 @@ export class AudioRingBuffer {
// silence, which is no underrun and survives the reset that re-anchors the next utterance.
#silent = false;
- // Samples in the most recent write, i.e. one decoded chunk. Catching up over a declared silence
- // tolerates this much above the target, so a ring sitting exactly on the target doesn't skip the
- // moment the next chunk lands. The most recent write rather than a running maximum: one oversized
- // decode would otherwise widen the band for the life of the ring.
- #chunk = 0;
-
- // Live playback converges on the floor by accelerating each window of received PCM on its way out.
- // The read index is past the staged window, which spans `#window` samples of the ring.
+ // Each window of received PCM stretches on its way out. The read index is past the staged window,
+ // which spans `#window` samples of the ring.
readonly #stretch: Stretch;
#window = 0;
- // The ring's length right after the latest write, less what the stretch has removed since. That
- // is where depth reads the clock: a frame has just arrived. Between arrivals the ring drains
- // toward the next one, which says nothing about how far behind the playhead is, so steering on
- // the current length would expand into every gap and settle behind the shared clock.
- #peak = 0;
// Whether the staged window lies in a declared silence, which plays unstretched as comfort noise.
#windowQuiet = false;
- // Buffered mode: play through everything buffered without skipping ahead.
- readonly #buffered: boolean;
+ // Where the newest audio received ends, decoded or not, unset until some arrives on this timeline.
+ #received?: number;
+ // How far the playhead trailed that audio when it arrived, less what the stretch and skips removed
+ // since: the buffer level the stretch decides on. Between arrivals the ring drains toward the next
+ // one, which says nothing about how far behind the playhead is, so the level ignores that drain.
+ #level = 0;
+ // Where the shared clock stands at the first frame of the current read, unset without a clock.
+ #clock?: number;
+ // The size of the latest read, which a refill leaves room for.
+ #period = QUANTUM;
+
// Un-stall threshold in samples (how much to buffer before playback starts).
#latencySamples: number;
+ // The playout delay and how far audio may trail the clock, in samples.
+ #target: number;
+ #bound: number;
// Whether the read/write indices have been anchored to the first inserted sample.
#anchored = false;
- constructor(props: {
- rate: number;
- channels: number;
- latency: number;
- buffered?: boolean;
- }) {
+ constructor(props: { rate: number; channels: number } & Pick & Partial) {
if (props.channels <= 0) throw new Error("invalid channels");
if (props.rate <= 0) throw new Error("invalid sample rate");
if (props.latency <= 0) throw new Error("invalid latency");
@@ -79,12 +95,12 @@ export class AudioRingBuffer {
this.rate = props.rate;
this.channels = props.channels;
- this.#buffered = props.buffered ?? false;
+ this.#target = this.#samples(props.target ?? props.latency);
+ this.#bound = this.#samples(props.bound ?? 0);
// The ring holds the latency floor as PCM, with headroom above it, at a capacity the receiver
// fixes rather than the target: whatever of the target is above the floor stays encoded
- // upstream (the main thread holds it back). The headroom tolerates a chunk above the floor in
- // live mode, and in buffered mode keeps the paced decode loop from overflow-dropping.
+ // upstream (the main thread holds it back), and the headroom keeps a burst from overflowing.
const capacity = ringCapacity(this.rate);
if (this.#latencySamples > capacity) throw new Error("latency exceeds capacity");
@@ -95,6 +111,10 @@ export class AudioRingBuffer {
this.#stretch = new Stretch(this.rate, this.channels);
}
+ #samples(ms: number): number {
+ return Math.round((ms / 1000) * this.rate);
+ }
+
get stalled(): boolean {
return this.#stalled;
}
@@ -134,9 +154,14 @@ export class AudioRingBuffer {
/** The playhead, with the staged window's unplayed output mapped back onto the samples it came from. */
get timestamp(): number {
+ return (this.#playhead() / this.rate) * 1_000_000;
+ }
+
+ // The playhead in samples: the read index less the staged window's unplayed share.
+ #playhead(): number {
const stretch = this.#stretch;
const staged = stretch.length > 0 ? (stretch.available * this.#window) / stretch.length : 0;
- return ((this.#readIndex - staged) / this.rate) * 1_000_000;
+ return this.#readIndex - staged;
}
/** Whether an insert has anchored the indices since creation or the last reset. */
@@ -153,35 +178,63 @@ export class AudioRingBuffer {
return this.#buffer[0]?.length;
}
- /** Move the floor, which never exceeds the fixed capacity, and end a park. */
- setLatency(latency: number): void {
+ /**
+ * Move the floor, which never exceeds the fixed capacity, and the target and bound when given. A
+ * moved target never stalls playback: the stretch slows down onto a rise and speeds up onto a fall.
+ */
+ setLatency(latency: number, terms?: Omit): void {
const latencySamples = Math.ceil(this.rate * (latency / 1000));
if (latencySamples <= 0) throw new Error("empty buffer");
if (latencySamples > this.capacity) throw new Error("latency exceeds capacity");
-
- // A deeper floor parks playback until it refills. Video holds the extra delay on its own, so
- // audio that kept draining at the old depth would run ahead by the difference. Parking keeps
- // what is buffered, so a rise costs only its own size in silence, and none if the buffer
- // already covers the new floor.
- if (latencySamples > this.#latencySamples) this.#stalled = true;
this.#latencySamples = latencySamples;
- this.#parked = false;
+ this.#target = this.#samples(terms?.target ?? latency);
+ this.#bound = this.#samples(terms?.bound ?? 0);
- // Resume a refill that the buffer already covers, which `write` would only notice on the
- // next frame, never for a source that has stopped.
- if (this.length >= latencySamples) this.#stalled = false;
+ // Start a ring that the buffer already covers, which `write` would only notice on the next
+ // frame, never for a source that has stopped.
+ this.#resume();
}
/**
- * Hold playback until the next floor update, keeping what is buffered.
- *
- * For a rise in the part of the target held encoded upstream: the ring already covers its floor,
- * so only the main thread knows when the received media covers the rest. A write does not resume
- * it, since output decoded before the park may still land.
+ * Note that media ending at `timestamp` arrived, decoded or not. How far the playhead trails it
+ * then is the level the stretch decides on, and what lies past the playhead the depth it may
+ * speed through.
*/
- park(): void {
- this.#stalled = true;
- this.#parked = true;
+ received(timestamp: number): void {
+ const end = Math.round((timestamp / 1_000_000) * this.rate);
+ if (this.#received !== undefined && end <= this.#received) return;
+ this.#received = end;
+ if (this.#anchored) this.#level = end - this.#playhead();
+ }
+
+ /**
+ * Note that the shared clock's reference moved `micros` earlier, jumping the clock ahead by as much.
+ * A jump counts as far as the ring holds audio past its target, which the early media that moved
+ * the reference left it. Once they add up to more than a stretch window, a read skips them, as far as
+ * the playhead trails the clock; less, as a publisher's clock drifting fast moves it, is the
+ * stretch's to catch up. A fall in the target moves the clock ahead too, which is always caught up
+ * by speeding up.
+ */
+ jumped(micros: number): void {
+ this.#incoming += Math.round((micros / 1_000_000) * this.rate);
+ }
+
+ /** Where the shared clock stands at the first frame of the next read, in microseconds, if it has one. */
+ clock(timestamp: number | undefined): void {
+ this.#clock = timestamp === undefined ? undefined : Math.round((timestamp / 1_000_000) * this.rate);
+ }
+
+ // Un-stall once the buffer holds the floor to start on, or after running dry, one stretch window
+ // and a read: the least that resumes without running dry again at once, and never more than the
+ // floor. Slowing down then rebuilds the level rather than a wait in silence.
+ #resume(): void {
+ if (!this.#stalled) return;
+ const cushion = Math.min(this.#latencySamples, this.#stretch.window + this.#period);
+ const need = this.#started ? cushion : this.#latencySamples;
+ if (this.length < need) return;
+ this.#stalled = false;
+ this.#started = true;
+ this.#stretch.level.reset();
}
write(timestamp: number, data: Float32Array[]): void {
@@ -197,6 +250,7 @@ export class AudioRingBuffer {
this.#readIndex = start;
this.#writeIndex = start;
this.#anchored = true;
+ this.#level = Math.max(this.#received ?? start, start) - start;
}
// Ignore samples behind the read index, which runs ahead of the playhead by the staged window:
@@ -214,48 +268,22 @@ export class AudioRingBuffer {
}
const end = start + samples;
- this.#chunk = data[0].length;
// The next utterance's first audio: whatever lies between it and the previous utterance's audio
// is declared silence.
if (this.#next) {
this.#next = false;
this.#previous = undefined;
- this.#tail = this.#writeIndex;
if (start > this.#writeIndex) this.#pauses.push({ start: this.#writeIndex, end: start });
}
- // Bound the ring. While playing live, the read path accelerates the ring back onto its floor, so
- // dropping the oldest audio and landing on the floor is an emergency re-anchor, as moq-audio
- // does: at NetEq's fast threshold, four times the floor plus 20 ms, or once the ring holds three
- // quarters of its headroom above the floor, whichever comes first. While stalled the
- // reader is not consuming, so only the hard capacity applies; dropping would throw away the very
- // audio the refill is accumulating. Buffered mode plays through everything, so it is
- // capacity-bound too.
- const playing = !this.#stalled && !this.#buffered;
- // Until the tail before a declared silence has played, only the capacity bounds the ring; then
- // a ring a chunk above its floor catches up by skipping the silence, which drops no audio.
- const band = playing && (this.#tail === undefined || this.#readIndex >= this.#tail);
+ // Bound the ring at its capacity, dropping the oldest audio. The stretch keeps a ring near its
+ // floor, so only a burst the decode gate let through overflows it.
this.#prune();
- const pause = this.#pauses[0];
- const latency = this.#latencySamples;
- const queued = end - this.#readIndex + this.#stretch.available;
if (end - this.#readIndex > this.capacity) {
+ // An empty ring only re-anchors over a gap it never held, which drops nothing.
+ if (this.length > 0) this.#skips++;
this.#skip(end - this.capacity, start);
- } else if (band && pause && pause.start <= this.#readIndex && queued > latency + this.#chunk) {
- // Skip only the silence past the queued floor, which the staged window counts toward, so the
- // read index only moves forward and never re-stages PCM the window already holds.
- this.#readIndex = Math.min(pause.end, this.#readIndex + queued - latency);
- this.#prune();
- } else if (
- band &&
- queued >=
- Math.min(
- 4 * (latency + Math.floor((this.rate * 20) / 1000)),
- latency + Math.floor(((this.capacity - latency) * 3) / 4),
- )
- ) {
- this.#skip(end - latency, start);
}
// Fill gaps with zeros if there's a discontinuity
@@ -292,14 +320,9 @@ export class AudioRingBuffer {
if (end > this.#writeIndex) {
this.#writeIndex = end;
}
- this.#peak = this.length;
-
- // Start playback once we've buffered the latency target. This is the only way out of a
- // stall but a park, so an underrun mid-playback refills to the target before resuming rather
- // than playing the next chunk on an empty cushion.
- if (!this.#parked && this.length >= this.#latencySamples) {
- this.#stalled = false;
- }
+ // Decoded audio arrived too, if no message said so first.
+ this.received((end / this.rate) * 1_000_000);
+ this.#resume();
}
/**
@@ -310,24 +333,30 @@ export class AudioRingBuffer {
*/
truncate(timestamp: number): void {
const target = Math.round((timestamp / 1_000_000) * this.rate);
- if (target >= this.#writeIndex) return;
- const length = this.length;
- // A staged window runs the read index ahead of the playhead. Unstage it so the successor
- // replaces whatever of it is still unplayed, resuming the ring at the sample now playing.
- const stretch = this.#stretch;
- if (target < this.#readIndex && stretch.available > 0) {
- const staged = (stretch.available * this.#window) / stretch.length;
- const capacity = this.#buffer[0].length;
- this.#readIndex = Math.max(Math.round(this.#readIndex - staged), this.#writeIndex - capacity);
- stretch.clear();
- this.#window = 0;
+ if (target < this.#writeIndex) {
+ // A staged window runs the read index ahead of the playhead. Unstage it so the successor
+ // replaces whatever of it is still unplayed, resuming the ring at the sample now playing.
+ const stretch = this.#stretch;
+ if (target < this.#readIndex && stretch.available > 0) {
+ const staged = (stretch.available * this.#window) / stretch.length;
+ const capacity = this.#buffer[0].length;
+ this.#readIndex = Math.max(Math.round(this.#readIndex - staged), this.#writeIndex - capacity);
+ stretch.clear();
+ this.#window = 0;
+ }
+ // Never retreat past the playhead: those samples are already due.
+ this.#writeIndex = Math.max(target, this.#readIndex);
+ this.#pauses = this.#pauses.filter((pause) => pause.start < this.#writeIndex);
+ const last = this.#pauses.at(-1);
+ if (last) last.end = Math.min(last.end, this.#writeIndex);
+ }
+ // What arrived past the cut, decoded or not, was the predecessor's, so the successor's arrivals
+ // count from there.
+ const received = Math.max(target, this.#writeIndex);
+ if (this.#received !== undefined && this.#received > received) {
+ this.#received = received;
+ if (this.#anchored) this.#level = received - this.#playhead();
}
- // Never retreat past the playhead: those samples are already due.
- this.#writeIndex = Math.max(target, this.#readIndex);
- this.#pauses = this.#pauses.filter((pause) => pause.start < this.#writeIndex);
- const last = this.#pauses.at(-1);
- if (last) last.end = Math.min(last.end, this.#writeIndex);
- this.#peak = Math.max(0, this.#peak - (length - this.length));
}
/**
@@ -343,22 +372,20 @@ export class AudioRingBuffer {
}
// Move the read index forward to `index`, dropping any queued audio it passes, staged included,
- // and the front of the incoming chunk at `start` when `index` lies past it. An empty ring only
- // re-anchors over a gap it never held: neither counter moves, as in moq-audio.
+ // and the front of the incoming chunk at `start` when `index` lies past it. The caller counts the
+ // skip; an empty ring only re-anchors over a gap it never held, so the trim does not move, as in
+ // moq-audio.
#skip(index: number, start: number): void {
- if (index > this.#readIndex && this.length > 0) {
- this.#skips++;
- this.#trimmed += this.#discarded(index, start);
- }
+ if (index > this.#readIndex && this.length > 0) this.#trimmed += this.#discarded(index, start);
// The staged window lies behind the new index, so it is dropped with the rest, as moq-audio's
- // trim discards staged output first.
- const length = this.length;
+ // skip discards staged output first. The playhead jumps, so the level drops with it.
+ const playhead = this.#playhead();
if (index > this.#readIndex) {
this.#stretch.clear();
this.#window = 0;
}
this.#readIndex = index;
- this.#peak = Math.max(0, this.#peak - (length - this.length));
+ this.#level -= Math.max(0, index - playhead);
this.#prune();
}
@@ -415,36 +442,83 @@ export class AudioRingBuffer {
this.#readIndex = 0;
this.#writeIndex = 0;
this.#stalled = true;
- this.#parked = false;
+ this.#started = false;
this.#anchored = false;
this.#end = undefined;
this.#previous = undefined;
this.#next = false;
- this.#tail = undefined;
this.#pauses = [];
this.#stretch.clear();
+ this.#stretch.level.reset();
this.#window = 0;
- this.#peak = 0;
+ this.#received = undefined;
+ this.#level = 0;
+ this.#clock = undefined;
+ this.#joined = false;
+ this.#incoming = 0;
+ this.#jumped = 0;
+ this.#skipping = false;
}
+ /**
+ * Fill `output` from the playhead, returning how many frames it supplied. Pass where the shared
+ * clock stands at its first frame through {@link clock} beforehand.
+ */
read(output: Float32Array[]): number {
if (output.length !== this.channels) throw new Error("wrong number of channels");
for (const dst of output) if (dst.length !== output[0].length) throw new Error("mismatching number of samples");
this.#quiet = undefined;
+ this.#period = output[0].length;
if (this.#stalled) return 0;
if (this.length <= 0) {
- // Ran dry mid-playback: re-stall so write() refills to the target before resuming.
+ // Ran dry mid-playback: re-stall so write() refills a cushion before resuming.
this.#stalled = true;
if (this.#ended()) this.#silent = true;
else this.#underruns++;
return 0;
}
+ // Skip what trails the clock past its bound and one window, back to the bound, or the jumps of
+ // its reference past one window. Less than that is the stretch's to catch up. A join has the
+ // bound of nothing heard yet, so it lands on the clock past whatever backlog it was handed, and a
+ // jump before it was never heard. A finite tail plays as received.
+ let skip = 0;
+ if (this.#clock !== undefined) {
+ const inputs = this.#inputs(0);
+ const bound = this.#joined ? inputs.bound : 0;
+ // Never more than the audio held past the target, so a skip never drains the ring below it.
+ const excess = Math.max(0, inputs.raw - inputs.target);
+ const jumped = Math.min(this.#jumped + this.#incoming, excess, Math.max(0, inputs.lag));
+ this.#jumped = this.#joined ? jumped : 0;
+ this.#incoming = 0;
+ this.#joined = true;
+ if (!this.#ending()) {
+ if (inputs.lag > bound + this.#stretch.window) skip = inputs.lag - bound;
+ if (this.#jumped > this.#stretch.window) skip = Math.max(skip, this.#jumped);
+ }
+ }
+ if (skip > 0) {
+ const playhead = this.#playhead();
+ const index = Math.min(this.#writeIndex, Math.ceil(playhead + skip));
+ // What a skip leaves of a jump, as past what is decoded, skips once it lands rather than
+ // staying inside the bound for the stretch.
+ this.#jumped = Math.max(0, this.#jumped - (index - playhead));
+ if (!this.#skipping) this.#skips++;
+ this.#skipping = true;
+ this.#skip(index, this.#writeIndex);
+ if (this.length <= 0) {
+ this.#stalled = true;
+ return 0;
+ }
+ } else {
+ this.#skipping = false;
+ }
+
const stretch = this.#stretch;
let samples = 0;
while (samples < output[0].length) {
- if (stretch.available === 0 && !this.#stage(output[0].length - samples)) break;
+ if (stretch.available === 0 && !this.#stage(output[0].length - samples, samples)) break;
const from = samples;
samples += stretch.read(output, samples);
// A quantum overlaps at most one declared silence unless an utterance is shorter than it,
@@ -469,23 +543,50 @@ export class AudioRingBuffer {
return samples;
}
- // Move the next window off the ring into the stretch, which accelerates a live ring onto its floor.
- // A window stops at the edge of a declared silence, which plays unstretched, so the stretch never
- // moves where one starts or ends. Whatever cannot stretch stages only the `need` this quantum
- // reads, so a write still lands anywhere past the playhead. Returns false once the ring is empty.
- #stage(need: number): boolean {
- const buffered = this.#writeIndex - this.#readIndex;
- if (buffered <= 0) return false;
+ // Whether the playhead is in audio before a declared end: a finite tail, which drains as received,
+ // since the next utterance re-anchors the ring anyway and its depth says nothing about the clock.
+ #ending(): boolean {
+ const end = this.#previous ?? this.#end;
+ return end !== undefined && this.#readIndex < end;
+ }
+
+ // What a window starting `elapsed` frames into this read decides on. Without a clock, the target
+ // is where the playhead belongs and nothing trails it far enough to skip.
+ #inputs(elapsed: number): Inputs {
+ const playhead = this.#playhead();
+ const level = Math.max(0, this.#level);
+ const raw = Math.max(0, Math.max(this.#received ?? 0, this.#writeIndex) - playhead);
+ if (this.#clock === undefined) {
+ return { target: this.#target, level, raw, lag: level - this.#target, bound: Number.POSITIVE_INFINITY };
+ }
+ return { target: this.#target, level, raw, lag: this.#clock + elapsed - playhead, bound: this.#bound };
+ }
+ // Move the next window off the ring into the stretch. A window stops at the edge of a declared
+ // silence, which plays unstretched, so the stretch never moves where one starts or ends; a playhead
+ // behind the clock skips the silence instead, which drops no audio. Whatever cannot stretch stages
+ // only the `need` this quantum reads, so a write still lands anywhere past the playhead. `elapsed`
+ // frames of this read have played. Returns false once the ring is empty.
+ #stage(need: number, elapsed: number): boolean {
this.#prune();
- const pause = this.#pauses[0];
+ let pause = this.#pauses[0];
+ if (pause !== undefined && pause.start <= this.#readIndex && this.#clock !== undefined) {
+ const lag = this.#clock + elapsed - this.#readIndex;
+ if (lag > 0) {
+ const index = Math.min(pause.end, this.#writeIndex, this.#readIndex + lag);
+ this.#level -= index - this.#readIndex;
+ this.#readIndex = index;
+ this.#prune();
+ pause = this.#pauses[0];
+ }
+ }
+
+ const buffered = this.#writeIndex - this.#readIndex;
+ if (buffered <= 0) return false;
const quiet = pause !== undefined && pause.start <= this.#readIndex;
- // Audio before a declared end is a finite tail: it drains as received, since the next utterance
- // re-anchors the ring anyway.
- const end = this.#previous ?? this.#end;
- const ending = end !== undefined && this.#readIndex < end;
- const enabled = !quiet && !ending && !this.#buffered;
- let frames = Math.min(buffered, this.#stretch.window, enabled ? buffered : need);
+ const enabled = !quiet && !this.#ending();
+ const window = this.#stretch.window;
+ let frames = Math.min(buffered, window, enabled ? buffered : need);
if (pause) frames = Math.min(frames, (quiet ? pause.end : pause.start) - this.#readIndex);
for (let channel = 0; channel < this.channels; channel++) {
@@ -494,8 +595,8 @@ export class AudioRingBuffer {
for (let i = 0; i < frames; i++) dst[i] = src[(this.#readIndex + i) % src.length];
}
const stretch = this.#stretch;
- stretch.process(frames, this.#peak - this.#latencySamples, enabled);
- this.#peak -= frames - stretch.length;
+ stretch.process(frames, enabled && frames === window ? this.#inputs(elapsed) : undefined);
+ this.#level -= frames - stretch.length;
this.#accelerated += Math.max(0, frames - stretch.length);
this.#readIndex += frames;
this.#window = frames;
diff --git a/js/watch/src/audio/stretch.test.ts b/js/watch/src/audio/stretch.test.ts
index 9d6de528ad0..2749be9c572 100644
--- a/js/watch/src/audio/stretch.test.ts
+++ b/js/watch/src/audio/stretch.test.ts
@@ -2,61 +2,63 @@ import { expect, it } from "bun:test";
import { readFileSync } from "node:fs";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
-import { type Decision, Stretch } from "./stretch";
+import { type Decision, decide, Level, Stretch } from "./stretch";
-interface Vector {
+interface Pcm {
name: string;
rate: number;
channels: number;
input: number[];
- buffered: number;
- target: number;
- cooldown: number;
- /** As native playback decides: it also expands, which this player never does. */
- decision: Decision | "Expand";
+ decision: Decision;
+ allowance: number;
lag: number;
output: number[];
}
+interface Vectors {
+ decisions: {
+ name: string;
+ rate: number;
+ target: number;
+ level: number;
+ raw: number;
+ lag: number;
+ bound: number;
+ cooldown: number;
+ decision: Decision;
+ allowance: number;
+ }[];
+ filter: {
+ name: string;
+ rate: number;
+ target: number;
+ steps: [number, number][];
+ filtered: number[];
+ }[];
+}
+
// Shared with moq-audio, so neither implementation is graded against the other.
-const corpus = join(
- dirname(fileURLToPath(import.meta.url)),
- "../../../../doc/concept/audio-jitter/stretch/corpus.json",
-);
-const vectors = JSON.parse(readFileSync(corpus, "utf8")) as Vector[];
+const shared = join(dirname(fileURLToPath(import.meta.url)), "../../../../doc/concept/audio-jitter/stretch");
+const pcm = JSON.parse(readFileSync(join(shared, "corpus.json"), "utf8")) as Pcm[];
+const vectors = JSON.parse(readFileSync(join(shared, "decisions.json"), "utf8")) as Vectors;
// The corpus is interleaved; the ring is planar.
const planar = (samples: number[], channels: number) =>
Array.from({ length: channels }, (_, channel) => samples.filter((_, i) => i % channels === channel));
-// The corpus decides as NetEq does, on the buffered depth against a band around the target. This
-// player decides on the excess above the target alone and never expands, so the corpus grades its
-// seams: an excess past the 20 ms band fast-accelerates, one inside it accelerates, and a ring at or
-// short of its target, where NetEq holds or expands, plays its window unchanged.
-const excess = (vector: Vector) => {
- const band = Math.floor((vector.rate * 20) / 1000);
- if (vector.decision === "Fast") return vector.buffered - vector.target;
- if (vector.decision === "Accelerate") return band - 1;
- return Math.min(0, vector.buffered - vector.target);
-};
-
-for (const vector of vectors) {
+for (const vector of pcm) {
it(`stretches the corpus PCM: ${vector.name}`, () => {
const stretch = new Stretch(vector.rate, vector.channels);
const input = planar(vector.input, vector.channels);
input.forEach((samples, channel) => {
stretch.input[channel].set(samples);
});
- stretch.cooldown = vector.cooldown;
- const expand = vector.decision === "Expand";
- expect(stretch.decide(excess(vector))).toBe(vector.decision === "Expand" ? "Hold" : vector.decision);
-
- stretch.process(stretch.window, excess(vector), true);
- expect(stretch.window - stretch.available).toBe(expand ? 0 : vector.lag);
+ stretch.stage(stretch.window, { decision: vector.decision, allowance: vector.allowance });
+ expect(Math.abs(stretch.window - stretch.available)).toBe(vector.lag);
const output = Array.from({ length: vector.channels }, () => new Float32Array(stretch.available));
expect(stretch.read(output, 0)).toBe(output[0].length);
- const expected = expand ? input : planar(vector.output, vector.channels);
+ const expected = planar(vector.output, vector.channels);
for (let channel = 0; channel < vector.channels; channel++) {
expect(output[channel].length).toBe(expected[channel].length);
output[channel].forEach((sample, i) => {
@@ -66,54 +68,79 @@ for (const vector of vectors) {
});
}
-it("decides on the excess above the target, and never to slow down", () => {
+for (const vector of vectors.decisions) {
+ it(`decides the corpus inputs: ${vector.name}`, () => {
+ const { rate, cooldown, decision, allowance, ...inputs } = vector;
+ expect(decide(rate, inputs, cooldown)).toEqual({ decision, allowance });
+ });
+}
+
+for (const vector of vectors.filter) {
+ it(`filters the corpus levels: ${vector.name}`, () => {
+ const level = new Level();
+ vector.steps.forEach(([value, stretched], step) => {
+ const actual = level.update(vector.rate, vector.target, value, stretched);
+ expect(Math.abs(actual - vector.filtered[step])).toBeLessThanOrEqual(1e-9);
+ });
+ });
+}
+
+it("takes what the stretch changed out of the next filter update", () => {
+ // 48 kHz at a 100 ms target: a constant signal accelerates by its shortest period, 120 frames,
+ // from a buffer at the high limit.
+ const inputs = { target: 4800, level: 4800, raw: 4800, lag: 120, bound: 0 };
const stretch = new Stretch(48000, 1);
+ stretch.input[0].fill(0.25);
stretch.cooldown = 0;
- // At or short of the target is audio on the clock or arrivals running late: either way, hold.
- expect(stretch.decide(0)).toBe("Hold");
- expect(stretch.decide(-48000)).toBe("Hold");
- // Any excess accelerates, so the ring rests on its target rather than in a band above it.
- expect(stretch.decide(1)).toBe("Accelerate");
- expect(stretch.decide(959)).toBe("Accelerate");
- // From NetEq's 20 ms granularity, fast-accelerate, even straight after a stretch.
- expect(stretch.decide(960)).toBe("Fast");
- stretch.cooldown = 1;
- expect(stretch.decide(959)).toBe("Hold");
- expect(stretch.decide(960)).toBe("Fast");
+ expect(stretch.process(stretch.window, inputs)).toBe("Accelerate");
+ expect(stretch.available).toBe(stretch.window - 120);
+ // The first update takes the level as it is, the next takes out the period removed.
+ expect(stretch.level.filtered).toBe(4800);
+ stretch.process(stretch.window, inputs);
+ expect(stretch.level.filtered).toBe(4680);
+ // A reset starts again from the level.
+ stretch.level.reset();
+ stretch.process(stretch.window, inputs);
+ expect(stretch.level.filtered).toBe(4800);
});
-it("passes a short, disabled, exhausted, or late window through unchanged", () => {
+it("passes a short, input-less, or exhausted window through unchanged", () => {
for (const rate of [8000, 44100, 48000, 96000]) {
const stretch = new Stretch(rate, 2);
for (const channel of stretch.input) channel.fill(0.25);
stretch.cooldown = 0;
- // A ring well above its target accelerates a full window of received PCM.
- stretch.process(stretch.window, stretch.window, true);
- expect(stretch.available).toBeLessThan(stretch.window);
+ // Ahead of the clock by a second: expand back onto it.
+ const target = Math.floor(rate / 10);
+ const inputs = { target, level: target, raw: target, lag: -rate, bound: 0 };
+ expect(stretch.process(stretch.window, inputs)).toBe("Expand");
+ expect(stretch.available).toBeGreaterThan(stretch.window);
const output = [new Float32Array(rate), new Float32Array(rate)];
expect(stretch.read(output, 0)).toBe(stretch.length);
expect(stretch.read(output, 0)).toBe(0);
stretch.cooldown = 0;
- stretch.process(stretch.window, stretch.window, false);
+ stretch.process(stretch.window);
expect(stretch.available).toBe(stretch.window);
- stretch.process(stretch.window >> 1, stretch.window, true);
+ stretch.process(stretch.window >> 1, inputs);
expect(stretch.available).toBe(stretch.window >> 1);
- // A ring a whole window short of its target still plays it as received.
- stretch.process(stretch.window, -stretch.window, true);
- expect(stretch.available).toBe(stretch.window);
}
});
-it("accelerates only by a pitch period that fits in the excess", () => {
+it("never moves a seam past its allowance", () => {
// A constant signal's shortest period is 2.5 ms: 120 frames at 48 kHz.
- const stretch = new Stretch(48000, 1);
- stretch.input[0].fill(0.25);
- stretch.cooldown = 0;
- stretch.process(stretch.window, 119, true);
- expect(stretch.available).toBe(stretch.window);
- stretch.cooldown = 0;
- stretch.process(stretch.window, 120, true);
- expect(stretch.available).toBe(stretch.window - 120);
+ const cases: [Decision, number, number][] = [
+ ["Accelerate", 119, 1440],
+ ["Accelerate", 120, 1320],
+ ["Accelerate", 239, 1320],
+ ["Fast", 360, 1080],
+ ["Expand", 119, 1440],
+ ["Expand", 120, 1560],
+ ];
+ for (const [decision, allowance, length] of cases) {
+ const stretch = new Stretch(48000, 1);
+ stretch.input[0].fill(0.25);
+ stretch.stage(stretch.window, { decision, allowance });
+ expect(stretch.length).toBe(length);
+ }
});
diff --git a/js/watch/src/audio/stretch.ts b/js/watch/src/audio/stretch.ts
index 31ad99d53f7..1fc4310aaf1 100644
--- a/js/watch/src/audio/stretch.ts
+++ b/js/watch/src/audio/stretch.ts
@@ -1,16 +1,100 @@
/*!
- * Received-PCM adaptation of WebRTC NetEq's accelerate, the shared seams and pitch search
- * doc/concept/audio-jitter.md specifies for this player and moq-audio's. This player never expands:
- * its ring starts on its target, so slowing down only ever settles audio behind the shared clock.
+ * Received-PCM adaptation of WebRTC NetEq's accelerate and preemptive expand: the decision, buffer-level
+ * filter, seams, and pitch search doc/concept/audio-jitter.md specifies for this player and moq-audio's.
*
* Copyright (c) 2012 The WebRTC project authors. All Rights Reserved.
- * Ported from modules/audio_coding/neteq/{time_stretch,accelerate,dsp_helper,
- * audio_vector,decision_logic}.
+ * Ported from modules/audio_coding/neteq/{time_stretch,accelerate,preemptive_expand,dsp_helper,
+ * audio_vector,decision_logic,buffer_level_filter}.
* @license BSD-3-Clause, with WebRTC's patent grant: see WEBRTC-LICENSE and WEBRTC-PATENTS in this package.
*/
/** What a window boundary decides, named as the shared corpus names it. */
-export type Decision = "Hold" | "Accelerate" | "Fast";
+export type Decision = "Hold" | "Accelerate" | "Fast" | "Expand";
+
+/** What a window decides on, in frames at the device rate. */
+export interface Inputs {
+ /** The playout delay the shared clock plays at. */
+ target: number;
+ /**
+ * How far the playhead trailed the newest received audio when it arrived, less what the stretch
+ * removed since: the buffer level with the publisher's cadence taken out. Filtered before deciding.
+ */
+ level: number;
+ /** Received audio ahead of the playhead, decoded or not. */
+ raw: number;
+ /** How far the playhead trails the shared clock: negative when it plays ahead of it. */
+ lag: number;
+ /** How far behind the clock audio may fall: the subscription's max delay less the playout delay. */
+ bound: number;
+}
+
+/** A decision and the most a seam may change the output by in frames: zero when it holds. */
+export interface Verdict {
+ decision: Decision;
+ allowance: number;
+}
+
+/**
+ * The shared decision rule, on a filtered `level`. Fast accelerate and accelerate only ever catch a
+ * playhead behind the clock up to it; expand slows one whose buffer runs low, as far as `bound` allows,
+ * or one ahead of the clock back onto it. `min(level, raw)` is the buffer: a stall delivers no arrival
+ * to lower `level`, so `raw` stops a playhead behind the clock from speeding into it.
+ */
+export function decide(rate: number, inputs: Inputs, cooldown: number): Verdict {
+ const { target, level, raw, lag, bound } = inputs;
+ // NetEq's DecisionLogic limits, with its 85 ms deceleration offset and 20 ms granularity.
+ const low = Math.max(Math.floor((3 * target) / 4), target - Math.floor((rate * 85) / 1000));
+ const high = Math.max(target, low + Math.floor((rate * 20) / 1000));
+ const buffer = Math.min(level, raw);
+ if (lag > 0 && buffer >= 4 * high) return { decision: "Fast", allowance: lag };
+ if (cooldown > 0) return { decision: "Hold", allowance: 0 };
+ if (lag > 0 && buffer >= high) return { decision: "Accelerate", allowance: lag };
+ if (level < low && lag < bound) return { decision: "Expand", allowance: bound - lag };
+ if (lag < 0) return { decision: "Expand", allowance: -lag };
+ return { decision: "Hold", allowance: 0 };
+}
+
+/**
+ * NetEq's buffer-level filter, updated once per window: three of its 10 ms steps, its factor chosen
+ * by the target, less what the stretch changed since the last update. Unset until the first update
+ * after a start or a {@link reset}, which takes the level as it is.
+ */
+export class Level {
+ #filtered?: number;
+
+ /** The filtered level, unset before the first update. */
+ get filtered(): number | undefined {
+ return this.#filtered;
+ }
+
+ /** Fold in `level`, after a window whose stretch removed `stretched` frames (negative: added). */
+ update(rate: number, target: number, level: number, stretched: number): number {
+ const filtered = this.#filtered;
+ if (filtered === undefined) {
+ this.#filtered = level;
+ return level;
+ }
+ // NetEq's BufferLevelFilter factors, in 256ths per 10 ms step, by the target in milliseconds.
+ const factor =
+ target * 1000 <= 20 * rate
+ ? 251
+ : target * 1000 <= 60 * rate
+ ? 252
+ : target * 1000 <= 140 * rate
+ ? 253
+ : 254;
+ const weight = (factor * factor * factor) / 16777216;
+ this.#filtered = Math.max(0, weight * filtered + (1 - weight) * level - stretched);
+ return this.#filtered;
+ }
+
+ /** Forget the history, so the next update starts from its level: after an underrun or a new timeline. */
+ reset(): void {
+ this.#filtered = undefined;
+ }
+}
+
+const HOLD: Verdict = { decision: "Hold", allowance: 0 };
// NetEq's 48 kHz Q12 FIR. Sampling its taps at fractional positions also admits other device rates
// without altering the search domain.
@@ -39,17 +123,20 @@ export class Stretch {
readonly #rate: number;
// 15 ms: the seam's position, and the longest pitch period.
readonly #middle: number;
- // 20 ms: NetEq's adjustment granularity. An excess this large fast-accelerates.
- readonly #band: number;
// 50 ms: NetEq's five 10 ms ticks between stretches.
readonly #rest: number;
#position = 0;
#length = 0;
+ // What the stretch removed since the last filter update, which that update takes out of its history.
+ #stretched = 0;
/** Frames left to render before an ordinary stretch: NetEq's six startup ticks, then its five. */
cooldown: number;
+ /** The buffer-level filter the decisions read. */
+ readonly level = new Level();
+
readonly #down = new Float64Array(DOWN);
readonly #correlation = new Float64Array(LAGS);
@@ -57,11 +144,11 @@ export class Stretch {
this.#rate = rate;
this.window = Math.max(1, Math.floor((rate * 30) / 1000));
this.#middle = Math.floor((rate * 15) / 1000);
- this.#band = Math.floor((rate * 20) / 1000);
this.#rest = Math.floor((rate * 50) / 1000);
this.cooldown = Math.floor((rate * 60) / 1000);
this.input = Array.from({ length: channels }, () => new Float32Array(this.window));
- this.#output = Array.from({ length: channels }, () => new Float32Array(this.window));
+ // An expand adds up to one 15 ms period.
+ this.#output = Array.from({ length: channels }, () => new Float32Array(this.window + this.#middle));
}
/** Output frames staged and not yet read. */
@@ -95,27 +182,41 @@ export class Stretch {
}
/**
- * Stage the first `frames` of {@link input}, accelerated by at most `excess` frames: how far the ring
- * sits behind its target. Only a full window of received PCM can stretch, so a short block, or one
- * that is not `enabled`, passes unchanged and a finite input never becomes concealment.
+ * Stage the first `frames` of {@link input}, stretched as `inputs` decide. Only a full window of
+ * received PCM can stretch, so a short block, or one with no `inputs`, passes unchanged and a
+ * finite input never becomes concealment.
*/
- process(frames: number, excess: number, enabled: boolean): void {
- const decision = enabled && frames === this.window ? this.decide(excess) : "Hold";
- const lag = decision === "Hold" ? 0 : this.#pitch(decision === "Fast", excess);
+ process(frames: number, inputs?: Inputs): Decision {
+ if (!inputs || frames !== this.window) return this.stage(frames, HOLD);
+ const level = this.level.update(this.#rate, inputs.target, inputs.level, this.#stretched);
+ this.#stretched = 0;
+ return this.stage(frames, decide(this.#rate, { ...inputs, level }, this.cooldown));
+ }
+
+ /** Stage the first `frames` of {@link input} as `verdict` decides, if a pitch period admits it. */
+ stage(frames: number, verdict: Verdict): Decision {
+ const { decision } = verdict;
+ const lag = decision === "Hold" ? 0 : this.#pitch(decision, verdict.allowance);
this.#position = 0;
- this.#length = frames - lag;
+ const expand = decision === "Expand";
+ this.#length = expand ? frames + lag : frames - lag;
+ this.#stretched += frames - this.#length;
if (lag === 0) {
for (let channel = 0; channel < this.input.length; channel++) {
const input = this.input[channel];
const output = this.#output[channel];
for (let i = 0; i < frames; i++) output[i] = input[i];
}
- return;
+ return decision;
}
- // Remove `lag` frames after the seam, crossfading the period before the middle into the one after.
+ // Accelerate removes `lag` frames after the seam, crossfading the period before the middle into
+ // the one after; expand adds them, crossfading the period after the middle into the one before.
const middle = this.#middle;
- const seam = middle - lag;
+ const seam = expand ? middle : middle - lag;
+ const first = expand ? middle : middle - lag;
+ const second = expand ? middle - lag : middle;
+ const tail = expand ? middle : middle + lag;
for (let channel = 0; channel < this.input.length; channel++) {
const input = this.input[channel];
const output = this.#output[channel];
@@ -124,32 +225,18 @@ export class Stretch {
// rather than duplicating either neighbour.
for (let i = 0; i < lag; i++) {
const weight = (i + 1) / (lag + 1);
- output[seam + i] = input[seam + i] * (1 - weight) + input[middle + i] * weight;
+ output[seam + i] = input[first + i] * (1 - weight) + input[second + i] * weight;
}
- for (let i = middle + lag; i < frames; i++) output[i - lag] = input[i];
+ for (let i = tail; i < frames; i++) output[seam + lag + i - tail] = input[i];
}
this.cooldown = this.#rest;
+ return decision;
}
- /**
- * What a full window decides given the ring's `excess` frames above its target, measured where a
- * frame has just arrived. Any excess accelerates, so the ring rests on its target rather than
- * anywhere in a band above it, and one past NetEq's 20 ms granularity fast-accelerates, which
- * catches up after a refill about as fast as dropping the backlog would. A ring at or below its
- * target holds: that is either audio on the clock or arrivals running late, and slowing down for
- * either adds latency.
- */
- decide(excess: number): Decision {
- if (excess >= this.#band) return "Fast";
- if (this.cooldown > 0 || excess <= 0) return "Hold";
- return "Accelerate";
- }
-
- // Search the first channel at 4 kHz, then admit the full-rate seam, returning its length or zero.
- // Fast mode admits a weaker match and takes every whole period that fits before the middle. An
- // ordinary accelerate removes one period, and only one that fits in the `excess`, so the ring never
- // lands ahead of its target.
- #pitch(fast: boolean, excess: number): number {
+ // Search the first channel at 4 kHz, then admit the full-rate seam, returning how many frames it
+ // changes the output by, or zero. Fast mode admits a weaker match and takes every whole period that
+ // fits before the middle; the others take one. None takes more than `allowance`.
+ #pitch(decision: Decision, allowance: number): number {
const middle = this.#middle;
if (middle === 0) return 0;
@@ -197,11 +284,16 @@ export class Stretch {
energy2 += b * b;
cross += a * b;
}
+ const fast = decision === "Fast";
const passive = (energy1 + energy2) / (2 * lag) <= PASSIVE;
const threshold = (fast ? 8192 : 14746) / 16384;
if (!passive && cross <= threshold * Math.sqrt(energy1 * energy2)) return 0;
- if (fast) return Math.floor(middle / lag) * lag;
- return lag <= excess ? lag : 0;
+ const periods = fast
+ ? Math.min(Math.floor(middle / lag), Math.floor(allowance / lag))
+ : allowance >= lag
+ ? 1
+ : 0;
+ return periods * lag;
}
// The first channel at a fractional frame, linearly interpolated and clamped to the window.
diff --git a/js/watch/src/element.ts b/js/watch/src/element.ts
index 12d8680f046..fb2edb77f80 100644
--- a/js/watch/src/element.ts
+++ b/js/watch/src/element.ts
@@ -606,8 +606,10 @@ export default class MoqWatch extends HTMLElement {
*
* Zero (the default) minimizes latency. A larger value enables buffered playback: build up
* future-dated frames (e.g. TTS written faster than real-time) and only skip ahead
- * once they would sit further than `delay + lookahead` past the playhead. Call `reset()` at each
- * utterance boundary. Not a jitter buffer: live streams leave it at zero and tune `delay`.
+ * once they would sit further than `delay + lookahead` past the playhead. Audio that falls
+ * behind, as after a stall, catches up by playing faster, and skips only what trails by more
+ * than the lookahead. Call `reset()` at each utterance boundary. Not a jitter buffer: live
+ * streams leave it at zero and tune `delay`.
*/
get lookahead(): Duration {
return this.controls.lookahead.peek();
diff --git a/js/watch/src/sync.ts b/js/watch/src/sync.ts
index 836c91946f8..6da306434ff 100644
--- a/js/watch/src/sync.ts
+++ b/js/watch/src/sync.ts
@@ -25,8 +25,10 @@ export type SyncInput = {
*
* Zero minimizes latency, the live default: playback re-anchors as soon as anything arrives
* early. A larger value lets faster-than-real-time frames (a TTS response with future
- * timestamps) build up instead of being skipped. Always finite, so worst case the audio ring
- * drops its oldest samples rather than exhausting memory. With a nonzero lookahead, the first video
+ * timestamps) build up instead of being skipped. Audio that falls behind, as after a stall,
+ * catches up by playing faster, and skips only what trails by more than the lookahead. Always
+ * finite, so worst case the audio ring drops its oldest samples rather than exhausting memory.
+ * With a nonzero lookahead, the first video
* frame is a poster until every enabled media track holds the resolved delay; `lookahead` is
* the most held ahead, not the amount required to start.
*/
diff --git a/quest/m1/audio-jitter-target/README.md b/quest/m1/audio-jitter-target/README.md
index 8df019a18a8..80682a6374c 100644
--- a/quest/m1/audio-jitter-target/README.md
+++ b/quest/m1/audio-jitter-target/README.md
@@ -72,7 +72,7 @@ reach zero underruns, and some underruns during congestion are expected.
records.
- Tests: no gate may reward moving away from the rule, so the audio harness
grades steady-state lag, the player's own trims, and recovery from a stall
- before [the model](/quest/m1/audio-jitter-target/stretch.md) lands.
+ before [the model](https://github.com/moq-dev/moq/pull/8507) lands.
- Milestone: ✅ every new child in m1, here (rejected: the model and grading
in m0; the setting in m2).
diff --git a/quest/m1/audio-jitter-target/calibrate.md b/quest/m1/audio-jitter-target/calibrate.md
index 5d283405a1b..f99ed24f488 100644
--- a/quest/m1/audio-jitter-target/calibrate.md
+++ b/quest/m1/audio-jitter-target/calibrate.md
@@ -5,7 +5,7 @@
`test/audio-quality/budgets.json` carries every browser and native row's
ceilings re-measured, by the rule in its note, from seven scheduled Nightly
runs on a `main` that includes
-[the stretch model](/quest/m1/audio-jitter-target/stretch.md). The rows budget
+[the stretch model](https://github.com/moq-dev/moq/pull/8507). The rows budget
steady-state lag, the player's own trims and skips, and underruns as
measured, and the note says the model is in them.
@@ -38,6 +38,14 @@ frame above zero on any row but `step`, or a nonzero `player_skips_per_min`
on a row whose stalls fit inside the `stretch` bound. Either means the model
is not doing its job there.
+Watch native-opus near-zero: at [moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507)'s
+head a developer host at load average about 43 filed 64 to 77 tap underruns
+a minute against its ceiling of 42, where two earlier runs filed 0. Decided
+2026-11-11 (maintainer, /quest-merge pick of #8507): ✅ merge and treat it as
+host noise for these nightlies to settle (rejected: re-measure on a quiet
+host first; investigate #8507's review fixes first). If the nightlies
+reproduce it, report it rather than budget it.
+
Native rows stay report-only; enforcing them belongs to
[Audio outlives video under bufferbloat](/quest/m4/audio-bufferbloat-av.md).
@@ -48,7 +56,3 @@ Native rows stay report-only; enforcing them belongs to
schedule` and download each with `gh run download -n
audio-quality-run` as it lands, since artifacts expire after 7 days. Never
dispatch the nightly by hand.
-
-## Required
-
-- [Stretch model](/quest/m1/audio-jitter-target/stretch.md) - the player these ceilings measure
diff --git a/quest/m1/audio-jitter-target/clock-step.md b/quest/m1/audio-jitter-target/clock-step.md
new file mode 100644
index 00000000000..2668f1760b8
--- /dev/null
+++ b/quest/m1/audio-jitter-target/clock-step.md
@@ -0,0 +1,33 @@
+# [M] The shared clock follows a lasting path-delay step
+
+## Goal
+
+After a lasting increase in path delay, audio and video settle on the shared
+clock at the playout delay again, in the browser and natively, instead of
+audio resting behind a clock whose reference never moves later. The `step`
+rows of the audio-quality harness show a steady-state `av_lag_ms` p50 within
+a frame of zero.
+
+## Plan
+
+Decided 2026-11-11 (maintainer, /quest-merge pick of
+[moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507)): keep.
+
+The stretch model deferred this so it would be decided with numbers. At
+#8507's head the live `step` rows rested 35.2 and 41.8 ms behind in the
+browser (AAC, Opus) and 5.7 and 17.3 ms natively, p95 37.5 to 65.1 ms, with
+native AAC at 29.3 underruns a minute (614 under
+[moq-dev/moq#8361](https://github.com/moq-dev/moq/pull/8361)).
+
+Recommendations:
+
+- Let the reference move later only on sustained evidence, such as the
+ minimum offset over a window well past the target, and treat the move like
+ a target rise: audio glides to it by stretching, video holds a frame.
+- Keep one rule in `js/watch` `Sync` and `moq_mux::sync`, with a shared test
+ vector, as the stretch model's decisions corpus does.
+- Do not add latency to reach zero underruns on the step.
+
+## Required
+
+- [Calibrate](/quest/m1/audio-jitter-target/calibrate.md) - the step rows' ceilings to measure against
diff --git a/quest/m1/audio-jitter-target/hole-wait.md b/quest/m1/audio-jitter-target/hole-wait.md
new file mode 100644
index 00000000000..d214765ccc0
--- /dev/null
+++ b/quest/m1/audio-jitter-target/hole-wait.md
@@ -0,0 +1,29 @@
+# [M] Split the consumer's hole wait from its age limit
+
+## Goal
+
+The container consumer waits on a missing group for `delay + lookahead` but
+keeps delivering aged media up to the subscription's max delay, in `js/hang`
+and Rust alike, so a reordered join backlog no longer raises the jitter
+target while a stall's late audio is still delivered for the stretch to
+catch up on.
+
+## Plan
+
+Decided 2026-11-11 (maintainer, /quest-merge pick of
+[moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507)): keep.
+
+#8507 set the browser consumer's single limit to the subscription's max
+delay, as native had, because at `delay + lookahead` the age skip dropped
+what the stretch would catch up on (1651.8 tap underruns a minute on
+relay-bbb, 2123.6 on relay-mic-stall). The cost: relay-mic-rtt40's consumer
+waits up to the max on its reordered join backlog's holes, which the
+estimator counts as jitter, raising its target p95 from 320 to 780 ms.
+
+Recommendations:
+
+- Two limits: skip a hole after `delay + lookahead`, drop media only once it
+ is older than the max.
+- Change both consumers in one PR and keep their tests mirrored.
+- Pass when relay-mic-rtt40's target p95 returns near 320 ms and the stall
+ rows keep #8507's underruns and trims.
diff --git a/quest/m1/audio-jitter-target/render-callback.md b/quest/m1/audio-jitter-target/render-callback.md
new file mode 100644
index 00000000000..e283a045a8e
--- /dev/null
+++ b/quest/m1/audio-jitter-target/render-callback.md
@@ -0,0 +1,32 @@
+# [M] The browser's target counts its render callback
+
+## Goal
+
+The browser's audio target includes the audio its render callback pulls at
+once, and its clock is paired on the wall clock, so the browser rests on the
+shared clock like native does, without the underruns that pairing alone
+caused.
+
+## Plan
+
+Decided 2026-11-11 (maintainer, /quest-merge pick of
+[moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507)): keep.
+
+Headless Chromium renders 2048-frame callbacks (`baseLatency` 42.7 ms), and
+the clock is paired with `currentTime` just after each callback, so the
+browser rests 28 to 35 ms behind the clock where native rests within a few
+ms. A max-filtered wall-clock pairing put audio on the clock (p50 -1.3 to
+4.5 ms) but underran near-zero Opus 45 times a minute, since the target did
+not count the callback.
+
+Recommendations, in order and in one PR:
+
+1. Add the render quantum (`baseLatency`, or the measured callback size) to
+ the browser ring's target.
+2. Then pair the clock on the wall clock with the max filter.
+
+Measure the browser live rows before and after; native is unchanged.
+
+## Required
+
+- [Calibrate](/quest/m1/audio-jitter-target/calibrate.md) - the browser ceilings this is measured against
diff --git a/quest/m1/audio-jitter-target/speech-replay.md b/quest/m1/audio-jitter-target/speech-replay.md
new file mode 100644
index 00000000000..9a305cef15b
--- /dev/null
+++ b/quest/m1/audio-jitter-target/speech-replay.md
@@ -0,0 +1,30 @@
+# [S] Replays play a speech-like signal
+
+## Goal
+
+The audio-quality replays drive the players with a signal whose pitch
+period and level vary like speech, so a replay's lag p95 measures how fast
+the player catches up, not how slowly the time stretch removes periods from
+a constant tone.
+
+## Plan
+
+Decided 2026-11-11 (maintainer, /quest-merge pick of
+[moq-dev/moq#8507](https://github.com/moq-dev/moq/pull/8507)): keep.
+
+On the replays' constant tone, the pitch search removes one 2.5 ms period a
+30 ms window, so every replay p95 lag after a stall is a slow catch-up that
+speech would finish faster (#8507's relay-mic-stall p95: 501.8 ms).
+
+Recommendations:
+
+- Synthesize the signal in the harness (a varying fundamental with
+ syllable-rate level changes and short pauses above the -60 dBFS silence
+ classifier), seeded, so replays stay deterministic and no clip needs a
+ license.
+- Re-measure the replay ceilings in `test/audio-quality/budgets.json` in the
+ same PR, noting the signal change.
+
+## Related
+
+- [ViSQOL](/quest/m4/audio-visqol.md) - scores a recorded clip on the live rows; this changes only the replays' synthetic input
diff --git a/quest/m1/audio-jitter-target/stretch-setting.md b/quest/m1/audio-jitter-target/stretch-setting.md
index 3fac8727b62..1627d08054f 100644
--- a/quest/m1/audio-jitter-target/stretch-setting.md
+++ b/quest/m1/audio-jitter-target/stretch-setting.md
@@ -63,7 +63,3 @@ Recommendations:
Public API: an additive attribute, property, and flag. Wire: none; a
fixed-delay player's audio subscription asks for a longer max delay.
-
-## Required
-
-- [Stretch model](/quest/m1/audio-jitter-target/stretch.md) - the slow-down and catch-up this setting bounds and turns off
diff --git a/quest/m1/audio-jitter-target/stretch.md b/quest/m1/audio-jitter-target/stretch.md
deleted file mode 100644
index 25da7023dd2..00000000000
--- a/quest/m1/audio-jitter-target/stretch.md
+++ /dev/null
@@ -1,133 +0,0 @@
-# [XL] Audio slows down and speeds up around its target in every player
-
-## Goal
-
-`moq play`, a live ``, and a buffered one (`lookahead` above zero)
-follow one decision rule, written once in `doc/concept/audio-jitter.md` and
-held by one set of shared vectors:
-
-- In steady state audio plays on the shared clock at the playout delay.
-- When the audio received ahead of the playhead runs low, playback slows down
- and audio falls behind the clock.
-- Once that buffer is back above the target, playback speeds up until audio is
- on the clock again.
-- After an underrun, audio resumes on a small cushion and slows down until
- the buffer is back at the target.
-- When the target moves, audio follows by stretching. It neither parks in
- silence on a rise nor skips buffered audio on a fall.
-- Audio skips ahead only once it trails by more than the subscription's max
- delay still delivers.
-
-So a congestion stall costs a gap and a catch-up, not the audio that arrived
-late. A buffered player follows the same rule and never skips inside its
-lookahead, which fixes the permanent lag and dropped audio groups of
-[moq-dev/moq#8031](https://github.com/moq-dev/moq/issues/8031).
-
-Boundaries: the estimator, the target formula, and video are unchanged. No
-packet loss concealment. Zero underruns is not a goal. No new public setting:
-the bound stays the max delay each player subscribes with today, until
-[the `stretch` setting](/quest/m1/audio-jitter-target/stretch-setting.md).
-
-## Plan
-
-Planned 2026-11-09 (maintainer, /quest-plan). The
-[epic](/quest/m1/audio-jitter-target/README.md) records the model and why.
-Decisions for this quest:
-
-- Refill after an underrun: ✅ resume on a small cushion, about one stretch
- window plus the render slack, then slow down to the target (rejected:
- refill to the full target, as the browser does today; measure both and pick
- in the PR). Refilling adds up to one target of silence and leaves audio
- that much further behind.
-- Speed: ✅ NetEq's thresholds and rates, untuned, in both directions, with
- fast accelerate at four times the high limit (rejected: a few percent both
- ways; a capped slow-down with NetEq's catch-up). A few percent recovers a
- 2 s lag in over a minute and saves about 1 ms near empty. The browser's
- fast accelerate at 20 ms of excess goes.
-- Buffered mode: ✅ folded in here, and
- [moq-dev/moq#8360](https://github.com/moq-dev/moq/pull/8360) is closed
- (rejected: merge it as is and unify after; fix its four audit items first).
- Its branch `quest/m1/audio-jitter-target/buffered-converge` at `7683e4e076`
- anchors buffered audio to the shared clock and adds buffered replay rows;
- both are worth taking.
-- Order: ✅ after native backpressure (rejected: this first; one quest for
- both). The level this rule reads is everything received ahead of the
- playhead, decoded or still encoded. The browser keeps the part of the
- target above its ring's floor encoded, and a buffered player its whole
- lookahead, while native decodes everything at once. One mirrored rule needs
- both players to split the buffer the same way.
-- Clock step: ✅ deferred (rejected: plan it now; drop it). The shared
- clock's reference only moves earlier, so after a lasting path-delay step
- audio would settle behind a stale clock. Report what the `step` rows' lag
- does in the PR body so the follow-up is decided with numbers.
-
-What the audit of `main` found (2026-11-09, `10772522a5`):
-
-- Native (`rs/moq-audio/src/stretch.rs`, `decide`) expands whenever the raw
- depth is below the target, which is true between every arrival, so audio
- settled 60 to 88 ms behind. NetEq's `DecisionLogic` compares a filtered
- level against `low = max(3/4 target, target - 85 ms)` and
- `high = max(target, low + 20 ms)`; the port kept neither the filter nor
- the low limit.
-- The browser (`js/watch/src/audio/ring-buffer.ts`, `stretch.ts`) never
- expands, fast-accelerates at 20 ms of excess, stays silent after an
- underrun until the whole target refills, parks on a target rise, and turns
- the stretch off in buffered mode.
-- [moq-dev/moq#8361](https://github.com/moq-dev/moq/pull/8361) held native
- audio on the clock and took AAC `step` from 0 to 614 underruns a minute.
- Its own analysis: the clock's reference never moves later, so after the 5
- to 60 ms step the clock-aligned depth sits about 55 ms short of the
- estimator's target, less than AAC's 160 ms PES spacing, and `main` had
- hidden that by settling a burst behind. It already admitted a speed-up only
- inside the excess, so that rule alone prevents nothing here. Under this
- model the slow-down lets audio settle behind the stale clock rather than
- underrun, which is the lag the deferred clock decision is about.
-
-Recommendations:
-
-- NetEq assumes evenly paced packets. A batched publisher such as AAC in TS
- delivers a whole PES at once, so the depth drains by a burst between
- arrivals and its mean sits well under the target on a perfect path.
- NetEq's limits on that mean would settle audio behind, as native does
- today. Keep NetEq's limits, filter, and rates, and choose a level with the
- publisher's cadence removed: the depth right after each arrival less what
- the stretch changed since (the browser's `peak`), or the playhead's lag
- behind the shared clock. Keep a check on the raw depth too, since a stall
- delivers no arrival to decide on. The `bursty-aac` replay row holds this:
- its steady-state lag should match the Opus rows'.
-- A target rise leaves audio ahead of the clock and a fall leaves it behind,
- so both become ordinary slow-down and speed-up. `AudioRingBuffer.park` and
- the skip on a lowered target have no caller left.
-- Rewrite the unit tests that pin the old rule rather than work around them:
- `stretch.test.ts` ("never to slow down"), `ring-buffer.test.ts` ("re-stalls
- and refills to the target", "never expands", "underruns every burst ...
- rather than settling later", the 20 ms fast case), `replay.test.ts` ("never
- behind it", "rather than settling behind the clock"), and
- `stretch_tracks_both_clock_directions_and_target_changes` in
- `rs/moq-audio/src/ring.rs`. The shared corpus
- (`doc/concept/audio-jitter/stretch/corpus.json`) gains decision vectors for
- the new inputs.
-- Re-measure every replay row in this PR; the rule changes all of them.
-- Docs in the same PR: one decision section in `doc/concept/audio-jitter.md`
- replaces the native order and "Browser decision", its "Across renditions"
- sentence on parking and skipping goes, and the `lookahead` docs in
- `js/watch/src/element.ts`, `js/watch/src/sync.ts`, and
- `doc/lib/js/watch.md` say a buffered player catches up by playing faster.
-
-Carried from the native port
-([moq-dev/moq#7856](https://github.com/moq-dev/moq/pull/7856)): the
-maintainer listened on 2026-11-05 and found the stretch inaudible on speech,
-and it costs 16 to 43% of a callback, 1.02 ms of a 42.7 ms period at worst.
-Rerun `just rs bench-playback` and `bun watch/bench/ring.ts`, since the
-decision now runs a filter.
-
-Public API: none expected. Wire: none.
-
-## Closes
-
-- [#8031](https://github.com/moq-dev/moq/issues/8031) - buffered playback loses lip sync and cuts audio out on a live stream
-
-## Related
-
-- [The `stretch` setting](/quest/m1/audio-jitter-target/stretch-setting.md) - makes this quest's skip bound configurable, and zero turns the stretch off
-- [Audio outlives video under bufferbloat](/quest/m4/audio-bufferbloat-av.md) - the audio and video congestion rows this behavior is asserted on
diff --git a/quest/m4/audio-bufferbloat-av.md b/quest/m4/audio-bufferbloat-av.md
index 3c1be80564f..cbb7b123053 100644
--- a/quest/m4/audio-bufferbloat-av.md
+++ b/quest/m4/audio-bufferbloat-av.md
@@ -26,7 +26,7 @@ Decided in planning (2026-10-20):
follows audio, so congested video lags it by design, and steady-state lip
sync belongs to the audio ring's time stretch. Since 2026-11-09 audio may
trail the clock through the congestion and catch up after it
- ([stretch model](/quest/m1/audio-jitter-target/stretch.md)), so these rows
+ ([stretch model](https://github.com/moq-dev/moq/pull/8507)), so these rows
gate audio dropped, not audio late.
- Folded in: the native `pulse` rows become enforced. They report only
because an intermittent two-second device stall voids about one AAC native
diff --git a/rs/moq-audio/src/bus.rs b/rs/moq-audio/src/bus.rs
index a10e13a0181..de6a63640b6 100644
--- a/rs/moq-audio/src/bus.rs
+++ b/rs/moq-audio/src/bus.rs
@@ -153,8 +153,9 @@ struct Feed {
/// Where a channel's audio waits to be mixed.
enum Queue {
- /// A [`Live`]'s jitter buffer, filled without the mixer's lock.
- Live(ring::Consumer),
+ /// A [`Live`]'s jitter buffer, filled without the mixer's lock. Boxed,
+ /// since it dwarfs a positioned queue.
+ Live(Box),
/// A [`Positioned`] channel's interleaved samples, the first of which is
/// mixed at the bus's next position. Filled under the mixer's lock, so a
/// write sees exactly where the mix is.
@@ -379,7 +380,7 @@ impl Bus {
comfort: false,
},
);
- let port = self.open(input, strip, Queue::Live(cons))?;
+ let port = self.open(input, strip, Queue::Live(Box::new(cons)))?;
Ok(Live { prod, port })
}
diff --git a/rs/moq-audio/src/playback/driver.rs b/rs/moq-audio/src/playback/driver.rs
index 75bcce70297..fa9ef91be44 100644
--- a/rs/moq-audio/src/playback/driver.rs
+++ b/rs/moq-audio/src/playback/driver.rs
@@ -59,10 +59,6 @@ const COMMAND_QUEUE: usize = 2 * mixer::MAX_SINKS;
const SYNC_ATTEMPTS: u32 = 8;
const SYNC_DELAY: Duration = Duration::from_millis(4);
-/// Frames the sample-format conversion buffer holds. Comfortably more than any
-/// host's period, so the loop over it almost always runs once.
-const SCRATCH_FRAMES: usize = 2048;
-
/// State shared between the caller's [`Engine`](super::Engine) handles, their
/// sinks, and the driver thread.
///
@@ -863,14 +859,8 @@ impl Driver {
where
T: cpal::SizedSample + cpal::FromSample,
{
- // The mixer works in `f32`, so anything else needs a staging buffer.
- // Allocated once and a whole number of frames long, so however big a
- // buffer the device asks for, the callback loops over this rather than
- // resizing (allocating on the audio thread is the one thing it must
- // never do).
let channels = config.channels as usize;
let sample_rate = config.sample_rate as f64;
- let mut scratch = vec![0.0f32; SCRATCH_FRAMES * channels];
let mut epoch = None;
device
@@ -884,13 +874,7 @@ impl Driver {
+ Duration::from_secs_f64((data.len() / channels) as f64 / sample_rate);
mixer.complete(now);
mixer.latency(timestamp.playback.checked_duration_since(timestamp.callback));
- for chunk in data.chunks_mut(scratch.len()) {
- let scratch = &mut scratch[..chunk.len()];
- mixer.fill(scratch, played_at);
- for (out, sample) in chunk.iter_mut().zip(scratch.iter()) {
- *out = T::from_sample(*sample);
- }
- }
+ mixer.fill(data, played_at, crate::ring::host());
},
move |error| {
failures.report(&error);
@@ -1066,11 +1050,11 @@ mod tests {
let (retired, _rx) = sync_channel(8);
let mut mixer = Mixer::new(wired.mixer, retired, 48_000, Layout::Stereo, Default::default()).unwrap();
let mut out = vec![0.0; 4096 * 2];
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
let samples: Vec<_> = [0.25f32; 123 * 2].iter().flat_map(|s| s.to_le_bytes()).collect();
assert_eq!(sink.write(&samples).unwrap().accepted_sample_frames, 123);
let mut drain = std::pin::pin!(sink.finish());
- mixer.fill(&mut out, Duration::from_millis(100));
+ mixer.fill(&mut out, Duration::from_millis(100), crate::ring::host());
mixer.complete(Duration::from_millis(100));
wired.shared.unbind();
assert!(wired.shared.sync());
diff --git a/rs/moq-audio/src/playback/mixer.rs b/rs/moq-audio/src/playback/mixer.rs
index 8b66edcadb2..920e6264845 100644
--- a/rs/moq-audio/src/playback/mixer.rs
+++ b/rs/moq-audio/src/playback/mixer.rs
@@ -102,7 +102,7 @@ pub(super) enum Command {
/// Start mixing a sink.
Add {
id: u64,
- cons: ring::Consumer,
+ cons: Box,
gain: Arc,
completion: Arc,
},
@@ -117,7 +117,7 @@ pub(super) enum Command {
/// One sink being mixed, owned by the audio thread until it is retired.
pub(super) struct Entry {
id: u64,
- cons: ring::Consumer,
+ cons: Box,
gain: Arc,
completion: Arc,
/// Device deadline of the last period containing this sink's samples.
@@ -160,6 +160,8 @@ pub(super) struct Mixer {
/// Channels the device takes, which is also the bus layout's: every sink
/// remixes into it on its way here.
channels: usize,
+ /// The device's rate, which every sink is resampled to.
+ rate: u32,
/// Per-frame gain step, so any change spans [`RAMP`] regardless of rate.
step: f32,
/// Accumulator for one chunk, in the device's layout.
@@ -205,6 +207,7 @@ impl Mixer {
commands,
retired,
channels,
+ rate,
step: 1.0 / (rate as f32 * RAMP),
bus: vec![0.0; CHUNK * channels],
scratch: vec![0.0; CHUNK * channels],
@@ -247,8 +250,19 @@ impl Mixer {
self.ledger.latency(latency);
}
- /// Fill one device buffer, interleaved at the device's channel count.
- pub(super) fn fill(&mut self, out: &mut [f32], played_at: Duration) {
+ /// Fill one device buffer, interleaved at the device's channel count and in
+ /// its sample format, whose first frame the device reads at `start` on the
+ /// host clock.
+ ///
+ /// The whole buffer in one call, converting as each chunk goes out, so a
+ /// buffer of any size is read against one clock reading rather than one per
+ /// staging pass.
+ pub(super) fn fill>(
+ &mut self,
+ out: &mut [T],
+ played_at: Duration,
+ start: ring::Instant,
+ ) {
loop {
match self.commands.try_recv() {
Ok(Command::Add {
@@ -304,6 +318,7 @@ impl Mixer {
let Self {
entries,
channels,
+ rate,
step,
bus,
scratch,
@@ -319,10 +334,14 @@ impl Mixer {
let channels = *channels;
let total = out.len() / channels;
+ // Each chunk starts as many frames past the callback's start as precede
+ // it, so a later chunk is not read as ahead of the clock by the chunks
+ // before it.
let mut done = 0;
while done < total {
let frames = (total - done).min(CHUNK);
let samples = frames * channels;
+ let now = start + Duration::from_secs_f64(done as f64 / *rate as f64);
bus[..samples].fill(0.0);
@@ -344,7 +363,7 @@ impl Mixer {
// stays on the timeline instead of queueing up and jumping when
// it unmutes. The status only reports under/overflow, which the
// channel has already corrected for.
- let status = entry.cons.read(&mut scratch[..samples]);
+ let status = entry.cons.read_at(&mut scratch[..samples], now, total - done);
if !matches!(status, Read::NotReady | Read::Underflow { frames: 0 }) {
entry.played_at = played_at;
}
@@ -417,7 +436,12 @@ impl Mixer {
reference.push(stereo);
}
- out[done * channels..(done + frames) * channels].copy_from_slice(&bus[..samples]);
+ for (out, sample) in out[done * channels..(done + frames) * channels]
+ .iter_mut()
+ .zip(&bus[..samples])
+ {
+ *out = T::from_sample(*sample);
+ }
done += frames;
}
@@ -495,7 +519,7 @@ mod tests {
self.commands
.send(Command::Add {
id,
- cons,
+ cons: Box::new(cons),
gain,
completion: Arc::new(super::super::sink::Completion::new(
super::super::driver::Commands::default(),
@@ -513,7 +537,7 @@ mod tests {
/// its first callback.
fn fill(&mut self, out: &mut [f32]) {
self.mixer.complete(Duration::ZERO);
- self.mixer.fill(out, Duration::ZERO);
+ self.mixer.fill(out, Duration::ZERO, ring::host());
}
/// Fill twice and hand back the buffer, so assertions look at gain that has
@@ -574,6 +598,62 @@ mod tests {
prod.push(&vec![value; frames * STEREO]);
}
+ /// A callback longer than a chunk reads each chunk where it starts, so a
+ /// stretched sink anchored exactly on its clock stays there with nothing
+ /// to correct, rather than reading every later chunk as ahead of it.
+ #[test]
+ fn each_chunk_of_a_callback_reads_the_clock_where_it_starts() {
+ let mut harness = Harness::new(Layout::Stereo);
+ let (mut prod, mut cons) = ring::channel(
+ STEREO,
+ RATE,
+ RATE,
+ ring::Config {
+ latency: Duration::from_millis(10),
+ capacity: Duration::from_secs(2),
+ pad: false,
+ trim: false,
+ comfort: true,
+ },
+ );
+ cons.stretch(RATE);
+ harness
+ .commands
+ .send(Command::Add {
+ id: 0,
+ cons: Box::new(cons),
+ gain: Arc::new(Gain::new()),
+ completion: Arc::new(super::super::sink::Completion::new(
+ super::super::driver::Commands::default(),
+ )),
+ })
+ .unwrap();
+
+ // One read marks the sink ready, then a second of tone is due exactly
+ // as it would play from the next callback on.
+ let start = ring::host();
+ harness.mixer.fill(&mut [0.0; 128 * STEREO], Duration::ZERO, start);
+ let tone: Vec = (0..RATE as usize)
+ .flat_map(|i| [(i as f32 * std::f32::consts::TAU * 200.0 / RATE as f32).sin() * 0.25; STEREO])
+ .collect();
+ assert_eq!(prod.push(&tone), Push::Ok);
+ // Four chunks to a callback, in a format the mixer converts to as it
+ // goes, so none of them restarts the clock.
+ let callback = 4 * CHUNK;
+ let mut now = start + Duration::from_millis(10);
+ prod.due(now + prod.buffered(), Duration::from_secs(2), Duration::ZERO);
+
+ let period = Duration::from_secs_f64(callback as f64 / RATE as f64);
+ let mut out = vec![0i16; callback * STEREO];
+ for _ in 0..10 {
+ harness.mixer.fill(&mut out, Duration::ZERO, now);
+ now += period;
+ }
+ let entry = &harness.mixer.entries[0];
+ assert_eq!(entry.cons.accelerated(), 0);
+ assert_eq!(entry.cons.skips(), 0);
+ }
+
#[test]
fn discards_writes_until_the_device_reads() {
let mut harness = Harness::new(Layout::Stereo);
diff --git a/rs/moq-audio/src/playback/sink.rs b/rs/moq-audio/src/playback/sink.rs
index 2e01e096fcf..2d3a4ab7c34 100644
--- a/rs/moq-audio/src/playback/sink.rs
+++ b/rs/moq-audio/src/playback/sink.rs
@@ -7,6 +7,8 @@ use std::sync::{Arc, Mutex};
use std::task::{Context, Poll};
use std::time::Duration;
+use kio::clock::Instant;
+
use super::driver::Shared;
use super::mixer::{self, Gain};
use crate::resample::Remix;
@@ -166,10 +168,11 @@ impl Sink {
/// how far ahead of the speaker you are, which is the anchor an A/V sync
/// clock steers video by.
///
- /// Received audio is time-stretched toward the current latency. A writer
- /// that outruns that correction eventually overflows and drops the excess;
- /// one that leaves no received audio underruns and plays silence. Both are
- /// logged and neither is an error, since the next write can recover.
+ /// Received audio is time-stretched toward the current latency, or toward
+ /// the clock [`due`](Self::due) anchors it to. A writer that outruns that
+ /// correction eventually overflows and drops the excess; one that leaves no
+ /// received audio underruns and plays silence. Both are logged and neither
+ /// is an error, since the next write can recover.
/// The returned [`Write`] counts input sample frames accepted and dropped;
/// dropped live audio should be observed for telemetry, not retried.
pub fn write(&mut self, samples: &[u8]) -> Result {
@@ -254,6 +257,10 @@ impl Sink {
/// Set the playout target, converging by stretching received PCM.
///
+ /// This is the buffer the sink keeps against jitter: it slows down while
+ /// what it holds runs short of it, and speeds up past it only to catch up
+ /// with the clock [`due`](Self::due) anchors it to.
+ ///
/// Must be non-zero, at most [`Input::LATENCY_MAX`], and fit in the current
/// ring. Each device open or rebuild allocates the then-current target plus
/// three seconds, less a 30 ms processing window reserved as headroom.
@@ -272,6 +279,30 @@ impl Sink {
Ok(())
}
+ /// Anchor the audio written so far to the caller's clock: its end is due to
+ /// sound at `at`.
+ ///
+ /// The sink then plays onto that clock rather than at its latency alone.
+ /// Behind it, it speeds up once it holds more than its latency; ahead of
+ /// it, it slows down. Its buffer running low slows it down too, falling
+ /// behind the clock by at most `late`, and audio trailing the clock by more
+ /// than `late` plus one 30 ms window is skipped. `jumped` is how far the
+ /// clock's reference moved earlier since the last call, which skips as much
+ /// audio as trails the clock, where a lower latency moves the clock ahead
+ /// by speeding up. Call it after each write and whenever the clock moves;
+ /// until the first call, or after the device changes, the latency alone is
+ /// the clock.
+ pub fn due(&mut self, at: Instant, late: Duration, jumped: Duration) {
+ // The device thread reads std's clock, so carry the distance from now
+ // over to it rather than the instant.
+ let (now, host) = (Instant::now(), crate::ring::host());
+ let at = match at.checked_duration_since(now) {
+ Some(ahead) => host + ahead,
+ None => host.checked_sub(now.duration_since(at)).unwrap_or(host),
+ };
+ self.channel.lock().unwrap().prod.due(at, late, jumped);
+ }
+
/// The PCM layout this sink was built with.
pub fn input(&self) -> &Input {
&self.input
@@ -400,8 +431,9 @@ pub(super) struct Registration {
channel: Arc>,
gain: Arc,
/// The consumer waiting to be handed to a mixer. Taken once it is attached,
- /// and refilled by [`rebuild`](Self::rebuild).
- pending: Option,
+ /// and refilled by [`rebuild`](Self::rebuild). Boxed, since it moves
+ /// through the mixer's command channel by value.
+ pending: Option>,
}
impl Registration {
@@ -448,7 +480,7 @@ impl Registration {
let (mut channel, cons) = channel(&input, rate, bus);
channel.prod.carry(¤t.prod);
*current = channel;
- self.pending = Some(cons);
+ self.pending = Some(Box::new(cons));
}
}
@@ -490,7 +522,7 @@ pub(super) fn new(
input: sink.input.clone(),
channel,
gain,
- pending: Some(cons),
+ pending: Some(Box::new(cons)),
};
Ok((sink, registration))
@@ -553,7 +585,7 @@ mod tests {
commands: super::super::driver::Commands::default(),
});
let (sink, mut registration) = new(0, output_rate, bus, input, shared, engine).unwrap();
- (sink, registration.pending.take().unwrap())
+ (sink, *registration.pending.take().unwrap())
}
#[test]
@@ -567,8 +599,8 @@ mod tests {
let (retired, _rx) = std::sync::mpsc::sync_channel(8);
let mut mixer = mixer::Mixer::new(rx, retired, 48_000, Layout::Stereo, Default::default()).unwrap();
registration.attach(&commands);
- let mut out = [0.0; 960];
- mixer.fill(&mut out, Duration::from_millis(100));
+ let mut out = [0.0f32; 960];
+ mixer.fill(&mut out, Duration::from_millis(100), crate::ring::host());
let samples = [0.25f32; 123 * 2];
let bytes: Vec<_> = samples.iter().flat_map(|s| s.to_le_bytes()).collect();
assert_eq!(sink.write(&bytes).unwrap().accepted_sample_frames, 123);
@@ -577,14 +609,14 @@ mod tests {
assert!(Pin::new(&mut drain).poll(&mut cx).is_pending());
let mut heard = 0;
while heard < 123 * 2 {
- mixer.fill(&mut out, Duration::from_millis(100));
+ mixer.fill(&mut out, Duration::from_millis(100), crate::ring::host());
heard += out.iter().filter(|s| **s == 0.25).count();
registration.complete();
assert!(Pin::new(&mut drain).poll(&mut cx).is_pending());
}
assert_eq!(heard, 123 * 2);
// Another staging chunk in the same callback does not finish the drain.
- mixer.fill(&mut out, Duration::from_millis(100));
+ mixer.fill(&mut out, Duration::from_millis(100), crate::ring::host());
registration.complete();
assert!(Pin::new(&mut drain).poll(&mut cx).is_pending());
// Later callbacks can still run before the device plays its queued tail.
@@ -613,35 +645,40 @@ mod tests {
let mut mixer = mixer::Mixer::new(rx, retired, 48_000, Layout::Stereo, shared.observe.clone()).unwrap();
registration.attach(&commands);
let mut out = [0.0; 1024];
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
assert!(observer.read().unwrap().stalled, "nothing written yet");
let tone: Vec<_> = [0.25f32; 2048].iter().flat_map(|s| s.to_le_bytes()).collect();
let _ = sink.write(&tone).unwrap();
for _ in 0..4 {
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
}
- let _ = sink.write(&tone).unwrap();
- mixer.fill(&mut out, Duration::ZERO);
+ // Running dry waits for a 30 ms window and the device period to resume.
+ let refill: Vec<_> = [0.25f32; 2 * (1440 + 512)]
+ .iter()
+ .flat_map(|s| s.to_le_bytes())
+ .collect();
+ let _ = sink.write(&refill).unwrap();
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
let observation = observer.read().unwrap();
assert!(!observation.stalled);
assert_eq!(observation.gaps.len(), 1, "the sink ran dry once: {observation:?}");
// A finished sink still plays its tail: no stall while it lasts.
let _drain = sink.finish();
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
let observation = observer.read().unwrap();
assert!(!observation.stalled && observation.stalls.is_empty(), "{observation:?}");
let tail = observation.clock;
for _ in 0..4 {
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
}
{
let mut cx = Context::from_waker(std::task::Waker::noop());
let mut finish = std::pin::pin!(observer.finish());
assert!(finish.as_mut().poll(&mut cx).is_pending());
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
assert!(finish.as_mut().poll(&mut cx).is_ready());
}
// Once the tail runs out nothing plays. The gap where it ran out lands in
@@ -683,8 +720,8 @@ mod tests {
};
let rms =
|samples: &[f32]| (samples.iter().map(|&s| s as f64 * s as f64).sum::() / samples.len() as f64).sqrt();
- let mut out = [0.0; 960];
- mixer.fill(&mut out, Duration::ZERO);
+ let mut out = [0.0f32; 960];
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
// 100 ms of a quiet background, then a declared end.
let quiet = 0.0002;
@@ -692,7 +729,7 @@ mod tests {
sink.end();
let mut heard = Vec::new();
for _ in 0..150 {
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
heard.extend_from_slice(&out);
}
// The primed silence, the background, then a second of noise at its level.
@@ -705,7 +742,7 @@ mod tests {
let _ = sink.write(&pcm(0.5, 960)).unwrap();
let mut until = 0;
loop {
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
if let Some(at) = out.iter().position(|s| *s == 0.5) {
until += at / 2;
break;
@@ -716,7 +753,7 @@ mod tests {
assert_eq!(until, 48_000 * 40 / 1000);
for _ in 0..10 {
let _ = sink.write(&pcm(0.5, 480)).unwrap();
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
}
let observation = observer.read().unwrap();
@@ -725,10 +762,10 @@ mod tests {
// Once the next utterance plays, running dry is a gap again.
for _ in 0..20 {
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
}
let _ = sink.write(&pcm(0.5, 2_400)).unwrap();
- mixer.fill(&mut out, Duration::ZERO);
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
assert_eq!(observer.read().unwrap().gaps.len(), 1);
}
@@ -761,8 +798,8 @@ mod tests {
let (retired, _rx) = std::sync::mpsc::sync_channel(8);
let mut mixer = mixer::Mixer::new(rx, retired, 48_000, Layout::Stereo, Default::default()).unwrap();
registration.attach(&commands);
- let mut out = [0.0; 960];
- mixer.fill(&mut out, Duration::ZERO);
+ let mut out = [0.0f32; 960];
+ mixer.fill(&mut out, Duration::ZERO, crate::ring::host());
let bytes: Vec<_> = [0.25f32]
.repeat(frames * 2)
@@ -778,7 +815,7 @@ mod tests {
let mut heard = 0;
for period in 1..100 {
let now = Duration::from_millis(10 * period);
- mixer.fill(&mut out, now);
+ mixer.fill(&mut out, now, crate::ring::host());
mixer.complete(now);
registration.complete();
heard += out.iter().step_by(2).filter(|left| **left > 0.125).count();
diff --git a/rs/moq-audio/src/ring.rs b/rs/moq-audio/src/ring.rs
index 9019ad02a69..936653aab43 100644
--- a/rs/moq-audio/src/ring.rs
+++ b/rs/moq-audio/src/ring.rs
@@ -4,8 +4,10 @@
//! reader-rate frames and both its size and what it admits are counted in them.
//! It opens primed with silence to its latency, can pad back up to it when the
//! writer falls behind, and can trim back down to it when the writer runs ahead.
-//! Native playback instead enables received-audio stretch; only an actual
-//! underrun refills with emergency silence, which never enters the pitch search.
+//! Native playback instead enables received-audio stretch, which plays the
+//! queue onto the writer's clock (see [`Producer::due`]) and skips only audio
+//! that trails it too far; an actual underrun refills with emergency silence,
+//! which never enters the pitch search, until one window is queued again.
//! A silence the writer declared is not an underrun: the reader fills it with
//! comfort noise, and the next utterance plays one latency after it resumes.
//! Neither end allocates, locks, or blocks after construction, so either may
@@ -30,6 +32,17 @@ use rubato::{Async, FixedAsync, PolynomialDegree, Resampler as _};
/// The most input frames resampled at once.
pub(crate) const CHUNK: usize = 512;
+/// An instant on the host clock. The reader runs on the device thread, outside
+/// any runtime, so both ends count time on the clock its callbacks play on.
+#[expect(clippy::disallowed_types, reason = "the device thread runs on the host clock")]
+pub(crate) type Instant = std::time::Instant;
+
+/// The host clock's current instant.
+#[expect(clippy::disallowed_methods, reason = "the device thread runs on the host clock")]
+pub(crate) fn host() -> Instant {
+ Instant::now()
+}
+
/// How a queue is sized and corrected, in time at the reader's rate.
pub(crate) struct Config {
/// Audio held between a push and the read that returns it.
@@ -87,6 +100,24 @@ struct Shared {
/// correction. One publication avoids sampling the ring and staging buffer
/// on opposite sides of a window transfer.
consumed: AtomicUsize,
+ /// Frames of the writer's stream the reader has played or skipped:
+ /// `consumed` before the stretch's length correction.
+ played: AtomicUsize,
+ /// The writer's end less `played`, at its latest push. Adding the reader's
+ /// `played` less `consumed` gives the depth that push left, less what the
+ /// stretch removed since: the level the stretch decides on.
+ level: AtomicUsize,
+ /// The writer's clock, once [`clocked`](Self::clocked): the writer's end
+ /// less the frames from [`Producer::base`] to when it is due, as an `i32`
+ /// in the low half, and how far the reader may trail it before it skips
+ /// ahead in the high half. One word, so the reader never pairs one anchor's
+ /// clock with another's bound. Adding the frames to now less `consumed` to
+ /// the clock gives how far the reader trails it.
+ anchor: AtomicU64,
+ clocked: AtomicBool,
+ /// Frames the writer's clock jumped ahead by moving its reference earlier,
+ /// since the reader last looked. See [`Producer::due`].
+ jumped: AtomicUsize,
target: AtomicUsize,
finished: AtomicBool,
stretch: AtomicBool,
@@ -135,12 +166,15 @@ pub(crate) fn channel(channels: usize, input_rate: u32, output_rate: u32, config
let shared = Arc::new(Shared {
target: AtomicUsize::new(latency),
+ level: AtomicUsize::new(latency),
..Default::default()
});
+ let base = host();
let producer = Producer {
ring: prod,
channels,
rate: output_rate,
+ base,
convert,
latency: config.latency,
written: latency,
@@ -152,10 +186,15 @@ pub(crate) fn channel(channels: usize, input_rate: u32, output_rate: u32, config
let consumer = Consumer {
ring: cons,
channels,
+ rate: output_rate,
+ base,
+ now: 0,
+ played: 0,
stretch: None,
staged_prime: false,
refill: false,
prime: latency,
+ demand: 0,
consumed: 0,
position: 0,
cleared: 0,
@@ -163,6 +202,8 @@ pub(crate) fn channel(channels: usize, input_rate: u32, output_rate: u32, config
comfort: crate::comfort::Comfort::new(output_rate, channels),
measure: config.comfort,
trim: config.trim,
+ jumped: 0,
+ joined: false,
skips: 0,
trimmed: 0,
accelerated: 0,
@@ -177,6 +218,8 @@ pub(crate) struct Producer {
channels: usize,
/// The reader's rate, which every queued frame is at.
rate: u32,
+ /// Where both ends count time from.
+ base: Instant,
/// Boxed, since the resampler state dwarfs the rest and a producer moves
/// through the mixer's command channel by value.
convert: Option>,
@@ -267,6 +310,13 @@ impl Producer {
if !self.shared.read.load(Ordering::Relaxed) {
return Push::NotReady;
}
+ let push = self.queue(input);
+ let level = self.head().wrapping_sub(self.shared.played.load(Ordering::Relaxed));
+ self.shared.level.store(level, Ordering::Relaxed);
+ push
+ }
+
+ fn queue(&mut self, input: &[f32]) -> Push {
if input.len() >= self.channels && std::mem::take(&mut self.next) {
self.resume();
}
@@ -306,6 +356,38 @@ impl Producer {
Push::Ok
}
+ /// Where the audio pushed so far ends, in frames: what was queued, and what
+ /// the resampler still owes for input it holds.
+ fn head(&self) -> usize {
+ let owed = self.convert.as_ref().map_or(0, |convert| convert.owed());
+ self.written.wrapping_add(owed)
+ }
+
+ /// Anchor what was pushed so far to the writer's clock: its end is due to
+ /// play at `at`. The stretch then plays the queue onto that clock, and the
+ /// reader skips audio trailing it by more than `late` plus one window.
+ /// `jumped` is how far the clock's reference moved earlier since the last
+ /// anchor, which the reader skips too, as far as it trails the clock.
+ /// Without an anchor, the latency is the clock.
+ pub(crate) fn due(&mut self, at: Instant, late: Duration, jumped: Duration) {
+ let frames = |duration: Duration| (duration.as_secs_f64() * self.rate as f64).round() as usize;
+ let at = match at.checked_duration_since(self.base) {
+ Some(since) => frames(since),
+ None => frames(self.base - at).wrapping_neg(),
+ };
+ // Within a few seconds of the writer's end, unless the caller's clock is
+ // off by hours, which saturates rather than wraps.
+ let clock = (self.head().wrapping_sub(at) as isize).clamp(i32::MIN as isize, i32::MAX as isize) as i32;
+ let late = u32::try_from(frames(late)).unwrap_or(u32::MAX);
+ // Released, so a reader that sees it also sees the target set before it.
+ self.shared
+ .anchor
+ .store(u64::from(late) << 32 | u64::from(clock as u32), Ordering::Release);
+ self.shared.clocked.store(true, Ordering::Release);
+ // After the anchor, so a reader that sees the jump also sees the clock it moved.
+ self.shared.jumped.fetch_add(frames(jumped), Ordering::Release);
+ }
+
/// Convert silence until the resampler has emitted the full duration of
/// everything pushed so far, returning false if the queue filled first.
///
@@ -437,6 +519,14 @@ impl Convert {
accepted
}
+ /// Output frames the input admitted so far is still owed, the filter's lag
+ /// included.
+ fn owed(&self) -> usize {
+ let owed = (self.fed as u128 * self.output_rate as u128 / self.input_rate as u128) as u64
+ + self.resampler.output_delay() as u64;
+ owed.saturating_sub(self.emitted) as usize
+ }
+
/// Pad with silence a chunk at a time until the output covers the filter's
/// lag and the input's whole duration, returning false if the ring filled
/// first.
@@ -483,14 +573,24 @@ impl Convert {
pub(crate) struct Consumer {
ring: HeapCons,
channels: usize,
+ rate: u32,
+ /// Where both ends count time from.
+ base: Instant,
+ /// Frames from `base` to the start of the current read.
+ now: usize,
+ /// See [`Shared::played`].
+ played: usize,
stretch: Option>,
/// Whether the window staged in the stretcher is startup cushion.
staged_prime: bool,
- /// An actual underrun waits for received PCM, emitting emergency silence
- /// directly rather than putting padding through the pitch search.
+ /// An actual underrun waits for one window of received PCM and the read
+ /// that plays it, emitting emergency silence directly rather than putting
+ /// padding through the pitch search.
refill: bool,
/// The startup cushion is silence, not received audio to stretch.
prime: usize,
+ /// Frames left in the device's callback as of the current read.
+ demand: usize,
consumed: usize,
/// Frames taken off the ring plus the comfort noise of each resume played
/// out: the writer's `written`, as far as the reader has reached.
@@ -502,8 +602,18 @@ pub(crate) struct Consumer {
comfort: crate::comfort::Comfort,
/// Whether the comfort noise measures what plays; unmeasured, it fills plain silence.
measure: bool,
- /// Trim only after filling three quarters of the current target's headroom.
+ /// Without stretch, trim only after filling three quarters of the current
+ /// target's headroom. Stretch skips by the clock instead.
trim: bool,
+ /// Frames the writer's clock jumped ahead by moving its reference earlier,
+ /// not yet skipped, as far as the queue holds audio past its target, which
+ /// the early media that moved it leaves, and the reader still trails the
+ /// clock.
+ jumped: usize,
+ /// Whether received audio has played on the clock. Until it has, nothing
+ /// trailing the clock was heard, so a join lands on it rather than at the
+ /// bound.
+ joined: bool,
/// Times [`Read::Trimmed`] was returned, ever.
skips: u64,
/// Received frames [`Read::Trimmed`] discarded, ever.
@@ -522,7 +632,17 @@ impl Consumer {
/// Fill `output` with queued interleaved frames, and silence past them.
pub(crate) fn read(&mut self, output: &mut [f32]) -> Read {
+ let demand = output.len() / self.channels;
+ self.read_at(output, host(), demand)
+ }
+
+ /// [`read`](Self::read), starting at `now`, with `demand` frames left in
+ /// the device's callback from there, which may outlast `output`: a refill
+ /// after an underrun waits until they are queued.
+ pub(crate) fn read_at(&mut self, output: &mut [f32], now: Instant, demand: usize) -> Read {
self.start();
+ self.demand = demand;
+ self.now = (now.saturating_duration_since(self.base).as_secs_f64() * self.rate as f64).round() as usize;
let whole = output.len() - output.len() % self.channels;
output[whole..].fill(0.0);
if !self.shared.pushed.load(Ordering::Relaxed) {
@@ -535,6 +655,7 @@ impl Consumer {
return Read::NotReady;
}
+ let skipped = self.skip();
let read = self.fill(&mut output[..whole]);
if read < whole {
if self.declared() {
@@ -550,7 +671,11 @@ impl Consumer {
};
}
- if self.trim {
+ if skipped > 0 {
+ return Read::Trimmed { frames: skipped };
+ }
+
+ if self.trim && self.stretch.is_none() {
let queued = self.available();
let latency = self.shared.target.load(Ordering::Relaxed);
let capacity = self.ring.capacity().get() / self.channels;
@@ -566,6 +691,95 @@ impl Consumer {
Read::Ok
}
+ /// What a stretch window starting `elapsed` frames into this read decides on.
+ fn inputs(&self, elapsed: usize) -> crate::stretch::Inputs {
+ // The anchor before the target, so a new anchor brings the target set
+ // before it.
+ let anchor = self
+ .shared
+ .clocked
+ .load(Ordering::Acquire)
+ .then(|| self.shared.anchor.load(Ordering::Acquire));
+ let target = self.shared.target.load(Ordering::Relaxed);
+ let level = self
+ .shared
+ .level
+ .load(Ordering::Relaxed)
+ .wrapping_add(self.played)
+ .wrapping_sub(self.consumed) as isize;
+ let (lag, bound) = if let Some(anchor) = anchor {
+ let clock = anchor as u32 as i32 as isize as usize;
+ let lag = clock.wrapping_add(self.now + elapsed).wrapping_sub(self.consumed) as isize;
+ (lag, (anchor >> 32) as usize)
+ } else {
+ // Without a clock, the latency is where the reader belongs, and
+ // only an excess past three quarters of the headroom is skipped.
+ let capacity = self.ring.capacity().get() / self.channels;
+ (level - target as isize, capacity.saturating_sub(target) * 3 / 4)
+ };
+ crate::stretch::Inputs {
+ target,
+ level: level.max(0) as f64,
+ raw: self.available(),
+ lag: lag as i64,
+ bound: bound as i64,
+ }
+ }
+
+ /// Skip what trails the clock by more than its bound plus one window, back
+ /// to the bound, or the jumps of its reference past one window, returning
+ /// the frames skipped. Less than that is the stretch's to catch up, as a
+ /// writer's clock drifting fast moves the reference. A join has the bound
+ /// of nothing heard yet, so it lands on the clock past whatever backlog it
+ /// was handed, and a jump before it was never heard. A finished or silenced
+ /// stream plays as written.
+ fn skip(&mut self) -> usize {
+ // Acquired before the anchor, so a jump comes with the clock it moved.
+ let jumped = self.shared.jumped.swap(0, Ordering::Acquire);
+ let Some(window) = self.stretch.as_ref().map(|stretch| stretch.window) else {
+ return 0;
+ };
+ if self.shared.finished.load(Ordering::Relaxed) || self.declared() {
+ return 0;
+ }
+ let inputs = self.inputs(0);
+ let clocked = self.shared.clocked.load(Ordering::Acquire);
+ let bound = if self.joined || !clocked { inputs.bound } else { 0 };
+ self.jumped = if self.joined {
+ // Never more than the audio held past the target, so a skip never
+ // drains the queue below it.
+ let excess = inputs.raw.saturating_sub(inputs.target);
+ (self.jumped + jumped).min(excess).min(inputs.lag.max(0) as usize)
+ } else {
+ 0
+ };
+ self.joined |= clocked && self.prime == 0;
+ let late = if inputs.lag > bound.saturating_add(window as i64) {
+ (inputs.lag - bound) as usize
+ } else {
+ 0
+ };
+ let skip = if self.jumped > window {
+ late.max(self.jumped)
+ } else {
+ late
+ };
+ if skip == 0 {
+ return 0;
+ }
+ let available = self.available();
+ let pushed = self.discard(skip);
+ let skipped = available - self.available();
+ // What a skip leaves of a jump, as past what is queued, skips once it
+ // arrives rather than staying inside the bound for the stretch.
+ self.jumped = self.jumped.saturating_sub(skipped);
+ if skipped > 0 {
+ self.skips += 1;
+ self.trimmed += pushed as u64;
+ }
+ skipped
+ }
+
/// Whether running dry now is a silence the writer declared: an end it has
/// not resumed from, on a queue that is not playing out its last samples.
fn declared(&self) -> bool {
@@ -594,6 +808,7 @@ impl Consumer {
let frames = frames.min(left);
if let Some(output) = output {
self.comfort.fill(&mut output[..frames * self.channels]);
+ self.played = self.played.wrapping_add(frames);
}
self.cushioned += frames;
self.consumed = self.consumed.wrapping_add(frames);
@@ -608,11 +823,20 @@ impl Consumer {
/// Fill `output` from the queue, returning the samples written.
fn fill(&mut self, output: &mut [f32]) -> usize {
if self.refill && !self.shared.finished.load(Ordering::Relaxed) {
+ // One window for the stretch and what is left of the device's
+ // callback: the least that resumes without running dry again at
+ // once, and never more than the start waits for. Expanding then
+ // rebuilds the level rather than this wait.
+ let window = self.stretch.as_ref().map_or(0, |stretch| stretch.window);
+ let cushion = (window + self.demand).min(self.shared.target.load(Ordering::Relaxed));
// Nothing more comes before a declared silence, so its tail plays now.
- if !self.declared() && self.available() < self.shared.target.load(Ordering::Relaxed) {
+ if !self.declared() && self.available() < cushion {
return 0;
}
self.refill = false;
+ if let Some(stretch) = &mut self.stretch {
+ stretch.level.reset();
+ }
}
let channels = self.channels;
let mut done = 0;
@@ -624,11 +848,11 @@ impl Consumer {
}
done += frames * channels;
self.consumed = self.consumed.wrapping_add(frames);
+ self.played = self.played.wrapping_add(frames);
continue;
}
- let buffered = self.ring.occupied_len() / channels;
- let mut frames = buffered;
+ let mut frames = self.ring.occupied_len() / channels;
if let Some((ahead, _)) = self.resume() {
if ahead == 0 {
let room = (output.len() - done) / channels;
@@ -640,7 +864,7 @@ impl Consumer {
// A declared end's tail drains as received: running dry after it is
// silence, not an underrun, so nothing should expand toward the target.
let declared = self.declared();
- let Some(stretch) = self.stretch.as_mut() else {
+ let Some(window) = self.stretch.as_ref().map(|stretch| stretch.window) else {
let frames = frames.min((output.len() - done) / channels);
if frames == 0 {
break;
@@ -654,29 +878,34 @@ impl Consumer {
}
self.position = self.position.wrapping_add(frames);
self.consumed = self.consumed.wrapping_add(frames);
+ self.played = self.played.wrapping_add(frames);
done += read;
continue;
};
- let frames = frames.min(stretch.window);
+ let frames = frames.min(window);
if frames == 0 {
break;
}
// Never transform startup padding or a finite stream's tail.
let enabled = self.prime == 0 && !self.shared.finished.load(Ordering::Relaxed) && !declared;
+ let inputs = (enabled && frames == window).then(|| self.inputs(done / channels));
+ let stretch = self.stretch.as_mut().expect("a window comes from the stretch");
let frames = if self.prime > 0 { frames.min(self.prime) } else { frames };
self.staged_prime = self.prime > 0;
self.prime -= frames.min(self.prime);
let read = self.ring.pop_slice(&mut stretch.input[..frames * channels]);
debug_assert_eq!(read, frames * channels);
self.position = self.position.wrapping_add(frames);
- stretch.process(frames, buffered, self.shared.target.load(Ordering::Relaxed), enabled);
- self.accelerated += frames.saturating_sub(stretch.available()) as u64;
+ stretch.process(frames, inputs);
+ let removed = frames.saturating_sub(stretch.available());
+ self.accelerated += removed as u64;
// The writer still counts the input while it is being analysed.
// Publish only the resulting duration change, then the rendered
// frames. Moving PCM into staging never makes it disappear.
self.consumed = self.consumed.wrapping_add(frames).wrapping_sub(stretch.available());
}
self.shared.consumed.store(self.consumed, Ordering::Relaxed);
+ self.shared.played.store(self.played, Ordering::Relaxed);
done
}
@@ -720,6 +949,7 @@ impl Consumer {
/// Drop up to `frames` queued frames unread, returning how many of them were
/// pushed rather than the startup silence or a resume's comfort noise.
pub(crate) fn discard(&mut self, frames: usize) -> usize {
+ // Skipped audio leaves `played` behind, so the level drops with it.
let staged = self.stretch.as_mut().map_or(0, |stretch| stretch.discard(frames));
self.consumed = self.consumed.wrapping_add(staged);
let mut pushed = if self.staged_prime { 0 } else { staged };
@@ -745,6 +975,7 @@ impl Consumer {
left -= skipped;
}
self.shared.consumed.store(self.consumed, Ordering::Relaxed);
+ self.shared.played.store(self.played, Ordering::Relaxed);
pushed
}
}
@@ -755,6 +986,12 @@ mod tests {
const STEREO: usize = 2;
+ /// A read whose device callback is `out` alone.
+ fn pull(cons: &mut Consumer, out: &mut [f32], now: Instant) -> Read {
+ let frames = out.len() / STEREO;
+ cons.read_at(out, now, frames)
+ }
+
fn queue(input_rate: u32, output_rate: u32, capacity: Duration) -> (Producer, Consumer) {
let (prod, cons) = channel(
STEREO,
@@ -964,64 +1201,360 @@ mod tests {
}
}
- /// Clock-controlled packet/device drift, without padding to hide a dropout.
+ /// A stretching queue fed 20 ms of a 200 Hz tone every `interval`
+ /// microseconds against a 10 ms device period.
+ fn tone(rate: u32, latency: Duration) -> (Producer, Consumer, Vec) {
+ let (prod, mut cons) = channel(
+ 2,
+ rate,
+ rate,
+ Config {
+ latency,
+ capacity: Duration::from_secs(3),
+ pad: false,
+ trim: false,
+ comfort: true,
+ },
+ );
+ cons.stretch(rate);
+ cons.start();
+ let mut pcm = vec![0.0; rate as usize / 50 * 2];
+ for (i, frame) in pcm.as_chunks_mut::<2>().0.iter_mut().enumerate() {
+ frame.fill((i as f32 * std::f32::consts::TAU * 200.0 / rate as f32).sin() * 0.25);
+ }
+ (prod, cons, pcm)
+ }
+
+ /// Packets on a clock that drifts from the device's, each due one target
+ /// after it arrives. The queue plays onto that clock, in either direction
+ /// of drift, and onto it again after the target rises (slowing down rather
+ /// than parking) or falls (speeding up rather than skipping). Without a
+ /// clock, the target alone is where it settles.
#[test]
- fn stretch_tracks_both_clock_directions_and_target_changes() {
- for rate in [8_000, 44_100, 48_000, 96_000] {
- for interval in [19_960u64, 20_040] {
- let (mut prod, mut cons) = channel(
- 2,
- rate,
- rate,
- Config {
- latency: Duration::from_millis(80),
- capacity: Duration::from_secs(3),
- pad: false,
- trim: false,
- comfort: true,
- },
- );
- cons.stretch(rate);
- cons.start();
- let mut pcm = vec![0.0; rate as usize / 50 * 2];
- for (i, frame) in pcm.as_chunks_mut::<2>().0.iter_mut().enumerate() {
- frame.fill((i as f32 * std::f32::consts::TAU * 200.0 / rate as f32).sin() * 0.25);
- }
- let mut out = vec![0.0; rate as usize / 100 * 2];
- let mut arrival = 0;
- for tick in 0..9000u64 {
- if tick == 3000 {
- assert!(prod.set_latency(Duration::from_millis(120)));
- }
- if tick == 6000 {
- assert!(prod.set_latency(Duration::from_millis(60)));
- }
- while arrival <= tick * 10_000 {
- assert_eq!(prod.push(&pcm), Push::Ok);
- arrival += interval;
- }
- let (status, allocations) = crate::allocation::measure(|| cons.read(&mut out));
- assert_eq!(allocations.operations, 0, "callback allocated");
- assert_eq!(status, Read::Ok, "{rate} Hz, {interval} us, tick {tick}");
- if tick % 3000 > 500 {
- let target = if tick < 3000 {
- 80.0
- } else if tick < 6000 {
- 120.0
- } else {
- 60.0
- };
- let buffered = prod.buffered().as_secs_f64() * 1000.0;
- assert!(
- (target - 40.0..=target + 40.0).contains(&buffered),
- "{rate} Hz tick {tick}: {buffered} ms, target {target}"
- );
+ fn stretch_plays_onto_the_clock_through_drift_and_target_changes() {
+ for clocked in [true, false] {
+ for rate in [8_000, 44_100, 48_000, 96_000] {
+ for interval in [19_960u64, 20_040] {
+ let (mut prod, mut cons, pcm) = tone(rate, Duration::from_millis(80));
+ let base = prod.base;
+ let mut out = vec![0.0; rate as usize / 100 * 2];
+ let mut arrival = 0;
+ let mut target = Duration::from_millis(80);
+ for tick in 0..9000u64 {
+ if tick == 3000 || tick == 6000 {
+ target = Duration::from_millis(if tick == 3000 { 120 } else { 60 });
+ assert!(prod.set_latency(target));
+ }
+ while arrival <= tick * 10_000 {
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ if clocked {
+ prod.due(
+ base + Duration::from_micros(arrival) + target,
+ Duration::from_secs(2),
+ Duration::ZERO,
+ );
+ }
+ arrival += interval;
+ }
+ let now = base + Duration::from_millis(tick * 10);
+ let (status, allocations) = crate::allocation::measure(|| pull(&mut cons, &mut out, now));
+ assert_eq!(allocations.operations, 0, "callback allocated");
+ assert_eq!(status, Read::Ok, "{rate} Hz, {interval} us, tick {tick}");
+ if tick % 3000 > 500 {
+ let target = target.as_secs_f64() * 1000.0;
+ let buffered = prod.buffered().as_secs_f64() * 1000.0;
+ assert!(
+ (target - 40.0..=target + 40.0).contains(&buffered),
+ "{rate} Hz tick {tick}: {buffered} ms, target {target}"
+ );
+ if clocked {
+ let lag = cons.inputs(0).lag as f64 * 1000.0 / rate as f64;
+ assert!(
+ lag.abs() <= 20.0,
+ "{rate} Hz {interval} us tick {tick}: {lag} ms behind"
+ );
+ }
+ }
}
+ assert_eq!(cons.skips(), 0);
}
}
}
}
+ /// A 300 ms stall runs the queue dry. It resumes on one window and a device
+ /// period rather than its target, then slows down to rebuild the target,
+ /// falling behind the clock without running dry again. Media that keeps
+ /// arriving that late leaves it there; media that arrives in a burst once
+ /// the stall clears lets it speed back onto the clock.
+ #[test]
+ fn a_stall_resumes_early_and_catches_up_once_the_buffer_returns() {
+ for burst in [false, true] {
+ let rate = 48_000;
+ let (mut prod, mut cons, pcm) = tone(rate, Duration::from_millis(80));
+ let base = prod.base;
+ let target = Duration::from_millis(80);
+ let mut out = vec![0.0; 480 * 2];
+ let mut underruns = Vec::new();
+ let mut lag = 0.0;
+ let mut sent = 0;
+ for tick in 0..1500u64 {
+ let now = tick * 10;
+ // Packets sent every 20 ms, each due one target after it is sent.
+ // From 1000 ms they arrive 300 ms late, or all at 1300 ms and
+ // on time after.
+ let arrival = |sent: u64| match sent {
+ ..1000 => sent,
+ _ if burst => sent.max(1300),
+ _ => sent + 300,
+ };
+ while arrival(sent) <= now {
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ prod.due(
+ base + Duration::from_millis(sent) + target,
+ Duration::from_secs(2),
+ Duration::ZERO,
+ );
+ sent += 20;
+ }
+ let read = pull(&mut cons, &mut out, base + Duration::from_millis(now));
+ if matches!(read, Read::Underflow { .. }) {
+ underruns.push(now);
+ }
+ lag = cons.inputs(0).lag as f64 * 1000.0 / rate as f64;
+ }
+ // Running dry once the buffer drains, until a cushion arrives: two
+ // packets rather than the four of the target.
+ assert!(
+ underruns.iter().all(|&at| (1040..1330).contains(&at)),
+ "{burst}: {underruns:?}"
+ );
+ if burst {
+ assert!(lag.abs() <= 20.0, "back on the clock, {lag} ms behind");
+ } else {
+ assert!(
+ (250.0..1000.0).contains(&lag),
+ "settled behind the clock, {lag} ms behind"
+ );
+ }
+ // Rebuilt into NetEq's band: from three quarters of the target to it.
+ let level = cons.inputs(0).level * 1000.0 / rate as f64;
+ assert!((60.0..=100.0).contains(&level), "{burst}: rebuilt to {level} ms");
+ assert_eq!(cons.skips(), 0);
+ }
+ }
+
+ /// Audio trailing the clock by more than its bound and a window skips back
+ /// to the bound; less than that is the stretch's to catch up.
+ #[test]
+ fn only_audio_past_the_bound_skips() {
+ let rate = 48_000;
+ let (mut prod, mut cons, pcm) = tone(rate, Duration::from_millis(10));
+ let base = prod.base;
+ let mut out = vec![0.0; 480 * 2];
+ // Two seconds queued behind the 10 ms prime, played on time through the
+ // prime and the first 10 ms of audio, which joins the clock.
+ for _ in 0..100 {
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ }
+ prod.due(
+ base + Duration::from_millis(2010),
+ Duration::from_millis(1970),
+ Duration::ZERO,
+ );
+ assert_eq!(pull(&mut cons, &mut out, base), Read::Ok);
+ assert_eq!(pull(&mut cons, &mut out, base + Duration::from_millis(10)), Read::Ok);
+ // Now all due at 10 ms: the reader trails by 2000 ms.
+ let due = base + Duration::from_millis(10);
+ prod.due(due, Duration::from_millis(1970), Duration::ZERO);
+ assert_eq!(pull(&mut cons, &mut out, base + Duration::from_millis(20)), Read::Ok);
+ assert_eq!(cons.skips(), 0, "a window past the bound is the stretch's");
+ prod.due(due, Duration::from_millis(1960), Duration::ZERO);
+ let read = pull(&mut cons, &mut out, base + Duration::from_millis(30));
+ assert_eq!(read, Read::Trimmed { frames: 1920 });
+ assert_eq!(cons.trimmed(), 1920, "back to the bound");
+ // A fixed delay has no bound: whatever trails the clock goes, here
+ // what is due before the second the end is due at.
+ prod.due(base + Duration::from_secs(1), Duration::ZERO, Duration::ZERO);
+ let read = pull(&mut cons, &mut out, base + Duration::from_millis(40));
+ assert!(matches!(read, Read::Trimmed { .. }), "{read:?}");
+ assert!(cons.inputs(0).lag <= 0);
+ assert_eq!(cons.skips(), 2);
+ let left = cons.available() as f64 * 1000.0 / rate as f64;
+ assert!((940.0..960.0).contains(&left), "{left} ms left");
+ }
+
+ /// A clock whose reference moved earlier skips as much, as far as the
+ /// reader trails it, where the same lag from a fallen target is caught up
+ /// by speeding up.
+ #[test]
+ fn a_reference_jump_skips_as_much() {
+ for jumped in [false, true] {
+ let rate = 48_000;
+ let (mut prod, mut cons, pcm) = tone(rate, Duration::from_millis(200));
+ let base = prod.base;
+ let mut out = vec![0.0; 480 * 2];
+ for _ in 0..100 {
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ }
+ // On time, through the 200 ms prime, which joins the clock.
+ let late = Duration::from_secs(2);
+ prod.due(base + Duration::from_millis(2200), late, Duration::ZERO);
+ let mut now = base;
+ for _ in 0..25 {
+ assert_eq!(pull(&mut cons, &mut out, now), Read::Ok);
+ now += Duration::from_millis(10);
+ }
+ // The clock moves 100 ms ahead: its reference, or a fall in the target.
+ let at = base + Duration::from_millis(2100);
+ if jumped {
+ prod.due(at, late, Duration::from_millis(150));
+ } else {
+ assert!(prod.set_latency(Duration::from_millis(100)));
+ prod.due(at, late, Duration::ZERO);
+ }
+ let read = pull(&mut cons, &mut out, now);
+ let lag = cons.inputs(0).lag as f64 * 1000.0 / rate as f64;
+ if jumped {
+ // Never past the clock, however far the reference moved.
+ assert!(matches!(read, Read::Trimmed { .. }), "{read:?}");
+ assert!((-20.0..=5.0).contains(&lag), "{lag} ms behind");
+ } else {
+ assert_eq!(read, Read::Ok);
+ assert!(lag >= 90.0, "{lag} ms behind");
+ for _ in 0..50 {
+ now += Duration::from_millis(10);
+ assert_eq!(pull(&mut cons, &mut out, now), Read::Ok);
+ }
+ assert_eq!(cons.skips(), 0);
+ assert!(cons.accelerated() > 0);
+ }
+ }
+ }
+
+ /// A jump counts only as far as the queue holds audio past its target,
+ /// which the early media that moved the reference leaves it, so one on a
+ /// queue running low is the stretch's to catch up, inside the bound.
+ #[test]
+ fn a_jump_skips_only_audio_held_past_the_target() {
+ for deep in [false, true] {
+ let rate = 48_000;
+ let (mut prod, mut cons, pcm) = tone(rate, Duration::from_millis(80));
+ let base = prod.base;
+ let late = Duration::from_secs(2);
+ let mut out = vec![0.0; 480 * 2];
+ let packets = if deep { 50 } else { 5 };
+ for _ in 0..packets {
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ }
+ // On time through the 80 ms prime, which joins the clock.
+ let end = Duration::from_millis(80 + 20 * packets);
+ prod.due(base + end, late, Duration::ZERO);
+ let mut now = base;
+ for _ in 0..9 {
+ assert!(!matches!(pull(&mut cons, &mut out, now), Read::Trimmed { .. }));
+ now += Duration::from_millis(10);
+ }
+ // The reference jumps 50 ms.
+ let shift = Duration::from_millis(50);
+ prod.due(base + end - shift, late, shift);
+ let read = pull(&mut cons, &mut out, now);
+ let lag = cons.inputs(0).lag as f64 * 1000.0 / rate as f64;
+ if deep {
+ assert!(matches!(read, Read::Trimmed { .. }), "{read:?}");
+ assert!((-20.0..=5.0).contains(&lag), "{lag} ms behind");
+ } else {
+ assert_eq!(cons.skips(), 0, "{read:?}");
+ assert!(lag >= 30.0, "{lag} ms behind");
+ }
+ }
+ }
+
+ /// Pending jumps never add up past the audio held over the target, so a
+ /// skip never drains the queue below it.
+ #[test]
+ fn a_pending_jump_never_skips_below_the_target() {
+ let rate = 48_000;
+ let (mut prod, mut cons, pcm) = tone(rate, Duration::from_millis(20));
+ let base = prod.base;
+ let late = Duration::from_secs(2);
+ let mut out = vec![0.0; 480 * 2];
+ for _ in 0..5 {
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ }
+ // On time through the 20 ms prime, which joins the clock.
+ let end = Duration::from_millis(120);
+ prod.due(base + end, late, Duration::ZERO);
+ let mut now = base;
+ for _ in 0..3 {
+ assert!(!matches!(pull(&mut cons, &mut out, now), Read::Trimmed { .. }));
+ now += Duration::from_millis(10);
+ }
+ // 25 ms pending, short of a window.
+ let mut shift = Duration::from_millis(25);
+ prod.due(base + end - shift, late, shift);
+ assert!(!matches!(pull(&mut cons, &mut out, now), Read::Trimmed { .. }));
+ now += Duration::from_millis(10);
+ // Drained to about 35 ms held, 15 ms past the target, then 15 ms more.
+ while cons.available() > rate as usize * 45 / 1000 {
+ assert!(!matches!(pull(&mut cons, &mut out, now), Read::Trimmed { .. }));
+ now += Duration::from_millis(10);
+ }
+ shift += Duration::from_millis(15);
+ prod.due(base + end - shift, late, Duration::from_millis(15));
+ let read = pull(&mut cons, &mut out, now);
+ assert!(
+ !matches!(read, Read::Trimmed { .. } | Read::Underflow { .. }),
+ "{read:?}"
+ );
+ assert_eq!(cons.skips(), 0);
+ }
+
+ /// A refill after an underrun waits for the whole device callback, not
+ /// the slice of it one read fills.
+ #[test]
+ fn a_refill_waits_for_the_whole_callback() {
+ let rate = 48_000;
+ let (mut prod, mut cons, pcm) = tone(rate, Duration::from_millis(100));
+ let mut out = vec![0.0; 1024 * 2];
+ let callback = 4096;
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ let now = host();
+ while !matches!(cons.read_at(&mut out, now, callback), Read::Underflow { .. }) {}
+ // 60 ms covers one window and a slice, but not the 85 ms callback.
+ for _ in 0..3 {
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ }
+ assert_eq!(cons.read_at(&mut out, now, callback), Read::Underflow { frames: 0 });
+ for _ in 0..2 {
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ }
+ assert_eq!(cons.read_at(&mut out, now, callback), Read::Ok);
+ }
+
+ /// Nothing trailing the clock was heard before the first audio plays on
+ /// it, so a join lands on the clock past a backlog the bound would keep.
+ #[test]
+ fn a_join_lands_on_the_clock() {
+ let rate = 48_000;
+ let (mut prod, mut cons, pcm) = tone(rate, Duration::from_millis(10));
+ let base = prod.base;
+ let mut out = vec![0.0; 480 * 2];
+ for _ in 0..100 {
+ assert_eq!(prod.push(&pcm), Push::Ok);
+ }
+ // All due at 10 ms, inside a 1970 ms bound.
+ prod.due(
+ base + Duration::from_millis(10),
+ Duration::from_millis(1970),
+ Duration::ZERO,
+ );
+ let read = pull(&mut cons, &mut out, base);
+ assert!(matches!(read, Read::Trimmed { .. }), "{read:?}");
+ assert_eq!(cons.skips(), 1);
+ assert!(cons.inputs(0).lag <= 0);
+ }
+
#[test]
fn staged_audio_is_buffered_and_finish_drains_it() {
let (mut prod, mut cons) = channel(
@@ -1104,15 +1637,18 @@ mod tests {
assert_eq!(prod.push(&[]), Push::Ok);
let mut out = vec![1.0; 4800 * 2];
assert_eq!(cons.read(&mut out), Read::Underflow { frames: 2400 });
- for _ in 0..2 {
- assert_eq!(prod.push(&[0.25; 960 * 2]), Push::Ok, "no padding inserted in the ring");
- assert_eq!(cons.read(&mut out[..128 * 2]), Read::Underflow { frames: 0 });
- assert!(out[..128 * 2].iter().all(|&sample| sample == 0.0));
- }
- assert_eq!(prod.push(&[0.25; 960 * 2]), Push::Ok);
+ // One 30 ms window and the 128-frame read, not the 50 ms target: 1568 frames.
+ assert_eq!(
+ prod.push(&[0.25; 1567 * 2]),
+ Push::Ok,
+ "no padding inserted in the ring"
+ );
+ assert_eq!(cons.read(&mut out[..128 * 2]), Read::Underflow { frames: 0 });
+ assert!(out[..128 * 2].iter().all(|&sample| sample == 0.0));
+ assert_eq!(prod.push(&[0.25; 2]), Push::Ok);
assert_eq!(cons.read(&mut out[..128 * 2]), Read::Ok);
assert!(out[..128 * 2].iter().all(|&sample| sample == 0.25));
- // A final short input must not wait forever to reach the refill target.
+ // A final short input must not wait forever to reach the cushion.
assert!(prod.flush());
while cons.available() > 0 {
cons.read(&mut out);
diff --git a/rs/moq-audio/src/stretch.rs b/rs/moq-audio/src/stretch.rs
index e3496279b4a..730f722ba47 100644
--- a/rs/moq-audio/src/stretch.rs
+++ b/rs/moq-audio/src/stretch.rs
@@ -3,17 +3,110 @@
//!
//! Copyright (c) 2012 The WebRTC project authors. All Rights Reserved.
//! Ported from modules/audio_coding/neteq/{time_stretch,accelerate,
-//! preemptive_expand,dsp_helper,audio_vector,decision_logic}.
+//! preemptive_expand,dsp_helper,audio_vector,decision_logic,buffer_level_filter}.
//! See WEBRTC-LICENSE and WEBRTC-PATENTS in this crate.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
-enum Operation {
+pub(crate) enum Operation {
Hold,
Accelerate,
Fast,
Expand,
}
+/// What a window decides on, in frames at the device rate.
+#[derive(Clone, Copy, Debug, PartialEq)]
+pub(crate) struct Inputs {
+ /// The playout delay the shared clock plays at.
+ pub target: usize,
+ /// How far the playhead trailed the newest received audio when it
+ /// arrived, less what the stretch removed since: the buffer level with
+ /// the publisher's cadence taken out. Filtered before deciding.
+ pub level: f64,
+ /// Received audio ahead of the playhead.
+ pub raw: usize,
+ /// How far the playhead trails the shared clock: negative when ahead.
+ pub lag: i64,
+ /// How far behind the clock audio may fall.
+ pub bound: i64,
+}
+
+/// The shared decision rule on a filtered level, with the most a seam may
+/// change the output by in frames. Accelerating only ever catches a playhead
+/// behind the clock up to it; expanding slows one whose buffer runs low, as
+/// far as `bound` allows, or one ahead of the clock back onto it. A stall
+/// delivers no arrival to lower `level`, so `raw` stops a playhead behind the
+/// clock from speeding into it.
+pub(crate) fn decide(rate: u32, inputs: Inputs, cooldown: usize) -> (Operation, i64) {
+ let Inputs {
+ target,
+ level,
+ raw,
+ lag,
+ bound,
+ } = inputs;
+ let rate = rate as usize;
+ // NetEq's DecisionLogic limits, with its 85 ms deceleration offset and
+ // 20 ms granularity.
+ let low = (3 * target / 4).max(target.saturating_sub(rate * 85 / 1000));
+ let high = target.max(low + rate * 20 / 1000);
+ let buffer = level.min(raw as f64);
+ if lag > 0 && buffer >= (4 * high) as f64 {
+ (Operation::Fast, lag)
+ } else if cooldown > 0 {
+ (Operation::Hold, 0)
+ } else if lag > 0 && buffer >= high as f64 {
+ (Operation::Accelerate, lag)
+ } else if level < low as f64 && lag < bound {
+ (Operation::Expand, bound - lag)
+ } else if lag < 0 {
+ (Operation::Expand, -lag)
+ } else {
+ (Operation::Hold, 0)
+ }
+}
+
+/// NetEq's buffer-level filter, updated once per window: three of its 10 ms
+/// steps, its factor chosen by the target, less what the stretch changed since
+/// the last update. Unset until the first update after a start or a
+/// [`reset`](Self::reset), which takes the level as it is.
+#[derive(Default)]
+pub(crate) struct Level {
+ filtered: Option,
+}
+
+impl Level {
+ /// Fold in `level`, after windows whose stretch removed `stretched` frames
+ /// (negative: added).
+ pub fn update(&mut self, rate: u32, target: usize, level: f64, stretched: i64) -> f64 {
+ let Some(filtered) = self.filtered else {
+ self.filtered = Some(level);
+ return level;
+ };
+ // NetEq's BufferLevelFilter factors, in 256ths per 10 ms step, by the
+ // target in milliseconds.
+ let rate = rate as usize;
+ let factor: u64 = if target * 1000 <= 20 * rate {
+ 251
+ } else if target * 1000 <= 60 * rate {
+ 252
+ } else if target * 1000 <= 140 * rate {
+ 253
+ } else {
+ 254
+ };
+ let weight = (factor * factor * factor) as f64 / 16_777_216.0;
+ let filtered = (weight * filtered + (1.0 - weight) * level - stretched as f64).max(0.0);
+ self.filtered = Some(filtered);
+ filtered
+ }
+
+ /// Forget the history, so the next update starts from its level.
+ pub fn reset(&mut self) {
+ self.filtered = None;
+ }
+}
+
/// One bounded block and its seam, allocated before entering the callback.
pub(crate) struct Stretch {
pub input: Vec,
@@ -24,6 +117,9 @@ pub(crate) struct Stretch {
position: usize,
length: usize,
cooldown: usize,
+ /// What the stretch removed since the last filter update.
+ stretched: i64,
+ pub level: Level,
}
impl Stretch {
@@ -38,6 +134,8 @@ impl Stretch {
position: 0,
length: 0,
cooldown: rate as usize * 60 / 1000,
+ stretched: 0,
+ level: Level::default(),
}
}
@@ -60,24 +158,34 @@ impl Stretch {
frames
}
- /// All input is new, received PCM. Short blocks and finished sinks cannot
- /// stretch, so a finite input never becomes concealment of an absent frame.
- pub fn process(&mut self, frames: usize, buffered: usize, target: usize, enabled: bool) {
- let operation = if enabled && frames == self.window {
- self.decide(buffered, target)
- } else {
- Operation::Hold
+ /// All input is new, received PCM. Short blocks and windows without
+ /// `inputs` cannot stretch, so a finite input never becomes concealment of
+ /// an absent frame.
+ pub fn process(&mut self, frames: usize, inputs: Option) -> Operation {
+ let Some(inputs) = inputs.filter(|_| frames == self.window) else {
+ return self.stage(frames, Operation::Hold, 0);
};
+ let level = self
+ .level
+ .update(self.rate, inputs.target, inputs.level, self.stretched);
+ self.stretched = 0;
+ let (operation, allowance) = decide(self.rate, Inputs { level, ..inputs }, self.cooldown);
+ self.stage(frames, operation, allowance)
+ }
+
+ /// Stage the first `frames` of the input as `operation` decides, if a
+ /// pitch period within `allowance` admits it.
+ fn stage(&mut self, frames: usize, operation: Operation, allowance: i64) -> Operation {
let lag = if operation == Operation::Hold {
0
} else {
- pitch(&self.input, self.channels, self.rate, operation == Operation::Fast)
+ pitch(&self.input, self.channels, self.rate, operation, allowance)
};
self.position = 0;
self.length = frames;
if lag == 0 {
self.output[..frames * self.channels].copy_from_slice(&self.input[..frames * self.channels]);
- return;
+ return operation;
}
let middle = self.rate as usize * 15 / 1000;
@@ -92,6 +200,7 @@ impl Stretch {
}
Operation::Hold => unreachable!(),
};
+ self.stretched += frames as i64 - self.length as i64;
let channels = self.channels;
self.output[..seam * channels].copy_from_slice(&self.input[..seam * channels]);
for frame in 0..lag {
@@ -107,26 +216,16 @@ impl Stretch {
self.output[(seam + lag) * channels..self.length * channels]
.copy_from_slice(&self.input[tail * channels..frames * channels]);
self.cooldown = self.rate as usize * 50 / 1000;
- }
-
- fn decide(&self, buffered: usize, target: usize) -> Operation {
- let high = target + self.rate as usize * 20 / 1000;
- if buffered >= high * 4 {
- Operation::Fast
- } else if self.cooldown > 0 {
- Operation::Hold
- } else if buffered >= high {
- Operation::Accelerate
- } else if buffered < target {
- Operation::Expand
- } else {
- Operation::Hold
- }
+ operation
}
}
-/// Search the first channel at 4 kHz, then validate the full-rate seam.
-fn pitch(input: &[f32], channels: usize, rate: u32, fast: bool) -> usize {
+/// Search the first channel at 4 kHz, then validate the full-rate seam,
+/// returning how many frames it changes the output by, or zero. Fast mode
+/// admits a weaker match and takes every whole period that fits before the
+/// middle; the others take one. None takes more than `allowance`.
+fn pitch(input: &[f32], channels: usize, rate: u32, operation: Operation, allowance: i64) -> usize {
+ let fast = operation == Operation::Fast;
// NetEq's 48 kHz Q12 FIR. Sampling its taps at fractional positions also
// admits native 44.1/96 kHz devices without altering the search domain.
const FILTER: [f64; 7] = [1019.0, 390.0, 427.0, 440.0, 427.0, 390.0, 1019.0];
@@ -191,7 +290,13 @@ fn pitch(input: &[f32], channels: usize, rate: u32, fast: bool) -> usize {
if !passive && cross <= threshold * (energy1 * energy2).sqrt() {
return 0;
}
- if fast { middle / lag * lag } else { lag }
+ let allowance = usize::try_from(allowance).unwrap_or(0);
+ let periods = if fast {
+ (middle / lag).min(allowance / lag)
+ } else {
+ usize::from(allowance >= lag)
+ };
+ periods * lag
}
#[cfg(test)]
@@ -200,34 +305,68 @@ mod tests {
use serde::Deserialize;
#[derive(Deserialize)]
- struct Vector {
+ struct Pcm {
name: String,
rate: u32,
channels: usize,
input: Vec,
- buffered: usize,
- target: usize,
- cooldown: usize,
decision: String,
+ allowance: i64,
lag: usize,
output: Vec,
}
+ #[derive(Deserialize)]
+ struct Vectors {
+ decisions: Vec,
+ filter: Vec,
+ }
+
+ #[derive(Deserialize)]
+ struct Decision {
+ name: String,
+ rate: u32,
+ target: usize,
+ level: f64,
+ raw: usize,
+ lag: i64,
+ bound: i64,
+ cooldown: usize,
+ decision: String,
+ allowance: i64,
+ }
+
+ #[derive(Deserialize)]
+ struct Filter {
+ name: String,
+ rate: u32,
+ target: usize,
+ steps: Vec<(f64, i64)>,
+ filtered: Vec,
+ }
+
+ fn operation(name: &str) -> Operation {
+ match name {
+ "Hold" => Operation::Hold,
+ "Accelerate" => Operation::Accelerate,
+ "Fast" => Operation::Fast,
+ "Expand" => Operation::Expand,
+ _ => panic!("unknown decision {name}"),
+ }
+ }
+
+ fn vectors() -> Vectors {
+ serde_json::from_str(include_str!("../../../doc/concept/audio-jitter/stretch/decisions.json")).unwrap()
+ }
+
#[test]
fn shared_pcm_vectors() {
- let vectors: Vec =
+ let vectors: Vec =
serde_json::from_str(include_str!("../../../doc/concept/audio-jitter/stretch/corpus.json")).unwrap();
for vector in vectors {
let mut stretch = Stretch::new(vector.rate, vector.channels);
stretch.input.copy_from_slice(&vector.input);
- stretch.cooldown = vector.cooldown;
- assert_eq!(
- format!("{:?}", stretch.decide(vector.buffered, vector.target)),
- vector.decision,
- "{}",
- vector.name
- );
- stretch.process(stretch.window, vector.buffered, vector.target, true);
+ stretch.stage(stretch.window, operation(&vector.decision), vector.allowance);
assert_eq!(stretch.length.abs_diff(stretch.window), vector.lag, "{}", vector.name);
assert_eq!(stretch.length * vector.channels, vector.output.len(), "{}", vector.name);
for (index, (&actual, &expected)) in stretch.output[..vector.output.len()]
@@ -244,13 +383,88 @@ mod tests {
}
}
+ #[test]
+ fn shared_decision_vectors() {
+ for vector in vectors().decisions {
+ let inputs = Inputs {
+ target: vector.target,
+ level: vector.level,
+ raw: vector.raw,
+ lag: vector.lag,
+ bound: vector.bound,
+ };
+ assert_eq!(
+ decide(vector.rate, inputs, vector.cooldown),
+ (operation(&vector.decision), vector.allowance),
+ "{}",
+ vector.name
+ );
+ }
+ }
+
+ #[test]
+ fn shared_filter_vectors() {
+ for vector in vectors().filter {
+ let mut level = Level::default();
+ for (step, (&(value, stretched), &expected)) in vector.steps.iter().zip(&vector.filtered).enumerate() {
+ let actual = level.update(vector.rate, vector.target, value, stretched);
+ assert!(
+ (actual - expected).abs() <= 1e-9,
+ "{} step {step}: {actual} != {expected}",
+ vector.name
+ );
+ }
+ }
+ }
+
+ #[test]
+ fn the_filter_takes_out_what_the_stretch_changed() {
+ // 48 kHz at a 100 ms target: a constant signal accelerates by its
+ // shortest period, 120 frames, from a buffer at the high limit.
+ let inputs = Inputs {
+ target: 4800,
+ level: 4800.0,
+ raw: 4800,
+ lag: 120,
+ bound: 0,
+ };
+ let mut stretch = Stretch::new(48_000, 1);
+ stretch.input.fill(0.25);
+ stretch.cooldown = 0;
+ assert_eq!(stretch.process(stretch.window, Some(inputs)), Operation::Accelerate);
+ assert_eq!(stretch.length, stretch.window - 120);
+ assert_eq!(stretch.level.filtered, Some(4800.0), "the first update takes the level");
+ stretch.cooldown = 1;
+ stretch.process(stretch.window, Some(inputs));
+ assert_eq!(
+ stretch.level.filtered,
+ Some(4680.0),
+ "the next takes out the period removed"
+ );
+ stretch.level.reset();
+ stretch.process(stretch.window, Some(inputs));
+ assert_eq!(
+ stretch.level.filtered,
+ Some(4800.0),
+ "a reset starts again from the level"
+ );
+ }
+
#[test]
fn a_finite_input_cannot_expand_forever() {
for rate in [8_000, 44_100, 48_000, 96_000] {
let mut stretch = Stretch::new(rate, 2);
stretch.input.fill(0.25);
stretch.cooldown = 0;
- stretch.process(stretch.window, stretch.window, rate as usize, true);
+ // Ahead of the clock by a second: expand back onto it.
+ let inputs = Some(Inputs {
+ target: rate as usize / 10,
+ level: rate as f64 / 10.0,
+ raw: rate as usize / 10,
+ lag: -(rate as i64),
+ bound: 0,
+ });
+ assert_eq!(stretch.process(stretch.window, inputs), Operation::Expand);
assert!(stretch.length > stretch.window);
let mut output = vec![0.0; 2 * rate as usize];
let (_, allocations) = crate::allocation::measure(|| {
@@ -258,10 +472,29 @@ mod tests {
assert_eq!(stretch.read(&mut output), 0);
});
assert_eq!(allocations.operations, 0);
- stretch.process(stretch.window, stretch.window, rate as usize, false);
+ stretch.cooldown = 0;
+ stretch.process(stretch.window, None);
assert_eq!(stretch.length, stretch.window, "a finished sink drains unchanged");
- stretch.process(stretch.window / 2, stretch.window, rate as usize, true);
+ stretch.process(stretch.window / 2, inputs);
assert_eq!(stretch.length, stretch.window / 2, "a short block drains unchanged");
}
}
+
+ #[test]
+ fn a_seam_never_overshoots_the_allowance() {
+ // A constant signal's shortest period is 2.5 ms: 120 frames at 48 kHz.
+ for (operation, allowance, length) in [
+ (Operation::Accelerate, 119, 1440),
+ (Operation::Accelerate, 120, 1320),
+ (Operation::Accelerate, 239, 1320),
+ (Operation::Fast, 360, 1080),
+ (Operation::Expand, 119, 1440),
+ (Operation::Expand, 120, 1560),
+ ] {
+ let mut stretch = Stretch::new(48_000, 1);
+ stretch.input.fill(0.25);
+ stretch.stage(stretch.window, operation, allowance);
+ assert_eq!(stretch.length, length, "{operation:?} {allowance}");
+ }
+ }
}
diff --git a/rs/moq-cli/src/play/fake.rs b/rs/moq-cli/src/play/fake.rs
index 7b59498e3ab..6243f2f546a 100644
--- a/rs/moq-cli/src/play/fake.rs
+++ b/rs/moq-cli/src/play/fake.rs
@@ -132,6 +132,10 @@ impl Speaker for Recorder {
}
}
+/// What an underrun waits for before playing again: the real ring's 30 ms
+/// stretch window and a 10 ms device period.
+const CUSHION: Duration = Duration::from_millis(40);
+
/// A speaker draining in real time from the moment the sink opens.
pub(super) struct FakeSink {
state: Arc>,
@@ -158,6 +162,9 @@ impl Sink for FakeSink {
Ok(())
}
+ /// Playing onto the clock is the real ring's stretch, which this fake leaves out.
+ fn due(&mut self, _at: Instant, _late: Duration, _jumped: Duration) {}
+
async fn finish(mut self) {
// A finite short tail drains even when it cannot refill the target.
if self.prime.take().is_some() {
@@ -174,7 +181,8 @@ impl Sink for FakeSink {
let now = Instant::now();
// DSP and clock drift are tested in moq-audio's real ring. This fake
// models its emergency recovery: no synthetic padding, and no PCM
- // heard until fresh writes refill the target after a genuine gap.
+ // heard until fresh writes refill one stretch window and a device
+ // period after a genuine gap.
if let Some(prime) = self.prime.take() {
self.end = now.saturating_add(prime);
// The opening silence already holds the first utterance one latency out.
@@ -209,7 +217,7 @@ impl Sink for FakeSink {
if self.refill {
self.pending.push((sample, duration));
self.queued += duration;
- if self.queued >= self.latency {
+ if self.queued >= CUSHION {
self.resume();
}
} else {
@@ -281,25 +289,24 @@ mod tests {
use super::*;
#[tokio::test]
- async fn underrun_waits_for_received_audio_at_the_new_target() {
+ async fn underrun_waits_for_a_cushion_of_received_audio() {
tokio::time::pause();
let recorder = Recorder::default();
let mut sink = recorder.sink(Input::default()).unwrap();
let pcm = [0u8; 960 * 2 * 4];
assert_eq!(sink.write(&pcm).unwrap().accepted_sample_frames, 960);
tokio::time::advance(Duration::from_millis(100)).await;
- sink.set_latency(Duration::from_millis(80)).unwrap();
+ // The cushion, not the target, however deep the target is.
+ sink.set_latency(Duration::from_millis(500)).unwrap();
let heard = recorder.played().len();
- for frames in 1..4 {
- assert_eq!(sink.write(&pcm).unwrap().accepted_sample_frames, 960);
- assert_eq!(sink.buffered(), Duration::from_millis(frames * 20));
- assert_eq!(recorder.played().len(), heard);
- }
+ assert_eq!(sink.write(&pcm).unwrap().accepted_sample_frames, 960);
+ assert_eq!(sink.buffered(), Duration::from_millis(20));
+ assert_eq!(recorder.played().len(), heard);
assert_eq!(sink.write(&pcm).unwrap().accepted_sample_frames, 960);
let played = recorder.played();
- assert_eq!(played.len(), heard + 4);
+ assert_eq!(played.len(), heard + 2);
assert_eq!(played[heard].from, Instant::now());
- assert_eq!(sink.buffered(), Duration::from_millis(80));
+ assert_eq!(sink.buffered(), Duration::from_millis(40));
}
#[tokio::test]
diff --git a/rs/moq-cli/src/play/media.rs b/rs/moq-cli/src/play/media.rs
index 36f5625b1ca..7e50b7a5b61 100644
--- a/rs/moq-cli/src/play/media.rs
+++ b/rs/moq-cli/src/play/media.rs
@@ -14,7 +14,7 @@ use kio::clock::Instant;
use super::args::Args;
use super::output::{Output, Sink, Speaker};
use super::playback::{Kind, Playback, Track, joined};
-use super::timeline::{AudioTimeline, Playhead, fit};
+use super::timeline::{AudioTimeline, Playhead};
use super::video::{Refresh, Video};
use super::window::Event;
use moq_mux::sync::{Producer as Presentation, Transition};
@@ -32,11 +32,6 @@ enum Crossing {
/// network jitter, and one with no depth at all can never be read from.
const AUDIO_BUFFER_MIN: Duration = Duration::from_millis(50);
-/// NetEq's delay adjustment granularity. The callback handles ordinary
-/// differences by stretching; the write path re-anchors only at four times
-/// the target plus this granularity.
-const AUDIO_SLACK: Duration = Duration::from_millis(20);
-
/// Maximum sample frames per silence write, independent of the catalog rate.
const AUDIO_SILENCE: usize = 960;
@@ -598,9 +593,9 @@ async fn play_audio(
// later. That estimate times each packet when `read` hands it over, so this
// receiver never waits on the speaker between reads. Live playback lets
// the callback stretch toward the target; a declared end plays comfort noise
- // until the old media drains and its replacement is due. Each write steers
- // the sink toward when the shared clock says its media is due, which keeps
- // audio with video in the steady state without letting a stall hold video.
+ // until the old media drains and its replacement is due. Each write anchors
+ // the sink to when the shared clock says its media is due, so the callback
+ // plays onto that clock in the steady state without letting a stall hold video.
let role = presentation.track(moq_mux::container::Kind::Audio);
let received = Mutex::new(VecDeque::new());
let arrived = tokio::sync::Notify::new();
@@ -622,7 +617,6 @@ async fn play_audio(
// One sample across every channel, the unit a write has to stay aligned to.
let stride = channels as usize * size_of::();
let samples = |duration: Duration| (duration.as_secs_f64() * sample_rate as f64).round() as u64;
- let slack = samples(AUDIO_SLACK);
let mut silence = Vec::new();
// The longest hole worth playing through, in samples. A hole this player would
@@ -661,15 +655,39 @@ async fn play_audio(
}
};
steer(None);
- // The depth that makes media ending at `end` sound when the shared clock says it is due.
- let hold = |end: Duration| {
- moq_net::Timestamp::try_from(end)
- .ok()
- .and_then(|end| presentation.due(end))
- .map(|due| {
- due.saturating_duration_since(Instant::now())
- .clamp(AUDIO_BUFFER_MIN, cap)
- })
+ // What the sink holds against jitter: the shared delay, floored at what the
+ // device needs and capped at what the sink was sized for.
+ let target = || schedule.delay().clamp(AUDIO_BUFFER_MIN, cap);
+ // Anchor the sink to the shared clock: the media written so far ends at
+ // `end`, and sounds when the clock says it is due. The callback stretches
+ // onto that clock, and skips only what trails it past the age budget the
+ // subscription still delivers, which a fixed delay leaves none of.
+ // The last anchor, as when its end was due, the end, and the delay. The
+ // clock's reference is when media zero arrived, `at - end - delay`: one that
+ // moved earlier jumped the clock ahead, which the sink skips rather than
+ // speeds through, as video skips frames it passed over.
+ let mut anchored = None::<(Instant, Duration, Duration)>;
+ let anchor = |sink: &mut ::Sink,
+ anchored: &mut Option<(Instant, Duration, Duration)>,
+ end: Duration| {
+ let Ok(timestamp) = moq_net::Timestamp::try_from(end) else {
+ return;
+ };
+ if let Some(at) = presentation.due(timestamp) {
+ // The end as the clock placed it, so rounding never reads as a jump.
+ let end = Duration::from(timestamp);
+ let delay = schedule.delay();
+ let nanos = |duration: Duration| duration.as_nanos() as i128;
+ let jumped = anchored.replace((at, end, delay)).map_or(0, |(was, from, before)| {
+ let earlier = match was.checked_duration_since(at) {
+ Some(ahead) => nanos(ahead),
+ None => -nanos(at.duration_since(was)),
+ };
+ earlier + nanos(end) - nanos(from) + nanos(delay) - nanos(before)
+ });
+ let jumped = Duration::from_nanos(jumped.clamp(0, u64::MAX as i128) as u64);
+ sink.due(at, max_delay.saturating_sub(delay), jumped);
+ }
};
// Tracks whether the last read failed, so a stream the decoder can't read at
@@ -824,8 +842,9 @@ async fn play_audio(
// No write may come to carry a delay change, so the speaker follows it now.
_ = role.delay_changed(seen) => {
steer(wrote);
- if let Some(held) = wrote.and_then(|(_, end)| hold(end)) {
- sink.set_latency(held)?;
+ sink.set_latency(target())?;
+ if let Some((_, end)) = wrote {
+ anchor(&mut sink, &mut anchored, end);
}
}
}
@@ -968,8 +987,7 @@ async fn play_audio(
// the delay was known, and a held frame behind only what was already
// written, so the rest of the lead goes in as silence ahead of the frame,
// which already covers any hole before it. A frame starting an utterance is
- // instead placed one target out by the sink itself, and `hold` below sizes
- // that target to the due time.
+ // instead placed one target out by the sink itself.
let short = presentation
.due(mapped)
.filter(|_| (fresh || held_back) && !utterance)
@@ -978,24 +996,15 @@ async fn play_audio(
.saturating_sub(sink.buffered())
})
.map_or(0, samples);
- // The depth that makes this frame's end sound when the shared clock says
- // it is due, so the callback stretches the speaker onto the clock: behind
- // it after a stall, it plays faster until it catches up. Floored at what
- // the device needs, and capped at the age budget the sink was sized for;
- // the consumer holds anything due later encoded.
- let buffered = samples(sink.buffered());
- let held = hold(timing.end).unwrap_or(input.latency.clamp(AUDIO_BUFFER_MIN, cap));
- let target = samples(held);
- sink.set_latency(held)?;
- // Leave normal convergence to the callback. Only an excess at NetEq's
- // fast threshold (four times target + granularity) is an emergency
- // re-anchor; a 20 ms trim would compete with every pitch correction.
- let fit = fit(buffered, target, slack, timing.silence.max(short), length as u64);
+ // Convergence is the callback's: it slows down when its buffer runs low and
+ // catches up with the clock once it holds more than its target, and only
+ // skips what trails the clock past what the subscription still delivers.
+ sink.set_latency(target())?;
// The packet pays for the channel count; the catalog cannot allocate
// a duration's worth of silence before any samples arrive.
silence.resize(length.min(AUDIO_SILENCE) * stride, 0);
- let mut quiet = fit.silence;
+ let mut quiet = timing.silence.max(short);
while quiet > 0 && !silence.is_empty() {
let frames = quiet.min((silence.len() / stride) as u64) as usize;
let written = sink.write(&silence[..frames * stride])?;
@@ -1006,9 +1015,9 @@ async fn play_audio(
}
quiet -= frames as u64;
}
- let skip = fit.skip as usize * stride;
- let written = sink.write(&frame.data[skip.min(frame.data.len())..])?;
- let trimmed = fit.skip + written.dropped_sample_frames as u64;
+ let written = sink.write(&frame.data)?;
+ anchor(&mut sink, &mut anchored, timing.end);
+ let trimmed = written.dropped_sample_frames as u64;
if trimmed > 0 {
output.send(Event::Trimmed {
audio: Duration::from_secs_f64(trimmed as f64 / sample_rate as f64),
@@ -1020,8 +1029,7 @@ async fn play_audio(
// A dropped write leaves the utterance to start on the next frame the sink accepts.
if utterance && written.accepted_sample_frames > 0 {
utterance = false;
- let skipped = Duration::from_secs_f64(fit.skip as f64 / sample_rate as f64);
- output.send(Event::Utterance(Duration::from(frame.timestamp) + skipped));
+ output.send(Event::Utterance(Duration::from(frame.timestamp)));
}
if std::mem::take(&mut declare) {
sink.end();
diff --git a/rs/moq-cli/src/play/output.rs b/rs/moq-cli/src/play/output.rs
index ec5435508fa..6b5440378db 100644
--- a/rs/moq-cli/src/play/output.rs
+++ b/rs/moq-cli/src/play/output.rs
@@ -5,6 +5,8 @@
use std::time::Duration;
+use kio::clock::Instant;
+
use moq_audio::playback::Input;
use super::window::Event;
@@ -39,6 +41,11 @@ pub(super) trait Sink: Send + Sync + 'static {
/// Update the callback's playout target without replacing the sink.
fn set_latency(&mut self, latency: Duration) -> anyhow::Result<()>;
+ /// The audio written so far is due to finish sounding at `at`; trailing that
+ /// by more than `late` is skipped, and by as much as the clock's reference
+ /// `jumped` earlier since the last call.
+ fn due(&mut self, at: Instant, late: Duration, jumped: Duration);
+
/// How much audio is queued ahead of the speaker.
fn buffered(&self) -> Duration;
@@ -65,6 +72,10 @@ impl Sink for moq_audio::playback::Sink {
Ok(moq_audio::playback::Sink::set_latency(self, latency)?)
}
+ fn due(&mut self, at: Instant, late: Duration, jumped: Duration) {
+ moq_audio::playback::Sink::due(self, at, late, jumped);
+ }
+
fn write(&mut self, samples: &[u8]) -> anyhow::Result {
Ok(moq_audio::playback::Sink::write(self, samples)?)
}
diff --git a/rs/moq-cli/src/play/timeline.rs b/rs/moq-cli/src/play/timeline.rs
index 32bb3f64991..4e9f49889d2 100644
--- a/rs/moq-cli/src/play/timeline.rs
+++ b/rs/moq-cli/src/play/timeline.rs
@@ -148,27 +148,6 @@ impl AudioTimeline {
}
}
-/// Keep the silence before a frame unless the distance is an emergency.
-/// `buffered` excludes this frame, so normal packet cadence never skips it.
-pub(super) fn fit(buffered: u64, target: u64, slack: u64, silence: u64, frame: u64) -> Fit {
- let ahead = buffered + silence;
- let skip = if ahead >= 4 * (target + slack) {
- (ahead - target).min(silence + frame)
- } else {
- 0
- };
- Fit {
- silence: silence.saturating_sub(skip),
- skip: skip.saturating_sub(silence),
- }
-}
-
-#[derive(Debug, PartialEq, Eq)]
-pub(super) struct Fit {
- pub silence: u64,
- pub skip: u64,
-}
-
#[cfg(test)]
mod tests {
use super::*;
@@ -274,24 +253,4 @@ mod tests {
assert!(!next.reset_sink);
assert_eq!(next.silence, 0);
}
-
- #[test]
- fn emergency_skip_consumes_hole_silence_before_frame_audio() {
- assert_eq!(fit(23_039, 4_800, 960, 0, 960), Fit { silence: 0, skip: 0 });
- assert_eq!(fit(23_040, 4_800, 960, 0, 960), Fit { silence: 0, skip: 960 });
- assert_eq!(
- fit(6_000, 4_800, 960, 18_000, 2_400),
- Fit {
- silence: 0,
- skip: 1_200
- }
- );
- assert_eq!(
- fit(1_000, 4_800, 960, 23_040, 960),
- Fit {
- silence: 3_800,
- skip: 0
- }
- );
- }
}
diff --git a/test/audio-quality/README.md b/test/audio-quality/README.md
index 9327afc3bf3..2cfc4c6f887 100644
--- a/test/audio-quality/README.md
+++ b/test/audio-quality/README.md
@@ -94,8 +94,8 @@ buffer it plays into 128-frame quanta at the device rate and classifies each by
supplied (`moq_audio::playback::Observer`). The meanings are the browser's, with three differences
in what the native player does:
-- Its ring stretches received PCM toward the target. A ring that still runs dry
- emits emergency silence while received PCM refills it; `silence` counts that padding.
+- Its ring stretches received PCM onto the shared clock by the page's rule. A ring that still runs
+ dry emits silence until a cushion of received PCM arrives; `silence` counts that padding.
Native `stalled` is only a sink with nothing written yet.
- The playhead is projected from the player's last write at the wall rate and held at its end.
- `device` is the host's reported output latency, with no `baseLatency`, and `render_load` is
@@ -234,7 +234,7 @@ writes the same samples a page does, so `analyze.ts` reduces both alike, and the
and clock checks have nothing to void. Decoding is taken as instant. A replay is deterministic, so
its budgets are exactly what it measured.
-The ring time-stretches live playback as the page's does, but the replay's audio is a constant, so
+The ring time-stretches live and buffered playback as the page's does, but the replay's audio is a constant, so
every seam admits at the shortest pitch period, 2.5 ms. Accelerate therefore moves less per stretch
than it would from speech; only fast accelerate, which takes every whole period that fits, moves its
most.
@@ -297,11 +297,11 @@ leaves the rest zero, so the tap reads every quantum's fill level from the outpu
player's own count, which its stats panel shows, is read alongside:
- `player_underruns`: `audio.out.stats.underruns`, each time the ring ran dry mid-playback and
- re-stalled to refill, plus each quantum it filled only part-way. The tap files the quanta after a
+ re-stalled to wait for a cushion, plus each quantum it filled only part-way. The tap files the quanta after a
run-dry as stalled time, so `underruns` leaves those out and this counts them. Unset natively.
- `player_skips`: `audio.out.stats.skips`, each jump that dropped buffered audio rather than playing
- it faster. Speeding up is not a skip. Natively, each trim of the ring, and each run of
- consecutive frames `moq play` skipped at the stretch's fast threshold or a full ring refused.
+ it faster. Speeding up is not a skip. Natively, each skip of the ring past its bound, and each
+ run of consecutive frames a full ring refused.
- `underruns`: quanta the ring filled only partly, or not at all, while playing.
- `short_quanta`: the underruns that were partly filled: the ring ran dry part-way through.
- `underrun_episodes`: maximal runs of consecutive underruns, one audible gap each.
diff --git a/test/audio-quality/budgets.json b/test/audio-quality/budgets.json
index ec8faf57025..d80f8797bf1 100644
--- a/test/audio-quality/budgets.json
+++ b/test/audio-quality/budgets.json
@@ -1,5 +1,5 @@
{
- "note": "The browser rows are measured on the nightly runner (ubuntu-24.04-arm, headless Chromium, seed 7, 60 s a row), from the audio-quality-run artifacts of the seven scheduled Nightly runs on main from 2026-10-29 to 2026-11-04 (40885347517, 41065371094, 41204380716, 41313466339, 41483654426, 41656613559, 41852517544), the first after a fresh subscriber stopped receiving the relay's cached history. The six AAC rows other than congestion are measured the same way from the seven runs from 2026-11-05 to 2026-11-11 (42050683248, 42236107954, 42377867523, 42487375545, 42659882487, 42842827718, 43041909115), the first after the TS importer published each PES as one group; the first four predate the browser's time stretch, and the six rows keep the four provisional ceilings set below from 2026-11-09 and 2026-11-10. Each ceiling is the worst of its seven nights times 1.25, plus one runner hiccup, rounded up to two significant figures. The hiccup is sized from the single gaps the 2026-10-08 night put on otherwise clean rows, the fixed-250 control among them: 250 underruns/min, 700 gap ms/min, 3.3 skips/min, 280 dropped ms/min (both sized on the playhead's skip-ahead inference, and now applied to player_skips_per_min and trimmed_ms_per_min), 0.01 stalled, 0.015 silence, and one 20 ms bucket of target. Budgeted this way, every night passes when it is held out of its row's measurement except one, and that row's own ceiling includes it: the 2026-11-03 Opus bursty row raised its target to 300 ms 2.6 s in and held it for one 20 s half-life before settling at 180. Such startup holds, Opus near-zero's 100 ms on 2026-10-30 and 10-31 among them, are budgeted as measured. These ceilings record today's player as the runner hears it, not target audio quality. To re-measure, download the same artifact from the latest scheduled nightlies (`gh run download -n audio-quality-run`), which keep every run, and apply the same rule. On 2026-11-10 the inferred skip_aheads and discarded_ms keys were dropped, since they read a time stretch's catch-up as dropped audio, and every browser row gained player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, by the same rule from the only two scheduled nights on main with today's browser player and its trim counter, 2026-11-09 and 2026-11-10 (42659882487, 42842827718). A lag is signed, so its ceiling is the worst night plus a quarter of its size plus one 20 ms bucket. These four keys are provisional until seven nights with the stretch model re-measure every row. The grade follows the model: audio rests on the shared clock, trails it through a stall, and catches up after. When a change moves a row, a steady-state lag (av_lag_ms p50) that rises, or audio the player trimmed, is a regression to fix rather than budget; a p95 lag that rises through a stall, or an underrun count that moves, is re-measured in. No ceiling is fixed at zero underruns or zero stalls; a replay row's is exactly what it measured. The replay rows are deterministic, so their ceilings are exactly what they measured, through the end of each recording, with no headroom: any regression fails them, and an improvement should be tightened in. They were last re-measured on 2026-10-22 using version 3 traces with encoded frame durations: AAC-LC uses its fixed 1024 samples, and relay-mic retains Opus packet TOC durations. relay-mic was re-recorded on 2026-11-03 from a real microphone, Chrome 156 on a MacBook Air publishing its built-in microphone through the public relay, recorded by headless Chromium on a Linux host (35 s, minimum RTT 2 ms), which raised its target ceiling from 60 to 80 ms and its silence from 0 to 0.001 as measured. The recording path itself has no added RTT. relay-mic-rtt40 is the 2026-11-03 Chrome row at 40 ms of added RTT on that laptop (180 s, minimum RTT 47 ms), kept as the baseline time stretch improves on; like every replay row, its player_underruns_per_min ceiling is what it measures, not a fixed zero. player_underruns_per_min, the player's own counter that its stats panel shows, is budgeted on the replay rows only; it counts the run-dry re-stalls the tap files as stalled time. On 2026-11-07, after the ordered read stopped delivering groups below its cursor (#8071) and took Rust's hole and age bounds (#8218), relay-bbb was re-measured at 16.1 player underruns/min, 0.025 stalled and 0.02 silence, and relay-mic-rtt40 at 8.2, its discarded audio at 70.9 ms/min and its target p95 at 320 ms rather than 780, since the stale backlog it never played no longer reaches the estimator; neither row lost audio the previous player heard. On 2026-11-08 the player began time-stretching live playback with NetEq's accelerate, deciding on the ring's depth right after each arrival and never expanding, and every replay row was re-measured with it, the browser rows left to the nightly runner. Each row's av_lag_ms, how far the playhead trails the shared clock video plays from, is what it measured: the ring rests one frame ahead of the newest arrival's playout, as the trim it replaces left it, so audio leads the clock by 19 to 23 ms at p50. Against the trim, the A/V lag p50 fell from 0.7, -17.5, -21.5, -18.2 and -15.6 ms to -22.4 on bursty-aac, -22.2 on local-aac, -22.7 on relay-bbb, -19 on relay-mic and -20.1 on relay-mic-rtt40, and its p95 fell on every row, most on relay-bbb (10.5 to -4.2) and relay-mic-rtt40 (-0.9 to -13.4). Underruns stayed where the trim had them, as the latency they would cost is not spent: player_underruns_per_min held at 4.4, 12.1, 16.1 and 6.1 and rose from 8.2 to 10.6 on relay-mic-rtt40, and relay-mic's tap now files 2 short quanta a minute, 2 ms of missing audio, where it filed none. What the stretch buys is the audio the trim dropped: discarded_ms_per_min fell from 87.8 to 0 on relay-bbb and from 127.5 to 100.6 on local-aac, whose remainder drops nothing: two receive stalls 14 and 15.3 s in leave audio 159 and 132 ms behind the clock, and the fast accelerate brings it back within half a second. player_skips_per_min and trimmed_ms_per_min, the ring's own count of re-anchors that dropped buffered audio and the media they dropped, are budgeted beside it, at 0 on every recorded row. relay-mic-stall, added 2026-11-10, is relay-mic with nothing arriving from 15 s to 16.5 s and the held frames then trickling in at twice real time until 18 s; today's player trims 2152.5 ms/min of it and its target p95 rises to 1540 ms. Once durations became whole nanoseconds, relay-bbb's delay, 200 ms plus one 1024-sample AAC frame, rounds up by a fraction of a sample, so its ring holds one more sample (23 µs) and its A/V lag p50 measures -22.6 rather than -22.7. Neither replay calibration changed the browser or native ceilings. bursty-aac is the 2026-10-18 AAC bursty isolated capture, recorded with the observer subscribed at the decoder's 2 s budget, so it holds every group the player saw. The native rows are provisional: measured on a loaded developer host (x86_64 Linux, the private PulseAudio null sink, seed 7, 60 s a row) on 2026-10-17, three runs of every row from the debug build nightly uses plus three more of the AAC rows from a release build, which played alike. Each ceiling is the worst clean run times 1.25, plus one native hiccup sized from the unimpaired Opus row's worst run (20 underruns/min, 50 gap ms/min, 2.2 skips/min, 110 discarded ms/min, 0.01 stalled, 0.005 silence, one 20 ms bucket of target), rounded up to two significant figures. On 2026-11-10, after native playback began reporting its A/V lag (#8361), the native rows' player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, and every key of the native and browser congestion rows, were measured by the same rules from three runs of each on a loaded developer host (x86_64 Linux, the debug build, seed 7, 60 s a row), with no void; native audio rests 9 to 95 ms behind its clock at p50. These are provisional until the stretch model's nightly re-measure, which also moves the browser congestion rows onto the runner. On the 2026-10-17 host the null sink's clock ran about 150 ppm slow against the monotonic clock, read within 5 ms at p95 per sample, and its output latency sat between 14 and 28 ms. Five of the 36 AAC rows (none of the 18 Opus rows) voided on the clock: the device stopped pulling for about two seconds mid-row, sometimes after a backend error and a stream restart, which is a native playback defect rather than noise, and the ceilings exclude those runs. Re-measure from the nightly runner's artifacts once it has a week of native rows.",
+ "note": "The browser rows are measured on the nightly runner (ubuntu-24.04-arm, headless Chromium, seed 7, 60 s a row), from the audio-quality-run artifacts of the seven scheduled Nightly runs on main from 2026-10-29 to 2026-11-04 (40885347517, 41065371094, 41204380716, 41313466339, 41483654426, 41656613559, 41852517544), the first after a fresh subscriber stopped receiving the relay's cached history. The six AAC rows other than congestion are measured the same way from the seven runs from 2026-11-05 to 2026-11-11 (42050683248, 42236107954, 42377867523, 42487375545, 42659882487, 42842827718, 43041909115), the first after the TS importer published each PES as one group; the first four predate the browser's time stretch, and the six rows keep the four provisional ceilings set below from 2026-11-09 and 2026-11-10. Each ceiling is the worst of its seven nights times 1.25, plus one runner hiccup, rounded up to two significant figures. The hiccup is sized from the single gaps the 2026-10-08 night put on otherwise clean rows, the fixed-250 control among them: 250 underruns/min, 700 gap ms/min, 3.3 skips/min, 280 dropped ms/min (both sized on the playhead's skip-ahead inference, and now applied to player_skips_per_min and trimmed_ms_per_min), 0.01 stalled, 0.015 silence, and one 20 ms bucket of target. Budgeted this way, every night passes when it is held out of its row's measurement except one, and that row's own ceiling includes it: the 2026-11-03 Opus bursty row raised its target to 300 ms 2.6 s in and held it for one 20 s half-life before settling at 180. Such startup holds, Opus near-zero's 100 ms on 2026-10-30 and 10-31 among them, are budgeted as measured. These ceilings record today's player as the runner hears it, not target audio quality. To re-measure, download the same artifact from the latest scheduled nightlies (`gh run download -n audio-quality-run`), which keep every run, and apply the same rule. On 2026-11-10 the inferred skip_aheads and discarded_ms keys were dropped, since they read a time stretch's catch-up as dropped audio, and every browser row gained player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, by the same rule from the only two scheduled nights on main with today's browser player and its trim counter, 2026-11-09 and 2026-11-10 (42659882487, 42842827718). A lag is signed, so its ceiling is the worst night plus a quarter of its size plus one 20 ms bucket. These four keys are provisional until seven nights with the stretch model re-measure every row. The grade follows the model: audio rests on the shared clock, trails it through a stall, and catches up after. When a change moves a row, a steady-state lag (av_lag_ms p50) that rises, or audio the player trimmed, is a regression to fix rather than budget; a p95 lag that rises through a stall, or an underrun count that moves, is re-measured in. No ceiling is fixed at zero underruns or zero stalls; a replay row's is exactly what it measured. The replay rows are deterministic, so their ceilings are exactly what they measured, through the end of each recording, with no headroom: any regression fails them, and an improvement should be tightened in. They were last re-measured on 2026-10-22 using version 3 traces with encoded frame durations: AAC-LC uses its fixed 1024 samples, and relay-mic retains Opus packet TOC durations. relay-mic was re-recorded on 2026-11-03 from a real microphone, Chrome 156 on a MacBook Air publishing its built-in microphone through the public relay, recorded by headless Chromium on a Linux host (35 s, minimum RTT 2 ms), which raised its target ceiling from 60 to 80 ms and its silence from 0 to 0.001 as measured. The recording path itself has no added RTT. relay-mic-rtt40 is the 2026-11-03 Chrome row at 40 ms of added RTT on that laptop (180 s, minimum RTT 47 ms), kept as the baseline time stretch improves on; like every replay row, its player_underruns_per_min ceiling is what it measures, not a fixed zero. player_underruns_per_min, the player's own counter that its stats panel shows, is budgeted on the replay rows only; it counts the run-dry re-stalls the tap files as stalled time. On 2026-11-07, after the ordered read stopped delivering groups below its cursor (#8071) and took Rust's hole and age bounds (#8218), relay-bbb was re-measured at 16.1 player underruns/min, 0.025 stalled and 0.02 silence, and relay-mic-rtt40 at 8.2, its discarded audio at 70.9 ms/min and its target p95 at 320 ms rather than 780, since the stale backlog it never played no longer reaches the estimator; neither row lost audio the previous player heard. On 2026-11-08 the player began time-stretching live playback with NetEq's accelerate, deciding on the ring's depth right after each arrival and never expanding, and every replay row was re-measured with it, the browser rows left to the nightly runner. Each row's av_lag_ms, how far the playhead trails the shared clock video plays from, is what it measured: the ring rests one frame ahead of the newest arrival's playout, as the trim it replaces left it, so audio leads the clock by 19 to 23 ms at p50. Against the trim, the A/V lag p50 fell from 0.7, -17.5, -21.5, -18.2 and -15.6 ms to -22.4 on bursty-aac, -22.2 on local-aac, -22.7 on relay-bbb, -19 on relay-mic and -20.1 on relay-mic-rtt40, and its p95 fell on every row, most on relay-bbb (10.5 to -4.2) and relay-mic-rtt40 (-0.9 to -13.4). Underruns stayed where the trim had them, as the latency they would cost is not spent: player_underruns_per_min held at 4.4, 12.1, 16.1 and 6.1 and rose from 8.2 to 10.6 on relay-mic-rtt40, and relay-mic's tap now files 2 short quanta a minute, 2 ms of missing audio, where it filed none. What the stretch buys is the audio the trim dropped: discarded_ms_per_min fell from 87.8 to 0 on relay-bbb and from 127.5 to 100.6 on local-aac, whose remainder drops nothing: two receive stalls 14 and 15.3 s in leave audio 159 and 132 ms behind the clock, and the fast accelerate brings it back within half a second. player_skips_per_min and trimmed_ms_per_min, the ring's own count of re-anchors that dropped buffered audio and the media they dropped, are budgeted beside it, at 0 on every recorded row. relay-mic-stall, added 2026-11-10, is relay-mic with nothing arriving from 15 s to 16.5 s and the held frames then trickling in at twice real time until 18 s; today's player trims 2152.5 ms/min of it and its target p95 rises to 1540 ms. Once durations became whole nanoseconds, relay-bbb's delay, 200 ms plus one 1024-sample AAC frame, rounds up by a fraction of a sample, so its ring holds one more sample (23 µs) and its A/V lag p50 measures -22.6 rather than -22.7. Neither replay calibration changed the browser or native ceilings. bursty-aac is the 2026-10-18 AAC bursty isolated capture, recorded with the observer subscribed at the decoder's 2 s budget, so it holds every group the player saw. The native rows are provisional: measured on a loaded developer host (x86_64 Linux, the private PulseAudio null sink, seed 7, 60 s a row) on 2026-10-17, three runs of every row from the debug build nightly uses plus three more of the AAC rows from a release build, which played alike. Each ceiling is the worst clean run times 1.25, plus one native hiccup sized from the unimpaired Opus row's worst run (20 underruns/min, 50 gap ms/min, 2.2 skips/min, 110 discarded ms/min, 0.01 stalled, 0.005 silence, one 20 ms bucket of target), rounded up to two significant figures. On 2026-11-10, after native playback began reporting its A/V lag (#8361), the native rows' player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, and every key of the native and browser congestion rows, were measured by the same rules from three runs of each on a loaded developer host (x86_64 Linux, the debug build, seed 7, 60 s a row), with no void; native audio rests 9 to 95 ms behind its clock at p50. These are provisional until the stretch model's nightly re-measure, which also moves the browser congestion rows onto the runner. On the 2026-10-17 host the null sink's clock ran about 150 ppm slow against the monotonic clock, read within 5 ms at p95 per sample, and its output latency sat between 14 and 28 ms. Five of the 36 AAC rows (none of the 18 Opus rows) voided on the clock: the device stopped pulling for about two seconds mid-row, sometimes after a backend error and a stream restart, which is a native playback defect rather than noise, and the ceilings exclude those runs. Re-measure from the nightly runner's artifacts once it has a week of native rows. On 2026-11-11 every player began following one stretch rule on the shared clock: it slows down below its target, speeds up above it, and skips only past its bound, which in the browser is the subscription's max delay. Every replay row was re-measured with it, the browser and native rows left to the nightly runner. Audio now rests on the clock rather than one frame ahead of it, so the A/V lag p50 rose from about -20 ms to 0.4 ms on relay-bbb, 1.1 on relay-mic, 1.2 on relay-mic-rtt40 and 1.8 on local-aac. Its p95 rose on every row, since audio that falls behind is now caught up by speeding up rather than trimmed: the shared clock steps forward as its reference converges through a tune-in backlog, and again when a target falls, and audio follows at the accelerate's pace, which on the replay's constant tone is one 2.5 ms period a window, slower than on speech. bursty-aac rests 5 ms behind its clock, so its p50 of 10.4 ms is mostly its catch-ups, and relay-mic-stall's p50 of 66.1 ms is the catch-up after its target falls from 1540 to 80 ms, which outlasts half the row. In exchange nothing is trimmed: relay-mic-stall's trimmed_ms_per_min fell from 2152.5 to 0, its skips from 6.1 to 0, and its underruns from 24.2 to 0 a minute, and player_underruns_per_min fell from 12.1 to 4 on local-aac, 16.1 to 0 on relay-bbb, 6.1 to 0 on relay-mic, 10.6 to 2.7 on relay-mic-rtt40 and 6.1 to 4 on relay-mic-stall, holding at 4.4 on bursty-aac. The tap now files 3.3 to 6.1 short quanta a minute on the AAC rows, under 0.3 ms of missing audio, where it filed none. To deliver the audio a stall delays, the container consumer now waits and skips at the subscription's max delay, as native already does, rather than at the delay plus lookahead. relay-mic-rtt40's join backlog arrives reordered, so the consumer now waits on its holes and the estimator observes the frames that fill them, up to 775 ms late, and its target p95 returns from 320 to 780 ms, held for one half-life after the join. Later on 2026-11-11 audio began skipping as far as the shared clock's reference moves earlier, once those moves add up to one stretch window and only as far as it holds audio past its target, rather than speeding through them, and every replay row was re-measured. The tune-in backlog's jumps now land before the graded window, so the A/V lag p95 fell from 453.1 to 2.1 ms on relay-bbb (its p50 from 0.4 to -1.3), 743.7 to 1.9 on relay-mic-rtt40, whose p50 fell from 1.2 to 0.4 and its tap's missing audio rose from 0.6 to 0.9 ms a minute, since a jump past what is decoded skips the rest once it lands,, 335.5 to 39.2 on local-aac and 88.6 to 0.3 on relay-mic, and the p50 from 66.1 to 25.3 on relay-mic-stall, with nothing trimmed in any graded window. The tap's underruns fell from 6.1 to 2 a minute on local-aac and 6.1 to 0 on relay-bbb; local-aac's silence rose from 0.001 to 0.002.",
"rows": [
{
"runtime": "chromium",
@@ -428,94 +428,94 @@
"profile": "bursty-aac",
"player_underruns_per_min": 4.4,
"player_skips_per_min": 0,
- "underruns_per_min": 0,
- "underrun_ms_per_min": 0,
+ "underruns_per_min": 3.3,
+ "underrun_ms_per_min": 0.1,
"trimmed_ms_per_min": 0,
- "stalled_share": 0.009,
- "silence_share": 0.012,
+ "stalled_share": 0.005,
+ "silence_share": 0.007,
"target_ms_p95": 343.2,
- "av_lag_ms_p50": -22.4,
- "av_lag_ms_p95": -22.4
+ "av_lag_ms_p50": 10.4,
+ "av_lag_ms_p95": 162.1
},
{
"runtime": "replay",
"codec": "aac",
"rate": 44100,
"profile": "local-aac",
- "player_underruns_per_min": 12.1,
+ "player_underruns_per_min": 4,
"player_skips_per_min": 0,
- "underruns_per_min": 0,
- "underrun_ms_per_min": 0,
+ "underruns_per_min": 2,
+ "underrun_ms_per_min": 0.1,
"trimmed_ms_per_min": 0,
- "stalled_share": 0.017,
- "silence_share": 0.021,
+ "stalled_share": 0,
+ "silence_share": 0.002,
"target_ms_p95": 203.2,
- "av_lag_ms_p50": -22.2,
- "av_lag_ms_p95": -4.2
+ "av_lag_ms_p50": 1.8,
+ "av_lag_ms_p95": 39.2
},
{
"runtime": "replay",
"codec": "aac",
"rate": 44100,
"profile": "relay-bbb",
- "player_underruns_per_min": 16.1,
+ "player_underruns_per_min": 0,
"player_skips_per_min": 0,
"underruns_per_min": 0,
"underrun_ms_per_min": 0,
"trimmed_ms_per_min": 0,
- "stalled_share": 0.025,
- "silence_share": 0.02,
+ "stalled_share": 0,
+ "silence_share": 0,
"target_ms_p95": 223.2,
- "av_lag_ms_p50": -22.6,
- "av_lag_ms_p95": -4.2
+ "av_lag_ms_p50": -1.3,
+ "av_lag_ms_p95": 2.1
},
{
"runtime": "replay",
"codec": "opus",
"rate": 48000,
"profile": "relay-mic",
- "player_underruns_per_min": 6.1,
+ "player_underruns_per_min": 0,
"player_skips_per_min": 0,
- "underruns_per_min": 2,
- "underrun_ms_per_min": 2,
+ "underruns_per_min": 0,
+ "underrun_ms_per_min": 0,
"trimmed_ms_per_min": 0,
"stalled_share": 0,
- "silence_share": 0.002,
+ "silence_share": 0,
"target_ms_p95": 80,
- "av_lag_ms_p50": -19,
- "av_lag_ms_p95": -18.5
+ "av_lag_ms_p50": 0.3,
+ "av_lag_ms_p95": 0.3
},
{
"runtime": "replay",
"codec": "opus",
"rate": 48000,
"profile": "relay-mic-rtt40",
- "player_underruns_per_min": 10.6,
+ "player_underruns_per_min": 2.7,
"player_skips_per_min": 0,
"underruns_per_min": 0.7,
- "underrun_ms_per_min": 0.5,
+ "underrun_ms_per_min": 0.9,
"trimmed_ms_per_min": 0,
- "stalled_share": 0.003,
- "silence_share": 0.004,
- "target_ms_p95": 320,
- "av_lag_ms_p50": -20.1,
- "av_lag_ms_p95": -13.4
+ "stalled_share": 0,
+ "silence_share": 0,
+ "target_ms_p95": 780,
+ "av_lag_ms_p50": 0.4,
+ "av_lag_ms_p95": 1.9
},
{
"runtime": "replay",
"codec": "opus",
"rate": 48000,
"profile": "relay-mic-stall",
- "player_underruns_per_min": 6.1,
- "player_skips_per_min": 6.1,
- "underruns_per_min": 24.2,
- "underrun_ms_per_min": 61.2,
- "trimmed_ms_per_min": 2152.5,
- "stalled_share": 0.067,
- "silence_share": 0.076,
+ "player_underruns_per_min": 4,
+ "player_skips_per_min": 0,
+ "underruns_per_min": 0,
+ "underrun_ms_per_min": 0,
+ "trimmed_ms_per_min": 0,
+ "stalled_share": 0.05,
+ "silence_share": 0.049,
"target_ms_p95": 1540,
- "av_lag_ms_p50": -19.5,
- "av_lag_ms_p95": -9.5
+ "av_lag_ms_p50": 25.3,
+ "av_lag_ms_p95": 501.8
}
]
}