From 40ad9909c83df1f778cc20947de27e5d3cfdcb43 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 22:54:15 -0700 Subject: [PATCH 1/7] chore(quest): claim quest/m1/audio-jitter-target/capped-budgets.md Quest-Claim: d9bd610d15dbd882 From 8add6f8e107849e2fa4ef1f05ed7ce34830fd7ed Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 22:56:48 -0700 Subject: [PATCH 2/7] quest: record the history fill behind the October 25 to 28 browser rows Co-Authored-By: Claude Opus 5.5 --- .../m1/audio-jitter-target/capped-budgets.md | 37 ++++++++++++++----- 1 file changed, 27 insertions(+), 10 deletions(-) diff --git a/quest/m1/audio-jitter-target/capped-budgets.md b/quest/m1/audio-jitter-target/capped-budgets.md index 80e93ef7724..91f40549cae 100644 --- a/quest/m1/audio-jitter-target/capped-budgets.md +++ b/quest/m1/audio-jitter-target/capped-budgets.md @@ -23,9 +23,9 @@ nightly, because estimator and player changes landed after the cap [moq-dev/moq#6525](https://github.com/moq-dev/moq/pull/6525), [moq-dev/moq#6484](https://github.com/moq-dev/moq/pull/6484), [moq-dev/moq#6609](https://github.com/moq-dev/moq/pull/6609)), the last on -2026-10-24. That makes 3 of 7 so far (2026-10-25, 10-26 and 10-27), with the -seventh due about 2026-10-31. The earlier post-cap runs below no longer -count toward the seven. Re-measure by the rule in the budgets note, and drop the note's sentence +2026-10-24. Five have run (2026-10-25 to 10-29), with the seventh due about +2026-10-31, but whether the first four count is open (below). The earlier +post-cap runs below no longer count toward the seven. Re-measure by the rule in the budgets note, and drop the note's sentence that budgets the AAC bursty discards against Opus's as measured. Nightlies from before the one-ring removal also carry `-isolated` rows; grade only the `-plain` rows, which are the browser rows now. @@ -70,11 +70,28 @@ seven-frame PES flush is seven one-frame groups, delivered newest first, so the oldest frame of a flush waits behind every newer group on a path that cannot carry them together. -The fourth run, [2026-10-28 nightly](https://github.com/moq-dev/moq/actions/runs/40698329026/job/122851001011), -breaches six browser ceilings: AAC wide discards 1790.9 ms/min against 460 -and target p95 2023.2 ms against 450; Opus bursty target p95 540 ms against -370, mild 160 against 120, and wide discards 1913.9 ms/min against 420 and -target p95 2020 ms against 570. Preserve this run with the other post-cap -measurements and distinguish startup/runner stalls from sustained estimator -behavior before re-measuring. The separate publisher fairness timeout was +The 2026-10-25 to 10-28 nightlies +([40192013303](https://github.com/moq-dev/moq/actions/runs/40192013303), +[40332938121](https://github.com/moq-dev/moq/actions/runs/40332938121), +[40506938673](https://github.com/moq-dev/moq/actions/runs/40506938673), +[40698329026](https://github.com/moq-dev/moq/actions/runs/40698329026)) +measured a relay bug rather than the player: 2 to 4.5 s into every row except +`fixed-250` and Opus `near-zero`, the subscriber received 15 to 25 s of the +publisher's cached history, newest first. On the `wide` rows the estimator +took that as a late interval and held a 2020 ms target for about 20 s, +then discarded about 1.8 to 1.9 s of audio a minute falling back, every night; +the 10-28 Opus `bursty` 540 ms target is the same arrival. No `wide` or +`bursty` row logged more than one long task, of at most 58 ms. +[moq-dev/moq#7168](https://github.com/moq-dev/moq/pull/7168) stopped a fresh +subscriber filling history below its start, merged 2026-10-29T00:52Z, and the +[2026-10-29 nightly](https://github.com/moq-dev/moq/actions/runs/40885347517) +is the first after it: the oldest frame any row received is about 2.6 s +behind live, every browser row is within budget, and AAC `wide` discards +205.6 ms/min at a 323.2 ms target p95, Opus `wide` 243.4 at 460. Across all +five nights the AAC `bursty` rows discard 0 to 238.1 ms a minute against 0 to +370.9 for Opus `bursty`, and Opus `near-zero` has no underrun episodes, at a +60 ms target p95 on three nights and 80 on two (the 120 ms prior decays to 60 +only at 21.4 s, after the 5 s warmup). Each night's browser metrics are kept +in [moq-dev/moq#7286](https://github.com/moq-dev/moq/pull/7286), since the +artifacts expire. The separate publisher fairness timeout was excess per-object work, fixed by [moq-dev/moq#7133](https://github.com/moq-dev/moq/pull/7133). From f2b4c64f69536a9a4e11378c80467ec49de414f2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 09:27:21 -0800 Subject: [PATCH 3/7] chore(quest): claim quest/m1/audio-jitter-target/capped-budgets.md Quest-Claim: a20f6dd06c45c2c2 From b6c6ef7f7d4e240db3514f0595c58d59669adeb2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 09:30:58 -0800 Subject: [PATCH 4/7] quest: name the nightlies still to grade in capped-budgets Co-Authored-By: Claude Haiku 5.5 --- quest/m1/audio-jitter-target/capped-budgets.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/quest/m1/audio-jitter-target/capped-budgets.md b/quest/m1/audio-jitter-target/capped-budgets.md index 586b61f9f80..a8f0004855c 100644 --- a/quest/m1/audio-jitter-target/capped-budgets.md +++ b/quest/m1/audio-jitter-target/capped-budgets.md @@ -11,11 +11,13 @@ every AAC frame on those nights arrives in order. ## Waiting - moq-dev/moq: seven scheduled Nightly runs on `main` after - [moq-dev/moq#7898](https://github.com/moq-dev/moq/pull/7898) merges. Find them - with `gh run list --workflow nightly.yml --event schedule` and download each - with `gh run download -n audio-quality-run`; never dispatch the nightly by - hand. The artifacts expire after 7 days, so download each run's as it lands - rather than waiting for the seventh. + [moq-dev/moq#7898](https://github.com/moq-dev/moq/pull/7898) merges. Four are + graded in [moq-dev/moq#8309](https://github.com/moq-dev/moq/pull/8309)'s body + (2026-11-05 to 11-08); the remaining three are the 2026-11-09, 11-10 and 11-11 + scheduled runs. Find them with `gh run list --workflow nightly.yml --event + schedule` and download each with `gh run download -n audio-quality-run`; + never dispatch the nightly by hand. The artifacts expire after 7 days, so + download each run's as it lands rather than waiting for the seventh. ## Plan From 0b4563f9be06c08c4cfff5637a147073b6bf4d34 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Mon, 9 Nov 2026 19:34:33 -0800 Subject: [PATCH 5/7] quest: five nightlies graded, two left in capped-budgets Co-Authored-By: Claude Haiku 5.5 --- quest/m1/audio-jitter-target/capped-budgets.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/quest/m1/audio-jitter-target/capped-budgets.md b/quest/m1/audio-jitter-target/capped-budgets.md index a8f0004855c..d33ed463d3c 100644 --- a/quest/m1/audio-jitter-target/capped-budgets.md +++ b/quest/m1/audio-jitter-target/capped-budgets.md @@ -11,10 +11,10 @@ every AAC frame on those nights arrives in order. ## Waiting - moq-dev/moq: seven scheduled Nightly runs on `main` after - [moq-dev/moq#7898](https://github.com/moq-dev/moq/pull/7898) merges. Four are + [moq-dev/moq#7898](https://github.com/moq-dev/moq/pull/7898) merges. Five are graded in [moq-dev/moq#8309](https://github.com/moq-dev/moq/pull/8309)'s body - (2026-11-05 to 11-08); the remaining three are the 2026-11-09, 11-10 and 11-11 - scheduled runs. Find them with `gh run list --workflow nightly.yml --event + (2026-11-05 to 11-09); the remaining two are the 2026-11-10 and 11-11 scheduled + runs. Find them with `gh run list --workflow nightly.yml --event schedule` and download each with `gh run download -n audio-quality-run`; never dispatch the nightly by hand. The artifacts expire after 7 days, so download each run's as it lands rather than waiting for the seventh. From 793b8444fe0c75507ed84ba0f0c8c589c1705b11 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 13:30:03 -0800 Subject: [PATCH 6/7] test(audio-quality): re-measure the AAC browser budgets from seven nights of grouped PES The six AAC browser rows other than congestion take their ceilings from the 2026-11-05 to 2026-11-11 scheduled nightlies, all after the TS importer began publishing each PES as one group, by the note's rule. The note drops its sentence that the AAC rows predate the grouping, and the finished quest goes. Co-Authored-By: Claude Opus 5.5 --- quest/m1/audio-jitter-target/calibrate.md | 4 -- .../m1/audio-jitter-target/capped-budgets.md | 54 ------------------- test/audio-quality/budgets.json | 18 +++---- 3 files changed, 9 insertions(+), 67 deletions(-) delete mode 100644 quest/m1/audio-jitter-target/capped-budgets.md diff --git a/quest/m1/audio-jitter-target/calibrate.md b/quest/m1/audio-jitter-target/calibrate.md index 3bb09ee1561..5d283405a1b 100644 --- a/quest/m1/audio-jitter-target/calibrate.md +++ b/quest/m1/audio-jitter-target/calibrate.md @@ -52,7 +52,3 @@ Native rows stay report-only; enforcing them belongs to ## Required - [Stretch model](/quest/m1/audio-jitter-target/stretch.md) - the player these ceilings measure - -## Related - -- [Capped budgets](/quest/m1/audio-jitter-target/capped-budgets.md) - re-measures the AAC browser rows after the TS importer change; if it has not finished when the model lands, these seven nights serve both diff --git a/quest/m1/audio-jitter-target/capped-budgets.md b/quest/m1/audio-jitter-target/capped-budgets.md deleted file mode 100644 index 2e46a833c5b..00000000000 --- a/quest/m1/audio-jitter-target/capped-budgets.md +++ /dev/null @@ -1,54 +0,0 @@ -# [XS] Re-measure the AAC browser budgets after the TS importer groups each PES - -## Goal - -`test/audio-quality/budgets.json` carries the six AAC browser rows' ceilings -re-measured, by the rule in its note, from seven scheduled nightlies on `main` -that all ran after the TS importer began publishing each PES as one group, and -the note's sentence that the AAC rows predate the grouping is dropped. Nearly -every AAC frame on those nights arrives in order. - -## Waiting - -- moq-dev/moq: seven scheduled Nightly runs on `main` after - [moq-dev/moq#7898](https://github.com/moq-dev/moq/pull/7898) merges. Four are - graded in [moq-dev/moq#8309](https://github.com/moq-dev/moq/pull/8309)'s body - (2026-11-05 to 11-08); the remaining three are the 2026-11-09, 11-10 and 11-11 - scheduled runs. Find them with `gh run list --workflow nightly.yml --event - schedule` and download each with `gh run download -n audio-quality-run`; - never dispatch the nightly by hand. The artifacts expire after 7 days, so - download each run's as it lands rather than waiting for the seventh. - -## Plan - -Decided 2026-11-04 (maintainer, on -[moq-dev/moq#7898](https://github.com/moq-dev/moq/pull/7898)): land one group -per PES as is. The extra AAC `wide` re-stalls are for Watch's time stretch to -absorb, and this quest no longer expects AAC `wide` to stop running dry. -Options shown: - -- ✅ Land as is -- Land only the budgets, holding the importer change until time stretch lands -- Keep the change and deepen AAC's target - -[moq-dev/moq#7898](https://github.com/moq-dev/moq/pull/7898) re-measured every -browser row from the 2026-10-29 to 2026-11-04 nightlies (its body keeps each -night's metrics) and changed the publisher: `moq import ts` used to send each -seven-frame AAC PES as seven one-frame groups, which the transport delivers -newest first, so 82 to 86% of AAC frames arrived behind a newer one. Each PES is -now one group, its frames in presentation order. - -On those nights, before the change, AAC `bursty` discarded up to 368.3 ms a -minute against Opus `bursty`'s 117.4. Replaying their captures with each PES's -arrival slots handed to its frames oldest first, as one group, halved the AAC -`bursty` and `wide` discards (`bursty` summed over the seven nights 1156 to 584 -ms a minute) and lowered both rows' target by 20 to 60 ms. That shallower target doubled AAC -`wide`'s player re-stalls (mean 3.0 to 6.6 a minute), the 0.95 quantile's trade -that the browser's time stretch absorbs. - -AAC `bursty` still discards about three times what Opus `bursty` does (584 -against 174 summed), and parity is not expected: the harness keeps ffmpeg's -default PES packing on purpose, so AAC frames reach the viewer a median 210 ms -behind their media time on `bursty` against Opus's 41. Grade the same -`chromium-aac-*` rows, and check the in-order claim with the share of each row's -`.arrivals.ndjson` frames that arrive behind a newer one. diff --git a/test/audio-quality/budgets.json b/test/audio-quality/budgets.json index 42cf24ec64c..edfcc1f2512 100644 --- a/test/audio-quality/budgets.json +++ b/test/audio-quality/budgets.json @@ -1,5 +1,5 @@ { - "note": "The browser rows are measured on the nightly runner (ubuntu-24.04-arm, headless Chromium, seed 7, 60 s a row), from the audio-quality-run artifacts of the seven scheduled Nightly runs on main from 2026-10-29 to 2026-11-04 (40885347517, 41065371094, 41204380716, 41313466339, 41483654426, 41656613559, 41852517544), the first after a fresh subscriber stopped receiving the relay's cached history. The AAC rows predate the TS importer publishing each PES as one group. Each ceiling is the worst of its seven nights times 1.25, plus one runner hiccup, rounded up to two significant figures. The hiccup is sized from the single gaps the 2026-10-08 night put on otherwise clean rows, the fixed-250 control among them: 250 underruns/min, 700 gap ms/min, 3.3 skips/min, 280 dropped ms/min (both sized on the playhead's skip-ahead inference, and now applied to player_skips_per_min and trimmed_ms_per_min), 0.01 stalled, 0.015 silence, and one 20 ms bucket of target. Budgeted this way, every night passes when it is held out of its row's measurement except one, and that row's own ceiling includes it: the 2026-11-03 Opus bursty row raised its target to 300 ms 2.6 s in and held it for one 20 s half-life before settling at 180. Such startup holds, Opus near-zero's 100 ms on 2026-10-30 and 10-31 among them, are budgeted as measured. These ceilings record today's player as the runner hears it, not target audio quality. To re-measure, download the same artifact from the latest scheduled nightlies (`gh run download -n audio-quality-run`), which keep every run, and apply the same rule. On 2026-11-10 the inferred skip_aheads and discarded_ms keys were dropped, since they read a time stretch's catch-up as dropped audio, and every browser row gained player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, by the same rule from the only two scheduled nights on main with today's browser player and its trim counter, 2026-11-09 and 2026-11-10 (42659882487, 42842827718). A lag is signed, so its ceiling is the worst night plus a quarter of its size plus one 20 ms bucket. These four keys are provisional until seven nights with the stretch model re-measure every row. The grade follows the model: audio rests on the shared clock, trails it through a stall, and catches up after. When a change moves a row, a steady-state lag (av_lag_ms p50) that rises, or audio the player trimmed, is a regression to fix rather than budget; a p95 lag that rises through a stall, or an underrun count that moves, is re-measured in. No ceiling is fixed at zero underruns or zero stalls; a replay row's is exactly what it measured. The replay rows are deterministic, so their ceilings are exactly what they measured, through the end of each recording, with no headroom: any regression fails them, and an improvement should be tightened in. They were last re-measured on 2026-10-22 using version 3 traces with encoded frame durations: AAC-LC uses its fixed 1024 samples, and relay-mic retains Opus packet TOC durations. relay-mic was re-recorded on 2026-11-03 from a real microphone, Chrome 156 on a MacBook Air publishing its built-in microphone through the public relay, recorded by headless Chromium on a Linux host (35 s, minimum RTT 2 ms), which raised its target ceiling from 60 to 80 ms and its silence from 0 to 0.001 as measured. The recording path itself has no added RTT. relay-mic-rtt40 is the 2026-11-03 Chrome row at 40 ms of added RTT on that laptop (180 s, minimum RTT 47 ms), kept as the baseline time stretch improves on; like every replay row, its player_underruns_per_min ceiling is what it measures, not a fixed zero. player_underruns_per_min, the player's own counter that its stats panel shows, is budgeted on the replay rows only; it counts the run-dry re-stalls the tap files as stalled time. On 2026-11-07, after the ordered read stopped delivering groups below its cursor (#8071) and took Rust's hole and age bounds (#8218), relay-bbb was re-measured at 16.1 player underruns/min, 0.025 stalled and 0.02 silence, and relay-mic-rtt40 at 8.2, its discarded audio at 70.9 ms/min and its target p95 at 320 ms rather than 780, since the stale backlog it never played no longer reaches the estimator; neither row lost audio the previous player heard. On 2026-11-08 the player began time-stretching live playback with NetEq's accelerate, deciding on the ring's depth right after each arrival and never expanding, and every replay row was re-measured with it, the browser rows left to the nightly runner. Each row's av_lag_ms, how far the playhead trails the shared clock video plays from, is what it measured: the ring rests one frame ahead of the newest arrival's playout, as the trim it replaces left it, so audio leads the clock by 19 to 23 ms at p50. Against the trim, the A/V lag p50 fell from 0.7, -17.5, -21.5, -18.2 and -15.6 ms to -22.4 on bursty-aac, -22.2 on local-aac, -22.7 on relay-bbb, -19 on relay-mic and -20.1 on relay-mic-rtt40, and its p95 fell on every row, most on relay-bbb (10.5 to -4.2) and relay-mic-rtt40 (-0.9 to -13.4). Underruns stayed where the trim had them, as the latency they would cost is not spent: player_underruns_per_min held at 4.4, 12.1, 16.1 and 6.1 and rose from 8.2 to 10.6 on relay-mic-rtt40, and relay-mic's tap now files 2 short quanta a minute, 2 ms of missing audio, where it filed none. What the stretch buys is the audio the trim dropped: discarded_ms_per_min fell from 87.8 to 0 on relay-bbb and from 127.5 to 100.6 on local-aac, whose remainder drops nothing: two receive stalls 14 and 15.3 s in leave audio 159 and 132 ms behind the clock, and the fast accelerate brings it back within half a second. player_skips_per_min and trimmed_ms_per_min, the ring's own count of re-anchors that dropped buffered audio and the media they dropped, are budgeted beside it, at 0 on every recorded row. relay-mic-stall, added 2026-11-10, is relay-mic with nothing arriving from 15 s to 16.5 s and the held frames then trickling in at twice real time until 18 s; today's player trims 2152.5 ms/min of it and its target p95 rises to 1540 ms. Once durations became whole nanoseconds, relay-bbb's delay, 200 ms plus one 1024-sample AAC frame, rounds up by a fraction of a sample, so its ring holds one more sample (23 µs) and its A/V lag p50 measures -22.6 rather than -22.7. Neither replay calibration changed the browser or native ceilings. bursty-aac is the 2026-10-18 AAC bursty isolated capture, recorded with the observer subscribed at the decoder's 2 s budget, so it holds every group the player saw. The native rows are provisional: measured on a loaded developer host (x86_64 Linux, the private PulseAudio null sink, seed 7, 60 s a row) on 2026-10-17, three runs of every row from the debug build nightly uses plus three more of the AAC rows from a release build, which played alike. Each ceiling is the worst clean run times 1.25, plus one native hiccup sized from the unimpaired Opus row's worst run (20 underruns/min, 50 gap ms/min, 2.2 skips/min, 110 discarded ms/min, 0.01 stalled, 0.005 silence, one 20 ms bucket of target), rounded up to two significant figures. On 2026-11-10, after native playback began reporting its A/V lag (#8361), the native rows' player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, and every key of the native and browser congestion rows, were measured by the same rules from three runs of each on a loaded developer host (x86_64 Linux, the debug build, seed 7, 60 s a row), with no void; native audio rests 9 to 95 ms behind its clock at p50. These are provisional until the stretch model's nightly re-measure, which also moves the browser congestion rows onto the runner. On the 2026-10-17 host the null sink's clock ran about 150 ppm slow against the monotonic clock, read within 5 ms at p95 per sample, and its output latency sat between 14 and 28 ms. Five of the 36 AAC rows (none of the 18 Opus rows) voided on the clock: the device stopped pulling for about two seconds mid-row, sometimes after a backend error and a stream restart, which is a native playback defect rather than noise, and the ceilings exclude those runs. Re-measure from the nightly runner's artifacts once it has a week of native rows.", + "note": "The browser rows are measured on the nightly runner (ubuntu-24.04-arm, headless Chromium, seed 7, 60 s a row), from the audio-quality-run artifacts of the seven scheduled Nightly runs on main from 2026-10-29 to 2026-11-04 (40885347517, 41065371094, 41204380716, 41313466339, 41483654426, 41656613559, 41852517544), the first after a fresh subscriber stopped receiving the relay's cached history. The six AAC rows other than congestion are measured the same way from the seven runs from 2026-11-05 to 2026-11-11 (42050683248, 42236107954, 42377867523, 42487375545, 42659882487, 42842827718, 43041909115), the first after the TS importer published each PES as one group; the first four predate the browser's time stretch, and their four provisional keys below are unchanged. Each ceiling is the worst of its seven nights times 1.25, plus one runner hiccup, rounded up to two significant figures. The hiccup is sized from the single gaps the 2026-10-08 night put on otherwise clean rows, the fixed-250 control among them: 250 underruns/min, 700 gap ms/min, 3.3 skips/min, 280 dropped ms/min (both sized on the playhead's skip-ahead inference, and now applied to player_skips_per_min and trimmed_ms_per_min), 0.01 stalled, 0.015 silence, and one 20 ms bucket of target. Budgeted this way, every night passes when it is held out of its row's measurement except one, and that row's own ceiling includes it: the 2026-11-03 Opus bursty row raised its target to 300 ms 2.6 s in and held it for one 20 s half-life before settling at 180. Such startup holds, Opus near-zero's 100 ms on 2026-10-30 and 10-31 among them, are budgeted as measured. These ceilings record today's player as the runner hears it, not target audio quality. To re-measure, download the same artifact from the latest scheduled nightlies (`gh run download -n audio-quality-run`), which keep every run, and apply the same rule. On 2026-11-10 the inferred skip_aheads and discarded_ms keys were dropped, since they read a time stretch's catch-up as dropped audio, and every browser row gained player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, by the same rule from the only two scheduled nights on main with today's browser player and its trim counter, 2026-11-09 and 2026-11-10 (42659882487, 42842827718). A lag is signed, so its ceiling is the worst night plus a quarter of its size plus one 20 ms bucket. These four keys are provisional until seven nights with the stretch model re-measure every row. The grade follows the model: audio rests on the shared clock, trails it through a stall, and catches up after. When a change moves a row, a steady-state lag (av_lag_ms p50) that rises, or audio the player trimmed, is a regression to fix rather than budget; a p95 lag that rises through a stall, or an underrun count that moves, is re-measured in. No ceiling is fixed at zero underruns or zero stalls; a replay row's is exactly what it measured. The replay rows are deterministic, so their ceilings are exactly what they measured, through the end of each recording, with no headroom: any regression fails them, and an improvement should be tightened in. They were last re-measured on 2026-10-22 using version 3 traces with encoded frame durations: AAC-LC uses its fixed 1024 samples, and relay-mic retains Opus packet TOC durations. relay-mic was re-recorded on 2026-11-03 from a real microphone, Chrome 156 on a MacBook Air publishing its built-in microphone through the public relay, recorded by headless Chromium on a Linux host (35 s, minimum RTT 2 ms), which raised its target ceiling from 60 to 80 ms and its silence from 0 to 0.001 as measured. The recording path itself has no added RTT. relay-mic-rtt40 is the 2026-11-03 Chrome row at 40 ms of added RTT on that laptop (180 s, minimum RTT 47 ms), kept as the baseline time stretch improves on; like every replay row, its player_underruns_per_min ceiling is what it measures, not a fixed zero. player_underruns_per_min, the player's own counter that its stats panel shows, is budgeted on the replay rows only; it counts the run-dry re-stalls the tap files as stalled time. On 2026-11-07, after the ordered read stopped delivering groups below its cursor (#8071) and took Rust's hole and age bounds (#8218), relay-bbb was re-measured at 16.1 player underruns/min, 0.025 stalled and 0.02 silence, and relay-mic-rtt40 at 8.2, its discarded audio at 70.9 ms/min and its target p95 at 320 ms rather than 780, since the stale backlog it never played no longer reaches the estimator; neither row lost audio the previous player heard. On 2026-11-08 the player began time-stretching live playback with NetEq's accelerate, deciding on the ring's depth right after each arrival and never expanding, and every replay row was re-measured with it, the browser rows left to the nightly runner. Each row's av_lag_ms, how far the playhead trails the shared clock video plays from, is what it measured: the ring rests one frame ahead of the newest arrival's playout, as the trim it replaces left it, so audio leads the clock by 19 to 23 ms at p50. Against the trim, the A/V lag p50 fell from 0.7, -17.5, -21.5, -18.2 and -15.6 ms to -22.4 on bursty-aac, -22.2 on local-aac, -22.7 on relay-bbb, -19 on relay-mic and -20.1 on relay-mic-rtt40, and its p95 fell on every row, most on relay-bbb (10.5 to -4.2) and relay-mic-rtt40 (-0.9 to -13.4). Underruns stayed where the trim had them, as the latency they would cost is not spent: player_underruns_per_min held at 4.4, 12.1, 16.1 and 6.1 and rose from 8.2 to 10.6 on relay-mic-rtt40, and relay-mic's tap now files 2 short quanta a minute, 2 ms of missing audio, where it filed none. What the stretch buys is the audio the trim dropped: discarded_ms_per_min fell from 87.8 to 0 on relay-bbb and from 127.5 to 100.6 on local-aac, whose remainder drops nothing: two receive stalls 14 and 15.3 s in leave audio 159 and 132 ms behind the clock, and the fast accelerate brings it back within half a second. player_skips_per_min and trimmed_ms_per_min, the ring's own count of re-anchors that dropped buffered audio and the media they dropped, are budgeted beside it, at 0 on every recorded row. relay-mic-stall, added 2026-11-10, is relay-mic with nothing arriving from 15 s to 16.5 s and the held frames then trickling in at twice real time until 18 s; today's player trims 2152.5 ms/min of it and its target p95 rises to 1540 ms. Once durations became whole nanoseconds, relay-bbb's delay, 200 ms plus one 1024-sample AAC frame, rounds up by a fraction of a sample, so its ring holds one more sample (23 µs) and its A/V lag p50 measures -22.6 rather than -22.7. Neither replay calibration changed the browser or native ceilings. bursty-aac is the 2026-10-18 AAC bursty isolated capture, recorded with the observer subscribed at the decoder's 2 s budget, so it holds every group the player saw. The native rows are provisional: measured on a loaded developer host (x86_64 Linux, the private PulseAudio null sink, seed 7, 60 s a row) on 2026-10-17, three runs of every row from the debug build nightly uses plus three more of the AAC rows from a release build, which played alike. Each ceiling is the worst clean run times 1.25, plus one native hiccup sized from the unimpaired Opus row's worst run (20 underruns/min, 50 gap ms/min, 2.2 skips/min, 110 discarded ms/min, 0.01 stalled, 0.005 silence, one 20 ms bucket of target), rounded up to two significant figures. On 2026-11-10, after native playback began reporting its A/V lag (#8361), the native rows' player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, and every key of the native and browser congestion rows, were measured by the same rules from three runs of each on a loaded developer host (x86_64 Linux, the debug build, seed 7, 60 s a row), with no void; native audio rests 9 to 95 ms behind its clock at p50. These are provisional until the stretch model's nightly re-measure, which also moves the browser congestion rows onto the runner. On the 2026-10-17 host the null sink's clock ran about 150 ppm slow against the monotonic clock, read within 5 ms at p95 per sample, and its output latency sat between 14 and 28 ms. Five of the 36 AAC rows (none of the 18 Opus rows) voided on the clock: the device stopped pulling for about two seconds mid-row, sometimes after a backend error and a stream restart, which is a native playback defect rather than noise, and the ceilings exclude those runs. Re-measure from the nightly runner's artifacts once it has a week of native rows.", "rows": [ { "runtime": "chromium", @@ -7,11 +7,11 @@ "rate": 44100, "profile": "bursty", "player_skips_per_min": 3.3, - "underruns_per_min": 260, - "underrun_ms_per_min": 710, + "underruns_per_min": 250, + "underrun_ms_per_min": 700, "trimmed_ms_per_min": 280, - "stalled_share": 0.022, - "silence_share": 0.03, + "stalled_share": 0.017, + "silence_share": 0.023, "target_ms_p95": 450, "av_lag_ms_p50": 190, "av_lag_ms_p95": 200 @@ -67,8 +67,8 @@ "rate": 44100, "profile": "step", "player_skips_per_min": 3.3, - "underruns_per_min": 260, - "underrun_ms_per_min": 710, + "underruns_per_min": 250, + "underrun_ms_per_min": 700, "trimmed_ms_per_min": 280, "stalled_share": 0.022, "silence_share": 0.023, @@ -86,8 +86,8 @@ "underrun_ms_per_min": 710, "trimmed_ms_per_min": 280, "stalled_share": 0.033, - "silence_share": 0.034, - "target_ms_p95": 430, + "silence_share": 0.042, + "target_ms_p95": 400, "av_lag_ms_p50": 25, "av_lag_ms_p95": 85 }, From c1a0db06b9795aba8320cec37e5782a9205d4f7d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 13:50:19 -0800 Subject: [PATCH 7/7] test(audio): say the AAC rows keep the provisional ceilings from 11-09 and 11-10 Co-Authored-By: Claude Opus 5.5 --- test/audio-quality/budgets.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/audio-quality/budgets.json b/test/audio-quality/budgets.json index edfcc1f2512..ec8faf57025 100644 --- a/test/audio-quality/budgets.json +++ b/test/audio-quality/budgets.json @@ -1,5 +1,5 @@ { - "note": "The browser rows are measured on the nightly runner (ubuntu-24.04-arm, headless Chromium, seed 7, 60 s a row), from the audio-quality-run artifacts of the seven scheduled Nightly runs on main from 2026-10-29 to 2026-11-04 (40885347517, 41065371094, 41204380716, 41313466339, 41483654426, 41656613559, 41852517544), the first after a fresh subscriber stopped receiving the relay's cached history. The six AAC rows other than congestion are measured the same way from the seven runs from 2026-11-05 to 2026-11-11 (42050683248, 42236107954, 42377867523, 42487375545, 42659882487, 42842827718, 43041909115), the first after the TS importer published each PES as one group; the first four predate the browser's time stretch, and their four provisional keys below are unchanged. Each ceiling is the worst of its seven nights times 1.25, plus one runner hiccup, rounded up to two significant figures. The hiccup is sized from the single gaps the 2026-10-08 night put on otherwise clean rows, the fixed-250 control among them: 250 underruns/min, 700 gap ms/min, 3.3 skips/min, 280 dropped ms/min (both sized on the playhead's skip-ahead inference, and now applied to player_skips_per_min and trimmed_ms_per_min), 0.01 stalled, 0.015 silence, and one 20 ms bucket of target. Budgeted this way, every night passes when it is held out of its row's measurement except one, and that row's own ceiling includes it: the 2026-11-03 Opus bursty row raised its target to 300 ms 2.6 s in and held it for one 20 s half-life before settling at 180. Such startup holds, Opus near-zero's 100 ms on 2026-10-30 and 10-31 among them, are budgeted as measured. These ceilings record today's player as the runner hears it, not target audio quality. To re-measure, download the same artifact from the latest scheduled nightlies (`gh run download -n audio-quality-run`), which keep every run, and apply the same rule. On 2026-11-10 the inferred skip_aheads and discarded_ms keys were dropped, since they read a time stretch's catch-up as dropped audio, and every browser row gained player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, by the same rule from the only two scheduled nights on main with today's browser player and its trim counter, 2026-11-09 and 2026-11-10 (42659882487, 42842827718). A lag is signed, so its ceiling is the worst night plus a quarter of its size plus one 20 ms bucket. These four keys are provisional until seven nights with the stretch model re-measure every row. The grade follows the model: audio rests on the shared clock, trails it through a stall, and catches up after. When a change moves a row, a steady-state lag (av_lag_ms p50) that rises, or audio the player trimmed, is a regression to fix rather than budget; a p95 lag that rises through a stall, or an underrun count that moves, is re-measured in. No ceiling is fixed at zero underruns or zero stalls; a replay row's is exactly what it measured. The replay rows are deterministic, so their ceilings are exactly what they measured, through the end of each recording, with no headroom: any regression fails them, and an improvement should be tightened in. They were last re-measured on 2026-10-22 using version 3 traces with encoded frame durations: AAC-LC uses its fixed 1024 samples, and relay-mic retains Opus packet TOC durations. relay-mic was re-recorded on 2026-11-03 from a real microphone, Chrome 156 on a MacBook Air publishing its built-in microphone through the public relay, recorded by headless Chromium on a Linux host (35 s, minimum RTT 2 ms), which raised its target ceiling from 60 to 80 ms and its silence from 0 to 0.001 as measured. The recording path itself has no added RTT. relay-mic-rtt40 is the 2026-11-03 Chrome row at 40 ms of added RTT on that laptop (180 s, minimum RTT 47 ms), kept as the baseline time stretch improves on; like every replay row, its player_underruns_per_min ceiling is what it measures, not a fixed zero. player_underruns_per_min, the player's own counter that its stats panel shows, is budgeted on the replay rows only; it counts the run-dry re-stalls the tap files as stalled time. On 2026-11-07, after the ordered read stopped delivering groups below its cursor (#8071) and took Rust's hole and age bounds (#8218), relay-bbb was re-measured at 16.1 player underruns/min, 0.025 stalled and 0.02 silence, and relay-mic-rtt40 at 8.2, its discarded audio at 70.9 ms/min and its target p95 at 320 ms rather than 780, since the stale backlog it never played no longer reaches the estimator; neither row lost audio the previous player heard. On 2026-11-08 the player began time-stretching live playback with NetEq's accelerate, deciding on the ring's depth right after each arrival and never expanding, and every replay row was re-measured with it, the browser rows left to the nightly runner. Each row's av_lag_ms, how far the playhead trails the shared clock video plays from, is what it measured: the ring rests one frame ahead of the newest arrival's playout, as the trim it replaces left it, so audio leads the clock by 19 to 23 ms at p50. Against the trim, the A/V lag p50 fell from 0.7, -17.5, -21.5, -18.2 and -15.6 ms to -22.4 on bursty-aac, -22.2 on local-aac, -22.7 on relay-bbb, -19 on relay-mic and -20.1 on relay-mic-rtt40, and its p95 fell on every row, most on relay-bbb (10.5 to -4.2) and relay-mic-rtt40 (-0.9 to -13.4). Underruns stayed where the trim had them, as the latency they would cost is not spent: player_underruns_per_min held at 4.4, 12.1, 16.1 and 6.1 and rose from 8.2 to 10.6 on relay-mic-rtt40, and relay-mic's tap now files 2 short quanta a minute, 2 ms of missing audio, where it filed none. What the stretch buys is the audio the trim dropped: discarded_ms_per_min fell from 87.8 to 0 on relay-bbb and from 127.5 to 100.6 on local-aac, whose remainder drops nothing: two receive stalls 14 and 15.3 s in leave audio 159 and 132 ms behind the clock, and the fast accelerate brings it back within half a second. player_skips_per_min and trimmed_ms_per_min, the ring's own count of re-anchors that dropped buffered audio and the media they dropped, are budgeted beside it, at 0 on every recorded row. relay-mic-stall, added 2026-11-10, is relay-mic with nothing arriving from 15 s to 16.5 s and the held frames then trickling in at twice real time until 18 s; today's player trims 2152.5 ms/min of it and its target p95 rises to 1540 ms. Once durations became whole nanoseconds, relay-bbb's delay, 200 ms plus one 1024-sample AAC frame, rounds up by a fraction of a sample, so its ring holds one more sample (23 µs) and its A/V lag p50 measures -22.6 rather than -22.7. Neither replay calibration changed the browser or native ceilings. bursty-aac is the 2026-10-18 AAC bursty isolated capture, recorded with the observer subscribed at the decoder's 2 s budget, so it holds every group the player saw. The native rows are provisional: measured on a loaded developer host (x86_64 Linux, the private PulseAudio null sink, seed 7, 60 s a row) on 2026-10-17, three runs of every row from the debug build nightly uses plus three more of the AAC rows from a release build, which played alike. Each ceiling is the worst clean run times 1.25, plus one native hiccup sized from the unimpaired Opus row's worst run (20 underruns/min, 50 gap ms/min, 2.2 skips/min, 110 discarded ms/min, 0.01 stalled, 0.005 silence, one 20 ms bucket of target), rounded up to two significant figures. On 2026-11-10, after native playback began reporting its A/V lag (#8361), the native rows' player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, and every key of the native and browser congestion rows, were measured by the same rules from three runs of each on a loaded developer host (x86_64 Linux, the debug build, seed 7, 60 s a row), with no void; native audio rests 9 to 95 ms behind its clock at p50. These are provisional until the stretch model's nightly re-measure, which also moves the browser congestion rows onto the runner. On the 2026-10-17 host the null sink's clock ran about 150 ppm slow against the monotonic clock, read within 5 ms at p95 per sample, and its output latency sat between 14 and 28 ms. Five of the 36 AAC rows (none of the 18 Opus rows) voided on the clock: the device stopped pulling for about two seconds mid-row, sometimes after a backend error and a stream restart, which is a native playback defect rather than noise, and the ceilings exclude those runs. Re-measure from the nightly runner's artifacts once it has a week of native rows.", + "note": "The browser rows are measured on the nightly runner (ubuntu-24.04-arm, headless Chromium, seed 7, 60 s a row), from the audio-quality-run artifacts of the seven scheduled Nightly runs on main from 2026-10-29 to 2026-11-04 (40885347517, 41065371094, 41204380716, 41313466339, 41483654426, 41656613559, 41852517544), the first after a fresh subscriber stopped receiving the relay's cached history. The six AAC rows other than congestion are measured the same way from the seven runs from 2026-11-05 to 2026-11-11 (42050683248, 42236107954, 42377867523, 42487375545, 42659882487, 42842827718, 43041909115), the first after the TS importer published each PES as one group; the first four predate the browser's time stretch, and the six rows keep the four provisional ceilings set below from 2026-11-09 and 2026-11-10. Each ceiling is the worst of its seven nights times 1.25, plus one runner hiccup, rounded up to two significant figures. The hiccup is sized from the single gaps the 2026-10-08 night put on otherwise clean rows, the fixed-250 control among them: 250 underruns/min, 700 gap ms/min, 3.3 skips/min, 280 dropped ms/min (both sized on the playhead's skip-ahead inference, and now applied to player_skips_per_min and trimmed_ms_per_min), 0.01 stalled, 0.015 silence, and one 20 ms bucket of target. Budgeted this way, every night passes when it is held out of its row's measurement except one, and that row's own ceiling includes it: the 2026-11-03 Opus bursty row raised its target to 300 ms 2.6 s in and held it for one 20 s half-life before settling at 180. Such startup holds, Opus near-zero's 100 ms on 2026-10-30 and 10-31 among them, are budgeted as measured. These ceilings record today's player as the runner hears it, not target audio quality. To re-measure, download the same artifact from the latest scheduled nightlies (`gh run download -n audio-quality-run`), which keep every run, and apply the same rule. On 2026-11-10 the inferred skip_aheads and discarded_ms keys were dropped, since they read a time stretch's catch-up as dropped audio, and every browser row gained player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, by the same rule from the only two scheduled nights on main with today's browser player and its trim counter, 2026-11-09 and 2026-11-10 (42659882487, 42842827718). A lag is signed, so its ceiling is the worst night plus a quarter of its size plus one 20 ms bucket. These four keys are provisional until seven nights with the stretch model re-measure every row. The grade follows the model: audio rests on the shared clock, trails it through a stall, and catches up after. When a change moves a row, a steady-state lag (av_lag_ms p50) that rises, or audio the player trimmed, is a regression to fix rather than budget; a p95 lag that rises through a stall, or an underrun count that moves, is re-measured in. No ceiling is fixed at zero underruns or zero stalls; a replay row's is exactly what it measured. The replay rows are deterministic, so their ceilings are exactly what they measured, through the end of each recording, with no headroom: any regression fails them, and an improvement should be tightened in. They were last re-measured on 2026-10-22 using version 3 traces with encoded frame durations: AAC-LC uses its fixed 1024 samples, and relay-mic retains Opus packet TOC durations. relay-mic was re-recorded on 2026-11-03 from a real microphone, Chrome 156 on a MacBook Air publishing its built-in microphone through the public relay, recorded by headless Chromium on a Linux host (35 s, minimum RTT 2 ms), which raised its target ceiling from 60 to 80 ms and its silence from 0 to 0.001 as measured. The recording path itself has no added RTT. relay-mic-rtt40 is the 2026-11-03 Chrome row at 40 ms of added RTT on that laptop (180 s, minimum RTT 47 ms), kept as the baseline time stretch improves on; like every replay row, its player_underruns_per_min ceiling is what it measures, not a fixed zero. player_underruns_per_min, the player's own counter that its stats panel shows, is budgeted on the replay rows only; it counts the run-dry re-stalls the tap files as stalled time. On 2026-11-07, after the ordered read stopped delivering groups below its cursor (#8071) and took Rust's hole and age bounds (#8218), relay-bbb was re-measured at 16.1 player underruns/min, 0.025 stalled and 0.02 silence, and relay-mic-rtt40 at 8.2, its discarded audio at 70.9 ms/min and its target p95 at 320 ms rather than 780, since the stale backlog it never played no longer reaches the estimator; neither row lost audio the previous player heard. On 2026-11-08 the player began time-stretching live playback with NetEq's accelerate, deciding on the ring's depth right after each arrival and never expanding, and every replay row was re-measured with it, the browser rows left to the nightly runner. Each row's av_lag_ms, how far the playhead trails the shared clock video plays from, is what it measured: the ring rests one frame ahead of the newest arrival's playout, as the trim it replaces left it, so audio leads the clock by 19 to 23 ms at p50. Against the trim, the A/V lag p50 fell from 0.7, -17.5, -21.5, -18.2 and -15.6 ms to -22.4 on bursty-aac, -22.2 on local-aac, -22.7 on relay-bbb, -19 on relay-mic and -20.1 on relay-mic-rtt40, and its p95 fell on every row, most on relay-bbb (10.5 to -4.2) and relay-mic-rtt40 (-0.9 to -13.4). Underruns stayed where the trim had them, as the latency they would cost is not spent: player_underruns_per_min held at 4.4, 12.1, 16.1 and 6.1 and rose from 8.2 to 10.6 on relay-mic-rtt40, and relay-mic's tap now files 2 short quanta a minute, 2 ms of missing audio, where it filed none. What the stretch buys is the audio the trim dropped: discarded_ms_per_min fell from 87.8 to 0 on relay-bbb and from 127.5 to 100.6 on local-aac, whose remainder drops nothing: two receive stalls 14 and 15.3 s in leave audio 159 and 132 ms behind the clock, and the fast accelerate brings it back within half a second. player_skips_per_min and trimmed_ms_per_min, the ring's own count of re-anchors that dropped buffered audio and the media they dropped, are budgeted beside it, at 0 on every recorded row. relay-mic-stall, added 2026-11-10, is relay-mic with nothing arriving from 15 s to 16.5 s and the held frames then trickling in at twice real time until 18 s; today's player trims 2152.5 ms/min of it and its target p95 rises to 1540 ms. Once durations became whole nanoseconds, relay-bbb's delay, 200 ms plus one 1024-sample AAC frame, rounds up by a fraction of a sample, so its ring holds one more sample (23 µs) and its A/V lag p50 measures -22.6 rather than -22.7. Neither replay calibration changed the browser or native ceilings. bursty-aac is the 2026-10-18 AAC bursty isolated capture, recorded with the observer subscribed at the decoder's 2 s budget, so it holds every group the player saw. The native rows are provisional: measured on a loaded developer host (x86_64 Linux, the private PulseAudio null sink, seed 7, 60 s a row) on 2026-10-17, three runs of every row from the debug build nightly uses plus three more of the AAC rows from a release build, which played alike. Each ceiling is the worst clean run times 1.25, plus one native hiccup sized from the unimpaired Opus row's worst run (20 underruns/min, 50 gap ms/min, 2.2 skips/min, 110 discarded ms/min, 0.01 stalled, 0.005 silence, one 20 ms bucket of target), rounded up to two significant figures. On 2026-11-10, after native playback began reporting its A/V lag (#8361), the native rows' player_skips_per_min, trimmed_ms_per_min, and av_lag_ms p50 and p95, and every key of the native and browser congestion rows, were measured by the same rules from three runs of each on a loaded developer host (x86_64 Linux, the debug build, seed 7, 60 s a row), with no void; native audio rests 9 to 95 ms behind its clock at p50. These are provisional until the stretch model's nightly re-measure, which also moves the browser congestion rows onto the runner. On the 2026-10-17 host the null sink's clock ran about 150 ppm slow against the monotonic clock, read within 5 ms at p95 per sample, and its output latency sat between 14 and 28 ms. Five of the 36 AAC rows (none of the 18 Opus rows) voided on the clock: the device stopped pulling for about two seconds mid-row, sometimes after a backend error and a stream restart, which is a native playback defect rather than noise, and the ceilings exclude those runs. Re-measure from the nightly runner's artifacts once it has a week of native rows.", "rows": [ { "runtime": "chromium",