Skip to content

fix: resolve the fork-only issues on PR #3 - #57

Merged
fperex merged 125 commits into
debug-findings-solutionfrom
codex/fork-only-unslop
Oct 1, 2026
Merged

fperex merged 125 commits into
debug-findings-solutionfrom
codex/fork-only-unslop

Conversation

@fperex

@fperex fperex commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Problem

PR #3 (1217ea74b) had 24 open fork-only issues (#31 to #53, #56). Its Interop job was red on media reattach: the relay served a returning viewer media from before it left (#51). #55, labelled upstream-code, is the player painting that older media over newer pictures, through fork-only reattach code. On Linux its Audio quality lane underran: the original failed 4 of 36 enforced checks. When this PR was handed over (f21282589), its Interop check was red, its last Nightly (84e98b3a8) failed the Linux Audio quality lane, and the relay cost more CPU than the original at 4 publishers x 10 viewers.

Approach

The issue fixes include targeted regression tests. #34 closes by decision, and #52 with PR #3's rewritten description. No budget, timeout or duration was relaxed.

Merge with a merge commit, not a squash, so each issue can name its fixing commit.

Impact

Public API, all additive:

  • moq_net::track::Producer::break_at(sequence) and @moq/net Track.Producer.breakAt(sequence): a subscription made afterwards starts at sequence at the earliest; a fetch still reaches the groups before it.
  • @moq/hang Container.Legacy.writeMarker(group, end): writes the endpoint marker as the whole group and closes it.
  • moq_tokio::Client::close(): closes the client's QUIC connections once each peer was sent the close (fix(cli): close the relay connection on SIGINT and SIGTERM moq-dev/moq#4287). The NoqClient::close() it calls is crate-private.

The takeover CSP documentation also states that the blob audio worklet requires script-src to allow blob:, including the main-thread audio path.

Wire: no format change. One reset code moves (#46):

  • A JS publisher resets an expired group with Old (0x34) on moq-lite, like Rust, instead of DELIVERY_TIMEOUT (0x2). IETF drafts have no Old, so there it now sends INTERNAL. @moq/net Expired carries 0x34, a peer's 0x34 decodes to it, and a peer's 0x2 stays a plain stream error. The moq-lite draft says so.

Behaviour:

  • moq-mux container::Producer::discontinuity declares the track's break, so a late joiner starts at the marker.
  • @moq/hang Container.Consumer: a stale group dropped behind returned media counts in skipped but raises no discontinuity, so the live GOP is not cut.
  • The relay's io_uring path refuses a new session with 503 while draining, as the tokio listeners do.
  • moq-shaper --report-interval 0s is a usage error.
  • @moq/watch and @moq/hang no longer ship test-only files.

Historical relay CPU and RAM against the original 1217ea74b:

Measure Measured relay heads against original
VPS Linux pairs, 3 rounds, CPU at 1x1 / 1x10 / 4x1 / 4x10 +0.0% / -3.5% / -0.3% / +1.7%; ranges overlap in every cell, zero loss, equal median p99
Mac 4x100, 4 alternating pairs CPU 0.961x, peak RSS 0.948x
VPS soak, 4x10 for 2 h each RSS flat in both; CPU 0.1835 against 0.1866 cores
Criterion time fanout -4.6%, 1-segment resume +0.7%, 2 to 3 segment resume +21%

The VPS relay rows in the table measured 0df56d52a, before the final catalog correction. The Mac and Criterion rows ran the relay branch head 23354da3d, before the publisher break and the finished-cache confirmation correction. The 2 to 3 segment cost exists only while a splice holds several live routes (a takeover); it is the price of moq-dev#4103's drift anchor. Full tables go in the results comment on #3.

Alternatives

Follow-ups

Actual-head long rows, original / df3d2b444, each 480 s:

Browser Whole underruns Concealed ms Maximum frame gap ms Minimum painted FPS Minimum video-input FPS
Chromium 1 / 1 780 / 760 602 / 601.8 8.2 / 7.4 22 / 22.4
Firefox 0 / 0 0 / 0 84 / 83 22.2 / 22.6 22.6 / 22.8
Playwright WebKit 2 / 2 200 / 220 169 / 166 19.9 / 17 22.1 / 21.5

All six raw graders are OK; these retain the nonzero events and low FPS. Source windows differ across browsers; both Chromium windows include the known quiet passage. These are quality observations, without CPU performance attribution. Earlier 0df rows remain separately attributed.
Linux completed 40 rows, a coverage count: one guarded tiles cell interrupted, 11 absent and two older catalog joins failed. The real-Origin regression fixes their shared-cache mechanism; seven once-only a3f1a74f8 rows retained five grader OK and two sampler failures, detailed below. Firefox's 800 ms quiet stretch is valid but source-phase confounded: original missed that phase, and matched original/final replay has identical underrun, skip and concealment counters. It cannot be attributed to this PR.

  • Focused Stall verification: At bc2680217, three Playwright-owned real-task fixtures passed: 0/23 healthy burst heads flagged with and without the probe; actual 150 ms freezes and 660 ms busy tasks flagged. Stall source is byte-identical at a3f1a74f8 and df3d2b444. Six playback measurements retain five oversize-metadata provenance voids and one compact-metadata WebKit main pass, with a later underrun/760 ms concealment. Chromium had 20 hidden samples per row, Firefox/WebKit none; their tab actions did not prove hiding. No void was retried/regraded. These bc268 diagnostics verify neither the catalog fix nor real Safari.
  • Real Safari: Plan18 original/20 actual df3d2b444. Prior31/38 coordinator rows:18 OK/four strict-CSP voids/nine problems; prior32 raws one partial/six untouched, V5raw33/two partials/five untouched. V6 ran those five once:four OK/one problem. Final36 coordinator rows:22 OK/four CSP voids/ten problems;38 raws include two interrupted partials, none untouched. Six worker-only CSP PASS; four script-header refusals,two/build. Six restarts FAIL suspended/time0 despite input/video; resume/activation cause unproved. Both DF memory rows fail all10 recoveries like original r1/r2; contexts/nodes10/1 are not heap proof. DFtilesweep1:three players/expected one, BSD seq 1 0 emitted1,0. DF10 partial SIGTERM:painted0,sampler1,001>1,000ms,missing second input read. V5DF50 guard SIGTERM/driver VOID: designated paint6.9median/4.7minfps,video-input FPS24.1median;5.4s,all50 readiness unknown,user severe lag/possible regression. Before V6 original50 never ran; V6original50 FAIL:paint5.6/0fps,gap6745ms,skew5537.5ms,5underruns/4460ms concealment;45s versus partial5.4s cannot grade a regression. Original/DF480s long OK each1underrun/560ms concealment,gap616ms,paint minima13.5/9.5fps. Raw decoded/decodeFps observes compressed frames before submission, not proof of submission/output; paint measures one designated renderer. Prior lock/V4 refusal retained; no retry/regrade. Checkpoint75/17cells/disjoint21names retained. Other2,713 objects/Cargo.lock match ab075bbb0; metadata DF. AAC13-/16-frame source explanation/receiver ambiguity retained; WebKit separate.

V6 original1/10 passed:designatedpaint24.1fps median,min23.1/22.8,gap67ms,skewp9539.2/33.5ms,zero underruns/concealment;all1/10 input snapshots complete. Original50 failed:paint5.6/0fps,video-input FPS17.8/0,gap6745ms,skew5537.5ms,5underruns/4460ms concealment,13video stalls/3153ms/four rebuilds. All50 input reads complete,28 meet raw input-rate threshold/22 flags including two null,physicalonScreen18; output readiness unknown. Audioadvance40402.2/45000ms,PCMoutput45322.4ms. Original/DF480s long both graderOK:each1under/560ms concealment/gap616ms,paint24median/min13.5/9.5,skew40.8/39.3ms;audioadvance464703/464677 and464736.3/464764ms,PCMoutput465008.6/465023.1ms,two audio-stalled samples each. No recorded listening/amplitude evidence or equivalent completeDF50 window; setup/clock progress is not subjective sound-quality proof. All raw decoded/decodedFps/perTile.decodeFps/decoding fields observe compressed frames before submission (both1217 andDF). A closed-decoder break precedes decode, so these counters prove neither successful submission nor decoded output. Painted counts one designated renderer timestamp,not all50. Raw fields/grades stay unchanged. V6guard clear/no retries; all recorded owned processes and private ports absent after teardown.

The user reported, in consecutive messages, "I noticed video is laggin very very hard", "This feel like a regression", then "The Safari test with many players". The V5 partial retains that adverse experience; it is neither a pass nor proof of a source regression. Metrics refer to selected bbb.hang, not all50: painted6.9fps median/4.7 minimum, video-input FPS24.1 median/22.9 minimum, skewp95=143.9ms; all five measured seconds below15fps. Fifty DOM players/visible-always gates existed, but perTile/decoding/onScreen are null because the final all-tile read failed. Only5.4s/20 samples followed15s warmup; frame log37 paints/243ms maxgap, sampler806ms maxgap, two rebuilds for5s/10s no frame. Twelve CPU samples span23.1s; browser-tree RSS peaks1,391.8MB/system load1 peaks21.97, excluding publisher RSS. AudioContext advances5,364.4ms/5,388ms wall and PCM-output counters5,674.4ms; zero sampled underrun/concealment is not a valid quality pass for an interrupted driver-void. No original50 comparator had run at V5; V6 later measured it once, with a different45s versus5.4s partial window. All recorded owned Safari/WebKit/50publisher/relay/page-server/caffeinate PIDs and private ports were gone in the retained teardown. Those five untouched cells completed once in V6; actualDF50 stayed excluded from repetition. At V5, shipping PR57 was19b/Interop red; separate8509 diagnostic was not a shipping-head replacement.

  • Retained full Audio quality: Successful manual bc2680217 Audio quality passed 172 replay checks/14 rows, 36 enforced browser checks/24 measured rows and a separate 24-check fixed-250 isolated main-thread control/two rows. It also reports eight non-enforced metric breaches/six rows and three non-enforced playout voids; two raw silence exceedances pass complete quiet-source proof. A separate bc268 Nightly fixed-250 shared/main run failed: 12 enforced AAC metric breaches in its full matrix; its separate shared/main control had Opus 6.5 underruns/min, 0.032 stalled share, 6.5 skips/min and 905.7 skipped ms/min (four enforced breaches), plus one AAC playout void. Manual success is not a universal pass.

The a3f1a74f8 Nightly Audio quality job (run 36819237894) passed 172 replay checks/14 rows, 36 enforced worker checks/24 rows and 24 shared/main checks/two rows. It retained 11 non-enforced metric breaches/seven rows and four non-enforced playout voids; one worker Opus and one main AAC raw silence exceedance passed complete quiet-source proof. Its main Nightly job failed the yanked yoke-derive 0.8.3 audit, addressed by the later lock update.

At df3d2b444, manual Audio quality 36822658662 and Nightly Audio quality job 110241399977 both passed 172 replay checks/14 rows, 36 enforced worker checks/24 rows and 24 shared/main checks/two rows. Excluded diagnostics remain: manual 12 metric breaches/eight rows and one playout void; Nightly five breaches/four rows and five voids. Complete quiet-source proof accepted one/two worker raw silence exceedances respectively and one main AAC exceedance in each run. Overall Nightly 36822661339 failed the macOS auth publisher-connect test; only its Audio quality job passed.

At actual 2a78f7425, manual AQ 36831617235 succeeded: 172 replay checks/14 rows, 36 worker/24 rows, 24 shared/main/two rows. Worker exclusions: 23 metric breaches/seven rows, five playout voids; three worker/one main silence exceedances pass complete quiet-source proof. Nightly AQ job 110269314921 failed eight enforced fixed-250 main-control metrics, with zero main voids; replay172/14 and worker36/24 passed. Worker exclusions: 17 breaches/six rows, six voids; three worker/one main quiet-source passes. Both main codecs had 1.1 underruns/min, 0.009 stalled share and 1.1 skips/min; skipped ms/min were AAC90.6, Opus370.9, all against zero ceilings. Overall Nightly 36831620398 failed: 75 successful jobs, one AQ failure, 15 skipped. Failure owner is unknown; production source equality with DF/AB does not relabel this as an AB measurement.

At actual ab075bbb0, manual AQ 36835459388 succeeded: 172 replay checks/14 rows, 36 worker/24 rows and 24 main/two rows. Worker exclusions: 11 metric breaches/five rows and one playout void; six silence exceedances pass complete quiet-source proof. Main has zero breaches/voids and one quiet-source accept. These remain AB measurements, not a99 results.

Actual AB Interop 36835086137/job 110280467248 failed: 24 pairs passed; all eight Python-publisher pairs failed at 30 s. Python connected, announced catalog and produced audio; browser audioBytes19514/audioTS16860 advanced with running worker audio, but videoFrames0/paintedfalse. Native video reads reset Old 0x34 at frame0 and later groups throughout 30 s. Cause is unresolved. All 32 local AB pairs and earlier hosted DF Interop passed; Python source matches DF.

The source/artifact census found no recurring healthy five-second actor in the reviewed app, AQ harness or Playwright code. The starvation initiator remains unknown. Worker/main comparisons differ in feeding thread/session, ring backend and source phase, so a worker PASS does not establish causation. Local movie timing does not prove CI source timing.

ac6461ae2 changes only a_stage_broadcast_picks_the_catalog_to_read: full parser/real TCP/Origin selects literal ["hd"]/["stereo"] at elapsed ZERO (.070 s); both sessions/drivers/server join before clock release. Production BUDGET500/CEILING1500/SETTLE30 ms and other TLS/QUIC tests stay unchanged. A99 broad red: 838 pass/one fail/three skip, 4,127 not run, [] versus ["stereo"]; focused original .103 s PASS, paused-only .099 s PASS without stasis proof, frozen QUIC60.008 s TIMEOUT discarded. Local check/Interop passed; completed AC hosted outcomes appear below. No CPU or product-cause claim.

Actual AB Nightly 36835462958 succeeded: 76 jobs successful/15 skipped. Its AQjob 110281696946 passed 172 replay checks/14 rows, 36 worker/24 rows, 24 main/two rows; worker exclusions10 metric breaches/four rows and four voids, four complete quiet-source accepts; main zero breaches/voids, one quiet-source accept. This remains AB, not AC evidence; AB PR Interop failed separately.

The interrupted VPS diagnostic retained five 540.8–568.7 ms page gaps with zero sampled underruns/concealment. Explicit executed-block markers were lost; Opus controls never ran. The unchanged pressure guard stopped the attempt, so it has no completed quality verdict and does not explain the earlier Nightly starvation.

  • PR feat(watch): add adaptive audio playout and worker offload #3 base conflict: PR feat(watch): add adaptive audio playout and worker offload #3 already conflicts with current main (CONFLICTING/DIRTY) at its pre-merge head 1217ea74b; its old Interop failure remains historical evidence. GitHub does not launch pull_request workflows for a conflicting base. GitHub documents this trigger limit. PR feat(watch): add adaptive audio playout and worker offload #3 keeps its existing target. Verification of the merged debug branch is reported separately and does not verify a merge into current main.
  • Frozen local gates: Earlier df3d2b444 Nix check from 0df56d52a stays green. At ab075bbb0, Nix just check from DF passes, including 339 relay tests/two skipped; just test interop --all passes all 32 pairs across Rust, Python, Go, browser JS, Node, Bun, C and GStreamer.
  • Final-head gates: Eight PR fix: resolve the fork-only issues on PR #3 #57 contexts succeeded. Overall Nightly 36822661339 failed auth_lifetime::an_outage_keeps_the_session_until_expires on macOS: publisher connect timeout; its Audio quality job passed.
    Actual AC PR Test36846958575/job110319278310 FAIL: 4,769 pass/one fail/six skip/318 unrun; Linux dropping_a_server_keeps_its_socket counts one versus two after both drops. Test unchanged from 1217ea74b; observer counts network namespace. Actual initiating owner unproved; no CPU/production-retainer attribution. Other AC hosted runs continue once.

Actual ac6461ae2 manual AQ36847001121 and Nightly36847004709 PASS; Nightly76 successful/15 skipped jobs. Each AQ: replay172/14, worker36/24, main24/two rows; zero enforced breaches/voids. Worker exclusions, manual/Nightly:14 breaches/six rows/four voids/five quiet accepts; nine/seven/four/three. Each main: zero breaches/voids, one quiet-source accept. Nightly has56 release artifacts, no AQ archive. Full raw rows remain in PR #57; no source-cause or19b result follows.

Actual AC Nightly worker raw rows:

row underrun ep/min underrun ms/min skips/min skipped ms/min groups/min silence quiet proven stalled target p95 converge s void
chromium-aac-44100-bursty-isolated 1.1 271.7 0 0 8.7 0.135 0/30 unproven 0.032 1100 59.2 playout
chromium-aac-44100-bursty-plain 2.2 818.3 1.1 251.4 2.2 0.308 0/68 unproven 0.005 660 56 playout
chromium-aac-44100-fixed-250-isolated 0 0 0 0 0 0 0/0 0 250 0 -
chromium-aac-44100-fixed-250-plain 0 0 0 0 0 0.218 48/48 0 250 0 -
chromium-aac-44100-high-rtt-isolated 0 0 4.4 71.4 0 0.317 69/70 unproven 0.005 220 57.7 -
chromium-aac-44100-high-rtt-plain 4.4 1093.9 0 0 2.2 0.236 0/52 unproven 0.014 220 51 playout
chromium-aac-44100-mild-isolated 0 0 0 0 0 0.227 44/50 unproven 0.009 200 13.2 playout
chromium-aac-44100-mild-plain 0 0 0 0 0 0 0/0 0 200 54.8 -
chromium-aac-44100-near-zero-isolated 0 0 0 0 0 0.068 15/15 0 200 49 -
chromium-aac-44100-near-zero-plain 0 0 0 0 0 0.414 91/91 0 180 41.7 -
chromium-aac-44100-step-isolated 0 0 0 0 0 0.132 15/29 unproven 0 240 43.5 -
chromium-aac-44100-step-plain 0 0 0 0 0 0.073 16/16 0 280 51.7 -
chromium-opus-48000-bursty-isolated 0 0 1.1 109.7 0 0.168 35/37 unproven 0.005 460 58 -
chromium-opus-48000-bursty-plain 1.1 272.7 0 0 0 0.295 0/65 unproven 0 1100 59.5 -
chromium-opus-48000-fixed-250-isolated 0 0 0 0 0 0.309 68/68 0 250 0 -
chromium-opus-48000-fixed-250-plain 0 0 0 0 0 0.114 0/25 unproven 0 250 0 -
chromium-opus-48000-high-rtt-isolated 0 0 0 0 0 0.036 0/8 unproven 0 220 56.2 -
chromium-opus-48000-high-rtt-plain 0 0 0 0 0 0.245 53/54 unproven 0 160 50.5 -
chromium-opus-48000-mild-isolated 0 0 0 0 0 0.277 0/61 unproven 0 320 55.5 -
chromium-opus-48000-mild-plain 0 0 0 0 0 0.082 18/18 0 380 46.5 -
chromium-opus-48000-near-zero-isolated 0 0 0 0 0 0.122 27/27 0 100 21.2 -
chromium-opus-48000-near-zero-plain 0 0 0 0 0 0.282 62/62 0 120 35.5 -
chromium-opus-48000-step-isolated 0 0 0 0 0 0.223 49/49 0 320 49.2 -
chromium-opus-48000-step-plain 0 0 0 0 0 0 0/0 0 400 55.3 -

Actual AC Nightly sharedMain raw rows:

row underrun ep/min underrun ms/min skips/min skipped ms/min groups/min silence quiet proven stalled target p95 converge s void
chromium-aac-44100-fixed-250-isolated 0 0 0 0 0 0.268 59/59 0 250 0 -
chromium-opus-48000-fixed-250-isolated 0 0 0 0 0 0.114 25/25 0 250 0 -

Actual AC worker raw rows:

row underrun ep/min underrun ms/min skips/min skipped ms/min groups/min silence quiet proven stalled target p95 converge s void
chromium-aac-44100-bursty-isolated 0 0 1.1 143.2 1.1 0.155 24/34 unproven 0.027 1160 56.3 playout
chromium-aac-44100-bursty-plain 0 0 2.2 59.9 1.1 0.312 0/69 unproven 0.005 640 60.2 playout
chromium-aac-44100-fixed-250-isolated 0 0 0 0 0 0.041 9/9 0 250 0 -
chromium-aac-44100-fixed-250-plain 0 0 0 0 0 0.294 65/65 0 250 0 -
chromium-aac-44100-high-rtt-isolated 0 0 0 0 0 0.309 67/68 unproven 0 260 54.5 -
chromium-aac-44100-high-rtt-plain 1.1 274.9 0 0 0 0.195 43/43 0 260 58.5 -
chromium-aac-44100-mild-isolated 2.2 546.5 0 0 0 0.227 46/50 unproven 0 220 51.2 -
chromium-aac-44100-mild-plain 0 0 0 0 0 0 0/0 0 180 35.5 -
chromium-aac-44100-near-zero-isolated 4.4 1910.2 7.6 209.5 0 0.164 13/36 unproven 0.159 180 42.5 playout
chromium-aac-44100-near-zero-plain 0 0 0 0 0 0.391 86/86 0 180 5.2 -
chromium-aac-44100-step-isolated 0 0 1.1 9.1 0 0.123 16/27 unproven 0 400 57.7 -
chromium-aac-44100-step-plain 0 0 0 0 0 0.086 19/19 0 320 43 -
chromium-opus-48000-bursty-isolated 0 0 0 0 0 0.159 35/35 0 500 58.5 -
chromium-opus-48000-bursty-plain 0 0 1.1 120.5 1.1 0.332 66/73 unproven 0.009 700 58 playout
chromium-opus-48000-fixed-250-isolated 0 0 0 0 0 0.359 79/79 0 250 0 -
chromium-opus-48000-fixed-250-plain 0 0 0 0 0 0.118 26/26 0 250 0 -
chromium-opus-48000-high-rtt-isolated 1.1 272.7 1.1 152.7 0 0.045 0/10 unproven 0.005 260 59.7 -
chromium-opus-48000-high-rtt-plain 0 0 0 0 0 0.25 55/55 0 180 38.2 -
chromium-opus-48000-mild-isolated 0 0 0 0 0 0.253 52/56 unproven 0 240 36 -
chromium-opus-48000-mild-plain 0 0 2.2 175.1 0 0.086 17/19 unproven 0 400 58.3 -
chromium-opus-48000-near-zero-isolated 0 0 0 0 0 0.109 24/24 0 100 22 -
chromium-opus-48000-near-zero-plain 0 0 0 0 0 0.266 59/59 0 60 0 -
chromium-opus-48000-step-isolated 0 0 1.1 98.3 0 0.205 45/45 0 240 35 -
chromium-opus-48000-step-plain 0 0 0 0 0 0 0/0 0 360 55.3 -

Actual AC sharedMain raw rows:

row underrun ep/min underrun ms/min skips/min skipped ms/min groups/min silence quiet proven stalled target p95 converge s void
chromium-aac-44100-fixed-250-isolated 0 0 0 0 0 0.25 55/55 0 250 0 -
chromium-opus-48000-fixed-250-isolated 0 0 0 0 0 0.123 27/27 0 250 0 -

19bcd9d63 changes only the Linux worker test, +66/-26 lines. It snapshots process-owned socket inode IDs without opening/duplicating a socket: exactly two after split, the same IDs after each unused member drop, none after shutdown. A plain bind must return AddrInUse while alive and succeed after shutdown. The single-test child isolates parallel libtest siblings; its fresh literal receipt after every assertion prevents a zero-test filter passing. Procfs errors fail except disappeared unrelated FDs; unrelated non-UTF8 file targets are ignored. Docker barrier: foreign same-port exit changes namespace count 3 to 2 while owned IDs stay stable; deliberate owned closure gives 1 then 0 and both bind controls pass. This proves the old observer attribution defect, not the initiating owner of AC's historical 2-to-1 failure. No CPU/production-retainer attribution or production/API/wire change. Actual Linux Test36853755318/job110341245926 passed:5,088 pass/six skipped78.659s; socket test0.052s PASS. Mac skips this fixture. PR Interop remains failed below.

  • Exact AC gates: ac6461ae2 is pushed. Local: 4,966 pass/three skip + 84 pass/zero skip; JS/build/lint; Interop 32/32 PASS. Seven PR checks pass; Test fails above. Manual AQ36847001121 PASS; Nightly36847004709 PASS:76 success/15 skip. Earlier 2a/DF Nightly and AB Interop failures stay retained; AB manual AQ and whole Nightly succeeded.
    Actual 19bcd9d63 PR Interop36853755161/job110341245092 FAIL: all 32 publisher/subscriber pairs and worker media lifecycle40s passed. Main-thread lifecycle failed at14s: after resume,67/81 samples carried the fixture tone above the noise floor. Four negative controls passed: frozen video6s, silent audio6s, offset audio7s, leaked session12s. This differs from AB's eight Python-publisher pair failures. The initiating owner is unresolved; local32-pair PASS remains separate. That 19bcd9d63 candidate was not approved for merge.

  • 19b historical gates: 19bcd9d63 is pushed; local 4,966 pass/three skip + 84 pass/zero skip, full JS/build/lint and Interop32/32 PASS. Linux Test36853755318/110341245926 PASS:5,088 pass/6 skip78.659s; socket0.052s PASS; Mac skips it.

  • Candidate gates: 09a783a78 is pushed; Local 4,966 passed/three skipped (72.071s) + 84 passed/zero skipped (4.513s); JS/build/lint; 32/32 Interop PASS. Eight actual PR checks SUCCESS.

Actual 09a783a78 hosted results: Manual 36874501140 SUCCESS (2 jobs: 1 skipped/1 success); replay 172/14 SUCCESS; worker 36/24 SUCCESS, reported 10 metrics/6 rows/3 voids/3 quiet accepts; main 24/2 SUCCESS, reported 0 metrics/0 rows/0 voids/1 quiet accepts. Nightly 36874506339 SUCCESS (91 jobs: 15 skipped/76 success); replay 172/14 SUCCESS; worker 36/24 SUCCESS, reported 11 metrics/9 rows/4 voids/0 quiet accepts; main 24/2 SUCCESS, reported 0 metrics/0 rows/0 voids/1 quiet accepts.

09a783a78: three harness files (+105/-2) surface browser-close errors after every close attempt. Swallow mutant:1 PASS/2 FAIL; fix:3 tests/6 assertions PASS, package6 PASS. Prior midwindow owner remains unknown. No playback/API/wire/budget change. DF differs only in12 test/doc/harness objects;2,705 others/eight AQ inputs match. Binaries stay DF; no compiled-binary/CPU equivalence.

Actual 19bcd9d63 hosted results: Manual36853751819 PASS: replay172/14, worker36/24, main24/2; worker11 excluded breaches/6 rows/4 voids/no quiet allowance. Nightly36853756326 FAIL:75 success/1 AQ fail/15 skip; replay172/14, main24/2 PASS; worker3 enforced fixed250/plain AAC breaches:1.1 underruns/min,1.1 episodes/min,220.5 underrun ms/min, zero ceilings;26 excluded/10 rows/2 voids. Each main0/0+1 quiet. Nightly timing stays outside merge gates; 19b Interop failed. No failure-owner attribution follows. The worker failure is not36 passed checks. The failure artifact retains24 traces with completed page.close, including both void rows; trace preservation does not make their playback valid.

Actual19b manualAQ worker raw rows:

row underrun ep/min underrun ms/min skips/min skipped ms/min groups/min silence quiet proven stalled target p95 converge s void
chromium-aac-44100-bursty-isolated 0 0 1.1 13.7 0 0.086 18/19 unproven 0 740 59.7 -
chromium-aac-44100-bursty-plain 1.1 267.5 2.2 26.9 7.6 0.281 0/62 unproven 0.023 340 45.5 playout
chromium-aac-44100-fixed-250-isolated 0 0 0 0 0 0 0/0 0 250 0 -
chromium-aac-44100-fixed-250-plain 0 0 0 0 0 0.191 42/42 0 250 0 -
chromium-aac-44100-high-rtt-isolated 0 0 0 0 2.2 0.303 0/67 unproven 0.009 240 50.8 playout
chromium-aac-44100-high-rtt-plain 0 0 0 0 0 0.222 0/49 unproven 0.014 240 55.5 playout
chromium-aac-44100-mild-isolated 0 0 0 0 0 0.227 32/50 unproven 0.014 180 35 playout
chromium-aac-44100-mild-plain 0 0 0 0 0 0 0/0 0 180 59.3 -
chromium-aac-44100-near-zero-isolated 0 0 0 0 0 0.072 16/16 0 200 6.8 -
chromium-aac-44100-near-zero-plain 0 0 0 0 0 0.409 89/90 unproven 0 180 56.5 -
chromium-aac-44100-step-isolated 0 0 0 0 0 0.14 12/31 unproven 0 280 51.5 -
chromium-aac-44100-step-plain 0 0 0 0 0 0.073 15/16 unproven 0 360 53.2 -
chromium-opus-48000-bursty-isolated 1.1 272.5 3.3 434.5 0 0.176 38/39 unproven 0.005 460 58 -
chromium-opus-48000-bursty-plain 0 0 0 0 0 0.295 63/65 unproven 0 540 57 -
chromium-opus-48000-fixed-250-isolated 0 0 0 0 0 0.306 67/67 0 250 0 -
chromium-opus-48000-fixed-250-plain 0 0 0 0 0 0.127 28/28 0 250 0 -
chromium-opus-48000-high-rtt-isolated 0 0 0 0 0 0.041 0/9 unproven 0 220 37 -
chromium-opus-48000-high-rtt-plain 1.1 274.4 0 0 0 0.214 46/47 unproven 0 260 50 -
chromium-opus-48000-mild-isolated 0 0 0 0 0 0.305 57/67 unproven 0 360 56 -
chromium-opus-48000-mild-plain 0 0 0 0 0 0.091 18/20 unproven 0 320 47.7 -
chromium-opus-48000-near-zero-isolated 0 0 0 0 0 0.113 25/25 0 100 0 -
chromium-opus-48000-near-zero-plain 0 0 0 0 0 0.249 55/55 0 60 0 -
chromium-opus-48000-step-isolated 0 0 0 0 0 0.236 48/52 unproven 0 540 49.5 -
chromium-opus-48000-step-plain 0 0 1.1 87.9 0 0 0/0 0 440 56.7 -

Actual19b manualAQ sharedMain raw rows:

row underrun ep/min underrun ms/min skips/min skipped ms/min groups/min silence quiet proven stalled target p95 converge s void
chromium-aac-44100-fixed-250-isolated 0 0 0 0 0 0.258 57/57 0 250 0 -
chromium-opus-48000-fixed-250-isolated 0 0 0 0 0 0.123 27/27 0 250 0 -

Actual19b manualAQ recorded-only exclusions:

  • chromium-aac-44100-near-zero-plain converge_s_seconds: 56.5 > 3
  • chromium-aac-44100-near-zero-plain silence_share_share: 0.409 > 0.075; 89/90 quiet windows over quiet source (log RMS r 1.00000, level 1.000, 115 audible windows); 1 of 90 quiet windows unproven, first at 20.00 s: audible source
  • chromium-aac-44100-step-isolated silence_share_share: 0.14 > 0.061; 12/31 quiet windows over quiet source (log RMS r 1.00000, level 1.000, 131 audible windows, 2 analyser lag segments); 19 of 31 quiet windows unproven, first at 5.75 s: alignment: 37 audible windows prove its lag, fewer than 40
  • chromium-opus-48000-bursty-isolated underrun_episodes_per_min: 1.1 > 0
  • chromium-opus-48000-bursty-isolated underrun_samples_per_min: 272.5 > 0
  • chromium-opus-48000-bursty-isolated silence_share_share: 0.176 > 0.15; 38/39 quiet windows over quiet source (log RMS r 1.00000, level 1.000, 101 audible windows); 1 of 39 quiet windows unproven, first at 59.25 s: discontinuity
  • chromium-opus-48000-high-rtt-isolated silence_share_share: 0.041 > 0.007; 0/9 quiet windows over quiet source (log RMS r 0.99964, level 1.000, 93 audible windows, 3 analyser lag segments); 9 of 9 quiet windows unproven, first at 45.25 s: lag change
  • chromium-opus-48000-high-rtt-plain underrun_episodes_per_min: 1.1 > 0
  • chromium-opus-48000-high-rtt-plain underrun_samples_per_min: 274.4 > 0
  • chromium-opus-48000-high-rtt-plain silence_share_share: 0.214 > 0.102; 46/47 quiet windows over quiet source (log RMS r 0.99973, level 1.000, 120 audible windows); 1 of 47 quiet windows unproven, first at 21.00 s: audible source
  • chromium-opus-48000-step-isolated silence_share_share: 0.236 > 0.088; 48/52 quiet windows over quiet source (log RMS r 1.00000, level 1.000, 106 audible windows); 4 of 52 quiet windows unproven, first at 36.26 s: audible source
  • chromium-aac-44100-bursty-plain: playout (the audio graph, timeline, or counters changed during the measured window)
  • chromium-aac-44100-high-rtt-isolated: playout (the audio graph, timeline, or counters changed during the measured window)
  • chromium-aac-44100-high-rtt-plain: playout (the audio graph, timeline, or counters changed during the measured window)
  • chromium-aac-44100-mild-isolated: playout (the audio graph, timeline, or counters changed during the measured window)

Actual19b Nightly worker raw rows:

row underrun ep/min underrun ms/min skips/min skipped ms/min groups/min silence quiet proven stalled target p95 converge s void
chromium-aac-44100-bursty-isolated 1.1 272.8 0 0 17.5 0.1 0/22 unproven 0.041 520 58.2 playout
chromium-aac-44100-bursty-plain 1.1 272.6 2.2 38.5 0 0.25 53/55 unproven 0.005 440 58.5 -
chromium-aac-44100-fixed-250-isolated 0 0 0 0 0 0 0/0 0 250 0 -
chromium-aac-44100-fixed-250-plain 1.1 220.5 0 0 0 0.072 0/16 unproven 0 250 0 -
chromium-aac-44100-high-rtt-isolated 0 0 1.1 133.5 0 0.255 52/56 unproven 0.005 440 57.7 -
chromium-aac-44100-high-rtt-plain 2.2 476.6 0 0 0 0.027 6/6 0 220 55.2 -
chromium-aac-44100-mild-isolated 2.2 376.4 0 0 0 0.145 28/32 unproven 0 220 26.2 -
chromium-aac-44100-mild-plain 0 0 0 0 0 0 0/0 0 200 28.2 -
chromium-aac-44100-near-zero-isolated 2.2 469.9 0 0 0 0.054 10/12 unproven 0 180 3 -
chromium-aac-44100-near-zero-plain 2.2 478.5 0 0 0 0.344 74/76 unproven 0 180 50.3 -
chromium-aac-44100-step-isolated 0 0 0 0 0 0.345 73/76 unproven 0 400 38.2 -
chromium-aac-44100-step-plain 1.1 178.5 3.3 98.9 0 0.09 16/20 unproven 0 440 55.5 -
chromium-opus-48000-bursty-isolated 1.1 272.7 1.1 87.8 0 0.177 36/39 unproven 0 360 56.2 -
chromium-opus-48000-bursty-plain 0 0 0 0 0 0.327 71/72 unproven 0 420 57.2 -
chromium-opus-48000-fixed-250-isolated 0 0 0 0 0 0.259 57/57 0 250 0 -
chromium-opus-48000-fixed-250-plain 0 0 0 0 0 0.072 15/16 unproven 0 250 0 -
chromium-opus-48000-high-rtt-isolated 0 0 0 0 0 0.045 8/10 unproven 0.005 300 59 -
chromium-opus-48000-high-rtt-plain 0 0 0 0 0 0.154 33/34 unproven 0 280 57 clock
chromium-opus-48000-mild-isolated 0 0 0 0 0 0.277 0/61 unproven 0 160 23.2 -
chromium-opus-48000-mild-plain 0 0 1.1 87.4 0 0.1 22/22 0 220 59.2 -
chromium-opus-48000-near-zero-isolated 0 0 0 0 0 0.114 25/25 0 100 50.2 -
chromium-opus-48000-near-zero-plain 0 0 0 0 0 0.255 56/56 0 100 18.7 -
chromium-opus-48000-step-isolated 0 0 5.5 2902.2 0 0.23 45/48 unproven 0.005 440 53.2 -
chromium-opus-48000-step-plain 0 0 0 0 0 0 0/0 0 420 57 -

Actual19b Nightly sharedMain raw rows:

row underrun ep/min underrun ms/min skips/min skipped ms/min groups/min silence quiet proven stalled target p95 converge s void
chromium-aac-44100-fixed-250-isolated 0 0 0 0 0 0.241 53/53 0 250 0 -
chromium-opus-48000-fixed-250-isolated 0 0 0 0 0 0.118 26/26 0 250 0 -

Actual19b Nightly recorded-only exclusions:

  • chromium-aac-44100-bursty-plain silence_share_share: 0.25 > 0.075; 53/55 quiet windows over quiet source (log RMS r 1.00000, level 1.000, 65 audible windows); 2 of 55 quiet windows unproven, first at 42.75 s: audible source

  • chromium-aac-44100-high-rtt-isolated skipped_samples_per_min: 133.5 > 85.9

  • chromium-aac-44100-high-rtt-isolated target_ms_p95: 440 > 330

  • chromium-aac-44100-high-rtt-isolated converge_s_seconds: 57.7 > 11.6

  • chromium-aac-44100-high-rtt-plain underruns_per_min: 2.2 > 0

  • chromium-aac-44100-high-rtt-plain underrun_episodes_per_min: 2.2 > 0

  • chromium-aac-44100-high-rtt-plain underrun_samples_per_min: 476.6 > 0

  • chromium-aac-44100-mild-isolated underruns_per_min: 2.2 > 1.7

  • chromium-aac-44100-mild-isolated underrun_episodes_per_min: 2.2 > 0

  • chromium-aac-44100-mild-isolated underrun_samples_per_min: 376.4 > 0

  • chromium-aac-44100-near-zero-isolated underruns_per_min: 2.2 > 0

  • chromium-aac-44100-near-zero-isolated underrun_episodes_per_min: 2.2 > 0

  • chromium-aac-44100-near-zero-isolated underrun_samples_per_min: 469.9 > 0

  • chromium-aac-44100-near-zero-plain underruns_per_min: 2.2 > 0

  • chromium-aac-44100-near-zero-plain underrun_episodes_per_min: 2.2 > 0

  • chromium-aac-44100-near-zero-plain underrun_samples_per_min: 478.5 > 0

  • chromium-aac-44100-near-zero-plain converge_s_seconds: 50.3 > 3

  • chromium-aac-44100-near-zero-plain silence_share_share: 0.344 > 0.075; 74/76 quiet windows over quiet source (log RMS r 0.99994, level 1.000, 110 audible windows); 2 of 76 quiet windows unproven, first at 27.27 s: discontinuity

  • chromium-aac-44100-step-isolated silence_share_share: 0.345 > 0.061; 73/76 quiet windows over quiet source (log RMS r 0.99995, level 1.000, 80 audible windows, 2 analyser lag segments); 3 of 76 quiet windows unproven, first at 23.25 s: audible source

  • chromium-opus-48000-bursty-isolated underrun_episodes_per_min: 1.1 > 0

  • chromium-opus-48000-bursty-isolated underrun_samples_per_min: 272.7 > 0

  • chromium-opus-48000-bursty-isolated silence_share_share: 0.177 > 0.15; 36/39 quiet windows over quiet source (log RMS r 0.99995, level 1.000, 117 audible windows); 3 of 39 quiet windows unproven, first at 39.50 s: audible source

  • chromium-opus-48000-high-rtt-isolated silence_share_share: 0.045 > 0.007; 8/10 quiet windows over quiet source (log RMS r 0.99923, level 1.000, 145 audible windows); 2 of 10 quiet windows unproven, first at 9.60 s: discontinuity

  • chromium-opus-48000-step-isolated skip_aheads_per_min: 5.5 > 4.9

  • chromium-opus-48000-step-isolated skipped_samples_per_min: 2902.2 > 419.4

  • chromium-opus-48000-step-isolated silence_share_share: 0.23 > 0.088; 45/48 quiet windows over quiet source (log RMS r 1.00000, level 1.000, 81 audible windows); 3 of 48 quiet windows unproven, first at 21.16 s: discontinuity

  • chromium-aac-44100-bursty-isolated: playout (the audio graph, timeline, or counters changed during the measured window)

  • chromium-opus-48000-high-rtt-plain: clock (AudioContext.currentTime drifted 3.06% from wall clock over 10s)

  • Catalog browser follow-up at a3f1a74f8: Seven once-only Linux rows used a3f1a74f8: five grader OK; two retained sampler failures. Chromium reattach r1 had a 2,078.6 ms gap after attach, one whole-window underrun/1,160 ms concealed; r2 had a 1,748.8 ms gap in warmup, zero underruns/concealment. Other whole-window counts were r3 reattach 2/120 ms, r3 hidden 1/120 ms, Firefox hidden 2/40 ms and r1/r2 hidden 0/0. Matched r2/r3 hidden catalog-to-media was 61.2/64.4 ms and click-to-ready 145/179 ms; Firefox was 22/1,341 ms. No Lagged or no-route warning appears in the relay log. These warm joins do not force the internal two-front overlap; the native real-Origin test does. Production Rust/JS matches df3d2b444; a3 used the prior lock. ab075bbb0 adds only auth-test changes; ac6461ae2 adds only trace/docs and completion-test changes; production remains unchanged. These native binaries remain attributed to a3f1a74f8, with its prior dependency lock.

Drafted by Claude Opus 5.5. Edited and posted by Codex.

(Written by GPT-6)

fperex and others added 30 commits September 28, 2026 16:03
Cover every Bun workspace with the scope regression check. Fixes #48.

Co-Authored-By: Codex <noreply@openai.com>
Exercise real replay output paths and probe sampling, and test large recording floors. Each regression fails when its corresponding safeguard is reverted. Covers the harness portion of #50.

Co-Authored-By: Codex <noreply@openai.com>
Reject zero report intervals at argument parsing. Count datagrams by active profile without adding a lock or an extra atomic update to forwarding. Cover stepped impairments and benchmark clients against step counts in nightly CI. Fixes #47.

Co-Authored-By: Codex <noreply@openai.com>
Refs #31.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex <noreply@openai.com>
Preserve decoded audio across bounded timeline holes. Commit short fragments before the splice instead of discarding them, and drain finite tracks without waiting for an impossible refill.

Refs #32 and #33.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex <noreply@openai.com>
Refs #39.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex <noreply@openai.com>
Refs #45. Verified raw QUIC and WebTransport admission against Linux io_uring workers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex <noreply@openai.com>
Fixes #31 for the JavaScript playout engine.

Co-Authored-By: Codex <noreply@openai.com>
Fixes #43.

Co-Authored-By: Codex <noreply@openai.com>
Fixes #40.

Co-Authored-By: Codex <noreply@openai.com>
Bound worker startup including lazy imports, terminate late workers, and preserve the refusal for later players. Fixes #42.

Co-Authored-By: Codex <noreply@openai.com>
Fixes #39E.

Co-Authored-By: Codex <noreply@openai.com>
Keep the first playback report and errors on the page while the worker receives ongoing ring state. Fixes #39D.

Co-Authored-By: Codex <noreply@openai.com>
Fixes #39A.

Co-Authored-By: Codex <noreply@openai.com>
Reject incomplete device timestamps, map shared rings to output time, and sample again on worker progress after resume. Fixes #37, #41, #56, and #39B.

Co-Authored-By: Codex <noreply@openai.com>
Distinguish codec recovery from the stall watchdog and remove load-sensitive deadlines. Fixes the watch portion of #53.

Co-Authored-By: Codex <noreply@openai.com>
Keep preview timestamps out of promotion and preserve healthy sparse subscriptions. Reset learned cadence when the source changes. Fixes #36 and #38A.

Co-Authored-By: Codex <noreply@openai.com>
Fixes #38B.

Co-Authored-By: Codex <noreply@openai.com>
Fail an interrupted UDP benchmark batch instead of waiting until the nightly job timeout.

Co-Authored-By: Codex <noreply@openai.com>
Start guarded writes only while the group is live. Release partial subscriptions when setup fails, and keep rejected signal listeners out of the subscriber set. Preserve lite priority updates through FIN acknowledgment without expiry listeners.

Map local expiry and remote OLD resets to Expired, matching Rust and allowing JSON window consumers to resume at checkpoints. DELIVERY_TIMEOUT remains a distinct stream error.

Co-Authored-By: Codex <noreply@openai.com>
Apply positive age budgets to completed groups when the reader stops pulling. Preserve zero-budget burst behavior. Report covered marker groups as discontinuities and avoid mistaking sparse arrivals for event-loop stalls under timer throttling.

Pin arrival measurement before parsing and preserve a long GOP after a shorter completed group.

Co-Authored-By: Codex <noreply@openai.com>
Keep new frames readable when a returning frame callback predates the last worker tick by no more than one frame period. Preserve rejection of larger clock reversals and make the suspended-callback grace test use a controlled clock.

Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Extend #40 coverage to detach and reattach with a different codec configuration.

Co-Authored-By: Codex <noreply@openai.com>
…s it (moq-dev#4104)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…dev#4103)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Grok 4.7 <noreply@x.ai>
An IETF live join delivers nothing below the group its SUBSCRIBE_OK names as
Largest, so declare that as the copy's start. A parked track's warm cache
waiting on the copy then sees the upstream moved past it and live readers
skip it, as they already do on lite-06+.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Use the shared shutdown signal in the player and send the QUIC close after the window exits. Exercise both process signals with real signals in isolated nextest processes.

Refs #51.

Co-Authored-By: Codex <noreply@openai.com>
Continue splicing across holes when an endpoint or finite track drains the buffer, so the playhead still reaches the final media timestamp.

Refs #32 and #33.

Co-Authored-By: Codex <noreply@openai.com>
@fperex fperex changed the title fix: address fork-only playback and transport issues fix: resolve the fork-only issues on PR #3 Oct 1, 2026
@fperex
fperex marked this pull request as ready for review October 1, 2026 16:16
@fperex
fperex merged commit 9b2d051 into debug-findings-solution Oct 1, 2026
104 checks passed
This was referenced Oct 1, 2026
@fperex
fperex deleted the codex/fork-only-unslop branch October 2, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants