Repository navigation
Conversation
- decodeControlMessage normalizes RangeError from truncated frame bodies to MoqtProtocolError, so malformed frames terminate the session on every path instead of being swallowed by the accept loops (the control-stream case previously left a zombie session). - Request timeouts now cancel the request stream before failing the record, so a late SUBSCRIBE_OK can no longer fire onOk after onError and resurrect the alias route. - Pending TRACK_STATUS requests are tracked by the session and failed on transport teardown with the same semantics as subscriptions and fetches (failure callback fired, response timer cleared). - MAX_OBJECT_PROPERTIES_LENGTH raised to 1 MiB: a single legal maximum-size KVP (0xffff value bytes) encodes to 65,539 bytes and could not fit under the old 64 KiB aggregate cap. - De-flaked the REQUEST_OK fetch-settle test (10ms -> 25ms deadline, still inside the 30ms observation window).
- parseChannelConfig no longer defaults unrecognized layouts to stereo: it returns undefined and requires a positive safe integer total <= 255, while still summing dotted layouts like 5.1/7.1.4. The presentation projection substitutes 2 for the required channels field (same pattern as sampleRate). - toAudioDecoderConfig derives numberOfChannels from the raw channelConfig and rejects a declared-but-unresolvable layout instead of silently configuring the decoder for stereo; sample rates must be positive integers in the u32 range before a config is returned (callers already disable the track on null).
- volume: a NaN write is dropped (last valid volume kept) so a non-finite value can never reach the GainNode; infinities clamp. - play()/autoplay right after a src-change or pause() suspend no longer reads stale AudioContext state: the in-flight suspend is tracked and every resume path treats it as suspended and chains behind it. - suspendMediaWhilePaused falls through its hold-window derivation chain when pauseHoldSeconds/targetLatency are not non-negative finite numbers, instead of letting setTimeout coerce NaN/negative delays into an immediate suspend. - resolveCatalog ignores joining-fetch entries after the settle deadline, cancels the fetch handle at the deadline, and guards all fetch/subscription callbacks with a per-attempt token so a superseded attempt (e.g. the auth-expiry retry) cannot settle or overwrite the fresh attempt's state.
A <canvas> defaults to 300x150, so the event bridge's first dimension poll dispatched a resize exposing the HTML default as the stream's dimensions before anything was decoded; native <video> reports 0x0 until metadata. Zero the canvas in the constructor to match the bridge's initial 0/0 trackers. Also rewrote the shadow-root reuse test to genuinely exercise the pre-attached-root path: happy-dom neither parses declarative shadow DOM nor implements customElements.upgrade(), so the test overrides shadowRoot on a throwaway subclass and asserts attachShadow is skipped and the existing canvas is reused.
Gives the URL input an accessible name, matching the label pattern in the spf-background-video template.
The audio renderer anchored its schedule at the oldest frame in the jitter buffer, and the video self-clock at its first decoded frame. A draft-19 relay replays several recent groups to every joining subscriber, so both start seconds behind the live edge and stay there: the latency controller only drains the excess at a 5% rate nudge, or skips to the newest keyframe-led group and plays that whole GOP. Place the first anchor at (newest buffered - target latency) instead, and again after a catch-up skip re-anchors: - Audio discards the backlog unheard (no fast-forward without pitch artifacts) and, when a drop lands, closes the decoder so the jumped-to audio re-anchors rather than butt-joining after a matching stretch of scheduled silence - which would keep the latency the drop just shed. - Video needs nothing new: the backlog is still fed to the decoder in order from its keyframe, and the existing drop-late-at-presentation path discards it against the edge-anchored clock. Only the self-clock (video-only playback, no audio master clock) takes the anchor directly. Both anchors come from the renderer behaviors, which own renderer construction and can see each subscriber's jitter-buffer stats; the target is resolved through the same chain the controller uses, hoisted into media/moq/timeline alongside the rest of the live-edge math. The knob is latency.joinAtEdge, reachable through MoqEngineConfig.latency with the controller's other tunables, and defaults on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRbJbAjmewRV3MxtTpEtSz
Actor-level: the audio renderer discards the pre-anchor backlog unheard and its clock reads the anchor rather than sitting behind ~800ms of scheduled silence; a buffer shallower than the target is left alone; a catch-up skip re-anchors at the edge instead of at the surviving group's start. The video renderer feeds every backlog frame to the decoder in order and drops them late against both an edge-anchored master clock and its own edge-anchored self-clock, and re-anchors at the edge across a catch-up discontinuity. Behavior-level: the anchor each renderer receives is the live edge of its own subscriber's jitter buffer, with the target resolved consumer -> catalog -> default, and no anchor at all when joinAtEdge is off. Engine-level: the in-memory relay replays 30 groups over 3s; playout starts within the target of the newest frame with the knob on and at the oldest replayed group with it off. The canvas attaches after the replay lands so the assertion is about the anchor rather than burst timing, and the controller is parked so its catch-up cannot do the work instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRbJbAjmewRV3MxtTpEtSz
Three coupled fixes to MoQ playout timing. `syncLatency` measured newest-buffered minus oldest-buffered and called it latency. The renderers hold media past the jitter buffer — the video renderer alone keeps `decodeAhead` frames of decoded lookahead — so the reading understated real latency by ~267ms at 30fps, and a shallow buffer that was in fact on target read as "too little latency" and drove the low-side nudge, raising real latency until enough un-decoded backlog reappeared. It now measures the delivery edge against `state.currentTime`, which is what `bufferDepthSeconds`' second parameter already documented itself as wanting. The video self-clock anchored once and extrapolated from `performance.now()` forever, so publisher/consumer clock drift integrated without bound, an anchor placed mid join-burst was held for the life of the stream, and every short pause ratcheted latency up permanently — only a >1s forward jump ever resynced it. It now re-reads the delivery edge on every clock read and slews back onto it, bounded by `clockSlewRate` (50ms/s by default) outside a `clockSlewTolerance` band. The bound is what keeps the clock monotonically forward while being pulled backwards, and it makes the join-burst race self-correcting rather than permanent. The option is renamed `getJoinAnchorUs` → `getTargetClockUs` to match: it is consulted continuously, not once. Publishing `state.currentTime` moves out of `setupAudioRenderer` into `trackPlayoutTime`. Gated on the AudioContext it went silent in exactly the video-only case it exists to cover — and it is now the latency controller's setpoint, so a starved clock is a stopped controller. Also clamps the audio join anchor forward to a running video clock: a deferred-audio unlock creates a fresh subscription whose edge can land behind the video self-clock, which only re-anchors on forward discontinuities and would otherwise freeze until the new master clock caught up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRbJbAjmewRV3MxtTpEtSz
The slew budget accrues with real time since the last evaluation, and the master-clock path returned before ever recording one. An audio track that ended mid-stream would hand presentation back to the self-clock with the whole master-clock interval banked, cashing it in as a single lurch instead of a 50ms/s correction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRbJbAjmewRV3MxtTpEtSz
Introduces `adaptLatencyTarget`, a behavior that proposes a target latency from observed delivery, plus the plumbing it needs — and leaves the existing resolution chain untouched. The precedence rule is one exported helper (`preferredTargetLatencySeconds`): an explicit `state.targetLatency` always wins, `state.adaptiveTargetLatency` fills in behind it, and both absent leaves the catalog → default chain byte-for-byte as it was. With adaptation off (the default) the reactor stays inactive, registers no timer, and writes nothing, so every reader resolves exactly what it resolved before. New observability, because the A/B between the two mechanisms is unreadable without it: - `state.effectiveTargetLatency` — the setpoint `syncLatency` actually resolved. Nothing exposed the resolved target before; every other slot is an input to the resolution. - `state.catchUpSkips`, `state.framesDropped`, `state.audioUnderruns` — the cost counters, published in both arms by the new always-on `trackPlayoutHealth`. The audio underrun counter is new all the way down: `AudioRendererContext.underruns` counts the rising edge of a schedule that ran dry, which is the only direct evidence that a target is below what the path sustains. - `TrackSubscriberContext.arrivalJitter` — a decaying arrival-offset envelope. Its *spread* is the jitter primitive; the bounds forget with a 4s time constant, because an unbounded minimum is what makes an offset-based estimate drift permanently pessimistic. - MSF `jitter` (§5.2.9) now survives catalog parsing. Separation of timescales is validated rather than documented: `resolveAdaptiveLatencyConfig` derives its defaults from the latency config so an untouched adaptive config can never violate the bounds, and throws on an explicit value that does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRbJbAjmewRV3MxtTpEtSz
New coverage, all as siblings of the fixed-setpoint tests rather than edits to them — every assertion pinning the untouched chain still runs verbatim. `adaptLatencyTarget`: inactive and timer-free while off; the warm-up gate falling through to the existing chain; the proposal's composition (floor, observed spread, catalog jitter, clamps); prompt widening on an underrun against four-times-slower narrowing behind the quiet window; the setpoint held still while the inner loop is mid-correction; the last proposal held across a handoff instead of stepping back to the fallback; late-frame drops ignored in ones and acted on in bulk; failure counters baselined at activation. `resolveAdaptiveLatencyConfig`: defaults derived so any latency tuning is safe, explicit values that outrun the inner loops throwing. Elsewhere: the precedence rule itself; the resolved target published for every layer of the chain; catch-up skips counted; the arrival envelope's spread and its decay; the audio underrun rising edge; `trackPlayoutHealth` publishing in both arms; catalog `jitter`; the pause hold window following the target actually being held; the element attribute routing to its own slot beside an explicit `target-latency`; and the engine composing all of it — including refusing a configuration whose loops cannot settle. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRbJbAjmewRV3MxtTpEtSz
a-bro
force-pushed
the
feat/spf-moq-latency-control
branch
from
August 1, 2026 18:12
732081f to
591f5e6
Compare
Author
|
Closing: superseded by the MoQ work on mmcc/moq-publisher and #63 (relayed joins), which our stack now tracks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six commits. The first four are bug fixes to MoQ playout timing; the last two add an
adaptive-latency controller that is off by default.
Why this exists: two real defects
Both were found while trying to hold a stable sub-second latency against a draft-19
relay, and both have observable symptoms rather than being theoretical.
1. The video self-clock free-ran from a single anchor
createVideoRendererActoranchored its presentation clock once — at the firstdecoded frame — and extrapolated from
performance.now()forever after. Only aforward jump greater than 1s ever resynced it.
Three consequences, all of them one-directional:
mechanism at all to pull the clock back onto the delivery edge.
anchor placed mid-join-burst was seconds behind the live edge — and was then
held for the life of the stream. The audio renderer had the same problem from the
other end: it anchored its schedule at the oldest frame in the jitter buffer,
i.e. deliberately at the back of the replay.
Symptom: playback that starts behind the live edge and drifts further behind the
longer it runs. The only recovery paths were the latency controller's 5% rate nudge
(which drains a multi-second excess very slowly) or a catch-up skip to the newest
keyframe-led group, which then plays that entire GOP.
2. The sync controller was measuring the wrong quantity
syncLatencycomputednewest-buffered − oldest-bufferedand called that "latency".That is jitter-buffer depth, not latency — and the renderers hold media past the
jitter buffer. The video renderer alone keeps
decodeAheadframes of decodedlookahead (default 8), which at 30fps is ~267ms sitting entirely outside the
measured window.
So a buffer that was in fact on target read as "too little latency", and the
controller applied its low-side nudge — which raises latency — until enough
un-decoded backlog reappeared to make the reading look right again.
Symptom: latency that is inconsistent between sessions and gets worse over
time, with the controller actively working against the target it was given.
Notably,
bufferDepthSeconds' second parameter already documented itself as wantinga playout timestamp. The call site was passing the oldest buffered frame instead.
The fix
Join at the live edge. Anchor at
(newest buffered − target latency)instead ofat the oldest buffered frame, and re-anchor there after a catch-up skip. Audio
discards the pre-anchor backlog unheard — there is no way to fast-forward audio
without pitch artifacts — and closes the decoder on a drop, so the jumped-to audio
re-anchors rather than butt-joining after a matching stretch of scheduled silence
(which would give back exactly the latency the drop just shed). Video needs nothing
new on the decode path: the backlog is still fed to the decoder in order from its
keyframe and the existing drop-late-at-presentation path discards it against the
edge-anchored clock.
Both anchors come from the renderer behaviors, which own renderer construction and
can see each subscriber's jitter-buffer stats. The knob is
latency.joinAtEdge,reachable through
MoqEngineConfig.latency, and it defaults on.Measure edge-to-playout.
syncLatencynow measures the delivery edge againststate.currentTime— real latency, lookahead included. Publishingstate.currentTimealso moved out of
setupAudioRendererinto a newtrackPlayoutTime: gated on theAudioContext, it went silent in exactly the video-only case it exists to cover, and
it is now the controller's setpoint, so a starved clock was a stopped controller.
Bounded slew. The self-clock re-reads the delivery edge on every clock read and
slews back onto it, bounded by
clockSlewRate(50ms/s default) outside aclockSlewToleranceband. The bound is what keeps the clock monotonically forwardwhile being pulled backwards, and it makes the join-burst race self-correcting rather
than permanent.
getJoinAnchorUsis renamedgetTargetClockUsto match — it isconsulted continuously now, not once.
A fourth commit fixes a related slew bug: the budget accrues with real time since the
last evaluation, and the master-clock path returned before ever recording one. An
audio track that ended mid-stream handed presentation back to the self-clock with the
entire master-clock interval banked, cashing it in as a single lurch instead of a
50ms/s correction.
The adaptive half is opt-in and off by default
adaptLatencyTargetproposes the lowest target the observed path can sustain. It isdisabled unless asked for:
MoqEngineConfig.adaptiveLatency.enabled— defaults tofalse.<simple-moq-video adaptive-latency>— presence-based, exactly like mmcc'sautoplayattribute, and explicitly modelled on it.With it off the reactor stays
inactive: no timer, no slot writes, and every readerresolves precisely the target it resolved before this behavior existed. That is the
control arm of the A/B this is meant to make measurable.
Precedence is one exported helper,
preferredTargetLatencySeconds, giving threemodes:
adaptiveTargetLatencyfills in.
state.targetLatencyalways wins,even with adaptation on.
catalog → default chain byte-for-byte as it was.
Supporting observability came along with it, because the A/B between the fixed and
adaptive mechanisms is unreadable without it:
state.effectiveTargetLatency(thesetpoint actually resolved — nothing exposed this before), the cost counters
catchUpSkips/framesDropped/audioUnderrunspublished by a new always-ontrackPlayoutHealth,TrackSubscriberContext.arrivalJitter(a decayingarrival-offset envelope, whose spread is the jitter primitive), and MSF
jitter(§5.2.9) surviving catalog parsing.
Caveat, stated plainly: the adaptive constants are reasoned, not measured
This matters for review, so it should not be buried. The defect fixes above are
backed by observed behaviour. The adaptive controller's tuning constants are not:
maxWidenRatePerSecond = 0.02s/smaxNarrowRatePerSecond = 0.005s/s (a quarter of the widen rate)widenStep = 0.15s,widenMax = 1s,widenDecayPerSecond = 0.01latency.clockSlewRate / 2(0.025 by default), behind aSchmitt trigger sized off the finest band any inner loop acts on.
These were derived from a separation-of-timescales argument, not from measurement.
The argument is that three negative-feedback loops act on one variable — the
bang-bang
rateNudge, the self-clock's continuous slew, and this behavior's movingsetpoint — and a setpoint that moves faster than the inner loops can follow is a
setpoint they never reach. The bound is validated rather than merely documented:
resolveAdaptiveLatencyConfigderives its defaults from the latency config so anuntouched adaptive config can never violate it, and throws on an explicit value
that does.
But the specific numbers inside that bound are a judgement call.
floorSecondsislikewise a flat constant, not a derivation (hang's precedent is the same shape — a
flat 100ms fallback). A reviewer should treat them as a starting point to be tuned
against real paths, and that is a large part of why the whole thing ships off by
default.
There is also a hard observability limit worth knowing: there is no transport RTT
available. spf speaks its own draft-19 stack (no PROBE — that is a moq-lite
extension) and browsers do not populate
WebTransport.getStats().smoothedRtt, soabsolute one-way delay simply cannot be measured. The estimate is built from arrival
jitter spread, the publisher's declared
jitter, a constant cushion, and failurefeedback.
Relationship to the autoplay work
Integrated, not redundant — this is likely the first question, so: the three
autoplay commits (
23d3ef62,cd36ce9a,0817a879) touch none of the fivefiles this work centres on.
packages/spf/src/playback/behaviors/sync-latency.tspackages/spf/src/playback/actors/dom/audio-renderer.tspackages/spf/src/playback/actors/dom/video-renderer.tspackages/spf/src/playback/behaviors/dom/setup-moq-renderers.tspackages/spf/src/media/moq/timeline.tsWhere the two do meet, the change is purely additive:
packages/spf/src/playback/engines/moq/engine.tspackages/spf/src/playback/engines/moq/adapter.tspackages/html/src/media/simple-moq-video/index.tspackages/html/src/media/simple-moq-video/tests/index.test.tssubscribe-selected-tracks.ts, its tests, andmoq/tests/adapter.test.ts— the restof the autoplay surface — are untouched entirely.
Concretely:
adaptive-latencysits besideautoplayinobservedAttributesand copies itspresence-based semantics deliberately, so the two attributes behave the same way.
accident. A deferred-audio unlock creates a fresh subscription whose edge can land
behind the video self-clock, which only re-anchors on forward discontinuities and
would otherwise freeze until the new master clock caught up — so the audio join
anchor is clamped forward to a running video clock.
suspendMediaWhilePausedand theaudioSuspendedgate both rejoin at the liveedge, which is the join-at-edge path this PR fixes.
Tests
1449 passing, up +39 from the 1410 baseline on
mmcc/moq-sandbox.New coverage was added as siblings of the existing fixed-setpoint tests rather
than as edits to them, so every assertion pinning the untouched chain still runs
verbatim. No autoplay test file loses a line (see the table above). The 28 removed
lines across three spf test files are helper extraction and import rewrites — for
example, the
currentTimefallback test moved describe blocks because publishingcurrentTimemoved fromsetupAudioRenderertotrackPlayoutTime. No assertion wasdeleted or weakened.
New test files:
packages/spf/src/playback/behaviors/tests/adapt-latency-target.test.ts(+410)packages/spf/src/playback/actors/dom/tests/video-renderer.test.ts(+305)packages/spf/src/playback/actors/dom/tests/audio-renderer.test.ts(+134)Verified locally on this branch
pnpm install --frozen-lockfilepnpm lintresolve-catalog.ts, a file this PR does not touch)pnpm build:packagespnpm typecheckpnpm turbo run test --filter="@videojs/spf" --env-mode=loosepnpm check:workspace.claude/plansonly (see below)Known CI failure, pre-existing
The
workspacejob (pnpm check:workspace) will fail with:This is not caused by this PR.
.claude/plans/moq-engine-implementation.mdis atracked file added in
03cdae43and modified by the autoplay commit23d3ef62;check:workspaceexpects.claude/plansto be a generated alias of.agents/plans,and a real tracked directory there breaks that invariant.
pnpm installsays thesame thing at postinstall time:
warning: refusing to replace non-generated .claude/plans.Verified pre-existing: running
pnpm check:workspaceonorigin/mmcc/moq-sandbox(
0817a879) with none of this PR's commits applied produces the identical result —9 passed, 1 failed, same message.
Left alone deliberately — it is the sandbox branch owner's call whether the file
moves to
.agents/plansor the check gets an exemption.Summary by cubic
Fixes MoQ playout timing to hold steady near the live edge and adds an optional adaptive-latency controller (off by default). Also hardens session/protocol handling and validates audio config to make playback safer and more predictable.
Bug Fixes
state.currentTime(not jitter-buffer depth). Video self-clock now slews toward the edge continuously with a bounded rate; prevents drift and join-burst lock-in.trackPlayoutTimeso the controller stays active in video-only cases; fix slew budget carryover when audio ends.MoqtProtocolError; request timeouts cancel request streams so lateSUBSCRIBE_OKcannot resurrect routes; pendingTRACK_STATUSrequests tracked and failed on teardown; raiseMAX_OBJECT_PROPERTIES_LENGTHto 1 MiB.parseChannelConfigrequires a safe, finite total and handles dotted layouts;toAudioDecoderConfigderivesnumberOfChannelsfromchannelConfig, validates positive integersampleRate, and rejects bad tracks.suspendMediaWhilePausedderives its hold window from the effective target (explicit or adaptive) and ignores non-finite/negative inputs; resume paths chain behind an in-flight suspend;resolveCatalogignores joining-fetch entries after the settle deadline and guards callbacks by attempt.simple-moq-video: report 0×0 dimensions before the first frame (canvases default to 300×150); drop NaN volume writes and clamp infinities.New Features
adaptLatencyTargetwith precedence: explicitstate.targetLatency>state.adaptiveTargetLatency> catalog → default. Disabled unlessMoqEngineConfig.adaptiveLatency.enabledorsimple-moq-videosetsadaptive-latency.state.effectiveTargetLatency,catchUpSkips,framesDropped,audioUnderruns,TrackSubscriberContext.arrivalJitter, and catalogjitterparsed and exposed.resolveTargetLatencySecondsandpreferredTargetLatencySecondsunify target resolution across consumer input, adaptive proposal, and catalog.simple-moq-videoacceptsadaptive-latencyattribute, routed to the engine without altering numerictarget-latency.Written for commit 591f5e6. Summary will update on new commits.