Repository navigation
fix(spf): serve relay pulls whose fill cannot be reset reliably - #63
Conversation
Moq-relay (0.14.17 through 0.17.0) subscribes upstream with Next Object plus FILL_PARAMETERS for the current group. Once a track has content that fill range is nonempty, and the publish session refused the whole SUBSCRIBE with NOT_SUPPORTED when the transport's writers lack WebTransport's commit() (RESET_STREAM_AT). Chromium 147 has no commit(), so in the browser a relay's second pull of a live track always failed: the real subscribe after a cross-relay TRACK_INFO probe, or any re-subscribe after the last viewer left. Players on another relay never got a catalog. Serve the subscription, and fail the fill as before: write the FETCH_HEADER, commit it where the writer can, and reset. Without commit() the reset may discard the header, and the subscriber sees no fill, which leaves its join at the next group boundary, as a failed fill does. REQUEST_UPDATE fills get the same treatment instead of ending the subscription. Stacked on mmcc#62, which ends tracks with PUBLISH_DONE and an exact Stream Count. There the fill counted only past the commit() gate this change removes. It now counts wherever the session opens one, whether or not its header survives the reset: §10.12 counts the streams the publisher opened, fill fetch streams included. A subscriber waits out a timeout for a stream reset before its header, but one that receives more streams than the count may close the session. moq-relay 0.17.0 never closes on either: it ends the track once the counted streams and the fill have arrived, or after a 1 s grace. A lost fill header leaves its fill outstanding, so that track end waits out the grace whatever the count says. The relay hub can now model a transport without commit(). The session tests cover the relay's probe-then-subscribe sequence, and the Stream Count of a fill reset without commit(), with its header delivered and lost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ts6nhCDi2pBF9h5ooBNbss
f883aba to
f1067b3
Compare
There was a problem hiding this comment.
2 issues found across 5 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/spf/src/publish/session/tests/publish-session.test.ts">
<violation number="1" location="packages/spf/src/publish/session/tests/publish-session.test.ts:1370">
P3: `bindings.at(-1) === 237` races with the probe's unbind: the probe subscription's `onTrackBinding(trackAlias: undefined)` and pull 237's bind are processed in independent handlers, so the unbind can land after the bind, leaving `at(-1)` undefined and timing the waitFor out. Wait for the probe's unbind before opening the second pull.</violation>
</file>
<file name="packages/spf/src/publish/session/publish-session.ts">
<violation number="1" location="packages/spf/src/publish/session/publish-session.ts:670">
P2: The fill Stream Count snapshot can undercount a fill that is still opening when the track ends. `#finTrack` reads `subscriber.fillStreams` and sends PUBLISH_DONE immediately, but `#openAndResetFill` increments that counter only after `await this.openUniStream()` resumes. If `end()` (or the drain in `close()`) runs while that open is pending, the peer has already received the uni stream by the time it resolves, yet the PUBLISH_DONE count excludes it — and the finished-guard at the top of the async block skips both the increment and the header write, so the peer sees a bare, uncounted, headerless stream. That is exactly the "count short of the streams it does receive" condition this comment says may close the session. The mirror direction also counts a fill whose `write()` then rejects (transport/peer dying), over-reporting a stream that never carried anything. The new unconditional fill service on non-`commit()` transports (the removed `hasMethods(writer, ['commit'])` guard) makes this reachable on Chromium too.</violation>
</file>
Reply to a comment to ask cubic a question or push back. It learns from your replies.
View guided diff | Re-trigger cubic
| // writes a header it cannot preserve through the reset. | ||
| if (!hasMethods(writer, ['commit'])) return; | ||
|
|
||
| // PUBLISH_DONE's Stream Count is the streams this side opened, |
There was a problem hiding this comment.
P2: The fill Stream Count snapshot can undercount a fill that is still opening when the track ends. #finTrack reads subscriber.fillStreams and sends PUBLISH_DONE immediately, but #openAndResetFill increments that counter only after await this.openUniStream() resumes. If end() (or the drain in close()) runs while that open is pending, the peer has already received the uni stream by the time it resolves, yet the PUBLISH_DONE count excludes it — and the finished-guard at the top of the async block skips both the increment and the header write, so the peer sees a bare, uncounted, headerless stream. That is exactly the "count short of the streams it does receive" condition this comment says may close the session. The mirror direction also counts a fill whose write() then rejects (transport/peer dying), over-reporting a stream that never carried anything. The new unconditional fill service on non-commit() transports (the removed hasMethods(writer, ['commit']) guard) makes this reachable on Chromium too.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/spf/src/publish/session/publish-session.ts, line 670:
<comment>The fill Stream Count snapshot can undercount a fill that is still opening when the track ends. `#finTrack` reads `subscriber.fillStreams` and sends PUBLISH_DONE immediately, but `#openAndResetFill` increments that counter only after `await this.openUniStream()` resumes. If `end()` (or the drain in `close()`) runs while that open is pending, the peer has already received the uni stream by the time it resolves, yet the PUBLISH_DONE count excludes it — and the finished-guard at the top of the async block skips both the increment and the header write, so the peer sees a bare, uncounted, headerless stream. That is exactly the "count short of the streams it does receive" condition this comment says may close the session. The mirror direction also counts a fill whose `write()` then rejects (transport/peer dying), over-reporting a stream that never carried anything. The new unconditional fill service on non-`commit()` transports (the removed `hasMethods(writer, ['commit'])` guard) makes this reachable on Chromium too.</comment>
<file context>
@@ -660,20 +667,20 @@ class MoqtPublishSessionImpl implements MoqtPublishSession {
- // writes a header it cannot preserve through the reset.
- if (!hasMethods(writer, ['commit'])) return;
-
+ // PUBLISH_DONE's Stream Count is the streams this side opened,
+ // fill fetch streams included (§10.12), so count this one even
+ // when the reset may discard its header. The subscriber waits
</file context>
There was a problem hiding this comment.
Checked by an agent working on this PR for me (relay-1q0.16). Its findings:
The code behaves correctly, but the comment was misleading. I fixed the comment and added a test in 8f3e84f.
A fill whose stream is still opening when the track ends is not counted. That is intended, and it cannot leave the subscriber with more streams than Stream Count says. Here is why.
When openUniStream() resolves after #finTrack, the guard if (!subscriber.fills.has(fill) || subscriber.finished) return; returns before subscriber.fillStreams++ and before writer.write(encodeFetchHeader(...)). The finally then aborts the writer. So the stream is reset without one MoQT byte on it. The subscriber has no FETCH_HEADER and no Request ID to tie that stream to this subscription. That means it cannot count the stream against this PUBLISH_DONE. A fill that does carry a header is always counted, because fillStreams++ runs in the same synchronous step that hands the header to write(). So whenever PUBLISH_DONE goes out, every fill with a header is already in the count.
The track publisher counts subgroup streams by this same rule, which #62 already ships. In track-publisher.ts, the JSDoc on streamCount says the count is taken "at the commit to writing the subgroup header (the openedGroups boundary), so a stream reset before its header — one the peer cannot attribute to the subscription — is excluded". An open that resolves after abort is aborted there and not counted.
draft-ietf-moq-transport-20 §10.12 (PUBLISH_DONE):
Stream Count: An integer indicating the number of data streams the publisher opened for this subscription, including streams that contained no Objects (e.g., an empty Subgroup) and including any fill fetch streams (see Section 5.1.3). [...] Subscribers SHOULD use a timeout or other mechanism to remove subscription state in case the publisher set an incorrect value, reset a stream before the SUBGROUP_HEADER, or set the maximum value. If a subscriber receives more streams for a subscription than specified in Stream Count, it MAY close the session with a PROTOCOL_VIOLATION.
The PROTOCOL_VIOLATION applies to streams received "for a subscription". A stream reset before its header was never received for any subscription.
The reverse case is a fill that is counted and whose write() then rejects, or one whose header is discarded by a reset without commit(). Either one over-counts by one. The draft covers that case: the subscriber "SHOULD use a timeout [...] in case the publisher set an incorrect value, reset a stream before the SUBGROUP_HEADER". The subgroup path does the same thing when writeObject rejects.
Removing the commit() gate makes this path reachable on Chromium, but Chromium behaves just as it does on transports that have commit(). The finished-guard comes before any commit() question.
What changed in 8f3e84f:
- Comment. The old comment said "count this one even when the reset may discard its header ... a count short of the streams it does receive may close the session". That reads as "count every stream opened", and it is what made this case look like a bug. The new comment states the actual rule: count at the commit to the header, like the track publisher. It says why a fill still opening at the end is left out, and why the over-count side is safe.
- Test. I added 'leaves a fill still opening when the track ends out of the Stream Count, headerless', in two variants: with and without reliable reset. It ends the track while the fill's open is still pending, with
getStreamCountreturning 3. PUBLISH_DONE carriesstreamCount: 3. Then the open resolves, and the fill stream gets no write and is aborted. I checked that the test catches a change of rule: when I movedfillStreams++to the start of the open, both variants failed.
To count fills while their open is still pending, a design choice is needed first. Their late streams would then be counted but carry no header, and the fill rule would no longer match the subgroup rule. I did not make that change.
| // cancelled it and pulled again for the viewer — now with a nonempty | ||
| // fill this transport cannot fail reliably. The pull is served anyway. | ||
| largest = { group: 0, object: 0 }; | ||
| await probe.reset(); |
There was a problem hiding this comment.
P3: bindings.at(-1) === 237 races with the probe's unbind: the probe subscription's onTrackBinding(trackAlias: undefined) and pull 237's bind are processed in independent handlers, so the unbind can land after the bind, leaving at(-1) undefined and timing the waitFor out. Wait for the probe's unbind before opening the second pull.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/spf/src/publish/session/tests/publish-session.test.ts, line 1370:
<comment>`bindings.at(-1) === 237` races with the probe's unbind: the probe subscription's `onTrackBinding(trackAlias: undefined)` and pull 237's bind are processed in independent handlers, so the unbind can land after the bind, leaving `at(-1)` undefined and timing the waitFor out. Wait for the probe's unbind before opening the second pull.</comment>
<file context>
@@ -1320,39 +1327,164 @@ describe('createMoqtPublishSession', () => {
+ // cancelled it and pulled again for the viewer — now with a nonempty
+ // fill this transport cannot fail reliably. The pull is served anyway.
+ largest = { group: 0, object: 0 };
+ await probe.reset();
+ const pull = await relayPull(237);
+
</file context>
| await probe.reset(); | |
| await probe.reset(); | |
| await vi.waitFor(() => expect(bindings.at(-1)).toBeUndefined()); |
There was a problem hiding this comment.
Checked by an agent working on this PR for me (relay-1q0.16). Its findings:
Not a race. f0577b4 adds a test that covers the ordering this thread worries about.
The probe's end cannot leave undefined as the last binding once pull 237 is accepted. When a subscription ends, the finally in its subscribe handler does not report a plain unbind. It reports the newest live binding: track.subscribers.filter((s) => !s.finished && s.accepted && s.forwarding).at(-1). If the probe's end is handled after 237 is accepted, it reports 237 again. If it is handled before, it reports undefined, and 237's acceptance then reports 237. Either way the last binding is 237.
I did not take the suggested waitFor(... toBeUndefined()). It would make the test wait for the unbind before the second SUBSCRIBE. moq-relay does not wait for that, so the test would stop covering the overlapping case. f0577b4 instead runs the relay test both ways (it.each):
probeEnds: 'before'is the existing ordering, unchanged.probeEnds: 'after'opens pull 237 while the probe is still live, cancels the probe only once 237 is bound, and waits for the probe'sonSubscribeEnd. It then asserts thatbindings.at(-1)is still 237.
I checked that the new case catches the race: when I made the finally report trackAlias: undefined, the 'after' case failed, and so did the existing 'hands the binding to the newest subscription...' test.
Loop runs, each one a full Vitest run of the test:
… end Review (cubic, mmcc#63) read the fill counter's comment as "count every stream this side opens" and flagged a fill whose stream is still opening when the track ends: PUBLISH_DONE goes out without it. That is the intended rule, the one the track publisher applies to subgroup streams (mmcc#62): a stream counts in the step that commits it to its header, so a stream reset before its header, which the subscriber cannot attribute to the subscription, is left out. The late fill is reset with no header, so the count can never fall short of the streams the subscriber can tie to the subscription. Reword the comment to say so, including the over-count side (a counted fill whose reset discards its header, or whose write rejects), which §10.12 covers with the subscriber's timeout. Add a test that ends the track while the fill's open is pending, with and without reliable reset: Stream Count 3, the fill stream gets no write and is aborted.
Review (cubic, mmcc#63) suspected the probe's unbind could land after pull 237's bind and leave the binding undefined. It cannot: a subscription's end reports the newest live binding, not an unbind, and pull 237 is already accepted. Run the relay test both ways, with the probe cancelled before and after the second pull, and check the binding is still 237 once the probe's end has been handled. Reporting an unbind there fails the new case.
Summary
A v10 publisher could not be watched from any relay other than the one it publishes to. When moq-relay pulls a track that already has content, its SUBSCRIBE asks for a fill. The publish session refused that SUBSCRIBE with NOT_SUPPORTED, because the browser's WebTransport writers have no
commit(). A relay pulls a track a second time whenever a viewer joins from another relay, so the catalog never reached that viewer. This change serves the subscription and still fails the fill, as the draft requires.The problem, measured on our fleet
The fleet is four moq-relay 0.17.0 relays (
oci1.{phx,ord,iad,fra}.relay.mux.global) in a star with iad as the hub. The test is a v10MoqPublishMediapublisher at phx and a v10<simple-moq-video>player, both in headless Chromium 147 on one host, with glass-to-glass measured canvas to canvas.At
mmcc/moq-publisher(d65c0c6ba, anddafbe32d6before it), the player plays when it is on the publisher's relay (phx/phx 3/3, TTFF 0.26–0.31 s). On any other relay it stays atreadyState 0with no catalog: phx→iad, phx→ord and phx→fra failed 9 of 9 today. Last nightdafbe32d6failed the same paths 84 of 84.The player is not the cause. The same player build plays phx→fra from two other MSF+LOC publishers (relay-sdks and media-shim, 3 of 3 each).
The publisher's relay logs the refusal for every failed run. Below is phx, for one phx→iad run:
phx logged
err=unsupportedin every one of today's failed runs (94 lines across 8 of them). iad reportsinternal errordownstream andtrack info failedon every retry.Root cause
FILL_PARAMETERSfor the current group (subscribe_joininmoq-net/src/ietf/subscriber.rs).#handleIncomingSubscribeanswered a nonempty fill withREQUEST_ERROR NOT_SUPPORTED "fill requires reliable stream reset"whenever the control writer had nocommit(). Chromium 147 hasWebTransportbut noWebTransportWriterand nocommit(), so every second pull was refused.commit(), so no test exercised the browser's case.Changes
commit()where the writer has it, then reset.commit(): the reset may discard the header, and the subscriber then sees no fill at all. That leaves its join at the next group boundary, just as a failed fill does. For this publisher that costs nothing: groups never span bindings, and every bind replays the current group from object 0 under a fresh Group ID. moq-relay 0.17.0 waits on a fill only to stitch a mid-group tail, which this publisher never sends.createRelayHub({ reliableReset: false })models a transport withoutcommit().Stacked on #62
This branch is now based on #62 (
fix/moq-publish-done-track-endedat10606e84e), which ends tracks with PUBLISH_DONE and an exact Stream Count. Only the top commit belongs to this PR, and it merges after #62.The two PRs met in
#openAndResetFill. #62 counted a fill fetch stream (subscriber.fillStreams++) just past thecommit()gate that this PR removes. The fill now counts wherever the session opens one, whether or not its header survives the reset:poll_settledinrs/moq-net/src/ietf/subscriber.rs,GRACEinrs/moq-net/src/tail.rs). A fill whose header was lost stays outstanding, so that track end waits out the grace whatever the count. That 1 s delay before "track ended" is the cost of a reset withoutcommit(). It comes from reading the relay's code; on the fleet it did not appear in 21 unpublish runs (the fill headers reached the relay before the reset), so the header-lost case is still unmeasured.New test: counts a fill reset without reliable reset in the PUBLISH_DONE Stream Count, with the fill's header delivered and with it lost. Both cases report 3 track streams plus the fill, so 4. Both fail if the count is gated on
commit()(they get 3), and both fail on #62 alone, where the SUBSCRIBE is refused.Testing
publish-session.test.ts:relay-hub.test.tsruns its relay-pull case with and without reliable reset.pnpm -F @videojs/spf testpassed 3 runs in a row (169 files, 2,727 tests, 11 skipped).pnpm build:packages,pnpm lintandpnpm typecheckpass.f1067b323, 18:39–18:54Z, fleet on its default config): the player at target 0.5 s passed 9/9 with the same G2G as before the rebase (phx/phx 554 ms, phx→iad 608 ms, phx→fra 688 ms; TTFF 0.31 / 0.89 / 2.10 s). A relay-sdks Python subscriber decoded the broadcast (video and audio) on phx/phx, phx→fra and two rejoin cases, 12/12, with every frame decoded.unpublish(), viewers got PUBLISH_DONE status 2 "track ended" on catalog and video in 9/9 player runs, about 46 / 211 / 514 ms after the stop at phx / iad / fra, and the relays logged no errors. One thing still to look at: with audio in the broadcast, 5 of 12 subscriber runs saw one track (audio in all 5, video once) end with status 0 "internal error" although the publisher sent status 2. It looks like a race with the WebTransport session closing 24–40 ms afterunpublish()(not yet confirmed). Closing the page withoutunpublish()still ends every track with status 0.7e1a9002e(this change without fix(spf): end published tracks with PUBLISH_DONE track ended #62).target-latency0.5 s (the default) and 0.1 s, on 2026-10-08 between 14:52 and 15:20Z, with the fleet on its default config. Each cell shows passes, the median G2G p50 (range) and the median TTFF (range). The last row is a control on two local moq-relay 0.17.0 relays clustered over loopback.d65c0c6ba), target 0.5 serr=unsupportedlines for the cross-relay runs.e3c77e8ebuild (moqt-19) joined in 2.1–2.5 s at similar G2G (phx→iad 612 ms, phx→fra 719 ms at 0.5 s).🤖 Generated with Claude Code
https://claude.ai/code/session_01Ts6nhCDi2pBF9h5ooBNbss