Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 11 additions & 21 deletions .claude/skills/verify-subscribe/SKILL.md

Large diffs are not rendered by default.

9 changes: 9 additions & 0 deletions doc/concept/moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,15 @@ A reader of a reported group, including one still arriving, ends with the publis
Before moq-lite 07 that code is application-specific, so every reported group ends with `OLD`.
Datagram delivery stays best effort.

A peer that grants fewer streams than its subscriptions have groups in flight, as a
browser does at about 100, gets them for the highest-priority groups. To make room
the publisher resets the lowest-priority group's stream with `TOO_FAR_BEHIND`, on
every moq-lite version, and from moq-lite 05 also reports that group as dropped.
Its reader ends with that error after the frames that arrived, and no later
stream carries the rest, even for a reader with no max delay. A reader that still
wants the group fetches it, which returns it whole on a stream the subscriber
opens, outside the cap.

Ranges are both a request and a filter. A subscriber reads only the groups its
own ranges cover, resolved again whenever it replaces them, even when another
subscriber of the same track asked for more. Widening them, including lowering
Expand Down
41 changes: 10 additions & 31 deletions drafts/draft-lcurley-moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,8 +203,8 @@ A frame is used to represent a chunk of data with an upfront size.
The contents are opaque to the moq-lite layer.

Frames within a Group are numbered from 0 in the order they were produced.
This index is not transmitted per frame; it is implied by position within the Group, anchored by the [GROUP](#group-message) message's `Frame Start`.
A Group Stream normally starts at frame 0.
This index is not transmitted per frame; it is implied by position within the Group.
A Group Stream starts at frame 0, and a Fetch Stream at its FETCH's `Frame Start`.

Each frame carries a presentation timestamp expressed in the Track's `Timescale` (see [TRACK_INFO](#track-info)), used by the moq-lite layer for [expiration](#expiration) decisions.
A Track with a `Timescale` of 0 is scale-less: its timestamps and Group starts are bare numbers in no known unit, and each frame's timestamp is its Group's start.
Expand All @@ -224,7 +224,7 @@ A subscriber that has received frames 0 through `N-1` of the Group starting at `
This is what lets a subscriber move a Track to a different publisher partway through a Group instead of waiting for the next Group to start, which matters for Tracks whose current Group may stay open indefinitely.
A Group can therefore be assembled from more than one publisher: each contributes a disjoint run of frames, and the subscriber concatenates them in index order.

A partial Group is only ever delivered to a subscriber that asked for one with FETCH, or as the [continuation](#yield) of a paused Group.
A partial Group is only ever delivered to a subscriber that asked for one with FETCH.
A publisher that holds only the later frames of a Group a subscription covers names it in a [SUBSCRIBE_DROP](#subscribe-drop) rather than deliver the part it holds.
One that holds its earlier frames, such as a relay whose upstream closed the Group or a Group expiring as it is sent, delivers them and closes the Group Stream without a GROUP_END.
A Group is the unit of decodability: its earlier Frames may carry a keyframe or compression state the later Frames need.
Expand Down Expand Up @@ -362,8 +362,6 @@ Sent when resetting a stream (RESET_STREAM), or when refusing to receive one (ST
| ------- | ------------- | ----------- |
| 0x3B | UNAUTHORIZED | The endpoint does not authorize this request, or no longer does, such as when the [scope](#auth-stream) does not cover it. The session stays up. |
| ------- | ------------- | ----------- |
| 0x3C | YIELDED | The Group Stream was reset to make room for a higher-priority stream. The Group is not given up: it [continues](#yield) on a later stream. |
| ------- | ------------- | ----------- |

Note that CANCELLED is 0x1, not 0x0: a stream reset with 0x0 is an INTERNAL_ERROR, not a routine cancellation.
An endpoint terminating a stream because the session is ending SHOULD use SESSION_CLOSED rather than the session's own code, since the two spaces are disjoint.
Expand Down Expand Up @@ -675,7 +673,7 @@ Drops and Groups name starts, so the subscriber judges them against its latest R
The publisher never closes its side first: a subscription whose every Range has an `End` other than None and whose every Group has been delivered, skipped, or dropped stays open for an update to widen it.

A subscriber fills Groups it misses by asking for them as Ranges, adding them to an open subscription with SUBSCRIBE_UPDATE or opening a bounded SUBSCRIBE when none is open, rather than with a FETCH of each Group; FETCH remains for one Group, or a slice of one.
A publisher never sends a Group twice on one subscription, except to [continue](#yield) one, so a Group the subscription already delivered or named in a SUBSCRIBE_DROP is asked for with a SUBSCRIBE of its own.
A publisher never sends a Group twice on one subscription, so a Group the subscription already delivered or named in a SUBSCRIBE_DROP is asked for with a SUBSCRIBE of its own.
That includes a Group dropped with OLD that the subscriber still wants, such as a relay reader whose `Subscriber Max Age` is wider than the one the drop was judged under: no SUBSCRIBE_UPDATE reopens a drop.

The subscriber closes its side (FIN) once it wants nothing more from the subscription, and the publisher then closes its side (FIN), cancelling any Group Stream still open.
Expand Down Expand Up @@ -929,28 +927,19 @@ A Group Stream MUST start with a GROUP message and MAY be followed by any number
A Group MAY contain zero FRAME messages, potentially indicating a gap in the track.
A frame MAY contain an empty payload, potentially indicating a gap in the group.

The GROUP message's `Frame Start` gives the index of the first FRAME on the stream, so a stream carrying only part of a Group is self-describing.

A publisher ends a Group Stream the way its Group ended (see [Group](#group)): with a GROUP_END and a FIN for a finished Group, with a FIN alone for a closed one, and with a reset for an aborted one.
A publisher MUST NOT send a GROUP_END for a Group that has not finished at this hop, and a subscriber treats bytes after a GROUP_END as a malformed Track.

A publisher MUST NOT send two Group Streams for the same Group Start on one subscription, except to [continue](#yield) one.
A publisher MUST NOT send two Group Streams for the same Group Start on one subscription.
A subscriber assembling a Group from more than one publisher does so across separate subscriptions, and is responsible for concatenating the runs in index order and for ignoring any frame it has already received.

Both the publisher and subscriber MAY reset the stream at any time.
This is not a fatal error and the session remains active.
The subscriber MAY cache the error and potentially retry later.

#### Yield {#yield}
A publisher out of the peer's stream credit MAY reset the lowest-priority Group Stream with YIELDED to make room for a higher-priority one.
A Group Stream of a Track carrying a MoqTransport Version under the `ietf` extension, whose object records name whole objects, is never reset with YIELDED.
Any other Group Stream may be, including one on a session with the extension: a continuation's records start at its `Frame Start`.

A Group whose stream was reset with YIELDED is paused, not given up: the subscriber keeps the frames it received, and the publisher MUST later continue the Group on another Group Stream or name it in [SUBSCRIBE_DROP](#subscribe-drop).
A continuation is a Group Stream for the paused Group whose `Frame Start` is the reset stream's own `Frame Start`, since a reset can discard frames the transport acknowledged.
The subscriber ignores the frames it already holds, including the part of a frame the reset cut short, and appends the rest; a continuation that starts past the frames it holds whole is malformed.
A continuation MAY overtake the earlier stream's reset, and waits unread until that stream ends; since the publisher reset that stream, the subscriber MAY instead end it at once and read the continuation, ignoring the frames it holds, as a browser can lose a reset that reaches a stream before its application does.
A subscriber whose transport reports a Group Stream's reset without its error code, as a browser can, pauses the Group as for YIELDED: the publisher's continuation or SUBSCRIBE_DROP settles it.
A publisher out of the peer's stream credit MAY reset the lowest-priority Group Stream with TOO_FAR_BEHIND to make room for a higher-priority one.
That gives the Group up on the subscription: the publisher names it in [SUBSCRIBE_DROP](#subscribe-drop), and a subscriber that wants the rest of it FETCHes the Group from the first frame it lacks (see [Positions](#positions)).
The drop settles the Group even where the subscriber's transport loses the reset or its error code, as a browser can.

## Datagrams
QUIC datagrams provide unreliable, unordered delivery for latency-sensitive content that does not need retransmission.
Expand Down Expand Up @@ -2122,8 +2111,7 @@ For every start in the subscription's range, the publisher MUST send a Group Str
This includes streams reset before their headers were delivered, and streams the subscriber stopped (STOP_SENDING), which it may have discarded unread; such a drop carries the STOP_SENDING's Error Code.
A publisher that names a Group in a drop after opening its stream also names, in a SUBSCRIBE_SKIP sent before the drop, the skipped starts that stream's `Previous Start` passed over, since the GROUP may never arrive.
A drop naming a stopped duplicate of a Group that another Group Stream already completed changes nothing for that Group: its readers keep it whole.
A reliably delivered reset retaining the Group Stream header accounts for that group instead, unless it is YIELDED or answers a STOP_SENDING.
A [paused](#yield) Group is accounted for only once a continuation completes it, a reset of one other than YIELDED ends it, or a SUBSCRIBE_DROP names it.
A reliably delivered reset retaining the Group Stream header accounts for that group instead, unless it answers a STOP_SENDING.
The publisher SHOULD name a start as soon as it gives it up, without waiting for SUBSCRIBE_FINISH.

## SUBSCRIBE_SKIP {#subscribe-skip}
Expand Down Expand Up @@ -2356,7 +2344,6 @@ GROUP Message {
Subscribe ID (i)
Group Start (i)
Previous Start (i)
Frame Start (i)
}
~~~

Expand All @@ -2374,12 +2361,6 @@ No Group has a start between the two, so the subscriber accounts for the starts
A publisher knows it when the starts below this Group were declared skipped down to a Group, or a Group starts just below it.
A value at or above `Group Start` plus one is a malformed Track.

**Frame Start**:
The index of the first FRAME message on this stream within the group (see [Positions](#positions)).
A plain absolute index; 0 means the stream carries the group from its beginning, which is the common case.
A non-zero value means the leading frames are not on this stream.
The subscriber MUST NOT assume it will receive the missing frames on another stream; they are a gap unless a FETCH covers them, or the stream [continues](#yield) a paused Group.


## FRAME
The FRAME message is a payload within a group.
Expand Down Expand Up @@ -2549,12 +2530,10 @@ A stream that FINs without a GROUP_END carries a closed Group, which may have he
- Replaced group datagrams with datagram Tracks, a namespace apart from Tracks of Groups. A subscriber subscribes to one on the Datagrams Stream (0x9) with DATAGRAMS and DATAGRAMS_UPDATE, which carry only a priority, and the publisher answers with DATAGRAMS_INFO. A datagram carries the Track's own Sequence and a Timestamp, with no Group, Inflated Size, or Range. Datagrams no longer count toward SUBSCRIBE_DROP, and a subscriber no longer deduplicates them against Group Streams.
- Added a leading Type to datagrams, 0x0 for a Track Datagram; a receiver discards one of an unknown Type.
- FRAME and datagrams always carry a timestamp, a bare number on a scale-less Track.
- Replaced GROUP's `Group Sequence` with `Group Start`, and added `Previous Start`, the start of the Group before it, so a subscriber sees the gap before a Group without a SUBSCRIBE_SKIP.
- Replaced GROUP's `Group Sequence` with `Group Start`, added `Previous Start`, the start of the Group before it, so a subscriber sees the gap before a Group without a SUBSCRIBE_SKIP, and removed `Frame Start`: a subscription reads whole Groups, so every Group Stream starts at frame 0.
- Added GROUP_END, a `Payload Length` of 0 followed by the Group's exclusive end: a Group or Fetch Stream ends with it when its Group finished, a FIN alone ends a closed Group whose frames may not all exist, and a reset aborts. A Group's own end bounds its reach.
- Expiration also measures a group's arrival age, the time since its successor arrived at this hop.
- A publisher closes the open Group Stream of a group that expires, without a GROUP_END, rather than resetting it, and a relay closes a Group it holds only the start of.
- Added [YIELDED](#yield) (0x3C): a publisher out of stream credit resets the lowest-priority Group Stream with it, which pauses the Group for a later Group Stream to continue from where the reset stream started.
- A subscriber MAY end a Group Stream once a later stream for its Group arrives, rather than wait for its reset, and pauses a Group whose reset arrives without its error code as for YIELDED.
- FRAME leads with `Payload Length`, the payload size plus one, in place of the trailing `Message Length`; 0 is malformed.
- A publisher SHOULD drop a datagram rather than block when the transport has no room for it, where it previously sent one only if the congestion controller could transmit it immediately.
- Added an optional `Subscriber Max Age` to FETCH, encoded as milliseconds plus one with zero meaning absent. It can tighten, but never relax, `Publisher Max Age`.
Expand Down
1 change: 0 additions & 1 deletion js/net/api/index.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -649,7 +649,6 @@ export const StreamCode: ((code: number) => StreamCode) & {
readonly TooFarBehind: StreamCode;
readonly Unauthorized: StreamCode;
readonly Unroutable: StreamCode;
readonly Yielded: StreamCode;
};

export type StreamCode = number & {
Expand Down
4 changes: 0 additions & 4 deletions js/net/src/error.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,6 @@ test("stream codes match the draft's Stream Error Codes table", () => {
FrameTooLarge: 0x38,
TimestampMismatch: 0x39,
Unauthorized: 0x3b,
Yielded: 0x3c,
});
});

Expand All @@ -276,7 +275,6 @@ test("the code tables keep their registry ranges", () => {
StreamCode.FrameTooLarge,
StreamCode.TimestampMismatch,
StreamCode.Unauthorized,
StreamCode.Yielded,
];
for (const code of Object.values(StreamCode)) {
if (assignedLite.includes(code)) {
Expand Down Expand Up @@ -404,7 +402,6 @@ test("toStreamCode and fromTransport agree on what a code means", () => {
StreamCode.Unroutable,
StreamCode.FrameTooLarge,
StreamCode.Unauthorized,
StreamCode.Yielded,
StreamCode(70),
]) {
expect(toStreamCode(fromTransport(toTransport(code, "reset")))).toBe(code);
Expand Down Expand Up @@ -438,7 +435,6 @@ test("toStreamCode: lite-only codes do not reach an IETF peer", () => {
StreamCode.GroupTooLarge,
StreamCode.Unroutable,
StreamCode.FrameTooLarge,
StreamCode.Yielded,
]) {
expect(toStreamCode(new StreamError(code), { version: Version.DRAFT_22 })).toBe(StreamCode.Internal);
}
Expand Down
5 changes: 0 additions & 5 deletions js/net/src/error.ts
Original file line number Diff line number Diff line change
Expand Up @@ -117,11 +117,6 @@ export const StreamCode = Object.freeze(
Unroutable: 0x36 as StreamCode,
/** The grant does not cover this request, or no longer does. The session stays up. */
Unauthorized: 0x3b as StreamCode,
/**
* The publisher reset a group's stream to make room for a higher-priority one, pausing the
* group for a later stream to continue rather than giving it up (lite-07).
*/
Yielded: 0x3c as StreamCode,
/** A group grew past its cache budget and was aborted. */
GroupTooLarge: 0x32 as StreamCode,
/** A frame's timedness or timestamp doesn't match its track's timescale. */
Expand Down
8 changes: 5 additions & 3 deletions js/net/src/integers.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,9 +145,11 @@ for (const version of [Lite.DRAFT_06, Lite.DRAFT_07]) {
await Writer.collect(version, (w) => LiteMessage.encode(w, (w) => w.write(body))),
version,
);
// Lite-07 carries the Previous Start ahead of the Frame Start.
const group = version === Lite.DRAFT_07 ? fields(version, 0, 0, 0, UNSAFE) : fields(version, 0, 0, UNSAFE);
await expect(LiteGroup.decode(await framed(group), version)).rejects.toThrow(RangeError);
// Only lite-06 carries a Frame Start.
if (version === Lite.DRAFT_06)
await expect(LiteGroup.decode(await framed(fields(version, 0, 0, UNSAFE)), version)).rejects.toThrow(
RangeError,
);
if (version === Lite.DRAFT_07)
await expect(SubscribeStart.decode(await framed(fields(version, 1, UNSAFE)), version)).rejects.toThrow(
RangeError,
Expand Down
31 changes: 16 additions & 15 deletions js/net/src/integration.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2995,10 +2995,10 @@ function qmuxPair(protocol: string, maxStreams: bigint): { client: Qmux; server:
return { client, server };
}

// On lite-07 a group whose stream a newer one reset to make room only pauses: once credit returns,
// a new stream continues it, and its reader receives every frame without an error. Mirrors the Rust
// `a_reset_group_resumes_without_loss`.
test("integration: lite-07 resumes a group reset to make room without loss", async () => {
// On lite-07 a group whose stream a newer one reset to make room is dropped: its reader ends with
// TOO_FAR_BEHIND, and a reader that wants the rest FETCHes it whole. Mirrors the Rust
// `a_reset_group_is_fetched_whole`.
test("integration: lite-07 drops a group reset to make room, and a FETCH reads it whole", async () => {
using clock = manual();
// Advance the clock until `work` settles: a waiting open makes room once it has waited about a
// round trip.
Expand All @@ -3024,29 +3024,30 @@ test("integration: lite-07 resumes a group reset to make room without loss", asy
// Unfinished, so its stream holds the only credit until the newer group resets it.
const older = producer.appendGroup();
older.writeFrame(textFrame("a"));
const paused = await settled(subscriber.recvGroup(), "the older group");
if (!paused) throw new Error("no older group");
expect(paused.sequence).toBe(0n);
expect(await settled(paused.readString(), "its first frame")).toBe("a");
const reset = await settled(subscriber.recvGroup(), "the older group");
if (!reset) throw new Error("no older group");
expect(reset.sequence).toBe(0n);
expect(await settled(reset.readString(), "its first frame")).toBe("a");

const newer = producer.appendGroup();
newer.writeFrame(textFrame("new"));
newer.finish();
const group = await settled(subscriber.recvGroup(), "the newer group taking the older one's credit");
expect(group?.sequence).toBe(1n);
expect(await settled(Promise.resolve(reset.closed), "the older group's end")).toMatchObject({
code: StreamCode.TooFarBehind,
});

// Written after the reset: a continuation carries them, behind the frame already held.
older.writeFrame(textFrame("b"));
older.writeFrame(textFrame("c"));
older.finish();
const rest: string[] = [];
const fetched = await settled(remote.track("0").fetchGroup(0n), "the FETCH of the dropped group");
const frames: string[] = [];
for (;;) {
const frame = await settled(paused.readString(), "the paused group's continuation");
const frame = await settled(fetched.readString(), "the fetched group");
if (frame === undefined) break;
rest.push(frame);
frames.push(frame);
}
expect(rest).toEqual(["b", "c"]);
expect(await paused.closed).toBeNull();
expect(frames).toEqual(["a", "b"]);
} finally {
subscriber.close();
remote.close();
Expand Down
Loading