Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions doc/concept/moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,21 @@ sides explicitly enable it.

On moq-lite 07, `SUBSCRIBE_END` counts the group streams opened for the
subscription. Rust and TypeScript stop waiting for missing streams once that
many headers have arrived; skipped group sequences add no wait. Groups already
being received continue until their own stream ends or resets.
many headers have arrived; skipped group sequences add no wait. A stream whose
header arrived is always read to its end before the subscription completes.

A stream reset before its header arrived cannot be counted, so the subscriber
still allows a grace period for late streams. The grace uses the subscription's
nonzero effective maximum age, or one second when no maximum age is set.
moq-lite 05 and 06 instead account for group sequences using received headers
and `SUBSCRIBE_DROP`.
moq-lite 05 and 06 instead account for group sequences using received headers,
datagrams, and `SUBSCRIBE_DROP`, from the start group the subscription last
asked for. A lost datagram is not owed, but its hole waits out the grace like a
lost stream.

From moq-lite 06, an end that contradicts the groups received aborts the track:
a group at or past `SUBSCRIBE_END`, or a `SUBSCRIBE_END` below a group already
received. moq-lite 05 specified an inclusive end, so there it only drops that
group or the early boundary.

## Discovery

Expand Down
13 changes: 11 additions & 2 deletions js/net/src/ietf/subscriber.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { Cost, type Route, routesEqual, UNKNOWN_HOP } from "../hop.ts";
import { hiddenBelow, hooks, scopeCaptures, scopeHead, scopeOverlaps } from "../internal.ts";
import * as Path from "../path.ts";
import type { Cursor, Reader, Stream } from "../stream.ts";
import { TAIL_GRACE_MS, Tail } from "../tail.ts";
import { Tail } from "../tail.ts";
import { type Timescale, Timestamp } from "../time.ts";
import type * as track from "../track.ts";
import { TimeoutError, withTimeout } from "../util/timeout.ts";
Expand Down Expand Up @@ -52,6 +52,8 @@ type Subscription = {
// The group streams received, so the subscription can wait for the ones PUBLISH_DONE
// says are still owed.
tail: Tail;
// The track's exclusive end, once an END_OF_TRACK declares it.
end?: number;
};

// Out-parameter for #openSubscribe: lets the caller observe partial progress
Expand Down Expand Up @@ -674,7 +676,7 @@ export class Subscriber {

const { tail, track } = subscription;
const complete = () => count > 0n && BigInt(tail.streams) >= count;
await tail.settle(complete, TAIL_GRACE_MS, track.closed);
await tail.settle(complete, track.closed);
}

/**
Expand Down Expand Up @@ -1018,6 +1020,12 @@ export class Subscriber {
let producer: netGroup.Producer | undefined;
const open = () => {
if (!producer) {
// The publisher contradicted its own end, which no later group can repair.
if (subscription.end !== undefined && group.groupId >= subscription.end) {
throw new ProtocolViolation(
`group ${group.groupId} is at or past the declared end ${subscription.end}`,
);
}
producer = new netGroup.Producer(group.groupId);
track.writeGroup(producer);
}
Expand Down Expand Up @@ -1068,6 +1076,7 @@ export class Subscriber {
} catch (err: unknown) {
throw new ProtocolViolation(`invalid END_OF_TRACK: ${reason(error(err))}`);
}
subscription.end ??= end;
return;
}
if (frame.payload === undefined) break;
Expand Down
14 changes: 14 additions & 0 deletions js/net/src/ietf/tail.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,20 @@ test("END_OF_TRACK at object 0 ends the track before its group, which never exis
expect(reader.final()).toBe(2);
});

test("a group at or past the END_OF_TRACK aborts the track", async () => {
const { subscriber, reader } = await subscribed();
const end = groupStream(subscriber, 2);
end.write(END_OF_TRACK);
end.finish();
expect(await reader.finished()).toBe(2);

const late = groupStream(subscriber, 2);
late.write(object("2.0"));
late.finish();
await late.handled;
expect(await reader.closed).toBeInstanceOf(ProtocolViolation);
});

test("a bare FIN without PUBLISH_DONE aborts the track", async () => {
const { reader, fin } = await subscribed();
await fin();
Expand Down
62 changes: 52 additions & 10 deletions js/net/src/lite/subscriber.ts
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ interface SubscribeEntry {
// The group streams received, so the subscription can wait for the ones still owed
// after the publisher ends it.
tail: Tail;
// The first group the SUBSCRIBE asked for, if any.
requested?: number;
// The first group the publisher serves (SUBSCRIBE_START) and the track's exclusive end
// (SUBSCRIBE_END), once it declares them.
start?: number;
Expand Down Expand Up @@ -587,7 +589,7 @@ export class Subscriber {
const subscriptionUpdates =
this.version === Version.DRAFT_01 || this.version === Version.DRAFT_02
? undefined
: this.#runSubscriptionUpdates(id, broadcast, producer, msg, stream);
: this.#runSubscriptionUpdates(id, broadcast, entry, msg, stream);

// Terminal conditions (stream end, track close, a failed subscription update) settle at most
// once; race them into one stable promise so the demand loop doesn't re-subscribe each pass.
Expand Down Expand Up @@ -652,7 +654,20 @@ export class Subscriber {
}

// Register before opening SUBSCRIBE so a racing GROUP stream finds the entry.
const entry: SubscribeEntry = { track: producer, timescale, tail: new Tail() };
const entry: SubscribeEntry = {
track: producer,
timescale,
// The effective max age is the stopgap grace: the wrong clock (it bounds
// presentation-time drift), but it is how long the subscriber was willing to wait
// for a late group anyway. Already the smaller of the subscriber's and the track's.
tail: new Tail({
grace: () => {
const maxAge = producer.subscription.peek()?.maxAge ?? Time.Milli.zero;
return maxAge > 0 ? maxAge : TAIL_GRACE_MS;
},
}),
requested: msg.startGroup,
};
this.#subscribes.set(id, entry);

state.stream = await Stream.open(this.#quic);
Expand Down Expand Up @@ -872,6 +887,9 @@ export class Subscriber {

if ("start" in resp) {
entry.start = resp.start.group;
// The groups the SUBSCRIBE asked for below it are unavailable, whatever the
// demand asks later.
if (entry.requested !== undefined) entry.tail.account(entry.requested, entry.start);
} else if ("end" in resp) {
if (entry.end !== undefined) throw new ProtocolViolation("duplicate SUBSCRIBE_END");
entry.end = resp.end.group;
Expand All @@ -881,7 +899,13 @@ export class Subscriber {
try {
entry.track.finishAt(entry.end);
} catch (err) {
throw new ProtocolViolation(`invalid SUBSCRIBE_END: ${reason(error(err))}`);
// lite-05 specified an inclusive end, and @moq/net 0.1.3 to 0.1.9 sent one, so
// there an end below a received group only costs the early boundary: the FIN
// still finishes the track. Later drafts made it exclusive.
if (this.version !== Version.DRAFT_05) {
throw new ProtocolViolation(`invalid SUBSCRIBE_END: ${reason(error(err))}`);
}
console.warn(`invalid SUBSCRIBE_END: ${reason(error(err))}`);
}
} else if ("drop" in resp) {
entry.tail.account(resp.drop.start, resp.drop.end + 1);
Expand All @@ -897,23 +921,24 @@ export class Subscriber {
// headers arrived keep reading until their own FIN or reset.
#settleTail(entry: SubscribeEntry): Promise<void> {
const { tail, track } = entry;
// Already the smaller of the subscriber's and the track's max age.
const maxAge = track.subscription.peek()?.maxAge ?? Time.Milli.zero;
const grace = maxAge > 0 ? maxAge : TAIL_GRACE_MS;

const complete = () => {
if (entry.streams !== undefined) return tail.streams >= entry.streams;
// Without SUBSCRIBE_END (older drafts) nothing says which groups are owed.
if (entry.end === undefined) return false;
// Without SUBSCRIBE_START the publisher served no group at all.
if (entry.start === undefined) return true;
const bounds = groupBounds(track.subscription.peek()?.groups ?? {});
const start = Math.max(entry.start, bounds.start);
// Owed from the floor the demand last asked for, which an update can move either
// way, or where SUBSCRIBE_START resolved a live-edge one. The groups the SUBSCRIBE
// asked for below its SUBSCRIBE_START were accounted for when it arrived.
const groups = track.subscription.peek()?.groups;
const bounds = groupBounds(groups ?? {});
const start = groups?.start === undefined ? entry.start : bounds.start;
const end = bounds.end === undefined ? entry.end : Math.min(entry.end, bounds.end);
return tail.covers(start, end);
};

return tail.settle(complete, grace, track.closed);
return tail.settle(complete, track.closed);
}

/**
Expand All @@ -930,10 +955,11 @@ export class Subscriber {
async #runSubscriptionUpdates(
id: bigint,
broadcast: Path.Valid,
track: track.Producer,
entry: SubscribeEntry,
msg: Subscribe,
stream: Stream,
): Promise<void> {
const track = entry.track;
const stopped: Promise<null> = race([track.closed, stream.reader.closed]).then(() => null);
let lastSent: track.Subscription = {
priority: msg.priority,
Expand All @@ -958,6 +984,12 @@ export class Subscriber {
const bounds = groupBounds(current.groups);
if (emptyRange({ startGroup: bounds.start, endGroup: bounds.end })) throw new Error(EMPTY_RANGE);

// A lowered floor owes groups nobody asked for until now.
if (current.groups?.start !== undefined) {
const floor = lastSent.groups?.start === undefined ? entry.start : groupBounds(lastSent.groups).start;
entry.tail.demand(bounds.start, floor ?? Number.POSITIVE_INFINITY);
}

// Round-trip the other Subscribe parameters so the publisher doesn't
// interpret SUBSCRIBE_UPDATE as a reset of ordered/maxAge/etc.
const update = new SubscribeUpdate({
Expand Down Expand Up @@ -1005,6 +1037,16 @@ export class Subscriber {
const read = tail.open(group.sequence);

try {
// The publisher contradicted its own end, which no later group can repair. lite-05
// specified an inclusive end, so its last group lands on it: the write below drops
// only that group there.
if (entry.end !== undefined && group.sequence >= entry.end && this.version !== Version.DRAFT_05) {
const violation = new ProtocolViolation(
`group ${group.sequence} is at or past the declared end ${entry.end}`,
);
track.close(violation);
throw violation;
}
track.writeGroup(producer);

// Block until the timescale is known; the group's stream can arrive before
Expand Down
86 changes: 81 additions & 5 deletions js/net/src/lite/tail.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { createMockTransportPair } from "../mock.ts";
import * as Path from "../path.ts";
import { Reader, Stream } from "../stream.ts";
import { Milli } from "../time.ts";
import type { Groups } from "../track.ts";
import { Group as GroupMessage } from "./group.ts";
import { StreamId } from "./stream.ts";
import {
Expand Down Expand Up @@ -51,10 +52,10 @@ function groupStream(subscriber: Subscriber, sequence: number) {
* it answers TRACK_INFO, then writes whatever responses the test asks for on the subscribe
* stream and FINs it when told.
*/
async function subscribed(version: Version, maxAge = GRACE) {
async function subscribed(version: Version, maxAge = GRACE, groups?: Groups) {
const pair = createMockTransportPair(ALPN_05);
const subscriber = new Subscriber(pair.client, version, randomHop());
const reader = subscriber.consume(Path.from("room")).track("video").subscribe({ maxAge });
const reader = subscriber.consume(Path.from("room")).track("video").subscribe({ maxAge, groups });

const info = await Stream.accept(pair.server);
if (!info) throw new Error("the subscriber never asked for TRACK_INFO");
Expand Down Expand Up @@ -210,16 +211,91 @@ describe.each([Version.DRAFT_05, Version.DRAFT_06, Version.DRAFT_07])("%s", (ver
expect(reader.final()).toBe(0);
});

test("a SUBSCRIBE_END below a group already received aborts the track", async () => {
// lite-05 specified an inclusive end, so there the group costs only its own stream.
test(`a group at or past SUBSCRIBE_END ${version === Version.DRAFT_05 ? "is dropped" : "aborts the track"}`, async () => {
const { subscriber, reader, respond } = await subscribed(version);
await respond({ start: new SubscribeStart(0) });
await respond({ end: new SubscribeEnd(2, 2) });
expect(await reader.finished()).toBe(2);

await groupStream(subscriber, 2).handled;
if (version === Version.DRAFT_05) {
expect(await settlesWithin(Promise.resolve(reader.closed), 50)).toBe(false);
} else {
expect(await reader.closed).toBeInstanceOf(ProtocolViolation);
}
});

test("a floor below SUBSCRIBE_START owes nothing below it", async () => {
const { subscriber, reader, respond, fin } = await subscribed(version, Milli(60_000), {
start: { included: 1 },
});
await respond({ start: new SubscribeStart(3) });
const group = groupStream(subscriber, 3);
group.finish();
await group.handled;
await respond({ end: new SubscribeEnd(4, 1) });
await fin();

expect((await reader.recvGroup())?.sequence).toBe(3);
expect(await settlesWithin(reader.recvGroup(), 1000)).toBe(true);
expect(await reader.closed).toBeNull();
});

test.skipIf(version === Version.DRAFT_07)("a lowered floor owes the groups it newly asked for", async () => {
const maxAge = Milli(60_000);
const { subscriber, respond, fin } = await subscribed(version, maxAge, { start: { included: 3 } });
await respond({ start: new SubscribeStart(3) });
const group = groupStream(subscriber, 3);
group.write("3.0");
group.finish();
await group.handled;

// A second reader lowers the floor, which the subscription forwards as an update.
const lower = subscriber
.consume(Path.from("room"))
.track("video")
.subscribe({ maxAge, groups: { start: { included: 1 } } });
await respond({ end: new SubscribeEnd(4, 1) });
await fin();
// Long enough for a subscription that owed nothing more to have ended.
await new Promise((resolve) => setTimeout(resolve, 50));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Replace the wall-clock synchronization with mocked time

This 50 ms sleep is the synchronization that determines whether the regression catches the old early-close behavior. On a loaded CI runner, a broken subscription may not finish teardown before the sleep returns, allowing groups 1 and 2 to arrive first and making the test pass spuriously. Advance mocked time and drain the relevant tasks deterministically instead of depending on scheduler timing.

AGENTS.md reference: AGENTS.md:L19-L19

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining. The mock transport is in-process with no timers, so the broken teardown is a microtask chain: every queued microtask drains before any timer fires, and a loaded runner only delays the timer, never the chain. The 50 ms is an event-loop drain, not a race. The age logic itself is now covered with an injected clock in tail.test.ts.

(Written by Claude Opus 5.5)


// QUIC does not order streams, so the lowered groups land after the FIN.
for (const sequence of [1, 2]) {
const late = groupStream(subscriber, sequence);
late.write(`${sequence}.0`);
late.finish();
await late.handled;
}

const received: number[] = [];
for (;;) {
const next = await lower.recvGroup();
if (!next) break;
received.push(next.sequence);
}
expect(received).toContain(1);
expect(received).toContain(2);
expect(await lower.closed).toBeNull();
});

// lite-05 specified an inclusive end, and @moq/net 0.1.3 to 0.1.9 sent one, so there an end
// below a received group only costs the early boundary.
test(`a SUBSCRIBE_END below a group already received ${version === Version.DRAFT_05 ? "finishes clean" : "aborts the track"}`, async () => {
const { subscriber, reader, respond, fin } = await subscribed(version);
await respond({ start: new SubscribeStart(0) });
const group = groupStream(subscriber, 3);
group.finish();
await group.handled;
await respond({ end: new SubscribeEnd(2, 2) });

const closed = await reader.closed;
expect(closed).toBeInstanceOf(Error);
if (version === Version.DRAFT_05) {
await fin();
expect(await reader.closed).toBeNull();
} else {
expect(await reader.closed).toBeInstanceOf(ProtocolViolation);
}
});
});

Expand Down
Loading
Loading