From 117ce6ef29644bf5d7293782b69360fa77048d8f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 28 Oct 2026 22:12:35 -0700 Subject: [PATCH 1/9] chore(quest): claim quest/m1/ietf-publisher-one-group.md Quest-Claim: ef91d518da7d7e1e From f4a5cc8a95a3d1853c925f2b962f9657741ae92e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 28 Oct 2026 22:29:51 -0700 Subject: [PATCH 2/9] refactor(net): serve IETF history one group at a time Co-Authored-By: GPT-6.1 Sol --- doc/concept/standard.md | 4 +- js/net/bench/fills.ts | 4 +- js/net/src/ietf/fetch.test.ts | 195 +---- js/net/src/ietf/publisher.test.ts | 215 +---- js/net/src/ietf/publisher.ts | 399 ++------- rs/moq-net/benches/session/mod.rs | 48 +- rs/moq-net/src/fuzz.rs | 7 - rs/moq-net/src/ietf/publisher.rs | 785 ++++-------------- rs/moq-net/src/ietf/subscriber.rs | 131 +-- rs/moq-net/src/model/resume.rs | 37 - rs/moq-net/src/model/track.rs | 280 ------- rs/moq-net/src/model/track/fill.rs | 40 - rs/moq-net/src/test_interop.rs | 8 +- rs/moq-net/tests/integration/route_change.rs | 23 +- .../tests/integration/subscribe_ranges.rs | 25 +- rs/moq-net/tests/integration/warm_backfill.rs | 41 +- 16 files changed, 389 insertions(+), 1853 deletions(-) diff --git a/doc/concept/standard.md b/doc/concept/standard.md index 670612e71b6..69fae436f31 100644 --- a/doc/concept/standard.md +++ b/doc/concept/standard.md @@ -28,13 +28,13 @@ subset drops. What a peer actually observes against this implementation: - **Pull.** Subscribers ask. Single-track `PUBLISH` offers are declined, unless they answer our `SUBSCRIBE_TRACKS` (below); announce a namespace and serve the resulting subscriptions. Announcements go out unsolicited. A client asks only for the prefixes its readers watch, and a server or relay link for every prefix it may discover. We never hold two overlapping requests at a peer that did not declare solicit: a broader one replaces the narrower ones once they are closed. The solicit `SETUP` option makes us wait to be asked. Without it, a peer also gets each match as a `NAMESPACE` on its `SUBSCRIBE_NAMESPACE` stream, so it hears the namespace twice. We send `PUBLISH` only to answer a `SUBSCRIBE_TRACKS` (below). - **`SUBSCRIBE_TRACKS` lists, it never pushes.** It lists every track of every broadcast under its prefix, later ones included. With `FORWARD=0`, each track gets a paused `PUBLISH` carrying its Track Properties, and a `PUBLISH_DONE` once the track is no longer listed: `TRACK_ENDED` if it finished, even just before its broadcast ended, and an error otherwise, since the track may go on elsewhere. A `REQUEST_UPDATE` with `FORWARD=1` resumes one, served from the cache like a `SUBSCRIBE` with the update's filter and answered with `LARGEST_OBJECT`; `FORWARD=0` pauses it again. Each paused `PUBLISH` opens only as the peer's stream credit admits it, so a peer that grants none holds none. A listing that omits `FORWARD` or sets it to 1 asks for every track's objects: on draft 22 each track gets `PUBLISH_SKIPPED`, and the subscriber `SUBSCRIBE`s to what it wants; draft 18 has no such skip, so the request is refused with `NOT_SUPPORTED`. A prefix overlapping another listing gets `PREFIX_OVERLAP`, a `TRACK_PROPERTY_FILTER` gets `NOT_SUPPORTED`, and a `REQUEST_UPDATE` on the listing itself ends it. On draft 18, a `SUBSCRIBE` to a track we `PUBLISH`ed gets `DUPLICATE_SUBSCRIPTION`. Cancelling the listing stops new `PUBLISH`es, while those sent live on. A broadcast is listed only while our publish grant covers it, from scratch once it does. Rust sends it upstream for a listing (below); JavaScript does not. - **A Rust TRACKS listing sends `SUBSCRIBE_TRACKS` upstream, only on demand.** Listing a broadcast from a moq-transport peer sends `SUBSCRIBE_TRACKS` with `FORWARD=0` while someone lists it, never on connect, announce, or subscribe, so a peer that refuses, resets, or ignores it affects only that listing. Each listed track's paused `PUBLISH` is its upstream copy, which a reader resumes rather than subscribing again. Only a `PUBLISH_DONE` with `TRACK_ENDED` removes a listed track, and it lists as finished; any other end, including the error our own answerer sends for a track that left unfinished, fails the listing, and readers list again. -- **History is fetched in ranges.** In Rust, Object ID gaps must be marked by Prior Object ID Gap properties for a group to be fetchable. A `FETCH` returns a range of groups from the cache, which a relay fills upstream with one `FETCH` per run of missing groups, or on draft 18 the saved prefix of the group a new subscription just joined. The range ends at Largest Object's group and reads it whole, so a `FETCH` of a group still being written waits for it to finish. That departs from draft 22's cap at Largest Object: a cache cannot know an open group's real Largest Object, and a cut group would be cached downstream as complete. On draft 18, `FETCH_OK` End Object 0 covers the entire End Group, matching the request; Rust also accepts the legacy answer naming the next group with Object 0. On draft 22, `FETCH_OK`'s End Location is the requested end, capped at Largest Object's group: a range holding no objects gets an empty fetch stream, and only a start past Largest Object is refused, with `INVALID_RANGE`. A Rust relay retains the upstream Largest Object from `TRACK_STATUS`, so even a fetch-only copy can resolve a draft 22 `LOCATION_FILTER` relative to it; without a publisher report or live feed, it refuses that filter as `NOT_SUPPORTED`. JavaScript publishing answers a `FETCH` from its cache alone, and a joining `FETCH` only for the head of its subscription's own group. It honors `FILL_TIMEOUT` on a `FETCH` or a fill as a total budget for waiting on objects that have not arrived, zero meaning cache only; once it runs out, the rest of each group still arriving is marked an Unknown gap on draft 18 and a Timed-Out gap on draft 22, never left implying the objects do not exist. It bounds the wait, not how old a cached object is. Rust still refuses `FILL_TIMEOUT` with `NOT_SUPPORTED`. Datagrams are never fetchable. A Rust reader that only fetches learns the track from `TRACK_STATUS` instead of subscribing, so a finished track stays fetchable; a reader whose publisher refuses `TRACK_STATUS` as `NOT_SUPPORTED` still subscribes. +- **History is served one group at a time.** Our moq-transport publishers answer a `FETCH` only within one group, a draft-18 joining `FETCH` for the subscription's current group head, and a draft-22 fill for the largest group's head. Whole-group fetches wait for an open group to finish. Both implementations accept and ignore `FILL_TIMEOUT`, including zero. Datagrams are never fetchable. Subscribers still request ranges upstream, one per run of missing groups; a third-party publisher can answer those ranges and mark their gaps. A Rust reader that only fetches learns the track from `TRACK_STATUS`, so a finished track stays fetchable; a reader whose publisher refuses `TRACK_STATUS` as `NOT_SUPPORTED` still subscribes. - **Compression.** Rust peers exchange compressed tracks when both offer the [Flate extension](/draft/moq-flate). Other peers receive one shared inflated copy. A publisher that seals the compression declaration makes that fallback unavailable: an incompatible peer is refused, preserving the sealed metadata. JavaScript does not offer the IETF extension. - **Timing.** A track whose `SUBSCRIBE_OK` declares no `TIMESCALE` is untimed. A Rust reader that never subscribed takes the units from `TRACK_STATUS`. - **Strict SETUP and parameters.** Repeated unknown `SETUP` options, GREASE included, are accepted if well-formed; a repeated known option is rejected. A `GROUP_ORDER` other than Ascending (1) or Descending (2) closes the session. An unknown message parameter, or one defined only for another message, closes the session. Other malformed control input closes the session with the draft's code too, such as a server `SETUP` carrying `PATH` or `AUTHORITY`, a namespace field ending in `/` (it would map to a path with an empty segment), a request cut short by FIN, or a second `GOAWAY` on one stream; a first `GOAWAY` on a request stream is legal. - **One `SETUP` credential**, forwarded to the [auth server](/bin/relay/auth#the-contract) unverified. A second `SETUP` token closes the session. We advertise no token cache, so an alias reference in `SETUP` closes it with `PROTOCOL_VIOLATION`, one on a request with `UNKNOWN_AUTH_TOKEN_ALIAS`, and a registration on a request with `AUTH_TOKEN_CACHE_OVERFLOW`. Where the Rust side authorizes the peer or verifies tokens, a token on a `SUBSCRIBE` or `PUBLISH_NAMESPACE` is verified like an in-band session token and authorizes that one request alone, even where that authorization already covers it; it never widens the session. It is refused with `NOT_SUPPORTED` when nothing verifies it, when it is empty, or when its Token Type is anything but 0 (out of band). A side that does neither, JavaScript included, ignores the token. Rust refuses a request carrying more than one distinct token with `NOT_SUPPORTED`, since it verifies one; JavaScript, which verifies none, ignores them all. A token-authorized `SUBSCRIBE` still stays within the grant the peer gave us. A token-authorized `PUBLISH_NAMESPACE` needs a grant covering everything beneath the namespace, and the tracks we subscribe under it rely on that announce instead of our limit, ending with it. An update renews the token, and a refused renewal ends only that request. Rust and JavaScript send the token a route names (`Route::auth`) on its `PUBLISH_NAMESPACE`, and swapping it renews the request with `REQUEST_UPDATE`; the inline `NAMESPACE` that answers a solicited `SUBSCRIBE_NAMESPACE` carries no token. A token added to a reconnecting connection's auth handle rides every session it makes, so the announcement keeps it across a reconnect; one added to a single session's handle ends with that session. A token on any other request is decoded and grants nothing, and a request repeating one is refused with `MALFORMED_AUTH_TOKEN`. The negotiated [MoQ Auth extension](/concept/moq-lite#authorization) accepts additional session tokens in band. - **Pause delivery.** `FORWARD=0` on a `SUBSCRIBE` or its update drops the subscription, closing its data streams and holding nothing upstream, while the request and its slot stay. `FORWARD=1` subscribes again as a new `SUBSCRIBE` with the same filter would, so nothing carries across the pause: its `REQUEST_OK` names the live edge it joined at. A paused update may change only priority and max age. -- **Refused, not fatal.** A legal request this stack does not serve is rejected on its own and the session stays up: the fetch forms above get `NOT_SUPPORTED`. Range Filters get `INVALID_FILTER` on draft 22 because the advertised filter limit is zero. On draft 22, namespaces whose first tuple field is exactly `.` or `.session` get `DOES_NOT_EXIST` before application lookup; other dot prefixes still reach the application. Subscription updates accept priority, negotiated subscriber max age, and delivery timeouts. Rust also carries range changes, keeps bounded subscriptions updatable, renews request tokens, and judges `NEW_GROUP_REQUEST` as a join (see dynamic groups below). On draft 22, widening fills cached history on a separate fetch stream, completing earlier groups and handing the newest head to its live subgroup tail; fills follow the requested group order. Other supported-but-unimplemented preferences end that subscription; a `TRACK_PROPERTY_FILTER` there closes the session with `PROTOCOL_VIOLATION`, since it belongs only to `SUBSCRIBE_TRACKS`. A namespace update may change its prefix; suffixes after the acknowledgement are relative to that new prefix. +- **Refused, not fatal.** A legal request this stack does not serve is rejected on its own and the session stays up: the fetch forms above get `NOT_SUPPORTED`. Range Filters get `INVALID_FILTER` on draft 22 because the advertised filter limit is zero. On draft 22, namespaces whose first tuple field is exactly `.` or `.session` get `DOES_NOT_EXIST` before application lookup; other dot prefixes still reach the application. Subscription updates accept priority, negotiated subscriber max age, and delivery timeouts. Rust also carries range changes, keeps bounded subscriptions updatable, renews request tokens, and judges `NEW_GROUP_REQUEST` as a join (see dynamic groups below). On draft 22, a wider or historical fill fails on its separate fetch stream while its subscription stays open; a largest-group fill hands its head to the live subgroup tail. Other supported-but-unimplemented preferences end that subscription; a `TRACK_PROPERTY_FILTER` there closes the session with `PROTOCOL_VIOLATION`, since it belongs only to `SUBSCRIBE_TRACKS`. A namespace update may change its prefix; suffixes after the acknowledgement are relative to that new prefix. - **Objects and datagrams.** Only subgroup 0 is accepted: a stream for any other subgroup, whether its header gives the ID or takes it from the first Object ID, is stopped and the session stays up. Rust relays preserve Object IDs, gaps, object properties, FIRST\_OBJECT, and END\_OF\_GROUP status objects between moq-transport peers for subgroup 0, including across a moq-lite-07 hop between two relays that negotiate the [ietf](/draft/moq-ietf) extension. Datagrams retain their IDs, properties, priority, and status. Plain Rust readers and moq-lite peers without the extension see consecutive frames and only normal datagram payloads at Object ID 0; other datagram objects and statuses are dropped on those surfaces. JavaScript supports only normal datagrams at Object ID 0. A malformed datagram closes the session; a relayed one the downstream draft cannot carry is dropped. A relayed Timestamp leaves in the units the downstream response declares, and is dropped where it declares none, as for a plain frame; an immutable one that would change or vanish is refused. On a timed track the frame holds each object's time, so a mutable Timestamp that only repeats the immutable one is not restated; a status object has no frame, so it keeps its Timestamp among its properties. Only property 0x10 is read as a Timestamp. Property 0x06 stays opaque and keeps its value regardless of timestamp units; it is dropped between drafts 18 and 22, or refused when immutable, because draft 22 assigns it to SUBGROUP\_DELIVERY\_TIMEOUT. A status object carrying properties is refused. A `FETCH` has no status objects, so its response ends at the last frame. A group cannot resume across route changes by frame ordinal once Object IDs skip ahead of the ordinals on either route; recovery refuses it instead of splicing different objects. - **Dynamic groups.** A track advertises `DYNAMIC_GROUPS` when its publisher sets a refresh ratio, and a track received with `DYNAMIC_GROUPS` gets a ratio of 0. A `NEW_GROUP_REQUEST` counts as one more subscriber joining, which a `SUBSCRIBE` already is: it asks for a new group only when the newest group's bytes after its first frame exceed the ratio times the first frame. Joins coalesce until the asked-for group arrives. A Rust relay sends `NEW_GROUP_REQUEST` upstream, on the `SUBSCRIBE` or in a `REQUEST_UPDATE`, once a join wants a group past the newest it holds; a relay holding nothing yet asks for none. On a track without a ratio, an update carrying one ends the subscription with `UPDATE_FAILED`. A moq-lite-07 hop (`moq-lite-07-wip`, opt-in) carries the same demand natively: the ratio in `TRACK_INFO` and the asked-for group in `SUBSCRIBE_UPDATE`. Earlier moq-lite versions carry neither, so a track received over one never refreshes. - **Priority.** Higher is served first. This stack's priorities run from -128 to 127 with 0 unset, and go out on the IETF wire, which ranks lower first, as `127 - priority`: an unset priority is IETF 127. An omitted subscriber or publisher priority reads as the drafts' default of IETF 128, one step below unset, so a relay forwards it unchanged. A relay's upstream subscription carries the highest priority its subscribers ask for and updates it as they come and go; its group order stays as the `SUBSCRIBE` asked, since no draft lets an update change it. diff --git a/js/net/bench/fills.ts b/js/net/bench/fills.ts index b93ab017bac..df3a607ab86 100644 --- a/js/net/bench/fills.ts +++ b/js/net/bench/fills.ts @@ -64,7 +64,7 @@ for (const groups of [2, 16, 128]) { const id = BigInt((round + 1) * 2); const parameters = new Parameters(); parameters.fillParameters = Filter.encodeFill( - { filter: { kind: "unfiltered" }, groupOrder: "ascending", rangeFilters: false }, + { filter: { kind: "relative", groups: 1n }, groupOrder: "ascending", rangeFilters: false }, version, ); await client.writer.u53(SubscribeUpdate.id); @@ -77,7 +77,7 @@ for (const groups of [2, 16, 128]) { if ((await reader.u53()) !== FetchHeader.type) throw new Error("wrong fill stream"); if ((await FetchHeader.decode(reader, version)).requestId !== id) throw new Error("wrong fill ID"); - if ((await reader.readAll()).byteLength < groups) throw new Error("short fill"); + if ((await reader.readAll()).byteLength === 0) throw new Error("short fill"); }), ); if (round > 1) samples.push((performance.now() - start) * 1000); diff --git a/js/net/src/ietf/fetch.test.ts b/js/net/src/ietf/fetch.test.ts index 49483a43678..2636bbd67b9 100644 --- a/js/net/src/ietf/fetch.test.ts +++ b/js/net/src/ietf/fetch.test.ts @@ -121,19 +121,6 @@ function at(group: number, object: number): Location { return { group: BigInt(group), object: BigInt(object) }; } -/** The last Object ID a varint holds, where a gap ends in a group whose end is unknown. */ -const MAX_OBJECT = 2n ** 62n - 1n; - -/** An End of Range as {@link readObjects} reports it, closing at `group`/`object`. */ -function gap(group: number, object: number | bigint, reason: string): [number, number, string] { - return [group, Number(object), `end ${reason}`]; -} - -/** What a FILL_TIMEOUT that ran out leaves: Unknown before draft-22 and Timed-Out from it. */ -function timedOut(version: IetfVersion): string { - return version === D18 ? "10c" : "20c"; -} - /** A manual clock installed until disposed, so no wait can lapse on its own. */ function manual(): Manual & Disposable { const clock = new Manual(); @@ -320,18 +307,17 @@ test("a FETCH stops at its end object", async () => { } }); -// The groups that do not exist are left out, and the rest go out in the order asked for. -test("a FETCH serves several groups in order", async () => { +test("a FETCH spanning groups is refused before FETCH_OK", async () => { for (const version of VERSIONS) { for (const order of ["ascending", "descending"] as const) { using fx = serve(version); publishPairs(fx.track, 5, 2); - const type = range(version, at(0, 1), at(3, 1)); - const answer = served(await fetch(fx, new Fetch({ requestId: FETCH_ID, type, groupOrder: order }))); - expect(answer.ok.endLocation).toEqual(endAt(version, 3, 0)); - const groups = [pairs([0]).slice(1), pairs([1]), pairs([3]).slice(0, 1)]; - if (order === "descending") groups.reverse(); - expect(answer.objects).toEqual(groups.flat()); + for (const end of [3, 9]) { + const type = range(version, at(0, 1), at(end, 0)); + expect(refused(await fetch(fx, new Fetch({ requestId: FETCH_ID, type, groupOrder: order })))).toBe( + NOT_SUPPORTED, + ); + } } } }); @@ -350,14 +336,6 @@ test("a FETCH through the final object reports the end of track", async () => { expect(last.ok.endOfTrack).toBe(true); expect(last.objects).toEqual(pairs([4]).slice(1)); - // Capped at the track's last group, so the End Location names its last object. - const past = served( - await fetch(fx, new Fetch({ requestId: FETCH_ID, type: range(version, at(3, 0), at(9, 0)) })), - ); - expect(past.ok.endLocation).toEqual(endAt(version, 4, 1)); - expect(past.ok.endOfTrack).toBe(true); - expect(past.objects).toEqual(pairs([3, 4])); - // A shorter prefix must not claim the track ends there. const prefix = served( await fetch(fx, new Fetch({ requestId: FETCH_ID, type: range(version, at(4, 0), at(4, 1)) })), @@ -374,7 +352,7 @@ test("a FETCH waits for the Largest Object's open group", async () => { publishPairs(fx.track, 4); const live = writeGroup(fx.track, 4, ["4-0"], false); let answered = false; - const answering = fetch(fx, new Fetch({ requestId: FETCH_ID, type: range(version, at(3, 0), at(9, 0)) })); + const answering = fetch(fx, new Fetch({ requestId: FETCH_ID, type: range(version, at(4, 0), at(4, 0)) })); void answering.finally(() => { answered = true; }); @@ -383,9 +361,9 @@ test("a FETCH waits for the Largest Object's open group", async () => { live.writeFrame({ payload: new TextEncoder().encode("4-1"), timestamp: new Timestamp(1, Timescale.MILLI) }); live.finish(); const answer = served(await answering); - expect(answer.ok.endLocation).toEqual(endAt(version, 4, 1)); + expect(answer.ok.endLocation).toEqual(version === D18 ? at(4, 0) : at(4, 1)); expect(answer.ok.endOfTrack).toBe(false); - expect(answer.objects).toEqual(pairs([3, 4])); + expect(answer.objects).toEqual(pairs([4])); } }); @@ -410,20 +388,13 @@ test("a FETCH the cache cannot answer is refused before FETCH_OK", async () => { }); // Datagrams are never cached, so a FETCH reaching one finds no group there. -test("a FETCH skips a datagram", async () => { +test("a FETCH of a datagram is refused", async () => { for (const version of VERSIONS) { using fx = serve(version); publishPairs(fx.track, 3); expect(fx.track.appendDatagram(new Timestamp(0, Timescale.MILLI), new Uint8Array([1]))).toBe(3); writeGroup(fx.track, 4, ["4-0"]); - const answer = served( - await fetch(fx, new Fetch({ requestId: FETCH_ID, type: range(version, at(1, 0), at(3, 0)) })), - ); - // The End Location names the last object sent rather than promising the datagram. - expect(answer.ok.endLocation).toEqual(endAt(version, 2, 1)); - expect(answer.objects).toEqual(pairs([1, 2])); - const only = new Fetch({ requestId: FETCH_ID, type: range(version, at(3, 0), at(3, 0)) }); expect(refused(await fetch(fx, only))).toBe(DOES_NOT_EXIST); } @@ -432,9 +403,10 @@ test("a FETCH skips a datagram", async () => { // A filter other than a bounded absolute one resolves against Largest Object. test("draft-22: a FETCH filter resolves against Largest Object", async () => { const cases: [Filter.Filter, number[] | number][] = [ - [{ kind: "unfiltered" }, [0, 1, 2]], + [{ kind: "unfiltered" }, NOT_SUPPORTED], [{ kind: "nextObject" }, INVALID_RANGE], - [{ kind: "relative", groups: 2n }, [1, 2]], + [{ kind: "relative", groups: 2n }, NOT_SUPPORTED], + [{ kind: "relative", groups: 1n }, [2]], [{ kind: "absolute", startGroup: 2n, startObject: 0n }, [2]], ]; for (const [filter, expected] of cases) { @@ -488,135 +460,30 @@ test("a cancelled FETCH stops waiting", async () => { } }); -// FILL_TIMEOUT=0 asks for only what is cached. The open group's missing tail is a marked gap -// rather than an end the requester would take for the group's, or a wait. -test("a zero FILL_TIMEOUT serves only what is cached", async () => { - for (const version of VERSIONS) { - using _clock = manual(); - using fx = serve(version); - publishPairs(fx.track, 4); - const live = writeGroup(fx.track, 4, ["4-0"], false); - const type = range(version, at(3, 0), at(9, 0)); - const answer = served(await fetch(fx, new Fetch({ requestId: FETCH_ID, type, fillTimeout: Milli(0) }))); - // Where the open group ends is unknown, so the gap and the End Location reach the last - // Object ID there could be: Object 0 covers the whole group before draft-22. - expect(answer.ok.endLocation).toEqual({ group: 4n, object: version === D18 ? 0n : MAX_OBJECT }); - expect(answer.ok.endOfTrack).toBe(false); - expect(answer.objects).toEqual([...pairs([3]), [4, 0, "4-0"], gap(4, MAX_OBJECT, timedOut(version))]); - - // A range ending inside the group ends its gap there. - const bounded = range(version, at(4, 0), at(4, 3)); - const short = served(await fetch(fx, new Fetch({ requestId: FETCH_ID, type: bounded, fillTimeout: Milli(0) }))); - expect(short.ok.endLocation).toEqual(endAt(version, 4, 2)); - expect(short.objects).toEqual([[4, 0, "4-0"], gap(4, 2, timedOut(version))]); - live.finish(); - } -}); - -// The budget bounds the total wait: what arrives within it is served, and the rest of the -// group becomes a gap once it runs out, even though the group never finishes. -test("a FILL_TIMEOUT bounds the wait and keeps what arrived within it", async () => { - for (const version of VERSIONS) { - using clock = manual(); - using fx = serve(version); - publishPairs(fx.track, 4); - const live = writeGroup(fx.track, 4, ["4-0"], false); - let answered = false; - const type = range(version, at(3, 0), at(9, 0)); - const answering = fetch(fx, new Fetch({ requestId: FETCH_ID, type, fillTimeout: Milli(100) })); - void answering.finally(() => { - answered = true; - }); - await clock.advance(ms(60)); - live.writeFrame({ payload: new TextEncoder().encode("4-1"), timestamp: new Timestamp(1, Timescale.MILLI) }); - // An object arriving does not restart the budget. - await clock.advance(ms(39)); - expect(answered).toBe(false); - await clock.advance(ms(1)); - const answer = served(await answering); - expect(answer.ok.endOfTrack).toBe(false); - expect(answer.objects).toEqual([...pairs([3, 4]), gap(4, MAX_OBJECT, timedOut(version))]); - live.finish(); - } -}); - -// Groups can arrive out of order. The Largest Object's group is read before FETCH_OK and an -// older open one after it, and both waits draw on the one budget. -test("a FILL_TIMEOUT is one budget across an answer's groups", async () => { - for (const version of VERSIONS) { - using clock = manual(); - using fx = serve(version); - const late = writeGroup(fx.track, 2, ["2-0"], false); - const newest = writeGroup(fx.track, 3, ["3-0"], false); - let answered = false; - const type = range(version, at(2, 0), at(9, 0)); - const answering = fetch(fx, new Fetch({ requestId: FETCH_ID, type, fillTimeout: Milli(100) })); - void answering.finally(() => { - answered = true; - }); - await clock.advance(ms(60)); - newest.writeFrame({ payload: new TextEncoder().encode("3-1"), timestamp: new Timestamp(1, Timescale.MILLI) }); - newest.finish(); - await clock.advance(ms(39)); - expect(answered).toBe(false); - await clock.advance(ms(1)); - const answer = served(await answering); - expect(answer.ok.endLocation).toEqual(endAt(version, 3, 1)); - // The object after a gap counts its group from the gap's. - expect(answer.objects).toEqual([[2, 0, "2-0"], gap(2, MAX_OBJECT, timedOut(version)), ...pairs([3])]); - late.finish(); - } -}); - -// A gap covers every Location since the prior object, so a group with nothing to send yet -// first marks the groups skipped before it as what their absence otherwise says: gone. -test("a timed-out group with nothing yet keeps the groups before it nonexistent", async () => { +// FILL_TIMEOUT is accepted without truncating a one-group answer, including zero. +test("FILL_TIMEOUT is ignored while a group finishes", async () => { for (const version of VERSIONS) { - for (const order of ["ascending", "descending"] as const) { - using _clock = manual(); + for (const fillTimeout of [Milli(0), Milli(100)]) { + using clock = manual(); using fx = serve(version); - // Group 1 never existed. - writeGroup(fx.track, 0, ["0-0", "0-1"]); - const empty = writeGroup(fx.track, 2, [], false); - writeGroup(fx.track, 3, ["3-0", "3-1"]); - const type = range(version, at(0, 0), at(3, 0)); - const answer = served( - await fetch(fx, new Fetch({ requestId: FETCH_ID, type, groupOrder: order, fillTimeout: Milli(0) })), + const live = writeGroup(fx.track, 4, ["4-0"], false); + let answered = false; + const answering = fetch( + fx, + new Fetch({ requestId: FETCH_ID, type: range(version, at(4, 0), at(4, 0)), fillTimeout }), ); - expect(answer.ok.endLocation).toEqual(version === D18 ? at(3, 0) : at(3, 1)); - const missing = gap(2, MAX_OBJECT, timedOut(version)); - expect(answer.objects).toEqual( - order === "ascending" - ? [...pairs([0]), gap(1, MAX_OBJECT, "8c"), missing, ...pairs([3])] - : [...pairs([3]), gap(3, MAX_OBJECT, "8c"), missing, ...pairs([0])], - ); - empty.finish(); + void answering.then(() => { + answered = true; + }); + await clock.advance(ms(1000)); + expect(answered).toBe(false); + live.writeFrame({ payload: new TextEncoder().encode("4-1"), timestamp: new Timestamp(1, Timescale.MILLI) }); + live.finish(); + expect(served(await answering).objects).toEqual(pairs([4])); } } }); -// Every FETCH keeps its own budget, even for the same open group, and one without FILL_TIMEOUT, -// as an older peer sends, waits for the group as it always has. -test("each FETCH keeps its own FILL_TIMEOUT", async () => { - for (const version of VERSIONS) { - using clock = manual(); - using fx = serve(version); - publishPairs(fx.track, 4); - const live = writeGroup(fx.track, 4, ["4-0"], false); - const type = range(version, at(4, 0), at(9, 0)); - const waiting = fetch(fx, new Fetch({ requestId: FETCH_ID, type })); - const cached = served(await fetch(fx, new Fetch({ requestId: FETCH_ID + 1n, type, fillTimeout: Milli(0) }))); - expect(cached.objects).toEqual([[4, 0, "4-0"], gap(4, MAX_OBJECT, timedOut(version))]); - - await clock.advance(ms(10_000)); - live.writeFrame({ payload: new TextEncoder().encode("4-1"), timestamp: new Timestamp(1, Timescale.MILLI) }); - live.finish(); - const full = served(await waiting); - expect(full.ok.endLocation).toEqual(endAt(version, 4, 1)); - expect(full.objects).toEqual(pairs([4])); - } -}); - /** Open a draft-18 SUBSCRIBE of `room/video` with `filter`, returning its SUBSCRIBE_OK. */ async function subscribe(fx: Served, filter: Filter.Filter = { kind: "nextObject" }) { const client = await Stream.open(fx.pair.client, { version: fx.version }); diff --git a/js/net/src/ietf/publisher.test.ts b/js/net/src/ietf/publisher.test.ts index f627896e771..2e47e34cf50 100644 --- a/js/net/src/ietf/publisher.test.ts +++ b/js/net/src/ietf/publisher.test.ts @@ -2831,7 +2831,7 @@ test("draft-22: a backwards range within one group serves nothing and ends the s } }); -test.each(["ascending", "descending"] as const)("draft-22: a sparse fill serves cached groups %s", async (order) => { +test.each(["ascending", "descending"] as const)("draft-22: a wider fill is refused in %s order", async (order) => { const fx = fixture(); const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); for (const sequence of [0, 3, 9]) { @@ -2857,12 +2857,18 @@ test.each(["ascending", "descending"] as const)("draft-22: a sparse fill serves const fill = await nextUni(fx.uni); if (!fill) throw new Error("missing fill stream"); const served = await readFill(fill, order); - expect(served.reset).toBeUndefined(); - expect(served.requestId).toBe(7n); - const sequences = order === "ascending" ? [0, 3, 9] : [9, 3, 0]; - expect(served.objects).toEqual( - sequences.flatMap((group) => [0, 1].map((id) => ({ group, id, payload: `${group}.${id}` }))), - ); + expect(served.reset).toBeDefined(); + expect(served.objects).toEqual([]); + const live = track.appendGroup(); + live.writeFrame({ payload: new TextEncoder().encode("live"), timestamp: Timestamp.now() }); + live.finish(); + const objects = []; + while (objects.length === 0) { + const stream = await nextUni(fx.uni); + if (!stream) throw new Error("subscription ended with its refused fill"); + objects.push(...(await readGroup(stream)).objects); + } + expect(objects).toEqual([{ id: 0, payload: "live" }]); } finally { fx.close(); client.close(); @@ -3224,80 +3230,12 @@ test("draft-22: a clean close past a bounded filter's end still sends PUBLISH_DO } }); -/** - * An absolute fill ending below the live edge with no end object reads until its group - * closes, which a group still being written may never do. The subscriber leaving has to end - * it: watching only the fetch stream would pin the group and its cache subscription for the - * life of the track. - */ -test("draft-22: the subscriber leaving ends a fill still reading its group", async () => { - const fx = fixture(); - const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); - - // Group 0 stays open, so a fill over it has no end of its own to wait for. Group 1 puts - // the live edge above it, which is what leaves the requested end object unset. - const open = track.appendGroup(); - open.writeFrame({ payload: new TextEncoder().encode("0.0"), timestamp: Timestamp.now() }); - writeGroup(track, 1); - - const { client } = await runSubscribe( - fx, - new Subscribe({ - requestId: 7n, - trackNamespace: Path.from("test"), - trackName: "video", - subscriberPriority: 0, - filter: { kind: "nextObject" }, - fill: { - filter: { kind: "absolute", startGroup: 0n, startObject: 0n, endGroup: 0n }, - rangeFilters: false, - }, - }), - ); - - try { - const fill = await nextUni(fx.uni); - if (!fill) throw new Error("no fetch stream for the requested fill"); - - // The fill is parked on a group that is never going to close on its own, so this only - // settles once the subscriber leaving cancels it. - const [data, observed] = fill.tee(); - const reading = readFill(data); - // Opening the stream precedes fetching the group. Observe an actual object so the - // cancellation tests a fill reading its open group, even when header writes yield. - const reader = new Reader(observed, undefined, V22); - expect(await reader.u53()).toBe(FetchHeader.type); - await FetchHeader.decode(reader, V22); - const flags = await reader.u53(); - if (flags & 0x08) await reader.u53(); - if (flags & 0x04) await reader.u53(); - if (flags & 0x10) await reader.u8(); - if (flags & 0x20) await reader.read(await reader.u53()); - expect(new TextDecoder().decode(await reader.read(await reader.u53()))).toBe("0.0"); - reader.stop(new Error("observation complete")); - client.close(); - - // A reset discards what the peer has not acknowledged, so the objects already written - // may or may not survive it. That it ends at all, reset as CANCELLED, is the whole point. - const served = await reading; - expect(served.reset).toMatchObject({ code: StreamCode.Cancel }); - } finally { - open.finish(); - fx.close(); - client.close(); - } -}); - -// A fill's FILL_TIMEOUT bounds its wait as a FETCH's does. Zero takes only what is cached, so -// the open group's missing tail is a Timed-Out gap rather than a wait the fill never ends. -test("draft-22: a fill's FILL_TIMEOUT marks what it did not wait for", async () => { +test.each([0, 1000])("draft-22: FILL_TIMEOUT=%s is ignored for a largest-group fill", async (timeout) => { using _clock = manual(); const fx = fixture(); const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); const open = track.appendGroup(); open.writeFrame({ payload: new TextEncoder().encode("0.0"), timestamp: Timestamp.now() }); - writeGroup(track, 1); - const { client } = await runSubscribe( fx, new Subscribe({ @@ -3306,23 +3244,13 @@ test("draft-22: a fill's FILL_TIMEOUT marks what it did not wait for", async () trackName: "video", subscriberPriority: 0, filter: { kind: "nextObject" }, - fill: { - filter: { kind: "absolute", startGroup: 0n, startObject: 0n, endGroup: 0n }, - fillTimeout: Milli(0), - rangeFilters: false, - }, + fill: { filter: { kind: "relative", groups: 1n }, fillTimeout: Milli(timeout), rangeFilters: false }, }), ); - try { const fill = await nextUni(fx.uni); - if (!fill) throw new Error("no fetch stream for the requested fill"); - const served = await readFill(fill); - expect(served.reset).toBeUndefined(); - expect(served.objects).toEqual([ - { group: 0, id: 0, payload: "0.0" }, - { group: 0, id: Number(2n ** 62n - 1n), payload: "end 20c" }, - ]); + if (!fill) throw new Error("missing fill"); + expect(await readFill(fill)).toMatchObject({ objects: [{ group: 0, id: 0, payload: "0.0" }] }); } finally { open.finish(); fx.close(); @@ -4172,7 +4100,7 @@ test("draft-22: an update fills its snapshot under its own request ID", async () const parameters = new Parameters(); parameters.subscriptionFilter = { kind: "nextObject" }; parameters.fillParameters = Filter.encodeFill( - { filter: { kind: "unfiltered" }, groupOrder: "ascending", rangeFilters: false }, + { filter: { kind: "relative", groups: 1n }, groupOrder: "ascending", rangeFilters: false }, fx.version, ); await client.writer.u53(SubscribeUpdate.id); @@ -4193,7 +4121,7 @@ test("draft-22: an update fills its snapshot under its own request ID", async () expect(served.requestId).toBe(2n); expect(served.reset).toBeUndefined(); expect(served.objects).toEqual( - [0, 1, 2].flatMap((group) => [0, 1].map((id) => ({ group, id, payload: `${group}.${id}` }))), + [2].flatMap((group) => [0, 1].map((id) => ({ group, id, payload: `${group}.${id}` }))), ); } finally { client.close(); @@ -4439,9 +4367,9 @@ test("draft-22: widening with a fill keeps the open live group", async () => { if (!live) throw new Error("missing initial open group"); const reading = readGroup(live); const parameters = new Parameters(); - parameters.subscriptionFilter = { kind: "absolute", startGroup: 0n, startObject: 0n }; + parameters.subscriptionFilter = { kind: "absolute", startGroup: 1n, startObject: 0n }; parameters.fillParameters = Filter.encodeFill( - { filter: { kind: "unfiltered" }, groupOrder: "ascending", rangeFilters: false }, + { filter: { kind: "relative", groups: 1n }, groupOrder: "ascending", rangeFilters: false }, fx.version, ); await client.writer.u53(SubscribeUpdate.id); @@ -4450,10 +4378,7 @@ test("draft-22: widening with a fill keeps the open live group", async () => { await RequestOk.decode(client.reader, fx.version); const fill = await nextUni(fx.uni); if (!fill) throw new Error("missing update fill"); - expect((await readFill(fill, "ascending")).objects).toEqual([ - { group: 0, id: 0, payload: "0.0" }, - { group: 1, id: 0, payload: "1.0" }, - ]); + expect((await readFill(fill, "ascending")).objects).toEqual([{ group: 1, id: 0, payload: "1.0" }]); open.writeFrame({ payload: new TextEncoder().encode("1.1"), timestamp: Timestamp.now() }); open.finish(); // The live stream still owns the rest of its group, so nothing else repeats it. @@ -4471,55 +4396,6 @@ test("draft-22: widening with a fill keeps the open live group", async () => { } }); -test("draft-22: widening with a fill resumes after the open live group's end", async () => { - const fx = fixture(); - const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); - writeGroup(track, 1); - const open = track.appendGroup(); - open.writeFrame({ payload: new TextEncoder().encode("1.0"), timestamp: Timestamp.now() }); - const { client } = await runSubscribe( - fx, - new Subscribe({ - requestId: 0n, - trackNamespace: Path.from("test"), - trackName: "video", - subscriberPriority: 0, - filter: { kind: "absolute", startGroup: 1n, startObject: 0n, endGroup: 1n, endObject: 1n }, - }), - ); - try { - const live = await nextUni(fx.uni); - if (!live) throw new Error("missing initial open group"); - const reading = readGroup(live); - const parameters = new Parameters(); - parameters.subscriptionFilter = { kind: "absolute", startGroup: 0n, startObject: 0n }; - parameters.fillParameters = Filter.encodeFill( - { filter: { kind: "unfiltered" }, groupOrder: "ascending", rangeFilters: false }, - fx.version, - ); - await client.writer.u53(SubscribeUpdate.id); - await new SubscribeUpdate({ requestId: 2n, parameters }).encode(client.writer, fx.version); - expect(await client.reader.u53()).toBe(RequestOk.id); - await RequestOk.decode(client.reader, fx.version); - for (const payload of ["1.1", "1.2"]) - open.writeFrame({ payload: new TextEncoder().encode(payload), timestamp: Timestamp.now() }); - open.finish(); - // The live stream keeps the objects up to its old end, so the tail starts after them. - expect((await reading).objects).toEqual([ - { id: 0, payload: "1.0" }, - { id: 1, payload: "1.1" }, - ]); - expect((await readServed(fx.uni, 2)).sort()).toEqual(["0.0", "1.0", "1.2"]); - expect(await nextUni(fx.uni)).toBeUndefined(); - } finally { - open.finish(); - client.close(); - track.finish(); - track.close(); - fx.close(); - } -}); - test("draft-22: a fill-only update short of the edge keeps the live join's floor", async () => { const fx = fixture(); const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); @@ -4549,14 +4425,14 @@ test("draft-22: a fill-only update short of the edge keeps the live join's floor await RequestOk.decode(client.reader, fx.version); const fill = await nextUni(fx.uni); if (!fill) throw new Error("missing update fill"); - expect((await readFill(fill, "ascending")).objects).toEqual([{ group: 0, id: 0, payload: "0.0" }]); + expect((await readFill(fill, "ascending")).reset).toBeDefined(); open.writeFrame({ payload: new TextEncoder().encode("1.1"), timestamp: Timestamp.now() }); open.finish(); expect((await reading).objects).toEqual([ { id: 0, payload: "1.0" }, { id: 1, payload: "1.1" }, ]); - // Only the fill carries group 0; the live cursor never owned it. + // The refused fill never changes the live cursor. expect(await nextUni(fx.uni)).toBeUndefined(); } finally { open.finish(); @@ -4629,13 +4505,13 @@ test.each([ const reading = readGroup(live).then(() => undefined, error); for (const [requestId, startGroup, startObject] of [ [2n, group, object], - [4n, 0n, 0n], + [4n, 1n, 0n], ] as const) { const parameters = new Parameters(); parameters.subscriptionFilter = { kind: "absolute", startGroup, startObject }; - if (startGroup === 0n) + if (requestId === 4n) parameters.fillParameters = Filter.encodeFill( - { filter: { kind: "unfiltered" }, groupOrder: "ascending", rangeFilters: false }, + { filter: { kind: "relative", groups: 1n }, groupOrder: "ascending", rangeFilters: false }, fx.version, ); await client.writer.u53(SubscribeUpdate.id); @@ -4646,7 +4522,7 @@ test.each([ expect(await reading).toMatchObject({ code: StreamCode.Cancel }); open.writeFrame({ payload: new TextEncoder().encode("1.1"), timestamp: Timestamp.now() }); open.finish(); - expect((await readServed(fx.uni, 2)).sort()).toEqual(["0.0", "1.0", "1.1"]); + expect((await readServed(fx.uni, 2)).sort()).toEqual(["1.0", "1.1"]); } finally { open.finish(); client.close(); @@ -4672,38 +4548,3 @@ async function readServed( } return served; } - -test("subscriber max age and REQUEST_UPDATE preserve omitted preferences", async () => { - const fx = fixture(Version.DRAFT_18); - const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); - const { client } = await runSubscribe( - fx, - new Subscribe({ - requestId: 0n, - trackNamespace: Path.from("test"), - trackName: "video", - subscriberPriority: 128, - maxAge: 250, - }), - ); - try { - expect(track.subscription.peek()?.maxDelay).toBe(Milli(250)); - const parameters = new Parameters(); - parameters.maxAge = 900; - await client.writer.u53(SubscribeUpdate.id); - await new SubscribeUpdate({ requestId: 2n, parameters }).encode(client.writer, fx.version); - expect(await client.reader.u53()).toBe(RequestOk.id); - await RequestOk.decode(client.reader, fx.version); - expect(track.subscription.peek()?.maxDelay).toBe(Milli(900)); - await client.writer.u53(SubscribeUpdate.id); - await new SubscribeUpdate({ requestId: 4n }).encode(client.writer, fx.version); - expect(await client.reader.u53()).toBe(RequestOk.id); - await RequestOk.decode(client.reader, fx.version); - expect(track.subscription.peek()?.maxDelay).toBe(Milli(900)); - } finally { - client.close(); - track.finish(); - track.close(); - fx.close(); - } -}); diff --git a/js/net/src/ietf/publisher.ts b/js/net/src/ietf/publisher.ts index 77cd37cccc0..fe8eaa09b9a 100644 --- a/js/net/src/ietf/publisher.ts +++ b/js/net/src/ietf/publisher.ts @@ -1,4 +1,4 @@ -import { Instant, Milli, Nano, now, sleep, sleepUntil } from "@moq/clock"; +import { Instant, Milli, Nano, now, sleep } from "@moq/clock"; import { type Dispose, type Getter, Once, race, Signal } from "@moq/signals"; import type { Credential, Grant } from "../auth.ts"; import { Gate } from "../auth_gate.ts"; @@ -29,7 +29,7 @@ import * as Path from "../path.ts"; import { Priority, sendOrder } from "../priority.ts"; import { type Stream, Writer } from "../stream.ts"; import { Timescale } from "../time.ts"; -import type { Groups, Info, Order, Position as TrackPosition, Subscriber as TrackSubscriber } from "../track.ts"; +import type { Groups, Info, Position as TrackPosition, Subscriber as TrackSubscriber } from "../track.ts"; import { untilAborted } from "../util/abort.ts"; import { nextChange } from "../util/change.ts"; import { Deadline, REQUEST_TIMEOUT_MS, requestOptions } from "../util/deadline.ts"; @@ -45,7 +45,7 @@ import * as Filter from "./filter.ts"; import { MAX_AGE_PARAMETER } from "./max_age.ts"; import * as Message from "./message.ts"; import * as Namespace from "./namespace.ts"; -import { EndOfRange, FetchEndOfRange, FetchFrame, Frame, Group as GroupMessage } from "./object.ts"; +import { FetchFrame, Frame, Group as GroupMessage } from "./object.ts"; import { Parameters } from "./parameters.ts"; import { DEFAULT_WIRE, fromWire, toWire } from "./priority.ts"; import * as Properties from "./properties.ts"; @@ -758,7 +758,7 @@ export class Publisher { const edge = liveEdge(track); const fill = msg.fill && msg.forward && Filter.isDraft22(version) - ? fillRange(msg.fill, msg.filter, edge.largest, order) + ? fillRange(msg.fill, msg.filter, edge.largest) : undefined; const requested = subscribeRange(msg, edge, version); // A copy, since an update moves the cursor in place and must not move the bounds too. @@ -919,7 +919,7 @@ export class Publisher { this.#applyPublished(msg, params); const selected = msg.fill && Filter.isDraft22(version) - ? fillRange(msg.fill, msg.filter, snapshot.largest, msg.groupOrder) + ? fillRange(msg.fill, msg.filter, snapshot.largest) : undefined; if (params.subscriptionFilter !== undefined || selected) { // An update's unfiltered range explicitly lowers the floor to zero; @@ -1405,8 +1405,7 @@ export class Publisher { } /** - * Serve a draft-22 fill on its own fetch stream: the requested range, read from the - * group cache, capped at the Largest Object snapshot. + * Serve the largest group's head on its own draft-22 fetch stream. * * A fill is a promise once requested. An empty range opens no stream, but a range we * cannot serve still opens one and resets it right after the FETCH_HEADER, the draft's @@ -1415,7 +1414,6 @@ export class Publisher { async #runFill(options: RunFill) { const { requestId, ranking, fill, cache, timescale, unsubscribed, cancelled, streams } = options; const version = this.#session.version; - const deadline = fillDeadline(fill.kind === "range" ? fill.timeout : undefined); // Everything is inside the try so the cache fork is released on every path out, // including a transport failure while opening the stream. @@ -1424,11 +1422,8 @@ export class Publisher { if (fill.kind === "empty") return; // A fill we cannot serve is reset right after its header, so it needs no rank. - const sequence = - fill.kind === "range" - ? Number(fill.order === "ascending" ? fill.start.group : fill.end.group) - : undefined; - const object = fill.kind === "range" ? Number(fill.start.object) : 0; + const sequence = fill.kind === "group" ? Number(fill.start.group) : undefined; + const object = fill.kind === "group" ? Number(fill.start.object) : 0; stream = await Writer.tryOpen(this.#quic, { cancel: unsubscribed, version, @@ -1443,31 +1438,17 @@ export class Publisher { await untilAborted(new FetchHeader({ requestId }).encode(stream, version), cancelled); - if (fill.kind !== "range") { - throw new Error("fill range filters are not supported"); + if (fill.kind !== "group") { + throw new Error("fill is not supported"); } - const sequences = cachedSequences(cache, rangeGroups(fill), options.priority); - if (fill.order === "descending") sequences.reverse(); - const out = new FetchWriter({ - stream, - version, - timescale, - order: fill.order, - start: - fill.order === "ascending" || fill.start.group === fill.end.group - ? fill.start - : { group: fill.end.group, object: 0n }, - }); - for (const sequence of sequences) { - const group = hooks.cachedGroup(cache, sequence); - if (!group) continue; - const slice = groupSlice(fill, sequence); - try { - await this.#writeFillGroup({ out, group, slice, deadline, cancelled }); - } finally { - group.close(); - } + const out = new FetchWriter({ stream, version, timescale }); + const group = hooks.cachedGroup(cache, Number(fill.start.group)); + if (!group) throw new NotFound("fill group unavailable"); + try { + await this.#writeFillGroup({ out, group, slice: groupSlice(fill, group.sequence), cancelled }); + } finally { + group.close(); } stream.close(); @@ -1488,8 +1469,7 @@ export class Publisher { } /** - * Write one cached group's frames as fetch objects, then an End of Range for the rest of - * its slice if the FILL_TIMEOUT `deadline` ran out first. + * Write the largest group's cached head as fetch objects. * * The cap is the Largest Object snapshot: the group may keep growing, but everything * past the snapshot belongs to the subscription, not the fill. @@ -1498,13 +1478,11 @@ export class Publisher { out, group, slice, - deadline, cancelled, }: { out: FetchWriter; group: group.Consumer; slice: GroupSlice; - deadline: Instant | undefined; cancelled: AbortSignal; }): Promise { let next = slice.skip; @@ -1519,11 +1497,7 @@ export class Publisher { if (slice.until !== undefined && next >= slice.until) break; // Reading from the fill's start drops everything below it; see the same read in #runGroup. - const frame = await nextFrame(group, slice.skip, deadline, cancelled, out.stream.closed); - if (frame === TIMED_OUT) { - await out.timedOut(group.sequence, slice, cancelled); - break; - } + const frame = await nextFrame(group, slice.skip, cancelled, out.stream.closed); if (!frame) break; next = frame.sequence + 1; if (slice.until !== undefined && frame.sequence >= slice.until) break; @@ -2907,8 +2881,7 @@ export class Publisher { async #runFetch(msg: Fetch, request: RequestStream) { this.#pinned(msg.epoch); - // The FILL_TIMEOUT counts from the request's arrival, whatever it waits on. - const deadline = fillDeadline(msg.fillTimeout); + // FILL_TIMEOUT is accepted and ignored: a one-group read ends with the group. const version = this.#session.version; const type = msg.type; const named = type.kind === "standalone" || type.kind === "filtered" ? type : undefined; @@ -2977,8 +2950,8 @@ export class Publisher { let answer: FetchAnswer | Refusal | undefined; try { try { - if (named) answer = await this.#fetchRange(msg, named, deadline, stop.signal); - else if (joining && join) answer = await this.#fetchJoin(joining, join, deadline, stop.signal); + if (named) answer = await this.#fetchRange(msg, named, stop.signal); + else if (joining && join) answer = await this.#fetchJoin(joining, join, stop.signal); else throw new Error("unknown FETCH type"); } catch (err: unknown) { if (stop.signal.reason === cancel) { @@ -2997,7 +2970,7 @@ export class Publisher { return; } resolved = true; - await this.#answerFetch(msg, request, answer, deadline, stop.signal); + await this.#answerFetch(msg, request, answer, stop.signal); } finally { lapse?.(); watch?.dispose(); @@ -3010,15 +2983,12 @@ export class Publisher { * Resolve a standalone or draft-22 FETCH against its track and read the highest group it * holds, or the refusal to answer with. * - * The range ends at the Largest Object's group, which is read whole even while it is - * open: the cache does not know an open group's real Largest Object, and a downstream - * cache would take a cut group as complete. So the answer waits for it to finish, or - * until the FILL_TIMEOUT `deadline`, after which the rest of it is a marked gap. + * A whole-group request waits for the group to finish, so a downstream cache never + * mistakes a prefix for a complete group. */ async #fetchRange( msg: Fetch, named: Extract, - deadline: Instant | undefined, signal: AbortSignal, ): Promise { const version = this.#session.version; @@ -3050,6 +3020,9 @@ export class Publisher { return { condition: "invalid_range", reasonPhrase: "empty range" }; } + if (!relative && start.group !== end.group) + return { condition: "not_supported", reasonPhrase: "FETCH spans more than one group" }; + const broadcast = await this.#resolve(named.trackNamespace, signal, msg.epoch); const priority = fromWire(msg.subscriberPriority); const cache = broadcast.track(named.trackName).subscribe({ priority, maxDelay: Milli(Varint.MAX_U53) }); @@ -3067,9 +3040,11 @@ export class Publisher { }; } range.start = fetchStart(relative, largest); - // Past every group, so the cap below ends it with the Largest Object's group. - range.end = { group: MAX_GROUP, object: 0n }; + range.end = { group: largest.group, object: 0n }; } + if (range.start.group !== range.end.group) + return { condition: "not_supported", reasonPhrase: "FETCH spans more than one group" }; + if (largest) { // A start past Largest Object in a group the cache holds is refused INVALID_RANGE // (draft-22 section 9.11), rather than waiting on the live group. A group past @@ -3079,29 +3054,16 @@ export class Publisher { ? { condition: "invalid_range", reasonPhrase: "start past Largest Object" } : { condition: "does_not_exist", reasonPhrase: `group ${range.start.group}` }; } - if (range.end.group > largest.group) range.end = { group: largest.group, object: 0n }; } - const groups = cachedSequences( - cache, - { start: { included: safe(range.start.group) }, end: { included: safe(range.end.group) } }, - priority, - ); - // A group gone or never cached is absent from the answer, as the response skips it, - // unless it is the only one. - let last: FetchedGroup | undefined; - while (!last) { - const sequence = groups.pop(); - if (sequence === undefined) return { condition: "does_not_exist", reasonPhrase: "no group in range" }; - const group = hooks.cachedGroup(cache, sequence); - if (!group) continue; - try { - last = await readFetched(group, fetchSlice(range, sequence), deadline, signal); - } catch (err) { - if (signal.aborted || groups.length === 0) throw err; - } finally { - group.close(); - } + const selected = safe(range.start.group); + const group = hooks.cachedGroup(cache, selected); + if (!group) return { condition: "does_not_exist", reasonPhrase: `group ${range.start.group}` }; + let last: FetchedGroup; + try { + last = await readFetched(group, fetchSlice(range, selected), signal); + } finally { + group.close(); } // Only a finished track's final object ends it, never a live track's Largest Object. @@ -3114,25 +3076,8 @@ export class Publisher { const groupEnd = BigInt(last.end); const until = fetchSlice(range, last.sequence).until; let endLocation: Location; - if (last.timedOut) { - // The gap that ends the answer runs to the end of the slice, which the End - // Location has to cover. Where an open group ends is unknown, so a whole-group - // slice reaches the last Object ID there could be: Object 0 says that before - // draft-22. - endLocation = { - group: sequence, - object: - until !== undefined - ? BigInt(Filter.isDraft22(version) ? until - 1 : until) - : Filter.isDraft22(version) - ? MAX_OBJECT - : 0n, - }; - } else if (Filter.isDraft22(version)) { - // Draft-22's End Location is inclusive: the end, already capped at the Largest - // Object's group (section 10.14). Objects missing before it do not exist. A - // whole-group end has no End Object to report, so it covers what the finished - // group holds, and at least the start an empty answer covered. + if (Filter.isDraft22(version)) { + // Draft-22's End Location is inclusive. An empty answer still covers its start. const object = until === undefined ? max(saturatingSub(groupEnd, 1n), BigInt(fetchSlice(range, last.sequence).skip)) @@ -3140,17 +3085,8 @@ export class Publisher { ? min(BigInt(until) - 1n, saturatingSub(groupEnd, 1n)) : BigInt(until) - 1n; endLocation = { group: sequence, object }; - } else if ( - (endOfTrack && range.end.object > 0n) || - sequence < range.end.group || - range.end.group !== end.group || - range.end.object !== end.object - ) { - // Cap a bounded response at the track's last object, or at the last object sent - // when the end group is not held or the range was capped: before draft-22 the - // End Location promises every object before it. A whole-group response to a - // request for that group keeps Object 0 rather than claiming that the next group - // is covered. + } else if (endOfTrack && range.end.object > 0n) { + // A bounded response cannot promise objects past the track's end. endLocation = { group: sequence, object: groupEnd }; } else { endLocation = range.end; @@ -3160,14 +3096,12 @@ export class Publisher { cache, owned: true, range, - groups, last, ok: new FetchOk({ endOfTrack, endLocation, properties: trackProperties(info, msg.propertiesWanted) }), // A Timestamp goes out only when FETCH_OK actually carries TIMESCALE. timescale: msg.propertiesWanted ? (info.timescale ?? undefined) : undefined, // Its own rank, so it competes with group streams by its priority alone. sendOrder: sendOrder({ priority }), - order: msg.groupOrder ?? "ascending", }; return answer; } finally { @@ -3218,14 +3152,15 @@ export class Publisher { async #fetchJoin( type: Extract, join: Joinable, - deadline: Instant | undefined, signal: AbortSignal, ): Promise { if (join.kind === "unsupported") return { condition: "not_supported", reasonPhrase: "joining filter not supported" }; const largest = join.largest; if (!largest) return { condition: "invalid_range", reasonPhrase: "no objects at subscription start" }; - const start = type.kind === "relativeJoining" ? saturatingSub(largest.group, type.offset) : type.group; + if (type.kind === "relativeJoining" && type.offset !== 0n) + return { condition: "not_supported", reasonPhrase: "joining FETCH reaches before the joined group" }; + const start = type.kind === "relativeJoining" ? largest.group : type.group; if (start > largest.group) return { condition: "invalid_range", reasonPhrase: "joining group past the subscription" }; // The groups before the subscription's are a standalone FETCH's to ask for. @@ -3239,7 +3174,7 @@ export class Publisher { if (!group) return unavailable; let last: FetchedGroup; try { - last = await readFetched(group, { skip: 0, until: Number(end.object) }, deadline, signal); + last = await readFetched(group, { skip: 0, until: Number(end.object) }, signal); } catch (err) { if (signal.aborted) throw err; return unavailable; @@ -3247,16 +3182,13 @@ export class Publisher { group.close(); } // The subscription starts right after the saved Largest Object, so the head has to be - // here in full or the two leave a gap, unless the FILL_TIMEOUT marks the rest of a - // prefix as one. Frames read are consecutive, so a full count starts at object 0. - const whole = last.frames.length === Number(end.object); - const prefix = last.timedOut && (last.frames[0]?.sequence ?? 0) === 0; - if (!whole && !prefix) return unavailable; + // here in full or the two leave a gap. Frames read are consecutive, so a full count + // starts at object 0. + if (last.frames.length !== Number(end.object)) return unavailable; return { cache: join.track, owned: false, range: { start: { group: end.group, object: 0n }, end }, - groups: [], last, // Where the subscription starts, never the end of the track. ok: new FetchOk({ endOfTrack: false, endLocation: end, properties: join.properties }), @@ -3264,26 +3196,16 @@ export class Publisher { // Ranked as the group it heads, ahead of that group's live rest. sendOrder: join.ranking.rank(sequence, 0), ranking: join.ranking, - order: "ascending", }; } /** - * Write FETCH_OK, then the answer's groups on one fetch stream named by the FETCH's own - * request ID, in the order asked for. Nothing past FETCH_OK can be refused, so a failure + * Write FETCH_OK, then the group on a fetch stream named by the FETCH's own request ID. Nothing past FETCH_OK can be refused, so a failure * resets both streams instead. */ - async #answerFetch( - msg: Fetch, - request: RequestStream, - answer: FetchAnswer, - deadline: Instant | undefined, - stop: AbortSignal, - ) { + async #answerFetch(msg: Fetch, request: RequestStream, answer: FetchAnswer, stop: AbortSignal) { const version = this.#session.version; - const { cache, range, last, timescale, ranking } = answer; - const order = [...answer.groups, last.sequence]; - if (answer.order === "descending") order.reverse(); + const { range, last, timescale, ranking } = answer; let stream: Writer | undefined; try { @@ -3299,34 +3221,8 @@ export class Publisher { ranking?.add(stream, Number(range.start.group), Number(range.start.object)); await untilAborted(new FetchHeader({ requestId: msg.requestId }).encode(stream, version), stop); - const end = answer.ok.endLocation.group; - const out = new FetchWriter({ - stream, - version, - timescale, - order: answer.order, - // A descending answer across groups starts at its End Group's first object. - start: - answer.order === "ascending" || range.start.group === end - ? range.start - : { group: end, object: 0n }, - }); - for (const sequence of order) { - if (sequence === last.sequence) { - for (const frame of last.frames) await out.object(sequence, frame, stop); - if (last.timedOut) await out.timedOut(sequence, fetchSlice(range, sequence), stop); - continue; - } - // Gone since it was listed: absent from the answer, which says it does not exist. - const group = hooks.cachedGroup(cache, sequence); - if (!group) continue; - try { - const slice = fetchSlice(range, sequence); - await this.#writeFillGroup({ out, group, slice, deadline, cancelled: stop }); - } finally { - group.close(); - } - } + const out = new FetchWriter({ stream, version, timescale }); + for (const frame of last.frames) await out.object(last.sequence, frame, stop); stream.close(); // The request stream ends once the answer is delivered. await untilAborted(stream.closed, stop); @@ -3462,8 +3358,6 @@ interface FetchAnswer { /** Whether the answer subscribed `cache` itself, and so closes it. */ owned: boolean; range: FetchRange; - /** The other groups the answer holds, lowest first. */ - groups: number[]; /** The highest group the answer holds, read ahead so FETCH_OK can say where it ends. */ last: FetchedGroup; ok: FetchOk; @@ -3473,8 +3367,6 @@ interface FetchAnswer { sendOrder: number; /** The subscription ranking the fetch stream joins, if any. */ ranking?: Priority; - /** The order its groups go out in. */ - order: Order; } /** A group read for a FETCH, so a later eviction cannot truncate what FETCH_OK promised. */ @@ -3485,16 +3377,8 @@ interface FetchedGroup { end: number; /** Whether the read reached the group's own end rather than stopping at the slice's. */ complete: boolean; - /** Whether the FILL_TIMEOUT ran out first, leaving the rest of the slice a gap. */ - timedOut: boolean; } -/** The largest group a FETCH can name. */ -const MAX_GROUP = 2n ** 62n - 1n; - -/** The largest Object ID a varint holds: the end of a gap in a group whose end is unknown. */ -const MAX_OBJECT = 2n ** 62n - 1n; - /** A wire sequence as a model one, saturating past what the model can name. */ function safe(sequence: bigint): number { return sequence > BigInt(Number.MAX_SAFE_INTEGER) ? Number.MAX_SAFE_INTEGER : Number(sequence); @@ -3536,31 +3420,15 @@ function fetchSlice(range: FetchRange, sequence: number): GroupSlice { }; } -/** - * Read the objects of `group` within `slice`, waiting for an open group to finish or the - * FILL_TIMEOUT `deadline` to pass. - */ -async function readFetched( - group: group.Consumer, - slice: GroupSlice, - deadline: Instant | undefined, - signal: AbortSignal, -): Promise { +/** Read a group's slice, waiting for its end when no object bound is set. */ +async function readFetched(group: group.Consumer, slice: GroupSlice, signal: AbortSignal): Promise { const frames: FetchedGroup["frames"] = []; let end = slice.skip; - const done = (complete: boolean, timedOut = false) => ({ - sequence: group.sequence, - frames, - end, - complete, - timedOut, - }); + const done = (complete: boolean) => ({ sequence: group.sequence, frames, end, complete }); for (;;) { - // A cap the finished group ends exactly at still reaches its end. if (slice.until !== undefined && end >= slice.until) return done(group.isClosed && group.frameCount === end && !(group.closed.peek() instanceof Error)); - const frame = await nextFrame(group, slice.skip, deadline, signal); - if (frame === TIMED_OUT) return done(false, true); + const frame = await nextFrame(group, slice.skip, signal); if (!frame) return done(true); if (slice.until !== undefined && frame.sequence >= slice.until) return done(false); frames.push(frame); @@ -3568,160 +3436,45 @@ async function readFetched( } } -/** What {@link nextFrame} returns once the FILL_TIMEOUT ran out with nothing buffered. */ -const TIMED_OUT = Symbol("timed out"); - -/** The instant a FILL_TIMEOUT of `timeout` from now runs out, or none without one. */ -function fillDeadline(timeout: Milli | undefined): Instant | undefined { - return timeout === undefined ? undefined : Instant.add(now(), Milli.toNano(timeout)); -} - -/** - * The next frame of `group` from `from` on, waiting while the group is open and until - * `closed` settles. Past the FILL_TIMEOUT `deadline` it takes only a frame already buffered, - * so a spent budget, or a zero one, still serves what is cached. - */ +/** Read the next selected frame until the requester leaves or the stream closes. */ async function nextFrame( group: group.Consumer, from: number, - deadline: Instant | undefined, signal: AbortSignal, closed?: Promise, -): Promise<({ sequence: number } & group.Frame) | undefined | typeof TIMED_OUT> { - const read = async () => { - const next = group.readFrameSequence({ from }); - // Raced as is: a reaction chained on `closed` would stay on it for every frame read. - return closed === undefined ? next : ((await race([next, closed])) ?? undefined); - }; - if (deadline === undefined) return untilAborted(read(), signal); - - for (let frame = group.tryReadFrameSequence(); frame; frame = group.tryReadFrameSequence()) { - if (frame.sequence >= from) return frame; - } - // A finished or failed group settles the read at once, which says which it was. - if (group.done) return untilAborted(read(), signal); - if (now() >= deadline) return TIMED_OUT; - const expiry = new AbortController(); - try { - const expired = sleepUntil(deadline, { signal: expiry.signal }).then((): typeof TIMED_OUT => TIMED_OUT); - return await untilAborted(race([read(), expired]), signal); - } finally { - expiry.abort(); - } +): Promise<({ sequence: number } & group.Frame) | undefined> { + const next = group.readFrameSequence({ from }); + return untilAborted(closed === undefined ? next : race([next, closed]).then((frame) => frame ?? undefined), signal); } -/** - * One fetch stream's objects and the End of Range markers between them, in the order its - * groups go out. Each refers back to the one before, so this tracks where that was. - */ +/** Write one group's fetch objects, inheriting the first object's priority. */ class FetchWriter { readonly stream: Writer; readonly #version: IetfVersion; readonly #timescale: Timescale | undefined; - readonly #order: Order; - /** The first Location the stream covers, in its own order. */ - readonly #start: Location; - /** The last object or End of Range written. */ - #prior?: Location; - /** Whether an object has been written, whose priority the rest inherit. */ #written = false; constructor({ stream, version, timescale, - order, - start, - }: { - stream: Writer; - version: IetfVersion; - /** The units objects are stamped in, if any; see {@link RunGroup.timescale}. */ - timescale: Timescale | undefined; - order: Order; - start: Location; - }) { + }: { stream: Writer; version: IetfVersion; timescale: Timescale | undefined }) { this.stream = stream; this.#version = version; this.#timescale = timescale; - this.#order = order; - this.#start = start; } /** Write `frame` of group `sequence`, following the prior object of that group, if any. */ async object(sequence: number, frame: { sequence: number } & group.Frame, stop: AbortSignal): Promise { - const group = BigInt(sequence); - const prior = this.#prior; const obj = new FetchFrame({ payload: frame.payload, timestamp: frame.timestamp }); const position = { group: sequence, object: frame.sequence, first: !this.#written, - previousGroup: prior !== undefined && prior.group !== group ? Number(prior.group) : undefined, }; await untilAborted(obj.encode(this.stream, position, this.#timescale, this.#version), stop); - this.#prior = { group, object: BigInt(frame.sequence) }; this.#written = true; } - - /** - * Mark the rest of group `sequence`'s `slice` as timed out, its objects having not - * arrived before the FILL_TIMEOUT ran out. Draft-18 has no such marker; its FILL_TIMEOUT - * reports them unknown. - * - * A marker covers everything since the prior one, so when nothing of this group went - * out, the groups skipped before it are first marked as what they otherwise imply: gone. - */ - async timedOut(sequence: number, slice: GroupSlice, stop: AbortSignal): Promise { - const group = BigInt(sequence); - const prior = this.#prior; - if (prior?.group !== group) { - const before = this.#before({ group, object: BigInt(slice.skip) }); - if (before && (prior ? this.#precedes(prior, before) : !this.#precedes(before, this.#start))) - await this.#end(EndOfRange.NonExistent, before, stop); - } - const reason = Filter.isDraft22(this.#version) ? EndOfRange.TimedOut : EndOfRange.Unknown; - const object = slice.until === undefined ? MAX_OBJECT : BigInt(slice.until - 1); - await this.#end(reason, { group, object }, stop); - } - - async #end(reason: EndOfRange, at: Location, stop: AbortSignal): Promise { - const marker = new FetchEndOfRange({ reason, group: at.group, object: at.object }); - await untilAborted(marker.encode(this.stream, this.#version), stop); - this.#prior = at; - } - - /** Whether `a` goes out before `b`: groups in the stream's order, objects ascending in one. */ - #precedes(a: Location, b: Location): boolean { - if (a.group !== b.group) return this.#order === "ascending" ? a.group < b.group : a.group > b.group; - return a.object < b.object; - } - - /** The Location right before `at` in the stream's order, if there is one. */ - #before(at: Location): Location | undefined { - if (at.object > 0n) return { group: at.group, object: at.object - 1n }; - if (this.#order === "descending") return { group: at.group + 1n, object: MAX_OBJECT }; - return at.group > 0n ? { group: at.group - 1n, object: MAX_OBJECT } : undefined; - } -} - -/** - * The groups `cache` holds within `groups`, lowest first. Only actual cached sequence - * numbers, never the peer's numeric span or a set of retained group handles, so a sparse - * range costs its retained groups. - */ -function cachedSequences(cache: TrackSubscriber, groups: Groups, priority: number): number[] { - const sequences: number[] = []; - const snapshot = cache.fork({ priority, maxDelay: Milli(Varint.MAX_U53) }); - try { - snapshot.withRange(groups); - for (let group = snapshot.tryRecvGroup(); group; group = snapshot.tryRecvGroup()) { - sequences.push(group.sequence); - group.close(); - } - } finally { - snapshot.close(); - } - return sequences.sort((a, b) => a - b); } /** The Track Properties a SUBSCRIBE_OK, TRACK_STATUS, or FETCH_OK answers with. */ @@ -3789,12 +3542,9 @@ type FillServe = /** The range is empty, so no fetch stream is opened at all. */ | { kind: "empty" } | { - kind: "range"; + kind: "group"; start: Location; end: EndLocation; - order: Order; - /** The FILL_TIMEOUT bounding its wait, if any. */ - timeout?: Milli; } /** Range Filters require object properties this publisher cannot select. */ | { kind: "unsupported" }; @@ -3895,7 +3645,7 @@ function excludes(range: ServeRange, sequence: number, slice: GroupSlice): boole /** A fill covering the whole requested prefix owns it; the live cursor starts after its snapshot. */ function afterFill(range: ServeRange, fill: FillServe | undefined, edge: LiveEdge): ServeRange { - if (fill?.kind !== "range" || !edge.largest || !edge.next) return range; + if (fill?.kind !== "group" || !edge.largest || !edge.next) return range; const start = range.start ?? { group: edge.latest ?? 0n, object: 0n }; if (start.group > edge.next.group || (start.group === edge.next.group && start.object >= edge.next.object)) return range; @@ -3911,12 +3661,7 @@ function afterFill(range: ServeRange, fill: FillServe | undefined, edge: LiveEdg * Resolve a fill request using the Fetch rules: relative to Largest Object and never * extending beyond it. An omitted Location Filter inherits the subscription's. */ -function fillRange( - fill: Filter.Fill, - subscription: Filter.Filter, - largest: Location | undefined, - order: Order, -): FillServe { +function fillRange(fill: Filter.Fill, subscription: Filter.Filter, largest: Location | undefined): FillServe { // A Range Filter narrows which objects pass, which we do not implement; serving the // unfiltered range instead would deliver objects the peer excluded, so refuse it. if (fill.rangeFilters) return { kind: "unsupported" }; @@ -3958,12 +3703,12 @@ function fillRange( ) { return { kind: "empty" }; } + if (start.group !== largest.group || end.group !== largest.group) return { kind: "unsupported" }; + return { - kind: "range", + kind: "group", start, end, - order: fill.groupOrder ?? order, - timeout: fill.fillTimeout, }; } diff --git a/rs/moq-net/benches/session/mod.rs b/rs/moq-net/benches/session/mod.rs index 3858423210c..ad6565d3865 100644 --- a/rs/moq-net/benches/session/mod.rs +++ b/rs/moq-net/benches/session/mod.rs @@ -16,13 +16,11 @@ //! groups have a `_cpu` suffix. Cache-clock advancement remains outside both clocks. //! `session_delivery_group_cost` sweeps publishers and viewers independently for SUBSCRIBE //! and FETCH; its constrained cases release held streams only after a real blocked write. -//! A one-relay FETCH fixture asserts one upstream FETCH per run during setup: one group on -//! moq-lite, a whole burst on moq-transport. -//! `session_delivery_fetch_range` FETCHes a burst of missing groups at once, swept over its -//! length, so a relay fills each run with one moq-transport FETCH. +//! A one-relay FETCH fixture asserts one upstream FETCH per group during setup. +//! `session_delivery_fetch_range` FETCHes missing groups independently, swept over burst length. //! `session_range_updates` widens established draft-22 subscriptions over a fresh cached -//! burst, swept over burst length and concurrent requests. Narrowing and writing the -//! next burst happen outside timing; the update and every filled group count. +//! burst's largest group, swept over burst length and concurrent requests. Narrowing and +//! writing the next burst happen outside timing; the update and largest-group fill count. //! `session_delivery_credit` grants a viewer one uni stream at a time, swept over its //! subscriptions and the backlog each holds, so every group waits for a stream grant. A //! publisher that parks state per backlogged group pays for it on each grant, which shows @@ -138,8 +136,7 @@ struct Shape { credit: Option, /// Publish compressed groups and exercise negotiated versus plain fallback egress. compress: bool, - /// Read each group via a fresh FETCH rather than a live subscription, a burst's groups - /// at once so the relays can fill them as one run. + /// Read each group via a fresh FETCH rather than a live subscription. fetch: bool, } @@ -677,12 +674,8 @@ impl Room { ); } if shape.fetch && shape.relays == 1 { - // A lite FETCH (stream type 3) names one group; a moq-transport FETCH (type 0x16) - // a whole run. Both types encode in one byte. - let (kind, run) = match room.cluster.version.is_lite() { - true => (3, 1), - false => (0x16, shape.burst), - }; + // Both FETCH types encode in one byte and every request names one group. + let kind = if room.cluster.version.is_lite() { 3 } else { 0x16 }; let upstream_fetches = room .cluster ._pairs @@ -695,7 +688,7 @@ impl Room { .count() }) .sum::(); - assert_eq!(upstream_fetches, room.groups / run, "one upstream FETCH per run"); + assert_eq!(upstream_fetches, room.groups, "one upstream FETCH per group"); } room } @@ -737,7 +730,7 @@ impl Room { } } let mut bytes = 0; - // A fetching viewer asks for a burst's groups at once; a subscription reads them + // A fetching viewer asks for each group independently; a subscription reads them // round-robin, so no subscription's backlog waits on another's being read. let passes = if self.shape.fetch { 1 } else { self.shape.burst }; let burst = self.shape.burst as u64; @@ -748,15 +741,6 @@ impl Room { let mut bytes = 0; for (broadcast, track) in &viewer.fetches { let newest = newest[*broadcast]; - // A burst is one span, as a downstream range FETCH asks for it. - let _filled = match burst { - 1 => None, - _ => Some( - moq_net::fuzz::fill_groups(track, newest + 1 - burst..newest + 1) - .await - .unwrap(), - ), - }; for sequence in newest + 1 - burst..=newest { let mut group = track.fetch_group(sequence, None).await.unwrap(); while let Some(frame) = group.read_frame().await.unwrap() { @@ -908,7 +892,7 @@ pub(crate) fn fetch_frames(c: &mut Criterion) { delivery::(c, "fetch_frames", shapes); } -/// Missing runs FETCHed at once across independent publishers, viewers, and run lengths. +/// Independent one-group FETCHes across publishers, viewers, and burst lengths. pub(crate) fn fetch_range(c: &mut Criterion) { if std::env::var_os("SESSION_ALLOCS").is_some() || std::env::var_os("SESSION_EXPIRY_ONLY").is_some() { return; @@ -927,7 +911,7 @@ pub(crate) fn fetch_range(c: &mut Criterion) { delivery::(c, "fetch_range", shapes); } -/// Range updates and their fill streams, over groups per fill and concurrent requests. +/// Largest-group fills, over cached backlog and concurrent requests. pub(crate) fn range_updates(c: &mut Criterion) { if std::env::var_os("SESSION_ALLOCS").is_some() || std::env::var_os("SESSION_EXPIRY_ONLY").is_some() { return; @@ -937,7 +921,7 @@ pub(crate) fn range_updates(c: &mut Criterion) { group.sample_size(10); for span in [2u64, 16] { for requests in [1usize, 16] { - group.throughput(Throughput::Elements(span * requests as u64)); + group.throughput(Throughput::Elements(requests as u64)); group.bench_function(BenchmarkId::new(format!("groups_{span}"), requests), |b| { let (mut cluster, mut viewers) = rt.block_on(async { let mut cluster = Cluster::::new( @@ -996,10 +980,14 @@ pub(crate) fn range_updates(c: &mut Criterion) { let started = Instant::now(); for viewer in &mut viewers { viewer.subscribers[0] - .update(track::Subscription::default().with_range(next..).with_max_delay(None)) + .update( + track::Subscription::default() + .with_range(future - 1..) + .with_max_delay(None), + ) .unwrap(); } - for _ in 0..span { + for _ in 0..1 { for viewer in &mut viewers { assert_eq!(read_group(&mut viewer.subscribers[0]).await, payload.len()); } diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index 86018d5e628..020057c3fc6 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -363,13 +363,6 @@ pub fn received(track: &track::Producer) { track.received(); } -/// Fill the `groups` of `track` its cache misses, a run per FETCH, as a publisher answers a -/// range FETCH, for the session bench. The returned guard keeps the filled groups cached. -pub async fn fill_groups(track: &track::Consumer, groups: std::ops::Range) -> crate::Result { - track.info().await?; - Ok(track.fill_groups(groups, 0).await) -} - /// `origin` as a session connected to `peer` serves it, for the viewers bench. pub fn excluding(origin: crate::origin::Consumer, peer: crate::Hop) -> crate::origin::Consumer { origin.excluding(peer) diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index 62e6c8aaada..cb38269ce74 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -154,24 +154,15 @@ impl FetchRange { } } -/// A FETCH resolved against its track, with the highest group it answers read ahead. +/// A FETCH resolved against its track, with its group read ahead. struct PreparedFetch { range: FetchRange, - /// The other groups the answer holds, lowest first. - groups: Vec, - /// The highest group the answer holds, read ahead so FETCH_OK can say where it ends. + /// Read ahead so FETCH_OK can say where the response ends. last: FetchedGroup, - /// Keeps the groups a relay filled for the answer cached while they are written. - _filled: Option, } -/// Resolve a FETCH's range against `track` and read its highest group, or the refusal -/// to answer with. `largest_filter` is a draft-22 filter ending at the Largest Object. -/// -/// The range ends at the Largest Object's group, known while a live feed writes the track -/// or once it ended, and reads that group whole even while it is open. A range spanning several -/// groups first fills what the cache misses, which on a relay is one upstream FETCH per -/// run of missing groups, then answers from the groups the cache holds. +/// Resolve a one-group FETCH and read its objects before promising them in FETCH_OK. +/// A whole-group request waits for an open group to finish. async fn prepare_fetch( track: &track::Consumer, mut range: FetchRange, @@ -214,12 +205,15 @@ async fn prepare_fetch( }, Filter::Absolute { start, .. } => start, }; - // Past every group, so the cap below ends it with the Largest Object's group. range.end = Location { - group: u64::MAX, + group: largest.group, object: 0, }; } + if range.start.group != range.end.group { + return Err((Error::Unsupported, "FETCH spans more than one group".into())); + } + if let Some(largest) = largest { // A start past Largest Object in a group the cache holds is refused INVALID_RANGE // (draft-22 section 9.11), rather than waiting on the live group. A group past every @@ -235,68 +229,19 @@ async fn prepare_fetch( false => failed(Error::NotFound), }); } - // The groups past the Largest Object's do not exist yet. That group itself is never - // cut, though draft-22 section 9.11 ends a FETCH at the Largest Object: the cache does - // not know an open group's real Largest Object, a peer has no way to enforce the cap, - // and a downstream cache would take a cut group as complete. So the answer waits for - // an open group to finish. - if range.end.group > largest.group { - range.end = Location { - group: largest.group, - object: 0, - }; - } } - let (filled, mut groups) = match range.start.group == range.end.group { - true => (None, vec![range.start.group]), - false => { - let span = range.start.group..range.end.group.saturating_add(1); - let filled = track.fill_groups(span.clone(), priority).await; - // Leaving out a group the source failed to serve would tell the peer it does not - // exist, so a failure that says nothing about the group refuses the whole answer. - if let Some(err) = filled.refused(span.clone()).into_iter().find(|err| { - !matches!( - err, - Error::NotFound | Error::NotFetchable | Error::Unsupported | Error::Old - ) - }) { - return Err(failed(err)); - } - // A group cached or filled for this answer exists even if the cache evicted it - // while the rest of the span filled: reading it again fetches it again. - let mut groups = filled.held(span.clone()); - groups.extend(track.cached_groups(span)); - (Some(filled), groups.into_iter().collect()) - } - }; - // A group gone or never fetchable is absent from the answer, as the response skips it, - // unless it is the only one. - let last = loop { - let Some(sequence) = groups.pop() else { - return Err((Error::NotFound, "no group in range".into())); - }; - match read_fetch( - track, - sequence, - range.skip(sequence), - range.until(sequence), - priority, - compression, - ) - .await - { - Ok(group) => break group, - Err(Error::NotFound | Error::NotFetchable | Error::Unsupported | Error::Old) if !groups.is_empty() => {} - Err(err) => return Err(failed(err)), - } - }; - Ok(PreparedFetch { - range, - groups, - last, - _filled: filled, - }) + let last = read_fetch( + track, + range.start.group, + range.start.object, + range.until(range.start.group), + priority, + compression, + ) + .await + .map_err(failed)?; + Ok(PreparedFetch { range, last }) } /// The group a FETCH answers with, read out of the cache so a later eviction cannot @@ -2285,8 +2230,7 @@ impl Publisher { super::request_stream::write_error(writer, request::Kind::Subscribe, err, reason, self.version).await } - /// Serve a draft-22 fill on its own fetch stream: the requested range, read from the - /// group cache, capped at the Largest Object snapshot. + /// Serve the largest group's head on its own draft-22 fetch stream. /// /// A fill is a promise once requested. An empty range opens no stream, but a range we /// cannot serve still opens one and resets it right after the FETCH_HEADER, the @@ -2311,7 +2255,6 @@ impl Publisher { group: sequence, object: skip, })), - FillServe::Range { start, .. } => Some(ranking.rank(start)), // Only the refusal goes out on its stream, so it competes with nothing. FillServe::Unsupported => None, }; @@ -2333,93 +2276,33 @@ impl Publisher { stream.varint(FetchHeader::TYPE).await?; stream.encode(&FetchHeader { request_id }).await?; - let (range, groups, _filled) = match fill { - FillServe::Group { sequence, skip, until } => ( - FetchRange { - start: Location { - group: sequence, - object: skip, - }, - end: Location { - group: sequence, - object: until.unwrap_or(0), - }, - }, - vec![sequence], - None, - ), - FillServe::Range { start, end, order } => { - let span = start.group..end.group.saturating_add(1); - let filled = track.fill_groups(span.clone(), priority).await; - if let Some(err) = filled.refused(span.clone()).into_iter().find(|err| { - !matches!( - err, - Error::NotFound | Error::NotFetchable | Error::Unsupported | Error::Old - ) - }) { - return Err(err); - } - let mut groups = filled.held(span.clone()); - groups.extend(track.cached_groups(span)); - ( - FetchRange { - start, - end: Location { - group: end.group, - object: end.object.map_or(0, |object| object.saturating_add(1)), - }, - }, - { - let mut groups: Vec<_> = groups.into_iter().collect(); - if order.map_or(ranking.order, GroupOrder::to_order) == Order::Descending { - groups.reverse(); - } - groups - }, - Some(filled), - ) - } - _ => return Err(Error::Unsupported), + let FillServe::Group { sequence, skip, until } = fill else { + return Err(Error::Unsupported); }; - let mut prior_group = None; - for sequence in groups { - let group = match track - .fetch_group( - sequence, - group::Fetch { - priority, - ..Default::default() - }, - ) - .await - { - Ok(group) => group, - Err(Error::NotFound | Error::NotFetchable | Error::Unsupported | Error::Old) - if matches!(fill, FillServe::Range { .. }) => - { - continue; - } - Err(err) => return Err(err), - }; - let group = if self.peer_setup.get().await.flate { - group.compressed() - } else { - group - }; - Self::write_fetch_group( - &mut stream, - group, + let group = track + .fetch_group( sequence, - GroupSlice { - skip: range.skip(sequence), - until: range.until(sequence), + group::Fetch { + priority, + ..Default::default() }, - timescale, - self.version, - &mut prior_group, ) .await?; - } + let group = if self.peer_setup.get().await.flate { + group.compressed() + } else { + group + }; + Self::write_fetch_group( + &mut stream, + group, + sequence, + GroupSlice { skip, until }, + timescale, + self.version, + &mut None, + ) + .await?; Ok(()) } .await; @@ -2801,13 +2684,7 @@ impl Publisher { .await; } - // FILL_TIMEOUT=0 asks for cache only, and any budget ends in Timed-Out gaps we - // don't write, so waiting on upstream regardless would ignore what was asked. - if msg.fill_timeout { - return self - .reject_fetch(stream, &Error::Unsupported, "FILL_TIMEOUT not supported") - .await; - } + // FILL_TIMEOUT is accepted and ignored: a one-group read ends with the group. // Pinned to the instance it names, if any: another is refused, never served. let epoch = msg.epoch.clone(); @@ -2886,6 +2763,12 @@ impl Publisher { return self.reject_fetch(stream, &Error::InvalidRange, "empty range").await; } + if largest_filter.is_none() && start.group != end.group { + return self + .reject_fetch(stream, &Error::Unsupported, "FETCH spans more than one group") + .await; + } + // The peer must have seen the announcement to name this namespace, so this // resolves like a SUBSCRIBE does. let broadcast = match gate.guard(self.resolve(&mut stream, &namespace, epoch)).await { @@ -2936,7 +2819,16 @@ impl Publisher { let gate = if token { gate.named() } else { gate }; lapses_with = Some(subscriber_request_id); let start = match fetch_type { - FetchType::RelativeJoining { group_offset, .. } => end.group.saturating_sub(group_offset), + FetchType::RelativeJoining { group_offset: 0, .. } => end.group, + FetchType::RelativeJoining { .. } => { + return self + .reject_fetch( + stream, + &Error::Unsupported, + "joining FETCH reaches before the joined group", + ) + .await; + } FetchType::AbsoluteJoining { group_id, .. } if group_id <= end.group => group_id, _ => { return self @@ -2974,10 +2866,7 @@ impl Publisher { }) }; - // The answer's highest group is read before replying: FETCH_OK names where the response - // ends, which a range running past the track only learns by reading it, and a refusal - // can still replace it until then. The subscriber cancelling is the only other way this - // ends early, and is owed nothing. + // Read the group before FETCH_OK promises it, while a refusal is still possible. let compression = self.peer_setup.get().await.flate; let prepared = { let mut prepare = std::pin::pin!(prepare_fetch( @@ -3003,18 +2892,11 @@ impl Publisher { }) .await }; - let PreparedFetch { - range, - groups, - last: group, - _filled, - } = match prepared { + let PreparedFetch { range, last: group } = match prepared { Some(Ok(prepared)) => prepared, Some(Err((err, reason))) => return self.reject_fetch(stream, &err, &reason).await, None => return Ok(()), }; - // The range ended short of the requested group, at the Largest Object's. - let capped = range.end != end; let end = range.end; let group = group.without_status(); // FETCH keeps each object's Timestamp, in the units FETCH_OK declares wherever the @@ -3049,10 +2931,7 @@ impl Publisher { } end } else if draft22 { - // Draft-22's End Location is inclusive: the end, already capped at the Largest - // Object's group (section 10.14). Objects missing before it do not exist. A whole-group end has no - // End Object to report, so it covers what the finished group holds, and at least - // the start an empty answer covered. + // Draft-22's End Location is inclusive. An empty answer still covers its start. let object = match range.until(group.sequence) { // The track's last object as this draft writes it, without a status object. Some(until) if end_of_track => (until - 1).min(group.end().saturating_sub(1)), @@ -3063,11 +2942,8 @@ impl Publisher { group: group.sequence, object, } - } else if (end_of_track && end.object > 0) || group.sequence < end.group || capped { - // Cap a bounded response at the track's last object, or at the last object sent - // when the end group is not held or the range was capped: before draft-22 the End - // Location promises every object before it. A whole-group response to a request - // for that group keeps Object 0 rather than claiming that the next group is covered. + } else if end_of_track && end.object > 0 { + // A bounded response cannot promise objects past the track's end. Location { group: group.sequence, object: group.end(), @@ -3076,15 +2952,6 @@ impl Publisher { end }; - // Groups go out in the order asked for, each read as its turn comes. - let last = group.sequence; - let mut order = groups; - order.push(last); - if matches!(msg.group_order, GroupOrder::Descending) { - order.reverse(); - } - let mut last_group = Some(group); - // The response, raced against the grant: a standalone FETCH whose grant narrows while it // waits on stream credit stops there instead of sending what it no longer may. let ok = async { @@ -3115,8 +2982,7 @@ impl Publisher { return Err(err); } - // The groups after FETCH_OK, which may wait on a group still being written, so the - // subscriber cancelling still releases everything the answer holds. + // Cancellation releases the buffered answer even while its stream is flow controlled. let respond = async { let mut rank = ranking.rank(range.start); let uni = self @@ -3132,28 +2998,7 @@ impl Publisher { }) .await?; let mut prior_group = None; - for sequence in order { - let group = match last_group.take_if(|group| group.sequence == sequence) { - Some(group) => group, - None => match read_fetch( - &track, - sequence, - range.skip(sequence), - range.until(sequence), - priority, - compression, - ) - .await - { - Ok(group) => group.without_status(), - // Gone since it was listed, or never fetchable: absent from the answer, - // which says it does not exist. - Err(Error::NotFound | Error::NotFetchable | Error::Unsupported | Error::Old) => continue, - Err(err) => return Err(err), - }, - }; - Self::write_fetched_group(&mut writer, group, &mut prior_group, self.version, timescale).await?; - } + Self::write_fetched_group(&mut writer, group, &mut prior_group, self.version, timescale).await?; settle(&mut writer).await?; Ok::<(), Error>(()) }; @@ -9287,30 +9132,11 @@ mod serve_tests { ("FETCH", ietf::Fetch::ID, &[ 0x03, // Number of Parameters 0x0A, 0x00, // FILL_TIMEOUT = 0 - 0x17, 0x01, 0x01, // LOCATION_FILTER (0x21): from one group back + 0x17, 0x01, 0x00, // LOCATION_FILTER (0x21): whole track 0x14, 0x01, // INCLUDE_PROPERTIES (0x35) = 1 ]), ]; - // A standalone FETCH we would otherwise serve, carrying FILL_TIMEOUT=0: cache only, - // with gaps reported as Timed-Out, which we can't write. - #[rustfmt::skip] - let fill_timeout = [ - 0x2B, // Request ID - 0x01, // Standalone - 0x01, 0x04, b'r', b'o', b'o', b'm', 0x05, b'v', b'i', b'd', b'e', b'o', - 0x00, 0x00, // Start Location - 0x00, 0x00, // End Location: the whole group 0 - 0x01, // Number of Parameters - 0x0A, 0x00, // FILL_TIMEOUT = 0 - ]; - let version = Version::Draft18; - assert_eq!( - refusal(version, ietf::Fetch::ID, fill_timeout.to_vec()).await, - NOT_SUPPORTED, - "{version}: FETCH with FILL_TIMEOUT" - ); - let version = Version::Draft22; for (label, id, params) in cases { assert_eq!( @@ -9540,7 +9366,7 @@ mod serve_tests { occurrences(&h.log, b"before-update") == 1 }) .await; - let mut updates = vec![(Location { group: 0, object: 0 }, true)]; + let mut updates = vec![(Location { group: 1, object: 0 }, true)]; if let Some(start) = narrowing { updates.insert(0, (start, false)); } @@ -9579,7 +9405,11 @@ mod serve_tests { group.finish().unwrap(); h.track.finish().unwrap(); serving.await.unwrap(); - assert_eq!(occurrences(&h.log, b"history"), 1, "the fill carries the history"); + assert_eq!( + occurrences(&h.log, b"history"), + 0, + "the fill only carries the current head" + ); for payload in [b"after-1", b"after-2"] { assert_eq!( occurrences(&h.log, payload), @@ -9645,7 +9475,7 @@ mod serve_tests { group.finish().unwrap(); h.track.finish().unwrap(); serving.await.unwrap(); - assert_eq!(occurrences(&h.log, b"history"), 1, "only the fill carries the history"); + assert_eq!(occurrences(&h.log, b"history"), 0, "the historical fill is refused"); assert_eq!(occurrences(&h.log, b"live"), 1, "the live group is not repeated"); } @@ -9715,7 +9545,7 @@ mod serve_tests { } h.track.finish().unwrap(); serving.await.unwrap(); - assert_eq!(occurrences(&h.log, b"history"), 1, "the historical fill is independent"); + assert_eq!(occurrences(&h.log, b"history"), 0, "the historical fill is refused"); assert_eq!(occurrences(&h.log, b"excluded"), 0, "the live floor stays at group 4"); assert_eq!( occurrences(&h.log, b"later-current"), @@ -9769,9 +9599,9 @@ mod serve_tests { } } - /// A fill spanning several groups carries every cached group on one fetch stream. + /// A wider fill resets its promised stream and leaves the live subscription open. #[moq_clock::test] - async fn a_multi_group_fill_serves_its_range() { + async fn a_multi_group_fill_is_refused_without_ending_the_subscription() { let mut h = serve(Version::Draft22); for sequence in 0..2 { @@ -9794,85 +9624,9 @@ mod serve_tests { .await; assert_eq!(occurrences(&h.log, FETCH_STREAM), 1, "the promised stream still opens"); - assert!(h.log.resets().is_empty(), "a supported fill must not reset"); - assert_eq!(occurrences(&h.log, b"frame"), 2, "both groups are served"); - } - - /// A sparse fill follows its requested order, inherits the subscription's when - /// omitted, and caps only the current group's head at the snapshot. - #[moq_clock::test] - async fn a_multi_group_fill_obeys_order_and_object_bounds() { - for order in [None, Some(GroupOrder::Ascending), Some(GroupOrder::Descending)] { - let h = serve(Version::Draft22); - for sequence in [2, 5, 8] { - let mut group = h.track.create_group(group::Info { sequence }).unwrap(); - for object in 0..2 { - group - .write_frame(timestamp(), format!("{sequence}-{object}").into_bytes()) - .unwrap(); - } - group.finish().unwrap(); - } - let ranking = Ranking { - queue: h.publisher.priority.clone(), - group: None, - request_id: RequestId(REQUEST_ID), - priority: kio::Producer::new(PRIORITY).consume(), - publisher: 0, - order: Order::Descending, - }; - let fill = fill_range( - ietf::Fill { - group_order: order, - range_filters: false, - filter: Some(Filter::Absolute { - start: Location { group: 2, object: 1 }, - end: None, - }), - }, - Filter::NextObject, - Some(Location { group: 8, object: 0 }), - ); - h.publisher - .run_fill( - fill, - &ranking, - h.track.consume(), - None, - &AtomicU64::new(0), - ranking.request_id, - ) - .await; - assert!(h.log.resets().is_empty()); - let wire = h.log.writes.lock().unwrap().clone(); - let mut buf = Decoder::new(&wire, Version::Draft22.into()); - assert_eq!(buf.varint().unwrap(), FetchHeader::TYPE); - assert_eq!( - FetchHeader::decode(&mut buf, Version::Draft22).unwrap().request_id, - ranking.request_id - ); - let mut headers = Vec::new(); - let mut payloads = Vec::new(); - while !buf.is_empty() { - let ietf::FetchObject::Object { group, object, .. } = - ietf::FetchObject::decode(&mut buf, Version::Draft22).unwrap() - else { - panic!("not an object") - }; - if let Some(group) = group { - headers.push((group, object)); - } - let size = buf.varint().unwrap() as usize; - payloads.push(String::from_utf8(buf.slice(size).unwrap().to_vec()).unwrap()); - } - if order == Some(GroupOrder::Ascending) { - assert_eq!(headers, [(2, Some(1)), (2, Some(0)), (2, Some(0))]); - assert_eq!(payloads, ["2-1", "5-0", "5-1", "8-0"]); - } else { - assert_eq!(headers, [(8, Some(0)), (2, Some(0)), (2, Some(1))]); - assert_eq!(payloads, ["8-0", "5-0", "5-1", "2-1"]); - } - } + assert_eq!(h.log.resets().len(), 1, "only the fill stream resets"); + assert_eq!(occurrences(&h.log, b"frame"), 0, "no history is served"); + assert!(h.log.closes().is_empty(), "the session stays open"); } /// Subscriber priority 128 on the wire, which every request in these tests carries. @@ -10654,7 +10408,7 @@ mod serve_tests { group_order, fetch_type, range_filters: false, - fill_timeout: false, + fill_timeout: true, properties_wanted, epoch: None, }, @@ -11200,36 +10954,6 @@ mod serve_tests { } } - /// A fill that fails for a reason other than the group's absence refuses the whole range, - /// rather than leaving the group out as though it did not exist. - #[moq_clock::test] - async fn a_range_fetch_refuses_a_failed_fill() { - for version in FETCH_DRAFTS { - let h = serve(version); - let request = h._broadcast.reserve_track("cam").unwrap(); - let dynamic = request.requests(); - let fetching = request.consume().fetch_group(1, None); - let mut fetched = dynamic.requested_fill_group().await.unwrap().accept(None).unwrap(); - fetched.write_frame(None, b"1-0".to_vec()).unwrap(); - fetched.finish().unwrap(); - fetching.await.unwrap(); - - let fetch_type = fetch_range( - version, - "cam", - Location { group: 0, object: 0 }, - Location { group: 1, object: 1 }, - ); - let refuse = async { - let fill = dynamic.requested_fill().await.unwrap(); - assert_eq!((fill.start().group, fill.end()), (0, 1), "{version}"); - fill.refuse_groups(0..1, Error::Unauthorized); - }; - let (buf, ()) = futures::join!(run_fetch(&h, fetch_type, GroupOrder::Ascending, true), refuse); - fetch_refusal(buf, version); - } - } - /// A track whose cache only ever answered a fetch has no live feed, so its newest /// object is not Largest Object: a draft-22 FETCH past it echoes the requested end. #[moq_clock::test] @@ -11321,6 +11045,7 @@ mod serve_tests { for version in [Version::Draft22] { for (filter, groups) in [ (Filter::Unfiltered, Some(vec![0, 1, 2])), + (Filter::Relative(1), Some(vec![2])), (Filter::NextObject, None), (Filter::Relative(2), Some(vec![1, 2])), ( @@ -11353,6 +11078,9 @@ mod serve_tests { (None, false) => { assert_eq!(fetch_refusal(buf, version), INVALID_RANGE, "{version}: {filter:?}") } + (Some(groups), false) if groups.len() > 1 => { + assert_eq!(fetch_refusal(buf, version), 0x3, "{version}: {filter:?}"); + } (Some(groups), false) => { let (ok, objects) = fetch_answer(buf, version); assert_eq!( @@ -11369,78 +11097,30 @@ mod serve_tests { } } - /// A range touching several groups is answered on one fetch stream in the order asked - /// for, from its start object to its end object, without the groups that do not exist. + /// Wider requests are refused before FETCH_OK, even if their end is past the track. #[moq_clock::test] - async fn a_standalone_fetch_serves_several_groups_in_order() { + async fn a_standalone_fetch_spanning_several_groups_is_refused() { for version in FETCH_DRAFTS { for order in [GroupOrder::Ascending, GroupOrder::Descending] { let mut h = serve(version); publish_pairs(&mut h, 5, Some(2)); settle().await; - - let buf = standalone_fetch( - &h, - Location { group: 0, object: 1 }, - Location { group: 3, object: 1 }, - order, - ) - .await; - let (ok, objects) = fetch_answer_in(buf, version, order); - assert_eq!(ok.end_location, end_location(version, 3, 0), "{version} {order:?}"); - assert!(!ok.end_of_track, "{version} {order:?}"); - let mut expected: Vec> = - vec![pairs([0])[1..].to_vec(), pairs([1]), pairs([3])[..1].to_vec()]; - if matches!(order, GroupOrder::Descending) { - expected.reverse(); + for end in [3, 9] { + let buf = standalone_fetch( + &h, + Location { group: 0, object: 1 }, + Location { group: end, object: 0 }, + order, + ) + .await; + assert_eq!(fetch_refusal(buf, version), 0x3, "{version} {order:?}"); + assert!(h.log.resets().is_empty()); } - assert_eq!(objects, expected.concat(), "{version} {order:?}"); } } } - /// A highest group that cannot be fetched is left out like any other, and the End Location - /// names the last object sent rather than promising the group left out. - #[moq_clock::test] - async fn a_fetch_skips_an_unfetchable_highest_group() { - for version in FETCH_DRAFTS { - let mut h = serve(version); - publish_pairs(&mut h, 3, None); - let mut datagram = h.track.create_group(group::Info { sequence: 3 }).unwrap(); - datagram - .set_wire(group::Wire { - sparse: false, - first_object: true, - first_id: 0, - extensions: false, - has_end: true, - fetchable: false, - last: None, - }) - .unwrap(); - datagram.write_frame(timestamp(), b"3-0".to_vec()).unwrap(); - datagram.finish().unwrap(); - let mut later = h.track.create_group(group::Info { sequence: 4 }).unwrap(); - later.write_frame(timestamp(), b"4-0".to_vec()).unwrap(); - later.finish().unwrap(); - settle().await; - - let buf = standalone_fetch( - &h, - Location { group: 1, object: 0 }, - Location { group: 3, object: 0 }, - GroupOrder::Ascending, - ) - .await; - let (ok, objects) = fetch_answer(buf, version); - assert_eq!(ok.end_location, end_location(version, 2, 1), "{version}"); - assert_eq!(objects, [pairs([1]), pairs([2])].concat(), "{version}"); - } - } - - /// A range reaching past a live track's Largest Object ends with its group, which it - /// waits on until the group finishes rather than cutting it at the Largest Object, and - /// never claims the track ends. + /// A whole live group waits for its end and never claims the track ends. #[moq_clock::test] async fn a_standalone_fetch_waits_for_the_largest_object_group() { for version in FETCH_DRAFTS { @@ -11452,8 +11132,8 @@ mod serve_tests { let fetch = standalone_fetch( &h, - Location { group: 3, object: 0 }, - Location { group: 9, object: 0 }, + Location { group: 4, object: 0 }, + Location { group: 4, object: 0 }, GroupOrder::Ascending, ); let finish = async { @@ -11463,68 +11143,17 @@ mod serve_tests { }; let (buf, ()) = futures::join!(fetch, finish); let (ok, objects) = fetch_answer(buf, version); - assert_eq!(ok.end_location, end_location(version, 4, 1), "{version}"); + assert_eq!( + ok.end_location, + if Filter::is_draft22(version) { + Location { group: 4, object: 1 } + } else { + Location { group: 4, object: 0 } + }, + "{version}" + ); assert!(!ok.end_of_track, "{version}"); - assert_eq!(objects, pairs([3, 4]), "{version}"); - } - } - - /// A range through a finished track's last object ends the track, wherever it starts. - #[moq_clock::test] - async fn a_fetch_of_several_groups_through_the_final_object_reports_the_end_of_track() { - for version in FETCH_DRAFTS { - let mut h = serve(version); - publish_pairs(&mut h, 5, None); - h.track.finish().unwrap(); - settle().await; - - let buf = standalone_fetch( - &h, - Location { group: 3, object: 0 }, - Location { group: 9, object: 0 }, - GroupOrder::Ascending, - ) - .await; - let (ok, objects) = fetch_answer(buf, version); - assert_eq!(ok.end_location, end_location(version, 4, 1), "{version}"); - assert!(ok.end_of_track, "{version}"); - assert_eq!(objects, pairs([3, 4]), "{version}"); - } - } - - /// A whole End Group at the wire's limit is valid: a range from the first group stops at - /// the track's end, and a missing single group is not found. - #[moq_clock::test] - async fn a_standalone_fetch_through_the_maximum_group_is_capped() { - for version in FETCH_DRAFTS { - let maximum = u64::MAX; - let mut h = serve(version); - publish_pairs(&mut h, 1, None); - h.track.finish().unwrap(); - settle().await; - for start in [0, maximum] { - let buf = standalone_fetch( - &h, - Location { - group: start, - object: 0, - }, - Location { - group: maximum, - object: 0, - }, - GroupOrder::Ascending, - ) - .await; - if start == maximum { - assert_eq!(fetch_refusal(buf, version), DOES_NOT_EXIST, "{version}"); - continue; - } - let (ok, objects) = fetch_answer(buf, version); - assert_eq!(ok.end_location, end_location(version, 0, 1), "{version}"); - assert!(ok.end_of_track, "{version}"); - assert_eq!(objects, pairs([0]), "{version}"); - } + assert_eq!(objects, pairs([4]), "{version}"); } } @@ -11716,101 +11345,6 @@ mod serve_tests { } } - /// A FETCH of groups 3 and 4 on a session whose request stream first carries `script`, - /// driven until FETCH_OK is out and the answer waits on group 3, still being written. - async fn fetch_waiting_after_ok( - version: Version, - script: Vec, - ) -> ( - ScriptedSession, - group::Producer, - std::pin::Pin>>>, - ) { - let origin = crate::origin::Config::new(crate::Hop::new(1).unwrap()).produce(); - let broadcast = origin.publish("room", crate::origin::Route::default()).unwrap(); - let track = broadcast.create_track("video", crate::track::Info::timed()).unwrap(); - let mut open = track.create_group(group::Info { sequence: 3 }).unwrap(); - open.write_frame(timestamp(), b"3-0".to_vec()).unwrap(); - let mut done = track.create_group(group::Info { sequence: 4 }).unwrap(); - done.write_frame(timestamp(), b"4-0".to_vec()).unwrap(); - done.finish().unwrap(); - settle().await; - - let session = ScriptedSession::new(script); - let peer_setup = peer::PeerSetup::default(); - peer_setup.set(peer::Peer::default()); - let publisher = Publisher::new( - crate::time::sim(), - crate::driver::Transport::new(session.clone()), - origin.consume(), - Control::new(false), - None, - peer_setup, - version, - ); - let stream = Stream::open(&mut crate::driver::Transport::new(session.clone()), version) - .await - .unwrap(); - let mark = session.log.writes.lock().unwrap().len(); - let mut fetch = Box::pin(publisher.run_fetch_stream( - stream, - ietf::Fetch { - request_id: FETCH_ID, - subscriber_priority: 128, - group_order: GroupOrder::Ascending, - fetch_type: FetchType::Standalone { - namespace: crate::Path::from_static("room"), - track: "video".into(), - start: Location { group: 3, object: 0 }, - end: Location { group: 4, object: 0 }, - }, - range_filters: false, - fill_timeout: false, - properties_wanted: true, - epoch: None, - }, - )); - for _ in 0..100 { - assert!(futures::poll!(fetch.as_mut()).is_pending(), "{version}: answered early"); - if session.log.writes.lock().unwrap().len() > mark { - break; - } - moq_clock::sim::yield_now().await; - } - assert!( - session.log.writes.lock().unwrap().len() > mark, - "{version}: FETCH_OK never went out" - ); - (session, open, fetch) - } - - /// A FETCH whose answer waits on an earlier group still being written, after FETCH_OK - /// is out, ends when the subscriber cancels it rather than holding the group. - #[moq_clock::test] - async fn a_fetch_cancelled_after_its_fetch_ok_stops_waiting() { - use crate::lite::test_transport::Close; - for version in FETCH_DRAFTS { - let (session, _open, mut fetch) = fetch_waiting_after_ok(version, Vec::new()).await; - session.close(Close::Reset); - let res = futures::poll!(fetch.as_mut()); - assert!(matches!(res, Poll::Ready(Ok(()))), "{version}: {res:?}"); - } - } - - /// A request stream carries at most one GOAWAY, counted across the whole FETCH: one - /// before FETCH_OK and another after it close the session. - #[moq_clock::test] - async fn a_fetch_counts_goaways_across_its_fetch_ok() { - const GOAWAY: [u8; 5] = [0x10, 0, 2, 0, 0]; - for version in [Version::Draft18, Version::Draft22] { - let (session, _open, mut fetch) = fetch_waiting_after_ok(version, GOAWAY.to_vec()).await; - assert!(session.log.closes().is_empty(), "{version}: one GOAWAY is allowed"); - session.push(&GOAWAY); - assert!(futures::poll!(fetch.as_mut()).is_ready(), "{version}: still answering"); - assert_eq!(session.log.closes()[0].0, 0x03, "{version}"); - } - } - /// A joining FETCH answers from its subscription's cache, which outlives the /// subscription, so it holds its own gate: one parked on stream credit when the grant /// narrows stops there, and nothing reaches the wire once credit returns. @@ -12446,10 +11980,9 @@ mod serve_tests { } } - /// A joining FETCH reaching back before the subscription's group is answered whole up to - /// where the subscription starts. + /// A joining FETCH reaching back before the subscription's group is refused. #[moq_clock::test] - async fn a_joining_fetch_reaching_back_is_served() { + async fn a_joining_fetch_spanning_several_groups_is_refused() { for version in JOINING_DRAFTS { let mut h = serve(version); publish_pairs(&mut h, 4, None); @@ -12467,7 +12000,7 @@ mod serve_tests { ); registered(&h, serving.as_mut()).await; - for (fetch_type, first) in [ + for (fetch_type, _first) in [ ( FetchType::RelativeJoining { subscriber_request_id: RequestId(REQUEST_ID), @@ -12506,11 +12039,7 @@ mod serve_tests { .unwrap(); let buf = bytes::Bytes::from(h.log.writes.lock().unwrap()[mark..].to_vec()); - let (ok, objects) = fetch_answer(buf, version); - assert_eq!(ok.end_location, Location { group: 4, object: 1 }, "{version}"); - let mut expected = pairs(first..4); - expected.push((4, 0, "4-0".into())); - assert_eq!(objects, expected, "{version}"); + assert_eq!(fetch_refusal(buf, version), 0x3, "{version}"); } } } @@ -16914,7 +16443,6 @@ impl ServeRange { object: until.and_then(|until| until.checked_sub(1)), }, ), - FillServe::Range { start, end, .. } => (start, end), _ => return None, }; ((requested.group, requested.object) >= (start.group, start.object) @@ -17031,12 +16559,6 @@ enum FillServe { skip: u64, until: Option, }, - /// Several groups, with the final group capped at the snapshot for its live tail. - Range { - start: Location, - end: EndLocation, - order: Option, - }, /// Object filters that this publisher cannot serve. Unsupported, } @@ -17086,12 +16608,8 @@ fn fill_range(fill: ietf::Fill, subscription: Filter, largest: Option) if start.group > end.group || (start.group == end.group && end.object.is_some_and(|object| object < start.object)) { return FillServe::Empty; } - if start.group != end.group { - return FillServe::Range { - start, - end, - order: fill.group_order, - }; + if start.group != largest.group || end.group != largest.group { + return FillServe::Unsupported; } FillServe::Group { @@ -17495,9 +17013,9 @@ mod fill_range_tests { ); } - /// A whole past group is served to its end; only the current group is capped. + /// A fill of a past group is refused even when it names only one group. #[test] - fn a_past_group_is_served_whole() { + fn a_past_group_fill_is_unsupported() { assert_eq!( fill_range( fill(Filter::Absolute { @@ -17507,11 +17025,7 @@ mod fill_range_tests { Filter::NextObject, LARGEST ), - FillServe::Group { - sequence: 7, - skip: 0, - until: None, - } + FillServe::Unsupported ); } @@ -17521,19 +17035,19 @@ mod fill_range_tests { assert_eq!( fill_range( fill(Filter::Absolute { - start: Location { group: 7, object: 2 }, + start: Location { group: 100, object: 2 }, end: Some(EndLocation { - group: 7, - object: Some(5) + group: 100, + object: Some(3) }), }), Filter::NextObject, LARGEST ), FillServe::Group { - sequence: 7, + sequence: 100, skip: 2, - until: Some(6), + until: Some(4), } ); } @@ -17561,43 +17075,22 @@ mod fill_range_tests { ); } - /// A fill can span cached history and the current group's head. + /// A fill never promises history spanning several groups, regardless of group order. #[test] - fn a_multi_group_fill_keeps_its_bounds() { - for (filter, start, end) in [ - ( - Filter::Relative(3), - Location { group: 98, object: 0 }, - EndLocation { - group: 100, - object: Some(4), - }, - ), - ( - Filter::Unfiltered, - Location { group: 0, object: 0 }, - EndLocation { - group: 100, - object: Some(4), - }, - ), - ( - Filter::Absolute { - start: Location { group: 7, object: 2 }, - end: Some(EndLocation { group: 9, object: None }), - }, - Location { group: 7, object: 2 }, - EndLocation { group: 9, object: None }, - ), + fn a_multi_group_fill_is_unsupported() { + for filter in [ + Filter::Unfiltered, + Filter::Relative(3), + Filter::Absolute { + start: Location { group: 7, object: 2 }, + end: Some(EndLocation { group: 9, object: None }), + }, ] { - assert_eq!( - fill_range(fill(filter), Filter::NextObject, LARGEST), - FillServe::Range { - start, - end, - order: None - } - ); + for order in [GroupOrder::Ascending, GroupOrder::Descending] { + let mut request = fill(filter); + request.group_order = Some(order); + assert_eq!(fill_range(request, Filter::NextObject, LARGEST), FillServe::Unsupported); + } } } diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index ed6fa325a39..be80a7a5946 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -14322,8 +14322,18 @@ mod stitch_tests { let mut run = GroupFetchRun::new(version, objects).await; let dynamic = run.track.requests(); let consumer = run.track.consume(); - let mut filling = Box::pin(consumer.fill_groups(SEQUENCE..last + 1, 0)); - assert!(futures::poll!(filling.as_mut()).is_pending()); + run.track.start_at(last + 1).unwrap(); + run.track.create_group((last + 1).into()).unwrap().finish().unwrap(); + let mut reading = consumer + .subscribe(Some( + track::Subscription::default() + .with_range(SEQUENCE..last + 1) + .with_order(crate::track::Order::Ascending) + .with_max_delay(None), + )) + .await + .unwrap(); + assert!(futures::FutureExt::now_or_never(reading.recv_group()).is_none()); let fill = dynamic.requested_fill().await.unwrap(); let bounded = consumer.fetch_group(last, None); let slot = kio::Producer::new(GroupFetch::Ready(RangeFetch { @@ -14389,103 +14399,29 @@ mod stitch_tests { assert!(run.track.consume().peek_group(SEQUENCE).is_none(), "nothing is cached"); } - /// A run of missing groups goes upstream as one FETCH, whose single stream our publisher - /// answers with every group it holds. The next relay caches those, and answers the readers - /// of the groups between them, which do not exist, with a miss. + /// A third-party publisher may answer a range even though ours refuses it. The subscriber + /// caches the supplied groups and reports the gaps between them as misses. #[moq_clock::test] - async fn a_range_fetch_round_trips_with_the_groups_between_dropped() { - use crate::lite::test_transport::Sent; + async fn a_range_fetch_caches_the_groups_with_the_groups_between_dropped() { for version in [Version::Draft18, Version::Draft22] { - let origin = crate::origin::Config::new(crate::Hop::new(2).unwrap()).produce(); - let broadcast = origin.publish("room", crate::origin::Route::default()).unwrap(); - let track = broadcast - .create_track("video", track::Info::default().with_timescale(None)) - .unwrap(); - for sequence in [4, 6] { - let mut group = track.create_group(group::Info { sequence }).unwrap(); - for object in 0..2 { - group - .write_frame(None, format!("{sequence}-{object}").into_bytes()) - .unwrap(); - } - group.finish().unwrap(); - } - track.finish().unwrap(); - - // The FETCH a relay sends for the run 3..=6. - let namespace = Path::from_static("room"); - let fetch_type = match Filter::is_draft22(version) { - true => ietf::FetchType::Filtered { - namespace, - track: "video".into(), - filter: Filter::Absolute { - start: ietf::Location { group: 3, object: 0 }, - end: Some(ietf::EndLocation { group: 6, object: None }), - }, - }, - false => ietf::FetchType::Standalone { - namespace, - track: "video".into(), - start: ietf::Location { group: 3, object: 0 }, - end: ietf::Location { group: 6, object: 0 }, - }, - }; - let mut body = Vec::new(); - ietf::Fetch { - request_id: RequestId(3), - subscriber_priority: 128, - group_order: ietf::GroupOrder::Ascending, - fetch_type, - range_filters: false, - fill_timeout: false, - properties_wanted: false, - epoch: None, - } - .encode_msg(&mut Encoder::new(&mut body, version.into()), version) - .unwrap(); - let downstream = ScriptedSession::per_stream(vec![vec![]]); - let peer = peer::PeerSetup::default(); - peer.set(peer::Peer::default()); - let publisher = crate::ietf::Publisher::new( - crate::time::sim(), - crate::driver::Transport::new(downstream.clone()), - origin.consume(), - Control::new(false), - None, - peer, - version, - ); - let stream = Stream::open(&mut crate::driver::Transport::new(downstream.clone()), version) - .await - .unwrap(); - moq_clock::timeout( - Duration::from_secs(1), - publisher - .handle_stream(ietf::Fetch::ID, ietf::Body(body.into()), stream, false) - .unwrap(), - ) - .await - .expect("publisher"); - let mut streams = std::collections::BTreeMap::>::new(); - for (id, event) in downstream.log.trail() { - if let Sent::Write(bytes) = event { - streams.entry(id).or_default().extend(bytes); - } - } - let bytes: Vec = streams.into_values().flatten().collect(); - let mut decoder = Decoder::new(&bytes, version.into()); - assert_eq!(decoder.varint().unwrap(), ietf::FetchOk::ID, "{version}"); - let ok = ietf::FetchOk::decode(&mut decoder, version).unwrap(); - assert!(ok.end_of_track, "{version}"); - assert_eq!(decoder.varint().unwrap(), ietf::FetchHeader::TYPE); - ietf::FetchHeader::decode(&mut decoder, version).unwrap(); - - // The next relay's fill of the same run reads that one stream. - let mut run = GroupFetchRun::new(version, decoder.rest().to_vec()).await; + let mut objects = group_fetch_objects(4, 0, &[b"4-0", b"4-1"]); + // Later Group IDs are deltas, so 1 advances from group 4 to group 6. + objects.extend(group_fetch_objects(1, 0, &[b"6-0", b"6-1"])); + let mut run = GroupFetchRun::new(version, objects).await; let dynamic = run.track.requests(); let consumer = run.track.consume(); - let mut filling = Box::pin(consumer.fill_groups(3..7, 0)); - assert!(futures::poll!(filling.as_mut()).is_pending()); + run.track.start_at(7).unwrap(); + run.track.create_group(7u64.into()).unwrap().finish().unwrap(); + let mut reading = consumer + .subscribe(Some( + track::Subscription::default() + .with_range(3..7) + .with_order(crate::track::Order::Ascending) + .with_max_delay(None), + )) + .await + .unwrap(); + assert!(futures::FutureExt::now_or_never(reading.recv_group()).is_none()); let fill = dynamic.requested_fill().await.unwrap(); assert_eq!( (fill.start().group, fill.end()), @@ -14509,10 +14445,7 @@ mod stitch_tests { .await .expect("recv"); assert!(res.is_ok(), "{version}: {res:?}"); - let _filled = moq_clock::timeout(Duration::from_secs(1), filling) - .await - .expect("filled"); - assert_eq!(consumer.cached_groups(0..10), [4, 6], "{version}"); + assert_eq!(consumer.cached_groups(0..7), [4, 6], "{version}"); for sequence in [4, 6] { let mut group = consumer.fetch_group(sequence, None).await.unwrap(); for object in 0..2 { diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 76c62edffd4..c668baec9ec 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -242,22 +242,6 @@ impl Consumer { self.state.read().copy.clone() } - /// Poll for the serving route's copy, as a fetch waits for one: `None` once the logical - /// track ends without one. - #[cfg(feature = "async")] - pub(super) fn poll_serving(&self, waiter: &kio::Waiter) -> Poll> { - loop { - let (generation, copy) = { - let route = self.state.read(); - (route.generation, route.copy.clone()) - }; - if copy.is_some() || self.end().is_some() { - return Poll::Ready(copy); - } - ready!(self.poll_changed(generation, waiter)); - } - } - /// The copy the front parked, while it serves none and may still serve one again. pub(super) fn parked(&self) -> Option { let route = self.state.read(); @@ -5190,27 +5174,6 @@ mod test { assert_eq!(&frame.payload[..], &[2]); } - /// A span fill made before the front serves a copy waits for one and asks it for the - /// whole span, rather than resolving with nothing filled. - #[test] - #[cfg(feature = "async")] - fn a_span_fill_waits_for_the_serving_copy() { - let routes = Producer::new(); - let logical = logical(&routes); - let consumer = logical.consume(); - let mut filling = Box::pin(consumer.fill_groups(0..4, 0)); - assert!(filling.as_mut().now_or_never().is_none(), "no copy serves yet"); - let a = copy(); - let dynamic = a.requests(); - routes.serve(a.consume()); - assert!( - filling.as_mut().now_or_never().is_none(), - "the copy's fill is unanswered" - ); - let fill = dynamic.requested_fill().now_or_never().unwrap().unwrap(); - assert_eq!((fill.start().group, fill.end()), (0, 4)); - } - #[test] #[cfg(feature = "async")] fn a_pending_recovery_fetch_does_not_disable_expiry() { diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index b0029e4fa99..526771538e5 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -4930,52 +4930,6 @@ impl Consumer { }) } - /// Fill the groups of `span` the cache misses and nothing recorded dropped, asking a - /// [`Requests`] once per run of misses so it fetches each run at once, and resolve once - /// every run is answered. The returned guard keeps the fills wanted, so the groups they - /// accepted stay cached while the caller reads them. A track with no handler resolves at - /// once, and a front's logical track fills through the copy the front assigns it. - #[cfg(feature = "async")] - pub(crate) fn fill_groups(&self, span: ops::Range, priority: i8) -> kio::Pending { - // Held while waiting: a front assigns its logical track a copy only while it is read. - let state = self.resolve(None).0; - let routes = state.read().routes.clone(); - if let Some(routes) = routes { - return kio::Pending::new(Filling(FillStage::Front { - routes, - read: state, - span, - priority, - serving: None, - })); - } - let (fills, holes, cached) = { - let state = state.read(); - let cached = state - .lookup - .range(span.clone()) - .filter(|(_, slot)| slot.serves_end()) - .map(|(sequence, _)| *sequence) - .collect(); - let holes: Vec> = miss::holes(&state, span) - .into_iter() - .flat_map(|hole| state.drops.gaps(&hole).collect::>()) - .collect(); - (state.fills.clone(), holes, cached) - }; - let joins = holes - .into_iter() - .flat_map(|hole| ask_span(&fills, hole, priority)) - .collect(); - kio::Pending::new(Filling(FillStage::Asked { - fills, - priority, - joins, - checked: 0, - cached, - })) - } - /// The sequences of `span` the cache holds, lowest first. #[cfg(feature = "async")] pub(crate) fn cached_groups(&self, span: ops::Range) -> Vec { @@ -5249,162 +5203,6 @@ fn queue(fills: &kio::Shared, sequence: u64, request: &FetchOptions) joins.into_iter().next().map(Box::new) } -/// The fills a [`Consumer::fill_groups`] asked, kept wanted until it drops. -#[cfg(feature = "async")] -pub(crate) struct Filled { - /// Held to keep the fills wanted. - joins: Vec, - /// The sequences the cache held when the fill began, lowest first. - cached: Vec, - /// A front's logical track, held read so the front keeps the filled copy serving it. - _read: Option>, -} - -#[cfg(feature = "async")] -impl Filled { - /// The sequences of `span` the cache held when the fill began or the fills accepted, - /// lowest first. Their source has them even once the cache evicts them, so reading one - /// again fetches it again. - pub(crate) fn held(&self, span: ops::Range) -> std::collections::BTreeSet { - let cached = self.cached.iter().copied().filter(|sequence| span.contains(sequence)); - self.joins - .iter() - .flat_map(|join| join.outcome().read().accepted_in(&span).collect::>()) - .chain(cached) - .collect() - } - - /// Why the fills refused sequences of `span`, lowest first. - pub(crate) fn refused(&self, span: ops::Range) -> Vec { - self.joins - .iter() - .flat_map(|join| join.outcome().read().refused_in(&span).cloned().collect::>()) - .collect() - } -} - -/// The pollable state of a [`Consumer::fill_groups`], resolving to the [`Filled`] guard. -#[cfg(feature = "async")] -pub(crate) struct Filling(FillStage); - -#[cfg(feature = "async")] -enum FillStage { - /// A front's logical track, filling through the copy the front serves it from. - Front { - routes: super::resume::Consumer, - read: kio::Consumer, - span: ops::Range, - priority: i8, - /// The serving copy's fill, once the front assigned one. - serving: Option>, - }, - /// Waiting for each run asked to be answered. - Asked { - fills: kio::Shared, - priority: i8, - joins: Vec, - /// How many of `joins` were answered and checked for a refusal to ask again. - checked: usize, - /// The sequences the cache held when the fill began. - cached: Vec, - }, - /// Resolved. - Done, -} - -#[cfg(feature = "async")] -impl kio::Task for Filling { - type Output = Filled; - - fn poll(&mut self, waiter: &kio::Waiter) -> Poll { - self.0.poll(waiter) - } -} - -#[cfg(feature = "async")] -impl FillStage { - fn poll(&mut self, waiter: &kio::Waiter) -> Poll { - match self { - Self::Front { - routes, - span, - priority, - serving, - .. - } => { - if serving.is_none() { - // The front ended without a copy, so nothing fills. - let Some(copy) = ready!(routes.poll_serving(waiter)) else { - return Poll::Ready(self.resolve(Vec::new(), Vec::new())); - }; - *serving = Some(Box::new(copy.fill_groups(span.clone(), *priority).into_inner().0)); - } - let filled = ready!(serving.as_mut().expect("set above").poll(waiter)); - Poll::Ready(self.resolve(filled.joins, filled.cached)) - } - Self::Asked { - fills, - priority, - joins, - checked, - cached, - } => loop { - for join in &joins[*checked..] { - ready!(join.poll_answered(waiter)); - } - // Joined after a tighter budget went on the wire, which judged a group too old - // for that budget, not ours: ask again with ours, as a fetch does. Terminates, - // since a new fill asks no budget at all. - let retry: Vec> = joins[*checked..] - .iter() - .flat_map(|join| join.outcome().read().budget_old_in(&join.groups)) - .collect(); - *checked = joins.len(); - if retry.is_empty() { - return Poll::Ready(Filled { - joins: std::mem::take(joins), - cached: std::mem::take(cached), - _read: None, - }); - } - for run in retry { - joins.extend(ask_span(fills, run, *priority)); - } - }, - Self::Done => panic!("polled after resolving"), - } - } - - /// Resolve a front's fill with what its serving copy held and asked. - fn resolve(&mut self, joins: Vec, cached: Vec) -> Filled { - let Self::Front { read, .. } = std::mem::replace(self, Self::Done) else { - unreachable!("only a front resolves through its copy"); - }; - Filled { - joins, - cached, - _read: Some(read), - } - } -} - -/// Ask the fills for the groups of `span`, joining any that cover them. -#[cfg(feature = "async")] -fn ask_span(fills: &kio::Shared, span: ops::Range, priority: i8) -> Vec { - let ask = fill::Ask { - start: Position::group(span.start), - end: span.end, - frame_end: None, - asking: fill::Asking { - priority, - // A fetch names its groups outright, so only the publisher's max age applies. - max_delay: None, - order: Order::Ascending, - }, - }; - fills.lock().join(fills, &ask).0 -} - /// One transport request, whose end may cut the stored copy short. #[derive(Clone)] pub(crate) struct FetchOptions { @@ -15583,7 +15381,6 @@ mod test { consumer.cached_groups(0..1).is_empty(), "a bounded prefix is no whole cached group" ); - let all = consumer.fill_groups(0..1, 0); // A narrower bound reuses the same stored prefix. assert!( consumer @@ -15606,7 +15403,6 @@ mod test { assert!(whole.read_frame().await.unwrap().is_some()); } assert!(whole.read_frame().await.unwrap().is_none()); - assert_eq!(all.await.held(0..1), BTreeSet::from([0])); assert_eq!(producer.state.read().lookup[&0].copies().count(), 1); } @@ -19784,82 +19580,6 @@ mod test { assert!(matches!(whole.read_frame().now_or_never(), Some(Ok(None)))); } - /// Filling a span asks once per run of misses, and a run is served up to a group the - /// cache gained since, which it skips: the handler fetches each run at once. - #[moq_clock::test] - #[cfg(feature = "async")] - async fn a_filled_span_is_served_a_run_at_a_time() { - let producer = track_producer("test", Info::timed()); - let dynamic = producer.requests(); - let consumer = producer.consume(); - producer - .create_group(group::Info { sequence: 2 }) - .unwrap() - .finish() - .unwrap(); - - let mut filling = Box::pin(consumer.fill_groups(0..8, 0)); - assert!(futures::poll!(filling.as_mut()).is_pending()); - let low = dynamic.requested_fill().await.unwrap(); - let high = dynamic.requested_fill().await.unwrap(); - assert_eq!((low.start().group, low.end()), (0, 2)); - assert_eq!((high.start().group, high.end()), (3, 8)); - - // Group 5 lands live while the high run waits on its source. - producer - .create_group(group::Info { sequence: 5 }) - .unwrap() - .finish() - .unwrap(); - assert_eq!(low.next_run(), Some(0..2)); - assert_eq!(high.next_run(), Some(3..5)); - high.drop_groups(3..5); - assert_eq!(high.next_run(), Some(6..8)); - high.accept(6, None).unwrap().finish().unwrap(); - assert_eq!(high.next_run(), Some(7..8)); - high.drop_groups(7..8); - assert_eq!(high.next_run(), None); - low.drop_groups(0..2); - - let std::task::Poll::Ready(filled) = futures::poll!(filling.as_mut()) else { - panic!("every run is answered"); - }; - assert_eq!(consumer.cached_groups(0..8), [2, 5, 6]); - - // A group cached when the fill began, or accepted since, stays named once the cache - // evicts it, so the answer reads it again. Group 5, which the live feed answered, - // counts too. - evict(&producer, 2); - evict(&producer, 6); - assert_eq!(consumer.cached_groups(0..8), [5]); - assert_eq!(filled.held(0..8).into_iter().collect::>(), [2, 5, 6]); - } - - /// A span fill that joined after a tighter budget went on the wire is not answered by - /// that budget's `Old`: it asks again with its own, as a fetch does. - #[moq_clock::test] - #[cfg(feature = "async")] - async fn a_span_fill_asks_again_after_another_budget_is_too_old() { - let producer = track_producer("test", Info::timed()); - let dynamic = producer.requests(); - let consumer = producer.consume(); - - let tight = consumer.fetch_group(5, group::Fetch::default().with_max_delay(Duration::from_secs(1))); - let req = dynamic.requested_fill_group().now_or_never().unwrap().unwrap(); - let mut filling = Box::pin(consumer.fill_groups(5..6, 0)); - assert!(futures::poll!(filling.as_mut()).is_pending()); - - req.reject(Error::Old); - assert!(matches!(tight.await, Err(Error::Old))); - assert!(futures::poll!(filling.as_mut()).is_pending()); - - let fill = dynamic.requested_fill().now_or_never().unwrap().unwrap(); - assert_eq!((fill.start().group, fill.end(), fill.max_delay()), (5, 6, None)); - fill.accept(5, None).unwrap().finish().unwrap(); - let filled = filling.await; - assert_eq!(filled.held(5..6).into_iter().collect::>(), [5]); - } - /// A drop judges the subscription, not a fetch: a fetched head still arriving keeps /// serving the group to its end after the drop ends the live copy. #[moq_clock::test] diff --git a/rs/moq-net/src/model/track/fill.rs b/rs/moq-net/src/model/track/fill.rs index c71757c73e3..c529bd39465 100644 --- a/rs/moq-net/src/model/track/fill.rs +++ b/rs/moq-net/src/model/track/fill.rs @@ -203,33 +203,6 @@ impl Outcome { self.partial.get(&sequence).filter(|copy| copy.covers(frames)).cloned() } - /// The sequences of `span` accepted, lowest first. - #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(super) fn accepted_in(&self, span: &ops::Range) -> impl Iterator { - self.accepted - .overlapping(span) - .flat_map(|run| run.start.max(span.start)..run.end.min(span.end)) - .collect::>() - .into_iter() - } - - /// Why the sequences of `span` were refused, lowest first. - #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(super) fn refused_in(&self, span: &ops::Range) -> impl Iterator { - self.refused.overlapping(span).map(|(_, refusal)| &refusal.err) - } - - /// The runs of `span` refused as too old under a budget, lowest first: a reader with no - /// budget of its own may still be served them. - #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(super) fn budget_old_in(&self, span: &ops::Range) -> Vec> { - self.refused - .overlapping(span) - .filter(|(_, refusal)| matches!(refusal.err, Error::Old) && refusal.asked.is_some()) - .map(|(run, _)| run.start.max(span.start)..run.end.min(span.end)) - .collect() - } - /// What was answered within `wanted`: accepted, or refused with why and the budget /// the fill asked with. #[expect( @@ -354,19 +327,6 @@ impl Join { pending.result.weak().same_channel(&self.attempt) } - /// Poll until every group this reader still reads is answered, or the fill ends. - #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(super) fn poll_answered(&self, waiter: &kio::Waiter) -> Poll<()> { - self.outcome() - .poll(waiter, |outcome| { - match self.held.iter().all(|run| outcome.resolved.gaps(run).next().is_none()) { - true => Poll::Ready(()), - false => Poll::Pending, - } - }) - .map(drop) - } - /// Whether the reader holds none of the fill any more. pub(super) fn is_empty(&self) -> bool { self.held.is_empty() diff --git a/rs/moq-net/src/test_interop.rs b/rs/moq-net/src/test_interop.rs index 68c639c2862..7ac8e15015a 100644 --- a/rs/moq-net/src/test_interop.rs +++ b/rs/moq-net/src/test_interop.rs @@ -25,7 +25,7 @@ pub(crate) fn fin(version: &str, started: bool, clean: bool, responses: Vec) } /// Rust updates keep the JS subscription alive, cover both unseen groups after narrowing, -/// and fill cached history with the update's ID and the actual group deltas on draft-22. +/// and fill the largest group's head under the update's ID on draft-22. #[test] #[ignore = "requires Bun; run by just test interop"] fn ietf_range_updates_interop() { @@ -89,10 +89,10 @@ fn ietf_range_updates_interop() { if draft22 { updates.push(update( 2, - range(0, None), + range(3, None), Some(ietf::Fill { group_order: Some(ietf::GroupOrder::Ascending), - filter: Some(range(0, None)), + filter: Some(range(3, None)), range_filters: false, }), )); @@ -192,7 +192,7 @@ fn ietf_range_updates_interop() { assert_eq!( filled, if draft22 { - (0..4).flat_map(|group| [(group, 0), (group, 1)]).collect::>() + (3..4).flat_map(|group| [(group, 0), (group, 1)]).collect::>() } else { vec![] }, diff --git a/rs/moq-net/tests/integration/route_change.rs b/rs/moq-net/tests/integration/route_change.rs index d384e438774..89dbd6a0288 100644 --- a/rs/moq-net/tests/integration/route_change.rs +++ b/rs/moq-net/tests/integration/route_change.rs @@ -304,20 +304,19 @@ async fn route_flaps(version: &str) { let triggers = [Trigger::BetterRoute, Trigger::BetterRouteDies, Trigger::Disconnect]; let mut expected = Vec::new(); let mut seen = Vec::new(); - for sequence in 0..=triggers.len() as u64 { - let mut group = topology.track.append_group().unwrap(); - for frame in 0..FRAMES { - if frame == FRAMES / 2 - && let Some(trigger) = triggers.get(sequence as usize) - { - topology.trigger(*trigger).await; - } - group.write_frame(Timestamp::ZERO, payload(sequence, frame)).unwrap(); - expected.push((sequence, payload(sequence, frame))); - seen.push(next(&mut rx).await); + // Keep every recovery within the largest group's head, which our IETF publishers serve. + let mut group = topology.track.append_group().unwrap(); + for frame in 0..FRAMES * (triggers.len() as u64 + 1) { + if frame % FRAMES == FRAMES / 2 + && let Some(trigger) = triggers.get((frame / FRAMES) as usize) + { + topology.trigger(*trigger).await; } - group.finish().unwrap(); + group.write_frame(Timestamp::ZERO, payload(0, frame)).unwrap(); + expected.push((0, payload(0, frame))); + seen.push(next(&mut rx).await); } + group.finish().unwrap(); assert_eq!(seen, expected, "{version}"); settle().await; diff --git a/rs/moq-net/tests/integration/subscribe_ranges.rs b/rs/moq-net/tests/integration/subscribe_ranges.rs index d317e2b6f42..46c5509ecdf 100644 --- a/rs/moq-net/tests/integration/subscribe_ranges.rs +++ b/rs/moq-net/tests/integration/subscribe_ranges.rs @@ -270,15 +270,15 @@ async fn initial_mixed_ranges_keep_the_absolute_history() { assert_eq!(next(&mut warm).await, None); } let subscription = Subscription::default() - .with_ranges([Range::groups(1..3), Range::live()]) + .with_ranges([Range::groups(1..2), Range::live()]) .with_max_delay(None); let mut reader = subscribe(&fixture.remote, subscription).await; let mut got = Vec::new(); - for _ in 0..3 { + for _ in 0..2 { got.push(next(&mut reader).await.expect("a mixed range group is missing")); } got.sort_unstable(); - assert_eq!(got, [1, 2, 5], "{version} relay={relay} finished={finished}"); + assert_eq!(got, [1, 5], "{version} relay={relay} finished={finished}"); } } } @@ -330,7 +330,7 @@ async fn draft22_updates_both_bounds_across_cached_groups() { ) .await .expect("the narrowed range never reached the publisher"); - reader.update(initial.with_range(0..)).unwrap(); + reader.update(initial.with_range(6..)).unwrap(); let mut group = fixture .track .create_group(moq_net::group::Info { sequence: 6 }) @@ -452,14 +452,14 @@ async fn draft18_updates_narrow_then_widen() { let version = "moq-transport-18"; for relay in [false, true] { let mut fixture = range_fixture_version(version, relay).await; - let initial = Subscription::default().with_range(0..).with_max_delay(None); + let initial = Subscription::default().with_range(5..).with_max_delay(None); let mut reader = subscribe(&fixture.remote, initial.clone()).await; let mut seen = Vec::new(); - for _ in 0..6 { + for _ in 0..1 { seen.push(next(&mut reader).await.expect("initial history")); } seen.sort_unstable(); - assert_eq!(seen, [0, 1, 2, 3, 4, 5], "{version} relay={relay}"); + assert_eq!(seen, [5], "{version} relay={relay}"); reader.update(initial.clone().with_range(4..5)).unwrap(); moq_clock::timeout( TIMEOUT, @@ -576,16 +576,15 @@ async fn empty_ranges_retire_the_wire_and_can_widen_again() { } } -/// A fill that widens an initially empty copy must finish every completed group, -/// including the snapshot head, before the publisher produces another group. +/// A largest-group fill finishes its snapshot before the publisher produces another group. #[moq_clock::test] -async fn draft22_cold_widening_finishes_every_filled_group() { +async fn draft22_cold_widening_finishes_the_largest_group() { for relay in [false, true] { let mut fixture = range_fixture_history("moq-transport-22", relay, 0).await; let initial = Subscription::default().with_range(2..).with_max_delay(None); let mut reader = subscribe(&fixture.remote, initial.clone()).await; - for start in [0, 2, 4] { - let future = start + 2; + for start in [0, 1, 2] { + let future = start + 1; reader.update(initial.clone().with_range(future..)).unwrap(); moq_clock::timeout( TIMEOUT, @@ -640,7 +639,7 @@ async fn draft22_cold_widening_finishes_every_filled_group() { ); } sequences.sort_unstable(); - assert_eq!(sequences, [start, start + 1]); + assert_eq!(sequences, [start]); } } } diff --git a/rs/moq-net/tests/integration/warm_backfill.rs b/rs/moq-net/tests/integration/warm_backfill.rs index 7d77ced22aa..63f282064bd 100644 --- a/rs/moq-net/tests/integration/warm_backfill.rs +++ b/rs/moq-net/tests/integration/warm_backfill.rs @@ -24,9 +24,8 @@ const GROUPS: u64 = 10; /// Reaches every group, however long the readers take. const BUDGET: Duration = Duration::from_secs(100); -/// Moq-transport-22 is left out: its cold relay serves an absolute floor only from the -/// live edge. -const VERSIONS: &[&str] = &["moq-lite-05", "moq-lite-06", "moq-lite-07-wip", "moq-transport-18"]; +/// Moq-transport publishers refuse multi-group history; the live subscription still works. +const VERSIONS: &[&str] = &["moq-lite-05", "moq-lite-06", "moq-lite-07-wip"]; fn origin(hop: u64) -> origin::Producer { let (origin, driver) = origin::Producer::new(origin::Config::new(Hop::new(hop).unwrap())); @@ -84,3 +83,39 @@ async fn a_newcomer_on_a_warm_relay_gets_the_backfill() { assert_eq!(warm, cold, "{version}: warm"); } } + +/// A refused history fill or FETCH never ends its live subscription. +#[moq_clock::test] +async fn a_refused_transport_backfill_keeps_delivering_live() { + for version in ["moq-transport-18", "moq-transport-22"] { + let publisher = origin(1); + let relay = origin(2); + let _link = peer(version.parse::().unwrap(), &publisher, &relay).await; + let broadcast = publisher.create_broadcast("live").unwrap(); + let source = broadcast + .create_track("video", Info::default().with_timescale(Timescale::MILLI)) + .unwrap(); + for sequence in 0..GROUPS { + let mut group = source.create_group(sequence.into()).unwrap(); + group + .write_frame(Timestamp::from_millis(sequence * 1000).unwrap(), &b"x"[..]) + .unwrap(); + group.finish().unwrap(); + } + broadcast.announce(Default::default()).unwrap(); + let mut subscriber = moq_clock::timeout(TIMEOUT, subscribe(&relay)).await.unwrap(); + let got = moq_clock::timeout(TIMEOUT, received(&mut subscriber)).await.unwrap(); + assert_eq!(got, BTreeSet::from([GROUPS - 1]), "{version}: only the live edge"); + let mut group = source.create_group(GROUPS.into()).unwrap(); + group + .write_frame(Timestamp::from_millis(GROUPS * 1000).unwrap(), &b"live"[..]) + .unwrap(); + group.finish().unwrap(); + let got = moq_clock::timeout(TIMEOUT, subscriber.recv_group()) + .await + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(got.sequence, GROUPS, "{version}: still live after the refused backfill"); + } +} From 5b2485fb4749dbed31e13eb9229b694d3956a5a7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 28 Oct 2026 22:40:48 -0700 Subject: [PATCH 3/9] docs(quest): record the refused-fill widening blocker Co-Authored-By: GPT-6.1 Sol --- doc/concept/standard.md | 2 +- quest/m1/ietf-publisher-one-group.md | 38 ++++--------------- .../tests/integration/subscribe_ranges.rs | 8 ++-- 3 files changed, 12 insertions(+), 36 deletions(-) diff --git a/doc/concept/standard.md b/doc/concept/standard.md index 69fae436f31..c43b645061b 100644 --- a/doc/concept/standard.md +++ b/doc/concept/standard.md @@ -34,7 +34,7 @@ subset drops. What a peer actually observes against this implementation: - **Strict SETUP and parameters.** Repeated unknown `SETUP` options, GREASE included, are accepted if well-formed; a repeated known option is rejected. A `GROUP_ORDER` other than Ascending (1) or Descending (2) closes the session. An unknown message parameter, or one defined only for another message, closes the session. Other malformed control input closes the session with the draft's code too, such as a server `SETUP` carrying `PATH` or `AUTHORITY`, a namespace field ending in `/` (it would map to a path with an empty segment), a request cut short by FIN, or a second `GOAWAY` on one stream; a first `GOAWAY` on a request stream is legal. - **One `SETUP` credential**, forwarded to the [auth server](/bin/relay/auth#the-contract) unverified. A second `SETUP` token closes the session. We advertise no token cache, so an alias reference in `SETUP` closes it with `PROTOCOL_VIOLATION`, one on a request with `UNKNOWN_AUTH_TOKEN_ALIAS`, and a registration on a request with `AUTH_TOKEN_CACHE_OVERFLOW`. Where the Rust side authorizes the peer or verifies tokens, a token on a `SUBSCRIBE` or `PUBLISH_NAMESPACE` is verified like an in-band session token and authorizes that one request alone, even where that authorization already covers it; it never widens the session. It is refused with `NOT_SUPPORTED` when nothing verifies it, when it is empty, or when its Token Type is anything but 0 (out of band). A side that does neither, JavaScript included, ignores the token. Rust refuses a request carrying more than one distinct token with `NOT_SUPPORTED`, since it verifies one; JavaScript, which verifies none, ignores them all. A token-authorized `SUBSCRIBE` still stays within the grant the peer gave us. A token-authorized `PUBLISH_NAMESPACE` needs a grant covering everything beneath the namespace, and the tracks we subscribe under it rely on that announce instead of our limit, ending with it. An update renews the token, and a refused renewal ends only that request. Rust and JavaScript send the token a route names (`Route::auth`) on its `PUBLISH_NAMESPACE`, and swapping it renews the request with `REQUEST_UPDATE`; the inline `NAMESPACE` that answers a solicited `SUBSCRIBE_NAMESPACE` carries no token. A token added to a reconnecting connection's auth handle rides every session it makes, so the announcement keeps it across a reconnect; one added to a single session's handle ends with that session. A token on any other request is decoded and grants nothing, and a request repeating one is refused with `MALFORMED_AUTH_TOKEN`. The negotiated [MoQ Auth extension](/concept/moq-lite#authorization) accepts additional session tokens in band. - **Pause delivery.** `FORWARD=0` on a `SUBSCRIBE` or its update drops the subscription, closing its data streams and holding nothing upstream, while the request and its slot stay. `FORWARD=1` subscribes again as a new `SUBSCRIBE` with the same filter would, so nothing carries across the pause: its `REQUEST_OK` names the live edge it joined at. A paused update may change only priority and max age. -- **Refused, not fatal.** A legal request this stack does not serve is rejected on its own and the session stays up: the fetch forms above get `NOT_SUPPORTED`. Range Filters get `INVALID_FILTER` on draft 22 because the advertised filter limit is zero. On draft 22, namespaces whose first tuple field is exactly `.` or `.session` get `DOES_NOT_EXIST` before application lookup; other dot prefixes still reach the application. Subscription updates accept priority, negotiated subscriber max age, and delivery timeouts. Rust also carries range changes, keeps bounded subscriptions updatable, renews request tokens, and judges `NEW_GROUP_REQUEST` as a join (see dynamic groups below). On draft 22, a wider or historical fill fails on its separate fetch stream while its subscription stays open; a largest-group fill hands its head to the live subgroup tail. Other supported-but-unimplemented preferences end that subscription; a `TRACK_PROPERTY_FILTER` there closes the session with `PROTOCOL_VIOLATION`, since it belongs only to `SUBSCRIBE_TRACKS`. A namespace update may change its prefix; suffixes after the acknowledgement are relative to that new prefix. +- **Refused, not fatal.** A legal request this stack does not serve is rejected on its own and the session stays up: multi-group `FETCH` and joining requests reaching before the joined group get `NOT_SUPPORTED`. Range Filters get `INVALID_FILTER` on draft 22 because the advertised filter limit is zero. On draft 22, namespaces whose first tuple field is exactly `.` or `.session` get `DOES_NOT_EXIST` before application lookup; other dot prefixes still reach the application. Subscription updates accept priority, negotiated subscriber max age, and delivery timeouts. Rust also carries range changes, keeps bounded subscriptions updatable, renews request tokens, and judges `NEW_GROUP_REQUEST` as a join (see dynamic groups below). On draft 22, a wider or historical fill fails on its separate fetch stream while its subscription stays open; a largest-group fill hands its head to the live subgroup tail. Other supported-but-unimplemented preferences end that subscription; a `TRACK_PROPERTY_FILTER` there closes the session with `PROTOCOL_VIOLATION`, since it belongs only to `SUBSCRIBE_TRACKS`. A namespace update may change its prefix; suffixes after the acknowledgement are relative to that new prefix. - **Objects and datagrams.** Only subgroup 0 is accepted: a stream for any other subgroup, whether its header gives the ID or takes it from the first Object ID, is stopped and the session stays up. Rust relays preserve Object IDs, gaps, object properties, FIRST\_OBJECT, and END\_OF\_GROUP status objects between moq-transport peers for subgroup 0, including across a moq-lite-07 hop between two relays that negotiate the [ietf](/draft/moq-ietf) extension. Datagrams retain their IDs, properties, priority, and status. Plain Rust readers and moq-lite peers without the extension see consecutive frames and only normal datagram payloads at Object ID 0; other datagram objects and statuses are dropped on those surfaces. JavaScript supports only normal datagrams at Object ID 0. A malformed datagram closes the session; a relayed one the downstream draft cannot carry is dropped. A relayed Timestamp leaves in the units the downstream response declares, and is dropped where it declares none, as for a plain frame; an immutable one that would change or vanish is refused. On a timed track the frame holds each object's time, so a mutable Timestamp that only repeats the immutable one is not restated; a status object has no frame, so it keeps its Timestamp among its properties. Only property 0x10 is read as a Timestamp. Property 0x06 stays opaque and keeps its value regardless of timestamp units; it is dropped between drafts 18 and 22, or refused when immutable, because draft 22 assigns it to SUBGROUP\_DELIVERY\_TIMEOUT. A status object carrying properties is refused. A `FETCH` has no status objects, so its response ends at the last frame. A group cannot resume across route changes by frame ordinal once Object IDs skip ahead of the ordinals on either route; recovery refuses it instead of splicing different objects. - **Dynamic groups.** A track advertises `DYNAMIC_GROUPS` when its publisher sets a refresh ratio, and a track received with `DYNAMIC_GROUPS` gets a ratio of 0. A `NEW_GROUP_REQUEST` counts as one more subscriber joining, which a `SUBSCRIBE` already is: it asks for a new group only when the newest group's bytes after its first frame exceed the ratio times the first frame. Joins coalesce until the asked-for group arrives. A Rust relay sends `NEW_GROUP_REQUEST` upstream, on the `SUBSCRIBE` or in a `REQUEST_UPDATE`, once a join wants a group past the newest it holds; a relay holding nothing yet asks for none. On a track without a ratio, an update carrying one ends the subscription with `UPDATE_FAILED`. A moq-lite-07 hop (`moq-lite-07-wip`, opt-in) carries the same demand natively: the ratio in `TRACK_INFO` and the asked-for group in `SUBSCRIBE_UPDATE`. Earlier moq-lite versions carry neither, so a track received over one never refreshes. - **Priority.** Higher is served first. This stack's priorities run from -128 to 127 with 0 unset, and go out on the IETF wire, which ranks lower first, as `127 - priority`: an unset priority is IETF 127. An omitted subscriber or publisher priority reads as the drafts' default of IETF 128, one step below unset, so a relay forwards it unchanged. A relay's upstream subscription carries the highest priority its subscribers ask for and updates it as they come and go; its group order stays as the `SUBSCRIBE` asked, since no draft lets an update change it. diff --git a/quest/m1/ietf-publisher-one-group.md b/quest/m1/ietf-publisher-one-group.md index 63de278a6b4..d6d3950b6ad 100644 --- a/quest/m1/ietf-publisher-one-group.md +++ b/quest/m1/ietf-publisher-one-group.md @@ -62,37 +62,13 @@ Decision prompts: - Rank: ✅ before the other FETCH quests, right after [moq-transport 18 and 22 only](/quest/m1/ietf-drop/README.md); at m1's end. -Implementation: - -- **Rust publisher.** Refuse a standalone FETCH spanning several groups with - NOT_SUPPORTED, alongside the existing joining-FETCH refusal - (`a_joining_fetch_spanning_several_groups_is_refused`). Resolve a - draft-22 fill to `FillServe::Group` for the largest group or - `FillServe::Unsupported`, deleting `FillServe::Range` and the range half of - `fill_range`. Accept FILL_TIMEOUT instead of the `"FILL_TIMEOUT not - supported"` refusal. Delete the end-location, `LOCATION_FILTER`, and - multi-group ordering rules only the range path needs, with their tests. -- **JS publisher.** Same rules in `#answerFetch` and `#runFill`. Remove the - FILL_TIMEOUT deadline and its timed-out markers from - [moq-dev/moq#7051](https://github.com/moq-dev/moq/pull/7051). Keep the End - of Range encoding, which - [JS evicted FETCH groups are unknown](/quest/m1/js-fetch-evicted-unknown.md) - uses for a group lost after FETCH_OK. -- **Tests.** Unit tests in both publishers for each refusal and for each - kept form on drafts 18 and 22. `warm_backfill` in - `rs/moq-net/tests/integration` keeps draft 22 out of its backfill and - instead asserts that a `0..` fill fails while the subscription delivers - live. This replaces the deleted cold-floor quest from - [moq-dev/moq#7168](https://github.com/moq-dev/moq/pull/7168): our publisher - now refuses that fill on purpose. Narrow `moq fetch` and the interop FETCH - cells to one group where they asked for more. -- **Compat.** If a released-version cell in `test/interop/compat.sh` breaks - (the FETCH capability cell, or a cell that fetched a range), add a planned - break to `test/interop/compat/planned-breaks.json`. -- **Docs.** Rewrite the "History is fetched in ranges" and "Refused, not - fatal" bullets of `doc/concept/standard.md`. - -Run `just check`, `just test interop --all`, and the compat lane. +Implementation is checkpointed in [moq-dev/moq#7204](https://github.com/moq-dev/moq/pull/7204): both publishers serve one-group FETCH and the largest-group fill, ignore FILL_TIMEOUT, and no longer serve ranges. Subscriber range requests are unchanged. The publisher-only model helpers and timeout machinery are removed; fixtures and benchmarks use the kept forms. + +Remaining decision: a refused fill can reset before its FETCH_HEADER is received. Our subscriber cannot associate that stream with its request, so its sole fill slot remains outstanding. `run_subscribe` defers every subsequent widening while that slot is outstanding. The existing `draft22_updates_both_bounds_across_cached_groups` integration test now stalls after `2..4 -> 0..6 -> 4..5 -> 0..`, losing live group 6 even though the subscription remains open. The memory transport's `reset_drops_queued_messages` test and the subscriber's deferred-update comment describe this case. + +Recommendation: permit a narrow subscriber fill-ownership fix that lets later updates retire an unidentifiable failed fill without changing the range FETCH or multi-group fill requests it sends. That exceeds the settled publisher-only scope and needs the maintainer's decision. Accepting permanently stalled widening is the alternative, not a test change to conceal it. + +After that decision, finish local checks and `just test interop --all`, then remove this quest and its links. The compatibility lane currently stops before session cells because accepted pins name moq-cli 0.14.8 while 0.14.9 is published. [moq-dev/moq#7061](https://github.com/moq-dev/moq/pull/7061) already owns that blocked remeasurement; do not repin it here without new evidence of a break from this change. Public API: none. Wire: our publishers refuse multi-group FETCH with NOT_SUPPORTED, fail multi-group fills, and ignore FILL_TIMEOUT. diff --git a/rs/moq-net/tests/integration/subscribe_ranges.rs b/rs/moq-net/tests/integration/subscribe_ranges.rs index 46c5509ecdf..494b14fb903 100644 --- a/rs/moq-net/tests/integration/subscribe_ranges.rs +++ b/rs/moq-net/tests/integration/subscribe_ranges.rs @@ -330,7 +330,7 @@ async fn draft22_updates_both_bounds_across_cached_groups() { ) .await .expect("the narrowed range never reached the publisher"); - reader.update(initial.with_range(6..)).unwrap(); + reader.update(initial.with_range(0..)).unwrap(); let mut group = fixture .track .create_group(moq_net::group::Info { sequence: 6 }) @@ -339,7 +339,7 @@ async fn draft22_updates_both_bounds_across_cached_groups() { .write_frame(Timestamp::from_millis(6000).unwrap(), b"later".as_slice()) .unwrap(); group.finish().unwrap(); - assert_eq!(next(&mut reader).await, Some(6)); + assert_eq!(next(&mut reader).await, Some(6), "relay={relay}"); } } @@ -460,7 +460,7 @@ async fn draft18_updates_narrow_then_widen() { } seen.sort_unstable(); assert_eq!(seen, [5], "{version} relay={relay}"); - reader.update(initial.clone().with_range(4..5)).unwrap(); + reader.update(initial.clone().with_range(6..7)).unwrap(); moq_clock::timeout( TIMEOUT, kio::wait(|waiter| { @@ -469,7 +469,7 @@ async fn draft18_updates_narrow_then_widen() { if fixture .track .subscription() - .is_some_and(|subscription| subscription.ranges == [Range::groups(4..5)]) + .is_some_and(|subscription| subscription.ranges == [Range::groups(6..7)]) { return std::task::Poll::Ready(()); } From 06d8fbf1aafc8d8f623b1477bcc68aa588686de7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 07:39:54 -0700 Subject: [PATCH 4/9] docs(quest): settle AFK fill ownership scope and record reproduction Co-Authored-By: GPT-6.1 Sol --- quest/m1/ietf-publisher-one-group.md | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/quest/m1/ietf-publisher-one-group.md b/quest/m1/ietf-publisher-one-group.md index d6d3950b6ad..fb324068ee6 100644 --- a/quest/m1/ietf-publisher-one-group.md +++ b/quest/m1/ietf-publisher-one-group.md @@ -15,7 +15,7 @@ gets the draft's fill-failure signal (its fetch stream reset right after FETCH_HEADER, as `FillServe::Unsupported` already does), so its SUBSCRIBE stays up. FILL_TIMEOUT is accepted and ignored on both sides. -Non-goals: our subscribers. `rs/moq-net/src/ietf/subscriber.rs` keeps sending +Non-goals: changing our subscribers' range requests. `rs/moq-net/src/ietf/subscriber.rs` keeps sending range FETCH and multi-group fills upstream, one per run of misses, as [moq-dev/moq#6724](https://github.com/moq-dev/moq/pull/6724) does. Sending is cheap and correct, since the upstream answers with its own gap markers. @@ -64,11 +64,23 @@ Decision prompts: Implementation is checkpointed in [moq-dev/moq#7204](https://github.com/moq-dev/moq/pull/7204): both publishers serve one-group FETCH and the largest-group fill, ignore FILL_TIMEOUT, and no longer serve ranges. Subscriber range requests are unchanged. The publisher-only model helpers and timeout machinery are removed; fixtures and benchmarks use the kept forms. -Remaining decision: a refused fill can reset before its FETCH_HEADER is received. Our subscriber cannot associate that stream with its request, so its sole fill slot remains outstanding. `run_subscribe` defers every subsequent widening while that slot is outstanding. The existing `draft22_updates_both_bounds_across_cached_groups` integration test now stalls after `2..4 -> 0..6 -> 4..5 -> 0..`, losing live group 6 even though the subscription remains open. The memory transport's `reset_drops_queued_messages` test and the subscriber's deferred-update comment describe this case. - -Recommendation: permit a narrow subscriber fill-ownership fix that lets later updates retire an unidentifiable failed fill without changing the range FETCH or multi-group fill requests it sends. That exceeds the settled publisher-only scope and needs the maintainer's decision. Accepting permanently stalled widening is the alternative, not a test change to conceal it. - -After that decision, finish local checks and `just test interop --all`, then remove this quest and its links. The compatibility lane currently stops before session cells because accepted pins name moq-cli 0.14.8 while 0.14.9 is published. [moq-dev/moq#7061](https://github.com/moq-dev/moq/pull/7061) already owns that blocked remeasurement; do not repin it here without new evidence of a break from this change. +AFK scope choice on 2026-10-29: ✅ adopt the draft's recommended narrow +subscriber fill-ownership fix; accept permanently stalled widening was rejected. +Keep subscriber range FETCH and multi-group fill requests unchanged. A reset may +discard FETCH_HEADER, so later widening must be able to retire a fill that has +no identified stream without allowing its late header to claim a newer fill. + +Reproduced before code changes at `5b2485fb4749dbed31e13eb9229b694d3956a5a7`: +`just rs test -p moq-net --test integration draft22_updates_both_bounds_across_cached_groups` +fails at the final live group assertion (`None` instead of `Some(6)`, +`relay=false`). The fill slot remains outstanding after the refused stream's +header is discarded. + +Remaining: implement request-owned fill state, resolve conflicts with current +main, and finish local checks and `just test interop --all`. Remove this quest +and its links once implementation and validation are complete. Compatibility +pin remeasurement remains owned by +[moq-dev/moq#7061](https://github.com/moq-dev/moq/pull/7061); do not repin here. Public API: none. Wire: our publishers refuse multi-group FETCH with NOT_SUPPORTED, fail multi-group fills, and ignore FILL_TIMEOUT. From 3f3b129ad572e350a2d5a984c62b7ccee3b1b408 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 07:46:13 -0700 Subject: [PATCH 5/9] fix(net): give widening fills their own receive state Co-Authored-By: GPT-6.1 Sol --- quest/m1/ietf-publisher-one-group.md | 16 +- rs/moq-net/src/ietf/subscriber.rs | 221 +++++++++++++++++++-------- 2 files changed, 169 insertions(+), 68 deletions(-) diff --git a/quest/m1/ietf-publisher-one-group.md b/quest/m1/ietf-publisher-one-group.md index fb324068ee6..3404878b463 100644 --- a/quest/m1/ietf-publisher-one-group.md +++ b/quest/m1/ietf-publisher-one-group.md @@ -76,10 +76,18 @@ fails at the final live group assertion (`None` instead of `Some(6)`, `relay=false`). The fill slot remains outstanding after the refused stream's header is discarded. -Remaining: implement request-owned fill state, resolve conflicts with current -main, and finish local checks and `just test interop --all`. Remove this quest -and its links once implementation and validation are complete. Compatibility -pin remeasurement remains owned by +The subscriber fix is implemented: widening atomically retires an unidentified +pending fill and reserves request-owned state. Active or ready heads retain their +ownership until handoff. Delayed initial headers and retired update headers +cannot claim a newer fill, including handlers already waiting for an answer. +The unchanged widening integration case now passes. All 3,139 enabled moq-net +tests pass (12 skipped), including the new delayed-header ownership case. + +Remaining: finish feasible local CI and `just test interop --all`. The initial +interop run passed all nine Rust/JS protocol tests, then hit the local Cargo +cache wrapper's `Argument list too long` error while building client binaries. +A run using the same pinned Nix toolchain directly is queued through `quest gate`. +Compatibility pin remeasurement remains owned by [moq-dev/moq#7061](https://github.com/moq-dev/moq/pull/7061); do not repin here. Public API: none. Wire: our publishers refuse multi-group FETCH with diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index be80a7a5946..266a3709b56 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -767,7 +767,7 @@ struct TrackState { // The exact sent update's fill floor; a snapshot below it opens no fill stream. fill_start: Option, - // The joining FETCH's own request id, when one was sent. + // The joining FETCH or latest fill update's own request id, when one was sent. fetch_id: Option, // The data streams read so far, which PUBLISH_DONE's Stream Count is checked against. @@ -3435,7 +3435,7 @@ impl Subscriber { Revoked, Done(Result, Error>), Fill(track::Fill), - Update(Update), + Update(RequestId, Update), Invalid(Error), } @@ -3568,21 +3568,7 @@ impl Subscriber { (Some(end), Some(prior)) => end > prior, _ => false, }; - // A widening's fill would own the head, so it waits for the prior fill to hand - // its head off. Nothing else does: a fill stream reset before its header names - // no request, so that fill never settles. - let deferred = widened - && Filter::is_draft22(self.version) - && fill - .poll(waiter, |fill| { - if fill.outstanding() { - Poll::Pending - } else { - Poll::Ready(()) - } - }) - .is_pending(); - let ranges = if bounds != sent_bounds && !deferred { + let ranges = if bounds != sent_bounds { let mut join = if Filter::is_draft22(self.version) { match subscribe_join(bounds.0, bounds.1, self.version) { Ok(join) => join, @@ -3622,13 +3608,37 @@ impl Subscriber { } else { None }; - let update = Update { + let mut update = Update { filter: ranges.as_ref().map(|join| join.filter), fill: ranges.and_then(|join| join.fill), priority: (priority != sent_priority).then_some(priority), max_age: wanted_max_age.filter(|_| wanted_max_age != sent_max_age), new_group, }; + let mut state = self.state.lock(); + let Some(held) = state.subscribes.get_mut(&request_id) else { + return Poll::Ready(End::Idle); + }; + // Reserve the new fill while holding the old slot. A stream that already + // owns a head must hand it off, but a reset may have discarded a pending + // fill's header, leaving nothing that could settle that request. + let prior_fill = held.fill.clone(); + let filling = if update.fill.is_some() { + match prior_fill.poll(waiter, |fill| match **fill { + Fill::Active | Fill::Ready { .. } => Poll::Pending, + _ => Poll::Ready(()), + }) { + Poll::Ready(Ok(fill)) => Some(fill), + Poll::Ready(Err(_)) => return Poll::Ready(End::Invalid(Error::Cancel)), + Poll::Pending => { + update.filter = None; + update.fill = None; + None + } + } + } else { + None + }; if update.priority.is_none() && update.max_age.is_none() && update.new_group.is_none() @@ -3636,7 +3646,24 @@ impl Subscriber { { return Poll::Pending; } - Poll::Ready(End::Update(update)) + let update_id = self.control.next_request_id(); + if let Some(mut filling) = filling { + // A handler may already have the old header and be waiting on its + // answer. It must keep the old state, never claim the new request. + filling.cancel(); + drop(filling); + held.fill = kio::Producer::new(Fill::Requested); + held.fill_start = match update.fill.and_then(|fill| fill.filter.or(update.filter)) { + Some(Filter::Absolute { start, .. }) => Some(start), + _ => None, + }; + let previous = held.fetch_id.replace(update_id); + if let Some(previous) = previous { + state.fetches.remove(&previous); + } + state.fetches.insert(update_id, FillRequest::Update(request_id)); + } + Poll::Ready(End::Update(update_id, update)) }) .await; @@ -3646,13 +3673,26 @@ impl Subscriber { let _ = track.clone().abort(err); break true; } - End::Update(update) => { + End::Update(update_id, update) => { let changed_range = update.filter.is_some(); if changed_range { sent_bounds = wanted_bounds.clone().expect("validated update bounds"); } let (asked, new_group, asked_age) = (update.priority, update.new_group, update.max_age); - if let Err(err) = self.write_update(&mut stream.writer, request_id, update).await { + if let Err(err) = stream + .writer + .encode_message(&ietf::SubscribeUpdate { + max_age: update.max_age, + request_id: update_id, + subscriber_priority: update.priority.map(super::priority::to_wire), + forward: None, + filter: update.filter, + authorization_token: None, + new_group: update.new_group, + fill: update.fill, + }) + .await + { tracing::debug!(%err, "failed to update the subscription"); ended = Some(Err(err.clone())); let _ = track.clone().abort(err); @@ -3983,43 +4023,6 @@ impl Subscriber { } } - /// Update an Established subscription: REQUEST_UPDATE carrying only what changed, under a - /// Request ID of its own. - async fn write_update( - &self, - writer: &mut Writer, - subscription: RequestId, - update: Update, - ) -> Result<(), Error> { - let request_id = self.control.next_request_id(); - if update.fill.is_some() { - let mut state = self.state.lock(); - let held = state.subscribes.get_mut(&subscription).ok_or(Error::Cancel)?; - *held.fill.write().map_err(|_| Error::Cancel)? = Fill::Requested; - held.fill_start = match update.fill.and_then(|fill| fill.filter.or(update.filter)) { - Some(Filter::Absolute { start, .. }) => Some(start), - _ => None, - }; - let previous = held.fetch_id.replace(request_id); - if let Some(previous) = previous { - state.fetches.remove(&previous); - } - state.fetches.insert(request_id, FillRequest::Update(subscription)); - } - writer - .encode_message(&ietf::SubscribeUpdate { - max_age: update.max_age, - request_id, - subscriber_priority: update.priority.map(super::priority::to_wire), - forward: None, - filter: update.filter, - authorization_token: None, - new_group: update.new_group, - fill: update.fill, - }) - .await - } - /// Tell the publisher to stop serving a subscription we are walking away from. /// /// Every path that abandons an Established subscription goes through here, because @@ -4980,6 +4983,10 @@ impl Subscriber { let joined = state.fetches.get(&header.request_id).copied(); let subscribe_id = joined.map(FillRequest::subscription).unwrap_or(header.request_id); let track = state.subscribes.get(&subscribe_id).ok_or(Error::NotFound)?; + // Once an update owns the fill, a delayed initial fill cannot claim its slot. + if joined.is_none() && track.fetch_id.is_some() { + return Err(Error::Unsupported); + } // A fill is one of the subscription's own data streams, so PUBLISH_DONE counts // it. A joining FETCH is a request of its own. let counted = (!matches!(joined, Some(FillRequest::Joining(_)))) @@ -16077,10 +16084,10 @@ mod new_group_tests { relay.serving.abort(); } - /// A fill whose stream reset before its header never settles. It holds back only a - /// widening, whose own fill would take the head; a narrowing still goes up. + /// An identified fill keeps ownership until its head is handed off. A fill with no + /// header can instead be superseded, since a reset may have discarded that header. #[moq_clock::test] - async fn an_unsettled_fill_holds_back_only_a_widening() { + async fn only_an_identified_fill_holds_back_a_widening() { let (mut relay, mut first) = relay().await; let (mut stream, _) = relay.accept().await; accept(&mut stream, false, 5).await; @@ -16094,7 +16101,7 @@ mod new_group_tests { .unwrap() .fill .clone(); - *fill.write().ok().unwrap() = Fill::Serving(None); + *fill.write().ok().unwrap() = Fill::Active; first.update(track::Subscription::default().with_range(6..)).unwrap(); let (id, body) = read(&mut stream).await; @@ -16107,11 +16114,97 @@ mod new_group_tests { first.update(track::Subscription::default()).unwrap(); quiet(&mut stream).await; - *fill.write().ok().unwrap() = Fill::Done; + // No stream claimed the head after all. This request can be retired without + // waiting for a failure signal that could never identify the request. + *fill.write().ok().unwrap() = Fill::Serving(None); let (id, body) = read(&mut stream).await; assert_eq!(id, ietf::SubscribeUpdate::ID); let msg = ietf::SubscribeUpdate::decode_msg(&mut body.decoder(VERSION), VERSION).unwrap(); - assert!(msg.fill.is_some(), "the widening went up once the fill settled"); + assert!(msg.fill.is_some(), "the widening superseded the unidentified fill"); + assert!(matches!(*fill.read(), Fill::Done), "the old request was retired"); + relay.serving.abort(); + } + + /// Delayed headers, including one already waiting for REQUEST_OK, belong to the + /// request they answer and cannot consume a later widening's fill. + #[moq_clock::test] + async fn superseded_fill_headers_cannot_claim_the_new_request() { + use crate::{coding::Encode, lite::test_transport::ScriptedSession}; + + let header = |request_id| async move { + let mut bytes = Vec::new(); + let mut encoder = crate::coding::Encoder::new(&mut bytes, VERSION.into()); + encoder.varint(ietf::FetchHeader::TYPE).unwrap(); + ietf::FetchHeader { request_id }.encode(&mut encoder, VERSION).unwrap(); + let session = ScriptedSession::per_stream_eof(vec![bytes]); + let (_, reader) = session.open_bi(Default::default()).await.unwrap(); + Reader::new(crate::transport::boxed::RecvStream::new(reader), VERSION) + }; + + let (mut relay, mut first) = relay().await; + let (mut stream, _) = relay.accept().await; + accept(&mut stream, false, 5).await; + let subscription = *relay.probe.state.lock().subscribes.keys().next().unwrap(); + first.update(track::Subscription::default().with_range(6..)).unwrap(); + let _ = read(&mut stream).await; + answer(&mut stream).await; + first.update(track::Subscription::default().with_range(4..)).unwrap(); + let (id, body) = read(&mut stream).await; + assert_eq!(id, ietf::SubscribeUpdate::ID); + let old = ietf::SubscribeUpdate::decode_msg(&mut body.decoder(VERSION), VERSION).unwrap(); + + // The initial SUBSCRIBE's delayed header cannot consume this update's slot. + let mut initial = header(subscription).await; + assert!(matches!( + relay.probe.clone().recv_fill(&mut initial).await, + Err(Error::Unsupported) + )); + + // This update's header overtakes its REQUEST_OK and parks on the old slot. + let mut early = header(old.request_id).await; + let mut receiving = relay.probe.clone(); + let mut early = std::pin::pin!(receiving.recv_fill(&mut early)); + assert!(futures::poll!(early.as_mut()).is_pending()); + stream + .writer + .encode_message(&ietf::RequestOk { + largest: Some(ietf::Location { group: 5, object: 0 }), + active: None, + }) + .await + .unwrap(); + settle().await; + + first.update(track::Subscription::default()).unwrap(); + let (id, body) = read(&mut stream).await; + assert_eq!(id, ietf::SubscribeUpdate::ID); + let newest = ietf::SubscribeUpdate::decode_msg(&mut body.decoder(VERSION), VERSION).unwrap(); + assert!(newest.fill.is_some()); + assert!(matches!(early.await, Err(Error::Unsupported))); + + // The same old header arriving after retirement has no mapping left, either. + let mut late = header(old.request_id).await; + assert!(matches!( + relay.probe.clone().recv_fill(&mut late).await, + Err(Error::NotFound) + )); + let fill = relay.probe.state.lock().subscribes[&subscription].fill.clone(); + assert!(matches!(*fill.read(), Fill::Requested)); + stream + .writer + .encode_message(&ietf::RequestOk { + largest: Some(ietf::Location { group: 5, object: 0 }), + active: None, + }) + .await + .unwrap(); + settle().await; + let mut current = header(newest.request_id).await; + relay.probe.clone().recv_fill(&mut current).await.unwrap(); + assert!( + matches!(*fill.read(), Fill::Done), + "only the current header claimed the slot" + ); relay.serving.abort(); } } From 8a4f14a3aa6ad375c8cb00a31d6a76cc4bbfc30f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 07:47:53 -0700 Subject: [PATCH 6/9] test(net): drop the merged multi-group fill eviction case Co-Authored-By: GPT-6.1 Sol --- js/net/src/ietf/publisher.test.ts | 50 ------------------------------- 1 file changed, 50 deletions(-) diff --git a/js/net/src/ietf/publisher.test.ts b/js/net/src/ietf/publisher.test.ts index 92c43a92dfc..2e47e34cf50 100644 --- a/js/net/src/ietf/publisher.test.ts +++ b/js/net/src/ietf/publisher.test.ts @@ -3258,56 +3258,6 @@ test.each([0, 1000])("draft-22: FILL_TIMEOUT=%s is ignored for a largest-group f } }); -// A group the cache evicts after the fill listed it is unknown, not skipped, which would say it -// never existed. Waiting on the open group the fill reaches first gives the older groups time -// to age out of the cache. -test("draft-22: a fill marks a group evicted after it was listed unknown", async () => { - using clock = manual(); - const fx = fixture(); - const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); - // Groups 0 and 1 age out; the open group 2 is the newest the fill covers. - writeGroup(track, 2); - writeGroup(track, 2); - await clock.advance(Nano.fromMilli(Milli(25_000))); - const open = track.appendGroup(); - open.writeFrame({ payload: new TextEncoder().encode("2.0"), timestamp: Timestamp.now() }); - writeGroup(track, 1); - - const { client } = await runSubscribe( - fx, - new Subscribe({ - requestId: 7n, - trackNamespace: Path.from("test"), - trackName: "video", - subscriberPriority: 0, - filter: { kind: "nextObject" }, - fill: { - filter: { kind: "absolute", startGroup: 0n, startObject: 0n, endGroup: 2n }, - fillTimeout: Milli(20_000), - rangeFilters: false, - }, - }), - ); - - try { - const fill = await nextUni(fx.uni); - if (!fill) throw new Error("no fetch stream for the requested fill"); - const served = await clock.drive(readFill(fill)); - expect(served.reset).toBeUndefined(); - const end = Number(2n ** 62n - 1n); - expect(served.objects).toEqual([ - { group: 2, id: 0, payload: "2.0" }, - { group: 2, id: end, payload: "end 20c" }, - { group: 1, id: end, payload: "end 10c" }, - { group: 0, id: end, payload: "end 10c" }, - ]); - } finally { - open.finish(); - fx.close(); - client.close(); - } -}); - /** * A broadcast served through `requested()` resolves its track on demand, and a dynamic serve * is deliberately one request per peer subscription. Resolving the track again to read the From 7c62950125340451dc83ccdfe23be8ce1bc8219e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 08:08:55 -0700 Subject: [PATCH 7/9] docs(quest): checkpoint the completed fill fix and interop validation Co-Authored-By: GPT-6.1 Sol --- quest/m1/ietf-publisher-one-group.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/quest/m1/ietf-publisher-one-group.md b/quest/m1/ietf-publisher-one-group.md index 3404878b463..ec9378df7cf 100644 --- a/quest/m1/ietf-publisher-one-group.md +++ b/quest/m1/ietf-publisher-one-group.md @@ -83,12 +83,19 @@ cannot claim a newer fill, including handlers already waiting for an answer. The unchanged widening integration case now passes. All 3,139 enabled moq-net tests pass (12 skipped), including the new delayed-header ownership case. -Remaining: finish feasible local CI and `just test interop --all`. The initial -interop run passed all nine Rust/JS protocol tests, then hit the local Cargo -cache wrapper's `Argument list too long` error while building client binaries. -A run using the same pinned Nix toolchain directly is queued through `quest gate`. -Compatibility pin remeasurement remains owned by -[moq-dev/moq#7061](https://github.com/moq-dev/moq/pull/7061); do not repin here. +`just test interop --all` passed in full, including all nine Rust/JS protocol +tests, every language client, finished streams, tails, compression, and resets. +Net TypeScript (including examples and benchmarks), Biome, Rust formatting, and +quest validation passed. The earlier JS fill benchmark completed its +cached-groups × subscribers sweep. + +Remaining: complete the running local CI gate, then remove this quest and its +links. The local Cargo-cache wrapper failed with `Argument list too long`; +interop succeeded using the pinned Nix toolchain with BASH_ENV disabled, and CI +is running in that same environment. Compatibility preflight confirms stale +planned pins (moq-cli 0.14.8 versus published 0.14.9). Pin remeasurement remains +owned by [moq-dev/moq#7061](https://github.com/moq-dev/moq/pull/7061); do not +repin here. Public API: none. Wire: our publishers refuse multi-group FETCH with NOT_SUPPORTED, fail multi-group fills, and ignore FILL_TIMEOUT. From 27caa7a2e2a98844faadedfd2103408358449dc2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 08:09:19 -0700 Subject: [PATCH 8/9] chore(quest): complete one-group IETF publisher work Co-Authored-By: GPT-6.1 Sol --- quest/m1/README.md | 1 - quest/m1/ietf-epoch-cut.md | 1 - quest/m1/ietf-js-join.md | 2 +- quest/m1/ietf-publisher-one-group.md | 107 --------------------- quest/m1/ietf-tracks-refuse.md | 4 - quest/m1/interop-coverage/js-ietf-fetch.md | 2 +- quest/m1/subscribe-ranges/README.md | 2 +- quest/m2/ietf-hole-marker.md | 3 +- 8 files changed, 4 insertions(+), 118 deletions(-) delete mode 100644 quest/m1/ietf-publisher-one-group.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 48bc378104c..0fd9245eb65 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -149,7 +149,6 @@ removed). - [Upgrade notes match the code](/quest/m1/upgrade-guide-facts.md) - the upgrade and CLI migration guides stop promising drafts 19-21, capped grants, and an ignored `MOQ_EPOCH` - [A smaller, safer API than release](/quest/m1/api-audit/README.md) - every surface added since release has a consumer or is gone, overlapping operations are collapsed, names mirror across Rust, JS, and the bindings, and every break says how to migrate - [moq-transport 18 and 22 only](/quest/m1/ietf-drop/README.md) - drafts 14-17 and 19-21 are deleted from Rust, JS, tests, interop lanes, and docs, removing the legacy control-stream adapter and its request limit -- [Our moq-transport publishers serve at most one group](/quest/m1/ietf-publisher-one-group.md) - a wider FETCH gets NOT_SUPPORTED and a wider draft-22 fill fails, FILL_TIMEOUT is ignored, and our subscribers still send ranges - [Our moq-transport publishers refuse SUBSCRIBE_TRACKS](/quest/m1/ietf-tracks-refuse.md) - a peer's listing gets NOT_SUPPORTED, while Rust still lists an IETF peer's tracks upstream for the archive - [A moq-transport request token is one auth per distinct token](/quest/m1/ietf-token-auth.md) - requests naming the same token share one verification and grant, on any request type, ending with the last such request - [moq-transport carries no epoch](/quest/m1/ietf-epoch-cut.md) - Rust and JS moq-transport sessions stop negotiating MoQ Epoch, so every moq-transport hop is epochless; the draft stays diff --git a/quest/m1/ietf-epoch-cut.md b/quest/m1/ietf-epoch-cut.md index 02d6787aae7..33a99624e54 100644 --- a/quest/m1/ietf-epoch-cut.md +++ b/quest/m1/ietf-epoch-cut.md @@ -91,4 +91,3 @@ offering EPOCH and never send its parameter. moq-lite is unchanged. ## Related - [Announcement takeover](/quest/m1/announce-takeover.md) - the lite-07 takeover, which no moq-transport counterpart follows -- [Our moq-transport publishers serve at most one group](/quest/m1/ietf-publisher-one-group.md) - the same narrowing for FETCH diff --git a/quest/m1/ietf-js-join.md b/quest/m1/ietf-js-join.md index b4fdc0ed942..0c4cb4a162e 100644 --- a/quest/m1/ietf-js-join.md +++ b/quest/m1/ietf-js-join.md @@ -31,7 +31,7 @@ Decided 2026-10-28 (maintainer), from [#7088](https://github.com/moq-dev/moq/iss ([#6407](https://github.com/moq-dev/moq/pull/6407)). - **No FETCH_OK deadline**, matching Rust: the joining FETCH lives as long as the subscription. Our publishers accept and ignore FILL_TIMEOUT - ([one group](/quest/m1/ietf-publisher-one-group.md)). + ([moq-dev/moq#7204](https://github.com/moq-dev/moq/pull/7204)). - **Rust's strictness on duplicates.** A whole group from object 0 that arrives before the head wins and the FETCH is discarded. One that arrives after the head is refused. diff --git a/quest/m1/ietf-publisher-one-group.md b/quest/m1/ietf-publisher-one-group.md deleted file mode 100644 index ec9378df7cf..00000000000 --- a/quest/m1/ietf-publisher-one-group.md +++ /dev/null @@ -1,107 +0,0 @@ -# [L] Our moq-transport publishers serve at most one group - -## Goal - -The Rust (`rs/moq-net/src/ietf/publisher.rs`) and JS -(`js/net/src/ietf/publisher.ts`) moq-transport publishers answer FETCH and -draft-22 fills only for one group: - -- a standalone FETCH whose start and end are in the same group, -- a Relative Joining FETCH at offset 0 (draft 18), or -- a fill of the largest group's head (draft 22). - -Any wider FETCH is refused with NOT_SUPPORTED before FETCH_OK. A wider fill -gets the draft's fill-failure signal (its fetch stream reset right after -FETCH_HEADER, as `FillServe::Unsupported` already does), so its SUBSCRIBE -stays up. FILL_TIMEOUT is accepted and ignored on both sides. - -Non-goals: changing our subscribers' range requests. `rs/moq-net/src/ietf/subscriber.rs` keeps sending -range FETCH and multi-group fills upstream, one per run of misses, as -[moq-dev/moq#6724](https://github.com/moq-dev/moq/pull/6724) does. Sending -is cheap and correct, since the upstream answers with its own gap markers. -Joining FETCH and the largest-group fill are unchanged. - -## Plan - -Decided with the maintainer on 2026-10-28 (`/quest-plan`, "what's not worth -supporting in IETF"). - -Why: answering a range authoritatively means never serving from an -incomplete cache, so the publisher must know every group in the range or -fetch every gap upstream. Since June, 21 of about 234 IETF fixes were FETCH, -join, or fill, mostly in the range logic: end locations -([moq-dev/moq#6295](https://github.com/moq-dev/moq/pull/6295), -[moq-dev/moq#5487](https://github.com/moq-dev/moq/pull/5487)), mid-group -starts ([moq-dev/moq#6465](https://github.com/moq-dev/moq/pull/6465)), -credit ([moq-dev/moq#6702](https://github.com/moq-dev/moq/pull/6702)), -datagram groups ([moq-dev/moq#4982](https://github.com/moq-dev/moq/pull/4982)), -and held fetches ([moq-dev/moq#5623](https://github.com/moq-dev/moq/pull/5623)). -The external interop runner (`englishm/moq-interop-runner`) has no FETCH -case, and our own publishers and subscribers talk moq-lite to each other. -Refusing a FETCH is allowed by the spec. The aim is to implement the simple -parts of moq-transport. - -Decision prompts: - -- Goal for FETCH: ✅ refuse multi-group; cache-only ranges marked `0x10C`; - joining only. -- Draft-22 multi-group fills: ✅ same rule as FETCH; keep multi-group fills. -- Who refuses: ✅ only our publishers (the maintainer, correcting an earlier - "both directions" pick: "subscriber.rs can do a range FETCH but - publisher.rs will return NOT_SUPPORTED"); both publishers and subscribers. -- A wider fill: ✅ the fill-failure reset, keeping the SUBSCRIBE; refusing - the whole SUBSCRIBE. -- FILL_TIMEOUT: refuse in both; ✅ accept and ignore in both; narrow the Rust - quest to one group. One live group ends on its own, so the wait is - bounded. A zero (cache-only) value may get objects that arrive later, - which is not quite conforming and is accepted. -- Hole-marker extension: ✅ its own m2 quest, - [moq-transport hole markers](/quest/m2/ietf-hole-marker.md), which lifts - this refusal wherever the cache knows every group in the range. -- Docs: ✅ inline only; a separate docs quest. -- Rank: ✅ before the other FETCH quests, right after - [moq-transport 18 and 22 only](/quest/m1/ietf-drop/README.md); at m1's end. - -Implementation is checkpointed in [moq-dev/moq#7204](https://github.com/moq-dev/moq/pull/7204): both publishers serve one-group FETCH and the largest-group fill, ignore FILL_TIMEOUT, and no longer serve ranges. Subscriber range requests are unchanged. The publisher-only model helpers and timeout machinery are removed; fixtures and benchmarks use the kept forms. - -AFK scope choice on 2026-10-29: ✅ adopt the draft's recommended narrow -subscriber fill-ownership fix; accept permanently stalled widening was rejected. -Keep subscriber range FETCH and multi-group fill requests unchanged. A reset may -discard FETCH_HEADER, so later widening must be able to retire a fill that has -no identified stream without allowing its late header to claim a newer fill. - -Reproduced before code changes at `5b2485fb4749dbed31e13eb9229b694d3956a5a7`: -`just rs test -p moq-net --test integration draft22_updates_both_bounds_across_cached_groups` -fails at the final live group assertion (`None` instead of `Some(6)`, -`relay=false`). The fill slot remains outstanding after the refused stream's -header is discarded. - -The subscriber fix is implemented: widening atomically retires an unidentified -pending fill and reserves request-owned state. Active or ready heads retain their -ownership until handoff. Delayed initial headers and retired update headers -cannot claim a newer fill, including handlers already waiting for an answer. -The unchanged widening integration case now passes. All 3,139 enabled moq-net -tests pass (12 skipped), including the new delayed-header ownership case. - -`just test interop --all` passed in full, including all nine Rust/JS protocol -tests, every language client, finished streams, tails, compression, and resets. -Net TypeScript (including examples and benchmarks), Biome, Rust formatting, and -quest validation passed. The earlier JS fill benchmark completed its -cached-groups × subscribers sweep. - -Remaining: complete the running local CI gate, then remove this quest and its -links. The local Cargo-cache wrapper failed with `Argument list too long`; -interop succeeded using the pinned Nix toolchain with BASH_ENV disabled, and CI -is running in that same environment. Compatibility preflight confirms stale -planned pins (moq-cli 0.14.8 versus published 0.14.9). Pin remeasurement remains -owned by [moq-dev/moq#7061](https://github.com/moq-dev/moq/pull/7061); do not -repin here. - -Public API: none. Wire: our publishers refuse multi-group FETCH with -NOT_SUPPORTED, fail multi-group fills, and ignore FILL_TIMEOUT. - -## Related - -- [Rust FETCH parity](/quest/m1/ietf-fetch-parity.md) - joining FETCH behavior this keeps -- [JS subscribers FETCH over moq-transport](/quest/m1/interop-coverage/js-ietf-fetch.md) - the JS subscriber's range FETCH, which stays -- [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - relays fill misses upstream by range, unchanged here diff --git a/quest/m1/ietf-tracks-refuse.md b/quest/m1/ietf-tracks-refuse.md index 2919883525f..17276e94b7c 100644 --- a/quest/m1/ietf-tracks-refuse.md +++ b/quest/m1/ietf-tracks-refuse.md @@ -41,7 +41,3 @@ covers it. Public API: none. Wire: our publishers refuse SUBSCRIBE_TRACKS with NOT_SUPPORTED. - -## Related - -- [Our moq-transport publishers serve at most one group](/quest/m1/ietf-publisher-one-group.md) - the same publisher-side narrowing for FETCH diff --git a/quest/m1/interop-coverage/js-ietf-fetch.md b/quest/m1/interop-coverage/js-ietf-fetch.md index 87058f7baf9..5455443b47e 100644 --- a/quest/m1/interop-coverage/js-ietf-fetch.md +++ b/quest/m1/interop-coverage/js-ietf-fetch.md @@ -22,7 +22,7 @@ mid-group join to a joining FETCH on draft 18 or a fill on draft 22, the way Rust's subscriber does. Then add the native JS runtimes as subscribers to the `run_fetch_round` cells in the tail suite, and a late-join case through a relay that does not replay the group. Our publishers answer only one group -([one group](/quest/m1/ietf-publisher-one-group.md), 2026-10-28), so a +([moq-dev/moq#7204](https://github.com/moq-dev/moq/pull/7204), 2026-10-28), so a past-range FETCH against them is refused. Test ranges against a third-party publisher, and one group against ours. diff --git a/quest/m1/subscribe-ranges/README.md b/quest/m1/subscribe-ranges/README.md index 82b512f8bfd..7d772940a18 100644 --- a/quest/m1/subscribe-ranges/README.md +++ b/quest/m1/subscribe-ranges/README.md @@ -44,7 +44,7 @@ relay bounds a sparse FETCH only if the model can request ranges: group's existing prefix but never wait for its future objects. Downstream, our publishers answer only a one-group FETCH or fill and refuse wider ones (maintainer, 2026-10-28, - [one group](/quest/m1/ietf-publisher-one-group.md)), until + [moq-dev/moq#7204](https://github.com/moq-dev/moq/pull/7204)), until [hole markers](/quest/m2/ietf-hole-marker.md) let a cache that knows every group answer a range. A FETCH of the open largest group reads it whole (maintainer, 2026-10-25): the cache cannot know an open group's real diff --git a/quest/m2/ietf-hole-marker.md b/quest/m2/ietf-hole-marker.md index 36ec0f84d0d..a6e3c74328f 100644 --- a/quest/m2/ietf-hole-marker.md +++ b/quest/m2/ietf-hole-marker.md @@ -16,7 +16,7 @@ drop judged under a narrower budget or range, as in [Drop set](/quest/m1/drop-set.md), proves nothing for a wider FETCH. That group is unknown, so the publisher fills it or the range is refused. Any other range stays refused, as -[Our moq-transport publishers serve at most one group](/quest/m1/ietf-publisher-one-group.md) +[moq-dev/moq#7204](https://github.com/moq-dev/moq/pull/7204) leaves it. ## Plan @@ -60,5 +60,4 @@ Our publishers answer multi-group FETCH and fills over known caches. ## Required -- [Our moq-transport publishers serve at most one group](/quest/m1/ietf-publisher-one-group.md) - the refusal this lifts - [Drop set](/quest/m1/drop-set.md) - the per-track runs that say which groups are known holes From 8dd35926e10bd91a8570a49e0464e4cea534bd99 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 29 Oct 2026 17:47:56 -0700 Subject: [PATCH 9/9] test(js): keep the REQUEST_UPDATE omitted-preferences case It covers subscriber max age, which stays supported without range serving. Co-Authored-By: Claude Opus 5.5 --- js/net/src/ietf/publisher.test.ts | 35 +++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/js/net/src/ietf/publisher.test.ts b/js/net/src/ietf/publisher.test.ts index 2e47e34cf50..3b5524fbe42 100644 --- a/js/net/src/ietf/publisher.test.ts +++ b/js/net/src/ietf/publisher.test.ts @@ -4548,3 +4548,38 @@ async function readServed( } return served; } + +test("subscriber max age and REQUEST_UPDATE preserve omitted preferences", async () => { + const fx = fixture(Version.DRAFT_18); + const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); + const { client } = await runSubscribe( + fx, + new Subscribe({ + requestId: 0n, + trackNamespace: Path.from("test"), + trackName: "video", + subscriberPriority: 128, + maxAge: 250, + }), + ); + try { + expect(track.subscription.peek()?.maxDelay).toBe(Milli(250)); + const parameters = new Parameters(); + parameters.maxAge = 900; + await client.writer.u53(SubscribeUpdate.id); + await new SubscribeUpdate({ requestId: 2n, parameters }).encode(client.writer, fx.version); + expect(await client.reader.u53()).toBe(RequestOk.id); + await RequestOk.decode(client.reader, fx.version); + expect(track.subscription.peek()?.maxDelay).toBe(Milli(900)); + await client.writer.u53(SubscribeUpdate.id); + await new SubscribeUpdate({ requestId: 4n }).encode(client.writer, fx.version); + expect(await client.reader.u53()).toBe(RequestOk.id); + await RequestOk.decode(client.reader, fx.version); + expect(track.subscription.peek()?.maxDelay).toBe(Milli(900)); + } finally { + client.close(); + track.finish(); + track.close(); + fx.close(); + } +});