diff --git a/.claude/skills/verify-subscribe/SKILL.md b/.claude/skills/verify-subscribe/SKILL.md index f4d91b001ab..9f8422b48ec 100644 --- a/.claude/skills/verify-subscribe/SKILL.md +++ b/.claude/skills/verify-subscribe/SKILL.md @@ -1,11 +1,11 @@ --- name: verify-subscribe -description: Model-check how a lite-07 subscription accounts for each group (`rs/moq-net/src/lite/publisher.rs` `TrackRun` and `GroupServe`, `lite/subscriber.rs` `ServeLoop::apply_response`, `src/tail.rs`) with TLA+. Use when a change touches SUBSCRIBE_DROP, SUBSCRIBE_SKIP, stale (OLD) drops, which groups a publisher serves or names once, Group Stream resets, preemption, and expiry (a lite-07 stream closed when its group expires), a GROUP's Previous Start and the skips it passes over or the publisher holds back for it, or the subscriber's tail, or when a drop bug needs every ordering of the Subscribe Stream against the Group Streams checked. +description: Model-check how a lite-07 subscription accounts for each group (`rs/moq-net/src/lite/publisher.rs` `TrackRun` and `GroupServe`, `lite/subscriber.rs` `ServeLoop::apply_response`, `src/tail.rs`) with TLA+. Use when a change touches SUBSCRIBE_DROP, SUBSCRIBE_SKIP, stale (OLD) drops, which groups a publisher serves or names once, Group Stream resets, preemption, and expiry (a stream reset with OLD when its group expires, even after its FIN), a GROUP's Previous Start and the skips it passes over or the publisher holds back for it, or the subscriber's tail, or when a drop bug needs every ordering of the Subscribe Stream against the Group Streams checked. --- # Verify subscription accounting -`rs/moq-net/verify/subscribe/SubscribeAccounting.tla` models the groups of one answered lite-07 subscription between the publisher's `TrackRun` and the subscriber's `TrackServe`. The draft promises that every sequence in range gets a Group Stream, a SUBSCRIBE_SKIP, or a SUBSCRIBE_DROP, and that every drop is final on its subscription, a stale (OLD) one included: a SUBSCRIBE_UPDATE reopens nothing, so the model leaves updates out. A skip still goes out over a drop of a group never served. Drops and skips travel in order on the Subscribe Stream, each Group Stream travels on its own, and a reset can lose a stream's header. A Group Stream whose group expires stale is closed, not reset: a FIN without GROUP_END, so the subscriber keeps what arrived as a closed group and no drop names it. A stream that never opened is still given up with a drop, and one whose FIN went out is never reset for expiry. A stream reset while serving, evicted or preempted with TOO_FAR_BEHIND for a group that outranks it, is named in a final drop. Each GROUP's Previous Start passes over the skipped run directly below it as a SUBSCRIBE_SKIP would, so the publisher holds such a skip back for the GROUP of a group it still owes, and sends it after all, ahead of the drop or joined to the skip, if it gives that group up instead. A drop of a group whose stream opened goes out after a skip of what its GROUP passed over, since the GROUP may never have arrived. Each model step is one poll of the code that handles one event. The track cursor's verdicts (hand a group out, give it up stale or final, skip it) are free choices, and so are when a stream fails and whether the cache holds the group above a skip, which decides whether the skip is held back. With `Pressure`, so is when the cache pool's debt folds the publisher's record of a group. TLC checks every ordering under the bounds in the config, and reports a deadlock for any state short of quiet where nothing can happen. +`rs/moq-net/verify/subscribe/SubscribeAccounting.tla` models the groups of one answered lite-07 subscription between the publisher's `TrackRun` and the subscriber's `TrackServe`. The draft promises that every sequence in range gets a Group Stream, a SUBSCRIBE_SKIP, or a SUBSCRIBE_DROP, and that every drop is final on its subscription, a stale (OLD) one included: a SUBSCRIBE_UPDATE reopens nothing, so the model leaves updates out. A skip still goes out over a drop of a group never served. Drops and skips travel in order on the Subscribe Stream, each Group Stream travels on its own, and a reset can lose a stream's header. A Group Stream whose group expires stale resets with OLD and is named in a stale drop, whether it never opened, is serving, or already sent its FIN and awaits the acknowledgement, so stale bytes never hold the send window; a subscriber that read it whole before the reset keeps the group. A stream reset while serving, evicted or preempted with TOO_FAR_BEHIND for a group that outranks it, is named in a final drop. Each GROUP's Previous Start passes over the skipped run directly below it as a SUBSCRIBE_SKIP would, so the publisher holds such a skip back for the GROUP of a group it still owes, and sends it after all, ahead of the drop or joined to the skip, if it gives that group up instead. A drop of a group whose stream opened goes out after a skip of what its GROUP passed over, since the GROUP may never have arrived. Each model step is one poll of the code that handles one event. The track cursor's verdicts (hand a group out, give it up stale or final, skip it) are free choices, and so are when a stream fails and whether the cache holds the group above a skip, which decides whether the skip is held back. With `Pressure`, so is when the cache pool's debt folds the publisher's record of a group. TLC checks every ordering under the bounds in the config, and reports a deadlock for any state short of quiet where nothing can happen. ## Rules @@ -14,7 +14,7 @@ Each rule is an INVARIANT; the harm says why it matters. | Rule | Holds when | Harm if broken | | --- | --- | --- | | `ServedOnce` | The publisher opens at most one Group Stream for a group. | The group's bytes go out twice, and the subscriber reads two streams of one group. | -| `NamedOnce` | The publisher names a group in at most one drop and at most one skip, and a skip only for a group it never served. The one exception repeats a skip a GROUP passed over, once, ahead of that group's drop. | A second name contradicts a final one: the subscriber accounts the group twice or ends a reader on a group it delivered. | +| `NamedOnce` | The publisher names a group in at most one drop and at most one skip, and a skip only for a group it never served. A drop may follow the group's own Group Stream, even one sent whole, once it resets. The one exception to one skip repeats a skip a GROUP passed over, once, ahead of that group's drop. | A second name contradicts a final one: the subscriber accounts the group twice or ends a reader on a group it delivered. | | `QuietIsSettled` | Once nothing is in flight and the publisher has nothing left to do, the subscriber's tail accounts for every group, a skip held back for a GROUP included. | The subscriber waits out its grace for a group nobody will send, and a relay holds its readers there. | ## Run @@ -23,7 +23,7 @@ Each rule is an INVARIANT; the harm says why it matters. quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla ``` -It checks every rule under `Pressure` over three groups, 1 to 3, so a skip of 1 can be held back for 2's GROUP and a run of 1 and 2 for 3's: about 1.97M distinct states, ten seconds. It passes. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. +It checks every rule under `Pressure` over three groups, 1 to 3, so a skip of 1 can be held back for 2's GROUP and a run of 1 and 2 for 3's: about 11.1M distinct states, two minutes. It passes. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. ## Reading a failure @@ -40,7 +40,7 @@ Sync commit: `3016af623807e939c2e99571a3ec02beb2493579`. Run `git log 3016af6238 | `rs/moq-net/src/lite/publisher.rs` `poll_forward_drop`, `forward_range`, `drop_range`, `report`, `hole`; `rs/moq-net/src/model/record.rs` `Record::report`, `tell`, `untold`, `Report::blocks` | `CursorGiveUp`, `dropped` | | `rs/moq-net/src/lite/publisher.rs` `TrackRun::elides`, `owes_group`, `send_elided` | `held`, the `hold` choice of `CursorGiveUp` | | `rs/moq-net/src/lite/publisher.rs` `TrackRun::poll_step` (outcomes) | `PubOutcomes`, `nahead` | -| `rs/moq-net/src/lite/publisher.rs` `GroupServe::poll_serve` (`Serve`, `Closed`), `settle_failed`, `poll_stopped`, `closes_expired`, `Ending`; `rs/moq-net/src/transport/boxed.rs` `ListedState::preempted` | `StreamFin`, `StreamFail`, `StreamAck` | +| `rs/moq-net/src/lite/publisher.rs` `GroupServe::poll_serve` (`Serve`, `Closed`), `settle_failed`, `poll_given_up`; `rs/moq-net/src/transport/boxed.rs` `ListedState::preempted` | `StreamFin`, `StreamFail`, `StreamAck` | | `rs/moq-net/src/model/record.rs` `Record::fold` | `PubFold`, `folded` | | `rs/moq-net/src/lite/subscriber.rs` `TrackEntry::open` (with the passed-over skip), `reopen`, `place`; `GroupRecv::poll_serve` | `SubHeader`, `SubFin`, `SubReset` | | `rs/moq-net/src/lite/subscriber.rs` `ServeLoop::apply_response` (`Drop`, `Skip`) | `OnDrop`, `OnSkip`, `SubRecv` | @@ -52,7 +52,8 @@ What the model asks of that code, each found by TLC as a broken rule without it: - The publisher serves no sequence its record already told, served or named (`Record::told`), and a skip goes out over a drop alone (`Report::blocks`), so a cursor that hands a group out again after giving it up, as after a route switch, sends nothing (`ServedOnce`, `NamedOnce`). - A skip held back for a group's GROUP goes out once that group is given up instead: ahead of its drop (`drop_range`, `forward_range`), and joined to its skip, since no GROUP passes over it then (`forward_range`). A skip of the group above that kept the run held would leave it unsent for good (`QuietIsSettled`). - A drop of a group whose stream opened goes out after a skip of what its GROUP passed over (`GroupOutcomes::dropped`), since a reset can lose the GROUP (`QuietIsSettled`). +- A stream reset for expiry after its FIN is still named in a drop (`settle_failed` from `GroupState::Closed`), since the reset can discard the header before it is read (`QuietIsSettled`). ## Not covered -The subscription starts answered and covers the groups, so SUBSCRIBE_OK, SUBSCRIBE_STARTED, ranges and Range IDs, fills (`RangeFills`) and the separate request (`Recovery`) a reader wider than a stale drop asks with, floors, SUBSCRIBE_UPDATE, and SUBSCRIBE_FINISH with `drop_unserved` are left out. The tail's grace and the ledger's folding are left out: `QuietIsSettled` asks for the publisher's answer, never the grace. So are priority and order (`Steer`), which stream a preempt picks, the Subscribe Stream's FIN and cancellation, sessions closing, versions before lite-07, relays forwarding drops downstream, the budget a relay records a stale drop under, datagrams, and the copy's readers (`poll_drop`). A route switch appears only as the cursor handing out once more a group it gave up. FETCH is left out: one request and one response, ended by a FIN or a reset, with nothing to race. GROUP_END takeover is left out, and so are frames: how much of a group an expired stream closes on, and GROUP_END's End, which only says where a whole group ends. With one Group Stream per group, the subscriber's wait for a duplicate stream (`Open::Duplicate`) is never reached. A GROUP's Previous Start is modeled as the skipped run directly below it; one naming a known group below, or nothing, passes over nothing, and a run released by an update (`release_elided`) is left out with updates. A pass covers three groups. +The subscription starts answered and covers the groups, so SUBSCRIBE_OK, SUBSCRIBE_STARTED, ranges and Range IDs, fills (`RangeFills`) and the separate request (`Recovery`) a reader wider than a stale drop asks with, floors, SUBSCRIBE_UPDATE, and SUBSCRIBE_FINISH with `drop_unserved` are left out. The tail's grace and the ledger's folding are left out: `QuietIsSettled` asks for the publisher's answer, never the grace. So are priority and order (`Steer`), which stream a preempt picks, the Subscribe Stream's FIN and cancellation, sessions closing, versions before lite-07, relays forwarding drops downstream, the budget a relay records a stale drop under, datagrams, and the copy's readers (`poll_drop`). A route switch appears only as the cursor handing out once more a group it gave up. FETCH is left out: one request and one response, ended by a FIN or a reset, with nothing to race. GROUP_END takeover is left out, and so are frames and GROUP_END's End, which only says where a whole group ends. With one Group Stream per group, the subscriber's wait for a duplicate stream (`Open::Duplicate`) is never reached. A GROUP's Previous Start is modeled as the skipped run directly below it; one naming a known group below, or nothing, passes over nothing, and a run released by an update (`release_elided`) is left out with updates. A pass covers three groups. diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 022454bad74..0fb276629cd 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -222,7 +222,7 @@ A Group can therefore be assembled from more than one publisher: each contribute A partial Group is only ever delivered to a subscriber that asked for one with FETCH. A publisher that holds only the later frames of a Group a subscription covers names it in a [SUBSCRIBE_DROP](#subscribe-drop) rather than deliver the part it holds. -One that holds its earlier frames, such as a relay whose upstream closed the Group or a Group expiring as it is sent, delivers them and closes the Group Stream without a GROUP_END. +One that holds its earlier frames, such as a relay whose upstream closed the Group, delivers them and closes the Group Stream without a GROUP_END. A Group is the unit of decodability: its earlier Frames may carry a keyframe or compression state the later Frames need. # Flow @@ -838,8 +838,9 @@ A Max Age of zero follows from the same rule without a special case. Timestamp age means a burst of groups delivered together still reads as its true age, so catching up never resets the clock. Arrival age bounds what timestamps cannot: a publisher whose timestamps stop advancing, a successor that never presents a frame, and a run of empty groups. -A publisher stops sending an expired group: it closes the group's open Group Stream without a GROUP_END, and names a group it opened no stream for in a [SUBSCRIBE_DROP](#subscribe-drop) with OLD. -The subscriber keeps the frames that arrived, and a GROUP_END never claims an expired group whole. +A publisher stops sending an expired group: it resets the group's Group Stream with OLD and names the group in a [SUBSCRIBE_DROP](#subscribe-drop) with OLD, as it does a group it opened no stream for. +This includes a stream whose GROUP_END was sent but not yet acknowledged, whose bytes would otherwise hold flow control the streams behind it need. +A subscriber that already read such a stream whole, through its GROUP_END and FIN, keeps the group: the OLD drop changes nothing for its readers. ## Unidirectional Streams Unidirectional streams are used for data transmission, the session's setup, and discovery and routing. @@ -1996,6 +1997,7 @@ See the [Prioritization](#prioritization) section for more information. **Subscriber Max Age**: The optional maximum age in milliseconds that the subscriber accepts, encoded as the value plus one; 0 omits the reader bound. The publisher still applies `Publisher Max Age`, and MUST use the tighter bound while serving and while the returned group remains open. +A response whose group expires under that bound before the subscriber acknowledges it is reset with OLD, even after its GROUP_END, as a Group Stream is (see [Expiration](#expiration)). **Group Start**: The start of the group to fetch. @@ -2351,10 +2353,10 @@ A stream that FINs without a GROUP_END carries a closed Group, which may have he - Replaced GROUP's `Group Sequence` with `Group Start`, added `Previous Start`, the start of the Group before it, so a subscriber sees the gap before a Group without a SUBSCRIBE_SKIP, and removed `Frame Start`: a subscription reads whole Groups, so every Group Stream starts at frame 0. - Added GROUP_END, a `Payload Length` of 0 followed by the Group's exclusive end: a Group or Fetch Stream ends with it when its Group finished, a FIN alone ends a closed Group whose frames may not all exist, and a reset aborts. A Group's own end bounds its reach. - Expiration also measures a group's arrival age, the time since its successor arrived at this hop. -- A publisher closes the open Group Stream of a group that expires, without a GROUP_END, rather than resetting it, and a relay closes a Group it holds only the start of. +- A relay closes a Group it holds only the start of, without a GROUP_END. A publisher resets the Group Stream of a group that expires with OLD, even one whose GROUP_END it sent but is not yet acknowledged, and names the group in a SUBSCRIBE_DROP with OLD. - FRAME leads with `Payload Length`, the payload size plus one, in place of the trailing `Message Length`; 0 is malformed. - A publisher SHOULD drop a datagram rather than block when the transport has no room for it, where it previously sent one only if the congestion controller could transmit it immediately. -- Added an optional `Subscriber Max Age` to FETCH, encoded as milliseconds plus one with zero meaning absent. It can tighten, but never relax, `Publisher Max Age`. +- Added an optional `Subscriber Max Age` to FETCH, encoded as milliseconds plus one with zero meaning absent. It can tighten, but never relax, `Publisher Max Age`. A response whose group expires before it is acknowledged is reset with OLD. - A receiver drops a claim whose resolved path it cannot hold, such as one past the path limits once joined beneath its own root, rather than closing the session ([Announce](#announce)). ## moq-lite-06 diff --git a/js/net/src/broadcast.ts b/js/net/src/broadcast.ts index 76365c08965..93d227f8ba3 100644 --- a/js/net/src/broadcast.ts +++ b/js/net/src/broadcast.ts @@ -341,7 +341,9 @@ async function fetchGroup( // now: an in-progress group must keep receiving frames for its lifetime // (mirrors Rust poll_fetch), and a failed group's unread frames stay in the // cache's reach, expirable, until the caller lets them go. - void hooks.groupDone(recv.group).then(() => subscriber.close()); + void ( + (options as FetchFrom).held ? hooks.groupReleased(recv.group) : hooks.groupDone(recv.group) + ).then(() => subscriber.close()); return recv.group; case "drop": // Its own reason, so a stale verdict stays OLD: a relay's fill asking under this @@ -368,6 +370,12 @@ async function fetchGroup( */ export interface FetchFrom { frameStart?: number; + /** + * Keep the fetch's subscription, which judges the group's budget, until the reader releases + * the group rather than once it reads the end: a publisher still owed an acknowledgement + * resets a response that goes stale meanwhile. + */ + held?: boolean; } // Settles once a read of `subscriber` may go differently. diff --git a/js/net/src/group.ts b/js/net/src/group.ts index d5b626a12a3..a02535d7d18 100644 --- a/js/net/src/group.ts +++ b/js/net/src/group.ts @@ -995,6 +995,8 @@ export class Consumer { #expired?: Expired; // Closed by its reader, which ends it whatever is left unread. #released = false; + // Settles once this reader is released, made only when something waits on it. + #releasing?: PromiseWithResolvers; // The arrival of the copy first read, kept once a replacement copy is adopted. #arrived?: Instant; // The last frame handed out to stream before it completed. Cut short, it is lost, so no @@ -1107,6 +1109,11 @@ export class Consumer { makeConsumer = (state) => new Consumer(state); hooks.releaseGroupReader = (group, abort) => group.#release(abort); hooks.groupDone = (group) => group.#done(); + hooks.groupReleased = (group) => { + if (group.#released) return Promise.resolve(); + group.#releasing ??= Promise.withResolvers(); + return group.#releasing.promise; + }; hooks.readsMirrored = (group) => group.#state === group.#mirrored; hooks.groupReads = (group, reads) => group.#mirrored.reads?.(reads); hooks.groupTimestamp = (group) => @@ -1134,7 +1141,6 @@ export class Consumer { group.#expiry = expiry; }; hooks.groupFailed = (group) => group.#failed(); - hooks.groupExpired = (group) => group.#expiry?.expired() === true; hooks.recoverGroup = (group, recover) => { group.#recover = recover; // A front serving another front attaches its recovery after the inner one handed the @@ -1562,6 +1568,7 @@ export class Consumer { this.#recover = undefined; // Closing is terminal for this reader, even with frames still unread. this.#released = true; + this.#releasing?.resolve(); if (this.#state.closed.peek() === undefined) this.#state.closed.set(abort ?? null); this.#refresh(); } diff --git a/js/net/src/ietf/publisher.test.ts b/js/net/src/ietf/publisher.test.ts index 031cc7d12a9..21a7f336cf3 100644 --- a/js/net/src/ietf/publisher.test.ts +++ b/js/net/src/ietf/publisher.test.ts @@ -21,6 +21,7 @@ import { hooks } from "../internal.ts"; import { createMockTransportPair, manual, SessionBroadcast, settlesWithin } from "../mock.ts"; import { Producer as OriginProducer } from "../origin.ts"; import * as Path from "../path.ts"; +import { Front } from "../resume.ts"; import { Cursor, Reader, Stream, Writer } from "../stream.ts"; import { Timescale, Timestamp } from "../time.ts"; import type { Producer as TrackProducer } from "../track.ts"; @@ -2303,6 +2304,7 @@ function fixture( version: IetfVersion = V22, grant?: Signal, pair = createMockTransportPair(ALPNS[version]), + routed = false, ): { pair: ReturnType; pub: Publisher; @@ -2313,7 +2315,20 @@ function fixture( } { const session = new NativeSession(pair.server, version, false); const { pub, origin } = publisher(pair.server, { session, grant }); - const broadcast = publish(origin, Path.from("test")); + let broadcast: BroadcastProducer; + let front: Front | undefined; + if (routed) { + // Served through a front, so every subscriber is a resume reader over its route copies. + broadcast = new BroadcastProducer(); + front = new Front(); + front.serve(broadcast.consume()); + const dynamic = origin.claim(Path.from("test")); + void (async () => { + for await (const request of dynamic.requested()) request.serve(front.consumer); + })(); + } else { + broadcast = publish(origin, Path.from("test")); + } const uni = pair.client.incomingUnidirectionalStreams.getReader() as ReadableStreamDefaultReader< ReadableStream >; @@ -2326,6 +2341,7 @@ function fixture( version, close: () => { uni.releaseLock(); + front?.consumer.close(); origin.close(); }, }; @@ -4397,7 +4413,8 @@ test("draft-22: an update fills its snapshot under its own request ID", async () test("draft-22: an empty newer group is not an object in Largest or a relative fill", async () => { const fx = fixture(); - const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); + // No max age, so the superseded group stays fresh enough to fill. + const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI, maxAge: null }); writeGroup(track, 2); const empty = hooks.appendNext(track); const { client, ok } = await runSubscribe( @@ -4571,6 +4588,153 @@ test("draft-22: cancellation releases a fill parked on flow control", async () = } }); +// A fill whose FIN waits on an acknowledgement still holds the connection, so the subscriber +// leaving resets it, as it does a fill still writing, rather than leave it to the session. +test("draft-22: cancellation resets a fill whose FIN is unacknowledged", async () => { + const fx = fixture(); + const finished = Promise.withResolvers(); + let reset = false; + const unacknowledged = new WritableStream({ + start(controller) { + controller.signal.addEventListener("abort", () => { + reset = true; + }); + }, + close() { + finished.resolve(); + return new Promise(() => {}); + }, + }); + spyOn(fx.pair.server, "createUnidirectionalStream").mockImplementationOnce(async () => unacknowledged); + const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); + writeGroup(track, 1); + const { client } = await runSubscribe( + fx, + new Subscribe({ + requestId: 0n, + trackNamespace: Path.from("test"), + trackName: "video", + subscriberPriority: 0, + filter: { kind: "absolute", startGroup: 10n, startObject: 0n }, + fill: { filter: { kind: "unfiltered" }, rangeFilters: false }, + }), + ); + try { + await finished.promise; + expect(reset).toBe(false); + client.close(); + while (track.subscription.peek() !== undefined) await track.subscription.changed(); + expect(reset).toBe(true); + } finally { + client.close(); + track.finish(); + track.close(); + fx.close(); + } +}); + +// A fill of a group still open ends at Largest Object, so a failure of a later frame upstream, +// while its FIN waits on an acknowledgement, is outside what it served and leaves it whole. +test("draft-22: a later frame failing leaves a fill whose FIN is unacknowledged", async () => { + const fx = fixture(); + const finished = Promise.withResolvers(); + let reset = false; + const unacknowledged = new WritableStream({ + start(controller) { + controller.signal.addEventListener("abort", () => { + reset = true; + }); + }, + close() { + finished.resolve(); + return new Promise(() => {}); + }, + }); + spyOn(fx.pair.server, "createUnidirectionalStream").mockImplementationOnce(async () => unacknowledged); + const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI, maxAge: null }); + const group = hooks.appendNext(track); + group.writeFrame({ payload: new TextEncoder().encode("0.0"), timestamp: Timestamp.fromMillis(0) }); + const { client } = await runSubscribe( + fx, + new Subscribe({ + requestId: 0n, + trackNamespace: Path.from("test"), + trackName: "video", + subscriberPriority: 0, + filter: { kind: "absolute", startGroup: 10n, startObject: 0n }, + fill: { filter: { kind: "relative", groups: 1n }, rangeFilters: false }, + }), + ); + try { + await finished.promise; + group.abort(new Error("upstream failed")); + // A task turn drains every microtask the failure schedules. + await yieldNow(); + expect(reset).toBe(false); + } finally { + client.close(); + track.finish(); + track.close(); + fx.close(); + } +}); + +// A fill whose FIN waits on an acknowledgement resets once its group goes stale under the +// subscription's budget, as the Rust fill does, rather than hold the connection for stale bytes. +// Routed through an instance, the fill reads a resume reader whose own timeline stays empty, so +// the group is judged by the copy that holds it. +test.each(["plain", "routed"])( + "draft-22: a %s fill whose FIN is unacknowledged resets once its group goes stale", + async (route) => { + const fx = fixture(V22, undefined, undefined, route === "routed"); + const finished = Promise.withResolvers(); + let reset = false; + const resets = Promise.withResolvers(); + const unacknowledged = new WritableStream({ + start(controller) { + controller.signal.addEventListener("abort", () => { + reset = true; + resets.resolve(); + }); + }, + close() { + finished.resolve(); + return new Promise(() => {}); + }, + }); + spyOn(fx.pair.server, "createUnidirectionalStream").mockImplementationOnce(async () => unacknowledged); + const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI, maxAge: Duration.millis(500) }); + writeGroup(track, 1); + const { client } = await runSubscribe( + fx, + new Subscribe({ + requestId: 0n, + trackNamespace: Path.from("test"), + trackName: "video", + subscriberPriority: 0, + filter: { kind: "absolute", startGroup: 10n, startObject: 0n }, + fill: { filter: { kind: "unfiltered" }, rangeFilters: false }, + }), + ); + try { + await finished.promise; + expect(reset).toBe(false); + // The track runs a second past the filled group's successor. + for (const ms of [1000, 2000]) { + const group = hooks.appendNext(track); + group.writeFrame({ payload: new TextEncoder().encode("edge"), timestamp: Timestamp.fromMillis(ms) }); + group.finish(); + } + await resets.promise; + } finally { + client.close(); + track.finish(); + track.close(); + fx.close(); + } + }, +); + // A fill reads the cache through forks of its own, ranked at the converted subscriber // priority. Copied unconverted, wire 100 would raise the track's aggregate from 27 to 100, // and wire 191 is outside the model's -128..=127, so the fork throws and the fill never runs. diff --git a/js/net/src/ietf/publisher.ts b/js/net/src/ietf/publisher.ts index 85f3104036a..1a650d165fa 100644 --- a/js/net/src/ietf/publisher.ts +++ b/js/net/src/ietf/publisher.ts @@ -1493,7 +1493,9 @@ export class Publisher { } const out = new FetchWriter({ stream, version, priority: publisherPriority }); - const group = hooks.cachedGroup(cache, fill.start.group); + // Judged by the subscription's budget, as the Rust fill's fetched group is, so a fill + // parked on flow control or awaiting its acknowledgement gives up once it goes stale. + const group = hooks.judgedGroup(cache, fill.start.group); if (!group) throw new NotFound("fill group unavailable"); const given = untilGivenUp(group, cancelled); try { @@ -1503,12 +1505,20 @@ export class Publisher { slice: groupSlice(fill, group.start), cancelled: given.signal, }); + stream.close(); + // The stream holds every byte it owes, so frames past its range failing upstream can + // no longer give it up: release the read, leaving the budget, as `#runGroup` does. + group.close(); + // Its bytes still hold the connection until acknowledged, so a group gone stale + // meanwhile, or the subscriber leaving, resets it, as the Rust `RunFill` does. + await untilAborted( + stream.closed.catch(() => undefined), + given.signal, + ); } finally { given.release(); group.close(); } - - stream.close(); console.debug(`fill complete: fill=${requestId}`); } catch (err: unknown) { // A fill never fails the subscription; its own stream carries the failure. diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 193870c6ce6..d517d3172da 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -548,6 +548,8 @@ export interface ResumeSource { fork(options?: Subscription): Subscriber; /** A fresh reader for a group still cached by one of this reader's copies. */ cachedGroup(key: bigint, from?: number): GroupConsumer | undefined; + /** As {@link cachedGroup} from its first frame, judged by the budget of the copy it comes from. */ + judgedGroup(key: bigint): GroupConsumer | undefined; /** Let go of every copy. */ close(abort?: Error): void; } @@ -874,6 +876,11 @@ export const hooks: { staleAt: (subscriber: Subscriber, key: bigint) => Expiry; /** A group the subscriber's track still caches, as a fresh read handle. */ cachedGroup: (subscriber: Subscriber, key: bigint, from?: number) => GroupConsumer | undefined; + /** + * As {@link cachedGroup} from its first frame, the reader given up once the subscription's + * budget gives up on it, judged by the route copy it is read from when routed. + */ + judgedGroup: (subscriber: Subscriber, key: bigint) => GroupConsumer | undefined; /** Whether {@link cachedGroup} would find the group, without making a read handle. */ holdsGroup: (subscriber: Subscriber, key: bigint) => boolean; /** @@ -977,6 +984,8 @@ export const hooks: { releaseGroupReader: (group: GroupConsumer, abort?: Error) => void; /** Settles once a group reader read or released everything it holds, its {@link GroupConsumer.done}. Assigned by `group.ts`. */ groupDone: (group: GroupConsumer) => GetPromise; + /** Settles once the reader releases the group (`close` or an abort), however far it read. */ + groupReleased: (group: GroupConsumer) => Promise; /** Whether a group reader still reads the copy it was mirrored from, not one its recovery moved it to. */ readsMirrored: (group: GroupConsumer) => boolean; /** @@ -1002,11 +1011,6 @@ export const hooks: { expireGroup: (group: GroupConsumer, expiry: Expiry) => void; /** The failure that ends a group's reader, as its `closed` would settle, without building that handle. */ groupFailed: (group: GroupConsumer) => Error | undefined; - /** - * Whether the subscription's budget gave the group up, rather than its copy failing or the - * cache evicting it. Mirrors the Rust crate-private `group::Consumer::is_expired`. - */ - groupExpired: (group: GroupConsumer) => boolean; /** * Carry a group across its logical track's route changes. Attached before the group is handed * out, so its `closed` never settles on a failure the recovery continues past. @@ -1253,6 +1257,9 @@ export const hooks: { cachedGroup: () => { throw new Error("track.ts not loaded"); }, + judgedGroup: () => { + throw new Error("track.ts not loaded"); + }, holdsGroup: () => { throw new Error("track.ts not loaded"); }, @@ -1373,6 +1380,9 @@ export const hooks: { releaseGroupReader: () => { throw new Error("group.ts not loaded"); }, + groupReleased: () => { + throw new Error("group.ts not loaded"); + }, groupDone: () => { throw new Error("group.ts not loaded"); }, @@ -1409,9 +1419,6 @@ export const hooks: { groupFailed: () => { throw new Error("group.ts not loaded"); }, - groupExpired: () => { - throw new Error("group.ts not loaded"); - }, recoverGroup: () => { throw new Error("group.ts not loaded"); }, diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 3ff309467b2..86f5cd3c6d3 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1407,21 +1407,25 @@ test("lite draft-06: a reported cut stays dropped when re-created", async () => // A group stream that never completes after its FIN still holds the connection. Like the // Rust `GroupServe`, the publisher waits on it only while the age budget keeps the group: // the newest group is never stale, and once its successor arrived a full budget ago the -// stream is reset with `Old` and the group dropped. -test("lite draft-06: a group stream that never completes is reset once the group goes stale", async () => { +// stream is reset with `Old` and the group dropped. Lite-07 resets it too, though its +// GROUP_END already went out: stale bytes must not hold the send window. +test.each([ + ["draft-06", Version.DRAFT_06], + ["draft-07", Version.DRAFT_07], +])("lite %s: a group stream that never completes is reset once the group goes stale", async (_, version) => { const clock = new Manual(); const installed = clock.install(); const budget = 1_000; // The track's max age caps the subscription's budget. const sub = await servedSubscription({ - version: Version.DRAFT_06, + version, startGroup: 0n, maxAge: Duration.millis(budget), }); try { sub.serve(0n); expect(await sub.servedSequence()).toBe(0n); - expect("start" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_06))).toBe(true); + expect("start" in (await decodeSubscribeResponse(sub.client.reader, version))).toBe(true); let reason: unknown; const reset = sub.stall().then((value) => { @@ -1440,7 +1444,7 @@ test("lite draft-06: a group stream that never completes is reset once the group await clock.advance(Duration.millis(1)); await reset; expect(toStreamCode(fromTransport(reason))).toBe(StreamCode.Old); - const resp = await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_06); + const resp = await decodeSubscribeResponse(sub.client.reader, version); if (!("drop" in resp)) throw new Error("expected SUBSCRIBE_DROP"); expect([resp.drop.start, resp.drop.end]).toEqual([1n, 2n]); } finally { @@ -1461,14 +1465,14 @@ async function holesToEnd(reader: Reader): Promise { } } -// A lite-07 stream that expired as stale is closed rather than reset, and accounts for its group, so -// no drop names it: a final drop the reader reaches for it later does not go out over it (one still -// arriving, as a relay's copy from upstream, is cut by it), and a widening that admits it again does -// not re-offer it (one cached whole, whose FIN went out first). +// A lite-07 stream that expires as stale resets with OLD, whether its group is still arriving (as a +// relay's copy from upstream) or went out whole with its FIN unacknowledged, and is named in one stale +// drop: a final drop the reader reaches for it later does not go out over it, and a widening that +// admits it again does not re-offer it, since every drop is final. test.each([ ["still arriving", true], - ["cached whole", false], -])("lite draft-07: an expired stream %s stays dropped", async (_, arriving) => { + ["sent whole", false], +])("lite draft-07: an expired stream %s resets and is dropped once", async (_, arriving) => { const clock = new Manual(); const installed = clock.install(); const budget = 1_000; @@ -1486,9 +1490,9 @@ test.each([ // The restated range resolves at once. expect("started" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07))).toBe(true); - let reset = false; - void sub.stall(true).then(() => { - reset = true; + let reset: unknown; + void sub.stall().then((reason) => { + reset = reason; }); if (arriving) sub.track.createGroup(1n).writeFrame(textFrame("hello", Timestamp.fromMillis(1))); else sub.serve(1n); @@ -1499,28 +1503,25 @@ test.each([ expect(await sub.servedSequence()).toBe(2n); await clock.advance(Duration.millis(budget)); await flush(); - expect(reset).toBe(false); + expect(toStreamCode(fromTransport(reset))).toBe(StreamCode.Old); hooks.dropTrackGroups(sub.track, 1n, 2n, { error: new StreamError(StreamCode(70), { message: "cut" }) }); await update(budget * 100, 1n); - expect("started" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07))).toBe(true); await flush(); sub.serve(3n); expect(await sub.servedSequence()).toBe(3n); sub.track.finish(); sub.track.close(); - expect(await holesToEnd(sub.client.reader)).toEqual([]); - expect(reset).toBe(false); + expect(await holesToEnd(sub.client.reader)).toEqual([["drop", 1n, 2n]]); } finally { await sub.close(); installed.close(); } }); -// A lite-07 stream whose group expires mid-frame waits for that frame before it closes, and resets -// with the frame's failure when the frame fails instead: a FIN would cut the frame short. Mirrors the -// Rust `GroupServe`, which closes an expired stream only at a frame boundary. -test("lite draft-07: an expired stream resets when its frame in flight fails", async () => { +// A lite-07 stream whose group expires mid-frame resets with OLD at once rather than wait for the +// frame, as the Rust `GroupServe` does. +test("lite draft-07: a stream expiring mid-frame resets at once", async () => { const clock = new Manual(); const installed = clock.install(); const budget = 1_000; @@ -1538,7 +1539,7 @@ test("lite draft-07: an expired stream resets when its frame in flight fails", a expect("started" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07))).toBe(true); let reset: unknown; - void sub.stall(true).then((reason) => { + void sub.stall().then((reason) => { reset = reason; }); const group = sub.track.createGroup(1n); @@ -1549,14 +1550,10 @@ test("lite draft-07: an expired stream resets when its frame in flight fails", a await clock.advance(Duration.millis(budget * 10)); sub.serve(2n); expect(await sub.servedSequence()).toBe(2n); - await clock.advance(Duration.millis(budget)); - await flush(); expect(reset).toBeUndefined(); - - const cut = new Error("frame cut"); - frame.abort(cut); + await clock.advance(Duration.millis(budget)); await flush(); - expect(reset).toBe(cut); + expect(toStreamCode(fromTransport(reset))).toBe(StreamCode.Old); } finally { await sub.close(); installed.close(); @@ -3452,6 +3449,69 @@ test("lite draft-07: a fetch whose budget gives up partway resets as too old", a } }); +// A fetch sent whole whose FIN the requester has yet to acknowledge still holds the connection, so +// it resets as too old once its budget gives the group up, as the Rust `FetchServe` does. +test("lite draft-07: a fetch sent whole resets as too old when it goes stale unacknowledged", async () => { + const pair = createMockTransportPair(ALPN_07_WIP); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, Version.DRAFT_07, randomHop(), origin.consume()); + const broadcast = publish(origin, Path.from("live")); + const track = broadcast.createTrack("video", { timescale: Timescale.MILLI, maxAge: null }); + const frame = (ms: number) => ({ payload: new TextEncoder().encode("x"), timestamp: Timestamp.fromMillis(ms) }); + const whole = track.createGroup(0n); + whole.writeFrame(frame(0)); + whole.finish(); + + const client = await Stream.open(pair.client, { version: Version.DRAFT_07 }); + const incoming = pair.server.incomingBidirectionalStreams.getReader(); + const accepted = await incoming.read(); + incoming.releaseLock(); + if (accepted.done) throw new Error("publisher never accepted the fetch stream"); + // The FIN is never acknowledged: the close stays pending until the stream resets. + const inner = accepted.value.writable.getWriter(); + let reset: unknown; + const writable = new WritableStream({ + start(controller) { + controller.signal.addEventListener("abort", () => { + reset = controller.signal.reason; + void inner.abort(controller.signal.reason); + }); + }, + write: (chunk) => inner.write(chunk), + close: () => new Promise(() => undefined), + }); + const server = new Stream({ readable: accepted.value.readable, writable, version: Version.DRAFT_07 }); + + try { + const msg = new Fetch({ + broadcast: Path.from("live"), + track: "video", + priority: 0, + maxDelay: 500, + group: 0n, + }); + const serving = publisher.runFetch(msg, server); + // The whole group and its FIN go out before any successor arrives. + await client.reader.u8(); + await flush(); + expect(reset).toBeUndefined(); + + // Then the track runs a second past its successor, with the FIN still unacknowledged. + for (const ms of [1000, 2000]) { + const group = track.createGroup(BigInt(ms / 1000)); + group.writeFrame(frame(ms)); + group.finish(); + } + await serving; + expect(toStreamCode(fromTransport(reset))).toBe(StreamCode.Old); + } finally { + publisher.close(); + client.close(); + broadcast.close(); + origin.close(); + } +}); + test.each([0, 1])("lite draft-07 reports the cached largest position when starting at time %s", async (start) => { const startGroup = BigInt(start); const version = Version.DRAFT_07; diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 28c4e2414a9..3f016ffef73 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -308,7 +308,10 @@ type ServeGroup = { }; /** What serving fetched frames needs beyond the group and destination stream. */ -type ServeFetch = Omit; +type ServeFetch = Omit & { + /** Why to stop writing the group, which a write parked on flow control races. */ + givenUp: GivenUp; +}; /** What the serving loop takes from the subscribe stream instead of serving a group. */ type Control = @@ -1261,6 +1264,7 @@ export class Publisher { // A FETCH requester leaves early by resetting its stream or losing its session, which // lets go of a lookup still waiting on the answer. let group: group.Consumer | undefined; + let givenUp: GivenUp | undefined; try { // The timescale is immutable, so serve exactly what TRACK_INFO advertised. Both // come off the same front, so the metadata and the frames are one generation. @@ -1279,6 +1283,8 @@ export class Publisher { signal: left.signal, // A copy holding the frames asked on serves them; one missing them is filled whole. frameStart: msg.startFrame, + // The budget judges the group until the response is acknowledged. + held: true, }; group = await front.track(msg.track).fetchGroup(msg.group, options); serving(group); @@ -1287,21 +1293,32 @@ export class Publisher { hooks.releaseGroupReader(group, unauthorized(msg.broadcast)); return; } - await this.#runFetchGroup(group, stream.writer, { - stamp: stamping(this.version, info), - compress: info.compress, - start: msg.startFrame, - end: msg.endFrame, - }); - console.debug(`fetch done: broadcast=${msg.broadcast} track=${msg.track} group=${msg.group}`); - stream.close(); - group.close(); - await acknowledged(stream.writer); + // The FETCH's own budget gives the group up, while a write waits on flow control and + // while its FIN waits for the acknowledgement, as the Rust `FetchServe` does: stale + // bytes must not hold the send window ahead of the streams behind them. + givenUp = new GivenUp(group); + try { + await this.#runFetchGroup(group, stream.writer, { + stamp: stamping(this.version, info), + compress: info.compress, + start: msg.startFrame, + end: msg.endFrame, + givenUp, + }); + console.debug(`fetch done: broadcast=${msg.broadcast} track=${msg.track} group=${msg.group}`); + stream.close(); + // Held until the acknowledgement, so its budget still judges it meanwhile. + const stop = await givenUp.write(() => acknowledged(stream.writer)); + if (stop) throw stop; + group.close(); + } finally { + givenUp.close(); + } } catch (err: unknown) { const e = error(err); - console.warn( - `fetch error: broadcast=${msg.broadcast} track=${msg.track} group=${msg.group} error=${reason(e)}`, - ); + // Giving up a group past its budget is routine, as in Rust's `RequestServe`. + const log = e === givenUp?.peek() ? console.debug : console.warn; + log(`fetch error: broadcast=${msg.broadcast} track=${msg.track} group=${msg.group} error=${reason(e)}`); if (group) hooks.releaseGroupReader(group, e); stream.abort(e); } @@ -2295,8 +2312,12 @@ export class Publisher { async #runFetchGroup( group: group.Consumer, stream: Writer, - { stamp, compress, start: startFrame, end: endFrame }: ServeFetch, + { stamp, compress, start: startFrame, end: endFrame, givenUp }: ServeFetch, ) { + const write = async (op: () => Promise) => { + const stop = await givenUp.write(op); + if (stop) throw stop; + }; // The response carries no header, so the receiver numbers the first frame it gets // as `startFrame`. Skipping the head here is the only thing keeping those numbers // honest; a group that ends before we reach it can't be served at all. @@ -2321,16 +2342,25 @@ export class Publisher { prevTs = ts; } - await stream.write( - frameHeader(stream.version, delta, read.inflated, readSize(read), hasLengthFirst(this.version)), + const header = frameHeader( + stream.version, + delta, + read.inflated, + readSize(read), + hasLengthFirst(this.version), ); - if (read.chunks) await sendPayload(read.chunks, (chunk) => stream.write(chunk), stream.closed); - else if (frame.payload.length > 0) await stream.write(frame.payload); + const chunks = read.chunks; + // One budget race covers the frame's header and payload. + await write(async () => { + await stream.write(header); + if (chunks) await sendPayload(chunks, (chunk) => stream.write(chunk), stream.closed); + else if (frame.payload.length > 0) await stream.write(frame.payload); + }); } // Lite-07: a response that reached a finished group's end says so before the FIN. const end = group.end; if (hasGroupEnd(this.version) && end !== undefined && hooks.groupIndex(group) >= group.frameCount) - await stream.write(groupEnd(stream.version, end)); + await write(() => stream.write(groupEnd(stream.version, end))); } /** @@ -2389,20 +2419,10 @@ export class Publisher { const { sub, group, givenUp, stamp, compress, priority, previous, start: startFrame, end: endFrame } = options; // The header names the requested position, so a copy missing it must be refused. const msg = new GroupMessage({ subscribe: sub, key: group.start, previous, frameStart: startFrame }); - // Lite-07 ends a finished group with GROUP_END and closes, rather than resets, the stream of - // a group that expires on it: the stream accounts for the group, so no drop names it. - const closes = hasGroupEnd(this.version); - // The write under way, which an expiry close lets complete, so the stream ends on a frame. - let writing: Promise | undefined; - // Whether the stream's end went out, after which it is never reset for expiry on lite-07. - let ended = false; // The subscription's budget, or an eviction of what is left to read, may give up on the // group mid-write, which abandons the stream. A group already given up never starts the write. const write = async (op: () => Promise) => { - const stop = await givenUp.write(() => { - writing = op(); - return writing; - }); + const stop = await givenUp.write(op); if (stop) throw stop; }; // Everything past this point runs inside the cleanup scope, so a failure never leaves @@ -2469,38 +2489,19 @@ export class Publisher { }); } - const end = closes && whole ? group.end : undefined; + // Lite-07 ends a finished group with GROUP_END. + const end = hasGroupEnd(this.version) && whole ? group.end : undefined; if (end !== undefined) await write(() => stream.write(groupEnd(stream.version, end))); stream.close(); - ended = true; group.close(); // A stream that fails after its FIN may have lost its header, so report it for a DROP. - // Its bytes still hold the connection until acknowledged, so before lite-07 a group gone - // stale meanwhile is reset like one still serving, as the Rust `GroupServe` does. - if (closes) await stream.closed; - else await write(() => stream.closed); + // Its bytes still hold the connection until acknowledged, so a group gone stale meanwhile + // is reset like one still serving, as the Rust `GroupServe` does: stale bytes must not fill + // the send window ahead of the streams behind them. + await write(() => stream.closed); return undefined; } catch (err: unknown) { - let e = error(err); - if (closes && !ended && hooks.groupExpired(group)) { - // Expired on its stream: what went out stands as a closed group, once the frame being - // written completes. One that fails resets the stream with its failure instead, since a - // FIN would cut it short. Mirrors the Rust `GroupServe`'s expiry close. - const failed = await writing?.then( - () => undefined, - (err: unknown) => error(err), - ); - if (failed === undefined) { - console.debug(`closing expired group: subscribe=${sub} key=${group.start}`); - stream.close(); - group.close(); - return await stream.closed.then( - () => undefined, - (err: unknown) => error(err), - ); - } - e = failed; - } + const e = error(err); stream.reset(e); hooks.releaseGroupReader(group, e); return e; diff --git a/js/net/src/lite/version.ts b/js/net/src/lite/version.ts index 47800bf27fc..406e4f9df48 100644 --- a/js/net/src/lite/version.ts +++ b/js/net/src/lite/version.ts @@ -730,9 +730,8 @@ export function hasLengthFirst(version: Version): boolean { /** * Whether GROUP carries the previous group's `Previous Start`, and a Group or Fetch Stream ends a - * finished group with GROUP_END and a closed one with a bare FIN. A publisher then closes an - * expired group's stream rather than resetting it. Added in lite-07; earlier versions end every - * group with a bare FIN. + * finished group with GROUP_END and a closed one with a bare FIN. Added in lite-07; earlier + * versions end every group with a bare FIN. */ export function hasGroupEnd(version: Version): boolean { switch (version) { diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 65b2abeb168..3d2d27fb58a 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -2,7 +2,7 @@ import { expect, spyOn, test } from "bun:test"; import { Duration, yieldNow } from "@moq/clock"; import { Manual } from "@moq/clock/test"; import { Signal } from "@moq/signals"; -import { Consumer as BroadcastConsumer, Producer as BroadcastProducer } from "./broadcast.ts"; +import { Consumer as BroadcastConsumer, Producer as BroadcastProducer, type FetchFrom } from "./broadcast.ts"; import { NotFound, SessionCode, SessionError, StreamCode, StreamError } from "./error.ts"; import { type Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; import { WireRoute } from "./hop.ts"; @@ -1949,6 +1949,35 @@ test("a fetched group continues from the next route once its own fails", async ( b.close(); }); +test("a held fetch lets go of the route it resumed onto once its reader releases the group", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + const stalled = ta.write(5n, 1, false); + front.serve(a.consume()); + // Held, as the lite publisher asks while a FETCH reply waits for its acknowledgement. + const held: track.FetchGroupOptions & FetchFrom = { held: true }; + const group = await front.consumer.track("seq").fetchGroup(5n, held); + expect(dec.decode((await group.readFrame())?.payload)).toBe("5.0"); + + tb.write(5n, 2); + front.serve(b.consume()); + stalled.abort(new StreamError(StreamCode.SessionClosed)); + expect(dec.decode((await group.readFrame())?.payload)).toBe("5.1"); + expect(await group.readFrame()).toBeUndefined(); + await settle(); + expect(tb.producer.demand().wanted.peek()).toBe(true); + + group.close(); + await unwanted(tb.producer.demand()); + + front.consumer.close(); + a.close(); + b.close(); +}); + test("a replacement spliced before the first copy answers is still checked against it", async () => { const front = new Front(); const a = new BroadcastProducer(); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index b3495b18153..ecb684f08c9 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -796,6 +796,18 @@ class Reader implements ResumeSource { return undefined; } + judgedGroup(key: bigint): GroupConsumer | undefined { + for (let index = this.#copies.length - 1; index >= 0; index--) { + const copy = this.#copies[index]; + const group = hooks.cachedGroup(copy.sub, key); + if (!group) continue; + // Judged by the copy it is read from: this reader's own timeline stays empty. + hooks.expireGroup(group, hooks.staleAt(copy.sub, key)); + return this.#handOut(copy, group); + } + return undefined; + } + /** * `copy` handed out `key`. The serving copy's first group bounds the copies it replaced * just below it: what they hold under it may be groups the new route never has, and what @@ -2359,6 +2371,9 @@ export class Front { adopt: () => { current.release(); current = next.source; + // Its fetch may hold its subscription until its reader lets go, and the + // reader is now `fetched.group`. + void hooks.groupReleased(fetched.group).then(() => next.group.close()); }, discard: () => { next.group.close(); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 9b48b8041fe..1c18fde31f4 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -4299,6 +4299,12 @@ export class Subscriber { const producer = subscriber.#state.producer; return producer && cachedMirror(producer, key, from); }; + hooks.judgedGroup = (subscriber, key) => { + if (subscriber.#resume) return subscriber.#resume.judgedGroup(key); + const group = hooks.cachedGroup(subscriber, key); + if (group) hooks.expireGroup(group, hooks.staleAt(subscriber, key)); + return group; + }; hooks.holdsGroup = (subscriber, key) => { const producer = subscriber.#state.producer; return producer !== undefined && cachedHolds(producer, key); diff --git a/js/net/src/util/given_up.ts b/js/net/src/util/given_up.ts index bd8acce971e..1904d37fb71 100644 --- a/js/net/src/util/given_up.ts +++ b/js/net/src/util/given_up.ts @@ -43,8 +43,8 @@ export class GivenUp implements GetPromise { async write(write: () => Promise, cancel?: Promise): Promise { // Judged afresh rather than read from the watch: a wall-clock deadline can pass before // its timer wakes the watch. - const stop = this.#stop ?? reason(this.#group); - if (stop) return stop; + this.#check(); + if (this.#stop) return this.#stop; const done = write().then(() => true as const); const result = await race(cancel === undefined ? [done, this] : [done, this, cancel]); return result === true || result === false ? undefined : result; diff --git a/quest/m0/js-relayed-fetch-budget.md b/quest/m0/js-relayed-fetch-budget.md new file mode 100644 index 00000000000..64923fa33ed --- /dev/null +++ b/quest/m0/js-relayed-fetch-budget.md @@ -0,0 +1,24 @@ +# [M] A JS relay judges a relayed FETCH by its budget + +## Goal + +A JS publisher serving a lite FETCH for a broadcast it consumes over the wire +(through a `Front`) resets the response with OLD once the group goes stale +while the reply waits on flow control or its acknowledgement, as it already +does for a broadcast it produces and as a Rust relay does. + +## Plan + +Found reviewing [moq-dev/moq#9140](https://github.com/moq-dev/moq/pull/9140). +`js/net/src/lite/publisher.ts` `#serveFetch` judges the group with `GivenUp`, +but a group from `Front.fetchGroup` comes from the wire +(`lite/subscriber.ts` `fetchGroup`) with no expiry attached, so only an +upstream reset or failure gives it up. Once the upstream reply is +acknowledged, nothing judges the downstream one. Rust has no gap: a relayed +fetch fills the track cache, whose policy judges it. + +Recommendation: attach an expiry judged by the route copy that holds the +track, as the routed draft-22 fill does through `ResumeSource.staleAt`, +without opening a subscription: a fetch-only relay must not start pulling the +track upstream. With no copy subscribed, nothing newer is known locally, which +matches Rust's cache. diff --git a/quest/m0/lite07-stale-reset.md b/quest/m0/lite07-stale-reset.md deleted file mode 100644 index 0abc206a3a2..00000000000 --- a/quest/m0/lite07-stale-reset.md +++ /dev/null @@ -1,15 +0,0 @@ -# [L] A lite-07 publisher resets a stale group's stream when it expires - -## Goal - -A lite-07 publisher aborts every group stream whose group expired, including one whose end it already wrote, so stale video can't fill the connection's send window and stall the streams behind it. The scheduled TS workflow's validation passes again: no audio expires on its 7 Mb/s path. - -## Plan - -Caused by [moq-dev/moq#8908](https://github.com/moq-dev/moq/pull/8908) (`fb443a5ac7`), found by bisecting the TS validation failure (run [44135366690](https://github.com/moq-dev/moq/actions/runs/44135366690), still failing on 11-18). Under lite-07, a stale group stream whose GROUP_END was already written is no longer reset when the group expires (`poll_stopped` and `closes_expired` in `rs/moq-net/src/lite/publisher.rs`). Stale video groups then fill moq-quic's 8 MiB connection send window, the control stream sending SUBSCRIBE_SKIP stalls 1–10 s behind them, and audio expires: 60–100 s lost out of 120. Removing the `!has_group_end()` guard brings audio loss to zero. - -Decided 2026-11-18 (maintainer, /quest-plan): ✅ reset stale groups again, as lite-06 did, in Rust, JS, the draft's Expiration section, and the `verify-subscribe` model (rejected: keep the wire rule and let higher-priority streams get past a full send window in moq-quic, since stale video would still use bottleneck bandwidth and other QUIC stacks would stall the same way; rejected: measure both first). This relaxes #8908's rule that each group is named only once; lite-07 is unpublished `-wip`, so the wire change is free. - -Decided 2026-11-18 (maintainer): moq-net never finishes a stream or producer and forgets it; something must abort it eventually. The rule is in `rs/moq-net/AGENTS.md`. As part of this quest, audit moq-net (Rust and `js/net`) for other streams or producers that are finished and then never aborted, such as fetch, datagram, and announce streams, and fix each one found or plan it as a follow-up. - -Public API: none. Wire: lite-07-wip expiry resets a finished group's stream again; update the draft in the same PR. diff --git a/quest/m0/test-flakes-3/fleet-drain-redial.md b/quest/m0/test-flakes-3/fleet-drain-redial.md new file mode 100644 index 00000000000..602498f52da --- /dev/null +++ b/quest/m0/test-flakes-3/fleet-drain-redial.md @@ -0,0 +1,19 @@ +# [S] A fleet drain's redial delivers the next group under load + +## Goal + +`moq-tokio` `connection::tests::a_fleet_drain_redials_through_a_fresh_resolve` +passes under a loaded `just check`. + +## Plan + +Seen once on 2026-11-18 during `just check` for +[moq-dev/moq#9140](https://github.com/moq-dev/moq/pull/9140), which does not +touch moq-tokio or GOAWAY: the subscriber's `recv_group` for `g1` failed with +`Session(GoawayTimeout)` (`rs/moq-tokio/src/connection.rs`, the `g1` wait). It +passed 5 of 5 alone and 40 of 40 under `--stress-count 40 --test-threads 16`, +taking 0.5 to 3 s alone, so it rides on wall-clock time. + +Find why the drained session's deadline can end the subscription before the +redial to B carries `g1`, and drive the timers with a manual clock or assert on +the handover event instead of real time, per the epic's rules. diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index 863b169f825..e3f11dee19a 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -2749,10 +2749,14 @@ impl RunFill { tracing::debug!(%err, fill = %self.request_id, "fill stream close failed"); return Poll::Ready(()); } - let closed = wait!(Settle => writer.poll_close(waiter)); + // Or until the group expires: stale bytes must not hold the connection. + let closed = wait!(Settle => match writer.poll_close(waiter) { + Poll::Pending if write.poll_stale(waiter) => Poll::Ready(Err(Error::Old)), + closed => closed, + }); match closed { Ok(()) => tracing::debug!(fill = %self.request_id, "fill complete"), - Err(err) => tracing::debug!(%err, fill = %self.request_id, "fill stream close failed"), + Err(err) => return self.fail(writer, err), } Poll::Ready(()) } @@ -5483,11 +5487,31 @@ impl FetchRespond { writer.buffer(&FetchHeader { request_id: self.request_id, })?; - wait!(Header => writer.poll_flush(waiter))?; - wait!(Body => self.write.poll(waiter, writer))?; - // Settle: the writer keeps the stream, and its rank, until the acknowledgement. + // Parked on flow control, a group gone stale resets its stream rather than pin it. + let header = wait!(Header => match writer.poll_flush(waiter) { + Poll::Pending if self.write.poll_stale(waiter) => Poll::Ready(Err(Error::Old)), + flushed => flushed, + }); + if let Err(err) = header { + writer.abort(&err); + return Poll::Ready(Err(err)); + } + let body = wait!(Body => self.write.poll(waiter, writer)); + if let Err(err) = body { + writer.abort(&err); + return Poll::Ready(Err(err)); + } + // Settle: the writer keeps the stream, and its rank, until the acknowledgement, or + // until the group expires, which resets it. writer.finish()?; - Poll::Ready(wait!(Settle => writer.poll_close(waiter))) + let settled = wait!(Settle => match writer.poll_close(waiter) { + Poll::Pending if self.write.poll_stale(waiter) => Poll::Ready(Err(Error::Old)), + closed => closed, + }); + if let Err(err) = &settled { + writer.abort(err); + } + Poll::Ready(settled) } } @@ -7451,6 +7475,12 @@ impl FetchGroupWrite { res } + /// Whether the group expired, so a written stream awaiting its acknowledgement resets + /// rather than hold the connection for stale bytes. + fn poll_stale(&mut self, waiter: &kio::Waiter) -> bool { + self.group.poll_expired(waiter) + } + fn poll_objects( &mut self, waiter: &kio::Waiter, @@ -7514,7 +7544,10 @@ impl FetchGroupWrite { } FetchGroupState::Batch { next, payload } => { if let Some(write) = payload { - ready!(write.poll(waiter, stream))?; + let Poll::Ready(written) = write.poll(waiter, stream) else { + return poll_parked(&mut self.group, waiter); + }; + written?; *payload = None; self.index += 1; self.group.keep_alive(); @@ -7556,9 +7589,15 @@ impl FetchGroupWrite { } } FetchGroupState::Partial { frame, chunk } => { - ready!(stream.poll_flush(waiter))?; + let Poll::Ready(flushed) = stream.poll_flush(waiter) else { + return poll_parked(&mut self.group, waiter); + }; + flushed?; if let Some(bytes) = chunk { - ready!(stream.poll_write_all(waiter, bytes))?; + let Poll::Ready(written) = stream.poll_write_all(waiter, bytes) else { + return poll_parked(&mut self.group, waiter); + }; + written?; *chunk = None; } if stream.poll_closed(waiter).is_ready() { @@ -7750,6 +7789,12 @@ fn poll_given_up(group: &mut group::Consumer, waiter: &kio::Waiter) -> Option Poll> { + poll_given_up(group, waiter).map_or(Poll::Pending, |err| Poll::Ready(Err(err))) +} + /// Serves one group on its own unidirectional stream in the moq-transport /// subgroup format. struct GroupServe { @@ -14771,6 +14816,125 @@ mod serve_tests { assert_eq!(fetch_object_priorities(&mut buf, version), [Some(55)]); } + /// A fill sent whole whose FIN the subscriber never acknowledges resets once its group goes + /// stale, rather than hold the connection for stale bytes until the subscription ends. + #[kio::test] + async fn a_fill_resets_once_its_unacknowledged_group_goes_stale() { + let version = Version::Draft22; + let budget = Duration::from_secs(2); + let session = ScriptedSession::per_stream(vec![Vec::new()]).with_unacked_uni_fins(1); + let h = serve_session(version, track::Info::timed().with_max_age(Some(budget)), session); + // Frozen timestamps, so only the wall clock since group 5 arrived convicts group 4. + let mut group = h.track.create_group(group::Info { start: 4 }).unwrap(); + group.write_frame(group.key_time(), b"fill".as_slice()).unwrap(); + group.finish().unwrap(); + + let ranking = Ranking { + queue: h.publisher.priority.clone(), + group: None, + request_id: RequestId(REQUEST_ID), + priority: kio::Producer::new(PRIORITY).consume(), + publisher: 0, + order: Order::Descending, + lapse: Lapse::default(), + }; + let opened = AtomicU64::new(0); + let fill = FillServe::Group { + key: 4, + skip: 0, + until: None, + }; + let mut run = + std::pin::pin!( + h.publisher + .run_fill(fill, &ranking, h.track.consume(), &opened, ranking.request_id) + ); + let finished = || h.log.trail().iter().any(|(_, sent)| *sent == Sent::Finish); + kio::clock::timeout(Duration::from_secs(1), async { + while !finished() { + assert!(futures::poll!(run.as_mut()).is_pending(), "the FIN is unacknowledged"); + kio::clock::sleep(Duration::from_millis(1)).await; + } + }) + .await + .expect("the fill went out whole"); + + let mut next = h.track.create_group(group::Info { start: 5 }).unwrap(); + next.write_frame(next.key_time(), b"next".as_slice()).unwrap(); + next.finish().unwrap(); + kio::clock::sim::advance(budget - Duration::from_millis(1)).await; + assert!(futures::poll!(run.as_mut()).is_pending(), "still within the budget"); + assert!(h.log.resets().is_empty()); + + kio::clock::sim::advance(Duration::from_millis(1)).await; + kio::clock::timeout(budget, run).await.expect("the stale fill resets"); + assert_eq!(h.log.resets().len(), 1, "the fill's stream reset"); + } + + /// A standalone FETCH whose FETCH_HEADER is parked on flow control resets once its group + /// goes stale, rather than hold the stream and its rank for a response nobody wants. + #[kio::test] + async fn a_fetch_parked_on_its_header_resets_once_its_group_goes_stale() { + for version in FETCH_DRAFTS { + let budget = Duration::from_secs(2); + // One byte of credit, so the FETCH_HEADER starts and then parks. + let session = ScriptedSession::per_stream(vec![Vec::new()]).with_uni_send_window(1); + let h = serve_session(version, track::Info::timed().with_max_age(Some(budget)), session); + // Frozen timestamps, so only the wall clock since group 1 arrived convicts group 0. + let mut group = h.track.create_group(group::Info { start: 0 }).unwrap(); + group.write_frame(group.key_time(), b"fetch".as_slice()).unwrap(); + group.finish().unwrap(); + + let stream = Stream::open(&crate::driver::Transport::new(h.session.clone()), version) + .await + .unwrap(); + let fetch_type = fetch_range( + version, + "video", + Location { group: 0, object: 0 }, + Location { group: 0, object: 0 }, + ); + let mut fetching = std::pin::pin!(h.publisher.clone().run_fetch_stream( + stream, + ietf::Fetch { + authorization_token: None, + request_id: FETCH_ID, + subscriber_priority: 128, + group_order: GroupOrder::Ascending, + fetch_type, + range_filters: false, + fill_timeout: true, + properties_wanted: false, + }, + )); + serve_until(fetching.as_mut(), "the FETCH_HEADER started", || { + !data_streams(&h.log).is_empty() + }) + .await; + + let mut next = h.track.create_group(group::Info { start: 1 }).unwrap(); + next.write_frame(next.key_time(), b"next".as_slice()).unwrap(); + next.finish().unwrap(); + kio::clock::sim::advance(budget - Duration::from_millis(1)).await; + assert!( + futures::poll!(fetching.as_mut()).is_pending(), + "{version}: within the budget" + ); + assert!(h.log.resets().is_empty(), "{version}"); + + kio::clock::sim::advance(Duration::from_millis(1)).await; + let _ = kio::clock::timeout(budget, fetching.as_mut()) + .await + .unwrap_or_else(|_| panic!("{version}: the stale fetch stayed parked")); + let resets: Vec<_> = data_streams(&h.log) + .into_values() + .flatten() + .filter(|sent| matches!(sent, Sent::Reset(_))) + .collect(); + assert_eq!(resets.len(), 1, "{version}: the fetch stream reset"); + } + } + /// The Track Properties block a publisher upstream of a relay sent on `version`: an even /// property the draft defines but moq-net does not implement (OBJECT_DELIVERY_TIMEOUT), /// and an odd one from the application range. diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index e4973db61ee..dcb79461b61 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -1608,6 +1608,12 @@ trait Request: Sized { const HOLDS: bool; /// The message that opens the stream, owned so it outlives the bytes it came from. type Message: std::fmt::Debug; + + /// The group a finished reply carried, kept until the peer acknowledges it so the stream + /// resets if the group expires first. + fn stale(self) -> Option { + None + } /// What the requester may send after the request, as it travels; [`NoUpdate`] for nothing. type Update: Resolve; @@ -1772,6 +1778,8 @@ enum RequestState { /// FIN and wait for the acknowledgement. Finish { finished: bool, + /// The group a reply carried, which resets the stream if it expires first. + stale: Option, }, } @@ -1827,6 +1835,9 @@ impl RequestServe { Error::Unauthorized | Error::Stream(crate::StreamError::Unauthorized) => { tracing::info!(broadcast = %crate::path::Rooted(self.shared.origin.root(), &self.broadcast), track = %self.track(), "{} unauthorized", R::KIND) } + Error::Old => { + tracing::debug!(broadcast = %crate::path::Rooted(self.shared.origin.root(), &self.broadcast), track = %self.track(), "{} expired", R::KIND) + } err => { tracing::warn!(broadcast = %crate::path::Rooted(self.shared.origin.root(), &self.broadcast), track = %self.track(), %err, "{} error", R::KIND) } @@ -1868,9 +1879,13 @@ impl RequestServe { // Drop the reply mid-flight: anything half-sent is cancelled rather than // completed for nobody. A request that never started moves its name into the log // context, as `start` would. - if let RequestState::Resolve { mut msg, .. } = - std::mem::replace(&mut self.state, RequestState::Finish { finished: false }) - { + if let RequestState::Resolve { mut msg, .. } = std::mem::replace( + &mut self.state, + RequestState::Finish { + finished: false, + stale: None, + }, + ) { self.track = R::take_track(&mut msg); } } @@ -1915,9 +1930,13 @@ impl RequestServe { } let broadcast = ready!(requesting.as_ref().expect("requesting").poll_ok(waiter))?; // Any placeholder but `Decode`, so a failed start is answered on the stream. - let RequestState::Resolve { mut msg, .. } = - std::mem::replace(&mut self.state, RequestState::Finish { finished: false }) - else { + let RequestState::Resolve { mut msg, .. } = std::mem::replace( + &mut self.state, + RequestState::Finish { + finished: false, + stale: None, + }, + ) else { unreachable!() }; self.track = R::take_track(&mut msg); @@ -1935,9 +1954,13 @@ impl RequestServe { stream.writer.poll_settle(waiter); } if ready!(step)?.is_break() { - let RequestState::Serve(request) = - std::mem::replace(&mut self.state, RequestState::Finish { finished: false }) - else { + let RequestState::Serve(request) = std::mem::replace( + &mut self.state, + RequestState::Finish { + finished: false, + stale: None, + }, + ) else { unreachable!() }; if R::HOLDS { @@ -1946,6 +1969,11 @@ impl RequestServe { finished: false, acked: false, }; + } else { + self.state = RequestState::Finish { + finished: false, + stale: request.stale(), + }; } } } @@ -1955,7 +1983,10 @@ impl RequestServe { // leaving before it is out cancels it and lets go of the request. if stream.writer.poll_flush(waiter)?.is_pending() { let closed = ready!(self.poll_closed(waiter)); - self.state = RequestState::Finish { finished: false }; + self.state = RequestState::Finish { + finished: false, + stale: None, + }; closed?; return Poll::Ready(Err(Error::Cancel)); } @@ -1970,7 +2001,10 @@ impl RequestServe { } let closed = ready!(self.poll_closed(waiter)); // Let go of the request now, not when the serve is dropped. - self.state = RequestState::Finish { finished: true }; + self.state = RequestState::Finish { + finished: true, + stale: None, + }; // A FIN or a reset both end the requester's interest; only a stray byte // is a fault. if let Err(err) = closed @@ -1982,9 +2016,14 @@ impl RequestServe { return Poll::Ready(Ok(())); } } - RequestState::Finish { finished } => { + RequestState::Finish { finished, .. } => { + // Those bytes hold the connection until acknowledged, so a reply gone stale + // while it waits is reset rather than left to stall the streams behind it. if !*finished { - ready!(stream.writer.poll_flush(waiter))?; + let Poll::Ready(flushed) = stream.writer.poll_flush(waiter) else { + return self.poll_stale(waiter); + }; + flushed?; stream.writer.finish()?; *finished = true; } @@ -1995,7 +2034,10 @@ impl RequestServe { ready!(self.poll_closed(waiter))?; } let stream = self.stream.as_mut().expect("stream present"); - return stream.writer.poll_close(waiter); + return match stream.writer.poll_close(waiter) { + Poll::Ready(closed) => Poll::Ready(closed), + Poll::Pending => self.poll_stale(waiter), + }; } } } @@ -2016,6 +2058,17 @@ impl RequestServe { Poll::Ready(res) } + /// Ready with `Old` once a finishing reply's group goes stale while its bytes wait. + fn poll_stale(&mut self, waiter: &kio::Waiter) -> Poll> { + let RequestState::Finish { stale: Some(group), .. } = &mut self.state else { + return Poll::Pending; + }; + match group.poll_expired(waiter) { + true => Poll::Ready(Err(Error::Old)), + false => Poll::Pending, + } + } + /// Ready once the requester closes its send direction, `Ok` for a FIN. Updates until then /// still reach the reply, each charged to the budget. fn poll_closed(&mut self, waiter: &kio::Waiter) -> Poll> { @@ -3097,6 +3150,13 @@ impl Request for FetchServe { match update {} } + fn stale(self) -> Option { + match self { + Self::Fetch { .. } => None, + Self::Serve { group, .. } => Some(group), + } + } + fn poll( &mut self, shared: &Shared, @@ -3155,12 +3215,20 @@ impl Request for FetchServe { batch_pos, records, } => loop { - ready!(writer.poll_flush(waiter))?; + // Parked on the transport, the group's budget still applies, as it does to + // a subscription's Group Stream: flow control must not pin a stale reply. + let Poll::Ready(flushed) = writer.poll_flush(waiter) else { + return poll_given_up(group, waiter).map_or(Poll::Pending, |err| Poll::Ready(Err(err))); + }; + flushed?; if let Some(records) = records { ready!(records.poll_state(writer, group, waiter))?; } if let Some(pending) = chunk { - ready!(writer.poll_write(waiter, pending))?; + let Poll::Ready(written) = writer.poll_write(waiter, pending) else { + return poll_given_up(group, waiter).map_or(Poll::Pending, |err| Poll::Ready(Err(err))); + }; + written?; if !bytes::Buf::has_remaining(pending) { *chunk = None; } @@ -5945,37 +6013,6 @@ fn poll_given_up(group: &mut group::Consumer, waiter: &kio::Waiter) -> Option Option { - if !version.has_group_end() { - return poll_given_up(group, waiter); - } - if *ending == Ending::Group && group.poll_expired(waiter) { - *ending = Ending::Expired; - } - // The cursor of an expired group fails its own end too, which closing replaces. - if *ending == Ending::Expired { - return None; - } - match group.poll_finished(waiter) { - Poll::Ready(Err(err)) => Some(err), - _ => None, - } -} - -/// Whether a lite-07 read that failed on `group` found it expired, which closes its stream -/// at this frame boundary rather than resetting it. -fn closes_expired(version: Version, group: &group::Consumer) -> bool { - version.has_group_end() && group.is_expired() -} - /// Buffer GROUP_END once a lite-07 stream without the `ietf` extension's object records read /// `group` to the end of a finished group: a `Payload Length` of 0, then the group's /// exclusive end in its track's timescale. Returns whether it did, so the FIN follows it; @@ -6046,7 +6083,7 @@ enum GroupState { ending: Ending, }, /// Every frame is written and the FIN sent: wait for the acknowledgement so a late - /// cancel can still reset the stream, and before lite-07 a late expiry too. + /// cancel or expiry can still reset the stream. Closed { writer: Writer, }, @@ -6059,9 +6096,6 @@ enum Ending { /// With the group: a FIN once its frames run out, after a GROUP_END on lite-07 if it /// finished whole. Group, - /// Lite-07: the group expired, so the stream FINs at the next frame boundary without a - /// GROUP_END. The subscriber keeps what arrived, and no drop names the group. - Expired, /// Lite-07: the GROUP_END is buffered, so the FIN follows once it flushes. Whole, } @@ -6264,7 +6298,7 @@ impl GroupServe { // flow control must not pin a stream that has gone stale, or one // whose unread rest the cache evicted. Poll::Pending => { - if let Some(err) = poll_stopped(&mut self.group, self.ctx.version, ending, waiter) { + if let Some(err) = poll_given_up(&mut self.group, waiter) { break 'serve Err(err); } return Poll::Pending; @@ -6290,9 +6324,7 @@ impl GroupServe { // flow control must not pin a stream that has gone stale, or one // whose unread rest the cache evicted. Poll::Pending => { - if let Some(err) = - poll_stopped(&mut self.group, self.ctx.version, ending, waiter) - { + if let Some(err) = poll_given_up(&mut self.group, waiter) { break 'serve Err(err); } return Poll::Pending; @@ -6305,9 +6337,8 @@ impl GroupServe { Poll::Ready(Err(err)) => break 'serve Err(err), Poll::Pending => return Poll::Pending, } - } else if *ending != Ending::Group { - // At a frame boundary with everything flushed: after the GROUP_END, - // or with the expired group's frames cut off here. + } else if *ending == Ending::Whole { + // Everything flushed after the GROUP_END. break 'serve Ok(()); } else if *batch_pos < batch.len() { let index = self.group.index() - batch.len() as u64 + *batch_pos as u64; @@ -6373,10 +6404,6 @@ impl GroupServe { } } }, - Poll::Ready(Err(_)) if closes_expired(self.ctx.version, &self.group) => { - *ending = Ending::Expired; - break 'serve Ok(()); - } Poll::Ready(Err(err)) => break 'serve Err(err), Poll::Pending => {} } @@ -6434,10 +6461,6 @@ impl GroupServe { } } }, - Poll::Ready(Err(_)) if closes_expired(self.ctx.version, &self.group) => { - *ending = Ending::Expired; - break 'serve Ok(()); - } Poll::Ready(Err(err)) => break 'serve Err(err), Poll::Pending => return Poll::Pending, } @@ -6445,16 +6468,12 @@ impl GroupServe { } }; - let GroupState::Serve { writer, ending, .. } = std::mem::replace(&mut self.state, GroupState::Done) + let GroupState::Serve { mut writer, .. } = std::mem::replace(&mut self.state, GroupState::Done) else { unreachable!() }; match outcome { Ok(()) => { - if ending == Ending::Expired { - tracing::debug!(key = self.key, "closing expired group"); - } - let mut writer = writer; // The buffer drained before the final frame resolved, so the // FIN follows the last byte. match writer.finish() { @@ -6476,11 +6495,11 @@ impl GroupServe { // discard bytes still retransmitting. let res = match writer.poll_close(waiter) { Poll::Ready(res) => res, - // Those bytes still hold the connection until acknowledged, so before - // lite-07 a group gone stale meanwhile releases them like one still - // serving. Lite-07 already said how the group ended, which a reset - // would take back. - Poll::Pending if !self.ctx.version.has_group_end() && self.group.poll_expired(waiter) => { + // Those bytes still hold the connection until acknowledged, so a group + // gone stale meanwhile releases them like one still serving, even after a + // lite-07 GROUP_END: stale bytes would fill the send window and stall the + // streams behind them. + Poll::Pending if self.group.poll_expired(waiter) => { let GroupState::Closed { writer } = std::mem::replace(&mut self.state, GroupState::Done) else { unreachable!() @@ -6887,10 +6906,10 @@ mod serve_group_test { } /// A lite-07 group expiring while its stream is open, here behind a blocked transport - /// write, closes the stream with what it sent rather than resetting it, sends none of the - /// frames written since, and needs no drop. + /// write, resets the stream with OLD, sends none of the frames written since, and fails so + /// its subscription names it in a stale drop. #[kio::test] - async fn lite07_closes_a_group_that_expires_on_its_stream() { + async fn lite07_resets_a_group_that_expires_on_its_stream() { let gate = kio::Producer::new(false); let session = SinkSession::gated_uni(gate.consume()); let log = session.log.clone(); @@ -6918,19 +6937,20 @@ mod serve_group_test { edge.write_frame(Timestamp::from_millis(1000).unwrap(), b"edge".as_slice()) .unwrap(); edge.finish().unwrap(); - assert!(futures::poll!(serve.as_mut()).is_pending(), "the frame is still owed"); // The group goes on, but its stream sends no more of it. old.write_frame(Timestamp::from_millis(10).unwrap(), b"more".as_slice()) .unwrap(); - *gate.write().ok().unwrap() = true; - serve.await.unwrap(); - assert!(log.resets().is_empty(), "reset"); + assert!(matches!(serve.await, Err(Error::Old))); + assert_eq!(log.resets(), [StreamError::Old.to_code()]); let (bytes, fin) = first_stream(&log); - assert!(fin, "no FIN"); - // Only the GROUP: the write blocked before the first frame, and none goes out after. - assert_eq!(bytes, [0, 3, 0, 0, 0]); - assert!(outcomes.dropped.lock().unwrap().is_empty(), "dropped"); + assert!(!fin, "FIN"); + // Nothing past the GROUP header the transport took before blocking. + assert!(!bytes.windows(4).any(|w| w == b"more"), "{bytes:?}"); + assert!( + outcomes.dropped.lock().unwrap().is_empty(), + "serve_group settles nothing" + ); } /// Egress counts only payload the transport took: a group reset while its frames wait in @@ -8649,11 +8669,10 @@ mod serve_group_test { } } - /// A group whose stream wrote its FIN but goes stale before the acknowledgement is never - /// served again, though the subscriber may have read the first stream whole. Before lite-07 - /// the stream resets with OLD and a stale drop names it, which is final even when a later - /// update widens the budget; lite-07 leaves a stream whose end it wrote alone, and needs no - /// drop. + /// A group whose stream wrote its FIN, and on lite-07 its GROUP_END, but goes stale before + /// the acknowledgement resets with OLD, so its bytes stop holding the send window, and a + /// stale drop names it. It is never served again, though the subscriber may have read the + /// first stream whole: the drop is final even when a later update widens the budget. #[kio::test] async fn a_stale_unacknowledged_fin_is_never_served_again() { for version in [Version::Lite06, Version::Lite07] { @@ -8691,12 +8710,9 @@ mod serve_group_test { write_group(&mut track, 1, 1000); write_group(&mut track, 2, 6000); while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} - let (dropped, resets): (&[(u64, u64)], &[u32]) = match version.has_group_end() { - true => (&[], &[]), - false => (&[(0, 0)], &[StreamError::Old.to_code()]), - }; + let dropped: &[(u64, u64)] = &[(0, 0)]; assert_eq!(drops(&mut writer, &log, version).await, dropped, "{version}"); - assert_eq!(session.resets(), resets, "{version}"); + assert_eq!(session.resets(), [StreamError::Old.to_code()], "{version}"); let opened = run.ctx.outcomes.opened.load(Ordering::Relaxed); run.update(update(Duration::from_secs(10))); @@ -11830,9 +11846,17 @@ mod tests { } /// A FETCH whose own budget gives up on its group partway resets as too old: a FIN - /// would tell the requester the prefix it got is the whole group. + /// would tell the requester the prefix it got is the whole group. One sent whole whose + /// FIN the requester has yet to acknowledge resets too, so stale bytes stop holding the + /// connection. #[kio::test] async fn an_expired_fetch_resets_rather_than_finishing() { + for whole in [false, true] { + expired_fetch_resets(whole).await; + } + } + + async fn expired_fetch_resets(whole: bool) { const VERSION: Version = Version::Lite07; let origin = crate::origin::Config::new(std::num::NonZeroU64::new(1).unwrap()).produce(); let broadcast = origin.route("room", crate::origin::RouteOptions::hidden()).unwrap(); @@ -11843,6 +11867,9 @@ mod tests { broadcast.announce().unwrap(); let mut open = track.append_next().unwrap(); open.write_frame(open.key_time(), b"a".to_vec()).unwrap(); + if whole { + open.finish().unwrap(); + } let peer_setup = crate::lite::PeerSetup::new(false, crate::transport::Address::Setup); peer_setup.set(crate::lite::Setup::default()).unwrap(); @@ -11875,7 +11902,10 @@ mod tests { } .encode(&mut crate::coding::Encoder::new(&mut script, VERSION.into()), VERSION) .unwrap(); - let session = ScriptedSession::new(script); + let session = match whole { + true => ScriptedSession::new(script).with_unacked_fin(), + false => ScriptedSession::new(script), + }; let (send, recv) = kio::wait(|waiter| { ::poll_open_bi(&session, waiter, Default::default()) }) @@ -11886,7 +11916,8 @@ mod tests { reader: crate::coding::Reader::new(crate::transport::boxed::RecvStream::new(recv), VERSION), }; let mut serve = RequestServe::::new(publisher.shared.clone(), stream); - assert!(!drive(&mut serve).await, "the open group holds the response"); + assert!(!drive(&mut serve).await, "whole={whole}: the response is held"); + assert!(session.log.resets().is_empty(), "whole={whole}"); // The track runs a full second past the group's successor. for ms in [1000, 2000] { @@ -11896,8 +11927,12 @@ mod tests { .unwrap(); group.finish().unwrap(); } - assert!(drive(&mut serve).await, "the budget ends the response"); - assert_eq!(session.log.resets(), vec![crate::StreamError::Old.to_code()]); + assert!(drive(&mut serve).await, "whole={whole}: the budget ends the response"); + assert_eq!( + session.log.resets(), + vec![crate::StreamError::Old.to_code()], + "whole={whole}" + ); } #[kio::test] diff --git a/rs/moq-net/src/lite/test_transport.rs b/rs/moq-net/src/lite/test_transport.rs index a3f895d2b63..c14bb7bd7bf 100644 --- a/rs/moq-net/src/lite/test_transport.rs +++ b/rs/moq-net/src/lite/test_transport.rs @@ -1037,6 +1037,8 @@ pub struct ScriptedSession { groups: bool, /// Set by [`Self::with_send_window`]. send_window: Option, + /// Set by [`Self::with_uni_send_window`]. + uni_send_window: Option, } impl ScriptedSession { @@ -1077,6 +1079,7 @@ impl ScriptedSession { unacked_unis: Default::default(), groups: false, send_window: None, + uni_send_window: None, } } @@ -1087,6 +1090,12 @@ impl ScriptedSession { self } + /// The same for each opened unidirectional stream. + pub fn with_uni_send_window(mut self, bytes: usize) -> Self { + self.uni_send_window = Some(bytes); + self + } + /// Open send groups, as native QUIC does, recording them in the log. pub fn with_groups(mut self) -> Self { self.groups = true; @@ -1312,7 +1321,14 @@ impl poll::Session for ScriptedSession { Poll::Ready(Err(_)) | Poll::Pending => return Poll::Pending, } } - let mut send = SinkSend::new(self.log.clone()); + let mut send = match self.uni_send_window { + // The gate's producer is gone, so the writes past the prefix park forever. + Some(window) => SinkSend { + prefix: window, + ..SinkSend::gated(self.log.clone(), kio::Producer::new(false).consume()) + }, + None => SinkSend::new(self.log.clone()), + }; send.unacked_fin = self .unacked_unis .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |count| count.checked_sub(1)) diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 77d0e21910b..a687d057f78 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -296,9 +296,8 @@ impl Version { } /// Whether GROUP carries the previous group's `Previous Start`, and a Group or Fetch Stream - /// ends a finished group with GROUP_END and a closed one with a bare FIN. A publisher then - /// closes an expired group's stream rather than resetting it. Added in lite-07; earlier - /// versions end every group with a bare FIN. + /// ends a finished group with GROUP_END and a closed one with a bare FIN. Added in lite-07; + /// earlier versions end every group with a bare FIN. pub(crate) fn has_group_end(self) -> bool { !matches!( self, diff --git a/rs/moq-net/src/model/group.rs b/rs/moq-net/src/model/group.rs index 585ecf92979..89486fe71b9 100644 --- a/rs/moq-net/src/model/group.rs +++ b/rs/moq-net/src/model/group.rs @@ -4113,13 +4113,6 @@ impl Consumer { self.poll_expiry(waiter, true) } - /// Whether the subscription this group was handed out under gave up on it as stale, as - /// [`Self::poll_expired`] or a read failing with [`Error::Old`] found. - #[cfg(feature = "async")] - pub(crate) fn is_expired(&self) -> bool { - self.expired - } - /// Apply the drift budget to a read that found nothing and is about to park, failing /// it with [`Error::Old`] once the budget gives up. /// diff --git a/rs/moq-net/tests/integration/partial_group.rs b/rs/moq-net/tests/integration/partial_group.rs index 38cdd31ce81..1b7c7c9b6a5 100644 --- a/rs/moq-net/tests/integration/partial_group.rs +++ b/rs/moq-net/tests/integration/partial_group.rs @@ -2,8 +2,7 @@ //! A group the subscription's max delay gives up on before the publisher finishes it never //! reaches the next hop as finished, so a relay does not cache the frames it already has as -//! the whole group. Before lite-07 the served stream resets with `Old`; lite-07 closes it -//! without a GROUP_END, which the next hop keeps as a closed group with no end. +//! the whole group: the served stream resets with `Old`. use crate::support; use moq_net::track::Delay; @@ -37,7 +36,6 @@ type Ended = (Result, moq_net::Error>, Option); /// return how a reader without a budget of its own ends group 0 at the downstream hop: /// `None` when that hop holds no copy of it. async fn round(version: &str) -> Option { - let closes = version == "moq-lite-07-wip"; let publisher = produce_origin(1); let downstream = produce_origin(2); let _pair = peer(version.parse::().unwrap(), &publisher, &downstream).await; @@ -90,14 +88,7 @@ async fn round(version: &str) -> Option { let tail = kio::clock::timeout(TIMEOUT, group.read_frame()) .await .expect("the stalled group never gave up"); - match closes { - true => assert!( - matches!(tail, Ok(None)) && group.end().is_none(), - "{version}: the live reader saw {tail:?} ending at {:?}", - group.end() - ), - false => assert!(tail.is_err(), "{version}: the live reader saw a clean end: {tail:?}"), - } + assert!(tail.is_err(), "{version}: the live reader saw a clean end: {tail:?}"); kio::clock::sleep(Duration::from_millis(100)).await; // A reader with no budget to apply sees only what the downstream hop cached. diff --git a/rs/moq-net/verify/subscribe/SubscribeAccounting.tla b/rs/moq-net/verify/subscribe/SubscribeAccounting.tla index a9adc40fc2a..ac02ad41c3e 100644 --- a/rs/moq-net/verify/subscribe/SubscribeAccounting.tla +++ b/rs/moq-net/verify/subscribe/SubscribeAccounting.tla @@ -13,12 +13,13 @@ (* order on the Subscribe Stream; each Group Stream travels on its own, *) (* unordered against them, and a reset can lose its header. *) (* *) -(* A lite-07 Group Stream whose group expires stale is closed, not reset: *) -(* FIN without GROUP_END, so the subscriber keeps what arrived as a closed *) -(* group, and no drop names it. A stream that never opened is still given *) -(* up with a drop, and a stream whose FIN went out is never reset for it. *) -(* A stream reset while serving, evicted or preempted with TOO_FAR_BEHIND *) -(* for a group that outranks it, is named in a final drop. *) +(* A Group Stream whose group expires stale resets with OLD and is named *) +(* in a stale drop, whether it never opened, is serving, or already sent *) +(* its FIN (after a GROUP_END) and awaits the acknowledgement: stale bytes *) +(* must not hold the send window. A subscriber that read it whole first *) +(* keeps the group. A stream reset while serving, evicted or preempted *) +(* with TOO_FAR_BEHIND for a group that outranks it, is named in a final *) +(* drop. *) (* *) (* Each GROUP carries the previous group's start, passing over the *) (* skipped starts directly below it as a SUBSCRIBE_SKIP would. The *) @@ -51,10 +52,12 @@ CONSTANTS \* A Group Stream: the publisher's side (`GroupState`), and the subscriber's. \* pub: "open" (serving), "fin" (FIN sent, awaiting its acknowledgement), \* "acked", or "reset" with `err` ("old" or "evicted") -\* sub: "none" (no header yet, or lost with a reset), "never" (never opened, so -\* nothing reached the wire), "header" (filling the copy), "fin", or "reset" +\* fin: the FIN went out before any reset, so every byte may arrive first +\* sub: "none" (no header yet, or lost with a reset, or never sent), "header" +\* (filling the copy), "fin", or "reset" \* passes: the skipped groups its GROUP's Previous Start passes over -NewStream(g, passes) == [g |-> g, pub |-> "open", err |-> "", sub |-> "none", passes |-> passes] +NewStream(g, passes) == + [g |-> g, pub |-> "open", fin |-> FALSE, err |-> "", sub |-> "none", passes |-> passes] \* A drop names one group; a skip names a run of them. DropMsg(g, err) == [type |-> "drop", gs |-> {g}, err |-> err] @@ -169,23 +172,23 @@ CursorGiveUp(g, kind, hold) == [] OTHER -> Named(nskip, held[g]) /\ UNCHANGED <> -\* A serving stream FINs (`GroupState::Closed`): after its last frame, or at a frame -\* boundary once its group expires stale (`Ending::Expired`), without a GROUP_END, so what -\* it wrote stands as a closed group. Nothing names the group either way. +\* A serving stream FINs after its last frame (`GroupState::Closed`), after a GROUP_END +\* if its group finished. Nothing names the group unless the stream resets later. StreamFinAt(i) == /\ streams[i].pub = "open" - /\ streams' = [streams EXCEPT ![i].pub = "fin"] + /\ streams' = [streams EXCEPT ![i].pub = "fin", ![i].fin = TRUE] /\ UNCHANGED <> -\* A stream fails and resets: it expires stale before it opened, waiting for stream credit -\* (`give_way`, `shed`), so nothing of it reached the wire; or it is reset while serving, -\* evicted or preempted with TOO_FAR_BEHIND for a group that outranks it, both final -\* ("evicted"). `settle_failed` queues the outcome, with what its GROUP passed over. +\* A stream fails and resets. It expires stale ("old", `poll_given_up`): before it opened, +\* waiting for stream credit (`give_way`, `shed`), so nothing of it reached the wire; while +\* serving; or after its FIN, before the acknowledgement (`GroupState::Closed`). Or it is +\* reset while serving, evicted or preempted with TOO_FAR_BEHIND for a group that outranks +\* it, both final ("evicted"). `settle_failed` queues the outcome, with what its GROUP +\* passed over. StreamFailAt(i, err) == - /\ streams[i].pub = "open" - /\ err = "old" => streams[i].sub = "none" - /\ streams' = [streams EXCEPT ![i].pub = "reset", ![i].err = err, - ![i].sub = IF err = "old" THEN "never" ELSE @] + /\ \/ streams[i].pub = "open" + \/ streams[i].pub = "fin" /\ err = "old" + /\ streams' = [streams EXCEPT ![i].pub = "reset", ![i].err = err] /\ outq' = Append(outq, <>) /\ UNCHANGED <> @@ -246,11 +249,12 @@ SubHeaderAt(i) == /\ rec' = SkipRec(unstale, passes) /\ UNCHANGED <> -\* The stream filling the copy is read to its FIN: the group is whole after a GROUP_END, -\* or closed where an expired one stopped, which the copy keeps alike ("whole"). +\* The stream filling the copy is read to its FIN, the group whole after a GROUP_END or +\* closed without one, which the copy keeps alike ("whole"). A reset sent after the FIN +\* may arrive too late to cut it. SubFinAt(i) == /\ streams[i].sub = "header" - /\ streams[i].pub \in {"fin", "acked"} + /\ streams[i].fin /\ streams' = [streams EXCEPT ![i].sub = "fin"] /\ cached' = [cached EXCEPT ![streams[i].g] = "whole"] /\ UNCHANGED <> @@ -304,10 +308,10 @@ SubRecv == ----------------------------------------------------------------------------- \* A stream is done once both sides stopped: acknowledged, or reset with the header -\* read to the reset, lost, or never sent. +\* read to the reset or to the FIN before it, lost, or never sent. StreamDone(s) == \/ s.pub = "acked" - \/ s.pub = "reset" /\ s.sub \in {"none", "never", "reset"} + \/ s.pub = "reset" /\ s.sub \in {"none", "reset", "fin"} \* Nothing in flight and nothing left for the publisher to do. Quiet ==