diff --git a/doc/bin/cli.md b/doc/bin/cli.md index cb76a414c12..c64dfd088f7 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -311,11 +311,11 @@ moq --connect https://relay.example.com/anon \ ## Redundant publishers -Two publishers that share a Hop ID (`--hop 42`) are treated as -interchangeable sources: relays hold both routes and fail over at a group -boundary. They must produce identical tracks with aligned groups. Everywhere -else leave `--hop` unset: a fresh id per run is what makes a restarted -encoder take over cleanly instead of splicing mid-stream. +Two publishers of the same broadcast name are interchangeable sources: +relays hold both routes and fail over between them mid-group. They must +produce identical tracks with aligned groups. A restarted encoder is the same +broadcast too, so one whose groups restart from 0 must publish under a new +name, or viewers wait for its sequence to catch up. ## Cluster diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index dfdc203ba25..635f7bbfcb8 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -29,23 +29,31 @@ has usually removed the other stale routes too, and the relay sends one retraction instead of advertising each stale path in turn. Requests still follow the current best route immediately; only the announcement waits. +A path names one broadcast, whoever publishes it. When the route serving a +broadcast dies, withdraws, or is beaten by a cheaper route, each subscription +continues on the new route from the first frame its readers lack, so they see +every frame once, mid-group included. A route that is still up finishes the +groups it has open, overlapping the new one. A group neither route delivers is +dropped once the readers' max age has passed it. A route through the subscribing peer +itself is never used. A publisher whose groups restart, such as an encoder +restarting from group 0, must publish under a new broadcast name; resumed under +the old one, readers wait for its sequence to catch up. + Failover routes must carry copies of the same broadcast. For each track, the relay requires matching timescale, retention window, publisher priority, and -group ordering. A source with different properties is refused before its groups -are spliced in. If no compatible source remains, the track fails with +group ordering. A source with different properties is refused before it serves +the track. If no compatible source remains, the track fails with `Unsupported`. New immutable properties require a new track name or broadcast -identity. +name. A route whose original publisher (its first hop) changes is updated in place on -both wire protocols, so the broadcast never briefly vanishes downstream. -Subscriptions already in flight keep draining the old publisher until it ends -and are never spliced onto the new one. New requests resolve through the updated -route as a fresh broadcast, without the old publisher's track properties. +both wire protocols, so the broadcast never briefly vanishes downstream, and +subscriptions in flight carry on through it. A publisher whose protocol names no hop (moq-transport without the cluster extension, moq-lite 01 through 03, or a peer that sends 0) gets a random first hop from the relay it connects to, fresh for each connection, followed by a 0. -Its reconnect is therefore a new publisher downstream, a reprice on the same +Its reconnect is therefore a new first hop downstream, a reprice on the same connection stays in place, and the 0 keeps it ranked as anonymous. ## Topology diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md index a69c9f75951..78c5a6c6e6e 100644 --- a/doc/concept/moq-lite.md +++ b/doc/concept/moq-lite.md @@ -87,9 +87,11 @@ always a prefix, on every wire version and on moq-transport alike; a service that serves only some of the paths beneath its prefix refuses the rest as they are requested. Each route carries the chain of relay identities it passed through, which is how forwarding loops are caught, and a cost, which is how a -subscriber picks among several routes to the same broadcast. A hop of 0 is the -anonymous mark and travels the chain unchanged; when it is the first hop, a relay -puts a random ID, fresh per connection, in front of it to name the publisher. A route that passed through an +subscriber picks among several routes to the same broadcast. A path names one +broadcast whoever publishes it, so a subscription moves between routes without +a seam; a publisher must not reuse a path for different content. A hop of 0 is +the anonymous mark and travels the chain unchanged; when it is the first hop, a +relay puts a random ID, fresh per connection, in front of it. A route that passed through an anonymous hop at any depth ranks below every fully identified route, whatever the costs say; among anonymous routes, cost keeps ordering. @@ -110,8 +112,7 @@ without that declaration fails the subscription with `ProtocolViolation`; older moq-lite versions use FIN alone. moq-transport requires `PUBLISH_DONE` before FIN. moq-transport sessions behave the same when a namespace is withdrawn. A route update that changes its first hop, the original publisher, is not a retraction: -subscriptions in flight drain the old publisher, and new requests resolve -through the new one. +subscriptions in flight carry on through it. A graceful session close withdraws its announcements and waits up to one second for transport acknowledgement before disconnecting. Rust uses @@ -291,12 +292,21 @@ anything on its own. Both ends apply it: the publisher skips a group rather than sending it, and the subscriber skips it again as it reads, since the publisher only ever sees the most tolerant budget across its subscribers. -Across a native route failover, the reader still judges buffered groups against -the logical track's live edge, including groups it is draining from a retired -route. A successor group with no timestamp leaves the preceding group's reach -unbounded until its first frame arrives; if it is dropped first, the next group -takes its place. A cached open group's prefix remains -readable across repeated takeovers and idle resumes. +A route failover is invisible to max age: a group open across the change carries +on from the new route at the frame where the old one stopped, and a group only a +replaced route that went quiet still holds is given up once it falls a full +budget behind the new route's live edge. A successor group with no timestamp +leaves the preceding group's reach unbounded until its first frame arrives; if +it is dropped first, the next group takes its place. + +A relay cancels its upstream subscription once nobody subscribes, but keeps its +copy of the track for 30 seconds after the last reader leaves, so a returning +reader or the next fetch finds its cache. That copy is not live meanwhile: +readers get nothing from its cache until the source answers again, since how +stale it is cannot be told. lite-07 and moq-transport answer with their largest +position, and older lite versions are asked from the head of the newest cached +group, so their first frame says where their feed is. A feed that has moved on +past everything cached leaves the cache below it to fetches. The publisher may declare a retention window per track. Omission sets no limit; zero keeps only the live edge. The origin cache ceiling and cache pool may still diff --git a/doc/concept/use-case/contribution.md b/doc/concept/use-case/contribution.md index 67931c160d0..4b45fdc9f9a 100644 --- a/doc/concept/use-case/contribution.md +++ b/doc/concept/use-case/contribution.md @@ -27,8 +27,8 @@ captions on. Because tracks are only pulled where they're needed, a publisher can open several connections that might be used. Primary and secondary ingest is two connections and no business logic: subscriptions ride the primary until it -fails, then move. Two encoders sharing a Hop ID become interchangeable -sources that relays fail over between at a group boundary; see +fails, then move. Two encoders publishing the same broadcast name are +interchangeable sources that relays fail over between mid-group; see [redundant publishers](/bin/cli#redundant-publishers). ## One protocol both ways diff --git a/drafts/draft-lcurley-moq-cluster.md b/drafts/draft-lcurley-moq-cluster.md index 97d0760c1e4..f9ffbf4dd8c 100644 --- a/drafts/draft-lcurley-moq-cluster.md +++ b/drafts/draft-lcurley-moq-cluster.md @@ -25,7 +25,7 @@ informative: This document defines a clustering extension for MoQ Transport {{moqt}}, used to build a mesh of relays. Each namespace advertisement carries the list of Hop IDs it has passed through, starting with the original publisher, and the accumulated cost of that path. -A receiver uses the list to detect loops and to tell which advertisements come from the same publisher, and the cost to choose between paths. +A receiver uses the list to detect loops, and the cost to choose between paths. Each endpoint declares its own Hop ID at setup, so a peer never advertises or serves it a path that already passed through it. --- note_Note_to_Readers @@ -99,8 +99,7 @@ A **Hop ID** is a variable-length integer naming one endpoint in a path. Hop IDs SHOULD be unique among the endpoints an advertisement can traverse. An endpoint MAY pick one at random, since collisions in a 64-bit space are unlikely, or use a configured identifier that survives restarts. -Loops and origins are detected by comparing Hop IDs for equality, so two endpoints sharing one are indistinguishable. -Redundant publishers of interchangeable content MAY share one deliberately, so the mesh treats their paths as failover options for the same content ({{selection}}). +Loops are detected by comparing Hop IDs for equality, so two endpoints sharing one are indistinguishable. ## The Reserved Hop ID 0 {#zero} **0 means "no identity"** and is reserved. @@ -109,11 +108,11 @@ It stands for an endpoint that did not negotiate this extension, and an endpoint Since any number of endpoints can be 0, it identifies nothing: - **Loop detection**: 0 in a HOP_PATH is never a loop. A receiver whose own Hop ID is 0 cannot detect loops through itself and MUST NOT discard an advertisement merely because the path contains 0. -- **Origin identity**: an advertisement whose first entry is 0 has an unknown publisher, which a relay names by putting a stamp of its own in front ({{stamping}}). A receiver MUST NOT treat two unstamped ones as interchangeable ({{selection}}). +- **Unknown publisher**: an advertisement whose first entry is 0 has an unknown publisher, which a relay names by putting a stamp of its own in front ({{stamping}}). - **Filtering**: a peer that declared 0 gave the receiver nothing to filter that session on. The receiver MAY assign an ID of its own ({{assigned}}) as local selection state and MUST NOT write it into HOP_PATH. Duplicate *non-zero* Hop IDs in one HOP_PATH are a loop; duplicate zeros are not. -Declaring 0 trades loop detection and failover for anonymity, except against a receiver that assigns an identity of its own. +Declaring 0 trades loop detection for anonymity, except against a receiver that assigns an identity of its own. ## Assigned Identities {#assigned} A receiver MAY assign a Hop ID to a peer that declared none, whether by declaring 0 or by not negotiating the extension. @@ -133,7 +132,7 @@ An anonymous accepted session cannot be correlated with anything, so it SHOULD g A relay that records an advertisement whose first HOP_PATH entry is 0 MUST insert a random non-zero Hop ID in front of it, picked once for the session the advertisement arrived on. One that arrived with no HOP_PATH ({{bridging}}) becomes that stamp followed by 0. -Nothing on the wire says whether an unnamed publisher that reconnects is the same one, so a fresh stamp per session makes its reconnect a change of publisher downstream ({{updating}}), while an update on the same session keeps the stamp and applies in place. +A fresh stamp per session keeps unnamed publishers on different sessions apart, while an update on the same session keeps the stamp and applies in place. The 0 behind the stamp keeps the path ranked below fully identified ones ({{selection}}), since the unnamed hop may hide any depth. A stamp names a session, not a peer: it MUST NOT be reused across sessions or taken as the peer's identity. @@ -235,10 +234,7 @@ A receiver MUST NOT treat the repeat as a duplicate or a protocol violation. An advertisement lives as long as its stream, so an update on a new stream would leave two streams claiming one namespace. An endpoint MUST NOT open a second stream for an advertisement it already maintains on the session. -An update replaces the old parameters atomically, so a receiver MUST NOT tear down subscriptions or drop cached state because one arrived. -If the first HOP_PATH entry is unchanged the content is continuous and subscriptions MAY resume on the new route at a group boundary. -If the first entry changed, the publisher changed, and the endpoint still sends an ordinary update. -The receiver keeps each subscription it is already serving on the old source until that source ends, MUST NOT resume or splice it onto the updated route, and serves new requests from the updated route without state cached from the old publisher. +An update replaces the old parameters atomically, so a receiver MUST NOT tear down subscriptions or drop cached state because one arrived, even when the first HOP_PATH entry changed: the namespace still names the same content ({{publishers}}). The expected update is a ROUTE_COST change, which is how a relay signals that it started or stopped carrying the namespace. @@ -258,8 +254,7 @@ A receiver SHOULD NOT cache refusals. A relay MUST NOT advertise a namespace merely because it resolved it: the covering advertisement stays the only one until the publisher advertises the concrete namespace, which it SHOULD do once producing, so a later request finds the running content by its exact namespace instead of resolving a second producer. -Two advertisements whose HOP_PATH begins with the same non-zero Hop ID come from the same publisher and carry interchangeable content: a receiver MAY hold them as redundant paths and fail an active subscription over to the survivor at a group boundary. -If the first entries differ, or either is 0, they are distinct publishers reusing a namespace ({{publishers}}). +Every advertisement of a namespace carries the same content ({{publishers}}): a receiver MAY hold several as redundant paths and fail an active subscription over to a survivor, continuing from the first object the subscriber lacks. An endpoint MUST NOT advertise a path whose HOP_PATH contains the Hop ID the peer declared: the peer could only discard it, and acting on it would form a loop. Of the paths that remain it SHOULD advertise the best, and advertises nothing when every path contains that Hop ID. @@ -274,13 +269,10 @@ One rule for advertisement and dispatch keeps advertised paths truthful and prev {{moqt}} lets several publishers advertise one namespace and leaves to the relay how it serves a SUBSCRIBE among them. Under this extension an advertisement is a path, so a session advertises a namespace at most once, a relay forwards only the best path it knows ({{selection}}), and a subscription is served from one source at a time. -A receiver MAY still hold paths to several publishers of one namespace and choose between them as it sees fit: serve from the cheapest and move to the next when it fails. +A namespace names one broadcast whichever publisher serves it, so several publishers of it are redundant copies: their groups are one sequence, and a subscription moves between them without a gap. +A publisher MUST NOT reuse a namespace for different content, and publishes a new instance under a new namespace instead. A refusal moves to another publisher only as {{selection}} allows: once, and only for NO_CAPACITY. -A relay that moves to another publisher MUST update its advertisement to the new path ({{updating}}), so the first Hop ID downstream names the publisher that new subscriptions reach. -Moving between distinct publishers is a discontinuity: their groups are not one sequence, so a subscriber sees an unrelated Location, and a FETCH that succeeds against one may fail against the other. - -Redundant publishers of the same content avoid this by sharing a Hop ID ({{hop-ids}}), which makes their paths interchangeable and lets a subscription fail over at a group boundary. -Publishers that do not share one are treated as reusing a name. +A relay that moves to another publisher MUST update its advertisement to the new path ({{updating}}). # Security Considerations {#security} @@ -337,9 +329,10 @@ This document requests one registration in the "REQUEST_ERROR Codes" registry. # Appendix A: Changelog ## moq-cluster-02 +- A namespace names one broadcast whichever publisher serves it, and a publisher MUST NOT reuse one for different content. A receiver MAY fail a subscription over between any of its advertisements, continuing from the first object the subscriber lacks instead of at a group boundary. Replaces the first-Hop-ID publisher identity. - Defined request resolution against the longest covering prefix and the NO_CAPACITY refusal with its single re-resolution; any other refusal is terminal, including between several publishers of one namespace. - A relay does not advertise a namespace because it resolved it; the publisher advertises the concrete namespace once producing. -- A change of original publisher is an ordinary update instead of a withdrawal and a new advertisement. Subscriptions already served drain the old source; new requests take the updated route. +- A change of original publisher is an ordinary update instead of a withdrawal and a new advertisement. - A relay puts a random Hop ID, picked per session, in front of an advertisement whose first HOP_PATH entry is 0, and writes that stamp followed by 0 for one with no HOP_PATH. ## moq-cluster-01 diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 7a018ff70a5..94e41fab3fd 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -116,8 +116,9 @@ Each client establishes a session with a CDN edge server, ideally the closest on Any broadcasts and subscriptions are transparently proxied by the CDN behind the scenes. ## Broadcast -A Broadcast is a collection of Tracks from a single publisher. +A Broadcast is a collection of Tracks named by a path. This corresponds to a MoqTransport's "track namespace". +A path names one Broadcast whichever publisher serves it: a publisher MUST NOT reuse a path for different content, and publishes a new instance under a new path instead. A publisher advertises what it can serve via ANNOUNCE_START messages, each carrying a path prefix: a route covering every broadcast path beneath it. A route is the only shape an advertisement takes: a publisher that serves only some of the paths beneath a prefix, such as a transcoder for any broadcast's derivative, advertises the covering prefix and refuses the requests it will not serve (see [Resolution](#resolution)); no message narrows a route. @@ -424,10 +425,7 @@ Applying one rule to both advertisement and dispatch keeps advertised paths trut When resolving a path covered by several routes (across any number of streams), the subscriber SHOULD prefer the most specific covering route (see [Resolution](#resolution)), then a path that contains no 0 Hop ID over one that does, then the lowest Warm Route Cost after adding each arriving link's cost (see [Cost Parameter](#cost-parameter)), breaking ties toward the lowest Cold Route Cost, then toward the shortest path, and then toward the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. -A route's identity is its first hop: the endpoint that originated it (see [ANNOUNCE_START](#announce-start)). -Two routes covering one path with the same non-zero first hop are the same origin reached different ways, and a relay MAY move a live subscription between them, resuming at a group boundary, so a route change the identity survives (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. -Across differing first hops, or where either is 0, the routes promise nothing about each other's content: a relay MUST NOT splice a live subscription across them, and when the serving session ends, in-flight subscriptions end with it (a reset) and the subscriber re-requests through the best remaining route. -Equal first hops promise the same origin, not interchangeable bytes; what a resuming relay serves next is whatever that origin publishes next at the group boundary. +Every route covering a path serves the same Broadcast, so a relay MAY move a live subscription between them, continuing from the first frame the subscriber lacks, and a route change (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. #### Resolution {#resolution} A SUBSCRIBE, FETCH, or TRACK request names a path, and the receiver resolves it against the routes covering that path, after the per-subscriber exclusion above. @@ -461,6 +459,7 @@ A subscriber opens Subscribe Streams to request a Track. The subscriber MUST start a Subscribe Stream with a SUBSCRIBE message followed by any number of SUBSCRIBE_UPDATE messages. The publisher replies with a SUBSCRIBE_OK message once the start group is resolved, followed by a SUBSCRIBE_END message once the subscription ends. For a live track the publisher MAY withhold SUBSCRIBE_OK until the first matching group resolves the start; if the track has already ended with no matching groups, it sends SUBSCRIBE_END with no preceding SUBSCRIBE_OK. +A requested start past the publisher's largest position is the exception: the publisher MUST send SUBSCRIBE_OK without waiting, with `Group` set to the requested start group, so a subscriber resuming just past what it holds learns where the live feed is from the answer alone (a quiet track may not reach the start for a long time). A rejection is a stream reset: a publisher that cannot serve the subscription (no such track, an ended broadcast, or any other refusal) MUST promptly reset the stream rather than leave it pending, so a subscriber distinguishes "pending" from "refused" by the reset, not by a timeout. A route claims capability rather than inventory, so a subscription for a covered path that names nothing is refused this way too. @@ -1140,13 +1139,15 @@ Common values include `1000` (milliseconds), `1000000` (microseconds), `48000` ( A SUBSCRIBE_OK message confirms a subscription and resolves its absolute start position. It is the first message the publisher sends on the Subscribe Stream, once the start position is known. -This is the trimmed-down counterpart of MoqTransport's SUBSCRIBE_OK: it retains the name and the role of the publisher's positive response, but carries only the resolved start position (all other per-track properties live in [TRACK_INFO](#track-info)). +This is the trimmed-down counterpart of MoqTransport's SUBSCRIBE_OK: it retains the name and the role of the publisher's positive response, but carries only the resolved start position and the publisher's largest position (all other per-track properties live in [TRACK_INFO](#track-info)). ~~~ SUBSCRIBE_OK Message { Type (i) = 0x0 Message Length (i) Group (i) + Largest Group (i) + [Largest Frame (i)] } ~~~ @@ -1167,6 +1168,11 @@ The subscriber derives the start frame from `Group` and its own request: The second case is easy to get wrong, so to be explicit: a subscriber that requested group 5 frame 15 and receives `Group` = 6 starts at **frame 0** of group 6, not frame 15. The frame offset belonged to group 5 and is gone along with the rest of it; it does not carry forward to whichever group the publisher resolved to. +**Largest Group** and **Largest Frame**: +The largest position the publisher has for the track when it answers: the last frame that exists, like moq-transport's Largest Location. +`Largest Group` is the group sequence plus one, and 0 means the track has nothing yet, in which case `Largest Frame` is absent. +A subscriber takes it as where the live feed is: a relay holding groups from an earlier subscription serves them from cache only once this says they are current, which a publisher answering promptly lets it do without waiting for a frame. + ## SUBSCRIBE_END {#subscribe-end} A SUBSCRIBE_END message is sent by the publisher to signal that no group at or after a given sequence will be produced. @@ -1351,6 +1357,8 @@ The `Message Length` describes the payload size on the wire. ## moq-lite-07 - Made TRACK_INFO Publisher Max Age optional, encoded as milliseconds plus one with zero meaning no limit. +- Added `Largest Group` and `Largest Frame` to SUBSCRIBE_OK: the publisher's largest position when it answers, which a subscriber takes as where the live feed is. A publisher MUST answer at once when the requested start is past it. Earlier versions carry no such position, so a subscriber takes the first frame instead. +- A path names one Broadcast whichever publisher serves it, and a publisher MUST NOT reuse a path for different content. A relay MAY move a subscription between any routes covering the path, continuing from the first frame the subscriber lacks instead of at a group boundary. Replaces the first-hop identity. - Assigned `moq-lite-07-wip` as this draft's protocol identifier until it is finalized as `moq-lite-07`. - Switched every variable-length integer, including SETUP parameter values, from QUIC's two-bit length prefix to moq-transport's leading-ones encoding, widening the range to 64 bits. diff --git a/js/net/src/ietf/subscriber.test.ts b/js/net/src/ietf/subscriber.test.ts index bdaf6b07d0b..e29c5d06ff6 100644 --- a/js/net/src/ietf/subscriber.test.ts +++ b/js/net/src/ietf/subscriber.test.ts @@ -653,8 +653,8 @@ test("a PUBLISH_NAMESPACE repricing is acknowledged in place", async () => { /** * An update whose first Hop ID differs names a different publisher. It still updates the - * advertisement in place and the stream stays open. A broadcast already held keeps - * draining, while the next consume starts fresh. + * advertisement in place, the stream stays open, and the path names the same broadcast, so + * the next consume shares it. */ test("a PUBLISH_NAMESPACE update that changes the publisher applies in place", async () => { const pair = createMockTransportPair(ALPN.DRAFT_19); @@ -694,8 +694,7 @@ test("a PUBLISH_NAMESPACE update that changes the publisher applies in place", a route: { hops: [HopSchema.parse(8n), PEER] }, }); - const fresh = subscriber.consume(Path.from("theirs")); - expect(fresh.closed).not.toBe(held.closed); + expect(subscriber.consume(Path.from("theirs")).closed).toBe(held.closed); expect(held.closed.peek()).toBeUndefined(); // Still announced: the stream ending is what retracts it. diff --git a/js/net/src/ietf/subscriber.ts b/js/net/src/ietf/subscriber.ts index 03dd535166e..6a93857d289 100644 --- a/js/net/src/ietf/subscriber.ts +++ b/js/net/src/ietf/subscriber.ts @@ -259,15 +259,12 @@ export class Subscriber { /** * Replace the stored route for a path that is already announced. A no-op when the * hops and cost did not change; otherwise consumers hear `update` so a forwarder - * can reprice without retracting. - * - * A new first hop is a new publisher: holders keep their broadcast to drain, but the - * next consume starts fresh rather than reusing the old publisher's cached track info. + * can reprice without retracting. The path still names the same broadcast, whatever + * the first hop now says, so the shared consume stays. */ #updateAnnounce(path: Path.Valid, route: Route) { const existing = this.#announced.get(path); if (existing === undefined || routesEqual(existing.route, route)) return; - if (existing.route.hops[0] !== route.hops[0]) this.#consumes.evict(path); existing.route = route; console.debug(`announced: broadcast=${path} rerouted`); for (const [consumer, filter] of this.#announcedConsumers) { diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index fa47bd12e32..81492ed6a91 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1642,3 +1642,43 @@ test("lite draft-05: a group that goes stale while its stream opens writes nothi origin.close(); } }); + +test.each([0, 1])("lite draft-07 reports the cached largest position when starting at group %s", async (startGroup) => { + const version = Version.DRAFT_07; + const pair = createMockTransportPair(ALPN_07_WIP); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, version, randomHop(), origin.consume()); + const broadcast = publish(origin, Path.from("quiet")); + const track = broadcast.createTrack("video"); + const group = new GroupProducer(0); + group.writeString("cached"); + group.close(); + track.writeGroup(group); + const client = await Stream.open(pair.client, { version }); + const server = await Stream.accept(pair.server, version); + if (!server) throw new Error("missing subscribe stream"); + const running = publisher.runSubscribe( + replaySubscribe({ + id: 0n, + broadcast: Path.from("quiet"), + track: "video", + priority: 0, + startGroup, + }), + server, + ); + try { + const response = await decodeSubscribeResponse(client.reader, version); + if (!("start" in response)) throw new Error("expected SUBSCRIBE_OK"); + expect(response.start.group).toBe(startGroup); + expect(response.start.largest).toEqual({ group: 0, frame: 0 }); + } finally { + client.close(); + publisher.close(); + broadcast.close(); + origin.close(); + pair.client.close(); + pair.server.close(); + await running; + } +}); diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 2b4bb7a93e6..80341512147 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -32,6 +32,7 @@ import { hasAnnounceId, hasAnnounceOk, hasDatagrams, + hasLargest, hasProbeRtt, hasRouteCost, hasStreamCount, @@ -818,11 +819,37 @@ export class Publisher { // Exactly-once arrival-order serving. This synchronous package-internal pop // and frameRange call are the operation's linearization point. + // Popping or filtering a group removes the subscriber's view of its edge. + const largest = !startSent && hasLargest(this.version) ? track.largest() : undefined; const recv = hooks.tryRecvGroup(track); switch (recv.kind) { case "error": throw recv.error; case "idle": + // A start past everything the track has (a subscriber resuming just after + // what it holds) is answered at once with the largest position, on + // versions that carry it: a quiet track may not reach that start for a + // while, and the subscriber judges what it holds against the answer. + if (emitRange && !startSent && hasLargest(this.version) && bounds.startGroup !== undefined) { + const startFrame = bounds.startFrame; + if ( + largest !== undefined && + (bounds.startGroup > largest.group || + (bounds.startGroup === largest.group && startFrame > largest.frame)) + ) { + startSent = true; + hooks.replaceGroups(track, { + start: { included: bounds.startGroup }, + end: bounds.endGroup === undefined ? undefined : { included: bounds.endGroup }, + }); + const start = new SubscribeStart(bounds.startGroup, largest); + if ( + !(await controls.response(encodeSubscribeResponse(stream, { start }, this.version))) + ) + return; + continue; + } + } // Before lite-07, an end declared ahead of the live edge goes out as // soon as it is known, while the remaining groups are still being // produced. The lite-07 count is not final until those groups open. @@ -874,7 +901,7 @@ export class Publisher { !(await controls.response( encodeSubscribeResponse( stream, - { start: new SubscribeStart(group.sequence) }, + { start: new SubscribeStart(group.sequence, largest) }, this.version, ), )) diff --git a/js/net/src/lite/subscribe.test.ts b/js/net/src/lite/subscribe.test.ts index 4da1af09185..79c6737f7a6 100644 --- a/js/net/src/lite/subscribe.test.ts +++ b/js/net/src/lite/subscribe.test.ts @@ -153,6 +153,22 @@ test("SubscribeStart round-trips on draft-05", async () => { expect(got.start.group).toBe(42); }); +test("SubscribeStart carries the largest position on draft-07", async () => { + // Type, length, group, largest group + 1, largest frame. + expect(await encode(Version.DRAFT_07, { start: new SubscribeStart(4, { group: 3, frame: 2 }) })).toEqual( + new Uint8Array([0, 3, 4, 4, 2]), + ); + for (const largest of [undefined, { group: 3, frame: 2 }]) { + const got = await responseRoundtrip(Version.DRAFT_07, { start: new SubscribeStart(4, largest) }); + if (!("start" in got)) throw new Error("expected start"); + expect([got.start.group, got.start.largest]).toEqual([4, largest]); + } + // Draft-06 has no largest position on the wire. + expect(await encode(Version.DRAFT_06, { start: new SubscribeStart(4, { group: 3, frame: 2 }) })).toEqual( + new Uint8Array([0, 1, 4]), + ); +}); + test("SubscribeEnd round-trips on draft-05", async () => { // Type, length, group: no stream count before draft-07. expect(await encode(Version.DRAFT_05, { end: new SubscribeEnd(7, 3) })).toEqual(new Uint8Array([1, 1, 7])); diff --git a/js/net/src/lite/subscribe.ts b/js/net/src/lite/subscribe.ts index 3d1a1f2711f..fecb237f40a 100644 --- a/js/net/src/lite/subscribe.ts +++ b/js/net/src/lite/subscribe.ts @@ -1,7 +1,8 @@ import * as Path from "../path.ts"; import type { Reader, Writer } from "../stream.ts"; +import type { Location } from "../track.ts"; import * as Message from "./message.ts"; -import { hasFrameBounds, hasGroupOrder, hasStreamCount, resolvesStart, Version } from "./version.ts"; +import { hasFrameBounds, hasGroupOrder, hasLargest, hasStreamCount, resolvesStart, Version } from "./version.ts"; /** * Encode the `Group Start` field shared by SUBSCRIBE and SUBSCRIBE_UPDATE. @@ -433,18 +434,39 @@ export class SubscribeOk { export class SubscribeStart { group: number; - constructor(group: number) { + /** + * The publisher's largest (group, frame) when it answered, or `undefined` for a track + * with nothing yet. Draft-07+ only; not on the wire before, where it decodes as `undefined`. + */ + largest?: Location; + + constructor(group: number, largest?: Location) { this.group = group; + this.largest = largest; } - async encode(w: Writer): Promise { + async encode(w: Writer, version: Version): Promise { return Message.encode(w, async (w) => { await w.u53(this.group); + if (!hasLargest(version)) return; + // Group + 1, so 0 is a track with nothing yet; the frame follows only otherwise. + if (this.largest === undefined) { + await w.u53(0); + } else { + await w.u53(this.largest.group + 1); + await w.u53(this.largest.frame); + } }); } - static async decode(r: Reader): Promise { - return Message.decode(r, async (r) => new SubscribeStart(await r.u53())); + static async decode(r: Reader, version: Version): Promise { + return Message.decode(r, async (r) => { + const group = await r.u53(); + if (!hasLargest(version)) return new SubscribeStart(group); + const largest = await r.u53(); + if (largest === 0) return new SubscribeStart(group); + return new SubscribeStart(group, { group: largest - 1, frame: await r.u53() }); + }); } } @@ -557,7 +579,7 @@ export async function encodeSubscribeResponse(w: Writer, resp: SubscribeResponse // Draft-05+: SUBSCRIBE_OK is gone; START/END/DROP carry the resolved range. if ("start" in resp) { await w.u53(0x0); - await resp.start.encode(w); + await resp.start.encode(w, version); } else if ("end" in resp) { await w.u53(0x1); await resp.end.encode(w, version); @@ -592,7 +614,7 @@ export async function decodeSubscribeResponse(r: Reader, version: Version): Prom const typ = await r.u53(); switch (typ) { case 0x0: - return { start: await SubscribeStart.decode(r) }; + return { start: await SubscribeStart.decode(r, version) }; case 0x1: return { end: await SubscribeEnd.decode(r, version) }; case 0x2: diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 79565c0c7e1..888eb444721 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -283,21 +283,18 @@ test("a restart updates the route in place, even from another publisher", async route: { hops: [PUBLISHER_B, PEER] }, }); - // The holder keeps its broadcast to drain, but the next consume starts fresh rather - // than reusing the old publisher's cached track info. - const fresh = subscriber.consume(room); - expect(fresh.closed).not.toBe(held.closed); + // The path still names the same broadcast, so the next consume shares it. + expect(subscriber.consume(room).closed).toBe(held.closed); expect(held.closed.peek()).toBeUndefined(); - // A third publisher is another update. The one above has to leave its own publisher on - // record, or this one would not read as a change. + // A third publisher is another update, still the same broadcast. await send((w) => encodeAnnounceBroadcast(w, { status: "restart", id: 0n, hops: [PUBLISHER_C] }, Version.DRAFT_06)); expect(await announced.next()).toMatchObject({ prefix: room, kind: "update", route: { hops: [PUBLISHER_C, PEER] }, }); - expect(subscriber.consume(room).closed).not.toBe(fresh.closed); + expect(subscriber.consume(room).closed).toBe(held.closed); await send((w) => encodeAnnounceBroadcast(w, { status: "endedId", id: 0n }, Version.DRAFT_06)); expect(await nextRoute(announced)).toMatchObject({ prefix: room, kind: "end" }); @@ -366,7 +363,7 @@ test("a lite-05 duplicate announce follows the same restart rule", async () => { // A responder that withholds its Hop ID sends the reserved 0, and an empty chain means it // originated the path itself, so the advertisement names nobody and the connection stamps it. // A restart on the same connection keeps that stamp, so it is a reprice that updates in place -// and keeps the shared broadcast. Only a first hop that changes names a new publisher. +// and keeps the shared broadcast, as does one that names a publisher. test("a restart from an unidentified publisher updates in place", async () => { const { subscriber, send, settle } = announceHarness(Version.DRAFT_06); const announced = subscriber.announced(); @@ -388,10 +385,10 @@ test("a restart from an unidentified publisher updates in place", async () => { expect(await nextRoute(announced)).toMatchObject({ prefix: room, kind: "update" }); expect(subscriber.consume(room).closed).toBe(held.closed); - // Naming a publisher is a change of publisher: the next consume starts fresh. + // Naming a publisher changes the route, not the broadcast. await send((w) => encodeAnnounceBroadcast(w, { status: "restart", id: 0n, hops: [PUBLISHER_A] }, Version.DRAFT_06)); expect(await nextRoute(announced)).toMatchObject({ prefix: room, kind: "update" }); - expect(subscriber.consume(room).closed).not.toBe(held.closed); + expect(subscriber.consume(room).closed).toBe(held.closed); announced.close(); subscriber.close(); diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 5f7e0fb5dd3..87f4e26637a 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -249,16 +249,11 @@ export class Subscriber { // per path is current, and every announce on this stream shares `prefix`). // // An advertisement skipped locally as a reflected loop is recorded with - // `publisher: undefined` and `live: false`: the peer numbered it and will retract - // it regardless of what we made of it, so its path is not free. Dropping it from - // the map instead would let a later announce take the path, and the skipped one's - // `endedId` would then retract that one's state. - // - // `publisher` is what lets a restart tell a route change (same publisher) from a - // new publisher on the route, whose content the next consume must not share - // with the old one's. + // `live: false`: the peer numbered it and will retract it regardless of what we + // made of it, so its path is not free. Dropping it from the map instead would let + // a later announce take the path, and the skipped one's `endedId` would then + // retract that one's state. type Advertisement = { - publisher: Hop | undefined; live: boolean; route: Route; captures: Path.Pattern[] | undefined; @@ -275,7 +270,7 @@ export class Subscriber { // they go on record and obey the same one-per-path rule: the initial set // naming a path twice is the same violation as two ANNOUNCE_STARTs for it, // and the record is what catches either. Draft01/02 carry no hop ids and no - // ANNOUNCE_OK, so this connection's stamp names the publisher. + // ANNOUNCE_OK, so this connection's stamp is the first hop. for (const suffix of init.suffixes) { const path = Path.join(prefix, suffix); if (advertised.has(path)) { @@ -284,7 +279,7 @@ export class Subscriber { const route = { hops: [this.#stamp, UNKNOWN_HOP], cost: Cost.zero }; const live = visible(path); const captures = scopeCaptures(scope, path); - advertised.set(path, { publisher: this.#stamp, live, route, captures }); + advertised.set(path, { live, route, captures }); if (!live) continue; console.debug(`announced: broadcast=${path} active=true`); announced.append({ prefix: path, captures, kind: "start", route }); @@ -414,7 +409,6 @@ export class Subscriber { // resolves here. retract(); advertised.set(path, { - publisher: undefined, live: false, route: { hops: full, cost: Cost.zero }, captures: undefined, @@ -428,9 +422,9 @@ export class Subscriber { continue; } - // The first hop identifies the original publisher; an empty chain means the - // peer itself originated it. One that names nobody (lite-01..03, or a peer - // reporting 0) gets this connection's stamp in front of its 0. + // The first hop is the original publisher; an empty chain means the peer itself + // originated it. One that names nobody (lite-01..03, or a peer reporting 0) + // gets this connection's stamp in front of its 0. const fullHops = stampHops( hops !== undefined && responderOrigin !== undefined ? [...hops, responderOrigin] @@ -442,18 +436,16 @@ export class Subscriber { if (fullHops === undefined || fullHops.length > MAX_HOPS) { console.debug(`announced: broadcast=${path} dropped (hop chain at MAX_HOPS)`); advertised.set(path, { - publisher: undefined, live: false, route: { hops: [], cost: Cost.zero }, captures: undefined, }); continue; } - const publisher = fullHops[0]; const route: Route = { hops: fullHops, cost: cost ?? Cost.zero }; const captures = scopeCaptures(scope, path); if (!visible(path)) { - advertised.set(path, { publisher, live: false, route, captures }); + advertised.set(path, { live: false, route, captures }); continue; } @@ -462,11 +454,9 @@ export class Subscriber { // route in place, so a forwarder re-prices without retracting. const previous = advertised.get(path); if (previous?.live) { - // A different publisher took the path. Subscriptions already open drain - // the old copy, but the next consume starts fresh rather than reusing the - // old publisher's cached track info. - if (previous.publisher !== publisher) this.#consumes.evict(path); - advertised.set(path, { publisher, live: true, route, captures }); + // Even from another publisher: the path still names the same broadcast, so + // the shared consume stays. + advertised.set(path, { live: true, route, captures }); console.debug(`announced: broadcast=${path} rerouted`); if (!routesEqual(previous.route, route)) { announced.append({ prefix: path, captures, kind: "update", route }); @@ -474,7 +464,7 @@ export class Subscriber { continue; } - advertised.set(path, { publisher, live: true, route, captures }); + advertised.set(path, { live: true, route, captures }); console.debug(`announced: broadcast=${path} active=true`); announced.append({ prefix: path, captures, kind: "start", route }); diff --git a/js/net/src/lite/version.ts b/js/net/src/lite/version.ts index 2d83114b6ff..f9b1d3fd65a 100644 --- a/js/net/src/lite/version.ts +++ b/js/net/src/lite/version.ts @@ -299,3 +299,22 @@ const VERSION_NAMES: Record = { export function versionName(v: Version): string { return VERSION_NAMES[v] ?? `unknown(0x${v.toString(16)})`; } + +/** + * Whether SUBSCRIBE_START carries the publisher's largest (group, frame), which a subscriber + * takes as where the live feed is. Added in lite-07; an earlier answer says nothing about it. + */ +export function hasLargest(version: Version): boolean { + // Explicitly list older versions so future versions keep the lite-07+ behavior. + switch (version) { + case Version.DRAFT_01: + case Version.DRAFT_02: + case Version.DRAFT_03: + case Version.DRAFT_04: + case Version.DRAFT_05: + case Version.DRAFT_06: + return false; + default: + return true; + } +} diff --git a/quest/m1/README.md b/quest/m1/README.md index e87d02526eb..acfd40b83be 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -32,6 +32,7 @@ QUIC studies there on that rule. - [Delete removed cluster flags](/quest/m1/cluster-shims.md) - once a release has carried their refusals, `mesh` and `linger` leave `cluster::Config` - [Track tail interop](/quest/m1/track-tail-interop.md) - a Rust publisher ending a track with a group in flight is read to its end by the JS subscriber, and the reverse, in `just test interop` - [Demand everywhere](/quest/m1/demand-everywhere.md) - requests and producers at every level watch subscribers through `demand()` alone, ahead of the bindings exposing it +- [Request linger](/quest/m1/request-linger.md) - an upstream FETCH or SUBSCRIBE outlives its last reader by a short linger, so quick re-requests don't churn upstream - [Merge queue](/quest/m1/merge-queue.md) - the required checks run on `merge_group`, so a stale green check can no longer break main - [Binding audio delay](/quest/m1/binding-surface.md) - moq-ffi and every wrapper configure and observe audio playout delay - [FFI shape](/quest/m1/ffi-shape/README.md) - the bindings mirror Rust's layers: net at the root, then media, json, flate, audio, and video namespaces built from the handle below @@ -46,6 +47,7 @@ QUIC studies there on that rule. - [Browser close code](/quest/m1/browser-close-code.md) - a playwright case proves the page reads a relay's close code and reason, on every web-transport backend - [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - every stream group in a lite subscription arrives or is dropped by name, and lite-07 drops its stream count for it - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges in either order and replaces FETCH; relays fill misses by range, including over moq-transport +- [Live media time](/quest/m1/subscribe-live-time.md) - a lite-07 SUBSCRIBE_OK carries the publisher's current media time, so readers judge a cache against the live edge - [Cross-relay bursts](/quest/m1/cross-relay-bursts.md) - bursty small-group tracks cross two relays without lost groups, unanswered FETCHes, or stalls - [Late lower groups](/quest/m1/lite-late-lower-group.md) - a moq-lite subscriber with a floor receives a group created below the first served one, as moq-transport does - [JS cache window](/quest/m1/js-prune-cache-window.md) - js/net ages idle groups on a cache window and keeps max_age as media-time staleness, as Rust does diff --git a/quest/m1/request-linger.md b/quest/m1/request-linger.md new file mode 100644 index 00000000000..02c35615248 --- /dev/null +++ b/quest/m1/request-linger.md @@ -0,0 +1,30 @@ +# [M] Upstream requests linger past their demand + +## Goal + +An upstream FETCH or SUBSCRIBE outlives the last reader wanting it by a short +linger, so a request issued and abandoned in quick succession (a reader that +re-subscribes, seeks, or blips) does not churn upstream. A reader returning +within the linger rides the request already in flight. + +## Plan + +Decided in planning (2026-10-03, from [#4741](https://github.com/moq-dev/moq/pull/4741)): + +- **Both request types.** The session's upstream SUBSCRIBE, today canceled the + moment the last subscriber leaves, and FETCH, today cut short (all the way + upstream) once no fetcher waits and no reader holds its group. +- **Demand decides, per request type, unsplit.** A FETCH is canceled once its + demand stays unused through the linger. A subscription's groups ignore group + demand: a front's reader still gives up a stale group a fetcher happens to + hold, as a relay's subscription would. Group demand is not split into fetch + and subscription halves. +- **A short fixed window**, around a second: enough to absorb a re-subscribe or + a seek, far below the front's 30 s cache linger. A constant, measured during + the quest. +- Watches demand through the `demand()` handles + [Demand everywhere](/quest/m1/demand-everywhere.md) plumbs. + +## Required + +- [Demand everywhere](/quest/m1/demand-everywhere.md) - requests and producers at every level watch subscribers through `demand()` alone diff --git a/quest/m1/subscribe-live-time.md b/quest/m1/subscribe-live-time.md new file mode 100644 index 00000000000..3068133a146 --- /dev/null +++ b/quest/m1/subscribe-live-time.md @@ -0,0 +1,53 @@ +# [L] SUBSCRIBE_OK carries the live media time + +## Goal + +When a route answers a lite-07 subscription, the subscriber learns the +publisher's current media time, extrapolated to now rather than the newest +frame's timestamp, so every reader's budget judges a cache against the real +live edge, even across a gap. + +Today (since [#4741](https://github.com/moq-dev/moq/pull/4741)) a front caches +nothing: its readers read the serving route's copy directly, and a track nobody +reads drops that copy, so an idle cache is never served. lite-07 SUBSCRIBE_OK +carries the largest position from the publisher's cache, which nothing reads yet. +A budget judges a group by where its successor starts, so past a gap an old +group's reach stays open. Re-scope before starting: the idle-cache case that +motivated this is gone. + +## Plan + +Decided in planning (2026-10-03): + +- **Estimator: least-delayed reference.** Each track keeps + `offset = min(wall − pts)` over a recent sliding window of the frames written + to it (the least-delayed frame sets the reference, the window follows clock + drift), and reports `now − offset` in the track's timescale. Reason: the + newest frame's timestamp is stale on a sparse track and carries that frame's + delivery jitter. +- **Lives in moq-net track state**, fed by every frame a `track::Producer` + writes, so origin publishers and relays share one implementation and any + SUBSCRIBE_OK reads it. js/net gets the same in its track state. +- **Each hop estimates on its own.** A relay reports its own estimate from the + `(pts, arrival)` pairs it received, never the upstream's value, so the delay + every hop adds shows up honestly. +- **Optional on the wire.** An untimed track (see + [Untimed lite-07](/quest/m1/lite-untimed.md)) has no media time, so the field + is absent, encoded the same way as an absent frame timestamp. A track whose + frames are all untimed never reports one, and no receiver invents one. +- **lite-07 only.** It is still WIP (`moq-lite-07-wip`), so the field is added + without negotiation. Older versions keep the gap rule. +- **API: `track::Subscriber::live().await`** resolves once the route answered, to + `Live { start, latest: Option, time: Option }`: the resolved + start, the largest group/frame, and the live media time. Mirrored in js/net. +- **Independent of [Subscribe ranges](/quest/m1/subscribe-ranges/README.md)**, + which rewrites the same messages: whichever lands second rebases. +- `js/watch/src/sync.ts` stays as it is: its latency range (from #1620) is + intended, and only the publisher's estimate uses the least-delayed reference. +- No new docs page: the lite draft and the max-age paragraph in + `doc/concept/moq-lite.md` are updated inline. + +## Related + +- [Untimed model](/quest/m1/untimed-model.md) - absent timestamps end to end; only real pairs feed the estimate +- [Shared clock](/quest/m2/shared-clock.md) - the hang catalog's `{wall, timescale}` anchor sits a layer up; this stays media-agnostic diff --git a/rs/moq-hls/src/export/mod.rs b/rs/moq-hls/src/export/mod.rs index f37ed43892f..a5773ce0d02 100644 --- a/rs/moq-hls/src/export/mod.rs +++ b/rs/moq-hls/src/export/mod.rs @@ -1701,11 +1701,10 @@ mod tests { (rendition, watcher) } - // A same-path republish takes the origin leaf over with a brand new broadcast (an ordinary - // publisher is `Hop::UNKNOWN`, which never counts as the same publisher, so even a plain - // reconnect qualifies). Renditions derived from the old broadcast's catalog must not serve the - // replacement's media: its group numbering restarts, so those bytes would be served under the - // replaced broadcast's segment number, duration, and PROGRAM-DATE-TIME. + // A path published again after its broadcast ended is a brand new broadcast. Renditions + // derived from the old broadcast's catalog must not serve the replacement's media: its group + // numbering restarts, so those bytes would be served under the replaced broadcast's segment + // number, duration, and PROGRAM-DATE-TIME. #[tokio::test] async fn a_replacement_publisher_is_not_served_under_the_replaced_catalog() { const OLD: &[u8] = b"OLDOLDOLDOLDOLDO"; @@ -1753,10 +1752,13 @@ mod tests { source, }; - // The publisher reconnects before the catalog snapshot is reconciled. That gap is wide in - // practice: `watch_catalog` retries a not-yet-written catalog track with backoff, so - // seconds can pass between resolving the broadcast and syncing its first snapshot. + // The publisher goes away and comes back before the catalog snapshot is reconciled. That + // gap is wide in practice: `watch_catalog` retries a not-yet-written catalog track with + // backoff, so seconds can pass between resolving the broadcast and syncing its first + // snapshot. The old broadcast ends first: a path still routed is one broadcast, so a + // publisher back before then would resume it instead. drop(old); + settle().await; let (new, _) = publish(&origin, NEW); settle().await; @@ -1865,9 +1867,10 @@ mod tests { .await .expect("the timeline reaches the rendition"); - // The media publisher reconnects before any segment is requested, taking the origin leaf - // over with a brand new broadcast whose group numbering restarts from zero. + // The media publisher goes away and comes back before any segment is requested, as a + // brand new broadcast whose group numbering restarts from zero. drop(old_media); + settle().await; let new_media = publish_media(&origin, NEW, None); settle().await; @@ -1925,8 +1928,8 @@ mod tests { moq_net::origin::Route::default().with_hops(hops(&[first])) } - /// A standalone media publisher (not an origin leaf) so the sibling is served through a - /// routed front whose first hop can change. + /// A standalone media publisher (not an origin leaf), so the sibling is served through a + /// routed front. fn write_routed_media( broadcast: &mut moq_net::broadcast::Producer, payload: &'static [u8], @@ -1971,12 +1974,12 @@ mod tests { } } - // A bound sibling whose publisher is replaced through a different first hop ends with - // Dropped. The exporter must drop rows listed for the old publisher, re-bind, and list - // only rows that arrive after the new bind resolves: fetching an old row from the + // A bound sibling whose route goes away ends with Dropped, and a new publisher may serve + // the path afterwards. The exporter must drop rows listed for the old publisher, re-bind, + // and list only rows that arrive after the new bind resolves: fetching an old row from the // replacement would serve restarted groups under the previous publisher's segment numbers. #[tokio::test] - async fn a_replaced_first_hop_sibling_drops_old_rows_and_rebinds() { + async fn a_replaced_sibling_drops_old_rows_and_rebinds() { tokio::time::pause(); const OLD: &[u8] = b"OLDOLDOLDOLDOLDO"; const NEW: &[u8] = b"NEWNEWNEWNEWNEWN"; @@ -2023,9 +2026,6 @@ mod tests { let mut early = rendition.segments(); assert_eq!(early.next().await.unwrap().unwrap().discontinuity, 0); - // The replacement is already announced before the incumbent is dropped, matching a - // rival publisher that appears while the current first hop is still serving. - let new_server = origin.dynamic("media", sibling_route(11)).unwrap(); drop((old_server, old_media, _old_track)); until_empty(&rendition).await; assert!( @@ -2033,6 +2033,12 @@ mod tests { "a row listed for the old publisher answers 404 after replacement" ); + // A publisher still routed when the incumbent goes would resume its broadcast + // instead: a path is one broadcast, whoever serves it. The bind failed while nothing + // served the path, so the next poll re-issues it. + let new_server = origin.dynamic("media", sibling_route(11)).unwrap(); + let _ = rendition.snapshot(); + let mut new_media = moq_net::broadcast::Info::new().produce(); accept_sibling(&new_server, &new_media).await; // The rendition skips timeline rows until this bind resolves. Joining the diff --git a/rs/moq-hls/src/export/upstream.rs b/rs/moq-hls/src/export/upstream.rs index 7b064de716e..e3e836e28cc 100644 --- a/rs/moq-hls/src/export/upstream.rs +++ b/rs/moq-hls/src/export/upstream.rs @@ -3,11 +3,11 @@ /// The one broadcast an export is bound to: the already-resolved catalog consumer, plus the /// [`moq_mux::Source`] it came from so a rendition can still follow a cross-broadcast reference. /// -/// The two travel together because resolving the catalog path by name is not idempotent. A -/// same-path republish is a takeover (`Hop::UNKNOWN` never counts as the same publisher, so -/// every ordinary publisher reconnect qualifies), and that installs a brand new broadcast at the -/// leaf. A rendition that looked its media up by path would then serve the replacement's groups -/// under the manifest, segment numbering, and `PROGRAM-DATE-TIME` of the broadcast it replaced. +/// The two travel together because resolving the catalog path by name is not idempotent. Once +/// a broadcast ends, its path can be published again as a brand new broadcast, such as a +/// publisher reconnecting after its old route is gone, with its group numbering restarted. A +/// rendition that looked its media up by path would then serve the replacement's groups under +/// the manifest, segment numbering, and `PROGRAM-DATE-TIME` of the broadcast it replaced. #[derive(Clone)] pub(crate) struct Upstream { /// Origin plus catalog path, for resolving a rendition's sibling `broadcast` reference. diff --git a/rs/moq-mux/src/container/consumer.rs b/rs/moq-mux/src/container/consumer.rs index 7f38cc050ed..9d99686ce9b 100644 --- a/rs/moq-mux/src/container/consumer.rs +++ b/rs/moq-mux/src/container/consumer.rs @@ -1644,7 +1644,7 @@ mod tests { } #[tokio::test(start_paused = true)] - async fn truncated_spliced_group_skips_to_the_next_clean_group() { + async fn truncated_resumed_group_skips_to_the_next_clean_group() { let origin = crate::source::produce_origin(); let hops = moq_net::Hops::try_from(vec![moq_net::Hop::new(10).unwrap()]).unwrap(); let first_route = origin @@ -1681,8 +1681,10 @@ mod tests { ) .unwrap(); - // Each cheaper route advances beyond the seam without serving its continuation. - // Enough takeovers prune every route that could still cover group 0. + // Each cheaper route takes over beyond group 0 without holding its continuation, + // while the first route stays up but silent. They carry the same broadcast, so each + // holds every group from 1 on. The open group is given up once the track runs a + // full budget past it. let mut routes = Vec::new(); let mut sources = Vec::new(); for sequence in 1..=4 { @@ -1701,15 +1703,17 @@ mod tests { if sequence == 1 { assert_eq!(consumer.read().await.unwrap().unwrap().timestamp, ts(0)); } - write_group(&mut track, sequence, &[ts(sequence * 100_000)]); + for sequence in 1..=sequence { + write_group(&mut track, sequence, &[ts(sequence * 1_000_000)]); + } routes.push(route); sources.push((source, track)); } let waiter = kio::Waiter::noop(); - // Group 1 was never handed out before its segment was pruned. An abort - // skips straight to the retained successor; a short clean group emits GroupEnd. + // The aborted group 0 skips straight to its successor; a short clean group emits + // GroupEnd. let event = consumer.poll_event(&waiter); let Poll::Ready(Ok(Some(Event::Frame(frame)))) = event else { panic!( @@ -1718,14 +1722,14 @@ mod tests { event.is_pending() ); }; - assert_eq!(frame.timestamp, ts(200_000)); + assert_eq!(frame.timestamp, ts(1_000_000)); assert!(frame.keyframe); - assert_eq!(consumer.current, 2); + assert_eq!(consumer.current, 1); assert!( matches!(consumer.poll_event(&waiter), Poll::Ready(Ok(Some(Event::GroupEnd)))), "the complete successor still emits its clean boundary" ); - assert_eq!(consumer.read().await.unwrap().unwrap().timestamp, ts(300_000)); + assert_eq!(consumer.read().await.unwrap().unwrap().timestamp, ts(2_000_000)); } // ---- Eviction recovery (pause/resume) ---- diff --git a/rs/moq-mux/src/container/ts/export_test.rs b/rs/moq-mux/src/container/ts/export_test.rs index 273ed39ecab..0bacdf53784 100644 --- a/rs/moq-mux/src/container/ts/export_test.rs +++ b/rs/moq-mux/src/container/ts/export_test.rs @@ -5975,6 +5975,10 @@ async fn resume_after(finish: bool) { before.extend(rest); assert_eq!(end.is_ok(), finish, "a finish ends cleanly and a drop fails: {end:?}"); + // The old broadcast ends before the publisher comes back: a path still routed is one + // broadcast, so a publisher back before then would resume it instead. + ended.closed().await; + // The returned catalog lists the track only in its second snapshot, and the restarted // publisher's clock starts over. let (mut broadcast, mut catalog) = publish(); @@ -6111,6 +6115,9 @@ async fn export_stats_skip_output_a_failure_discards() { assert_eq!(units(export.stats()), pes_count(&frames)); drop((broadcast, catalog, track)); + // The old broadcast ends before the publisher comes back: a path still routed is one + // broadcast, so a publisher back before then would resume it instead. + ended.closed().await; let (mut broadcast, mut catalog) = publish(); let mut track = aac_rendition(&mut broadcast, &mut catalog, "a.aac"); source.returned(&ended).await.unwrap(); diff --git a/rs/moq-net/benches/origin.rs b/rs/moq-net/benches/origin.rs index fde970c50df..a6cceec63b6 100644 --- a/rs/moq-net/benches/origin.rs +++ b/rs/moq-net/benches/origin.rs @@ -1,5 +1,6 @@ //! Fan-out benchmarks for the origin: its route table, the announce cursors -//! watching it, request resolution, and the handoff between local sources. +//! watching it, request resolution, the handoff between local sources, and the +//! delivery through a front. //! //! Every shape is swept over both axes, publishers (routes) and subscribers //! (cursors), so a cost that grows with the size of the table rather than with @@ -480,6 +481,115 @@ fn bench_handoff(c: &mut Criterion) { group.finish(); } +/// `(tracks, readers per track)` shapes for [`bench_relay`]. +const RELAY: [(usize, usize); 4] = [(1, 1), (1, 100), (100, 1), (10, 100)]; + +/// Frames per group in [`bench_relay`]. +const RELAY_FRAMES: usize = 10; + +/// Steady-state delivery through a front: one group of [`RELAY_FRAMES`] frames per +/// track, read in full by every reader of the broadcast the front serves. Readers +/// drain the serving route's shared group cache directly, so measure both the track +/// and reader axes. +fn bench_relay(c: &mut Criterion) { + let mut group = c.benchmark_group("origin/relay"); + for (tracks, readers) in RELAY { + group.throughput(Throughput::Elements((tracks * readers * RELAY_FRAMES) as u64)); + group.bench_function(BenchmarkId::from_parameter(format!("{tracks}t_{readers}r")), |b| { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let (producer, driver) = origin::Producer::new(origin::Config::default()); + runtime.spawn(moq_net::time::run(driver)); + let broadcast = producer.publish("room/live", origin::Route::default()).unwrap(); + let sources: Vec<_> = (0..tracks) + .map(|i| broadcast.create_track(format!("{i}"), None).unwrap()) + .collect(); + let mut subscriptions = runtime.block_on(async { + let resolved = producer.consume().request_broadcast("room/live").await.unwrap(); + let mut subscriptions = Vec::new(); + for i in 0..tracks { + let track = resolved.track(&format!("{i}")).unwrap(); + for _ in 0..readers { + subscriptions.push(track.subscribe(None).await.unwrap()); + } + } + subscriptions + }); + + b.iter_custom(|iterations| { + runtime.block_on(async { + let started = std::time::Instant::now(); + for _ in 0..iterations { + for source in &sources { + let mut group = source.append_group().unwrap(); + for _ in 0..RELAY_FRAMES { + group.write_frame(Timestamp::ZERO, b"frame".as_ref()).unwrap(); + } + group.finish().unwrap(); + } + for subscription in &mut subscriptions { + let mut group = subscription.recv_group().await.unwrap().expect("group"); + for _ in 0..RELAY_FRAMES { + group.read_frame().await.unwrap().expect("frame"); + } + } + } + started.elapsed() + }) + }); + }); + } + group.finish(); +} + +/// Re-poll stalled groups through a front, including their subscription budget. +/// Sweep tracks and readers separately to expose unrelated table scans. +fn bench_parked(c: &mut Criterion) { + let mut group = c.benchmark_group("origin/parked"); + for tracks in [1, 100] { + for readers in [1, 100] { + group.throughput(Throughput::Elements((tracks * readers) as u64)); + group.bench_function(BenchmarkId::from_parameter(format!("{tracks}t_{readers}r")), |b| { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let (producer, driver) = origin::Producer::new(origin::Config::default()); + runtime.spawn(moq_net::time::run(driver)); + let broadcast = producer.publish("room/live", origin::Route::default()).unwrap(); + let sources: Vec<_> = (0..tracks) + .map(|i| broadcast.create_track(format!("{i}"), None).unwrap()) + .collect(); + let _writers: Vec<_> = sources.iter().map(|source| source.append_group().unwrap()).collect(); + let (subscriptions, mut groups) = runtime.block_on(async { + let resolved = producer.consume().request_broadcast("room/live").await.unwrap(); + let mut subscriptions = Vec::new(); + let mut groups = Vec::new(); + for i in 0..tracks { + let track = resolved.track(&format!("{i}")).unwrap(); + for _ in 0..readers { + let mut sub = track.subscribe(None).await.unwrap(); + groups.push(sub.recv_group().await.unwrap().unwrap()); + subscriptions.push(sub); + } + } + (subscriptions, groups) + }); + let waiter = kio::Waiter::noop(); + b.iter(|| { + for group in &mut groups { + assert!(group.poll_read_frame(&waiter).is_pending()); + } + }); + drop(subscriptions); + }); + } + } + group.finish(); +} + criterion_group!( benches, bench_announce, @@ -491,6 +601,8 @@ criterion_group!( bench_serve_idle, bench_subscribe, bench_request, - bench_handoff + bench_handoff, + bench_relay, + bench_parked ); criterion_main!(benches); diff --git a/rs/moq-net/src/coding/reader.rs b/rs/moq-net/src/coding/reader.rs index 8b0605a9edc..7805e3cc4ec 100644 --- a/rs/moq-net/src/coding/reader.rs +++ b/rs/moq-net/src/coding/reader.rs @@ -213,6 +213,23 @@ impl Reader { Poll::Ready(Ok(self.buffer.split_to(size).freeze())) } + /// Poll to discard `remaining` bytes, counting it down as they arrive. Nothing is + /// buffered past what one read brings, so a peer cannot make a skipped payload cost + /// memory. + pub(crate) fn poll_skip(&mut self, cx: &mut Context<'_>, remaining: &mut usize) -> Poll> { + loop { + let skipped = self.buffer.len().min(*remaining); + self.buffer.advance(skipped); + *remaining -= skipped; + if *remaining == 0 { + return Poll::Ready(Ok(())); + } + if !ready!(self.poll_read_more(cx))? { + return Poll::Ready(Err(DecodeError::Short.into())); + } + } + } + /// Read exactly the given number of bytes from the stream. pub async fn read_exact(&mut self, size: usize) -> Result { std::future::poll_fn(|cx| self.poll_read_exact(cx, size)).await @@ -413,6 +430,25 @@ mod tests { } } + /// A skipped payload is discarded across reads, leaving what follows it, and a skip + /// waiting on more has counted down what it already discarded. + #[test] + fn skip_discards_across_reads() { + let chunks = [b"abc".as_slice(), b"defg", b"hi"]; + let mut reader = Reader::new(Chunks(chunks.into()), crate::lite::Version::Lite05); + let mut cx = Context::from_waker(std::task::Waker::noop()); + + let mut remaining = 6; + assert!(matches!(reader.poll_skip(&mut cx, &mut remaining), Poll::Ready(Ok(())))); + assert_eq!(&reader.read_exact(3).now_or_never().unwrap().unwrap()[..], b"ghi"); + + let mut reader = Reader::new(Chunks([b"ab".as_slice(), b"cd"].into()), crate::lite::Version::Lite05); + let mut remaining = 5; + assert!(reader.poll_skip(&mut cx, &mut remaining).is_pending()); + assert_eq!(remaining, 1); + assert!(reader.buffer.is_empty(), "nothing skipped is kept"); + } + fn refuses_frame + Debug, V: StreamCodes + Clone>(prefix: &[u8], version: V, oversized: bool) { let mut reader = Reader::new(Chunks([b"following bytes".as_slice()].into()), version); reader.buffer.extend_from_slice(prefix); diff --git a/rs/moq-net/src/error.rs b/rs/moq-net/src/error.rs index 148a857f646..eae8cb053b8 100644 --- a/rs/moq-net/src/error.rs +++ b/rs/moq-net/src/error.rs @@ -379,10 +379,10 @@ pub enum Error { Closed, /// The reader asked for a frame the group never held: below - /// [`crate::group::Producer::start_at`], or skipped past a splice. Named from the - /// consumer's side; distinct from [`Self::GroupTooLarge`], which aborts the whole - /// group when a write exceeds the cache budget, and from [`Self::Evicted`], which - /// drops a whole group under the pool's memory pressure. + /// [`crate::group::Producer::start_at`]. Named from the consumer's side; distinct + /// from [`Self::GroupTooLarge`], which aborts the whole group when a write exceeds + /// the cache budget, and from [`Self::Evicted`], which drops a whole group under the + /// pool's memory pressure. #[error("lagged")] Lagged, diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index 686a91aed1a..9f5acaa69ba 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -629,6 +629,10 @@ where } }; + // A relay's copy that went idle cannot say where its live edge is until its route + // answers again; answering from its cache would advertise a stale one. + kio::wait(|waiter| track.poll_live(waiter)).await; + // The filter and any fill are relative to the live edge, so snapshot it once: // the fill ends exactly where a Next Object subscription begins, which is what // lets the draft's current-group join (Next Object plus a StartGroup=1 fill) @@ -6225,8 +6229,8 @@ struct LiveEdge { /// The newest group sequence, `None` before any group exists. latest: Option, /// The precise Largest Object. `None` when the track is empty, or when the newest - /// group's frames cannot be read right now (none written yet, or a spliced track - /// between segments), in which case nothing is advertised and no fill is servable. + /// group's frames cannot be read right now (none written yet), in which case nothing + /// is advertised and no fill is servable. largest: Option, /// One past the Largest Object, which is where a Next Object subscription begins. /// When the edge is imprecise this falls back to the next group boundary: never below diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index be6cfc35d37..09544cde2ce 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -153,6 +153,11 @@ struct State { // Each broadcast created by a PUBLISH_NAMESPACE message. broadcasts: HashMap, + + // Copies nobody subscribes to, kept for the linger with no subscription upstream: + // a session ending ends them too. + lingering: HashMap, + next_lingering: u64, } impl State { @@ -172,6 +177,9 @@ impl State { let _ = producer.clone().abort(err.clone()); } } + for (_, producer) in self.lingering.drain() { + let _ = producer.abort_session(err.clone()); + } } fn close(&mut self) { for (_, mut track) in self.subscribes.drain() { @@ -182,6 +190,9 @@ impl State { let _ = producer.close(); } } + for (_, producer) in self.lingering.drain() { + let _ = producer.close(); + } } } @@ -369,6 +380,49 @@ struct Accepted { largest: Option, } +/// What a subscription is for: the origin's request, until the first SUBSCRIBE_OK +/// accepts it, then the copy it became, subscribed again after lingering. +enum Target { + Request(track::Request), + Resume(Idle), +} + +impl Target { + /// The subscribers' aggregate demand. + fn subscription(&self) -> Option { + match self { + Self::Request(request) => request.subscription(), + Self::Resume(idle) => idle.track.subscription(), + } + } + + fn producer(&self) -> Option<&track::Producer> { + match self { + Self::Request(_) => None, + Self::Resume(idle) => Some(&idle.track), + } + } + + /// The subscription could not be made: the track fails with `err`. + fn fail(self, err: Error) { + match self { + Self::Request(request) => request.reject(err), + Self::Resume(idle) => { + let _ = idle.track.abort(err); + } + } + } +} + +/// A copy with no subscription upstream, kept for fetches and a returning subscriber. +struct Idle { + track: track::Producer, + /// Serves its cache misses with a group FETCH, subscribed or not. + dynamic: track::Dynamic, + demand: track::Demand, + timescale: Option, +} + struct TrackState { producer: Option, /// The origin request, until SUBSCRIBE_OK accepts it. Abort rejects this: the @@ -400,6 +454,11 @@ struct TrackState { // A pre-draft-20 joining FETCH, which reuses the fill rendezvous. joining: Option, + // Where the subscription asked to start, when that is partway through a group: a + // resumed subscription picking up after the frames a previous route delivered. The + // stream for that group legitimately starts there, without a head. + resume: Option, + // The data streams read so far, which PUBLISH_DONE's Stream Count is checked against. tail: kio::Producer, } @@ -429,6 +488,7 @@ impl TrackState { largest: None, fetch_id: None, joining, + resume: None, tail: Default::default(), } } @@ -1680,9 +1740,41 @@ where return; } - let subscription = request.subscription(); + let track_name = request.name().to_owned(); + // Group FETCHes for cache misses: standalone, so they outlive each subscription. + let mut group_fetches = TaskSet::owned(); + let mut target = Target::Request(request); + loop { + let Some(idle) = self + .subscribe_once(&broadcast_path, &track_name, target, &mut group_fetches) + .await + else { + return; + }; + let Some(next) = self + .linger(&broadcast_path, &track_name, idle, &mut group_fetches) + .await + else { + return; + }; + target = Target::Resume(next); + } + } + + /// Subscribe upstream for `target`, until the subscription ends: `Some` once nobody + /// subscribes any more, with the copy kept for [`Self::linger`], `None` once the track + /// ended or failed. + async fn subscribe_once( + &mut self, + broadcast_path: &Path<'_>, + track_name: &str, + target: Target, + group_fetches: &mut TaskSet, + ) -> Option { + let subscription = target.subscription(); + let start = subscription.as_ref().and_then(|s| s.start); // A live join delivers nothing below the group SUBSCRIBE_OK names as Largest. - let live = subscription.as_ref().and_then(|s| s.start).is_none(); + let live = start.is_none(); let join = match subscribe_join( subscription.as_ref().and_then(|s| s.start), subscription.as_ref().and_then(|s| s.end), @@ -1690,16 +1782,16 @@ where ) { Ok(join) => join, Err(err) => { - request.reject(err); - return; + target.fail(err); + return None; } }; let request_id = match self.control.next_request_id(&self.runtime).await { Ok(id) => id, Err(err) => { - request.reject(err); - return; + target.fail(err); + return None; } }; @@ -1707,8 +1799,8 @@ where Ok(s) => s, Err(err) => { tracing::debug!(%err, "failed to open subscribe stream"); - request.reject(err); - return; + target.fail(err); + return None; } }; @@ -1723,35 +1815,48 @@ where let mut state = self.state.lock(); state.subscribes.insert( request_id, - TrackState::pending(request.name().to_owned(), broadcast_path.to_owned(), fill, joining), + TrackState { + resume: subscription + .as_ref() + .and_then(|s| s.start) + .filter(|start| start.frame != 0), + // A resumed copy is the session's to abort from here on. + producer: target.producer().cloned(), + ..TrackState::pending(track_name.to_owned(), broadcast_path.to_owned(), fill, joining) + }, ); } - // Write Subscribe message + // Write Subscribe message. The aggregate is read now: a subscriber can join while + // the request ID and stream were awaited, and nothing updates the priority after. + let priority = target.subscription().map(|s| s.priority).unwrap_or(0); if let Err(err) = self - .write_subscribe(&mut stream, request_id, &broadcast_path, &request, join) + .write_subscribe(&mut stream, request_id, broadcast_path, track_name, priority, join) .await { tracing::debug!(%err, "failed to write subscribe"); self.remove_subscribe(request_id); - request.reject(err); - return; + target.fail(err); + return None; } - tracing::info!(broadcast = %self.origin.absolute(&broadcast_path), track = %request.name(), "subscribe started"); + tracing::info!(broadcast = %self.origin.absolute(broadcast_path), track = %track_name, "subscribe started"); // Park the origin request where a session abort can reject it. The producer // does not exist until SUBSCRIBE_OK, and dropping this task would otherwise // end the track as `Dropped`. - let track_name = request.name().to_owned(); - { - let mut state = self.state.lock(); - let Some(held) = state.subscribes.get_mut(&request_id) else { - request.reject(Error::Cancel); - return; - }; - held.pending = Some(request); - } + let resumed = match target { + Target::Request(request) => { + let mut state = self.state.lock(); + let Some(held) = state.subscribes.get_mut(&request_id) else { + request.reject(Error::Cancel); + return None; + }; + held.pending = Some(request); + None + } + Target::Resume(idle) => Some(idle), + }; // A publisher can be serving before its SUBSCRIBE_OK reaches us, since the data // streams are independent of the request stream. Waiting for the response alone would @@ -1778,6 +1883,16 @@ where return Poll::Ready(Setup::Response(res)); } let mut state = self.state.lock(); + if let Some(idle) = &resumed { + // A session abort drained the subscription, and the copy with it. + if !state.subscribes.contains_key(&request_id) { + return Poll::Ready(Setup::Gone); + } + return match idle.demand.poll_unused(waiter) { + Poll::Ready(_) => Poll::Ready(Setup::Unused), + Poll::Pending => Poll::Pending, + }; + } let Some(pending) = state .subscribes .get_mut(&request_id) @@ -1795,6 +1910,8 @@ where match setup { Setup::Response(res) => break Some(res), Setup::Gone => break None, + // Nobody holds the resumed copy any more: back to lingering. + Setup::Unused if resumed.is_some() => break None, Setup::Unused => { let mut state = self.state.lock(); let Some(pending) = state @@ -1817,15 +1934,16 @@ where let Some(response) = setup else { tracing::info!( - broadcast = %self.origin.absolute(&broadcast_path), + broadcast = %self.origin.absolute(broadcast_path), track = %track_name, "subscribe abandoned before it was accepted" ); // The publisher may already be serving before it answers. A session abort // already rejected the parked request; dropping what remains is not a second one. - self.remove_subscribe(request_id); + let aborted = self.remove_subscribe(request_id).is_none(); self.cancel_subscribe(stream, request_id).await; - return; + // A resumed copy goes back to lingering, unless the session took it. + return resumed.filter(|_| !aborted); }; // SUBSCRIBE_OK commits the model's immutable metadata before the alias releases @@ -1837,15 +1955,21 @@ where pending.reject(Error::UnexpectedMessage); } self.remove_subscribe(request_id); - return; + if let Some(idle) = resumed { + let _ = idle.track.abort(Error::UnexpectedMessage); + } + return None; } Err(err) => { tracing::debug!(%err, "subscribe response error"); if let Some(pending) = self.take_pending(request_id) { - pending.reject(err); + pending.reject(err.clone()); } self.remove_subscribe(request_id); - return; + if let Some(idle) = resumed { + let _ = idle.track.abort(err); + } + return None; } }; let Accepted { @@ -1859,24 +1983,42 @@ where .with_timescale(Timescale::MICRO) .with_max_age(max_age) .with_priority(super::priority::from_wire(priority.unwrap_or(128))); - // Declared before the track is released to readers, so a warm cache waiting on - // this copy judges itself against where the live feed actually starts. - let Some(request) = self.take_pending(request_id) else { - // Aborted while the answer was in hand. The parked request is already rejected. - self.remove_subscribe(request_id); - return; + let (mut track, dynamic) = match resumed { + // The copy already holds the track: it is current again up to the answer's + // Largest Location. + Some(idle) => { + if !self.state.lock().subscribes.contains_key(&request_id) { + // Aborted with the session while the answer was in hand. + return None; + } + let mut track = idle.track; + track.set_live(largest.map(|largest| track::Position { + group: largest.group, + frame: largest.object, + })); + (track, idle.dynamic) + } + None => { + let Some(request) = self.take_pending(request_id) else { + // Aborted while the answer was in hand. The parked request is already rejected. + self.remove_subscribe(request_id); + return None; + }; + let request = match live { + true => request.resolving_start(), + false => request, + }; + // Serves cache misses with a group FETCH. Registered before accepting, so a + // miss queued meanwhile waits for it rather than failing for want of a handler. + let dynamic = request.dynamic(); + (request.accept(info), dynamic) + } }; - let request = match live { - true => request.resolving_start(), - false => request, + // A live join starts at the publisher's edge; an absolute one where it asked. + let _ = match live { + true => track.start_at(largest.map(|largest| largest.group)), + false => track.start_at(start.map(|start| start.group)), }; - // Serves cache misses with a group FETCH. Registered before accepting, so a miss - // queued meanwhile waits for it rather than failing for want of a handler. - let dynamic = request.dynamic(); - let mut track = request.accept(info); - if live { - let _ = track.start_at(largest.map(|largest| largest.group)); - } let mut fetching: Option> = None; { let mut state = self.state.lock(); @@ -1905,7 +2047,7 @@ where } self.remove_subscribe(request_id); let _ = track.abort(err); - return; + return None; } if let Some(joining) = joining && largest.is_some() @@ -1913,20 +2055,20 @@ where fetching = self.start_joining_fetch(request_id, &track, joining).await; } - // One event ends the subscription: the last consumer leaving, or the + // One event ends the subscription: the last subscriber leaving, or the // publisher's PUBLISH_DONE. The broadcast ending does not: a retraction // does not disturb subscriptions already in flight. enum End { - Unused, + Idle, Done(Result), Fetch(group::Request), } let mut fetch_done = fetching.is_none(); let demand = track.demand(); - // Group FETCHes for cache misses, cancelled with the subscription. - let mut group_fetches = TaskSet::owned(); - let cancelled = { + // Nobody subscribing at all (only fetches asked) needs no subscription. + let mut subscribed = track.subscription().is_some(); + let idle = { let mut done = std::pin::pin!(Self::read_publish_done(&mut stream.reader, self.version)); loop { let end = kio::wait(|waiter| { @@ -1941,8 +2083,13 @@ where return Poll::Ready(End::Fetch(request)); } let _ = group_fetches.poll(waiter); - if demand.poll_unused(waiter).is_ready() { - return Poll::Ready(End::Unused); + // The last subscriber left: the upstream subscription goes with it, as on + // lite, and the copy lingers for fetches and a returning subscriber. + while let Poll::Ready(Ok(subscription)) = track.poll_subscription_changed(waiter) { + subscribed = subscription.is_some(); + } + if !subscribed || demand.poll_unused(waiter).is_ready() { + return Poll::Ready(End::Idle); } waiter.poll_future(done.as_mut()).map(End::Done) }) @@ -1952,23 +2099,20 @@ where End::Fetch(request) => { let fetch = self.clone().run_group_fetch( broadcast_path.to_owned(), - track_name.clone(), + track_name.to_owned(), request, timescale, ); group_fetches.push(fetch); } - End::Unused => match track.abort_unused(Error::Cancel) { - Ok(()) => { - tracing::info!(broadcast = %self.origin.absolute(&broadcast_path), track = %track_name, "subscribe cancelled"); - break true; - } - Err(used) => track = used, - }, + End::Idle => { + tracing::info!(broadcast = %self.origin.absolute(broadcast_path), track = %track_name, "subscribe cancelled (idle)"); + break true; + } End::Done(res) => { match res { Ok(count) => { - tracing::info!(broadcast = %self.origin.absolute(&broadcast_path), track = %track_name, "subscribe complete"); + tracing::info!(broadcast = %self.origin.absolute(broadcast_path), track = %track_name, "subscribe complete"); // The publisher sends PUBLISH_DONE once every data stream it opened // is closed, but QUIC does not order them, so some can still be on // their way. Wait until Stream Count of their headers arrived and @@ -1999,7 +2143,7 @@ where } Err(err) => { tracing::debug!(%err, "subscribe ended with error"); - let _ = track.abort(err); + let _ = track.clone().abort(err); } } // The publisher already ended the request, so there is nothing to cancel. @@ -2010,15 +2154,116 @@ where }; // Clean up - self.remove_subscribe(request_id); + let aborted = self.remove_subscribe(request_id).is_none(); - match cancelled { + match idle { true => self.cancel_subscribe(stream, request_id).await, // The publisher already ended the request, so a FIN is all we owe it. false => { stream.writer.finish().ok(); + return None; } } + // A session abort took the copy too. + if aborted { + return None; + } + // What the copy cached goes stale from here. + track.set_idle(); + Some(Idle { + demand: track.demand(), + track, + dynamic, + timescale, + }) + } + + /// Keep a copy nobody subscribes to, cache and all, for fetches and a subscriber + /// returning soon: `Some` once one does, to subscribe again, `None` once nobody held + /// it through the linger, or the session ended. + async fn linger( + &mut self, + broadcast_path: &Path<'_>, + track_name: &str, + mut idle: Idle, + group_fetches: &mut TaskSet, + ) -> Option { + // Registered so a session abort ends the copy with its error. + let id = { + let mut state = self.state.lock(); + state.next_lingering += 1; + let id = state.next_lingering; + state.lingering.insert(id, idle.track.clone()); + id + }; + let mut linger = crate::runtime::Deadline::new(&self.runtime); + let mut subscribed = idle.track.subscription().is_some(); + enum Step { + Fetch(group::Request), + Subscribe, + Expired, + Closed, + } + let resume = loop { + let step = kio::wait(|waiter| { + if idle.track.poll_closed(waiter).is_ready() { + return Poll::Ready(Step::Closed); + } + if let Poll::Ready(Ok(request)) = idle.dynamic.poll_requested_group(waiter) { + return Poll::Ready(Step::Fetch(request)); + } + let _ = group_fetches.poll(waiter); + while let Poll::Ready(Ok(subscription)) = idle.track.poll_subscription_changed(waiter) { + subscribed = subscription.is_some(); + } + if subscribed { + return Poll::Ready(Step::Subscribe); + } + // Nobody holds it: let it go after the linger. A reader waiting on a fetch holds + // it too. A holder returning restarts the countdown when it next leaves. + if idle.demand.poll_unused(waiter).is_ready() { + if linger.deadline().is_none() { + linger.set(self.runtime.now().checked_add(track::IDLE_LINGER)); + } + if linger.poll(waiter).is_ready() { + return Poll::Ready(Step::Expired); + } + let _ = idle.demand.poll_used(waiter); + } else { + linger.set(None); + } + Poll::Pending + }) + .await; + + match step { + Step::Fetch(request) => { + let fetch = self.clone().run_group_fetch( + broadcast_path.to_owned(), + track_name.to_owned(), + request, + idle.timescale, + ); + group_fetches.push(fetch); + } + Step::Subscribe => break true, + Step::Expired => match idle.track.abort_unused(Error::Cancel) { + Ok(()) => { + tracing::info!(broadcast = %self.origin.absolute(broadcast_path), track = %track_name, "track released (idle)"); + self.state.lock().lingering.remove(&id); + return None; + } + Err(used) => { + idle.track = used; + linger.set(None); + } + }, + Step::Closed => break false, + } + }; + // The session's abort took it out of the registry already, or this does. + let registered = self.state.lock().lingering.remove(&id).is_some(); + (resume && registered).then_some(idle) } /// Read the PUBLISH_DONE that ends an Established subscription, as the end it reports @@ -2096,19 +2341,17 @@ where stream: &mut Stream, request_id: RequestId, broadcast: &Path<'_>, - request: &track::Request, + track_name: &str, + priority: u8, join: Join, ) -> Result<(), Error> { - // Read the aggregate now: a subscriber can join while the request ID and stream - // were awaited, and nothing updates the priority after SUBSCRIBE. - let priority = request.subscription().map(|s| s.priority).unwrap_or(0); stream.writer.encode(&ietf::Subscribe::ID).await?; stream .writer .encode(&ietf::Subscribe { request_id, track_namespace: broadcast.to_owned(), - track_name: request.name().into(), + track_name: track_name.into(), subscriber_priority: super::priority::to_wire(priority), group_order: GroupOrder::Descending, filter: join.filter, @@ -2133,15 +2376,21 @@ where track: &track::Producer, joining: JoiningFetch, ) -> Option> { - let fill = { + let (fill, largest) = { let state = self.state.lock(); - state.subscribes.get(&subscribe_id)?.fill.clone() + let held = state.subscribes.get(&subscribe_id)?; + (held.fill.clone(), held.largest) + }; + // Where the subscription starts if the join falls back to live. + let live = Live { + track: track.clone(), + start: largest.map(|largest| largest.group), }; let fetch_id = match self.control.next_request_id(&self.runtime).await { Ok(id) => id, Err(_) => { - settle_join_live(&fill); + settle_join_live(&fill, live); return None; } }; @@ -2157,7 +2406,7 @@ where Ok(s) => s, Err(err) => { tracing::debug!(%err, "failed to open joining FETCH stream"); - settle_join_live(&fill); + settle_join_live(&fill, live); return None; } }; @@ -2193,22 +2442,22 @@ where .await { tracing::debug!(%err, "failed to write joining FETCH"); - settle_join_live(&fill); + settle_join_live(&fill, live); return None; } let mut this = self.clone(); Some( async move { - this.finish_joining_fetch(stream, fill).await; + this.finish_joining_fetch(stream, fill, live).await; } .maybe_boxed(), ) } - async fn finish_joining_fetch(&mut self, mut stream: Stream, fill: kio::Producer) { + async fn finish_joining_fetch(&mut self, mut stream: Stream, fill: kio::Producer, live: Live) { if !matches!(self.read_fetch_response(&mut stream).await, Ok(true)) { - settle_join_live(&fill); + settle_join_live(&fill, live); let _ = stream.writer.close().await; return; } @@ -2315,13 +2564,15 @@ where } }; - let (mut track, timescale, fill, mut reading) = { + let (mut track, timescale, fill, resume, mut reading) = { let state = self.state.lock(); let track = state.subscribes.get(&request_id).ok_or(Error::NotFound)?; ( track.producer.clone().ok_or(Error::NotFound)?, track.timescale, track.fill.clone(), + // Only the group the subscription resumes partway through. + track.resume.filter(|resume| resume.group == group.group_id), // Every data stream counts toward PUBLISH_DONE's Stream Count, even one // dropped below, and the subscription's end waits until it is read. Reading::open(&track.tail, Some(group.group_id), self.runtime.now()), @@ -2341,13 +2592,14 @@ where // degradation as a publisher that no longer holds the head. // // A fill we asked for is the exception, since its fetch stream is carrying exactly - // that head for [`Self::open_group`] to stitch this onto. + // that head for [`Self::open_group`] to stitch this onto. So is the group a resumed + // subscription asked to start partway through: the head came from another route. // // The bit is only the publisher's claim, so what is enforced is the object ids // themselves: [`next_object_id`] holds every object to starting where the head // stopped and incrementing by 1, whatever the header said and on the drafts that // have no such bit to read. - if !group.flags.first_object && !fill.read().outstanding() { + if !group.flags.first_object && !fill.read().outstanding() && resume.is_none() { tracing::debug!( track_alias = %group.track_alias, group = %group.group_id, @@ -2361,7 +2613,7 @@ where // Otherwise dropping the local subscriber cannot end this handler. let opened = { let mut opening = track.clone(); - let mut open = std::pin::pin!(self.open_group(stream, &mut opening, &fill, &group, &mut reading)); + let mut open = std::pin::pin!(self.open_group(stream, &mut opening, &fill, resume, &group, &mut reading)); kio::wait(|waiter| { if let Poll::Ready(err) = track.poll_closed(waiter) { return Poll::Ready(Err(err)); @@ -2592,6 +2844,7 @@ where stream: &mut Reader, track: &mut track::Producer, fill: &kio::Producer, + resume: Option, header: &ietf::GroupHeader, reading: &mut Reading, ) -> Result { @@ -2627,6 +2880,17 @@ where } if !fill.read().outstanding() { + // The group a resumed subscription picks up partway through starts where it + // asked, and a stream with no objects is the end of a group complete there. + // A publisher sending more of it (a pre-draft-20 join asks for all of it) has + // the objects below the start dropped as they arrive. + if let Some(resume) = resume + && first.is_none_or(|first| first.id <= resume.frame) + { + let mut producer = create(track)?; + producer.start_at(resume.frame)?; + return Ok(Opened::Group(producer, first.map_or(resume.frame, |first| first.id))); + } return Ok(Opened::Group(create(track)?, 0)); } @@ -2727,6 +2991,8 @@ struct GroupIngest { version: Version, prior_object: Option, start: u64, + /// The object being read is one the group started past, so it is read and dropped. + dropping: bool, phase: IngestPhase, budget: frame::Budget, } @@ -2744,6 +3010,8 @@ enum IngestPhase { Status { timestamp: Option }, /// Streaming the object payload. Payload { frame: frame::ProducerOwned }, + /// Discarding a dropped object's payload: the bytes of it still to read. + Skip { size: usize }, /// An explicit end-of-group or end-of-track status arrived. Finished(Ended), } @@ -2763,6 +3031,7 @@ impl GroupIngest { version: subscriber.version, prior_object: None, start, + dropping: false, phase: IngestPhase::Delta, budget: subscriber.frames.clone(), } @@ -2791,7 +3060,7 @@ where return self.recv_group_fetch(stream, slot).await; } - let (subscribe_id, fill, joining, largest, _counted) = { + let (subscribe_id, fill, joining, largest, resume, _counted) = { let state = self.state.lock(); // A draft-20 fill is named by the SUBSCRIBE's request id. A pre-draft-20 joining // FETCH has its own id, which `fetches` maps back to that subscription. @@ -2803,7 +3072,14 @@ where let counted = joined .is_none() .then(|| Reading::open(&track.tail, None, self.runtime.now())); - (subscribe_id, track.fill.clone(), track.joining, track.largest, counted) + ( + subscribe_id, + track.fill.clone(), + track.joining, + track.largest, + track.resume, + counted, + ) }; // SUBSCRIBE_OK declares the units these object timestamps are in, and this stream can @@ -2848,7 +3124,7 @@ where // track does not close a group producer, since those lifecycles are independent. let res = { let mut serving = track.clone(); - let mut serve = std::pin::pin!(self.run_fill(stream, &mut serving, timescale, joining, largest)); + let mut serve = std::pin::pin!(self.run_fill(stream, &mut serving, timescale, joining, largest, resume)); kio::wait(|waiter| { if let Poll::Ready(err) = track.poll_closed(waiter) { return Poll::Ready(Err(err)); @@ -2895,11 +3171,12 @@ where timescale: Option, joining: Option, largest: Option, + resume: Option, ) -> Result { let mut head: Option<(u64, u64, crate::recv::Group)> = None; match self - .run_fill_objects(stream, track, timescale, joining, largest, &mut head) + .run_fill_objects(stream, track, timescale, joining, largest, resume, &mut head) .await { Ok(()) => Ok(match head { @@ -2939,6 +3216,10 @@ where /// Location; each complete group below that is finished, and the last one is the head /// the live stream continues. Anything else is a head the model cannot represent, and /// refusing the stream leaves the subscription itself alone. + /// + /// A pre-draft-20 join resuming partway through a group asks for all of it, so the + /// objects below `resume` are dropped: another route already delivered them. + #[allow(clippy::too_many_arguments)] async fn run_fill_objects( &mut self, stream: &mut Reader, @@ -2946,6 +3227,7 @@ where timescale: Option, joining: Option, largest: Option, + resume: Option, head: &mut Option<(u64, u64, crate::recv::Group)>, ) -> Result<(), Error> { let mut prior_group = None; @@ -3009,9 +3291,15 @@ where } } - let (_, next, producer) = head.as_mut().expect("the head was created above"); + let (sequence, next, producer) = head.as_mut().expect("the head was created above"); + if *next == 0 + && let Some(resume) = resume.filter(|resume| resume.group == *sequence) + { + producer.start_at(resume.frame)?; + } + let keep = *next >= producer.frame_count() as u64; if !self - .recv_fetch_payload(stream, producer, object.properties, timescale) + .recv_fetch_payload(stream, producer, object.properties, timescale, keep) .await? { return Err(Error::Unsupported); @@ -3022,7 +3310,8 @@ where Ok(()) } - /// Read one fetch object's length and payload into `producer`, after its header. + /// Read one fetch object's length and payload into `producer`, after its header, or + /// past it unless `keep`. /// /// Returns `false` for a draft-14 or 15 end-of-group or end-of-track marker, which /// is a status rather than a frame. @@ -3032,6 +3321,7 @@ where producer: &mut group::Producer, properties: Option>, timescale: Option, + keep: bool, ) -> Result { // The properties carry the frame's presentation timestamp (the Timestamp Object // Property) in the units the track declared. A track that declared none opted @@ -3055,6 +3345,11 @@ where _ => return Err(Error::Unsupported), } } + if !keep { + let mut remaining = usize::try_from(size).map_err(|_| Error::FrameTooLarge)?; + std::future::poll_fn(|cx| stream.poll_skip(cx, &mut remaining)).await?; + return Ok(true); + } // `create_frame_owned` is the allocation chokepoint: it rejects an oversized `size` // and allocates up front only within the budget, so no pre-check is needed. @@ -3417,7 +3712,7 @@ where } match self - .recv_fetch_payload(stream, producer, object.properties, timescale) + .recv_fetch_payload(stream, producer, object.properties, timescale, true) .await? { true => next += 1, @@ -3580,15 +3875,25 @@ fn poll_settled(settle: &mut Settle, waiter: &kio::Waiter, fill: &kio::Producer< settle.poll(waiter, |tail| filled && count > 0 && tail.streams() >= count) } +/// The track a joining FETCH serves, and the group its subscription starts at should the +/// join fall back to live: the publisher's edge, Largest. +struct Live { + track: track::Producer, + start: Option, +} + /// A refused or missing joining FETCH continues the subscription live: drop the outstanding -/// fill so a mid-group tail is not left waiting on a head that is never coming. -fn settle_join_live(fill: &kio::Producer) { - let Ok(mut state) = fill.write() else { - return; - }; - if matches!(*state, Fill::Requested | Fill::Serving(_)) { +/// fill so a mid-group tail is not left waiting on a head that is never coming, and declare +/// the start at the edge, since nothing below it is coming now. +fn settle_join_live(fill: &kio::Producer, live: Live) { + if let Ok(mut state) = fill.write() + && matches!(*state, Fill::Requested | Fill::Serving(_)) + { *state = Fill::Done; } + if let Some(start) = live.start { + let _ = live.track.clone().start_at(start); + } } impl GroupIngest { @@ -3608,7 +3913,9 @@ impl GroupIngest { let Some(id_delta) = ready!(reader.poll_decode_maybe::(&mut cx))? else { return Poll::Ready(Ok(Ended::Group)); }; - self.prior_object = Some(next_object_id(self.prior_object, id_delta, self.start)?); + let id = next_object_id(self.prior_object, id_delta, self.start)?; + self.prior_object = Some(id); + self.dropping = id < group.frame_count() as u64; self.phase = match self.has_extensions { true => IngestPhase::ExtSize, false => IngestPhase::Size { timestamp: None }, @@ -3637,6 +3944,11 @@ impl GroupIngest { self.phase = IngestPhase::Status { timestamp: *timestamp }; continue; } + if self.dropping { + let size = usize::try_from(size).map_err(|_| Error::FrameTooLarge)?; + self.phase = IngestPhase::Skip { size }; + continue; + } // `create_frame_owned` is the allocation chokepoint: it rejects an // oversized `size` and allocates up front only within the budget, so // no pre-check is needed. @@ -3647,9 +3959,11 @@ impl GroupIngest { IngestPhase::Status { timestamp } => { let status: u64 = ready!(reader.poll_decode(&mut cx))?; if status == 0 { - let timestamp = timestamp.unwrap_or_else(|| crate::Timestamp::from(self.runtime.now())); - let frame = group.create_frame_owned(frame::Info { size: 0, timestamp }, &self.budget)?; - frame.finish()?; + if !self.dropping { + let timestamp = timestamp.unwrap_or_else(|| crate::Timestamp::from(self.runtime.now())); + let frame = group.create_frame_owned(frame::Info { size: 0, timestamp }, &self.budget)?; + frame.finish()?; + } self.phase = IngestPhase::Delta; } else if status == END_OF_GROUP && !self.has_end { self.phase = IngestPhase::Finished(Ended::Group); @@ -3677,6 +3991,10 @@ impl GroupIngest { } } } + IngestPhase::Skip { size } => { + ready!(reader.poll_skip(&mut cx, size))?; + self.phase = IngestPhase::Delta; + } IngestPhase::Finished(ended) => { let ended = std::mem::replace(ended, Ended::Group); return Poll::Ready(Ok(ended)); @@ -4777,12 +5095,13 @@ mod tests { "{version:?}: alias 7 must be bound before we cancel", ); - // The last consumer leaves: nothing wants this track any more. + // The last consumer leaves: nothing wants this track any more. The upstream + // subscription is cancelled at once, and the copy lingers for a returning reader. drop(subscription); drop(track); drop(consumer); - tokio::time::timeout(std::time::Duration::from_secs(1), serving) + tokio::time::timeout(crate::track::IDLE_LINGER * 2, serving) .await .expect("run_subscribe did not finish") .unwrap(); @@ -6291,7 +6610,9 @@ fn subscribe_join( version: Version, ) -> Result { if !Filter::is_draft20(version) { - if start.is_some_and(|start| start.frame != 0) || end.is_some() { + // No pre-draft-20 join starts partway through a group, so a resume point there + // asks for its whole group; the objects below it are dropped on arrival. + if end.is_some() { return Err(Error::Unsupported); } return Ok(Join { @@ -6526,16 +6847,15 @@ mod filter_tests { } } - /// A frame-level start has no joining-FETCH spelling, so it is refused rather than - /// rounded down to the group. + /// A frame-level start has no joining-FETCH spelling, so a resumed subscription asks + /// for its whole group; the frames below the start are already cached elsewhere. #[test] - fn older_drafts_refuse_a_frame_level_start() { + fn older_drafts_widen_a_frame_level_start_to_its_group() { for version in JOINING_DRAFTS { + let join = subscribe_join(Some(track::Position { group: 7, frame: 1 }), None, version) + .unwrap_or_else(|err| panic!("{version}: {err}")); assert!( - matches!( - subscribe_join(Some(track::Position { group: 7, frame: 1 }), None, version), - Err(Error::Unsupported) - ), + matches!(join.fetch, Some(JoiningFetch::Absolute { group_id: 7 })), "{version}" ); } @@ -6750,6 +7070,15 @@ mod stitch_tests { self } + /// Resume the subscription partway through a group another route delivered the + /// head of. + fn with_resume(self, resume: track::Position) -> Self { + if let Some(track) = self.subscriber.state.lock().subscribes.get_mut(&REQUEST) { + track.resume = Some(resume); + } + self + } + /// A reader over the next scripted stream, standing in for one the peer opened. async fn stream(&self) -> Reader<::RecvStream, Version> { let mut session = self.session.clone(); @@ -7166,6 +7495,59 @@ mod stitch_tests { assert_eq!(frames.len(), 1, "the group is whatever one producer wrote, not both"); } + /// A resumed subscription that gets its group whole (a pre-draft-20 join can only ask + /// for all of it) keeps only the objects from where it asked: another route already + /// delivered the rest. + #[tokio::test] + async fn a_resumed_whole_group_drops_the_delivered_head() { + let h = Harness::new(Fill::Done, vec![tail_stream(SEQUENCE, 0, &[b"0", b"1", b"2"])]).with_resume( + track::Position { + group: SEQUENCE, + frame: 2, + }, + ); + let mut consumer = h.track.subscribe(None); + let mut whole = h.stream().await; + h.subscriber.clone().recv_group(&mut whole).await.expect("the group"); + + let mut group = consumer.recv_group().await.unwrap().expect("the group arrived"); + group.start_at(0); + assert_eq!(group.index(), 2, "the group starts where the subscription asked"); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"2"); + assert!(group.read_frame().await.unwrap().is_none()); + } + + /// The same for a joining FETCH's head: the objects below the resume point are + /// dropped, and the live tail continues what is left. + #[tokio::test] + async fn a_resumed_fill_drops_the_delivered_head() { + let h = Harness::new( + Fill::Serving(Some(Timescale::MICRO)), + vec![ + fill_stream(SEQUENCE, &[b"0", b"1", b"2"]), + tail_stream(SEQUENCE, 3, &[b"3"]), + ], + ) + .with_resume(track::Position { + group: SEQUENCE, + frame: 2, + }); + let mut consumer = h.track.subscribe(None); + let mut fill = h.stream().await; + let mut tail = h.stream().await; + h.subscriber.clone().recv_fill(&mut fill).await.expect("fill"); + h.subscriber.clone().recv_group(&mut tail).await.expect("tail"); + + let mut group = consumer.recv_group().await.unwrap().expect("the group arrived"); + group.start_at(0); + assert_eq!(group.index(), 2, "the group starts where the subscription asked"); + let mut frames = Vec::new(); + while let Some(frame) = group.read_frame().await.unwrap() { + frames.push(frame.payload.to_vec()); + } + assert_eq!(frames, [b"2".to_vec(), b"3".to_vec()]); + } + /// Without a head there is nothing to stitch onto, so a stream that starts part way /// through a group is dropped and the join degrades to the next group boundary. This is /// what a strict publisher gives a subscriber that asks for no fill. @@ -7875,41 +8257,6 @@ mod joining_fetch_tests { } } - /// A frame-level start is refused before SUBSCRIBE is written, rather than rounded down. - #[tokio::test(start_paused = true)] - async fn a_frame_level_start_never_reaches_the_wire() { - let version = Version::Draft19; - let session = ScriptedSession::new(Vec::new()); - let log = session.log.clone(); - let (tasks, _task_set) = crate::util::TaskSet::new(); - let mut subscriber = Subscriber::new( - crate::time::Clock::tokio(), - session, - crate::origin::Config::new(crate::Hop::new(1).unwrap()).produce(), - Control::new(None, false), - None, - peer::PeerSetup::default(), - crate::Hop::new(1).unwrap(), - None, - version, - tasks, - Default::default(), - ); - - let producer = crate::broadcast::Info::default().produce(); - let mut dynamic = producer.dynamic(); - let consumer = producer.consume(); - let track = consumer.track("video").unwrap(); - let _subscription = - track.subscribe(track::Subscription::default().with_start(track::Position { group: 7, frame: 1 })); - let request = dynamic.requested_track().await.expect("no track requested"); - - subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await; - - let writes = log.writes.lock().unwrap().clone(); - assert!(writes.is_empty(), "a refused join must not write SUBSCRIBE"); - } - #[derive(Clone, Copy, PartialEq, Eq)] enum FetchStage { Unanswered, diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 64a767eefed..f33ad313af2 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -95,12 +95,15 @@ fn serving_max_age(version: Version, requested: Duration) -> Duration { /// carry a Max Age, but there it is a staleness tolerance only; Lite01/02 additionally get /// an unbounded budget so nothing is dropped under them (see [`serving_max_age`]), which /// must not read as a request to replay the whole cache on join. -fn position_cursor(track: &mut track::Subscriber, version: Version, start_group: Option) { +/// +/// `latest` is the newest group when the SUBSCRIBE arrived, not once the subscription +/// resolved: anything written in between is newer, so it is delivered. +fn position_cursor(track: &mut track::Subscriber, version: Version, start_group: Option, latest: Option) { if version.resolves_start() || start_group.is_some() { return; } - if let Some(latest) = track.latest() { + if let Some(latest) = latest { track.start_at(latest); } } @@ -1206,6 +1209,8 @@ enum SubscribeServe { Confirm { msg: lite::Subscribe<'static>, subscribing: track::Subscribing, + /// The newest group when the SUBSCRIBE arrived; see [`position_cursor`]. + latest: Option, update: Option, }, /// Streaming groups and datagrams. Boxed: by far the largest state, and the enum @@ -1236,10 +1241,13 @@ impl Request for SubscribeServe { // One subscriber for the whole subscription: the run loop polls its groups and its // best-effort datagrams from this single cursor, so a group-only or datagram-only // track opens exactly one subscription (no duplicate demand). - let subscribing = broadcast.track(&msg.track)?.subscribe(subscription).into_inner(); + let track = broadcast.track(&msg.track)?; + let latest = track.latest(); + let subscribing = track.subscribe(subscription).into_inner(); Ok(Self::Confirm { msg, subscribing, + latest, update: None, }) } @@ -1262,13 +1270,16 @@ impl Request for SubscribeServe { loop { match self { Self::Confirm { subscribing, .. } => { - let track = ready!(subscribing.poll_ok(waiter))?; - let Self::Confirm { msg, update, .. } = std::mem::replace( + let mut track = ready!(subscribing.poll_ok(waiter))?; + let Self::Confirm { + msg, latest, update, .. + } = std::mem::replace( self, Self::Drain { children: Default::default(), }, - ) else { + ) + else { unreachable!() }; @@ -1315,7 +1326,9 @@ impl Request for SubscribeServe { opens: Default::default(), }; - let mut run = TrackRun::new(sub, track, Bounds::from(&msg), track_priority_tx); + let bounds = Bounds::from(&msg); + position_cursor(&mut track, shared.version, bounds.start_group, latest); + let mut run = TrackRun::new(sub, track, bounds, track_priority_tx); if let Some(update) = update { run.update(update); } @@ -1546,20 +1559,20 @@ mod test { ); let mut legacy = served(Version::Lite01); - position_cursor(&mut legacy, Version::Lite01, None); + position_cursor(&mut legacy, Version::Lite01, None, producer.latest()); assert_eq!(drain(&mut legacy), vec![2]); // Lite03-05 declare a budget, but their drafts define it as a staleness tolerance // and an absent start as the latest group, so they are pinned all the same. let mut tolerant = producer.subscribe(track::Subscription::default().with_max_age(std::time::Duration::from_secs(5))); - position_cursor(&mut tolerant, Version::Lite05, None); + position_cursor(&mut tolerant, Version::Lite05, None, producer.latest()); assert_eq!(drain(&mut tolerant), vec![2]); // On lite-06 the declared budget is what resolves the start, so it stands. let mut declared = producer.subscribe(track::Subscription::default().with_max_age(std::time::Duration::from_secs(5))); - position_cursor(&mut declared, Version::Lite06, None); + position_cursor(&mut declared, Version::Lite06, None, producer.latest()); assert_eq!(drain(&mut declared), vec![0, 1, 2]); } @@ -2321,8 +2334,6 @@ impl TrackRun { bounds: Bounds, track_priority_tx: kio::Producer, ) -> Self { - position_cursor(&mut track, ctx.version, bounds.start_group); - // Apply the initial cap from the original Subscribe. Subsequent updates // flow through the SUBSCRIBE_UPDATE arm below. track.end_at(bounds.end_group.map_or(Bound::Unbounded, Bound::Included)); @@ -2419,6 +2430,26 @@ impl TrackRun { } } + // A start past everything the live feed has (a subscriber resuming just after what it + // holds) is answered at once with the largest position, on versions that carry it: + // a quiet track may not reach that start for a while, and the subscriber judges + // what it holds against the answer. + if self.emit_range + && !self.start_sent + && self.ctx.version.has_largest() + && let Some(start) = self.track.subscription().start + && let Poll::Ready(Some(largest)) = self.track.poll_live(waiter) + && start > largest + { + self.start_sent = true; + writer.buffer(&lite::SubscribeResponse::Start(lite::SubscribeStart { + group: start.group, + largest: Some(largest), + }))?; + self.track.raise_start_to(start.group); + return Poll::Ready(Ok(ControlFlow::Continue(()))); + } + // One cursor drives the whole subscription: poll the cap-aware arrival-order // group and, when enabled, the next best-effort datagram. Groups are polled // first so a datagram burst can't starve them; datagrams flow whenever no @@ -2495,8 +2526,16 @@ impl TrackRun { self.skipped.clear(); self.start_sent = true; // Only the group: the subscriber derives the start frame from its own request - // (see `lite::SubscribeStart`). - writer.buffer(&lite::SubscribeResponse::Start(lite::SubscribeStart { group: start }))?; + // (see `lite::SubscribeStart`). The track is live by now, since its first group + // was readable. + let largest = match self.track.poll_live(&kio::Waiter::noop()) { + Poll::Ready(largest) => largest, + Poll::Pending => None, + }; + writer.buffer(&lite::SubscribeResponse::Start(lite::SubscribeStart { + group: start, + largest, + }))?; // SUBSCRIBE_START is an implicit drop of everything below the resolved start (the // subscriber records it as a permanent miss), so a lower group arriving late must // not be served after all. A widening SUBSCRIBE_UPDATE re-lowers the floor, @@ -3343,8 +3382,8 @@ mod serve_group_test { track.finish().unwrap(); run.await.unwrap(); - // SUBSCRIBE_START at 0, then SUBSCRIBE_END at 3 with 2 streams. - assert_eq!(*log.writes.lock().unwrap(), [0, 1, 0, 1, 2, 3, 2]); + // SUBSCRIBE_START at 0 (largest 0.0), then SUBSCRIBE_END at 3 with 2 streams. + assert_eq!(*log.writes.lock().unwrap(), [0, 3, 0, 1, 0, 1, 2, 3, 2]); } /// A lite-07 run over a relay's track, whose upstream subscription still waits on the @@ -3408,9 +3447,13 @@ mod serve_group_test { /// Whether the first thing written was SUBSCRIBE_START at `group`. fn started_at(&self, group: u64) -> bool { - let start = lite::SubscribeResponse::Start(lite::SubscribeStart { group }); - let start = start.encode_bytes(Version::Lite07).unwrap(); - self.session.log.writes.lock().unwrap().starts_with(&start) + use crate::coding::Decode; + let writes = self.session.log.writes.lock().unwrap(); + let mut slice = writes.as_slice(); + matches!( + lite::SubscribeResponse::decode(&mut slice, Version::Lite07), + Ok(lite::SubscribeResponse::Start(start)) if start.group == group + ) } } @@ -3521,7 +3564,11 @@ mod serve_group_test { write_group(&mut track, 1, 1000); track.finish().unwrap(); assert!(futures::poll!(run.as_mut()).is_pending(), "group 1 is still opening"); - assert_eq!(*log.writes.lock().unwrap(), [0, 1, 0], "only SUBSCRIBE_START so far"); + assert_eq!( + *log.writes.lock().unwrap(), + [0, 3, 0, 1, 0], + "only SUBSCRIBE_START so far" + ); let Ok(mut open) = gate.write() else { panic!("transport gate closed"); @@ -3529,7 +3576,7 @@ mod serve_group_test { *open = true; drop(open); run.await.unwrap(); - assert_eq!(*log.writes.lock().unwrap(), [0, 1, 0, 1, 2, 2, 1]); + assert_eq!(*log.writes.lock().unwrap(), [0, 3, 0, 1, 0, 1, 2, 2, 1]); } } diff --git a/rs/moq-net/src/lite/subscribe.rs b/rs/moq-net/src/lite/subscribe.rs index 4656e945219..1d09f01c792 100644 --- a/rs/moq-net/src/lite/subscribe.rs +++ b/rs/moq-net/src/lite/subscribe.rs @@ -295,6 +295,9 @@ impl Message for SubscribeOk { #[derive(Clone, Debug)] pub struct SubscribeStart { pub group: u64, + /// The publisher's largest (group, frame) when it answered, `None` for a track with + /// nothing yet. Lite07+ only; not on the wire before, where it decodes as `None`. + pub largest: Option, } impl Message for SubscribeStart { @@ -302,16 +305,40 @@ impl Message for SubscribeStart { if !version.has_track_stream() { return Err(DecodeError::Version); } - Ok(Self { - group: u64::decode(r, version)?, - }) + let group = u64::decode(r, version)?; + let largest = match version.has_largest() { + // Group + 1, so 0 is a track with nothing yet; the frame follows only otherwise. + true => match u64::decode(r, version)?.checked_sub(1) { + Some(group) => Some(crate::track::Position { + group, + frame: u64::decode(r, version)?, + }), + None => None, + }, + false => None, + }; + Ok(Self { group, largest }) } fn encode_msg(&self, w: &mut W, version: Version) -> Result<(), EncodeError> { if !version.has_track_stream() { return Err(EncodeError::Version); } - self.group.encode(w, version) + self.group.encode(w, version)?; + if version.has_largest() { + match self.largest { + Some(largest) => { + largest + .group + .checked_add(1) + .ok_or(EncodeError::BoundsExceeded)? + .encode(w, version)?; + largest.frame.encode(w, version)?; + } + None => 0u64.encode(w, version)?, + } + } + Ok(()) } } @@ -578,7 +605,10 @@ mod test { #[test] fn subscribe_start_roundtrips_on_lite05() { - let resp = SubscribeResponse::Start(SubscribeStart { group: 42 }); + let resp = SubscribeResponse::Start(SubscribeStart { + group: 42, + largest: None, + }); let mut buf = Vec::new(); resp.encode(&mut buf, Version::Lite05).unwrap(); let mut slice = buf.as_slice(); @@ -588,6 +618,29 @@ mod test { } } + /// Lite-07 carries the publisher's largest position; earlier versions leave it off the + /// wire, so it decodes as `None` there. + #[test] + fn subscribe_start_carries_the_largest_position_on_lite07() { + for largest in [None, Some(crate::track::Position { group: 3, frame: 2 })] { + let resp = SubscribeResponse::Start(SubscribeStart { group: 4, largest }); + let mut buf = Vec::new(); + resp.encode(&mut buf, Version::Lite07).unwrap(); + let mut slice = buf.as_slice(); + match SubscribeResponse::decode(&mut slice, Version::Lite07).unwrap() { + SubscribeResponse::Start(start) => assert_eq!((start.group, start.largest), (4, largest)), + other => panic!("expected Start, got {other:?}"), + } + } + let resp = SubscribeResponse::Start(SubscribeStart { + group: 4, + largest: Some(crate::track::Position { group: 3, frame: 2 }), + }); + let mut buf = Vec::new(); + resp.encode(&mut buf, Version::Lite06).unwrap(); + assert_eq!(buf, [0, 1, 4], "lite-06 has no largest position"); + } + #[test] fn subscribe_end_roundtrips_on_lite05() { let resp = SubscribeResponse::End(SubscribeEnd { group: 7, streams: 3 }); diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 137aa17c480..57734d0e52a 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -468,6 +468,10 @@ impl Subscriber { if let Ok(mut tail) = entry.tail.write() { tail.account(dg.sequence..dg.sequence.saturating_add(1), self.runtime.now()); } + // A datagram says where the live feed is as well as a group does. + if !entry.producer.is_live() { + entry.producer.set_live(Some(Position::group(dg.sequence))); + } entry.producer.insert_datagram(dg.sequence, timestamp, dg.payload)?; Ok(()) } @@ -789,6 +793,15 @@ impl GroupRecv { // The subscription's end waits until this stream is read. let reading = Reading::open(&entry.tail, Some(hdr.sequence), self.subscriber.runtime.now()); + // A route's first group says where its live feed is, when its answer did + // not: even one the cache already holds, as an idle copy asked from its + // head gets back. + if !entry.producer.is_live() { + entry.producer.set_live(Some(Position { + group: hdr.sequence, + frame: hdr.frame_start, + })); + } let group_info = group::Info { sequence: hdr.sequence }; // Stats (groups/frames/bytes) are counted in the model as the group // is written, through the tagged `track::Producer`. @@ -813,8 +826,8 @@ impl GroupRecv { Err(err) => return Poll::Ready(Err(err)), }; // The stream may carry only the tail of the group; number the frames - // from where the publisher said they start so a reader splicing - // across routes lines them up. + // from where the publisher said they start so a front continuing the + // group across routes lines them up. group.start_at(hdr.frame_start)?; (group, entry.producer.clone(), entry.timescale, reading) }; @@ -1590,9 +1603,12 @@ mod tests { async fn a_subscribe_end_below_a_received_group() { for version in [Version::Lite05, Version::Lite06, Version::Lite07] { let mut responses = Vec::new(); - lite::SubscribeResponse::Start(lite::SubscribeStart { group: 0 }) - .encode(&mut responses, version) - .unwrap(); + lite::SubscribeResponse::Start(lite::SubscribeStart { + group: 0, + largest: None, + }) + .encode(&mut responses, version) + .unwrap(); lite::SubscribeResponse::End(lite::SubscribeEnd { group: 2, streams: 1 }) .encode(&mut responses, version) .unwrap(); @@ -1693,9 +1709,12 @@ mod tests { fn fin_responses(version: Version, started: bool, clean: bool) -> Vec { let mut responses = Vec::new(); if started { - lite::SubscribeResponse::Start(lite::SubscribeStart { group: 0 }) - .encode(&mut responses, version) - .unwrap(); + lite::SubscribeResponse::Start(lite::SubscribeStart { + group: 0, + largest: None, + }) + .encode(&mut responses, version) + .unwrap(); } if clean { lite::SubscribeResponse::End(lite::SubscribeEnd { group: 0, streams: 0 }) @@ -2017,9 +2036,8 @@ mod tests { } } - /// The `(group, frame)` bounds `resume::slice` produces after a mid-group takeover: a - /// subscriber resuming at frame 3 of group 5, capped by a later boundary in the same - /// group. + /// Mid-group demand: a subscriber resuming at frame 3 of group 5, capped by a later + /// bound in the same group. fn mid_group_demand() -> Subscription { Subscription::default() .with_start(Position { group: 5, frame: 3 }) @@ -2031,8 +2049,8 @@ mod tests { /// /// The codec rejects a frame bound such a peer cannot carry, so passing the demand /// through unchanged fails the SUBSCRIBE and hands the track back. The origin then - /// re-splices the same route indefinitely, since the splice itself keeps succeeding - /// and the retry budget never trips. + /// asks the same route again indefinitely, since asking keeps succeeding and the + /// retry budget never trips. #[tokio::test] async fn frame_bounds_widen_for_an_older_peer() { let mut h = Harness::new(Version::Lite05); @@ -2061,32 +2079,36 @@ mod tests { assert_eq!((msg.start_frame, msg.end_frame), (0, None)); } - /// The same demand on a lite-06 peer keeps its frame offsets, so the widening is - /// version-gated rather than unconditional. + /// A lite-07 SUBSCRIBE keeps a mid-group start: its answer carries the largest position, + /// which says whether what the subscriber holds is current. Lite-06's answer does not, + /// so it asks from the head of the group instead, and the first frame says where the + /// live feed is; its end keeps the frame bound. #[tokio::test] - async fn frame_bounds_survive_on_a_lite06_peer() { - let mut h = Harness::new(Version::Lite06); - let mut sub = Sub::None; - - h.serve - .handle_subscription( - &mut h.producer, - &mut sub, - Some(mid_group_demand()), - true, - Some(Timescale::default()), - ) - .await - .unwrap(); + async fn a_mid_group_start_survives_only_where_the_answer_has_the_largest() { + for (version, start_frame) in [(Version::Lite07, 3), (Version::Lite06, 0)] { + let mut h = Harness::new(version); + let mut sub = Sub::None; + + h.serve + .handle_subscription( + &mut h.producer, + &mut sub, + Some(mid_group_demand()), + true, + Some(Timescale::default()), + ) + .await + .unwrap(); - let wire = h.wire(); - let mut wire = wire.as_slice(); - assert_eq!( - lite::ControlType::decode(&mut wire, Version::Lite06).unwrap(), - lite::ControlType::Subscribe - ); - let msg = lite::Subscribe::decode(&mut wire, Version::Lite06).unwrap(); - assert_eq!((msg.start_frame, msg.end_frame), (3, Some(7))); + let wire = h.wire(); + let mut wire = wire.as_slice(); + assert_eq!( + lite::ControlType::decode(&mut wire, version).unwrap(), + lite::ControlType::Subscribe + ); + let msg = lite::Subscribe::decode(&mut wire, version).unwrap(); + assert_eq!((msg.start_frame, msg.end_frame), (start_frame, Some(7)), "{version:?}"); + } } /// The model's exclusive end maps back to the wire's inclusive pair. @@ -3311,15 +3333,15 @@ impl std::task::Wake for RouteWake { enum ServeEnd { /// The upstream FIN'd: the track is over for good. Finished, - /// The route or session failed: abort the track so the origin re-splices it - /// from another source. + /// The route or session failed: abort the track so the origin serves it from + /// another source. GiveBack(Error), /// No consumers or in-flight fetches remain; the owner must commit the idle abort. Idle, } /// Serves one requested track for a relay: owns this session's copy of the -/// track (spliced into the origin's logical track), driving the single upstream +/// track (pumped into the origin's logical track), driving the single upstream /// subscription (opened lazily on the first downstream subscriber, canceled when /// the last one leaves) concurrently with any number of one-shot fetches. #[derive(Clone)] @@ -3330,7 +3352,15 @@ struct TrackServe { } impl TrackServe { + /// The mid-group start a peer without frame bounds can't be asked for, recorded so + /// the frames below it are dropped when its whole group arrives. fn widen_frame_bounds(&self, subscription: &mut Subscription) { + // An answer that does not carry the largest position cannot vouch for what a + // subscriber holds, so ask from the head of its group instead: the first frame then + // arrives at once and says where the live feed is. + if !self.subscriber.version.has_largest() { + subscription.start = subscription.start.map(|start| Position::group(start.group)); + } if self.subscriber.version.has_frame_bounds() { return; } @@ -3371,9 +3401,7 @@ impl TrackServe { // An empty half-open range asks for nothing, and the wire cannot say that: its // bounds are inclusive, so the nearest encoding either hands back the position // the caller excluded or inverts the range outright once the two bounds meet. No - // demand at all is the faithful translation. `resume::slice` reaches this on its - // own: a subscriber resuming exactly at a segment's cap owes that segment nothing. - // An absent start is the live edge, wherever that lands, so the only end that is + // demand at all is the faithful translation. An absent start is the live edge, wherever that lands, so the only end that is // certainly empty is the very first position, which is what `Position::default()` // stands in for. let pref = pref.filter(|sub| sub.end.is_none_or(|end| end > sub.start.unwrap_or_default())); @@ -3383,6 +3411,12 @@ impl TrackServe { self.widen_frame_bounds(&mut subscription); match sub { Sub::None => { + // An idle copy is asked from the head of the newest group it cached when + // the subscriber has no start of its own: a quiet track (a catalog) sends + // that group again, which says the cache is current. + if subscription.start.is_none() { + subscription.start = producer.idle_newest().map(Position::group); + } // Open an upstream SUBSCRIBE for the first subscriber. Ok(Begin::Establish(self.prepare_establish( producer, @@ -3433,6 +3467,8 @@ impl TrackServe { let _ = active.stream.writer.finish(); tracing::info!(track = %self.name, "subscribe canceled (idle)"); *sub = Sub::None; + // The copy is still held: what it cached goes stale from here. + producer.set_idle(); } Ok(Begin::None) } @@ -3448,11 +3484,7 @@ impl TrackServe { /// in the map yet is dropped, stalling the track forever). The caller /// deregisters `id` if the establish fails. /// - /// The subscription's bounds come straight from the demand aggregate. After a - /// route change a takeover boundary caps the *previous* segment's `end_group`; - /// the segment resuming the track keeps whatever start the subscribers asked for, - /// so a live-edge subscriber gets the live edge upstream rather than a replay of - /// the outage. + /// The subscription's bounds come straight from the demand aggregate. fn prepare_establish( &self, producer: &mut track::Producer, @@ -3889,15 +3921,15 @@ impl TrackInfoFetch { } } -/// The serve loop proper: owns this session's copy of the track (spliced into +/// The serve loop proper: owns this session's copy of the track (pumped into /// the origin's logical track), driving the single upstream subscription /// (opened lazily on the first downstream subscriber, canceled when the last /// one leaves) concurrently with any number of one-shot fetches. struct ServeLoop { - /// This session's copy, accepted with the resolved info. The origin splices + /// This session's copy, accepted with the resolved info. The origin pumps /// it into the logical track; demand from the logical subscribers arrives - /// through the producer's aggregate, sliced to this segment's bounds - /// (including the resume floor after a source change). + /// through the producer's aggregate (including the resume floor after a source + /// change). serving: track::Producer, /// Watches `serving`'s subscribers, to release the copy once nobody reads it. demand: track::Demand, @@ -3913,6 +3945,9 @@ struct ServeLoop { supports_fetch: bool, timescale: Option, mode: ServeMode, + /// Armed while nobody holds the copy: it stays, cache and all, for a returning + /// reader until this fires. + linger: crate::runtime::Deadline, } // A state machine's enum is its storage: one transient instance per stream, so the @@ -3959,6 +3994,7 @@ impl ServeLoop { supports_fetch: serve.subscriber.version.has_track_stream(), timescale, mode: ServeMode::Select, + linger: crate::runtime::Deadline::new(&serve.subscriber.runtime), } } @@ -4062,13 +4098,22 @@ impl ServeLoop { // (2) In-flight fetches; completions just retire. let _ = self.fetches.poll(waiter); - // (3) Nobody reads this copy anymore: the origin dropped its source - // copy when demand ended, so drop it instead of holding the track - // state (and its TRACK_INFO) for a reader that may never return. - // In-flight fetches keep it alive: work already accepted still - // gets finished. + // (3) Nobody holds this copy anymore. Its upstream subscription already + // went with the last subscriber, so it lingers, cache and all, for a + // reader or fetch that asks again soon, then is dropped. In-flight + // fetches keep it alive: work already accepted still gets finished. if self.fetches.is_empty() && self.demand.poll_unused(waiter).is_ready() { - return Poll::Ready(ServeEnd::Idle); + if self.linger.deadline().is_none() { + let now = serve.subscriber.runtime.now(); + self.linger.set(now.checked_add(track::IDLE_LINGER)); + } + if self.linger.poll(waiter).is_ready() { + return Poll::Ready(ServeEnd::Idle); + } + // A reader returning restarts the countdown when it next leaves. + let _ = self.demand.poll_used(waiter); + } else { + self.linger.set(None); } // (4) The upstream subscribe stream closed, or carried a START/END/DROP. @@ -4110,9 +4155,13 @@ impl ServeLoop { // SUBSCRIBE_START names the first group this feed serves: // the publisher skipped everything below it (e.g. it could // not serve the requested frame). Record it as a drop - // signal, so a spliced reader waiting on a skipped group + // signal, so a reader waiting on a skipped group // fails over instead of stalling on a live route. lite::SubscribeResponse::Start(start) => { + // Where the live feed is, on versions whose answer says. + if serve.subscriber.version.has_largest() { + self.serving.set_live(start.largest); + } // A START describes the demand the SUBSCRIBE carried. // It applies only while the current start still matches // that demand (updates get no fresh START, so an update diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 33a1d52bd02..5a0115c0257 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -39,6 +39,18 @@ impl Version { } } + /// Whether SUBSCRIBE_OK carries the publisher's largest (group, frame), which a + /// subscriber takes as where the live feed is. Added in lite-07; an earlier answer + /// says nothing about the live edge. + #[allow(clippy::match_like_matches_macro)] + pub(crate) fn has_largest(self) -> bool { + // Match form so future versions default forward (AGENTS.md convention). + match self { + Self::Lite01 | Self::Lite02 | Self::Lite03 | Self::Lite04 | Self::Lite05 | Self::Lite06 => false, + _ => true, + } + } + /// Whether the version has lite-05's dedicated TRACK stream and related stream /// layout changes. /// diff --git a/rs/moq-net/src/model/broadcast.rs b/rs/moq-net/src/model/broadcast.rs index 0acb6631676..e2cd88c3d2f 100644 --- a/rs/moq-net/src/model/broadcast.rs +++ b/rs/moq-net/src/model/broadcast.rs @@ -8,7 +8,6 @@ //! [Info] is the broadcast's static metadata, fixed for its lifetime. use crate::{cache, stats, track}; use std::{ - collections::{HashMap, VecDeque}, sync::Arc, task::{Poll, ready}, }; @@ -90,27 +89,11 @@ struct BroadcastState { // coalescing onto it there). requests: Requests, track::Request>, - // Route-fed mode (a relay/origin "front"): tracks are spliced logical tracks - // joined across per-session tracks. `None` for an ordinary broadcast. - spliced: Option, - // Set once the broadcast ends: `Producer::close()` or the last producer-side // handle dropping. Every lookup after it answers `Unroutable`. closing: bool, } -/// The spliced (route-fed) half of a broadcast: logical tracks that outlive any -/// single session, plus the queue of tracks awaiting a serving route. -#[derive(Default)] -struct SplicedState { - // Logical tracks by name, owned strongly: they live as long as the broadcast - // (the origin's front), not as long as any consumer. - tracks: HashMap, super::resume::Producer>, - - // Names awaiting assignment to a route, in request order. - pending: VecDeque>, -} - impl BroadcastState { /// Insert a track weak handle into the lookup, returning an error if a live /// track already holds the name. A closed entry under the name is reclaimed. @@ -137,12 +120,8 @@ impl BroadcastState { } } - /// Live demand: a subscribed spliced track (route-fed broadcast), or a - /// pending request / consumed track (ordinary broadcast). See [`Demand`]. + /// Live demand: a pending request or a consumed track. See [`Demand`]. fn is_used(&self) -> bool { - if let Some(spliced) = &self.spliced { - return spliced.tracks.values().any(|track| track.is_used()); - } !self.requests.is_empty() || self.tracks.iter().any(|track| track.is_used()) } @@ -151,15 +130,6 @@ impl BroadcastState { /// state, so a watcher registered here alone would miss the edge. `want` /// picks the direction; each channel only arms while its side is unmet. fn register_demand(&self, waiter: &kio::Waiter, want: bool) { - if let Some(spliced) = &self.spliced { - for track in spliced.tracks.values() { - let _ = match want { - true => track.poll_used(waiter), - false => track.poll_unused(waiter), - }; - } - return; - } for track in self.tracks.iter() { match want { true => track.poll_used(waiter), @@ -255,24 +225,6 @@ impl Producer { self.alive.unannounce(); } - /// Create a route-fed (spliced) broadcast: consumer track lookups mint logical - /// tracks that are spliced across per-session tracks, queued for a route to - /// serve. Used by the origin for broadcasts reached over the network. - pub(crate) fn new_spliced(info: Info) -> Self { - let state = kio::Shared::new(BroadcastState { - spliced: Some(SplicedState::default()), - ..Default::default() - }); - Self { - info: Arc::new(info), - alive: Alive::new(state.clone()), - state, - // The origin-owned spliced broadcast stays untagged: egress attribution is - // applied when a tagged `origin::Consumer` hands the consumer out. - stats: stats::Scope::default(), - } - } - /// The broadcast's static metadata, fixed when it was created. pub fn info(&self) -> &Info { &self.info @@ -381,64 +333,10 @@ impl Producer { ) } - /// Poll for the next spliced track awaiting a serving route, returning its name - /// and logical producer. Route-fed broadcasts only. - pub(crate) fn poll_spliced_assigned(&self, waiter: &kio::Waiter) -> Poll<(Arc, super::resume::Producer)> { - let mut state = ready!(self.state.poll(waiter, |state| { - match &state.spliced { - Some(spliced) if !spliced.pending.is_empty() => Poll::Ready(()), - _ => Poll::Pending, - } - })); - - let spliced = state.spliced.as_mut().expect("predicate guaranteed spliced"); - let name = spliced.pending.pop_front().expect("predicate guaranteed a request"); - let producer = spliced.tracks.get(&name).expect("pending name without a track").clone(); - Poll::Ready((name, producer)) - } - - /// Let go of every spliced track, aborting with `err` the ones never handed - /// out by [`Self::poll_spliced_assigned`]. Called when the broadcast ends: - /// whoever took the others decides how they end. - pub(crate) fn release_spliced(&self, err: Error) { - let mut state = self.state.lock(); - if let Some(spliced) = state.spliced.as_mut() { - for name in std::mem::take(&mut spliced.pending) { - if let Some(producer) = spliced.tracks.get_mut(&name) { - let _ = producer.abort(err.clone()); - } - } - spliced.tracks.clear(); - } - } - - /// Remove the spliced track `producer` from under `name`, unless it has a reader. - /// - /// Returns false only when a reader holds it: lookups hand out readers under the - /// same lock, so one arriving after the caller decided is never cut off. Anything - /// else (removed, or already replaced under the name) is gone as far as the caller - /// is concerned. - pub(crate) fn forget_spliced(&self, name: &str, producer: &super::resume::Producer) -> bool { - let mut state = self.state.lock(); - let Some(spliced) = state.spliced.as_mut() else { - return true; - }; - match spliced.tracks.get(name) { - Some(current) if current.is_clone(producer) => { - if current.is_used() { - return false; - } - spliced.tracks.remove(name); - true - } - _ => true, - } - } - /// Create a consumer of this one publisher's broadcast. /// /// A view of this broadcast object, not of its path: a new publisher at the same - /// path is never spliced into it, so it ends when this broadcast does. Go through + /// path never feeds it, so it ends when this broadcast does. Go through /// an origin for a consumer that should not care which publisher serves the path. pub fn consume(&self) -> Consumer { Consumer { @@ -783,41 +681,6 @@ impl Consumer { return Err(Error::Unroutable); } - // A route-fed broadcast mints spliced logical tracks: they outlive any - // session, and a route is asked (via the pending queue) to start serving. - if let Some(spliced) = state.spliced.as_mut() { - // An aborted logical track is a verdict from the sources attached at - // the time, not a property of the name: a publisher that had not yet - // created the track may have it now. Drop it so this request reaches a - // source again, exactly as the plain lookup below reclaims a closed - // entry. A *finished* one stays, since its cache is still readable, - // until the front forgets it after going unread for its linger. - // - // So a name, once finished, is never spliced onto again: a publisher - // that finishes a track and publishes it again is serving new content, - // not resuming this one, and a subscriber has to re-read the catalog and - // re-initialize rather than be spliced onto it. Resuming the same - // content across routes is the transparent case, and that is what - // `resume::Producer` already does. Publish new content under a new - // name. - if spliced.tracks.get(name).is_some_and(|track| track.is_aborted()) { - spliced.tracks.remove(name); - } - if let Some(producer) = spliced.tracks.get(name) { - return Ok(track::Consumer::spliced( - name.into(), - self.info.clone(), - producer.consume(), - )); - } - let name: Arc = name.into(); - let producer = super::resume::Producer::new(); - let consumer = producer.consume(); - spliced.tracks.insert(name.clone(), producer); - spliced.pending.push_back(name.clone()); - return Ok(track::Consumer::spliced(name, self.info.clone(), consumer)); - } - // Reuse a live producer if one is already publishing the track. `get` drops a // closed entry and returns `None`, so we fall through to a fresh request. if let Some(weak) = state.tracks.get(name) { @@ -951,8 +814,7 @@ impl super::WeakEntry for WeakConsumer { /// /// Obtained from [`Producer::demand`] or [`Consumer::demand`]; the broadcast-level sibling of /// [`track::Demand`](crate::track::Demand). Demand means live interest in the -/// broadcast's content: a subscribed spliced track on a route-fed broadcast, or -/// a pending track request / a consumed track on an ordinary one. A publisher +/// broadcast's content: a pending track request or a consumed track. A publisher /// uses it to run expensive work only while someone is watching, and routing /// uses it to advertise a warm copy at zero cost. /// @@ -1129,41 +991,13 @@ mod test { assert!(matches!(demand.unused().await, Err(Error::Dropped))); } - /// Demand on a spliced (route-fed) broadcast follows the logical tracks' - /// consumers, which is what flips a relay's advertised cost. - #[tokio::test] - async fn demand_spliced() { - tokio::time::pause(); - - let producer = Producer::new_spliced(Info::new()); - let consumer = producer.consume(); - let demand = producer.demand(); - let watched = consumer.demand(); - - assert!(!demand.is_used()); - assert!(!watched.is_used()); - let track = consumer.track("video").unwrap(); - assert!(demand.is_used()); - assert!(watched.is_used()); - - // Dropping the only consumer wakes a parked `unused`, even though the - // logical track itself stays cached in the broadcast. - let (unused, ()) = tokio::join!(expect(watched.unused()), async { drop(track) }); - unused.unwrap(); - assert!(!demand.is_used()); - assert!(!watched.is_used()); - - // A repeat consumer for the cached track counts again. - let _track = consumer.track("video").unwrap(); - assert!(demand.is_used()); - } - /// A consumer demand handle distinguishes lost demand from a dropped producer. #[tokio::test] async fn consumer_demand_reports_dropped_producer() { - let producer = Producer::new_spliced(Info::new()); + let producer = Info::new().produce(); let consumer = producer.consume(); let watched = consumer.demand(); + let _video = producer.create_track("video", None).unwrap(); let track = consumer.track("video").unwrap(); assert!(watched.is_used()); diff --git a/rs/moq-net/src/model/frame.rs b/rs/moq-net/src/model/frame.rs index da7f4d8cd5b..ca8f1e87027 100644 --- a/rs/moq-net/src/model/frame.rs +++ b/rs/moq-net/src/model/frame.rs @@ -676,6 +676,9 @@ pub struct Consumer { // The parent subscription can expire after this frame handle is returned. expiry: Option, expired: bool, + // Read from a front's logical track: carries the read across route changes, with + // this frame's index in its group. Boxed: it is the rare case. + recover: Option>, } impl std::ops::Deref for Consumer { @@ -696,6 +699,69 @@ impl Consumer { stats: stats::Meter::default(), expiry: None, expired: false, + recover: None, + } + } + + /// Carry this frame across a front's route changes; see [`super::resume`]. + pub(crate) fn with_recover(mut self, recover: super::resume::Recover, index: u64) -> Self { + self.recover = Some(Box::new((recover, index))); + self + } + + /// Run `read`, and once this copy fails with its route, or stalls while a newer route + /// serves, continue from the serving route's copy of the same frame, past the bytes + /// already read. + fn poll_resumed( + &mut self, + waiter: &kio::Waiter, + mut read: impl FnMut(&mut Self, &kio::Waiter) -> Poll>, + ) -> Poll> { + loop { + let res = read(self, waiter); + if self.expired { + return res; + } + let Some(recover) = self.recover.as_mut() else { + return res; + }; + let failed = match &res { + Poll::Ready(Ok(_)) => return res, + Poll::Ready(Err(err)) => Some(err.clone()), + Poll::Pending => None, + }; + let (recover, index) = &mut **recover; + if !recover.wants(failed.as_ref(), waiter) { + return res; + } + let mut replacement = match recover.poll(*index, failed.as_ref(), waiter) { + Poll::Ready(Ok(group)) => group, + Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), + Poll::Pending => return Poll::Pending, + }; + // A cached group can precede this frame's header. Keep the old generation + // until the replacement frame is actually available. + if let Some(expiry) = &mut self.expiry { + expiry.policy = expiry.policy.for_track(&replacement.copy); + } + let frame = match replacement.group.poll_next_frame(waiter) { + Poll::Ready(Ok(Some(frame))) => frame, + Poll::Ready(Ok(None)) => return Poll::Ready(Err(Error::WrongSize)), + Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), + Poll::Pending => { + return match self.poll_expired(waiter) { + true => Poll::Ready(Err(Error::Old)), + false => Poll::Pending, + }; + } + }; + // Same name, same content: a different size is the routes disagreeing. + if frame.info.size != self.info.size { + return Poll::Ready(Err(Error::ProtocolViolation)); + } + recover.adopt(&replacement); + self.state = frame.state; + self.source = frame.source; } } @@ -730,7 +796,8 @@ impl Consumer { let Some(expiry) = &self.expiry else { return false; }; - if !expiry.policy.is_expired(waiter) { + let budget = self.recover.as_ref().and_then(|recover| recover.0.poll_budget(waiter)); + if !expiry.policy.is_expired(budget, waiter) { return false; } @@ -749,6 +816,13 @@ impl Consumer { /// /// Returns `None` once the frame is finished and all bytes have been consumed. pub fn poll_read_chunk(&mut self, waiter: &kio::Waiter) -> Poll>> { + if self.recover.is_none() { + return self.poll_read_chunk_once(waiter); + } + self.poll_resumed(waiter, Self::poll_read_chunk_once) + } + + fn poll_read_chunk_once(&mut self, waiter: &kio::Waiter) -> Poll>> { if self.expired { return Poll::Ready(Err(Error::Old)); } @@ -806,6 +880,13 @@ impl Consumer { /// Poll for all remaining bytes, resolving once the frame is finished. pub fn poll_read_all(&mut self, waiter: &kio::Waiter) -> Poll> { + if self.recover.is_none() { + return self.poll_read_all_once(waiter); + } + self.poll_resumed(waiter, Self::poll_read_all_once) + } + + fn poll_read_all_once(&mut self, waiter: &kio::Waiter) -> Poll> { if self.expired { return Poll::Ready(Err(Error::Old)); } diff --git a/rs/moq-net/src/model/front.rs b/rs/moq-net/src/model/front.rs index dce21083f66..b7b00c19bdb 100644 --- a/rs/moq-net/src/model/front.rs +++ b/rs/moq-net/src/model/front.rs @@ -1,14 +1,19 @@ //! The origin's failover as a state machine. //! -//! A [`Front`] is one path served through a spliced broadcast: it picks the -//! source it serves from and splices each logical track from that source's -//! copy, re-splicing when the source is replaced and refusing what a source -//! will not serve. Everything here is plain data: [`Front::step`] takes one +//! A [`Front`] is one path served through one broadcast: it picks the source it +//! serves from and splices each logical track from that source's copy (its readers +//! read the copy; see `super::resume`), re-splicing when the source is replaced and +//! refusing what a source will not serve. Everything here is plain data: [`Front::step`] takes one //! [`Event`] and returns the [`Action`]s to perform, without locks or waiters. //! The origin's driver task feeds events from the world (the route table, the //! sources, the tracks, the clock) and executes the actions; see //! `origin::run_front`. //! +//! A path names one broadcast, whoever serves it, so any route covering the path +//! may take over from another and its tracks resume where they stopped. A +//! publisher reusing a name for different content is a bug; it publishes under a +//! new name (an epoch) instead. +//! //! Sources and tracks are named by ids and names, never handles, so a //! transition can be checked in a unit test by comparing the actions it emits. @@ -18,14 +23,13 @@ use std::{ time::Duration, }; -use crate::{Error, Hop, runtime::Instant, track}; +use crate::{Error, runtime::Instant, track}; -/// A route the table selected for the front: the entry id, the endpoint that -/// originated it, and whether it is a broadcast published on this origin. +/// A route the table selected for the front: the entry id, and whether it is a +/// broadcast published on this origin. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(super) struct Candidate { pub route: u64, - pub first: Option, pub local: bool, } @@ -42,7 +46,7 @@ pub(super) struct Refusal { /// What happened in the world. #[derive(Clone, Debug)] pub(super) enum Event { - /// The routes covering the path were re-read: the best qualifying route, if + /// The routes covering the path were re-read: the best route, if /// any, and whether the serving source has begun closing. Selected { best: Option, @@ -53,7 +57,7 @@ pub(super) enum Event { Resolved { route: u64, result: Result }, /// A source closed: it will never serve again. SourceClosed { source: u64 }, - /// The spliced broadcast handed out a new logical track to serve. It has no + /// The front's broadcast handed out a new logical track to serve. It has no /// reader until [`Event::Used`] says so. TrackAssigned { track: Arc, now: Instant }, /// A source answered a track query: its copy's metadata, or a refusal. @@ -91,7 +95,7 @@ pub(super) enum Event { /// What the driver does in the world. #[derive(Clone, Debug)] pub(super) enum Action { - /// Ask the route table for the best qualifying route and feed it back as + /// Ask the route table for the best route and feed it back as /// [`Event::Selected`]. Reselect, /// Request the path through `route`; feed the outcome back as @@ -104,14 +108,14 @@ pub(super) enum Action { /// Ask `source` for its copy of `track`; feed the answer back as /// [`Event::TrackInfo`]. Query { track: Arc, source: u64 }, - /// Splice `source`'s copy of `track` in, resuming where the segments stop. + /// Splice `source`'s copy of `track` in: each reader resumes on it where it stopped. Splice { track: Arc, source: u64 }, - /// Drop the source copy of `track` but keep the delivered groups spliced, - /// so resume stays seamless while nobody reads. + /// Drop the source copy of `track`: nobody reads it, and a returning reader gets a + /// fresh one. Park { track: Arc }, /// Remove `track` from the broadcast and drop everything behind it, unless a reader - /// arrived meanwhile: it went unread for the linger, or the source is local and - /// keeps its own cache. Feed back [`Event::Forgotten`] once it is gone. + /// arrived meanwhile: it went unread for the linger, or the source is local. + /// Feed back [`Event::Forgotten`] once it is gone. Forget { track: Arc }, /// The logical track completed. Finish { track: Arc }, @@ -125,64 +129,17 @@ pub(super) enum Action { End { err: Error }, } -/// The content identity of a front: who its first source came from. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub(super) enum Identity { - /// No source has attached yet: the first request may resolve through any - /// covering route, and whoever serves it fixes the identity. - Undetermined, - /// The first source was a broadcast published on this origin. A newer local - /// publisher takes over while the incumbent is live; once the incumbent is - /// closing the front ends instead, so a newcomer gets a fresh broadcast - /// rather than being spliced into one that is over. - Local, - /// The serving route's first hop was absent or [`Hop::UNKNOWN`], which - /// identifies nobody: the front cannot resume through any other route, so - /// its source ending, `route` leaving the table, or `route` gaining a first - /// hop ends it. - Anonymous { route: u64 }, - /// The first hop of the serving route. Routes sharing it are the same - /// origin reached another way and safe to resume through. - Publisher(Hop), -} - -/// Which routes qualify for a front's (re)selection. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub(super) enum Pin { - /// Any served route. - Any, - /// Only broadcasts published on this origin. - Local, - /// Only routes originated by this first hop. - Publisher(Hop), - /// Only this route, while its publisher stays unknown: the front never fails - /// over, and an update naming a publisher ends it. - Route(u64), -} - -impl Identity { - fn pin(self) -> Pin { - match self { - Self::Undetermined => Pin::Any, - Self::Local => Pin::Local, - Self::Anonymous { route } => Pin::Route(route), - Self::Publisher(hop) => Pin::Publisher(hop), - } - } -} - /// Where a track stands with respect to its source copy. #[derive(Clone, Debug, PartialEq, Eq)] enum TrackState { - /// No copy: nothing spliced, or the segment stays spliced from a source - /// that left. + /// No copy: nothing spliced, or the source that served it left. Idle, /// A source was asked for its copy. Querying { source: u64 }, /// A source's copy is spliced in and being served. Spliced { source: u64 }, - /// Nobody reads it: the copy was dropped, and whatever the track delivered - /// stays spliced until the linger expires and the track is forgotten. + /// Nobody reads it: the copy was dropped, and the track stays until the linger + /// expires and it is forgotten. Parked { since: Instant }, } @@ -196,6 +153,10 @@ struct Track { refusal: Option, /// Whether anyone reads the track: a copy is only spliced in for a reader. used: bool, + /// A detached source whose copy still feeds the track until the serving source's + /// replaces it: a route being beaten keeps serving until then, and keeps serving if + /// the newcomer refuses the track. + draining: Option, /// The track finished or aborted: nothing is spliced again, and it stays only /// until it goes unread for the linger. ended: bool, @@ -204,14 +165,13 @@ struct Track { /// One front's state; see the module docs. #[derive(Clone, Debug)] pub(super) struct Front { - identity: Identity, /// The attached source and the route that produced it. - serving: Option<(u64, u64)>, + serving: Option<(u64, Candidate)>, /// Whether the serving source has begun closing, as of the last event that /// said. A closing source is never asked for anything again. serving_closing: bool, /// The route an upstream request is in flight through. - upstream: Option, + upstream: Option, /// Routes excluded from selection: they refused the path while another /// source was serving, or their source ended while still advertised. refused: HashSet, @@ -235,7 +195,6 @@ impl Front { /// track stays before it is forgotten. pub(super) fn new(linger: Duration) -> Self { Self { - identity: Identity::Undetermined, serving: None, serving_closing: false, upstream: None, @@ -250,11 +209,6 @@ impl Front { } } - /// Which routes qualify for the next selection. - pub(super) fn pin(&self) -> Pin { - self.identity.pin() - } - /// The routes excluded from selection; the driver skips them. pub(super) fn refused_routes(&self) -> &HashSet { &self.refused @@ -299,6 +253,7 @@ impl Front { refusal: None, used: false, ended: false, + draining: None, }, ); } @@ -337,32 +292,27 @@ impl Front { if self.serving.is_some() { self.serving_closing = serving_closing; } - let serving_route = self.serving.map(|(_, route)| route); + let serving_route = self.serving.map(|(_, serving)| serving.route); match best { - // The serving route is still the best qualifying one. + // The serving route is still the best one. Some(candidate) if Some(candidate.route) == serving_route => { self.upstream = None; } // Already asking through it. - Some(candidate) if self.upstream == Some(candidate.route) => {} - // A local newcomer must not be spliced into a broadcast whose - // publisher is on the way out: end, and a fresh front serves it. - Some(candidate) if candidate.local && self.identity == Identity::Local && serving_closing => { - self.end(Error::Dropped, actions); - } - // A better (or replacement) qualifying route: request through it. + Some(candidate) if self.upstream.is_some_and(|upstream| upstream.route == candidate.route) => {} + // A better (or replacement) route: request through it. Some(candidate) => { - self.upstream = Some(candidate.route); + self.upstream = Some(candidate); actions.push(Action::Request { route: candidate.route }); } - // Nothing qualifies: the front is over, even with a live source. Its - // route left the table and nothing with the same content replaced it, - // so it serves nobody new; a later announcement gets a fresh front. + // Nothing serves the path: the front is over, even with a live source. Its + // route left the table and nothing replaced it, so it serves nobody new; + // a later announcement gets a fresh front. None => { self.upstream = None; - let err = match self.identity { - Identity::Undetermined => self.last_err.take().unwrap_or(Error::Unroutable), - _ => self.last_err.take().unwrap_or(Error::Dropped), + let err = match self.resolved { + true => Error::Dropped, + false => self.last_err.take().unwrap_or(Error::Unroutable), }; self.end(err, actions); } @@ -370,14 +320,18 @@ impl Front { } fn resolved(&mut self, route: u64, result: Result, actions: &mut Vec) { - match result { + let Some(upstream) = self.upstream.filter(|upstream| upstream.route == route) else { // A source for a request we no longer want: let it go. - Ok(source) if self.upstream != Some(route) => actions.push(Action::Detach { source }), + if let Ok(source) = result { + actions.push(Action::Detach { source }); + } + return; + }; + match result { Ok(source) => { self.upstream = None; - self.attach(source, route, actions); + self.attach(source, upstream, actions); } - Err(_) if self.upstream != Some(route) => {} // The route retracted before serving: the table already reflects // it, so the next selection retries the survivor. Err(Refusal { standing: false, .. }) => { @@ -385,35 +339,26 @@ impl Front { self.last_err = Some(Error::Unroutable); actions.push(Action::Reselect); } - // An authoritative refusal of the path. It ends a front with no - // other source; a serving front merely skips the refuser. + // An authoritative refusal of the path: skip the refuser. Another route may + // still serve, and with none left the selection ends the front with this + // error. Err(Refusal { err, standing: true }) => { self.upstream = None; - match self.serving { - Some(_) => { - self.refused.insert(route); - self.last_err = Some(err); - actions.push(Action::Reselect); - } - None => self.end(err, actions), - } + self.refused.insert(route); + self.last_err = Some(err); + actions.push(Action::Reselect); } } } - fn attach(&mut self, source: u64, route: u64, actions: &mut Vec) { + fn attach(&mut self, source: u64, candidate: Candidate, actions: &mut Vec) { if let Some((old, _)) = self.serving.take() { actions.push(Action::Detach { source: old }); - // Its copies are gone with it; the segments they delivered stay - // spliced and the replacement resumes past them. - for track in self.tracks.values_mut() { - if matches!(track.state, TrackState::Querying { source: s } | TrackState::Spliced { source: s } if s == old) - { - track.state = TrackState::Idle; - } - } } - self.serving = Some((source, route)); + // Every copy is from an older source, including one a track still drains from a + // source the previous one replaced. + self.drain_copies(); + self.serving = Some((source, candidate)); self.serving_closing = false; if !self.resolved { self.resolved = true; @@ -431,46 +376,39 @@ impl Front { } } - /// Fix the identity from the first source's candidate. Called by the driver - /// with the candidate it requested through, before the source resolves. - pub(super) fn identify(&mut self, candidate: Candidate) { - if self.identity != Identity::Undetermined { - return; - } - self.identity = match (candidate.local, candidate.first) { - (true, _) => Identity::Local, - (false, Some(hop)) if hop != Hop::UNKNOWN => Identity::Publisher(hop), - (false, _) => Identity::Anonymous { route: candidate.route }, - }; - } - fn source_closed(&mut self, source: u64, actions: &mut Vec) { - let Some((serving, route)) = self.serving else { + let Some((serving, candidate)) = self.serving else { return; }; if serving != source { return; } // A standing route can outlive the source it produced. Asking it again - // would re-request the broadcast that just ended; another route to the - // same publisher may still resume it. - self.refused.insert(route); + // would re-request the broadcast that just ended; another route may still + // resume it. + self.refused.insert(candidate.route); self.serving = None; self.serving_closing = false; actions.push(Action::Detach { source }); + self.drain_copies(); + self.last_err = Some(Error::Dropped); + actions.push(Action::Reselect); + } + + /// The serving source is going: a spliced copy keeps feeding its track (readers read it + /// until a replacement's is spliced, or it runs out), and one still being asked for is + /// dropped. + fn drain_copies(&mut self) { for track in self.tracks.values_mut() { - if matches!(track.state, TrackState::Querying { source: s } | TrackState::Spliced { source: s } if s == source) - { - track.state = TrackState::Idle; + match track.state { + TrackState::Spliced { source } => { + track.draining = Some(source); + track.state = TrackState::Idle; + } + TrackState::Querying { .. } => track.state = TrackState::Idle, + TrackState::Idle | TrackState::Parked { .. } => {} } } - self.last_err = Some(Error::Dropped); - match self.identity { - // A local publisher ending ends its broadcast; a newcomer at the - // path gets a fresh one. An anonymous source can never be resumed. - Identity::Local | Identity::Anonymous { .. } | Identity::Undetermined => self.end(Error::Dropped, actions), - Identity::Publisher(_) => actions.push(Action::Reselect), - } } fn track_info( @@ -508,6 +446,7 @@ impl Front { }; match verdict { Ok(()) => { + track.draining = None; track.state = TrackState::Spliced { source }; actions.push(Action::Splice { track: name.clone(), @@ -530,7 +469,22 @@ impl Front { let Some(track) = self.tracks.get_mut(&name) else { return; }; - if track.state != (TrackState::Spliced { source }) { + let spliced = track.state == (TrackState::Spliced { source }); + if track.draining == Some(source) { + track.draining = None; + // A path is one broadcast, so a copy that completed ends the track whoever + // serves it now. One that failed leaves the track to the serving source. + if result.is_ok() && !track.ended { + track.state = TrackState::Idle; + track.ended = true; + actions.push(Action::Finish { track: name }); + return; + } + if !spliced { + return; + } + } + if !spliced { return; } match result { @@ -564,6 +518,12 @@ impl Front { } track.refused.insert(source); track.refusal = Some(err); + // A source being replaced still serves it: keep reading that copy, and the + // refusal stands once it runs out. + if let Some(draining) = track.draining { + track.state = TrackState::Spliced { source: draining }; + return; + } self.redispatch(name, actions); } @@ -607,22 +567,33 @@ impl Front { return; }; track.used = false; + // Nothing reads it, so a copy it still drains goes with the park. + track.draining = None; match track.state { - // A local source keeps its own cache, so a warm copy would only be a staler - // duplicate of it: forget the track outright, and a returning reader - // re-splices the source and reads its cache against the real live edge. - TrackState::Spliced { .. } if self.identity == Identity::Local => { + // A local source is reached again for free: forget the track outright, and a + // returning reader re-splices the source. + TrackState::Spliced { source } + if self + .serving + .is_some_and(|(serving, candidate)| serving == source && candidate.local) => + { track.state = TrackState::Idle; actions.push(Action::Forget { track: name }); } - // Drop the copy so the source goes idle at once; the delivered - // groups stay spliced for the linger. + // Drop the copy so the source goes idle at once; the track lingers. TrackState::Spliced { .. } => { track.state = TrackState::Parked { since: now }; actions.push(Action::Park { track: name }); } TrackState::Parked { .. } => {} - TrackState::Idle | TrackState::Querying { .. } => track.state = TrackState::Parked { since: now }, + // A copy still being asked for is dropped with the park. + TrackState::Idle | TrackState::Querying { .. } => { + track.state = TrackState::Parked { since: now }; + // An ended track has nothing to drop. + if !track.ended { + actions.push(Action::Park { track: name }); + } + } } } @@ -669,24 +640,12 @@ mod tests { const LINGER: Duration = Duration::from_secs(30); - fn hop(id: u64) -> Hop { - Hop::new(id).unwrap() - } - - fn remote(route: u64, first: u64) -> Candidate { - Candidate { - route, - first: Some(hop(first)), - local: false, - } + fn remote(route: u64) -> Candidate { + Candidate { route, local: false } } fn local(route: u64) -> Candidate { - Candidate { - route, - first: None, - local: true, - } + Candidate { route, local: true } } fn info() -> track::Info { @@ -713,7 +672,6 @@ mod tests { }), &[Action::Request { route: candidate.route }], ); - front.identify(candidate); assert_actions( front.step(Event::Resolved { route: candidate.route, @@ -756,9 +714,8 @@ mod tests { #[test] fn first_source_resolves_and_serves_read_tracks() { - let front = serving(remote(1, 10), 100); - assert_eq!(front.identity, Identity::Publisher(hop(10))); - assert_eq!(front.pin(), Pin::Publisher(hop(10))); + let front = serving(remote(1), 100); + assert_eq!(front.serving(), Some(100)); } #[test] @@ -777,10 +734,10 @@ mod tests { #[test] fn unchanged_selection_is_a_no_op() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); assert_actions( front.step(Event::Selected { - best: Some(remote(1, 10)), + best: Some(remote(1)), serving_closing: false, }), &[], @@ -789,10 +746,10 @@ mod tests { #[test] fn better_route_takes_over_and_resplices() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); assert_actions( front.step(Event::Selected { - best: Some(remote(2, 10)), + best: Some(remote(2)), serving_closing: false, }), &[Action::Request { route: 2 }], @@ -810,20 +767,20 @@ mod tests { }, ], ); - assert_eq!(front.serving, Some((200, 2))); + assert_eq!(front.serving, Some((200, remote(2)))); } #[test] fn a_stale_resolution_is_let_go() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); front.step(Event::Selected { - best: Some(remote(2, 10)), + best: Some(remote(2)), serving_closing: false, }); // The table moved back before the request landed. assert_actions( front.step(Event::Selected { - best: Some(remote(1, 10)), + best: Some(remote(1)), serving_closing: false, }), &[], @@ -838,8 +795,8 @@ mod tests { } #[test] - fn dead_source_reselects_through_the_same_publisher() { - let mut front = serving(remote(1, 10), 100); + fn dead_source_reselects_another_route() { + let mut front = serving(remote(1), 100); assert_actions( front.step(Event::SourceClosed { source: 100 }), &[Action::Detach { source: 100 }, Action::Reselect], @@ -847,7 +804,7 @@ mod tests { assert!(front.refused_routes().contains(&1)); assert_actions( front.step(Event::Selected { - best: Some(remote(3, 10)), + best: Some(remote(3)), serving_closing: false, }), &[Action::Request { route: 3 }], @@ -878,23 +835,10 @@ mod tests { ); } - /// An anonymous front can only ever serve from the route it started on, and - /// that route standing keeps it serving. #[test] - fn anonymous_front_keeps_its_standing_route() { - let candidate = Candidate { - route: 1, - first: Some(Hop::UNKNOWN), - local: false, - }; - let mut front = serving(candidate, 100); - assert_actions( - front.step(Event::Selected { - best: Some(candidate), - serving_closing: false, - }), - &[], - ); + fn dead_source_with_no_replacement_ends() { + let mut front = serving(remote(1), 100); + front.step(Event::SourceClosed { source: 100 }); assert_actions( front.step(Event::Selected { best: None, @@ -904,58 +848,69 @@ mod tests { ); } + /// A refusal skips the refuser rather than ending the front: another route to the + /// same content may serve. With none left, the front ends with the refusal. #[test] - fn dead_source_with_no_replacement_ends() { - let mut front = serving(remote(1, 10), 100); - front.step(Event::SourceClosed { source: 100 }); + fn standing_refusal_tries_the_next_route_then_ends() { + let mut front = Front::new(LINGER); + front.step(Event::Selected { + best: Some(remote(1)), + serving_closing: false, + }); + assert_actions( + front.step(Event::Resolved { + route: 1, + result: Err(Refusal { + err: Error::NotFound, + standing: true, + }), + }), + &[Action::Reselect], + ); + assert_eq!(front.refused_routes(), &HashSet::from([1])); assert_actions( front.step(Event::Selected { best: None, serving_closing: false, }), - &[Action::End { err: Error::Dropped }], - ); - } - - #[test] - fn anonymous_source_never_resumes() { - let candidate = Candidate { - route: 1, - first: Some(Hop::UNKNOWN), - local: false, - }; - let mut front = serving(candidate, 100); - assert_eq!(front.pin(), Pin::Route(1)); - assert_actions( - front.step(Event::SourceClosed { source: 100 }), - &[Action::Detach { source: 100 }, Action::End { err: Error::Dropped }], + &[Action::End { err: Error::NotFound }], ); } + /// The audit's F2: the serving source died, and the first replacement refuses the + /// path. A second route is still tried. #[test] - fn standing_refusal_ends_an_unresolved_front() { - let mut front = Front::new(LINGER); + fn standing_refusal_after_the_source_died_tries_another_route() { + let mut front = serving(remote(1), 100); + front.step(Event::SourceClosed { source: 100 }); front.step(Event::Selected { - best: Some(remote(1, 10)), + best: Some(remote(2)), serving_closing: false, }); assert_actions( front.step(Event::Resolved { - route: 1, + route: 2, result: Err(Refusal { err: Error::NotFound, standing: true, }), }), - &[Action::End { err: Error::NotFound }], + &[Action::Reselect], + ); + assert_actions( + front.step(Event::Selected { + best: Some(remote(3)), + serving_closing: false, + }), + &[Action::Request { route: 3 }], ); } #[test] fn standing_refusal_while_serving_skips_the_refuser() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); front.step(Event::Selected { - best: Some(remote(2, 10)), + best: Some(remote(2)), serving_closing: false, }); assert_actions( @@ -969,14 +924,14 @@ mod tests { &[Action::Reselect], ); assert!(front.refused_routes().contains(&2)); - assert_eq!(front.serving, Some((100, 1))); + assert_eq!(front.serving, Some((100, remote(1)))); } #[test] fn retracted_route_is_not_a_refusal() { let mut front = Front::new(LINGER); front.step(Event::Selected { - best: Some(remote(1, 10)), + best: Some(remote(1)), serving_closing: false, }); assert_actions( @@ -995,7 +950,7 @@ mod tests { #[test] fn a_source_refusing_a_track_aborts_it_and_nothing_else() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); front.step(Event::TrackAssigned { track: name("audio"), now: Instant::now(), @@ -1018,7 +973,7 @@ mod tests { #[test] fn a_closing_source_refusal_is_not_a_verdict() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); front.step(Event::TrackAssigned { track: name("audio"), now: Instant::now(), @@ -1037,7 +992,7 @@ mod tests { // The replacement is asked afresh. front.step(Event::SourceClosed { source: 100 }); front.step(Event::Selected { - best: Some(remote(2, 10)), + best: Some(remote(2)), serving_closing: false, }); let actions = front.step(Event::Resolved { @@ -1053,7 +1008,7 @@ mod tests { #[test] fn a_copy_dying_after_delivering_resplices() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); assert_actions( front.step(Event::TrackEnded { track: name("video"), @@ -1071,7 +1026,7 @@ mod tests { #[test] fn a_copy_dying_before_delivering_is_a_refusal() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); assert_actions( front.step(Event::TrackEnded { track: name("video"), @@ -1089,7 +1044,7 @@ mod tests { #[test] fn a_completed_copy_finishes_the_track() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); assert_actions( front.step(Event::TrackEnded { track: name("video"), @@ -1104,9 +1059,9 @@ mod tests { #[test] fn incompatible_copy_is_refused() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); front.step(Event::Selected { - best: Some(remote(2, 10)), + best: Some(remote(2)), serving_closing: false, }); front.step(Event::Resolved { @@ -1117,6 +1072,7 @@ mod tests { max_age: Some(Duration::from_secs(1)), ..track::Info::default() }; + // The source it replaced keeps feeding the track (the audit's F3)... assert_actions( front.step(Event::TrackInfo { track: name("video"), @@ -1124,6 +1080,17 @@ mod tests { closing: false, result: Ok(other), }), + &[], + ); + // ...and the refusal stands once that runs out. + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(Error::Dropped), + delivered: true, + }), &[Action::Abort { track: name("video"), err: Error::Unsupported, @@ -1133,7 +1100,7 @@ mod tests { #[test] fn unread_track_parks_then_is_forgotten_after_the_linger() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); let t0 = Instant::now(); assert_actions( front.step(Event::Unused { @@ -1165,7 +1132,7 @@ mod tests { /// it: the driver feeds `Used` instead of `Forgotten`, and the track re-splices. #[test] fn a_reader_racing_the_forget_keeps_the_track() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); let t0 = Instant::now(); front.step(Event::Unused { track: name("video"), @@ -1188,7 +1155,7 @@ mod tests { /// after the last one, is never spliced again, and is then forgotten. #[test] fn a_finished_track_lingers_then_is_forgotten() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); assert_actions( front.step(Event::TrackEnded { track: name("video"), @@ -1214,7 +1181,7 @@ mod tests { ); // A replacement source does not re-splice it either. front.step(Event::Selected { - best: Some(remote(2, 10)), + best: Some(remote(2)), serving_closing: false, }); assert_actions( @@ -1241,7 +1208,7 @@ mod tests { /// the front. #[test] fn an_aborted_track_lingers_then_is_forgotten() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); front.step(Event::TrackEnded { track: name("video"), source: 100, @@ -1280,7 +1247,7 @@ mod tests { #[test] fn a_returning_reader_cancels_the_linger() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); let t0 = Instant::now(); front.step(Event::Unused { track: name("video"), @@ -1302,7 +1269,7 @@ mod tests { /// forgotten after the linger rather than kept as a stub. #[test] fn unread_track_is_never_spliced() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); let t0 = Instant::now(); front.step(Event::TrackAssigned { track: name("audio"), @@ -1315,44 +1282,42 @@ mod tests { ); } + /// A newer publisher at the path takes over, local or not, and closing or not: the + /// path is the same broadcast. #[test] - fn local_newcomer_takes_over_a_live_incumbent() { - let mut front = serving(local(1), 100); - assert_eq!(front.identity, Identity::Local); - assert_eq!(front.pin(), Pin::Local); - assert_actions( - front.step(Event::Selected { - best: Some(local(2)), - serving_closing: false, - }), - &[Action::Request { route: 2 }], - ); + fn a_newcomer_takes_over_the_incumbent() { + for closing in [false, true] { + let mut front = serving(local(1), 100); + assert_actions( + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: closing, + }), + &[Action::Request { route: 2 }], + ); + } } + /// The serving source ending resumes through whatever else serves the path. #[test] - fn local_newcomer_never_splices_into_a_closing_incumbent() { + fn the_incumbent_ending_resumes_elsewhere() { let mut front = serving(local(1), 100); assert_actions( - front.step(Event::Selected { - best: Some(local(2)), - serving_closing: true, - }), - &[Action::End { err: Error::Dropped }], + front.step(Event::SourceClosed { source: 100 }), + &[Action::Detach { source: 100 }, Action::Reselect], ); - } - - #[test] - fn local_incumbent_ending_ends_the_front() { - let mut front = serving(local(1), 100); assert_actions( - front.step(Event::SourceClosed { source: 100 }), - &[Action::Detach { source: 100 }, Action::End { err: Error::Dropped }], + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }), + &[Action::Request { route: 2 }], ); } #[test] fn teardown_ends_everything() { - let mut front = serving(remote(1, 10), 100); + let mut front = serving(remote(1), 100); assert_actions(front.step(Event::Closed), &[Action::End { err: Error::Dropped }]); } @@ -1363,11 +1328,11 @@ mod tests { let t0 = Instant::now(); let alphabet: Vec = vec![ Event::Selected { - best: Some(remote(1, 10)), + best: Some(remote(1)), serving_closing: false, }, Event::Selected { - best: Some(remote(2, 10)), + best: Some(remote(2)), serving_closing: false, }, Event::Selected { @@ -1476,4 +1441,175 @@ mod tests { walk(&Front::new(LINGER), &alphabet, 5, &mut sequences); assert!(sequences > 100_000); } + + /// The newcomer refused the track, so it stays on the source the newcomer replaced. + /// A third source still asks for it: every copy the front holds is older than it. + #[test] + fn a_new_source_queries_a_track_draining_an_older_one() { + let mut front = serving(remote(1), 100); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Err(Error::NotFound), + }); + front.step(Event::Selected { + best: Some(remote(3)), + serving_closing: false, + }); + assert_actions( + front.step(Event::Resolved { + route: 3, + result: Ok(300), + }), + &[ + Action::Detach { source: 200 }, + Action::Query { + track: name("video"), + source: 300, + }, + ], + ); + } + + /// The newcomer refused the track, so it drained the incumbent's copy. Going unread + /// parks the track and drops that copy, so a later refusal cannot put the track back + /// on a copy nothing feeds any more. + #[test] + fn an_unread_track_lets_go_of_the_copy_it_drains() { + let mut front = serving(remote(1), 100); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Err(Error::NotFound), + }); + let t0 = Instant::now(); + assert_actions( + front.step(Event::Unused { + track: name("video"), + now: t0, + }), + &[ + Action::Park { track: name("video") }, + Action::Arm { at: Some(t0 + LINGER) }, + ], + ); + front.step(Event::Selected { + best: Some(remote(3)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 3, + result: Ok(300), + }); + front.step(Event::Used { track: name("video") }); + let actions = front.step(Event::TrackInfo { + track: name("video"), + source: 300, + closing: false, + result: Err(Error::NotFound), + }); + assert_ne!( + front.tracks[&name("video")].state, + TrackState::Spliced { source: 100 }, + "spliced to a copy the park dropped; actions {actions:?}" + ); + } + + /// A track draining the incumbent while the newcomer's answer is pending goes unread: + /// the copy it drains is parked at once, like any unread track's. + #[test] + fn a_draining_track_parks_its_copy_when_unread() { + let mut front = serving(remote(1), 100); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + let t0 = Instant::now(); + assert_actions( + front.step(Event::Unused { + track: name("video"), + now: t0, + }), + &[ + Action::Park { track: name("video") }, + Action::Arm { at: Some(t0 + LINGER) }, + ], + ); + } + + /// A front that served ends `Dropped` when its route leaves, not with the refusal or + /// retraction of a challenger that never took over. + #[test] + fn a_front_that_served_ends_dropped() { + for standing in [true, false] { + let mut front = serving(remote(1), 100); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Err(Refusal { + err: Error::NotFound, + standing, + }), + }); + front.step(Event::Selected { + best: Some(remote(1)), + serving_closing: false, + }); + assert_actions( + front.step(Event::Selected { + best: None, + serving_closing: false, + }), + &[Action::End { err: Error::Dropped }], + ); + } + } + + /// The serving source closed and the front reselected, but the pump still reads the + /// closed source's copy. That copy ending cleanly ends the track: a path is one + /// broadcast, whoever serves it. + #[test] + fn a_closed_sources_clean_end_finishes_the_track() { + let mut front = serving(remote(1), 100); + front.step(Event::SourceClosed { source: 100 }); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Ok(()), + delivered: true, + }), + &[Action::Finish { track: name("video") }], + ); + } } diff --git a/rs/moq-net/src/model/group.rs b/rs/moq-net/src/model/group.rs index db1e74f5a7f..4fefbf4e32e 100644 --- a/rs/moq-net/src/model/group.rs +++ b/rs/moq-net/src/model/group.rs @@ -773,16 +773,6 @@ impl Producer { self.close_aborted(err) } - /// Abort with `err` unless another handle still owns the group, so a retained clone - /// lets go without the unfinished-drop warning and never cuts off a live writer. - /// Sound without a lock: only an existing handle can mint another. - pub(crate) fn abort_if_last(&self, err: Error) -> Result<()> { - if Arc::strong_count(&self.alive) > 1 { - return Ok(()); - } - self.close_aborted(err) - } - /// Abort with `err` only while nothing consumes the group, returning whether it is /// closed. Consumer creation and the check share a lock, so a reader arriving after /// [`poll_unused`](Self::poll_unused) keeps the group alive instead of reading the abort. @@ -852,7 +842,7 @@ impl Producer { /// One past the last frame committed to an unfinished group, when that is past its /// first: where a replacement route resumes. `None` once the group is finished, or - /// while it holds nothing a replacement could splice onto. + /// while it holds nothing a replacement could continue. /// /// The *committed* count, not the written one: a route dying midway through a /// chunked frame leaves that frame unusable, so the replacement has to send it @@ -944,7 +934,7 @@ impl Producer { Consumer { info: self.info, track: self.track.clone(), - inner: ConsumerKind::Plain(Plain { + cursor: Cursor { state, index: 0, end: None, @@ -952,15 +942,15 @@ impl Producer { cache: self.cache.clone(), access: self.alive.access.clone(), refreshed: self.cache.pool().now(), - }), + }, // Untagged: a tagged track attaches the egress meter via `with_meter` // when it hands the consumer to a subscriber/fetch. stats: stats::Meter::default(), - stale_stats: stats::Meter::default(), expiry: None, expired: false, ended: false, stale_counted: Arc::default(), + recover: None, } } @@ -1093,12 +1083,8 @@ impl Drop for Prefetch { } /// Consume a group, frame-by-frame. -/// -/// Usually a view of one [`Producer`], but a group served across a route change is -/// *spliced*: it reads each contributing route's copy in turn, joined at the frame the -/// takeover happened on, so the reader never sees the seam. pub struct Consumer { - inner: ConsumerKind, + cursor: Cursor, // Immutable stream state. info: Info, @@ -1108,11 +1094,8 @@ pub struct Consumer { track: track::Info, // Egress payload meter, set by a tagged track via [`Self::with_meter`]. Empty - // (no-op) for an untagged group. + // (no-op) for an untagged group. Also owns unread content discarded by expiry. stats: stats::Meter, - // The meter that owns unread content discarded by expiry. Route-specific - // cursors inside a spliced group inherit this without metering delivery twice. - stale_stats: stats::Meter, // Subscriber-specific drift policy. A group can become stale after the track // hands it out, while its reader is waiting for the first or next frame. @@ -1126,6 +1109,9 @@ pub struct Consumer { // Cloned cursors are parallel views of one handed-out delivery. Whichever // observes expiry first records its unread tail; the others must not repeat it. stale_counted: Arc, + // Handed out from a front's logical track: carries the read across route changes. + // Boxed: it is the rare case. + recover: Option>, } /// Subscriber-specific policy for expiring a group after it was handed out. @@ -1135,20 +1121,15 @@ pub struct Consumer { pub(crate) trait Expiry: Send + Sync + std::panic::UnwindSafe + std::panic::RefUnwindSafe { /// Return whether the group is stale, registering `waiter` for anything that /// could change the answer while the group remains live. - fn is_expired(&self, waiter: &kio::Waiter) -> bool; -} + /// A logical reader supplies its current budget after the original copy is gone. + fn is_expired(&self, max_age: Option, waiter: &kio::Waiter) -> bool; -// `Plain` is the hot path and carries an inline frame prefetch, so boxing it to even the -// variants out would cost an allocation per group to save a pointer chase on the rare one. -#[expect(clippy::large_enum_variant)] -enum ConsumerKind { - Plain(Plain), - // Boxed: the spliced cursor set dwarfs the plain one, and splicing is the rare case. - Spliced(Box), + /// Keep the reader's budget and cap while following a replacement track's edge. + fn for_track(&self, track: &track::Consumer) -> Arc; } -/// The cursor state for a group backed by a single [`Producer`]. -struct Plain { +/// The read cursor over a [`Producer`]'s shared state. +struct Cursor { // Shared state with the producer. state: kio::Consumer, @@ -1168,7 +1149,7 @@ struct Plain { refreshed: u64, } -impl Clone for Plain { +impl Clone for Cursor { fn clone(&self) -> Self { // A clone shares the channel and inherits `index`, but starts with an empty // prefetch: it re-reads its batch from the shared state, in parallel. @@ -1187,20 +1168,17 @@ impl Clone for Plain { impl Clone for Consumer { fn clone(&self) -> Self { Self { - inner: match &self.inner { - ConsumerKind::Plain(plain) => ConsumerKind::Plain(plain.clone()), - ConsumerKind::Spliced(spliced) => ConsumerKind::Spliced(Box::new((**spliced).clone())), - }, + cursor: self.cursor.clone(), info: self.info, track: self.track.clone(), // Inherit the meter without re-counting the group: the original already // counted it when the track handed it out. stats: self.stats.clone(), - stale_stats: self.stale_stats.clone(), expiry: self.expiry.clone(), expired: self.expired, ended: self.ended, stale_counted: self.stale_counted.clone(), + recover: self.recover.clone(), } } } @@ -1216,69 +1194,74 @@ impl std::ops::Deref for Consumer { impl Consumer { /// Snapshot the content this cursor would discard if its group were skipped. pub(crate) fn content(&self) -> stats::Content { - match &self.inner { - ConsumerKind::Plain(plain) => plain.state.read().content(), - // Drift is evaluated before a segment copy is wrapped as a spliced group. - // Keep the group count honest if a future caller reaches this fallback. - ConsumerKind::Spliced(_) => stats::Content { - groups: 1, - ..Default::default() - }, - } - } - - /// Content not already attributed as delivered by this handed-out cursor. - fn unread_content(&self) -> stats::Content { - match &self.inner { - ConsumerKind::Plain(plain) => plain.unread_content(), - // Each route-specific plain cursor enforces expiry inside a spliced group. - ConsumerKind::Spliced(_) => stats::Content::default(), - } - } - - /// Rebuild this consumer as the head of a group assembled across route changes, - /// keeping the group's identity and its track's properties. See [`super::resume`]. - pub(crate) fn into_spliced(self, mut spliced: super::resume::Group) -> Self { - spliced.set_stale_meter(self.stale_stats.clone()); - Self { - inner: ConsumerKind::Spliced(Box::new(spliced)), - info: self.info, - track: self.track, - stats: self.stats, - stale_stats: self.stale_stats, - // Each segment keeps its own route-specific expiry policy. Applying the - // head segment's policy to the assembled group would use the wrong edge - // after a takeover. - expiry: None, - expired: false, - ended: false, - stale_counted: self.stale_counted, - } + self.cursor.state.read().content() } /// Attach an egress payload meter, counting this as one delivered group. /// Called by a tagged track when it hands the consumer to a subscriber or fetch. pub(crate) fn with_meter(mut self, meter: stats::Meter) -> Self { meter.group(); - self.stats = meter.clone(); - self.set_stale_meter(meter); + self.stats = meter; self } - /// Attach only the meter that owns content discarded by expiry. - pub(crate) fn set_stale_meter(&mut self, meter: stats::Meter) { - if let ConsumerKind::Spliced(spliced) = &mut self.inner { - spliced.set_stale_meter(meter.clone()); - } - self.stale_stats = meter; - } - /// Keep applying this subscription's drift budget while the group is read. pub(crate) fn with_expiry(mut self, expiry: Arc) -> Self { self.expiry = Some(expiry); self } + /// Carry this group across a front's route changes; see [`super::resume`]. + pub(crate) fn with_recover(mut self, recover: super::resume::Recover) -> Self { + self.recover = Some(Box::new(recover)); + self + } + + /// Run `read`, and once this copy fails with its route, or stalls while a newer route + /// serves, continue from the serving route's copy at the same frame and read again. + fn poll_resumed( + &mut self, + waiter: &kio::Waiter, + mut read: impl FnMut(&mut Self, &kio::Waiter) -> Poll>>, + ) -> Poll>> { + loop { + let res = read(self, waiter); + // The reader's own budget gave up on it: that is no route's doing. + if self.expired || self.ended { + return res; + } + let Some(recover) = self.recover.as_mut() else { + return res; + }; + let failed = match &res { + Poll::Ready(Ok(Some(_))) => return res, + // Read to its end: no route is needed for it any more. + Poll::Ready(Ok(None)) => { + self.recover = None; + return res; + } + Poll::Ready(Err(err)) => Some(err.clone()), + Poll::Pending => None, + }; + if !recover.wants(failed.as_ref(), waiter) { + return res; + } + let replacement = match recover.poll(self.cursor.index as u64, failed.as_ref(), waiter) { + Poll::Ready(Ok(group)) => group, + Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), + Poll::Pending => return Poll::Pending, + }; + // Same frames by index, so only the channel changes; read progress and the + // cap stay this cursor's. + let (index, end) = (self.cursor.index, self.cursor.end); + recover.adopt(&replacement); + self.expiry = self.expiry.as_ref().map(|expiry| expiry.for_track(&replacement.copy)); + self.cursor = replacement.group.cursor; + self.cursor.index = index; + self.cursor.end = end; + } + } + /// Check the parent subscription while a wire publisher drains detached payload. pub(crate) fn poll_expired(&mut self, waiter: &kio::Waiter) -> bool { self.poll_expired_while_pending(waiter, false) @@ -1319,7 +1302,7 @@ impl Consumer { /// one that ended. A cursor that still holds unread content (a wire publisher with /// buffered frames, a half-read payload) is genuinely truncated and reports it. fn expired_truncates(&self) -> bool { - let unread = self.unread_content(); + let unread = self.cursor.unread_content(); unread.frames > 0 || unread.bytes > 0 } @@ -1327,11 +1310,13 @@ impl Consumer { pub(crate) fn poll_expired_while_pending(&mut self, waiter: &kio::Waiter, pending: bool) -> bool { if !self.expired && (pending || self.expiry_pending()) - && self.expiry.as_ref().is_some_and(|expiry| expiry.is_expired(waiter)) - { + && self.expiry.as_ref().is_some_and(|expiry| { + let budget = self.recover.as_ref().and_then(|recover| recover.poll_budget(waiter)); + expiry.is_expired(budget, waiter) + }) { self.expired = true; if !self.stale_counted.swap(true, Ordering::Relaxed) { - self.stale_stats.stale(self.unread_content()); + self.stats.stale(self.cursor.unread_content()); } } self.expired @@ -1339,35 +1324,24 @@ impl Consumer { /// Whether expiry can still discard content or unblock a group that may grow. fn expiry_pending(&self) -> bool { - match &self.inner { - ConsumerKind::Plain(plain) => plain.expiry_pending(), - // The route-specific plain cursors own expiry for a spliced group. - ConsumerKind::Spliced(_) => false, - } + self.cursor.expiry_pending() } /// Whether this cursor failed because its subscription max age budget expired. + #[cfg(test)] pub(crate) fn latency_expired(&self) -> bool { self.expired } /// Whether the group has been aborted (including pool eviction); the abort /// dropped the cached frames, so a held consumer has nothing left to read. - /// - /// A spliced group spans several routes, so no single abort empties it; only a - /// plain cursor can answer. pub(crate) fn is_aborted(&self) -> bool { - match &self.inner { - ConsumerKind::Plain(plain) => plain.state.read().abort.is_some(), - ConsumerKind::Spliced(_) => false, - } + self.cursor.state.read().abort.is_some() } /// Mark the group as still being read, so a slow batch drain does not expire it. pub fn keep_alive(&self) { - if let ConsumerKind::Plain(plain) = &self.inner { - plain.state.read().charge.refresh(); - } + self.cursor.state.read().charge.refresh(); } /// Record a cache access from the consumer side: a parked group re-offered to @@ -1376,18 +1350,11 @@ impl Consumer { self.keep_alive(); } - /// Park `waiter` until the group closes (finish, abort, or eviction). Spliced - /// subscribers register on parked groups so an eviction wakes them; a group - /// that already closed cleanly can never abort, so no waiter is needed. - /// - /// A spliced group reads as closed without registering anything: no single abort - /// empties it, so [`Self::is_aborted`] can never turn true and there is nothing - /// a wakeup would change. + /// Park `waiter` until the group closes (finish, abort, or eviction). Subscribers + /// register on parked groups so an eviction wakes them; a group that already closed + /// cleanly can never abort, so no waiter is needed. pub(crate) fn poll_closed(&self, waiter: &kio::Waiter) -> Poll<()> { - match &self.inner { - ConsumerKind::Plain(plain) => plain.state.poll_closed(waiter), - ConsumerKind::Spliced(_) => Poll::Ready(()), - } + self.cursor.state.poll_closed(waiter) } /// The parent track's timescale. @@ -1400,10 +1367,7 @@ impl Consumer { /// Starts at 0, or at the group's first available frame once [`Self::set_frames`] has /// clamped it, and advances by one per frame read. pub fn index(&self) -> u64 { - match &self.inner { - ConsumerKind::Plain(plain) => plain.index as u64, - ConsumerKind::Spliced(spliced) => spliced.index(), - } + self.cursor.index as u64 } /// Limit subsequent reads to these frame indices without rewinding read progress. @@ -1427,10 +1391,7 @@ impl Consumer { /// Only moves forward; a lower `index` is ignored, since the frames behind the /// cursor may already have been handed out. pub(crate) fn start_at(&mut self, index: u64) { - match &mut self.inner { - ConsumerKind::Plain(plain) => plain.start_at(index), - ConsumerKind::Spliced(spliced) => spliced.start_at(index), - } + self.cursor.start_at(index); } /// Advance the read cursor to `index`, skipping every frame below it. @@ -1439,10 +1400,7 @@ impl Consumer { /// never held. A [`Producer::start_at`] floor above `index` still surfaces as /// [`Error::Lagged`]. pub fn skip_to(&mut self, index: u64) { - match &mut self.inner { - ConsumerKind::Plain(plain) => plain.skip_to(index), - ConsumerKind::Spliced(spliced) => spliced.start_at(index), - } + self.cursor.skip_to(index); } /// Stop reading at `end`, or remove the cap with `..`. @@ -1453,24 +1411,14 @@ impl Consumer { /// frames that are still cached. pub(crate) fn end_at(&mut self, end: impl Into) { let end = end.into().exclusive(); - match &mut self.inner { - ConsumerKind::Plain(plain) => { - plain.end = end.map(|end| usize::try_from(end).unwrap_or(usize::MAX)); - } - ConsumerKind::Spliced(spliced) => spliced.end_at(end), - } + self.cursor.end = end.map(|end| usize::try_from(end).unwrap_or(usize::MAX)); } /// The number of frames written so far (completed plus any in-flight), independent of /// how many this consumer has read. The final total once the group is finished. pub fn frame_count(&self) -> usize { - match &self.inner { - ConsumerKind::Plain(plain) => { - let state = plain.state.read(); - state.fin.unwrap_or(state.next_index) - } - ConsumerKind::Spliced(spliced) => spliced.frame_count(), - } + let state = self.cursor.state.read(); + state.fin.unwrap_or(state.next_index) } /// Return a consumer for the next frame for chunked reading. @@ -1483,6 +1431,20 @@ impl Consumer { /// Returns None if the group is finished and the index is out of range, or the cursor /// passed the [`Self::set_frames`] cap. pub fn poll_next_frame(&mut self, waiter: &kio::Waiter) -> Poll>> { + if self.recover.is_none() { + return self.poll_next_frame_once(waiter); + } + let res = self.poll_resumed(waiter, Self::poll_next_frame_once); + match (res, &self.recover) { + // A frame read from a copy that may fail too carries on the same way. + (Poll::Ready(Ok(Some(frame))), Some(recover)) => Poll::Ready(Ok(Some( + frame.with_recover((**recover).clone(), self.cursor.index as u64 - 1), + ))), + (res, _) => res, + } + } + + fn poll_next_frame_once(&mut self, waiter: &kio::Waiter) -> Poll>> { if self.ended { return Poll::Ready(Ok(None)); } @@ -1493,19 +1455,8 @@ impl Consumer { let expiry = self .expiry .as_ref() - .map(|policy| frame::Expiry::new(policy.clone(), self.stale_stats.clone(), self.stale_counted.clone())); - let res = match &mut self.inner { - ConsumerKind::Plain(plain) => plain.poll_next_frame(waiter, &stats, expiry), - ConsumerKind::Spliced(spliced) => { - // The per-route copies underneath are untagged, so meter the spliced - // stream here: it is the one the subscriber actually reads. - let res = ready!(spliced.poll_next_frame(waiter))?; - if res.is_some() { - stats.frames(1); - } - Poll::Ready(Ok(res.map(|frame| frame.with_meter(stats)))) - } - }; + .map(|policy| frame::Expiry::new(policy.clone(), stats.clone(), self.stale_counted.clone())); + let res = self.cursor.poll_next_frame(waiter, &stats, expiry); match res.is_pending().then(|| self.poll_expired_if_blocked(waiter)).flatten() { Some(true) => Poll::Ready(Err(Error::Old)), Some(false) => Poll::Ready(Ok(None)), @@ -1515,24 +1466,20 @@ impl Consumer { /// Read the next frame (timestamp and payload) all at once, without blocking. pub fn poll_read_frame(&mut self, waiter: &kio::Waiter) -> Poll>> { + if self.recover.is_none() { + return self.poll_read_frame_once(waiter); + } + self.poll_resumed(waiter, Self::poll_read_frame_once) + } + + fn poll_read_frame_once(&mut self, waiter: &kio::Waiter) -> Poll>> { if self.ended { return Poll::Ready(Ok(None)); } if self.expired { return Poll::Ready(Err(Error::Old)); } - let stats = self.stats.clone(); - let res = match &mut self.inner { - ConsumerKind::Plain(plain) => plain.poll_read_frame(waiter, &stats), - ConsumerKind::Spliced(spliced) => { - let res = ready!(spliced.poll_read_frame(waiter))?; - if let Some(frame) = &res { - stats.frames(1); - stats.bytes(frame.payload.len() as u64); - } - Poll::Ready(Ok(res)) - } - }; + let res = self.cursor.poll_read_frame(waiter, &self.stats); match res.is_pending().then(|| self.poll_expired_if_blocked(waiter)).flatten() { Some(true) => Poll::Ready(Err(Error::Old)), Some(false) => Poll::Ready(Ok(None)), @@ -1544,17 +1491,14 @@ impl Consumer { pub async fn read_frame(&mut self) -> Result> { // A prefetched frame is already buffered, so the drift budget (which only judges // a read that would park) can never apply to it. + // Serve from the prefetched batch without building a future or allocating a waker. if !self.expired - && let ConsumerKind::Plain(plain) = &mut self.inner + && !self.cursor.capped() + && let Some(frame) = self.cursor.prefetch.pop() { - // Serve from the prefetched batch without building a future or allocating a waker. - if !plain.capped() - && let Some(frame) = plain.prefetch.pop() - { - plain.refresh_if_stale(); - plain.index += 1; - return Ok(Some(frame)); - } + self.cursor.refresh_if_stale(); + self.cursor.index += 1; + return Ok(Some(frame)); } kio::wait(|waiter| self.poll_read_frame(waiter)).await } @@ -1604,21 +1548,27 @@ impl Consumer { /// Poll until the group terminates, returning this cursor's next frame index. pub fn poll_finished(&mut self, waiter: &kio::Waiter) -> Poll> { + if self.recover.is_none() { + return self.poll_finished_once(waiter); + } + let res = self.poll_resumed(waiter, |this, waiter| { + this.poll_finished_once(waiter).map(|res| res.map(Some)) + }); + res.map(|res| res.map(|index| index.expect("finished with an index"))) + } + + fn poll_finished_once(&mut self, waiter: &kio::Waiter) -> Poll> { if self.ended { return Poll::Ready(Ok(self.index())); } if self.expired { return Poll::Ready(Err(Error::Old)); } - let res = match &mut self.inner { - ConsumerKind::Plain(plain) => { - let index = plain.index; - plain - .poll(waiter, |state| state.poll_end(index)) - .map(|res| res.map(|()| index as u64)) - } - ConsumerKind::Spliced(spliced) => spliced.poll_finished(waiter), - }; + let index = self.cursor.index; + let res = self + .cursor + .poll(waiter, |state| state.poll_end(index)) + .map(|res| res.map(|()| index as u64)); match res.is_pending().then(|| self.poll_expired_if_blocked(waiter)).flatten() { Some(true) => Poll::Ready(Err(Error::Old)), // The group ended where the cursor stands, so that is its frame count. @@ -1638,7 +1588,7 @@ impl Consumer { } } -impl Plain { +impl Cursor { /// Whether this cursor still has unread content or may receive another frame. fn expiry_pending(&self) -> bool { if self.capped() { @@ -2477,7 +2427,7 @@ mod test { )); } - /// Seeking to the group's first available frame is how a spliced reader picks up + /// Seeking to the group's first available frame is how a front's pump picks up /// the tail; a lower index clamps up rather than failing. #[test] fn start_at_clamps_up_to_the_first_frame() { diff --git a/rs/moq-net/src/model/mod.rs b/rs/moq-net/src/model/mod.rs index 870cd97b4ef..a2e6a3b4142 100644 --- a/rs/moq-net/src/model/mod.rs +++ b/rs/moq-net/src/model/mod.rs @@ -18,7 +18,7 @@ mod bytes; pub(crate) mod clock; mod datagram; mod requests; -pub(crate) mod resume; +mod resume; mod subscription; mod time; mod timed; diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 5f96414526c..00cd050446c 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -1,4 +1,4 @@ -use crate::{broadcast, cache, group, stats, track}; +use crate::{broadcast, cache, stats, track}; use kio::Task; use std::{ cmp::Reverse, @@ -14,7 +14,7 @@ use rand::RngExt; use super::{ Requests, WeakCache, WeakEntry, - front::{Action, Candidate, Event, Front, Pin, Refusal}, + front::{Action, Candidate, Event, Front, Refusal}, }; use crate::{ AsPath, Error, InvalidPattern, Path, PathOwned, Pattern, Patterns, @@ -855,8 +855,8 @@ struct RouteEntry { /// matches this as well as [`Self::hops`], so an anonymous hop 0 still /// cannot echo back to the session it came from. via: Hop, - /// Whether this is a broadcast published on this origin: a front that - /// starts from one only fails over to another local publisher. + /// Whether this is a broadcast published on this origin, which wins a cost tie and + /// keeps its own cache. local: bool, /// Whether a handle marked [`Producer::peer`] inserted the entry, so it /// entered from a cluster peer rather than here. @@ -910,17 +910,6 @@ impl RouteEntry { self.server.is_some() || (self.source.is_some() && self.prefix == *path) } - /// Whether `pin` admits this entry for a front's selection. - fn qualifies(&self, pin: Pin) -> bool { - match pin { - Pin::Any => true, - Pin::Local => self.local, - Pin::Publisher(first) => self.hops.iter().next() == Some(&first), - // An update that names a publisher is a different one, even in place. - Pin::Route(id) => self.id == id && self.hops.iter().next().is_none_or(|first| *first == Hop::UNKNOWN), - } - } - /// Whether this entry may be observed or served to a requester excluding `peer`. /// /// A non-zero peer is hidden when it is the announcing session (`via`) or @@ -997,20 +986,17 @@ impl Horizon { } } -/// One remotely-served front in [`OriginState::fronts`]: the shared spliced -/// broadcast at a path plus the channel requesters resolve through. +/// One remotely-served front in [`OriginState::fronts`]: the shared broadcast at a +/// path plus the channel requesters resolve through. #[derive(Clone)] struct RemoteFront { /// Resolves requesters with the front's consumer (or the error that ended it /// unresolved). The producer lives here so the teardown can reject requesters /// still parked on a front whose watcher was cancelled. request: kio::Producer, - /// The front's spliced broadcast, weak: dead once the front ends, so a + /// The front's broadcast, weak: dead once the front ends, so a /// later request re-creates the front instead of joining a corpse. broadcast: broadcast::WeakConsumer, - /// Which routes serve the front's content, fixed by its first source. A - /// request joins the front only while the best route is one of them. - pin: kio::Lock, } /// The last route a cursor observed: entry id, metadata, servability, and captures. @@ -1532,10 +1518,10 @@ impl Producer { /// /// This is how local content enters an origin. The returned /// [`broadcast::Producer`] is a source: the origin owns the broadcast - /// consumers actually see, and splices its tracks across every source created - /// at the same path, preferring the newest. When the serving source changes, - /// tracks resume from the replacement at the first missing group; consumers - /// never observe the swap. + /// consumers actually see, and feeds its tracks from every source created at the + /// same path, preferring the newest. When the serving source changes, tracks + /// resume from the replacement where they left off; consumers never observe the + /// swap. /// /// The broadcast exists for nobody until [`broadcast::Producer::announce`]: /// until then no announce cursor lists it and a request for its path fails @@ -2013,13 +1999,6 @@ impl AnnounceProducer { let Some(entry) = shared.routes.entry_mut(prefix, *id) else { return Err(Error::Closed); }; - // A new first hop is a new publisher: a later request goes back to the - // handler rather than joining what the old one served. - if entry.hops.iter().next() != route.hops.iter().next() - && let Some(server) = &entry.server - { - drop(std::mem::take(&mut server.lock().served)); - } entry.hops = route.hops.clone(); entry.stale = stale; entry.cost = route.cost; @@ -2232,210 +2211,11 @@ impl Drop for DriverState { } } -/// How long a spliced track stays warm after its last reader leaves. -/// -/// Within the window a returning viewer, or the next of a run of back-to-back -/// fetches, reads the groups the front already cached: no second round trip for -/// `TRACK_INFO`. Groups past that cached edge cost a fresh source splice. After -/// the window the track is forgotten, finished and aborted ones included, so a -/// long-lived broadcast only holds the tracks read recently. -/// -/// Sized above the fetch cadence of a segmented consumer: HLS polls every -/// `TARGETDURATION` seconds, commonly 6 or 10, so a shorter window would drop the -/// copy between every segment and re-request the track each time. A warm copy -/// holds no upstream subscription (that is canceled as soon as demand ends), so -/// waiting longer costs cached state, not a viewer. -const TRACK_IDLE_LINGER: Duration = Duration::from_secs(30); - -/// A local copy of groups the front already delivered, so resume stays spliced -/// after the source track is dropped. Cache misses stay pending while demand -/// re-splices the upstream source. Finished on drop so an idle linger does not -/// warn about an abandoned producer. -struct WarmCopy { - track: track::Producer, - _dynamic: track::Dynamic, - /// The newest group, where the copy spliced after this one picks up (see - /// [`TrackIo::head`]). - edge: Option, -} - -impl Drop for WarmCopy { - fn drop(&mut self) { - let _ = self.track.finish(); - } -} - -/// A warm copy's newest group, which the copy spliced after it continues. -/// -/// Kept past that splice: an open one must stay open for the continuation (dropping -/// an unfinished producer clears its frames), and either kind supplies the head the -/// continuation lacks when the next park rebuilds the group. Nothing will ever finish -/// an open one, so it aborts on drop. -struct WarmGroup(group::Producer); - -impl Drop for WarmGroup { - fn drop(&mut self) { - if !self.0.is_finished() { - let _ = self.0.clone().abort(Error::Cancel); - } - } -} - -/// A spliced copy's newest group, held so its head outlives the copy's route (see -/// [`TrackIo::held`]). Borrowed from the route, which may still be writing it for other -/// readers, so it aborts on drop only once it is the last handle: a dead route never -/// finishes it. -struct HeldGroup(group::Producer); - -impl Drop for HeldGroup { - fn drop(&mut self) { - if !self.0.is_finished() { - let _ = self.0.abort_if_last(Error::Cancel); - } - } -} - -/// Cache what `source` delivered on a new local track the origin owns: its complete -/// groups, and its open live edge rebuilt from the frames already delivered. -/// -/// `head` is the previous park's edge. A copy spliced after a warm cache continues its -/// edge group from the next frame, so its copy of that group lacks the head, which -/// `head` supplies. `held` is the copy's newest group as [`TrackIo::held`] kept it: a -/// copy that died tore its open groups out of its own cache, but not out of the held one. -fn warm_copy(source: &track::Consumer, head: Option<&WarmGroup>, held: Option<&group::Producer>) -> Option { - let info = source.cached_info()?; - let mut track = track::Producer::new(Arc::new(source.broadcast().clone()), source.name(), info); - let head = head.map(|head| &head.0); - let mut groups = source.cached_groups(); - if let Some(held) = held - && !groups.iter().any(|(group, _)| group.sequence == held.sequence) - { - groups.push((held.clone(), true)); - } - // Not `source.latest()`: datagrams share the sequence counter and can run past it. - let latest = groups - .iter() - .filter(|(_, visible)| *visible) - .map(|(group, _)| group.sequence) - .max(); - let mut edge = None; - let mut first = None; - - // A spliced copy hides the group it continued (its halves sit in two segments), so - // carry the previous edge over when the copy has no version of it at all. First, - // since it arrived before anything the copy holds. - if let Some(head) = head - && !groups.iter().any(|(group, _)| group.sequence == head.sequence) - { - let is_latest = latest.is_none_or(|latest| head.sequence >= latest); - if head.is_finished() { - let _ = track.adopt_group(head.clone(), true); - first = Some(head.sequence); - if is_latest { - edge = Some(WarmGroup(head.clone())); - } - } else if is_latest { - edge = warm_rebuild(&track, head, None); - first = edge.as_ref().map(|_| head.sequence); - } - } - - for (group, visible) in groups { - let finished = group.is_finished(); - let whole = group.live_first_frame() == Some(0); - let is_latest = visible && Some(group.sequence) == latest; - let warm = if finished && whole { - let _ = track.adopt_group(group.clone(), visible); - Some(WarmGroup(group)) - } else if (finished || is_latest) && (whole || head.is_some_and(|head| head.sequence == group.sequence)) { - // Dropping the source copy resets an open live edge mid-transfer, so rebuild - // it from the frames already delivered: the re-splice asks for the next - // frame, and a group that stays open for good (a JSON log in group 0) - // continues instead of being re-sent whole on every resume. A continuation - // is rebuilt whole from the previous edge's head the same way. - warm_rebuild(&track, &group, head) - } else { - // Mid-transfer backlog, or a continuation with no head to complete it. - None - }; - if visible && let Some(warm) = &warm { - first = Some(first.map_or(warm.0.sequence, |first: u64| first.min(warm.0.sequence))); - } - if is_latest { - edge = warm; - } - } - // Arrival order can put the live group before cached history. Declare the - // cache's oldest group so a new subscription cannot resolve its floor from - // whichever group arrived first and permanently skip that history. - track.start_at(first).ok()?; - let dynamic = track.dynamic(); - Some(WarmCopy { - track, - _dynamic: dynamic, - edge, - }) -} - -/// Rebuild `live` on `track` from its delivered frames, prefixed by `head`'s when `live` -/// only holds a continuation of it. Finished like `live`, or left open. -fn warm_rebuild(track: &track::Producer, live: &group::Producer, head: Option<&group::Producer>) -> Option { - // A continuation holds nothing below its offset. - let mut start = live.live_first_frame()? as u64; - let mut tail = live.consume(); - tail.start_at(start); - let mut frames = Vec::new(); - if start > 0 - && let Some(head) = head.filter(|head| head.sequence == live.sequence) - && let Some(head_start) = head.live_first_frame() - { - let mut head = head.consume(); - head.start_at(head_start as u64); - while head.index() < start { - match head.poll_read_frame(&kio::Waiter::noop()) { - Poll::Ready(Ok(Some(frame))) => frames.push(frame), - _ => break, - } - } - match head.index() == start { - true => start = head_start as u64, - // The head doesn't reach the continuation: keep only the continuation. - false => frames.clear(), - } - } - while let Poll::Ready(Ok(Some(frame))) = tail.poll_read_frame(&kio::Waiter::noop()) { - frames.push(frame); - } - if frames.is_empty() { - return None; - } - - // Wrapped first, so a failed write aborts it rather than dropping it unfinished. - let rebuilt = WarmGroup( - track - .create_group(group::Info { - sequence: live.sequence, - }) - .ok()?, - ); - let mut writer = rebuilt.0.clone(); - if start > 0 { - writer.start_at(start).ok()?; - } - for frame in frames { - writer.write_frame(frame.timestamp, frame.payload).ok()?; - } - if live.is_finished() { - writer.finish().ok()?; - } - Some(rebuilt) -} - /// Everything [`run_front`] owns, queued by [`Consumer::request_broadcast`]. struct FrontTask { /// The route table the front selects from. shared: kio::Shared, - /// The spliced broadcast the front serves. + /// The broadcast the front serves. broadcast: broadcast::Producer, /// Absolute path of the front. path: PathOwned, @@ -2445,86 +2225,78 @@ struct FrontTask { watch: Watch, /// Resolves the requesters parked on the front's channel. request: kio::Producer, - /// Published for requesters once the first source fixes it; see [`RemoteFront::pin`]. - pin: kio::Lock, timers: Clock, } +/// A route asked for its copy of a track: the copy, its info once its query +/// resolved, and a subscription with what the readers want. +struct Asked { + source: u64, + copy: track::Consumer, + info: Option, + /// Some sessions only learn a track's info by subscribing (moq-transport's + /// SUBSCRIBE_OK), and read the demand only then, so the readers' demand rides the + /// query. Kept once spliced until the last reader leaves, so the copy is never left + /// with nobody subscribed while readers are still subscribing to it on their own. + sub: Option>, +} + /// The driver's side of one logical track: the handles behind the names the /// machine uses. struct TrackIo { - resume: super::resume::Producer, + /// The logical track, until the first copy says what it is. + request: Option, + /// The logical track once accepted. Nothing writes it: its readers read the + /// serving route's copy through `routes`. + accepted: Option, + /// Whether anyone reads the logical track. + weak: track::TrackWeak, + /// Which copy serves the track, and how it ends; see [`super::resume`]. + routes: super::resume::Producer, + /// A copy whose info is still in flight. + query: Option<(Asked, track::Querying)>, /// The copy whose info resolved, waiting for the machine to splice it. - staged: Option<(u64, track::Consumer)>, - /// A query in flight: the source asked, its copy, and the pending info. - query: Option<(u64, track::Consumer, track::Querying)>, - /// The spliced copy: its source and the track. + staged: Option, + /// The source whose copy serves the track, and the subscription its query made. copy: Option<(u64, track::Consumer)>, - /// The delivered edge when the copy spliced in: a copy that dies without - /// advancing it delivered nothing. Snapshotted per splice, not per wake, so an - /// unrelated wake between the copy's last frame and its death cannot launder - /// its progress away. - edge: Option, - /// Delivered groups kept after the copy was dropped, so resume stays spliced - /// through the linger without pinning the source as a reader. - warm: Option, - /// The last warm copy's newest group, outliving it so the copy spliced after it - /// can continue the group (see [`WarmGroup`]). Released at the next park. - head: Option, - /// The spliced copy's newest group. A route that ends abruptly releases its open - /// groups, and the takeover after it asks the next copy only for the frames past - /// the break, so without this handle no route holds the group's head for a reader - /// that arrives later. See [`TrackIo::retire`]. - held: Option, - /// The spliced copy, once it died: its segment keeps ending readers with its error - /// unless a takeover follows, which retires it. - dead: Option, + held: Option>, + /// The last copy ended because its session closed locally: a close, not an + /// error, so the track ends cleanly if nothing takes over. + closed: bool, /// Whether the track had a reader as of the last demand edge. used: bool, } impl TrackIo { - /// Let go of every source handle once the logical track ended: its segments keep - /// what readers drain, and only its demand is still watched, until it is forgotten. - fn end(&mut self) { - self.staged = None; - self.query = None; - self.copy = None; - self.warm = None; - self.head = None; - self.held = None; - self.dead = None; - } - - /// Keep the spliced copy as dead. The front only refreshes `held` between wakes, so - /// first catch a newest group the copy surfaced since, while its cache still has it. - fn bury(&mut self) { - let Some((_, copy)) = self.copy.take() else { return }; - let after = self.held.as_ref().map(|held| held.0.sequence); - if let Poll::Ready(group) = copy.poll_latest_group(after, &kio::Waiter::noop()) { - self.held = Some(HeldGroup(group)); + /// Serve the track from `asked`'s copy, accepting it with that copy's info first. + fn splice(&mut self, asked: Asked) { + if let Some(request) = self.request.take() { + let info = asked.info.clone().unwrap_or_default(); + self.accepted = Some(request.routes(self.routes.consume()).accept(info)); } - self.dead = Some(copy); + self.routes.serve(asked.copy.clone()); + self.copy = Some((asked.source, asked.copy)); + self.held = asked.sub; } - /// Retire `copy` as a takeover replaces it. Mid-group, park its delivery warm, open - /// group included, so a reader arriving later still gets the group's head and the - /// takeover continues it. Between groups its own cache keeps everything finished, so - /// there is nothing to do. - fn retire(&mut self, copy: &track::Consumer) -> Result<(), Error> { - let Some(held) = self.held.take() else { - return Ok(()); - }; - if held.0.is_finished() { - return Ok(()); + /// End the track: cleanly with `Ok`, or failed for good. Readers drain what the + /// serving copy still holds. + fn end(&mut self, result: Result<(), Error>) { + self.query = None; + self.staged = None; + self.copy = None; + self.held = None; + match self.request.take() { + // Ended before any route served it: nothing to read. + Some(request) => request.reject(result.err().unwrap_or(Error::NotFound)), + None => { + // A failed track refuses newcomers too, so they ask afresh. + if let (Err(err), Some(accepted)) = (&result, self.accepted.take()) { + let _ = accepted.abort(err.clone()); + } + self.routes.end(result); + } } - let Some(warm) = warm_copy(copy, self.head.as_ref(), Some(&held.0)) else { - return Ok(()); - }; - self.head = None; - self.resume.park(&warm.track)?; - self.warm = Some(warm); - Ok(()) } } @@ -2539,24 +2311,25 @@ async fn run_front(task: FrontTask) { horizon, watch, request, - pin, timers, } = task; /// What the wait below returns: one thing that happened. enum Step { - Assigned(Arc, super::resume::Producer), + Assigned(track::Request), Resolved(u64, Result), SourceClosed(u64), Info(Arc, u64, Result), - Ended(Arc, u64, Result<(), Error>), - Held(Arc, group::Producer), + Ended(Arc, u64, Result<(), Error>, bool), Demand(Arc), Deadline, Table, } - let mut front = Front::new(TRACK_IDLE_LINGER); + // The front serves its broadcast on demand: every track a reader names is + // handed here, and its readers read it from whichever source serves the path. + let mut dynamic = broadcast.dynamic(); + let mut front = Front::new(track::IDLE_LINGER); let mut sources: HashMap = HashMap::new(); let mut next_source = 0u64; // The in-flight upstream request: the route and its pending channel. @@ -2567,7 +2340,7 @@ async fn run_front(task: FrontTask) { let mut seen = 0; let mut events: VecDeque = VecDeque::new(); - // Read the table for the machine: the best qualifying route and whether + // Read the table for the machine: the best route and whether // the serving source is on its way out. Also what the watch wakes for. let select = |front: &mut Front, sources: &HashMap, seen: &mut u64| -> Event { let table = shared.read(); @@ -2578,10 +2351,9 @@ async fn run_front(task: FrontTask) { *seen = watch.seen(); front.retain_routes(|route| table.routes.covers(&path.as_path(), route)); let best = table - .best_route(&path.as_path(), horizon, front.pin(), front.refused_routes()) + .best_route(&path.as_path(), horizon, front.refused_routes()) .map(|entry| Candidate { route: entry.id, - first: entry.hops.iter().next().copied(), local: entry.local, }); let serving_closing = front @@ -2595,30 +2367,24 @@ async fn run_front(task: FrontTask) { loop { while let Some(event) = events.pop_front() { + let answered = match &event { + Event::TrackInfo { track, source, .. } => Some((track.clone(), *source)), + _ => None, + }; for action in front.step(event) { match action { Action::Reselect => events.push_back(select(&mut front, &sources, &mut seen)), Action::Request { route } => { - // The entry, its identity for the front, and what it serves. + // The entry and what it serves. let found = { let table = shared.read(); table .routes .covering(&path.as_path()) .find(|entry| entry.id == route && entry.live()) - .map(|entry| { - ( - Candidate { - route, - first: entry.hops.iter().next().copied(), - local: entry.local, - }, - entry.source.clone(), - entry.server.clone(), - ) - }) + .map(|entry| (entry.source.clone(), entry.server.clone())) }; - let Some((candidate, source, server)) = found else { + let Some((source, server)) = found else { events.push_back(Event::Resolved { route, result: Err(Refusal { @@ -2628,8 +2394,6 @@ async fn run_front(task: FrontTask) { }); continue; }; - front.identify(candidate); - *pin.lock() = front.pin(); if let Some(source) = source { let id = next_source; next_source += 1; @@ -2697,17 +2461,15 @@ async fn run_front(task: FrontTask) { upstream = Some((route, pending)); } Action::Detach { source } => { + // Readers keep reading the source's copy until a replacement is + // spliced, so a route being beaten keeps serving until then; a + // route that died ends its copy, which says so. sources.remove(&source); - // Its copies go with it; the segments they delivered stay - // spliced until a replacement resumes past them. for io in tracks.values_mut() { - if io.copy.as_ref().is_some_and(|(s, _)| *s == source) { - io.bury(); - } - if io.query.as_ref().is_some_and(|(s, ..)| *s == source) { + if io.query.as_ref().is_some_and(|(asked, _)| asked.source == source) { io.query = None; } - if io.staged.as_ref().is_some_and(|(s, _)| *s == source) { + if io.staged.as_ref().is_some_and(|asked| asked.source == source) { io.staged = None; } } @@ -2722,10 +2484,19 @@ async fn run_front(task: FrontTask) { let closing = sources.get(&source).is_some_and(|s| s.is_closing()); match sources.get(&source).map(|s| s.track(&name)) { Some(Ok(copy)) => { + let sub = io.weak.subscription().map(|demand| copy.subscribe(demand)); // `into_inner` sheds the `Pending` future wrapper so only // the pollable (which is `Sync`) is held across the wait. - let query = copy.query().into_inner(); - io.query = Some((source, copy, query)); + let info = copy.query().into_inner(); + io.query = Some(( + Asked { + source, + copy, + info: None, + sub, + }, + info, + )); } Some(Err(err)) => events.push_back(Event::TrackInfo { track: name, @@ -2738,54 +2509,21 @@ async fn run_front(task: FrontTask) { } Action::Splice { track: name, source } => { let Some(io) = tracks.get_mut(&name) else { continue }; - let Some((staged, copy)) = io.staged.take() else { - continue; - }; - if staged != source { + let Some(asked) = io.staged.take() else { continue; - } - let outgoing = io.copy.take().map(|(_, copy)| copy).or_else(|| io.dead.take()); - let retired = match outgoing { - Some(outgoing) => io.retire(&outgoing), - None => Ok(()), }; - if let Err(err) = retired.and_then(|()| io.resume.takeover(©)) { - // Closed means the logical track already ended. Anything - // else is a boundary bug; abort rather than strand - // subscribers on a track nobody serves. - let _ = io.resume.abort(err); - tracks.remove(&name); + if asked.source != source { continue; } - if let Some(head) = io.warm.take().and_then(|mut warm| warm.edge.take()) { - io.head = Some(head); - } - // The new segment has produced nothing yet: this is the - // edge the copy is asked to advance. - io.edge = io.resume.resume_position(); - io.held = None; - io.copy = Some((source, copy)); + io.splice(asked); } Action::Park { track: name } => { let Some(io) = tracks.get_mut(&name) else { continue }; - let Some((_, copy)) = io.copy.take() else { continue }; - // Drop the source copy so its producer goes idle at once; keep - // the groups it delivered on a local track so resume stays - // spliced until the linger expires. - let warm = warm_copy(©, io.head.as_ref(), io.held.as_ref().map(|held| &held.0)); - drop(copy); - io.head = None; + // Drop the copy so its source goes idle at once. A returning + // reader gets a fresh one, so nothing cached can be stale. + io.routes.park(); + io.copy = None; io.held = None; - io.dead = None; - let parked = match &warm { - Some(warm) => io.resume.park(&warm.track), - None => io.resume.release(), - }; - if parked.is_err() { - tracks.remove(&name); - continue; - } - io.warm = warm; } Action::Forget { track: name } => { // A reader that looked the track up since the machine decided keeps @@ -2793,7 +2531,7 @@ async fn run_front(task: FrontTask) { // current level, so a reader gone before the next poll would // otherwise leave the track unread with no linger armed. if let Some(io) = tracks.get_mut(&name) - && !broadcast.forget_spliced(&name, &io.resume) + && !io.weak.abort_unused(Error::Dropped) { if !io.used { io.used = true; @@ -2801,20 +2539,21 @@ async fn run_front(task: FrontTask) { } continue; } - tracks.remove(&name); + // Nobody reads it, so nobody follows its copy: let the route go idle. + if let Some(io) = tracks.remove(&name) { + io.routes.park(); + } events.push_back(Event::Forgotten { track: name }); } Action::Finish { track: name } => { if let Some(io) = tracks.get_mut(&name) { - io.end(); - let _ = io.resume.finish(); + io.end(Ok(())); } } Action::Abort { track: name, err } => { if let Some(io) = tracks.get_mut(&name) { tracing::debug!(name = %name, %err, "aborting track"); - io.end(); - let _ = io.resume.abort(err); + io.end(Err(err)); } } Action::Arm { at } => deadline.set(at), @@ -2825,39 +2564,46 @@ async fn run_front(task: FrontTask) { // Ending the broadcast only retracts it: no new requesters or // tracks, and a newcomer at the path gets a fresh front. Tracks // in flight carry on (moq-lite: retraction does not disturb - // subscriptions already in flight): dropping their producers - // leaves each reader on the copy it was spliced from, ending - // when and as that copy ends. + // subscriptions already in flight): their readers follow the copy + // they read to its end, since no front is left to replace it. broadcast.close(); - broadcast.release_spliced(err.clone()); for (_, mut io) in tracks.drain() { + let used = io.weak.is_used(); // A reader still waiting on its source's answer is in flight - // too: splice the copy it asked, past any warm cache, so it - // ends as that copy does. - let waiting = io.staged.take().map(|(_, copy)| copy); - let waiting = waiting.or_else(|| io.query.take().map(|(_, copy, _)| copy)); - if let Some(copy) = waiting - && io.resume.is_used() + // too: serve it the copy it asked, so it ends as that copy does. + let waiting = io.staged.take().or_else(|| io.query.take().map(|(asked, _)| asked)); + if io.copy.is_none() + && used && let Some(asked) = waiting { - if io.resume.takeover(©).is_err() { - continue; - } - io.warm = None; + io.splice(asked); } - // Nothing in flight: unread, never spliced, or only a warm cache. - if !io.resume.is_used() || !io.resume.is_spliced() || io.warm.is_some() { - let _ = io.resume.abort(err.clone()); + // Nothing in flight: unread, or nothing serving it. One whose copy + // went with a session closed locally ends cleanly. + if !used || io.copy.is_none() { + io.end(match io.closed && used { + true => Ok(()), + false => Err(err.clone()), + }); } + // Dropping `io.routes` concludes the rest: readers follow the copy. } return; } } } + // A staged copy the machine did not splice was refused or is no longer wanted: + // let it go, or its source stays subscribed for nobody. + if let Some((name, source)) = answered + && let Some(io) = tracks.get_mut(&name) + && io.staged.as_ref().is_some_and(|asked| asked.source == source) + { + io.staged = None; + } } let step = kio::wait(|waiter| { - if let Poll::Ready((name, resume)) = broadcast.poll_spliced_assigned(waiter) { - return Poll::Ready(Step::Assigned(name, resume)); + if let Poll::Ready(Ok(request)) = dynamic.poll_requested_track(waiter) { + return Poll::Ready(Step::Assigned(request)); } if let Some((route, pending)) = &upstream && let Poll::Ready(result) = pending.poll(waiter, |p| match &p.resolved { @@ -2881,28 +2627,38 @@ async fn run_front(task: FrontTask) { return Poll::Ready(Step::SourceClosed(id)); } for (name, io) in &mut tracks { - if let Some((source, _, query)) = &mut io.query + if let Some((asked, query)) = &mut io.query && let Poll::Ready(result) = query.poll(waiter) { - return Poll::Ready(Step::Info(name.clone(), *source, result)); + return Poll::Ready(Step::Info(name.clone(), asked.source, result)); } - if let Some((source, copy)) = &io.copy - && let Poll::Ready(result) = copy.poll_complete(waiter) + // The query's subscription goes with the last reader. + if io.held.is_some() + && let Some(accepted) = &mut io.accepted { - return Poll::Ready(Step::Ended(name.clone(), *source, result)); + while let Poll::Ready(Ok(_)) = accepted.poll_subscription_changed(waiter) {} + if accepted.subscription().is_none() { + io.held = None; + } } - if let Some((_, copy)) = &io.copy - && let Poll::Ready(group) = - copy.poll_latest_group(io.held.as_ref().map(|held| held.0.sequence), waiter) + // Settled once the copy closes: a group still open below a declared end is + // owed until then, and a session closing first means it never came. + if let Some((source, copy)) = &io.copy + && copy.poll_closed(waiter).is_ready() { - return Poll::Ready(Step::Held(name.clone(), group)); + let result = match copy.poll_complete(&kio::Waiter::noop()) { + Poll::Ready(result) => result, + Poll::Pending => Err(Error::Dropped), + }; + let delivered = copy.latest().is_some(); + return Poll::Ready(Step::Ended(name.clone(), *source, result, delivered)); } // Watch the demand edge in whichever direction is unmet. - let edge = match io.used { - true => io.resume.poll_unused(waiter), - false => io.resume.poll_used(waiter), - }; - if edge.is_ready() { + match io.used { + true => io.weak.poll_unused(waiter), + false => io.weak.poll_used(waiter), + } + if io.weak.is_used() != io.used { return Poll::Ready(Step::Demand(name.clone())); } } @@ -2914,19 +2670,20 @@ async fn run_front(task: FrontTask) { .await; let event = match step { - Step::Assigned(name, resume) => { + Step::Assigned(request) => { + let name: Arc = request.name().into(); tracks.insert( name.clone(), TrackIo { - resume, - staged: None, + weak: request.weak(), + request: Some(request), + accepted: None, + routes: super::resume::Producer::new(), query: None, + staged: None, copy: None, - edge: None, - warm: None, - head: None, held: None, - dead: None, + closed: false, used: false, }, ); @@ -2961,11 +2718,11 @@ async fn run_front(task: FrontTask) { Step::Info(name, source, result) => { let closing = sources.get(&source).is_some_and(|s| s.is_closing()); let Some(io) = tracks.get_mut(&name) else { continue }; - let Some((_, copy, _)) = io.query.take() else { continue }; + let Some((asked, _)) = io.query.take() else { continue }; // A copy that is already aborted cannot be spliced; its error is // the source's answer for the track. let result = match result { - Ok(info) => match copy.poll_complete(&kio::Waiter::noop()) { + Ok(info) => match asked.copy.poll_complete(&kio::Waiter::noop()) { Poll::Ready(Err(err)) => Err(err), _ => Ok(info), }, @@ -2973,8 +2730,13 @@ async fn run_front(task: FrontTask) { }; // Staged only while the track has a reader: without one the machine // will not splice, and a held copy would keep the source subscribed. - if result.is_ok() && io.used { - io.staged = Some((source, copy)); + if let Ok(info) = &result + && io.used + { + io.staged = Some(Asked { + info: Some(info.clone()), + ..asked + }); } Event::TrackInfo { track: name, @@ -2983,11 +2745,12 @@ async fn run_front(task: FrontTask) { result, } } - Step::Ended(name, source, result) => { + Step::Ended(name, source, result, delivered) => { let closing = sources.get(&source).is_some_and(|s| s.is_closing()); let Some(io) = tracks.get_mut(&name) else { continue }; - io.bury(); - let delivered = io.resume.resume_position() != io.edge; + io.copy = None; + io.held = None; + io.closed = matches!(result, Err(Error::Closed)); Event::TrackEnded { track: name, source, @@ -2996,15 +2759,9 @@ async fn run_front(task: FrontTask) { delivered, } } - Step::Held(name, group) => { - if let Some(io) = tracks.get_mut(&name) { - io.held = Some(HeldGroup(group)); - } - continue; - } Step::Demand(name) => { let Some(io) = tracks.get_mut(&name) else { continue }; - io.used = io.resume.is_used(); + io.used = io.weak.is_used(); if !io.used { // Nothing will be spliced now: let go of the copies a query // holds, or the source stays subscribed with nobody reading. @@ -3364,10 +3121,10 @@ struct OriginState { cursors: HashMap, // The remotely-served fronts, keyed by absolute path and the requester's - // split-horizon exclusion. Each is a spliced broadcast whose watcher task - // materializes it from the best covering route and re-splices it through - // routes sharing its first hop, so a route change the identity survives is - // invisible to subscribers. Keyed per exclusion so a front's failover can + // split-horizon exclusion. Each is a broadcast whose watcher task + // materializes it from the best covering route and switches it between + // routes, so a route change is invisible to subscribers. Keyed per exclusion + // so a front's failover can // never adopt a route flowing back through one of its own readers. Weak, so // a front dies with its watcher and a later request re-creates it. fronts: WeakCache, @@ -3605,12 +3362,10 @@ impl OriginState { /// prefix, the cheapest served one is picked by [`route_order`]. /// /// Only announced routes are candidates: an unannounced broadcast serves - /// nobody, and does not shadow anything either. `pin` is the front's - /// identity: only routes it admits are candidates, since a route from anyone - /// else is different content rather than an alternate path (see [`Front`]). - /// A broadcast published on this origin competes on cost like any other - /// route and wins a tie. - fn best_route(&self, path: &Path, horizon: Horizon, pin: Pin, refused: &HashSet) -> Option<&RouteEntry> { + /// nobody, and does not shadow anything either. Routes `refused` for the + /// front's path are skipped. A broadcast published on this origin competes + /// on cost like any other route and wins a tie. + fn best_route(&self, path: &Path, horizon: Horizon, refused: &HashSet) -> Option<&RouteEntry> { // Covering prefixes of one path form a chain, so the deepest node with a // candidate holds the unique longest prefix; walking down, the last such // node decides. @@ -3623,7 +3378,6 @@ impl OriginState { .filter(|entry| entry.live()) .filter(|entry| entry.scope.matches(path.as_str())) .filter(|entry| horizon.admits(entry)) - .filter(|entry| entry.qualifies(pin)) .filter(|entry| !refused.contains(&entry.id)) .peekable(); if candidates.peek().is_some() { @@ -3743,10 +3497,9 @@ impl Dynamic { /// Re-price the route in place: replace its hops and cost. /// /// Consumers observe another active update for the same prefix; sessions - /// forward it as a restart, so route churn never looks like new content. A - /// new first hop is a new publisher: broadcasts already served from the old - /// one drain, and later requests reach the handler again. The prefix is fixed - /// at announce time: to move a route, drop this and call + /// forward it as a restart, so route churn never looks like new content. + /// Broadcasts already served through the route keep serving whatever its hops + /// now say. The prefix is fixed at announce time: to move a route, drop this and call /// [`Producer::dynamic`] again. Fails with [`Error::Closed`] once the origin's /// [`Driver`] has been dropped. pub fn update(&self, route: Route) -> Result<(), Error> { @@ -4460,12 +4213,11 @@ impl Consumer { /// the best announced route covering it (the most specific prefix, then the /// cheapest, a broadcast published on this origin winning ties) and /// materializes it, from the broadcast itself or from the peer that - /// announced the route. When its serving source dies or a better qualifying - /// route appears, the front re-splices through the best route sharing its - /// first hop at a group boundary, invisibly to subscribers. A change that - /// does not preserve the first hop ends the broadcast instead, as does its - /// route retracting with no replacement, and the next request re-serves the - /// path. Tracks already in flight carry on to their own end. + /// announced the route. When its serving source dies or a better route + /// appears, the front switches to the best remaining route, invisibly to + /// subscribers: a path names one broadcast, whoever serves it. Its route + /// retracting with no replacement ends the broadcast, and the next request + /// re-serves the path. Tracks already in flight carry on to their own end. /// /// The returned future fails with [`Error::Unroutable`] at once when no /// announced route covers the path, including a broadcast created on this @@ -4508,7 +4260,7 @@ impl Consumer { // Checked before joining a front, so a front still draining after its // route retracted takes no newcomers. if state - .best_route(&absolute.as_path(), self.horizon, Pin::Any, &HashSet::new()) + .best_route(&absolute.as_path(), self.horizon, &HashSet::new()) .is_none() { return kio::Pending::new(Requesting::failed(Error::Unroutable)); @@ -4516,31 +4268,21 @@ impl Consumer { // Join the live front for this path and exclusion, if any: its watcher // resolves (or already resolved) the request channel with the front's - // spliced broadcast, so repeat requests share one upstream - // subscription. Only while the best route still serves the front's - // content, though: once a different publisher wins (a cheaper route), a - // newcomer gets a fresh front from it, and the old front keeps serving the - // readers it has, since other content can't be spliced into it. + // broadcast, so repeat requests share one upstream subscription. Whichever + // route serves the path, it is the same broadcast. let key = (absolute.clone(), self.horizon); if let Some(front) = state.fronts.get(&key) { - let pin = *front.pin.lock(); - let current = state - .best_route(&absolute.as_path(), self.horizon, Pin::Any, &HashSet::new()) - .is_some_and(|entry| entry.qualifies(pin)); - if current { - let pending = Requesting::queued(front.request.consume()) - .with_path(requested) - .with_stats(scope); - return kio::Pending::new(pending); - } - state.fronts.remove(&key); + let pending = Requesting::queued(front.request.consume()) + .with_path(requested) + .with_stats(scope); + return kio::Pending::new(pending); } // A route covers the path: mint the front and hand its watcher the // request. The watcher materializes the path from the best covering - // route, resolves the channel, and re-splices the front through - // routes sharing its first hop for as long as one serves. - let broadcast = broadcast::Producer::new_spliced(broadcast::Info { + // route, resolves the channel, and switches the front between routes for + // as long as one serves. + let broadcast = broadcast::Producer::new(broadcast::Info { pool: self.pool.clone(), cache_duration: self.cache_duration, path: absolute.clone(), @@ -4548,13 +4290,11 @@ impl Consumer { let request = kio::Producer::::default(); let consumer = request.consume(); let watch = state.watch(&self.shared, &absolute); - let pin = kio::Lock::new(Pin::Any); state.fronts.insert( key, RemoteFront { request: request.clone(), broadcast: broadcast.consume().weak(), - pin: pin.clone(), }, ); // Released before the push: a set whose handles are gone drops the task, @@ -4567,7 +4307,6 @@ impl Consumer { horizon: self.horizon, watch, request, - pin, timers: self.timers.clone(), })); kio::Pending::new(Requesting::queued(consumer).with_path(requested).with_stats(scope)) @@ -5933,11 +5672,10 @@ mod tests { pending.await.expect("resolves through the cheaper route"); } - /// A cheaper route that appears after a front was minted wins new requests too: - /// the cached front serves other content, so a newcomer gets a fresh front from - /// the winner, while the old front keeps serving the readers it already has. + /// A cheaper route that appears after a front was minted takes the front over: the + /// path is one broadcast whoever serves it, so a newcomer joins the front. #[tokio::test] - async fn cheaper_route_after_a_front_wins_new_requests() { + async fn cheaper_route_after_a_front_takes_it_over() { let producer = origin(1).produce(); let consumer = producer.consume(); @@ -5950,15 +5688,12 @@ mod tests { let server = producer .dynamic("room/alice", Route::default().with_hops(hops(&[10])).with_cost(1)) .unwrap(); - - let pending = consumer.request_broadcast("room/alice"); let request = queued(&server).await; let upstream = broadcast::Info::new().produce(); request.accept(&upstream); - let second = pending.await.expect("resolves through the cheaper route"); - assert!(!first.is_closed(), "the old front must keep serving its readers"); - assert!(!first.is_clone(&second), "the newcomer must not join the old front"); + let second = consumer.request_broadcast("room/alice").await.expect("resolves"); + assert!(first.is_clone(&second), "the newcomer joins the front"); } /// At equal cost the local broadcast wins even over a route with no hops of its @@ -6109,124 +5844,6 @@ mod tests { (server, upstream, dynamic, resolved) } - /// A reader returning to a parked track waits for the fresh copy to resolve its - /// start, and skips the warm cache when the copy resolves past it: the source - /// judged the groups in between stale, so the older cache is stale too. Without the - /// hold the reader was handed the whole warm cache first, seconds behind live. - #[tokio::test] - async fn returning_reader_skips_a_warm_cache_the_copy_resolved_past() { - let ms = |v: u64| crate::Timestamp::from_millis(v).unwrap(); - let (_server, _upstream, mut dynamic, resolved) = served_front().await; - let budget = track::Subscription::default().with_max_age(Duration::from_millis(100)); - - let track = resolved.track("audio").unwrap(); - let b = budget.clone(); - let subscribing = tokio::spawn(async move { track.subscribe(b).await }); - let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) - .await - .expect("the front asked the source") - .expect("request"); - let source = request.resolving_start().accept(None); - for seq in 0..4u64 { - let mut group = source.create_group(seq.into()).unwrap(); - group.write_frame(ms(seq * 20), b"old".as_ref()).unwrap(); - group.finish().unwrap(); - } - let mut subscription = subscribing.await.unwrap().expect("subscribe"); - subscription.recv_group().await.unwrap().expect("the live group"); - drop(subscription); - tokio::time::timeout(Duration::from_secs(1), source.demand().unused()) - .await - .expect("parked") - .expect("source open"); - drop(source); - - let track = resolved.track("audio").unwrap(); - let subscribing = tokio::spawn(async move { track.subscribe(budget).await }); - let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) - .await - .expect("the front asked the source again") - .expect("request"); - let mut source = request.resolving_start().accept(None); - let mut subscription = subscribing.await.unwrap().expect("resubscribe"); - - // The copy has not resolved its start: nothing is handed out yet. - assert!( - tokio::time::timeout(Duration::from_millis(50), subscription.recv_group()) - .await - .is_err(), - "the warm cache was served before the copy resolved its start" - ); - - // The source resolves past the floor (lite-06 skipped 4..20 as stale). - source.start_at(20).unwrap(); - let mut group = source.create_group(20u64.into()).unwrap(); - group.write_frame(ms(2000), b"new".as_ref()).unwrap(); - group.finish().unwrap(); - let group = subscription.recv_group().await.unwrap().expect("the live group"); - assert_eq!(group.sequence, 20, "a stale warm group was served"); - } - - /// A warm cache whose newest group finished still resumes when the source has - /// nothing newer: the re-splice asks for that group's tail, which a source that - /// resolves starts lazily (with its first served group) can answer at once. Asking - /// past it left a returning catalog reader waiting for the next catalog change. - #[tokio::test] - async fn returning_reader_replays_a_current_warm_cache() { - let (_server, _upstream, mut dynamic, resolved) = served_front().await; - - let track = resolved.track("catalog").unwrap(); - let subscribing = tokio::spawn(async move { track.subscribe(None).await }); - let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) - .await - .expect("the front asked the source") - .expect("request"); - let source = request.resolving_start().accept(None); - let mut group = source.create_group(0u64.into()).unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"snapshot".as_ref()).unwrap(); - group.finish().unwrap(); - let mut subscription = subscribing.await.unwrap().expect("subscribe"); - subscription.recv_group().await.unwrap().expect("the catalog"); - drop(subscription); - tokio::time::timeout(Duration::from_secs(1), source.demand().unused()) - .await - .expect("parked") - .expect("source open"); - drop(source); - - let track = resolved.track("catalog").unwrap(); - let subscribing = tokio::spawn(async move { track.subscribe(None).await }); - let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) - .await - .expect("the front asked the source again") - .expect("request"); - let mut source = request.resolving_start().accept(None); - let mut subscription = subscribing.await.unwrap().expect("resubscribe"); - - // The source still has group 0 as its newest: it serves the empty tail, and - // that is when its start resolves. - let reading = tokio::spawn(async move { - let mut group = subscription.recv_group().await.unwrap().expect("the catalog"); - assert_eq!(group.sequence, 0); - group.read_frame().await.unwrap().expect("the snapshot").payload - }); - tokio::task::yield_now().await; - assert_eq!( - source.subscription().and_then(|sub| sub.start), - Some(track::Position { group: 0, frame: 1 }), - "the re-splice asked past the cached catalog" - ); - source.start_at(0).unwrap(); - let mut tail = source.create_group(0u64.into()).unwrap(); - tail.start_at(1).unwrap(); - tail.finish().unwrap(); - let payload = tokio::time::timeout(Duration::from_secs(1), reading) - .await - .expect("the returning reader never got the catalog") - .unwrap(); - assert_eq!(&payload[..], b"snapshot"); - } - /// A finished track stays readable from the front while it is read and for the /// linger after, then leaves the broadcast so it stops pinning its cache: the next /// reader asks the source afresh. @@ -6278,7 +5895,7 @@ mod tests { ); // Paused time runs the front's earlier deadline before this sleep returns. - tokio::time::sleep(TRACK_IDLE_LINGER).await; + tokio::time::sleep(track::IDLE_LINGER).await; let track = resolved.track("catalog").unwrap(); let _subscribing = tokio::spawn(async move { track.subscribe(None).await }); @@ -6288,85 +5905,14 @@ mod tests { .expect("request"); } - /// A group that stays open for good (a JSON log in group 0) survives a park: the - /// returning reader gets the frames delivered before it from the warm cache, and the - /// re-splice asks the source only for the frames after them, across repeated parks. - /// A datagram sequenced past the group does not hide it as the live edge. - #[tokio::test] - async fn returning_reader_continues_an_open_warm_group() { - let (_server, _upstream, mut dynamic, resolved) = served_front().await; - - async fn read(group: &mut group::Consumer) -> Vec { - let frame = tokio::time::timeout(Duration::from_secs(1), group.read_frame()) - .await - .expect("frame") - .unwrap() - .expect("group ended"); - frame.payload.to_vec() - } - - let mut expect: Vec<&[u8]> = Vec::new(); - let mut floor: Option = None; - for (round, payload) in [b"a".as_ref(), b"b", b"c"].into_iter().enumerate() { - let track = resolved.track("log").unwrap(); - let subscribing = tokio::spawn(async move { track.subscribe(None).await }); - let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) - .await - .expect("the front asked the source") - .expect("request"); - let mut source = request.resolving_start().accept(None); - let mut subscription = subscribing.await.unwrap().expect("subscribe"); - - // The source resolves at the floor's group, continuing group 0. - source.start_at(0).unwrap(); - let mut group = source.create_group(0u64.into()).unwrap(); - if let Some(floor) = floor { - group.start_at(floor.frame).unwrap(); - } - group.write_frame(crate::Timestamp::ZERO, payload).unwrap(); - expect.push(payload); - source - .insert_datagram(10, crate::Timestamp::ZERO, b"datagram".as_ref()) - .unwrap(); - - let mut reading = tokio::time::timeout(Duration::from_secs(1), subscription.recv_group()) - .await - .expect("group 0") - .unwrap() - .expect("track ended"); - assert_eq!(reading.sequence, 0); - for frame in &expect { - assert_eq!(read(&mut reading).await, *frame, "round {round}"); - } - assert_eq!( - source.subscription().and_then(|sub| sub.start), - floor, - "round {round} asked for the wrong continuation" - ); - - drop(reading); - drop(subscription); - tokio::time::timeout(Duration::from_secs(1), source.demand().unused()) - .await - .expect("parked") - .expect("source open"); - drop(group); - drop(source); - floor = Some(track::Position { - group: 0, - frame: expect.len() as u64, - }); - } - } - - /// A route that ends abruptly releases its open groups, and the takeover asks the - /// next route only for the frames past the break. The front keeps the head it - /// spliced, so a reader arriving afterwards still gets the whole group: whether the - /// old route dies before the new one takes over (a reconnect) or after (a cheaper - /// route preempting it). A reader joining mid-outage gets it too, and the reader - /// already mid-group is not handed it a second time. + /// A route that ends abruptly releases its open groups, and the next route may + /// deliver only the frames past the break. The reader mid-group carries on from + /// there, and a reader arriving afterwards fetches the whole group from that route: + /// whether the old route dies before the new one takes over (a reconnect) or after + /// (a cheaper route preempting it). A reader joining mid-outage gets it too, and the + /// reader already mid-group is not handed it a second time. #[tokio::test] - async fn a_takeover_keeps_the_open_group_head_for_later_readers() { + async fn a_takeover_serves_the_open_group_head_to_later_readers() { tokio::time::pause(); for dies_first in [true, false] { takeover_keeps_the_open_group_head(dies_first).await; @@ -6384,7 +5930,7 @@ mod tests { queued(&first_server).await.accept(&upstream); let resolved = pending.await.unwrap(); - async fn read(group: &mut group::Consumer) -> Vec { + async fn read(group: &mut crate::group::Consumer) -> Vec { let frame = tokio::time::timeout(Duration::from_secs(1), group.read_frame()) .await .expect("frame") @@ -6428,26 +5974,38 @@ mod tests { let replacement = broadcast::Info::new().produce(); let mut replacement_dynamic = replacement.dynamic(); queued(&replacement_server).await.accept(&replacement); - let mut next = tokio::time::timeout(Duration::from_secs(1), replacement_dynamic.requested_track()) + let request = tokio::time::timeout(Duration::from_secs(1), replacement_dynamic.requested_track()) .await .expect("the front asked the replacement") - .unwrap() - .resolving_start() - .accept(None); + .unwrap(); + // The replacement serves the whole group to a fetch, frames still to come included. + let fetches = request.dynamic(); + let fetched = Arc::new(std::sync::Mutex::new(Vec::::new())); + let serving = fetched.clone(); + let _fetching = tokio::spawn(async move { + while let Ok(request) = fetches.requested_group().await { + let mut group = request.accept(None).unwrap(); + for frame in [b"a", b"b", b"c"] { + group.write_frame(crate::Timestamp::ZERO, frame.as_ref()).unwrap(); + } + serving.lock().unwrap().push(group); + } + }); + let mut next = request.resolving_start().accept(None); next.start_at(0).unwrap(); let mut next_group = next.create_group(0u64.into()).unwrap(); next_group.start_at(2).unwrap(); next_group.write_frame(crate::Timestamp::ZERO, b"c".as_ref()).unwrap(); assert_eq!(read(&mut reading).await, b"c", "dies_first={dies_first}"); - // A preempted route still owns its open group: its writer and other readers carry on. + // A preempted route still owns its open group: its writer and other readers carry + // on, and its copy of the frame the replacement already wrote is a duplicate. if let Some((group, ..)) = &mut first { let mut independent = group.consume(); - group.write_frame(crate::Timestamp::ZERO, b"x".as_ref()).unwrap(); - for expect in [b"a", b"b", b"x"] { + group.write_frame(crate::Timestamp::ZERO, b"c".as_ref()).unwrap(); + for expect in [b"a", b"b", b"c"] { assert_eq!(read(&mut independent).await, expect); } - group.finish().unwrap(); } drop(first); @@ -6488,6 +6046,9 @@ mod tests { // The continuation still flows to both. next_group.write_frame(crate::Timestamp::ZERO, b"d".as_ref()).unwrap(); + for group in fetched.lock().unwrap().iter_mut() { + group.write_frame(crate::Timestamp::ZERO, b"d".as_ref()).unwrap(); + } assert_eq!(read(&mut reading).await, b"d"); assert_eq!(read(&mut late).await, b"d"); if let Some((_, group)) = &mut joined { @@ -6507,220 +6068,66 @@ mod tests { )); } - /// A returning reader is handed the parked open group spliced onto the source's - /// continuation, which starts past the warm head. The group's end stays unknown - /// until the source finishes it, even before a frame of it is read: readers drop a - /// group ahead of their cursor whose end resolves to an error. + /// The same holds for a reader returning to a parked track: its warm cache + /// does not stand in for the copy it is waiting on. #[tokio::test] - async fn returning_reader_gets_a_live_open_warm_group() { - use futures::FutureExt; - - tokio::time::pause(); - let (_server, _upstream, mut dynamic, resolved) = served_front().await; + async fn unannounce_keeps_a_returning_reader_awaiting_its_info() { + let producer = origin(1).produce(); + let consumer = producer.consume(); - // The first reader leaves mid-group. + let broadcast = producer.publish("room/alice", Route::default()).unwrap(); + let mut dynamic = broadcast.dynamic(); + let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); let track = resolved.track("video").unwrap(); let subscribing = tokio::spawn(async move { track.subscribe(None).await }); let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) .await .expect("the front asked the source") .expect("request"); - let mut source = request.resolving_start().accept(None); + let source = request.accept(None); + let mut group = source.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"cached".as_ref()).unwrap(); + group.finish().unwrap(); let mut subscription = subscribing.await.unwrap().expect("subscribe"); - source.start_at(0).unwrap(); - let mut group = source.create_group(0u64.into()).unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"a".as_ref()).unwrap(); - let mut reading = subscription.recv_group().await.unwrap().expect("group 0"); - assert_eq!(reading.read_frame().await.unwrap().unwrap().payload.as_ref(), b"a"); - drop(reading); + subscription.recv_group().await.unwrap().expect("the cached group"); drop(subscription); + + // Parked: the source copy goes, the delivered group stays warm. The source + // then tears its idle track down, so a returning reader asks it afresh. tokio::time::timeout(Duration::from_secs(1), source.demand().unused()) .await .expect("parked") .expect("source open"); - drop(group); drop(source); - - // The next reader returns while the group is still live upstream. let track = resolved.track("video").unwrap(); let subscribing = tokio::spawn(async move { track.subscribe(None).await }); let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) .await - .expect("the front asked the source") + .expect("the front asked the source again") .expect("request"); - let mut source = request.resolving_start().accept(None); - let mut subscription = subscribing.await.unwrap().expect("subscribe"); - source.start_at(0).unwrap(); - let mut group = source.create_group(0u64.into()).unwrap(); - group.start_at(1).unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"b".as_ref()).unwrap(); - let mut reading = tokio::time::timeout(Duration::from_secs(1), subscription.recv_group()) - .await - .expect("group 0") - .unwrap() - .expect("track ended"); - assert_eq!(reading.sequence, 0); - let early = reading.finished().now_or_never(); - assert!( - early.is_none(), - "the live group already resolved its end before a frame was read: {early:?}" - ); + broadcast.unannounce(); + settle(|| resolved.is_closed()).await; - // The end answers for this reader's cursor, not the seam it was probed from. + // A fresh source copy numbers groups past what the cache already delivered. + let source = request.accept(None); + let mut group = source.create_group(1u64.into()).unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"late".as_ref()).unwrap(); group.finish().unwrap(); - assert_eq!(reading.finished().await.unwrap(), 0); - assert_eq!(reading.read_frame().await.unwrap().unwrap().payload.as_ref(), b"a"); - assert_eq!(reading.finished().await.unwrap(), 1); - assert_eq!(reading.read_frame().await.unwrap().unwrap().payload.as_ref(), b"b"); - assert_eq!(reading.finished().await.unwrap(), 2); + source.finish().unwrap(); + + let mut subscription = subscribing.await.unwrap().expect("subscribe survives the retraction"); + let mut payloads = Vec::new(); + while let Some(mut group) = subscription.recv_group().await.expect("ends cleanly") { + payloads.push(group.read_frame().await.unwrap().unwrap().payload); + } + assert_eq!(payloads.last().map(|p| &p[..]), Some(&b"late"[..])); } + /// A re-announce that lands before the front acts on the retraction reuses + /// the same route entry, so the front carries on. #[tokio::test] - async fn warm_head_survives_another_takeover_before_park() { - tokio::time::pause(); - let producer = origin(1).produce(); - let _server = producer - .dynamic("room/alice", Route::default().with_hops(hops(&[10])).with_cost(5)) - .unwrap(); - let pending = producer.consume().request_broadcast("room/alice"); - let upstream = broadcast::Info::new().produce(); - let mut dynamic = upstream.dynamic(); - queued(&_server).await.accept(&upstream); - let resolved = pending.await.unwrap(); - - let track = resolved.track("log").unwrap(); - let subscribing = tokio::spawn(async move { track.subscribe(None).await }); - let source = dynamic.requested_track().await.unwrap().accept(None); - let mut group = source.create_group(0u64.into()).unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"a".as_ref()).unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"b".as_ref()).unwrap(); - let mut subscription = subscribing.await.unwrap().unwrap(); - let mut reading = subscription.recv_group().await.unwrap().unwrap(); - assert_eq!(&reading.read_frame().await.unwrap().unwrap().payload[..], b"a"); - drop(reading); - drop(subscription); - source.demand().unused().await.unwrap(); - drop(group); - drop(source); - - let track = resolved.track("log").unwrap(); - let subscribing = tokio::spawn(async move { track.subscribe(None).await }); - let mut resumed = dynamic.requested_track().await.unwrap().resolving_start().accept(None); - resumed.start_at(0).unwrap(); - let mut subscription = subscribing.await.unwrap().unwrap(); - let mut reading = subscription.recv_group().await.unwrap().unwrap(); - assert_eq!(&reading.read_frame().await.unwrap().unwrap().payload[..], b"a"); - - let replacement_server = producer - .dynamic("room/alice", Route::default().with_hops(hops(&[10]))) - .unwrap(); - let replacement = broadcast::Info::new().produce(); - let mut replacement_dynamic = replacement.dynamic(); - queued(&replacement_server).await.accept(&replacement); - let mut source = replacement_dynamic - .requested_track() - .await - .unwrap() - .resolving_start() - .accept(None); - source.start_at(0).unwrap(); - let mut group = source.create_group(0u64.into()).unwrap(); - group.start_at(2).unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"c".as_ref()).unwrap(); - let mut continuation = reading.clone(); - continuation.start_at(2); - let frame = tokio::time::timeout(Duration::from_secs(1), continuation.read_frame()) - .await - .expect("replacement frame") - .unwrap() - .unwrap(); - assert_eq!(&frame.payload[..], b"c"); - drop(continuation); - assert_eq!(&reading.read_frame().await.unwrap().unwrap().payload[..], b"b"); - assert_eq!(&reading.read_frame().await.unwrap().unwrap().payload[..], b"c"); - drop(reading); - drop(subscription); - source.demand().unused().await.unwrap(); - drop(group); - drop(source); - - let track = resolved.track("log").unwrap(); - let subscribing = tokio::spawn(async move { track.subscribe(None).await }); - let mut next = replacement_dynamic - .requested_track() - .await - .unwrap() - .resolving_start() - .accept(None); - next.start_at(0).unwrap(); - let mut subscription = subscribing.await.unwrap().unwrap(); - let mut reading = subscription.recv_group().await.unwrap().unwrap(); - for expected in [b"a", b"b", b"c"] { - assert_eq!(&reading.read_frame().await.unwrap().unwrap().payload[..], expected); - } - } - - /// The same holds for a reader returning to a parked track: its warm cache - /// does not stand in for the copy it is waiting on. - #[tokio::test] - async fn unannounce_keeps_a_returning_reader_awaiting_its_info() { - let producer = origin(1).produce(); - let consumer = producer.consume(); - - let broadcast = producer.publish("room/alice", Route::default()).unwrap(); - let mut dynamic = broadcast.dynamic(); - let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); - let track = resolved.track("video").unwrap(); - let subscribing = tokio::spawn(async move { track.subscribe(None).await }); - let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) - .await - .expect("the front asked the source") - .expect("request"); - let source = request.accept(None); - let mut group = source.append_group().unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"cached".as_ref()).unwrap(); - group.finish().unwrap(); - let mut subscription = subscribing.await.unwrap().expect("subscribe"); - subscription.recv_group().await.unwrap().expect("the cached group"); - drop(subscription); - - // Parked: the source copy goes, the delivered group stays warm. The source - // then tears its idle track down, so a returning reader asks it afresh. - tokio::time::timeout(Duration::from_secs(1), source.demand().unused()) - .await - .expect("parked") - .expect("source open"); - drop(source); - let track = resolved.track("video").unwrap(); - let subscribing = tokio::spawn(async move { track.subscribe(None).await }); - let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) - .await - .expect("the front asked the source again") - .expect("request"); - - broadcast.unannounce(); - settle(|| resolved.is_closed()).await; - - // A fresh source copy numbers groups past what the cache already delivered. - let source = request.accept(None); - let mut group = source.create_group(1u64.into()).unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"late".as_ref()).unwrap(); - group.finish().unwrap(); - source.finish().unwrap(); - - let mut subscription = subscribing.await.unwrap().expect("subscribe survives the retraction"); - let mut payloads = Vec::new(); - while let Some(mut group) = subscription.recv_group().await.expect("ends cleanly") { - payloads.push(group.read_frame().await.unwrap().unwrap().payload); - } - assert_eq!(payloads.last().map(|p| &p[..]), Some(&b"late"[..])); - } - - /// A re-announce that lands before the front acts on the retraction reuses - /// the same route entry, so the front carries on. - #[tokio::test] - async fn reannounce_before_the_front_acts_keeps_it() { + async fn reannounce_before_the_front_acts_keeps_it() { let producer = origin(1).produce(); let consumer = producer.consume(); @@ -7795,7 +7202,7 @@ mod tests { let replacement = broadcast::Info::new().produce(); let track = replacement.create_track("video", None).unwrap(); // The same content: group 0 was already delivered through the old route, - // so the splice resumes at group 1. + // so the track resumes at group 1. let mut group = track.append_group().unwrap(); group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); group.finish().unwrap(); @@ -7829,19 +7236,6 @@ mod tests { drop(consumer); } - #[tokio::test] - async fn remote_source_resumes_through_same_first_hop() { - let (mut rig, incumbent, source) = ResumeRig::new(&[10]).await; - let standby_server = rig.standby(&[10, 20]); - - // The serving route dies: retraction plus source abort, like a session. - drop(incumbent); - drop(source); - - // The standby shares the first hop, so the subscription resumes there. - assert_resumes(&mut rig, &standby_server).await; - } - /// A source claiming the same content cannot change immutable track metadata: /// the successor is refused instead of the subscriber's samples being read on /// a different grid, and the verdict outlives the aborted logical track. @@ -7857,8 +7251,9 @@ mod tests { drop(incumbent); drop(source); - // The standby shares the first hop, so the front re-requests through it, - // but its copy of the track is on another grid. + // The front re-requests through the standby, but its copy of the track is + // on another grid. The incumbent's copy serves until it runs out, then the + // refusal stands. let request = queued(&standby_server).await; let successor = broadcast::Info::new().produce(); let track = successor.create_track("video", replacement).unwrap(); @@ -7866,6 +7261,7 @@ mod tests { group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); group.finish().unwrap(); request.accept(&successor); + rig.incumbent_track.clone().abort(Error::Dropped).unwrap(); assert!( matches!(rig.subscription.recv_group().await, Err(Error::Unsupported)), @@ -7879,124 +7275,121 @@ mod tests { } } + /// A path is one broadcast whoever publishes it: when the serving route dies, the + /// subscription resumes through another route, whatever its first hop, or with + /// none at all. #[tokio::test] - async fn different_first_hop_ends_the_subscription() { - let (mut rig, incumbent, source) = ResumeRig::new(&[10]).await; - // Another publisher entirely: same path, different first hop. - let rival_server = rig.standby(&[11]); - - // The incumbent's session dies, taking its track with it: a live copy - // would otherwise keep serving after the front ends. - drop(incumbent); - drop(source); - rig.incumbent_track.abort(Error::Dropped).unwrap(); + async fn any_route_at_the_path_resumes_the_subscription() { + let cases = [ + (&[10][..], &[10, 20][..]), + (&[10][..], &[11][..]), + (&[][..], &[][..]), + (&[0][..], &[12][..]), + ]; + for (first, other) in cases { + let (mut rig, incumbent, source) = ResumeRig::new(first).await; + let standby = rig.standby(other); - // The subscription ends rather than splicing onto the rival's frames. - let err = rig.subscription.recv_group().await.err().expect("subscription ends"); - assert!(matches!(err, Error::Dropped), "unexpected end: {err}"); + drop(incumbent); + drop(source); + rig.incumbent_track.clone().abort(Error::Dropped).unwrap(); - // A fresh request resolves through the rival. - let consumer = rig.producer.consume(); - let pending = consumer.request_broadcast("room/alice"); - let request = queued(&rival_server).await; - let replacement = broadcast::Info::new().produce(); - request.accept(&replacement); - pending.await.expect("re-request resolves through the rival"); + assert_resumes(&mut rig, &standby).await; + } } - /// A route updated in place to a new first hop names a new publisher, whether the - /// front started anonymous or named. The in-flight subscription drains the old copy - /// until it ends and never splices the new one; a new request gets a fresh front - /// through the new publisher, free of the old broadcast's track info. + /// A route updated in place to a new first hop still serves the same broadcast: + /// the subscription carries on through it, and a new request joins the front. #[tokio::test] - async fn a_first_hop_update_drains_the_old_publisher() { + async fn a_first_hop_update_keeps_serving() { for first in [&[][..], &[0][..], &[10][..]] { let (mut rig, server, _source) = ResumeRig::new(first).await; + let standby = rig.standby(&[10, 20]); server.update(Route::default().with_hops(hops(&[11]))).unwrap(); - // The old copy keeps flowing to the subscription already reading it. let mut group = rig.incumbent_track.append_group().unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"draining".as_ref()).unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"after".as_ref()).unwrap(); group.finish().unwrap(); let mut group = next_group(&mut rig.subscription) .await - .expect("the in-flight subscription survives the update") - .expect("track ended early"); - let frame = group.read_frame().await.expect("read frame").expect("frame"); - assert_eq!(&frame.payload[..], b"draining", "first hop {first:?}"); - - // A new request is a new broadcast from the new publisher, whose track info - // differs from what the old front cached. - let consumer = rig.producer.consume(); - let pending = consumer.request_broadcast("room/alice"); - let request = queued(&server).await; - let replacement = broadcast::Info::new().produce(); - let track = replacement - .create_track("video", track::Info::default().with_priority(7)) - .unwrap(); - let mut group = track.append_group().unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"new".as_ref()).unwrap(); - group.finish().unwrap(); - request.accept(&replacement); - - let resolved = pending.await.expect("resolves through the new publisher"); - assert!( - !resolved.is_clone(&rig.resolved), - "first hop {first:?} joined the old front" - ); - let mut fresh = resolved - .track("video") - .unwrap() - .subscribe(None) - .await - .expect("the old publisher's track info does not apply"); - let mut group = next_group(&mut fresh) - .await - .expect("recv group") + .expect("the subscription survives the update") .expect("track ended early"); let frame = group.read_frame().await.expect("read frame").expect("frame"); - assert_eq!(&frame.payload[..], b"new"); + assert_eq!(&frame.payload[..], b"after", "first hop {first:?}"); - // The old copy ending ends the drained subscription, without a group from - // the new publisher. - rig.incumbent_track.finish().unwrap(); - let end = next_group(&mut rig.subscription).await; + let resolved = rig.producer.consume().request_broadcast("room/alice").await.unwrap(); + assert!(resolved.is_clone(&rig.resolved), "first hop {first:?} left the front"); assert!( - !matches!(end, Ok(Some(_))), - "first hop {first:?} spliced the new publisher into a live subscription" + standby.poll_requested_broadcast(&kio::Waiter::noop()).is_pending(), + "first hop {first:?} moved off a route that still serves" ); } } - /// A named front whose route moves to a new publisher resumes through another route - /// from its own publisher, never through the updated one. + /// A session never subscribes to itself: when the route serving a peer dies, its + /// front never resumes onto a route through that peer, even one from the same + /// publisher. A reader that can see the route resumes there. #[tokio::test] - async fn a_first_hop_update_resumes_through_the_same_publisher() { - let (mut rig, incumbent, _source) = ResumeRig::new(&[10]).await; - let standby_server = rig.standby(&[10, 20]); - - incumbent.update(Route::default().with_hops(hops(&[11]))).unwrap(); - - assert_resumes(&mut rig, &standby_server).await; - assert!( - incumbent.poll_requested_broadcast(&kio::Waiter::noop()).is_pending(), - "the front never asks the new publisher" - ); - } + async fn resume_never_routes_through_the_requester() { + for peer in [Some(origin(7)), None] { + let producer = origin(1).produce(); + let consumer = match peer { + Some(peer) => producer.consume().excluding(peer), + None => producer.consume(), + }; + let incumbent = producer + .dynamic("room", Route::default().with_hops(hops(&[10]))) + .unwrap(); + // The same publisher, reached through the requesting peer. + let echo = producer + .dynamic("room", Route::default().with_hops(hops(&[10, 7]))) + .unwrap(); - #[tokio::test] - async fn anonymous_routes_never_resume() { - // An empty hop chain identifies nobody, so two of them must not pass for - // one publisher reconnecting. - let (mut rig, incumbent, source) = ResumeRig::new(&[]).await; - let _twin_server = rig.standby(&[]); + let pending = consumer.request_broadcast("room/alice"); + let request = queued(&incumbent).await; + let source = broadcast::Info::new().produce(); + let track = source.create_track("video", None).unwrap(); + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); + group.finish().unwrap(); + request.accept(&source); - drop(incumbent); - drop(source); - rig.incumbent_track.abort(Error::Dropped).unwrap(); + let resolved = pending.await.expect("resolves through the incumbent"); + let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + let mut group = next_group(&mut subscription).await.unwrap().expect("first group"); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"before"); - let err = rig.subscription.recv_group().await.err().expect("subscription ends"); - assert!(matches!(err, Error::Dropped), "unexpected end: {err}"); + // The serving route dies, like a session. + drop(incumbent); + drop(source); + track.abort(Error::Dropped).unwrap(); + + match peer { + Some(_) => { + let err = next_group(&mut subscription).await.err().expect("subscription ends"); + assert!(matches!(err, Error::Dropped), "unexpected end: {err}"); + assert!( + echo.poll_requested_broadcast(&kio::Waiter::noop()).is_pending(), + "the front asked the requester for its own copy" + ); + } + None => { + let request = queued(&echo).await; + let replacement = broadcast::Info::new().produce(); + let track = replacement.create_track("video", None).unwrap(); + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); + group.finish().unwrap(); + request.accept(&replacement); + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"resumed".as_ref()).unwrap(); + group.finish().unwrap(); + + let mut group = next_group(&mut subscription).await.unwrap().expect("resumed group"); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"resumed"); + } + } + } } /// An anonymous publisher that dies without unannouncing is replaced by the @@ -8292,9 +7685,9 @@ mod tests { } /// An origin front drops the source track as soon as its last reader leaves, - /// so the publisher's `unused()` resolves far below `TRACK_IDLE_LINGER`. + /// so the publisher's `unused()` resolves far below `track::IDLE_LINGER`. /// Cached groups stay on the front for the linger; a returning reader - /// replays them and re-splices for groups past that edge. + /// replays them and asks the source again for groups past that edge. #[tokio::test] async fn origin_front_drops_the_source_when_unused() { let producer = origin(1).produce(); @@ -8320,7 +7713,7 @@ mod tests { tokio::time::timeout(Duration::from_secs(1), track.demand().unused()) .await - .expect("source unused should resolve far below TRACK_IDLE_LINGER") + .expect("source unused should resolve far below track::IDLE_LINGER") .expect("source closed"); // Cached groups stay on the front for the linger; a returning reader @@ -8354,81 +7747,81 @@ mod tests { assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"live"); } - /// A returning reader judges the warm cache against the logical track's live - /// edge, not the parked segment's own frozen one: groups the fresh source - /// has left behind by more than the budget are skipped, exactly as they would - /// be on one unspliced track. - /// - /// A local source is released rather than parked (it keeps its own cache), so - /// this goes through a served front, which is what actually holds the warm copy. - /// The copy resolves at the cached edge, not past it: resolving past it drops - /// the cache outright, which is a different case. - #[tokio::test] - async fn resumed_reader_skips_warm_groups_behind_the_new_edge() { - let ms = |v: u64| crate::Timestamp::from_millis(v).unwrap(); - let (_server, _upstream, mut dynamic, resolved) = served_front().await; - let budget = track::Subscription::default().with_max_age(Duration::from_millis(100)); - let write = |source: &track::Producer, sequence: u64, millis: u64| { - let mut group = source.create_group(sequence.into()).unwrap(); - group.write_frame(ms(millis), b"x".as_ref()).unwrap(); - group.finish().unwrap(); - }; - let drain = |subscription: &mut track::Subscriber| { - let mut sequences = Vec::new(); - while let Poll::Ready(group) = subscription.poll_recv_group(&kio::Waiter::noop()) { - sequences.push(group.unwrap().expect("track ended").sequence); - } - sequences - }; + /// A reader still holding an open group after its track goes unread gets the rest + /// of the group: parking the track reads no new group, but the upstream stays + /// subscribed until the held one ends. + #[tokio::test(start_paused = true)] + async fn a_held_group_continues_after_its_track_goes_unread() { + let producer = origin(1).produce(); + let server = producer + .dynamic("live", Route::default().with_hops(hops(&[10]))) + .unwrap(); + let pending = producer.consume().request_broadcast("live"); + let source = broadcast::Info::new().produce(); + let track = source.create_track("video", None).unwrap(); + queued(&server).await.accept(&source); + let resolved = pending.await.unwrap(); + + let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + let mut writing = track.append_group().unwrap(); + writing.write_frame(crate::Timestamp::ZERO, b"head".as_ref()).unwrap(); + let mut reading = subscription.recv_group().await.unwrap().unwrap(); + assert_eq!(&reading.read_frame().await.unwrap().unwrap().payload[..], b"head"); - let track = resolved.track("video").unwrap(); - let first = budget.clone(); - let subscribing = tokio::spawn(async move { track.subscribe(first).await }); - let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) - .await - .expect("the front asked the source") - .expect("request"); - let source = request.resolving_start().accept(None); - for (sequence, millis) in [(0, 0), (1, 20), (2, 40), (3, 60)] { - write(&source, sequence, millis); - } - let mut subscription = subscribing.await.unwrap().expect("subscribe"); - next_group(&mut subscription).await.unwrap().expect("a cached group"); - drain(&mut subscription); drop(subscription); + tokio::time::sleep(Duration::from_millis(10)).await; + writing.write_frame(crate::Timestamp::ZERO, b"tail".as_ref()).unwrap(); + writing.finish().unwrap(); - tokio::time::timeout(Duration::from_secs(1), source.demand().unused()) + let tail = tokio::time::timeout(Duration::from_secs(1), reading.read_frame()) .await - .expect("parked") - .expect("source open"); - drop(source); - - let track = resolved.track("video").unwrap(); - let second = budget.clone(); - let subscribing = tokio::spawn(async move { track.subscribe(second).await }); - let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) + .expect("the held group lost its source") + .unwrap() + .expect("the group ended early"); + assert_eq!(&tail.payload[..], b"tail"); + let end = tokio::time::timeout(Duration::from_secs(1), reading.read_frame()).await; + assert!(matches!(end, Ok(Ok(None))), "the group should finish: {end:?}"); + tokio::time::timeout(Duration::from_secs(1), track.demand().unused()) .await - .expect("the front asked the source again") - .expect("request"); - let mut source = request.resolving_start().accept(None); - for (sequence, millis) in [(20, 400), (21, 420), (22, 440)] { - write(&source, sequence, millis); - } - // At the cached edge, not past it, so the warm copy stays and the budget - // decides. Past it, the copy has already judged the cache stale. - source.start_at(3).unwrap(); - let mut subscription = subscribing.await.unwrap().expect("resubscribe"); - settle(|| subscription.latest() == Some(22)).await; + .expect("the front lets the source go once the held group ends") + .expect("source closed"); + } + + /// A source that aborts one group while its track carries on ends that group for + /// the reader too, rather than leaving it waiting for a route change that never + /// comes. + #[tokio::test(start_paused = true)] + async fn a_group_the_source_aborts_ends_while_the_track_lives() { + let producer = origin(1).produce(); + let broadcast = producer.publish("live", Route::default()).unwrap(); + let track = broadcast.create_track("video", None).unwrap(); + let resolved = producer.consume().request_broadcast("live").await.unwrap(); + let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + + let mut aborted = track.append_group().unwrap(); + aborted.write_frame(crate::Timestamp::ZERO, b"head".as_ref()).unwrap(); + let mut reading = subscription.recv_group().await.unwrap().unwrap(); + assert_eq!(&reading.read_frame().await.unwrap().unwrap().payload[..], b"head"); + aborted.abort(Error::Cancel).unwrap(); + + let end = tokio::time::timeout(Duration::from_secs(1), reading.read_frame()).await; + assert!(matches!(end, Ok(Err(_))), "the group should fail: {end:?}"); - // Groups 0..=2 reach at most 60ms against an edge at 440ms. Group 3 reaches - // where group 20 starts, 40ms behind that edge, inside the 100ms budget. - assert_eq!(drain(&mut subscription), [3, 20, 21, 22]); + // The track carries on. + let mut next = track.append_group().unwrap(); + next.write_frame(crate::Timestamp::ZERO, b"next".as_ref()).unwrap(); + next.finish().unwrap(); + let mut group = tokio::time::timeout(Duration::from_secs(1), subscription.recv_group()) + .await + .expect("the next group") + .unwrap() + .unwrap(); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"next"); } - /// A front serving from another front's spliced copy has no snapshot to keep: - /// it still drops upstream on the unused edge, so the publisher's `unused()` - /// resolves far below `TRACK_IDLE_LINGER` through the whole chain. The next - /// reader re-splices, paying `TRACK_INFO` again. + /// A front serving from another front still drops upstream on the unused edge, + /// so the publisher's `unused()` resolves far below `track::IDLE_LINGER` through + /// the whole chain, and a returning reader reaches the publisher's cache afresh. #[tokio::test] async fn chained_front_drops_the_source_when_unused() { let leaf = origin(1).produce(); @@ -8440,8 +7833,7 @@ mod tests { group.write_frame(crate::Timestamp::ZERO, b"cached".as_ref()).unwrap(); group.finish().unwrap(); - // The leaf's front view: a spliced broadcast, so any front serving from - // it holds a spliced source copy with nothing to snapshot. + // The leaf's front view, which the next front serves from. let leaf_front = leaf_consumer.request_broadcast("room/alice").await.expect("resolves"); let mid = origin(2).produce(); @@ -8469,16 +7861,9 @@ mod tests { tokio::time::timeout(Duration::from_secs(5), track.demand().unused()) .await - .expect("chained unused should resolve far below TRACK_IDLE_LINGER") + .expect("chained unused should resolve far below track::IDLE_LINGER") .expect("source closed"); - let cached = edge_resolved.track("video").unwrap().cached_groups(); - assert_eq!( - cached.iter().map(|(group, _)| group.sequence).collect::>(), - vec![0], - "every front keeps the delivered groups after releasing its source" - ); - // A budget spanning the cache, so the returning reader replays it. let mut subscription = edge_resolved .track("video") @@ -8503,16 +7888,14 @@ mod tests { tokio::time::timeout(Duration::from_secs(5), track.demand().unused()) .await - .expect("second chained unused should resolve far below TRACK_IDLE_LINGER") + .expect("second chained unused should resolve far below track::IDLE_LINGER") .expect("source closed"); - let cached = edge_resolved.track("video").unwrap().cached_groups(); - assert_eq!( - cached.iter().map(|(group, _)| group.sequence).collect::>(), - vec![0, 1], - "repeated demand keeps every complete group while releasing its source" - ); - + // A group the leaf produced while every front was parked: the fetch re-splices + // each hop to reach the leaf. + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"fetched".as_ref()).unwrap(); + group.finish().unwrap(); let fetch = edge_resolved.track("video").unwrap().fetch_group(2, None); let mut fetch = std::pin::pin!(fetch); assert!(futures::poll!(fetch.as_mut()).is_pending(), "fetch should re-splice"); @@ -8520,9 +7903,6 @@ mod tests { .await .expect("fetch should reach the leaf") .expect("source open"); - let mut group = track.append_group().unwrap(); - group.write_frame(crate::Timestamp::ZERO, b"fetched".as_ref()).unwrap(); - group.finish().unwrap(); let mut group = tokio::time::timeout(Duration::from_secs(5), fetch) .await .expect("re-spliced source should answer the fetch") @@ -8575,6 +7955,233 @@ mod tests { assert!(matches!(subscription.recv_group().await, Err(Error::Unsupported))); } + /// A remote front serving "room/alice" with one read track, "video", whose first + /// group was delivered. Returns the producer, the incumbent's track, and the reader. + async fn remote_front() -> ( + Producer, + Dynamic, + broadcast::Producer, + track::Producer, + track::Subscriber, + ) { + let producer = origin(1).produce(); + let server = producer + .dynamic("room", Route::default().with_hops(hops(&[10]))) + .unwrap(); + let pending = producer.consume().request_broadcast("room/alice"); + let upstream = broadcast::Info::new().produce(); + let old = upstream.create_track("video", None).unwrap(); + queued(&server).await.accept(&upstream); + let resolved = pending.await.expect("resolves"); + let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + let mut group = old.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); + group.finish().unwrap(); + let mut group = next_group(&mut subscription).await.unwrap().expect("first group"); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"before"); + (producer, server, upstream, old, subscription) + } + + /// A local newcomer takes over while its answer for "video" is pending, so the track + /// drains the incumbent. The reader leaves: the incumbent's copy is dropped at once, + /// like any unread track's, not held until the linger. + #[tokio::test(start_paused = true)] + async fn an_unread_draining_track_releases_the_old_source() { + let (producer, _server, _upstream, old, subscription) = remote_front().await; + + let newcomer = producer.create_broadcast("room/alice").unwrap(); + let mut handler = newcomer.dynamic(); + newcomer.announce(Route::default()).unwrap(); + let _unanswered = tokio::time::timeout(Duration::from_secs(1), handler.requested_track()) + .await + .expect("the front asked the newcomer") + .expect("request"); + + drop(subscription); + tokio::time::timeout(Duration::from_secs(1), old.demand().unused()) + .await + .expect("the draining source stays subscribed with nobody reading") + .expect("open"); + } + + /// The newcomer refuses "video", so the track drains the incumbent. The reader leaves + /// (the park drops the incumbent's copy), a third source that also lacks the track + /// takes over, and the reader returns: it gets an outcome rather than waiting on the + /// dropped copy. + #[tokio::test(start_paused = true)] + async fn a_returning_reader_is_not_stranded_on_a_dropped_copy() { + let (producer, _server, upstream, old, mut subscription) = remote_front().await; + let resolved = producer.consume().request_broadcast("room/alice").await.unwrap(); + + // No handler and no track: refuses "video" with NotFound. + let _second = producer.publish("room/alice", Route::default()).unwrap(); + for _ in 0..10 { + tokio::task::yield_now().await; + } + let mut group = old.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"draining".as_ref()).unwrap(); + group.finish().unwrap(); + let mut group = next_group(&mut subscription).await.unwrap().expect("draining group"); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"draining"); + drop(group); + + drop(subscription); + tokio::time::timeout(Duration::from_secs(1), old.demand().unused()) + .await + .expect("parked") + .expect("open"); + + let _third = producer.publish("room/alice", Route::default()).unwrap(); + for _ in 0..10 { + tokio::task::yield_now().await; + } + + let mut again = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + let mut group = old.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"after".as_ref()).unwrap(); + group.finish().unwrap(); + let outcome = tokio::time::timeout(Duration::from_secs(5), async { + loop { + match again.recv_group().await { + Ok(Some(mut group)) => { + if &group.read_frame().await.unwrap().unwrap().payload[..] == b"after" { + return Ok(()); + } + } + Ok(None) => return Ok(()), + Err(err) => return Err(err), + } + } + }) + .await; + assert!(outcome.is_ok(), "the returning reader waits on a copy the park dropped"); + drop(upstream); + } + + /// A newer local source's copy of "video" is refused for its metadata while the + /// incumbent keeps serving: the refused copy is let go, not kept subscribed. + #[tokio::test(start_paused = true)] + async fn a_refused_copy_is_not_kept_subscribed() { + let producer = origin(1).produce(); + let consumer = producer.consume(); + let first = producer.publish("room/alice", Route::default()).unwrap(); + let track = first.create_track("video", None).unwrap(); + let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); + group.finish().unwrap(); + let mut group = subscription.recv_group().await.unwrap().unwrap(); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"before"); + + let second = producer.publish("room/alice", Route::default()).unwrap(); + let incompatible = second + .create_track("video", track::Info::default().with_timescale(crate::Timescale::MICRO)) + .unwrap(); + for _ in 0..10 { + tokio::task::yield_now().await; + } + assert!(!incompatible.demand().is_used(), "the refused copy is still subscribed"); + + // The refusal happened: the incumbent leaving surfaces it. + drop(track); + first.close(); + assert!(matches!(subscription.recv_group().await, Err(Error::Unsupported))); + } + + /// As above, the newer source's copy was refused and the incumbent keeps serving. + /// Then both withdraw and the front ends: the track in flight carries on with the + /// incumbent, never the refused copy. + #[tokio::test] + async fn the_end_never_feeds_a_refused_copy() { + let producer = origin(1).produce(); + let consumer = producer.consume(); + let first = producer.publish("room/alice", Route::default()).unwrap(); + let track = first.create_track("video", None).unwrap(); + let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); + group.finish().unwrap(); + let mut group = subscription.recv_group().await.unwrap().unwrap(); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"before"); + drop(group); + + let second = producer.publish("room/alice", Route::default()).unwrap(); + let incompatible = second + .create_track("video", track::Info::default().with_timescale(crate::Timescale::MICRO)) + .unwrap(); + for _ in 0..10 { + tokio::task::yield_now().await; + } + + first.unannounce(); + second.unannounce(); + settle(|| resolved.is_closed()).await; + + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"still".as_ref()).unwrap(); + group.finish().unwrap(); + for payload in [b"wrong0".as_ref(), b"wrong1".as_ref()] { + let mut group = incompatible.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, payload).unwrap(); + group.finish().unwrap(); + } + + let mut group = next_group(&mut subscription) + .await + .expect("the in-flight track carries on") + .expect("track ended early"); + let frame = group.read_frame().await.unwrap().unwrap(); + assert_eq!( + &frame.payload[..], + b"still", + "the front's end spliced the copy it refused" + ); + } + + /// The track drains the incumbent while a newer source's answer is pending, then both + /// withdraw and the front ends: the track carries on with the incumbent's copy rather + /// than the unanswered one. + #[tokio::test] + async fn the_end_keeps_the_copy_still_serving() { + let producer = origin(1).produce(); + let consumer = producer.consume(); + let first = producer.publish("room/alice", Route::default()).unwrap(); + let track = first.create_track("video", None).unwrap(); + let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); + let mut subscription = resolved.track("video").unwrap().subscribe(None).await.unwrap(); + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); + group.finish().unwrap(); + let mut group = subscription.recv_group().await.unwrap().unwrap(); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"before"); + drop(group); + + let second = producer.create_broadcast("room/alice").unwrap(); + let mut handler = second.dynamic(); + second.announce(Route::default()).unwrap(); + let request = tokio::time::timeout(Duration::from_secs(1), handler.requested_track()) + .await + .expect("the front asked the newcomer") + .expect("request"); + + first.unannounce(); + second.unannounce(); + settle(|| resolved.is_closed()).await; + + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"still".as_ref()).unwrap(); + group.finish().unwrap(); + request.reject(Error::NotFound); + + let mut group = next_group(&mut subscription) + .await + .expect("the in-flight track carries on") + .expect("track ended early"); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"still"); + } + #[tokio::test] async fn multiple_scopes_present_one_broad_prefix() { let producer = origin(1).produce(); diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index e7a5a2ce23a..f1f2f4eaa54 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -1,6125 +1,1778 @@ -//! Splice multiple per-session tracks into one logical track, switching at frame -//! boundaries, so a subscription survives route and connection changes. +//! Serve a front's logical track straight from its routes' copies. //! -//! A [`Producer`] holds an ordered list of segments, each a [`track::Consumer`] -//! bounded to a half-open range of [`Position`]s. [`Producer::switch`] appends a -//! segment starting at position `P` and caps the previous one there, so the segments -//! always partition the track. A [`Subscriber`] reads across the segments as if they -//! were one track: bounds are enforced on the read side (a route delivering outside -//! its range is silently filtered), a segment dying stalls the subscriber instead of -//! erroring (the next [`Producer::switch`] resumes it), and demand is forwarded to -//! each underlying track capped at its segment end, so a session serving a segment -//! just sees an ordinary subscription that happens to end at a boundary. A -//! group-aligned boundary never becomes a *start*: it says which groups this segment -//! owns, not which ones anyone wants, so a subscriber asking for the live edge still -//! asks for it after a switch. +//! A front serves each track through one route at a time and switches when that route +//! dies, withdraws, or is beaten. A path names one broadcast whoever serves it, so every +//! route's copy of a track holds the same groups and frames. A reader of the logical +//! track therefore reads the serving route's copy directly, through its own +//! [`Subscriber`]: nothing is copied, and with one route it is a passthrough. //! -//! Boundaries are frame-precise, so a takeover does not have to wait for the next -//! group. When one lands inside a group the subscriber has already handed out, the -//! group itself is spliced: [`Group`] reads each route's copy in turn and the reader -//! sees one continuous frame stream. This is what lets a track whose current group -//! stays open indefinitely (a JSON append log, a catalog with deltas) survive a route -//! change at all. That is the one boundary a live-edge subscriber does demand, since -//! only demand carries the frame offset the continuation has to start from. - -use std::collections::{BTreeMap, HashSet}; -use std::ops::Bound; +//! When the front switches, each reader subscribes to the new copy from the newest group +//! it handed out, and keeps the replaced copy only for what it already handed out: the +//! replaced subscription is bounded after that group, and dropped once no group read from +//! it is left. A group carries on across the switch through its [`Recover`]: once its copy +//! fails with its route, or stalls while a newer route serves, it continues from the +//! serving route's copy of the same group at the same frame (and byte), found in that +//! copy's cache or fetched from it. + +use std::collections::BTreeSet; +use std::sync::{Arc, Mutex, MutexGuard, Weak}; use std::task::{Poll, ready}; -use crate::{Datagram, Error, Result, Timestamp, frame, group, track}; -use track::{Anchor, LiveEdge}; +use crate::{Datagram, Error, Result, StreamError, group, stats, track}; -use super::subscription::{Cap, Position, Subscription, max_some, min_some}; +use super::subscription::{Cap, Position, Subscription, max_some}; -/// One spliced source: a track bounded to a half-open range of positions. -#[derive(Clone)] -struct Segment { - /// Monotonic id, used by subscribers to reconcile their cursor set. - id: u64, - /// First position this segment serves, or `None` for no lower bound (the - /// initial segment, which may start at the live edge). - start: Option, - /// First position this segment no longer serves, or `None` while it is the - /// newest segment. Exclusive, so it is exactly the next segment's `start`. - end: Option, - /// The underlying per-session track. - track: track::Consumer, - /// Where the source is asked to start: `start`, except after a warm cache (see - /// [`Segment::warm_edge`]). - ask: Option, - /// A parked track's warm cache (see [`Producer::park`]), which live readers hold - /// until the next segment's copy resolves its start. - warm: bool, +/// How many delivered group sequences a reader remembers, so a group two routes both +/// deliver is handed out once. +const MAX_DELIVERED: usize = 1024; + +/// What the front says about a logical track: the copy to read now, and how it ended. +#[derive(Default)] +struct Route { + /// Bumped whenever `copy` changes, so a reader can tell a replacement apart. + generation: u64, + /// The serving route's copy; `None` while nobody reads the track, or between routes. + copy: Option, + /// How the track ended, once the front decided. + end: Option>, } -impl Segment { - /// Whether this segment serves `position`. - fn covers(&self, position: Position) -> bool { - self.start.is_none_or(|start| position >= start) && self.end.is_none_or(|end| position < end) - } +/// Every reader of a logical track, so a new copy subscribes them all as it is served: +/// one that is not reading right now still gets what that copy delivers. +type Readers = Arc>>>>; - /// One past the last position this segment produced within its own range, or - /// `None` if nothing in range exists yet (out-of-range groups, e.g. a fetch into - /// the track below the segment's start, don't count). - /// - /// Clamped to `end`: once the track's edge races past the cap the range - /// reads as settled through it, even if a position inside it never arrived. A - /// late arrival is still served (bounds filter reads, not access); the next - /// takeover splices above the cap either way. - fn produced(&self) -> Option { - let produced = self.track.resume_position()?; - // Nothing in range yet. An unbounded segment starts at the very first frame, so - // a track holding only an empty group still counts as having produced nothing. - if produced <= self.start.unwrap_or_default() { - return None; - } - Some(match self.end { - Some(end) => produced.min(end), - None => produced, - }) - } +/// The front's side of a logical track: which copy serves it, and how it ends. +/// +/// Dropping it concludes the track: readers follow the last copy to its end, since no +/// front is left to replace it. +pub(crate) struct Producer { + state: kio::Producer, + readers: Readers, +} - /// Where a copy spliced after this warm cache is asked to start: the end of the - /// cache's newest group, even a finished one, rather than the head of the next. - /// - /// A source only resolves a start once it has a group to serve, so asking past its - /// newest group would leave a returning reader waiting on the next one. Asking for the - /// newest group's tail lets a source that is still there answer at once, and one that - /// moved on answer past it. Only the ask moves: the boundary stays where the cache - /// stops, so whatever the source sends for a finished group's empty tail (a FIN, or a - /// reset from a publisher that refuses empty ranges) sits outside the new segment. - fn warm_edge(&self) -> Option { - let group = self.track.peek_latest()?; - Some(Position { - group: group.sequence, - frame: group.frame_count() as u64, - }) +impl Producer { + pub(crate) fn new() -> Self { + Self { + state: kio::Producer::default(), + readers: Readers::default(), + } } - /// The newest cached group this segment serves below `before` (exclusive), or its - /// newest cached group at all when `before` is `None`. - /// - /// Bounded at both ends, since a segment's track can hold groups outside the range - /// it serves: a fetch can backfill below `start`, and the track's own edge can race - /// past `end` before the next takeover splices above it. Neither belongs to this - /// segment, so the search is capped at `end` and anything below `start` reads as a - /// miss (there is nothing in range, or the caller would have found it). - fn peek_below(&self, before: Option) -> Option { - // The segment end is exclusive, matching the exclusive group limit `peek_before` - // expects. - let limit = min_some(before, last_group(self.end)); - let group = match limit { - Some(limit) => self.track.peek_before(limit)?, - None => self.track.peek_latest()?, + /// Serve the track from `copy` from now on, subscribing every reader to it. + pub(crate) fn serve(&self, copy: track::Consumer) { + if let Ok(mut route) = self.state.write() { + route.generation += 1; + route.copy = Some(copy); + } + let readers = { + let mut readers = self.readers.lock().expect("readers poisoned"); + readers.retain(|reader| reader.strong_count() > 0); + readers.clone() }; - if self.start.is_some_and(|start| group.sequence < start.group) { - return None; + for reader in readers.iter().filter_map(Weak::upgrade) { + reader.lock().expect("reader poisoned").sync(&kio::Waiter::noop()); } - Some(group) - } -} - -/// The demand to register on an underlying track: the subscriber's own -/// preferences intersected with a segment's bounds. -fn slice(prefs: &Subscription, start: Option, end: Option) -> Subscription { - // A segment's bounds and a subscription's are both half-open, so they intersect - // directly with no inclusive/exclusive conversion in between. Both sides use the - // `_some` family: this is an intersection of two independent ranges, not an - // aggregate across subscribers, so an absent bound on either side is neutral rather - // than absorbing. - // - // For the start that means a takeover boundary becomes demand even for a live-edge - // subscriber, so the replacement resumes exactly where the dead route stopped and the - // splice loses nothing. The catch-up that buys is bounded by how far the boundary - // trails the replacement's live edge, which is failover max_age: a broadcast closes - // with its last source (there is no linger), so a takeover only happens between - // overlapping routes. A consumer that would rather skip even that much drops it - // downstream on its own max age budget, which is what `Duration::ZERO` means. - Subscription { - start: max_some(prefs.start, start), - end: min_some(prefs.end, end), - ..prefs.clone() } -} - -/// Where the first servable group in `from..cap` across `segments` starts presenting; -/// see [`track::Consumer::poll_first_start`]. An unstamped group stops the search: -/// skipping it for a later start would shrink a reach that is not yet proven. -fn poll_first_start<'a>( - segments: impl IntoIterator, - waiter: &kio::Waiter, - from: u64, - cap: Option, -) -> Option> { - segments.into_iter().find_map(|segment| { - let start = segment.start.map_or(0, |start| start.group).max(from); - segment - .track - .poll_first_start(waiter, start, min_some(cap, last_group(segment.end))) - }) -} -/// Where a splice continues past the exclusive group `boundary` of segment `after`, -/// below the reader's `cap`: the first group the later segments serve there. Holds the -/// question rather than a cached answer, so a group aborted or evicted before its first -/// frame hands the bound to whichever group serves there next. -#[derive(Clone)] -pub(crate) struct Successor { - state: kio::ConsumerWeak, - after: u64, - boundary: u64, - cap: Option, -} - -impl Successor { - /// Where the successor starts presenting: `None` while no later group is cached or - /// the first one is unstamped. Registers `waiter` for anything that could move it. - /// Takes the splice's and its segments' locks, so never call it under a track's. - pub(crate) fn poll_start(&self, waiter: &kio::Waiter) -> Option { - let mut start = None; - let _ = self.state.poll(waiter, |state| { - // A draining reader's own segment may already have been pruned. - let later = state.segments.iter().filter(|segment| segment.id > self.after); - start = poll_first_start(later, waiter, self.boundary, self.cap).flatten(); - Poll::<()>::Pending - }); - start + /// Let go of the serving copy: nobody reads the track, so its route goes idle. + pub(crate) fn park(&self) { + if let Ok(mut route) = self.state.write() + && route.copy.is_some() + { + route.generation += 1; + route.copy = None; + } } -} -impl PartialEq for Successor { - fn eq(&self, other: &Self) -> bool { - self.after == other.after - && self.boundary == other.boundary - && self.cap == other.cap - && self.state.same_channel(&other.state) + /// The track ended: cleanly with `Ok`, or failed for good. + pub(crate) fn end(&self, result: Result<()>) { + if let Ok(mut route) = self.state.write() + && route.end.is_none() + { + route.end = Some(result); + } } -} - -/// How many segments a logical track keeps before pruning terminal ones from the -/// front: the live segment plus a couple of predecessors still draining to slow -/// readers. Without a bound, every failover leaves one dead segment (pinning a -/// dead session's [`track::Consumer`] and cache) behind for the life of the track. -const MAX_SEGMENTS: usize = 3; -struct ResumeState { - /// Segments in switch order; ranges are disjoint and ascending. - segments: Vec, - /// Everything below this position was covered by segments that have since been - /// pruned: no future segment can serve it (boundaries only move forward), so - /// readers below it give up rather than parking for a replacement. `None` until - /// the first prune; reset by [`Producer::release`] along with the segments. - pruned: Option, - /// Bumped on every mutation so subscribers know to reconcile. - epoch: u64, - /// No more switches will happen; the logical track ends with its last segment. - finished: bool, - /// The logical track was aborted; surfaced to every subscriber. - abort: Option, -} - -impl Default for ResumeState { - fn default() -> Self { - Self { - segments: Vec::new(), - pruned: None, - epoch: 1, - finished: false, - abort: None, + pub(crate) fn consume(&self) -> Consumer { + Consumer { + state: self.state.consume(), + readers: self.readers.clone(), } } } -/// A point-in-time copy of the producer state, reconciled into a -/// [`Subscriber`]'s cursor set by [`Subscriber::apply`]. -struct Snapshot { - epoch: u64, - finished: bool, - abort: Option, - segments: Vec, +/// A reader's handle on a logical track's [`Producer`]. +#[derive(Clone)] +pub(crate) struct Consumer { + state: kio::Consumer, + readers: Readers, } -impl ResumeState { - fn snapshot(&self) -> Snapshot { - Snapshot { - epoch: self.epoch, - finished: self.finished, - abort: self.abort.clone(), - segments: self.segments.clone(), +impl Consumer { + /// The serving route's copy, if any. + pub(crate) fn serving(&self) -> Option { + self.state.read().copy.clone() + } + + /// Subscribe to the logical track with the reader's own preferences. + pub(crate) fn subscribe(&self, subscription: kio::Producer) -> Subscriber { + let mirrored = subscription.read().clone(); + let reader = Arc::new_cyclic(|this| { + Mutex::new(Reader { + this: this.clone(), + route: self.clone(), + subscription: subscription.clone(), + mirrored, + generation: None, + copies: Vec::new(), + delivered: BTreeSet::new(), + newest: None, + ordered: None, + datagram: None, + groups: (0, Cap::from(..)), + }) + }); + // Subscribed to the serving copy at once, so groups it has now are the reader's + // even if the route changes before the first read, and to each copy after. + { + let mut readers = self.readers.lock().expect("readers poisoned"); + // Sweep the readers that left once the list fills, so it tracks the live count. + if readers.len() == readers.capacity() { + readers.retain(|reader| reader.strong_count() > 0); + let live = readers.len(); + readers.reserve(live.max(1)); + } + readers.push(Arc::downgrade(&reader)); } + reader.lock().expect("reader poisoned").sync(&kio::Waiter::noop()); + Subscriber { reader, subscription } } - /// One past the newest position across the segments, clamped to their bounds: where - /// a replacement segment should pick the track up. - /// - /// The pruned floor participates: a pruned segment produced exactly through its - /// cap, so dropping it must not let the boundary collapse below what it served - /// (a takeover would re-splice under the delivered edge). - fn resume_position(&self) -> Option { - self.segments - .iter() - .filter_map(Segment::produced) - .chain(self.pruned) - .max() - } - - /// The latest group sequence across the segments, clamped to their bounds. - fn latest(&self) -> Option { - let position = self.resume_position()?; - match position.frame { - // The resume point sits at the head of a group, so the newest group actually - // produced is the one below it. - 0 => position.group.checked_sub(1), - _ => Some(position.group), + /// Fetch one group from whichever route serves the track. + pub(crate) fn fetch_group(&self, sequence: u64, options: group::Fetch) -> Fetching { + Fetching { + route: self.clone(), + sequence, + options, + pending: None, } } - /// The newest live edge across the segments below the exclusive `cap`, each clamped - /// to its own range: the edge the logical track measures drift against. - fn live_edge(&self, cap: Option) -> Option { - self.segments - .iter() - .filter_map(|segment| { - let edge = segment.track.live_edge(min_some(cap, last_group(segment.end)))?; - // Out-of-range content below the segment is not its to serve. - let start = segment.start.map_or(0, |start| start.group); - (edge.sequence >= start).then_some(edge) - }) - .max_by_key(|edge| edge.sequence) - } - - /// Append a segment serving the track from `start` onward, capping (or replacing) - /// the previous segments so the ranges stay disjoint and ascending. - fn switch(&mut self, track: track::Consumer, start: Option) -> Result<()> { - if !self.segments.is_empty() { - // A boundary is required once a segment exists. - let Some(start) = start else { - return Err(crate::coding::BoundsExceeded.into()); - }; - - // Segments the new range fully covers are replaced outright, provided - // they never produced anything in range (nothing to splice around). - while let Some(prev) = self.segments.last() { - let prev_start = prev.start.unwrap_or_default(); - if start > prev_start { - break; - } - if prev.produced().is_some() { - return Err(crate::coding::BoundsExceeded.into()); - } - self.segments.pop(); - } - - // Cap whatever remains at the boundary. The bound is exclusive, so the new - // segment's start is exactly the previous segment's end. The cap may land - // below the produced edge (a manual boundary re-serving delivered - // positions from the replacement); readers reconcile on every frame, so a - // moved cap re-routes them, and the subscriber's floor dedups re-delivery. - if let Some(prev) = self.segments.last_mut() { - prev.end = Some(start); + /// Poll for the route state to move past `generation`, or the track to end. + fn poll_changed(&self, generation: u64, waiter: &kio::Waiter) -> Poll<()> { + match self.state.poll(waiter, |route| { + match route.generation != generation || route.end.is_some() { + true => Poll::Ready(()), + false => Poll::Pending, } + }) { + Poll::Pending => Poll::Pending, + Poll::Ready(_) => Poll::Ready(()), } - - let id = self.epoch; - self.segments.push(Segment { - id, - start, - end: None, - track, - ask: start, - warm: false, - }); - self.epoch += 1; - self.prune(); - Ok(()) } - /// Drop retired segments from the front once the list outgrows - /// [`MAX_SEGMENTS`], recording the range they covered in [`Self::pruned`]. - /// - /// A front segment is retired when it owes nothing more: it produced through - /// its cap (a takeover boundary is the resume position, so this is every - /// takeover-capped segment, alive or not) or its track is terminal. What it - /// holds is a cache for slow readers, and a reader mid-drain keeps its own - /// positioned cursor (see [`Group::poll_current`]'s missing-segment - /// fallback). Only a manually spliced boundary can sit above the produced - /// edge; that segment is still expected to backfill, so it blocks the sweep - /// (and the segments behind it) until it does or dies. - fn prune(&mut self) { - while self.segments.len() > MAX_SEGMENTS { - let front = &self.segments[0]; - let Some(end) = front.end else { break }; - let owes_more = - front.produced() < Some(end) && front.track.poll_complete(&kio::Waiter::noop()).is_pending(); - if owes_more { - break; - } - self.pruned = self.pruned.max(Some(end)); - self.segments.remove(0); + /// How the logical track ended: `None` while the front may still replace its copy, + /// `Some(None)` once it concluded (readers follow the last copy), else the front's + /// decision. + fn end(&self) -> Option>> { + let route = self.state.read(); + match (&route.end, route.is_closed()) { + (Some(end), _) => Some(Some(end.clone())), + (None, true) => Some(None), + (None, false) => None, } } } -/// Splices tracks into one logical track by switching at group boundaries. -/// -/// Created with [`Self::new`]; hand out read access via [`Self::consume`]. Call -/// [`Self::switch`] (or [`Self::takeover`]) whenever the serving route changes; -/// subscribers migrate transparently. The producer only manages boundaries: the -/// actual groups are written by whoever owns each underlying [`track::Producer`]. -#[derive(Clone, Default)] -pub struct Producer { - state: kio::Producer, +/// Whether `err` is the route failing rather than refusing: another route may still +/// deliver what it failed. +fn route_failed(err: &Error) -> bool { + matches!( + err, + Error::Transport(_) + | Error::Session(_) + | Error::SessionClosed + | Error::GoingAway + | Error::GoawayTimeout + | Error::Dropped + | Error::Closed + | Error::Stream(StreamError::Session(_) | StreamError::GoingAway) + ) } -impl Producer { - /// Create a logical track with no segments; subscribers stall until the first - /// [`Self::switch`]. - pub fn new() -> Self { - Self::default() - } +/// Whether `copy` failed rather than ended: its route went away, so an error read from it +/// is no verdict on what it carried. +fn copy_failed(copy: &track::Consumer) -> bool { + matches!(copy.poll_complete(&kio::Waiter::noop()), Poll::Ready(Err(_))) +} - /// Splice in a track serving the track from `start` onward, capping the previous - /// segment there. - /// - /// The first switch may pass `None` to leave the segment unbounded (it serves - /// whatever the subscriber asks for, typically the live edge). Every later - /// switch must pass `Some(start)`. A previous segment whose range the new one - /// fully covers is replaced outright, provided it never produced anything in - /// range (there is nothing to splice around); otherwise the boundary must - /// advance past it, or this fails with [`Error::BoundsExceeded`] and the - /// segment list is unchanged. - /// - /// Bounds are enforced when reading: a previous segment's session may keep - /// delivering past its new cap (the switch races the network) and those groups - /// and frames are simply never surfaced. - // Production callers go through `takeover`; this is the entry point an explicit - // wire-driven boundary (a future manual-splice surface) would use, and the - // boundary tests drive it directly. - #[cfg_attr(not(test), expect(dead_code))] - pub fn switch( - &mut self, - track: impl super::origin_impl::Consume, - start: impl Into>, - ) -> Result<()> { - let track = track.consume(); - let start = start.into(); - let mut state = self.state.write().map_err(|_| Error::Dropped)?; - if state.finished || state.abort.is_some() { - return Err(Error::Closed); - } - state.switch(track, start) - } +/// A route's copy, as one reader reads it. +struct Copy { + generation: u64, + track: track::Consumer, + sub: Sub, + /// Where the subscription was asked to start, so demand updates never ask for less. + floor: Option, + /// Replaced by a newer route: nothing past this group, the newest it had then, is + /// taken from it. + until: Option>, + /// The group channel ran out. + done: Option>, + /// The datagram channel ran out. + datagrams_done: Option>, + /// The newest datagram handed out before this copy was subscribed: a copy that + /// buffers datagrams replays them, and those were already handed out. + datagrams_seen: Option, + /// Held by every group handed out from this copy, so a replaced copy stays + /// subscribed while one is still read. + lease: Arc<()>, +} - /// Splice in a track that resumes wherever the current segments stop: one past - /// the newest spliced frame. - /// - /// This is [`Self::switch`] with the boundary computed from the current state, - /// for callers reacting to a route change rather than choosing a boundary. The - /// boundary is frame-precise, so a group that was mid-transfer when its route - /// died is continued rather than abandoned: the replacement is asked for the - /// frames from the break onward, and subscribers read across the two copies as - /// one group. It rolls to the next group only once the current one is complete, - /// since nothing more can be appended to it. - pub fn takeover(&mut self, track: impl super::origin_impl::Consume) -> Result<()> { - let track = track.consume(); - // Compute the boundary and apply it under one write guard: a boundary - // computed under a separate read lock could race the old route delivering - // more frames, splicing the new segment below the delivered edge. - let mut state = self.state.write().map_err(|_| Error::Dropped)?; - if state.finished || state.abort.is_some() { - return Err(Error::Closed); - } - // One past the newest delivered position. With nothing delivered (or no - // segments at all) there is nothing to splice around, so the replacement - // replaces them outright and starts unbounded, exactly like a first splice. - // `switch` rejects a `None` start once a segment exists, hence the clear. - let ask = state - .segments - .last() - .filter(|last| last.warm) - .and_then(Segment::warm_edge); - let start = state.resume_position(); - if start.is_none() { - state.segments.clear(); - } - state.switch(track, start)?; - if let Some(ask) = ask - && start.is_some() - && let Some(last) = state.segments.last_mut() - { - last.ask = Some(ask); +enum Sub { + Pending(kio::Pending), + Ready(track::Subscriber), +} + +impl Copy { + /// What this copy is asked for: the reader's own preferences, from no earlier than + /// where it was asked to start, and through no later than where it was replaced. + fn wire(&self, reader: &Subscription) -> Subscription { + let mut wire = Subscription { + start: max_some(reader.start, self.floor), + ..reader.clone() + }; + if let Some(until) = self.until { + let end = until.and_then(Position::after_group).unwrap_or_default(); + wire.end = Some(wire.end.map_or(end, |requested| requested.min(end))); } - Ok(()) + wire } - /// Drop every segment, releasing the underlying tracks while keeping the - /// logical track alive for a later [`Self::takeover`]. - /// - /// For a track nobody is reading: releasing the last consumer of a segment lets - /// the serving session tear its copy down, so an idle track stops costing an - /// upstream subscription and a cached [`track::Info`]. The next takeover starts - /// unbounded again, since with no segments there is no boundary to splice - /// around. - pub(crate) fn release(&mut self) -> Result<()> { - let mut state = self.state.write().map_err(|_| Error::Dropped)?; - if state.finished || state.abort.is_some() { - return Err(Error::Closed); - } - if state.segments.is_empty() { - return Ok(()); - } - state.segments.clear(); - // The next takeover starts unbounded and may restart the numbering, so a - // floor from the old numbering must not cut into it. - state.pruned = None; - state.epoch += 1; - Ok(()) + fn update(&mut self, reader: &Subscription) { + let wire = self.wire(reader); + let _ = match &mut self.sub { + Sub::Pending(pending) => pending.update(wire), + Sub::Ready(sub) => sub.update(wire), + }; } - /// Replace every segment with `warm`, a cache of what they delivered, for a track - /// nobody reads anymore: [`Self::release`] followed by an unbounded first splice. - /// - /// Live readers hold the cache until the next [`Self::takeover`]'s copy resolves - /// where its feed starts. A copy that picks up at the cache's edge proves the cache - /// still leads into the live feed, so it is read as usual (and a group the cache - /// holds open continues from its next frame). A copy that starts past the edge - /// skipped groups its source already judged stale, so the older cache is stale too - /// and live readers skip it. - pub(crate) fn park(&mut self, warm: impl super::origin_impl::Consume) -> Result<()> { - let track = warm.consume(); - let mut state = self.state.write().map_err(|_| Error::Dropped)?; - if state.finished || state.abort.is_some() { - return Err(Error::Closed); + /// The subscription, once the copy answered. `Err` once it refused. + fn poll_ready(&mut self, groups: (u64, Cap), waiter: &kio::Waiter) -> Poll> { + if let Sub::Pending(pending) = &self.sub { + let mut sub = ready!(pending.poll_ok(waiter))?; + sub.raise_start_to(groups.0); + sub.end_at(groups.1); + self.sub = Sub::Ready(sub); } - state.segments.clear(); - state.pruned = None; - state.switch(track, None)?; - if let Some(segment) = state.segments.last_mut() { - segment.warm = true; - } - Ok(()) + let Sub::Ready(sub) = &mut self.sub else { unreachable!() }; + Poll::Ready(Ok(sub)) } - /// Whether any segment is spliced in, and so whether there is anything for - /// [`Self::release`] to drop. - /// - /// A segment outlives the route that produced it: the source can leave, or its - /// copy can die, while the segment stays spliced so readers keep what it already - /// delivered. That makes this, not the caller's own handle on the serving route, - /// the condition for arming an idle release. - pub(crate) fn is_spliced(&self) -> bool { - !self.state.read().segments.is_empty() + /// Whether this copy is still needed: it serves, or a group read from it is still out. + fn needed(&self, serving: bool) -> bool { + serving || (Arc::strong_count(&self.lease) > 1 && self.done.is_none()) } +} + +/// One reader's subscription to a logical track; see the module docs. +pub(crate) struct Subscriber { + /// Shared with the groups it hands out, so one stuck on a dead route can follow the + /// reader onto the next route without the reader polling for groups. + reader: Arc>, + subscription: kio::Producer, +} - /// One past the newest delivered position across the segments. The origin's - /// serve task reads this as its delivered-progress signal. - pub(crate) fn resume_position(&self) -> Option { - self.state.read().resume_position() +impl Subscriber { + fn reader(&self) -> MutexGuard<'_, Reader> { + self.reader.lock().expect("reader poisoned") } - /// Mark the logical track as complete: no further switches. Subscribers see a - /// clean end once the final segment's track finishes. - pub fn finish(&mut self) -> Result<()> { - let mut state = self.state.write().map_err(|_| Error::Dropped)?; - if state.finished || state.abort.is_some() { - return Err(Error::Closed); - } - state.finished = true; - state.epoch += 1; - Ok(()) + /// The reader's preferences channel, shared with its [`track::Control`]. + pub(crate) fn subscription(&self) -> &kio::Producer { + &self.subscription } - /// Abort the logical track, releasing every subscriber with `err`. - /// - /// Fails once the track [`finish`](Self::finish)ed: a clean end is terminal, - /// so a late abort (e.g. route churn re-queueing an already-completed track) - /// cannot turn it into an error for subscribers still draining. - pub fn abort(&mut self, err: Error) -> Result<()> { - let mut state = self.state.write().map_err(|_| Error::Dropped)?; - if state.finished || state.abort.is_some() { - return Err(Error::Closed); - } - state.abort = Some(err); - state.epoch += 1; - Ok(()) + /// Poll for the next group: in arrival order, or in sequence order for `ordered`. + pub(crate) fn poll_group(&mut self, ordered: bool, waiter: &kio::Waiter) -> Poll>> { + self.reader().poll_group(ordered, waiter) } - /// Whether the logical track ended in an error, as opposed to still being - /// servable or cleanly [`finish`](Self::finish)ed. - pub(crate) fn is_aborted(&self) -> bool { - self.state.read().abort.is_some() + /// Poll for the next datagram in arrival order. + pub(crate) fn poll_recv_datagram(&mut self, waiter: &kio::Waiter) -> Poll>> { + self.reader().poll_recv_datagram(waiter) } - /// Whether `other` produces the same logical track. - pub(crate) fn is_clone(&self, other: &Self) -> bool { - self.state.same_channel(&other.state) + /// Poll for the track's declared final sequence. + pub(crate) fn poll_finished(&mut self, waiter: &kio::Waiter) -> Poll> { + self.reader().poll_finished(waiter) } - /// Create a read handle for the logical track. - pub fn consume(&self) -> Consumer { - Consumer { - state: self.state.consume(), + /// Poll for where the serving copy's live feed starts; see + /// [`track::Subscriber::poll_start`]. + pub(crate) fn poll_start(&mut self, waiter: &kio::Waiter) -> Poll> { + let mut reader = self.reader(); + let sub = ready!(reader.poll_serving(waiter)); + match sub { + Some(sub) => sub.poll_start(waiter), + None => Poll::Ready(None), } } - /// Whether any read handle for the logical track currently exists. - /// - /// This is the demand signal: a spliced track with no consumers is cached - /// state nobody is watching. - pub fn is_used(&self) -> bool { - self.state.is_used() + /// Raise the local read floor; see [`track::Subscriber::set_groups`]. + pub(crate) fn raise_start_to(&mut self, start: u64) { + self.reader().raise_start_to(start) } - /// Poll for a consumer appearing, parking `waiter` until one does. Ready - /// immediately once one exists (or the track closed). Feeds - /// [`crate::broadcast::Demand`], which recomputes on wake. - pub(crate) fn poll_used(&self, waiter: &kio::Waiter) -> Poll<()> { - self.state.poll_used(waiter).map(|_| ()) + /// Cap local reads at `end`; see [`track::Subscriber::set_groups`]. + pub(crate) fn end_at(&mut self, end: Cap) { + self.reader().end_at(end) } - /// Poll for the last consumer going away, parking `waiter` until it does. - /// Ready immediately once none remain (or the track closed). - pub(crate) fn poll_unused(&self, waiter: &kio::Waiter) -> Poll<()> { - self.state.poll_unused(waiter).map(|_| ()) + /// The newest sequence any copy has. + pub(crate) fn latest(&self) -> Option { + self.reader().latest() } -} -/// A cheap, cloneable read handle for a spliced logical track. -#[derive(Clone)] -pub struct Consumer { - state: kio::Consumer, -} - -impl Consumer { - /// Complete groups retained across segments, clipped to the ranges that belong - /// to each segment. A frame-aligned switch splits one group across two tracks; - /// that seam group is skipped while complete groups on either side are kept. - pub(crate) fn cached_groups(&self) -> Vec<(group::Producer, bool)> { - let segments = self.state.read().segments.clone(); - let mut seen = HashSet::new(); - let mut groups = Vec::new(); - for segment in segments { - let start = segment - .start - .map(|start| start.group.saturating_add(u64::from(start.frame != 0))); - let end = segment.end.map(|end| end.group); - for (group, visible) in segment.track.cached_groups() { - if start.is_some_and(|start| group.sequence < start) - || end.is_some_and(|end| group.sequence >= end) - || !seen.insert(group.sequence) - { - continue; - } - groups.push((group, visible)); + /// The groups the copies' drift budget skipped since the last call, counted against + /// the logical track: the copies themselves are untagged. + pub(crate) fn take_stale(&mut self) -> stats::Content { + let mut stale = stats::Content::default(); + for copy in &mut self.reader().copies { + if let Sub::Ready(sub) = &mut copy.sub { + stale.add(sub.take_stale()); } } - groups - } - - /// Publisher properties resolved by the first segment. Every takeover is - /// validated against the front's accepted track info before it is spliced in. - pub(crate) fn cached_info(&self) -> Option { - self.state.read().segments.first()?.track.cached_info() + stale } - /// A complete cached copy of `sequence`, newest segment first. - /// - /// A segment can answer a FETCH only when it owns the requested head and the - /// rest of the group. A frame boundary inside the group therefore falls - /// through to another segment or to the live route's upstream FETCH. - pub(crate) fn cached_group(&self, sequence: u64, frame_start: u64) -> Option { - let segments = self.state.read().segments.clone(); - segments.iter().rev().find_map(|segment| { - let (start, end) = frames(segment.start, segment.end, sequence)?; - (start <= frame_start && end.is_none()) - .then(|| segment.track.cached_group(sequence, frame_start)) - .flatten() - }) + /// Poll for the serving copy's cache reflecting its live feed; see + /// [`track::Subscriber::poll_live`]. + pub(crate) fn poll_live(&mut self, waiter: &kio::Waiter) -> Poll> { + let mut reader = self.reader(); + let sub = ready!(reader.poll_serving(waiter)); + match sub { + Some(sub) => sub.poll_live(waiter), + None => Poll::Ready(None), + } } +} - /// Open a live subscription across every segment. - /// - /// The subscription's preferences are forwarded to each underlying track - /// intersected with its segment bounds, so each serving session sees plain - /// demand for its own range. Demand registers as the subscriber is polled. - /// Pass `None` for [`Subscription::default`]. - #[cfg(test)] - pub fn subscribe(&self, subscription: impl Into>) -> Subscriber { - let prefs = kio::Producer::new(subscription.into().unwrap_or_default()); - self.subscribe_shared(prefs) - } +/// A [`Subscriber`]'s state. +struct Reader { + this: Weak>, + route: Consumer, + /// The reader's own preferences, mirrored into every copy's subscription. + subscription: kio::Producer, + mirrored: Subscription, + /// The route generation last read. + generation: Option, + /// Oldest first: the last one is the serving route's, when it is subscribed. + copies: Vec, + /// The groups handed out, so a group two copies deliver is handed out once. + delivered: BTreeSet, + /// The newest group handed out: where a new copy is asked to start. + newest: Option, + /// Reading in sequence order: the last group handed out. + ordered: Option, + /// The newest datagram handed out. + datagram: Option, + /// The local group filter; see [`track::Subscriber::set_groups`]. + groups: (u64, Cap), +} - /// Subscribe with an externally-owned preferences channel, so a - /// [`track::Control`]-style handle can update it. - pub(crate) fn subscribe_shared(&self, prefs: kio::Producer) -> Subscriber { - let last_prefs = prefs.read().clone(); - Subscriber { - state: self.state.clone(), - prefs, - last_prefs, - epoch: 0, - finished: false, - abort: None, - closed: false, - segments: Vec::new(), - next_sequence: 0, - min_sequence: 0, - end_sequence: None, - outer: Anchor::default(), - drift_anchor: kio::Producer::new(Anchor::default()), +impl Reader { + /// Follow the route: subscribe to a new copy, bound the ones it replaced, and mirror + /// the reader's preferences into each. + fn sync(&mut self, waiter: &kio::Waiter) { + // Snapshot and re-arm together, including when a change was observed: another + // update must wake a reader that parks after applying this one. + let mut changed = None; + let _ = self.subscription.poll_ref(waiter, |sub| { + if **sub != self.mirrored { + changed = Some((**sub).clone()); + } + Poll::<()>::Pending + }); + if let Some(sub) = changed { + self.mirrored = sub; + for copy in &mut self.copies { + copy.update(&self.mirrored); + } } - } - /// Poll for the track's [`track::Info`], resolved from the first segment. - /// - /// Stays pending until a segment exists and its track's info is known (the - /// serving session may not have accepted it yet). - pub fn poll_info(&self, waiter: &kio::Waiter) -> Poll> { - // Wait for the first segment (or a terminal state), then poll its info. - let track = match ready!(self.state.poll(waiter, |state| { - if state.abort.is_some() || !state.segments.is_empty() { - Poll::Ready( - state - .abort - .clone() - .map_or_else(|| Ok(state.segments[0].track.clone()), Err), - ) - } else { - Poll::Pending + let (generation, serving) = { + let route = self.route.state.read(); + (route.generation, route.copy.clone()) + }; + if self.generation == Some(generation) { + return; + } + self.generation = Some(generation); + let Some(track) = serving else { return }; + if self.copies.last().is_some_and(|copy| copy.generation == generation) { + return; + } + // The replaced copies finish the groups they had when replaced, and nothing newer. + for copy in &mut self.copies { + if copy.until.is_none() { + copy.until = Some(self.newest.max(copy.track.latest())); + copy.update(&self.mirrored); } - })) { - Ok(res) => res?, - Err(state) => match (&state.abort, state.segments.first()) { - (Some(err), _) => return Poll::Ready(Err(err.clone())), - (None, Some(segment)) => segment.track.clone(), - // Closed without ever getting a segment: nothing will resolve this. - (None, None) => return Poll::Ready(Err(Error::Dropped)), - }, + } + let floor = self.newest.map(Position::group); + // The wire request carries the floor from the first message: some sessions read + // the demand only once. + let wire = Subscription { + start: max_some(self.mirrored.start, floor), + ..self.mirrored.clone() }; + self.copies.push(Copy { + generation, + sub: Sub::Pending(track.subscribe(wire)), + track, + floor, + until: None, + done: None, + datagrams_done: None, + datagrams_seen: self.datagram, + lease: Arc::new(()), + }); + } - track.query().poll_ok(waiter) + /// Drop the replaced copies nothing is read from any more. + fn let_go(&mut self) { + let serving = self.generation; + self.copies.retain(|copy| copy.needed(Some(copy.generation) == serving)); } - /// Return the track's [`track::Info`], resolved from the first segment. - #[cfg(test)] - pub async fn info(&self) -> Result { - kio::wait(|waiter| self.poll_info(waiter)).await + fn deliverable(&self, copy: &Copy, sequence: u64) -> bool { + if copy + .until + .is_some_and(|until| until.is_none_or(|until| sequence > until)) + { + return false; + } + match self.ordered { + Some(last) => sequence > last, + None => !self.delivered.contains(&sequence), + } } - /// Fetch a single past group without a live subscription. - /// - /// Routed to the most recent segment's track: old segments' sessions are - /// usually gone by the time history is fetched, and a live route can serve - /// groups outside its subscription bounds (bounds slice demand, not access). - /// In-flight fetches on older segments are unaffected. With no segment yet - /// (no route has served the track), the fetch waits for the first one. - pub fn fetch_group(&self, sequence: u64, options: impl Into>) -> kio::Pending { - kio::Pending::new(Fetching { - state: self.state.clone(), + fn hand_out(&mut self, index: usize, group: group::Consumer, ordered: bool) -> group::Consumer { + let sequence = group.sequence; + if ordered { + self.ordered = Some(sequence); + } + self.delivered.insert(sequence); + if self.delivered.len() > MAX_DELIVERED { + self.delivered.pop_first(); + } + self.newest = self.newest.max(Some(sequence)); + let copy = &self.copies[index]; + group.with_recover(Recover { sequence, - options: options.into().unwrap_or_default(), - inner: kio::Lock::new(None), + reader: self.this.clone(), + route: self.route.clone(), + generation: copy.generation, + copy: copy.track.clone(), + lease: Some(copy.lease.clone()), + fetch: None, }) } - /// The latest group sequence across the segments, clamped to their bounds. - pub fn latest(&self) -> Option { - self.state.read().latest() + fn poll_group(&mut self, ordered: bool, waiter: &kio::Waiter) -> Poll>> { + loop { + if let Some(res) = self.poll_group_once(ordered, waiter) { + return res; + } + } } - /// The newest segment's declared exclusive end, where fetches are routed. - pub(crate) fn final_sequence(&self) -> Option { - // Copied out: the segment's track takes its own lock. - let track = self.state.read().segments.last().map(|segment| segment.track.clone())?; - track.final_sequence() + /// One pass over the copies; `None` when the route moved meanwhile and it is worth + /// another. + fn poll_group_once( + &mut self, + ordered: bool, + waiter: &kio::Waiter, + ) -> Option>>> { + self.sync(waiter); + let groups = self.groups; + // The serving copy first: it is the one that keeps delivering. + for index in (0..self.copies.len()).rev() { + let copy = &mut self.copies[index]; + if copy.done.is_some() { + continue; + } + loop { + let copy = &mut self.copies[index]; + let sub = match copy.poll_ready(groups, waiter) { + Poll::Pending => break, + Poll::Ready(Err(err)) => { + copy.done = Some(Err(err)); + break; + } + Poll::Ready(Ok(sub)) => sub, + }; + let res = match ordered { + true => sub.poll_next_group(waiter), + false => sub.poll_recv_group(waiter), + }; + match res { + Poll::Pending => break, + Poll::Ready(Ok(Some(group))) => { + let copy = &self.copies[index]; + if !self.deliverable(copy, group.sequence) { + continue; + } + let group = self.hand_out(index, group, ordered); + return Some(Poll::Ready(Ok(Some(group)))); + } + // Drained: the track ended as the copy did, which is not cleanly when + // its session went away short of the declared end. + Poll::Ready(Ok(None)) => { + copy.done = Some(match copy.track.poll_complete(&kio::Waiter::noop()) { + Poll::Ready(end) => end, + Poll::Pending => Ok(()), + }); + break; + } + Poll::Ready(Err(err)) => { + copy.done = Some(Err(err)); + break; + } + } + } + } + self.let_go(); + self.poll_end(waiter, |copy| copy.done.clone()) } - /// One past the newest position across the segments: where a route taking this - /// logical track over would resume. - pub(crate) fn resume_position(&self) -> Option { - self.state.read().resume_position() + /// Whether the track ended, once the serving copy's channel (as `done` reads it) ran + /// out. `None` when the route moved since it was read: the caller looks again. + fn poll_end( + &self, + waiter: &kio::Waiter, + done: impl Fn(&Copy) -> Option>, + ) -> Option>>> { + let serving = self + .copies + .last() + .filter(|copy| Some(copy.generation) == self.generation); + // The serving copy drained a complete track: that is its end, whichever route + // would serve it next. A copy that failed is the front's to replace. + if let Some(Some(Ok(()))) = serving.map(&done) { + return Some(Poll::Ready(Ok(None))); + } + match self.route.end() { + Some(Some(Err(err))) => return Some(Poll::Ready(Err(err))), + // Ended cleanly: once the serving copy (if any) drained. + Some(Some(Ok(()))) => match serving.map(&done) { + None | Some(Some(_)) => return Some(Poll::Ready(Ok(None))), + Some(None) => {} + }, + // No front is left to replace the copy: the track ends as it does. A copy whose + // session closed locally ends it cleanly. + Some(None) => match serving.map(&done) { + None => return Some(Poll::Ready(Err(Error::Dropped))), + Some(Some(Ok(()) | Err(Error::Closed))) => return Some(Poll::Ready(Ok(None))), + Some(Some(Err(err))) => return Some(Poll::Ready(Err(err))), + Some(None) => {} + }, + None => {} + } + match self.route.poll_changed(self.generation.unwrap_or_default(), waiter) { + Poll::Ready(()) if self.route.end().is_none() => None, + _ => Some(Poll::Pending), + } } - /// The newest live edge across the segments; see [`track::Consumer::live_edge`]. - pub(crate) fn live_edge(&self, cap: Option) -> Option { - self.state.read().live_edge(cap) + fn poll_recv_datagram(&mut self, waiter: &kio::Waiter) -> Poll>> { + loop { + if let Some(res) = self.poll_datagram_once(waiter) { + return res; + } + } } - /// Where the first servable group in `from..cap` starts presenting; see - /// [`track::Consumer::poll_first_start`]. - pub(crate) fn poll_first_start( - &self, - waiter: &kio::Waiter, - from: u64, - cap: Option, - ) -> Option> { - let mut start = None; - let _ = self.state.poll(waiter, |state| { - start = poll_first_start(&state.segments, waiter, from, cap); - Poll::<()>::Pending - }); - start + fn poll_datagram_once(&mut self, waiter: &kio::Waiter) -> Option>>> { + self.sync(waiter); + self.let_go(); + let groups = self.groups; + for index in (0..self.copies.len()).rev() { + loop { + let copy = &mut self.copies[index]; + if copy.datagrams_done.is_some() || copy.until.is_some() { + break; + } + let sub = match copy.poll_ready(groups, waiter) { + Poll::Pending => break, + Poll::Ready(Err(err)) => { + copy.datagrams_done = Some(Err(err)); + break; + } + Poll::Ready(Ok(sub)) => sub, + }; + match sub.poll_recv_datagram(waiter) { + Poll::Pending => break, + Poll::Ready(Ok(Some(datagram))) => { + if copy.datagrams_seen.is_some_and(|seen| datagram.sequence <= seen) { + continue; + } + self.datagram = self.datagram.max(Some(datagram.sequence)); + return Some(Poll::Ready(Ok(Some(datagram)))); + } + Poll::Ready(Ok(None)) => { + copy.datagrams_done = Some(match copy.track.poll_complete(&kio::Waiter::noop()) { + Poll::Ready(end) => end, + Poll::Pending => Ok(()), + }); + break; + } + Poll::Ready(Err(err)) => { + copy.datagrams_done = Some(Err(err)); + break; + } + } + } + } + self.poll_end(waiter, |copy| copy.datagrams_done.clone()) } - /// The newest cached group across every spliced segment; see - /// [`track::Consumer::peek_latest`]. - pub(crate) fn peek_latest(&self) -> Option { - self.peek_below(None) + fn poll_finished(&mut self, waiter: &kio::Waiter) -> Poll> { + self.sync(waiter); + let groups = self.groups; + let concluded = self.route.end(); + if let Some(Some(Err(err))) = concluded { + return Poll::Ready(Err(err)); + } + let serving = self.generation; + if let Some(copy) = self.copies.last_mut().filter(|copy| Some(copy.generation) == serving) { + match copy.poll_ready(groups, waiter) { + Poll::Ready(Ok(sub)) => match sub.poll_finished(waiter) { + Poll::Ready(Ok(fin)) => return Poll::Ready(Ok(fin)), + // The copy failed: only the end of the track if no front replaces it. + Poll::Ready(Err(err)) if concluded.is_some() => return Poll::Ready(Err(err)), + _ => {} + }, + Poll::Ready(Err(err)) if concluded.is_some() => return Poll::Ready(Err(err)), + _ => {} + } + } else if concluded.is_some() { + return Poll::Ready(Err(Error::Dropped)); + } + let _ = self.route.poll_changed(serving.unwrap_or_default(), waiter); + Poll::Pending } - /// A cached group by sequence from the newest segment; see - /// [`track::Consumer::peek_group`]. - pub(crate) fn peek_group(&self, sequence: u64) -> Option { - let track = self.state.read().segments.last().map(|segment| segment.track.clone())?; - track.peek_group(sequence) + /// The serving copy's subscription, once it answered; `None` once the track ended + /// without one. + fn poll_serving(&mut self, waiter: &kio::Waiter) -> Poll> { + self.sync(waiter); + let groups = self.groups; + let serving = self.generation; + let concluded = self.route.end().is_some(); + let copy = self.copies.last_mut().filter(|copy| Some(copy.generation) == serving); + match copy { + Some(copy) => match copy.poll_ready(groups, waiter) { + Poll::Ready(Ok(sub)) => Poll::Ready(Some(sub)), + Poll::Ready(Err(_)) if concluded => Poll::Ready(None), + _ => { + let _ = self.route.poll_changed(serving.unwrap_or_default(), waiter); + Poll::Pending + } + }, + None if concluded => Poll::Ready(None), + None => { + let _ = self.route.poll_changed(serving.unwrap_or_default(), waiter); + Poll::Pending + } + } } - /// The nearest cached group below `sequence` across every spliced segment; see - /// [`track::Consumer::peek_before`]. - pub(crate) fn peek_before(&self, sequence: u64) -> Option { - self.peek_below(Some(sequence)) + fn raise_start_to(&mut self, start: u64) { + self.groups.0 = self.groups.0.max(start); + for copy in &mut self.copies { + if let Sub::Ready(sub) = &mut copy.sub { + sub.raise_start_to(start); + } + } } - /// Walk the segments newest-first for the first one holding a group in range. - /// - /// Every segment, not just the newest: a takeover splices in a fresh track with no - /// groups at all, and a peek that stopped there would read the whole logical track - /// as empty for the entire failover window. Ranges are disjoint and ascending, so - /// the first hit walking backwards is the newest. - /// - /// The segment list is copied out rather than held, since resolving each candidate - /// takes the underlying track's own lock. - fn peek_below(&self, before: Option) -> Option { - let segments = self.state.read().segments.clone(); - segments.iter().rev().find_map(|segment| segment.peek_below(before)) + fn end_at(&mut self, end: Cap) { + self.groups.1 = end; + for copy in &mut self.copies { + if let Sub::Ready(sub) = &mut copy.sub { + sub.end_at(end); + } + } + } + + fn latest(&self) -> Option { + self.copies.iter().filter_map(|copy| copy.track.latest()).max() } } -/// The pollable state of a [`Consumer::fetch_group`]; awaited via the -/// [`kio::Pending`] wrapper. +/// Carries a group across route changes; see the module docs. /// -/// Waits for a segment to exist (no route may have served the track yet), then -/// first resolves from any segment that still caches a complete copy, then -/// issues the fetch against the newest segment's track. A fetch whose copy dies -/// fails over: it re-latches onto a newer segment if one already spliced in, or -/// parks for the next takeover like a live subscription (the front aborting the -/// track ends the wait). An error from a copy that is still live (e.g. the group -/// is gone upstream) is authoritative and surfaces. -pub struct Fetching { - state: kio::Consumer, +/// Held by a group handed out from a logical track. It is consulted only once the +/// group's copy fails or stalls, so a passthrough read never touches it. +pub(crate) struct Recover { sequence: u64, - options: group::Fetch, - // The latched segment (id, its track, the in-flight fetch), set once a - // segment exists. Behind a shared lock both to allow `&self` polling and to - // break the type recursion with `track::Fetching` (which can wrap a resume - // [`Fetching`]). - #[allow(clippy::type_complexity)] - inner: kio::Lock)>>, + /// The reader it was handed out to, followed onto each new route. + reader: Weak>, + route: Consumer, + /// The route generation of the copy being read, and that copy. + generation: u64, + copy: track::Consumer, + /// Keeps a replaced copy subscribed while this group is read from it. + lease: Option>, + /// A fetch of the rest of the group from the serving route, by its generation. + fetch: Option<(u64, Fetch)>, } -impl Fetching { - /// Poll for a segment to latch the fetch onto: the newest one, strictly newer - /// than `latched` when given. `Err` when the logical track aborted; - /// `Ok(None)` when the producer is gone with no qualifying segment, so - /// nothing will ever answer. - fn poll_latch(&self, waiter: &kio::Waiter, latched: Option) -> Poll>> { - let newest = move |segments: &[Segment]| { - segments - .last() - .filter(|segment| latched.is_none_or(|latched| segment.id > latched)) - .map(|segment| (segment.id, segment.track.clone())) - }; - match self.state.poll(waiter, |s| match (&s.abort, newest(&s.segments)) { - (Some(err), _) => Poll::Ready(Err(err.clone())), - (None, Some(next)) => Poll::Ready(Ok(next)), - (None, None) => Poll::Pending, - }) { - Poll::Ready(Ok(res)) => Poll::Ready(res.map(Some)), - // The producer is gone; whatever segment it froze with is all - // there will ever be. - Poll::Ready(Err(state)) => Poll::Ready(match &state.abort { - Some(err) => Err(err.clone()), - None => Ok(newest(&state.segments)), - }), - Poll::Pending => Poll::Pending, +enum Fetch { + Pending(kio::Pending), + /// The serving route will not fetch it: wait for its subscription instead. + Refused(Error), +} + +impl Clone for Recover { + fn clone(&self) -> Self { + Self { + sequence: self.sequence, + reader: self.reader.clone(), + route: self.route.clone(), + generation: self.generation, + copy: self.copy.clone(), + lease: self.lease.clone(), + // A clone reads in parallel, so it asks for itself. + fetch: None, } } } -impl kio::Task for Fetching { - type Output = Result; - - fn poll(&mut self, waiter: &kio::Waiter) -> Poll { - if let Some(mut group) = (Consumer { - state: self.state.clone(), - }) - .cached_group(self.sequence, self.options.frame_start) - { - // Sitting where the caller asked, as a fetch from the segment's own track would. - group.start_at(self.options.frame_start); - return Poll::Ready(Ok(group)); +impl Recover { + /// Whether to look past the copy being read: it stalled while a newer route serves, or + /// it failed without a verdict on the group. Only a route that serves can refuse a + /// group on purpose (it is too old, evicted, or the application said so). + pub(crate) fn wants(&self, err: Option<&Error>, waiter: &kio::Waiter) -> bool { + let verdict = err.is_some_and(|err| { + matches!( + StreamError::from(err), + StreamError::Old | StreamError::Evicted | StreamError::App(_) + ) + }); + match err { + Some(_) if !verdict || copy_failed(&self.copy) => true, + _ => self.route.poll_changed(self.generation, waiter).is_ready(), } + } - let mut inner = self.inner.lock(); - + /// Poll for the serving route's copy of the group, positioned at frame `index`. + /// + /// `failed` is the error the current copy ended with, if it did; a stall is `None`. + /// Returns the copy's error when nothing can continue the group. + pub(crate) fn poll( + &mut self, + index: u64, + failed: Option<&Error>, + waiter: &kio::Waiter, + ) -> Poll> { + // Subscribe the reader to a new route from its newest group, which is this one or + // later, so that route's subscription delivers the rest of it. + if let Some(reader) = self.reader.upgrade() { + reader.lock().expect("reader poisoned").sync(waiter); + } loop { - if inner.is_none() { - // Wait for the first segment; the newest wins if several arrived. - let (id, track) = match ready!(self.poll_latch(waiter, None))? { - Some(next) => next, - // The producer died without a route ever serving the track. - None => return Poll::Ready(Err(Error::NotFound)), - }; - let fetch = track.fetch_group(self.sequence, self.options.clone()); - *inner = Some((id, track, fetch)); + let (generation, serving, end, closed) = { + let route = self.route.state.read(); + ( + route.generation, + route.copy.clone(), + route.end.clone(), + route.is_closed(), + ) + }; + // Registered before looking, so a route change from here on wakes the reader. + if self.route.poll_changed(generation, waiter).is_ready() && end.is_none() && !closed { + continue; } + if let Some(Err(err)) = &end { + return Poll::Ready(Err(err.clone())); + } + let Some(serving) = serving else { + // Nothing serves: a failed copy waits for a route, unless none will come. + return match (failed, closed || end.is_some()) { + (Some(err), true) => Poll::Ready(Err(err.clone())), + _ => Poll::Pending, + }; + }; + // The copy being read still serves: only a failure asks it again. + if generation == self.generation && failed.is_none() { + return Poll::Pending; + } + let res = self.poll_serving(generation, &serving, index, failed, waiter); + // The serving route failed too, and no front is left to replace it. + if let (Poll::Pending, Some(err)) = (&res, failed) + && (closed || end.is_some()) + && copy_failed(&serving) + { + return Poll::Ready(Err(match end { + Some(Err(end)) => end, + _ => err.clone(), + })); + } + if res.is_pending() + && let Some(budget) = self.poll_budget(waiter) + && serving.poll_stale(self.sequence, budget, waiter).is_ready() + { + return Poll::Ready(Err(Error::Old)); + } + return res; + } + } - let (latched, track, fetch) = inner.as_mut().expect("latched above"); - let err = match kio::Task::poll(&mut **fetch, waiter) { - Poll::Ready(Err(err)) => err, - Poll::Ready(Ok(group)) => return Poll::Ready(Ok(group)), - Poll::Pending => { - // A warm cache deliberately parks misses while demand re-splices - // upstream. Follow that newer segment without waiting for the warm - // copy to reject its queued request. - let latched = *latched; - match self.poll_latch(waiter, Some(latched)) { - Poll::Ready(Ok(Some((id, track)))) => { - let fetch = track.fetch_group(self.sequence, self.options.clone()); - *inner = Some((id, track, fetch)); - continue; - } - Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), - Poll::Ready(Ok(None)) | Poll::Pending => {} - } + /// [`Self::poll`] once a newer route serves, through its copy `serving`. + fn poll_serving( + &mut self, + generation: u64, + serving: &track::Consumer, + index: u64, + failed: Option<&Error>, + waiter: &kio::Waiter, + ) -> Poll> { + // A newer copy's subscription delivered it: continue from there. The copy that + // failed delivered it already, so its subscription will not again. + let same = generation == self.generation; + if !same && let Poll::Ready(Some(mut group)) = serving.poll_group(self.sequence, waiter) { + group.start_at(index); + if group.index() == index { + return Poll::Ready(Ok(Replacement { + group, + generation, + copy: serving.clone(), + })); + } + } - // Park on the resume state too: the front aborting must end an - // in-flight fetch even when the latched copy never answers it. - return match self.state.poll(waiter, |s| match &s.abort { - Some(err) => Poll::Ready(err.clone()), - None => Poll::Pending, - }) { - Poll::Ready(Ok(err)) => Poll::Ready(Err(err)), - // The producer froze without aborting; only the copy can - // answer now. - _ => Poll::Pending, - }; + // Otherwise ask for the rest. The subscription may still deliver it first. + if self.fetch.as_ref().is_none_or(|(asked, _)| *asked != generation) { + let options = group::Fetch::default().with_frame_start(index); + self.fetch = Some((generation, Fetch::Pending(serving.fetch_group(self.sequence, options)))); + } + let Some((_, fetch)) = &mut self.fetch else { + unreachable!("asked above") + }; + let err = match fetch { + // That route failed: the next one is asked. + Fetch::Refused(err) if route_failed(err) || copy_failed(serving) => return Poll::Pending, + Fetch::Refused(err) => err.clone(), + Fetch::Pending(pending) => match ready!(kio::Task::poll(&mut **pending, waiter)) { + Ok(mut group) => { + group.start_at(index); + if group.index() == index { + return Poll::Ready(Ok(Replacement { + group, + generation, + copy: serving.clone(), + })); + } + *fetch = Fetch::Refused(Error::NotFound); + Error::NotFound } - }; - - // The latched copy failed the fetch. Fail over to a segment spliced in - // above it, if any; ids are monotonic, so "newer" is a plain compare. - let next = match self.poll_latch(waiter, Some(*latched)) { - Poll::Ready(res) => res?, - // No replacement yet, and the waiter is registered for the next - // switch. A dead copy's failure is the route's, not the group's: - // park for the takeover, exactly like a live subscription stalls. - // A live copy's answer stands. - Poll::Pending => match track.poll_complete(&kio::Waiter::noop()) { - Poll::Ready(Err(_)) => return Poll::Pending, - _ => return Poll::Ready(Err(err)), - }, - }; - // The producer froze with nothing newer spliced in: the latched - // copy's answer is final. - let Some((id, track)) = next else { - return Poll::Ready(Err(err)); - }; - let fetch = track.fetch_group(self.sequence, self.options.clone()); - *inner = Some((id, track, fetch)); - // Loop: poll the replacement fetch in this same pass. + // That route is failing too: wait for the next one. + Err(err) if route_failed(&err) || copy_failed(serving) => { + *fetch = Fetch::Refused(err); + return Poll::Pending; + } + Err(err) => { + *fetch = Fetch::Refused(err.clone()); + err + } + }, + }; + // A refusal is a verdict only for a copy that failed, and once the serving + // subscription will not deliver the group either. + if failed.is_some() && (same || serving.poll_group(self.sequence, waiter).is_ready()) { + return Poll::Ready(Err(err)); } + + Poll::Pending } -} -/// The frames of `sequence` a `[start, end)` segment range serves, as a half-open index -/// range. `None` when the group falls outside the range entirely. -fn frames(start: Option, end: Option, sequence: u64) -> Option<(u64, Option)> { - let start = match start { - Some(start) if start.group > sequence => return None, - Some(start) if start.group == sequence => start.frame, - _ => 0, - }; - let end = match end { - // A boundary at the head of a later group leaves this one whole. - Some(end) if end.group > sequence => None, - Some(end) if end.group == sequence => Some(end.frame), - Some(_) => return None, - None => None, - }; - Some((start, end)) + /// The reader's budget for a late group; `None` for a fetch, which has no live edge + /// to be late against. + pub(crate) fn poll_budget(&self, waiter: &kio::Waiter) -> Option { + let reader = self.reader.upgrade()?; + let mut reader = reader.lock().expect("reader poisoned"); + // A held group or frame may be the only thing being polled. Mirror and watch + // preferences here too, before judging its budget against the live edge. + reader.sync(waiter); + Some(reader.mirrored.max_age) + } + + /// Commit the replacement only once the caller has acquired its cursor or frame. + pub(crate) fn adopt(&mut self, replacement: &Replacement) { + self.generation = replacement.generation; + self.copy = replacement.copy.clone(); + // A later switch must keep this copy subscribed while the recovered group + // still reads it, just like a group handed out from that copy directly. + self.lease = self.reader.upgrade().and_then(|reader| { + let reader = reader.lock().expect("reader poisoned"); + reader + .copies + .iter() + .find(|copy| copy.generation == replacement.generation) + .map(|copy| copy.lease.clone()) + }); + self.fetch = None; + } } -/// The exclusive group cap a half-open segment can serve, or no bound for the newest segment. -fn last_group(end: Option) -> Option { - end.and_then(|end| Cap::from(end.group_end()).exclusive()) +/// A candidate cursor, committed only when the requested read can use it. +pub(crate) struct Replacement { + pub(crate) group: group::Consumer, + pub(crate) copy: track::Consumer, + generation: u64, } -/// A group assembled from several routes' copies, joined at frame boundaries. -/// -/// Rather than being fed, it pulls: on every frame it asks the segment list which route -/// owns the next frame of this group and reads that route's copy. A boundary landing -/// mid-group therefore reaches an already-handed-out group with no bookkeeping, and a -/// reader that never touches the subscription again still picks the continuation up. -/// -/// A copy that dies waits for a replacement route. A missing continuation subscribes -/// that route from the frame this group needs; a refusal ends the wait, as does a live -/// edge past this group when the route declares no start. -pub(crate) struct Group { - state: kio::Consumer, - /// The logical subscription's live max age budget. - subscription: kio::Consumer, - /// The logical reader's drift anchor, used only to judge each route's copy. - anchor: kio::Consumer, - - /// The logical group being assembled. +/// A fetch of one group through whichever route serves the logical track. +pub(crate) struct Fetching { + route: Consumer, sequence: u64, - - /// The index of the next frame to return. - index: u64, - - /// Exclusive cap from [`Self::end_at`]. - end: Option, - - /// The route being read: its segment, the exclusive frame bound the segment puts on - /// this group, and this reader's positioned copy. - current: Option, - - /// Demand held on a replacement copy until this group ends, since the parent - /// subscriber may be idle. - waiting: Option, - - /// The segment whose copy died under us, and why. - dead: Option<(u64, Error)>, - - /// The tagged logical subscriber's meter for route-copy expiry only. - stale_stats: crate::stats::Meter, + options: group::Fetch, + /// The fetch in flight, and the route generation it went to. + pending: Option<(u64, kio::Pending)>, } -/// A group reader's own subscription on one segment's route, starting at the frame -/// it needs. A floor never widens the parent's request past it, where a live-edge -/// start would erase the parent's continuation floor from the aggregate. -struct Waiting { - segment: u64, - start: Position, - track: track::Consumer, - pending: kio::Pending, -} +impl kio::Task for Fetching { + type Output = Result; -impl Waiting { - fn new(segment: u64, start: Position, track: &track::Consumer) -> Self { - Self { - segment, - start, - track: track.clone(), - pending: track.subscribe(track::Subscription::default().with_start(start)), + fn poll(&mut self, waiter: &kio::Waiter) -> Poll { + loop { + let (generation, serving) = { + let route = self.route.state.read(); + (route.generation, route.copy.clone()) + }; + let Some(serving) = serving else { + return match self.route.end() { + Some(Some(Err(err))) => Poll::Ready(Err(err)), + Some(_) => Poll::Ready(Err(Error::NotFound)), + None => { + ready!(self.route.poll_changed(generation, waiter)); + continue; + } + }; + }; + if self.route.poll_changed(generation, waiter).is_ready() && self.route.end().is_none() { + continue; + } + if self.pending.as_ref().is_none_or(|(asked, _)| *asked != generation) { + self.pending = Some((generation, serving.fetch_group(self.sequence, self.options.clone()))); + } + let (_, pending) = self.pending.as_mut().expect("asked above"); + let result = match kio::Task::poll(&mut **pending, waiter) { + Poll::Ready(result) => result, + Poll::Pending => { + return match self.route.end() { + Some(Some(Err(err))) => Poll::Ready(Err(err)), + _ => Poll::Pending, + }; + } + }; + return match result { + Ok(group) => Poll::Ready(Ok(group.with_recover(Recover { + sequence: self.sequence, + reader: Weak::new(), + route: self.route.clone(), + generation, + copy: serving.clone(), + lease: None, + fetch: None, + }))), + // The route failed: ask the next one, unless none will come. + Err(err) if (route_failed(&err) || copy_failed(&serving)) && self.route.end().is_none() => { + ready!(self.route.poll_changed(generation, waiter)); + continue; + } + Err(err) => Poll::Ready(Err(err)), + }; } } } -struct Current { - segment: u64, - cap: Option, - bound: Option, - group: group::Consumer, -} +#[cfg(test)] +mod test { + use std::time::Duration; -/// A route covering one position: the segment id, its track, and the segment's -/// exclusive frame bound on the group (outer `None` when the group escapes the -/// segment's range entirely, from [`frames`]). -type Covering = (u64, track::Consumer, Option>, Option); + use futures::FutureExt; -impl Clone for Group { - fn clone(&self) -> Self { - // Cursors are per-reader, so the clone re-latches its own cursor over the - // same copy at this reader's position. Re-resolving from the segment list - // instead would break fanout for a pruned segment's group: the list has - // forgotten the route, but the latch still holds its frames. - let current = self.current.as_ref().and_then(|current| { - let mut group = current.group.clone(); - group.start_at(self.index); - // The copy no longer holds this position (evicted); the clone - // re-resolves like any unlatched reader rather than misaligning. - (group.index() == self.index).then_some(Current { - segment: current.segment, - cap: current.cap, - bound: current.bound, - group, - }) - }); - Self { - state: self.state.clone(), - subscription: self.subscription.clone(), - anchor: self.anchor.clone(), - sequence: self.sequence, - index: self.index, - end: self.end, - current, - // Each reader holds its own demand, so dropping the original cannot - // strand a clone still reading the tail. - waiting: self - .waiting - .as_ref() - .map(|w| Waiting::new(w.segment, w.start, &w.track)), - dead: self.dead.clone(), - stale_stats: self.stale_stats.clone(), - } + use super::*; + use crate::{Timestamp, broadcast}; + + /// A route's copy of the track. + fn copy() -> track::Producer { + track::Producer::new(Arc::new(broadcast::Info::default()), "video", None) } -} -impl Group { - fn new( - state: kio::Consumer, - subscription: kio::Consumer, - anchor: kio::Consumer, - sequence: u64, - index: u64, - ) -> Self { - Self { - state, - subscription, - anchor, - sequence, - index, - end: None, - current: None, - waiting: None, - dead: None, - stale_stats: Default::default(), - } + /// The logical track `routes` serves. + fn logical(routes: &Producer) -> track::Producer { + track::Request::new(Arc::new(broadcast::Info::default()), "video") + .routes(routes.consume()) + .accept(None) } - /// The number of frames known on the current route copy, never below this - /// logical cursor's position. - pub(crate) fn frame_count(&self) -> usize { - let count = self - .current - .as_ref() - .map_or(self.index, |current| current.group.frame_count() as u64) - .max(self.index); - usize::try_from(count).unwrap_or(usize::MAX) + fn subscribe(logical: &track::Producer, max_age: Duration) -> track::Subscriber { + let subscription = Subscription::default().with_max_age(max_age); + logical + .consume() + .subscribe(subscription) + .now_or_never() + .unwrap() + .unwrap() } - /// Attribute expiry for route-specific copies behind this logical group. - pub(crate) fn set_stale_meter(&mut self, meter: crate::stats::Meter) { - if let Some(current) = &mut self.current { - current.group.set_stale_meter(meter.clone()); - } - self.stale_stats = meter; + fn recv(sub: &mut track::Subscriber) -> group::Consumer { + sub.recv_group() + .now_or_never() + .expect("a group") + .unwrap() + .expect("track ended") } - /// Pre-latch the delivering route's own copy, so the payload it already - /// delivered survives even if its segment is pruned before the reader drains - /// it. Costs nothing otherwise: the per-frame reuse check validates the latch - /// against the live segment list, so a moved cap still re-routes the reader. - /// - /// The copy must actually hold this reader's position: a partial copy that - /// starts higher (a peer delivering a partial group nobody asked for) is not - /// latched, since the reuse path trusts the latch's alignment and would - /// misnumber its frames. The reader re-resolves instead, and the peek path's - /// own check buries the copy as lagged. - fn latched(mut self, segment: u64, cap: Option, bound: Option, mut group: group::Consumer) -> Self { - group.end_at(cap.map_or(Bound::Unbounded, Bound::Excluded)); - group.start_at(self.index); - group.set_stale_meter(self.stale_stats.clone()); - if group.index() == self.index { - self.current = Some(Current { - segment, - cap, - bound, - group, - }); - } - self + fn read(group: &mut group::Consumer) -> Option> { + let frame = group.read_frame().now_or_never().expect("a frame").unwrap(); + frame.map(|frame| frame.payload.to_vec()) } - pub fn index(&self) -> u64 { - self.index + fn ts(ms: u64) -> Timestamp { + Timestamp::from_millis(ms).unwrap() } - pub fn start_at(&mut self, index: u64) { - if index <= self.index { - return; - } - self.index = index; - // Keep the latch when its copy still covers the new cursor: for a pruned - // segment it is the only copy left, and the per-read reuse check still - // re-routes if a different route owns the position. A copy that cannot - // land exactly there (or whose bound the cursor passed) is dropped, and - // the next read re-resolves. - if let Some(current) = &mut self.current { - current.group.start_at(index); - if current.group.index() != index || current.cap.is_some_and(|cap| index >= cap) { - self.current = None; - } - } + #[test] + fn a_reader_reads_the_serving_copy() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::ZERO); + + let mut group = a.create_group(group::Info { sequence: 0 }).unwrap(); + group.write_frame(ts(0), b"a".as_ref()).unwrap(); + group.finish().unwrap(); + + let mut reading = recv(&mut sub); + assert_eq!(read(&mut reading), Some(b"a".to_vec())); + assert_eq!(read(&mut reading), None); } - /// Cap the reader at `index` (exclusive), the form [`Cap::exclusive`] produces. - pub fn end_at(&mut self, index: Option) { - self.end = index; + #[test] + fn a_stalled_group_continues_from_the_replacement() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + + let mut open = a.create_group(group::Info { sequence: 0 }).unwrap(); + open.write_frame(ts(0), b"a".as_ref()).unwrap(); + open.write_frame(ts(1), b"b".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + assert_eq!(read(&mut reading), Some(b"a".to_vec())); + assert_eq!(read(&mut reading), Some(b"b".to_vec())); + + // The replacement holds the whole group; the replaced route stays quiet. + let b = copy(); + let mut whole = b.create_group(group::Info { sequence: 0 }).unwrap(); + for (ms, frame) in [(0, b"a"), (1, b"b"), (2, b"c")] { + whole.write_frame(ts(ms), frame.as_ref()).unwrap(); + } + routes.serve(b.consume()); + + assert_eq!(read(&mut reading), Some(b"c".to_vec())); + whole.finish().unwrap(); + assert_eq!(read(&mut reading), None); + assert!( + sub.recv_group().now_or_never().is_none(), + "the group was handed out twice" + ); } - /// Locate the route owning `position`: the shared resolve behind the read - /// cursor and the finish probe. `dead` is the segment already given up on - /// for these frames. `Ready(None)` once nothing can ever serve them. - fn poll_covering(&self, position: Position, dead: Option, waiter: &kio::Waiter) -> Poll> { - let sequence = self.sequence; - let located = self.state.poll(waiter, |state| { - // Waiting for a replacement route only makes sense while one can still - // arrive. A finished logical track has no more switches coming, and an - // aborted one is over outright, so a dead copy is the end of the group - // rather than a gap to park on. - // - // Nor can one arrive below the resume point. [`Producer::takeover`] - // derives every boundary from [`ResumeState::resume_position`], which - // only moves forward, so once it is past this position no future segment - // will ever cover it and nobody will be asked for these frames again. - // - // Only once a route has been given up on, though: a live route that has - // not delivered this group yet may still do so out of order, and its own - // progress is what moved the resume point past us. - let stranded = dead.is_some() && state.resume_position().is_some_and(|resume| resume > position); - // Below the pruned floor no segment exists and none can arrive: - // the coverage was dropped along with the segments that held it. - let lost = state.pruned.is_some_and(|floor| position < floor); - let terminal = state.finished || state.abort.is_some() || stranded || lost; - match state.segments.iter().find(|segment| segment.covers(position)) { - // The route that owns these frames was given up on. The verdict is - // reversible: a peek miss can come from the route's declared start, - // which demand moving backward lowers, so reconsider before the - // terminal checks. It has to come first because the arrival itself - // is what strands us (it moves the resume point past this - // position), which would otherwise condemn the very copy that - // resolves the wait. Watching the cache is also what wakes this - // poll when the copy lands. - Some(segment) if dead == Some(segment.id) => { - match segment.track.poll_serving_group(sequence, position.frame, waiter) { - Poll::Ready(()) => Poll::Ready(Some(( - segment.id, - segment.track.clone(), - frames(segment.start, segment.end, sequence).map(|(_, end)| end), - last_group(segment.end), - ))), - Poll::Pending => match terminal { - true => Poll::Ready(None), - false => Poll::Pending, - }, - } - } - Some(segment) => Poll::Ready(Some(( - segment.id, - segment.track.clone(), - frames(segment.start, segment.end, sequence).map(|(_, end)| end), - last_group(segment.end), - ))), - // No route owns them yet; park unless none is coming. - None if terminal => Poll::Ready(None), - None => Poll::Pending, - } - }); + #[test] + fn a_half_read_frame_continues_from_the_replacement() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + + let mut open = a.create_group(group::Info { sequence: 0 }).unwrap(); + let mut frame = open + .create_frame(crate::frame::Info { + size: 4, + timestamp: ts(0), + }) + .unwrap(); + frame.write(b"ab".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + let mut chunks = reading.next_frame().now_or_never().unwrap().unwrap().unwrap(); + assert_eq!( + &chunks.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], + b"ab" + ); - match located { - Poll::Ready(Ok(found)) => Poll::Ready(found), - // The producer is gone, so the segment list is frozen. - Poll::Ready(Err(_)) => Poll::Ready(None), - Poll::Pending => Poll::Pending, - } + // The route dies mid-frame, and the replacement holds the frame whole. + drop(frame); + a.clone().abort(Error::Dropped).unwrap(); + let b = copy(); + let mut whole = b.create_group(group::Info { sequence: 0 }).unwrap(); + whole.write_frame(ts(0), b"abcd".as_ref()).unwrap(); + routes.serve(b.consume()); + + assert_eq!( + &chunks.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], + b"cd" + ); + assert!(chunks.read_chunk().now_or_never().unwrap().unwrap().is_none()); } - /// Ask the replacement for this group's continuation even when the parent subscriber - /// is idle. - /// - /// The demand outlives the peek: a copy whose header has arrived, even before - /// this reader first looked, still needs it for the tail. It is released only - /// once the group ends or moves routes. - fn poll_copy( - &mut self, - segment: u64, - start: Position, - track: &track::Consumer, - waiter: &kio::Waiter, - ) -> Poll>> { - // A route is reused whatever `start` asks for, since the reader's position only - // moves forward and the segments are disjoint: keeping the earlier floor asks for - // a superset of what a later read needs, and re-subscribing to narrow it would - // drop the demand in between. - if self.waiting.as_ref().is_none_or(|w| w.segment != segment) { - self.waiting = Some(Waiting::new(segment, start, track)); - } - let result = (|| { - if let Poll::Ready(Some(copy)) = track.poll_peek_group(self.sequence, waiter) { - return Poll::Ready(Ok(Some(copy))); - } - // The pending handle owns the demand; accepting it checks refusals without - // moving this group's cursor or asking for historical data via FETCH. - ready!(self.waiting.as_ref().expect("registered above").pending.poll_ok(waiter))?; - // A declared start already tells the peek whether this group is coming, even - // when a newer group's stream overtakes it. Without one, a live edge past - // this group is the only sign it never will. - if ready!(track.poll_start(waiter)).is_none() - && track - .peek_latest() - .is_some_and(|latest| latest.sequence > self.sequence) - { - return Poll::Ready(Err(Error::NotFound)); + #[test] + fn a_group_both_routes_deliver_is_handed_out_once() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + let b = copy(); + for (track, sequences) in [(&a, 0..2), (&b, 0..3)] { + for sequence in sequences { + let mut group = track.create_group(group::Info { sequence }).unwrap(); + group.write_frame(ts(sequence), b"x".as_ref()).unwrap(); + group.finish().unwrap(); } - track.poll_peek_group(self.sequence, waiter).map(Ok) - })(); - if !matches!(result, Poll::Pending | Poll::Ready(Ok(Some(_)))) { - self.waiting = None; } - result - } - - /// Point `current` at the route owning frame `index` of this group. - /// - /// `Ready(Err(_))` once no route can ever serve those frames: a group only ends - /// cleanly through its unbounded tail copy, so this is always a truncation. - fn poll_current(&mut self, waiter: &kio::Waiter) -> Poll> { - loop { - let position = Position { - group: self.sequence, - frame: self.index, - }; - let dead = self.dead.as_ref().map(|(segment, _)| *segment); - let found = ready!(self.poll_covering(position, dead, waiter)); - let Some((segment, track, Some(cap), bound)) = found else { - self.waiting = None; - // No segment covers the position, but a latched copy still drains: - // a pruned segment's cursor holds exactly the frames it owned (its - // cap was its produced edge), so read it dry before giving up. Once - // it ends or dies, `current` clears and the next resolve settles it. - if self.current.is_some() { - return Poll::Ready(Ok(())); - } - return Poll::Ready(Err(self.give_up())); - }; + assert_eq!(recv(&mut sub).sequence, 0); + assert_eq!(recv(&mut sub).sequence, 1); + routes.serve(b.consume()); + assert_eq!(recv(&mut sub).sequence, 2); + assert!(sub.recv_group().now_or_never().is_none()); + } - // Reuse the positioned handle while the route and its bound hold: it carries - // a frame prefetch that re-resolving would throw away. - if self - .current - .as_ref() - .is_some_and(|current| current.segment == segment && current.cap == cap && current.bound == bound) - { - return Poll::Ready(Ok(())); - } + #[test] + fn a_replaced_route_finishes_only_the_groups_it_had() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + let mut first = a.create_group(group::Info { sequence: 0 }).unwrap(); + first.write_frame(ts(0), b"a0".as_ref()).unwrap(); + assert_eq!(recv(&mut sub).sequence, 0); + + let b = copy(); + routes.serve(b.consume()); + // Both routes go on to group 1; only the serving one's copy counts. + for (track, payload) in [(&a, b"a1"), (&b, b"b1")] { + let mut group = track.create_group(group::Info { sequence: 1 }).unwrap(); + group.write_frame(ts(1), payload.as_ref()).unwrap(); + group.finish().unwrap(); + } + let mut next = recv(&mut sub); + assert_eq!(next.sequence, 1); + assert_eq!(read(&mut next), Some(b"b1".to_vec())); + } - // The route may not have delivered this group yet, so wait on its cache. - let Some(group) = ready!(self.poll_copy(segment, position, &track, waiter))? else { - // This route will never have it; fall back to whichever segment replaces it. - self.dead = Some((segment, Error::NotFound)); - continue; - }; + #[test] + fn a_group_no_route_continues_is_given_up_once_stale() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(2)); + let mut open = a.create_group(group::Info { sequence: 0 }).unwrap(); + open.write_frame(ts(0), b"a".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + assert_eq!(read(&mut reading), Some(b"a".to_vec())); + + // The replacement starts past the group and cannot fetch it; the replaced route + // stays up but quiet. + let b = copy(); + routes.serve(b.consume()); + for sequence in 1..=2 { + let mut group = b.create_group(group::Info { sequence }).unwrap(); + group.write_frame(ts(sequence * 1000), b"b".as_ref()).unwrap(); + group.finish().unwrap(); + } + assert!(reading.read_frame().now_or_never().is_none(), "not stale yet"); + + let mut group = b.create_group(group::Info { sequence: 3 }).unwrap(); + group.write_frame(ts(3000), b"b".as_ref()).unwrap(); + assert!(matches!(reading.read_frame().now_or_never(), Some(Err(Error::Old)))); + } - // `start_at` clamps up to the first frame the copy still holds, so landing - // higher than asked means this route can't cover the seam after all. Treat it - // like a dead copy and wait for one that can. - let mut group = track.guard_group(group, self.subscription.clone(), self.anchor.clone(), bound); - group.set_stale_meter(self.stale_stats.clone()); - group.start_at(self.index); - if group.index() != self.index { - self.dead = Some((segment, Error::Lagged)); - continue; + #[test] + fn datagrams_a_new_copy_replays_are_handed_out_once() { + let routes = Producer::new(); + let logical = logical(&routes); + let mut a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + let mut b = copy(); + for track in [&mut a, &mut b] { + for sequence in [1, 2] { + track.insert_datagram(sequence, ts(sequence), b"d".as_ref()).unwrap(); } - - group.end_at(cap.map_or(Bound::Unbounded, Bound::Excluded)); - self.current = Some(Current { - segment, - cap, - bound, - group, - }); - self.dead = None; - return Poll::Ready(Ok(())); } - } + b.insert_datagram(3, ts(3), b"d".as_ref()).unwrap(); - /// No replacement can arrive: report the loss that cut the group short, from the - /// route that died, or [`Error::Dropped`] when its segment was pruned away. - /// - /// The dead route stays recorded, so every later poll reaches the same verdict. - /// Clearing it would re-resolve the route's reclaimed copy as one still to come - /// and park, and a reader probing `poll_finished` for the loss would hang. - fn give_up(&self) -> Error { - let err = self.dead.as_ref().map_or(Error::Dropped, |(_, err)| err.clone()); - // The only place a spliced group's loss becomes visible, so say which frames - // went missing rather than leaving a stuck group to explain itself. An old - // group was skipped on purpose (something newer superseded it), so it is not - // a loss worth reporting. - if !matches!(crate::StreamError::from(&err), crate::StreamError::Old) { - tracing::warn!( - group = self.sequence, - frame = self.index, - %err, - "no route can serve the rest of this group" - ); - } - err + let recv = + |sub: &mut track::Subscriber| sub.recv_datagram().now_or_never().map(|d| d.unwrap().unwrap().sequence); + assert_eq!(recv(&mut sub), Some(1)); + assert_eq!(recv(&mut sub), Some(2)); + routes.serve(b.consume()); + assert_eq!(recv(&mut sub), Some(3)); + assert_eq!(recv(&mut sub), None); } - /// Advance past a copy that ran out at its boundary, or report the group's end. - /// `true` to keep reading from the next route. - fn roll(&mut self) -> bool { - let Some(current) = &self.current else { - return false; - }; - match current.cap { - Some(cap) => { - // Reaching the bound is the only clean end a bounded copy has: the - // boundary was derived from what that route produced. - debug_assert_eq!(self.index, cap, "bounded copy ended below its boundary"); - self.index = self.index.max(cap); - self.current = None; - true - } - // Unbounded: this route owns the rest of the group, so its end is the group's. - None => false, + #[derive(Default)] + struct Wake(std::sync::atomic::AtomicBool); + + impl std::task::Wake for Wake { + fn wake(self: Arc) { + self.0.store(true, std::sync::atomic::Ordering::Relaxed); } } - /// Mark the current route dead so the reader waits for a replacement instead of - /// failing the group outright. - fn bury(&mut self, err: Error) { - self.dead = self.current.take().map(|current| (current.segment, err)); + #[test] + fn a_parked_group_wakes_when_the_route_changes() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + let _open = a.create_group(group::Info { sequence: 0 }).unwrap(); + let mut reading = recv(&mut sub); + let wake = Arc::new(Wake::default()); + let waiter = kio::Waiter::new(wake.clone().into()); + assert!(reading.poll_read_frame(&waiter).is_pending()); + let b = copy(); + routes.serve(b.consume()); + assert!(wake.0.load(std::sync::atomic::Ordering::Relaxed)); } - pub fn poll_read_frame(&mut self, waiter: &kio::Waiter) -> Poll>> { - loop { - if self.end.is_some_and(|end| self.index >= end) { - // Raising the cap re-registers the demand on the next read. - self.waiting = None; - return Poll::Ready(Ok(None)); - } - ready!(self.poll_current(waiter))?; - let result = { - let current = self.current.as_mut().expect("resolved above"); - ready!(current.group.poll_read_frame(waiter)) - }; - let latency_expired = self - .current - .as_ref() - .is_some_and(|current| current.group.latency_expired()); - match result { - Ok(Some(frame)) => { - self.index += 1; - return Poll::Ready(Ok(Some(frame))); - } - Ok(None) if self.roll() => continue, - Ok(None) => { - self.waiting = None; - return Poll::Ready(Ok(None)); - } - // Terminal, and the reader may be retained past the error, so the - // demand outliving the group would pin upstream demand for good. - Err(err) if latency_expired => { - self.waiting = None; - return Poll::Ready(Err(err)); + #[test] + fn a_half_read_frame_waits_for_the_replacement_header() { + for arrives in [true, false] { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + let mut open = a.create_group(group::Info { sequence: 0 }).unwrap(); + let mut frame = open + .create_frame(crate::frame::Info { + size: 4, + timestamp: ts(0), + }) + .unwrap(); + frame.write(b"ab".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + let mut chunks = reading.next_frame().now_or_never().unwrap().unwrap().unwrap(); + assert_eq!( + &chunks.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], + b"ab" + ); + + let b = copy(); + let mut replacement = b.create_group(group::Info { sequence: 0 }).unwrap(); + routes.serve(b.consume()); + assert!(chunks.read_chunk().now_or_never().is_none()); + if !arrives { + for sequence in 1..=4 { + let mut group = b.create_group(group::Info { sequence }).unwrap(); + group.write_frame(ts(sequence * 10_000), b"new".as_ref()).unwrap(); + group.finish().unwrap(); } - Err(err) => self.bury(err), + assert!(matches!(chunks.read_chunk().now_or_never(), Some(Err(Error::Old)))); + continue; } + replacement.write_frame(ts(0), b"abcd".as_ref()).unwrap(); + assert_eq!( + &chunks + .read_chunk() + .now_or_never() + .expect("replacement header arrived") + .unwrap() + .unwrap()[..], + b"cd" + ); } } - pub fn poll_next_frame(&mut self, waiter: &kio::Waiter) -> Poll>> { - loop { - if self.end.is_some_and(|end| self.index >= end) { - // Raising the cap re-registers the demand on the next read. - self.waiting = None; - return Poll::Ready(Ok(None)); + #[test] + fn recovered_groups_and_frames_expire_against_the_new_route() { + for partial in [false, true] { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(2)); + let mut open = a.create_group(group::Info { sequence: 0 }).unwrap(); + let mut frame = open + .create_frame(crate::frame::Info { + size: 4, + timestamp: ts(0), + }) + .unwrap(); + frame.write(b"a".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + let mut chunks = reading.next_frame().now_or_never().unwrap().unwrap().unwrap(); + assert!(chunks.read_chunk().now_or_never().unwrap().is_ok()); + + let b = copy(); + let mut replacement = b.create_group(group::Info { sequence: 0 }).unwrap(); + let mut replacement_frame = replacement + .create_frame(crate::frame::Info { + size: 4, + timestamp: ts(0), + }) + .unwrap(); + replacement_frame.write(b"ab".as_ref()).unwrap(); + routes.serve(b.consume()); + assert_eq!(&chunks.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], b"b"); + // Transfer the group cursor too, while its next frame is still pending. + assert!(reading.read_frame().now_or_never().is_none()); + for sequence in 1..=4 { + let mut next = b.create_group(group::Info { sequence }).unwrap(); + next.write_frame(ts(sequence * 1000), b"x".as_ref()).unwrap(); + next.finish().unwrap(); } - ready!(self.poll_current(waiter))?; - let result = { - let current = self.current.as_mut().expect("resolved above"); - ready!(current.group.poll_next_frame(waiter)) - }; - let latency_expired = self - .current - .as_ref() - .is_some_and(|current| current.group.latency_expired()); - match result { - Ok(Some(frame)) => { - self.index += 1; - return Poll::Ready(Ok(Some(frame))); - } - Ok(None) if self.roll() => continue, - Ok(None) => { - self.waiting = None; - return Poll::Ready(Ok(None)); - } - // Terminal, and the reader may be retained past the error, so the - // demand outliving the group would pin upstream demand for good. - Err(err) if latency_expired => { - self.waiting = None; - return Poll::Ready(Err(err)); - } - Err(err) => self.bury(err), + if partial { + assert!(matches!(chunks.read_chunk().now_or_never(), Some(Err(Error::Old)))); + } else { + assert!(matches!(reading.read_frame().now_or_never(), Some(Ok(None)))); } } } - /// Resolve once the logical group ends, answering with the read cursor. Only the - /// unbounded tail copy knows that end; a group cut short at a seam no route can - /// serve fails instead, so a truncated group never passes for a complete one. - pub fn poll_finished(&mut self, waiter: &kio::Waiter) -> Poll> { - ready!(self.poll_current(waiter))?; - let current = self.current.as_mut().expect("resolved above"); - let Some(cap) = current.cap else { - let count = ready!(current.group.poll_finished(waiter)); - // A complete copy is cached, so it no longer needs the demand that pulled it. - self.waiting = None; - return Poll::Ready(count); - }; - - // A bounded copy can't declare the end; the continuation does, and when it - // can never arrive the group is truncated at the cap. Probed here, not - // left to `poll_current`: a latched bounded copy resolves without - // consulting the segment list, so this is the poll that must park on the - // seam (or a caller that never drains to it would hang with no waiter - // registered). The covering route's own copy is consulted too, since a - // route that skip-declared this group (SUBSCRIBE_START above it) never - // delivers the seam even though its segment covers it. - let seam = Position { - group: self.sequence, - frame: cap, - }; - loop { - let dead = self.dead.as_ref().map(|(segment, _)| *segment); - let Some((segment, track, _, bound)) = ready!(self.poll_covering(seam, dead, waiter)) else { - self.waiting = None; - return Poll::Ready(Err(self.give_up())); - }; - match ready!(self.poll_copy(segment, seam, &track, waiter))? { - // The continuation's copy declares the count: its own count - // already includes the frames it skipped. - Some(continuation) => { - let mut continuation = - track.guard_group(continuation, self.subscription.clone(), self.anchor.clone(), bound); - continuation.set_stale_meter(self.stale_stats.clone()); - // Ask from the seam, as `poll_current` reads from it. A fresh cursor - // sits at frame 0, below a continuation that starts past the head, - // and would report a live group as lagged. The probe's index is the - // seam, not this reader's, so answer with our own. - continuation.start_at(cap); - if continuation.index() != cap { - self.dead = Some((segment, Error::Lagged)); - continue; - } - let index = self.index; - let end = ready!(continuation.poll_finished(waiter)); - self.waiting = None; - return Poll::Ready(end.map(|_| index)); - } - // This route will never have it; wait for whatever replaces it. - None => self.dead = Some((segment, Error::NotFound)), - } - } + #[test] + fn a_pending_fetch_wakes_and_moves_to_the_new_route() { + let routes = Producer::new(); + let a = copy(); + let dynamic = a.dynamic(); + routes.serve(a.consume()); + let mut fetching = routes.consume().fetch_group(0, group::Fetch::default()); + let wake = Arc::new(Wake::default()); + let waiter = kio::Waiter::new(wake.clone().into()); + assert!(kio::Task::poll(&mut fetching, &waiter).is_pending()); + let _request = dynamic.requested_group().now_or_never().unwrap().unwrap(); + wake.0.store(false, std::sync::atomic::Ordering::Relaxed); + let b = copy(); + let mut group = b.create_group(group::Info { sequence: 0 }).unwrap(); + group.write_frame(ts(0), b"b".as_ref()).unwrap(); + group.finish().unwrap(); + routes.serve(b.consume()); + assert!(wake.0.load(std::sync::atomic::Ordering::Relaxed)); + assert!(matches!(kio::Task::poll(&mut fetching, &waiter), Poll::Ready(Ok(_)))); } -} - -/// A subscriber's cursor over one segment. -struct SegmentSub { - id: u64, - start: Option, - end: Option, - /// Groups below this were handed out before the cursor subscribed: a warm cache - /// taking over a reader's cursors must not surface them again. - floor: u64, - /// Where the source is asked to start; see [`Segment::ask`]. - ask: Option, - sub: SubState, - /// A completed segment's cursor, retained while parked groups may need their - /// max age budget re-evaluated after the outer cap rises. - terminal: Option, - /// The drift anchor for this segment's cursor as of the last - /// [`Subscriber::refresh_anchor`], applied when a pending cursor activates. - anchor: Anchor, - /// The producer dropped this segment (pruned, or replaced before producing). - /// The cursor drains what it already holds, then retires; see - /// [`Self::retired`]. - pruned: bool, - /// Received groups held back by the subscriber's [`Subscriber::end_at`] cap, - /// re-offered once the cap rises (arrival-order reads consume the underlying - /// cursor, so they are parked here instead of dropped). Keyed by sequence so - /// the lowest is re-offered first; holding them here (rather than blocking on - /// the first) keeps in-range groups that arrive behind a capped one flowing. - parked: BTreeMap, - /// Set while this is a warm segment (see [`Producer::park`]) that has not been - /// cleared for live reads: the copy spliced after it, once there is one. - warm: Option, -} - -/// A warm segment waiting on the copy spliced after it; see [`Subscriber::poll_activate`]. -struct Warm { - /// The cache's newest group. - edge: Option, - next: Option, -} -impl SegmentSub { - /// The cursor that can evaluate staleness, whether the segment is live or has - /// already completed. - fn stale_sub_mut(&mut self) -> Option<&mut track::Subscriber> { - match &mut self.sub { - SubState::Active(sub) => Some(sub.as_mut()), - _ => self.terminal.as_mut(), + #[test] + fn a_pending_recovery_fetch_does_not_disable_expiry() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(2)); + let mut open = a.create_group(group::Info { sequence: 0 }).unwrap(); + open.write_frame(ts(0), b"a".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + assert_eq!(read(&mut reading), Some(b"a".to_vec())); + let b = copy(); + let _dynamic = b.dynamic(); + routes.serve(b.consume()); + assert!(reading.read_frame().now_or_never().is_none()); + for sequence in 1..=4 { + let mut group = b.create_group(group::Info { sequence }).unwrap(); + group.write_frame(ts(sequence * 1000), b"b".as_ref()).unwrap(); + group.finish().unwrap(); } + assert!(matches!(reading.read_frame().now_or_never(), Some(Err(Error::Old)))); } - /// Move an active cursor into terminal retention and mark the segment done. - fn complete(&mut self, end: Result>) { - let previous = std::mem::replace(&mut self.sub, SubState::Done(end)); - if let SubState::Active(sub) = previous { - self.terminal = Some(*sub); - } + #[test] + fn a_fetched_group_wakes_when_the_route_changes() { + let routes = Producer::new(); + let a = copy(); + let _open = a.create_group(group::Info { sequence: 0 }).unwrap(); + routes.serve(a.consume()); + let mut reading = kio::Pending::new(routes.consume().fetch_group(0, group::Fetch::default())) + .now_or_never() + .unwrap() + .unwrap(); + let wake = Arc::new(Wake::default()); + let waiter = kio::Waiter::new(wake.clone().into()); + assert!(reading.poll_read_frame(&waiter).is_pending()); + routes.serve(copy().consume()); + assert!(wake.0.load(std::sync::atomic::Ordering::Relaxed)); } - /// The first group this segment can serve, for the underlying read cursor. - fn first_group(&self) -> u64 { - self.start.map_or(0, |start| start.group).max(self.floor) + #[test] + fn ending_the_front_releases_a_stalled_group() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + let _open = a.create_group(group::Info { sequence: 0 }).unwrap(); + let mut reading = recv(&mut sub); + assert!(reading.read_frame().now_or_never().is_none()); + routes.end(Err(Error::Unroutable)); + assert!(matches!( + reading.read_frame().now_or_never(), + Some(Err(Error::Unroutable)) + )); } - /// The exclusive group cap this segment can serve, for the underlying read - /// cursor. `None` while it is the newest segment. - fn last_group(&self) -> Option { - // An empty segment would serve nothing, and no switch produces one: every - // boundary comes from `resume_position`, which sits at or above the first frame. - // `None` here reads as "no cap", which is why it is asserted rather than relied - // on; the authoritative filter is `Segment::covers`, which uses the exclusive - // bound directly. - debug_assert!( - self.end != Some(Position::default()), - "a segment cannot end at the start of the track" + #[test] + fn a_datagram_route_failure_waits_for_replacement() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + a.abort(Error::Dropped).unwrap(); + assert!(sub.recv_datagram().now_or_never().is_none()); + let mut b = copy(); + routes.serve(b.consume()); + b.insert_datagram(1, ts(0), b"b".as_ref()).unwrap(); + assert_eq!( + sub.recv_datagram().now_or_never().unwrap().unwrap().unwrap().sequence, + 1 ); - last_group(self.end) } - /// Whether a producer-dropped segment is spent and can be removed. A capped - /// cursor is kept until it drains (it may hold delivered-but-unread groups, - /// including one parked at the subscriber's cap, whose re-offer latches the - /// delivering copy and so still reads out post-prune); an uncapped one was - /// replaced before producing, so it holds nothing. The straggler bound in - /// `reap` cuts what lingers too long, parked group and all. - fn retired(&self) -> bool { - self.pruned && (self.end.is_none() || (matches!(self.sub, SubState::Done(_)) && self.parked.is_empty())) + #[test] + fn datagram_readers_release_replaced_subscriptions() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + let b = copy(); + routes.serve(b.consume()); + assert!(sub.recv_datagram().now_or_never().is_none()); + let readers = routes.readers.lock().unwrap(); + let reader = readers[0].upgrade().unwrap(); + assert_eq!(reader.lock().unwrap().copies.len(), 1); } -} - -enum SubState { - /// Waiting for the underlying track's info (it may not be accepted yet). - Pending(kio::Pending), - /// Live cursor over the underlying track. Boxed: the subscriber dwarfs the other - /// variants, and every segment holds this enum. - Active(Box), - /// The underlying track ended: `Ok` with the group count when it finished - /// cleanly, `Err` with the cause when it aborted or was dropped. An abort only - /// surfaces once no switch can follow: until then a dead route stalls the - /// logical track for the next switch to replace. - Done(Result>), -} - -/// A live subscription spliced across every segment of a logical track. -/// -/// Reads switch between the underlying [`track::Subscriber`]s at the segment -/// boundaries. A segment's session failing does not error the subscription; it -/// stalls until [`Producer::switch`] provides a replacement. Once the producer -/// [`finish`](Producer::finish)es, or goes away without a terminal state, no -/// replacement can follow, so the track ends as its final segment's track ended -/// once the remaining segments drain: cleanly if it finished, with its error if it -/// died. With nobody left to splice a replacement, a stall would never end. -pub struct Subscriber { - state: kio::Consumer, - /// This subscriber's preferences; shared with control handles, so changes are - /// picked up in [`Self::poll_sync`] and re-sliced onto every segment. - prefs: kio::Producer, - last_prefs: Subscription, - - /// Last observed producer epoch; a mismatch triggers a reconcile. - epoch: u64, - finished: bool, - abort: Option, - /// The producer is gone without a terminal state: the segment list is frozen, - /// so once it drains the read paths end as the final segment did (no takeover - /// can ever resume the track). - closed: bool, - - /// Cursors over the segments, in segment order. - segments: Vec, - - /// One past the highest sequence returned by [`Self::next_group`]. - next_sequence: u64, - /// Minimum sequence to surface, set by [`Self::start_at`]. - min_sequence: u64, - /// Exclusive cap for [`Self::next_group`], set by [`Self::end_at`]. - end_sequence: Option, - /// The anchor imposed by a reader wrapping this subscriber (a nested splice - /// segment), folded into every drift anchor pushed onto the segments. Its cap never - /// bounds delivery: the outer reader enforces its own window, and an inner cap would - /// hide a segment's completion from it. - outer: Anchor, - /// The logical drift anchor ([`Self::end_sequence`] and [`Self::outer`] combined, - /// with the newest edge across the segments), shared with groups that outlive this - /// cursor poll. Refreshed on every [`Self::poll_sync`]. - drift_anchor: kio::Producer, -} + #[test] + fn ending_the_front_releases_a_pending_fetch() { + let routes = Producer::new(); + let a = copy(); + let _dynamic = a.dynamic(); + routes.serve(a.consume()); + let mut fetching = routes.consume().fetch_group(0, group::Fetch::default()); + let waiter = kio::Waiter::noop(); + assert!(kio::Task::poll(&mut fetching, &waiter).is_pending()); + routes.end(Err(Error::Unroutable)); + assert!(matches!( + kio::Task::poll(&mut fetching, &waiter), + Poll::Ready(Err(Error::Unroutable)) + )); + } -impl Subscriber { - /// Sync with the producer and preferences: pick up new segments, apply moved - /// boundaries, re-slice demand, and register the waiter for the next change. - fn poll_sync(&mut self, waiter: &kio::Waiter) { - self.sync(waiter); - self.reap(); - self.refresh_anchor(); + #[test] + fn a_recovered_group_keeps_its_new_copy_subscribed_during_another_switch() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + let _open = a.create_group(group::Info { sequence: 0 }).unwrap(); + let mut reading = recv(&mut sub); + let b = copy(); + let mut replacement = b.create_group(group::Info { sequence: 0 }).unwrap(); + replacement.write_frame(ts(0), b"b".as_ref()).unwrap(); + routes.serve(b.consume()); + assert_eq!(read(&mut reading), Some(b"b".to_vec())); + let c = copy(); + routes.serve(c.consume()); + assert!(sub.recv_group().now_or_never().is_none()); + let readers = routes.readers.lock().unwrap(); + let reader = readers[0].upgrade().unwrap(); + assert!(reader.lock().unwrap().copies.iter().any(|copy| copy.generation == 2)); } - /// Reap retired cursors, then bound the live stragglers: a pruned segment's - /// cursor keeps draining (groups below its cap may still arrive out of - /// order, and its demand keeps the upstream serving them), but only the - /// newest few. Beyond the bound the oldest are cut, mirroring the - /// producer-side policy: a reader that far behind loses the range. - /// - /// Runs from [`Self::poll_sync`] so every polling entry point enforces the - /// bound; a subscriber driven only through datagrams or `poll_finished` - /// accumulates cursors all the same. - fn reap(&mut self) { - self.segments.retain(|s| !s.retired()); - let mut cut = self - .segments - .iter() - .filter(|s| s.pruned) - .count() - .saturating_sub(MAX_SEGMENTS); - if cut > 0 { - for seg in &mut self.segments { - if cut == 0 { - break; + #[test] + fn held_read_preferences_follow_the_reader() { + enum Read { + Group(Box), + Frame(crate::frame::Consumer), + } + impl Read { + fn poll(&mut self, waiter: &kio::Waiter) -> Poll> { + match self { + Self::Group(group) => group + .poll_read_frame(waiter) + .map(|result| result.map(|frame| assert!(frame.is_none()))), + Self::Frame(frame) => frame + .poll_read_chunk(waiter) + .map(|result| result.map(|frame| assert!(frame.is_none()))), } - if seg.pruned { - seg.sub = SubState::Done(Err(Error::Dropped)); - seg.terminal = None; - seg.parked.clear(); - cut -= 1; + } + } + for recovered in [false, true] { + for widen in [false, true] { + for chunked in [false, true] { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(if widen { 2 } else { 10 })); + let b = copy(); + // Both routes carry the same content, including the unfinished payload. + let mut writers = [&a, &b].map(|source| source.create_group(group::Info { sequence: 0 }).unwrap()); + let _frames: Vec<_> = writers + .iter_mut() + .map(|group| { + group.write_frame(ts(0), b"first".as_ref()).unwrap(); + chunked.then(|| { + let mut frame = group + .create_frame(crate::frame::Info { + size: 4, + timestamp: ts(1), + }) + .unwrap(); + frame.write(b"ab".as_ref()).unwrap(); + frame + }) + }) + .collect(); + let mut group = recv(&mut sub); + assert_eq!(read(&mut group), Some(b"first".to_vec())); + // Give both read APIs the same wake-driven schedule. A detached frame + // must observe preferences without its parent group being polled. + let mut reading = if chunked { + let mut frame = group.next_frame().now_or_never().unwrap().unwrap().unwrap(); + assert_eq!(&frame.read_chunk().now_or_never().unwrap().unwrap().unwrap()[..], b"ab"); + drop(group); + Read::Frame(frame) + } else { + Read::Group(Box::new(group)) + }; + let source = if recovered { + routes.serve(b.consume()); + &b + } else { + &a + }; + let wake = Arc::new(Wake::default()); + let waiter = kio::Waiter::new(wake.clone().into()); + assert!(reading.poll(&waiter).is_pending()); + if recovered { + // Retire the original copy so its preferences stop being mirrored. + assert!(sub.recv_group().now_or_never().is_none()); + assert!(a.subscription().is_none()); + } + wake.0.store(false, std::sync::atomic::Ordering::Relaxed); + sub.control() + .update( + Subscription::default() + .with_priority(7) + .with_max_age(Duration::from_secs(if widen { 10 } else { 2 })), + ) + .unwrap(); + assert!( + wake.0.load(std::sync::atomic::Ordering::Relaxed), + "recovered={recovered}, widen={widen}, chunked={chunked}: preference update did not wake the held read" + ); + for sequence in 1..=4 { + let mut next = source.create_group(group::Info { sequence }).unwrap(); + next.write_frame(ts(sequence * 1000), b"next".as_ref()).unwrap(); + next.finish().unwrap(); + } + let result = reading.poll(&waiter); + if widen { + assert!( + result.is_pending(), + "recovered={recovered}, chunked={chunked}: widened budget was ignored" + ); + } else if chunked { + assert!( + matches!(result, Poll::Ready(Err(Error::Old))), + "recovered={recovered}: narrowed budget did not truncate the frame" + ); + } else { + assert!( + matches!(result, Poll::Ready(Ok(()))), + "recovered={recovered}: narrowed budget did not end the drained group" + ); + } + assert_eq!(source.subscription().unwrap().priority, 7); + if widen { + // Observing one change must re-arm the waiter for the next change. + wake.0.store(false, std::sync::atomic::Ordering::Relaxed); + sub.control() + .update( + Subscription::default() + .with_priority(8) + .with_max_age(Duration::from_secs(10)), + ) + .unwrap(); + assert!( + wake.0.load(std::sync::atomic::Ordering::Relaxed), + "recovered={recovered}, chunked={chunked}: second update did not wake the held read" + ); + assert!(reading.poll(&waiter).is_pending()); + assert_eq!(source.subscription().unwrap().priority, 8); + } } } - self.segments.retain(|s| !s.retired()); } } - fn sync(&mut self, waiter: &kio::Waiter) { - // Preference changes re-derive every segment's demand. Loop: a poll that - // consumes a change leaves no waiter registered, so re-poll until Pending - // (mirroring the state loop below), or the next update is silently lost. - loop { - let prefs = { - let last = &self.last_prefs; - match self - .prefs - .poll(waiter, |p| if **p != *last { Poll::Ready(()) } else { Poll::Pending }) - { - Poll::Ready(Ok(guard)) => (*guard).clone(), - Poll::Ready(Err(_)) | Poll::Pending => break, + /// Replay every ordering of independent actions, with the return to A following + /// the switch to B. The payload is the oracle, independent of route bookkeeping. + #[test] + fn route_flap_interleavings_preserve_readers() { + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + enum Event { + SwitchToB, + ReturnToA, + FinishA, + FinishB, + Update, + Cancel, + } + + fn run(events: &[Event], wake_only: bool) { + let routes = Producer::new(); + let logical = logical(&routes); + let copies = [copy(), copy()]; + routes.serve(copies[0].consume()); + let mut writers: Vec<_> = copies + .iter() + .map(|copy| Some(copy.create_group(group::Info { sequence: 0 }).unwrap())) + .collect(); + writers[0] + .as_mut() + .unwrap() + .write_frame(ts(0), b"first".as_ref()) + .unwrap(); + let mut subscriptions = [ + Some(subscribe(&logical, Duration::from_secs(10))), + Some(subscribe(&logical, Duration::from_secs(10))), + ]; + let control = subscriptions[1].as_ref().unwrap().control(); + let mut groups = subscriptions.each_mut().map(|sub| Some(recv(sub.as_mut().unwrap()))); + let wakes = [Arc::new(Wake::default()), Arc::new(Wake::default())]; + let waiters = wakes.each_ref().map(|wake| kio::Waiter::new(wake.clone().into())); + let mut received: [Vec>; 2] = Default::default(); + let mut ended = [false; 2]; + + let mut poll = |groups: &mut [Option; 2], initial: bool| { + for i in 0..2 { + let woken = wakes[i].0.swap(false, std::sync::atomic::Ordering::Relaxed); + if ended[i] || (!initial && wake_only && !woken) { + continue; + } + let Some(group) = &mut groups[i] else { continue }; + loop { + match group.poll_read_frame(&waiters[i]) { + Poll::Ready(Ok(Some(frame))) => received[i].push(frame.payload.to_vec()), + Poll::Ready(Ok(None)) => { + ended[i] = true; + break; + } + Poll::Ready(Err(err)) => panic!("{events:?}, wake_only={wake_only}, reader={i}: {err}"), + Poll::Pending => break, + } + } } }; - self.last_prefs = prefs; - for seg in &mut self.segments { - let prefs = slice(&self.last_prefs, seg.ask, seg.end); - if let Some(sub) = seg.stale_sub_mut() { - let _ = sub.update(prefs); - } - } - } - - loop { - let epoch = self.epoch; - // Snapshot inside the predicate: `kio::Consumer::poll` yields the - // predicate's value on change, or the final state once closed. Inline - // the poll so its state borrow ends with this statement. - let (snapshot, closed) = match self.state.poll(waiter, |state| { - if state.epoch != epoch { - Poll::Ready(state.snapshot()) - } else { - Poll::Pending - } - }) { - Poll::Ready(Ok(snapshot)) => (Some(snapshot), false), - // The producer is gone; the state is frozen, so reconcile one last - // time and stop watching (existing segments can still drain). - Poll::Ready(Err(state)) => { - let snapshot = (state.epoch != epoch).then(|| state.snapshot()); - (snapshot, true) - } - // Unchanged, and the waiter is now registered for the next switch. - Poll::Pending => return, - }; - - if let Some(snapshot) = snapshot { - self.apply(snapshot); - } - if closed { - self.closed = true; - return; - } - // Loop: re-poll so the waiter is registered for the next change. - } - } - - /// Apply a producer snapshot: move boundaries on known segments and subscribe - /// to new ones. - fn apply(&mut self, snapshot: Snapshot) { - let Snapshot { - epoch, - finished, - abort, - segments, - } = snapshot; - self.epoch = epoch; - self.finished = finished; - self.abort = abort; - - // Mark segments the producer dropped: replaced (never produced anything, - // so their cursor holds nothing and retires at once) or pruned (capped; - // the cursor may still hold delivered-but-unread groups, so it drains - // before retiring; see `poll_segment`). A parked group survives the - // prune: its re-offer latches the delivering copy (see `hand_out`), so it - // still reads out, and the straggler bound in `reap` is what keeps such - // entries from pinning their cursors forever. - for s in &mut self.segments { - s.pruned = !segments.iter().any(|n| n.id == s.id); - } - // A park replaces every segment with a cache of what they delivered (see - // [`Producer::park`]). Cursors still draining the replaced segments would hand - // that content out a second time, so the cache takes over from them, past what - // this reader was already handed. Groups they parked at the cap are in the - // cache too, and park again from there. - let parked = segments - .iter() - .any(|n| n.warm && !self.segments.iter().any(|s| s.id == n.id)); - if parked { - self.segments.retain(|s| !s.pruned); - } - self.segments.retain(|s| !s.retired()); - - let nexts: Vec<_> = segments.iter().skip(1).map(|next| Some(next.track.clone())).collect(); - let nexts = nexts.into_iter().chain(std::iter::once(None)); - for (segment, next) in segments.into_iter().zip(nexts) { - match self.segments.iter_mut().find(|s| s.id == segment.id) { - Some(existing) => { - if let Some(warm) = &mut existing.warm { - warm.next = next; - } - if existing.end != segment.end { - // The boundary bounds the drift anchor as well as the demand; the - // anchor follows in `refresh_anchor`. - existing.end = segment.end; - if let Some(sub) = existing.stale_sub_mut() { - // Shrink the demand so the session can cap upstream. The - // read bounds stay on this subscriber (see `poll_recv_group`): - // an inner `end_at` would park boundary-crossing groups in the - // inner cursor, hiding the segment's completion. - let _ = sub.update(slice(&self.last_prefs, segment.ask, segment.end)); - } - // A still-pending subscription picks the moved boundary up - // when it activates (see `poll_activate`). Groups already handed - // out need nothing: a [`Group`] re-reads the segment list on - // every frame, so it sees the new boundary by itself. - } - } - None => { - let sub = segment - .track - .subscribe(slice(&self.last_prefs, segment.ask, segment.end)); - self.segments.push(SegmentSub { - id: segment.id, - start: segment.start, - end: segment.end, - // A cache taking over (see above) starts past what this reader has. - floor: if segment.warm { self.next_sequence } else { 0 }, - ask: segment.ask, - sub: SubState::Pending(sub), - terminal: None, - anchor: Anchor::default(), - pruned: false, - parked: BTreeMap::new(), - warm: segment.warm.then(|| Warm { - edge: segment.track.latest(), - next, - }), - }); - } - } - } - } - - /// Wrap a group just received from a segment so the reader can follow it across - /// later route changes. - /// - /// Returns `None` when the copy continues a group already handed out: its segment - /// starts partway into the group, which only happens after an earlier segment served - /// the head. Those frames reach the reader through the group it already holds, so - /// surfacing the copy again would duplicate them. - fn hand_out(&self, segment: usize, group: group::Consumer) -> Option { - let seg = &self.segments[segment]; - let sequence = group.sequence; - let (start, end) = frames(seg.start, seg.end, sequence)?; - if start != 0 { - return None; - } - // Latch the delivering copy: the spliced reader otherwise re-resolves it - // through the segment list, which forgets this route the moment its - // segment is pruned, turning a group the cursor already delivered into an - // empty husk. - let spliced = Group::new( - self.state.clone(), - self.prefs.consume(), - self.drift_anchor.consume(), - sequence, - 0, - ) - .latched(seg.id, end, seg.last_group(), group.clone()); - Some(group.into_spliced(spliced)) - } - - /// The anchor a segment's cursor measures drift against: the logical `anchor`, with - /// its cap lowered to the segment's boundary. - /// - /// A segment's inner cursor is deliberately left uncapped (an inner `end_at` would - /// park boundary-crossing groups where its completion can't be seen), so this is how - /// both bounds reach the drift anchor. Without them a segment measures staleness - /// against groups it will never surface: the route running past the boundary, or the - /// reader's own cap holding content back. The edge is left as is: it is the newest - /// content the logical track holds, which a segment's own track never sees. - /// - /// When the boundary lowers the cap, the reader's next group past it lives in a - /// later segment of `state` ([`Anchor::successor`]). Otherwise the logical anchor's - /// own successor (a wrapping splice's) still holds. - fn segment_anchor(seg: &SegmentSub, anchor: Anchor, state: &kio::Consumer) -> Anchor { - let Some(boundary) = seg.last_group() else { - return anchor; - }; - let cap = anchor.cap; - let mut capped = anchor.capped(Some(boundary)); - if capped.cap != cap { - capped.successor = Some(Successor { - state: state.weak(), - after: seg.id, - boundary, - cap, - }); - } - capped - } - - /// The logical drift anchor, as of the last [`Self::refresh_anchor`]. - fn anchor(&self) -> Anchor { - self.drift_anchor.read().clone() - } - - /// Re-derive the logical drift anchor and push it onto every segment cursor. - /// - /// The cap is the tightest any reader of this subscriber imposes: its own - /// [`Self::end_at`] and whatever a wrapping splice pushed down. The edge is the - /// newest across the producer's segments within that cap, or a wrapping splice's if - /// newer. Resolved on every sync, since each segment is a separate track and the - /// logical edge moves whenever any of them grows. - fn refresh_anchor(&mut self) { - let outer = self.outer.clone().capped(self.end_sequence); - let edge = self - .state - .read() - .live_edge(outer.cap) - .into_iter() - .chain(outer.edge.clone()) - .max_by_key(|edge| edge.sequence); - let anchor = Anchor { edge, ..outer }; - // Skip a no-op write: every handed-out group's expiry watches this channel. - if self.anchor() != anchor - && let Ok(mut current) = self.drift_anchor.write() - { - *current = anchor.clone(); - } - for seg in &mut self.segments { - let anchor = Self::segment_anchor(seg, anchor.clone(), &self.state); - seg.anchor = anchor.clone(); - if let Some(sub) = seg.stale_sub_mut() { - sub.set_anchor(anchor); - } - } - } - - /// Bound every segment's drift anchor from outside; the spliced arm of - /// [`track::Subscriber::set_anchor`], for this subscriber nested as a segment of - /// another splice. - pub(crate) fn set_anchor(&mut self, anchor: Anchor) { - self.outer = anchor; - self.refresh_anchor(); - } - - /// Count each segment's seek convictions behind the deliverer's `committed` - /// watermark; the spliced arm of [`track::Subscriber::commit_seek_stale`]. - pub(crate) fn commit_seek_stale(&mut self, committed: u64) { - for seg in &mut self.segments { - if let Some(sub) = seg.stale_sub_mut() { - sub.commit_seek_stale(committed); - } - } - } - - /// Drop every segment's conviction for `sequence` without counting it; the - /// spliced arm of [`track::Subscriber::discard_seek_conviction`]. - pub(crate) fn discard_seek_conviction(&mut self, sequence: u64) { - for seg in &mut self.segments { - if let Some(sub) = seg.stale_sub_mut() { - sub.discard_seek_conviction(sequence); - } - } - } - - /// Whether the drift budget says to skip `group`, asked of the segment whose - /// window covers it; the spliced arm of [`track::Subscriber::poll_stale`], for a - /// nested splice's parked group being re-offered after a cap rise. - /// - /// A group no current segment covers gets a `false`: with no cursor left that - /// could judge it, delivering is the safe direction. - pub(crate) fn poll_stale(&mut self, group: &group::Consumer, waiter: &kio::Waiter) -> Poll> { - for seg in &mut self.segments { - if seg.first_group() <= group.sequence - && super::subscription::before_end(group.sequence, seg.last_group()) - && let Some(sub) = seg.stale_sub_mut() - { - return sub.poll_stale(group, waiter); - } - } - Poll::Ready(Ok(false)) - } - - /// Resolve a segment's pending subscription, if any. Ready once the segment is - /// `Active` or `Done`; a rejected or closed track becomes `Done` (stall, not - /// error). Never consumes groups, so terminal-state pollers can share it. - fn poll_activate(seg: &mut SegmentSub, prefs: &Subscription, min_sequence: u64, waiter: &kio::Waiter) -> Poll<()> { - if matches!(seg.sub, SubState::Pending(_)) - && let Some(warm) = &seg.warm - { - // Nothing spliced after the cache yet, so nothing says it still leads into - // the live feed. A splice bumps the epoch, which wakes this waiter. - let Some(next) = &warm.next else { - return Poll::Pending; - }; - let start = ready!(next.poll_start(waiter)); - let edge = warm.edge; - seg.warm = None; - // The copy was asked for the cache's newest group and started past it: its - // source judged that group stale, so the older cache is no use to live reads. - if start.is_some_and(|start| edge.is_some_and(|edge| start > edge)) { - // Discarded, not finished: there is no group count, same as a dropped segment. - seg.complete(Err(Error::Dropped)); - return Poll::Ready(()); - } - } - - if let SubState::Pending(pending) = &mut seg.sub { - match ready!(pending.poll_ok(waiter)) { - Ok(mut sub) => { - // Enforce the floor on the read cursor, and re-slice demand in - // case a boundary moved while the subscription was pending. The - // upper bounds (segment boundary and `end_at` cap) are enforced by - // this subscriber, never on the inner cursor: an inner cap would - // park groups there and hide the segment's completion. Raised, not - // assigned: the inner subscription resolved its own start from its - // budget and floor, and this must not rewind past it. - sub.raise_start_to(seg.first_group().max(min_sequence)); - sub.set_anchor(seg.anchor.clone()); - let _ = sub.update(slice(prefs, seg.ask, seg.end)); - seg.sub = SubState::Active(Box::new(sub)); - } - // The underlying track was rejected or closed: stall, not error. - Err(err) => seg.sub = SubState::Done(Err(err)), - } - } - Poll::Ready(()) - } - - /// Drive one segment cursor: resolve a pending subscription, then poll for an - /// in-bounds group. Out-of-bounds groups (a route racing its cap) are skipped. - fn poll_segment( - seg: &mut SegmentSub, - prefs: &Subscription, - min_sequence: u64, - waiter: &kio::Waiter, - ) -> Poll> { - loop { - match &mut seg.sub { - SubState::Pending(_) => { - ready!(Self::poll_activate(seg, prefs, min_sequence, waiter)); - } - SubState::Active(sub) => match ready!(sub.poll_recv_group(waiter)) { - Ok(Some(group)) => { - // `start_at` already floors the cursor; enforce the cap here since - // arrival-order reads don't honor `end_at`. - if !super::subscription::before_end(group.sequence, seg.last_group()) { - continue; + poll(&mut groups, true); + for event in events { + match event { + Event::SwitchToB => routes.serve(copies[1].consume()), + Event::ReturnToA => routes.serve(copies[0].consume()), + Event::FinishA | Event::FinishB => { + let index = usize::from(*event == Event::FinishB); + let mut writer = writers[index].take().unwrap(); + if index == 1 { + writer.write_frame(ts(0), b"first".as_ref()).unwrap(); } - return Poll::Ready(Some(group)); + writer.write_frame(ts(1), b"second".as_ref()).unwrap(); + writer.finish().unwrap(); } - Ok(None) => { - let end = match sub.poll_end(waiter) { - Poll::Ready(end) => end, - Poll::Pending => Err(Error::Dropped), - }; - seg.complete(end); - return Poll::Ready(None); - } - // A dead segment stalls the logical track rather than erroring; - // the next switch resumes it. - Err(err) => { - seg.complete(Err(err)); - return Poll::Ready(None); - } - }, - SubState::Done(_) => return Poll::Ready(None), - } - } - } - - /// Seek this subscriber's lowest servable group at or above `floor` (bounded by - /// `end`, exclusive), without advancing any cursor. The shared engine behind - /// [`Self::poll_next_group`] and [`Self::poll_seek_group`]. - /// - /// Each segment's own cursor applies the drift budget as it seeks, so a group the - /// budget has convicted is stepped over here exactly as [`Self::poll_segment`] - /// steps over one. Nothing is *consumed*, though: each segment is seeked rather - /// than read, so a candidate a poll does not deliver stays where it was, and a cap - /// lowered between polls never strands a group behind a segment cursor that had - /// already stepped past it. - /// - /// `deliver` marks the caller as the one handing the group out, which stamps the - /// winner's cache entry. A nested seek passes `false`: its candidate may lose at - /// the level above, and a loser re-seeked every poll must not be shielded from - /// eviction. - fn poll_seek( - &mut self, - floor: u64, - end: Option, - deliver: bool, - waiter: &kio::Waiter, - ) -> Poll>> { - self.poll_sync(waiter); - - // Delivery is the only irrevocable signal: the sequence cursor never returns - // below `next_sequence`, so every segment's convictions behind it are final. - // The floor is no substitute; it includes `start_at`, which can be lowered - // again. A nested seek (`deliver` false) commits nothing: the outer deliverer - // reaches these segments through its own commit recursion. - if deliver { - let committed = self.next_sequence; - self.commit_seek_stale(committed); - } - - let mut floor = floor; - 'retry: loop { - let mut all_done = true; - let mut best: Option<(usize, group::Consumer)> = None; - - for index in 0..self.segments.len() { - if matches!(self.segments[index].sub, SubState::Pending(_)) - && Self::poll_activate(&mut self.segments[index], &self.last_prefs, self.min_sequence, waiter) - .is_pending() - { - all_done = false; - continue; - } - - let seg = &mut self.segments[index]; - // A boundary behind the floor has nothing to serve this poll. A pure - // skip, deliberately: the floor can come back down (a lowered - // `start_at` with nothing delivered yet), so the segment must not be - // completed or otherwise committed past its cache. - if seg.last_group().is_some_and(|last| last <= floor) { - continue; - } - - let cap = min_some(end, seg.last_group()); - match &mut seg.sub { - SubState::Active(sub) => match sub.poll_seek_group(floor, cap, waiter) { - Poll::Ready(Ok(Some(group))) => { - all_done = false; - // Strict, so the earlier segment wins a boundary tie: it - // holds the head copy, the one `hand_out` can surface. - if best.as_ref().is_none_or(|(_, best)| group.sequence < best.sequence) { - best = Some((index, group)); - } - } - // The track ran out at or below the floor: the segment drained. - Poll::Ready(Ok(None)) => { - let end = match sub.poll_end(waiter) { - Poll::Ready(end) => end, - Poll::Pending => Err(Error::Dropped), - }; - seg.complete(end); - } - // A dead segment stalls the logical track rather than erroring; - // the next switch resumes it. - Poll::Ready(Err(err)) => seg.complete(Err(err)), - Poll::Pending => all_done = false, - }, - SubState::Pending(_) => all_done = false, - SubState::Done(_) => {} - } - } - - if let Some((index, group)) = best { - let sequence = group.sequence; - // Delivery is a cache access; a candidate merely considered is not. - if deliver { - group.cache_refresh(); - } - // A copy continuing a group already handed out splices into it rather - // than surfacing again; step past it and look for the next sequence. - match self.hand_out(index, group) { - Some(group) => return Poll::Ready(Ok(Some(group))), - None => { - floor = sequence.saturating_add(1); - continue 'retry; + Event::Update => control + .update( + Subscription::default() + .with_priority(7) + .with_max_age(Duration::from_secs(2)), + ) + .unwrap(), + Event::Cancel => { + groups[0] = None; + subscriptions[0] = None; } } + poll(&mut groups, false); } - - if let Some(err) = &self.abort { - return Poll::Ready(Err(err.clone())); - } - // Finished or gone, no switch can follow: the track ends as its final - // segment did. - if all_done && (self.finished || self.closed) { - return Poll::Ready(self.final_end().map(|()| None)); + assert_eq!( + received[1], + [b"first".to_vec(), b"second".to_vec()], + "{events:?}, wake_only={wake_only}" + ); + assert!(ended[1], "{events:?}, wake_only={wake_only}: missing group end"); + assert!( + [b"first".to_vec(), b"second".to_vec()].starts_with(&received[0]), + "{events:?}: cancelled reader replayed content" + ); + assert!( + subscriptions[1].as_mut().unwrap().recv_group().now_or_never().is_none(), + "{events:?}: duplicate group" + ); + let demand = copies[0].subscription().expect("surviving reader"); + assert_eq!(demand.priority, 7, "{events:?}"); + assert_eq!(demand.max_age, Duration::from_secs(2), "{events:?}"); + drop(groups); + drop(subscriptions); + drop(control); + for copy in &copies { + assert!(copy.subscription().is_none(), "{events:?}: cancelled demand survived"); } - return Poll::Pending; } - } - - /// Poll for the next group in arrival order across the segments. - /// - /// A group that continues one already returned (a boundary landed inside it) is - /// spliced onto that group rather than surfaced again, so each sequence is handed - /// out exactly once no matter how many routes contributed to it. - /// - /// Returns `Poll::Ready(Ok(None))` once every segment completed and the - /// producer finished or was dropped, with the final segment's track finished. - /// Returns `Poll::Ready(Err(_))` if the producer aborted, or if it finished or - /// was dropped and the final segment's track died (with that track's error). - pub fn poll_recv_group(&mut self, waiter: &kio::Waiter) -> Poll>> { - self.poll_sync(waiter); - - let end_sequence = self.end_sequence; - let min_sequence = self.min_sequence; - let beyond_cap = |sequence: u64| !super::subscription::before_end(sequence, end_sequence); - - let mut all_done = true; - // An eviction aborts a parked group without touching any cursor this - // subscriber polls, so each entry needs a waiter or this poll would never - // rerun. `poll_closed` observes-or-registers under one lock: `Pending` parks - // the waiter while the group is open (an open group cannot be aborted), and - // `Ready` means closed, where only an abort invalidates the entry. A cleanly - // closed group can never gain an abort, so it needs no waiter. Checking - // `is_aborted` separately from the registration would leave a window where an - // abort lands between the two and wakes nobody. - let watch = |group: &group::Consumer| match group.poll_closed(waiter) { - Poll::Pending => true, - Poll::Ready(()) => !group.is_aborted(), - }; - - for index in 0..self.segments.len() { - // A `start_at` overtook these parked groups; drop them and read on. - // Eviction/expiry (which aborts a cached group) drops its entry too, - // bounding parking by the track's cache policy rather than retaining - // every group a long-capped subscription ever observed. - self.segments[index] - .parked - .retain(|sequence, group| *sequence >= min_sequence && watch(group)); - - // Re-offer the lowest parked group back inside the cap once it rises. - while let Some(&sequence) = self.segments[index].parked.keys().next() { - if beyond_cap(sequence) { - break; - } - let group = self.segments[index] - .parked - .remove(&sequence) - .expect("parked key just observed"); - // The cap rising widens the live edge too, so a group parked while it - // was current can be a backlog by the time it is owed again. Re-check it - // rather than hand back content the budget has since given up on. - if let Some(sub) = self.segments[index].stale_sub_mut() - && matches!(sub.poll_stale(&group, waiter), Poll::Ready(Ok(true))) - { - continue; - } - // A re-offer is a delivery: stamp it like a fresh hand-out. - group.cache_refresh(); - // Folded into a group already handed out: try the next parked one. - if let Some(group) = self.hand_out(index, group) { - self.next_sequence = self.next_sequence.max(sequence.saturating_add(1)); - return Poll::Ready(Ok(Some(group))); + fn walk(events: &mut Vec, remaining: &[Event]) { + if remaining.is_empty() { + for wake_only in [false, true] { + run(events, wake_only); } + return; } - - loop { - let polled = Self::poll_segment(&mut self.segments[index], &self.last_prefs, min_sequence, waiter); - match polled { - Poll::Ready(Some(group)) => { - if beyond_cap(group.sequence) { - // `end_at` holds the group until the cap rises rather than - // dropping it; keep draining so an in-range group that - // arrived behind it still flows. Watch it from the moment it - // parks: the retain pass above already ran, so an entry - // admitted here would otherwise sit unwatched for the rest - // of this poll, and an abort could wake nobody. - if watch(&group) { - self.segments[index].parked.insert(group.sequence, group); - } - continue; - } - if group.sequence < min_sequence { - // A `start_at` raced an already-delivered group; skip it - // and re-poll the same segment for what's behind it. - continue; - } - let sequence = group.sequence; - // Folded into a group already handed out: keep reading this - // segment rather than surfacing the same sequence twice. - let Some(group) = self.hand_out(index, group) else { - continue; - }; - self.next_sequence = self.next_sequence.max(sequence.saturating_add(1)); - return Poll::Ready(Ok(Some(group))); - } - Poll::Ready(None) => break, - Poll::Pending => break, + for (i, event) in remaining.iter().enumerate() { + if *event == Event::ReturnToA && !events.contains(&Event::SwitchToB) { + continue; } - } - - // Parked groups become deliverable if the cap rises, and a segment that - // hasn't completed can still produce; either way the track isn't over. - let seg = &self.segments[index]; - if !seg.parked.is_empty() || !matches!(seg.sub, SubState::Done(_)) { - all_done = false; - } - } - - if let Some(err) = &self.abort { - return Poll::Ready(Err(err.clone())); - } - // Finished or gone, no switch can follow: the track ends as its final - // segment did rather than stalling forever. - if all_done && (self.finished || self.closed) { - return Poll::Ready(self.final_end().map(|()| None)); - } - Poll::Pending - } - - /// Receive the next group in arrival order across the segments. - #[cfg(test)] - pub async fn recv_group(&mut self) -> Result> { - kio::wait(|waiter| self.poll_recv_group(waiter)).await - } - - /// Poll for the next group with a higher sequence than any previously - /// returned, skipping late arrivals, across the segments. - /// - /// Mirrors [`track::Ordered`]: segments are seeked rather than read, and each - /// segment's own cursor applies [`Subscription::max_age`] as it seeks, so a group - /// the budget has convicted is stepped over here too. See [`Self::poll_seek`]. - pub fn poll_next_group(&mut self, waiter: &kio::Waiter) -> Poll>> { - let floor = self.next_sequence.max(self.min_sequence); - let Some(group) = ready!(self.poll_seek(floor, self.end_sequence, true, waiter))? else { - return Poll::Ready(Ok(None)); - }; - self.next_sequence = self.next_sequence.max(group.sequence.saturating_add(1)); - // A boundary inside the delivered group leaves a copy of this sequence in the - // next segment, which may have convicted it; the delivery splices into that - // copy, so its content is served and must not be counted. - self.discard_seek_conviction(group.sequence); - // The delivery commits every segment's convictions the cursor just passed. - self.commit_seek_stale(self.next_sequence); - Poll::Ready(Ok(Some(group))) - } - - /// Seek the lowest servable group at or above `floor` (bounded by `end`) without - /// advancing this subscriber's own cursor: the nested-splice counterpart of - /// [`track::Subscriber::poll_seek_group`]. - pub(crate) fn poll_seek_group( - &mut self, - floor: u64, - end: Option, - waiter: &kio::Waiter, - ) -> Poll>> { - let floor = floor.max(self.min_sequence); - let end = min_some(end, self.end_sequence); - self.poll_seek(floor, end, false, waiter) - } - - /// Return the next group with a higher sequence than any previously returned. - #[cfg(test)] - pub async fn next_group(&mut self) -> Result> { - kio::wait(|waiter| self.poll_next_group(waiter)).await - } - - /// Poll for the next datagram, from the newest segment only (datagrams are a - /// live best-effort channel; there is nothing to resume from older segments). - pub fn poll_recv_datagram(&mut self, waiter: &kio::Waiter) -> Poll>> { - self.poll_sync(waiter); - - // Drive the newest segment's activation too: a subscriber polling only - // datagrams must still resolve the subscription (registering demand) and - // be woken when it activates. - let mut pending_activation = false; - if let Some(seg) = self.segments.last_mut() { - if Self::poll_activate(seg, &self.last_prefs, self.min_sequence, waiter).is_pending() { - pending_activation = true; - } else if let SubState::Active(sub) = &mut seg.sub - && let Ok(Some(datagram)) = ready!(sub.poll_recv_datagram(waiter)) - { - return Poll::Ready(Ok(Some(datagram))); - } - } - - if let Some(err) = &self.abort { - return Poll::Ready(Err(err.clone())); - } - // The producer finished, so this channel ends now. A final segment that - // already died is that death: the datagram read above drops a Ready(Err) - // on the floor, and a clean Ok(None) here would hide it. A segment that - // has not activated yet has no end to report. - if self.finished { - if pending_activation { - return Poll::Ready(Ok(None)); - } - return match ready!(self.poll_final(waiter)) { - Some(end) => Poll::Ready(end.map(|_| None)), - None => Poll::Ready(Ok(None)), - }; - } - // The producer is gone: no takeover is coming, so the track ends when - // its newest segment does, and the way it did. - if self.closed && !pending_activation { - return match ready!(self.poll_final(waiter)) { - Some(end) => Poll::Ready(end.map(|_| None)), - None => Poll::Ready(Err(Error::Dropped)), - }; - } - Poll::Pending - } - - /// Block until the logical track ends: `Ok` after a clean finish, `Err` after - /// an abort. Readers use `finished()`; this just discards the group count. - #[cfg(test)] - pub async fn closed(&mut self) -> Result<()> { - kio::wait(|waiter| self.poll_finished(waiter)).await.map(|_| ()) - } - - /// Poll for the logical track finishing, returning the final segment's group - /// count (one past its last sequence). - pub fn poll_finished(&mut self, waiter: &kio::Waiter) -> Poll> { - self.poll_end(waiter) - .map(|res| res.and_then(|end| end.ok_or(Error::Closed))) - } - - pub(crate) fn poll_end(&mut self, waiter: &kio::Waiter) -> Poll>> { - self.poll_sync(waiter); - - if let Some(err) = &self.abort { - return Poll::Ready(Err(err.clone())); - } - if !self.finished && !self.closed { - return Poll::Pending; - } - // The track ends as its final segment did, or with nothing spliced, cleanly - // only if it finished. - match ready!(self.poll_final(waiter)) { - Some(end) => Poll::Ready(end), - None if self.finished => Poll::Ready(Ok(Some(0))), - None => Poll::Ready(Err(Error::Dropped)), - } - } - - /// Poll for where the source of the first live segment reaching this cursor's floor - /// starts, raised to the floor, once resolved; see [`track::Subscriber::poll_start`]. - /// A segment ending at or below the floor, or one that already ended, serves nothing - /// more here, so its source is not waited on. Like [`Self::poll_final`], this only - /// resolves the subscription and consumes no groups. - pub(crate) fn poll_start(&mut self, waiter: &kio::Waiter) -> Poll> { - self.poll_sync(waiter); - let floor = self.min_sequence; - for seg in &mut self.segments { - if seg.last_group().is_some_and(|last| last <= floor) { - continue; - } - ready!(Self::poll_activate(seg, &self.last_prefs, floor, waiter)); - if let SubState::Active(sub) = &mut seg.sub { - return Poll::Ready(ready!(sub.poll_start(waiter)).map(|start| start.max(floor))); + events.push(*event); + let mut next = remaining.to_vec(); + next.remove(i); + walk(events, &next); + events.pop(); } } - Poll::Ready(None) - } - - /// Wait for the final segment's track to end: its group count when it - /// finished, its error when it died, and `None` when there is no segment. Earlier segments don't - /// decide the end. Only the subscription is resolved here: consuming groups, or - /// completing the segment, would steal them from a `recv_group` caller on the - /// same subscriber. - fn poll_final(&mut self, waiter: &kio::Waiter) -> Poll>>> { - let Some(seg) = self.segments.last_mut() else { - return Poll::Ready(None); - }; - ready!(Self::poll_activate(seg, &self.last_prefs, self.min_sequence, waiter)); - match &mut seg.sub { - SubState::Done(end) => Poll::Ready(Some(end.clone())), - // Observe only: the cursor may still hold groups, so the read path - // completes the segment once it drains. - SubState::Active(sub) => Poll::Ready(Some(ready!(sub.poll_end(waiter)))), - SubState::Pending(_) => unreachable!("poll_activate resolved above"), - } - } - - /// How a logical track ends once every segment drained and no switch can follow - /// (the producer finished or went away): as its final segment's track ended, since - /// that is where the content ended. A finished track with no segment ends cleanly; - /// an orphaned one with [`Error::Dropped`]. - fn final_end(&self) -> Result<()> { - match self.segments.last().map(|seg| &seg.sub) { - Some(SubState::Done(end)) => end.clone().map(|_| ()), - None if self.finished => Ok(()), - _ => Err(Error::Dropped), - } - } - - /// Block until the logical track is finished, returning the final group count. - #[cfg(test)] - pub async fn finished(&mut self) -> Result { - kio::wait(|waiter| self.poll_finished(waiter)).await - } - - /// Start the subscriber at the specified sequence. - pub fn start_at(&mut self, sequence: u64) { - self.min_sequence = sequence; - for seg in &mut self.segments { - let floor = seg.first_group().max(sequence); - if let SubState::Active(sub) = &mut seg.sub { - sub.start_at(floor); - } - } - } - - /// Raise the floor to `sequence`, keeping any higher floor already set. See - /// [`track::Subscriber::raise_start_to`]. - pub(crate) fn raise_start_to(&mut self, sequence: u64) { - self.min_sequence = self.min_sequence.max(sequence); - for seg in &mut self.segments { - let floor = seg.first_group().max(sequence); - if let SubState::Active(sub) = &mut seg.sub { - sub.raise_start_to(floor); - } - } - } - - /// Cap the subscriber at `end`, or remove the cap with `..`. - /// - /// Enforced on this subscriber's reads (see [`Self::poll_recv_group`]), never - /// on the inner segment cursors, so a capped group parks here and a rising cap - /// re-offers it. - pub fn end_at(&mut self, end: impl Into) { - self.end_sequence = end.into().exclusive(); - // The cap bounds each segment's drift anchor as well as this reader's own - // delivery: a segment must not measure against groups this cap hides. - self.refresh_anchor(); - } - - /// The shared preferences channel, so `track::Control` can wrap it. - pub(crate) fn prefs(&self) -> kio::Producer { - self.prefs.clone() - } - - /// Take the groups every segment's drift budget skipped since the last call. - /// - /// The segments are subscribed untagged, so their skips have nowhere to be counted - /// until they reach the [`track::Subscriber`] that owns the stats scope. A retired - /// segment's tail is lost with it, which is the same bound its groups already had. - pub(crate) fn take_stale(&mut self) -> crate::stats::Content { - let mut stale = crate::stats::Content::default(); - for seg in &mut self.segments { - if let Some(sub) = seg.stale_sub_mut() { - stale.add(sub.take_stale()); - } - } - stale - } - - /// Replace this subscriber's preferences; each segment's demand is re-derived - /// on the next poll. - pub fn update(&mut self, subscription: Subscription) { - if let Ok(mut prefs) = self.prefs.write() { - *prefs = subscription; - } - } - - /// The latest group sequence across the segments, clamped to their bounds. - pub fn latest(&self) -> Option { - self.state.read().latest() - } - - /// Whether `other` reads the same logical track. - pub fn is_clone(&self, other: &Self) -> bool { - self.state.same_channel(&other.state) - } -} - -#[cfg(test)] -mod test { - use super::*; - use crate::broadcast; - use futures::FutureExt; - use std::sync::Arc; - use std::time::Duration; - - fn track_pair(name: &str) -> (track::Producer, track::Consumer) { - let producer = track::Producer::new(Arc::new(broadcast::Info::default()), name, None); - let consumer = producer.consume(); - (producer, consumer) - } - - /// [`track_pair`] with explicit publisher properties, for tests that care about the - /// retention window a subscriber's drift budget is clamped to. - fn track_pair_with(name: &str, info: track::Info) -> (track::Producer, track::Consumer) { - let producer = track::Producer::new(Arc::new(broadcast::Info::default()), name, info); - let consumer = producer.consume(); - (producer, consumer) - } - - /// A bounded replay window for tests whose subject requires every buffered group. - fn replay() -> Subscription { - Subscription::default().with_max_age(std::time::Duration::from_secs(30)) - } - - fn write_group(producer: &mut track::Producer, sequence: u64, payload: &str) { - let mut group = producer.create_group(group::Info { sequence }).unwrap(); - group.write_frame(Timestamp::ZERO, payload.as_bytes().to_vec()).unwrap(); - group.finish().unwrap(); - } - - /// Like [`write_group`], but placing the group on a real media timeline so the - /// drift budget has something to measure. Most tests here are about splicing - /// mechanics and use [`write_group`], paired with [`replay`] when they need every - /// buffered group rather than the real-time live edge. - fn write_group_at(producer: &mut track::Producer, sequence: u64, payload: &str, at: Duration) { - let mut group = producer.create_group(group::Info { sequence }).unwrap(); - group - .write_frame(at.try_into().unwrap(), payload.as_bytes().to_vec()) - .unwrap(); - group.finish().unwrap(); - } - - fn recv(sub: &mut Subscriber) -> u64 { - sub.recv_group() - .now_or_never() - .expect("should not block") - .expect("should not error") - .expect("should not be finished") - .sequence - } - - fn read(group: &mut group::Consumer) -> Vec { - group - .read_frame() - .now_or_never() - .expect("should not block") - .expect("should not error") - .expect("should not be finished") - .payload - .to_vec() - } - - fn recv_pending(sub: &mut Subscriber) { - assert!(sub.recv_group().now_or_never().is_none(), "should have blocked"); - } - - /// A waker that counts its wakes, for asserting a pending poll left a live - /// registration behind. - struct CountWaker(std::sync::atomic::AtomicUsize); - - impl std::task::Wake for CountWaker { - fn wake(self: Arc) { - self.0.fetch_add(1, std::sync::atomic::Ordering::SeqCst); - } - } - - impl CountWaker { - fn new() -> (Arc, std::task::Waker) { - let counter = Arc::new(Self(std::sync::atomic::AtomicUsize::new(0))); - (counter.clone(), std::task::Waker::from(counter)) - } - - fn count(&self) -> usize { - self.0.load(std::sync::atomic::Ordering::SeqCst) - } - } - - #[tokio::test] - async fn switch_splices_groups() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - - let mut sub = producer.consume().subscribe(replay()); - - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - assert_eq!(recv(&mut sub), 0); - assert_eq!(recv(&mut sub), 1); - - // Switch to B at group 2. A racing past its cap is filtered. - producer.switch(&consumer_b, Position::group(2)).unwrap(); - write_group(&mut track_a, 2, "a2-over-cap"); - write_group(&mut track_b, 2, "b2"); - write_group(&mut track_b, 3, "b3"); - - assert_eq!(recv(&mut sub), 2); - assert_eq!(recv(&mut sub), 3); - recv_pending(&mut sub); - } - - /// A segment that ends at or below the cursor's floor serves it nothing, so its - /// source's unresolved start must not hold up the start of the segment that does. - #[tokio::test] - async fn poll_start_skips_a_segment_below_the_floor() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - track_a.request_start(Some(0)).unwrap(); - track_b.request_start(Some(5)).unwrap(); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - producer.switch(&consumer_b, Position::group(5)).unwrap(); - - let mut sub = producer.consume().subscribe(replay()); - sub.start_at(10); - assert!( - kio::wait(|waiter| sub.poll_start(waiter)).now_or_never().is_none(), - "B's source has not resolved yet" - ); - - // A's source never resolves: it serves nothing at or above the floor. - track_b.start_at(12).unwrap(); - let start = kio::wait(|waiter| sub.poll_start(waiter)).now_or_never(); - assert_eq!(start, Some(Some(12)), "waited on a segment below the floor"); - } - - /// A segment that already ended serves nothing more, so the start comes from the next - /// one rather than falling back to whichever group arrives first. - #[tokio::test] - async fn poll_start_skips_an_ended_segment() { - let (track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - track_b.request_start(Some(5)).unwrap(); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - producer.switch(&consumer_b, Position::group(5)).unwrap(); - track_a.abort(Error::Cancel).unwrap(); - - let mut sub = producer.consume().subscribe(replay()); - // Reading drains A to its end, leaving nothing for it to serve. - recv_pending(&mut sub); - assert!( - kio::wait(|waiter| sub.poll_start(waiter)).now_or_never().is_none(), - "B's source has not resolved yet" - ); - - track_b.start_at(5).unwrap(); - let start = kio::wait(|waiter| sub.poll_start(waiter)).now_or_never(); - assert_eq!(start, Some(Some(5))); - } - - #[tokio::test] - async fn demand_reflects_boundaries() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - - let mut sub = producer - .consume() - .subscribe(Subscription::default().with_start(Position::group(0))); - // Poll once so the subscriber registers on segment A. - recv_pending(&mut sub); - assert_eq!(track_a.subscription().unwrap().end, None); - - producer.switch(&consumer_b, Position::group(5)).unwrap(); - recv_pending(&mut sub); - - // The old session sees its demand capped; the new one starts at the boundary. - assert_eq!(track_a.subscription().unwrap().end, Some(Position::group(5))); - assert_eq!(track_b.subscription().unwrap().start, Some(Position::group(5))); - } - - #[tokio::test] - async fn update_reslices_demand() { - let (track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - - let mut sub = producer.consume().subscribe(None); - recv_pending(&mut sub); - assert_eq!(track_a.subscription().unwrap().priority, 0); - - sub.update(Subscription::default().with_priority(7)); - recv_pending(&mut sub); - assert_eq!(track_a.subscription().unwrap().priority, 7); - } - - #[tokio::test] - async fn dead_segment_stalls_until_switch() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - - // The route dies: the subscriber stalls, it does not error. - track_a.abort(Error::Dropped).unwrap(); - recv_pending(&mut sub); - - // A replacement resumes exactly where the old route left off. - producer.switch(&consumer_b, Position::group(1)).unwrap(); - write_group(&mut track_b, 1, "b1"); - assert_eq!(recv(&mut sub), 1); - } - - #[tokio::test] - async fn takeover_computes_boundary() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - - // No segments yet: the takeover is unbounded. - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - assert_eq!(recv(&mut sub), 0); - assert_eq!(recv(&mut sub), 1); - - // Groups exist: the takeover resumes one past the newest, even when the old - // route's cache died with it (a group mid-transfer is lost like any loss, - // never re-delivered live to subscribers that may already have it). - track_a.abort(Error::Dropped).unwrap(); - producer.takeover(&consumer_b).unwrap(); - write_group(&mut track_b, 2, "b2"); - assert_eq!(recv(&mut sub), 2); - } - - /// A route that stalls while a replacement runs on leaves the boundary trailing, - /// so the takeover asks the new route to replay a backlog. That backlog is bounded - /// by the drift budget and nothing else: a REAL_TIME subscriber jumps to the live - /// edge, while one that declared a budget covering the gap reads it whole. - #[tokio::test] - async fn a_takeover_backlog_is_bounded_by_the_budget() { - // Both routes retain a minute, so the budget is the only thing bounding the - // backlog (a budget past the publisher's window is clamped to it). - let retain = track::Info::default().with_max_age(std::time::Duration::from_secs(60)); - let (mut track_a, consumer_a) = track_pair_with("a", retain.clone()); - let (mut track_b, consumer_b) = track_pair_with("b", retain); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut live = producer.consume().subscribe(None); - let mut patient = producer - .consume() - .subscribe(Subscription::default().with_max_age(std::time::Duration::from_secs(60))); - - write_group_at(&mut track_a, 0, "a0", std::time::Duration::ZERO); - assert_eq!(recv(&mut live), 0); - assert_eq!(recv(&mut patient), 0); - - // The old route stalls; the replacement resumes at the boundary and replays - // half a minute of media as fast as the wire allows. - track_a.abort(Error::Dropped).unwrap(); - producer.takeover(&consumer_b).unwrap(); - recv_pending(&mut live); - for second in 1..=30 { - write_group_at(&mut track_b, second, "b", std::time::Duration::from_secs(second)); - } - - // The live subscriber takes the edge and writes the backlog off; the patient - // one asked to tolerate it and gets every group. - assert_eq!(recv(&mut live), 30); - recv_pending(&mut live); - - let mut backfill = Vec::new(); - while let Some(Ok(Some(group))) = patient.recv_group().now_or_never() { - backfill.push(group.sequence); - } - assert_eq!(backfill, (1..=30).collect::>()); - } - - /// A capped spliced subscriber measures drift against its cap, exactly like a plain - /// one. The segment is deliberately left uncapped so its completion stays visible, so - /// the cap has to reach its drift anchor by another route or the groups the reader - /// still wants read as ancient. - #[tokio::test] - async fn a_capped_spliced_subscriber_measures_drift_against_its_cap() { - let retain = track::Info::default().with_max_age(std::time::Duration::from_secs(60)); - let (mut track_a, consumer_a) = track_pair_with("a", retain); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - sub.end_at(..1); - - write_group_at(&mut track_a, 0, "a0", std::time::Duration::ZERO); - write_group_at(&mut track_a, 1, "a1", std::time::Duration::from_secs(30)); - - // Group 1 is past the cap, so it is not a live edge this reader can jump to. - assert_eq!(recv(&mut sub), 0); - } - - /// A group parked above the cap is re-checked when the cap rises: the live edge moved - /// while it waited, so handing it back unconditionally would deliver a backlog the - /// budget has already given up on. - #[tokio::test] - async fn a_raised_cap_rechecks_parked_groups() { - let retain = track::Info::default().with_max_age(std::time::Duration::from_secs(60)); - let (mut track_a, consumer_a) = track_pair_with("a", retain); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - sub.end_at(..1); - - write_group_at(&mut track_a, 0, "a0", std::time::Duration::ZERO); - assert_eq!(recv(&mut sub), 0); - - // Group 1 parks above the cap; group 2 lands half a minute further on. - write_group_at(&mut track_a, 1, "a1", std::time::Duration::from_secs(1)); - recv_pending(&mut sub); - write_group_at(&mut track_a, 2, "a2", std::time::Duration::from_secs(30)); - - // Raising the cap owes the reader group 1 again, but by now it is a backlog. - sub.end_at(..); - assert_eq!(recv(&mut sub), 2); - recv_pending(&mut sub); - } - - /// Completing the segment must not discard the cursor that evaluates parked - /// groups. A later cap increase still uses the terminal track's live edge. - #[tokio::test] - async fn a_raised_cap_rechecks_parked_groups_after_segment_finish() { - let retain = track::Info::default().with_max_age(std::time::Duration::from_secs(60)); - let (mut track_a, consumer_a) = track_pair_with("a", retain); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - sub.end_at(..1); - - write_group_at(&mut track_a, 0, "a0", std::time::Duration::ZERO); - assert_eq!(recv(&mut sub), 0); - - write_group_at(&mut track_a, 1, "a1", std::time::Duration::from_secs(1)); - write_group_at(&mut track_a, 2, "a2", std::time::Duration::from_secs(30)); - track_a.finish().unwrap(); - // Drive the inner cursor through its clean end while both newer groups are - // parked above the cap. - recv_pending(&mut sub); - - sub.end_at(..); - assert_eq!(recv(&mut sub), 2, "the stale parked group is skipped after finish"); - recv_pending(&mut sub); - } - - #[tokio::test] - async fn takeover_replaces_empty_segment() { - let (track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - recv_pending(&mut sub); - - // A never produced anything, so B replaces it outright and group 0 is - // still reachable. - drop(track_a); - producer.takeover(&consumer_b).unwrap(); - write_group(&mut track_b, 0, "b0"); - assert_eq!(recv(&mut sub), 0); - } - - #[tokio::test] - async fn finish_ends_after_final_segment() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - - // Finishing the logical track alone isn't the end; the segment must drain. - producer.finish().unwrap(); - recv_pending(&mut sub); - - track_a.finish().unwrap(); - assert!( - sub.recv_group() - .now_or_never() - .expect("should not block") - .expect("should not error") - .is_none(), - "should be finished" - ); - assert_eq!(sub.finished().now_or_never().unwrap().unwrap(), 1); - assert!(sub.closed().now_or_never().unwrap().is_ok()); - } - - #[tokio::test] - async fn next_group_across_segments() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - - write_group(&mut track_a, 0, "a0"); - producer.switch(&consumer_b, Position::group(1)).unwrap(); - write_group(&mut track_b, 1, "b1"); - - let mut read = |expected: &[u8]| { - let mut group = sub.next_group().now_or_never().unwrap().unwrap().unwrap(); - let frame = group.read_frame().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(&frame.payload[..], expected); - }; - read(b"a0"); - read(b"b1"); - } - - #[tokio::test] - async fn info_from_first_segment() { - let (_track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - let consumer = producer.consume(); - - // No segments: info is parked. - assert!(consumer.info().now_or_never().is_none()); - - producer.switch(&consumer_a, None).unwrap(); - let info = consumer.info().now_or_never().unwrap().unwrap(); - assert_eq!(info.timescale, crate::Timescale::default()); - } - - #[tokio::test] - async fn fetch_uses_an_older_complete_cache_before_the_newest_segment() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - producer.switch(&consumer_b, Position::group(10)).unwrap(); - - // The older segment owns this range and still has a complete cached copy. - // Prefer it over asking the newest live route, which may have already - // evicted the group upstream. - write_group(&mut track_a, 3, "a3"); - let consumer = producer.consume(); - let mut group = consumer - .fetch_group(3, None) - .now_or_never() - .expect("cached fetch should resolve") - .unwrap(); - assert_eq!(group.sequence, 3); - assert_eq!(read(&mut group), b"a3"); - - // A cached group on the newest segment still resolves immediately, even - // below its subscribe boundary: bounds slice demand, not access. - write_group(&mut track_b, 4, "b4"); - let mut group = consumer - .fetch_group(4, None) - .now_or_never() - .expect("newest cached fetch should resolve") - .unwrap(); - assert_eq!(read(&mut group), b"b4"); - } - - /// A cached copy is handed back sitting at the requested frame, the same as a fetch - /// the segment's own track answers. - #[tokio::test] - async fn a_cached_fetch_starts_at_the_requested_frame() { - let (track_a, consumer_a) = track_pair("a"); - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - for payload in ["f0", "f1", "f2"] { - group - .write_frame(crate::Timestamp::ZERO, payload.as_bytes().to_vec()) - .unwrap(); - } - group.finish().unwrap(); - - let mut group = producer - .consume() - .fetch_group(0, group::Fetch::default().with_frame_start(1)) - .now_or_never() - .expect("cached fetch should resolve") - .unwrap(); - assert_eq!(group.index(), 1); - let frame = group.read_frame().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(frame.payload.as_ref(), b"f1"); - } - - #[tokio::test] - async fn fetch_waits_for_first_segment() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - let consumer = producer.consume(); - - // No segment yet: the fetch parks instead of failing (a route may serve the - // track any moment). - let fetch = consumer.fetch_group(0, None); - let mut fetch = std::pin::pin!(fetch); - assert!(futures::poll!(fetch.as_mut()).is_pending(), "fetch should wait"); - - // The first segment arrives with the group cached: the fetch resolves. - write_group(&mut track_a, 0, "a0"); - producer.switch(&consumer_a, None).unwrap(); - let group = fetch.await.expect("fetch should resolve"); - assert_eq!(group.sequence, 0); - } - - #[tokio::test] - async fn pending_fetch_follows_a_newer_segment() { - let (track_a, consumer_a) = track_pair("a"); - let dynamic_a = track_a.dynamic(); - let (track_b, consumer_b) = track_pair("b"); - let dynamic_b = track_b.dynamic(); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let consumer = producer.consume(); - let fetch = tokio::spawn(async move { consumer.fetch_group(0, None).await }); - - let _request_a = tokio::time::timeout(Duration::from_secs(1), dynamic_a.requested_group()) - .await - .expect("first segment should receive the fetch") - .unwrap(); - - producer.takeover(&consumer_b).unwrap(); - let request_b = tokio::time::timeout(Duration::from_secs(1), dynamic_b.requested_group()) - .await - .expect("pending fetch should follow the takeover") - .unwrap(); - let mut served = request_b.accept(None).unwrap(); - served.write_frame(Timestamp::ZERO, b"b0".to_vec()).unwrap(); - served.finish().unwrap(); - - let mut group = tokio::time::timeout(Duration::from_secs(1), fetch) - .await - .expect("replacement should answer the fetch") - .unwrap() - .unwrap(); - assert_eq!(read(&mut group), b"b0"); - } - - #[test] - fn warm_snapshot_skips_only_the_frame_seam() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - write_group(&mut track_a, 0, "a0"); - let mut seam = track_a.create_group(group::Info { sequence: 1 }).unwrap(); - seam.write_frame(Timestamp::ZERO, b"a1".to_vec()).unwrap(); - - producer.takeover(&consumer_b).unwrap(); - let mut seam = track_b.create_group(group::Info { sequence: 1 }).unwrap(); - seam.start_at(1).unwrap(); - seam.write_frame(Timestamp::ZERO, b"b1".to_vec()).unwrap(); - seam.finish().unwrap(); - write_group(&mut track_b, 2, "b2"); - - let cached = producer.consume().cached_groups(); - assert_eq!( - cached.iter().map(|(group, _)| group.sequence).collect::>(), - vec![0, 2], - "only the group split across both segments is unsafe to snapshot" - ); - } - - #[tokio::test] - async fn takeover_survives_dead_empty_segment() { - let (mut track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - let (mut track_c, consumer_c) = track_pair("c"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - - // A dies; B takes over at the boundary but dies before producing. - track_a.abort(Error::Dropped).unwrap(); - producer.takeover(&consumer_b).unwrap(); - drop(track_b); - - // C replaces B's empty segment instead of failing forever on the - // unadvanceable boundary. - producer.takeover(&consumer_c).unwrap(); - write_group(&mut track_c, 1, "c1"); - assert_eq!(recv(&mut sub), 1); - } - - #[tokio::test] - async fn finished_does_not_consume_groups() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - write_group(&mut track_a, 0, "a0"); - producer.finish().unwrap(); - - // Waiting for the end must not steal the buffered group from recv. - assert!(sub.finished().now_or_never().is_none(), "final segment still open"); - assert_eq!(recv(&mut sub), 0); - - track_a.finish().unwrap(); - assert_eq!(sub.finished().now_or_never().unwrap().unwrap(), 1); - } - - #[tokio::test] - async fn datagram_only_subscriber_activates() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // Polling only datagrams must still resolve the subscription. - assert!( - kio::wait(|waiter| sub.poll_recv_datagram(waiter)) - .now_or_never() - .is_none(), - "no datagram yet" - ); - track_a.append_datagram(Timestamp::ZERO, b"d0".as_ref()).unwrap(); - let datagram = kio::wait(|waiter| sub.poll_recv_datagram(waiter)) - .now_or_never() - .expect("datagram should be ready") - .expect("should not error") - .expect("track should not be finished"); - assert_eq!(&datagram.payload[..], b"d0"); - } - - #[tokio::test] - async fn end_at_parks_at_cap() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - - // The cap parks the subscriber; the group beyond it is held, not dropped. - sub.end_at(..1); - assert_eq!(recv(&mut sub), 0); - recv_pending(&mut sub); - - // Raising the cap re-offers the parked group. - sub.end_at(..2); - assert_eq!(recv(&mut sub), 1); - } - - /// A park that replaces a reader's segments with a cache of what they delivered - /// hands that reader nothing twice, however many times one open group (a catalog - /// taking deltas) changes route. A newcomer still gets the whole group, head first, - /// after the takeovers have pruned every segment that started at its head. - #[tokio::test] - async fn repeated_parks_mid_group_hand_the_head_out_once() { - let mut producer = Producer::new(); - let (track, consumer) = track_pair("a"); - producer.takeover(&consumer).unwrap(); - let mut sub = producer.consume().subscribe(None); - let mut group = track.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"0".to_vec()).unwrap(); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"0"); - - let mut routes = vec![(track, consumer, group)]; - let mut heads = Vec::new(); - let rounds = MAX_SEGMENTS as u64 + 1; - for frame in 1..=rounds { - // The cache holds the whole group so far, the way the origin rebuilds it. - let (cache, cache_consumer) = track_pair("cache"); - let mut whole = cache.create_group(group::Info { sequence: 0 }).unwrap(); - for f in 0..frame { - whole.write_frame(Timestamp::ZERO, f.to_string().into_bytes()).unwrap(); - } - producer.park(&cache_consumer).unwrap(); - - let (next, next_consumer) = track_pair("next"); - producer.takeover(&next_consumer).unwrap(); - let mut continuation = next.create_group(group::Info { sequence: 0 }).unwrap(); - continuation.start_at(frame).unwrap(); - continuation - .write_frame(Timestamp::ZERO, frame.to_string().into_bytes()) - .unwrap(); - - assert_eq!(read(&mut reading), frame.to_string().as_bytes()); - recv_pending(&mut sub); - routes.push((next, next_consumer, continuation)); - // The origin finishes a cache once the takeover lands, but holds its open - // group: dropping that would clear its frames. - cache.finish().unwrap(); - heads.push(whole); - } - - let mut late = producer.consume().subscribe(None); - let mut group = late.recv_group().now_or_never().unwrap().unwrap().unwrap(); - for frame in 0..=rounds { - assert_eq!(read(&mut group), frame.to_string().as_bytes()); - } - } - - /// A group held back at the reader's cap when a park replaces its segments comes - /// from the cache once the cap rises, exactly once. - #[tokio::test] - async fn a_park_reoffers_a_group_held_at_the_cap_once() { - let (mut track_a, consumer_a) = track_pair("a"); - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - sub.end_at(..1); - write_group(&mut track_a, 0, "0"); - write_group(&mut track_a, 1, "1"); - assert_eq!(recv(&mut sub), 0); - recv_pending(&mut sub); - - let (mut cache, cache_consumer) = track_pair("cache"); - write_group(&mut cache, 0, "0"); - write_group(&mut cache, 1, "1"); - producer.park(&cache_consumer).unwrap(); - let (mut track_b, consumer_b) = track_pair("b"); - producer.takeover(&consumer_b).unwrap(); - write_group(&mut track_b, 2, "2"); - - sub.end_at(..3); - assert_eq!(recv(&mut sub), 1); - assert_eq!(recv(&mut sub), 2); - recv_pending(&mut sub); - } - - /// A parked beyond-cap group must not block in-range groups that arrive - /// behind it: a relay can ingest a burst micro-reordered (newest first). - #[tokio::test] - async fn end_at_reoffers_reordered_arrivals() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - - sub.end_at(..2); - - // Reordered burst: the beyond-cap group arrives first. - write_group(&mut track_a, 2, "a2"); - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - - // The capped group parks without blocking the in-range late arrivals. - assert_eq!(recv(&mut sub), 0); - assert_eq!(recv(&mut sub), 1); - recv_pending(&mut sub); - - // Raising the cap re-offers the parked group. - sub.end_at(..3); - assert_eq!(recv(&mut sub), 2); - } - - /// A parked group the producer aborts (eviction/expiry) is dropped rather - /// than re-offered when the cap rises. - #[tokio::test] - async fn evicted_parked_groups_are_dropped() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - sub.end_at(..1); - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - - let straggler = track_a.create_group(group::Info { sequence: 1 }).unwrap(); - recv_pending(&mut sub); - straggler.abort(Error::Old).unwrap(); - - sub.end_at(..); - write_group(&mut track_a, 2, "a2"); - assert_eq!(recv(&mut sub), 2, "the evicted parked group is dropped, not re-offered"); - } - - #[tokio::test] - async fn next_group_skips_boundary_duplicate() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - - let next = |sub: &mut Subscriber| { - kio::wait(|waiter| sub.poll_next_group(waiter)) - .now_or_never() - .expect("should not block") - .expect("should not error") - .expect("should not be finished") - .sequence - }; - - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - assert_eq!(next(&mut sub), 0); - assert_eq!(next(&mut sub), 1); - - // A boundary at the delivered edge: B re-serves group 1, which was already - // returned and must not be delivered twice. - producer.switch(&consumer_b, Position::group(1)).unwrap(); - write_group(&mut track_b, 1, "b1"); - write_group(&mut track_b, 2, "b2"); - assert_eq!(next(&mut sub), 2); - } - - /// A beyond-cap group arriving (and being polled) before an in-range one must not - /// advance the sequence cursor past the late arrival: the inner cursor is capped, - /// so the reordered burst is delivered in order once the cap admits each group. - #[tokio::test] - async fn next_group_cap_holds_a_reordered_group_without_losing_late_arrivals() { - let (mut track_a, consumer_a) = track_pair("a"); - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - - let next = |sub: &mut Subscriber| { - kio::wait(|waiter| sub.poll_next_group(waiter)) - .now_or_never() - .expect("should not block") - .expect("should not error") - .expect("should not be finished") - .sequence - }; - let next_pending = |sub: &mut Subscriber| { - assert!( - kio::wait(|waiter| sub.poll_next_group(waiter)).now_or_never().is_none(), - "should have blocked" - ); - }; - - sub.end_at(..2); - - // The beyond-cap group arrives first and is polled: it must hold, not advance - // the cursor past the in-range groups still on their way. - write_group(&mut track_a, 2, "a2"); - next_pending(&mut sub); - - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - assert_eq!(next(&mut sub), 0); - assert_eq!(next(&mut sub), 1); - next_pending(&mut sub); - - // Raising the cap admits the held group. - sub.end_at(..3); - assert_eq!(next(&mut sub), 2); - } - - /// Lowering the cap after the cursor already considered (but did not deliver) a - /// higher group must not strand a late in-range arrival: seeking consumes - /// nothing, so no segment cursor ever steps past an undelivered sequence. - #[tokio::test] - async fn next_group_cap_lowering_after_a_lookahead_keeps_late_arrivals() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - producer.switch(&consumer_b, Position::group(1)).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - - let next = |sub: &mut Subscriber| { - kio::wait(|waiter| sub.poll_next_group(waiter)) - .now_or_never() - .expect("should not block") - .expect("should not error") - .expect("should not be finished") - .sequence - }; - let next_pending = |sub: &mut Subscriber| { - assert!( - kio::wait(|waiter| sub.poll_next_group(waiter)).now_or_never().is_none(), - "should have blocked" - ); - }; - - // The new segment's group 2 arrives early, and the old segment delivers 0. - write_group(&mut track_b, 2, "b2"); - write_group(&mut track_a, 0, "a0"); - assert_eq!(next(&mut sub), 0); - - // Cap below the group the cursor could already see. It must hold, not have - // committed the segment past the still-missing group 1. - sub.end_at(..2); - next_pending(&mut sub); - - write_group(&mut track_b, 1, "b1"); - assert_eq!(next(&mut sub), 1); - next_pending(&mut sub); - - sub.end_at(..); - assert_eq!(next(&mut sub), 2); - } - - /// The spliced sequence path inherits the [`track::Ordered`] contract: the drift - /// budget applies as it reads, and a backlog inside the budget still crosses a - /// takeover boundary whole, exactly as a plain track's sequence cursor delivers it. - #[tokio::test] - async fn next_group_sheds_a_stale_backlog_like_a_plain_track() { - // Groups spaced 10s apart on the media timeline, so with the default (zero) - // max age budget every group behind the newest one's reach is stale. - let stamp = |sequence: u64| Duration::from_secs(10 * sequence); - - // Plain baseline: the same content through `track::Ordered`. - let (mut plain, _plain_consumer) = track_pair("plain"); - for sequence in 0..4 { - write_group_at(&mut plain, sequence, "data", stamp(sequence)); - } - let mut plain_ordered = plain.subscribe(None).ordered(); - let baseline: Vec = std::iter::from_fn(|| { - plain_ordered - .next_group() - .now_or_never()? - .expect("should not error") - .map(|group| group.sequence) - }) - .collect(); - assert_eq!(baseline, vec![3], "a plain sequence cursor sheds the backlog"); - - // The arrival path on the same content lands in the same place. - let mut arrival = plain.subscribe(None); - assert_eq!( - arrival.recv_group().now_or_never().unwrap().unwrap().unwrap().sequence, - 3, - "the arrival path skips straight to the live edge" - ); - - // Spliced: the same backlog split across a takeover boundary. Retained long - // enough that the replay budget below is not clamped to the default window. - let retain = track::Info::default().with_max_age(Duration::from_secs(60)); - let (mut track_a, consumer_a) = track_pair_with("a", retain.clone()); - let (mut track_b, consumer_b) = track_pair_with("b", retain); - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - producer.switch(&consumer_b, Position::group(2)).unwrap(); - let mut sub = producer.consume().subscribe(None); - - write_group_at(&mut track_a, 0, "a0", stamp(0)); - write_group_at(&mut track_a, 1, "a1", stamp(1)); - write_group_at(&mut track_b, 2, "b2", stamp(2)); - write_group_at(&mut track_b, 3, "b3", stamp(3)); - - let spliced: Vec = std::iter::from_fn(|| { - kio::wait(|waiter| sub.poll_next_group(waiter)) - .now_or_never()? - .expect("should not error") - .map(|group| group.sequence) - }) - .collect(); - - // Segment A's groups are judged as the plain cursor judges them: against the - // logical edge (group 3), with group 1's reach bounded by its successor in - // segment B. The sequence path inherits that from the arrival path rather than - // inventing its own anchor, so the two agree. - assert_eq!(spliced, baseline, "a splice sheds the backlog like one track"); - - let mut arrival = producer.consume().subscribe(None); - let arrival: Vec = std::iter::from_fn(|| { - kio::wait(|waiter| arrival.poll_recv_group(waiter)) - .now_or_never()? - .expect("should not error") - .map(|group| group.sequence) - }) - .collect(); - assert_eq!(arrival, spliced, "both spliced cursors shed the same backlog"); - - // A budget spanning the history bursts it in full across the boundary. - let mut replay = producer - .consume() - .subscribe(Subscription::default().with_max_age(Duration::from_secs(60))); - let replayed: Vec = std::iter::from_fn(|| { - kio::wait(|waiter| replay.poll_next_group(waiter)) - .now_or_never()? - .expect("should not error") - .map(|group| group.sequence) - }) - .collect(); - assert_eq!(replayed, vec![0, 1, 2, 3], "a backlog inside the budget crosses whole"); - } - - #[tokio::test] - async fn unstamped_successor_segment_keeps_the_previous_group_unbounded() { - let (mut a, a_read) = track_pair("a"); - let (b, b_read) = track_pair("b"); - let (mut c, c_read) = track_pair("c"); - write_group_at(&mut a, 0, "a0", Duration::ZERO); - let _unstamped = b.create_group(1u64.into()).unwrap(); - write_group_at(&mut c, 2, "c2", Duration::from_secs(10)); - write_group_at(&mut c, 3, "c3", Duration::from_secs(20)); - let mut producer = Producer::new(); - producer.switch(a_read, None).unwrap(); - producer.switch(b_read, Position::group(1)).unwrap(); - producer.switch(c_read, Position::group(2)).unwrap(); - let mut sub = producer.consume().subscribe(None); - assert_eq!( - recv(&mut sub), - 0, - "the unstamped successor must not borrow segment C's start" - ); - } - - /// A handed-out group parked at its tail is re-judged once its unstamped successor in - /// the next segment presents a first frame. That frame touches only the successor's - /// group, not either track or the drift anchor, so the read has to watch it directly. - #[tokio::test] - async fn unstamped_successor_first_frame_wakes_a_parked_read() { - use std::task::Context; - - let (a, a_read) = track_pair("a"); - let (mut b, b_read) = track_pair("b"); - let mut head = a.create_group(0u64.into()).unwrap(); - head.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - let mut successor = b.create_group(1u64.into()).unwrap(); - write_group_at(&mut b, 2, "edge", Duration::from_secs(20)); - let mut producer = Producer::new(); - producer.switch(a_read, None).unwrap(); - producer.switch(b_read, Position::group(1)).unwrap(); - let mut sub = producer.consume().subscribe(None); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(reading.sequence, 0, "an unbounded reach is not stale"); - assert_eq!(read(&mut reading), b"a0"); - - let (counter, waker) = CountWaker::new(); - let mut cx = Context::from_waker(&waker); - let mut next = std::pin::pin!(reading.read_frame()); - assert!(next.as_mut().poll(&mut cx).is_pending()); - - let before = counter.count(); - successor - .write_frame(Duration::from_secs(1).try_into().unwrap(), b"b1".to_vec()) - .unwrap(); - assert!(counter.count() > before, "the successor's first frame lost its wakeup"); - let result = next.as_mut().poll(&mut cx); - assert!(matches!(result, Poll::Ready(Ok(None))), "the head is stale: {result:?}"); - } - - /// An unstamped successor aborted before its first frame no longer names where the - /// track continues, so a read parked at the previous group's tail re-resolves it: - /// first to a later group already cached, then to one that arrives after the abort. - #[tokio::test] - async fn aborted_unstamped_successor_re_resolves_a_parked_read() { - use std::task::Context; - - let (a, a_read) = track_pair("a"); - let (mut b, b_read) = track_pair("b"); - let mut head = a.create_group(0u64.into()).unwrap(); - head.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - let successor = b.create_group(1u64.into()).unwrap(); - write_group_at(&mut b, 3, "edge", Duration::from_secs(20)); - let mut producer = Producer::new(); - producer.switch(a_read, None).unwrap(); - producer.switch(b_read, Position::group(1)).unwrap(); - let budget = Subscription::default().with_max_age(Duration::from_secs(10)); - let mut sub = producer.consume().subscribe(budget); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(reading.sequence, 0, "an unbounded reach is not stale"); - assert_eq!(read(&mut reading), b"a0"); - - let (counter, waker) = CountWaker::new(); - let mut cx = Context::from_waker(&waker); - let mut next = std::pin::pin!(reading.read_frame()); - assert!(next.as_mut().poll(&mut cx).is_pending()); - - successor.abort(Error::Cancel).unwrap(); - // Group 3 is now the successor, and the edge itself: within the budget. - assert!(next.as_mut().poll(&mut cx).is_pending()); - - let before = counter.count(); - write_group_at(&mut b, 2, "b2", Duration::from_secs(1)); - assert!(counter.count() > before, "the replacement successor lost its wakeup"); - let result = next.as_mut().poll(&mut cx); - assert!(matches!(result, Poll::Ready(Ok(None))), "the head is stale: {result:?}"); - } - - /// The same across segments: once the next segment's only group is aborted, the - /// segment after it says where the track continues. - #[tokio::test] - async fn aborted_unstamped_successor_re_resolves_across_segments() { - use std::task::Context; - - let (a, a_read) = track_pair("a"); - let (b, b_read) = track_pair("b"); - let (mut c, c_read) = track_pair("c"); - let mut head = a.create_group(0u64.into()).unwrap(); - head.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - let successor = b.create_group(1u64.into()).unwrap(); - write_group_at(&mut c, 2, "c2", Duration::from_secs(1)); - write_group_at(&mut c, 3, "c3", Duration::from_secs(20)); - let mut producer = Producer::new(); - producer.switch(a_read, None).unwrap(); - producer.switch(b_read, Position::group(1)).unwrap(); - producer.switch(c_read, Position::group(2)).unwrap(); - let mut sub = producer.consume().subscribe(None); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(reading.sequence, 0, "an unbounded reach is not stale"); - assert_eq!(read(&mut reading), b"a0"); - - let (counter, waker) = CountWaker::new(); - let mut cx = Context::from_waker(&waker); - let mut next = std::pin::pin!(reading.read_frame()); - assert!(next.as_mut().poll(&mut cx).is_pending()); - - let before = counter.count(); - successor.abort(Error::Cancel).unwrap(); - assert!(counter.count() > before, "the successor's abort lost its wakeup"); - let result = next.as_mut().poll(&mut cx); - assert!(matches!(result, Poll::Ready(Ok(None))), "the head is stale: {result:?}"); - } - - #[tokio::test] - async fn aborted_stamped_successor_wakes_a_parked_read() { - tokio::time::pause(); - let (a, a_read) = track_pair("a"); - let (mut b, b_read) = track_pair("b"); - let mut head = a.create_group(0u64.into()).unwrap(); - head.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - let mut successor = b.create_group(1u64.into()).unwrap(); - successor - .write_frame(Duration::from_secs(30).try_into().unwrap(), b"b1".to_vec()) - .unwrap(); - write_group_at(&mut b, 2, "b2", Duration::from_secs(1)); - write_group_at(&mut b, 3, "edge", Duration::from_secs(20)); - let mut producer = Producer::new(); - producer.switch(a_read, None).unwrap(); - producer.switch(b_read, Position::group(1)).unwrap(); - let mut sub = producer.consume().subscribe(None); - let mut reading = sub.recv_group().await.unwrap().unwrap(); - assert_eq!(reading.sequence, 0, "the rewound successor extends the reach"); - assert_eq!(read(&mut reading), b"a0"); - - let (counter, waker) = CountWaker::new(); - let mut cx = std::task::Context::from_waker(&waker); - let mut next = std::pin::pin!(reading.read_frame()); - assert!(next.as_mut().poll(&mut cx).is_pending()); - let before = counter.count(); - successor.abort(Error::Cancel).unwrap(); - assert!( - counter.count() > before, - "the stamped successor's abort lost its wakeup" - ); - let result = next.as_mut().poll(&mut cx); - assert!(matches!(result, Poll::Ready(Ok(None))), "the head is stale: {result:?}"); - } - - /// Cache eviction removes an unstamped successor from its track, so the next - /// segment's first group becomes the bound for the read already parked on A. - #[tokio::test] - async fn evicted_unstamped_successor_re_resolves_across_segments() { - tokio::time::pause(); - let (a, a_read) = track_pair("a"); - let pool = crate::cache::Pool::new(crate::cache::Config::default().with_expiry(Duration::from_secs(1))); - let info = broadcast::Info { - pool: pool.clone(), - ..Default::default() - }; - let b = track::Producer::new(Arc::new(info), "b", None); - let (mut c, c_read) = track_pair("c"); - let mut head = a.create_group(0u64.into()).unwrap(); - head.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - let successor = b.create_group(1u64.into()).unwrap(); - // The live edge is protected from eviction and lies beyond B's segment. - let _protected = b.create_group(4u64.into()).unwrap(); - write_group_at(&mut c, 2, "c2", Duration::from_secs(1)); - write_group_at(&mut c, 3, "edge", Duration::from_secs(20)); - let mut producer = Producer::new(); - producer.switch(a_read, None).unwrap(); - producer.switch(b.consume(), Position::group(1)).unwrap(); - producer.switch(c_read, Position::group(2)).unwrap(); - let mut sub = producer.consume().subscribe(None); - let mut reading = sub.recv_group().await.unwrap().unwrap(); - assert_eq!(reading.sequence, 0, "the unstamped successor leaves reach unbounded"); - assert_eq!(read(&mut reading), b"a0"); - - let (counter, waker) = CountWaker::new(); - let mut cx = std::task::Context::from_waker(&waker); - let mut next = std::pin::pin!(reading.read_frame()); - assert!(next.as_mut().poll(&mut cx).is_pending()); - let before = counter.count(); - crate::model::clock::advance(Duration::from_secs(2)); - pool.gc(crate::model::clock::now()); - assert!(successor.is_aborted(), "cache GC evicted the successor"); - assert!( - b.consume().peek_group(1).is_none(), - "the slot was removed from the cache" - ); - assert!(counter.count() > before, "the successor's eviction lost its wakeup"); - let result = next.as_mut().poll(&mut cx); - assert!(matches!(result, Poll::Ready(Ok(None))), "the head is stale: {result:?}"); - } - - #[tokio::test] - async fn pruned_segment_boundary_is_judged_against_later_segments() { - let (mut a, a_read) = track_pair("a"); - let mut producer = Producer::new(); - producer.switch(a_read, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - // Keep A's cursor draining while its newest group advances beyond the late boundary group. - write_group_at(&mut a, 0, "a0", Duration::ZERO); - assert_eq!(recv(&mut sub), 0); - write_group_at(&mut a, 3, "past-boundary", Duration::from_secs(3)); - for sequence in 3..=(2 + MAX_SEGMENTS as u64) { - let (mut track, consumer) = track_pair("later"); - producer.switch(consumer, Position::group(sequence)).unwrap(); - write_group_at(&mut track, sequence, "later", Duration::from_secs(sequence * 10)); - assert_eq!(recv(&mut sub), sequence); - } - assert!(producer.state.read().pruned.is_some()); - // A's boundary group arrives after A was pruned; it is stale against group 3's start. - write_group_at(&mut a, 2, "a2", Duration::from_secs(2)); - recv_pending(&mut sub); - } - - /// A segment's track never sees the groups of the segments after it: its own edge - /// freezes once a takeover caps it, and its last group has no successor there. - /// Both spliced cursors judge its backlog as one plain track holding the same groups - /// would: against the logical edge, with the last group's reach bounded by where the - /// next segment picks up. - #[tokio::test] - async fn a_capped_segment_is_judged_like_one_track() { - let budget = Subscription::default().with_max_age(Duration::from_millis(100)); - let old = [(0, 0), (1, 20), (2, 40), (3, 60)]; - let new = [(20, 400), (21, 420), (22, 440), (23, 460), (24, 480), (25, 500)]; - - let (mut plain, _plain_consumer) = track_pair("plain"); - for (sequence, millis) in old.into_iter().chain(new) { - write_group_at(&mut plain, sequence, "p", Duration::from_millis(millis)); - } - let mut baseline = plain.subscribe(budget.clone()); - let baseline: Vec = std::iter::from_fn(|| { - baseline - .recv_group() - .now_or_never()? - .expect("should not error") - .map(|group| group.sequence) - }) - .collect(); - // Group 3 reaches group 20's start at 400ms, 100ms behind the edge at 500ms. - assert_eq!(baseline, vec![20, 21, 22, 23, 24, 25]); - - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - producer.switch(&consumer_b, Position::group(4)).unwrap(); - for (sequence, millis) in old { - write_group_at(&mut track_a, sequence, "a", Duration::from_millis(millis)); - } - for (sequence, millis) in new { - write_group_at(&mut track_b, sequence, "b", Duration::from_millis(millis)); - } - - let mut arrival = producer.consume().subscribe(budget.clone()); - let arrival: Vec = std::iter::from_fn(|| { - kio::wait(|waiter| arrival.poll_recv_group(waiter)) - .now_or_never()? - .expect("should not error") - .map(|group| group.sequence) - }) - .collect(); - assert_eq!(arrival, baseline); - - let mut ordered = producer.consume().subscribe(budget); - let ordered: Vec = std::iter::from_fn(|| { - kio::wait(|waiter| ordered.poll_next_group(waiter)) - .now_or_never()? - .expect("should not error") - .map(|group| group.sequence) - }) - .collect(); - assert_eq!(ordered, baseline); - } - - /// A nested splice's leaves are judged within the *outer* boundary. The outer - /// window reaches them two ways, through the seek's own `end` and through - /// [`track::Subscriber::set_anchor`] recursing into the spliced segment; without - /// either, a leaf anchors drift on a group past the outer boundary and convicts - /// everything the outer segment still owes its reader. - #[tokio::test] - async fn nested_splice_judges_within_the_outer_boundary() { - let stamp = |sequence: u64| Duration::from_secs(10 * sequence); - let retain = track::Info::default().with_max_age(Duration::from_secs(60)); - let budget = Subscription::default().with_max_age(Duration::from_secs(15)); - - // Inner splice: one segment carrying groups 0..=2. Group 2 sits past the - // outer boundary below and rewinds to 0s, so it is exactly the successor the - // outer window must hide from the inner leaves: taken as group 1's, it would - // collapse group 1's reach to 0s, 30s behind the edge. - let (mut track_a, consumer_a) = track_pair_with("a", retain.clone()); - let mut inner = Producer::new(); - inner.switch(&consumer_a, None).unwrap(); - write_group_at(&mut track_a, 0, "a0", stamp(0)); - write_group_at(&mut track_a, 1, "a1", stamp(1)); - write_group_at(&mut track_a, 2, "a2", stamp(0)); - - // Outer splice: the inner spliced track up to group 2, then a plain track. - let inner_track = - track::Consumer::spliced("inner".into(), Arc::new(broadcast::Info::default()), inner.consume()); - let (mut track_b, consumer_b) = track_pair_with("b", retain); - let mut outer = Producer::new(); - outer.switch(&inner_track, None).unwrap(); - outer.switch(&consumer_b, Position::group(2)).unwrap(); - write_group_at(&mut track_b, 2, "b2", stamp(2)); - write_group_at(&mut track_b, 3, "b3", stamp(3)); - - // Group 0 is 20s behind the edge, past the 15s budget. Group 1 reaches B's - // group 2 at 20s, only 10s behind, so it survives. - let mut sub = outer.consume().subscribe(budget.clone()); - let sequences: Vec = std::iter::from_fn(|| { - kio::wait(|waiter| sub.poll_next_group(waiter)) - .now_or_never()? - .expect("should not error") - .map(|group| group.sequence) - }) - .collect(); - assert_eq!( - sequences, - vec![1, 2, 3], - "the nested segment is judged within the outer boundary" - ); - - let mut arrival = outer.consume().subscribe(budget); - let arrival: Vec = std::iter::from_fn(|| { - kio::wait(|waiter| arrival.poll_recv_group(waiter)) - .now_or_never()? - .expect("should not error") - .map(|group| group.sequence) - }) - .collect(); - assert_eq!(arrival, sequences, "the arrival path sheds the same backlog"); - } - - /// A seek's conviction is speculative: the spliced path seeks every segment and - /// delivers from one, so a losing segment's conviction must not reach the stale - /// count until the cursor commits past it. A budget that widens in between delivers - /// the group after all, and delivered content never counts. - #[tokio::test] - async fn seek_conviction_counts_only_once_committed() { - // Group 2 starts after group 3 (a rewind), so group 1's reach runs past the live - // edge and it survives a zero budget, while groups 0 and 2 are convicted. - let stamp = |sequence: u64| Duration::from_secs([0, 10, 40, 30][sequence as usize]); - - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - producer.switch(&consumer_b, Position::group(2)).unwrap(); - write_group_at(&mut track_a, 0, "a0", stamp(0)); - write_group_at(&mut track_a, 1, "a1", stamp(1)); - write_group_at(&mut track_b, 2, "b2", stamp(2)); - write_group_at(&mut track_b, 3, "b3", stamp(3)); - - let mut sub = producer.consume().subscribe(None); - let next = |sub: &mut Subscriber| { - kio::wait(|waiter| sub.poll_next_group(waiter)) - .now_or_never() - .expect("should not block") - .expect("should not error") - .expect("should not be finished") - .sequence - }; - - // The zero budget delivers group 1 (segment A's newest) and convicts group 0 - // there and group 2 in segment B, which loses this round: group 1 is lower. - // The delivery commits past group 0, so it counts; group 2 stays speculative. - assert_eq!(next(&mut sub), 1); - assert_eq!( - sub.take_stale().groups, - 1, - "only the conviction the delivery passed counts" - ); - - // Widen the budget: the convicted-but-uncommitted group 2 is delivered after - // all, so it must never reach the stale count. - sub.update(Subscription::default().with_max_age(Duration::from_secs(60))); - assert_eq!(next(&mut sub), 2); - assert_eq!(next(&mut sub), 3); - assert_eq!(sub.take_stale().groups, 0, "delivered content never counts as stale"); - } - - /// The seek floor is not a commit signal: `start_at` can raise it and lower it - /// again, so a losing segment's conviction must survive a floor that briefly - /// jumped past it, and still be deliverable (uncounted) once the budget widens. - #[tokio::test] - async fn a_reversible_floor_does_not_commit_a_conviction() { - // Group 2 starts after group 3 (a rewind), so group 1's reach runs past the live - // edge and it survives a zero budget, while groups 0 and 2 are convicted. - let stamp = |sequence: u64| Duration::from_secs([0, 10, 40, 30][sequence as usize]); - - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - producer.switch(&consumer_b, Position::group(2)).unwrap(); - write_group_at(&mut track_a, 0, "a0", stamp(0)); - write_group_at(&mut track_a, 1, "a1", stamp(1)); - write_group_at(&mut track_b, 2, "b2", stamp(2)); - write_group_at(&mut track_b, 3, "b3", stamp(3)); - - let mut sub = producer.consume().subscribe(None); - let next = |sub: &mut Subscriber| { - kio::wait(|waiter| sub.poll_next_group(waiter)) - .now_or_never() - .expect("should not block") - .expect("should not error") - .expect("should not be finished") - .sequence - }; - - // Group 1 wins; group 2's conviction in segment B stays speculative. - assert_eq!(next(&mut sub), 1); - assert_eq!(sub.take_stale().groups, 1, "the delivery commits past group 0 only"); - - // Jump the floor past the conviction without delivering anything, then bring - // it back. Nothing was delivered, so nothing was committed. - sub.start_at(10); - assert!( - kio::wait(|waiter| sub.poll_next_group(waiter)).now_or_never().is_none(), - "nothing at or above the raised floor" - ); - sub.start_at(0); - - // The widened budget delivers the convicted group; it must not have been - // counted while the floor sat above it. - sub.update(Subscription::default().with_max_age(Duration::from_secs(60))); - assert_eq!(next(&mut sub), 2); - assert_eq!(next(&mut sub), 3); - assert_eq!(sub.take_stale().groups, 0, "a floor jump commits nothing"); - } - - /// A boundary inside a group leaves a copy of the same sequence in two segments: - /// the earlier serves the head, the later the continuation the delivery splices - /// into. The later segment's cursor can convict its copy, but the delivery serves - /// that content, so the conviction must be discarded rather than counted. - #[tokio::test] - async fn a_delivered_continuation_is_not_counted_stale() { - // Group 2 starts after group 3 (a rewind), so group 1's reach runs past the live - // edge and it survives a zero budget, while groups 0 and 2 are convicted. - let stamp = |sequence: u64| Duration::from_secs([0, 10, 40, 30][sequence as usize]); - - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - // Mid-group boundary: group 1's head lives in segment A, its tail in B. - producer.switch(&consumer_b, Position { group: 1, frame: 1 }).unwrap(); - write_group_at(&mut track_a, 0, "a0", stamp(0)); - write_group_at(&mut track_a, 1, "a1", stamp(1)); - write_group_at(&mut track_b, 1, "b1", stamp(1)); - write_group_at(&mut track_b, 2, "b2", stamp(2)); - write_group_at(&mut track_b, 3, "b3", stamp(3)); - - let mut sub = producer.consume().subscribe(None); - let next = |sub: &mut Subscriber| { - kio::wait(|waiter| sub.poll_next_group(waiter)) - .now_or_never() - .expect("should not block") - .expect("should not error") - .expect("should not be finished") - .sequence - }; - - // Group 1 wins from segment A (it holds the head copy) while segment B convicts - // group 2. B's continuation copy of 1 is delivered through the head, so it - // must not be counted. - assert_eq!(next(&mut sub), 1); - assert_eq!(next(&mut sub), 3); - assert_eq!( - sub.take_stale().groups, - 2, - "groups 0 and 2 are stale; the delivered continuation of 1 is not" - ); - } - - /// A finalized losing segment's convictions must not be counted when its cursor - /// runs out: a mid-group boundary leaves the convicted sequence servable through - /// the *previous* segment (whose delivery splices into this segment's copy), and a - /// raised floor that forced the `None` can be lowered again. Only a delivery - /// commit may count; a conviction never committed is dropped with the cursor. - #[tokio::test] - async fn a_finalized_segment_flushes_nothing_without_a_delivery() { - // Group 1 starts after group 3 (a rewind), so group 0's reach runs past the live - // edge and it survives a zero budget, while groups 1 and 2 are convicted. - let stamp = |sequence: u64| Duration::from_secs([0, 40, 20, 30][sequence as usize]); - - // C owns group 0, A owns group 1's head, finalized B owns its tail onward. - // Retained long enough that the widened budget below is not clamped. - let retain = track::Info::default().with_max_age(Duration::from_secs(60)); - let (mut track_c, consumer_c) = track_pair_with("c", retain.clone()); - let (mut track_a, consumer_a) = track_pair_with("a", retain.clone()); - let (mut track_b, consumer_b) = track_pair_with("b", retain); - let mut producer = Producer::new(); - producer.switch(&consumer_c, None).unwrap(); - producer.switch(&consumer_a, Position::group(1)).unwrap(); - producer.switch(&consumer_b, Position { group: 1, frame: 1 }).unwrap(); - write_group_at(&mut track_c, 0, "c0", stamp(0)); - write_group_at(&mut track_a, 1, "a1", stamp(1)); - write_group_at(&mut track_b, 1, "b1", stamp(1)); - write_group_at(&mut track_b, 2, "b2", stamp(2)); - write_group_at(&mut track_b, 3, "b3", stamp(3)); - track_b.finish().unwrap(); - - let mut sub = producer.consume().subscribe(None); - let next = |sub: &mut Subscriber| { - kio::wait(|waiter| sub.poll_next_group(waiter)) - .now_or_never() - .expect("should not block") - .expect("should not error") - .expect("should not be finished") - .sequence - }; - - // Delivering 0 leaves B's convictions of its copy of 1 (and of 2) pending. - assert_eq!(next(&mut sub), 0); - - // A floor past B's final sequence runs its cursor out and completes the - // segment; the pending convictions must be dropped, not counted, because... - sub.start_at(10); - assert!( - kio::wait(|waiter| sub.poll_next_group(waiter)).now_or_never().is_none(), - "nothing at or above the raised floor" - ); - - // ...group 1 is still deliverable: A holds its head, and the delivery splices - // into the copy B's cursor convicted. Counted-and-delivered is the one thing - // the stale stat must never say. - sub.start_at(0); - sub.update(Subscription::default().with_max_age(Duration::from_secs(60))); - assert_eq!(next(&mut sub), 1); - assert_eq!( - sub.take_stale().groups, - 0, - "no delivery ever committed past a conviction" - ); - } - - /// A parked group re-offered through a *nested* splice is re-checked against the - /// drift budget exactly as a plain segment's would be: the cap rising widened the - /// live edge too, and the covering segment's leaf is the cursor that can judge it. - #[tokio::test] - async fn nested_parked_group_is_rechecked_when_the_cap_rises() { - let stamp = |sequence: u64| Duration::from_secs(10 * sequence); - - // Inner splice over one plain track holding groups 0..=2. - let (mut track_a, consumer_a) = track_pair("a"); - let mut inner = Producer::new(); - inner.switch(&consumer_a, None).unwrap(); - write_group_at(&mut track_a, 0, "a0", stamp(0)); - write_group_at(&mut track_a, 1, "a1", stamp(1)); - write_group_at(&mut track_a, 2, "a2", stamp(2)); - - let inner_track = - track::Consumer::spliced("inner".into(), Arc::new(broadcast::Info::default()), inner.consume()); - let mut outer = Producer::new(); - outer.switch(&inner_track, None).unwrap(); - - // Capped at group 0: group 0 is the whole window (and so its own live edge), - // while groups 1 and 2 park above the cap. - let mut sub = outer.consume().subscribe(None); - sub.end_at(..1); - let recv = |sub: &mut Subscriber| { - kio::wait(|waiter| sub.poll_recv_group(waiter)) - .now_or_never() - .map(|group| { - group - .expect("should not error") - .expect("should not be finished") - .sequence - }) - }; - assert_eq!(recv(&mut sub), Some(0)); - assert_eq!(recv(&mut sub), None, "groups beyond the cap park"); - - // Raising the cap re-offers the parked groups, but it also widened the live - // edge to group 2, so group 1 is stale now and must be skipped, not served. - sub.end_at(..); - assert_eq!(recv(&mut sub), Some(2), "the re-offered backlog is re-checked"); - assert_eq!(sub.take_stale().groups, 1, "the skipped park is counted once"); - } - - #[tokio::test] - async fn consecutive_updates_wake() { - use std::task::Context; - - let (track_a, consumer_a) = track_pair("a"); - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - let prefs = sub.prefs(); - - let (counter, waker) = CountWaker::new(); - let mut cx = Context::from_waker(&waker); - - let mut fut = std::pin::pin!(sub.recv_group()); - assert!(fut.as_mut().poll(&mut cx).is_pending()); - - // First update wakes and is applied on the next poll. - *prefs.write().ok().unwrap() = Subscription::default().with_priority(1); - assert_eq!(counter.count(), 1); - assert!(fut.as_mut().poll(&mut cx).is_pending()); - assert_eq!(track_a.subscription().unwrap().priority, 1); - - // The poll that consumed the change must have re-registered: a second - // update, with no other activity in between, still wakes. Count the delta - // rather than the total: `kio::Park` reuses a waiter that still holds - // registrations, so applying the change above notifies a list this poll is - // itself parked on and self-wakes once. That extra wake costs a redundant - // poll and nothing else, while a lost wakeup parks the task forever. - let before = counter.count(); - *prefs.write().ok().unwrap() = Subscription::default().with_priority(2); - assert!(counter.count() > before, "second update lost its wakeup"); - assert!(fut.as_mut().poll(&mut cx).is_pending()); - assert_eq!(track_a.subscription().unwrap().priority, 2); - } - - /// A route dying partway through a group is resumed at the frame it stopped on: - /// the reader keeps the same group handle, sees no duplicate frames, and never - /// learns a route changed. This is the whole point of frame-precise boundaries - /// (a JSON append log's group may never roll). - #[tokio::test] - async fn takeover_splices_mid_group() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // A opens group 0 and writes two frames, leaving it open. - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"a1".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(reading.sequence, 0); - assert_eq!(read(&mut reading), b"a0", "first"); - assert_eq!(read(&mut reading), b"a1", "second"); - assert!(reading.read_frame().now_or_never().is_none(), "group is still open"); - - // The route dies mid-group and B takes over inside it. - producer.takeover(&consumer_b).unwrap(); - track_a.abort(Error::Dropped).unwrap(); - recv_pending(&mut sub); - - let demand = track_b.subscription().unwrap(); - assert_eq!( - demand.start, - Some(Position { group: 0, frame: 2 }), - "resumes in the same group, at the frame the old route stopped on" - ); - - let mut group = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - group.start_at(2).unwrap(); - group.write_frame(Timestamp::ZERO, b"b2".to_vec()).unwrap(); - group.finish().unwrap(); - - // Same handle, no seam: the continuation is not surfaced as a second group. - assert_eq!(read(&mut reading), b"b2"); - assert!(reading.read_frame().now_or_never().unwrap().unwrap().is_none()); - recv_pending(&mut sub); - } - - #[tokio::test] - async fn a_replacement_copy_keeps_the_handed_out_group_latency_budget() { - let (track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - let mut head = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - head.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().expect("head group"); - assert_eq!(read(&mut reading), b"a0"); - - producer.takeover(&consumer_b).unwrap(); - track_a.abort(Error::Dropped).unwrap(); - recv_pending(&mut sub); - - let mut continuation = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - continuation.start_at(1).unwrap(); - continuation.write_frame(Timestamp::ZERO, b"b1".to_vec()).unwrap(); - assert_eq!(read(&mut reading), b"b1"); - - assert!( - reading.read_frame().now_or_never().is_none(), - "the replacement group is still the live edge" - ); - - crate::model::clock::advance(std::time::Duration::from_secs(1)); - write_group_at(&mut track_b, 1, "edge", std::time::Duration::from_secs(1)); - - let result = reading - .read_frame() - .now_or_never() - .expect("the newer group changes the verdict"); - // Drained, so the budget ends the replacement rather than truncating it. - assert!(matches!(result, Ok(None)), "the replacement ends: {result:?}"); - } - - /// A replacement that serves the whole group instead of the requested tail still - /// splices cleanly: the reader picks up at the seam and never sees the frames it - /// already has. - /// - /// This is what a peer too old for frame bounds delivers. The lite subscriber widens - /// the request to the whole group for such a peer rather than failing to encode it - /// (see `TrackServe::widen_frame_bounds`), so the extra frames are filtered here. - #[tokio::test] - async fn takeover_splices_a_replacement_that_resends_the_head() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"a1".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - assert_eq!(read(&mut reading), b"a1"); - - producer.takeover(&consumer_b).unwrap(); - track_a.abort(Error::Dropped).unwrap(); - recv_pending(&mut sub); - - // B numbers the group from 0, as an older peer that cannot carry the offset does. - let mut group = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"dup0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"dup1".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"b2".to_vec()).unwrap(); - group.finish().unwrap(); - - // The reader resumes at frame 2; the re-sent head is filtered, not replayed. - assert_eq!(read(&mut reading), b"b2"); - assert!(reading.read_frame().now_or_never().unwrap().unwrap().is_none()); - - // And the group is not surfaced a second time just because B served it whole. - recv_pending(&mut sub); - } - - /// A route dying midway through a chunked frame resumes *at* that frame, not after - /// it. Only the dead route ever saw its payload, and only part of it, so nothing - /// downstream can use it. - #[tokio::test] - async fn takeover_redelivers_an_incomplete_frame() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - - // Frame 1 is opened and half written when the route dies. Dropping the frame - // producer without finishing aborts the group, as a broken stream would. - { - let mut frame = group - .create_frame(frame::Info { - size: 6, - timestamp: Timestamp::ZERO, - }) - .unwrap(); - frame.write(b"foo".to_vec()).unwrap(); - } - track_a.abort(Error::Dropped).unwrap(); - - producer.takeover(&consumer_b).unwrap(); - recv_pending(&mut sub); - - let demand = track_b.subscription().unwrap(); - assert_eq!( - demand.start, - Some(Position { group: 0, frame: 1 }), - "the half-written frame must be redelivered, not skipped" - ); - - let mut group = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - group.start_at(1).unwrap(); - group.write_frame(Timestamp::ZERO, b"b1".to_vec()).unwrap(); - - // The reader was parked on frame 1 and picks it up from the replacement. - assert_eq!(read(&mut reading), b"b1"); - } - - /// The old route racing past its frame boundary is filtered, exactly as an - /// out-of-range group is. - #[tokio::test] - async fn mid_group_boundary_caps_the_old_route() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - let mut group_a = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group_a.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - - producer.takeover(&consumer_b).unwrap(); - recv_pending(&mut sub); - // The old copy's demand is capped just below the boundary. - let demand = track_a.subscription().unwrap(); - // Exclusive: serve group 0 up to and including frame 0. - assert_eq!(demand.end, Some(Position { group: 0, frame: 1 })); - - // A keeps writing anyway; those frames belong to B's range now. - group_a.write_frame(Timestamp::ZERO, b"a1-over-cap".to_vec()).unwrap(); - - let mut group_b = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - group_b.start_at(1).unwrap(); - group_b.write_frame(Timestamp::ZERO, b"b1".to_vec()).unwrap(); - - assert_eq!(read(&mut reading), b"b1"); - } - - /// A takeover only rolls to the next group once the current one is complete; - /// there is nothing left to append to it. - #[tokio::test] - async fn takeover_rolls_past_a_finished_group() { - let (mut track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - - producer.takeover(&consumer_b).unwrap(); - recv_pending(&mut sub); - - // The boundary rolls to the next group once the current one is complete, and - // the replacement is asked to resume there. - assert_eq!(producer.resume_position(), Some(Position::group(1))); - assert_eq!(track_b.subscription().unwrap().start, Some(Position::group(1))); - } - - /// A copy dying mid-group stalls its readers instead of erroring them, the same - /// way a dead segment stalls the track. The next takeover resumes them. - #[tokio::test] - async fn dead_copy_stalls_until_the_continuation() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - - // The copy fails outright; the reader must not see the error. - group.abort(Error::Dropped).unwrap(); - assert!(reading.read_frame().now_or_never().is_none(), "should stall, not error"); - - producer.takeover(&consumer_b).unwrap(); - let mut group = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - group.start_at(1).unwrap(); - group.write_frame(Timestamp::ZERO, b"b1".to_vec()).unwrap(); - assert_eq!(read(&mut reading), b"b1"); - } - - /// A dead copy stalls only while a replacement can still arrive. Once the logical - /// track aborts, no switch is coming, so the reader has to surface the loss rather - /// than park on it forever. - #[tokio::test] - async fn dead_copy_ends_once_the_track_aborts() { - let (track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - - // The route dies. A switch could still replace it, so the reader parks. - group.abort(Error::Dropped).unwrap(); - assert!( - reading.read_frame().now_or_never().is_none(), - "a switch could still come" - ); - - // The logical track aborts: nothing can replace it now. - producer.abort(Error::Cancel).unwrap(); - assert!( - matches!(reading.read_frame().now_or_never(), Some(Err(_))), - "an aborted track must not leave the reader parked" - ); - } - - /// A group the reader gave up on stays lost: every later poll reports the same - /// loss. `finished` is how a reader tells a transport loss from a bad payload, so - /// it must not park once the route's aborted copy has been reclaimed from its cache. - #[tokio::test] - async fn lost_group_stays_lost() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - - // The relay resets the rest of the group, then moves on. The next write - // reclaims the aborted slot, so the route no longer knows the group at all. - group.abort(Error::Stream(crate::StreamError::Old)).unwrap(); - write_group(&mut track_a, 1, "a1"); - - assert!(matches!( - reading.read_frame().now_or_never(), - Some(Err(Error::Stream(crate::StreamError::Old))) - )); - assert!( - matches!( - reading.finished().now_or_never(), - Some(Err(Error::Stream(crate::StreamError::Old))) - ), - "a lost group must not park or change its answer" - ); - assert!(matches!( - reading.read_frame().now_or_never(), - Some(Err(Error::Stream(crate::StreamError::Old))) - )); - } - - /// A route whose copy of the group is missing the frames the reader needs is treated - /// like a dead one. Reading the tail as if it were the head would silently renumber - /// every frame in the group. - /// - /// The loss is reported rather than parked on, because the route already produced - /// past the missing frames. Boundaries come from `resume_position`, which only moves - /// forward, so no future takeover will ever ask anyone for frame 0 again. - #[tokio::test] - async fn copy_missing_the_head_is_lost() { - let (track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // This route only ever held the tail: its copy starts at frame 2. - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.start_at(2).unwrap(); - group.write_frame(Timestamp::ZERO, b"a2".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(reading.sequence, 0); - assert_eq!(reading.index(), 0, "the reader still wants frame 0"); - assert!( - matches!(reading.read_frame().now_or_never(), Some(Err(Error::Lagged))), - "must report the loss rather than serve the tail as the head" - ); - } - - /// Latest-only recovery ends a missing continuation once the replacement is ahead. - #[tokio::test] - async fn live_route_ahead_of_the_seam_ends_the_group() { - let (track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"a1".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - assert_eq!(read(&mut reading), b"a1"); - - // B takes over inside group 0, then runs ahead to group 1 without ever serving - // the rest of group 0. That pushes the resume point past the seam. - producer.takeover(&consumer_b).unwrap(); - write_group(&mut track_b, 1, "b1"); - assert_eq!(recv(&mut sub), 1); - - assert!( - matches!(reading.read_frame().now_or_never(), Some(Err(Error::NotFound))), - "latest recovery cannot request an older continuation" - ); - } - - /// A declared start at or below the seam promises the group, so a newer group - /// overtaking it on the wire does not end the wait. - #[tokio::test] - async fn declared_start_waits_for_an_overtaken_group() { - let (track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"a1".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - assert_eq!(read(&mut reading), b"a1"); - - producer.takeover(&consumer_b).unwrap(); - track_b.start_at(0).unwrap(); - write_group(&mut track_b, 1, "b1"); - assert_eq!(recv(&mut sub), 1); - assert!( - reading.read_frame().now_or_never().is_none(), - "the declared start still promises group 0" - ); - - let mut group = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - group.start_at(2).unwrap(); - group.write_frame(Timestamp::ZERO, b"b2".to_vec()).unwrap(); - assert_eq!(read(&mut reading), b"b2"); - } - - #[tokio::test] - async fn takeover_after_empty_segment_keeps_live_edge() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - recv_pending(&mut sub); - assert_eq!(track_a.subscription().unwrap().start, None); - - // A dies before producing anything; B takes over. - drop(track_a); - producer.takeover(&consumer_b).unwrap(); - recv_pending(&mut sub); - - // The replacement must inherit live-edge demand, not a full backfill. - assert_eq!(track_b.subscription().unwrap().start, None); - } - - #[tokio::test] - async fn fetch_fails_over_to_a_newer_segment() { - let (track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let consumer = producer.consume(); - - // The latched copy is dead: the fetch parks for a takeover instead of - // surfacing the route's failure as the group's. - track_a.abort(Error::Dropped).unwrap(); - let fetch = consumer.fetch_group(0, None); - let mut fetch = std::pin::pin!(fetch); - assert!(futures::poll!(fetch.as_mut()).is_pending(), "a dead copy should park"); - - // The replacement has the group cached; the fetch fails over to it. - write_group(&mut track_b, 0, "b0"); - producer.takeover(&consumer_b).unwrap(); - let group = fetch.await.expect("fetch should fail over"); - assert_eq!(group.sequence, 0); - } - - #[tokio::test] - async fn fetch_aborts_with_the_track() { - let (track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let consumer = producer.consume(); - - track_a.abort(Error::Dropped).unwrap(); - let fetch = consumer.fetch_group(0, None); - let mut fetch = std::pin::pin!(fetch); - assert!(futures::poll!(fetch.as_mut()).is_pending(), "a dead copy should park"); - - // The logical track aborting ends the wait with its error. - producer.abort(Error::Cancel).unwrap(); - assert!(matches!(fetch.await, Err(Error::Cancel))); - } - - #[tokio::test] - async fn fetch_pending_ends_when_the_track_aborts() { - let (track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let consumer = producer.consume(); - - // A fetch handler exists but never answers: the latched fetch parks. - let handler = track_a.dynamic(); - let fetch = consumer.fetch_group(0, None); - let mut fetch = std::pin::pin!(fetch); - assert!( - futures::poll!(fetch.as_mut()).is_pending(), - "unanswered fetch should park" - ); - - // The front aborting must end the in-flight fetch, not strand it on the - // copy that will never answer. - producer.abort(Error::Cancel).unwrap(); - assert!(matches!(fetch.await, Err(Error::Cancel))); - drop(handler); - } - - #[tokio::test] - async fn fetch_error_from_a_live_copy_is_authoritative() { - let (_track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let consumer = producer.consume(); - - // The copy is alive and will never carry group 0 (no fetch handler): - // its answer surfaces instead of parking for a takeover. - let result = consumer - .fetch_group(0, None) - .now_or_never() - .expect("a live copy's answer must resolve immediately"); - assert!(matches!(result, Err(Error::NotFound))); - } - - #[tokio::test] - async fn takeover_after_produced_segment_resumes_at_the_boundary() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - recv_pending(&mut sub); - assert_eq!(track_a.subscription().unwrap().start, None); - - // A produced groups before its route died; B takes over. - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - assert_eq!(recv(&mut sub), 0); - assert_eq!(recv(&mut sub), 1); - drop(track_a); - producer.takeover(&consumer_b).unwrap(); - recv_pending(&mut sub); - - // A live-edge subscriber resumes at the boundary rather than at B's own live - // edge, so the splice loses nothing. What that can replay is bounded by how - // far the boundary trails B, which is failover latency; a consumer that wants - // none of it skips on its own max age budget downstream. - assert_eq!(track_b.subscription().unwrap().start, Some(Position::group(2))); - - // The segment range still filters a group below the boundary, so a replacement - // that serves one anyway never delivers it twice. - write_group(&mut track_b, 1, "b1"); - recv_pending(&mut sub); - - // Groups at the live edge still arrive past the boundary. - write_group(&mut track_b, 5, "b5"); - assert_eq!(recv(&mut sub), 5); - } - - #[tokio::test] - async fn takeover_keeps_an_explicit_start() { - let (mut track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - // The budget is what makes the backfill readable: an explicit start says which - // groups the publisher should send, not that the subscriber will wait for them. - let mut sub = producer.consume().subscribe( - Subscription::default() - .with_start(Position::group(0)) - .with_max_age(std::time::Duration::from_secs(60)), - ); - recv_pending(&mut sub); - assert_eq!(track_a.subscription().unwrap().start, Some(Position::group(0))); - - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - assert_eq!(recv(&mut sub), 0); - assert_eq!(recv(&mut sub), 1); - drop(track_a); - producer.takeover(&consumer_b).unwrap(); - recv_pending(&mut sub); - - // A subscriber that asked for history keeps its gapless backfill: the - // boundary raises the explicit start to the first missing group. - assert_eq!(track_b.subscription().unwrap().start, Some(Position::group(2))); - } - - #[tokio::test] - async fn switch_validates_boundaries() { - let (mut track_a, consumer_a) = track_pair("a"); - let (_track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - - // A later switch requires an explicit, advancing boundary; 0 is only legal - // when the previous segment never produced a group. - assert!(producer.switch(&consumer_b, None).is_err()); - write_group(&mut track_a, 0, "a0"); - assert!(producer.switch(&consumer_b, Position::group(0)).is_err()); - producer.switch(&consumer_b, Position::group(1)).unwrap(); - } - - /// Repeated failovers must not accumulate a segment per takeover: dead - /// predecessors are pruned once the list outgrows [`MAX_SEGMENTS`], and the - /// boundary stays where the pruned segments left it rather than collapsing. - #[tokio::test] - async fn prune_bounds_segments_and_keeps_the_boundary() { - let mut producer = Producer::new(); - let mut sub = producer.consume().subscribe(None); - - // Each route serves one group and dies; the next takeover resumes past it. - let rounds = 2 * MAX_SEGMENTS as u64; - for sequence in 0..rounds { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert_eq!(recv(&mut sub), sequence); - track.abort(Error::Dropped).unwrap(); - } - assert_eq!( - producer.state.read().segments.len(), - MAX_SEGMENTS, - "dead predecessors should have been pruned" - ); - - // The floor carries what the pruned segments served: a replacement splices - // one past the newest group, never back at the start. - let (mut track, consumer) = track_pair("final"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, 0, "below-the-floor"); - recv_pending(&mut sub); - write_group(&mut track, rounds, "resumed"); - assert_eq!(recv(&mut sub), rounds); - } - - /// Pruning must not depend on predecessors dying: a takeover boundary is the - /// resume position, so a capped segment already produced everything it owns - /// and is retired even while its track is alive. Route churn with long-lived - /// sessions would otherwise grow the list without bound, since nothing - /// re-prunes between switches. - #[tokio::test] - async fn prune_retires_live_predecessors() { - let mut producer = Producer::new(); - let mut sub = producer.consume().subscribe(None); - - // Every route stays alive: only the takeover cap retires them. - let mut tracks = Vec::new(); - let rounds = 2 * MAX_SEGMENTS as u64; - for sequence in 0..rounds { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert_eq!(recv(&mut sub), sequence); - tracks.push(track); - } - assert_eq!( - producer.state.read().segments.len(), - MAX_SEGMENTS, - "live predecessors should still be pruned" - ); - } - - /// A route that skips a group for good (SUBSCRIBE_START names a later first - /// group) must fail the readers waiting on it over, not stall them: the route - /// is alive, so nothing else would ever mark the gap as permanent. - #[tokio::test] - async fn declared_start_fails_over_a_skipped_group() { - let (track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // A serves one frame of an open group, then dies mid-group. - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - assert!( - reading.read_frame().now_or_never().is_none(), - "the seam parks for a continuation" - ); - // The session dies mid-group: its group producers go with it. - drop(group); - track_a.abort(Error::Dropped).unwrap(); - - // B takes over but declares group 1 as its first: the seam is skipped for - // good, not merely late. - producer.takeover(&consumer_b).unwrap(); - track_b.start_at(1).unwrap(); - write_group(&mut track_b, 1, "b1"); - assert_eq!(recv(&mut sub), 1); - - // The reader waiting on group 0's continuation surfaces the loss instead - // of parking forever on a live route. - assert!( - reading - .read_frame() - .now_or_never() - .expect("the skipped seam must resolve") - .is_err(), - "the skipped frames are a loss, not a clean end" - ); - } - - /// A reader latched on a capped copy must follow a moved boundary: popping an - /// empty successor can re-cap its predecessor lower, and the frames past the - /// revised cap belong to the replacement route. Serving them from the stale - /// copy would substitute (or duplicate) the replacement's frames. - #[tokio::test] - async fn latched_reader_follows_a_moved_boundary() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - let (track_c, consumer_c) = track_pair("c"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // A serves two frames of an open group; B splices at the produced edge, so - // the reader latches A's copy with a cap of (0, 2). - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"a1".to_vec()).unwrap(); - producer.switch(&consumer_b, Position { group: 0, frame: 2 }).unwrap(); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - - // B dies empty; C replaces it with an earlier boundary, moving A's cap down - // to (0, 1). Frame 1 now belongs to C, and the latched reader must fetch - // C's copy rather than serving A's past the revised cap. - drop(track_b); - producer.switch(&consumer_c, Position { group: 0, frame: 1 }).unwrap(); - let mut group = track_c.create_group(group::Info { sequence: 0 }).unwrap(); - group.start_at(1).unwrap(); - group.write_frame(Timestamp::ZERO, b"c1".to_vec()).unwrap(); - assert_eq!(read(&mut reading), b"c1"); - } - - /// A long-lived subscriber must not accumulate a cursor per takeover when the - /// old routes stay alive: their tracks never end, so nothing else would ever - /// cut them. A bounded number of pruned cursors linger to drain out-of-order - /// stragglers; beyond the bound the oldest are cut, subscription, demand, - /// and all. - #[tokio::test] - async fn pruned_cursors_stay_bounded() { - let mut producer = Producer::new(); - let mut sub = producer.consume().subscribe(None); - - let mut tracks = Vec::new(); - let rounds = 3 * MAX_SEGMENTS as u64; - for sequence in 0..rounds { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert_eq!(recv(&mut sub), sequence); - tracks.push(track); - } - - // The cut happens while polling and the next poll reaps the entries. - recv_pending(&mut sub); - recv_pending(&mut sub); - assert_eq!( - sub.segments.len(), - 2 * MAX_SEGMENTS, - "pruned cursors beyond the bound must be cut" - ); - assert!(tracks[0].subscription().is_none(), "a cut cursor releases its demand"); - assert!( - tracks[rounds as usize - MAX_SEGMENTS - 1].subscription().is_some(), - "a pruned cursor within the bound keeps draining" - ); - } - - /// A capped subscriber riding out route churn keeps its parked groups across - /// prunes (their re-offer latches the delivering copy, so they still read - /// out) while the straggler bound cuts the oldest entries whole, parked group - /// and all, so nothing accumulates without bound. - #[tokio::test] - async fn capped_subscriber_bounds_parked_segments() { - let mut producer = Producer::new(); - let mut sub = producer.consume().subscribe(replay()); - sub.end_at(..1); - - // Every round parks one group beyond the cap, then fails over to a live - // replacement route. - let mut tracks = Vec::new(); - let rounds = 3 * MAX_SEGMENTS as u64; - for round in 0..rounds { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, round + 1, "beyond-the-cap"); - recv_pending(&mut sub); - tracks.push(track); - } - - recv_pending(&mut sub); - assert_eq!( - sub.segments.len(), - 2 * MAX_SEGMENTS, - "parked entries must stay bounded, not accumulate" - ); - assert!(tracks[0].subscription().is_none(), "a cut entry releases its demand"); - - // Raising the cap re-offers every retained parked group: the pruned ones - // within the bound deliver through their latched copies, and only the cut - // ranges are lost. - sub.end_at(..); - for sequence in (rounds - 2 * MAX_SEGMENTS as u64 + 1)..=rounds { - assert_eq!(recv(&mut sub), sequence); - } - recv_pending(&mut sub); - } - - /// A route given up on is not gone for good: the peek miss may have come from - /// its declared start (SUBSCRIBE_START), which demand moving backward lowers. - /// When the copy lands after all, the reader revives the route instead of - /// reporting the frames lost. - /// - /// The seam is reached with no latched copy in hand (A's is capped at the - /// boundary and rolls clean), so the revival is the only way through: a stale - /// latch would otherwise error, re-point `dead` at its own segment, and - /// un-guard this route by accident. - #[tokio::test] - async fn buried_route_revives_when_the_copy_lands() { - let (track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // A serves one frame of a group and stops there. - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"a0".to_vec()).unwrap(); - - // B takes over at the seam and declares it starts at group 1, so the peek - // for the continuation is a permanent miss and the route is buried. - producer.takeover(&consumer_b).unwrap(); - track_b.start_at(1).unwrap(); - - // Handed out after the boundary exists, so the copy is capped at the seam. - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"a0"); - assert!(reading.read_frame().now_or_never().is_none(), "the seam parks"); - - // Demand widens backward: the floor drops and B serves the continuation - // after all. Reviving has to beat the strand check, since B producing the - // frames is itself what moves the resume point past the seam. - track_b.start_at(None).unwrap(); - let mut cont = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - cont.start_at(1).unwrap(); - cont.write_frame(Timestamp::ZERO, b"b1".to_vec()).unwrap(); - assert_eq!(read(&mut reading), b"b1"); - } - - /// A partial copy nobody asked for (its first frame above the reader's - /// position, from a protocol-violating peer) is never latched: the latch is - /// trusted for alignment, so a misaligned one would surface its frames under - /// the wrong indices. The reader buries the copy as lagged instead. - /// - /// This is also the guard on the revival in [`Group::poll_covering`], which - /// reconsiders a buried route ahead of the terminal checks: revival keys on - /// [`track::Consumer::poll_serving_group`], and if that stopped requiring the - /// copy to serve the requested frame, the copy here would revive, re-bury on - /// the very next peek, and loop forever inside one poll. That regression - /// surfaces as a hang rather than a failed assertion, which nextest reports - /// as a TIMEOUT. - #[tokio::test] - async fn misaligned_copy_is_lost_without_spinning() { - let (track, consumer) = track_pair("t"); - let mut producer = Producer::new(); - producer.takeover(&consumer).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // The copy of group 0 starts at frame 5 while the segment owes the whole - // group. - let mut group = track.create_group(group::Info { sequence: 0 }).unwrap(); - group.start_at(5).unwrap(); - group.write_frame(Timestamp::ZERO, b"f5".to_vec()).unwrap(); - - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(reading.index(), 0); - - // The head is gone for good, so the group ends as a loss naming the copy - // that could not cover it, rather than surfacing misnumbered frames. - let err = reading - .read_frame() - .now_or_never() - .expect("the loss must resolve rather than park") - .expect_err("a misaligned copy is a loss, never misnumbered frames"); - assert!(matches!(err, Error::Lagged), "expected a lagged copy, got {err:?}"); - } - - /// A takeover splices in a track with no groups at all. A peek that stopped at the - /// newest segment would read the logical track as empty until the new route - /// produces, which is the whole failover window. - #[tokio::test] - async fn peeks_walk_past_an_empty_segment() { - let (mut track_a, consumer_a) = track_pair("a"); - let (_track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - producer.switch(&consumer_b, Position { group: 2, frame: 0 }).unwrap(); - - let consumer = producer.consume(); - assert_eq!(consumer.peek_latest().map(|group| group.sequence), Some(1)); - assert_eq!(consumer.peek_before(1).map(|group| group.sequence), Some(0)); - assert_eq!(consumer.peek_before(0).map(|group| group.sequence), None); - } - - /// A segment's track can hold groups outside the range the segment serves: its own - /// edge can race past the cap before the next takeover splices above it. Those - /// belong to the newer segment's range, so a peek must not return them. - #[tokio::test] - async fn peeks_respect_segment_bounds() { - let (mut track_a, consumer_a) = track_pair("a"); - let (_track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - write_group(&mut track_a, 0, "a0"); - // B takes over at 1, capping A at 0. A then races past its own cap. - producer.switch(&consumer_b, Position { group: 1, frame: 0 }).unwrap(); - write_group(&mut track_a, 5, "a5"); - - let consumer = producer.consume(); - assert_eq!( - consumer.peek_latest().map(|group| group.sequence), - Some(0), - "group 5 is above A's cap and belongs to B's range" - ); - } - - /// Seeking forward keeps a latched copy that still covers the new position: - /// for a pruned segment it is the only copy left, so clearing it would lose - /// frames the reader still holds. - #[tokio::test] - async fn seek_keeps_a_pruned_latch() { - let (track, consumer) = track_pair("t"); - let mut producer = Producer::new(); - producer.takeover(&consumer).unwrap(); - let mut sub = producer.consume().subscribe(None); - - let mut group = track.create_group(group::Info { sequence: 0 }).unwrap(); - for payload in [b"f0", b"f1", b"f2"] { - group.write_frame(Timestamp::ZERO, payload.to_vec()).unwrap(); - } - group.finish().unwrap(); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"f0"); - track.abort(Error::Dropped).unwrap(); - - // Failovers prune the segment; the latch is all that remains of group 0. - for sequence in 1..=MAX_SEGMENTS as u64 { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert_eq!(recv(&mut sub), sequence); - track.abort(Error::Dropped).unwrap(); - } - assert!( - producer.state.read().pruned.is_some(), - "the first segment should be pruned" - ); - - // The seek lands inside the latch: the frame survives the prune. - reading.start_at(2); - assert_eq!(read(&mut reading), b"f2"); - } - - /// The straggler bound holds on every polling entry point: a subscriber - /// driven only through datagrams accumulates a cursor per takeover all the - /// same, so the reap must run from the shared sync, not just the group path. - #[tokio::test] - async fn datagram_poller_bounds_pruned_cursors() { - let mut producer = Producer::new(); - let mut sub = producer.consume().subscribe(None); - - for sequence in 0..(3 * MAX_SEGMENTS as u64) { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert!( - kio::wait(|waiter| sub.poll_recv_datagram(waiter)) - .now_or_never() - .is_none(), - "no datagram expected" - ); - } - - assert_eq!( - sub.segments.len(), - 2 * MAX_SEGMENTS, - "the reap must run on the datagram path too" - ); - } - - /// An out-of-order group that lands after its segment was pruned still drains: - /// the cursor lingers (within the straggler bound) exactly because an empty - /// poll is not proof of completeness, and its demand is what keeps the - /// upstream serving the stragglers. - #[tokio::test] - async fn late_group_drains_from_a_pruned_cursor() { - let mut producer = Producer::new(); - let mut sub = producer.consume().subscribe(replay()); - - // A delivers group 1 first; group 0 is still in flight when A is outranked - // and enough failovers prune its segment. - let (mut track_a, consumer_a) = track_pair("a"); - producer.takeover(&consumer_a).unwrap(); - write_group(&mut track_a, 1, "a1"); - assert_eq!(recv(&mut sub), 1); - - for sequence in 2..=(1 + MAX_SEGMENTS as u64) { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert_eq!(recv(&mut sub), sequence); - } - assert!( - producer.state.read().pruned.is_some(), - "the first segment should be pruned" - ); - - // The straggler finally lands and surfaces through the lingering cursor. - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - } - - /// `finished()` on a group bounded by a mid-group takeover fails once no - /// segment can serve the seam: the continuation owns the end, and when the - /// covering segments are pruned away the group is truncated, not complete. - /// Polled without draining first, which is exactly the caller the seam check - /// must park (and wake) rather than hang. - #[tokio::test] - async fn finished_fails_for_a_pruned_bounded_group() { - let (track_a, consumer_a) = track_pair("a"); - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // A serves two frames of an open group; B splices at the seam, so the - // reader's copy is handed out bounded at (0, 2). - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"f0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"f1".to_vec()).unwrap(); - let (mut track_b, consumer_b) = track_pair("b"); - producer.takeover(&consumer_b).unwrap(); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - drop(group); - track_a.abort(Error::Dropped).unwrap(); - - // Until B advances, it may still serve frame 2. - assert!(reading.finished().now_or_never().is_none()); - write_group(&mut track_b, 1, "b1"); - assert_eq!(recv(&mut sub), 1); - - // Enough failovers prune A and B: nothing can serve the seam anymore, so - // the group was cut short. - for sequence in 2..=(1 + MAX_SEGMENTS as u64) { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert_eq!(recv(&mut sub), sequence); - } - let err = reading - .finished() - .now_or_never() - .expect("the lost seam must resolve") - .unwrap_err(); - assert!(matches!(err, Error::Dropped), "{err:?}"); - } - - /// A reader that drained a bounded copy to its cap before the seam was lost - /// still learns the group was truncated: `finished()` and the read cursor - /// answer for the original group, not for where the copy stopped. - #[tokio::test] - async fn finished_fails_for_a_drained_pruned_bounded_group() { - let (track_a, consumer_a) = track_pair("a"); - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"f0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"f1".to_vec()).unwrap(); - let (mut track_b, consumer_b) = track_pair("b"); - producer.takeover(&consumer_b).unwrap(); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"f0"); - assert_eq!(read(&mut reading), b"f1"); - drop(group); - track_a.abort(Error::Dropped).unwrap(); - - write_group(&mut track_b, 1, "b1"); - assert_eq!(recv(&mut sub), 1); - for sequence in 2..=(1 + MAX_SEGMENTS as u64) { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert_eq!(recv(&mut sub), sequence); - } - - let mut cloned = reading.clone(); - let err = reading - .finished() - .now_or_never() - .expect("the lost seam must resolve") - .unwrap_err(); - assert!(matches!(err, Error::Dropped), "{err:?}"); - - // Reading past the cap rolls off the bounded copy; the end it reaches is - // still the truncation, never a clean `None`. - let err = reading - .read_frame() - .now_or_never() - .expect("the lost seam must resolve") - .unwrap_err(); - assert!(matches!(err, Error::Dropped), "{err:?}"); - let err = reading.finished().now_or_never().unwrap().unwrap_err(); - assert!(matches!(err, Error::Dropped), "{err:?}"); - - // Same for `next_frame`. - let next = cloned.next_frame().now_or_never().unwrap(); - assert!(matches!(next, Err(Error::Dropped)), "{:?}", next.err()); - } - - /// A handed-out group survives its segment's prune, for every reader: the - /// delivering copy is latched at hand-out and a clone re-latches it at its - /// own position (fanout must not depend on the segment list remembering the - /// route), so both read the payload out and end cleanly instead of stalling. - #[tokio::test] - async fn group_reader_gives_up_below_the_pruned_floor() { - let mut producer = Producer::new(); - let mut sub = producer.consume().subscribe(None); - - // Group 0 is handed out but never read; its route dies and enough - // failovers follow to prune the segment that held it. - let (mut track, consumer) = track_pair("t0"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, 0, "kept"); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(reading.sequence, 0); - let mut cloned = reading.clone(); - track.abort(Error::Dropped).unwrap(); - - for sequence in 1..=MAX_SEGMENTS as u64 { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert_eq!(recv(&mut sub), sequence); - track.abort(Error::Dropped).unwrap(); - } - assert!( - producer.state.read().pruned.is_some(), - "the first segment should be pruned" - ); - - // Both readers still deliver, then the group ends cleanly at what the - // pruned route produced. - for reader in [&mut reading, &mut cloned] { - assert_eq!(read(reader), b"kept"); - assert!( - reader.read_frame().now_or_never().unwrap().unwrap().is_none(), - "the group ends at what the route produced" - ); - } - } - - #[tokio::test(start_paused = true)] - async fn group_only_read_requests_its_continuation() { - group_only_continuation(false).await; - } - - #[tokio::test(start_paused = true)] - async fn group_only_finish_requests_its_continuation() { - group_only_continuation(true).await; - } - - async fn group_only_continuation(finish: bool) { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"head".to_vec()).unwrap(); - // `finished` probes the seam past a bounded copy; a read waits past a dead one. - if finish { - producer.takeover(&consumer_b).unwrap(); - } - let mut reading = sub.recv_group().await.unwrap().unwrap(); - // The parent subscriber never asks B for anything. - drop(sub); - let parked = |reading: &mut group::Consumer| match finish { - true => reading.finished().now_or_never().is_none(), - false => reading.read_frame().now_or_never().is_none(), - }; - if !finish { - assert_eq!(read(&mut reading), b"head"); - producer.takeover(&consumer_b).unwrap(); - track_a.abort(Error::Dropped).unwrap(); - } - assert!(parked(&mut reading)); - // Demand starts at the frame the reader needs: its cursor, or the seam past A's copy. - assert_eq!( - track_b.subscription().expect("group-only demand").start, - Some(Position { group: 0, frame: 1 }) - ); - - // The header arriving does not end the wait: the tail still needs the demand. - let mut copy = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - copy.write_frame(Timestamp::ZERO, b"head".to_vec()).unwrap(); - assert!(parked(&mut reading)); - assert!(track_b.subscription().is_some(), "the tail still needs demand"); - - copy.write_frame(Timestamp::ZERO, b"tail".to_vec()).unwrap(); - copy.finish().unwrap(); - if finish { - reading.finished().await.unwrap(); - assert_eq!(read(&mut reading), b"head"); - } - assert_eq!(read(&mut reading), b"tail"); - assert!(reading.read_frame().await.unwrap().is_none()); - assert!(track_b.subscription().is_none(), "the finished group releases demand"); - } - - /// A replacement cached before the reader first looks still needs demand for its - /// tail, and a clone keeps that demand after the original reader is dropped. - #[tokio::test(start_paused = true)] - async fn cached_replacement_holds_demand_across_clones() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"head".to_vec()).unwrap(); - let mut reading = sub.recv_group().await.unwrap().unwrap(); - drop(sub); - assert_eq!(read(&mut reading), b"head"); - - producer.takeover(&consumer_b).unwrap(); - track_a.abort(Error::Dropped).unwrap(); - let mut copy = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - copy.write_frame(Timestamp::ZERO, b"head".to_vec()).unwrap(); - assert!(reading.read_frame().now_or_never().is_none()); - assert!(track_b.subscription().is_some(), "a cached header still needs demand"); - - let mut clone = reading.clone(); - drop(reading); - assert!(track_b.subscription().is_some(), "the clone keeps its own demand"); - - copy.write_frame(Timestamp::ZERO, b"tail".to_vec()).unwrap(); - copy.finish().unwrap(); - assert_eq!(read(&mut clone), b"tail"); - assert!(clone.read_frame().await.unwrap().is_none()); - assert!(track_b.subscription().is_none(), "the finished group releases demand"); - } - - /// A reader that stops at its cap may be retained, so the cap releases the demand - /// on the replacement rather than the drop. - #[tokio::test(start_paused = true)] - async fn capped_read_releases_replacement_demand() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"head".to_vec()).unwrap(); - let mut reading = sub.recv_group().await.unwrap().unwrap(); - drop(sub); - assert_eq!(read(&mut reading), b"head"); - - producer.takeover(&consumer_b).unwrap(); - track_a.abort(Error::Dropped).unwrap(); - let mut copy = track_b.create_group(group::Info { sequence: 0 }).unwrap(); - copy.write_frame(Timestamp::ZERO, b"head".to_vec()).unwrap(); - copy.write_frame(Timestamp::ZERO, b"mid".to_vec()).unwrap(); - reading.set_frames(..2); - assert_eq!(read(&mut reading), b"mid"); - assert!(track_b.subscription().is_some(), "the read still holds demand"); - - assert!(reading.read_frame().now_or_never().unwrap().unwrap().is_none()); - assert!(track_b.subscription().is_none(), "the cap releases demand"); - } - - /// `finished()` fails when the seam's covering route skip-declared the group: - /// its segment geometrically covers the continuation, but its SUBSCRIBE_START - /// floor proves the group will never arrive, so the group is truncated at the - /// cap. Polled without draining, and woken by the successor's track (the seam - /// probe parks on the peek), not just the segment list. - #[tokio::test] - async fn finished_fails_when_the_successor_skips_the_seam() { - let (track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // A serves two frames of an open group; B splices at the seam. - let mut group = track_a.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"f0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"f1".to_vec()).unwrap(); - producer.takeover(&consumer_b).unwrap(); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - drop(group); - track_a.abort(Error::Dropped).unwrap(); - - // B covers the seam, so the count is still open. - assert!(reading.finished().now_or_never().is_none(), "the seam is coverable"); - - // B declares it starts at group 1 and produces it: group 0's continuation - // is skipped for good, so the group was cut short. - track_b.start_at(1).unwrap(); - write_group(&mut track_b, 1, "b1"); - assert_eq!(recv(&mut sub), 1); - let err = reading - .finished() - .now_or_never() - .expect("a skip-declared seam must resolve") - .unwrap_err(); - assert!(matches!(err, Error::NotFound), "{err:?}"); - } - - /// A reader that already latched a pruned segment's copy keeps draining it: the - /// cursor holds the buffered frames, and a pruned segment produced everything it - /// owned, so the copy runs out exactly at the boundary. - #[tokio::test] - async fn reader_drains_a_pruned_segments_copy() { - let mut producer = Producer::new(); - let mut sub = producer.consume().subscribe(None); - - // Group 0 has two frames; the reader consumes one, latching the copy. - let (track, consumer) = track_pair("t0"); - producer.takeover(&consumer).unwrap(); - let mut group = track.create_group(group::Info { sequence: 0 }).unwrap(); - group.write_frame(Timestamp::ZERO, b"f0".to_vec()).unwrap(); - group.write_frame(Timestamp::ZERO, b"f1".to_vec()).unwrap(); - group.finish().unwrap(); - let mut reading = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); - assert_eq!(read(&mut reading), b"f0"); - track.abort(Error::Dropped).unwrap(); - - // Failovers prune the segment out from under the latched cursor. - for sequence in 1..=MAX_SEGMENTS as u64 { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "payload"); - assert_eq!(recv(&mut sub), sequence); - track.abort(Error::Dropped).unwrap(); - } - assert!( - producer.state.read().pruned.is_some(), - "the first segment should be pruned" - ); - - // The second frame still arrives from the latched copy. - assert_eq!(read(&mut reading), b"f1"); - } - - #[tokio::test] - async fn switch_replaces_a_run_of_empty_segments() { - let (mut track_a, consumer_a) = track_pair("a"); - let (_track_b, consumer_b) = track_pair("b"); - let (_track_c, consumer_c) = track_pair("c"); - let (mut track_d, consumer_d) = track_pair("d"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - - // B and C splice in but die before producing anything. - producer.switch(&consumer_b, Position::group(1)).unwrap(); - producer.switch(&consumer_c, Position::group(2)).unwrap(); - - // D's boundary covers both empty segments: one switch replaces the run, - // and the group they never produced is D's to serve. - producer.switch(&consumer_d, Position::group(1)).unwrap(); - write_group(&mut track_d, 1, "d1"); - assert_eq!(recv(&mut sub), 1); - assert_eq!(producer.state.read().segments.len(), 2); - } - - #[tokio::test] - async fn abort_drains_before_erroring() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - write_group(&mut track_a, 0, "a0"); - producer.abort(Error::Cancel).unwrap(); - - // Buffered groups drain first; then the abort surfaces. Waiting on the - // end sees the abort immediately (it consumes nothing). - assert!(matches!(sub.finished().now_or_never().unwrap(), Err(Error::Cancel))); - assert_eq!(recv(&mut sub), 0); - assert!(matches!(sub.recv_group().now_or_never().unwrap(), Err(Error::Cancel))); - } - - #[tokio::test] - async fn terminal_states_are_exclusive() { - let (_track_a, consumer_a) = track_pair("a"); - let (_track_b, consumer_b) = track_pair("b"); - - // A finished track accepts no further transitions: a late abort (route - // churn re-queueing a completed track) must not error draining readers. - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - producer.finish().unwrap(); - assert!(matches!(producer.abort(Error::Cancel), Err(Error::Closed))); - assert!(matches!(producer.finish(), Err(Error::Closed))); - assert!(matches!( - producer.switch(&consumer_b, Position::group(1)), - Err(Error::Closed) - )); - assert!(matches!(producer.takeover(&consumer_b), Err(Error::Closed))); - assert!(matches!(producer.release(), Err(Error::Closed))); - - // An aborted track is just as terminal. - let mut producer = Producer::new(); - producer.abort(Error::Cancel).unwrap(); - assert!(matches!(producer.finish(), Err(Error::Closed))); - assert!(matches!(producer.abort(Error::Cancel), Err(Error::Closed))); - assert!(matches!(producer.takeover(&consumer_b), Err(Error::Closed))); - } - - #[tokio::test] - async fn dropped_producer_errors_once_drained() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - - // The segment dies, then the producer goes away without finish/abort: - // no takeover can ever come, so stalling would hang forever. The track - // ends with the segment's own error. - track_a.abort(Error::Timeout).unwrap(); - drop(producer); - - let result = sub.recv_group().now_or_never().expect("must not stall forever"); - assert!(matches!(result, Err(Error::Timeout))); - } - - /// A finished logical track ends as its final segment did: a segment cut off - /// after the finish (its session died with a group still in flight) is an - /// error, not a clean end. - #[tokio::test] - async fn finished_producer_ends_with_a_dead_final_segment() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - - producer.finish().unwrap(); - track_a.abort(Error::Timeout).unwrap(); - - let result = sub.recv_group().now_or_never().expect("must not stall forever"); - assert!(matches!(result, Err(Error::Timeout))); - } - - /// A datagram-only reader of a finished logical track ends with the final - /// segment's error. The datagram poll drops a segment error that is not - /// `Ok(Some)`, so the finished path has to ask the segment itself. - #[tokio::test] - async fn finished_producer_ends_datagrams_with_a_dead_final_segment() { - let (track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - assert!( - kio::wait(|waiter| sub.poll_recv_datagram(waiter)) - .now_or_never() - .is_none(), - "no datagram yet" - ); - - producer.finish().unwrap(); - track_a.abort(Error::Timeout).unwrap(); - - let result = kio::wait(|waiter| sub.poll_recv_datagram(waiter)) - .now_or_never() - .expect("must not stall forever"); - assert!(matches!(result, Err(Error::Timeout))); - } - - #[tokio::test] - async fn dropped_producer_keeps_a_live_segment_serving() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - drop(producer); - - // The frozen state still has a live segment: groups keep flowing. - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - recv_pending(&mut sub); - - // The segment finishing ends the logical track cleanly: that is where - // the content ended, and no takeover can come. - track_a.finish().unwrap(); - let result = sub.recv_group().now_or_never().expect("must not stall forever"); - assert!(matches!(result, Ok(None))); - } - - #[tokio::test] - async fn dropped_producer_ends_finished_waiters_with_the_segment() { - for clean in [true, false] { - let (track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - recv_pending(&mut sub); - - // Once dropped, the producer can never finish the logical track: an - // end-waiter waits on the live segment and ends as it does. - drop(producer); - assert!( - sub.finished().now_or_never().is_none(), - "ended while the segment is live" - ); - match clean { - true => track_a.finish().unwrap(), - false => track_a.abort(Error::Timeout).unwrap(), - } - let result = sub.finished().now_or_never().expect("must not stall forever"); - match clean { - true => assert!(matches!(result, Ok(0))), - false => assert!(matches!(result, Err(Error::Timeout))), - } - } - } - - /// Waiting for the end, or polling datagrams, must not consume the final - /// segment: groups still queued in its cursor are delivered afterwards, for a - /// finished producer and a dropped one alike. - #[tokio::test] - async fn end_waiters_leave_queued_groups_readable() { - for dropped in [false, true] { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(replay()); - - write_group(&mut track_a, 0, "a0"); - track_a.finish().unwrap(); - match dropped { - true => drop(producer), - false => producer.finish().unwrap(), - } - - let count = sub.finished().now_or_never().expect("the end is known"); - assert!(matches!(count, Ok(1)), "{count:?}"); - let datagram = kio::wait(|waiter| sub.poll_recv_datagram(waiter)).now_or_never(); - assert!(matches!(datagram, Some(Ok(None))), "{datagram:?}"); - - assert_eq!(recv(&mut sub), 0, "dropped={dropped}"); - let end = sub.recv_group().now_or_never().expect("must not stall forever"); - assert!(matches!(end, Ok(None)), "dropped={dropped}"); - } - } - - #[tokio::test] - async fn dropped_producer_ends_datagrams() { - let (track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - // The segment is terminal and the producer is gone: a datagram-only - // poller must observe the drop, not park forever. - drop(track_a); - drop(producer); - let result = kio::wait(|waiter| sub.poll_recv_datagram(waiter)) - .now_or_never() - .expect("must not stall forever"); - assert!(matches!(result, Err(Error::Dropped))); - } - - #[tokio::test] - async fn evicted_parked_group_wakes_the_clean_end() { - use std::task::Context; - - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - sub.end_at(..1); - write_group(&mut track_a, 0, "a0"); - let straggler = track_a.create_group(group::Info { sequence: 1 }).unwrap(); - assert_eq!(recv(&mut sub), 0); - recv_pending(&mut sub); // the straggler parks beyond the cap - - track_a.finish().unwrap(); - producer.finish().unwrap(); - - let (counter, waker) = CountWaker::new(); - let mut cx = Context::from_waker(&waker); - let mut fut = std::pin::pin!(sub.recv_group()); - assert!( - fut.as_mut().poll(&mut cx).is_pending(), - "the parked group holds the end open" - ); - - // Eviction aborts the parked group behind the subscriber's back: it must - // wake and observe the clean end rather than sleeping forever. - straggler.abort(Error::Old).unwrap(); - assert!(counter.count() > 0, "the eviction wakeup was lost"); - let result = fut.as_mut().poll(&mut cx); - assert!(matches!(result, Poll::Ready(Ok(None)))); - } - - /// The counterpart of [`evicted_parked_group_wakes_the_clean_end`] with the - /// terminal states already in place when the straggler is first observed: - /// the poll that parks it is the same poll that sees the segment finish, so - /// the group must be watched from the moment it parks, not from the next - /// poll (which nothing would trigger). - #[tokio::test] - async fn straggler_parked_after_finish_still_wakes() { - use std::task::Context; - - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - sub.end_at(..1); - write_group(&mut track_a, 0, "a0"); - assert_eq!(recv(&mut sub), 0); - - let straggler = track_a.create_group(group::Info { sequence: 1 }).unwrap(); - track_a.finish().unwrap(); - producer.finish().unwrap(); - - let (counter, waker) = CountWaker::new(); - let mut cx = Context::from_waker(&waker); - let mut fut = std::pin::pin!(sub.recv_group()); - assert!( - fut.as_mut().poll(&mut cx).is_pending(), - "the parked group holds the end open" - ); - - straggler.abort(Error::Old).unwrap(); - assert!(counter.count() > 0, "the abort wakeup was lost"); - assert!(matches!(fut.as_mut().poll(&mut cx), Poll::Ready(Ok(None)))); - } - - #[tokio::test] - async fn release_restarts_unbounded() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.takeover(&consumer_a).unwrap(); - write_group(&mut track_a, 7, "a7"); - assert!(producer.is_spliced()); - - // Nobody is reading: release the segments, keeping the track alive. - producer.release().unwrap(); - assert!(!producer.is_spliced()); - assert_eq!(producer.resume_position(), None); - - // The next takeover starts unbounded: a fresh reader gets the live edge - // even below the old numbering (the source may have restarted). - producer.takeover(&consumer_b).unwrap(); - let mut sub = producer.consume().subscribe(None); - recv_pending(&mut sub); - assert_eq!(track_b.subscription().unwrap().start, None); - write_group(&mut track_b, 2, "b2"); - assert_eq!(recv(&mut sub), 2); - } - - #[tokio::test] - async fn release_resets_the_pruned_floor() { - let mut producer = Producer::new(); - - // Enough takeovers that the front segments prune, leaving a floor. - let count = 2 * MAX_SEGMENTS as u64; - let mut tracks = Vec::new(); - for sequence in 0..count { - let (mut track, consumer) = track_pair("t"); - producer.takeover(&consumer).unwrap(); - write_group(&mut track, sequence, "g"); - tracks.push(track); - } - producer.release().unwrap(); - - // The old numbering went with the segments: a restarted source's low - // sequences must not be filtered by the stale floor. - let (mut track, consumer) = track_pair("fresh"); - producer.takeover(&consumer).unwrap(); - let mut sub = producer.consume().subscribe(None); - write_group(&mut track, 0, "g0"); - assert_eq!(recv(&mut sub), 0); - } - - #[tokio::test] - async fn fetch_fails_when_the_producer_dies_segmentless() { - let producer = Producer::new(); - let consumer = producer.consume(); - - // No segment yet: the fetch waits for a route to serve the track. - let fetch = consumer.fetch_group(0, None); - let mut fetch = std::pin::pin!(fetch); - assert!(futures::poll!(fetch.as_mut()).is_pending(), "fetch should wait"); - - // The producer dies without one ever arriving: nothing can serve it. - drop(producer); - assert!(matches!(fetch.await, Err(Error::NotFound))); - } - - #[tokio::test] - async fn info_fails_when_the_producer_dies_segmentless() { - let producer = Producer::new(); - let consumer = producer.consume(); - - let info = consumer.info(); - let mut info = std::pin::pin!(info); - assert!(futures::poll!(info.as_mut()).is_pending(), "info should wait"); - - drop(producer); - assert!(matches!(info.await, Err(Error::Dropped))); - } - - #[tokio::test] - async fn start_at_drops_parked_groups_below_the_floor() { - let (mut track_a, consumer_a) = track_pair("a"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - - sub.end_at(..1); - write_group(&mut track_a, 0, "a0"); - write_group(&mut track_a, 1, "a1"); - write_group(&mut track_a, 2, "a2"); - assert_eq!(recv(&mut sub), 0); - recv_pending(&mut sub); // groups 1 and 2 park beyond the cap - - // The reader skips ahead: the parked range below the floor is dropped, - // while the parked group at the floor is still re-offered. - sub.start_at(2); - sub.end_at(..); - assert_eq!(recv(&mut sub), 2, "group 1 was overtaken by start_at"); - recv_pending(&mut sub); - } - - #[tokio::test] - async fn demand_intersects_subscriber_end_with_boundary() { - let (track_a, consumer_a) = track_pair("a"); - let (track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe( - Subscription::default() - .with_start(Position::group(0)) - .with_end(Position::after_group(3)), - ); - recv_pending(&mut sub); - assert_eq!(track_a.subscription().unwrap().end, Position::after_group(3)); - - producer.switch(&consumer_b, Position::group(2)).unwrap(); - recv_pending(&mut sub); - - // The old segment's demand caps at whichever end is tighter (here the - // boundary); the new one keeps the subscriber's own end. - assert_eq!(track_a.subscription().unwrap().end, Some(Position::group(2))); - assert_eq!(track_b.subscription().unwrap().end, Position::after_group(3)); - } - - #[tokio::test] - async fn subscribers_read_independently() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let consumer = producer.consume(); - let mut sub1 = consumer.subscribe(replay()); - let mut sub2 = consumer.subscribe(replay()); - recv_pending(&mut sub1); - recv_pending(&mut sub2); - - write_group(&mut track_a, 0, "a0"); - producer.switch(&consumer_b, Position::group(1)).unwrap(); - write_group(&mut track_b, 1, "b1"); - - // Each subscriber holds its own cursors over the shared segments. - assert_eq!(recv(&mut sub1), 0); - assert_eq!(recv(&mut sub1), 1); - assert_eq!(recv(&mut sub2), 0); - assert_eq!(recv(&mut sub2), 1); - assert!(sub1.is_clone(&sub2)); - } - - #[tokio::test] - async fn latest_clamps_to_segment_bounds() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let consumer = producer.consume(); - assert_eq!(consumer.latest(), None); - - write_group(&mut track_a, 0, "a0"); - assert_eq!(consumer.latest(), Some(0)); - - producer.switch(&consumer_b, Position::group(2)).unwrap(); - - // The old route races past its cap: the logical edge stays clamped to - // the segment's range (its owed range is settled once the track's edge - // passes the cap, even if some groups in it never arrived). - write_group(&mut track_a, 5, "a5"); - assert_eq!(consumer.latest(), Some(1)); - - // A below-boundary group on the new route (unbounded demand racing the - // splice) doesn't drag the edge backwards either. - write_group(&mut track_b, 0, "b0"); - assert_eq!(consumer.latest(), Some(1)); - - write_group(&mut track_b, 3, "b3"); - assert_eq!(consumer.latest(), Some(3)); - } - - #[tokio::test] - async fn datagrams_come_from_the_newest_segment() { - let (mut track_a, consumer_a) = track_pair("a"); - let (mut track_b, consumer_b) = track_pair("b"); - - let mut producer = Producer::new(); - producer.switch(&consumer_a, None).unwrap(); - let mut sub = producer.consume().subscribe(None); - assert!( - kio::wait(|waiter| sub.poll_recv_datagram(waiter)) - .now_or_never() - .is_none(), - "no datagram yet" - ); - - producer.switch(&consumer_b, Position::group(1)).unwrap(); - - // Datagrams are a live best-effort channel: one from a replaced segment - // is stale and never surfaces, only the live route's flow does. - track_a.append_datagram(Timestamp::ZERO, b"old".as_ref()).unwrap(); - assert!( - kio::wait(|waiter| sub.poll_recv_datagram(waiter)) - .now_or_never() - .is_none(), - "stale datagram must not surface" + walk( + &mut Vec::new(), + &[ + Event::SwitchToB, + Event::ReturnToA, + Event::FinishA, + Event::FinishB, + Event::Update, + Event::Cancel, + ], ); - track_b.append_datagram(Timestamp::ZERO, b"new".as_ref()).unwrap(); - let datagram = kio::wait(|waiter| sub.poll_recv_datagram(waiter)) - .now_or_never() - .expect("datagram should be ready") - .expect("should not error") - .expect("track should not be finished"); - assert_eq!(&datagram.payload[..], b"new"); } } diff --git a/rs/moq-net/src/model/subscription.rs b/rs/moq-net/src/model/subscription.rs index 0c814da566a..cec094b20e8 100644 --- a/rs/moq-net/src/model/subscription.rs +++ b/rs/moq-net/src/model/subscription.rs @@ -306,16 +306,7 @@ pub(super) fn before_end(sequence: u64, end: Option) -> bool { // family silently narrows or widens what the publisher sends, so the suffix, not the // `min`/`max`, is the part to read. -/// The lower of two optional bounds, `None` neutral. Pairs with [`max_some`]. -pub(super) fn min_some(a: Option, b: Option) -> Option { - match (a, b) { - (Some(a), Some(b)) => Some(a.min(b)), - (Some(a), None) | (None, Some(a)) => Some(a), - (None, None) => None, - } -} - -/// The higher of two optional bounds, `None` neutral. Pairs with [`min_some`]. +/// The higher of two optional bounds, `None` neutral. pub(super) fn max_some(a: Option, b: Option) -> Option { match (a, b) { (Some(a), Some(b)) => Some(a.max(b)), diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index ee1da6e8812..09ecb6bb550 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -19,11 +19,10 @@ use crate::{broadcast, cache, group, stats}; use super::{Datagram, Requests}; use super::Cap; -use super::resume::Successor; pub use super::subscription::{Position, Subscription}; use std::{ - collections::{BTreeMap, HashSet, VecDeque}, + collections::{BTreeMap, VecDeque}, ops::{Bound, RangeBounds}, sync::Arc, sync::OnceLock, @@ -65,6 +64,16 @@ pub(super) struct ExpiryScan { gc: bool, } +/// How long a track nobody reads stays, with its cache, before it is let go. +/// +/// Within the window a returning viewer, or the next of a run of back-to-back +/// fetches, finds it: a front its verdicts, a session's copy the groups it cached, +/// without another round trip upstream. Sized above the fetch cadence of a segmented +/// consumer: HLS polls every `TARGETDURATION` seconds, commonly 6 or 10. A lingering +/// track holds no upstream subscription, so waiting longer costs cached state, not a +/// viewer. +pub(crate) const IDLE_LINGER: Duration = Duration::from_secs(30); + /// Publisher-side properties of a track. /// /// These are fixed by the publisher when the track is created and don't change @@ -241,9 +250,21 @@ pub(crate) struct TrackState { // Whether `start_sequence` is only the floor a subscription asked for, still // waiting on the serving session to resolve where the live feed begins (a // lite-06+ SUBSCRIBE_START). Readers that must know the resolved start (see - // [`Consumer::poll_start`]) wait on it; everything else treats the floor as usual. + // [`Subscriber::poll_start`]) wait on it; everything else treats the floor as usual. start_pending: bool, + // Whether the cache reflects the live feed: readers get nothing from it while it does + // not, since how stale it is cannot be told; fetches still do. A track is live from + // creation, and only a session's copy goes idle: when its upstream subscription ends + // with the copy still held, until the route answers again. + live: bool, + // Readers start at this group: the route's live feed went on past a gap after + // everything cached, so nothing bounds how old the cache below it is. + live_floor: Option, + // The newest group cached when the track went idle: what the route's answer is + // judged against, whatever lands before it. + idle_newest: Option, + // Where production stopped, snapshotted when the open groups are released (an // abort, or the last producer dropping). Computed live from the cache otherwise; // see [`Self::resume_position`]. @@ -264,6 +285,10 @@ pub(crate) struct TrackState { // The reverse fetch queue (see [`FetchState`]), same reasoning: cache-miss // `fetch_group` calls enqueue here and a `Dynamic` drains. fetch: kio::Shared, + + // A front's logical track: read straight from its routes' copies rather than this + // cache, which stays empty; see [`super::resume`]. + routes: Option, } /// A cached group plus its bookkeeping in the track's `lookup` map. @@ -351,9 +376,17 @@ impl TrackState { /// Returns the group and its absolute index so the consumer can advance past it. /// The cached producer rather than a consumer: `consume` reads the clock and can /// take the group's own lock, which the caller does once the track guard is gone. + /// Whether readers may take from the cache: it reflects the live feed, or the track + /// ended, which makes the cache the whole of it. + fn readable(&self) -> bool { + self.live || self.sealed || self.final_sequence.is_some() || self.abort.is_some() + } + fn poll_recv_group(&self, index: usize, min_sequence: u64) -> Poll>> { let start = index.saturating_sub(self.offset); - for (i, (sequence, stamp)) in self.arrival.iter().enumerate().skip(start) { + let readable = self.readable(); + let min_sequence = min_sequence.max(self.live_floor.unwrap_or(0)); + for (i, (sequence, stamp)) in self.arrival.iter().enumerate().skip(start).filter(|_| readable) { if *sequence >= min_sequence && let Some(slot) = self.lookup.get(sequence) && slot.stamp == *stamp @@ -380,7 +413,9 @@ impl TrackState { /// resumes at the oldest still-buffered datagram, skipping the lost ones. fn poll_recv_datagram(&self, index: usize) -> Poll>> { let start = index.saturating_sub(self.datagram_offset); - if let Some(datagram) = self.datagrams.get(start) { + if self.readable() + && let Some(datagram) = self.datagrams.get(start) + { return Poll::Ready(Ok(Some((datagram.clone(), self.datagram_offset + start)))); } @@ -443,16 +478,17 @@ impl TrackState { let best = self .lookup - .range(next_sequence..) + .range(next_sequence.max(self.live_floor.unwrap_or(0))..) + .filter(|_| self.readable()) .map(|(_, slot)| &slot.group) .take_while(|group| super::subscription::before_end(group.sequence, end_sequence)) .find(|group| !group.is_aborted()); if let Some(group) = best { - // Deliberately no cache refresh here: this is a pure seek, and the spliced - // merge consults candidates it may not deliver. The deliverer stamps the - // winner; a loser re-seeked every poll must not be shielded from eviction. - // The caller consumes it with the track guard released, like `poll_recv_group`. + // Deliberately no cache refresh here: this is a pure seek, and the caller may + // convict the candidate rather than deliver it. The caller stamps what it + // delivers; a group walked off must not be shielded from eviction. The caller + // consumes it with the track guard released, like `poll_recv_group`. return Poll::Ready(Ok(Some(group.clone()))); } @@ -462,7 +498,7 @@ impl TrackState { // A closed track produces nothing more either, so a gap below its // boundary is the end, not a wait. Cached in-range groups were // returned above, so an aborted track drains what finished first. - // This is the cursor the ordered and spliced readers use. + // This is the cursor the ordered reader uses. if closed || self.final_sequence.is_some_and(|fin| next_sequence >= fin) { return Poll::Ready(end()); } @@ -497,13 +533,13 @@ impl TrackState { /// stamped is servable (an unstamped track drives no expiry at all; the cache's /// own wall-clock policy is what bounds it). /// - /// One scan answers a whole poll. The highest-sequence anchor in range is above + /// One scan answers a whole poll. The highest-sequence edge in range is above /// every candidate below it and above none at or past it. Recomputing per candidate /// would make walking a backlog of N groups off in one poll cost O(N^2). /// /// Capped because a group can only be late relative to data that would actually be - /// served in its place: a subscription ending at a takeover boundary can't jump past - /// it, so the groups it still wants aren't stale just because the route ran on. + /// served in its place: a reader capped by [`Subscriber::set_groups`] can't jump past + /// the cap, so the groups it still wants aren't stale just because the track ran on. /// Fetched backfill is absent from `arrival`, so it cannot age subscription content /// as though it were a live replacement. fn live_edge(&self, cap: Option) -> Option { @@ -526,15 +562,11 @@ impl TrackState { }) } - /// This track's own edge under the exclusive `cap`, for measuring drift. An outer - /// edge and a successor live on other tracks; the caller revalidates those before - /// taking this lock and passes them in, so the locks never nest. - fn drift_edge(&self, cap: Option, outer: Option<(u64, Timestamp)>, successor: Option) -> Edge { + /// This track's own edge under the exclusive `cap`, for measuring drift. + fn drift_edge(&self, cap: Option) -> Edge { Edge { presentation: self.live_edge(cap), - outer, cap, - successor, } } @@ -554,16 +586,10 @@ impl TrackState { /// successor counts: timestamps need not rise with sequence (a rewind reorders them), /// so a later stamped group proves nothing about where an unstamped successor will /// begin, and shrinking the bound is the unsafe direction. An unstamped successor - /// therefore leaves the reach unbounded until it presents its first frame. - /// - /// With no servable successor below `cap`, the reader's next group is past the cap, - /// and `beyond` is where it starts when another track serves it (a splice's next - /// segment; see [`Anchor::successor`]). - fn reach(&self, sequence: u64, cap: Option, beyond: Option) -> Option { - match self.first_servable(sequence.saturating_add(1), cap) { - Some(group) => group.timestamp(), - None => beyond, - } + /// therefore leaves the reach unbounded until it presents its first frame, and so + /// does having no servable successor below `cap` at all. + fn reach(&self, sequence: u64, cap: Option) -> Option { + self.first_servable(sequence.saturating_add(1), cap)?.timestamp() } /// The first servable group in `from..cap`, stamped or not. @@ -599,34 +625,25 @@ impl TrackState { /// /// The edge must sit strictly above the candidate. The live edge is never late /// against itself, and backfill or the tail of a rewound timeline can carry a high - /// timestamp on a low sequence without being an edge at all. Of the edges that - /// qualify, the highest sequence is the newest content, whichever track holds it: - /// this one's, or the `outer` edge a splice pushed from another segment. + /// timestamp on a low sequence without being an edge at all. fn is_stale(&self, sequence: u64, edge: &Edge, budget: Duration) -> bool { - if !self.lookup.contains_key(&sequence) { - return false; - } + self.lookup.contains_key(&sequence) && self.drifted(sequence, edge, budget) + } - // The anchor was resolved under an earlier lock, so confirm it still names - // the same servable incarnation before it convicts a candidate. Failing safe - // (delivering) is right, since the next poll resolves fresh anchors. - let local = edge + /// [`Self::is_stale`] for group `sequence`, whether this track holds it or not: its + /// successor here starts a full `budget` behind the edge. + fn drifted(&self, sequence: u64, edge: &Edge, budget: Duration) -> bool { + // The edge was resolved under an earlier lock, so confirm it still names the + // same servable incarnation before it convicts a candidate. Failing safe + // (delivering) is right, since the next poll resolves a fresh edge. + let Some(live) = edge .presentation - .filter(|live| self.holds(live.sequence, live.stamp)) - .map(|live| (live.sequence, live.timestamp)); - // `outer` was revalidated and `successor` resolved on their own tracks before this - // lock was taken ([`LiveEdge::is_live`], [`Successor::poll_start`]). There is no - // slot for them here, and taking their locks here would nest. - let Some((_, timestamp)) = local - .into_iter() - .chain(edge.outer) - .filter(|(live, _)| *live > sequence) - .max_by_key(|(live, _)| *live) + .filter(|live| live.sequence > sequence && self.holds(live.sequence, live.stamp)) else { return false; }; - self.reach(sequence, edge.cap, edge.successor) - .is_some_and(|reach| matches!(timestamp.checked_sub(reach), Ok(age) if Duration::from(age) >= budget)) + self.reach(sequence, edge.cap) + .is_some_and(|reach| matches!(live.timestamp.checked_sub(reach), Ok(age) if Duration::from(age) >= budget)) } /// Resolve a one-shot fetch from the track side: the cached group, or an [`Error`] @@ -888,6 +905,7 @@ impl TrackState { fn spawn(broadcast: Arc) -> kio::Producer { let state = kio::Producer::new(Self { broadcast: broadcast.clone(), + live: true, ..Default::default() }); let cache = cache::Track::new(broadcast.pool.clone(), state.downgrade()); @@ -1160,7 +1178,7 @@ impl TrackState { group: max, frame: frame as u64, }), - // A copy that wrote nothing has no frames to splice onto, and the reader + // A copy that wrote nothing has no frames to continue, and the reader // already holds its (empty) handle. Pointing a replacement at frame 0 would // hand the same sequence out twice, so roll to the next group exactly as a // finished one does. @@ -1168,6 +1186,54 @@ impl TrackState { } } + /// The upstream subscription ended with the copy still held: what it cached may go + /// stale, so readers get nothing from it until the route answers again. Buffered + /// datagrams go too, since a reader returning later must not be handed them. + fn set_idle(&mut self) { + self.live = false; + self.idle_newest = self + .lookup + .iter() + .rev() + .find(|(_, slot)| slot.visible && !slot.group.is_aborted()) + .map(|(sequence, _)| *sequence); + self.datagram_offset += self.datagrams.len(); + self.datagrams.clear(); + } + + /// The route answered with its largest position (`None` for nothing yet): the cache + /// is current up to there. A feed that went on past a gap after everything cached + /// leaves the cache unjudgeable, since nothing bounds how far an old group reached, so + /// readers skip it; whatever the feed delivers stays. + fn set_live(&mut self, largest: Option) { + if self.live { + return; + } + self.live = true; + let cached = self.idle_newest.take(); + self.live_floor = match (largest, cached) { + (Some(largest), Some(cached)) if largest.group > cached.saturating_add(1) => Some(cached + 1), + // The route has nothing, so whatever is cached is not its feed. + (None, Some(cached)) => Some(cached.saturating_add(1)), + _ => self.live_floor, + }; + } + + /// The newest live group the cache holds, and the number of frames it has so far. + fn newest(&self) -> Option<(u64, u64)> { + let (sequence, slot) = self.lookup.iter().rev().find(|(_, slot)| slot.visible)?; + Some((*sequence, slot.group.frame_count() as u64)) + } + + /// The largest position the cache holds: the newest live group's last frame (its head, + /// while it has none). + fn largest(&self) -> Option { + self.newest().map(|(group, frames)| Position { + group, + frame: frames.saturating_sub(1), + }) + } + fn poll_finished(&self) -> Poll> { if let Some(fin) = self.final_sequence { Poll::Ready(Ok(fin)) @@ -1180,15 +1246,6 @@ impl TrackState { } } - /// Distinguish a locally closed receive track from a declared content end. - fn poll_end(&self) -> Poll>> { - if self.sealed && self.final_sequence.is_none() && self.abort.is_none() { - Poll::Ready(Ok(None)) - } else { - self.poll_finished().map(|res| res.map(Some)) - } - } - fn modify(producer: &kio::Producer) -> Result> { producer.write().map_err(|r| r.abort.clone().unwrap_or(Error::Dropped)) } @@ -1328,24 +1385,6 @@ impl Producer { &self.broadcast } - /// Cache an already-produced group without rewriting its frames. - /// - /// Used by an origin front to keep a warm copy of groups it already delivered - /// after dropping the source track that produced them. - pub(crate) fn adopt_group(&mut self, group: group::Producer, visible: bool) -> Result<()> { - let mut state = self.modify()?; - if let Some(fin) = state.final_sequence - && group.sequence >= fin - { - return Err(Error::Closed); - } - if state.lookup.contains_key(&group.sequence) { - return Err(Error::Duplicate); - } - state.insert_group(&group, visible); - Ok(()) - } - /// Create a new group with the given sequence number. pub fn create_group(&self, group: group::Info) -> Result { let mut state = self.modify()?; @@ -1558,9 +1597,37 @@ impl Producer { Ok(()) } + /// Readers get nothing from the cache until [`Self::set_live`]: the upstream + /// subscription ended with this copy still held, so what it cached may go stale. + pub(crate) fn set_idle(&mut self) { + if let Ok(mut state) = self.modify() { + state.set_idle(); + } + } + + /// The route answered with its largest position, `None` for nothing yet; see + /// [`Self::set_idle`]. A no-op while live. + pub(crate) fn set_live(&mut self, largest: Option) { + if let Ok(mut state) = self.modify() { + state.set_live(largest); + } + } + + /// Whether readers may take from the cache; see [`Self::set_idle`]. + pub(crate) fn is_live(&self) -> bool { + self.state.read().live + } + + /// While idle, the newest group cached when the track went idle: a route asked from + /// its head sends it again, and its first frame says whether the cache is current. + pub(crate) fn idle_newest(&self) -> Option { + let state = self.state.read(); + state.idle_newest.filter(|_| !state.live) + } + /// Declare the floor a subscription asked for while the serving session has yet to /// resolve its start: nothing below `sequence` arrives, exactly as [`Self::start_at`], - /// but [`Consumer::poll_start`] keeps waiting until a later [`Self::start_at`] + /// but [`Subscriber::poll_start`] keeps waiting until a later [`Self::start_at`] /// resolves it. pub(crate) fn request_start(&mut self, sequence: Option) -> Result<()> { self.modify()?.set_start(sequence, true); @@ -1681,7 +1748,7 @@ impl Producer { /// Unlike a wire subscription, the info is already known, so a subscription /// opened from this handle resolves immediately. pub fn consume(&self) -> Consumer { - Consumer::plain(self.name.clone(), self.state.consume()) + Consumer::new(self.name.clone(), self.state.consume()) } /// Subscribing to this in-process track, resolving synchronously. @@ -1701,10 +1768,8 @@ impl Producer { // subscriber surfaces the close/abort on its first read; the preferences are // simply never registered (nothing aggregates them anymore). let info = self.info.clone(); - let min_sequence = floor_of(&preferences); let subscription = kio::Producer::new(preferences); register_subscription(self.state.read(), &subscription); - let drift_anchor = kio::Producer::new(Anchor::default()); // Hoisted: an inline `read()` guard would live to the end of the struct literal, // deadlocking against the `consume()` below. @@ -1713,20 +1778,7 @@ impl Producer { name: self.name.clone(), broadcast, info, - inner: SubscriberKind::Plain(PlainSubscriber { - state: self.state.consume(), - subscription, - min_sequence, - index: 0, - datagram_index: 0, - next_sequence: 0, - end_sequence: None, - parked: BTreeMap::new(), - stale_cap: None, - drift_anchor, - stale: stats::Content::default(), - seek_pending: BTreeMap::new(), - }), + inner: Inner::Plain(Cursor::new(self.state.consume(), subscription)), // A producer-side (in-process) subscribe is not egress: stay untagged. stats: stats::Scope::default(), _stats_sub: stats::Subscription::default(), @@ -2094,24 +2146,6 @@ fn snapshot_subscription(subs: &kio::Shared, bound: Option, outer: Option) -> Option { - super::subscription::min_some(cursor, outer) -} - /// The read cursor's floor: the group the subscription named, or 0 (no floor). /// /// A floor is the only thing a start contributes; [`Subscription::max_age`] is what asks @@ -2185,7 +2219,7 @@ impl TrackWeak { /// decline the teardown, or gets nothing and re-requests the track. It is never /// handed a track that is already on its way out. pub fn try_consume(&self) -> Option { - Some(Consumer::plain(self.name.clone(), self.state.try_consume()?)) + Some(Consumer::new(self.name.clone(), self.state.try_consume()?)) } /// The shared name handle, for use as a broadcast lookup key (clone is a @@ -2223,6 +2257,31 @@ impl TrackWeak { !self.state.is_closed() && self.state.is_used() } + /// The readers' aggregate demand, or `None` while nobody subscribes. + pub(crate) fn subscription(&self) -> Option { + let state = self.state.read(); + let (subs, bound) = (state.subscriptions.clone(), state.max_age_bound()); + drop(state); + snapshot_subscription(&subs, bound) + } + + /// End the track with `err` unless a reader holds it, atomically with lookups: a + /// lookup either gets a consumer in time to keep it, or finds it closed and asks + /// afresh. True once the track is gone. + pub(crate) fn abort_unused(&self, err: Error) -> bool { + let Some(producer) = self.state.produce() else { + return true; + }; + match producer.write_unused() { + kio::Unused::Idle(guard) => { + commit_abort(guard, err); + true + } + kio::Unused::Closed => true, + kio::Unused::Used => false, + } + } + /// Park `waiter` for the next consumer appearing; a no-op once one exists. /// Feeds [`crate::broadcast::Demand`], which recomputes on wake. pub(crate) fn poll_used(&self, waiter: &kio::Waiter) { @@ -2362,10 +2421,6 @@ pub(crate) enum DemandState { /// to the publisher; it just names a track you can [`subscribe`](Self::subscribe) /// to (a live, ongoing stream of groups) later. The same handle can be subscribed /// to multiple times, and clones are cheap. -/// -/// A track reached through a route-fed broadcast is *spliced*: it is backed by one -/// or more per-session tracks joined at group boundaries, and this handle reads -/// across them transparently. #[derive(Clone)] pub struct Consumer { name: Arc, @@ -2373,35 +2428,19 @@ pub struct Consumer { // relative references resolve against. Rebound by `broadcast::Consumer::track` to // that handle's view, which may name the broadcast differently than its producer did. broadcast: Arc, - inner: ConsumerKind, + state: kio::Consumer, // Egress stats scope, set by a tagged [`broadcast::Consumer`] via // [`Self::with_stats`]. Empty (no-op) for an untagged track. stats: stats::Scope, } -#[derive(Clone)] -enum ConsumerKind { - Plain(kio::Consumer), - Spliced(super::resume::Consumer), -} - impl Consumer { - fn plain(name: Arc, state: kio::Consumer) -> Self { + fn new(name: Arc, state: kio::Consumer) -> Self { let broadcast = state.read().broadcast.clone(); Self { name, broadcast, - inner: ConsumerKind::Plain(state), - stats: stats::Scope::default(), - } - } - - /// A consumer over a spliced logical track (a route-fed broadcast's track). - pub(crate) fn spliced(name: Arc, broadcast: Arc, resume: super::resume::Consumer) -> Self { - Self { - name, - broadcast, - inner: ConsumerKind::Spliced(resume), + state, stats: stats::Scope::default(), } } @@ -2421,61 +2460,6 @@ impl Consumer { self } - /// Groups this copy still holds, so an origin can keep them after dropping the source. - /// - /// Publisher-produced groups come back in arrival order, matching what - /// `recv_group` would deliver; fetched backfill absent from arrival follows - /// in sequence order for sequence fetches. - pub(crate) fn cached_groups(&self) -> Vec<(group::Producer, bool)> { - match &self.inner { - ConsumerKind::Plain(state) => { - let state = state.read(); - let mut out = Vec::with_capacity(state.lookup.len()); - for (sequence, stamp) in state.arrival.iter() { - if let Some(slot) = state.lookup.get(sequence) - && slot.stamp == *stamp - && !slot.group.is_aborted() - { - out.push((slot.group.clone(), slot.visible)); - } - } - // Fetched backfill never enters arrival; keep it for sequence fetches. - let mut copied: HashSet = out.iter().map(|(group, _)| group.sequence).collect(); - for (sequence, slot) in state.lookup.iter() { - if !slot.group.is_aborted() && copied.insert(*sequence) { - out.push((slot.group.clone(), slot.visible)); - } - } - out - } - ConsumerKind::Spliced(resume) => resume.cached_groups(), - } - } - - /// A cached group that contains every frame from `frame_start` onward. - /// - /// Unlike [`Self::peek_group`], this is suitable for satisfying a FETCH: a - /// partial cached copy is a miss so the caller can ask upstream for its head. - pub(crate) fn cached_group(&self, sequence: u64, frame_start: u64) -> Option { - match &self.inner { - ConsumerKind::Plain(state) => { - let state = state.read(); - let group = state.covering_group(sequence, frame_start)?; - group.cache_refresh(); - Some(group.consume()) - } - ConsumerKind::Spliced(resume) => resume.cached_group(sequence, frame_start), - } - } - - /// Publisher properties already resolved on this copy, if any. - pub(crate) fn cached_info(&self) -> Option { - match &self.inner { - ConsumerKind::Plain(state) => state.read().info.clone(), - ConsumerKind::Spliced(resume) => resume.cached_info(), - } - } - /// The track name this handle is bound to. pub fn name(&self) -> &str { &self.name @@ -2501,21 +2485,14 @@ impl Consumer { pub fn subscribe(&self, subscription: impl Into>) -> kio::Pending { let subscription = kio::Producer::new(subscription.into().unwrap_or_default()); - let inner = match &self.inner { - ConsumerKind::Plain(state) => { - // Register the subscription if the track is live. If it is already closed, the - // returned future resolves to the abort error via `Subscribing::poll_ok`. - register_subscription(state.read(), &subscription); - SubscribingKind::Plain(state.clone()) - } - // A spliced subscription registers per segment once the subscriber polls. - ConsumerKind::Spliced(resume) => SubscribingKind::Spliced(resume.clone()), - }; + // Register the subscription if the track is live. If it is already closed, the + // returned future resolves to the abort error via `Subscribing::poll_ok`. + register_subscription(self.state.read(), &subscription); kio::Pending::new(Subscribing { name: self.name.clone(), broadcast: self.broadcast.clone(), - inner, + state: self.state.clone(), subscription, stats: self.stats.clone(), }) @@ -2526,14 +2503,6 @@ impl Consumer { /// source that never declares one). Parks while a lite-06+ session still owes its /// SUBSCRIBE_START (see [`Producer::request_start`]); a closed track is ready with /// whatever it last declared, since nothing will resolve it anymore. - pub(crate) fn poll_start(&self, waiter: &kio::Waiter) -> Poll> { - match &self.inner { - ConsumerKind::Plain(state) => Self::poll_state_start(state, waiter), - ConsumerKind::Spliced(_) => Poll::Ready(None), - } - } - - /// [`Self::poll_start`] over one track's state, shared with [`Subscriber::poll_start`]. fn poll_state_start(state: &kio::Consumer, waiter: &kio::Waiter) -> Poll> { let res = state.poll(waiter, |state| match state.start_pending && state.abort.is_none() { true => Poll::Pending, @@ -2549,81 +2518,28 @@ impl Consumer { /// counting as a fetch or a delivery. The IETF publisher snapshots its frame count to /// resolve Largest Object; a group that is not immediately available reads as no edge. pub(crate) fn peek_latest(&self) -> Option { - match &self.inner { - ConsumerKind::Plain(state) => { - let sequence = state.read().max_sequence?; - self.peek_group(sequence) - } - ConsumerKind::Spliced(resume) => resume.peek_latest(), - } - } - - /// The live edge below the exclusive `cap` that drift is measured against; see - /// [`TrackState::live_edge`]. A splice reports the newest across its segments. - pub(crate) fn live_edge(&self, cap: Option) -> Option { - match &self.inner { - ConsumerKind::Plain(state) => { - let edge = state.read().live_edge(cap)?; - Some(LiveEdge { - sequence: edge.sequence, - timestamp: edge.timestamp, - stamp: edge.stamp, - track: state.weak(), - }) - } - ConsumerKind::Spliced(resume) => resume.live_edge(cap), - } - } - - /// Where the first servable group in `from..cap` starts presenting: `None` while no - /// such group is cached, `Some(None)` while it has no frame yet. Registers `waiter` - /// for whatever could move that answer: a group arriving or leaving, or the unstamped - /// one's first frame. A splice answers from the first segment holding one. - pub(crate) fn poll_first_start( - &self, - waiter: &kio::Waiter, - from: u64, - cap: Option, - ) -> Option> { - match &self.inner { - ConsumerKind::Plain(state) => { - let mut start = None; - let _ = state.poll(waiter, |state| { - while let Some(group) = state.first_servable(from, cap) { - // A stamped group can abort without changing its track. Register - // before checking it again, so an abort either wakes or re-resolves. - let _ = group.poll_closed(waiter); - let _ = group.poll_timestamp(waiter); - if !group.is_aborted() { - start = Some(group.timestamp()); - break; - } - } - Poll::<()>::Pending - }); - start - } - ConsumerKind::Spliced(resume) => resume.poll_first_start(waiter, from, cap), + if let Some(serving) = self.serving() { + return serving.peek_latest(); } + let sequence = self.state.read().max_sequence?; + self.peek_group(sequence) } /// The nearest cached group below `sequence`, under the same terms as /// [`Self::peek_group`]. Walks the cache's own order, so gaps in the group numbering /// are crossed and aborted (evicted) entries are skipped. pub(crate) fn peek_before(&self, sequence: u64) -> Option { - match &self.inner { - ConsumerKind::Plain(state) => { - let state = state.read(); - state - .lookup - .range(..sequence) - .rev() - .map(|(_, slot)| &slot.group) - .find(|group| !group.is_aborted()) - .map(|group| group.consume()) - } - ConsumerKind::Spliced(resume) => resume.peek_before(sequence), + if let Some(serving) = self.serving() { + return serving.peek_before(sequence); } + let state = self.state.read(); + state + .lookup + .range(..sequence) + .rev() + .map(|(_, slot)| &slot.group) + .find(|group| !group.is_aborted()) + .map(|group| group.consume()) } /// A cached group by sequence, under the same terms as [`Self::peek_latest`]. Unlike a @@ -2631,139 +2547,61 @@ impl Consumer { /// group alive over one a subscriber actually read; an aborted (evicted) group is a /// miss. pub(crate) fn peek_group(&self, sequence: u64) -> Option { - match &self.inner { - ConsumerKind::Plain(state) => { - let state = state.read(); - let slot = state.lookup.get(&sequence)?; - if slot.group.is_aborted() { - return None; - } - Some(slot.group.consume()) - } - ConsumerKind::Spliced(resume) => resume.peek_group(sequence), + let state = self.state.read(); + let slot = state.lookup.get(&sequence)?; + if slot.group.is_aborted() { + return None; } + Some(slot.group.consume()) } - /// Attach a subscription's drift policy to a cached group resolved outside its cursor. - pub(crate) fn guard_group( - &self, - group: group::Consumer, - subscription: kio::Consumer, - anchor: kio::Consumer, - bound: Option, - ) -> group::Consumer { - let ConsumerKind::Plain(state) = &self.inner else { - return group; - }; - let sequence = group.sequence; - group.with_expiry(Arc::new(GroupExpiry { - state: state.weak(), - subscription, - anchor, - bound, - sequence, - })) - } - - /// Poll for the newest cached group once its sequence passes `after`, as a producer - /// handle: holding one keeps its frames when the track's own producer tears down - /// abruptly, which otherwise releases every unfinished group. Pending once closed. - pub(crate) fn poll_latest_group(&self, after: Option, waiter: &kio::Waiter) -> Poll { - let ConsumerKind::Plain(state) = &self.inner else { - return Poll::Pending; - }; - let res = state.poll(waiter, |state| { - match state - .latest_group - .filter(|latest| after.is_none_or(|after| *latest > after)) + /// Poll for group `sequence` the way the live feed delivers it: `Some` once it is + /// cached, `None` once the feed will not deliver it (it went past it, starts past it, + /// or ended), and pending while it still may. + pub(crate) fn poll_group(&self, sequence: u64, waiter: &kio::Waiter) -> Poll> { + let res = self.state.poll(waiter, |state| { + if let Some(slot) = state.lookup.get(&sequence) + && !slot.group.is_aborted() { - Some(latest) => match state.lookup.get(&latest) { - Some(slot) => Poll::Ready(slot.group.clone()), - None => Poll::Pending, - }, - None => Poll::Pending, + return Poll::Ready(Some(slot.group.consume())); } - }); - match res { - Poll::Ready(Ok(group)) => Poll::Ready(group), - _ => Poll::Pending, - } - } - - /// Poll for a cached group by sequence, parking `waiter` until it lands. - /// - /// `Ready(None)` once it can never arrive: the track ended below the sequence, or - /// closed. Unlike [`Self::fetch_group`] this never asks anyone to produce the group, - /// so it only resolves for one a live subscription is already pulling. That is what - /// the spliced reader in [`super::resume`] wants: it waits for the route serving a - /// group to deliver it, rather than opening a redundant fetch alongside. - pub(crate) fn poll_peek_group(&self, sequence: u64, waiter: &kio::Waiter) -> Poll> { - let ConsumerKind::Plain(state) = &self.inner else { - // A segment's track is never itself spliced. - return Poll::Pending; - }; - - let res = state.poll(waiter, |state| { - match state.lookup.get(&sequence) { - Some(slot) if !slot.group.is_aborted() => Poll::Ready(Some(slot.group.consume())), - // An aborted slot is a hole this route can no longer fill. - Some(_) => Poll::Ready(None), - // Past the declared end, so it can never exist. - None if state.final_sequence.is_some_and(|fin| sequence >= fin) => Poll::Ready(None), - // Below the declared start, so the live feed skipped it for good. - None if state.start_sequence.is_some_and(|start| sequence < start) => Poll::Ready(None), - None => Poll::Pending, + let passed = state.max_sequence.is_some_and(|newest| newest > sequence) + || state.start_sequence.is_some_and(|start| start > sequence) + || state.final_sequence.is_some_and(|fin| fin <= sequence); + match passed { + true => Poll::Ready(None), + false => Poll::Pending, } }); - match res { - Poll::Ready(Ok(res)) => Poll::Ready(res), - // The track died; whatever it cached went with it. + Poll::Ready(Ok(group)) => Poll::Ready(group), Poll::Ready(Err(_)) => Poll::Ready(None), Poll::Pending => Poll::Pending, } } - /// Poll for a live cached copy of `sequence` that can serve frame `index`, - /// parking until one exists. - /// - /// Unlike [`Self::poll_peek_group`] this never gives a verdict: a missing - /// group parks (registered for its arrival) even below the declared start, - /// since demand may move backward and revive it. The spliced reader uses it - /// to reconsider a route it gave up on, so it must be exact about what - /// "available" means: a copy that cannot start at `index` (its head is gone) - /// leaves the route buried rather than reviving it into a peek that would - /// bury it again. That exactness is load-bearing, since the reader consults - /// this ahead of its terminal checks: relaxing it to "the group exists" makes - /// revive and re-bury alternate forever inside one poll - /// (`resume::test::misaligned_copy_is_lost_without_spinning`, where the - /// regression surfaces as a hang). - pub(crate) fn poll_serving_group(&self, sequence: u64, index: u64, waiter: &kio::Waiter) -> Poll<()> { - let ConsumerKind::Plain(state) = &self.inner else { - // A segment's track is never itself spliced. - return Poll::Pending; - }; - let res = state.poll(waiter, |state| match state.lookup.get(&sequence) { - Some(slot) if !slot.group.is_aborted() => { - // `start_at` clamps up, so landing higher means the copy no longer - // holds this position. - let mut group = slot.group.consume(); - group.start_at(index); - match group.index() == index { - true => Poll::Ready(()), - false => Poll::Pending, - } + /// Poll for group `sequence` falling a full `budget` behind this track's live edge, + /// as a reader would judge it, whether the track holds it or not; see + /// `TrackState::is_stale`. Pending for good once the track closes. + pub(crate) fn poll_stale(&self, sequence: u64, budget: Duration, waiter: &kio::Waiter) -> Poll<()> { + let res = self.state.poll(waiter, |state| { + match state.drifted(sequence, &state.drift_edge(None), budget) { + true => Poll::Ready(()), + false => Poll::Pending, } - _ => Poll::Pending, }); match res { Poll::Ready(Ok(())) => Poll::Ready(()), - // The track died; whatever would arrive never will, and the caller's - // terminal checks settle the wait. - Poll::Ready(Err(_)) | Poll::Pending => Poll::Pending, + _ => Poll::Pending, } } + /// The serving route's copy, for a front's logical track; see [`super::resume`]. + fn serving(&self) -> Option { + let routes = self.state.read().routes.clone()?; + routes.serving() + } + /// Fetching a single past group, without holding a live subscription. /// /// Returns a [`kio::Pending`] that resolves to the [`group::Consumer`]: @@ -2778,26 +2616,32 @@ impl Consumer { /// handler request. pub fn fetch_group(&self, sequence: u64, options: impl Into>) -> kio::Pending { let options = options.into().unwrap_or_default(); + let resume = self.state.read().routes.clone(); // One fetch per calling context, counted here (coalesced upstream work is // still one request served). Independent of `subscriptions` and the viewer // refcount. self.stats.fetch(); - let state = match &self.inner { - ConsumerKind::Plain(state) => state, - // Spliced: routed to the newest segment's (plain) track, waiting for a - // segment to exist if no route has served the track yet. - ConsumerKind::Spliced(resume) => { - return kio::Pending::new(Fetching { - inner: FetchingKind::Spliced(resume.fetch_group(sequence, options)), - stats: self.stats.clone(), - }); - } - }; + let state = &self.state; let mut result = None; + // A front's logical track caches nothing: the serving route answers. + if let Some(resume) = resume { + return kio::Pending::new(Fetching { + state: state.clone(), + fetch: state.read().fetch.clone(), + sequence, + frame_start: options.frame_start, + priority: options.priority, + result: None, + hit: None, + stats: self.stats.clone(), + resume: Some(Box::new(resume.fetch_group(sequence, options))), + }); + } + // Queue a request only when the group isn't already resolvable from the track // (cached, aborted, or past-final all resolve through `Fetching::poll` without // a queue entry). @@ -2849,15 +2693,15 @@ impl Consumer { } kio::Pending::new(Fetching { - inner: FetchingKind::Plain { - state: state.clone(), - fetch, - sequence, - frame_start: options.frame_start, - result, - hit, - }, + state: state.clone(), + fetch, + sequence, + frame_start: options.frame_start, + priority: options.priority, + result, + hit, stats: self.stats.clone(), + resume: None, }) } @@ -2869,52 +2713,47 @@ impl Consumer { /// [`Subscriber::info`] is the already-resolved counterpart. pub fn query(&self) -> kio::Pending { kio::Pending::new(Querying { - inner: match &self.inner { - ConsumerKind::Plain(state) => QueryingKind::Plain(state.clone()), - ConsumerKind::Spliced(resume) => QueryingKind::Spliced(resume.clone()), - }, + state: self.state.clone(), }) } /// Return the latest group sequence in the track, or `None` before any group. pub fn latest(&self) -> Option { - match &self.inner { - ConsumerKind::Plain(state) => state.read().max_sequence, - ConsumerKind::Spliced(resume) => resume.latest(), + if let Some(serving) = self.serving() { + return serving.latest(); } + self.state.read().max_sequence } /// The declared exclusive final sequence, or `None` while the track is open ended. - /// - /// A spliced track answers for its newest segment, which is where fetches go. pub(crate) fn final_sequence(&self) -> Option { - match &self.inner { - ConsumerKind::Plain(state) => state.read().final_sequence, - ConsumerKind::Spliced(resume) => resume.final_sequence(), + if let Some(serving) = self.serving() { + return serving.final_sequence(); } + self.state.read().final_sequence } /// The frame-precise point a replacement route should resume from: one past the /// last frame this copy produced. `None` if it produced nothing. /// /// Survives the track aborting, which is when a route change asks. + #[cfg(test)] pub(crate) fn resume_position(&self) -> Option { - match &self.inner { - ConsumerKind::Plain(state) => state.read().resume_position(), - ConsumerKind::Spliced(resume) => resume.resume_position(), - } + self.state.read().resume_position() + } + + /// Poll for the track closing: no producer is left to write it, so nothing more + /// will ever arrive. + pub(crate) fn poll_closed(&self, waiter: &kio::Waiter) -> Poll<()> { + self.state.poll_closed(waiter) } /// Poll for the track reaching a terminal state: `Ok(())` once it is complete /// (the final group was produced), `Err` once it closed or aborted before - /// completing. The origin's dispatcher uses this to tell a track that truly - /// ended from one whose serving route died mid-stream. + /// completing. This tells a track that truly ended from one whose serving route + /// died mid-stream. A session closed locally, on purpose, is [`Error::Closed`]. pub(crate) fn poll_complete(&self, waiter: &kio::Waiter) -> Poll> { - let ConsumerKind::Plain(state) = &self.inner else { - // Spliced tracks are compositions; the dispatcher never monitors one. - return Poll::Pending; - }; - match ready!(state.poll(waiter, |state| { + match ready!(self.state.poll(waiter, |state| { // A local close without a declared end is a dead route, not finished content. if state.final_sequence.is_some() && state.is_complete() { Poll::Ready(()) @@ -2925,7 +2764,11 @@ impl Consumer { Ok(_) => Poll::Ready(Ok(())), // Closed before completing. Read through the returned guard: it holds // the lock, so re-locking the channel here would deadlock. - Err(closed) => Poll::Ready(Err(closed.abort.clone().unwrap_or(Error::Dropped))), + Err(closed) => Poll::Ready(Err(match (&closed.abort, closed.sealed) { + (Some(err), _) => err.clone(), + (None, true) => Error::Closed, + (None, false) => Error::Dropped, + })), } } } @@ -2935,65 +2778,35 @@ impl Consumer { pub struct Subscribing { name: Arc, broadcast: Arc, - inner: SubscribingKind, + state: kio::Consumer, subscription: kio::Producer, stats: stats::Scope, } -enum SubscribingKind { - Plain(kio::Consumer), - Spliced(super::resume::Consumer), -} - impl Subscribing { /// Poll until the peer confirms the subscription, yielding the [`Subscriber`]. /// Errors if the track is aborted or not found. pub fn poll_ok(&self, waiter: &kio::Waiter) -> Poll> { - match &self.inner { - SubscribingKind::Plain(state) => { - // Wait until the track info is available - let info = ready!(state.poll(waiter, |state| state.poll_info())) - .map_err(|e| e.abort.clone().unwrap_or(Error::Dropped))??; - - let drift_anchor = kio::Producer::new(Anchor::default()); - let min_sequence = floor_of(&self.subscription.read()); - Poll::Ready(Ok(Subscriber { - name: self.name.clone(), - broadcast: self.broadcast.clone(), - info, - inner: SubscriberKind::Plain(PlainSubscriber { - state: state.clone(), - subscription: self.subscription.clone(), - min_sequence, - index: 0, - datagram_index: 0, - next_sequence: 0, - end_sequence: None, - parked: BTreeMap::new(), - stale_cap: None, - drift_anchor, - stale: stats::Content::default(), - seek_pending: BTreeMap::new(), - }), - stats: self.stats.clone(), - _stats_sub: self.stats.subscribe(), - })) - } - SubscribingKind::Spliced(resume) => { - // Resolved from the first segment's track. The publisher's max age - // window is applied to each per-session aggregate, not here. - let info = ready!(resume.poll_info(waiter))?; - - Poll::Ready(Ok(Subscriber { - name: self.name.clone(), - broadcast: self.broadcast.clone(), - info, - inner: SubscriberKind::Spliced(Box::new(resume.subscribe_shared(self.subscription.clone()))), - stats: self.stats.clone(), - _stats_sub: self.stats.subscribe(), - })) - } - } + // Wait until the track info is available + let info = ready!(self.state.poll(waiter, |state| state.poll_info())) + .map_err(|e| e.abort.clone().unwrap_or(Error::Dropped))??; + + let resume = self.state.read().routes.clone(); + let inner = match resume { + Some(resume) => Inner::Resume( + Box::new(resume.subscribe(self.subscription.clone())), + self.state.clone(), + ), + None => Inner::Plain(Cursor::new(self.state.clone(), self.subscription.clone())), + }; + Poll::Ready(Ok(Subscriber { + name: self.name.clone(), + broadcast: self.broadcast.clone(), + info, + inner, + stats: self.stats.clone(), + _stats_sub: self.stats.subscribe(), + })) } /// Change the subscription preferences before (or after) it resolves. @@ -3018,26 +2831,16 @@ impl kio::Task for Subscribing { /// The pollable state of a [`Consumer::query`]; awaited via the /// [`kio::Pending`] wrapper. pub struct Querying { - inner: QueryingKind, -} - -enum QueryingKind { - Plain(kio::Consumer), - Spliced(super::resume::Consumer), + state: kio::Consumer, } impl Querying { /// Poll until the track's [`Info`] is known, without subscribing to its groups. pub fn poll_ok(&self, waiter: &kio::Waiter) -> Poll> { - match &self.inner { - QueryingKind::Plain(state) => { - // Wait until the track info is available - let info = ready!(state.poll(waiter, |state| state.poll_info())) - .map_err(|e| e.abort.clone().unwrap_or(Error::Dropped))??; - Poll::Ready(Ok(info)) - } - QueryingKind::Spliced(resume) => resume.poll_info(waiter), - } + // Wait until the track info is available + let info = ready!(self.state.poll(waiter, |state| state.poll_info())) + .map_err(|e| e.abort.clone().unwrap_or(Error::Dropped))??; + Poll::Ready(Ok(info)) } } @@ -3167,51 +2970,52 @@ impl Drop for group::Request { /// [`group::Consumer`] once the group lands in the track's cache (already present, /// or produced after a wire FETCH), or [`Error::NotFound`] if it can never exist. pub struct Fetching { - inner: FetchingKind, + state: kio::Consumer, + fetch: kio::Shared, + sequence: u64, + // This caller's own start, so a cached group that begins above it is a miss + // rather than a short answer. + frame_start: u64, + // This caller's priority, for a fetch that moves to a front's routes. + priority: u8, + // The joined attempt's result channel; `None` on a cache hit, or when no handler + // existed to queue on. + result: Option>, + // The group already cached when the fetch was made, held so it stays wanted. + // Boxed: a group consumer dwarfs the rest of the handle. + hit: Option>, // Egress stats scope, so the resolved group carries a payload meter (and counts // as one delivered group). Empty (no-op) for an untagged track. stats: stats::Scope, -} - -enum FetchingKind { - Plain { - state: kio::Consumer, - fetch: kio::Shared, - sequence: u64, - // This caller's own start, so a cached group that begins above it is a miss - // rather than a short answer. - frame_start: u64, - // The joined attempt's result channel; `None` on a cache hit, or when no handler - // existed to queue on. - result: Option>, - // The group already cached when the fetch was made, held so it stays wanted. - // Boxed: a group consumer dwarfs the rest of the handle. - hit: Option>, - }, - /// A spliced track's fetch: waits for a segment, then fetches from it. - Spliced(kio::Pending), + // A front's logical track: fetched from whichever route serves it. Boxed: rare. + resume: Option>, } impl kio::Task for Fetching { type Output = Result; fn poll(&mut self, waiter: &kio::Waiter) -> Poll { - let (state, fetch, sequence, frame_start, result, hit) = match &mut self.inner { - FetchingKind::Plain { - state, - fetch, - sequence, - frame_start, - result, - hit, - } => (state, fetch, *sequence, *frame_start, result.as_ref(), hit), - FetchingKind::Spliced(spliced) => { - // A fetched group is metered here (once), at the tagged handle: the - // spliced source track it comes from is the origin's own, untagged. - return kio::Task::poll(&mut **spliced, waiter) - .map(|res| res.map(|group| group.with_meter(self.stats.meter()))); - } - }; + // Made before a front accepted its logical track, which caches nothing: the + // serving route answers instead. + if self.resume.is_none() + && let Some(routes) = self.state.read().routes.clone() + { + let options = group::Fetch::default() + .with_priority(self.priority) + .with_frame_start(self.frame_start); + self.resume = Some(Box::new(routes.fetch_group(self.sequence, options))); + } + if let Some(resume) = &mut self.resume { + let group = ready!(kio::Task::poll(&mut **resume, waiter))?; + return Poll::Ready(Ok(group.with_meter(self.stats.meter()))); + } + let (state, fetch, sequence, frame_start, result) = ( + &self.state, + &self.fetch, + self.sequence, + self.frame_start, + self.result.as_ref(), + ); // Hand back a consumer sitting where the caller asked rather than at the group's // own start. Coverage was checked first, so this can only skip frames they excluded. @@ -3219,7 +3023,7 @@ impl kio::Task for Fetching { group.start_at(frame_start); group.with_meter(self.stats.meter()) }; - if let Some(group) = hit { + if let Some(group) = &self.hit { return Poll::Ready(Ok(resolve((**group).clone()))); } @@ -3299,7 +3103,7 @@ pub struct Subscriber { // The broadcast this track belongs to; see [`Self::broadcast`]. broadcast: Arc, info: Info, - inner: SubscriberKind, + inner: Inner, // Egress stats scope, used to meter the groups this subscriber reads. Empty // (no-op) for an untagged track. stats: stats::Scope, @@ -3308,22 +3112,25 @@ pub struct Subscriber { _stats_sub: stats::Subscription, } -enum SubscriberKind { - Plain(PlainSubscriber), - // Boxed: the spliced cursor set dwarfs the plain cursor. - Spliced(Box), +/// How a [`Subscriber`] reads: its track's own cache, or a front's routes. +enum Inner { + Plain(Cursor), + /// A front's logical track, read straight from its routes' copies; see + /// [`super::resume`]. Holds the logical track so it counts as a reader. + /// Boxed: the plain cursor is the hot path. + Resume( + Box, + #[allow(dead_code)] kio::Consumer, + ), } /// One poll's view of how far this subscription may drift: the clamped budget and the /// live edge to measure a candidate group against. Resolved once, then applied to every -/// group that poll considers. `outer` is revalidated and `successor` resolved per -/// candidate, outside this track's lock. -#[derive(Clone)] +/// group that poll considers. +#[derive(Clone, Copy)] struct Drift { budget: Duration, edge: Edge, - outer: Option, - successor: Option, } /// Keeps one handed-out group tied to the subscription whose cursor selected it. @@ -3335,35 +3142,36 @@ struct GroupExpiry { /// the last real subscriber. state: kio::ConsumerWeak, subscription: kio::Consumer, - anchor: kio::Consumer, - bound: Option, + /// The cursor's drift cap; see [`Cursor::drift_cap`]. + cap: kio::Consumer>, sequence: u64, } impl group::Expiry for GroupExpiry { - fn is_expired(&self, waiter: &kio::Waiter) -> bool { - let mut max_age = Duration::default(); - let _ = self.subscription.poll(waiter, |subscription| { - max_age = subscription.max_age; - Poll::<()>::Pending + fn for_track(&self, track: &Consumer) -> Arc { + Arc::new(Self { + state: track.state.weak(), + subscription: self.subscription.clone(), + cap: self.cap.clone(), + sequence: self.sequence, + }) + } + + fn is_expired(&self, max_age: Option, waiter: &kio::Waiter) -> bool { + let max_age = max_age.unwrap_or_else(|| { + let mut max_age = Duration::default(); + let _ = self.subscription.poll(waiter, |subscription| { + max_age = subscription.max_age; + Poll::<()>::Pending + }); + max_age }); - let mut anchor = Anchor::default(); - let _ = self.anchor.poll(waiter, |current| { - anchor = (**current).clone(); + let mut cap = None; + let _ = self.cap.poll(waiter, |current| { + cap = **current; Poll::<()>::Pending }); - let anchor = anchor.capped(self.bound); - let cap = anchor.cap; - // Before this track's lock: both may name another track, and nesting deadlocks. - let outer = anchor - .edge - .filter(LiveEdge::is_live) - .map(|live| (live.sequence, live.timestamp)); - let successor = anchor - .successor - .as_ref() - .and_then(|successor| successor.poll_start(waiter)); let mut expired = false; let _ = self.state.poll(waiter, |state| { @@ -3374,7 +3182,7 @@ impl group::Expiry for GroupExpiry { if let Some(group) = state.first_servable(self.sequence.saturating_add(1), cap) { let _ = group.poll_closed(waiter); } - let edge = state.drift_edge(cap, outer, successor); + let edge = state.drift_edge(cap); expired = state.is_stale(self.sequence, &edge, budget); if expired { break; @@ -3419,7 +3227,7 @@ impl group::Expiry for GroupExpiry { } // Register on track changes even though the current answer is known: a - // newer group can move either live edge while this group read is pending. + // newer group can move the live edge while this group read is pending. Poll::<()>::Pending }); @@ -3433,75 +3241,9 @@ impl group::Expiry for GroupExpiry { struct Edge { /// This track's own edge, revalidated before it convicts anything. presentation: Option, - /// A newer edge on another track, already revalidated: its sequence and the newest - /// frame it had presented. See [`Anchor::edge`]. - outer: Option<(u64, Timestamp)>, /// The cap the edge was resolved under, so per-candidate reach lookups measure /// against the same servable window. cap: Option, - /// Where the next group past `cap` starts, already revalidated. See [`Anchor::successor`]. - successor: Option, -} - -/// How a reader wrapping a cursor bounds its drift anchor from outside: pushed by a -/// splice onto each segment's cursor, and shared with the groups a cursor hands out. -#[derive(Clone, Default, PartialEq)] -pub(crate) struct Anchor { - /// The exclusive sequence cap on what the reader could be handed; see [`servable_cap`]. - pub cap: Option, - /// The newest edge across a splice's segments. Each segment is a separate track - /// that only sees its own groups, so without this a parked segment measures against - /// its own frozen edge while the logical track has moved on. Revalidated on its own - /// track before it convicts anything, since it may be judged long after it was pushed. - pub edge: Option, - /// Where the reader's next group past `cap` starts presenting, when another track - /// serves it (a splice's next segment). The last group below the cap has no - /// successor in its own track, so without this nothing bounds its reach and it is - /// never judged stale. Resolved afresh on every judgment rather than cached, so a - /// group that goes away hands the bound to whichever group serves there next. - pub successor: Option, -} - -impl Anchor { - /// This anchor under a further `cap`. A lower cap drops the successor: it named - /// where the reader continues past the old cap, which is no longer served. - pub fn capped(mut self, cap: Option) -> Self { - let capped = servable_cap(cap, self.cap); - if capped != self.cap { - self.cap = capped; - self.successor = None; - } - self - } -} - -/// The newest stamped group of a track: its sequence and the newest frame it presented, -/// plus enough identity for a reader on another track to revalidate it. -#[derive(Clone)] -pub(crate) struct LiveEdge { - pub sequence: u64, - pub timestamp: Timestamp, - stamp: u32, - track: kio::ConsumerWeak, -} - -impl LiveEdge { - /// Whether the edge still names the same servable group on its own track, the - /// check [`TrackState::is_stale`] runs on a local edge. An eviction or abort since - /// the splice resolved it must not convict anything. Takes that track's lock, so - /// never call it under another's. - fn is_live(&self) -> bool { - self.track.read().holds(self.sequence, self.stamp) - } -} - -impl PartialEq for LiveEdge { - fn eq(&self, other: &Self) -> bool { - self.sequence == other.sequence - && self.timestamp == other.timestamp - && self.stamp == other.stamp - && self.track.same_channel(&other.track) - } } /// The newest servable group that has presented at least one frame. @@ -3509,7 +3251,7 @@ impl PartialEq for LiveEdge { struct PresentationEdge { sequence: u64, /// The slot incarnation this timestamp was read from, so an eviction or a re-served - /// sequence between resolving the anchor and using it is detectable. + /// sequence between resolving the edge and using it is detectable. stamp: u32, /// The newest frame this group has presented, not its first. The candidate side of the /// comparison is an upper bound on what a group could still reach, so the edge side has @@ -3518,8 +3260,8 @@ struct PresentationEdge { timestamp: Timestamp, } -/// The cursor state for a subscription over a single (per-session) track. -struct PlainSubscriber { +/// The read cursor behind a [`Subscriber`]. +struct Cursor { state: kio::Consumer, subscription: kio::Producer, @@ -3543,39 +3285,50 @@ struct PlainSubscriber { /// are parked here instead of dropped). Keyed by sequence so the lowest is /// re-offered first. parked: BTreeMap, - /// A cap imposed by a reader wrapping this cursor (a [`super::resume::Subscriber`] - /// segment), folded into the drift anchor only. Delivery is still bounded by - /// `end_sequence`, which stays unset on a segment so its completion is visible. - stale_cap: Option, - /// Shared effective anchor used by groups after this cursor hands them out. The - /// only copy of the outer edge a wrapping reader pushed (see [`Anchor::edge`]). - drift_anchor: kio::Producer, + /// [`Self::end_sequence`], shared with the groups this cursor hands out so their + /// expiry measures drift against the same servable window after a cap moves. + /// + /// Deliberately not [`Subscription::end`]. That is a *request to the publisher*, + /// folded in with every other subscriber's, and it does not filter this handle (see + /// [`Subscriber`]): another unbounded subscriber widens the aggregate and the groups + /// arrive here anyway. Capping the drift edge with it would pin the live edge at the + /// requested end while delivery ran past it, and everything above would then have + /// nothing newer to be late against. + drift_cap: kio::Producer>, /// Groups the drift budget skipped since the count was last drained. Accumulated - /// here rather than metered in place because the handle that owns the stats scope - /// is the outer [`Subscriber`], which may be reading this cursor through a - /// [`super::resume::Subscriber`] segment (untagged, so the outer wrapper is the - /// only place attribution happens once). + /// here rather than metered in place because the stats scope lives on the owning + /// [`Subscriber`], which drains this after every read. stale: stats::Content, /// Groups the seek path has convicted but whose sequences no caller has committed /// past yet, keyed by sequence; see [`Self::commit_seek_stale`]. seek_pending: BTreeMap, } -impl PlainSubscriber { - /// The drift anchor for a read bounded by `end`. Every read folds `end_sequence` - /// into `end`, so capping the shared anchor (which already holds it) is exact. - fn anchor(&self, end: Option) -> Anchor { - self.drift_anchor.read().clone().capped(end) +impl Cursor { + fn new(state: kio::Consumer, subscription: kio::Producer) -> Self { + let min_sequence = floor_of(&subscription.read()); + Self { + state, + subscription, + min_sequence, + index: 0, + datagram_index: 0, + next_sequence: 0, + end_sequence: None, + parked: BTreeMap::new(), + drift_cap: kio::Producer::new(None), + stale: stats::Content::default(), + seek_pending: BTreeMap::new(), + } } - /// Publish the `outer` anchor under this cursor's own cap. - fn update_drift_anchor(&mut self, outer: Anchor) { - let anchor = outer.capped(self.end_sequence); + /// Publish [`Self::end_sequence`] to the groups already handed out. + fn update_drift_cap(&mut self) { // Skip a no-op write: every handed-out group's expiry watches this channel. - if *self.drift_anchor.read() != anchor - && let Ok(mut current) = self.drift_anchor.write() + if *self.drift_cap.read() != self.end_sequence + && let Ok(mut current) = self.drift_cap.write() { - *current = anchor; + *current = self.end_sequence; } } @@ -3596,68 +3349,35 @@ impl PlainSubscriber { std::mem::take(&mut self.stale) } - /// Note a group an outer reader skipped on this cursor's behalf, so it lands in the - /// same counter as the ones skipped here. - fn note_stale(&mut self, group: &group::Consumer) { - self.stale.add(group.content()); - } - /// This subscriber's clamped drift budget and the live edge to measure against, /// resolved once per poll. /// - /// `cap` bounds the anchor: the caller passes the same window it reads from, so a + /// `cap` bounds the edge: the caller passes the same window it reads from, so a /// group is only ever judged against content that could actually be served in its /// place. Read fresh each poll, so a mid-stream [`Control::update`] applies /// to the very next group, and shared across every candidate that poll walks off, so /// discarding a backlog of N groups costs one scan rather than N. Only ever /// [`Poll::Ready`]; the track ending surfaces as the error the caller was going to /// get anyway. - fn poll_drift(&self, anchor: Anchor, waiter: &kio::Waiter) -> Poll> { + fn poll_drift(&self, cap: Option, waiter: &kio::Waiter) -> Poll> { let mut max_age = Duration::default(); let _ = self.subscription.poll(waiter, |subscription| { max_age = subscription.max_age; Poll::<()>::Pending }); - let cap = anchor.cap; - let outer = anchor.edge; - let successor = anchor.successor; self.poll(waiter, |state| { - // Local edge only. The pushed edge and successor are revalidated in - // [`Self::poll_stale`], outside this lock. Poll::Ready(Ok(Drift { budget: clamp_max_age(max_age, state.max_age_bound()), - edge: state.drift_edge(cap, None, None), - outer: outer.clone(), - successor: successor.clone(), + edge: state.drift_edge(cap), })) }) } /// Whether the drift budget says to skip `group`, against a [`Drift`] already resolved /// for this poll. - fn poll_stale(&self, group: &group::Consumer, drift: &Drift, waiter: &kio::Waiter) -> Poll> { - // Revalidate before this track's lock. Both can name another track, including - // one whose own judgment is waiting on this one. - let outer = drift - .outer - .as_ref() - .filter(|live| live.is_live()) - .map(|live| (live.sequence, live.timestamp)); - let successor = drift - .successor - .as_ref() - .and_then(|successor| successor.poll_start(waiter)); - let presentation = drift.edge.presentation; - let cap = drift.edge.cap; - let budget = drift.budget; - self.poll(waiter, move |state| { - let edge = Edge { - presentation, - outer, - cap, - successor, - }; - Poll::Ready(Ok(state.is_stale(group.sequence, &edge, budget))) + fn poll_stale(&self, group: &group::Consumer, drift: Drift, waiter: &kio::Waiter) -> Poll> { + self.poll(waiter, |state| { + Poll::Ready(Ok(state.is_stale(group.sequence, &drift.edge, drift.budget))) }) } @@ -3666,8 +3386,7 @@ impl PlainSubscriber { group.with_expiry(Arc::new(GroupExpiry { state: self.state.weak(), subscription: self.subscription.consume(), - anchor: self.drift_anchor.consume(), - bound: None, + cap: self.drift_cap.consume(), sequence, })) } @@ -3694,7 +3413,7 @@ impl PlainSubscriber { .retain(|sequence, group| *sequence >= min_sequence && watch(group)); // One scan for the whole poll, so walking a backlog off stays linear in its size. - let drift = ready!(self.poll_drift(self.anchor(self.end_sequence), waiter))?; + let drift = ready!(self.poll_drift(self.end_sequence, waiter))?; loop { // Re-offer the lowest parked group back inside the cap once it rises, @@ -3740,7 +3459,7 @@ impl PlainSubscriber { // Drop a group the drift budget has given up on and keep scanning, so one // poll walks a whole backlog off rather than handing it out group by group. - if ready!(self.poll_stale(&consumer, &drift, waiter))? { + if ready!(self.poll_stale(&consumer, drift, waiter))? { self.stale.add(consumer.content()); continue; } @@ -3759,61 +3478,31 @@ impl PlainSubscriber { Poll::Ready(Ok(Some(datagram))) } + /// The lowest servable group past the last one returned, walking off everything the + /// drift budget has convicted on the way. fn poll_next_group(&mut self, waiter: &kio::Waiter) -> Poll>> { - let floor = self.next_sequence.max(self.min_sequence); - let Some(group) = ready!(self.poll_seek_group(floor, self.end_sequence, waiter))? else { - return Poll::Ready(Ok(None)); - }; - self.next_sequence = group.sequence.saturating_add(1); - // The delivery commits everything the seek stepped over to reach this group. - self.commit_seek_stale(self.next_sequence); - // Delivery is a cache access, same as the arrival-order path. - group.cache_refresh(); - Poll::Ready(Ok(Some(group))) - } - - /// Seek the lowest servable group in `floor..end` without advancing any cursor, - /// walking off everything the drift budget has convicted on the way. - /// - /// The drift anchor is built from the same window the seek reads (`end` folded into - /// the cap), so a candidate is only judged against content that could be served in - /// its place. A nested splice depends on this: its outer boundary arrives only as - /// this call's `end`, and an anchor past it would convict groups the caller still - /// owes its reader. - /// - /// Repeated polls return the same group until the caller's own floor passes it, - /// which is the point: the spliced sequence path picks the lowest candidate across - /// segments and must not commit a segment past a group it has not delivered. - fn poll_seek_group( - &mut self, - floor: u64, - end: Option, - waiter: &kio::Waiter, - ) -> Poll>> { - let mut floor = floor.max(self.min_sequence); - let end = super::subscription::min_some(end, self.end_sequence); + let mut floor = self.next_sequence.max(self.min_sequence); + let end = self.end_sequence; // One scan for the whole poll, so walking a backlog off stays linear in its size. - let drift = ready!(self.poll_drift(self.anchor(end), waiter))?; + let drift = ready!(self.poll_drift(end, waiter))?; - loop { + let group = loop { let Some(producer) = ready!(self.poll(waiter, |state| state.poll_next_in_range(floor, end))?) else { // Deliberately no flush of `seek_pending` here: only a delivery commit // may count a conviction. This `None` can be an artifact of a floor - // that will lower again, and a mid-group boundary can serve a - // convicted sequence through another segment even after this cursor - // retires. A conviction never committed is dropped uncounted with the - // cursor, the same tail bound a retired segment already has. + // that will lower again. A conviction never committed is dropped + // uncounted with the cursor. return Poll::Ready(Ok(None)); }; let group = producer.consume(); // Skip a group the budget has given up on and keep scanning, so one poll // walks a whole backlog off rather than handing it out group by group. - if ready!(self.poll_stale(&group, &drift, waiter))? { - // Not counted yet: a seek advances no cursor, and a conviction is not - // permanent (the budget can widen, the edge can be evicted), so the - // group may still be delivered. Re-snapshot on every re-examination so - // the eventual count reflects the group's latest observed content. + if ready!(self.poll_stale(&group, drift, waiter))? { + // Not counted yet: a conviction is not permanent (the budget can widen, + // the edge can be evicted), so the group may still be delivered. + // Re-snapshot on every re-examination so the eventual count reflects the + // group's latest observed content. self.seek_pending.insert(group.sequence, group.content()); floor = group.sequence.saturating_add(1); continue; @@ -3822,33 +3511,32 @@ impl PlainSubscriber { // A conviction the budget walked back: the group is handed over after all, // so it must never reach the stale count. self.seek_pending.remove(&group.sequence); - return Poll::Ready(Ok(Some(self.with_expiry(group)))); - } + break self.with_expiry(group); + }; + + self.next_sequence = group.sequence.saturating_add(1); + // The delivery commits everything the seek stepped over to reach this group. + self.commit_seek_stale(self.next_sequence); + // Delivery is a cache access, same as the arrival-order path. + group.cache_refresh(); + Poll::Ready(Ok(Some(group))) } - /// Count the seek path's convictions below `committed` into [`Self::stale`], + /// Count the sequence path's convictions below `committed` into [`Self::stale`], /// exactly once each. /// - /// A seek is speculative: the spliced path seeks every segment and delivers from - /// one, so a conviction only becomes a real skip once a delivery commits past it. - /// Until then the entry waits in [`Self::seek_pending`], where a delivered group - /// removes itself (see [`Self::poll_seek_group`]). `committed` must be the - /// deliverer's sequence watermark (one past its last delivery), which only rises. - /// The seek's floor is NOT that: it includes `start_at`, which can be lowered - /// again, and a conviction it flushed could then be delivered after all. + /// A conviction only becomes a real skip once a delivery commits past it. Until then + /// the entry waits in [`Self::seek_pending`], where a delivered group removes itself + /// (see [`Self::poll_next_group`]). `committed` must be the sequence watermark (one + /// past the last delivery), which only rises. The seek's floor is NOT that: it + /// includes `start_at`, which can be lowered again, and a conviction it flushed could + /// then be delivered after all. fn commit_seek_stale(&mut self, committed: u64) { let keep = self.seek_pending.split_off(&committed); for (_, content) in std::mem::replace(&mut self.seek_pending, keep) { self.stale.add(content); } } - - /// Drop a conviction for `sequence` without counting it: the sequence was - /// delivered by another cursor over the same content (a splice boundary inside - /// the group leaves a copy in two segments), so its content is not stale. - fn discard_seek_conviction(&mut self, sequence: u64) { - self.seek_pending.remove(&sequence); - } } /// A cloneable handle to a subscriber's delivery preferences. @@ -3900,97 +3588,27 @@ impl Subscriber { } /// Attribute the groups the drift budget skipped since the last read. - /// - /// Drained rather than metered where the skip happens, for the same reason a - /// delivered group is metered here: a spliced subscriber reads through untagged - /// per-segment cursors, so this wrapper is the one place that counts exactly once. fn count_stale(&mut self, meter: &stats::Meter) { - // An untagged subscriber leaves the count where it is. A spliced reader polls - // its segments through this same method, and those segments are untagged, so - // draining here would throw the count away before the tagged handle wrapping - // them ever sees it. + // An untagged subscriber leaves the count where it is. if meter.is_tracked() { meter.stale(self.take_stale()); } } - /// Take the groups the drift budget skipped since the last call, so a nesting - /// handle (a spliced subscriber reading this one as a segment) can attribute them. + /// The groups the drift budget skipped since the last call; see [`Cursor::take_stale`]. pub(crate) fn take_stale(&mut self) -> stats::Content { match &mut self.inner { - SubscriberKind::Plain(plain) => plain.take_stale(), - SubscriberKind::Spliced(spliced) => spliced.take_stale(), - } - } - - /// Bound the drift anchor from outside, for a reader that caps this subscriber - /// without capping its cursor, and splices it with other tracks. - /// - /// A [`super::resume::Subscriber`] segment is deliberately left uncapped - /// ([`Self::set_groups`] would park boundary-crossing groups where its completion can't - /// be seen), so its own cap has to reach the anchor this way or the segment measures - /// drift against groups its reader will never be served. The same goes for the edge: - /// a segment's track never sees the groups of the segments after it. A spliced - /// segment folds the anchor into what it pushes onto its own segments, so it reaches - /// the plain cursors at the leaves however deep the splices nest. - pub(crate) fn set_anchor(&mut self, anchor: Anchor) { - match &mut self.inner { - SubscriberKind::Plain(plain) => { - plain.stale_cap = anchor.cap; - plain.update_drift_anchor(anchor); - } - SubscriberKind::Spliced(spliced) => spliced.set_anchor(anchor), - } - } - - /// Count the seek path's convictions below the deliverer's `committed` watermark - /// as stale, exactly once each; see [`PlainSubscriber::commit_seek_stale`]. - /// - /// The spliced seek calls this on every segment before it delivers, so a losing - /// segment's convictions are counted once the winning delivery moves the cursor - /// past them, and never before. - pub(crate) fn commit_seek_stale(&mut self, committed: u64) { - match &mut self.inner { - SubscriberKind::Plain(plain) => plain.commit_seek_stale(committed), - SubscriberKind::Spliced(spliced) => spliced.commit_seek_stale(committed), - } - } - - /// Drop any conviction for `sequence` without counting it; see - /// [`PlainSubscriber::discard_seek_conviction`]. The spliced deliverer calls this - /// for the sequence it just handed out, since a boundary inside that group leaves - /// a convictable copy in the next segment that the delivery splices into. - pub(crate) fn discard_seek_conviction(&mut self, sequence: u64) { - match &mut self.inner { - SubscriberKind::Plain(plain) => plain.discard_seek_conviction(sequence), - SubscriberKind::Spliced(spliced) => spliced.discard_seek_conviction(sequence), - } - } - - /// Whether the drift budget says to skip `group`, for a reader holding a group this - /// subscriber handed it earlier (a parked one, re-offered once a cap rose). Counts - /// the skip here so it reaches the stats with the rest. A spliced subscriber asks - /// the segment whose window covers the group, so a nested park is re-checked - /// against the same anchor a fresh read would use. - pub(crate) fn poll_stale(&mut self, group: &group::Consumer, waiter: &kio::Waiter) -> Poll> { - let plain = match &mut self.inner { - SubscriberKind::Plain(plain) => plain, - SubscriberKind::Spliced(spliced) => return spliced.poll_stale(group, waiter), - }; - let drift = ready!(plain.poll_drift(plain.anchor(plain.end_sequence), waiter))?; - let stale = ready!(plain.poll_stale(group, &drift, waiter))?; - if stale { - plain.note_stale(group); + Inner::Plain(cursor) => cursor.take_stale(), + Inner::Resume(resume, _) => resume.take_stale(), } - Poll::Ready(Ok(stale)) } /// Create a handle for updating this subscriber's delivery preferences. pub fn control(&self) -> Control { Control { subscription: match &self.inner { - SubscriberKind::Plain(plain) => plain.subscription.clone(), - SubscriberKind::Spliced(spliced) => spliced.prefs(), + Inner::Plain(cursor) => cursor.subscription.clone(), + Inner::Resume(resume, _) => resume.subscription().clone(), }, } } @@ -4026,8 +3644,8 @@ impl Subscriber { pub fn poll_recv_group(&mut self, waiter: &kio::Waiter) -> Poll>> { let meter = self.stats.meter(); let res = match &mut self.inner { - SubscriberKind::Plain(plain) => plain.poll_recv_group(waiter), - SubscriberKind::Spliced(spliced) => spliced.poll_recv_group(waiter), + Inner::Plain(cursor) => cursor.poll_recv_group(waiter), + Inner::Resume(resume, _) => resume.poll_group(false, waiter), }; self.count_stale(&meter); res.map(|res| res.map(|group| group.map(|group| group.with_meter(meter)))) @@ -4056,8 +3674,8 @@ impl Subscriber { pub fn poll_recv_datagram(&mut self, waiter: &kio::Waiter) -> Poll>> { let meter = self.stats.meter(); let res = match &mut self.inner { - SubscriberKind::Plain(plain) => plain.poll_recv_datagram(waiter), - SubscriberKind::Spliced(spliced) => spliced.poll_recv_datagram(waiter), + Inner::Plain(cursor) => cursor.poll_recv_datagram(waiter), + Inner::Resume(resume, _) => resume.poll_recv_datagram(waiter), }; // Unlike a group (metered lazily as its frames are read), a datagram is // delivered whole here, so count it as the single-frame group it stands in for. @@ -4078,30 +3696,11 @@ impl Subscriber { } /// The sequence cursor behind [`Ordered`], which owns the only public door to it. - fn poll_next_group(&mut self, waiter: &kio::Waiter) -> Poll>> { + pub(crate) fn poll_next_group(&mut self, waiter: &kio::Waiter) -> Poll>> { let meter = self.stats.meter(); let res = match &mut self.inner { - SubscriberKind::Plain(plain) => plain.poll_next_group(waiter), - SubscriberKind::Spliced(spliced) => spliced.poll_next_group(waiter), - }; - self.count_stale(&meter); - res.map(|res| res.map(|group| group.map(|group| group.with_meter(meter)))) - } - - /// Seek the lowest cached group in `floor..end` without advancing any cursor. - /// Crate-visible for [`super::resume::Subscriber`], whose sequence path picks the - /// lowest candidate across its segments and advances only its own floor; see - /// [`PlainSubscriber::poll_seek_group`]. - pub(crate) fn poll_seek_group( - &mut self, - floor: u64, - end: Option, - waiter: &kio::Waiter, - ) -> Poll>> { - let meter = self.stats.meter(); - let res = match &mut self.inner { - SubscriberKind::Plain(plain) => plain.poll_seek_group(floor, end, waiter), - SubscriberKind::Spliced(spliced) => spliced.poll_seek_group(floor, end, waiter), + Inner::Plain(cursor) => cursor.poll_next_group(waiter), + Inner::Resume(resume, _) => resume.poll_group(true, waiter), }; self.count_stale(&meter); res.map(|res| res.map(|group| group.map(|group| group.with_meter(meter)))) @@ -4123,38 +3722,48 @@ impl Subscriber { /// Whether `other` was cloned from this subscriber (shares the same underlying state). pub fn is_clone(&self, other: &Self) -> bool { match (&self.inner, &other.inner) { - (SubscriberKind::Plain(a), SubscriberKind::Plain(b)) => a.state.same_channel(&b.state), - (SubscriberKind::Spliced(a), SubscriberKind::Spliced(b)) => a.is_clone(b), + (Inner::Plain(a), Inner::Plain(b)) => a.state.same_channel(&b.state), + (Inner::Resume(a, _), Inner::Resume(b, _)) => a.subscription().same_channel(b.subscription()), _ => false, } } /// Poll for where the source's feed starts, raised to this cursor's floor, once - /// resolved; see [`Consumer::poll_start`]. A feed starting below the floor serves the + /// resolved; see [`Subscriber::poll_start`]. A feed starting below the floor serves the /// floor's group too. `None` when the source declares none. pub(crate) fn poll_start(&mut self, waiter: &kio::Waiter) -> Poll> { - match &mut self.inner { - SubscriberKind::Plain(plain) => { - let start = ready!(Consumer::poll_state_start(&plain.state, waiter)); - Poll::Ready(start.map(|start| start.max(plain.min_sequence))) - } - SubscriberKind::Spliced(spliced) => spliced.poll_start(waiter), - } + let cursor = match &mut self.inner { + Inner::Plain(cursor) => cursor, + Inner::Resume(resume, _) => return resume.poll_start(waiter), + }; + let start = ready!(Consumer::poll_state_start(&cursor.state, waiter)); + Poll::Ready(start.map(|start| start.max(cursor.min_sequence))) } - /// Poll for the track's declared final sequence, without blocking. - pub fn poll_finished(&mut self, waiter: &kio::Waiter) -> Poll> { - match &mut self.inner { - SubscriberKind::Plain(plain) => plain.poll(waiter, |state| state.poll_finished()), - SubscriberKind::Spliced(spliced) => spliced.poll_finished(waiter), + /// Poll for the track's cache reflecting its live feed (or the track ending), with the + /// largest position it holds; `None` for a track with nothing yet. A front's logical + /// track answers for the route serving it. See [`Producer::set_idle`]. + pub(crate) fn poll_live(&mut self, waiter: &kio::Waiter) -> Poll> { + let cursor = match &mut self.inner { + Inner::Plain(cursor) => cursor, + Inner::Resume(resume, _) => return resume.poll_live(waiter), + }; + let res = cursor.state.poll(waiter, |state| match state.readable() { + true => Poll::Ready(state.largest()), + false => Poll::Pending, + }); + match res { + Poll::Ready(Ok(largest)) => Poll::Ready(largest), + Poll::Ready(Err(state)) => Poll::Ready(state.largest()), + Poll::Pending => Poll::Pending, } } - /// Observe a declared end or a clean local close for an origin's spliced cursor. - pub(crate) fn poll_end(&mut self, waiter: &kio::Waiter) -> Poll>> { + /// Poll for the track's declared final sequence, without blocking. + pub fn poll_finished(&mut self, waiter: &kio::Waiter) -> Poll> { match &mut self.inner { - SubscriberKind::Plain(plain) => plain.poll(waiter, |state| state.poll_end()), - SubscriberKind::Spliced(spliced) => spliced.poll_end(waiter), + Inner::Plain(cursor) => cursor.poll(waiter, |state| state.poll_finished()), + Inner::Resume(resume, _) => resume.poll_finished(waiter), } } @@ -4191,20 +3800,16 @@ impl Subscriber { /// See [Local cursor vs wire preference](Self#local-cursor-vs-wire-preference). pub(crate) fn start_at(&mut self, sequence: u64) { match &mut self.inner { - SubscriberKind::Plain(plain) => plain.min_sequence = sequence, - SubscriberKind::Spliced(spliced) => spliced.start_at(sequence), + Inner::Plain(cursor) => cursor.min_sequence = sequence, + Inner::Resume(resume, _) => resume.raise_start_to(sequence), } } /// Raise the read cursor's floor to `sequence`, keeping any higher floor already set. - /// - /// The spliced layer positions a segment's inner cursor with this instead of - /// [`Self::set_groups`]: the inner subscription already resolved a start from its own - /// budget and floor, and an assignment would discard it. pub(crate) fn raise_start_to(&mut self, sequence: u64) { match &mut self.inner { - SubscriberKind::Plain(plain) => plain.min_sequence = plain.min_sequence.max(sequence), - SubscriberKind::Spliced(spliced) => spliced.raise_start_to(sequence), + Inner::Plain(cursor) => cursor.min_sequence = cursor.min_sequence.max(sequence), + Inner::Resume(resume, _) => resume.raise_start_to(sequence), } } @@ -4222,19 +3827,12 @@ impl Subscriber { /// Lowering the cap below the consumer's current cursor parks the consumer until the /// cap is raised. pub(crate) fn end_at(&mut self, end: impl Into) { - let end = end.into(); match &mut self.inner { - SubscriberKind::Plain(plain) => { - plain.end_sequence = end.exclusive(); - // A successor dropped by a lower cap stays dropped until the wrapping - // reader pushes its anchor again, which it does on every poll. - let outer = Anchor { - cap: plain.stale_cap, - ..plain.drift_anchor.read().clone() - }; - plain.update_drift_anchor(outer); + Inner::Plain(cursor) => { + cursor.end_sequence = end.into().exclusive(); + cursor.update_drift_cap(); } - SubscriberKind::Spliced(spliced) => spliced.end_at(end), + Inner::Resume(resume, _) => resume.end_at(end.into()), } } @@ -4249,21 +3847,20 @@ impl Subscriber { /// here (see [`Producer::subscription`]). Returns [`Error::Closed`] if the track /// already ended; the update is meaningless at that point and can usually be ignored. pub fn update(&mut self, subscription: Subscription) -> Result<()> { - match &mut self.inner { - SubscriberKind::Plain(plain) => { - let mut state = plain.subscription.write().map_err(|_| Error::Closed)?; - *state = subscription; - } - SubscriberKind::Spliced(spliced) => spliced.update(subscription), - } + let channel = match &self.inner { + Inner::Plain(cursor) => &cursor.subscription, + Inner::Resume(resume, _) => resume.subscription(), + }; + let mut state = channel.write().map_err(|_| Error::Closed)?; + *state = subscription; Ok(()) } /// Return the latest sequence number in the track. pub fn latest(&self) -> Option { match &self.inner { - SubscriberKind::Plain(plain) => plain.state.read().max_sequence, - SubscriberKind::Spliced(spliced) => spliced.latest(), + Inner::Plain(cursor) => cursor.state.read().max_sequence, + Inner::Resume(resume, _) => resume.latest(), } } } @@ -4442,6 +4039,9 @@ pub struct Request { // The serving session resolves the start of each subscription itself, so the // accepted track's start is unknown until it says (see [`Self::resolving_start`]). resolving_start: bool, + + // Served from a front's routes; see [`Self::routes`]. + routes: Option, } impl Request { @@ -4459,11 +4059,20 @@ impl Request { _dynamic: dynamic, stats: stats::Scope::default(), resolving_start: false, + routes: None, } } + /// Serve the track from a front's routes, read straight from their copies; see + /// [`super::resume`]. Applied atomically with [`Self::accept`], so no reader ever sees + /// the accepted track without it. + pub(crate) fn routes(mut self, routes: super::resume::Consumer) -> Self { + self.routes = Some(routes); + self + } + /// Mark the track as served by a session that resolves each subscription's start - /// (lite-06+), so [`Consumer::poll_start`] waits for its declaration instead of + /// (lite-06+), so [`Subscriber::poll_start`] waits for its declaration instead of /// reading the requested floor as the start. Applied atomically with /// [`Self::accept`], before any reader can see the track. pub(crate) fn resolving_start(mut self) -> Self { @@ -4485,7 +4094,7 @@ impl Request { /// A [`Consumer`] for the eventual track, usable before the request is accepted. pub fn consume(&self) -> Consumer { - Consumer::plain(self.name.clone(), self.state.consume()) + Consumer::new(self.name.clone(), self.state.consume()) } /// Create a [`Dynamic`] handle that serves on-demand fetches of uncached @@ -4536,6 +4145,7 @@ impl Request { if let Ok(mut state) = self.state.write() { state.accept(info.clone()); state.start_pending = self.resolving_start; + state.routes = self.routes; } // Accepting the request creates the track producer: count it as one ingress // subscription (closed when the last handle drops). No-op when untagged. @@ -4678,37 +4288,6 @@ mod test { .expect("track was closed") } - /// A declared start (SUBSCRIBE_START) resolves a peek below it as a permanent - /// miss, while a group that is already cached below the start stays readable - /// (a fetch can create one; the declaration only covers the live feed). - #[tokio::test] - async fn peek_resolves_below_the_declared_start() { - let mut producer = track_producer("test", None); - let consumer = producer.consume(); - - let mut cached = producer.create_group(group::Info { sequence: 1 }).unwrap(); - cached.write_frame(Timestamp::ZERO, b"backfill".to_vec()).unwrap(); - cached.finish().unwrap(); - - // Nothing declared yet: a missing group parks. - let waiter = kio::Waiter::noop(); - assert!(consumer.poll_peek_group(0, &waiter).is_pending()); - - // The declaration turns the missing group into a permanent miss, but the - // cached one below it still resolves. - producer.start_at(3).unwrap(); - assert!(matches!(consumer.poll_peek_group(0, &waiter), Poll::Ready(None))); - assert!(matches!(consumer.poll_peek_group(1, &waiter), Poll::Ready(Some(_)))); - assert!(consumer.poll_peek_group(3, &waiter).is_pending()); - - // The declaration follows the demand in either direction: forward retires - // the skipped range, backward reopens it. - producer.start_at(4).unwrap(); - assert!(matches!(consumer.poll_peek_group(3, &waiter), Poll::Ready(None))); - producer.start_at(0).unwrap(); - assert!(consumer.poll_peek_group(0, &waiter).is_pending()); - } - #[tokio::test] async fn append_datagram_shares_group_sequence() { let mut producer = track_producer("test", None); @@ -6583,9 +6162,7 @@ mod test { let state = producer.state.read(); let drift = Drift { budget: Duration::ZERO, - edge: state.drift_edge(None, None, None), - outer: None, - successor: None, + edge: state.drift_edge(None), }; assert!( state.is_stale(0, &drift.edge, drift.budget), @@ -6604,48 +6181,6 @@ mod test { ); } - /// The same holds for an edge a splice pushed from another segment: a handed-out - /// group judges it long after the splice resolved it, so it is revalidated on its own - /// track before it convicts anything. - #[tokio::test] - async fn an_evicted_outer_edge_convicts_nothing() { - let mut producer = track_producer("a", None); - let mut subscriber = producer.subscribe(Subscription::default().with_max_age(Duration::from_secs(1))); - let mut open = producer.append_group().unwrap(); - open.write_frame(Timestamp::ZERO, bytes::Bytes::from_static(b"a")) - .unwrap(); - let mut group = subscriber.recv_group().await.unwrap().expect("group"); - assert!(group.read_frame().await.unwrap().is_some()); - // A successor bounds the open group's reach, without being late against it. - append_at(&mut producer, 10); - - let mut next = track_producer("b", None); - append_at(&mut next, 30_000); - let edge = append_at(&mut next, 30_010); - subscriber.set_anchor(Anchor { - cap: None, - edge: next.consume().live_edge(None), - successor: None, - }); - - let mut control = group.clone(); - assert!( - matches!(control.read_frame().now_or_never(), Some(Ok(None))), - "the open group ends against the outer edge" - ); - - // The outer edge dies before the group is judged again. - let slot = next.modify().unwrap().lookup.remove(&edge).unwrap(); - let _ = slot.group.abort(Error::Evicted); - - assert!( - group.read_frame().now_or_never().is_none(), - "a vanished outer edge is no reason to drop what is left" - ); - assert!(!group.latency_expired()); - open.finish().unwrap(); - } - #[tokio::test] async fn a_lower_sequence_is_never_the_live_edge() { let producer = track_producer("test", None); @@ -6691,9 +6226,7 @@ mod test { append_at(&mut producer, 1000); // Capped at group 0: the route running on past the cap is data this subscriber - // can never be served, so it isn't a live edge to be late against. This is what - // keeps a spliced segment from dropping the groups either side of a takeover - // boundary. + // can never be served, so it isn't a live edge to be late against. let mut subscriber = producer.subscribe(Subscription::default().with_end(Position::after_group(0))); subscriber.set_groups(..1); assert_eq!(drain(&mut subscriber), vec![0]); @@ -7036,21 +6569,6 @@ mod test { assert!(done.is_none(), "missing groups below the boundary are not an error"); } - #[tokio::test] - async fn cached_groups_preserve_arrival_order() { - let producer = track_producer("test", None); - producer.create_group(group::Info { sequence: 5 }).unwrap(); - producer.create_group(group::Info { sequence: 3 }).unwrap(); - - let groups = producer.consume().cached_groups(); - let sequences: Vec = groups.iter().map(|(group, _)| group.sequence).collect(); - assert_eq!( - sequences, - vec![5, 3], - "warm snapshot must follow arrival, not sequence order" - ); - } - #[test] fn append_finish_cannot_be_rewritten() { let producer = track_producer("test", None); diff --git a/rs/moq-net/tests/fetch_cancel.rs b/rs/moq-net/tests/fetch_cancel.rs index 5e2d45ff7c5..53ab1386265 100644 --- a/rs/moq-net/tests/fetch_cancel.rs +++ b/rs/moq-net/tests/fetch_cancel.rs @@ -152,3 +152,67 @@ async fn abandoned_fetch_reaches_the_publisher_lite05() { async fn abandoned_fetch_reaches_the_publisher_lite06() { abandoned_fetches_reach_the_publisher("moq-lite-06").await; } + +/// A group fetched through a relay is still there for the next fetch shortly after the +/// track went unread: the relay's copy lingers with its cache, so a segmented reader +/// (HLS fetching each segment) does not go upstream again for every request. +async fn a_refetch_within_the_linger_stays_on_the_relay(version: &str) { + let version: Version = version.parse().unwrap(); + let publisher = produce_origin(1); + let relay = produce_origin(2); + let _pair = peer(version, &publisher, &relay).await; + + let broadcast = publisher.create_broadcast("room").unwrap(); + let mut dynamic = broadcast.dynamic(); + broadcast.announce(Default::default()).unwrap(); + tokio::time::sleep(Duration::from_secs(1)).await; + + let consumer = relay.consume().request_broadcast("room").await.unwrap(); + let mut waiting = Box::pin(consumer.track("video").unwrap().fetch_group(0, None)); + let request = tokio::select! { + request = dynamic.requested_track() => request.expect("the track request reaches the publisher"), + _ = &mut waiting => panic!("nothing answered the track"), + }; + let groups = request.dynamic(); + let _track = request.accept(track::Info::default()); + let request = tokio::select! { + request = groups.requested_group() => request.expect("the fetch reaches the publisher"), + _ = &mut waiting => panic!("nothing answered the fetch"), + }; + let mut group = request.accept(None).unwrap(); + group + .write_frame(Timestamp::ZERO, Bytes::from_static(b"segment")) + .unwrap(); + group.finish().unwrap(); + let mut fetched = waiting.await.unwrap(); + assert_eq!(&fetched.read_frame().await.unwrap().unwrap().payload[..], b"segment"); + drop(fetched); + // The publisher lets the group go, so only another FETCH would bring it back. + group.abort(moq_net::Error::Cancel).unwrap(); + + // The track goes unread between segments, well inside the linger. + tokio::time::sleep(Duration::from_secs(5)).await; + + let mut refetch = Box::pin(consumer.track("video").unwrap().fetch_group(0, None)); + let mut fetched = tokio::select! { + fetched = &mut refetch => fetched.expect("the relay serves the group again"), + _ = dynamic.requested_track() => panic!("{version}: the relay asked for the track again"), + _ = groups.requested_group() => panic!("{version}: the relay fetched the group again"), + }; + assert_eq!(&fetched.read_frame().await.unwrap().unwrap().payload[..], b"segment"); +} + +#[tokio::test(start_paused = true)] +async fn a_refetch_within_the_linger_stays_on_the_relay_lite06() { + a_refetch_within_the_linger_stays_on_the_relay("moq-lite-06").await; +} + +#[tokio::test(start_paused = true)] +async fn a_refetch_within_the_linger_stays_on_the_relay_lite07() { + a_refetch_within_the_linger_stays_on_the_relay("moq-lite-07-wip").await; +} + +#[tokio::test(start_paused = true)] +async fn a_refetch_within_the_linger_stays_on_the_relay_ietf19() { + a_refetch_within_the_linger_stays_on_the_relay("moq-transport-19").await; +} diff --git a/rs/moq-net/tests/legacy_reconnect.rs b/rs/moq-net/tests/legacy_reconnect.rs index 143f902f1e1..f883b635b0b 100644 --- a/rs/moq-net/tests/legacy_reconnect.rs +++ b/rs/moq-net/tests/legacy_reconnect.rs @@ -1,10 +1,9 @@ -//! A publisher on a wire with no hop ids, reconnecting through a relay, is a new source -//! downstream. +//! A publisher on a wire with no hop ids, reconnecting through a relay, is a new first +//! hop downstream. //! //! moq-transport without the Cluster extension names no publisher, so the relay it //! connects to stamps each connection with a random Hop ID of its own. A reconnect is a -//! new connection and so a new first hop, which a downstream relay reads as a new -//! source rather than splicing it onto the old one. +//! new connection and so a new first hop. mod support; diff --git a/rs/moq-net/tests/rejoin.rs b/rs/moq-net/tests/rejoin.rs index b6684c46ce0..46bfff2f86d 100644 --- a/rs/moq-net/tests/rejoin.rs +++ b/rs/moq-net/tests/rejoin.rs @@ -83,14 +83,15 @@ async fn rejoin_recovers_the_group_reset_on_leave() { let mut sub = remote.track("video").unwrap().subscribe(prefs()).await.unwrap(); let mut rejoined = Vec::new(); - loop { + // Arrival order: the reset group comes back once the route delivers it again, + // which may be after a newer group. + let wanted: &[u64] = if advance { &[1, 2] } else { &[1] }; + while !wanted.iter().all(|want| rejoined.iter().any(|(seq, _)| seq == want)) { let mut group = sub.recv_group().await.unwrap().unwrap(); let sequence = group.sequence; rejoined.push((sequence, read_all(&mut group).await)); - if sequence == if advance { 2 } else { 1 } { - break; - } } + rejoined.sort_by_key(|(seq, _)| *seq); let reset = rejoined.iter().find(|(seq, _)| *seq == 1).expect("reset group"); // The reader asks for the group it is missing, so the reset group comes back // whole even once the live edge has moved past it. @@ -108,3 +109,69 @@ async fn rejoin_recovers_the_group_reset_on_leave() { } } } + +/// A reader rejoining a track another handle kept open is not handed what the relay cached +/// before it went idle. +/// +/// A fetch-only reader holds the relay's copy of the track, so the relay keeps it when the +/// last subscriber leaves and only cancels the upstream subscription. The publisher moves +/// on meanwhile. The cache is stale until the route answers again, so the rejoining reader +/// starts at the live edge, whether the answer carries the largest position (lite-07) or +/// the first frame says where the feed is. +#[tokio::test(start_paused = true)] +async fn rejoin_skips_a_cache_kept_by_another_handle() { + for version in [ + "moq-lite-05", + "moq-lite-06", + "moq-lite-07-wip", + "moq-transport-19", + "moq-transport-22", + ] { + tokio::time::timeout(TEST_TIMEOUT, async { + let publisher = produce_origin(1); + let relay = produce_origin(2); + + let broadcast = publisher.create_broadcast("bench").unwrap(); + let track = broadcast.create_track("video", None).unwrap(); + broadcast.announce(Default::default()).unwrap(); + + let mut options = MockConnectOptions::new(version.parse::().unwrap()); + options.server_publish = Some(publisher.consume()); + options.client_subscribe = Some(relay.clone()); + let _pair = connect_mock(options).await; + + let consumer = relay.consume(); + consumer.routed("bench").await.unwrap(); + let remote = consumer.request_broadcast("bench").await.unwrap(); + let ts = |ms| Timestamp::from_millis(ms).unwrap(); + + // Held for fetches only: it keeps the relay's copy without subscribing. + let _held = remote.track("video").unwrap(); + + let mut group = track.append_group().unwrap(); + group.write_frame(ts(0), b"old".as_ref()).unwrap(); + group.finish().unwrap(); + let mut sub = remote.track("video").unwrap().subscribe(None).await.unwrap(); + let mut group = sub.recv_group().await.unwrap().unwrap(); + assert_eq!(read_all(&mut group).await.unwrap(), [b"old".to_vec()], "{version}"); + drop((group, sub)); + track.demand().unused().await.unwrap(); + + // The publisher moves on while nobody subscribes. + for sequence in 1..=3u64 { + let mut group = track.append_group().unwrap(); + group.write_frame(ts(sequence * 1000), b"new".as_ref()).unwrap(); + group.finish().unwrap(); + } + + let mut sub = remote.track("video").unwrap().subscribe(None).await.unwrap(); + let group = sub.recv_group().await.unwrap().unwrap(); + assert_eq!( + group.sequence, 3, + "{version}: the rejoining reader got the stale cache first" + ); + }) + .await + .unwrap_or_else(|_| panic!("{version}: timed out")); + } +} diff --git a/rs/moq-net/tests/route_change.rs b/rs/moq-net/tests/route_change.rs new file mode 100644 index 00000000000..417d1fc249b --- /dev/null +++ b/rs/moq-net/tests/route_change.rs @@ -0,0 +1,463 @@ +//! A subscription survives its route changing, end to end over real sessions. +//! +//! A publisher `P` is pulled by two relays `A` and `B`, both of which re-advertise +//! it to the subscribing relay `R`. A path is one broadcast whoever serves it, so `R` +//! may resume a subscription served through one onto the other. The reader on `R` +//! must see every frame exactly once, in order, whether the route changes between +//! groups or in the middle of one, and however it changes. + +mod support; + +use std::time::Duration; + +use moq_net::{Error, Hop, Timestamp, Version, broadcast, origin, track}; +use support::harness::{MockConnectOptions, MockPair, connect_mock}; +use tokio::sync::mpsc; + +const TEST_TIMEOUT: Duration = Duration::from_secs(30); + +/// Frames per group. +const FRAMES: u64 = 4; + +fn produce_origin(hop: u64) -> origin::Producer { + let (producer, driver) = origin::Producer::new(origin::Config::new(Hop::new(hop).unwrap())); + tokio::spawn(support::harness::run(driver)); + producer +} + +/// Have `to` pull everything `from` publishes. +async fn link(version: Version, from: &origin::Producer, to: &origin::Producer) -> MockPair { + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(from.consume()); + options.client_subscribe = Some(to.clone()); + connect_mock(options).await +} + +fn payload(group: u64, frame: u64) -> Vec { + format!("{group}.{frame}").into_bytes() +} + +/// Let every task settle. Time is paused, so this returns once the runtime is idle. +async fn settle() { + tokio::time::sleep(Duration::from_millis(500)).await; +} + +/// How the serving route goes away (or is beaten). +#[derive(Clone, Copy, Debug)] +enum Trigger { + /// The session to `A` dies abruptly. + Disconnect, + /// `A` loses the broadcast and withdraws it, while its session to `R` stays up. + Unannounce, + /// `R` gains a direct (shorter) route to `P` while `A` stays healthy. + BetterRoute, + /// The direct route from [`Trigger::BetterRoute`] dies, sending the track back. + BetterRouteDies, +} + +/// Where in the stream the route changes. +#[derive(Clone, Copy, Debug)] +enum Position { + /// After a group finished, before the next one starts. + BetweenGroups, + /// After the reader saw half of an open group. + MidGroup, +} + +struct Topology { + version: Version, + publisher: origin::Producer, + relay_b: origin::Producer, + subscriber: origin::Producer, + _broadcast: broadcast::Producer, + track: track::Producer, + p_to_a: Option, + a_to_r: Option, + direct: Option, + _links: Vec, +} + +impl Topology { + /// `P -> A -> R` serving, with `P -> B` pulled and `B -> R` connected after the + /// subscription started on `A`, so `A` is the incumbent. + async fn new(version: Version) -> (Self, track::Subscriber) { + let publisher = produce_origin(1); + let relay_a = produce_origin(2); + let relay_b = produce_origin(3); + let subscriber = produce_origin(4); + + let broadcast = publisher.create_broadcast("live").unwrap(); + let track = broadcast.create_track("video", None).unwrap(); + broadcast.announce(Default::default()).unwrap(); + + let p_to_a = link(version, &publisher, &relay_a).await; + let p_to_b = link(version, &publisher, &relay_b).await; + let a_to_r = link(version, &relay_a, &subscriber).await; + + let consumer = subscriber.consume(); + consumer.routed("live").await.unwrap(); + let remote = consumer.request_broadcast("live").await.unwrap(); + let prefs = track::Subscription::default().with_max_age(Duration::from_secs(60)); + let sub = remote.track("video").unwrap().subscribe(prefs).await.unwrap(); + + let topology = Self { + version, + publisher, + relay_b, + subscriber, + _broadcast: broadcast, + track, + p_to_a: Some(p_to_a), + a_to_r: Some(a_to_r), + direct: None, + _links: vec![p_to_b], + }; + (topology, sub) + } + + /// Connect the standby route `B -> R`. + async fn standby(&mut self) { + let b_to_r = link(self.version, &self.relay_b, &self.subscriber).await; + self._links.push(b_to_r); + settle().await; + } + + async fn trigger(&mut self, trigger: Trigger) { + match trigger { + Trigger::Disconnect => { + let pair = self.a_to_r.take().unwrap(); + pair.server.abort(Error::Cancel); + pair.client.abort(Error::Cancel); + } + Trigger::Unannounce => { + let pair = self.p_to_a.take().unwrap(); + pair.server.abort(Error::Cancel); + pair.client.abort(Error::Cancel); + } + Trigger::BetterRoute => { + self.direct = Some(link(self.version, &self.publisher, &self.subscriber).await); + } + Trigger::BetterRouteDies => { + let pair = self.direct.take().unwrap(); + pair.server.abort(Error::Cancel); + pair.client.abort(Error::Cancel); + } + } + settle().await; + } +} + +/// Read every group in full, reporting each frame as it arrives. +fn read(mut sub: track::Subscriber) -> mpsc::UnboundedReceiver<(u64, moq_net::Result>)> { + let (tx, rx) = mpsc::unbounded_channel(); + tokio::spawn(async move { + loop { + let mut group = match sub.recv_group().await { + Ok(Some(group)) => group, + Ok(None) => return, + Err(err) => { + let _ = tx.send((u64::MAX, Err(err))); + return; + } + }; + loop { + match group.read_frame().await { + Ok(Some(frame)) => { + if tx.send((group.sequence, Ok(frame.payload.to_vec()))).is_err() { + return; + } + } + Ok(None) => break, + Err(err) => { + let _ = tx.send((group.sequence, Err(err))); + break; + } + } + } + } + }); + rx +} + +/// Wait for the next frame the reader reports, failing on an error or a hang. +async fn next(rx: &mut mpsc::UnboundedReceiver<(u64, moq_net::Result>)>) -> (u64, Vec) { + let (group, frame) = tokio::time::timeout(Duration::from_secs(10), rx.recv()) + .await + .expect("reader hung") + .expect("reader ended"); + (group, frame.unwrap_or_else(|err| panic!("group {group} failed: {err}"))) +} + +async fn route_change(version: &str, trigger: Trigger, position: Position) { + let version: Version = version.parse().unwrap(); + let (mut topology, sub) = Topology::new(version).await; + let mut rx = read(sub); + topology.standby().await; + + let mut expected = Vec::new(); + let mut seen = Vec::new(); + + // Group 0 in full through `A`. + let mut group = topology.track.append_group().unwrap(); + for frame in 0..FRAMES { + group.write_frame(Timestamp::ZERO, payload(0, frame)).unwrap(); + expected.push((0, payload(0, frame))); + seen.push(next(&mut rx).await); + } + group.finish().unwrap(); + + // Group 1: half through `A`, then the route changes. Between groups, the change + // lands before the group starts at all. + let split = match position { + Position::BetweenGroups => 0, + Position::MidGroup => FRAMES / 2, + }; + if split == 0 { + topology.trigger(trigger).await; + } + let mut group = topology.track.append_group().unwrap(); + for frame in 0..FRAMES { + if frame == split && split != 0 { + topology.trigger(trigger).await; + } + group.write_frame(Timestamp::ZERO, payload(1, frame)).unwrap(); + expected.push((1, payload(1, frame))); + seen.push(next(&mut rx).await); + } + group.finish().unwrap(); + + // Group 2 entirely through the replacement. + let mut group = topology.track.append_group().unwrap(); + for frame in 0..FRAMES { + group.write_frame(Timestamp::ZERO, payload(2, frame)).unwrap(); + expected.push((2, payload(2, frame))); + seen.push(next(&mut rx).await); + } + group.finish().unwrap(); + + let render = |frames: &[(u64, Vec)]| { + frames + .iter() + .map(|(group, frame)| format!("{group}:{}", String::from_utf8_lossy(frame))) + .collect::>() + .join(" ") + }; + assert_eq!(render(&seen), render(&expected), "{version} {trigger:?} {position:?}"); + + // Nothing trails: no duplicate group or frame arrives later. + settle().await; + assert!( + rx.try_recv().is_err(), + "{version} {trigger:?} {position:?}: trailing delivery" + ); +} + +/// The route changes again and again in one stream, each time mid-group: a direct +/// route beats `A`, then dies, then `A`'s session dies too, leaving `B`. The reader +/// still sees every frame exactly once. +async fn route_flaps(version: &str) { + let version: Version = version.parse().unwrap(); + let (mut topology, sub) = Topology::new(version).await; + let mut rx = read(sub); + topology.standby().await; + + let triggers = [Trigger::BetterRoute, Trigger::BetterRouteDies, Trigger::Disconnect]; + let mut expected = Vec::new(); + let mut seen = Vec::new(); + for sequence in 0..=triggers.len() as u64 { + let mut group = topology.track.append_group().unwrap(); + for frame in 0..FRAMES { + if frame == FRAMES / 2 + && let Some(trigger) = triggers.get(sequence as usize) + { + topology.trigger(*trigger).await; + } + group.write_frame(Timestamp::ZERO, payload(sequence, frame)).unwrap(); + expected.push((sequence, payload(sequence, frame))); + seen.push(next(&mut rx).await); + } + group.finish().unwrap(); + } + + assert_eq!(seen, expected, "{version}"); + settle().await; + assert!(rx.try_recv().is_err(), "{version}: trailing delivery"); +} + +macro_rules! route_change_tests { + ($($name:ident: $version:literal,)*) => { + $( + mod $name { + use super::*; + + async fn run(trigger: Trigger, position: Position) { + tokio::time::timeout(TEST_TIMEOUT, route_change($version, trigger, position)) + .await + .expect("timed out"); + } + + #[tokio::test(start_paused = true)] + async fn disconnect_between_groups() { + run(Trigger::Disconnect, Position::BetweenGroups).await; + } + + #[tokio::test(start_paused = true)] + async fn disconnect_mid_group() { + run(Trigger::Disconnect, Position::MidGroup).await; + } + + #[tokio::test(start_paused = true)] + async fn unannounce_between_groups() { + run(Trigger::Unannounce, Position::BetweenGroups).await; + } + + #[tokio::test(start_paused = true)] + async fn unannounce_mid_group() { + run(Trigger::Unannounce, Position::MidGroup).await; + } + + #[tokio::test(start_paused = true)] + async fn better_route_between_groups() { + run(Trigger::BetterRoute, Position::BetweenGroups).await; + } + + #[tokio::test(start_paused = true)] + async fn better_route_mid_group() { + run(Trigger::BetterRoute, Position::MidGroup).await; + } + + } + )* + }; +} + +route_change_tests! { + lite_07: "moq-lite-07-wip", + lite_06: "moq-lite-06", + lite_05: "moq-lite-05", + lite_04: "moq-lite-04", + ietf_19: "moq-transport-19", + ietf_22: "moq-transport-22", +} + +/// `P` feeds `A` slowly and `B` promptly, and `R` reads through `A` until that route dies +/// partway through group 1: `A` has handed on (1, 0) and (1, 1) while `P` has already +/// moved on to group 2. `B` (warmed by another reader on `W`) holds everything, so `R` +/// resumes there from (1, 2). +async fn lagging_route_dies(version: Version) -> mpsc::UnboundedReceiver<(u64, moq_net::Result>)> { + async fn lagged(version: Version, from: &origin::Producer, to: &origin::Producer, lag: Duration) -> MockPair { + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(from.consume()); + options.client_subscribe = Some(to.clone()); + options.latency = lag; + connect_mock(options).await + } + async fn subscribe(origin: &origin::Producer) -> track::Subscriber { + let consumer = origin.consume(); + consumer.routed("live").await.unwrap(); + let remote = consumer.request_broadcast("live").await.unwrap(); + let preferences = track::Subscription::default().with_max_age(Duration::from_secs(60)); + remote.track("video").unwrap().subscribe(preferences).await.unwrap() + } + fn write(group: &mut moq_net::group::Producer, sequence: u64, frame: u64) { + let timestamp = Timestamp::from_micros(1_000_000 + sequence * 100_000 + frame * 1_000).unwrap(); + group.write_frame(timestamp, payload(sequence, frame)).unwrap(); + } + + let (p, a, b, r, w) = ( + produce_origin(1), + produce_origin(2), + produce_origin(3), + produce_origin(4), + produce_origin(5), + ); + let broadcast = p.create_broadcast("live").unwrap(); + let track = broadcast.create_track("video", None).unwrap(); + broadcast.announce(Default::default()).unwrap(); + let p_a = lagged(version, &p, &a, Duration::from_millis(300)).await; + let p_b = lagged(version, &p, &b, Duration::ZERO).await; + let a_r = lagged(version, &a, &r, Duration::ZERO).await; + let b_w = lagged(version, &b, &w, Duration::ZERO).await; + let warm = read(subscribe(&w).await); + let mut rx = read(subscribe(&r).await); + let b_r = lagged(version, &b, &r, Duration::ZERO).await; + settle().await; + + let mut group = track.append_group().unwrap(); + for frame in 0..4 { + write(&mut group, 0, frame); + } + group.finish().unwrap(); + for frame in 0..4 { + assert_eq!(next(&mut rx).await, (0, payload(0, frame)), "{version}"); + } + + let mut group = track.append_group().unwrap(); + write(&mut group, 1, 0); + write(&mut group, 1, 1); + assert_eq!(next(&mut rx).await, (1, payload(1, 0)), "{version}"); + assert_eq!(next(&mut rx).await, (1, payload(1, 1)), "{version}"); + // P moves on while A is still behind. + write(&mut group, 1, 2); + write(&mut group, 1, 3); + group.finish().unwrap(); + let mut group = track.append_group().unwrap(); + write(&mut group, 2, 0); + + a_r.server.abort(Error::Cancel); + a_r.client.abort(Error::Cancel); + settle().await; + tokio::spawn(async move { + let _keep = (broadcast, track, group, p_a, p_b, b_w, b_r, warm, a_r, p, a, b, r, w); + std::future::pending::<()>().await + }); + rx +} + +/// Lite carries the resume point upstream, so the rest of the group and the next one +/// arrive in order. +#[tokio::test(start_paused = true)] +async fn lite_resumes_after_a_lagging_route_dies() { + for version in ["moq-lite-05", "moq-lite-06"] { + let version: Version = version.parse().unwrap(); + let mut rx = lagging_route_dies(version).await; + assert_eq!(next(&mut rx).await, (1, payload(1, 2)), "{version}"); + assert_eq!(next(&mut rx).await, (1, payload(1, 3)), "{version}"); + assert_eq!(next(&mut rx).await, (2, payload(2, 0)), "{version}"); + } +} + +/// Before draft 20 the resume rides a joining FETCH from group 1 through Largest, which +/// a moq-rs publisher refuses for spanning several groups, so the subscription joins +/// live instead. The rest of group 1 is still fetched on its own. +#[tokio::test(start_paused = true)] +async fn an_older_ietf_draft_still_resumes_the_open_group() { + let version: Version = "moq-transport-19".parse().unwrap(); + let mut rx = lagging_route_dies(version).await; + assert_eq!(next(&mut rx).await, (1, payload(1, 2))); + assert_eq!(next(&mut rx).await, (1, payload(1, 3))); +} + +/// moq-transport 22 is left out: its relay fills a group's missing frames with a FETCH, +/// which draft 20+ does not serve yet (see `quest/m1/ietf-fetch-location.md`). +macro_rules! route_flap_tests { + ($($name:ident: $version:literal,)*) => { + $( + #[tokio::test(start_paused = true)] + async fn $name() { + tokio::time::timeout(TEST_TIMEOUT, route_flaps($version)) + .await + .expect("timed out"); + } + )* + }; +} + +route_flap_tests! { + flaps_lite_07: "moq-lite-07-wip", + flaps_lite_06: "moq-lite-06", + flaps_lite_05: "moq-lite-05", + flaps_lite_04: "moq-lite-04", + flaps_ietf_19: "moq-transport-19", + flaps_ietf_22: "moq-transport-22", +} diff --git a/rs/moq-tokio/tests/broadcast.rs b/rs/moq-tokio/tests/broadcast.rs index 28148e95756..70dc6ab3ca6 100644 --- a/rs/moq-tokio/tests/broadcast.rs +++ b/rs/moq-tokio/tests/broadcast.rs @@ -1232,10 +1232,10 @@ async fn rejoin_skips_a_stale_warm_cache(version: &str) { .expect("request timeout") .expect("broadcast resolves"); let budget = moq_net::track::Subscription::default().with_max_age(Duration::from_millis(100)); - async fn recv(sub: &mut moq_net::track::Subscriber) -> u64 { + async fn recv(sub: &mut moq_net::track::Subscriber, version: &str) -> u64 { tokio::time::timeout(TIMEOUT, sub.recv_group()) .await - .expect("recv timeout") + .unwrap_or_else(|_| panic!("{version}: recv timeout")) .expect("recv failed") .expect("track ended") .sequence @@ -1247,7 +1247,7 @@ async fn rejoin_skips_a_stale_warm_cache(version: &str) { .subscribe(budget.clone()) .await .expect("subscribe"); - recv(&mut sub).await; + recv(&mut sub, version).await; drop(sub); // The front parks the track and cancels upstream, while the publisher moves on. @@ -1265,14 +1265,14 @@ async fn rejoin_skips_a_stale_warm_cache(version: &str) { .subscribe(budget) .await .expect("resubscribe"); - let first = recv(&mut sub).await; + let first = recv(&mut sub, version).await; assert!( first > 4, "{version}: a rejoining reader was served the stale cache first: group {first}" ); let mut sequence = first; while sequence < 20 { - sequence = recv(&mut sub).await; + sequence = recv(&mut sub, version).await; assert!( sequence >= 4, "{version}: a rejoining reader was served stale group {sequence}"