diff --git a/doc/concept/identity.md b/doc/concept/identity.md index 5f73e058273..a9a567375a7 100644 --- a/doc/concept/identity.md +++ b/doc/concept/identity.md @@ -148,7 +148,8 @@ one holds a follower. `@moq/watch`, `moq play`, and `moqsrc` follow this way, an request failing is never the signal to restart; the follower handing out a new broadcast is. Nor is a track's error kind: a track withdrawn unfinished ends `Unroutable` over moq-lite but `Internal` over moq-transport, -whose wire has no code for it. +whose wire has no code for it, and one its publisher releases unfinished ends +`Internal` over both. **Hold one stream** when the output must not splice two, as an exporter or recorder does. Rust and JavaScript `request(path)` and `moq_mux::Source` keep diff --git a/doc/lib/cpp/index.md b/doc/lib/cpp/index.md index bc483a497b6..fa58e9c4f97 100644 --- a/doc/lib/cpp/index.md +++ b/doc/lib/cpp/index.md @@ -269,9 +269,10 @@ its `moq::MediaCatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast->close()` takes it offline. How long to keep it up is yours to decide. A track still open at the -close, or released without a finish, ends with `Unroutable` (`Internal` over -moq-transport, which has no such code). Readers follow a publisher's runs by -announcement, not by that error: see [identity and recovery](/concept/identity). +close ends with `Unroutable` (`Internal` over moq-transport, which has no such +code), and one released without a finish ends `Dropped`, which readers see as +`Internal`. Readers follow a publisher's runs by announcement, not by that error: +see [identity and recovery](/concept/identity). `moq::shutdown()` stops the `moq-ffi` runtime workers, then the executor. Pending calls resolve `Cancelled`, and every object stays safe to drop. Call diff --git a/doc/lib/dart/index.md b/doc/lib/dart/index.md index 51a7b2046a9..29d92080cb0 100644 --- a/doc/lib/dart/index.md +++ b/doc/lib/dart/index.md @@ -156,7 +156,7 @@ shows which module holds each. - **Media reads.** `media.TrackConsumer`, `audio.Consumer`, and `video.Consumer` return a `Read` in order: a `ReadFrame`, a `ReadGap` where media was lost, or a `ReadEnd` where the publisher stopped its encoder. A gap breaks continuity. An encoder end is not stream EOF: following frames before its endpoint are the old encoder's drain; the first at or past it starts fresh. A decoder reads everything the track still holds unless a live player sets `start: media.DecodeStart.latest` in its decoder output, which joins at the newest group and skips the cached backlog. - **Decoded frames hold decoder buffers.** `dispose()` each frame from a `video.Consumer` promptly rather than waiting for the garbage collector, or the decoder stalls. `resize` is best effort, so read each frame's `width()` and `height()`. - **Closing.** `await moq.close()` drains for up to one second and reports its result. `session.shutdown()` requests the drain synchronously; `await session.closed()` waits for its result. `session.cancel(code: 0)` closes at once. Finish or abort live tracks first. Awaiting cancellation of an announcement, grant, or request subscription releases its pending native read. Announcements and grants also release their consumer; server listeners leave their shared server usable. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. -- **Ending cleanly.** When the input ends, finish every track, then call `finish()` on its `CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.close()` or disposing its `MoqRoute` takes it offline. How long to keep it up is yours to decide. A track still open at the close, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). +- **Ending cleanly.** When the input ends, finish every track, then call `finish()` on its `CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.close()` or disposing its `MoqRoute` takes it offline. How long to keep it up is yours to decide. A track still open at the close ends with `Unroutable` (`Internal` over moq-transport, which has no such code), and one released without a finish ends `Dropped`, which readers see as `Internal`. Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). - **Stats.** `session.stats()` reports `rtt`, `estimatedSendRateBps`, and the byte and packet counters (`bytesSent`, `bytesReceived`, `bytesLost`, `packetsSent`, `packetsReceived`, `packetsLost`), and `datagramsUnrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is `null` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe reconnects. A first dial that reaches no session ends the session with that error rather than retrying. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recvBandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rateChanged()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `client.viewers()` divides the connection's receive estimate among the broadcasts you watch, max-min fair by each one's ceiling. `watch(maxBps: ...)` returns a `Viewer` claiming up to what the largest rendition the player would switch to needs; `rate()` reads its slice in bits per second, `null` while there is no estimate, so hold the current selection. `update(maxBps: ...)` changes the ceiling, such as when the catalog enables or disables a rendition. Holding a viewer requests PROBE, and the session sends the viewers' summed ceilings as the PROBE target; `dispose()` it once the broadcast stops being watched. `setTarget(targetBps: ...)` sends another target, `0` sends none, and `null` follows the viewers again. Ceilings or a target past 2^53 - 1 bps, the most a PROBE target carries, throw `OutOfRangeMoqException`. Client viewers survive reconnects and read `null` while disconnected. - **Authorization.** On `moq-lite-07-wip` (opt in with `ConnectOptions(versions: ...)`), listen to `moq.grants().updates()` for the `Grant` the relay sent back and each change, `null` until it answers and while a client session reconnects. diff --git a/doc/lib/go/index.md b/doc/lib/go/index.md index d9b12e693d9..26216e9c9a5 100644 --- a/doc/lib/go/index.md +++ b/doc/lib/go/index.md @@ -132,7 +132,7 @@ shows which module holds each. - **Media reads.** `media.TrackConsumer`, `audio.Consumer`, and `video.Consumer` return a `Read` in order: `ReadFrame`, a `ReadGap` where media was lost, or a `ReadEnd` where the publisher stopped its encoder. A gap breaks continuity. An encoder end is not stream EOF: following frames before its endpoint are the old encoder's drain; the first at or past it starts fresh. A decoder reads everything the track still holds unless a live player sets `Start: media.DecodeStartLatest` in its decoder output, which joins at the newest group and skips the cached backlog. - **Decoded frames hold decoder buffers.** `Close` each `VideoDecodedFrame` promptly, or the decoder stalls. `Resize` is best effort, so read each frame's `Width()` and `Height()`. - **Closing.** `client.Close()` drains for up to one second and returns an error on failure. `session.Shutdown()` requests that drain immediately; `session.Closed(ctx)` waits for its result, and `session.Cancel(code)` closes at once. Finish or abort live tracks first. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. -- **Ending cleanly.** When the input ends, finish every track, then call `Finish()` on its `media.CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.Close()` takes it offline. How long to keep it up is yours to decide. A track still open at the close, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). +- **Ending cleanly.** When the input ends, finish every track, then call `Finish()` on its `media.CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.Close()` takes it offline. How long to keep it up is yours to decide. A track still open at the close ends with `Unroutable` (`Internal` over moq-transport, which has no such code), and one released without a finish ends `Dropped`, which readers see as `Internal`. Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). - **Stats.** `Session().Stats()` reports `RTT`, `EstimatedSendRateBps`, and the byte and packet counters (`BytesSent`, `BytesReceived`, `BytesLost`, `PacketsSent`, `PacketsReceived`, `PacketsLost`), and `DatagramsUnrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is nil when the transport does not report it, which is not the same as zero. `Version` names the negotiated protocol. `Dial` returns a connecting session; read `Session().State()` and wait with `StateChanged(ctx, state.Revision)` to observe reconnects. A first dial that reaches no session ends the session with that error rather than retrying. Hold the owned `BandwidthConsumer` returned by `Session().RecvBandwidth()` to receive bandwidth estimates. `Rate()` reads the current estimate in bits per second and `RateChanged(ctx)` waits for a new one. Calling `Cancel()` on the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `Session().Viewers()` divides the connection's receive estimate among the broadcasts you watch, max-min fair by each one's ceiling. `Watch(maxBps)` returns a `Viewer` claiming up to what the largest rendition the player would switch to needs; `Rate()` reads its slice in bits per second, nil while there is no estimate, so hold the current selection. `Update(maxBps)` changes the ceiling, such as when the catalog enables or disables a rendition. Holding a viewer requests PROBE, and the session sends the viewers' summed ceilings as the PROBE target; `Close()` it once the broadcast stops being watched, from the goroutine that last used it. `SetTarget(&bps)` sends another target, zero sends none, and nil follows the viewers again. `Watch`, `Update`, and `SetTarget` return `moq.ErrOutOfRange` when the ceilings or the target would pass 2^53 - 1 bps, the most a PROBE target carries. Client viewers survive reconnects and read nil while disconnected. - **Authorization.** On `moq-lite-07-wip` (opt in with `ClientConfig.Versions`), `Session().Grants()` returns a `GrantConsumer`: `Grant()` is the `Grant` the relay sent back, nil until it answers and while a client session reconnects, and `Changed(ctx)` waits for the next one. `Cancel()` it when done. diff --git a/doc/lib/kt/index.md b/doc/lib/kt/index.md index dd016e06ee0..f6a6458f813 100644 --- a/doc/lib/kt/index.md +++ b/doc/lib/kt/index.md @@ -109,7 +109,7 @@ shows which module holds each. - **Media reads.** `dev.moq.media.TrackConsumer`, `dev.moq.audio.Consumer`, and `dev.moq.video.Consumer` return a `Read` in order: `ReadFrame`, a `ReadGap` where media was lost, or a `ReadEnd` where the publisher stopped its encoder. A gap breaks continuity. An encoder end is not stream EOF: following frames before its endpoint are the old encoder's drain; the first at or past it starts fresh. A decoder reads everything the track still holds unless a live player sets `start = DecodeStart.LATEST` in its decoder output, which joins at the newest group and skips the cached backlog. - **Decoded frames hold decoder buffers.** `close()` each frame from a `dev.moq.video.Consumer` promptly (or `use { }` it), or the decoder stalls. `resize` is best effort, so read each frame's `width()` and `height()`. - **Closing.** `moq.shutdown()` requests a drain immediately; suspending `moq.session.closed()` waits up to one second and reports its result. `Moq.close()`, and so `use { }`, cancels at once. Finish or abort live tracks first. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. -- **Ending cleanly.** When the input ends, finish every track, then call `finish()` on its `CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.end()` takes it offline. How long to keep it up is yours to decide. A track still open at the end, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). +- **Ending cleanly.** When the input ends, finish every track, then call `finish()` on its `CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.end()` takes it offline. How long to keep it up is yours to decide. A track still open at the end ends with `Unroutable` (`Internal` over moq-transport, which has no such code), and one released without a finish ends `Dropped`, which readers see as `Internal`. Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). - **Stats.** `session.stats()` reports `rtt`, `estimatedSendRateBps`, and the byte and packet counters (`bytesSent`, `bytesReceived`, `bytesLost`, `packetsSent`, `packetsReceived`, `packetsLost`), and `datagramsUnrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is `null` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe reconnects. A first dial that reaches no session ends the session with that error rather than retrying. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recvBandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rateChanged()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `moq.viewers()` divides the connection's receive estimate among the broadcasts you watch, max-min fair by each one's ceiling. `watch(maxBps)` returns a `Viewer` claiming up to what the largest rendition the player would switch to needs; `rate()` reads its slice in bits per second, `null` while there is no estimate, so hold the current selection. `update(maxBps)` changes the ceiling, such as when the catalog enables or disables a rendition. Holding a viewer requests PROBE, and the session sends the viewers' summed ceilings as the PROBE target; `close()` it, or leave `use { }`, once the broadcast stops being watched. `setTarget(bps)` sends another target, `0` sends none, and `null` follows the viewers again. Ceilings or a target past 2^53 - 1 bps, the most a PROBE target carries, throw `MoqException.OutOfRange`. Client viewers survive reconnects and read `null` while disconnected. - **Authorization.** On `moq-lite-07-wip` (opt in with `ClientConfig(versions = ...)`), collect `moq.grants().updates()` for the `Grant` the relay sent back and each change, `null` until it answers and while a client session reconnects. Ending the collection cancels the consumer. diff --git a/doc/lib/py/index.md b/doc/lib/py/index.md index 1474b95bd02..478ba5e995d 100644 --- a/doc/lib/py/index.md +++ b/doc/lib/py/index.md @@ -124,7 +124,7 @@ shows which module holds each. - **Media reads.** `media.TrackConsumer`, `moq.audio.Consumer`, and `moq.video.Consumer` yield, in order, frames, a `Gap` where media was lost, and an `End` where the publisher stopped its encoder. A gap breaks continuity. An encoder end is not stream EOF: following frames before its endpoint are the old encoder's drain; the first at or past it starts fresh. A decoder reads everything the track still holds unless a live player sets `start=moq.media.DecodeStart.LATEST` in its decoder output, which joins at the newest group and skips the cached backlog. - **Decoded frames hold decoder buffers.** Drop each frame from a `moq.video.Consumer` promptly, or the decoder stalls. `resize` is best effort, so read each frame's `width()` and `height()`. - **Closing.** `session.shutdown()` requests a drain immediately; `await session.closed()` waits up to one second and reports its result. `cancel(code)` closes at once. Finish or abort live tracks first. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. -- **Ending cleanly.** When the input ends, finish every track, then call `finish()` on its `moq.media.CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.close()` takes it offline. How long to keep it up is yours to decide. A track still open at the close, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). +- **Ending cleanly.** When the input ends, finish every track, then call `finish()` on its `moq.media.CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.close()` takes it offline. How long to keep it up is yours to decide. A track still open at the close ends with `Unroutable` (`Internal` over moq-transport, which has no such code), and one released without a finish ends `Dropped`, which readers see as `Internal`. Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). - **Stats.** `session.stats()` reports `rtt`, `estimated_send_rate_bps`, and the byte and packet counters (`bytes_sent`, `bytes_received`, `bytes_lost`, `packets_sent`, `packets_received`, `packets_lost`), and `datagrams_unrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is `None` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe reconnects. A first dial that reaches no session ends the session with that error rather than retrying. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recv_bandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rate_changed()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `session.viewers()` divides the receive estimate among the broadcasts you watch. `watch(max_bps)` returns a `Viewer` claiming up to a ceiling, the bitrate of the largest rendition you would switch to; its `rate()` reads that viewer's fair share in bits per second, `None` while there is no estimate (keep the current rendition), and `update(max_bps)` changes the ceiling. Holding a viewer requests the estimate like `recv_bandwidth()`, and dropping it releases the claim. The session sends the summed ceilings as its PROBE target; `set_target(bps)` overrides that, `0` sends none, and `None` follows the viewers again. Ceilings or a target past 2^53 - 1 bps, the most a PROBE target carries, raise `moq.Error.OutOfRange`. - **Authorization.** On `moq-lite-07-wip` (opt in with `versions=`), `async for grant in session.grants()` yields the `Grant` the relay sent back and each change, `None` until it answers and while a client session reconnects. `grants().grant()` reads it once; cancelling or leaving an `async with` releases the wait. diff --git a/doc/lib/swift/index.md b/doc/lib/swift/index.md index 82a7a1b64b0..b7673a195b9 100644 --- a/doc/lib/swift/index.md +++ b/doc/lib/swift/index.md @@ -106,7 +106,7 @@ shows which module holds each. - **Media reads.** `Media.TrackConsumer`, `Audio.Consumer`, and `Video.Consumer` yield `Read` cases in order: `.frame`, a `.gap` where media was lost, or an `.end` where the publisher stopped its encoder. A gap breaks continuity. An encoder end is not stream EOF: following frames before its endpoint are the old encoder's drain; the first at or past it starts fresh. A decoder reads everything the track still holds unless a live player sets `start: .latest` in its decoder output, which joins at the newest group and skips the cached backlog. - **Decoded frames hold decoder buffers.** Release each frame from a `Video.Consumer` promptly, or the decoder stalls. `resize` is best effort, so read each frame's `width()` and `height()`. - **Closing.** `session.shutdown()` requests a drain immediately; `try await session.closed()` waits up to one second and reports its result. `session.cancel(code:)` closes at once. Finish or abort live tracks first. moq-lite-05 and newer wait for the peer to receive finished tails; older Lite versions and IETF wait only for transport ACKs. -- **Ending cleanly.** When the input ends, finish every track, then call `try finish()` on its `Media.CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.close()`, or releasing its route, takes it offline. How long to keep it up is yours to decide. A track still open at the close, or released without a finish, ends with `Unroutable` (`Internal` over moq-transport, which has no such code). Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). +- **Ending cleanly.** When the input ends, finish every track, then call `try finish()` on its `Media.CatalogProducer` to finish the catalog, which ends the content. The broadcast stays announced and keeps serving the finished tracks, so a subscriber still catching up can drain them, until `broadcast.close()`, or releasing its route, takes it offline. How long to keep it up is yours to decide. A track still open at the close ends with `Unroutable` (`Internal` over moq-transport, which has no such code), and one released without a finish ends `Dropped`, which readers see as `Internal`. Readers follow a publisher's runs by announcement, not by that error: see [identity and recovery](/concept/identity). - **Stats.** `session.stats()` reports `rtt`, `estimatedSendRateBps`, and the byte and packet counters (`bytesSent`, `bytesReceived`, `bytesLost`, `packetsSent`, `packetsReceived`, `packetsLost`), and `datagramsUnrouted`, the datagrams received for a subscription with no datagram track, reported only on moq-lite 05 and later, which carry datagrams. A field is `nil` when the transport does not report it, which is not the same as zero. `version` names the negotiated protocol. Connect returns a connecting session; read its coherent `state()` snapshot and wait with that snapshot's revision to observe reconnects. A first dial that reaches no session ends the session with that error rather than retrying. Each watcher keeps its own revision. Hold the owned `BandwidthConsumer` returned by `session.recvBandwidth()` to receive bandwidth estimates. `rate()` reads the current estimate in bits per second and `rateChanged()` waits for a new one. Release the consumer or call `cancel()` when finished; releasing the last consumer stops PROBE; the session and stats snapshots do not request it. Client consumers survive reconnects and report no estimate while disconnected. - **Viewers.** `session.viewers()` divides the receive estimate among the broadcasts you watch. `watch(maxBps:)` returns a `Viewer` claiming up to a ceiling, the bitrate of the largest rendition you would switch to; its `rate()` reads that viewer's fair share in bits per second, `nil` while there is no estimate (keep the current rendition), and `update(maxBps:)` changes the ceiling. Holding a viewer requests the estimate like `recvBandwidth()`, and releasing it releases the claim. The session sends the summed ceilings as its PROBE target; `setTarget(targetBps:)` overrides that, `0` sends none, and `nil` follows the viewers again. `watch`, `update`, and `setTarget` throw `MoqError.OutOfRange` when the ceilings or the target would pass 2^53 - 1 bps, the most a PROBE target carries. - **Authorization.** On `moq-lite-07-wip` (opt in with `Client(versions:)`), `for try await grant in session.grants()` yields the `Grant` the relay sent back and each change, `nil` until it answers and while a client session reconnects. Cancelling the iterating task cancels the native wait. diff --git a/js/net/src/origin.ts b/js/net/src/origin.ts index bded4b83e16..2cd64b99fe8 100644 --- a/js/net/src/origin.ts +++ b/js/net/src/origin.ts @@ -262,6 +262,11 @@ interface RequestSlot { readonly handles: Set>; /** Pending plain handles, each ending itself once synced discovery leaves it no route. */ readonly absent: Set<() => void>; + /** + * Called each time the table moves for the path while the slot keeps its route and no takeover + * is pending: the evidence a read stranded on that route waits for. + */ + readonly moved: Set<() => void>; readonly route: Signal; /** The front a detached slot took from the table, closed with its last handle. */ retired?: broadcast.Consumer; @@ -281,6 +286,10 @@ interface Resolution { readonly instance?: Instance.Valid; /** The local broadcast or route entry's identity the front came from, when it did. */ readonly source?: object; + /** Whether the front is the local broadcast, whose word on a track no route news overtakes. */ + readonly local?: boolean; + /** Kept while a cheaper route of the instance has yet to answer: a takeover is pending. */ + readonly pending?: boolean; } /** @@ -862,6 +871,16 @@ class OriginState { * stops serving, it ends, except for an answer retained across an in-place Restart. */ reroute(path: Path.Valid, slot: RequestSlot): void { + const before = slot.route.peek()?.source; + this.#reroute(path, slot); + // Only the same route still winning, with no takeover pending, is news for a read stranded + // on it, as the Rust front unstrands only when the selected route is the serving one. + const after = slot.route.peek(); + if (before === undefined || after?.source !== before || after.pending || slot.failover) return; + for (const moved of [...slot.moved]) moved(); + } + + #reroute(path: Path.Valid, slot: RequestSlot): void { if (this.retained.has(slot)) return; const current = slot.route.peek(); // A pinned slot follows its instance from the start, and a detached one once it resolved. @@ -902,6 +921,10 @@ class OriginState { instance: Instance.Valid | undefined, source: object | undefined, ): void { + // A slot it follows records a pending takeover in `failover`, so one the slot was taken + // out of the table with is stale. + const current = slot.route.peek(); + if (current?.pending) slot.route.set({ ...current, pending: false }); if (instance === undefined) { if (source && this.serves(path, source, instance, true)) return; this.drop(path, slot, unroutable()); @@ -918,7 +941,7 @@ class OriginState { slot.failover = undefined; if (source === local) return; this.retain(path, slot, undefined); - slot.route.set({ front: local, instance, source: local }); + slot.route.set({ front: local, instance, source: local, local: true }); return; } if (!entry?.server) { @@ -1197,7 +1220,12 @@ class OriginState { if (local && this.localWins(path, entry)) { // Nothing reads a remote front the local broadcast replaced, so close its session subscription. this.releaseMaterialized(path); - return { front: local, instance: this.advertisedLocal.peek().get(path)?.instance, source: local }; + return { + front: local, + instance: this.advertisedLocal.peek().get(path)?.instance, + source: local, + local: true, + }; } let cached = this.materialized.get(path); @@ -1223,7 +1251,7 @@ class OriginState { entry.server.enqueue(path); // The same instance over another route: the old front serves until the new one answers. const kept = cached?.instance === instance ? cached : undefined; - return { front: kept?.front, instance, source: kept?.source }; + return { front: kept?.front, instance, source: kept?.source, pending: true }; } } @@ -2344,6 +2372,7 @@ export class Consumer implements Reader { blind: 0, handles: new Set(), absent: new Set(), + moved: new Set(), route: new Signal(undefined), pin, }; @@ -2365,6 +2394,7 @@ export class Consumer implements Reader { blind: 0, handles: new Set(), absent: new Set(), + moved: new Set(), route: new Signal(this.#state.route(path, {})), }; slot = created; @@ -2402,7 +2432,7 @@ export class Consumer implements Reader { const front = instance === undefined || resolution?.instance === instance ? offered : undefined; if (logical && source?.instance !== undefined && resolution?.instance === source.instance) { // Between routes of the instance, readers wait for the next one to answer. - if (front) logical.serve(front); + if (front) logical.serve(front, resolution?.local); else logical.park(); return handle; } @@ -2414,7 +2444,7 @@ export class Consumer implements Reader { // route may answer a re-request with another stream, so nothing re-sources it. const carried = resolution?.instance !== undefined && !Instance.isMinted(resolution.instance); logical = front && carried ? new Front() : undefined; - logical?.serve(front as broadcast.Consumer); + logical?.serve(front as broadcast.Consumer, resolution?.local); handle = logical ? logical.consumer : front?.clone(); if (handle) { hooks.stampPath(handle, relative); @@ -2452,6 +2482,18 @@ export class Consumer implements Reader { release(); }; if (!announced) taken.absent.add(absent); + // The table moved while the same route may still win: a read stranded on it asks again. + // Queued behind the route's own flush, so a route that changed is served first, and + // checked again then: a takeover gone pending meanwhile is no news. + const moved = () => { + const at = route.peek()?.source; + queueMicrotask(() => { + const now = route.peek(); + if (released || now?.source !== at || now?.pending || taken.failover) return; + logical?.moved(); + }); + }; + taken.moved.add(moved); const unsubscribe = route.subscribe((resolution) => { own(resolution); @@ -2464,6 +2506,7 @@ export class Consumer implements Reader { released = true; unsubscribe(); taken.absent.delete(absent); + taken.moved.delete(moved); logical?.conclude(); logical = undefined; handle?.close(); diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index da641f81f51..0d3531f56bc 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -22,6 +22,7 @@ import { overrideBroadcastWire, wireOf } from "./wire.ts"; const INFO: track.Info = { compression: "none", timescale: null, maxAge: null, priority: 0, refresh: null }; const INSTANCE = Instance.parse("01900000-0000-7000-8000-000000000001"); +const NEWER = Instance.parse("01900000-0000-7000-8000-000000000002"); const path = Path.from("live"); const enc = new TextEncoder(); const dec = new TextDecoder(); @@ -850,6 +851,37 @@ test("a replacement that answers and then aborts before its first group leaves t close(); }); +// Mirrors `a_stranded_track_still_draining_asks_again_when_the_table_moves` in rs/moq-net/src/model/front.rs: +// a replacement losing its route judged the route, so it refuses nothing the next route serves. +test("a replacement that loses its route before its first group leaves the next route free to serve", async () => { + const front = new Front(); + const [a, b, c] = [new BroadcastProducer(), new BroadcastProducer(), new BroadcastProducer()]; + const [ta, tb, tc] = [copy(a), copy(b), copy(c)]; + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0n); + await settle(); + expect(drain(sub)).toEqual([0n]); + + front.serve(b.consume()); + await settle(); + tb.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + ta.write(1n); + await settle(); + expect(drain(sub)).toEqual([1n]); + + front.serve(c.consume()); + tc.write(2n); + await settle(); + expect(drain(sub)).toEqual([2n]); + expect(sub.closed.peek()).toBeUndefined(); + + sub.close(); + front.consumer.close(); + for (const broadcast of [a, b, c]) broadcast.close(); +}); + // Mirrors the Rust front: an incumbent that finished cleanly ends the track whoever serves it. test("a replacement that aborts before its first group after the incumbent finished ends the track cleanly", async () => { const { a, b, swap, active, close } = await setup(); @@ -1236,19 +1268,368 @@ test("a copy that fails mid-serve on a route still serving is subscribed afresh" close(); }); -// Mirrors `a_copy_verdict_aborts_without_querying_again` in rs/moq-net/src/model/front.rs. -test("a copy that loses its route mid-serve on a route still serving ends the read", async () => { - const { a, active, close } = await setup(); +// Mirrors `an_unroutable_route_strands_the_track_until_the_table_moves` in rs/moq-net/src/model/front.rs. +for (const delivered of [true, false]) { + test(`a copy that loses its route ${delivered ? "mid-serve" : "before delivering"} waits for the next route`, async () => { + const { a, b, active, swap, close } = await setup(); + const ta = copy(a); + const sub = active.track("seq").subscribe(); + if (delivered) { + ta.write(0n); + expect(drain(sub)).toEqual([0n]); + } + + ta.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + expect(sub.closed.peek()).toBeUndefined(); + + const tb = copy(b); + const disposeB = await swap(); + tb.write(1n); + expect(drain(sub)).toEqual([1n]); + + sub.close(); + disposeB(); + close(); + }); +} + +// Mirrors `an_unroutable_route_strands_the_track_until_the_table_moves` in rs/moq-net/src/model/front.rs: +// the same route winning again once the table moves asks it afresh. +test("a read stranded on a route asks it again once the table moves", async () => { + const { origin, a, b, active, request, close } = await setup(); + const first = copy(a); + const sub = active.track("seq").subscribe(); + first.write(0n); + await settle(); + expect(drain(sub)).toEqual([0n]); + + first.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + // The route could serve again, but reading the handle is no news: the read keeps waiting. + const second = copy(a); + second.write(1n); + expect(request.active.peek()).toBe(active); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + expect(sub.closed.peek()).toBeUndefined(); + + // A dearer route of the instance moves the table while the same route still wins. + const disposeB = serve(origin, b, 10n); + await settle(); + await settle(); + expect(drain(sub)).toEqual([1n]); + + sub.close(); + disposeB(); + close(); +}); + +// Mirrors `a_pending_takeover_leaves_a_stranded_track_waiting` in rs/moq-net/src/model/front.rs: +// the table moving toward a cheaper route that has yet to answer is no news for the stranded one. +for (const answers of [true, false]) { + test(`a read stranded on a route waits out a takeover still pending, until it ${answers ? "answers" : "leaves"}`, async () => { + const { origin, a, b, active, close } = await setup(); + const first = copy(a); + const sub = active.track("seq").subscribe(); + first.write(0n); + await settle(); + expect(drain(sub)).toEqual([0n]); + + first.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + // The stranded route could serve again, but a cheaper one is announced and has not answered. + const second = copy(a); + second.write(1n); + const next = wireOf(origin).receive(path, WireRoute.normalize({ instance: INSTANCE, cost: 1n })); + const asked = next.requested(); + await settle(); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + expect(sub.closed.peek()).toBeUndefined(); + + if (answers) { + // It answers, and takes over. + const tb = copy(b); + const { value: pending } = await asked.next(); + pending.serve(b.consume()); + await settle(); + await settle(); + tb.write(2n); + await settle(); + expect(drain(sub)).toEqual([2n]); + } else { + // It leaves first: the stranded route wins again, which is the news the read waited for. + next.close(); + await settle(); + await settle(); + expect(drain(sub)).toEqual([1n]); + } + + sub.close(); + next.close(); + close(); + }); +} + +// The same, after another instance won the path: the request leaves the table, and its own +// instance's takeover leaving is still the news the stranded read waits for. +test("a read stranded on a route asks it again once a pending takeover leaves, after another instance won", async () => { + const { origin, a, active, close } = await setup(); + const first = copy(a); + const sub = active.track("seq").subscribe(); + first.write(0n); + await settle(); + expect(drain(sub)).toEqual([0n]); + + first.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + const second = copy(a); + second.write(1n); + const next = wireOf(origin).receive(path, WireRoute.normalize({ instance: INSTANCE, cost: 1n })); + await settle(); + const newer = wireOf(origin).receive(path, WireRoute.normalize({ instance: NEWER, cost: 1n })); + await settle(); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + + // The takeover leaves: the stranded route wins its instance again. + next.close(); + await settle(); + await settle(); + expect(drain(sub)).toEqual([1n]); + expect(sub.closed.peek()).toBeUndefined(); + + sub.close(); + newer.close(); + close(); +}); + +// Mirrors the refusal half of `a_stranded_track_still_draining_asks_again_when_the_table_moves` in +// rs/moq-net/src/model/front.rs: a replacement refusing the track as unroutable judged its route. +test("a replacement that refuses a track as unroutable is asked again once the table moves, the replaced copy feeding meanwhile", async () => { + const front = new Front(); + const [a, b] = [new BroadcastProducer(), new BroadcastProducer()]; const ta = copy(a); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0n); + await settle(); + expect(drain(sub)).toEqual([0n]); + + const requested = wireOf(b).requested(); + front.serve(b.consume()); + const request = await requested; + if (!request) throw new Error("expected on-demand track request"); + request.reject(new StreamError(StreamCode.Unroutable)); + await settle(); + ta.write(1n); + await settle(); + expect(drain(sub)).toEqual([1n]); + + const tb = copy(b); + tb.write(2n); + front.moved(); + await settle(); + expect(drain(sub)).toEqual([2n]); + expect(sub.closed.peek()).toBeUndefined(); + + sub.close(); + front.consumer.close(); + for (const broadcast of [a, b]) broadcast.close(); +}); + +// The table moving twice in one turn, the second time toward a cheaper route that has yet to +// answer: the first move's news is stale by the time the read could act on it. +test("a read stranded on a route waits out a takeover that went pending in the same turn", async () => { + const { origin, a, b, active, close } = await setup(); + const first = copy(a); const sub = active.track("seq").subscribe(); + first.write(0n); + await settle(); + expect(drain(sub)).toEqual([0n]); + + first.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + const second = copy(a); + second.write(1n); + // A dearer route moves the table while the stranded one still wins, then a cheaper one does. + const dearer = wireOf(origin).receive(path, WireRoute.normalize({ instance: INSTANCE, cost: 10n })); + const next = wireOf(origin).receive(path, WireRoute.normalize({ instance: INSTANCE, cost: 1n })); + const asked = next.requested(); + await settle(); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + expect(sub.closed.peek()).toBeUndefined(); + + const tb = copy(b); + const { value: pending } = await asked.next(); + pending.serve(b.consume()); + await settle(); + await settle(); + tb.write(2n); + await settle(); + expect(drain(sub)).toEqual([2n]); + + sub.close(); + dearer.close(); + next.close(); + close(); +}); + +// A concluded front has no route left to move: the copy that serves it losing its route ends the +// read rather than strand it, even when it serves again after a replacement failed. +test("a concluded front's copy that loses its route ends the read", async () => { + const front = new Front(); + const [a, b] = [new BroadcastProducer(), new BroadcastProducer()]; + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); ta.write(0n); + await settle(); expect(drain(sub)).toEqual([0n]); + front.serve(b.consume()); + await settle(); + tb.producer.abort(new StreamError(StreamCode.Unroutable)); + front.conclude(); + await settle(); + ta.write(1n); + await settle(); + expect(drain(sub)).toEqual([1n]); + ta.producer.abort(new StreamError(StreamCode.Unroutable)); - await expect(sub.recvGroup()).rejects.toMatchObject({ code: StreamCode.Unroutable }); + await settle(); + let ended: unknown; + try { + sub.tryRecvGroup(); + } catch (err) { + ended = err; + } + expect(ended).toBeInstanceOf(StreamError); sub.close(); - close(); + front.consumer.close(); + for (const broadcast of [a, b]) broadcast.close(); +}); + +// Mirrors `a_stranded_track_still_draining_asks_again_when_the_table_moves` in rs/moq-net/src/model/front.rs. +test("a replacement that loses its route is asked again once the table moves, the replaced copy feeding meanwhile", async () => { + const front = new Front(); + const [a, b] = [new BroadcastProducer(), new BroadcastProducer()]; + const ta = copy(a); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0n); + await settle(); + expect(drain(sub)).toEqual([0n]); + + const lost = copy(b); + front.serve(b.consume()); + await settle(); + lost.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + ta.write(1n); + await settle(); + expect(drain(sub)).toEqual([1n]); + + const tb = copy(b); + tb.write(2n); + front.moved(); + await settle(); + expect(drain(sub)).toEqual([2n]); + expect(sub.closed.peek()).toBeUndefined(); + + sub.close(); + front.consumer.close(); + for (const broadcast of [a, b]) broadcast.close(); +}); + +// Mirrors `a_drained_copy_lost_as_unroutable_waits_on_the_serving_source` in rs/moq-net/src/model/front.rs. +test("a replaced copy that loses its route after its replacement did waits on the serving source", async () => { + const front = new Front(); + const [a, b] = [new BroadcastProducer(), new BroadcastProducer()]; + const ta = copy(a); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0n); + await settle(); + expect(drain(sub)).toEqual([0n]); + + const lost = copy(b); + front.serve(b.consume()); + await settle(); + lost.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + ta.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + expect(sub.closed.peek()).toBeUndefined(); + + // The serving source could serve again, but reading it is no news: the read keeps waiting. + const tb = copy(b); + tb.write(1n); + await settle(); + expect(sub.tryRecvGroup()).toBeUndefined(); + + front.moved(); + await settle(); + expect(drain(sub)).toEqual([1n]); + + sub.close(); + front.consumer.close(); + for (const broadcast of [a, b]) broadcast.close(); +}); + +// Mirrors `a_drained_copy_lost_as_unroutable_leaves_the_serving_refusal_standing` in rs/moq-net/src/model/front.rs. +test("a replaced copy that loses its route leaves its replacement's refusal standing", async () => { + const front = new Front(); + const [a, b] = [new BroadcastProducer(), new BroadcastProducer()]; + const ta = copy(a); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe(); + await settle(); + + // The replacement has no such track. + front.serve(b.consume()); + await settle(); + ta.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + let ended: unknown; + try { + sub.tryRecvGroup(); + } catch (err) { + ended = err; + } + expect(ended).toBeInstanceOf(Error); + expect(routeFailed(ended as Error)).toBe(false); + + front.consumer.close(); + for (const broadcast of [a, b]) broadcast.close(); +}); + +// A local broadcast's word on a track is its verdict: no route news can overtake it. +test("a local broadcast failing a track as unroutable ends the read", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const ta = copy(a); + front.serve(a.consume(), true); + const sub = front.consumer.track("seq").subscribe(); + ta.write(0n); + await settle(); + expect(drain(sub)).toEqual([0n]); + + ta.producer.abort(new StreamError(StreamCode.Unroutable)); + await settle(); + expect(() => sub.tryRecvGroup()).toThrow(); + + front.consumer.close(); + a.close(); }); test("between routes a reader keeps reading the copy it has", async () => { diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index cc73611d8c4..ac9a55e89f7 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -105,6 +105,13 @@ export function routeFailed(err: Error): boolean { return err instanceof SessionError || sessionClosed(err); } +// Whether `err` from `source` judges its route rather than the track: a received route that lost +// the content refuses nothing, and the read waits for the next route. A local broadcast's word is +// the track's verdict. Mirrors the Rust front's `strands`. +function strands(err: Error, source: Source): boolean { + return unroutable(err) && !source.local; +} + // Whether `err` says the route is going away rather than that it gave up on the copy: its // answer is no verdict, and the next route is asked. Mirrors the Rust front's `closing`. function closing(err: Error): boolean { @@ -174,11 +181,14 @@ class Source { readonly wire: BroadcastWire; /** The table's handle this was served from, so serving it again is a no-op. */ readonly from: broadcast.Consumer; + /** The local broadcast rather than a received route: its word on a track is the verdict. */ + readonly local: boolean; #consumer: broadcast.Consumer; #holds = 0; - constructor(from: broadcast.Consumer) { + constructor(from: broadcast.Consumer, local = false) { this.from = from; + this.local = local; this.#consumer = from.clone(); this.wire = wireOf(this.#consumer); } @@ -212,6 +222,8 @@ class Route { * a replacement failing first leaves it serving. The Rust front only swaps once one delivers. */ incumbent?: Source; + /** The source a read idles on, having lost the route; cleared once asked again or replaced. */ + #stranded?: Source; /** Bumped on every change above, waking parked reads. */ readonly changed = new Signal(0); readonly readers = new Set(); @@ -230,6 +242,7 @@ class Route { serve(source: Source): void { // A refusal ends the track: no later source continues it, compatible or not. if (this.concluded || this.refusal || this.source === source) return; + this.#stranded = undefined; this.generation++; // An undelivered source hands over the incumbent it held, not itself. if (!this.incumbent) this.#hold(this.source); @@ -265,6 +278,18 @@ class Route { this.#bump(); } + /** A read idles on `source`, which lost the route, until the table moves: see {@link Front.moved}. */ + strand(source: Source): void { + this.#stranded = source; + } + + /** Whether a read idles on a lost route, which the table moving now ends. */ + unstrand(): boolean { + if (!this.#stranded) return false; + this.#stranded = undefined; + return true; + } + /** Subscribe every reader afresh to the source that still serves, once its copy failed. */ resubscribe(): void { if (!this.source) return; @@ -1015,10 +1040,18 @@ class Reader implements ResumeSource { if (!routeFailed(ended)) return this.#fail(ended); if (route.refusal) return this.#fail(route.refusal); // The route that failed it still serves and is not going away, so no move replaces - // the copy. As in the Rust front, one that delivered nothing is the route refusing - // the track, and one that did is subscribed afresh, unless its source lost the route: - // asking it again could only replace the withdrawal's cause with a refusal. + // the copy. As in the Rust front, a received route that lost the content judged the + // route, not the track: the read waits for the next route, since asking the same one + // again could only replace the withdrawal's cause with a refusal, and that route's + // news follows on the same session. A local broadcast's word is the track's verdict. + // Otherwise one that delivered nothing is the route refusing the track, and one that + // did is subscribed afresh. A concluded route has no front left to move it, so its + // loss ends the read. if (serving && route.source === serving.source && !closing(ended)) { + if (!route.concluded && strands(ended, serving.source)) { + route.strand(serving.source); + return { kind: "idle" }; + } if (!serving.delivered) { route.refuse(ended); return this.#fail(ended); @@ -1053,7 +1086,12 @@ class Reader implements ResumeSource { copy.sub.closed.peek() === undefined, ); if (!incumbent || !this.route.revert(serving.source, incumbent.source)) return false; - if (!closing(failed)) this.route.refuse(failed); + if (!closing(failed)) { + // A replacement losing its route judged the route: the replaced copy keeps feeding the + // read, which asks the replacement again once the table moves, as the Rust front does. + if (strands(failed, serving.source)) this.route.strand(serving.source); + else this.route.refuse(failed); + } return true; } @@ -1940,9 +1978,9 @@ export class Front { } /** Serve every track from `from` from now on, once each copy proves compatible. */ - serve(from: broadcast.Consumer): void { + serve(from: broadcast.Consumer, local = false): void { if (this.#concluded || this.#source?.from === from) return; - const source = new Source(from); + const source = new Source(from, local); source.retain(); const previous = this.#source; this.#source = source; @@ -1951,6 +1989,23 @@ export class Front { previous?.release(); } + /** + * The table moved for the path while the same route still wins, with no takeover pending: a + * read stranded on a lost route asks the serving source again, the evidence it waited for. + * Mirrors the Rust front's `unstrand`. + */ + moved(): void { + const current = this.#source; + if (this.#concluded || !current) return; + for (const route of this.#routes.values()) { + if (!route.unstrand()) continue; + // Read from the serving source: subscribe afresh. Otherwise the read is on a copy the + // serving source replaced, which takes over again as any handover does. + if (route.source === current) route.resubscribe(); + else this.#splice(route, current); + } + } + /** Between routes: readers keep what they have and wait for the next one. */ park(): void { if (this.#concluded || !this.#source) return; @@ -2021,8 +2076,11 @@ export class Front { (err: unknown) => { if (this.#source !== source) return; const error = err instanceof Error ? err : new Error(String(err)); - // A route on its way out is no verdict: the next one is asked. - if (!routeFailed(error)) route.refuse(error); + // A route that lost the content leaves the replaced copy feeding the read, which asks + // again once the table moves, as the Rust front's refusal does. A route on its way + // out is no verdict: the next one is asked. + if (strands(error, source)) route.strand(source); + else if (!routeFailed(error)) route.refuse(error); }, ) .finally(() => { diff --git a/quest/m2/README.md b/quest/m2/README.md index 44f31144ac8..bef2f984b36 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -227,7 +227,7 @@ maintainer asked that what remains be unblocked or punted, and picked these on [moq-dev/moq#8768](https://github.com/moq-dev/moq/pull/8768)). m2 keeps only what would need a breaking release had it landed right after the cut: -- ✅ Left m2: [the request loop](/quest/m2/multicast/requests.md) +- ✅ Left m2: [the request loop](https://github.com/moq-dev/moq/pull/8881) to m3's cluster-routing epic (a fixture and a fix, no wire), until it returned to m2 on 2026-11-15 (maintainer, on [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)) and became diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md index 858eac6c026..90fbcf554c3 100644 --- a/quest/m2/lite07-finalize.md +++ b/quest/m2/lite07-finalize.md @@ -129,7 +129,7 @@ maintainer picked these on [moq-dev/moq#8768](https://github.com/moq-dev/moq/pull/8768)), applying the maintainer's rule that m2 keeps only what would need a breaking release had it landed right after the cut: -- ✅ [The request loop](/quest/m2/multicast/requests.md) moved +- ✅ [The request loop](https://github.com/moq-dev/moq/pull/8881) moved to m3 and no longer gates this: a fixture and a fix with no wire change. It returned to m2 on 2026-11-15 and still does not gate this. - ✅ [Route layer acceptance](/quest/m2/cluster-routes/acceptance.md) stays diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index a9c81669e2e..ceac46221a9 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -98,7 +98,7 @@ Settled: dead hop remembered past the routes that name it. Routing state is a hint. A request is confirmed only once its upstream is, data is identified by Instance, track, group, and frame, and a stale or stalled route is left - on evidence ([requests](/quest/m2/multicast/requests.md), + on evidence ([requests](https://github.com/moq-dev/moq/pull/8881), [route health](/quest/m3/route-health/README.md)). What a hold protected, a claim flickering while routes settle, becomes a reader's local debounce from unavailable to offline, never on the wire. This reverses items 19 and @@ -147,7 +147,7 @@ flagged as costly, on has no path, chain, or Instance, so a moq-transport relay announcing back a name it learned from us cannot be told from a nearer publisher. That is unsupported: it must cost bounded work and touch no other - broadcast ([requests](/quest/m2/multicast/requests.md)), and nothing + broadcast ([requests](https://github.com/moq-dev/moq/pull/8881)), and nothing detects it (rejected: ignoring a claim for a name already announced to that session, which also hides a real second publisher behind it). 11. ✅ **Redundant ingest is one Instance from synchronized encoders.** Two diff --git a/quest/m2/multicast/accept.md b/quest/m2/multicast/accept.md index 95c220c747e..f931314f652 100644 --- a/quest/m2/multicast/accept.md +++ b/quest/m2/multicast/accept.md @@ -44,5 +44,4 @@ Public API: none. Wire: none expected. ## Required - [The switch](/quest/m2/multicast/switch.md) - the sessions these cases run on -- [Requests](/quest/m2/multicast/requests.md) - a request fails closed and follows its Instance - [The proof](/quest/m2/multicast/proof.md) - what these cases need not show diff --git a/quest/m2/multicast/proof.md b/quest/m2/multicast/proof.md index 0343dc184ac..109851d473d 100644 --- a/quest/m2/multicast/proof.md +++ b/quest/m2/multicast/proof.md @@ -41,7 +41,7 @@ What to show, each a lemma about one message handler: State plainly what the theorem does not give, and where each is covered: -- No loop at every instant: [requests](/quest/m2/multicast/requests.md) fail +- No loop at every instant: [requests](https://github.com/moq-dev/moq/pull/8881) fail closed instead. - A cost that depends on who subscribed: [warmth](/quest/m2/multicast/warmth.md) argues it separately. diff --git a/quest/m2/multicast/requests.md b/quest/m2/multicast/requests.md deleted file mode 100644 index dd28f3db54d..00000000000 --- a/quest/m2/multicast/requests.md +++ /dev/null @@ -1,88 +0,0 @@ -# [L] A request treats routing as a hint - -## Goal - -No subscription, fetch, or track request depends on routing state being -fresh. A relay confirms a request only once its upstream has, and serves -every request for one Instance and track from one upstream subscription, -whichever origin node the requests named. A request whose route ends, is -refused as unroutable, or loops moves to another route of its Instance -instead of ending the track, hanging, or ending with `Unroutable` while its -relay shows the broadcast available. No group crosses one link twice for one -subscription, and the old and new copies of a move overlap for a bounded -time. Tests on real lite-07 relays with mocked time pin each. - -## Plan - -Decided 2026-11-15 (maintainer, /quest-plan): this is the data half of -"evidence, not timers" ([epic](/quest/m2/multicast/README.md) item 4). With -no hold and no doubt, selected routes may loop or go stale while news is in -flight, and that is safe only if a request fails closed and follows its -Instance. It started as the request loop quest, which this absorbs. - -Decided 2026-11-15 (maintainer, on -[moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832)): ✅ m2 -(rejected: m3). Relays that show a broadcast available while ending readers' -subscriptions should block the milestone. - -Decided 2026-11-13 (maintainer, /quest-plan on the BGP failure modes): - -- ✅ A fixture on real relays, with no TLA+ for pending requests and - aggregation (rejected: adding them to the route model). -- ✅ A relay's pending upstream request follows its route when the selected - next hop changes, even before the request is answered, so a loop clears - with the routes (rejected: a hop count on every request that fails one at - the limit, which adds a field to SUBSCRIBE, FETCH, and the other requests - and turns a loop that would heal into an error the subscriber sees). - -Why a request cannot find its own loop: once relays aggregate requests, one -fed around a loop meets a relay already asking upstream and joins it, so -neither a nonce nor a carried path ever returns to its sender. Nothing -errors and nobody is fed. What holds instead: a pending request confirms -nothing downstream, and every relay re-asks when its selected route changes. - -Hazards to reproduce first, from the 2026-11-15 audit and the acceptance -fixtures: - -- `route_failover::live_mesh_link_failure_keeps_presence` cuts all but one - of a publishing relay's links on a 34-relay mesh, and 3 readers on relays - that stayed available end with `Unroutable`. Flip it once they read on. -- An upstream that loses its own route resets the subscription `Unroutable` - and sends the route news on another stream. If the reset is handled first, - the front aborts a track that has delivered and never splices it again - (`rs/moq-net/src/model/front.rs` `track_ended`), while - `rs/moq-net/src/model/resume.rs` treats an unroutable copy as waiting for - another route. Found by reading; reproduce it two hops below a live - upstream that loses its route, and for a fresh TRACK naming a route that - just ended. -- During a move the old and new upstream subscriptions overlap with no time - bound if the old route keeps winning, and the newest group is asked for - again. -- Two route entries for one claim through one session send two SUBSCRIBEs - that differ only in depth. -- A moq-transport peer that announces back a name it learned from this - relay looks like a nearer publisher, and a request sent to it comes - straight back. Unsupported - ([epic](/quest/m2/multicast/README.md) item 10): pin that the pair of - requests stays pending without retrying in a loop, and that other - broadcasts on both sessions read on. - -The loop fixture: form the loop the churn sweeps found (two unrelated -failures at once, so two relays briefly route a publisher through each -other), subscribe at both while it lasts, then let the routes settle. Both -subscribers receive the next group and no upstream request is left pending. -Cover a request answered before the loop and one still unanswered when it -forms. - -Fix in the per-reader path, which stays: the producer-replacement redesign -that would have deleted it was dropped 2026-11-15 (maintainer, -/quest-complete). Run the `verify-failover` and `verify-resume` skills on -what they cover. - -Public API: none. Wire: none expected. If a refusal must tell "this route is -gone" from "no such track", that is a lite-07 wire change to report, which is -why this sits in m2. - -## Related - -- [moq-dev/moq#8734](https://github.com/moq-dev/moq/pull/8734) - its doubt rule lets a publisher leaving three or more relays at once loop a request until the news arrives diff --git a/quest/m2/multicast/switch.md b/quest/m2/multicast/switch.md index 21ab958223a..4ff344cf7c3 100644 --- a/quest/m2/multicast/switch.md +++ b/quest/m2/multicast/switch.md @@ -27,7 +27,11 @@ is a decision to list in the PR: Lite-06 Chain and its Setup Parameter: the nodes that announced it again across a scoped grant or an older version. - A request keeps its Instance and the origin node its first hop chose, and - loses Route Depth. + loses Route Depth. That ends the second subscription a move from a claim's + route to its answer's opens on one link, which + `route_failover::residual_two_depths_of_one_claim_subscribe_twice` pins + ([moq-dev/moq#8881](https://github.com/moq-dev/moq/pull/8881)): flip it to + one subscription. - An endpoint that never forwards declares no Node ID. A peer holding a scoped publish grant sends claims and no routes, and the relay announces them under its own Node ID. Peers that trust each other with the whole diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index 8aa8595761b..205eda1a9eb 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -23,7 +23,7 @@ a node that says it carries a broadcast). What stands and what goes: - ✅ Joins stop at the first relay carrying the Instance, which the route layer gives with no message - ([requests](/quest/m2/multicast/requests.md)). + ([requests](https://github.com/moq-dev/moq/pull/8881)). - ✅ How a relay learns of a warm neighbour off its path is picked from measurements ([warmth](/quest/m2/multicast/warmth.md)), which [the implementation](/quest/m3/cache-routing/select.md) then builds. diff --git a/rs/moq-ffi/src/audio.rs b/rs/moq-ffi/src/audio.rs index e13d7bb108f..c186308a35a 100644 --- a/rs/moq-ffi/src/audio.rs +++ b/rs/moq-ffi/src/audio.rs @@ -396,8 +396,9 @@ impl TryFrom for moq_audio::Frame { /// is PCM in the format declared by the [`MoqAudioEncoderInput`] /// passed at publish time. /// -/// Released before its finish, or still open when its broadcast closes, it ends its track with -/// `Unroutable`, which a moq-transport reader sees as `Internal`: its wire has no such code. +/// Released before its finish, it ends its track with `Dropped`, which readers elsewhere see as +/// `Internal`. Still open when its broadcast closes, it ends it with `Unroutable`, which a +/// moq-transport reader sees as `Internal`: its wire has no such code. #[derive(uniffi::Object)] pub struct MoqAudioProducer { inner: std::sync::Mutex>, @@ -420,7 +421,7 @@ impl crate::producer::Child for MoqAudioProducer { impl Drop for MoqAudioProducer { fn drop(&mut self) { - crate::producer::Child::abort(self, moq_net::Error::Unroutable); + crate::producer::Child::abort(self, moq_net::Error::Dropped); } } diff --git a/rs/moq-ffi/src/datagram.rs b/rs/moq-ffi/src/datagram.rs index c713b70a0a3..8a0577ca86d 100644 --- a/rs/moq-ffi/src/datagram.rs +++ b/rs/moq-ffi/src/datagram.rs @@ -99,7 +99,7 @@ impl Child for MoqDatagramProducer { impl Drop for MoqDatagramProducer { fn drop(&mut self) { - Child::abort(self, moq_net::Error::Unroutable); + Child::abort(self, moq_net::Error::Dropped); } } diff --git a/rs/moq-ffi/src/json.rs b/rs/moq-ffi/src/json.rs index 436d6fe6c57..4293463ea02 100644 --- a/rs/moq-ffi/src/json.rs +++ b/rs/moq-ffi/src/json.rs @@ -88,8 +88,9 @@ mod tests { /// in a fresh group, by a task this producer runs until it is finished or dropped. That joiner /// still reads the cached group first. /// -/// Released before its finish, or still open when its broadcast closes, it ends its track with -/// `Unroutable`, which a moq-transport reader sees as `Internal`: its wire has no such code. +/// Released before its finish, it ends its track with `Dropped`, which readers elsewhere see as +/// `Internal`. Still open when its broadcast closes, it ends it with `Unroutable`, which a +/// moq-transport reader sees as `Internal`: its wire has no such code. #[derive(uniffi::Object)] pub struct MoqJsonSnapshotProducer { inner: Arc>>>, @@ -135,7 +136,7 @@ impl Child for MoqJsonSnapshotProducer { impl Drop for MoqJsonSnapshotProducer { fn drop(&mut self) { - self.abort(moq_net::Error::Unroutable); + self.abort(moq_net::Error::Dropped); } } @@ -280,8 +281,9 @@ impl MoqJsonSnapshotConsumer { /// Publishes an ordered log of JSON records, one record per append. /// -/// Released before its finish, or still open when its broadcast closes, it ends its track with -/// `Unroutable`, which a moq-transport reader sees as `Internal`: its wire has no such code. +/// Released before its finish, it ends its track with `Dropped`, which readers elsewhere see as +/// `Internal`. Still open when its broadcast closes, it ends it with `Unroutable`, which a +/// moq-transport reader sees as `Internal`: its wire has no such code. #[derive(uniffi::Object)] pub struct MoqJsonStreamProducer { inner: std::sync::Mutex>>, @@ -299,7 +301,7 @@ impl Child for MoqJsonStreamProducer { impl Drop for MoqJsonStreamProducer { fn drop(&mut self) { - self.abort(moq_net::Error::Unroutable); + self.abort(moq_net::Error::Dropped); } } @@ -404,8 +406,9 @@ impl MoqJsonStreamConsumer { /// A join that finds the open group's edits outweigh the track's `refresh` ratio gets the window /// restated in a fresh group, by a task this producer runs until it is finished or dropped. /// -/// Released before its finish, or still open when its broadcast closes, it ends its track with -/// `Unroutable`, which a moq-transport reader sees as `Internal`: its wire has no such code. +/// Released before its finish, it ends its track with `Dropped`, which readers elsewhere see as +/// `Internal`. Still open when its broadcast closes, it ends it with `Unroutable`, which a +/// moq-transport reader sees as `Internal`: its wire has no such code. #[derive(uniffi::Object)] pub struct MoqJsonWindowProducer { inner: Arc>>>, @@ -464,7 +467,7 @@ impl Child for MoqJsonWindowProducer { impl Drop for MoqJsonWindowProducer { fn drop(&mut self) { - self.abort(moq_net::Error::Unroutable); + self.abort(moq_net::Error::Dropped); } } diff --git a/rs/moq-ffi/src/media.rs b/rs/moq-ffi/src/media.rs index 14abfe21b5f..d1050f88ad0 100644 --- a/rs/moq-ffi/src/media.rs +++ b/rs/moq-ffi/src/media.rs @@ -935,8 +935,9 @@ struct MediaStreamProducer { /// Imports a container into a broadcast, from chunks split anywhere such as a raw byte stream. /// -/// Released before its finish, or still open when its broadcast closes, it ends its tracks with -/// `Unroutable`, which a moq-transport reader sees as `Internal`: its wire has no such code. +/// Released before its finish, it ends its tracks with `Dropped`, which readers elsewhere see as +/// `Internal`. Still open when its broadcast closes, it ends them with `Unroutable`, which a +/// moq-transport reader sees as `Internal`: its wire has no such code. #[derive(uniffi::Object)] pub struct MoqMediaContainerProducer { driver: crate::ffi::Background, @@ -946,8 +947,9 @@ pub struct MoqMediaContainerProducer { /// Imports complete encoded frames onto one track. /// -/// Released before its finish, or still open when its broadcast closes, it ends its track with -/// `Unroutable`, which a moq-transport reader sees as `Internal`: its wire has no such code. +/// Released before its finish, it ends its track with `Dropped`, which readers elsewhere see as +/// `Internal`. Still open when its broadcast closes, it ends it with `Unroutable`, which a +/// moq-transport reader sees as `Internal`: its wire has no such code. #[derive(uniffi::Object)] pub struct MoqMediaTrackProducer { inner: std::sync::Mutex>, @@ -956,8 +958,9 @@ pub struct MoqMediaTrackProducer { /// Imports a video byte stream onto one track. /// -/// Released before its finish, or still open when its broadcast closes, it ends its track with -/// `Unroutable`, which a moq-transport reader sees as `Internal`: its wire has no such code. +/// Released before its finish, it ends its track with `Dropped`, which readers elsewhere see as +/// `Internal`. Still open when its broadcast closes, it ends it with `Unroutable`, which a +/// moq-transport reader sees as `Internal`: its wire has no such code. #[derive(uniffi::Object)] pub struct MoqMediaTrackStreamProducer { inner: std::sync::Mutex>, @@ -977,7 +980,7 @@ impl Child for MoqMediaContainerProducer { impl Drop for MoqMediaContainerProducer { fn drop(&mut self) { - self.abort(moq_net::Error::Unroutable); + self.abort(moq_net::Error::Dropped); } } @@ -992,7 +995,7 @@ impl Child for MoqMediaTrackProducer { impl Drop for MoqMediaTrackProducer { fn drop(&mut self) { - self.abort(moq_net::Error::Unroutable); + self.abort(moq_net::Error::Dropped); } } @@ -1007,7 +1010,7 @@ impl Child for MoqMediaTrackStreamProducer { impl Drop for MoqMediaTrackStreamProducer { fn drop(&mut self) { - self.abort(moq_net::Error::Unroutable); + self.abort(moq_net::Error::Dropped); } } diff --git a/rs/moq-ffi/src/producer.rs b/rs/moq-ffi/src/producer.rs index 0c8eeb4b679..f50a971ad4c 100644 --- a/rs/moq-ffi/src/producer.rs +++ b/rs/moq-ffi/src/producer.rs @@ -695,8 +695,9 @@ impl MoqTrackRequest { /// A stream track to which groups and frames can be published. /// -/// Released before its finish, or still open when its broadcast closes, it ends its track with -/// `Unroutable`, which a moq-transport reader sees as `Internal`: its wire has no such code. +/// Released before its finish, it ends its track with `Dropped`, which readers elsewhere see as +/// `Internal`. Still open when its broadcast closes, it ends it with `Unroutable`, which a +/// moq-transport reader sees as `Internal`: its wire has no such code. #[derive(uniffi::Object)] pub struct MoqTrackProducer { phase: std::sync::Mutex, @@ -748,7 +749,7 @@ impl Drop for MoqTrackProducer { fn drop(&mut self) { let _guard = crate::ffi::enter(); // A finished track lives on in its broadcast, and may still take groups below its end. - std::mem::take(self.phase.get_mut().unwrap()).abort(moq_net::Error::Unroutable); + std::mem::take(self.phase.get_mut().unwrap()).abort(moq_net::Error::Dropped); } } @@ -981,7 +982,7 @@ impl Drop for MoqGroupProducer { if let GroupPhase::Writing(group) = std::mem::take(self.inner.get_mut()) && group.poll_closed(&kio::Waiter::noop()).is_pending() { - let _ = group.abort(moq_net::Error::Unroutable); + let _ = group.abort(moq_net::Error::Dropped); } } } @@ -1113,7 +1114,7 @@ mod metadata_tests { drop(binding); assert!(matches!( consumer.poll_read_frame(&kio::Waiter::noop()), - std::task::Poll::Ready(Err(moq_net::Error::Unroutable)) + std::task::Poll::Ready(Err(moq_net::Error::Dropped)) )); } diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index 2ec21c542d1..be4cf4e1120 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -1942,17 +1942,14 @@ async fn track_request_demand_survives_accept() { demand.until_wanted().await.unwrap(); consumer.cancel(); demand.until_unwanted().await.unwrap(); - // Released unfinished, the track aborts, and its watcher keeps the cause. + // Released unfinished, the track ends dropped, which its watcher reports as `Closed`, as a + // track released without an abort. drop(track); broadcast.close().unwrap(); let Err(err) = demand.until_wanted().await else { panic!("the watcher outlived its track"); }; - assert_protocol( - &err, - crate::error::MoqErrorScope::Stream, - crate::error::MoqProtocolKind::Unroutable, - ); + assert!(matches!(err, MoqError::Closed), "{err:?}"); } /// A datagram handler keeps its broadcast alive independently of the stream handler. @@ -4853,6 +4850,18 @@ async fn assert_left(readers: [Arc; N]) { } } +/// Each reader ends without a cause, as a handle released before its finish or abort leaves it: +/// `Dropped`, which keeps `Unroutable` for a route losing the content. +async fn assert_released(readers: [Arc; N]) { + for reader in readers { + let read = kio::clock::timeout(TIMEOUT, reader.read_frame()).await; + let Err(err) = read.expect("the track carried on") else { + panic!("the track ended cleanly"); + }; + assert!(matches!(err, MoqError::Internal { .. }), "{err:?}"); + } +} + /// Closing a broadcast ends every track handle still open on it the way a publisher leaving /// does, and the handles refuse later calls. #[tokio::test(start_paused = true)] @@ -4872,14 +4881,14 @@ async fn closing_a_broadcast_ends_its_open_tracks() { assert!(matches!(media.finish(), Err(MoqError::Closed))); } -/// A track handle released without a finish, as a garbage collector does, ends its track as a -/// publisher leaving rather than a cause-less drop. +/// A track handle released without a finish, as a garbage collector does, ends its track as +/// dropped: its broadcast stays, so no route was lost. #[tokio::test(start_paused = true)] async fn dropping_an_unfinished_track_handle_ends_it() { let broadcast = MoqBroadcastProducer::new().unwrap(); let (raw, media, readers) = children(&broadcast).await; drop((raw, media)); - assert_left(readers).await; + assert_released(readers).await; broadcast.close().unwrap(); } @@ -4949,13 +4958,13 @@ async fn closing_a_broadcast_ends_its_open_data_tracks() { assert!(matches!(json_window.push("1".into(), us(0)), Err(MoqError::Closed))); } -/// A data writer released without a finish ends its track as a publisher leaving. +/// A data writer released without a finish ends its track as dropped. #[tokio::test(start_paused = true)] async fn dropping_an_unfinished_data_writer_ends_it() { let broadcast = MoqBroadcastProducer::new().unwrap(); let (writers, readers) = data_children(&broadcast).await; drop(writers); - assert_left(readers).await; + assert_released(readers).await; broadcast.close().unwrap(); } diff --git a/rs/moq-ffi/src/video.rs b/rs/moq-ffi/src/video.rs index 31e19910f59..56ec93c09de 100644 --- a/rs/moq-ffi/src/video.rs +++ b/rs/moq-ffi/src/video.rs @@ -599,8 +599,9 @@ impl VideoProducer { /// [`write`](Self::write) accepts a [`MoqVideoFrame`] whose `data` is in the /// pixel format declared by the [`MoqVideoEncoderInput`] passed at publish time. /// -/// Released before its finish, or still open when its broadcast closes, it ends its track with -/// `Unroutable`, which a moq-transport reader sees as `Internal`: its wire has no such code. +/// Released before its finish, it ends its track with `Dropped`, which readers elsewhere see as +/// `Internal`. Still open when its broadcast closes, it ends it with `Unroutable`, which a +/// moq-transport reader sees as `Internal`: its wire has no such code. #[derive(uniffi::Object)] pub struct MoqVideoProducer { /// `None` once finished or aborted. @@ -670,7 +671,7 @@ impl crate::producer::Child for MoqVideoProducer { impl Drop for MoqVideoProducer { fn drop(&mut self) { - crate::producer::Child::abort(self, moq_net::Error::Unroutable); + crate::producer::Child::abort(self, moq_net::Error::Dropped); } } diff --git a/rs/moq-net/src/error.rs b/rs/moq-net/src/error.rs index 1a95d70a295..7690a5e522a 100644 --- a/rs/moq-net/src/error.rs +++ b/rs/moq-net/src/error.rs @@ -471,8 +471,9 @@ pub enum Error { #[error("unknown ALPN: {0}")] UnknownAlpn(String), - /// The producer was dropped without finishing, so the content is incomplete. - #[error("dropped")] + /// The producer was dropped without finishing or aborting, so the content is incomplete: + /// released by its owner, as a binding's garbage collector does, with no cause to report. + #[error("dropped without a finish or abort")] Dropped, /// The handle was already closed by this side. diff --git a/rs/moq-net/src/model/front.rs b/rs/moq-net/src/model/front.rs index aa69378c2a8..122db890cc5 100644 --- a/rs/moq-net/src/model/front.rs +++ b/rs/moq-net/src/model/front.rs @@ -194,6 +194,11 @@ struct Track { /// replaces it: a route being beaten keeps serving until then, and keeps serving if /// the newcomer refuses the track. draining: Option, + /// The source whose route lost the track, failing or refusing it as unroutable, which + /// judges the route, not the track. The track waits for another source, or for the + /// table to move, rather than ask that source again at once or end; a draining copy + /// keeps feeding it meanwhile. + stranded: Option, /// The track finished or aborted: nothing is spliced again, and it stays only /// until it goes unread for the linger. ended: bool, @@ -320,6 +325,7 @@ impl Front { wanted: false, ended: false, draining: None, + stranded: None, }, ); } @@ -389,9 +395,11 @@ impl Front { } let serving_route = self.serving.map(|(_, serving)| serving.route); match best { - // The serving route is still the best one. + // The serving route is still the best one. The table moved, which is the + // evidence a track it lost waits for: ask it again. Some(candidate) if Some(candidate.route) == serving_route => { self.upstream = None; + self.unstrand(actions); } // Already asking through it. Some(candidate) if self.upstream.is_some_and(|upstream| upstream.route == candidate.route) => {} @@ -534,6 +542,7 @@ impl Front { Ok(()) => { track.draining = None; track.lost = None; + track.stranded = None; track.state = TrackState::Spliced { source }; actions.push(Action::Splice { track: name.clone(), @@ -553,6 +562,7 @@ impl Front { delivered: bool, actions: &mut Vec, ) { + let strands = self.strands(); let Some(track) = self.tracks.get_mut(&name) else { return; }; @@ -592,6 +602,18 @@ impl Front { track.lost = Some(err); self.redispatch(name, actions); } + // The route lost the content, which another route of the instance continues: + // wait for one rather than ask this route again, which can only refuse. A copy + // drained from a replaced source leaves the track to the serving one. + Err(err) if delivered && err.unroutable() && strands => { + track.state = TrackState::Idle; + track.lost = Some(err); + if self.serving.is_some_and(|(serving, _)| serving == source) { + track.stranded = Some(source); + } else { + self.redispatch(name, actions); + } + } // A content verdict survives every copy. Under a minted instance, even a route // failure cannot be retried against the same content. Err(err) if delivered => { @@ -618,6 +640,8 @@ impl Front { refusal: (Error, Option), actions: &mut Vec, ) { + let strands = self.strands(); + let serving = self.serving.is_some_and(|(serving, _)| serving == source); let track = self.tracks.get_mut(&name).expect("refusing a known track"); track.state = TrackState::Idle; // A closing source's answer is not a verdict: it is on its way out and @@ -625,6 +649,21 @@ impl Front { if closing { return; } + // A route that lost the content says nothing of the track: wait for another of its + // instance, keeping a draining copy meanwhile. A replaced source's loss leaves the + // track to the serving one, whose own answer stands. + if refusal.0.unroutable() && strands { + if let Some(draining) = track.draining { + track.state = TrackState::Spliced { source: draining }; + } else if !serving { + self.redispatch(name, actions); + return; + } + if serving { + track.stranded = Some(source); + } + return; + } track.refused.insert(source); track.refusal = Some(match track.lost.take() { Some(lost) => (lost, None), @@ -647,7 +686,7 @@ impl Front { return; }; let track = self.tracks.get_mut(&name).expect("dispatching a known track"); - if !track.wanted || track.ended || track.state != TrackState::Idle { + if !track.wanted || track.ended || track.state != TrackState::Idle || track.stranded == Some(source) { return; } if track.refused.contains(&source) { @@ -667,6 +706,45 @@ impl Front { actions.push(Action::Query { track: name, source }); } + /// Whether an unroutable end from the serving source judges its route rather than the + /// track, so the track waits for the table to move. Only a received route can be stale: + /// a local publisher's word is the track's verdict, and no news about it is in flight. A + /// minted instance has no other route to wait for. + fn strands(&self) -> bool { + self.serving + .is_some_and(|(_, candidate)| candidate.carried && !candidate.local) + } + + /// Ask the serving source again for every track it lost as unroutable. One still read from + /// a draining copy keeps reading it until the answer, as during any handover. + fn unstrand(&mut self, actions: &mut Vec) { + let Some((source, _)) = self.serving else { + return; + }; + let mut idle = Vec::new(); + for (name, track) in &mut self.tracks { + if track.stranded != Some(source) { + continue; + } + track.stranded = None; + match track.state { + TrackState::Spliced { source: draining } + if track.draining == Some(draining) && track.wanted && !track.ended && !self.serving_closing => + { + track.state = TrackState::Querying { source }; + actions.push(Action::Query { + track: name.clone(), + source, + }); + } + _ => idle.push(name.clone()), + } + } + for name in idle { + self.redispatch(name, actions); + } + } + fn wanted(&mut self, name: Arc, actions: &mut Vec) { let Some(track) = self.tracks.get_mut(&name) else { return; @@ -1224,8 +1302,6 @@ mod tests { Error::from(crate::StreamError::MalformedTrack), Error::App(7), Error::NotFound, - Error::Unroutable, - Error::from(crate::StreamError::Unroutable), Error::Unsupported, ] { let mut front = serving(candidate, 100); @@ -1248,6 +1324,330 @@ mod tests { } } + /// A route losing the content judges the route, not the track: whether its copy fails + /// unroutable after delivering or it refuses the track so, the track waits for another + /// source rather than end, and asks the same one again only once the table moves. + #[test] + fn an_unroutable_route_strands_the_track_until_the_table_moves() { + for err in [Error::Unroutable, Error::from(crate::StreamError::Unroutable)] { + let mut front = serving(remote(1), 100); + let lost = Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(err.clone()), + delivered: true, + }; + assert_actions(front.step(lost), &[]); + // The table moved while the same route still wins: ask it again. + let query = [Action::Query { + track: name("video"), + source: 100, + }]; + let same = Event::Selected { + best: Some(remote(1)), + serving_closing: false, + }; + assert_actions(front.step(same.clone()), &query); + // Refused the same way: still waiting, and a reader returning asks nothing. + let refused = Event::TrackInfo { + track: name("video"), + source: 100, + closing: false, + result: Err(err.clone()), + retry: None, + }; + assert_actions(front.step(refused), &[]); + assert_actions(front.step(Event::Wanted { track: name("video") }), &[]); + // Another route serves it. + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + assert_actions( + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }), + &[ + Action::Detach { + source: 100, + drain: false, + }, + Action::Query { + track: name("video"), + source: 200, + }, + ], + ); + assert!(!front.ended()); + // With no route left the front ends, and the track with the route's loss. + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Err(err.clone()), + retry: None, + }); + assert_actions( + front.step(Event::Selected { + best: None, + serving_closing: false, + }), + &[Action::End { err: Error::Unroutable }], + ); + assert_eq!(front.lost("video").map(|err| err.to_string()), Some(err.to_string())); + + // A minted instance has no other route, and a local publisher's word is the track's + // verdict with no news in flight to overtake it: either refusal ends the track, as + // does a local copy failing so. + for (local, carried) in [(false, false), (true, true)] { + let candidate = Candidate { + route: 1, + local, + carried, + }; + let abort = [Action::Abort { + track: name("video"), + err: err.clone(), + retry: None, + }]; + let mut front = querying(candidate, 100); + let refused = Event::TrackInfo { + track: name("video"), + source: 100, + closing: false, + result: Err(err.clone()), + retry: None, + }; + assert_actions(front.step(refused), &abort); + if local { + let mut front = serving(candidate, 100); + let lost = Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(err.clone()), + delivered: true, + }; + assert_actions(front.step(lost), &abort); + } + } + } + } + + /// The table moving toward a route that has yet to answer is no news for a stranded track: + /// it waits for that answer, or asks its own route again once that route wins back. + #[test] + fn a_pending_takeover_leaves_a_stranded_track_waiting() { + let stranded = || { + let mut front = serving(remote(1), 100); + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(Error::Unroutable), + delivered: true, + }); + // A cheaper route wins: ask it, and nothing else. + let cheaper = Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }; + assert_actions(front.step(cheaper.clone()), &[Action::Request { route: 2 }]); + assert_actions(front.step(cheaper), &[]); + front + }; + + // It answers: the track asks it. + let mut front = stranded(); + assert_actions( + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }), + &[ + Action::Detach { + source: 100, + drain: false, + }, + Action::Query { + track: name("video"), + source: 200, + }, + ], + ); + + // It leaves first: the serving route wins again, which is the news the track waited for. + let mut front = stranded(); + assert_actions( + front.step(Event::Selected { + best: Some(remote(1)), + serving_closing: false, + }), + &[Action::Query { + track: name("video"), + source: 100, + }], + ); + } + + /// The serving source refuses a track unroutable while the copy it replaces still drains. + /// Once the table moves, the source is asked again with the draining copy still read, and + /// its answer replaces it. A request for the track after it aborted starts unstranded. + #[test] + fn a_stranded_track_still_draining_asks_again_when_the_table_moves() { + let replaced = || { + let mut front = serving(remote(1), 100); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + let refused = Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Err(Error::from(crate::StreamError::Unroutable)), + retry: None, + }; + assert_actions(front.step(refused), &[]); + front + }; + let query = [Action::Query { + track: name("video"), + source: 200, + }]; + + let mut front = replaced(); + let moved = Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }; + assert_actions(front.step(moved), &query); + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Ok(info()), + retry: None, + }), + &[Action::Splice { + track: name("video"), + source: 200, + }], + ); + + // The draining copy fails for good; a fresh request asks the serving source. + let mut front = replaced(); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(Error::App(7)), + delivered: true, + }), + &[Action::Abort { + track: name("video"), + err: Error::App(7), + retry: None, + }], + ); + front.step(Event::TrackAssigned { + track: name("video"), + now: Instant::now(), + }); + assert_actions( + front.step(Event::Wanted { track: name("video") }), + &[query[0].clone(), Action::Arm { at: None }], + ); + } + + /// A replaced copy that delivered nothing loses its route after its replacement refused + /// the track: the replacement's answer stands, so the track ends with it rather than wait + /// on a route nothing serves from. + #[test] + fn a_drained_copy_lost_as_unroutable_leaves_the_serving_refusal_standing() { + let mut front = serving(remote(1), 100); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + let refused = Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Err(Error::NotFound), + retry: None, + }; + assert_actions(front.step(refused), &[]); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(Error::from(crate::StreamError::Unroutable)), + delivered: false, + }), + &[Action::Abort { + track: name("video"), + err: Error::NotFound, + retry: None, + }], + ); + } + + /// A replaced copy still draining fails unroutable after its replacement refused the + /// track so: the track waits on the serving source, the one it asks once the table moves. + #[test] + fn a_drained_copy_lost_as_unroutable_waits_on_the_serving_source() { + let mut front = serving(remote(1), 100); + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + let unroutable = || Error::from(crate::StreamError::Unroutable); + let refused = Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Err(unroutable()), + retry: None, + }; + assert_actions(front.step(refused), &[]); + let drained = Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(unroutable()), + delivered: true, + }; + assert_actions(front.step(drained), &[]); + assert_actions( + front.step(Event::Selected { + best: Some(remote(2)), + serving_closing: false, + }), + &[Action::Query { + track: name("video"), + source: 200, + }], + ); + } + /// Under a minted instance a re-request could reach another stream, so the track ends /// with the error instead, whether the route is a peer's or a local broadcast. #[test] diff --git a/rs/moq-net/tests/integration/route_failover.rs b/rs/moq-net/tests/integration/route_failover.rs index 09807194186..cecf4ac3df8 100644 --- a/rs/moq-net/tests/integration/route_failover.rs +++ b/rs/moq-net/tests/integration/route_failover.rs @@ -21,7 +21,7 @@ use std::collections::BTreeMap; use std::time::Duration; use futures::channel::mpsc; -use moq_net::{Error, Pattern, Patterns, StreamError, Timestamp, group, origin, stats, track}; +use moq_net::{Error, Pattern, Patterns, Timestamp, group, origin, stats, track}; use support::Kind; use support::harness::{MockConnectOptions, MockPair, connect_mock, peer_with_latency}; @@ -173,9 +173,19 @@ type Got = moq_net::Result<(u64, Duration)>; /// Read every frame of `video` at `path` through `node`, following it from the start. async fn reader(node: &origin::Producer, path: &str, start: kio::clock::Instant) -> mpsc::UnboundedReceiver { + reader_of(node, path, "video", start).await +} + +/// Read every frame of `name` at `path` through `node`, following it from the start. +async fn reader_of( + node: &origin::Producer, + path: &str, + name: &str, + start: kio::clock::Instant, +) -> mpsc::UnboundedReceiver { let remote = node.consume().request(path, None).await.unwrap(); let prefs = track::Subscription::default().with_delay(track::Delay::default().with_max(Duration::from_secs(60))); - let mut sub = remote.track("video").unwrap().subscribe(prefs).await.unwrap(); + let mut sub = remote.track(name).unwrap().subscribe(prefs).await.unwrap(); let (tx, rx) = mpsc::unbounded(); kio::clock::sim::spawn(async move { let _keep = remote; @@ -281,6 +291,48 @@ impl Links { connect_mock(options).await } + /// Frames each link sent, by name. + fn frames(&self) -> BTreeMap { + let mut frames = BTreeMap::new(); + for (tier, role, traffic) in self.registry.snapshot().traffic() { + if role == stats::Role::Publisher { + *frames.entry(tier.as_str().to_string()).or_default() += traffic.frames; + } + } + frames + } + + /// Track subscriptions each link served, by name. + fn subscriptions(&self) -> BTreeMap { + let mut subscriptions = BTreeMap::new(); + for (tier, role, traffic) in self.registry.snapshot().traffic() { + if role == stats::Role::Publisher { + *subscriptions.entry(tier.as_str().to_string()).or_default() += traffic.subscriptions_started; + } + } + subscriptions + } + + /// Track subscriptions each link serves now, by name. + fn open(&self) -> BTreeMap { + let mut open = BTreeMap::new(); + for (tier, role, traffic) in self.registry.snapshot().traffic() { + if role == stats::Role::Publisher { + *open.entry(tier.as_str().to_string()).or_default() += traffic.active_subscriptions(); + } + } + open + } + + /// Control messages of every kind each link sent, by name. + fn control(&self) -> BTreeMap { + let mut sent = BTreeMap::new(); + for (tier, _, totals) in self.registry.snapshot().control() { + *sent.entry(tier.as_str().to_string()).or_default() += totals.tx_messages; + } + sent + } + /// Announce-stream messages each link sent, by name. fn sent(&self) -> BTreeMap { let mut sent = BTreeMap::new(); @@ -538,22 +590,17 @@ async fn live_mesh_withdraw_retracts_once_without_latency() { /// Ports moq-dev/moq#4644's `live_link_failure_fails_over`, which bounded the longest any /// relay went without a path by two hold-downs. Proves every relay serves the broadcast /// again with at most one retraction and one re-announcement each (no hunting), and every -/// reader on a relay whose route stayed usable gets every frame but the pinned ones below. +/// reader on a relay whose route stayed usable gets every frame, once and in order. /// /// Every route left begins at the publisher's relay, the far end of each closed link, so /// doubt keeps no relay off one. A relay whose own link closed moves onto a neighbour's /// direct route, which may be stale too, until the news arrives. Measured over the five -/// publishers: 18 relays retracted and announced again, most within milliseconds, and 6 -/// readers lost their subscription. +/// publishers: 18 relays retracted and announced again, most within milliseconds, and the +/// readers on 3 of them lost their subscription. /// -/// Pins behaviour the contract's continuity goal contradicts, owned by the request-loop -/// quest: a reader on a relay that stayed available may end with `Unroutable` (3 readers, -/// all for publisher 22, listed in [`UNROUTABLE`]). The replaced doubt rule, which held -/// those relays instead, ended none of them. -/// The (publisher, relay) readers [`live_mesh_link_failure_keeps_presence`] pins as ending -/// with `Unroutable` though their relay stayed available. -const UNROUTABLE: [(usize, usize); 3] = [(22, 17), (22, 24), (22, 31)]; - +/// A stale route resets a subscription `Unroutable` before its withdrawal arrives (3 +/// readers' relays, all for publisher 22). That judges the route, not the track, so those +/// readers wait for the next route rather than end. #[kio::test] async fn live_mesh_link_failure_keeps_presence() { let version = lite07(); @@ -636,11 +683,8 @@ async fn live_mesh_link_failure_keeps_presence() { let live = Liveness::of(rx, &feed); if !live.seamless() { lost_readers += 1; - // Pinned: only these readers end with `Unroutable` on a relay that stayed available. - let pinned = UNROUTABLE.contains(&(publisher, *relay)) - && matches!(live.ended, Some(Error::Stream(StreamError::Unroutable))); assert!( - dark.contains(relay) || pinned, + dark.contains(relay), "publisher {publisher}: relay {relay} kept a usable route but its reader got {live:?}" ); } @@ -660,3 +704,404 @@ async fn live_mesh_link_failure_keeps_presence() { PUBLISHERS.len() ); } + +/// Seeded random meshes of 5 to 8 relays at random link latencies, the publisher on relay +/// 0 and a reader on every other, each cut one or two links at once. These are the seeds +/// a sweep of 400 found with a reader ending on a relay that stayed available. +/// +/// In each, a relay's front moves as the news of the cut arrives, for a moment onto a +/// stale route whose next hop already lost its own: that hop refuses the fresh track, or +/// resets a copy that delivered, `Unroutable`. Proves the refusal judges the route, not the +/// track: the front waits for the next route, and no reader on a relay that stayed +/// available ends, nor waits for good on any relay. +#[kio::test] +async fn a_stale_route_refusing_a_track_ends_no_reader() { + for seed in [13, 17, 92, 114] { + stale_route_case(seed).await; + } +} + +async fn stale_route_case(seed: u64) { + let mut rng = support::Rng(seed); + let n = rng.range(5, 8) as usize; + let nodes: Vec<_> = (1..=n as u64).map(crate::mesh_withdraw::produce_origin).collect(); + let mut edges = Vec::new(); + for i in 1..n { + edges.push((rng.pick(i), i)); + } + for _ in 0..rng.range(1, n as u64) { + let (a, b) = (rng.pick(n), rng.pick(n)); + if a != b && !edges.contains(&(a.min(b), a.max(b))) { + edges.push((a.min(b), a.max(b))); + } + } + let mut links = Vec::new(); + for &(a, b) in &edges { + let latency = Duration::from_millis(rng.range(1, 80)); + links.push(Some(peer_with_latency(lite07(), &nodes[a], &nodes[b], latency).await)); + } + let start = kio::clock::now(); + let broadcast = nodes[0].route("live", origin::RouteOptions::hidden()).unwrap(); + let track = broadcast.broadcast().create_track("video", timed()).unwrap(); + let mut feed = Feed::new(track, start, MESH_STEP); + broadcast.announce().unwrap(); + kio::clock::sleep(Duration::from_secs(1)).await; + let mut watched: Vec<_> = nodes.iter().map(|node| Watch::new(node, "live", start)).collect(); + let mut readers = Vec::new(); + for (relay, node) in nodes.iter().enumerate().skip(1) { + readers.push((relay, reader(node, "live", start).await)); + } + feed.run(5).await; + for _ in 0..rng.range(1, 2) { + if let Some(pair) = links[rng.pick(edges.len())].take() { + cut(&pair); + } + } + feed.run(30).await; + kio::clock::sleep(Duration::from_secs(1)).await; + + let dark: Vec<_> = (watched.iter_mut()) + .map(|watch| { + watch.seen()[1..] + .iter() + .any(|seen| !seen.available || matches!(seen.kind, Kind::End | Kind::Offline)) + }) + .collect(); + for (relay, rx) in &mut readers { + let live = Liveness::of(rx, &feed); + match &live.ended { + // A relay that lost every route may end its readers. + Some(_) => assert!(dark[*relay], "seed {seed}: relay {relay} stayed available: {live:?}"), + None => assert!(live.seamless(), "seed {seed}: relay {relay}: {live:?}"), + } + } + drop(broadcast); +} + +/// `P` publishes `live` to `U` and `B`, and `D` pulls both, preferring `U`, whose link takes +/// two seconds each way; `E` pulls `D`. `U`'s link to `P` times out and comes back at once, +/// and `U` serves `live` again a second later, faster than its news reaches `D`. +/// +/// `U` resets the subscription below it `Unroutable` only once it lost every route, and it +/// sends that loss to `D` on the same session as a retraction, which the announce cursor +/// neither holds nor cancels when `U` announces again. Proves `D`'s front hears the +/// retraction and moves on, whichever arrives first, so the wait on a stranded track ends: +/// readers on `D` and on `E`, two hops below `U`, get every frame once, in order. +#[kio::test] +async fn an_upstream_flapping_faster_than_its_news_strands_no_reader() { + let links = Links::new(); + let [p, u, b, d, e] = [1, 2, 3, 4, 5].map(produce_origin); + let p_u = links.link("p-u", &p, &u, 1).await; + let _p_b = links.link("p-b", &p, &b, 1).await; + let mut options = links.options("u-d", &u, &d); + options.cost = Some(1); + options.latency = Duration::from_secs(2); + let _u_d = connect_mock(options).await; + let _b_d = links.link("b-d", &b, &d, 100).await; + let _d_e = links.link("d-e", &d, &e, 1).await; + let broadcast = p.route("live", origin::RouteOptions::hidden()).unwrap(); + let track = broadcast.broadcast().create_track("video", timed()).unwrap(); + broadcast.announce().unwrap(); + kio::clock::sleep(Duration::from_secs(10)).await; + + let start = kio::clock::now(); + let mut at_u = Watch::new(&u, "live", start); + let mut feed = Feed::new(track, start, STEP); + let mut readers = [ + ("D", reader(&d, "live", start).await), + ("E", reader(&e, "live", start).await), + ]; + feed.run(MOVE).await; + p_u.server.abort(Error::IdleTimeout); + p_u.client.abort(Error::IdleTimeout); + let _p_u = links.link("p-u2", &p, &u, 1).await; + feed.run(MOVE * 4).await; + kio::clock::sleep(Duration::from_secs(5)).await; + + let seen = at_u.seen(); + let kinds: Vec<_> = seen.iter().map(|seen| seen.kind).collect(); + assert_eq!(kinds, [Kind::Start, Kind::Offline, Kind::Start], "U flapped: {seen:?}"); + assert!( + seen[2].at - seen[1].at < Duration::from_secs(2), + "faster than the news: {seen:?}" + ); + for (node, rx) in &mut readers { + let live = Liveness::of(rx, &feed); + assert!(live.seamless(), "{node}: {live:?}"); + assert!(live.worst > Duration::from_secs(2), "{node} read through U: {live:?}"); + } + drop(broadcast); +} + +/// `P` publishes `live` to `A` and `B`, and the relay `R` reads it through `A`. A cheaper +/// link from `B` then moves `R`'s route to `B` while the publisher pauses for `pause` +/// between two groups, so the new route has nothing new to deliver for that long. +/// +/// Proves the move's cost is bounded by evidence, not time: `R` keeps `A`'s copy until +/// `B`'s delivers a group the reader lacks, however long that takes, and then `A`'s stops. +/// The two routes overlap by at most the group in progress and the one after, each on its +/// own link, so no frame crosses a link twice, and the reader gets every frame once, in +/// order. +#[kio::test] +async fn a_move_overlaps_the_routes_until_the_new_one_delivers() { + for pause in [Duration::ZERO, Duration::from_secs(30)] { + let links = Links::new(); + let [p, a, b, r] = [1, 2, 3, 4].map(produce_origin); + let broadcast = p.route("live", origin::RouteOptions::hidden()).unwrap(); + let track = broadcast.broadcast().create_track("video", timed()).unwrap(); + broadcast.announce().unwrap(); + let _p_a = links.link("p-a", &p, &a, 1).await; + let _p_b = links.link("p-b", &p, &b, 1).await; + let _a_r = links.link("a-r", &a, &r, 10).await; + kio::clock::sleep(Duration::from_secs(1)).await; + + let start = kio::clock::now(); + let mut feed = Feed::new(track, start, STEP); + let mut rx = reader(&r, "live", start).await; + // Into the third group, so the move lands mid-group. + feed.run(2 * FRAMES + 2).await; + let _b_r = links.link("b-r", &b, &r, 1).await; + feed.run(FRAMES - 2).await; + kio::clock::sleep(pause).await; + feed.run(4 * FRAMES).await; + let overlap = links.frames(); + feed.run(4 * FRAMES).await; + kio::clock::sleep(Duration::from_secs(1)).await; + + let frames = links.frames(); + let live = Liveness::of(&mut rx, &feed); + println!("a move pausing {pause:?}: frames per link {frames:?}, reader {live:?}"); + assert!(live.seamless(), "{pause:?}: {live:?}"); + assert_eq!(frames["a-r"], overlap["a-r"], "{pause:?}: A's copy stopped: {frames:?}"); + let written = feed.written.len() as u64; + for link in ["a-r", "b-r"] { + assert!( + frames[link] <= written, + "{pause:?}: {link} carried a frame twice: {frames:?}" + ); + } + let twice = frames["a-r"] + frames["b-r"] - written; + assert!( + twice <= 2 * FRAMES, + "{pause:?}: {twice} frames on both links: {frames:?}" + ); + drop(broadcast); + } +} + +/// Pins a duplicate subscription, a decision on moq-dev/moq#8881. A worker `W` claims +/// `pool` and answers a request for `pool/p` with the exact route, under the claim's +/// instance; `R` pulls `W`, and `E1`, `E2`, and `E3` pull `R`. `E1` asks for `pool/p` +/// through the claim, a request naming `W`'s route at depth 1; once the answer's route +/// arrives, `E1` moves onto it, now naming depth 2, and `R` serves that from another front, +/// which asks `W` again: one Instance and track, two subscriptions on the one link, +/// differing only in depth. The first closes once the move lands, as any move's overlap +/// does, so one stays open, and `E2` and `E3`, asking after the answer, join it. A fix +/// serves both depths from one subscription, so the move costs none. +#[kio::test] +async fn residual_two_depths_of_one_claim_subscribe_twice() { + use crate::claim_continues::{Worker, settle}; + let links = Links::new(); + let worker = Worker::new(1, "pool", "w"); + let [r, e1, e2, e3] = [2, 3, 4, 5].map(produce_origin); + let _w_r = links.link("w-r", &worker.origin, &r, 1).await; + let _r_e1 = links.link("r-e1", &r, &e1, 1).await; + let _r_e2 = links.link("r-e2", &r, &e2, 1).await; + let _r_e3 = links.link("r-e3", &r, &e3, 1).await; + settle().await; + let mut subscribed = Vec::new(); + for node in [&e1, &e2, &e3] { + let broadcast = node.consume().request("pool/p", None).await.unwrap(); + subscribed.push(broadcast.track("video").unwrap().subscribe(None).await.unwrap()); + settle().await; + assert_eq!(links.subscriptions()["w-r"], 2, "{:?}", links.subscriptions()); + assert_eq!(links.open()["w-r"], 1, "the first closed: {:?}", links.open()); + } + assert_eq!(worker.served(), 1); +} + +/// `P` publishes `live` through `X` to `A` and through `Y` to `B`, and through `C` to +/// both at a higher cost. `A` and `B` link to each other cheaply, at 50 ms each way, so +/// each prefers the other's route over `C`'s. Two unrelated links close at once, `X`'s to +/// `A` and `Y`'s to `B`: each relay doubts only its own lost hop, so `A` moves onto `B`'s +/// direct route and `B` onto `A`'s, both stale, and the routes loop until the news +/// crosses. Readers on `A` and `B` follow `video` from before the cut, and readers asking +/// for `audio`, which nobody read yet, join while the loop stands. +/// +/// Proves the loop formed (each relay asked the other), that it ends with the routes: once +/// they settle on `C`, every reader gets every frame once, in order, and no request is left +/// between `A` and `B`. +#[kio::test] +async fn a_request_into_a_routing_loop_follows_the_routes_out() { + let links = Links::new(); + let [p, x, y, c, a, b] = [1, 2, 3, 4, 5, 6].map(produce_origin); + let broadcast = p.route("live", origin::RouteOptions::hidden()).unwrap(); + let video = broadcast.broadcast().create_track("video", timed()).unwrap(); + let audio = broadcast.broadcast().create_track("audio", timed()).unwrap(); + broadcast.announce().unwrap(); + let _p_x = links.link("p-x", &p, &x, 1).await; + let _p_y = links.link("p-y", &p, &y, 1).await; + let _p_c = links.link("p-c", &p, &c, 1).await; + let x_a = links.link("x-a", &x, &a, 1).await; + let y_b = links.link("y-b", &y, &b, 1).await; + let mut between = Vec::new(); + for (name, from, to) in [("a-b", &a, &b), ("b-a", &b, &a)] { + let mut options = links.options(name, from, to); + options.cost = Some(1); + options.latency = Duration::from_millis(50); + between.push(connect_mock(options).await); + } + let _c_a = links.link("c-a", &c, &a, 10).await; + let _c_b = links.link("c-b", &c, &b, 10).await; + kio::clock::sleep(Duration::from_secs(1)).await; + + let start = kio::clock::now(); + let mut feeds = [Feed::new(video, start, STEP), Feed::new(audio, start, STEP)]; + let run = async |feeds: &mut [Feed; 2], count| { + let [video, audio] = feeds; + futures::join!(video.run(count), audio.run(count)); + }; + let mut readers = Vec::new(); + for (name, node) in [("A", &a), ("B", &b)] { + readers.push((name, 0, reader_of(node, "live", "video", start).await)); + } + run(&mut feeds, MOVE).await; + cut(&x_a); + cut(&y_b); + kio::clock::sleep(Duration::from_millis(5)).await; + // Asked while the loop stands, so their subscriptions answer only once it ends. + let asked = async { + let at_a = reader_of(&a, "live", "audio", start).await; + let at_b = reader_of(&b, "live", "audio", start).await; + [("A", 1, at_a), ("B", 1, at_b)] + }; + let (asked, ()) = futures::join!(asked, run(&mut feeds, MOVE)); + readers.extend(asked); + kio::clock::sleep(Duration::from_secs(1)).await; + + let subscriptions = links.subscriptions(); + let open = links.open(); + println!("a request into a loop: subscriptions {subscriptions:?}, open {open:?}"); + for link in ["a-b", "b-a"] { + assert!( + subscriptions.get(link).is_some_and(|count| *count > 0), + "{link} looped: {subscriptions:?}" + ); + assert_eq!(open[link], 0, "{link} left a request: {open:?}"); + } + for (name, index, rx) in &mut readers { + let live = Liveness::of(rx, &feeds[*index]); + assert!(live.seamless(), "{name}, track {index}: {live:?}"); + } + drop(between); + drop(broadcast); +} + +/// A moq-transport relay that announces back a name it learned from us, which is +/// unsupported (the multicast epic's item 10): its wire carries no path or Instance, so the +/// name looks like a nearer publisher. `R` pulls `live` from its publisher `P` over a link +/// it ranks below the moq-transport ones, and publishes to `M`, which `N` pulls. `O`, a +/// relay of another make, claims `live` toward `R` and answers by reading it from `N`, +/// naming nobody it came through. `R`'s reader of `live` follows `O`, whose request comes +/// back to `R` through `M`, and `R` sends it to `O` again, where it joins the request +/// already waiting. +/// +/// Proves the loop costs bounded work and touches nothing else: the reader of `live` +/// stays pending, no message or subscription is sent once it formed, `O` is asked once, +/// and the other broadcasts on both sides of it read on. +#[kio::test] +async fn a_name_announced_back_over_moq_transport_waits_without_retrying() { + let links = Links::new(); + let transport: moq_net::Version = "moq-transport-18".parse().unwrap(); + let [p, r, m, n, o] = [1, 2, 3, 4, 5].map(produce_origin); + let live = p.route("live", origin::RouteOptions::hidden()).unwrap(); + live.broadcast().create_track("video", timed()).unwrap(); + live.announce().unwrap(); + let mut options = links.options("p-r", &p, &r); + options.client_subscribe = Some(r.clone().with_preference(origin::Preference::new(-1))); + let _p_r = connect_mock(options).await; + kio::clock::sleep(Duration::from_secs(1)).await; + let mut pairs = Vec::new(); + for (name, from, to) in [("r-m", &r, &m), ("m-n", &m, &n), ("o-r", &o, &r)] { + let mut options = links.options(name, from, to); + options.version = transport; + pairs.push(connect_mock(options).await); + } + let claim = o.claim("live", origin::RouteOptions::default()).unwrap(); + let from = n.consume(); + let asked = std::rc::Rc::new(std::cell::Cell::new(0)); + let counted = asked.clone(); + drop(kio::clock::sim::spawn(async move { + while let Some(request) = claim.requested().await { + counted.set(counted.get() + 1); + match from.request("live", None).await { + Ok(broadcast) => request.serve(broadcast), + Err(err) => request.reject(err), + } + } + })); + let at_r = r.route("at-r", origin::RouteOptions::hidden()).unwrap(); + let at_o = o.route("at-o", origin::RouteOptions::hidden()).unwrap(); + let mut feeds = Vec::new(); + let start = kio::clock::now(); + for route in [&at_r, &at_o] { + let track = route.broadcast().create_track("video", timed()).unwrap(); + route.announce().unwrap(); + feeds.push(Feed::new(track, start, STEP)); + } + kio::clock::sleep(Duration::from_secs(1)).await; + + let mut others = [reader(&m, "at-r", start).await, reader(&r, "at-o", start).await]; + let looped = r.consume().request("live", None).await.unwrap(); + let mut looped = looped.track("video").unwrap().subscribe(None); + let run = async |feeds: &mut Vec, count| { + futures::future::join_all(feeds.iter_mut().map(|feed| feed.run(count))).await; + }; + run(&mut feeds, MOVE).await; + let (control, subscriptions) = (links.control(), links.subscriptions()); + run(&mut feeds, 10 * MOVE).await; + + assert!( + kio::clock::timeout(Duration::from_millis(1), &mut looped) + .await + .is_err(), + "the looped request stays pending" + ); + assert_eq!(links.control(), control, "nothing sent once the loop formed"); + assert_eq!(links.subscriptions(), subscriptions); + assert_eq!(asked.get(), 1, "O asked once"); + for (rx, feed) in others.iter_mut().zip(&feeds) { + let live = Liveness::of(rx, feed); + assert!(live.seamless(), "{live:?}"); + } + drop((pairs, live, at_r, at_o)); +} + +/// `P` publishes `live` and `R` pulls it. `P` ends its track `Dropped` while `live` stays +/// announced, as a binding releasing a handle before its finish does. That says nothing of a +/// route, so readers at `P` and at `R` both end at once rather than wait for the table to +/// move, which nothing does while `live` stands: only `Unroutable` reports a route losing +/// the content. +#[kio::test] +async fn a_track_released_while_announced_ends_every_reader() { + let links = Links::new(); + let [p, r] = [1, 2].map(produce_origin); + let broadcast = p.route("live", origin::RouteOptions::hidden()).unwrap(); + let track = broadcast.broadcast().create_track("video", timed()).unwrap(); + broadcast.announce().unwrap(); + let _p_r = links.link("p-r", &p, &r, 1).await; + kio::clock::sleep(Duration::from_secs(1)).await; + + let start = kio::clock::now(); + let mut feed = Feed::new(track.clone(), start, STEP); + let mut at_p = reader(&p, "live", start).await; + let mut at_r = reader(&r, "live", start).await; + feed.run(2 * FRAMES).await; + track.abort(Error::Dropped).unwrap(); + kio::clock::sleep(Duration::from_secs(1)).await; + let ended = + |rx: &mut mpsc::UnboundedReceiver| std::iter::from_fn(|| rx.try_recv().ok()).find_map(|got| got.err()); + assert!(ended(&mut at_p).is_some(), "the publisher's own reader ends"); + assert!(ended(&mut at_r).is_some(), "the next relay's reader ends too"); + drop(broadcast); +}