From e182bf6000b169804ca868edfbb5bb8a3bb19a73 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 1 Sep 2026 09:13:44 -0700 Subject: [PATCH 1/4] chore(quest): claim route-resume CAEC5E00-71FA-4DE3-8A0A-6062410325A5 From 06aacfd8a13c3450b7745ebdba74a456cb57dcdc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 1 Sep 2026 18:48:05 -0700 Subject: [PATCH 2/4] feat(net)!: resume subscriptions across routes sharing a first hop A subscription whose serving route dies now resumes through the next best covering route when both routes name the same non-zero first hop, re-splicing at a group boundary. Different or unknown (0) first hops still end the subscription, so one publisher's subscribers are never spliced onto another's frames and two anonymous peers never pass for one reconnecting. This restores the seamless failover that #3225 removed, keyed on the routing identity that stayed on the wire. request_broadcast now resolves a routed path to a per-(path, exclusion) front: a spliced broadcast whose watcher materializes it from the best covering route through the existing per-route request queues, re-selects when the table changes (a drain reprice migrates before the session dies; a metadata-only reprice changes nothing), and retries a retracted route's failure through the survivor. Fronts are keyed per split-horizon exclusion so a failover can never adopt a route flowing back through one of its own readers. The spliced-track machinery (resume.rs, serve_track) is reused unchanged. The draft's no-splice paragraph becomes the first-hop rule, recorded in the moq-lite-06 changelog. The goaway cluster and route migration tests return to their pre-#3225 seamless expectations. Co-Authored-By: Claude Fable 5 --- drafts/draft-lcurley-moq-lite.md | 7 +- rs/moq-net/src/model/origin.rs | 842 +++++++++++++++++++++++---- rs/moq-net/src/model/weak_cache.rs | 7 + rs/moq-relay/tests/goaway_cluster.rs | 90 +-- rs/moq-tokio/tests/broadcast.rs | 32 +- 5 files changed, 782 insertions(+), 196 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index ee0470b623f..1ebad80ae44 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -371,8 +371,10 @@ Applying one rule to both advertisement and dispatch keeps advertised paths trut When resolving a path covered by several routes (across any number of streams), the subscriber SHOULD prefer the most specific covering prefix, then the lowest Warm Route Cost after adding each arriving link's cost (see [Cost Parameter](#cost-parameter)), breaking ties toward the lowest Cold Route Cost, then toward the shortest path, and then toward the most recently received, so a reconnecting publisher is not outranked by the stale session it replaced. -A route carries no content identity: nothing on the wire promises that two routes covering one path serve interchangeable bytes. -A relay MUST NOT splice a live subscription across sources reached through different routes; when a serving session ends, in-flight subscriptions end with it (a reset), and the subscriber re-requests through the best remaining route. +A route's identity is its first hop: the endpoint that originated it (see [ANNOUNCE_START](#announce-start)). +Two routes covering one path with the same non-zero first hop are the same origin reached different ways, and a relay MAY move a live subscription between them, resuming at a group boundary, so a route change the identity survives (a reconnect, a cheaper path, a draining session) is invisible to the subscriber. +Across differing first hops, or where either is 0, the routes promise nothing about each other's content: a relay MUST NOT splice a live subscription across them, and when the serving session ends, in-flight subscriptions end with it (a reset) and the subscriber re-requests through the best remaining route. +Equal first hops promise the same origin, not interchangeable bytes; what a resuming relay serves next is whatever that origin publishes next at the group boundary. ### Subscribe A subscriber opens Subscribe Streams to request a Track. @@ -1283,6 +1285,7 @@ The `Message Length` describes the payload size on the wire. - Removed `Subscriber Ordered` from SUBSCRIBE and SUBSCRIBE_UPDATE, and `Publisher Ordered` from TRACK_INFO and SUBSCRIBE_OK. Group order within a Track is now normatively newest-first, with no field to invert it: a subscriber that wants sequence order reads in sequence order, which costs the network nothing and does not let one subscriber's preference reach a Track a relay is fanning out to many. Note this removes a byte from the middle of each message, so a lite-06 peer cannot parse a lite-05 one's SUBSCRIBE (the earlier drafts keep the byte, and an implementation serving them SHOULD write 0 and ignore what it reads). - Made `Group Start` an absolute floor (the raw minimum group sequence, default 0) rather than the sequence + 1 with 0 meaning the latest group. The start resolves from `Subscriber Max Age` instead: the oldest group at or above the floor within the budget, so a subscriber that buffers is handed the head of what it can still play. A zero budget still resolves to the latest group, which was the only start the old encoding could ask for by default. - Removed the actively-carrying Warm discount, its ceiling exemption, and the `(Cold cost, hash)` adoption rank with its re-parenting delay: costs are forwarded accumulated only. The `Warm` and `Cold` fields and the selection order are unchanged. +- Replaced the no-splice rule with a first-hop identity: two routes covering one path with the same non-zero first hop are the same origin reached another way, and a relay MAY move a live subscription between them at a group boundary. Splicing across differing or unknown (0) first hops remains prohibited. ## moq-lite-05 - Renamed ANNOUNCE_INTEREST to ANNOUNCE_REQUEST and ANNOUNCE to ANNOUNCE_BROADCAST. diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 8bb590a846c..14e48a03479 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -14,7 +14,7 @@ use rand::RngExt; use web_async::Lock; use web_transport_trait::{MaybeSend, MaybeSync}; -use super::{Requests, WeakCache}; +use super::{Requests, WeakCache, WeakEntry}; use crate::{ AsPath, Error, Path, PathOwned, PathPrefixes, coding::{BoundsExceeded, Decode, DecodeError, Encode, EncodeError}, @@ -767,6 +767,35 @@ struct ServeState { closed: bool, } +/// Key of a remotely-served front: the absolute path and the requester's +/// split-horizon exclusion. Requesters excluding different peers get separate +/// fronts, so a front's failover never adopts a route flowing back through one +/// of its own readers. +type FrontKey = (PathOwned, Option); + +/// One remotely-served front in [`OriginState::fronts`]: the shared spliced +/// broadcast at a path plus the channel requesters resolve through. +#[derive(Clone)] +struct RemoteFront { + /// Resolves requesters with the front's consumer (or the error that ended it + /// unresolved). The producer lives here so the teardown can reject requesters + /// still parked on a front whose watcher was cancelled. + request: kio::Producer, + /// The front's spliced broadcast, weak: dead once its watcher exits, so a + /// later request re-creates the front instead of joining a corpse. + broadcast: broadcast::WeakConsumer, +} + +impl WeakEntry for RemoteFront { + fn is_closed(&self) -> bool { + self.broadcast.is_closed() + } + + fn same_channel(&self, other: &Self) -> bool { + self.broadcast.same_channel(&other.broadcast) + } +} + /// One registered announce cursor: which prefixes it may see, how they are /// re-rooted, and the per-cursor delivery buffer. struct TableCursor { @@ -1383,13 +1412,7 @@ impl Producer { pub fn consume(&self) -> Consumer { // Untagged: a session tags the egress consumer separately via // `origin::Consumer::with_stats` (ingress and egress are distinct sides). - Consumer::new( - self.info, - self.root.clone(), - self.nodes.clone(), - self.shared.clone(), - stats::Session::default(), - ) + Consumer::from_producer(self, stats::Session::default()) } /// Returns a new Producer that automatically strips out the provided prefix. @@ -1663,12 +1686,13 @@ impl DriverState { // `create_broadcast` holds across its attach: a concurrent create either // finishes before this (the walk below cleans its entry up) or observes // `closed` and fails with `Closed`. - let (pending, servers, cursors) = { + let (pending, servers, cursors, fronts) = { let mut shared = self.shared.lock(); shared.closed = true; let servers: Vec<_> = shared.routes.iter().filter_map(|entry| entry.server.clone()).collect(); let cursors: Vec<_> = shared.cursors.values().map(|cursor| cursor.state.clone()).collect(); - (shared.fallback.drain_all(), servers, cursors) + let fronts: Vec<_> = shared.fronts.values().map(|front| front.request.clone()).collect(); + (shared.fallback.drain_all(), servers, cursors, fronts) }; // Reject every pending request, including those already handed to a @@ -1679,6 +1703,14 @@ impl DriverState { request.resolved.get_or_insert(Err(Error::Dropped)); } } + + // Reject requesters still parked on a remote front's channel: its watcher + // was cancelled above and will never resolve them. + for producer in fronts { + if let Ok(mut request) = producer.write() { + request.resolved.get_or_insert(Err(Error::Dropped)); + } + } for server in servers { let mut server = server.lock(); server.closed = true; @@ -2345,6 +2377,372 @@ async fn serve_track( } } +/// The content identity of a remotely-served front: who originated the route +/// its first source arrived through. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Identity { + /// No source has attached yet: the first request may resolve through any + /// covering route, and whoever serves it fixes the identity. + Undetermined, + /// The serving route's first hop was absent or [`Hop::UNKNOWN`], which + /// identifies nobody and never matches itself: the front cannot resume, so + /// its source ending ends it. Two anonymous publishers must never pass for + /// one reconnecting. + Anonymous, + /// The first hop of the serving route: the endpoint that originated it. + /// Routes sharing it are the same origin reached another way and safe to + /// resume through; anything else is different content at the same path. + Publisher(Hop), +} + +impl Identity { + /// The first-hop pin for [`OriginState::best_route`], or `None` when any + /// route qualifies (no source attached yet). + fn pin(&self) -> Option { + match self { + Self::Publisher(hop) => Some(*hop), + _ => None, + } + } + + /// Whether route selection applies at all: an anonymous front never adopts + /// another route. + fn routable(&self) -> bool { + !matches!(self, Self::Anonymous) + } +} + +/// Everything [`run_remote_front`] owns, queued by [`Consumer::request_broadcast`]. +struct RemoteFrontTask { + /// The route table the front selects and re-selects from. + shared: kio::Shared, + /// The front's source table, shared with its [`serve_track`] tasks. + state: kio::Producer, + /// The spliced broadcast the front serves. + broadcast: broadcast::Producer, + /// Absolute path of the front. + path: PathOwned, + /// The requesters' split-horizon exclusion, applied to every (re)selection. + exclude: Option, + /// Resolves the requesters parked on the front's channel. + request: kio::Producer, + tasks: Tasks, + timers: TimersSlot, +} + +/// Owns a remotely-served front: materializes the path from the best covering +/// route, dispatches its spliced tracks, and re-splices through routes sharing +/// the front's first-hop identity when the serving source dies or a better +/// qualifying route appears, so subscribers never observe a survivable route +/// change. The front ends (aborting its subscribers) when the source dies with +/// no qualifying route left, when a qualifying route refuses the path, or when +/// the origin tears down. +async fn run_remote_front(task: RemoteFrontTask) { + let RemoteFrontTask { + shared, + state, + broadcast, + path, + exclude, + request, + tasks, + timers, + } = task; + + enum Step { + /// A spliced track needs a serve task. + Serve(Arc, super::resume::Producer), + /// The in-flight upstream request resolved. + Resolved(Result), + /// The attached source closed. + SourceDead, + /// The route table changed in a way the decision below cares about. + Table, + /// The origin tore down. + Closed, + } + + let mut front = FrontDriver { + state, + broadcast, + initial: Some(request), + identity: Identity::Undetermined, + serving: None, + }; + // The in-flight upstream request: the targeted route entry, its first hop, + // and the pending channel. + let mut upstream: Option<(u64, Option, kio::Consumer)> = None; + // Routes that refused the path (an authoritative reject while another source + // was serving, or a queue with no live handler). Never retried while the + // entry stands; a reconnect is a fresh entry. + let mut refused: HashSet = HashSet::new(); + // Why the last candidate fell through, reported if the front dies unresolved. + let mut last_err: Option = None; + + 'run: loop { + // Decide what the table means for us, then fire at most one upstream + // request. Locks are sequential, never nested: the table's, then the + // chosen route's queue. + // What the wait below compares table changes against: the qualifying + // route this pass decided on, recomputed identically by the wait's + // predicate so only a table change that would alter the decision wakes it. + let decided; + let fire = { + let table = shared.read(); + if table.closed { + break 'run; + } + refused.retain(|id| table.routes.iter().any(|entry| entry.id == *id)); + let best = match front.identity.routable() { + true => table.best_route(&path.as_path(), exclude, front.identity.pin(), &refused), + false => None, + }; + decided = best.map(|entry| entry.id); + let serving_route = front.serving.as_ref().map(|(_, route, _)| *route); + match best { + // The serving route is still the best qualifying one. + Some(entry) if Some(entry.id) == serving_route => { + upstream = None; + None + } + // A better (or replacement) qualifying route: request through it, + // unless we already are. + Some(entry) => match upstream.as_ref().is_some_and(|(route, ..)| *route == entry.id) { + true => None, + false => Some(( + entry.id, + entry.hops.iter().next().copied(), + entry.server.clone().expect("best_route yields served entries"), + )), + }, + // Nothing qualifies. A live source keeps serving (its route may + // return); without one the front is over. + None => { + upstream = None; + match &front.serving { + Some(_) => None, + None => { + let err = match front.identity { + Identity::Undetermined => last_err.take().unwrap_or(Error::Unroutable), + _ => last_err.take().unwrap_or(Error::Dropped), + }; + front.end(err); + return; + } + } + } + } + }; + + if let Some((route, first, server)) = fire { + let mut serve = server.lock(); + match serve.closed { + // Retracted between the table read and its lock: the table has + // already moved on, so just re-decide. + true => continue 'run, + false => { + // A source this route already materialized for the path + // attaches without another upstream round trip. + if let Some(weak) = serve.served.get(&path) { + drop(serve); + front.attach(weak.consume(), route, first); + continue 'run; + } + let pending = match serve.requests.join(&path) { + Some(producer) => producer.consume(), + None => { + let producer = kio::Producer::::default(); + let consumer = producer.consume(); + match serve.requests.insert(path.clone(), producer) { + Ok(()) => consumer, + // No live handler behind the route: it cannot + // serve, whatever the table says. + Err(_) => { + refused.insert(route); + last_err = Some(Error::Unroutable); + continue 'run; + } + } + } + }; + upstream = Some((route, first, pending)); + } + } + } + + let pin = front.identity.pin(); + let routable = front.identity.routable(); + + let step = kio::wait(|waiter| { + if let Poll::Ready((name, resume)) = front.broadcast.poll_spliced_assigned(waiter) { + return Poll::Ready(Step::Serve(name, resume)); + } + + if let Some((.., pending)) = &upstream + && let Poll::Ready(result) = pending.poll(waiter, |p| match &p.resolved { + Some(result) => Poll::Ready(result.clone()), + None => Poll::Pending, + }) { + return Poll::Ready(Step::Resolved(match result { + Ok(resolved) => resolved, + // The queue died unresolved (its handler dropped): the route + // could not serve. + Err(_closed) => Err(Error::Unroutable), + })); + } + + if let Some((.., source)) = &front.serving + && source.poll_closed(waiter).is_ready() + { + return Poll::Ready(Step::SourceDead); + } + + match shared.poll(waiter, |table| { + if table.closed { + return Poll::Ready(()); + } + let best = match routable { + true => table.best_route(&path.as_path(), exclude, pin, &refused).map(|e| e.id), + false => None, + }; + match best == decided { + true => Poll::Pending, + false => Poll::Ready(()), + } + }) { + Poll::Ready(table) => { + let closed = table.closed; + drop(table); + Poll::Ready(match closed { + true => Step::Closed, + false => Step::Table, + }) + } + Poll::Pending => Poll::Pending, + } + }) + .await; + + match step { + Step::Serve(name, resume) => { + tasks.push(serve_track(front.state.clone(), name, resume, timers.clone())); + } + Step::Resolved(result) => { + let (route, first, _) = upstream.take().expect("resolved an in-flight request"); + match result { + Ok(source) => front.attach(source, route, first), + // The route retracted before serving: the table already + // reflects it, so the next pass retries the survivor. Each + // such retry consumed a real retraction, so this cannot spin. + Err(Error::Unroutable) => { + last_err = Some(Error::Unroutable); + } + // An authoritative refusal of the path. It ends a front with + // no other source (a refusal is never retried); a serving + // front merely skips the refuser. + Err(err) => match &front.serving { + Some(_) => { + refused.insert(route); + last_err = Some(err); + } + None => { + front.end(err); + return; + } + }, + } + } + Step::SourceDead => { + front.detach_dead(); + last_err = Some(Error::Dropped); + } + Step::Table => {} + Step::Closed => break 'run, + } + } + + // The origin tore down; its teardown already rejected parked requesters. + front.end(Error::Dropped); +} + +/// The mutable half of a remote front's watcher: the source table and spliced +/// broadcast it manages, who is serving, and the requesters awaiting the first +/// source. +struct FrontDriver { + /// The front's source table, shared with its [`serve_track`] tasks. + state: kio::Producer, + /// The spliced broadcast the front serves. + broadcast: broadcast::Producer, + /// Resolves the requesters parked on the front once the first source + /// attaches (or the front dies first). + initial: Option>, + /// The front's content identity, fixed by the first source to attach. + identity: Identity, + /// The attached source: its id in the front's table, the route entry that + /// served it, and the source itself. + serving: Option<(u64, u64, broadcast::Consumer)>, +} + +impl FrontDriver { + /// Attach a materialized source: replace the previous one (its tracks + /// re-splice at a group boundary), fix the front's identity on the first + /// attach, and resolve the requesters parked on the front's channel. + fn attach(&mut self, source: broadcast::Consumer, route: u64, first: Option) { + let Ok(mut front) = self.state.write() else { return }; + if let Some((id, ..)) = self.serving.take() { + front.sources.retain(|entry| entry.id != id); + } + let id = front.next_source; + front.next_source += 1; + front.sources.push(FrontSource { + id, + source: source.clone(), + }); + front.reselect(); + drop(front); + + self.serving = Some((id, route, source)); + if self.identity == Identity::Undetermined { + self.identity = match first { + Some(hop) if hop != Hop::UNKNOWN => Identity::Publisher(hop), + _ => Identity::Anonymous, + }; + } + if let Some(request) = self.initial.take() + && let Ok(mut pending) = request.write() + { + pending.resolved.get_or_insert(Ok(self.broadcast.consume())); + } + } + + /// Remove the dead source from the table. Its spliced tracks park on the + /// empty table until a replacement attaches or the front ends. + fn detach_dead(&mut self) { + if let Some((id, ..)) = self.serving.take() + && let Ok(mut front) = self.state.write() + { + front.sources.retain(|entry| entry.id != id); + front.reselect(); + } + } + + /// End the front: reject requesters still parked on its channel, close the + /// source table so its serve tasks exit, and abort the spliced broadcast so + /// its subscribers observe the end. + fn end(&mut self, err: Error) { + if let Some(request) = self.initial.take() + && let Ok(mut pending) = request.write() + { + pending.resolved.get_or_insert(Err(err.clone())); + } + if let Ok(mut front) = self.state.write() { + front.closed = true; + } + self.broadcast.abort_spliced(err); + self.broadcast.finish(); + } +} + /// The origin's shared state: the route table, the announce cursors observing /// it, and the fallback request queue. /// @@ -2371,6 +2769,15 @@ struct OriginState { // consumers drop. served: WeakCache, + // The remotely-served fronts, keyed by absolute path and the requester's + // split-horizon exclusion. Each is a spliced broadcast whose watcher task + // materializes it from the best covering route and re-splices it through + // routes sharing its first hop, so a route change the identity survives is + // invisible to subscribers. Keyed per exclusion so a front's failover can + // never adopt a route flowing back through one of its own readers. Weak, so + // a front dies with its watcher and a later request re-creates it. + fronts: WeakCache, + // Set when the origin's driver dropped: new requests fail with `Closed` // immediately and handlers observe the end instead of parking forever. closed: bool, @@ -2464,14 +2871,26 @@ impl OriginState { self.cursors.insert(id, cursor); } - /// The queue behind the best served route covering `path` (absolute) for a - /// requester excluding `exclude`. + /// The best served route covering `path` (absolute) for a requester excluding + /// `exclude`, skipping the `refused` entry ids. /// /// The most specific covering prefix wins outright, so a narrow advertise-only /// announcement shadows a broad served one: requests under it fall through to /// the fallback handler instead of being routed around it. Among routes at the /// winning prefix, the cheapest served one is picked by [`route_order`]. - fn best_server(&self, path: &Path, exclude: Option) -> Option> { + /// + /// With `publisher` set, only routes originated by that first hop are + /// candidates: this is the identity a front resumes through, and a route from + /// anyone else is different content rather than an alternate path (see + /// [`run_remote_front`]). `Hop::UNKNOWN` identifies nobody, so callers never + /// pin it. + fn best_route( + &self, + path: &Path, + exclude: Option, + publisher: Option, + refused: &HashSet, + ) -> Option<&RouteEntry> { let candidates: Vec<&RouteEntry> = self .routes .iter() @@ -2480,6 +2899,11 @@ impl OriginState { Some(peer) if peer != Hop::UNKNOWN => !entry.hops.contains(&peer), _ => true, }) + .filter(|entry| match publisher { + Some(first) => entry.hops.iter().next() == Some(&first), + None => true, + }) + .filter(|entry| !refused.contains(&entry.id)) .collect(); // Covering prefixes of one path form a chain, so the longest is unique. @@ -2488,7 +2912,6 @@ impl OriginState { .into_iter() .filter(|entry| entry.prefix.len() == most && entry.server.is_some()) .min_by_key(|entry| route_order(path, entry)) - .and_then(|entry| entry.server.clone()) } } @@ -2873,13 +3296,7 @@ impl Consume for Producer { // Mirrors the inherent `Producer::consume`; inlined to avoid the // inherent-vs-trait `consume` ambiguity. Untagged: egress is tagged // separately from ingress. - Consumer::new( - self.info, - self.root.clone(), - self.nodes.clone(), - self.shared.clone(), - stats::Session::default(), - ) + Consumer::from_producer(self, stats::Session::default()) } } @@ -2941,6 +3358,18 @@ pub struct Consumer { // `announced` and skipped by `request_broadcast`, so a peer is never served // (or advertised) its own content back. `None` (the default) filters nothing. exclude: Option, + + // The origin config remote fronts inherit (identity, cache pool, retention), + // mirroring what `create_broadcast` gives a local front. + origin: Info, + + // Submission handle to the origin's [`Driver`], for the front watcher a + // routed `request_broadcast` spawns. Closed once the driver drops. + tasks: Tasks, + + // The driver's clock and timers, threaded into fronts for the track idle + // linger. + timers: TimersSlot, } impl std::ops::Deref for Consumer { @@ -2952,20 +3381,17 @@ impl std::ops::Deref for Consumer { } impl Consumer { - fn new( - info: Hop, - root: PathOwned, - nodes: OriginNodes, - shared: kio::Shared, - stats: stats::Session, - ) -> Self { + fn from_producer(producer: &Producer, stats: stats::Session) -> Self { Self { - info, - nodes, - root, - shared, + info: producer.info, + nodes: producer.nodes.clone(), + root: producer.root.clone(), + shared: producer.shared.clone(), stats, exclude: None, + origin: producer.info(), + tasks: producer.tasks.clone(), + timers: producer.timers.clone(), } } @@ -3002,12 +3428,8 @@ impl Consumer { /// rather than tearing the stream down. pub(crate) fn empty(&self) -> Self { Self { - info: self.info, nodes: OriginNodes { nodes: Vec::new() }, - root: self.root.clone(), - shared: self.shared.clone(), - stats: self.stats.clone(), - exclude: self.exclude, + ..self.clone() } } @@ -3149,12 +3571,8 @@ impl Consumer { pub fn scope(&self, prefixes: &[Path]) -> Option { let prefixes = PathPrefixes::new(prefixes); Some(Consumer { - info: self.info, - root: self.root.clone(), nodes: self.nodes.select(&prefixes)?, - shared: self.shared.clone(), - stats: self.stats.clone(), - exclude: self.exclude, + ..self.clone() }) } @@ -3169,8 +3587,12 @@ impl Consumer { /// 2. Otherwise the best announced route covering the path (most specific /// prefix first, then cheapest) serves it on demand: sessions materialize /// the path from the peer that announced the route. Concurrent requests - /// for the same path coalesce, and the served broadcast is shared for as - /// long as it stays live; a closed one is re-served on the next request. + /// for the same path coalesce onto one shared front, which outlives any + /// single route: when its serving route dies or a better one appears, the + /// front re-splices through the best route sharing its first hop at a + /// group boundary, invisibly to subscribers. A route change that does not + /// preserve the first hop ends the broadcast instead, and the next + /// request re-serves the path. /// 3. Otherwise a live [`Dynamic`] handler (see [`Producer::dynamic`]) /// receives the request as a fallback. /// @@ -3208,39 +3630,58 @@ impl Consumer { return kio::Pending::new(Requesting::failed(Error::Closed)); } - if in_scope && let Some(server) = state.best_server(&absolute.as_path(), self.exclude) { - // Two sequential locks, never nested: the entry's queue has its own. - drop(state); - let mut serve = server.lock(); - if !serve.closed { - // Reuse a still-live broadcast already served for this path, so repeat - // requests share one upstream subscription. - if let Some(weak) = serve.served.get(&absolute) { - let resolved = Requesting::ready(weak.consume()).with_path(requested).with_stats(scope); - return kio::Pending::new(resolved); - } - - // Coalesce onto a pending request for the same path; otherwise - // register a new one. - let consumer = if let Some(producer) = serve.requests.join(&absolute) { - producer.consume() - } else { - let producer = kio::Producer::::default(); - let consumer = producer.consume(); - if serve.requests.insert(absolute, producer).is_err() { - return kio::Pending::new(Requesting::failed(Error::Unroutable)); - } - consumer - }; - return kio::Pending::new(Requesting::pending(consumer).with_path(requested).with_stats(scope)); + if in_scope { + // Join the live front for this path and exclusion, if any: its watcher + // resolves (or already resolved) the request channel with the front's + // spliced broadcast, so repeat requests share one upstream + // subscription. A front whose route has since retracted still serves + // for as long as its session does. + let key = (absolute.clone(), self.exclude); + if let Some(front) = state.fronts.get(&key) { + let pending = Requesting::pending(front.request.consume()) + .with_path(requested) + .with_stats(scope); + return kio::Pending::new(pending); } - // The route was retracted between the lookup and its lock; fall - // through to the fallback handler. - drop(serve); - state = self.shared.lock(); - if state.closed { - return kio::Pending::new(Requesting::failed(Error::Closed)); + if state + .best_route(&absolute.as_path(), self.exclude, None, &HashSet::new()) + .is_some() + { + // A route covers the path: mint the front and hand its watcher the + // request. The watcher materializes the path from the best covering + // route, resolves the channel, and re-splices the front through + // routes sharing its first hop for as long as one serves. + let broadcast = broadcast::Producer::new_spliced(broadcast::Info { + origin: self.origin.clone(), + path: absolute.clone(), + }); + let front_state = kio::Producer::new(FrontState { + next_source: 0, + sources: Vec::new(), + active: None, + closed: false, + }); + let request = kio::Producer::::default(); + let consumer = request.consume(); + state.fronts.insert( + key, + RemoteFront { + request: request.clone(), + broadcast: broadcast.consume().weak(), + }, + ); + self.tasks.push(run_remote_front(RemoteFrontTask { + shared: self.shared.clone(), + state: front_state, + broadcast, + path: absolute, + exclude: self.exclude, + request, + tasks: self.tasks.clone(), + timers: self.timers.clone(), + })); + return kio::Pending::new(Requesting::pending(consumer).with_path(requested).with_stats(scope)); } } @@ -3275,12 +3716,9 @@ impl Consumer { let prefix = prefix.as_path(); Some(Self { - info: self.info, root: self.root.join(&prefix).to_owned(), nodes: self.nodes.root(&prefix)?, - shared: self.shared.clone(), - stats: self.stats.clone(), - exclude: self.exclude, + ..self.clone() }) } @@ -3549,6 +3987,20 @@ mod tests { panic!("condition never settled"); } + /// Yield to the driver until the server's front watcher delivers a request. + async fn queued(server: &mut RouteServer) -> Request { + let mut request = None; + settle(|| match server.poll_requested_broadcast(&kio::Waiter::noop()) { + Poll::Ready(Ok(popped)) => { + request = Some(popped); + true + } + _ => false, + }) + .await; + request.unwrap() + } + #[tokio::test] async fn announce_and_retract() { let producer = origin(1).produce(); @@ -3787,16 +4239,13 @@ mod tests { let (_announcement, mut server) = producer.announce_served("room", Route::default()).unwrap(); let pending = consumer.request_broadcast("room/alice"); - let request = match server.poll_requested_broadcast(&kio::Waiter::noop()) { - Poll::Ready(Ok(request)) => request, - _ => panic!("expected a queued request"), - }; + let request = queued(&mut server).await; assert_eq!(request.path().as_str(), "room/alice"); let source = broadcast::Info::new().produce(); request.accept(&source); - let resolved = pending.now_or_never().expect("accepted").expect("resolves"); + let resolved = pending.await.expect("resolves"); // The handle is named by what the requester asked for. assert_eq!(resolved.info().path.as_str(), "room/alice"); @@ -3818,18 +4267,15 @@ mod tests { let first = consumer.request_broadcast("room/alice"); let second = consumer.request_broadcast("room/alice"); - let request = match server.poll_requested_broadcast(&kio::Waiter::noop()) { - Poll::Ready(Ok(request)) => request, - _ => panic!("expected a queued request"), - }; + let request = queued(&mut server).await; // Only one request reaches the server. assert!(server.poll_requested_broadcast(&kio::Waiter::noop()).is_pending()); let source = broadcast::Info::new().produce(); request.accept(&source); - let first = first.now_or_never().expect("accepted").expect("resolves"); - let second = second.now_or_never().expect("accepted").expect("resolves"); + let first = first.await.expect("resolves"); + let second = second.await.expect("resolves"); assert!(first.is_clone(&second)); } @@ -3843,7 +4289,7 @@ mod tests { drop(announcement); drop(server); - let err = pending.now_or_never().expect("rejected").err().unwrap(); + let err = pending.await.err().unwrap(); assert!(matches!(err, Error::Unroutable)); // With the route gone, later requests are unroutable immediately. @@ -3870,9 +4316,9 @@ mod tests { let mut resolving = Box::pin(consumer.routed_broadcast("room/alice")); assert!((&mut resolving).now_or_never().is_none()); - // Each incumbent dies with the request still queued on it: the request - // fails Unroutable, and the resolve must retry through the next standby - // instead of parking on an announce update that never comes. Two + // Each incumbent dies with the front's request in flight on it: the + // front's watcher observes the retraction and retries through the next + // standby instead of parking on an announce update that never comes. Two // retractions in a row, so the announce stream's initial coverage replay // cannot paper over the missing retry. drop(incumbent); @@ -3882,17 +4328,11 @@ mod tests { drop(second_server); assert!((&mut resolving).now_or_never().is_none()); - let request = match standby_server.poll_requested_broadcast(&kio::Waiter::noop()) { - Poll::Ready(Ok(request)) => request, - _ => panic!("expected the retry to queue on the standby"), - }; + let request = queued(&mut standby_server).await; let source = broadcast::Info::new().produce(); request.accept(&source); - let resolved = resolving - .now_or_never() - .expect("resolves via the standby") - .expect("resolves"); + let resolved = resolving.await.expect("resolves via the standby"); assert_eq!(resolved.info().path.as_str(), "room/alice"); } @@ -3939,7 +4379,8 @@ mod tests { // Everything else still routes to the broad server. let _pending = consumer.request_broadcast("room/alice"); - assert!(broad_server.poll_requested_broadcast(&kio::Waiter::noop()).is_ready()); + let request = queued(&mut broad_server).await; + assert_eq!(request.path().as_str(), "room/alice"); } #[tokio::test] @@ -4030,6 +4471,201 @@ mod tests { assert!(late.next().now_or_never().expect("ended").is_none()); } + /// One live subscription reading a track through a remote front, plus the + /// bookkeeping to kill and replace its serving route. + struct ResumeRig { + producer: Producer, + resolved: broadcast::Consumer, + subscription: track::Subscriber, + /// Keeps the incumbent's track producing; dropping it would abort the + /// track out from under the front mid-test. + _incumbent_track: track::Producer, + } + + impl ResumeRig { + /// Announce a served route with `first` as its first hop, materialize + /// "room/alice" through it with a one-group "before" track, and subscribe. + async fn new(first: &[u64]) -> (Self, AnnounceProducer, broadcast::Producer) { + let producer = origin(1).produce(); + let consumer = producer.consume(); + + let (announcement, mut server) = producer + .announce_served("room", Route::default().with_hops(hops(first))) + .unwrap(); + + let pending = consumer.request_broadcast("room/alice"); + let request = queued(&mut server).await; + let mut source = broadcast::Info::new().produce(); + let mut track = source.create_track("video", None).unwrap(); + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); + group.finish().unwrap(); + request.accept(&source); + + let resolved = pending.await.expect("resolves"); + let mut subscription = resolved + .track("video") + .unwrap() + .subscribe(None) + .await + .expect("subscribe"); + let mut group = subscription + .recv_group() + .await + .expect("recv group") + .expect("track ended early"); + let frame = group.read_frame().await.expect("read frame").expect("frame"); + assert_eq!(&frame.payload[..], b"before"); + + ( + Self { + producer, + resolved, + subscription, + _incumbent_track: track, + }, + announcement, + source, + ) + } + + /// Stand up a second served route with `first` as its first hop and hand + /// back its server, ready to answer the front's re-request. + fn standby(&self, first: &[u64]) -> (AnnounceProducer, RouteServer) { + self.producer + .announce_served("room", Route::default().with_hops(hops(first))) + .unwrap() + } + } + + /// Accept the front's re-request on `server` with a source carrying the same + /// content stream (the delivered group plus its successor) and prove the + /// rig's subscription resumes onto it: the successor group is delivered on + /// the same subscription, at the group boundary. + async fn assert_resumes(rig: &mut ResumeRig, server: &mut RouteServer) { + let request = queued(server).await; + let mut replacement = broadcast::Info::new().produce(); + let mut track = replacement.create_track("video", None).unwrap(); + // The same content: group 0 was already delivered through the old route, + // so the splice resumes at group 1. + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"before".as_ref()).unwrap(); + group.finish().unwrap(); + request.accept(&replacement); + + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"resumed".as_ref()).unwrap(); + group.finish().unwrap(); + + let mut group = rig + .subscription + .recv_group() + .await + .expect("subscription survives the failover") + .expect("track ended early"); + let frame = group.read_frame().await.expect("read frame").expect("frame"); + assert_eq!(&frame.payload[..], b"resumed"); + } + + #[tokio::test] + async fn remote_source_resumes_through_same_first_hop() { + let (mut rig, incumbent, source) = ResumeRig::new(&[10]).await; + let (_standby, mut standby_server) = rig.standby(&[10, 20]); + + // The serving route dies: retraction plus source abort, like a session. + drop(incumbent); + drop(source); + + // The standby shares the first hop, so the subscription resumes there. + assert_resumes(&mut rig, &mut standby_server).await; + } + + #[tokio::test] + async fn different_first_hop_ends_the_subscription() { + let (mut rig, incumbent, source) = ResumeRig::new(&[10]).await; + // Another publisher entirely: same path, different first hop. + let (_rival, mut rival_server) = rig.standby(&[11]); + + drop(incumbent); + drop(source); + + // The subscription ends rather than splicing onto the rival's frames. + let err = rig.subscription.recv_group().await.err().expect("subscription ends"); + assert!(matches!(err, Error::Dropped), "unexpected end: {err}"); + + // A fresh request resolves through the rival. + let consumer = rig.producer.consume(); + let pending = consumer.request_broadcast("room/alice"); + let request = queued(&mut rival_server).await; + let replacement = broadcast::Info::new().produce(); + request.accept(&replacement); + pending.await.expect("re-request resolves through the rival"); + } + + #[tokio::test] + async fn anonymous_routes_never_resume() { + // An empty hop chain identifies nobody, so two of them must not pass for + // one publisher reconnecting. + let (mut rig, incumbent, source) = ResumeRig::new(&[]).await; + let (_twin, _twin_server) = rig.standby(&[]); + + drop(incumbent); + drop(source); + + let err = rig.subscription.recv_group().await.err().expect("subscription ends"); + assert!(matches!(err, Error::Dropped), "unexpected end: {err}"); + } + + #[tokio::test] + async fn reprice_is_invisible_to_the_subscription() { + let (rig, incumbent, mut source) = ResumeRig::new(&[10]).await; + + // A metadata-only reprice of the only route: nothing re-requests and the + // subscription keeps flowing from the same source. + incumbent + .update(Route::default().with_hops(hops(&[10])).with_cost(9)) + .unwrap(); + + let mut track = source.create_track("audio", None).unwrap(); + let mut group = track.append_group().unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"steady".as_ref()).unwrap(); + group.finish().unwrap(); + + let mut audio = rig + .resolved + .track("audio") + .unwrap() + .subscribe(None) + .await + .expect("subscribe survives the reprice"); + let mut group = audio + .recv_group() + .await + .expect("recv group") + .expect("track ended early"); + let frame = group.read_frame().await.expect("read frame").expect("frame"); + assert_eq!(&frame.payload[..], b"steady"); + } + + #[tokio::test] + async fn drain_reprice_migrates_before_the_session_dies() { + let (mut rig, incumbent, source) = ResumeRig::new(&[10]).await; + let (_standby, mut standby_server) = rig.standby(&[10, 20]); + + // The serving route drains: repriced to the ceiling while its session + // keeps serving. The front migrates to the standby without waiting for + // the death. + incumbent + .update(Route::default().with_hops(hops(&[10])).with_cost(DRAIN_COST)) + .unwrap(); + + assert_resumes(&mut rig, &mut standby_server).await; + + // The drained source outlived the migration. + drop(incumbent); + drop(source); + } + #[tokio::test] async fn local_sources_splice_newest_first() { let producer = origin(1).produce(); diff --git a/rs/moq-net/src/model/weak_cache.rs b/rs/moq-net/src/model/weak_cache.rs index dc791b293f5..acdc445d111 100644 --- a/rs/moq-net/src/model/weak_cache.rs +++ b/rs/moq-net/src/model/weak_cache.rs @@ -102,6 +102,13 @@ where None } + /// Iterate every entry, closed ones included, for a teardown walk. A closed + /// entry may still carry side state the walk must settle (a cancelled front's + /// parked requesters), so nothing is filtered here. + pub fn values(&self) -> impl Iterator { + self.map.values() + } + /// Remove and return the entry for `key`, if any. pub fn remove(&mut self, key: &Q) -> Option where diff --git a/rs/moq-relay/tests/goaway_cluster.rs b/rs/moq-relay/tests/goaway_cluster.rs index 6901dbdc307..5550c967221 100644 --- a/rs/moq-relay/tests/goaway_cluster.rs +++ b/rs/moq-relay/tests/goaway_cluster.rs @@ -373,9 +373,9 @@ async fn spawn_relay_with_upstream( /// gated: MID-B's first inbound connection can only be that reconnect). /// 3. The draining MID-A leg keeps serving through the handover window, so /// every group published across the swap arrives exactly once. -/// 4. Once the old leg closes, the subscription through it ends (failover is -/// a resubscribe, not a transparent splice) and a fresh subscription -/// through MID-B carries the rest. +/// 4. Once the old leg closes, the SAME subscription keeps flowing: both legs' +/// routes share TOP as their first hop, so BOTTOM's front re-splices onto +/// the MID-B leg at a group boundary instead of ending the subscription. /// 5. No GOAWAY leaks to the subscriber's own session, and the path never /// retracts under the subscriber. async fn cluster_diamond_goaway_seamless_failover_inner() { @@ -552,36 +552,10 @@ async fn cluster_diamond_goaway_seamless_failover_inner() { // BOTTOM, positively proving the new leg carries the resubscribe. drop(mid_a_upstream); - // The subscription was served through the MID-A leg, so its close ends it: - // failover is a resubscribe, not a transparent splice. Drain any groups - // still in flight, then observe the end. - within("old subscription ends with the drained leg", async { - while let Ok(Some(_)) = sub.recv_group().await {} - }) - .await; - - // Resubscribe on the same broadcast handle: the subscriber's session with - // BOTTOM is intact, and BOTTOM now resolves the track through MID-B. - drop(bc); - let bc = within("broadcast re-resolves after the failover", async { - let consumer = sub_origin.consume(); - consumer.request_broadcast("diamond").await.ok() - }) - .await - .expect("broadcast re-resolves"); - let mut sub = within( - "resubscribe after the failover", - bc.track("video") - .expect("track handle") - .subscribe(moq_net::track::Subscription::default().with_max_age(Duration::from_secs(60))), - ) - .await - .expect("resubscribe"); - - // A lite-05 subscription starts at the live edge (its Max Age is a staleness - // tolerance, not a replay request), so publish at a steady cadence and collect - // what lands once the fresh chain establishes. MID-A is severed, so any - // post-drain group can only have flowed TOP -> MID-B -> BOTTOM. + // The subscription was served through the MID-A leg, but its close does not + // end it: both legs' routes name TOP as their first hop, so BOTTOM's front + // re-splices onto the MID-B leg at a group boundary and the subscription + // rides through. Keep publishing and keep reading the SAME subscription. const POST_DRAIN_LAST: u64 = LAST_GROUP + 40; let post_publisher = tokio::spawn(async move { for seq in (LAST_GROUP + 1)..=POST_DRAIN_LAST { @@ -596,13 +570,19 @@ async fn cluster_diamond_goaway_seamless_failover_inner() { } }); - // Three distinct post-drain groups, every frame verified, proves the new leg - // carries the subscription (the establish may still see the pre-swap edge). + // Three distinct post-drain groups on the same subscription, every frame + // verified, proves the new leg carries it: MID-A is severed, so they can + // only have flowed TOP -> MID-B -> BOTTOM. let mut post = BTreeSet::new(); while post.iter().filter(|seq| **seq > LAST_GROUP).count() < 3 { collect_group(&mut sub, &mut post, FRAMES_PER_GROUP, "post-drain (MID-B leg only)").await; } post_publisher.abort(); + // The splice must not re-deliver anything the old leg already served. + assert!( + post.iter().all(|seq| *seq > LAST_GROUP), + "the splice re-delivered pre-failover groups: {post:?}" + ); // ── no GOAWAY cascade to the downstream subscriber ─────────────────── assert!( @@ -671,8 +651,8 @@ async fn collect_group(sub: &mut moq_net::track::Subscriber, seen: &mut BTreeSet } /// An empty-URI GOAWAY ("reconnect to me") makes the cluster redial the same -/// endpoint. The subscription through the drained session ends with it, and a -/// resubscribe through the redialed session resumes delivery. +/// endpoint. Both sessions' routes name the same first hop, so the subscription +/// through the drained session re-splices onto the redial and keeps delivering. async fn cluster_reconnects_on_empty_uri_goaway_inner() { let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); @@ -744,39 +724,17 @@ async fn cluster_reconnects_on_empty_uri_goaway_inner() { within("old session drains", first_dial.closed()).await; - // The broadcast was materialized through the first session, so its close ends - // the subscription: failover is a resubscribe, not a transparent splice. - within("old subscription ends with the session", async { - while let Ok(Some(_)) = sub.recv_group().await {} - }) - .await; - - // Re-request through the redialed session's route and resubscribe. - let bc = within("broadcast resolves through the redial", async { - cluster.origin.consume().request_broadcast("cam").await.ok() - }) - .await - .expect("broadcast resolves"); - let mut sub = within("resubscribe", async { - bc.track("video").expect("track handle").subscribe(None).await - }) - .await - .expect("resubscribe"); - + // The redialed session's route names the same first hop, so the broadcast + // re-splices onto it: the SAME subscription delivers the next group, with + // nothing re-delivered and no visible end. let mut g = track.append_group().expect("append group"); g.write_frame(moq_net::Timestamp::ZERO, b"empty_g1".as_ref()) .expect("write frame"); g.finish().expect("finish"); - // The fresh subscription may start at the retained group 0; skip up to g1. - let mut g1 = loop { - let g = within("recv g1 after the redial", sub.recv_group()) - .await - .expect("recv") - .expect("track ended early"); - if g.sequence >= 1 { - break g; - } - }; + let mut g1 = within("recv g1 through the redial", sub.recv_group()) + .await + .expect("subscription survives the redial") + .expect("track ended early"); assert_eq!(g1.sequence, 1, "the redialed session must deliver the new group"); assert_eq!( g1.read_frame().await.expect("read").expect("frame").payload[..], diff --git a/rs/moq-tokio/tests/broadcast.rs b/rs/moq-tokio/tests/broadcast.rs index 1c3d5e0e5ad..e161596deac 100644 --- a/rs/moq-tokio/tests/broadcast.rs +++ b/rs/moq-tokio/tests/broadcast.rs @@ -744,10 +744,10 @@ async fn read_payloads(sub: &mut moq_net::track::Subscriber, count: usize) -> Ve /// A path stays routed across two publisher sessions announcing it. /// /// The client connects to two servers announcing the same route. The preferred -/// (cheaper) route serves the track; when that session dies, the materialized -/// broadcast aborts and a fresh request resolves through the standby, without -/// the path ever being retracted. Routes claim capability, not content -/// identity, so the failover is an explicit re-request rather than a splice. +/// (cheaper) route serves the track; when that session dies, the broadcast +/// re-splices through the standby at a group boundary, without the path ever +/// being retracted: both routes name the same first hop, so they are the same +/// origin reached different ways and the subscription rides the failover. #[tracing_test::traced_test] #[tokio::test] async fn broadcast_route_migration() { @@ -872,28 +872,10 @@ async fn broadcast_route_migration() { .expect("subscribe failed"); assert_eq!(read_payloads(&mut sub, 1).await, ["a1"]); - // Kill the serving session. The materialized broadcast aborts; the next - // request resolves through the standby route and serves its content. + // Kill the serving session. Both routes name the same first hop, so the + // broadcast re-splices through the standby at the group boundary and the + // SAME subscription carries B's continuation. drop(session_a); - tokio::time::timeout(TIMEOUT, async { while sub.recv_group().await.is_ok() {} }) - .await - .expect("the dead session's subscription should abort"); - - let mut sub = tokio::time::timeout(TIMEOUT, async { - loop { - // The dead session's entry may still be routable for an instant; keep - // re-requesting until the standby serves the subscription. - if let Ok(broadcast) = sub_consumer.request_broadcast("test").await - && let Ok(track) = broadcast.track("video") - && let Ok(sub) = track.subscribe(subscription.clone()).await - { - return sub; - } - tokio::time::sleep(Duration::from_millis(10)).await; - } - }) - .await - .expect("the standby route should serve a fresh subscription"); assert_eq!(read_payloads(&mut sub, 2).await, ["b2", "b3"]); // The new subscription is live: what B produces from here on flows through it. From 93c610ed29ab9a25d96f424254615312cec63f5f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 1 Sep 2026 18:49:47 -0700 Subject: [PATCH 3/4] docs(cli): 1+1 publishers with a shared id hand over seamlessly Co-Authored-By: Claude Fable 5 --- doc/bin/cli.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/doc/bin/cli.md b/doc/bin/cli.md index 8c65ead3576..3c65c5956c2 100644 --- a/doc/bin/cli.md +++ b/doc/bin/cli.md @@ -380,14 +380,16 @@ identical routes, which relays deduplicate: the mesh sees one route while either publisher is alive, so killing one retracts nothing and the survivor keeps serving. -A relay never splices a live subscription from one publisher to another. -When the publisher serving a subscription goes away, that subscription ends -and the viewer re-subscribes, landing on the best remaining route; players do -this automatically when the path is (still) announced. So 1+1 buys a fast -resubscribe instead of a dead stream, not a seamless mid-group handover, and -the two encoders should still produce equivalent broadcasts (same track -names, comparable timelines) so a viewer that fails over keeps decoding. Run -the same command from two encoders, pinning the same id on both: +A relay resumes a live subscription across routes that name the same first +hop. When the publisher serving a subscription goes away, the relay +re-splices it through the best remaining route with that first hop at a group +boundary, and the viewer keeps watching. So 1+1 with a shared id buys a +seamless handover, and the two encoders must produce equivalent broadcasts +(same track names, aligned group sequences, comparable timelines) so the +spliced stream keeps decoding. Publishers with different ids (or none) never +splice: that subscription ends and the viewer re-subscribes, landing on the +best remaining route. Run the same command from two encoders, pinning the +same id on both: ```bash moq --hop 42 --connect https://relay-a.example.com/anon --broadcast event.hang import ts From 5bc765efe96ff74ecbf02410d2c5a18fbdf57ba5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 1 Sep 2026 19:21:30 -0700 Subject: [PATCH 4/4] fix(net): refuse a dead route's server and keep read handles weak Two Codex review findings on the front watcher: A RouteServer dropped while its announcement stands leaves the entry in the table with a closed queue; the watcher re-picked it and spun the origin driver on one core. A closed server now marks the route refused, retained for as long as the entry stands. Consumers cloned the owning Tasks handle, so a lingering read handle kept TaskSet::poll (and the documented Driver completion) from ever resolving after the producers dropped. Read handles now carry a non-owning TasksWeak that submits while the driver lives but never extends it. Co-Authored-By: Claude Fable 5 --- rs/moq-net/src/model/origin.rs | 56 ++++++++++++++++++++++++++++------ rs/moq-net/src/util.rs | 29 ++++++++++++++++++ 2 files changed, 76 insertions(+), 9 deletions(-) diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 14e48a03479..fce36959767 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -19,7 +19,7 @@ use crate::{ AsPath, Error, Path, PathOwned, PathPrefixes, coding::{BoundsExceeded, Decode, DecodeError, Encode, EncodeError}, runtime::{AnyTimers, Instant, Timers, TimersSlot}, - util::{TaskSet, Tasks}, + util::{TaskSet, Tasks, TasksWeak}, }; /// One relay's identity in a broadcast's hop chain: a 62-bit varint on the wire. @@ -2426,7 +2426,7 @@ struct RemoteFrontTask { exclude: Option, /// Resolves the requesters parked on the front's channel. request: kio::Producer, - tasks: Tasks, + tasks: TasksWeak, timers: TimersSlot, } @@ -2537,9 +2537,14 @@ async fn run_remote_front(task: RemoteFrontTask) { if let Some((route, first, server)) = fire { let mut serve = server.lock(); match serve.closed { - // Retracted between the table read and its lock: the table has - // already moved on, so just re-decide. - true => continue 'run, + // The server is gone: retracted under us, or its handler dropped + // while the announcement stands. Either way the route cannot + // serve, so skip it rather than re-picking it forever. + true => { + refused.insert(route); + last_err = Some(Error::Unroutable); + continue 'run; + } false => { // A source this route already materialized for the path // attaches without another upstream round trip. @@ -3363,9 +3368,10 @@ pub struct Consumer { // mirroring what `create_broadcast` gives a local front. origin: Info, - // Submission handle to the origin's [`Driver`], for the front watcher a - // routed `request_broadcast` spawns. Closed once the driver drops. - tasks: Tasks, + // Non-owning submission handle to the origin's [`Driver`], for the front + // watcher a routed `request_broadcast` spawns. Non-owning so a lingering + // read handle never keeps the driver from finishing. + tasks: TasksWeak, // The driver's clock and timers, threaded into fronts for the track idle // linger. @@ -3390,7 +3396,7 @@ impl Consumer { stats, exclude: None, origin: producer.info(), - tasks: producer.tasks.clone(), + tasks: producer.tasks.downgrade(), timers: producer.timers.clone(), } } @@ -4567,6 +4573,38 @@ mod tests { assert_eq!(&frame.payload[..], b"resumed"); } + /// A `RouteServer` dropped while its announcement stands leaves the entry in + /// the table with a closed queue. The front must refuse it rather than + /// re-pick it forever, which would spin the origin driver on one core. + #[tokio::test] + async fn dropped_server_with_live_announcement_is_refused() { + let producer = origin(1).produce(); + let consumer = producer.consume(); + let (_announcement, server) = producer.announce_served("room", Route::default()).unwrap(); + drop(server); + + let err = tokio::time::timeout(Duration::from_secs(5), consumer.request_broadcast("room/alice")) + .await + .expect("the front must give up, not spin") + .err() + .unwrap(); + assert!(matches!(err, Error::Unroutable)); + } + + /// The driver's completion contract: it resolves once every producer handle + /// drops, however many read handles remain. + #[tokio::test] + async fn driver_resolves_with_live_consumers() { + let (producer, driver) = Producer::new(Info::new(origin(1))); + let consumer = producer.consume(); + let run = driver.run(crate::runtime::tokio_test::Tokio::<()>::new()); + drop(producer); + tokio::time::timeout(Duration::from_secs(5), run) + .await + .expect("driver must finish once the producers are gone"); + drop(consumer); + } + #[tokio::test] async fn remote_source_resumes_through_same_first_hop() { let (mut rig, incumbent, source) = ResumeRig::new(&[10]).await; diff --git a/rs/moq-net/src/util.rs b/rs/moq-net/src/util.rs index 8f835e07e1b..d26001a5955 100644 --- a/rs/moq-net/src/util.rs +++ b/rs/moq-net/src/util.rs @@ -85,6 +85,17 @@ impl Tasks { } } +impl Tasks { + /// A non-owning submission handle: pushes work while the driver lives, but + /// neither keeps the set from finishing nor counts as a sender. For read + /// handles, whose existence must not extend the origin's lifecycle. + pub fn downgrade(&self) -> TasksWeak { + TasksWeak { + state: self.state.clone(), + } + } +} + impl Clone for Tasks { fn clone(&self) -> Self { self.state.lock().senders += 1; @@ -94,6 +105,24 @@ impl Clone for Tasks { } } +/// Non-owning [`Tasks`]: same submissions, no claim on the set's lifetime, so +/// [`TaskSet::poll`] still finishes once the owning handles drop. +#[derive(Clone)] +pub(crate) struct TasksWeak { + state: kio::Shared, +} + +impl TasksWeak { + /// Queue a future for polling by the associated [`TaskSet`], dropped if the + /// set (or every owning [`Tasks`] handle) is already gone. + pub fn push(&self, task: impl MaybeBoxedExt<'static, Output = ()>) { + let mut state = self.state.lock(); + if !state.closed && state.senders > 0 { + state.queued.push_back(task.maybe_boxed()); + } + } +} + impl Drop for Tasks { fn drop(&mut self) { // The mutation wakes a parked set, which may be what lets it finish.