diff --git a/quest/main/README.md b/quest/main/README.md index f3a8f9cc22a..26185fe2b02 100644 --- a/quest/main/README.md +++ b/quest/main/README.md @@ -2,8 +2,7 @@ ## Goal -Make the origin's failover a state machine with a runtime-free test per -transition, the correctness core of the relay. Alongside it, settle the public contracts of `moq-archive`, `moq-e2ee`, `moq-sock`, +Settle the public contracts of `moq-archive`, `moq-e2ee`, `moq-sock`, `moq-uring`, `moq-audio`, `moq-video`, `moq-transcode`, and `moq-nvenc` before the imminent release, while supplying the reusable GPU media support needed to remove raw-pixel CPU transfers from the Pronto CARLA demo. These are @@ -90,9 +89,6 @@ do not add another media abstraction or a renderer crate during stabilization. ## Quests - [Binding parity](/quest/main/binding-parity.md) - every wrapper reaches every moq-ffi method in its own idiom with moq-net's verbs -- [Origin failover machine](/quest/main/origin-failover-machine.md) - a front's - source selection and track splicing become a pure step function with a - runtime-free test per transition ## Related diff --git a/quest/main/origin-failover-machine.md b/quest/main/origin-failover-machine.md deleted file mode 100644 index dd2ad80c188..00000000000 --- a/quest/main/origin-failover-machine.md +++ /dev/null @@ -1,101 +0,0 @@ -# [L] Origin failover as a state machine - -## Goal - -The origin's failover, how a front picks the source it serves from and how -each spliced track follows that choice, is a pure step function: events in, -actions out, no locks and no waiters inside. The async task around it only -feeds events and executes actions. Every transition has a test that needs no -runtime, and a bounded enumeration over event sequences checks the invariants -a wake predicate used to guard by hand: an event that changes nothing produces -no action, a source that refused a track is never dispatched for it again, a -source change produces at most one takeover per track. - -Subscribers keep today's guarantees: the newest local publisher at a path wins -the moment it attaches, a remote front resumes only through routes sharing the -first hop that served it, a source refusing one track is skipped for that -track alone, and a front with nothing left to serve ends with the last -refusal's error. Behavior only: everything stays private to -`rs/moq-net/src/model/origin.rs` except the action surface of -`resume::Producer`, which may move to fit the machine. No wire or public API -change; lands on main. - -## Plan - -### Root problem - -Every livelock and spin this file has had was a wake predicate disagreeing -with the decision it guarded: the resume memo of #3312, the `NoRoute` edge -that kept firing, the `dead` set that starved the watcher. #3882 and #3884 -removed the table-wide predicates (fronts now sleep on a `Watch` at their -path node), but `serve_track` still polls `FrontState` with a predicate that -must mirror `serve_route`, and the decisions are spread across five pieces of -state per source (`refused`, `dead`, `refusal`, `spliced_edge`, `Identity`) -plus a multi-source table only local fronts use. Make the decision data, not -a closure, and the class disappears. - -### Shape - -- One machine per front, owning its per-track state. One driver task per - front polls exactly the waits the machine names (the watch, the in-flight - upstream request, the serving source's close, each spliced track's - completion, each track's demand edge, one deadline) and hands the result - back as an event. Tracks are entries in the machine's state with their own - demand gate and idle linger; there is no per-track task. -- Events, roughly: a covering route changed, the upstream request resolved or - refused, the serving source closed, a track was assigned, a track completed - or failed (with whether it delivered anything since its splice), a reader - arrived or the last one left, the linger deadline passed, the origin closed. - Actions, roughly: request through a route, attach or detach a source, - splice a track from a source, release a track's segment, abort or finish a - track, end the front, arm a deadline. Adjust the alphabet as the tests - demand; the point is that both are plain data. -- One source slot per front. Local publishers are already route entries - (`local: true`) and `route_order` prefers the newest, so the table selects - for local and remote alike and `FrontState.sources` goes away. Keep what - `attach_source` guarantees today: a front whose sources have all closed is - replaced by a fresh broadcast, never spliced into, and the join-or-create - decision happens under one lock. -- Per-track verdicts stay per track (a standby that has not created every - track must not cost the incumbent anything), but they become one enum per - track (unspliced, waiting on info, spliced from a source, refused by these - sources, done) instead of parallel sets. The corpse distinction goes: a - source that is closing is never dispatched, and its watcher's detach is the - only signal the machine waits for. -- The identity pin (first hop of the route that first served the front) stays - the rule for which routes qualify on re-selection. - -### Proof - -- A transition table: for each state and event, the expected actions and next - state, as plain unit tests. -- A bounded exhaustive enumerator over the event alphabet with a small model - (two sources, two tracks, sequences up to a handful of steps) asserting the - invariants above. No new dependency; a hand-rolled enumerator is enough. -- The existing tokio tests stay as the end-to-end proof and pass unmodified - except where a dropped rule (the corpse distinction) changes what they - assert. `benches/origin.rs` `handoff` is the before/after for the handover - latency; it must not regress. - -### Look out for - -- `Watch::seen` is read under the table lock beside the decision it guards; - a watch is never dropped while holding that lock. -- The demand gate: a release on the unused edge must not re-splice and spin; - both directions key off the same signal today, keep that. -- `resume::Producer::takeover` errors other than `Closed` are boundary bugs; - the machine aborts the track rather than retrying. -- Track metadata compatibility (`accept_track_info`) is a refusal, not a - failover. -- Teardown order: `closed` is set and every watch poked under the table lock - before requesters are rejected. - -## Required - -- [#3884](https://github.com/moq-dev/moq/pull/3884) merged: the front watch this builds on - -## Related - -- [Origin narrowing](/quest/next/origin-narrowing.md) - a live grant narrowing ends broadcasts the origin handed out; rebases onto the new shape -- [Dynamic track sequences](/quest/next/2991-net-coalesce-dynamic-tracks-and-preserve-sequences-across.md) - the takeover edge this machine drives -- [Relay memory](/quest/next/relay-memory.md) - per-front state is part of what it measures diff --git a/rs/moq-net/benches/origin.rs b/rs/moq-net/benches/origin.rs index 795fdd6560f..576d5ce5371 100644 --- a/rs/moq-net/benches/origin.rs +++ b/rs/moq-net/benches/origin.rs @@ -25,7 +25,7 @@ const SHAPES: [(usize, usize); 3] = [(100, 10), (1_000, 100), (1_000, 1_000)]; struct Fanout { producer: origin::Producer, consumer: origin::Consumer, - _driver: origin::Driver, + driver: origin::Driver, _publishers: Vec, subscribers: Vec, } @@ -44,7 +44,7 @@ fn fanout(publishers: usize, subscribers: usize) -> Fanout { Fanout { producer, consumer, - _driver: driver, + driver, _publishers: publishers, subscribers, } @@ -165,24 +165,27 @@ fn bench_subscribe(c: &mut Criterion) { group.finish(); } -/// Resolving one broadcast among `publishers`: a local hit walks the tree to -/// the exact path, and a miss under an advertise-only route walks the route -/// table to prove nothing serves it. Neither may depend on `publishers`. +/// Resolving one broadcast among `publishers`: a local hit walks the table to +/// the exact path and joins the front serving it, and a miss under a broadcast +/// published above it walks the table to prove nothing serves it. Neither may +/// depend on `publishers`. fn bench_request(c: &mut Criterion) { let mut group = c.benchmark_group("origin/request"); // A request costs nothing per subscriber, so this sweeps the publisher counts // in `SHAPES` rather than its shapes, whose last two share one. for publishers in [100, 1_000] { - let fleet = fanout(publishers, 0); - // An advertise-only route above the misses: it covers them without serving them. + let mut fleet = fanout(publishers, 0); + // A broadcast above the misses covers them without serving them. let _covering = fleet.producer.publish("room", origin::Route::default()).unwrap(); + let waiter = kio::Waiter::noop(); group.bench_function(BenchmarkId::new("local", publishers), |b| { b.iter(|| { - fleet - .consumer - .request_broadcast("room/0") + let pending = fleet.consumer.request_broadcast("room/0"); + // The front's driver resolves the first request; later ones join it. + fleet.driver.poll(moq_net::time::Instant::now(), &waiter).unwrap(); + pending .now_or_never() - .expect("local resolves synchronously") + .expect("resolves once driven") .expect("local broadcast"); }); }); diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 67a7b9baeed..1b2fab15980 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -343,7 +343,7 @@ impl TrackState { /// the broadcast now or holds it open for a replacement. /// /// Only the detach that drops the refcount to zero decides, matching the model's rule -/// for several sources at one path (`detach_source`): an earlier owner that vanished +/// for several sources at one path (the front's source selection): an earlier owner that vanished /// does not outvote the last one still on the path. That keeps two advertisements on /// one session behaving like the same two on separate sessions, where the model sees /// two independent sources and the last one out decides. @@ -4220,7 +4220,8 @@ mod tests { /// is the one that counts: the broadcast survives the first stop and closes on the /// last, whatever kind each detach is. /// - /// That is the model's own rule for several sources at one path (`detach_source`), + /// That is the model's own rule for several sources at one path (the front's source + /// selection), /// which is what these advertisements would be had they arrived on two sessions. The /// refcount is a detail of sharing one `SourceGuard` per session; it must not change /// what the origin sees. diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index c8bbadf0ccc..7350301cc76 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -2121,11 +2121,7 @@ mod tests { // The peer's own subscription, excluding the hop the server minted for // it, is served from the local front before anything is announced back. let peer = origin.consume().excluding(assigned); - let resolved = peer - .request_broadcast("room/host") - .now_or_never() - .expect("local lookup is synchronous") - .expect("resolves"); + let resolved = peer.request_broadcast("room/host").await.expect("resolves"); let mut sub = resolved .track("video") .unwrap() @@ -2166,10 +2162,11 @@ mod tests { .unwrap(); assert!(!accepted, "an announce that already names this origin must be dropped"); - // The local front is still the one at the path, and still serving. - let still = origin - .consume() - .get_broadcast("room/host") + // The local front is still the one at the path, and still serving: the + // peer's next request joins it rather than minting another. + let still = peer + .request_broadcast("room/host") + .await .expect("the local front keeps serving"); assert!( still.is_clone(&resolved), diff --git a/rs/moq-net/src/model/broadcast.rs b/rs/moq-net/src/model/broadcast.rs index 0897a446cb0..52acc81924d 100644 --- a/rs/moq-net/src/model/broadcast.rs +++ b/rs/moq-net/src/model/broadcast.rs @@ -524,19 +524,19 @@ impl Alive { }) } - /// Retract the path's advertisement, if any. + /// Withdraw the path's advertisement, if any; the broadcast stays servable. fn unannounce(&self) { - let announcement = self.announcer.lock().as_mut().and_then(Announcer::take); - // Retracted outside the announcer lock: the retraction re-syncs the origin's - // announce cursors under the origin's own lock. - drop(announcement); + if let Some(announcer) = self.announcer.lock().as_mut() { + announcer.withdraw(); + } } /// End the broadcast's advertising for good: retract the standing advertisement /// and drop the announcer, so a later `announce` fails with `Closed`. fn retire(&self) { let announcer = self.announcer.lock().take(); - // Dropped outside the announcer lock, like `unannounce`. + // Dropped outside the announcer lock: the entry's removal re-syncs the + // origin's cursors under the origin's own lock. drop(announcer); } } diff --git a/rs/moq-net/src/model/front.rs b/rs/moq-net/src/model/front.rs new file mode 100644 index 00000000000..6ee9d797592 --- /dev/null +++ b/rs/moq-net/src/model/front.rs @@ -0,0 +1,1270 @@ +//! The origin's failover as a state machine. +//! +//! A [`Front`] is one path served through a spliced broadcast: it picks the +//! source it serves from and splices each logical track from that source's +//! copy, re-splicing when the source is replaced and refusing what a source +//! will not serve. Everything here is plain data: [`Front::step`] takes one +//! [`Event`] and returns the [`Action`]s to perform, without locks or waiters. +//! The origin's driver task feeds events from the world (the route table, the +//! sources, the tracks, the clock) and executes the actions; see +//! `origin::run_front`. +//! +//! Sources and tracks are named by ids and names, never handles, so a +//! transition can be checked in a unit test by comparing the actions it emits. + +use std::{ + collections::{BTreeMap, HashSet}, + sync::Arc, + time::Duration, +}; + +use crate::{Error, Hop, runtime::Instant, track}; + +/// A route the table selected for the front: the entry id, the endpoint that +/// originated it, and whether it is a broadcast published on this origin. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) struct Candidate { + pub route: u64, + pub first: Option, + pub local: bool, +} + +/// Why an upstream request through a route did not produce a source. +#[derive(Clone, Debug)] +pub(super) struct Refusal { + pub err: Error, + /// Whether the route is still in the table. A refusal from a standing route + /// is the handler's answer and is never re-asked; one from a retracted route + /// only means the table moved. + pub standing: bool, +} + +/// What happened in the world. +#[derive(Clone, Debug)] +pub(super) enum Event { + /// The routes covering the path were re-read: the best qualifying route, if + /// any, and whether the serving source has begun closing. + Selected { + best: Option, + serving_closing: bool, + }, + /// The request through `route` resolved with a source (registered under the + /// given id by the driver) or was refused. + Resolved { route: u64, result: Result }, + /// A source closed: it will never serve again. + SourceClosed { source: u64 }, + /// The spliced broadcast handed out a new logical track to serve. + TrackAssigned { track: Arc }, + /// A source answered a track query: its copy's metadata, or a refusal. + /// `closing` is whether the source has begun closing, in which case a + /// refusal is not held against it: it is on its way out. + TrackInfo { + track: Arc, + source: u64, + closing: bool, + result: Result, + }, + /// The spliced copy of a track ended: completed, or died. `delivered` is + /// whether it produced anything since it was spliced in; a copy that died + /// without delivering refused the track. + TrackEnded { + track: Arc, + source: u64, + closing: bool, + result: Result<(), Error>, + delivered: bool, + }, + /// A reader arrived on the track. + Used { track: Arc }, + /// The last reader left the track. + Unused { track: Arc, now: Instant }, + /// The armed deadline passed. + Deadline { now: Instant }, + /// The origin is tearing down. + Closed, +} + +/// What the driver does in the world. +#[derive(Clone, Debug)] +pub(super) enum Action { + /// Ask the route table for the best qualifying route and feed it back as + /// [`Event::Selected`]. + Reselect, + /// Request the path through `route`; feed the outcome back as + /// [`Event::Resolved`]. + Request { route: u64 }, + /// Drop the driver's handle on a source. + Detach { source: u64 }, + /// Resolve the requesters parked on the front: it has a source. + Resolve, + /// Ask `source` for its copy of `track`; feed the answer back as + /// [`Event::TrackInfo`]. + Query { track: Arc, source: u64 }, + /// Splice `source`'s copy of `track` in, resuming where the segments stop. + Splice { track: Arc, source: u64 }, + /// Drop the source copy of `track` but keep the delivered groups spliced, + /// so resume stays seamless while nobody reads. + Park { track: Arc }, + /// Drop the delivered groups of `track` too: the linger expired unread. + Release { track: Arc }, + /// The logical track completed. + Finish { track: Arc }, + /// The logical track failed for good. + Abort { track: Arc, err: Error }, + /// Arm (or clear) the deadline the front wants to be woken at. + Arm { at: Option }, + /// The front is over: abort every spliced track and reject the parked + /// requesters with `err`. + End { err: Error }, +} + +/// The content identity of a front: who its first source came from. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum Identity { + /// No source has attached yet: the first request may resolve through any + /// covering route, and whoever serves it fixes the identity. + Undetermined, + /// The first source was a broadcast published on this origin. A newer local + /// publisher takes over while the incumbent is live; once the incumbent is + /// closing the front ends instead, so a newcomer gets a fresh broadcast + /// rather than being spliced into one that is over. + Local, + /// The serving route's first hop was absent or [`Hop::UNKNOWN`], which + /// identifies nobody: the front cannot resume, so its source ending ends it. + Anonymous, + /// The first hop of the serving route. Routes sharing it are the same + /// origin reached another way and safe to resume through. + Publisher(Hop), +} + +/// Which routes qualify for a front's (re)selection. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(super) enum Pin { + /// Any served route. + Any, + /// Only broadcasts published on this origin. + Local, + /// Only routes originated by this first hop. + Publisher(Hop), + /// Nothing: the front never re-selects. + None, +} + +impl Identity { + fn pin(self) -> Pin { + match self { + Self::Undetermined => Pin::Any, + Self::Local => Pin::Local, + Self::Anonymous => Pin::None, + Self::Publisher(hop) => Pin::Publisher(hop), + } + } +} + +/// Where a track stands with respect to its source copy. +#[derive(Clone, Debug, PartialEq, Eq)] +enum TrackState { + /// No copy: nothing spliced, or the segment stays spliced from a source + /// that left. + Idle, + /// A source was asked for its copy. + Querying { source: u64 }, + /// A source's copy is spliced in and being served. + Spliced { source: u64 }, + /// Nobody reads it: the copy was dropped, the delivered groups stay + /// spliced until the linger expires. + Parked { since: Instant }, +} + +#[derive(Clone, Debug)] +struct Track { + state: TrackState, + /// Sources that refused this track, by id. A refusal is authoritative for + /// that source; a source that reattaches has a new id and is asked afresh. + refused: HashSet, + /// The most recent refusal, reported when every source has refused. + refusal: Option, + /// Whether anyone reads the track: a copy is only spliced in for a reader. + used: bool, +} + +/// One front's state; see the module docs. +#[derive(Clone, Debug)] +pub(super) struct Front { + identity: Identity, + /// The attached source and the route that produced it. + serving: Option<(u64, u64)>, + /// Whether the serving source has begun closing, as of the last event that + /// said. A closing source is never asked for anything again. + serving_closing: bool, + /// The route an upstream request is in flight through. + upstream: Option, + /// Routes that refused the path while another source was serving. + refused: HashSet, + /// Why the last candidate fell through, reported if the front ends unresolved. + last_err: Option, + /// Whether the parked requesters were resolved (the first source attached). + resolved: bool, + tracks: BTreeMap, Track>, + /// Immutable track metadata, fixed by the first copy of each track. Every + /// source of the broadcast must serve the same content. + info: BTreeMap, track::Info>, + /// How long an unread track keeps its delivered groups spliced. + linger: Duration, + /// The deadline last armed, so a step only re-arms on change. + armed: Option, + ended: bool, +} + +impl Front { + /// A front with no source and no tracks; `linger` is how long an unread + /// track keeps its delivered groups spliced. + pub(super) fn new(linger: Duration) -> Self { + Self { + identity: Identity::Undetermined, + serving: None, + serving_closing: false, + upstream: None, + refused: HashSet::new(), + last_err: None, + resolved: false, + tracks: BTreeMap::new(), + info: BTreeMap::new(), + linger, + armed: None, + ended: false, + } + } + + /// Which routes qualify for the next selection. + pub(super) fn pin(&self) -> Pin { + self.identity.pin() + } + + /// The routes that refused the path; the driver skips them when selecting. + pub(super) fn refused_routes(&self) -> &HashSet { + &self.refused + } + + /// Forget refused routes that left the table (a reconnect is a fresh entry). + pub(super) fn retain_routes(&mut self, standing: impl Fn(u64) -> bool) { + self.refused.retain(|route| standing(*route)); + } + + /// The attached source, if any. + pub(super) fn serving(&self) -> Option { + self.serving.map(|(source, _)| source) + } + + /// Whether the front is over. + #[cfg(test)] + fn ended(&self) -> bool { + self.ended + } + + /// Apply one event and return what to do about it. Nothing after [`Action::End`]. + pub(super) fn step(&mut self, event: Event) -> Vec { + let mut actions = Vec::new(); + if self.ended { + return actions; + } + match event { + Event::Selected { best, serving_closing } => self.selected(best, serving_closing, &mut actions), + Event::Resolved { route, result } => self.resolved(route, result, &mut actions), + Event::SourceClosed { source } => self.source_closed(source, &mut actions), + // A fresh logical track, even under a name served before: an earlier + // verdict belonged to that request, and a later one asks afresh. The + // broadcast's track metadata is what persists. + Event::TrackAssigned { track } => { + self.tracks.insert( + track, + Track { + state: TrackState::Idle, + refused: HashSet::new(), + refusal: None, + used: false, + }, + ); + } + Event::TrackInfo { + track, + source, + closing, + result, + } => self.track_info(track, source, closing, result, &mut actions), + Event::TrackEnded { + track, + source, + closing, + result, + delivered, + } => self.track_ended(track, source, closing, result, delivered, &mut actions), + Event::Used { track } => self.used(track, &mut actions), + Event::Unused { track, now } => self.unused(track, now, &mut actions), + Event::Deadline { now } => self.deadline(now, &mut actions), + Event::Closed => self.end(Error::Dropped, &mut actions), + } + if !self.ended { + let at = self.next_deadline(); + if at != self.armed { + self.armed = at; + actions.push(Action::Arm { at }); + } + } + actions + } + + fn selected(&mut self, best: Option, serving_closing: bool, actions: &mut Vec) { + if self.serving.is_some() { + self.serving_closing = serving_closing; + } + let serving_route = self.serving.map(|(_, route)| route); + match best { + // The serving route is still the best qualifying one. + Some(candidate) if Some(candidate.route) == serving_route => { + self.upstream = None; + } + // Already asking through it. + Some(candidate) if self.upstream == Some(candidate.route) => {} + // A local newcomer must not be spliced into a broadcast whose + // publisher is on the way out: end, and a fresh front serves it. + Some(candidate) if candidate.local && self.identity == Identity::Local && serving_closing => { + self.end(Error::Dropped, actions); + } + // A better (or replacement) qualifying route: request through it. + Some(candidate) => { + self.upstream = Some(candidate.route); + actions.push(Action::Request { route: candidate.route }); + } + // Nothing qualifies. A live source keeps serving (its route may + // return); without one the front is over. + None => { + self.upstream = None; + if self.serving.is_none() { + let err = match self.identity { + Identity::Undetermined => self.last_err.take().unwrap_or(Error::Unroutable), + _ => self.last_err.take().unwrap_or(Error::Dropped), + }; + self.end(err, actions); + } + } + } + } + + fn resolved(&mut self, route: u64, result: Result, actions: &mut Vec) { + match result { + // A source for a request we no longer want: let it go. + Ok(source) if self.upstream != Some(route) => actions.push(Action::Detach { source }), + Ok(source) => { + self.upstream = None; + self.attach(source, route, actions); + } + Err(_) if self.upstream != Some(route) => {} + // The route retracted before serving: the table already reflects + // it, so the next selection retries the survivor. + Err(Refusal { standing: false, .. }) => { + self.upstream = None; + self.last_err = Some(Error::Unroutable); + actions.push(Action::Reselect); + } + // An authoritative refusal of the path. It ends a front with no + // other source; a serving front merely skips the refuser. + Err(Refusal { err, standing: true }) => { + self.upstream = None; + match self.serving { + Some(_) => { + self.refused.insert(route); + self.last_err = Some(err); + actions.push(Action::Reselect); + } + None => self.end(err, actions), + } + } + } + } + + fn attach(&mut self, source: u64, route: u64, actions: &mut Vec) { + if let Some((old, _)) = self.serving.take() { + actions.push(Action::Detach { source: old }); + // Its copies are gone with it; the segments they delivered stay + // spliced and the replacement resumes past them. + for track in self.tracks.values_mut() { + if matches!(track.state, TrackState::Querying { source: s } | TrackState::Spliced { source: s } if s == old) + { + track.state = TrackState::Idle; + } + } + } + self.serving = Some((source, route)); + self.serving_closing = false; + if !self.resolved { + self.resolved = true; + actions.push(Action::Resolve); + } + // A read track is never parked, so idle is the only state to re-query. + for (name, track) in &mut self.tracks { + if track.used && track.state == TrackState::Idle { + track.state = TrackState::Querying { source }; + actions.push(Action::Query { + track: name.clone(), + source, + }); + } + } + } + + /// Fix the identity from the first source's candidate. Called by the driver + /// with the candidate it requested through, before the source resolves. + pub(super) fn identify(&mut self, candidate: Candidate) { + if self.identity != Identity::Undetermined { + return; + } + self.identity = match (candidate.local, candidate.first) { + (true, _) => Identity::Local, + (false, Some(hop)) if hop != Hop::UNKNOWN => Identity::Publisher(hop), + (false, _) => Identity::Anonymous, + }; + } + + fn source_closed(&mut self, source: u64, actions: &mut Vec) { + let Some((serving, _)) = self.serving else { + return; + }; + if serving != source { + return; + } + self.serving = None; + self.serving_closing = false; + actions.push(Action::Detach { source }); + for track in self.tracks.values_mut() { + if matches!(track.state, TrackState::Querying { source: s } | TrackState::Spliced { source: s } if s == source) + { + track.state = TrackState::Idle; + } + } + self.last_err = Some(Error::Dropped); + match self.identity { + // A local publisher ending ends its broadcast; a newcomer at the + // path gets a fresh one. An anonymous source can never be resumed. + Identity::Local | Identity::Anonymous | Identity::Undetermined => self.end(Error::Dropped, actions), + Identity::Publisher(_) => actions.push(Action::Reselect), + } + } + + fn track_info( + &mut self, + name: Arc, + source: u64, + closing: bool, + result: Result, + actions: &mut Vec, + ) { + let Some(track) = self.tracks.get_mut(&name) else { + return; + }; + if track.state != (TrackState::Querying { source }) { + return; + } + let verdict = match result { + // A copy claiming the same content with different metadata is + // refused rather than reinterpreted. + Ok(info) => match self.info.get(&name) { + Some(expected) + if expected.timescale != info.timescale + || expected.max_age != info.max_age + || expected.priority != info.priority => + { + Err(Error::Unsupported) + } + Some(_) => Ok(()), + None => { + self.info.insert(name.clone(), info); + Ok(()) + } + }, + Err(err) => Err(err), + }; + match verdict { + Ok(()) => { + track.state = TrackState::Spliced { source }; + actions.push(Action::Splice { + track: name.clone(), + source, + }); + } + Err(err) => self.refuse(name, source, closing, err, actions), + } + } + + fn track_ended( + &mut self, + name: Arc, + source: u64, + closing: bool, + result: Result<(), Error>, + delivered: bool, + actions: &mut Vec, + ) { + let Some(track) = self.tracks.get_mut(&name) else { + return; + }; + if track.state != (TrackState::Spliced { source }) { + return; + } + match result { + // Over for good: the track leaves the machine, so a name served again + // later starts afresh and a long-lived front does not keep a stub per + // name it ever served. + Ok(()) => { + self.tracks.remove(&name); + actions.push(Action::Finish { track: name }); + } + // Died mid-serve after delivering: normal failover, re-splice from + // the serving source at the first missing group. + Err(_) if delivered => { + track.state = TrackState::Idle; + self.redispatch(name, actions); + } + // Died before producing anything: a refusal. + Err(err) => self.refuse(name, source, closing, err, actions), + } + } + + /// Record that `source` will not serve `name`, then either ask another + /// source or, when the serving source itself refused, abort the track. + fn refuse(&mut self, name: Arc, source: u64, closing: bool, err: Error, actions: &mut Vec) { + let track = self.tracks.get_mut(&name).expect("refusing a known track"); + track.state = TrackState::Idle; + // A closing source's answer is not a verdict: it is on its way out and + // its replacement is asked afresh. + if closing { + return; + } + track.refused.insert(source); + track.refusal = Some(err); + self.redispatch(name, actions); + } + + /// Splice `name` from the serving source if it can serve it, or abort the + /// track once the serving source has refused it: refusals are never + /// retried, and no other source will be asked while this one serves. + fn redispatch(&mut self, name: Arc, actions: &mut Vec) { + let Some((source, _)) = self.serving else { + return; + }; + let track = self.tracks.get_mut(&name).expect("dispatching a known track"); + if !track.used || track.state != TrackState::Idle { + return; + } + if track.refused.contains(&source) { + let err = track.refusal.clone().unwrap_or(Error::NotFound); + self.tracks.remove(&name); + actions.push(Action::Abort { track: name, err }); + return; + } + if self.serving_closing { + return; + } + track.state = TrackState::Querying { source }; + actions.push(Action::Query { track: name, source }); + } + + fn used(&mut self, name: Arc, actions: &mut Vec) { + let Some(track) = self.tracks.get_mut(&name) else { + return; + }; + track.used = true; + if let TrackState::Parked { .. } = track.state { + track.state = TrackState::Idle; + } + self.redispatch(name, actions); + } + + fn unused(&mut self, name: Arc, now: Instant, actions: &mut Vec) { + let Some(track) = self.tracks.get_mut(&name) else { + return; + }; + track.used = false; + match track.state { + // Drop the copy so the source goes idle at once; the delivered + // groups stay spliced for the linger. + TrackState::Spliced { .. } => { + track.state = TrackState::Parked { since: now }; + actions.push(Action::Park { track: name }); + } + TrackState::Querying { .. } => track.state = TrackState::Idle, + _ => {} + } + } + + fn deadline(&mut self, now: Instant, actions: &mut Vec) { + // The driver's deadline fired and cleared itself: re-arm whatever is still + // parked, even if nothing was due yet, or it would never be released. + self.armed = None; + for (name, track) in &mut self.tracks { + if let TrackState::Parked { since } = track.state + && since + self.linger <= now + { + track.state = TrackState::Idle; + actions.push(Action::Release { track: name.clone() }); + } + } + } + + fn next_deadline(&self) -> Option { + self.tracks + .values() + .filter_map(|track| match track.state { + TrackState::Parked { since } => Some(since + self.linger), + _ => None, + }) + .min() + } + + fn end(&mut self, err: Error, actions: &mut Vec) { + if self.ended { + return; + } + self.ended = true; + if let Some((source, _)) = self.serving.take() { + actions.push(Action::Detach { source }); + } + actions.push(Action::End { err }); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const LINGER: Duration = Duration::from_secs(30); + + fn hop(id: u64) -> Hop { + Hop::new(id).unwrap() + } + + fn remote(route: u64, first: u64) -> Candidate { + Candidate { + route, + first: Some(hop(first)), + local: false, + } + } + + fn local(route: u64) -> Candidate { + Candidate { + route, + first: None, + local: true, + } + } + + fn info() -> track::Info { + track::Info::default() + } + + fn name(s: &str) -> Arc { + Arc::from(s) + } + + /// Actions compare by their debug form: `Error` has no equality. + #[track_caller] + fn assert_actions(actual: Vec, expected: &[Action]) { + assert_eq!(format!("{actual:?}"), format!("{expected:?}")); + } + + /// A front serving `source` through `candidate`, with `video` spliced and read. + fn serving(candidate: Candidate, source: u64) -> Front { + let mut front = Front::new(LINGER); + assert_actions( + front.step(Event::Selected { + best: Some(candidate), + serving_closing: false, + }), + &[Action::Request { route: candidate.route }], + ); + front.identify(candidate); + assert_actions( + front.step(Event::Resolved { + route: candidate.route, + result: Ok(source), + }), + &[Action::Resolve], + ); + assert_actions(front.step(Event::TrackAssigned { track: name("video") }), &[]); + assert_actions( + front.step(Event::Used { track: name("video") }), + &[Action::Query { + track: name("video"), + source, + }], + ); + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source, + closing: false, + result: Ok(info()), + }), + &[Action::Splice { + track: name("video"), + source, + }], + ); + front + } + + #[test] + fn first_source_resolves_and_serves_read_tracks() { + let front = serving(remote(1, 10), 100); + assert_eq!(front.identity, Identity::Publisher(hop(10))); + assert_eq!(front.pin(), Pin::Publisher(hop(10))); + } + + #[test] + fn nothing_routable_ends_an_unresolved_front() { + let mut front = Front::new(LINGER); + assert_actions( + front.step(Event::Selected { + best: None, + serving_closing: false, + }), + &[Action::End { err: Error::Unroutable }], + ); + assert!(front.ended()); + assert_actions(front.step(Event::Closed), &[]); + } + + #[test] + fn unchanged_selection_is_a_no_op() { + let mut front = serving(remote(1, 10), 100); + assert_actions( + front.step(Event::Selected { + best: Some(remote(1, 10)), + serving_closing: false, + }), + &[], + ); + } + + #[test] + fn better_route_takes_over_and_resplices() { + let mut front = serving(remote(1, 10), 100); + assert_actions( + front.step(Event::Selected { + best: Some(remote(2, 10)), + serving_closing: false, + }), + &[Action::Request { route: 2 }], + ); + assert_actions( + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }), + &[ + Action::Detach { source: 100 }, + Action::Query { + track: name("video"), + source: 200, + }, + ], + ); + assert_eq!(front.serving, Some((200, 2))); + } + + #[test] + fn a_stale_resolution_is_let_go() { + let mut front = serving(remote(1, 10), 100); + front.step(Event::Selected { + best: Some(remote(2, 10)), + serving_closing: false, + }); + // The table moved back before the request landed. + assert_actions( + front.step(Event::Selected { + best: Some(remote(1, 10)), + serving_closing: false, + }), + &[], + ); + assert_actions( + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }), + &[Action::Detach { source: 200 }], + ); + } + + #[test] + fn dead_source_reselects_through_the_same_publisher() { + let mut front = serving(remote(1, 10), 100); + assert_actions( + front.step(Event::SourceClosed { source: 100 }), + &[Action::Detach { source: 100 }, Action::Reselect], + ); + assert_actions( + front.step(Event::Selected { + best: Some(remote(3, 10)), + serving_closing: false, + }), + &[Action::Request { route: 3 }], + ); + assert_actions( + front.step(Event::Resolved { + route: 3, + result: Ok(300), + }), + &[Action::Query { + track: name("video"), + source: 300, + }], + ); + } + + #[test] + fn dead_source_with_no_replacement_ends() { + let mut front = serving(remote(1, 10), 100); + front.step(Event::SourceClosed { source: 100 }); + assert_actions( + front.step(Event::Selected { + best: None, + serving_closing: false, + }), + &[Action::End { err: Error::Dropped }], + ); + } + + #[test] + fn anonymous_source_never_resumes() { + let candidate = Candidate { + route: 1, + first: Some(Hop::UNKNOWN), + local: false, + }; + let mut front = serving(candidate, 100); + assert_eq!(front.pin(), Pin::None); + assert_actions( + front.step(Event::SourceClosed { source: 100 }), + &[Action::Detach { source: 100 }, Action::End { err: Error::Dropped }], + ); + } + + #[test] + fn standing_refusal_ends_an_unresolved_front() { + let mut front = Front::new(LINGER); + front.step(Event::Selected { + best: Some(remote(1, 10)), + serving_closing: false, + }); + assert_actions( + front.step(Event::Resolved { + route: 1, + result: Err(Refusal { + err: Error::NotFound, + standing: true, + }), + }), + &[Action::End { err: Error::NotFound }], + ); + } + + #[test] + fn standing_refusal_while_serving_skips_the_refuser() { + let mut front = serving(remote(1, 10), 100); + front.step(Event::Selected { + best: Some(remote(2, 10)), + serving_closing: false, + }); + assert_actions( + front.step(Event::Resolved { + route: 2, + result: Err(Refusal { + err: Error::NotFound, + standing: true, + }), + }), + &[Action::Reselect], + ); + assert!(front.refused_routes().contains(&2)); + assert_eq!(front.serving, Some((100, 1))); + } + + #[test] + fn retracted_route_is_not_a_refusal() { + let mut front = Front::new(LINGER); + front.step(Event::Selected { + best: Some(remote(1, 10)), + serving_closing: false, + }); + assert_actions( + front.step(Event::Resolved { + route: 1, + result: Err(Refusal { + err: Error::Unroutable, + standing: false, + }), + }), + &[Action::Reselect], + ); + assert!(front.refused_routes().is_empty()); + assert!(!front.ended()); + } + + #[test] + fn a_source_refusing_a_track_aborts_it_and_nothing_else() { + let mut front = serving(remote(1, 10), 100); + front.step(Event::TrackAssigned { track: name("audio") }); + front.step(Event::Used { track: name("audio") }); + assert_actions( + front.step(Event::TrackInfo { + track: name("audio"), + source: 100, + closing: false, + result: Err(Error::NotFound), + }), + &[Action::Abort { + track: name("audio"), + err: Error::NotFound, + }], + ); + assert_eq!(front.tracks[&name("video")].state, TrackState::Spliced { source: 100 }); + } + + #[test] + fn a_closing_source_refusal_is_not_a_verdict() { + let mut front = serving(remote(1, 10), 100); + front.step(Event::TrackAssigned { track: name("audio") }); + front.step(Event::Used { track: name("audio") }); + assert_actions( + front.step(Event::TrackInfo { + track: name("audio"), + source: 100, + closing: true, + result: Err(Error::NotFound), + }), + &[], + ); + assert!(front.tracks[&name("audio")].refused.is_empty()); + // The replacement is asked afresh. + front.step(Event::SourceClosed { source: 100 }); + front.step(Event::Selected { + best: Some(remote(2, 10)), + serving_closing: false, + }); + let actions = front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + assert!( + actions + .iter() + .any(|action| matches!(action, Action::Query { track, source: 200 } if track.as_ref() == "audio")) + ); + } + + #[test] + fn a_copy_dying_after_delivering_resplices() { + let mut front = serving(remote(1, 10), 100); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(Error::Dropped), + delivered: true, + }), + &[Action::Query { + track: name("video"), + source: 100, + }], + ); + } + + #[test] + fn a_copy_dying_before_delivering_is_a_refusal() { + let mut front = serving(remote(1, 10), 100); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Err(Error::Dropped), + delivered: false, + }), + &[Action::Abort { + track: name("video"), + err: Error::Dropped, + }], + ); + } + + #[test] + fn a_completed_copy_finishes_the_track() { + let mut front = serving(remote(1, 10), 100); + assert_actions( + front.step(Event::TrackEnded { + track: name("video"), + source: 100, + closing: false, + result: Ok(()), + delivered: true, + }), + &[Action::Finish { track: name("video") }], + ); + } + + #[test] + fn incompatible_copy_is_refused() { + let mut front = serving(remote(1, 10), 100); + front.step(Event::Selected { + best: Some(remote(2, 10)), + serving_closing: false, + }); + front.step(Event::Resolved { + route: 2, + result: Ok(200), + }); + let other = track::Info { + max_age: Duration::from_secs(1), + ..track::Info::default() + }; + assert_actions( + front.step(Event::TrackInfo { + track: name("video"), + source: 200, + closing: false, + result: Ok(other), + }), + &[Action::Abort { + track: name("video"), + err: Error::Unsupported, + }], + ); + } + + #[test] + fn unread_track_parks_then_releases_after_the_linger() { + let mut front = serving(remote(1, 10), 100); + let t0 = Instant::now(); + assert_actions( + front.step(Event::Unused { + track: name("video"), + now: t0, + }), + &[ + Action::Park { track: name("video") }, + Action::Arm { at: Some(t0 + LINGER) }, + ], + ); + // Too early: nothing released, but the deadline is armed again since the + // driver clears it on firing. + assert_actions( + front.step(Event::Deadline { now: t0 + LINGER / 2 }), + &[Action::Arm { at: Some(t0 + LINGER) }], + ); + // The driver cleared the fired deadline itself; nothing is parked, so + // nothing re-arms. + assert_actions( + front.step(Event::Deadline { now: t0 + LINGER }), + &[Action::Release { track: name("video") }], + ); + // A returning reader re-splices from the serving source. + assert_actions( + front.step(Event::Used { track: name("video") }), + &[Action::Query { + track: name("video"), + source: 100, + }], + ); + } + + #[test] + fn a_returning_reader_cancels_the_linger() { + let mut front = serving(remote(1, 10), 100); + let t0 = Instant::now(); + front.step(Event::Unused { + track: name("video"), + now: t0, + }); + assert_actions( + front.step(Event::Used { track: name("video") }), + &[ + Action::Query { + track: name("video"), + source: 100, + }, + Action::Arm { at: None }, + ], + ); + } + + #[test] + fn unread_track_is_never_spliced() { + let mut front = serving(remote(1, 10), 100); + assert_actions(front.step(Event::TrackAssigned { track: name("audio") }), &[]); + assert_eq!(front.tracks[&name("audio")].state, TrackState::Idle); + } + + #[test] + fn local_newcomer_takes_over_a_live_incumbent() { + let mut front = serving(local(1), 100); + assert_eq!(front.identity, Identity::Local); + assert_eq!(front.pin(), Pin::Local); + assert_actions( + front.step(Event::Selected { + best: Some(local(2)), + serving_closing: false, + }), + &[Action::Request { route: 2 }], + ); + } + + #[test] + fn local_newcomer_never_splices_into_a_closing_incumbent() { + let mut front = serving(local(1), 100); + assert_actions( + front.step(Event::Selected { + best: Some(local(2)), + serving_closing: true, + }), + &[Action::Detach { source: 100 }, Action::End { err: Error::Dropped }], + ); + } + + #[test] + fn local_incumbent_ending_ends_the_front() { + let mut front = serving(local(1), 100); + assert_actions( + front.step(Event::SourceClosed { source: 100 }), + &[Action::Detach { source: 100 }, Action::End { err: Error::Dropped }], + ); + } + + #[test] + fn teardown_ends_everything() { + let mut front = serving(remote(1, 10), 100); + assert_actions( + front.step(Event::Closed), + &[Action::Detach { source: 100 }, Action::End { err: Error::Dropped }], + ); + } + + /// Every sequence of events up to a small depth, over a small alphabet, + /// checked against the invariants the driver relies on. + #[test] + fn bounded_sequences_hold_the_invariants() { + let t0 = Instant::now(); + let alphabet: Vec = vec![ + Event::Selected { + best: Some(remote(1, 10)), + serving_closing: false, + }, + Event::Selected { + best: Some(remote(2, 10)), + serving_closing: false, + }, + Event::Selected { + best: None, + serving_closing: false, + }, + Event::Resolved { + route: 1, + result: Ok(100), + }, + Event::Resolved { + route: 2, + result: Ok(200), + }, + Event::Resolved { + route: 2, + result: Err(Refusal { + err: Error::NotFound, + standing: true, + }), + }, + Event::SourceClosed { source: 100 }, + Event::TrackAssigned { track: name("v") }, + Event::Used { track: name("v") }, + Event::Unused { + track: name("v"), + now: t0, + }, + Event::TrackInfo { + track: name("v"), + source: 100, + closing: false, + result: Ok(info()), + }, + Event::TrackInfo { + track: name("v"), + source: 100, + closing: false, + result: Err(Error::NotFound), + }, + Event::TrackEnded { + track: name("v"), + source: 100, + closing: false, + result: Err(Error::Dropped), + delivered: false, + }, + Event::Deadline { now: t0 + LINGER }, + ]; + + fn walk(front: &Front, alphabet: &[Event], depth: usize, sequences: &mut usize) { + for event in alphabet { + // Ids are minted once per resolution, so a source can never resolve + // while it is already serving; the alphabet reuses ids for brevity. + if let Event::Resolved { result: Ok(source), .. } = event + && front.serving.map(|(serving, _)| serving) == Some(*source) + { + continue; + } + let mut next = front.clone(); + let before = format!("{next:?}"); + let actions = next.step(event.clone()); + *sequences += 1; + + // An event that changes nothing produces nothing, except letting + // go of a source the front never took. + if format!("{next:?}") == before { + assert!( + actions.iter().all(|action| matches!(action, Action::Detach { .. })), + "no-op event {event:?} produced {actions:?}" + ); + } + // Nothing after the end. + if front.ended() { + assert!(actions.is_empty()); + } + // A source is only ever asked for a track it has not refused. + for action in &actions { + if let Action::Query { track, source } = action { + assert!(!next.tracks[track].refused.contains(source)); + } + } + // At most one takeover per track per event. + let splices = actions + .iter() + .filter(|action| matches!(action, Action::Splice { .. })) + .count(); + assert!(splices <= 1); + // A Detach names a source the front no longer serves from. + for action in &actions { + if let Action::Detach { source } = action { + assert_ne!(next.serving.map(|(s, _)| s), Some(*source)); + } + } + if depth > 1 { + walk(&next, alphabet, depth - 1, sequences); + } + } + } + + let mut sequences = 0; + walk(&Front::new(LINGER), &alphabet, 5, &mut sequences); + assert!(sequences > 100_000); + } +} diff --git a/rs/moq-net/src/model/mod.rs b/rs/moq-net/src/model/mod.rs index 9dede1e34a0..3b259d15408 100644 --- a/rs/moq-net/src/model/mod.rs +++ b/rs/moq-net/src/model/mod.rs @@ -5,11 +5,14 @@ pub mod frame; pub mod group; pub mod track; -// The origin + announce subsystem shares one implementation (a broadcast tree). +// The origin + announce subsystem shares one implementation (a route table). // It stays in a single private module and is surfaced as two curated public // modules so neither leaks the other's plumbing. #[path = "origin.rs"] mod origin_impl; +// The failover state machine origin fronts run; pure, so its transitions are +// tested without a runtime. +mod front; mod bytes; pub(crate) mod clock; diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 35cbfe2e7ff..d90a96040a9 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -2,7 +2,7 @@ use crate::{broadcast, cache, stats, track}; use kio::Pollable; use std::{ cmp::Reverse, - collections::{BTreeMap, BTreeSet, HashMap, HashSet}, + collections::{BTreeMap, BTreeSet, HashMap, HashSet, VecDeque}, fmt, sync::Arc, sync::atomic::{AtomicU64, Ordering}, @@ -10,16 +10,18 @@ use std::{ time::Duration, }; -use kio::Lock; use rand::RngExt; -use super::{Requests, WeakCache, WeakEntry}; +use super::{ + Requests, WeakCache, WeakEntry, + front::{Action, Candidate, Event, Front, Pin, Refusal}, +}; use crate::{ AsPath, Error, InvalidPattern, Path, PathOwned, Pattern, Patterns, coding::{BoundsExceeded, Decode, DecodeError, Encode, EncodeError}, runtime::{Instant, Timers}, time::Clock, - util::{TaskSet, Tasks, TasksWeak}, + util::{Keepalive, TaskSet, Tasks, TasksWeak}, }; /// One relay's identity in a broadcast's hop chain: a 62-bit varint on the wire. @@ -523,18 +525,6 @@ impl ConsumerId { } } -// The origin-owned broadcast at a leaf: the spliced broadcast consumers see and -// the table of local sources feeding it. Local broadcasts are reachable by exact -// path; whether anything is *advertised* is a separate concern, owned by the -// route table (see [`Producer::announce`]). -struct OriginBroadcast { - /// The shared, spliced broadcast; its `consume()` is what consumers get. - broadcast: broadcast::Producer, - /// The source table, shared with every source watcher and the front task. - /// Also the broadcast's identity for stale-teardown checks. - state: kio::Producer, -} - /// FNV-1a over a path and a sequence of origin ids. /// /// FNV-1a, not the std hasher: its output is fixed across Rust versions and @@ -698,9 +688,15 @@ struct RouteEntry { local: bool, /// The queue requests under this route are served from, when the announcer /// serves content on demand (a [`Dynamic`]). `None` for an advertise-only - /// announcement (a broadcast's exact path, or [`Producer::announce`]), whose - /// covered paths resolve only through the tree. + /// announcement ([`Producer::announce`]) and for a local broadcast. server: Option>, + /// The broadcast published on this origin at exactly `prefix`, when the + /// entry is one: requests resolve to it directly, and the newest one at a + /// path wins through [`route_order`]. + source: Option, + /// Whether announce cursors see the entry. A local broadcast is servable + /// from the moment it is created but advertised only once it announces. + advertised: bool, /// [`prefix_claim`] of [`Self::prefix`], built once at announce time. /// /// The announce sync evaluates a route's claim once per (cursor, route) pair, @@ -714,6 +710,23 @@ impl RouteEntry { self.hops.iter().any(|hop| *hop == Hop::UNKNOWN) } + /// Whether a request for `path` can be served through this entry. A served + /// route covers everything beneath its prefix; a broadcast published here + /// is only itself, so it serves its exact path and shadows what is beneath. + fn serves(&self, path: &Path) -> bool { + self.server.is_some() || (self.source.is_some() && self.prefix == *path) + } + + /// Whether `pin` admits this entry for a front's selection. + fn qualifies(&self, pin: Pin) -> bool { + match pin { + Pin::Any => true, + Pin::Local => self.local, + Pin::Publisher(first) => self.hops.iter().next() == Some(&first), + Pin::None => false, + } + } + /// Whether this entry may be observed or served to a requester excluding `peer`. /// /// A non-zero peer is hidden when it is the announcing session (`via`) or @@ -853,117 +866,24 @@ impl TableCursor { .map(|(_, captures)| captures) } - /// Whether this cursor may observe `entry` at all (split horizon). + /// Whether this cursor may observe `entry` at all: advertised, not behind + /// the excluded peer (split horizon), and within the cursor's patterns. fn visible(&self, entry: &RouteEntry) -> bool { - entry.visible_to(self.exclude) && entry.overlaps(&self.allowed) - } -} - -struct OriginNode { - // The origin-owned broadcast published at this node, if any (see - // [`Producer::create_broadcast`]). - broadcast: Option, - - // Nested nodes, one level down the tree. - nested: HashMap>, -} - -impl OriginNode { - fn new() -> Self { - Self { - broadcast: None, - nested: HashMap::new(), - } - } - - fn leaf(&mut self, path: &Path) -> Lock { - let (dir, rest) = path.next_part().expect("leaf called with empty path"); - - let next = self.entry(dir); - if rest.is_empty() { next } else { next.lock().leaf(&rest) } - } - - fn entry(&mut self, dir: &str) -> Lock { - match self.nested.get(dir) { - Some(next) => next.clone(), - None => { - let next = Lock::new(OriginNode::new()); - self.nested.insert(dir.to_string(), next.clone()); - next - } - } - } - - fn resolve_broadcast(&self, rest: impl AsPath) -> Option { - let rest = rest.as_path(); - - if let Some((dir, rest)) = rest.next_part() { - let node = self.nested.get(dir)?; - let node = node.lock(); - return node.resolve_broadcast(&rest); - } - - Some(self.broadcast.as_ref()?.broadcast.consume()) - } - - /// Remove the broadcast at `relative` if it is `expect`, pruning empty nodes on - /// the way back up. The identity check keeps a stale teardown from clobbering a - /// replacement. - fn remove(&mut self, expect: &kio::Producer, relative: impl AsPath) { - let relative = relative.as_path(); - - if let Some((dir, relative)) = relative.next_part() { - let Some(nested) = self.nested.get(dir) else { return }; - let nested = nested.clone(); - let mut locked = nested.lock(); - locked.remove(expect, &relative); - - if locked.is_empty() { - drop(locked); - self.nested.remove(dir); - } - } else if let Some(existing) = &self.broadcast - && existing.state.same_channel(expect) - { - self.broadcast = None; - } - } - - fn is_empty(&self) -> bool { - self.broadcast.is_none() && self.nested.is_empty() - } - - /// Nodes in this subtree, counting self. Test-only: pruning is invisible - /// through the public surface, so the tests assert on the tree's size. - #[cfg(test)] - fn count(&self) -> usize { - 1 + self.nested.values().map(|nested| nested.lock().count()).sum::() + entry.advertised && entry.visible_to(self.exclude) && entry.overlaps(&self.allowed) } } -/// A handle's view of an origin's path tree: the tree itself plus the absolute -/// patterns the handle may reach in it. -/// -/// Paths, because pruning removes empty nodes: a pinned `Lock` outlives -/// the prune as an orphan that publishes and resolves where no lookup can reach. -/// Only [`Self::tree`] is stable, so every operation resolves against it and node -/// identity lives in exactly one place. Scoping a handle therefore creates nothing -/// in the tree; only a broadcast does. +/// A handle's view of an origin: the absolute patterns it may reach. #[derive(Clone)] struct OriginScope { - // The tree root, shared by every handle derived from one origin. Never pruned: - // it hangs off no parent. - tree: Lock, - - // The paths this handle may reach, absolute under `tree`. + // The paths this handle may reach, absolute. allowed: Patterns, } impl OriginScope { - /// A view over a fresh tree that reaches nothing. + /// A view that reaches nothing. fn empty() -> Self { Self { - tree: Lock::new(OriginNode::new()), allowed: Patterns::new(), } } @@ -974,10 +894,7 @@ impl OriginScope { if allowed.is_empty() { None } else { - Some(Self { - tree: self.tree.clone(), - allowed, - }) + Some(Self { allowed }) } } @@ -995,7 +912,6 @@ impl OriginScope { impl Default for OriginScope { fn default() -> Self { Self { - tree: Lock::new(OriginNode::new()), allowed: Patterns::from(Pattern::all()), } } @@ -1061,7 +977,7 @@ pub struct Producer { // downstream relays can detect loops and prefer the shortest path. hop: Hop, - // The tree and the absolute patterns this handle may publish under. + // The absolute patterns this handle may publish under. scope: OriginScope, // The prefix that is automatically stripped from all paths. @@ -1125,7 +1041,6 @@ impl Producer { let driver = Driver { state: DriverState { set, - tree: scope.tree, shared, done: false, }, @@ -1221,27 +1136,13 @@ impl Producer { pub fn create_broadcast(&self, path: impl AsPath) -> Result { let path = path.as_path(); - // Held across the whole attach: the driver's teardown sets `closed` under - // this lock, so a create either completes before the teardown (whose walk - // then cleans the entry up) or observes `closed` here and fails. - let lifecycle = self.shared.lock(); - if lifecycle.closed { - return Err(Error::Closed); - } - - // `get` resolves the path against the tree root, which is the same absolute - // path the handle's own root produces: an allowed prefix is always stored - // alongside its absolute position. So one path serves both the front's - // identity and its position in the tree. let full = self.root.join(&path).to_owned(); if !self.scope.permits(&full) { return Err(Error::Unauthorized); } - let tree = self.scope.tree.clone(); - // A decoded prefix and suffix are each within the wire limit, but their - // join might not be. Enforcing here bounds the tree depth and guarantees the path - // can be re-encoded when forwarded. + // join might not be. Enforcing here bounds the table depth and guarantees the + // path can be re-encoded when forwarded. if full.parts().count() > Path::MAX_PARTS { return Err(BoundsExceeded.into()); } @@ -1252,62 +1153,37 @@ impl Producer { // Resolve the ingress counters once, keyed by the absolute broadcast path. let ingress = self.stats.ingress(&full); - // The broadcast advertises its own exact path, already checked against the scope. - let announcer = Announcer { - announcing: Announcing { - hop: self.hop, - shared: self.shared.clone(), - requested: full.clone(), - prefixes: vec![(full.clone(), claim)], - scope: self.scope.allowed.clone(), - local: true, - stats: self.stats.clone(), - }, - current: None, + // The broadcast is a route table entry at its exact path from the start, + // so requests resolve to it and the newest publisher at a path wins; + // cursors see it only once it announces. The entry lives as long as the + // broadcast: its announcer drops on finish, abort, or the last handle. + let announcing = Announcing { + hop: self.hop, + shared: self.shared.clone(), + requested: full.clone(), + prefixes: vec![(full.clone(), claim)], + scope: self.scope.allowed.clone(), + local: true, + stats: self.stats.clone(), }; - - let source = broadcast::Info { + let info = broadcast::Info { pool: self.pool.clone(), cache_duration: self.cache_duration, - path: full.clone(), - } - .produce() - .with_stats(ingress.clone()) - .with_announcer(announcer); - let consumer = source.consume(); - - // Attach synchronously: the source is visible to exact lookups before this - // returns; only lifecycle work needs the driver. - let origin = self.config(); - let ctx = AttachContext { - origin: &origin, - tree: &tree, - full: &full, - tasks: &self.tasks, - timers: &self.timers, - }; - // The root is its own leaf and is never pruned. - let leaf = if full.is_empty() { - tree.clone() - } else { - tree.lock().leaf(&full) + path: full, }; - let (state, broadcast, id) = attach_source(&ctx, &leaf, &consumer); - - self.tasks.push(run_source(SourceTask { - source: consumer, - timers: self.timers.clone(), - leaf, - state, - broadcast, - id, - })); - // A local broadcast changes what the exact path resolves to, so a requester - // parked on it (see `routed_broadcast`) retries. - lifecycle.routes.poke_at(&full); - drop(lifecycle); - - Ok(source) + let source = info.produce().with_stats(ingress); + let entry = announcing.announce( + Route::default(), + Serving { + server: None, + source: Some(source.consume()), + advertised: false, + }, + )?; + Ok(source.with_announcer(Announcer { + entry, + _keepalive: self.tasks.keepalive(), + })) } /// Create and advertise a broadcast in one call. @@ -1319,9 +1195,9 @@ impl Producer { /// Mint a standalone source broadcast for a served-route request: it carries /// this origin's cache policy and ingress attribution, but - /// is *not* inserted into the broadcast tree. Sessions answer + /// is *not* entered into the route table. Sessions answer /// [`Dynamic`] requests with one of these; the requester already holds - /// the request's result channel, so the tree never needs to resolve it. + /// the request's result channel, so the table never needs to resolve it. pub(crate) fn create_source(&self, path: impl AsPath) -> broadcast::Producer { let path = path.as_path(); let full = self.root.join(&path).to_owned(); @@ -1345,7 +1221,14 @@ impl Producer { /// [`Dynamic`] handler, which serve what they claim. #[cfg(test)] pub(crate) fn announce(&self, prefix: impl AsPath, route: Route) -> Result { - Announcing::new(self, prefix)?.announce(route, None) + Announcing::new(self, prefix)?.announce( + route, + Serving { + server: None, + source: None, + advertised: true, + }, + ) } /// Advertise a route over `prefix` and serve the requests beneath it. @@ -1373,7 +1256,14 @@ impl Producer { let announcing = Announcing::new(self, prefix)?; let serve = kio::Shared::::default(); serve.lock().requests.add_handler(); - let announcement = announcing.announce(route, Some(serve.clone()))?; + let announcement = announcing.announce( + route, + Serving { + server: Some(serve.clone()), + source: None, + advertised: true, + }, + )?; Ok(Dynamic { announcement, state: serve, @@ -1404,7 +1294,7 @@ impl Producer { }) } - /// Cheap read handle over this origin's broadcast tree. + /// Cheap read handle over this origin's route table. /// /// Use [`Consumer::announced`] to register interest and start receiving /// announcement events; the consumer itself does not allocate any channels. @@ -1428,12 +1318,6 @@ impl Producer { pub fn absolute(&self, path: impl AsPath) -> Path<'_> { self.root.join(path) } - - /// Nodes in the whole path tree, counting the root. Test-only. - #[cfg(test)] - pub(crate) fn node_count(&self) -> usize { - self.scope.tree.lock().count() - } } /// What it takes to insert a route: the prefixes it covers and the origin table @@ -1476,7 +1360,7 @@ impl Announcing { }) } - fn announce(&self, route: Route, server: Option>) -> Result { + fn announce(&self, route: Route, serving: Serving) -> Result { debug_assert!( !route.hops.contains(&self.hop), "announce called with a looping hop chain", @@ -1502,7 +1386,9 @@ impl Announcing { cost: meta.1, via, local: self.local, - server: server.clone(), + server: serving.server.clone(), + source: serving.source.clone(), + advertised: serving.advertised, claim: claim.clone(), }); shared.sync_route(prefix, claim); @@ -1521,32 +1407,37 @@ impl Announcing { } } -/// The advertisement a broadcast owns: its exact path, announced and retracted -/// through [`broadcast::Producer::announce`] and [`broadcast::Producer::unannounce`], -/// and retracted for good when the broadcast ends. +/// What a route entry serves and whether cursors see it. +struct Serving { + server: Option>, + source: Option, + advertised: bool, +} + +/// The table entry a broadcast owns: its exact path, advertised and withdrawn +/// through [`broadcast::Producer::announce`] and +/// [`broadcast::Producer::unannounce`], and removed when the broadcast ends. /// /// Handed to the broadcast by [`Producer::create_broadcast`], so a standalone /// broadcast has none and cannot announce. pub(crate) struct Announcer { - announcing: Announcing, - current: Option, + entry: AnnounceProducer, + /// A published broadcast is lifecycle work: the origin's driver keeps + /// running for as long as one lives, even once every producer handle is + /// gone, so a session handed a producer can drop it and keep serving. + _keepalive: Keepalive, } impl Announcer { /// Advertise the broadcast's path with `route`, or re-price the standing /// advertisement in place. pub(crate) fn announce(&mut self, route: Route) -> Result<(), Error> { - if let Some(current) = &self.current { - return current.update(route); - } - self.current = Some(self.announcing.announce(route, None)?); - Ok(()) + self.entry.update(route) } - /// Take the standing advertisement, if any, so the caller retracts it by - /// dropping it outside whatever lock guards this announcer. - pub(crate) fn take(&mut self) -> Option { - self.current.take() + /// Withdraw the advertisement; the broadcast stays servable. + pub(crate) fn withdraw(&mut self) { + self.entry.withdraw(); } } @@ -1582,17 +1473,35 @@ impl AnnounceProducer { for (prefix, id) in &self.entries { // Each entry keeps its advertised prefix; only the metadata moves. let Some(entry) = shared.routes.entry_mut(prefix, *id) else { - continue; + return Err(Error::Closed); }; entry.hops = route.hops.clone(); entry.cost = route.cost; entry.via = route.via; + entry.advertised = true; let claim = entry.claim.clone(); shared.sync_route(prefix, &claim); } Ok(()) } + /// Hide the entries from announce cursors; requests still resolve through + /// them. What [`broadcast::Producer::unannounce`] does. + fn withdraw(&self) { + let mut shared = self.shared.lock(); + for (prefix, id) in &self.entries { + let Some(entry) = shared.routes.entry_mut(prefix, *id) else { + continue; + }; + if !entry.advertised { + continue; + } + entry.advertised = false; + let claim = entry.claim.clone(); + shared.sync_route(prefix, &claim); + } + } + /// Retract the route now: remove its table entries and reject anything still /// waiting on its queue. Idempotent, and what dropping the advertisement does. fn retract(&self) { @@ -1646,10 +1555,8 @@ pub struct Driver { /// Lifecycle work and the state it tears down. struct DriverState { - /// Source watchers, fronts, and serve tasks: producers submit, this polls. + /// The front drivers: producers submit, this polls. set: TaskSet, - /// The whole broadcast tree, for the teardown walk on drop. - tree: Lock, /// The route table, announce cursors, and the remotely-served fronts, for /// ending everything on drop. shared: kio::Shared, @@ -1696,8 +1603,8 @@ impl DriverState { /// front, retract every route, end announcement cursors, and reject pending /// requests. fn teardown(&mut self) { - // Cancel queued and running lifecycle work first, so nothing re-attaches - // or serves while the walks below empty the tree. + // Cancel queued and running lifecycle work first, so no front serves + // while the table is ended below. drop(std::mem::replace(&mut self.set, TaskSet::owned())); // Refuse new work and take the pending requests, under the same lock @@ -1746,8 +1653,6 @@ impl DriverState { state.ended = true; } } - - teardown_broadcasts(&self.tree); } } @@ -1757,28 +1662,6 @@ impl Drop for DriverState { } } -/// Abort and unpublish every broadcast under `node` with [`Error::Dropped`]. The -/// lifecycle tasks are already cancelled, so this finishes the teardown they -/// would have run. -fn teardown_broadcasts(node: &Lock) { - let (entry, children) = { - let mut guard = node.lock(); - let children: Vec<_> = guard.nested.values().cloned().collect(); - (guard.broadcast.take(), children) - }; - if let Some(entry) = entry { - // Close the front so anything still holding its table observes the end. - if let Ok(mut state) = entry.state.write() { - state.closed = true; - } - entry.broadcast.abort_spliced(Error::Dropped); - entry.broadcast.finish(); - } - for child in children { - teardown_broadcasts(&child); - } -} - /// How long a spliced track stays warm after its last reader leaves. /// /// Within the window a returning viewer, or the next of a run of back-to-back @@ -1793,312 +1676,6 @@ fn teardown_broadcasts(node: &Lock) { /// waiting longer costs cached state, not a viewer. const TRACK_IDLE_LINGER: Duration = Duration::from_secs(30); -/// One attached source in a [`FrontState`] table. -struct FrontSource { - id: u64, - /// The source broadcast tracks are served from. - source: broadcast::Consumer, -} - -/// Shared state behind a front: the attached local sources and which one is -/// active. -struct FrontState { - /// Attach counter, handed to each [`FrontSource`] so selection can break a - /// tie toward the newest source. - next_source: u64, - sources: Vec, - /// Immutable track metadata, retained across idle release and aborted attempts. - /// Every source of this broadcast must serve the same content. - track_info: HashMap, track::Info>, - /// The source tracks are dispatched to: the newest attached. Backups park - /// until promoted. - active: Option, - /// Terminal: no more sources may attach and every poller stops. Set - /// synchronously by the detach that empties the table. - closed: bool, -} - -impl FrontState { - /// Admit only copies with the broadcast's established track properties. - fn accept_track_info(&mut self, name: &Arc, info: track::Info) -> Result<(), Error> { - if self.closed { - return Err(Error::Closed); - } - if let Some(expected) = self.track_info.get(name) { - let track::Info { - timescale, - max_age, - priority, - } = info; - if timescale != expected.timescale || max_age != expected.max_age || priority != expected.priority { - return Err(Error::Unsupported); - } - } else { - self.track_info.insert(name.clone(), info); - } - Ok(()) - } - - /// The newest attached source: the one new work dispatches to. Local sources - /// carry no route metadata, so recency is the whole order: a publisher - /// re-creating a path over a fresh handle wins the moment it attaches instead - /// of waiting for the old handle to be torn down. - fn best_source(&self) -> Option { - self.sources.iter().map(|s| s.id).max() - } - - /// The source one track should be served from: the front's active source - /// unless `skip` rules it out, then the newest source that survives. - /// - /// Whether a source carries a given track is a per-track property (a standby - /// that has not created it yet, a publisher whose encoder is still starting), - /// so a source refusing one track is ruled out of that track only, never out - /// of the front. - fn serve_route(&self, skip: impl Fn(u64) -> bool) -> Option { - if let Some(active) = self.active - && !skip(active) - && self.sources.iter().any(|s| s.id == active) - { - return Some(active); - } - self.sources.iter().map(|s| s.id).filter(|id| !skip(*id)).max() - } - - /// Re-pick the active source after the table changed. Serve tasks watch - /// `active` and re-splice on their own, so a replacement takes over seamlessly - /// at a group boundary. - fn reselect(&mut self) { - self.active = self.best_source(); - } -} - -/// Detach source `id`, promoting the newest remaining source; the tracks it was -/// serving re-splice on their own. Idempotent. -/// -/// Detaching the last source closes the broadcast synchronously, however the -/// source ended, which guarantees a following create at the path is a *new* -/// broadcast rather than splicing new content into this one. -fn detach_source(state: &kio::Producer, broadcast: &broadcast::Producer, leaf: &Lock, id: u64) { - let close = { - let Ok(mut s) = state.write() else { return }; - let Some(pos) = s.sources.iter().position(|entry| entry.id == id) else { - return; - }; - s.sources.remove(pos); - s.reselect(); - if s.sources.is_empty() && !s.closed { - // Last one out: close now. The front task observes `closed` and - // finishes the teardown (unpublish). - s.closed = true; - true - } else { - false - } - }; - if close { - broadcast.abort_spliced(Error::Dropped); - } - // The tree is pruned by the front task once it observes the close. - let _ = leaf; -} - -/// Everything a queued source watcher continues with after -/// [`Producer::create_broadcast`] performed the synchronous attach. -struct SourceTask { - /// The source broadcast, watched for its end. - source: broadcast::Consumer, - timers: Clock, - /// The leaf the attach landed on. - leaf: Lock, - /// The front's source table. - state: kio::Producer, - /// The spliced broadcast the front serves. - broadcast: broadcast::Producer, - /// The source's id in the table. - id: u64, -} - -/// Owns one source's lifecycle after its synchronous attach: waits for the -/// source to end (finish, abort, or drop), then detaches it. Queued on the -/// origin's [`Driver`] by [`Producer::create_broadcast`]. -async fn run_source(task: SourceTask) { - let SourceTask { - source, - timers, - leaf, - state, - broadcast, - id, - } = task; - let _ = timers; - - kio::wait(|waiter| source.poll_closed(waiter).map(Ok::<(), Error>)) - .await - .ok(); - - // The source ended, deliberately or not: detach it. If it was the last one - // the front closes with it. - detach_source(&state, &broadcast, &leaf, id); -} - -/// Everything about a source's attach that does not change between attempts. -struct AttachContext<'a> { - origin: &'a Config, - /// The origin's tree root: the one node pruning never removes, so the leaf is - /// resolved from here rather than pinned by a handle that a prune can orphan. - tree: &'a Lock, - /// Absolute path: the front's identity, its log lines, and its position under - /// `tree` are all the same path. - full: &'a PathOwned, - /// Driver submission handle, for queueing a fresh front's task. - tasks: &'a Tasks, - /// The driver's clock, threaded into fronts for the track idle linger. - timers: &'a Clock, -} - -/// Attach a source to the broadcast at `leaf`, creating (and publishing) the -/// broadcast if none is live. One lock acquisition covers the whole -/// join-or-create decision, so concurrent attaches cannot race each other. -/// -/// A later source joins the live front and immediately becomes the active one -/// (newest wins), so a publisher re-creating a path over a fresh handle takes -/// over without waiting for the old handle to be torn down; tracks re-splice at -/// the first missing group. A front whose sources have all closed is replaced by -/// a fresh broadcast instead, so new content is never spliced into subscribers -/// of a broadcast that is over. -fn attach_source( - ctx: &AttachContext, - leaf: &Lock, - source: &broadcast::Consumer, -) -> (kio::Producer, broadcast::Producer, u64) { - let mut leaf_guard = leaf.lock(); - - // Join the live broadcast if the leaf already has one. A closed one (torn - // down, awaiting teardown, or evicted just below) is replaced instead. - if let Some(existing) = &leaf_guard.broadcast { - let mut joined = None; - if let Ok(mut s) = existing.state.write() - && !s.closed - { - if !s.sources.is_empty() && s.sources.iter().all(|entry| entry.source.is_closing()) { - // Every attached source has already closed; only the driver's - // detach sweep is outstanding. Splicing requires overlapping - // *live* sources, so joining now would splice new content into - // subscribers of a broadcast that is over. Close the front and - // create a fresh one below; its own task finishes the teardown, - // finding the leaf slot already taken. - s.closed = true; - } else { - let id = s.next_source; - s.next_source += 1; - s.sources.push(FrontSource { - id, - source: source.clone(), - }); - s.reselect(); - joined = Some(id); - } - } - if let Some(id) = joined { - let state = existing.state.clone(); - let broadcast = existing.broadcast.clone(); - return (state, broadcast, id); - } - } - - // First source: create the broadcast and publish it into the tree. - let broadcast = broadcast::Producer::new_spliced(broadcast::Info { - pool: ctx.origin.pool.clone(), - cache_duration: ctx.origin.cache_duration, - path: ctx.full.clone(), - }); - let state = kio::Producer::new(FrontState { - next_source: 1, - sources: vec![FrontSource { - id: 0, - source: source.clone(), - }], - track_info: HashMap::new(), - active: Some(0), - closed: false, - }); - - // A stale (closed) entry is replaced; its own teardown task then finds the - // slot already taken and leaves it alone. - leaf_guard.broadcast = Some(OriginBroadcast { - broadcast: broadcast.clone(), - state: state.clone(), - }); - drop(leaf_guard); - - ctx.tasks.push(run_front( - state.clone(), - broadcast.clone(), - ctx.tree.clone(), - ctx.full.clone(), - ctx.tasks.clone(), - ctx.timers.clone(), - )); - - (state, broadcast, 0) -} - -/// Owns a front's lifecycle: dispatches each requested track to a serve task -/// until the last source detaches, then unpublishes the broadcast. -async fn run_front( - state: kio::Producer, - broadcast: broadcast::Producer, - tree: Lock, - full: PathOwned, - tasks: Tasks, - slot: Clock, -) { - enum Step { - Serve(Arc, super::resume::Producer), - Closed, - } - - loop { - let step = { - kio::wait(|waiter| { - if let Poll::Ready((name, resume)) = broadcast.poll_spliced_assigned(waiter) { - return Poll::Ready(Step::Serve(name, resume)); - } - // The close is set synchronously by the detach that empties the - // table; this task only finishes the teardown. `Err` is the - // channel itself dying, which also ends the front. - match state.poll_ref(waiter, |s| match s.closed { - true => Poll::Ready(()), - false => Poll::Pending, - }) { - Poll::Ready(_) => Poll::Ready(Step::Closed), - Poll::Pending => Poll::Pending, - } - }) - .await - }; - - match step { - Step::Serve(name, resume) => { - // Serve tasks self-terminate when the track completes or the - // front closes. - tasks.push(serve_track(state.clone(), name, resume, slot.clone())); - } - Step::Closed => break, - } - } - - // Abort the logical tracks (releasing their subscribers) and unpublish. - broadcast.abort_spliced(Error::Dropped); - - // Deliberate end; suppresses the dropped-without-finish warning. - broadcast.finish(); - - // Remove the broadcast from the tree (identity-checked, so a replacement is - // untouched) and prune empty nodes. - tree.lock().remove(&state, &full); -} - /// A local copy of groups the front already delivered, so resume stays spliced /// after the source track is dropped. Cache misses stay pending while demand /// re-splices the upstream source. Finished on drop so an idle linger does not @@ -2128,390 +1705,10 @@ fn warm_copy(source: &track::Consumer) -> Option { }) } -/// Serves one spliced logical track: splices in the best source's copy of the -/// track, re-splicing on handover or failure, until the track completes or the -/// front closes. A refusal (a source rejecting the track, returning incompatible -/// metadata, or dying before delivering anything) is authoritative and never -/// retried: the refuser is skipped for this track so a joining standby cannot kill a subscription -/// the incumbent is serving, and once every attached source has refused, the -/// track aborts with the last refusal's error. The verdict belongs to this -/// request; a later consumer request asks afresh (see `track_inner`). Failures -/// after delivered progress (a serving session dying mid-stream) are normal -/// failover and re-splice from the next source at the first missing group; a -/// source that fails while *closing* is a corpse to fail over past (its watcher -/// is about to detach it), not a verdict on the track. -async fn serve_track( - state: kio::Producer, - name: Arc, - mut resume: super::resume::Producer, - slot: Clock, -) { - enum Step { - Closed, - Splice(u64, broadcast::Consumer), - Complete, - Failed(Error), - /// The route we were serving from left the table with nothing servable to - /// replace it: drop our handle; the verdict block at the top of the loop - /// decides between aborting (all refused) and parking (corpses detaching). - NoRoute, - /// The linger expired with the track still unread: release the segment. - Idle, - /// A reader arrived or the last one left: recompute the demand gate. - Demand, - } - - // The source whose copy is currently spliced in, and that copy. - let mut serving: Option<(u64, track::Consumer)> = None; - // Local cache of groups the front already delivered, held after the source - // copy is dropped so resume stays spliced for the linger without pinning - // the source as a reader. Cache misses wait for demand to re-splice the - // source, and the copy is finished on drop so an idle linger is quiet. - let mut warm: Option = None; - // The delivered edge when that copy spliced in. A copy that dies without - // advancing it never delivered anything, which is what [`Step::Failed`] - // uses to tell a refusal from a mid-stream failover. Snapshotted per splice, - // not per wake: an unrelated wake between the copy's last frame and its - // death must not launder its delivered progress away. - let mut spliced_edge: Option = None; - // Sources that refused this track, and the most recent refusal's error. A - // standby joining a live front wins dispatch the moment it attaches, which is - // before a real publisher has created every track, so its refusal must cost - // the incumbent nothing: we keep serving from a route that has the track. - let mut refused: HashSet = HashSet::new(); - let mut refusal: Option = None; - // Sources whose splice failed because they had already closed. Their watchers - // are about to detach them (closing the front if nothing else remains), so - // wait for the table to move on rather than treating a corpse's error as a - // refusal (ids are never reused, so this cannot wedge). - let mut dead: HashSet = HashSet::new(); - // When the spliced segment stopped being read, starting the release countdown. - let mut idle_since: Option = None; - // Only the driver polls this body, after advancing the shared clock. - let timers = slot.clone(); - let mut deadline = crate::runtime::Deadline::new(&timers); - - loop { - let serving_id = serving.as_ref().map(|(id, _)| *id); - - // The table's verdict: once every attached source has refused, nothing - // will ever serve the track (refusals are never retried) and it aborts - // with the last refusal's error. A detached refuser leaves the set, so a - // source that reattaches (under a fresh id) is asked anew; a table blocked - // only by corpses awaiting detach parks instead, since their replacement - // (a reconnect) deserves the seamless splice. - { - let s = state.read(); - refused.retain(|id| s.sources.iter().any(|r| r.id == *id)); - dead.retain(|id| s.sources.iter().any(|r| r.id == *id)); - let exhausted = !s.sources.is_empty() - && s.serve_route(|id| refused.contains(&id) || dead.contains(&id)) - .is_none(); - if exhausted && dead.is_empty() { - drop(s); - let err = refusal.take().unwrap_or(Error::NotFound); - tracing::debug!(name = %name, %err, "every source refused track; aborting"); - let _ = resume.abort(err); - return; - } - } - - // Demand gates both directions: an unread track never splices a source in, - // and a spliced one is released once the idle window expires. Both sides use - // the same signal, so a release can't immediately re-splice and spin. - // - // The source copy is dropped on the unused edge, not at the linger: holding - // it would count the front as a reader for 30s after the real one left. - // The countdown keys off the segment, not our handle on the route that - // produced it: a route that leaves (or a copy that dies) drops the handle - // while the segment stays spliced, and that segment is exactly what the - // release exists to reclaim. Keying off the handle strands it until the front - // closes, pinning the departed source's cached groups and leaving a dead - // segment's edge behind for the next takeover to splice above. - let used = resume.is_used(); - idle_since = match (resume.is_spliced(), used) { - (true, false) => idle_since.or_else(|| Some(timers.now())), - _ => None, - }; - deadline.set(idle_since.and_then(|at| at.checked_add(TRACK_IDLE_LINGER))); - - let step = { - let skip = |id: u64| refused.contains(&id) || dead.contains(&id); - kio::wait(|waiter| { - // Watch the source table: the front closing, a better servable - // source than the one spliced in (skipping any we already know - // can't serve this track), or the served route leaving the table, - // which retires the refusals collected against it. Splicing waits - // for a reader. - match state.poll(waiter, |s| { - let gone = serving_id.is_some_and(|id| !s.sources.iter().any(|r| r.id == id)); - if s.closed - || (used && (gone || matches!(s.serve_route(skip), Some(next) if Some(next) != serving_id))) - { - Poll::Ready(()) - } else { - Poll::Pending - } - }) { - Poll::Ready(Ok(guard)) => { - if guard.closed { - return Poll::Ready(Step::Closed); - } - let Some(next) = guard.serve_route(skip) else { - return Poll::Ready(Step::NoRoute); - }; - let source = guard - .sources - .iter() - .find(|r| r.id == next) - .expect("servable source in table") - .source - .clone(); - return Poll::Ready(Step::Splice(next, source)); - } - Poll::Ready(Err(_)) => return Poll::Ready(Step::Closed), - Poll::Pending => {} - } - - // Watch the demand edge in whichever direction is unmet. This has to end - // the wait, not just wake it: `used` and the countdown are computed by - // the outer loop, so a wake that stayed inside would re-poll with the - // stale value and never arm (or cancel) the linger. - let edge = match used { - true => resume.poll_unused(waiter), - false => resume.poll_used(waiter), - }; - if edge.is_ready() { - return Poll::Ready(Step::Demand); - } - - // Watch the spliced copy for its end: complete means the logical - // track is over; anything else means the serving copy died. - if let Some((_, track)) = &serving - && let Poll::Ready(result) = track.poll_complete(waiter) - { - return Poll::Ready(match result { - Ok(()) => Step::Complete, - Err(err) => Step::Failed(err), - }); - } - - deadline.poll(waiter).map(|_| Step::Idle) - }) - .await - }; - - match step { - // The front's teardown aborts the logical track. - Step::Closed => return, - Step::Complete => { - let _ = resume.finish(); - return; - } - Step::Failed(err) => { - // The spliced copy died mid-serve. With delivered progress since - // its splice it's a normal failover: re-splice from the (possibly - // same) active source. A copy that died before producing anything - // is a refusal (a source whose track keeps dying right after - // acceptance must not re-splice forever), unless the source - // itself is closing: that corpse parks for its detach so a - // reconnect gets the seamless splice. - if resume.resume_position() == spliced_edge - && let Some(id) = serving_id - { - let closing = state - .read() - .sources - .iter() - .find(|r| r.id == id) - .is_some_and(|r| r.source.is_closing()); - if closing { - dead.insert(id); - } else { - refused.insert(id); - refusal = Some(err); - } - } - serving = None; - } - // The outer loop recomputes `used` and the countdown on the next pass. - // Drop the source copy on the unused edge so its producer goes idle - // immediately; keep the groups it already delivered on a local track - // so resume stays spliced until the linger expires. A source with no - // snapshot (a spliced copy from another front) still releases its - // segment: the next reader re-splices, paying `TRACK_INFO` again. - Step::Demand => { - if !resume.is_used() - && let Some((_, source)) = serving.take() - { - drop(warm.take()); - let local = warm_copy(&source); - drop(source); - if resume.release().is_err() { - return; - } - if let Some(local) = local { - if let Err(err) = resume.takeover(&local.track) { - let _ = resume.abort(err); - return; - } - warm = Some(local); - } - } - } - // Forget which route we were serving from, or the `gone` edge that woke - // us keeps firing: the id stays absent from the table, the wait returns - // Ready at once, and the loop spins on a full core without ever parking. - // The segment itself stays spliced into `resume` (readers keep whatever - // it delivered) until a replacement is proven servable. - Step::NoRoute => serving = None, - Step::Idle => { - // Nobody has read the track for the linger: release the cached - // segment. The source's copy was already dropped on the unused - // edge. The logical track stays alive and re-splices on the next - // reader. - if resume.release().is_err() { - // Finished or aborted meanwhile; the track is over either way. - return; - } - serving = None; - drop(warm.take()); - } - Step::Splice(id, source) => { - // Ask the source for its copy and wait for the info to resolve, - // proving it servable, before splicing it in. Bail out early if - // the table moves on while waiting. - let attempt = match source.track(&name) { - Ok(track) => { - // `into_inner` sheds the `Pending` future wrapper so only - // the pollable (which is `Sync`) is held across the await. - let query = track.query().into_inner(); - let skip = |id: u64| refused.contains(&id) || dead.contains(&id); - let info = kio::wait(|waiter| { - if let Poll::Ready(result) = query.poll(waiter) { - return Poll::Ready(Some(result)); - } - match state.poll(waiter, |s| { - if s.closed || s.serve_route(skip) != Some(id) { - Poll::Ready(()) - } else { - Poll::Pending - } - }) { - Poll::Ready(_) => Poll::Ready(None), - Poll::Pending => Poll::Pending, - } - }) - .await; - match info { - // The table changed under us; re-pick from the top. - None => continue, - // A copy that is already aborted can't be spliced; - // its error is the source's answer for the track. - // One that claims the same content with different - // metadata is refused rather than reinterpreted. - Some(Ok(info)) => match track.poll_complete(&kio::Waiter::noop()) { - Poll::Ready(Err(err)) => Err(err), - _ => match state.write() { - Ok(mut state) => state.accept_track_info(&name, info).map(|()| track), - Err(_) => Err(Error::Dropped), - }, - }, - Some(Err(err)) => Err(err), - } - } - Err(err) => Err(err), - }; - - match attempt { - Ok(track) => { - if let Err(err) = resume.takeover(&track) { - // Closed means the logical track already ended - // (finished or aborted). Anything else is a boundary - // bug; abort rather than strand subscribers on a - // track no task serves (a no-op after a clean end). - let _ = resume.abort(err); - return; - } - // `dead` must survive the takeover. A dead source can never - // serve again (ids are never reused; `is_closing` is - // terminal), and the retain above reclaims its entry once - // its watcher detaches it. Re-admitting a still-attached - // closing route here would let `serve_route`'s active - // preference re-dispatch it, and because both its instant - // failure and a cached standby splice resolve without - // awaiting, the loop would spin inside a single poll, - // starving the watcher whose detach ends the cycle. - // The new segment has produced nothing yet, so this is - // the edge the copy is asked to advance. - spliced_edge = resume.resume_position(); - serving = Some((id, track)); - } - // The source itself closed or deliberately ended: not a - // verdict on the track. Park until its watcher detaches - // it and the table promotes a replacement. - Err(_) if source.is_closing() => { - dead.insert(id); - serving = None; - } - // The dispatched source does not carry the track: a publisher - // announces a broadcast only once its tracks exist, so the - // answer is authoritative and never re-asked. Skip the source - // for this track; the verdict block aborts once every source - // has refused. - Err(err) => { - tracing::debug!(name = %name, source = id, %err, "source refused track"); - refused.insert(id); - refusal = Some(err); - serving = None; - } - } - } - } - } -} - -/// The content identity of a remotely-served front: who originated the route -/// its first source arrived through. -#[derive(Clone, Copy, PartialEq, Eq)] -enum Identity { - /// No source has attached yet: the first request may resolve through any - /// covering route, and whoever serves it fixes the identity. - Undetermined, - /// The serving route's first hop was absent or [`Hop::UNKNOWN`], which - /// identifies nobody and never matches itself: the front cannot resume, so - /// its source ending ends it. Two anonymous publishers must never pass for - /// one reconnecting. - Anonymous, - /// The first hop of the serving route: the endpoint that originated it. - /// Routes sharing it are the same origin reached another way and safe to - /// resume through; anything else is different content at the same path. - Publisher(Hop), -} - -impl Identity { - /// The first-hop pin for [`OriginState::best_route`], or `None` when any - /// route qualifies (no source attached yet). - fn pin(&self) -> Option { - match self { - Self::Publisher(hop) => Some(*hop), - _ => None, - } - } - - /// Whether route selection applies at all: an anonymous front never adopts - /// another route. - fn routable(&self) -> bool { - !matches!(self, Self::Anonymous) - } -} - -/// Everything [`run_remote_front`] owns, queued by [`Consumer::request_broadcast`]. -struct RemoteFrontTask { - /// The route table the front selects and re-selects from. +/// Everything [`run_front`] owns, queued by [`Consumer::request_broadcast`]. +struct FrontTask { + /// The route table the front selects from. shared: kio::Shared, - /// The front's source table, shared with its [`serve_track`] tasks. - state: kio::Producer, /// The spliced broadcast the front serves. broadcast: broadcast::Producer, /// Absolute path of the front. @@ -2522,304 +1719,465 @@ struct RemoteFrontTask { watch: Watch, /// Resolves the requesters parked on the front's channel. request: kio::Producer, - tasks: TasksWeak, timers: Clock, } -/// Owns a remotely-served front: materializes the path from the best covering -/// route, dispatches its spliced tracks, and re-splices through routes sharing -/// the front's first-hop identity when the serving source dies or a better -/// qualifying route appears, so subscribers never observe a survivable route -/// change. The front ends (aborting its subscribers) when the source dies with -/// no qualifying route left, when a qualifying route refuses the path, or when -/// the origin tears down. -async fn run_remote_front(task: RemoteFrontTask) { - let RemoteFrontTask { +/// The driver's side of one logical track: the handles behind the names the +/// machine uses. +struct TrackIo { + resume: super::resume::Producer, + /// The copy whose info resolved, waiting for the machine to splice it. + staged: Option<(u64, track::Consumer)>, + /// A query in flight: the source asked, its copy, and the pending info. + query: Option<(u64, track::Consumer, track::Querying)>, + /// The spliced copy: its source and the track. + copy: Option<(u64, track::Consumer)>, + /// The delivered edge when the copy spliced in: a copy that dies without + /// advancing it delivered nothing. Snapshotted per splice, not per wake, so an + /// unrelated wake between the copy's last frame and its death cannot launder + /// its progress away. + edge: Option, + /// Delivered groups kept after the copy was dropped, so resume stays spliced + /// through the linger without pinning the source as a reader. + warm: Option, + /// Whether the track had a reader as of the last demand edge. + used: bool, +} + +/// Drives one front: feeds the world's events to a [`Front`] and performs the +/// actions it returns, until the front ends. The decisions live in the machine; +/// this only waits and executes, so nothing here decides anything twice. +async fn run_front(task: FrontTask) { + let FrontTask { shared, - state, broadcast, path, exclude, watch, request, - tasks, timers, } = task; + /// What the wait below returns: one thing that happened. enum Step { - /// A spliced track needs a serve task. - Serve(Arc, super::resume::Producer), - /// The in-flight upstream request resolved. - Resolved(Result), - /// The attached source closed. - SourceDead, - /// A route covering the path changed, or the origin tore down. + Assigned(Arc, super::resume::Producer), + Resolved(u64, Result), + SourceClosed(u64), + Info(Arc, u64, Result), + Ended(Arc, u64, Result<(), Error>), + Demand(Arc), + Deadline, Table, } - let mut front = FrontDriver { - state, - broadcast, - initial: Some(request), - identity: Identity::Undetermined, - serving: None, + let mut front = Front::new(TRACK_IDLE_LINGER); + let mut sources: HashMap = HashMap::new(); + let mut next_source = 0u64; + // The in-flight upstream request: the route and its pending channel. + let mut upstream: Option<(u64, kio::Consumer)> = None; + let mut tracks: HashMap, TrackIo> = HashMap::new(); + let mut deadline = crate::runtime::Deadline::new(&timers); + // The watch generation the last selection saw. + let mut seen = 0; + let mut events: VecDeque = VecDeque::new(); + + // Read the table for the machine: the best qualifying route and whether + // the serving source is on its way out. Also what the watch wakes for. + let select = |front: &mut Front, sources: &HashMap, seen: &mut u64| -> Event { + let table = shared.read(); + if table.closed { + return Event::Closed; + } + // Read alongside the decision, under the lock a poke takes first. + *seen = watch.seen(); + front.retain_routes(|route| table.routes.covers(&path.as_path(), route)); + let best = table + .best_route(&path.as_path(), exclude, front.pin(), front.refused_routes()) + .map(|entry| Candidate { + route: entry.id, + first: entry.hops.iter().next().copied(), + local: entry.local, + }); + let serving_closing = front + .serving() + .and_then(|id| sources.get(&id)) + .is_some_and(|source| source.is_closing()); + Event::Selected { best, serving_closing } }; - // The in-flight upstream request: the targeted route entry, its first hop, - // and the pending channel. - let mut upstream: Option<(u64, Option, kio::Consumer)> = None; - // Routes that refused the path (an authoritative reject while another source - // was serving, or a queue with no live handler). Never retried while the - // entry stands; a reconnect is a fresh entry. - let mut refused: HashSet = HashSet::new(); - // Why the last candidate fell through, reported if the front dies unresolved. - let mut last_err: Option = None; - - 'run: loop { - // Decide what the table means for us, then fire at most one upstream - // request. Locks are sequential, never nested: the table's, then the - // chosen route's queue. - // The watch generation this decision saw; the wait below sleeps until the - // covering routes move past it. - let seen; - let fire = { - let table = shared.read(); - if table.closed { - break 'run; - } - seen = watch.seen(); - refused.retain(|id| table.routes.covers(&path.as_path(), *id)); - let best = match front.identity.routable() { - true => table.best_route(&path.as_path(), exclude, front.identity.pin(), &refused), - false => None, - }; - let serving_route = front.serving.as_ref().map(|(_, route, _)| *route); - match best { - // The serving route is still the best qualifying one. - Some(entry) if Some(entry.id) == serving_route => { - upstream = None; - None - } - // A better (or replacement) qualifying route: request through it, - // unless we already are. - Some(entry) => match upstream.as_ref().is_some_and(|(route, ..)| *route == entry.id) { - true => None, - false => Some(( - entry.id, - entry.hops.iter().next().copied(), - entry.server.clone().expect("best_route yields served entries"), - )), - }, - // Nothing qualifies. A live source keeps serving (its route may - // return); without one the front is over. - None => { - upstream = None; - match &front.serving { - Some(_) => None, - None => { - let err = match front.identity { - Identity::Undetermined => last_err.take().unwrap_or(Error::Unroutable), - _ => last_err.take().unwrap_or(Error::Dropped), - }; - front.end(err); - return; + + events.push_back(select(&mut front, &sources, &mut seen)); + + loop { + while let Some(event) = events.pop_front() { + for action in front.step(event) { + match action { + Action::Reselect => events.push_back(select(&mut front, &sources, &mut seen)), + Action::Request { route } => { + // The entry, its identity for the front, and what it serves. + let found = { + let table = shared.read(); + table + .routes + .covering(&path.as_path()) + .find(|entry| entry.id == route) + .map(|entry| { + ( + Candidate { + route, + first: entry.hops.iter().next().copied(), + local: entry.local, + }, + entry.source.clone(), + entry.server.clone(), + ) + }) + }; + let Some((candidate, source, server)) = found else { + events.push_back(Event::Resolved { + route, + result: Err(Refusal { + err: Error::Unroutable, + standing: false, + }), + }); + continue; + }; + front.identify(candidate); + if let Some(source) = source { + let id = next_source; + next_source += 1; + sources.insert(id, source); + events.push_back(Event::Resolved { route, result: Ok(id) }); + continue; + } + let Some(server) = server else { + events.push_back(Event::Resolved { + route, + result: Err(Refusal { + err: Error::Unroutable, + standing: true, + }), + }); + continue; + }; + let mut serve = server.lock(); + if serve.closed { + // Retracted under us, or its handler dropped while the + // announcement stands: it cannot serve. + drop(serve); + events.push_back(Event::Resolved { + route, + result: Err(Refusal { + err: Error::Unroutable, + standing: true, + }), + }); + continue; + } + // A source this route already materialized for the path + // attaches without another upstream round trip. + if let Some(weak) = serve.served.get(&path) { + drop(serve); + let id = next_source; + next_source += 1; + sources.insert(id, weak.consume()); + events.push_back(Event::Resolved { route, result: Ok(id) }); + continue; } + let pending = match serve.requests.join(&path) { + Some(producer) => producer.consume(), + None => { + let producer = kio::Producer::::default(); + let consumer = producer.consume(); + match serve.requests.insert(path.clone(), producer) { + Ok(()) => consumer, + // No live handler behind the route: it cannot + // serve, whatever the table says. + Err(_) => { + drop(serve); + events.push_back(Event::Resolved { + route, + result: Err(Refusal { + err: Error::Unroutable, + standing: true, + }), + }); + continue; + } + } + } + }; + upstream = Some((route, pending)); } - } - } - }; - - if let Some((route, first, server)) = fire { - let mut serve = server.lock(); - match serve.closed { - // The server is gone: retracted under us, or its handler dropped - // while the announcement stands. Either way the route cannot - // serve, so skip it rather than re-picking it forever. - true => { - refused.insert(route); - last_err = Some(Error::Unroutable); - continue 'run; - } - false => { - // A source this route already materialized for the path - // attaches without another upstream round trip. - if let Some(weak) = serve.served.get(&path) { - drop(serve); - front.attach(weak.consume(), route, first); - continue 'run; + Action::Detach { source } => { + sources.remove(&source); + // Its copies go with it; the segments they delivered stay + // spliced until a replacement resumes past them. + for io in tracks.values_mut() { + if io.copy.as_ref().is_some_and(|(s, _)| *s == source) { + io.copy = None; + } + if io.query.as_ref().is_some_and(|(s, ..)| *s == source) { + io.query = None; + } + if io.staged.as_ref().is_some_and(|(s, _)| *s == source) { + io.staged = None; + } + } } - let pending = match serve.requests.join(&path) { - Some(producer) => producer.consume(), - None => { - let producer = kio::Producer::::default(); - let consumer = producer.consume(); - match serve.requests.insert(path.clone(), producer) { - Ok(()) => consumer, - // No live handler behind the route: it cannot - // serve, whatever the table says. - Err(_) => { - refused.insert(route); - last_err = Some(Error::Unroutable); - continue 'run; - } + Action::Resolve => { + if let Ok(mut pending) = request.write() { + pending.resolved.get_or_insert(Ok(broadcast.consume())); + } + } + Action::Query { track: name, source } => { + let Some(io) = tracks.get_mut(&name) else { continue }; + let closing = sources.get(&source).is_some_and(|s| s.is_closing()); + match sources.get(&source).map(|s| s.track(&name)) { + Some(Ok(copy)) => { + // `into_inner` sheds the `Pending` future wrapper so only + // the pollable (which is `Sync`) is held across the wait. + let query = copy.query().into_inner(); + io.query = Some((source, copy, query)); } + Some(Err(err)) => events.push_back(Event::TrackInfo { + track: name, + source, + closing, + result: Err(err), + }), + None => {} + } + } + Action::Splice { track: name, source } => { + let Some(io) = tracks.get_mut(&name) else { continue }; + let Some((staged, copy)) = io.staged.take() else { + continue; + }; + if staged != source { + continue; + } + if let Err(err) = io.resume.takeover(©) { + // Closed means the logical track already ended. Anything + // else is a boundary bug; abort rather than strand + // subscribers on a track nobody serves. + let _ = io.resume.abort(err); + tracks.remove(&name); + continue; } - }; - upstream = Some((route, first, pending)); + io.warm = None; + // The new segment has produced nothing yet: this is the + // edge the copy is asked to advance. + io.edge = io.resume.resume_position(); + io.copy = Some((source, copy)); + } + Action::Park { track: name } => { + let Some(io) = tracks.get_mut(&name) else { continue }; + let Some((_, copy)) = io.copy.take() else { continue }; + // Drop the source copy so its producer goes idle at once; keep + // the groups it delivered on a local track so resume stays + // spliced until the linger expires. + let warm = warm_copy(©); + drop(copy); + if io.resume.release().is_err() { + tracks.remove(&name); + continue; + } + if let Some(warm) = warm { + if let Err(err) = io.resume.takeover(&warm.track) { + let _ = io.resume.abort(err); + tracks.remove(&name); + continue; + } + io.warm = Some(warm); + } + } + Action::Release { track: name } => { + let Some(io) = tracks.get_mut(&name) else { continue }; + io.warm = None; + if io.resume.release().is_err() { + tracks.remove(&name); + } + } + Action::Finish { track: name } => { + if let Some(mut io) = tracks.remove(&name) { + let _ = io.resume.finish(); + } + } + Action::Abort { track: name, err } => { + if let Some(mut io) = tracks.remove(&name) { + tracing::debug!(name = %name, %err, "aborting track"); + let _ = io.resume.abort(err); + } + } + Action::Arm { at } => deadline.set(at), + Action::End { err } => { + if let Ok(mut pending) = request.write() { + pending.resolved.get_or_insert(Err(err.clone())); + } + broadcast.abort_spliced(err); + broadcast.finish(); + return; + } } } } let step = kio::wait(|waiter| { - if let Poll::Ready((name, resume)) = front.broadcast.poll_spliced_assigned(waiter) { - return Poll::Ready(Step::Serve(name, resume)); + if let Poll::Ready((name, resume)) = broadcast.poll_spliced_assigned(waiter) { + return Poll::Ready(Step::Assigned(name, resume)); } - - if let Some((.., pending)) = &upstream + if let Some((route, pending)) = &upstream && let Poll::Ready(result) = pending.poll(waiter, |p| match &p.resolved { Some(result) => Poll::Ready(result.clone()), None => Poll::Pending, }) { - return Poll::Ready(Step::Resolved(match result { - Ok(resolved) => resolved, - // The queue died unresolved (its handler dropped): the route - // could not serve. - Err(_closed) => Err(Error::Unroutable), - })); + return Poll::Ready(Step::Resolved( + *route, + match result { + Ok(resolved) => resolved, + // The queue died unresolved (its handler dropped): the route + // could not serve. + Err(_closed) => Err(Error::Unroutable), + }, + )); } - - if let Some((.., source)) = &front.serving + if let Some(id) = front.serving() + && let Some(source) = sources.get(&id) && source.poll_closed(waiter).is_ready() { - return Poll::Ready(Step::SourceDead); + return Poll::Ready(Step::SourceClosed(id)); + } + for (name, io) in &tracks { + if let Some((source, _, query)) = &io.query + && let Poll::Ready(result) = query.poll(waiter) + { + return Poll::Ready(Step::Info(name.clone(), *source, result)); + } + if let Some((source, copy)) = &io.copy + && let Poll::Ready(result) = copy.poll_complete(waiter) + { + return Poll::Ready(Step::Ended(name.clone(), *source, result)); + } + // Watch the demand edge in whichever direction is unmet. + let edge = match io.used { + true => io.resume.poll_unused(waiter), + false => io.resume.poll_used(waiter), + }; + if edge.is_ready() { + return Poll::Ready(Step::Demand(name.clone())); + } + } + if deadline.poll(waiter).is_ready() { + return Poll::Ready(Step::Deadline); } - - // Only a change to a route covering this path can alter the decision - // above, and only those poke the watch: an unrelated announcement - // never wakes this front. watch.poll_changed(waiter, seen).map(|()| Step::Table) }) .await; - match step { - Step::Serve(name, resume) => { - tasks.push(serve_track(front.state.clone(), name, resume, timers.clone())); + let event = match step { + Step::Assigned(name, resume) => { + tracks.insert( + name.clone(), + TrackIo { + resume, + staged: None, + query: None, + copy: None, + edge: None, + warm: None, + used: false, + }, + ); + Event::TrackAssigned { track: name } } - Step::Resolved(result) => { - let (route, first, _) = upstream.take().expect("resolved an in-flight request"); + Step::Resolved(route, result) => { + upstream = None; match result { - Ok(source) => front.attach(source, route, first), - // The route retracted before serving: the table already - // reflects it, so the next pass retries the survivor. Each - // such retry consumed a real retraction, so this cannot spin. - // A retraction and a handler's rejection resolve alike, so the - // table tells them apart: an `Unroutable` from a route that - // still stands is the handler's answer, and re-asking it - // would spin forever. - Err(Error::Unroutable) if !shared.lock().routes.covers(&path.as_path(), route) => { - last_err = Some(Error::Unroutable); + Ok(source) => { + let id = next_source; + next_source += 1; + sources.insert(id, source); + Event::Resolved { route, result: Ok(id) } } - // An authoritative refusal of the path. It ends a front with - // no other source (a refusal is never retried); a serving - // front merely skips the refuser. - Err(err) => match &front.serving { - Some(_) => { - refused.insert(route); - last_err = Some(err); - } - None => { - front.end(err); - return; + Err(err) => { + // A retraction and a handler's rejection resolve alike, so + // the table tells them apart: an `Unroutable` from a route + // that still stands is the handler's answer. + let standing = + !matches!(err, Error::Unroutable) || shared.read().routes.covers(&path.as_path(), route); + Event::Resolved { + route, + result: Err(Refusal { err, standing }), } + } + } + } + Step::SourceClosed(source) => Event::SourceClosed { source }, + Step::Info(name, source, result) => { + let closing = sources.get(&source).is_some_and(|s| s.is_closing()); + let Some(io) = tracks.get_mut(&name) else { continue }; + let Some((_, copy, _)) = io.query.take() else { continue }; + // A copy that is already aborted cannot be spliced; its error is + // the source's answer for the track. + let result = match result { + Ok(info) => match copy.poll_complete(&kio::Waiter::noop()) { + Poll::Ready(Err(err)) => Err(err), + _ => Ok(info), }, + Err(err) => Err(err), + }; + // Staged only while the track has a reader: without one the machine + // will not splice, and a held copy would keep the source subscribed. + if result.is_ok() && io.used { + io.staged = Some((source, copy)); + } + Event::TrackInfo { + track: name, + source, + closing, + result, } } - Step::SourceDead => { - front.detach_dead(); - last_err = Some(Error::Dropped); + Step::Ended(name, source, result) => { + let closing = sources.get(&source).is_some_and(|s| s.is_closing()); + let Some(io) = tracks.get_mut(&name) else { continue }; + io.copy = None; + let delivered = io.resume.resume_position() != io.edge; + Event::TrackEnded { + track: name, + source, + closing, + result, + delivered, + } } - Step::Table => {} - } - } - - // The origin tore down; its teardown already rejected parked requesters. - front.end(Error::Dropped); -} - -/// The mutable half of a remote front's watcher: the source table and spliced -/// broadcast it manages, who is serving, and the requesters awaiting the first -/// source. -struct FrontDriver { - /// The front's source table, shared with its [`serve_track`] tasks. - state: kio::Producer, - /// The spliced broadcast the front serves. - broadcast: broadcast::Producer, - /// Resolves the requesters parked on the front once the first source - /// attaches (or the front dies first). - initial: Option>, - /// The front's content identity, fixed by the first source to attach. - identity: Identity, - /// The attached source: its id in the front's table, the route entry that - /// served it, and the source itself. - serving: Option<(u64, u64, broadcast::Consumer)>, -} - -impl FrontDriver { - /// Attach a materialized source: replace the previous one (its tracks - /// re-splice at a group boundary), fix the front's identity on the first - /// attach, and resolve the requesters parked on the front's channel. - fn attach(&mut self, source: broadcast::Consumer, route: u64, first: Option) { - let Ok(mut front) = self.state.write() else { return }; - if let Some((id, ..)) = self.serving.take() { - front.sources.retain(|entry| entry.id != id); - } - let id = front.next_source; - front.next_source += 1; - front.sources.push(FrontSource { - id, - source: source.clone(), - }); - front.reselect(); - drop(front); - - self.serving = Some((id, route, source)); - if self.identity == Identity::Undetermined { - self.identity = match first { - Some(hop) if hop != Hop::UNKNOWN => Identity::Publisher(hop), - _ => Identity::Anonymous, - }; - } - if let Some(request) = self.initial.take() - && let Ok(mut pending) = request.write() - { - pending.resolved.get_or_insert(Ok(self.broadcast.consume())); - } - } - - /// Remove the dead source from the table. Its spliced tracks park on the - /// empty table until a replacement attaches or the front ends. - fn detach_dead(&mut self) { - if let Some((id, ..)) = self.serving.take() - && let Ok(mut front) = self.state.write() - { - front.sources.retain(|entry| entry.id != id); - front.reselect(); - } - } - - /// End the front: reject requesters still parked on its channel, close the - /// source table so its serve tasks exit, and abort the spliced broadcast so - /// its subscribers observe the end. - fn end(&mut self, err: Error) { - if let Some(request) = self.initial.take() - && let Ok(mut pending) = request.write() - { - pending.resolved.get_or_insert(Err(err.clone())); - } - if let Ok(mut front) = self.state.write() { - front.closed = true; - } - self.broadcast.abort_spliced(err); - self.broadcast.finish(); + Step::Demand(name) => { + let Some(io) = tracks.get_mut(&name) else { continue }; + io.used = io.resume.is_used(); + if !io.used { + // Nothing will be spliced now: let go of the copies a query + // holds, or the source stays subscribed with nobody reading. + io.query = None; + io.staged = None; + } + match io.used { + true => Event::Used { track: name }, + false => Event::Unused { + track: name, + now: timers.now(), + }, + } + } + Step::Deadline => { + // Cleared here so a fired deadline cannot keep firing; the machine + // re-arms what is still parked. + deadline.set(None); + Event::Deadline { now: timers.now() } + } + Step::Table => select(&mut front, &sources, &mut seen), + }; + events.push_back(event); } } @@ -2853,7 +2211,7 @@ struct RouteNode { } /// What a [`Watch`] observes: bumped by every change to a route covering its -/// path, a local broadcast attaching at the path, and the origin's teardown. +/// path (a broadcast published here is one) and by the origin's teardown. #[derive(Default)] struct Watched { generation: u64, @@ -3103,13 +2461,6 @@ impl RouteTable { } } - /// Wake the watches of exactly `path`: a local broadcast attached there. - fn poke_at(&self, path: &Path) { - if let Some(node) = self.root.find(path.parts()) { - node.poke(); - } - } - /// Wake every watch: the origin is tearing down. fn poke_all(&self) { self.root.walk(&mut |node| node.poke()); @@ -3135,8 +2486,8 @@ impl RouteTable { /// it, and the remotely-served fronts. /// /// Carried in a [`kio::Shared`], so producers, consumers, and handlers work -/// under one lock. Local broadcasts live in the tree ([`OriginNode`]) instead; -/// this holds everything advertised or served on demand. +/// under one lock. Broadcasts published here are route table entries like the +/// routes announced from elsewhere; this holds everything that serves a path. #[derive(Default)] struct OriginState { // The announced routes, keyed by prefix. The table holds one entry per live @@ -3299,18 +2650,11 @@ impl OriginState { /// unroutable instead of being routed around it. Among routes at the winning /// prefix, the cheapest served one is picked by [`route_order`]. /// - /// With `publisher` set, only routes originated by that first hop are - /// candidates: this is the identity a front resumes through, and a route from - /// anyone else is different content rather than an alternate path (see - /// [`run_remote_front`]). `Hop::UNKNOWN` identifies nobody, so callers never - /// pin it. - fn best_route( - &self, - path: &Path, - exclude: Option, - publisher: Option, - refused: &HashSet, - ) -> Option<&RouteEntry> { + /// `pin` is the front's identity: only routes it admits are candidates, since + /// a route from anyone else is different content rather than an alternate + /// path (see [`Front`]). A broadcast published on this origin outranks any + /// remote route at the same prefix. + fn best_route(&self, path: &Path, exclude: Option, pin: Pin, refused: &HashSet) -> Option<&RouteEntry> { // Covering prefixes of one path form a chain, so the deepest node with a // candidate holds the unique longest prefix; walking down, the last such // node decides. @@ -3322,16 +2666,13 @@ impl OriginState { .iter() .filter(|entry| entry.scope.matches(path.as_str())) .filter(|entry| entry.visible_to(exclude)) - .filter(|entry| match publisher { - Some(first) => entry.hops.iter().next() == Some(&first), - None => true, - }) + .filter(|entry| entry.qualifies(pin)) .filter(|entry| !refused.contains(&entry.id)) .peekable(); if candidates.peek().is_some() { best = candidates - .filter(|entry| entry.server.is_some()) - .min_by_key(|entry| route_order(&entry.prefix, entry)); + .filter(|entry| entry.serves(path)) + .min_by_key(|entry| (!entry.local, route_order(&entry.prefix, entry))); } } best @@ -3538,8 +2879,6 @@ pub struct Requesting { } enum RequestState { - // Already announced: resolves immediately with a clone of this broadcast. - Ready(broadcast::Consumer), // Unroutable at request time: resolves immediately with this error. Baked in so // `request_broadcast` itself stays infallible. Failed(Error), @@ -3548,10 +2887,6 @@ enum RequestState { } impl Requesting { - fn ready(broadcast: broadcast::Consumer) -> Self { - Self::new(RequestState::Ready(broadcast)) - } - fn failed(error: Error) -> Self { Self::new(RequestState::Failed(error)) } @@ -3599,7 +2934,6 @@ impl Requesting { /// Poll for the requested broadcast without blocking. pub fn poll_ok(&self, waiter: &kio::Waiter) -> Poll> { match &self.inner { - RequestState::Ready(broadcast) => Poll::Ready(Ok(self.hand_out(broadcast.clone()))), RequestState::Failed(error) => Poll::Ready(Err(error.clone())), RequestState::Pending(consumer) => Poll::Ready( match ready!(consumer.poll(waiter, |state| match &state.resolved { @@ -3681,7 +3015,7 @@ impl Consume for track::Consumer { } } -/// Cheap read handle over an origin's broadcast tree and route table. +/// Cheap read handle over an origin's route table. /// /// Clones share the underlying state without allocating any per-cursor /// resources. To receive route announcements, call [`Self::announced`]; to @@ -3807,19 +3141,21 @@ impl Consumer { self.clone() } - /// Internal synchronous lookup: the local broadcast at `path`, if any. - fn resolve(&self, path: impl AsPath) -> Option { + /// The newest broadcast published on this origin at exactly `path`, if any. + /// Test-only: a request goes through the table like any other. + #[cfg(test)] + pub(crate) fn get_broadcast(&self, path: impl AsPath) -> Option { let full = self.root.join(path).to_owned(); if !self.scope.permits(&full) { return None; } - self.scope.tree.lock().resolve_broadcast(&full) - } - - /// [`Self::resolve`] as the peek the tests assert on. - #[cfg(test)] - pub(crate) fn get_broadcast(&self, path: impl AsPath) -> Option { - self.resolve(path) + let table = self.shared.lock(); + table + .routes + .at(&full) + .filter(|entry| entry.local) + .min_by_key(|entry| route_order(&entry.prefix, entry)) + .and_then(|entry| entry.source.clone()) } /// Block until an announced route covers `path`, and return it. @@ -3933,23 +3269,21 @@ impl Consumer { /// Resolve a broadcast by exact path. /// - /// Returns a [`kio::Pending`] future (resolved synchronously where possible), - /// mirroring [`track::Consumer::fetch_group`](track::Consumer::fetch_group). - /// The lookup order: - /// - /// 1. A local broadcast at the exact path ([`Producer::create_broadcast`]), - /// announced or not, resolves immediately. - /// 2. Otherwise the best announced route covering the path (most specific - /// prefix first, then cheapest) serves it on demand: sessions materialize - /// the path from the peer that announced the route. Concurrent requests - /// for the same path coalesce onto one shared front, which outlives any - /// single route: when its serving route dies or a better one appears, the - /// front re-splices through the best route sharing its first hop at a - /// group boundary, invisibly to subscribers. A route change that does not - /// preserve the first hop ends the broadcast instead, and the next - /// request re-serves the path. + /// Returns a [`kio::Pending`] future, mirroring + /// [`track::Consumer::fetch_group`](track::Consumer::fetch_group). Every + /// path resolves through a front the origin's [`Driver`] runs: the request + /// mints one or joins the one already serving the path, and the front picks + /// the best route covering it (a broadcast published on this origin at the + /// exact path first, announced or not; then the most specific prefix, then + /// the cheapest) and materializes it, from the broadcast itself or from the + /// peer that announced the route. When its serving source dies or a better + /// qualifying route appears, the front re-splices through the best route + /// sharing its first hop at a group boundary, invisibly to subscribers. A + /// change that does not preserve the first hop ends the broadcast instead, + /// and the next request re-serves the path. /// - /// The returned future resolves to [`Error::Unroutable`] when neither exists. + /// The returned future fails with [`Error::Unroutable`] at once when nothing + /// covers the path. /// A route claims capability, not inventory: resolving a covered path /// succeeds optimistically, and a path that names nothing surfaces as /// [`Error::NotFound`] on its tracks instead. @@ -3966,12 +3300,6 @@ impl Consumer { // catalog it reads may reference. let requested = path.to_owned(); - // A local broadcast at the exact path wins. - if let Some(broadcast) = self.resolve(&path) { - let resolved = Requesting::ready(broadcast).with_path(requested).with_stats(scope); - return kio::Pending::new(resolved); - } - // Routes only cover paths within this consumer's scope. if !self.scope.permits(&absolute) { return kio::Pending::new(Requesting::failed(Error::Unauthorized)); @@ -3999,7 +3327,7 @@ impl Consumer { // Nothing serves the path: no local broadcast and no served route. if state - .best_route(&absolute.as_path(), self.exclude, None, &HashSet::new()) + .best_route(&absolute.as_path(), self.exclude, Pin::Any, &HashSet::new()) .is_none() { return kio::Pending::new(Requesting::failed(Error::Unroutable)); @@ -4014,13 +3342,6 @@ impl Consumer { cache_duration: self.cache_duration, path: absolute.clone(), }); - let front_state = kio::Producer::new(FrontState { - next_source: 0, - sources: Vec::new(), - track_info: HashMap::new(), - active: None, - closed: false, - }); let request = kio::Producer::::default(); let consumer = request.consume(); let watch = state.watch(&self.shared, &absolute); @@ -4034,15 +3355,13 @@ impl Consumer { // Released before the push: a set whose handles are gone drops the task, // and the `Watch` it carries unregisters under this same lock. drop(state); - self.tasks.push(run_remote_front(RemoteFrontTask { + self.tasks.push(run_front(FrontTask { shared: self.shared.clone(), - state: front_state, broadcast, path: absolute, exclude: self.exclude, watch, request, - tasks: self.tasks.clone(), timers: self.timers.clone(), })); kio::Pending::new(Requesting::queued(consumer).with_path(requested).with_stats(scope)) @@ -4394,11 +3713,7 @@ mod tests { // Created unannounced: reachable by exact path, invisible to the cursor. let broadcast = producer.create_broadcast("room/alice").unwrap(); announced.assert_next_wait(); - let local = consumer - .request_broadcast("room/alice") - .now_or_never() - .expect("local") - .expect("resolves"); + let local = consumer.request_broadcast("room/alice").await.expect("resolves"); assert_eq!(local.info().path.as_str(), "room/alice"); broadcast.announce(Route::default().with_cost(3)).unwrap(); @@ -4415,11 +3730,7 @@ mod tests { announced.assert_next_ended("room/alice"); broadcast.unannounce(); announced.assert_next_wait(); - let local = consumer - .request_broadcast("room/alice") - .now_or_never() - .expect("local") - .expect("resolves"); + let local = consumer.request_broadcast("room/alice").await.expect("resolves"); assert_eq!(local.info().path.as_str(), "room/alice"); // Back on the air, then the end of the broadcast retracts for good. @@ -4478,54 +3789,6 @@ mod tests { announced.assert_next_wait(); } - /// A handle names its subtrees by path, so scoping or rooting one creates - /// nothing in the tree; only a broadcast does, and its teardown prunes it back - /// out. Otherwise a relay grows by one node per distinct scope for as long as - /// it runs. - #[tokio::test] - async fn scoping_creates_no_nodes() { - let producer = origin(1).produce(); - let bare = producer.node_count(); - - let scoped = producer.consume().scope("", &scopes(&["room/a", "room/b"])).unwrap(); - assert_eq!(producer.node_count(), bare, "scoping should not create nodes"); - assert!( - producer - .consume() - .scope("room/c", &Patterns::from(Pattern::all())) - .is_ok() - ); - assert_eq!(producer.node_count(), bare, "rooting should not create nodes"); - drop(scoped); - - let broadcast = producer.create_broadcast("room/a/chat").unwrap(); - assert_eq!(producer.node_count(), bare + 3, "room, a and chat"); - broadcast.finish(); - settle(|| producer.node_count() == bare).await; - } - - /// A scoped handle resolves by path against the tree rather than through a - /// node handle, so pruning its subtree between two broadcasts cannot strand it - /// on an orphan no lookup reaches. - #[tokio::test] - async fn scoped_handles_survive_a_prune() { - let producer = origin(1).produce(); - let scoped = producer.scope("", &scopes(&["channel"])).unwrap(); - let consumer = producer.consume().scope("", &scopes(&["channel"])).unwrap(); - - // Create the scoped subtree and prune it straight back out. - let first = scoped.create_broadcast("channel/chat").unwrap(); - assert!(consumer.get_broadcast("channel/chat").is_some()); - first.finish(); - settle(|| consumer.get_broadcast("channel/chat").is_none()).await; - - let _second = scoped.create_broadcast("channel/chat").unwrap(); - assert!( - consumer.get_broadcast("channel/chat").is_some(), - "the source attached into an orphan" - ); - } - #[tokio::test] async fn announce_keeps_its_prefix_under_a_producer_scope() { let producer = origin(1).produce(); @@ -4823,10 +4086,7 @@ mod tests { let broadcast = producer.create_broadcast("room/alice").unwrap(); let _announcement = producer.announce("room/alice", Route::default()).unwrap(); - let resolved = resolving - .now_or_never() - .expect("resolves once announced") - .expect("resolves"); + let resolved = resolving.await.expect("resolves"); assert_eq!(resolved.info().path.as_str(), "room/alice"); drop(broadcast); } @@ -4837,11 +4097,7 @@ mod tests { let consumer = producer.consume(); let broadcast = producer.create_broadcast("room/alice").unwrap(); - let resolved = consumer - .request_broadcast("room/alice") - .now_or_never() - .expect("local lookup is synchronous") - .expect("resolves"); + let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); assert_eq!(resolved.info().path.as_str(), "room/alice"); drop(broadcast); @@ -5039,11 +4295,7 @@ mod tests { assert_eq!(resolved.info().path.as_str(), "room/alice"); // A repeat request shares the served broadcast instead of re-asking. - let again = consumer - .request_broadcast("room/alice") - .now_or_never() - .expect("cached") - .expect("resolves"); + let again = consumer.request_broadcast("room/alice").await.expect("resolves"); assert!(again.is_clone(&resolved)); } @@ -5788,19 +5040,11 @@ mod tests { let consumer = producer.consume(); let first = producer.create_broadcast("room/alice").unwrap(); - let resolved = consumer - .request_broadcast("room/alice") - .now_or_never() - .expect("resolves") - .expect("resolves"); + let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); // A second source at the same path joins the same front. let second = producer.create_broadcast("room/alice").unwrap(); - let again = consumer - .request_broadcast("room/alice") - .now_or_never() - .expect("resolves") - .expect("resolves"); + let again = consumer.request_broadcast("room/alice").await.expect("resolves"); assert!(again.is_clone(&resolved)); // Losing one source keeps the front alive; losing both closes it. @@ -5829,11 +5073,7 @@ mod tests { group.write_frame(crate::Timestamp::ZERO, b"cached".as_ref()).unwrap(); group.finish().unwrap(); - let resolved = consumer - .request_broadcast("room/alice") - .now_or_never() - .expect("resolves") - .expect("resolves"); + let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); let mut subscription = resolved .track("video") .unwrap() @@ -5898,11 +5138,7 @@ mod tests { // The leaf's front view: a spliced broadcast, so any front serving from // it holds a spliced source copy with nothing to snapshot. - let leaf_front = leaf_consumer - .request_broadcast("room/alice") - .now_or_never() - .expect("resolves") - .expect("resolves"); + let leaf_front = leaf_consumer.request_broadcast("room/alice").await.expect("resolves"); let mid = origin(2).produce(); let mid_server = mid.dynamic("room", Route::default().with_hops(hops(&[10]))).unwrap(); @@ -5999,11 +5235,7 @@ mod tests { let first = producer.create_broadcast("room/alice").unwrap(); let track = first.create_track("video", None).unwrap(); - let resolved = consumer - .request_broadcast("room/alice") - .now_or_never() - .expect("resolves") - .expect("resolves"); + let resolved = consumer.request_broadcast("room/alice").await.expect("resolves"); let mut subscription = resolved .track("video") .unwrap() @@ -6111,6 +5343,23 @@ mod tests { assert_eq!(table.routes.root.cursors_below, 0); } + /// A session handed an `origin::Producer` drops it once it has its own + /// handles, so the driver must keep running while a published broadcast + /// lives, and finish once the last one is gone. + #[test] + fn a_published_broadcast_keeps_the_driver_running() { + let (producer, mut driver) = Producer::new(Config::new(origin(1))); + let waiter = kio::Waiter::noop(); + let broadcast = producer.create_broadcast("room/a").unwrap(); + drop(producer); + assert!( + driver.poll(Instant::now(), &waiter).is_ok(), + "the broadcast is lifecycle work" + ); + drop(broadcast); + assert!(matches!(driver.poll(Instant::now(), &waiter), Err(Error::Closed))); + } + #[test] fn watch_wakes_only_for_covering_changes() { let producer = origin(1).produce(); diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 0123e60cc2d..28226cf633b 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -377,6 +377,7 @@ impl Producer { /// copy can die, while the segment stays spliced so readers keep what it already /// delivered. That makes this, not the caller's own handle on the serving route, /// the condition for arming an idle release. + #[cfg(test)] pub(crate) fn is_spliced(&self) -> bool { !self.state.read().segments.is_empty() } diff --git a/rs/moq-net/src/util.rs b/rs/moq-net/src/util.rs index d26001a5955..0d2eab0bab8 100644 --- a/rs/moq-net/src/util.rs +++ b/rs/moq-net/src/util.rs @@ -53,20 +53,27 @@ pub(crate) async fn err_only(fut: impl Future>) -> E { } /// Cloneable handle for submitting futures to a driver-owned [`TaskSet`]. +#[derive(Clone)] pub(crate) struct Tasks { state: kio::Shared, + /// The set's lifetime: it finishes once every owning handle, [`Tasks`] or + /// [`Keepalive`], is gone. A kio token rather than a count under the queue's + /// lock, so a claim on the lifetime is `Send` even where the queued futures + /// are not (wasm). + alive: kio::Producer<()>, +} + +/// A claim on a [`TaskSet`]'s lifetime with no submission queue: what a +/// broadcast published on an origin holds, so the driver outlives the producer +/// handles a session was given and dropped. +pub(crate) struct Keepalive { + _alive: kio::Producer<()>, } -/// The submission queue between [`Tasks`] handles and their [`TaskSet`], -/// following kio's convention for [`kio::Shared`] liveness: the handle count -/// and the set's closure are plain fields maintained under the same lock that -/// queues the work. +/// The submission queue between [`Tasks`] handles and their [`TaskSet`]. #[derive(Default)] struct Submissions { queued: VecDeque>, - /// Live [`Tasks`] handles; the set finishes once this reaches zero and the - /// remaining work drains. - senders: usize, /// The set dropped: submissions are discarded, as the driver that would /// have polled them has torn down. closed: bool, @@ -87,20 +94,19 @@ impl Tasks { impl Tasks { /// A non-owning submission handle: pushes work while the driver lives, but - /// neither keeps the set from finishing nor counts as a sender. For read + /// neither keeps the set from finishing nor counts as an owner. For read /// handles, whose existence must not extend the origin's lifecycle. pub fn downgrade(&self) -> TasksWeak { TasksWeak { state: self.state.clone(), + alive: self.alive.consume(), } } -} -impl Clone for Tasks { - fn clone(&self) -> Self { - self.state.lock().senders += 1; - Self { - state: self.state.clone(), + /// A claim on the set's lifetime alone; see [`Keepalive`]. + pub fn keepalive(&self) -> Keepalive { + Keepalive { + _alive: self.alive.clone(), } } } @@ -110,26 +116,20 @@ impl Clone for Tasks { #[derive(Clone)] pub(crate) struct TasksWeak { state: kio::Shared, + alive: kio::Consumer<()>, } impl TasksWeak { /// Queue a future for polling by the associated [`TaskSet`], dropped if the - /// set (or every owning [`Tasks`] handle) is already gone. + /// set (or every owning handle) is already gone. pub fn push(&self, task: impl MaybeBoxedExt<'static, Output = ()>) { let mut state = self.state.lock(); - if !state.closed && state.senders > 0 { + if !state.closed && !self.alive.is_closed() { state.queued.push_back(task.maybe_boxed()); } } } -impl Drop for Tasks { - fn drop(&mut self) { - // The mutation wakes a parked set, which may be what lets it finish. - self.state.lock().senders -= 1; - } -} - /// A dynamic set of child tasks polled by its parent driver, built on /// [`kio::Tasks`] (per-child wakeups, drop cancels). /// @@ -139,24 +139,21 @@ impl Drop for Tasks { pub(crate) struct TaskSet { state: kio::Shared, active: kio::Tasks, + /// Closed once every owning handle is gone; see [`Tasks::alive`]. + alive: kio::Consumer<()>, } impl TaskSet { /// Create a task submission handle and its driver-owned set. pub fn new() -> (Tasks, Self) { - let state = kio::Shared::new(Submissions { - queued: VecDeque::new(), - senders: 1, - closed: false, - }); - let tasks = Tasks { state: state.clone() }; - ( - tasks, - Self { - state, - active: kio::Tasks::new(), - }, - ) + let state = kio::Shared::::default(); + let alive = kio::Producer::<()>::default(); + let set = Self { + state: state.clone(), + active: kio::Tasks::new(), + alive: alive.consume(), + }; + (Tasks { state, alive }, set) } /// Create a set that only its owner can push to, for a loop that accepts streams @@ -165,6 +162,8 @@ impl TaskSet { Self { state: kio::Shared::default(), active: kio::Tasks::new(), + // No owning handle ever exists, so the set finishes when its children drain. + alive: kio::Producer::<()>::default().consume(), } } @@ -183,19 +182,24 @@ impl TaskSet { // children run below: a child pushing through a `Tasks` clone mid-pass // wakes whoever is registered, and without it that push would be lost. let mut submissions_done = false; - while let Poll::Ready(mut state) = self.state.poll(waiter, |state| { - if state.queued.is_empty() && state.senders > 0 { - Poll::Pending - } else { - Poll::Ready(()) - } - }) { + loop { + // Registers `waiter` for the last owning handle dropping, which may be + // what lets the set finish. + let orphaned = self.alive.poll_closed(waiter).is_ready(); + let Poll::Ready(mut state) = self.state.poll(waiter, |state| { + if state.queued.is_empty() && !orphaned { + Poll::Pending + } else { + Poll::Ready(()) + } + }) else { + break; + }; while let Some(task) = state.queued.pop_front() { self.active.push(future_task(task)); } - // No handles left: nothing can ever submit again, so no - // registration is needed. - if state.senders == 0 { + // No handles left: nothing can ever submit again. + if orphaned { submissions_done = true; break; }