diff --git a/quest/m2/cluster-routes/acceptance.md b/quest/m2/cluster-routes/acceptance.md index 849fd746ae..5fdd652d05 100644 --- a/quest/m2/cluster-routes/acceptance.md +++ b/quest/m2/cluster-routes/acceptance.md @@ -38,7 +38,7 @@ router's forwarding and operation-count fixtures went with it, and `route/tests/multicast*.rs` covers the new one. The route fan-out quest those pinned was deleted with the old rule, its one surviving cost moved to -[route benches](/quest/m3/route-benches.md). +[route wakes](/quest/m4/route-wakes.md). Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): the switch makes `origin::Config::update_hold` crate-private, since nothing diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index fa5620962e..00b60381a1 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -38,7 +38,6 @@ children, [the uncovered cases](/quest/m2/multicast/accept.md), [route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring the lite-06 mitigations. After the cut come warmth's wire ([cache-aware routing](/quest/m3/cache-routing/README.md)), -[route benches](/quest/m3/route-benches.md), [the proof binding](/quest/m3/route-proof-binding.md), [route health](/quest/m3/route-health/README.md), and [outbound federation](/quest/m3/cluster-routing/federation.md). diff --git a/quest/m2/multicast/mesh-replay.md b/quest/m2/multicast/mesh-replay.md index 3411373d2a..91e561abb4 100644 --- a/quest/m2/multicast/mesh-replay.md +++ b/quest/m2/multicast/mesh-replay.md @@ -34,10 +34,13 @@ to the tests that exist: newest and previous release meshed, and lite-07 is chosen link by link: only two new relays speak it to each other. Nothing runs a mesh in that state, where one path crosses both versions. -- The same decision keeps the per-broadcast rewrite of - [route benches](/quest/m3/route-benches.md) in m3 on the condition that it - is measured on a deployed mesh's shape before that mesh's cluster links - move to lite-07. This quest's report is that measurement. +- The same decision kept the fix for the per-broadcast rewrite after the + cut, now in [route wakes](/quest/m4/route-wakes.md), on the condition that + the cost is measured on a deployed mesh's shape before that mesh's cluster + links move to lite-07. + [moq-dev/moq#9114](https://github.com/moq-dev/moq/pull/9114) benched it: + 10 origin writes per broadcast when an upstream reopens twice. This + quest's report is that measurement on a real mesh. Decided 2026-11-18 (maintainer, on [moq-dev/moq#9110](https://github.com/moq-dev/moq/pull/9110)): ✅ it blocks @@ -69,7 +72,7 @@ Public API: none. Wire: none. ## Related -- [Route benches](/quest/m3/route-benches.md) - the costs this measures on a real mesh +- [Route wakes](/quest/m4/route-wakes.md) - owns the rewrite this measures on a real mesh - [The cases no test covers run on real sessions](/quest/m2/multicast/accept.md) - the same failures on mock sessions with mocked time - [Random fault schedules](/quest/m4/cluster-fault-schedules.md) - searches fault timings on the simulated clock - [A nightly soak](/quest/m4/relay-soak.md) - one real relay under churn diff --git a/quest/m2/multicast/route-counters.md b/quest/m2/multicast/route-counters.md index dc6122356e..819f6dd497 100644 --- a/quest/m2/multicast/route-counters.md +++ b/quest/m2/multicast/route-counters.md @@ -37,9 +37,9 @@ Agent's choices, for the implementer to revisit: - Count in `rs/moq-net/src/stats.rs` beside `routes_refused`, per label and never per broadcast or per node, so the cost and the output stay flat as the mesh grows. -- The count of broadcasts rewritten is the cost - [route benches](/quest/m3/route-benches.md) measures, read from a running - relay. +- The count of broadcasts rewritten is the cost the `routes/reopen` bench + measures and [route wakes](/quest/m4/route-wakes.md) owns, read from a + running relay. - Say what the announce counters mean on a lite-07 session, where a route change is no longer one announce per broadcast, and fix the doc or the counter if they now mislead. @@ -49,5 +49,4 @@ shared role. Wire: none. ## Related -- [Route benches](/quest/m3/route-benches.md) - measures the rewrite these count - [Real relays replay a mesh and its faults](/quest/m2/multicast/mesh-replay.md) - reads these in its report diff --git a/quest/m3/route-benches.md b/quest/m3/route-benches.md deleted file mode 100644 index 09341f8638..0000000000 --- a/quest/m3/route-benches.md +++ /dev/null @@ -1,51 +0,0 @@ -# [M] Route benches on the router - -## Goal - -`route::relay::Router` has benches for what the old path rule measured, swept -over both axes, and the costs the switch measured and left are fixed or kept -with a number that says why. - -## Plan - -Found by [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) and -proposed by [moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053). -The origin rewrite below is what survived of the route fan-out quest, whose -other findings were the old path router's indexes and doubts. - -Decided 2026-11-18 (maintainer, on -[moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053#issuecomment-6975349236)): -✅ kept (rejected: dropping it). It follows the lite-07 cut, as the -2026-11-17 brief allowed for benches, so it sits in m3 rather than the -multicast epic the cut waits on. - -- The path-rule benches (drain, repath, follow) left `benches/routes.rs` - with the old router and have no equivalent on the new one. Add them, - swept over both axes: relays on the path and broadcasts per node. -- Measure each cost the switch left, then fix it or record why it stays: - - re-offering a route sends its node's claims again; - - a non-routing peer is told one ANNOUNCE_UPDATE per broadcast when a - route's cost changes; - - an edit swapping one prefix for another tells again the claims both - cover; - - a relay whose only route closes ends its subscriptions while the claim - stays shown; - - a change to a route rewrites every claim behind it into the origin: - `route::node` `State::flush` relanes each broadcast the route's node - claims, and `Served` carries the cost, so a re-price is one origin - write per broadcast, each waking the fronts below it. Recommended: one - shared record per route in the origin that its broadcasts refer to, - built with [route wakes](/quest/m4/route-wakes.md), which owns the - fronts' side. -- [moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053) sends a - whole copy of a node's claims wherever its filtered copy used to go - (`withheld` is gone), so a claim moving between a node's two identities - costs a revision under each. Bench a churn of adoptions over a marked - link with it. - -Public API: none. Wire: none, unless a fix changes what a claim carries, -which lands before the lite-07 cut. - -## Related - -- [Cache-aware multicast](/quest/m2/multicast/README.md) - the route layer these measure diff --git a/quest/m4/route-wakes.md b/quest/m4/route-wakes.md index 4b7e4f8b57..d74f686769 100644 --- a/quest/m4/route-wakes.md +++ b/quest/m4/route-wakes.md @@ -1,4 +1,4 @@ -# [M] A route change wakes only the fronts it affects +# [L] A route change writes once and wakes only the fronts it affects ## Goal @@ -37,9 +37,27 @@ Facts from `main` (`rs/moq-net/src/model/origin.rs`, since Wildcard landed in exactly the path skips both hashes. A join moves only paths the newcomer wins, a leave only paths the leaver was winning. - On lite-07 the route layer writes into this table through `route::node`, - one write per broadcast whose way changed; a change to one route rewrites - every broadcast behind it, which [route benches](/quest/m3/route-benches.md) - owns. This quest is the fronts' half: what each write wakes. + one write per broadcast whose way changed; a change to one route's cost + rewrites every broadcast behind it (`route::node` `State::flush` relanes + each, and `Served` carries the cost). This quest owns both halves: one + shared record per route in the origin that its broadcasts refer to, so a + re-price is one write, and what each write wakes. `routes/reopen/spare` + measures the route side (moq-dev/moq#9114): an upstream reopening twice + in a row costs 10 origin writes per broadcast behind it, 2.8 s at 16 + routes of 4,096 broadcasts, linear in each axis; `routes/drain` costs 2 + per broadcast, 480 ms at the same shape. A route that moves to another + path at the same rank through the same session already rewrites nothing; + one that moves to another session at the same rank relanes every + broadcast its node claims. + +Decided 2026-11-18 (maintainer, on +[moq-dev/moq#9114](https://github.com/moq-dev/moq/pull/9114)): ✅ the origin +record lives here, in m4 (rejected: moving this quest to m3; a separate m3 +quest for the record alone). It replaces that morning's decision to keep it +in m3 with the benches. The measured cost above is about 4 µs a write, so +the worst flap seen on a deployed mesh costs one relay a fraction of a +second, and [the mesh replay](/quest/m2/multicast/mesh-replay.md) measures +it on that mesh's graph before its links move to lite-07. Decisions: @@ -79,10 +97,9 @@ Decisions: Decided 2026-10-13: coordinate the index with the route model and per-track active/standby selection. Do not build a second route table per track or -turn one route's cost change into per-broadcast mutations. Whichever -implementation lands first, this or the origin record in -[route benches](/quest/m3/route-benches.md), preserves these operation-count -tests for the next. +turn one route's cost change into per-broadcast mutations. The origin +record and the index land together here, keeping these operation-count +tests. Verification: `origin/pool_churn` leaves flat in served paths at every pool width it already sweeps, and a unit test that counts `select` calls per @@ -95,7 +112,3 @@ Public API: none. Wire: none. ## Required - [Cache-aware routing](/quest/m3/cache-routing/README.md) - adds the warm pick to how a relay chooses its source, so the index is built once on the final selection - -## Related - -- [Route benches](/quest/m3/route-benches.md) - the route layer's side: one origin record per route instead of a write per broadcast diff --git a/rs/moq-net/benches/routes.rs b/rs/moq-net/benches/routes.rs index a5cd9435f8..7add2c7a5a 100644 --- a/rs/moq-net/benches/routes.rs +++ b/rs/moq-net/benches/routes.rs @@ -24,12 +24,43 @@ //! axis is expected to grow, since the stream hears its prefix's claims again; the //! prefixes axis no faster than the prefixes held. //! +//! - drain, repath, follow, reopen, swap, remark: a relay hearing every forwarding node +//! behind it from its upstream neighbours, telling one downstream neighbour every route +//! and one watcher every claim, and serving what it hears into a real origin as a +//! session does. Swept over the forwarding nodes whose routes the event changes and the +//! broadcasts behind each. Each prints what the event cost once per shape: routes and +//! claims sent downstream, claims told the watcher, and writes into the origin. +//! - drain: the cheapest upstream sends GOAWAY, so every route moves to the second. +//! - repath: the only upstream moves every route onto another path at the same cost. +//! Nothing anyone is told or served changes, so only the routes axis may grow. +//! - follow: the first upstream connects to a relay holding nothing and offers every +//! route with its claims. +//! - reopen/spare: the cheapest upstream closes its session and opens another, twice in +//! a row, as a relay restarting every link does, while a second upstream offers every +//! route dearer. Each route's cost changes four times, and each change rewrites every +//! claim behind it into the origin and tells the watcher each again. +//! - reopen/sole: the same with no second upstream, so each route ends and returns: +//! the claims stay shown, unavailable, and come back sent whole. +//! - swap: the watcher swaps the whole tree for the first forwarding node's prefix and +//! back. Each edit tells again the claims both prefixes cover. +//! - remark: the relay's own claim moves between its two identities, then back, as an +//! endpoint's does when it reconnects over a marked link. Swept over the claims the relay +//! holds and the downstream neighbours, which each hear a revision under both. +//! //! The route layer is crate-private, so this reaches it through the `fuzz` module. //! //! Run with `cargo bench -p moq-net --features fuzz --bench routes`. +#![expect( + clippy::disallowed_methods, + clippy::disallowed_types, + reason = "custom timings leave the unmeasured half of an event out of the sample" +)] + +use std::time::{Duration, Instant}; + use criterion::{BatchSize, BenchmarkId, Criterion, criterion_group, criterion_main}; -use moq_net::fuzz; +use moq_net::fuzz::{self, RouteMesh, RouteShape, RouteTally}; /// `(claims, streams)` shapes. const SHAPES: [(usize, usize); 4] = [(10, 10), (10_000, 10), (10, 1_000), (10_000, 1_000)]; @@ -46,6 +77,13 @@ const FOLLOWED: [(usize, usize); 4] = [(16, 1), (1_024, 1), (16, 16), (1_024, 16 /// `(claims, lag)` shapes for a relay hearing a stream that lags another. const LAGGING: [(usize, usize); 4] = [(100, 1), (10_000, 1), (100, 100), (10_000, 10_000)]; +/// `(relays, broadcasts)` shapes for a relay between routing neighbours: the forwarding +/// nodes whose routes one event changes, and the broadcasts behind each. +const ROUTED: [(usize, usize); 6] = [(1, 16), (1, 4_096), (16, 16), (16, 4_096), (64, 16), (64, 1_024)]; + +/// `(broadcasts, downstream)` shapes for the relay's own claim moving identity. +const REMARKED: [(usize, usize); 4] = [(16, 1), (4_096, 1), (16, 16), (4_096, 16)]; + fn bench(c: &mut Criterion, name: &str, interest: fn(usize) -> String, sent: usize) { let mut group = c.benchmark_group(format!("routes/{name}")); for (claims, streams) in SHAPES { @@ -150,6 +188,156 @@ fn bench_refollow(c: &mut Criterion) { group.finish(); } +/// A relay shaped `(relays, broadcasts)` with `upstream` upstream neighbours. +fn mesh(relays: usize, broadcasts: usize, upstream: usize) -> RouteShape { + RouteShape { + relays, + broadcasts, + upstream, + downstream: 1, + watchers: 1, + } +} + +fn label(relays: usize, broadcasts: usize) -> String { + format!("{relays}r_{broadcasts}b") +} + +/// Print what one event cost, once per shape. +fn report(name: &str, id: &str, tally: RouteTally) { + let RouteTally { + routes, + claims, + told, + writes, + } = tally; + eprintln!("routes/{name}/{id}: routes={routes} claims={claims} told={told} writes={writes}"); +} + +/// Time `event` on a mesh built once per shape, and `restore` after it unmeasured. +fn bench_event( + c: &mut Criterion, + name: &str, + upstream: usize, + event: fn(&mut RouteMesh) -> RouteTally, + restore: fn(&mut RouteMesh), +) { + let mut group = c.benchmark_group(format!("routes/{name}")); + group.sample_size(10); + for (relays, broadcasts) in ROUTED { + let id = label(relays, broadcasts); + let mut fixture = None; + group.bench_function(BenchmarkId::from_parameter(&id), |b| { + let mesh = fixture.get_or_insert_with(|| { + let mut mesh = RouteMesh::new(mesh(relays, broadcasts, upstream)); + // Once to settle, then once to report. + event(&mut mesh); + restore(&mut mesh); + report(name, &id, event(&mut mesh)); + restore(&mut mesh); + mesh + }); + b.iter_custom(|iters| { + let mut spent = Duration::ZERO; + for _ in 0..iters { + let start = Instant::now(); + std::hint::black_box(event(mesh)); + spent += start.elapsed(); + restore(mesh); + } + spent + }); + }); + } + group.finish(); +} + +fn bench_drain(c: &mut Criterion) { + bench_event(c, "drain", 2, |mesh| mesh.goaway(0), |mesh| _ = mesh.reopen(0)); +} + +fn bench_repath(c: &mut Criterion) { + bench_event(c, "repath", 1, |mesh| mesh.repath(0), |_| {}); +} + +/// Close and open the first upstream twice. +fn twice(mesh: &mut RouteMesh) -> RouteTally { + let mut tally = mesh.reopen(0); + tally += mesh.reopen(0); + tally +} + +fn bench_reopen_spare(c: &mut Criterion) { + bench_event(c, "reopen/spare", 2, twice, |_| {}); +} + +fn bench_reopen_sole(c: &mut Criterion) { + bench_event(c, "reopen/sole", 1, twice, |_| {}); +} + +fn bench_swap(c: &mut Criterion) { + bench_event( + c, + "swap", + 1, + |mesh| { + let mut tally = mesh.swap(true); + tally += mesh.swap(false); + tally + }, + |_| {}, + ); +} + +fn bench_follow(c: &mut Criterion) { + let mut group = c.benchmark_group("routes/follow"); + group.sample_size(10); + for (relays, broadcasts) in ROUTED { + let id = label(relays, broadcasts); + let mut reported = false; + group.bench_function(BenchmarkId::from_parameter(&id), |b| { + if !std::mem::replace(&mut reported, true) { + report("follow", &id, RouteMesh::new(mesh(relays, broadcasts, 0)).open(0)); + } + b.iter_batched_ref( + || RouteMesh::new(mesh(relays, broadcasts, 0)), + |mesh| mesh.open(0), + BatchSize::PerIteration, + ); + }); + } + group.finish(); +} + +fn bench_remark(c: &mut Criterion) { + let mut group = c.benchmark_group("routes/remark"); + for (broadcasts, downstream) in REMARKED { + let id = format!("{broadcasts}b_{downstream}d"); + let mut fixture = None; + group.bench_function(BenchmarkId::from_parameter(&id), |b| { + let mesh = fixture.get_or_insert_with(|| { + let shape = RouteShape { + downstream, + ..mesh(0, 0, 0) + }; + let mut mesh = RouteMesh::new(shape); + mesh.own(broadcasts); + mesh.remark(); + mesh.remark(); + let mut tally = mesh.remark(); + tally += mesh.remark(); + report("remark", &id, tally); + mesh + }); + b.iter(|| { + mesh.remark(); + mesh.remark() + }); + }); + } + group.finish(); +} + criterion_group!( benches, bench_watched, @@ -159,5 +347,12 @@ criterion_group!( bench_confirmed, bench_teardown, bench_refollow, + bench_drain, + bench_repath, + bench_follow, + bench_reopen_spare, + bench_reopen_sole, + bench_swap, + bench_remark, ); criterion_main!(benches); diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index 27f4dea0a1..8add046551 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -12,10 +12,12 @@ //! It tracks the crate's internals, so it can change in any release. See `fuzz/README.md` for the workflow. mod ingest; +mod mesh; mod request; mod session; pub use ingest::ingest; +pub use mesh::{RouteMesh, RouteShape, RouteTally}; pub use request::request; pub use session::{session_ietf, session_lite}; diff --git a/rs/moq-net/src/fuzz/mesh.rs b/rs/moq-net/src/fuzz/mesh.rs new file mode 100644 index 0000000000..6ef28b0e1f --- /dev/null +++ b/rs/moq-net/src/fuzz/mesh.rs @@ -0,0 +1,416 @@ +//! A relay's lite-07 route node between routing neighbours, serving what it hears into a +//! real origin as a session does, for the routes bench. +//! +//! The relay hears every forwarding node behind it from each upstream neighbour, the +//! first cheapest, and tells each downstream neighbour every route and each watcher every +//! claim. An event returns a [`RouteTally`] of what it cost the relay. + +use std::collections::HashMap; +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; + +use crate::Path; +use crate::route::NodeId; +use crate::route::node::{self, Msg, Output, Stream}; +use crate::route::relay::{Change, Edit, Hop, Offer, Port, Role}; + +use super::{route_ask, route_claim, route_clock, route_id, route_leaf}; + +/// The shape of a [`RouteMesh`]. +#[derive(Clone, Copy, Debug, Default)] +pub struct RouteShape { + /// Forwarding nodes behind the upstream neighbours, each offered by every one. + pub relays: usize, + /// Claims each forwarding node holds. + pub broadcasts: usize, + /// Routing neighbours offering a route to every forwarding node, the first cheapest. + pub upstream: usize, + /// Routing neighbours asking for every route. + pub downstream: usize, + /// Endpoint sessions asking for the whole tree. + pub watchers: usize, +} + +/// What one event cost the relay, by what it sent and wrote. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct RouteTally { + /// Routes, route ends, DEAD and DRAIN sent to the downstream neighbours. + pub routes: usize, + /// Claim changes sent to the downstream neighbours. + pub claims: usize, + /// Claim changes told to the watchers, each an ANNOUNCE_* message. + pub told: usize, + /// Writes into the origin: a claim served, served differently, or let go, each one a + /// closing session marks unavailable as it leaves it behind, and each such orphan the + /// node releases. + pub writes: usize, +} + +impl std::ops::AddAssign for RouteTally { + fn add_assign(&mut self, other: Self) { + self.routes += other.routes; + self.claims += other.claims; + self.told += other.told; + self.writes += other.writes; + } +} + +/// A claim the origin holds through one session, and the route it was last given. +struct Held { + claim: crate::origin::Claim, + route: crate::origin::RouteInfo, +} + +type ServedKey = (NodeId, Path, crate::Instance); + +/// One upstream session and what it serves into the origin. +struct Upstream { + stream: Stream, + served: HashMap, +} + +/// A claim a closed session left behind, held until the node lets it go, which counts as +/// a write into the origin. +struct Left(crate::origin::Claim, Arc); + +impl node::Orphan for Left { + fn release(self: Box, ended: bool) { + self.1.fetch_add(1, Ordering::Relaxed); + if ended { + self.0.withdrawn(); + } + } +} + +/// A relay's route node shaped as a [`RouteShape`], settled, with the origin it serves. +pub struct RouteMesh { + // Dropped before the node: a stream disconnects from its node on drop. + up: Vec>, + down: Vec, + watchers: Vec, + node: node::Node, + origin: crate::origin::Producer, + _driver: crate::origin::Driver, + /// Orphans the node released since the last [`Self::collect`]. + released: Arc, + shape: RouteShape, + instance: crate::Instance, + next_tag: u64, + /// The middle hop every route takes, which [`Self::repath`] moves. + via: u64, + /// Whether the relay files its own claim as one that crossed a marked link. + marked: bool, +} + +impl RouteMesh { + /// The middle hop of every route. + const VIA: u64 = 500; + + /// A relay shaped as `shape`, everything heard, told, and served. + pub fn new(shape: RouteShape) -> Self { + let (origin, driver) = crate::origin::Producer::new(crate::origin::Config::default()); + let mut mesh = Self { + up: Vec::new(), + down: Vec::new(), + watchers: Vec::new(), + node: node::Node::new(route_id(2), true, route_clock()), + origin, + _driver: driver, + released: Arc::default(), + shape, + instance: crate::Instance::random(), + next_tag: 1, + via: Self::VIA, + marked: false, + }; + for j in 0..shape.downstream { + let port = Port { + cost: 1, + priority: 0, + upstream: false, + role: Role::Routing(route_id(300 + j as u64)), + }; + let tag = mesh.tag(); + let stream = mesh.node.connect(tag, port).unwrap(); + stream.recv(Msg::Edited).unwrap(); + stream.recv(route_ask(Path::empty())).unwrap(); + stream.flush(); + mesh.down.push(stream); + } + for _ in 0..shape.watchers { + let tag = mesh.tag(); + let stream = mesh.node.connect(tag, route_leaf()).unwrap(); + stream.recv(route_ask(Path::empty())).unwrap(); + stream.flush(); + mesh.watchers.push(stream); + } + for i in 0..shape.upstream { + mesh.open(i); + } + mesh + } + + fn tag(&mut self) -> u64 { + self.next_tag += 1; + self.next_tag + } + + /// The forwarding node `r` behind the upstream neighbours. + fn relay(r: usize) -> NodeId { + route_id(1_000 + r as u64) + } + + /// The upstream neighbour `i`. + fn upstream(i: usize) -> NodeId { + route_id(100 + i as u64) + } + + /// What upstream neighbour `i` offers of forwarding node `r`. + fn offer(&self, i: usize, r: usize) -> Offer { + let path = [ + Hop { + node: Self::relay(r), + session: 0, + }, + Hop { + node: route_id(self.via), + session: 1, + }, + Hop { + node: Self::upstream(i), + session: r as u64 + 1, + }, + ]; + Offer { + path: path.into(), + draining: false, + priority: i8::MAX, + cost: 2, + marked: false, + } + } + + /// Upstream neighbour `i` connects and offers every route with its claims. + pub fn open(&mut self, i: usize) -> RouteTally { + let port = Port { + cost: 1 + i as u64, + priority: 0, + upstream: false, + role: Role::Routing(Self::upstream(i)), + }; + let tag = self.tag(); + let stream = self.node.connect(tag, port).unwrap(); + // The neighbour answered what the relay asks it for. + stream.recv(Msg::Edited).unwrap(); + for r in 0..self.shape.relays { + let origin = Self::relay(r); + let offer = Some(self.offer(i, r)); + stream.recv(Msg::Route { origin, offer }).unwrap(); + let changes = (0..self.shape.broadcasts) + .map(|b| { + let path = Path::new(&format!("live/{r}/{b}")).unwrap(); + ((path, self.instance.clone()), Change::Live(route_claim(0, 0))) + }) + .collect(); + stream + .recv(Msg::Claims { + origin, + changes, + floor: 1, + }) + .unwrap(); + } + stream.flush(); + if self.up.len() <= i { + self.up.resize_with(i + 1, || None); + } + self.up[i] = Some(Upstream { + stream, + served: HashMap::new(), + }); + self.collect() + } + + /// Upstream neighbour `i`'s session closes, leaving what it served behind, as a + /// session does. + pub fn close(&mut self, i: usize) -> RouteTally { + let mut tally = RouteTally::default(); + let up = self.up[i].take().expect("an open upstream"); + for ((origin, path, instance), held) in up.served { + let route = held.route.with_available(false); + held.claim.update(route).unwrap(); + tally.writes += 1; + up.stream.orphan( + origin, + &(path, instance), + Box::new(Left(held.claim, self.released.clone())), + ); + } + drop(up.stream); + tally += self.collect(); + tally + } + + /// Upstream neighbour `i` closes its session and opens another. + pub fn reopen(&mut self, i: usize) -> RouteTally { + let mut tally = self.close(i); + tally += self.open(i); + tally + } + + /// Upstream neighbour `i` sends GOAWAY. + pub fn goaway(&mut self, i: usize) -> RouteTally { + self.up[i].as_ref().expect("an open upstream").stream.goaway(); + self.collect() + } + + /// Upstream neighbour `i` moves every route onto another path at the same cost. + pub fn repath(&mut self, i: usize) -> RouteTally { + self.via = match self.via { + Self::VIA => Self::VIA + 1, + _ => Self::VIA, + }; + let offers: Vec<_> = (0..self.shape.relays) + .map(|r| (Self::relay(r), self.offer(i, r))) + .collect(); + let stream = &self.up[i].as_ref().expect("an open upstream").stream; + for (origin, offer) in offers { + stream + .recv(Msg::Route { + origin, + offer: Some(offer), + }) + .unwrap(); + } + stream.flush(); + self.collect() + } + + /// The first watcher swaps the whole tree for the first forwarding node's prefix, or + /// back. + pub fn swap(&mut self, narrow: bool) -> RouteTally { + let (whole, part) = (Path::empty(), Path::new("live/0").unwrap()); + let edit = match narrow { + true => Edit { + remove: vec![whole], + add: vec![part], + }, + false => Edit { + remove: vec![part], + add: vec![whole], + }, + }; + let watcher = self.watchers.first().expect("a watcher"); + watcher.recv(Msg::Ask(edit)).unwrap(); + watcher.flush(); + self.collect() + } + + /// The relay claims `claims` more broadcasts of its own, beside the one + /// [`Self::remark`] moves. + pub fn own(&mut self, claims: usize) -> RouteTally { + for c in 1..=claims { + let broadcast = (Path::new(&format!("own/{c}")).unwrap(), self.instance.clone()); + self.node.own(false, broadcast, Change::Live(route_claim(0, 0))); + } + self.collect() + } + + /// The relay's own claim moves between its two identities: filed as having crossed a + /// marked link, or not. + pub fn remark(&mut self) -> RouteTally { + self.marked = !self.marked; + let broadcast = (Path::new("own/0").unwrap(), self.instance.clone()); + self.node.own(self.marked, broadcast, Change::Live(route_claim(0, 0))); + self.collect() + } + + /// Take everything the relay queued, serving into the origin what it asks to serve + /// and confirming each, until nothing more is queued. + fn collect(&mut self) -> RouteTally { + let mut tally = RouteTally::default(); + for stream in &self.down { + while let Some(msg) = stream.try_wire() { + match msg { + Msg::Claims { changes, .. } => tally.claims += changes.len(), + Msg::Route { .. } | Msg::Dead { .. } | Msg::Drain { .. } => tally.routes += 1, + _ => {} + } + } + } + for stream in &self.watchers { + while let Some(msg) = stream.try_wire() { + if let Msg::Announce { changes } = msg { + tally.told += changes.len(); + } + } + } + loop { + let mut wrote = 0; + for up in self.up.iter_mut().flatten() { + while up.stream.try_wire().is_some() {} + while let Some(output) = up.stream.try_serve() { + wrote += 1; + Self::apply(&self.origin, up, output); + } + } + if wrote == 0 { + break; + } + tally.writes += wrote; + } + tally.writes += self.released.swap(0, Ordering::Relaxed); + tally + } + + /// Serve `output` into the origin as a session does, and confirm it. + fn apply(origin: &crate::origin::Producer, up: &mut Upstream, output: Output) { + match output { + Output::Serve { + origin: node, + path, + instance, + served, + } => { + let lane = crate::origin::Lane { + primary: served.primary, + priority: Some(served.priority), + above: u8::try_from(served.attr.above).unwrap_or(u8::MAX), + }; + let route = crate::origin::RouteInfo::new() + .with_instance(instance.clone(), served.attr.epoch) + .with_cost(served.cost) + .with_node(Some(node)) + .with_lane(lane) + .with_draining(served.draining) + .with_available(served.available) + .with_marked(served.marked); + let key = (node, path, instance); + match up.served.get_mut(&key) { + Some(held) => { + held.claim.update(route.clone()).unwrap(); + held.route = route; + } + None => { + let claim = origin.forward(key.1.clone(), route.clone()).unwrap(); + up.served.insert(key.clone(), Held { claim, route }); + } + } + let (node, path, instance) = key; + up.stream.serving(node, &(path, instance), true); + } + Output::Unserve { + origin: node, + path, + instance, + ended, + } => { + if let Some(held) = up.served.remove(&(node, path, instance)) + && ended + { + held.claim.withdrawn(); + } + } + } + } +} diff --git a/rs/moq-net/src/route/relay.rs b/rs/moq-net/src/route/relay.rs index 95cb333634..c9ec8e6ede 100644 --- a/rs/moq-net/src/route/relay.rs +++ b/rs/moq-net/src/route/relay.rs @@ -425,6 +425,14 @@ struct Best { /// path, the next hop in use, the shorter path, the neighbour's Node ID, and its session. type Order = (bool, Reverse, Metric, bool, bool, usize, NodeId, N); +/// What a forwarding node's usable routes give the ways to its claims ([`Router::ways`]): +/// each one's next hop, rank, and mark, and the next hop of the one in use. +#[derive(Clone, Debug, PartialEq, Eq)] +struct Ways { + routes: Vec<(N, Rank, bool)>, + best: Option, +} + /// What changed at a router since its caller last asked. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) struct Changes { @@ -524,6 +532,8 @@ pub(crate) struct Router { /// Forwarding nodes whose route changed since the neighbours were last told. stale: BTreeSet, best: BTreeMap>, + /// What each forwarding node's usable routes give the ways to its claims. + ways: BTreeMap>, /// The forwarding nodes whose last selected route was marked, kept while no route /// reaches them and their claims stay known, so the claims keep the mark. marks: BTreeSet, @@ -634,6 +644,7 @@ impl Router { draining: BTreeSet::new(), stale: BTreeSet::new(), best: BTreeMap::new(), + ways: BTreeMap::new(), marks: BTreeSet::new(), tx_hops: BTreeMap::new(), sent_to: BTreeMap::new(), @@ -1519,12 +1530,14 @@ impl Router { fn reselect(&mut self, origin: NodeId) { let current = self.best.get(&origin).map(|best| best.via); let mut pick: Option<(Best, Order)> = None; - // A route that is not the best still serves the origin's claims, so each change - // to one reaches whoever follows them. - if let Some(log) = &mut self.log { - log.routes.insert(origin); - } - for mine in self.usable(origin) { + let usable = self.usable(origin); + let mut ways = Ways { + routes: (usable.iter()) + .map(|mine| (mine.via, mine.offer.rank(), mine.offer.marked)) + .collect(), + best: None, + }; + for mine in usable { let via = mine.via; let offer = mine.offer.clone(); let (draining, priority, cost) = mine.offer.rank(); @@ -1545,6 +1558,8 @@ impl Router { } } let pick = pick.map(|(best, _)| best); + ways.best = pick.as_ref().map(|best| best.via); + self.relog(origin, ways); let was = self.best.get(&origin); if was == pick.as_ref() { return; @@ -1563,9 +1578,6 @@ impl Router { Some(best) => self.best.insert(origin, best), None => self.best.remove(&origin), }; - if let Some(log) = &mut self.log { - log.routes.insert(origin); - } // Whether its claims are reachable follows the route, and so does the rank of each // that the rank reaches. if (flipped || remarked) @@ -1580,6 +1592,20 @@ impl Router { self.stale.insert(origin); } + /// Log `origin`'s route as changed once what its ways carry does. A route that is not + /// the best still serves the origin's claims, so a change to any usable route counts, + /// but one that leaves every way's next hop, rank, and mark alone, such as a new path + /// at the same cost, changes nothing a reader follows. + fn relog(&mut self, origin: NodeId, ways: Ways) { + let changed = match ways.routes.is_empty() { + true => self.ways.remove(&origin).is_some(), + false => self.ways.insert(origin, ways.clone()).as_ref() != Some(&ways), + }; + if changed && let Some(log) = &mut self.log { + log.routes.insert(origin); + } + } + /// What `peer` should hear of `origin` now. fn offer_for(&self, origin: NodeId, peer: N) -> Option { let to = self.peers.get(&peer)?; @@ -2708,6 +2734,7 @@ impl Router { draining, stale, best, + ways: _, marks, tx_hops: _, sent_to: _, @@ -2837,6 +2864,20 @@ impl Router { } assert_eq!(self.claimants, claimants, "claimants"); assert_eq!(self.ranked, ranked, "ranked claims"); + // Rebuilt with what the work count leaves out. + let work = self.work.get(); + let origins: BTreeSet<&NodeId> = self.heard.keys().chain(self.ways.keys()).collect(); + for origin in origins { + let routes: Vec<_> = (self.usable(*origin).iter()) + .map(|mine| (mine.via, mine.offer.rank(), mine.offer.marked)) + .collect(); + let ways = (!routes.is_empty()).then(|| Ways { + routes, + best: self.best.get(origin).map(|best| best.via), + }); + assert_eq!(self.ways.get(origin), ways.as_ref(), "the ways to {origin:?}"); + } + self.work.set(work); let deadlines: BTreeSet<(Instant, Broadcast)> = (self.waiting.iter()) .map(|(broadcast, at)| (*at, broadcast.clone())) .collect(); @@ -3155,6 +3196,33 @@ mod tests { assert_eq!(router.own().1[&(p("near"), 7)].cost, Metric::MAX, "an endpoint's claim"); } + /// A route that moves onto another path at the same rank changes no way to its claims, + /// so its caller relanes none of them; one that costs more does. + #[test] + fn a_route_logs_a_change_only_when_its_ways_do() { + let mut router = Router::::new(node(1), debounce(HOP)); + router.follow(); + open(&mut router, 2, Role::Routing(node(2))); + router.recv(2, route(5, 2, 1)).unwrap(); + assert_eq!(router.changes().routes, BTreeSet::from([node(5)]), "a new route"); + let mut moved = route(5, 2, 1); + if let Msg::Route { offer: Some(offer), .. } = &mut moved { + let mut path = offer.path.to_vec(); + path.insert( + 1, + Hop { + node: node(4), + session: 1, + }, + ); + offer.path = path.into(); + } + router.recv(2, moved).unwrap(); + assert!(router.changes().routes.is_empty(), "the same rank on another path"); + router.recv(2, route(5, 2, 2)).unwrap(); + assert_eq!(router.changes().routes, BTreeSet::from([node(5)]), "a dearer route"); + } + /// A claim reached over a route ranks as the route does too: draining if the route is, /// at the lower of the two priorities, and at the sum of the two costs. Beside it, this /// node's own copy leads to the neighbour that gave it.