diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index c3a17568a5..61a22a95ee 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -41,9 +41,14 @@ whose fix changes the wire, [the mesh replay](/quest/m2/multicast/mesh-replay.md), and [draining dialed links](/quest/m2/multicast/drain-dialed-links.md), and on [route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring -the lite-06 mitigations. After the cut come warmth's wire -([cache-aware routing](/quest/m3/cache-routing/README.md)), -[route health](/quest/m3/route-health/README.md), and +the lite-06 mitigations. It does not wait on +[warmth's port](/quest/m3/cache-routing/select.md): its wire joins lite-07 +only if the port has landed and passed the mesh replay when the cut is made, +and otherwise moves to the first wip version after it (maintainer, +2026-11-19). The port is measured and not yet built, pending the maintainer +(the coordinating session, under the maintainer's delegation, 2026-11-19). +After the cut come +[route health](/quest/m3/route-health/README.md) and [outbound federation](/quest/m3/cluster-routing/federation.md). Decided 2026-11-15 (maintainer, /quest-plan), after an audit of the lite-07 diff --git a/quest/m3/cache-routing/README.md b/quest/m3/cache-routing/README.md index c630d3ea93..23c2b5cb69 100644 --- a/quest/m3/cache-routing/README.md +++ b/quest/m3/cache-routing/README.md @@ -36,10 +36,11 @@ a node that says it carries a broadcast). What stands and what goes: - ✅ A pull moves to a cheaper warm copy only from a reader ranked below its own: ranked readers, which a TLA+ model shows settle (maintainer, 2026-11-17, [moq-dev/moq#8986](https://github.com/moq-dev/moq/pull/8986); - rejected: settled pulls). Its port started 2026-11-19, and its wire - joins lite-07 only if the port is ready when the cut's other blockers - clear (maintainer, 2026-11-19); otherwise it lands in the first wip - version after the cut. + rejected: settled pulls). Its wire joins lite-07 only if the port is + ready when the cut's other blockers clear (maintainer, 2026-11-19); + otherwise it lands in the first wip version after the cut. The port is + measured and not yet built, pending the maintainer (the coordinating + session, under the maintainer's delegation, 2026-11-19). - The non-goal of "one copy across a multi-hop sparse overlay rather than among direct neighbors" is dropped: a mesh is the product this serves. diff --git a/quest/m3/cache-routing/select.md b/quest/m3/cache-routing/select.md index 2a3bf2b82c..f1ddef8de0 100644 --- a/quest/m3/cache-routing/select.md +++ b/quest/m3/cache-routing/select.md @@ -63,13 +63,71 @@ reason for the change: a deployment expects warm pulls to cut what its backbone carries by a large part, and a release is the only way its production mesh gets them. -First, before building: measure that saving. The simulator's shapes are -generated, and against path joins the saving runs from under 1% on random -region meshes to 8% on tiers and 70% on rings (the table below), so it -depends on the graph. Let `warmth.rs` read a mesh from a file (relays, links -with costs and marks, and where readers and publishers sit), run ranked -readers and path joins on it, and report link-groups carried by each. A -deployment's graph is private: take the file by path and commit none. +Decided 2026-11-19 (the coordinating session, to which the maintainer +delegated the call, on +[moq-dev/moq#9157](https://github.com/moq-dev/moq/pull/9157#issuecomment-7005346052)): +✅ **the measurement lands as the port's first step, and the port is not +built until the maintainer answers the item under Waiting** (rejected: +building the port now regardless; deciding there to drop or defer it). + +Measured 2026-11-18, before building. `warmth.rs` reads a +mesh from a file (relays, links with costs, marks, and a label, and where +publishers and readers sit) and `benchmark_warmth_mesh` reports the +link-groups ranked readers and path joins each put on it, in all, per label, +and per link. A deployment's graph is private: pass its file by path as +`MOQ_WARMTH_MESH` and commit none. `warmth.mesh` beside it is the example. + +A tree feeding the same readers spans as many links under either rule, give +or take a relay carrying for others, so the saving is in which links carry +it. Generated meshes, five seeds each summed, every one settled right under +both rules, path joins / ranked readers: + +| mesh | backbone link-groups | backbone cold cost | every link-group | control messages | +|---|---|---|---|---| +| CDN, full core mesh, equal uplinks | 95 / 95 | 1048 / 760 (-27%) | 339 / 339 | 718 / 2902 | +| CDN, full core mesh, unequal uplinks | 125 / 90 (-28%) | 1408 / 736 (-48%) | 368 / 368 | 776 / 3193 | +| CDN, ring of regions, equal uplinks | 97 / 95 (-2%) | 776 / 760 (-2%) | 341 / 339 | 722 / 2234 | +| CDN, ring of regions, unequal uplinks | 134 / 94 (-30%) | 1072 / 752 (-30%) | 377 / 372 | 794 / 2735 | +| CDN, one hub region, equal uplinks | 97 / 97 | 1088 / 1088 | 341 / 341 | 722 / 1914 | +| CDN, one hub region, unequal uplinks | 128 / 93 (-27%) | 1432 / 1052 (-27%) | 371 / 371 | 782 / 2482 | +| 8 relays with an uplink each, peered every pair | 71 / 20 (-72%) | 710 / 200 (-72%) | 71 / 71 | 182 / 1236 | +| 8 relays with an uplink each, peered in a ring | 71 / 42 (-41%) | 710 / 420 (-41%) | 71 / 71 | 182 / 508 | + +A CDN here is six regions on a circle, two cores each, and four edges a +region marking both cores upstream; the backbone is the links between +regions, priced 4 plus 4 per step around the circle, warm a third. Four +broadcasts are published at random edges and each read by a random half of +the others. Equal uplinks cost 3, unequal ones 3 or 4 at random. In the last +two rows the backbone is the uplinks to the publisher's relay (10 cold, 4 +warm, peer links 2), and half the relays read each of four broadcasts. + +What it shows: + +- With equal uplinks a region's edges already land on one core by the + rendezvous hash, the path joins already share the shortest-path tree, and + warmth saves no backbone link-group: 0% to 2%. +- On a full core mesh it carries as many link-groups over shorter links: a + far region pulls from a nearer warm one instead of straight from the + publisher's, 27% off the cost where a link is priced by its length and + nothing where every backbone link costs the same. Each such pull adds a + relay hop. +- With unequal uplinks a region's edges split over its cores and each core + pulls its own copy: 27% to 30% fewer backbone link-groups. Pricing each + edge's uplinks equally gets the same tree with no wire. +- Relays with an uplink each and links between them, the drone meshes, save + 41% to 72%, the whole of what item 9 of the + [epic](/quest/m2/multicast/README.md) asked for. +- It costs 2.6 to 6.8 times the control messages. + +On a deployment's graph, measured by path and not committed: under its +hop-count link costs, ranked readers cut backbone link-groups 4% against +path joins. With latency link costs and a warm price at a third of the cold, +they cut 20% (39% of latency-weighted backbone cost, 46% of intercontinental +link-groups), at +7 to 21% mean reader latency and up to 8.7x the control +messages. With latency costs and no warm discount, 2%. Latency link costs +alone, with path joins, remove a 5 to 12% mean reader-latency stretch over +the shortest path, with no wire change; the relay's cluster config already +prices each peer link (`cost`). **Why ranked readers settle.** A relay's choice reads only copies feeding lower ranks, and what those feed never depends on a higher rank's choice. @@ -432,6 +490,10 @@ later step, a query naming an Instance, answered once with an offer, path, mark, and rank or none. A rank is a route cost and a Node ID. Nothing lite-07 freezes changes. +## Waiting + +- Maintainer: whether to build the port. Its saving on the deployment's graph depends on a warm discount that trades reader latency for backbone: 20% of backbone link-groups at +7 to 21% mean reader latency with a warm price at a third of the cold, and 2% to 4% without one. Options: (1) adopt latency link costs on the deployment first, which needs no change here, and build the port after the lite-07 cut only if that latency trade is accepted, its wire then in the first wip version after the cut, recommended, since the costs alone remove a 5 to 12% reader-latency stretch and the port's saving is the trade itself; (2) build the port now for lite-07, at the risk of rushing an XL wire into the cut for a trade not yet accepted; (3) drop the port. An agent cannot take the step: it is a product trade between reader latency and backbone cost, and no check or measurement decides it. Done when the answer is recorded in this Plan. + ## Related - [Real relays replay a mesh and its faults](/quest/m2/multicast/mesh-replay.md) - the port runs through it before its wire may join lite-07 diff --git a/rs/moq-net/src/route/tests/warm.rs b/rs/moq-net/src/route/tests/warm.rs index 09ef9efbc9..e1e514c3e4 100644 --- a/rs/moq-net/src/route/tests/warm.rs +++ b/rs/moq-net/src/route/tests/warm.rs @@ -486,7 +486,7 @@ fn close(streams: &mut BTreeMap>, peer: Node, stream: u64) { } } -fn pair(a: Node, b: Node) -> (Node, Node) { +pub(super) fn pair(a: Node, b: Node) -> (Node, Node) { (a.min(b), a.max(b)) } @@ -621,8 +621,9 @@ impl WarmSim { self.net.link(node, home, 1, Tie::Peer); self.net.publish(node, id); (0..claims) - .map(|claim| { - let path = format!("live/{claim}"); + .map(|_| { + // Numbered across publishers, so no two announce one path. + let path = format!("live/{}", self.casts.len()); self.net.claim(node, &path, true); self.casts.push((id, p(&path))); self.casts.len() - 1 @@ -1531,7 +1532,7 @@ impl WarmSim { carry.since = reading.then_some(round); // A reader at a relay already fed has its first frame at once, whether or not // anything is sent. - if reading && self.fed(node, cast) && self.live(node, cast).is_some() { + if reading && self.feeds(node, cast) { self.entry(node, cast).since = None; self.waits.push(0); } @@ -1752,6 +1753,11 @@ impl WarmSim { // --- What is true --- + /// Whether `node` is fed `cast` from a publisher still live. + pub fn feeds(&self, node: Node, cast: usize) -> bool { + self.fed(node, cast) && self.live(node, cast).is_some() + } + /// Whether `node` is fed `cast`: its pulls lead to the publisher, each received. pub fn fed(&self, node: Node, cast: usize) -> bool { let mut at = node; @@ -1885,7 +1891,7 @@ impl WarmSim { let round = self.round; let mut reach: BTreeMap> = BTreeMap::new(); for (node, cast) in self.readers() { - let fed = self.fed(node, cast) && self.live(node, cast).is_some(); + let fed = self.feeds(node, cast); let start = self.endpoints[&self.casts[cast].0]; let path = reach .entry(start) @@ -1916,10 +1922,7 @@ impl WarmSim { if self.circles(node, cast) { return Err(format!("{}: broadcast {cast} circles at relay {node}", self.arm.name())); } - if self.reading(node, cast) - && reach.contains(&node) - && !(self.fed(node, cast) && self.live(node, cast).is_some()) - { + if self.reading(node, cast) && reach.contains(&node) && !self.feeds(node, cast) { return Err(format!("{}: relay {node} unfed for broadcast {cast}", self.arm.name())); } let carry = self.carry(node, cast); diff --git a/rs/moq-net/src/route/tests/warmth.mesh b/rs/moq-net/src/route/tests/warmth.mesh new file mode 100644 index 0000000000..56990bad7b --- /dev/null +++ b/rs/moq-net/src/route/tests/warmth.mesh @@ -0,0 +1,26 @@ +# An example mesh for `benchmark_warmth_mesh` in `warmth.rs`, which documents the format. + +relay origin near middle far pop edge-a edge-b + +# Cores on a line, with long links from the origin that pass over the cores between, +# labelled so the report totals them. +link origin near 6 1 peer +link near middle 4 1 peer +link middle far 4 1 peer +link origin middle 6 6 peer backbone +link origin far 8 8 peer backbone + +# A pop beside the far core, with its own long link. +link far pop 2 1 peer +link origin pop 9 4 peer backbone + +# Edges mark their cores upstream. +link edge-a far 1 1 up +link edge-a middle 2 1 up +link edge-b pop 1 1 up + +publish live origin +publish back edge-b + +read live far near middle pop edge-a edge-b +read back origin edge-a near diff --git a/rs/moq-net/src/route/tests/warmth.rs b/rs/moq-net/src/route/tests/warmth.rs index b5cf594c1c..33864731e3 100644 --- a/rs/moq-net/src/route/tests/warmth.rs +++ b/rs/moq-net/src/route/tests/warmth.rs @@ -6,6 +6,7 @@ //! costs against the cheapest, how many copies leave the publisher's relay, the //! messages and rounds it took, and whether pulls ever formed a cycle or left a reader //! unfed. The ignored benchmarks print the tables; the tests pin what they show. +//! [`benchmark_warmth_mesh`] measures a mesh read from a file instead. use super::multi::Tie; use super::warm::*; @@ -1479,3 +1480,305 @@ fn ranked_pulls_settle_random_case_503() { } assert_eq!(churn(NOT_DOWN_QUERY, 503..504, random).failed, 1); } + +// --- A mesh from a file --- + +/// A mesh read from a file, one statement per line, `#` starting a comment: +/// +/// - `relay ...` names relays. +/// - `link [