From abd19bd2c8745d76d681baed2b0a620b1fba8e3c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 13:51:18 -0700 Subject: [PATCH 1/5] quest: claim cross-node-flapping From 792e839f6c68d5967d923ea38b106b203cbf6c0b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 14:35:13 -0700 Subject: [PATCH 2/5] test(net): pin cross-node close flapping and its residuals A client publishing on one relay closes a broadcast, and a client watching from another relay must see one retraction. The drill reproduces #4349's retract/announce/retract on the release before the retraction quarantine and passes on main; a zero-hold control shows it still detects the toggle. Residual counterexamples pin what main does today: a stale path outlasting the 300 ms hold on a six-relay ring, a revival the quarantine misses on nine relays, and 179 hunting relays (3 reviving) over 200 seeded random meshes. The route rewrite must bring each to a single retraction. Drains now wait out the two-second quarantine, so a late revival is seen. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 - quest/m1/cluster-routing/README.md | 1 - quest/m1/cluster-routing/handoff.md | 1 - quest/m1/cluster-routing/routes.md | 12 +- quest/m1/cross-node-flapping.md | 53 ------ rs/moq-net/tests/mesh_withdraw.rs | 273 +++++++++++++++++++++++++++- 6 files changed, 274 insertions(+), 67 deletions(-) delete mode 100644 quest/m1/cross-node-flapping.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 80e8c0927c2..61188ee6fc4 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -62,7 +62,6 @@ above them. - [qmux credit](/quest/m0/qmux-credit.md) - dropped/stopped streams return credit and close is delivered - [Restart followers](/quest/m1/restart-followers.md) - first-party consumers follow an epochless `Restart`, from a GOAWAY drain or a publisher replacement, by resolving the path again -- [Cross-node flapping](/quest/m1/cross-node-flapping.md) - incident-sensitive baseline and residual stale-revival counterexamples before the route rewrite - [Cluster routing](/quest/m1/cluster-routing/README.md) - prove shared routing contracts, implement per-origin routes, and connect outbound local clusters - [P2P](/quest/m1/p2p/README.md) - application-selected peer sessions use shared priorities, routing, and same-epoch handover - [Routing guide](/quest/m1/routing-guide.md) - concepts and practical outbound federation/P2P setup diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index dc8648e871b..1b2bbd7995c 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -94,7 +94,6 @@ setup after the capabilities land. - [Route health](/quest/m2/route-health/README.md) - epoch-only active/passive delivery monitoring and automatic recovery - [P2P](/quest/m1/p2p/README.md) - application-selected peer sessions feed the shared routing model - [Routing guide](/quest/m1/routing-guide.md) - concept page and practical federation/P2P setup -- [Cross-node flapping](/quest/m1/cross-node-flapping.md) - incident-sensitive regressions for residual withdrawal hunting - [Route wakes](/quest/m2/route-wakes.md) - local work follows affected fronts rather than the whole covered table - [Link quality](/quest/m2/link-quality.md) - optional measured link cost, distinct from delivery-health failover - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - independent origins serving the same explicit epoch diff --git a/quest/m1/cluster-routing/handoff.md b/quest/m1/cluster-routing/handoff.md index 6df7268763a..27a42b22be0 100644 --- a/quest/m1/cluster-routing/handoff.md +++ b/quest/m1/cluster-routing/handoff.md @@ -55,5 +55,4 @@ bounded seqno-request attempt ends and no further unsolicited update arrives. ## Related -- [Cross-node flapping](/quest/m1/cross-node-flapping.md) - residual stale-revival fixtures - [Lazy discovery](/quest/m1/lazy-discovery/README.md) - interest changes must not erase freshness protection diff --git a/quest/m1/cluster-routing/routes.md b/quest/m1/cluster-routing/routes.md index fa101f5b882..90bb65e3da7 100644 --- a/quest/m1/cluster-routing/routes.md +++ b/quest/m1/cluster-routing/routes.md @@ -176,6 +176,15 @@ Open, for the implementer to settle and record: with deterministic acceptance tests before the wire is written. - How much of today's route trie, fronts, and `route_order` survives intact. +The #4349 counterexamples are the `residual_*` tests in +`rs/moq-net/tests/mesh_withdraw.rs`. They pin today's hunting on lite-06: +a stale path outlasting the 300 ms hold, a revival the quarantine misses, +and 179 hunting relays (3 reviving) over 200 seeded meshes. Turn each into a +single retraction everywhere, and keep `cross_node_close_retracts_once` +passing. On `main` an epochless switch to a stale path is a Restart, which a +lite-06 client sees as a retraction and an announcement; the release sent an +in-place update. + Tests, time mocked: #4644's live-graph withdrawal and failover tests (`quest/m0/path-hunting` branch) as acceptance, run on the wip version; a link flap with many broadcasts behind one origin costs one ROUTE change per @@ -219,11 +228,10 @@ release. - [Routing contracts](/quest/m1/cluster-routing/contracts.md) - authenticated incarnation, local priorities, forwarding progress, and export views - [Next-hop handoff](/quest/m1/cluster-routing/handoff.md) - complete and fresh claims with bounded state -- [Cross-node flapping](/quest/m1/cross-node-flapping.md) - incident-sensitive baseline and residual counterexamples ## Closes -- [#4349](https://github.com/moq-dev/moq/issues/4349) - close once the residual cross-node flapping counterexamples pass +- [#4349](https://github.com/moq-dev/moq/issues/4349) - close once the `residual_*` tests in `mesh_withdraw.rs` retract once everywhere ## Related diff --git a/quest/m1/cross-node-flapping.md b/quest/m1/cross-node-flapping.md deleted file mode 100644 index 04be7ce47c9..00000000000 --- a/quest/m1/cross-node-flapping.md +++ /dev/null @@ -1,53 +0,0 @@ -# [M] Reproduce cross-node announce flapping - -## Goal - -A time-mocked drill with three or more clustered relays, extending -`rs/moq-net/tests/mesh_withdraw.rs` (which already checks one retraction on a -34-node partial mesh) or added beside it: a publisher on one node cleanly closes a broadcast, and a client -watching its prefix on another node sees one retraction, with no -re-announce, for that path. The drill runs against `main` and lands as a -regression test. - -Reported in [#4349](https://github.com/moq-dev/moq/issues/4349) against the -release on cdn.moq.pro (2026-10-11): after a clean close, the subscriber's -announce stream saw the path Announced/Retracted about 1140 times over about -2 s, and subscribes meanwhile answered `internal error` or `unroutable`. -It never happened on one node or on a self-hosted two-relay 0.17.2 cluster. -Loss and FETCH were clean in every run (about 31k FETCHes, all ok). - -## Plan - -- Updated 2026-10-13: #5375's two-second same-publisher retraction quarantine - and #5391's repeated-retraction queue dedup landed. #5375 reports 1,042 - revived broadcasts reduced to 15 over 200 randomized meshes; the #5376 - release backport reports 457 reduced to 2. These are mitigation results, - not proof that main is clean. Existing mesh_withdraw tests include the - eight-node regression. Preserve those and reduce residual failures to - deterministic public fixtures, with pre-fix/release positive controls. - Git backports establish release inclusion, not a verified deployment build. -- The stale-revival mechanism investigated in #5375: path hunting in today's lite-06 path - vector. A peer's re-announce of a withdrawn prefix clears its withdrawal - (`reannounced` in `rs/moq-net/src/model/origin.rs`) and revives stale - alternate routes through it, so each node's late withdrawal toggles the - path again. A two-relay cluster has no alternates, which fits the report. - #4642's 300 ms hold did not cover fresh retract/announce toggles. The newer - quarantine holds strictly worse stale alternatives, leaving equal/better - stale ranks. Do not lengthen the hold to declare the issue solved. -- Land the incident-sensitive baseline and residual fixtures before the route - rewrite, without requiring a redesign here. Routes and announces owns - closing #4349, so this quest can complete with the counterexample harness - rather than depending on the implementation it gates. A passing single topology is not enough to close - the residual concern. Fix a small independent root cause only if one is found. -- Not planned here: the reporter's self-hosted two-relay rejoin stall - (`subscribe canceled (idle)`, then `old`, then no state). On 0.17.2 a - rejoin could read a stale cached group as the live edge, which #4741 and - #4914 changed on `main`; revisit it if their re-run on a release - carrying those still stalls. -- Harness for reference: https://github.com/fperex/moq-burst-repro (published - crates only); its traces hold per-session announcement sequences. - -## Related - -- [Routes and announces](/quest/m1/cluster-routing/routes.md) - the lite-07 redesign that must pass the residual counterexamples -- [Dropped sources](/quest/m1/dropped-sources.md) - the `internal error` replies seen during the same churn diff --git a/rs/moq-net/tests/mesh_withdraw.rs b/rs/moq-net/tests/mesh_withdraw.rs index 639c7f109ef..9fb92c22c02 100644 --- a/rs/moq-net/tests/mesh_withdraw.rs +++ b/rs/moq-net/tests/mesh_withdraw.rs @@ -12,7 +12,7 @@ use std::{collections::HashMap, time::Duration}; use futures::{StreamExt, channel::mpsc}; use moq_net::{Hop, Version, announce, broadcast, origin}; -use support::harness::{MockPair, peer, peer_with_latency}; +use support::harness::{MockConnectOptions, MockPair, connect_mock, peer, peer_with_latency}; fn produce_origin(hop: u64) -> origin::Producer { let (producer, driver) = origin::Producer::new(origin::Config::new(Hop::new(hop).unwrap())); @@ -30,10 +30,11 @@ enum Kind { } /// Every update per prefix until the watcher goes quiet. Time is paused, so the -/// timeout fires only once every task is idle. +/// timeout fires only once every task is idle. Quiet means longer than a +/// retraction's two-second quarantine, which could release a stale path late. async fn drain(watched: &mut mpsc::UnboundedReceiver<(String, Kind)>) -> HashMap> { let mut updates = HashMap::>::new(); - while let Ok(Some((prefix, kind))) = moq_net_sim::timeout(Duration::from_secs(1), watched.next()).await { + while let Ok(Some((prefix, kind))) = moq_net_sim::timeout(Duration::from_secs(3), watched.next()).await { updates.entry(prefix).or_default().push(kind); } updates @@ -232,21 +233,27 @@ async fn partial_mesh_withdraw_retracts_once() { } } +/// A link between two relays, by index, and its one-way latency in ms. +type Link = ((usize, usize), u64); + +/// Eight relays where, without the retraction quarantine, relays 1 and 5 +/// retract a broadcast published on relay 7 and then announce it again. +#[rustfmt::skip] +const STALE_PATHS: &[Link] = &[ + ((0, 1), 35), ((1, 2), 90), ((0, 3), 129), ((2, 4), 140), ((1, 5), 137), ((4, 6), 148), + ((2, 7), 30), ((3, 5), 124), ((1, 4), 111), ((0, 5), 21), ((3, 4), 34), +]; + /// A relay that already retracted a withdrawn broadcast must not announce it /// again. Its neighbors, two or more hops from the publisher's relay, still hunt /// through stale paths; one that falls back to a path not through this relay /// offers it as a new route after the relay forgot the withdrawal. #[moq_net_sim::test] async fn retracted_relay_ignores_a_stale_path() { - #[rustfmt::skip] - const EDGES: &[((usize, usize), u64)] = &[ - ((0, 1), 35), ((1, 2), 90), ((0, 3), 129), ((2, 4), 140), ((1, 5), 137), ((4, 6), 148), - ((2, 7), 30), ((3, 5), 124), ((1, 4), 111), ((0, 5), 21), ((3, 4), 34), - ]; let version: Version = "moq-lite-06".parse().unwrap(); let nodes: Vec<_> = (1..=8).map(produce_origin).collect(); let mut pairs = Vec::new(); - for &((a, b), ms) in EDGES { + for &((a, b), ms) in STALE_PATHS { pairs.push(peer_with_latency(version, &nodes[a], &nodes[b], Duration::from_millis(ms)).await); } let mut watched: Vec<_> = nodes.iter().map(|node| watch(node.consume().announced())).collect(); @@ -268,3 +275,251 @@ async fn retracted_relay_ignores_a_stale_path() { } assert!(revived.is_empty(), "{revived:?}"); } + +/// Relays peered over `edges`, each +/// holding route updates for `hold`, with a client publishing on relay +/// `publisher` and a client watching on relay `watcher`. Every relay is watched +/// too. +/// +/// The watching client holds nothing, so it reports every toggle its relay +/// sends, as a browser's announce stream does. The broadcast has no epoch, so a +/// relay moving to another path restarts it, which lite-06 sends as a +/// retraction then an announcement. +struct Cluster { + _pairs: Vec, + publisher: origin::Producer, + client: mpsc::UnboundedReceiver<(String, Kind)>, + relays: Vec>, +} + +impl Cluster { + async fn new(version: &str, edges: &[Link], hold: Duration, publisher: usize, watcher: usize) -> Self { + let version: Version = version.parse().unwrap(); + let produce = |id: u64, hold: Duration| { + let mut config = origin::Config::new(Hop::new(id).unwrap()); + config.update_hold = hold; + let (producer, driver) = origin::Producer::new(config); + support::harness::spawn(driver); + producer + }; + let n = edges.iter().map(|&((a, b), _)| a.max(b) + 1).max().unwrap(); + let relays: Vec<_> = (1..=n as u64).map(|id| produce(id, hold)).collect(); + let mut pairs = Vec::new(); + for &((a, b), ms) in edges { + pairs.push(peer_with_latency(version, &relays[a], &relays[b], Duration::from_millis(ms)).await); + } + + let publishing = produce(1000, Duration::ZERO); + let mut options = MockConnectOptions::new(version); + options.client_publish = Some(publishing.consume()); + options.server_subscribe = Some(relays[publisher].clone()); + pairs.push(connect_mock(options).await); + + let watching = produce(1001, Duration::ZERO); + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(relays[watcher].consume()); + options.client_subscribe = Some(watching.clone()); + pairs.push(connect_mock(options).await); + + let mut cluster = Self { + _pairs: pairs, + publisher: publishing, + client: watch(watching.consume().announced()), + relays: relays.iter().map(|relay| watch(relay.consume().announced())).collect(), + }; + // Let every session's announce interest land before anything is published. + assert!(drain(&mut cluster.client).await.is_empty()); + cluster + } + + /// Publish `path` from the client and require everyone to see it. Returns + /// what the watching client saw. + async fn publish(&mut self, path: &str) -> (broadcast::Producer, Vec) { + let broadcast = self.publisher.create_broadcast(path).unwrap(); + broadcast.announce(Default::default()).unwrap(); + let client = drain(&mut self.client).await.remove(path).unwrap(); + assert_ne!(client.last(), Some(&Kind::End), "client: {client:?}"); + for (relay, watched) in self.relays.iter_mut().enumerate() { + let kinds = drain(watched).await.remove(path).unwrap(); + assert_ne!(kinds.last(), Some(&Kind::End), "relay {relay}: {kinds:?}"); + } + (broadcast, client) + } + + /// Close `broadcast` cleanly: what the watching client, then each relay, saw. + async fn close(&mut self, path: &str, broadcast: broadcast::Producer) -> (Vec, Vec>) { + drop(broadcast); + let client = drain(&mut self.client).await.remove(path).unwrap(); + let mut relays = Vec::new(); + for watched in self.relays.iter_mut() { + relays.push(drain(watched).await.remove(path).unwrap()); + } + (client, relays) + } +} + +/// #4349: a publisher on one relay cleanly closes a broadcast, and a client +/// watching from another relay sees one retraction and no re-announce. On the +/// release before the retraction quarantine (#5376), this client saw it +/// retracted, announced, and retracted again. Relay 0 still restarts onto a +/// stale path after the hold; see [`residual_slow_ring_reannounces`]. +#[moq_net_sim::test] +async fn cross_node_close_retracts_once_lite06() { + cross_node_close_retracts_once("moq-lite-06").await; +} + +#[moq_net_sim::test] +async fn cross_node_close_retracts_once_lite07() { + cross_node_close_retracts_once("moq-lite-07-wip").await; +} + +async fn cross_node_close_retracts_once(version: &str) { + let mut cluster = Cluster::new(version, STALE_PATHS, origin::DEFAULT_UPDATE_HOLD, 7, 1).await; + for round in 0..3 { + let (broadcast, client) = cluster.publish("room/a").await; + assert_eq!(client, [Kind::Start], "round {round}"); + let (client, relays) = cluster.close("room/a", broadcast).await; + assert_eq!(client, [Kind::End], "round {round}"); + for (relay, kinds) in relays.iter().enumerate() { + assert_eq!( + kinds.last(), + Some(&Kind::End), + "round {round}, relay {relay}: {kinds:?}" + ); + assert!( + !kinds.windows(2).any(|w| w == [Kind::End, Kind::Start]), + "round {round}, relay {relay}: {kinds:?}" + ); + } + } +} + +/// The positive control for [`cross_node_close_retracts_once`]: without the +/// update hold and the quarantine, the client sees the broadcast come back +/// before it ends. +#[moq_net_sim::test] +async fn cross_node_close_without_hold_reannounces() { + let mut cluster = Cluster::new("moq-lite-06", STALE_PATHS, Duration::ZERO, 7, 1).await; + let (broadcast, _) = cluster.publish("room/a").await; + let (client, _) = cluster.close("room/a", broadcast).await; + assert_eq!(client, [Kind::End, Kind::Start, Kind::End]); +} + +// Residual counterexamples on `main`, with the hold and the retraction +// quarantine. Each pins what the cluster does today, so a change either way +// fails here. Routes and announces (quest/m1/cluster-routing/routes.md) must +// bring each to a single retraction everywhere. + +/// A ring of six relays with links under 150 ms. Relay 3 hears the withdrawal +/// from relay 2 at 84 ms, but relay 5's equal-cost copy, the long way round, +/// stays live until 446 ms. That outlasts the hold, so relay 3 restarts onto +/// the stale copy and its client sees the broadcast come back. +#[moq_net_sim::test] +async fn residual_slow_ring_reannounces() { + #[rustfmt::skip] + const EDGES: &[Link] = &[ + ((0, 1), 141), ((0, 2), 55), ((2, 3), 29), ((1, 4), 135), ((4, 5), 148), ((3, 5), 22), + ]; + use Kind::*; + let mut cluster = Cluster::new("moq-lite-06", EDGES, origin::DEFAULT_UPDATE_HOLD, 0, 3).await; + let (broadcast, _) = cluster.publish("room/a").await; + let (client, relays) = cluster.close("room/a", broadcast).await; + assert_eq!(client, [End, Start, End]); + let relays: Vec<&[Kind]> = relays.iter().map(Vec::as_slice).collect(); + assert_eq!(relays, [&[End][..], &[End], &[End], &[Restart, End], &[End], &[End]]); +} + +/// Nine relays, reduced from a seeded random mesh with [`random_mesh`]'s +/// links. Relays 2 and 8 retract the broadcast and then announce it again: a +/// revival the quarantine misses. +#[moq_net_sim::test] +async fn residual_mesh_revives() { + #[rustfmt::skip] + const EDGES: &[Link] = &[ + ((3, 4), 10), ((3, 5), 12), ((1, 6), 149), ((0, 7), 73), ((6, 8), 20), ((1, 4), 26), + ((0, 6), 121), ((3, 7), 81), ((0, 2), 96), ((2, 8), 93), ((5, 8), 89), + ]; + use Kind::*; + let mut cluster = Cluster::new("moq-lite-06", EDGES, origin::DEFAULT_UPDATE_HOLD, 4, 8).await; + let (broadcast, _) = cluster.publish("room/a").await; + let (client, relays) = cluster.close("room/a", broadcast).await; + assert_eq!(client, [End, Start, End, Start, End]); + let relays: Vec<&[Kind]> = relays.iter().map(Vec::as_slice).collect(); + assert_eq!( + relays, + [ + &[Restart, End][..], + &[End], + &[End, Start, End], + &[End], + &[End], + &[Restart, End], + &[End], + &[Restart, End], + &[Restart, End, Start, End], + ] + ); +} + +/// splitmix64, so every platform builds the same meshes. +struct Rng(u64); + +impl Rng { + fn next(&mut self) -> u64 { + self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15); + let mut z = self.0; + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + z ^ (z >> 31) + } + + /// A value in `lo..=hi`. + fn range(&mut self, lo: u64, hi: u64) -> u64 { + lo + self.next() % (hi - lo + 1) + } +} + +/// A connected mesh of 8 to 17 relays (a random spanning tree plus up to as +/// many extra links), each link 1 to 150 ms, and the relay publishing on it. +fn random_mesh(seed: u64) -> (Vec, usize) { + let mut rng = Rng(seed); + let n = rng.range(8, 17) as usize; + let mut edges = Vec::new(); + let mut linked = std::collections::HashSet::new(); + for b in 1..n { + let a = rng.range(0, b as u64 - 1) as usize; + linked.insert((a, b)); + edges.push(((a, b), rng.range(1, 150))); + } + for _ in 0..rng.range(0, n as u64) { + let a = rng.range(0, n as u64 - 1) as usize; + let b = rng.range(0, n as u64 - 1) as usize; + let (a, b) = (a.min(b), a.max(b)); + if a != b && linked.insert((a, b)) { + edges.push(((a, b), rng.range(1, 150))); + } + } + (edges, rng.range(0, n as u64 - 1) as usize) +} + +/// One clean close on each of 200 seeded random meshes: how many relays +/// announce the broadcast again before their last retraction (by restarting +/// onto a stale path, or by reviving it after retracting), and how many of +/// those revive. Pinned at today's counts; Routes and announces brings both +/// to zero. +#[moq_net_sim::test] +async fn residual_random_meshes() { + let (mut hunted, mut revived) = (0, 0); + for seed in 0..200 { + let (edges, publisher) = random_mesh(seed); + let mut cluster = Cluster::new("moq-lite-06", &edges, origin::DEFAULT_UPDATE_HOLD, publisher, 0).await; + let (broadcast, _) = cluster.publish("room/a").await; + let (_, relays) = cluster.close("room/a", broadcast).await; + for kinds in relays { + assert_eq!(kinds.last(), Some(&Kind::End), "seed {seed}: {kinds:?}"); + hunted += usize::from(kinds.len() > 1); + revived += usize::from(kinds.windows(2).any(|w| w == [Kind::End, Kind::Start])); + } + } + assert_eq!((hunted, revived), (179, 3)); +} From a8146b324d374f13a8a4d4ee399d7403dc20cbae Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 14:39:43 -0700 Subject: [PATCH 3/5] quest: require a fixture for the publish-time Restart toggle Co-Authored-By: Claude Opus 5.5 --- quest/m1/cluster-routing/routes.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/quest/m1/cluster-routing/routes.md b/quest/m1/cluster-routing/routes.md index 90bb65e3da7..386a7d031b1 100644 --- a/quest/m1/cluster-routing/routes.md +++ b/quest/m1/cluster-routing/routes.md @@ -185,6 +185,12 @@ passing. On `main` an epochless switch to a stale path is a Restart, which a lite-06 client sees as a retraction and an announcement; the release sent an in-place update. +Publishing toggles too, decided 2026-10-13: add a time-mocked fixture where, +on some rings, a longer path arrives first and the better route then replaces +it as a Restart, so a lite-06 client sees `Start, End, Start` while the +broadcast stays live. Pin today's sequence beside the `residual_*` tests; +the epochless Restart rule above decides what it becomes. + Tests, time mocked: #4644's live-graph withdrawal and failover tests (`quest/m0/path-hunting` branch) as acceptance, run on the wip version; a link flap with many broadcasts behind one origin costs one ROUTE change per From 1778ded00efd6a0e6e34bd1942911ca22db38497 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 14:53:30 -0700 Subject: [PATCH 4/5] test(net): type link latency as Duration and pin the sweep's extra updates Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/tests/mesh_withdraw.rs | 41 +++++++++++++++++-------------- 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/rs/moq-net/tests/mesh_withdraw.rs b/rs/moq-net/tests/mesh_withdraw.rs index 9fb92c22c02..7870c2c7a54 100644 --- a/rs/moq-net/tests/mesh_withdraw.rs +++ b/rs/moq-net/tests/mesh_withdraw.rs @@ -233,15 +233,19 @@ async fn partial_mesh_withdraw_retracts_once() { } } -/// A link between two relays, by index, and its one-way latency in ms. -type Link = ((usize, usize), u64); +/// A link between two relays, by index, and its one-way latency. +type Link = ((usize, usize), Duration); + +const fn ms(millis: u64) -> Duration { + Duration::from_millis(millis) +} /// Eight relays where, without the retraction quarantine, relays 1 and 5 /// retract a broadcast published on relay 7 and then announce it again. #[rustfmt::skip] const STALE_PATHS: &[Link] = &[ - ((0, 1), 35), ((1, 2), 90), ((0, 3), 129), ((2, 4), 140), ((1, 5), 137), ((4, 6), 148), - ((2, 7), 30), ((3, 5), 124), ((1, 4), 111), ((0, 5), 21), ((3, 4), 34), + ((0, 1), ms(35)), ((1, 2), ms(90)), ((0, 3), ms(129)), ((2, 4), ms(140)), ((1, 5), ms(137)), ((4, 6), ms(148)), + ((2, 7), ms(30)), ((3, 5), ms(124)), ((1, 4), ms(111)), ((0, 5), ms(21)), ((3, 4), ms(34)), ]; /// A relay that already retracted a withdrawn broadcast must not announce it @@ -253,8 +257,8 @@ async fn retracted_relay_ignores_a_stale_path() { let version: Version = "moq-lite-06".parse().unwrap(); let nodes: Vec<_> = (1..=8).map(produce_origin).collect(); let mut pairs = Vec::new(); - for &((a, b), ms) in STALE_PATHS { - pairs.push(peer_with_latency(version, &nodes[a], &nodes[b], Duration::from_millis(ms)).await); + for &((a, b), latency) in STALE_PATHS { + pairs.push(peer_with_latency(version, &nodes[a], &nodes[b], latency).await); } let mut watched: Vec<_> = nodes.iter().map(|node| watch(node.consume().announced())).collect(); @@ -305,8 +309,8 @@ impl Cluster { let n = edges.iter().map(|&((a, b), _)| a.max(b) + 1).max().unwrap(); let relays: Vec<_> = (1..=n as u64).map(|id| produce(id, hold)).collect(); let mut pairs = Vec::new(); - for &((a, b), ms) in edges { - pairs.push(peer_with_latency(version, &relays[a], &relays[b], Duration::from_millis(ms)).await); + for &((a, b), latency) in edges { + pairs.push(peer_with_latency(version, &relays[a], &relays[b], latency).await); } let publishing = produce(1000, Duration::ZERO); @@ -418,7 +422,7 @@ async fn cross_node_close_without_hold_reannounces() { async fn residual_slow_ring_reannounces() { #[rustfmt::skip] const EDGES: &[Link] = &[ - ((0, 1), 141), ((0, 2), 55), ((2, 3), 29), ((1, 4), 135), ((4, 5), 148), ((3, 5), 22), + ((0, 1), ms(141)), ((0, 2), ms(55)), ((2, 3), ms(29)), ((1, 4), ms(135)), ((4, 5), ms(148)), ((3, 5), ms(22)), ]; use Kind::*; let mut cluster = Cluster::new("moq-lite-06", EDGES, origin::DEFAULT_UPDATE_HOLD, 0, 3).await; @@ -436,8 +440,8 @@ async fn residual_slow_ring_reannounces() { async fn residual_mesh_revives() { #[rustfmt::skip] const EDGES: &[Link] = &[ - ((3, 4), 10), ((3, 5), 12), ((1, 6), 149), ((0, 7), 73), ((6, 8), 20), ((1, 4), 26), - ((0, 6), 121), ((3, 7), 81), ((0, 2), 96), ((2, 8), 93), ((5, 8), 89), + ((3, 4), ms(10)), ((3, 5), ms(12)), ((1, 6), ms(149)), ((0, 7), ms(73)), ((6, 8), ms(20)), ((1, 4), ms(26)), + ((0, 6), ms(121)), ((3, 7), ms(81)), ((0, 2), ms(96)), ((2, 8), ms(93)), ((5, 8), ms(89)), ]; use Kind::*; let mut cluster = Cluster::new("moq-lite-06", EDGES, origin::DEFAULT_UPDATE_HOLD, 4, 8).await; @@ -489,14 +493,14 @@ fn random_mesh(seed: u64) -> (Vec, usize) { for b in 1..n { let a = rng.range(0, b as u64 - 1) as usize; linked.insert((a, b)); - edges.push(((a, b), rng.range(1, 150))); + edges.push(((a, b), ms(rng.range(1, 150)))); } for _ in 0..rng.range(0, n as u64) { let a = rng.range(0, n as u64 - 1) as usize; let b = rng.range(0, n as u64 - 1) as usize; let (a, b) = (a.min(b), a.max(b)); if a != b && linked.insert((a, b)) { - edges.push(((a, b), rng.range(1, 150))); + edges.push(((a, b), ms(rng.range(1, 150)))); } } (edges, rng.range(0, n as u64 - 1) as usize) @@ -504,12 +508,12 @@ fn random_mesh(seed: u64) -> (Vec, usize) { /// One clean close on each of 200 seeded random meshes: how many relays /// announce the broadcast again before their last retraction (by restarting -/// onto a stale path, or by reviving it after retracting), and how many of -/// those revive. Pinned at today's counts; Routes and announces brings both -/// to zero. +/// onto a stale path, or by reviving it after retracting), how many of those +/// revive, and how many updates they send beyond the one retraction. Pinned at +/// today's counts; Routes and announces brings all three to zero. #[moq_net_sim::test] async fn residual_random_meshes() { - let (mut hunted, mut revived) = (0, 0); + let (mut hunted, mut revived, mut extra) = (0, 0, 0); for seed in 0..200 { let (edges, publisher) = random_mesh(seed); let mut cluster = Cluster::new("moq-lite-06", &edges, origin::DEFAULT_UPDATE_HOLD, publisher, 0).await; @@ -519,7 +523,8 @@ async fn residual_random_meshes() { assert_eq!(kinds.last(), Some(&Kind::End), "seed {seed}: {kinds:?}"); hunted += usize::from(kinds.len() > 1); revived += usize::from(kinds.windows(2).any(|w| w == [Kind::End, Kind::Start])); + extra += kinds.len() - 1; } } - assert_eq!((hunted, revived), (179, 3)); + assert_eq!((hunted, revived, extra), (179, 3, 192)); } From ae7bebacd9c8cda12ca563e16d3dd612c9d6c83b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 13 Oct 2026 19:47:38 -0700 Subject: [PATCH 5/5] test(net): settle the mesh on global idle instead of a quiet timeout moq_net_sim::idle completes once every task is parked and nothing is armed, the point the run loop used to report only as a deadlock. mesh_withdraw waits for it before draining any receiver, so a stale path released after a quarantine can no longer land after its relay was read. Test origins use an unbounded cache pool: the idle-expiry sweep would otherwise stay armed forever. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/sim/src/lib.rs | 61 +++++++++++++++++++++++----- rs/moq-net/tests/mesh_withdraw.rs | 66 ++++++++++++++++++------------- 2 files changed, 91 insertions(+), 36 deletions(-) diff --git a/rs/moq-net/sim/src/lib.rs b/rs/moq-net/sim/src/lib.rs index 91ae336ae1a..527316ebb91 100644 --- a/rs/moq-net/sim/src/lib.rs +++ b/rs/moq-net/sim/src/lib.rs @@ -3,13 +3,14 @@ //! Time never moves on its own. Once every task has stalled, [`run`] jumps the clock //! to the earliest armed timer and wakes it, like tokio's paused clock, so a test //! never waits on the wall clock. A run whose tasks are all parked with nothing -//! armed can never finish, so it panics rather than hang. +//! armed can never finish, so it panics rather than hang, unless a task awaits +//! [`idle`], which completes at exactly that point. //! //! Protocol drivers take time from their caller: [`drive`] polls one with the //! simulated instant, and [`attach`] keeps an externally owned clock in step. use std::{ - cell::RefCell, + cell::{Cell, RefCell}, collections::BTreeMap, future::{Future, IntoFuture}, pin::{Pin, pin}, @@ -41,6 +42,8 @@ struct Runtime { spawned: RefCell>, time: RefCell