diff --git a/quest/m1/README.md b/quest/m1/README.md index 80e8c0927c2..61188ee6fc4 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -62,7 +62,6 @@ above them. - [qmux credit](/quest/m0/qmux-credit.md) - dropped/stopped streams return credit and close is delivered - [Restart followers](/quest/m1/restart-followers.md) - first-party consumers follow an epochless `Restart`, from a GOAWAY drain or a publisher replacement, by resolving the path again -- [Cross-node flapping](/quest/m1/cross-node-flapping.md) - incident-sensitive baseline and residual stale-revival counterexamples before the route rewrite - [Cluster routing](/quest/m1/cluster-routing/README.md) - prove shared routing contracts, implement per-origin routes, and connect outbound local clusters - [P2P](/quest/m1/p2p/README.md) - application-selected peer sessions use shared priorities, routing, and same-epoch handover - [Routing guide](/quest/m1/routing-guide.md) - concepts and practical outbound federation/P2P setup diff --git a/quest/m1/cluster-routing/README.md b/quest/m1/cluster-routing/README.md index dc8648e871b..1b2bbd7995c 100644 --- a/quest/m1/cluster-routing/README.md +++ b/quest/m1/cluster-routing/README.md @@ -94,7 +94,6 @@ setup after the capabilities land. - [Route health](/quest/m2/route-health/README.md) - epoch-only active/passive delivery monitoring and automatic recovery - [P2P](/quest/m1/p2p/README.md) - application-selected peer sessions feed the shared routing model - [Routing guide](/quest/m1/routing-guide.md) - concept page and practical federation/P2P setup -- [Cross-node flapping](/quest/m1/cross-node-flapping.md) - incident-sensitive regressions for residual withdrawal hunting - [Route wakes](/quest/m2/route-wakes.md) - local work follows affected fronts rather than the whole covered table - [Link quality](/quest/m2/link-quality.md) - optional measured link cost, distinct from delivery-health failover - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - independent origins serving the same explicit epoch diff --git a/quest/m1/cluster-routing/handoff.md b/quest/m1/cluster-routing/handoff.md index 6df7268763a..27a42b22be0 100644 --- a/quest/m1/cluster-routing/handoff.md +++ b/quest/m1/cluster-routing/handoff.md @@ -55,5 +55,4 @@ bounded seqno-request attempt ends and no further unsolicited update arrives. ## Related -- [Cross-node flapping](/quest/m1/cross-node-flapping.md) - residual stale-revival fixtures - [Lazy discovery](/quest/m1/lazy-discovery/README.md) - interest changes must not erase freshness protection diff --git a/quest/m1/cluster-routing/routes.md b/quest/m1/cluster-routing/routes.md index fa101f5b882..386a7d031b1 100644 --- a/quest/m1/cluster-routing/routes.md +++ b/quest/m1/cluster-routing/routes.md @@ -176,6 +176,21 @@ Open, for the implementer to settle and record: with deterministic acceptance tests before the wire is written. - How much of today's route trie, fronts, and `route_order` survives intact. +The #4349 counterexamples are the `residual_*` tests in +`rs/moq-net/tests/mesh_withdraw.rs`. They pin today's hunting on lite-06: +a stale path outlasting the 300 ms hold, a revival the quarantine misses, +and 179 hunting relays (3 reviving) over 200 seeded meshes. Turn each into a +single retraction everywhere, and keep `cross_node_close_retracts_once` +passing. On `main` an epochless switch to a stale path is a Restart, which a +lite-06 client sees as a retraction and an announcement; the release sent an +in-place update. + +Publishing toggles too, decided 2026-10-13: add a time-mocked fixture where, +on some rings, a longer path arrives first and the better route then replaces +it as a Restart, so a lite-06 client sees `Start, End, Start` while the +broadcast stays live. Pin today's sequence beside the `residual_*` tests; +the epochless Restart rule above decides what it becomes. + Tests, time mocked: #4644's live-graph withdrawal and failover tests (`quest/m0/path-hunting` branch) as acceptance, run on the wip version; a link flap with many broadcasts behind one origin costs one ROUTE change per @@ -219,11 +234,10 @@ release. - [Routing contracts](/quest/m1/cluster-routing/contracts.md) - authenticated incarnation, local priorities, forwarding progress, and export views - [Next-hop handoff](/quest/m1/cluster-routing/handoff.md) - complete and fresh claims with bounded state -- [Cross-node flapping](/quest/m1/cross-node-flapping.md) - incident-sensitive baseline and residual counterexamples ## Closes -- [#4349](https://github.com/moq-dev/moq/issues/4349) - close once the residual cross-node flapping counterexamples pass +- [#4349](https://github.com/moq-dev/moq/issues/4349) - close once the `residual_*` tests in `mesh_withdraw.rs` retract once everywhere ## Related diff --git a/quest/m1/cross-node-flapping.md b/quest/m1/cross-node-flapping.md deleted file mode 100644 index 04be7ce47c9..00000000000 --- a/quest/m1/cross-node-flapping.md +++ /dev/null @@ -1,53 +0,0 @@ -# [M] Reproduce cross-node announce flapping - -## Goal - -A time-mocked drill with three or more clustered relays, extending -`rs/moq-net/tests/mesh_withdraw.rs` (which already checks one retraction on a -34-node partial mesh) or added beside it: a publisher on one node cleanly closes a broadcast, and a client -watching its prefix on another node sees one retraction, with no -re-announce, for that path. The drill runs against `main` and lands as a -regression test. - -Reported in [#4349](https://github.com/moq-dev/moq/issues/4349) against the -release on cdn.moq.pro (2026-10-11): after a clean close, the subscriber's -announce stream saw the path Announced/Retracted about 1140 times over about -2 s, and subscribes meanwhile answered `internal error` or `unroutable`. -It never happened on one node or on a self-hosted two-relay 0.17.2 cluster. -Loss and FETCH were clean in every run (about 31k FETCHes, all ok). - -## Plan - -- Updated 2026-10-13: #5375's two-second same-publisher retraction quarantine - and #5391's repeated-retraction queue dedup landed. #5375 reports 1,042 - revived broadcasts reduced to 15 over 200 randomized meshes; the #5376 - release backport reports 457 reduced to 2. These are mitigation results, - not proof that main is clean. Existing mesh_withdraw tests include the - eight-node regression. Preserve those and reduce residual failures to - deterministic public fixtures, with pre-fix/release positive controls. - Git backports establish release inclusion, not a verified deployment build. -- The stale-revival mechanism investigated in #5375: path hunting in today's lite-06 path - vector. A peer's re-announce of a withdrawn prefix clears its withdrawal - (`reannounced` in `rs/moq-net/src/model/origin.rs`) and revives stale - alternate routes through it, so each node's late withdrawal toggles the - path again. A two-relay cluster has no alternates, which fits the report. - #4642's 300 ms hold did not cover fresh retract/announce toggles. The newer - quarantine holds strictly worse stale alternatives, leaving equal/better - stale ranks. Do not lengthen the hold to declare the issue solved. -- Land the incident-sensitive baseline and residual fixtures before the route - rewrite, without requiring a redesign here. Routes and announces owns - closing #4349, so this quest can complete with the counterexample harness - rather than depending on the implementation it gates. A passing single topology is not enough to close - the residual concern. Fix a small independent root cause only if one is found. -- Not planned here: the reporter's self-hosted two-relay rejoin stall - (`subscribe canceled (idle)`, then `old`, then no state). On 0.17.2 a - rejoin could read a stale cached group as the live edge, which #4741 and - #4914 changed on `main`; revisit it if their re-run on a release - carrying those still stalls. -- Harness for reference: https://github.com/fperex/moq-burst-repro (published - crates only); its traces hold per-session announcement sequences. - -## Related - -- [Routes and announces](/quest/m1/cluster-routing/routes.md) - the lite-07 redesign that must pass the residual counterexamples -- [Dropped sources](/quest/m1/dropped-sources.md) - the `internal error` replies seen during the same churn diff --git a/rs/moq-net/sim/src/lib.rs b/rs/moq-net/sim/src/lib.rs index 91ae336ae1a..527316ebb91 100644 --- a/rs/moq-net/sim/src/lib.rs +++ b/rs/moq-net/sim/src/lib.rs @@ -3,13 +3,14 @@ //! Time never moves on its own. Once every task has stalled, [`run`] jumps the clock //! to the earliest armed timer and wakes it, like tokio's paused clock, so a test //! never waits on the wall clock. A run whose tasks are all parked with nothing -//! armed can never finish, so it panics rather than hang. +//! armed can never finish, so it panics rather than hang, unless a task awaits +//! [`idle`], which completes at exactly that point. //! //! Protocol drivers take time from their caller: [`drive`] polls one with the //! simulated instant, and [`attach`] keeps an externally owned clock in step. use std::{ - cell::RefCell, + cell::{Cell, RefCell}, collections::BTreeMap, future::{Future, IntoFuture}, pin::{Pin, pin}, @@ -41,6 +42,8 @@ struct Runtime { spawned: RefCell>, time: RefCell