diff --git a/quest/m2/cluster-routes/README.md b/quest/m2/cluster-routes/README.md index 1007dff1568..fb6cabff2c4 100644 --- a/quest/m2/cluster-routes/README.md +++ b/quest/m2/cluster-routes/README.md @@ -53,10 +53,27 @@ The items below were decided 2026-11-10 (maintainer, /quest-plan on lite-07 rout The maintainer delegated these decisions to the coordinator while away, recorded 2026-11-13: "The user is away; the coordinator takes recommendations on their behalf, open to change in review." - Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. [The fresh-publisher exemption](/quest/m2/cluster-routes/fresh-publisher.md) is modeled and measured separately before deciding whether to adopt it. + Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, while the maintainer was away, open to change in review): ✅ keep doubt until the one-second timer fires (rejected: early collection, which allows the request loop `Loop.cfg` recorded). The model and CI's path-rule sweeps use this timer baseline. ✅ keep separate claim handling until the switch removes Babel (rejected: requiring one shared claim engine now); `Reader` and `View` remain shared. A fresh publisher's exemption was to be modeled and measured separately; the decision below covers it. Measured on 24 relays (the mesh sweep's sparse shape, 16 seeds), against today's router, with the doubt lasting 4 hops of delay: a detour settles in 12 hops of delay where the distance takes 15.7, and in 6.2 where it does not run through the far end of the closed link; the publisher's relay dying costs 124 route messages where the distance costs 276, with no loop and 6 claims sent again; and a link closing in a clique costs 44 messages with one publisher behind it and 198 with eight, where the distance costs 621 and 5,024. Accepted: a detour through the far end of a closed link waits the timer out; selected routes can loop for a moment when two unrelated things fail at once (32 of 120,000 churn steps), which the distance never does; and holds and doubt have passed sweeps. The model covers both, but its no-stale-route property holds only for one failure and a patient timer; `StaleIsChased` bounds the rest. The timer's length is the model's and the router's to settle, from the maintainer's 1 s. Rejected: Babel's distance; paths with nothing else, which hunt through stale paths (65,219 messages for one publisher leaving a clique of 12); no doubt, which loops when a relay with several neighbours dies; and a doubt lifted by an epoch the node floods, which resolves in a couple of hops but costs a message per session on each relay-to-relay close ("Epoches are a bit scary in terms of messages"). + Declined 2026-11-14 (maintainer), timer-only doubt retained over it: a fresh publisher's exemption, under which a relay moves at once onto a doubted publisher's own route when an authenticated session the publisher opened after the doubt began offers it. [moq-dev/moq#8735](https://github.com/moq-dev/moq/pull/8735) modeled and measured it. The model found no broken rule (1.8M states, every ordering of the publisher rejoining two joined relays), but the exemption frees only the publisher's own relay: every other relay still doubts the publisher until its timer fires, and a publisher that moves to another relay gains nothing. Under churn it shows each flap of a publisher's link that the timer damps. Measured on 24 relays (sparse: 16 meshes; churn: 400 cases), recovery in ms, and dark counting observations of a reachable publisher with no route: + + | Case | Rule | Recovery | Dark | Messages | Claims | + | --- | --- | ---: | ---: | ---: | ---: | + | Clique rejoin | timer | 1,050 | 984 | 1,105 | 96 | + | Clique rejoin | exempt | 1,025 | 921 | 1,105 | 96 | + | Ring rejoin | timer | 1,875 | 1,512 | 118 | 96 | + | Ring rejoin | exempt | 1,800 | 1,380 | 120 | 88 | + | Sparse rejoin | timer | 1,372 | 1,194 | 164 | 100 | + | Sparse rejoin | exempt | 1,285 | 1,056 | 187 | 91 | + | Clique move | either | 1,100 | 1,032 | 1,612 | 2,124 | + | Ring move | either | 1,850 | 1,488 | 117 | 92 | + | Churn | timer | | 162,654 | 8,785 | 5,698 | + | Churn | exempt | | 116,404 | 13,646 | 10,524 | + + Neither rule looped, and churn's stale observations stayed near 3,300 under both. + 28. ✅ Decided 2026-11-12 (maintainer, /quest-plan on the quests still written for Babel), asked how item 27's removal is split and ordered: **the router first with no wire change, then a switch of the sessions, and the switch lands before the route stream.** [The router](https://github.com/moq-dev/moq/pull/8622) runs the path rule with a route per scope, sans-IO, beside today's router. [The switch](/quest/m2/cluster-routes/switch.md) puts it on the lite-07-wip wire, the draft, the Rust sessions, and `js/net`, and deletes Babel's router. [The interest set](https://github.com/moq-dev/moq/pull/8728) then edits the path router only, and [the route stream](/quest/m2/cluster-routes/route-stream.md) follows the switch with no echo or route request to carry. The switch's recommended wire puts the answer that ends a hold on the publisher half, behind the routes its sender sent first, so the path rule needs no order between streams that today's sessions lack for one prefix. [Acceptance](/quest/m2/cluster-routes/acceptance.md) is of the switched sessions, and the inductive invariant stays in m1, rewritten for the path, without `LoopFree` as an invariant. Rejected: the router quest in m1; one quest for router, wire, draft, sessions, and `js/net`; the route stream first, which builds the echo and ROUTE_REQUEST on the new stream and then deletes them; one wire change for the path rule and the one stream together; moving the inductive invariant to m4; deleting it. Reversed 2026-11-13 (maintainer, /quest-complete): ✅ the inductive invariant is deleted, since Apalache cannot step even one transition of the route model ([moq-dev/moq#8564](https://github.com/moq-dev/moq/pull/8564)), no Rust TLA+ checker qualifies ([moq-dev/moq#8600](https://github.com/moq-dev/moq/pull/8600), [moq-dev/moq#8669](https://github.com/moq-dev/moq/pull/8669)), and no more JVM model checking is wanted here (rejected: keeping it paused). 29. ✅ Decided 2026-11-12 (maintainer, /quest-audit before the cut), asked how the path rule keeps the `upstream` guarantee, which the prototype left out and [the switch](/quest/m2/cluster-routes/switch.md) had listed for removal with Babel's machinery: **a route keeps one sticky down-only flag beside its path.** A relay sets it on a route it learns over a link marked `upstream`, it is never cleared on a later hop, and a relay never sends a flagged route over an upstream link. The flag is export policy, not loop avoidance, so the path does not replace it: a relay that hears a CDN route from a LAN peer learned it on a link that is not upstream, and only the flag tells it the route came down from an upstream. That case is a local cluster with more than one uplink ([outbound federation](/quest/m3/cluster-routing/federation.md)), where a relay whose own uplink died reaches the CDN through a peer; an edge that dials only its cores needs just the local rule. This is the Only-To-Customer attribute of BGP roles (RFC 9234). The router (`rs/moq-net/src/route/path.rs`, built in [#8622](https://github.com/moq-dev/moq/pull/8622)) carries the flag and its export check, [the switch](/quest/m2/cluster-routes/switch.md) adds them to `RouteHandoff.tla` and keeps the flag on ROUTE_START and ROUTE_UPDATE, and [acceptance](/quest/m2/cluster-routes/acceptance.md)'s two-uplink mesh proves it. Sender side only, as today. Rejected: a mark on each hop of the path, which gives the same answer from more bytes and model state; the local rule with no flag, which either leaks through a LAN peer or gives up the dead-uplink fallback; also refusing a flagged route that arrives from a link whose grant marks the sender as a customer, which can be added later without a wire change. diff --git a/quest/m2/cluster-routes/fresh-publisher.md b/quest/m2/cluster-routes/fresh-publisher.md deleted file mode 100644 index 10e5a978847..00000000000 --- a/quest/m2/cluster-routes/fresh-publisher.md +++ /dev/null @@ -1,51 +0,0 @@ -# [L] Model and measure a fresh publisher's doubt exemption - -## Goal - -Determine whether a fresh authenticated direct publisher session can recover -its own route before the one-second doubt expires, without admitting a stale -return or a request loop. Compare it with timer-only doubt before deciding -whether to adopt it. - -## Plan - -Decided 2026-11-13 (maintainer via the authorized coordinator, `/quest-afk`, -while the maintainer was away): -timer-only doubt is the baseline in -[moq-dev/moq#8622](https://github.com/moq-dev/moq/pull/8622), and this exemption -is modeled and measured separately. Early collection is rejected because it -permits the request loop recorded by `Loop.cfg`. These choices remain open -to change in review. - -The fixed 24-relay reconnect cases take 1,050 ms in a clique and 1,875 ms in -a ring, including propagation at 25 ms per hop, versus 50 ms and 325 ms for -the distance rule. Both path policies wait because heard routes still name -the doubted publisher. The exemption could remove that wait for a publisher -returning directly, but not for a relay presenting somebody else's route. - -Model what authenticates the publisher and binds the offer to the new direct -session, including delayed offers from the old session, reconnects, and -concurrent link failures. Keep the timer-only model as the control. Reproduce -counterexamples on the router and measure the same reconnect and churn -scripts on both policies, reporting traffic, request recovery, stale routes, -and reachable but unrouted observations. Preserve the baseline's safety -checks; a faster reconnect alone does not justify the exemption. - -Recommend limiting any exemption to the new authenticated publisher's own -direct offer. A forwarded offer or a new transport session alone does not -establish that the origin is alive. Record the evidence and adoption decision -before changing the default, and mirror an adopted rule in Rust, JS, and the -draft wherever the wire switch has already landed. - -## Waiting - -- Maintainer: after the model and measurements, adopt the narrowly authenticated - direct-publisher exemption or retain timer-only doubt. Recommend adoption only - if it preserves the safety checks and improves the measured reconnect cases; - otherwise retain the baseline. An agent can gather that evidence, but adopting - an exception changes the coordinator's accepted routing policy and needs an - explicit decision. - -## Related - -- [lite-07 sessions run the path router](/quest/m2/cluster-routes/switch.md) - supplies the authenticated session boundary when it puts the rule on the wire diff --git a/rs/moq-srt/src/server.rs b/rs/moq-srt/src/server.rs index 1af2858c62d..4b365bb9b08 100644 --- a/rs/moq-srt/src/server.rs +++ b/rs/moq-srt/src/server.rs @@ -868,6 +868,8 @@ mod tests { use moq_mux::container::{Producer, ts}; use moq_net::Timestamp; + kio::clock::host_time(); // real SRT sockets + // Short enough to keep the test quick, long enough that the release instants // either side of the marker are separated by more than clock noise. const LATENCY: Duration = Duration::from_millis(300);