From 8e2b7aa0c38727a69d83946caf8501dd10a600b1 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 09:46:32 -0800 Subject: [PATCH 01/11] chore(quest): claim quest/m2/route-latency.md Quest-Claim: b14401aa6e4877d6 From 7dff2820d9e9c2ce8023505ce544f8cfc2e98cdf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 09:58:27 -0800 Subject: [PATCH 02/11] quest: size the route latency sum for the maintainer's lite-07 decision Co-Authored-By: Claude Opus 5.5 --- quest/m2/route-latency.md | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/quest/m2/route-latency.md b/quest/m2/route-latency.md index 05707c0a64e..ec377356580 100644 --- a/quest/m2/route-latency.md +++ b/quest/m2/route-latency.md @@ -47,12 +47,39 @@ If the field is taken, it holds to these: - The draft, the Rust and JS codecs, and where the application reads it land together, with `just test interop --all`. -Size the field against the lite-07 draft first, then bring the maintainer the -choice below with that size and the update count. +Sized 2026-11-10 against the lite-07 draft, by a bufferbloat simulation +whose script and tables are in +[moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458): + +- Wire: `Down Only (8)` becomes `Flags (8)`, bit 0 down-only and bit 1 + latency present, followed by `Latency (i)` in milliseconds, PROBE's RTT + unit, only when bit 1 is set. Unknown costs 0 bytes, a sum up to 127 ms 1 + byte, up to 16.4 s 2 bytes: ROUTE_UPDATE grows from 6-9 bytes to 7-11, + ROUTE_START from 14-16 to 15-18. +- Filter: a node reads its link's smoothed RTT once a second and reports the + minimum of the last 30 reads, re-advertising only when that leaves + max(5 ms, a quarter) of what it last advertised. Under bufferbloat, reading + every ACK sent 292 updates per link-hour, an EWMA of 1 Hz reads 37, a 20 s + median 1.3, and the 30 s minimum 0. A rise shows after 30 s, a drop at + once, and a standing queue shorter than 30 s sends nothing. +- Updates: a relay adds its upstream link's value to the sum it received, + so one real shift sends one ROUTE_UPDATE per origin on each stream + downstream of the link: R for the first link of a chain of R relays, 1 for + the last, measured at 1.0 to 1.4 per downstream relay for R = 1 to 8. + Jitter alone sent 0 per origin-hour up to R = 2, and 2.2 on average (9 at + most) at R = 8. +- Cost: the fan-out of a Cost change today, every route selected over the + link, but triggered by measurement: a link carrying 10,000 origins sends + about 80 KB per downstream stream per shift. +- Accuracy: each hop is within max(5 ms, 25%) of its link's RTT floor, so + the sum is within the sum of those bounds. +- An endpoint whose transport reports no RTT can still learn its first hop + from PROBE, whose replies carry the relay's smoothed RTT; only one without + a Probe Stream reads the sum as excluding the first hop. ## Waiting -- Maintainer: decide whether lite-07's ROUTE carries the latency sum. An agent cannot take it, since it adds a field to a wire about to freeze. Options: a quantized, informational sum on ROUTE_START and ROUTE_UPDATE as above, or no wire change, leaving applications to measure with a track of their own. Recommendation: the field, if the sizing shows a shift costs a ROUTE_UPDATE per origin at most; check that the choice is recorded in this Plan. +- Maintainer: decide whether lite-07's ROUTE carries the latency sum. An agent cannot take it, since it adds a field to a wire about to freeze. Options: the sum as sized above, costing 0 to 2 bytes on ROUTE_START and ROUTE_UPDATE and one ROUTE_UPDATE per origin per downstream stream for each real shift, none for jitter; or no wire change, leaving applications to measure with a track of their own. Recommendation: the field, since the sizing meets the bar of at most one ROUTE_UPDATE per origin per shift; check that the choice is recorded in this Plan. ## Related From 3041b06b14c877b638ffbb67e816cb43c67fde1a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 12:25:51 -0800 Subject: [PATCH 03/11] feat(net): lite-07 routes carry an informational min RTT Co-Authored-By: Claude Opus 5.5 --- dart/moq/lib/src/aliases.dart | 7 +- dart/moq_ffi/lib/src/moq.dart | 91 ++++++++++++- doc/bin/relay/cluster.md | 19 +++ doc/bin/relay/config.md | 1 + drafts/draft-lcurley-moq-lite.md | 36 +++-- go/wrapper/types.go | 4 +- .../kotlin/dev/moq/Aliases.kt | 6 + py/moq-rs/moq/__init__.py | 2 + py/moq-rs/moq/types.py | 4 + quest/m2/README.md | 2 +- quest/m2/lite07-finalize.md | 2 +- quest/m2/route-latency.md | 115 ++++++---------- rs/moq-ffi/src/origin.rs | 28 ++++ rs/moq-net/src/fuzz.rs | 1 + rs/moq-net/src/lite/announce.rs | 52 +++++++- rs/moq-net/src/lite/subscriber.rs | 1 + rs/moq-net/src/lite/subscriber/routed.rs | 24 +++- rs/moq-net/src/model/mod.rs | 4 +- rs/moq-net/src/model/origin.rs | 77 ++++++++++- rs/moq-net/src/route/mod.rs | 76 +++++++++-- rs/moq-net/src/route/node.rs | 50 +++++-- rs/moq-net/src/route/rtt.rs | 125 ++++++++++++++++++ rs/moq-net/src/route/tests/min_rtt.rs | 109 +++++++++++++++ rs/moq-net/src/route/tests/mod.rs | 22 +++ rs/moq-net/src/route/tests/routing.rs | 3 + rs/moq-relay/src/cluster.rs | 13 ++ rs/moq-relay/src/settings.rs | 3 + rs/moq-tokio/tests/integration/broadcast.rs | 69 ++++++++++ swift/Sources/Moq/Aliases.swift | 3 + 29 files changed, 819 insertions(+), 130 deletions(-) create mode 100644 rs/moq-net/src/route/rtt.rs create mode 100644 rs/moq-net/src/route/tests/min_rtt.rs diff --git a/dart/moq/lib/src/aliases.dart b/dart/moq/lib/src/aliases.dart index ab7391cf40e..6aea471ffab 100644 --- a/dart/moq/lib/src/aliases.dart +++ b/dart/moq/lib/src/aliases.dart @@ -67,9 +67,14 @@ typedef AnnounceConfig = MoqAnnounceConfig; /// Follows a specific broadcast path across the publisher instances that serve it. typedef Following = MoqFollowing; -/// The read-only facts on a received route: a static cost and the stream's instance. +/// The read-only facts on a received route: a static cost, the stream's instance, and +/// its [MinRtt]. typedef RouteInfo = MoqRouteInfo; +/// The minimum round trip to a route's origin node over moq-lite-07: `upstream` past the +/// session's peer, `link` to the peer, and their `total`. Informational; it never ranks a route. +typedef MinRtt = MoqMinRtt; + /// One continuous stream at a path: `Instance.parse` names one, and only a session mints one. /// /// Two are equal only when their text and minted flag both match, so pass an diff --git a/dart/moq_ffi/lib/src/moq.dart b/dart/moq_ffi/lib/src/moq.dart index 3cca3b647d9..3e8fe8ae4e6 100644 --- a/dart/moq_ffi/lib/src/moq.dart +++ b/dart/moq_ffi/lib/src/moq.dart @@ -2962,6 +2962,77 @@ class FfiConverterMoqAnnounceConfig { } } +class MoqMinRtt { + final Duration? upstream; + final Duration? link; + final Duration? total; + MoqMinRtt({this.upstream, this.link, this.total}); +} + +class FfiConverterMoqMinRtt { + static MoqMinRtt lift(RustBuffer buf) { + return FfiConverterMoqMinRtt.read(buf.asUint8List()).value; + } + + static LiftRetVal read(Uint8List buf) { + int new_offset = buf.offsetInBytes; + final upstream_lifted = FfiConverterOptionalDuration.read( + Uint8List.view(buf.buffer, new_offset), + ); + final upstream = upstream_lifted.value; + new_offset += upstream_lifted.bytesRead; + final link_lifted = FfiConverterOptionalDuration.read( + Uint8List.view(buf.buffer, new_offset), + ); + final link = link_lifted.value; + new_offset += link_lifted.bytesRead; + final total_lifted = FfiConverterOptionalDuration.read( + Uint8List.view(buf.buffer, new_offset), + ); + final total = total_lifted.value; + new_offset += total_lifted.bytesRead; + return LiftRetVal( + MoqMinRtt(upstream: upstream, link: link, total: total), + new_offset - buf.offsetInBytes, + ); + } + + static RustBuffer lower(MoqMinRtt value) { + final total_length = + FfiConverterOptionalDuration.allocationSize(value.upstream) + + FfiConverterOptionalDuration.allocationSize(value.link) + + FfiConverterOptionalDuration.allocationSize(value.total) + + 0; + final buf = Uint8List(total_length); + write(value, buf); + return toRustBuffer(buf); + } + + static int write(MoqMinRtt value, Uint8List buf) { + int new_offset = buf.offsetInBytes; + new_offset += FfiConverterOptionalDuration.write( + value.upstream, + Uint8List.view(buf.buffer, new_offset), + ); + new_offset += FfiConverterOptionalDuration.write( + value.link, + Uint8List.view(buf.buffer, new_offset), + ); + new_offset += FfiConverterOptionalDuration.write( + value.total, + Uint8List.view(buf.buffer, new_offset), + ); + return new_offset - buf.offsetInBytes; + } + + static int allocationSize(MoqMinRtt value) { + return FfiConverterOptionalDuration.allocationSize(value.upstream) + + FfiConverterOptionalDuration.allocationSize(value.link) + + FfiConverterOptionalDuration.allocationSize(value.total) + + 0; + } +} + class MoqOriginConfig { final int? cacheCapacityBytes; MoqOriginConfig({this.cacheCapacityBytes = null}); @@ -3012,10 +3083,12 @@ class MoqRouteInfo { final int cost; final MoqInstance instance; final bool available; + final MoqMinRtt minRtt; MoqRouteInfo({ required this.cost, required this.instance, required this.available, + required this.minRtt, }); } @@ -3041,8 +3114,18 @@ class FfiConverterMoqRouteInfo { ); final available = available_lifted.value; new_offset += available_lifted.bytesRead; + final minRtt_lifted = FfiConverterMoqMinRtt.read( + Uint8List.view(buf.buffer, new_offset), + ); + final minRtt = minRtt_lifted.value; + new_offset += minRtt_lifted.bytesRead; return LiftRetVal( - MoqRouteInfo(cost: cost, instance: instance, available: available), + MoqRouteInfo( + cost: cost, + instance: instance, + available: available, + minRtt: minRtt, + ), new_offset - buf.offsetInBytes, ); } @@ -3052,6 +3135,7 @@ class FfiConverterMoqRouteInfo { FfiConverterUInt64.allocationSize(value.cost) + FfiConverterMoqInstance.allocationSize(value.instance) + FfiConverterBool.allocationSize(value.available) + + FfiConverterMoqMinRtt.allocationSize(value.minRtt) + 0; final buf = Uint8List(total_length); write(value, buf); @@ -3072,6 +3156,10 @@ class FfiConverterMoqRouteInfo { value.available, Uint8List.view(buf.buffer, new_offset), ); + new_offset += FfiConverterMoqMinRtt.write( + value.minRtt, + Uint8List.view(buf.buffer, new_offset), + ); return new_offset - buf.offsetInBytes; } @@ -3079,6 +3167,7 @@ class FfiConverterMoqRouteInfo { return FfiConverterUInt64.allocationSize(value.cost) + FfiConverterMoqInstance.allocationSize(value.instance) + FfiConverterBool.allocationSize(value.available) + + FfiConverterMoqMinRtt.allocationSize(value.minRtt) + 0; } } diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index d17024b76ce..68e6ffdfc57 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -219,6 +219,25 @@ draining process. A route's price is the publisher's production cost plus the link costs it crosses. A live publisher seeds 0, and caching never changes it. +## Route latency + +A moq-lite-07-wip route also carries its min RTT: each relay adds its link's +lowest smoothed RTT over the last 30 seconds to the sum it received, so an +application reads the round trip to a publisher through any number of relays +from the route it was announced on. It is informational and never picks a +route; the link cost does that. A link's value changes only when its minimum +moves past the larger of 5 ms and a quarter, so jitter and short queues send +no route updates. + +Set `withhold_min_rtt` to advertise none, and every route crossing this relay +reports none from here on. The same setting is `--cluster-withhold-min-rtt` or +`MOQ_CLUSTER_WITHHOLD_MIN_RTT`. + +```toml +[cluster] +withhold_min_rtt = true +``` + ## LAN discovery On a LAN there may be no one to list. `[cluster.lan]` advertises this relay diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index 2df66a6ea03..54bf00129eb 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -307,6 +307,7 @@ node = "https://us-west.example.com/" # This relay's own URL. token = "cluster.jwt" # JWT for dials without an inline ?jwt=. route_key = "route.key" # Route key; relays sharing the file fail over as one to scoped peers. max_hops = 32 # Longest accepted/forwarded hop path (1 to 32). +withhold_min_rtt = false # Advertise no min RTT to moq-lite-07 peers. [cluster.api] url = "https://api.example.com/peers" # Live JSON peer list; a local file also works. diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index f026f67e54a..80f682f4f96 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -407,7 +407,7 @@ The subscriber creates the stream with an ANNOUNCE_REQUEST message. The publisher replies with a single ANNOUNCE_OK message followed by the matching routes and claims and any future changes: - ROUTE_START: the publisher reaches an origin node. -- ROUTE_UPDATE: a route's seqno, metric, or down-only bit changed. +- ROUTE_UPDATE: a route's seqno, metric, down-only bit, or Min RTT changed. - ROUTE_HOLD: the publisher has no usable route but holds it while it recovers. - ROUTE_END: the publisher let go of a route and every claim under it. - ANNOUNCE_START: a matching claim of a route's origin node. @@ -492,12 +492,23 @@ A receiver adds the session's link cost (see [Cost Parameter](#cost-parameter)) Each link adds at least 1, so a route's metric strictly increases away from its origin. A metric that overflows 64 bits makes the route unusable. +A route may also carry a Min RTT: the round trip to its origin node in milliseconds, summed link by link. +It is informational: nothing ranks or selects a route, or judges it feasible, by its Min RTT. +An origin advertises its own route with a Min RTT of 0. +A receiver adds its link min RTT for the session to the Min RTT of every route the peer sends; the sum is its Min RTT through that peer, which it advertises on the route it selects. +A link min RTT is the lowest smoothed RTT [RFC9002] the connection measured over the last 30 seconds, read once a second, in milliseconds rounded up. +A lifetime minimum never rises after a path change, so it is not a link min RTT. +A node SHOULD keep the link min RTT it adds until the lowest reading differs from it by more than the larger of 5 ms and a quarter of it, so jitter and short queues send no ROUTE_UPDATE and a lasting change sends one per route. +A node MAY withhold Min RTT, and a node whose connection reports no RTT has no link min RTT; either sends its routes without one, and a route without one stays without one past every later node. +An endpoint learns its round trip to an origin node by adding its own link min RTT to the Min RTT of the route it selected; without its own, the Min RTT alone covers every hop past its first. +Like the Metric, a Min RTT is unverified: a peer can misstate only what crosses it. + A node keeps a feasibility distance per Node ID: the highest Seqno of any route it advertised, and the lowest metric it advertised at that Seqno, as in [RFC8966] Section 3.5.1. A route is feasible when no feasibility distance is kept for its Node ID yet, when its Seqno is greater than the distance's, or when its Seqno is equal and its advertised Metric is lower. A node selects, per Node ID, among its feasible routes that are not held: a session that has not received a GOAWAY, then the highest local priority, then the lowest metric. It MUST NOT select an infeasible route, and a distance only ever falls within a Seqno, so advertising a longer route never raises it. A node MAY mark a session upstream. -A route is down-only when it arrived on an upstream session or with its Down Only bit set; a node sends a down-only route with the bit set, and never on an upstream session. +A route is down-only when it arrived on an upstream session or with its down-only flag set; a node sends a down-only route with the flag set, and never on an upstream session. A node sends its selected route to an origin node on an Announce Stream it answers only while the route has a claim the stream's request presents (see [Announce](#announce)): it sends ROUTE_START before the first such claim, and once the last one is withdrawn, the batch carrying that withdrawal, closed by its floor (which certifies the withdrawal where the node can certify the request), then ROUTE_END. It starts that route on the stream again only once the subscriber echoed the ROUTE_END. A subscriber therefore holds no route for an origin node its request does not meet. A node that does not forward, such as a client, sends only its own route, and holds no feasibility distance for any other. @@ -1142,7 +1153,8 @@ ROUTE_START Message { Node ID (i), Seqno (i), Metric (i), - Down Only (8), + Flags (8), + [Min RTT (i),] } ~~~ @@ -1160,9 +1172,13 @@ The origin's sequence number of the route, which only the origin advances. **Metric**: The publisher's metric to the origin node, 0 when it is the origin. -**Down Only**: -1 if the route is down-only (see [Routing](#routing)), 0 otherwise. -Any other value is a PROTOCOL_VIOLATION. +**Flags**: +A bit field: 0x1 when the route is down-only (see [Routing](#routing)), and 0x2 when a Min RTT follows. +Any other bit set is a PROTOCOL_VIOLATION. + +**Min RTT**: +The publisher's Min RTT to the origin node in milliseconds (see [Routing](#routing)), 0 when it is the origin. +Present only when Flags has 0x2 set, and absent when the publisher withholds it or has none. ## ROUTE_UPDATE {#route-update} @@ -1175,7 +1191,8 @@ ROUTE_UPDATE Message { Route ID (i), Seqno (i), Metric (i), - Down Only (8), + Flags (8), + [Min RTT (i),] } ~~~ @@ -1185,14 +1202,14 @@ Set to 0x7 to indicate a ROUTE_UPDATE message. **Route ID**: The ordinal implicitly assigned by a prior ROUTE_START on this stream. -**Seqno**, **Metric**, and **Down Only**: +**Seqno**, **Metric**, **Flags**, and **Min RTT**: As defined for [ROUTE_START](#route-start), replacing the route's. A ROUTE_UPDATE for a held route makes it usable again. ## ROUTE_HOLD {#route-hold} A publisher sends a ROUTE_HOLD message when it has no usable route to the origin node but still holds the route and its claims while it recovers (see [Routing](#routing)), referencing its Route ID. -The receiver keeps the route's last Seqno, Metric, and Down Only, and does not select it until a ROUTE_UPDATE. +The receiver keeps the route's last Seqno, Metric, Flags, and Min RTT, and does not select it until a ROUTE_UPDATE. ~~~ ROUTE_HOLD Message { @@ -2230,6 +2247,7 @@ The `Message Length` describes the payload size on the wire. - A receiver mints an Instance for each announcement a session of an earlier version delivers, and a `Minted` field beside every Instance (ANNOUNCE_START, ANNOUNCE_RESTART, TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS) says whether a receiver minted it, so no Instance text is reserved for minting. - Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live claim's suffix. - Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list and ANNOUNCE_OK's Hop ID, and replaced the SETUP Hop parameter (0x5) with a [Node ID parameter](#node-id-parameter), so an endpoint that reaches itself detects it at setup. +- ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional Min RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route. - Added ROUTE_HOLD (0xB), which holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back. - Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible. - Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. diff --git a/go/wrapper/types.go b/go/wrapper/types.go index df7661cc642..0e45840f11c 100644 --- a/go/wrapper/types.go +++ b/go/wrapper/types.go @@ -29,8 +29,10 @@ type ( ProtocolKind = ffi.MoqProtocolKind // OriginConfig configures a new origin, such as its maximum cache size in bytes. OriginConfig = ffi.MoqOriginConfig - // RouteInfo is the read-only facts on a received route: the Instance of the stream it serves (routes with one instance are one continuous stream) and its static production and link cost (lower wins). + // RouteInfo is the read-only facts on a received route: the Instance of the stream it serves (routes with one instance are one continuous stream), its static production and link cost (lower wins), and its MinRtt. RouteInfo = ffi.MoqRouteInfo + // MinRtt is the minimum round trip to a route's origin node over moq-lite-07: Upstream past the session's peer, Link to the peer, and their Total. Informational; it never ranks a route. + MinRtt = ffi.MoqMinRtt // Instance names one continuous stream at a path. Compare two with Eq: a session mints one // for a route whose wire carries none, and it differs from a named instance with the same Text. Instance = ffi.MoqInstance diff --git a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt index 948c7424528..8e6f158d577 100644 --- a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt +++ b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt @@ -193,6 +193,12 @@ typealias RouteOptions = uniffi.moq.MoqRouteOptions */ typealias RouteInfo = uniffi.moq.MoqRouteInfo +/** + * The minimum round trip to a route's origin node over moq-lite-07: `upstream` past the session's + * peer, `link` to the peer, and their `total`. Informational; it never ranks a route. + */ +typealias MinRtt = uniffi.moq.MoqMinRtt + /** Tunes how a track subscription is delivered: priority, delay, ranges, and group ordering. */ typealias Subscription = uniffi.moq.MoqSubscription diff --git a/py/moq-rs/moq/__init__.py b/py/moq-rs/moq/__init__.py index 997c926ca91..fce747ad4f0 100644 --- a/py/moq-rs/moq/__init__.py +++ b/py/moq-rs/moq/__init__.py @@ -77,6 +77,7 @@ Frame, Grant, Instance, + MinRtt, Order, Position, ProtocolError, @@ -152,6 +153,7 @@ "OriginConsumer", "OriginProducer", "Order", + "MinRtt", "Position", "Range", "RateControl", diff --git a/py/moq-rs/moq/types.py b/py/moq-rs/moq/types.py index 2628f449bc9..a1b6acb4421 100644 --- a/py/moq-rs/moq/types.py +++ b/py/moq-rs/moq/types.py @@ -47,6 +47,9 @@ from moq_ffi import ( MoqRateTarget as RateTarget, ) +from moq_ffi import ( + MoqMinRtt as MinRtt, +) from moq_ffi import ( MoqRouteInfo as RouteInfo, ) @@ -106,6 +109,7 @@ "FetchGroupOptions", "Grant", "Instance", + "MinRtt", "Order", "Position", "ProtocolError", diff --git a/quest/m2/README.md b/quest/m2/README.md index 763c3a74b6e..197245d6434 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -199,7 +199,7 @@ quests that require them. - [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - the wire names every abandoned group and the drop signal is public, so readers count losses instead of stalling - [Stream credit by priority](/quest/m1/stream-credit/README.md) - every stream opens through one priority entry point, and a higher-priority group takes over or resets a lower one when credit runs out (held in m1 by an open claim) - [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - warm routes and the eligibility rule hold in route-layer sweeps, and the warm message's wire placement is chosen before the lite-07 cut -- [Route latency](/quest/m2/route-latency.md) - the maintainer decides whether a lite-07 ROUTE carries an informational latency sum, and it lands before the cut if so +- [Route latency in js/net](/quest/m2/route-latency.md) - js/net reads and writes the min RTT a lite-07 ROUTE carries, once its route layer lands - [Finalize moq-lite-07](/quest/m2/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it - [Unbounded groups](/quest/m2/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh - [Small API fixes before the cut](/quest/m2/api-shape/README.md) - one path and one spelling per item, names by role, and no dead or wrong-role public surface across net, transport, media, native, apps, JS, and bindings diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md index f284423a969..f168493ccff 100644 --- a/quest/m2/lite07-finalize.md +++ b/quest/m2/lite07-finalize.md @@ -127,7 +127,7 @@ whose lite-07 encoding [lite-07 by start](/quest/m2/time-addressing/lite07.md) c - [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - may add SUBSCRIBE_DROP for a stopped group; leaves this list if it takes the no-wire path - [Delete the old start bookkeeping](/quest/m2/subscribe-ranges/start-bookkeeping.md) - may change START per update on lite-07 - [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - may add the warm-route message to lite-07; leaves this list if the proof places it in a SETUP extension -- [Route latency](/quest/m2/route-latency.md) - may add a latency sum to ROUTE; leaves this list if the maintainer takes the no-wire path +- [Route latency in js/net](/quest/m2/route-latency.md) - js/net reads and writes ROUTE's Min RTT ## Related diff --git a/quest/m2/route-latency.md b/quest/m2/route-latency.md index ec377356580..96681660f0b 100644 --- a/quest/m2/route-latency.md +++ b/quest/m2/route-latency.md @@ -1,87 +1,48 @@ -# [M] Route latency +# [S] Route latency in js/net ## Goal -The maintainer has decided whether a lite-07 ROUTE carries an informational -latency sum, and the choice is carried out before the cut. If it does, each -node adds its link's smoothed RTT to the sum on the route it selected before -advertising it, so an endpoint learns the latency to an origin node through -any number of relays by adding its own first hop, with no extra track and no -knowledge of the relays between. The sum never ranks or selects a route. +`js/net` reads a lite-07 route's Min RTT as Rust does: it decodes and encodes +ROUTE_START's and ROUTE_UPDATE's Flags and optional Min RTT, a browser +publishes its own route at 0, and its announced routes carry the sum the +relay advertised plus its own first hop, marked as excluding the first hop +when the browser's transport reports no RTT. The sum never ranks or selects +a route. ## Plan -Planned 2026-11-09 (maintainer, /quest-plan from Kixel's audit of its fleet -plan): ✅ settle lite-07's route-layer additions before the cut, since a -field on ROUTE after it needs another wire version. - -Why it is worth deciding: an application that wants to show the round trip -between two peers through a cluster has no way to today without publishing a -track for it. A per-hop message keyed by the announce hop list was proposed, -but lite-07 removed that list for one distance-vector route per origin, and -such a route cannot name its path, so a sum is all the wire can carry. The -[status track](/quest/m3/mesh/status.md) publishes per-link RTT for mesh -nodes only, so it cannot answer for a path through an operator's relays. -[moq-dev/moq#3769](https://github.com/moq-dev/moq/pull/3769) priced links by -measurement and was closed for "measurement first, with operator-owned -routing policy"; this is the measurement alone. - -If the field is taken, it holds to these: - -- `route_order` and feasibility never read it. The metric stays the - operator's static cost, and [link quality](/quest/m5/link-quality.md) - stays the opt-in way to price a link by measurement. -- It comes from what QUIC already measures on the link, quantized, with - hysteresis, so jitter sends no ROUTE_UPDATE - ([RFC 9616](https://www.rfc-editor.org/rfc/rfc9616.html) is the reference - for RTT that feeds a routing protocol). Report the ROUTE_UPDATEs one real - shift causes across a chain of relays. -- A node may withhold it, and a sum that crosses such a node is unknown from - there on, marked without a sentinel value. An operator decides whether its - relays report. -- The endpoint's own first hop can be unknown too, such as a browser whose - WebTransport reports no RTT. The application then reads the sum from the - first relay alone, marked as excluding the first hop, never a guess. -- It is unverified, like the metric: a peer can misstate only what crosses - it, and nothing acts on the value. -- The draft, the Rust and JS codecs, and where the application reads it land - together, with `just test interop --all`. - -Sized 2026-11-10 against the lite-07 draft, by a bufferbloat simulation -whose script and tables are in -[moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458): - -- Wire: `Down Only (8)` becomes `Flags (8)`, bit 0 down-only and bit 1 - latency present, followed by `Latency (i)` in milliseconds, PROBE's RTT - unit, only when bit 1 is set. Unknown costs 0 bytes, a sum up to 127 ms 1 - byte, up to 16.4 s 2 bytes: ROUTE_UPDATE grows from 6-9 bytes to 7-11, - ROUTE_START from 14-16 to 15-18. -- Filter: a node reads its link's smoothed RTT once a second and reports the - minimum of the last 30 reads, re-advertising only when that leaves - max(5 ms, a quarter) of what it last advertised. Under bufferbloat, reading - every ACK sent 292 updates per link-hour, an EWMA of 1 Hz reads 37, a 20 s - median 1.3, and the 30 s minimum 0. A rise shows after 30 s, a drop at - once, and a standing queue shorter than 30 s sends nothing. -- Updates: a relay adds its upstream link's value to the sum it received, - so one real shift sends one ROUTE_UPDATE per origin on each stream - downstream of the link: R for the first link of a chain of R relays, 1 for - the last, measured at 1.0 to 1.4 per downstream relay for R = 1 to 8. - Jitter alone sent 0 per origin-hour up to R = 2, and 2.2 on average (9 at - most) at R = 8. -- Cost: the fan-out of a Cost change today, every route selected over the - link, but triggered by measurement: a link carrying 10,000 origins sends - about 80 KB per downstream stream per shift. -- Accuracy: each hop is within max(5 ms, 25%) of its link's RTT floor, so - the sum is within the sum of those bounds. -- An endpoint whose transport reports no RTT can still learn its first hop - from PROBE, whose replies carry the relay's smoothed RTT; only one without - a Probe Stream reads the sum as excluding the first hop. - -## Waiting - -- Maintainer: decide whether lite-07's ROUTE carries the latency sum. An agent cannot take it, since it adds a field to a wire about to freeze. Options: the sum as sized above, costing 0 to 2 bytes on ROUTE_START and ROUTE_UPDATE and one ROUTE_UPDATE per origin per downstream stream for each real shift, none for jitter; or no wire change, leaving applications to measure with a track of their own. Recommendation: the field, since the sizing meets the bar of at most one ROUTE_UPDATE per origin per shift; check that the choice is recorded in this Plan. +Decided 2026-11-10 (maintainer, on +[moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458)): ✅ lite-07's +ROUTE carries the latency sum, as min RTT. Each node adds its link's windowed +minimum, the lowest of the last 30 once-a-second reads of the smoothed RTT, +re-advertised only past max(5 ms, 25%), never QUIC's lifetime min_rtt, which +never rises after a path change. The endpoint adds its own first hop from its +connection. Rejected: no wire change, leaving applications a track of their +own. + +[moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458) landed the +draft, the Rust codec and route layer, `origin::RouteInfo::min_rtt`, the +bindings' `MinRtt`, and the relay's `--cluster-withhold-min-rtt`. Its PR body +keeps the sizing simulation: one real shift sends one ROUTE_UPDATE per origin +on each stream downstream of the link, and jitter sends none. + +What remains is the JS side, which needs the JS route layer first: + +- Decode Flags (0x1 down-only, 0x2 Min RTT present, any other bit a + PROTOCOL_VIOLATION) and the Min RTT that follows, in milliseconds, and + encode a browser's own route with a Min RTT of 0. +- Add `MinRtt` (`upstream`, `link`, `total()`) to `js/net`'s `RouteInfo`, + named as Rust's. `link` follows the same windowed minimum over + WebTransport `getStats()`'s smoothed RTT, and is `undefined` where it + reports none. +- Pin the codec to Rust's with the shared corpus, and run + `just test interop --all`. + +## Required + +- [js/net route layer](/quest/m2/cluster-routing/routes/js.md) - the ROUTE codec and route-backed `RouteInfo` this extends ## Related -- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the ROUTE messages this would extend +- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the ROUTE messages this extends - [Link quality](/quest/m5/link-quality.md) - measured link cost, which feeds selection where this never does diff --git a/rs/moq-ffi/src/origin.rs b/rs/moq-ffi/src/origin.rs index b3febfd5573..4f234c1a326 100644 --- a/rs/moq-ffi/src/origin.rs +++ b/rs/moq-ffi/src/origin.rs @@ -94,6 +94,33 @@ pub struct MoqRouteInfo { pub instance: Arc, /// Whether a usable path reaches the route's origin; requests may fail while it is not. pub available: bool, + /// The minimum round trip to the route's origin node, informational; never ranks a route. + pub min_rtt: MoqMinRtt, +} + +/// The minimum round trip to a route's origin node over moq-lite-07, in two parts. +/// +/// Each node on the path adds its link's lowest smoothed RTT over the last 30 seconds. +/// The sum is unknown past a node that withholds it or cannot measure its link, and on +/// any other wire. +#[derive(Clone, Copy, Debug, PartialEq, Eq, uniffi::Record)] +pub struct MoqMinRtt { + /// Past the session's peer: the sum the peer advertised, absent when unknown. + pub upstream: Option, + /// The session's own link to the peer, absent when its transport reports no RTT. + pub link: Option, + /// The whole path, present when both parts are. + pub total: Option, +} + +impl From for MoqMinRtt { + fn from(min_rtt: moq_net::origin::MinRtt) -> Self { + Self { + upstream: min_rtt.upstream, + link: min_rtt.link, + total: min_rtt.total(), + } + } } impl From for MoqRouteInfo { @@ -102,6 +129,7 @@ impl From for MoqRouteInfo { cost: route.cost().value(), instance: MoqInstance::wrap(route.instance().clone()), available: route.available(), + min_rtt: route.min_rtt().into(), } } } diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index 4cbcf800809..07338f37bc0 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -677,6 +677,7 @@ impl RouteRelay { seqno: 0, metric: 0, down_only: false, + min_rtt: None, }; let receive = (0..streams) .map(|_| { diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index d23df0e92e9..e0e690a9ab0 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -114,15 +114,30 @@ pub enum AnnounceBroadcast { Floor { id: u64, revision: Option }, } -/// Encode a seqno, metric, and down-only bit. No origin advertises the last seqno, -/// which would leave no room for the successor a ROUTE_REQUEST names. +/// ROUTE_START and ROUTE_UPDATE flag: the route is down-only. +const DOWN_ONLY: u8 = 0x1; +/// ROUTE_START and ROUTE_UPDATE flag: a Min RTT follows. +const MIN_RTT: u8 = 0x2; + +/// Encode a seqno, metric, flags, and min RTT in milliseconds. No origin advertises the +/// last seqno, which would leave no room for the successor a ROUTE_REQUEST names. fn encode_advert(w: &mut Encoder<'_>, advert: &Advert) -> Result<(), EncodeError> { if advert.seqno == route::MAX { return Err(EncodeError::InvalidState); } w.varint(advert.seqno)?; w.varint(advert.metric)?; - w.bool(advert.down_only); + let mut flags = 0; + if advert.down_only { + flags |= DOWN_ONLY; + } + if advert.min_rtt.is_some() { + flags |= MIN_RTT; + } + w.u8(flags); + if let Some(min_rtt) = advert.min_rtt { + w.varint(u64::try_from(min_rtt.as_millis()).map_err(|_| EncodeError::InvalidState)?)?; + } Ok(()) } @@ -131,10 +146,20 @@ fn decode_advert(r: &mut Decoder<'_>) -> Result { if seqno == route::MAX { return Err(DecodeError::InvalidValue); } + let metric = r.varint()?; + let flags = r.u8()?; + if flags & !(DOWN_ONLY | MIN_RTT) != 0 { + return Err(DecodeError::InvalidValue); + } + let min_rtt = match flags & MIN_RTT { + 0 => None, + _ => Some(std::time::Duration::from_millis(r.varint()?)), + }; Ok(Advert { seqno, - metric: r.varint()?, - down_only: r.bool()?, + metric, + down_only: flags & DOWN_ONLY != 0, + min_rtt, }) } @@ -1541,6 +1566,7 @@ mod tests { seqno, metric, down_only, + min_rtt: None, } } @@ -1582,6 +1608,20 @@ mod tests { id: 0, advert: advert(42, 0, true), }, + AnnounceBroadcast::RouteStart { + node: 0x7a40, + advert: Advert { + min_rtt: Some(std::time::Duration::ZERO), + ..advert(1, 0, false) + }, + }, + AnnounceBroadcast::RouteUpdate { + id: 1, + advert: Advert { + min_rtt: Some(std::time::Duration::from_millis(u64::MAX)), + ..advert(2, 3, true) + }, + }, AnnounceBroadcast::RouteHold { id: 3 }, AnnounceBroadcast::RouteEnd { id: 0 }, AnnounceBroadcast::Floor { @@ -1696,6 +1736,8 @@ mod tests { let malformed = [ forge(ROUTE_START, advert(0, 0, 1, 0)), forge(ROUTE_START, advert(1, route::MAX, 1, 0)), + // A flag no version defines, and a Min RTT flag without its field. + forge(ROUTE_START, advert(1, 0, 1, 4)), forge(ROUTE_START, advert(1, 0, 1, 2)), // ROUTE_UPDATE's leading field is the Route ID, so 0 is a valid id here. forge(ROUTE_UPDATE, advert(0, route::MAX, 1, 0)), diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index ab10fbfd8a2..67ac68a3206 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -9024,6 +9024,7 @@ mod tests { seqno: 0, metric: 0, down_only: false, + min_rtt: None, }; lite::AnnounceBroadcast::RouteStart { node, advert } .encode(w, Version::Lite07) diff --git a/rs/moq-net/src/lite/subscriber/routed.rs b/rs/moq-net/src/lite/subscriber/routed.rs index 41a55264a6a..6d9a16e761a 100644 --- a/rs/moq-net/src/lite/subscriber/routed.rs +++ b/rs/moq-net/src/lite/subscriber/routed.rs @@ -14,6 +14,7 @@ use crate::{ Error, Path, lite::{self, AnnounceReply}, route::{self, node}, + transport::{Metrics as _, poll::Session as _}, }; use super::{AnnouncedRoute, Subscriber}; @@ -96,6 +97,10 @@ pub(super) struct Routed { ready: kio::Queue<(route::Id, Path)>, /// The peer sent a GOAWAY: every claim served here ranks last. drained: bool, + /// The link's min RTT, which the peer's routes add. + rtt: route::LinkRtt, + /// When to read the link's RTT next. + read_rtt: crate::time::Deadline, } impl Routed { @@ -128,6 +133,8 @@ impl Routed { let stream = node .receive(scoped(&link, ®ions, key), prefix.clone()) .map_err(refused)?; + let mut rtt = route::LinkRtt::default(); + stream.min_rtt(rtt.read(subscriber.session.metrics().rtt())); Ok(Self { node, stream, @@ -150,6 +157,8 @@ impl Routed { previous: HashMap::new(), ready: Default::default(), drained: false, + rtt, + read_rtt: crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL), }) } @@ -340,6 +349,7 @@ impl Routed { .node .receive(scoped(&self.link, &self.regions, key), self.prefix.clone()) .map_err(refused)?; + stream.min_rtt(self.rtt.reported()); // What the old stream served stays in the origin until the new one serves it // again, or lets go after. let previous = std::mem::take(&mut self.served); @@ -458,7 +468,8 @@ impl Routed { .with_via(subscriber.via(None)) .with_node(Some(id)) .with_draining(self.drained) - .with_available(served.available); + .with_available(served.available) + .with_min_rtt(served.min_rtt); if let Some(entry) = self.previous.remove(&key) { self.served.insert(key.clone(), entry); } @@ -495,6 +506,16 @@ impl Routed { writer: &mut crate::coding::Writer, waiter: &kio::Waiter, ) -> Result<(), Error> { + // Only a route the peer sent adds the link's min RTT, so a peer sending none + // costs no timer. + while !self.routes.is_empty() && self.read_rtt.poll(waiter).is_ready() { + let last = self.rtt.reported(); + let now = self.rtt.read(subscriber.session.metrics().rtt()); + if now != last { + self.stream.min_rtt(now); + } + self.read_rtt = crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL); + } while let Poll::Ready(output) = self.stream.poll_output(waiter) { self.output(subscriber, output); } @@ -589,6 +610,7 @@ mod tests { seqno: 0, metric: 0, down_only: false, + min_rtt: None, }; fn start(path: &'static str) -> Msg { diff --git a/rs/moq-net/src/model/mod.rs b/rs/moq-net/src/model/mod.rs index 76a6bffd228..180d971cd32 100644 --- a/rs/moq-net/src/model/mod.rs +++ b/rs/moq-net/src/model/mod.rs @@ -64,8 +64,8 @@ pub mod origin { #[cfg(feature = "async")] pub(crate) use super::origin_impl::NodeFilter; pub use super::origin_impl::{ - Claim, Config, Consumer, Cost, Driver, Link, Preference, Producer, Request, Requesting, Route, RouteInfo, - RouteOptions, Source, + Claim, Config, Consumer, Cost, Driver, Link, MinRtt, Preference, Producer, Request, Requesting, Route, + RouteInfo, RouteOptions, Source, }; pub use crate::route::RouteKey; diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 81a53d8e248..0286c245c8a 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -108,6 +108,10 @@ pub struct Config { /// proves nothing, so such a peer files them under a key of its session alone. pub route_key: Option, + /// Advertise no min RTT to lite-07 peers, so no route crossing this origin carries + /// one past it (default false). Its own readers still see the [`MinRtt`] it measures. + pub withhold_min_rtt: bool, + /// Longest hop path a session using this origin accepts or forwards (default 32). /// /// Applies to moq-lite-03+ hop paths and the negotiated MoQ Cluster extension on @@ -166,6 +170,7 @@ impl Default for Config { Self { hop: random_id(), route_key: None, + withhold_min_rtt: false, max_hops: MAX_HOP_LIMIT, pool, cache_duration: None, @@ -376,6 +381,28 @@ impl<'a, V: Copy> Decode<'a, V> for Hops { } } +/// The minimum round trip to a route's origin node over moq-lite-07, in two parts. +/// Informational: it never ranks or selects a route. +/// +/// Each node on the path adds its link's min RTT, the lowest smoothed RTT the link +/// measured over the last 30 seconds, to the sum it received. The sum is unknown past +/// a node that withholds it or cannot measure its link, and on any other wire. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub struct MinRtt { + /// Past the session's peer: the sum the peer advertised, `None` when unknown. + pub upstream: Option, + /// The session's own link to the peer, `None` when its transport reports no RTT. + pub link: Option, +} + +impl MinRtt { + /// The whole path to the origin node, when both parts are known. + pub fn total(&self) -> Option { + self.upstream?.checked_add(self.link?) + } +} + /// The highest route cost, shared by every wire version. const MAX_COST: u64 = (1 << 62) - 1; @@ -512,6 +539,10 @@ pub struct RouteInfo { /// Whether a usable path reaches the route's origin; see [`Self::available`]. pub(crate) available: bool, + + /// The round trip to the route's origin node; see [`Self::min_rtt`]. Never forwarded: + /// each node measures its own link. + pub(crate) min_rtt: MinRtt, } impl RouteInfo { @@ -529,6 +560,7 @@ impl RouteInfo { node: None, draining: false, available: true, + min_rtt: MinRtt::default(), } } } @@ -566,6 +598,12 @@ impl RouteInfo { self.cost } + /// The minimum round trip to the route's origin node, as far as it is known; see + /// [`MinRtt`]. Unknown on a route this origin serves itself. + pub fn min_rtt(&self) -> MinRtt { + self.min_rtt + } + /// Whether a usable path reaches the route's origin. An unavailable route is held: /// the network lost every path to it but may recover one, so the announcement stays. /// It ranks with a draining route, below every available one, and requests through @@ -635,6 +673,13 @@ impl RouteInfo { self } + /// Set the round trip to the route's origin node; see [`Self::min_rtt`]. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn with_min_rtt(mut self, min_rtt: MinRtt) -> Self { + self.min_rtt = min_rtt; + self + } + /// Whether this route passed through an anonymous hop. /// /// True when the chain holds a 0 anywhere, including Lite03 hop-count @@ -847,8 +892,8 @@ type RouteOrder = ( Reverse, ); -/// The `(hops, cost, source, instance, epoch, auth, node, available)` metadata an announce -/// cursor delivers alongside a prefix. +/// The `(hops, cost, source, instance, epoch, auth, node, available, min_rtt)` metadata +/// an announce cursor delivers alongside a prefix. type RouteMeta = ( Hops, Cost, @@ -858,6 +903,7 @@ type RouteMeta = ( Option, Option, bool, + MinRtt, ); /// The earlier of two optional deadlines. @@ -1256,6 +1302,7 @@ impl OriginConsumerState { node: meta.6, draining: false, available: meta.7, + min_rtt: meta.8, }, }; (RouteEvent { kind, announce }, owed) @@ -1298,6 +1345,8 @@ struct RouteEntry { draining: bool, /// Whether a usable path reaches the route's origin; see [`RouteInfo::available`]. available: bool, + /// The round trip to the route's origin node; see [`RouteInfo::min_rtt`]. + min_rtt: MinRtt, /// The queue requests under this route are served from, when the announcer /// serves content on demand (a [`Claim`]). `None` for an advertise-only /// announcement ([`Producer::announce`]) and for a local broadcast. @@ -1366,12 +1415,13 @@ impl RouteEntry { self.auth.clone(), self.node, self.available, + self.min_rtt, ) } /// Whether [`Self::meta`] would equal `meta`, without building it. fn delivers(&self, meta: &RouteMeta) -> bool { - let (hops, cost, source, instance, epoch, auth, node, available) = meta; + let (hops, cost, source, instance, epoch, auth, node, available, min_rtt) = meta; *hops == self.hops && *cost == self.cost && *source == self.entered() @@ -1380,6 +1430,7 @@ impl RouteEntry { && *auth == self.auth && *node == self.node && *available == self.available + && *min_rtt == self.min_rtt } /// Whether a request for `path` can be served through this entry. A served @@ -2208,7 +2259,11 @@ impl Producer { let (tasks, set) = TaskSet::new(); let scope = OriginScope::default(); let timers = Deadlines::new(Instant::ZERO); - let shared = SendCell::new(kio::Shared::new(OriginState::new(config.update_hold, config.route_key))); + let shared = SendCell::new(kio::Shared::new(OriginState::new( + config.update_hold, + config.route_key, + config.withhold_min_rtt, + ))); let pool = config.pool.clone(); let producer = Self { hop: config.hop, @@ -2340,6 +2395,7 @@ impl Producer { cache_duration: self.cache_duration, update_hold: shared.update_hold, route_key: shared.route_key.clone(), + withhold_min_rtt: shared.withhold_min_rtt, } } @@ -2380,7 +2436,7 @@ impl Producer { scope: OriginScope::empty(), max_hops: MAX_HOP_LIMIT, root: Path::default(), - shared: SendCell::new(kio::Shared::new(OriginState::new(DEFAULT_UPDATE_HOLD, None))), + shared: SendCell::new(kio::Shared::new(OriginState::new(DEFAULT_UPDATE_HOLD, None, false))), pool: cache::Pool::default(), cache_duration: None, stats: stats::Session::default(), @@ -2973,6 +3029,7 @@ impl Announcing { node: route.node, draining: route.draining, available: route.available, + min_rtt: route.min_rtt, server: serving.server.clone(), source: serving.source.clone(), advertised: serving.advertised, @@ -3589,6 +3646,7 @@ impl AnnounceProducer { entry.node = route.node; entry.draining = route.draining; entry.available = route.available; + entry.min_rtt = route.min_rtt; entry.advertised = true; let claim = entry.claim.clone(); shared.sync_route(prefix, &claim, Gone::End); @@ -3650,6 +3708,7 @@ impl AnnounceProducer { node: entry.node, draining: entry.draining, available: entry.available, + min_rtt: entry.min_rtt, }) } @@ -6091,6 +6150,8 @@ struct OriginState { update_hold: Duration, // See [`Config::route_key`]. route_key: Option, + // See [`Config::withhold_min_rtt`]. + withhold_min_rtt: bool, // The announced routes, keyed by prefix. The table holds one entry per live // advertisement, not one per broadcast consumer. @@ -6134,10 +6195,11 @@ struct OriginState { } impl OriginState { - fn new(update_hold: Duration, route_key: Option) -> Self { + fn new(update_hold: Duration, route_key: Option, withhold_min_rtt: bool) -> Self { Self { update_hold, route_key, + withhold_min_rtt, routes: RouteTable::default(), next_route: 0, next_watch: 0, @@ -7346,7 +7408,8 @@ impl Consumer { .serving() }; // Built before taking the lock, since opening a cursor takes it. - let node = crate::route::Node::origin(self.hop()); + let withhold = self.shared.lock().withhold_min_rtt; + let node = crate::route::Node::origin(self.hop(), withhold); let claims = OwnClaims { routes: own(false).routes(), downstream: own(true).routes(), diff --git a/rs/moq-net/src/route/mod.rs b/rs/moq-net/src/route/mod.rs index 6a74dacea35..bb5f5851a91 100644 --- a/rs/moq-net/src/route/mod.rs +++ b/rs/moq-net/src/route/mod.rs @@ -18,6 +18,7 @@ pub(crate) mod node; mod proof; +mod rtt; mod view; #[cfg(test)] @@ -29,15 +30,17 @@ use std::{ collections::{BTreeMap, BTreeSet}, fmt, sync::Arc, + time::Duration, }; -use crate::Path; +use crate::{Path, origin::MinRtt}; pub(crate) use node::Node; #[cfg(feature = "async")] pub(crate) use proof::EXPORTER_LABEL; pub use proof::RouteKey; pub(crate) use proof::{Proof, Transcript}; +pub(crate) use rtt::LinkRtt; pub(crate) use view::{View, within}; use view::{ancestors, below, covering, narrow}; @@ -156,6 +159,10 @@ pub(crate) struct Advert { pub seqno: Seqno, pub metric: Metric, pub down_only: bool, + /// The sum of each link's min RTT to the identity: informational, never read to + /// rank or select a route. `None` once a node on the path withheld it or could not + /// measure its link. + pub min_rtt: Option, } /// A ROUTE's state on a stream. @@ -323,6 +330,8 @@ impl Heard { struct Peer { link: Link, + /// The link's min RTT, which routes learned here add; `None` while unmeasured. + min_rtt: Option, /// The neighbor sent GOAWAY: rank it below every other eligible route. draining: bool, /// The prefix the neighbor asked this node for, once it opened a stream. @@ -424,6 +433,8 @@ pub(crate) struct Router { changed: BTreeMap>, /// Mints a fresh identity on rotation. rng: u64, + /// Advertise every route without a min RTT. + withhold: bool, /// Per claim this node has shown, how many times it forgot that claim. #[cfg(test)] forgot: BTreeMap<(Id, Path), u32>, @@ -468,6 +479,7 @@ impl Router { ctxs: BTreeMap::new(), changed: BTreeMap::new(), rng: seed | 1, + withhold: false, #[cfg(test)] forgot: BTreeMap::new(), #[cfg(test)] @@ -477,6 +489,12 @@ impl Router { } } + /// Advertise every route without a min RTT, so none crossing this node carries one. + pub fn withhold(&mut self) { + self.withhold = true; + self.update_all(); + } + /// Take the identities recomputed since the last call, whose selection may have /// changed, each with the claims whose shown state did. pub fn changed(&mut self) -> BTreeMap> { @@ -504,6 +522,7 @@ impl Router { neighbor, Peer { link, + min_rtt: None, draining: false, watch: None, asked: None, @@ -570,13 +589,28 @@ impl Router { if let Some(peer) = self.neighbors.get_mut(&neighbor) { peer.draining = true; } - let ids: Vec = self - .rx - .iter() + self.update(self.through(neighbor)); + } + + /// The link to `neighbor` has `min_rtt`, or `None` unmeasured: every route learned + /// there adds it. + pub fn min_rtt(&mut self, neighbor: N, min_rtt: Option) { + let Some(peer) = self.neighbors.get_mut(&neighbor) else { + return; + }; + if peer.min_rtt == min_rtt { + return; + } + peer.min_rtt = min_rtt; + self.update(self.through(neighbor)); + } + + /// The identities `neighbor` sent a route for. + fn through(&self, neighbor: N) -> Vec { + (self.rx.iter()) .filter(|(_, routes)| routes.contains_key(&neighbor)) .map(|(id, _)| *id) - .collect(); - self.update(ids); + .collect() } #[cfg(test)] @@ -1032,18 +1066,39 @@ impl Router { best.map(|(_, selected)| selected) } - /// Every usable feasible route to `id`, with this node's metric through each. Any - /// of them forwards without a loop, if it [carries](Self::carries) the claim. - pub fn routes(&self, id: Id) -> Vec<(N, Metric)> { + /// Every usable feasible route to `id`, with this node's metric and min RTT + /// through each. Any of them forwards without a loop, if it + /// [carries](Self::carries) the claim. + pub fn routes(&self, id: Id) -> Vec<(N, Metric, MinRtt)> { if self.origins.contains_key(&id) { return Vec::new(); } (self.rx.get(&id).into_iter().flatten()) .filter(|(_, entry)| !entry.held && self.feasible(id, entry.advert)) - .filter_map(|(&next, entry)| Some((next, add(entry.advert.metric, self.neighbors[&next].link.cost)?))) + .filter_map(|(&next, entry)| { + let peer = &self.neighbors[&next]; + let min_rtt = MinRtt { + upstream: entry.advert.min_rtt, + link: peer.min_rtt, + }; + Some((next, add(entry.advert.metric, peer.link.cost)?, min_rtt)) + }) .collect() } + /// This node's min RTT to `id` along `route`: 0 at the identity's own node. + fn route_min_rtt(&self, id: Id, route: &Selected) -> Option { + let Some(next) = route.next else { + return Some(Duration::ZERO); + }; + let entry = self.rx.get(&id)?.get(&next)?; + MinRtt { + upstream: entry.advert.min_rtt, + link: self.neighbors[&next].min_rtt, + } + .total() + } + /// Whether a request for a claim at `claim` may go through `next`. pub fn carries(&self, next: N, claim: &Path) -> bool { self.neighbors.get(&next).is_some_and(|peer| peer.carries(claim)) @@ -1621,6 +1676,7 @@ impl Router { seqno: route.seqno, metric: route.metric, down_only: route.down_only, + min_rtt: self.route_min_rtt(id, &route).filter(|_| !self.withhold), })), None if serves && ctx.live && was.is_some() => Some(Reach::Held), _ => None, diff --git a/rs/moq-net/src/route/node.rs b/rs/moq-net/src/route/node.rs index c17fc75ddbc..c65a2e1a7b5 100644 --- a/rs/moq-net/src/route/node.rs +++ b/rs/moq-net/src/route/node.rs @@ -25,6 +25,7 @@ use crate::{Instance, Path}; use rand::RngExt; use super::{Error, GENERATED, Id, Link, Metric, Out, QUALIFIED, Reach, Rev, Router, Seqno}; +use crate::origin::MinRtt; /// One announce stream in one direction. pub(crate) type Key = u64; @@ -58,6 +59,8 @@ pub(crate) struct Served { /// The selected route's metric: a path through the claim costs the Announce /// Cost plus this. pub metric: Metric, + /// The route's min RTT, informational. + pub min_rtt: MinRtt, /// Whether a usable route serves it; a held one leaves it shown but unavailable. pub available: bool, } @@ -356,8 +359,9 @@ struct State { routes: HashMap, } -/// An identity's usable feasible routes: each stream, its metric, and the peer's name. -type Routes = Vec<(Key, Metric, Id)>; +/// An identity's usable feasible routes: each stream, its metric and min RTT, and +/// the peer's name. +type Routes = Vec<(Key, Metric, MinRtt, Id)>; /// Where one shown claim is served: each stream, what it was last sent, and /// whether it confirmed serving it. @@ -420,11 +424,16 @@ impl Node { /// A node declaring the origin's hop id, so the origin has one identity on every /// wire. A declared identity is never qualified, so an id carrying that marker - /// loses it, or is replaced by a random one if nothing is left. - pub fn origin(hop: std::num::NonZeroU64) -> Self { + /// loses it, or is replaced by a random one if nothing is left. One that + /// `withholds` advertises no min RTT. + pub fn origin(hop: std::num::NonZeroU64, withhold: bool) -> Self { let mut rng = kio::clock::rng(); let id = declared_for(hop).unwrap_or_else(|| rng.random_range(1..=GENERATED)); - Self::new(id, rng.random()) + let node = Self::new(id, rng.random()); + if withhold { + node.lock().router.withhold(); + } + node } fn lock(&self) -> Locked<'_> { @@ -588,6 +597,11 @@ impl Stream { self.with(|router, key| router.goaway(key)); } + /// The link to the peer has `min_rtt`, or `None` unmeasured. + pub fn min_rtt(&self, min_rtt: Option) { + self.with(|router, key| router.min_rtt(key, min_rtt)); + } + /// This stream's session now serves `claim` of `id`, so a stream that no longer /// carries a route to it lets go. pub fn serving(&self, id: Id, claim: &Path) { @@ -679,7 +693,7 @@ impl State { /// the routes changed too. fn home(&mut self, id: Id, claims: BTreeSet, ended: &BTreeMap<(Id, Path), bool>) { let routes: Routes = (self.router.routes(id).into_iter()) - .map(|(key, metric)| (key, metric, self.router.name(key, id))) + .map(|(key, metric, min_rtt)| (key, metric, min_rtt, self.router.name(key, id))) .collect(); let last = self.routes.get(&id).map_or(&[][..], Vec::as_slice); let claims = match last == routes.as_slice() { @@ -704,28 +718,29 @@ impl State { for claim in shown.into_iter().rev() { let shown = &self.router.shown()[&(id, claim.clone())]; let (item, available) = (shown.item.clone(), shown.available); - let carrying: Vec<(Key, Metric, Id)> = (routes.iter()) - .filter(|(key, _, _)| self.router.carries(*key, &claim)) + let carrying: Vec<(Key, Metric, MinRtt, Id)> = (routes.iter()) + .filter(|(key, ..)| self.router.carries(*key, &claim)) .copied() .collect(); let home = self.homes.entry((id, claim.clone())).or_default(); // With no route left, the router no longer knows the name a stream's peer // gave the identity, so it keeps the one it was served under. - let wanted: Vec<(Key, Metric, bool, Id)> = match carrying.is_empty() { + let wanted: Vec<(Key, Metric, MinRtt, bool, Id)> = match carrying.is_empty() { true => (home.keys.iter()) - .map(|(key, slot)| (*key, slot.served.metric, false, slot.named)) + .map(|(key, slot)| (*key, slot.served.metric, slot.served.min_rtt, false, slot.named)) .collect(), false => (carrying.into_iter()) - .map(|(key, metric, named)| (key, metric, available, named)) + .map(|(key, metric, min_rtt, named)| (key, metric, min_rtt, available, named)) .collect(), }; for slot in home.keys.values_mut() { slot.leaving = true; } - for (key, metric, available, named) in wanted { + for (key, metric, min_rtt, available, named) in wanted { let served = Served { claim: item.clone(), metric, + min_rtt, available, }; match home.keys.get_mut(&key) { @@ -802,7 +817,7 @@ impl State { let home = self.homes.get(&(id, claim.clone())); let slots = home.into_iter().flat_map(|home| home.keys.iter()); let carrying: Vec<_> = (routes.iter()) - .filter(|(key, _, _)| self.router.carries(*key, claim)) + .filter(|(key, ..)| self.router.carries(*key, claim)) .collect(); if carrying.is_empty() { for (_, slot) in slots { @@ -813,10 +828,11 @@ impl State { } let slots: BTreeMap = slots.filter(|(_, slot)| !slot.leaving).map(|(k, s)| (*k, s)).collect(); let want: BTreeMap = (carrying.iter()) - .map(|&&(key, metric, named)| { + .map(|&&(key, metric, min_rtt, named)| { let served = Served { claim: shown.item.clone(), metric, + min_rtt, available: shown.available, }; (key, (served, named)) @@ -919,6 +935,10 @@ mod tests { served: Served { claim: claim(&instance, 5), metric: 1, + min_rtt: MinRtt { + upstream: Some(std::time::Duration::ZERO), + link: None, + }, available: true, }, }] @@ -1203,6 +1223,7 @@ mod tests { seqno, metric: 0, down_only: false, + min_rtt: None, })) }; for origin in 10..110 { @@ -1326,6 +1347,7 @@ mod tests { seqno: 0, metric: 0, down_only: false, + min_rtt: None, }; receive.route(1, Some(Reach::Finite(advert))); receive.batch(1, vec![(p("live/x"), Some(claim(&instance, 0)))], Some(1)); diff --git a/rs/moq-net/src/route/rtt.rs b/rs/moq-net/src/route/rtt.rs new file mode 100644 index 00000000000..f701c82e96e --- /dev/null +++ b/rs/moq-net/src/route/rtt.rs @@ -0,0 +1,125 @@ +//! A link's min RTT, as the route layer adds it to the routes learned over the link. + +use std::{collections::VecDeque, time::Duration}; + +/// Reads a link's smoothed RTT once per [`Self::INTERVAL`] and reports the lowest of the +/// last 30 reads in whole milliseconds, changed only once it moves past the larger of +/// 5 ms and a quarter of the value reported. +/// +/// The smoothed RTT follows every queue, and a lifetime minimum never rises after a +/// path change; a windowed minimum ignores a queue shorter than the window and turns a +/// path change into one new value. +#[derive(Debug, Default)] +pub(crate) struct LinkRtt { + reads: VecDeque, + reported: Option, +} + +impl LinkRtt { + /// How often to [read](Self::read) the smoothed RTT. + pub const INTERVAL: Duration = Duration::from_secs(1); + /// How many reads the minimum covers. + const WINDOW: usize = 30; + /// The smallest change reported. + const FLOOR: Duration = Duration::from_millis(5); + + /// The min RTT last reported. + pub fn reported(&self) -> Option { + self.reported + } + + /// Take one read of the smoothed RTT, `None` when the transport reports none, and + /// return the min RTT to report. + pub fn read(&mut self, rtt: Option) -> Option { + let Some(rtt) = rtt else { + self.reads.clear(); + self.reported = None; + return None; + }; + if self.reads.len() == Self::WINDOW { + self.reads.pop_front(); + } + self.reads.push_back(rtt); + let min = self.reads.iter().min().expect("just pushed"); + let min = Duration::from_millis(u64::try_from(min.as_micros().div_ceil(1000)).unwrap_or(u64::MAX)); + match self.reported { + Some(reported) if min.abs_diff(reported) <= (reported / 4).max(Self::FLOOR) => {} + _ => self.reported = Some(min), + } + self.reported + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn ms(ms: u64) -> Option { + Some(Duration::from_millis(ms)) + } + + /// Feed `reads` one per interval, returning each report. + fn feed(link: &mut LinkRtt, reads: impl IntoIterator>) -> Vec> { + reads.into_iter().map(|rtt| link.read(rtt)).collect() + } + + #[test] + fn the_first_read_reports_at_once() { + let mut link = LinkRtt::default(); + assert_eq!(link.read(ms(30)), ms(30)); + } + + #[test] + fn a_queue_shorter_than_the_window_reports_nothing() { + let mut link = LinkRtt::default(); + link.read(ms(30)); + // 29 seconds of a growing queue, then it drains. + let queued = (1..30).map(|i| ms(30 + i * 3)); + assert!(feed(&mut link, queued).iter().all(|report| *report == ms(30))); + assert_eq!(link.read(ms(31)), ms(30)); + } + + #[test] + fn a_rise_reports_once_the_window_passes_it() { + let mut link = LinkRtt::default(); + link.read(ms(30)); + let reports = feed(&mut link, std::iter::repeat_n(ms(55), 30)); + // The window still holds the last 30 ms read until the 30th read of 55 ms. + assert!(reports[..29].iter().all(|report| *report == ms(30))); + assert_eq!(reports[29], ms(55)); + } + + #[test] + fn a_drop_reports_at_once() { + let mut link = LinkRtt::default(); + link.read(ms(80)); + assert_eq!(link.read(ms(40)), ms(40)); + } + + #[test] + fn a_change_within_the_band_reports_nothing() { + let mut link = LinkRtt::default(); + link.read(ms(10)); + // A quarter of 10 ms is below the 5 ms floor. + assert_eq!(link.read(ms(5)), ms(10)); + assert_eq!(link.read(ms(4)), ms(4)); + let mut link = LinkRtt::default(); + link.read(ms(100)); + assert_eq!(link.read(ms(75)), ms(100)); + assert_eq!(link.read(ms(74)), ms(74)); + } + + #[test] + fn a_transport_without_rtt_reports_none_and_starts_over() { + let mut link = LinkRtt::default(); + link.read(ms(80)); + assert_eq!(link.read(None), None); + assert_eq!(link.read(ms(90)), ms(90)); + } + + #[test] + fn a_fraction_of_a_millisecond_rounds_up() { + let mut link = LinkRtt::default(); + assert_eq!(link.read(Some(Duration::from_micros(300))), ms(1)); + } +} diff --git a/rs/moq-net/src/route/tests/min_rtt.rs b/rs/moq-net/src/route/tests/min_rtt.rs new file mode 100644 index 00000000000..1b00ba27997 --- /dev/null +++ b/rs/moq-net/src/route/tests/min_rtt.rs @@ -0,0 +1,109 @@ +//! The min RTT a route carries: each node adds its link's to the sum it received, and +//! nothing ranks or selects a route by it. + +use std::time::Duration; + +use super::*; + +fn originate(sim: &mut Sim, node: Node, id: Id) { + sim.originate(node, id, 0); + sim.announce(node, "", 0, None); +} + +fn ms(ms: u64) -> Option { + Some(Duration::from_millis(ms)) +} + +/// The min RTT `node` advertises to `peer` for `id`. +fn advertised(sim: &Sim, node: Node, peer: Node, id: Id) -> Option { + match sim.sent(node, peer, id) { + Some(Reach::Finite(advert)) => advert.min_rtt, + other => panic!("{node} sends {peer} {other:?}"), + } +} + +/// A chain origin, a, b, c: each relay measures its link toward the origin. +fn chain() -> Sim { + let (origin, a, b, c) = (0, 1, 2, 3); + let mut sim = Sim::new(4); + sim.join(origin, a, 1); + sim.join(a, b, 1); + sim.join(b, c, 1); + sim.min_rtt(a, origin, Some(10)); + sim.min_rtt(b, a, Some(20)); + sim.min_rtt(c, b, Some(30)); + originate(&mut sim, origin, 7); + assert!(sim.run()); + sim +} + +#[test] +fn each_relay_adds_its_link_to_the_sum() { + let (origin, a, b, c) = (0, 1, 2, 3); + let sim = chain(); + assert_eq!(advertised(&sim, origin, a, 7), ms(0), "the origin's own route"); + assert_eq!(advertised(&sim, a, b, 7), ms(10)); + assert_eq!(advertised(&sim, b, c, 7), ms(30)); + let routes = sim.nodes[c].routes(7); + let [(next, _, min_rtt)] = routes[..] else { + panic!("one route: {routes:?}"); + }; + assert_eq!(next, b); + assert_eq!((min_rtt.upstream, min_rtt.link, min_rtt.total()), (ms(30), ms(30), ms(60))); +} + +/// A shift on one link sends one ROUTE_UPDATE on each stream carrying the route from +/// there on, and the same value again sends nothing. +#[test] +fn a_shift_updates_each_stream_downstream_once() { + let (origin, a, b, c) = (0, 1, 2, 3); + let mut sim = chain(); + let downstream: usize = [a, b, c].iter().map(|&node| sim.sending(node, 7)).sum(); + let before = sim.stats.route_msgs; + sim.min_rtt(a, origin, Some(40)); + assert!(sim.run()); + assert_eq!(sim.stats.route_msgs - before, downstream); + assert_eq!(advertised(&sim, b, c, 7), ms(60)); + + let before = sim.stats.route_msgs; + sim.min_rtt(a, origin, Some(40)); + assert!(sim.run()); + assert_eq!(sim.stats.route_msgs, before); +} + +/// A link with no measurement, or a node that withholds, leaves the sum unknown from +/// there on, while the nodes before it still carry theirs. +#[test] +fn an_unknown_link_or_a_withholding_node_ends_the_sum() { + let (a, b, c) = (1, 2, 3); + let mut sim = chain(); + sim.min_rtt(b, a, None); + assert!(sim.run()); + assert_eq!(advertised(&sim, a, b, 7), ms(10)); + assert_eq!(advertised(&sim, b, c, 7), None); + + let mut sim = chain(); + sim.withhold(b); + assert!(sim.run()); + assert_eq!(advertised(&sim, b, c, 7), None); + let routes = sim.nodes[c].routes(7); + assert_eq!(routes[0].2.link, ms(30), "c still measures its own link"); + assert_eq!(routes[0].2.total(), None); +} + +/// The metric selects; a far lower min RTT through a costlier link changes nothing. +#[test] +fn min_rtt_never_selects_a_route() { + let (origin, cheap, fast, edge) = (0, 1, 2, 3); + let mut sim = Sim::new(4); + sim.join(origin, cheap, 1); + sim.join(origin, fast, 5); + sim.join(cheap, edge, 1); + sim.join(fast, edge, 1); + sim.min_rtt(cheap, origin, Some(200)); + sim.min_rtt(fast, origin, Some(1)); + originate(&mut sim, origin, 7); + assert!(sim.run()); + assert_eq!(sim.route(edge, 7).and_then(|route| route.next), Some(cheap)); + assert_eq!(advertised(&sim, cheap, edge, 7), ms(200)); +} diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index f3060404833..7e9756b986c 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -6,6 +6,7 @@ //! lite-07 claim already covers the paths below it, so the slash is dropped. mod handoff; +mod min_rtt; mod routing; use std::{ @@ -374,6 +375,27 @@ impl Sim { self.flush(); } + /// `node` measures its link to `peer` at `ms`, or nothing. + pub fn min_rtt(&mut self, node: Node, peer: Node, ms: Option) { + self.nodes[node].min_rtt(peer, ms.map(std::time::Duration::from_millis)); + self.flush(); + } + + pub fn withhold(&mut self, node: Node) { + self.nodes[node].withhold(); + self.flush(); + } + + /// What `node` last sent `peer` for `id`. + pub fn sent(&self, node: Node, peer: Node, id: Id) -> Option { + self.nodes[node].sent.get(&(id, peer)).copied() + } + + /// The streams `node` sends a route for `id` on. + pub fn sending(&self, node: Node, id: Id) -> usize { + Router::::of(&self.nodes[node].sent, id).count() + } + pub fn hold(&mut self, from: Node, to: Node) { self.held.insert((from, to)); } diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index 4f948014415..6666a4770f2 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -783,6 +783,7 @@ fn a_newer_request_raises_the_pending_one() { seqno: 0, metric: 1, down_only: false, + min_rtt: None, }; relay.route(up, 7, Some(Reach::Finite(advert))); relay.drain(); @@ -870,6 +871,7 @@ fn an_overflowing_route_neither_certifies_nor_holds() { seqno: 0, metric, down_only: false, + min_rtt: None, })) }; let item = Item { @@ -921,6 +923,7 @@ fn a_fresh_uncertified_claim_waits_on_an_older_floor() { seqno: 0, metric, down_only: false, + min_rtt: None, })) }; let item = Item { diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index 844ee1e1ba6..090d96f4de9 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -767,6 +767,18 @@ pub struct Config { )] pub max_hops: Option, + /// Advertise no min RTT on the routes this relay sends moq-lite-07 peers, so a + /// route crossing it reports none from here on. Boolean flag: pass + /// `--cluster-withhold-min-rtt` (or `=true` / `=false`). + #[usage( + name = "cluster-withhold-min-rtt", + long = "cluster-withhold-min-rtt", + env = "MOQ_CLUSTER_WITHHOLD_MIN_RTT", + setting = "cluster.withhold_min_rtt", + bool_value + )] + pub withhold_min_rtt: bool, + /// Connect to one or more other cluster nodes. Each entry is a full URL, e.g. /// `https://host/?jwt=TOKEN`, or an object with `url`, `cost`, `egress`, /// `token`, and `upstream`. A bare host or `host:port` is refused; pass the @@ -1466,6 +1478,7 @@ impl Cluster { .into(), None => origin::RouteKey::generate(), }); + origin_config.withhold_min_rtt = config.withhold_min_rtt; if let Some(max) = config.max_hops { origin_config.max_hops = NonZeroUsize::new(max) .filter(|max| max.get() <= 32) diff --git a/rs/moq-relay/src/settings.rs b/rs/moq-relay/src/settings.rs index 0f6aece673a..333a0323785 100644 --- a/rs/moq-relay/src/settings.rs +++ b/rs/moq-relay/src/settings.rs @@ -77,6 +77,9 @@ struct Cluster { #[usage(env = "MOQ_CLUSTER_MAX_HOPS", cli("--cluster-max-hops"))] max_hops: Option, + #[usage(env = "MOQ_CLUSTER_WITHHOLD_MIN_RTT", cli("--cluster-withhold-min-rtt"))] + withhold_min_rtt: Option, + #[usage(env = "MOQ_CLUSTER_CONNECT", cli("--cluster-connect"), parse = "list_by_comma")] connect: Option>, diff --git a/rs/moq-tokio/tests/integration/broadcast.rs b/rs/moq-tokio/tests/integration/broadcast.rs index 07eaf63b1ce..97e055ea043 100644 --- a/rs/moq-tokio/tests/integration/broadcast.rs +++ b/rs/moq-tokio/tests/integration/broadcast.rs @@ -952,6 +952,75 @@ async fn announce_lifecycle_test(version: &str) { .expect("server task failed"); } +/// A lite-07 route carries the publisher's min RTT, 0 at its own node, and the +/// subscriber adds its own link's; a publisher that withholds it sends none. +#[tokio::test] +async fn broadcast_moq_lite_07_route_min_rtt() { + let reported = first_route(moq_net::origin::Config::default()).await; + let min_rtt = reported.min_rtt(); + assert_eq!(min_rtt.upstream, Some(std::time::Duration::ZERO)); + let link = min_rtt.link.expect("QUIC reports an RTT"); + assert!(link > std::time::Duration::ZERO, "rounded up to a whole millisecond"); + assert_eq!(min_rtt.total(), Some(link)); + + let mut config = moq_net::origin::Config::default(); + config.withhold_min_rtt = true; + let withheld = first_route(config).await; + assert_eq!(withheld.min_rtt().upstream, None); + assert!(withheld.min_rtt().link.is_some(), "the subscriber still measures its link"); + assert_eq!(withheld.min_rtt().total(), None); +} + +/// The route a lite-07 subscriber first sees for a broadcast an origin built from +/// `config` serves. +async fn first_route(config: moq_net::origin::Config) -> moq_net::origin::RouteInfo { + let version = "moq-lite-07-wip"; + let pub_origin = moq_tokio::origin::spawn_config(config); + let _route = pub_origin + .route("room/first", moq_net::origin::RouteOptions::default()) + .expect("create broadcast"); + + let mut server_config = moq_tokio::listen::Config::default(); + server_config.bind = Some("[::]:0".parse().unwrap()); + server_config.tls.generate = vec!["localhost".into()]; + server_config.version = vec![version.parse().unwrap()]; + let server = server_config.init(Default::default()).expect("init server"); + let mut server = server.listen().await.expect("failed to listen"); + let addr = server.local_addr().expect("local addr"); + + let sub_origin = moq_tokio::origin::spawn(); + let mut announcements = sub_origin.consume().announced(); + + let mut client_config = moq_tokio::connect::Config::default(); + client_config.tls.insecure = Some(true); + client_config.version = vec![version.parse().unwrap()]; + let client = client_config.init(Default::default()).expect("init client"); + let url: url::Url = format!("moqt://localhost:{}", addr.port()).parse().unwrap(); + + let server_handle = tokio::spawn(async move { + let request = server.accept().await.expect("accept"); + let session = request.with_publisher(&pub_origin).ok().await?; + let _ = session.closed().await; + Ok::<_, anyhow::Error>(()) + }); + + let client = client.with_subscriber(sub_origin); + let (_client, connection) = kio::clock::timeout(TIMEOUT, connect_once(client, url)) + .await + .expect("connect timeout") + .expect("connect failed"); + let (update, active) = next_announce(&mut announcements).await; + assert_eq!(update.prefix.as_str(), "room/first"); + assert!(active, "expected the initial announce"); + + drop(connection); + server_handle + .await + .expect("server task panicked") + .expect("server task failed"); + update.route +} + /// Read `count` groups (one frame each) off the subscriber, returning the sorted /// payloads. Sorted because delivery order across groups isn't guaranteed. async fn read_payloads(sub: &mut moq_net::track::Subscriber, count: usize) -> Vec { diff --git a/swift/Sources/Moq/Aliases.swift b/swift/Sources/Moq/Aliases.swift index 1c6f74ba093..973eaa39a0b 100644 --- a/swift/Sources/Moq/Aliases.swift +++ b/swift/Sources/Moq/Aliases.swift @@ -57,6 +57,9 @@ public typealias Instance = MoqFFI.MoqInstance /// The read-only facts on a received route: the `Instance` of the stream it serves, /// relay hop ids (oldest first), and its static production and link cost, lower wins. public typealias RouteInfo = MoqFFI.MoqRouteInfo +/// The minimum round trip to a route's origin node over moq-lite-07: `upstream` past the +/// session's peer, `link` to the peer, and their `total`. Informational; it never ranks a route. +public typealias MinRtt = MoqFFI.MoqMinRtt /// A route over a prefix: the origin-relative `prefix`, what each filter /// wildcard matched (`captures`, a required list, empty without wildcards), and the `route` serving it. /// Resolve a path with `OriginConsumer.request`. From 4af6bc0c9170b453299f83e6b2d8eeb30ace4ad8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 13:01:38 -0800 Subject: [PATCH 04/11] fix(net): gate the link RTT sampler on async, and format Co-Authored-By: Claude Opus 5.5 --- py/moq-rs/moq/types.py | 6 +++--- rs/moq-net/src/route/mod.rs | 1 + rs/moq-net/src/route/tests/min_rtt.rs | 5 ++++- rs/moq-tokio/tests/integration/broadcast.rs | 5 ++++- 4 files changed, 12 insertions(+), 5 deletions(-) diff --git a/py/moq-rs/moq/types.py b/py/moq-rs/moq/types.py index a1b6acb4421..20add55f6d3 100644 --- a/py/moq-rs/moq/types.py +++ b/py/moq-rs/moq/types.py @@ -32,6 +32,9 @@ from moq_ffi import ( MoqInstance as Instance, ) +from moq_ffi import ( + MoqMinRtt as MinRtt, +) from moq_ffi import ( MoqOrder as Order, ) @@ -47,9 +50,6 @@ from moq_ffi import ( MoqRateTarget as RateTarget, ) -from moq_ffi import ( - MoqMinRtt as MinRtt, -) from moq_ffi import ( MoqRouteInfo as RouteInfo, ) diff --git a/rs/moq-net/src/route/mod.rs b/rs/moq-net/src/route/mod.rs index bb5f5851a91..727a42c398f 100644 --- a/rs/moq-net/src/route/mod.rs +++ b/rs/moq-net/src/route/mod.rs @@ -40,6 +40,7 @@ pub(crate) use node::Node; pub(crate) use proof::EXPORTER_LABEL; pub use proof::RouteKey; pub(crate) use proof::{Proof, Transcript}; +#[cfg(feature = "async")] pub(crate) use rtt::LinkRtt; pub(crate) use view::{View, within}; use view::{ancestors, below, covering, narrow}; diff --git a/rs/moq-net/src/route/tests/min_rtt.rs b/rs/moq-net/src/route/tests/min_rtt.rs index 1b00ba27997..6f7e8810f18 100644 --- a/rs/moq-net/src/route/tests/min_rtt.rs +++ b/rs/moq-net/src/route/tests/min_rtt.rs @@ -49,7 +49,10 @@ fn each_relay_adds_its_link_to_the_sum() { panic!("one route: {routes:?}"); }; assert_eq!(next, b); - assert_eq!((min_rtt.upstream, min_rtt.link, min_rtt.total()), (ms(30), ms(30), ms(60))); + assert_eq!( + (min_rtt.upstream, min_rtt.link, min_rtt.total()), + (ms(30), ms(30), ms(60)) + ); } /// A shift on one link sends one ROUTE_UPDATE on each stream carrying the route from diff --git a/rs/moq-tokio/tests/integration/broadcast.rs b/rs/moq-tokio/tests/integration/broadcast.rs index 97e055ea043..5d80f79134c 100644 --- a/rs/moq-tokio/tests/integration/broadcast.rs +++ b/rs/moq-tokio/tests/integration/broadcast.rs @@ -967,7 +967,10 @@ async fn broadcast_moq_lite_07_route_min_rtt() { config.withhold_min_rtt = true; let withheld = first_route(config).await; assert_eq!(withheld.min_rtt().upstream, None); - assert!(withheld.min_rtt().link.is_some(), "the subscriber still measures its link"); + assert!( + withheld.min_rtt().link.is_some(), + "the subscriber still measures its link" + ); assert_eq!(withheld.min_rtt().total(), None); } From c28ca280de142beb0662a91670fbc894e241904e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 13:13:15 -0800 Subject: [PATCH 05/11] fix(net): stop the link RTT timer on a transport that reports none A transport with no RTT never starts reporting one, and a perpetual 1 Hz timer kept the simulated clock from ever going idle. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/lite/subscriber/routed.rs | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/rs/moq-net/src/lite/subscriber/routed.rs b/rs/moq-net/src/lite/subscriber/routed.rs index 6d9a16e761a..24009cb8056 100644 --- a/rs/moq-net/src/lite/subscriber/routed.rs +++ b/rs/moq-net/src/lite/subscriber/routed.rs @@ -99,8 +99,8 @@ pub(super) struct Routed { drained: bool, /// The link's min RTT, which the peer's routes add. rtt: route::LinkRtt, - /// When to read the link's RTT next. - read_rtt: crate::time::Deadline, + /// When to read the link's RTT next, or `None` once the transport reported none. + read_rtt: Option, } impl Routed { @@ -134,7 +134,10 @@ impl Routed { .receive(scoped(&link, ®ions, key), prefix.clone()) .map_err(refused)?; let mut rtt = route::LinkRtt::default(); - stream.min_rtt(rtt.read(subscriber.session.metrics().rtt())); + let first = rtt.read(subscriber.session.metrics().rtt()); + stream.min_rtt(first); + // A transport that reports no RTT never starts to, so it costs no timer. + let read_rtt = first.map(|_| crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL)); Ok(Self { node, stream, @@ -158,7 +161,7 @@ impl Routed { ready: Default::default(), drained: false, rtt, - read_rtt: crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL), + read_rtt, }) } @@ -508,13 +511,16 @@ impl Routed { ) -> Result<(), Error> { // Only a route the peer sent adds the link's min RTT, so a peer sending none // costs no timer. - while !self.routes.is_empty() && self.read_rtt.poll(waiter).is_ready() { + while !self.routes.is_empty() + && let Some(read_rtt) = &mut self.read_rtt + && read_rtt.poll(waiter).is_ready() + { let last = self.rtt.reported(); let now = self.rtt.read(subscriber.session.metrics().rtt()); if now != last { self.stream.min_rtt(now); } - self.read_rtt = crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL); + self.read_rtt = now.map(|_| crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL)); } while let Poll::Ready(output) = self.stream.poll_output(waiter) { self.output(subscriber, output); From d99d0e65dcae839796c12c315a95529f15764205 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 13:38:23 -0800 Subject: [PATCH 06/11] fix(net): forward a Min RTT past the field's range as unknown A peer's largest Min RTT plus a link exceeded u64 milliseconds, which failed the encode and ended the downstream announce stream. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 1 + rs/moq-net/src/lite/announce.rs | 26 +++++++++++++++++++++++--- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 80f682f4f96..e1ba3ca274d 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -500,6 +500,7 @@ A link min RTT is the lowest smoothed RTT [RFC9002] the connection measured over A lifetime minimum never rises after a path change, so it is not a link min RTT. A node SHOULD keep the link min RTT it adds until the lowest reading differs from it by more than the larger of 5 ms and a quarter of it, so jitter and short queues send no ROUTE_UPDATE and a lasting change sends one per route. A node MAY withhold Min RTT, and a node whose connection reports no RTT has no link min RTT; either sends its routes without one, and a route without one stays without one past every later node. +A sum too large to encode is sent without one too. An endpoint learns its round trip to an origin node by adding its own link min RTT to the Min RTT of the route it selected; without its own, the Min RTT alone covers every hop past its first. Like the Metric, a Min RTT is unverified: a peer can misstate only what crosses it. diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index e0e690a9ab0..5d18bd225ad 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -127,16 +127,20 @@ fn encode_advert(w: &mut Encoder<'_>, advert: &Advert) -> Result<(), EncodeError } w.varint(advert.seqno)?; w.varint(advert.metric)?; + // A sum past the field's range, which only a peer's absurd value reaches, is unknown. + let min_rtt = advert + .min_rtt + .and_then(|min_rtt| u64::try_from(min_rtt.as_millis()).ok()); let mut flags = 0; if advert.down_only { flags |= DOWN_ONLY; } - if advert.min_rtt.is_some() { + if min_rtt.is_some() { flags |= MIN_RTT; } w.u8(flags); - if let Some(min_rtt) = advert.min_rtt { - w.varint(u64::try_from(min_rtt.as_millis()).map_err(|_| EncodeError::InvalidState)?)?; + if let Some(min_rtt) = min_rtt { + w.varint(min_rtt)?; } Ok(()) } @@ -1638,6 +1642,22 @@ mod tests { } } + /// A peer's largest Min RTT plus a link goes past the field: it is forwarded as + /// unknown, never an error that would end the stream. + #[test] + fn a_min_rtt_past_the_field_is_unknown() { + let msg = |min_rtt| AnnounceBroadcast::RouteUpdate { + id: 1, + advert: Advert { + min_rtt, + ..advert(2, 3, false) + }, + }; + let past = std::time::Duration::from_millis(u64::MAX).checked_add(std::time::Duration::from_millis(1)); + assert!(past.is_some()); + assert_eq!(broadcast_round_trip(&msg(past), Version::Lite07), msg(None)); + } + /// A floor is a marker: the Route ID and the revision plus one, 0 when it certifies /// nothing. #[test] From c2468ba61339b8c973bd3ab3f29ba827b36a11f2 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Tue, 10 Nov 2026 18:39:33 -0800 Subject: [PATCH 07/11] feat(net)!: name the route's round trip `rtt` and drop withholding `origin::RouteInfo::rtt()` returns the whole path as `Option`, replacing `origin::MinRtt`; the bindings' `RouteInfo.rtt` and the draft's `RTT (i)` field follow. `--cluster-withhold-min-rtt` and `origin::Config::withhold_min_rtt` are removed: the value is informational. The link sampler keeps reading after a read of no RTT, since qmux reports its first only after a ping, and stops after a window of misses. Co-Authored-By: Claude Opus 5.5 --- dart/moq/lib/src/aliases.dart | 6 +- dart/moq_ffi/lib/src/moq.dart | 91 ++----------------- doc/bin/relay/cluster.md | 20 ++-- doc/bin/relay/config.md | 1 - drafts/draft-lcurley-moq-lite.md | 34 +++---- go/wrapper/types.go | 6 +- .../kotlin/dev/moq/Aliases.kt | 9 +- py/moq-rs/moq/__init__.py | 2 - py/moq-rs/moq/types.py | 4 - quest/m2/README.md | 2 +- quest/m2/lite07-finalize.md | 2 +- quest/m2/route-latency.md | 30 +++--- rs/moq-ffi/src/origin.rs | 32 +------ rs/moq-net/src/fuzz.rs | 2 +- rs/moq-net/src/lite/announce.rs | 40 ++++---- rs/moq-net/src/lite/subscriber.rs | 2 +- rs/moq-net/src/lite/subscriber/routed.rs | 50 ++++++++-- rs/moq-net/src/model/mod.rs | 4 +- rs/moq-net/src/model/origin.rs | 91 ++++++------------- rs/moq-net/src/route/mod.rs | 64 ++++++------- rs/moq-net/src/route/node.rs | 55 +++++------ rs/moq-net/src/route/rtt.rs | 19 ++++ rs/moq-net/src/route/tests/mod.rs | 11 +-- rs/moq-net/src/route/tests/routing.rs | 6 +- .../src/route/tests/{min_rtt.rs => rtt.rs} | 52 ++++------- rs/moq-relay/src/cluster.rs | 13 --- rs/moq-relay/src/settings.rs | 3 - rs/moq-tokio/tests/integration/broadcast.rs | 34 ++----- swift/Sources/Moq/Aliases.swift | 6 +- 29 files changed, 254 insertions(+), 437 deletions(-) rename rs/moq-net/src/route/tests/{min_rtt.rs => rtt.rs} (62%) diff --git a/dart/moq/lib/src/aliases.dart b/dart/moq/lib/src/aliases.dart index 6aea471ffab..c26cc8ee539 100644 --- a/dart/moq/lib/src/aliases.dart +++ b/dart/moq/lib/src/aliases.dart @@ -68,13 +68,9 @@ typedef AnnounceConfig = MoqAnnounceConfig; typedef Following = MoqFollowing; /// The read-only facts on a received route: a static cost, the stream's instance, and -/// its [MinRtt]. +/// its informational `rtt` to the origin node. typedef RouteInfo = MoqRouteInfo; -/// The minimum round trip to a route's origin node over moq-lite-07: `upstream` past the -/// session's peer, `link` to the peer, and their `total`. Informational; it never ranks a route. -typedef MinRtt = MoqMinRtt; - /// One continuous stream at a path: `Instance.parse` names one, and only a session mints one. /// /// Two are equal only when their text and minted flag both match, so pass an diff --git a/dart/moq_ffi/lib/src/moq.dart b/dart/moq_ffi/lib/src/moq.dart index 3e8fe8ae4e6..4a41463c6da 100644 --- a/dart/moq_ffi/lib/src/moq.dart +++ b/dart/moq_ffi/lib/src/moq.dart @@ -2962,77 +2962,6 @@ class FfiConverterMoqAnnounceConfig { } } -class MoqMinRtt { - final Duration? upstream; - final Duration? link; - final Duration? total; - MoqMinRtt({this.upstream, this.link, this.total}); -} - -class FfiConverterMoqMinRtt { - static MoqMinRtt lift(RustBuffer buf) { - return FfiConverterMoqMinRtt.read(buf.asUint8List()).value; - } - - static LiftRetVal read(Uint8List buf) { - int new_offset = buf.offsetInBytes; - final upstream_lifted = FfiConverterOptionalDuration.read( - Uint8List.view(buf.buffer, new_offset), - ); - final upstream = upstream_lifted.value; - new_offset += upstream_lifted.bytesRead; - final link_lifted = FfiConverterOptionalDuration.read( - Uint8List.view(buf.buffer, new_offset), - ); - final link = link_lifted.value; - new_offset += link_lifted.bytesRead; - final total_lifted = FfiConverterOptionalDuration.read( - Uint8List.view(buf.buffer, new_offset), - ); - final total = total_lifted.value; - new_offset += total_lifted.bytesRead; - return LiftRetVal( - MoqMinRtt(upstream: upstream, link: link, total: total), - new_offset - buf.offsetInBytes, - ); - } - - static RustBuffer lower(MoqMinRtt value) { - final total_length = - FfiConverterOptionalDuration.allocationSize(value.upstream) + - FfiConverterOptionalDuration.allocationSize(value.link) + - FfiConverterOptionalDuration.allocationSize(value.total) + - 0; - final buf = Uint8List(total_length); - write(value, buf); - return toRustBuffer(buf); - } - - static int write(MoqMinRtt value, Uint8List buf) { - int new_offset = buf.offsetInBytes; - new_offset += FfiConverterOptionalDuration.write( - value.upstream, - Uint8List.view(buf.buffer, new_offset), - ); - new_offset += FfiConverterOptionalDuration.write( - value.link, - Uint8List.view(buf.buffer, new_offset), - ); - new_offset += FfiConverterOptionalDuration.write( - value.total, - Uint8List.view(buf.buffer, new_offset), - ); - return new_offset - buf.offsetInBytes; - } - - static int allocationSize(MoqMinRtt value) { - return FfiConverterOptionalDuration.allocationSize(value.upstream) + - FfiConverterOptionalDuration.allocationSize(value.link) + - FfiConverterOptionalDuration.allocationSize(value.total) + - 0; - } -} - class MoqOriginConfig { final int? cacheCapacityBytes; MoqOriginConfig({this.cacheCapacityBytes = null}); @@ -3083,12 +3012,12 @@ class MoqRouteInfo { final int cost; final MoqInstance instance; final bool available; - final MoqMinRtt minRtt; + final Duration? rtt; MoqRouteInfo({ required this.cost, required this.instance, required this.available, - required this.minRtt, + this.rtt, }); } @@ -3114,17 +3043,17 @@ class FfiConverterMoqRouteInfo { ); final available = available_lifted.value; new_offset += available_lifted.bytesRead; - final minRtt_lifted = FfiConverterMoqMinRtt.read( + final rtt_lifted = FfiConverterOptionalDuration.read( Uint8List.view(buf.buffer, new_offset), ); - final minRtt = minRtt_lifted.value; - new_offset += minRtt_lifted.bytesRead; + final rtt = rtt_lifted.value; + new_offset += rtt_lifted.bytesRead; return LiftRetVal( MoqRouteInfo( cost: cost, instance: instance, available: available, - minRtt: minRtt, + rtt: rtt, ), new_offset - buf.offsetInBytes, ); @@ -3135,7 +3064,7 @@ class FfiConverterMoqRouteInfo { FfiConverterUInt64.allocationSize(value.cost) + FfiConverterMoqInstance.allocationSize(value.instance) + FfiConverterBool.allocationSize(value.available) + - FfiConverterMoqMinRtt.allocationSize(value.minRtt) + + FfiConverterOptionalDuration.allocationSize(value.rtt) + 0; final buf = Uint8List(total_length); write(value, buf); @@ -3156,8 +3085,8 @@ class FfiConverterMoqRouteInfo { value.available, Uint8List.view(buf.buffer, new_offset), ); - new_offset += FfiConverterMoqMinRtt.write( - value.minRtt, + new_offset += FfiConverterOptionalDuration.write( + value.rtt, Uint8List.view(buf.buffer, new_offset), ); return new_offset - buf.offsetInBytes; @@ -3167,7 +3096,7 @@ class FfiConverterMoqRouteInfo { return FfiConverterUInt64.allocationSize(value.cost) + FfiConverterMoqInstance.allocationSize(value.instance) + FfiConverterBool.allocationSize(value.available) + - FfiConverterMoqMinRtt.allocationSize(value.minRtt) + + FfiConverterOptionalDuration.allocationSize(value.rtt) + 0; } } diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index 68e6ffdfc57..54d7e5233e6 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -221,22 +221,14 @@ crosses. A live publisher seeds 0, and caching never changes it. ## Route latency -A moq-lite-07-wip route also carries its min RTT: each relay adds its link's +A moq-lite-07-wip route also carries its RTT: each relay adds its link's lowest smoothed RTT over the last 30 seconds to the sum it received, so an application reads the round trip to a publisher through any number of relays -from the route it was announced on. It is informational and never picks a -route; the link cost does that. A link's value changes only when its minimum -moves past the larger of 5 ms and a quarter, so jitter and short queues send -no route updates. - -Set `withhold_min_rtt` to advertise none, and every route crossing this relay -reports none from here on. The same setting is `--cluster-withhold-min-rtt` or -`MOQ_CLUSTER_WITHHOLD_MIN_RTT`. - -```toml -[cluster] -withhold_min_rtt = true -``` +from the route it was announced on (`RouteInfo::rtt`). The value is absent +while any link on the path has no RTT measurement. It is informational and +never picks a route; the link cost does that. A link's value changes only +when its minimum moves past the larger of 5 ms and a quarter, so jitter and +short queues send no route updates. ## LAN discovery diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index 54bf00129eb..2df66a6ea03 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -307,7 +307,6 @@ node = "https://us-west.example.com/" # This relay's own URL. token = "cluster.jwt" # JWT for dials without an inline ?jwt=. route_key = "route.key" # Route key; relays sharing the file fail over as one to scoped peers. max_hops = 32 # Longest accepted/forwarded hop path (1 to 32). -withhold_min_rtt = false # Advertise no min RTT to moq-lite-07 peers. [cluster.api] url = "https://api.example.com/peers" # Live JSON peer list; a local file also works. diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index e1ba3ca274d..f95ffed1d57 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -407,7 +407,7 @@ The subscriber creates the stream with an ANNOUNCE_REQUEST message. The publisher replies with a single ANNOUNCE_OK message followed by the matching routes and claims and any future changes: - ROUTE_START: the publisher reaches an origin node. -- ROUTE_UPDATE: a route's seqno, metric, down-only bit, or Min RTT changed. +- ROUTE_UPDATE: a route's seqno, metric, down-only bit, or RTT changed. - ROUTE_HOLD: the publisher has no usable route but holds it while it recovers. - ROUTE_END: the publisher let go of a route and every claim under it. - ANNOUNCE_START: a matching claim of a route's origin node. @@ -492,17 +492,17 @@ A receiver adds the session's link cost (see [Cost Parameter](#cost-parameter)) Each link adds at least 1, so a route's metric strictly increases away from its origin. A metric that overflows 64 bits makes the route unusable. -A route may also carry a Min RTT: the round trip to its origin node in milliseconds, summed link by link. -It is informational: nothing ranks or selects a route, or judges it feasible, by its Min RTT. -An origin advertises its own route with a Min RTT of 0. -A receiver adds its link min RTT for the session to the Min RTT of every route the peer sends; the sum is its Min RTT through that peer, which it advertises on the route it selects. +A route may also carry an RTT: the round trip to its origin node in milliseconds, summed link by link. +It is informational: nothing ranks or selects a route, or judges it feasible, by its RTT. +An origin advertises its own route with an RTT of 0. +A receiver adds its link min RTT for the session to the RTT of every route the peer sends; the sum is its RTT through that peer, which it advertises on the route it selects. A link min RTT is the lowest smoothed RTT [RFC9002] the connection measured over the last 30 seconds, read once a second, in milliseconds rounded up. A lifetime minimum never rises after a path change, so it is not a link min RTT. A node SHOULD keep the link min RTT it adds until the lowest reading differs from it by more than the larger of 5 ms and a quarter of it, so jitter and short queues send no ROUTE_UPDATE and a lasting change sends one per route. -A node MAY withhold Min RTT, and a node whose connection reports no RTT has no link min RTT; either sends its routes without one, and a route without one stays without one past every later node. +A node whose connection reports no RTT has no link min RTT, so it sends the routes it learned there without one, and a route without one stays without one past every later node. A sum too large to encode is sent without one too. -An endpoint learns its round trip to an origin node by adding its own link min RTT to the Min RTT of the route it selected; without its own, the Min RTT alone covers every hop past its first. -Like the Metric, a Min RTT is unverified: a peer can misstate only what crosses it. +An endpoint learns its round trip to an origin node by adding its own link min RTT to the RTT of the route it selected; without its own, the RTT alone covers every hop past its first. +Like the Metric, an RTT is unverified: a peer can misstate only what crosses it. A node keeps a feasibility distance per Node ID: the highest Seqno of any route it advertised, and the lowest metric it advertised at that Seqno, as in [RFC8966] Section 3.5.1. A route is feasible when no feasibility distance is kept for its Node ID yet, when its Seqno is greater than the distance's, or when its Seqno is equal and its advertised Metric is lower. @@ -1155,7 +1155,7 @@ ROUTE_START Message { Seqno (i), Metric (i), Flags (8), - [Min RTT (i),] + [RTT (i),] } ~~~ @@ -1174,12 +1174,12 @@ The origin's sequence number of the route, which only the origin advances. The publisher's metric to the origin node, 0 when it is the origin. **Flags**: -A bit field: 0x1 when the route is down-only (see [Routing](#routing)), and 0x2 when a Min RTT follows. +A bit field: 0x1 when the route is down-only (see [Routing](#routing)), and 0x2 when an RTT follows. Any other bit set is a PROTOCOL_VIOLATION. -**Min RTT**: -The publisher's Min RTT to the origin node in milliseconds (see [Routing](#routing)), 0 when it is the origin. -Present only when Flags has 0x2 set, and absent when the publisher withholds it or has none. +**RTT**: +The publisher's RTT to the origin node in milliseconds (see [Routing](#routing)), 0 when it is the origin. +Present only when Flags has 0x2 set, and absent when the publisher has none. ## ROUTE_UPDATE {#route-update} @@ -1193,7 +1193,7 @@ ROUTE_UPDATE Message { Seqno (i), Metric (i), Flags (8), - [Min RTT (i),] + [RTT (i),] } ~~~ @@ -1203,14 +1203,14 @@ Set to 0x7 to indicate a ROUTE_UPDATE message. **Route ID**: The ordinal implicitly assigned by a prior ROUTE_START on this stream. -**Seqno**, **Metric**, **Flags**, and **Min RTT**: +**Seqno**, **Metric**, **Flags**, and **RTT**: As defined for [ROUTE_START](#route-start), replacing the route's. A ROUTE_UPDATE for a held route makes it usable again. ## ROUTE_HOLD {#route-hold} A publisher sends a ROUTE_HOLD message when it has no usable route to the origin node but still holds the route and its claims while it recovers (see [Routing](#routing)), referencing its Route ID. -The receiver keeps the route's last Seqno, Metric, Flags, and Min RTT, and does not select it until a ROUTE_UPDATE. +The receiver keeps the route's last Seqno, Metric, Flags, and RTT, and does not select it until a ROUTE_UPDATE. ~~~ ROUTE_HOLD Message { @@ -2248,7 +2248,7 @@ The `Message Length` describes the payload size on the wire. - A receiver mints an Instance for each announcement a session of an earlier version delivers, and a `Minted` field beside every Instance (ANNOUNCE_START, ANNOUNCE_RESTART, TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS) says whether a receiver minted it, so no Instance text is reserved for minting. - Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live claim's suffix. - Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list and ANNOUNCE_OK's Hop ID, and replaced the SETUP Hop parameter (0x5) with a [Node ID parameter](#node-id-parameter), so an endpoint that reaches itself detects it at setup. -- ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional Min RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route. +- ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route. - Added ROUTE_HOLD (0xB), which holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back. - Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible. - Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows. diff --git a/go/wrapper/types.go b/go/wrapper/types.go index 0e45840f11c..ca456c12c20 100644 --- a/go/wrapper/types.go +++ b/go/wrapper/types.go @@ -29,10 +29,8 @@ type ( ProtocolKind = ffi.MoqProtocolKind // OriginConfig configures a new origin, such as its maximum cache size in bytes. OriginConfig = ffi.MoqOriginConfig - // RouteInfo is the read-only facts on a received route: the Instance of the stream it serves (routes with one instance are one continuous stream), its static production and link cost (lower wins), and its MinRtt. + // RouteInfo is the read-only facts on a received route: the Instance of the stream it serves (routes with one instance are one continuous stream) and its static production and link cost (lower wins), and its informational Rtt to the origin node. RouteInfo = ffi.MoqRouteInfo - // MinRtt is the minimum round trip to a route's origin node over moq-lite-07: Upstream past the session's peer, Link to the peer, and their Total. Informational; it never ranks a route. - MinRtt = ffi.MoqMinRtt // Instance names one continuous stream at a path. Compare two with Eq: a session mints one // for a route whose wire carries none, and it differs from a named instance with the same Text. Instance = ffi.MoqInstance @@ -223,8 +221,6 @@ const ( ProtocolKindVersion = ffi.MoqProtocolKindVersion // ProtocolKindDeliveryTimeout identifies delivery timeout. ProtocolKindDeliveryTimeout = ffi.MoqProtocolKindDeliveryTimeout - // ProtocolKindSessionClosed identifies session closed. - ProtocolKindSessionClosed = ffi.MoqProtocolKindSessionClosed // ProtocolKindGoingAway identifies going away. ProtocolKindGoingAway = ffi.MoqProtocolKindGoingAway // ProtocolKindTooFarBehind identifies too far behind. diff --git a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt index 8e6f158d577..46481f133bd 100644 --- a/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt +++ b/kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt @@ -189,16 +189,11 @@ typealias RouteOptions = uniffi.moq.MoqRouteOptions /** * The read-only facts on a received route: the [Instance] of the stream it serves, relay hop ids - * (oldest first), and static production and link cost (lower wins). + * (oldest first), static production and link cost (lower wins), and its informational `rtt` to the + * origin node. */ typealias RouteInfo = uniffi.moq.MoqRouteInfo -/** - * The minimum round trip to a route's origin node over moq-lite-07: `upstream` past the session's - * peer, `link` to the peer, and their `total`. Informational; it never ranks a route. - */ -typealias MinRtt = uniffi.moq.MoqMinRtt - /** Tunes how a track subscription is delivered: priority, delay, ranges, and group ordering. */ typealias Subscription = uniffi.moq.MoqSubscription diff --git a/py/moq-rs/moq/__init__.py b/py/moq-rs/moq/__init__.py index fce747ad4f0..997c926ca91 100644 --- a/py/moq-rs/moq/__init__.py +++ b/py/moq-rs/moq/__init__.py @@ -77,7 +77,6 @@ Frame, Grant, Instance, - MinRtt, Order, Position, ProtocolError, @@ -153,7 +152,6 @@ "OriginConsumer", "OriginProducer", "Order", - "MinRtt", "Position", "Range", "RateControl", diff --git a/py/moq-rs/moq/types.py b/py/moq-rs/moq/types.py index 20add55f6d3..2628f449bc9 100644 --- a/py/moq-rs/moq/types.py +++ b/py/moq-rs/moq/types.py @@ -32,9 +32,6 @@ from moq_ffi import ( MoqInstance as Instance, ) -from moq_ffi import ( - MoqMinRtt as MinRtt, -) from moq_ffi import ( MoqOrder as Order, ) @@ -109,7 +106,6 @@ "FetchGroupOptions", "Grant", "Instance", - "MinRtt", "Order", "Position", "ProtocolError", diff --git a/quest/m2/README.md b/quest/m2/README.md index 197245d6434..f2342d65f2a 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -199,7 +199,7 @@ quests that require them. - [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - the wire names every abandoned group and the drop signal is public, so readers count losses instead of stalling - [Stream credit by priority](/quest/m1/stream-credit/README.md) - every stream opens through one priority entry point, and a higher-priority group takes over or resets a lower one when credit runs out (held in m1 by an open claim) - [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - warm routes and the eligibility rule hold in route-layer sweeps, and the warm message's wire placement is chosen before the lite-07 cut -- [Route latency in js/net](/quest/m2/route-latency.md) - js/net reads and writes the min RTT a lite-07 ROUTE carries, once its route layer lands +- [Route latency in js/net](/quest/m2/route-latency.md) - js/net reads and writes the RTT a lite-07 ROUTE carries, once its route layer lands - [Finalize moq-lite-07](/quest/m2/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it - [Unbounded groups](/quest/m2/unbounded-groups/README.md) - groups and frames lose their size caps; a group keeps a 32 MB tail, and readers behind it replay through paced flights or refresh - [Small API fixes before the cut](/quest/m2/api-shape/README.md) - one path and one spelling per item, names by role, and no dead or wrong-role public surface across net, transport, media, native, apps, JS, and bindings diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md index f168493ccff..ae8167f4cc3 100644 --- a/quest/m2/lite07-finalize.md +++ b/quest/m2/lite07-finalize.md @@ -127,7 +127,7 @@ whose lite-07 encoding [lite-07 by start](/quest/m2/time-addressing/lite07.md) c - [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - may add SUBSCRIBE_DROP for a stopped group; leaves this list if it takes the no-wire path - [Delete the old start bookkeeping](/quest/m2/subscribe-ranges/start-bookkeeping.md) - may change START per update on lite-07 - [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - may add the warm-route message to lite-07; leaves this list if the proof places it in a SETUP extension -- [Route latency in js/net](/quest/m2/route-latency.md) - js/net reads and writes ROUTE's Min RTT +- [Route latency in js/net](/quest/m2/route-latency.md) - js/net reads and writes ROUTE's RTT ## Related diff --git a/quest/m2/route-latency.md b/quest/m2/route-latency.md index 96681660f0b..925090486a6 100644 --- a/quest/m2/route-latency.md +++ b/quest/m2/route-latency.md @@ -2,18 +2,17 @@ ## Goal -`js/net` reads a lite-07 route's Min RTT as Rust does: it decodes and encodes -ROUTE_START's and ROUTE_UPDATE's Flags and optional Min RTT, a browser -publishes its own route at 0, and its announced routes carry the sum the -relay advertised plus its own first hop, marked as excluding the first hop -when the browser's transport reports no RTT. The sum never ranks or selects -a route. +`js/net` reads a lite-07 route's RTT as Rust does: it decodes and encodes +ROUTE_START's and ROUTE_UPDATE's Flags and optional RTT, a browser publishes +its own route at 0, and its announced routes carry the sum the relay +advertised plus its own first hop, unknown when the browser's transport +reports no RTT. The sum never ranks or selects a route. ## Plan Decided 2026-11-10 (maintainer, on [moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458)): ✅ lite-07's -ROUTE carries the latency sum, as min RTT. Each node adds its link's windowed +ROUTE carries the latency sum, as RTT. Each node adds its link's windowed minimum, the lowest of the last 30 once-a-second reads of the smoothed RTT, re-advertised only past max(5 ms, 25%), never QUIC's lifetime min_rtt, which never rises after a path change. The endpoint adds its own first hop from its @@ -21,20 +20,19 @@ connection. Rejected: no wire change, leaving applications a track of their own. [moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458) landed the -draft, the Rust codec and route layer, `origin::RouteInfo::min_rtt`, the -bindings' `MinRtt`, and the relay's `--cluster-withhold-min-rtt`. Its PR body +draft, the Rust codec and route layer, `origin::RouteInfo::rtt`, and the +bindings' `RouteInfo.rtt`, with no option to withhold it. Its PR body keeps the sizing simulation: one real shift sends one ROUTE_UPDATE per origin on each stream downstream of the link, and jitter sends none. What remains is the JS side, which needs the JS route layer first: -- Decode Flags (0x1 down-only, 0x2 Min RTT present, any other bit a - PROTOCOL_VIOLATION) and the Min RTT that follows, in milliseconds, and - encode a browser's own route with a Min RTT of 0. -- Add `MinRtt` (`upstream`, `link`, `total()`) to `js/net`'s `RouteInfo`, - named as Rust's. `link` follows the same windowed minimum over - WebTransport `getStats()`'s smoothed RTT, and is `undefined` where it - reports none. +- Decode Flags (0x1 down-only, 0x2 RTT present, any other bit a + PROTOCOL_VIOLATION) and the RTT that follows, in milliseconds, and encode + a browser's own route with an RTT of 0. +- Add `rtt` to `js/net`'s `RouteInfo`, named as Rust's: the advertised sum + plus the same windowed minimum over WebTransport `getStats()`'s smoothed + RTT, and `undefined` while either is unknown. - Pin the codec to Rust's with the shared corpus, and run `just test interop --all`. diff --git a/rs/moq-ffi/src/origin.rs b/rs/moq-ffi/src/origin.rs index 4f234c1a326..d0b8d4db89e 100644 --- a/rs/moq-ffi/src/origin.rs +++ b/rs/moq-ffi/src/origin.rs @@ -94,33 +94,9 @@ pub struct MoqRouteInfo { pub instance: Arc, /// Whether a usable path reaches the route's origin; requests may fail while it is not. pub available: bool, - /// The minimum round trip to the route's origin node, informational; never ranks a route. - pub min_rtt: MoqMinRtt, -} - -/// The minimum round trip to a route's origin node over moq-lite-07, in two parts. -/// -/// Each node on the path adds its link's lowest smoothed RTT over the last 30 seconds. -/// The sum is unknown past a node that withholds it or cannot measure its link, and on -/// any other wire. -#[derive(Clone, Copy, Debug, PartialEq, Eq, uniffi::Record)] -pub struct MoqMinRtt { - /// Past the session's peer: the sum the peer advertised, absent when unknown. - pub upstream: Option, - /// The session's own link to the peer, absent when its transport reports no RTT. - pub link: Option, - /// The whole path, present when both parts are. - pub total: Option, -} - -impl From for MoqMinRtt { - fn from(min_rtt: moq_net::origin::MinRtt) -> Self { - Self { - upstream: min_rtt.upstream, - link: min_rtt.link, - total: min_rtt.total(), - } - } + /// The round trip to the route's origin node over moq-lite-07, absent while any link + /// has no measurement. Informational; it never ranks a route. + pub rtt: Option, } impl From for MoqRouteInfo { @@ -129,7 +105,7 @@ impl From for MoqRouteInfo { cost: route.cost().value(), instance: MoqInstance::wrap(route.instance().clone()), available: route.available(), - min_rtt: route.min_rtt().into(), + rtt: route.rtt(), } } } diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index 07338f37bc0..bf2055d1c4d 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -677,7 +677,7 @@ impl RouteRelay { seqno: 0, metric: 0, down_only: false, - min_rtt: None, + rtt: None, }; let receive = (0..streams) .map(|_| { diff --git a/rs/moq-net/src/lite/announce.rs b/rs/moq-net/src/lite/announce.rs index 5d18bd225ad..45779ca9611 100644 --- a/rs/moq-net/src/lite/announce.rs +++ b/rs/moq-net/src/lite/announce.rs @@ -116,10 +116,10 @@ pub enum AnnounceBroadcast { /// ROUTE_START and ROUTE_UPDATE flag: the route is down-only. const DOWN_ONLY: u8 = 0x1; -/// ROUTE_START and ROUTE_UPDATE flag: a Min RTT follows. -const MIN_RTT: u8 = 0x2; +/// ROUTE_START and ROUTE_UPDATE flag: an RTT follows. +const RTT: u8 = 0x2; -/// Encode a seqno, metric, flags, and min RTT in milliseconds. No origin advertises the +/// Encode a seqno, metric, flags, and RTT in milliseconds. No origin advertises the /// last seqno, which would leave no room for the successor a ROUTE_REQUEST names. fn encode_advert(w: &mut Encoder<'_>, advert: &Advert) -> Result<(), EncodeError> { if advert.seqno == route::MAX { @@ -128,19 +128,17 @@ fn encode_advert(w: &mut Encoder<'_>, advert: &Advert) -> Result<(), EncodeError w.varint(advert.seqno)?; w.varint(advert.metric)?; // A sum past the field's range, which only a peer's absurd value reaches, is unknown. - let min_rtt = advert - .min_rtt - .and_then(|min_rtt| u64::try_from(min_rtt.as_millis()).ok()); + let rtt = advert.rtt.and_then(|rtt| u64::try_from(rtt.as_millis()).ok()); let mut flags = 0; if advert.down_only { flags |= DOWN_ONLY; } - if min_rtt.is_some() { - flags |= MIN_RTT; + if rtt.is_some() { + flags |= RTT; } w.u8(flags); - if let Some(min_rtt) = min_rtt { - w.varint(min_rtt)?; + if let Some(rtt) = rtt { + w.varint(rtt)?; } Ok(()) } @@ -152,10 +150,10 @@ fn decode_advert(r: &mut Decoder<'_>) -> Result { } let metric = r.varint()?; let flags = r.u8()?; - if flags & !(DOWN_ONLY | MIN_RTT) != 0 { + if flags & !(DOWN_ONLY | RTT) != 0 { return Err(DecodeError::InvalidValue); } - let min_rtt = match flags & MIN_RTT { + let rtt = match flags & RTT { 0 => None, _ => Some(std::time::Duration::from_millis(r.varint()?)), }; @@ -163,7 +161,7 @@ fn decode_advert(r: &mut Decoder<'_>) -> Result { seqno, metric, down_only: flags & DOWN_ONLY != 0, - min_rtt, + rtt, }) } @@ -1570,7 +1568,7 @@ mod tests { seqno, metric, down_only, - min_rtt: None, + rtt: None, } } @@ -1615,14 +1613,14 @@ mod tests { AnnounceBroadcast::RouteStart { node: 0x7a40, advert: Advert { - min_rtt: Some(std::time::Duration::ZERO), + rtt: Some(std::time::Duration::ZERO), ..advert(1, 0, false) }, }, AnnounceBroadcast::RouteUpdate { id: 1, advert: Advert { - min_rtt: Some(std::time::Duration::from_millis(u64::MAX)), + rtt: Some(std::time::Duration::from_millis(u64::MAX)), ..advert(2, 3, true) }, }, @@ -1642,14 +1640,14 @@ mod tests { } } - /// A peer's largest Min RTT plus a link goes past the field: it is forwarded as + /// A peer's largest RTT plus a link goes past the field: it is forwarded as /// unknown, never an error that would end the stream. #[test] - fn a_min_rtt_past_the_field_is_unknown() { - let msg = |min_rtt| AnnounceBroadcast::RouteUpdate { + fn an_rtt_past_the_field_is_unknown() { + let msg = |rtt| AnnounceBroadcast::RouteUpdate { id: 1, advert: Advert { - min_rtt, + rtt, ..advert(2, 3, false) }, }; @@ -1756,7 +1754,7 @@ mod tests { let malformed = [ forge(ROUTE_START, advert(0, 0, 1, 0)), forge(ROUTE_START, advert(1, route::MAX, 1, 0)), - // A flag no version defines, and a Min RTT flag without its field. + // A flag no version defines, and an RTT flag without its field. forge(ROUTE_START, advert(1, 0, 1, 4)), forge(ROUTE_START, advert(1, 0, 1, 2)), // ROUTE_UPDATE's leading field is the Route ID, so 0 is a valid id here. diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 67ac68a3206..085c27fd18f 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -9024,7 +9024,7 @@ mod tests { seqno: 0, metric: 0, down_only: false, - min_rtt: None, + rtt: None, }; lite::AnnounceBroadcast::RouteStart { node, advert } .encode(w, Version::Lite07) diff --git a/rs/moq-net/src/lite/subscriber/routed.rs b/rs/moq-net/src/lite/subscriber/routed.rs index 24009cb8056..bd31a5a5768 100644 --- a/rs/moq-net/src/lite/subscriber/routed.rs +++ b/rs/moq-net/src/lite/subscriber/routed.rs @@ -99,7 +99,7 @@ pub(super) struct Routed { drained: bool, /// The link's min RTT, which the peer's routes add. rtt: route::LinkRtt, - /// When to read the link's RTT next, or `None` once the transport reported none. + /// When to read the link's RTT next, or `None` once it stopped reading. read_rtt: Option, } @@ -134,10 +134,12 @@ impl Routed { .receive(scoped(&link, ®ions, key), prefix.clone()) .map_err(refused)?; let mut rtt = route::LinkRtt::default(); - let first = rtt.read(subscriber.session.metrics().rtt()); - stream.min_rtt(first); - // A transport that reports no RTT never starts to, so it costs no timer. - let read_rtt = first.map(|_| crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL)); + stream.rtt(rtt.read(subscriber.session.metrics().rtt())); + // A transport can report its first RTT later, such as qmux after its first ping. + let read_rtt = Some(crate::time::Deadline::after( + &subscriber.runtime, + route::LinkRtt::INTERVAL, + )); Ok(Self { node, stream, @@ -352,7 +354,7 @@ impl Routed { .node .receive(scoped(&self.link, &self.regions, key), self.prefix.clone()) .map_err(refused)?; - stream.min_rtt(self.rtt.reported()); + stream.rtt(self.rtt.reported()); // What the old stream served stays in the origin until the new one serves it // again, or lets go after. let previous = std::mem::take(&mut self.served); @@ -472,7 +474,7 @@ impl Routed { .with_node(Some(id)) .with_draining(self.drained) .with_available(served.available) - .with_min_rtt(served.min_rtt); + .with_rtt(served.rtt); if let Some(entry) = self.previous.remove(&key) { self.served.insert(key.clone(), entry); } @@ -518,9 +520,10 @@ impl Routed { let last = self.rtt.reported(); let now = self.rtt.read(subscriber.session.metrics().rtt()); if now != last { - self.stream.min_rtt(now); + self.stream.rtt(now); } - self.read_rtt = now.map(|_| crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL)); + self.read_rtt = (self.rtt.reading()) + .then(|| crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL)); } while let Poll::Ready(output) = self.stream.poll_output(waiter) { self.output(subscriber, output); @@ -616,7 +619,7 @@ mod tests { seqno: 0, metric: 0, down_only: false, - min_rtt: None, + rtt: None, }; fn start(path: &'static str) -> Msg { @@ -968,6 +971,33 @@ mod tests { ); } + /// A transport that reports its first RTT only after the stream opened, such as + /// qmux before its first ping returns, still gets its link measured. + #[kio::test] + async fn a_late_rtt_is_still_read() { + let session = SinkSession::new(Default::default()); + let (subscriber, _consumer) = super::super::tests::restart_subscriber(session.clone()); + let prefix = Path::empty(); + let mut routed = super::Routed::new(&subscriber, &prefix, 1, &Default::default()).unwrap(); + let mut decoder = AnnounceDecoder::default(); + feed(&mut routed, &mut decoder, &[]).unwrap(); + let mut writer = crate::coding::Writer::new( + crate::lite::test_transport::SinkSend::new(Default::default()), + crate::lite::Version::Lite07, + ); + let waiter = kio::Waiter::noop(); + routed.poll(&subscriber, &mut writer, &waiter).unwrap(); + kio::clock::sim::advance(super::route::LinkRtt::INTERVAL).await; + routed.poll(&subscriber, &mut writer, &waiter).unwrap(); + assert_eq!(routed.rtt.reported(), None); + + let rtt = std::time::Duration::from_millis(20); + session.set_stats(crate::lite::test_transport::SinkStats::default().with_rtt(rtt)); + kio::clock::sim::advance(super::route::LinkRtt::INTERVAL).await; + routed.poll(&subscriber, &mut writer, &waiter).unwrap(); + assert_eq!(routed.rtt.reported(), Some(rtt)); + } + /// What the stream keeps between floors is the net change to its committed claims, /// however many messages the peer sends without one. #[kio::test] diff --git a/rs/moq-net/src/model/mod.rs b/rs/moq-net/src/model/mod.rs index 180d971cd32..76a6bffd228 100644 --- a/rs/moq-net/src/model/mod.rs +++ b/rs/moq-net/src/model/mod.rs @@ -64,8 +64,8 @@ pub mod origin { #[cfg(feature = "async")] pub(crate) use super::origin_impl::NodeFilter; pub use super::origin_impl::{ - Claim, Config, Consumer, Cost, Driver, Link, MinRtt, Preference, Producer, Request, Requesting, Route, - RouteInfo, RouteOptions, Source, + Claim, Config, Consumer, Cost, Driver, Link, Preference, Producer, Request, Requesting, Route, RouteInfo, + RouteOptions, Source, }; pub use crate::route::RouteKey; diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 0286c245c8a..6625772a4b2 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -108,10 +108,6 @@ pub struct Config { /// proves nothing, so such a peer files them under a key of its session alone. pub route_key: Option, - /// Advertise no min RTT to lite-07 peers, so no route crossing this origin carries - /// one past it (default false). Its own readers still see the [`MinRtt`] it measures. - pub withhold_min_rtt: bool, - /// Longest hop path a session using this origin accepts or forwards (default 32). /// /// Applies to moq-lite-03+ hop paths and the negotiated MoQ Cluster extension on @@ -170,7 +166,6 @@ impl Default for Config { Self { hop: random_id(), route_key: None, - withhold_min_rtt: false, max_hops: MAX_HOP_LIMIT, pool, cache_duration: None, @@ -381,28 +376,6 @@ impl<'a, V: Copy> Decode<'a, V> for Hops { } } -/// The minimum round trip to a route's origin node over moq-lite-07, in two parts. -/// Informational: it never ranks or selects a route. -/// -/// Each node on the path adds its link's min RTT, the lowest smoothed RTT the link -/// measured over the last 30 seconds, to the sum it received. The sum is unknown past -/// a node that withholds it or cannot measure its link, and on any other wire. -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] -#[non_exhaustive] -pub struct MinRtt { - /// Past the session's peer: the sum the peer advertised, `None` when unknown. - pub upstream: Option, - /// The session's own link to the peer, `None` when its transport reports no RTT. - pub link: Option, -} - -impl MinRtt { - /// The whole path to the origin node, when both parts are known. - pub fn total(&self) -> Option { - self.upstream?.checked_add(self.link?) - } -} - /// The highest route cost, shared by every wire version. const MAX_COST: u64 = (1 << 62) - 1; @@ -540,9 +513,9 @@ pub struct RouteInfo { /// Whether a usable path reaches the route's origin; see [`Self::available`]. pub(crate) available: bool, - /// The round trip to the route's origin node; see [`Self::min_rtt`]. Never forwarded: + /// The round trip to the route's origin node; see [`Self::rtt`]. Never forwarded: /// each node measures its own link. - pub(crate) min_rtt: MinRtt, + pub(crate) rtt: Option, } impl RouteInfo { @@ -560,7 +533,7 @@ impl RouteInfo { node: None, draining: false, available: true, - min_rtt: MinRtt::default(), + rtt: None, } } } @@ -598,10 +571,15 @@ impl RouteInfo { self.cost } - /// The minimum round trip to the route's origin node, as far as it is known; see - /// [`MinRtt`]. Unknown on a route this origin serves itself. - pub fn min_rtt(&self) -> MinRtt { - self.min_rtt + /// The round trip to the route's origin node over moq-lite-07, informational: it + /// never ranks or selects a route. + /// + /// Each link on the path adds its min RTT, the lowest smoothed RTT it measured over + /// the last 30 seconds, this session's own link included. `None` on any other wire, + /// on a route this origin serves itself, and while any link on the path has no + /// measurement. + pub fn rtt(&self) -> Option { + self.rtt } /// Whether a usable path reaches the route's origin. An unavailable route is held: @@ -673,10 +651,10 @@ impl RouteInfo { self } - /// Set the round trip to the route's origin node; see [`Self::min_rtt`]. + /// Set the round trip to the route's origin node; see [`Self::rtt`]. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(crate) fn with_min_rtt(mut self, min_rtt: MinRtt) -> Self { - self.min_rtt = min_rtt; + pub(crate) fn with_rtt(mut self, rtt: Option) -> Self { + self.rtt = rtt; self } @@ -892,7 +870,7 @@ type RouteOrder = ( Reverse, ); -/// The `(hops, cost, source, instance, epoch, auth, node, available, min_rtt)` metadata +/// The `(hops, cost, source, instance, epoch, auth, node, available, rtt)` metadata /// an announce cursor delivers alongside a prefix. type RouteMeta = ( Hops, @@ -903,7 +881,7 @@ type RouteMeta = ( Option, Option, bool, - MinRtt, + Option, ); /// The earlier of two optional deadlines. @@ -1302,7 +1280,7 @@ impl OriginConsumerState { node: meta.6, draining: false, available: meta.7, - min_rtt: meta.8, + rtt: meta.8, }, }; (RouteEvent { kind, announce }, owed) @@ -1345,8 +1323,8 @@ struct RouteEntry { draining: bool, /// Whether a usable path reaches the route's origin; see [`RouteInfo::available`]. available: bool, - /// The round trip to the route's origin node; see [`RouteInfo::min_rtt`]. - min_rtt: MinRtt, + /// The round trip to the route's origin node; see [`RouteInfo::rtt`]. + rtt: Option, /// The queue requests under this route are served from, when the announcer /// serves content on demand (a [`Claim`]). `None` for an advertise-only /// announcement ([`Producer::announce`]) and for a local broadcast. @@ -1415,13 +1393,13 @@ impl RouteEntry { self.auth.clone(), self.node, self.available, - self.min_rtt, + self.rtt, ) } /// Whether [`Self::meta`] would equal `meta`, without building it. fn delivers(&self, meta: &RouteMeta) -> bool { - let (hops, cost, source, instance, epoch, auth, node, available, min_rtt) = meta; + let (hops, cost, source, instance, epoch, auth, node, available, rtt) = meta; *hops == self.hops && *cost == self.cost && *source == self.entered() @@ -1430,7 +1408,7 @@ impl RouteEntry { && *auth == self.auth && *node == self.node && *available == self.available - && *min_rtt == self.min_rtt + && *rtt == self.rtt } /// Whether a request for `path` can be served through this entry. A served @@ -2259,11 +2237,7 @@ impl Producer { let (tasks, set) = TaskSet::new(); let scope = OriginScope::default(); let timers = Deadlines::new(Instant::ZERO); - let shared = SendCell::new(kio::Shared::new(OriginState::new( - config.update_hold, - config.route_key, - config.withhold_min_rtt, - ))); + let shared = SendCell::new(kio::Shared::new(OriginState::new(config.update_hold, config.route_key))); let pool = config.pool.clone(); let producer = Self { hop: config.hop, @@ -2395,7 +2369,6 @@ impl Producer { cache_duration: self.cache_duration, update_hold: shared.update_hold, route_key: shared.route_key.clone(), - withhold_min_rtt: shared.withhold_min_rtt, } } @@ -2436,7 +2409,7 @@ impl Producer { scope: OriginScope::empty(), max_hops: MAX_HOP_LIMIT, root: Path::default(), - shared: SendCell::new(kio::Shared::new(OriginState::new(DEFAULT_UPDATE_HOLD, None, false))), + shared: SendCell::new(kio::Shared::new(OriginState::new(DEFAULT_UPDATE_HOLD, None))), pool: cache::Pool::default(), cache_duration: None, stats: stats::Session::default(), @@ -3029,7 +3002,7 @@ impl Announcing { node: route.node, draining: route.draining, available: route.available, - min_rtt: route.min_rtt, + rtt: route.rtt, server: serving.server.clone(), source: serving.source.clone(), advertised: serving.advertised, @@ -3646,7 +3619,7 @@ impl AnnounceProducer { entry.node = route.node; entry.draining = route.draining; entry.available = route.available; - entry.min_rtt = route.min_rtt; + entry.rtt = route.rtt; entry.advertised = true; let claim = entry.claim.clone(); shared.sync_route(prefix, &claim, Gone::End); @@ -3708,7 +3681,7 @@ impl AnnounceProducer { node: entry.node, draining: entry.draining, available: entry.available, - min_rtt: entry.min_rtt, + rtt: entry.rtt, }) } @@ -6150,8 +6123,6 @@ struct OriginState { update_hold: Duration, // See [`Config::route_key`]. route_key: Option, - // See [`Config::withhold_min_rtt`]. - withhold_min_rtt: bool, // The announced routes, keyed by prefix. The table holds one entry per live // advertisement, not one per broadcast consumer. @@ -6195,11 +6166,10 @@ struct OriginState { } impl OriginState { - fn new(update_hold: Duration, route_key: Option, withhold_min_rtt: bool) -> Self { + fn new(update_hold: Duration, route_key: Option) -> Self { Self { update_hold, route_key, - withhold_min_rtt, routes: RouteTable::default(), next_route: 0, next_watch: 0, @@ -7408,8 +7378,7 @@ impl Consumer { .serving() }; // Built before taking the lock, since opening a cursor takes it. - let withhold = self.shared.lock().withhold_min_rtt; - let node = crate::route::Node::origin(self.hop(), withhold); + let node = crate::route::Node::origin(self.hop()); let claims = OwnClaims { routes: own(false).routes(), downstream: own(true).routes(), diff --git a/rs/moq-net/src/route/mod.rs b/rs/moq-net/src/route/mod.rs index 727a42c398f..38235c843ba 100644 --- a/rs/moq-net/src/route/mod.rs +++ b/rs/moq-net/src/route/mod.rs @@ -33,7 +33,7 @@ use std::{ time::Duration, }; -use crate::{Path, origin::MinRtt}; +use crate::Path; pub(crate) use node::Node; #[cfg(feature = "async")] @@ -160,10 +160,15 @@ pub(crate) struct Advert { pub seqno: Seqno, pub metric: Metric, pub down_only: bool, - /// The sum of each link's min RTT to the identity: informational, never read to - /// rank or select a route. `None` once a node on the path withheld it or could not + /// The round trip to the identity, the sum of each link's min RTT: informational, + /// never read to rank or select a route. `None` once a node on the path could not /// measure its link. - pub min_rtt: Option, + pub rtt: Option, +} + +/// The RTT through `peer` along a route it advertised: unknown when either part is. +fn through_rtt(advert: Advert, peer: &Peer) -> Option { + advert.rtt?.checked_add(peer.rtt?) } /// A ROUTE's state on a stream. @@ -332,7 +337,7 @@ impl Heard { struct Peer { link: Link, /// The link's min RTT, which routes learned here add; `None` while unmeasured. - min_rtt: Option, + rtt: Option, /// The neighbor sent GOAWAY: rank it below every other eligible route. draining: bool, /// The prefix the neighbor asked this node for, once it opened a stream. @@ -434,8 +439,6 @@ pub(crate) struct Router { changed: BTreeMap>, /// Mints a fresh identity on rotation. rng: u64, - /// Advertise every route without a min RTT. - withhold: bool, /// Per claim this node has shown, how many times it forgot that claim. #[cfg(test)] forgot: BTreeMap<(Id, Path), u32>, @@ -480,7 +483,6 @@ impl Router { ctxs: BTreeMap::new(), changed: BTreeMap::new(), rng: seed | 1, - withhold: false, #[cfg(test)] forgot: BTreeMap::new(), #[cfg(test)] @@ -490,12 +492,6 @@ impl Router { } } - /// Advertise every route without a min RTT, so none crossing this node carries one. - pub fn withhold(&mut self) { - self.withhold = true; - self.update_all(); - } - /// Take the identities recomputed since the last call, whose selection may have /// changed, each with the claims whose shown state did. pub fn changed(&mut self) -> BTreeMap> { @@ -523,7 +519,7 @@ impl Router { neighbor, Peer { link, - min_rtt: None, + rtt: None, draining: false, watch: None, asked: None, @@ -593,16 +589,16 @@ impl Router { self.update(self.through(neighbor)); } - /// The link to `neighbor` has `min_rtt`, or `None` unmeasured: every route learned + /// The link to `neighbor` has `rtt`, or `None` unmeasured: every route learned /// there adds it. - pub fn min_rtt(&mut self, neighbor: N, min_rtt: Option) { + pub fn rtt(&mut self, neighbor: N, rtt: Option) { let Some(peer) = self.neighbors.get_mut(&neighbor) else { return; }; - if peer.min_rtt == min_rtt { + if peer.rtt == rtt { return; } - peer.min_rtt = min_rtt; + peer.rtt = rtt; self.update(self.through(neighbor)); } @@ -1067,10 +1063,10 @@ impl Router { best.map(|(_, selected)| selected) } - /// Every usable feasible route to `id`, with this node's metric and min RTT - /// through each. Any of them forwards without a loop, if it - /// [carries](Self::carries) the claim. - pub fn routes(&self, id: Id) -> Vec<(N, Metric, MinRtt)> { + /// Every usable feasible route to `id`, with this node's metric and RTT through + /// each. Any of them forwards without a loop, if it [carries](Self::carries) the + /// claim. + pub fn routes(&self, id: Id) -> Vec<(N, Metric, Option)> { if self.origins.contains_key(&id) { return Vec::new(); } @@ -1078,26 +1074,22 @@ impl Router { .filter(|(_, entry)| !entry.held && self.feasible(id, entry.advert)) .filter_map(|(&next, entry)| { let peer = &self.neighbors[&next]; - let min_rtt = MinRtt { - upstream: entry.advert.min_rtt, - link: peer.min_rtt, - }; - Some((next, add(entry.advert.metric, peer.link.cost)?, min_rtt)) + Some(( + next, + add(entry.advert.metric, peer.link.cost)?, + through_rtt(entry.advert, peer), + )) }) .collect() } - /// This node's min RTT to `id` along `route`: 0 at the identity's own node. - fn route_min_rtt(&self, id: Id, route: &Selected) -> Option { + /// This node's RTT to `id` along `route`: 0 at the identity's own node. + fn route_rtt(&self, id: Id, route: &Selected) -> Option { let Some(next) = route.next else { return Some(Duration::ZERO); }; let entry = self.rx.get(&id)?.get(&next)?; - MinRtt { - upstream: entry.advert.min_rtt, - link: self.neighbors[&next].min_rtt, - } - .total() + through_rtt(entry.advert, &self.neighbors[&next]) } /// Whether a request for a claim at `claim` may go through `next`. @@ -1677,7 +1669,7 @@ impl Router { seqno: route.seqno, metric: route.metric, down_only: route.down_only, - min_rtt: self.route_min_rtt(id, &route).filter(|_| !self.withhold), + rtt: self.route_rtt(id, &route), })), None if serves && ctx.live && was.is_some() => Some(Reach::Held), _ => None, diff --git a/rs/moq-net/src/route/node.rs b/rs/moq-net/src/route/node.rs index c65a2e1a7b5..5248e30c17a 100644 --- a/rs/moq-net/src/route/node.rs +++ b/rs/moq-net/src/route/node.rs @@ -25,7 +25,6 @@ use crate::{Instance, Path}; use rand::RngExt; use super::{Error, GENERATED, Id, Link, Metric, Out, QUALIFIED, Reach, Rev, Router, Seqno}; -use crate::origin::MinRtt; /// One announce stream in one direction. pub(crate) type Key = u64; @@ -59,8 +58,8 @@ pub(crate) struct Served { /// The selected route's metric: a path through the claim costs the Announce /// Cost plus this. pub metric: Metric, - /// The route's min RTT, informational. - pub min_rtt: MinRtt, + /// The round trip to the route's origin node, informational. + pub rtt: Option, /// Whether a usable route serves it; a held one leaves it shown but unavailable. pub available: bool, } @@ -359,9 +358,9 @@ struct State { routes: HashMap, } -/// An identity's usable feasible routes: each stream, its metric and min RTT, and -/// the peer's name. -type Routes = Vec<(Key, Metric, MinRtt, Id)>; +/// An identity's usable feasible routes: each stream, its metric and RTT, and the +/// peer's name. +type Routes = Vec<(Key, Metric, Option, Id)>; /// Where one shown claim is served: each stream, what it was last sent, and /// whether it confirmed serving it. @@ -424,16 +423,11 @@ impl Node { /// A node declaring the origin's hop id, so the origin has one identity on every /// wire. A declared identity is never qualified, so an id carrying that marker - /// loses it, or is replaced by a random one if nothing is left. One that - /// `withholds` advertises no min RTT. - pub fn origin(hop: std::num::NonZeroU64, withhold: bool) -> Self { + /// loses it, or is replaced by a random one if nothing is left. + pub fn origin(hop: std::num::NonZeroU64) -> Self { let mut rng = kio::clock::rng(); let id = declared_for(hop).unwrap_or_else(|| rng.random_range(1..=GENERATED)); - let node = Self::new(id, rng.random()); - if withhold { - node.lock().router.withhold(); - } - node + Self::new(id, rng.random()) } fn lock(&self) -> Locked<'_> { @@ -597,9 +591,9 @@ impl Stream { self.with(|router, key| router.goaway(key)); } - /// The link to the peer has `min_rtt`, or `None` unmeasured. - pub fn min_rtt(&self, min_rtt: Option) { - self.with(|router, key| router.min_rtt(key, min_rtt)); + /// The link to the peer has `rtt`, or `None` unmeasured. + pub fn rtt(&self, rtt: Option) { + self.with(|router, key| router.rtt(key, rtt)); } /// This stream's session now serves `claim` of `id`, so a stream that no longer @@ -693,7 +687,7 @@ impl State { /// the routes changed too. fn home(&mut self, id: Id, claims: BTreeSet, ended: &BTreeMap<(Id, Path), bool>) { let routes: Routes = (self.router.routes(id).into_iter()) - .map(|(key, metric, min_rtt)| (key, metric, min_rtt, self.router.name(key, id))) + .map(|(key, metric, rtt)| (key, metric, rtt, self.router.name(key, id))) .collect(); let last = self.routes.get(&id).map_or(&[][..], Vec::as_slice); let claims = match last == routes.as_slice() { @@ -718,29 +712,29 @@ impl State { for claim in shown.into_iter().rev() { let shown = &self.router.shown()[&(id, claim.clone())]; let (item, available) = (shown.item.clone(), shown.available); - let carrying: Vec<(Key, Metric, MinRtt, Id)> = (routes.iter()) + let carrying: Vec<(Key, Metric, Option, Id)> = (routes.iter()) .filter(|(key, ..)| self.router.carries(*key, &claim)) .copied() .collect(); let home = self.homes.entry((id, claim.clone())).or_default(); // With no route left, the router no longer knows the name a stream's peer // gave the identity, so it keeps the one it was served under. - let wanted: Vec<(Key, Metric, MinRtt, bool, Id)> = match carrying.is_empty() { + let wanted: Vec<(Key, Metric, Option, bool, Id)> = match carrying.is_empty() { true => (home.keys.iter()) - .map(|(key, slot)| (*key, slot.served.metric, slot.served.min_rtt, false, slot.named)) + .map(|(key, slot)| (*key, slot.served.metric, slot.served.rtt, false, slot.named)) .collect(), false => (carrying.into_iter()) - .map(|(key, metric, min_rtt, named)| (key, metric, min_rtt, available, named)) + .map(|(key, metric, rtt, named)| (key, metric, rtt, available, named)) .collect(), }; for slot in home.keys.values_mut() { slot.leaving = true; } - for (key, metric, min_rtt, available, named) in wanted { + for (key, metric, rtt, available, named) in wanted { let served = Served { claim: item.clone(), metric, - min_rtt, + rtt, available, }; match home.keys.get_mut(&key) { @@ -828,11 +822,11 @@ impl State { } let slots: BTreeMap = slots.filter(|(_, slot)| !slot.leaving).map(|(k, s)| (*k, s)).collect(); let want: BTreeMap = (carrying.iter()) - .map(|&&(key, metric, min_rtt, named)| { + .map(|&&(key, metric, rtt, named)| { let served = Served { claim: shown.item.clone(), metric, - min_rtt, + rtt, available: shown.available, }; (key, (served, named)) @@ -935,10 +929,7 @@ mod tests { served: Served { claim: claim(&instance, 5), metric: 1, - min_rtt: MinRtt { - upstream: Some(std::time::Duration::ZERO), - link: None, - }, + rtt: None, available: true, }, }] @@ -1223,7 +1214,7 @@ mod tests { seqno, metric: 0, down_only: false, - min_rtt: None, + rtt: None, })) }; for origin in 10..110 { @@ -1347,7 +1338,7 @@ mod tests { seqno: 0, metric: 0, down_only: false, - min_rtt: None, + rtt: None, }; receive.route(1, Some(Reach::Finite(advert))); receive.batch(1, vec![(p("live/x"), Some(claim(&instance, 0)))], Some(1)); diff --git a/rs/moq-net/src/route/rtt.rs b/rs/moq-net/src/route/rtt.rs index f701c82e96e..21434b5f453 100644 --- a/rs/moq-net/src/route/rtt.rs +++ b/rs/moq-net/src/route/rtt.rs @@ -13,6 +13,8 @@ use std::{collections::VecDeque, time::Duration}; pub(crate) struct LinkRtt { reads: VecDeque, reported: Option, + /// Reads in a row that found no RTT. + misses: usize, } impl LinkRtt { @@ -28,14 +30,22 @@ impl LinkRtt { self.reported } + /// Whether to keep reading: false once a window of reads found no RTT, since a + /// transport that reports none that long, past a first ping, never will. + pub fn reading(&self) -> bool { + self.misses < Self::WINDOW + } + /// Take one read of the smoothed RTT, `None` when the transport reports none, and /// return the min RTT to report. pub fn read(&mut self, rtt: Option) -> Option { let Some(rtt) = rtt else { self.reads.clear(); self.reported = None; + self.misses += 1; return None; }; + self.misses = 0; if self.reads.len() == Self::WINDOW { self.reads.pop_front(); } @@ -117,6 +127,15 @@ mod tests { assert_eq!(link.read(ms(90)), ms(90)); } + #[test] + fn a_window_without_rtt_stops_reading() { + let mut link = LinkRtt::default(); + feed(&mut link, std::iter::repeat_n(None, 29)); + assert!(link.reading()); + link.read(None); + assert!(!link.reading()); + } + #[test] fn a_fraction_of_a_millisecond_rounds_up() { let mut link = LinkRtt::default(); diff --git a/rs/moq-net/src/route/tests/mod.rs b/rs/moq-net/src/route/tests/mod.rs index 7e9756b986c..f4204b673a1 100644 --- a/rs/moq-net/src/route/tests/mod.rs +++ b/rs/moq-net/src/route/tests/mod.rs @@ -6,8 +6,8 @@ //! lite-07 claim already covers the paths below it, so the slash is dropped. mod handoff; -mod min_rtt; mod routing; +mod rtt; use std::{ collections::{BTreeMap, BTreeSet, VecDeque}, @@ -376,13 +376,8 @@ impl Sim { } /// `node` measures its link to `peer` at `ms`, or nothing. - pub fn min_rtt(&mut self, node: Node, peer: Node, ms: Option) { - self.nodes[node].min_rtt(peer, ms.map(std::time::Duration::from_millis)); - self.flush(); - } - - pub fn withhold(&mut self, node: Node) { - self.nodes[node].withhold(); + pub fn rtt(&mut self, node: Node, peer: Node, ms: Option) { + self.nodes[node].rtt(peer, ms.map(std::time::Duration::from_millis)); self.flush(); } diff --git a/rs/moq-net/src/route/tests/routing.rs b/rs/moq-net/src/route/tests/routing.rs index 6666a4770f2..626c197fe54 100644 --- a/rs/moq-net/src/route/tests/routing.rs +++ b/rs/moq-net/src/route/tests/routing.rs @@ -783,7 +783,7 @@ fn a_newer_request_raises_the_pending_one() { seqno: 0, metric: 1, down_only: false, - min_rtt: None, + rtt: None, }; relay.route(up, 7, Some(Reach::Finite(advert))); relay.drain(); @@ -871,7 +871,7 @@ fn an_overflowing_route_neither_certifies_nor_holds() { seqno: 0, metric, down_only: false, - min_rtt: None, + rtt: None, })) }; let item = Item { @@ -923,7 +923,7 @@ fn a_fresh_uncertified_claim_waits_on_an_older_floor() { seqno: 0, metric, down_only: false, - min_rtt: None, + rtt: None, })) }; let item = Item { diff --git a/rs/moq-net/src/route/tests/min_rtt.rs b/rs/moq-net/src/route/tests/rtt.rs similarity index 62% rename from rs/moq-net/src/route/tests/min_rtt.rs rename to rs/moq-net/src/route/tests/rtt.rs index 6f7e8810f18..0abdd554600 100644 --- a/rs/moq-net/src/route/tests/min_rtt.rs +++ b/rs/moq-net/src/route/tests/rtt.rs @@ -1,5 +1,5 @@ -//! The min RTT a route carries: each node adds its link's to the sum it received, and -//! nothing ranks or selects a route by it. +//! The RTT a route carries: each node adds its link's min RTT to the sum it received, +//! and nothing ranks or selects a route by it. use std::time::Duration; @@ -14,10 +14,10 @@ fn ms(ms: u64) -> Option { Some(Duration::from_millis(ms)) } -/// The min RTT `node` advertises to `peer` for `id`. +/// The RTT `node` advertises to `peer` for `id`. fn advertised(sim: &Sim, node: Node, peer: Node, id: Id) -> Option { match sim.sent(node, peer, id) { - Some(Reach::Finite(advert)) => advert.min_rtt, + Some(Reach::Finite(advert)) => advert.rtt, other => panic!("{node} sends {peer} {other:?}"), } } @@ -29,9 +29,9 @@ fn chain() -> Sim { sim.join(origin, a, 1); sim.join(a, b, 1); sim.join(b, c, 1); - sim.min_rtt(a, origin, Some(10)); - sim.min_rtt(b, a, Some(20)); - sim.min_rtt(c, b, Some(30)); + sim.rtt(a, origin, Some(10)); + sim.rtt(b, a, Some(20)); + sim.rtt(c, b, Some(30)); originate(&mut sim, origin, 7); assert!(sim.run()); sim @@ -45,14 +45,7 @@ fn each_relay_adds_its_link_to_the_sum() { assert_eq!(advertised(&sim, a, b, 7), ms(10)); assert_eq!(advertised(&sim, b, c, 7), ms(30)); let routes = sim.nodes[c].routes(7); - let [(next, _, min_rtt)] = routes[..] else { - panic!("one route: {routes:?}"); - }; - assert_eq!(next, b); - assert_eq!( - (min_rtt.upstream, min_rtt.link, min_rtt.total()), - (ms(30), ms(30), ms(60)) - ); + assert_eq!(routes, [(b, 3, ms(60))]); } /// A shift on one link sends one ROUTE_UPDATE on each stream carrying the route from @@ -63,48 +56,41 @@ fn a_shift_updates_each_stream_downstream_once() { let mut sim = chain(); let downstream: usize = [a, b, c].iter().map(|&node| sim.sending(node, 7)).sum(); let before = sim.stats.route_msgs; - sim.min_rtt(a, origin, Some(40)); + sim.rtt(a, origin, Some(40)); assert!(sim.run()); assert_eq!(sim.stats.route_msgs - before, downstream); assert_eq!(advertised(&sim, b, c, 7), ms(60)); let before = sim.stats.route_msgs; - sim.min_rtt(a, origin, Some(40)); + sim.rtt(a, origin, Some(40)); assert!(sim.run()); assert_eq!(sim.stats.route_msgs, before); } -/// A link with no measurement, or a node that withholds, leaves the sum unknown from -/// there on, while the nodes before it still carry theirs. +/// A link with no measurement leaves the sum unknown from there on, while the nodes +/// before it still carry theirs. #[test] -fn an_unknown_link_or_a_withholding_node_ends_the_sum() { +fn an_unknown_link_ends_the_sum() { let (a, b, c) = (1, 2, 3); let mut sim = chain(); - sim.min_rtt(b, a, None); + sim.rtt(b, a, None); assert!(sim.run()); assert_eq!(advertised(&sim, a, b, 7), ms(10)); assert_eq!(advertised(&sim, b, c, 7), None); - - let mut sim = chain(); - sim.withhold(b); - assert!(sim.run()); - assert_eq!(advertised(&sim, b, c, 7), None); - let routes = sim.nodes[c].routes(7); - assert_eq!(routes[0].2.link, ms(30), "c still measures its own link"); - assert_eq!(routes[0].2.total(), None); + assert_eq!(sim.nodes[c].routes(7)[0].2, None); } -/// The metric selects; a far lower min RTT through a costlier link changes nothing. +/// The metric selects; a far lower RTT through a costlier link changes nothing. #[test] -fn min_rtt_never_selects_a_route() { +fn rtt_never_selects_a_route() { let (origin, cheap, fast, edge) = (0, 1, 2, 3); let mut sim = Sim::new(4); sim.join(origin, cheap, 1); sim.join(origin, fast, 5); sim.join(cheap, edge, 1); sim.join(fast, edge, 1); - sim.min_rtt(cheap, origin, Some(200)); - sim.min_rtt(fast, origin, Some(1)); + sim.rtt(cheap, origin, Some(200)); + sim.rtt(fast, origin, Some(1)); originate(&mut sim, origin, 7); assert!(sim.run()); assert_eq!(sim.route(edge, 7).and_then(|route| route.next), Some(cheap)); diff --git a/rs/moq-relay/src/cluster.rs b/rs/moq-relay/src/cluster.rs index 090d96f4de9..844ee1e1ba6 100644 --- a/rs/moq-relay/src/cluster.rs +++ b/rs/moq-relay/src/cluster.rs @@ -767,18 +767,6 @@ pub struct Config { )] pub max_hops: Option, - /// Advertise no min RTT on the routes this relay sends moq-lite-07 peers, so a - /// route crossing it reports none from here on. Boolean flag: pass - /// `--cluster-withhold-min-rtt` (or `=true` / `=false`). - #[usage( - name = "cluster-withhold-min-rtt", - long = "cluster-withhold-min-rtt", - env = "MOQ_CLUSTER_WITHHOLD_MIN_RTT", - setting = "cluster.withhold_min_rtt", - bool_value - )] - pub withhold_min_rtt: bool, - /// Connect to one or more other cluster nodes. Each entry is a full URL, e.g. /// `https://host/?jwt=TOKEN`, or an object with `url`, `cost`, `egress`, /// `token`, and `upstream`. A bare host or `host:port` is refused; pass the @@ -1478,7 +1466,6 @@ impl Cluster { .into(), None => origin::RouteKey::generate(), }); - origin_config.withhold_min_rtt = config.withhold_min_rtt; if let Some(max) = config.max_hops { origin_config.max_hops = NonZeroUsize::new(max) .filter(|max| max.get() <= 32) diff --git a/rs/moq-relay/src/settings.rs b/rs/moq-relay/src/settings.rs index 333a0323785..0f6aece673a 100644 --- a/rs/moq-relay/src/settings.rs +++ b/rs/moq-relay/src/settings.rs @@ -77,9 +77,6 @@ struct Cluster { #[usage(env = "MOQ_CLUSTER_MAX_HOPS", cli("--cluster-max-hops"))] max_hops: Option, - #[usage(env = "MOQ_CLUSTER_WITHHOLD_MIN_RTT", cli("--cluster-withhold-min-rtt"))] - withhold_min_rtt: Option, - #[usage(env = "MOQ_CLUSTER_CONNECT", cli("--cluster-connect"), parse = "list_by_comma")] connect: Option>, diff --git a/rs/moq-tokio/tests/integration/broadcast.rs b/rs/moq-tokio/tests/integration/broadcast.rs index 5d80f79134c..f7a941ab4e2 100644 --- a/rs/moq-tokio/tests/integration/broadcast.rs +++ b/rs/moq-tokio/tests/integration/broadcast.rs @@ -952,33 +952,12 @@ async fn announce_lifecycle_test(version: &str) { .expect("server task failed"); } -/// A lite-07 route carries the publisher's min RTT, 0 at its own node, and the -/// subscriber adds its own link's; a publisher that withholds it sends none. -#[tokio::test] -async fn broadcast_moq_lite_07_route_min_rtt() { - let reported = first_route(moq_net::origin::Config::default()).await; - let min_rtt = reported.min_rtt(); - assert_eq!(min_rtt.upstream, Some(std::time::Duration::ZERO)); - let link = min_rtt.link.expect("QUIC reports an RTT"); - assert!(link > std::time::Duration::ZERO, "rounded up to a whole millisecond"); - assert_eq!(min_rtt.total(), Some(link)); - - let mut config = moq_net::origin::Config::default(); - config.withhold_min_rtt = true; - let withheld = first_route(config).await; - assert_eq!(withheld.min_rtt().upstream, None); - assert!( - withheld.min_rtt().link.is_some(), - "the subscriber still measures its link" - ); - assert_eq!(withheld.min_rtt().total(), None); -} - -/// The route a lite-07 subscriber first sees for a broadcast an origin built from -/// `config` serves. -async fn first_route(config: moq_net::origin::Config) -> moq_net::origin::RouteInfo { +/// A lite-07 route carries the publisher's RTT, 0 at its own node, and the subscriber +/// adds its own link's. +#[tokio::test] +async fn broadcast_moq_lite_07_route_rtt() { let version = "moq-lite-07-wip"; - let pub_origin = moq_tokio::origin::spawn_config(config); + let pub_origin = moq_tokio::origin::spawn(); let _route = pub_origin .route("room/first", moq_net::origin::RouteOptions::default()) .expect("create broadcast"); @@ -1021,7 +1000,8 @@ async fn first_route(config: moq_net::origin::Config) -> moq_net::origin::RouteI .await .expect("server task panicked") .expect("server task failed"); - update.route + let rtt = update.route.rtt().expect("QUIC reports an RTT"); + assert!(rtt > std::time::Duration::ZERO, "rounded up to a whole millisecond"); } /// Read `count` groups (one frame each) off the subscriber, returning the sorted diff --git a/swift/Sources/Moq/Aliases.swift b/swift/Sources/Moq/Aliases.swift index 973eaa39a0b..b349a961bb5 100644 --- a/swift/Sources/Moq/Aliases.swift +++ b/swift/Sources/Moq/Aliases.swift @@ -55,11 +55,9 @@ public typealias Datagram = MoqFFI.MoqDatagram /// one, and `==` also compares `isMinted()`, so pin a reported stream with the instance, not its text. public typealias Instance = MoqFFI.MoqInstance /// The read-only facts on a received route: the `Instance` of the stream it serves, -/// relay hop ids (oldest first), and its static production and link cost, lower wins. +/// relay hop ids (oldest first), its static production and link cost, lower wins, and its +/// informational `rtt` to the origin node. public typealias RouteInfo = MoqFFI.MoqRouteInfo -/// The minimum round trip to a route's origin node over moq-lite-07: `upstream` past the -/// session's peer, `link` to the peer, and their `total`. Informational; it never ranks a route. -public typealias MinRtt = MoqFFI.MoqMinRtt /// A route over a prefix: the origin-relative `prefix`, what each filter /// wildcard matched (`captures`, a required list, empty without wildcards), and the `route` serving it. /// Resolve a path with `OriginConsumer.request`. From 3fe08a6e71192120a0e85f0f9750c026b7b423e0 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Tue, 10 Nov 2026 19:11:10 -0800 Subject: [PATCH 08/11] fix(net): restart the link RTT window after a pause in routes Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/lite/subscriber/routed.rs | 71 ++++++++++++++++++++---- 1 file changed, 59 insertions(+), 12 deletions(-) diff --git a/rs/moq-net/src/lite/subscriber/routed.rs b/rs/moq-net/src/lite/subscriber/routed.rs index bd31a5a5768..dacef6aec76 100644 --- a/rs/moq-net/src/lite/subscriber/routed.rs +++ b/rs/moq-net/src/lite/subscriber/routed.rs @@ -101,6 +101,8 @@ pub(super) struct Routed { rtt: route::LinkRtt, /// When to read the link's RTT next, or `None` once it stopped reading. read_rtt: Option, + /// Whether the peer has a route here, so the link's RTT is read. + sampling: bool, } impl Routed { @@ -133,13 +135,6 @@ impl Routed { let stream = node .receive(scoped(&link, ®ions, key), prefix.clone()) .map_err(refused)?; - let mut rtt = route::LinkRtt::default(); - stream.rtt(rtt.read(subscriber.session.metrics().rtt())); - // A transport can report its first RTT later, such as qmux after its first ping. - let read_rtt = Some(crate::time::Deadline::after( - &subscriber.runtime, - route::LinkRtt::INTERVAL, - )); Ok(Self { node, stream, @@ -162,8 +157,9 @@ impl Routed { previous: HashMap::new(), ready: Default::default(), drained: false, - rtt, - read_rtt, + rtt: Default::default(), + read_rtt: None, + sampling: false, }) } @@ -512,9 +508,25 @@ impl Routed { waiter: &kio::Waiter, ) -> Result<(), Error> { // Only a route the peer sent adds the link's min RTT, so a peer sending none - // costs no timer. - while !self.routes.is_empty() - && let Some(read_rtt) = &mut self.read_rtt + // costs no timer. Reading restarts its window on resuming, so a pause never + // keeps a reading older than the window. + match (self.routes.is_empty(), self.sampling) { + (true, true) => { + self.sampling = false; + self.rtt = Default::default(); + self.read_rtt = None; + self.stream.rtt(None); + } + (false, false) => { + self.sampling = true; + self.read_rtt = Some(crate::time::Deadline::after( + &subscriber.runtime, + std::time::Duration::ZERO, + )); + } + _ => {} + } + while let Some(read_rtt) = &mut self.read_rtt && read_rtt.poll(waiter).is_ready() { let last = self.rtt.reported(); @@ -998,6 +1010,41 @@ mod tests { assert_eq!(routed.rtt.reported(), Some(rtt)); } + /// A stream whose peer ends every route stops reading, and starts a fresh window + /// once one returns, so a reading from before the pause never stays reported. + #[kio::test] + async fn a_pause_restarts_the_window() { + use crate::lite::test_transport::SinkStats; + let ms = std::time::Duration::from_millis; + let session = SinkSession::new(Default::default()).with_stats(SinkStats::default().with_rtt(ms(10))); + let (subscriber, _consumer) = super::super::tests::restart_subscriber(session.clone()); + let prefix = Path::empty(); + let mut routed = super::Routed::new(&subscriber, &prefix, 1, &Default::default()).unwrap(); + let mut decoder = AnnounceDecoder::default(); + let mut writer = crate::coding::Writer::new( + crate::lite::test_transport::SinkSend::new(Default::default()), + crate::lite::Version::Lite07, + ); + let waiter = kio::Waiter::noop(); + feed(&mut routed, &mut decoder, &[]).unwrap(); + routed.poll(&subscriber, &mut writer, &waiter).unwrap(); + assert_eq!(routed.rtt.reported(), Some(ms(10))); + + routed.receive(&prefix, Msg::RouteEnd { id: 0 }, &mut decoder).unwrap(); + routed.poll(&subscriber, &mut writer, &waiter).unwrap(); + assert!(routed.read_rtt.is_none(), "no route, no timer"); + session.set_stats(SinkStats::default().with_rtt(ms(100))); + kio::clock::sim::advance(std::time::Duration::from_secs(60)).await; + + let route = Msg::RouteStart { + node: 7, + advert: ADVERT, + }; + routed.receive(&prefix, route, &mut decoder).unwrap(); + routed.poll(&subscriber, &mut writer, &waiter).unwrap(); + assert_eq!(routed.rtt.reported(), Some(ms(100))); + } + /// What the stream keeps between floors is the net change to its committed claims, /// however many messages the peer sends without one. #[kio::test] From e71cb1fbf35cd4794b67ba5af3b9e7dc8fc08d47 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 00:27:14 -0800 Subject: [PATCH 09/11] fix(net): sample a link's RTT before serving the first claims Synced drained the node's outputs before poll started sampling, so the first claims served carried no RTT until the next read. Note in the cluster doc that a route this node serves itself has no RTT. Co-Authored-By: Claude Opus 5.5 --- doc/bin/relay/cluster.md | 3 +- rs/moq-net/src/lite/subscriber/routed.rs | 81 ++++++++++++++++++------ 2 files changed, 62 insertions(+), 22 deletions(-) diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index 54d7e5233e6..c6ad3a17580 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -225,7 +225,8 @@ A moq-lite-07-wip route also carries its RTT: each relay adds its link's lowest smoothed RTT over the last 30 seconds to the sum it received, so an application reads the round trip to a publisher through any number of relays from the route it was announced on (`RouteInfo::rtt`). The value is absent -while any link on the path has no RTT measurement. It is informational and +while any link on the path has no RTT measurement, and on a route this node +serves itself, such as a broadcast published to it. It is informational and never picks a route; the link cost does that. A link's value changes only when its minimum moves past the larger of 5 ms and a quarter, so jitter and short queues send no route updates. diff --git a/rs/moq-net/src/lite/subscriber/routed.rs b/rs/moq-net/src/lite/subscriber/routed.rs index dacef6aec76..b8d808b24eb 100644 --- a/rs/moq-net/src/lite/subscriber/routed.rs +++ b/rs/moq-net/src/lite/subscriber/routed.rs @@ -404,6 +404,7 @@ impl Routed { /// Act on everything the node queued for the stream: serve the claims it hands /// over. pub(super) fn outputs(&mut self, subscriber: &Subscriber) { + self.sample(subscriber); while let Some(output) = self.stream.try_output() { self.output(subscriber, output); } @@ -500,16 +501,12 @@ impl Routed { } } - /// Write the queued replies, and serve requests on every claim whose queue woke. - pub(super) fn poll( - &mut self, - subscriber: &Subscriber, - writer: &mut crate::coding::Writer, - waiter: &kio::Waiter, - ) -> Result<(), Error> { - // Only a route the peer sent adds the link's min RTT, so a peer sending none - // costs no timer. Reading restarts its window on resuming, so a pause never - // keeps a reading older than the window. + /// Start or stop reading the link's min RTT as the peer's routes come and go. + /// + /// Only a route the peer sent adds the link's min RTT, so a peer sending none costs + /// no timer. Starting reads at once, so the first claims served already carry it, + /// and in a fresh window, so a pause never keeps a reading older than the window. + fn sample(&mut self, subscriber: &Subscriber) { match (self.routes.is_empty(), self.sampling) { (true, true) => { self.sampling = false; @@ -519,23 +516,35 @@ impl Routed { } (false, false) => { self.sampling = true; - self.read_rtt = Some(crate::time::Deadline::after( - &subscriber.runtime, - std::time::Duration::ZERO, - )); + self.read_link(subscriber); } _ => {} } + } + + /// Read the link's smoothed RTT once and schedule the next read. + fn read_link(&mut self, subscriber: &Subscriber) { + let last = self.rtt.reported(); + let now = self.rtt.read(subscriber.session.metrics().rtt()); + if now != last { + self.stream.rtt(now); + } + self.read_rtt = (self.rtt.reading()) + .then(|| crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL)); + } + + /// Write the queued replies, and serve requests on every claim whose queue woke. + pub(super) fn poll( + &mut self, + subscriber: &Subscriber, + writer: &mut crate::coding::Writer, + waiter: &kio::Waiter, + ) -> Result<(), Error> { + self.sample(subscriber); while let Some(read_rtt) = &mut self.read_rtt && read_rtt.poll(waiter).is_ready() { - let last = self.rtt.reported(); - let now = self.rtt.read(subscriber.session.metrics().rtt()); - if now != last { - self.stream.rtt(now); - } - self.read_rtt = (self.rtt.reading()) - .then(|| crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL)); + self.read_link(subscriber); } while let Poll::Ready(output) = self.stream.poll_output(waiter) { self.output(subscriber, output); @@ -1010,6 +1019,36 @@ mod tests { assert_eq!(routed.rtt.reported(), Some(rtt)); } + /// Claims served when the peer reports Synced, before the stream is first polled, + /// already carry the link's RTT. + #[kio::test] + async fn the_first_claims_served_carry_the_rtt() { + use crate::lite::test_transport::SinkStats; + let ms = std::time::Duration::from_millis; + let session = SinkSession::new(Default::default()).with_stats(SinkStats::default().with_rtt(ms(10))); + let (subscriber, _consumer) = super::super::tests::restart_subscriber(session); + let prefix = Path::empty(); + let mut routed = super::Routed::new(&subscriber, &prefix, 1, &Default::default()).unwrap(); + let mut decoder = AnnounceDecoder::default(); + let route = Msg::RouteStart { + node: 7, + advert: Advert { + rtt: Some(ms(5)), + ..ADVERT + }, + }; + let floor = Msg::Floor { + id: 0, + revision: Some(0), + }; + for msg in [route, start("live"), floor] { + routed.receive(&prefix, msg, &mut decoder).unwrap(); + } + routed.outputs(&subscriber); + let rtts: Vec<_> = routed.served.values().map(|entry| entry.route.route.rtt()).collect(); + assert_eq!(rtts, [Some(ms(15))]); + } + /// A stream whose peer ends every route stops reading, and starts a fresh window /// once one returns, so a reading from before the pause never stays reported. #[kio::test] From 6cdeb936050320cfc0f88249292cda25a3c021b8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 02:17:04 -0800 Subject: [PATCH 10/11] style: rustfmt Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/lite/subscriber/routed.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rs/moq-net/src/lite/subscriber/routed.rs b/rs/moq-net/src/lite/subscriber/routed.rs index b8d808b24eb..ee42ed88291 100644 --- a/rs/moq-net/src/lite/subscriber/routed.rs +++ b/rs/moq-net/src/lite/subscriber/routed.rs @@ -529,8 +529,8 @@ impl Routed { if now != last { self.stream.rtt(now); } - self.read_rtt = (self.rtt.reading()) - .then(|| crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL)); + self.read_rtt = + (self.rtt.reading()).then(|| crate::time::Deadline::after(&subscriber.runtime, route::LinkRtt::INTERVAL)); } /// Write the queued replies, and serve requests on every claim whose queue woke. From 11e4198e8a2c2a5815fd0478aba9519a4aaa21b3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 11 Nov 2026 03:36:46 -0800 Subject: [PATCH 11/11] docs(relay): route RTT ends at the origin node, not the publisher Co-Authored-By: Claude Opus 5.5 --- doc/bin/relay/cluster.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index c6ad3a17580..723828dc151 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -223,8 +223,10 @@ crosses. A live publisher seeds 0, and caching never changes it. A moq-lite-07-wip route also carries its RTT: each relay adds its link's lowest smoothed RTT over the last 30 seconds to the sum it received, so an -application reads the round trip to a publisher through any number of relays -from the route it was announced on (`RouteInfo::rtt`). The value is absent +application reads the round trip through any number of relays to the route's +origin node, the one the broadcast is published to, from the route it was +announced on (`RouteInfo::rtt`). The publisher's own connection to that node is +not included. The value is absent while any link on the path has no RTT measurement, and on a route this node serves itself, such as a broadcast published to it. It is informational and never picks a route; the link cost does that. A link's value changes only