Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion dart/moq/lib/src/aliases.dart
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,8 @@ typedef AnnounceConfig = MoqAnnounceConfig;
/// Follows a specific broadcast path across the publisher instances that serve it.
typedef Following = MoqFollowing;

/// The read-only facts on a received route: a static cost and the stream's instance.
/// The read-only facts on a received route: a static cost, the stream's instance, and
/// its informational `rtt` to the origin node.
typedef RouteInfo = MoqRouteInfo;

/// One continuous stream at a path: `Instance.parse` names one, and only a session mints one.
Expand Down
20 changes: 19 additions & 1 deletion dart/moq_ffi/lib/src/moq.dart
Original file line number Diff line number Diff line change
Expand Up @@ -3012,10 +3012,12 @@ class MoqRouteInfo {
final int cost;
final MoqInstance instance;
final bool available;
final Duration? rtt;
MoqRouteInfo({
required this.cost,
required this.instance,
required this.available,
this.rtt,
});
}

Expand All @@ -3041,8 +3043,18 @@ class FfiConverterMoqRouteInfo {
);
final available = available_lifted.value;
new_offset += available_lifted.bytesRead;
final rtt_lifted = FfiConverterOptionalDuration.read(
Uint8List.view(buf.buffer, new_offset),
);
final rtt = rtt_lifted.value;
new_offset += rtt_lifted.bytesRead;
return LiftRetVal(
MoqRouteInfo(cost: cost, instance: instance, available: available),
MoqRouteInfo(
cost: cost,
instance: instance,
available: available,
rtt: rtt,
),
new_offset - buf.offsetInBytes,
);
}
Expand All @@ -3052,6 +3064,7 @@ class FfiConverterMoqRouteInfo {
FfiConverterUInt64.allocationSize(value.cost) +
FfiConverterMoqInstance.allocationSize(value.instance) +
FfiConverterBool.allocationSize(value.available) +
FfiConverterOptionalDuration.allocationSize(value.rtt) +
0;
final buf = Uint8List(total_length);
write(value, buf);
Expand All @@ -3072,13 +3085,18 @@ class FfiConverterMoqRouteInfo {
value.available,
Uint8List.view(buf.buffer, new_offset),
);
new_offset += FfiConverterOptionalDuration.write(
value.rtt,
Uint8List.view(buf.buffer, new_offset),
);
return new_offset - buf.offsetInBytes;
}

static int allocationSize(MoqRouteInfo value) {
return FfiConverterUInt64.allocationSize(value.cost) +
FfiConverterMoqInstance.allocationSize(value.instance) +
FfiConverterBool.allocationSize(value.available) +
FfiConverterOptionalDuration.allocationSize(value.rtt) +
0;
}
}
Expand Down
14 changes: 14 additions & 0 deletions doc/bin/relay/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,20 @@ draining process.
A route's price is the publisher's production cost plus the link costs it
crosses. A live publisher seeds 0, and caching never changes it.

## Route latency

A moq-lite-07-wip route also carries its RTT: each relay adds its link's
lowest smoothed RTT over the last 30 seconds to the sum it received, so an
application reads the round trip through any number of relays to the route's
origin node, the one the broadcast is published to, from the route it was
announced on (`RouteInfo::rtt`). The publisher's own connection to that node is
not included. The value is absent
while any link on the path has no RTT measurement, and on a route this node
serves itself, such as a broadcast published to it. It is informational and
never picks a route; the link cost does that. A link's value changes only
when its minimum moves past the larger of 5 ms and a quarter, so jitter and
short queues send no route updates.

## LAN discovery

On a LAN there may be no one to list. `[cluster.lan]` advertises this relay
Expand Down
37 changes: 28 additions & 9 deletions drafts/draft-lcurley-moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -407,7 +407,7 @@ The subscriber creates the stream with an ANNOUNCE_REQUEST message.
The publisher replies with a single ANNOUNCE_OK message followed by the matching routes and claims and any future changes:

- ROUTE_START: the publisher reaches an origin node.
- ROUTE_UPDATE: a route's seqno, metric, or down-only bit changed.
- ROUTE_UPDATE: a route's seqno, metric, down-only bit, or RTT changed.
- ROUTE_HOLD: the publisher has no usable route but holds it while it recovers.
- ROUTE_END: the publisher let go of a route and every claim under it.
- ANNOUNCE_START: a matching claim of a route's origin node.
Expand Down Expand Up @@ -492,12 +492,24 @@ A receiver adds the session's link cost (see [Cost Parameter](#cost-parameter))
Each link adds at least 1, so a route's metric strictly increases away from its origin.
A metric that overflows 64 bits makes the route unusable.

A route may also carry an RTT: the round trip to its origin node in milliseconds, summed link by link.
It is informational: nothing ranks or selects a route, or judges it feasible, by its RTT.
An origin advertises its own route with an RTT of 0.
A receiver adds its link min RTT for the session to the RTT of every route the peer sends; the sum is its RTT through that peer, which it advertises on the route it selects.
A link min RTT is the lowest smoothed RTT [RFC9002] the connection measured over the last 30 seconds, read once a second, in milliseconds rounded up.
A lifetime minimum never rises after a path change, so it is not a link min RTT.
A node SHOULD keep the link min RTT it adds until the lowest reading differs from it by more than the larger of 5 ms and a quarter of it, so jitter and short queues send no ROUTE_UPDATE and a lasting change sends one per route.
A node whose connection reports no RTT has no link min RTT, so it sends the routes it learned there without one, and a route without one stays without one past every later node.
A sum too large to encode is sent without one too.
An endpoint learns its round trip to an origin node by adding its own link min RTT to the RTT of the route it selected; without its own, the RTT alone covers every hop past its first.
Like the Metric, an RTT is unverified: a peer can misstate only what crosses it.

A node keeps a feasibility distance per Node ID: the highest Seqno of any route it advertised, and the lowest metric it advertised at that Seqno, as in [RFC8966] Section 3.5.1.
A route is feasible when no feasibility distance is kept for its Node ID yet, when its Seqno is greater than the distance's, or when its Seqno is equal and its advertised Metric is lower.
A node selects, per Node ID, among its feasible routes that are not held: a session that has not received a GOAWAY, then the highest local priority, then the lowest metric.
It MUST NOT select an infeasible route, and a distance only ever falls within a Seqno, so advertising a longer route never raises it.
A node MAY mark a session upstream.
A route is down-only when it arrived on an upstream session or with its Down Only bit set; a node sends a down-only route with the bit set, and never on an upstream session.
A route is down-only when it arrived on an upstream session or with its down-only flag set; a node sends a down-only route with the flag set, and never on an upstream session.

A node sends its selected route to an origin node on an Announce Stream it answers only while the route has a claim the stream's request presents (see [Announce](#announce)): it sends ROUTE_START before the first such claim, and once the last one is withdrawn, the batch carrying that withdrawal, closed by its floor (which certifies the withdrawal where the node can certify the request), then ROUTE_END. It starts that route on the stream again only once the subscriber echoed the ROUTE_END. A subscriber therefore holds no route for an origin node its request does not meet.
A node that does not forward, such as a client, sends only its own route, and holds no feasibility distance for any other.
Expand Down Expand Up @@ -1142,7 +1154,8 @@ ROUTE_START Message {
Node ID (i),
Seqno (i),
Metric (i),
Down Only (8),
Flags (8),
[RTT (i),]
}
~~~

Expand All @@ -1160,9 +1173,13 @@ The origin's sequence number of the route, which only the origin advances.
**Metric**:
The publisher's metric to the origin node, 0 when it is the origin.

**Down Only**:
1 if the route is down-only (see [Routing](#routing)), 0 otherwise.
Any other value is a PROTOCOL_VIOLATION.
**Flags**:
A bit field: 0x1 when the route is down-only (see [Routing](#routing)), and 0x2 when an RTT follows.
Any other bit set is a PROTOCOL_VIOLATION.

**RTT**:
The publisher's RTT to the origin node in milliseconds (see [Routing](#routing)), 0 when it is the origin.
Present only when Flags has 0x2 set, and absent when the publisher has none.


## ROUTE_UPDATE {#route-update}
Expand All @@ -1175,7 +1192,8 @@ ROUTE_UPDATE Message {
Route ID (i),
Seqno (i),
Metric (i),
Down Only (8),
Flags (8),
[RTT (i),]
}
~~~

Expand All @@ -1185,14 +1203,14 @@ Set to 0x7 to indicate a ROUTE_UPDATE message.
**Route ID**:
The ordinal implicitly assigned by a prior ROUTE_START on this stream.

**Seqno**, **Metric**, and **Down Only**:
**Seqno**, **Metric**, **Flags**, and **RTT**:
As defined for [ROUTE_START](#route-start), replacing the route's.
A ROUTE_UPDATE for a held route makes it usable again.


## ROUTE_HOLD {#route-hold}
A publisher sends a ROUTE_HOLD message when it has no usable route to the origin node but still holds the route and its claims while it recovers (see [Routing](#routing)), referencing its Route ID.
The receiver keeps the route's last Seqno, Metric, and Down Only, and does not select it until a ROUTE_UPDATE.
The receiver keeps the route's last Seqno, Metric, Flags, and RTT, and does not select it until a ROUTE_UPDATE.

~~~
ROUTE_HOLD Message {
Expand Down Expand Up @@ -2230,6 +2248,7 @@ The `Message Length` describes the payload size on the wire.
- A receiver mints an Instance for each announcement a session of an earlier version delivers, and a `Minted` field beside every Instance (ANNOUNCE_START, ANNOUNCE_RESTART, TRACK, TRACKS, SUBSCRIBE, FETCH, and DATAGRAMS) says whether a receiver minted it, so no Instance text is reserved for minting.
- Added announce compression: ANNOUNCE_START gains `Path Base` and `Path Keep` to copy the head of a live claim's suffix.
- Split routing into two layers on the Announce Stream. ROUTE_START (0x6), ROUTE_UPDATE (0x7), and ROUTE_END (0x8) advertise how the publisher reaches an origin node, named by a Node ID, with a seqno, an additive metric, and a down-only bit, and loops are prevented by Babel's feasibility condition. Every ANNOUNCE_START names its route's Route ID in place of the Hop ID list, so a link change sends one ROUTE_UPDATE per origin rather than an update per claim. Removed the Hop ID list and ANNOUNCE_OK's Hop ID, and replaced the SETUP Hop parameter (0x5) with a [Node ID parameter](#node-id-parameter), so an endpoint that reaches itself detects it at setup.
- ROUTE_START and ROUTE_UPDATE carry Flags, whose 0x2 bit adds an optional RTT: the informational sum of each link's lowest smoothed RTT over the last 30 seconds to the origin node, which never selects a route.
- Added ROUTE_HOLD (0xB), which holds a route while it recovers; ROUTE_END lets it go. Neither withdraws the origin's claims, and the subscriber echoes every ROUTE_END back.
- Added ROUTE_REQUEST (0xA), sent by the subscriber on the Announce Stream to ask for a route with a newer seqno when every route it has is infeasible.
- Added ANNOUNCE_FLOOR (0x9), closing each batch of a route's claims with the origin Revision at which the stream's claims across the requested prefix are certified plus one, or 0 to certify nothing, so a stale copy never revives a claim whose withdrawal a newer one delivered. A floor certifies only the claims its own request shows.
Expand Down
2 changes: 1 addition & 1 deletion go/wrapper/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ type (
ProtocolKind = ffi.MoqProtocolKind
// OriginConfig configures a new origin, such as its maximum cache size in bytes.
OriginConfig = ffi.MoqOriginConfig
// RouteInfo is the read-only facts on a received route: the Instance of the stream it serves (routes with one instance are one continuous stream) and its static production and link cost (lower wins).
// RouteInfo is the read-only facts on a received route: the Instance of the stream it serves (routes with one instance are one continuous stream) and its static production and link cost (lower wins), and its informational Rtt to the origin node.
RouteInfo = ffi.MoqRouteInfo
// Instance names one continuous stream at a path. Compare two with Eq: a session mints one
// for a route whose wire carries none, and it differs from a named instance with the same Text.
Expand Down
3 changes: 2 additions & 1 deletion kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,8 @@ typealias RouteOptions = uniffi.moq.MoqRouteOptions

/**
* The read-only facts on a received route: the [Instance] of the stream it serves, relay hop ids
* (oldest first), and static production and link cost (lower wins).
* (oldest first), static production and link cost (lower wins), and its informational `rtt` to the
* origin node.
*/
typealias RouteInfo = uniffi.moq.MoqRouteInfo

Expand Down
2 changes: 1 addition & 1 deletion quest/m2/lite07-finalize.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ transcript freezes at the cut and a flaw in it can only be fixed on the wire.
- [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - may add SUBSCRIBE_DROP for a stopped group; leaves this list if it takes the no-wire path
- [Delete the old start bookkeeping](/quest/m2/subscribe-ranges/start-bookkeeping.md) - may change START per update on lite-07
- [Prove cache-aware routing](/quest/m2/cache-routing-proof.md) - may add the warm-route message to lite-07; leaves this list if the proof places it in a SETUP extension
- [Route latency](/quest/m2/route-latency.md) - may add a latency sum to ROUTE; leaves this list if the maintainer takes the no-wire path
- [Route latency in js/net](/quest/m2/route-latency.md) - js/net reads and writes ROUTE's RTT
- [A symbolic model proves a route key proof binds to its session and side](/quest/m1/route-key-model.md) - may change what the Route Key Parameter signs, if the model finds a replay, reflection, or relaying attack

## Related
Expand Down
86 changes: 36 additions & 50 deletions quest/m2/route-latency.md
Original file line number Diff line number Diff line change
@@ -1,60 +1,46 @@
# [M] Route latency
# [S] Route latency in js/net

## Goal

The maintainer has decided whether a lite-07 ROUTE carries an informational
latency sum, and the choice is carried out before the cut. If it does, each
node adds its link's smoothed RTT to the sum on the route it selected before
advertising it, so an endpoint learns the latency to an origin node through
any number of relays by adding its own first hop, with no extra track and no
knowledge of the relays between. The sum never ranks or selects a route.
`js/net` reads a lite-07 route's RTT as Rust does: it decodes and encodes
ROUTE_START's and ROUTE_UPDATE's Flags and optional RTT, a browser publishes
its own route at 0, and its announced routes carry the sum the relay
advertised plus its own first hop, unknown when the browser's transport
reports no RTT. The sum never ranks or selects a route.

## Plan

Planned 2026-11-09 (maintainer, /quest-plan from Kixel's audit of its fleet
plan): ✅ settle lite-07's route-layer additions before the cut, since a
field on ROUTE after it needs another wire version.

Why it is worth deciding: an application that wants to show the round trip
between two peers through a cluster has no way to today without publishing a
track for it. A per-hop message keyed by the announce hop list was proposed,
but lite-07 removed that list for one distance-vector route per origin, and
such a route cannot name its path, so a sum is all the wire can carry. The
[status track](/quest/m3/mesh/status.md) publishes per-link RTT for mesh
nodes only, so it cannot answer for a path through an operator's relays.
[moq-dev/moq#3769](https://github.com/moq-dev/moq/pull/3769) priced links by
measurement and was closed for "measurement first, with operator-owned
routing policy"; this is the measurement alone.

If the field is taken, it holds to these:

- `route_order` and feasibility never read it. The metric stays the
operator's static cost, and [link quality](/quest/m5/link-quality.md)
stays the opt-in way to price a link by measurement.
- It comes from what QUIC already measures on the link, quantized, with
hysteresis, so jitter sends no ROUTE_UPDATE
([RFC 9616](https://www.rfc-editor.org/rfc/rfc9616.html) is the reference
for RTT that feeds a routing protocol). Report the ROUTE_UPDATEs one real
shift causes across a chain of relays.
- A node may withhold it, and a sum that crosses such a node is unknown from
there on, marked without a sentinel value. An operator decides whether its
relays report.
- The endpoint's own first hop can be unknown too, such as a browser whose
WebTransport reports no RTT. The application then reads the sum from the
first relay alone, marked as excluding the first hop, never a guess.
- It is unverified, like the metric: a peer can misstate only what crosses
it, and nothing acts on the value.
- The draft, the Rust and JS codecs, and where the application reads it land
together, with `just test interop --all`.

Size the field against the lite-07 draft first, then bring the maintainer the
choice below with that size and the update count.

## Waiting

- Maintainer: decide whether lite-07's ROUTE carries the latency sum. An agent cannot take it, since it adds a field to a wire about to freeze. Options: a quantized, informational sum on ROUTE_START and ROUTE_UPDATE as above, or no wire change, leaving applications to measure with a track of their own. Recommendation: the field, if the sizing shows a shift costs a ROUTE_UPDATE per origin at most; check that the choice is recorded in this Plan.
Decided 2026-11-10 (maintainer, on
[moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458)): ✅ lite-07's
ROUTE carries the latency sum, as RTT. Each node adds its link's windowed
minimum, the lowest of the last 30 once-a-second reads of the smoothed RTT,
re-advertised only past max(5 ms, 25%), never QUIC's lifetime min_rtt, which
never rises after a path change. The endpoint adds its own first hop from its
connection. Rejected: no wire change, leaving applications a track of their
own.

[moq-dev/moq#8458](https://github.com/moq-dev/moq/pull/8458) landed the
draft, the Rust codec and route layer, `origin::RouteInfo::rtt`, and the
bindings' `RouteInfo.rtt`, with no option to withhold it. Its PR body
keeps the sizing simulation: one real shift sends one ROUTE_UPDATE per origin
on each stream downstream of the link, and jitter sends none.

What remains is the JS side, which needs the JS route layer first:

- Decode Flags (0x1 down-only, 0x2 RTT present, any other bit a
PROTOCOL_VIOLATION) and the RTT that follows, in milliseconds, and encode
a browser's own route with an RTT of 0.
- Add `rtt` to `js/net`'s `RouteInfo`, named as Rust's: the advertised sum
plus the same windowed minimum over WebTransport `getStats()`'s smoothed
RTT, and `undefined` while either is unknown.
- Pin the codec to Rust's with the shared corpus, and run
`just test interop --all`.

## Required

- [js/net route layer](/quest/m2/cluster-routing/routes/js.md) - the ROUTE codec and route-backed `RouteInfo` this extends

## Related

- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the ROUTE messages this would extend
- [Routes and announces](/quest/m2/cluster-routing/routes/README.md) - the ROUTE messages this extends
- [Link quality](/quest/m5/link-quality.md) - measured link cost, which feeds selection where this never does
4 changes: 4 additions & 0 deletions rs/moq-ffi/src/origin.rs
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,9 @@ pub struct MoqRouteInfo {
pub instance: Arc<MoqInstance>,
/// Whether a usable path reaches the route's origin; requests may fail while it is not.
pub available: bool,
/// The round trip to the route's origin node over moq-lite-07, absent while any link
/// has no measurement. Informational; it never ranks a route.
pub rtt: Option<std::time::Duration>,
}

impl From<moq_net::origin::RouteInfo> for MoqRouteInfo {
Expand All @@ -102,6 +105,7 @@ impl From<moq_net::origin::RouteInfo> for MoqRouteInfo {
cost: route.cost().value(),
instance: MoqInstance::wrap(route.instance().clone()),
available: route.available(),
rtt: route.rtt(),
}
}
}
Expand Down
1 change: 1 addition & 0 deletions rs/moq-net/src/fuzz.rs
Original file line number Diff line number Diff line change
Expand Up @@ -677,6 +677,7 @@ impl RouteRelay {
seqno: 0,
metric: 0,
down_only: false,
rtt: None,
};
let receive = (0..streams)
.map(|_| {
Expand Down
Loading
Loading