Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 25 additions & 17 deletions doc/bin/relay/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,26 +100,34 @@ For a full mesh, list every other relay, or serve the list from `api.url`.
A session carries both directions, so one dial per pair is enough; listing a
pair on both sides opens a redundant second session.

`cluster.max_hops` bounds the hop path this relay accepts or forwards, from 1
to 32, with 32 as the default. Set it near the mesh's diameter after any single
link or node failure, plus a margin, to cut off long detours during withdrawal
path hunting. Count the publisher's hop too. A cap below that diameter can
partition the mesh; a lower cap only drops longer paths and preserves route
selection among the remaining paths. A peer hears the best route with room
for this relay's hop, even when a cheaper route at the cap serves locally.
`cluster.max_hops` is a legacy setting: it bounds the hop path that
moq-lite-03 through moq-lite-06 carry, and moq-transport drafts 18 through 22
when the MoQ Cluster extension is negotiated, and it leaves with moq-lite-06.
A moq-lite-07-wip route crosses at most 32 nodes whatever this is set to: it
does not hunt for paths, so a lower cap would only hide a short path behind a
cheaper long one. In a mesh that mixes the two, a moq-lite-07-wip route that
crossed in from moq-lite-06 still carries the moq-lite-06 chain: a relay drops
the route when that chain plus the route's publisher is longer than
`max_hops`.

On those older versions it bounds the hop path this relay accepts or forwards,
from 1 to 32, with 32 as the default. Set it near the mesh's diameter after any
single link or node failure, plus a margin, to cut off long detours during
withdrawal path hunting. Count the publisher's hop too. A cap below that
diameter can partition the mesh; a lower cap only drops longer paths and
preserves route selection among the remaining paths. A peer hears the best
route with room for this relay's hop, even when a cheaper route at the cap
serves locally.

```toml
[cluster]
max_hops = 6
```

The same setting is `--cluster-max-hops` or `MOQ_CLUSTER_MAX_HOPS`. It covers
moq-lite-03 through moq-lite-06 and moq-transport drafts 18 through 22 when
the MoQ Cluster extension is negotiated. A moq-lite-07-wip route crosses at
most 32 nodes whatever this is set to: it does not hunt for paths, so a lower
cap would only hide a short path behind a cheaper long one. A forwarded path counts this relay's
own hop even when its peer appends it on receipt. Older versions and peers
without hop paths hide their depth, which this setting cannot bound.
The same setting is `--cluster-max-hops` or `MOQ_CLUSTER_MAX_HOPS`. A forwarded
path counts this relay's own hop even when its peer appends it on receipt.
Versions before moq-lite-03 and peers without hop paths hide their depth, which
this setting cannot bound.

## Upstream links

Expand Down Expand Up @@ -277,8 +285,8 @@ url = "https://api.example.com/cluster/peers"
HTTPS peer-list fetches present the same client certificate as cluster dials,
but authenticate the API with system roots and the hostname in its URL.
Mesh roots, fingerprints, insecure mode, and the mesh hostname override do not
apply to the API. For a private API, set `cluster.api.tls.root` to PEM
paths (or repeat `--cluster-api-tls-root`); these replace system trust
apply to the API. For a private API, set `cluster.api.root` to PEM
paths (or repeat `--cluster-api-root`); these replace system trust
for API fetches only. Malformed roots fail startup, and files reload for new
connections when rotated.

Expand Down Expand Up @@ -355,7 +363,7 @@ services.moq-relay = {
auth.enable = true;
tls.certs = [ { chain = "/etc/moq/leaf.pem"; key = "/run/secrets/moq-leaf.key"; } ];
cluster = {
rootUrl = [ "https://root.example.com:4443" ];
connect = [ "https://root.example.com:4443" ];
certFile = "/run/secrets/moq-cluster.pem";
keyFile = "/run/secrets/moq-cluster.key";
};
Expand Down
8 changes: 3 additions & 5 deletions doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,6 @@ A stream-only relay refuses `preferred_v4`, `preferred_v6`, `lb.id`, and
The io\_uring runtime refuses `preferred_v4`, `preferred_v6`, and `lb.id`;
use the tokio runtime when the listener needs these settings.
`[listen.lb] nonce` needs an `id` in the same table; unset nonce uses 8.
The `--listen-quic-lb-id` and `--listen-quic-lb-nonce` flags update that table.
`unix.allow` needs `unix.bind`.

`[listen.lb.forward]` keeps a connection alive when its packets land on another relay behind
Expand All @@ -107,8 +106,7 @@ so one file serves the whole cluster:
A packet for an unlisted server ID is dropped. A forward is accepted only from a listed address
and under the shared `key` (`openssl rand -hex 32 > forward.key`). A changed file replaces the
map; an invalid one is logged and the last valid map kept, or stops startup. Forwarding needs
`lb.id` and all three settings, which the `--listen-quic-lb-forward-bind`, `-key`, and `-peers`
flags update.
`lb.id` and all three settings.

## \[quic]

Expand Down Expand Up @@ -278,11 +276,11 @@ connect = ["https://us-east.example.com/?cost=10"] # Peers to dial. ?cost pric
node = "https://us-west.example.com/" # This relay's own URL.
token = "cluster.jwt" # JWT for dials without an inline ?jwt=.
route_key = "route.key" # Route key; relays sharing the file fail over as one to scoped peers.
max_hops = 32 # Longest accepted/forwarded hop path (1 to 32).
max_hops = 32 # Legacy: longest hop path on moq-lite-03 to 06 (1 to 32).

[cluster.api]
url = "https://api.example.com/peers" # Live JSON peer list; a local file also works.
# tls.root = ["api-ca.pem"] # API server trust, independent from connect.tls.
# root = ["api-ca.pem"] # API server trust, independent from connect.tls.

[cluster.lan] # Find peers on the LAN over mDNS.
enabled = true
Expand Down
22 changes: 14 additions & 8 deletions nix/modules/moq-relay.nix
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ let
authUrl = "http://127.0.0.1:${toString cfg.auth.port}/";
in
{
imports = [
(lib.mkRemovedOptionModule [ "services" "moq-relay" "cluster" "rootUrl" ]
"Use services.moq-relay.cluster.connect, a list of full peer URLs such as [ \"https://peer.example.com:4443\" ]."
)
];

options.services.moq-relay = {
enable = lib.mkEnableOption "moq-relay";

Expand Down Expand Up @@ -107,7 +113,7 @@ in
};

cluster = {
rootUrl = lib.mkOption {
connect = lib.mkOption {
type = lib.types.nullOr (lib.types.nonEmptyListOf lib.types.str);
default = null;
example = [
Expand Down Expand Up @@ -141,7 +147,7 @@ in
default = null;
example = "/run/secrets/moq-cluster.pem";
description = ''
Client certificate chain (PEM) this node presents when it dials `rootUrl`, issued
Client certificate chain (PEM) this node presents when it dials `connect`, issued
by the peers' `ca`. Reloaded when it changes. Needs `keyFile`.
'';
};
Expand All @@ -159,7 +165,7 @@ in
type = lib.types.nullOr lib.types.path;
default = null;
description = ''
A JWT this node presents when it dials `rootUrl`, for peers whose auth server
A JWT this node presents when it dials `connect`, for peers whose auth server
verifies tokens instead of certificates. Re-read on every dial. Prefer `certFile`.
'';
};
Expand Down Expand Up @@ -229,12 +235,12 @@ in
warnings =
lib.optional
(
cfg.cluster.rootUrl != null
cfg.cluster.connect != null
&& cfg.cluster.certFile == null
&& cfg.cluster.tokenFile == null
&& lib.any (url: builtins.match ".*[?&]jwt=[^&#].*" url == null) (lib.toList cfg.cluster.rootUrl)
&& lib.any (url: builtins.match ".*[?&]jwt=[^&#].*" url == null) (lib.toList cfg.cluster.connect)
)
"services.moq-relay: this relay dials cluster.rootUrl with no credential, so its peer refuses it or admits it only as an anonymous client; set cluster.certFile and cluster.keyFile (or cluster.tokenFile). The module no longer signs a cluster token.";
"services.moq-relay: this relay dials cluster.connect with no credential, so its peer refuses it or admits it only as an anonymous client; set cluster.certFile and cluster.keyFile (or cluster.tokenFile). The module no longer signs a cluster token.";

# Create user and group
users.users.${cfg.user} = {
Expand Down Expand Up @@ -373,9 +379,9 @@ in
// lib.optionalAttrs (!cfg.auth.enable) {
MOQ_AUTH_PUBLIC = publicPattern;
}
// lib.optionalAttrs (cfg.cluster.rootUrl != null) {
// lib.optionalAttrs (cfg.cluster.connect != null) {
# Cluster configuration
MOQ_CLUSTER_CONNECT = lib.concatStringsSep "," cfg.cluster.rootUrl;
MOQ_CLUSTER_CONNECT = lib.concatStringsSep "," cfg.cluster.connect;
}
// lib.optionalAttrs (cfg.cluster.ca != [ ]) {
MOQ_LISTEN_TLS_ROOT = lib.concatMapStringsSep "," toString cfg.cluster.ca;
Expand Down
1 change: 0 additions & 1 deletion quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ removal, a rename, or a wire, catalog, or format change that is free until
item re-checked for a downstream caller first.
- **Relay and transports.** [CPU](/quest/m2/relay-cpu.md), which the
memory pairing found; [congestion names](/quest/m2/congestion-names.md);
[config names](/quest/m2/relay-config-names.md) and
[the 0.14 leftovers](/quest/m2/relay-014-leftovers.md); and
[transport names](/quest/m2/transport-names.md).
- **Auth and crypto.** [Grants](/quest/m2/auth-grants.md),
Expand Down
6 changes: 4 additions & 2 deletions quest/m2/multicast/switch.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,10 @@ Delete with the old rule:
discount that formed pull cycles here, and lets a link cost 0 (line 88),
which `route::Error::ZeroCost` refuses. The code implements neither, and
item 10 of the epic makes a moq-transport session an endpoint, which has
no path to carry. `cluster.max_hops` then bounds lite-03 to lite-06 only
([config names](/quest/m2/relay-config-names.md)).
no path to carry. `cluster.max_hops` then bounds lite-03 to lite-06 only:
drop the MoQ Cluster extension from its docs (`cluster::Config::max_hops`,
`origin::Config::max_hops`, `doc/bin/relay/cluster.md`, and
`doc/bin/relay/config.md`).
- `rs/moq-net/verify/route/` and the `verify-route` skill, or rewrite the
skill for the exhaustive search.
- [Firefox tabs prove their route key](/quest/m5/routes-browser-proof.md),
Expand Down
4 changes: 0 additions & 4 deletions quest/m2/relay-014-leftovers.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,3 @@ For the PR:
Public API: none (`Legacy` is crate-private). Config and CLI: ten hidden
flags, their variables, and six hidden keys stop parsing; a wildcard-free
public or mTLS pattern is accepted. Wire: none.

## Related

- [The relay's config names](/quest/m2/relay-config-names.md) - the old spellings this cut does refuse by name
95 changes: 0 additions & 95 deletions quest/m2/relay-config-names.md

This file was deleted.

6 changes: 4 additions & 2 deletions quest/m3/release/migrate-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,10 @@ m1 quest on 2026-10-22.
- Known break, a Nix option rather than a Rust API (added 2026-10-31, from
[moq-dev/moq#7343](https://github.com/moq-dev/moq/pull/7343)): the NixOS
module option `services.moq-relay.cluster.rootUrl` takes a non-empty
list instead of a string. Migrate `rootUrl = "https://root:4443";` to
`rootUrl = [ "https://root:4443" ];`; an empty list fails evaluation.
list instead of a string, and is renamed `cluster.connect`
([moq-dev/moq#8924](https://github.com/moq-dev/moq/pull/8924)). Migrate
`rootUrl = "https://root:4443";` to `connect = [ "https://root:4443" ];`;
an empty list fails evaluation, and `rootUrl` fails it naming `connect`.
Write it into `rs/moq-relay/MIGRATE.md`'s transition section (its
`## Unreleased` section until the cut names the version).

Expand Down
6 changes: 3 additions & 3 deletions quest/m5/nix-relay-peers-test.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
## Goal

`nix flake check` evaluates the moq-relay NixOS module with no, one, and
several `cluster.rootUrl` peers and refuses an empty list, asserting the
several `cluster.connect` peers and refuses an empty list, asserting the
`MOQ_CLUSTER_CONNECT` value each produces. Optionally a NixOS VM test runs a
root and two leaves from the module and sees them connect.

Expand All @@ -15,14 +15,14 @@ Decided 2026-10-31 (user, /quest-plan of the follow-ups from the 2026-10-31

Moved to m5 2026-11-12 (maintainer, /quest-audit): no consumer; parked rather than deleted.

[moq-dev/moq#7343](https://github.com/moq-dev/moq/pull/7343) made `services.moq-relay.cluster.rootUrl` a
[moq-dev/moq#7343](https://github.com/moq-dev/moq/pull/7343) made `services.moq-relay.cluster.rootUrl` (now `cluster.connect`) a
`nullOr (nonEmptyListOf str)` (`nix/modules/moq-relay.nix:120` at 4bf501b535), joined into
`MOQ_CLUSTER_CONNECT` at `:365`, and verified it only by a manual `nix eval`.

- Add an eval check beside `nix/runner/config-test.nix`, wired into
`flake.nix`'s `checks` (`:750` at 4bf501b535): `null` sets no `MOQ_CLUSTER_CONNECT`, one
and two URLs join with commas, `[ ]` fails the type check, and a relay with
`rootUrl` but no `certFile` or `tokenFile` raises the missing-credential warning.
`connect` but no `certFile` or `tokenFile` raises the missing-credential warning.
- A VM test (root plus two leaves) only if it stays cheap in CI; otherwise
leave it out and say so.

Expand Down
2 changes: 1 addition & 1 deletion rs/moq-cli/src/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2771,7 +2771,7 @@ mod tests {
&["--listen-version", "moq-lite-02"],
&["--listen-tls-generate", "localhost"],
&["--listen-preferred-v4", "127.0.0.1:443"],
&["--listen-quic-lb-id", "01"],
&["--listen-lb-id", "01"],
&["--listen-tcp-bind", "127.0.0.1:0"],
&["--cluster-node", "https://self.example"],
&["--auth-public", "**"],
Expand Down
10 changes: 6 additions & 4 deletions rs/moq-net/src/model/origin.rs
Original file line number Diff line number Diff line change
Expand Up @@ -113,12 +113,14 @@ pub struct Config {

/// Longest hop path a session using this origin accepts or forwards (default 32).
///
/// Applies to moq-lite-03+ hop paths and the negotiated MoQ Cluster extension on
/// moq-transport. A forwarded path counts this origin's own hop too, including
/// Legacy: applies to moq-lite-03 through 06 and the negotiated MoQ Cluster extension
/// on moq-transport, and leaves with moq-lite-06. A moq-lite-07 route's own hops stop
/// at 32 whatever this is, but a route that crossed into moq-lite-07 from moq-lite-06
/// still carries that chain: with its publisher's hop appended it must fit this
/// bound, or the route is dropped (logged at debug). A forwarded path counts this origin's own hop too, including
/// when the receiving peer reconstructs that hop from the announce handshake.
/// Protocols without a hop path cannot reveal or bound the depth they hide. No wire
/// carries a path longer than 32, so a larger value acts as 32, and a moq-lite-07
/// route's own hops stop there whatever this is.
/// carries a path longer than 32, so a larger value acts as 32.
pub max_hops: NonZeroUsize,

/// The cache pool broadcasts under this origin charge their groups into. It flows
Expand Down
Loading