Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/bin/relay/cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ node = "us-west.local:4443"

[cluster.lan]
enabled = true
# secret = "/etc/moq/cluster.key" # Optional. 64 hex chars, or a file holding them.
# secret = "/etc/moq/cluster.key" # Optional. A file holding 64 hex chars (openssl rand -hex 32).
# app = "default" # DNS-SD subtype; moq-cli shares this name.
```

Expand Down
18 changes: 15 additions & 3 deletions doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ bind = "[::]:443" # QUIC (UDP), as --listen. Omit for a strea
version = ["moq-lite-05"] # Restrict accepted versions. Omit for all.
timeout = "10s" # Handshake deadline. "0" waits forever.

[listen.reset]
secret = "/etc/moq/reset.key" # File holding 64 hex chars. Omit for a random key.

[listen.tls]
cert = "cert.pem" # Certificate chain and key. Reloaded on change.
key = "key.pem"
Expand All @@ -46,9 +49,18 @@ admission, so a peer that connects and never speaks, or whose admission never
gets an answer, is closed instead of held open by keep-alives. An HTTP read's
admission is bounded by the same value from the request's arrival.

`reset.secret` lets a relay restarted on the same address reset its
predecessor's QUIC connections, so clients reconnect at once instead of
waiting out the idle timeout. Keep it secret, and give it to one relay
only: any holder resets every other holder's connections, so live relays behind
one address would reset each other's clients unless the load balancer routes by
connection ID (`lb_id`, QUIC-LB). Relays on different addresses gain nothing from sharing
it, since a client accepts a reset only from its relay's address.
Create one with `openssl rand -hex 32 > reset.key`.

A setting no configured listener reads stops startup rather than being ignored.
A stream-only relay refuses `preferred_v4`, `preferred_v6`, and `lb_id` (or `load_balancer`), which
only QUIC reads, and a `[listen.tls]` `cert`, `key`, or `generate` unless
A stream-only relay refuses `preferred_v4`, `preferred_v6`, `lb_id` (or `load_balancer`), and
`reset.secret`, which only QUIC reads, and a `[listen.tls]` `cert`, `key`, or `generate` unless
`tcp.tls` serves it.
`lb_nonce` needs `lb_id`. `lb_id` and `lb_nonce` cannot be combined with `load_balancer`.
`unix.allow` needs `unix.bind`.
Expand Down Expand Up @@ -267,7 +279,7 @@ id = 12345 # Stable Hop ID across res

[cluster.lan] # Find peers on the LAN over mDNS.
enabled = true
# secret = "/etc/moq/cluster.key" # Optional: 64 hex chars, or a file holding them.
# secret = "/etc/moq/cluster.key" # Optional: a file holding 64 hex chars.
# app = "default" # DNS-SD subtype; moq-cli shares this name.
```

Expand Down
1 change: 0 additions & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,6 @@ lands before the cut.
- [Audio jitter target](/quest/m1/audio-jitter-target/README.md) - the playout target is default in both languages; a manual browser proof and a native trace replay remain
- [A/V clock](/quest/m1/av-clock.md) - the audio playhead drives Sync.reference while audio plays, through per-track sync handles
- [Publish channel count](/quest/m1/publish-audio-channel-count.md) - forcing a channel count on an Audio.Capture stops costing the subscriber gaps of silence, and the media fixture's cold-start audio-tone flake is fixed at its cause
- [Restart reset](/quest/m1/stateless-reset-key.md) - a relay restarted with the same `listen.reset_key` stateless-resets its predecessor's connections, so clients reconnect within an RTT instead of the idle timeout
- [Drain cleanup](/quest/m1/drain-cleanup.md) - sessions force-closed at the drain deadline record their lease end and bytes before the relay or CLI exits, while unfinished handshakes stay bounded
- [Rust redirect loops](/quest/m1/tokio-redirect-loop.md) - moq-tokio escalates its backoff when GOAWAY redirects bounce between peers, as JS does
- [JS request owner](/quest/m1/js-request-owner.md) - every JS IETF request stream has one owning handle with an `ended` promise and one closer
Expand Down
35 changes: 0 additions & 35 deletions quest/m1/stateless-reset-key.md

This file was deleted.

1 change: 0 additions & 1 deletion quest/m2/quic-outage-probe.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,3 @@ Public API: none.

- [Keep-alive from the idle timeout](/quest/m2/quic-keep-alive.md) - the other outage-time knob
- [Per-stream deadlines](/quest/m2/quic-deadline.md) - an early probe ahead of a deadline
- [Restart reset](/quest/m1/stateless-reset-key.md) - the restart half of #5953's item 4
5 changes: 5 additions & 0 deletions rs/moq-cli/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@ How to update your invocation and config for each breaking change, newest first.

## Unreleased

### `--cluster-lan-secret` is a file path only (#5910)

`--cluster-lan-secret`, `MOQ_CLUSTER_LAN_SECRET`, and `[cluster.lan] secret` no longer accept 64 hex characters inline.
Write the hex to a file (`openssl rand -hex 32 > cluster.key`) and pass its path.

### `--listen-timeout` bounds admission (#5765)

`moq --listen` now finishes auth admission within the `--listen-timeout` handshake deadline (default `10s`, counted from the first packet) instead of a separate fixed ten seconds, so a short timeout now refuses a slow auth server sooner and `0s` waits on it forever.
Expand Down
5 changes: 4 additions & 1 deletion rs/moq-cli/src/args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2542,7 +2542,10 @@ mod tests {
])
.expect("parse");
assert!(cli.moq.validate().is_ok());
assert_eq!(cli.moq.cluster.lan.secret.as_deref(), Some("cluster.key"));
assert_eq!(
cli.moq.cluster.lan.secret.as_deref(),
Some(std::path::Path::new("cluster.key"))
);
}

/// The app is only read by the mesh, so configuring one without it is an
Expand Down
1 change: 1 addition & 0 deletions rs/moq-quic/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ Changes on top of the upstream commit, besides the renames:
- **Late stream limits**: peer transport parameters that raise a stream limit after an open was refused emit `StreamEvent::Available`, as `MAX_STREAMS` does. A server accepted before the rest of a multi-packet ClientHello (a post-quantum key share is enough) otherwise leaves a 0.5-RTT `open_uni` parked until the idle timeout. Not offered upstream yet.
- **Stream credit on stop**: `RecvStream::stop` on a stream whose final size is already known (finished or reset) frees it and queues `MAX_STREAMS` at once, as reading the reset does, instead of leaving the credit until the next packet arrives. Not offered upstream yet; quinn `main` has the same gap.
- **Drop resets** (`quic::driver::SendStream`): a send stream dropped before `finish` is reset with code 0, or with the peer's `STOP_SENDING` code, instead of quinn's implicit `finish`, so a cancelled writer never ends a truncated stream with a FIN. This matches moq-uring's driver and moq-net's transport contract. `quic::web` resets after the WebTransport header with the mapped code. quinn's `stream_stopped_2` finishes its stream before dropping it. Not offered upstream: it reverses a documented quinn behaviour.
- **Reset secret** (`EndpointConfig::reset_secret`): derives the stateless reset key from an operator's secret with HKDF-SHA256, and keys a `HashedConnectionIdGenerator` from it too, since a restarted endpoint drops packets for connection IDs it can't validate before it would reset them. It also lowers `min_reset_interval` to 1ms, so a restart resets thousands of connections in seconds. Both drivers take one, so a restarted server resets its predecessor's connections. Not offered upstream: it only combines quinn's `reset_key` and `HashedConnectionIdGenerator::from_key`.
- **Connection limits** (`Limits`, `Connection::set_limits`, and the drivers' `set_limits`): one validated call changes a live connection's stream limit and windows, refusing values a peer would reject and changing nothing then. moq-tokio and moq-uring re-export the type, so both runtimes check limits the same way. Not offered upstream: it only bundles quinn's existing setters.

Recheck the BBR3 changes when cherry-picking quinn#2481 updates, or if it merges.
Expand Down
51 changes: 49 additions & 2 deletions rs/moq-quic/src/config/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,53 @@ impl EndpointConfig {
self
}

/// Derive the reset key from a random 32-byte `secret` with HKDF-SHA256, so an endpoint
/// restarted with the same secret resets its predecessor's connections
///
/// Every endpoint holding the secret resets every other holder's connections, so RFC 9000
/// section 21.11 requires that packets for a live connection only ever reach its owner: share
/// it across live endpoints only behind one address that routes by connection ID.
///
/// Also keys a [`HashedConnectionIdGenerator`] from it, since an endpoint drops packets for
/// connection IDs it can't validate instead of resetting them. Set
/// [`cid_generator`](Self::cid_generator) afterwards to use another generator; it must accept
/// the predecessor's IDs for the reset to reach them.
///
/// Lowers [`min_reset_interval`](Self::min_reset_interval) to 1ms too, since a restart leaves
/// every peer waiting on a reset at once: 1000 a second resets 2,000 connections in 2s, where
/// 20ms takes 40s. A reset is always smaller than the packet that drew it, so this never
/// amplifies, and the limit still caps what a flood of spoofed packets can reflect.
#[cfg(any(feature = "aws-lc-rs", feature = "ring"))]
pub fn reset_secret(&mut self, secret: &[u8; 32]) -> &mut Self {
#[cfg(all(feature = "aws-lc-rs", not(feature = "ring")))]
use aws_lc_rs::{hkdf, hmac};
#[cfg(feature = "ring")]
use ring::{hkdf, hmac};

struct Len(usize);
impl hkdf::KeyType for Len {
fn len(&self) -> usize {
self.0
}
}

let prk = hkdf::Salt::new(hkdf::HKDF_SHA256, &[]).extract(secret);
let reset_key = prk
.expand(&[b"moq-quic reset key"], hmac::HMAC_SHA256)
.expect("an HMAC key fits HKDF-SHA256");
self.reset_key = Arc::new(hmac::Key::from(reset_key));

let mut cid_key = [0; 8];
prk.expand(&[b"moq-quic cid key"], Len(cid_key.len()))
.and_then(|okm| okm.fill(&mut cid_key))
.expect("8 bytes fit HKDF-SHA256");
let cid_key = u64::from_le_bytes(cid_key);
self.connection_id_generator_factory =
Arc::new(move || Box::new(HashedConnectionIdGenerator::from_key(cid_key)));
self.min_reset_interval = Duration::from_millis(1);
self
}

/// Maximum UDP payload size accepted from peers (excluding UDP and IP overhead).
///
/// Must be greater or equal than 1200.
Expand Down Expand Up @@ -136,8 +183,8 @@ impl EndpointConfig {

/// Minimum interval between outgoing stateless reset packets
///
/// Defaults to 20ms. Limits the impact of attacks which flood an endpoint with garbage packets,
/// e.g. [ISAKMP/IKE amplification]. Larger values provide a stronger defense, but may delay
/// Defaults to 20ms, or 1ms after [`reset_secret`](Self::reset_secret). Limits the impact of
/// attacks which flood an endpoint with garbage packets, e.g. [ISAKMP/IKE amplification]. Larger values provide a stronger defense, but may delay
/// detection of some error conditions by clients. Using a [`ConnectionIdGenerator`] with a low
/// rate of false positives in [`validate`](ConnectionIdGenerator::validate) reduces the risk
/// incurred by a small minimum reset interval.
Expand Down
71 changes: 71 additions & 0 deletions rs/moq-quic/src/tests/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -577,6 +577,77 @@ fn client_stateless_reset() {
assert_matches!(conn.poll(), None);
}

/// Restart the server from a fresh config built from `restart`, and report whether the client's
/// next packet for the old connection drew a stateless reset
fn restarted_server_resets(restart: &[u8; 32]) -> bool {
let _guard = subscribe();
let config = |secret: &[u8; 32]| {
let mut config = EndpointConfig::default();
config.reset_secret(secret);
Arc::new(config)
};

let mut pair = Pair::new(config(&[1; 32]), server_config());
let (client_ch, _) = pair.connect();
pair.drive(); // Flush any post-handshake frames
pair.server.endpoint = Endpoint::new(config(restart), Some(Arc::new(server_config())), true);
pair.client.connections.get_mut(&client_ch).unwrap().ping();
pair.drive();
matches!(
pair.client_conn_mut(client_ch).poll(),
Some(Event::ConnectionLost {
reason: ConnectionError::Reset
})
)
}

#[test]
fn reset_secret_survives_a_restart() {
assert!(restarted_server_resets(&[1; 32]));
}

#[test]
fn reset_secret_must_match() {
assert!(!restarted_server_resets(&[2; 32]));
}

/// After a restart every peer's next packet draws a reset at once, so a reset secret answers
/// 1000 connections a second, still dropping a packet that arrives within 1ms of the last reset
#[test]
fn reset_secret_resets_connections_each_millisecond() {
let _guard = subscribe();
let mut config = EndpointConfig::default();
config.reset_secret(&[1; 32]);
let mut cids = (config.connection_id_generator_factory)();
let mut endpoint = Endpoint::new(Arc::new(config), Some(Arc::new(server_config())), true);

let interval = Duration::from_millis(1);
let start = Instant::now();
let mut buf = Vec::new();
for i in 0..2_000u16 {
let remote = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 1024 + i);
let mut packet = vec![0x40];
packet.extend_from_slice(&cids.generate_cid());
packet.resize(100, 0);

let now = start + interval * u32::from(i);
let event = endpoint.handle(now, remote, None, None, packet[..].into(), &mut buf);
assert!(
matches!(event, Some(DatagramEvent::Response(_))),
"connection {i} was not reset"
);
let event = endpoint.handle(
now + interval / 2,
remote,
None,
None,
packet[..].into(),
&mut buf,
);
assert!(event.is_none(), "connection {i} was reset twice within 1ms");
}
}

/// The client reads CONNECTION_CLOSE after the server forgets the connection.
/// The client's reply triggers a reset before the application polls.
#[test]
Expand Down
5 changes: 5 additions & 0 deletions rs/moq-relay/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ Rolling a cluster across a release also follows the [upgrade guide](https://moq.

## Unreleased

### `cluster.lan.secret` is a file path only (#5910)

`--cluster-lan-secret`, `MOQ_CLUSTER_LAN_SECRET`, and `[cluster.lan] secret` no longer accept 64 hex characters inline.
Write the hex to a file (`openssl rand -hex 32 > cluster.key`) and pass its path; `LanConfig::secret` is now a `PathBuf`.

### Admission runs under the handshake deadline (#5765)

`listen.timeout` now bounds auth admission as part of the handshake, replacing the fixed ten-second admission timeout, so `"0"` also lets admission wait forever.
Expand Down
22 changes: 14 additions & 8 deletions rs/moq-relay/src/cluster.rs
Original file line number Diff line number Diff line change
Expand Up @@ -876,8 +876,8 @@ pub struct LanConfig {
)]
pub enabled: bool,

/// The shared key admitting a peer to the LAN mesh, as 64 hexadecimal
/// characters or a path to a file containing them.
/// A file holding the shared key admitting a peer to the LAN mesh, as 64
/// hexadecimal characters (`openssl rand -hex 32`).
///
/// Optional. Without it, anyone who can reach the listener joins, so leave
/// it unset only on networks you trust. With it, only peers that prove they
Expand All @@ -886,10 +886,10 @@ pub struct LanConfig {
name = "cluster-lan-secret",
long = "cluster-lan-secret",
env = "MOQ_CLUSTER_LAN_SECRET",
value_name = "HEX_OR_PATH",
setting = "cluster.lan.secret"
)]
pub secret: Option<String>,
#[usage(value_hint = usage_rs::ValueHint::FilePath)]
pub secret: Option<std::path::PathBuf>,

/// DNS-SD application this relay advertises under. Peers using a different
/// name never discover this one. Defaults to `default`, which moq-cli
Expand Down Expand Up @@ -2129,7 +2129,7 @@ impl Cluster {
async fn lan_discovery(
advertise: &LanAdvertise,
node: Option<&str>,
secret: Option<&str>,
secret: Option<&std::path::Path>,
app: moq_tokio::mdns::App,
) -> anyhow::Result<moq_tokio::mdns::Discovery> {
let mut config = moq_tokio::mdns::Config::new(app, advertise.port);
Expand Down Expand Up @@ -2160,7 +2160,7 @@ async fn lan_discovery(
config = config.with_node(url);
}
if let Some(secret) = secret {
let secret = moq_tokio::mdns::Secret::load(secret).context("invalid --cluster-lan-secret")?;
let secret = moq_tokio::secret::Secret::read(secret).context("invalid --cluster-lan-secret")?;
config = config.with_secret(secret);
}
Ok(config.advertise().await?)
Expand Down Expand Up @@ -3267,7 +3267,10 @@ mod tests {
let args = vec![std::ffi::OsString::from("moq-relay"), std::ffi::OsString::from(&path)];
let config = RelayConfig::parse_and_merge(args).expect("config load");
assert!(config.cluster.lan.enabled);
assert_eq!(config.cluster.lan.secret.as_deref(), Some("cluster.key"));
assert_eq!(
config.cluster.lan.secret.as_deref(),
Some(std::path::Path::new("cluster.key"))
);
assert_eq!(
config.cluster.lan.app.as_ref().map(ToString::to_string).as_deref(),
Some("custom")
Expand Down Expand Up @@ -3298,7 +3301,10 @@ mod tests {
std::ffi::OsString::from("from-cli"),
];
let config = RelayConfig::parse_and_merge(args).expect("config load");
assert_eq!(config.cluster.lan.secret.as_deref(), Some("from-cli.key"));
assert_eq!(
config.cluster.lan.secret.as_deref(),
Some(std::path::Path::new("from-cli.key"))
);
assert_eq!(
config.cluster.lan.app.as_ref().map(ToString::to_string).as_deref(),
Some("from-cli")
Expand Down
Loading