Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,8 +174,8 @@ first cores, so two relays on one host, or one limited to fewer CPUs than the
host has, would crowd onto them. Turn it on for a host the relay has to itself.

`io_uring` additionally needs Linux 6.12+ and the `io-uring` cargo feature; it
shares the same reloadable TLS identity, serves every configured version
like the shared runtime, and refuses `mtu_discovery`. A setting the build or host cannot deliver refuses to
shares the same reloadable TLS identity and serves every configured version
like the shared runtime. A setting the build or host cannot deliver refuses to
start rather than being ignored. If io\_uring workers fail to start naming
`RLIMIT_MEMLOCK`, raise it (`LimitMEMLOCK=` under systemd).

Expand Down
1 change: 0 additions & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,6 @@ tracks, so per-group stats stop costing cluster announces.
- [No reused track names](/quest/m1/name-reuse.md) - no first-party publisher re-creates a finished track under its old name
- [Routing guide](/quest/m1/routing-guide.md) - concepts and practical outbound federation setup
- [Controlled time](/quest/m1/time/README.md) - every crate and package reads time through moq-clock or @moq/clock, with regression tests for the changed contracts
- [io_uring MTU matches tokio](/quest/m1/uring-mtu.md) - a 1200-byte floor and `quic.mtu_discovery` honored, so the ring stops black-holing smaller paths
- [io_uring listen extensions](/quest/m1/uring-listen-extensions.md) - ring listeners offer the configured `[listen] extensions`
- [Rust request credit](/quest/m1/rs-request-credit.md) - a moq-net request fails at once without stream credit instead of waiting
- [Generated C bindings](/quest/m1/c/README.md) - C generated from moq-ffi ships as `moq-c` 0.8.0 and replaces the hand-written libmoq
Expand Down
46 changes: 0 additions & 46 deletions quest/m1/uring-mtu.md

This file was deleted.

4 changes: 2 additions & 2 deletions quest/m2/perf/uring-profile.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ profile, and plan a quest only for one that shows up:
- atomic RMWs in worker metrics (remote wake counters genuinely have
several writers);
- receive rearm churn that `IORING_RECVSEND_POLL_FIRST` would cut;
- GSO batching on ordinary and jumbo paths after
[MTU parity](/quest/m1/uring-mtu.md).
- GSO batching on ordinary and jumbo paths, now that discovery can raise
the MTU.

Decided 2026-10-22 from the io_uring audit.

Expand Down
1 change: 0 additions & 1 deletion quest/m2/relay-io-uring-package.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ Public API: none. Wire: none.

## Required

- [io_uring MTU matches tokio](/quest/m1/uring-mtu.md) - `quic.mtu_discovery` honored and smaller paths reachable
- [io_uring handle lifecycle](/quest/m2/uring-lifecycle.md) - handles fail with their worker and close on last drop
- [Ring drain GOAWAY](/quest/m2/uring-ring-drain.md) - closes are delivered on the ring for moq-transport too
- [One port on the io_uring workers](/quest/m2/uring-demux.md) - a one-port config is served on the ring too
16 changes: 10 additions & 6 deletions rs/moq-relay/src/uring.rs
Original file line number Diff line number Diff line change
Expand Up @@ -414,12 +414,6 @@ impl std::fmt::Debug for Workers {
/// One qlog sink is built for the whole group and cloned into every worker, so
/// the traces share a single writer thread however many workers there are.
fn transport(quic: &moq_tokio::quic::Resolved) -> anyhow::Result<moq_uring::quic::Transport> {
// The datagram path fixes both payload ceilings at SEGMENT and hands
// `conn.send` slices of exactly that, so there is no larger size to find.
anyhow::ensure!(
!quic.mtu_discovery,
"io_uring workers send a fixed-size UDP payload, so quic.mtu_discovery has nothing to discover"
);
// The tokio path reports this through `moq_tokio::Error::QlogUnsupported`,
// which nothing builds here: this listener is the io_uring stack.
#[cfg(not(feature = "qlog"))]
Expand Down Expand Up @@ -449,6 +443,7 @@ fn transport(quic: &moq_tokio::quic::Resolved) -> anyhow::Result<moq_uring::quic
transport.idle_timeout = quic.idle_timeout;
transport.max_streams = quic.max_streams;
transport.keep_alive = quic.keep_alive;
transport.mtu_discovery = quic.mtu_discovery;
// `[quic]` resolves every window, defaults included, so both runtimes run
// the same ones rather than each backend's own.
transport.receive_window = Some(quic.receive_window);
Expand Down Expand Up @@ -906,6 +901,15 @@ mod tests {
assert_eq!(transport.send_window, Some(32 << 20));
}

/// `quic.mtu_discovery` reaches the workers as it does the tokio backends: off unless set.
#[test]
fn mtu_discovery_is_applied() {
let mut quic = moq_tokio::quic::Config::default();
assert!(!transport(&quic.resolve()).expect("default").mtu_discovery);
quic.mtu_discovery = Some(true);
assert!(transport(&quic.resolve()).expect("discovery").mtu_discovery);
}

/// Incomplete messages hold connection credit, so the window has to fit the largest.
#[test]
fn a_receive_window_below_the_largest_message_is_refused() {
Expand Down
21 changes: 12 additions & 9 deletions rs/moq-uring/src/quic/driver/connection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ use bytes::Bytes;
use moq_quic::{ConnectionHandle, Dir, StreamId, VarInt};
use rustc_hash::FxHashMap;

use super::super::{Error, SEGMENT};
use super::super::Error;
use super::endpoint;
use crate::udp;
use crate::worker::Owner;
Expand Down Expand Up @@ -881,16 +881,19 @@ impl Driver {
Poll::Pending => return Poll::Pending,
};

let segments = (tx.capacity() / SEGMENT).min(TRAIN_SEGMENTS);
if segments == 0 {
return Poll::Ready(Err(Error::Io(format!(
"transmit buffer of {} bytes holds no {SEGMENT} byte segment",
tx.capacity()
))));
}

let (transmit, closing) = {
let mut conn = self.shared.conn.borrow_mut();
// moq-quic packs every full datagram at the current MTU, which
// discovery can raise, so a train sized for any smaller one would
// outgrow the buffer.
let mtu = usize::from(conn.current_mtu());
let segments = (tx.capacity() / mtu).min(TRAIN_SEGMENTS);
if segments == 0 {
return Poll::Ready(Err(Error::Io(format!(
"transmit buffer of {} bytes holds no {mtu} byte datagram",
tx.capacity()
))));
}
// moq-quic encrypts the train straight into the socket's buffer.
// Nothing to send, or paced; the buffer returns to the pool on drop.
let Some(transmit) = conn.poll_transmit(self.shared.owner.now(), segments, &mut tx) else {
Expand Down
9 changes: 5 additions & 4 deletions rs/moq-uring/src/quic/driver/endpoint.rs
Original file line number Diff line number Diff line change
Expand Up @@ -102,12 +102,13 @@ impl Endpoint {
None => None,
};
let accepting = server.is_some().then(|| Accepting { queue: VecDeque::new() });
// MTU discovery is off (the GSO pool sends fixed SEGMENT datagrams),
// so the endpoint has no reason to allow it either.
// Every `udp::Socket` refuses to fragment, so a probe too large for the
// path is lost rather than delivered in pieces, and discovery can trust
// what arrives. Each connection's transport still decides whether to probe.
let endpoint = moq_quic::Endpoint::new(
super::endpoint_config(socket.shard(), config.reset_secret.as_ref())?,
super::endpoint_config(socket.shard(), config.reset_secret.as_ref()),
server,
false,
true,
);

let inner = Rc::new(Inner {
Expand Down
25 changes: 10 additions & 15 deletions rs/moq-uring/src/quic/driver/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ use rustls::pki_types::{CertificateDer, PrivateKeyDer};

#[cfg(feature = "qlog")]
use super::qlog;
use super::{Congestion, Error, Identity, SEGMENT, Transport, client, server};
use super::{Congestion, Error, Identity, Transport, client, server};
use crate::udp;

/// Per-stream flow control credit, which returns as bytes are read, so it only paces a stream.
Expand All @@ -41,8 +41,8 @@ const CONNECTION_WINDOW: u32 = 128 * 1024 * 1024;
const _: () = assert!(CONNECTION_WINDOW as u64 >= moq_net::transport::MIN_RECEIVE_WINDOW);
/// Unacknowledged outgoing data to buffer, whatever the peer allows.
const SEND_WINDOW: u32 = 16 * 1024 * 1024;
/// How many datagrams to buffer in each direction.
const DATAGRAM_WINDOW: usize = 64 * SEGMENT;
/// Datagram bytes to buffer in each direction, about 64 full-size datagrams.
const DATAGRAM_WINDOW: usize = 96 * 1024;

fn ecn_to_quic(ecn: udp::Ecn) -> moq_quic::EcnCodepoint {
match ecn {
Expand Down Expand Up @@ -96,12 +96,11 @@ fn provider() -> Arc<rustls::crypto::CryptoProvider> {
.clone()
}

/// The endpoint-wide configuration: how connection ids are minted, the reset
/// key, and the largest datagram we tell peers we can receive.
/// The endpoint-wide configuration: how connection ids are minted and the reset key.
pub(crate) fn endpoint_config(
shard: Option<moq_sock::shard::Shard>,
reset_secret: Option<&[u8; 32]>,
) -> Result<Arc<moq_quic::EndpointConfig>, Error> {
) -> Arc<moq_quic::EndpointConfig> {
let mut config = moq_quic::EndpointConfig::default();
// The random ids below validate a predecessor's too, so the reset reaches them.
if let Some(secret) = reset_secret {
Expand All @@ -119,10 +118,7 @@ pub(crate) fn endpoint_config(
}
Box::new(cids)
}));
config
.max_udp_payload_size(SEGMENT as u16)
.map_err(|err| Error::Quic(err.to_string()))?;
Ok(Arc::new(config))
Arc::new(config)
}

/// Dial as `config` says.
Expand Down Expand Up @@ -282,11 +278,10 @@ fn transport_config(config: &Transport) -> Result<moq_quic::TransportConfig, Err
transport.send_window(send_window);
transport.datagram_receive_buffer_size(Some(DATAGRAM_WINDOW));
transport.datagram_send_buffer_size(DATAGRAM_WINDOW);
// Every datagram in a GSO train is one SEGMENT, so the packet size is not
// moq-quic's to discover: pin it and turn the probing off.
transport.initial_mtu(SEGMENT as u16);
transport.min_mtu(SEGMENT as u16);
transport.mtu_discovery_config(None);
// moq-quic probes by default; the 1200-byte floor reaches every path.
if !config.mtu_discovery {
transport.mtu_discovery_config(None);
}
Comment on lines +281 to +284

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bound MTU discovery by the socket's transmit-buffer ceiling

When a direct moq-uring consumer enables discovery with a valid smaller udp::Config::tx_buffer_len, such as 1350 bytes, the handshake succeeds but discovery eventually terminates the connection. The default discovery upper bound is 1452 bytes, and moq_quic::Connection::poll_transmit emits individual probes larger than current_mtu() regardless of the calculated segment count. udp::TxBuf::send rejects these probes because their length exceeds the configured buffer ceiling, and flush_one treats that rejection as a terminal I/O error. Bound discovery's upper limit by the socket's transmit-buffer size so enabling discovery does not break connections using smaller buffers.

Useful? React with 👍 / 👎.

transport.congestion_controller_factory(match config.congestion {
Congestion::Loss => Arc::new(moq_quic::congestion::CubicConfig::default())
as Arc<dyn moq_quic::congestion::ControllerFactory + Send + Sync>,
Expand Down
9 changes: 5 additions & 4 deletions rs/moq-uring/src/quic/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,6 @@ pub use endpoint::Endpoint;
/// The flow-control limits a live connection can change; see [`Connection::set_limits`].
pub use moq_quic::{Limits, LimitsError};

/// The QUIC payload size every full datagram in a GSO train uses, and the
/// stride GRO coalesces with.
pub(crate) const SEGMENT: usize = 1350;

/// A TLS certificate chain and the private key that signs for it, as PEM.
/// One value, so neither half can be configured alone.
///
Expand Down Expand Up @@ -138,6 +134,10 @@ pub struct Transport {
/// connection, or `None` (the default) to send none and let the idle
/// timeout decide.
pub keep_alive: Option<std::time::Duration>,
/// Probe for a path MTU above the 1200-byte floor every QUIC path carries,
/// off by default. A probe the path drops costs only itself, and a path
/// that later shrinks falls back to the floor.
pub mtu_discovery: bool,
/// Where to write qlog traces, or `None` (the default) to write none.
///
/// moq-quic takes one sink per transport config, so this writes one file
Expand All @@ -160,6 +160,7 @@ impl Default for Transport {
send_window: None,
congestion: Congestion::default(),
keep_alive: None,
mtu_discovery: false,
#[cfg(feature = "qlog")]
qlog: None,
}
Expand Down
1 change: 1 addition & 0 deletions rs/moq-uring/tests/integration/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ mod ecn;
mod endpoint;
mod identity;
mod local_addr;
mod mtu;
mod qlog;
mod receive_credit;
mod session;
Expand Down
Loading
Loading