From 6915606fd9438bf774d3b4d8d13f9686fa0d63f0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 13:30:24 -0800 Subject: [PATCH 1/4] chore(quest): claim quest/m1/session-policy/uring.md Quest-Claim: 86ca4e5f9743cff5 From 10c2c6098925164b171fcd7167eaad3eb5d5e7ec Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 14:32:27 -0800 Subject: [PATCH 2/4] refactor(relay): io_uring sessions run through Connection and moq-net's accept policy The io_uring workers now build a uring `Request` that implements the sealed `incoming::Sealed`, so `Connection::run` admits, answers and supervises every runtime's sessions. Admission, the drain-bounded SETUP reply and supervision hop to the shared tokio runtime through the new `Sealed::shared` hook; the driver and bitrate pacing stay on the worker. The hand-copied `Deadline`, `serve_request`, and the `set_limits`/window plumbing are gone: moq-net's accept deadline and `with_receive_window` do that work. `moq_tokio::listen::Config::server` is the one mapping of `listen` onto the moq-net server, used by moq-tokio's listeners, the io_uring workers, and the relay's WebSocket upgrades. Co-Authored-By: Claude Opus 5.5 --- quest/m1/session-policy/README.md | 17 +- quest/m1/session-policy/uring.md | 37 -- quest/m2/cut-gate/cut-audit/README.md | 6 +- quest/m3/relay-io-uring-package.md | 8 +- rs/moq-relay/src/connection.rs | 113 +++--- rs/moq-relay/src/incoming.rs | 44 ++- rs/moq-relay/src/relay.rs | 9 +- rs/moq-relay/src/uring.rs | 494 ++++++++++++-------------- rs/moq-relay/src/web.rs | 42 +-- rs/moq-relay/src/websocket.rs | 11 +- rs/moq-tokio/src/server.rs | 23 +- 11 files changed, 387 insertions(+), 417 deletions(-) delete mode 100644 quest/m1/session-policy/uring.md diff --git a/quest/m1/session-policy/README.md b/quest/m1/session-policy/README.md index e06bf901738..0d722d09c30 100644 --- a/quest/m1/session-policy/README.md +++ b/quest/m1/session-policy/README.md @@ -7,9 +7,7 @@ SETUP, admission and the reply, closing with Timeout), the admission bound, widening the receive window on admit, and reconnect with backoff come from moq-net for tokio, io_uring and WebRTC sessions alike. moq-tokio keeps only sockets, DNS, TLS, accept loops, the fallback runtime and spawning. The epic is -done when moq-relay's hand-copied policy in `uring.rs` is deleted in favour of -the moq-net calls, and the WebRTC transport's session runs through the same -policy. +done when the WebRTC transport's session runs through the same policy. ## Plan @@ -28,8 +26,8 @@ in moq-net, and decided: - ✅ **Reconnect is in scope**: the reconnect, backoff and GOAWAY candidate walk move too, as a poll machine. Not chosen: leaving them in moq-tokio until io_uring dials. -- ✅ **Fold** the m2 relay-serve-path quest into - [the io_uring path](/quest/m1/session-policy/uring.md) and the +- ✅ **Fold** the m2 relay-serve-path quest into the io_uring path, which landed + in [moq-dev/moq#8082](https://github.com/moq-dev/moq/pull/8082), and the poll-first server-machines quest into accept policy, which landed in [moq-dev/moq#8003](https://github.com/moq-dev/moq/pull/8003). - ✅ **Break moq-net's public API in m1**, before the cut, with MIGRATE notes. @@ -47,13 +45,7 @@ Research (2026-11-04, `origin/main` at e6be13d5f8): `Goaway`, `run`). All of it runs on `kio::clock`, so none of it needs tokio. Only sockets, DNS, Happy Eyeballs, TLS, accept loops, `runtime::scope`, spawning and WebRTC's own UDP socket are tokio-specific. -- moq-relay's io_uring path re-implements the server side by hand - (`rs/moq-relay/src/uring.rs`: its own `Deadline`, the admission bound, - the drain-bounded reply, and the `with_deadline` and `set_limits` calls - that wire them up). Eight commits - patched drift between the copies (58eaa6afd5, 8a13d58910, 6bd1306d6e, - 1f56330d21, d17b84ee7f, 4576bb1231, c11192c7e6, 67df70012d). moq-uring - itself has no client side. +- moq-uring has no client side. - Widening on admit needs a transport hook: moq-net's `transport::poll` session exposes only `receive_window()`, while tokio QUIC, qmux and uring's `web::Session` all have `set_limits`. Add a trait method whose default @@ -65,5 +57,4 @@ Research (2026-11-04, `origin/main` at e6be13d5f8): ## Required -- [The io_uring relay path uses moq-net's policy](/quest/m1/session-policy/uring.md) - deletes `uring.rs`'s copy - [moq-tokio becomes a thin shell](/quest/m1/session-policy/tokio-handoff.md) - sockets, TLS, accept loops and spawning, plus the transport handoffs WebRTC needs diff --git a/quest/m1/session-policy/uring.md b/quest/m1/session-policy/uring.md deleted file mode 100644 index 8fae589be23..00000000000 --- a/quest/m1/session-policy/uring.md +++ /dev/null @@ -1,37 +0,0 @@ -# [M] The io_uring relay path uses moq-net's policy - -## Goal - -moq-relay serves io_uring sessions through the same moq-net accept policy and -the same `Connection::run` and listen-config mapping as tokio, so a -lifecycle or config fix lands once. The hand-copied deadline, admission bound -and drain-bounded reply in `rs/moq-relay/src/uring.rs`, and the `with_deadline` -and `set_limits` calls that wire them up, are deleted. - -## Plan - -Part of [Session policy lives in moq-net](/quest/m1/session-policy/README.md); -see it for the decisions and research. - -- Folded in 2026-11-04 (maintainer): the m2 relay-serve-path quest - ([last version](https://github.com/moq-dev/moq/blob/e6be13d5f8/quest/m2/relay-serve-path.md)). - Its finding stands: `uring::serve_request` re-implements moq-relay's - `Connection::run` (`rs/moq-relay/src/connection.rs`; the old quest called it - `Connection::serve`) step for step, `Workers::bind` re-implements the config - mapping also at moq-tokio's `server.rs` and `websocket.rs`, and eight commits - patched the drift (58eaa6afd5, 8a13d58910, 6bd1306d6e, 1f56330d21, - d17b84ee7f, 4576bb1231, c11192c7e6, 67df70012d). -- Implement the sealed `incoming::Sealed` for a uring `Request` whose driver - runs on the worker handle, with admission hopping to the shared runtime - through `Lifecycle::shared`, so one `Connection::run` serves every runtime. - Use one listen-config mapping at all three sites. -- Do this before [the io_uring packaging work](/quest/m3/relay-io-uring-package.md) - publishes the duplicated path. -- The relay's existing drain, admission and handshake-deadline tests must - pass on both runtimes. - -Public API: none. Wire: none. - -## Related - -- [io_uring relay package](/quest/m3/relay-io-uring-package.md) - ships the path this unifies diff --git a/quest/m2/cut-gate/cut-audit/README.md b/quest/m2/cut-gate/cut-audit/README.md index 304e8fd5ad9..cd386f6ed0c 100644 --- a/quest/m2/cut-gate/cut-audit/README.md +++ b/quest/m2/cut-gate/cut-audit/README.md @@ -42,9 +42,9 @@ internal and can ship after the cut: [moq-transport request leases](/quest/m3/ietf-request-lease.md), [group heads filled in place](/quest/m3/ietf-fill-refill.md), moqsrc pad actors and publish rendition identity (both since landed). The -[one relay serve path](/quest/m1/session-policy/uring.md) proposal moved into -m1 on 2026-11-04, as part of moving session policy into moq-net before the -cut. Other redesign evidence went into the quests that already own the +one relay serve path proposal moved into m1 on 2026-11-04, as part of moving +session policy into moq-net before the cut, and landed in +[moq-dev/moq#8082](https://github.com/moq-dev/moq/pull/8082). Other redesign evidence went into the quests that already own the mechanism: [subscribe ranges](/quest/m2/subscribe-ranges/README.md), [drop set](/quest/m1/drop-set.md), diff --git a/quest/m3/relay-io-uring-package.md b/quest/m3/relay-io-uring-package.md index 7ed518050cb..69ec9821dae 100644 --- a/quest/m3/relay-io-uring-package.md +++ b/quest/m3/relay-io-uring-package.md @@ -38,13 +38,13 @@ ships, not the default. (`listen.lb_id` QUIC-LB server ids). That refusal is a recorded exception to parity, not a prerequisite, until a consumer asks. -Decided 2026-11-05 (maintainer, /quest-audit): ✅ this waits for -[the io_uring relay path using moq-net's policy](/quest/m1/session-policy/uring.md), -so packaging never publishes the hand-copied lifecycle path. +Decided 2026-11-05 (maintainer, /quest-audit): ✅ this waited for the io_uring +relay path to serve through moq-net's policy, so packaging never publishes the +hand-copied lifecycle path; that landed in +[moq-dev/moq#8082](https://github.com/moq-dev/moq/pull/8082). Public API: none. Wire: none. ## Required - [One port on the io_uring workers](/quest/m3/uring-demux.md) - a one-port config is served on the ring too -- [The io_uring relay path uses moq-net's policy](/quest/m1/session-policy/uring.md) - the ring serves through the shared accept policy before it ships diff --git a/rs/moq-relay/src/connection.rs b/rs/moq-relay/src/connection.rs index 4c824abed5e..4f0e1947c7c 100644 --- a/rs/moq-relay/src/connection.rs +++ b/rs/moq-relay/src/connection.rs @@ -1,17 +1,17 @@ use crate::{auth, cluster, incoming::Incoming, refusals::RefusalKind}; +use futures::FutureExt as _; use tracing::Instrument; -#[cfg(all(target_os = "linux", feature = "_uring"))] use tracing::instrument::WithSubscriber; /// An admission refusal carrying the session code to close the request with. /// /// Used only on the pre-accept auth path so the caller can close once with /// the right code instead of sprinkling close/return at each failure site. -pub(crate) struct Rejection { - pub(crate) code: moq_net::SessionError, +struct Rejection { + code: moq_net::SessionError, refusal: Option, - pub(crate) source: anyhow::Error, + source: anyhow::Error, } impl From for Rejection { @@ -30,8 +30,8 @@ impl From for Rejection { /// [`run`](Self::run) to admit the request through its lease, wire up /// publish/subscribe origins, and serve the session until it closes. /// -/// The request is a moq-tokio [`Request`](moq_tokio::server::Request), or an -/// in-memory accepted session when the `test-support` feature is enabled. +/// The request is a moq-tokio [`Request`](moq_tokio::server::Request), an io_uring +/// worker's, or an in-memory accepted session when the `test-support` feature is enabled. pub struct Connection { request: R, lifecycle: Lifecycle, @@ -39,7 +39,7 @@ pub struct Connection { /// Shared owner of admission, supervision and the connection's log span. #[derive(Clone)] -pub(crate) struct Lifecycle { +struct Lifecycle { id: u64, span: tracing::Span, cluster: cluster::Cluster, @@ -83,21 +83,34 @@ impl Connection { /// ends only this connection, as an error. pub async fn run(self) { let Self { request, lifecycle } = self; - lifecycle.run(Self::serve(request, &lifecycle)).await; + match request.shared().cloned() { + Some(shared) => lifecycle.start(&shared, Self::start(request, &lifecycle)).await, + None => { + lifecycle + .run(async { Self::start(request, &lifecycle).await?.await }) + .await + } + } } - async fn serve(mut request: R, lifecycle: &Lifecycle) -> anyhow::Result<()> { + /// Admit the request and answer its SETUP, returning the session's supervision. + async fn start(mut request: R, lifecycle: &Lifecycle) -> anyhow::Result { + let shared = request.shared().cloned(); + let admit = { + let admission = request.admission(&lifecycle.auth); + let pacer = request.pacer(); + let lifecycle = lifecycle.clone(); + async move { lifecycle.admit(admission, pacer).await } + }; // Abandoned at the drain deadline, so the deadline bounds this task and its // socket like it bounds a session, and at the handshake deadline, so a peer // cannot hold the connection open behind a decision that never comes. // Boxed: the admission future is large, and nesting it inline overflows debug stacks. let admitted = request - .within(Box::pin( - lifecycle.admit(request.admission(&lifecycle.auth), request.pacer()), - )) + .within(Box::pin(lifecycle.on(shared.as_ref(), admit))) .await .inspect_err(|_| lifecycle.cluster.refusals.record(RefusalKind::Unavailable)) - .map_err(|err| err.context("admission timed out"))?; + .map_err(|err| err.context("admission timed out"))??; let (admitted, registration) = match admitted { Ok(admitted) => admitted, Err(err) => { @@ -142,23 +155,34 @@ impl Connection { let serving = lifecycle.shutdown.serve(); // Bounded too: a peer that stalls the SETUP reply must not outlive the relay. // Abandoning the reply refuses the session with the timeout code, which its stats count. - let (session, driver) = lifecycle - .shutdown - .until_deadline(request.ok()) - .await - .ok_or_else(|| anyhow::anyhow!("the SETUP reply was still pending at the drain deadline"))??; + let expired = { + let shutdown = lifecycle.shutdown.clone(); + lifecycle.on(shared.as_ref(), async move { shutdown.expired().await }) + }; + let (session, driver) = tokio::select! { + biased; + _ = expired => anyhow::bail!("the SETUP reply was still pending at the drain deadline"), + reply = request.ok() => reply?, + }; // Joined rather than awaited in turn, for a driver that runs only when polled. A // spawned driver's panic still surfaces here, so it cannot look like cancellation. - let (result, driven) = futures::join!( - lifecycle.supervise(serving, session, lease, registration, transport), - R::drive(driver), - ); - driven?; - result + let lifecycle = lifecycle.clone(); + Ok(async move { + let (result, driven) = futures::join!( + lifecycle.supervise(serving, session, lease, registration, transport), + R::drive(driver), + ); + driven?; + result + } + .boxed()) } } +/// A started session's supervision, which runs until the session closes. +type Supervision = futures::future::BoxFuture<'static, anyhow::Result<()>>; + /// The span every line about one connection logs under, on whichever runtime emits it. fn span(id: u64, remote: Option) -> tracing::Span { tracing::info_span!( @@ -171,12 +195,7 @@ fn span(id: u64, remote: Option) -> tracing::Span { } impl Lifecycle { - pub(crate) fn new( - cluster: cluster::Cluster, - auth: auth::Auth, - id: u64, - remote: Option, - ) -> Self { + fn new(cluster: cluster::Cluster, auth: auth::Auth, id: u64, remote: Option) -> Self { Self { id, span: span(id, remote), @@ -187,26 +206,25 @@ impl Lifecycle { } } - pub(crate) fn with_shutdown(mut self, shutdown: crate::shutdown::Observer) -> Self { + fn with_shutdown(mut self, shutdown: crate::shutdown::Observer) -> Self { self.shutdown = shutdown; self } - pub(crate) fn with_sessions(mut self, sessions: crate::session::Registry) -> Self { + fn with_sessions(mut self, sessions: crate::session::Registry) -> Self { self.sessions = Some(sessions); self } /// Run `task` in this connection's span, then log its close; a panic closes it as an error. - pub(crate) async fn run(&self, task: impl std::future::Future>) { + async fn run(&self, task: impl std::future::Future>) { let result = self.isolate(task).await; self.closed(result); } - /// Run the ring worker's half, then supervise the started session on `runtime`, which logs + /// Run the connection task's half, then supervise the started session on `runtime`, which logs /// the close there: a worker stopped at shutdown drops its tasks, and the line with them. - #[cfg(all(target_os = "linux", feature = "_uring"))] - pub(crate) async fn start( + async fn start( &self, runtime: &tokio::runtime::Handle, start: impl std::future::Future>, @@ -240,10 +258,21 @@ impl Lifecycle { } } + /// Run `task` on `runtime`, or in place without one; see [`shared`](Self::shared). + async fn on( + &self, + runtime: Option<&tokio::runtime::Handle>, + task: impl std::future::Future + Send + 'static, + ) -> anyhow::Result { + match runtime { + Some(runtime) => self.shared(runtime, task).await, + None => Ok(task.await), + } + } + /// Keep the connection span across a worker-to-shared-runtime handoff and return panics to its owner. /// Dropping the returned future aborts the task, so it never outlives its connection. - #[cfg(all(target_os = "linux", feature = "_uring"))] - pub(crate) async fn shared( + async fn shared( &self, runtime: &tokio::runtime::Handle, task: impl std::future::Future + Send + 'static, @@ -270,7 +299,7 @@ impl Lifecycle { /// Admit `request`, holding its connection to a capped grant through `pacer`, or /// refusing a capped grant without one. - pub(crate) async fn admit( + async fn admit( &self, request: moq_auth::Request, pacer: Option, @@ -343,7 +372,7 @@ impl Lifecycle { Ok((admitted, registration)) } - pub(crate) async fn supervise( + async fn supervise( &self, serving: crate::shutdown::Serving, session: moq_net::Session, @@ -438,11 +467,11 @@ pub(crate) fn session_bytes(session: &moq_net::Session) -> moq_auth::Bytes { } } -#[cfg(all(test, target_os = "linux", feature = "_uring"))] +#[cfg(test)] #[path = "../tests/integration/logs.rs"] mod logs; -#[cfg(all(test, target_os = "linux", feature = "_uring"))] +#[cfg(test)] mod tests { use super::*; diff --git a/rs/moq-relay/src/incoming.rs b/rs/moq-relay/src/incoming.rs index 9033af43289..4bd24d35f99 100644 --- a/rs/moq-relay/src/incoming.rs +++ b/rs/moq-relay/src/incoming.rs @@ -4,11 +4,15 @@ use crate::auth; /// A session whose SETUP awaits the relay's reply: what a [`Connection`](crate::Connection) serves. /// -/// Sealed: native requests, plus in-memory accepted sessions with `test-support`. +/// Sealed: native requests, io_uring workers' requests, plus in-memory accepted sessions +/// with `test-support`. pub trait Incoming: Sealed {} impl Incoming for moq_tokio::server::Request {} +#[cfg(all(target_os = "linux", feature = "_uring"))] +impl Incoming for crate::uring::Request {} + #[cfg(feature = "test-support")] impl Incoming for Accepted {} @@ -40,13 +44,20 @@ impl Accepted { )] pub trait Sealed: Sized { /// The running session's protocol driver. - type Driver; + type Driver: Send + 'static; /// The peer's socket address, for the connection's log span. fn remote(&self) -> Option; /// The transport, for the line logged once the session is negotiated. - fn transport(&self) -> impl std::fmt::Display + use; + fn transport(&self) -> &'static str; + + /// The tokio runtime that admits and supervises the session, when the connection's own + /// task cannot: one on an io_uring worker, which has no tokio reactor for the auth + /// client and stops before the drain ends. `None` does both in place. + fn shared(&self) -> Option<&tokio::runtime::Handle> { + None + } /// What the auth decider is asked about this session. fn admission(&self, auth: &auth::Auth) -> moq_auth::Request; @@ -81,7 +92,7 @@ pub trait Sealed: Sized { async fn ok(self) -> anyhow::Result<(moq_net::Session, Self::Driver)>; /// Run the session's protocol until it ends, alongside its supervision. - async fn drive(driver: Self::Driver) -> anyhow::Result<()>; + fn drive(driver: Self::Driver) -> impl std::future::Future> + Send; } impl Sealed for moq_tokio::server::Request { @@ -92,8 +103,8 @@ impl Sealed for moq_tokio::server::Request { self.remote_addr() } - fn transport(&self) -> impl std::fmt::Display + use<> { - moq_tokio::server::Request::transport(self) + fn transport(&self) -> &'static str { + moq_tokio::server::Request::transport(self).as_str() } fn admission(&self, auth: &auth::Auth) -> moq_auth::Request { @@ -173,7 +184,7 @@ impl Sealed for Accepted { None } - fn transport(&self) -> impl std::fmt::Display + use<> { + fn transport(&self) -> &'static str { self.transport.as_str() } @@ -182,11 +193,7 @@ impl Sealed for Accepted { } async fn within(&mut self, work: impl std::future::Future) -> anyhow::Result { - let expired = self.handshake.expired(); - match futures::future::select(std::pin::pin!(work), std::pin::pin!(expired)).await { - futures::future::Either::Left((done, _)) => Ok(done), - futures::future::Either::Right(_) => Err(moq_net::Error::Timeout.into()), - } + within(&mut self.handshake, work).await } async fn reject(self, code: moq_net::SessionError) { @@ -236,3 +243,16 @@ impl Sealed for Accepted { Ok(()) } } + +/// Finish `work` before `handshake`'s accept deadline, which refuses the session if it passes. +#[cfg(any(feature = "test-support", all(target_os = "linux", feature = "_uring")))] +pub(crate) async fn within( + handshake: &mut moq_net::server::Handshake, + work: impl std::future::Future, +) -> anyhow::Result { + let expired = handshake.expired(); + match futures::future::select(std::pin::pin!(work), std::pin::pin!(expired)).await { + futures::future::Either::Left((done, _)) => Ok(done), + futures::future::Either::Right(_) => Err(moq_net::Error::Timeout.into()), + } +} diff --git a/rs/moq-relay/src/relay.rs b/rs/moq-relay/src/relay.rs index c8c218ca7db..cb0f7b94df4 100644 --- a/rs/moq-relay/src/relay.rs +++ b/rs/moq-relay/src/relay.rs @@ -154,7 +154,7 @@ impl Relay { .implementation .get_or_insert_with(|| format!("moq-relay/{} {}", env!("CARGO_PKG_VERSION"), moq_net::IMPLEMENTATION)) .clone(); - config.connect.implementation = Some(implementation.clone()); + config.connect.implementation = Some(implementation); let resolved_config = Box::new(config.clone()); let drain_timeout = config.drain_timeout(); // The name this relay reports in every auth request: the stats node label, @@ -185,9 +185,7 @@ impl Relay { // Every listener shares admission ownership before its workers can accept. let (shutdown_trigger, shutdown) = shutdown::Observer::new(drain_timeout); - let server_versions = config.listen.versions(); let handshake_timeout = config.listen.resolved_timeout(); - let server_extensions = config.listen.extensions; // Bind the QUIC workers first: they own the listen address when configured, // so the server below must not also try to bind it. @@ -334,10 +332,7 @@ impl Relay { let (ready, _) = tokio::sync::watch::channel(Startup::Starting); let web = web::Web::new(auth.clone(), cluster.clone(), certificates, config.web) .with_shutdown(shutdown.clone()) - .with_versions(server_versions) - .with_extensions(server_extensions) - .with_implementation(implementation) - .with_timeout(handshake_timeout) + .with_listen(&config.listen) .with_quic(&config.quic) .with_sessions(sessions.clone()) .with_tls_acceptor(server.tls_acceptor()); diff --git a/rs/moq-relay/src/uring.rs b/rs/moq-relay/src/uring.rs index b2020c79249..eddcce74638 100644 --- a/rs/moq-relay/src/uring.rs +++ b/rs/moq-relay/src/uring.rs @@ -9,12 +9,13 @@ //! moq-transport and the SETUP-negotiated versions run through //! [`moq_uring::transport::Local`], since their drivers need `Send`. //! -//! The split of work mirrors the relay's topology: a worker owns everything -//! transport-shaped (the handshake, the session's protocol machine), while -//! authentication and session supervision run on the shared tokio runtime, -//! which owns the HTTP client, the timers, and the origins. The session -//! handle is `Send + Sync` whatever drives its transport, which is what makes -//! the handoff free. +//! Each accepted session runs through the same [`Connection`] as the tokio +//! listeners. The split of work mirrors the relay's topology: a worker owns +//! everything transport-shaped (the handshake, the session's protocol machine), +//! while authentication and session supervision run on the shared tokio runtime, +//! which owns the HTTP client and the origins. The session handle is +//! `Send + Sync` whatever drives its transport, which is what makes the handoff +//! free. use std::net::SocketAddr; use std::num::NonZeroU64; @@ -26,7 +27,7 @@ use anyhow::Context as _; use futures::FutureExt as _; use tracing::Instrument; -use crate::{auth, cluster, connection::Lifecycle, shutdown}; +use crate::{Connection, auth, cluster, shutdown}; /// A stop signal a worker parks on, wakeable from the shared runtime. #[derive(Default)] @@ -71,8 +72,8 @@ struct Serve { /// The moq ALPNs this listener speaks, for the WebTransport subprotocol /// pick. alpns: Arc>, - /// The moq handshake every connection runs: the listener's versions and - /// the extensions and implementation it offers in SETUP. + /// The moq handshake every connection runs, mapped from `listen` as the + /// tokio listeners map it, opening `[quic] receive_window` on admission. moq: moq_net::Server, /// The address every worker socket shares, reported as a session's `local`. local: SocketAddr, @@ -82,9 +83,6 @@ struct Serve { /// `listen.reset.secret`, shared so any worker resets any worker's /// connections, a restarted group's included. reset_secret: Option<[u8; 32]>, - /// `[quic] receive_window`, opened once a session is admitted: until then a - /// connection holds [`moq_net::transport::SETUP_RECEIVE_WINDOW`]. - window: Option, /// The served TLS policy, which builds each client's identity against its roots. tls: moq_tokio::tls::Serve, } @@ -105,8 +103,6 @@ pub struct Workers { members: Vec, addr: SocketAddr, server: moq_uring::quic::server::Config, - /// `[quic] receive_window`, which admission opens; see [`Serve::window`]. - window: Option, /// The TLS policy every worker serves, and the live resolver behind the /// fingerprint endpoint. tls: moq_tokio::tls::Serve, @@ -149,8 +145,9 @@ impl Workers { let moq_tokio::listen::Resolved { bind, versions, - extensions, - implementation, + // With the versions and timeout, mapped onto the moq handshake by `listen.server()`. + extensions: _, + implementation: _, timeout, tls: listen_tls, preferred_v4, @@ -164,6 +161,8 @@ impl Workers { }, reset_secret, } = server.listen.resolve()?; + // The one mapping of `listen` onto the moq handshake, shared with the tokio listeners. + let moq = server.listen.server(); let quic = &server.quic; for (configured, setting) in [ (preferred_v4.is_some(), "listen.preferred_v4"), @@ -245,7 +244,11 @@ impl Workers { } let mut server = moq_uring::quic::server::Config::new(tls); server.transport = transport(&quic)?; - let window = server.transport.limits.receive_window; + // Held back until admission, which opens the configured window. + let moq = match server.transport.limits.receive_window { + Some(window) => moq.with_receive_window(window), + None => moq, + }; server.transport.limits.receive_window = NonZeroU64::new(moq_net::transport::SETUP_RECEIVE_WINDOW); // GSO is a property of the socket, not the connection, so it rides the @@ -268,20 +271,11 @@ impl Workers { members, addr, server, - window, tls: serve, udp, pin: config.pin, alpns: Arc::new(alpns), - moq: { - let mut moq = moq_net::Server::new() - .with_versions(versions) - .with_extensions(extensions); - if let Some(implementation) = &implementation { - moq = moq.with_implementation(implementation); - } - moq - }, + moq, timeout, reset_secret, threads: Vec::new(), @@ -339,7 +333,6 @@ impl Workers { local: self.addr, timeout: self.timeout, reset_secret: self.reset_secret, - window: self.window, tls: self.tls.clone(), }; @@ -658,23 +651,7 @@ fn serve_worker(spawn: Spawn) -> bool { loop { match endpoint.accept().await { Ok(conn) => { - let id = serve.cluster.next_connection_id(); - let conn_handle = accept_handle.clone(); - let serve = serve.clone(); - let lifecycle = Lifecycle::new( - serve.cluster.clone(), - serve.auth.clone(), - id, - Some(conn.remote_address()), - ) - .with_shutdown(serve.shutdown.clone()) - .with_sessions(serve.sessions.clone()); - accept_handle.spawn(async move { - let shared = serve.tokio.clone(); - lifecycle - .start(&shared, serve_connection(conn_handle, conn, serve, &lifecycle)) - .await; - }); + accept_handle.spawn(serve_connection(accept_handle.clone(), conn, serve.clone())); } Err(err) => { let _ = accept_failures @@ -692,58 +669,32 @@ fn serve_worker(spawn: Spawn) -> bool { true } -/// What remains of a connection's handshake deadline, and how to tell the peer -/// it passed. -struct Deadline { - /// When the handshake must be done, if ever. - at: Option, - timer: moq_uring::Timer, - /// Closes with the timeout code once a session exists to carry it. Until - /// then a timed-out handshake is just dropped, and its guard closes in the - /// handshake's own error space, as HTTP/3 has no MoQ codes. - close: Option, -} - -impl Deadline { - fn new(handle: &moq_uring::Handle, conn: &moq_uring::quic::Connection) -> Self { - let at = conn.handshake().deadline; - let mut timer = handle.timer(); - timer.set(at); - Self { at, timer, close: None } - } - - async fn bound>( - &mut self, - handshake: impl Future>, - ) -> anyhow::Result { - let timer = std::pin::pin!(self.timer.wait()); - let handshake = std::pin::pin!(handshake); - match futures::future::select(timer, handshake).await { - futures::future::Either::Left(_) => { - // Close before dropping the pending handshake, whose cancellation guard - // would otherwise replace the timeout with an abandoned-handshake error. - let err = moq_net::Error::Timeout; - if let Some(close) = &mut self.close { - close.close(moq_net::SessionError::Timeout.to_code(), &err.to_string()); - } - Err(err.into()) - } - futures::future::Either::Right((result, _)) => result.map_err(Into::into), - } - } +/// Accept one connection on its worker, then serve it through [`Connection`]. +async fn serve_connection(handle: moq_uring::Handle, conn: moq_uring::quic::Connection, serve: Serve) { + let remote = conn.remote_address(); + // Like the tokio listener, a connection that never reaches its SETUP is no session. + let request = match moq_tokio::panic::isolate(accept(handle, conn, &serve)).await { + Some(Ok(request)) => request, + Some(Err(err)) => return tracing::debug!(%remote, %err, "failed to accept session"), + // `isolate` logged the panic; only this handshake is lost. + None => return, + }; + Connection::new(request, serve.cluster.clone(), serve.auth.clone()) + .with_id(serve.cluster.next_connection_id()) + .with_shutdown(serve.shutdown.clone()) + .with_sessions(serve.sessions.clone()) + .run() + .await; } -/// Start one accepted connection on its worker: the handshakes here, then -/// [`serve_request`]. -async fn serve_connection( +/// Run the transport handshake and read the moq SETUP, by the deadline the endpoint +/// started at the connection's first packet and enforced through the QUIC handshake. +async fn accept( handle: moq_uring::Handle, conn: moq_uring::quic::Connection, - serve: Serve, - lifecycle: &Lifecycle, -) -> anyhow::Result { - // The endpoint started this deadline at the connection's first packet and - // enforced it through the QUIC handshake; it covers SETUP OK too. - let mut deadline = Deadline::new(&handle, &conn); + serve: &Serve, +) -> anyhow::Result { + let deadline = conn.handshake().deadline; // Read the link facts now: the handshake below consumes the connection. let remote = conn.remote_address(); @@ -754,12 +705,10 @@ async fn serve_connection( // Browsers speak WebTransport; everyone else raw QUIC. The moq version // rides the ALPN for raw QUIC and the WebTransport subprotocol for `h3`. - // The negotiated moq protocol: the WebTransport sub-protocol for `h3`, else the ALPN. let mut alpn = conn.protocol().map(str::to_owned); - let (transport, url) = match conn.protocol() { + let (conn, url) = match conn.protocol() { Some("h3") => { - let request = deadline - .bound(moq_uring::quic::web::Request::accept(conn)) + let request = within(deadline, moq_uring::quic::web::Request::accept(conn)) .await .context("WebTransport handshake failed")?; let url = request.url.clone(); @@ -772,9 +721,8 @@ async fn serve_connection( if let Some(protocol) = &protocol { response = response.with_protocol(protocol); } - alpn = protocol.clone(); - let session = deadline - .bound(request.respond(response)) + alpn = protocol; + let session = within(deadline, request.respond(response)) .await .context("WebTransport response failed")?; (session, Some(url)) @@ -782,197 +730,223 @@ async fn serve_connection( _ => (moq_uring::quic::web::Session::raw(conn), None), }; - deadline.close = Some(transport.clone()); - let conn = transport.clone(); - let server = serve.moq.clone(); - let session = moq_uring::transport::Local::new(transport); - let now = moq_net::time::Instant::now(); - let mut accepted = Accepted { + let session = moq_uring::transport::Local::new(conn.clone()); + let handshake = serve + .moq + .accept_request(moq_net::time::Instant::now(), session) + .with_deadline(deadline) + .await + .context("moq handshake failed")?; + Ok(Request { handle, + shared: serve.tokio.clone(), conn, - deadline, - serve, + handshake, + local: serve.local, remote, sni, identity, alpn, url, + caps: kio::Producer::new(None), + quic_limits: Default::default(), + }) +} + +/// Finish a transport handshake by `deadline`, or fail with a timeout. +/// +/// Nothing is established yet that could carry a close code, so an expired +/// handshake is dropped, which closes its connection. +async fn within>( + deadline: Option, + handshake: impl Future>, +) -> anyhow::Result { + let done = match deadline { + Some(deadline) => kio::clock::timeout_at(deadline, handshake) + .await + .map_err(|_| moq_net::Error::Timeout)?, + None => handshake.await, }; - let at = accepted.deadline.at; - let request = accepted - .deadline - .bound(server.accept_request(now, session).with_deadline(at)) - .await - .context("moq handshake failed")?; - serve_request(accepted, request, lifecycle).await + done.map_err(Into::into) } -/// A connection past its transport handshake, with the facts read off it -/// before the moq handshake consumed it. -struct Accepted { +/// A session on an io_uring worker whose SETUP awaits the relay's reply. +/// +/// It never leaves its worker, which drives the connection and, once the session +/// starts, its protocol; admission and supervision hop to the shared runtime. +pub(crate) struct Request { handle: moq_uring::Handle, + /// The runtime that owns authentication and session supervision. + shared: tokio::runtime::Handle, conn: moq_uring::quic::web::Session, - deadline: Deadline, - serve: Serve, + handshake: moq_net::server::Handshake, + local: SocketAddr, remote: SocketAddr, sni: Option, identity: Option, alpn: Option, + /// The request URL, on WebTransport only. url: Option, + /// The latest caps the lease paced, `None` until it paces any. The lease lives on the + /// shared runtime, so a task on this worker applies them to the connection. + caps: kio::Producer>, + /// What [`ok`](crate::incoming::Sealed::ok) applies before answering SETUP. + quic_limits: moq_quic::Limits, } -/// A started session's supervision. -type Supervision = futures::future::BoxFuture<'static, anyhow::Result<()>>; - -/// Authenticate on the shared runtime, then start the session on the worker. -/// Returns the session's supervision, which runs on the shared runtime too. -async fn serve_request( - accepted: Accepted, - request: moq_net::server::Handshake, - lifecycle: &Lifecycle, -) -> anyhow::Result { - let Accepted { - handle, - conn, - mut deadline, - serve, - remote, - sni, - identity, - alpn, - url, - } = accepted; +impl crate::incoming::Sealed for Request { + /// Spawned on the worker by [`ok`](Self::ok). + type Driver = (); - // WebTransport addresses its path through the URL, but both transports can - // carry a credential in the MoQ SETUP. - let transport = match url { - Some(_) => moq_auth::Transport::WebTransport, - None => moq_auth::Transport::Quic, - }; - let mut auth_request = crate::auth::request_for_setup(&serve.auth, &request, transport); - if let Some(url) = &url { - auth_request.path = url.path().to_string(); - auth_request.query = url.query().map(str::to_owned); + fn remote(&self) -> Option { + Some(self.remote) + } + + fn transport(&self) -> &'static str { + self.auth_transport().as_str() } - auth_request.remote = Some(remote); - auth_request.local = Some(serve.local); - // Only the SNI, like the tokio listener: the CONNECT host selected no certificate. - auth_request.server_name = sni; - auth_request.alpn = alpn.clone(); - auth_request.tls = identity.as_ref().and_then(crate::auth::peer); - let admission = lifecycle.clone(); - // The connection is pinned to this worker, so a cap that changes on the shared - // runtime reaches it through `caps`, which a task here applies. - let caps = kio::Producer::new(auth::Caps::default()); - let pacer = { - let caps = caps.clone(); - auth::Pacer::new(move |new| { + + fn shared(&self) -> Option<&tokio::runtime::Handle> { + Some(&self.shared) + } + + fn admission(&self, auth: &auth::Auth) -> moq_auth::Request { + let mut request = auth::request_for_setup(auth, &self.handshake, self.auth_transport()); + // WebTransport addresses its path through the URL; raw QUIC carries it in SETUP. + if let Some(url) = &self.url { + request.path = url.path().to_string(); + request.query = url.query().map(str::to_owned); + } + request.remote = Some(self.remote); + request.local = Some(self.local); + // Only the SNI, like the tokio listener: the CONNECT host selected no certificate. + request.server_name = self.sni.clone(); + request.alpn = self.alpn.clone(); + request.tls = self.identity.as_ref().and_then(auth::peer); + request + } + + async fn within(&mut self, work: impl Future) -> anyhow::Result { + crate::incoming::within(&mut self.handshake, work).await + } + + async fn reject(self, code: moq_net::SessionError) { + self.handshake.close(code.into()); + } + + fn pacer(&self) -> Option { + let caps = self.caps.clone(); + Some(auth::Pacer::new(move |new| { if let Ok(mut caps) = caps.write() { - *caps = new; - } - }) - }; - // The handshake deadline covers admission too, so a peer cannot hold the connection - // open behind a decision that never comes. Dropping the shared task withdraws it. - let decided = deadline - .bound(lifecycle.shared( - &serve.tokio, - async move { admission.admit(auth_request, Some(pacer)).await }, - )) - .await - .inspect_err(|err| { - if matches!(err.downcast_ref::(), Some(moq_net::Error::Timeout)) { - serve.cluster.refusals.record(crate::refusals::RefusalKind::Unavailable); + *caps = Some(new); } - })?; - let (admitted, registration) = match decided { - Ok(admitted) => admitted, - Err(err) => { - request.close(err.code.into()); - return Err(err.source); - } - }; + })) + } + + fn with_quic_limits(mut self, limits: moq_quic::Limits) -> anyhow::Result { + limits.validate()?; + self.quic_limits = limits; + Ok(self) + } - let cluster::Admitted { - lease, - publisher, - subscriber, - stats, - } = admitted; + fn auth(&self) -> moq_net::auth::Handle { + self.handshake.auth() + } - let mut request = request.with_stats(stats).with_limits(serve.cluster.limits(&lease)); - if let Some(subscriber) = subscriber { - request = request.with_publisher(subscriber); + fn requests(&mut self) -> moq_net::auth::Requests { + self.handshake.requests() } - if let Some(publisher) = publisher { - request = request.with_subscriber(publisher); + + fn role(&self) -> Option { + self.handshake.role() } - // A grant that no longer holds is refused, so the session records why it ended. - let lease = match lease.preparing(request.auth(), request.requests(), request.role()) { - Ok(lease) => lease, - Err(err) => { - request.close(err.clone()); - return Err(err.into()); - } - }; - // Before SETUP OK, so the first requests see them: a peer's wider limits, the caps, - // and then the window the handshake held back, which the caps pace. - let mut limits = serve.cluster.quic_limits(&lease); - limits.receive_window = limits.receive_window.or(serve.window); - conn.set_limits(limits)?; - // The task holds only a consumer: dropping the lease, and with it `pacer`, closes - // `caps`, which ends the task and releases its handle on the connection. - let mut applied = lease.caps(); - if let Ok(mut caps) = caps.write() { - *caps = applied; + + fn with_stats(mut self, stats: moq_net::stats::Session) -> Self { + self.handshake = self.handshake.with_stats(stats); + self } - let caps = caps.consume(); - let pacing = async move { - while let Ok(next) = caps - .wait(|caps| match **caps != applied { - true => std::task::Poll::Ready(**caps), - false => std::task::Poll::Pending, - }) - .await - { - if let Err(err) = conn.set_limits(next.limits()) { - tracing::warn!(%err, "failed to apply bitrate caps"); - } - applied = next; + + fn with_limits(mut self, limits: moq_net::session::Limits) -> Self { + self.handshake = self.handshake.with_limits(limits); + self + } + + fn with_publisher(mut self, publish: moq_net::origin::Consumer) -> Self { + self.handshake = self.handshake.with_publisher(publish); + self + } + + fn with_subscriber(mut self, subscribe: moq_net::origin::Producer) -> Self { + self.handshake = self.handshake.with_subscriber(subscribe); + self + } + + async fn ok(self) -> anyhow::Result<(moq_net::Session, Self::Driver)> { + let Self { + handle, + conn, + mut handshake, + caps, + mut quic_limits, + .. + } = self; + // The other limits go first; moq-net opens the receive window after them, so a + // `receive_rate` paces the opened window. + if let Some(window) = quic_limits.receive_window.take() { + handshake = handshake.with_receive_window(window); } - }; - handle.spawn(pacing.in_current_span()); - let serving = serve.shutdown.serve(); - // Bounded by the drain deadline too, like the tokio path, so a peer that stalls - // the SETUP reply cannot hold the drain. That is a tokio timer, so it runs on the - // shared runtime, and dropping it aborts the task. Dropping the reply refuses the - // session with the timeout code, which its stats count. - let shutdown = serve.shutdown.clone(); - let expired = std::pin::pin!(lifecycle.shared(&serve.tokio, async move { shutdown.expired().await })); - let reply = std::pin::pin!(request.ok()); - let (session, driver) = match futures::future::select(reply, expired).await { - futures::future::Either::Left((reply, _)) => reply?, - futures::future::Either::Right(_) => anyhow::bail!("the SETUP reply was still pending at the drain deadline"), - }; - let driver_handle = handle.clone(); - let driver = async move { - match driver_handle.run(driver).await { - moq_net::Error::Closed => {} - err => tracing::debug!(%err, "session driver ended"), + conn.set_limits(quic_limits).expect("validated by with_quic_limits"); + + // Holds only a consumer: dropping the lease, and with it the pacer, closes `caps`, + // which ends the task and releases its handle on the connection. + let caps = caps.consume(); + let pacing = { + let conn = conn.clone(); + async move { + let mut applied = None; + while let Ok(next) = caps + .wait(|caps| match **caps != applied { + true => std::task::Poll::Ready(**caps), + false => std::task::Poll::Pending, + }) + .await + { + if let Some(caps) = next + && let Err(err) = conn.set_limits(caps.limits()) + { + tracing::warn!(%err, "failed to apply bitrate caps"); + } + applied = next; + } + } + }; + handle.spawn(pacing.in_current_span()); + + let (session, driver) = handshake.ok().await?; + let driver_handle = handle.clone(); + let driver = async move { + match driver_handle.run(driver).await { + moq_net::Error::Closed => {} + err => tracing::debug!(%err, "session driver ended"), + } + }; + handle.spawn(moq_tokio::panic::isolate(driver).in_current_span().map(drop)); + Ok((session, ())) + } + + async fn drive((): Self::Driver) -> anyhow::Result<()> { + Ok(()) + } +} + +impl Request { + /// WebTransport when the request carried a URL, else raw QUIC. + fn auth_transport(&self) -> moq_auth::Transport { + match self.url { + Some(_) => moq_auth::Transport::WebTransport, + None => moq_auth::Transport::Quic, } - }; - handle.spawn(moq_tokio::panic::isolate(driver).in_current_span().map(drop)); - - // The driver stays on this worker. The session handle is Send + Sync, so its - // supervision (credential expiry, GOAWAY drain) runs with the timers and the - // shutdown broadcast on the shared runtime. - let lifecycle = lifecycle.clone(); - Ok(async move { - lifecycle - .supervise(serving, session, lease, registration, transport) - .await } - .boxed()) } #[cfg(test)] diff --git a/rs/moq-relay/src/web.rs b/rs/moq-relay/src/web.rs index 37a8d1a210e..e86a86ec2a3 100644 --- a/rs/moq-relay/src/web.rs +++ b/rs/moq-relay/src/web.rs @@ -168,6 +168,8 @@ pub(crate) struct WebState { /// handshake and partial headers included), and a WebSocket wait for its MoQ /// SETUP; `None` waits forever. pub(crate) timeout: Option, + /// The moq-net server each upgraded WebSocket accepts its session with. + pub(crate) server: moq_net::Server, /// The `[quic]` windows and stream count each WebSocket session starts with. pub(crate) quic: moq_tokio::quic::Resolved, } @@ -182,18 +184,12 @@ pub struct Addrs { pub https: Option, } -/// The implementation WebSocket sessions advertise in SETUP, or `None` for the library's. -#[derive(Clone, Default)] -#[cfg_attr(not(feature = "websocket"), allow(dead_code, reason = "websocket-only"))] -pub(crate) struct Implementation(pub Option); - /// Run a HTTP server using Axum pub struct Web { state: Arc, config: Config, + /// The versions [`WebState::server`] accepts, which pick the WebSocket subprotocol. versions: moq_net::Versions, - extensions: moq_net::setup::Extensions, - implementation: Implementation, health: moq_tokio::accept::Health, http_listener: Option, https_listener: Option, @@ -214,21 +210,21 @@ impl Web { certificates: moq_tokio::tls::Certificates, config: Config, ) -> Self { + let listen = moq_tokio::listen::Config::default(); let state = Arc::new(WebState { auth, cluster, certificates, shutdown: crate::shutdown::Observer::disabled(), sessions: crate::session::Registry::new(), - timeout: moq_tokio::listen::Config::default().resolved_timeout(), + timeout: listen.resolved_timeout(), + server: listen.server(), quic: Default::default(), }); Self { state, config, - versions: moq_net::Versions::all(), - extensions: Default::default(), - implementation: Default::default(), + versions: listen.versions(), health: moq_tokio::accept::Health::new("web"), http_listener: None, https_listener: None, @@ -311,21 +307,28 @@ impl Web { self.state.certificates.fingerprints() } - /// Restrict which MoQ versions WebSocket sessions accept, in preference order. - pub fn with_versions(mut self, versions: moq_net::Versions) -> Self { - self.versions = versions; + /// Accept WebSocket sessions as `listen` describes, and bound connections by its timeout + /// (see [`with_timeout`](Self::with_timeout)). + pub(crate) fn with_listen(mut self, listen: &moq_tokio::listen::Config) -> Self { + let state = Arc::get_mut(&mut self.state).expect("with_listen called after routes were built"); + state.timeout = listen.resolved_timeout(); + state.server = listen.server(); + self.versions = listen.versions(); self } - /// Choose which moq-transport extensions WebSocket sessions offer in SETUP. - pub(crate) fn with_extensions(mut self, extensions: moq_net::setup::Extensions) -> Self { - self.extensions = extensions; + /// Restrict which MoQ versions WebSocket sessions accept, in preference order. + pub fn with_versions(mut self, versions: moq_net::Versions) -> Self { + let state = Arc::get_mut(&mut self.state).expect("with_versions called after routes were built"); + state.server = state.server.clone().with_versions(versions.clone()); + self.versions = versions; self } /// Set the implementation WebSocket sessions advertise in SETUP; empty advertises nothing. pub fn with_implementation(mut self, implementation: impl Into) -> Self { - self.implementation = Implementation(Some(implementation.into())); + let state = Arc::get_mut(&mut self.state).expect("with_implementation called after routes were built"); + state.server = state.server.clone().with_implementation(implementation); self } @@ -369,6 +372,7 @@ impl Web { // hyper adds the header timeout to the clock unchecked, so an unreachable // one would panic on every connection. state.timeout = timeout.filter(|timeout| kio::clock::now().checked_add(*timeout).is_some()); + state.server = state.server.clone().with_timeout(state.timeout); self } @@ -444,8 +448,6 @@ impl Web { }; app.layer(Extension(self.versions.clone())) - .layer(Extension(self.extensions)) - .layer(Extension(self.implementation.clone())) .with_state(self.state.clone()) } diff --git a/rs/moq-relay/src/websocket.rs b/rs/moq-relay/src/websocket.rs index bb994cc0688..b56c79e0622 100644 --- a/rs/moq-relay/src/websocket.rs +++ b/rs/moq-relay/src/websocket.rs @@ -28,8 +28,6 @@ pub(crate) async fn serve_ws( socket_stats: Option>, admission: Option>, Extension(versions): Extension, - Extension(extensions): Extension, - Extension(crate::web::Implementation(implementation)): Extension, State(state): State>, mut request: Request, ) -> axum::response::Result { @@ -59,14 +57,7 @@ pub(crate) async fn serve_ws( uri.path_and_query().map_or("/", |value| value.as_str()) )) .map_err(|_| StatusCode::BAD_REQUEST)?; - let mut server = moq_net::Server::new() - .with_timeout(state.timeout) - .with_versions(versions) - .with_extensions(extensions); - if let Some(implementation) = implementation { - server = server.with_implementation(implementation); - } - let mut options = moq_tokio::websocket::Options::new(url, server, active); + let mut options = moq_tokio::websocket::Options::new(url, state.server.clone(), active); options.link.remote = Some(remote); options.link.alpn = Some(protocol.clone()); options.link.server_name = server_name.map(|Extension(ServerName(name))| name); diff --git a/rs/moq-tokio/src/server.rs b/rs/moq-tokio/src/server.rs index 891e99161d3..a3786da3d63 100644 --- a/rs/moq-tokio/src/server.rs +++ b/rs/moq-tokio/src/server.rs @@ -63,6 +63,19 @@ impl crate::listen::Config { } } + /// The moq-net server every listener accepts sessions with: its versions, the + /// extensions and implementation its SETUP offers, and its accept deadline. + pub fn server(&self) -> moq_net::Server { + let mut server = moq_net::Server::new() + .with_timeout(self.resolved_timeout()) + .with_versions(self.versions()) + .with_extensions(self.extensions); + if let Some(implementation) = &self.implementation { + server = server.with_implementation(implementation); + } + server + } + /// Whether a QUIC, TCP, or Unix bind is explicitly configured. pub fn has_explicit_bind(&self) -> bool { self.bind.is_some() || self.has_stream_listener() @@ -278,7 +291,6 @@ impl Server { config.validate()?; let versions = config.versions(); - let timeout = config.resolved_timeout(); // Build a QUIC backend when `--listen` is set, or when nothing else // is (the default). A stream-only server (`--listen-unix-bind` with no @@ -408,14 +420,7 @@ impl Server { #[cfg(all(feature = "tcp", feature = "_certs"))] let tls_stream = tls.map(|tls| TcpTls::new(&config, &versions, tls)).transpose()?; - let mut moq = moq_net::Server::new() - .with_timeout(timeout) - .with_versions(versions.clone()) - .with_extensions(config.extensions) - .with_stats(stats); - if let Some(implementation) = &config.implementation { - moq = moq.with_implementation(implementation); - } + let mut moq = config.server().with_stats(stats); if let Some(window) = window { moq = moq.with_receive_window(window); } From 3614429e280577098a0a914e26281ca9c9df2fe8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 15:15:18 -0800 Subject: [PATCH 3/4] feat(relay)!: Web takes the listen config through one public with_listen Web::with_versions, with_implementation and with_timeout are replaced by Web::with_listen(&listen::Config), so an embedder sets the WebSocket SETUP and the web timeout the one way the relay's other listeners read them. Co-Authored-By: Claude Opus 5.5 --- rs/moq-relay/benches/http_auth.rs | 4 ++- rs/moq-relay/src/web.rs | 60 ++++++++++++------------------- 2 files changed, 25 insertions(+), 39 deletions(-) diff --git a/rs/moq-relay/benches/http_auth.rs b/rs/moq-relay/benches/http_auth.rs index b67b5f70d94..97954763471 100644 --- a/rs/moq-relay/benches/http_auth.rs +++ b/rs/moq-relay/benches/http_auth.rs @@ -74,8 +74,10 @@ impl Fixture { let driver = tokio::spawn(async move { driver.run().await.unwrap() }); // No admission timeout, so held requests stay waiting however long setup and the // measurement take. + let mut listen = moq_tokio::listen::Config::default(); + listen.timeout = Some(std::time::Duration::ZERO.into()); let web = web::Web::new(auth, cluster, certificates, config) - .with_timeout(None) + .with_listen(&listen) .bind() .unwrap(); let addr = web.addrs().http.unwrap(); diff --git a/rs/moq-relay/src/web.rs b/rs/moq-relay/src/web.rs index e86a86ec2a3..ec9f453050f 100644 --- a/rs/moq-relay/src/web.rs +++ b/rs/moq-relay/src/web.rs @@ -307,9 +307,15 @@ impl Web { self.state.certificates.fingerprints() } - /// Accept WebSocket sessions as `listen` describes, and bound connections by its timeout - /// (see [`with_timeout`](Self::with_timeout)). - pub(crate) fn with_listen(mut self, listen: &moq_tokio::listen::Config) -> Self { + /// Accept WebSocket sessions as `listen` describes, the way the relay's other listeners do: + /// its versions, the extensions and implementation SETUP offers, and its timeout. + /// + /// The timeout closes a connection that sits that long with no complete request in + /// flight, including one stalled in its TLS handshake or headers or idle in keep-alive, + /// bounds an HTTP request's auth admission (answering 502 past it), and bounds a + /// WebSocket's MoQ SETUP after the upgrade. Zero, or a timeout past the clock's range, + /// waits forever. Defaults to [`moq_tokio::listen::Config::default`]. + pub fn with_listen(mut self, listen: &moq_tokio::listen::Config) -> Self { let state = Arc::get_mut(&mut self.state).expect("with_listen called after routes were built"); state.timeout = listen.resolved_timeout(); state.server = listen.server(); @@ -317,21 +323,6 @@ impl Web { self } - /// Restrict which MoQ versions WebSocket sessions accept, in preference order. - pub fn with_versions(mut self, versions: moq_net::Versions) -> Self { - let state = Arc::get_mut(&mut self.state).expect("with_versions called after routes were built"); - state.server = state.server.clone().with_versions(versions.clone()); - self.versions = versions; - self - } - - /// Set the implementation WebSocket sessions advertise in SETUP; empty advertises nothing. - pub fn with_implementation(mut self, implementation: impl Into) -> Self { - let state = Arc::get_mut(&mut self.state).expect("with_implementation called after routes were built"); - state.server = state.server.clone().with_implementation(implementation); - self - } - /// A live handle to the accept-loop health of the HTTP/HTTPS listeners, for an /// embedder that publishes it (see [`moq_tokio::accept`]). /// @@ -361,21 +352,6 @@ impl Web { self } - /// Close a connection that sits this long with no complete request in flight, - /// including one stalled in its TLS handshake or headers or idle in keep-alive, - /// bound an HTTP request's auth admission (answering 502 past it), and bound a - /// WebSocket's MoQ SETUP after the upgrade; `None`, or a timeout past the clock's - /// range, waits forever. Defaults to the - /// [`moq_tokio::listen::Config::resolved_timeout`] default. - pub fn with_timeout(mut self, timeout: Option) -> Self { - let state = Arc::get_mut(&mut self.state).expect("with_timeout called after routes were built"); - // hyper adds the header timeout to the clock unchecked, so an unreachable - // one would panic on every connection. - state.timeout = timeout.filter(|timeout| kio::clock::now().checked_add(*timeout).is_some()); - state.server = state.server.clone().with_timeout(state.timeout); - self - } - /// Start each WebSocket session at `quic`'s windows and stream count, the relay's /// `[quic]` section, as its QUIC and `tcp://` sessions do. Defaults to the `[quic]` /// defaults. @@ -1172,6 +1148,13 @@ impl IntoResponse for ServeGroup { mod tests { use super::*; + /// A listen config with `timeout`, where `None` waits forever. + fn listen(timeout: Option) -> moq_tokio::listen::Config { + let mut listen = moq_tokio::listen::Config::default(); + listen.timeout = Some(timeout.unwrap_or_default().into()); + listen + } + /// The state an acceptor built outside [`Web::serve`] hands a `tls://` session. #[cfg(all(feature = "websocket", any(unix, windows)))] fn acceptor_state() -> Arc { @@ -2246,7 +2229,7 @@ mod tests { let certificates = moq_tokio::tls::Certificates::from_pem(&std::fs::read(&cert).unwrap()).unwrap(); // No idle deadline, so a stalled connection outlives serving below. let mut web = Web::new(auth, cluster, certificates, config) - .with_timeout(None) + .with_listen(&listen(None)) .bind() .unwrap(); let incoming = rtmp.then(|| web.rtmp()); @@ -2408,7 +2391,8 @@ mod tests { let dir = TempDir::new().unwrap(); let (_, cert, _) = make_certs(&dir); let certificates = moq_tokio::tls::Certificates::from_pem(&std::fs::read(&cert).unwrap()).unwrap(); - let web = Web::new(auth, cluster, certificates, Config::default()).with_timeout(Some(std::time::Duration::MAX)); + let web = Web::new(auth, cluster, certificates, Config::default()) + .with_listen(&listen(Some(std::time::Duration::MAX))); assert_eq!(web.state.timeout, None); } @@ -2434,7 +2418,7 @@ mod tests { let (_, cert, _) = make_certs(&dir); let certificates = moq_tokio::tls::Certificates::from_pem(&std::fs::read(&cert).unwrap()).unwrap(); let web = Web::new(auth, cluster, certificates, config) - .with_timeout(Some(timeout)) + .with_listen(&listen(Some(timeout))) .bind() .unwrap(); let http = web.addrs().http.expect("the HTTP listener is configured"); @@ -2471,7 +2455,7 @@ mod tests { let (_, cert, _) = make_certs(&dir); let certificates = moq_tokio::tls::Certificates::from_pem(&std::fs::read(&cert).unwrap()).unwrap(); let web = Web::new(auth, cluster, certificates, config) - .with_timeout(Some(timeout)) + .with_listen(&listen(Some(timeout))) .bind() .unwrap(); let http = web.addrs().http.expect("the HTTP listener is configured"); @@ -2647,7 +2631,7 @@ mod tests { let (_, cert, _) = make_certs(&dir); let certificates = moq_tokio::tls::Certificates::from_pem(&std::fs::read(&cert).unwrap()).unwrap(); let web = Web::new(auth, cluster, certificates, config) - .with_timeout(None) + .with_listen(&listen(None)) .bind() .unwrap(); let http = web.addrs().http.expect("the HTTP listener is configured"); From d49b61ca1cd730461d6cf522b5ce4a09569af5a3 Mon Sep 17 00:00:00 2001 From: quest fixer Date: Fri, 6 Nov 2026 21:39:30 -0800 Subject: [PATCH 4/4] fix(quest): size session policy after its last child closes --- quest/m1/session-policy/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/quest/m1/session-policy/README.md b/quest/m1/session-policy/README.md index e8058f1b9d4..3b5bcac73d1 100644 --- a/quest/m1/session-policy/README.md +++ b/quest/m1/session-policy/README.md @@ -1,4 +1,4 @@ -# Session policy lives in moq-net +# [M] Session policy lives in moq-net ## Goal