diff --git a/doc/bin/relay/auth.md b/doc/bin/relay/auth.md index 438c3fe2102..c2da90a7e79 100644 --- a/doc/bin/relay/auth.md +++ b/doc/bin/relay/auth.md @@ -480,3 +480,7 @@ parent directory or an allowlist. dials such as [cluster links](/bin/relay/cluster#tls-links). It asks for a client certificate as the QUIC listener does, under the same `listen.tls.root`, so a peer on it presents a certificate or a token. +The HTTPS listener (`web.https`) accepts `tls://` too, with no setting: a +ClientHello that offers a moq ALPN is served the listen certificate and the +same client-certificate request, while every other connection keeps +`web.https.cert`. diff --git a/doc/bin/relay/cluster.md b/doc/bin/relay/cluster.md index 1f191c3644b..0c6dfab9e42 100644 --- a/doc/bin/relay/cluster.md +++ b/doc/bin/relay/cluster.md @@ -134,8 +134,9 @@ topology check; whatever generates the peer list applies the layout's rules. A link inside a region is rarely congested, so it can skip QUIC: a `tls://` peer URL dials qmux over TLS on TCP, verified with the same `connect.tls` settings as any other dial. The accepting relay serves it from its TCP -listener with TLS on. That listener applies the same client-certificate policy -as the QUIC listener, so a peer on it authenticates with its `connect.tls` +listener with TLS on, or from its HTTPS listener, which serves `tls://` with +the listen certificate. Both apply the same client-certificate policy as the +QUIC listener, so a peer on them authenticates with its `connect.tls` certificate against `listen.tls.root`, or with a token (`cluster.token`, `token`, or `?jwt=`). diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index df987c82f87..3710eba5161 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -201,7 +201,7 @@ start rather than being ignored. If io\_uring workers fail to start naming listen = "[::]:4443" # HTTP: fingerprint, announced, fetch, health. [web.https] -listen = "[::]:443" # HTTPS plus the WebSocket fallback. +listen = "[::]:443" # HTTPS, the WebSocket fallback, and tls:// with the listen certificate. cert = "cert.pem" # cert, key, and root need listen. Omit cert and key with [acme]. key = "key.pem" diff --git a/doc/bin/relay/index.md b/doc/bin/relay/index.md index 79af9ecd559..2551a588057 100644 --- a/doc/bin/relay/index.md +++ b/doc/bin/relay/index.md @@ -99,6 +99,10 @@ relay.with_web(web).run().await?; send its path in the SETUP. - To decide admissions yourself, leave `[auth]` empty and answer `relay.admissions()`; see [Authentication](/bin/relay/auth#in-process). +- To serve RTMP on the web ports, take `relay.rtmp()` and pass each accepted + stream to `moq_rtmp::accept_stream`. The HTTP and HTTPS listeners hand it + plaintext RTMP, and RTMPS (TLS with no ALPN) already decrypted; untaken, the + relay closes them. See [docs.rs/moq-relay](https://docs.rs/moq-relay) and [`rs/moq-relay/examples/embed.rs`](https://github.com/moq-dev/moq/blob/main/rs/moq-relay/examples/embed.rs). diff --git a/quest/m2/one-port/README.md b/quest/m2/one-port/README.md index 862fb1b9c6a..3fe9ba70501 100644 --- a/quest/m2/one-port/README.md +++ b/quest/m2/one-port/README.md @@ -6,7 +6,8 @@ A relay, or a binary embedding the gateway crates beside one, speaks everything it serves on one UDP port and one TCP port. UDP carries QUIC (WebTransport and raw moq), the WebRTC media path (STUN, DTLS, SRTP) for WHIP and WHEP, and SRT. TCP carries TLS-terminated -HTTP (WebSocket qmux, WHIP and WHEP signaling, HLS, ops), RTMP, and RTMPS. +HTTP (WebSocket qmux, WHIP and WHEP signaling, HLS, ops), `tls://` qmux, +RTMP, and RTMPS. Upstream delivers the demux and stacks that accept a fed socket or stream; `moq-relay` itself serves QUIC and HTTP on them, and an embedder such as moq.pro's edge wires WebRTC, RTMP, and SRT, which the @@ -81,22 +82,9 @@ and goes to WebRTC; any other is dropped and counted. Decided 2026-10-13: the responder binds its own port ([relay STUN](/doc/bin/relay/config.md#stun)), so the shared port never answers public Binding requests. -### TCP - -The 1935 listener downstream already peeks one byte to split RTMPS from RTMP. -The unified acceptor generalizes it: 0x16 is TLS, terminated here, then the -first decrypted byte is peeked again; 0x03 is an RTMP handshake and anything -else is HTTP, which goes to the axum router. A plaintext 0x03 is RTMP and a -plaintext ASCII method is HTTP. RTMPS clients send no ALPN, so ALPN cannot do -this. The acceptor yields classified connections; `moq-rtmp`'s -`accept_stream` already takes any `AsyncRead + AsyncWrite`, and -`axum_server::Server::from_listener` takes a listener that a channel of -pre-accepted streams can stand behind. - ## Required - [WebRTC on the shared socket](/quest/m2/one-port/rtc-feed.md) - `moq-rtc` serves WHIP and WHEP media from the WebRTC hook and pins ICE tuples -- [TCP acceptor](/quest/m2/one-port/tcp-demux.md) - one listener carries TLS-terminated HTTP, RTMP, and RTMPS - [SRT on the shared socket](/quest/m2/one-port/srt-demux.md) - moq-srt drives `srt-protocol` on demuxed packets and the flow table pins its 4-tuples ## Related diff --git a/quest/m2/one-port/tcp-demux.md b/quest/m2/one-port/tcp-demux.md deleted file mode 100644 index a86ee9225da..00000000000 --- a/quest/m2/one-port/tcp-demux.md +++ /dev/null @@ -1,61 +0,0 @@ -# [M] TCP acceptor - -## Goal - -`moq-relay`'s listener accepts TLS-terminated HTTP, RTMP, and RTMPS on one -TCP port and yields classified connections, so an embedder routes RTMP to -`moq-rtmp` and everything else reaches the axum router as today. The relay -binary has no RTMP consumer and gains none here: it refuses a classified -RTMP connection with a log line, and the embedder that carries RTMP today -(moq.pro's edge) is the consumer. - -## Plan - -`moq-relay/src/listener.rs` grows the acceptor: peek the first byte; 0x16 is -TLS, accepted with the served identity, then the first decrypted byte is -peeked; 0x03 is RTMP, anything else is HTTP. A plaintext 0x03 is RTMP; a -plaintext ASCII method is HTTP. The acceptor yields -`Accepted::{Http(stream), Rtmp(stream)}` where the stream is a boxed -`AsyncRead + AsyncWrite` that may already be TLS. HTTP connections are pushed -into a channel that stands behind `axum_server::Server::from_listener`, so -the router and its ALPN list are untouched. RTMP connections are the -embedder's to hand to `moq_rtmp::Server::accept_stream`; upstream nothing -consumes them, so the relay logs and drops them unless a consumer is wired. - -The accept loop never peeks or handshakes. Each accepted socket is spawned -onto its own task with a bounded read timeout; that task peeks, optionally -terminates TLS, and yields `Accepted`. An idle or slow client stalls only -its own task, so a peer that connects and sends nothing cannot block later -accepts. The peek is non-destructive on `TcpStream` and on the rustls -stream alike. Reuse the handshake bound `listen.timeout` already sets -(`moq_tokio::listen::Config::timeout`) rather than a new knob. - -The TLS arm also carries `tls://` qmux, served today by its own `listen.tcp` -listener with TLS. It negotiates a moq ALPN in the TLS handshake, so split -the decrypted arm on ALPN: a moq ALPN goes to the qmux server and an HTTP -ALPN to the router. A connection with no ALPN keeps the decrypted-byte -sniff, since RTMPS clients negotiate none (`moq-rtmp` documents an empty -ALPN list): `0x03` is RTMP, anything else is HTTP. - -Decided 2026-10-08: under one acceptor, the client-certificate request the -`tls://` listener makes (since -[moq-dev/moq#7358](https://github.com/moq-dev/moq/pull/7358)) sits on the moq-ALPN TLS -config only, so browsers and RTMPS clients on the same port are never asked -for one. ✅ Decided 2026-10-30 (maintainer): it is optional under CA roots and -required under pinned `tls.peers`, as on QUIC, so pinned mode still means -the listener knows who connected. Keep `tls://` client-certificate -verification and the `MtlsUnsupported` lift working. - -Keep the accepted `TcpStream`'s handle reachable after it is boxed: the -relay already captures socket stats at accept for qmux (`SocketStats` in -`web.rs`), and [WebSocket bitrate caps](/quest/m2/rate-websocket.md) set -socket buffer sizes on the same socket. - -Tests: an HTTP request, a WebSocket upgrade, a raw RTMP C0, an RTMPS C0 with -no ALPN, and a `tls://` qmux client against one listener each land in the -right arm. - -## Related - -- [UDP demux](/quest/m2/p2p/udp-demux.md) - the UDP half -- [WebSocket bitrate caps](/quest/m2/rate-websocket.md) - sizes the kernel buffers of the socket this acceptor boxes diff --git a/quest/m2/rate-websocket.md b/quest/m2/rate-websocket.md index bf8286db419..df3277e6a32 100644 --- a/quest/m2/rate-websocket.md +++ b/quest/m2/rate-websocket.md @@ -33,7 +33,8 @@ excess: `SO_SNDBUF` to about the burst, since paced reads cannot stop bytes the kernel already accepted and paced writes can queue a send buffer that drains faster than the cap. Fixing the size disables autotuning, which is the - point. The buffers bound the burst only; the sustained cap comes from the + point. The web acceptor keeps the accepted socket reachable through + `listener::Stream::tcp` (`rs/moq-relay/src/listener/demux.rs`) for this. The buffers bound the burst only; the sustained cap comes from the paced IO. Behind a TCP-terminating proxy, the buffers bound only the relay's side. - Reads stop at the bucket, so the client's TCP window closes; writes stop @@ -48,7 +49,3 @@ live session applies. ## Required - [QUIC caps](/quest/m2/rate-quic.md) - introduces the moq-auth cap type this reuses - -## Related - -- [TCP acceptor](/quest/m2/one-port/tcp-demux.md) - boxes the accepted stream, so it must keep the socket handle these buffer sizes need diff --git a/rs/moq-relay/src/listener.rs b/rs/moq-relay/src/listener.rs index 73be9f37515..979b9b123a6 100644 --- a/rs/moq-relay/src/listener.rs +++ b/rs/moq-relay/src/listener.rs @@ -14,7 +14,9 @@ //! back. Because the wrapper never yields an error upward, `axum_server`'s flat //! sleep is unreachable. +mod demux; mod idle; +pub(crate) use demux::{Classified, Stream, Tls, classify}; pub(crate) use idle::Admission; use std::{ @@ -41,6 +43,17 @@ impl Accepted { } } +#[cfg(all(test, unix, feature = "websocket"))] +impl Accepted { + /// A socket accepted outside [`Listener`], holding no admission. + pub(crate) fn new(stream: TcpStream) -> Self { + Self { + stream, + admission: None, + } + } +} + impl AsyncRead for Accepted { fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll> { Pin::new(&mut self.stream).poll_read(cx, buf) @@ -57,6 +70,11 @@ impl AsyncWrite for Accepted { Pin::new(&mut self.stream).poll_shutdown(cx) } } +impl AsRef for Accepted { + fn as_ref(&self) -> &TcpStream { + &self.stream + } +} #[cfg(unix)] impl std::os::fd::AsFd for Accepted { fn as_fd(&self) -> std::os::fd::BorrowedFd<'_> { diff --git a/rs/moq-relay/src/listener/demux.rs b/rs/moq-relay/src/listener/demux.rs new file mode 100644 index 00000000000..c138de2d414 --- /dev/null +++ b/rs/moq-relay/src/listener/demux.rs @@ -0,0 +1,181 @@ +//! Splits a public web connection into HTTP, RTMP, and `tls://` qmux by its first bytes. +//! +//! `axum_server` runs the acceptor on each connection's own task, so a peer that +//! connects and sends nothing stalls only itself, under the same idle deadline +//! that bounds its TLS handshake and first request. + +use std::{ + io, + pin::Pin, + task::{Context, Poll}, + time::Duration, +}; + +use axum_server::tls_rustls::RustlsConfig; +use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, ReadBuf}; +use tokio::net::TcpStream; +use tokio_rustls::{LazyConfigAcceptor, server::TlsStream}; + +use super::Accepted; + +/// The RTMP version byte (C0) an RTMP handshake opens with. +const RTMP_VERSION: u8 = 0x03; + +/// The bound on a TLS handshake, as `axum_server`'s own rustls acceptor sets. +const TLS_HANDSHAKE: Duration = Duration::from_secs(10); + +/// A classified connection: the protocol its first bytes named. +pub(crate) enum Classified { + Http(Stream), + Rtmp(Stream), + /// `tls://`: its ClientHello offered a moq ALPN, and the handshake negotiated one. + Qmux(Box>), +} + +/// The TLS a listener terminates: HTTP's config, and `tls://`'s when it is served. +#[derive(Clone)] +pub(crate) struct Tls { + pub(crate) http: RustlsConfig, + pub(crate) qmux: Option, +} + +/// Classify `accepted`, terminating TLS with `tls` when the listener serves it. +/// +/// A plaintext C0 is RTMP on either listener. On a TLS listener anything else is +/// handed to rustls, which refuses whatever is not a ClientHello (0x16), so +/// plaintext HTTP is still refused there. A ClientHello that offers a moq ALPN is +/// `tls://`, finished with the listen identity. Otherwise a negotiated ALPN names +/// HTTP; RTMPS clients offer none, so a connection without one is sniffed again +/// on its first decrypted byte. +pub(crate) async fn classify(accepted: Accepted, tls: Option<&Tls>) -> io::Result { + let mut first = [0u8; 1]; + if accepted.stream.peek(&mut first).await? == 0 { + return Err(io::ErrorKind::UnexpectedEof.into()); + } + let Some(tls) = tls.filter(|_| first[0] != RTMP_VERSION) else { + let stream = Stream { + transport: Transport::Plain(accepted), + peeked: None, + }; + return Ok(match first[0] { + RTMP_VERSION => Classified::Rtmp(stream), + _ => Classified::Http(stream), + }); + }; + + let handshake = async { + let start = LazyConfigAcceptor::new(rustls::server::Acceptor::default(), accepted).await?; + let alpns = start.client_hello().alpn(); + let qmux = tls + .qmux + .as_ref() + .filter(|qmux| alpns.is_some_and(|alpns| qmux.offered(alpns))); + let config = match qmux { + Some(qmux) => qmux.config(), + None => tls.http.get_inner(), + }; + Ok::<_, io::Error>((start.into_stream(config).await?, qmux.is_some())) + }; + let (mut tls, qmux) = kio::clock::timeout(TLS_HANDSHAKE, handshake) + .await + .map_err(|_| io::Error::from(io::ErrorKind::TimedOut))??; + if qmux { + return Ok(Classified::Qmux(Box::new(tls))); + } + if tls.get_ref().1.alpn_protocol().is_some() { + return Ok(Classified::Http(Stream { + transport: Transport::Tls(Box::new(tls)), + peeked: None, + })); + } + let first = tls.read_u8().await?; + let stream = Stream { + transport: Transport::Tls(Box::new(tls)), + peeked: Some(first), + }; + Ok(match first { + RTMP_VERSION => Classified::Rtmp(stream), + _ => Classified::Http(stream), + }) +} + +/// An accepted connection after classification, plaintext or with TLS terminated. +pub(crate) struct Stream { + transport: Transport, + /// The decrypted byte read to classify the connection, replayed to the first read. + peeked: Option, +} + +enum Transport { + Plain(Accepted), + Tls(Box>), +} + +impl Stream { + fn accepted(&self) -> &Accepted { + match &self.transport { + Transport::Plain(accepted) => accepted, + Transport::Tls(tls) => tls.get_ref().0, + } + } + + /// The accepted socket, for socket options and kernel statistics. + pub(crate) fn tcp(&self) -> &TcpStream { + &self.accepted().stream + } + + /// The TLS session, when the connection carried one. + pub(crate) fn tls(&self) -> Option<&rustls::ServerConnection> { + match &self.transport { + Transport::Plain(_) => None, + Transport::Tls(tls) => Some(tls.get_ref().1), + } + } + + pub(crate) fn admission(&self) -> Option { + self.accepted().admission.clone() + } +} + +impl AsyncRead for Stream { + fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll> { + if buf.remaining() > 0 + && let Some(byte) = self.peeked.take() + { + buf.put_slice(&[byte]); + return Poll::Ready(Ok(())); + } + match &mut self.transport { + Transport::Plain(accepted) => Pin::new(accepted).poll_read(cx, buf), + Transport::Tls(tls) => Pin::new(tls.as_mut()).poll_read(cx, buf), + } + } +} + +impl AsyncWrite for Stream { + fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { + match &mut self.transport { + Transport::Plain(accepted) => Pin::new(accepted).poll_write(cx, buf), + Transport::Tls(tls) => Pin::new(tls.as_mut()).poll_write(cx, buf), + } + } + fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + match &mut self.transport { + Transport::Plain(accepted) => Pin::new(accepted).poll_flush(cx), + Transport::Tls(tls) => Pin::new(tls.as_mut()).poll_flush(cx), + } + } + fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + match &mut self.transport { + Transport::Plain(accepted) => Pin::new(accepted).poll_shutdown(cx), + Transport::Tls(tls) => Pin::new(tls.as_mut()).poll_shutdown(cx), + } + } +} + +#[cfg(unix)] +impl std::os::fd::AsFd for Stream { + fn as_fd(&self) -> std::os::fd::BorrowedFd<'_> { + std::os::fd::AsFd::as_fd(self.tcp()) + } +} diff --git a/rs/moq-relay/src/relay.rs b/rs/moq-relay/src/relay.rs index 44a90d1e2d2..fef60393f97 100644 --- a/rs/moq-relay/src/relay.rs +++ b/rs/moq-relay/src/relay.rs @@ -329,7 +329,8 @@ impl Relay { .with_extensions(server_extensions) .with_implementation(implementation) .with_timeout(handshake_timeout) - .with_sessions(sessions.clone()); + .with_sessions(sessions.clone()) + .with_tls_acceptor(server.tls_acceptor()); #[cfg(feature = "acme")] let web = web.with_acme(acme.as_ref().map(crate::acme::Acme::resolver)); let web = web.bind()?; @@ -446,6 +447,12 @@ impl Relay { self.admissions.take() } + /// The RTMP and RTMPS connections the public web listeners classify, for the + /// embedder to serve; see [`web::Web::rtmp`]. Untaken, they are logged and closed. + pub fn rtmp(&mut self) -> web::rtmp::Incoming { + self.web.rtmp() + } + /// The shared cluster: the origin every session and peer publishes into. pub fn cluster(&self) -> &cluster::Cluster { &self.cluster diff --git a/rs/moq-relay/src/web.rs b/rs/moq-relay/src/web.rs index b37fcd8a42e..f3763c9130a 100644 --- a/rs/moq-relay/src/web.rs +++ b/rs/moq-relay/src/web.rs @@ -14,18 +14,15 @@ use axum::{ response::{Html, IntoResponse, Response}, routing::get, }; -use axum_server::{ - accept::{Accept, DefaultAcceptor}, - tls_rustls::{RustlsAcceptor, RustlsConfig}, -}; +use axum_server::{accept::Accept, tls_rustls::RustlsConfig}; use bytes::Bytes; use futures::{FutureExt, future::BoxFuture}; -use tokio::io::{AsyncRead, AsyncWrite}; -use tokio_rustls::server::TlsStream; use tower_service::Service; use crate::{auth, cluster}; +pub mod rtmp; + /// Configuration for the HTTP/HTTPS web server. #[derive(usage_rs::Args, Clone, Debug, serde::Deserialize, serde::Serialize)] #[usage(unknown_flags = "error", args_override_self = false)] @@ -161,19 +158,11 @@ pub(crate) struct WebState { pub(crate) cluster: cluster::Cluster, /// TLS certificate information served at `/certificate.sha256`. pub(crate) certificates: moq_tokio::tls::Certificates, - /// Relay-wide shutdown broadcast; WebSocket sessions drain with a GOAWAY - /// when it fires. Defaults to a handle that never fires. - #[cfg_attr( - not(feature = "websocket"), - allow(dead_code, reason = "only the `websocket` feature uses it") - )] + /// Relay-wide shutdown broadcast; WebSocket and `tls://` sessions drain with a + /// GOAWAY when it fires. Defaults to a handle that never fires. pub(crate) shutdown: crate::shutdown::Observer, - /// Live sessions on this node, so a WebSocket session is listed and nudged - /// like a QUIC one. - #[cfg_attr( - not(feature = "websocket"), - allow(dead_code, reason = "only the `websocket` feature uses it") - )] + /// Live sessions on this node, so a WebSocket or `tls://` session is listed and + /// nudged like a QUIC one. pub(crate) sessions: crate::session::Registry, /// How long a connection may sit with no complete request in flight (TLS /// handshake and partial headers included), and a WebSocket wait for its MoQ @@ -209,6 +198,9 @@ pub struct Web { https_tls: Option, /// The ACME certificate resolver HTTPS serves instead of `https.cert`/`https.key`. acme: Option>, + rtmp: rtmp::Handoff, + /// Serves `tls://` on the HTTPS listener, when the relay has a listen identity. + tls_stream: Option, addrs: Addrs, } @@ -241,6 +233,8 @@ impl Web { https_listener: None, https_tls: None, acme: None, + rtmp: rtmp::Handoff::default(), + tls_stream: None, addrs: Addrs::default(), } } @@ -298,6 +292,29 @@ impl Web { self.https_listener.as_ref() } + /// Take the RTMP and RTMPS connections the HTTP and HTTPS listeners classify. + /// + /// RTMP opens with a byte no HTTP request or TLS ClientHello starts with, and an + /// RTMPS client negotiates no ALPN, so both share the web ports. Until this is + /// called, or once the returned consumer is dropped, they are logged and closed. + /// A later call replaces the earlier consumer. + pub fn rtmp(&mut self) -> rtmp::Incoming { + let (handoff, incoming) = rtmp::Handoff::new(); + self.rtmp = handoff; + incoming + } + + /// Serve `tls://` on the HTTPS listener with `acceptor`, usually + /// [`moq_tokio::Server::tls_acceptor`]. + /// + /// A ClientHello that offers one of its moq ALPNs finishes with the listen TLS + /// identity and becomes a MoQ session. Every other connection keeps + /// `https.cert`, so browsers and RTMPS clients are unaffected. + pub fn with_tls_acceptor(mut self, acceptor: Option) -> Self { + self.tls_stream = acceptor; + self + } + /// The actual bound addresses after [`bind`](Self::bind) or [`crate::Relay::load`]. pub fn addrs(&self) -> Addrs { self.addrs @@ -456,6 +473,8 @@ impl Web { https_listener, https_tls, acme, + rtmp, + tls_stream, .. } = self.bind()?; let app = app @@ -463,6 +482,9 @@ impl Web { .layer(axum::middleware::from_fn(get_cors)) .into_make_service_with_connect_info::(); let ws = config.resolved_ws(); + // Connections still classifying outlive this future, and must not keep the + // RTMP consumer waiting once it returns. + let _rtmp = rtmp.close_on_drop(); // Each request is admitted on its own, so a connection may have as many in flight // as a session may have credentials verifying; the client queues the rest. let streams = u32::try_from(state.auth.verifications()).unwrap_or(u32::MAX); @@ -475,7 +497,7 @@ impl Web { timeout: state.timeout, activity: Some(state.shutdown.activity.clone()), }, - WebAcceptor::plain(ws), + WebAcceptor::plain(ws, rtmp.clone(), state.clone()), )?; server.http_builder().http2().max_concurrent_streams(streams); Some(server.serve(app.clone())) @@ -504,7 +526,15 @@ impl Web { timeout: state.timeout, activity: Some(state.shutdown.activity.clone()), }, - WebAcceptor::tls(rustls_config, ws), + WebAcceptor::tls( + crate::listener::Tls { + http: rustls_config, + qmux: tls_stream, + }, + ws, + rtmp, + state.clone(), + ), )?; server.http_builder().http2().max_concurrent_streams(streams); Some(server.serve(app)) @@ -685,152 +715,127 @@ pub(crate) struct ServerName(pub(crate) String); /// Accepts a connection on a public web listener. /// -/// Captures the socket, then hands the stream to `inner`: [`DefaultAcceptor`] for -/// `http://`, or [`RustlsAcceptor`] for `https://`, which also gives the connection -/// a peer certificate to report. Both listeners wrap this same acceptor, so the -/// capture cannot reach one and miss the other. +/// Classifies the connection (terminating TLS on `https://`), hands RTMP to +/// [`rtmp::Incoming`], serves `tls://`, and captures the socket of an HTTP one. +/// Both listeners wrap this same acceptor, so the capture cannot reach one and +/// miss the other. #[derive(Clone)] -struct WebAcceptor { - inner: A, +struct WebAcceptor { + /// Present on `https://`, which also gives the connection a peer certificate to report. + tls: Option, /// Whether a WebSocket route exists to use the handle. The capture holds a /// duplicated descriptor for the life of the connection, so doing it when /// nothing can read it would cost every API, HLS, and health-check connection a /// second descriptor for nothing. ws: bool, + rtmp: rtmp::Handoff, + /// What a `tls://` session runs against. + state: Arc, } -impl WebAcceptor { +impl WebAcceptor { /// Accepts plain HTTP, which has no handshake of its own to run. - fn plain(ws: bool) -> Self { + fn plain(ws: bool, rtmp: rtmp::Handoff, state: Arc) -> Self { Self { - inner: DefaultAcceptor::new(), + tls: None, ws, + rtmp, + state, } } -} -impl WebAcceptor> { /// Accepts HTTPS, running the TLS handshake and surfacing a verified peer /// certificate as [`MtlsPeer`]. /// /// When no client CA is configured the inner verifier is `NoClientAuth` and /// `peer_certificates()` always returns None, so the marker never appears. - fn tls(config: RustlsConfig, ws: bool) -> Self { + fn tls(tls: crate::listener::Tls, ws: bool, rtmp: rtmp::Handoff, state: Arc) -> Self { Self { - inner: RustlsAcceptor::new(config), + tls: Some(tls), ws, + rtmp, + state, } } -} - -/// HTTP protocol and peer metadata available after a connection finishes TLS. -/// -/// Which streams can present a client certificate is a property of the stream type, -/// not of the acceptor: only rustls has one to report, and a plain TCP connection -/// never does. -trait HttpStream { - /// The marker for this connection's verified peer certificate. - fn mtls_peer(&self) -> Option; - /// The SNI that selected the certificate, when the connection carried TLS. - fn server_name(&self) -> Option<&str> { - None - } - fn admission(&self) -> Option { - None - } - fn is_http2(&self) -> bool { - false - } - /// Whether this connection is an ACME TLS-ALPN-01 validation, which carries no HTTP. - fn acme_challenge(&self) -> bool { - false - } -} -impl HttpStream for tokio::net::TcpStream { - fn mtls_peer(&self) -> Option { - None - } -} - -impl HttpStream for crate::listener::Accepted { - fn mtls_peer(&self) -> Option { - None - } - fn admission(&self) -> Option { - self.admission.clone() - } -} - -impl HttpStream for TlsStream { - fn admission(&self) -> Option { - self.get_ref().0.admission() - } - fn is_http2(&self) -> bool { - self.get_ref().1.alpn_protocol() == Some(b"h2") - } - fn acme_challenge(&self) -> bool { - self.get_ref().1.alpn_protocol() == Some(b"acme-tls/1") - } - fn server_name(&self) -> Option<&str> { - self.get_ref().1.server_name() - } - fn mtls_peer(&self) -> Option { - self.get_ref() - .1 - .peer_certificates() - .filter(|certs| !certs.is_empty()) - .map(|certs| MtlsPeer(moq_tokio::tls::PeerIdentity::from_chain(certs.to_vec()))) + /// Run a `tls://` connection as a MoQ session on its own task, past the idle + /// deadline that bounds this acceptor. + fn serve_qmux(&self, stream: Box>) { + let Some(acceptor) = self.tls.as_ref().and_then(|tls| tls.qmux.clone()) else { + return; + }; + let state = self.state.clone(); + // The admission carries into the session, as a WebSocket upgrade's does. + let Some(active) = stream + .get_ref() + .0 + .admission + .clone() + .and_then(|admission| admission.take()) + .or_else(|| state.shutdown.activity.enter()) + else { + return; + }; + tokio::spawn(async move { + let Some(request) = state.shutdown.until_deadline(acceptor.accept(*stream, active)).await else { + return; + }; + let request = match request { + Ok(request) => request, + Err(err) => { + tracing::debug!(%err, "failed to accept tls:// on the HTTPS listener"); + return; + } + }; + crate::connection::Connection::new(request, state.cluster.clone(), state.auth.clone()) + .with_id(state.cluster.next_connection_id()) + .with_shutdown(state.shutdown.clone()) + .with_sessions(state.sessions.clone()) + .run() + .await; + }); } } -/// The connection [`WebAcceptor`] is handed: a byte stream, plus a descriptor to -/// read socket statistics from on platforms that expose them. -/// -/// The stream owns the accepted TCP socket, so the extra bound narrows nothing; -/// it exists because a `cfg` can't be written on a `where` clause directly. -#[cfg(any(target_os = "linux", target_os = "macos"))] -pub(crate) trait AcceptStream: AsyncRead + AsyncWrite + Unpin + Send + std::os::fd::AsFd + 'static {} -#[cfg(any(target_os = "linux", target_os = "macos"))] -impl AcceptStream for T {} - -/// As above, where kernel TCP statistics are unavailable. -#[cfg(not(any(target_os = "linux", target_os = "macos")))] -pub(crate) trait AcceptStream: AsyncRead + AsyncWrite + Unpin + Send + 'static {} -#[cfg(not(any(target_os = "linux", target_os = "macos")))] -impl AcceptStream for T {} - -impl Accept for WebAcceptor -where - I: AcceptStream, - S: Send + 'static, - A: Accept, - A::Stream: HttpStream + Send + 'static, - A::Service: Send + 'static, - A::Future: Send + 'static, -{ - type Stream = A::Stream; - type Service = SetConnectionExtensions; +impl Accept for WebAcceptor { + type Stream = crate::listener::Stream; + type Service = SetConnectionExtensions; type Future = BoxFuture<'static, std::io::Result<(Self::Stream, Self::Service)>>; - fn accept(&self, stream: I, service: S) -> Self::Future { - // Read the socket here, while it is still a plain `TcpStream`. Once TLS and - // then hyper's upgrade have wrapped it there is no descriptor left to find. - let socket = self.ws.then(|| socket_stats(&stream)).flatten(); - let inner = self.inner.accept(stream, service); + fn accept(&self, stream: crate::listener::Accepted, service: S) -> Self::Future { + let this = self.clone(); async move { - let (stream, service) = inner.await?; - if stream.acme_challenge() { + let stream = match crate::listener::classify(stream, this.tls.as_ref()).await? { + crate::listener::Classified::Http(stream) => stream, + crate::listener::Classified::Rtmp(stream) => { + this.rtmp.send(stream).await; + return Err(std::io::Error::other("handed off an RTMP connection")); + } + crate::listener::Classified::Qmux(stream) => { + this.serve_qmux(stream); + return Err(std::io::Error::other("handed off a tls:// connection")); + } + }; + let tls = stream.tls(); + if tls.is_some_and(|tls| tls.alpn_protocol() == Some(b"acme-tls/1")) { // The validator checked the certificate during the handshake; RFC 8737 has the server close. return Err(std::io::Error::other("answered an ACME TLS-ALPN-01 challenge")); } - let peer = stream.mtls_peer(); - let server_name = stream.server_name().map(|name| ServerName(name.to_owned())); + let peer = tls + .and_then(|tls| tls.peer_certificates()) + .filter(|certs| !certs.is_empty()) + .map(|certs| MtlsPeer(moq_tokio::tls::PeerIdentity::from_chain(certs.to_vec()))); + let server_name = tls + .and_then(|tls| tls.server_name()) + .map(|name| ServerName(name.to_owned())); let admission = stream.admission(); - if stream.is_http2() { + if tls.is_some_and(|tls| tls.alpn_protocol() == Some(b"h2")) { // Extended CONNECT is not enabled, so h2 cannot become a WebSocket. drop(admission.as_ref().and_then(crate::listener::Admission::take)); } + // Read the socket now: once hyper's upgrade has wrapped the stream there is + // no descriptor left to find. + let socket = this.ws.then(|| socket_stats(stream.tcp())).flatten(); Ok(( stream, SetConnectionExtensions { @@ -849,14 +854,14 @@ where /// The kernel's view of `stream`, when this platform exposes one and there is a /// qmux session that could use it. #[cfg(all(any(target_os = "linux", target_os = "macos"), feature = "websocket"))] -fn socket_stats(stream: &I) -> Option { +fn socket_stats(stream: &tokio::net::TcpStream) -> Option { qmux::TcpStats::new(stream) .ok() .map(|s| SocketStats(std::sync::Arc::new(s))) } #[cfg(not(all(any(target_os = "linux", target_os = "macos"), feature = "websocket")))] -fn socket_stats(_stream: &I) -> Option { +fn socket_stats(_stream: &tokio::net::TcpStream) -> Option { None } @@ -1229,9 +1234,25 @@ impl IntoResponse for ServeGroup { #[cfg(test)] mod tests { use super::*; + + /// The state an acceptor built outside [`Web::serve`] hands a `tls://` session. + #[cfg(all(feature = "websocket", any(unix, windows)))] + fn acceptor_state() -> Arc { + let cluster = cluster::Cluster::new(crate::cluster::Options::default()).unwrap(); + let cert = rcgen::generate_simple_self_signed(vec!["localhost".into()]).unwrap(); + let certificates = moq_tokio::tls::Certificates::from_pem(cert.cert.pem().as_bytes()).unwrap(); + Web::new( + crate::auth::Auth::refuse("test"), + cluster, + certificates, + Config::default(), + ) + .state + } use rcgen::{CertificateParams, KeyPair}; use std::io::Write; use tempfile::TempDir; + use tokio::io::{AsyncRead, AsyncWrite}; #[tokio::test(start_paused = true)] async fn authenticated_http_responses_are_private_and_cannot_be_sniffed() { @@ -1838,9 +1859,13 @@ mod tests { let addr = listener.local_addr().unwrap(); let connecting = tokio::spawn(async move { tokio::net::TcpStream::connect(addr).await.unwrap() }); let (accepted, _) = listener.accept().await.unwrap(); - let _client = connecting.await.unwrap(); + let mut client = connecting.await.unwrap(); + // The acceptor waits for a first byte to classify the connection. + tokio::io::AsyncWriteExt::write_all(&mut client, b"GET").await.unwrap(); - let (stream, mut service) = Accept::<_, EchoSocket>::accept(&WebAcceptor::plain(true), accepted, EchoSocket) + let acceptor = WebAcceptor::plain(true, rtmp::Handoff::default(), acceptor_state()); + let accepted = crate::listener::Accepted::new(accepted); + let (stream, mut service) = Accept::<_, EchoSocket>::accept(&acceptor, accepted, EchoSocket) .await .unwrap(); @@ -1883,11 +1908,12 @@ mod tests { let addr = listener.local_addr().unwrap(); let connecting = tokio::spawn(async move { tokio::net::TcpStream::connect(addr).await.unwrap() }); let (accepted, _) = listener.accept().await.unwrap(); - let _client = connecting.await.unwrap(); + let mut client = connecting.await.unwrap(); + tokio::io::AsyncWriteExt::write_all(&mut client, b"GET").await.unwrap(); - let (stream, service) = Accept::<_, ()>::accept(&WebAcceptor::plain(false), accepted, ()) - .await - .unwrap(); + let acceptor = WebAcceptor::plain(false, rtmp::Handoff::default(), acceptor_state()); + let accepted = crate::listener::Accepted::new(accepted); + let (stream, service) = Accept::<_, ()>::accept(&acceptor, accepted, ()).await.unwrap(); assert!( service.socket.is_none(), @@ -2089,6 +2115,267 @@ mod tests { serving.abort(); } + /// A `tls://` dial to the HTTPS port lands a MoQ session served with the listen + /// identity rather than `https.cert`, while the port keeps serving HTTPS. + #[cfg(feature = "quic")] + #[tokio::test] + async fn https_listener_serves_tls_qmux() { + use rustls::pki_types::{CertificateDer, pem::PemObject}; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + let dir = TempDir::new().unwrap(); + let (ca, cert, key) = make_certs(&dir); + let mut config = crate::Config::default(); + config.listen.bind = Some("127.0.0.1:0".parse().unwrap()); + config.listen.tls.generate = vec!["localhost".into()]; + config.web.https.listen = Some("127.0.0.1:0".parse().unwrap()); + config.web.https.cert = vec![cert]; + config.web.https.key = vec![key]; + config.auth.public = vec![moq_auth::Pattern::all()]; + let relay = crate::Relay::load(config).await.unwrap().with_signals(false); + let https = relay.web_addrs().https.unwrap(); + let ready = relay.ready(); + let trigger = relay.shutdown_trigger().clone(); + let running = tokio::spawn(relay.run()); + ready.wait().await.unwrap(); + + let url: url::Url = format!("tls://localhost:{}/room", https.port()).parse().unwrap(); + let dial = |tls: moq_tokio::tls::Connect| { + let mut config = moq_tokio::connect::Config::default(); + config.tls = tls; + let connection = config + .init(Default::default()) + .unwrap() + .with_reconnect(false) + .connect(url.clone()); + kio::clock::timeout(std::time::Duration::from_secs(10), connection.established()) + }; + let mut https_ca = moq_tokio::tls::Connect::default(); + https_ca.root = vec![ca.clone()]; + https_ca.system_roots = Some(false); + let refused = dial(https_ca).await.expect("the dial with the HTTPS CA timed out"); + assert!(refused.is_err(), "tls:// was served with https.cert"); + let mut insecure = moq_tokio::tls::Connect::default(); + insecure.insecure = Some(true); + let _session = dial(insecure) + .await + .expect("the tls:// dial timed out") + .expect("tls:// did not land a MoQ session"); + + let mut roots = rustls::RootCertStore::empty(); + for cert in CertificateDer::pem_slice_iter(&std::fs::read(&ca).unwrap()) { + roots.add(cert.unwrap()).unwrap(); + } + let mut client = rustls::ClientConfig::builder() + .with_root_certificates(roots) + .with_no_client_auth(); + client.alpn_protocols = vec![b"http/1.1".to_vec()]; + let tcp = tokio::net::TcpStream::connect(https).await.unwrap(); + let mut client = tokio_rustls::TlsConnector::from(Arc::new(client)) + .connect("localhost".try_into().unwrap(), tcp) + .await + .expect("HTTPS still serves https.cert"); + client + .write_all(b"GET /health HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n") + .await + .unwrap(); + let mut response = String::new(); + let _ = client.read_to_string(&mut response).await; + assert!(response.starts_with("HTTP/1.1 200"), "{response}"); + + trigger.start(); + running.await.unwrap().unwrap(); + } + + /// `tls://` on the HTTPS port applies the listen client-certificate policy, as the + /// `listen.tcp` `tls://` listener does: under pinned peers a dial without a pinned + /// certificate is refused. HTTPS on the same port is never asked for one. + #[cfg(feature = "quic")] + #[tokio::test] + async fn https_listener_verifies_tls_qmux_client_certificates() { + use rustls::pki_types::{CertificateDer, pem::PemObject}; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + let dir = TempDir::new().unwrap(); + let (ca, cert, key) = make_certs(&dir); + let mut config = crate::Config::default(); + config.listen.bind = Some("127.0.0.1:0".parse().unwrap()); + config.listen.tls.generate = vec!["localhost".into()]; + config.listen.tls.peers = Some(moq_tokio::tls::Peers::new()); + config.web.https.listen = Some("127.0.0.1:0".parse().unwrap()); + config.web.https.cert = vec![cert]; + config.web.https.key = vec![key]; + config.auth.public = vec![moq_auth::Pattern::all()]; + let relay = crate::Relay::load(config).await.unwrap().with_signals(false); + let https = relay.web_addrs().https.unwrap(); + let ready = relay.ready(); + let trigger = relay.shutdown_trigger().clone(); + let running = tokio::spawn(relay.run()); + ready.wait().await.unwrap(); + + let url: url::Url = format!("tls://localhost:{}/room", https.port()).parse().unwrap(); + let mut config = moq_tokio::connect::Config::default(); + config.tls.insecure = Some(true); + let connection = config + .init(Default::default()) + .unwrap() + .with_reconnect(false) + .connect(url); + let refused = kio::clock::timeout(std::time::Duration::from_secs(10), connection.established()) + .await + .expect("the tls:// dial timed out"); + assert!(refused.is_err(), "tls:// accepted a dial without a pinned certificate"); + + let mut roots = rustls::RootCertStore::empty(); + for cert in CertificateDer::pem_slice_iter(&std::fs::read(&ca).unwrap()) { + roots.add(cert.unwrap()).unwrap(); + } + let mut client = rustls::ClientConfig::builder() + .with_root_certificates(roots) + .with_no_client_auth(); + client.alpn_protocols = vec![b"http/1.1".to_vec()]; + let tcp = tokio::net::TcpStream::connect(https).await.unwrap(); + let mut client = tokio_rustls::TlsConnector::from(Arc::new(client)) + .connect("localhost".try_into().unwrap(), tcp) + .await + .expect("HTTPS asked for a client certificate"); + client + .write_all(b"GET /health HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n") + .await + .unwrap(); + let mut response = String::new(); + let _ = client.read_to_string(&mut response).await; + assert!(response.starts_with("HTTP/1.1 200"), "{response}"); + + trigger.start(); + running.await.unwrap().unwrap(); + } + + /// Each web listener splits its connections by their first bytes: HTTP reaches the + /// router, while plaintext RTMP and RTMPS (TLS with no ALPN) reach the RTMP + /// consumer with the classifying byte intact, until serving stops. Untaken, RTMP + /// is closed. + #[tokio::test] + async fn web_listeners_hand_rtmp_to_its_consumer() { + use rustls::pki_types::{CertificateDer, pem::PemObject}; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + trait Io: AsyncRead + AsyncWrite + Unpin + Send + 'static {} + impl Io for T {} + + let dir = TempDir::new().unwrap(); + let (ca, cert, key) = make_certs(&dir); + let mut roots = rustls::RootCertStore::empty(); + for cert in CertificateDer::pem_slice_iter(&std::fs::read(&ca).unwrap()) { + roots.add(cert.unwrap()).unwrap(); + } + let connector = |alpn: &[&[u8]]| { + let mut config = rustls::ClientConfig::builder() + .with_root_certificates(roots.clone()) + .with_no_client_auth(); + config.alpn_protocols = alpn.iter().map(|alpn| alpn.to_vec()).collect(); + tokio_rustls::TlsConnector::from(Arc::new(config)) + }; + + let serve = |rtmp: bool| { + let mut config = Config::default(); + config.http.listen = Some("127.0.0.1:0".parse().unwrap()); + config.https.listen = Some("127.0.0.1:0".parse().unwrap()); + config.https.cert = vec![cert.clone()]; + config.https.key = vec![key.clone()]; + let auth = crate::auth::Config { + public_subscribe: vec![moq_auth::Pattern::all()], + ..Default::default() + } + .init("test", &moq_tokio::tls::Connect::default(), false) + .unwrap(); + let cluster = cluster::Cluster::new(crate::cluster::Options::default()).unwrap(); + let certificates = moq_tokio::tls::Certificates::from_pem(&std::fs::read(&cert).unwrap()).unwrap(); + // No idle deadline, so a stalled connection outlives serving below. + let mut web = Web::new(auth, cluster, certificates, config) + .with_timeout(None) + .bind() + .unwrap(); + let incoming = rtmp.then(|| web.rtmp()); + let Addrs { + http: Some(http), + https: Some(https), + } = web.addrs() + else { + panic!("both listeners are configured"); + }; + let routes = web.routes(); + (tokio::spawn(web.serve(routes)), http, https, incoming) + }; + let connect = async |addr: net::SocketAddr, alpn: Option<&[&[u8]]>| -> Box { + let tcp = tokio::net::TcpStream::connect(addr).await.unwrap(); + match alpn { + None => Box::new(tcp) as Box, + Some(alpn) => Box::new( + connector(alpn) + .connect("localhost".try_into().unwrap(), tcp) + .await + .expect("TLS handshake"), + ), + } + }; + + let (serving, http, https, incoming) = serve(true); + let mut incoming = incoming.unwrap(); + for (addr, alpn) in [(http, None), (https, None), (https, Some(&[][..]))] { + let mut client = connect(addr, alpn).await; + client.write_all(&[3, 1, 2, 3]).await.unwrap(); + let mut rtmp = kio::clock::timeout(std::time::Duration::from_secs(10), incoming.accept()) + .await + .expect("RTMP reached its consumer") + .unwrap(); + assert_eq!( + rtmp.tcp().local_addr().unwrap(), + addr, + "the accepted socket stays reachable" + ); + let mut c0 = [0u8; 4]; + rtmp.read_exact(&mut c0).await.unwrap(); + assert_eq!(c0, [3, 1, 2, 3], "addr={addr} tls={}", alpn.is_some()); + } + // Silent, so it stays classifying; the requests below are accepted after it. + let _stalled = tokio::net::TcpStream::connect(http).await.unwrap(); + for (addr, alpn) in [ + (http, None), + (https, Some(&[b"http/1.1".as_slice()][..])), + (https, Some(&[][..])), + ] { + let mut client = connect(addr, alpn).await; + client + .write_all(b"GET /health HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n") + .await + .unwrap(); + let mut response = String::new(); + let _ = client.read_to_string(&mut response).await; + assert!( + response.starts_with("HTTP/1.1 200"), + "addr={addr} alpn={alpn:?}: {response}" + ); + } + // The connection still classifying does not keep the consumer open past serving. + serving.abort(); + let _ = serving.await; + let ended = kio::clock::timeout(std::time::Duration::from_secs(10), incoming.accept()).await; + assert!(matches!(ended, Ok(None)), "the consumer ends with serving"); + + let (serving, http, https, _) = serve(false); + for (addr, alpn) in [(http, None), (https, Some(&[][..]))] { + let mut client = connect(addr, alpn).await; + client.write_all(&[3]).await.unwrap(); + let mut rest = Vec::new(); + let read = kio::clock::timeout(std::time::Duration::from_secs(10), client.read_to_end(&mut rest)) + .await + .expect("an unserved RTMP connection is closed"); + assert!(rest.is_empty(), "{read:?}"); + } + serving.abort(); + } + /// An embedded route can read the accepted address with Axum's standard /// extractor, regardless of client-supplied proxy headers. #[tokio::test] @@ -2645,7 +2932,11 @@ mod tests { .await .unwrap(); assert!(activity.close().now_or_never().is_none(), "TLS was not counted"); - let acceptor = WebAcceptor::tls(RustlsConfig::from_config(tls), false); + let tls = crate::listener::Tls { + http: RustlsConfig::from_config(tls), + qmux: None, + }; + let acceptor = WebAcceptor::tls(tls, false, rtmp::Handoff::default(), acceptor_state()); let ((stream, _service), _client) = socket_step(async { tokio::try_join!( acceptor.accept(accepted, ()), @@ -2654,7 +2945,7 @@ mod tests { }) .await .unwrap(); - assert!(stream.is_http2()); + assert_eq!(stream.tls().unwrap().alpn_protocol(), Some(&b"h2"[..])); assert!( activity.close().now_or_never().is_some(), "h2 awaited request headers before releasing its admission" diff --git a/rs/moq-relay/src/web/rtmp.rs b/rs/moq-relay/src/web/rtmp.rs new file mode 100644 index 00000000000..b0a6e5da99d --- /dev/null +++ b/rs/moq-relay/src/web/rtmp.rs @@ -0,0 +1,100 @@ +//! RTMP and RTMPS connections the public web listeners classified, for an embedder to serve. +//! +//! The relay speaks no RTMP itself: without a consumer taken from +//! [`Web::rtmp`](super::Web::rtmp) it logs and closes them. + +use std::{ + io, + pin::Pin, + sync::{Arc, Mutex, PoisonError}, + task::{Context, Poll}, +}; + +use tokio::io::{AsyncRead, AsyncWrite, ReadBuf}; +use tokio::sync::mpsc; + +/// The RTMP connections accepted on the public web listeners. +pub struct Incoming(mpsc::Receiver); + +impl Incoming { + /// The next RTMP connection, or `None` once the web listeners stop. + pub async fn accept(&mut self) -> Option { + self.0.recv().await + } +} + +/// An RTMP connection, plaintext or with TLS already terminated, for `moq_rtmp::accept_stream`. +pub struct Stream(crate::listener::Stream); + +impl Stream { + /// The accepted TCP socket, for socket options and statistics. + pub fn tcp(&self) -> &tokio::net::TcpStream { + self.0.tcp() + } +} + +impl AsyncRead for Stream { + fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll> { + Pin::new(&mut self.0).poll_read(cx, buf) + } +} + +impl AsyncWrite for Stream { + fn poll_write(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8]) -> Poll> { + Pin::new(&mut self.0).poll_write(cx, buf) + } + fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.0).poll_flush(cx) + } + fn poll_shutdown(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { + Pin::new(&mut self.0).poll_shutdown(cx) + } +} + +/// Where the listeners send classified RTMP connections; empty refuses them. +/// +/// Connections still classifying hold clones of this, so the sender sits behind +/// one shared slot that [`Close`] empties when serving stops, ending +/// [`Incoming::accept`] regardless of them. +#[derive(Clone, Default)] +pub(crate) struct Handoff(Arc>>>); + +impl Handoff { + /// A handoff and the consumer it feeds. + /// + /// One slot: each connection waits on its own task for the embedder to take it, + /// which the listener's idle deadline bounds like any other stalled handshake. + pub(crate) fn new() -> (Self, Incoming) { + let (tx, rx) = mpsc::channel(1); + (Self(Arc::new(Mutex::new(Some(tx)))), Incoming(rx)) + } + + /// Refuse every later connection once the returned guard drops. + pub(crate) fn close_on_drop(&self) -> Close { + Close(self.clone()) + } + + /// Give `stream` to the consumer, or close it when there is none. + pub(crate) async fn send(&self, stream: crate::listener::Stream) { + // The embedder owns an RTMP session's lifetime, so the drain stops counting it here. + drop(stream.admission().as_ref().and_then(crate::listener::Admission::take)); + let peer = stream.tcp().peer_addr().ok(); + let tx = self.0.lock().unwrap_or_else(PoisonError::into_inner).clone(); + let refused = match tx { + Some(tx) => tx.send(Stream(stream)).await.is_err(), + None => true, + }; + if refused { + tracing::debug!(?peer, "closed an RTMP connection: nothing serves RTMP here"); + } + } +} + +/// Empties a [`Handoff`] when dropped. +pub(crate) struct Close(Handoff); + +impl Drop for Close { + fn drop(&mut self) { + self.0.0.lock().unwrap_or_else(PoisonError::into_inner).take(); + } +} diff --git a/rs/moq-tokio/Cargo.toml b/rs/moq-tokio/Cargo.toml index cba109df9a1..fe5b46194a1 100644 --- a/rs/moq-tokio/Cargo.toml +++ b/rs/moq-tokio/Cargo.toml @@ -47,7 +47,7 @@ mdns = ["dep:mdns-sd", "dep:hmac", "dep:sha2", "dep:subtle"] jemalloc = ["dep:tikv-jemallocator", "dep:tikv-jemalloc-ctl"] websocket = ["_transport", "dep:qmux", "watch", "dep:dns-lookup"] # Qmux over TCP: plaintext `tcp://`, and `tls://` once a crypto provider is on. Server-side and client-side. -tcp = ["_transport", "dep:qmux", "dep:dns-lookup"] +tcp = ["_transport", "dep:qmux", "dep:dns-lookup", "dep:tokio-rustls"] # Unix-domain-socket qmux transport (`unix://`), unix-only. Adds peer-credential # inspection on accept. Pulls in `tcp` (shares qmux's stream transport). uds = ["tcp", "qmux/uds"] @@ -101,6 +101,8 @@ tikv-jemalloc-ctl = { version = "0.7", optional = true } tikv-jemallocator = { version = "0.7", features = ["profiling", "unprefixed_malloc_on_supported_platforms"], optional = true } time = { workspace = true } tokio = { workspace = true, features = ["full"] } +# The TLS stream `tcp::TlsAcceptor` accepts from a listener sharing its port. +tokio-rustls = { workspace = true, optional = true } tokio-util = { workspace = true, features = ["io"], optional = true } toml = { workspace = true } tracing = "0.1" diff --git a/rs/moq-tokio/src/server.rs b/rs/moq-tokio/src/server.rs index d01c16ff9cd..5d8b847792b 100644 --- a/rs/moq-tokio/src/server.rs +++ b/rs/moq-tokio/src/server.rs @@ -247,6 +247,9 @@ pub struct Server { accept: FuturesUnordered>>, #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] streams: StreamListeners, + /// The `tls://` config another listener can serve, when a listen identity is loaded. + #[cfg(all(feature = "tcp", feature = "_certs"))] + tls_stream: Option>, #[cfg(feature = "iroh")] iroh: Option, #[cfg(feature = "quic")] @@ -414,6 +417,12 @@ impl Server { unix_allow, ); + #[cfg(all(feature = "tcp", feature = "_certs"))] + let tls_stream = tls + .map(|tls| TcpTls::new(&config, &versions, tls)) + .transpose()? + .map(|tls| tls.config); + // The endpoint was bound before this server's versions were known, so it // accepts nothing until they name its ALPNs. #[cfg(feature = "iroh")] @@ -450,6 +459,8 @@ impl Server { timeout, #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] streams, + #[cfg(all(feature = "tcp", feature = "_certs"))] + tls_stream, #[cfg(feature = "iroh")] iroh, #[cfg(feature = "quic")] @@ -516,6 +527,18 @@ impl Server { Ok(()) } + /// Serve `tls://` on connections another listener accepts, such as an HTTPS + /// server that shares its port with it (see [`crate::tcp::TlsAcceptor`]). + /// + /// `None` when this server loaded no listen TLS identity, as a stream-only + /// server without `--listen-tcp-tls` does not. + #[cfg(all(feature = "tcp", feature = "_certs"))] + pub fn tls_acceptor(&self) -> Option { + let config = self.tls_stream.clone()?; + let server = self.moq.clone().with_versions(self.streams.versions.clone()); + Some(crate::tcp::TlsAcceptor::new(config, server, self.timeout)) + } + /// A live handle to the certificates this server is serving. /// /// Use it to publish the SHA-256 fingerprints of a generated certificate at @@ -1164,7 +1187,7 @@ impl Deadline { all(feature = "uds", unix), feature = "webrtc" ))] -async fn within( +pub(crate) async fn within( deadline: Option, handshake: impl Future>, ) -> crate::Result @@ -1475,33 +1498,8 @@ fn spawn_tcp_loop( handshakes.spawn(crate::panic::isolate(async move { match within(deadline, pending).await { Ok((session, remote)) => { - // Set only on `tls://`: the SNI, and the client certificate its - // verifier already accepted. - let server_name = session.server_name().map(str::to_owned); - let identity = session - .peer_certificates() - .map(|chain| crate::tls::PeerIdentity::from_chain(chain.to_vec())); - let link = Link { - remote: Some(remote), - local, - server_name: server_name.clone(), - ..Default::default() - }; - stream_request( - session, - &setup, - tx, - Incoming { - transport: Transport::Tcp, - link, - // Like raw QUIC, the SNI is the only authority a `tls://` client offers. - authority: server_name, - identity, - deadline, - active, - }, - ) - .await + let incoming = Incoming::tcp(&session, Some(remote), local, deadline, active); + stream_request(session, &setup, tx, incoming).await; } // Per-connection: a failed `accept(2)` is the listener's own to // classify and pace, and never surfaces here. @@ -1561,7 +1559,7 @@ fn spawn_unix_loop( } #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] -struct Incoming { +pub(crate) struct Incoming { transport: Transport, link: Link, authority: Option, @@ -1570,6 +1568,45 @@ struct Incoming { active: moq_net::server::Active, } +#[cfg(feature = "tcp")] +impl Incoming { + /// A TCP session's facts. On `tls://` these are the SNI, and the client + /// certificate its verifier already accepted. + pub(crate) fn tcp( + session: &qmux::Session, + remote: Option, + local: Option, + deadline: Option, + active: moq_net::server::Active, + ) -> Self { + let server_name = session.server_name().map(str::to_owned); + let identity = session + .peer_certificates() + .map(|chain| crate::tls::PeerIdentity::from_chain(chain.to_vec())); + Self { + transport: Transport::Tcp, + link: Link { + remote, + local, + server_name: server_name.clone(), + ..Default::default() + }, + // Like raw QUIC, the SNI is the only authority a `tls://` client offers. + authority: server_name, + identity, + deadline, + active, + } + } + + /// Report the ALPN the connection's TLS handshake negotiated. + #[cfg(feature = "_certs")] + pub(crate) fn with_alpn(mut self, alpn: Option) -> Self { + self.link.alpn = alpn; + self + } +} + /// Read SETUP in the connection task and forward the resulting request. #[cfg(any(feature = "tcp", all(feature = "uds", unix)))] async fn stream_request( @@ -1578,6 +1615,21 @@ async fn stream_request( tx: tokio::sync::mpsc::Sender, incoming: Incoming, ) { + match accept_stream(&setup.server, session, incoming).await { + Ok(request) => { + let _ = tx.send(request).await; + } + Err(err) => tracing::debug!(%err, "stream SETUP handshake failed"), + } +} + +/// Read SETUP on a stream session, which carries its path in SETUP rather than a URL. +#[cfg(any(feature = "tcp", all(feature = "uds", unix)))] +pub(crate) async fn accept_stream( + server: &moq_net::Server, + session: qmux::Session, + incoming: Incoming, +) -> crate::Result { let Incoming { transport, link, @@ -1586,24 +1638,19 @@ async fn stream_request( deadline, active, } = incoming; - match accept_request(&setup.server, session, deadline).await { - Ok((request, deadline)) => { - let request = Request { - transport, - url: None, - authority, - identity, - link, - handshake: Box::new(request), - #[cfg(feature = "quic")] - quic: None, - deadline, - active, - }; - let _ = tx.send(request).await; - } - Err(err) => tracing::debug!(%err, "stream SETUP handshake failed"), - } + let (request, deadline) = accept_request(server, session, deadline).await?; + Ok(Request { + transport, + url: None, + authority, + identity, + link, + handshake: Box::new(request), + #[cfg(feature = "quic")] + quic: None, + deadline, + active, + }) } /// The transport-level facts a backend captures while accepting a connection, before the diff --git a/rs/moq-tokio/src/tcp.rs b/rs/moq-tokio/src/tcp.rs index a48e68997b7..ee2db381109 100644 --- a/rs/moq-tokio/src/tcp.rs +++ b/rs/moq-tokio/src/tcp.rs @@ -233,6 +233,89 @@ async fn connect_addrs( .await } +/// Serves `tls://` on TCP connections another listener accepted, such as a web +/// server that reads each ClientHello so HTTPS and `tls://` share its port. +/// +/// From [`crate::Server::tls_acceptor`]: the listen TLS identity, offering the +/// same `qmux-01.` ALPNs as a [`Listener`] serving `tls://`. +#[cfg(feature = "_certs")] +#[derive(Clone)] +pub struct TlsAcceptor { + config: std::sync::Arc, + qmux: qmux::tls::Server, + server: moq_net::Server, + timeout: Option, +} + +#[cfg(feature = "_certs")] +impl TlsAcceptor { + pub(crate) fn new( + config: std::sync::Arc, + server: moq_net::Server, + timeout: Option, + ) -> Self { + let qmux = qmux::tls::Server::new(config.clone()).with_session_config(crate::qmux_config()); + Self { + config, + qmux, + server, + timeout, + } + } + + /// The config to finish the TLS handshake with: the listen identity and its moq ALPNs. + pub fn config(&self) -> std::sync::Arc { + self.config.clone() + } + + /// Whether a ClientHello offering `alpns` asks for `tls://`: it names one of + /// [`config`](Self::config)'s ALPNs. + pub fn offered<'a>(&self, mut alpns: impl Iterator) -> bool { + alpns.any(|offered| self.config.alpn_protocols.iter().any(|alpn| alpn == offered)) + } + + /// Run the qmux handshake and read SETUP on a connection whose TLS handshake + /// finished with [`config`](Self::config). + /// + /// The negotiated ALPN picks the qmux wire format. The request reports the + /// socket's addresses, the SNI, the negotiated ALPN, and a client certificate + /// the config's verifier accepted, as a [`Listener`] serving `tls://` does. + /// The listen timeout bounds the handshake from this call, and `active` keeps + /// the connection counted toward the drain. + pub async fn accept( + &self, + stream: tokio_rustls::server::TlsStream, + active: moq_net::server::Active, + ) -> crate::Result + where + T: tokio::io::AsyncRead + tokio::io::AsyncWrite + AsRef + Send + Unpin + 'static, + { + crate::runtime::scope(async move { + let deadline = self.timeout.and_then(|timeout| kio::clock::now().checked_add(timeout)); + let tcp: &tokio::net::TcpStream = stream.get_ref().0.as_ref(); + let remote = tcp.peer_addr().ok(); + let local = tcp.local_addr().ok(); + let stats = qmux::tcp::adopt(tcp).map_err(Error::Io)?; + let alpn = stream + .get_ref() + .1 + .alpn_protocol() + .and_then(|alpn| std::str::from_utf8(alpn).ok()) + .map(str::to_owned); + let handshake = async { + self.qmux + .accept_established(stream, stats) + .await + .map_err(|err| Error::Accept(crate::error::message(err))) + }; + let session = crate::server::within(deadline, handshake).await?; + let incoming = crate::server::Incoming::tcp(&session, remote, local, deadline, active).with_alpn(alpn); + crate::server::accept_stream(&self.server, session, incoming).await + }) + .await + } +} + /// Listens for incoming qmux connections on a TCP port, plaintext or TLS. pub struct Listener { listener: tokio::net::TcpListener, diff --git a/rs/qmux/src/tcp.rs b/rs/qmux/src/tcp.rs index 66fb5e41f9f..da82d4b551d 100644 --- a/rs/qmux/src/tcp.rs +++ b/rs/qmux/src/tcp.rs @@ -30,7 +30,10 @@ async fn finish(stream: TcpStream, config: crate::Config, is_server: bool) -> Re /// Disables Nagle: the session already batches records and flushes once per batch, /// so Nagle has nothing left to coalesce and would only hold a batch's tail until /// the peer's delayed ACK. -pub(crate) fn adopt(stream: &TcpStream) -> std::io::Result> { +/// +/// For a socket whose session starts elsewhere, such as +/// `tls::Server::accept_established`. +pub fn adopt(stream: &TcpStream) -> std::io::Result> { stream.set_nodelay(true)?; // The kernel already tracks this socket's RTT (and, on Linux, its delivery diff --git a/rs/qmux/src/tls.rs b/rs/qmux/src/tls.rs index 5416ae3bebb..fa507621b73 100644 --- a/rs/qmux/src/tls.rs +++ b/rs/qmux/src/tls.rs @@ -157,18 +157,34 @@ impl Server { let socket_stats = crate::tcp::adopt(&stream)?; let acceptor = TlsAcceptor::from(self.config.clone()); let tls_stream = acceptor.accept(stream).await?; + self.accept_established(tls_stream, socket_stats).await + } - let negotiated = tls_stream.get_ref().1.alpn_protocol(); - let negotiated_str = negotiated.and_then(|a| std::str::from_utf8(a).ok()); - tracing::debug!(?negotiated_str, "TLS accepted, negotiated ALPN"); - - let (version, protocol) = alpn::parse(negotiated_str)?; + /// Accept a session over a TLS stream whose handshake the caller already ran, + /// such as a listener that reads the ClientHello to pick a config for its port. + /// + /// The ALPN that handshake negotiated names the wire-format version, and the + /// session reports the client's certificate chain and SNI as an + /// [`accept`](Self::accept)ed one does. `socket_stats` reports on the TCP + /// connection underneath; take it before TLS wraps the socket, as + /// [`accept`](Self::accept) does. + pub async fn accept_established( + &self, + stream: tokio_rustls::server::TlsStream, + socket_stats: Option, + ) -> Result + where + T: tokio::io::AsyncRead + tokio::io::AsyncWrite + Send + Unpin + 'static, + { + let conn = stream.get_ref().1; + let negotiated = conn.alpn_protocol().and_then(|a| std::str::from_utf8(a).ok()); + tracing::debug!(?negotiated, "TLS accepted, negotiated ALPN"); + let (version, protocol) = alpn::parse(negotiated)?; tracing::debug!(?version, ?protocol, "parsed ALPN"); let session_config = self.session.clone().with_negotiated(version, protocol); - let conn = tls_stream.get_ref().1; let handshake = Handshake::new(conn.peer_certificates(), conn.server_name()); - let mut transport = Stream::new(tls_stream); + let mut transport = Stream::new(stream); if let Some(stats) = socket_stats { transport = transport.with_socket_stats(stats); }