diff --git a/quest/m1/quic/README.md b/quest/m1/quic/README.md index 06bd6a53d82..90542b510a6 100644 --- a/quest/m1/quic/README.md +++ b/quest/m1/quic/README.md @@ -62,7 +62,6 @@ cache limits remain separate; bitrate pacing composes with this memory cap. ## Required -- [Retain connection credit](/quest/m1/quic/receive-credit.md) - native receivers own connection credit independently of stream reads - [Bound message assembly](/quest/m1/quic/receive-memory.md) - complete headers and frames return credit, and receiver priority cancellation restores progress - [Reliable stream reset](/quest/m1/quic/reliable-reset.md) - `RESET_STREAM_AT`, so a reset WebTransport stream still delivers its header diff --git a/quest/m1/quic/receive-credit.md b/quest/m1/quic/receive-credit.md deleted file mode 100644 index 79d7c589ef4..00000000000 --- a/quest/m1/quic/receive-credit.md +++ /dev/null @@ -1,54 +0,0 @@ -# [L] Return connection credit when the receiver releases bytes - -## Goal - -Native receivers can read QUIC stream bytes without returning connection -credit until their consumer releases those bytes. The same ownership works -through the tokio and io_uring adapters, including cancellation and teardown. -Stream credit remains independent, so a finite stream window does not force -an entire application message to fit before more bytes arrive. - -## Plan - -Decided in the 2026-10-10 receive-memory interview: use `moq-quic`, not a -new feature in the frozen moq-dev/noq fork. This quest -provides the transport primitive; [message assembly](/quest/m1/quic/receive-memory.md) -activates it with the recovery policy needed for progress. - -Today `Chunks::finalize` returns both kinds of credit before the parser has -a complete message. Separate returning borrowed transport state from -returning connection credit. Carry ownership through the native adapters -without copying payloads solely to track credit. Prefer an owned handle or -explicit consumption accounting over callbacks; keep the surface limited to -what the message-assembly consumer needs. - -Replenish `MAX_STREAM_DATA` on reads. Defer `MAX_DATA` until the owner releases -the bytes, exactly once, even after FIN, STOP_SENDING, RESET_STREAM, or stream -state reclamation. Unread discarded bytes, late data on stopped streams, -reset final-size gaps, and already-read retained bytes must not leak or -return the same credit twice. Releasing useful credit must eventually wake -a blocked peer even below the normal window-update threshold; do not depend -on optional DATA_BLOCKED frames. Runtime window shrink debt still applies. - -Expose enough credit accounting and events for MoQ to detect assembly -pressure. QUIC performs cancellation but does not choose application victims. -A sender's write being blocked, or its data being acknowledged, does not -prove receiver assembly deadlock. No sender-side shedding is added. - -Tests use the transport simulator and controlled time: stream credit advances -while connection credit stays held, explicit release resumes the peer, and -all terminal paths account once even when retained bytes outlive the stream. -Cover a useful release smaller than the update threshold and compose with -reliable reset when available. Wire adapter tests into CI and measure the -accounting cost across stream counts and payload sizes. - -Public API: a native receive-credit ownership/observation seam, finalized -with its consumer. Wire: existing QUIC flow-control and cancellation frames; -no new MoQ message or negotiation. Browser and iroh receivers do not gain a -capability their transports cannot provide. - -## Related - -- [Bound message assembly](/quest/m1/quic/receive-memory.md) - retains connection credit until a complete message is handed off -- [Reliable stream reset](/quest/m1/quic/reliable-reset.md) - compose retained credit with reliable prefixes and final-size accounting -- [Relay peers get wider limits](/quest/m1/quic/peer-limits.md) - runtime receive-window changes preserve retained-credit accounting diff --git a/quest/m1/quic/receive-memory.md b/quest/m1/quic/receive-memory.md index f027c782966..5e2bd90a1b7 100644 --- a/quest/m1/quic/receive-memory.md +++ b/quest/m1/quic/receive-memory.md @@ -18,6 +18,17 @@ peer-visible QUIC flow control instead of a separate header quota error. The native transport primitive is a separate prerequisite; do not activate whole-message retention without its recovery policy in this quest. +That primitive landed as a provisional seam to finalize here: both native +drivers' receive streams have a copying `poll_read_held` that yields a +`Credit` for exactly the bytes read. Dropping a `Credit` returns it, so a +reader buffer keeps one merged `Credit` and drops `split(n)` for each +complete unit; read-ahead stays in the remainder. io_uring's `poll_closed` +read-ahead keeps its credit until the application reads it. Both drivers' +`Connection::stats().credit` reports held and still-available credit. +`poll_read_held` stays separate until this quest folds it into `poll_read`. +moq-net's transport trait and the WebTransport wrappers do not carry it yet; +reshape any of it to fit the consumer. + Hold connection credit through reader buffers, type/length prefixes, headers, extensions, and frame assembly. Attribute read-ahead bytes to the next unit rather than releasing them with the completed one. Release on complete-unit @@ -86,10 +97,6 @@ Browser/iroh receive-side guarantees and completed-cache limits are outside this quest. Update existing configuration and transport docs inline; no separate guide quest is needed. -## Required - -- [Retain connection credit](/quest/m1/quic/receive-credit.md) - core and native adapters support separate byte ownership and stream-credit release - ## Related - [Scope track priority](/quest/m1/track-priority-scope.md) - broadcast fairness domains and subscription priority semantics diff --git a/rs/moq-quic/src/connection/mod.rs b/rs/moq-quic/src/connection/mod.rs index e828b460551..7580f622668 100644 --- a/rs/moq-quic/src/connection/mod.rs +++ b/rs/moq-quic/src/connection/mod.rs @@ -76,7 +76,22 @@ use spaces::Retransmits; use spaces::{PacketNumberFilter, PacketSpace, SendableFrames, SentPacket, ThinRetransmits}; mod stats; -pub use stats::{ConnectionStats, FrameStats, PathStats, UdpStats}; + +/// Identifies one connection, so state handed out by it cannot be returned to another. +#[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] +pub(crate) struct ConnectionTag(u64); + +impl ConnectionTag { + /// No connection, for state that has not come from one yet. + pub(crate) const NONE: Self = Self(0); + + /// A tag no other connection in this process has. + pub(crate) fn next() -> Self { + static NEXT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(1); + Self(NEXT.fetch_add(1, std::sync::atomic::Ordering::Relaxed)) + } +} +pub use stats::{ConnectionStats, CreditStats, FrameStats, PathStats, UdpStats}; mod streams; #[cfg(fuzzing)] @@ -84,7 +99,7 @@ pub use streams::StreamsState; #[cfg(not(fuzzing))] use streams::StreamsState; pub use streams::{ - Chunks, ClosedStream, FinishError, ReadError, ReadableError, RecvStream, SendStream, + Chunks, ClosedStream, Credit, FinishError, ReadError, ReadableError, RecvStream, SendStream, ShouldTransmit, StreamEvent, Streams, WriteError, Written, }; @@ -1358,10 +1373,24 @@ impl Connection { stats.path.cwnd = self.path.congestion.window(); stats.path.bandwidth_estimate = self.path.congestion.metrics().bandwidth_estimate; stats.path.current_mtu = self.path.mtud.current_mtu(); + stats.credit = self.streams.credit_stats(); stats } + /// Return connection flow control credit taken by [`Chunks::hold()`] to the peer. + /// + /// If [`ShouldTransmit::should_transmit()`] returns `true`, call + /// [`Self::poll_transmit()`] soon so the peer learns of the larger window. + /// + /// # Panics + /// If `credit` came from another connection. + pub fn release(&mut self, credit: Credit) -> ShouldTransmit { + let transmit = self.streams.release(credit); + self.spaces[SpaceId::Data].pending.max_data |= transmit.should_transmit(); + transmit + } + /// Ping the remote endpoint /// /// Causes an ACK-eliciting packet to be transmitted. diff --git a/rs/moq-quic/src/connection/stats.rs b/rs/moq-quic/src/connection/stats.rs index b62cede34eb..1a6dab4ba3b 100644 --- a/rs/moq-quic/src/connection/stats.rs +++ b/rs/moq-quic/src/connection/stats.rs @@ -179,4 +179,17 @@ pub struct ConnectionStats { pub frame_rx: FrameStats, /// Statistics related to the current transmission path pub path: PathStats, + /// Connection-level receive flow control + pub credit: CreditStats, +} + +/// Connection-level receive flow control, from the receiver's side +#[derive(Debug, Default, Copy, Clone, PartialEq, Eq)] +#[non_exhaustive] +pub struct CreditStats { + /// Bytes read with [`Chunks::hold()`](crate::Chunks::hold) whose credit is not released yet + pub held: u64, + /// Bytes the peer may still send before it needs more credit, including credit not yet + /// advertised + pub available: u64, } diff --git a/rs/moq-quic/src/connection/streams/mod.rs b/rs/moq-quic/src/connection/streams/mod.rs index 745555db11b..b9ef4c3dec0 100644 --- a/rs/moq-quic/src/connection/streams/mod.rs +++ b/rs/moq-quic/src/connection/streams/mod.rs @@ -12,7 +12,7 @@ use crate::{Dir, StreamId, VarInt, connection::streams::state::get_or_insert_rec mod recv; use recv::Recv; -pub use recv::{Chunks, ReadError, ReadableError}; +pub use recv::{Chunks, Credit, ReadError, ReadableError}; mod send; pub(crate) use send::{ByteSlice, BytesArray}; diff --git a/rs/moq-quic/src/connection/streams/recv.rs b/rs/moq-quic/src/connection/streams/recv.rs index c7fcf5d773b..526b4343e37 100644 --- a/rs/moq-quic/src/connection/streams/recv.rs +++ b/rs/moq-quic/src/connection/streams/recv.rs @@ -6,6 +6,7 @@ use tracing::debug; use super::state::get_or_insert_recv; use super::{ClosedStream, Retransmits, ShouldTransmit, StreamId, StreamsState}; +use crate::connection::ConnectionTag; use crate::connection::assembler::{Assembler, Chunk, IllegalOrderedRead}; use crate::connection::streams::state::StreamRecv; use crate::{TransportError, VarInt, frame}; @@ -336,10 +337,25 @@ impl<'a> Chunks<'a> { } } + /// Keep the connection flow control credit for the bytes read so far. + /// + /// Stream credit still advances on [`Self::finalize()`], but the connection credit for these + /// bytes is withheld from the peer until the returned [`Credit`] is passed to + /// [`Connection::release()`][crate::Connection::release]. + pub fn hold(&mut self) -> Credit { + let bytes = mem::take(&mut self.read); + self.streams.held += bytes; + Credit { + bytes, + tag: self.streams.tag, + } + } + /// Mark the read data as consumed from the stream. /// /// The number of read bytes will be released from the congestion window, /// allowing the remote peer to send more data if it was previously blocked. + /// Bytes taken by [`Self::hold()`] keep their connection credit. /// /// If [`ShouldTransmit::should_transmit()`] returns `true`, /// a packet needs to be sent to the peer informing them that the stream is unblocked. @@ -388,6 +404,69 @@ impl Drop for Chunks<'_> { } } +/// Connection flow control credit for bytes read with [`Chunks::hold()`]. +/// +/// The peer cannot reuse this credit until it is passed to +/// [`Connection::release()`][crate::Connection::release] on the connection it came from, so +/// dropping it leaks receive window. A default `Credit` is empty and belongs to no connection +/// until it merges one that does. +#[must_use = "dropping Credit leaks connection flow control credit"] +#[derive(Debug, Default, PartialEq, Eq)] +pub struct Credit { + bytes: u64, + pub(super) tag: ConnectionTag, +} + +impl Credit { + /// The number of bytes this credit covers. + pub fn len(&self) -> u64 { + self.bytes + } + + /// Whether this credit covers no bytes. + pub fn is_empty(&self) -> bool { + self.bytes == 0 + } + + /// Split off the credit for the first `bytes`, keeping the rest. + /// + /// # Panics + /// If `bytes` exceeds [`Self::len()`]. + pub fn split(&mut self, bytes: u64) -> Self { + assert!( + bytes <= self.bytes, + "split {bytes} of {} held bytes", + self.bytes + ); + self.bytes -= bytes; + Self { + bytes, + tag: self.tag, + } + } + + /// Take over the bytes covered by `other`. + /// + /// # Panics + /// If `other` came from another connection. + pub fn merge(&mut self, other: Self) { + if other.tag != self.tag { + assert!( + self.tag == ConnectionTag::NONE || other.tag == ConnectionTag::NONE, + "merged credit from another connection" + ); + if self.tag == ConnectionTag::NONE { + self.tag = other.tag; + } + } + self.bytes += other.bytes; + } + + pub(super) fn into_bytes(self) -> u64 { + self.bytes + } +} + enum ChunksState { Readable(Box), Reset(VarInt), diff --git a/rs/moq-quic/src/connection/streams/state.rs b/rs/moq-quic/src/connection/streams/state.rs index 0c47d3df7f2..56820451f1f 100644 --- a/rs/moq-quic/src/connection/streams/state.rs +++ b/rs/moq-quic/src/connection/streams/state.rs @@ -9,13 +9,14 @@ use rustc_hash::FxHashMap; use tracing::{debug, trace}; use super::{ - PendingStreamsQueue, Recv, Retransmits, Send, SendState, ShouldTransmit, StreamEvent, + Credit, PendingStreamsQueue, Recv, Retransmits, Send, SendState, ShouldTransmit, StreamEvent, StreamHalf, ThinRetransmits, }; use crate::{ Dir, MAX_STREAM_COUNT, Side, StreamId, TransportError, VarInt, coding::BufMutExt, - connection::stats::FrameStats, + connection::ConnectionTag, + connection::stats::{CreditStats, FrameStats}, frame::{self, FrameStruct, StreamMetaVec}, transport_parameters::TransportParameters, }; @@ -121,6 +122,11 @@ pub struct StreamsState { pub(super) data_sent: u64, /// Sum of end offsets of all receive streams. Includes gaps, so it's an upper bound. data_recvd: u64, + /// Bytes read with [`Chunks::hold`](super::Chunks::hold) whose connection credit the + /// application has not released yet + pub(super) held: u64, + /// Which connection [`Credit`]s handed out here belong to + pub(super) tag: ConnectionTag, /// Total quantity of outgoing data retained in send buffers, including acknowledged data pub(super) buffered_data: u64, /// Configured upper bound for `buffered_data`. @@ -180,6 +186,8 @@ impl StreamsState { sent_max_data: receive_window, data_sent: 0, data_recvd: 0, + held: 0, + tag: ConnectionTag::next(), buffered_data: 0, send_window, stream_receive_window: stream_receive_window.into(), @@ -282,7 +290,9 @@ impl StreamsState { if !rs.stopped { self.events.push_back(StreamEvent::Readable { id }); - return Ok(ShouldTransmit(false)); + // Credit released while the peer still had room may not have been worth a frame + // then; it is now that the peer is about to block. + return Ok(ShouldTransmit(self.max_data_due())); } // Stopped streams become closed instantly on FIN, so check whether we need to clean up @@ -991,17 +1001,46 @@ impl StreamsState { self.receive_window_shrink_debt -= credits; } + ShouldTransmit(self.max_data_due()) + } + + /// Return the connection credit held by `credit` to the peer. + pub(crate) fn release(&mut self, credit: Credit) -> ShouldTransmit { + assert!( + credit.is_empty() || credit.tag == self.tag, + "released credit from another connection" + ); + let bytes = credit.into_bytes(); + assert!( + bytes <= self.held, + "released {bytes} bytes of credit with only {} held", + self.held + ); + self.held -= bytes; + self.add_read_credits(bytes) + } + + /// Whether the window has grown enough since the last `MAX_DATA` to be worth announcing. + /// + /// Updates are batched to a fraction of the receive window, which accommodates bigger windows + /// with fewer frames. Once the peer is within that fraction of its limit any increase is + /// worth a frame, because the credit an application holds may never add up to the fraction. + fn max_data_due(&self) -> bool { if self.local_max_data > VarInt::MAX.into_inner() { - return ShouldTransmit(false); + return false; } - // Only announce a window update if it's significant enough - // to make it worthwhile sending a MAX_DATA frame. - // We use a fraction of the configured connection receive window to make - // the decision, to accommodate for connection using bigger windows requiring - // less updates. - let diff = self.local_max_data - self.sent_max_data.into_inner(); - ShouldTransmit(diff >= (self.receive_window / 8)) + let sent = self.sent_max_data.into_inner(); + let threshold = self.receive_window / 8; + let diff = self.local_max_data - sent; + diff >= threshold || (diff > 0 && sent.saturating_sub(self.data_recvd) < threshold) + } + + pub(crate) fn credit_stats(&self) -> CreditStats { + CreditStats { + held: self.held, + available: self.local_max_data.saturating_sub(self.data_recvd), + } } /// Update counters for removal of a stream @@ -2528,4 +2567,246 @@ mod tests { server.next_remote[Dir::Bi as usize] ); } + + /// How a held stream ends in [`held_credit_returns_once`]. + #[derive(Debug, Clone, Copy)] + enum End { + Fin, + Stop, + Reset, + StopThenReset, + } + + fn data(client: &mut StreamsState, id: StreamId, offset: u64, len: usize, fin: bool) { + let _ = client + .received( + frame::Stream { + id, + offset, + fin, + data: Bytes::from(vec![0; len]), + }, + len, + ) + .unwrap(); + } + + fn reset(client: &mut StreamsState, id: StreamId, final_offset: u32) { + let _ = client + .received_reset(frame::ResetStream { + id, + error_code: 0u32.into(), + final_offset: final_offset.into(), + }) + .unwrap(); + } + + /// Read everything readable now, holding its connection credit. + fn read_held(client: &mut StreamsState, id: StreamId) -> Credit { + let mut pending = Retransmits::default(); + let mut recv = RecvStream { + id, + state: client, + pending: &mut pending, + }; + let mut chunks = recv.read(true).unwrap(); + while let Ok(Some(_)) = chunks.next(usize::MAX) {} + let credit = chunks.hold(); + let _ = chunks.finalize(); + credit + } + + /// Every byte of a stream returns its connection credit exactly once, whichever terminal path + /// it takes and whether the held bytes are released before or after its state is reclaimed. + #[test] + fn held_credit_returns_once() { + for end in [End::Fin, End::Stop, End::Reset, End::StopThenReset] { + let mut client = make(Side::Client); + let initial = client.local_max_data; + let id = StreamId::new(Side::Server, Dir::Uni, 0); + + // A readable prefix, then a gap the transport must account for itself. + data(&mut client, id, 0, 32, false); + data(&mut client, id, 64, 16, false); + let credit = read_held(&mut client, id); + assert_eq!(credit.len(), 32, "{end:?}"); + assert_eq!(client.credit_stats().held, 32, "{end:?}"); + assert_eq!( + client.local_max_data, initial, + "{end:?}: held, not returned" + ); + + let size = match end { + End::Fin => { + data(&mut client, id, 32, 32, false); + data(&mut client, id, 80, 16, true); + let rest = read_held(&mut client, id); + assert_eq!(rest.len(), 64, "{end:?}"); + let _ = client.release(rest); + 96 + } + End::Stop => { + let mut pending = Retransmits::default(); + RecvStream { + id, + state: &mut client, + pending: &mut pending, + } + .stop(0u32.into()) + .unwrap(); + // Late data below and beyond the stop is credited on arrival. + data(&mut client, id, 32, 32, false); + data(&mut client, id, 80, 16, true); + 96 + } + End::Reset => { + reset(&mut client, id, 128); + let mut pending = Retransmits::default(); + let mut recv = RecvStream { + id, + state: &mut client, + pending: &mut pending, + }; + let mut chunks = recv.read(true).unwrap(); + assert!(chunks.next(usize::MAX).is_err(), "{end:?}"); + let _ = chunks.finalize(); + 128 + } + End::StopThenReset => { + let mut pending = Retransmits::default(); + RecvStream { + id, + state: &mut client, + pending: &mut pending, + } + .stop(0u32.into()) + .unwrap(); + reset(&mut client, id, 128); + // A duplicate reset after reclamation changes nothing. + reset(&mut client, id, 128); + 128 + } + }; + assert!( + !client.recv.contains_key(&id), + "{end:?}: state reclaimed while credit is held" + ); + assert_eq!( + client.local_max_data, + initial + size - 32, + "{end:?}: everything but the held prefix" + ); + + let _ = client.release(credit); + assert_eq!(client.local_max_data, initial + size, "{end:?}"); + assert_eq!(client.credit_stats().held, 0, "{end:?}"); + } + } + + #[test] + fn held_credit_pays_shrink_debt() { + let mut client = make(Side::Client); + let initial = client.local_max_data; + let id = StreamId::new(Side::Server, Dir::Uni, 0); + data(&mut client, id, 0, 32, false); + let credit = read_held(&mut client, id); + client.set_receive_window((client.receive_window - 16).try_into().unwrap()); + let _ = client.release(credit); + assert_eq!(client.local_max_data, initial + 16); + } + + /// Another connection holding at least as much credit still refuses it. + #[test] + #[should_panic(expected = "released credit from another connection")] + fn release_from_another_connection() { + let id = StreamId::new(Side::Server, Dir::Uni, 0); + let mut a = make(Side::Client); + data(&mut a, id, 0, 32, false); + let credit = read_held(&mut a, id); + let mut b = make(Side::Client); + data(&mut b, id, 0, 32, false); + let _held = read_held(&mut b, id); + let _ = b.release(credit); + } + + #[test] + #[should_panic(expected = "merged credit from another connection")] + fn merge_from_another_connection() { + let id = StreamId::new(Side::Server, Dir::Uni, 0); + let mut a = make(Side::Client); + data(&mut a, id, 0, 32, false); + let mut b = make(Side::Client); + data(&mut b, id, 0, 32, false); + let mut credit = read_held(&mut a, id); + credit.merge(read_held(&mut b, id)); + } + + /// An empty default credit takes on the connection of the first credit merged into it. + #[test] + fn default_credit_adopts_its_connection() { + let id = StreamId::new(Side::Server, Dir::Uni, 0); + let mut a = make(Side::Client); + data(&mut a, id, 0, 32, false); + let mut credit = Credit::default(); + credit.merge(read_held(&mut a, id)); + assert_eq!(credit.len(), 32); + let _ = a.release(credit); + assert_eq!(a.credit_stats().held, 0); + } + + /// Ingest and read rounds across many streams, with and without holding and releasing each + /// read's connection credit. The held cost must stay flat as the stream count grows. + #[test] + #[ignore = "release benchmark, run by nightly"] + fn benchmark_receive_credit() { + const BYTES: usize = 1 << 30; + println!("streams,payload,plain_ns_per_read,held_ns_per_read"); + for streams in [1u64, 64, 4096] { + for payload in [1200usize, 64 * 1024] { + let rounds = (BYTES / payload / streams as usize).max(1) as u64; + let chunk = Bytes::from(vec![0; payload]); + let mut ns = [0f64; 2]; + for (mode, held) in [false, true].into_iter().enumerate() { + let mut client = StreamsState::new( + Side::Client, + VarInt::from_u64(streams).unwrap(), + 0u32.into(), + 0, + VarInt::from_u64(1 << 50).unwrap(), + VarInt::from_u64(1 << 40).unwrap(), + ); + let mut pending = Retransmits::default(); + let start = std::time::Instant::now(); + for round in 0..rounds { + for index in 0..streams { + let id = StreamId::new(Side::Server, Dir::Uni, index); + let frame = frame::Stream { + id, + offset: round * payload as u64, + fin: false, + data: chunk.clone(), + }; + let _ = client.received(frame, payload).unwrap(); + let mut recv = RecvStream { + id, + state: &mut client, + pending: &mut pending, + }; + let mut chunks = recv.read(true).unwrap(); + std::hint::black_box(chunks.next(usize::MAX).unwrap()); + if held { + let credit = chunks.hold(); + let _ = chunks.finalize(); + let _ = client.release(credit); + } else { + let _ = chunks.finalize(); + } + } + } + ns[mode] = start.elapsed().as_nanos() as f64 / (rounds * streams) as f64; + } + println!("{streams},{payload},{:.1},{:.1}", ns[0], ns[1]); + } + } + } } diff --git a/rs/moq-quic/src/lib.rs b/rs/moq-quic/src/lib.rs index a5c39cb9576..78a1d34c52e 100644 --- a/rs/moq-quic/src/lib.rs +++ b/rs/moq-quic/src/lib.rs @@ -44,10 +44,10 @@ pub use bloom_token_log::BloomTokenLog; mod connection; pub use crate::connection::{ - Chunk, Chunks, ClosedStream, Connection, ConnectionError, ConnectionStats, Datagrams, Event, - FinishError, FrameStats, PathStats, ReadError, ReadableError, RecvStream, RttEstimator, - SendDatagramError, SendStream, ShouldTransmit, StreamEvent, Streams, UdpStats, WriteError, - Written, + Chunk, Chunks, ClosedStream, Connection, ConnectionError, ConnectionStats, Credit, CreditStats, + Datagrams, Event, FinishError, FrameStats, PathStats, ReadError, ReadableError, RecvStream, + RttEstimator, SendDatagramError, SendStream, ShouldTransmit, StreamEvent, Streams, UdpStats, + WriteError, Written, }; #[cfg(feature = "qlog")] pub use connection::qlog::QlogStream; diff --git a/rs/moq-quic/src/tests/mod.rs b/rs/moq-quic/src/tests/mod.rs index 17154b9f8ec..60ff3f285ee 100644 --- a/rs/moq-quic/src/tests/mod.rs +++ b/rs/moq-quic/src/tests/mod.rs @@ -6032,6 +6032,123 @@ fn partial_poll_reports_starvation() { ); } +/// Write what the client can and read what the server can, holding its credit, until neither +/// makes progress. Returns the bytes read. +fn pump_held( + pair: &mut Pair, + (client_ch, server_ch): (ConnectionHandle, ConnectionHandle), + s: StreamId, + msg: &[u8], + sent: &mut usize, + held: &mut Credit, +) -> usize { + let mut total = 0; + loop { + // Deliver whatever credit the server queued since the last round. + pair.drive(); + let mut wrote = 0; + if *sent < msg.len() + && let Ok(n) = pair.client_send(client_ch, s).write(&msg[*sent..]) + { + wrote = n; + *sent += n; + if *sent == msg.len() { + pair.client_send(client_ch, s).finish().unwrap(); + } + } + pair.drive(); + + let mut read = 0; + let mut recv = pair.server_recv(server_ch, s); + if let Ok(mut chunks) = recv.read(true) { + while let Ok(Some(chunk)) = chunks.next(usize::MAX) { + read += chunk.bytes.len(); + } + held.merge(chunks.hold()); + let _ = chunks.finalize(); + } + if read == 0 && wrote == 0 { + return total; + } + total += read; + } +} + +/// Held reads keep replenishing a stream window much smaller than the connection's, while the +/// connection window stays spent until the bytes are released. A release far below the usual +/// `MAX_DATA` threshold still unblocks the peer, without relying on its DATA_BLOCKED. +#[test] +fn held_credit_spends_connection_window_only() { + const WINDOW: usize = 4000; + let _guard = subscribe(); + let mut transport = cubic_transport(); + transport.receive_window((WINDOW as u32).into()); + transport.stream_receive_window(1000u32.into()); + let mut pair = Pair::new( + Default::default(), + ServerConfig { + transport: Arc::new(transport), + ..server_config() + }, + ); + let chs = pair.connect(); + let (client_ch, server_ch) = chs; + let s = pair.client_streams(client_ch).open(Dir::Uni).unwrap(); + let msg = vec![0xAB; 3 * WINDOW]; + let (mut sent, mut held) = (0, Credit::default()); + + assert_eq!( + pump_held(&mut pair, chs, s, &msg, &mut sent, &mut held), + WINDOW + ); + assert_eq!(held.len(), WINDOW as u64); + let stats = pair.server_conn_mut(server_ch).stats().credit; + assert_eq!((stats.held, stats.available), (WINDOW as u64, 0)); + + let small = held.split(100); + let _ = pair.server_conn_mut(server_ch).release(small); + assert_eq!( + pump_held(&mut pair, chs, s, &msg, &mut sent, &mut held), + 100 + ); + + // Credit too small to announce while the peer still has room goes out once the peer spends + // that room, even though the receiver reads nothing in between. + let big = held.split(600); + let _ = pair.server_conn_mut(server_ch).release(big); + pair.drive_server(); + let small = held.split(100); + let _ = pair.server_conn_mut(server_ch).release(small); + let before = sent; + for _ in 0..4 { + if let Ok(n) = pair.client_send(client_ch, s).write(&msg[sent..]) { + sent += n; + } + pair.drive(); + } + assert_eq!(sent - before, 700); + assert_eq!( + pump_held(&mut pair, chs, s, &msg, &mut sent, &mut held), + 700 + ); + + let mut total = WINDOW + 100 + 700; + while total < msg.len() { + let all = mem::take(&mut held); + let _ = pair.server_conn_mut(server_ch).release(all); + let read = pump_held(&mut pair, chs, s, &msg, &mut sent, &mut held); + assert!(read > 0, "a release unblocks the peer"); + total += read; + } + assert_eq!( + pair.server_recv(server_ch, s).read(true).err(), + Some(ReadableError::ClosedStream), + "finished and reclaimed while its tail is still held" + ); + let _ = pair.server_conn_mut(server_ch).release(held); + assert_eq!(pair.server_conn_mut(server_ch).stats().credit.held, 0); +} + #[test] fn sender_window_is_not_starvation() { let _guard = subscribe(); diff --git a/rs/moq-tokio/src/quic/driver/connection.rs b/rs/moq-tokio/src/quic/driver/connection.rs index e86cecef417..44f79933e7c 100644 --- a/rs/moq-tokio/src/quic/driver/connection.rs +++ b/rs/moq-tokio/src/quic/driver/connection.rs @@ -1015,7 +1015,7 @@ impl RemoteAddressWatcher { } #[derive(Debug)] -pub(crate) struct ConnectionRef(Arc); +pub(crate) struct ConnectionRef(pub(crate) Arc); impl ConnectionRef { fn stable_id(&self) -> usize { diff --git a/rs/moq-tokio/src/quic/driver/mod.rs b/rs/moq-tokio/src/quic/driver/mod.rs index d758cc9c7c6..2a927ce4c8e 100644 --- a/rs/moq-tokio/src/quic/driver/mod.rs +++ b/rs/moq-tokio/src/quic/driver/mod.rs @@ -73,7 +73,7 @@ pub use crate::quic::driver::connection::{ pub use crate::quic::driver::endpoint::{Accept, Endpoint, EndpointStats}; pub use crate::quic::driver::incoming::{Incoming, IncomingFuture, RetryError}; pub use crate::quic::driver::recv_stream::{ - ReadError, ReadExactError, ReadToEndError, RecvStream, ResetError, + Credit, ReadError, ReadExactError, ReadToEndError, RecvStream, ResetError, }; #[cfg(test)] pub(crate) use crate::quic::driver::runtime::wrap_udp_socket; diff --git a/rs/moq-tokio/src/quic/driver/recv_stream.rs b/rs/moq-tokio/src/quic/driver/recv_stream.rs index 2bf91f62310..bde840c96c6 100644 --- a/rs/moq-tokio/src/quic/driver/recv_stream.rs +++ b/rs/moq-tokio/src/quic/driver/recv_stream.rs @@ -1,7 +1,9 @@ use std::{ + fmt, future::{Future, poll_fn}, - io, + io, mem, pin::Pin, + sync::Arc, task::{Context, Poll, ready}, }; @@ -10,7 +12,10 @@ use moq_quic::{Chunk, Chunks, ClosedStream, ConnectionError, ReadableError, Stre use thiserror::Error; use tokio::io::ReadBuf; -use crate::quic::driver::{VarInt, connection::ConnectionRef}; +use crate::quic::driver::{ + VarInt, + connection::{ConnectionInner, ConnectionRef}, +}; /// A stream that can only be used to receive data /// @@ -94,6 +99,50 @@ impl RecvStream { .await } + /// Read data contiguously from the stream, keeping its connection flow control credit. + /// + /// Like [`read()`], but yields the [`Credit`] for exactly the bytes read into `buf`, or `None` + /// if the stream was finished. Stream credit still advances, so a stream window smaller than + /// an application message keeps the message flowing, while the connection window these bytes + /// occupy stays spent until the credit is dropped. + /// + /// This operation is cancel-safe. + /// + /// [`read()`]: RecvStream::read + pub async fn read_held(&mut self, buf: &mut [u8]) -> Result, ReadError> { + poll_fn(|cx| self.poll_read_held(cx, buf)).await + } + + /// Attempts to read from the stream into the provided buffer, keeping the bytes' connection + /// flow control credit. + /// + /// See [`read_held()`] for details. + /// + /// [`read_held()`]: RecvStream::read_held + pub fn poll_read_held( + &mut self, + cx: &mut Context<'_>, + buf: &mut [u8], + ) -> Poll, ReadError>> { + let mut buf = ReadBuf::new(buf); + let mut held = moq_quic::Credit::default(); + let read = match buf.remaining() { + 0 => Poll::Ready(Ok(Some(()))), + _ => self.poll_read_generic(cx, true, |chunks| { + let status = fill(chunks, &mut buf); + held.merge(chunks.hold()); + status + }), + }; + // Bytes are read exactly when the read yields `Some`. + read.map_ok(|read| { + read.map(|()| Credit { + conn: self.conn.0.clone(), + inner: mem::take(&mut held), + }) + }) + } + /// Attempts to read from the stream into the provided buffer /// /// On success, returns `Poll::Ready(Ok(num_bytes_read))` and places data into `buf`. If this @@ -136,24 +185,8 @@ impl RecvStream { return Poll::Ready(Ok(())); } - self.poll_read_generic(cx, true, |chunks| { - let mut read = false; - loop { - if buf.remaining() == 0 { - // We know `read` is `true` because `buf.remaining()` was not 0 before - return ReadStatus::Readable(()); - } - - match chunks.next(buf.remaining()) { - Ok(Some(chunk)) => { - buf.put_slice(&chunk.bytes); - read = true; - } - res => return (if read { Some(()) } else { None }, res.err()).into(), - } - } - }) - .map(|res| res.map(|_| ())) + self.poll_read_generic(cx, true, |chunks| fill(chunks, buf)) + .map(|res| res.map(|_| ())) } /// Read the next segment of data @@ -429,6 +462,91 @@ impl RecvStream { } } +/// Copy from `chunks` until `buf` is full or nothing more is readable. +fn fill(chunks: &mut Chunks<'_>, buf: &mut ReadBuf<'_>) -> ReadStatus<()> { + let mut read = false; + loop { + if buf.remaining() == 0 { + // We know `read` is `true` because `buf.remaining()` was not 0 before + return ReadStatus::Readable(()); + } + + match chunks.next(buf.remaining()) { + Ok(Some(chunk)) => { + buf.put_slice(&chunk.bytes); + read = true; + } + res => return (if read { Some(()) } else { None }, res.err()).into(), + } + } +} + +/// Connection flow control credit for bytes read with [`RecvStream::read_held`]. +/// +/// The peer cannot reuse the connection window these bytes occupy until this is dropped, which +/// returns it even after the stream or connection is gone. +pub struct Credit { + conn: Arc, + inner: moq_quic::Credit, +} + +impl Credit { + /// The number of bytes this credit covers. + pub fn len(&self) -> u64 { + self.inner.len() + } + + /// Whether this credit covers no bytes. + pub fn is_empty(&self) -> bool { + self.inner.is_empty() + } + + /// Split off the credit for the first `bytes`, keeping the rest. + /// + /// # Panics + /// If `bytes` exceeds [`Self::len()`]. + pub fn split(&mut self, bytes: u64) -> Self { + Self { + conn: self.conn.clone(), + inner: self.inner.split(bytes), + } + } + + /// Take over the bytes covered by `other`. + /// + /// # Panics + /// If `other` came from another connection. + pub fn merge(&mut self, mut other: Self) { + assert!( + Arc::ptr_eq(&self.conn, &other.conn), + "merged credit from another connection" + ); + self.inner.merge(mem::take(&mut other.inner)); + } +} + +impl Drop for Credit { + fn drop(&mut self) { + if self.inner.is_empty() { + return; + } + let mut conn = self.conn.state.lock("Credit::drop"); + if conn + .inner + .release(mem::take(&mut self.inner)) + .should_transmit() + { + conn.wake(); + } + } +} + +impl fmt::Debug for Credit { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("Credit").field("len", &self.len()).finish() + } +} + enum ReadStatus { Readable(T), Finished(Option), diff --git a/rs/moq-tokio/src/quic/driver/tests.rs b/rs/moq-tokio/src/quic/driver/tests.rs index aae7054d83a..4b63539caf3 100755 --- a/rs/moq-tokio/src/quic/driver/tests.rs +++ b/rs/moq-tokio/src/quic/driver/tests.rs @@ -43,7 +43,9 @@ use tracing::instrument::Instrument as _; use tracing::{error_span, info}; use tracing_subscriber::EnvFilter; -use super::{ClientConfig, Endpoint, EndpointConfig, RecvStream, SendStream, TransportConfig}; +use super::{ + ClientConfig, Credit, Endpoint, EndpointConfig, RecvStream, SendStream, TransportConfig, +}; #[test] fn handshake_timeout() { @@ -1851,6 +1853,65 @@ async fn dropped_connection_cleans_up() { endpoint.wait_idle().await; } +/// Held reads keep the connection window spent until their credit drops, while a stream window +/// far smaller than it keeps replenishing; credit outlives its stream. +#[tokio::test] +async fn held_reads_spend_the_connection_window() { + const WINDOW: usize = 32 * 1024; + let _guard = subscribe(); + let mut transport = TransportConfig::default(); + transport.receive_window((WINDOW as u32).into()); + transport.stream_receive_window(4096u32.into()); + let endpoint = endpoint_with_config(transport); + + let (client, server) = tokio::try_join!( + endpoint + .connect(endpoint.local_addr().unwrap(), "localhost") + .unwrap(), + async { endpoint.accept().await.unwrap().await } + ) + .unwrap(); + let mut send = client.open_uni().await.unwrap(); + let writer = tokio::spawn(async move { + // The receiver walks away before this finishes. + let _ = send.write_all(&[7; 4 * WINDOW]).await; + }); + + let mut recv = server.accept_uni().await.unwrap(); + let mut buf = vec![0; WINDOW]; + let mut held = read_held_exact(&mut recv, &mut buf).await; + assert_eq!(held.len(), WINDOW as u64); + let stats = server.stats().credit; + assert_eq!((stats.held, stats.available), (WINDOW as u64, 0)); + + // Far below the usual update threshold, and still enough to unblock. + drop(held.split(100)); + held.merge(read_held_exact(&mut recv, &mut buf[..100]).await); + assert_eq!(server.stats().credit.held, WINDOW as u64); + + drop(held); + let tail = read_held_exact(&mut recv, &mut buf[..500]).await; + drop(recv); + assert_eq!( + server.stats().credit.held, + 500, + "credit outlives its stream" + ); + drop(tail); + assert_eq!(server.stats().credit.held, 0); + writer.await.unwrap(); +} + +/// Fill `buf` with held reads, returning the credit for all of it. +async fn read_held_exact(recv: &mut RecvStream, buf: &mut [u8]) -> Credit { + let mut held = recv.read_held(buf).await.unwrap().unwrap(); + while (held.len() as usize) < buf.len() { + let read = held.len() as usize; + held.merge(recv.read_held(&mut buf[read..]).await.unwrap().unwrap()); + } + held +} + #[derive(Default)] struct WakeCounter { wakes: AtomicUsize, diff --git a/rs/moq-uring/src/quic/driver/connection.rs b/rs/moq-uring/src/quic/driver/connection.rs index fb70a5a1126..a672a9417b2 100644 --- a/rs/moq-uring/src/quic/driver/connection.rs +++ b/rs/moq-uring/src/quic/driver/connection.rs @@ -337,6 +337,11 @@ impl Connection { pub fn handshake(&self) -> &crate::quic::handshake::Info { &self.handshake } + + /// Returns connection statistics. + pub fn stats(&self) -> moq_quic::ConnectionStats { + self.shared.conn.borrow().stats() + } } impl Clone for Connection { diff --git a/rs/moq-uring/src/quic/driver/mod.rs b/rs/moq-uring/src/quic/driver/mod.rs index 7a6b265f89a..30e53a05a34 100644 --- a/rs/moq-uring/src/quic/driver/mod.rs +++ b/rs/moq-uring/src/quic/driver/mod.rs @@ -15,7 +15,7 @@ mod stream; pub use connection::Connection; pub use endpoint::Endpoint; -pub use stream::{RecvStream, SendStream}; +pub use stream::{Credit, RecvStream, SendStream}; pub(crate) use connection::{End, Shared}; diff --git a/rs/moq-uring/src/quic/driver/stream.rs b/rs/moq-uring/src/quic/driver/stream.rs index e415f57f308..1be7b4e6a99 100644 --- a/rs/moq-uring/src/quic/driver/stream.rs +++ b/rs/moq-uring/src/quic/driver/stream.rs @@ -223,6 +223,9 @@ pub struct RecvStream { stopped: bool, /// Bytes `poll_closed` read ahead, handed to `poll_read` before moq-quic's. backlog: BytesMut, + /// The connection credit for `backlog`, kept until the application reads + /// it, as if it were still in moq-quic's reassembly buffer. + backlog_credit: moq_quic::Credit, } impl RecvStream { @@ -234,6 +237,63 @@ impl RecvStream { finished: false, stopped: false, backlog: BytesMut::new(), + backlog_credit: moq_quic::Credit::default(), + } + } + + /// Read into `dst`, keeping the connection flow control credit for exactly + /// the bytes read, or `None` once the peer has finished the stream. + /// + /// Stream credit still advances, so a stream window smaller than an + /// application message keeps the message flowing, while the connection + /// window these bytes occupy stays spent until the credit is dropped. + pub fn poll_read_held(&mut self, cx: &mut Context<'_>, dst: &mut [u8]) -> Poll, Error>> { + let mut held = moq_quic::Credit::default(); + self.poll_read_into(cx, dst, Some(&mut held)).map_ok(|read| { + read.map(|_| Credit { + shared: self.shared.clone(), + inner: held, + }) + }) + } + + /// [`poll_read`](web_transport_trait::poll::RecvStream::poll_read), with + /// the credit for the bytes read moved into `held` instead of returned to + /// the peer. + fn poll_read_into( + &mut self, + cx: &mut Context<'_>, + dst: &mut [u8], + held: Option<&mut moq_quic::Credit>, + ) -> Poll, Error>> { + let waiter = self.park.hold(cx); + if dst.is_empty() { + return Poll::Ready(Ok(Some(0))); + } + if !self.backlog.is_empty() { + return Poll::Ready(Ok(Some(self.drain(dst, held)))); + } + if self.finished { + return Poll::Ready(Ok(None)); + } + match Self::read(&self.shared, self.id, dst.len(), held) { + Read::Chunk(bytes) => { + // moq-quic hands out at most `dst.len()`. + dst[..bytes.len()].copy_from_slice(&bytes); + Poll::Ready(Ok(Some(bytes.len()))) + } + Read::Finished => { + self.finished = true; + Poll::Ready(Ok(None)) + } + Read::Blocked => { + if let Some(err) = self.shared.closed() { + return Poll::Ready(Err(err)); + } + self.shared.park_readable(self.id, waiter); + Poll::Pending + } + Read::Reset(code) => Poll::Ready(Err(Error::Reset(code))), } } @@ -246,6 +306,7 @@ impl RecvStream { /// [`stop`](web_transport_trait::poll::RecvStream::stop) with a /// full-width code, for the WebTransport HTTP/3 error mapping. pub(crate) fn stop_code(&mut self, code: u64) { + release(&self.shared, std::mem::take(&mut self.backlog_credit)); self.backlog.clear(); if self.stopped || self.finished { return; @@ -264,11 +325,12 @@ impl RecvStream { /// Take up to `max` bytes out of moq-quic's reassembly buffer. /// /// Reading is what returns the peer's flow control credit, so a read that - /// owes it a frame kicks the driver. + /// owes it a frame kicks the driver. The connection credit for the bytes + /// read moves into `held` instead, when given. /// /// Takes the shared state rather than `&mut self` so a caller can hold a /// waiter from `self.park` across the read. - fn read(shared: &Shared, id: StreamId, max: usize) -> Read { + fn read(shared: &Shared, id: StreamId, max: usize, held: Option<&mut moq_quic::Credit>) -> Read { let mut conn = shared.conn.borrow_mut(); let mut recv = conn.recv_stream(id); let mut chunks = match recv.read(true) { @@ -282,6 +344,9 @@ impl RecvStream { Err(moq_quic::ReadError::Blocked) => Read::Blocked, Err(moq_quic::ReadError::Reset(code)) => Read::Reset(code.into_inner()), }; + if let Some(held) = held { + held.merge(chunks.hold()); + } let transmit = chunks.finalize().should_transmit(); drop(conn); if transmit { @@ -290,14 +355,20 @@ impl RecvStream { read } - /// Move up to `dst.len()` read-ahead bytes out of the backlog. + /// Move up to `dst.len()` read-ahead bytes out of the backlog, with their + /// credit moved into `held` or else returned to the peer. /// /// Wakes a `poll_closed` parked at the read-ahead cap: the room it was /// waiting for is what this just made. - fn drain(&mut self, dst: &mut [u8]) -> usize { + fn drain(&mut self, dst: &mut [u8], held: Option<&mut moq_quic::Credit>) -> usize { let n = dst.len().min(self.backlog.len()); dst[..n].copy_from_slice(&self.backlog[..n]); self.backlog.advance(n); + let credit = self.backlog_credit.split(n as u64); + match held { + Some(held) => held.merge(credit), + None => release(&self.shared, credit), + } if n > 0 { self.shared.wake_readable(self.id); } @@ -309,35 +380,7 @@ impl web_transport_trait::poll::RecvStream for RecvStream { type Error = Error; fn poll_read(&mut self, cx: &mut Context<'_>, dst: &mut [u8]) -> Poll, Self::Error>> { - let waiter = self.park.hold(cx); - if dst.is_empty() { - return Poll::Ready(Ok(Some(0))); - } - if !self.backlog.is_empty() { - return Poll::Ready(Ok(Some(self.drain(dst)))); - } - if self.finished { - return Poll::Ready(Ok(None)); - } - match Self::read(&self.shared, self.id, dst.len()) { - Read::Chunk(bytes) => { - let n = bytes.len().min(dst.len()); - dst[..n].copy_from_slice(&bytes[..n]); - Poll::Ready(Ok(Some(n))) - } - Read::Finished => { - self.finished = true; - Poll::Ready(Ok(None)) - } - Read::Blocked => { - if let Some(err) = self.shared.closed() { - return Poll::Ready(Err(err)); - } - self.shared.park_readable(self.id, waiter); - Poll::Pending - } - Read::Reset(code) => Poll::Ready(Err(Error::Reset(code))), - } + self.poll_read_into(cx, dst, None) } fn stop(&mut self, code: u32) { @@ -367,7 +410,7 @@ impl web_transport_trait::poll::RecvStream for RecvStream { self.shared.park_readable(self.id, waiter); return Poll::Pending; } - match Self::read(&self.shared, self.id, READ_CHUNK) { + match Self::read(&self.shared, self.id, READ_CHUNK, Some(&mut self.backlog_credit)) { Read::Chunk(bytes) => self.backlog.extend_from_slice(&bytes), Read::Finished => { self.finished = true; @@ -388,6 +431,7 @@ impl web_transport_trait::poll::RecvStream for RecvStream { impl Drop for RecvStream { fn drop(&mut self) { + release(&self.shared, std::mem::take(&mut self.backlog_credit)); self.shared.forget_recv(self.id); if !self.finished && !self.stopped { let _ = self @@ -406,3 +450,70 @@ impl std::fmt::Debug for RecvStream { f.debug_struct("RecvStream").field("id", &self.id).finish() } } + +/// Return `credit` to the peer, kicking the driver if the peer should hear. +fn release(shared: &Shared, credit: moq_quic::Credit) { + if credit.is_empty() { + return; + } + if shared.conn.borrow_mut().release(credit).should_transmit() { + shared.kick(); + } +} + +/// Connection flow control credit for bytes read with +/// [`RecvStream::poll_read_held`]. +/// +/// The peer cannot reuse the connection window these bytes occupy until this +/// is dropped, which returns it even after the stream or connection is gone. +pub struct Credit { + shared: Shared, + inner: moq_quic::Credit, +} + +impl Credit { + /// The number of bytes this credit covers. + pub fn len(&self) -> u64 { + self.inner.len() + } + + /// Whether this credit covers no bytes. + pub fn is_empty(&self) -> bool { + self.inner.is_empty() + } + + /// Split off the credit for the first `bytes`, keeping the rest. + /// + /// # Panics + /// If `bytes` exceeds [`Self::len()`]. + pub fn split(&mut self, bytes: u64) -> Self { + Self { + shared: self.shared.clone(), + inner: self.inner.split(bytes), + } + } + + /// Take over the bytes covered by `other`. + /// + /// # Panics + /// If `other` came from another connection. + pub fn merge(&mut self, mut other: Self) { + assert!( + std::rc::Rc::ptr_eq(&self.shared, &other.shared), + "merged credit from another connection" + ); + self.inner.merge(std::mem::take(&mut other.inner)); + } +} + +impl Drop for Credit { + fn drop(&mut self) { + release(&self.shared, std::mem::take(&mut self.inner)); + } +} + +impl std::fmt::Debug for Credit { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Credit").field("len", &self.len()).finish() + } +} diff --git a/rs/moq-uring/src/quic/mod.rs b/rs/moq-uring/src/quic/mod.rs index f6e84c8959f..b8ccd230f33 100644 --- a/rs/moq-uring/src/quic/mod.rs +++ b/rs/moq-uring/src/quic/mod.rs @@ -30,7 +30,7 @@ pub mod web; mod driver; -pub use driver::{Connection, RecvStream, SendStream}; +pub use driver::{Connection, Credit, RecvStream, SendStream}; pub use endpoint::Endpoint; /// The QUIC payload size every full datagram in a GSO train uses, and the diff --git a/rs/moq-uring/tests/credit.rs b/rs/moq-uring/tests/credit.rs new file mode 100644 index 00000000000..6a9fcf3787b --- /dev/null +++ b/rs/moq-uring/tests/credit.rs @@ -0,0 +1,186 @@ +//! Held receive credit through the io_uring driver: held reads keep the +//! connection window spent until their credit drops, and bytes `poll_closed` +//! reads ahead keep theirs until the application reads them. +//! +//! Unsupported local kernels skip loudly; MOQ_STRICT makes CI refuse them. + +#![cfg(all(target_os = "linux", feature = "quic"))] + +#[path = "support.rs"] +mod support; + +use std::net::UdpSocket; +use std::task::Poll; +use std::time::Duration; + +use moq_uring::{Config, Error, Worker, quic, udp}; +use web_transport_trait::poll::{RecvStream as _, SendStream as _, Session as _}; + +const ALPN: &str = "moq-uring-credit"; +/// The server's connection window; under `poll_closed`'s read-ahead cap so +/// one window fits in its backlog. +const WINDOW: usize = 32 * 1024; + +fn worker() -> Option { + match Worker::new(Config::default()) { + Ok(worker) => Some(worker), + Err(Error::Unsupported(reason)) if std::env::var_os("MOQ_STRICT").is_none() => { + eprintln!("skipping io_uring credit test: {reason}"); + None + } + Err(err) => panic!("worker setup failed: {err}"), + } +} + +/// Await `future`, failing loudly rather than hanging if it never resolves. +async fn within(handle: &moq_uring::Handle, what: &str, future: impl Future) -> T { + let mut deadline = moq_uring::Timer::after(handle, Duration::from_secs(5)); + let mut future = std::pin::pin!(future); + kio::wait(|waiter| { + let mut cx = waiter.context(); + if let Poll::Ready(value) = future.as_mut().poll(&mut cx) { + return Poll::Ready(Some(value)); + } + deadline.poll(waiter).map(|()| None) + }) + .await + .unwrap_or_else(|| panic!("timed out waiting for {what}")) +} + +/// Read exactly `len` bytes, keeping their credit. +async fn read_held(handle: &moq_uring::Handle, recv: &mut quic::RecvStream, len: usize) -> quic::Credit { + let mut buf = vec![0u8; len]; + let mut held: Option = None; + while held.as_ref().map_or(0, quic::Credit::len) < len as u64 { + let filled = held.as_ref().map_or(0, |held| held.len() as usize); + let credit = within( + handle, + "stream data", + std::future::poll_fn(|cx| recv.poll_read_held(cx, &mut buf[filled..])), + ) + .await + .expect("read") + .expect("not finished"); + match &mut held { + Some(held) => held.merge(credit), + None => held = Some(credit), + } + } + held.expect("len > 0") +} + +/// Read exactly `len` bytes. +async fn read(handle: &moq_uring::Handle, recv: &mut quic::RecvStream, len: usize) { + let mut buf = vec![0u8; len]; + let mut filled = 0; + while filled < len { + let n = within( + handle, + "stream data", + std::future::poll_fn(|cx| recv.poll_read(cx, &mut buf[filled..])), + ) + .await + .expect("read") + .expect("not finished"); + filled += n; + } +} + +#[test] +fn held_reads_spend_the_connection_window() { + let Some(mut worker) = worker() else { return }; + let handle = worker.handle(); + let certs = support::certs().expect("certificates"); + + let socket = |handle: &moq_uring::Handle| { + handle + .udp(UdpSocket::bind("127.0.0.1:0").expect("bind"), udp::Config::default()) + .expect("socket") + }; + let mut server_config = quic::server::Config::new( + quic::Identity::open(&certs.cert, &certs.key) + .expect("identity") + .server(&[ALPN], quic::server::ClientAuth::None) + .expect("tls"), + ); + server_config.transport.receive_window = Some(WINDOW as u64); + // Far smaller than the window: held reads must keep replenishing it. + server_config.transport.stream_receive_window = Some(4096); + let server = quic::Endpoint::new( + socket(&handle), + quic::endpoint::Config::default().with_server(server_config), + ) + .expect("server endpoint"); + let client = quic::Endpoint::new(socket(&handle), quic::endpoint::Config::default()).expect("dial-only endpoint"); + let mut dial = quic::client::Config::new(server.local_addr(), "localhost"); + dial.alpn = vec![ALPN.to_string()]; + dial.verify = false; + + worker + .block_on(async move { + let mut dialed = client.connect(&dial).await.expect("dial"); + let mut accepted = server.accept().await.expect("accept"); + + let mut send = std::future::poll_fn(|cx| dialed.poll_open_uni(cx)) + .await + .expect("open_uni"); + handle.spawn(async move { + let payload = vec![7u8; 4 * WINDOW]; + let mut buf = &payload[..]; + while !buf.is_empty() { + match std::future::poll_fn(|cx| send.poll_write(cx, buf)).await { + Ok(n) => buf = &buf[n..], + // The receiver stops once it has seen enough. + Err(_) => return, + } + } + let _ = send.finish(); + let _ = std::future::poll_fn(|cx| send.poll_closed(cx)).await; + }); + + let mut recv = within( + &handle, + "the stream", + std::future::poll_fn(|cx| accepted.poll_accept_uni(cx)), + ) + .await + .expect("accept_uni"); + + let mut held = read_held(&handle, &mut recv, WINDOW).await; + let stats = accepted.stats().credit; + assert_eq!((stats.held, stats.available), (WINDOW as u64, 0)); + + // Far below the usual update threshold, and still enough to unblock. + drop(held.split(100)); + held.merge(read_held(&handle, &mut recv, 100).await); + assert_eq!(accepted.stats().credit.held, WINDOW as u64); + + // Read-ahead keeps the credit the application has not taken yet. + drop(held); + within( + &handle, + "a window of read-ahead", + std::future::poll_fn(|cx| match recv.poll_closed(cx) { + Poll::Ready(res) => panic!("stream ended early: {res:?}"), + Poll::Pending if accepted.stats().credit.held == WINDOW as u64 => Poll::Ready(()), + Poll::Pending => Poll::Pending, + }), + ) + .await; + + // A plain read returns it as it takes the bytes; a held read keeps it. + read(&handle, &mut recv, 1000).await; + assert_eq!(accepted.stats().credit.held, WINDOW as u64 - 1000); + let taken = read_held(&handle, &mut recv, 500).await; + assert_eq!(accepted.stats().credit.held, WINDOW as u64 - 1000); + + // Stopping returns the read-ahead; the application still owns what it took. + recv.stop(0); + assert_eq!(accepted.stats().credit.held, 500); + drop(recv); + assert_eq!(accepted.stats().credit.held, 500, "credit outlives its stream"); + drop(taken); + assert_eq!(accepted.stats().credit.held, 0); + }) + .expect("worker"); +} diff --git a/rs/moq-uring/tests/web.rs b/rs/moq-uring/tests/web.rs index 2fbf2157ba4..ffdd71de508 100644 --- a/rs/moq-uring/tests/web.rs +++ b/rs/moq-uring/tests/web.rs @@ -737,8 +737,7 @@ fn cancelling_a_pending_web_handshake(send_settings: bool) { let mut watch = conn.clone(); let mut control = None; if send_settings { - use web_transport_trait::Stats as _; - let before = watch.stats().bytes_received().expect("receive stats"); + let before = watch.stats().udp_rx.bytes; let mut stream = std::future::poll_fn(|cx| peer.poll_open_uni(cx)) .await .expect("control stream"); @@ -756,7 +755,7 @@ fn cancelling_a_pending_web_handshake(send_settings: bool) { control = Some(stream); within(&handle, "peer SETTINGS to arrive", async { loop { - if watch.stats().bytes_received().expect("receive stats") > before { + if watch.stats().udp_rx.bytes > before { break; } let mut tick = moq_uring::Timer::after(&handle, std::time::Duration::from_millis(10)); diff --git a/sh/gh/nightly.sh b/sh/gh/nightly.sh index e3f82a30f85..3570cf01028 100755 --- a/sh/gh/nightly.sh +++ b/sh/gh/nightly.sh @@ -24,6 +24,7 @@ case "${1:-}" in --run-ignored only --success-output immediate run just rs bench-feed-fetch run just rs test -p moq-relay -E 'test(benchmark_stun_sources)' --run-ignored only --success-output immediate + run just rs test -p moq-quic --release -E 'test(benchmark_receive_credit)' --run-ignored only --success-output immediate # Fuzz exposes moq-net's codec benches; it is not a feature of the other crates. run cargo bench --locked --no-fail-fast -p moq-net --features fuzz --bench '*' -- --test # Each kio routine once, including the contended bench's worker threads.