diff --git a/quest/m1/credit-model.md b/quest/m1/credit-model.md index a0dc11b8e6..cd8d3c18b5 100644 --- a/quest/m1/credit-model.md +++ b/quest/m1/credit-model.md @@ -65,6 +65,6 @@ Public API: none. Wire: none. ## Related -- [Paused-stream allowance](/quest/m1/paused-allowance.md) - the design this model checks before it merges +- [Unsealed frame credit](/quest/m4/unsealed-frame-credit.md) - returns credit as bytes are read, which the model must then cover - [Overcommitted receive-window cells](/quest/m3/receive-windows-evicted.md) - the benchmark side of the same mechanism - [Send-credit stall counters](/quest/m3/send-credit-stalls.md) - what an operator sees when the sender waits on credit diff --git a/quest/m1/paused-allowance.md b/quest/m1/paused-allowance.md deleted file mode 100644 index 559567d8a6..0000000000 --- a/quest/m1/paused-allowance.md +++ /dev/null @@ -1,24 +0,0 @@ -# [S] Any number of streams pause within half the connection window - -## Goal - -[#9210](https://github.com/moq-dev/moq/pull/9210) merges with its design -checked by the [credit model](/quest/m1/credit-model.md): each paused stream -holds a slot-sized allowance of credit instead of a stream window, so any -number of streams pause within half the connection window. #9210 deletes -this quest. - -## Plan - -Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control -audit): ✅ #9210 ships in this cut (rejected: landing after the cut to soak). -✅ It merges once the credit model passes on its design (rejected: merging -first and modeling the result). - -#9210's own design decisions, settled 2026-11-19, and its two open `Waiting` -items stay on the PR. When the model finds a violation in its design, fix it -there before merging. - -## Required - -- [Credit model](/quest/m1/credit-model.md) - checks the slot allowances and `set_window` before they ship diff --git a/quest/m3/release/cut.md b/quest/m3/release/cut.md index 9e1aeced74..e68094c664 100644 --- a/quest/m3/release/cut.md +++ b/quest/m3/release/cut.md @@ -234,8 +234,7 @@ Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): Decided 2026-11-20 (maintainer, /quest-plan of the pre-cut flow-control audit): ✅ the cut waits on the [credit model](/quest/m1/credit-model.md), [#9210](https://github.com/moq-dev/moq/pull/9210)'s -[paused-stream allowance](/quest/m1/paused-allowance.md), which merges once -the model passes on it, and a [bounded relay cache by +paused-stream allowance, which merges once the model passes on it, and a [bounded relay cache by default](/quest/m1/relay-memory-default.md). Rejected: cutting with flow control unverified; landing #9210 after the cut. The [JS frame cap](/quest/m1/js-frame-cap.md) ships with the cut only if it @@ -260,6 +259,5 @@ check that Release remeasure opened or updated its pin PR, as RELEASE.md describ - [Interop and Video pass on main](/quest/m0/interop-green/README.md) - the cut PR's release gate runs `interop.yml`, which failed on `main` every day through 2026-11-12 - [Nightly passes on main](/quest/m0/nightly/README.md) - the release gate runs `features.yml`, red in the 2026-11-12 Nightly - [Credit model](/quest/m1/credit-model.md) - receive credit, pausing, and the sender's credit wait never deadlock -- [Paused-stream allowance](/quest/m1/paused-allowance.md) - #9210 merges with its design checked - [Relay memory default](/quest/m1/relay-memory-default.md) - a stock relay's cache is bounded - [Claim new npm names](/quest/m3/release/npm-first-publish.md) - trusted publishing can publish every new `@moq/*` package diff --git a/quest/m4/unsealed-frame-credit.md b/quest/m4/unsealed-frame-credit.md new file mode 100644 index 0000000000..8df7713593 --- /dev/null +++ b/quest/m4/unsealed-frame-credit.md @@ -0,0 +1,42 @@ +# [L] A frame's bytes return their credit as they are read, not at its seal + +## Goal + +About 200 concurrent groups, each writing 64 KiB frames over one connection, +arrive losslessly when every reader is slower than its writer. Today they +evict with no paused stream refused, at the default 4 MiB connection window. + +A stream mid-frame holds its unsealed bytes' connection credit until the +segment seals, so N such streams in flight at once need N frames of +connection window, less what paused streams hold. [#9210](https://github.com/moq-dev/moq/pull/9210) +sized a paused stream's allowance so any number pause, and the sim at 64 KiB +frames now meets this limit first: about 150 groups pass and 200 evict. The +limit holds for streams that have not paused yet, so no allowance split lifts +it. + +## Plan + +Possible direction, not decided: return a stream's credit as its bytes are +read, instead of at the seal. The obstacle is that a segment's bytes are +allocated against the origin's budget until it seals (`rs/moq-net/src/model`), +and the lease is released per segment +(`rs/moq-quic/src/connection/streams/lease.rs`), so crediting earlier means +either the budget charges a segment as it fills or the transport holds +credit for bytes the model already counts. It is a design change of its own, +as [unbounded groups](/quest/m3/unbounded-groups/README.md) made the release +per segment. + +Reproduce with `quic_sim::many_groups_carry_bulk_transfers_losslessly` at 200 +groups instead of `MANY`. The fix raises `MANY` until a frame's worth of +credit per stream no longer bounds it, or says what bounds it instead. + +Whatever lands updates the [credit model](/quest/m1/credit-model.md): early +return changes when credit comes back, and its safety property (credit held +together never exceeds the window) must still hold. + +Public API: none expected. Wire: none. + +## Related + +- [Credit model](/quest/m1/credit-model.md) - models the credit this changes +- [Unbounded groups](/quest/m3/unbounded-groups/README.md) - decided the per-segment release diff --git a/rs/justfile b/rs/justfile index 29177ac0d6..0746564f52 100644 --- a/rs/justfile +++ b/rs/justfile @@ -387,6 +387,8 @@ quic-sim: # Forward delay against RTT as a queue-growth signal: deterministic in the simulation, then over real sockets through each cellular moq-shaper preset, one at a time so they do not add to each other's noise. bash sh/rs/nextest.sh cargo nextest run --locked --release -p moq-quic --lib --run-ignored only --no-capture -E 'test(/^tests::receive_ts::/)' bash sh/rs/nextest.sh cargo nextest run --locked --release -p moq-tokio --test integration --run-ignored only --no-capture --test-threads 1 -E 'test(/^receive_ts::/)' + # A hundred bulk transfers on one relay connection all pause losslessly, where the merge gate runs twelve. + bash sh/rs/nextest.sh cargo nextest run --locked --release -p moq-relay --features test-support --test integration --run-ignored only -E 'test(/^quic_sim::many_/)' # Each realtime congestion controller against BBR3 over sim replicas of the cellular presets: queueing delay, throughput, reaction to the drop, and share beside CUBIC. bash sh/rs/nextest.sh cargo nextest run --locked --release -p moq-quic --lib --run-ignored only --no-capture -E 'test(/^tests::realtime::/)' diff --git a/rs/moq-net/src/coding/reader.rs b/rs/moq-net/src/coding/reader.rs index 57f23e52af..af402d5614 100644 --- a/rs/moq-net/src/coding/reader.rs +++ b/rs/moq-net/src/coding/reader.rs @@ -28,6 +28,10 @@ pub struct Reader { scratch: Vec, /// Bytes taken into a frame's unsealed segment, whose receive credit returns once it seals. unreleased: u64, + /// The declared size of the last frame admitted. A group pauses when its next frame does + /// not fit, up to a frame short of its window, so the stream's credit stops that short + /// too, judging the next frame by this one. + frame: u64, stats: crate::stats::Session, } @@ -76,14 +80,18 @@ pub(crate) trait Pause { /// How far the group's writer reads this stream ahead of its fastest reader before /// pausing: the stream's receive credit, capped at the live tail's ahead half (see /// [`crate::model::pace::window`]). A stream outside any session takes the cap. - fn window(&self) -> u64 { + fn window(&mut self) -> u64 { crate::model::pace::AHEAD } - /// Reserve the credit this stream may hold while paused, its stream window and the `taken` - /// bytes it read whose credit it has yet to return, returning whether it may pause. A + /// Let the peer send only what the group's writer has `room` for, at least the `unsealed` + /// rest of the segment being written, so the stream holds little once it pauses; `None` + /// while no reader holds the writer back. + fn shape(&mut self, _room: Option, _unsealed: u64) {} + + /// Reserve the credit this stream holds while paused, returning whether it may pause. A /// stream outside any session always may. - fn pause(&mut self, _taken: u64) -> bool { + fn pause(&mut self) -> bool { true } @@ -92,12 +100,16 @@ pub(crate) trait Pause { } impl Pause for crate::transport::boxed::RecvStream { - fn window(&self) -> u64 { + fn window(&mut self) -> u64 { crate::transport::boxed::RecvStream::window(self) } - fn pause(&mut self, taken: u64) -> bool { - crate::transport::boxed::RecvStream::pause(self, taken) + fn shape(&mut self, room: Option, unsealed: u64) { + crate::transport::boxed::RecvStream::shape(self, room, unsealed) + } + + fn pause(&mut self) -> bool { + crate::transport::boxed::RecvStream::pause(self) } fn resume(&mut self) { @@ -129,6 +141,7 @@ impl Reader { want: 0, scratch: Vec::new(), unreleased: 0, + frame: 0, stats: Default::default(), } } @@ -454,8 +467,17 @@ impl Reader { let result = loop { // A writer too far ahead of the group's fastest reader leaves the rest in the - // stream, so flow control pushes back upstream. - let lease = match self.poll_pace(waiter, frame.poll_ready(waiter, self.stream.window())) { + // stream, so flow control pushes back upstream. One partway into a segment reads + // on to its seal first, so a paused stream never holds a segment's credit. + let paced = match self.unreleased { + 0 => { + let window = self.stream.window(); + let paced = frame.poll_ready(waiter, window); + self.poll_pace(waiter, paced, || frame.room(window), frame.unsealed() as u64) + } + _ => Poll::Ready(Ok(())), + }; + let mut lease = match match paced { // The stream alone, so `unreleased` can count while the lease is out. Poll::Ready(Ok(())) => { let version = &self.version; @@ -465,8 +487,7 @@ impl Reader { } Poll::Ready(Err(err)) => break Err(err), Poll::Pending => Poll::Pending, - }; - let mut lease = match lease { + } { Poll::Pending => { if owed > 0 { frame.notify(); @@ -515,6 +536,11 @@ impl Reader { frame.notify(); owed = 0; } + // The transport grants credit from what was taken when it next sends, so the + // room is measured from here, the chunk taken. + let window = self.stream.window(); + let room = frame.room(window).map(|room| room.saturating_sub(self.frame)); + self.stream.shape(room, frame.unsealed() as u64); } if complete { break Ok(()); @@ -543,24 +569,37 @@ impl Reader { where S: Pause, { - self.poll_pace(waiter, group.poll_admit(waiter, header, self.stream.window())) + self.frame = header.info.size; + let window = self.stream.window(); + let paced = group.poll_admit(waiter, header, window); + self.poll_pace(waiter, paced, || group.room(window), 0) } /// Wait out `paced`, the group's pace gate, if the session has room for the credit this /// stream holds while paused (see [`Pause`]); otherwise read on, and the group sheds. + /// Once open, the stream's credit is sized to the `room` the writer has left, at least + /// the `unsealed` rest of its segment. /// /// A paused stream is not read, so it is watched for a reset instead, which fails the /// wait rather than leaving it to the group's reader to notice. - fn poll_pace(&mut self, waiter: &kio::Waiter, paced: Poll<()>) -> Poll> + fn poll_pace( + &mut self, + waiter: &kio::Waiter, + paced: Poll<()>, + room: impl FnOnce() -> Option, + unsealed: u64, + ) -> Poll> where S: Pause, { match paced { Poll::Ready(()) => { self.stream.resume(); + let room = room().map(|room| room.saturating_sub(self.frame)); + self.stream.shape(room, unsealed); Poll::Ready(Ok(())) } - Poll::Pending if self.stream.pause(self.unreleased) => { + Poll::Pending if self.stream.pause() => { let err = ready!(self.stream.poll_reset(waiter)); Poll::Ready(Err(self.version.transport_error(err))) } @@ -630,6 +669,7 @@ impl Reader { want: self.want, scratch: self.scratch, unreleased: self.unreleased, + frame: self.frame, stats: self.stats, } } @@ -1260,22 +1300,28 @@ mod tests { released: usize, /// The pacing window, or the cap when zero. window: u64, - /// The taken bytes each pause reserved for. - paused: Vec, + /// The credit held at each pause. + paused: Vec, + /// The room and unsealed rest each shaping was given. + shaped: Vec<(Option, u64)>, } struct CreditedLease<'a>(&'a mut Credited); impl Pause for Credited { - fn window(&self) -> u64 { + fn window(&mut self) -> u64 { match self.window { 0 => crate::model::pace::AHEAD, window => window, } } - fn pause(&mut self, taken: u64) -> bool { - self.paused.push(taken); + fn shape(&mut self, room: Option, unsealed: u64) { + self.shaped.push((room, unsealed)); + } + + fn pause(&mut self) -> bool { + self.paused.push(self.held); true } } @@ -1367,10 +1413,11 @@ mod tests { assert_eq!(read, data); } - /// A stream paused partway through a segment still holds the credit of what it took of - /// it, so the pause reserves that along with the stream window. + /// A stream paced partway through a segment reads on to its seal before pausing, so a + /// paused stream never holds the credit of a segment it has yet to seal, and its credit + /// stays sized to at least the rest of that segment meanwhile. #[test] - fn a_pause_mid_segment_reserves_what_it_took() { + fn a_stream_reads_on_to_the_seal_before_pausing() { use crate::frame::SEGMENT; const CHUNK: usize = 100_000; @@ -1392,14 +1439,61 @@ mod tests { ..Default::default() }; let mut reader = Reader::new(stream, crate::lite::Version::Lite05); - for _ in 0..SEGMENT / CHUNK { + // One chunk straddles the seal. + for _ in 0..SEGMENT / CHUNK + 1 { reader.stream.chunks.push_back(Bytes::from(vec![0u8; CHUNK])); } let noop = &kio::Waiter::noop(); - assert!(reader.poll_read_frame(noop, &mut frame).is_pending(), "paced mid-frame"); + assert!( + reader.poll_read_frame(noop, &mut frame).is_pending(), + "paced at the seal" + ); let stream = &reader.stream; - assert!(stream.held > 0 && stream.held < SEGMENT, "{} bytes held", stream.held); - assert_eq!(stream.paused, [stream.held as u64], "the pause reserves what it holds"); + assert_eq!(stream.released, SEGMENT, "read on past the window to the seal"); + assert_eq!(stream.paused, [0], "paused holding nothing taken"); + let (room, unsealed) = *stream.shaped.last().expect("shaped as it read"); + assert_eq!(room, Some(0), "no room left"); + assert_eq!(unsealed, SEGMENT as u64, "but the next segment to seal"); + } + + /// While a reader holds the group back, ingest sizes the stream's credit to the room the + /// group has left, a frame short, since it pauses once the next frame does not fit. + #[test] + fn credit_is_shaped_a_frame_short_of_the_room() { + const FRAME: usize = 1000; + const WINDOW: u64 = 100_000; + + let group = crate::group::Info { start: 0 }.produce(); + let _idle = group.consume(); + let mut group = crate::recv::Group::new(group); + let info = crate::frame::Info { + size: FRAME as u64, + timestamp: crate::Timestamp::ZERO, + }; + let stream = Credited { + window: WINDOW, + ..Default::default() + }; + let mut reader = Reader::new(stream, crate::lite::Version::Lite05); + let noop = &kio::Waiter::noop(); + let mut room = None; + for _ in 0..3 { + assert!(matches!( + reader.poll_admit(noop, &group, &info.into()), + Poll::Ready(Ok(())) + )); + let mut frame = group.create_object(info.into()).unwrap(); + reader.stream.chunks.push_back(Bytes::from(vec![0u8; FRAME])); + assert!(matches!(reader.poll_read_frame(noop, &mut frame), Poll::Ready(Ok(())))); + room = frame.room(WINDOW); + group.restore(frame.finish().unwrap()); + } + + let room = room.expect("a reader holds it back"); + assert!(room <= WINDOW - 3 * FRAME as u64, "the frames took room"); + let (shaped, unsealed) = *reader.stream.shaped.last().unwrap(); + assert_eq!(shaped, Some(room - FRAME as u64), "a frame short"); + assert_eq!(unsealed, 0, "between frames"); } } diff --git a/rs/moq-net/src/model/frame.rs b/rs/moq-net/src/model/frame.rs index 89ef4065f7..784db1b603 100644 --- a/rs/moq-net/src/model/frame.rs +++ b/rs/moq-net/src/model/frame.rs @@ -1390,6 +1390,13 @@ impl Producer { .poll_paced(waiter, self.buf.written(Ordering::Acquire) as u64, window) } + /// How many more bytes of this frame the group's writer may take before + /// [`Self::poll_ready`] holds it at `window`, or `None` while nothing would. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn room(&self, window: u64) -> Option { + self.group().room(self.buf.written(Ordering::Acquire) as u64, window) + } + /// Publish what has been written so far, waking consumers parked on the group. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub(crate) fn notify(&self) { diff --git a/rs/moq-net/src/model/group.rs b/rs/moq-net/src/model/group.rs index 40c06bf6cc..53b56227a5 100644 --- a/rs/moq-net/src/model/group.rs +++ b/rs/moq-net/src/model/group.rs @@ -2407,6 +2407,17 @@ impl Producer { } } + /// How many more bytes the writer, `partial` bytes into a frame it has not committed, may + /// take before [`Self::poll_paced`] holds it at `window`, or `None` once nothing would; + /// see [`super::pace::Pace::room`]. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn room(&self, partial: u64, window: u64) -> Option { + if self.handle.state.is_closed() { + return None; + } + self.handle.alive.pace.room(partial, window, self.handle.state.wanted()) + } + /// The non-writing observer used by the cache and built-in sessions. // Sessions need an abortable observer while public callers use Demand. pub(crate) fn handle(&self) -> Handle { diff --git a/rs/moq-net/src/model/pace.rs b/rs/moq-net/src/model/pace.rs index 6aaa986a5f..0fe9e244b3 100644 --- a/rs/moq-net/src/model/pace.rs +++ b/rs/moq-net/src/model/pace.rs @@ -8,7 +8,9 @@ //! //! Ingest's window is the stream receive credit its transport grants, capped at [`AHEAD`] //! (see [`window`]), so a hop holds about what its upstream peer can have in flight on -//! that stream. A local writer, which has no transport, uses [`AHEAD`] itself. +//! that stream; twice that on a transport that shapes a paced stream's credit to the room +//! left, which then buffers in the group what the transport no longer does. A local +//! writer, which has no transport, uses [`AHEAD`] itself. //! //! Progress is in bytes on one scale per group copy: each frame entering the copy's log //! ends where the frames before it, payload and slot, add up to. A frame larger than the @@ -129,6 +131,25 @@ impl Pace { self.paced(self.read.load(Ordering::SeqCst), partial, window) } + /// How many more bytes a writer `partial` bytes into an uncommitted frame may take + /// before it is more than `window` past the fastest reader, or `None` once nothing holds + /// it back: that reader fell off, or the writer is past the window with no reader + /// (`wanted` false), so it reads on and sheds. + /// + /// Measured from where readers got even before one holds the group, since the first to + /// arrive starts there. + pub fn room(&self, partial: u64, window: u64, wanted: bool) -> Option { + let read = self.read.load(Ordering::SeqCst); + if read < self.front.load(Ordering::SeqCst) { + return None; + } + let lead = self.written().saturating_add(partial).saturating_sub(read); + match window.checked_sub(lead) { + None if !wanted => None, + room => Some(room.unwrap_or(0)), + } + } + /// Register `waiter` for the next read or shed, returning how far readers got, read /// after registering so a read in between is never missed; [`Self::paced`] reads the /// front after it too. @@ -504,4 +525,32 @@ mod tests { assert!(flag.woken()); assert_eq!(group.poll_paced(&waiter, 0, AHEAD), Poll::Ready(())); } + + /// The room a writer has left runs from where its fastest reader got, even before one + /// holds the group; it is unbounded once that reader fell off, or past the window with + /// nobody reading, since the writer then reads on. + #[test] + fn room_runs_from_the_fastest_reader() { + let window = 4 * MB as u64; + let group = written(2); + let room = group.room(0, window).expect("within the window"); + assert!(room < window - 2 * MB as u64, "frames cost more than their payload"); + assert_eq!( + group.room(MB as u64, window), + Some(room - MB as u64), + "less what is partly in" + ); + assert_eq!(group.room(0, MB as u64), None, "past the window with nobody reading"); + + let mut reader = group.consume(); + assert_eq!(group.room(0, MB as u64), Some(0), "a reader holds it back"); + read(&mut reader); + read(&mut reader); + assert_eq!(group.room(0, MB as u64), Some(MB as u64), "the reader caught up"); + + let group = written(17); + let _reader = group.consume(); + group.handle().shed(u64::MAX, u64::MAX); + assert_eq!(group.room(0, AHEAD), None, "the reader fell off"); + } } diff --git a/rs/moq-net/src/recv.rs b/rs/moq-net/src/recv.rs index 02cc058273..760204420d 100644 --- a/rs/moq-net/src/recv.rs +++ b/rs/moq-net/src/recv.rs @@ -75,6 +75,15 @@ impl Group { .poll_paced(waiter, header.info.size.saturating_add(metadata), window) } + /// How many more bytes the writer may take, a frame's included, before + /// [`Self::poll_admit`] holds it at `window`, or `None` while nothing would. + pub fn room(&self, window: u64) -> Option { + self.writer + .as_ref() + .expect("group writer is available between frames") + .room(0, window) + } + /// Restore the writer returned by a completed frame. pub fn restore(&mut self, writer: group::Producer) { debug_assert!(self.writer.is_none()); diff --git a/rs/moq-net/src/transport.rs b/rs/moq-net/src/transport.rs index 0ed13b3839..6d676413b5 100644 --- a/rs/moq-net/src/transport.rs +++ b/rs/moq-net/src/transport.rs @@ -144,11 +144,13 @@ impl Metrics for Untracked {} /// stops reading still holds. /// /// A session's ingest stops reading a group's stream while the group's fastest reader is far -/// behind, so flow control pushes back to the publisher. Such a paused stream keeps up to -/// `stream` bytes it has not read, plus what it read of a frame's unsealed segment, counted -/// against the shared `connection` window, so the session pauses only as many streams as fit -/// in half of it; the rest keep reading. See -/// [`poll::Session::receive_window`]. +/// behind, so flow control pushes back to the publisher. Such a paused stream holds the +/// connection credit it was granted, counted against the shared `connection` window, so the +/// session pauses only as many streams as fit in half of it; the rest keep reading. A +/// transport that shapes a stream's credit ([`poll::RecvStream::set_window`]) holds a paused +/// one to a small allowance, which shrinks with each stream the session shapes yet sums to +/// under that half however many there are, so every one pauses; any other holds its `stream` +/// window. See [`poll::Session::receive_window`]. #[derive(Clone, Copy, Debug, PartialEq, Eq)] #[non_exhaustive] pub struct ReceiveWindow { @@ -925,6 +927,19 @@ pub mod poll { Poll::Pending } + /// Let the peer send at most `window` bytes past what the reader took, rather than the + /// stream's full receive credit, returning the connection credit the stream may hold from + /// now on: the bytes that arrived and were not released, and those the peer may still send. + /// + /// A session pacing a stream (see [`Session::receive_window`]) shrinks its window as the + /// group runs out of room, so the stream holds little once paused, and many fit in the + /// connection window. Credit already granted stays granted. `None` when the transport + /// cannot, the default, whose paused streams hold their full stream credit. + fn set_window(&mut self, window: u64) -> Option { + let _ = window; + None + } + /// Poll for the next chunk, up to `max` bytes, or `None` once the peer finished the /// stream. Its credit returns as it is read. /// diff --git a/rs/moq-net/src/transport/boxed.rs b/rs/moq-net/src/transport/boxed.rs index 4639dca791..1a54134fa8 100644 --- a/rs/moq-net/src/transport/boxed.rs +++ b/rs/moq-net/src/transport/boxed.rs @@ -6,6 +6,7 @@ //! awaits them. Each transport only instantiates the thin forwarding impls below. use std::any::Any; +use std::collections::BTreeSet; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Arc, Mutex, OnceLock, Weak}; use std::task::{Poll, ready}; @@ -32,6 +33,7 @@ impl Session { pub(crate) fn new(session: S) -> Self { let inner: Arc> = Arc::new(Shared { paused: AtomicU64::new(0), + slots: Mutex::default(), session, }); Self { @@ -142,30 +144,88 @@ impl Session { /// A boxed transport and the receive credit its paused streams hold. pub(super) struct Shared { - /// The stream credit reserved by streams this session stopped reading to pace a group's + /// The connection credit held by streams this session stopped reading to pace a group's /// writer, at most half the connection window; see [`ReceiveWindow`]. paused: AtomicU64, + /// The slots of the streams this session shapes the credit of; see [`Pacing::allowance`]. + slots: Mutex, session: S, } impl Shared { - /// Reserve the credit of one more paused stream, its stream window and the `taken` bytes it - /// read without returning their credit, unless it would push the paused total over half the - /// connection window, returning how much it reserved. Without a connection window a paused - /// stream holds only its own credit, so it reserves nothing. - fn reserve(&self, taken: u64) -> Option { - let window = self.session.receive_window()?; - let Some(connection) = window.connection else { - return Some(0); - }; - let charge = window.stream.saturating_add(taken).min(connection.get()); - let cap = connection.get() / 2; + /// Reserve `charge` for one more paused stream, unless it would push the paused total over + /// half the `connection` window. + fn reserve(&self, charge: u64, connection: u64) -> bool { + let cap = connection / 2; self.paused .fetch_update(Ordering::AcqRel, Ordering::Acquire, |paused| { paused.checked_add(charge).filter(|total| *total <= cap) }) - .ok()?; - Some(charge) + .is_ok() + } +} + +/// The slots of a session's shaped streams, the lowest free one first, so a session with few +/// streams gives each a large allowance. +#[derive(Default)] +struct Slots { + next: u64, + free: BTreeSet, +} + +impl Slots { + fn take(&mut self) -> u64 { + self.free.pop_first().unwrap_or_else(|| { + self.next += 1; + self.next - 1 + }) + } + + fn give(&mut self, slot: u64) { + self.free.insert(slot); + } +} + +/// How ingest paces a stream under its session's receive windows. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct Pacing { + /// How far the group's writer reads ahead of its fastest reader before pausing, on a + /// transport that cannot shape credit, whose paused streams hold their stream credit. + window: u64, + /// The stream's full receive credit. + stream: u64, + /// What the session's paused streams may hold together, half its connection window; zero + /// without one to share. + cap: u64, +} + +impl Pacing { + fn new(receive: Option) -> Self { + let window = crate::model::pace::window(receive); + let Some(receive) = receive else { + return Self { + window, + stream: 0, + cap: 0, + }; + }; + Self { + window, + stream: receive.stream, + cap: receive.connection.map_or(0, |connection| connection.get() / 2), + } + } + + /// The credit a paced stream in `slot` lets the peer send past the room its group has + /// left, so a paused one holds about this much. + /// + /// Credit once granted cannot be taken back, so a paused stream holds its allowance until + /// it resumes, however many streams pause after it. The cap is therefore split by slot, + /// not by how many are paused: slot `i` gets `cap / ((i + 1) * (i + 2))`, which sums to + /// under the cap over any number of slots, so every stream fits at its own allowance. + fn allowance(&self, slot: u64) -> u64 { + let share = slot.saturating_add(1).saturating_mul(slot.saturating_add(2)); + (self.cap / share).min(self.stream) } } @@ -457,24 +517,36 @@ impl RecvStream { Self(Box::new(Budgeted { stream, session, - held: 0, - window: OnceLock::new(), + held: None, + pacing: OnceLock::new(), + shaped: None, + unshaped: false, + probed: false, + slot: None, })) } /// How far ingest reads this stream ahead of its group's fastest reader before pausing: /// the credit the stream may hold, so a paced hop buffers about what its peer can have - /// in flight on it. - pub(crate) fn window(&self) -> u64 { + /// in flight on it, and on a transport that shapes credit a stream window more, which a + /// paused stream no longer holds in the transport. + pub(crate) fn window(&mut self) -> u64 { self.0.window() } - /// Reserve the receive credit this stream may hold while ingest stops reading it, its stream - /// window and the `taken` bytes read without returning their credit, returning whether it - /// may stop: not once the session's paused streams hold half its connection window, nor on - /// a transport that reports no window. - pub(crate) fn pause(&mut self, taken: u64) -> bool { - self.0.pause(taken) + /// Let the peer send only what the group's writer has `room` for, at least the `unsealed` + /// rest of the segment being written, plus an allowance, or the full stream credit while + /// no reader holds the writer back (`None`), so the stream holds about the allowance once + /// it pauses. See [`poll::RecvStream::set_window`]. + pub(crate) fn shape(&mut self, room: Option, unsealed: u64) { + self.0.shape(room, unsealed) + } + + /// Reserve the connection credit this stream holds while ingest stops reading it, returning + /// whether it may stop: not once the session's paused streams hold half its connection + /// window, nor on a transport that reports no window. + pub(crate) fn pause(&mut self) -> bool { + self.0.pause() } /// Return the credit [`Self::pause`] reserved, as ingest reads the stream again. @@ -487,28 +559,89 @@ impl RecvStream { struct Budgeted { stream: T, session: Budget, - /// The credit reserved while paused, zero while read. - held: u64, - /// The pacing window, read from the session's receive windows when ingest first paces - /// the stream, so one that arrived before admission widened them paces at the wider ones. - window: OnceLock, + /// The credit reserved while paused, `None` while read. + held: Option, + /// Read from the session's receive windows when ingest first paces the stream, so one + /// that arrived before admission widened them paces at the wider ones. + pacing: OnceLock, + /// The window last set with [`poll::RecvStream::set_window`], `None` for the full credit. + shaped: Option, + /// The transport cannot set a stream's window, so a paused stream holds its full credit. + unshaped: bool, + /// Whether the transport has been asked yet to set the stream's window. + probed: bool, + /// The stream's slot among the session's shaped streams, until it drops. + slot: Option, } impl Budgeted { /// Return the credit reserved while paused, if any. fn release(&mut self) { - let held = std::mem::take(&mut self.held); - if held > 0 + if let Some(held) = self.held.take() + && held > 0 && let Some(shared) = self.session.upgrade() { shared.paused.fetch_sub(held, Ordering::AcqRel); } } + + /// The credit this stream lets the peer send past the room its group has left, from its + /// slot, taking one on first use; see [`Pacing::allowance`]. + fn allowance(&mut self, pacing: &Pacing) -> u64 { + let slot = match self.slot { + Some(slot) => slot, + None => { + let Some(shared) = self.session.upgrade() else { + return 0; + }; + let slot = shared.slots.lock().expect("slots poisoned").take(); + *self.slot.insert(slot) + } + }; + pacing.allowance(slot) + } + + /// Give the slot back, once the stream drops or its transport cannot shape credit. + fn free(&mut self) { + if let Some(slot) = self.slot.take() + && let Some(shared) = self.session.upgrade() + { + shared.slots.lock().expect("slots poisoned").give(slot); + } + } + + fn pacing(&self) -> Pacing { + *self.pacing.get_or_init(|| { + let receive = self + .session + .upgrade() + .and_then(|shared| shared.session.receive_window()); + Pacing::new(receive) + }) + } +} + +impl Budgeted { + /// Set the stream's window to `window`, returning the connection credit it may hold, or + /// `None` once the transport turned out not to support it. + fn set_window(&mut self, window: u64) -> Option { + if self.unshaped { + return None; + } + let credit = self.stream.set_window(window); + self.unshaped = credit.is_none(); + if self.unshaped { + self.free(); + } + self.shaped = credit.map(|_| window); + credit + } } impl Drop for Budgeted { fn drop(&mut self) { self.release(); + self.free(); } } @@ -614,8 +747,10 @@ trait ErasedRecv: Send + Sync { fn set_priority(&mut self, priority: i64); fn stop(&mut self, code: u32); fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll; - fn window(&self) -> u64; - fn pause(&mut self, taken: u64) -> bool; + fn set_window(&mut self, window: u64) -> Option; + fn window(&mut self) -> u64; + fn shape(&mut self, room: Option, unsealed: u64); + fn pause(&mut self) -> bool; fn resume(&mut self); } @@ -815,24 +950,89 @@ impl ErasedRecv for Budgeted { poll::RecvStream::poll_reset(&mut self.stream, waiter).map(Error::new) } - fn window(&self) -> u64 { - *self.window.get_or_init(|| { - let receive = self - .session - .upgrade() - .and_then(|shared| shared.session.receive_window()); - crate::model::pace::window(receive) - }) + fn set_window(&mut self, window: u64) -> Option { + poll::RecvStream::set_window(&mut self.stream, window) + } + + fn window(&mut self) -> u64 { + let pacing = self.pacing(); + // Whether the transport shapes credit decides the window, so ask before the first one + // rather than learn it from the first shaping that moves. + if pacing.cap > 0 && !self.probed { + self.probed = true; + Budgeted::set_window(self, pacing.stream); + } + match self.unshaped || pacing.cap == 0 { + true => pacing.window, + // A shaped stream holds a stream window less once paused, so its group holds it + // instead, and a paced reader has as much buffered ahead as before. + false => pacing + .window + .saturating_add(pacing.stream) + .min(crate::model::pace::AHEAD), + } + } + + fn shape(&mut self, room: Option, unsealed: u64) { + let pacing = self.pacing(); + // Nothing is shared without a connection window, so nothing needs to stay small. + if pacing.cap == 0 || self.unshaped { + return; + } + let allowance = self.allowance(&pacing); + let target = match room { + Some(room) => room.max(unsealed).saturating_add(allowance).min(pacing.stream), + None => pacing.stream, + }; + let current = self.shaped.unwrap_or(pacing.stream); + // Credit granted stays granted, so a window left above its target is credit the stream + // holds once paused, and a smaller one always pays for the transport's lock. A larger + // one only does once it moves half an allowance. + if target < current || target - current >= allowance / 2 { + Budgeted::set_window(self, target); + } } - fn pause(&mut self, taken: u64) -> bool { - if self.held > 0 { + fn pause(&mut self) -> bool { + if self.held.is_some() { return true; } - let Some(held) = self.session.upgrade().and_then(|shared| shared.reserve(taken)) else { + let Some(shared) = self.session.upgrade() else { + return false; + }; + let Some(receive) = shared.session.receive_window() else { + return false; + }; + // Without a connection window a paused stream holds only its own credit. + let Some(connection) = receive.connection.map(std::num::NonZeroU64::get) else { + self.held = Some(0); + return true; + }; + let pacing = self.pacing(); + // The least it could hold, checked first, so a stream past the cap reads on without + // taking the transport's lock. + let least = match self.unshaped { + true => receive.stream, + false => 0, + }; + if shared.paused.load(Ordering::Acquire).saturating_add(least) > pacing.cap { return false; + } + // A transport that shapes says what the stream holds; any other holds its full credit. + let credit = match self.unshaped { + true => None, + false => { + let allowance = self.allowance(&pacing); + Budgeted::set_window(self, allowance) + } }; - self.held = held; + let charge = credit.unwrap_or(receive.stream).min(connection); + // Refused, it reads on and sheds, still holding only its shaped credit, so it may pause + // once others resume. + if !shared.reserve(charge, connection) { + return false; + } + self.held = Some(charge); true } @@ -1031,6 +1231,10 @@ impl poll::RecvStream for RecvStream { fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll { self.0.poll_reset(waiter) } + + fn set_window(&mut self, window: u64) -> Option { + self.0.set_window(window) + } } impl Buf for Lease<'_> { @@ -1078,7 +1282,7 @@ mod test { recv } - /// Each paused stream reserves its stream window and what it took of an unsealed segment, + /// On a transport that cannot shape credit, each paused stream reserves its stream window, /// and the session pauses no more than fit in half the connection window; a stream /// resuming or dropped returns its share. #[test] @@ -1089,31 +1293,123 @@ mod test { .with_receive_window(ReceiveWindow::new(std::num::NonZeroU64::new(100), 20)), ); let mut streams: Vec<_> = (0..3).map(|_| accept(&session)).collect(); - assert!(streams[0].pause(0)); - assert!(streams[0].pause(0), "a paused stream reserves once"); - assert!(streams[1].pause(0)); - assert!(!streams[2].pause(0), "a third would hold 60 of 100"); + assert!(streams[0].pause()); + assert!(streams[0].pause(), "a paused stream reserves once"); + assert!(streams[1].pause()); + assert!(!streams[2].pause(), "a third would hold 60 of 100"); streams[0].resume(); - assert!(streams[2].pause(0), "a resumed stream returned its share"); + assert!(streams[2].pause(), "a resumed stream returned its share"); drop(streams.remove(1)); - assert!(accept(&session).pause(0), "so did a dropped one"); - - // A stream partway through a segment also holds the credit of what it took of it. - let session = Session::new( - SinkSession::accepted_bi(gate.consume()) - .with_receive_window(ReceiveWindow::new(std::num::NonZeroU64::new(100), 20)), - ); - let mut first = accept(&session); - assert!(first.pause(25), "holds 45 of 100"); - assert!(!accept(&session).pause(0), "a second would hold 65 of 100"); + assert!(accept(&session).pause(), "so did a dropped one"); // Without a connection window nothing is shared, so every stream may pause. let session = Session::new( SinkSession::accepted_bi(gate.consume()).with_receive_window(ReceiveWindow::new(None, u64::MAX)), ); let mut streams: Vec<_> = (0..3).map(|_| accept(&session)).collect(); - assert!(streams.iter_mut().all(|stream| stream.pause(0))); + assert!(streams.iter_mut().all(|stream| stream.pause())); + } + + /// A stream that records each window it is set to, and holds that much credit once set. + struct Shaping { + windows: Arc>>, + } + + impl poll::RecvStream for Shaping { + type Error = crate::lite::test_transport::SinkError; + type Lease<'a> = poll::BufferedLease<'a>; + + fn poll_lease(&mut self, _: &kio::Waiter, _: usize) -> Poll, Self::Error>> { + Poll::Pending + } + + fn set_priority(&mut self, _: i64) {} + + fn stop(&mut self, _: u32) {} + + fn set_window(&mut self, window: u64) -> Option { + self.windows.lock().unwrap().push(window); + Some(window) + } + } + + const CONNECTION: u64 = 128 * 1024; + const STREAM: u64 = 64 * 1024; + /// What a session's paused streams may hold together in these tests. + const CAP: u64 = CONNECTION / 2; + + fn shaping_session() -> Session { + let gate = kio::Producer::new(true); + Session::new( + SinkSession::accepted_bi(gate.consume()) + .with_receive_window(ReceiveWindow::new(std::num::NonZeroU64::new(CONNECTION), STREAM)), + ) + } + + fn shaping(session: &Session) -> (RecvStream, Arc>>) { + let windows = Arc::new(Mutex::new(Vec::new())); + let stream = Shaping { + windows: windows.clone(), + }; + (RecvStream::budgeted(stream, session.budget()), windows) + } + + /// A stream on a transport that shapes credit lets the peer send what its group has room + /// for plus the allowance of its slot, and pauses holding what the transport says it holds. + #[test] + fn a_shaped_stream_pauses_at_the_credit_it_holds() { + let session = shaping_session(); + let (mut first, first_windows) = shaping(&session); + let (mut stream, windows) = shaping(&session); + assert_eq!( + stream.window(), + 2 * STREAM, + "its group holds what the transport does not" + ); + // The first to shape takes the first slot, the next the second: a sixth of the cap. + first.shape(None, 0); + assert!(first_windows.lock().unwrap().is_empty(), "already at its full credit"); + let allowance = CAP / 6; + stream.shape(Some(10_000), 0); + stream.shape(Some(9_800), 0); + stream.shape(Some(100), 2_000); + stream.shape(Some(100), 2_100); + stream.shape(None, 0); + assert_eq!( + *windows.lock().unwrap(), + [STREAM, 10_000 + allowance, 9_800 + allowance, 2_000 + allowance, STREAM,], + "asked once, then a smaller window always applies, a larger only past half an allowance, and never short of the segment's rest" + ); + + // Each pauses at its allowance: the first a half of the cap, the second a sixth. + assert!(first.pause()); + assert!(stream.pause()); + assert_eq!(*first_windows.lock().unwrap(), [CAP / 2]); + assert_eq!(windows.lock().unwrap().last(), Some(&allowance)); + first.resume(); + stream.resume(); + } + + /// However many streams pause at once, and however they arrive, every one fits: the + /// allowances of any number of slots add up to under the cap, and a stream that ends gives + /// its slot, and so its allowance, to the next. + #[test] + fn any_number_of_shaped_streams_pause() { + let session = shaping_session(); + let mut streams: Vec<_> = (0..1000).map(|_| shaping(&session)).collect(); + for (stream, _) in &mut streams { + assert!(stream.pause(), "every stream fits"); + } + let held = streams.iter().map(|(_, w)| *w.lock().unwrap().last().unwrap()); + assert!(held.sum::() <= CAP, "within the cap"); + + // The lowest free slot goes first: a new stream takes the one just given back. + let (_, windows) = streams.remove(2); + let allowance = *windows.lock().unwrap().last().unwrap(); + let (mut stream, windows) = shaping(&session); + assert!(stream.pause()); + assert_eq!(*windows.lock().unwrap(), [allowance]); } /// A stream paces at the credit its session grants it, so a hop holds about what its @@ -1127,7 +1423,7 @@ mod test { SinkSession::accepted_bi(gate.consume()) .with_receive_window(ReceiveWindow::new(std::num::NonZeroU64::new(100), 20)), ); - assert_eq!(accept(&session).window(), 20); + assert_eq!(accept(&session).window(), 20, "a transport that cannot shape credit"); let session = Session::new( SinkSession::accepted_bi(gate.consume()).with_receive_window(ReceiveWindow::new(None, u64::MAX)), @@ -1317,7 +1613,7 @@ mod test { SinkSession::accepted_bi(gate.consume()) .with_receive_window(ReceiveWindow::new(std::num::NonZeroU64::new(10), 20)), ); - let early = accept(&session); + let mut early = accept(&session); let limits = Limits { receive_window: std::num::NonZeroU64::new(100), }; @@ -1331,7 +1627,7 @@ mod test { fn an_unbounded_stream_never_pauses() { let gate = kio::Producer::new(true); let session = Session::new(SinkSession::accepted_bi(gate.consume())); - assert!(!accept(&session).pause(0)); + assert!(!accept(&session).pause()); let session = Session::new( SinkSession::accepted_bi(gate.consume()) @@ -1339,6 +1635,6 @@ mod test { ); let mut stream = accept(&session); drop(session); - assert!(!stream.pause(0)); + assert!(!stream.pause()); } } diff --git a/rs/moq-quic/README.md b/rs/moq-quic/README.md index a762990064..70174d66d2 100644 --- a/rs/moq-quic/README.md +++ b/rs/moq-quic/README.md @@ -102,6 +102,7 @@ Changes on top of the upstream commit, besides the renames: - **Draining endpoint event**: `EndpointEvent::is_draining` fires once per connection when it stops being active, before the drained event, so the driver's `Endpoint::wait_all_draining` can end a shutdown without sitting out every closing period. - **Late stream limits**: peer transport parameters that raise a stream limit after an open was refused wake `Streams::poll_open`, as `MAX_STREAMS` does. A server accepted before the rest of a multi-packet ClientHello (a post-quantum key share is enough) otherwise leaves a 0.5-RTT `open_uni` parked until the idle timeout. Not offered upstream yet. - **Receive leases** (`streams/lease.rs`, `recv::Stream::poll_lease`), building on [#5675](https://github.com/moq-dev/moq/pull/5675)'s held credit: a lease reads a stream's ordered bytes out of the assembler but keeps their connection credit, counted in `StreamsState::held`, until the reader `advance`s or `take`s past them and `release`s. Stream credit runs from the bytes the reader took rather than leased (`Leased::untaken`), widened to a waiting reader's `want`, so a reader that stops taking holds at most its stream window of bytes it has not taken, besides those it took and has not released. A lease copies the bytes it reads out of their datagram buffers (`Leased::extend`), so a byte of credit never pins a receive batch, whether its reader waits, parks, or takes it. The lease lives in the stream's `Recv`, which a leased stream keeps, with its stream ID, until the application stops it, even past the FIN. When the peer has spent the connection window and no reader can make progress, the stream ranked lowest by `set_priority` (the newest on a tie) is evicted: its bytes and credit are freed, the stream is stopped without `STOP_SENDING`, and its next lease fails with `recv::Error::Evicted`; the application's `stop` then tells the peer and frees the stream once its final size is known. Leased streams that may hold credit are kept in `StreamsState::ranks`, in eviction order, so an eviction costs a logarithm rather than a scan of every stream. `MAX_DATA` goes out for any increase once the peer is within an eighth of the window (`max_data_due`), since what leases release may never add up to the usual threshold. Streams with data their reader has not looked at hold eviction off; they are kept in `StreamsState::unlooked`, so the stuck check stays constant time however many readers wait. `ConnectionStats::credit` reports the leased and available bytes, the streams evicted, and the peak credit spent. A `recv::Lease` owns the chunks it took, so a reader holds no connection borrow while it parses; `restore` or `release` hands it back, and an eviction pass waits while one is out. A port touching `Chunks` or `add_read_credits` has to keep `held` balanced. +- **Per-stream windows** (`recv::Stream::set_window`): a reader may let the peer send less than the stream receive window past what it took, which it re-sets as it goes, and learn the connection credit the stream may still hold. moq-net's ingest shrinks a paced stream's window to the room its group has left, so a stream it pauses holds little and many fit in the connection window. `MAX_STREAM_DATA` goes out once credit grows by an eighth of the stream's own window. Not offered upstream: no quinn reader pauses streams. - **Owned datagrams**: a received datagram is copied out of its packet when queued, since `datagram_receive_buffer_size` charges only its payload and a slice keeps the whole receive batch alive. quinn has the same gap; not offered upstream yet. - **Stream credit on stop**: `recv::Stream::stop` on a stream whose final size is already known (finished or reset) frees it and queues `MAX_STREAMS` at once, as reading the reset does, instead of leaving the credit until the next packet arrives. Not offered upstream yet; quinn `main` has the same gap. - **Drop resets** (`quic::driver::send::Stream`): a send stream dropped before `finish` is reset with code 0, or with the peer's `STOP_SENDING` code, instead of quinn's implicit `finish`, so a cancelled writer never ends a truncated stream with a FIN. This matches moq-uring's driver and moq-net's transport contract. `quic::web` resets after the WebTransport header with the mapped code. quinn's `stream_stopped_2` finishes its stream before dropping it. Not offered upstream: it reverses a documented quinn behaviour. diff --git a/rs/moq-quic/src/connection/streams/lease.rs b/rs/moq-quic/src/connection/streams/lease.rs index b3913df4d8..f60f64959a 100644 --- a/rs/moq-quic/src/connection/streams/lease.rs +++ b/rs/moq-quic/src/connection/streams/lease.rs @@ -31,7 +31,7 @@ pub(super) struct Leased { lent: u64, /// Bytes whose connection credit is withheld: `len`, plus bytes taken or advanced past but /// not yet released. - held: u64, + pub(super) held: u64, /// How the stream ended, once the last byte is leased. end: Option, priority: i64, @@ -498,6 +498,30 @@ impl Stream<'_> { self.queue_max_stream_data() } + /// Let the peer send at most `window` bytes past what the reader took, rather than the + /// connection's stream receive window, returning the connection credit the stream may hold + /// from now on: what arrived and was not released, and what the peer may still send. + /// + /// A reader about to stop taking shrinks its window first, so the stream holds little + /// while it waits. Credit already granted stays granted, so a smaller window holds only + /// once the reader takes what was granted before it. A larger window, up to the + /// connection's, grants the added credit once it is an eighth of the window, as a read + /// would, or at once when a reader waits for it. A reader waiting on a larger unit still + /// gets room for it. + pub fn set_window(&mut self, window: u64) -> u64 { + let stream_receive_window = self.state.stream_receive_window; + let Some(slot) = self.state.recv.get_mut(&self.id) else { + return 0; + }; + // A stream this side opened has no receive state until something arrives or reads it. + let recv = get_or_insert_recv(slot, self.state.initial_stream_receive_window); + recv.set_window(window); + let credit = recv.credit(stream_receive_window); + let transmit = self.queue_max_stream_data(); + self.transmit(transmit); + credit + } + /// Rank this stream for eviction: when the connection window is stuck, the stream with the /// lowest priority goes first, and the newest of those that tie. pub fn set_priority(&mut self, priority: i64) { diff --git a/rs/moq-quic/src/connection/streams/recv.rs b/rs/moq-quic/src/connection/streams/recv.rs index e5aec58975..c175d6144c 100644 --- a/rs/moq-quic/src/connection/streams/recv.rs +++ b/rs/moq-quic/src/connection/streams/recv.rs @@ -159,6 +159,9 @@ pub(super) struct Recv { pub(super) reader: Reader, /// What a reader leased, which lives as long as the stream does. pub(super) lease: Leased, + /// The credit its reader lets the peer send past what it took, when smaller than the + /// connection's stream receive window; see [`Stream::set_window`](super::lease). + window: Option, } /// Whether the application still reads a receive stream @@ -183,6 +186,7 @@ impl Recv { data_end: 0, reader: Reader::Reading, lease: Leased::default(), + window: None, }) } @@ -277,11 +281,14 @@ impl Recv { /// transmission of the value is recommended. If the boolean value is /// `false` the new window should only be transmitted if a previous transmission /// had failed. - pub(super) fn max_stream_data(&mut self, stream_receive_window: u64) -> (u64, bool) { + pub(super) fn max_stream_data(&self, stream_receive_window: u64) -> (u64, bool) { // Credit runs from what the reader took, not what it leased: a reader that stops taking // holds at most its window of bytes it has not taken. One waiting on a unit larger than // the window gets room for the unit, which the connection window bounds. let taken = self.assembler.bytes_read() - self.lease.untaken(); + let stream_receive_window = self + .window + .map_or(stream_receive_window, |window| window.min(stream_receive_window)); let window = stream_receive_window.max(self.lease.wanted()); // A window shrunk since the last announcement cannot take back credit already granted. let max_stream_data = (taken + window).max(self.sent_max_stream_data); @@ -297,10 +304,33 @@ impl Recv { // takes in the meantime would add to it. let diff = max_stream_data - self.sent_max_stream_data; let short = self.sent_max_stream_data < taken + self.lease.wanted(); - let transmit = self.can_send_flow_control() && (diff >= (stream_receive_window / 8) || short); + // At least a byte, so a tiny window never resends an unchanged limit. + let threshold = (stream_receive_window / 8).max(1); + let transmit = self.can_send_flow_control() && (diff >= threshold || short); (max_stream_data, transmit) } + /// Let the peer send at most `window` bytes past what the reader took, or the connection's + /// stream receive window if that is smaller. + pub(super) fn set_window(&mut self, window: u64) { + self.window = Some(window); + } + + /// The connection credit this stream may hold under `stream_receive_window`: the bytes that + /// arrived and were not released, and those the peer may still send. + pub(super) fn credit(&self, stream_receive_window: u64) -> u64 { + let (max_stream_data, _) = self.max_stream_data(stream_receive_window); + let mut limit = max_stream_data.max(self.sent_max_stream_data); + if let Some(size) = self.final_offset() { + limit = limit.min(size); + } + match self.reader { + Reader::Reading => limit.saturating_sub(self.end) + self.unread_credit() + self.lease.held, + // It keeps no data, and the peer's further bytes return their credit as they arrive. + Reader::Evicted | Reader::Stopped => self.lease.held, + } + } + /// Records that a `MAX_STREAM_DATA` announcing a certain window was sent /// /// This will suppress enqueuing further `MAX_STREAM_DATA` frames unless @@ -566,7 +596,7 @@ impl<'a> Chunks<'a> { let mut should_transmit = self.streams.queue_max_stream_id(self.pending); // If the stream hasn't finished, we may need to issue stream-level flow control credit - if let ChunksState::Readable(mut rs) = state { + if let ChunksState::Readable(rs) = state { let (_, max_stream_data) = rs.max_stream_data(self.streams.stream_receive_window); should_transmit |= max_stream_data; if max_stream_data { diff --git a/rs/moq-quic/src/tests/mod.rs b/rs/moq-quic/src/tests/mod.rs index aee620d558..9e3abe99b6 100644 --- a/rs/moq-quic/src/tests/mod.rs +++ b/rs/moq-quic/src/tests/mod.rs @@ -7161,6 +7161,109 @@ fn leases_hold_one_stream_window() { assert_eq!(pair.server_recv(server_ch, s).lease(0).err(), Some(recv::Error::Closed)); } +/// A reader can let the peer send less than the stream window past what it took, so a +/// reader about to stop holds less: credit already granted stays granted, a smaller window +/// limits what is granted next, and a larger one grants at once. Each change reports the +/// connection credit the stream may hold. +#[test] +fn a_smaller_window_limits_the_next_grant() { + const WINDOW: usize = 4000; + const STREAM: usize = 1000; + let _guard = subscribe(); + let mut transport = transport::Config::default(); + transport.limits.receive_window = NonZeroU64::new(WINDOW as u64); + transport.limits.stream_receive_window = NonZeroU64::new(STREAM as u64); + let mut pair = Pair::new( + Default::default(), + server::Config { + transport, + ..server_config() + }, + ); + let chs = pair.connect(); + let (client_ch, server_ch) = chs; + let s = pair + .client_streams(client_ch) + .open(Dir::Uni, Default::default()) + .unwrap(); + let msg = vec![0xAB; WINDOW]; + let all = msg.len(); + let mut sent = 0; + let take_all = |pair: &mut Pair| { + let mut recv = pair.server_recv(server_ch, s); + let mut lease = recv.lease(0).unwrap(); + lease.advance(lease.remaining()); + recv.release(lease); + }; + + assert_eq!(pump_leased(&mut pair, chs, s, (&msg, 100), &mut sent), 100); + take_all(&mut pair); + assert_eq!( + pair.server_recv(server_ch, s).set_window(200), + (STREAM - 100) as u64, + "the first stream window stays granted" + ); + assert_eq!(pump_leased(&mut pair, chs, s, (&msg, all), &mut sent), STREAM - 100); + + take_all(&mut pair); + assert_eq!( + pump_leased(&mut pair, chs, s, (&msg, all), &mut sent), + 200, + "only the smaller window" + ); + assert_eq!(pair.server_recv(server_ch, s).set_window(200), 200, "all of it arrived"); + + assert_eq!(pair.server_recv(server_ch, s).set_window(STREAM as u64), STREAM as u64); + assert_eq!( + pump_leased(&mut pair, chs, s, (&msg, all), &mut sent), + STREAM - 200, + "a larger window grants at once" + ); +} + +/// A window set on a bidirectional stream this side opened, before anything arrived on it, +/// holds once the first grant is spent. +#[test] +fn a_window_holds_on_a_stream_opened_locally() { + const STREAM: usize = 1000; + let _guard = subscribe(); + let mut client = client_config(); + client.transport.limits.stream_receive_window = NonZeroU64::new(STREAM as u64); + let mut pair = Pair::default(); + let (client_ch, server_ch) = pair.connect_with(client); + let s = pair + .client_streams(client_ch) + .open(Dir::Bi, Default::default()) + .unwrap(); + assert_eq!( + pair.client_recv(client_ch, s).set_window(200), + STREAM as u64, + "the first stream window stays granted" + ); + + pair.client_send(client_ch, s).write(&mut &b"x"[..]).unwrap(); + pair.drive(); + assert_matches!(pair.server_streams(server_ch).accept(Dir::Bi), Some(stream) if stream == s); + let msg = vec![0xAB; 2 * STREAM]; + let mut sent = 0; + let mut send = |pair: &mut Pair| { + let before = sent; + while let Poll::Ready(Ok(n)) = pair.server_send(server_ch, s).try_write(&mut &msg[sent..]) { + sent += n; + } + pair.drive(); + sent - before + }; + assert_eq!(send(&mut pair), STREAM); + + let mut recv = pair.client_recv(client_ch, s); + let mut lease = recv.lease(0).unwrap(); + lease.advance(lease.remaining()); + recv.release(lease); + pair.drive(); + assert_eq!(send(&mut pair), 200, "only the smaller window"); +} + /// Lease `want` bytes of `s` once they arrive, driving the pair until they do, and hand them /// back unread. fn await_unit( diff --git a/rs/moq-relay/tests/integration/quic_sim.rs b/rs/moq-relay/tests/integration/quic_sim.rs index f130593f1f..7d5c71baed 100644 --- a/rs/moq-relay/tests/integration/quic_sim.rs +++ b/rs/moq-relay/tests/integration/quic_sim.rs @@ -236,46 +236,90 @@ const CHUNK: usize = 8 * 1024; /// connection window fills with bytes nobody takes and the transport evicts the stream. #[test] fn a_group_carries_a_bulk_transfer_losslessly() { - memory::run(|| async { - let network = Network::new(0); - let auth = memory::public(&["**"], &[], &[]); - let cluster = memory::start(memory::cluster(), moq_tokio::connect::Config::default()).await; - let relay = serve(&network, SocketAddr::new(ip(1), PORT), &cluster, &auth).await; - - let origin = moq_tokio::origin::spawn(); - let broadcast = origin - .route("bulk", moq_net::origin::RouteOptions::default()) - .expect("route"); - let mut track = broadcast - .broadcast() - .create_track("data", moq_net::track::Info::default()) - .expect("track"); - broadcast.announce().expect("announce"); - let _publisher = client(&network, PUBLISHER, &relay) - .with_publisher(&origin) - .connect(url(&relay)); - - let viewer = moq_tokio::origin::spawn(); - let _subscriber = client(&network, SUBSCRIBER, &relay) - .with_subscriber(viewer.clone()) - .connect(url(&relay)); - - let write = async { - while track.subscription_changed().await.unwrap().is_none() {} - let mut group = track.append_group(moq_net::Timestamp::unscaled(0)).unwrap(); - let info = moq_net::frame::Info { - size: CHUNK as u64, - timestamp: moq_net::Timestamp::unscaled(0), - }; - for index in 0..BULK / CHUNK { - group.ready(&info).await; - group - .write_frame(moq_net::Timestamp::unscaled(0), vec![index as u8; CHUNK]) - .unwrap(); - } - group.finish().unwrap(); + memory::run(|| bulk(1, CHUNK, BULK)); +} + +/// Many groups on one connection each carry a bulk transfer without loss, every one read +/// slower than it is written, so every hop has all of them paused at once. A group that +/// could not pause would read on and shed past the live tail before its reader got far. +#[test] +fn concurrent_groups_carry_bulk_transfers_losslessly() { + memory::run(|| bulk(SESSIONS, 64 * 1024, 1024 * 1024)); +} + +/// [`concurrent_groups_carry_bulk_transfers_losslessly`] at [`MANY`] sessions, past what any +/// fixed credit per paused stream fits in half the connection window. Every one still +/// pauses, since a paused stream holds a share of that half set by its slot, not by how many +/// pause. +#[test] +#[ignore = "a hundred bulk transfers, about a minute in debug: run in release by `just rs quic-sim`"] +fn many_groups_carry_bulk_transfers_losslessly() { + memory::run(|| bulk(MANY, 64 * 1024, 1024 * 1024)); +} + +/// How many bulk transfers share one connection in +/// [`concurrent_groups_carry_bulk_transfers_losslessly`]: well past the two that pause when +/// each paused stream holds its whole stream window. +const SESSIONS: usize = 12; + +/// How many share one in [`many_groups_carry_bulk_transfers_losslessly`]. Past a couple of +/// hundred, the frames in flight at once each hold credit until they seal, and that fills +/// the connection window whatever the paused streams hold. +const MANY: usize = 100; + +/// Write `tracks` tracks of one group each, [`BULK`] bytes in `chunk`-byte frames, through +/// one relay, and read the first `read` bytes of each slower than they are written, +/// asserting every one arrives. +async fn bulk(tracks: usize, chunk: usize, read: usize) { + let network = Network::new(0); + let auth = memory::public(&["**"], &[], &[]); + let cluster = memory::start(memory::cluster(), moq_tokio::connect::Config::default()).await; + let relay = serve(&network, SocketAddr::new(ip(1), PORT), &cluster, &auth).await; + + let origin = moq_tokio::origin::spawn(); + let broadcast = origin + .route("bulk", moq_net::origin::RouteOptions::default()) + .expect("route"); + let sources: Vec<_> = (0..tracks) + .map(|index| { + broadcast + .broadcast() + .create_track(format!("data{index}"), moq_net::track::Info::default()) + .expect("track") + }) + .collect(); + broadcast.announce().expect("announce"); + let _publisher = client(&network, PUBLISHER, &relay) + .with_publisher(&origin) + .connect(url(&relay)); + + let viewer = moq_tokio::origin::spawn(); + let _subscriber = client(&network, SUBSCRIBER, &relay) + .with_subscriber(viewer.clone()) + .connect(url(&relay)); + + let write = |mut track: moq_net::track::Producer| async move { + while track.subscription_changed().await.unwrap().is_none() {} + let mut group = track.append_group(moq_net::Timestamp::unscaled(0)).unwrap(); + let info = moq_net::frame::Info { + size: chunk as u64, + timestamp: moq_net::Timestamp::unscaled(0), }; - let read = async { + for index in 0..BULK / chunk { + group.ready(&info).await; + group + .write_frame(moq_net::Timestamp::unscaled(0), vec![index as u8; chunk]) + .unwrap(); + // Other tasks run between frames, as on a real runtime. + kio::clock::sim::yield_now().await; + } + group.finish().unwrap(); + // Held until the reads end: dropping a track unfinished aborts it. + futures::future::pending::<()>().await; + }; + let read = |name: String| { + let viewer = viewer.clone(); + async move { let remote = loop { match viewer.consume().request("bulk", None).await { Ok(remote) => break remote, @@ -284,7 +328,7 @@ fn a_group_carries_a_bulk_transfer_losslessly() { } }; let consumer = remote - .track("data") + .track(&name) .unwrap() .subscribe( moq_net::track::Subscription::default() @@ -294,27 +338,36 @@ fn a_group_carries_a_bulk_transfer_losslessly() { .unwrap(); let mut ordered = consumer.ordered(); let mut group = ordered.next_group().await.unwrap().expect("a group"); - let mut read = 0; - while let Some(frame) = group - .read_frame() - .await - .unwrap_or_else(|err| panic!("after {read} bytes: {err}")) - { - let index = read / CHUNK; - assert_eq!(frame.payload.len(), CHUNK, "frame {index} is cut short"); + for index in 0..read / chunk { + let frame = group + .read_frame() + .await + .unwrap_or_else(|err| panic!("{name} frame {index}: {err}")) + .unwrap_or_else(|| panic!("{name} ended at frame {index}")); + assert_eq!(frame.payload.len(), chunk, "{name} frame {index} is cut short"); assert!( frame.payload.iter().all(|&byte| byte == index as u8), - "frame {index} is corrupted" + "{name} frame {index} is corrupted" ); - read += CHUNK; // A consumer draining at about 64 MB/s, slower than the writer. - kio::clock::sleep(Duration::from_micros(CHUNK as u64 / 64)).await; + kio::clock::sleep(Duration::from_micros(chunk as u64 / 64)).await; } - read - }; - let read = kio::clock::timeout(Duration::from_secs(600), async { futures::join!(write, read).1 }) - .await - .expect("the transfer stalled"); - assert_eq!(read, BULK); - }); + // A group read whole ends cleanly right after, never with more or a failure. + if read == BULK { + let end = group + .read_frame() + .await + .unwrap_or_else(|err| panic!("{name} end: {err}")); + assert!(end.is_none(), "{name} carries more than {BULK} bytes"); + } + } + }; + let writes = futures::future::join_all(sources.into_iter().map(write)); + let reads = futures::future::join_all((0..tracks).map(|index| read(format!("data{index}")))); + kio::clock::timeout(Duration::from_secs(600), async { + futures::pin_mut!(writes, reads); + futures::future::select(writes, reads).await; + }) + .await + .expect("the transfer stalled"); } diff --git a/rs/moq-tokio/src/quic.rs b/rs/moq-tokio/src/quic.rs index 1c06ea06f2..b8238f4fd0 100644 --- a/rs/moq-tokio/src/quic.rs +++ b/rs/moq-tokio/src/quic.rs @@ -140,8 +140,9 @@ pub const DEFAULT_MAX_STREAMS: u64 = 10_000; /// Bounds how much unread data and incomplete messages one peer can make us buffer across /// all of its streams, which the backends otherwise leave unlimited. It has to be at least moq-net's /// `transport::MIN_RECEIVE_WINDOW`, one 1 MiB segment. 4 MiB sustains about 270 Mbps at a -/// 100 ms RTT, room for any one viewer or publisher, and lets a session pause two group -/// streams for slow readers (half the window, a stream window each). +/// 100 ms RTT, room for any one viewer or publisher, and lets a session pause any number of +/// group streams for slow readers: half the window is split among them, the first taking the +/// most. pub const DEFAULT_RECEIVE_WINDOW: u64 = 4 << 20; const _: () = assert!(DEFAULT_RECEIVE_WINDOW >= moq_net::transport::MIN_RECEIVE_WINDOW); @@ -149,7 +150,8 @@ const _: () = assert!(DEFAULT_RECEIVE_WINDOW >= moq_net::transport::MIN_RECEIVE_ /// /// Stream credit returns as bytes are read, whether or not their message is complete, /// so this only paces one stream: enough for a single group stream to reach about -/// 67 Mbps at a 100 ms RTT. It is also what a stream paused for a slow reader holds. +/// 67 Mbps at a 100 ms RTT. A stream paced for a slow reader holds less: its credit +/// shrinks to the room its group has left. pub const DEFAULT_STREAM_RECEIVE_WINDOW: u64 = 1 << 20; /// Default cap on unacknowledged outgoing data, used when [`Config::send_window`] is unset. diff --git a/rs/moq-tokio/src/quic/driver/recv.rs b/rs/moq-tokio/src/quic/driver/recv.rs index 17a9dbbbbc..15424fcb03 100644 --- a/rs/moq-tokio/src/quic/driver/recv.rs +++ b/rs/moq-tokio/src/quic/driver/recv.rs @@ -198,6 +198,13 @@ impl Stream { conn.inner.recv_stream(self.reader.id).set_priority(priority); } + /// Let the peer send at most `window` bytes past what the reader took, returning the + /// connection credit the stream may hold. See `moq_quic`'s `RecvStream::set_window`. + pub fn set_window(&mut self, window: u64) -> u64 { + let mut conn = self.reader.conn.state.lock(); + conn.inner.recv_stream(self.reader.id).set_window(window) + } + /// Stop accepting data /// /// Discards unread data and notifies the peer to stop transmitting. diff --git a/rs/moq-tokio/src/quic/web/recv.rs b/rs/moq-tokio/src/quic/web/recv.rs index 1ed5ab5725..160356ab43 100644 --- a/rs/moq-tokio/src/quic/web/recv.rs +++ b/rs/moq-tokio/src/quic/web/recv.rs @@ -71,6 +71,13 @@ impl Stream { } } + /// Let the peer send at most `window` bytes past what the reader took, returning the + /// connection credit the stream may hold, none once stopped. See + /// [`recv::Stream::set_window`]. + pub fn set_window(&mut self, window: u64) -> u64 { + self.inner.as_mut().map_or(0, |inner| inner.set_window(window)) + } + /// Tell the other end to stop sending data with the given error code. See /// [`recv::Stream::stop`]. This is a u32 with WebTransport since it shares the error /// space with HTTP/3. A raw QUIC session sends the code as is. Does nothing once stopped. @@ -223,6 +230,9 @@ impl moq_net::transport::poll::RecvStream for Stream { fn set_priority(&mut self, priority: i64) { Self::set_priority(self, priority); } + fn set_window(&mut self, window: u64) -> Option { + Some(Self::set_window(self, window)) + } fn stop(&mut self, code: u32) { Self::stop_mut(self, code); } diff --git a/rs/moq-uring/src/quic/driver/recv.rs b/rs/moq-uring/src/quic/driver/recv.rs index c0049793df..663a9607d5 100644 --- a/rs/moq-uring/src/quic/driver/recv.rs +++ b/rs/moq-uring/src/quic/driver/recv.rs @@ -185,6 +185,12 @@ impl Stream { .set_priority(priority); } + /// Let the peer send at most `window` bytes past what the reader took, returning the + /// connection credit the stream may hold. See `moq_quic`'s `RecvStream::set_window`. + pub fn set_window(&mut self, window: u64) -> u64 { + self.shared.conn.borrow_mut().recv_stream(self.id).set_window(window) + } + /// Stop accepting data. /// /// Discards unread data and notifies the peer to stop transmitting. diff --git a/rs/moq-uring/src/quic/web/recv.rs b/rs/moq-uring/src/quic/web/recv.rs index 3179e9bdc6..3ef050bbc8 100644 --- a/rs/moq-uring/src/quic/web/recv.rs +++ b/rs/moq-uring/src/quic/web/recv.rs @@ -69,6 +69,13 @@ impl Stream { } } + /// Let the peer send at most `window` bytes past what the reader took, returning the + /// connection credit the stream may hold, none once stopped. See + /// [`recv::Stream::set_window`]. + pub fn set_window(&mut self, window: u64) -> u64 { + self.inner.as_mut().map_or(0, |inner| inner.set_window(window)) + } + /// Tell the other end to stop sending data with the given error code. See /// [`recv::Stream::stop`]. This is a u32 with WebTransport since it shares the error /// space with HTTP/3. A raw QUIC session sends the code as is. Does nothing once stopped. @@ -212,6 +219,9 @@ impl moq_net::transport::poll::RecvStream for Stream { fn set_priority(&mut self, priority: i64) { Self::set_priority(self, priority); } + fn set_window(&mut self, window: u64) -> Option { + Some(Self::set_window(self, window)) + } fn stop(&mut self, code: u32) { Self::stop_mut(self, code); } diff --git a/rs/moq-uring/src/transport/local.rs b/rs/moq-uring/src/transport/local.rs index 04395615f8..7ae897312e 100644 --- a/rs/moq-uring/src/transport/local.rs +++ b/rs/moq-uring/src/transport/local.rs @@ -176,6 +176,10 @@ impl poll::RecvStream for Local { self.0.set_priority(priority) } + fn set_window(&mut self, window: u64) -> Option { + self.0.set_window(window) + } + fn stop(&mut self, code: u32) { self.0.stop(code) } diff --git a/rs/qmux/src/session.rs b/rs/qmux/src/session.rs index 6d4766d94a..1a11ba99b0 100644 --- a/rs/qmux/src/session.rs +++ b/rs/qmux/src/session.rs @@ -731,6 +731,13 @@ impl poll::RecvStream for RecvStream { self.inner.lock().conn.recv_stream(self.id).set_priority(priority); } + fn set_window(&mut self, window: u64) -> Option { + if self.end != RecvEnd::Open { + return Some(0); + } + Some(self.inner.lock().conn.recv_stream(self.id).set_window(window)) + } + fn stop(&mut self, code: u32) { if self.end != RecvEnd::Open { return;