diff --git a/doc/bin/relay/config.md b/doc/bin/relay/config.md index 441352ce03b..cb32e1f7e8a 100644 --- a/doc/bin/relay/config.md +++ b/doc/bin/relay/config.md @@ -391,6 +391,15 @@ newest 32 MB. A reader of a track's newest group that reaches a shed frame asks for a new group on tracks that opt into refresh. Any other reader of a shed frame sees `Evicted`. +While a group has readers, the relay receives at most 16 MB of it past the +fastest one, then stops reading its stream until that reader catches up, so +flow control slows the upstream hop and the publisher. A slower reader is never +waited on: one that trails the fastest by more than the rest of the 32 MB +reaches a shed frame. A group nobody reads keeps arriving and shedding. +A stopped stream keeps its receive credit, so the relay stops reading at most as +many streams per connection as fit in half of `quic.receive_window`; past that, +groups keep arriving and shedding as if unpaced. + ## \[stats] ```toml diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md index 4e5b157ef7d..27cbea25374 100644 --- a/doc/concept/moq-lite.md +++ b/doc/concept/moq-lite.md @@ -383,6 +383,18 @@ Put together, a conference might use: Under light congestion video drops the tail of a group; under heavy congestion video stops and audio lags by at most 500 ms. No protocol change, just knobs. +A long group is paced by its fastest reader instead. Each hop receives at most +16 MB of a group past its fastest reader, then stops reading the group's stream, +so stream flow control pushes back hop by hop to the publisher, and a reader +that keeps up loses nothing. A slower reader is never waited on: one that trails +the fastest by more than the group's 32 MB tail reaches a shed frame, and asks +for a new group on tracks that opt into [refresh](#refresh) or ends with +`Evicted`. A hop with no reader keeps receiving and shedding, so pacing never +stalls an unwatched track, and the publisher's own writes never wait unless it +asks to. A stopped stream keeps its receive credit, so a hop stops at most as +many streams per connection as fit in half its connection window, and the rest +keep receiving and shedding. + ### Refresh A publisher can opt a track into refresh by setting its byte ratio. A new diff --git a/quest/m1/unbounded-groups/fastest-reader.md b/quest/m1/unbounded-groups/fastest-reader.md index 4c87b863a70..71c725010e6 100644 --- a/quest/m1/unbounded-groups/fastest-reader.md +++ b/quest/m1/unbounded-groups/fastest-reader.md @@ -73,6 +73,18 @@ Decided 2026-10-28 (maintainer, after an audit of the plan): credit, in Rust and JS. Test it: enough parked groups to fill the connection window, with TRACK_INFO still arriving and the groups then read. +Decided 2026-10-30 (maintainer, in the +[moq-dev/moq#7388](https://github.com/moq-dev/moq/pull/7388) iteration): + +- moq-net caps the receive credit paused groups hold per session at about half + the connection window, read live from the transport + (`poll::Session::receive_window`). A group that would push the total past + the cap keeps reading and shedding instead of pausing. This stands until the + [receive pool](/quest/m2/receive-pool.md) folds paused credit into its + per-worker budget. +- The readiness future is `group::Producer::poll_ready` and `async fn ready()`, + pending until a reader exists and the 16 MB window has room. + Prior art splits the same way (2026-10-28 survey): Varnish streaming, ATS read-while-writer, and WHATWG `tee()` fan out at the writer's pace and buffer everything; Varnish `transit_buffer`, nginx unbuffered proxying, and ATS flow @@ -80,15 +92,48 @@ control pace a single client; Node `pipe()` to several destinations paces the slowest. This is `tee()` with a bounded buffer that drops the branch that falls off it ([whatwg/streams#1235](https://github.com/whatwg/streams/issues/1235)). -Open, for the implementer to measure and record: the readiness API's shape, -which is an API decision to bring to the maintainer before coding. qmux transports carry the same rule through their -own stream credit ([qmux receive lease](/quest/m2/qmux-receive-lease.md)). +Landed in [moq-dev/moq#7388](https://github.com/moq-dev/moq/pull/7388) (Rust, +session ingest only): readers report the byte end of each frame they take +(`Pace` in `rs/moq-net/src/model/pace.rs`), and the shared transport reader's +payload loop (`coding::Reader::poll_read_frame`) waits on +`group::Producer::poll_paced` before taking more, and admits a frame only once +its declared size fits the window (`coding::Reader::poll_admit`), so moq-lite +and moq-transport ingest both pause past the 16 MB window. A reader that took every +committed frame counts as waiting on the writer, so a frame larger than the +window never stalls it. A paused stream is not read, so it is watched for a +reset instead (`poll::RecvStream::poll_reset`, which moq-quic answers through +moq-tokio and moq-uring without taking bytes). Tests cover one hop on lite-06, +lite-07-wip, and moq-transport-22: an idle reader holds the relay copy near the +window and then receives every frame, a slow reader ends `Evicted` while the +fast one keeps up, a session past its credit cap reads on, and a publisher's +reset ends a paused relay copy. The same PR landed the credit +cap (each paused stream reserves its stream window from a per-session budget +in `rs/moq-net/src/transport/boxed.rs`; moq-quic reports its windows through +moq-tokio and moq-uring, and a transport that reports none never pauses) and +the public readiness future. + +What remains: -Tests: a relay chain where a fast reader paces a publisher that writes faster -than the link, with memory held near the window; a second, slower reader ending -`Evicted` without slowing the fast one; a track with no readers never blocking -its publisher; Rust and JS parity. Benchmark the hot path in `group.rs` and -record any regression. +- Receive-window reporting for the transports that report none yet (quinn, + iroh, qmux, WebTransport in the browser, and the FFI adapters), whose ingest + never pauses until they do. Each must also answer + `poll::RecvStream::poll_reset`, or a reset goes unseen while paused. +- A sized readiness check for local writers: `group::Producer::ready()` + waits for room but not for the next frame's size, so a local frame larger + than the room left still sheds what the egress has yet to send. +- JS parity. +- A two-relay chain test with memory measured near the window. +- A quiet-host comparison against the base of `group_read_frames`, + `group_contended`, and `group_relay_frames`, plus `group_paced` (a writer + parked at the window, over 1, 8, and 64 readers), which the first slice + added. The cursor's report is one relaxed load per frame or batch, and a + `fetch_max` only for the fastest reader. On the busy host that landed it, + runs swung by up to 50% either way between repeats. +- Fills that write a whole batch of groups at once evict their own earlier + groups before the reader takes them (the follow-up from + [moq-dev/moq#7339](https://github.com/moq-dev/moq/pull/7339)). Pacing within + a group does not cover this; decide whether it belongs here or with + [flights](/quest/m1/unbounded-groups/flights.md). -Docs: the moq-lite concept page's congestion section and the relay cache docs -change inline. +qmux transports carry the same rule through their own stream credit +([qmux receive lease](/quest/m2/qmux-receive-lease.md)). diff --git a/quest/m2/receive-pool.md b/quest/m2/receive-pool.md index ee1333748f4..25af3e931dc 100644 --- a/quest/m2/receive-pool.md +++ b/quest/m2/receive-pool.md @@ -46,6 +46,15 @@ Open, for the implementer to bring to the maintainer: to move to the cache without a window where the bytes are billed twice or not at all. +Idea for later (maintainer, 2026-10-30, from the +[fastest-reader pacing](/quest/m1/unbounded-groups/fastest-reader.md) iteration +in [moq-dev/moq#7388](https://github.com/moq-dev/moq/pull/7388)): instead of +moq-quic granting `MAX_STREAM_DATA` automatically as a stream is read, the +application could pause a stream, or grant it an explicit +`allow(fastest + 16 MB)`, so flow control never grants credit past the pacing +window. Paused streams would then hold no credit beyond that window, replacing +moq-net's per-session cap on paused credit. + Lessons from other stacks: QUICHE frees 8 KiB blocks as they are consumed, msquic never advertises credit it could not back, and quinn and lsquic both let a slice pin its packet buffer. #5878 already handles the last with compaction. diff --git a/rs/moq-net/benches/group.rs b/rs/moq-net/benches/group.rs index 6b838ff303c..7ca89ed39f0 100644 --- a/rs/moq-net/benches/group.rs +++ b/rs/moq-net/benches/group.rs @@ -22,6 +22,10 @@ //! holds (a small pool's debt, or the 32 MiB live tail) and the frames it sheds, so //! shedding the front stays a flat cost per frame. //! +//! `group_paced` keeps a writer one frame past the 16 MiB window ahead of its fastest +//! reader, as a paced ingest runs, swept over readers: each frame read reports progress, +//! and the fastest reader's read wakes the parked writer. +//! //! Run with `cargo bench -p moq-net`. #![expect( @@ -758,6 +762,57 @@ fn bench_shed(c: &mut Criterion) { g.finish(); } +/// Counts wakes, so a bench can check that reads let a parked writer on. +#[derive(Default)] +struct Wakes(AtomicU64); + +impl std::task::Wake for Wakes { + fn wake(self: Arc) { + self.0.fetch_add(1, Ordering::Relaxed); + } +} + +/// A writer held at the pacing window: each round it polls [`group::Producer::poll_ready`], +/// parking whenever it is past the window, every reader takes one frame, and the writer +/// writes the next. A reader reports a prefetched batch at once, so the fastest one's batch +/// read wakes the parked writer. Throughput counts frames read, so the per-read progress +/// report and the park and wake show up together, and the readers trailing the fastest one +/// as a slope across readers. +fn bench_paced(c: &mut Criterion) { + // Payloads are shared clones, so a megabyte costs a refcount and the window is 16 frames. + let payload = Bytes::from(vec![0u8; 1 << 20]); + let mut g = c.benchmark_group("group_paced"); + for readers in [1, 8, 64] { + g.throughput(Throughput::Elements(readers as u64)); + g.bench_function(BenchmarkId::from_parameter(readers), |b| { + let mut ctx = fresh_group(); + let mut consumers: Vec<_> = (0..readers).map(|_| ctx.group.consume()).collect(); + // One frame past the window, so the writer is paced from the first round. + for _ in 0..17 { + ctx.group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + } + let wakes = Arc::new(Wakes::default()); + let writer = kio::Waiter::new(wakes.clone().into()); + let reader = kio::Waiter::noop(); + let mut parked = 0u64; + b.iter(|| { + parked += ctx.group.poll_ready(&writer).is_pending() as u64; + for consumer in &mut consumers { + let Poll::Ready(Ok(Some(frame))) = consumer.poll_read_frame(&reader) else { + panic!("a frame is ready"); + }; + black_box(frame); + } + ctx.group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + }); + // A writer still paced parks again on the next round, so wakes trail parks. + assert!(parked > 0, "the writer never parked"); + assert!(wakes.0.load(Ordering::Relaxed) > 0, "no read woke the writer"); + }); + } + g.finish(); +} + criterion_group!( benches, bench_write, @@ -768,6 +823,7 @@ criterion_group!( bench_relay_frames, bench_inflate, bench_contended, - bench_shed + bench_shed, + bench_paced ); criterion_main!(benches); diff --git a/rs/moq-net/src/coding/reader.rs b/rs/moq-net/src/coding/reader.rs index eb47277624f..0457e4e9f97 100644 --- a/rs/moq-net/src/coding/reader.rs +++ b/rs/moq-net/src/coding/reader.rs @@ -40,6 +40,31 @@ pub struct Reader { /// enough that an ordinary burst is still one wake. const WAKE_BUDGET: usize = 64 * 1024; +/// A stream ingest may stop reading to pace a group's writer, which holds its receive +/// credit meanwhile. +pub(crate) trait Pause { + /// Reserve the credit this stream may hold while paused, returning whether it may pause. + /// A stream outside any session always may. + fn pause(&mut self) -> bool { + true + } + + /// Return what [`Self::pause`] reserved, as the stream is read again. + fn resume(&mut self) {} +} + +impl Pause for crate::transport::boxed::RecvStream { + fn pause(&mut self) -> bool { + crate::transport::boxed::RecvStream::pause(self) + } + + fn resume(&mut self) { + crate::transport::boxed::RecvStream::resume(self) + } +} + +impl Pause for crate::transport::poll::Buffered {} + /// The largest frame [`Reader::poll_read_frame`] copies out of a chunk holding all of it, /// rather than caching the chunk as it arrived. /// @@ -370,7 +395,9 @@ impl Reader { } } - /// Fill a frame's payload from the stream, returning `Pending` once it would block. + /// Fill a frame's payload from the stream, returning `Pending` once it would block or + /// the group's writer is paced by its fastest reader (see + /// [`crate::frame::Producer::poll_ready`]) and the stream may pause (see [`Pause`]). /// /// The payload reaches the frame as it arrives, but keeps its receive credit until the /// segment it lands in seals, so an incomplete frame holds at most one segment of the @@ -392,12 +419,22 @@ impl Reader { &mut self, waiter: &kio::Waiter, frame: &mut crate::frame::Producer, - ) -> Poll> { + ) -> Poll> + where + S: Pause, + { self.want = 0; let mut owed = 0; let result = loop { - let mut lease = match self.poll_lease(waiter, frame.remaining().min(1)) { + // A writer too far ahead of the group's fastest reader leaves the rest in the + // stream, so flow control pushes back upstream. + let lease = match self.poll_pace(waiter, frame.poll_ready(waiter)) { + Poll::Ready(Ok(())) => self.poll_lease(waiter, frame.remaining().min(1)), + Poll::Ready(Err(err)) => break Err(err), + Poll::Pending => Poll::Pending, + }; + let mut lease = match lease { Poll::Pending => { if owed > 0 { frame.notify(); @@ -457,6 +494,46 @@ impl Reader { Poll::Ready(result) } + /// Poll until `group`'s writer may admit the frame `header` opens from this stream: until + /// that leaves no byte its fastest reader has yet to take for the live tail to shed. + /// + /// Pacing before the frame exists matters because admitting it trims the tail to fit + /// its declared size, so a large frame would otherwise shed what that reader still wants. + /// An error means the stream failed while paused. + pub(crate) fn poll_admit( + &mut self, + waiter: &kio::Waiter, + group: &crate::recv::Group, + header: &crate::frame::Header, + ) -> Poll> + where + S: Pause, + { + self.poll_pace(waiter, group.poll_admit(waiter, header)) + } + + /// Wait out `paced`, the group's pace gate, if the session has room for the credit this + /// stream holds while paused (see [`Pause`]); otherwise read on, and the group sheds. + /// + /// A paused stream is not read, so it is watched for a reset instead, which fails the + /// wait rather than leaving it to the group's reader to notice. + fn poll_pace(&mut self, waiter: &kio::Waiter, paced: Poll<()>) -> Poll> + where + S: Pause, + { + match paced { + Poll::Ready(()) => { + self.stream.resume(); + Poll::Ready(Ok(())) + } + Poll::Pending if self.stream.pause() => { + let err = ready!(self.stream.poll_reset(waiter)); + Poll::Ready(Err(self.version.transport_error(err))) + } + Poll::Pending => Poll::Ready(Ok(())), + } + } + /// Poll for exactly `size` bytes. pub fn poll_read_exact(&mut self, waiter: &kio::Waiter, size: usize) -> Poll> { self.want = 0; @@ -691,11 +768,12 @@ mod tests { /// Hands back one queued chunk per poll, then blocks: a stream whose socket has /// delivered a burst and has nothing more for this turn. Finishes once `fin` is set and - /// the queue is empty. + /// the queue is empty, and reports a reset once `reset` is set. #[derive(Default)] struct Queue { chunks: VecDeque<&'static [u8]>, fin: bool, + reset: bool, } impl Chunks for Queue { @@ -719,13 +797,20 @@ mod tests { } fn stop(&mut self, _code: u32) {} + + fn poll_reset(&mut self, _: &kio::Waiter) -> Poll { + match self.reset { + true => Poll::Ready(Default::default()), + false => Poll::Pending, + } + } } fn queue(chunks: &[&'static [u8]], version: V) -> Reader, V> { Reader::new( Buffered::new(Queue { chunks: chunks.iter().copied().collect(), - fin: false, + ..Default::default() }), version, ) @@ -906,6 +991,48 @@ mod tests { )); } + /// A group 17 MB past an idle reader, so its writer is paced. + fn paced_group() -> (crate::group::Producer, crate::group::Consumer) { + let mut group = crate::group::Info { sequence: 0 }.produce(); + let idle = group.consume(); + let payload = Bytes::from(vec![0u8; 1 << 20]); + for _ in 0..17 { + group.write_frame(crate::Timestamp::ZERO, payload.clone()).unwrap(); + } + (group, idle) + } + + /// A paused stream is not read, so it is watched for a reset instead: one ends the wait + /// at once, whether a frame waits to be admitted or to fill, rather than once the + /// group's reader moves. + #[test] + fn a_reset_ends_a_paused_stream() { + let noop = &kio::Waiter::noop(); + + let (group, _idle) = paced_group(); + let group = crate::recv::Group::new(group); + let header = crate::frame::Header::from(crate::frame::Info { + size: 1, + timestamp: Some(crate::Timestamp::ZERO), + }); + let mut reader = queue(&[], crate::lite::Version::Lite05); + assert!(reader.poll_admit(noop, &group, &header).is_pending(), "paused to admit"); + reader.stream.get_mut().reset = true; + assert!(matches!(reader.poll_admit(noop, &group, &header), Poll::Ready(Err(_)))); + + let (group, _idle) = paced_group(); + let mut frame = group + .create_frame(crate::frame::Info { + size: 1, + timestamp: Some(crate::Timestamp::ZERO), + }) + .unwrap(); + let mut reader = queue(&[b"x"], crate::lite::Version::Lite05); + assert!(reader.poll_read_frame(noop, &mut frame).is_pending(), "paused to fill"); + reader.stream.get_mut().reset = true; + assert!(matches!(reader.poll_read_frame(noop, &mut frame), Poll::Ready(Err(_)))); + } + /// A frame that arrives whole in a larger chunk, as in a datagram it shares with other /// frames, is copied out of it when it is at most [`COPY_WHOLE`], so caching it does not /// keep the rest of the receive allocation alive. A larger one is cached as the chunk. @@ -1008,7 +1135,7 @@ mod tests { Buffered::new(Burst { chunks: Queue { chunks: std::iter::repeat_n(CHUNK, chunks).collect(), - fin: false, + ..Default::default() }, payload, waiter, @@ -1033,6 +1160,8 @@ mod tests { struct CreditedLease<'a>(&'a mut Credited); + impl Pause for Credited {} + impl RecvStream for Credited { type Error = crate::lite::test_transport::SinkError; type Lease<'a> = CreditedLease<'a>; diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 5d928d470bb..563ea6fa225 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -5105,6 +5105,9 @@ enum IngestPhase { Size { timestamp: Option }, /// Reading the status of an empty object. Status { timestamp: Option }, + /// Waiting to admit the object, or an empty one's properties, until the group's + /// fastest reader leaves room for it. + Admit { header: frame::Header }, /// Streaming the object payload. Payload { frame: frame::Producer }, /// An explicit end-of-group or end-of-track status arrived. @@ -6022,11 +6025,12 @@ impl FetchPayloadRecv { properties: decoded.properties, }) }); - let header = frame::Header { + let header: frame::Header = frame::Header { info: frame::Info { size, timestamp }, fields, inflated: super::object::inflated(decoded.inflated, producer.compressed())?, }; + wait!(Admit => stream.poll_admit(waiter, producer, header))?; let mut frame: crate::frame::Producer = producer.create_object(header)?; let read: Result<(), Error> = wait!(Read => stream.poll_read_frame(waiter, frame)); if let Err(err) = read { @@ -6182,7 +6186,7 @@ impl GroupIngest { /// `Ready(Ok(_))` once the stream FINs on an object boundary, or an explicit /// end-of-group or end-of-track status arrives. The caller finishes or aborts the /// group; an object cut short mid-payload was already aborted here with the reason. - pub(crate) fn poll( + pub(crate) fn poll( &mut self, reader: &mut Reader, group: &mut crate::recv::Group, @@ -6232,22 +6236,32 @@ impl GroupIngest { self.phase = IngestPhase::Status { timestamp: *timestamp }; continue; } + let timestamp = object_time(self.timescale, *timestamp)?; + let header = self.header(group, frame::Info { size, timestamp })?; + self.phase = IngestPhase::Admit { header }; + } + IngestPhase::Admit { header } => { + ready!(reader.poll_admit(waiter, group, header))?; + let IngestPhase::Admit { header } = std::mem::replace(&mut self.phase, IngestPhase::Delta) else { + unreachable!() + }; // `create_object` is the allocation chokepoint: it rejects an oversized // `size` and allocates up front only within the origin's budget, so no // pre-check is needed. - let timestamp = object_time(self.timescale, *timestamp)?; - let header = self.header(group, frame::Info { size, timestamp })?; + let empty = header.info.size == 0; let frame = group.create_object(header)?; - self.phase = IngestPhase::Payload { frame }; + match empty { + // An empty object has no payload to read, only properties to admit. + true => group.restore(frame.finish()?), + false => self.phase = IngestPhase::Payload { frame }, + } } IngestPhase::Status { timestamp } => { let status = ready!(reader.poll_varint(waiter))?; if status == 0 { let timestamp = object_time(self.timescale, *timestamp)?; let header = self.header(group, frame::Info { size: 0, timestamp })?; - let frame = group.create_object(header)?; - group.restore(frame.finish()?); - self.phase = IngestPhase::Delta; + self.phase = IngestPhase::Admit { header }; continue; } // A status object has no frame to hold its time, so its block keeps it. diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 1ddef7bac5a..bb5a71c72e3 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -1400,6 +1400,8 @@ enum IngestPhase { timestamp: Option, inflated: Option, }, + /// Waiting to admit the frame until the group's fastest reader leaves room for it. + Admit { header: frame::Header }, /// Streaming the frame payload. Payload { frame: frame::Producer }, } @@ -1445,7 +1447,7 @@ impl FrameIngest { /// `Ready(Ok(()))` once the stream FINs on a frame boundary. The caller /// finishes or aborts the group; a frame cut short mid-payload was already /// aborted here with the reason. - pub(crate) fn poll( + pub(crate) fn poll( &mut self, reader: &mut Reader, group: &mut crate::recv::Group, @@ -1585,16 +1587,25 @@ impl FrameIngest { } None => None, }; + self.phase = IngestPhase::Admit { + header: frame::Header { + info: frame::Info { size, timestamp }, + fields, + inflated, + }, + }; + } + IngestPhase::Admit { header } => { + ready!(reader.poll_admit(waiter, group, header))?; + let IngestPhase::Admit { header } = std::mem::replace(&mut self.phase, IngestPhase::Timing) else { + unreachable!() + }; // `create_object` is the allocation chokepoint: it rejects an oversized // `size` and allocates up front only within the origin's budget, so no // pre-check is needed. No wire timestamp (pre-lite-05) means an untimed // frame. A compressed frame stores the slice as it arrives, its inflated // size checked against the same limits before anything inflates. - let frame = group.create_object(frame::Header { - info: frame::Info { size, timestamp }, - fields, - inflated, - })?; + let frame = group.create_object(header)?; self.phase = IngestPhase::Payload { frame }; } IngestPhase::Payload { frame } => { diff --git a/rs/moq-net/src/lite/test_transport.rs b/rs/moq-net/src/lite/test_transport.rs index 231559b8777..5071888d88e 100644 --- a/rs/moq-net/src/lite/test_transport.rs +++ b/rs/moq-net/src/lite/test_transport.rs @@ -616,6 +616,8 @@ pub struct SinkSession { stats: Arc>, /// Set by [`Self::with_probe`]. probe: Option, + /// Set by [`Self::with_receive_window`]. + receive_window: Option, } impl SinkSession { @@ -645,6 +647,12 @@ impl SinkSession { self } + /// Report `window` as the receive windows in force. + pub fn with_receive_window(mut self, window: crate::transport::ReceiveWindow) -> Self { + self.receive_window = Some(window); + self + } + /// Give the transport a sender for the session to hand its probe target. pub fn with_probe(mut self) -> Self { self.probe = Some(crate::transport::Probe::new()); @@ -830,6 +838,10 @@ impl poll::Session for SinkSession { fn probe(&self) -> Option { self.probe.clone() } + + fn receive_window(&self) -> Option { + self.receive_window + } } /// Connection statistics a test can dictate. Every metric defaults to unknown, diff --git a/rs/moq-net/src/model/frame.rs b/rs/moq-net/src/model/frame.rs index 497243e03e9..b3c8ca71a01 100644 --- a/rs/moq-net/src/model/frame.rs +++ b/rs/moq-net/src/model/frame.rs @@ -1199,6 +1199,14 @@ impl Producer { } } + /// Poll until the group's writer may take more of this frame from upstream; see + /// [`group::Producer::poll_paced`]. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn poll_ready(&self, waiter: &kio::Waiter) -> Poll<()> { + self.group() + .poll_paced(waiter, self.buf.written(Ordering::Acquire) as u64) + } + /// Publish what has been written so far, waking consumers parked on the group. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub(crate) fn notify(&self) { diff --git a/rs/moq-net/src/model/group.rs b/rs/moq-net/src/model/group.rs index 9660405062d..719acf55121 100644 --- a/rs/moq-net/src/model/group.rs +++ b/rs/moq-net/src/model/group.rs @@ -19,6 +19,7 @@ //! //! The stream is closed with [Error] when all writers or readers are dropped. use super::flate; +use super::pace::Pace; mod content; use crate::cache; use crate::frame::{self, Cached, FrameBuf, Payload}; @@ -57,6 +58,9 @@ pub(super) const CACHE_OVERHEAD: u64 = (kio::Producer::::HEAP // straight after. + 2 * size_of::() + size_of::() + // The `Pace` its writer and readers share, behind its own `Arc`. + + 2 * size_of::() + + size_of::() + kio::log::Writer::::HEAP) as u64; /// A group contains a sequence number because they can arrive out of order. @@ -143,6 +147,8 @@ pub(super) struct Entry { fields: Option>, // A compressed group's inflated size of the stored slice; zero on a plain group. inflated: u64, + // Where the frame ends on its copy's [`Pace`] scale, set as it enters the log. + end: u64, } impl Clone for Entry { @@ -152,6 +158,7 @@ impl Clone for Entry { frame: self.frame.clone(), fields: self.fields.clone(), inflated: self.inflated, + end: self.end, } } } @@ -164,6 +171,7 @@ impl Entry { frame, fields: None, inflated, + end: 0, } } @@ -258,6 +266,9 @@ pub(super) struct GroupState { // Whether an adopted head (see [`Handle::adopt_head`]) begins at the group's first // object, for a wire header that arrives after it. adopted: Option, + + // How far readers got, which each frame entering the log is measured against. + pace: Arc, } /// The frames a copy of a group holds, from where it starts, which a successor copy of @@ -652,6 +663,7 @@ impl Inflate { }, fields: slice.fields.clone(), inflated: 0, + end: slice.end, }; let cost = Plain::cost(&entry); self.plain.bill.add(cost); @@ -871,13 +883,15 @@ impl GroupState { } let base = self.log.base() as usize; if base < end { - let (mut charged, mut payload, mut fields) = (0, 0, 0); + let (mut charged, mut payload, mut fields, mut shed) = (0, 0, 0, 0); for entry in (base..end).map(|index| self.log.get(index as u64).expect("committed below the end")) { + shed = entry.end; charged += self.entry_bytes(entry); payload += entry.frame.payload.len() as u64; fields += entry.fields.as_ref().map_or(0, |fields| fields.heap()); } self.forget(payload, fields); + self.pace.shed(shed); let heap = kio::log::Writer::::HEAP; let before = self.log.heap().max(heap); self.log.evict(end as u64); @@ -953,8 +967,15 @@ impl GroupState { } /// Append a completed frame, returning the slot growth it charged. - fn push(&mut self, entry: Entry) -> u64 { + /// + /// The frame's place on the pace scale costs its payload, metadata, and slot, as the + /// live tail does, so a paced writer of tiny frames is bounded too. + fn push(&mut self, mut entry: Entry) -> u64 { let slots = self.reserve_frame(); + let metadata = entry.fields.as_ref().map_or(0, |fields| fields.heap()); + entry.end = self + .pace + .commit(entry.frame.payload.len() as u64 + metadata + size_of::() as u64); self.log.push(entry); slots } @@ -1062,6 +1083,7 @@ impl GroupState { /// /// A compressed group hands a plain reader each frame whole once it completes, as the /// slices to inflate, and a `raw` one the stored slice, streaming one still arriving. + /// A frame handed out counts as taken, pacing the writer (see [`Pace`]). fn poll_frame_source(&self, index: usize, raw: bool) -> Poll>> { self.check_read(index, raw)?; if index < self.held() @@ -1075,6 +1097,7 @@ impl GroupState { // walk spare a group a consumer is actively draining. self.charge.refresh(); if let Some(flate) = self.flate.as_ref().filter(|_| !raw) { + self.pace.took(entry.end); return Poll::Ready(Ok(Some(Source::Inflate { job: self.inflate(flate, index, index)?, local: index, @@ -1083,6 +1106,7 @@ impl GroupState { }))); } let (header, source) = entry.read(self.flate.is_some()); + self.pace.took(entry.end); return Poll::Ready(Ok(Some(Source::Ready(header, source)))); } if index == self.committed() @@ -1091,6 +1115,8 @@ impl GroupState { && (raw || self.flate.is_none()) { self.charge.refresh(); + // A reader streaming the open frame is at the writer's edge. + self.pace.took_committed(); let info = frame::Info { size: p.buf.size() as u64, timestamp: p.timestamp, @@ -1119,7 +1145,14 @@ impl GroupState { (Some(total), Some(err)) if index < total => Poll::Ready(Err(err.clone())), (Some(_), _) => Poll::Ready(Ok(())), (None, Some(err)) => Poll::Ready(Err(err.clone())), - (None, None) => Poll::Pending, + (None, None) => { + // Waiting on frames the writer has yet to commit, wherever the reader skipped + // to, so it holds the writer back from none of them. + if index >= self.committed() { + self.pace.took_committed(); + } + Poll::Pending + } } } @@ -1446,6 +1479,8 @@ struct Alive { // A reader of the frame log, cloned into each cursor so it reads frames without the // group lock. log: kio::log::Reader, + // The `GroupState::pace` cursors report their reads to. + pace: Arc, // When the group was created here, on the ambient clock; see [`Consumer::arrived`]. arrived: crate::time::Instant, } @@ -1558,7 +1593,7 @@ impl Producer { let state = kio::Producer::::default(); let charge = cache.charge(); let access = charge.access(); - let log = { + let (log, pace) = { let mut state = state.write().ok().expect("a new group is open"); if track.compress { state.flate = Some(Box::new(Flate::new( @@ -1567,7 +1602,7 @@ impl Producer { ))); } state.charge = charge; - state.log.reader() + (state.log.reader(), state.pace.clone()) }; let alive = Arc::new(Alive { aborted: AtomicBool::new(false), @@ -1577,6 +1612,7 @@ impl Producer { started: AtomicBool::new(false), access, log, + pace, arrived: super::clock::now(), }); Self { @@ -2090,6 +2126,7 @@ impl Producer { frame, fields, inflated, + end: 0, }; if !owned { state.canonicalize(state.next_index - 1, &mut entry)?; @@ -2181,6 +2218,52 @@ impl Producer { pub async fn closed(&self) -> Result<()> { kio::wait(|waiter| self.handle.poll_closed(waiter)).await } + /// Poll until a reader holds the group and the writer is at most 16 MB past the end of + /// the last frame its fastest reader took, so a local writer runs no further ahead than + /// the live tail keeps behind that reader. + /// + /// Ready once the group is closed, so the next write surfaces why. A reader that took + /// every committed frame is waiting on the writer, so it never holds the writer back, + /// and one whose next frame was shed has fallen off the group and is not waited on. + /// Writing without waiting never fails: the group sheds its front instead, as it also + /// does for a frame larger than the room left, which trims the tail to fit. + pub fn poll_ready(&self, waiter: &kio::Waiter) -> Poll<()> { + ready!(self.poll_paced(waiter, 0)); + // A closed group reports `Closed`, which is ready too. + self.handle.state.poll_wanted_changed(waiter, false).map(|_| ()) + } + + /// Wait until a reader holds the group and the writer has room; see [`Self::poll_ready`]. + pub async fn ready(&self) { + kio::wait(|waiter| self.poll_ready(waiter)).await + } + + /// Poll until the writer may take more from upstream, `partial` bytes into a frame + /// it has not committed: until it is at most [`super::pace::AHEAD`] past the end of the last + /// frame its fastest reader took. + /// + /// Ready at once while no reader holds the group, so a group nobody watches keeps + /// writing and sheds its front, and once it is closed, so the next write surfaces why. + /// A reader that took every committed frame is waiting on the writer, so a frame + /// larger than the window never stalls it, and one whose next frame was shed has + /// fallen off the group and is not waited on. + /// + /// A session's ingest waits on this; a local writer waits on [`Self::poll_ready`]. + pub(crate) fn poll_paced(&self, waiter: &kio::Waiter, partial: u64) -> Poll<()> { + let pace = &self.handle.alive.pace; + if !pace.ahead(partial) { + return Poll::Ready(()); + } + // Either wakes the writer: the last reader leaving, or the group closing. + if self.handle.poll_unwanted(waiter).is_ready() || self.handle.state.poll_closed(waiter).is_ready() { + return Poll::Ready(()); + } + match pace.paced(pace.park(waiter), partial) { + true => Poll::Pending, + false => Poll::Ready(()), + } + } + /// The non-writing observer used by the cache and built-in sessions. // Sessions need an abortable observer while public callers use Demand. pub(crate) fn handle(&self) -> Handle { @@ -2395,6 +2478,8 @@ impl Handle { if state.canonicalize(index, &mut entry).is_err() { return; } + // Below the log, so off this copy's pace scale: taking it reports nothing. + entry.end = 0; adopted.push(entry); } // Below the log's first frame, so these never enter it: readers find them here. @@ -2790,7 +2875,7 @@ impl Handle { raw: false, compressed: self.track.compress, cache: self.cache.clone(), - access: self.alive.access.clone(), + alive: self.alive.clone(), refreshed: self.cache.pool().now(), }, // Untagged: a tagged track attaches the egress meter via `with_meter` @@ -3042,8 +3127,9 @@ impl Prefetch { } } - /// Buffer `entry` at the end of the batch, with its inflated size if `inflated`. - fn push(&mut self, entry: &Entry, inflated: bool) { + /// Buffer `entry` at the end of the batch, with its inflated size if `inflated`, + /// returning where it ends on the group's pace scale. + fn push(&mut self, entry: &Entry, inflated: bool) -> u64 { let at = self.len; let frames = self .frames @@ -3074,6 +3160,7 @@ impl Prefetch { self.inflated.get_or_insert_default()[at] = entry.inflated; } self.len += 1; + entry.end } /// Refill with up to `budget` of the completed frames from `index` on, under the @@ -3110,25 +3197,39 @@ impl Prefetch { Some(flate) if !raw && count > 0 => Some(state.inflate(flate, index, index + count - 1)?), _ => None, }; + let mut end = 0; for entry in state.entries(index, index + count) { - self.push(entry, compressed && raw); + end = self.push(entry, compressed && raw); } + // The batch is the reader's whatever the group sheds, so it counts as taken. + state.pace.took(end); Ok(job) } /// Refill with up to `budget` of the completed frames from `index` on, reading the /// log without the group's guard, and return how many. A plain read of a `compressed` /// group inflates under the guard instead, through [`Self::fill`]. - fn fill_log(&mut self, log: &mut kio::log::Reader, index: usize, budget: usize, compressed: bool) -> usize { + /// + /// The batch is the reader's whatever the group sheds, so it reports it taken to `pace`. + fn fill_log( + &mut self, + log: &mut kio::log::Reader, + pace: &Pace, + index: usize, + budget: usize, + compressed: bool, + ) -> usize { self.reset(compressed); let mut read = log.read(); + let mut end = 0; for index in (index as u64..).take(budget.min(Self::CAP)) { // Evicted or not yet committed: the guard resolves which. let Ok(Some(entry)) = read.get(index) else { break; }; - self.push(entry, compressed); + end = self.push(entry, compressed); } + pace.took(end); self.len } @@ -3274,9 +3375,10 @@ struct Cursor { // Whether the group is compressed, so a plain read inflates under the state's lock. compressed: bool, - // Record prefetched reads without entering the group's state on every frame. + // Record prefetched reads without entering the group's state on every frame, through + // the group's access stamp, and report the frames taken to its pace. cache: Arc, - access: Arc, + alive: Arc, refreshed: u64, } @@ -3293,7 +3395,7 @@ impl Clone for Cursor { raw: self.raw, compressed: self.compressed, cache: self.cache.clone(), - access: self.access.clone(), + alive: self.alive.clone(), refreshed: self.refreshed, } } @@ -3893,7 +3995,7 @@ impl Cursor { /// Record prefetched reads, updating the eviction rank once per sampled tick. fn refresh_if_stale(&mut self) { - self.access.touch(); + self.alive.access.touch(); let tick = self.cache.pool().now(); if tick != self.refreshed { self.state.read().charge.refresh(); @@ -3977,13 +4079,10 @@ impl Cursor { // A committed frame comes from the log without the lock, unless a plain read of a // compressed group has to inflate it under the lock. let committed = match !self.compressed || raw { - true => self - .log - .read() - .get(index as u64) - .ok() - .flatten() - .map(|entry| entry.read(self.compressed)), + true => self.log.read().get(index as u64).ok().flatten().map(|entry| { + self.alive.pace.took(entry.end); + entry.read(self.compressed) + }), false => None, }; let (info, source) = match committed { @@ -4054,7 +4153,12 @@ impl Cursor { // The batch is drained: refill it from the log without the lock, unless a plain // read of a compressed group has to inflate under it. - if (!self.compressed || self.raw) && self.prefetch.fill_log(&mut self.log, index, budget, self.compressed) > 0 { + if (!self.compressed || self.raw) + && self + .prefetch + .fill_log(&mut self.log, &self.alive.pace, index, budget, self.compressed) + > 0 + { self.refresh_if_stale(); let (frames, bytes) = self.prefetch.buffered(); stats.frames(frames); @@ -4854,6 +4958,7 @@ mod test { frame: Frame::new(None, Bytes::from_static(b"b")).into(), fields: fields(1), inflated: 0, + end: 0, }, ], compressed: false, diff --git a/rs/moq-net/src/model/mod.rs b/rs/moq-net/src/model/mod.rs index 00a715a1efb..9123fd25f95 100644 --- a/rs/moq-net/src/model/mod.rs +++ b/rs/moq-net/src/model/mod.rs @@ -23,6 +23,8 @@ pub(crate) mod datagram; mod delivered; mod expiry; mod skipped; +// How far a group's writer reads ahead of its fastest reader. +mod pace; // The DEFLATE codec behind compressed tracks. mod flate; pub(crate) mod object; diff --git a/rs/moq-net/src/model/pace.rs b/rs/moq-net/src/model/pace.rs new file mode 100644 index 00000000000..bb1ca8ff938 --- /dev/null +++ b/rs/moq-net/src/model/pace.rs @@ -0,0 +1,387 @@ +//! Fastest-reader pacing: how far a group's writer reads ahead of its furthest reader. +//! +//! Readers report the frames they take; a paced writer (a session's ingest) stops reading +//! its upstream stream while it is more than [`AHEAD`] past the end of the last frame the +//! fastest reader took, so stream flow control pushes back to the publisher. Only the +//! fastest reader counts: a slower one is never waited on, and falls off the group's +//! front as the live tail sheds. +//! +//! Progress is in bytes on one scale per group copy: each frame entering the copy's log +//! ends where the frames before it, payload and slot, add up to. A frame larger than the +//! window never stalls its writer, since a reader that took every committed frame is +//! waiting on the writer. + +use std::sync::{ + Mutex, PoisonError, + atomic::{AtomicBool, AtomicU64, Ordering}, +}; +use std::task::Waker; + +use super::group::MAX_CACHE_BYTES; + +/// How far a paced writer runs ahead of its group's fastest reader: half the live tail, +/// so the other half stays behind the fastest reader for slower ones. +pub(super) const AHEAD: u64 = MAX_CACHE_BYTES / 2; + +/// One group copy's reader progress, shared by its writer and every cursor. +#[derive(Default)] +pub(super) struct Pace { + // Where the last frame the fastest reader took ends. + read: AtomicU64, + // Where the last committed frame ends. + written: AtomicU64, + // Where the last frame shed from the copy's front ends: a reader short of it fell off. + front: AtomicU64, + // Set while the writer waits on `read`, so a read wakes it without locking otherwise. + parked: AtomicBool, + // The writer's waker: a group has one writer, so one slot, far smaller than a list. + writer: Mutex>, +} + +impl Pace { + /// A reader took the frame ending at `end`, and every frame before it. + pub fn took(&self, end: u64) { + // The common case, a reader trailing the fastest one, writes nothing. + if self.read.load(Ordering::Relaxed) >= end { + return; + } + // SeqCst pairs with `park`: either this sees the writer parked, or it sees the read. + self.read.fetch_max(end, Ordering::SeqCst); + self.wake(); + } + + /// Wake the writer, if parked, since what it waits on changed. + fn wake(&self) { + // A load first, so a writer not parked costs a read rather than a locked swap; still + // SeqCst, to stay in the order `park` pairs with. + if self.parked.load(Ordering::SeqCst) && self.parked.swap(false, Ordering::SeqCst) { + let writer = self.writer.lock().unwrap_or_else(PoisonError::into_inner).take(); + if let Some(writer) = writer { + writer.wake(); + } + } + } + + /// A reader took every frame committed so far, such as by taking the open one. + pub fn took_committed(&self) { + self.took(self.written()); + } + + /// Count a committed frame costing `cost`, returning where it ends. + /// + /// Called under the group's lock, so commits never race. + pub fn commit(&self, cost: u64) -> u64 { + let end = self.written() + cost; + self.written.store(end, Ordering::Release); + end + } + + /// The copy shed every frame up to the one ending at `end`, which may leave the + /// fastest reader fallen off, so the writer no longer waits on it. + pub fn shed(&self, end: u64) { + // SeqCst pairs with `park`, as a read does. + self.front.fetch_max(end, Ordering::SeqCst); + self.wake(); + } + + fn written(&self) -> u64 { + self.written.load(Ordering::Acquire) + } + + /// Whether a writer `partial` bytes into an uncommitted frame is more than [`AHEAD`] + /// past the fastest reader, given it read up to `read`. + /// + /// Never while that reader took every committed frame, since it is waiting on the + /// writer, nor once the frames it has yet to take were shed, since it fell off. + pub fn paced(&self, read: u64, partial: u64) -> bool { + let written = self.written(); + read < written && read >= self.front.load(Ordering::SeqCst) && written.saturating_add(partial) - read > AHEAD + } + + /// Whether the writer is paced, as [`Self::paced`] judges it now. + pub fn ahead(&self, partial: u64) -> bool { + self.paced(self.read.load(Ordering::SeqCst), partial) + } + + /// Register `waiter` for the next read or shed, returning how far readers got, read + /// after registering so a read in between is never missed; [`Self::paced`] reads the + /// front after it too. + pub fn park(&self, waiter: &kio::Waiter) -> u64 { + let mut writer = self.writer.lock().unwrap_or_else(PoisonError::into_inner); + match &mut *writer { + Some(waker) => waker.clone_from(waiter.waker()), + None => *writer = Some(waiter.waker().clone()), + } + drop(writer); + self.parked.store(true, Ordering::SeqCst); + self.read.load(Ordering::SeqCst) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + use std::task::Poll; + + use bytes::Bytes; + + use super::*; + use crate::{Error, frame, group}; + + const MB: usize = 1 << 20; + + struct Flag(AtomicBool); + + impl std::task::Wake for Flag { + fn wake(self: Arc) { + self.0.store(true, Ordering::SeqCst); + } + } + + impl Flag { + fn woken(&self) -> bool { + self.0.swap(false, Ordering::SeqCst) + } + } + + fn flag() -> (Arc, kio::Waiter) { + let flag = Arc::new(Flag(AtomicBool::new(false))); + let waiter = kio::Waiter::new(std::task::Waker::from(flag.clone())); + (flag, waiter) + } + + /// An untimed group with `frames` frames of a megabyte each. + fn written(frames: usize) -> group::Producer { + let mut group = group::Info { sequence: 0 }.produce_untimed(); + let payload = Bytes::from(vec![0u8; MB]); + for _ in 0..frames { + group.write_frame(None, payload.clone()).unwrap(); + } + group + } + + fn read(reader: &mut group::Consumer) { + let Poll::Ready(Ok(Some(_))) = reader.poll_read_frame(&kio::Waiter::noop()) else { + panic!("a committed frame is ready"); + }; + } + + /// A writer more than the window ahead of its fastest reader waits, and that reader + /// taking frames lets it on; a slower reader is never waited on. + #[test] + fn the_fastest_reader_paces_the_writer() { + let (flag, waiter) = flag(); + let mut group = written(15); + let mut fast = group.consume(); + let _slow = group.consume(); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Ready(()), "within the window"); + + group.write_frame(None, Bytes::from(vec![0u8; MB])).unwrap(); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Pending, "past the window"); + assert!(!flag.woken()); + + // A batch read takes several frames at once, which the writer counts as read. + read(&mut fast); + assert!(flag.woken(), "the fastest reader's read wakes the writer"); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Ready(())); + } + + /// A one-byte frame whose properties hold `metadata` bytes. + #[cfg(feature = "async")] + fn fielded(id: u64, metadata: usize) -> frame::Header { + use crate::model::object::{Block, Fields, Properties}; + let block = Block { + bytes: Bytes::from(vec![0u8; metadata]), + immutable: false, + }; + frame::Header { + info: frame::Info { + size: 1, + timestamp: None, + }, + fields: Some(Arc::new(Fields { + id, + properties: Properties { + block: Some(block), + ..Default::default() + }, + })), + inflated: None, + } + } + + /// Metadata counts toward the window as it does toward the live tail, both committed + /// and to admit a frame, so frames whose properties outweigh their payloads are paced. + #[test] + #[cfg(feature = "async")] + fn metadata_counts_toward_the_window() { + let (_flag, waiter) = flag(); + let mut group = group::Info { sequence: 0 }.produce_untimed(); + let _reader = group.consume(); + for id in 0..15 { + let mut frame = group.create_object(fielded(id, MB)).unwrap(); + frame.write(Bytes::from_static(b"f")).unwrap(); + group = frame.finish().unwrap(); + } + let group = crate::recv::Group::new(group); + assert_eq!(group.poll_admit(&waiter, &fielded(15, 0)), Poll::Ready(())); + assert_eq!( + group.poll_admit(&waiter, &fielded(15, 2 * MB)), + Poll::Pending, + "its metadata would pass the window" + ); + } + + /// Bytes measure the window, not frames, so an open frame counts as it fills. + #[test] + fn an_open_frame_counts_toward_the_window() { + let (_flag, waiter) = flag(); + let group = written(15); + let _reader = group.consume(); + let mut frame = group + .create_frame(frame::Info { + size: 2 * MB as u64, + timestamp: None, + }) + .unwrap(); + assert_eq!(frame.poll_ready(&waiter), Poll::Ready(())); + frame.write(Bytes::from(vec![0u8; MB])).unwrap(); + assert_eq!(frame.poll_ready(&waiter), Poll::Pending); + } + + /// A frame is admitted only once its declared size fits the window, since admitting + /// it trims the tail to fit, which would otherwise shed what the reader still wants. + #[test] + fn a_large_frame_waits_to_be_admitted() { + let (flag, waiter) = flag(); + let group = written(15); + let mut reader = group.consume(); + assert_eq!(group.poll_paced(&waiter, 20 * MB as u64), Poll::Pending); + while reader.poll_read_frame(&kio::Waiter::noop()).is_ready() {} + assert!(flag.woken()); + assert_eq!( + group.poll_paced(&waiter, 20 * MB as u64), + Poll::Ready(()), + "the reader took everything, so the frame sheds nothing it wants" + ); + } + + /// A group nobody reads keeps writing, and a parked writer moves on once its last + /// reader leaves. + #[test] + fn a_group_without_readers_never_waits() { + let (flag, waiter) = flag(); + let group = written(16); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Ready(())); + + let reader = group.consume(); + assert_eq!( + group.poll_paced(&waiter, 0), + Poll::Pending, + "a reader that took nothing" + ); + drop(reader); + assert!(flag.woken(), "the last reader leaving wakes the writer"); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Ready(())); + } + + /// A local writer waits for a reader before it writes, then for room in the window, and + /// a reader arriving or reading wakes it. + #[test] + fn a_local_writer_waits_for_a_reader_and_room() { + let (flag, waiter) = flag(); + let mut group = group::Info { sequence: 0 }.produce_untimed(); + assert_eq!(group.poll_ready(&waiter), Poll::Pending, "nobody reads yet"); + let mut reader = group.consume(); + assert!(flag.woken(), "a reader arriving wakes the writer"); + assert_eq!(group.poll_ready(&waiter), Poll::Ready(())); + + for _ in 0..17 { + group.write_frame(None, Bytes::from(vec![0u8; MB])).unwrap(); + } + assert_eq!(group.poll_ready(&waiter), Poll::Pending, "past the window"); + read(&mut reader); + assert!(flag.woken(), "the reader's read wakes the writer"); + assert_eq!(group.poll_ready(&waiter), Poll::Ready(())); + + drop(reader); + assert_eq!(group.poll_ready(&waiter), Poll::Pending, "the reader left"); + group.handle().abort(Error::Cancel).unwrap(); + assert_eq!(group.poll_ready(&waiter), Poll::Ready(()), "a closed group never waits"); + } + + /// A reader that took every committed frame is waiting on the writer, so a frame + /// larger than the window never stalls against it. + #[test] + fn a_reader_at_the_edge_never_stalls_a_large_frame() { + let (_flag, waiter) = flag(); + let group = written(1); + let mut reader = group.consume(); + read(&mut reader); + assert!(reader.poll_read_frame(&kio::Waiter::noop()).is_pending()); + + let mut frame = group + .create_frame(frame::Info { + size: 20 * MB as u64, + timestamp: None, + }) + .unwrap(); + frame.write(Bytes::from(vec![0u8; 18 * MB])).unwrap(); + assert_eq!(frame.poll_ready(&waiter), Poll::Ready(())); + } + + /// A reader whose next frame the tail shed fell off the group, so the writer stops + /// waiting on it. + #[test] + fn a_reader_that_fell_off_is_not_waited_on() { + let (_flag, waiter) = flag(); + let mut group = written(1); + let _reader = group.consume(); + // A local writer never waits, so it sheds the front past the 32 MB tail. + for _ in 0..40 { + group.write_frame(None, Bytes::from(vec![0u8; MB])).unwrap(); + } + assert_eq!(group.poll_paced(&waiter, 0), Poll::Ready(())); + } + + /// A reader that skipped past what the writer has written is waiting on it, so it + /// never holds the writer back from the frame it wants. + #[test] + fn a_reader_ahead_of_the_writer_never_stalls_it() { + let (_flag, waiter) = flag(); + let mut group = group::Info { sequence: 0 }.produce_untimed(); + let mut reader = group.consume().with_range(20..); + assert!(reader.poll_read_frame(&kio::Waiter::noop()).is_pending()); + for _ in 0..17 { + group.write_frame(None, Bytes::from(vec![0u8; MB])).unwrap(); + } + // Each commit wakes the parked reader, which polls again and reports the edge. + assert!(reader.poll_read_frame(&kio::Waiter::noop()).is_pending()); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Ready(())); + } + + /// Shedding past the fastest reader under cache pressure wakes a parked writer: that + /// reader fell off. + #[test] + fn shedding_past_the_reader_wakes_the_writer() { + let (flag, waiter) = flag(); + let group = written(17); + let _reader = group.consume(); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Pending); + group.handle().shed(u64::MAX, u64::MAX); + assert!(flag.woken()); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Ready(())); + } + + /// An abort wakes a parked writer, so its next write surfaces why. + #[test] + fn an_abort_wakes_a_parked_writer() { + let (flag, waiter) = flag(); + let group = written(17); + let _reader = group.consume(); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Pending); + group.handle().abort(Error::Cancel).unwrap(); + assert!(flag.woken()); + assert_eq!(group.poll_paced(&waiter, 0), Poll::Ready(())); + } +} diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 086bb1b603b..778d23c1ab7 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -17655,7 +17655,7 @@ mod test { #[kio::test] #[cfg(feature = "async")] async fn fetch_refresh_survives_eviction() { - let (mut producer, _pool) = pooled_producer(10_000); + let (mut producer, _pool) = pooled_producer(11_000); let consumer = producer.consume(); finished_group(&mut producer, 3_000); // seq 0 @@ -18445,7 +18445,7 @@ mod test { #[cfg(feature = "async")] async fn same_tick_fetch_protects() { // No time advances at all: every timestamp lands in the same tick. - let (mut producer, _pool) = pooled_producer(10_000); + let (mut producer, _pool) = pooled_producer(11_000); let consumer = producer.consume(); finished_group(&mut producer, 3_000); // seq 0 diff --git a/rs/moq-net/src/recv.rs b/rs/moq-net/src/recv.rs index d005fd209c9..8e87b288d58 100644 --- a/rs/moq-net/src/recv.rs +++ b/rs/moq-net/src/recv.rs @@ -30,6 +30,17 @@ impl Group { .map(frame::Producer::cancel_on_drop) } + /// Poll until the writer may admit the frame `header` opens, its payload and metadata, + /// without shedding a byte its fastest reader has yet to take; see + /// [`group::Producer::poll_paced`]. + pub fn poll_admit(&self, waiter: &kio::Waiter, header: &frame::Header) -> std::task::Poll<()> { + let metadata = header.fields.as_ref().map_or(0, |fields| fields.heap()); + self.writer + .as_ref() + .expect("group writer is available between frames") + .poll_paced(waiter, header.info.size.saturating_add(metadata)) + } + /// Restore the writer returned by a completed frame. pub fn restore(&mut self, writer: group::Producer) { debug_assert!(self.writer.is_none()); diff --git a/rs/moq-net/src/transport.rs b/rs/moq-net/src/transport.rs index 2420752f0c9..1db50e6d18d 100644 --- a/rs/moq-net/src/transport.rs +++ b/rs/moq-net/src/transport.rs @@ -138,6 +138,30 @@ pub trait Metrics { struct Untracked; impl Metrics for Untracked {} +/// The receive credit a transport grants its peer, which bounds what a stream the session +/// stops reading still holds. +/// +/// A session's ingest stops reading a group's stream while the group's fastest reader is far +/// behind, so flow control pushes back to the publisher. Such a paused stream keeps up to +/// `stream` bytes counted against the shared `connection` window, so the session pauses only +/// as many streams as fit in half of it; the rest keep reading. See +/// [`poll::Session::receive_window`]. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub struct ReceiveWindow { + /// The connection-wide receive window, in bytes, or [`u64::MAX`] when streams share none. + pub connection: u64, + /// The most receive credit one stream may hold, in bytes. + pub stream: u64, +} + +impl ReceiveWindow { + /// A connection-wide window of `connection` bytes, of which one stream holds at most `stream`. + pub fn new(connection: u64, stream: u64) -> Self { + Self { connection, stream } + } +} + /// The smallest connection receive window a transport that leases may offer. /// /// A session keeps each unit it is still assembling leased, and a unit larger than the @@ -327,6 +351,16 @@ pub mod poll { None } + /// Return the receive windows now in force, or `None` when the transport cannot say. + /// + /// Read whenever the session considers pausing a stream, so a window changed on a live + /// connection is followed. Without one, the session cannot bound the credit a paused + /// stream holds, so it never pauses one: ingest reads every stream as it arrives and + /// sheds what its readers fall behind on. + fn receive_window(&self) -> Option { + None + } + /// Accept the next unidirectional stream opened by the peer. fn accept_uni(&self) -> AcceptUni<'_, Self> { AcceptUni::new(self) @@ -535,6 +569,10 @@ pub mod poll { fn probe(&self) -> Option { (**self).probe() } + + fn receive_window(&self) -> Option { + (**self).receive_window() + } } /// A transport that the async client and server can box: @@ -814,6 +852,18 @@ pub mod poll { /// Call this method manually if you want to specify a code yourself. fn stop(&mut self, code: u32); + /// Poll until the stream fails without reading it: the peer resets it, or the + /// connection closes. Pending for good once the peer finished it. + /// + /// A session that stops reading a stream to pace it (see + /// [`Session::receive_window`]) watches this instead, so a reset still ends the group + /// it carries. It takes no bytes and grants no credit. The default never completes, + /// which suits a transport reporting no receive window, since none of its streams pause. + fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll { + let _ = waiter; + Poll::Pending + } + /// Poll for the next chunk, up to `max` bytes, or `None` once the peer finished the /// stream. Its credit returns as it is read. /// @@ -891,6 +941,12 @@ pub mod poll { /// An implementation MUST also do this on Drop: [`Buffered`] forwards an explicit stop /// but leaves the drop to the wrapped stream. fn stop(&mut self, code: u32); + + /// Poll until the stream fails without reading it. See [`RecvStream::poll_reset`]. + fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll { + let _ = waiter; + Poll::Pending + } } /// Leases over a transport that cannot keep bytes unconsumed, buffering what it reads. @@ -961,6 +1017,14 @@ pub mod poll { self.len = 0; self.inner.stop(code); } + + fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll { + // Every byte arrived, so nothing is left to reset. + match self.fin { + true => Poll::Pending, + false => self.inner.poll_reset(waiter), + } + } } /// A [`Lease`] of a [`Buffered`] stream. diff --git a/rs/moq-net/src/transport/boxed.rs b/rs/moq-net/src/transport/boxed.rs index 459f7abe283..f541e489990 100644 --- a/rs/moq-net/src/transport/boxed.rs +++ b/rs/moq-net/src/transport/boxed.rs @@ -6,17 +6,19 @@ //! awaits them. Each transport only instantiates the thin forwarding impls below. use std::any::Any; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Weak}; use std::task::Poll; use bytes::{Buf, Bytes}; use super::poll::{self, SendOptions}; -use super::{Address, CloseKind, Probe, Wire}; +use super::{Address, CloseKind, Probe, ReceiveWindow, Wire}; /// A boxed [`poll::Boxable`] session, shared by every task that uses it. #[derive(Clone)] pub(crate) struct Session { - inner: std::sync::Arc, + inner: Arc>, /// Shared by every clone, so whichever task closes the transport first says why. closing: std::sync::Arc, /// Shared by every clone, so each task counts the subscriptions and fetches it carries. @@ -26,12 +28,20 @@ pub(crate) struct Session { impl Session { pub(crate) fn new(session: S) -> Self { Self { - inner: std::sync::Arc::new(session), + inner: Arc::new(Shared { + paused: AtomicU64::new(0), + session, + }), closing: Default::default(), work: Default::default(), } } + /// The handle each stream this session yields reserves its paused credit through. + fn budget(&self) -> Budget { + Arc::downgrade(&self.inner) + } + /// How this side closed the transport. pub(crate) fn closing(&self) -> &std::sync::Arc { &self.closing @@ -45,10 +55,38 @@ impl Session { /// Close with `code` over `cause`, this side's reason for ending the session. pub(crate) fn abort(&self, cause: &crate::Error, code: u32, reason: &str) { self.closing.sent(code, Some(cause)); - self.inner.close(code, reason); + self.inner.session.close(code, reason); } } +/// A boxed transport and the receive credit its paused streams hold. +struct Shared { + /// The stream credit reserved by streams this session stopped reading to pace a group's + /// writer, at most half the connection window; see [`ReceiveWindow`]. + paused: AtomicU64, + session: S, +} + +impl Shared { + /// Reserve the credit of one more paused stream, unless it would push the paused total + /// over half the connection window, returning how much it reserved. + fn reserve(&self) -> Option { + let window = self.session.receive_window()?; + let charge = window.stream.min(window.connection); + let cap = window.connection / 2; + self.paused + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |paused| { + paused.checked_add(charge).filter(|total| *total <= cap) + }) + .ok()?; + Some(charge) + } +} + +/// The session a stream reserves paused credit from, which a stream outliving it never keeps +/// open: once it is gone, nothing is paused. +type Budget = Weak>; + /// A boxed transport error, keeping both code registries and the message. #[derive(Debug)] pub(crate) struct Error(Box); @@ -115,9 +153,56 @@ impl SendStream { pub(crate) struct RecvStream(Box); impl RecvStream { + /// Box a stream outside any session, which therefore never pauses. #[cfg(test)] pub(crate) fn new(stream: impl poll::RecvStream + Send + Sync) -> Self { - Self(Box::new(stream)) + Self::budgeted(stream, Weak::>::new()) + } + + fn budgeted(stream: impl poll::RecvStream + Send + Sync, session: Budget) -> Self { + Self(Box::new(Budgeted { + stream, + session, + held: 0, + })) + } + + /// Reserve the receive credit this stream may hold while ingest stops reading it, returning + /// whether it may stop: not once the session's paused streams hold half its connection + /// window, nor on a transport that reports no window. + pub(crate) fn pause(&mut self) -> bool { + self.0.pause() + } + + /// Return the credit [`Self::pause`] reserved, as ingest reads the stream again. + pub(crate) fn resume(&mut self) { + self.0.resume() + } +} + +/// A transport's receive stream and the paused credit it reserved from its session. +struct Budgeted { + stream: T, + session: Budget, + /// The credit reserved while paused, zero while read. + held: u64, +} + +impl Budgeted { + /// Return the credit reserved while paused, if any. + fn release(&mut self) { + let held = std::mem::take(&mut self.held); + if held > 0 + && let Some(shared) = self.session.upgrade() + { + shared.paused.fetch_sub(held, Ordering::AcqRel); + } + } +} + +impl Drop for Budgeted { + fn drop(&mut self) { + self.release(); } } @@ -175,18 +260,20 @@ impl super::Metrics for Metrics { } trait ErasedSession: Send + Sync { - fn poll_accept_uni(&self, waiter: &kio::Waiter) -> Poll>; - fn poll_accept_bi(&self, waiter: &kio::Waiter) -> Poll>; + fn poll_accept_uni(&self, waiter: &kio::Waiter, budget: Budget) -> Poll>; + fn poll_accept_bi(&self, waiter: &kio::Waiter, budget: Budget) -> Poll>; fn poll_open_uni(&self, waiter: &kio::Waiter, options: SendOptions<'_, Group>) -> Poll>; fn poll_open_bi( &self, waiter: &kio::Waiter, options: SendOptions<'_, Group>, + budget: Budget, ) -> Poll>; fn poll_open_request( &self, waiter: &kio::Waiter, options: SendOptions<'_, Group>, + budget: Budget, ) -> Poll, Error>>; fn send_group(&self, priority: i64) -> Option; fn send_datagram(&self, payload: Bytes, options: SendOptions<'_, Group>) -> Result<(), Error>; @@ -199,6 +286,7 @@ trait ErasedSession: Send + Sync { fn metrics(&self) -> Metrics; fn wire(&self) -> Option; fn probe(&self) -> Option; + fn receive_window(&self) -> Option; } trait ErasedGroup: Send + Sync { @@ -222,6 +310,9 @@ trait ErasedRecv: Send + Sync { fn poll_read_chunk(&mut self, waiter: &kio::Waiter, max: usize) -> Poll, Error>>; fn set_priority(&mut self, priority: i64); fn stop(&mut self, code: u32); + fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll; + fn pause(&mut self) -> bool; + fn resume(&mut self); } // Not `Send`: a lease may hold its transport's lock, and never outlives a poll. @@ -247,18 +338,18 @@ fn unbox<'a, G: 'static>(options: SendOptions<'a, Group>) -> SendOptions<'a, G> } } -fn bi((send, recv): poll::BiStreams) -> (SendStream, RecvStream) { - (SendStream(Box::new(send)), RecvStream(Box::new(recv))) +fn bi((send, recv): poll::BiStreams, budget: Budget) -> (SendStream, RecvStream) { + (SendStream(Box::new(send)), RecvStream::budgeted(recv, budget)) } impl ErasedSession for S { - fn poll_accept_uni(&self, waiter: &kio::Waiter) -> Poll> { + fn poll_accept_uni(&self, waiter: &kio::Waiter, budget: Budget) -> Poll> { poll::Session::poll_accept_uni(self, waiter) - .map(|res| res.map(|recv| RecvStream(Box::new(recv))).map_err(Error::new)) + .map(|res| res.map(|recv| RecvStream::budgeted(recv, budget)).map_err(Error::new)) } - fn poll_accept_bi(&self, waiter: &kio::Waiter) -> Poll> { - poll::Session::poll_accept_bi(self, waiter).map(|res| res.map(bi::).map_err(Error::new)) + fn poll_accept_bi(&self, waiter: &kio::Waiter, budget: Budget) -> Poll> { + poll::Session::poll_accept_bi(self, waiter).map(|res| res.map(|bis| bi::(bis, budget)).map_err(Error::new)) } fn poll_open_uni(&self, waiter: &kio::Waiter, options: SendOptions<'_, Group>) -> Poll> { @@ -270,17 +361,22 @@ impl ErasedSession for S { &self, waiter: &kio::Waiter, options: SendOptions<'_, Group>, + budget: Budget, ) -> Poll> { - poll::Session::poll_open_bi(self, waiter, unbox(options)).map(|res| res.map(bi::).map_err(Error::new)) + poll::Session::poll_open_bi(self, waiter, unbox(options)) + .map(|res| res.map(|bis| bi::(bis, budget)).map_err(Error::new)) } fn poll_open_request( &self, waiter: &kio::Waiter, options: SendOptions<'_, Group>, + budget: Budget, ) -> Poll, Error>> { - poll::Session::poll_open_request(self, waiter, unbox(options)) - .map(|res| res.map(|streams| streams.map(bi::)).map_err(Error::new)) + poll::Session::poll_open_request(self, waiter, unbox(options)).map(|res| { + res.map(|streams| streams.map(|bis| bi::(bis, budget))) + .map_err(Error::new) + }) } fn send_group(&self, priority: i64) -> Option { @@ -326,6 +422,10 @@ impl ErasedSession for S { fn probe(&self) -> Option { poll::Session::probe(self) } + + fn receive_window(&self) -> Option { + poll::Session::receive_window(self) + } } impl ErasedGroup for G { @@ -378,23 +478,42 @@ impl + Send + Sync> ErasedSend for T { } } -impl ErasedRecv for T { +impl ErasedRecv for Budgeted { fn poll_lease(&mut self, waiter: &kio::Waiter, want: usize) -> Poll, Error>> { - poll::RecvStream::poll_lease(self, waiter, want) + poll::RecvStream::poll_lease(&mut self.stream, waiter, want) .map(|res| res.map(|lease| Lease(Box::new(lease))).map_err(Error::new)) } // Forwarded rather than left to the provided method, so reading a chunk never boxes a lease. fn poll_read_chunk(&mut self, waiter: &kio::Waiter, max: usize) -> Poll, Error>> { - poll::RecvStream::poll_read_chunk(self, waiter, max).map_err(Error::new) + poll::RecvStream::poll_read_chunk(&mut self.stream, waiter, max).map_err(Error::new) } fn set_priority(&mut self, priority: i64) { - poll::RecvStream::set_priority(self, priority) + poll::RecvStream::set_priority(&mut self.stream, priority) } fn stop(&mut self, code: u32) { - poll::RecvStream::stop(self, code) + poll::RecvStream::stop(&mut self.stream, code) + } + + fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll { + poll::RecvStream::poll_reset(&mut self.stream, waiter).map(Error::new) + } + + fn pause(&mut self) -> bool { + if self.held > 0 { + return true; + } + let Some(held) = self.session.upgrade().and_then(|shared| shared.reserve()) else { + return false; + }; + self.held = held; + true + } + + fn resume(&mut self) { + self.release(); } } @@ -418,15 +537,15 @@ impl poll::Session for Session { type Error = Error; fn poll_accept_uni(&self, waiter: &kio::Waiter) -> Poll> { - self.inner.poll_accept_uni(waiter) + self.inner.session.poll_accept_uni(waiter, self.budget()) } fn poll_accept_bi(&self, waiter: &kio::Waiter) -> Poll> { - self.inner.poll_accept_bi(waiter) + self.inner.session.poll_accept_bi(waiter, self.budget()) } fn poll_open_uni(&self, waiter: &kio::Waiter, options: SendOptions<'_, Group>) -> Poll> { - self.inner.poll_open_uni(waiter, options) + self.inner.session.poll_open_uni(waiter, options) } fn poll_open_bi( @@ -434,7 +553,7 @@ impl poll::Session for Session { waiter: &kio::Waiter, options: SendOptions<'_, Group>, ) -> Poll> { - self.inner.poll_open_bi(waiter, options) + self.inner.session.poll_open_bi(waiter, options, self.budget()) } fn poll_open_request( @@ -442,52 +561,56 @@ impl poll::Session for Session { waiter: &kio::Waiter, options: SendOptions<'_, Group>, ) -> Poll, Error>> { - self.inner.poll_open_request(waiter, options) + self.inner.session.poll_open_request(waiter, options, self.budget()) } fn send_group(&self, priority: i64) -> Option { - self.inner.send_group(priority) + self.inner.session.send_group(priority) } fn send_datagram(&self, payload: Bytes, options: SendOptions<'_, Group>) -> Result<(), Error> { - self.inner.send_datagram(payload, options) + self.inner.session.send_datagram(payload, options) } fn poll_recv_datagram(&self, waiter: &kio::Waiter) -> Poll> { - self.inner.poll_recv_datagram(waiter) + self.inner.session.poll_recv_datagram(waiter) } fn max_datagram_size(&self) -> usize { - self.inner.max_datagram_size() + self.inner.session.max_datagram_size() } fn protocol(&self) -> Option<&str> { - self.inner.protocol() + self.inner.session.protocol() } fn address(&self) -> Address { - self.inner.address() + self.inner.session.address() } fn close(&self, code: u32, reason: &str) { self.closing.sent(code, None); - self.inner.close(code, reason) + self.inner.session.close(code, reason) } fn poll_closed(&self, waiter: &kio::Waiter) -> Poll { - self.inner.poll_closed(waiter) + self.inner.session.poll_closed(waiter) } fn metrics(&self) -> impl super::Metrics { - self.inner.metrics() + self.inner.session.metrics() } fn wire(&self) -> Option { - self.inner.wire() + self.inner.session.wire() } fn probe(&self) -> Option { - self.inner.probe() + self.inner.session.probe() + } + + fn receive_window(&self) -> Option { + self.inner.session.receive_window() } } @@ -543,6 +666,10 @@ impl poll::RecvStream for RecvStream { fn stop(&mut self, code: u32) { self.0.stop(code) } + + fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll { + self.0.poll_reset(waiter) + } } impl Buf for Lease<'_> { @@ -577,3 +704,50 @@ impl poll::Lease for Lease<'_> { self.0.release() } } + +#[cfg(test)] +mod test { + use super::*; + use crate::lite::test_transport::SinkSession; + + fn accept(session: &Session) -> RecvStream { + let Poll::Ready(Ok((_, recv))) = poll::Session::poll_accept_bi(session, &kio::Waiter::noop()) else { + panic!("the sink accepts at once"); + }; + recv + } + + /// Each paused stream reserves its stream window, and the session pauses no more than + /// fit in half the connection window; a stream resuming or dropped returns its share. + #[test] + fn paused_streams_hold_at_most_half_the_connection_window() { + let gate = kio::Producer::new(true); + let session = + Session::new(SinkSession::accepted_bi(gate.consume()).with_receive_window(ReceiveWindow::new(100, 20))); + let mut streams: Vec<_> = (0..3).map(|_| accept(&session)).collect(); + assert!(streams[0].pause()); + assert!(streams[0].pause(), "a paused stream reserves once"); + assert!(streams[1].pause()); + assert!(!streams[2].pause(), "a third would hold 60 of 100"); + + streams[0].resume(); + assert!(streams[2].pause(), "a resumed stream returned its share"); + drop(streams.remove(1)); + assert!(accept(&session).pause(), "so did a dropped one"); + } + + /// A transport that reports no window never pauses a stream, since nothing bounds what + /// it would hold; nor does a stream that outlived its session. + #[test] + fn an_unbounded_stream_never_pauses() { + let gate = kio::Producer::new(true); + let session = Session::new(SinkSession::accepted_bi(gate.consume())); + assert!(!accept(&session).pause()); + + let session = + Session::new(SinkSession::accepted_bi(gate.consume()).with_receive_window(ReceiveWindow::new(100, 20))); + let mut stream = accept(&session); + drop(session); + assert!(!stream.pause()); + } +} diff --git a/rs/moq-net/src/transport/memory.rs b/rs/moq-net/src/transport/memory.rs index d41267eafd6..3a12c52602d 100644 --- a/rs/moq-net/src/transport/memory.rs +++ b/rs/moq-net/src/transport/memory.rs @@ -404,6 +404,14 @@ impl poll::Session for Session { fn poll_closed(&self, waiter: &kio::Waiter) -> Poll { poll_close(&self.side.conn.close, waiter) } + + /// Streams share no connection window, and one holds at most its buffer. + fn receive_window(&self) -> Option { + Some(super::ReceiveWindow::new( + u64::MAX, + self.conn().config.stream_buffer as u64, + )) + } } /// One direction of a stream: the messages its writer queued for its reader. @@ -647,6 +655,23 @@ impl poll::Chunks for Chunks { drop(channel); self.release(); } + + fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll { + if self.stopped { + return Poll::Ready(Error::Closed); + } + if let Poll::Ready(err) = poll_close(&self.close, waiter) { + return Poll::Ready(err); + } + let channel = ready!(self.channel.poll(waiter, |channel| match channel.end { + Some(End::Reset(_)) => Poll::Ready(()), + _ => Poll::Pending, + })); + let Some(End::Reset(code)) = channel.end else { + unreachable!("checked above") + }; + Poll::Ready(Error::Reset(code)) + } } impl Drop for Chunks { diff --git a/rs/moq-net/tests/integration/fastest_reader.rs b/rs/moq-net/tests/integration/fastest_reader.rs new file mode 100644 index 00000000000..d4112857e9b --- /dev/null +++ b/rs/moq-net/tests/integration/fastest_reader.rs @@ -0,0 +1,250 @@ +#![cfg(feature = "async")] + +//! A relay's ingest buffers at most a window past the group's fastest reader, then leaves +//! the rest in the stream, so flow control pushes back to the publisher. A slower reader +//! is never waited on: it falls off the group's front. + +use std::time::Duration; + +use bytes::Bytes; +use moq_net::{Hop, Timestamp, Version, origin, track}; + +use crate::support::harness::{MockPair, peer, spawn}; + +const TIMEOUT: Duration = Duration::from_secs(10); + +const MB: usize = 1 << 20; + +const VERSIONS: &[&str] = &["moq-lite-06", "moq-lite-07-wip", "moq-transport-22"]; + +fn origin(hop: u64) -> origin::Producer { + let (origin, driver) = origin::Producer::new(origin::Config::new(Hop::new(hop).unwrap())); + spawn(driver); + origin +} + +/// A publisher's track and a relay's subscription to it, over one peering. +struct Relay { + track: track::Producer, + subscriber: track::Subscriber, + _broadcast: origin::Publication, + pair: MockPair, + _origins: [origin::Producer; 2], +} + +async fn relay(version: &str) -> Relay { + let publisher = origin(1); + let relay = origin(2); + let pair = peer(version.parse::().unwrap(), &publisher, &relay).await; + + let broadcast = publisher.create_broadcast("live").unwrap(); + let track = broadcast + .create_track("data", track::Info::default().with_timescale(moq_net::Timescale::MILLI)) + .unwrap(); + broadcast.announce(Default::default()).unwrap(); + + let remote = kio::clock::timeout(TIMEOUT, relay.consume().follow("live", None).unwrap().available()) + .await + .expect("resolve timeout") + .expect("broadcast resolves"); + let subscriber = kio::clock::timeout( + TIMEOUT, + remote + .track("data") + .unwrap() + .subscribe(track::Subscription::default().with_max_delay(Duration::from_secs(3600))), + ) + .await + .expect("subscribe timeout") + .expect("subscribe"); + Relay { + track, + subscriber, + _broadcast: broadcast, + pair, + _origins: [publisher, relay], + } +} + +#[kio::test] +async fn an_idle_reader_paces_the_relay_and_loses_nothing() { + let payload = Bytes::from(vec![7u8; MB]); + for version in VERSIONS { + let mut relay = relay(version).await; + + let mut group = relay.track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + let mut reader = kio::clock::timeout(TIMEOUT, relay.subscriber.recv_group()) + .await + .expect("group timeout") + .unwrap() + .expect("group 0"); + // Within the publisher's own 32 MB tail, so only the relay's pacing holds it back. + for _ in 1..30 { + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + } + group.finish().unwrap(); + + // Simulated time moves only once every task is parked: the relay's ingest stopped. + kio::clock::sleep(Duration::from_secs(1)).await; + let held = reader.frame_count(); + assert!( + (15..=18).contains(&held), + "{version}: the relay read {held} MB ahead of a reader that took nothing" + ); + + for index in 0..30 { + let frame = kio::clock::timeout(TIMEOUT, reader.read_frame()) + .await + .unwrap_or_else(|_| panic!("{version}: frame {index} never arrived")) + .unwrap_or_else(|err| panic!("{version}: frame {index}: {err}")) + .expect("a frame"); + assert_eq!(frame.payload.len(), MB, "{version}: frame {index}"); + } + let end = kio::clock::timeout(TIMEOUT, reader.read_frame()) + .await + .expect("end timeout"); + assert!(matches!(end, Ok(None)), "{version}: {end:?}"); + } +} + +/// A paused stream is not read, so the relay watches it for a reset instead: a publisher +/// aborting the group ends the relay's copy while its reader still takes nothing. +#[kio::test] +async fn a_reset_reaches_a_paused_relay() { + let payload = Bytes::from(vec![7u8; MB]); + for version in VERSIONS { + let mut relay = relay(version).await; + + let mut group = relay.track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + let mut reader = kio::clock::timeout(TIMEOUT, relay.subscriber.recv_group()) + .await + .expect("group timeout") + .unwrap() + .expect("group 0"); + for _ in 1..30 { + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + } + kio::clock::sleep(Duration::from_secs(1)).await; + assert!(reader.frame_count() < 30, "{version}: the relay never paused"); + + group.abort(moq_net::Error::Cancel).unwrap(); + let end = kio::clock::timeout(TIMEOUT, reader.finished()) + .await + .unwrap_or_else(|_| panic!("{version}: the reset never reached the paused relay")); + assert!(end.is_err(), "{version}: {end:?}"); + } +} + +/// A frame larger than the room left waits to be admitted, since admitting it trims the +/// tail to fit: an idle reader loses nothing to a large frame either. +#[kio::test] +async fn a_large_frame_sheds_nothing_an_idle_reader_wants() { + let payload = Bytes::from(vec![7u8; MB]); + for version in VERSIONS { + let mut relay = relay(version).await; + + let mut group = relay.track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + let mut reader = kio::clock::timeout(TIMEOUT, relay.subscriber.recv_group()) + .await + .expect("group timeout") + .unwrap() + .expect("group 0"); + for _ in 1..15 { + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + } + // Let the relay take what fits, so the publisher's egress has sent every frame the + // large one sheds from the publisher's own tail. + kio::clock::sleep(Duration::from_secs(1)).await; + group + .write_frame(Timestamp::ZERO, Bytes::from(vec![7u8; 20 * MB])) + .unwrap(); + group.finish().unwrap(); + + kio::clock::sleep(Duration::from_secs(1)).await; + assert_eq!( + reader.frame_count(), + 15, + "{version}: the large frame waits for the reader" + ); + for index in 0..16 { + let frame = kio::clock::timeout(TIMEOUT, reader.read_frame()) + .await + .unwrap_or_else(|_| panic!("{version}: frame {index} never arrived")) + .unwrap_or_else(|err| panic!("{version}: frame {index}: {err}")) + .expect("a frame"); + let size = if index < 15 { MB } else { 20 * MB }; + assert_eq!(frame.payload.len(), size, "{version}: frame {index}"); + } + } +} + +/// A session whose paused streams would hold more than half its connection window reads +/// on instead, and its idle reader keeps what the live tail holds. +#[kio::test] +async fn a_session_past_its_credit_cap_reads_on() { + let payload = Bytes::from(vec![7u8; MB]); + for version in VERSIONS { + let mut relay = relay(version).await; + // One paused stream's credit is already more than half the connection window. + let window = moq_net::transport::ReceiveWindow::new(MB as u64, MB as u64); + relay.pair.server_transport.set_receive_window(Some(window)); + + let mut group = relay.track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + let reader = kio::clock::timeout(TIMEOUT, relay.subscriber.recv_group()) + .await + .expect("group timeout") + .unwrap() + .expect("group 0"); + for _ in 1..30 { + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + } + group.finish().unwrap(); + + kio::clock::sleep(Duration::from_secs(1)).await; + let held = reader.frame_count(); + assert_eq!(held, 30, "{version}: the relay paused past its credit cap"); + } +} + +#[kio::test] +async fn a_slow_reader_falls_off_without_slowing_the_fast_one() { + let payload = Bytes::from(vec![7u8; MB]); + for version in VERSIONS { + let mut relay = relay(version).await; + + let mut group = relay.track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + let mut fast = kio::clock::timeout(TIMEOUT, relay.subscriber.recv_group()) + .await + .expect("group timeout") + .unwrap() + .expect("group 0"); + let mut slow = fast.clone(); + + // Well past the 32 MB tail: the fast reader keeps up the whole way. + for index in 0..48 { + if index > 0 { + group.write_frame(Timestamp::ZERO, payload.clone()).unwrap(); + } + let frame = kio::clock::timeout(TIMEOUT, fast.read_frame()) + .await + .unwrap_or_else(|_| panic!("{version}: the fast reader stalled at frame {index}")) + .unwrap_or_else(|err| panic!("{version}: frame {index}: {err}")) + .expect("a frame"); + assert_eq!(frame.payload.len(), MB, "{version}: frame {index}"); + } + group.finish().unwrap(); + + let fell = kio::clock::timeout(TIMEOUT, slow.read_frame()) + .await + .expect("read timeout"); + assert!( + matches!(fell, Err(moq_net::Error::Evicted)), + "{version}: the slow reader got {fell:?}" + ); + } +} diff --git a/rs/moq-net/tests/integration/main.rs b/rs/moq-net/tests/integration/main.rs index 6c4c2ad082b..c92b429c3ac 100644 --- a/rs/moq-net/tests/integration/main.rs +++ b/rs/moq-net/tests/integration/main.rs @@ -23,6 +23,7 @@ mod drained_claim; mod empty_group; mod errored_end; mod extensions; +mod fastest_reader; mod fetch_cancel; mod fetch_fanout; mod fetch_from_feed; diff --git a/rs/moq-net/tests/integration/support/mock.rs b/rs/moq-net/tests/integration/support/mock.rs index 8f870f3d42e..0b7b82880f7 100644 --- a/rs/moq-net/tests/integration/support/mock.rs +++ b/rs/moq-net/tests/integration/support/mock.rs @@ -413,6 +413,14 @@ impl poll::Chunks for MockChunks { fn stop(&mut self, code: u32) { self.stop_after_latency(code); } + + // Unlike data, a reset skips the link's latency: no test times one. + fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll { + match &mut self.inner { + Some(inner) => poll::RecvStream::poll_reset(inner, waiter), + None => Poll::Ready(memory::Error::Closed), + } + } } impl Drop for MockChunks { @@ -454,6 +462,8 @@ struct Side { panic_on_stats: AtomicBool, /// The wire counters [`poll::Session::wire`] exposes, which a test updates as a transport would. wire: moq_net::transport::Wire, + /// What [`poll::Session::receive_window`] reports instead of the memory transport's, if set. + receive_window: Mutex>>, } /// The connection statistics a [`MockSession`] reports. Unavailable by default. @@ -643,6 +653,13 @@ impl poll::Session for MockSession { fn wire(&self) -> Option { Some(self.side.wire.clone()) } + + fn receive_window(&self) -> Option { + match *self.side.receive_window.lock().unwrap() { + Some(window) => window, + None => self.inner.receive_window(), + } + } } #[allow(dead_code, reason = "only some test binaries steer delivery")] @@ -676,6 +693,11 @@ impl MockSession { *self.side.stats.lock().unwrap() = stats; } + /// Report `window` as the receive windows in force, as a transport whose limits changed. + pub fn set_receive_window(&self, window: Option) { + *self.side.receive_window.lock().unwrap() = Some(window); + } + /// How many times the stats were read. pub fn stats_reads(&self) -> usize { self.side.stats_reads.load(Ordering::Relaxed) @@ -913,6 +935,7 @@ pub fn create_mock_session_pair(config: memory::Config) -> (MockSession, MockSes stats: Default::default(), stats_reads: Default::default(), panic_on_stats: Default::default(), + receive_window: Default::default(), wire: Default::default(), }), }; diff --git a/rs/moq-quic/src/connection/mod.rs b/rs/moq-quic/src/connection/mod.rs index 8327cb7028a..e0000595b5b 100644 --- a/rs/moq-quic/src/connection/mod.rs +++ b/rs/moq-quic/src/connection/mod.rs @@ -1672,6 +1672,17 @@ impl Connection { } } + /// The connection-wide receive window now in force; see [`Limits::receive_window`] + pub fn receive_window(&self) -> u64 { + self.streams.receive_window() + } + + /// The most receive credit one stream may hold: the largest per-stream window ever in force, + /// since credit already granted outlives a shrink; see [`Limits::stream_receive_window`] + pub fn stream_receive_window(&self) -> u64 { + self.streams.peak_stream_receive_window() + } + /// Change every limit `limits` sets, leaving the others as they are /// /// Errors, changing nothing, on limits [`Limits::validate`] refuses. diff --git a/rs/moq-quic/src/connection/streams/state.rs b/rs/moq-quic/src/connection/streams/state.rs index 400e7a10c6c..e1318f948a9 100644 --- a/rs/moq-quic/src/connection/streams/state.rs +++ b/rs/moq-quic/src/connection/streams/state.rs @@ -184,6 +184,9 @@ pub(crate) struct StreamsState { /// new stream starts with whatever [`stream_receive_window`](Self::stream_receive_window) /// became since pub(super) initial_stream_receive_window: u64, + /// The largest stream receive window ever in force, which bounds the credit a stream may + /// still hold after the window shrinks + peak_stream_receive_window: u64, // Pertinent state from the TransportParameters supplied by the peer initial_max_stream_data_uni: VarInt, @@ -255,6 +258,7 @@ impl StreamsState { send_window, stream_receive_window: stream_receive_window.into(), initial_stream_receive_window: stream_receive_window.into(), + peak_stream_receive_window: stream_receive_window.into(), initial_max_stream_data_uni: 0u32.into(), initial_max_stream_data_bidi_local: 0u32.into(), initial_max_stream_data_bidi_remote: 0u32.into(), @@ -1115,10 +1119,22 @@ impl StreamsState { self.unblock(); } + /// The connection receive window now in force. + pub(crate) fn receive_window(&self) -> u64 { + self.receive_window + } + + /// The most credit one receive stream may hold: the largest window ever in force, since + /// credit already granted outlives a shrink. + pub(crate) fn peak_stream_receive_window(&self) -> u64 { + self.peak_stream_receive_window + } + /// Set the receive window of each stream, which applies as each stream's credit is next /// extended: credit already granted cannot be taken back. pub(crate) fn set_stream_receive_window(&mut self, window: u64) { self.stream_receive_window = window; + self.peak_stream_receive_window = self.peak_stream_receive_window.max(window); } /// Set the receive_window and returns whether the receive_window has been @@ -2952,6 +2968,17 @@ mod tests { assert!(should_transmit.should_transmit()); } + /// A stream may still hold credit granted under a larger window, so the bound reported to + /// the application never falls when the window shrinks. + #[test] + fn stream_receive_window_reports_its_peak() { + let mut server = make(Side::Server); + let initial = server.peak_stream_receive_window(); + server.set_stream_receive_window(initial * 4); + server.set_stream_receive_window(initial / 2); + assert_eq!(server.peak_stream_receive_window(), initial * 4); + } + #[test] fn shrink_receive_window() { let mut server = make(Side::Server); diff --git a/rs/moq-tokio/src/quic/driver/connection.rs b/rs/moq-tokio/src/quic/driver/connection.rs index 6ed11b2bd4c..65174659d3c 100644 --- a/rs/moq-tokio/src/quic/driver/connection.rs +++ b/rs/moq-tokio/src/quic/driver/connection.rs @@ -528,6 +528,13 @@ impl Connection { self.0.state.lock().inner.stats() } + /// The receive windows now in force: the connection's, and the most one stream holds. + #[cfg(feature = "net")] + pub(crate) fn receive_window(&self) -> moq_net::transport::ReceiveWindow { + let state = self.0.state.lock(); + moq_net::transport::ReceiveWindow::new(state.inner.receive_window(), state.inner.stream_receive_window()) + } + /// How many send streams the connection still holds state for #[cfg(test)] pub(crate) fn send_streams(&self) -> usize { diff --git a/rs/moq-tokio/src/quic/driver/recv.rs b/rs/moq-tokio/src/quic/driver/recv.rs index cedb3567fa4..9139ef770cd 100644 --- a/rs/moq-tokio/src/quic/driver/recv.rs +++ b/rs/moq-tokio/src/quic/driver/recv.rs @@ -227,36 +227,39 @@ impl Stream { /// /// This operation is cancel-safe. pub async fn received_reset(&mut self) -> Result, Error> { - poll_fn(|cx| { - let mut conn = self.conn.state.lock(); + poll_fn(|cx| self.poll_received_reset(cx)).await + } - if let Some(code) = self.reset { - return Poll::Ready(Ok(Some(code))); - } + /// Poll form of [`Self::received_reset`]. It takes no bytes, so a lease in progress is + /// left as it was. + pub(crate) fn poll_received_reset(&mut self, cx: &mut Context<'_>) -> Poll, Error>> { + let mut conn = self.conn.state.lock(); - match conn.inner.recv_stream(self.id).received_reset() { - Err(_) => Poll::Ready(Ok(None)), - Ok(Some(error_code)) => { - // Stream state has just now been freed, so the connection may need to issue new - // stream ID flow control credit, and later reads fail with the cached code. - self.reset = Some(error_code); - conn.wake(); - Poll::Ready(Ok(Some(error_code))) - } - Ok(None) => { - if let Some(e) = &conn.error { - return Poll::Ready(Err(e.clone().into())); - } - // Resets always notify readers, since a reset is an immediate read error. We - // could introduce a dedicated channel to reduce the risk of spurious wakeups, - // but that increased complexity is probably not justified, as an application - // that is expecting a reset is not likely to receive large amounts of data. - conn.blocked_readers.insert(self.id, cx.waker().clone()); - Poll::Pending + if let Some(code) = self.reset { + return Poll::Ready(Ok(Some(code))); + } + + match conn.inner.recv_stream(self.id).received_reset() { + Err(_) => Poll::Ready(Ok(None)), + Ok(Some(error_code)) => { + // Stream state has just now been freed, so the connection may need to issue new + // stream ID flow control credit, and later reads fail with the cached code. + self.reset = Some(error_code); + conn.wake(); + Poll::Ready(Ok(Some(error_code))) + } + Ok(None) => { + if let Some(e) = &conn.error { + return Poll::Ready(Err(e.clone().into())); } + // Resets always notify readers, since a reset is an immediate read error. We + // could introduce a dedicated channel to reduce the risk of spurious wakeups, + // but that increased complexity is probably not justified, as an application + // that is expecting a reset is not likely to receive large amounts of data. + conn.blocked_readers.insert(self.id, cx.waker().clone()); + Poll::Pending } - }) - .await + } } /// Handle common logic related to reading out of a receive stream diff --git a/rs/moq-tokio/src/quic/web/recv.rs b/rs/moq-tokio/src/quic/web/recv.rs index 541774131fb..af1188809c9 100644 --- a/rs/moq-tokio/src/quic/web/recv.rs +++ b/rs/moq-tokio/src/quic/web/recv.rs @@ -2,7 +2,7 @@ use std::{ io, pin::Pin, sync::Arc, - task::{Context, Poll}, + task::{Context, Poll, ready}, }; use bytes::{BufMut, Bytes}; @@ -45,6 +45,20 @@ impl RecvStream { } } + /// Poll until the peer resets the stream or the session closes, without reading it; pending + /// for good once every byte arrived. See [`recv::Stream::received_reset`]. + pub fn poll_reset(&mut self, cx: &mut Context<'_>) -> Poll { + let close = &self.close; + let Some(inner) = &mut self.inner else { + return Poll::Ready(ReadError::ClosedStream); + }; + match ready!(inner.poll_received_reset(cx)) { + Ok(None) => Poll::Pending, + Ok(Some(code)) => Poll::Ready(map_error(close, recv::Error::Reset(code))), + Err(err) => Poll::Ready(map_error(close, err)), + } + } + /// Rank this stream for eviction. See [`recv::Stream::set_priority`]. pub fn set_priority(&mut self, priority: i64) { if let Some(inner) = &mut self.inner { diff --git a/rs/moq-tokio/src/transport.rs b/rs/moq-tokio/src/transport.rs index 1cd7f00d23c..fba3824bb32 100644 --- a/rs/moq-tokio/src/transport.rs +++ b/rs/moq-tokio/src/transport.rs @@ -908,6 +908,10 @@ impl Grouped for crate::quic::web::Session { Some(crate::quic::driver::connection::Connection::stats(self).credit.evicted) } + fn receive_window(&self) -> Option { + Some(crate::quic::driver::connection::Connection::receive_window(self)) + } + fn send_group(&self, priority: i64) -> Option { Some(Self::send_group(self, priority)) } diff --git a/rs/moq-tokio/src/transport/grouped.rs b/rs/moq-tokio/src/transport/grouped.rs index 4a10d7ecddd..89cc0c54089 100644 --- a/rs/moq-tokio/src/transport/grouped.rs +++ b/rs/moq-tokio/src/transport/grouped.rs @@ -42,6 +42,12 @@ pub trait Grouped: web_transport_trait::Session { None } + /// The receive windows now in force, or `None` when the backend cannot say; see + /// [`poll::Session::receive_window`](moq_net::transport::poll::Session::receive_window). + fn receive_window(&self) -> Option { + None + } + /// Open a send group at `priority`, or `None` when the backend has none. fn send_group(&self, priority: i64) -> Option<::Group> { let _ = priority; diff --git a/rs/moq-tokio/src/transport/owned.rs b/rs/moq-tokio/src/transport/owned.rs index 9ded0bbba8b..f5cd027d453 100644 --- a/rs/moq-tokio/src/transport/owned.rs +++ b/rs/moq-tokio/src/transport/owned.rs @@ -200,6 +200,9 @@ impl> poll::Session fn probe(&self) -> Option { super::Grouped::probe(&self.session) } + fn receive_window(&self) -> Option { + super::Grouped::receive_window(&self.session) + } } impl poll::SendStream for BridgedSend { type Error = Error; @@ -259,6 +262,9 @@ impl poll::RecvStream for crate::quic::web::RecvStream { // A stream already closed has nobody left to tell. crate::quic::web::RecvStream::stop(self, code); } + fn poll_reset(&mut self, waiter: &moq_net::kio::Waiter) -> Poll { + crate::quic::web::RecvStream::poll_reset(self, &mut waiter.context()).map(Error) + } } #[cfg(feature = "quic")] diff --git a/rs/moq-uring/src/quic/driver/connection.rs b/rs/moq-uring/src/quic/driver/connection.rs index bca44657742..5dc203793fa 100644 --- a/rs/moq-uring/src/quic/driver/connection.rs +++ b/rs/moq-uring/src/quic/driver/connection.rs @@ -393,6 +393,12 @@ impl Connection { pub(crate) fn streams_evicted(&self) -> u64 { self.shared.conn.borrow().stats().credit.evicted } + + /// The receive windows now in force: the connection's, and the most one stream holds. + pub(crate) fn receive_window(&self) -> moq_net::transport::ReceiveWindow { + let conn = self.shared.conn.borrow(); + moq_net::transport::ReceiveWindow::new(conn.receive_window(), conn.stream_receive_window()) + } } impl Clone for Connection { diff --git a/rs/moq-uring/src/quic/driver/recv.rs b/rs/moq-uring/src/quic/driver/recv.rs index 4a4fbe19f96..c205f6c7f36 100644 --- a/rs/moq-uring/src/quic/driver/recv.rs +++ b/rs/moq-uring/src/quic/driver/recv.rs @@ -215,7 +215,13 @@ impl Stream { /// /// Cancel-safe. pub async fn received_reset(&mut self) -> Result, Error> { - poll_fn(|cx| self.parked(cx, Self::wait_received_reset)).await + poll_fn(|cx| self.poll_received_reset(cx)).await + } + + /// Poll form of [`Self::received_reset`]. It takes no bytes, so a lease in progress is + /// left as it was. + pub(crate) fn poll_received_reset(&mut self, cx: &Context<'_>) -> Poll, Error>> { + self.parked(cx, Self::wait_received_reset) } fn wait_received_reset(&mut self, waiter: &kio::Waiter) -> Poll, Error>> { diff --git a/rs/moq-uring/src/quic/web.rs b/rs/moq-uring/src/quic/web.rs index 3cbc17bba1c..751ca45747f 100644 --- a/rs/moq-uring/src/quic/web.rs +++ b/rs/moq-uring/src/quic/web.rs @@ -459,6 +459,11 @@ impl Session { self.conn.streams_evicted() } + /// The receive windows now in force: the connection's, and the most one stream holds. + pub(crate) fn receive_window(&self) -> moq_net::transport::ReceiveWindow { + self.conn.receive_window() + } + /// Open a send group at `priority`, for streams to join with /// [`SendStream::set_group`]. The HTTP/3 streams stay outside every /// group, so they go first. @@ -1119,6 +1124,20 @@ impl RecvStream { inner.set_priority(priority); } } + + /// Poll until the peer resets the stream or the connection closes, without reading it; + /// pending for good once every byte arrived. See [`recv::Stream::received_reset`]. + pub fn poll_reset(&mut self, cx: &mut Context<'_>) -> Poll { + let web = self.web; + let Some(inner) = &mut self.inner else { + return Poll::Ready(Error::Closed); + }; + match ready!(inner.poll_received_reset(cx)) { + Ok(None) => Poll::Pending, + Ok(Some(code)) => Poll::Ready(map(web, Error::Reset(code.into_inner()))), + Err(err) => Poll::Ready(map(web, err.into())), + } + } } /// Unmap a stream error's code from the session's code space. diff --git a/rs/moq-uring/src/transport.rs b/rs/moq-uring/src/transport.rs index 6b043479fd7..7489caeef04 100644 --- a/rs/moq-uring/src/transport.rs +++ b/rs/moq-uring/src/transport.rs @@ -37,6 +37,10 @@ impl Grouped for quic::web::Session { Some(quic::web::Session::streams_evicted(self)) } + fn receive_window(&self) -> Option { + Some(quic::web::Session::receive_window(self)) + } + fn send_group(&mut self, priority: i64) -> Option { Some(quic::web::Session::send_group(self, priority)) } @@ -92,6 +96,9 @@ impl moq_net::transport::poll::RecvStream for quic::web::RecvStream { fn stop(&mut self, code: u32) { web_transport_trait::poll::RecvStream::stop(self, code) } + fn poll_reset(&mut self, waiter: &kio::Waiter) -> std::task::Poll { + quic::web::RecvStream::poll_reset(self, &mut waiter.context()).map(adapter::Error) + } } #[cfg(feature = "quic")] diff --git a/rs/moq-uring/src/transport/adapter.rs b/rs/moq-uring/src/transport/adapter.rs index d54a97220f1..db65d110dc6 100644 --- a/rs/moq-uring/src/transport/adapter.rs +++ b/rs/moq-uring/src/transport/adapter.rs @@ -89,6 +89,12 @@ pub trait Grouped: backend::Session { None } + /// The receive windows now in force, or `None` when the backend cannot say; see + /// [`poll::Session::receive_window`](moq_net::transport::poll::Session::receive_window). + fn receive_window(&self) -> Option { + None + } + /// Open a send group at `priority`, or `None` when the backend has none. fn send_group(&mut self, priority: i64) -> Option<::Group> { let _ = priority; @@ -254,6 +260,9 @@ impl + 'static> poll::Sessi fn probe(&self) -> Option { Grouped::probe(&self.session) } + fn receive_window(&self) -> Option { + Grouped::receive_window(&self.session) + } } /// An outgoing WebTransport stream adapted to moq-net. /// diff --git a/rs/moq-uring/src/transport/local.rs b/rs/moq-uring/src/transport/local.rs index 571cbc2b246..90890356063 100644 --- a/rs/moq-uring/src/transport/local.rs +++ b/rs/moq-uring/src/transport/local.rs @@ -119,6 +119,10 @@ impl poll::Session for Local { fn probe(&self) -> Option { self.0.probe() } + + fn receive_window(&self) -> Option { + self.0.receive_window() + } } impl poll::SendStream for Local { @@ -173,6 +177,10 @@ impl poll::RecvStream for Local { self.0.stop(code) } + fn poll_reset(&mut self, waiter: &kio::Waiter) -> Poll { + self.0.poll_reset(waiter) + } + fn poll_read_chunk(&mut self, waiter: &kio::Waiter, max: usize) -> Poll, Self::Error>> { self.0.poll_read_chunk(waiter, max) }