From 3525adcf81c6720e0a5402e7c2e10b3b2a0b974b Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 11:05:00 -0700 Subject: [PATCH 1/9] chore: claim subscriber close tail drain From b7afeba251c81da47af9a0734236a19057c14db8 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 12:05:32 -0700 Subject: [PATCH 2/9] fix(net): drain received group tails after session close Retain subscription state while the closed transport drains buffered streams and settle IETF control waits. Deliver pending WebTransport uni streams before surfacing connection close. Regress published Lite versions and both IETF session paths, including control-first polling. Co-Authored-By: GPT-6 --- quest/m1/README.md | 1 - quest/m1/close-tail-drain.md | 83 ---------- rs/moq-net/src/ietf/adapter.rs | 45 ++++-- rs/moq-net/src/ietf/session.rs | 222 ++++++++++++++------------ rs/moq-net/src/ietf/subscriber.rs | 114 +++++++++++++ rs/moq-net/src/lite/session.rs | 71 +++++--- rs/moq-net/src/lite/subscriber.rs | 12 ++ rs/moq-net/src/lite/test_transport.rs | 92 ++++++++--- rs/moq-net/src/session.rs | 3 + rs/moq-net/src/transport/memory.rs | 16 +- rs/moq-net/tests/session_close.rs | 66 +++++++- rs/moq-net/tests/support/harness.rs | 31 +++- rs/moq-tokio/src/quic/web/session.rs | 45 ++++-- rs/moq-uring/src/quic/web.rs | 15 +- rs/moq-uring/tests/web.rs | 35 ++++ 15 files changed, 569 insertions(+), 282 deletions(-) delete mode 100644 quest/m1/close-tail-drain.md diff --git a/quest/m1/README.md b/quest/m1/README.md index e0654e4f0ef..2de6c03159a 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -146,7 +146,6 @@ above them. - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS - [A/V sync across a break](/quest/m1/watch-break-av-sync.md) - `@moq/watch` never plays pre-break audio out of sync with pre-break video - [Plan: HLS export failure signal](/quest/m1/hls-failure-signal.md) - decide what players see when the exporter's timeline fails, live or history, so a truncated recording never looks complete -- [Close tail drain](/quest/m1/close-tail-drain.md) - after a peer close, subscribers read the group streams their transport already delivered instead of aborting them, so a finished tail arrives on every version - [Publisher linger](/quest/m1/publisher-linger.md) - every finishing publisher keeps its broadcast up for the media max-age so subscribers can drain it - [Vulkan Video encode on AMD](/quest/m1/vulkan-encode.md) - H.264 and H.265 from an external Vulkan image on RADV, hand-rolled on ash, proven on an RX 9070 - [VA-API encodes an external Vulkan image](/quest/m1/vaapi-vulkan-import.md) - an explicit-sync DMA-BUF reaches Intel's encoder, proven on Arrow Lake diff --git a/quest/m1/close-tail-drain.md b/quest/m1/close-tail-drain.md deleted file mode 100644 index 755f5c56d74..00000000000 --- a/quest/m1/close-tail-drain.md +++ /dev/null @@ -1,83 +0,0 @@ -# [M] Subscribers drain the tail after a peer close - -## Goal - -A group the publisher finished before a graceful session close reaches the -subscriber on every version (lite-01 through lite-07, and IETF): after a peer -close, the subscriber keeps reading the group streams its transport already -delivered instead of aborting them. Reproduce the loss first with a regression -test that fails without the fix. No wire change. - -## Plan - -Found while reworking the in-memory transport (#5530). Facts at 813a92b48: - -- On a peer close, lite `UniAccept::poll` returns the accept error - (`rs/moq-net/src/lite/subscriber.rs:712`), and `Driver::poll` - (`rs/moq-net/src/lite/session.rs:381-411`) aborts every subscription when - `local_close` is false, then drops the accepted `UniServe` children - mid-read. IETF `run_unis` (`rs/moq-net/src/ietf/session.rs:882`, - `:931-942`) does the same, and `ietf::session::start` aborts again after - the race in each of its two arms, drafts 14-16 (`:371-374`) and drafts 17+ - (`:588-591`). -- Real QUIC keeps that data readable: moq-tokio's `quic::driver` `poll_accept` hands out - already-received streams before the connection error, and reads return - buffered chunks and the FIN, reporting `ConnectionLost` only when blocked. - moq-uring's raw QUIC follows the same order, but its WebTransport layer - does not: `poll_accept_uni` (`rs/moq-uring/src/quic/web.rs:566-569`) - returns the connection error before classifying streams already in - `pending_uni`, so a tail stream accepted in the same poll as the close is - lost below the subscriber. -- The closer (`Session::close`, `rs/moq-net/src/session.rs:201`) waits for - its group streams' FIN acks. An ACK only means delivery to the peer's - transport, not that the peer processed it, so on lite-01 through lite-06 - and IETF the receiver throws away a tail its transport already holds. - lite-07 already avoids this at the protocol level: the closer waits for the - subscriber's Subscribe-stream FIN (`drafts/draft-lcurley-moq-lite.md`). -- The test mock (`rs/moq-net/tests/support/mock.rs`) hid it: its FIN counts as - acked only once the reading application stops or drops the stream. - `older_lite_close_keeps_ack_drain` never checks what the reader got. - -Decided (maintainer, 2026-10-14): - -- Fix at the receiver. After a peer close, the lite and IETF subscribers - stop aborting accepted group streams and read each to its FIN or until the - transport reports it closed. Buffered data bounds the drain, so no timer. - Streams not yet accepted are still accepted and drained in the same way. - The closer-side wait stays lite-07 only, since older published versions - can't gain a wire change. - Make the drain a session-level phase, not just a change to `UniAccept` and - `run_unis`: a control-stream or publisher error can end the driver first, - and subscription cleanup can remove a track a data stream still needs. - Keep the subscription state the draining streams depend on, and settle - anything that can't complete after the close (such as IETF children waiting - on control state) so the no-timer argument holds. That argument needs a - closed transport: a non-transport terminal error (such as legacy lite's - session-stream FIN returning `Cancel`, `rs/moq-net/src/lite/session.rs:652-660`) - must not start a drain over a live connection that could hang. Close the - transport first, or abort as today if buffered data does not survive a - local close. -- Tests ack a FIN on delivery, as QUIC does, so the regression for lite-01 - through lite-06 and IETF (one draft from each `start` arm) asserts the - reader got the tail and fails before the fix. Cover both the buffered-tail - case and one where a control or accept error wins the poll order. Give - `older_lite_close_keeps_ack_drain` that reader assertion. -- Once #5530 has landed, switch `moq_net::transport::memory`'s - `SendStream::poll_closed` to resolve on `finish` (delivery into the - channel), dropping its "finished and empty" rule. Do this unless the - memory bench (`rs/moq-net/benches/memory.rs`) shows a regression; if it - does, change only the tests. - -The receiver drain also needs moq-uring WebTransport to classify and hand -out its pending uni streams before reporting the connection error, with a -regression for a tail stream and CONNECTION_CLOSE arriving in the same poll. -Check the other `poll_accept_uni` adapters (such as `rs/moq-ffi` and -moq-tokio's `quic::web` path) for the same ordering. - -Public API: none. Wire: none. - -## Related - -- [Publisher linger](/quest/m1/publisher-linger.md) - keeps a finished broadcast servable so subscribers can drain it, a layer above this -- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - groups cut short by a session close -- [QUIC ack hook](/quest/m2/quic-ack-hook.md) - a FIN/ack hook in the QUIC layer diff --git a/rs/moq-net/src/ietf/adapter.rs b/rs/moq-net/src/ietf/adapter.rs index 03ab85c3f76..03e453309c8 100644 --- a/rs/moq-net/src/ietf/adapter.rs +++ b/rs/moq-net/src/ietf/adapter.rs @@ -450,18 +450,45 @@ impl crate::transport::poll::SendStream for } } +/// Preserve the real receive stream's registries through the shared-control adapter. +/// Virtual streams have only local errors, while data streams still carry raw codes. +#[derive(Debug, thiserror::Error)] +#[non_exhaustive] +pub enum RecvError { + #[error(transparent)] + Real(E), + #[error(transparent)] + Virtual(crate::Error), +} + +impl crate::transport::Error for RecvError { + fn session_error(&self) -> Option<(u32, String)> { + match self { + Self::Real(err) => err.session_error(), + Self::Virtual(err) => err.session_error(), + } + } + + fn stream_error(&self) -> Option { + match self { + Self::Real(err) => err.stream_error(), + Self::Virtual(err) => err.stream_error(), + } + } +} + pub enum AdapterRecv { Real(S::RecvStream), Virtual(VirtualRecvStream), } impl crate::transport::poll::RecvStream for AdapterRecv { - type Error = crate::Error; + type Error = RecvError<::Error>; fn poll_read(&mut self, cx: &mut Context<'_>, dst: &mut [u8]) -> Poll, Self::Error>> { match self { - Self::Real(s) => s.poll_read(cx, dst).map(|r| r.map_err(|_| crate::Error::Closed)), - Self::Virtual(s) => s.poll_read(cx, dst), + Self::Real(s) => s.poll_read(cx, dst).map(|r| r.map_err(RecvError::Real)), + Self::Virtual(s) => s.poll_read(cx, dst).map_err(RecvError::Virtual), } } @@ -471,15 +498,15 @@ impl crate::transport::poll::RecvStream for buf: &mut B, ) -> Poll, Self::Error>> { match self { - Self::Real(s) => s.poll_read_buf(cx, buf).map(|r| r.map_err(|_| crate::Error::Closed)), - Self::Virtual(s) => s.poll_read_buf(cx, buf), + Self::Real(s) => s.poll_read_buf(cx, buf).map(|r| r.map_err(RecvError::Real)), + Self::Virtual(s) => s.poll_read_buf(cx, buf).map_err(RecvError::Virtual), } } fn poll_read_chunk(&mut self, cx: &mut Context<'_>, max: usize) -> Poll, Self::Error>> { match self { - Self::Real(s) => s.poll_read_chunk(cx, max).map(|r| r.map_err(|_| crate::Error::Closed)), - Self::Virtual(s) => s.poll_read_chunk(cx, max), + Self::Real(s) => s.poll_read_chunk(cx, max).map(|r| r.map_err(RecvError::Real)), + Self::Virtual(s) => s.poll_read_chunk(cx, max).map_err(RecvError::Virtual), } } @@ -492,8 +519,8 @@ impl crate::transport::poll::RecvStream for fn poll_closed(&mut self, cx: &mut Context<'_>) -> Poll> { match self { - Self::Real(s) => s.poll_closed(cx).map(|r| r.map_err(|_| crate::Error::Closed)), - Self::Virtual(s) => s.poll_closed(cx), + Self::Real(s) => s.poll_closed(cx).map(|r| r.map_err(RecvError::Real)), + Self::Virtual(s) => s.poll_closed(cx).map_err(RecvError::Virtual), } } } diff --git a/rs/moq-net/src/ietf/session.rs b/rs/moq-net/src/ietf/session.rs index af9a871587a..5c90e31cc99 100644 --- a/rs/moq-net/src/ietf/session.rs +++ b/rs/moq-net/src/ietf/session.rs @@ -284,7 +284,7 @@ where // Every half only ends the session on error (err_only parks on clean // completion); the task set draining is the one clean exit. let mut adapter_run = std::pin::pin!(err_only(adapter.run(setup.reader, setup.writer, goaway.clone()))); - let mut unis = std::pin::pin!(err_only(run_unis( + let mut unis = Unis::new( adapter.clone(), subscriber.clone(), UniSetup { @@ -293,9 +293,8 @@ where early: Vec::new(), version, goaway, - local_close: closing.clone() }, - ))); + ); let mut dispatch = std::pin::pin!(err_only(run_dispatch( dispatch_session, publisher.clone(), @@ -347,8 +346,8 @@ where if let Poll::Ready(err) = waiter.poll_future(adapter_run.as_mut()) { return Poll::Ready(Err::<(), Error>(err)); } - if let Poll::Ready(err) = waiter.poll_future(unis.as_mut()) { - return Poll::Ready(Err(err)); + if let Poll::Ready(res) = unis.poll(waiter) { + return Poll::Ready(res); } if let Poll::Ready(err) = waiter.poll_future(dispatch.as_mut()) { return Poll::Ready(Err(err)); @@ -368,6 +367,10 @@ where Poll::Pending }) .await; + close_transport(&mut session, &res); + if !closing.load(std::sync::atomic::Ordering::Relaxed) { + kio::wait(|waiter| unis.poll_drain(waiter)).await; + } if closing.load(std::sync::atomic::Ordering::Relaxed) { subscriber.close(); } else if let Err(err) = &res { @@ -491,7 +494,7 @@ where // Every half only ends the session on error (err_only parks on clean // completion); `setup` never resolves (it holds the stream open) and the // task set draining is the one clean exit. - let mut unis = std::pin::pin!(err_only(run_unis( + let mut unis = Unis::new( session.clone(), subscriber.clone(), UniSetup { @@ -500,9 +503,8 @@ where early: early_unis, version, goaway, - local_close: closing.clone() }, - ))); + ); let mut dispatch = std::pin::pin!(err_only(run_dispatch( session.clone(), publisher.clone(), @@ -546,8 +548,8 @@ where let res = kio::wait(|waiter| { use std::task::Poll; - if let Poll::Ready(err) = waiter.poll_future(unis.as_mut()) { - return Poll::Ready(Err::<(), Error>(err)); + if let Poll::Ready(res) = unis.poll(waiter) { + return Poll::Ready(res); } if let Poll::Ready(err) = waiter.poll_future(dispatch.as_mut()) { return Poll::Ready(Err(err)); @@ -586,6 +588,10 @@ where Ok(()) => Error::Cancel, Err(err) => err.clone(), }); + close_transport(&mut session, &res); + if !closing.load(std::sync::atomic::Ordering::Relaxed) { + kio::wait(|waiter| unis.poll_drain(waiter)).await; + } if closing.load(std::sync::atomic::Ordering::Relaxed) { subscriber.close(); } else if let Err(err) = &res { @@ -601,24 +607,6 @@ where Err(err) => err.clone(), }); - match &res { - Err(err @ Error::Transport(_)) => { - tracing::info!(%err, "session terminated"); - session.close(super::error::to_session_code(&SessionError::Internal), ""); - } - Err(err) => { - tracing::warn!(%err, "session error"); - session.close( - super::error::to_session_code(&SessionError::from(err)), - err.to_string().as_ref(), - ); - } - _ => { - tracing::info!("session closed"); - session.close(super::error::to_session_code(&SessionError::Cancel), ""); - } - } - res } .maybe_boxed(); @@ -864,6 +852,26 @@ impl UniType { } } +fn close_transport(session: &mut S, res: &Result<(), Error>) { + match res { + Err(err @ Error::Transport(_)) => { + tracing::info!(%err, "session terminated"); + session.close(super::error::to_session_code(&SessionError::Internal), ""); + } + Err(err) => { + tracing::warn!(%err, "session error"); + session.close( + super::error::to_session_code(&SessionError::from(err)), + err.to_string().as_ref(), + ); + } + _ => { + tracing::info!("session closed"); + session.close(super::error::to_session_code(&SessionError::Cancel), ""); + } + } +} + /// Setup and lifecycle state for the incoming uni stream dispatcher. struct UniSetup { peer: Option, @@ -872,7 +880,6 @@ struct UniSetup { early: Vec>, version: Version, goaway: crate::goaway::Protocol, - local_close: std::sync::Arc, } /// Accept incoming uni streams, including SETUP and GOAWAY on draft-17+. @@ -882,81 +889,87 @@ struct UniSetup { /// the peer has not sent yet, held back by connection flow control that the later /// streams' unread data is using up. Waiting here for its type would deadlock the /// connection. -async fn run_unis(mut session: S, subscriber: Subscriber, setup: UniSetup) -> Result<(), Error> -where - S: crate::transport::poll::Boxable, -{ - use std::task::Poll; +struct Unis { + session: S, + uni: Uni, + early: std::collections::VecDeque>, + tasks: TaskSet, + fatal: kio::Shared>, + ended: Option, +} - let UniSetup { - peer: peer_setup, - read: setup_read, - early, - version, - goaway, - local_close, - } = setup; - let outer_version = crate::Version::Ietf(version); - let mut tasks = TaskSet::owned(); - let uni = Uni { - session: session.clone(), - subscriber: subscriber.clone(), - peer_setup, - // A gated server accept already read the peer's one SETUP off its own uni stream, - // so anything arriving here is a second one. - seen_setup: std::sync::Arc::new(std::sync::atomic::AtomicBool::new(setup_read)), - goaway, - version, - }; - // The first stream to break the session, recorded by the child that read it. - let fatal = kio::Shared::>::default(); - let mut early = early.into_iter(); +impl Unis { + fn new(session: S, subscriber: Subscriber, setup: UniSetup) -> Self { + Self { + uni: Uni { + session: session.clone(), + subscriber, + peer_setup: setup.peer, + seen_setup: std::sync::Arc::new(std::sync::atomic::AtomicBool::new(setup.read)), + goaway: setup.goaway, + version: setup.version, + }, + session, + early: setup.early.into(), + tasks: TaskSet::owned(), + fatal: Default::default(), + ended: None, + } + } - loop { - let reader = match early.next() { - Some(reader) => reader, - None => { - let recv = tasks - .drive(|waiter| { - let broken = fatal.poll(waiter, |fatal| match fatal.is_some() { - true => Poll::Ready(()), - false => Poll::Pending, - }); - if let Poll::Ready(mut fatal) = broken - && let Some(err) = fatal.take() - { - return Poll::Ready(Err(err)); - } - let mut cx = waiter.context(); - session.poll_accept_uni(&mut cx).map(Ok) - }) - .await; - let recv = match recv { - Ok(Ok(recv)) => recv, - Ok(Err(err)) => { - let err = Error::from_transport(err); - // Settle tracks and open groups before the owned receive tasks drop - // their cancellation guards, which would otherwise record Cancel. - if local_close.load(std::sync::atomic::Ordering::Relaxed) { - subscriber.close(); - } else { - subscriber.abort(&err); - } - return Err(err); + fn accept(&mut self, waiter: &kio::Waiter) { + while self.ended.is_none() { + let reader = match self.early.pop_front() { + Some(reader) => reader, + None => match self.session.poll_accept_uni(&mut waiter.context()) { + std::task::Poll::Ready(Ok(recv)) => Reader::new(recv, crate::Version::Ietf(self.uni.version)), + std::task::Poll::Ready(Err(err)) => { + self.ended = Some(Error::from_transport(err)); + break; } - Err(fatal) => return Err(fatal), - }; - Reader::new(recv, outer_version) - } - }; + std::task::Poll::Pending => break, + }, + }; + let uni = self.uni.clone(); + let fatal = self.fatal.clone(); + self.tasks.push(async move { + if let Err(err) = uni.serve(reader).await { + fatal.lock().get_or_insert(err); + } + }); + } + } - let uni = uni.clone(); - let fatal = fatal.clone(); - tasks.push(async move { - if let Err(err) = uni.serve(reader).await { - fatal.lock().get_or_insert(err); - } - }); + fn poll(&mut self, waiter: &kio::Waiter) -> std::task::Poll> { + let _ = self.tasks.poll(waiter); + if let Some(err) = self.fatal.lock().take() { + return std::task::Poll::Ready(Err(err)); + } + self.accept(waiter); + let _ = self.tasks.poll(waiter); + if let Some(err) = self.fatal.lock().take() { + return std::task::Poll::Ready(Err(err)); + } + match &self.ended { + Some(err) => std::task::Poll::Ready(Err(err.clone())), + None => std::task::Poll::Pending, + } + } + + fn poll_drain(&mut self, waiter: &kio::Waiter) -> std::task::Poll<()> { + self.accept(waiter); + let drained = self.tasks.poll(waiter).is_ready(); + if self.ended.is_none() { + return std::task::Poll::Pending; + } + // Every accepted child has first consumed its buffered header. A fill + // still waiting to be promised or opened can no longer arrive. + self.uni.subscriber.settle_control(); + if drained { + std::task::Poll::Ready(()) + } else { + self.tasks.poll(waiter) + } } } @@ -2454,7 +2467,7 @@ mod tests { // group stream racing ahead of it lands in. let read = accepted.is_some(); let early = accepted.map(|accepted| accepted.early).unwrap_or_default(); - let mut unis = std::pin::pin!(run_unis( + let mut unis = Unis::new( session, subscriber, UniSetup { @@ -2463,12 +2476,11 @@ mod tests { early, version, goaway, - local_close: Default::default() - } - )); + }, + ); for _ in 0..100 { - if let std::task::Poll::Ready(result) = futures::poll!(unis.as_mut()) { + if let std::task::Poll::Ready(result) = unis.poll(&kio::Waiter::noop()) { return (log, Some(result)); } if !log.stops().is_empty() { diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index dbe54f0f891..9d8021bfa77 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -59,6 +59,8 @@ enum Alias { /// still remember. #[derive(Default)] struct AliasTable { + /// No control response can bind another alias once receive draining starts. + ended: bool, map: HashMap, /// Retired aliases in retirement order, so the oldest is forgotten first. @@ -723,6 +725,7 @@ async fn resolve_track_alias( // our STOP_SENDING. Discard the group now rather than waiting out the timeout // for a binding that is never coming. Some(Alias::Retired) => Poll::Ready(Err(Error::Cancel)), + None if aliases.ended => Poll::Ready(Err(Error::Cancel)), None => Poll::Pending, }); if let Poll::Ready(result) = resolved { @@ -838,6 +841,29 @@ where self.state.lock().close(); } + /// Settle waits for control state that cannot arrive on the closed transport, + /// retaining known aliases and tracks until their buffered data is consumed. + pub fn settle_control(&self) { + let state = self.state.lock(); + if let Ok(mut aliases) = state.aliases.write() { + aliases.ended = true; + } + for track in state.subscribes.values() { + if let Ok(mut fill) = track.fill.write() + && matches!(*fill, Fill::Requested | Fill::Serving(_)) + { + *fill = Fill::Done; + } + } + for fetch in state.group_fetches.values() { + if let Ok(mut fetch) = fetch.write() + && matches!(*fetch, GroupFetch::Pending) + { + *fetch = GroupFetch::Done; + } + } + } + /// Leave `alias` in the state a cancelled subscription leaves behind: bound to a /// subscription, then retired. /// @@ -12414,6 +12440,94 @@ mod stitch_tests { assert_eq!(frames.len(), 2, "the head is published as the prefix it is"); } + #[moq_net_sim::test] + async fn receive_drain_keeps_bound_aliases_and_ends_unknown_alias_waits() { + let h = Harness::new(Fill::Done, vec![]); + let aliases = h.subscriber.state.lock().aliases.consume(); + let mut unknown = Box::pin(resolve_track_alias(&h.subscriber.runtime, aliases.clone(), ALIAS + 1)); + assert!(futures::poll!(unknown.as_mut()).is_pending()); + h.subscriber.settle_control(); + assert!(matches!( + futures::FutureExt::now_or_never(unknown), + Some(Err(Error::Cancel)) + )); + assert!(matches!(resolve_track_alias(&h.subscriber.runtime, aliases, ALIAS).await, Ok(id) if id == REQUEST)); + } + + #[moq_net_sim::test] + async fn receive_drain_ends_a_fill_waiting_for_subscribe_ok() { + let h = Harness::new(Fill::Requested, vec![fill_stream(SEQUENCE, &[b"head"])]); + let mut stream = h.stream().await; + let mut subscriber = h.subscriber.clone(); + let mut receiving = Box::pin(subscriber.recv_fill(&mut stream)); + assert!(futures::poll!(receiving.as_mut()).is_pending()); + h.subscriber.settle_control(); + assert!(matches!( + futures::FutureExt::now_or_never(receiving), + Some(Err(Error::Unsupported)) + )); + } + + #[moq_net_sim::test] + async fn receive_drain_ends_a_tail_waiting_for_a_missing_fill() { + let h = Harness::new( + Fill::Serving(Some(Timescale::MICRO)), + vec![tail_stream(SEQUENCE, 1, &[b"tail"])], + ); + let mut stream = h.stream().await; + let mut subscriber = h.subscriber.clone(); + let mut receiving = Box::pin(subscriber.recv_group(&mut stream)); + assert!(futures::poll!(receiving.as_mut()).is_pending()); + h.subscriber.settle_control(); + assert!(futures::FutureExt::now_or_never(receiving).is_some()); + } + + #[moq_net_sim::test] + async fn receive_drain_preserves_a_buffered_fill_and_tail() { + let h = Harness::new( + Fill::Serving(Some(Timescale::MICRO)), + vec![tail_stream(SEQUENCE, 1, &[b"tail"]), fill_stream(SEQUENCE, &[b"head"])], + ); + let mut consumer = h.track.subscribe(None); + let mut tail = h.stream().await; + let mut head = h.stream().await; + let mut tail_subscriber = h.subscriber.clone(); + let mut head_subscriber = h.subscriber.clone(); + let mut receiving = Box::pin(futures::future::join( + tail_subscriber.recv_group(&mut tail), + head_subscriber.recv_fill(&mut head), + )); + assert!(futures::poll!(receiving.as_mut()).is_pending()); + h.subscriber.settle_control(); + let (tail, head) = receiving.await; + tail.expect("buffered tail"); + head.expect("buffered head"); + let (_, frames) = read_group(&mut consumer).await; + assert_eq!( + frames.iter().map(|(_, bytes)| bytes.as_slice()).collect::>(), + vec![b"head".as_slice(), b"tail".as_slice()] + ); + } + + #[moq_net_sim::test] + async fn receive_drain_ends_a_group_fetch_waiting_for_fetch_ok() { + let h = Harness::new(Fill::Done, vec![fill_stream(SEQUENCE, &[b"head"])]); + h.subscriber + .state + .lock() + .group_fetches + .insert(REQUEST, kio::Producer::new(GroupFetch::Pending)); + let mut stream = h.stream().await; + let mut subscriber = h.subscriber.clone(); + let mut receiving = Box::pin(subscriber.recv_fill(&mut stream)); + assert!(futures::poll!(receiving.as_mut()).is_pending()); + h.subscriber.settle_control(); + assert!(matches!( + futures::FutureExt::now_or_never(receiving), + Some(Err(Error::Unsupported)) + )); + } + /// A fetch stream can arrive before SUBSCRIBE_OK commits the pending track. It waits /// for that response instead of looking up a producer that does not exist yet. #[moq_net_sim::test] diff --git a/rs/moq-net/src/lite/session.rs b/rs/moq-net/src/lite/session.rs index 4395977ff34..b241bbd0e33 100644 --- a/rs/moq-net/src/lite/session.rs +++ b/rs/moq-net/src/lite/session.rs @@ -343,6 +343,7 @@ where .then(|| SendSetup::new(session.clone(), our_setup, version)), goaway: Some(SendGoaway::new(runtime, session.clone(), goaway, version)), session_stream: setup_stream, + ending: None, publisher, subscriber: SubscriberDriver::new(subscriber, early), session, @@ -371,6 +372,8 @@ pub(crate) struct Driver { /// The legacy session stream (pre-lite-03). Only its *error* ends the race, so /// the publisher and subscriber keep running while it sits idle. session_stream: Option>, + /// The first terminal outcome, retained while received streams drain. + ending: Option>, publisher: Publisher, subscriber: SubscriberDriver, /// For the terminal close: the machine's last act reports the outcome to @@ -383,33 +386,44 @@ where S: crate::transport::poll::Session, { pub(crate) fn poll(&mut self, waiter: &kio::Waiter) -> Poll> { - let res = std::task::ready!(self.poll_protocol(waiter)); - self.auth.handle.close(match &res { - Ok(()) => Error::Cancel, - Err(err) => err.clone(), - }); - if self.local_close.load(std::sync::atomic::Ordering::Relaxed) { - self.subscriber.close(); - } else if let Err(err) = &res { - // Every track this session was receiving ends with its error. - self.subscriber.abort(err); - } - match &res { - Err(Error::Transport(_)) => { - tracing::info!("session terminated"); - self.session.close(SessionError::Internal.to_code(), ""); - } - Err(err) => { - tracing::warn!(%err, "session error"); - self.session.close( - crate::lite::error::to_session_code(&SessionError::from(err)), - err.to_string().as_ref(), - ); + if self.ending.is_none() { + let res = std::task::ready!(self.poll_protocol(waiter)); + self.auth.handle.close(match &res { + Ok(()) => Error::Cancel, + Err(err) => err.clone(), + }); + match &res { + Err(Error::Transport(_)) => { + tracing::info!("session terminated"); + self.session.close(SessionError::Internal.to_code(), ""); + } + Err(err) => { + tracing::warn!(%err, "session error"); + self.session.close( + crate::lite::error::to_session_code(&SessionError::from(err)), + err.to_string().as_ref(), + ); + } + _ => { + tracing::info!("session closed"); + self.session.close(SessionError::Cancel.to_code(), ""); + } } - _ => { - tracing::info!("session closed"); - self.session.close(SessionError::Cancel.to_code(), ""); + if self.local_close.load(std::sync::atomic::Ordering::Relaxed) { + self.subscriber.close(); + return Poll::Ready(res); } + self.ending = Some(res); + } + // Closing the transport first bounds every read, including when a control + // stream ended the protocol while the connection itself was still live. + // Keep control machines and their subscription entries until data drains. + std::task::ready!(self.subscriber.poll_drain(waiter)); + let res = self.ending.take().expect("draining a terminal session"); + if let Err(err) = &res { + self.subscriber.abort(err); + } else { + self.subscriber.close(); } Poll::Ready(res) } @@ -903,6 +917,13 @@ mod tests { } } + /// A protocol error on a live connection must end partial receive reads before + /// waiting for them, otherwise receive draining could wait for new peer data. + #[moq_net_sim::test] + async fn malformed_auth_closes_before_draining_a_partial_group_header() { + assert_session_closes(ScriptedSession::per_stream(vec![vec![99]]).with_incoming_unis(vec![vec![0]])).await; + } + async fn malformed_auth_reply(script: Vec) { // The setup token's AUTH stream is the first one the session opens. let transport = ScriptedSession::per_stream(vec![script]); diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 1ad386c4fc5..d0bcc999439 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -620,6 +620,18 @@ impl SubscriberDriver { self.cleanup.close(); } + /// Read only the closed transport's buffered uni streams. Control machines + /// stay owned, so their cancellation guards cannot remove subscription state. + pub fn poll_drain(&mut self, waiter: &kio::Waiter) -> Poll<()> { + let closed = self.uni.poll(waiter).is_ready(); + let drained = self.uni.children.poll(waiter).is_ready(); + if closed && drained { + Poll::Ready(()) + } else { + Poll::Pending + } + } + pub fn poll(&mut self, waiter: &kio::Waiter) -> Poll> { let mut i = 0; while i < self.prefixes.len() { diff --git a/rs/moq-net/src/lite/test_transport.rs b/rs/moq-net/src/lite/test_transport.rs index f47528eec5d..c0051c9a916 100644 --- a/rs/moq-net/src/lite/test_transport.rs +++ b/rs/moq-net/src/lite/test_transport.rs @@ -48,6 +48,7 @@ pub struct Log { streams: Arc, stops: Arc>>, closes: Arc>>, + closed: kio::Shared, bi_opens: Arc, priorities: Arc>>, } @@ -61,6 +62,20 @@ pub enum Sent { } impl Log { + fn close(&self, code: u32, reason: &str) { + self.closes.lock().unwrap().push((code, reason.to_owned())); + *self.closed.lock() = true; + } + + fn poll_closed(&self, waiter: &kio::Waiter) -> Poll { + self.closed + .poll(waiter, |closed| match **closed { + true => Poll::Ready(()), + false => Poll::Pending, + }) + .map(|_| SinkError) + } + /// What every send stream did, in call order, tagged with the stream's id (its /// creation order). Unlike [`Self::writes`] and [`Self::resets`], this keeps the /// order across streams, so a test can check one stream's message against @@ -276,6 +291,7 @@ impl poll::RecvStream for DeadRecv { /// peer with nothing more to say. Sends record to [`Log`] like [`SinkSession`]. #[derive(Clone)] pub struct DeadStreamSession { + park: kio::Park, /// What the session's send streams recorded, for test assertions. pub log: Log, unis: Arc>, @@ -287,6 +303,7 @@ impl DeadStreamSession { /// `count` uni streams that die before their first byte, then silence. pub fn unis(count: usize) -> Self { Self { + park: Default::default(), log: Log::default(), unis: Arc::new(Mutex::new(count)), bis: Arc::new(Mutex::new(0)), @@ -297,6 +314,7 @@ impl DeadStreamSession { /// `count` bidi streams that die before their first byte, then silence. pub fn bis(count: usize) -> Self { Self { + park: Default::default(), log: Log::default(), unis: Arc::new(Mutex::new(0)), bis: Arc::new(Mutex::new(count)), @@ -328,17 +346,17 @@ impl poll::Session for DeadStreamSession { type RecvStream = DeadRecv; type Error = SinkError; - fn poll_accept_uni(&mut self, _cx: &mut Context<'_>) -> Poll> { + fn poll_accept_uni(&mut self, cx: &mut Context<'_>) -> Poll> { match Self::take(&self.unis) { true => Poll::Ready(Ok(DeadRecv(self.reset))), - false => Poll::Pending, + false => self.log.poll_closed(self.park.hold(cx)).map(Err), } } - fn poll_accept_bi(&mut self, _cx: &mut Context<'_>) -> Poll, Self::Error>> { + fn poll_accept_bi(&mut self, cx: &mut Context<'_>) -> Poll, Self::Error>> { match Self::take(&self.bis) { true => Poll::Ready(Ok((SinkSend::new(self.log.clone()), DeadRecv(self.reset)))), - false => Poll::Pending, + false => self.log.poll_closed(self.park.hold(cx)).map(Err), } } @@ -371,11 +389,11 @@ impl poll::Session for DeadStreamSession { } fn close(&mut self, code: u32, reason: &str) { - self.log.closes.lock().unwrap().push((code, reason.to_owned())); + self.log.close(code, reason); } - fn poll_closed(&mut self, _cx: &mut Context<'_>) -> Poll { - Poll::Pending + fn poll_closed(&mut self, cx: &mut Context<'_>) -> Poll { + self.log.poll_closed(self.park.hold(cx)) } fn stats(&self) -> impl crate::transport::Stats { @@ -386,6 +404,7 @@ impl poll::Session for DeadStreamSession { /// Send streams record their bytes while peer reads park forever. #[derive(Clone, Default)] pub struct SinkSession { + park: kio::Park, bi_prefix: usize, pub log: Log, /// Set by [`Self::gated_bi`]. `None` parks `open_bi` itself forever, which is all @@ -499,8 +518,8 @@ impl poll::Session for SinkSession { type RecvStream = PendingRecv; type Error = SinkError; - fn poll_accept_uni(&mut self, _cx: &mut Context<'_>) -> Poll> { - Poll::Pending + fn poll_accept_uni(&mut self, cx: &mut Context<'_>) -> Poll> { + self.log.poll_closed(self.park.hold(cx)).map(Err) } fn poll_accept_bi(&mut self, _cx: &mut Context<'_>) -> Poll, Self::Error>> { @@ -559,11 +578,11 @@ impl poll::Session for SinkSession { } fn close(&mut self, code: u32, reason: &str) { - self.log.closes.lock().unwrap().push((code, reason.to_owned())); + self.log.close(code, reason); } - fn poll_closed(&mut self, _cx: &mut Context<'_>) -> Poll { - Poll::Pending + fn poll_closed(&mut self, cx: &mut Context<'_>) -> Poll { + self.log.poll_closed(self.park.hold(cx)) } fn stats(&self) -> impl crate::transport::Stats { @@ -609,6 +628,7 @@ impl crate::transport::Stats for SinkStats { /// Parks once the script is exhausted rather than reporting EOF: a read loop that saw /// EOF would exit, and a test usually wants to assert against the loop still running. pub struct ScriptedRecv { + park: kio::Park, script: Arc>>, /// How the peer's send side ends once the script runs out; `None` parks. close: Arc>>, @@ -625,7 +645,7 @@ pub enum Close { impl poll::RecvStream for ScriptedRecv { type Error = SinkError; - fn poll_read(&mut self, _cx: &mut Context<'_>, dst: &mut [u8]) -> Poll, Self::Error>> { + fn poll_read(&mut self, cx: &mut Context<'_>, dst: &mut [u8]) -> Poll, Self::Error>> { let take = { let mut script = self.script.lock().unwrap(); if script.is_empty() { @@ -642,7 +662,7 @@ impl poll::RecvStream for ScriptedRecv { 0 => match *self.close.lock().unwrap() { Some(Close::Fin) => Poll::Ready(Ok(None)), Some(Close::Reset) => Poll::Ready(Err(SinkError)), - None => Poll::Pending, + None => self.log.poll_closed(self.park.hold(cx)).map(Err), }, take => Poll::Ready(Ok(Some(take))), } @@ -801,24 +821,26 @@ impl poll::Session for ScriptedSession { type RecvStream = ScriptedRecv; type Error = SinkError; - fn poll_accept_uni(&mut self, _cx: &mut Context<'_>) -> Poll> { + fn poll_accept_uni(&mut self, cx: &mut Context<'_>) -> Poll> { let Some(script) = self.incoming_unis.lock().unwrap().pop_front() else { - return Poll::Pending; + return self.log.poll_closed(self.park.hold(cx)).map(Err); }; Poll::Ready(Ok(ScriptedRecv { + park: Default::default(), script: Arc::new(Mutex::new(script)), close: self.close.clone(), log: self.log.clone(), })) } - fn poll_accept_bi(&mut self, _cx: &mut Context<'_>) -> Poll, Self::Error>> { + fn poll_accept_bi(&mut self, cx: &mut Context<'_>) -> Poll, Self::Error>> { let Some(script) = self.incoming_bidis.lock().unwrap().pop_front() else { - return Poll::Pending; + return self.log.poll_closed(self.park.hold(cx)).map(Err); }; Poll::Ready(Ok(( SinkSend::new(self.log.clone()), ScriptedRecv { + park: Default::default(), script: Arc::new(Mutex::new(script)), close: self.close.clone(), log: self.log.clone(), @@ -849,6 +871,7 @@ impl poll::Session for ScriptedSession { Poll::Ready(Ok(( send, ScriptedRecv { + park: Default::default(), script, close: self.close.clone(), log: self.log.clone(), @@ -885,14 +908,41 @@ impl poll::Session for ScriptedSession { } fn close(&mut self, code: u32, reason: &str) { - self.log.closes.lock().unwrap().push((code, reason.to_owned())); + self.log.close(code, reason); } - fn poll_closed(&mut self, _cx: &mut Context<'_>) -> Poll { - Poll::Pending + fn poll_closed(&mut self, cx: &mut Context<'_>) -> Poll { + self.log.poll_closed(self.park.hold(cx)) } fn stats(&self) -> impl crate::transport::Stats { self.stats } } + +#[test] +fn a_local_close_preserves_scripted_uni_bytes_and_ends_accepts() { + use poll::{RecvStream as _, Session as _}; + for fin in [false, true] { + let mut session = ScriptedSession::new(Vec::new()).with_incoming_unis(vec![b"last".to_vec()]); + if fin { + session.close(Close::Fin); + } + poll::Session::close(&mut session, 0, "done"); + let mut cx = Context::from_waker(std::task::Waker::noop()); + assert!(session.poll_closed(&mut cx).is_ready()); + let Poll::Ready(Ok(mut recv)) = session.poll_accept_uni(&mut cx) else { + panic!("buffered uni lost") + }; + let mut bytes = [0; 4]; + assert!(matches!(recv.poll_read(&mut cx, &mut bytes), Poll::Ready(Ok(Some(4))))); + assert_eq!(&bytes, b"last"); + let end = recv.poll_read(&mut cx, &mut bytes); + assert!(if fin { + matches!(end, Poll::Ready(Ok(None))) + } else { + matches!(end, Poll::Ready(Err(_))) + }); + assert!(matches!(session.poll_accept_uni(&mut cx), Poll::Ready(Err(_)))); + } +} diff --git a/rs/moq-net/src/session.rs b/rs/moq-net/src/session.rs index 4a825756d50..70200efaac2 100644 --- a/rs/moq-net/src/session.rs +++ b/rs/moq-net/src/session.rs @@ -224,6 +224,9 @@ impl Session { /// Block until the transport session is closed, returning the reason. /// + /// On a peer close, the driver first drains group streams already delivered to + /// the transport, including streams the subscriber has not yet accepted. + /// /// A close code the peer sent is decoded through the session registry (so an auth /// rejection arrives as `Error::Session(SessionError::Unauthorized)`); every peer code is /// preserved as [`Error::Session`], and a close carrying no application code surfaces as diff --git a/rs/moq-net/src/transport/memory.rs b/rs/moq-net/src/transport/memory.rs index 1bccc9948bb..98ca019a83c 100644 --- a/rs/moq-net/src/transport/memory.rs +++ b/rs/moq-net/src/transport/memory.rs @@ -367,10 +367,10 @@ impl Channel { self.queued = 0; } - /// How the writer sees the channel close: reset, stopped, or finished and drained. + /// How the writer sees the channel close: reset, stopped, or FIN delivered. fn sent(&self) -> Poll> { match self.end { - Some(End::Fin) if self.messages.is_empty() => Poll::Ready(Ok(())), + Some(End::Fin) => Poll::Ready(Ok(())), Some(End::Reset(_)) => Poll::Ready(Ok(())), Some(End::Stop(code)) => Poll::Ready(Err(Error::Stopped(code))), _ => Poll::Pending, @@ -460,9 +460,9 @@ impl poll::SendStream for SendStream { } } - /// Ready once this side reset the stream, the reader stopped it, or the reader took - /// every message of a finished stream. That stands in for a FIN acknowledgement, so a - /// writer that waits for it before closing the session knows the peer has the data. + /// Ready once this side reset the stream, the reader stopped it, or FIN was delivered + /// into the channel. Like a QUIC acknowledgement, delivery does not wait for the + /// receiving application to read the buffered data. fn poll_closed(&mut self, cx: &mut Context<'_>) -> Poll> { let waiter = self.park.hold(cx); // Check the close first, so a drain that happened before it is still seen below. @@ -735,7 +735,7 @@ mod tests { } #[test] - fn finish_closes_once_drained() { + fn finish_acknowledges_delivery_before_the_reader_drains() { let (mut a, mut b) = pair(Config::default()); let (mut send, mut recv) = uni(&mut a, &mut b); @@ -744,8 +744,8 @@ mod tests { assert!(matches!(send.finish(), Err(Error::Closed))); assert!(ready(recv.poll_closed(&mut noop())).is_ok()); - // The writer's close waits for the reader to take every message. - assert!(send.poll_closed(&mut noop()).is_pending()); + // FIN acknowledges delivery, while the payload remains buffered for the reader. + assert!(ready(send.poll_closed(&mut noop())).is_ok()); assert_eq!(text(read(&mut recv, usize::MAX)), "done"); assert!(ready(send.poll_closed(&mut noop())).is_ok()); assert_eq!(ready(read(&mut recv, usize::MAX)).unwrap(), None); diff --git a/rs/moq-net/tests/session_close.rs b/rs/moq-net/tests/session_close.rs index 92f2c79cf73..ac8a58f4d4a 100644 --- a/rs/moq-net/tests/session_close.rs +++ b/rs/moq-net/tests/session_close.rs @@ -37,6 +37,10 @@ struct Setup { } async fn setup(version: &str) -> Setup { + setup_with_gate(version, None).await +} + +async fn setup_with_gate(version: &str, gate: Option>) -> Setup { let publisher = produce_origin(1); let broadcast = publisher.create_broadcast("bcast").unwrap(); let track = broadcast.create_track("video", timed()).unwrap(); @@ -46,6 +50,7 @@ async fn setup(version: &str) -> Setup { let mut options = MockConnectOptions::new(version.parse::().unwrap()); options.client_publish = Some(publisher.consume()); options.server_subscribe = Some(subscriber.clone()); + options.server_gate = gate; let pair = connect_mock(options).await; let consumer = subscriber.consume(); @@ -304,19 +309,66 @@ async fn close_requires_subscriber_fin() { #[moq_net_sim::test] async fn older_lite_close_keeps_ack_drain() { for version in ["moq-lite-05", "moq-lite-06"] { + peer_close_reads_buffered_tail(version).await; + } +} + +#[moq_net_sim::test] +async fn earlier_lite_peer_close_reads_buffered_tail() { + for version in ["moq-lite-01", "moq-lite-02", "moq-lite-03", "moq-lite-04"] { + peer_close_reads_buffered_tail(version).await; + } +} + +#[moq_net_sim::test] +async fn ietf_peer_close_reads_buffered_tail() { + for version in ["moq-transport-14", "moq-transport-17", "moq-transport-22"] { + peer_close_reads_buffered_tail(version).await; + } +} + +async fn peer_close_reads_buffered_tail(version: &str) { + let Setup { + pair, track, reader, .. + } = setup(version).await; + // Delivery to the transport acknowledges the FIN before the receiving + // protocol driver has consumed the buffered stream. + pair.client_transport.ack_fins(); + let mut group = track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, b"last".as_slice()).unwrap(); + group.finish().unwrap(); + track.finish().unwrap(); + pair.client.close().await.expect("released draft still drains on ACKs"); + let (got, _) = reader.await.unwrap(); + assert_eq!(got, vec![b"last".to_vec()], "{version}"); +} + +/// Legacy session/control streams and the accept error race the buffered group. +/// Holding the whole receiver ensures the close is visible on its first poll. +#[moq_net_sim::test] +async fn peer_close_before_receiver_poll_keeps_buffered_tail() { + for version in [ + "moq-lite-01", + "moq-lite-02", + "moq-lite-05", + "moq-lite-06", + "moq-transport-14", + "moq-transport-17", + ] { + let gate = kio::Producer::new(true); let Setup { - pair, - track, - reader: _reader, - .. - } = setup(version).await; + pair, track, reader, .. + } = setup_with_gate(version, Some(gate.consume())).await; + *gate.write().ok().expect("receiver gate remains open") = false; pair.client_transport.ack_fins(); - pair.client_transport.hold_unis(); let mut group = track.append_group().unwrap(); group.write_frame(Timestamp::ZERO, b"last".as_slice()).unwrap(); group.finish().unwrap(); track.finish().unwrap(); - pair.client.close().await.expect("released draft still drains on ACKs"); + pair.client.close().await.expect("FIN delivery needs no receiver poll"); + *gate.write().ok().expect("receiver gate remains open") = true; + let (got, _) = reader.await.unwrap(); + assert_eq!(got, vec![b"last".to_vec()], "{version}"); } } diff --git a/rs/moq-net/tests/support/harness.rs b/rs/moq-net/tests/support/harness.rs index 0bcd0109e74..286e036b5cf 100644 --- a/rs/moq-net/tests/support/harness.rs +++ b/rs/moq-net/tests/support/harness.rs @@ -11,6 +11,7 @@ use std::sync::{ Arc, atomic::{AtomicUsize, Ordering}, }; +use std::task::Poll; use moq_net::{Client, Server, Session, Version, origin}; @@ -35,18 +36,28 @@ pub fn spawn(mut driver: D) { } } -struct Counted { +struct Controlled { driver: D, - polls: Arc, + polls: Option>, + gate: Option>, } -impl moq_net::time::Driver for Counted { +impl moq_net::time::Driver for Controlled { fn poll( &mut self, now: moq_net::time::Instant, waiter: &kio::Waiter, ) -> Result, moq_net::Error> { - self.polls.fetch_add(1, Ordering::Relaxed); + if let Some(gate) = &self.gate + && gate + .poll(waiter, |open| if **open { Poll::Ready(()) } else { Poll::Pending }) + .is_pending() + { + return Ok(None); + } + if let Some(polls) = &self.polls { + polls.fetch_add(1, Ordering::Relaxed); + } self.driver.poll(now, waiter) } } @@ -86,6 +97,8 @@ pub struct MockConnectOptions { pub stats: MockStats, /// Count server driver polls into this counter when supplied. pub server_polls: Option>, + /// Pause the receiving driver while its transport keeps delivering data. + pub server_gate: Option>, } impl MockConnectOptions { @@ -105,6 +118,7 @@ impl MockConnectOptions { cost: None, stats: MockStats::default(), server_polls: None, + server_gate: None, } } } @@ -182,10 +196,11 @@ pub async fn connect_mock(opts: MockConnectOptions) -> MockPair { .accept(now(), server_transport) .await .expect("server handshake failed"); - match opts.server_polls { - Some(polls) => spawn(Counted { driver, polls }), - None => spawn(driver), - } + spawn(Controlled { + driver, + polls: opts.server_polls, + gate: opts.server_gate, + }); session }; let (client_session, server_session) = futures::join!(client_fut, server_fut); diff --git a/rs/moq-tokio/src/quic/web/session.rs b/rs/moq-tokio/src/quic/web/session.rs index 38e01d66847..d1a317ab063 100644 --- a/rs/moq-tokio/src/quic/web/session.rs +++ b/rs/moq-tokio/src/quic/web/session.rs @@ -700,6 +700,8 @@ pub struct SessionAccept { qpack_decoder: Option, accept_uni: Pin>, + // Pending WebTransport headers must be decoded before returning this error. + uni_end: Option, accept_bi: Pin>, // Keep track of work being done to read/write the WebTransport stream header. @@ -745,6 +747,7 @@ impl SessionAccept { qpack_encoder: None, accept_uni, + uni_end: None, accept_bi, pending_uni: FuturesUnordered::new(), @@ -782,18 +785,17 @@ impl SessionAccept { loop { // Accept any new streams. - if let Poll::Ready(Some(res)) = self.accept_uni.poll_next_unpin(cx) { + if self.uni_end.is_none() + && let Poll::Ready(Some(res)) = self.accept_uni.poll_next_unpin(cx) + { // Start decoding the header and add the future to the list of pending streams. - let recv = match res { - Ok(recv) => recv, - Err(e) => { - return Poll::Ready(Err(e.into())); + match res { + Ok(recv) => { + self.pending_uni.push(Box::pin(Self::decode_uni(recv, self.session_id))); + continue; } - }; - let pending = Self::decode_uni(recv, self.session_id); - self.pending_uni.push(Box::pin(pending)); - - continue; + Err(err) => self.uni_end = Some(err.into()), + } } // Poll the list of pending streams. @@ -803,7 +805,8 @@ impl SessionAccept { log_header_error(err, "unidirectional"); continue; } - Poll::Ready(None) | Poll::Pending => return Poll::Pending, + Poll::Ready(None) => return self.uni_end.clone().map_or(Poll::Pending, |err| Poll::Ready(Err(err))), + Poll::Pending => return Poll::Pending, }; // Decide if we keep looping based on the type. @@ -1266,6 +1269,26 @@ mod tests { ); } + #[tokio::test] + async fn webtransport_accepts_a_buffered_uni_after_the_peer_close() { + let (client, server, _endpoints) = pair().await; + let mut send = client.open_uni().await.unwrap(); + // WebTransport stream type, session ID 0, then the application payload. + send.write_all(&[0x40, 0x54, 0x00, b'l', b'a', b's', b't']) + .await + .unwrap(); + send.finish().unwrap(); + assert!(send.stopped().await.unwrap().is_none()); + client.close(0u32.into(), b"done"); + server.closed().await; + let mut accept = SessionAccept::new(server, SESSION, Arc::new(CloseReason::new(false))); + let mut recv = kio::wait(|waiter| accept.poll_accept_uni(waiter)) + .await + .expect("buffered uni precedes the close"); + assert_eq!(recv.read_to_end(64).await.unwrap(), b"last"); + assert!(kio::wait(|waiter| accept.poll_accept_uni(waiter)).await.is_err()); + } + /// A uni stream reset before its header arrived reports the reset, not a foreign session. #[tokio::test] async fn uni_reset_before_header() { diff --git a/rs/moq-uring/src/quic/web.rs b/rs/moq-uring/src/quic/web.rs index 192592f9b21..40284abe900 100644 --- a/rs/moq-uring/src/quic/web.rs +++ b/rs/moq-uring/src/quic/web.rs @@ -562,15 +562,22 @@ impl web_transport_trait::poll::Session for Session { // Adopt every new arrival, then try to classify the pending set. // Each mid-header stream parks the caller in its own stream's // waiters, so progress on any of them re-polls us. - loop { + let ended = loop { match web_transport_trait::poll::Session::poll_accept_uni(&mut self.conn, cx) { Poll::Ready(Ok(recv)) => web.state.borrow_mut().pending_uni.push(PendingUni::new(recv)), - Poll::Ready(Err(err)) => return Poll::Ready(Err(unmap_err(err))), - Poll::Pending => break, + Poll::Ready(Err(err)) => break Some(err), + Poll::Pending => break None, } - } + }; if !classify_uni(&web, cx) { + // Buffered headers can finish after the connection error. Hand those + // streams out before reporting that no more can arrive. + if web.state.borrow().pending_uni.is_empty() + && let Some(err) = ended + { + return Poll::Ready(Err(unmap_err(err))); + } return Poll::Pending; } } diff --git a/rs/moq-uring/tests/web.rs b/rs/moq-uring/tests/web.rs index 2fbf2157ba4..f8a159d69bb 100644 --- a/rs/moq-uring/tests/web.rs +++ b/rs/moq-uring/tests/web.rs @@ -208,6 +208,41 @@ fn webtransport_echo_end_to_end() { client.join().expect("client thread"); } +#[test] +fn webtransport_accepts_a_buffered_uni_after_the_peer_close() { + let Some(mut worker) = worker() else { return }; + let handle = worker.handle(); + let certs = support::certs().expect("certificates"); + let endpoint = h3_endpoint(&handle, &certs); + let client = tokio_client(format!("https://{}/tail", endpoint.local_addr()), |session| { + Box::pin(async move { + let mut send = session.open_uni().await.expect("open uni"); + send.write_all(PAYLOAD).await.expect("write"); + send.finish().expect("finish"); + assert!(send.stopped().await.expect("FIN delivered").is_none()); + session.close(0, b"done"); + session.closed().await; + }) + }); + worker + .block_on(async move { + let conn = endpoint.accept().await.expect("accept"); + let request = quic::web::Request::accept(conn).await.expect("handshake"); + let mut session = request + .respond(quic::web::Response::default().with_protocol(PROTO)) + .await + .expect("respond"); + std::future::poll_fn(|cx| session.poll_closed(cx)).await; + let mut recv = std::future::poll_fn(|cx| session.poll_accept_uni(cx)) + .await + .expect("buffered uni precedes the close"); + assert_eq!(drain(&mut recv).await, PAYLOAD); + assert!(std::future::poll_fn(|cx| session.poll_accept_uni(cx)).await.is_err()); + }) + .expect("worker"); + client.join().expect("client thread"); +} + /// A whole moq-lite session over WebTransport: subprotocol negotiation, SETUP /// on the bidirectional stream, announce, subscribe, and a group on a /// unidirectional stream, with the tokio stack as the subscriber. This is the From 7f825acf7d5f80654c5d4af583536315f7f7bad4 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 12:16:11 -0700 Subject: [PATCH 3/9] fix(net): keep delivery acknowledgements confined to close regressions The optional production memory-transport acknowledgement change regressed the high-stream-count benchmark, so retain its existing semantics and use the test transport delivery-ACK knob for receiver-close coverage. Co-Authored-By: GPT-6 --- rs/moq-net/src/transport/memory.rs | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/rs/moq-net/src/transport/memory.rs b/rs/moq-net/src/transport/memory.rs index 98ca019a83c..1bccc9948bb 100644 --- a/rs/moq-net/src/transport/memory.rs +++ b/rs/moq-net/src/transport/memory.rs @@ -367,10 +367,10 @@ impl Channel { self.queued = 0; } - /// How the writer sees the channel close: reset, stopped, or FIN delivered. + /// How the writer sees the channel close: reset, stopped, or finished and drained. fn sent(&self) -> Poll> { match self.end { - Some(End::Fin) => Poll::Ready(Ok(())), + Some(End::Fin) if self.messages.is_empty() => Poll::Ready(Ok(())), Some(End::Reset(_)) => Poll::Ready(Ok(())), Some(End::Stop(code)) => Poll::Ready(Err(Error::Stopped(code))), _ => Poll::Pending, @@ -460,9 +460,9 @@ impl poll::SendStream for SendStream { } } - /// Ready once this side reset the stream, the reader stopped it, or FIN was delivered - /// into the channel. Like a QUIC acknowledgement, delivery does not wait for the - /// receiving application to read the buffered data. + /// Ready once this side reset the stream, the reader stopped it, or the reader took + /// every message of a finished stream. That stands in for a FIN acknowledgement, so a + /// writer that waits for it before closing the session knows the peer has the data. fn poll_closed(&mut self, cx: &mut Context<'_>) -> Poll> { let waiter = self.park.hold(cx); // Check the close first, so a drain that happened before it is still seen below. @@ -735,7 +735,7 @@ mod tests { } #[test] - fn finish_acknowledges_delivery_before_the_reader_drains() { + fn finish_closes_once_drained() { let (mut a, mut b) = pair(Config::default()); let (mut send, mut recv) = uni(&mut a, &mut b); @@ -744,8 +744,8 @@ mod tests { assert!(matches!(send.finish(), Err(Error::Closed))); assert!(ready(recv.poll_closed(&mut noop())).is_ok()); - // FIN acknowledges delivery, while the payload remains buffered for the reader. - assert!(ready(send.poll_closed(&mut noop())).is_ok()); + // The writer's close waits for the reader to take every message. + assert!(send.poll_closed(&mut noop()).is_pending()); assert_eq!(text(read(&mut recv, usize::MAX)), "done"); assert!(ready(send.poll_closed(&mut noop())).is_ok()); assert_eq!(ready(read(&mut recv, usize::MAX)).unwrap(), None); From 57313c41fcaaf8a05fdbc7d10f3c528576da17fe Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 16 Oct 2026 12:27:22 -0700 Subject: [PATCH 4/9] docs(net): retain close-tail quest while qmux teardown blocks draining The existing TCP publisher restart regression passes on main but hangs with receive draining. A direct qmux 0.6.3 reproduction confirms unfinished stream reads remain pending after session and accept-queue closure while Session handles retain the shared stream table. Record the upstream dependency blocker and clarify that Session::closed does not finish receive draining. Co-Authored-By: GPT-6 --- quest/m1/README.md | 1 + quest/m1/close-tail-drain.md | 92 ++++++++++++++++++++++++++++++++++++ rs/moq-net/src/session.rs | 4 +- 3 files changed, 95 insertions(+), 2 deletions(-) create mode 100644 quest/m1/close-tail-drain.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 13caa8e951b..8f90fa49061 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -142,6 +142,7 @@ above them. - [JS pending tail](/quest/m1/js-pending-tail.md) - a JS reader holds for a track's pending tail like Rust - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS - [Plan: HLS export failure signal](/quest/m1/hls-failure-signal.md) - decide what players see when the exporter's timeline fails, live or history, so a truncated recording never looks complete +- [Close tail drain](/quest/m1/close-tail-drain.md) - after a peer close, subscribers read the group streams their transport already delivered instead of aborting them, so a finished tail arrives on every version - [Publisher linger](/quest/m1/publisher-linger.md) - every finishing publisher keeps its broadcast up for the media max-age so subscribers can drain it - [Vulkan Video encode on AMD](/quest/m1/vulkan-encode.md) - H.264 and H.265 from an external Vulkan image on RADV, hand-rolled on ash, proven on an RX 9070 - [VA-API encodes an external Vulkan image](/quest/m1/vaapi-vulkan-import.md) - an explicit-sync DMA-BUF reaches Intel's encoder, proven on Arrow Lake diff --git a/quest/m1/close-tail-drain.md b/quest/m1/close-tail-drain.md new file mode 100644 index 00000000000..b6da3726fd6 --- /dev/null +++ b/quest/m1/close-tail-drain.md @@ -0,0 +1,92 @@ +# [M] Subscribers drain the tail after a peer close + +## Goal + +A group the publisher finished before a graceful session close reaches the +subscriber on every version (lite-01 through lite-07, and IETF): after a peer +close, the subscriber keeps reading the group streams its transport already +delivered instead of aborting them. Reproduce the loss first with a regression +test that fails without the fix. No wire change. + +## Plan + +Found while reworking the in-memory transport (#5530). Facts at 813a92b48: + +- On a peer close, lite `UniAccept::poll` returns the accept error + (`rs/moq-net/src/lite/subscriber.rs:712`), and `Driver::poll` + (`rs/moq-net/src/lite/session.rs:381-411`) aborts every subscription when + `local_close` is false, then drops the accepted `UniServe` children + mid-read. IETF `run_unis` (`rs/moq-net/src/ietf/session.rs:882`, + `:931-942`) does the same, and `ietf::session::start` aborts again after + the race in each of its two arms, drafts 14-16 (`:371-374`) and drafts 17+ + (`:588-591`). +- Real QUIC keeps that data readable: moq-tokio's `quic::driver` `poll_accept` hands out + already-received streams before the connection error, and reads return + buffered chunks and the FIN, reporting `ConnectionLost` only when blocked. + moq-uring's raw QUIC follows the same order, but its WebTransport layer + does not: `poll_accept_uni` (`rs/moq-uring/src/quic/web.rs:566-569`) + returns the connection error before classifying streams already in + `pending_uni`, so a tail stream accepted in the same poll as the close is + lost below the subscriber. +- The closer (`Session::close`, `rs/moq-net/src/session.rs:201`) waits for + its group streams' FIN acks. An ACK only means delivery to the peer's + transport, not that the peer processed it, so on lite-01 through lite-06 + and IETF the receiver throws away a tail its transport already holds. + lite-07 already avoids this at the protocol level: the closer waits for the + subscriber's Subscribe-stream FIN (`drafts/draft-lcurley-moq-lite.md`). +- The test mock (`rs/moq-net/tests/support/mock.rs`) hid it: its FIN counts as + acked only once the reading application stops or drops the stream. + `older_lite_close_keeps_ack_drain` never checks what the reader got. + +Decided (maintainer, 2026-10-14): + +- Fix at the receiver. After a peer close, the lite and IETF subscribers + stop aborting accepted group streams and read each to its FIN or until the + transport reports it closed. Buffered data bounds the drain, so no timer. + Streams not yet accepted are still accepted and drained in the same way. + The closer-side wait stays lite-07 only, since older published versions + can't gain a wire change. + Make the drain a session-level phase, not just a change to `UniAccept` and + `run_unis`: a control-stream or publisher error can end the driver first, + and subscription cleanup can remove a track a data stream still needs. + Keep the subscription state the draining streams depend on, and settle + anything that can't complete after the close (such as IETF children waiting + on control state) so the no-timer argument holds. That argument needs a + closed transport: a non-transport terminal error (such as legacy lite's + session-stream FIN returning `Cancel`, `rs/moq-net/src/lite/session.rs:652-660`) + must not start a drain over a live connection that could hang. Close the + transport first, or abort as today if buffered data does not survive a + local close. +- Tests ack a FIN on delivery, as QUIC does, so the regression for lite-01 + through lite-06 and IETF (one draft from each `start` arm) asserts the + reader got the tail and fails before the fix. Cover both the buffered-tail + case and one where a control or accept error wins the poll order. Give + `older_lite_close_keeps_ack_drain` that reader assertion. +- Once #5530 has landed, switch `moq_net::transport::memory`'s + `SendStream::poll_closed` to resolve on `finish` (delivery into the + channel), dropping its "finished and empty" rule. Do this unless the + memory bench (`rs/moq-net/benches/memory.rs`) shows a regression; if it + does, change only the tests. + +The receiver drain also needs moq-uring WebTransport to classify and hand +out its pending uni streams before reporting the connection error, with a +regression for a tail stream and CONNECTION_CLOSE arriving in the same poll. +Check the other `poll_accept_uni` adapters (such as `rs/moq-ffi` and +moq-tokio's `quic::web` path) for the same ordering. + +Public API: none. Wire: none. + +The receive drain also requires qmux to end unfinished stream reads after the +session closes, once their buffered data is consumed. In qmux 0.6.3, +`SessionState` teardown leaves stream channel senders in the shared `Streams` +map retained by `Session` handles, so such reads can remain pending forever. +Fix that lifecycle in `moq-dev/web-transport` and ship a dependency update +before enabling the drain across qmux transports. Keep the TCP publisher +restart regression in `moq-gst` passing; it checks that the closed session's +source withdraws without waiting on an unfinished media stream. + +## Related + +- [Publisher linger](/quest/m1/publisher-linger.md) - keeps a finished broadcast servable so subscribers can drain it, a layer above this +- [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - groups cut short by a session close +- [QUIC ack hook](/quest/m2/quic-ack-hook.md) - a FIN/ack hook in the QUIC layer diff --git a/rs/moq-net/src/session.rs b/rs/moq-net/src/session.rs index 70200efaac2..e1ccc6a8804 100644 --- a/rs/moq-net/src/session.rs +++ b/rs/moq-net/src/session.rs @@ -224,8 +224,8 @@ impl Session { /// Block until the transport session is closed, returning the reason. /// - /// On a peer close, the driver first drains group streams already delivered to - /// the transport, including streams the subscriber has not yet accepted. + /// The driver keeps draining group streams already delivered to the transport + /// after this resolves, so keep driving it until it completes. /// /// A close code the peer sent is decoded through the session registry (so an auth /// rejection arrives as `Error::Session(SessionError::Unauthorized)`); every peer code is From c34cba5227444df0f0072e4e431c85f4bf9b74bb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 20 Oct 2026 13:23:07 -0700 Subject: [PATCH 5/9] test(net): keep close regressions agreeing with main's close codes A post-close transport error now carries the session's first close code, so a driver that ends on that error reports the same code the session closed with. The tail regressions wait until both ends hold the subscription, so they test the group tail rather than a SUBSCRIBE_OK still in flight. Record that main still pins qmux 0.6.3 and that the fix must come from moq's own stack. Co-Authored-By: Claude Opus 5.5 --- quest/m1/close-tail-drain.md | 7 ++++-- rs/moq-net/src/lite/test_transport.rs | 23 ++++++++++++------- rs/moq-net/tests/integration/session_close.rs | 8 +++++++ 3 files changed, 28 insertions(+), 10 deletions(-) diff --git a/quest/m1/close-tail-drain.md b/quest/m1/close-tail-drain.md index f70e691f624..1cbaf870ef2 100644 --- a/quest/m1/close-tail-drain.md +++ b/quest/m1/close-tail-drain.md @@ -80,8 +80,11 @@ The receive drain also requires qmux to end unfinished stream reads after the session closes, once their buffered data is consumed. In qmux 0.6.3, `SessionState` teardown leaves stream channel senders in the shared `Streams` map retained by `Session` handles, so such reads can remain pending forever. -Fix that lifecycle in `moq-dev/web-transport` and ship a dependency update -before enabling the drain across qmux transports. Keep the TCP publisher +Main still depends on that release (rechecked 2026-10-20), and the hang +reproduces there. moq takes no new work in, or dependency on, moq-dev/web-transport +(maintainer, 2026-10-20), so the fix comes from moq's own stack: the in-tree qmux +that [qmux on moq-quic](/quest/m2/quic-qmux.md) builds, unless the maintainer picks +an interim guard in moq-tokio's async-backend adapter. Keep the TCP publisher restart regression in `moq-gst` passing; it checks that the closed session's source withdraws without waiting on an unfinished media stream. diff --git a/rs/moq-net/src/lite/test_transport.rs b/rs/moq-net/src/lite/test_transport.rs index 6f79090e939..fe462796cfb 100644 --- a/rs/moq-net/src/lite/test_transport.rs +++ b/rs/moq-net/src/lite/test_transport.rs @@ -18,8 +18,12 @@ use std::{ use crate::transport::poll; +/// A failed transport call. After a session close, it carries that close's code, as a +/// real transport reports the code the session ended with. #[derive(Debug, Clone, Default)] -pub struct SinkError; +pub struct SinkError { + code: u32, +} impl std::fmt::Display for SinkError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { @@ -31,7 +35,7 @@ impl std::error::Error for SinkError {} impl crate::transport::Error for SinkError { fn session_error(&self) -> Option<(u32, String)> { - Some((0, "closed".to_string())) + Some((self.code, "closed".to_string())) } } @@ -53,7 +57,8 @@ pub struct Log { streams: Arc, stops: Arc>>, closes: Arc>>, - closed: kio::Shared, + /// The first session close's code, once one happened. + closed: kio::Shared>, /// How long the trail and the stops were at the first session close. closed_at: Arc>>, bi_opens: Arc, @@ -84,10 +89,12 @@ impl Log { fn poll_closed(&self, waiter: &kio::Waiter) -> Poll { self.closed .poll(waiter, |closed| match **closed { - true => Poll::Ready(()), - false => Poll::Pending, + Some(_) => Poll::Ready(()), + None => Poll::Pending, + }) + .map(|closed| SinkError { + code: closed.expect("closed"), }) - .map(|_| SinkError) } /// What every send stream did, in call order, tagged with the stream's id (its @@ -176,7 +183,7 @@ impl Log { .unwrap() .get_or_insert_with(|| (self.trail.lock().unwrap().len(), self.stops.lock().unwrap().len())); self.closes.lock().unwrap().push((code, reason.to_owned())); - *self.closed.lock() = true; + self.closed.lock().get_or_insert(code); } /// How many bidi streams the session has opened, so a test can pin down how many @@ -854,7 +861,7 @@ impl poll::RecvStream for ScriptedRecv { match take { 0 => match *self.close.lock().unwrap() { Some(Close::Fin) => Poll::Ready(Ok(None)), - Some(Close::Reset) => Poll::Ready(Err(SinkError)), + Some(Close::Reset) => Poll::Ready(Err(SinkError::default())), None => self.log.poll_closed(self.park.hold(cx)).map(Err), }, take => Poll::Ready(Ok(Some(take))), diff --git a/rs/moq-net/tests/integration/session_close.rs b/rs/moq-net/tests/integration/session_close.rs index 26b4c4bd17b..1c2aedd57b8 100644 --- a/rs/moq-net/tests/integration/session_close.rs +++ b/rs/moq-net/tests/integration/session_close.rs @@ -64,8 +64,10 @@ async fn setup_with_gate(version: &str, gate: Option>) -> Se .expect("broadcast resolves"); // The publisher only learns of a subscription once the subscriber polls it. + let (ready, active) = futures::channel::oneshot::channel(); let reader = moq_clock::sim::spawn(async move { let mut sub = remote.track("video").unwrap().subscribe(None).await.expect("subscribe"); + let _ = ready.send(()); let mut got = Vec::new(); loop { let mut group = match sub.recv_group().await { @@ -86,6 +88,12 @@ async fn setup_with_gate(version: &str, gate: Option>) -> Se moq_clock::timeout(TIMEOUT, support::harness::subscribed(&track)) .await .expect("no subscriber appeared"); + // Both ends hold the subscription, so a close tests only the group tail and not a + // control response still in flight. + moq_clock::timeout(TIMEOUT, active) + .await + .expect("the subscription never became active") + .expect("reader alive"); Setup { pair, From 922c73f5bc17d58314d77ab16782942fe9a38b84 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 17:01:58 -0700 Subject: [PATCH 6/9] quest: fix the qmux teardown hang in qmux, consumed by a release bump Co-Authored-By: Claude Opus 5.5 --- quest/m1/close-tail-drain.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/quest/m1/close-tail-drain.md b/quest/m1/close-tail-drain.md index 074ce4f36c7..e644b853a15 100644 --- a/quest/m1/close-tail-drain.md +++ b/quest/m1/close-tail-drain.md @@ -77,16 +77,16 @@ moq-tokio's `quic::web` path) for the same ordering. Public API: none. Wire: none. The receive drain also requires qmux to end unfinished stream reads after the -session closes, once their buffered data is consumed. In qmux 0.6.3, -`SessionState` teardown leaves stream channel senders in the shared `Streams` -map retained by `Session` handles, so such reads can remain pending forever. -Main still depends on that release (rechecked 2026-10-20), and the hang -reproduces there. moq takes no new work in, or dependency on, moq-dev/web-transport -(maintainer, 2026-10-20), so the fix comes from moq's own stack: the in-tree qmux -that [qmux on moq-quic](/quest/m2/quic-qmux.md) builds, unless the maintainer picks -an interim guard in moq-tokio's async-backend adapter. Keep the TCP publisher -restart regression in `moq-gst` passing; it checks that the closed session's -source withdraws without waiting on an unfinished media stream. +session closes, once their buffered data is consumed. qmux 0.6.3 and 0.6.4 leave +the stream channel senders in the shared `Streams` table that `Session` handles +retain after teardown, so such reads stay pending forever. Decided (maintainer, +2026-10-21): fix it at the source in qmux now, ahead of +[qmux on the QUIC stream state machine](/quest/m1/quic-qmux.md). +[moq-dev/web-transport#437](https://github.com/moq-dev/web-transport/pull/437) +drops those senders at teardown; this PR bumps `qmux` to the first release that +contains it, with no `[patch]`. Keep the TCP publisher restart regressions in +`moq-gst` passing; they check that the closed session's source withdraws +without waiting on an unfinished media stream. ## Related From fbed668b986df17a65f3167e69f74255328f0845 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 18:48:24 -0700 Subject: [PATCH 7/9] fix(deps): bump qmux to 0.6.5 for the teardown read fix qmux 0.6.5 (moq-dev/web-transport#437) ends receive-stream reads at session teardown, which the close-tail drain relies on. It uses trait API from web-transport-trait 0.5.1, so the lockfile moves that too. Co-Authored-By: Claude Opus 5.5 --- Cargo.lock | 10 +++++----- Cargo.toml | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index a51cdadcec9..1dbed131557 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1959,7 +1959,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c54e03a951783e8b327515db3f2a2fd0e3bed362a96b066f341ce66ed49b4ead" dependencies = [ "data-encoding", - "syn 3.0.6", + "syn 1.0.109", ] [[package]] @@ -7528,9 +7528,9 @@ dependencies = [ [[package]] name = "qmux" -version = "0.6.3" +version = "0.6.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "542edbed299cac6d624d3964c423c2b2a94b19ffcde4eef5fde09b74c64799e4" +checksum = "34b4c766e407d739307a8e7e978af264bbd7ad177615bce98ceb795e0b7912b4" dependencies = [ "bytes", "futures", @@ -11094,9 +11094,9 @@ dependencies = [ [[package]] name = "web-transport-trait" -version = "0.5.0" +version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f1e801f997864ab0074afb1c7d3262df46d5d9ad9b78274c47694544db2dba7" +checksum = "60221f03172c9e8e8d19d1c038c87ef2804d05f8a71264817f60a28bbd03edab" dependencies = [ "bytes", ] diff --git a/Cargo.toml b/Cargo.toml index 479044152d1..ee1b6062a99 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -221,7 +221,7 @@ percent-encoding = "2" pollster = "1.0" # default-features off to skip the fork and timeout machinery; crates opt into what they use. proptest = { version = "1.12.0", default-features = false, features = ["std"] } -qmux = { version = "0.6.3", default-features = false } +qmux = { version = "0.6.5", default-features = false } rand = "0.10.1" rangemap = "1.8" # default-features off so each consumer picks its crypto provider. From 4329029ac47a2f3c6ff1e0710287215e37fa3447 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 19:16:01 -0700 Subject: [PATCH 8/9] fix(net): end the lite drain when groups wait on an undeliverable SETUP With the ietf extension offered, a GROUP stream parks on the peer's SETUP before reading its header. Once the transport is closed and every remaining uni stream is parked that way, no stream can still deliver the SETUP (it was truncated or never opened), so the drain waited forever. Count parked streams and stop draining when they are all that is left. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/lite/session.rs | 40 +++++++++++++++++++++++++++---- rs/moq-net/src/lite/subscriber.rs | 37 +++++++++++++++++++++++++--- 2 files changed, 69 insertions(+), 8 deletions(-) diff --git a/rs/moq-net/src/lite/session.rs b/rs/moq-net/src/lite/session.rs index 2a90de59572..9ad74a989dd 100644 --- a/rs/moq-net/src/lite/session.rs +++ b/rs/moq-net/src/lite/session.rs @@ -923,7 +923,37 @@ mod tests { /// waiting for them, otherwise receive draining could wait for new peer data. #[moq_clock::test] async fn malformed_auth_closes_before_draining_a_partial_group_header() { - assert_session_closes(ScriptedSession::per_stream(vec![vec![99]]).with_incoming_unis(vec![vec![0]])).await; + assert_session_closes( + ScriptedSession::per_stream(vec![vec![99]]).with_incoming_unis(vec![vec![0]]), + Setup::default(), + ) + .await; + } + + /// After close, no stream can deliver the peer SETUP a GROUP stream waits on to read + /// its header, whether the SETUP stream was truncated or never opened. + #[moq_clock::test] + async fn drain_ends_when_groups_wait_on_an_undeliverable_setup() { + let version = Version::Lite07; + let mut group = Vec::new(); + DataType::Group + .encode(&mut crate::coding::Encoder::new(&mut group, version.into()), version) + .unwrap(); + let mut setup = Vec::new(); + DataType::Setup + .encode(&mut crate::coding::Encoder::new(&mut setup, version.into()), version) + .unwrap(); + let ietf = Setup { + ietf: true, + ..Default::default() + }; + for unis in [vec![group.clone(), setup], vec![group]] { + assert_session_closes( + ScriptedSession::per_stream(vec![vec![99]]).with_incoming_unis(unis), + ietf.clone(), + ) + .await; + } } async fn malformed_auth_reply(script: Vec) { @@ -1026,7 +1056,7 @@ mod tests { w.varint(0).unwrap(); w.varint(4).unwrap(); w.varint(0).unwrap(); - assert_session_closes(ScriptedSession::per_stream_eof(vec![script])).await; + assert_session_closes(ScriptedSession::per_stream_eof(vec![script]), Setup::default()).await; } #[moq_clock::test] @@ -1044,10 +1074,10 @@ mod tests { } async fn assert_malformed_reply_closes(script: Vec) { - assert_session_closes(ScriptedSession::per_stream(vec![script])).await; + assert_session_closes(ScriptedSession::per_stream(vec![script]), Setup::default()).await; } - async fn assert_session_closes(transport: ScriptedSession) { + async fn assert_session_closes(transport: ScriptedSession, our_setup: Setup) { let log = transport.log.clone(); let start = start(Config { runtime: crate::time::sim(), @@ -1059,7 +1089,7 @@ mod tests { subscribe: None, peer_hop: None, version: Version::Lite07, - our_setup: Setup::default(), + our_setup, peer_setup: None, auth: crate::auth::Handle::new(true), }) diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 2e032978c16..fa3738f0772 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -77,6 +77,8 @@ pub(super) struct Subscriber { version: Version, /// The peer's advertised SETUP (lite-05+), set when its Setup stream is read. peer_setup: super::PeerSetup, + /// GROUP streams parked until the peer's SETUP says how to read their header. + setup_waits: Arc, /// Local policy overriding the peer's declared egress price. See `poll_link_cost`. cost: Option, /// Sources minted by the announce half, drained by the driver into @@ -119,6 +121,7 @@ impl Subscriber { next_id: Default::default(), version: config.version, peer_setup: config.peer_setup, + setup_waits: Default::default(), cost: config.cost, sources: kio::Queue::new(), going_away: config.going_away, @@ -642,8 +645,12 @@ impl SubscriberDriver { /// stay owned, so their cancellation guards cannot remove subscription state. pub fn poll_drain(&mut self, waiter: &kio::Waiter) -> Poll<()> { let closed = self.uni.poll(waiter).is_ready(); - let drained = self.uni.children.poll(waiter).is_ready(); - if closed && drained { + let _ = self.uni.children.poll(waiter); + // Once every remaining stream waits on a SETUP that never arrived, none of + // them can deliver it: the peer can open no more. + let stuck = self.subscriber.peer_setup.ietf().is_none() + && self.uni.children.len() == self.subscriber.setup_waits.load(atomic::Ordering::Relaxed); + if closed && (self.uni.children.is_empty() || stuck) { Poll::Ready(()) } else { Poll::Pending @@ -860,6 +867,24 @@ struct GroupRecv { subscriber: Subscriber, reader: Reader, state: GroupRecvState, + /// Held while parked on the peer's SETUP, so a drain can tell it never arrives. + setup_wait: Option, +} + +/// Counts one GROUP stream parked on the peer's SETUP for as long as it lives. +struct SetupWait(Arc); + +impl SetupWait { + fn new(waits: &Arc) -> Self { + waits.fetch_add(1, atomic::Ordering::Relaxed); + Self(waits.clone()) + } +} + +impl Drop for SetupWait { + fn drop(&mut self) { + self.0.fetch_sub(1, atomic::Ordering::Relaxed); + } } // A state machine's enum is its storage: one transient instance per stream, so the @@ -887,6 +912,7 @@ impl GroupRecv { subscriber, reader, state: GroupRecvState::Header, + setup_wait: None, } } @@ -895,7 +921,12 @@ impl GroupRecv { match &mut self.state { GroupRecvState::Header => { // Before the header, which a wait after it would lose. - let ietf = ready!(self.subscriber.peer_setup.poll_ietf(waiter)); + let Poll::Ready(ietf) = self.subscriber.peer_setup.poll_ietf(waiter) else { + let waits = &self.subscriber.setup_waits; + self.setup_wait.get_or_insert_with(|| SetupWait::new(waits)); + return Poll::Pending; + }; + self.setup_wait = None; let mut cx = waiter.context(); let hdr = ready!(self.reader.poll_decode::(&mut cx))?; From acbe35ef24cc33dc51d7e9459b0c3ad88c7bc0b7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 20:32:20 -0700 Subject: [PATCH 9/9] quest: delete the finished close-tail drain quest Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index 318785162ab..dd3ff4b5743 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -111,7 +111,6 @@ lite-07 stream type here. - [JS answers SUBSCRIBE_START like Rust](/quest/m1/js-start-parity.md) - a JS publisher's START names where its backlog starts, not its newest group - [Received timestamps are media time](/quest/m1/lite-time-provenance.md) - a received timed track's timestamps are media time, so the rewind check can run across hops - [WebGPU on Safari](/quest/m1/webgpu-safari.md) - the WebGPU renderer is verified on Safari 26 for macOS and iOS -- [Close tail drain](/quest/m1/close-tail-drain.md) - after a peer close, subscribers read the group streams their transport already delivered instead of aborting them, so a finished tail arrives on every version - [Silent-peer close](/quest/m1/quic-silent-peer-drain.md) - moq-quic closes a connection that never heard its peer without the 3 s drain - [Binding linger](/quest/m1/linger-ffi.md) - `moq-ffi` and every binding finish a broadcast cleanly, so subscribers drain it until its media is stale - [JS linger](/quest/m1/linger-js.md) - `js/net` finishes a broadcast cleanly like Rust, and `@moq/publish` uses it when a source stops