From c8c58be3dc0069f564f24ba34d1644e7b806ba0a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 30 Oct 2026 00:52:12 -0700 Subject: [PATCH 1/4] chore(quest): claim quest/m2/ietf-join-after-finish.md Quest-Claim: cc38b3a19ea65f83 From a2afd006a2373bfd861dd6262f76659bcc9321cf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 30 Oct 2026 02:30:11 -0700 Subject: [PATCH 2/4] fix(net): a draft-18 join reaching a finished track keeps its head A subscription whose track ended cleanly keeps its joining-FETCH entry past PUBLISH_DONE, so a joining FETCH sent on SUBSCRIBE_OK that arrives after the finish is answered with the Largest group's head instead of refused. The entry lasts until the subscriber closes the request, the session closes, or the request token's grant or the session grant stops covering the track. Co-Authored-By: Claude Opus 5.5 --- quest/m1/ietf-fetch-parity.md | 9 +- quest/m2/README.md | 1 - quest/m2/audit-ci.md | 6 +- quest/m2/ietf-fill-refill.md | 2 +- quest/m2/ietf-finish.md | 8 +- quest/m2/ietf-join-after-finish.md | 80 ----- quest/m2/ietf-js-joining-fetch.md | 12 +- rs/moq-net/src/ietf/publisher.rs | 369 ++++++++++++++++++++- rs/moq-net/src/ietf/publisher/tracks.rs | 4 +- rs/moq-net/src/lite/test_transport.rs | 5 + rs/moq-net/tests/integration/track_tail.rs | 57 ++++ 11 files changed, 436 insertions(+), 117 deletions(-) delete mode 100644 quest/m2/ietf-join-after-finish.md diff --git a/quest/m1/ietf-fetch-parity.md b/quest/m1/ietf-fetch-parity.md index cf987ee5fb4..9aa6a874adb 100644 --- a/quest/m1/ietf-fetch-parity.md +++ b/quest/m1/ietf-fetch-parity.md @@ -17,7 +17,7 @@ three FETCH behaviors: the requester has no timescale to read. Non-goals: answering a joining FETCH that arrives after its subscription -ended ([Join after a quick finish](/quest/m2/ietf-join-after-finish.md)), +ended ([moq-dev/moq#7306](https://github.com/moq-dev/moq/pull/7306)), and draft-22 fills. ## Plan @@ -32,9 +32,9 @@ publishers answer FETCH the same way at the cut. held and the requester can still cancel, so the FETCH keeps its own lifetime instead of the subscription's. It holds nothing a hostile peer could not already make it hold through an open FETCH. Check whether this - is part of the draft-18 `tail` flake with a Rust publisher that - [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) describes, - and note the result there. + is part of the draft-18 `tail` flake with a Rust publisher. That flake was a + joining FETCH arriving after PUBLISH_DONE, fixed in + [moq-dev/moq#7306](https://github.com/moq-dev/moq/pull/7306). - **Lost head.** Refuse with DOES_NOT_EXIST, as a missing prefix already is (`a_joining_fetch_refuses_a_missing_prefix_before_fetch_ok`). Update `a_joining_fetch_refuses_a_prefix_group_it_lost`. @@ -51,5 +51,4 @@ Properties. ## Related -- [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) - the late joining FETCH this leaves out - [Headless moq-transport groups](/quest/m2/ietf-headless-group.md) - what a subscriber does with a tail whose head never comes diff --git a/quest/m2/README.md b/quest/m2/README.md index f6d54a23242..62c0ffe8352 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -232,7 +232,6 @@ lite-07; bitrate enforcement is not part of this release. - [Relay metrics through prometheus-client](/quest/m2/relay-prometheus-client.md) - `/metrics` renders through the maintained client with the same series - [JS IETF request streams](/quest/m2/js-request-stream.md) - `RequestStream` owns deadlines and cancellation, and TRACK trailing bytes close the session like Rust - [moq-transport finish interop](/quest/m2/ietf-finish.md) - the interop `finish` case runs on every even moq-transport draft instead of being skipped -- [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) - a pre-draft-22 joining FETCH that arrives after its subscription finished is still answered, within a bound the peer already negotiates - [JS joining FETCH](/quest/m2/ietf-js-joining-fetch.md) - the JS publisher answers a joining FETCH that arrives after a quick finish, mirroring the Rust bound - [ashpd uses maintained D-Bus crates](/quest/m2/ashpd-maintained.md) - wait for a stable zbus 6 port, then remove the portal dependency exceptions - [Askama uses a maintained TOML parser](/quest/m2/askama-maintained.md) - wait for a stable parser migration, then remove the generator dependency exception diff --git a/quest/m2/audit-ci.md b/quest/m2/audit-ci.md index df19a6c3cca..3506e2a9005 100644 --- a/quest/m2/audit-ci.md +++ b/quest/m2/audit-ci.md @@ -68,7 +68,7 @@ verdicts: broadcast` (`remote error: code=0`). Bisected to [moq-dev/moq#6348](https://github.com/moq-dev/moq/pull/6348): the harness finishes its track before the pre-draft-22 joining FETCH arrives, which - [join after a quick finish](/quest/m2/ietf-join-after-finish.md) owns. The + [moq-dev/moq#7306](https://github.com/moq-dev/moq/pull/7306) fixes on the Rust publisher; recheck it on Nightly. The merge queue does not run the browser harness, so only Nightly sees it. - Audio quality: two Chromium Opus `wide` rows exceed their `target_ms_p95` budget. The [audio jitter target](/quest/m1/audio-jitter-target/README.md) @@ -92,8 +92,8 @@ verdicts: [#6317](https://github.com/moq-dev/moq/pull/6317)). The moq-mux large-enum and moq-net minimal-feature lints pass on main `f42b336f0e`. - Interop's finite-track tails fail before draft 22 (`Error: old`, a JS - `RESET_STREAM`); [join after a quick finish](/quest/m2/ietf-join-after-finish.md) - owns them. TS fails T-STD overflow and late audio starts; the TS reliability + `RESET_STREAM`); [moq-dev/moq#7306](https://github.com/moq-dev/moq/pull/7306) fixes the Rust publisher's + half, and [JS joining FETCH](/quest/m2/ietf-js-joining-fetch.md) owns the JS one. TS fails T-STD overflow and late audio starts; the TS reliability quests own them. The relay-log `FAIL` lines in Interop's self-test are negative controls, not failures. - Infrastructure flakes in the sample: DNS failures installing Nix diff --git a/quest/m2/ietf-fill-refill.md b/quest/m2/ietf-fill-refill.md index 3b29aa93f91..c72f3259ae0 100644 --- a/quest/m2/ietf-fill-refill.md +++ b/quest/m2/ietf-fill-refill.md @@ -21,7 +21,7 @@ Group stitching across two streams drew repeated fixes: [#6348](https://github.com/moq-dev/moq/pull/6348), [#6407](https://github.com/moq-dev/moq/pull/6407), plus [headless groups](/quest/m2/ietf-headless-group.md) and -[join after finish](/quest/m2/ietf-join-after-finish.md). The `Fill` enum +join after finish ([moq-dev/moq#7306](https://github.com/moq-dev/moq/pull/7306)). The `Fill` enum and its handoff (`rs/moq-net/src/ietf/subscriber.rs` about lines 273-396 and 4712-4862) exist only because the model has one producer per group. diff --git a/quest/m2/ietf-finish.md b/quest/m2/ietf-finish.md index 9c214702090..9109cbd1bcd 100644 --- a/quest/m2/ietf-finish.md +++ b/quest/m2/ietf-finish.md @@ -17,8 +17,8 @@ Observed on 2026-10-22 with `just test interop --finish --all --versions ietf` - Draft 18: late readers see the finished catalog end empty, a short tail, or a refused joining FETCH (`publish done: internal error`). This is the pre-draft-22 join after a finish, which - [join after a quick finish](/quest/m2/ietf-join-after-finish.md) fixes; rerun - these drafts once it lands. + [moq-dev/moq#7306](https://github.com/moq-dev/moq/pull/7306) fixes for Rust publishers; rerun + these drafts. - Draft 22: several readers time out after the withdrawal (C and C++ on the health broadcast), JS readers see `publish done: internal error`, and native JS refuses a "malformed request stream message" from the relay. @@ -29,7 +29,3 @@ every reader's acknowledgement, so judge each cell from its reader's log. Public API: none. Wire: none expected; any fix that changes framing updates its draft. - -## Required - -- [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) - the pre-draft-22 half of this, without which draft 18 cannot pass diff --git a/quest/m2/ietf-join-after-finish.md b/quest/m2/ietf-join-after-finish.md deleted file mode 100644 index fa1f5271071..00000000000 --- a/quest/m2/ietf-join-after-finish.md +++ /dev/null @@ -1,80 +0,0 @@ -# [M] A quick finish keeps its pre-draft-22 join - -## Goal - -On draft 18, a joining FETCH that reaches our publisher after its -subscription finished cleanly is still answered from the subscription's -saved Largest group. A track that ends within about one RTT of SUBSCRIBE_OK -then finishes cleanly at the subscriber, head included, instead of losing -its last group. -What the publisher retains for this is bounded by design (the session, the -subscription's request, or a limit the peer already negotiates), never by a -timer constant. - -## Plan - -Every pre-draft-22 join is a SUBSCRIBE followed, after SUBSCRIBE_OK, by a -`RelativeJoining{0}` FETCH ([moq-dev/moq#6348](https://github.com/moq-dev/moq/pull/6348)). -A refused one costs the subscriber that group's head and nothing more -([moq-dev/moq#6407](https://github.com/moq-dev/moq/pull/6407), decided with -the maintainer on 2026-10-22): a stream starting mid-group is dropped and -the subscription goes on. - -State: `rs/moq-net/src/ietf/publisher.rs` registers a `Join` entry in -`Publisher::joins` when it dispatches the SUBSCRIBE, fills it with a `Joined` -snapshot as SUBSCRIBE_OK goes out, and drops it when the subscription ends, -before PUBLISH_DONE. A joining FETCH that arrives later finds no entry, waits -out the registration wait in `joined()`, and is refused as an unknown -subscription, so the subscriber loses the Largest group. Short or -already-finished tracks hit this often. - -It flakes the pinned-pass interop `tail` cells with a Rust publisher, about -one run in six (seen 2026-10-22 while landing -[moq-dev/moq#6407](https://github.com/moq-dev/moq/pull/6407)): the tail test -finishes its track before SUBSCRIBE_OK, so PUBLISH_DONE follows within -milliseconds. The relay's settle grace (about a second) ends long before the -10 s registration wait refuses the FETCH, and the empty subscription stream -for the Largest group then becomes an empty group, which `rust -> rust` and -`rust -> js-native-*` read as "empty group". - -Fix it at the publisher: the subscriber cannot recover a head nobody serves, -and other publishers may refuse the same way. - -A finished entry holds a `track::Consumer`, which keeps the track's cache -alive, so retaining it costs memory a peer controls. Candidate bounds: - -- **The request.** Draft 18 gives each SUBSCRIBE its own request stream; - keep the entry until the subscriber closes its side, with our subscriber - holding it open until its joining FETCH resolves. A peer that never sends - the FETCH costs no more than an open subscription, which stream credit - already bounds. -- **The session alone.** Grows with every subscription a long-lived relay - session ever makes, so probably not bounded enough. - -Recommendation: the request close. Keep only entries that can be answered (`Joined::Group`); an -unsubscribed or refused subscription still drops its entry at once. The -entry must not outlive what authorized the subscription: when its request -token's grant expires or is revoked, or the session grant no longer covers -the track, drop it, since a late FETCH reads the cache on that authority. A head -evicted from the cache meanwhile stays a loud refusal. The existing wait in -`joined()` bounds a FETCH whose subscription has not arrived yet; do not -stretch it to cover this. - -Tests, on mocked time with no sleeps, on draft 18: - -- A track that finishes right after SUBSCRIBE_OK, with the joining FETCH - delivered an RTT after PUBLISH_DONE: the FETCH returns the Largest group's - head and the subscriber's track finishes cleanly with it. Today the FETCH - is refused and the group is lost. -- Closing the subscription's request drops its entry. -- A request token that expires (by mocked time) or is revoked after the - finish drops the entry, and the late FETCH is refused. -- An unsubscribed subscription's FETCH is refused, and session close frees - every entry. - -Public API: none. Wire: none; a late joining FETCH is answered where it was -refused. - -## Related - -- [JS joining FETCH](/quest/m2/ietf-js-joining-fetch.md) - the JS publisher mirrors the bound settled here diff --git a/quest/m2/ietf-js-joining-fetch.md b/quest/m2/ietf-js-joining-fetch.md index 9bed251e77b..481c08c34f1 100644 --- a/quest/m2/ietf-js-joining-fetch.md +++ b/quest/m2/ietf-js-joining-fetch.md @@ -16,9 +16,11 @@ in m2. A FETCH that resolves before PUBLISH_DONE already finishes its head. In the tail test the track is finished before SUBSCRIBE_OK goes out, so PUBLISH_DONE follows at once and the FETCH arrives to no subscription. Keep a -finished subscription's join answerable under the bound -[Join after a quick finish](/quest/m2/ietf-join-after-finish.md) settles for -Rust, and mirror it on the joinable-SUBSCRIBE registry js-fetch adds. +finished subscription's join answerable under the bound Rust settled in +[moq-dev/moq#7306](https://github.com/moq-dev/moq/pull/7306), and mirror it on the joinable-SUBSCRIBE registry js-fetch adds: +only a track that ended cleanly keeps its join, until the subscriber closes the +SUBSCRIBE's request stream, the session closes, or the request token's grant +or the session grant stops covering the track. Tests: a unit test for a FETCH that arrives after PUBLISH_DONE, the bound's refusals mirrored from Rust, and the tail cells @@ -28,10 +30,6 @@ plus the browser publisher. Public API: none. Wire: none; a late joining FETCH is answered where it was refused. -## Required - -- [Join after a quick finish](/quest/m2/ietf-join-after-finish.md) - settles the bound this reuses - ## Related - [Headless moq-transport groups](/quest/m2/ietf-headless-group.md) - what a subscriber does with a tail whose head never comes diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index 53e3bd2cac0..9a4a6dfcb65 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -835,7 +835,8 @@ enum Joined { }, } -/// One subscription's entry in [`Publisher::joins`], removed when the subscription ends. +/// One subscription's entry in [`Publisher::joins`], removed when the subscription ends, or +/// for a track that ended cleanly, once its [`Retain`] lets go. /// /// A fetch that arrives afterwards then finds nothing and is refused, rather than being /// answered from a group that is no longer the edge of anything. @@ -844,6 +845,32 @@ struct Join { request_id: RequestId, } +impl Join { + /// Whether a joining FETCH would be answered from this entry rather than refused. + fn answers(&self) -> bool { + matches!( + self.joins.read().get(&self.request_id), + Some(Some(Joined::Group { .. })) + ) + } +} + +/// A cleanly finished draft-18 subscription's join, kept past PUBLISH_DONE. +/// +/// The subscriber sends its joining FETCH once SUBSCRIBE_OK arrives, so on a track that ends +/// within about an RTT of it the FETCH reaches us after PUBLISH_DONE, and would lose the +/// Largest group's head. The entry holds the track's cache, which a peer controls, so it is +/// bounded by the request: kept until the subscriber closes its side (ours holds it open until +/// the FETCH's head lands or its settle grace ends), and no longer than what authorized the +/// subscription, since a late FETCH reads the cache on that authority. The open request still takes its stream +/// credit and subscription slot meanwhile. +struct Retain { + _join: Join>, + grant: Option, + gate: crate::auth::Gate, + _slot: crate::auth::Hold, +} + impl Drop for Join { fn drop(&mut self) { self.joins.lock().remove(&self.request_id); @@ -1338,6 +1365,8 @@ impl Publisher { // releasing what it holds upstream, and only then is the peer told. `acked` holds the // group stream count once SUBSCRIBE_OK is buffered, which decides how. let mut acked: Option> = None; + // Set once the track ran to its end, which is all that leaves a join worth keeping. + let mut track_ended = false; // An instance never comes back, so a request pinned to a gone one is missing, not // worth waiting for. let rendezvous = msg.rendezvous && msg.epoch.is_none(); @@ -1535,6 +1564,7 @@ impl Publisher { let served = served.unwrap_or(Served::Track(Ok(()))); if completed && matches!(served, Served::Track(Ok(()))) { + track_ended = true; self.write_end_of_track(&mut stream, &mut track_serve, &mut request_finished, &mut goaways) .await; } @@ -1562,10 +1592,24 @@ impl Publisher { }) .await }; - // A FETCH joining this subscription ends with it, before the peer reads a word. - drop(_join); - // The subscription's demand is gone; only its credit may still wait on a verification. - drop((request_grant, _slot)); + // A track that ended cleanly keeps what a joining FETCH needs (see [`Retain`]); any + // other end takes the join with it, before the peer reads a word. + let retain = match (&subscribed, _join) { + (Ok(Ok(Subscribed::Served(Served::Track(Ok(())), _))), Some(join)) if track_ended && join.answers() => { + Some(Retain { + _join: join, + grant: request_grant, + gate, + _slot, + }) + } + _ => { + // The subscription's demand is gone; only its credit may still wait on a + // verification. + drop((request_grant, _slot)); + None + } + }; let (served, streams) = match subscribed { Ok(Ok(Subscribed::Served(served, streams))) => (served, streams), @@ -1585,7 +1629,8 @@ impl Publisher { } }, }; - self.write_publish_done(stream, served, streams, Some(&credit)).await + self.write_publish_done(stream, served, streams, Some(&credit), retain) + .await } } @@ -1668,13 +1713,14 @@ impl Publisher { /// End a subscription on the wire with the PUBLISH_DONE `served` reports, after `streams` /// group streams, returning how serving ended. The stream closes only once no verification - /// holds `credit`'s request, if it has one. + /// holds `credit`'s request, if it has one, and once `retain` lets go of its join. async fn write_publish_done( &self, mut stream: Stream, served: Served, streams: u64, credit: Option<&kio::Producer<()>>, + retain: Option, ) -> Result<(), Error> { let (status, reason) = served.status(); let res = match served { @@ -1692,6 +1738,10 @@ impl Publisher { }) .await; + if let Some(retain) = retain { + self.retain_join(&mut stream, retain).await; + } + // The peer learns the subscription ended at once, but the stream holds its credit until // the application lets go of every token it took for it. if let Some(credit) = credit { @@ -1703,6 +1753,33 @@ impl Publisher { res } + /// Keep a finished subscription's join until the subscriber closes the request, the + /// session closes, or what authorized the subscription stops covering it. + async fn retain_join(&self, stream: &mut Stream, mut retain: Retain) { + let session = self.session.clone(); + kio::wait(|waiter| { + if session.poll_closed(waiter).is_ready() || stream.writer.poll_closed(waiter).is_ready() { + return Poll::Ready(()); + } + // An update that crossed PUBLISH_DONE is skipped, since nothing answers it now; the + // subscriber's FIN or a reset ends the request. + loop { + match stream.reader.poll_read_chunk(waiter, usize::MAX) { + Poll::Ready(Ok(Some(_))) => continue, + Poll::Ready(_) => return Poll::Ready(()), + Poll::Pending => break, + } + } + if let Some(grant) = &retain.grant + && grant.poll_ended(waiter).is_ready() + { + return Poll::Ready(()); + } + retain.gate.poll_denied(waiter) + }) + .await; + } + async fn run_subscription( &self, stream: &mut Stream, @@ -6722,7 +6799,7 @@ mod subscribe_cursor_test { mod serve_tests { use super::*; use crate::coding::{Decode, Decoder}; - use crate::lite::test_transport::{Log, ScriptedSession, Sent, SinkSession}; + use crate::lite::test_transport::{Close, Log, ScriptedSession, Sent, SinkSession}; use crate::model::ProduceTest; #[moq_clock::test] @@ -8389,7 +8466,20 @@ mod serve_tests { } h.track.finish().unwrap(); - serve.await.unwrap(); + // A draft-18 subscription whose track ended keeps its join until the subscriber + // closes the request, so close it once PUBLISH_DONE is out. + while futures::poll!(serve.as_mut()).map(Result::unwrap).is_pending() { + if publish_done_sent(h) { + h.session.close(Close::Fin); + } + moq_clock::sim::yield_now().await; + } + h.session.reopen(); + } + + /// Whether the subscription's request stream carries PUBLISH_DONE yet. + fn publish_done_sent(h: &Serve) -> bool { + !h.log.trail().is_empty() && Wire::read(&h.log, h.publisher.version).publish_done.is_some() } /// A subscribe for a broadcast we do not serve is refused with the negotiated draft's own @@ -9868,8 +9958,13 @@ mod serve_tests { /// Drive a subscription until its snapshot is available, failing if it ends first. async fn registered(h: &Serve, serving: impl std::future::Future>) { + registered_as(h, RequestId(REQUEST_ID), serving).await + } + + /// [`registered`], for the subscription with request id `id`. + async fn registered_as(h: &Serve, id: RequestId, serving: impl std::future::Future>) { let joined = h.publisher.joins.wait(|joins| { - if matches!(joins.get(&RequestId(REQUEST_ID)), Some(Some(_))) { + if matches!(joins.get(&id), Some(Some(_))) { Poll::Ready(()) } else { Poll::Pending @@ -10193,8 +10288,9 @@ mod serve_tests { ); } - /// A joining FETCH that arrives after its subscription ended has no live edge to name, so - /// it is refused rather than answered from a group that is no longer the edge of anything. + /// A joining FETCH that arrives after its subscription ended and the subscriber closed the + /// request has no live edge to name, so it is refused rather than answered from a group + /// that is no longer the edge of anything. #[moq_clock::test] async fn a_joining_fetch_without_its_subscription_is_refused() { for version in JOINING_DRAFTS { @@ -10217,6 +10313,255 @@ mod serve_tests { } } + /// The refusal code of the REQUEST_ERROR a joining FETCH is answered with, failing on a + /// FETCH_OK. + async fn joining_refusal(h: &Serve, version: Version) -> u64 { + let mark = h.log.writes.lock().unwrap().len(); + let mut buf = bytes::Bytes::from(joining_fetch(h, mark).await.unwrap()); + assert_eq!( + crate::coding::decode_varint(&mut buf, version).unwrap(), + ietf::RequestError::ID, + "{version}: the joining FETCH was answered" + ); + crate::coding::decode_buf(&mut buf, version, |r, v| { + Ok(ietf::RequestError::decode(r, v)?.error_code) + }) + .unwrap() + } + + /// Finish the track and drive `serving` until its PUBLISH_DONE is out, failing if the + /// request ends with it: a finished subscription keeps its join until the subscriber + /// closes the request. + async fn finish_and_retain(h: &Serve, mut serving: std::pin::Pin<&mut impl Future>>) { + h.track.finish().unwrap(); + for _ in 0..200 { + assert!( + futures::poll!(serving.as_mut()).is_pending(), + "the request ended with the subscription" + ); + if publish_done_sent(h) { + return; + } + moq_clock::sim::yield_now().await; + } + panic!("PUBLISH_DONE never went out"); + } + + /// A track that ends within an RTT of SUBSCRIBE_OK ends its subscription before the + /// joining FETCH sent on that SUBSCRIBE_OK arrives. The FETCH is still answered with the + /// Largest group's head, until the subscriber closes the request. + #[moq_clock::test] + async fn a_joining_fetch_after_the_track_ends_is_answered() { + for version in JOINING_DRAFTS { + let mut h = serve(version); + publish_groups(&mut h, 4); + let mut group = h.track.create_group(group::Info { sequence: 5 }).unwrap(); + group.write_frame(timestamp(), b"the-head".as_slice()).unwrap(); + group.finish().unwrap(); + settle().await; + + let stream = Stream::open(&mut crate::driver::Transport::new(h.session.clone()), version) + .await + .unwrap(); + let mut serving = std::pin::pin!( + h.publisher + .clone() + .run_subscribe_stream(stream, subscribe(Filter::NextObject, None)) + ); + registered(&h, serving.as_mut()).await; + finish_and_retain(&h, serving.as_mut()).await; + + // The FETCH lands an RTT after PUBLISH_DONE. + moq_clock::sleep(Duration::from_millis(100)).await; + let mark = h.log.writes.lock().unwrap().len(); + let mut buf = bytes::Bytes::from(joining_fetch(&h, mark).await.unwrap()); + assert_eq!( + crate::coding::decode_varint(&mut buf, version).unwrap(), + ietf::FetchOk::ID, + "{version}: the joining FETCH was refused" + ); + let ok = crate::coding::decode_buf(&mut buf, version, |r, v| ietf::FetchOk::decode(r, v)).unwrap(); + assert_eq!(ok.end_location, Location { group: 5, object: 1 }, "{version}"); + assert_eq!( + occurrences(&h.log, b"the-head"), + 1, + "{version}: the head was not served" + ); + assert!(h.log.resets().is_empty(), "{version}"); + + // Closing the request drops the join, and a FETCH after that is refused. + assert!(futures::poll!(serving.as_mut()).is_pending(), "{version}"); + assert!(!h.publisher.joins.read().is_empty(), "{version}"); + h.session.close(Close::Fin); + serving.await.unwrap(); + assert!( + h.publisher.joins.read().is_empty(), + "{version}: the join outlived its request" + ); + h.session.reopen(); + assert_eq!(joining_refusal(&h, version).await, INVALID_JOINING_REQUEST_ID); + } + } + + /// A REQUEST_UPDATE that crossed PUBLISH_DONE is skipped, not taken as a protocol + /// violation, and the join stays until the subscriber's FIN. + #[moq_clock::test] + async fn an_update_after_publish_done_keeps_the_join() { + for version in JOINING_DRAFTS { + // One script shared by every stream, so the update can arrive after PUBLISH_DONE. + let mut h = serve_session( + version, + crate::track::Info::timed(), + crate::origin::Route::default(), + ScriptedSession::new(Vec::new()), + ); + publish_groups(&mut h, 5); + settle().await; + let stream = Stream::open(&mut crate::driver::Transport::new(h.session.clone()), version) + .await + .unwrap(); + let mut serving = std::pin::pin!( + h.publisher + .clone() + .run_subscribe_stream(stream, subscribe(Filter::NextObject, None)) + ); + registered(&h, serving.as_mut()).await; + finish_and_retain(&h, serving.as_mut()).await; + + // REQUEST_UPDATE, Request ID 2, no parameters. + h.session.push(&[0x02, 0, 2, 2, 0]); + for _ in 0..10 { + assert!(futures::poll!(serving.as_mut()).is_pending(), "{version}"); + moq_clock::sim::yield_now().await; + } + assert!(h.log.closes().is_empty(), "{version}: the update closed the session"); + assert!( + !h.publisher.joins.read().is_empty(), + "{version}: the update dropped the join" + ); + + h.session.close(Close::Fin); + serving.await.unwrap(); + assert!(h.publisher.joins.read().is_empty(), "{version}"); + } + } + + /// A subscription the subscriber cancels before the track ends keeps nothing for a later + /// joining FETCH. + #[moq_clock::test] + async fn an_unsubscribed_join_is_not_kept() { + for version in JOINING_DRAFTS { + let mut h = serve(version); + publish_groups(&mut h, 5); + settle().await; + let stream = Stream::open(&mut crate::driver::Transport::new(h.session.clone()), version) + .await + .unwrap(); + let mut serving = std::pin::pin!( + h.publisher + .clone() + .run_subscribe_stream(stream, subscribe(Filter::NextObject, None)) + ); + registered(&h, serving.as_mut()).await; + h.session.close(Close::Fin); + serving.await.unwrap(); + assert!(h.publisher.joins.read().is_empty(), "{version}"); + h.session.reopen(); + assert_eq!(joining_refusal(&h, version).await, INVALID_JOINING_REQUEST_ID); + } + } + + /// The session closing frees every join a finished subscription kept. + #[moq_clock::test] + async fn a_session_close_frees_every_kept_join() { + for version in JOINING_DRAFTS { + let mut h = serve(version); + publish_groups(&mut h, 5); + settle().await; + let mut servings = Vec::new(); + for id in [REQUEST_ID, REQUEST_ID + 2] { + let stream = Stream::open(&mut crate::driver::Transport::new(h.session.clone()), version) + .await + .unwrap(); + let mut msg = subscribe(Filter::NextObject, None); + msg.request_id = RequestId(id); + let mut serving = Box::pin(h.publisher.clone().run_subscribe_stream(stream, msg)); + registered_as(&h, RequestId(id), serving.as_mut()).await; + servings.push(serving); + } + h.track.finish().unwrap(); + for _ in 0..50 { + for serving in &mut servings { + assert!(futures::poll!(serving.as_mut()).is_pending(), "{version}"); + } + moq_clock::sim::yield_now().await; + } + assert_eq!( + h.publisher.joins.read().len(), + 2, + "{version}: the finished joins were not kept" + ); + + crate::transport::poll::Session::close(&h.session, 0, "done"); + for serving in servings { + serving.await.unwrap(); + } + assert!(h.publisher.joins.read().is_empty(), "{version}"); + } + } + + /// How a token subscription's grant ends after its track did. + #[derive(Clone, Copy, Debug)] + enum GrantEnd { + Expired, + Revoked, + } + + /// A join kept past the track's end reads the cache on the subscription's request token, + /// so it goes once that grant expires or is revoked, and a late FETCH is refused. + #[moq_clock::test] + async fn a_kept_join_ends_with_its_request_token() { + const VERSION: Version = Version::Draft18; + for end in [GrantEnd::Expired, GrantEnd::Revoked] { + let (auth, mut requests, _cred) = auth_covering_other(); + let mut h = serve_with_auth(VERSION, auth, Vec::new()); + let rt = h.publisher.runtime.clone(); + publish_groups(&mut h, 5); + settle().await; + + let stream = Stream::open(&mut crate::driver::Transport::new(h.session.clone()), VERSION) + .await + .unwrap(); + let mut serving = std::pin::pin!(h.publisher.clone().run_subscribe_stream(stream, token_subscribe())); + let accepted = async { requests.next().await.expect("a request") }; + let request = match futures::future::select(std::pin::pin!(accepted), serving.as_mut()).await { + futures::future::Either::Left((request, _)) => request, + futures::future::Either::Right((res, _)) => panic!("{end:?}: ended unverified: {res:?}"), + }; + let enforced = request.accept(grant_all_expiring(&rt, Some(60))); + registered(&h, serving.as_mut()).await; + finish_and_retain(&h, serving.as_mut()).await; + + match end { + GrantEnd::Expired => moq_clock::sleep(Duration::from_secs(61)).await, + GrantEnd::Revoked => { + moq_clock::sleep(Duration::from_secs(1)).await; + assert!(futures::poll!(serving.as_mut()).is_pending(), "{end:?}"); + enforced.revoke(crate::SessionError::Unauthorized, "revoked"); + } + } + let _ = moq_clock::timeout(Duration::from_secs(1), serving.as_mut()) + .await + .unwrap_or_else(|_| panic!("{end:?}: the join outlived its grant")); + assert!(h.publisher.joins.read().is_empty(), "{end:?}"); + assert_eq!( + joining_refusal(&h, VERSION).await, + INVALID_JOINING_REQUEST_ID, + "{end:?}" + ); + } + } + #[moq_clock::test] async fn a_joining_fetch_rejects_unsupported_filters() { for version in JOINING_DRAFTS { diff --git a/rs/moq-net/src/ietf/publisher/tracks.rs b/rs/moq-net/src/ietf/publisher/tracks.rs index 066a563f983..03a08943af2 100644 --- a/rs/moq-net/src/ietf/publisher/tracks.rs +++ b/rs/moq-net/src/ietf/publisher/tracks.rs @@ -975,7 +975,7 @@ impl Publisher { if let Some(done) = refused { drop(join); - return self.write_publish_done(stream, done, 0, None).await; + return self.write_publish_done(stream, done, 0, None, None).await; } // Group streams opened over every time the subscription forwarded, which @@ -1109,7 +1109,7 @@ impl Publisher { }; drop(join); - self.write_publish_done(stream, done, opened, None).await + self.write_publish_done(stream, done, opened, None, None).await } /// Mark a paused subscription for joining FETCHes: draft-18 section 5.1 refuses one diff --git a/rs/moq-net/src/lite/test_transport.rs b/rs/moq-net/src/lite/test_transport.rs index a17bf539c68..ba121a11608 100644 --- a/rs/moq-net/src/lite/test_transport.rs +++ b/rs/moq-net/src/lite/test_transport.rs @@ -1083,6 +1083,11 @@ impl ScriptedSession { *self.close.lock().unwrap() = Some(close); } + /// Undo [`Self::close`], so streams opened from here park once their script runs out. + pub fn reopen(&self) { + *self.close.lock().unwrap() = None; + } + /// Answer each stream from `scripts`, but only once the gate opens: a peer that /// replies normally after its transport header gets send capacity. pub fn gated_open(scripts: Vec>, gate: kio::Consumer) -> Self { diff --git a/rs/moq-net/tests/integration/track_tail.rs b/rs/moq-net/tests/integration/track_tail.rs index 1264fd07afd..ebbd7efedef 100644 --- a/rs/moq-net/tests/integration/track_tail.rs +++ b/rs/moq-net/tests/integration/track_tail.rs @@ -262,6 +262,63 @@ async fn lite07_empty_track_ends_without_the_grace() { assert!(outcome.elapsed < GRACE / 10, "ended after {:?}", outcome.elapsed); } +/// Before draft 22, a group written before SUBSCRIBE_OK sits below its Largest Object, so +/// its head arrives on the joining FETCH sent once SUBSCRIBE_OK lands. A track that ends as +/// the subscription starts has ended it by the time that FETCH reaches the publisher, an RTT +/// later, and still ends cleanly with the head. +#[moq_clock::test] +async fn draft18_a_join_reaching_a_finished_track_keeps_its_head() { + const RTT: Duration = Duration::from_millis(100); + + let Pair { + pair, + track, + remote, + _keep, + } = connect("moq-transport-18").await; + pair.client_transport.set_latency(RTT / 2); + let mut group = track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, PAYLOAD).unwrap(); + group.finish().unwrap(); + + let reader = moq_clock::sim::spawn(async move { + let mut sub = remote + .track("video") + .unwrap() + .subscribe(moq_net::track::Subscription::default().with_range(0..)) + .await + .expect("subscribe"); + let mut frames = Vec::new(); + let err = loop { + let mut group = match sub.recv_group().await { + Ok(Some(group)) => group, + Ok(None) => break None, + Err(err) => break Some(err), + }; + while let Some(frame) = group.read_frame().await.expect("group failed") { + frames.push(frame.payload.to_vec()); + } + }; + (frames, err) + }); + + moq_clock::timeout(TIMEOUT, support::harness::subscribed(&track)) + .await + .expect("no subscriber appeared"); + // Kept alive, as a publisher draining its session does: a finished track whose producer + // is gone lets its cache reclaim the head, and that FETCH is refused. + track.finish().unwrap(); + let _track = track; + + let (frames, err) = moq_clock::timeout(TIMEOUT, reader) + .await + .expect("the subscription never ended") + .expect("reader panicked"); + assert!(err.is_none(), "{err:?}"); + assert_eq!(frames, [PAYLOAD], "the head was lost"); + drop((pair, _keep)); +} + /// IETF drafts with a joining FETCH (18) and with subscription fills (22). const IETF: &[&str] = &["moq-transport-18", "moq-transport-22"]; From 1ea2c29b632f4df9bdcf7f23089a816026e3da34 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 30 Oct 2026 03:25:06 -0700 Subject: [PATCH 3/4] fix(net): a kept join watches its grant while PUBLISH_DONE waits, and drains cooperatively Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/ietf/publisher.rs | 123 +++++++++++++++++++++++++++---- 1 file changed, 109 insertions(+), 14 deletions(-) diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index bb02ad006a8..d23846c115a 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -871,6 +871,18 @@ struct Retain { _slot: crate::auth::Hold, } +impl Retain { + /// Ready once what authorized the subscription no longer covers it. + fn poll_lapsed(&mut self, waiter: &kio::Waiter) -> Poll<()> { + if let Some(grant) = &self.grant + && grant.poll_ended(waiter).is_ready() + { + return Poll::Ready(()); + } + self.gate.poll_denied(waiter) + } +} + impl Drop for Join { fn drop(&mut self) { self.joins.lock().remove(&self.request_id); @@ -1729,14 +1741,27 @@ impl Publisher { Served::Paused(_) => Ok(()), }; - let _ = stream - .writer - .encode_message(&ietf::PublishDone { + let mut retain = retain; + { + let done = ietf::PublishDone { status_code: status.code(), stream_count: streams, reason_phrase: reason.into(), + }; + // A peer holding back flow control parks PUBLISH_DONE, and the kept join must still + // go once the request closes or its authorization lapses meanwhile. + let mut encode = std::pin::pin!(stream.writer.encode_message(&done)); + let reader = &mut stream.reader; + let _ = kio::wait(|waiter| { + if let Some(kept) = &mut retain + && (reader.poll_discard(waiter).is_ready() || kept.poll_lapsed(waiter).is_ready()) + { + retain = None; + } + waiter.poll_future(encode.as_mut()) }) .await; + } if let Some(retain) = retain { self.retain_join(&mut stream, retain).await; @@ -1763,19 +1788,10 @@ impl Publisher { } // An update that crossed PUBLISH_DONE is skipped, since nothing answers it now; the // subscriber's FIN or a reset ends the request. - loop { - match stream.reader.poll_read_chunk(waiter, usize::MAX) { - Poll::Ready(Ok(Some(_))) => continue, - Poll::Ready(_) => return Poll::Ready(()), - Poll::Pending => break, - } - } - if let Some(grant) = &retain.grant - && grant.poll_ended(waiter).is_ready() - { + if stream.reader.poll_discard(waiter).is_ready() { return Poll::Ready(()); } - retain.gate.poll_denied(waiter) + retain.poll_lapsed(waiter) }) .await; } @@ -10457,6 +10473,85 @@ mod serve_tests { } } + /// A peer holding back flow control parks PUBLISH_DONE, and the kept join still goes + /// once the subscription's request token is revoked meanwhile. + #[moq_clock::test] + async fn a_kept_join_ends_with_its_token_while_publish_done_waits() { + for version in JOINING_DRAFTS { + let origin = crate::origin::Config::new(crate::Hop::new(1).unwrap()).produce(); + let broadcast = origin.publish("room", crate::origin::Route::default()).unwrap(); + let track = broadcast.create_track("video", crate::track::Info::timed()).unwrap(); + let mut group = track.create_group(group::Info { sequence: 0 }).unwrap(); + group.write_frame(timestamp(), b"frame".to_vec()).unwrap(); + group.finish().unwrap(); + settle().await; + + let bi = kio::Producer::new(true); + let session = SinkSession::gated_bi(bi.consume()); + let peer_setup = peer::PeerSetup::default(); + peer_setup.set(peer::Peer::default()); + let auth = crate::auth::Link::new(false); + let mut requests = auth.handle().requests(); + // A limit of nothing, so only the token admits the subscription. + auth.handle().authorize(&crate::auth::Grant::default()); + let publisher = Publisher::new( + crate::time::sim(), + crate::driver::Transport::new(session.clone()), + origin.consume(), + Control::new(false), + None, + peer_setup, + version, + ) + .with_auth(auth.clone()); + + let stream = Stream::open(&mut crate::driver::Transport::new(session.clone()), version) + .await + .unwrap(); + let mut serving = std::pin::pin!(publisher.clone().run_subscribe_stream(stream, token_subscribe())); + let accepted = async { requests.next().await.expect("a request") }; + let request = match futures::future::select(std::pin::pin!(accepted), serving.as_mut()).await { + futures::future::Either::Left((request, _)) => request, + futures::future::Either::Right((res, _)) => panic!("{version}: ended unverified: {res:?}"), + }; + let enforced = request.accept(crate::auth::Grant::all()); + let joined = publisher.joins.wait(|joins| match joins.get(&RequestId(REQUEST_ID)) { + Some(Some(_)) => Poll::Ready(()), + _ => Poll::Pending, + }); + match futures::future::select(std::pin::pin!(joined), serving.as_mut()).await { + futures::future::Either::Left(_) => {} + futures::future::Either::Right((res, _)) => panic!("{version}: subscription ended: {res:?}"), + } + + // The request stream takes nothing more, so PUBLISH_DONE parks. + let Ok(mut writable) = bi.write() else { + panic!("request stream gate closed"); + }; + *writable = false; + drop(writable); + track.finish().unwrap(); + for _ in 0..50 { + assert!(futures::poll!(serving.as_mut()).is_pending(), "{version}"); + settle().await; + } + assert!( + !publisher.joins.read().is_empty(), + "{version}: the finished join was not kept" + ); + + enforced.revoke(crate::SessionError::Unauthorized, "revoked"); + for _ in 0..10 { + let _ = futures::poll!(serving.as_mut()); + settle().await; + } + assert!( + publisher.joins.read().is_empty(), + "{version}: the join outlived its grant while PUBLISH_DONE waited" + ); + } + } + /// A subscription the subscriber cancels before the track ends keeps nothing for a later /// joining FETCH. #[moq_clock::test] From 8c1fd92f9af58b6b29b9c51c9293d197624c808b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 1 Nov 2026 09:30:00 -0800 Subject: [PATCH 4/4] test(net): close the request once a cold-copy subscription's PUBLISH_DONE is out main's a_cold_reported_copy_serves_its_first_wire_object awaited a finished draft-18 subscription, which now keeps its join until the subscriber closes the request. Share run_live's drive-and-close loop as served(). Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/ietf/publisher.rs | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index 30d6c01925f..edbf55b26fd 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -8542,15 +8542,20 @@ mod serve_tests { } h.track.finish().unwrap(); - // A draft-18 subscription whose track ended keeps its join until the subscriber - // closes the request, so close it once PUBLISH_DONE is out. - while futures::poll!(serve.as_mut()).map(Result::unwrap).is_pending() { + served(h, serve).await; + h.session.reopen(); + } + + /// Drive `serving` to a clean end once its track has finished. A draft-18 subscription + /// whose track ended keeps its join until the subscriber closes the request, so close it + /// once PUBLISH_DONE is out. + async fn served(h: &Serve, mut serving: std::pin::Pin<&mut impl Future>>) { + while futures::poll!(serving.as_mut()).map(Result::unwrap).is_pending() { if publish_done_sent(h) { h.session.close(Close::Fin); } kio::clock::sim::yield_now().await; } - h.session.reopen(); } /// Whether the subscription's request stream carries PUBLISH_DONE yet. @@ -9925,7 +9930,7 @@ mod serve_tests { } frame.finish().unwrap().finish().unwrap(); h.track.finish().unwrap(); - serving.await.unwrap(); + served(&h, serving.as_mut()).await; assert_eq!(occurrences(&h.log, payload), 1, "{version} streamed={streamed}"); assert!(h.log.resets().is_empty(), "{version} streamed={streamed}");