From 341e338aaa79509e4c74e8d439c9734f743244d7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 21:31:37 -0800 Subject: [PATCH 1/5] chore(quest): claim quest/m0/backport-fill-head-race.md Quest-Claim: 48dfb52e60b522fa From 18c7a3e78630069aae7781f06524695c692bccd6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 22:50:00 -0800 Subject: [PATCH 2/5] docs(quest): record the joined-head backport and remaining release work Co-Authored-By: GPT-6.1 Sol --- quest/m0/backport-fill-head-race.md | 66 +++++++++++++++-------------- 1 file changed, 35 insertions(+), 31 deletions(-) diff --git a/quest/m0/backport-fill-head-race.md b/quest/m0/backport-fill-head-race.md index 1b19f3d6daf..a3065cc9262 100644 --- a/quest/m0/backport-fill-head-race.md +++ b/quest/m0/backport-fill-head-race.md @@ -1,37 +1,41 @@ -# [M] Backport the joined-head FETCH race fix to release +# [S] Ship the joined-head FETCH race backport ## Goal -A relay on `release` no longer FETCHes the head of a group that its own -SUBSCRIBE is already bringing. In that race the fetched objects can land first, -so the relay caches the group as fetched, refuses the live tail stream that ends -it as `duplicate`, and leaves live readers waiting on the group until they time -out. This ports the relay fix of -[moq-dev/moq#7135](https://github.com/moq-dev/moq/pull/7135) and ships it in a -`moq-net` patch release. +The `release` relay no longer races its subscription's joining head with a +cache-miss FETCH, and the fix reaches users in a `moq-net` patch release. +Public API and wire stay unchanged. ## Plan -Decided 2026-10-30 (maintainer): port only the relay fix. Why: #7135's other -bug, the JS publisher answering an `INCLUDE_PROPERTIES=0` FETCH without -timestamps, cannot happen on `release`, whose `js/net/src/ietf/publisher.ts` -`runFetch` refuses every FETCH. Decided 2026-11-01 (maintainer): #7135 itself -ships only the relay fix; `main` fixed that bug from the relay side instead. - -Decided 2026-10-30 (maintainer): m0, beside the other `release` backports. -Why: it is an intermittent relay hang on the shipped line. - -`release` has the same machinery as main before #7135: the joining-head -`Fill` in `rs/moq-net/src/ietf/subscriber.rs` and the group FETCH that fills a -cache miss (`run_group_fetch`). It has no `rs/moq-net/src/model/track/fill.rs`, -so the fix is hand-ported, not cherry-picked. Port the behavior: a cache miss -for the joined group waits until the subscription's own head arrives or its -request settles, then the cache answers it; a failed head request still falls -back to a FETCH. Only that group waits, so older history still FETCHes at once, -and the wait ends when no reader wants the group or the track aborts. Port -#7135's subscriber tests that cover this alongside it. - -Start by reproducing the race on `release` with one of those tests. If -`release` cannot hit it, delete this quest and say why in the PR. - -Public API: none. Wire: none. +[moq-dev/moq#7135](https://github.com/moq-dev/moq/pull/7135) merged on `main`. +The relay fix is hand-ported in [moq-dev/moq#7680](https://github.com/moq-dev/moq/pull/7680), targeting `release`. + +The race reproduced on `release`: both joined-head miss tests sent an extra +FETCH while the subscription's own fill was still outstanding. The older-history +control fetched immediately. The port preserves those cases and covers reader +cancellation and track abort using release's paused-time harness. + +Decided 2026-10-30 and reaffirmed 2026-11-01 (maintainer): port only the relay +fix. Release's JS publisher refuses every FETCH, so the old publisher timestamp +bug cannot happen there. The settled main PR also carries only the relay fix. + +The port uses release's existing per-group requests and joining `Fill`; it +imports none of main's range-fill or subscription-widening machinery. The +joined group waits for its own head or a settled fill, the cache answers a head +that arrives, and a remaining miss still FETCHes. Older history does not wait. + +Local review also exposed a pending miss no longer being polled during the +PUBLISH_DONE grace. The backport adds that poll and a paused-time reproduction. +Main's corresponding grace also omits it. Land that main-side correction before +the backport, following the main-first backport rule. + +After the backport lands, use the existing patch-release process. Package +version bumps, tags, and publication are separate from this implementation +attempt. Verify that the published `moq-net` patch contains the backport, then +remove this quest and its links. + +## Waiting + +- moq-dev/moq: land the PUBLISH_DONE grace correction on main before [moq-dev/moq#7680](https://github.com/moq-dev/moq/pull/7680) lands; the backport contains the release-side poll and a paused-time reproduction. +- Maintainer: review and merge [moq-dev/moq#7680](https://github.com/moq-dev/moq/pull/7680) after the release checks pass, then publish the `moq-net` patch through the normal release process; verify its source contains the joined-head wait. From 163e92d1b3a465b5ab9e97dc05c8a479b3c23a31 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 10:02:28 -0800 Subject: [PATCH 3/5] chore(quest): claim quest/m0/backport-fill-head-race.md Quest-Claim: ae079970e0949fb7 From 3bc3a80e6a90c56f3bddae3b0caf7bff048d1fb3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 10:33:19 -0800 Subject: [PATCH 4/5] fix(net): keep driving cache-miss fetches through subscription waits The PUBLISH_DONE grace and a resumed subscription's setup waits stopped polling the track's group fetches, so a miss the joined head released, or a reader cancelling its FETCH, stalled until the wait ended. Co-Authored-By: Claude Opus 5.5 --- quest/m0/backport-fill-head-race.md | 10 +- rs/moq-net/src/ietf/subscriber.rs | 150 +++++++++++++++++++++++++++- 2 files changed, 154 insertions(+), 6 deletions(-) diff --git a/quest/m0/backport-fill-head-race.md b/quest/m0/backport-fill-head-race.md index a3065cc9262..09cd8bc0f09 100644 --- a/quest/m0/backport-fill-head-race.md +++ b/quest/m0/backport-fill-head-race.md @@ -27,8 +27,11 @@ that arrives, and a remaining miss still FETCHes. Older history does not wait. Local review also exposed a pending miss no longer being polled during the PUBLISH_DONE grace. The backport adds that poll and a paused-time reproduction. -Main's corresponding grace also omits it. Land that main-side correction before -the backport, following the main-first backport rule. +[moq-dev/moq#7903](https://github.com/moq-dev/moq/pull/7903) lands the same +correction on main first, following the main-first backport rule. Its other +change, driving a lingering copy's fetches while a resumed SUBSCRIBE is set up, +has no `release` counterpart: release creates its fetch set after SUBSCRIBE_OK +and never lingers. After the backport lands, use the existing patch-release process. Package version bumps, tags, and publication are separate from this implementation @@ -37,5 +40,4 @@ remove this quest and its links. ## Waiting -- moq-dev/moq: land the PUBLISH_DONE grace correction on main before [moq-dev/moq#7680](https://github.com/moq-dev/moq/pull/7680) lands; the backport contains the release-side poll and a paused-time reproduction. -- Maintainer: review and merge [moq-dev/moq#7680](https://github.com/moq-dev/moq/pull/7680) after the release checks pass, then publish the `moq-net` patch through the normal release process; verify its source contains the joined-head wait. +- Maintainer: review and merge [moq-dev/moq#7680](https://github.com/moq-dev/moq/pull/7680) after the release checks pass, then publish the `moq-net` patch through the normal release process; verify its source contains the joined-head wait. Agents never merge into `release`, bump versions, tag, or publish, so these steps stay manual. diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 036acdc4865..770de89305e 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -3046,6 +3046,8 @@ impl Subscriber { } loop { let adopt = kio::wait(|waiter| { + // Every wait here drives the fetches a lingering copy started. + let _ = group_fetches.poll(waiter); if gate.poll_denied(waiter).is_ready() { return Poll::Ready(Err(Abandon::Failed(Error::Unauthorized))); } @@ -3117,6 +3119,7 @@ impl Subscriber { None => { let opening = self.session.clone(); let stream = match kio::wait(|waiter| { + let _ = group_fetches.poll(waiter); if gate.poll_denied(waiter).is_ready() { return Poll::Ready(Err(Error::Unauthorized)); } @@ -3270,6 +3273,7 @@ impl Subscriber { }); loop { let setup = kio::wait(|waiter| { + let _ = group_fetches.poll(waiter); let response = waiter.poll_future(response.as_mut()); if let Poll::Ready(Err(err)) = response { return Poll::Ready(Setup::Response(Err(err))); @@ -3366,7 +3370,8 @@ impl Subscriber { // The publisher may already be serving before it answers. A session abort // already rejected the parked request; dropping what remains is not a second one. let aborted = self.remove_subscribe(request_id).is_none(); - self.cancel_subscribe(stream).await; + let mut cancelling = std::pin::pin!(self.cancel_subscribe(stream)); + group_fetches.drive(|waiter| waiter.poll_future(cancelling.as_mut())).await; // A resumed copy goes back to lingering, unless the session took it. return resumed.filter(|_| !aborted); }; @@ -3935,6 +3940,9 @@ impl Subscriber { tracing::debug!(%err, "joining FETCH failed, joining at the next whole group"); } } + // Cache misses keep going meanwhile, such as one the fill + // settling just released to FETCH the joined group. + let _ = group_fetches.poll(waiter); poll_settled(&mut settle, waiter, &fill, count) }) .await; @@ -8690,6 +8698,81 @@ mod tests { } } + /// A lingering copy's fetches keep going while a returning subscriber's SUBSCRIBE awaits + /// its answer: a reader leaving cancels its FETCH then, not once the answer arrives. + #[kio::test] + async fn a_resubscribe_keeps_driving_the_lingering_fetches() { + let version = Version::Draft22; + let status_ok = message( + ietf::TrackStatusOk::ID, + &ietf::TrackStatusOk { + largest: Some(ietf::Location { group: 3, object: 42 }), + properties: Default::default(), + }, + version, + ); + // Neither the FETCH nor the SUBSCRIBE is ever answered. + let session = crate::lite::test_transport::ScriptedSession::per_stream(vec![status_ok]); + let log = session.log.clone(); + let (tasks, _task_set) = crate::util::TaskSet::new(); + let mut subscriber = Subscriber::new( + crate::time::sim(), + crate::driver::Transport::new(session.clone()), + crate::origin::Config::new(crate::Hop::new(1).unwrap()).produce(), + Control::new(false), + None, + peer::PeerSetup::declared(peer::Peer::default()), + crate::Hop::new(1).unwrap(), + None, + version, + tasks, + Default::default(), + ); + let producer = crate::broadcast::Info::default().produce(); + let mut dynamic = producer.requests(); + let consumer = producer.consume(); + let track = consumer.track("video").unwrap(); + // Demand with nobody subscribing learns the track from TRACK_STATUS, then lingers. + let mut query = std::pin::pin!(track.info()); + assert!(futures::poll!(query.as_mut()).is_pending()); + let request = dynamic.requested_track().await.expect("no track requested"); + let serving = kio::clock::sim::spawn(async move { + subscriber + .run_subscribe(Path::from_static("broadcast"), dynamic, None, request) + .await; + }); + settle().await; + + let mut fetch = Box::pin(track.fetch_group(2, None)); + assert!(futures::poll!(fetch.as_mut()).is_pending()); + settle().await; + let _subscription = track.subscribe(None); + settle().await; + let requests = |id| { + let writes = log.writes.lock().unwrap().clone(); + let mut buf = Decoder::new(&writes, version.into()); + let mut count = 0; + while !buf.is_empty() { + let type_id = buf.varint().unwrap(); + let size = buf.u16().unwrap(); + buf.slice(size as usize).unwrap(); + count += usize::from(type_id == id); + } + count + }; + assert_eq!(requests(ietf::Fetch::ID), 1, "the miss FETCHed while lingering"); + assert_eq!(requests(ietf::Subscribe::ID), 1, "the returning subscriber subscribed"); + + let stops = log.stops().len(); + drop(fetch); + settle().await; + assert!( + log.stops().len() > stops, + "the abandoned FETCH is cancelled while SUBSCRIBE awaits its answer" + ); + serving.abort(); + } + /// The publisher opens no fetch stream for an empty range, so a fill against a track /// with no content is owed nothing. Leaving it outstanding would withhold every later /// group behind a head that is never coming. @@ -15411,7 +15494,10 @@ mod joining_fetch_tests { scripts: [Vec; 2], control: Control, ) -> Self { - let session = ScriptedSession::per_stream(Vec::from(scripts)); + Self::subscribe(version, start, ScriptedSession::per_stream(Vec::from(scripts)), control).await + } + + async fn subscribe(version: Version, start: Option, session: ScriptedSession, control: Control) -> Self { let (tasks, _task_set) = crate::util::TaskSet::new(); let subscriber = Subscriber::new( crate::time::sim(), @@ -15916,6 +16002,66 @@ mod joining_fetch_tests { assert_eq!(fetches(), 1, "a settled fill leaves the miss to a FETCH"); } + /// The grace PUBLISH_DONE waits out for reordered data streams keeps driving cache + /// misses: one parked on the joined head FETCHes as soon as the fill settles, rather + /// than once the grace ends. + #[kio::test] + async fn a_miss_fetches_during_the_publish_done_grace() { + let version = Version::Draft22; + let largest = Some(ietf::Location { group: 4, object: 1 }); + let session = ScriptedSession::new(subscribe_ok(version, largest)).with_read_wakes(); + let run = JoinRun::subscribe(version, None, session, Control::new(false)).await; + let fetches = || { + decode_messages(&run.session.log, version) + .iter() + .filter(|(id, _)| *id == ietf::Fetch::ID) + .count() + }; + let mut fetch = Box::pin(run._hold.1.fetch_group(4, None)); + assert!(futures::poll!(fetch.as_mut()).is_pending()); + settle().await; + assert_eq!(fetches(), 0, "the miss waits for the subscription's fill"); + + run.session.push(&message_bytes( + ietf::PublishDone::ID, + &ietf::PublishDone { + status_code: ietf::PublishDoneStatus::TrackEnded.code(), + stream_count: 0, + reason_phrase: "done".into(), + }, + version, + )); + settle().await; + assert!( + !run.subscriber.state.lock().subscribes.is_empty(), + "the grace is still running" + ); + + let fill = run + .subscriber + .state + .lock() + .subscribes + .values() + .next() + .unwrap() + .fill + .clone(); + *fill.write().ok().unwrap() = Fill::Done; + settle().await; + assert!( + !run.subscriber.state.lock().subscribes.is_empty(), + "the grace is still running" + ); + assert_eq!(fetches(), 1, "a settled fill leaves the miss to a FETCH during the grace"); + + kio::clock::sleep(crate::ledger::GRACE * 2).await; + assert!( + run.subscriber.state.lock().subscribes.is_empty(), + "PUBLISH_DONE ended the subscription" + ); + } + /// A malformed answer to the joining FETCH is not a refusal: the peer broke the /// protocol, so the session closes. #[kio::test] From 527e4c9c1713ff025c5c56372fe73881816c70a6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 10:57:02 -0800 Subject: [PATCH 5/5] style(net): format the fetch-driving changes Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/ietf/subscriber.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 770de89305e..976434e033c 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -3371,7 +3371,9 @@ impl Subscriber { // already rejected the parked request; dropping what remains is not a second one. let aborted = self.remove_subscribe(request_id).is_none(); let mut cancelling = std::pin::pin!(self.cancel_subscribe(stream)); - group_fetches.drive(|waiter| waiter.poll_future(cancelling.as_mut())).await; + group_fetches + .drive(|waiter| waiter.poll_future(cancelling.as_mut())) + .await; // A resumed copy goes back to lingering, unless the session took it. return resumed.filter(|_| !aborted); }; @@ -16053,7 +16055,11 @@ mod joining_fetch_tests { !run.subscriber.state.lock().subscribes.is_empty(), "the grace is still running" ); - assert_eq!(fetches(), 1, "a settled fill leaves the miss to a FETCH during the grace"); + assert_eq!( + fetches(), + 1, + "a settled fill leaves the miss to a FETCH during the grace" + ); kio::clock::sleep(crate::ledger::GRACE * 2).await; assert!(