diff --git a/quest/m0/backport-fill-head-race.md b/quest/m0/backport-fill-head-race.md index 999fed9f17f..1b19f3d6daf 100644 --- a/quest/m0/backport-fill-head-race.md +++ b/quest/m0/backport-fill-head-race.md @@ -6,16 +6,17 @@ A relay on `release` no longer FETCHes the head of a group that its own SUBSCRIBE is already bringing. In that race the fetched objects can land first, so the relay caches the group as fetched, refuses the live tail stream that ends it as `duplicate`, and leaves live readers waiting on the group until they time -out. This ports fix 2 of +out. This ports the relay fix of [moq-dev/moq#7135](https://github.com/moq-dev/moq/pull/7135) and ships it in a `moq-net` patch release. ## Plan Decided 2026-10-30 (maintainer): port only the relay fix. Why: #7135's other -fix, the JS publisher answering an `INCLUDE_PROPERTIES=0` FETCH without +bug, the JS publisher answering an `INCLUDE_PROPERTIES=0` FETCH without timestamps, cannot happen on `release`, whose `js/net/src/ietf/publisher.ts` -`runFetch` refuses every FETCH. +`runFetch` refuses every FETCH. Decided 2026-11-01 (maintainer): #7135 itself +ships only the relay fix; `main` fixed that bug from the relay side instead. Decided 2026-10-30 (maintainer): m0, beside the other `release` backports. Why: it is an intermittent relay hang on the shipped line. @@ -34,7 +35,3 @@ Start by reproducing the race on `release` with one of those tests. If `release` cannot hit it, delete this quest and say why in the PR. Public API: none. Wire: none. - -## Required - -- [The browser backend baseline completes on IETF22](/quest/m0/nightly/browser-baseline.md) - lands the fix this quest ports, on main diff --git a/quest/m0/nightly/README.md b/quest/m0/nightly/README.md index 4f4ffdee91c..720f0a44712 100644 --- a/quest/m0/nightly/README.md +++ b/quest/m0/nightly/README.md @@ -1,4 +1,4 @@ -# Nightly passes on main +# [S] Nightly passes on main ## Goal @@ -32,7 +32,3 @@ This README's own work, after the children: threshold to [bench CI](/quest/m1/bench-ci.md). Public API: none. Wire: none. - -## Required - -- [The browser backend baseline completes on IETF22](/quest/m0/nightly/browser-baseline.md) - the WASM and UniFFI backend baseline completes every supported transport case without an internal error diff --git a/quest/m0/nightly/browser-baseline.md b/quest/m0/nightly/browser-baseline.md deleted file mode 100644 index ff173f85878..00000000000 --- a/quest/m0/nightly/browser-baseline.md +++ /dev/null @@ -1,20 +0,0 @@ -# [M] The browser backend baseline completes on IETF22 - -## Goal - -The WASM and UniFFI backend baseline completes every supported transport case without an internal error. - -## Plan - -[The browser baseline job](https://github.com/moq-dev/moq/actions/runs/40698329026/job/122851001597) fails under -`just test wasm-baseline --uniffi` with `evaluate: MoqError.Internal` during -the moq-transport-22 benchmark. Its relay log also contains a uni-stream -`not found` error; that is evidence to investigate, not an established cause. -The retained `wasm-backend-baseline` artifact contains the run's diagnostics. - -Reproduce the failing cell on current main and trace the original transport -or model error through the bindings. Preserve completion and frame-integrity -checks. Review the already-landed reset/error mapping fixes before attributing -this to UniFFI itself. - -Public API: none expected. Wire: none expected. diff --git a/quest/m2/fill-wait-cancel-test.md b/quest/m2/fill-wait-cancel-test.md index 5cd3cc04464..024533d9a27 100644 --- a/quest/m2/fill-wait-cancel-test.md +++ b/quest/m2/fill-wait-cancel-test.md @@ -23,7 +23,3 @@ that group is the documented contract and a late FETCH risks the duplicate group #7135 fixed. Public API: none. Wire: none. - -## Required - -- [The browser backend baseline completes on IETF22](/quest/m0/nightly/browser-baseline.md) - its PR, #7135, adds the wait this tests diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 5316a6109ad..e3b5e247c24 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -3832,6 +3832,7 @@ impl Subscriber { track_name.to_owned(), requested, Later::Known(timescale), + Some(Heading { request_id }), ); group_fetches.push(fetch); } @@ -4076,6 +4077,7 @@ impl Subscriber { track_name.to_owned(), fill, Later::Known(idle.properties.timescale), + None, ); group_fetches.push(fetch); } @@ -4483,6 +4485,7 @@ impl Subscriber { track_name.to_owned(), fill, timescale.later(), + None, ); group_fetches.push(fetch); } @@ -5464,6 +5467,13 @@ impl Subscriber { /// hold the earlier groups until it finishes, or be cut short at the Largest Object by a /// publisher following draft-22 section 9.11. /// + /// `heading` is the subscription's own fill, when it may be bringing a joined group's + /// head. That group waits for it instead: a FETCH racing it can cache the group first, + /// and then the tail stream, which is what ends the group for live readers, is refused + /// as a duplicate. The cache answers the group once the head lands, and a fill that + /// settles without one leaves it to a FETCH. Only that group waits, and only while a + /// reader still wants it. + /// /// `_working` counts the fill from when it is queued, before its first poll. async fn serve_fill( self, @@ -5472,23 +5482,46 @@ impl Subscriber { name: String, fill: track::Fill, timescale: Later>, + heading: Option, ) { let fill = std::sync::Arc::new(fill); let mut last = None; - while let Some(run) = fill.next_run() { - let run = match fill.live_group() { - Some(live) if run.contains(&live) => { - let mut pieces = [run.start..live, live..live + 1, live + 1..run.end] - .into_iter() - .filter(|piece| !piece.is_empty()); - let piece = match fill.order() { - track::Order::Ascending => pieces.next(), - track::Order::Descending => pieces.next_back(), - }; - piece.expect("the run holds the live group") - } - _ => run, + loop { + // The groups around a pending head go first, so it holds back only its own group. + let pending = match heading.as_ref().and_then(|heading| heading.pending(&self.state)) { + Some(Head::Group(group)) => fill.next_run_except(group), + _ => None, + }; + let Some(run) = pending.or_else(|| fill.next_run()) else { + break; }; + let mut run = match fill.live_group() { + Some(live) => first_piece(run, live, fill.order()), + None => run, + }; + if let Some(heading) = &heading + && let Some(head) = heading.pending(&self.state) + { + if let Head::Group(group) = head { + run = first_piece(run, group, fill.order()); + } + if head.overlaps(&run) { + let waited = kio::wait(|waiter| { + if let Poll::Ready(err) = fill.poll_aborted(waiter) { + return Poll::Ready(Err(err)); + } + if fill.poll_unwanted_groups(run.clone(), waiter).is_ready() { + return Poll::Ready(Ok(())); + } + heading.poll_changed(&self.state, head, waiter).map(Ok) + }) + .await; + if let Err(err) = waited { + fill.reject_groups(run, err); + } + continue; + } + } // Every fetch answers its run, so the same one coming back is a bug; ending the // fill names it dropped rather than fetching it forever. if last.as_ref() == Some(&run) { @@ -6183,6 +6216,79 @@ fn poll_headed(fill: &kio::Producer, waiter: &kio::Waiter) -> Poll<()> { } } +/// A subscription's own fill, which may still be bringing the head of the group it joined. +/// It is looked up on each read, since a widening replaces the fill with its own. +struct Heading { + request_id: RequestId, +} + +/// The group a subscription's fill may still bring a head for. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Head { + /// A widening's fill awaits the answer that names its group, so it could be any. + Unnamed, + /// The group the subscription joined, whose head is on its way. + Group(u64), +} + +impl Head { + fn overlaps(self, run: &std::ops::Range) -> bool { + match self { + Head::Unnamed => true, + Head::Group(group) => run.contains(&group), + } + } +} + +impl Heading { + /// The group whose head the fill may still bring, or `None` once the head landed or + /// none is coming. + fn pending(&self, state: &Lock) -> Option { + self.read(state, None) + } + + /// Ready once [`Self::pending`] no longer answers `head`. + fn poll_changed(&self, state: &Lock, head: Head, waiter: &kio::Waiter) -> Poll<()> { + match self.read(state, Some(waiter)) == Some(head) { + true => Poll::Pending, + false => Poll::Ready(()), + } + } + + /// [`Self::pending`], registering `waiter` on the fill read before reading it, so a change + /// after the read still wakes it. The group only changes with the fill's state, and a + /// widening cancels the fill it replaces, which wakes a waiter on that one. + fn read(&self, state: &Lock, waiter: Option<&kio::Waiter>) -> Option { + // The state lock first, as SUBSCRIBE_OK takes it to name the group and serve the fill. + // Holding it across the read keeps a widening from swapping the fill in between. + let state = state.lock(); + let held = state.subscribes.get(&self.request_id)?; + if let Some(waiter) = waiter { + let _ = held.fill.poll_ref(waiter, |_| Poll::<()>::Pending); + } + match *held.fill.read() { + Fill::Requested => Some(Head::Unnamed), + Fill::Serving(_) | Fill::Active => held.fill_largest.map(Head::Group), + Fill::Ready { .. } | Fill::Done | Fill::Failed { .. } | Fill::Ended => None, + } + } +} + +/// The first piece of `run` in `order` once `group` is split off into a run of its own. +fn first_piece(run: std::ops::Range, group: u64, order: track::Order) -> std::ops::Range { + if !run.contains(&group) { + return run; + } + let mut pieces = [run.start..group, group..group + 1, group + 1..run.end] + .into_iter() + .filter(|piece| !piece.is_empty()); + let piece = match order { + track::Order::Ascending => pieces.next(), + track::Order::Descending => pieces.next_back(), + }; + piece.expect("the run holds the group") +} + /// A Normal object's timestamp: none on an untimed track, and required on a timed one, /// where an object without a Timestamp is malformed. fn object_time( @@ -15629,6 +15735,149 @@ mod joining_fetch_tests { } } + /// A reader missing the head of the group the subscription joined waits for the + /// subscription's own fill, which is bringing that head. A cache-miss FETCH racing it + /// could cache the group first, refusing the tail that ends it for live readers. Only a + /// fill that settled without the head leaves the miss to a FETCH. + #[kio::test] + async fn a_miss_waits_for_the_subscriptions_own_fill() { + let version = Version::Draft22; + let largest = Some(ietf::Location { group: 4, object: 1 }); + let run = JoinRun::start(version, None, subscribe_ok(version, largest), fetch_ok(version)).await; + let fetches = || { + decode_messages(&run.session.log, version) + .iter() + .filter(|(id, _)| *id == ietf::Fetch::ID) + .count() + }; + assert!(run.fill_outstanding(), "the subscription asked for a fill"); + + let mut fetch = Box::pin(run._hold.1.fetch_group(4, None)); + assert!(futures::poll!(fetch.as_mut()).is_pending()); + settle().await; + assert_eq!(fetches(), 0, "the miss waits for the subscription's fill"); + + // The fill fails, so the head is the miss's to fetch after all. + let fill = run + .subscriber + .state + .lock() + .subscribes + .values() + .next() + .unwrap() + .fill + .clone(); + *fill.write().ok().unwrap() = Fill::Done; + settle().await; + assert_eq!(fetches(), 1, "a settled fill leaves the miss to a FETCH"); + } + + /// The head landing ends the wait, and the cache answers the miss: no FETCH duplicates + /// the group the fill wrote. + #[kio::test] + async fn a_miss_is_answered_by_the_head_its_fill_lands() { + let version = Version::Draft22; + let largest = Some(ietf::Location { group: 4, object: 1 }); + let run = JoinRun::start(version, None, subscribe_ok(version, largest), fetch_ok(version)).await; + let fetches = || { + decode_messages(&run.session.log, version) + .iter() + .filter(|(id, _)| *id == ietf::Fetch::ID) + .count() + }; + + let mut fetch = Box::pin(run._hold.1.fetch_group(4, None)); + assert!(futures::poll!(fetch.as_mut()).is_pending()); + settle().await; + assert_eq!(fetches(), 0, "the miss waits for the subscription's fill"); + + // The fill writes the head, as `run_fill` does. + let (mut track, fill) = { + let state = run.subscriber.state.lock(); + let held = state.subscribes.values().next().unwrap(); + (held.producer.clone().unwrap(), held.fill.clone()) + }; + let mut head = None; + open_fill_group(&mut track, &mut head, 4).unwrap(); + let (sequence, next, producer) = head.unwrap(); + *fill.write().ok().unwrap() = Fill::Ready { + sequence, + next, + producer: producer.into_inner(), + }; + settle().await; + assert_eq!(fetches(), 0, "the cache answers the miss"); + let Poll::Ready(Ok(group)) = futures::poll!(fetch.as_mut()) else { + panic!("the miss must resolve from the cache"); + }; + assert_eq!(group.sequence, 4); + } + + /// Only the joined group waits for the subscription's fill. A head that never arrives + /// leaves its fill outstanding for as long as the subscription lasts, which must not hold + /// back older groups it brings nothing for. + #[kio::test] + async fn a_miss_for_older_history_skips_the_fill_wait() { + let version = Version::Draft22; + let largest = Some(ietf::Location { group: 4, object: 1 }); + let run = JoinRun::start(version, None, subscribe_ok(version, largest), fetch_ok(version)).await; + let fetches = || { + decode_messages(&run.session.log, version) + .iter() + .filter(|(id, _)| *id == ietf::Fetch::ID) + .count() + }; + assert!(run.fill_outstanding(), "the subscription asked for a fill"); + + let mut fetch = Box::pin(run._hold.1.fetch_group(3, None)); + assert!(futures::poll!(fetch.as_mut()).is_pending()); + settle().await; + assert_eq!(fetches(), 1, "group 3 is fetched without waiting for group 4's head"); + } + + /// A widening replaces the subscription's fill with its own, so a miss follows the new + /// one: until its answer names the group it heads, every miss waits for it. + #[kio::test] + async fn a_miss_follows_the_fill_a_widening_requested() { + let version = Version::Draft22; + let largest = Some(ietf::Location { group: 4, object: 1 }); + let run = JoinRun::start(version, None, subscribe_ok(version, largest), fetch_ok(version)).await; + let fetches = || { + decode_messages(&run.session.log, version) + .iter() + .filter(|(id, _)| *id == ietf::Fetch::ID) + .count() + }; + + // As the update arm does: cancel the old fill and request a new one in its place. + { + let mut state = run.subscriber.state.lock(); + let held = state.subscribes.values_mut().next().unwrap(); + held.fill.write().ok().unwrap().cancel(); + held.fill = kio::Producer::new(Fill::Requested); + } + + let mut fetch = Box::pin(run._hold.1.fetch_group(3, None)); + assert!(futures::poll!(fetch.as_mut()).is_pending()); + settle().await; + assert_eq!(fetches(), 0, "the miss waits for the widening's fill"); + + let fill = run + .subscriber + .state + .lock() + .subscribes + .values() + .next() + .unwrap() + .fill + .clone(); + *fill.write().ok().unwrap() = Fill::Done; + settle().await; + assert_eq!(fetches(), 1, "a settled fill leaves the miss to a FETCH"); + } + /// A malformed answer to the joining FETCH is not a refusal: the peer broke the /// protocol, so the session closes. #[kio::test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index b5ee4fdf2ac..c1f6b13ae63 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -21592,6 +21592,41 @@ mod test { ); } + /// A run that passes over a group another request is bringing serves the groups around + /// it, across the runs that answered groups split the fill into, and leaves that group + /// to [`Fill::next_run`] once nothing else is owed. + #[test] + #[cfg(feature = "async")] + fn a_run_passes_over_a_skipped_group() { + let mut producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + let mut descending = producer.subscribe(replay().with_range(0..)); + producer.start_at(6).unwrap(); + write_group(&producer, 6); + let _ = descending.poll_recv_group(&kio::Waiter::noop()); + let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&kio::Waiter::noop()) else { + panic!("the miss below the start is asked"); + }; + assert_eq!(fill.order(), Order::Descending); + assert_eq!(fill.next_run(), Some(0..6)); + assert_eq!(fill.next_run_except(5), Some(0..5)); + assert_eq!(fill.next_run_except(2), Some(3..6)); + + fill.drop_groups(3..4); + assert_eq!(fill.next_run(), Some(4..6)); + assert_eq!(fill.next_run_except(5), Some(4..5)); + assert_eq!(fill.next_run_except(4), Some(5..6)); + fill.drop_groups(4..5); + assert_eq!( + fill.next_run_except(5), + Some(0..3), + "older runs are served while 5 waits" + ); + fill.drop_groups(0..3); + assert_eq!(fill.next_run_except(5), None); + assert_eq!(fill.next_run(), Some(5..6)); + } + /// A reader changing its priority, budget, or order moves the fill it waits on, queued /// or taken, so the handler serves the rest under the new values, even ones lower than /// it was asked with. diff --git a/rs/moq-net/src/model/track/fill.rs b/rs/moq-net/src/model/track/fill.rs index f582f0f4a18..66ee59971cc 100644 --- a/rs/moq-net/src/model/track/fill.rs +++ b/rs/moq-net/src/model/track/fill.rs @@ -646,6 +646,15 @@ impl FillState { } } +/// The pieces of `gap` once `skip` is taken out of it. +fn without(gap: ops::Range, skip: Option) -> impl Iterator> { + let pieces = match skip { + Some(skip) if gap.contains(&skip) => [gap.start..skip, skip + 1..gap.end], + _ => [gap, 0..0], + }; + pieces.into_iter().filter(|piece| !piece.is_empty()) +} + /// Stop routing readers wanting `groups` to fill `id`: they were answered. Reads first, so /// an answer that changes nothing wakes no handler parked on the queue. fn uncover(fills: &kio::Shared, id: u64, groups: ops::Range) { @@ -866,6 +875,11 @@ impl Fill { /// neither answered nor delivered live since it was asked. `None` once none is owed or /// no reader waits on the fill, so stop serving it. pub fn wanted(&self) -> Option { + self.wanted_except(None) + } + + /// [`Self::wanted`], passing over `skip`. + fn wanted_except(&self, skip: Option) -> Option { if !self.inner.result.wanted() { return None; } @@ -873,7 +887,10 @@ impl Fill { let wanted = self.inner.wanted(); let sequence = { let outcome = self.inner.result.read(); - let mut gaps = wanted.iter().flat_map(|run| outcome.resolved.gaps(run)); + let mut gaps = wanted + .iter() + .flat_map(|run| outcome.resolved.gaps(run)) + .flat_map(|gap| without(gap, skip)); match self.order() { Order::Ascending => gaps.next().map(|gap| gap.start), Order::Descending => gaps.last().map(|gap| gap.end - 1), @@ -940,10 +957,29 @@ impl Fill { #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub(crate) fn next_run(&self) -> Option> { let sequence = self.wanted()?; + Some(self.run_from(sequence, self.inner.owed_around(sequence))) + } + + /// [`Self::next_run`], passing over `skip`: a group another request is already bringing, + /// so the groups around it are served while it waits. `None` once nothing else is owed. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn next_run_except(&self, skip: u64) -> Option> { + let sequence = self.wanted_except(Some(skip))?; let owed = self.inner.owed_around(sequence); + let owed = match skip < sequence { + true => owed.start.max(skip + 1)..owed.end, + false => owed.start..owed.end.min(skip), + }; + Some(self.run_from(sequence, owed)) + } + + /// The run from `sequence` over the `owed` groups beside it, in the fill's order, up to + /// the nearest one the cache holds. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + fn run_from(&self, sequence: u64, owed: ops::Range) -> ops::Range { let state = self.inner.state.read(); let held = |(_, slot): &(&u64, &super::Slot)| !slot.is_aborted(); - Some(match self.order() { + match self.order() { Order::Ascending => { let end = state .lookup @@ -961,7 +997,7 @@ impl Fill { .map_or(owed.start, |(held, _)| held + 1); start..sequence + 1 } - }) + } } /// The newest group the track's live feed produced, while a feed still writes the track: