diff --git a/quest/m2/README.md b/quest/m2/README.md
index 2ff204b9437..b0f73146035 100644
--- a/quest/m2/README.md
+++ b/quest/m2/README.md
@@ -27,7 +27,7 @@ removal, a rename, or a wire, catalog, or format change that is free until
gains its reader, reversing 2026-11-14; and
[lite-07 by start](/quest/m2/time-addressing/lite07.md) carries the
smaller wire fixes. [Refresh](/quest/m2/refresh-interval.md) and
- [group drops](/quest/m2/group-drops.md) took simpler contracts.
+ group drops took simpler contracts.
[moq-transport time addressing](/quest/m2/ietf-time-addressing.md)
returned from m4, and the two moq-transport interop quests in m3 were
deleted. Kept as they are: datagram tracks with their lite-05/06 bridge,
@@ -322,7 +322,7 @@ what would need a breaking release had it landed right after the cut:
the routes epic and gates the cut, per the 2026-10-13 rule that routes'
tier and partition tests run before it (maintainer; rejected: moving it to
m3 and fixing late routing flaws in lite-08).
-- ✅ [Group drops](/quest/m2/group-drops.md) split: its lite-07 wire question
+- ✅ Group drops split: its lite-07 wire question
was its own quest, settled without a wire-format change in
[moq-dev/moq#8774](https://github.com/moq-dev/moq/pull/8774); the public
`Ordered` API waits for the model.
diff --git a/quest/m2/group-drops.md b/quest/m2/group-drops.md
deleted file mode 100644
index 69d44aa0fe5..00000000000
--- a/quest/m2/group-drops.md
+++ /dev/null
@@ -1,83 +0,0 @@
-# [S] A group a subscriber gives up on is always reported
-
-## Goal
-
-Every group a moq-lite subscriber gives up on surfaces to the application.
-In the API, `track::Subscriber`'s drop signal (today the crate-private
-`poll_drop`, which the ordered reader and resume use to skip gaps) is public,
-so the receive-windows benchmark and applications can count losses instead of
-waiting forever, and that benchmark's overcommitted cells finish with
-`delivered_frames` plus the reported drops accounting for every group sent.
-
-## Plan
-
-Planned 2026-11-04 with the maintainer, from
-[moq-dev/moq#7838](https://github.com/moq-dev/moq/pull/7838): no public API
-reports a group the subscriber dropped without `recv_group` returning it, so a
-reader that paces on delivered-or-evicted groups stalls. Split 2026-11-14 (the
-m2 triage before the cut, under /quest-afk, for the maintainer's review):
-the wire half made every abandoned group known and decided lite-07's part;
-this publishes it. It stays in m2 because the `Ordered` change below is
-breaking.
-
-The wire half ([moq-dev/moq#8774](https://github.com/moq-dev/moq/pull/8774))
-found every group the overcommitted cells lose already in the drop set: a
-scratch reader polling `poll_drop` beside `recv_group` finished every cell of
-the receive-windows benchmark with delivered, evicted-after-header and dropped
-groups summing to those sent, on lite-06 and lite-07. So surfacing
-`poll_drop` alone unskips those cells.
-
-- ✅ **Milestone** (maintainer, 2026-11-04): it changes public API, so it
- lands before the cut.
-- **After the drop set.** Drop set finished in
- [moq-dev/moq#8476](https://github.com/moq-dev/moq/pull/8476) on 2026-11-12.
- `poll_drop` now takes no `Delivered` argument and yields the given-up run
- with its `Hole` (`rs/moq-net/src/model/track.rs`, `track/hole.rs`). Publish
- the API against that record. The drop set owns the per-cursor drop runs and
- their reasons; this reports what a cursor gives up.
-- Mirror it in `@moq/net` and the bindings only for a named consumer (for
- example a player or container reader that skips gaps), per the API audit
- rule that new surface needs one; name it in the PR or leave the mirror out.
-
-Decided 2026-11-07 (maintainer, /quest-plan from the API sweep of open PRs and
-remaining quests): ✅ key the drop signal by group start and land after
-[time addressing's model](https://github.com/moq-dev/moq/blob/7d1bee79cb8fcb4d816fc53f0547ef8b2b19756a/quest/m2/time-addressing/model.md), so its public
-API never ships sequence-keyed (rejected: landing first and rekeying with
-the model). ✅ Drops surface as a `Gap { range, error }` item on
-`track::Ordered`, which already skips gaps through `poll_drop`, mirroring
-the container consumer's `Read::Gap` (`rs/moq-mux/src/container/consumer.rs`);
-`recv_group` is unchanged (rejected: a separate `recv_drop` read; a drop
-variant on `recv_group`).
-
-Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): ✅ a
-drop is a span and nothing more: SUBSCRIBE_DROP carries no group count on
-lite-07, and `Gap` reports its start-keyed `range` with no count (rejected:
-a count on SUBSCRIBE_DROP that `Gap` passes on, which would also have let a
-skip become a drop with a count of 0). This settles the question left open
-here on 2026-11-07, against its recommendation. Once starts are sparse a
-span's length is not a number of groups, and only the node that dropped
-them could count; a relay's downstream would need the count carried on the
-wire at every hop for a number few readers use. A reader that needs a count
-measures what it lost in time, which the span gives, or counts the groups
-it did receive. So "count losses" in the Goal means account for every span:
-the receive-windows benchmark's overcommitted cells finish when delivered
-groups plus reported spans cover everything sent. At HEAD
-`SubscribeDrop`'s `Group Count` (`rs/moq-net/src/lite/subscribe.rs`) is a
-count of keys, which [the wire rewrite](/quest/m2/time-addressing/lite07.md)
-turns into the span's end.
-
-Public API: breaking for `track::Ordered` only. Its one read,
-`poll_next_group` (and its async wrapper), returns
-`Result>` instead of `Result >`, where
-`Item` is a group or a `Gap { range, error }`; `Ok(None)` still ends the
-read and `Err` still carries a track abort.
-Callers match the new item, and a caller that ignores drops takes the group
-arm; the PR's MIGRATE notes say so. `recv_group` and the unordered cursor are
-unchanged, so a reader that needs drop accounting reads through `Ordered`.
-`@moq/net` and the bindings mirror it only with a named consumer.
-Wire: none.
-
-## Related
-
-- [The lite-07 wire addresses groups by start](/quest/m2/time-addressing/lite07.md) - SUBSCRIBE_DROP names the time spans `Gap` reports
-- [The receive-windows benchmark](/quest/m3/receive-windows-evicted.md) - its overcommitted cells wait on this (skipped by [moq-dev/moq#7838](https://github.com/moq-dev/moq/pull/7838) until it lands)
diff --git a/quest/m2/lite07-finalize.md b/quest/m2/lite07-finalize.md
index e2802592593..53436fe4919 100644
--- a/quest/m2/lite07-finalize.md
+++ b/quest/m2/lite07-finalize.md
@@ -149,7 +149,7 @@ would need a breaking release had it landed right after the cut:
waits on nixpkgs as well as the cut; dropping the check, which would find a
transcript flaw only after the freeze.
- ✅ The wire half of group drops split out as its own quest, which this
- waited on in place of [the public drop signal](/quest/m2/group-drops.md),
+ waited on in place of the public drop signal,
an API change that doesn't touch the wire. It settled without a
wire-format change ([moq-dev/moq#8774](https://github.com/moq-dev/moq/pull/8774)):
both publishers already name a Group Stream the subscriber stopped in
diff --git a/quest/m2/time-addressing/lite07.md b/quest/m2/time-addressing/lite07.md
index 639f29e9acb..a568d7ad9de 100644
--- a/quest/m2/time-addressing/lite07.md
+++ b/quest/m2/time-addressing/lite07.md
@@ -138,7 +138,7 @@ is free only until lite-07 freezes:
- ✅ SUBSCRIBE_DROP names a span and carries no group count (rejected: a
count, which would also have let SUBSCRIBE_SKIP become a drop with a
count of 0). Today's `Group Count` is a count of keys, so it becomes the
- span's end. [Group drops](/quest/m2/group-drops.md) reports the span.
+ span's end. `track::Ordered`'s `Gap` reports the span.
Recommendations for the same PR, not decided:
@@ -188,5 +188,4 @@ TRACKS_ADD change, and the Padding stream and datagram types go.
## Related
- [YIELDED deferred](/quest/m2/yield-defer.md) - removes continuations, the last use of GROUP's `Frame Start`
-- [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - reads the spans SUBSCRIBE_DROP names
- [Refresh a track](/quest/m2/refresh-interval.md) - raises the `Wanted Group` this turns into a start time
diff --git a/quest/m2/unused-api/net.md b/quest/m2/unused-api/net.md
index 6ab781f3d9b..55848253cc4 100644
--- a/quest/m2/unused-api/net.md
+++ b/quest/m2/unused-api/net.md
@@ -62,8 +62,8 @@ Not in this sweep, with their owners:
- `Subscriber::next_in_range`, `is_skipped`, and `poll_start` have one
consumer, moq-mux's container reader, which orders groups itself.
- [Group drops](/quest/m2/group-drops.md) adds `Gap` to `track::Ordered`;
- once the container reader moves onto it these three can go private.
+ `track::Ordered` now reports a `Gap`; once the container reader moves onto it
+ these three can go private.
- `socket::is_endpoint` leaves with
[socket endpoints](/quest/m2/sockets/remove.md).
- `origin::Producer::update_hold`, `with_proxy`, `Source`, `source()`,
@@ -82,4 +82,3 @@ Public API: breaking, removals only (`Position::after`, `after_group`,
## Related
- [moq-net's node, version, handshake, and route priority](/quest/m2/net-renames.md) - makes `Version` opaque
-- [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - gives the container reader the `Gap` that frees three more methods
diff --git a/quest/m3/receive-windows-evicted.md b/quest/m3/receive-windows-evicted.md
index aadefaebfa9..75ccededee1 100644
--- a/quest/m3/receive-windows-evicted.md
+++ b/quest/m3/receive-windows-evicted.md
@@ -8,7 +8,7 @@ overcommitted cells again, and Nightly's `Rust benchmark tests` job
(`gh nightly rust-test`) still passes. Those cells are 256 streams,
4 MiB frames, and 4 MiB frames with a following stream window. Until
then, the benchmark skips them and prints each one with
-`quest/m2/group-drops.md` as the reason.
+`quest/m3/receive-windows-evicted.md` as the reason.
Every cell keeps the workload it offers. A cell whose incomplete units
fit the connection window fails on any eviction. A cell that
@@ -24,8 +24,8 @@ pulling the m2 blocker into m1). It ranks right after m2, when it can start.
Moved to m1 2026-11-04 with the maintainer, after
[moq-dev/moq#7838](https://github.com/moq-dev/moq/pull/7838) landed the
-fitting cells: what remains waits on
-[group-drops](/quest/m2/group-drops.md), an m2 quest, and with the
+fitting cells: what remained waited on
+`quest/m2/group-drops.md`, an m2 quest, and with the
overcommitted cells skipped the benchmark no longer fails Nightly, so this
no longer holds the m0
[Nightly passes on main](/quest/m0/nightly/README.md) epic open. It replaces
@@ -42,7 +42,7 @@ Overload policy, decided 2026-11-04 (maintainer):
Silent loss, decided 2026-11-04 (maintainer):
-- ✅ Fix it at the source in `quest/m2/group-drops.md`, planned in
+- ✅ Fix it at the source in `quest/m2/group-drops.md` (landed), planned in
[moq-dev/moq#7890](https://github.com/moq-dev/moq/pull/7890). There,
the wire always names an abandoned group, and `track::Subscriber`'s
drop signal becomes public.
@@ -84,10 +84,18 @@ What remains:
(traced as `fetch abandoned`). Such a group's in-flight permit never
returns, so the writer stalls and the reader times out. The 4 MiB-frame
cell hit this on every run, and the 256-stream cell intermittently.
-- Once group-drops lands, the reader returns a dropped group's permit and
- counts the group as lost. Then remove the skip (`OVERCOMMITTED` and the
- `Cell::fits` partition in `benchmark_receive_windows`), and run the full
- sweep through `quest gate`.
+- The drop signal landed in `quest/m2/group-drops.md` as `track::Item::Gap`
+ on `track::Ordered`, which the reader must read through: its subscription
+ anchors at group 0 (`Anchor::Time(0)`), and `next_group` returns each
+ group or the span given up. The reader returns a permit per group in a
+ span (the writer's `sent` map holds the keys) and counts them lost. A
+ scratch run found two things for this PR to settle: `Ordered` reports a
+ group landing after a higher one was returned as `Old`, so a fitting cell
+ loses reordered frames with no eviction and `measure`'s "lost a frame
+ without an eviction" check must count `Old` gaps; and the default cell
+ runs only once the reader is anchored at group 0. Then remove the skip
+ (`OVERCOMMITTED` and the `Cell::fits` partition in
+ `benchmark_receive_windows`), and run the full sweep through `quest gate`.
- Retain the stream/frame/window axes and the actual offered workload
for a shared receive budget, if one is planned again.
- Repro: `just rs test -p moq-tokio --test integration --release -E
@@ -101,7 +109,3 @@ connection-wide admission semaphore are recorded in
[moq-dev/moq#7838](https://github.com/moq-dev/moq/pull/7838).
Public API: none. Wire: none.
-
-## Required
-
-- [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - the public drop signal the reader needs to return a dropped group's permit
diff --git a/rs/moq-archive/src/consumer/tests.rs b/rs/moq-archive/src/consumer/tests.rs
index 03de452f01b..2078ab237b8 100644
--- a/rs/moq-archive/src/consumer/tests.rs
+++ b/rs/moq-archive/src/consumer/tests.rs
@@ -144,21 +144,27 @@ fn from(start: u64) -> track::Range {
}
}
+/// The next group, past the gaps the read reports for what the archive never held.
+async fn group(reader: &mut track::Ordered) -> Option {
+ loop {
+ let item = timeout(Duration::from_secs(10), reader.next_group())
+ .await
+ .expect("a group arrives")
+ .unwrap()?;
+ if let track::Item::Group(group) = item {
+ return Some(group);
+ }
+ }
+}
+
/// The next group's start, without waiting for it to end.
async fn sequence(reader: &mut track::Ordered) -> Option {
- timeout(Duration::from_secs(10), reader.next_group())
- .await
- .expect("a group arrives")
- .unwrap()
- .map(|group| group.start)
+ group(reader).await.map(|group| group.start)
}
/// The next group and every frame in it, once the group ends.
async fn next(reader: &mut track::Ordered) -> Option<(u64, Vec)> {
- let mut group = timeout(Duration::from_secs(10), reader.next_group())
- .await
- .expect("a group arrives")
- .unwrap()?;
+ let mut group = group(reader).await?;
let mut frames = Vec::new();
while let Some(frame) = timeout(Duration::from_secs(10), group.read_frame())
.await
@@ -226,7 +232,7 @@ async fn a_recording_round_trips() {
let mut reader = subscribe(&broadcast, "video").await;
for sequence in 0..6 {
- let mut group = reader.next_group().await.unwrap().unwrap();
+ let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert_eq!(group.start, sequence);
for index in 0..3 {
let frame = group.read_frame().await.unwrap().unwrap();
@@ -450,7 +456,7 @@ async fn an_archive_without_markers_stays_open() {
let (broadcast, _run) = serve(&archive.store).await;
let mut reader = subscribe(&broadcast, "video").await;
assert_eq!(next(&mut reader).await.unwrap(), (0, expected(true, 0, 0..1)));
- let mut group = reader.next_group().await.unwrap().unwrap();
+ let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert!(group.read_frame().await.unwrap().is_some());
let pending = timeout(30 * REFRESH, group.read_frame()).await;
assert!(pending.is_err(), "the newest group stays open");
@@ -466,7 +472,7 @@ async fn closing_the_broadcast_cancels_the_live_tracks() {
let (broadcast, run) = serve(&archive.store).await;
let mut reader = subscribe(&broadcast, "video").await;
- let mut group = reader.next_group().await.unwrap().unwrap();
+ let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert!(group.read_frame().await.unwrap().is_some());
broadcast.close();
@@ -485,7 +491,7 @@ async fn a_gapped_continuation_aborts_the_open_group() {
let (broadcast, run) = serve(&archive.store).await;
let mut reader = subscribe(&broadcast, "video").await;
- let mut group = reader.next_group().await.unwrap().unwrap();
+ let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert!(group.read_frame().await.unwrap().is_some());
// Frame 1 of group 0 is missing.
@@ -508,7 +514,7 @@ async fn a_store_failure_aborts_the_served_tracks() {
let (broadcast, run) = serve(&archive.store).await;
let mut reader = subscribe(&broadcast, "video").await;
- let mut group = reader.next_group().await.unwrap().unwrap();
+ let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert!(group.read_frame().await.unwrap().is_some());
archive.store.inner().fail_lists();
@@ -564,7 +570,7 @@ async fn a_completed_archive_ends_as_recorded() {
let end = timeout(Duration::from_secs(10), reader.next_group())
.await
.expect("the track ends");
- let end = end.map(|group| group.map(|group| group.start));
+ let end = end.map(|item| item.map(|item| item.group().expect("a group").start));
assert!(
matches!(end, Err(moq_net::Error::Stream(moq_net::StreamError::App(7)))),
"{end:?}"
@@ -697,7 +703,9 @@ async fn read_range(broadcast: &broadcast::Producer, track: &str, range: track::
let mut read = Vec::new();
while let Ok(group) = timeout(5 * REFRESH, reader.next_group()).await {
match group.unwrap() {
- Some(group) => read.push(group.start),
+ Some(track::Item::Group(group)) => read.push(group.start),
+ // Nothing is stored below the first group, which the read reports as a gap.
+ Some(track::Item::Gap { .. }) => {}
None => break,
}
}
diff --git a/rs/moq-archive/src/producer/track.rs b/rs/moq-archive/src/producer/track.rs
index 204d0a362ce..722c9f23421 100644
--- a/rs/moq-archive/src/producer/track.rs
+++ b/rs/moq-archive/src/producer/track.rs
@@ -8,7 +8,7 @@ use std::time::Duration;
use bytes::Bytes;
use kio::clock::Instant;
-use moq_net::track::{Anchor, Delay, Order, Position, Range, Subscription};
+use moq_net::track::{Anchor, Delay, Item, Order, Position, Range, Subscription};
use moq_net::{StreamError, Timescale, group};
use object_store::path::Path;
use object_store::{ObjectStore, ObjectStoreExt, PutMode, PutPayload};
@@ -218,8 +218,12 @@ impl Track {
}
}
None => {
- if let Poll::Ready(next) = ordered.poll_next_group(waiter) {
- return Poll::Ready(Event::Next(next));
+ // A group the track gave up on is stored as the hole it leaves.
+ while let Poll::Ready(next) = ordered.poll_next_group(waiter) {
+ match next {
+ Ok(Some(Item::Gap { .. })) => continue,
+ next => return Poll::Ready(Event::Next(next.map(|item| item.and_then(Item::group)))),
+ }
}
}
}
diff --git a/rs/moq-bench/src/connection.rs b/rs/moq-bench/src/connection.rs
index 358c4ac6ab2..c9a79cf184f 100644
--- a/rs/moq-bench/src/connection.rs
+++ b/rs/moq-bench/src/connection.rs
@@ -896,7 +896,12 @@ mod tests {
.await
.unwrap()
.ordered();
- let mut group = sub.next_group().await.unwrap().expect("a group");
+ let mut group = sub
+ .next_group()
+ .await
+ .unwrap()
+ .and_then(moq_net::track::Item::group)
+ .expect("a group");
let keyframe = group.read_frame().await.unwrap().expect("keyframe");
let header: serde_json::Value = serde_json::from_slice(&keyframe.payload).unwrap();
@@ -943,7 +948,12 @@ mod tests {
.await
.unwrap()
.ordered();
- let mut group = sub.next_group().await.unwrap().expect("a group");
+ let mut group = sub
+ .next_group()
+ .await
+ .unwrap()
+ .and_then(moq_net::track::Item::group)
+ .expect("a group");
// Just the keyframe, then the group ends.
assert!(group.read_frame().await.unwrap().is_some(), "keyframe");
@@ -975,7 +985,12 @@ mod tests {
tokio::time::advance(Duration::from_millis(250)).await;
let mut sub = consumer.track(TRACK).unwrap().subscribe(None).await.unwrap().ordered();
- let mut group = sub.next_group().await.unwrap().expect("a group");
+ let mut group = sub
+ .next_group()
+ .await
+ .unwrap()
+ .and_then(moq_net::track::Item::group)
+ .expect("a group");
let keyframe = group.read_frame().await.unwrap().expect("keyframe").payload;
assert_eq!(keyframe.len(), 300, "keyframe padded to the rolled frame size");
@@ -1008,7 +1023,12 @@ mod tests {
tokio::time::advance(Duration::from_millis(250)).await;
let mut sub = consumer.track(TRACK).unwrap().subscribe(None).await.unwrap().ordered();
- let mut group = sub.next_group().await.unwrap().expect("a group");
+ let mut group = sub
+ .next_group()
+ .await
+ .unwrap()
+ .and_then(moq_net::track::Item::group)
+ .expect("a group");
let keyframe = group.read_frame().await.unwrap().expect("keyframe").payload;
assert!(keyframe.len() > 50, "header is the floor; no truncation to fit");
diff --git a/rs/moq-c/src/consume.rs b/rs/moq-c/src/consume.rs
index 73c62840e97..8960779052b 100644
--- a/rs/moq-c/src/consume.rs
+++ b/rs/moq-c/src/consume.rs
@@ -878,9 +878,13 @@ impl Consume {
if Pin::new(&mut close).poll(&mut waiter.context()).is_ready() {
return Poll::Ready(Ok(RawStep::Stop));
}
- match ready!(track.poll_next_group(waiter))? {
- Some(group) => Poll::Ready(Ok(RawStep::Item(group))),
- None => Poll::Ready(Ok(RawStep::End)),
+ loop {
+ match ready!(track.poll_next_group(waiter))? {
+ Some(moq_net::track::Item::Group(group)) => return Poll::Ready(Ok(RawStep::Item(group))),
+ // The C API reports no gaps.
+ Some(moq_net::track::Item::Gap { .. }) => continue,
+ None => return Poll::Ready(Ok(RawStep::End)),
+ }
}
})
.await?
diff --git a/rs/moq-cli/src/main.rs b/rs/moq-cli/src/main.rs
index 9d1c55ae6f6..55346288adf 100644
--- a/rs/moq-cli/src/main.rs
+++ b/rs/moq-cli/src/main.rs
@@ -1420,7 +1420,7 @@ mod tests {
.await
.unwrap()
.ordered();
- let mut group = track.next_group().await.unwrap().unwrap();
+ let mut group = track.next_group().await.unwrap().unwrap().group().unwrap();
group.read_frame().await.unwrap().unwrap().payload
})
.await
diff --git a/rs/moq-ffi/src/consumer.rs b/rs/moq-ffi/src/consumer.rs
index 01bab12a2fe..599c7fa99fe 100644
--- a/rs/moq-ffi/src/consumer.rs
+++ b/rs/moq-ffi/src/consumer.rs
@@ -509,7 +509,7 @@ impl TrackInner {
waiter.waker().wake_by_ref();
return Poll::Ready(Ok(Some(group)));
}
- ordered.track.poll_next_group(waiter).map(|r| r.map_err(Into::into))
+ poll_group(&mut ordered.track, waiter).map(|r| r.map_err(Into::into))
}
Err(e) => Poll::Ready(Err(e)),
}
@@ -529,7 +529,7 @@ impl TrackInner {
loop {
let group = match &mut ordered.pending {
Some(group) => group,
- None => match ready!(ordered.track.poll_next_group(waiter)) {
+ None => match ready!(poll_group(&mut ordered.track, waiter)) {
Ok(Some(group)) => ordered.pending.insert(group),
Ok(None) => return Poll::Ready(Ok(None)),
Err(err) => return Poll::Ready(Err(err.into())),
@@ -730,3 +730,18 @@ impl MoqGroupConsumer {
self.reads.cancel();
}
}
+
+/// The next group of an ordered read. The bindings report no gaps, so a group the track
+/// gave up on is passed over.
+fn poll_group(
+ track: &mut moq_net::track::Ordered,
+ waiter: &kio::Waiter,
+) -> Poll, moq_net::Error>> {
+ loop {
+ match ready!(track.poll_next_group(waiter))? {
+ Some(moq_net::track::Item::Group(group)) => return Poll::Ready(Ok(Some(group))),
+ Some(moq_net::track::Item::Gap { .. }) => continue,
+ None => return Poll::Ready(Ok(None)),
+ }
+ }
+}
diff --git a/rs/moq-ffi/src/media.rs b/rs/moq-ffi/src/media.rs
index 21adba3b955..a3f4de2cfc6 100644
--- a/rs/moq-ffi/src/media.rs
+++ b/rs/moq-ffi/src/media.rs
@@ -2244,7 +2244,11 @@ mod timebase_tests {
.await
.unwrap()
.ordered();
- let mut group = catalogs.next_group().await.unwrap().unwrap();
+ let mut group = loop {
+ if let Some(group) = catalogs.next_group().await.unwrap().unwrap().group() {
+ break group;
+ }
+ };
let stamp = group.read_frame().await.unwrap().unwrap().timestamp;
// Catalogs published in the same millisecond, such as a rendition and its first estimate,
// start a unit apart, since each starts its own group.
diff --git a/rs/moq-gst/src/sink/imp.rs b/rs/moq-gst/src/sink/imp.rs
index 489ccdb05d5..fdf10b9a140 100644
--- a/rs/moq-gst/src/sink/imp.rs
+++ b/rs/moq-gst/src/sink/imp.rs
@@ -2076,6 +2076,7 @@ mod tests {
.now_or_never()
.expect("a group is ready")
.unwrap()
+ .and_then(moq_net::track::Item::group)
.expect("the resumed keyframe opens a group");
let frame = group
.read_frame()
diff --git a/rs/moq-gst/src/sink/pad.rs b/rs/moq-gst/src/sink/pad.rs
index 2a8e1b03a2d..54b511b4c63 100644
--- a/rs/moq-gst/src/sink/pad.rs
+++ b/rs/moq-gst/src/sink/pad.rs
@@ -2068,7 +2068,12 @@ mod tests {
.expect("subscribe to the opaque track")
.ordered();
- let mut group = subscriber.next_group().await.unwrap().expect("a first group");
+ let mut group = subscriber
+ .next_group()
+ .await
+ .unwrap()
+ .and_then(moq_net::track::Item::group)
+ .expect("a first group");
let frame = group.read_frame().await.unwrap().expect("a frame in the first group");
assert_eq!(
frame.payload.as_ref(),
@@ -2085,7 +2090,12 @@ mod tests {
"one buffer produces one group with one frame"
);
- let mut group = subscriber.next_group().await.unwrap().expect("a second group");
+ let mut group = subscriber
+ .next_group()
+ .await
+ .unwrap()
+ .and_then(moq_net::track::Item::group)
+ .expect("a second group");
let frame = group.read_frame().await.unwrap().expect("a frame in the second group");
assert_eq!(frame.payload.as_ref(), b"second");
assert_eq!(
@@ -2191,7 +2201,12 @@ mod tests {
.await
.expect("subscribe to the opaque track")
.ordered();
- let mut group = subscriber.next_group().await.unwrap().expect("a group");
+ let mut group = subscriber
+ .next_group()
+ .await
+ .unwrap()
+ .and_then(moq_net::track::Item::group)
+ .expect("a group");
let frame = group.read_frame().await.unwrap().expect("a frame");
assert_eq!(frame.payload.as_ref(), b"no pts", "the unstamped buffer was published");
assert_eq!(
diff --git a/rs/moq-hls/src/export/rendition.rs b/rs/moq-hls/src/export/rendition.rs
index 8ff9afbd49e..d2b39b595aa 100644
--- a/rs/moq-hls/src/export/rendition.rs
+++ b/rs/moq-hls/src/export/rendition.rs
@@ -2416,7 +2416,9 @@ impl WatchMedia {
}
let groups = self.groups.as_mut().expect("subscribed above");
match ready!(groups.poll_next_group(waiter))? {
- Some(group) => self.step = Step::First(group),
+ Some(moq_net::track::Item::Group(group)) => self.step = Step::First(group),
+ // A segment is listed from the groups that arrive, so one given up on is read past.
+ Some(moq_net::track::Item::Gap { .. }) => {}
None => match follow.latest.take() {
Some((group, _, reach)) => self.step = Step::End(group, Some(reach)),
None => {
diff --git a/rs/moq-hls/src/import.rs b/rs/moq-hls/src/import.rs
index 79ddd36b5aa..3bcc2b46844 100644
--- a/rs/moq-hls/src/import.rs
+++ b/rs/moq-hls/src/import.rs
@@ -2405,7 +2405,8 @@ mod tests {
let mut out = Vec::new();
loop {
let mut group = match subscriber.poll_next_group(&kio::Waiter::noop()) {
- Poll::Ready(Ok(Some(group))) => group,
+ Poll::Ready(Ok(Some(moq_net::track::Item::Group(group)))) => group,
+ Poll::Ready(Ok(Some(moq_net::track::Item::Gap { .. }))) => continue,
Poll::Pending | Poll::Ready(Ok(None)) => break,
Poll::Ready(Err(error)) => return (out, Some(error)),
};
diff --git a/rs/moq-json/examples/telemetry.rs b/rs/moq-json/examples/telemetry.rs
index 310085307e7..935d5f0ef26 100644
--- a/rs/moq-json/examples/telemetry.rs
+++ b/rs/moq-json/examples/telemetry.rs
@@ -111,7 +111,10 @@ fn frame_bytes(config: snapshot::Config, ticks: u64) -> usize {
let waiter = kio::Waiter::noop();
let mut total = 0;
let mut track = consumer.ordered();
- while let Poll::Ready(Ok(Some(mut group))) = track.poll_next_group(&waiter) {
+ while let Poll::Ready(Ok(Some(item))) = track.poll_next_group(&waiter) {
+ let moq_net::track::Item::Group(mut group) = item else {
+ continue;
+ };
while let Poll::Ready(Ok(Some(frame))) = group.poll_read_frame(&waiter) {
total += frame.payload.len();
}
diff --git a/rs/moq-json/src/snapshot/consumer.rs b/rs/moq-json/src/snapshot/consumer.rs
index 5df238786b5..12f24f16094 100644
--- a/rs/moq-json/src/snapshot/consumer.rs
+++ b/rs/moq-json/src/snapshot/consumer.rs
@@ -203,9 +203,16 @@ impl Track {
if let Self::Subscribing(pending) = self {
*self = Self::Ordered(Box::new(std::task::ready!(pending.poll_ok(waiter))?.ordered()));
}
- match self {
- Self::Ordered(track) => track.poll_next_group(waiter),
- Self::Subscribing(_) => unreachable!("resolved above"),
+ let Self::Ordered(track) = self else {
+ unreachable!("resolved above")
+ };
+ // A snapshot is the newest group's content, so a group given up on changes nothing.
+ loop {
+ match std::task::ready!(track.poll_next_group(waiter))? {
+ Some(moq_net::track::Item::Group(group)) => return Poll::Ready(Ok(Some(group))),
+ Some(moq_net::track::Item::Gap { .. }) => continue,
+ None => return Poll::Ready(Ok(None)),
+ }
}
}
}
diff --git a/rs/moq-json/src/snapshot/mod.rs b/rs/moq-json/src/snapshot/mod.rs
index 33a2cc9bd5b..9bd2f5e893e 100644
--- a/rs/moq-json/src/snapshot/mod.rs
+++ b/rs/moq-json/src/snapshot/mod.rs
@@ -1073,8 +1073,13 @@ mod test {
));
let waiter = kio::Waiter::noop();
- let Poll::Ready(Ok(Some(mut group))) = subscriber.poll_next_group(&waiter) else {
- panic!("the group was published, so a subscriber sees it");
+ let mut group = loop {
+ let Poll::Ready(Ok(Some(item))) = subscriber.poll_next_group(&waiter) else {
+ panic!("the group was published, so a subscriber sees it");
+ };
+ if let Some(group) = item.group() {
+ break group;
+ }
};
assert!(
matches!(group.poll_read_frame(&waiter), Poll::Ready(Ok(None))),
diff --git a/rs/moq-json/src/window/consumer.rs b/rs/moq-json/src/window/consumer.rs
index f1355071729..f0feb4f82d4 100644
--- a/rs/moq-json/src/window/consumer.rs
+++ b/rs/moq-json/src/window/consumer.rs
@@ -60,11 +60,13 @@ impl Consumer {
let Some(group) = &mut self.group else {
match self.track.poll_next_group(waiter)? {
- Poll::Ready(Some(group)) => {
+ Poll::Ready(Some(moq_net::track::Item::Group(group))) => {
self.codec = Some(Codec::new());
self.group = Some(group);
continue;
}
+ // The window advances by index, so a group given up on is read past.
+ Poll::Ready(Some(moq_net::track::Item::Gap { .. })) => continue,
Poll::Ready(None) => return Poll::Ready(Ok(None)),
Poll::Pending => return Poll::Pending,
}
diff --git a/rs/moq-json/src/window/mod.rs b/rs/moq-json/src/window/mod.rs
index 588cae01b41..530ea035a6b 100644
--- a/rs/moq-json/src/window/mod.rs
+++ b/rs/moq-json/src/window/mod.rs
@@ -877,8 +877,13 @@ mod test {
assert!(producer.window().is_empty());
let waiter = kio::Waiter::noop();
- let Poll::Ready(Ok(Some(mut group))) = subscriber.poll_next_group(&waiter) else {
- panic!("the rejected group's header was published");
+ let mut group = loop {
+ let Poll::Ready(Ok(Some(item))) = subscriber.poll_next_group(&waiter) else {
+ panic!("the rejected group's header was published");
+ };
+ if let Some(group) = item.group() {
+ break group;
+ }
};
assert!(matches!(group.poll_read_frame(&waiter), Poll::Ready(Ok(None))));
}
@@ -938,7 +943,7 @@ mod test {
let _finished = producer.clone().finish().unwrap();
let waiter = kio::Waiter::noop();
- let Poll::Ready(Ok(Some(mut group))) = subscriber.poll_next_group(&waiter) else {
+ let Poll::Ready(Ok(Some(moq_net::track::Item::Group(mut group)))) = subscriber.poll_next_group(&waiter) else {
panic!("expected the window group");
};
while let Poll::Ready(Ok(Some(_))) = group.poll_read_frame(&waiter) {}
diff --git a/rs/moq-mux/src/catalog/msf/consumer.rs b/rs/moq-mux/src/catalog/msf/consumer.rs
index 81ceb084a05..29715ac6b4c 100644
--- a/rs/moq-mux/src/catalog/msf/consumer.rs
+++ b/rs/moq-mux/src/catalog/msf/consumer.rs
@@ -43,7 +43,9 @@ impl Consumer {
// so we can distinguish "more groups may arrive" from "no more groups, ever".
let track_finished = loop {
match self.track.poll_next_group(waiter)? {
- Poll::Ready(Some(group)) => self.group = Some(group),
+ Poll::Ready(Some(moq_net::track::Item::Group(group))) => self.group = Some(group),
+ // Only the newest catalog matters, so a group given up on changes nothing.
+ Poll::Ready(Some(moq_net::track::Item::Gap { .. })) => {}
Poll::Ready(None) => break true,
Poll::Pending => break false,
}
diff --git a/rs/moq-mux/src/catalog/producer.rs b/rs/moq-mux/src/catalog/producer.rs
index e24be062aa8..342942285d4 100644
--- a/rs/moq-mux/src/catalog/producer.rs
+++ b/rs/moq-mux/src/catalog/producer.rs
@@ -1543,7 +1543,7 @@ mod test {
.await
.unwrap()
.ordered();
- let mut group = track.next_group().await.unwrap().unwrap();
+ let mut group = track.next_group().await.unwrap().unwrap().group().unwrap();
let frame = group.read_frame().await.unwrap().unwrap();
let wire: serde_json::Value = serde_json::from_slice(&frame.payload).unwrap();
assert_eq!(wire["tracks"][0]["namespace"], expected);
@@ -2344,7 +2344,11 @@ mod test {
home.block_on(async {
for (name, mut track) in readers {
let (mut stamps, mut starts) = (Vec::new(), Vec::::new());
- while let Some(mut group) = track.next_group().await.unwrap() {
+ while let Some(item) = track.next_group().await.unwrap() {
+ // Nothing exists below the first group, which the read reports as a gap.
+ let moq_net::track::Item::Group(mut group) = item else {
+ continue;
+ };
let first = stamps.len();
while let Some(frame) = group.read_frame().await.unwrap() {
stamps.push(frame.timestamp.as_micros().unwrap());
diff --git a/rs/moq-mux/src/container/fmp4/import_test.rs b/rs/moq-mux/src/container/fmp4/import_test.rs
index a307f645a4e..08dffbec410 100644
--- a/rs/moq-mux/src/container/fmp4/import_test.rs
+++ b/rs/moq-mux/src/container/fmp4/import_test.rs
@@ -1675,8 +1675,11 @@ async fn starts(consumer: &moq_net::broadcast::Consumer, track: &str) -> Vec {
+ Poll::Ready(Ok(Some(moq_net::track::Item::Group(group)))) => {
self.pending = Some((group, Default::default(), Timestamp::ZERO));
}
+ // SI carries the last snapshot over a group given up on.
+ Poll::Ready(Ok(Some(moq_net::track::Item::Gap { .. }))) => {}
Poll::Ready(Ok(None)) => {
ended = true;
break;
diff --git a/rs/moq-mux/src/container/ts/si.rs b/rs/moq-mux/src/container/ts/si.rs
index 22f6e61d653..5c4d75fc4b5 100644
--- a/rs/moq-mux/src/container/ts/si.rs
+++ b/rs/moq-mux/src/container/ts/si.rs
@@ -822,8 +822,11 @@ mod tests {
.ordered();
let waiter = kio::Waiter::noop();
let mut read = Vec::new();
- while let std::task::Poll::Ready(Ok(Some(group))) = reader.poll_next_group(&waiter) {
- read.push(group.start);
+ while let std::task::Poll::Ready(Ok(Some(item))) = reader.poll_next_group(&waiter) {
+ // A group given up on is a gap, which reads no group.
+ if let Some(group) = item.group() {
+ read.push(group.start);
+ }
}
assert_eq!(read, expected);
}
diff --git a/rs/moq-net/benches/group.rs b/rs/moq-net/benches/group.rs
index 05379f0d2d8..b2ced6deae8 100644
--- a/rs/moq-net/benches/group.rs
+++ b/rs/moq-net/benches/group.rs
@@ -319,7 +319,14 @@ fn bench_track_recv(c: &mut Criterion) {
},
|(ctx, mut subscriber)| {
for _ in 0..n {
- let group = subscriber.next_group().now_or_never().unwrap().unwrap().unwrap();
+ let group = subscriber
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap();
black_box(group);
}
(ctx, subscriber)
diff --git a/rs/moq-net/benches/origin.rs b/rs/moq-net/benches/origin.rs
index 0d06852bc0b..b56295aa578 100644
--- a/rs/moq-net/benches/origin.rs
+++ b/rs/moq-net/benches/origin.rs
@@ -1707,7 +1707,13 @@ impl ResumeReader {
fn poll_group(&mut self, waiter: &kio::Waiter) -> Poll>> {
match self {
Self::Arrival(reader) => reader.poll_recv_group(waiter),
- Self::Key(reader) => reader.poll_next_group(waiter),
+ Self::Key(reader) => loop {
+ match std::task::ready!(reader.poll_next_group(waiter))? {
+ Some(track::Item::Group(group)) => return Poll::Ready(Ok(Some(group))),
+ Some(track::Item::Gap { .. }) => continue,
+ None => return Poll::Ready(Ok(None)),
+ }
+ },
}
}
}
diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs
index ab3492e94ec..6445dccbf12 100644
--- a/rs/moq-net/src/ietf/subscriber.rs
+++ b/rs/moq-net/src/ietf/subscriber.rs
@@ -16561,9 +16561,15 @@ mod stitch_tests {
let mut next = h.stream().await;
h.subscriber.clone().recv_group(&mut next).await.expect("next group");
+ // The refused group comes back as a gap, then the next group.
+ let Poll::Ready(Ok(Some(track::Item::Gap { range, .. }))) = ordered.poll_next_group(&kio::Waiter::noop())
+ else {
+ panic!("the refused group is not reported");
+ };
+ assert!(range.contains(&KEY), "{range:?}");
let group = match ordered.poll_next_group(&kio::Waiter::noop()) {
- Poll::Ready(Ok(Some(group))) => group,
- Poll::Ready(other) => panic!("no next group: {:?}", other.map(|group| group.map(|g| g.start))),
+ Poll::Ready(Ok(Some(track::Item::Group(group)))) => group,
+ Poll::Ready(_) => panic!("no next group"),
Poll::Pending => panic!("the ordered reader still waits on the refused group"),
};
assert_eq!(group.start, KEY + 1);
diff --git a/rs/moq-net/src/lite/subscriber/fills.rs b/rs/moq-net/src/lite/subscriber/fills.rs
index 61fea0638dd..c06bd083d14 100644
--- a/rs/moq-net/src/lite/subscriber/fills.rs
+++ b/rs/moq-net/src/lite/subscriber/fills.rs
@@ -867,7 +867,7 @@ mod tests {
group.finish().unwrap();
}
for key in 0..3 {
- let Poll::Ready(Ok(Some(group))) = reader.poll_next_group(&waiter) else {
+ let Poll::Ready(Ok(Some(track::Item::Group(group)))) = reader.poll_next_group(&waiter) else {
panic!("group {key} was delivered");
};
assert_eq!(group.start, key);
diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs
index bedaed5d033..276a5726b61 100644
--- a/rs/moq-net/src/model/resume.rs
+++ b/rs/moq-net/src/model/resume.rs
@@ -922,6 +922,21 @@ impl Subscriber {
self.reader().take(candidate)
}
+ /// The keys that landed on any copy since the last call, below `below`; see
+ /// [`track::Subscriber::note_late`].
+ pub(super) fn late(&mut self, below: u64) -> Vec {
+ let mut reader = self.reader();
+ reader
+ .copies
+ .iter_mut()
+ .filter_map(|copy| match &mut copy.sub {
+ Sub::Ready(sub) => Some(sub.late(below)),
+ Sub::Pending(_) => None,
+ })
+ .flatten()
+ .collect()
+ }
+
/// Forget the gaps in `delivered` below each group past the reader's budget, judged on
/// the newest copy that answered; see [`track::Subscriber::forget_past_budget`].
pub(super) fn forget_past_budget(&self, delivered: &mut Delivered) {
@@ -5454,6 +5469,99 @@ mod test {
}
}
+ /// A front's key-order reader reports a group that lands on its copy after the read passed
+ /// its key, as old, and never one it handed out.
+ #[test]
+ fn a_front_reports_a_late_arrival_as_a_gap() {
+ let routes = Producer::new();
+ let logical = logical(&routes);
+ let a = copy();
+ routes.serve(a.consume());
+ let mut sub = subscribe(&logical, HOUR).ordered();
+ let waiter = kio::Waiter::noop();
+ let read = |sub: &mut track::Ordered| -> Vec {
+ let mut items = Vec::new();
+ while let Poll::Ready(Ok(Some(item))) = sub.poll_next_group(&waiter) {
+ items.push(match item {
+ track::Item::Group(group) => format!("group {}", group.start),
+ track::Item::Gap { range, error } => format!("gap {range:?} ({error})"),
+ });
+ }
+ items
+ };
+ write(&a, 0);
+ write(&a, 5);
+ assert_eq!(read(&mut sub), ["group 0", "group 5"]);
+ write(&a, 3);
+ assert_eq!(read(&mut sub), ["gap 3..4 (old)"]);
+ write(&a, 6);
+ assert_eq!(read(&mut sub), ["group 6"]);
+ }
+
+ /// A front served by another front reports a late arrival on the inner copy too.
+ #[test]
+ fn a_nested_front_reports_a_late_arrival_as_a_gap() {
+ let inner_routes = Producer::new();
+ let inner = logical(&inner_routes);
+ let b = copy();
+ inner_routes.serve(b.consume());
+ let routes = Producer::new();
+ let outer = logical(&routes);
+ routes.serve(inner.consume());
+ let mut sub = subscribe(&outer, HOUR).ordered();
+ let waiter = kio::Waiter::noop();
+ let read = |sub: &mut track::Ordered| -> Vec {
+ let mut items = Vec::new();
+ while let Poll::Ready(Ok(Some(item))) = sub.poll_next_group(&waiter) {
+ items.push(match item {
+ track::Item::Group(group) => format!("group {}", group.start),
+ track::Item::Gap { range, error } => format!("gap {range:?} ({error})"),
+ });
+ }
+ items
+ };
+ write(&b, 0);
+ write(&b, 5);
+ assert_eq!(read(&mut sub), ["group 0", "group 5"]);
+ write(&b, 3);
+ assert_eq!(read(&mut sub), ["gap 3..4 (old)"]);
+ }
+
+ /// A late arrival the reader's ranges leave out is no gap of its own: it is neither reported
+ /// nor held, and widening the ranges does not report it later, as the read never goes back.
+ #[test]
+ fn a_front_ignores_a_late_arrival_its_ranges_exclude() {
+ let routes = Producer::new();
+ let logical = logical(&routes);
+ let a = copy();
+ routes.serve(a.consume());
+ let mut sub = subscribe(&logical, HOUR).ordered().with_range(50..);
+ let waiter = kio::Waiter::noop();
+ let read = |sub: &mut track::Ordered| -> Vec {
+ let mut items = Vec::new();
+ while let Poll::Ready(Ok(Some(item))) = sub.poll_next_group(&waiter) {
+ items.push(match item {
+ track::Item::Group(group) => format!("group {}", group.start),
+ track::Item::Gap { range, error } => format!("gap {range:?} ({error})"),
+ });
+ }
+ items
+ };
+ write(&a, 50);
+ write(&a, 60);
+ assert_eq!(read(&mut sub), ["group 50", "group 60"]);
+ for key in (0..40).step_by(2) {
+ write(&a, key);
+ assert!(read(&mut sub).is_empty(), "group {key} is outside the ranges");
+ }
+ let mut sub = sub.with_range(..);
+ let widened = read(&mut sub);
+ assert!(
+ widened.is_empty(),
+ "an ordered read never goes back, so widening reports no gap and returns no group: {widened:?}"
+ );
+ }
+
/// Lowering a front reader's local floor offers the cached groups below the old one it
/// passed over, as a plain reader does.
#[test]
diff --git a/rs/moq-net/src/model/resume_model.rs b/rs/moq-net/src/model/resume_model.rs
index 6cafa7de9c7..1052f776483 100644
--- a/rs/moq-net/src/model/resume_model.rs
+++ b/rs/moq-net/src/model/resume_model.rs
@@ -305,6 +305,20 @@ impl Reader {
ranges && self.bounds.is_none_or(|(start, end)| (start..end).contains(&group))
}
+ /// Record a run this reader gave up, failing if it handed one of its groups out.
+ fn report_drop(&mut self, index: usize, range: std::ops::Range, why: &str) -> Result<(), TestCaseError> {
+ for group in range.clone() {
+ if self.reading.contains_key(&group) {
+ return Err(fail(
+ index,
+ format!("dropped {range:?} ({why}) after handing out {group}"),
+ ));
+ }
+ }
+ self.dropped.extend(range.start..range.end.min(GROUPS));
+ Ok(())
+ }
+
/// Everything observed, to tell whether a poll found anything new.
fn observed(&self) -> (usize, usize, bool, usize) {
let frames = self
@@ -653,7 +667,14 @@ impl Model {
while reader.end.is_none() {
let polled = match reader.cursor.as_mut().unwrap() {
Cursor::Arrival(sub) => sub.poll_recv_group(&waiter),
- Cursor::Ordered(sub) => sub.poll_next_group(&waiter),
+ Cursor::Ordered(sub) => match sub.poll_next_group(&waiter) {
+ // An ordered read reports its drops in turn.
+ Poll::Ready(Ok(Some(track::Item::Gap { range, error }))) => {
+ reader.report_drop(index, range, &format!("{error:?}"))?;
+ continue;
+ }
+ polled => polled.map(|res| res.map(|item| item.map(|item| item.group().expect("a group")))),
+ },
};
match polled {
Poll::Pending => break,
@@ -728,19 +749,15 @@ impl Model {
}
}
- if reader.spec.drops
- && let Some(Cursor::Arrival(sub)) = &mut reader.cursor
- {
- while let Poll::Ready((range, err)) = sub.poll_drop(&waiter) {
- for group in range.clone() {
- if reader.reading.contains_key(&group) {
- return Err(fail(
- index,
- format!("dropped {range:?} ({err:?}) after handing out {group}"),
- ));
- }
- }
- reader.dropped.extend(range.start..range.end.min(GROUPS));
+ if reader.spec.drops && matches!(reader.cursor, Some(Cursor::Arrival(_))) {
+ loop {
+ let Some(Cursor::Arrival(sub)) = &mut reader.cursor else {
+ unreachable!("matched above")
+ };
+ let Poll::Ready((range, err)) = sub.poll_drop(&waiter) else {
+ break;
+ };
+ reader.report_drop(index, range, &format!("{err:?}"))?;
}
}
diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs
index d811278fb0d..76c2c3efc0f 100644
--- a/rs/moq-net/src/model/track.rs
+++ b/rs/moq-net/src/model/track.rs
@@ -8027,6 +8027,8 @@ struct Cursor {
/// One past the highest key returned by `next_group`.
/// Used only by that method to skip late arrivals; does not affect `recv_group`.
next_key: u64,
+ /// The absolute arrival index [`Self::late`] has read up to.
+ late: usize,
/// Exclusive upper key bound for `next_group` and `recv_group`. `None` means no
/// cap. Set by `set_range`;
/// can be raised, lowered, or unset at any time. Groups at or past the cap stay in
@@ -8046,8 +8048,9 @@ struct Cursor {
/// [`Subscriber`], which drains this after every read.
stale: stats::Content,
/// Groups the seek path has convicted but whose keys no caller has committed
- /// past yet, keyed by key; see [`Self::commit_seek_stale`].
- seek_pending: BTreeMap,
+ /// past yet, keyed by key, with the budget that convicted each; see
+ /// [`Self::commit_seek_stale`].
+ seek_pending: BTreeMap,
/// What this cursor handed out, skipped, gave up, and reported.
record: Record,
/// The drops a wire publisher forwards, once it first polls for one.
@@ -8392,6 +8395,7 @@ impl Cursor {
filter: Box::new(filter),
index: 0,
next_key: 0,
+ late: 0,
end_key: None,
// Uncapped, the servable groups are the window's.
servable: published.servable.clone(),
@@ -8458,6 +8462,21 @@ impl Cursor {
}
}
+ /// The keys that landed in this cursor's window since it last looked, below `below`: a key
+ /// cursor already read past them, so it never hands them out.
+ fn late(&mut self, below: u64) -> Vec {
+ let state = self.state.read();
+ let from = self.late.max(state.offset);
+ self.late = state.offset + state.arrival.len();
+ state
+ .arrival
+ .iter()
+ .skip(from - state.offset)
+ .map(|(key, _)| *key)
+ .filter(|key| *key < below && self.filter.window.contains(*key))
+ .collect()
+ }
+
/// Resolve [`Self::servable`] again once the window or cap moved, and publish it to the
/// groups already handed out. Costs the ranges, never the cache.
fn update_servable(&mut self) {
@@ -9623,7 +9642,7 @@ impl Cursor {
// the edge can be evicted), so the group may still be delivered.
// Re-snapshot on every re-examination so the eventual count reflects the
// group's latest observed content.
- self.seek_pending.insert(group.start, group.content());
+ self.seek_pending.insert(group.start, (group.content(), drift.budget));
floor = group.start.saturating_add(1);
continue;
}
@@ -9690,8 +9709,15 @@ impl Cursor {
/// then be delivered after all.
fn commit_seek_stale(&mut self, committed: u64) {
let keep = self.seek_pending.split_off(&committed);
- for (_, content) in std::mem::replace(&mut self.seek_pending, keep) {
+ for (key, (content, budget)) in std::mem::replace(&mut self.seek_pending, keep) {
self.stale.add(content);
+ // Given up for good now, so it is as reported as a write-off is: a key-order
+ // reader tells its caller the group will not come.
+ if self.drops.is_some()
+ && let Some(end) = key.checked_add(1)
+ {
+ self.record.pending.insert(key..end, Pending::own(Hole::stale(budget)));
+ }
}
}
}
@@ -10233,6 +10259,54 @@ impl Subscriber {
}) || owed())
}
+ /// The keys that landed on this copy since the last call, below `below`.
+ pub(super) fn late(&mut self, below: u64) -> Vec {
+ match &mut self.inner {
+ Inner::Plain(cursor) => cursor.late(below),
+ Inner::Resume(resume, _) => resume.late(below),
+ }
+ }
+
+ /// Report the groups that landed below where this key-order reader had read, which it
+ /// never hands out, as given up `Old`. A group it handed out or judged stale is already
+ /// settled, so the record blocks it.
+ fn note_late(&mut self) {
+ let mut late = match &mut self.inner {
+ Inner::Plain(cursor) => cursor.late(cursor.next_key),
+ Inner::Resume(resume, logical) => resume.late(logical.record.delivered.floor()),
+ };
+ // A copy's window can be wider than this reader's ranges and bounds, so a late key they
+ // leave out is no gap of this reader's, as the plain path's window already clips it.
+ if matches!(self.inner, Inner::Resume(..)) {
+ late.retain(|key| !self.runs_in_range(*key, key.saturating_add(1)).is_empty());
+ }
+ let record = match &mut self.inner {
+ Inner::Plain(cursor) => &mut cursor.record,
+ Inner::Resume(_, logical) => &mut logical.record,
+ };
+ for key in late {
+ if let Some(end) = key.checked_add(1) {
+ DropCursor::give_up(record, key..end, Hole::drop(StreamError::Old));
+ }
+ }
+ }
+
+ /// Record `key` as served, so no cut or drop of it is reported afterwards.
+ fn settle(&mut self, key: u64) {
+ if let Some(end) = key.checked_add(1) {
+ self.settle_range(key..end);
+ }
+ }
+
+ /// Record `keys` as served or reported, so no cut or drop of them is reported again.
+ fn settle_range(&mut self, keys: ops::Range) {
+ let record = match &mut self.inner {
+ Inner::Plain(cursor) => &mut cursor.record,
+ Inner::Resume(_, logical) => &mut logical.record,
+ };
+ record.tell(keys, Report::Settled);
+ }
+
/// The key cursor behind [`Ordered`], which owns the only public door to it.
pub(super) fn poll_next_group(&mut self, waiter: &kio::Waiter) -> Poll>> {
self.poll_group(true, waiter)
@@ -10245,7 +10319,14 @@ impl Subscriber {
/// The two advance independently, and interleaving them produces a group stream that
/// is neither, which is why the choice is a handle rather than a method.
pub fn ordered(self) -> Ordered {
- Ordered { inner: self }
+ Ordered {
+ inner: self,
+ held: None,
+ gaps: BTreeMap::new(),
+ at: 0,
+ started: false,
+ idle: None,
+ }
}
/// Whether `other` was cloned from this subscriber (shares the same underlying state).
@@ -10832,7 +10913,7 @@ impl Subscriber {
/// Created by [`Subscriber::ordered`], which consumes the subscriber, so a track is read
/// one way or the other and never both. Every group it returns has a higher key
/// than the last, so a late arrival (network reordering, or a gap the cache filled after
-/// the fact) is skipped rather than delivered out of turn.
+/// the fact) is reported as a [gap](Item::Gap) rather than delivered out of turn.
///
/// # Age and skipping
///
@@ -10863,11 +10944,56 @@ impl Subscriber {
/// ordered moq-transport FETCH. A moq-transport SUBSCRIBE signals no gaps, so there a group
/// lost in transit can hold the ordered reader at its hole until the track ends.
///
+/// A group this reader gives up on is not lost silently: it comes back as an
+/// [`Item::Gap`], so a reader that paces on what was sent counts the loss instead of
+/// waiting for a group that will never come. That is every group the publisher or a relay
+/// dropped, the transport evicted, the budget aged out, a fetch could not find, or that
+/// landed after the read passed its key (reported as `Old`). A gap carries the keys it
+/// spans and is returned once, after any group the read still owes below it and before
+/// the next group above it, or when the track ends. Keys the publisher declared skipped
+/// (see [`Producer::skip_groups`]) were never groups, so they are no gap;
+/// [`Self::is_skipped`] answers for them.
+///
/// The budget is updatable mid-stream through [`Self::control`]. [`Duration::ZERO`] (the
/// default) keeps only what nothing newer has superseded, so a consumer that stalls and
/// resumes rejoins the live edge instead of replaying at 1x.
pub struct Ordered {
inner: Subscriber,
+ /// A group read and not yet returned, because a gap below it goes first.
+ held: Option>,
+ /// The gaps found and not yet returned, as `start => (end, error)`. They never overlap, and
+ /// a drop found while others wait goes in order among them.
+ gaps: BTreeMap,
+ /// One past the highest key returned, as a group or the end of a gap.
+ at: u64,
+ /// Whether the drop record exists. It holds only what happened after it began, so it
+ /// starts before the first read.
+ started: bool,
+ /// The read that last parked, answered again until something it parked on fires.
+ idle: Option>,
+}
+
+/// What an [`Ordered`] read returns next.
+pub enum Item {
+ /// A group, with a higher key than any group returned before.
+ Group(group::Consumer),
+ /// Groups this reader gave up on, which it will not return.
+ Gap {
+ /// The keys given up, as group starts.
+ range: ops::Range,
+ /// Why: what a read of those groups would have ended with.
+ error: Error,
+ },
+}
+
+impl Item {
+ /// The group, or `None` for a gap.
+ pub fn group(self) -> Option {
+ match self {
+ Self::Group(group) => Some(group),
+ Self::Gap { .. } => None,
+ }
+ }
}
impl Ordered {
@@ -10886,7 +11012,8 @@ impl Ordered {
self.inner.broadcast()
}
- /// The next group with a higher key than any previously returned.
+ /// The next group with a higher key than any previously returned, or a gap this reader
+ /// gave up on (see [Gaps](Self#gaps)).
///
/// Honors the group range set by [`Self::with_range`]:
/// a group past the cap stays in the producer's cache and becomes eligible again if
@@ -10895,12 +11022,192 @@ impl Ordered {
/// Later reads return `Ok(None)` immediately too; widening the ranges with
/// [`Self::update`] resumes reading above the last key returned.
///
- /// Returns `Ok(Some(group))` when a group is available, `Ok(None)` when the track is
+ /// Returns `Ok(Some(item))` when a group or gap is available, `Ok(None)` when the track is
/// finished (including an abort after its declared end settled, see [`Producer::abort`])
/// or the ranges are read, or the error when the track has been aborted short of its end.
+ /// Every gap is returned before the end, even a clean one.
#[cfg_attr(feature = "async", kio::wrapper)]
- pub fn poll_next_group(&mut self, waiter: &kio::Waiter) -> Poll>> {
- self.inner.poll_next_group(waiter)
+ pub fn poll_next_group(&mut self, waiter: &kio::Waiter) -> Poll>> {
+ if idle::Idle::parked(&self.idle, (), waiter) {
+ return Poll::Pending;
+ }
+ let mut idle = idle::Idle::start(&mut self.idle, waiter);
+ let polled = self.poll_item(idle.waiter());
+ idle.finish(&mut self.idle, (), polled.is_pending());
+ polled
+ }
+
+ fn poll_item(&mut self, waiter: &kio::Waiter) -> Poll>> {
+ // Gaps are recorded only once polled for, so before the first read.
+ if !self.started {
+ self.started = true;
+ self.find_gap(true, waiter);
+ }
+ let read = self.read(waiter);
+ // After the read, which judged what it walked past: a group it handed out leaves only
+ // what that queued, and anything else judges the live edge for the gaps it parks on.
+ self.find_gap(!matches!(read, Poll::Ready(Ok(Some(_)))), waiter);
+ match read {
+ Poll::Ready(Ok(Some(group))) => {
+ let key = group.start;
+ match self.lowest_gap() {
+ // The gap lies wholly below it, or around it: the part below goes first.
+ Some((at, span)) if span.start < key => {
+ // What lies above the group, or above the run when it ends before it, waits.
+ let resume = if span.end <= key {
+ span.end
+ } else {
+ key.saturating_add(1)
+ };
+ let below = span.start..span.end.min(key);
+ let item = self.take_gap(at, below, resume);
+ self.held = Some(Box::new(group));
+ Poll::Ready(Ok(Some(item)))
+ }
+ // A group handed out is no gap, even one a drop also named, as a cut group is.
+ lowest => {
+ if let Some((at, span)) = lowest
+ && span.start == key
+ {
+ self.cut_gap(at, span.start, key.saturating_add(1));
+ }
+ self.at = self.at.max(key.saturating_add(1));
+ Poll::Ready(Ok(Some(Item::Group(group))))
+ }
+ }
+ }
+ // What the read still owes lies above the gap, so the gap goes first.
+ Poll::Pending => match self.next_gap() {
+ Some(item) => Poll::Ready(Ok(Some(item))),
+ None => Poll::Pending,
+ },
+ // Every gap is returned before the end.
+ Poll::Ready(end) => match self.lowest_gap() {
+ Some((at, span)) => {
+ let resume = span.end;
+ Poll::Ready(Ok(Some(self.take_gap(at, span, resume))))
+ }
+ None => Poll::Ready(end.map(|_| None)),
+ },
+ }
+ }
+
+ /// The next group in key order: the one held behind a gap, if the ranges still cover it.
+ fn read(&mut self, waiter: &kio::Waiter) -> Poll>> {
+ if let Some(group) = self.held.take() {
+ let key = group.start;
+ if !self.inner.runs_in_range(key, key.saturating_add(1)).is_empty() {
+ return Poll::Ready(Ok(Some(*group)));
+ }
+ // A range moved off it. With nothing in range above, it waits for the range to widen
+ // again, and the read waits or ends as it does for capped ranges; otherwise the read
+ // passes it.
+ if self.inner.runs_in_range(key, u64::MAX).is_empty() {
+ self.held = Some(group);
+ }
+ }
+ let read = self.inner.poll_next_group(waiter);
+ if let Poll::Ready(Ok(Some(group))) = &read {
+ // Handed out, so no later cut or drop of it is a gap.
+ self.inner.settle(group.start);
+ }
+ read
+ }
+
+ /// The lowest run of a gap the ranges cover now, as the gap's key in `gaps` and the keys of
+ /// it that run covers, since the ranges may have moved since the gap was found. A gap they
+ /// exclude waits, in case they widen again.
+ fn lowest_gap(&mut self) -> Option<(u64, ops::Range)> {
+ // The fields borrow apart, so the map is walked in place and the walk stops at the
+ // first gap the ranges cover.
+ let Self { inner, gaps, .. } = self;
+ gaps.iter().find_map(|(start, (end, _))| {
+ let runs = inner.runs_in_range(*start, *end);
+ Some((*start, runs.into_iter().next()?))
+ })
+ }
+
+ /// Take `given` from the gap `lowest_gap` found, as an item. What lies from `resume` on stays
+ /// a gap, to be clipped and returned later, as does what lies below `given`.
+ fn take_gap(&mut self, at: u64, given: ops::Range, resume: u64) -> Item {
+ let error = self.cut_gap(at, given.start, resume);
+ self.give(given, error)
+ }
+
+ /// Remove `from..resume` from the gap held at `at`, returning its error. What lies below
+ /// `from`, which the ranges leave out, and from `resume` on stay gaps, to be clipped and
+ /// returned later.
+ fn cut_gap(&mut self, at: u64, from: u64, resume: u64) -> Error {
+ let (end, error) = self.gaps.remove(&at).expect("the lowest gap is held");
+ if at < from {
+ self.gaps.insert(at, (from, error.clone()));
+ }
+ if end > resume {
+ self.gaps.insert(resume, (end, error.clone()));
+ }
+ error
+ }
+
+ /// Look for every drop recorded, in whatever order they come.
+ fn find_gap(&mut self, judge: bool, waiter: &kio::Waiter) {
+ self.inner.note_late();
+ loop {
+ let polled = match judge {
+ true => self.inner.poll_drop(waiter),
+ false => self.inner.poll_queued_drop(waiter),
+ };
+ let Poll::Ready((range, hole)) = polled else {
+ return;
+ };
+ match hole {
+ Hole::Drop { error, .. } => {
+ // A final drop replaces a stale one still waiting, rather than overlap it.
+ self.forget_gaps(range.clone());
+ self.gaps.insert(range.start, (range.end, error.into()));
+ }
+ // A skip was never a group: nothing was given up, and it replaces a drop
+ // still waiting.
+ Hole::Skip => self.forget_gaps(range),
+ }
+ }
+ }
+
+ /// Remove `skipped` from the gaps waiting, splitting those it only partly covers.
+ fn forget_gaps(&mut self, skipped: ops::Range) {
+ // The gaps never overlap, so their ends rise with their starts: walk down from the
+ // highest one that starts below the skip's end, and stop at the first that ends before it.
+ let hit: Vec = self
+ .gaps
+ .range(..skipped.end)
+ .rev()
+ .take_while(|(_, (end, _))| *end > skipped.start)
+ .map(|(start, _)| *start)
+ .collect();
+ for start in hit {
+ let (end, error) = self.gaps.remove(&start).expect("the gap was just found");
+ if start < skipped.start {
+ self.gaps.insert(start, (skipped.start, error.clone()));
+ }
+ if end > skipped.end {
+ self.gaps.insert(skipped.end, (end, error));
+ }
+ }
+ }
+
+ /// The lowest gap, once nothing the read still owes lies below it.
+ fn next_gap(&mut self) -> Option- {
+ let (at, span) = self.lowest_gap()?;
+ let ready = span.start <= self.at || self.inner.next_in_range(self.at).is_none_or(|owed| owed >= span.start);
+ let resume = span.end;
+ ready.then(|| self.take_gap(at, span, resume))
+ }
+
+ /// A gap as an item, which the read has now moved past.
+ fn give(&mut self, range: ops::Range
, error: Error) -> Item {
+ self.at = self.at.max(range.end);
+ // Told once: no later drop of these keys is a gap again.
+ self.inner.settle_range(range.clone());
+ Item::Gap { range, error }
}
/// Limit subsequent reads to these group keys, returning `self` for chaining.
@@ -10914,6 +11221,8 @@ impl Ordered {
/// [`Self::with_range`] on a reader held in place.
fn set_range(&mut self, groups: impl RangeBounds) {
+ // A moved range notifies nothing a parked read waits on.
+ idle::Idle::disarm(&mut self.idle);
self.inner.set_range(groups);
}
@@ -10932,6 +11241,7 @@ impl Ordered {
///
/// Returns [`Error::Closed`] if the track already ended.
pub fn update(&mut self, subscription: Subscription) -> Result<()> {
+ idle::Idle::disarm(&mut self.idle);
self.inner.update(subscription)
}
@@ -11907,6 +12217,30 @@ mod test {
}
/// Every group a reader hands out with the frame indices it reads, until one would block.
+ /// What an ordered read returns until it ends, as `group 2` or `gap 1..2 (error)`.
+ #[cfg(feature = "async")]
+ fn items(ordered: &mut Ordered) -> Vec {
+ let mut items = Vec::new();
+ while let Some(next) = ordered.next_group().now_or_never() {
+ match next {
+ Ok(Some(Item::Group(group))) => items.push(format!("group {}", group.start)),
+ Ok(Some(Item::Gap { range, error })) => items.push(format!("gap {range:?} ({error})")),
+ Ok(None) => return items,
+ Err(err) => {
+ items.push(format!("error ({err})"));
+ return items;
+ }
+ }
+ }
+ items.push("pending".to_string());
+ items
+ }
+
+ /// An ordered read that expects no gap, as the groups it returns.
+ fn groups_only(polled: Poll>>) -> Poll>> {
+ polled.map(|res| res.map(|item| item.map(|item| item.group().expect("a group, not a gap"))))
+ }
+
fn read_groups(
mut next: impl FnMut(&kio::Waiter) -> Poll>>,
) -> Vec<(u64, Vec)> {
@@ -11946,7 +12280,7 @@ mod test {
let mut arrival = bounded();
assert_eq!(read_groups(|waiter| arrival.poll_recv_group(waiter)), want);
let mut ordered = bounded().ordered();
- assert_eq!(read_groups(|waiter| ordered.poll_next_group(waiter)), want);
+ assert_eq!(read_groups(|waiter| groups_only(ordered.poll_next_group(waiter))), want);
}
/// A group missing its head is never served short: reading it is [`Error::Lagged`],
@@ -15681,8 +16015,10 @@ mod test {
#[cfg(feature = "async")]
fn drain_ordered(subscriber: &mut Ordered) -> Vec {
let mut keys = Vec::new();
- while let Some(Ok(Some(group))) = subscriber.next_group().now_or_never() {
- keys.push(group.start);
+ while let Some(Ok(Some(item))) = subscriber.next_group().now_or_never() {
+ if let Item::Group(group) = item {
+ keys.push(group.start);
+ }
}
keys
}
@@ -16061,8 +16397,8 @@ mod test {
let mut subscriber = producer.subscribe(Subscription::default().with_range(0..2));
assert_eq!(drain(&mut subscriber), vec![1]);
let mut subscriber = producer.subscribe(Subscription::default().with_range(0..2)).ordered();
- let next = subscriber.next_group().now_or_never().unwrap().unwrap().unwrap();
- assert_eq!(next.start, 1);
+ // Group 0 is stale against it, which the read reports before the group.
+ assert_eq!(items(&mut subscriber), ["gap 0..1 (old)", "group 1"]);
// A budget that covers it delivers the range, and nothing above it.
let mut subscriber = producer.subscribe(replay().with_end(2));
@@ -16918,12 +17254,8 @@ mod test {
let mut ordered = producer.subscribe(replay()).ordered();
drop(producer);
- let first = ordered.next_group().now_or_never().expect("group 0").unwrap().unwrap();
- assert_eq!(first.start, 0);
- let second = ordered.next_group().now_or_never().expect("group 2").unwrap().unwrap();
- assert_eq!(second.start, 2);
- let done = ordered.next_group().now_or_never().expect("end").unwrap();
- assert!(done.is_none(), "missing groups below the boundary are not an error");
+ // A group never produced below a later one is given up, and the end is clean.
+ assert_eq!(items(&mut ordered), ["group 0", "gap 1..2 (not found)", "group 2"]);
}
/// A finished track ends both readers with nothing left in range, whether the range
@@ -16943,7 +17275,7 @@ mod test {
}
let _producer = producer.finish().unwrap();
for key in 0..3 {
- let Poll::Ready(Ok(Some(group))) = ordered.poll_next_group(&kio::Waiter::noop()) else {
+ let Poll::Ready(Ok(Some(Item::Group(group)))) = ordered.poll_next_group(&kio::Waiter::noop()) else {
panic!("group {key}, given_up={given_up}")
};
assert_eq!(group.start, key);
@@ -16952,6 +17284,18 @@ mod test {
};
assert_eq!(group.start, key);
}
+ // The group given up is reported, then the read ends.
+ if given_up {
+ let Poll::Ready(Ok(Some(Item::Gap { range, error }))) = ordered.poll_next_group(&kio::Waiter::noop())
+ else {
+ panic!("the given up group is reported")
+ };
+ assert_eq!(range, 3..4);
+ assert!(
+ matches!(error, Error::Stream(StreamError::DeliveryTimeout)),
+ "{error:?}"
+ );
+ }
assert!(
matches!(ordered.poll_next_group(&kio::Waiter::noop()), Poll::Ready(Ok(None))),
"given_up={given_up}"
@@ -17006,9 +17350,23 @@ mod test {
late.write_frame(late.key_time(), &b"late"[..]).unwrap();
let late = late.finish().unwrap();
assert_eq!(late.frame_count(), 1);
- let first = ordered.next_group().now_or_never().expect("group 0").unwrap().unwrap();
+ let first = ordered
+ .next_group()
+ .now_or_never()
+ .expect("group 0")
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap();
assert_eq!(first.start, 0);
- let second = ordered.next_group().now_or_never().expect("group 1").unwrap().unwrap();
+ let second = ordered
+ .next_group()
+ .now_or_never()
+ .expect("group 1")
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap();
assert_eq!(second.start, 1);
let done = ordered.next_group().now_or_never().expect("end").unwrap();
assert!(done.is_none(), "the finished track ends after its late group");
@@ -17224,7 +17582,7 @@ mod test {
let mut ordered = producer.subscribe(replay().with_range(0..)).ordered();
for key in 0..2 {
assert_eq!(arrival.assert_group().start, key);
- assert_eq!(ordered.next_group().await.unwrap().unwrap().start, key);
+ assert_eq!(ordered.next_group().await.unwrap().unwrap().group().unwrap().start, key);
}
assert!(matches!(arrival.recv_group().await, Err(Error::App(7))));
assert!(matches!(ordered.next_group().await, Err(Error::App(7))));
@@ -17262,7 +17620,7 @@ mod test {
let mut reader = producer.subscribe(replay().with_range(0..)).ordered();
let _finished = producer.clone().fail(Error::App(7)).unwrap();
for key in 0..3 {
- assert_eq!(reader.next_group().await.unwrap().unwrap().start, key);
+ assert_eq!(reader.next_group().await.unwrap().unwrap().group().unwrap().start, key);
}
assert!(matches!(reader.next_group().await, Err(Error::App(7))));
assert!(matches!(producer.append_next(), Err(Error::Closed)));
@@ -17342,7 +17700,12 @@ mod test {
// Missing lower groups do not hold either reader for the wire's grace.
// The last group's own stream remains open independently of the track end.
assert_eq!(arrival.assert_group().start, 2);
- assert_eq!(ordered.next_group().await.unwrap().unwrap().start, 2);
+ // They are reported as one gap, since nothing was produced below the group.
+ let Some(Item::Gap { range, .. }) = ordered.next_group().await.unwrap() else {
+ panic!("the missing groups are reported")
+ };
+ assert_eq!(range, 0..2);
+ assert_eq!(ordered.next_group().await.unwrap().unwrap().group().unwrap().start, 2);
assert!(arrival.recv_group().now_or_never().unwrap().unwrap().is_none());
assert!(ordered.next_group().now_or_never().unwrap().unwrap().is_none());
@@ -17352,10 +17715,15 @@ mod test {
arrival.recv_group().now_or_never().unwrap(),
Err(Error::Timeout)
));
- assert!(matches!(
- ordered.next_group().now_or_never().unwrap(),
- Err(Error::Timeout)
- ));
+ let next = ordered.next_group().now_or_never().unwrap();
+ assert!(
+ matches!(next, Err(Error::Timeout)),
+ "{:?}",
+ next.map(|i| i.map(|i| match i {
+ Item::Gap { range, error } => format!("{range:?} {error}"),
+ _ => "group".into(),
+ }))
+ );
}
/// A received copy's end holds readers at a hole below it, since a lower group's stream
@@ -17971,7 +18339,7 @@ mod test {
.next_group()
.now_or_never()
.expect("should not block")
- .map(|group| group.map(|group| group.start));
+ .map(|item| item.map(|item| item.group().expect("a group").start));
assert!(matches!(res, Err(Error::Timeout)), "expected the abort, got {res:?}");
}
@@ -18102,7 +18470,7 @@ mod test {
.next_group()
.now_or_never()
.expect("should not block")
- .map(|group| group.map(|group| group.start));
+ .map(|item| item.map(|item| item.group().expect("a group").start));
assert!(matches!(end, Ok(None)), "expected the clean end, got {end:?}");
}
@@ -18130,7 +18498,7 @@ mod test {
.next_group()
.now_or_never()
.expect("should not block")
- .map(|group| group.map(|group| group.start));
+ .map(|item| item.map(|item| item.group().expect("a group").start));
assert!(
matches!(end, Ok(None)),
"capped at the end: expected the clean end, got {end:?}"
@@ -18154,7 +18522,7 @@ mod test {
.next_group()
.now_or_never()
.expect("should not block")
- .map(|group| group.map(|group| group.start));
+ .map(|item| item.map(|item| item.group().expect("a group").start));
assert!(
matches!(end, Ok(None)),
"gap below the cap: expected the clean end, got {end:?}"
@@ -18962,6 +19330,309 @@ mod test {
assert_eq!(drain_ordered(&mut consumer), [0, 1]);
}
+ /// A group the track gives up on comes back as a gap, in key order among the groups, with
+ /// why; a group already returned is no gap, nor is a key the publisher skipped.
+ #[test]
+ #[cfg(feature = "async")]
+ fn an_ordered_read_reports_what_it_gives_up() {
+ let producer = track_producer("test", None);
+ let mut ordered = producer.subscribe(replay()).ordered();
+ put(&producer, 0, None);
+ put(&producer, 3, None);
+ producer.drop_groups(1..3, StreamError::DeliveryTimeout);
+ producer.skip_groups(4..6);
+ put(&producer, 6, None);
+ assert_eq!(
+ items(&mut ordered),
+ [
+ "group 0",
+ "gap 1..3 (delivery timeout)",
+ "group 3",
+ "group 6",
+ "pending"
+ ],
+ "a skip is no gap"
+ );
+
+ // A group handed out and then cut is read as the failure it is, never reported twice.
+ let cut = producer.create_group(group::Info { start: 7 }).unwrap();
+ let Poll::Ready(Ok(Some(Item::Group(read)))) = ordered.poll_next_group(&kio::Waiter::noop()) else {
+ panic!("group 7 is ready")
+ };
+ assert_eq!(read.start, 7);
+ cut.abort(Error::Cancel).unwrap();
+ assert_eq!(items(&mut ordered), ["pending"]);
+ }
+
+ /// A gap waits behind a group the read still owes below it, and goes out once nothing is
+ /// owed, or the end comes: every gap is returned before the track ends.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_gap_waits_behind_what_is_owed_below_it() {
+ let producer = received_producer(None);
+ let mut ordered = producer
+ .subscribe(Subscription::default().with_delay(Delay::default().with_max(None)))
+ .ordered();
+ put(&producer, 0, None);
+ put(&producer, 2, None);
+ producer.drop_groups(4..5, StreamError::DeliveryTimeout);
+ assert_eq!(
+ items(&mut ordered),
+ ["group 0", "pending"],
+ "group 1 is owed below the gap"
+ );
+ put(&producer, 1, None);
+ assert_eq!(
+ items(&mut ordered),
+ ["group 1", "group 2", "pending"],
+ "group 3 is owed below the gap"
+ );
+ put(&producer, 3, None);
+ assert_eq!(
+ items(&mut ordered),
+ ["group 3", "gap 4..5 (delivery timeout)", "pending"]
+ );
+
+ let producer = track_producer("test", None);
+ let mut ordered = producer.subscribe(replay()).ordered();
+ put(&producer, 0, None);
+ producer.drop_groups(1..2, StreamError::DeliveryTimeout);
+ let _producer = producer.finish().unwrap();
+ assert_eq!(
+ items(&mut ordered),
+ ["group 0", "gap 1..2 (delivery timeout)"],
+ "the end is clean after every gap"
+ );
+ }
+
+ /// A gap returned is told once: a later drop of the same keys, even a final one that would
+ /// replace a stale report on the wire, is no second gap.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_gap_is_returned_once() {
+ let producer = received_producer(None);
+ let budget = Duration::from_millis(50);
+ let mut ordered = producer
+ .subscribe(Subscription::default().with_delay(Delay::default().with_max(budget)))
+ .ordered();
+ put(&producer, 0, None);
+ put(&producer, 2, None);
+ assert_eq!(items(&mut ordered), ["group 0", "pending"]);
+ // The budget gives up on the group still owed, as stale.
+ elapse(&producer, budget);
+ assert_eq!(items(&mut ordered), ["gap 1..2 (old)", "group 2", "pending"]);
+ producer.drop_groups(1..2, StreamError::DeliveryTimeout);
+ put(&producer, 3, None);
+ assert_eq!(items(&mut ordered), ["group 3", "pending"]);
+ }
+
+ /// A skip replaces a drop still waiting for the groups below it, so the keys it names are
+ /// no gap, and the part of a drop it leaves still is.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_skip_replaces_a_buffered_gap() {
+ let producer = received_producer(None);
+ let mut ordered = producer
+ .subscribe(Subscription::default().with_delay(Delay::default().with_max(None)))
+ .ordered();
+ put(&producer, 0, None);
+ put(&producer, 2, None);
+ producer.drop_groups(4..8, StreamError::DeliveryTimeout);
+ assert_eq!(items(&mut ordered), ["group 0", "pending"]);
+ producer.skip_groups(5..7);
+ producer.drop_groups(1..2, StreamError::DeliveryTimeout);
+ assert_eq!(
+ items(&mut ordered),
+ ["gap 1..2 (delivery timeout)", "group 2", "pending"]
+ );
+ put(&producer, 3, None);
+ assert_eq!(
+ items(&mut ordered),
+ [
+ "group 3",
+ "gap 4..5 (delivery timeout)",
+ "gap 7..8 (delivery timeout)",
+ "pending"
+ ]
+ );
+ }
+
+ /// A group landing after the read passed its key is given up as old, since the read never
+ /// goes back; the group it was passed for is no gap.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_cached_group_the_seek_passes_as_stale_is_a_gap() {
+ let producer = received_producer(None);
+ put(&producer, 0, None);
+ put(&producer, 1, None);
+ elapse(&producer, Duration::from_millis(100));
+ put(&producer, 2, None);
+ let budget = Duration::from_millis(50);
+ let mut ordered = producer
+ .subscribe(Subscription::default().with_delay(Delay::default().with_max(budget)))
+ .ordered();
+ assert_eq!(items(&mut ordered), ["gap 0..1 (old)", "group 1", "group 2", "pending"]);
+ }
+
+ /// A final drop replaces the part of a stale gap it covers while that gap still waits, so
+ /// the buffered gaps never overlap: the stale run keeps what lies either side of it.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_buffered_gap_gives_way_to_what_replaces_it() {
+ let producer = received_producer(None);
+ let mut ordered = producer.subscribe(replay()).ordered();
+ ordered.gaps.insert(4, (8, StreamError::Old.into()));
+ ordered.forget_gaps(5..7);
+ ordered.gaps.insert(5, (7, StreamError::DeliveryTimeout.into()));
+ let held: Vec<_> = ordered.gaps.iter().map(|(start, (end, _))| (*start, *end)).collect();
+ assert_eq!(held, [(4, 5), (5, 7), (7, 8)]);
+ ordered.forget_gaps(0..100);
+ assert!(ordered.gaps.is_empty());
+ }
+
+ /// What a range excludes from a gap that is partly covered stays a gap, so a range that
+ /// widens again over it reports it.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_gap_a_range_clips_keeps_what_the_range_excluded() {
+ let producer = received_producer(None);
+ let mut ordered = producer
+ .subscribe(Subscription::default().with_delay(Delay::default().with_max(None)))
+ .ordered();
+ put(&producer, 2, None);
+ producer.drop_groups(0..2, StreamError::DeliveryTimeout);
+ producer.drop_groups(3..6, StreamError::DeliveryTimeout);
+ let first = ordered.next_group().now_or_never().expect("ready").unwrap().unwrap();
+ assert!(matches!(first, Item::Gap { range, .. } if range == (0..2)));
+ ordered.set_range(4..5);
+ assert_eq!(items(&mut ordered), ["gap 4..5 (delivery timeout)", "pending"]);
+ ordered.set_range(..);
+ assert_eq!(
+ items(&mut ordered),
+ ["gap 3..4 (delivery timeout)", "gap 5..6 (delivery timeout)", "pending"]
+ );
+ }
+
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_late_arrival_is_a_gap() {
+ let producer = track_producer("test", None);
+ let mut ordered = producer.subscribe(replay()).ordered();
+ put(&producer, 0, None);
+ put(&producer, 5, None);
+ assert_eq!(items(&mut ordered), ["group 0", "group 5", "pending"]);
+ put(&producer, 3, None);
+ assert_eq!(items(&mut ordered), ["gap 3..4 (old)", "pending"]);
+ put(&producer, 6, None);
+ assert_eq!(
+ items(&mut ordered),
+ ["group 6", "pending"],
+ "reported once, nothing else"
+ );
+ }
+
+ /// A gap is clipped to what the reader asked for.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_gap_is_clipped_to_the_reader_range() {
+ let producer = track_producer("test", None);
+ let mut ordered = producer.subscribe(replay()).ordered().with_range(2..4);
+ producer.drop_groups(0..6, StreamError::DeliveryTimeout);
+ put(&producer, 5, None);
+ assert_eq!(items(&mut ordered), ["gap 2..4 (delivery timeout)", "pending"]);
+ }
+
+ /// A drop found while a higher gap waits goes out first, in key order, rather than staying
+ /// hidden behind it.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_lower_drop_found_late_goes_before_a_buffered_gap() {
+ let producer = received_producer(None);
+ let mut ordered = producer
+ .subscribe(Subscription::default().with_delay(Delay::default().with_max(None)))
+ .ordered();
+ put(&producer, 0, None);
+ put(&producer, 2, None);
+ producer.drop_groups(4..5, StreamError::DeliveryTimeout);
+ assert_eq!(items(&mut ordered), ["group 0", "pending"]);
+ producer.drop_groups(1..2, StreamError::DeliveryTimeout);
+ assert_eq!(
+ items(&mut ordered),
+ ["gap 1..2 (delivery timeout)", "group 2", "pending"]
+ );
+ put(&producer, 3, None);
+ assert_eq!(
+ items(&mut ordered),
+ ["group 3", "gap 4..5 (delivery timeout)", "pending"]
+ );
+ }
+
+ /// A range narrowed after a gap went out ahead of a group read keeps that group back,
+ /// handing it out again once the range widens, and clips the gaps still waiting.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_range_moved_after_a_gap_applies_to_what_is_buffered() {
+ let producer = track_producer("test", None);
+ let mut ordered = producer.subscribe(replay()).ordered();
+ put(&producer, 2, None);
+ producer.drop_groups(0..2, StreamError::DeliveryTimeout);
+ producer.drop_groups(3..6, StreamError::DeliveryTimeout);
+ let first = ordered.next_group().now_or_never().expect("ready").unwrap().unwrap();
+ assert!(matches!(first, Item::Gap { range, .. } if range == (0..2)));
+ ordered.set_range(..2);
+ assert_eq!(
+ items(&mut ordered),
+ ["pending"],
+ "capped below the held group, which waits for the range"
+ );
+ ordered.set_range(..4);
+ assert_eq!(
+ items(&mut ordered),
+ ["group 2", "gap 3..4 (delivery timeout)", "pending"],
+ "the held group returns, and the gap above is clipped to the range"
+ );
+
+ let producer = track_producer("test", None);
+ let mut ordered = producer.subscribe(replay()).ordered();
+ put(&producer, 2, None);
+ put(&producer, 3, None);
+ producer.drop_groups(0..2, StreamError::DeliveryTimeout);
+ let first = ordered.next_group().now_or_never().expect("ready").unwrap().unwrap();
+ assert!(matches!(first, Item::Gap { range, .. } if range == (0..2)));
+ ordered.set_range(3..);
+ assert_eq!(
+ items(&mut ordered),
+ ["group 3", "pending"],
+ "a floor raised over the held group passes it"
+ );
+ }
+
+ /// A gap a moved range only partly covers returns that part, and the rest stays for the
+ /// range to widen over.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_moved_range_returns_a_buffered_gap_by_the_part_it_covers() {
+ let producer = track_producer("test", None);
+ let mut ordered = producer.subscribe(replay()).ordered();
+ put(&producer, 2, None);
+ producer.drop_groups(0..2, StreamError::DeliveryTimeout);
+ producer.drop_groups(3..6, StreamError::DeliveryTimeout);
+ let first = ordered.next_group().now_or_never().expect("ready").unwrap().unwrap();
+ assert!(matches!(first, Item::Gap { range, .. } if range == (0..2)));
+ ordered.set_range(..4);
+ assert_eq!(
+ items(&mut ordered),
+ ["group 2", "gap 3..4 (delivery timeout)", "pending"],
+ "capped inside the gap"
+ );
+ ordered.set_range(..);
+ assert_eq!(
+ items(&mut ordered),
+ ["gap 4..6 (delivery timeout)", "pending"],
+ "widening returns the rest"
+ );
+ }
+
/// The wait follows the reader's budget: a zero budget waits for nothing, a budget
/// narrowed mid-wait ends it, and an unbounded one waits until the hole resolves.
#[test]
@@ -19012,7 +19683,9 @@ mod test {
.now_or_never()
.expect("should not block")
.expect("would have errored")
- .expect("track should not be closed");
+ .expect("track should not be closed")
+ .group()
+ .expect("a group");
assert_eq!(group.start, 5);
// Seq 3 arrives late, skipped because 3 <= 5.
@@ -19027,26 +19700,14 @@ mod test {
.unwrap()
.finish()
.unwrap();
- // Seq 7 arrives and is returned.
+ // Seq 7 arrives and is returned, after the groups it passed are reported.
producer
.create_group(group::Info { start: 7 })
.unwrap()
.finish()
.unwrap();
- let group = consumer
- .next_group()
- .now_or_never()
- .expect("should not block")
- .expect("would have errored")
- .expect("track should not be closed");
- assert_eq!(group.start, 7);
-
- // No more groups. This would block.
- assert!(
- consumer.next_group().now_or_never().is_none(),
- "should block waiting for a higher key"
- );
+ assert_eq!(items(&mut consumer), ["gap 3..5 (old)", "group 7", "pending"]);
}
#[test]
@@ -19072,7 +19733,9 @@ mod test {
.now_or_never()
.expect("should not block")
.expect("would have errored")
- .expect("track should not be closed");
+ .expect("track should not be closed")
+ .group()
+ .expect("a group");
assert_eq!(group.start, 3);
let group = consumer
@@ -19080,7 +19743,9 @@ mod test {
.now_or_never()
.expect("should not block")
.expect("would have errored")
- .expect("track should not be closed");
+ .expect("track should not be closed")
+ .group()
+ .expect("a group");
assert_eq!(group.start, 5);
}
@@ -19277,7 +19942,9 @@ mod test {
.now_or_never()
.expect("should not block")
.expect("would have errored")
- .expect("track should not be closed");
+ .expect("track should not be closed")
+ .group()
+ .expect("a group");
assert_eq!(group.start, 3);
// The plain handle walks arrivals, so it still starts at seq 5.
@@ -19301,9 +19968,42 @@ mod test {
consumer.set_range(..3);
// Groups 0, 1, 2 are within the cap.
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 0);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 1);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 2);
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 0
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 1
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 2
+ );
// Group 3 is beyond the cap: next_group parks even though cached groups exist.
assert!(
@@ -19327,20 +20027,86 @@ mod test {
}
consumer.set_range(..2);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 0);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 1);
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 0
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 1
+ );
assert!(consumer.next_group().now_or_never().is_none(), "capped at 2");
// Raise the cap; previously-blocked cached groups become available again.
consumer.set_range(..5);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 2);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 3);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 4);
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 2
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 3
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 4
+ );
assert!(consumer.next_group().now_or_never().is_none(), "capped at 5");
// Remove the cap; everything remaining flows.
consumer.set_range(..);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 5);
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 5
+ );
assert!(consumer.next_group().now_or_never().is_none(), "no more groups");
}
@@ -19359,9 +20125,42 @@ mod test {
}
// Drain everything with no cap.
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 0);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 1);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 2);
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 0
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 1
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 2
+ );
// Lower the cap below the cursor. New groups beyond the cap are blocked.
consumer.set_range(..2);
@@ -19382,8 +20181,30 @@ mod test {
// Restoring the cap to no-limit (or any value >= cursor) releases them.
consumer.set_range(..);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 3);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 4);
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 3
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 4
+ );
}
#[test]
@@ -19423,16 +20244,71 @@ mod test {
.unwrap();
// next_group walks in key order through everything <= cap.
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 2);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 3);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 4);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 5);
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 2
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 3
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 4
+ );
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 5
+ );
// Now blocked: 8 is still beyond the cap.
assert!(consumer.next_group().now_or_never().is_none());
// Raise the cap; cached seq 8 is finally served.
consumer.set_range(..11);
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 8);
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 8
+ );
}
/// `recv_group` (arrival order) honors the `end_at` cap like `next_group`: a group
@@ -19690,8 +20566,26 @@ mod test {
group.finish().unwrap();
}
- assert_eq!(consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start, 0);
- let mut last = consumer.next_group().now_or_never().unwrap().unwrap().unwrap();
+ assert_eq!(
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
+ 0
+ );
+ let mut last = consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap();
assert_eq!(last.start, 1);
last = last.with_range(..end.frame);
assert_eq!(
@@ -19732,7 +20626,15 @@ mod test {
.unwrap();
assert!(matches!(producer.append_next(), Err(Error::BoundsExceeded(_))));
assert_eq!(
- consumer.next_group().now_or_never().unwrap().unwrap().unwrap().start,
+ consumer
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap()
+ .start,
u64::MAX - 1
);
// The track's end lands on the reserved key.
@@ -20372,7 +21274,14 @@ mod test {
assert_eq!(cut.start, 0);
assert!(matches!(cut.with_range(200..).read_frame().await, Err(Error::App(5))));
arrival.assert_no_group();
- let cut = ordered.next_group().now_or_never().unwrap().unwrap().unwrap();
+ let cut = ordered
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap();
assert_eq!(cut.start, 0);
assert!(matches!(cut.with_range(200..).read_frame().await, Err(Error::App(5))));
assert!(
@@ -20827,8 +21736,11 @@ mod test {
let mut sub = producer.subscribe(replay().with_range(0..2)).ordered();
let reading = kio::clock::sim::spawn(async move {
let mut read = Vec::new();
- while let Some(group) = sub.next_group().await.unwrap() {
- read.push(group.start);
+ while let Some(item) = sub.next_group().await.unwrap() {
+ read.push(match item {
+ Item::Group(group) => format!("group {}", group.start),
+ Item::Gap { range, .. } => format!("gap {range:?}"),
+ });
}
read
});
@@ -20837,7 +21749,7 @@ mod test {
.expect("the head is asked")
.unwrap();
request.reject(Error::NotFound);
- assert_eq!(reading.await.unwrap(), [1]);
+ assert_eq!(reading.await.unwrap(), ["gap 0..1", "group 1"]);
}
/// A refused mid-group end caps none of its group's copies, even one that fit under it.
@@ -24066,8 +24978,8 @@ mod test {
while let Poll::Ready(Some(key)) = fill.poll_wanted_changed(&waiter) {
write_filled(fill.accept(key, None), key);
served.push(key);
- while let Poll::Ready(Ok(Some(group))) = ordered.poll_next_group(&waiter) {
- read.push(group.start);
+ while let Poll::Ready(Ok(Some(item))) = ordered.poll_next_group(&waiter) {
+ read.push(item.group().expect("a group").start);
}
}
assert_eq!(served, [3, 2, 1, 0]);
@@ -26701,7 +27613,14 @@ mod test {
assert_eq!(arrival.assert_group().start, 1);
assert!(matches!(arrival.recv_group().now_or_never(), Some(Ok(None))));
- let mut first = ordered.next_group().now_or_never().unwrap().unwrap().unwrap();
+ let mut first = ordered
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap();
assert_eq!(first.start, 0);
assert!(matches!(first.read_frame().now_or_never(), Some(Err(Error::App(5)))));
assert_eq!(drain_ordered(&mut ordered), [1]);
@@ -26805,7 +27724,14 @@ mod test {
assert_eq!(drain_ordered(&mut ordered), [0]);
ordered = ordered.with_range(..);
- let mut reoffered = ordered.next_group().now_or_never().unwrap().unwrap().unwrap();
+ let mut reoffered = ordered
+ .next_group()
+ .now_or_never()
+ .unwrap()
+ .unwrap()
+ .unwrap()
+ .group()
+ .unwrap();
assert_eq!(reoffered.start, 1);
assert!(matches!(
reoffered.read_frame().now_or_never(),
diff --git a/rs/moq-net/tests/integration/drop_record.rs b/rs/moq-net/tests/integration/drop_record.rs
index 9b7da2eee2f..f2a7fbd2d22 100644
--- a/rs/moq-net/tests/integration/drop_record.rs
+++ b/rs/moq-net/tests/integration/drop_record.rs
@@ -471,6 +471,71 @@ async fn an_origin_eviction_drop_is_forwarded_final() {
}
}
+/// A group the publisher's origin reclaimed before it was sent reaches an ordered reader on
+/// the far side of a relay as a gap, so the groups it receives plus the spans it is told of
+/// cover every group written, with nothing held waiting for one that will not come.
+#[kio::test]
+async fn an_ordered_reader_is_told_of_every_group_the_origin_evicted() {
+ const EXPIRY: Duration = Duration::from_secs(10);
+ const PAYLOAD: usize = 16 * 1024;
+ const WRITTEN: u64 = 4;
+ for versions in [[LITE06, LITE06], [LITE07, LITE07]] {
+ kio::clock::timeout(TIMEOUT * 10, async {
+ // One stream of credit past the Announce Stream that lite-07 holds: group 0's, held from the relay,
+ // so the rest queue.
+ let credit = if versions == [LITE07, LITE07] { 2 } else { 1 };
+ let chain = Chain::with_credit(versions, [expiring(EXPIRY), forever(), forever()], Some(credit)).await;
+ // A budget bounds the wait on a group a hop judged stale; none would fill it instead.
+ let mut reader = chain
+ .subscribe(reading(Duration::from_secs(3600), [support::keys(0..)]))
+ .await
+ .ordered();
+ chain.upstream.server_transport.hold_unis();
+ for key in 0..WRITTEN {
+ let mut group = chain.source.create_group(moq_net::group::Info { start: key }).unwrap();
+ group
+ .write_frame(Timestamp::from_millis(key * 1000).unwrap(), vec![0u8; PAYLOAD])
+ .unwrap();
+ group.finish().unwrap();
+ }
+ // Nobody reads the queued groups, so the origin reclaims all but the newest.
+ kio::clock::sleep(EXPIRY * 3).await;
+ chain.upstream.server_transport.release_unis();
+ let finished = chain.source.clone().finish().unwrap();
+
+ let mut got = BTreeSet::new();
+ let mut told = BTreeSet::new();
+ while let Some(item) = kio::clock::timeout(TIMEOUT, reader.next_group())
+ .await
+ .expect("the reader stalled on a group that will not come")
+ .expect("the reader runs on")
+ {
+ match item {
+ track::Item::Group(group) => assert!(got.insert(group.start), "group {} twice", group.start),
+ track::Item::Gap { range, .. } => {
+ for key in range {
+ assert!(told.insert(key), "group {key} reported twice");
+ }
+ }
+ }
+ }
+ assert!(got.is_disjoint(&told), "{versions:?}: {got:?} also reported {told:?}");
+ assert!(
+ got.contains(&(WRITTEN - 1)),
+ "{versions:?}: the newest is delivered: {got:?}"
+ );
+ assert_eq!(
+ got.union(&told).copied().collect::>(),
+ (0..WRITTEN).collect::>(),
+ "{versions:?}: delivered {got:?} and reported {told:?}"
+ );
+ drop(finished);
+ })
+ .await
+ .unwrap_or_else(|_| panic!("{versions:?}: timed out"));
+ }
+}
+
/// After a widening from max delay B1 to B2 for a group the publisher no longer holds, its
/// stale drop under B1, still in flight when the relay sends the B2 update, settles no reader;
/// the drop the publisher sends again under the B2 update settles the B2 reader, over lite-07.
diff --git a/rs/moq-net/tests/integration/settled_end_survives_abort.rs b/rs/moq-net/tests/integration/settled_end_survives_abort.rs
index 23a591424ea..539883cebe3 100644
--- a/rs/moq-net/tests/integration/settled_end_survives_abort.rs
+++ b/rs/moq-net/tests/integration/settled_end_survives_abort.rs
@@ -59,7 +59,7 @@ async fn round(abort: bool, ordered: bool) -> (Vec, Option) {
let mut consumer = consumer.ordered();
loop {
match consumer.next_group().await {
- Ok(Some(group)) => got.push(group.start),
+ Ok(Some(item)) => got.push(item.group().expect("a group").start),
Ok(None) => return (got, None),
Err(err) => return (got, Some(err)),
}
diff --git a/rs/moq-net/tests/integration/socket.rs b/rs/moq-net/tests/integration/socket.rs
index c88a1431f42..9fb2165bdd9 100644
--- a/rs/moq-net/tests/integration/socket.rs
+++ b/rs/moq-net/tests/integration/socket.rs
@@ -108,9 +108,10 @@ fn send(track: &track::Producer, payload: &'static [u8]) {
/// Read the next message from a session, or `None` once the peer finished.
async fn recv(reader: &mut track::Ordered) -> Result>, Error> {
kio::clock::timeout(TIMEOUT, async {
- let Some(mut group) = reader.next_group().await? else {
+ let Some(item) = reader.next_group().await? else {
return Ok(None);
};
+ let mut group = item.group().expect("a group, not a gap");
let frame = group.read_frame().await?.expect("a frame");
Ok(Some(frame.payload.to_vec()))
})
diff --git a/rs/moq-relay/tests/integration/quic_sim.rs b/rs/moq-relay/tests/integration/quic_sim.rs
index 68f588923f0..c31b6c9833f 100644
--- a/rs/moq-relay/tests/integration/quic_sim.rs
+++ b/rs/moq-relay/tests/integration/quic_sim.rs
@@ -283,7 +283,12 @@ fn a_socket_session_carries_a_bulk_transfer_losslessly() {
};
let read = async {
let mut ordered = dialed.consumer.ordered();
- let mut group = ordered.next_group().await.unwrap().expect("a group");
+ let mut group = ordered
+ .next_group()
+ .await
+ .unwrap()
+ .and_then(moq_net::track::Item::group)
+ .expect("a group");
let mut read = 0;
while let Some(frame) = group
.read_frame()
diff --git a/rs/moq-relay/tests/integration/socket_auth.rs b/rs/moq-relay/tests/integration/socket_auth.rs
index d106423008b..a0b94295e0a 100644
--- a/rs/moq-relay/tests/integration/socket_auth.rs
+++ b/rs/moq-relay/tests/integration/socket_auth.rs
@@ -175,7 +175,12 @@ fn send(track: &moq_net::track::Producer, payload: &'static [u8]) {
/// Read the next message of a session.
async fn recv(reader: &mut moq_net::track::Ordered) -> Vec {
kio::clock::timeout(TIMEOUT, async {
- let mut group = reader.next_group().await.expect("group").expect("a group");
+ let mut group = reader
+ .next_group()
+ .await
+ .expect("group")
+ .and_then(moq_net::track::Item::group)
+ .expect("a group");
let frame = group.read_frame().await.expect("frame").expect("a frame");
frame.payload.to_vec()
})
diff --git a/rs/moq-rtmp/src/dial.rs b/rs/moq-rtmp/src/dial.rs
index 30f923e2cd9..3d51c3be4db 100644
--- a/rs/moq-rtmp/src/dial.rs
+++ b/rs/moq-rtmp/src/dial.rs
@@ -1274,6 +1274,7 @@ mod tests {
.await
.expect("catalog read timed out")
.expect("catalog read")
+ .and_then(moq_net::track::Item::group)
.expect("a catalog group");
let frame = group
.read_frame()
diff --git a/rs/moq-stats/src/produce.rs b/rs/moq-stats/src/produce.rs
index 1120aae38bb..461ac877258 100644
--- a/rs/moq-stats/src/produce.rs
+++ b/rs/moq-stats/src/produce.rs
@@ -1927,7 +1927,7 @@ mod tests {
/// The same, without blocking: `None` once nothing more is buffered.
fn try_next_frame(track: &mut track::Ordered) -> Option {
use futures::FutureExt;
- let mut group = track.next_group().now_or_never()?.expect("ok")?;
+ let mut group = track.next_group().now_or_never()?.expect("ok")?.group()?;
group.read_frame().now_or_never()?.expect("ok")
}
@@ -3202,7 +3202,7 @@ mod tests {
let mut track = subscribe(broadcast, &name).await;
let mut first = None;
while let Some(group) = futures::FutureExt::now_or_never(track.next_group()) {
- let start = group.unwrap().unwrap().start;
+ let start = group.unwrap().unwrap().group().expect("a group").start;
first = Some(first.map_or(start, |first: u64| first.min(start)));
}
let first = first.expect("a run's first group is listed");
diff --git a/rs/moq-tokio/tests/integration/receive_credit.rs b/rs/moq-tokio/tests/integration/receive_credit.rs
index 3a392eb02ef..103138562dd 100644
--- a/rs/moq-tokio/tests/integration/receive_credit.rs
+++ b/rs/moq-tokio/tests/integration/receive_credit.rs
@@ -1028,10 +1028,10 @@ const INFLIGHT: usize = 1 << 20;
/// The receiver's storage segment (`moq_net`'s crate-private `frame::SEGMENT`): a frame within
/// one keeps its credit until it completes, and a larger one returns each segment's as it seals.
const SEGMENT: usize = 1 << 20;
-/// Why cells whose incomplete units overcommit the window are skipped: the quest that makes
-/// `track::Subscriber` report each group it abandons, which their readers need to return its
+/// Why cells whose incomplete units overcommit the window are skipped: their readers must
+/// read through `track::Ordered`, which reports each group it abandons, to return its
/// in-flight credit.
-const OVERCOMMITTED: &str = "waits on quest/m2/group-drops.md";
+const OVERCOMMITTED: &str = "waits on quest/m3/receive-windows-evicted.md";
/// One benchmark configuration.
#[derive(Clone, Copy)]
@@ -1200,8 +1200,8 @@ async fn measure(cell: Cell, profile: Option, bytes: usize)
///
/// Cells whose units overcommit the window are skipped, each printed with [`OVERCOMMITTED`]
/// as the reason. They should keep their offered load and report what eviction cost them, but
-/// the subscriber can abandon a group without handing it out, so its in-flight credit never
-/// returns and the track's writer stalls.
+/// the reader abandons a group without handing it out, so its in-flight credit never returns
+/// and the track's writer stalls, until it reads the gaps `track::Ordered` reports.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "release benchmark, run by nightly"]
async fn benchmark_receive_windows() {
diff --git a/rs/moq-transcode/src/lib.rs b/rs/moq-transcode/src/lib.rs
index 8bd5f4075c8..c378a74280e 100644
--- a/rs/moq-transcode/src/lib.rs
+++ b/rs/moq-transcode/src/lib.rs
@@ -510,7 +510,7 @@ mod tests {
group.finish().unwrap();
for (name, subscriber) in &mut subscribers {
- let mut group = subscriber.next_group().await.unwrap().unwrap();
+ let mut group = subscriber.next_group().await.unwrap().unwrap().group().unwrap();
assert_eq!(group.start, sequence, "{name} changed the group start");
let mut received = Vec::new();
while let Some(payload) = group.read_frame().await.unwrap() {
@@ -1017,7 +1017,7 @@ mod tests {
.await
.unwrap()
.ordered();
- let mut group = subscriber.next_group().await.unwrap().unwrap();
+ let mut group = subscriber.next_group().await.unwrap().unwrap().group().unwrap();
assert!(group.start <= 1, "unexpected sequence {}", group.start);
let payload = group.read_frame().await.unwrap().unwrap();
let frame = hang::container::Frame::decode(payload.payload).unwrap();
@@ -1120,7 +1120,11 @@ mod tests {
.await
.unwrap()
.ordered();
- let mut group = live.next_group().await.unwrap().unwrap();
+ let mut group = loop {
+ if let Some(group) = live.next_group().await.unwrap().unwrap().group() {
+ break group;
+ }
+ };
assert!(group.read_frame().await.unwrap().is_some());
transcoder.abort();
@@ -1169,7 +1173,7 @@ mod tests {
assert!(update.encoding);
// Real frames, so the counters are counting encoding rather than intent.
- let mut group = subscriber.next_group().await.unwrap().unwrap();
+ let mut group = subscriber.next_group().await.unwrap().unwrap().group().unwrap();
group.read_frame().await.unwrap().unwrap();
assert!(rendition.frames() > 0);
assert!(rendition.bytes() > 0);
diff --git a/rs/moq-transcode/src/rung.rs b/rs/moq-transcode/src/rung.rs
index 94e1b800585..0624008180d 100644
--- a/rs/moq-transcode/src/rung.rs
+++ b/rs/moq-transcode/src/rung.rs
@@ -893,7 +893,11 @@ async fn drop_gap(
let mut groups = track.subscribe(subscription).await?.ordered();
// The newest source group below the miss, as the output start it maps to.
let mut before = None;
- while let Some(group) = groups.next_group().await? {
+ while let Some(item) = groups.next_group().await? {
+ // A group given up on is no source group.
+ let moq_net::track::Item::Group(group) = item else {
+ continue;
+ };
if group.start >= miss {
return Ok::<_, moq_net::Error>((before, output_start(group.start, scale)));
}
diff --git a/test/interop/clients/rust/hls.rs b/test/interop/clients/rust/hls.rs
index 001f2e2b523..80e3fe17a8e 100644
--- a/test/interop/clients/rust/hls.rs
+++ b/test/interop/clients/rust/hls.rs
@@ -145,7 +145,11 @@ async fn main() -> anyhow::Result<()> {
.with_delay(moq_net::track::Delay::default().with_max(Duration::from_secs(3600)));
let mut groups = broadcast.track(&name)?.subscribe(replay).await?.ordered();
let mut out = Vec::new();
- while let Some(mut group) = groups.next_group().await? {
+ while let Some(item) = groups.next_group().await? {
+ // A group given up on starts no segment.
+ let moq_net::track::Item::Group(mut group) = item else {
+ continue;
+ };
let Some(frame) = group.read_frame().await? else {
continue;
};
diff --git a/test/interop/clients/rust/socket-js.rs b/test/interop/clients/rust/socket-js.rs
index 5fd37c47a8f..ab38cd224a5 100644
--- a/test/interop/clients/rust/socket-js.rs
+++ b/test/interop/clients/rust/socket-js.rs
@@ -56,7 +56,11 @@ async fn main() -> anyhow::Result<()> {
if dial {
stream.producer.write_frame(key(i), vec![own + i])?;
}
- let mut group = reader.next_group().await?.context("short track half")?;
+ let mut group = reader
+ .next_group()
+ .await?
+ .and_then(moq_net::track::Item::group)
+ .context("short track half")?;
ensure!(group.start == u64::from(i), "wrong group key");
let frame = group.read_frame().await?.context("empty track message")?;
ensure!(frame.payload.as_ref() == [peer + i], "wrong track frame");
@@ -80,7 +84,11 @@ async fn main() -> anyhow::Result<()> {
ensure!(reset_reader.next_group().await.is_err(), "track RST became FIN");
eprintln!("socket reset verified dial");
} else {
- let mut group = reset_reader.next_group().await?.context("missing reset signal")?;
+ let mut group = reset_reader
+ .next_group()
+ .await?
+ .and_then(moq_net::track::Item::group)
+ .context("missing reset signal")?;
ensure!(
group
.read_frame()
diff --git a/test/interop/clients/rust/socket.rs b/test/interop/clients/rust/socket.rs
index 3734fbca69b..060f4f6c818 100644
--- a/test/interop/clients/rust/socket.rs
+++ b/test/interop/clients/rust/socket.rs
@@ -62,7 +62,11 @@ async fn main() -> anyhow::Result<()> {
}
let _producer = producer.finish()?;
for start in 0..3 {
- let mut next = consumer.next_group().await?.context("short track")?;
+ let mut next = consumer
+ .next_group()
+ .await?
+ .and_then(moq_net::track::Item::group)
+ .context("short track")?;
ensure!(next.start == start, "wrong group order");
group(&mut next).await?;
}