Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ removal, a rename, or a wire, catalog, or format change that is free until
gains its reader, reversing 2026-11-14; and
[lite-07 by start](/quest/m2/time-addressing/lite07.md) carries the
smaller wire fixes. [Refresh](/quest/m2/refresh-interval.md) and
[group drops](/quest/m2/group-drops.md) took simpler contracts.
group drops took simpler contracts.
[moq-transport time addressing](/quest/m2/ietf-time-addressing.md)
returned from m4, and the two moq-transport interop quests in m3 were
deleted. Kept as they are: datagram tracks with their lite-05/06 bridge,
Expand Down Expand Up @@ -322,7 +322,7 @@ what would need a breaking release had it landed right after the cut:
the routes epic and gates the cut, per the 2026-10-13 rule that routes'
tier and partition tests run before it (maintainer; rejected: moving it to
m3 and fixing late routing flaws in lite-08).
- ✅ [Group drops](/quest/m2/group-drops.md) split: its lite-07 wire question
- ✅ Group drops split: its lite-07 wire question
was its own quest, settled without a wire-format change in
[moq-dev/moq#8774](https://github.com/moq-dev/moq/pull/8774); the public
`Ordered` API waits for the model.
Expand Down
83 changes: 0 additions & 83 deletions quest/m2/group-drops.md

This file was deleted.

2 changes: 1 addition & 1 deletion quest/m2/lite07-finalize.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ would need a breaking release had it landed right after the cut:
waits on nixpkgs as well as the cut; dropping the check, which would find a
transcript flaw only after the freeze.
- ✅ The wire half of group drops split out as its own quest, which this
waited on in place of [the public drop signal](/quest/m2/group-drops.md),
waited on in place of the public drop signal,
an API change that doesn't touch the wire. It settled without a
wire-format change ([moq-dev/moq#8774](https://github.com/moq-dev/moq/pull/8774)):
both publishers already name a Group Stream the subscriber stopped in
Expand Down
3 changes: 1 addition & 2 deletions quest/m2/time-addressing/lite07.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ is free only until lite-07 freezes:
- ✅ SUBSCRIBE_DROP names a span and carries no group count (rejected: a
count, which would also have let SUBSCRIBE_SKIP become a drop with a
count of 0). Today's `Group Count` is a count of keys, so it becomes the
span's end. [Group drops](/quest/m2/group-drops.md) reports the span.
span's end. `track::Ordered`'s `Gap` reports the span.

Recommendations for the same PR, not decided:

Expand Down Expand Up @@ -188,5 +188,4 @@ TRACKS_ADD change, and the Padding stream and datagram types go.
## Related

- [YIELDED deferred](/quest/m2/yield-defer.md) - removes continuations, the last use of GROUP's `Frame Start`
- [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - reads the spans SUBSCRIBE_DROP names
- [Refresh a track](/quest/m2/refresh-interval.md) - raises the `Wanted Group` this turns into a start time
5 changes: 2 additions & 3 deletions quest/m2/unused-api/net.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ Not in this sweep, with their owners:

- `Subscriber::next_in_range`, `is_skipped`, and `poll_start` have one
consumer, moq-mux's container reader, which orders groups itself.
[Group drops](/quest/m2/group-drops.md) adds `Gap` to `track::Ordered`;
once the container reader moves onto it these three can go private.
`track::Ordered` now reports a `Gap`; once the container reader moves onto it
these three can go private.
- `socket::is_endpoint` leaves with
[socket endpoints](/quest/m2/sockets/remove.md).
- `origin::Producer::update_hold`, `with_proxy`, `Source`, `source()`,
Expand All @@ -82,4 +82,3 @@ Public API: breaking, removals only (`Position::after`, `after_group`,
## Related

- [moq-net's node, version, handshake, and route priority](/quest/m2/net-renames.md) - makes `Version` opaque
- [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - gives the container reader the `Gap` that frees three more methods
28 changes: 16 additions & 12 deletions quest/m3/receive-windows-evicted.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ overcommitted cells again, and Nightly's `Rust benchmark tests` job
(`gh nightly rust-test`) still passes. Those cells are 256 streams,
4 MiB frames, and 4 MiB frames with a following stream window. Until
then, the benchmark skips them and prints each one with
`quest/m2/group-drops.md` as the reason.
`quest/m3/receive-windows-evicted.md` as the reason.

Every cell keeps the workload it offers. A cell whose incomplete units
fit the connection window fails on any eviction. A cell that
Expand All @@ -24,8 +24,8 @@ pulling the m2 blocker into m1). It ranks right after m2, when it can start.

Moved to m1 2026-11-04 with the maintainer, after
[moq-dev/moq#7838](https://github.com/moq-dev/moq/pull/7838) landed the
fitting cells: what remains waits on
[group-drops](/quest/m2/group-drops.md), an m2 quest, and with the
fitting cells: what remained waited on
`quest/m2/group-drops.md`, an m2 quest, and with the
overcommitted cells skipped the benchmark no longer fails Nightly, so this
no longer holds the m0
[Nightly passes on main](/quest/m0/nightly/README.md) epic open. It replaces
Expand All @@ -42,7 +42,7 @@ Overload policy, decided 2026-11-04 (maintainer):

Silent loss, decided 2026-11-04 (maintainer):

- ✅ Fix it at the source in `quest/m2/group-drops.md`, planned in
- ✅ Fix it at the source in `quest/m2/group-drops.md` (landed), planned in
[moq-dev/moq#7890](https://github.com/moq-dev/moq/pull/7890). There,
the wire always names an abandoned group, and `track::Subscriber`'s
drop signal becomes public.
Expand Down Expand Up @@ -84,10 +84,18 @@ What remains:
(traced as `fetch abandoned`). Such a group's in-flight permit never
returns, so the writer stalls and the reader times out. The 4 MiB-frame
cell hit this on every run, and the 256-stream cell intermittently.
- Once group-drops lands, the reader returns a dropped group's permit and
counts the group as lost. Then remove the skip (`OVERCOMMITTED` and the
`Cell::fits` partition in `benchmark_receive_windows`), and run the full
sweep through `quest gate`.
- The drop signal landed in `quest/m2/group-drops.md` as `track::Item::Gap`
on `track::Ordered`, which the reader must read through: its subscription
anchors at group 0 (`Anchor::Time(0)`), and `next_group` returns each
group or the span given up. The reader returns a permit per group in a
span (the writer's `sent` map holds the keys) and counts them lost. A
scratch run found two things for this PR to settle: `Ordered` reports a
group landing after a higher one was returned as `Old`, so a fitting cell
loses reordered frames with no eviction and `measure`'s "lost a frame
without an eviction" check must count `Old` gaps; and the default cell
runs only once the reader is anchored at group 0. Then remove the skip
(`OVERCOMMITTED` and the `Cell::fits` partition in
`benchmark_receive_windows`), and run the full sweep through `quest gate`.
- Retain the stream/frame/window axes and the actual offered workload
for a shared receive budget, if one is planned again.
- Repro: `just rs test -p moq-tokio --test integration --release -E
Expand All @@ -101,7 +109,3 @@ connection-wide admission semaphore are recorded in
[moq-dev/moq#7838](https://github.com/moq-dev/moq/pull/7838).

Public API: none. Wire: none.

## Required

- [A group a subscriber gives up on is always reported](/quest/m2/group-drops.md) - the public drop signal the reader needs to return a dropped group's permit
40 changes: 24 additions & 16 deletions rs/moq-archive/src/consumer/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -144,21 +144,27 @@ fn from(start: u64) -> track::Range {
}
}

/// The next group, past the gaps the read reports for what the archive never held.
async fn group(reader: &mut track::Ordered) -> Option<moq_net::group::Consumer> {
loop {
let item = timeout(Duration::from_secs(10), reader.next_group())
.await
.expect("a group arrives")
.unwrap()?;
if let track::Item::Group(group) = item {
return Some(group);
}
}
}

/// The next group's start, without waiting for it to end.
async fn sequence(reader: &mut track::Ordered) -> Option<u64> {
timeout(Duration::from_secs(10), reader.next_group())
.await
.expect("a group arrives")
.unwrap()
.map(|group| group.start)
group(reader).await.map(|group| group.start)
}

/// The next group and every frame in it, once the group ends.
async fn next(reader: &mut track::Ordered) -> Option<(u64, Vec<String>)> {
let mut group = timeout(Duration::from_secs(10), reader.next_group())
.await
.expect("a group arrives")
.unwrap()?;
let mut group = group(reader).await?;
let mut frames = Vec::new();
while let Some(frame) = timeout(Duration::from_secs(10), group.read_frame())
.await
Expand Down Expand Up @@ -226,7 +232,7 @@ async fn a_recording_round_trips() {

let mut reader = subscribe(&broadcast, "video").await;
for sequence in 0..6 {
let mut group = reader.next_group().await.unwrap().unwrap();
let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert_eq!(group.start, sequence);
for index in 0..3 {
let frame = group.read_frame().await.unwrap().unwrap();
Expand Down Expand Up @@ -450,7 +456,7 @@ async fn an_archive_without_markers_stays_open() {
let (broadcast, _run) = serve(&archive.store).await;
let mut reader = subscribe(&broadcast, "video").await;
assert_eq!(next(&mut reader).await.unwrap(), (0, expected(true, 0, 0..1)));
let mut group = reader.next_group().await.unwrap().unwrap();
let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert!(group.read_frame().await.unwrap().is_some());
let pending = timeout(30 * REFRESH, group.read_frame()).await;
assert!(pending.is_err(), "the newest group stays open");
Expand All @@ -466,7 +472,7 @@ async fn closing_the_broadcast_cancels_the_live_tracks() {

let (broadcast, run) = serve(&archive.store).await;
let mut reader = subscribe(&broadcast, "video").await;
let mut group = reader.next_group().await.unwrap().unwrap();
let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert!(group.read_frame().await.unwrap().is_some());

broadcast.close();
Expand All @@ -485,7 +491,7 @@ async fn a_gapped_continuation_aborts_the_open_group() {

let (broadcast, run) = serve(&archive.store).await;
let mut reader = subscribe(&broadcast, "video").await;
let mut group = reader.next_group().await.unwrap().unwrap();
let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert!(group.read_frame().await.unwrap().is_some());

// Frame 1 of group 0 is missing.
Expand All @@ -508,7 +514,7 @@ async fn a_store_failure_aborts_the_served_tracks() {

let (broadcast, run) = serve(&archive.store).await;
let mut reader = subscribe(&broadcast, "video").await;
let mut group = reader.next_group().await.unwrap().unwrap();
let mut group = reader.next_group().await.unwrap().unwrap().group().unwrap();
assert!(group.read_frame().await.unwrap().is_some());

archive.store.inner().fail_lists();
Expand Down Expand Up @@ -564,7 +570,7 @@ async fn a_completed_archive_ends_as_recorded() {
let end = timeout(Duration::from_secs(10), reader.next_group())
.await
.expect("the track ends");
let end = end.map(|group| group.map(|group| group.start));
let end = end.map(|item| item.map(|item| item.group().expect("a group").start));
assert!(
matches!(end, Err(moq_net::Error::Stream(moq_net::StreamError::App(7)))),
"{end:?}"
Expand Down Expand Up @@ -697,7 +703,9 @@ async fn read_range(broadcast: &broadcast::Producer, track: &str, range: track::
let mut read = Vec::new();
while let Ok(group) = timeout(5 * REFRESH, reader.next_group()).await {
match group.unwrap() {
Some(group) => read.push(group.start),
Some(track::Item::Group(group)) => read.push(group.start),
// Nothing is stored below the first group, which the read reports as a gap.
Some(track::Item::Gap { .. }) => {}
None => break,
}
}
Expand Down
10 changes: 7 additions & 3 deletions rs/moq-archive/src/producer/track.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ use std::time::Duration;

use bytes::Bytes;
use kio::clock::Instant;
use moq_net::track::{Anchor, Delay, Order, Position, Range, Subscription};
use moq_net::track::{Anchor, Delay, Item, Order, Position, Range, Subscription};
use moq_net::{StreamError, Timescale, group};
use object_store::path::Path;
use object_store::{ObjectStore, ObjectStoreExt, PutMode, PutPayload};
Expand Down Expand Up @@ -218,8 +218,12 @@ impl<S: ObjectStore> Track<S> {
}
}
None => {
if let Poll::Ready(next) = ordered.poll_next_group(waiter) {
return Poll::Ready(Event::Next(next));
// A group the track gave up on is stored as the hole it leaves.
while let Poll::Ready(next) = ordered.poll_next_group(waiter) {
match next {
Ok(Some(Item::Gap { .. })) => continue,
next => return Poll::Ready(Event::Next(next.map(|item| item.and_then(Item::group)))),
}
}
}
}
Expand Down
28 changes: 24 additions & 4 deletions rs/moq-bench/src/connection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -896,7 +896,12 @@ mod tests {
.await
.unwrap()
.ordered();
let mut group = sub.next_group().await.unwrap().expect("a group");
let mut group = sub
.next_group()
.await
.unwrap()
.and_then(moq_net::track::Item::group)
.expect("a group");

let keyframe = group.read_frame().await.unwrap().expect("keyframe");
let header: serde_json::Value = serde_json::from_slice(&keyframe.payload).unwrap();
Expand Down Expand Up @@ -943,7 +948,12 @@ mod tests {
.await
.unwrap()
.ordered();
let mut group = sub.next_group().await.unwrap().expect("a group");
let mut group = sub
.next_group()
.await
.unwrap()
.and_then(moq_net::track::Item::group)
.expect("a group");

// Just the keyframe, then the group ends.
assert!(group.read_frame().await.unwrap().is_some(), "keyframe");
Expand Down Expand Up @@ -975,7 +985,12 @@ mod tests {
tokio::time::advance(Duration::from_millis(250)).await;

let mut sub = consumer.track(TRACK).unwrap().subscribe(None).await.unwrap().ordered();
let mut group = sub.next_group().await.unwrap().expect("a group");
let mut group = sub
.next_group()
.await
.unwrap()
.and_then(moq_net::track::Item::group)
.expect("a group");
let keyframe = group.read_frame().await.unwrap().expect("keyframe").payload;

assert_eq!(keyframe.len(), 300, "keyframe padded to the rolled frame size");
Expand Down Expand Up @@ -1008,7 +1023,12 @@ mod tests {
tokio::time::advance(Duration::from_millis(250)).await;

let mut sub = consumer.track(TRACK).unwrap().subscribe(None).await.unwrap().ordered();
let mut group = sub.next_group().await.unwrap().expect("a group");
let mut group = sub
.next_group()
.await
.unwrap()
.and_then(moq_net::track::Item::group)
.expect("a group");
let keyframe = group.read_frame().await.unwrap().expect("keyframe").payload;

assert!(keyframe.len() > 50, "header is the floor; no truncation to fit");
Expand Down
Loading