Skip to content
Merged
2 changes: 1 addition & 1 deletion quest/m1/compress/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ This README owns the end-to-end proof and the guide once every child lands:

## Required

- [Mid-group subscriptions widen](/quest/m1/compress/app-range-widen.md) - an application range starting mid-group on a compressed track widens to frame 0 at the subscriber instead of failing with `Lagged`
- [moq-transport mid-group subscriptions widen](/quest/m1/compress/ietf-range-widen.md) - an application range starting mid-group on a compressed track widens to frame 0 at a moq-transport subscriber instead of failing with `Lagged`
- [Migrate](/quest/m1/compress/migrate.md) - catalogs, stats, and timelines publish one compressed track; moq-flate, @moq/flate, and every `.z` name are gone

## Related
Expand Down
31 changes: 0 additions & 31 deletions quest/m1/compress/app-range-widen.md

This file was deleted.

58 changes: 58 additions & 0 deletions quest/m1/compress/ietf-range-widen.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# [M] A moq-transport mid-group subscription to a compressed track widens to frame 0

## Goal

An application range starting mid-group on a compressed track, requested over
moq-transport draft-17+ with the Flate extension negotiated, reads the frames
it asked for instead of failing with `Lagged` against a publisher that serves
exactly the requested Object range. The requesting IETF subscriber widens the
SUBSCRIBE start to Object 0 when it lacks the group's prefix, as lite-07 does
since [moq-dev/moq#6242](https://github.com/moq-dev/moq/pull/6242). The server
never widens and keeps serving exactly what is asked.

## Plan

The rule is the [epic](/quest/m1/compress/README.md)'s Ranges rule, decided
with the maintainer on 2026-10-20: the subscriber widens, never the server;
one that holds frames 0..N-1 still asks from N; the reader still starts at the
frame it asked for. Lite-07 applies it through `track::Producer::widen_compressed`
(`rs/moq-net/src/model/track.rs`), which a warm IETF copy can reuse.

Split from #6242 because a two-hop matrix (lite-07 and moq-transport-17, 18
and 22 on either hop) found three gaps under the IETF requester that are not
about compression:

- A cold SUBSCRIBE (`Target::Request` in `rs/moq-net/src/ietf/subscriber.rs`)
learns the track is compressed only from SUBSCRIBE_OK. Widening every cold
mid-group start on a Flate session would make plain tracks pay for the
prefix too. Decide between learning the properties first (TRACK_STATUS where
`TrackStatusOk::describes_track` holds) and widening cold starts only on
Flate sessions.
- Readers get IETF-received groups whole: `track::Subscriber::deliver` skips
the frame window when `group.relayed()`, since Object IDs may not be frame
ordinals. An application on an IETF hop therefore already reads frames below
its start on plain tracks, and widening would hand it more. Filter by
ordinal when the copy's Object IDs are dense, or by Object ID.
- REQUEST_UPDATE carries no range, so a live range change never reaches
upstream; [moq-transport ranges](/quest/m1/subscribe-ranges/ietf.md) owns
that.

In the matrix, our own IETF publisher served the whole group for a mid-group
Absolute start, so the `Lagged` failure needs a publisher that honors the
exact start; test against the publisher model directly.

`js/net` needs no mirror: its subscriptions carry group ranges only
(`Subscription.groups`), so a JS SUBSCRIBE always starts at frame 0, and a
compressed group that does not is refused with `TooFarBehind`.

Regression: extend
`application_ranges_keep_their_filter_after_requesting_the_compressed_prefix`
(`rs/moq-net/tests/integration/compress.rs`) over moq-transport-17, 18 and 22
hops.

Public API: none. Wire: none.

## Related

- [moq-transport ranges](/quest/m1/subscribe-ranges/ietf.md) - carries a live range change upstream on REQUEST_UPDATE
- [Bounded tails](/quest/m1/subscribe-ranges/bounded-tail.md) - a copy cut at a request's end must accept its tail before a warm prefix can be extended
1 change: 1 addition & 0 deletions quest/m1/subscribe-ranges/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ every sequence delivered or dropped.
- [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) - the lite-07 wire, Rust publisher and subscriber, and the draft
- [moq-transport ranges](/quest/m1/subscribe-ranges/ietf.md) - range FETCH upstream per missing run, and non-blocking FETCH served downstream
- [JS ranges](/quest/m1/subscribe-ranges/js.md) - `@moq/net` model and lite-07 parity
- [Bounded tails](/quest/m1/subscribe-ranges/bounded-tail.md) - a copy cut at a request's mid-group end accepts the rest of its group instead of rejecting it as `Duplicate`

## Related

Expand Down
42 changes: 42 additions & 0 deletions quest/m1/subscribe-ranges/bounded-tail.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# [M] A copy that stopped at a request's end bound accepts its tail

## Goal

A group received for a request whose end falls mid-group stays open to the
rest of that group. When later demand asks the same sequence past that end,
the tail is accepted instead of rejected as `Duplicate`, so a second
subscriber on the same relay reads the frames it asked for. This holds for
plain and compressed tracks on every version that carries frame bounds.

## Plan

Found on [moq-dev/moq#6242](https://github.com/moq-dev/moq/pull/6242) while
answering Codex's
[P1](https://github.com/moq-dev/moq/pull/6242#discussion_r4345457940): through
one relay, a subscriber reads group 5 frames 0..3 and a second then asks for
3..5. The tail never arrives, on a plain track and on a compressed one alike.

- `GroupRecv` (`rs/moq-net/src/lite/subscriber.rs`) calls `group.finish()`
when a group stream ends cleanly, including one that ended only because the
request's end bound was reached. The truncated copy then looks complete.
- `claim_sequence` and `covering_group` (`rs/moq-net/src/model/track.rs`)
judge a copy only by its first frame, so every later stream for that
sequence is `Duplicate`, whatever frame it was asked from.
- Fix it in the receive path: a stream that stopped at the request's end
bound is not the group's end. Weigh leaving the copy unfinished so a tail
appends from its end (one copy; a compressed group keeps inflating against
its own prefix, which `widen_compressed` relies on when it keeps a request at
N) against letting a bounded copy stop covering a request past its end so
the new stream replaces it (compressed requests would then widen to frame 0
again).
- A peer must not be able to pass off an early FIN as reaching the bound.

Test on mock time through a relay: a bounded first subscriber, then a second
asking past its end, on plain and compressed lite-07 tracks, and on
moq-transport draft-20+ if its Absolute end reaches the same path.

Public API: none expected. Wire: none.

## Related

- [Compressed tracks](/quest/m1/compress/README.md) - its Ranges rule keeps a warm request at N, which this makes reachable
42 changes: 41 additions & 1 deletion rs/moq-net/benches/codec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -68,5 +68,45 @@ fn bench(c: &mut Criterion) {
group.finish();
}

criterion_group!(benches, bench);
/// An outgoing request reads the counted aggregate and checks only its named group,
/// independent of the audience beside it and the cache's unrelated groups.
fn compressed_request(c: &mut Criterion) {
use moq_net::{broadcast, track};
use std::hint::black_box;
let mut bench = c.benchmark_group("codec_compressed_request");
for groups in [1, 64, 1024] {
for readers in [1, 64, 1024] {
for held in [false, true] {
let broadcast = broadcast::Info::default().produce();
let track = broadcast
.create_track("compressed", track::Info::default().with_compress(true))
.unwrap();
for _ in 0..groups {
let mut group = track.append_group().unwrap();
group.write_frame(None, bytes::Bytes::from_static(b"prefix")).unwrap();
group.finish().unwrap();
}
let sequence = if held { groups - 1 } else { groups };
let subscription = track::Subscription::default().with_ranges([track::Range {
start: track::Anchor::Absolute(track::Position {
group: sequence,
frame: 1,
}),
end: None,
}]);
let _readers: Vec<_> = (0..readers).map(|_| track.subscribe(subscription.clone())).collect();
bench.bench_function(BenchmarkId::new(format!("held_{held}_{readers}r"), groups), |b| {
b.iter(|| {
let mut subscription = track.subscription().unwrap();
moq_net::fuzz::widen_compressed(&track, &mut subscription);
black_box(subscription);
});
});
}
}
}
bench.finish();
}

criterion_group!(benches, bench, compressed_request);
criterion_main!(benches);
5 changes: 5 additions & 0 deletions rs/moq-net/src/fuzz.rs
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,11 @@ pub fn object_group(frames: usize, properties: bool) -> (broadcast::Producer, tr
(broadcast, track, group)
}

/// Normalize a compressed wire request for the private requesting-model benchmark.
pub fn widen_compressed(track: &track::Producer, subscription: &mut track::Subscription) {
track.widen_compressed(subscription);
}

/// Every target, keyed by the name of its `fuzz_targets/<name>.rs` shim.
pub const TARGETS: &[(&str, Target)] = &[
("lite", lite_wire),
Expand Down
49 changes: 49 additions & 0 deletions rs/moq-net/src/lite/subscriber.rs
Original file line number Diff line number Diff line change
Expand Up @@ -4308,6 +4308,54 @@ mod tests {
}
}

/// A partial prefix, a tail, or a released group cannot seed the inflater. Only
/// a complete live prefix permits requesting the tail without widening its start.
#[moq_clock::test]
async fn compressed_frame_bounds_require_the_entire_cached_prefix() {
for (first, count, aborted, expected) in [
(0, 0, false, 0),
(0, 2, false, 0),
(0, 3, false, 3),
(1, 3, false, 0),
(0, 3, true, 0),
] {
let mut h = Harness::new(Version::Lite07);
h.producer = h
._broadcast
.create_track("compressed", track::Info::timed().with_compress(true))
.unwrap();
let mut group = h.producer.create_group(crate::group::Info { sequence: 5 }).unwrap();
group.start_at(first).unwrap();
for frame in first..first + count {
group
.write_frame(crate::Timestamp::from_millis(frame).unwrap(), b"prefix".as_ref())
.unwrap();
}
if aborted {
group.abort(Error::Cancel).unwrap();
} else {
group.finish().unwrap();
}
let mut sub = Sub::None;
h.serve
.handle_subscription(&mut h.producer, &mut sub, Some(mid_group_demand()), true)
.await
.unwrap();
let wire = h.wire();
let mut wire = wire.as_slice();
assert_eq!(
crate::coding::decode_buf(&mut wire, Version::Lite07, lite::ControlType::decode).unwrap(),
lite::ControlType::Subscribe
);
let msg = crate::coding::decode_buf(&mut wire, Version::Lite07, lite::Subscribe::decode).unwrap();
assert_eq!(
(msg.start_group, msg.start_frame, msg.end_group, msg.end_frame),
(Some(5), expected, Some(5), Some(7)),
"first={first}, count={count}, aborted={aborted}"
);
}
}

/// The model's exclusive end maps back to the wire's inclusive pair.
///
/// The two disagree deliberately (see [`Position::after_group`]), so this pins the seam:
Expand Down Expand Up @@ -6217,6 +6265,7 @@ impl<S: crate::transport::poll::Session> TrackServe<S> {
// The wire carries one range: the span of every range asked for, which the
// subscribers' own reads narrow again.
subscription.set_bounds(subscription.start(), subscription.end());
producer.widen_compressed(&mut subscription);
self.widen_frame_bounds(&mut subscription);
match sub {
Sub::None => {
Expand Down
7 changes: 7 additions & 0 deletions rs/moq-net/src/model/group.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1932,6 +1932,13 @@ impl Handle {
state.abort.is_none().then_some(state.offset)
}

/// Whether this copy still holds every committed frame below `end`, so a compressed
/// tail requested there can inflate against it. Sparse object IDs are not ordinals.
pub(super) fn holds_prefix(&self, end: u64) -> bool {
let state = self.state.read();
state.abort.is_none() && state.offset == 0 && !state.sparse() && state.committed as u64 >= end
}

/// One past the last frame committed to an unfinished group, when that is past its
/// first: where a replacement route resumes. `None` once the group is finished, or
/// while it holds nothing a replacement could continue.
Expand Down
89 changes: 89 additions & 0 deletions rs/moq-net/src/model/track.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3340,6 +3340,45 @@ impl Producer {
poll_combined_changed(&subs, bound, &mut self.prev_subscription, waiter).map(Ok)
}

/// Widen outgoing compressed ranges that cannot inflate from a cached prefix.
/// Only requesters call this: the original preferences keep filtering local reads,
/// and a server serves the frame bounds it was actually asked for.
///
/// A range kept mid-group relies on the cached prefix, so it counts as a read of that
/// copy and normal eviction takes it last. Nothing pins it: a prefix still lost
/// before its tail lands fails that group with [`Error::Lagged`], which takes a
/// consumer slower than the cache's turnover.
#[cfg_attr(
not(feature = "async"),
allow(dead_code, reason = "Only session adapters request upstream.")
)]
// private: capability gap
pub(crate) fn widen_compressed(&self, subscription: &mut Subscription) {
if !self.info.compress {
return;
}
let state = self.state.read();
for range in &mut subscription.ranges {
if range.is_empty() {
continue;
}
let Anchor::Absolute(start) = &mut range.start else {
continue;
};
if start.frame == 0 {
continue;
}
let prefix = state
.lookup
.get(&start.group)
.and_then(|slot| slot.copies().find(|group| group.holds_prefix(start.frame)));
match prefix {
Some(group) => group.cache_refresh(),
Comment thread
kixelated marked this conversation as resolved.
Resolved
None => start.frame = 0,
}
}
}

/// Create a [`Dynamic`] handle that serves on-demand fetches of uncached
/// (old) groups. Most producers never need this; a relay creates one to fetch
/// past groups from upstream.
Expand Down Expand Up @@ -14694,6 +14733,56 @@ mod test {
assert!(consumer.peek_group(1).is_none(), "unread group is evicted instead");
}

/// Requesting a compressed tail from frame N reads the cached prefix it relies on, as a
/// FETCH hit does, so pressure evicts a never-read group first and the prefix stays to
/// seed the tail's inflater.
#[test]
fn a_requested_prefix_survives_eviction() {
let (producer, pool) = pooled_producer(u64::MAX);
let producer = Producer::new(
producer.broadcast.clone(),
"compressed",
Info::timed().with_compress(true),
);
let consumer = producer.consume();
// Incompressible, so each group caches about its payload.
let mut seed = 1u32;
let mut append = || {
let payload: Vec<u8> = (0..3_000)
.map(|_| {
seed = seed.wrapping_mul(1_103_515_245).wrapping_add(12_345);
(seed >> 16) as u8
})
.collect();
let mut group = producer.append_group().unwrap();
group.write_frame(Timestamp::ZERO, payload).unwrap();
group.finish().unwrap();
};

append(); // seq 0
elapse(&producer, Duration::from_secs(1));
append(); // seq 1
elapse(&producer, Duration::from_secs(1));
let used = pool.used();
append(); // seq 2
// Room for the three groups and a little more, so each later append evicts one.
pool.resize(pool.used() + (pool.used() - used) / 3);
elapse(&producer, Duration::from_millis(500));

let start = Anchor::Absolute(Position { group: 0, frame: 1 });
let mut request = Subscription::default().with_ranges([Range { start, end: None }]);
producer.widen_compressed(&mut request);
assert_eq!(request.ranges[0].start, start, "a held prefix keeps the request at N");
elapse(&producer, Duration::from_millis(500));

append(); // seq 3
elapse(&producer, Duration::from_secs(1));
append(); // seq 4

assert!(consumer.peek_group(0).is_some(), "the requested prefix survives");
assert!(consumer.peek_group(1).is_none(), "an unread group is evicted instead");
}

/// A consumer holding an evicted group surfaces the eviction, not a hang or a
/// truncated clean end.
#[moq_clock::test]
Expand Down
Loading
Loading