diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md
index 6d5678fca35..a05994b1124 100644
--- a/doc/concept/moq-lite.md
+++ b/doc/concept/moq-lite.md
@@ -308,7 +308,8 @@ starts) is asked of the track's dynamic handler
as a fill, which a relay serves by fetching upstream one group at a time, skipping
a group once every subscriber that asked for it narrows past it; with
no handler it is reported as dropped once the feed stops waiting for it. One
-evicted after the subscriber saw it cached, but before it read it, is skipped. A relay merges its subscribers' ranges into one upstream request, which
+evicted after the subscriber saw it cached, but before it read it, is asked the same
+way, once: evicted again before it is read, it is reported as dropped. A relay merges its subscribers' ranges into one upstream request, which
moq-lite-07 carries range by range and older wire versions as the single span
covering them. A relative or time anchor resolves on the hop that publishes the
track, and over moq-lite-07, which carries a relative or time start upstream and
diff --git a/quest/m1/unbounded-groups/flights.md b/quest/m1/unbounded-groups/flights.md
index b62dccdea2e..54c2f4adf9c 100644
--- a/quest/m1/unbounded-groups/flights.md
+++ b/quest/m1/unbounded-groups/flights.md
@@ -75,7 +75,3 @@ Public API: none. Wire: none, beyond the existing frame-offset fills.
## Required
- [Fastest-reader pacing](/quest/m1/unbounded-groups/fastest-reader.md) - flights pace on their fastest rider the same way the live edge does
-
-## Related
-
-- [Refill evicted groups](/quest/m2/evicted-refill.md) - refills whole groups evicted before a reader took them
diff --git a/quest/m2/README.md b/quest/m2/README.md
index 337557447b7..b29a01bf978 100644
--- a/quest/m2/README.md
+++ b/quest/m2/README.md
@@ -83,7 +83,6 @@ lite-07; bitrate enforcement is not part of this release.
- [Shared group serving and ingest](/quest/m2/shared-serve.md) - Rust's moq-lite and moq-transport sessions serve and ingest groups through one poll machine each, with only wire encoding per protocol
- [Shared group serving and ingest in JS](/quest/m2/shared-serve-js.md) - the same split in js/net, following the Rust seam
- [Paired bench BASE](/quest/m2/bench-base-paired.md) - `just bench ` uses the nightly's paired comparison and the old one is deleted
-- [Refill evicted groups](/quest/m2/evicted-refill.md) - a group evicted after a reader judged it but before it took it is refilled, never delivered twice, and guarded against thrash
- [Receive pool](/quest/m2/receive-pool.md) - one byte budget per worker covers every connection's receive credit and the cache, billing each frame once
- [Front parking](/quest/m2/origin-front-parks.md) - an unroutable request waits on a front instead of re-asking on every route-table move
- [Route wakes](/quest/m2/route-wakes.md) - a route change wakes only the fronts it can move, so pool churn stops scaling with served paths
diff --git a/quest/m2/evicted-refill.md b/quest/m2/evicted-refill.md
deleted file mode 100644
index bdd740325ee..00000000000
--- a/quest/m2/evicted-refill.md
+++ /dev/null
@@ -1,36 +0,0 @@
-# [M] Refill groups evicted before a reader took them
-
-## Goal
-
-A reader behind the live edge still gets a group the cache evicted after the
-reader judged it but before it took it: the group becomes a miss and is asked
-of the track's fills, like one evicted before the reader judged it. A reader
-never receives a group it already took twice, and slow readers under cache
-pressure cannot make the relay refetch without bound.
-
-## Plan
-
-From [moq-dev/moq#6171](https://github.com/moq-dev/moq/pull/6171), where the
-maintainer chose to land fills without this (2026-10-21). Today a reader
-judges each sequence once: a group cached when judged and evicted later is
-skipped silently.
-
-- Track evictions per reader. An ordered reader can judge again every
- eviction at or above its next sequence. An arrival-order reader delivers
- above its floor, so it must tell the groups it already walked past from
- the ones still ahead, for example by keeping each evicted group's arrival
- index. Bound what the track keeps by the slowest reader, not by the cache.
-- Guard against thrash. Under pool pressure a refill evicts another group,
- possibly the next one the same slow reader needs. Decide in the PR, with
- options and a recommendation, how a reader stops refilling once its
- refills only evict what it reads next: give up to the drop cursor, or rely
- on its max delay.
-- Test a slow ascending reader on a pooled track under bufferbloat: it
- receives every group its budget covers, takes none twice, and refills stay
- proportional to what it reads.
-
-Public API: none. Wire: none.
-
-## Related
-
-- [Unbounded groups](/quest/m1/unbounded-groups/README.md) - refills the parts of a group the cache sheds under pressure
diff --git a/rs/moq-net/benches/track.rs b/rs/moq-net/benches/track.rs
index 53776a22d15..ea391399f9c 100644
--- a/rs/moq-net/benches/track.rs
+++ b/rs/moq-net/benches/track.rs
@@ -46,6 +46,11 @@
//! wall clock: each empty group is handed out like any other, so a walk over the rest of
//! the run per group shows up as a slope over it.
//!
+//! `track_evict_fanout` writes groups into a full pool, each evicting one that every reader
+//! judged cached, swept over the readers and the groups written between their reads: each
+//! eviction leaves the group in every reader's inbox, so its cost grows with the readers,
+//! and a cost that grows with the groups written beyond that shows up as a slope.
+//!
//! `track_fanout_frame` writes frames into an open group while every subscriber is parked
//! on the next one, re-polling whichever reader the write woke: a frame wakes no track
//! reader, so waking them shows up as a slope over the subscribers.
@@ -553,6 +558,46 @@ fn bench_fanout(c: &mut Criterion) {
group.finish();
}
+/// Groups written between reads in `track_evict_fanout`, each evicting one.
+const EVICTED: [usize; 3] = [1, 8, 64];
+
+fn bench_evict_fanout(c: &mut Criterion) {
+ let mut group = c.benchmark_group("track_evict_fanout");
+ for subscribers in FANOUT {
+ for evicted in EVICTED {
+ group.throughput(Throughput::Elements(evicted as u64));
+ group.bench_function(BenchmarkId::new(format!("{subscribers}r"), evicted), |b| {
+ let mut fanout = Fanout::new(subscribers);
+ // Twice what the pool holds, so every write evicts.
+ for _ in 0..2 * CACHE_CAPACITY / PAYLOAD as u64 {
+ fanout.cycle();
+ }
+ // Only the writes and the evictions they cause are timed, not the reads that
+ // judge each written group cached again.
+ b.iter_custom(|iterations| {
+ let mut elapsed = Duration::ZERO;
+ for _ in 0..iterations {
+ let start = Instant::now();
+ for _ in 0..evicted {
+ let mut group = fanout.track.append_group().unwrap();
+ group.write_frame(Timestamp::ZERO, fanout.payload.clone()).unwrap();
+ group.finish().unwrap();
+ }
+ elapsed += start.elapsed();
+ for (subscriber, waiter) in fanout.subscribers.iter_mut().zip(&fanout.waiters) {
+ while let Poll::Ready(Ok(Some(group))) = subscriber.poll_recv_group(waiter) {
+ black_box(group);
+ }
+ }
+ }
+ elapsed
+ });
+ });
+ }
+ }
+ group.finish();
+}
+
/// Frames written per group in `track_fanout_frame`, after which the group rolls and every
/// subscriber takes the next one: enough that the roll's cost per frame stays small.
const FRAMES_PER_GROUP: usize = 256;
@@ -1946,6 +1991,7 @@ criterion_group!(
bench_budget_widening,
bench_fanout,
bench_fanout_frame,
+ bench_evict_fanout,
bench_fetch_expiry,
bench_fetch_budgets,
bench_fetch_park,
diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs
index fb451d50464..97264338d27 100644
--- a/rs/moq-net/src/model/track.rs
+++ b/rs/moq-net/src/model/track.rs
@@ -464,6 +464,15 @@ pub(crate) struct TrackState {
// Bumped on every insert into `drops`, so a cursor rescans it only after a change.
drops_revision: u64,
drops_changes: ledger::Changes,
+ // Every reader's eviction inbox. Its own lock, so a reader registers through the read
+ // guard it judges under.
+ inboxes: Mutex,
+ // Bumped whenever an eviction lands in an inbox, so a reader with nothing new skips
+ // its own lock.
+ evictions: u64,
+ // Bumped as each group is cached, stamped on its slot: a reader judged a slot cached
+ // only if it was cached by the count the reader last judged at.
+ cached: u64,
// Whether the cache reflects the live feed: readers get nothing from it while it does
// not, since how stale it is cannot be told; fetches still do. A track is live from
@@ -547,6 +556,9 @@ struct Slot {
// the replacement and deliver it twice or at the wrong position.
stamp: u32,
+ // [`TrackState::cached`] once this incarnation was cached.
+ cached: u64,
+
// Whether this incarnation came from the live publisher and can replace older
// subscription content. Fetch-only backfill stays cached but never anchors drift.
visible: bool,
@@ -938,6 +950,21 @@ impl TrackState {
Some(slot)
}
+ /// [`Self::uncache`] a servable group to free the cache and abort it with `err`, recording
+ /// it for the readers that judged it cached to judge again.
+ fn evict_group(&mut self, sequence: u64, err: Error) {
+ let slot = self.uncache(sequence).expect("a cached group");
+ slot.abort(err);
+ let landed = self
+ .inboxes
+ .get_mut()
+ .unwrap_or_else(PoisonError::into_inner)
+ .evicted(sequence, slot.cached);
+ if landed {
+ self.evictions += 1;
+ }
+ }
+
/// [`Self::uncache`] a group that another copy displaces rather than one that ends, so a
/// teardown still ends the copies its readers hold.
fn displace(&mut self, sequence: u64) {
@@ -1760,7 +1787,7 @@ impl TrackState {
// Take the group out of the cache and abort it, so any consumer
// still reading surfaces `Error::Old` instead of blocking forever
// on a frame that will never arrive.
- self.uncache(sequence).unwrap().abort(Error::Old);
+ self.evict_group(sequence, Error::Old);
}
}
@@ -2036,11 +2063,13 @@ impl TrackState {
if visible {
self.record_arrival(sequence, now);
}
+ self.cached += 1;
let replaced = self.lookup.insert(
sequence,
Slot {
group: group.clone(),
stamp,
+ cached: self.cached,
visible,
arrived: now,
head: None,
@@ -2230,7 +2259,7 @@ impl TrackState {
self.debt -= size;
paid = paid.saturating_add(size);
- self.uncache(sequence).unwrap().abort(Error::Evicted);
+ self.evict_group(sequence, Error::Evicted);
}
// No cold group left to pay with: shed the front of the groups that could not go
// whole, the ones being read and then the live edge, so neither grows without bound.
@@ -6235,7 +6264,7 @@ impl Cursor {
}
#[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))]
- fn poll_drop(&mut self, waiter: &kio::Waiter) -> Poll<(ops::Range, StreamError)> {
+ fn poll_drop(&mut self, taken: &Delivered, waiter: &kio::Waiter) -> Poll<(ops::Range, StreamError)> {
self.sync();
if self.drops.is_none() {
let mut drops = Box::new(DropCursor::new(&self.state.read()));
@@ -6249,7 +6278,9 @@ impl Cursor {
if self.filter.refused.is_none()
&& let Poll::Ready(Ok(drift)) = self.poll_drift(&self.servable, waiter)
{
- self.probe(self.floor(), &drift, waiter);
+ // From where the reader reads next: an ordered one asks nothing below what it took.
+ let from = self.next_sequence.unwrap_or(u64::MAX).max(self.floor());
+ self.probe(from, &drift, taken, waiter);
}
// A cap at or below the floor leaves nothing to report, not a reversed span.
let floor = self.floor();
@@ -6658,7 +6689,8 @@ impl Cursor {
/// Ask fills for what this cursor misses from `from`, and take in what they answered,
/// parking `waiter` on the fills still out. Whatever it gives up goes to the drop cursor.
- fn probe(&mut self, from: u64, drift: &Drift, waiter: &kio::Waiter) {
+ /// `taken` is what its subscriber handed out, which an eviction never makes a miss.
+ fn probe(&mut self, from: u64, drift: &Drift, taken: &Delivered, waiter: &kio::Waiter) {
let (window, cap, revision) = (&self.filter.window, self.end_sequence, self.filter.revision);
let asking = {
let subscription = self.subscription.read();
@@ -6691,6 +6723,7 @@ impl Cursor {
.map(|run| *run.start()..=(*run.end()).min(end))
.collect()
},
+ taken,
drift,
&mut given_up,
waiter,
@@ -6732,7 +6765,12 @@ impl Cursor {
self.misses.as_ref().is_some_and(|misses| misses.is_waiting())
}
- fn poll_recv_group(&mut self, take: bool, waiter: &kio::Waiter) -> Poll>> {
+ fn poll_recv_group(
+ &mut self,
+ take: bool,
+ taken: &Delivered,
+ waiter: &kio::Waiter,
+ ) -> Poll>> {
if let Some(peeked) = self.peeked.take().filter(|peeked| take && !peeked.ordered) {
// The reserved group is still the one handed out, but bounded by the ranges in
// force now, which a `Control` may have replaced since the peek.
@@ -6751,7 +6789,7 @@ impl Cursor {
// One scan for the whole poll, so walking a backlog off stays linear in its size.
let drift = ready!(self.poll_drift(&self.servable, waiter))?;
self.reoffer(drift.budget);
- self.probe(self.floor(), &drift, waiter);
+ self.probe(self.floor(), &drift, taken, waiter);
self.poll_recv_with(take, drift, waiter)
}
@@ -6935,7 +6973,12 @@ impl Cursor {
/// The lowest servable group past the last one returned, walking off everything the
/// drift budget has convicted on the way.
- fn poll_next_group(&mut self, take: bool, waiter: &kio::Waiter) -> Poll>> {
+ fn poll_next_group(
+ &mut self,
+ take: bool,
+ taken: &Delivered,
+ waiter: &kio::Waiter,
+ ) -> Poll>> {
if let Some(peeked) = self.peeked.take().filter(|peeked| take && peeked.ordered) {
// Bounded by the ranges in force now, as on the arrival path.
self.sync();
@@ -6951,7 +6994,7 @@ impl Cursor {
let floor = next.max(self.floor());
// One scan for the whole poll, so walking a backlog off stays linear in its size.
let drift = ready!(self.poll_drift(&self.servable, waiter))?;
- self.probe(floor, &drift, waiter);
+ self.probe(floor, &drift, taken, waiter);
self.poll_next_with(take, floor, drift, waiter)
}
@@ -7303,8 +7346,8 @@ impl Subscriber {
let res = match &mut self.inner {
Inner::Plain(cursor) => loop {
let res = match ordered {
- true => cursor.poll_next_group(true, waiter),
- false => cursor.poll_recv_group(true, waiter),
+ true => cursor.poll_next_group(true, &self.delivered, waiter),
+ false => cursor.poll_recv_group(true, &self.delivered, waiter),
};
match res {
Poll::Ready(Ok(Some(group))) if self.delivered.contains(group.sequence) => continue,
@@ -7366,8 +7409,8 @@ impl Subscriber {
match &mut self.inner {
Inner::Plain(cursor) => loop {
let group = ready!(match ordered {
- true => cursor.poll_next_group(false, waiter),
- false => cursor.poll_recv_group(false, waiter),
+ true => cursor.poll_next_group(false, &self.delivered, waiter),
+ false => cursor.poll_recv_group(false, &self.delivered, waiter),
})?;
let Some(group) = group else {
return Poll::Ready(Ok(None));
@@ -7377,8 +7420,8 @@ impl Subscriber {
}
// A replay advances only this copy's cursor, as an ordinary read would.
ready!(match ordered {
- true => cursor.poll_next_group(true, waiter),
- false => cursor.poll_recv_group(true, waiter),
+ true => cursor.poll_next_group(true, &self.delivered, waiter),
+ false => cursor.poll_recv_group(true, &self.delivered, waiter),
})?;
},
Inner::Resume(resume, _) => resume.poll_peek_group(ordered, &self.delivered, waiter),
@@ -7725,7 +7768,7 @@ impl Subscriber {
#[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))]
pub(crate) fn poll_drop(&mut self, waiter: &kio::Waiter) -> Poll<(ops::Range, StreamError)> {
match &mut self.inner {
- Inner::Plain(cursor) => cursor.poll_drop(waiter),
+ Inner::Plain(cursor) => cursor.poll_drop(&self.delivered, waiter),
Inner::Resume(resume, _) => resume.poll_drop(&self.delivered, waiter),
}
}
@@ -19222,6 +19265,294 @@ mod test {
assert_eq!(asked, [(4, 6), (0, 2)]);
}
+ /// A group evicted after an arrival-order reader judged it cached, before it took it, is
+ /// asked of a fill like one evicted before: the reader still receives it, and a group it
+ /// took is never asked again.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_group_evicted_before_its_reader_took_it_is_refilled() {
+ let producer = track_producer("test", Info::timed());
+ let dynamic = producer.requests();
+ let mut subscriber = producer.subscribe(replay());
+ for sequence in 0..3 {
+ write_group(&producer, sequence);
+ }
+ let waiter = kio::Waiter::noop();
+ let Poll::Ready(Ok(Some(first))) = subscriber.poll_recv_group(&waiter) else {
+ panic!("group 0 is cached");
+ };
+ assert_eq!(first.sequence, 0);
+ evict(&producer, 0);
+ evict(&producer, 1);
+ assert_eq!(drain_groups(&mut subscriber), [2]);
+ let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else {
+ panic!("group 1 is a miss now");
+ };
+ assert_eq!((fill.start().group, fill.end()), (1, 2), "group 0 was taken");
+ assert_eq!(fill.wanted(), Some(1));
+ write_filled(fill.accept(1, None), 1);
+ drop(fill);
+ assert_eq!(drain_groups(&mut subscriber), [1]);
+ assert!(dynamic.poll_requested_fill(&waiter).is_pending());
+ assert!(drained_drops(&mut subscriber).is_empty(), "nothing was given up");
+ let Inner::Plain(cursor) = &subscriber.inner else {
+ unreachable!("a plain subscription");
+ };
+ let misses = cursor.misses.as_ref().expect("the reader judged its misses");
+ assert!(misses.refilled().is_empty(), "a refill handed out is forgotten");
+ }
+
+ /// An eviction waits only for a reader that judged the evicted group cached: a reader
+ /// that judged once and stalled holds no more than what was cached then, however many
+ /// groups the feed caches and evicts after, late lower ones included.
+ #[test]
+ #[cfg(feature = "async")]
+ fn an_eviction_waits_only_for_readers_that_judged_it_cached() {
+ let producer = track_producer("test", Info::timed());
+ let _dynamic = producer.requests();
+ write_group(&producer, 0);
+ write_group(&producer, 1);
+ write_group(&producer, 1 << 40);
+ let held = || producer.state.read().inboxes.lock().unwrap().held();
+ evict(&producer, 0);
+ assert!(held().is_empty(), "no reader judged group 0");
+
+ let mut stalled = producer.subscribe(replay());
+ let mut active = producer.subscribe(replay());
+ drain_groups(&mut stalled);
+ drain_groups(&mut active);
+ // Late groups below the stalled reader's newest, cached after it judged.
+ for sequence in (2..200).step_by(2) {
+ write_group(&producer, sequence);
+ drain_groups(&mut active);
+ evict(&producer, sequence);
+ }
+ // The active reader judged each one cached, so it holds the last until it judges again.
+ assert_eq!(held(), [vec![], vec![(198, 199)]], "the stalled reader judged none");
+ drain_groups(&mut active);
+ evict(&producer, 1);
+ assert_eq!(held(), [vec![(1, 2)], vec![(1, 2)]], "both judged group 1 cached");
+ drain_groups(&mut active);
+ assert_eq!(held(), [vec![(1, 2)], vec![]], "the active reader took it in");
+ drop(stalled);
+ assert_eq!(held(), [vec![]], "a dropped reader holds nothing");
+ write_group(&producer, 3);
+ evict(&producer, 3);
+ assert_eq!(
+ producer.state.read().inboxes.lock().unwrap().len(),
+ 1,
+ "an eviction prunes a dropped reader's inbox"
+ );
+ }
+
+ /// An ordered reader forwarding drops asks nothing below what it took, though its drop
+ /// cursor reports from the window's floor.
+ #[test]
+ #[cfg(feature = "async")]
+ fn an_ordered_readers_drop_poll_asks_nothing_it_took() {
+ let producer = track_producer("test", Info::timed());
+ let dynamic = producer.requests();
+ let mut subscriber = producer.subscribe(replay().with_order(Order::Ascending));
+ for sequence in 0..3 {
+ write_group(&producer, sequence);
+ }
+ let waiter = kio::Waiter::noop();
+ for expected in 0..2 {
+ let Poll::Ready(Ok(Some(group))) = subscriber.poll_next_group(&waiter) else {
+ panic!("group {expected} is cached");
+ };
+ assert_eq!(group.sequence, expected);
+ }
+ evict(&producer, 0);
+ evict(&producer, 1);
+ assert!(drained_drops(&mut subscriber).is_empty());
+ assert!(dynamic.poll_requested_fill(&waiter).is_pending(), "both were taken");
+ }
+
+ /// An ordered reader waits on the refill of a group evicted at or above its next
+ /// sequence, and reads it in order.
+ #[test]
+ #[cfg(feature = "async")]
+ fn an_ordered_reader_refills_an_evicted_group_in_order() {
+ let producer = track_producer("test", Info::timed());
+ let dynamic = producer.requests();
+ let mut subscriber = producer.subscribe(replay().with_order(Order::Ascending));
+ for sequence in 0..4 {
+ write_group(&producer, sequence);
+ }
+ let waiter = kio::Waiter::noop();
+ let next = |subscriber: &mut Subscriber| match subscriber.poll_next_group(&kio::Waiter::noop()) {
+ Poll::Ready(Ok(Some(group))) => Some(group.sequence),
+ _ => None,
+ };
+ assert_eq!(next(&mut subscriber), Some(0));
+ evict(&producer, 1);
+ evict(&producer, 2);
+ assert_eq!(next(&mut subscriber), None, "group 1 comes before 3");
+ let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else {
+ panic!("groups 1 and 2 are misses now");
+ };
+ assert_eq!((fill.start().group, fill.end()), (1, 3));
+ while let Some(sequence) = fill.wanted() {
+ write_filled(fill.accept(sequence, None), sequence);
+ }
+ drop(fill);
+ assert_eq!([next(&mut subscriber), next(&mut subscriber)], [Some(1), Some(2)]);
+ assert_eq!(next(&mut subscriber), Some(3));
+ }
+
+ /// An ordered reader gives up a refill evicted again while it waits on a lower one,
+ /// naming it dropped and releasing its fill, rather than reading past it unnamed.
+ #[test]
+ #[cfg(feature = "async")]
+ fn an_ordered_reader_gives_up_a_refill_evicted_while_it_waits_below() {
+ let producer = track_producer("test", Info::timed());
+ let dynamic = producer.requests();
+ let mut subscriber = producer.subscribe(replay().with_order(Order::Ascending));
+ for sequence in 0..4 {
+ write_group(&producer, sequence);
+ }
+ let waiter = kio::Waiter::noop();
+ let next = |subscriber: &mut Subscriber| match subscriber.poll_next_group(&kio::Waiter::noop()) {
+ Poll::Ready(Ok(Some(group))) => Some(group.sequence),
+ _ => None,
+ };
+ assert_eq!(next(&mut subscriber), Some(0));
+ evict(&producer, 1);
+ evict(&producer, 2);
+ assert_eq!(next(&mut subscriber), None, "group 1 comes before 3");
+ let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else {
+ panic!("groups 1 and 2 are misses now");
+ };
+ write_filled(fill.accept(2, None), 2);
+ assert_eq!(next(&mut subscriber), None, "still waiting on group 1");
+ evict(&producer, 2);
+ assert_eq!(next(&mut subscriber), None, "still waiting on group 1");
+ assert_eq!(drained_drops(&mut subscriber), [(2, 3)]);
+ write_filled(fill.accept(1, None), 1);
+ drop(fill);
+ assert_eq!([next(&mut subscriber), next(&mut subscriber)], [Some(1), Some(3)]);
+ assert!(dynamic.poll_requested_fill(&waiter).is_pending(), "refilled once");
+ }
+
+ /// A refill evicted again before its reader took it is given up rather than asked a
+ /// second time, so refills that evict each other cannot loop.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_refill_evicted_again_is_given_up() {
+ let producer = track_producer("test", Info::timed());
+ let dynamic = producer.requests();
+ let mut subscriber = producer.subscribe(replay());
+ for sequence in 0..3 {
+ write_group(&producer, sequence);
+ }
+ let waiter = kio::Waiter::noop();
+ assert!(matches!(subscriber.poll_recv_group(&waiter), Poll::Ready(Ok(Some(_)))));
+ evict(&producer, 1);
+ assert_eq!(drain_groups(&mut subscriber), [2]);
+ let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else {
+ panic!("group 1 is a miss now");
+ };
+ assert_eq!(fill.wanted(), Some(1));
+ write_filled(fill.accept(1, None), 1);
+ drop(fill);
+ evict(&producer, 1);
+ assert!(drain_groups(&mut subscriber).is_empty());
+ assert!(dynamic.poll_requested_fill(&waiter).is_pending(), "refilled once");
+ assert_eq!(drained_drops(&mut subscriber), [(1, 2)]);
+ }
+
+ /// A slow ascending reader on a pooled track holding `cached` groups, taking `pace` of
+ /// every four its feed writes, as behind a bufferbloated link. Returns what it read, how
+ /// many groups the fills wrote, and what it gave up.
+ #[cfg(feature = "async")]
+ fn read_slowly(cached: usize, pace: u64) -> (Vec, u64, Vec<(u64, u64)>) {
+ let size = 10_000;
+ let (producer, _pool) = pooled_producer((cached * size) as u64);
+ let dynamic = producer.requests();
+ let mut subscriber = producer.subscribe(replay().with_order(Order::Ascending));
+ let write = |group: Result, sequence: u64| {
+ let mut group = group.unwrap();
+ group
+ .write_frame(Timestamp::from_millis(sequence).unwrap(), vec![0u8; size])
+ .unwrap();
+ group.finish().unwrap();
+ };
+ let waiter = kio::Waiter::noop();
+ let mut refilled = 0;
+ let mut serve = || {
+ while let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) {
+ while let Some(sequence) = fill.wanted() {
+ write(fill.accept(sequence, None), sequence);
+ refilled += 1;
+ }
+ }
+ };
+ let (mut read, mut drops) = (Vec::new(), Vec::new());
+ // A wire publisher forwards what the reader gives up as it reads.
+ let mut take = |subscriber: &mut Subscriber| {
+ let taken = match subscriber.poll_next_group(&kio::Waiter::noop()) {
+ Poll::Ready(Ok(Some(group))) => {
+ read.push(group.sequence);
+ true
+ }
+ _ => false,
+ };
+ drops.extend(drained_drops(subscriber));
+ taken
+ };
+ for sequence in 0..GROUPS {
+ write(producer.create_group(group::Info { sequence }), sequence);
+ if sequence % 4 < pace {
+ take(&mut subscriber);
+ }
+ serve();
+ }
+ // The feed pauses and the reader drains what it can, refills included.
+ loop {
+ serve();
+ if !take(&mut subscriber) {
+ serve();
+ if !take(&mut subscriber) {
+ break;
+ }
+ }
+ }
+ (read, refilled, drops)
+ }
+
+ #[cfg(feature = "async")]
+ const GROUPS: u64 = 40;
+
+ /// A backlog a little past the cache: what the reader judged and lost to eviction is
+ /// refilled, so it receives every group once, in order, refilling no more than it reads.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_slow_reader_on_a_pooled_track_receives_every_group() {
+ let (read, refilled, drops) = read_slowly(8, 3);
+ assert_eq!(read, (0..GROUPS).collect::>(), "every group once, in order");
+ assert!(drops.is_empty(), "nothing given up: {drops:?}");
+ assert!(refilled > 0, "the backlog outgrew the cache");
+ assert!(refilled <= GROUPS, "{refilled} refills for {GROUPS} groups read");
+ }
+
+ /// A backlog far past the cache, where refills evict the groups the reader reads next:
+ /// each group is refilled at most once, and one lost again is given up and named, never
+ /// handed out twice.
+ #[test]
+ #[cfg(feature = "async")]
+ fn a_slow_reader_past_the_cache_refills_each_group_once() {
+ let (read, refilled, drops) = read_slowly(4, 2);
+ let mut seen = read.clone();
+ seen.dedup();
+ assert_eq!(seen, read, "none twice");
+ assert!(read.is_sorted(), "in order");
+ assert!(refilled <= GROUPS, "{refilled} refills for {GROUPS} groups");
+ let lost: Vec = (0..GROUPS).filter(|sequence| !read.contains(sequence)).collect();
+ let named: Vec = drops.iter().flat_map(|&(start, end)| start..end).collect();
+ assert_eq!(lost, named, "every group lost is named dropped");
+ }
+
/// A reader forgets the misses it gave up once its window moves past them, as it forgets
/// what it judged, so a feed skipping every other group costs it only the gaps it can
/// still read.
@@ -20013,12 +20344,7 @@ mod test {
/// Take group `sequence` out of the cache as eviction does.
fn evict(producer: &Producer, sequence: u64) {
- producer
- .modify()
- .unwrap()
- .uncache(sequence)
- .unwrap()
- .abort(Error::Evicted);
+ producer.modify().unwrap().evict_group(sequence, Error::Evicted);
}
/// A rewound first frame is refused and leaves the group as it was, so the publisher can
diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs
index 6bc2dc064ec..45f5b710329 100644
--- a/rs/moq-net/src/model/track/miss.rs
+++ b/rs/moq-net/src/model/track/miss.rs
@@ -7,8 +7,9 @@
//! (evicted, or a gap the feed skipped). A hole the ledger still waits for is deferred and
//! judged again as the feed moves.
//!
-//! Each sequence is judged once, so a group evicted after the reader judged it cached is
-//! not a miss: judging it again would refill a group an arrival-order reader already took.
+//! Each sequence is judged once, except a group evicted after the reader judged it cached
+//! and before it took it: judged again as a miss, once. A refill evicted again before it is
+//! taken is given up, so a slow reader's refills cannot evict each other without bound.
//!
//! A start is no promise, since a group below it may still arrive late. A hole under it is
//! asked at once, but a refusal is final only once the ledger gives up on it too; until
@@ -18,12 +19,14 @@
use std::{
collections::{BTreeMap, BTreeSet},
ops::{self, RangeInclusive},
+ sync::{Arc, Mutex, PoisonError},
task::Poll,
time::Duration,
};
use rangemap::{RangeInclusiveSet, RangeSet};
+use super::super::delivered::Delivered;
pub(super) use super::fill::Asking;
use super::fill::{Ask, FillState, Join};
use super::{Drift, Order, Position, TrackState};
@@ -60,6 +63,12 @@ pub(super) struct Misses {
filled: BTreeSet,
/// Never asked again: named dropped to the reader.
given_up: RangeSet,
+ /// Where the track leaves what it evicts that this reader judged cached, once it judged.
+ inbox: Option,
+ /// The track's eviction count this reader last emptied its inbox at.
+ evictions: u64,
+ /// Judged again after an eviction, within the judged span: never refilled twice.
+ refilled: RangeSet,
}
/// A run of misses the reader gives up, for the drop cursor to name.
@@ -190,8 +199,14 @@ impl Misses {
&self.given_up
}
- /// The next answered group still cached in `order`, forgetting one evicted before it was
- /// read.
+ /// What was refilled after an eviction and not handed out yet.
+ #[cfg(all(test, feature = "async"))]
+ pub fn refilled(&self) -> &RangeSet {
+ &self.refilled
+ }
+
+ /// The next answered group still cached in `order`, giving up one evicted before it was
+ /// read: [`Self::judge`] asked a first eviction again, so this one was refilled already.
pub fn next_filled(
&mut self,
state: &TrackState,
@@ -213,8 +228,13 @@ impl Misses {
}
}
- /// Take `sequence` as handed out, if a fill answered it.
+ /// Take `sequence` as handed out, if a fill answered it. A refill handed out is never
+ /// judged again, so it is forgotten too.
pub fn take_filled(&mut self, sequence: u64) {
+ // The last sequence is never judged, so never refilled.
+ if let Some(end) = sequence.checked_add(1) {
+ self.refilled.remove(sequence..end);
+ }
if self.filled.remove(&sequence) {
self.release(sequence);
}
@@ -336,6 +356,10 @@ impl Misses {
taken.remove(sequence..sequence + 1);
}
}
+ // Seen cached, though judging may not run again before it is evicted.
+ if let Some(inbox) = &self.inbox {
+ inbox.saw(state.cached);
+ }
}
Err((Error::Old, asked)) if looser(max_delay, asked) => retry.insert(groups),
// A drop the track recorded is named by the drop cursor already.
@@ -351,7 +375,8 @@ impl Misses {
/// Judge the holes in `target` the last probe has not, returning the runs to ask a
/// fill for. Gives up a run already stale under this reader's budget. `named` is the
- /// lowest group the reader's ranges name, which may reach below the feed's start.
+ /// lowest group the reader's ranges name, which may reach below the feed's start, and
+ /// `taken` what the reader handed out, which an eviction never makes a miss again.
#[expect(
clippy::too_many_arguments,
reason = "one probe's inputs, borrowed apart from its cursor"
@@ -362,6 +387,7 @@ impl Misses {
key: (u64, u64, Option),
named: Option,
mut target: impl FnMut(RangeInclusive) -> RangeInclusiveSet,
+ taken: &Delivered,
drift: &Drift,
given_up: &mut GivenUp,
waiter: &kio::Waiter,
@@ -393,7 +419,8 @@ impl Misses {
&& self.budget == Some(drift.budget)
&& self.ended == ended
&& self.ledger_revision == ledger.revision()
- && self.drops_revision == drops.revision();
+ && self.drops_revision == drops.revision()
+ && self.evictions == state.evictions;
// The grace only times deferred gaps, so with none a new one has nothing to move.
// It is judged once something else changes.
if unchanged && self.deferred.runs.is_empty() {
@@ -436,6 +463,7 @@ impl Misses {
self.judged = clip(&self.judged);
self.refused = clip(&self.refused);
self.given_up = clip(&self.given_up);
+ self.refilled = clip(&self.refilled);
for (&start, &(end, _)) in &self.deferred.runs {
for run in span.overlapping(&(start..end)) {
candidates.insert(start.max(run.start)..end.min(run.end));
@@ -451,7 +479,12 @@ impl Misses {
// A moving read floor removes only its retired prefix. New production extends the
// judged span; neither path visits older deferred gaps.
if low > 0 {
- for set in [&mut self.judged, &mut self.refused, &mut self.given_up] {
+ for set in [
+ &mut self.judged,
+ &mut self.refused,
+ &mut self.given_up,
+ &mut self.refilled,
+ ] {
set.remove(0..low);
}
self.deferred.remove(0..low);
@@ -497,6 +530,65 @@ impl Misses {
self.refused.remove(sequence..sequence + 1);
}
}
+ // What it judged cached and the track evicted since, before this reader took it, is a
+ // miss now.
+ let inbox = self.inbox.get_or_insert_with(|| {
+ let inbox = Inbox::default();
+ state
+ .inboxes
+ .lock()
+ .unwrap_or_else(PoisonError::into_inner)
+ .register(inbox.clone());
+ inbox
+ });
+ let evicted = inbox.judge(state.cached, low, top);
+ self.evictions = state.evictions;
+ let mut refills = RangeSet::new();
+ let mut again = RangeSet::new();
+ for run in evicted.iter() {
+ let run = run.start.max(low)..run.end.min(top.saturating_add(1));
+ if run.is_empty() {
+ continue;
+ }
+ for part in target(run.start..=run.end - 1).iter() {
+ for gap in taken.gaps(*part.start()..*part.end() + 1) {
+ for judged in self.judged.overlapping(&gap) {
+ let judged = judged.start.max(gap.start)..judged.end.min(gap.end);
+ for fresh in self.refilled.gaps(&judged) {
+ refills.insert(fresh);
+ }
+ for refilled in self.refilled.overlapping(&judged) {
+ again.insert(refilled.start.max(judged.start)..refilled.end.min(judged.end));
+ }
+ }
+ }
+ }
+ }
+ // Settled here rather than when read, since an ordered reader never looks at what a
+ // fill answered: it reads past a group gone from the cache. Still asked, its fill
+ // settles it; given up already, it was named.
+ for run in self.wanted.iter().chain(self.given_up.iter()) {
+ again.remove(run.clone());
+ }
+ for (run, refill) in refills
+ .iter()
+ .map(|run| (run, true))
+ .chain(again.iter().map(|run| (run, false)))
+ {
+ // A fill's answer evicted before it was taken is no longer held for this reader.
+ let filled: Vec = self.filled.range(run.clone()).copied().collect();
+ for sequence in filled {
+ self.filled.remove(&sequence);
+ self.release(sequence);
+ }
+ match refill {
+ true => {
+ self.refilled.insert(run.clone());
+ candidates.insert(run.clone());
+ }
+ false => self.give_up(run.clone(), given_up),
+ }
+ }
self.ledger_revision = ledger.revision();
self.drops_revision = drops.revision();
self.budget = Some(drift.budget);
@@ -725,6 +817,96 @@ impl Misses {
}
}
+/// What the track evicted that one reader judged cached, and what it judged: a slot cached
+/// by the count it judged at, within the span it judged. Holds no more than the slots it
+/// judged cached, so a stalled reader pins nothing cached after.
+#[derive(Clone, Default)]
+pub(super) struct Inbox(Arc>);
+
+#[derive(Default)]
+struct Judging {
+ cached: u64,
+ span: Option>,
+ evicted: RangeSet,
+}
+
+impl Inbox {
+ /// Take what was evicted since the last call, and judge `low..=top` from here on, with
+ /// every slot cached by `cached`.
+ fn judge(&self, cached: u64, low: u64, top: u64) -> RangeSet {
+ let mut judging = self.0.lock().unwrap_or_else(PoisonError::into_inner);
+ judging.cached = cached;
+ judging.span = Some(low..=top);
+ std::mem::take(&mut judging.evicted)
+ }
+
+ /// Count every slot cached by `cached` as judged cached, within the span judged last.
+ fn saw(&self, cached: u64) {
+ let mut judging = self.0.lock().unwrap_or_else(PoisonError::into_inner);
+ judging.cached = judging.cached.max(cached);
+ }
+}
+
+/// Every reader's [`Inbox`], pruned of dropped readers on each eviction, or once
+/// registrations double it past what the last prune kept.
+#[derive(Default)]
+pub(super) struct Inboxes {
+ inboxes: Vec,
+ kept: usize,
+}
+
+impl Inboxes {
+ fn register(&mut self, inbox: Inbox) {
+ self.inboxes.push(inbox);
+ if self.inboxes.len() > 2 * self.kept.max(1) {
+ self.inboxes.retain(|inbox| Arc::strong_count(&inbox.0) > 1);
+ self.kept = self.inboxes.len();
+ }
+ }
+
+ /// How many inboxes are registered, live or not.
+ #[cfg(all(test, feature = "async"))]
+ pub(super) fn len(&self) -> usize {
+ self.inboxes.len()
+ }
+
+ /// What each live reader's inbox holds, in registration order.
+ #[cfg(all(test, feature = "async"))]
+ pub(super) fn held(&self) -> Vec> {
+ self.inboxes
+ .iter()
+ .filter(|inbox| Arc::strong_count(&inbox.0) > 1)
+ .map(|inbox| {
+ let judging = inbox.0.lock().unwrap_or_else(PoisonError::into_inner);
+ judging.evicted.iter().map(|run| (run.start, run.end)).collect()
+ })
+ .collect()
+ }
+
+ /// Leave `sequence`, cached as count `cached`, with each reader that judged it cached.
+ /// Returns whether any did.
+ pub(super) fn evicted(&mut self, sequence: u64, cached: u64) -> bool {
+ let Some(end) = sequence.checked_add(1) else {
+ return false;
+ };
+ let mut landed = false;
+ // Every inbox is visited anyway, so a dropped reader's goes here too.
+ self.inboxes.retain(|inbox| {
+ if Arc::strong_count(&inbox.0) == 1 {
+ return false;
+ }
+ let mut judging = inbox.0.lock().unwrap_or_else(PoisonError::into_inner);
+ if cached <= judging.cached && judging.span.as_ref().is_some_and(|span| span.contains(&sequence)) {
+ judging.evicted.insert(sequence..end);
+ landed = true;
+ }
+ true
+ });
+ self.kept = self.inboxes.len();
+ landed
+ }
+}
+
impl TrackState {
/// Split `hole`, which the cache holds nothing for, into what the live feed will never
/// deliver and the gaps it still waits for, with when each stops being waited for.
@@ -857,6 +1039,7 @@ mod tests {
(0, 0, None),
None,
|span| [span].into_iter().collect(),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter,
@@ -896,6 +1079,7 @@ mod tests {
(0, 0, None),
None,
|_| panic!("the edge has not moved"),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter,
@@ -920,6 +1104,7 @@ mod tests {
(0, 0, None),
None,
|_| panic!("recreation extends no span"),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter,
@@ -961,6 +1146,7 @@ mod tests {
(0, 0, None),
None,
|span| [span].into_iter().collect(),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter,
@@ -994,6 +1180,7 @@ mod tests {
(0, 0, None),
None,
|_| panic!("the edge has not moved"),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter,
@@ -1030,6 +1217,7 @@ mod tests {
(0, 0, None),
None,
|span| [span].into_iter().collect(),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter
@@ -1045,6 +1233,7 @@ mod tests {
(0, 0, None),
None,
|_| panic!("an unchanged probe rebuilds no span"),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter
@@ -1065,6 +1254,7 @@ mod tests {
(0, 0, None),
None,
|_| panic!("a late arrival extends no span"),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter
@@ -1081,6 +1271,7 @@ mod tests {
(0, 0, None),
None,
|_| panic!("a due gap extends no span"),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter,
@@ -1117,7 +1308,16 @@ mod tests {
let full = |span| [span].into_iter().collect();
assert!(
misses
- .judge(&state, (0, 0, None), None, full, &drift, &mut given_up, &waiter)
+ .judge(
+ &state,
+ (0, 0, None),
+ None,
+ full,
+ &Delivered::default(),
+ &drift,
+ &mut given_up,
+ &waiter
+ )
.is_empty()
);
assert!(misses.deferred.runs.is_empty(), "a contiguous feed defers nothing");
@@ -1134,6 +1334,7 @@ mod tests {
(0, 0, None),
None,
|_| panic!("nothing deferred: the new grace waits for a change"),
+ &Delivered::default(),
&drift,
&mut given_up,
&waiter
@@ -1151,7 +1352,16 @@ mod tests {
let opened = state.cache.pool().now_wall();
assert!(
misses
- .judge(&state, (0, 0, None), None, full, &drift, &mut given_up, &waiter)
+ .judge(
+ &state,
+ (0, 0, None),
+ None,
+ full,
+ &Delivered::default(),
+ &drift,
+ &mut given_up,
+ &waiter
+ )
.is_empty()
);
assert_eq!(misses.deferred.overlapping(0..5).collect::>(), vec![3..4]);