diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md index 6d5678fca35..a05994b1124 100644 --- a/doc/concept/moq-lite.md +++ b/doc/concept/moq-lite.md @@ -308,7 +308,8 @@ starts) is asked of the track's dynamic handler as a fill, which a relay serves by fetching upstream one group at a time, skipping a group once every subscriber that asked for it narrows past it; with no handler it is reported as dropped once the feed stops waiting for it. One -evicted after the subscriber saw it cached, but before it read it, is skipped. A relay merges its subscribers' ranges into one upstream request, which +evicted after the subscriber saw it cached, but before it read it, is asked the same +way, once: evicted again before it is read, it is reported as dropped. A relay merges its subscribers' ranges into one upstream request, which moq-lite-07 carries range by range and older wire versions as the single span covering them. A relative or time anchor resolves on the hop that publishes the track, and over moq-lite-07, which carries a relative or time start upstream and diff --git a/quest/m1/unbounded-groups/flights.md b/quest/m1/unbounded-groups/flights.md index b62dccdea2e..54c2f4adf9c 100644 --- a/quest/m1/unbounded-groups/flights.md +++ b/quest/m1/unbounded-groups/flights.md @@ -75,7 +75,3 @@ Public API: none. Wire: none, beyond the existing frame-offset fills. ## Required - [Fastest-reader pacing](/quest/m1/unbounded-groups/fastest-reader.md) - flights pace on their fastest rider the same way the live edge does - -## Related - -- [Refill evicted groups](/quest/m2/evicted-refill.md) - refills whole groups evicted before a reader took them diff --git a/quest/m2/README.md b/quest/m2/README.md index 337557447b7..b29a01bf978 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -83,7 +83,6 @@ lite-07; bitrate enforcement is not part of this release. - [Shared group serving and ingest](/quest/m2/shared-serve.md) - Rust's moq-lite and moq-transport sessions serve and ingest groups through one poll machine each, with only wire encoding per protocol - [Shared group serving and ingest in JS](/quest/m2/shared-serve-js.md) - the same split in js/net, following the Rust seam - [Paired bench BASE](/quest/m2/bench-base-paired.md) - `just bench ` uses the nightly's paired comparison and the old one is deleted -- [Refill evicted groups](/quest/m2/evicted-refill.md) - a group evicted after a reader judged it but before it took it is refilled, never delivered twice, and guarded against thrash - [Receive pool](/quest/m2/receive-pool.md) - one byte budget per worker covers every connection's receive credit and the cache, billing each frame once - [Front parking](/quest/m2/origin-front-parks.md) - an unroutable request waits on a front instead of re-asking on every route-table move - [Route wakes](/quest/m2/route-wakes.md) - a route change wakes only the fronts it can move, so pool churn stops scaling with served paths diff --git a/quest/m2/evicted-refill.md b/quest/m2/evicted-refill.md deleted file mode 100644 index bdd740325ee..00000000000 --- a/quest/m2/evicted-refill.md +++ /dev/null @@ -1,36 +0,0 @@ -# [M] Refill groups evicted before a reader took them - -## Goal - -A reader behind the live edge still gets a group the cache evicted after the -reader judged it but before it took it: the group becomes a miss and is asked -of the track's fills, like one evicted before the reader judged it. A reader -never receives a group it already took twice, and slow readers under cache -pressure cannot make the relay refetch without bound. - -## Plan - -From [moq-dev/moq#6171](https://github.com/moq-dev/moq/pull/6171), where the -maintainer chose to land fills without this (2026-10-21). Today a reader -judges each sequence once: a group cached when judged and evicted later is -skipped silently. - -- Track evictions per reader. An ordered reader can judge again every - eviction at or above its next sequence. An arrival-order reader delivers - above its floor, so it must tell the groups it already walked past from - the ones still ahead, for example by keeping each evicted group's arrival - index. Bound what the track keeps by the slowest reader, not by the cache. -- Guard against thrash. Under pool pressure a refill evicts another group, - possibly the next one the same slow reader needs. Decide in the PR, with - options and a recommendation, how a reader stops refilling once its - refills only evict what it reads next: give up to the drop cursor, or rely - on its max delay. -- Test a slow ascending reader on a pooled track under bufferbloat: it - receives every group its budget covers, takes none twice, and refills stay - proportional to what it reads. - -Public API: none. Wire: none. - -## Related - -- [Unbounded groups](/quest/m1/unbounded-groups/README.md) - refills the parts of a group the cache sheds under pressure diff --git a/rs/moq-net/benches/track.rs b/rs/moq-net/benches/track.rs index 53776a22d15..ea391399f9c 100644 --- a/rs/moq-net/benches/track.rs +++ b/rs/moq-net/benches/track.rs @@ -46,6 +46,11 @@ //! wall clock: each empty group is handed out like any other, so a walk over the rest of //! the run per group shows up as a slope over it. //! +//! `track_evict_fanout` writes groups into a full pool, each evicting one that every reader +//! judged cached, swept over the readers and the groups written between their reads: each +//! eviction leaves the group in every reader's inbox, so its cost grows with the readers, +//! and a cost that grows with the groups written beyond that shows up as a slope. +//! //! `track_fanout_frame` writes frames into an open group while every subscriber is parked //! on the next one, re-polling whichever reader the write woke: a frame wakes no track //! reader, so waking them shows up as a slope over the subscribers. @@ -553,6 +558,46 @@ fn bench_fanout(c: &mut Criterion) { group.finish(); } +/// Groups written between reads in `track_evict_fanout`, each evicting one. +const EVICTED: [usize; 3] = [1, 8, 64]; + +fn bench_evict_fanout(c: &mut Criterion) { + let mut group = c.benchmark_group("track_evict_fanout"); + for subscribers in FANOUT { + for evicted in EVICTED { + group.throughput(Throughput::Elements(evicted as u64)); + group.bench_function(BenchmarkId::new(format!("{subscribers}r"), evicted), |b| { + let mut fanout = Fanout::new(subscribers); + // Twice what the pool holds, so every write evicts. + for _ in 0..2 * CACHE_CAPACITY / PAYLOAD as u64 { + fanout.cycle(); + } + // Only the writes and the evictions they cause are timed, not the reads that + // judge each written group cached again. + b.iter_custom(|iterations| { + let mut elapsed = Duration::ZERO; + for _ in 0..iterations { + let start = Instant::now(); + for _ in 0..evicted { + let mut group = fanout.track.append_group().unwrap(); + group.write_frame(Timestamp::ZERO, fanout.payload.clone()).unwrap(); + group.finish().unwrap(); + } + elapsed += start.elapsed(); + for (subscriber, waiter) in fanout.subscribers.iter_mut().zip(&fanout.waiters) { + while let Poll::Ready(Ok(Some(group))) = subscriber.poll_recv_group(waiter) { + black_box(group); + } + } + } + elapsed + }); + }); + } + } + group.finish(); +} + /// Frames written per group in `track_fanout_frame`, after which the group rolls and every /// subscriber takes the next one: enough that the roll's cost per frame stays small. const FRAMES_PER_GROUP: usize = 256; @@ -1946,6 +1991,7 @@ criterion_group!( bench_budget_widening, bench_fanout, bench_fanout_frame, + bench_evict_fanout, bench_fetch_expiry, bench_fetch_budgets, bench_fetch_park, diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index fb451d50464..97264338d27 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -464,6 +464,15 @@ pub(crate) struct TrackState { // Bumped on every insert into `drops`, so a cursor rescans it only after a change. drops_revision: u64, drops_changes: ledger::Changes, + // Every reader's eviction inbox. Its own lock, so a reader registers through the read + // guard it judges under. + inboxes: Mutex, + // Bumped whenever an eviction lands in an inbox, so a reader with nothing new skips + // its own lock. + evictions: u64, + // Bumped as each group is cached, stamped on its slot: a reader judged a slot cached + // only if it was cached by the count the reader last judged at. + cached: u64, // Whether the cache reflects the live feed: readers get nothing from it while it does // not, since how stale it is cannot be told; fetches still do. A track is live from @@ -547,6 +556,9 @@ struct Slot { // the replacement and deliver it twice or at the wrong position. stamp: u32, + // [`TrackState::cached`] once this incarnation was cached. + cached: u64, + // Whether this incarnation came from the live publisher and can replace older // subscription content. Fetch-only backfill stays cached but never anchors drift. visible: bool, @@ -938,6 +950,21 @@ impl TrackState { Some(slot) } + /// [`Self::uncache`] a servable group to free the cache and abort it with `err`, recording + /// it for the readers that judged it cached to judge again. + fn evict_group(&mut self, sequence: u64, err: Error) { + let slot = self.uncache(sequence).expect("a cached group"); + slot.abort(err); + let landed = self + .inboxes + .get_mut() + .unwrap_or_else(PoisonError::into_inner) + .evicted(sequence, slot.cached); + if landed { + self.evictions += 1; + } + } + /// [`Self::uncache`] a group that another copy displaces rather than one that ends, so a /// teardown still ends the copies its readers hold. fn displace(&mut self, sequence: u64) { @@ -1760,7 +1787,7 @@ impl TrackState { // Take the group out of the cache and abort it, so any consumer // still reading surfaces `Error::Old` instead of blocking forever // on a frame that will never arrive. - self.uncache(sequence).unwrap().abort(Error::Old); + self.evict_group(sequence, Error::Old); } } @@ -2036,11 +2063,13 @@ impl TrackState { if visible { self.record_arrival(sequence, now); } + self.cached += 1; let replaced = self.lookup.insert( sequence, Slot { group: group.clone(), stamp, + cached: self.cached, visible, arrived: now, head: None, @@ -2230,7 +2259,7 @@ impl TrackState { self.debt -= size; paid = paid.saturating_add(size); - self.uncache(sequence).unwrap().abort(Error::Evicted); + self.evict_group(sequence, Error::Evicted); } // No cold group left to pay with: shed the front of the groups that could not go // whole, the ones being read and then the live edge, so neither grows without bound. @@ -6235,7 +6264,7 @@ impl Cursor { } #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - fn poll_drop(&mut self, waiter: &kio::Waiter) -> Poll<(ops::Range, StreamError)> { + fn poll_drop(&mut self, taken: &Delivered, waiter: &kio::Waiter) -> Poll<(ops::Range, StreamError)> { self.sync(); if self.drops.is_none() { let mut drops = Box::new(DropCursor::new(&self.state.read())); @@ -6249,7 +6278,9 @@ impl Cursor { if self.filter.refused.is_none() && let Poll::Ready(Ok(drift)) = self.poll_drift(&self.servable, waiter) { - self.probe(self.floor(), &drift, waiter); + // From where the reader reads next: an ordered one asks nothing below what it took. + let from = self.next_sequence.unwrap_or(u64::MAX).max(self.floor()); + self.probe(from, &drift, taken, waiter); } // A cap at or below the floor leaves nothing to report, not a reversed span. let floor = self.floor(); @@ -6658,7 +6689,8 @@ impl Cursor { /// Ask fills for what this cursor misses from `from`, and take in what they answered, /// parking `waiter` on the fills still out. Whatever it gives up goes to the drop cursor. - fn probe(&mut self, from: u64, drift: &Drift, waiter: &kio::Waiter) { + /// `taken` is what its subscriber handed out, which an eviction never makes a miss. + fn probe(&mut self, from: u64, drift: &Drift, taken: &Delivered, waiter: &kio::Waiter) { let (window, cap, revision) = (&self.filter.window, self.end_sequence, self.filter.revision); let asking = { let subscription = self.subscription.read(); @@ -6691,6 +6723,7 @@ impl Cursor { .map(|run| *run.start()..=(*run.end()).min(end)) .collect() }, + taken, drift, &mut given_up, waiter, @@ -6732,7 +6765,12 @@ impl Cursor { self.misses.as_ref().is_some_and(|misses| misses.is_waiting()) } - fn poll_recv_group(&mut self, take: bool, waiter: &kio::Waiter) -> Poll>> { + fn poll_recv_group( + &mut self, + take: bool, + taken: &Delivered, + waiter: &kio::Waiter, + ) -> Poll>> { if let Some(peeked) = self.peeked.take().filter(|peeked| take && !peeked.ordered) { // The reserved group is still the one handed out, but bounded by the ranges in // force now, which a `Control` may have replaced since the peek. @@ -6751,7 +6789,7 @@ impl Cursor { // One scan for the whole poll, so walking a backlog off stays linear in its size. let drift = ready!(self.poll_drift(&self.servable, waiter))?; self.reoffer(drift.budget); - self.probe(self.floor(), &drift, waiter); + self.probe(self.floor(), &drift, taken, waiter); self.poll_recv_with(take, drift, waiter) } @@ -6935,7 +6973,12 @@ impl Cursor { /// The lowest servable group past the last one returned, walking off everything the /// drift budget has convicted on the way. - fn poll_next_group(&mut self, take: bool, waiter: &kio::Waiter) -> Poll>> { + fn poll_next_group( + &mut self, + take: bool, + taken: &Delivered, + waiter: &kio::Waiter, + ) -> Poll>> { if let Some(peeked) = self.peeked.take().filter(|peeked| take && peeked.ordered) { // Bounded by the ranges in force now, as on the arrival path. self.sync(); @@ -6951,7 +6994,7 @@ impl Cursor { let floor = next.max(self.floor()); // One scan for the whole poll, so walking a backlog off stays linear in its size. let drift = ready!(self.poll_drift(&self.servable, waiter))?; - self.probe(floor, &drift, waiter); + self.probe(floor, &drift, taken, waiter); self.poll_next_with(take, floor, drift, waiter) } @@ -7303,8 +7346,8 @@ impl Subscriber { let res = match &mut self.inner { Inner::Plain(cursor) => loop { let res = match ordered { - true => cursor.poll_next_group(true, waiter), - false => cursor.poll_recv_group(true, waiter), + true => cursor.poll_next_group(true, &self.delivered, waiter), + false => cursor.poll_recv_group(true, &self.delivered, waiter), }; match res { Poll::Ready(Ok(Some(group))) if self.delivered.contains(group.sequence) => continue, @@ -7366,8 +7409,8 @@ impl Subscriber { match &mut self.inner { Inner::Plain(cursor) => loop { let group = ready!(match ordered { - true => cursor.poll_next_group(false, waiter), - false => cursor.poll_recv_group(false, waiter), + true => cursor.poll_next_group(false, &self.delivered, waiter), + false => cursor.poll_recv_group(false, &self.delivered, waiter), })?; let Some(group) = group else { return Poll::Ready(Ok(None)); @@ -7377,8 +7420,8 @@ impl Subscriber { } // A replay advances only this copy's cursor, as an ordinary read would. ready!(match ordered { - true => cursor.poll_next_group(true, waiter), - false => cursor.poll_recv_group(true, waiter), + true => cursor.poll_next_group(true, &self.delivered, waiter), + false => cursor.poll_recv_group(true, &self.delivered, waiter), })?; }, Inner::Resume(resume, _) => resume.poll_peek_group(ordered, &self.delivered, waiter), @@ -7725,7 +7768,7 @@ impl Subscriber { #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub(crate) fn poll_drop(&mut self, waiter: &kio::Waiter) -> Poll<(ops::Range, StreamError)> { match &mut self.inner { - Inner::Plain(cursor) => cursor.poll_drop(waiter), + Inner::Plain(cursor) => cursor.poll_drop(&self.delivered, waiter), Inner::Resume(resume, _) => resume.poll_drop(&self.delivered, waiter), } } @@ -19222,6 +19265,294 @@ mod test { assert_eq!(asked, [(4, 6), (0, 2)]); } + /// A group evicted after an arrival-order reader judged it cached, before it took it, is + /// asked of a fill like one evicted before: the reader still receives it, and a group it + /// took is never asked again. + #[test] + #[cfg(feature = "async")] + fn a_group_evicted_before_its_reader_took_it_is_refilled() { + let producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + let mut subscriber = producer.subscribe(replay()); + for sequence in 0..3 { + write_group(&producer, sequence); + } + let waiter = kio::Waiter::noop(); + let Poll::Ready(Ok(Some(first))) = subscriber.poll_recv_group(&waiter) else { + panic!("group 0 is cached"); + }; + assert_eq!(first.sequence, 0); + evict(&producer, 0); + evict(&producer, 1); + assert_eq!(drain_groups(&mut subscriber), [2]); + let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else { + panic!("group 1 is a miss now"); + }; + assert_eq!((fill.start().group, fill.end()), (1, 2), "group 0 was taken"); + assert_eq!(fill.wanted(), Some(1)); + write_filled(fill.accept(1, None), 1); + drop(fill); + assert_eq!(drain_groups(&mut subscriber), [1]); + assert!(dynamic.poll_requested_fill(&waiter).is_pending()); + assert!(drained_drops(&mut subscriber).is_empty(), "nothing was given up"); + let Inner::Plain(cursor) = &subscriber.inner else { + unreachable!("a plain subscription"); + }; + let misses = cursor.misses.as_ref().expect("the reader judged its misses"); + assert!(misses.refilled().is_empty(), "a refill handed out is forgotten"); + } + + /// An eviction waits only for a reader that judged the evicted group cached: a reader + /// that judged once and stalled holds no more than what was cached then, however many + /// groups the feed caches and evicts after, late lower ones included. + #[test] + #[cfg(feature = "async")] + fn an_eviction_waits_only_for_readers_that_judged_it_cached() { + let producer = track_producer("test", Info::timed()); + let _dynamic = producer.requests(); + write_group(&producer, 0); + write_group(&producer, 1); + write_group(&producer, 1 << 40); + let held = || producer.state.read().inboxes.lock().unwrap().held(); + evict(&producer, 0); + assert!(held().is_empty(), "no reader judged group 0"); + + let mut stalled = producer.subscribe(replay()); + let mut active = producer.subscribe(replay()); + drain_groups(&mut stalled); + drain_groups(&mut active); + // Late groups below the stalled reader's newest, cached after it judged. + for sequence in (2..200).step_by(2) { + write_group(&producer, sequence); + drain_groups(&mut active); + evict(&producer, sequence); + } + // The active reader judged each one cached, so it holds the last until it judges again. + assert_eq!(held(), [vec![], vec![(198, 199)]], "the stalled reader judged none"); + drain_groups(&mut active); + evict(&producer, 1); + assert_eq!(held(), [vec![(1, 2)], vec![(1, 2)]], "both judged group 1 cached"); + drain_groups(&mut active); + assert_eq!(held(), [vec![(1, 2)], vec![]], "the active reader took it in"); + drop(stalled); + assert_eq!(held(), [vec![]], "a dropped reader holds nothing"); + write_group(&producer, 3); + evict(&producer, 3); + assert_eq!( + producer.state.read().inboxes.lock().unwrap().len(), + 1, + "an eviction prunes a dropped reader's inbox" + ); + } + + /// An ordered reader forwarding drops asks nothing below what it took, though its drop + /// cursor reports from the window's floor. + #[test] + #[cfg(feature = "async")] + fn an_ordered_readers_drop_poll_asks_nothing_it_took() { + let producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + let mut subscriber = producer.subscribe(replay().with_order(Order::Ascending)); + for sequence in 0..3 { + write_group(&producer, sequence); + } + let waiter = kio::Waiter::noop(); + for expected in 0..2 { + let Poll::Ready(Ok(Some(group))) = subscriber.poll_next_group(&waiter) else { + panic!("group {expected} is cached"); + }; + assert_eq!(group.sequence, expected); + } + evict(&producer, 0); + evict(&producer, 1); + assert!(drained_drops(&mut subscriber).is_empty()); + assert!(dynamic.poll_requested_fill(&waiter).is_pending(), "both were taken"); + } + + /// An ordered reader waits on the refill of a group evicted at or above its next + /// sequence, and reads it in order. + #[test] + #[cfg(feature = "async")] + fn an_ordered_reader_refills_an_evicted_group_in_order() { + let producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + let mut subscriber = producer.subscribe(replay().with_order(Order::Ascending)); + for sequence in 0..4 { + write_group(&producer, sequence); + } + let waiter = kio::Waiter::noop(); + let next = |subscriber: &mut Subscriber| match subscriber.poll_next_group(&kio::Waiter::noop()) { + Poll::Ready(Ok(Some(group))) => Some(group.sequence), + _ => None, + }; + assert_eq!(next(&mut subscriber), Some(0)); + evict(&producer, 1); + evict(&producer, 2); + assert_eq!(next(&mut subscriber), None, "group 1 comes before 3"); + let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else { + panic!("groups 1 and 2 are misses now"); + }; + assert_eq!((fill.start().group, fill.end()), (1, 3)); + while let Some(sequence) = fill.wanted() { + write_filled(fill.accept(sequence, None), sequence); + } + drop(fill); + assert_eq!([next(&mut subscriber), next(&mut subscriber)], [Some(1), Some(2)]); + assert_eq!(next(&mut subscriber), Some(3)); + } + + /// An ordered reader gives up a refill evicted again while it waits on a lower one, + /// naming it dropped and releasing its fill, rather than reading past it unnamed. + #[test] + #[cfg(feature = "async")] + fn an_ordered_reader_gives_up_a_refill_evicted_while_it_waits_below() { + let producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + let mut subscriber = producer.subscribe(replay().with_order(Order::Ascending)); + for sequence in 0..4 { + write_group(&producer, sequence); + } + let waiter = kio::Waiter::noop(); + let next = |subscriber: &mut Subscriber| match subscriber.poll_next_group(&kio::Waiter::noop()) { + Poll::Ready(Ok(Some(group))) => Some(group.sequence), + _ => None, + }; + assert_eq!(next(&mut subscriber), Some(0)); + evict(&producer, 1); + evict(&producer, 2); + assert_eq!(next(&mut subscriber), None, "group 1 comes before 3"); + let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else { + panic!("groups 1 and 2 are misses now"); + }; + write_filled(fill.accept(2, None), 2); + assert_eq!(next(&mut subscriber), None, "still waiting on group 1"); + evict(&producer, 2); + assert_eq!(next(&mut subscriber), None, "still waiting on group 1"); + assert_eq!(drained_drops(&mut subscriber), [(2, 3)]); + write_filled(fill.accept(1, None), 1); + drop(fill); + assert_eq!([next(&mut subscriber), next(&mut subscriber)], [Some(1), Some(3)]); + assert!(dynamic.poll_requested_fill(&waiter).is_pending(), "refilled once"); + } + + /// A refill evicted again before its reader took it is given up rather than asked a + /// second time, so refills that evict each other cannot loop. + #[test] + #[cfg(feature = "async")] + fn a_refill_evicted_again_is_given_up() { + let producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + let mut subscriber = producer.subscribe(replay()); + for sequence in 0..3 { + write_group(&producer, sequence); + } + let waiter = kio::Waiter::noop(); + assert!(matches!(subscriber.poll_recv_group(&waiter), Poll::Ready(Ok(Some(_))))); + evict(&producer, 1); + assert_eq!(drain_groups(&mut subscriber), [2]); + let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else { + panic!("group 1 is a miss now"); + }; + assert_eq!(fill.wanted(), Some(1)); + write_filled(fill.accept(1, None), 1); + drop(fill); + evict(&producer, 1); + assert!(drain_groups(&mut subscriber).is_empty()); + assert!(dynamic.poll_requested_fill(&waiter).is_pending(), "refilled once"); + assert_eq!(drained_drops(&mut subscriber), [(1, 2)]); + } + + /// A slow ascending reader on a pooled track holding `cached` groups, taking `pace` of + /// every four its feed writes, as behind a bufferbloated link. Returns what it read, how + /// many groups the fills wrote, and what it gave up. + #[cfg(feature = "async")] + fn read_slowly(cached: usize, pace: u64) -> (Vec, u64, Vec<(u64, u64)>) { + let size = 10_000; + let (producer, _pool) = pooled_producer((cached * size) as u64); + let dynamic = producer.requests(); + let mut subscriber = producer.subscribe(replay().with_order(Order::Ascending)); + let write = |group: Result, sequence: u64| { + let mut group = group.unwrap(); + group + .write_frame(Timestamp::from_millis(sequence).unwrap(), vec![0u8; size]) + .unwrap(); + group.finish().unwrap(); + }; + let waiter = kio::Waiter::noop(); + let mut refilled = 0; + let mut serve = || { + while let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) { + while let Some(sequence) = fill.wanted() { + write(fill.accept(sequence, None), sequence); + refilled += 1; + } + } + }; + let (mut read, mut drops) = (Vec::new(), Vec::new()); + // A wire publisher forwards what the reader gives up as it reads. + let mut take = |subscriber: &mut Subscriber| { + let taken = match subscriber.poll_next_group(&kio::Waiter::noop()) { + Poll::Ready(Ok(Some(group))) => { + read.push(group.sequence); + true + } + _ => false, + }; + drops.extend(drained_drops(subscriber)); + taken + }; + for sequence in 0..GROUPS { + write(producer.create_group(group::Info { sequence }), sequence); + if sequence % 4 < pace { + take(&mut subscriber); + } + serve(); + } + // The feed pauses and the reader drains what it can, refills included. + loop { + serve(); + if !take(&mut subscriber) { + serve(); + if !take(&mut subscriber) { + break; + } + } + } + (read, refilled, drops) + } + + #[cfg(feature = "async")] + const GROUPS: u64 = 40; + + /// A backlog a little past the cache: what the reader judged and lost to eviction is + /// refilled, so it receives every group once, in order, refilling no more than it reads. + #[test] + #[cfg(feature = "async")] + fn a_slow_reader_on_a_pooled_track_receives_every_group() { + let (read, refilled, drops) = read_slowly(8, 3); + assert_eq!(read, (0..GROUPS).collect::>(), "every group once, in order"); + assert!(drops.is_empty(), "nothing given up: {drops:?}"); + assert!(refilled > 0, "the backlog outgrew the cache"); + assert!(refilled <= GROUPS, "{refilled} refills for {GROUPS} groups read"); + } + + /// A backlog far past the cache, where refills evict the groups the reader reads next: + /// each group is refilled at most once, and one lost again is given up and named, never + /// handed out twice. + #[test] + #[cfg(feature = "async")] + fn a_slow_reader_past_the_cache_refills_each_group_once() { + let (read, refilled, drops) = read_slowly(4, 2); + let mut seen = read.clone(); + seen.dedup(); + assert_eq!(seen, read, "none twice"); + assert!(read.is_sorted(), "in order"); + assert!(refilled <= GROUPS, "{refilled} refills for {GROUPS} groups"); + let lost: Vec = (0..GROUPS).filter(|sequence| !read.contains(sequence)).collect(); + let named: Vec = drops.iter().flat_map(|&(start, end)| start..end).collect(); + assert_eq!(lost, named, "every group lost is named dropped"); + } + /// A reader forgets the misses it gave up once its window moves past them, as it forgets /// what it judged, so a feed skipping every other group costs it only the gaps it can /// still read. @@ -20013,12 +20344,7 @@ mod test { /// Take group `sequence` out of the cache as eviction does. fn evict(producer: &Producer, sequence: u64) { - producer - .modify() - .unwrap() - .uncache(sequence) - .unwrap() - .abort(Error::Evicted); + producer.modify().unwrap().evict_group(sequence, Error::Evicted); } /// A rewound first frame is refused and leaves the group as it was, so the publisher can diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index 6bc2dc064ec..45f5b710329 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -7,8 +7,9 @@ //! (evicted, or a gap the feed skipped). A hole the ledger still waits for is deferred and //! judged again as the feed moves. //! -//! Each sequence is judged once, so a group evicted after the reader judged it cached is -//! not a miss: judging it again would refill a group an arrival-order reader already took. +//! Each sequence is judged once, except a group evicted after the reader judged it cached +//! and before it took it: judged again as a miss, once. A refill evicted again before it is +//! taken is given up, so a slow reader's refills cannot evict each other without bound. //! //! A start is no promise, since a group below it may still arrive late. A hole under it is //! asked at once, but a refusal is final only once the ledger gives up on it too; until @@ -18,12 +19,14 @@ use std::{ collections::{BTreeMap, BTreeSet}, ops::{self, RangeInclusive}, + sync::{Arc, Mutex, PoisonError}, task::Poll, time::Duration, }; use rangemap::{RangeInclusiveSet, RangeSet}; +use super::super::delivered::Delivered; pub(super) use super::fill::Asking; use super::fill::{Ask, FillState, Join}; use super::{Drift, Order, Position, TrackState}; @@ -60,6 +63,12 @@ pub(super) struct Misses { filled: BTreeSet, /// Never asked again: named dropped to the reader. given_up: RangeSet, + /// Where the track leaves what it evicts that this reader judged cached, once it judged. + inbox: Option, + /// The track's eviction count this reader last emptied its inbox at. + evictions: u64, + /// Judged again after an eviction, within the judged span: never refilled twice. + refilled: RangeSet, } /// A run of misses the reader gives up, for the drop cursor to name. @@ -190,8 +199,14 @@ impl Misses { &self.given_up } - /// The next answered group still cached in `order`, forgetting one evicted before it was - /// read. + /// What was refilled after an eviction and not handed out yet. + #[cfg(all(test, feature = "async"))] + pub fn refilled(&self) -> &RangeSet { + &self.refilled + } + + /// The next answered group still cached in `order`, giving up one evicted before it was + /// read: [`Self::judge`] asked a first eviction again, so this one was refilled already. pub fn next_filled( &mut self, state: &TrackState, @@ -213,8 +228,13 @@ impl Misses { } } - /// Take `sequence` as handed out, if a fill answered it. + /// Take `sequence` as handed out, if a fill answered it. A refill handed out is never + /// judged again, so it is forgotten too. pub fn take_filled(&mut self, sequence: u64) { + // The last sequence is never judged, so never refilled. + if let Some(end) = sequence.checked_add(1) { + self.refilled.remove(sequence..end); + } if self.filled.remove(&sequence) { self.release(sequence); } @@ -336,6 +356,10 @@ impl Misses { taken.remove(sequence..sequence + 1); } } + // Seen cached, though judging may not run again before it is evicted. + if let Some(inbox) = &self.inbox { + inbox.saw(state.cached); + } } Err((Error::Old, asked)) if looser(max_delay, asked) => retry.insert(groups), // A drop the track recorded is named by the drop cursor already. @@ -351,7 +375,8 @@ impl Misses { /// Judge the holes in `target` the last probe has not, returning the runs to ask a /// fill for. Gives up a run already stale under this reader's budget. `named` is the - /// lowest group the reader's ranges name, which may reach below the feed's start. + /// lowest group the reader's ranges name, which may reach below the feed's start, and + /// `taken` what the reader handed out, which an eviction never makes a miss again. #[expect( clippy::too_many_arguments, reason = "one probe's inputs, borrowed apart from its cursor" @@ -362,6 +387,7 @@ impl Misses { key: (u64, u64, Option), named: Option, mut target: impl FnMut(RangeInclusive) -> RangeInclusiveSet, + taken: &Delivered, drift: &Drift, given_up: &mut GivenUp, waiter: &kio::Waiter, @@ -393,7 +419,8 @@ impl Misses { && self.budget == Some(drift.budget) && self.ended == ended && self.ledger_revision == ledger.revision() - && self.drops_revision == drops.revision(); + && self.drops_revision == drops.revision() + && self.evictions == state.evictions; // The grace only times deferred gaps, so with none a new one has nothing to move. // It is judged once something else changes. if unchanged && self.deferred.runs.is_empty() { @@ -436,6 +463,7 @@ impl Misses { self.judged = clip(&self.judged); self.refused = clip(&self.refused); self.given_up = clip(&self.given_up); + self.refilled = clip(&self.refilled); for (&start, &(end, _)) in &self.deferred.runs { for run in span.overlapping(&(start..end)) { candidates.insert(start.max(run.start)..end.min(run.end)); @@ -451,7 +479,12 @@ impl Misses { // A moving read floor removes only its retired prefix. New production extends the // judged span; neither path visits older deferred gaps. if low > 0 { - for set in [&mut self.judged, &mut self.refused, &mut self.given_up] { + for set in [ + &mut self.judged, + &mut self.refused, + &mut self.given_up, + &mut self.refilled, + ] { set.remove(0..low); } self.deferred.remove(0..low); @@ -497,6 +530,65 @@ impl Misses { self.refused.remove(sequence..sequence + 1); } } + // What it judged cached and the track evicted since, before this reader took it, is a + // miss now. + let inbox = self.inbox.get_or_insert_with(|| { + let inbox = Inbox::default(); + state + .inboxes + .lock() + .unwrap_or_else(PoisonError::into_inner) + .register(inbox.clone()); + inbox + }); + let evicted = inbox.judge(state.cached, low, top); + self.evictions = state.evictions; + let mut refills = RangeSet::new(); + let mut again = RangeSet::new(); + for run in evicted.iter() { + let run = run.start.max(low)..run.end.min(top.saturating_add(1)); + if run.is_empty() { + continue; + } + for part in target(run.start..=run.end - 1).iter() { + for gap in taken.gaps(*part.start()..*part.end() + 1) { + for judged in self.judged.overlapping(&gap) { + let judged = judged.start.max(gap.start)..judged.end.min(gap.end); + for fresh in self.refilled.gaps(&judged) { + refills.insert(fresh); + } + for refilled in self.refilled.overlapping(&judged) { + again.insert(refilled.start.max(judged.start)..refilled.end.min(judged.end)); + } + } + } + } + } + // Settled here rather than when read, since an ordered reader never looks at what a + // fill answered: it reads past a group gone from the cache. Still asked, its fill + // settles it; given up already, it was named. + for run in self.wanted.iter().chain(self.given_up.iter()) { + again.remove(run.clone()); + } + for (run, refill) in refills + .iter() + .map(|run| (run, true)) + .chain(again.iter().map(|run| (run, false))) + { + // A fill's answer evicted before it was taken is no longer held for this reader. + let filled: Vec = self.filled.range(run.clone()).copied().collect(); + for sequence in filled { + self.filled.remove(&sequence); + self.release(sequence); + } + match refill { + true => { + self.refilled.insert(run.clone()); + candidates.insert(run.clone()); + } + false => self.give_up(run.clone(), given_up), + } + } self.ledger_revision = ledger.revision(); self.drops_revision = drops.revision(); self.budget = Some(drift.budget); @@ -725,6 +817,96 @@ impl Misses { } } +/// What the track evicted that one reader judged cached, and what it judged: a slot cached +/// by the count it judged at, within the span it judged. Holds no more than the slots it +/// judged cached, so a stalled reader pins nothing cached after. +#[derive(Clone, Default)] +pub(super) struct Inbox(Arc>); + +#[derive(Default)] +struct Judging { + cached: u64, + span: Option>, + evicted: RangeSet, +} + +impl Inbox { + /// Take what was evicted since the last call, and judge `low..=top` from here on, with + /// every slot cached by `cached`. + fn judge(&self, cached: u64, low: u64, top: u64) -> RangeSet { + let mut judging = self.0.lock().unwrap_or_else(PoisonError::into_inner); + judging.cached = cached; + judging.span = Some(low..=top); + std::mem::take(&mut judging.evicted) + } + + /// Count every slot cached by `cached` as judged cached, within the span judged last. + fn saw(&self, cached: u64) { + let mut judging = self.0.lock().unwrap_or_else(PoisonError::into_inner); + judging.cached = judging.cached.max(cached); + } +} + +/// Every reader's [`Inbox`], pruned of dropped readers on each eviction, or once +/// registrations double it past what the last prune kept. +#[derive(Default)] +pub(super) struct Inboxes { + inboxes: Vec, + kept: usize, +} + +impl Inboxes { + fn register(&mut self, inbox: Inbox) { + self.inboxes.push(inbox); + if self.inboxes.len() > 2 * self.kept.max(1) { + self.inboxes.retain(|inbox| Arc::strong_count(&inbox.0) > 1); + self.kept = self.inboxes.len(); + } + } + + /// How many inboxes are registered, live or not. + #[cfg(all(test, feature = "async"))] + pub(super) fn len(&self) -> usize { + self.inboxes.len() + } + + /// What each live reader's inbox holds, in registration order. + #[cfg(all(test, feature = "async"))] + pub(super) fn held(&self) -> Vec> { + self.inboxes + .iter() + .filter(|inbox| Arc::strong_count(&inbox.0) > 1) + .map(|inbox| { + let judging = inbox.0.lock().unwrap_or_else(PoisonError::into_inner); + judging.evicted.iter().map(|run| (run.start, run.end)).collect() + }) + .collect() + } + + /// Leave `sequence`, cached as count `cached`, with each reader that judged it cached. + /// Returns whether any did. + pub(super) fn evicted(&mut self, sequence: u64, cached: u64) -> bool { + let Some(end) = sequence.checked_add(1) else { + return false; + }; + let mut landed = false; + // Every inbox is visited anyway, so a dropped reader's goes here too. + self.inboxes.retain(|inbox| { + if Arc::strong_count(&inbox.0) == 1 { + return false; + } + let mut judging = inbox.0.lock().unwrap_or_else(PoisonError::into_inner); + if cached <= judging.cached && judging.span.as_ref().is_some_and(|span| span.contains(&sequence)) { + judging.evicted.insert(sequence..end); + landed = true; + } + true + }); + self.kept = self.inboxes.len(); + landed + } +} + impl TrackState { /// Split `hole`, which the cache holds nothing for, into what the live feed will never /// deliver and the gaps it still waits for, with when each stops being waited for. @@ -857,6 +1039,7 @@ mod tests { (0, 0, None), None, |span| [span].into_iter().collect(), + &Delivered::default(), &drift, &mut given_up, &waiter, @@ -896,6 +1079,7 @@ mod tests { (0, 0, None), None, |_| panic!("the edge has not moved"), + &Delivered::default(), &drift, &mut given_up, &waiter, @@ -920,6 +1104,7 @@ mod tests { (0, 0, None), None, |_| panic!("recreation extends no span"), + &Delivered::default(), &drift, &mut given_up, &waiter, @@ -961,6 +1146,7 @@ mod tests { (0, 0, None), None, |span| [span].into_iter().collect(), + &Delivered::default(), &drift, &mut given_up, &waiter, @@ -994,6 +1180,7 @@ mod tests { (0, 0, None), None, |_| panic!("the edge has not moved"), + &Delivered::default(), &drift, &mut given_up, &waiter, @@ -1030,6 +1217,7 @@ mod tests { (0, 0, None), None, |span| [span].into_iter().collect(), + &Delivered::default(), &drift, &mut given_up, &waiter @@ -1045,6 +1233,7 @@ mod tests { (0, 0, None), None, |_| panic!("an unchanged probe rebuilds no span"), + &Delivered::default(), &drift, &mut given_up, &waiter @@ -1065,6 +1254,7 @@ mod tests { (0, 0, None), None, |_| panic!("a late arrival extends no span"), + &Delivered::default(), &drift, &mut given_up, &waiter @@ -1081,6 +1271,7 @@ mod tests { (0, 0, None), None, |_| panic!("a due gap extends no span"), + &Delivered::default(), &drift, &mut given_up, &waiter, @@ -1117,7 +1308,16 @@ mod tests { let full = |span| [span].into_iter().collect(); assert!( misses - .judge(&state, (0, 0, None), None, full, &drift, &mut given_up, &waiter) + .judge( + &state, + (0, 0, None), + None, + full, + &Delivered::default(), + &drift, + &mut given_up, + &waiter + ) .is_empty() ); assert!(misses.deferred.runs.is_empty(), "a contiguous feed defers nothing"); @@ -1134,6 +1334,7 @@ mod tests { (0, 0, None), None, |_| panic!("nothing deferred: the new grace waits for a change"), + &Delivered::default(), &drift, &mut given_up, &waiter @@ -1151,7 +1352,16 @@ mod tests { let opened = state.cache.pool().now_wall(); assert!( misses - .judge(&state, (0, 0, None), None, full, &drift, &mut given_up, &waiter) + .judge( + &state, + (0, 0, None), + None, + full, + &Delivered::default(), + &drift, + &mut given_up, + &waiter + ) .is_empty() ); assert_eq!(misses.deferred.overlapping(0..5).collect::>(), vec![3..4]);