diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 90244e6a47a..df29ff5092a 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -14,10 +14,9 @@ narrower budget is served after a widening, as a cold relay would. ## Plan -Decided 2026-10-27 (maintainer): this runs after -[Range count cost](/quest/m1/subscribe-ranges/fill-linear.md), since both edit -`given_up` in `rs/moq-net/src/model/track/miss.rs` (made a `Required` entry -2026-10-30, in [moq-dev/moq#6995](https://github.com/moq-dev/moq/pull/6995)), and before +Decided 2026-10-27 (maintainer): this runs after the range count cost work, +which finished in [moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), +since both edit `given_up` in `rs/moq-net/src/model/track/miss.rs`, and before [Rust groups by start](/quest/m1/time-addressing/model.md), which requires it because runs keyed by group sequence convert mechanically to starts. @@ -304,10 +303,6 @@ a cancelled fetch below a track's end still holds the end open until the cache expiry scan removes it, so how fast a subscription ends depends on expiry timing. Settle it here, where cancelled and expired runs get a reason. -## Required - -- [Range count cost](/quest/m1/subscribe-ranges/fill-linear.md) - the remaining range-count walks stop growing per poll; both edit `miss.rs` - ## Related - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - fills the cache miss an evicted run leaves diff --git a/quest/m1/subscribe-ranges/README.md b/quest/m1/subscribe-ranges/README.md index d9028eb290c..0ef6ae40a02 100644 --- a/quest/m1/subscribe-ranges/README.md +++ b/quest/m1/subscribe-ranges/README.md @@ -103,7 +103,6 @@ every sequence delivered or dropped. ## Required - [Fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md) - `fetch_group` becomes a one-range subscription that ends once its range is accounted for -- [Range count cost](/quest/m1/subscribe-ranges/fill-linear.md) - time anchors, outstanding fills, sparse live numbering, and lite-07 frame bounds stop walking every range per poll - [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) - the lite-07 wire, Rust publisher and subscriber, and the draft - [JS ranges](/quest/m1/subscribe-ranges/js.md) - `@moq/net` model and lite-07 parity, and the on-demand request surface - [Remove FETCH from lite-07](/quest/m1/subscribe-ranges/lite07-fetch.md) - FETCH removed from lite-07 in Rust, JS, and the draft diff --git a/quest/m1/subscribe-ranges/fill-linear.md b/quest/m1/subscribe-ranges/fill-linear.md deleted file mode 100644 index 882e934c3af..00000000000 --- a/quest/m1/subscribe-ranges/fill-linear.md +++ /dev/null @@ -1,42 +0,0 @@ -# [M] A subscription's range count cannot multiply per-poll cost - -## Goal - -The model's cost per poll no longer grows with the number of ranges in a -subscription, or that number is bounded by the design (flow control or an -owned budget), not by a refusal constant. A benchmark sweeps range count for -each path below, so a slope shows up. - -## Plan - -The fill path itself went linear in -[moq-dev/moq#6995](https://github.com/moq-dev/moq/pull/6995): -`track_range_filling` holds flat over 1 to 256 ranges, any span, and up to 64 -readers sharing the fills, and the serving path's window end and coverage -checks stopped walking every range. Four walks still grow with range count. -A lite-07 SUBSCRIBE fits about 32K ranges in its 64 KiB message, so a hostile -peer can multiply each by that. - -- **Time anchors.** While any time end is unsettled, `Window::settle` advances - every span on each cursor sync, and `Window::watch_starts` and - `Window::speculative` walk every span on each park. A peer naming many time - ends past the live edge keeps them unsettled. Timed datagram admission - (`Window::admits_datagram`) walks every range per datagram. Index the - unsettled ends by the sequence each waits on, so a settle advances only the - ends new groups reach, and split datagram admission by anchor kind into - sorted lookups. -- **Outstanding fills.** While fills are out, each probe runs - `Misses::expire` over every wanted run and `Misses::resolve` collects the - joins over every wanted run, so a reader whose fills never answer pays - O(wanted runs) per live group. Expire by deadline as `Deferred` does, and - resolve only the joins whose outcome changed. -- **Sparse live numbering.** `TrackState::find_servable` leapfrogs runs and - live groups, so on a track with sparse live sequences (moq-transport), a - reader with ranges in the gaps costs O(min(ranges, gaps)) per search. -- **Lite-07 frame bounds.** The lite-07 publisher's `Frames::start` and - `Frames::end` (`rs/moq-net/src/lite/publisher.rs`) scan every decoded - range per served group and per datagram. Key them by group. This is the - 2026-10-27 security audit's NET-1, folded in here (maintainer, - 2026-10-27). - -Public API: none expected. Wire: none expected. diff --git a/rs/moq-net/Cargo.toml b/rs/moq-net/Cargo.toml index 3aded811901..2a8f7a49874 100644 --- a/rs/moq-net/Cargo.toml +++ b/rs/moq-net/Cargo.toml @@ -18,9 +18,9 @@ default = ["async"] async = ["dep:futures", "kio/async", "kio/runtime"] # Exposes the wire codecs to the `fuzz/` harness and the announce, codec, path, varint, frame, and canonical storage benches, -# session views of an origin to the viewers bench, and the lite publisher's run loop to the -# publisher bench, through the hidden `fuzz` module. Off by -# default and hidden from the docs. +# session views of an origin to the viewers bench, the lite publisher's run loop to the +# publisher bench, and a lite-07 publisher's frame bounds to the ranges bench, through the +# hidden `fuzz` module. Off by default and hidden from the docs. fuzz = ["async"] [dependencies] @@ -124,6 +124,12 @@ name = "frame" harness = false required-features = ["fuzz"] +# Reaches a lite-07 publisher's frame bounds through the hidden `fuzz` module. +[[bench]] +name = "ranges" +harness = false +required-features = ["fuzz"] + # Receives frames through the private session reader via the hidden `fuzz` module. [[bench]] name = "backing" diff --git a/rs/moq-net/benches/ranges.rs b/rs/moq-net/benches/ranges.rs new file mode 100644 index 00000000000..5d0bf6346c8 --- /dev/null +++ b/rs/moq-net/benches/ranges.rs @@ -0,0 +1,414 @@ +//! Per-poll cost over the number of ranges in one subscription. +//! +//! A lite-07 SUBSCRIBE fits about 32K ranges in its 64 KiB message, so a walk over every +//! range on each poll lets a peer multiply what one subscription costs by that much. Each +//! group here holds the ranges fixed and repeats one steady-state step, swept over the range +//! count, so such a walk shows up as a slope over it: +//! +//! - `ranges_time_end`: ranges whose time ends lie past the live edge, so none settles. A +//! group lands, starting before every end, and the reader drains it. +//! - `ranges_time_cap`: one time end past the live edge beside fixed ranges far ahead of +//! it. A group lands yet to start, which caps the end, then starts inside it, which lifts +//! the cap, so the window moves twice per group. +//! - `ranges_fills_out`: ranges below where the feed starts, asked of a handler that never +//! answers, beside one live range. A live group lands and the reader drains it while every +//! fill is still out. +//! - `ranges_sparse`: a feed numbered sparsely, as a moq-transport publisher may, with the +//! reader's ranges in the gaps between its groups. A group lands past them, and the reader +//! polls with nothing in range. +//! - `ranges_time_gaps`: time ranges each ending between two groups of a sparsely numbered +//! history, so each end waits on the missing sequences of a gap of its own, beside one time +//! end past the live edge whose cap moves twice per group, as in `ranges_time_cap`. +//! - `ranges_datagram`: timed ranges none of which admits the datagram that lands, so the +//! reader judges it against all of them. +//! - `ranges_frames`: a lite-07 publisher's frame bounds, each range starting and ending +//! partway through its own group, looked up for one group as a served group or datagram +//! does. +//! +//! Run with `cargo bench -p moq-net --features fuzz --bench ranges`. + +use std::hint::black_box; +use std::task::Poll; +use std::time::Duration; + +use bytes::Bytes; +use criterion::{BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; +use moq_net::{Timestamp, broadcast, group, track}; + +/// Ranges in the one subscription, up to about what a lite-07 SUBSCRIBE fits. +const RANGES: [u64; 4] = [1, 64, 1024, 16_384]; + +/// A budget the run never outlasts, so nothing goes stale. +const BUDGET: Duration = Duration::from_secs(3600); + +/// Where the live feed starts, in milliseconds, past every time a range names. +const LIVE: u64 = 1 << 32; + +fn timed() -> track::Info { + track::Info::default().with_timescale(moq_net::Timescale::MILLI) +} + +fn millis(millis: u64) -> Timestamp { + Timestamp::from_millis(millis).unwrap() +} + +/// Write a one-frame group at `sequence`, presented at `at` milliseconds. +fn land(track: &track::Producer, sequence: u64, at: u64) { + let mut group = track.create_group(group::Info { sequence }).unwrap(); + group.write_frame(millis(at), Bytes::from_static(b"x")).unwrap(); + group.finish().unwrap(); +} + +/// Hand out everything the reader has now, returning how many groups. +fn drain(reader: &mut track::Subscriber) -> usize { + let waiter = kio::Waiter::noop(); + let mut handed = 0; + while let Poll::Ready(Ok(Some(group))) = reader.poll_recv_group(&waiter) { + black_box(group); + handed += 1; + } + handed +} + +/// One reader whose time ends lie past the live edge; see `ranges_time_end`. +pub(crate) struct TimeEnds { + _broadcast: broadcast::Producer, + track: track::Producer, + reader: track::Subscriber, + next: u64, +} + +impl TimeEnds { + pub(crate) fn new(ranges: u64) -> Self { + let broadcast = broadcast::Info::default().produce(); + let track = broadcast.create_track("bench", timed()).unwrap(); + land(&track, 0, 0); + // Each range starts and ends far past the live edge, at a time of its own. + let subscription = track::Subscription::default() + .with_max_delay(BUDGET) + .with_ranges((0..ranges).map(|i| track::Range { + start: track::Anchor::Time(millis(LIVE + i * 2)), + end: Some(track::Anchor::Time(millis(LIVE + i * 2 + 1))), + })); + let mut reader = track.subscribe(subscription); + drain(&mut reader); + Self { + _broadcast: broadcast, + track, + reader, + next: 1, + } + } + + pub(crate) fn step(&mut self) { + land(&self.track, self.next, self.next); + self.next += 1; + assert_eq!(drain(&mut self.reader), 1, "every end is past the group"); + } +} + +/// One reader whose time end moves its cap beside fixed ranges; see `ranges_time_cap`. +pub(crate) struct TimeCap { + _broadcast: broadcast::Producer, + track: track::Producer, + reader: track::Subscriber, + next: u64, +} + +impl TimeCap { + pub(crate) fn new(ranges: u64) -> Self { + let broadcast = broadcast::Info::default().produce(); + let track = broadcast.create_track("bench", timed()).unwrap(); + land(&track, 0, 0); + // The time end, then disjoint groups far past anything the run lands. + let subscription = track::Subscription::default().with_max_delay(BUDGET).with_ranges( + std::iter::once(track::Range { + start: track::Anchor::Time(millis(0)), + end: Some(track::Anchor::Time(millis(LIVE))), + }) + .chain((1..ranges).map(|i| track::Range::groups(LIVE + i * 2..LIVE + i * 2 + 1))), + ); + let mut reader = track.subscribe(subscription); + drain(&mut reader); + Self { + _broadcast: broadcast, + track, + reader, + next: 1, + } + } + + pub(crate) fn step(&mut self) { + let mut group = self.track.create_group(group::Info { sequence: self.next }).unwrap(); + let mut handed = drain(&mut self.reader); + group.write_frame(millis(self.next), Bytes::from_static(b"x")).unwrap(); + group.finish().unwrap(); + handed += drain(&mut self.reader); + self.next += 1; + assert_eq!(handed, 1, "the group is in range"); + } +} + +/// One reader waiting on fills that never answer; see `ranges_fills_out`. +pub(crate) struct FillsOut { + _broadcast: broadcast::Producer, + track: track::Producer, + _requests: track::Requests, + reader: track::Subscriber, + next: u64, +} + +impl FillsOut { + pub(crate) fn new(ranges: u64) -> Self { + let broadcast = broadcast::Info::default().produce(); + let mut track = broadcast.create_track("bench", timed()).unwrap(); + // Held and never polled, so every fill stays out. + let requests = track.requests(); + // The feed starts past every range, with a sequence between each that no range asks + // for, so each range is a fill of its own. + let start = ranges * 2; + track.start_at(start).unwrap(); + land(&track, start, start); + let subscription = track::Subscription::default().with_max_delay(BUDGET).with_ranges( + (0..ranges) + .map(|i| track::Range::groups(i * 2..i * 2 + 1)) + .chain([track::Range::groups(start..)]), + ); + let mut reader = track.subscribe(subscription); + drain(&mut reader); + Self { + _broadcast: broadcast, + track, + _requests: requests, + reader, + next: start + 1, + } + } + + pub(crate) fn step(&mut self) { + land(&self.track, self.next, self.next); + self.next += 1; + black_box(drain(&mut self.reader)); + } +} + +/// One reader whose ranges sit in the gaps of a sparsely numbered feed; see `ranges_sparse`. +pub(crate) struct Sparse { + _broadcast: broadcast::Producer, + track: track::Producer, + reader: track::Subscriber, + next: u64, +} + +/// The step between a sparse feed's sequences. +const STRIDE: u64 = 4; + +impl Sparse { + pub(crate) fn new(ranges: u64) -> Self { + let broadcast = broadcast::Info::default().produce(); + let track = broadcast.create_track("bench", timed()).unwrap(); + // The feed's groups at every multiple of the stride, and the reader's ranges one past + // each, which no group fills, above group 0, the one group in range. + for i in 0..=ranges { + land(&track, i * STRIDE, i); + } + let subscription = track::Subscription::default().with_max_delay(BUDGET).with_ranges( + std::iter::once(track::Range::groups(0..1)) + .chain((1..=ranges).map(|i| track::Range::groups(i * STRIDE + 1..i * STRIDE + 2))), + ); + let mut reader = track.subscribe(subscription); + assert_eq!(drain(&mut reader), 1, "group 0 is the one in range"); + Self { + _broadcast: broadcast, + track, + reader, + next: ranges + 1, + } + } + + pub(crate) fn step(&mut self) { + land(&self.track, self.next * STRIDE, self.next); + self.next += 1; + assert_eq!(drain(&mut self.reader), 0, "nothing lands in range"); + } +} + +/// One reader whose time ends wait in the gaps of a sparsely numbered feed; see +/// `ranges_time_gaps`. +pub(crate) struct TimeGaps { + _broadcast: broadcast::Producer, + track: track::Producer, + reader: track::Subscriber, + /// Where the live feed went on from the history. + live: u64, + next: u64, +} + +impl TimeGaps { + pub(crate) fn new(ranges: u64) -> Self { + let broadcast = broadcast::Info::default().produce(); + let track = broadcast.create_track("bench", timed()).unwrap(); + // The history's groups at every multiple of the stride, ten milliseconds apart, and + // each range from just after one starts until halfway to the next: it ends before the + // next group, but the sequences between may still arrive inside it. The live feed goes + // on from the last, one sequence and millisecond at a time, read by a range from there. + for i in 0..=ranges { + land(&track, i * STRIDE, i * 10); + } + let live = track::Range { + start: track::Anchor::Time(millis(ranges * 10)), + end: Some(track::Anchor::Time(millis(LIVE))), + }; + let subscription = track::Subscription::default().with_max_delay(BUDGET).with_ranges( + (0..ranges) + .map(|i| track::Range { + start: track::Anchor::Time(millis(i * 10 + 1)), + end: Some(track::Anchor::Time(millis(i * 10 + 5))), + }) + .chain([live]), + ); + let mut reader = track.subscribe(subscription); + let groups = usize::try_from(ranges).unwrap() + 1; + assert_eq!(drain(&mut reader), groups, "each range's own group, and the live edge"); + Self { + _broadcast: broadcast, + track, + reader, + live: ranges * STRIDE, + next: ranges * STRIDE + 1, + } + } + + pub(crate) fn step(&mut self) { + // Each a millisecond after the last, from where the history ended. + let at = self.next - self.live + self.live / STRIDE * 10; + let mut group = self.track.create_group(group::Info { sequence: self.next }).unwrap(); + let mut handed = drain(&mut self.reader); + group.write_frame(millis(at), Bytes::from_static(b"x")).unwrap(); + group.finish().unwrap(); + handed += drain(&mut self.reader); + self.next += 1; + assert_eq!(handed, 1, "the live group is in range"); + } +} + +/// One reader judging datagrams no range admits; see `ranges_datagram`. +pub(crate) struct Datagrams { + _broadcast: broadcast::Producer, + track: track::Producer, + reader: track::Subscriber, + at: u64, +} + +impl Datagrams { + pub(crate) fn new(ranges: u64) -> Self { + let broadcast = broadcast::Info::default().produce(); + let mut track = broadcast.create_track("bench", timed()).unwrap(); + // Each range presents from a time of its own through group 0, which every datagram + // after the first passes. + let subscription = track::Subscription::default() + .with_max_delay(BUDGET) + .with_ranges((0..ranges).map(|i| track::Range { + start: track::Anchor::Time(millis(i)), + end: Some(track::Anchor::Absolute(1)), + })); + let mut reader = track.subscribe(subscription); + // Group 0 is in every range, so only the first datagram is admitted. + track.append_datagram(millis(LIVE), Bytes::from_static(b"x")).unwrap(); + let waiter = kio::Waiter::noop(); + assert!(matches!(reader.poll_recv_datagram(&waiter), Poll::Ready(Ok(Some(_))))); + Self { + _broadcast: broadcast, + track, + reader, + at: LIVE + 1, + } + } + + pub(crate) fn step(&mut self) { + self.track + .append_datagram(millis(self.at), Bytes::from_static(b"x")) + .unwrap(); + self.at += 1; + let waiter = kio::Waiter::noop(); + assert!( + self.reader.poll_recv_datagram(&waiter).is_pending(), + "no range admits the datagram" + ); + } +} + +/// A lite-07 publisher's frame bounds; see `ranges_frames`. +pub(crate) struct Frames { + bounds: moq_net::fuzz::FrameBounds, + ranges: u64, + next: u64, +} + +impl Frames { + pub(crate) fn new(ranges: u64) -> Self { + Self { + bounds: moq_net::fuzz::FrameBounds::new(ranges), + ranges, + next: 0, + } + } + + pub(crate) fn step(&mut self) { + // Each group in turn, a range's or the one between, so a lookup can't stop early. + black_box(self.bounds.serve(self.next % (self.ranges * 2))); + self.next += 1; + } +} + +fn sweep(c: &mut Criterion, name: &str, new: impl Fn(u64) -> S, step: impl Fn(&mut S)) { + let mut group = c.benchmark_group(name); + group.throughput(Throughput::Elements(1)); + for ranges in RANGES { + group.bench_function(BenchmarkId::from_parameter(ranges), |b| { + let mut scenario = new(ranges); + b.iter(|| step(&mut scenario)); + }); + } + group.finish(); +} + +fn bench_time_end(c: &mut Criterion) { + sweep(c, "ranges_time_end", TimeEnds::new, TimeEnds::step); +} + +fn bench_time_cap(c: &mut Criterion) { + sweep(c, "ranges_time_cap", TimeCap::new, TimeCap::step); +} + +fn bench_fills_out(c: &mut Criterion) { + sweep(c, "ranges_fills_out", FillsOut::new, FillsOut::step); +} + +fn bench_sparse(c: &mut Criterion) { + sweep(c, "ranges_sparse", Sparse::new, Sparse::step); +} + +fn bench_time_gaps(c: &mut Criterion) { + sweep(c, "ranges_time_gaps", TimeGaps::new, TimeGaps::step); +} + +fn bench_datagram(c: &mut Criterion) { + sweep(c, "ranges_datagram", Datagrams::new, Datagrams::step); +} + +fn bench_frames(c: &mut Criterion) { + sweep(c, "ranges_frames", Frames::new, Frames::step); +} + +criterion_group!( + benches, + bench_time_end, + bench_time_cap, + bench_fills_out, + bench_sparse, + bench_time_gaps, + bench_datagram, + bench_frames +); +criterion_main!(benches); diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index d93f7f8a16a..165f7f444fd 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -94,6 +94,22 @@ impl CanonicalStorage { } } +/// A lite-07 subscription's frame bounds over many ranges, for the range-count bench. +pub struct FrameBounds(crate::lite::FramesBench); + +impl FrameBounds { + /// `ranges` ranges, each starting and ending partway through its own group. + pub fn new(ranges: u64) -> Self { + Self(crate::lite::FramesBench::new(ranges)) + } + + /// The first and last frame served of `group`, as a served group or datagram looks + /// them up. + pub fn serve(&self, group: u64) -> (u64, Option) { + self.0.serve(group) + } +} + /// Every target, keyed by the name of its `fuzz_targets/.rs` shim. pub const TARGETS: &[(&str, Target)] = &[ ("lite", lite_wire), diff --git a/rs/moq-net/src/lite/mod.rs b/rs/moq-net/src/lite/mod.rs index 97cd9aae167..fbfc5a50db0 100644 --- a/rs/moq-net/src/lite/mod.rs +++ b/rs/moq-net/src/lite/mod.rs @@ -70,6 +70,9 @@ use subscriber::*; // The fuzz harness drives a group stream's ingest directly. #[cfg(all(feature = "async", any(test, feature = "fuzz")))] pub(crate) use subscriber::FrameIngest; +// The range-count bench looks up a subscription's frame bounds directly. +#[cfg(all(feature = "async", any(test, feature = "fuzz")))] +pub(crate) use publisher::FramesBench; pub use track::*; pub use tracks::*; pub use version::Version; diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index a9b081b6d16..552c338c7c3 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -7,7 +7,7 @@ use crate::{SessionError, announce, frame, group, origin, track}; use rangemap::{RangeMap, RangeSet}; use std::{ borrow::Cow, - collections::HashMap, + collections::{BTreeMap, HashMap}, ops::{Bound, ControlFlow, Range}, sync::{ Arc, Mutex, @@ -3676,32 +3676,66 @@ fn position_group(group: group::Consumer, frames: &Frames) -> Option)>, + /// What the ranges starting or ending partway through each group serve of it, keyed by + /// group so a lookup costs the same however many ranges there are. + parts: BTreeMap, +} + +/// What the ranges naming one group partway serve of it, merged. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct Part { + /// The lowest first frame any of them serves. + first: u64, + /// The highest last frame, inclusive, or `None` once one runs to the group's end. + last: Option, } impl Frames { + /// Add a range serving `group` from frame `first` through `last`, inclusive, or to the + /// end of the group for `None`. + fn add(&mut self, group: u64, first: u64, last: Option) { + self.parts + .entry(group) + .and_modify(|part| { + part.first = part.first.min(first); + part.last = part.last.zip(last).map(|(a, b)| a.max(b)); + }) + .or_insert(Part { first, last }); + } + /// The first frame to serve of `group`: 0 unless every range naming it starts partway. fn start(&self, group: u64) -> u64 { - self.parts - .iter() - .filter(|(part, ..)| *part == group) - .map(|(_, first, _)| *first) - .min() - .unwrap_or(0) + self.parts.get(&group).map_or(0, |part| part.first) } /// The last frame to serve of `group`, inclusive, or `None` for all of it: a range /// naming it with no end bound there runs to its end. fn end(&self, group: u64) -> Option { - let mut end = None; - for (_, _, last) in self.parts.iter().filter(|(part, ..)| *part == group) { - let last = (*last)?; - end = Some(end.map_or(last, |end: u64| end.max(last))); - } - end + self.parts.get(&group)?.last + } +} + +/// The frame bounds of `ranges` ranges, each starting and ending partway through its own +/// group, for the range-count bench. +#[cfg(all(feature = "async", any(test, feature = "fuzz")))] +pub(crate) struct FramesBench(Frames); + +#[cfg(all(feature = "async", any(test, feature = "fuzz")))] +impl FramesBench { + pub(crate) fn new(ranges: u64) -> Self { + let at = |group, frame| Some(lite::Anchor::Absolute(track::Position { group, frame })); + let ranges: Vec<_> = (0..ranges) + .map(|i| lite::Range { + start: at(i * 2, 1), + end: at(i * 2, 3), + }) + .collect(); + Self(Bounds::new(&ranges, None).expect("group anchors").frames) + } + + /// The frames served of `group`, as a served group or datagram looks them up. + pub(crate) fn serve(&self, group: u64) -> (u64, Option) { + (self.0.start(group), self.0.end(group)) } } @@ -3772,14 +3806,14 @@ impl Bounds { }; match (first, last) { (Some((group, first)), Some((end, last))) if group == end => { - bounds.frames.parts.push((group, first, Some(last))); + bounds.frames.add(group, first, Some(last)); } _ => { if let Some((group, first)) = first { - bounds.frames.parts.push((group, first, None)); + bounds.frames.add(group, first, None); } if let Some((group, last)) = last { - bounds.frames.parts.push((group, 0, Some(last))); + bounds.frames.add(group, 0, Some(last)); } } } @@ -4009,6 +4043,9 @@ struct TrackRun { steer_tx: kio::Producer, // The frame bounds of the requested ranges, which the model leaves to `position_group`. frames: Frames, + // The lowest group the requested ranges name, `None` unless each names its start: kept + // with the ranges, so a wake reads it without copying every range. + named: Option, // A SUBSCRIBE_UPDATE this track cannot serve, such as a time anchor on a track the // peer was told is untimed, which ends the subscription. refused: Option, @@ -4058,7 +4095,8 @@ impl TrackRun { ) -> Self { // Apply the initial cap from the original Subscribe. Subsequent updates // flow through the SUBSCRIBE_UPDATE arm below. - let cap = bounds.groups().end().map_or(Bound::Unbounded, Bound::Excluded); + let groups = bounds.groups(); + let cap = groups.end().map_or(Bound::Unbounded, Bound::Excluded); // The FIN of a covered subscription is the wire's to send, below. let track = track.with_range((Bound::Included(floor), cap)).parked(); @@ -4071,6 +4109,7 @@ impl TrackRun { Self { bounded, frames: bounds.frames, + named: groups.start(), refused: None, ctx, track, @@ -4132,7 +4171,7 @@ impl TrackRun { // Feed the full update into the model subscriber so the producer's // aggregate reflects it (and a relay re-forwards it upstream). // Read first: `update` replaces these preferences. - let requested = self.track.subscription().start(); + let requested = self.named; let floored = requested.is_some_and(|group| group > 0); let bounds = match Bounds::new(&upd.ranges, self.ctx.stamp.map(|stamp| stamp.timescale)) { Ok(bounds) => bounds, @@ -4143,12 +4182,18 @@ impl TrackRun { }; let groups = bounds.groups(); let (start, end) = (groups.start(), groups.end()); - let _ = self.track.update(crate::track::Subscription { - priority: upd.priority, - max_delay: serving_max_delay(self.ctx.version, upd.max_delay), - order: upd.order, - ..groups - }); + if self + .track + .update(crate::track::Subscription { + priority: upd.priority, + max_delay: serving_max_delay(self.ctx.version, upd.max_delay), + order: upd.order, + ..groups + }) + .is_ok() + { + self.named = start; + } // An explicit start moves the read cursor. Lite-06+ encodes an absent // start as no floor, so a subscription that had one above group 0 has to // drop back to 0: the cursor only rises on its own, and a finished group @@ -4313,7 +4358,7 @@ impl TrackRun { if self.emit_range && self.start.is_none() && self.ctx.version.has_largest() - && let Some(start) = self.track.subscription().start() + && let Some(start) = self.named && let Poll::Ready(Some(largest)) = self.track.poll_live(waiter) && (track::Position { group: start, @@ -4609,7 +4654,7 @@ impl TrackRun { // floor. Lite-06 encodes a floor of group 0 as 0, which decodes as no named floor; // that 0 is group 0, so it is not pinned. Raised, not assigned: an update that // landed while the group was held may already have raised it past. - if self.track.subscription().start().is_none() && !self.ctx.version.resolves_start() { + if self.named.is_none() && !self.ctx.version.resolves_start() { self.raise_floor(start); } self.serve(group); @@ -5260,20 +5305,25 @@ mod serve_group_test { /// Ranges naming the same group serve the span covering all of their frames in it. #[test] fn frames_widen_across_ranges() { - let frames = Frames { - parts: vec![ - (3, 5, None), - (3, 2, None), - (4, 1, Some(6)), - (7, 0, Some(1)), - (7, 0, Some(3)), - ], - }; + let mut frames = Frames::default(); + for (group, first, last) in [ + (3, 5, None), + (3, 2, None), + (4, 1, Some(6)), + (7, 0, Some(1)), + (7, 0, Some(3)), + (8, 0, Some(2)), + (8, 1, None), + ] { + frames.add(group, first, last); + } assert_eq!((frames.start(3), frames.start(4), frames.start(5)), (2, 1, 0)); assert_eq!( (frames.end(3), frames.end(4), frames.end(7), frames.end(6)), (None, Some(6), Some(3), None) ); + // One range running to the end of a group serves all of it past the others' ends. + assert_eq!((frames.start(8), frames.end(8)), (0, None)); // Disjoint ranges meeting partway through group 5 serve all of it between them. let group = |group, frame| Some(lite::Anchor::Absolute(track::Position { group, frame })); @@ -5314,8 +5364,10 @@ mod serve_group_test { let mut group = track.create_group(group::Info { sequence: 3 }).unwrap(); group.start_at(5).unwrap(); group.write_frame(Timestamp::ZERO, b"tail".to_vec()).unwrap(); - let starting = |frame| Frames { - parts: vec![(3, frame, None)], + let starting = |frame| { + let mut frames = Frames::default(); + frames.add(3, frame, None); + frames }; // Asked for whole groups, so the missing frames 0..5 make this unservable. @@ -5348,8 +5400,10 @@ mod serve_group_test { group.write_frame(Timestamp::ZERO, vec![i]).unwrap(); } group.finish().unwrap(); - let ending = |group, frame| Frames { - parts: vec![(group, 0, Some(frame))], + let ending = |group, frame| { + let mut frames = Frames::default(); + frames.add(group, 0, Some(frame)); + frames }; let mut consumer = position_group(group.consume(), &ending(7, 1)).unwrap(); diff --git a/rs/moq-net/src/model/subscription.rs b/rs/moq-net/src/model/subscription.rs index 08bf5ae5d66..f6025c13fd3 100644 --- a/rs/moq-net/src/model/subscription.rs +++ b/rs/moq-net/src/model/subscription.rs @@ -5,9 +5,11 @@ use std::{ use crate::Timestamp; +mod runs; mod window; +pub(super) use runs::{Layered, Runs}; pub(crate) use window::Accounted; -pub(super) use window::{Containing, Held, Holdings, Resolved, Unheld, Window, later}; +pub(super) use window::{Containing, Held, Holdings, Resolved, Seen, Unheld, Window, later}; /// Subscriber-side preferences for receiving a track. /// diff --git a/rs/moq-net/src/model/subscription/runs.rs b/rs/moq-net/src/model/subscription/runs.rs new file mode 100644 index 00000000000..cbce7e8f218 --- /dev/null +++ b/rs/moq-net/src/model/subscription/runs.rs @@ -0,0 +1,381 @@ +//! Sets of group sequences held as sorted runs, and the layered set a [`Window`](super::Window) +//! reads: fixed runs shared across its moves, with the few runs its time ends walk on top. +//! +//! A subscription may name thousands of ranges, yet an unsettled time end moves its cap on +//! each group. Rebuilding or copying every run on each move would let a peer multiply what +//! one group costs by its range count, so the runs that never move are kept apart, shared +//! by reference, and only the layer above them is rebuilt. + +use std::{ops::RangeInclusive, sync::Arc}; + +use rangemap::RangeInclusiveSet; + +/// Disjoint runs of groups, lowest first and never adjacent, each found from either end by +/// one search. +/// +/// Not a [`RangeInclusiveSet`]: its overlapping iterator walks back from the last run, so +/// finding the highest run below a point costs every run above it. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub(in crate::model) struct Runs(Vec>); + +impl Runs { + /// The runs of `runs`, in any order, overlapping or not. + pub fn new(runs: impl IntoIterator>) -> Self { + let set: RangeInclusiveSet = runs.into_iter().filter(|run| !run.is_empty()).collect(); + Self(set.into_iter().collect()) + } + + pub fn iter(&self) -> std::slice::Iter<'_, RangeInclusive> { + self.0.iter() + } + + pub fn contains(&self, sequence: u64) -> bool { + let index = self.0.partition_point(|run| *run.end() < sequence); + self.0.get(index).is_some_and(|run| *run.start() <= sequence) + } + + /// The lowest run reaching into `lo..=hi`. + pub fn first(&self, lo: u64, hi: u64) -> Option<&RangeInclusive> { + let index = self.0.partition_point(|run| *run.end() < lo); + self.0.get(index).filter(|run| *run.start() <= hi) + } + + /// The highest run reaching into `lo..=hi`. + pub fn last(&self, lo: u64, hi: u64) -> Option<&RangeInclusive> { + let index = self.0.partition_point(|run| *run.start() <= hi); + self.0.get(index.checked_sub(1)?).filter(|run| *run.end() >= lo) + } + + /// The runs reaching into `lo..=hi`, lowest first, clipped to it. + pub fn within(&self, lo: u64, hi: u64) -> impl DoubleEndedIterator> + '_ { + let from = self.0.partition_point(|run| *run.end() < lo); + let to = self.0.partition_point(|run| *run.start() <= hi).max(from); + self.0[from..to] + .iter() + .map(move |run| (*run.start()).max(lo)..=(*run.end()).min(hi)) + } + + /// Add `run`, merging the runs it overlaps or touches. Costs a search and a shift. + pub fn insert(&mut self, run: RangeInclusive) { + if run.is_empty() { + return; + } + let (mut start, mut end) = run.into_inner(); + // The first run reaching `start - 1` or later, and the first starting past `end + 1`. + let from = self.0.partition_point(|run| run.end().saturating_add(1) < start); + let to = self + .0 + .partition_point(|run| *run.start() <= end.saturating_add(1)) + .max(from); + if let (Some(first), Some(last)) = (self.0[from..to].first(), self.0[from..to].last()) { + start = start.min(*first.start()); + end = end.max(*last.end()); + } + self.0.splice(from..to, std::iter::once(start..=end)); + } +} + +/// A set of groups held as shared fixed runs with a layer of runs over them, which may +/// overlap them or each other. +/// +/// Each query answers from both by searches, never by visiting the fixed runs a layered run +/// covers, so a layer that moves costs its own runs, however many fixed ones lie under it. +/// Both are shared, so a copy costs nothing until the layer moves. +#[derive(Clone, Debug, Default)] +pub(in crate::model) struct Layered { + fixed: Arc, + layer: Arc, +} + +impl PartialEq for Layered { + /// Equal as held: the same fixed runs and layer, each compared by reference first. The + /// same groups split differently between the two compare unequal. + fn eq(&self, other: &Self) -> bool { + let same = |a: &Arc, b: &Arc| Arc::ptr_eq(a, b) || a == b; + same(&self.fixed, &other.fixed) && same(&self.layer, &other.layer) + } +} + +impl Layered { + pub fn new(fixed: Arc, layer: Runs) -> Self { + Self { + fixed, + layer: Arc::new(layer), + } + } + + pub fn fixed(&self) -> &Arc { + &self.fixed + } + + pub fn layer(&self) -> &Runs { + &self.layer + } + + /// Hold `fixed` under the layer from now on. The runs already held stay shared when they + /// are the same, so what shares them still differs from this only in the layer. + pub fn set_fixed(&mut self, fixed: Runs) { + if *self.fixed != fixed { + self.fixed = Arc::new(fixed); + } + } + + pub fn set_layer(&mut self, layer: Runs) { + self.layer = Arc::new(layer); + } + + /// Whether both hold the same fixed runs, so they differ only in their layers. + pub fn shares_fixed(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.fixed, &other.fixed) + } + + pub fn contains(&self, sequence: u64) -> bool { + self.fixed.contains(sequence) || self.layer.contains(sequence) + } + + /// The lowest whole run of the union reaching into `lo..=hi`, its start clipped to `lo`. + pub fn first(&self, lo: u64, hi: u64) -> Option> { + let starts = [self.fixed.first(lo, hi), self.layer.first(lo, hi)]; + let start = starts.into_iter().flatten().map(|run| (*run.start()).max(lo)).min()?; + // Grow the end through whatever either set holds next to it: each step crosses into + // the other set, so a layered run over many fixed ones costs a search, not a walk. + let mut end = start; + loop { + let grown = [&*self.fixed, &*self.layer] + .into_iter() + .filter_map(|runs| runs.first(end, end.saturating_add(1))) + .map(|run| *run.end()) + .max() + .filter(|grown| *grown > end); + match grown { + Some(grown) => end = grown, + None => return Some(start..=end), + } + if end == u64::MAX { + return Some(start..=end); + } + } + } + + /// The highest whole run of the union reaching into `lo..=hi`, its end clipped to `hi`. + pub fn last(&self, lo: u64, hi: u64) -> Option> { + let ends = [self.fixed.last(lo, hi), self.layer.last(lo, hi)]; + let end = ends.into_iter().flatten().map(|run| (*run.end()).min(hi)).max()?; + let mut start = end; + loop { + let grown = [&*self.fixed, &*self.layer] + .into_iter() + .filter_map(|runs| runs.last(start.saturating_sub(1), start)) + .map(|run| *run.start()) + .min() + .filter(|grown| *grown < start); + match grown { + Some(grown) => start = grown, + None => return Some(start..=end), + } + if start == 0 { + return Some(start..=end); + } + } + } + + /// The union's runs from `from` on, lowest first, each whole but clipped to `from`. + pub fn runs(&self, from: u64) -> impl Iterator> + '_ { + let mut next = Some(from); + std::iter::from_fn(move || { + let run = self.first(next?, u64::MAX)?; + next = run.end().checked_add(1); + Some(run) + }) + } + + /// The lowest group held. + pub fn lowest(&self) -> Option { + self.first(0, u64::MAX).map(|run| *run.start()) + } + + /// The highest group held. + pub fn highest(&self) -> Option { + self.last(0, u64::MAX).map(|run| *run.end()) + } + + /// What `self` holds and `old` doesn't within `lo..=hi`, lowest first, when both share + /// their fixed runs: `None` when they don't. + /// + /// Costs the two layers, and the fixed runs inside where they differ: a layered run that + /// grows over many fixed ones, from the live edge to past every group, costs them only + /// within `lo..=hi`. + pub fn gained(&self, old: &Self, lo: u64, hi: u64) -> Option>> { + if !self.shares_fixed(old) { + return None; + } + let mut gained = Vec::new(); + if lo > hi { + return Some(gained); + } + for run in self.layer.within(lo, hi) { + for piece in minus(run, &old.layer) { + gained.extend(minus(piece, &self.fixed)); + } + } + Some(gained) + } +} + +/// The parts of `run` that `runs` doesn't hold, lowest first. +fn minus(run: RangeInclusive, runs: &Runs) -> impl Iterator> + '_ { + let (lo, hi) = (*run.start(), *run.end()); + let mut from = (lo <= hi).then_some(lo); + let mut inside = runs.within(lo, hi).peekable(); + std::iter::from_fn(move || { + loop { + let at = from?; + match inside.peek() { + Some(held) if *held.start() <= at => { + from = held.end().checked_add(1).filter(|next| *next <= hi); + inside.next(); + } + Some(held) => { + let gap = at..=*held.start() - 1; + from = held.end().checked_add(1).filter(|next| *next <= hi); + inside.next(); + return Some(gap); + } + None => { + from = None; + return Some(at..=hi); + } + } + } + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A small deterministic generator, so a failure replays. + fn generator() -> impl FnMut(u64) -> u64 { + let mut x = 0x9e37_79b9_7f4a_7c15_u64; + move |bound| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x % bound + } + } + + fn random_runs(next: &mut impl FnMut(u64) -> u64, count: u64, span: u64) -> Vec> { + (0..next(count + 1)) + .map(|_| { + let start = next(span); + // Now and then a run reaching the last sequence. + let end = match next(8) { + 0 => u64::MAX, + _ => start + next(4), + }; + start..=end + }) + .collect() + } + + /// The points a layered set is probed at: each bound of every run, either side of it. + fn points(runs: &[RangeInclusive]) -> Vec { + let mut points: Vec = runs + .iter() + .flat_map(|run| [*run.start(), *run.end()]) + .flat_map(|point| [point.saturating_sub(1), point, point.saturating_add(1)]) + .chain([0, u64::MAX]) + .collect(); + points.sort_unstable(); + points.dedup(); + points + } + + /// A layered set answers every query as the set of its runs merged would. + #[test] + fn layered_runs_match_their_union() { + let mut next = generator(); + for round in 0..2000 { + let fixed = random_runs(&mut next, 6, 40); + let layer = random_runs(&mut next, 3, 40); + let layered = Layered::new(Arc::new(Runs::new(fixed.clone())), Runs::new(layer.clone())); + let union: RangeInclusiveSet = fixed.iter().chain(&layer).cloned().collect(); + let context = || format!("round {round}: {fixed:?} under {layer:?}"); + assert!(layered.runs(0).eq(union.iter().cloned()), "{}", context()); + let points = points(&union.iter().cloned().collect::>()); + for &lo in &points { + assert_eq!(layered.contains(lo), union.contains(&lo), "{} at {lo}", context()); + let from: Vec<_> = union + .overlapping(lo..=u64::MAX) + .map(|run| (*run.start()).max(lo)..=*run.end()) + .collect(); + assert!(layered.runs(lo).eq(from), "{} from {lo}", context()); + for &hi in points.iter().filter(|hi| **hi >= lo) { + let mut reaching = union.overlapping(lo..=hi); + let first = reaching.next().map(|run| (*run.start()).max(lo)..=*run.end()); + assert_eq!(layered.first(lo, hi), first, "{} first in {lo}..={hi}", context()); + let last = union + .overlapping(lo..=hi) + .next_back() + .map(|run| *run.start()..=(*run.end()).min(hi)); + assert_eq!(layered.last(lo, hi), last, "{} last in {lo}..={hi}", context()); + } + } + } + } + + /// What a moved layer gains is exactly the groups the new set holds and the old one + /// doesn't, within the span asked about. + #[test] + fn a_moved_layer_gains_exactly_the_new_groups() { + let mut next = generator(); + for round in 0..2000 { + let fixed = Arc::new(Runs::new(random_runs(&mut next, 6, 40))); + let old = Layered::new(fixed.clone(), Runs::new(random_runs(&mut next, 3, 40))); + let new = Layered::new(fixed, Runs::new(random_runs(&mut next, 3, 40))); + let (lo, hi) = (next(30), 20 + next(30)); + let gained: RangeInclusiveSet = new.gained(&old, lo, hi).unwrap().into_iter().collect(); + let expected: Vec = (lo..=hi) + .filter(|sequence| new.contains(*sequence) && !old.contains(*sequence)) + .collect(); + let found: Vec = (lo..=hi).filter(|sequence| gained.contains(sequence)).collect(); + assert_eq!(found, expected, "round {round}: {old:?} to {new:?} in {lo}..={hi}"); + } + let unshared = Layered::new(Arc::new(Runs::default()), Runs::default()); + assert!(unshared.gained(&Layered::default(), 0, 9).is_none()); + } + + /// Inserting a run merges what it overlaps or touches, as a set of the runs would. + #[test] + fn inserted_runs_merge_as_a_set() { + let mut next = generator(); + for _ in 0..2000 { + let runs = random_runs(&mut next, 8, 40); + let mut inserted = Runs::default(); + for run in &runs { + inserted.insert(run.clone()); + } + assert_eq!(inserted, Runs::new(runs.clone()), "{runs:?}"); + } + } + + /// A run minus a set is the parts of it the set doesn't hold. + #[test] + fn a_run_minus_runs_is_its_gaps() { + let mut next = generator(); + for _ in 0..2000 { + let runs = Runs::new(random_runs(&mut next, 6, 40)); + let start = next(40); + let run = start..=start + next(10); + let expected: Vec = run.clone().filter(|sequence| !runs.contains(*sequence)).collect(); + let found: Vec = minus(run.clone(), &runs).flatten().collect(); + assert_eq!(found, expected, "{run:?} minus {runs:?}"); + } + let whole = Runs::new(std::iter::once(0..=u64::MAX)); + assert!(minus(0..=u64::MAX, &whole).next().is_none()); + let low = Runs::new(std::iter::once(0..=9)); + let above: Vec<_> = minus(5..=u64::MAX, &low).collect(); + assert_eq!((above.len(), above.first()), (1, Some(&(10..=u64::MAX)))); + } +} diff --git a/rs/moq-net/src/model/subscription/window.rs b/rs/moq-net/src/model/subscription/window.rs index f35ef12e15c..6b7d76727ac 100644 --- a/rs/moq-net/src/model/subscription/window.rs +++ b/rs/moq-net/src/model/subscription/window.rs @@ -4,11 +4,13 @@ //! what arrives, not what one subscriber asked for. A [`Window`] is the per-subscriber //! side: resolved at the hop serving the subscriber, it filters that subscriber's reads. -use std::ops; +use std::{ + collections::{BTreeMap, BTreeSet}, + ops, + sync::Arc, +}; -use rangemap::RangeInclusiveSet; - -use super::{Anchor, Range}; +use super::{Anchor, Layered, Range, Runs}; use crate::{Error, Result, Timestamp}; /// One range resolved to groups: from `start` up to the exclusive `end`, `None` while it @@ -76,6 +78,25 @@ pub(in crate::model) trait Holdings { /// Whether nothing more can arrive or start here, so a time end has nothing to wait /// on and is left uncapped. fn complete(&self) -> bool; + + /// The runs of sequences whose arrivals or drops changed since `seen`, which this then + /// moves on to now, or `None` once anything else a parked walk reads may have changed + /// (see [`Window::parked`]): the declared start, completion, whether it + /// [looks up](Self::looks_up) its history, or a copy the live feed never delivered, which + /// may hold a group or give one a start. `None` by default, which wakes every parked walk. + fn changes(&self, seen: &mut Seen) -> Option>> { + let _ = seen; + None + } +} + +/// How far a [`Window`] has taken in its [`Holdings::changes`], in whatever marks the +/// holdings keep: the revisions of the indexes reporting changed runs, and what they report +/// no runs for. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(in crate::model) struct Seen { + pub runs: [u64; 2], + pub global: Option<[u64; 4]>, } /// Where a presentation time falls among the groups a hop holds. @@ -133,8 +154,13 @@ impl Holdings for Unheld { /// Whether `time` is strictly after `other`, as instants: exact across scales, unlike /// [`Timestamp`]'s `Ord`, which breaks a tie between scales. pub(in crate::model) fn later(time: Timestamp, other: Timestamp) -> bool { - u128::from(time.value()) * u128::from(other.scale().as_u64()) - > u128::from(other.value()) * u128::from(time.scale().as_u64()) + instant(time, other).is_gt() +} + +/// How `time` orders against `other` as instants, exact across scales. +fn instant(time: Timestamp, other: Timestamp) -> std::cmp::Ordering { + (u128::from(time.value()) * u128::from(other.scale().as_u64())) + .cmp(&(u128::from(other.value()) * u128::from(time.scale().as_u64()))) } /// A subscriber's [`Range`]s resolved to the group sequences it reads. @@ -143,10 +169,32 @@ pub(in crate::model) struct Window { /// What these were resolved from, so only replacing the ranges resolves again. ranges: Vec, spans: Vec, - /// The groups the spans cover, an unsettled time end through its cap. - groups: RangeInclusiveSet, - /// The part of `groups` a miss may ask for: see [`Self::missable`]. - missable: RangeInclusiveSet, + /// The spans still looking up their time start (see [`Self::found`]), which read nothing + /// meanwhile: kept apart so a probe finds them without visiting every span. + looking: Vec, + /// The unsettled time ends, grouped by walk. + walks: Vec, + /// The walks waiting out a gap below the group their ends fall before, which move only + /// once a sequence from the gap up to that group arrives or is given up, so a settle + /// wakes only the walks its [`Holdings::changes`] reach. Ends left in different gaps of + /// a sparse feed never merge, so walking each of them on every settle would cost the + /// ranges. Keyed by where each [`Parked`] starts. + parked: BTreeMap, + /// How far the parked walks have taken in the holdings' changes. + seen: Seen, + /// The groups the spans cover, an unsettled time end through its cap: every settled span + /// and parked walk in the fixed runs, shared with what reads them, and one run per walk + /// not parked layered over them, so a walk that moves rebuilds only the layer. + groups: Layered, + /// The part of `groups` a miss may ask for, layered the same way: see [`Self::missable`]. + missable: Layered, + /// The groups resolved for good: the spans with a settled end, which the fixed runs hold + /// with the parked walks' runs. + resolved: Runs, + resolved_missable: Runs, + /// The fixed runs no longer hold the settled spans and parked walks: rebuilt once the + /// settle under way is done, which a settle or a walk parking or waking does. + stale: bool, /// A relative anchor waits on a first group, so nothing is filtered yet. pending: bool, /// Nothing is left to resolve: see [`Self::is_settled`]. Kept so a read checks it once @@ -155,6 +203,21 @@ pub(in crate::model) struct Window { /// The floor the last [`Self::covered_by`] checked from, and where it stopped finding the /// groups accounted for. covered: Option<(u64, u64)>, + /// The lowest start among the spans that name theirs, and whether every span does: see + /// [`Self::named`] and [`Self::start`]. Kept so a probe reads them without a walk. + named: Option, + all_named: bool, + /// The ranges' bounds on a datagram, indexed for [`Self::admits_datagram`]. + admission: Admission, +} + +/// Parked walks whose waits overlap, so a change reaching any of them wakes them all: the +/// parked groups are disjoint, each found by one search. +#[derive(Debug)] +struct Parked { + /// The exclusive end of what they wait on, from the key it is parked under. + end: u64, + walks: Vec, } /// One non-empty range, resolved. @@ -166,24 +229,17 @@ struct Span { end: End, /// Whether the start names a group, rather than reaching back to the oldest one. floored: bool, - /// The lowest group a miss asks for. A time start before every group held asks none - /// below the oldest: which of them it reaches is unknown until they start. - reach: u64, - /// A time start before every group held, looked up from the track's history: the span - /// reads nothing until the lookup answers (see [`Window::found`]). - looking: Option, + /// A time start the groups held can't place, looked up from the track's history: the + /// span reads nothing until the lookup answers. + looking: Option, } -impl Span { - /// The exclusive end of the groups read now, `None` for none: an unsettled time end - /// reads up to its cap. - fn limit(&self) -> Option { - match self.end { - End::Before(end) => Some(end), - End::Open => None, - End::Time(time) => time.limit, - } - } +/// A time start being looked up: its time, and the exclusive end of the run below every +/// group held, which holds the group containing it if the track's history does. +#[derive(Clone, Copy, Debug)] +struct Looking { + pts: Timestamp, + below: u64, } #[derive(Clone, Copy, Debug)] @@ -192,23 +248,78 @@ enum End { Before(u64), /// Past every group. Open, - /// Before the first group starting after a time, once that is known. - Time(TimeEnd), + /// Before the first group starting after this time, once a [`Walk`] settles it. + Time(Timestamp), } -/// A time end not settled yet, and how far the walk to settle it got. +/// Unsettled time ends whose walks agree, so one walk serves them all. +/// +/// Ends that start alike walk the same groups until one starts between their times, so +/// the ends sharing a walk differ only in when they end. A group starting at time `t` +/// splits off the ends before `t`, which walk on, and settle, as a walk of their own. +/// Walks that come to agree merge. Each end settles once, and a settle costs the walks +/// rather than the ends. /// -/// Timestamps never rewind, so the end falls before the lowest group starting past `pts`, -/// once every sequence between it and the newest one starting at or before `pts` is -/// accounted for. Every position below only moves forward, so each group or run of -/// sequences is passed once, whatever a peer leaves missing. +/// Timestamps never rewind, so an end falls before the lowest group starting past it, once +/// every sequence between it and the newest group starting at or before it is accounted +/// for. Every position below only moves forward, so each group or run of sequences is +/// passed once, whatever a peer leaves missing. +#[derive(Debug)] +struct Walk { + /// The ends sharing it, earliest first. + ends: BTreeSet, + trail: Trail, + /// The lowest group any of its spans reads, and the lowest a miss asks for: each span + /// reads on from its start up to the cap, so the walk's groups are one run from here. + low: u64, + low_reach: u64, + /// The cap when the settle under way began, inherited by the walks split off since, so + /// each knows whether the groups it reads moved. + was: Option, +} + +/// An unsettled time end: the span it ends, ordered by when, as instants. #[derive(Clone, Copy, Debug)] -struct TimeEnd { +struct Ending { pts: Timestamp, - /// The newest group known to start at or before `pts`: it and everything below it in - /// the span is in range. + span: usize, +} + +impl Ord for Ending { + fn cmp(&self, other: &Self) -> std::cmp::Ordering { + instant(self.pts, other.pts).then(self.span.cmp(&other.span)) + } +} + +impl PartialOrd for Ending { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} + +impl PartialEq for Ending { + fn eq(&self, other: &Self) -> bool { + self.cmp(other).is_eq() + } +} + +impl Eq for Ending {} + +/// How far a [`Walk`] got: the same for every end sharing it, so walks merge once theirs +/// are equal. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +struct Trail { + /// The lowest group the walk has yet to put in range for good. Each of its spans starts + /// here, or at or below `known`, past which [`Walk::fix`] moves it. + floor: u64, + /// The lowest group a miss asks for in what the walk reads: a time start before every + /// group held asks none below the oldest, since which of them it reaches is unknown + /// until they start. + reach: u64, + /// The newest group known to start at or before every end: it and everything below it + /// in the spans is in range. known: Option, - /// The lowest group known to start after `pts`: where the end falls once settled. + /// The lowest group known to start after every end: where they fall once settled. after: Option, /// Where the walk over held groups resumes: those below it started, or never will. held: u64, @@ -217,14 +328,15 @@ struct TimeEnd { /// The exclusive cap on what is read meanwhile, `None` for none. limit: Option, /// The group the cap admits on speculation: the first above `known` that may still - /// start, which may start past `pts`. + /// start, which may start past the ends. speculative: Option, } -impl TimeEnd { - fn new(pts: Timestamp, start: u64) -> Self { +impl Trail { + fn new(start: u64, reach: u64) -> Self { Self { - pts, + floor: start, + reach, known: None, after: None, held: start, @@ -235,75 +347,118 @@ impl TimeEnd { } /// The lowest sequence not known to be in range, `None` once the last sequence is. - fn unknown(&self, start: u64) -> Option { + /// + /// `known` is only ever found at or above every span's start, so it alone decides once + /// set. + fn unknown(&self) -> Option { match self.known { - Some(known) => known.checked_add(1).map(|next| next.max(start)), - None => Some(start), + Some(known) => known.checked_add(1), + None => Some(self.floor), } } +} - /// Walk the held groups on from where the last call stopped. Returns where the end - /// falls once it settles. - fn advance(&mut self, start: u64, holdings: &impl Holdings) -> Option { +impl Walk { + /// Whether every end is before `time`, so a group starting then lies past each. + /// + /// When only some are, those leave as a walk of their own from here, queued in `forks` + /// to walk on, and this one keeps the rest: `false`. + fn past(&mut self, time: Timestamp, forks: &mut Vec) -> bool { + let (Some(first), Some(last)) = (self.ends.first(), self.ends.last()) else { + return false; + }; + if later(time, last.pts) { + return true; + } + if !later(time, first.pts) { + return false; + } + let rest = self.ends.split_off(&Ending { pts: time, span: 0 }); + forks.push(Walk { + ends: std::mem::replace(&mut self.ends, rest), + trail: self.trail, + low: self.low, + low_reach: self.low_reach, + was: self.was, + }); + false + } + + /// Walk the held groups on from where the last call stopped, splitting off into `forks` + /// the ends a group starts between. Returns where the ends fall once they settle. + /// + /// A split-off walks again from the state it split at: each call resumes where the + /// last stopped, so it takes the other branch there, passing nothing twice. + fn advance(&mut self, holdings: &impl Holdings, forks: &mut Vec) -> Option { // The starts index passes a group that may still start, so a stalled one never - // keeps a later group known to start inside the end from being read. - while let Some(from) = self.unknown(start) + // keeps a later group known to start inside the ends from being read. + while let Some(from) = self.trail.unknown() && let Some((sequence, at)) = holdings.next_start(from) - && self.after.is_none_or(|after| sequence < after) + && self.trail.after.is_none_or(|after| sequence < after) { - match later(at, self.pts) { - true => self.after = Some(sequence), - false => self.known = Some(sequence), + match self.past(at, forks) { + true => self.trail.after = Some(sequence), + false => self.trail.known = Some(sequence), } } // The first sequence above `known` that may still start is the one read on // speculation: a held group yet to start, or a missing one that may arrive. Only a - // group starting inside the end is jumped to, since it puts all below it in range. - self.speculative = None; - while let Some(from) = self.unknown(start) { - let at = self.held.max(from); - if self.after.is_some_and(|after| at >= after) { + // group starting inside the ends is jumped to, since it puts all below it in range. + self.trail.speculative = None; + while let Some(from) = self.trail.unknown() { + let at = self.trail.held.max(from); + if self.trail.after.is_some_and(|after| at >= after) { break; } let next = holdings .next_held(at) - .filter(|(sequence, _)| self.after.is_none_or(|after| *sequence < after)); + .filter(|(sequence, _)| self.trail.after.is_none_or(|after| *sequence < after)); + // A later group starting inside some ends but not others splits them first: those + // it starts past account for `at` on their own. + let jump = match next { + Some((sequence, Held::Started(time))) if sequence != at => { + (!self.past(time, forks)).then_some(sequence) + } + _ => None, + }; match next { Some((sequence, held)) if sequence == at => match held { - Held::Started(time) if later(time, self.pts) => self.after = Some(at), - Held::Started(_) => self.known = Some(at), + Held::Started(time) => match self.past(time, forks) { + true => self.trail.after = Some(at), + false => self.trail.known = Some(at), + }, Held::Never => {} Held::Pending => { - self.held = at; - self.speculative = Some(at); + self.trail.held = at; + self.trail.speculative = Some(at); break; } }, - Some((sequence, Held::Started(time))) if !later(time, self.pts) => self.known = Some(sequence), + Some(_) if jump.is_some() => self.trail.known = jump, // Past every held group: the next to arrive is the speculative one, judged // once a settle sees it, before it is read. - None if self.after.is_none() => { - self.held = at; + None if self.trail.after.is_none() => { + self.trail.held = at; break; } _ => match holdings.gone(at) { Some(end) if end > at => { - self.held = end; + self.trail.held = end; continue; } _ => { - self.held = at; - self.speculative = (!holdings.complete()).then_some(at); + self.trail.held = at; + self.trail.speculative = (!holdings.complete()).then_some(at); break; } }, } // Nothing sits above the last sequence, so the walk ends on it. let Some(next) = at.checked_add(1) else { break }; - self.held = next; + self.trail.held = next; } - self.limit = match (self.speculative, self.after) { + self.trail.limit = match (self.trail.speculative, self.trail.after) { _ if holdings.complete() => None, (Some(speculative), _) => speculative.checked_add(1), (None, Some(after)) => Some(after), @@ -311,39 +466,100 @@ impl TimeEnd { (None, None) => None, }; - // Settle once every sequence up to the first group starting past `pts` can no - // longer start inside it. + // Settle once every sequence up to the first group starting past the ends can no + // longer start inside them. // // Datagrams stay ephemeral and never index where their group starts, so a sequence // that arrived only as a datagram counts as gone here. A stream copy of that group - // arriving later is read even if it starts past `pts`: accepted, since a publisher - // that streams a group it also sent as a datagram is rare. - let after = self.after?; + // arriving later is read even if it starts past the ends: accepted, since a + // publisher that streams a group it also sent as a datagram is rare. + let after = self.trail.after?; // Everything at or below `known` is in range, and `after` lies above it. - self.accounted = self.accounted.max(self.unknown(start).unwrap_or(after)); - while self.accounted < after { - let next = holdings.next_held(self.accounted); + self.trail.accounted = self.trail.accounted.max(self.trail.unknown().unwrap_or(after)); + while self.trail.accounted < after { + let next = holdings.next_held(self.trail.accounted); match next { - Some((sequence, held)) if sequence == self.accounted => match held { + Some((sequence, held)) if sequence == self.trail.accounted => match held { Held::Pending => return None, - // It started past `pts` since the walk above stopped on it: everything - // below is accounted for, so the end falls here. - Held::Started(at) if later(at, self.pts) => { - self.after = Some(sequence); - return self.after; + // It started past the ends since the walk above stopped on it: everything + // below is accounted for, so they fall here. + Held::Started(at) if self.past(at, forks) => { + self.trail.after = Some(sequence); + return self.trail.after; } - _ => self.accounted += 1, + _ => self.trail.accounted += 1, }, _ => { let below = next.map_or(after, |(sequence, _)| sequence.min(after)); - self.accounted = match holdings.complete() { + self.trail.accounted = match holdings.complete() { true => below, - false => holdings.gone(self.accounted)?.min(below), + false => holdings.gone(self.trail.accounted)?.min(below), }; } } } - self.after + self.trail.after + } + + /// Walk on only from above `known`, what the walk has put in range for good staying in + /// its run from [`Self::low`]. Walks that started apart then agree once they get here. + fn fix(&mut self) { + let Some(known) = self.trail.known else { return }; + if self.trail.floor > known { + return; + } + self.trail.floor = known.saturating_add(1); + self.trail.reach = self.trail.reach.max(self.trail.floor); + } + + /// The walk's groups now, and the part a miss may ask for: one run each, from where its + /// spans start up to its cap. Everything from `low` up to `known` is in range for good, + /// and the cap never falls below it. + fn runs(&self) -> (Option>, Option>) { + let last = match self.trail.limit { + Some(end) => end.checked_sub(1).filter(|last| *last >= self.low), + None => Some(u64::MAX), + }; + let Some(last) = last else { + return (None, None); + }; + let missable = (self.low_reach <= last).then_some(self.low_reach..=last); + (Some(self.low..=last), missable) + } + + /// What this walk waits on, if it can park: settling stopped at a gap that may still + /// arrive, below the group its ends fall before, holding from above `known` up to that + /// group only groups below the gap that never start. + /// + /// Until a sequence there arrives or is given up, the walk reads the same answers. One + /// arriving late, even into a run given up, is in the holdings' changes, so the walk + /// wakes then. Costs the groups held there, which never start. + fn waits(&self, holdings: &impl Holdings) -> Option> { + let (gap, after, from) = (self.trail.accounted, self.trail.after?, self.trail.unknown()?); + if gap >= after || holdings.complete() || holdings.gone(gap).is_some() { + return None; + } + let mut at = from; + while let Some((sequence, held)) = holdings.next_held(at) + && sequence < after + { + if sequence >= gap || held != Held::Never { + return None; + } + at = sequence + 1; + } + Some(from..after) + } + + /// Take on `other`'s ends, its trail being this one's. Both runs end where the trails + /// put them, so the merged runs start at the lower of each. + fn merge(&mut self, mut other: Walk) { + self.low = self.low.min(other.low); + self.low_reach = self.low_reach.min(other.low_reach); + if other.ends.len() > self.ends.len() { + std::mem::swap(&mut self.ends, &mut other.ends); + } + self.ends.extend(other.ends); } } @@ -356,11 +572,21 @@ impl Window { let mut window = Self { ranges: ranges.to_vec(), spans: Vec::new(), - groups: RangeInclusiveSet::new(), - missable: RangeInclusiveSet::new(), + looking: Vec::new(), + walks: Vec::new(), + parked: BTreeMap::new(), + seen: Seen::default(), + groups: Layered::default(), + missable: Layered::default(), + resolved: Runs::default(), + resolved_missable: Runs::default(), + stale: false, pending: false, settled: false, covered: None, + named: None, + all_named: true, + admission: Admission::default(), }; window.place(latest, holdings)?; Ok(window) @@ -387,51 +613,147 @@ impl Window { let _ = self.place(latest, holdings); return !self.pending; } + if self.walks.is_empty() && self.parked.is_empty() { + return false; + } + let moved = self.walk(holdings); + if moved || self.stale { + self.rebuild(); + } + moved + } + + /// Walk each unsettled time end on, settling those that can be, and merge the walks + /// that come to agree. Returns whether the groups in range moved. + fn walk(&mut self, holdings: &impl Holdings) -> bool { let mut moved = false; - for span in self.spans.iter_mut().filter(|span| span.looking.is_none()) { - let End::Time(time) = &mut span.end else { continue }; - let limit = time.limit; - if let Some(end) = time.advance(span.start, holdings) { - span.end = End::Before(end); - moved = true; - } else { - moved |= time.limit != limit; + let mut walks = std::mem::take(&mut self.walks); + self.wake(holdings, &mut walks); + for walk in &mut walks { + walk.was = walk.trail.limit; + } + let mut kept = Vec::with_capacity(walks.len()); + while let Some(mut walk) = walks.pop() { + match walk.advance(holdings, &mut walks) { + Some(end) => { + // What settles joins the fixed runs, which readers share: copied once per + // settle under way, since a settle is once per end. + (moved, self.stale) = (true, true); + if walk.low < end { + self.resolved.insert(walk.low..=end - 1); + } + if walk.low_reach < end { + self.resolved_missable.insert(walk.low_reach..=end - 1); + } + for ending in walk.ends { + self.spans[ending.span].end = End::Before(end); + } + } + None => { + moved |= walk.trail.limit != walk.was; + walk.fix(); + match walk.waits(holdings) { + Some(waits) => self.park(waits, walk), + None => kept.push(walk), + } + } } } - if moved { - self.rebuild(); + kept.sort_unstable_by_key(|walk| walk.trail); + for walk in kept { + match self.walks.last_mut() { + Some(last) if last.trail == walk.trail => last.merge(walk), + _ => self.walks.push(walk), + } } moved } + /// Move into `walks` the parked walks a change since the last settle reaches: all of + /// them once the holdings can't say which. + fn wake(&mut self, holdings: &impl Holdings, walks: &mut Vec) { + if self.parked.is_empty() { + // Kept current, so the first walk to park wakes only on what changes after it. + let _ = holdings.changes(&mut self.seen); + return; + } + let woke = walks.len(); + match holdings.changes(&mut self.seen) { + None => walks.extend( + std::mem::take(&mut self.parked) + .into_values() + .flat_map(|parked| parked.walks), + ), + Some(changed) => { + for run in changed { + for (_, parked) in self.overlapping(run) { + walks.extend(parked.walks); + } + } + } + } + // The fixed runs held those it woke. + self.stale |= walks.len() > woke; + } + + /// Park `walk`, waiting on `waits`, with every parked walk whose wait overlaps it. + fn park(&mut self, waits: ops::Range, walk: Walk) { + self.stale = true; + let mut parked = Parked { + end: waits.end, + walks: vec![walk], + }; + let mut start = waits.start; + for overlapping in self.overlapping(waits.clone()) { + start = start.min(overlapping.0); + parked.end = parked.end.max(overlapping.1.end); + parked.walks.extend(overlapping.1.walks); + } + self.parked.insert(start, parked); + } + + /// Take out the parked walks whose waits overlap `run`. + fn overlapping(&mut self, run: ops::Range) -> Vec<(u64, Parked)> { + if run.is_empty() { + return Vec::new(); + } + // They are disjoint: the one holding the run's start, then each starting inside it. + let below = self + .parked + .range(..=run.start) + .next_back() + .filter(|(_, parked)| parked.end > run.start) + .map(|(start, _)| *start); + let inside = self + .parked + .range(run.start.saturating_add(1)..run.end) + .map(|(start, _)| *start); + let starts: Vec = below.into_iter().chain(inside).collect(); + starts + .into_iter() + .filter_map(|start| self.parked.remove(&start).map(|parked| (start, parked))) + .collect() + } + /// The groups an unsettled time end reads on speculation, which may start past it: a /// read parked on the cap wakes when one of them starts. + /// + /// A parked walk waits on a group arriving, never one starting, and an arrival changes + /// the track a read parks on anyway, so only the walks not parked count, here and in + /// [`Self::watch_starts`]. pub fn speculative(&self) -> impl Iterator + '_ { - self.spans - .iter() - .filter(|span| span.looking.is_none()) - .filter_map(|span| match span.end { - End::Time(time) => time.speculative, - _ => None, - }) + self.walks.iter().filter_map(|walk| walk.trail.speculative) } - /// The lowest sequence whose start could still move an unsettled time end, `None` - /// without one or once the last sequence is in range. + /// The lowest sequence whose start could still move an unsettled time end not parked, + /// `None` without one or once the last sequence is in range. pub fn watch_starts(&self) -> Option { - self.spans - .iter() - .filter(|span| span.looking.is_none()) - .filter_map(|span| match span.end { - End::Time(time) => time.unknown(span.start), - _ => None, - }) - .min() + self.walks.iter().filter_map(|walk| walk.trail.unknown()).min() } /// Whether group `sequence` is in range. Everything is while a relative anchor waits. pub fn contains(&self, sequence: u64) -> bool { - self.pending || self.groups.contains(&sequence) + self.pending || self.groups.contains(sequence) } /// Whether a datagram is in range, judged on its own and never through a time end's @@ -441,30 +763,7 @@ impl Window { let Some(timestamp) = timestamp else { return self.contains(sequence); }; - // Spans are in the order of the ranges they resolve, so one pass pairs them. - let mut spans = self.spans.iter().peekable(); - self.ranges - .iter() - .enumerate() - .filter(|(_, range)| !range.is_empty()) - .any(|(index, range)| { - // A relative anchor still waiting on a latest group bounds nothing yet. - let span = spans.next_if(|span| span.index == index); - let started = match range.start { - Anchor::Time(pts) => !later(pts, timestamp), - Anchor::Absolute(start) => sequence >= start, - Anchor::Relative(_) => span.is_none_or(|span| sequence >= span.start), - }; - let ended = match range.end { - None => true, - Some(Anchor::Time(pts)) => !later(timestamp, pts), - Some(Anchor::Absolute(end)) => sequence < end, - Some(Anchor::Relative(_)) => { - span.is_none_or(|span| !matches!(span.end, End::Before(end) if sequence >= end)) - } - }; - started && ended - }) + self.admission.admits(sequence, timestamp) } /// The lowest group in range, `None` while a relative anchor waits. Nothing in range @@ -472,17 +771,17 @@ impl Window { pub fn floor(&self) -> Option { match self.pending { true => None, - false => Some(self.groups.iter().next().map_or(0, |run| *run.start())), + false => Some(self.groups.lowest().unwrap_or(0)), } } /// The lowest group the ranges name, `None` while one reaches back to the oldest group /// or a relative anchor waits on a first group. pub fn start(&self) -> Option { - if self.pending || self.spans.iter().any(|span| !span.floored) { - return None; + match self.pending || !self.all_named { + true => None, + false => self.named, } - self.spans.iter().map(|span| span.start).min() } /// The lowest group a range names, `None` when none does or a relative anchor waits. @@ -490,12 +789,7 @@ impl Window { pub fn named(&self) -> Option { match self.pending { true => None, - false => self - .spans - .iter() - .filter(|span| span.floored) - .map(|span| span.start) - .min(), + false => self.named, } } @@ -536,8 +830,8 @@ impl Window { if !self.settled { return None; } - match self.groups.iter().next_back() { - Some(run) => run.end().checked_add(1), + match self.groups.highest() { + Some(last) => last.checked_add(1), None => Some(0), } } @@ -587,27 +881,29 @@ impl Window { self.from(&self.missable, from) } - fn from<'a>( - &'a self, - groups: &'a RangeInclusiveSet, - from: u64, - ) -> impl Iterator> + 'a { + fn from<'a>(&'a self, groups: &'a Layered, from: u64) -> impl Iterator> + 'a { let all = self.pending.then_some(from..=u64::MAX); - let runs = (!self.pending).then(|| { - groups - .overlapping(from..=u64::MAX) - .map(move |run| (*run.start()).max(from)..=*run.end()) - }); + let runs = (!self.pending).then(|| groups.runs(from)); all.into_iter().chain(runs.into_iter().flatten()) } + /// The groups in range, `None` while a relative anchor waits and every group is. + pub fn groups(&self) -> Option<&Layered> { + (!self.pending).then_some(&self.groups) + } + + /// The groups a miss may ask for, as [`Self::groups`]. + pub fn missable_groups(&self) -> Option<&Layered> { + (!self.pending).then_some(&self.missable) + } + /// The first run of `range` that is in range. pub fn clip(&self, range: ops::Range) -> Option> { let last = range.end.checked_sub(1).filter(|last| *last >= range.start)?; if self.pending { return Some(range); } - let run = self.groups.overlapping(range.start..=last).next()?; + let run = self.groups.first(range.start, last)?; let end = match run.end().checked_add(1) { Some(end) => end.min(range.end), None => range.end, @@ -618,9 +914,9 @@ impl Window { /// The time starts still looked up, each with the exclusive end of the run below every /// group held, which holds the group containing it if the track's history does. pub fn lookups(&self) -> impl Iterator + '_ { - self.spans + self.looking .iter() - .filter_map(|span| span.looking.map(|pts| (pts, span.reach))) + .filter_map(|span| self.spans[*span].looking.map(|looking| (looking.pts, looking.below))) } /// The lookup of time start `pts` answered with the group containing it, or the oldest @@ -629,31 +925,64 @@ impl Window { /// couldn't tell, so they start where the groups held say. Returns whether the groups in /// range moved. pub fn found(&mut self, pts: Timestamp, group: Option, holdings: &impl Holdings) -> bool { - let mut moved = false; - for span in self.spans.iter_mut().filter(|span| span.looking == Some(pts)) { - span.looking = None; - moved = true; - let started = group.filter(|group| holdings.next_start(*group).is_some_and(|(found, _)| found == *group)); - (span.start, span.floored, span.reach) = match (started, holdings.containing(pts)) { + let mut found = Vec::new(); + self.looking.retain(|span| match self.spans[*span].looking { + Some(looking) if looking.pts == pts => { + found.push(*span); + false + } + _ => true, + }); + if found.is_empty() { + return false; + } + let started = group.filter(|group| holdings.next_start(*group).is_some_and(|(found, _)| found == *group)); + for index in found { + let span = &mut self.spans[index]; + let Some(looking) = span.looking.take() else { continue }; + let (start, floored, reach) = match (started, holdings.containing(pts)) { (Some(group), _) | (None, Containing::Group(group)) => (group, true, group), (None, _) => ( 0, false, - holdings.next_start(0).map_or(span.reach, |(oldest, _)| oldest), + holdings.next_start(0).map_or(looking.below, |(oldest, _)| oldest), ), }; - // A time end walked from group 0 meanwhile walks again from where the span reaches: - // a start the history couldn't place asks nothing below the oldest group held, so - // none of those may hold the end open. - if let Some(Anchor::Time(end)) = self.ranges[span.index].end { - let mut time = TimeEnd::new(end, span.reach); - span.end = time.advance(span.start, holdings).map_or(End::Time(time), End::Before); + (span.start, span.floored) = (start, floored); + let limit = match span.end { + End::Before(end) => Some(end), + End::Open => None, + // It walks from where the span reaches: a start the history couldn't place asks + // nothing below the oldest group held, so none of those may hold the end open. + End::Time(end) => { + self.walks.push(Walk { + ends: BTreeSet::from([Ending { pts: end, span: index }]), + trail: Trail { + held: reach, + accounted: reach, + ..Trail::new(start, reach) + }, + low: start, + low_reach: reach, + was: Some(start), + }); + continue; + } + }; + if let Some(last) = limit.map_or(Some(u64::MAX), |end| end.checked_sub(1)) + && last >= start + { + self.resolved.insert(start..=last); + if reach <= last { + self.resolved_missable.insert(reach..=last); + } } } - if moved { - self.rebuild(); - } - moved + self.stale = true; + self.name(); + self.walk(holdings); + self.rebuild(); + true } fn place(&mut self, latest: Option, holdings: &impl Holdings) -> Result<()> { @@ -670,64 +999,256 @@ impl Window { return Err(Error::Unsupported); } self.spans.clear(); + self.looking.clear(); + self.walks.clear(); + self.parked.clear(); + let (mut fixed, mut fixed_missable) = (Vec::new(), Vec::new()); self.pending = false; + // The time ends that start alike walk alike, until a group starts between them. + let mut walks = std::collections::BTreeMap::<(u64, u64), BTreeSet>::new(); for (index, range) in self.ranges.iter().enumerate().filter(|(_, range)| !range.is_empty()) { let (Some((start, floored)), Some(end)) = (start(range.start, latest, holdings), end(range.end, latest)) else { self.pending = true; continue; }; - let end = match end { - End::Time(mut time) => { - time = TimeEnd::new(time.pts, start); - time.advance(start, holdings).map_or(End::Time(time), End::Before) - } - end => end, + // A time start the groups held can't place reads nothing until its lookup answers. + let looking = match range.start { + Anchor::Time(pts) => unresolved(start, floored, holdings).map(|below| Looking { pts, below }), + _ => None, }; - let (reach, looking) = match range.start { - Anchor::Time(pts) => match unresolved(start, floored, holdings) { - Some(searched) => (searched, Some(pts)), - None if floored => (start, None), - None => (holdings.next_start(0).map_or(start, |(oldest, _)| oldest), None), - }, - _ => (start, None), + if looking.is_some() { + self.looking.push(self.spans.len()); + self.spans.push(Span { + index, + start, + end, + floored, + looking, + }); + continue; + } + // The lowest group a miss asks for: a time start before every group held asks none + // below the oldest, since which of them it reaches is unknown until they start. + let reach = match range.start { + Anchor::Time(_) if !floored => holdings.next_start(0).map_or(start, |(oldest, _)| oldest), + _ => start, }; + let limit = match end { + End::Before(end) => Some(end), + End::Open => None, + End::Time(pts) => { + let span = self.spans.len(); + walks.entry((start, reach)).or_default().insert(Ending { pts, span }); + Some(start) + } + }; + if let Some(last) = limit.map_or(Some(u64::MAX), |end| end.checked_sub(1)) + && last >= start + { + fixed.push(start..=last); + if reach <= last { + fixed_missable.push(reach..=last); + } + } self.spans.push(Span { index, start, end, floored, - reach, - looking, + looking: None, }); } + self.admission = Admission::new(&self.ranges, &self.spans); + self.name(); + self.resolved = Runs::new(fixed); + self.resolved_missable = Runs::new(fixed_missable); + self.groups = Layered::new(Arc::new(self.resolved.clone()), Runs::default()); + self.missable = Layered::new(Arc::new(self.resolved_missable.clone()), Runs::default()); + self.stale = false; + self.walks = walks + .into_iter() + .map(|((start, reach), ends)| Walk { + ends, + trail: Trail::new(start, reach), + low: start, + low_reach: reach, + was: Some(start), + }) + .collect(); + self.walk(holdings); self.rebuild(); Ok(()) } + /// Keep [`Self::named`] and [`Self::start`] current with where the spans start. + fn name(&mut self) { + self.named = self + .spans + .iter() + .filter(|span| span.floored) + .map(|span| span.start) + .min(); + self.all_named = self.spans.iter().all(|span| span.floored); + } + + /// The groups in range: the fixed ones, and the run each walk not parked reads now + /// layered over them. Costs those walks, and the fixed runs only once a settle or a walk + /// parking or waking changed them. fn rebuild(&mut self) { self.covered = None; - self.settled = !self.pending - && !self - .spans - .iter() - .any(|span| span.looking.is_some() || matches!(span.end, End::Time(_))); - self.groups = RangeInclusiveSet::new(); - self.missable = RangeInclusiveSet::new(); // A span still looking up its start reads nothing until it knows which group that is. - for span in self.spans.iter().filter(|span| span.looking.is_none()) { - let last = match span.limit() { - Some(end) => match end.checked_sub(1) { - Some(last) if last >= span.start => last, - _ => continue, + self.settled = !self.pending && self.walks.is_empty() && self.parked.is_empty() && self.looking.is_empty(); + let (groups, missable): (Vec<_>, Vec<_>) = self.walks.iter().map(Walk::runs).unzip(); + self.groups.set_layer(Runs::new(groups.into_iter().flatten())); + self.missable.set_layer(Runs::new(missable.into_iter().flatten())); + if std::mem::take(&mut self.stale) { + let parked = || { + self.parked + .values() + .flat_map(|parked| parked.walks.iter()) + .map(Walk::runs) + }; + let groups = self + .resolved + .iter() + .cloned() + .chain(parked().filter_map(|(groups, _)| groups)); + let missable = + (self.resolved_missable.iter().cloned()).chain(parked().filter_map(|(_, missable)| missable)); + let (groups, missable) = (Runs::new(groups), Runs::new(missable)); + self.groups.set_fixed(groups); + self.missable.set_fixed(missable); + } + } +} + +/// The ranges' bounds on a datagram, split by the kind of anchor at each end, so judging one +/// costs a few sorted lookups however many ranges there are. +/// +/// A datagram is one frame, so a time anchor judges its own timestamp, and a group anchor +/// its sequence: each range bounds it from below by one and from above by one. Within each +/// pairing, the ranges sorted by their lower bound carry the highest upper bound of any +/// range at or below them, so the ranges whose lower bound admits a datagram admit it once +/// the last of them reaches it. +#[derive(Debug, Default)] +struct Admission { + groups: Stab, + group_to_time: Stab, + time_to_group: Stab, + times: Stab, +} + +/// One bound of a range on a datagram: the first or last sequence it admits, or the time +/// it presents from or through. +#[derive(Clone, Copy)] +enum Bound { + Group(u64), + Time(Timestamp), +} + +impl Admission { + /// Index `ranges`, with `spans` their resolution in the same order. + fn new(ranges: &[Range], spans: &[Span]) -> Self { + let mut admission = Self::default(); + let mut spans = spans.iter().peekable(); + for (index, range) in ranges.iter().enumerate().filter(|(_, range)| !range.is_empty()) { + // A relative anchor still waiting on a latest group bounds nothing yet. + let span = spans.next_if(|span| span.index == index); + let lower = match range.start { + Anchor::Time(pts) => Bound::Time(pts), + Anchor::Absolute(start) => Bound::Group(start), + Anchor::Relative(_) => Bound::Group(span.map_or(0, |span| span.start)), + }; + // The last sequence a group end admits: one ending at group 0 admits none. + let last = |end: u64| end.checked_sub(1).map(Bound::Group); + let upper = match range.end { + None => Some(Bound::Group(u64::MAX)), + Some(Anchor::Time(pts)) => Some(Bound::Time(pts)), + Some(Anchor::Absolute(end)) => last(end), + Some(Anchor::Relative(_)) => match span.map(|span| span.end) { + Some(End::Before(end)) => last(end), + _ => Some(Bound::Group(u64::MAX)), }, - None => u64::MAX, }; - self.groups.insert(span.start..=last); - if span.reach <= last { - self.missable.insert(span.reach..=last); + match (lower, upper) { + (_, None) => {} + (Bound::Group(lower), Some(Bound::Group(upper))) => admission.groups.push(lower, upper), + (Bound::Group(lower), Some(Bound::Time(upper))) => admission.group_to_time.push(lower, upper), + (Bound::Time(lower), Some(Bound::Group(upper))) => admission.time_to_group.push(lower, upper), + (Bound::Time(lower), Some(Bound::Time(upper))) => admission.times.push(lower, upper), } } + admission.groups.index(); + admission.group_to_time.index(); + admission.time_to_group.index(); + admission.times.index(); + admission + } + + /// Whether any one range admits a datagram's `sequence` and `timestamp` together. + fn admits(&self, sequence: u64, timestamp: Timestamp) -> bool { + self.groups.admits(sequence, sequence) + || self.group_to_time.admits(sequence, timestamp) + || self.time_to_group.admits(timestamp, sequence) + || self.times.admits(timestamp, timestamp) + } +} + +/// A point a datagram is judged at, ordered as instants for a time. +trait Point: Copy { + fn order(self, other: Self) -> std::cmp::Ordering; +} + +impl Point for u64 { + fn order(self, other: Self) -> std::cmp::Ordering { + self.cmp(&other) + } +} + +impl Point for Timestamp { + fn order(self, other: Self) -> std::cmp::Ordering { + instant(self, other) + } +} + +/// Ranges bounded below by an `L` and above by a `U`, both inclusive, sorted by the lower +/// bound, each carrying the highest upper bound at or before it once indexed. +#[derive(Debug)] +struct Stab { + bounds: Vec<(L, U)>, +} + +impl Default for Stab { + fn default() -> Self { + Self { bounds: Vec::new() } + } +} + +impl Stab { + fn push(&mut self, lower: L, upper: U) { + self.bounds.push((lower, upper)); + } + + fn index(&mut self) { + self.bounds.sort_by(|a, b| a.0.order(b.0)); + let mut highest: Option = None; + for (_, upper) in &mut self.bounds { + let top = match highest { + Some(top) if top.order(*upper).is_gt() => top, + _ => *upper, + }; + *upper = top; + highest = Some(top); + } + } + + /// Whether a range bounded at or below `lower` reaches `upper`. + fn admits(&self, lower: L, upper: U) -> bool { + let below = self.bounds.partition_point(|(start, _)| start.order(lower).is_le()); + below + .checked_sub(1) + .is_some_and(|last| upper.order(self.bounds[last].1).is_le()) } } @@ -775,7 +1296,7 @@ fn end(anchor: Option, latest: Option) -> Option { .saturating_sub(back) .checked_add(1) .map_or(End::Open, End::Before), - Some(Anchor::Time(pts)) => End::Time(TimeEnd::new(pts, 0)), + Some(Anchor::Time(pts)) => End::Time(pts), }) } @@ -783,6 +1304,8 @@ fn end(anchor: Option, latest: Option) -> Option { mod tests { use super::*; + use rangemap::RangeInclusiveSet; + use std::cell::Cell; fn at(millis: u64) -> Timestamp { @@ -791,6 +1314,9 @@ mod tests { /// A hop's groups by sequence, the runs that can no longer arrive, and how many held /// groups and starts a walk asked about. + /// + /// Reports changes as a track does: a group arriving and a run given up, never a group + /// starting or ending empty, which its ledger never sees. #[derive(Default)] struct Groups { held: std::collections::BTreeMap, @@ -798,6 +1324,12 @@ mod tests { complete: bool, looks_up: bool, asked: Cell, + /// How many fetched copies gave a group held past its frame 0 a start, which a + /// track counts rather than logs. + fetched: Cell, + /// Every change so far, oldest first, and what was last logged: the held sequences + /// and how many runs were given up. + log: std::cell::RefCell<(Vec>, BTreeSet, usize)>, } impl Groups { @@ -863,6 +1395,30 @@ mod tests { fn complete(&self) -> bool { self.complete } + + fn changes(&self, seen: &mut Seen) -> Option>> { + let mut log = self.log.borrow_mut(); + let (log, held, gone) = &mut *log; + let arrived = self.held.keys().filter(|sequence| !held.contains(sequence)); + log.extend(arrived.map(|sequence| *sequence..sequence.saturating_add(1))); + log.extend(self.gone[*gone..].iter().cloned()); + *held = self.held.keys().copied().collect(); + *gone = self.gone.len(); + let from = usize::try_from(seen.runs[0]).unwrap(); + let changed = log[from..].to_vec(); + let global = [ + u64::from(self.complete), + self.fetched.get(), + u64::from(self.looks_up), + 0, + ]; + let primed = seen.global == Some(global); + *seen = Seen { + runs: [log.len() as u64, 0], + global: Some(global), + }; + primed.then_some(changed) + } } struct Untimed; @@ -1092,6 +1648,180 @@ mod tests { assert!(asked < 8 * count as usize, "{asked} queries for {count} groups"); } + /// Time ends that start alike share one walk, so a settle costs the walk, not the ends. + /// A group starting between them splits off and settles the ends before it. + #[test] + fn time_ends_that_start_alike_share_one_walk() { + let ends = 1_000; + let mut groups = Groups::new([Groups::started(0, 0)]); + let ranges: Vec = (0..ends).map(|i| range(time(0), time(1_000_000 + i))).collect(); + let mut window = Window::new(&ranges, Some(0), &groups).unwrap(); + assert_eq!(window.walks.len(), 1); + + let count = 200; + let asked = groups.asked.get(); + for sequence in 1..count { + groups.held.insert(sequence, Held::Started(at(sequence))); + window.settle(Some(sequence), &groups); + } + let asked = groups.asked.get() - asked; + assert!(asked < 8 * count as usize, "{asked} queries for {count} groups"); + assert_eq!(window.walks.len(), 1); + assert!(!window.is_settled() && window.contains(count)); + + // Group 200 starts inside the later half of the ends: the earlier half settles before it. + groups.held.insert(count, Held::Started(at(1_000_000 + ends / 2))); + assert!(window.settle(Some(count), &groups)); + assert_eq!(window.walks.len(), 1); + let resolved = window.resolved(); + let split = usize::try_from(ends / 2).unwrap(); + assert!(resolved[..split].iter().all(|r| *r + == Some(Resolved { + start: 0, + end: Some(count) + }))); + assert!( + resolved[split..] + .iter() + .all(|r| *r == Some(Resolved { start: 0, end: None })) + ); + + // Once a group starts past every end, they all settle. + groups.held.insert(count + 1, Held::Started(at(2_000_000))); + assert!(window.settle(Some(count + 1), &groups)); + assert!(window.is_settled() && window.walks.is_empty()); + assert_eq!(covered(&window, count + 3), (0..=count).collect::>()); + } + + /// Shared walks resolve every range as walking it alone would, as groups arrive, start, + /// never start, or go missing in any order, with ends in mixed scales. + #[test] + fn shared_walks_match_each_end_walked_alone() { + // A small deterministic generator, so a failure replays. + let seed = Cell::new(0x2545_f491_4f6c_dd1d_u64); + let next = |bound: u64| { + let mut x = seed.get(); + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + seed.set(x); + x % bound + }; + let micros = |value| Anchor::Time(Timestamp::new(value, crate::Timescale::MICRO).unwrap()); + const SEQUENCES: u64 = 16; + for round in 0..400 { + // Each group starts in its own ten milliseconds, so starts ascend with the sequence. + let starts: Vec = (0..SEQUENCES).map(|sequence| sequence * 10 + next(10)).collect(); + let pts = |end: bool| match next(3) { + 0 => micros(next(SEQUENCES * 10_000)), + // Exactly a group's start, in either scale. + 1 if next(2) == 0 => time(starts[next(SEQUENCES) as usize]), + 1 => micros(starts[next(SEQUENCES) as usize] * 1000), + _ => time(next(SEQUENCES * 10) + u64::from(end) * 5), + }; + let ranges: Vec = (0..1 + next(12)) + .map(|_| { + let start = match next(3) { + 0 => Anchor::Absolute(next(SEQUENCES)), + 1 => Anchor::Relative(next(4)), + _ => pts(false), + }; + let end = match next(6) { + 0 => None, + 1 => Some(Anchor::Absolute(next(SEQUENCES + 2))), + _ => Some(pts(true)), + }; + range(start, end) + }) + .collect(); + + let mut groups = Groups::default(); + for _ in 0..next(6) { + let sequence = next(SEQUENCES); + groups + .held + .insert(sequence, Held::Started(at(starts[sequence as usize]))); + } + let latest = Some(next(SEQUENCES)); + let mut window = Window::new(&ranges, latest, &groups).unwrap(); + let mut alone: Vec = ranges + .iter() + .map(|range| Window::new(std::slice::from_ref(range), latest, &groups).unwrap()) + .collect(); + + for step in 0..40 { + let sequence = next(SEQUENCES); + let started = Held::Started(at(starts[sequence as usize])); + match next(6) { + 0 | 1 => { + groups.held.entry(sequence).or_insert(Held::Pending); + } + 2 | 3 => { + if groups.held.get(&sequence).is_none_or(|held| *held == Held::Pending) { + groups.held.insert(sequence, started); + } + } + 4 => match groups.held.get(&sequence) { + None | Some(Held::Pending) => { + groups.held.insert(sequence, Held::Never); + } + // A fetched head gives a group held past its frame 0 a start. + Some(Held::Never) if next(2) == 0 => { + groups.held.insert(sequence, started); + groups.fetched.set(groups.fetched.get() + 1); + } + _ => {} + }, + _ => { + let end = sequence + 1 + next(3); + if groups.held.range(sequence..end).next().is_none() { + groups.gone.push(sequence..end); + } + } + } + groups.complete |= step == 39 && round % 4 == 0; + + let before: Vec = covered(&window, SEQUENCES + 4); + let moved = window.settle(latest, &groups); + for window in &mut alone { + window.settle(latest, &groups); + } + let context = || format!("round {round} step {step}: {ranges:?} over {:?}", groups.held); + + let union: Vec = (0..SEQUENCES + 4) + .filter(|sequence| alone.iter().any(|window| window.contains(*sequence))) + .collect(); + let now = covered(&window, SEQUENCES + 4); + assert_eq!(now, union, "{}", context()); + assert!(moved || now == before, "{}: moved unreported", context()); + + let mut missable = RangeInclusiveSet::new(); + for run in alone.iter().flat_map(|window| window.missable(0)) { + missable.insert(run); + } + assert!(window.missable(0).eq(missable.iter().cloned()), "{}", context()); + + let resolved: Vec<_> = alone.iter().map(|window| window.resolved()[0]).collect(); + assert_eq!(window.resolved(), resolved, "{}", context()); + assert_eq!( + window.is_settled(), + alone.iter().all(Window::is_settled), + "{}", + context() + ); + let speculative: std::collections::BTreeSet = window.speculative().collect(); + let each: std::collections::BTreeSet = alone.iter().flat_map(Window::speculative).collect(); + assert_eq!(speculative, each, "{}", context()); + assert_eq!( + window.watch_starts(), + alone.iter().filter_map(Window::watch_starts).min(), + "{}", + context() + ); + } + } + } + /// A datagram is one frame, so its own timestamp places it against a time end. #[test] fn a_datagram_is_judged_by_its_own_timestamp() { @@ -1236,6 +1966,84 @@ mod tests { assert!(window.admits_datagram(6, Some(at(5000)))); } + /// The indexed admission agrees with judging each range on its own, over mixed anchors, + /// scales, and relative anchors resolved or still waiting. + #[test] + fn datagram_admission_matches_each_range_judged_alone() { + fn admits_alone(window: &Window, sequence: u64, timestamp: Timestamp) -> bool { + window.ranges.iter().enumerate().any(|(index, range)| { + if range.is_empty() { + return false; + } + let span = window.spans.iter().find(|span| span.index == index); + let started = match range.start { + Anchor::Time(pts) => !later(pts, timestamp), + Anchor::Absolute(start) => sequence >= start, + Anchor::Relative(_) => span.is_none_or(|span| sequence >= span.start), + }; + let ended = match range.end { + None => true, + Some(Anchor::Time(pts)) => !later(timestamp, pts), + Some(Anchor::Absolute(end)) => sequence < end, + Some(Anchor::Relative(_)) => { + span.is_none_or(|span| !matches!(span.end, End::Before(end) if sequence >= end)) + } + }; + started && ended + }) + } + + // A small deterministic generator, so a failure replays. + let seed = Cell::new(0x9e37_79b9_7f4a_7c15_u64); + let next = |bound: u64| { + let mut x = seed.get(); + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + seed.set(x); + x % bound + }; + let micros = |value| Timestamp::new(value, crate::Timescale::MICRO).unwrap(); + let groups = Groups::new((0..20).map(|sequence| Groups::started(sequence, sequence * 100))); + for round in 0..300 { + let anchor = |end: bool| match next(if end { 4 } else { 3 }) { + 0 => Anchor::Absolute(next(20)), + 1 => Anchor::Relative(next(6)), + 2 if next(2) == 0 => time(next(2000)), + 2 => Anchor::Time(micros(next(2_000_000))), + _ => Anchor::Absolute(u64::MAX), + }; + let ranges: Vec = (0..1 + next(6)) + .map(|_| { + let start = anchor(false); + let end = match next(4) { + 0 => None, + _ => Some(anchor(true)), + }; + range(start, end) + }) + .collect(); + let latest = (round % 3 != 0).then(|| next(20)); + let held = match latest { + Some(_) => &groups, + None => &Groups::default(), + }; + let window = Window::new(&ranges, latest, held).unwrap(); + for _ in 0..40 { + let sequence = next(24); + let timestamp = match next(2) { + 0 => at(next(2400)), + _ => micros(next(2_400_000)), + }; + assert_eq!( + window.admits_datagram(sequence, Some(timestamp)), + admits_alone(&window, sequence, timestamp), + "{ranges:?} at {latest:?}: datagram {sequence} at {timestamp:?}" + ); + } + } + } + /// A coverage check resumes where the last stopped, but not below its floor or past its /// cap: a lower floor or a wider cap reaches groups no check has seen. #[test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 214e500bdfb..66d4c97614d 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -22,7 +22,7 @@ use super::delivered::Delivered; use super::skipped::Skipped; use crate::ledger::{self, Ledger}; -use super::subscription::{Accounted, Containing, Held, Holdings, Resolved, Window, later}; +use super::subscription::{Accounted, Containing, Held, Holdings, Layered, Resolved, Runs, Seen, Window, later}; pub use super::subscription::{Anchor, Order, Position, Range, Subscription}; use rangemap::{RangeInclusiveSet, RangeMap, RangeSet}; @@ -45,6 +45,7 @@ use std::{ sync::Arc, sync::OnceLock, sync::PoisonError, + sync::Weak, sync::atomic::{AtomicBool, Ordering}, task::{Poll, ready}, time::Duration, @@ -501,6 +502,10 @@ pub(crate) struct TrackState { // Bumped on every insert into `drops`, so a cursor rescans it only after a change. drops_revision: u64, drops_changes: ledger::Changes, + // How many copies the cache took in that the live feed didn't deliver: fetched backfill, + // and fetched heads joining a live group, which may give it a start. Neither is in the + // arrival ledger, so a time end parked on a gap looks again. + fetched: u64, // Every reader's eviction inbox. Its own lock, so a reader registers through the read // guard it judges under. inboxes: Mutex, @@ -852,7 +857,7 @@ impl Holdings for TrackState { let dropped = self.drops.get_key_value(&sequence).map(|(dropped, _)| dropped.end); // The history a handler serves may still hold what the live feed never delivered, or // delivered and the cache evicted, starting anywhere: only a drop, or a reader's own - // refused fills (see `Seen`), says it is gone. + // refused fills (see `ReaderHoldings`), says it is gone. if self.looks_up() { return dropped; } @@ -867,17 +872,38 @@ impl Holdings for TrackState { fn complete(&self) -> bool { self.sealed || self.abort.is_some() } + + fn changes(&self, seen: &mut Seen) -> Option>> { + let global = [ + self.start_sequence.map_or(0, |start| start.saturating_add(1)), + u64::from(self.complete()), + self.fetched, + // A handler appearing or leaving changes which starts count and what is gone. + u64::from(self.looks_up()), + ]; + let (ledger, drops) = (self.arrivals.changes(), &self.drops_changes); + let changed = (seen.global == Some(global)) + .then(|| ledger.since(seen.runs[0]).chain(drops.since(seen.runs[1])).collect()); + *seen = Seen { + runs: [ledger.revision(), drops.revision()], + global: Some(global), + }; + changed + } } /// Where a cursor's groups start, for resolving its time anchors: its track's, with the /// runs its own fills refused or it gave up counted gone, so a time end walking history a /// handler serves passes what that history lacks. -struct Seen<'a> { +struct ReaderHoldings<'a> { state: &'a TrackState, misses: Option<&'a miss::Misses>, + /// What the reader's own fills gave up since its window last took it in (see + /// [`miss::Misses::take_gone`]): changes a parked time end wakes on beside the track's. + newly_gone: RangeSet, } -impl Holdings for Seen<'_> { +impl Holdings for ReaderHoldings<'_> { fn timed(&self) -> bool { self.state.timed() } @@ -909,6 +935,12 @@ impl Holdings for Seen<'_> { fn complete(&self) -> bool { self.state.complete() } + + fn changes(&self, seen: &mut Seen) -> Option>> { + let mut changed = self.state.changes(seen)?; + changed.extend(self.newly_gone.iter().cloned()); + Some(changed) + } } impl TrackState { @@ -1516,36 +1548,68 @@ impl TrackState { let end = end.map_or(Bound::Unbounded, Bound::Excluded); scan(&mut self.live((Bound::Included(from), end))) } - Servable::Runs(runs) => { - // What is left to search, shrinking from the end the search starts at. - let (mut lo, mut hi) = (from, u64::MAX); - while lo <= hi { - let mut left = runs.overlapping(lo..=hi); - let run = match rev { - true => left.next_back(), - false => left.next(), - }?; - let (start, end) = ((*run.start()).max(lo), (*run.end()).min(hi)); - // The cache from the run on, toward the far end: one lookup either finds - // the run's groups or the next cached group past a run holding none. - let leap = match rev { - true => leap(self.live(lo..=end).rev(), start..=end, &mut f), - false => leap(self.live(start..=hi), start..=end, &mut f), - }; - match leap { - Leap::Found(found) => return Some(found), - Leap::Empty => return None, - Leap::Past(sequence) if rev => hi = sequence, - Leap::Past(sequence) => lo = sequence, - Leap::Exhausted if rev => hi = start.checked_sub(1)?, - Leap::Exhausted => lo = end.checked_add(1)?, - } + Servable::Runs(several) => { + let hi = match several.cap { + Some(cap) => cap.checked_sub(1)?, + None => u64::MAX, + }; + if from > hi { + return None; } - None + let groups = &several.groups; + // The fixed runs through the reader's index of the live groups among them, then + // the walks' few runs layered over them, short of what the index found. + let fixed = several.live.find(self, groups.fixed(), from, hi, rev, &mut f); + // Nothing lies past a find at the far end, so only the find answers. + let short = match (&fixed, rev) { + (None, _) => Some((from, hi)), + (Some((found, _)), true) => found.checked_add(1).map(|lo| (lo, hi)), + (Some((found, _)), false) => found.checked_sub(1).map(|hi| (from, hi)), + }; + let layered = short + .filter(|(lo, hi)| lo <= hi) + .and_then(|(lo, hi)| self.leapfrog(groups.layer(), lo, hi, rev, &mut f)); + layered.or(fixed.map(|(_, found)| found)) } } } + /// [`Self::find_servable`] over `runs` within `lo..=hi`: several runs leapfrog the live + /// groups, so a run holding none costs one lookup rather than a walk over every run past + /// it. + fn leapfrog<'a, R>( + &'a self, + runs: &Runs, + mut lo: u64, + mut hi: u64, + rev: bool, + f: &mut impl FnMut(&'a Slot) -> Option, + ) -> Option { + // What is left to search, shrinking from the end the search starts at. + while lo <= hi { + let run = match rev { + true => runs.last(lo, hi), + false => runs.first(lo, hi), + }?; + let (start, end) = ((*run.start()).max(lo), (*run.end()).min(hi)); + // The cache from the run on, toward the far end: one lookup either finds the run's + // groups or the next cached group past a run holding none. + let leap = match rev { + true => leap(self.live(lo..=end).rev(), start..=end, f), + false => leap(self.live(start..=hi), start..=end, f), + }; + match leap { + Leap::Found(found) => return Some(found), + Leap::Empty => return None, + Leap::Past(sequence) if rev => hi = sequence, + Leap::Past(sequence) => lo = sequence, + Leap::Exhausted if rev => hi = start.checked_sub(1)?, + Leap::Exhausted => lo = end.checked_add(1)?, + } + } + None + } + /// Whether `sequence` still holds the servable incarnation `stamp`. fn holds(&self, sequence: u64, stamp: u32) -> bool { self.lookup @@ -2217,6 +2281,9 @@ impl TrackState { } self.advance(sequence, visible); + if !visible { + self.fetched += 1; + } // A publisher may re-create a group it cut; the sequence is no longer given up. if visible && self.drops.contains_key(&sequence) @@ -3064,6 +3131,8 @@ impl TrackState { if let Some(head) = slot.head.replace(Box::new(group.handle())) { self.detach(&head); } + // A head holding frame 0 gives a group received past it a start. + self.fetched += 1; } // Invisible to arrival-order subscribers: fetched on demand, not produced // live by the publisher. @@ -6078,9 +6147,70 @@ struct Edge { servable: Servable, } -/// The groups `window` covers from `from`, below the exclusive `cap`. -fn admitted(window: &Window, from: u64, cap: Option) -> RangeInclusiveSet { - below(window.runs(from), cap) +/// What a cursor's arrival walk reads: its window from the floor, below the cap. Holds the +/// window's groups as they are layered, so taking it again after a time end moves costs +/// the walks' runs, not every run of the window. +#[derive(Clone, Default)] +struct Admitted { + /// The window's groups, `None` while a relative anchor waits and every group is in it. + groups: Option, + floor: u64, + cap: Option, +} + +impl Admitted { + fn new(window: &Window, floor: u64, cap: Option) -> Self { + Self { + groups: window.groups().cloned(), + floor, + cap, + } + } + + /// The last group admitted, `None` for none. + fn last(&self) -> Option { + match self.cap { + Some(cap) => cap.checked_sub(1).filter(|last| *last >= self.floor), + None => Some(u64::MAX), + } + } + + /// The admitted runs reaching into `run`, clipped to it, lowest first. + fn overlapping(&self, run: ops::RangeInclusive) -> Vec> { + let Some(last) = self.last() else { + return Vec::new(); + }; + let (lo, hi) = ((*run.start()).max(self.floor), (*run.end()).min(last)); + if lo > hi { + return Vec::new(); + } + match &self.groups { + None => std::iter::once(lo..=hi).collect(), + Some(groups) => groups + .runs(lo) + .take_while(|run| *run.start() <= hi) + .map(|run| *run.start()..=(*run.end()).min(hi)) + .collect(), + } + } + + /// Every admitted run, lowest first. + fn runs(&self) -> Vec> { + self.overlapping(0..=u64::MAX) + } + + /// What this admits and `old` doesn't within `lo..=hi`, when the two differ only in the + /// window's walks: `None` otherwise. + fn gained(&self, old: &Self, lo: u64, hi: u64) -> Option>> { + let (groups, old_groups) = (self.groups.as_ref()?, old.groups.as_ref()?); + if (self.floor, self.cap) != (old.floor, old.cap) || !groups.shares_fixed(old_groups) { + return None; + } + let Some(last) = self.last() else { + return Some(Vec::new()); + }; + groups.gained(old_groups, lo.max(self.floor), hi.min(last)) + } } /// `runs`, lowest first, below the exclusive `cap`. @@ -6173,8 +6303,177 @@ enum Servable { /// case, kept as cheap to copy and look up as a cap. Its start still bounds a group /// handed out before the run moved above it, which is judged against the run. Run { start: u64, end: Option }, - /// Disjoint runs, lowest first. - Runs(Arc>), + /// Several runs, as the window holds them. + Runs(Arc), +} + +/// A servable window of several runs: the window's groups as it layers them, below the +/// exclusive `cap`, with the reader's index of the live groups inside its fixed runs. +/// +/// Built again whenever the window or cap moves, which a time end does on each group, so it +/// shares the fixed runs and their index, and copies only the walks' runs. +struct Several { + groups: Layered, + cap: Option, + live: Arc, +} + +impl PartialEq for Several { + fn eq(&self, other: &Self) -> bool { + self.groups == other.groups && self.cap == other.cap + } +} + +/// A reader's index of the live feed's cached groups inside its window's fixed runs, so a +/// search costs the live groups it passes, never the runs between them: thousands of runs +/// in the gaps of a sparsely numbered feed would otherwise each cost a lookup. +/// +/// Taken in from the arrival order, which every live group enters once, on the reader's own +/// search: each arrival costs it one lookup in the fixed runs. A slot that leaves or is +/// replaced leaves its entry stale, dropped when a search meets it or once stale entries +/// could outnumber the live groups. +/// +/// One index per track searched: a group's expiry that moved to a replacement copy (see +/// [`GroupExpiry::for_track`]) searches its reader's servable on that copy's track, whose +/// arrival order and stamps are its own. +#[derive(Default)] +struct Live(Mutex>); + +struct Indexed { + /// The track indexed, by its cache account. Held weakly, so a track that went away + /// leaves an entry another track's can't be mistaken for. + track: Weak, + /// How far into the arrival order this has taken in, as an absolute index. + taken: usize, + /// The stamp of each live group inside the fixed runs, by sequence, as it arrived. + groups: BTreeMap, +} + +/// How many index entries a search takes per lock: the search answers with the lock +/// released, since the answer reads each group's own state. +const LIVE_BATCH: usize = 16; + +impl Indexed { + /// Whether this indexes `state`'s track. + fn indexes(&self, state: &TrackState) -> bool { + std::ptr::eq(self.track.as_ptr(), Arc::as_ptr(&state.cache)) + } + + /// Take in what arrived since the last call. + fn catch_up(&mut self, state: &TrackState, fixed: &Runs) { + let skip = self.taken.saturating_sub(state.offset).min(state.arrival.len()); + for (sequence, stamp) in state.arrival.range(skip..) { + if fixed.contains(*sequence) { + self.groups.insert(*sequence, *stamp); + } + } + self.taken = state.offset + state.arrival.len(); + // A live entry is a visible slot, so past twice their count most entries are stale, + // and a prune costs each entry it drops one lookup. + if self.groups.len() > 2 * state.visible.len().max(LIVE_BATCH) { + self.groups + .retain(|sequence, stamp| current(state, *sequence, *stamp).is_some()); + } + } +} + +/// The live slot at `sequence`, if it is still the incarnation stamped `stamp`. +fn current(state: &TrackState, sequence: u64, stamp: u32) -> Option<&Slot> { + state + .lookup + .get(&sequence) + .filter(|slot| slot.stamp == stamp && slot.visible) +} + +impl Live { + /// The index of `state`'s track among `indexes`, made on its first search. Indexes of + /// tracks that went away are dropped then. + fn index<'a>(indexes: &'a mut Vec, state: &TrackState) -> &'a mut Indexed { + let position = match indexes.iter().position(|indexed| indexed.indexes(state)) { + Some(position) => position, + None => { + indexes.retain(|indexed| indexed.track.strong_count() > 0); + indexes.push(Indexed { + track: Arc::downgrade(&state.cache), + taken: 0, + groups: BTreeMap::new(), + }); + indexes.len() - 1 + } + }; + &mut indexes[position] + } + + /// The first live group inside `fixed`, within `lo..=hi`, for which `f` answers, lowest + /// first or, with `rev`, highest first, with its sequence. + fn find<'a, R>( + &self, + state: &'a TrackState, + fixed: &Runs, + mut lo: u64, + mut hi: u64, + rev: bool, + f: &mut impl FnMut(&'a Slot) -> Option, + ) -> Option<(u64, R)> { + let mut caught_up = false; + loop { + if lo > hi { + return None; + } + let mut batch = [(0, 0); LIVE_BATCH]; + let taken = { + let mut indexes = self.0.lock().unwrap_or_else(PoisonError::into_inner); + let indexed = Self::index(&mut indexes, state); + if !caught_up { + indexed.catch_up(state, fixed); + caught_up = true; + } + let entries = indexed + .groups + .range(lo..=hi) + .map(|(&sequence, &stamp)| (sequence, stamp)); + let mut taken = 0; + let mut put = |entry| { + batch[taken] = entry; + taken += 1; + }; + match rev { + true => entries.rev().take(LIVE_BATCH).for_each(&mut put), + false => entries.take(LIVE_BATCH).for_each(&mut put), + } + taken + }; + let batch = &batch[..taken]; + let mut stale = Vec::new(); + let mut found = None; + for &(sequence, stamp) in batch { + match current(state, sequence, stamp) { + Some(slot) => { + if let Some(answer) = f(slot) { + found = Some((sequence, answer)); + break; + } + } + None => stale.push(sequence), + } + } + if !stale.is_empty() { + let mut indexes = self.0.lock().unwrap_or_else(PoisonError::into_inner); + let indexed = Self::index(&mut indexes, state); + for sequence in stale { + indexed.groups.remove(&sequence); + } + } + if found.is_some() || taken < LIVE_BATCH { + return found; + } + let &(last, _) = batch.last().expect("a full batch"); + match rev { + true => hi = last.checked_sub(1)?, + false => lo = last.checked_add(1)?, + } + } + } } impl Servable { @@ -6188,27 +6487,47 @@ impl Servable { fn contains(&self, sequence: u64) -> bool { match self { Self::Run { start, end } => sequence >= *start && end.is_none_or(|end| sequence < end), - Self::Runs(runs) => runs.contains(&sequence), + Self::Runs(several) => several.cap.is_none_or(|cap| sequence < cap) && several.groups.contains(sequence), } } - /// The groups in `runs`, disjoint and lowest first. Only several runs allocate: every + /// The groups `window` reads. `previous`, what the same reader was served before, lends + /// its index of the live groups when the fixed runs are the same. + fn of(window: &Window, previous: &Self) -> Self { + match window.groups() { + None => Self::ALL, + Some(groups) => Self::layered(groups.clone(), None, previous), + } + } + + /// The groups in `groups` below the exclusive `cap`. Only several runs allocate: every /// subscriber resolves one, so the common single run stays off the heap. - fn new(runs: impl Iterator>) -> Self { - let mut runs = runs.peekable(); - let Some(first) = runs.next() else { + fn layered(groups: Layered, cap: Option, previous: &Self) -> Self { + let last = match cap { + Some(cap) => match cap.checked_sub(1) { + Some(last) => last, + None => return Self::NONE, + }, + None => u64::MAX, + }; + let (first, more) = { + let mut runs = groups.runs(0).take_while(|run| *run.start() <= last); + (runs.next(), runs.next().is_some()) + }; + let Some(first) = first else { return Self::NONE; }; - if runs.peek().is_none() { + if !more { return Self::Run { start: *first.start(), - end: first.end().checked_add(1), + end: (*first.end()).min(last).checked_add(1), }; } - let mut set = RangeInclusiveSet::new(); - set.insert(first); - set.extend(runs); - Self::Runs(Arc::new(set)) + let live = match previous { + Self::Runs(several) if several.groups.shares_fixed(&groups) => several.live.clone(), + _ => Arc::default(), + }; + Self::Runs(Arc::new(Several { groups, cap, live })) } /// These groups below the exclusive `cap`. @@ -6227,15 +6546,9 @@ impl Servable { false => Self::NONE, } } - Self::Runs(runs) => { - let Some(last) = cap.checked_sub(1) else { - return Self::NONE; - }; - Self::new( - runs.iter() - .take_while(|run| *run.start() <= last) - .map(|run| *run.start()..=(*run.end()).min(last)), - ) + Self::Runs(several) => { + let cap = several.cap.map_or(cap, |own| own.min(cap)); + Self::layered(several.groups.clone(), Some(cap), self) } } } @@ -6465,7 +6778,7 @@ struct Filter { refused: Option, /// What the arrival walk reads: the window from the floor, below the cap. The walk /// passes a group outside it holding nothing, so widening it re-scans the cache. - admitted: RangeInclusiveSet, + admitted: Admitted, /// The floor, [`Self::revision`], and cap [`Self::admitted`] was taken at. admitted_at: Option<(u64, u64, Option)>, /// The spans a widening newly covers, from the ranges, the floor, the cap, or the @@ -6517,7 +6830,7 @@ impl Cursor { resolved: ranges_revision, revision: 0, refused, - admitted: RangeInclusiveSet::new(), + admitted: Admitted::default(), admitted_at: None, owed: RangeInclusiveSet::new(), lookups: Vec::new(), @@ -6525,7 +6838,7 @@ impl Cursor { }; (filter, state.datagram_offset + start, state.cache.clone()) }; - let published = Published::new(Servable::new(filter.window.runs(0)), filter.resolved, None); + let published = Published::new(Servable::of(&filter.window, &Servable::NONE), filter.resolved, None); Self { state, cache, @@ -6598,7 +6911,7 @@ impl Cursor { fn update_servable(&mut self) { // A moved window or cap notifies nothing this cursor parked on. idle::Idle::disarm(&mut self.idle); - let runs = Servable::new(self.filter.window.runs(0)); + let runs = Servable::of(&self.filter.window, &self.published.read().runs); let (revision, cap) = (self.filter.resolved, self.end_sequence); self.servable = runs.below(cap); let next = |current: &Published| { @@ -6631,16 +6944,28 @@ impl Cursor { let revision = self.subscription.ranges_revision(); (revision != self.filter.resolved).then(|| (revision, preferences.ranges.clone())) }; + // Taken even when nothing walks, so it never piles up. + let newly_gone = self + .misses + .as_deref_mut() + .map(miss::Misses::take_gone) + .unwrap_or_default(); // A settled window has nothing to read from the track, whose lock every subscriber // of it takes. if replaced.is_none() && self.filter.window.is_settled() { return false; } let state = self.state.read(); + // Every walk reads the reader's own gone runs, so none parks on a gap they fill. + let holdings = ReaderHoldings { + state: &state, + misses: self.misses.as_deref(), + newly_gone, + }; let moved = match replaced { Some((revision, ranges)) => { self.filter.resolved = revision; - match Window::new(&ranges, state.latest_live(), &*state) { + match Window::new(&ranges, state.latest_live(), &holdings) { Ok(window) => { self.filter.window = window; self.filter.refused = None; @@ -6654,14 +6979,8 @@ impl Cursor { // The live feed had no group when the ranges resolved, so the newest one then was // the first to arrive, or the newest once it went live, however many have arrived // or left since. - None if self.filter.window.is_pending() => self.filter.window.settle(state.first_live(), &*state), - None => { - let seen = Seen { - state: &state, - misses: self.misses.as_deref(), - }; - self.filter.window.settle(state.latest_live(), &seen) - } + None if self.filter.window.is_pending() => self.filter.window.settle(state.first_live(), &holdings), + None => self.filter.window.settle(state.latest_live(), &holdings), }; drop(state); if moved { @@ -6747,24 +7066,26 @@ impl Cursor { super::subscription::before_end(sequence, self.end_sequence) && self.filter.window.contains(sequence) } - /// The window from `from`, below the exclusive `cap`. - fn admitted(&self, from: u64, cap: Option) -> RangeInclusiveSet { - admitted(&self.filter.window, from, cap) - } - - /// [`Filter::owed`] as of now: what was owed and what the walk newly reads since - /// [`Filter::admitted`] was taken, both cut to `admitted`. - fn owed(&self, admitted: &RangeInclusiveSet) -> RangeInclusiveSet { + /// [`Filter::owed`] as of `admitted`: what was owed and what the walk newly reads since + /// [`Filter::admitted`] was taken, both cut to `admitted`. Costs every run of both. + fn owed(&self, admitted: &Admitted) -> RangeInclusiveSet { // Nothing is owed before the walk first reads: it starts at the oldest arrival. if self.filter.admitted_at.is_none() { return RangeInclusiveSet::new(); } let mut owed = RangeInclusiveSet::new(); - for run in admitted.iter() { - for gap in self.filter.admitted.gaps(run) { - owed.insert(gap); + for run in admitted.runs() { + let mut from = Some(*run.start()); + for held in self.filter.admitted.overlapping(run.clone()) { + if let Some(gap) = from.filter(|from| from < held.start()) { + owed.insert(gap..=*held.start() - 1); + } + from = held.end().checked_add(1); } - for span in self.filter.owed.overlapping(run) { + if let Some(gap) = from.filter(|from| from <= run.end()) { + owed.insert(gap..=*run.end()); + } + for span in self.filter.owed.overlapping(&run) { owed.insert((*span.start()).max(*run.start())..=(*span.end()).min(*run.end())); } } @@ -6773,14 +7094,54 @@ impl Cursor { /// Take in a move of the floor, the ranges, or the cap: whatever it newly admits is /// owed. Costs the ranges, never the cache, and only when one moved. + /// + /// A time end whose cap moves on each group moves only the walks' runs layered over the + /// window's fixed ones, so that costs those runs: what they gain is owed only where the + /// cache holds a group now, the only place a scan finds one, and what they lose is cut + /// only from what is owed. fn admit(&mut self) { let at = (self.floor(), self.filter.revision, self.end_sequence); if self.filter.admitted_at == Some(at) { return; } // The arrival walk reads the window from the floor. - let admitted = self.admitted(self.floor(), self.end_sequence); - self.filter.owed = self.owed(&admitted); + let admitted = Admitted::new(&self.filter.window, self.floor(), self.end_sequence); + let moved = self.filter.admitted_at.and_then(|_| { + let cached = { + let state = self.state.read(); + state + .lookup + .first_key_value() + .map(|(first, _)| *first) + .zip(state.lookup.last_key_value().map(|(last, _)| *last)) + }; + let gained = match cached { + Some((first, last)) => admitted.gained(&self.filter.admitted, first, last)?, + None => admitted.gained(&self.filter.admitted, 1, 0)?, + }; + let owed = self + .filter + .owed + .first() + .map(|run| *run.start()) + .zip(self.filter.owed.last().map(|run| *run.end())); + let lost = match owed { + Some((first, last)) => self.filter.admitted.gained(&admitted, first, last)?, + None => Vec::new(), + }; + Some((gained, lost)) + }); + match moved { + Some((gained, lost)) => { + for run in lost { + self.filter.owed.remove(run); + } + for run in gained { + self.filter.owed.insert(run); + } + } + None => self.filter.owed = self.owed(&admitted), + } self.filter.admitted = admitted; self.filter.admitted_at = Some(at); } @@ -6894,13 +7255,7 @@ impl Cursor { drops.pending.remove(range); } } - let admitted: Vec<_> = self - .filter - .admitted - .overlapping(&run) - .map(|admitted| (*admitted.start()).max(*run.start())..=(*admitted.end()).min(*run.end())) - .collect(); - for owed in admitted { + for owed in self.filter.admitted.overlapping(run.clone()) { self.filter.owed.insert(owed); } } @@ -7128,7 +7483,13 @@ impl Cursor { continue; } lookup.join = None; - moved |= self.filter.window.found(lookup.pts, answer, &*state); + // What the reader's fills gave up since its last sync wakes its walks at the next. + let holdings = ReaderHoldings { + state: &state, + misses: self.misses.as_deref(), + newly_gone: RangeSet::new(), + }; + moved |= self.filter.window.found(lookup.pts, answer, &holdings); } if moved { self.filter.revision += 1; @@ -7152,7 +7513,9 @@ impl Cursor { let (window, cap, revision) = (&self.filter.window, self.end_sequence, self.filter.revision); let misses = self.misses.get_or_insert_with(Box::default); let mut given_up = Vec::new(); - misses.retain((from, revision, cap), || below(window.missable(from), cap)); + misses.retain((from, revision, cap), window.missable_groups(), || { + below(window.missable(from), cap) + }); // One lock for the judgement and the fills it asks, since every subscriber of the // track probes through it. let (asks, fills) = { @@ -7160,6 +7523,7 @@ impl Cursor { let asks = misses.judge( &state, (from, revision, cap), + window.missable_groups(), window.named(), window.end(), |span| { @@ -7589,7 +7953,7 @@ impl Republish { return; }; drop(state); - let runs = Servable::new(window.runs(0)); + let runs = Servable::of(&window, &Servable::NONE); if let Ok(mut current) = published.write() && current.revision < revision { @@ -7952,7 +8316,7 @@ impl Subscriber { }; // What a widening owes, the walk has yet to take in, or lifting the local cap would: // a group past it is unread, not gone. - let owed = cursor.owed(&cursor.admitted(cursor.floor(), None)); + let owed = cursor.owed(&Admitted::new(&cursor.filter.window, cursor.floor(), None)); let last = unread.end.map(|end| end.checked_sub(1)); let owed = || { owed.iter().any(|run| { @@ -13494,6 +13858,152 @@ mod test { } } + /// A search over a layered servable window, through the reader's index of the live + /// groups and the layer's runs, finds what a scan of every live group would, in either + /// direction, as groups arrive, start, abort, leave the cache, or arrive again between + /// searches that keep the same index. Each window is searched on two tracks, as a + /// group's expiry that moved to a replacement copy searches its reader's. + #[test] + fn layered_searches_match_a_scan_of_every_live_group() { + // A small deterministic generator, so a failure replays. + let seed = std::cell::Cell::new(0x2545_f491_4f6c_dd1d_u64); + let next = |bound: u64| { + let mut x = seed.get(); + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + seed.set(x); + x % bound + }; + const SEQUENCES: u64 = 40; + let at = |millis| Timestamp::from_millis(millis).unwrap(); + for round in 0..300 { + let producers = [ + track_producer("test", Info::timed()), + track_producer("copy", Info::timed()), + ]; + let runs = |count: u64| -> Runs { + Runs::new((0..next(count + 1)).map(|_| { + let start = next(SEQUENCES); + start..=start + next(5) + })) + }; + let fixed = Arc::new(runs(8)); + let mut groups = [Vec::new(), Vec::new()]; + let mut servable = Servable::NONE; + for step in 0..30 { + let sequence = next(SEQUENCES); + let track = next(2) as usize; + let (producer, groups) = (&producers[track], &mut groups[track]); + match next(6) { + 0 | 1 => { + if let Ok(mut group) = producer.create_group(group::Info { sequence }) { + if next(3) > 0 { + group + .write_frame(at(sequence * 10), bytes::Bytes::from_static(b"x")) + .unwrap(); + } + groups.push(group); + } + } + 2 => { + if let Some(index) = groups.iter().position(|group| group.sequence == sequence) { + let _ = groups.swap_remove(index).abort(Error::Cancel); + } + } + 3 => { + let _ = producer.modify().unwrap().uncache(sequence); + } + _ => {} + } + // Now and then the window's walks move, keeping the fixed runs and their index. + if step == 0 || next(4) == 0 { + let cap = (next(4) == 0).then(|| next(SEQUENCES + 2)); + let layered = Layered::new(fixed.clone(), runs(2)); + servable = Servable::layered(layered, cap, &servable); + } + for (track, producer) in producers.iter().enumerate() { + let state = producer.state.read(); + let shown = |slot: &Slot| slot.is_shown().then_some(slot.group.sequence); + for from in [0, next(SEQUENCES), next(SEQUENCES)] { + for rev in [false, true] { + let scan = state.live(from..).filter(|(sequence, _)| servable.contains(**sequence)); + let expected = match rev { + true => scan.map(|(_, slot)| slot).rev().find_map(shown), + false => scan.map(|(_, slot)| slot).find_map(shown), + }; + let found = state.find_servable(&servable, from, rev, shown); + let shape = match &servable { + Servable::Run { start, end } => format!("run {start}..{end:?}"), + Servable::Runs(several) => format!( + "{:?} under {:?} below {:?}", + several.groups.fixed(), + several.groups.layer(), + several.cap + ), + }; + let cached: Vec<_> = state + .live(..) + .map(|(sequence, slot)| (*sequence, slot.is_shown())) + .collect(); + assert_eq!( + found, expected, + "round {round} step {step} track {track} from {from}, rev {rev}: {shape}, cached {cached:?}" + ); + } + } + } + } + } + } + + /// Once churn leaves the index holding more entries than the cache, a cache that then + /// shrinks to its newest group still lets the index shrink, though every search answers + /// at the newest entry and never meets the stale ones. + #[test] + fn the_live_index_shrinks_with_the_cache() { + let producer = track_producer("test", Info::timed()); + let groups: Vec<_> = (0..100) + .map(|sequence| producer.create_group(group::Info { sequence }).unwrap()) + .collect(); + let fixed = Arc::new(Runs::new([0..=999, 1001..=u64::MAX])); + let servable = Servable::layered(Layered::new(fixed, Runs::default()), None, &Servable::NONE); + let Servable::Runs(several) = &servable else { + panic!("several runs"); + }; + let entries = || -> usize { + let indexes = several.live.0.lock().unwrap(); + indexes.iter().map(|indexed| indexed.groups.len()).sum() + }; + let newest = |slot: &Slot| Some(slot.group.sequence); + assert_eq!( + producer.state.read().find_servable(&servable, 0, true, newest), + Some(99) + ); + assert_eq!(entries(), 100); + + // Most of the cache leaves, so a prune keeps only the groups still cached. + for sequence in 0..60 { + producer.modify().unwrap().uncache(sequence).unwrap(); + } + assert_eq!( + producer.state.read().find_servable(&servable, 0, true, newest), + Some(99) + ); + assert_eq!(entries(), 40); + + // Then all but the newest leaves. + for sequence in 60..99 { + producer.modify().unwrap().uncache(sequence).unwrap(); + } + assert_eq!( + producer.state.read().find_servable(&servable, 0, true, newest), + Some(99) + ); + assert_eq!(entries(), 1, "the stale entries are pruned"); + drop(groups); + } + /// Capping a servable window in place keeps exactly the groups the set of its runs /// keeps below the cap, so the single run stays off the heap without moving a bound. #[test] @@ -13518,22 +14028,50 @@ mod test { Some(13), Some(u64::MAX), ]; + let points = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, u64::MAX - 1, u64::MAX]; for bounds in windows { let runs = || bounds.iter().map(|&(first, last)| first..=last); - let servable = Servable::new(runs()); - if let [(first, last)] = bounds { - assert!( - servable - == Servable::Run { - start: *first, - end: last.checked_add(1) - }, - "{bounds:?}" - ); - } - for cap in caps { - let expected = Servable::new(below(runs(), cap).into_iter()); - assert!(servable.below(cap) == expected, "{bounds:?} below {cap:?}"); + // The runs fixed, and again split between fixed runs and a layer over them. + let layered = [ + Layered::new(Arc::new(Runs::new(runs())), Runs::default()), + Layered::new(Arc::new(Runs::new(runs().skip(1))), Runs::new(runs().take(1))), + ]; + for groups in layered { + let servable = Servable::layered(groups, None, &Servable::NONE); + if let [(first, last)] = bounds { + assert!( + servable + == Servable::Run { + start: *first, + end: last.checked_add(1) + }, + "{bounds:?}" + ); + } + for cap in caps { + let expected = below(runs(), cap); + let capped = servable.below(cap); + for point in points { + assert_eq!( + capped.contains(point), + expected.contains(&point), + "{bounds:?} below {cap:?} at {point}" + ); + } + // One run below the cap stays off the heap. + if let Some(run) = expected.iter().next() + && expected.len() == 1 + { + assert!( + capped + == Servable::Run { + start: *run.start(), + end: run.end().checked_add(1) + }, + "{bounds:?} below {cap:?}" + ); + } + } } } } @@ -19900,6 +20438,52 @@ mod test { assert_eq!(read, [5, 6], "groups 7 and 8 start past the end"); } + /// A time end parked on a gap the history may still fill moves on once the reader's own + /// fill refuses it: what its fills give up counts gone for its walks, which wake on it. + #[test] + #[cfg(feature = "async")] + fn a_time_end_parked_on_a_gap_wakes_when_its_fill_refuses_it() { + let producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + let at = |millis| Timestamp::from_millis(millis).unwrap(); + for sequence in 0..7 { + write_group(&producer, sequence); + } + for sequence in [3, 4] { + evict(&producer, sequence); + } + let mut subscriber = producer.subscribe(replay().with_order(Order::Ascending).with_ranges([Range { + start: Anchor::Time(at(0)), + end: Some(Anchor::Time(at(4))), + }])); + // Polled as an executor would, again whenever the read wakes itself. + let (flag, waiter) = Flag::waiter(); + let mut poll = || loop { + match subscriber.poll_group(true, &waiter) { + Poll::Ready(res) => return Some(res.expect("no error").map(|group| group.sequence)), + Poll::Pending if flag.take() => continue, + Poll::Pending => return None, + } + }; + let (mut read, mut ended) = (Vec::new(), false); + loop { + while !ended && let Some(next) = poll() { + match next { + Some(sequence) => read.push(sequence), + None => ended = true, + } + } + let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&kio::Waiter::noop()) else { + break; + }; + while let Some(sequence) = fill.wanted() { + fill.group(sequence).reject(Error::App(sequence as u16)); + } + } + assert_eq!(read, [0, 1, 2]); + assert!(ended, "the end settles at group 5, past the refused 3 and 4"); + } + /// A wide fill refused group by group, each with its own error, holds only the answers its /// readers have yet to take, not one per refusal. #[test] @@ -21277,6 +21861,96 @@ mod test { assert_eq!(drain(&mut late), vec![4]); } + /// A time end whose cap moves beside fixed ranges owes, of every cached group, exactly + /// what was owed and is still admitted, and what is newly admitted: what the walks' runs + /// gain is taken in by their runs alone, and must still owe each cached group it reaches. + #[test] + fn a_moving_time_cap_owes_what_it_newly_admits() { + let seed = std::cell::Cell::new(0x2545_f491_4f6c_dd1d_u64); + let next = |bound: u64| { + let mut x = seed.get(); + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + seed.set(x); + x % bound + }; + let at = |millis| Timestamp::from_millis(millis).unwrap(); + const SEQUENCES: u64 = 40; + let mut owed = 0; + for round in 0..150 { + let producer = track_producer("test", Info::timed()); + // One time end, and fixed ranges around and above where the feed starts. + let ranges = std::iter::once(Range { + start: Anchor::Time(at(next(100))), + end: Some(Anchor::Time(at(200 + next(400)))), + }) + .chain((0..1 + next(6)).map(|_| { + let start = next(SEQUENCES); + Range::groups(start..start + 1 + next(3)) + })); + let mut subscriber = producer.subscribe(Subscription::default().with_ranges(ranges)); + let mut groups = Vec::new(); + for step in 0..30 { + let sequence = next(SEQUENCES); + match next(4) { + 0 | 1 => { + if let Ok(mut group) = producer.create_group(group::Info { sequence }) { + if next(2) == 0 { + group + .write_frame(at(sequence * 20), bytes::Bytes::from_static(b"x")) + .unwrap(); + } + groups.push(group); + } + } + 2 => { + if let Some(group) = groups.iter_mut().find(|group| group.sequence == sequence) + && group.handle().timestamp().is_none() + { + let _ = group.write_frame(at(sequence * 20), bytes::Bytes::from_static(b"x")); + } + } + _ => { + // Read now and then, so the walk passes groups and owes them again. + let _ = subscriber.poll_recv_group(&kio::Waiter::noop()); + } + } + let Inner::Plain(cursor) = &mut subscriber.inner else { + unreachable!("a local subscriber reads its own cursor"); + }; + cursor.sync(); + let (before, was, primed) = ( + cursor.filter.admitted.runs(), + cursor.filter.owed.clone(), + cursor.filter.admitted_at.is_some(), + ); + cursor.admit(); + let now = cursor.filter.admitted.runs(); + let held = + |runs: &[ops::RangeInclusive], sequence: u64| runs.iter().any(|run| run.contains(&sequence)); + let cached: Vec = cursor.state.read().lookup.keys().copied().collect(); + for sequence in cached { + let expected = + primed && held(&now, sequence) && (was.contains(&sequence) || !held(&before, sequence)); + assert_eq!( + cursor.filter.owed.contains(&sequence), + expected, + "round {round} step {step}: group {sequence}, admitted {before:?} then {now:?}, owed {was:?}" + ); + owed += usize::from(expected); + } + for run in cursor.filter.owed.iter() { + assert!( + cursor.filter.admitted.overlapping(run.clone()) == vec![run.clone()], + "round {round} step {step}: owed {run:?} beyond {now:?}" + ); + } + } + } + assert!(owed > 100, "only {owed} groups owed"); + } + /// A narrow reader under an open aggregate passes the groups outside its range holding /// nothing for them, however many arrive, and widening its ranges still serves the /// ones the cache holds in the new span, newest first and once each. @@ -21298,7 +21972,7 @@ mod test { unreachable!() }; assert!(cursor.filter.owed.is_empty(), "nothing owed for excluded groups"); - assert_eq!(cursor.filter.admitted.iter().count(), 1, "only the range is kept"); + assert_eq!(cursor.filter.admitted.runs().len(), 1, "only the range is kept"); narrow .update(replay().with_ranges([Range::groups(0..2), Range::groups(5..8)])) diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index 20b4e340f2f..2732671faa4 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -21,13 +21,14 @@ use std::{ collections::{BTreeMap, BTreeSet}, ops::{self, RangeInclusive}, sync::{Arc, Mutex, PoisonError}, - task::Poll, + task::{Poll, Wake, Waker}, time::Duration, }; use rangemap::{RangeInclusiveSet, RangeMap, RangeSet}; use super::super::delivered::Delivered; +use super::super::subscription::Layered; pub(super) use super::fill::Asking; use super::fill::{Ask, FillState, Join}; use super::{Drift, Order, Position, TrackState}; @@ -59,6 +60,12 @@ pub(super) struct Misses { /// The join holding each sequence: a reader's joins hold disjoint groups, so a change /// touches only the joins holding what it changes, never every fill the reader asked. holders: RangeMap, + /// What each join parks on, under its key: a waiter of its own, which records that its + /// fill answered before waking the reader, so [`Self::resolve`] looks only at the fills + /// that answered, never at every fill still out. + parked: BTreeMap, + /// The joins whose fills answered since the reader last looked. + answered: Arc, /// The key the next join takes. next_join: u64, /// What the reader asked its fills with last. @@ -69,8 +76,15 @@ pub(super) struct Misses { filled: BTreeSet, /// Never asked again: named dropped to the reader. given_up: RangeSet, - /// The window revision and cap what this holds was last clipped to. - clipped: Option<(u64, Option)>, + /// What became [`Self::gone`] since the cursor last took it, for its window's parked time + /// ends to wake on (see [`Self::take_gone`]). + newly_gone: RangeSet, + /// The window revision and cap what this holds was last clipped to, with the window's + /// missable groups then, `None` while every group was. + clipped: Option<(u64, Option, Option)>, + /// The window's missable groups the last probe judged against, `None` while every group + /// was, so a move of only its time ends' runs judges only what they moved over. + view: Option, /// Where the track leaves what it evicts that this reader judged cached, once it judged. inbox: Option, /// The track's eviction count this reader last emptied its inbox at. @@ -82,6 +96,45 @@ pub(super) struct Misses { /// A run of misses the reader gives up, for the drop cursor to name. pub(super) type GivenUp = Vec>; +/// The joins whose fills changed their answer, recorded by each join's own waiter before it +/// wakes the reader's task. +/// +/// The task is the one the reader last resolved for. A reader polled by another task since +/// still learns of every answer, since the joins' waiters stay registered and wake whichever +/// task the reader resolved for last. +#[derive(Default)] +struct Answered { + keys: Mutex>, + task: Mutex>, +} + +/// Wakes the reader once join `key`'s fill answers, recording which join it was. +struct JoinWake { + key: u64, + answered: Arc, +} + +impl Wake for JoinWake { + fn wake(self: Arc) { + self.wake_by_ref(); + } + + fn wake_by_ref(self: &Arc) { + // Recorded before the task wakes, so the resolve it triggers sees the change. + let answered = &self.answered; + answered + .keys + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(self.key); + // Woken with no lock held: an executor may poll inline. + let task = answered.task.lock().unwrap_or_else(PoisonError::into_inner).clone(); + if let Some(task) = task { + task.wake(); + } + } +} + /// The current deferred runs, indexed by sequence and by their next judgement deadline. /// Splits remove the old age entry before publishing either piece; no stale timers remain. #[derive(Default)] @@ -192,6 +245,25 @@ impl Misses { } } + /// The highest sequence still to come from a fill. + fn highest(&self) -> Option { + let wanted = self.wanted.iter().next_back().map(|run| run.end - 1); + wanted.max(self.filled.last().copied()) + } + + /// Forget `lost`, which the reader no longer reads: its fills no longer owe it those + /// groups. + fn lose(&mut self, lost: &RangeSet) { + for run in lost.iter() { + let filled: Vec = self.filled.range(run.clone()).copied().collect(); + for sequence in filled { + self.filled.remove(&sequence); + } + self.wanted.remove(run.clone()); + } + self.release(lost); + } + /// Whether a fill still owes this reader a group. pub fn is_waiting(&self) -> bool { !self.wanted.is_empty() @@ -221,6 +293,12 @@ impl Misses { .map(|run| run.end) } + /// What became [`Self::gone`] since the last call: a time end parked on a gap this + /// reader's own fills gave up on walks on once told. + pub fn take_gone(&mut self) -> RangeSet { + std::mem::take(&mut self.newly_gone) + } + /// What was refilled after an eviction and not handed out yet. #[cfg(all(test, feature = "async"))] pub fn refilled(&self) -> &RangeSet { @@ -277,6 +355,17 @@ impl Misses { self.next_join += 1; self.holders.insert(join.groups.clone(), key); self.joins.insert(key, join); + let wake = Arc::new(JoinWake { + key, + answered: self.answered.clone(), + }); + self.parked.insert(key, kio::Waiter::new(Waker::from(wake))); + // Looked at once, which parks it. + self.answered + .keys + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(key); } } @@ -293,6 +382,7 @@ impl Misses { join.release(&RangeSet::from_iter([piece])); if join.is_empty() { self.joins.remove(&key); + self.parked.remove(&key); } } } @@ -310,22 +400,33 @@ impl Misses { } /// Forget what the reader no longer reads: below `from`, or outside `admitted`, its window - /// at `revision` below the exclusive `cap`. Its fills no longer owe it those groups, and one - /// nobody waits on is withdrawn once its last join goes. + /// at `revision` below the exclusive `cap`, whose missable groups are `view`. Its fills no + /// longer owe it those groups, and one nobody waits on is withdrawn once its last join goes. /// /// Clips to the window only once it moves, since everything taken in since lies inside it, - /// so a poll under the same window costs only what falls below a raised floor. + /// so a poll under the same window costs only what falls below a raised floor. A move of + /// only the window's time ends costs what their runs no longer cover where this holds + /// anything, never every run of the window. pub fn retain( &mut self, (from, revision, cap): (u64, u64, Option), + view: Option<&Layered>, admitted: impl FnOnce() -> RangeInclusiveSet, ) { - let clipped = self.clipped.replace((revision, cap)); + let clipped = self.clipped.replace((revision, cap, view.cloned())); // A join holds only what is still wanted or filled. let Some(lowest) = self.lowest() else { return; }; - if clipped == Some((revision, cap)) { + let moved = match &clipped { + Some((clipped, at, _)) if (*clipped, *at) == (revision, cap) => Some(Vec::new()), + Some((_, at, Some(old))) if *at == cap => { + let highest = self.highest().unwrap_or(lowest); + view.and_then(|view| old.gained(view, from, highest)) + } + _ => None, + }; + if let Some(lost) = moved { if lowest < from { self.filled = self.filled.split_off(&from); self.wanted.remove(0..from); @@ -333,6 +434,13 @@ impl Misses { below.insert(0..from); self.release(&below); } + let lost: RangeSet = lost + .into_iter() + .map(|run| *run.start()..run.end().saturating_add(1)) + .collect(); + if !lost.is_empty() { + self.lose(&lost); + } return; } let admitted = &admitted(); @@ -355,8 +463,11 @@ impl Misses { self.release(&lost); } - /// Take in what the joined fills answered, parking `waiter` on the rest. Returns the + /// Take in what the joined fills answered, waking `waiter` once the rest do. Returns the /// runs to ask again: refused as too old for a budget tighter than this reader's. + /// + /// Looks only at the joins whose fills answered since the last call, and the joins new + /// since, so a reader waiting on thousands of fills costs the ones that answered. pub fn resolve( &mut self, state: &kio::Consumer, @@ -365,16 +476,17 @@ impl Misses { waiter: &kio::Waiter, ) -> RangeSet { let mut retry = RangeSet::new(); - // Only the joins holding something unanswered: one holding only filled groups until - // they are taken has nothing to answer. - let keys: BTreeSet = self - .wanted - .iter() - .flat_map(|run| self.holders.overlapping(run).map(|(_, key)| *key)) - .collect(); + { + let mut task = self.answered.task.lock().unwrap_or_else(PoisonError::into_inner); + if !task.as_ref().is_some_and(|task| task.will_wake(waiter.waker())) { + *task = Some(waiter.waker().clone()); + } + } + // Taken after the task is set: an answer landing since wakes the task set above. + let keys = std::mem::take(&mut *self.answered.keys.lock().unwrap_or_else(PoisonError::into_inner)); let mut keys = keys.into_iter().peekable(); while let Some(&key) = keys.peek() { - let Some(join) = self.joins.get(&key) else { + let (Some(join), Some(parked)) = (self.joins.get(&key), self.parked.get(&key)) else { keys.next(); continue; }; @@ -391,7 +503,9 @@ impl Misses { keys.next(); continue; } - let answers = match join.outcome().poll(waiter, |outcome| { + // Only what is unanswered: a join holding only filled groups until they are taken + // has nothing to answer. + let answers = match join.outcome().poll(parked, |outcome| { let answered = outcome.within(&wanted); match answered.is_empty() { true => Poll::Pending, @@ -471,6 +585,7 @@ impl Misses { &mut self, state: &TrackState, key: (u64, u64, Option), + view: Option<&Layered>, named: Option, bound: Option, mut target: impl FnMut(RangeInclusive) -> RangeInclusiveSet, @@ -527,8 +642,18 @@ impl Misses { state.cache.watch_wall(at, waiter); return asks; } + // A window that moved only its time ends' runs keeps its fixed runs, so only what those + // runs moved over is judged again below. + let layered = match (&self.view, view) { + (Some(old), Some(view)) if old.shares_fixed(view) => Some((old.clone(), view)), + _ => None, + }; let terms = self.at.is_none_or(|previous| { - previous.1 != at.1 || previous.2 != at.2 || previous.4 != at.4 || previous.5 != at.5 || at.0 < previous.0 + (previous.1 != at.1 && layered.is_none()) + || previous.2 != at.2 + || previous.4 != at.4 + || previous.5 != at.5 + || at.0 < previous.0 }) || self.budget != Some(drift.budget) || self.grace != Some(grace) || self.ended != ended; @@ -579,6 +704,34 @@ impl Misses { } self.deferred.remove(0..low); } + // What the time ends' runs no longer cover is forgotten, and what they newly cover + // judged, within what the probe judges: from the floor up to the newest group, + // below the cap. + let last = key + .2 + .map_or(Some(top), |cap| cap.checked_sub(1).map(|cap| cap.min(top))); + if let Some((old, view)) = &layered + && let Some(last) = last + && self.at.is_some_and(|previous| previous.1 != at.1) + { + let half = |run: RangeInclusive| *run.start()..run.end().saturating_add(1); + for run in old.gained(view, low, last).into_iter().flatten().map(half) { + for set in [ + &mut self.judged, + &mut self.refused, + &mut self.given_up, + &mut self.refilled, + ] { + set.remove(run.clone()); + } + self.deferred.remove(run); + } + for run in view.gained(old, low, last).into_iter().flatten().map(half) { + for fresh in self.judged.gaps(&run) { + candidates.insert(fresh); + } + } + } let first = self .at .and_then(|previous| previous.3) @@ -657,8 +810,18 @@ impl Misses { // Settled here rather than when read, since an ordered reader never looks at what a // fill answered: it reads past a group gone from the cache. Still asked, its fill // settles it; given up already, it was named. - for run in self.wanted.iter().chain(self.given_up.iter()) { - again.remove(run.clone()); + // Only what overlaps it, so a reader still asking thousands of fills pays nothing here. + let settled: Vec> = again + .iter() + .flat_map(|run| { + let wanted = self.wanted.overlapping(run); + wanted + .chain(self.given_up.overlapping(run)) + .map(move |settled| settled.start.max(run.start)..settled.end.min(run.end)) + }) + .collect(); + for run in settled { + again.remove(run); } for (run, refill) in refills .iter() @@ -685,6 +848,7 @@ impl Misses { self.grace = Some(grace); self.ended = ended; self.at = Some(at); + self.view = view.cloned(); for run in candidates.iter() { self.deferred.remove(run.clone()); self.judged.insert(run.clone()); @@ -768,31 +932,13 @@ impl Misses { /// as the reader skips a stale cached group, parking `waiter` on each run going stale. /// A fill answers on its handler's time, so an ordered reader would otherwise wait out /// a stalled one for good. + /// + /// Each stretch of a wanted run between the groups the live feed cached inside it is + /// judged whole, from its start. Every stretch starting below the first group the live + /// feed has after a stretch shares its successors, so one check judges all of them: + /// thousands of runs a fill still owes below the live edge cost one, not one each. pub fn expire(&mut self, state: &TrackState, drift: &Drift, given_up: &mut GivenUp, waiter: &kio::Waiter) { - let (mut stale, mut arrived) = (RangeSet::new(), RangeSet::new()); - for run in self.wanted.iter() { - // Each stretch between the groups cached inside the run shares one successor, so - // one check judges it whole. Groups arrive out of order, so a fresh late group - // keeps the stretch below it owed while the ones around it go stale. - let mut from = run.start; - // Only the live feed's arrivals split the run: an aborted group is a hole, and a - // backfill is a fill's answer, reaching the reader once that fill publishes it. - let cached = state - .lookup - .range(run.clone()) - .filter(|(_, slot)| slot.is_walked()) - .map(|(&sequence, _)| sequence); - for at in cached.chain(std::iter::once(run.end)) { - if from < at && state.poll_drifted(from, &drift.edge.servable, drift.budget, None, waiter) { - stale.insert(from..at); - } - // What arrived live is the feed's to serve, as the reader serves any cached group. - if at < run.end { - arrived.insert(at..at + 1); - } - from = at.saturating_add(1); - } - } + let (stale, arrived) = self.expired(state, drift, waiter); if stale.is_empty() && arrived.is_empty() { return; } @@ -809,6 +955,85 @@ impl Misses { } } + /// What [`Self::expire`] gives up as stale, and what arrived live, parking `waiter`. + fn expired(&self, state: &TrackState, drift: &Drift, waiter: &kio::Waiter) -> (RangeSet, RangeSet) { + let (mut stale, mut arrived) = (RangeSet::new(), RangeSet::new()); + let servable = &drift.edge.servable; + let mut next = self.wanted.iter().next().map(|run| run.start); + while let Some(from) = next { + let bound = state.arrival(from, servable).map(|slot| slot.group.sequence); + let drifted = state.poll_drifted(from, servable, drift.budget, None, waiter); + next = self.stretches(state, from, bound, drifted, &mut stale, &mut arrived); + } + (stale, arrived) + } + + /// Visit the stretches of the wanted runs starting at `from` and below `bound`, all judged + /// `drifted`, putting them in `stale` if so, and the groups the live feed cached inside a + /// run in `arrived`. Returns where the next stretch starts, at or past `bound`. + /// + /// Groups arrive out of order, so a fresh late group keeps the stretch below it owed while + /// the ones around it go stale. Only the live feed's arrivals split a run: an aborted + /// group is a hole, and a backfill is a fill's answer, reaching the reader once that fill + /// publishes it. A fresh stretch with no such group before the next live one is passed + /// in one step, however many runs lie between. + fn stretches( + &self, + state: &TrackState, + from: u64, + bound: Option, + drifted: bool, + stale: &mut RangeSet, + arrived: &mut RangeSet, + ) -> Option { + let walked = |lo: u64, hi: u64| { + state + .visible + .range(lo..hi) + .copied() + .find(|sequence| state.lookup.get(sequence).is_some_and(|slot| slot.serves_end())) + }; + // Where the visit is, and whether a stretch started below it is still running there. + let (mut at, mut within) = (from, false); + loop { + let run = self.wanted.overlapping(&(at..u64::MAX)).next()?.clone(); + if run.start > at { + (at, within) = (run.start, false); + } + if !within && bound.is_some_and(|bound| at >= bound) { + return Some(at); + } + match walked(at, run.end) { + // What arrived live is the feed's to serve, as the reader serves any cached group. + Some(group) => { + if drifted && at < group { + stale.insert(at..group); + } + arrived.insert(group..group + 1); + (at, within) = (group.checked_add(1)?, false); + } + None if drifted => { + stale.insert(at..run.end); + (at, within) = (run.end, false); + } + // The stretch began below the bound and ran past it: the next starts past it too. + None if bound.is_some_and(|bound| bound <= run.end) => (at, within) = (run.end, false), + None => { + // Fresh up to the next group the live feed holds, or the bound: every stretch + // starting below either is fresh too, holding no arrival. + let next = state.visible.range(run.end..).next().copied(); + let leap = match (next, bound) { + (Some(next), Some(bound)) => next.min(bound), + (next, bound) => next.or(bound)?, + }; + // A run holding the leap from below it is a stretch already running there. + within = self.wanted.get(&leap).is_some_and(|held| held.start < leap); + at = leap; + } + } + } + } + /// Ask the fills for `runs` in the reader's order, settling what no handler can take as /// refused. Fills queue in the order asked, so a descending reader's newest run comes first. pub fn ask( @@ -875,6 +1100,7 @@ impl Misses { } for run in left.iter() { self.refused.insert(run.clone()); + self.newly_gone.insert(run.clone()); self.deferred .insert(run.clone(), state.arrivals.folds_at(run.start, state.miss_grace())); } @@ -886,6 +1112,7 @@ impl Misses { return; } self.given_up.insert(groups.clone()); + self.newly_gone.insert(groups.clone()); given_up.push(groups); } } @@ -1124,6 +1351,7 @@ mod tests { (0, 0, None), None, None, + None, |span| [span].into_iter().collect(), &Delivered::default(), &drift, @@ -1165,6 +1393,7 @@ mod tests { (0, 0, None), None, None, + None, |_| panic!("the edge has not moved"), &Delivered::default(), &drift, @@ -1191,6 +1420,7 @@ mod tests { (0, 0, None), None, None, + None, |_| panic!("recreation extends no span"), &Delivered::default(), &drift, @@ -1263,6 +1493,7 @@ mod tests { (0, 0, None), None, None, + None, |span| [span].into_iter().collect(), &Delivered::default(), &drift, @@ -1298,6 +1529,7 @@ mod tests { (0, 0, None), None, None, + None, |_| panic!("the edge has not moved"), &Delivered::default(), &drift, @@ -1336,6 +1568,7 @@ mod tests { (0, 0, None), None, None, + None, |span| [span].into_iter().collect(), &Delivered::default(), &drift, @@ -1353,6 +1586,7 @@ mod tests { (0, 0, None), None, None, + None, |_| panic!("an unchanged probe rebuilds no span"), &Delivered::default(), &drift, @@ -1375,6 +1609,7 @@ mod tests { (0, 0, None), None, None, + None, |_| panic!("a late arrival extends no span"), &Delivered::default(), &drift, @@ -1393,6 +1628,7 @@ mod tests { (0, 0, None), None, None, + None, |_| panic!("a due gap extends no span"), &Delivered::default(), &drift, @@ -1436,6 +1672,7 @@ mod tests { (0, 0, None), None, None, + None, full, &Delivered::default(), &drift, @@ -1458,6 +1695,7 @@ mod tests { (0, 0, None), None, None, + None, |_| panic!("nothing deferred: the new grace waits for a change"), &Delivered::default(), &drift, @@ -1482,6 +1720,7 @@ mod tests { (0, 0, None), None, None, + None, full, &Delivered::default(), &drift, @@ -1498,6 +1737,331 @@ mod tests { ); } + /// Judging the stretches a fill still owes by their shared successor gives up exactly + /// what judging each stretch alone would, as groups arrive, start, abort, or leave the + /// cache in any order, on either clock, against a window of one run or several. + #[test] + fn expiring_by_successor_matches_each_stretch_judged_alone() { + use super::super::super::subscription::{Layered, Runs}; + use crate::Timestamp; + + /// The old judgement: each stretch between the live groups cached in a wanted run, by + /// its own start. + fn each(misses: &Misses, state: &TrackState, drift: &Drift) -> (RangeSet, RangeSet) { + let waiter = kio::Waiter::noop(); + let (mut stale, mut arrived) = (RangeSet::new(), RangeSet::new()); + for run in misses.wanted.iter() { + let mut from = run.start; + let cached = state + .lookup + .range(run.clone()) + .filter(|(_, slot)| slot.is_walked()) + .map(|(&sequence, _)| sequence); + for at in cached.chain(std::iter::once(run.end)) { + if from < at && state.poll_drifted(from, &drift.edge.servable, drift.budget, None, &waiter) { + stale.insert(from..at); + } + if at < run.end { + arrived.insert(at..at + 1); + } + from = at.saturating_add(1); + } + } + (stale, arrived) + } + + let seed = std::cell::Cell::new(0x2545_f491_4f6c_dd1d_u64); + let next = |bound: u64| { + let mut x = seed.get(); + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + seed.set(x); + x % bound + }; + const SEQUENCES: u64 = 32; + let at = |millis| Timestamp::from_millis(millis).unwrap(); + let mut judged = 0; + for round in 0..400 { + let broadcast = crate::broadcast::Info::default().produce(); + let track = broadcast + .create_track( + "expire", + crate::track::Info::default().with_timescale(crate::Timescale::MILLI), + ) + .unwrap(); + let mut groups = Vec::new(); + let budget = Duration::from_millis(50 + next(200)); + let runs = |count: u64| { + Runs::new((0..1 + next(count)).map(|_| { + let start = next(SEQUENCES); + start..=start + next(6) + })) + }; + let servable = match next(3) { + 0 => Servable::ALL, + 1 => Servable::Run { + start: next(8), + end: Some(SEQUENCES - next(8)), + }, + _ => Servable::layered( + Layered::new(Arc::new(runs(6)), runs(2)), + (next(3) == 0).then(|| next(SEQUENCES + 2)), + &Servable::NONE, + ), + }; + let mut misses = Misses::default(); + for _ in 0..1 + next(6) { + let start = next(SEQUENCES); + misses.wanted.insert(start..start + 1 + next(6)); + } + for step in 0..12 { + let sequence = next(SEQUENCES); + match next(6) { + 0..=2 => { + if let Ok(mut group) = track.create_group(crate::group::Info { sequence }) { + if next(4) > 0 { + group + .write_frame(at(sequence * 20), bytes::Bytes::from_static(b"x")) + .unwrap(); + } + groups.push(group); + } + } + 3 => { + if let Some(index) = groups.iter().position(|group| group.sequence == sequence) { + let _ = groups.swap_remove(index).abort(Error::Cancel); + } + } + 4 => { + let _ = track.modify().unwrap().uncache(sequence); + } + _ => broadcast.info().pool.step(Duration::from_millis(next(120))), + } + let state = track.state.read(); + let drift = Drift { + budget, + bounded: false, + edge: state.drift_edge(&servable), + }; + let expected = each(&misses, &state, &drift); + let found = misses.expired(&state, &drift, &kio::Waiter::noop()); + judged += expected.0.iter().count(); + assert_eq!( + ( + found.0.iter().cloned().collect::>(), + found.1.iter().cloned().collect::>() + ), + ( + expected.0.iter().cloned().collect::>(), + expected.1.iter().cloned().collect::>() + ), + "round {round} step {step}: wanted {:?}", + misses.wanted.iter().collect::>() + ); + } + } + assert!(judged > 100, "only {judged} stale runs judged"); + } + + /// A window whose time ends move only their runs over the same fixed runs is followed by + /// what those runs moved over, and ends up judging and holding exactly what judging the + /// whole window again would: the second reader below is handed an unshared copy of the + /// same fixed runs on each move, so it always judges the window whole. + #[kio::test] + async fn a_moved_layer_is_judged_as_the_whole_window_would_be() { + use super::super::super::subscription::{Layered, Runs}; + + let seed = std::cell::Cell::new(0x2545_f491_4f6c_dd1d_u64); + let next = |bound: u64| { + let mut x = seed.get(); + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + seed.set(x); + x % bound + }; + const SEQUENCES: u64 = 40; + let runs = |count: u64| { + Runs::new((0..next(count + 1)).map(|_| { + let start = next(SEQUENCES); + start..=start + next(5) + })) + }; + let snapshot = |misses: &Misses| { + let sets = [ + &misses.wanted, + &misses.judged, + &misses.refused, + &misses.given_up, + &misses.refilled, + ] + .map(|set| set.iter().cloned().collect::>()); + let deferred: Vec<_> = misses.deferred.overlapping(0..u64::MAX).collect(); + let filled: Vec<_> = misses.filled.iter().copied().collect(); + (sets, deferred, filled, misses.joins.len()) + }; + let mut asked = 0; + for round in 0..150 { + let broadcast = crate::broadcast::Info::default().produce(); + let mut track = broadcast.create_track("moved", crate::track::Info::default()).unwrap(); + let budget = Duration::from_secs(5); + let _reader = track.subscribe(Subscription::default().with_max_delay(budget)); + // A feed with holes, some of them under where it starts. + for sequence in 0..SEQUENCES { + if next(3) > 0 { + track + .create_group(crate::group::Info { sequence }) + .unwrap() + .finish() + .unwrap(); + } + } + if next(2) == 0 { + track.start_at(next(SEQUENCES / 2)).unwrap(); + } + let fills = track.state.read().fills.clone(); + fills.lock().add_handler(); + let fixed = Arc::new(runs(6)); + let cap = (next(3) == 0).then(|| next(SEQUENCES + 4)); + let asking = Asking { + priority: 0, + max_delay: Some(budget), + order: Order::Ascending, + }; + let (mut moved, mut whole) = (Misses::default(), Misses::default()); + let (mut given_moved, mut given_whole) = (Vec::new(), Vec::new()); + for revision in 0..8 { + let layer = runs(3); + let shared = Layered::new(fixed.clone(), layer.clone()); + let unshared = Layered::new(Arc::new((*fixed).clone()), layer); + let key = (0, revision, cap); + for (misses, view, given_up) in [ + (&mut moved, &shared, &mut given_moved), + (&mut whole, &unshared, &mut given_whole), + ] { + let target = |span: RangeInclusive| -> RangeInclusiveSet { + let end = cap.map_or(*span.end(), |cap| (*span.end()).min(cap.saturating_sub(1))); + if cap == Some(0) || *span.start() > end { + return RangeInclusiveSet::new(); + } + view.runs(*span.start()) + .take_while(|run| *run.start() <= end) + .map(|run| *run.start()..=(*run.end()).min(end)) + .collect() + }; + misses.retain(key, Some(view), || target(0..=u64::MAX - 1)); + let state = track.state.read(); + let drift = Drift { + budget, + bounded: false, + edge: state.drift_edge(&Servable::ALL), + }; + let asks = misses.judge( + &state, + key, + Some(view), + None, + None, + target, + &Delivered::default(), + &drift, + given_up, + &kio::Waiter::noop(), + ); + drop(state); + asked += asks.iter().count(); + misses.ask(&track.state.consume(), &fills, &asks, &asking, given_up); + } + assert_eq!( + snapshot(&moved), + snapshot(&whole), + "round {round} revision {revision}: {fixed:?} under {:?}", + shared_layer(&moved) + ); + assert_eq!(given_moved, given_whole, "round {round} revision {revision}"); + } + } + assert!(asked > 100, "only {asked} runs asked"); + + fn shared_layer(misses: &Misses) -> Option { + misses.view.as_ref().map(|view| view.layer().clone()) + } + } + + /// A reader waiting on many fills looks only at the ones that answered, yet each still + /// wakes it, and wakes the task that read it last: a reader read by another task since + /// parked learns of the answer too. + #[kio::test] + async fn an_answered_fill_wakes_the_task_that_read_last() { + use std::sync::atomic::{AtomicUsize, Ordering}; + + #[derive(Default)] + struct Count(AtomicUsize); + impl Wake for Count { + fn wake(self: Arc) { + self.0.fetch_add(1, Ordering::Relaxed); + } + } + + let broadcast = crate::broadcast::Info::default().produce(); + let mut track = broadcast.create_track("fills", crate::track::Info::default()).unwrap(); + let requests = track.requests(); + track.start_at(8).unwrap(); + track + .create_group(crate::group::Info { sequence: 8 }) + .unwrap() + .finish() + .unwrap(); + // Four groups below where the feed starts, each a fill of its own. + let ranges = [0..1, 2..3, 4..5, 6..7].map(crate::track::Range::groups); + let mut reader = track.subscribe( + Subscription::default() + .with_max_delay(Duration::from_secs(60)) + .with_ranges(ranges.into_iter().chain([crate::track::Range::groups(8..)])), + ); + let (first, second) = (Arc::new(Count::default()), Arc::new(Count::default())); + let waiter = |count: &Arc| kio::Waiter::new(Waker::from(count.clone())); + let first_waiter = waiter(&first); + while let Poll::Ready(Ok(Some(_))) = reader.poll_recv_group(&first_waiter) {} + let second_waiter = waiter(&second); + assert!(reader.poll_recv_group(&second_waiter).is_pending()); + let super::super::Inner::Plain(cursor) = &reader.inner else { + unreachable!("a local subscriber reads its own cursor"); + }; + let misses = cursor.misses.as_ref().expect("the reader asked fills"); + assert_eq!(misses.joins.len(), 4, "one join per fill"); + assert!( + misses.answered.keys.lock().unwrap().is_empty(), + "every join was looked at once and parked" + ); + + // One fill answers: only its join is looked at again, and the later task wakes. + let fill = std::future::poll_fn(|cx| requests.poll_requested_fill(&kio::Waiter::new(cx.waker().clone()))) + .await + .unwrap(); + let woken = second.0.load(Ordering::Relaxed); + fill.drop_groups(fill.start().group..fill.end()); + assert!( + second.0.load(Ordering::Relaxed) > woken, + "the task that read last wakes" + ); + let super::super::Inner::Plain(cursor) = &reader.inner else { + unreachable!("a local subscriber reads its own cursor"); + }; + let answered = cursor.misses.as_ref().unwrap().answered.keys.lock().unwrap().len(); + assert_eq!(answered, 1, "only the join that answered is looked at again"); + let _ = reader.poll_recv_group(&second_waiter); + let super::super::Inner::Plain(cursor) = &reader.inner else { + unreachable!("a local subscriber reads its own cursor"); + }; + assert_eq!( + cursor.misses.as_ref().unwrap().joins.len(), + 3, + "the answered fill's join is gone" + ); + } + #[test] fn deferred_splits_and_replacements_retire_the_old_age_entries() { let mut deferred = Deferred::default();