diff --git a/quest/m1/release-perf/README.md b/quest/m1/release-perf/README.md index c265a7c08c1..1f974bc788d 100644 --- a/quest/m1/release-perf/README.md +++ b/quest/m1/release-perf/README.md @@ -1,4 +1,4 @@ -# Main is no slower than release +# [S] Main is no slower than release ## Goal @@ -266,11 +266,26 @@ Decided 2026-10-30 (maintainer), in - The sparse rows are accepted as that per-group model cost. - The fanout mode is accidental and is fixed before the cut, in - [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md). + [moq-dev/moq#7493](https://github.com/moq-dev/moq/pull/7493). + +That fix cut the shared track lock from about 155 acquisitions per delivered +group to 34 (release takes 31), counted on one worker. Most were the +logical-track reader re-reading every copy's end, the retention edge, and a +probe's fills on each of several reads per wake, plus a whole track walk on +every group machine's wake. Its own pairing, 14 alternating rounds at load 7 +to 28, found no high-mode round on either side: relay CPU 0.066 to 0.090 cores +against release's 0.062 to 0.104, with mixed-sign paired deltas inside +release's own spread, and 5 to 36% fewer user-space instructions than release +in every round. The re-pair above, on a quieter host, is what remains. + +Decided 2026-11-01 (maintainer), in +[moq-dev/moq#7493](https://github.com/moq-dev/moq/pull/7493): ✅ the new +`session_fanout_threads` Criterion case joins the release pairing through a +bench-only backport to `release`, which carries it once that PR lands. Bench-only +backports are allowed from now on. ## Required -- [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md) - relay fanout subscribers stop contending one track lock, so main's relay CPU matches release's in every round - [Relay CPU per frame since #5926](/quest/m1/release-perf/relay-cpu-bisect.md) - the 18-56% rise in relay instructions per delivered frame is bisected, attributed, and its avoidable cost fixed ## Related diff --git a/quest/m1/release-perf/fanout-lock.md b/quest/m1/release-perf/fanout-lock.md deleted file mode 100644 index 4c22fcc1b05..00000000000 --- a/quest/m1/release-perf/fanout-lock.md +++ /dev/null @@ -1,61 +0,0 @@ -# [M] Relay fanout subscribers stop contending one track lock - -## Goal - -The relay's `fanout` workload (`bench/relay.sh`, 64 subscribers of one track) -uses no more relay CPU on `main` than on `release` in every alternating -round, with the same delivery. Subscribers of one track no longer serialize -on the track's shared `kio::Lock` for every frame, so spreading their tasks -across Tokio workers no longer multiplies the relay's CPU. - -## Plan - -Found by the 2026-10-30 re-run in the -[release-perf epic](/quest/m1/release-perf/README.md) -([moq-dev/moq#7299](https://github.com/moq-dev/moq/pull/7299)), `release` -`de0a2b0534` against `main` `7c8e255304`. Decided 2026-10-30 (maintainer): -this is accidental cost, fixed before the cut. - -What the profile found: - -- `main`'s relay is bimodal under `fanout`. Release used 0.062 to 0.082 - cores in all seven rounds; main used 0.062 to 0.072 in three and 0.157 to - 0.241 in four, at the same 640 frames/s and p99. -- In the high mode 21 to 24 Tokio workers are busy and the relay makes 9,400 - to 12,600 voluntary context switches/s, against 2 to 6 workers and 400 to - 1,300 switches/s in the low mode. Release never exceeded 4 busy workers or - 2,800 switches/s. Eight CPU-profiled runs of `main` (frame pointers, the - default shared runtime) reproduced the high mode in five. -- In a high run 54% of samples end in a syscall (3 of 227 in a low run), - mostly futex waits on the track's shared `kio::Lock` and the wakes after - them. Every subscriber's read takes it, in the lite publisher's - `poll_recv_next` → `poll_group` / `poll_peek_group` and `poll_forward_drop` - → `poll_drop` (`rs/moq-net/src/lite/publisher.rs`), which reach the track - state through `rs/moq-net/src/model/resume.rs` and - `rs/moq-net/src/model/track.rs`. -- Packed onto a few workers, the 64 tasks never contend. Spread over many, - each frame wakes them all onto one mutex. - -Approach: - -- First find what `release`'s read path did differently, since release ran in - the same rounds and never spread. Compare how many times one subscriber's - wake takes the track lock on each branch. -- Then cut the per-wake lock work: take the lock once per wake rather than - once per poll step, or keep per-reader state outside the producer's mutex - so a reader only locks when a new group or drop is published. -- Add a Criterion case that reproduces the contention without the relay - harness (many readers of one track on a multi-threaded runtime), and keep - it in the release pairing. -- Verify with seven alternating `fanout` rounds against `origin/release`, - recording `/proc/loadavg`, busy workers, and voluntary context switches - per run; a quiet host is preferred. `release` has no `listen.tcp.tls`, so a - `just bench origin/release` pairing passes `"quic ws wss tcp"` (or just - `quic`). `just bench-profile` cannot yet run the default shared runtime, - where the mode shows up. - -Public API: none expected. Wire: none. - -## Related - -- [Log lock profile](/quest/m1/unbounded-groups/log-lock-profile.md) - lock contention profiling of `GroupState`, not the track lock diff --git a/quest/m1/release-perf/relay-cpu-bisect.md b/quest/m1/release-perf/relay-cpu-bisect.md index b206f31d9c9..96ce6aa7863 100644 --- a/quest/m1/release-perf/relay-cpu-bisect.md +++ b/quest/m1/release-perf/relay-cpu-bisect.md @@ -23,7 +23,7 @@ Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-a - Bisect `cdff3c8f95..main` on chat over io_uring first (the largest rise and the tightest ranges), with a step size that keeps each build's cost bounded, then confirm the culprits on video and on the tokio workers. A rise that shows on io_uring but not on the tokio workers is attributed on its own, not folded into a shared culprit. - Attribute each step with `perf record` at a fixed period, as #7505 did, before deciding fix or accept. Record accepted costs in the epic's Plan with the feature they pay for. - Commit the driver this needs as a deterministic mode of `just bench-profile` or `just bench-runtime` (pinned cores, instructions per delivered frame, fixed-period samples) instead of another throwaway script; #7505's driver on `bench/relay.sh` was not committed. -- [Fanout lock contention](/quest/m1/release-perf/fanout-lock.md) covers the bimodal futex cost on the fanout workload; don't fix it twice. A bisect step that lands on the same lock goes there. +- [moq-dev/moq#7493](https://github.com/moq-dev/moq/pull/7493) fixed the bimodal futex cost on the fanout workload; don't fix it twice. A bisect step that lands on the same lock starts from that PR's lock counts. Public API: none expected. Wire: none. diff --git a/rs/moq-net/Cargo.toml b/rs/moq-net/Cargo.toml index 405dc55c83e..1868b1dbbd3 100644 --- a/rs/moq-net/Cargo.toml +++ b/rs/moq-net/Cargo.toml @@ -50,12 +50,13 @@ futures = { workspace = true } kio = { path = "../kio", features = ["runtime", "sim"] } proptest = { workspace = true } serde_json = { workspace = true } -# The benches run on tokio. test-util (tokio::time::pause/advance) is test-only, so -# it must not leak into the normal dependency feature set. -tokio = { workspace = true, features = ["rt", "test-util", "time"] } +# The benches run on tokio, the session fanout bench across worker threads. test-util +# (tokio::time::pause/advance) is test-only, so it must not leak into the normal +# dependency feature set. +tokio = { workspace = true, features = ["rt", "rt-multi-thread", "test-util", "time"] } tracing-test = { workspace = true } -# Optional per-thread CPU timing in the session benchmark. +# Thread and process CPU timing in the session benchmark. [target.'cfg(unix)'.dev-dependencies] libc = { workspace = true } diff --git a/rs/moq-net/benches/session.rs b/rs/moq-net/benches/session.rs index 304cc1a23d3..fdb194ff8d2 100644 --- a/rs/moq-net/benches/session.rs +++ b/rs/moq-net/benches/session.rs @@ -13,6 +13,7 @@ criterion_group!( session::fetch_range::, session::range_updates::, session::withdrawal::, + session::fanout_threads::, session::idle ); criterion_main!(benches); diff --git a/rs/moq-net/benches/session/mod.rs b/rs/moq-net/benches/session/mod.rs index 2032db72277..43faa4665e1 100644 --- a/rs/moq-net/benches/session/mod.rs +++ b/rs/moq-net/benches/session/mod.rs @@ -48,7 +48,13 @@ //! holds others open, swept over sessions and requests held per session, so a slope //! here is per-request bookkeeping growing with the sessions or the requests. //! -//! Everything runs on one current-thread runtime, so the only work measured is +//! `session_fanout_threads` serves one broadcast's one-frame groups through one relay to 64 +//! viewers on multi-threaded runtimes, timed in process CPU. Each group wakes every +//! subscription's serve task at once, so a read path that makes them take one shared lock +//! many times pays for it in lock waits and wakes across workers: the `workers=16` rows +//! climb above the `workers=1` ones. +//! +//! Everything else runs on one current-thread runtime, so the only work measured is //! protocol and relay work, never scheduling across threads. //! //! Run with `cargo bench -p moq-net --features fuzz --bench session`. @@ -283,40 +289,64 @@ fn runtime() -> tokio::runtime::Runtime { .unwrap() } -/// One round's clock; the CPU variant excludes scheduling delays from other processes. +/// A multi-threaded runtime with `workers` threads, for the scenarios timed in process CPU. +fn threaded(workers: usize) -> tokio::runtime::Runtime { + tokio::runtime::Builder::new_multi_thread() + .worker_threads(workers) + .enable_time() + .build() + .unwrap() +} + +/// What a round is timed in. +#[derive(Clone, Copy)] +enum Clock { + Wall, + /// This thread's CPU time, excluding scheduling delays from other processes. + Thread, + /// Every thread's CPU time, for a multi-threaded runtime: lock waits and wakes across + /// its workers count too. + Process, +} + +/// One round's clock. enum MeasurementClock { Wall(Instant), #[cfg(unix)] - Cpu(Duration), + Cpu(libc::clockid_t, Duration), } impl MeasurementClock { - fn start(cpu: bool) -> Self { - if cpu { - #[cfg(unix)] - return Self::Cpu(Self::cpu()); - #[cfg(not(unix))] - panic!("SESSION_CPU requires Unix thread CPU clocks"); + fn start(clock: Clock) -> Self { + #[cfg(unix)] + let id = match clock { + Clock::Wall => return Self::Wall(Instant::now()), + Clock::Thread => libc::CLOCK_THREAD_CPUTIME_ID, + Clock::Process => libc::CLOCK_PROCESS_CPUTIME_ID, + }; + #[cfg(unix)] + return Self::Cpu(id, Self::cpu(id)); + #[cfg(not(unix))] + match clock { + Clock::Wall => Self::Wall(Instant::now()), + _ => panic!("CPU-timed rounds require Unix CPU clocks"), } - Self::Wall(Instant::now()) } fn elapsed(self) -> Duration { match self { Self::Wall(start) => start.elapsed(), #[cfg(unix)] - Self::Cpu(start) => Self::cpu() - .checked_sub(start) - .expect("thread CPU clock moved backwards"), + Self::Cpu(id, start) => Self::cpu(id).checked_sub(start).expect("CPU clock moved backwards"), } } #[cfg(unix)] - fn cpu() -> Duration { + fn cpu(id: libc::clockid_t) -> Duration { let mut time = libc::timespec { tv_sec: 0, tv_nsec: 0 }; // SAFETY: `time` is a valid, initialized output pointer for this supported clock. - let result = unsafe { libc::clock_gettime(libc::CLOCK_THREAD_CPUTIME_ID, &mut time) }; - assert_eq!(result, 0, "thread CPU clock: {}", std::io::Error::last_os_error()); + let result = unsafe { libc::clock_gettime(id, &mut time) }; + assert_eq!(result, 0, "CPU clock: {}", std::io::Error::last_os_error()); Duration::new( u64::try_from(time.tv_sec).unwrap(), u32::try_from(time.tv_nsec).unwrap(), @@ -600,7 +630,7 @@ struct Room { /// The group each viewer's subscriber is reading, when paced, flattened in viewer order. reading: Vec>, cache_now: moq_net::time::Instant, - cpu: bool, + clock: Clock, } impl Room { @@ -667,7 +697,10 @@ impl Room { writing: Vec::new(), reading: Vec::new(), cache_now: moq_net::time::Instant::now(), - cpu: std::env::var_os("SESSION_CPU").is_some(), + clock: match std::env::var_os("SESSION_CPU") { + Some(_) => Clock::Thread, + None => Clock::Wall, + }, }; room.writing.resize_with(room.watched.len(), || None); room.reading.resize_with(shape.viewers * shape.watch, || None); @@ -848,7 +881,7 @@ impl Room { let mut next_report = Duration::from_secs(10); for _ in 0..iters { self.tick(); - let start = MeasurementClock::start(self.cpu); + let start = MeasurementClock::start(self.clock); self.round().await; elapsed += start.elapsed(); if report && began.elapsed() >= next_report { @@ -1415,6 +1448,42 @@ impl IdleRoom { } } +/// One relay serving one broadcast's one-frame groups to 64 viewers, as the relay load +/// harness's `fanout` workload does, on multi-threaded runtimes timed in process CPU; see +/// the module docs. +pub(crate) fn fanout_threads(c: &mut Criterion) { + // Process CPU is read through a Unix clock. + if !cfg!(unix) || std::env::var_os("SESSION_ALLOCS").is_some() || std::env::var_os("SESSION_EXPIRY_ONLY").is_some() + { + return; + } + let shape = Shape { + publishers: 1, + viewers: 64, + frame: 400, + group_frames: 1, + ..Shape::BASE + }; + let mut group = c.benchmark_group(format!("{}session_fanout_threads", B::PREFIX)); + // Groups delivered per round: one to each viewer. + group.throughput(Throughput::Elements(shape.viewers as u64)); + for workers in [1, 16] { + let rt = threaded(workers); + for version in ["moq-lite-06", "moq-lite-07-wip"] { + let mut room = None; + group.bench_function(format!("{version}/workers={workers}"), |b| { + let room = room.get_or_insert_with(|| { + let mut room = rt.block_on(Room::::new(version, shape)); + room.clock = Clock::Process; + room + }); + b.iter_custom(|iters| rt.block_on(room.measure(iters))) + }); + } + } + group.finish(); +} + pub(crate) fn idle(c: &mut Criterion) { let mut group = c.benchmark_group("session_idle"); for version in ["moq-lite-06", "moq-transport-22"] { diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 1393f68957c..15a8e94e377 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -23,6 +23,7 @@ use crate::{ priority::{Priority, PriorityHandle, PriorityQueue}, track::Order, transport::poll::{GroupOf, SendGroup as _, SendOptions}, + util::Idle, }; use super::Version; @@ -3990,6 +3991,8 @@ struct TrackRun { opening: Option, /// The groups with a stream, serving or awaiting the acknowledgement of their FIN. children: kio::Tasks, + /// The track side's last poll, if it parked; see [`Self::poll_step`]. + idle: Option>, } impl TrackRun { @@ -4029,6 +4032,7 @@ impl TrackRun { datagrams, opening: None, children: kio::Tasks::new(), + idle: None, } } @@ -4046,6 +4050,9 @@ impl TrackRun { /// what lets relays pause an upstream subscription across consumer churn without /// tearing it down. Lite-07 ends one whose every range is bounded once it is covered. fn update(&mut self, upd: lite::SubscribeUpdate) { + // The update moves the cursor and the bounds the track side judges against, which + // notifies nothing it parked on. + Idle::disarm(&mut self.idle); // Every update restates the wanted group. Past our largest it is a join, which // repeats the pending ask; at or below it the group already exists. if upd @@ -4170,6 +4177,8 @@ impl TrackRun { if let Some(opening) = &mut self.opening && let Poll::Ready(res) = opening.poll_open(&mut self.ctx.session, self.ctx.group.as_ref(), waiter) { + // The track side takes groups again. + Idle::disarm(&mut self.idle); let opening = self.opening.take().expect("polled above"); match res { Ok(()) => self.children.push(opening), @@ -4177,7 +4186,13 @@ impl TrackRun { } } // Drive the in-flight group machines; completions just retire. + let running = self.children.len(); let _ = self.children.poll(waiter); + // A retired machine releases its group, and the track side waits on the machines + // to finish or cover the subscription, neither of which the track notifies. + if self.children.len() < running { + Idle::disarm(&mut self.idle); + } // A group whose header does not fit this peer's varint wrote nothing. The // reset is on a stream the subscriber never tied to this subscription, so // forward it here or the reader waits forever. @@ -4193,6 +4208,25 @@ impl TrackRun { return Poll::Ready(Ok(ControlFlow::Continue(()))); } + // Most wakes are a group machine's frame or the writer's credit. Nothing the track + // side parked on fired then, so it would park again: skip it rather than lock the + // shared track state every subscriber of the track reads through. + if Idle::parked(&self.idle, (), waiter) { + return Poll::Pending; + } + let mut idle = Idle::start(&mut self.idle, waiter); + let polled = self.poll_track(writer, idle.waiter()); + idle.finish(&mut self.idle, (), polled.is_pending()); + polled + } + + /// The track side of [`Self::poll_step`]: forward drops, resolve the start, and take the + /// next group, datagram, or boundary. Every wait here parks on the track subscriber. + fn poll_track( + &mut self, + writer: &mut Writer, + waiter: &kio::Waiter, + ) -> Poll, Error>> { if self.poll_forward_drop(writer, waiter)? { return Poll::Ready(Ok(ControlFlow::Continue(()))); } diff --git a/rs/moq-net/src/model/cache.rs b/rs/moq-net/src/model/cache.rs index 4ba981694f1..a5524a702c8 100644 --- a/rs/moq-net/src/model/cache.rs +++ b/rs/moq-net/src/model/cache.rs @@ -43,7 +43,7 @@ use crate::util::SendCell; use std::collections::BTreeMap; use std::sync::Mutex; -use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; use std::sync::{Arc, OnceLock, PoisonError, Weak}; use std::task::{Poll, ready}; use std::time::Duration; @@ -714,6 +714,10 @@ pub(super) struct Track { // lock, and wakes only the readers following it. largest: kio::Shared, + // Whether the track ever declared an end, set under the track lock and never cleared, + // so a reader looking for one skips that lock until there is one to read. + declared: AtomicBool, + // What the frames this track's groups are still receiving may allocate up front, // shared with every track minted from the same origin handle. frames: super::frame::Budget, @@ -822,6 +826,7 @@ impl Track { wakes: Default::default(), aborts: OnceLock::new(), largest: Default::default(), + declared: AtomicBool::new(false), }); if let Some(key) = track.pool.register(&track) { let _ = track.sweep.set(key); @@ -849,6 +854,16 @@ impl Track { &self.wakes } + /// The track declared an end. Called under the track lock. + pub(super) fn declare_end(&self) { + self.declared.store(true, Ordering::Release); + } + + /// Whether the track ever declared an end, so the end may be worth the track lock. + pub(super) fn end_declared(&self) -> bool { + self.declared.load(Ordering::Acquire) + } + /// Watch every group the publisher cuts, rather than chosen sequences. pub(super) fn watch_cuts(&self) -> aborts::Watch { self.aborts.get_or_init(Arc::default).watch_cuts() diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 5fbc5137807..5e4e3b8ffb0 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -1200,10 +1200,7 @@ impl Reader { _ => None, }, None => match &mut copy.sub { - Sub::Ready(sub) => match sub.poll_finished(&kio::Waiter::noop()) { - Poll::Ready(Ok(end)) => Some(end), - _ => None, - }, + Sub::Ready(sub) => sub.declared_end(), Sub::Pending(_) => copy.track.declared_end(), }, }; @@ -1268,11 +1265,17 @@ impl Reader { self.learn_end(None); self.window(); self.out.retain(|(_, out)| out.strong_count() > 0); + // Only a replaced copy is judged against the edge, which costs the serving track's + // lock: a reader following one route never pays it. + let edge = match self.copies.iter().any(|copy| copy.replaced) { + true => self.copies.last().and_then(|copy| copy.track.latest_sequence()), + false => None, + }; let reader = Retain { groups: self.groups, window: &self.window, groups_read: self.groups_read, - edge: self.copies.last().and_then(|copy| copy.track.latest_sequence()), + edge, mirrored: &self.mirrored, delivered, out: &self.out, diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 81315adec95..441d1df08b2 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -2542,6 +2542,7 @@ impl TrackState { } self.end = Some(end); self.end_error = error; + self.cache.declare_end(); self.cache.wakes().arrived(fin..); Ok(()) } @@ -6495,6 +6496,11 @@ impl Cursor { let revision = self.subscription.ranges_revision(); (revision != self.filter.resolved).then(|| (revision, preferences.ranges.clone())) }; + // A settled window has nothing to read from the track, whose lock every subscriber + // of it takes. + if replaced.is_none() && self.filter.window.is_settled() { + return false; + } let state = self.state.read(); let moved = match replaced { Some((revision, ranges)) => { @@ -6857,13 +6863,14 @@ impl Cursor { order: subscription.order, } }; - let fills = self.state.read().fills.clone(); let misses = self.misses.get_or_insert_with(Box::default); let mut given_up = Vec::new(); misses.retain((from, revision, cap), || below(window.missable(from), cap)); - let asks = { + // One lock for the judgement and the fills it asks, since every subscriber of the + // track probes through it. + let (asks, fills) = { let state = self.state.read(); - misses.judge( + let asks = misses.judge( &state, (from, revision, cap), window.named(), @@ -6882,7 +6889,8 @@ impl Cursor { drift, &mut given_up, waiter, - ) + ); + (asks, state.fills.clone()) }; misses.ask(&self.state, &fills, &asks, &asking, &mut given_up); loop { @@ -6892,7 +6900,10 @@ impl Cursor { } misses.ask(&self.state, &fills, &retry, &asking, &mut given_up); } - misses.expire(&self.state.read(), drift, &mut given_up, waiter); + // Only what a fill still owes can expire. + if misses.is_waiting() { + misses.expire(&self.state.read(), drift, &mut given_up, waiter); + } if let Some(drops) = &mut self.drops { for run in given_up { drops.give_up(run); @@ -6903,7 +6914,7 @@ impl Cursor { /// The next group a fill answered for this cursor in its order, cached and not handed /// out yet. fn next_filled(&mut self) -> Option { - let misses = self.misses.as_mut()?; + let misses = self.misses.as_mut().filter(|misses| misses.has_filled())?; let order = self.subscription.read().order; let mut given_up = Vec::new(); let group = misses.next_filled(&self.state.read(), order, &mut given_up); @@ -7913,6 +7924,21 @@ impl Subscriber { } } + /// The track's declared exclusive end, as [`Self::poll_finished`] reports it, without + /// registering a wake. A track that never declared one answers without its lock. + pub(super) fn declared_end(&mut self) -> Option { + match &mut self.inner { + Inner::Plain(cursor) => match cursor.cache.end_declared() { + true => cursor.state.read().end, + false => None, + }, + Inner::Resume(..) => match self.poll_finished(&kio::Waiter::noop()) { + Poll::Ready(Ok(end)) => Some(end), + _ => None, + }, + } + } + /// Poll for the track's declared exclusive end, without blocking. pub fn poll_finished(&mut self, waiter: &kio::Waiter) -> Poll> { match &mut self.inner { diff --git a/rs/moq-net/src/model/track/idle.rs b/rs/moq-net/src/model/track/idle.rs index 21c0e02fb42..05b05b5e020 100644 --- a/rs/moq-net/src/model/track/idle.rs +++ b/rs/moq-net/src/model/track/idle.rs @@ -3,20 +3,14 @@ //! A read that parks registers on everything that can change its answer: the track, the //! subscription, the fills it waits on, and the wall deadlines it judged against. Each of //! those notifies on every change, so until one fires, the answer stays the same. [`Idle`] -//! parks reads through a waiter of its own that records any wake before passing it on to -//! the task, and answers a read repeated before that wake with the same `Pending`, rather -//! than walking the whole read path again to reach it. +//! answers a read repeated before that wake with the same `Pending`, rather than walking +//! the whole read path again to reach it. //! -//! Only what notifies can be trusted this way. A change the reader makes to its own cursor, -//! such as moving its range, notifies nothing, so it disarms the gate itself. +//! A change the reader makes to its own cursor, such as moving its range, notifies nothing, +//! so it disarms the gate itself. -use std::{ - sync::{ - Arc, - atomic::{AtomicBool, Ordering}, - }, - task::{Wake, Waker}, -}; +/// A cursor's parked group read, and the waiter it parked on. +pub(super) type Idle = crate::util::Idle; /// Which read parked: a [`Cursor`](super::Cursor) parks the arrival and sequence walks /// separately, since one finding nothing says nothing about the other. @@ -35,122 +29,3 @@ impl Walk { } } } - -/// A cursor's parked read, and the waiter it parked on. -pub(super) struct Idle { - /// The read that parked, or `None` while nothing is parked. - parked: Option, - woken: Arc, - /// Wakes [`Self::woken`]. - wake: Waker, - /// Holds the waiter reads park on with [`Self::wake`]. Kept across reads, so parking - /// again on a list that still holds it is free, as it would be for the task's own - /// waiter, and retired as a task's own park retires one that parked on more lists - /// than it can record. - park: kio::Park, -} - -/// Records a wake, then wakes the task. -struct Woken { - woken: AtomicBool, - task: Waker, -} - -impl Wake for Woken { - fn wake(self: Arc) { - self.wake_by_ref(); - } - - fn wake_by_ref(self: &Arc) { - // Before the task's wake, so the read it triggers sees the change. - self.woken.store(true, Ordering::Release); - self.task.wake_by_ref(); - } -} - -impl Idle { - fn new(task: &Waker) -> Self { - let woken = Arc::new(Woken { - woken: AtomicBool::new(false), - task: task.clone(), - }); - Self { - parked: None, - wake: Waker::from(woken.clone()), - woken, - park: kio::Park::default(), - } - } - - /// Whether `walk`, read again for `task`, would park exactly as it last did: it parked, - /// nothing it parked on has fired since, and it still wakes the same task. - pub(super) fn parked(this: &Option>, walk: Walk, task: &kio::Waiter) -> bool { - this.as_ref().is_some_and(|idle| { - idle.parked == Some(walk) - && !idle.woken.woken.load(Ordering::Acquire) - && idle.woken.task.will_wake(task.waker()) - }) - } - - /// Start a full read for `task`, returning the gate holding the waiter it must park - /// on. A wake from here on counts against the read about to happen. - pub(super) fn start(this: &mut Option>, task: &kio::Waiter) -> Box { - match this.take() { - Some(mut idle) if idle.woken.task.will_wake(task.waker()) => { - idle.parked = None; - // Cleared before the read looks at anything, so a change it misses wakes - // the waiter after this store and leaves the next read a full one. - idle.woken.woken.store(false, Ordering::Release); - idle - } - // Another task: parking anew drops the old waiter, whose registrations go dead. - _ => Box::new(Self::new(task.waker())), - } - } - - /// The waiter a full read parks on. - pub(super) fn waiter(&mut self) -> &kio::Waiter { - self.park.hold(&std::task::Context::from_waker(&self.wake)) - } - - /// Put the gate back after a full read of `walk`, armed if it parked. - pub(super) fn finish(mut self: Box, this: &mut Option>, walk: Walk, pending: bool) { - self.parked = pending.then_some(walk); - *this = Some(self); - } - - /// Forget the parked read: the cursor moved in a way nothing notifies. - pub(super) fn disarm(this: &mut Option>) { - if let Some(idle) = this { - idle.parked = None; - } - } -} - -#[cfg(test)] -mod test { - use super::*; - - /// The waiter reads park on is kept across reads, but retired once it parked on more - /// lists than it records, so re-parking cannot stack entries those lists never collect. - #[test] - fn a_waiter_parked_past_its_records_is_retired() { - let task = kio::Waiter::noop(); - let mut slot = None; - for (lists, retired) in [(1, false), (9, true)] { - let mut idle = Idle::start(&mut slot, &task); - let mut parked: Vec<_> = (0..lists).map(|_| kio::WaiterList::new()).collect(); - let waiter = idle.waiter(); - for list in &mut parked { - waiter.register(list); - } - idle.finish(&mut slot, Walk::Arrival, true); - assert!(Idle::parked(&slot, Walk::Arrival, &task)); - - let mut idle = Idle::start(&mut slot, &task); - idle.waiter(); - assert_eq!(parked.iter().all(kio::WaiterList::is_empty), retired); - idle.finish(&mut slot, Walk::Arrival, false); - } - } -} diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index feb76ffd73f..0667edcfd7c 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -196,6 +196,11 @@ impl Misses { !self.wanted.is_empty() } + /// Whether a fill answered a group not handed out yet. + pub fn has_filled(&self) -> bool { + !self.filled.is_empty() + } + /// The lowest sequence asked and not answered yet. pub fn lowest_wanted(&self) -> Option { self.wanted.iter().next().map(|run| run.start) diff --git a/rs/moq-net/src/util.rs b/rs/moq-net/src/util.rs index 529cea34d19..438ce870ef5 100644 --- a/rs/moq-net/src/util.rs +++ b/rs/moq-net/src/util.rs @@ -3,10 +3,12 @@ use std::{collections::VecDeque, future::Future, pin::Pin, task::Poll}; mod cell; +mod idle; #[cfg(feature = "async")] mod later; mod wake; pub(crate) use cell::SendCell; +pub(crate) use idle::Idle; #[cfg(feature = "async")] pub(crate) use later::{Later, Learning}; pub(crate) use wake::QueueWake; diff --git a/rs/moq-net/src/util/idle.rs b/rs/moq-net/src/util/idle.rs new file mode 100644 index 00000000000..75f55c57558 --- /dev/null +++ b/rs/moq-net/src/util/idle.rs @@ -0,0 +1,191 @@ +//! An early exit for a read that parked and was not woken since. +//! +//! A read that parks registers on everything that can change its answer, and each of those +//! notifies on every change. [`Idle`] parks the read through a waiter of its own, which +//! records any wake before passing it on to the task, so until one fires a repeated read +//! can answer `Pending` again without walking, and locking, everything it read before. +//! +//! Only what notifies can be trusted this way. A change the owner makes itself, which +//! notifies nothing, has to [`Idle::disarm`] the gate. +//! +//! An owner with more than one read keys each by `K`, since one parking says nothing +//! about another. + +use std::{ + sync::{ + Arc, + atomic::{AtomicBool, Ordering}, + }, + task::{Context, Wake, Waker}, +}; + +/// A read's parked state, and the waiter it parked on. +pub(crate) struct Idle { + /// The read that last parked, while nothing has disarmed it since. + parked: Option, + woken: Arc, + /// Wakes [`Self::woken`]. + wake: Waker, + /// Holds the waiter reads park on with [`Self::wake`]. Kept across reads, so parking + /// again on a list that still holds it is free, as it would be for the task's own + /// waiter, and retired as a task's own park retires one that parked on more lists + /// than it can record. + park: kio::Park, +} + +/// Records a wake, then wakes the task. +struct Woken { + woken: AtomicBool, + task: Waker, +} + +impl Wake for Woken { + fn wake(self: Arc) { + self.wake_by_ref(); + } + + fn wake_by_ref(self: &Arc) { + // Before the task's wake, so the read it triggers sees the change. + self.woken.store(true, Ordering::Release); + self.task.wake_by_ref(); + } +} + +impl Idle { + fn new(task: &Waker) -> Box { + let woken = Arc::new(Woken { + woken: AtomicBool::new(false), + task: task.clone(), + }); + Box::new(Self { + parked: None, + wake: Waker::from(woken.clone()), + woken, + park: kio::Park::default(), + }) + } + + /// Whether `read`, again for `task`, would park exactly as it last did: it parked, + /// nothing it parked on has fired since, nothing disarmed it, and it still wakes the + /// same task. + pub(crate) fn parked(this: &Option>, read: K, task: &kio::Waiter) -> bool { + this.as_ref().is_some_and(|idle| { + idle.parked == Some(read) + && !idle.woken.woken.load(Ordering::Acquire) + && idle.woken.task.will_wake(task.waker()) + }) + } + + /// Start a full read for `task`, returning the gate holding the waiter it must park on. + /// A wake from here on counts against the read about to happen. + pub(crate) fn start(this: &mut Option>, task: &kio::Waiter) -> Box { + match this.take() { + Some(mut idle) if idle.woken.task.will_wake(task.waker()) => { + idle.parked = None; + // Cleared before the read looks at anything, so a change it misses wakes the + // waiter after this store and leaves the next read a full one. + idle.woken.woken.store(false, Ordering::Release); + idle + } + // Another task: parking anew drops the old waiter, whose registrations go dead. + _ => Self::new(task.waker()), + } + } + + /// The waiter a full read parks on. + pub(crate) fn waiter(&mut self) -> &kio::Waiter { + self.park.hold(&Context::from_waker(&self.wake)) + } + + /// Put the gate back after a full `read`, armed if it parked. + pub(crate) fn finish(mut self: Box, this: &mut Option>, read: K, pending: bool) { + self.parked = pending.then_some(read); + *this = Some(self); + } + + /// Forget the parked read: something it depends on changed without notifying. + pub(crate) fn disarm(this: &mut Option>) { + if let Some(idle) = this { + idle.parked = None; + } + } +} + +#[cfg(test)] +mod test { + use super::*; + + /// A parked read stays parked until a list it parked on wakes it, a read for another + /// task parks anew, and a disarmed one walks again. + #[test] + fn a_read_stays_parked_until_woken_or_disarmed() { + let task = kio::Waiter::noop(); + let mut slot: Option> = None; + assert!(!Idle::parked(&slot, (), &task), "nothing parked yet"); + + let mut list = kio::WaiterList::new(); + let mut idle = Idle::start(&mut slot, &task); + idle.waiter().register(&mut list); + idle.finish(&mut slot, (), true); + assert!(Idle::parked(&slot, (), &task)); + + list.wake(); + assert!(!Idle::parked(&slot, (), &task), "a wake reopens the read"); + + let mut idle = Idle::start(&mut slot, &task); + idle.waiter().register(&mut list); + idle.finish(&mut slot, (), true); + assert!(Idle::parked(&slot, (), &task)); + Idle::disarm(&mut slot); + assert!(!Idle::parked(&slot, (), &task), "a disarm reopens the read"); + + let idle = Idle::start(&mut slot, &task); + idle.finish(&mut slot, (), false); + assert!( + !Idle::parked(&slot, (), &task), + "a read that got something is not parked" + ); + + let other = kio::Waiter::new(Waker::from(Arc::new(Woken { + woken: AtomicBool::new(false), + task: Waker::noop().clone(), + }))); + let idle = Idle::start(&mut slot, &task); + idle.finish(&mut slot, (), true); + assert!(!Idle::parked(&slot, (), &other), "another task reads anew"); + } + + /// A gate keyed by read answers only for the read that parked. + #[test] + fn a_parked_read_says_nothing_about_another() { + let task = kio::Waiter::noop(); + let mut slot = None; + let idle = Idle::start(&mut slot, &task); + idle.finish(&mut slot, 1u8, true); + assert!(Idle::parked(&slot, 1, &task)); + assert!(!Idle::parked(&slot, 2, &task), "another read walks"); + } + + /// The waiter reads park on is kept across reads, but retired once it parked on more + /// lists than it records, so re-parking cannot stack entries those lists never collect. + #[test] + fn a_waiter_parked_past_its_records_is_retired() { + let task = kio::Waiter::noop(); + let mut slot: Option> = None; + for (lists, retired) in [(1, false), (9, true)] { + let mut idle = Idle::start(&mut slot, &task); + let mut parked: Vec<_> = (0..lists).map(|_| kio::WaiterList::new()).collect(); + let waiter = idle.waiter(); + for list in &mut parked { + waiter.register(list); + } + idle.finish(&mut slot, (), true); + assert!(Idle::parked(&slot, (), &task)); + + let mut idle = Idle::start(&mut slot, &task); + idle.waiter(); + assert_eq!(parked.iter().all(kio::WaiterList::is_empty), retired); + idle.finish(&mut slot, (), false); + } + } +}