diff --git a/quest/m0/nightly/README.md b/quest/m0/nightly/README.md index 4f4ffdee91c..292cf384c17 100644 --- a/quest/m0/nightly/README.md +++ b/quest/m0/nightly/README.md @@ -35,4 +35,5 @@ Public API: none. Wire: none. ## Required +- [The paired bench baseline builds](/quest/m0/nightly/bench-baseline-build.md) - the next scheduled Nightly's paired bench jobs build their baseline and compare, or the baseline moves outside the checkout - [The browser backend baseline completes on IETF22](/quest/m0/nightly/browser-baseline.md) - the WASM and UniFFI backend baseline completes every supported transport case without an internal error diff --git a/quest/m0/nightly/bench-baseline-build.md b/quest/m0/nightly/bench-baseline-build.md new file mode 100644 index 00000000000..a2e0914292d --- /dev/null +++ b/quest/m0/nightly/bench-baseline-build.md @@ -0,0 +1,35 @@ +# [XS] The paired bench baseline builds + +## Goal + +The scheduled Nightly's paired bench jobs build their baseline and get to the +comparison. Check with `gh run list --workflow nightly.yml --event schedule` +and the "Rust bench targets" step of the newest run (the other paired jobs, +such as "io_uring benchmarks", share the cause). Once a scheduled run gets past +the base build, delete this quest. + +## Plan + +Decided 2026-11-01 (maintainer, `/quest-plan`): a wait-and-check quest that +stays ready, since #7245 likely fixed it already and only the next scheduled +run can tell. + +Scheduled run +[41204380716](https://github.com/moq-dev/moq/actions/runs/41204380716) +(2026-10-31) failed every paired bench job before comparing: `bench/run.sh` +checks the baseline out to `$RUN/base`, under `target/bench/nightly/` inside +the main checkout, and Cargo could not merge that checkout's +`.cargo/config.toml` `build-ffi` alias with the current one ("failed to merge +key `build-ffi`"). + +That run's head already had +[moq-dev/moq#7245](https://github.com/moq-dev/moq/pull/7245), which makes the +aliases strings, so a nested checkout's alias overrides instead of merging. +Its baseline (`previous-nightly`) predated #7245 and still had the list form. +The next scheduled run's baseline includes #7245, so it should build. + +If it still fails on the merge, build the baseline outside the checkout (a +`git worktree` beside the repository instead of under `target/`), so no +enclosing `.cargo/config.toml` applies. + +Found by [moq-dev/moq#7253](https://github.com/moq-dev/moq/pull/7253). diff --git a/quest/m1/README.md b/quest/m1/README.md index 08c012b00a0..dc9f82fc224 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -163,7 +163,6 @@ removed). - [Rust releases publish through OIDC](/quest/m1/release-rs-oidc.md) - release-plz publishes with trusted publishing and `CARGO_REGISTRY_TOKEN` is revoked - [kio's clock tests pass under cargo test](/quest/m1/clock-test-process.md) - `paused_runtimes_own_their_anchors` stops depending on being first in its process - [The JS stalled-consumer eviction test ages its groups](/quest/m1/js-cache-age-test.md) - the test drives the clock the cache reads, so it can fail -- [An idle track poll exits cheaply](/quest/m1/idle-poll-exit.md) - a reader's poll on a track with nothing new costs a small constant, not ~3,800 instructions - [Main is no slower than release](/quest/m1/release-perf/README.md) - every benchmark where main is slower than release is fixed or accepted with a reason, checked by a final paired run before the cut - [Interop coverage](/quest/m1/interop-coverage/README.md) - the riskiest moq-transport features moq-net exposes get cross-implementation tests over drafts 18 and 22 - [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - A/A and deliberately slowed Nightly runs set the paired comparison's threshold, so its regression alerts can be trusted diff --git a/quest/m1/idle-poll-exit.md b/quest/m1/idle-poll-exit.md deleted file mode 100644 index ca32d748d05..00000000000 --- a/quest/m1/idle-poll-exit.md +++ /dev/null @@ -1,32 +0,0 @@ -# [M] An idle track poll exits cheaply - -## Goal - -A reader's poll on a track with nothing new costs a small constant number of -instructions, not about 3,800, so catalog and media fanout scale with the -readers that have work rather than every pending reader. - -## Plan - -Found 2026-10-28 by [moq-dev/moq#7140](https://github.com/moq-dev/moq/pull/7140), -which moved the arrival-grace and clock checks behind the cheap "nothing -changed" checks in `Misses::judge`. Even at the old baseline, a pending catalog -poll costs about 3,800 instructions per reader, spread across the cursor's -poll path (`Cursor::probe` in `rs/moq-net/src/model/track.rs` and what it -calls) on a track with nothing new. - -Decided 2026-10-28 (maintainer): m1, for relay fanout scale. - -- Profile one idle poll with instruction counts (callgrind or - `perf stat --no-scale`) and find where the cost goes. -- Add one early exit the whole path shares, keyed on something every change - already bumps (for example a track revision the cursor last saw), rather - than a short-cut per function. -- Measure the moq-mux catalog fanout benches and the moq-net session benches - over tracks and readers, before and after. - -Public API: none. Wire: none. - -## Related - -- [Mesh session cost](/quest/m2/perf/mesh-session-cost.md) - per-session costs in the same model diff --git a/quest/m2/perf/README.md b/quest/m2/perf/README.md index 23a8199d184..fcb695d93a5 100644 --- a/quest/m2/perf/README.md +++ b/quest/m2/perf/README.md @@ -92,6 +92,7 @@ revisions on the same host and worker count, and profiles with - [io_uring buffer pool memory](/quest/m2/perf/uring-pool-memory.md) - measure reserved versus resident pool memory before any shrink policy - [An idle export poll is cheap](/quest/m2/perf/export-idle-poll.md) - an export poll nothing woke skips the track fill and catalog work it repeats today - [Mesh session cost](/quest/m2/perf/mesh-session-cost.md) - moq-net's per-session announce, route sync, and scope work drops, swept over sessions and announces +- [Adapter reads reuse their waker](/quest/m2/perf/adapter-waker-alloc.md) - a measured verdict on the per-read waker allocation moq-tokio and moq-uring reads pay since #7042, fixed if real - [Lite request allocations](/quest/m2/perf/lite-request-allocs.md) - `Path::concat` allocates once and the lite request handler keeps one track-name copy - [Coverage before START](/quest/m2/perf/covered-before-start.md) - the lite-07 publisher's coverage check costs O(1) per wake before SUBSCRIBE_START - [io_uring demux churn](/quest/m2/perf/uring-demux-churn.md) - the demux snapshots stats and builds its fed list once per completion, not per datagram diff --git a/quest/m2/perf/adapter-waker-alloc.md b/quest/m2/perf/adapter-waker-alloc.md new file mode 100644 index 00000000000..e8a3ffce37f --- /dev/null +++ b/quest/m2/perf/adapter-waker-alloc.md @@ -0,0 +1,36 @@ +# [S] Adapter reads reuse their waker + +## Goal + +A measured verdict on what a parking read costs in the production adapters +(moq-tokio, moq-uring) since +[moq-dev/moq#7042](https://github.com/moq-dev/moq/pull/7042), and a fix if +the cost is real: a stream read that has to wait stops allocating a fresh +shared waker each time. + +## Plan + +Decided 2026-11-01 (maintainer, `/quest-plan`): measure before fixing, since +no bench has shown the cost yet. + +Since #7042, transport polls take a `kio::Waiter` through `&self`, and every +`kio::wait` future (`rs/kio/src/waiter.rs`) holds a waiter of its own. The +moq-net coding layer (`rs/moq-net/src/coding/reader.rs`, `writer.rs`) wraps +each decode or flush in one, so a read that parks allocates the waiter's +shared `Arc` every time, where a per-stream `kio::Park` used to reuse +one. + +[moq-dev/moq#7262](https://github.com/moq-dev/moq/pull/7262) fixed the +in-memory transport's version (it registered on the session close per read). +The memory-transport benches rarely park, so they cannot see the adapters' +cost. + +Measure first: user-space instruction counts and allocations, pinned to one +core, on moq-uring's `session_lite` bench (`just rs bench-session`) and, for +moq-tokio, which has no crate-level session bench, the relay's +`rs/moq-relay/benches/session.rs` or `just bench BASE`. Measure a revision +before #7042 next to main. A measured no-win deletes this quest. + +If it is real, let a stream keep one waiter (or `kio::Park`) across reads +again, for example in `coding::Reader`, instead of one per `kio::wait` +future. diff --git a/quest/m2/perf/mesh-session-cost.md b/quest/m2/perf/mesh-session-cost.md index c0242e697e5..7d505c1e621 100644 --- a/quest/m2/perf/mesh-session-cost.md +++ b/quest/m2/perf/mesh-session-cost.md @@ -15,3 +15,12 @@ cost is production code). Cache `OriginScope::relative`'s patterns first; benchmark before changing the rest. Public API: none expected. + +Decided 2026-11-01 (maintainer, `/quest-plan`): the per-reader cost of a +delivered group belongs here, beside the rest of the per-session cost. +Measured by [moq-dev/moq#7263](https://github.com/moq-dev/moq/pull/7263): +a track read that does get a group still costs about 12k to 26k user-space +instructions per reader per group (1 to 16 readers per track). Part of it is +copying the large `group::Consumer` value, inside +`Poll>>`, through each layer that returns it; +consider boxing parts of `group::Consumer`, and measure it on the same read. diff --git a/rs/moq-net/benches/track.rs b/rs/moq-net/benches/track.rs index 87d050c879b..1bfd838bf6f 100644 --- a/rs/moq-net/benches/track.rs +++ b/rs/moq-net/benches/track.rs @@ -26,8 +26,14 @@ //! `track_sparse_read` hands out groups from a feed with every other sequence missing, //! holding a fixed number of gaps the reader still remembers: each hand-out must cost the //! same however many gaps it holds, so a walk over them shows up as a slope. -//! `track_sparse_poll` repeats a parked read without changing the feed, so revisiting -//! deferred gaps on an otherwise unchanged poll shows up independently of delivery. +//! `track_sparse_poll` repeats a parked read over a feed holding deferred gaps, without +//! changing it: the repeat parks again without walking, so a slope over the gaps means a +//! repeated read revisits them again. +//! +//! `track_idle_poll` repeats every parked read over readers sharing one track, in arrival +//! and sequence order, with nothing new: a repeated read parks again without walking the +//! read path, so each costs a small constant, and a slope over the readers, or a jump in +//! that constant, means the early exit stopped applying. //! //! `track_sparse_expire` opens one new gap while one old gap expires, so a full expiry //! rescan on each arrival shows up as a slope over the gaps still within budget. @@ -1361,6 +1367,73 @@ fn bench_sparse_poll(c: &mut Criterion) { group.finish(); } +/// Readers parked on one track with nothing new, each having taken the one cached group. +struct Idle { + _broadcast: broadcast::Producer, + _track: track::Producer, + _open: group::Producer, + recv: Vec, + next: Vec, + waiter: kio::Waiter, +} + +impl Idle { + fn new(readers: usize, ordered: bool) -> Self { + let broadcast = broadcast::Info::default().produce(); + let track = broadcast.create_track("bench", timed()).unwrap(); + let mut open = track.append_group().unwrap(); + open.write_frame(Timestamp::ZERO, Bytes::from_static(b"x")).unwrap(); + let waiter = kio::Waiter::noop(); + let (mut recv, mut next) = (Vec::new(), Vec::new()); + for _ in 0..readers { + let reader = track.subscribe(track::Subscription::default().with_max_delay(AGED_BUDGET)); + if ordered { + let mut reader = reader.ordered(); + assert!(matches!(reader.poll_next_group(&waiter), Poll::Ready(Ok(Some(_))))); + next.push(reader); + } else { + let mut reader = reader; + assert!(matches!(reader.poll_recv_group(&waiter), Poll::Ready(Ok(Some(_))))); + recv.push(reader); + } + } + let mut idle = Self { + _broadcast: broadcast, + _track: track, + _open: open, + recv, + next, + waiter, + }; + // The first park walks the read path; only the repeats are measured. + idle.poll(); + idle + } + + fn poll(&mut self) { + for reader in &mut self.recv { + assert!(black_box(reader.poll_recv_group(&self.waiter)).is_pending()); + } + for reader in &mut self.next { + assert!(black_box(reader.poll_next_group(&self.waiter)).is_pending()); + } + } +} + +fn bench_idle_poll(c: &mut Criterion) { + let mut group = c.benchmark_group("track_idle_poll"); + for (name, ordered) in [("recv", false), ("next", true)] { + for readers in FANOUT { + group.throughput(Throughput::Elements(readers as u64)); + group.bench_function(BenchmarkId::new(name, readers), |b| { + let mut idle = Idle::new(readers, ordered); + b.iter(|| idle.poll()); + }); + } + } + group.finish(); +} + /// A sparse feed that opens one gap per millisecond, keeping `gaps` within its budget. struct Expiring { sparse: Sparse, @@ -2026,6 +2099,7 @@ criterion_group!( bench_parked_read, bench_sparse_read, bench_sparse_poll, + bench_idle_poll, bench_sparse_expire, bench_gap_flush, bench_wall, diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 24c93496831..8ce096d3f6b 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -31,6 +31,7 @@ pub(super) use demand::Preferences; use demand::Subscriptions; mod fill; +mod idle; mod miss; pub use fill::Fill; use fill::FillState; @@ -6164,6 +6165,9 @@ struct Cursor { drops: Option>, /// What this cursor misses and asks fills for, once it first reads. misses: Option>, + /// The group read that last parked, answered again without the walk until something + /// it parked on fires, once it first parks. + idle: Option>, } /// A group a cursor found but has not moved past; see [`Cursor::take`]. @@ -6378,6 +6382,7 @@ impl Cursor { skipped: Skipped::default(), drops: None, misses: None, + idle: None, } } @@ -6428,6 +6433,8 @@ impl Cursor { /// Resolve [`Self::servable`] again once the window or cap moved, and publish it to the /// groups already handed out. Costs the ranges, never the cache. fn update_servable(&mut self) { + // A moved window or cap notifies nothing this cursor parked on. + idle::Idle::disarm(&mut self.idle); let runs = Servable::new(self.filter.window.runs(0)); let (revision, cap) = (self.filter.resolved, self.end_sequence); self.servable = runs.below(cap); @@ -6676,6 +6683,11 @@ impl Cursor { }) } + /// Whether a group read in `walk` order would park again at once; see [`idle`]. + fn parked(&self, walk: idle::Walk, waiter: &kio::Waiter) -> bool { + idle::Idle::parked(&self.idle, walk, waiter) + } + /// Take the groups skipped since the last call, for the owner to meter. fn take_stale(&mut self) -> stats::Content { std::mem::take(&mut self.stale) @@ -6884,10 +6896,17 @@ impl Cursor { /// Find the next group in arrival order, or in sequence order for `ordered`, without /// moving past it. fn poll_peek(&mut self, ordered: bool, taken: &Delivered, waiter: &kio::Waiter) -> Poll>> { - match ordered { - true => self.poll_peek_next(taken, waiter), - false => self.poll_peek_recv(taken, waiter), + let walk = idle::Walk::new(ordered); + if self.parked(walk, waiter) { + return Poll::Pending; } + let mut idle = idle::Idle::start(&mut self.idle, waiter); + let polled = match ordered { + true => self.poll_peek_next(taken, idle.waiter()), + false => self.poll_peek_recv(taken, idle.waiter()), + }; + idle.finish(&mut self.idle, walk, polled.is_pending()); + polled } /// Move past the group `peeked` found and hand it out: the exact group, even if its @@ -7051,6 +7070,7 @@ impl Cursor { if next == self.next_sequence { return; } + idle::Idle::disarm(&mut self.idle); self.next_sequence = next; // A group delivered on another copy is not a stale skip on this one. if !self.seek_pending.is_empty() { @@ -7408,6 +7428,15 @@ impl Subscriber { /// The next group in arrival order, or in sequence order for `ordered`, skipping any /// already handed out. fn poll_group(&mut self, ordered: bool, waiter: &kio::Waiter) -> Poll>> { + // Ahead of the meter: a read that parks again counts only what a peek skipped. + if let Inner::Plain(cursor) = &self.inner + && cursor.parked(idle::Walk::new(ordered), waiter) + { + if cursor.stale != stats::Content::default() && self.stats.is_tagged() { + self.count_stale(&self.stats.meter()); + } + return Poll::Pending; + } match self.poll_peek_group(ordered, waiter) { Poll::Ready(Ok(Some(reserved))) => Poll::Ready(self.take(reserved).map(Some)), res => { @@ -8955,6 +8984,34 @@ mod test { } } + /// A read that found nothing parks again at once until something it parked on fires, + /// but parks anew for another task, which the next group then wakes instead. + #[test] + fn an_idle_read_parks_for_the_task_polling_it() { + for ordered in [false, true] { + let producer = track_producer("test", Info::default()); + let mut first = framed(&producer, 0, 1); + first.finish().unwrap(); + let mut reader = producer.subscribe(replay()); + let mut read = |waiter: &kio::Waiter| match ordered { + true => reader.poll_group(true, waiter), + false => reader.poll_group(false, waiter), + }; + let (flag, waiter) = Flag::waiter(); + assert!(matches!(read(&waiter), Poll::Ready(Ok(Some(_))))); + assert!(read(&waiter).is_pending()); + assert!(read(&waiter).is_pending(), "nothing fired, so it parks again"); + assert!(!flag.take()); + + let (other, moved) = Flag::waiter(); + assert!(read(&moved).is_pending()); + let mut next = framed(&producer, 1, 1); + next.finish().unwrap(); + assert!(other.take(), "the next group wakes the task that polled last"); + assert!(matches!(read(&moved), Poll::Ready(Ok(Some(_))))); + } + } + /// Write group `sequence` with `frames` frames, left open. fn framed(producer: &Producer, sequence: u64, frames: usize) -> group::Producer { let mut group = producer.create_group(group::Info { sequence }).unwrap(); diff --git a/rs/moq-net/src/model/track/idle.rs b/rs/moq-net/src/model/track/idle.rs new file mode 100644 index 00000000000..21c0e02fb42 --- /dev/null +++ b/rs/moq-net/src/model/track/idle.rs @@ -0,0 +1,156 @@ +//! The shared early exit for a group read that found nothing. +//! +//! A read that parks registers on everything that can change its answer: the track, the +//! subscription, the fills it waits on, and the wall deadlines it judged against. Each of +//! those notifies on every change, so until one fires, the answer stays the same. [`Idle`] +//! parks reads through a waiter of its own that records any wake before passing it on to +//! the task, and answers a read repeated before that wake with the same `Pending`, rather +//! than walking the whole read path again to reach it. +//! +//! Only what notifies can be trusted this way. A change the reader makes to its own cursor, +//! such as moving its range, notifies nothing, so it disarms the gate itself. + +use std::{ + sync::{ + Arc, + atomic::{AtomicBool, Ordering}, + }, + task::{Wake, Waker}, +}; + +/// Which read parked: a [`Cursor`](super::Cursor) parks the arrival and sequence walks +/// separately, since one finding nothing says nothing about the other. +#[derive(Clone, Copy, PartialEq, Eq)] +pub(super) enum Walk { + Arrival, + Sequence, +} + +impl Walk { + /// The walk a read in sequence order for `ordered`, or arrival order otherwise, takes. + pub(super) fn new(ordered: bool) -> Self { + match ordered { + true => Self::Sequence, + false => Self::Arrival, + } + } +} + +/// A cursor's parked read, and the waiter it parked on. +pub(super) struct Idle { + /// The read that parked, or `None` while nothing is parked. + parked: Option, + woken: Arc, + /// Wakes [`Self::woken`]. + wake: Waker, + /// Holds the waiter reads park on with [`Self::wake`]. Kept across reads, so parking + /// again on a list that still holds it is free, as it would be for the task's own + /// waiter, and retired as a task's own park retires one that parked on more lists + /// than it can record. + park: kio::Park, +} + +/// Records a wake, then wakes the task. +struct Woken { + woken: AtomicBool, + task: Waker, +} + +impl Wake for Woken { + fn wake(self: Arc) { + self.wake_by_ref(); + } + + fn wake_by_ref(self: &Arc) { + // Before the task's wake, so the read it triggers sees the change. + self.woken.store(true, Ordering::Release); + self.task.wake_by_ref(); + } +} + +impl Idle { + fn new(task: &Waker) -> Self { + let woken = Arc::new(Woken { + woken: AtomicBool::new(false), + task: task.clone(), + }); + Self { + parked: None, + wake: Waker::from(woken.clone()), + woken, + park: kio::Park::default(), + } + } + + /// Whether `walk`, read again for `task`, would park exactly as it last did: it parked, + /// nothing it parked on has fired since, and it still wakes the same task. + pub(super) fn parked(this: &Option>, walk: Walk, task: &kio::Waiter) -> bool { + this.as_ref().is_some_and(|idle| { + idle.parked == Some(walk) + && !idle.woken.woken.load(Ordering::Acquire) + && idle.woken.task.will_wake(task.waker()) + }) + } + + /// Start a full read for `task`, returning the gate holding the waiter it must park + /// on. A wake from here on counts against the read about to happen. + pub(super) fn start(this: &mut Option>, task: &kio::Waiter) -> Box { + match this.take() { + Some(mut idle) if idle.woken.task.will_wake(task.waker()) => { + idle.parked = None; + // Cleared before the read looks at anything, so a change it misses wakes + // the waiter after this store and leaves the next read a full one. + idle.woken.woken.store(false, Ordering::Release); + idle + } + // Another task: parking anew drops the old waiter, whose registrations go dead. + _ => Box::new(Self::new(task.waker())), + } + } + + /// The waiter a full read parks on. + pub(super) fn waiter(&mut self) -> &kio::Waiter { + self.park.hold(&std::task::Context::from_waker(&self.wake)) + } + + /// Put the gate back after a full read of `walk`, armed if it parked. + pub(super) fn finish(mut self: Box, this: &mut Option>, walk: Walk, pending: bool) { + self.parked = pending.then_some(walk); + *this = Some(self); + } + + /// Forget the parked read: the cursor moved in a way nothing notifies. + pub(super) fn disarm(this: &mut Option>) { + if let Some(idle) = this { + idle.parked = None; + } + } +} + +#[cfg(test)] +mod test { + use super::*; + + /// The waiter reads park on is kept across reads, but retired once it parked on more + /// lists than it records, so re-parking cannot stack entries those lists never collect. + #[test] + fn a_waiter_parked_past_its_records_is_retired() { + let task = kio::Waiter::noop(); + let mut slot = None; + for (lists, retired) in [(1, false), (9, true)] { + let mut idle = Idle::start(&mut slot, &task); + let mut parked: Vec<_> = (0..lists).map(|_| kio::WaiterList::new()).collect(); + let waiter = idle.waiter(); + for list in &mut parked { + waiter.register(list); + } + idle.finish(&mut slot, Walk::Arrival, true); + assert!(Idle::parked(&slot, Walk::Arrival, &task)); + + let mut idle = Idle::start(&mut slot, &task); + idle.waiter(); + assert_eq!(parked.iter().all(kio::WaiterList::is_empty), retired); + idle.finish(&mut slot, Walk::Arrival, false); + } + } +} diff --git a/rs/moq-net/src/stats.rs b/rs/moq-net/src/stats.rs index 78efbf003a8..b784fcfb366 100644 --- a/rs/moq-net/src/stats.rs +++ b/rs/moq-net/src/stats.rs @@ -1970,6 +1970,13 @@ impl Clone for Scope { } impl Scope { + /// Whether this scope was tagged, a cheap pre-check that never resolves: an untagged + /// scope's [`Meter`] counts nothing, but a tagged one's may count nothing either, when + /// the registry is disabled or excludes its path, so [`Meter::is_tracked`] decides. + pub(crate) fn is_tagged(&self) -> bool { + self.session.inner.is_some() && self.resolved.is_some() + } + /// The counters for the session's current tier. Skips the registry unless the /// tier changed since the last call, so the per-group path stays cheap. fn counters(&self) -> Option> {