diff --git a/bench/README.md b/bench/README.md index c314333f0e2..bae2512267c 100644 --- a/bench/README.md +++ b/bench/README.md @@ -64,9 +64,9 @@ nix develop --command env SESSION_ALLOCS=1 cargo bench --locked -p moq-net --fea Throughput counts delivered datagrams. Every one must arrive with its sequence and payload; the memory transport loses none at these cadences, so a drop fails -instead of looking cheaper. `SESSION_ALLOCS=1` prints allocations, allocated -bytes and, where Linux perf counters are readable, user instructions per -delivered datagram or viewer-group. All three exclude the cache tick. Bytes sum +instead of looking cheaper. `SESSION_ALLOCS=1` prints the group consumer size, +allocations, allocated bytes and, where Linux perf counters are readable, user instructions per +delivered datagram or viewer-group. The counts exclude the cache tick. Bytes sum requested allocation and reallocation sizes, including growth in place; they measure neither live memory nor bytes copied. A multiplexed instruction counter fails the run instead of reporting an undercount. Every task runs on the bench's diff --git a/quest/m1/release-perf/README.md b/quest/m1/release-perf/README.md index 725f2cab901..ffabf4712d9 100644 --- a/quest/m1/release-perf/README.md +++ b/quest/m1/release-perf/README.md @@ -315,7 +315,6 @@ subscribers. ## Required - [Relay CPU per frame since #5926](/quest/m1/release-perf/relay-cpu-bisect.md) - the 18-56% rise in relay instructions per delivered frame is bisected, attributed, and its avoidable cost fixed -- [A read moves a small group consumer](/quest/m1/release-perf/group-consumer-size.md) - `group::Consumer` shrinks from 280 bytes, so a read stops copying it through every layer ## Related diff --git a/quest/m1/release-perf/group-consumer-size.md b/quest/m1/release-perf/group-consumer-size.md deleted file mode 100644 index 057debc66c2..00000000000 --- a/quest/m1/release-perf/group-consumer-size.md +++ /dev/null @@ -1,21 +0,0 @@ -# [M] A read moves a small group consumer - -## Goal - -`moq_net::group::Consumer` shrinks from 280 bytes by boxing what a read rarely touches, so handing a group to each reader stops copying 280 bytes through every layer it passes. `just bench-fanout` shows the drop, and the common read path gains no allocation per group. - -## Plan - -Decided 2026-11-03 (maintainer, on [moq-dev/moq#7605](https://github.com/moq-dev/moq/pull/7605)): a follow-up in this epic. - -Found by [moq-dev/moq#7605](https://github.com/moq-dev/moq/pull/7605)'s session scenario (a publisher, a relay origin, and viewers over the in-memory transport): `memcpy` is its largest self cost, about 9.4k of 113k instructions per delivered group at 64 viewers, mostly moves of `group::Consumer` and the `Peeked` that carries one, by value through `Poll>>` layers. The largest callers per group: `track::Subscriber::poll_peek_group` (982), the viewer's `lite::subscriber::UniServe::poll_serve` (715), `track::Subscriber::take` (566, 442, and 428 across its inlined copies), `track::Cursor::poll_peek` (507), and `kio::Tasks::insert` (454). - -- Find which fields a read touches per group and box the rest, or the rarely-set state, behind one pointer. Check `size_of` before and after. -- Measure with `just bench-fanout "" "model session" "64 512"`, and `BENCH_FANOUT_CALLGRIND=1` with `bench/fanout.py compare` to see the moves shrink. Keep the session scenario's allocations per group flat: the [relay allocations](/quest/m2/perf/relay-allocs.md) quest tracks them. - -Public API: `group::Consumer`'s layout only; no signature change expected. Wire: none. - -## Related - -- [Relay CPU per frame since #5926](/quest/m1/release-perf/relay-cpu-bisect.md) - found this -- [The relay allocates less per delivered frame](/quest/m2/perf/relay-allocs.md) - boxing must not add an allocation per delivered group diff --git a/quest/m1/release-perf/relay-cpu-bisect.md b/quest/m1/release-perf/relay-cpu-bisect.md index c64a984f82d..e4f6313f667 100644 --- a/quest/m1/release-perf/relay-cpu-bisect.md +++ b/quest/m1/release-perf/relay-cpu-bisect.md @@ -39,7 +39,7 @@ Decided 2026-11-03 (maintainer, on [moq-dev/moq#7605](https://github.com/moq-dev - ✅ A follow-up at m0, as a correctness bug: [a waiter's record limit changes no behavior](/quest/m0/kio-waiter-records.md). - Leave the limit at 8. 3. `group::Consumer` moves: - - ✅ A follow-up in this epic: [a read moves a small group consumer](/quest/m1/release-perf/group-consumer-size.md). + - ✅ A follow-up in this epic, implemented in [moq-dev/moq#7791](https://github.com/moq-dev/moq/pull/7791). - Leave it. Remaining: @@ -55,7 +55,6 @@ Public API: none expected. Wire: none. ## Related - [A waiter's record limit changes no behavior](/quest/m0/kio-waiter-records.md) - found here; the session's waiter registration cost waits on it -- [A read moves a small group consumer](/quest/m1/release-perf/group-consumer-size.md) - found here; the session's largest remaining self cost - [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - the nightly series that should have caught this growth - [Performance audit](/quest/m2/audit-perf.md) - counts instructions when bisecting, for the same reason diff --git a/rs/moq-net/benches/session/mod.rs b/rs/moq-net/benches/session/mod.rs index d8554982cdc..d3ff91e097d 100644 --- a/rs/moq-net/benches/session/mod.rs +++ b/rs/moq-net/benches/session/mod.rs @@ -10,8 +10,9 @@ //! as a slope. Unwatched broadcasts stay announced and silent: their cost is the //! route table they occupy, not a publisher writing into its own cache. //! -//! `SESSION_ALLOCS=1` prints allocations, allocated bytes and, where Linux perf counters are -//! readable, user instructions per viewer-group or delivered datagram instead of timing. +//! `SESSION_ALLOCS=1` prints the group consumer size, allocations, allocated bytes and, +//! where Linux perf counters are readable, user instructions per viewer-group or delivered +//! datagram instead of timing. //! `SESSION_CPU=1` measures thread CPU time on Unix instead of elapsed time, excluding //! preemption while retaining all protocol work on this current-thread runtime. CPU-timed //! groups have a `_cpu` suffix. Cache-clock advancement remains outside both clocks. @@ -1487,11 +1488,13 @@ fn dash(c: &mut Criterion, name: &str, shapes: impl IntoIterator() { const ROUNDS: usize = 256; + println!("group::Consumer: {} bytes", size_of::()); + let rt = runtime(); let fanout = Shape { publishers: 1, diff --git a/rs/moq-net/src/model/frame.rs b/rs/moq-net/src/model/frame.rs index acc087ad273..e5a2c09a10d 100644 --- a/rs/moq-net/src/model/frame.rs +++ b/rs/moq-net/src/model/frame.rs @@ -1465,7 +1465,7 @@ pub(super) enum Source { pub(super) struct Expiry { policy: Arc, stale_stats: stats::Meter, - stale_counted: Arc, + delivery: Arc, tail: Range, count_payload: bool, } @@ -1474,12 +1474,12 @@ impl Expiry { pub(super) fn new( policy: Arc, stale_stats: stats::Meter, - stale_counted: Arc, + delivery: Arc, ) -> Self { Self { policy, stale_stats, - stale_counted, + delivery, tail: 0..0, count_payload: false, } @@ -1663,7 +1663,7 @@ impl Consumer { } self.expired = true; - if !expiry.stale_counted.swap(true, Ordering::Relaxed) { + if !expiry.delivery.stale_counted.swap(true, Ordering::Relaxed) { let mut stale = self.state.read().content_range(expiry.tail.start, expiry.tail.end); if expiry.count_payload { stale.bytes += self.size().saturating_sub(self.read_idx) as u64; diff --git a/rs/moq-net/src/model/group.rs b/rs/moq-net/src/model/group.rs index 2b3c4570184..33fa9cbe502 100644 --- a/rs/moq-net/src/model/group.rs +++ b/rs/moq-net/src/model/group.rs @@ -530,7 +530,7 @@ impl Plain { for entry in state.frames.range(start..).take(end.saturating_sub(local)) { prefetch.push(entry, false); } - prefetch.len + prefetch.len() } /// Shed the oldest inflated frames past [`MAX_CACHE_BYTES`], keeping the newest, once @@ -2871,7 +2871,13 @@ impl Handle { fn consumer(&self, state: kio::Consumer) -> Consumer { Consumer { info: self.info, - track: self.track.clone(), + delivery: Arc::new(Delivery { + timescale: self.track.timescale, + #[cfg(feature = "async")] + ietf: self.track.ietf(), + arrived: self.alive.arrived, + stale_counted: AtomicBool::new(false), + }), cursor: Cursor { state, log: self.alive.log.clone(), @@ -2890,11 +2896,9 @@ impl Handle { expiry: None, expired: false, adopted: false, - stale_counted: Arc::default(), recover: None, idle: None, parked: false, - arrived: self.alive.arrived, } } @@ -3046,19 +3050,25 @@ impl Demand { /// pays nothing. Inline storage would triple the size of every [`Consumer`], which /// moves by value through each layer that hands out a group. /// +/// The frame buffer's allocation also holds its cursor and optional metadata pointers, +/// keeping an empty prefetch to one pointer without another allocation on the first fill. /// Slots hold the public [`frame::Frame`], with a segmented payload set aside in `chains`, /// so a whole read of a small frame moves it out of its slot as it did before segments. #[derive(Default)] struct Prefetch { + batch: Option>, +} + +struct Batch { // Initialized, not-yet-taken frames are `frames[pos..len]`; the rest are uninitialized. // A slot whose frame is in `chains` holds an empty payload. - frames: Option; Self::CAP]>>, + frames: [MaybeUninit; Prefetch::CAP], // Segmented payloads for `frames`, allocated by the first batch that carries one. - chains: Option; Self::CAP]>>, + chains: Option; Prefetch::CAP]>>, // Relay metadata for `frames`, allocated by the first batch that carries any. - fields: Option>; Self::CAP]>>, + fields: Option>; Prefetch::CAP]>>, // Inflated sizes for `frames`, filled only by a batch of a compressed group's stored slices. - inflated: Option>, + inflated: Option>, pos: usize, len: usize, } @@ -3068,7 +3078,8 @@ impl Prefetch { /// The inflated size of the stored slice [`Self::pop`] just returned. fn take_inflated(&self) -> Option { - self.inflated.as_ref()?.get(self.pos.checked_sub(1)?).copied() + let batch = self.batch.as_ref()?; + batch.inflated.as_ref()?.get(batch.pos.checked_sub(1)?).copied() } /// Take the next buffered frame whole, copying a segmented payload into one buffer. @@ -3077,7 +3088,7 @@ impl Prefetch { #[inline(always)] fn pop(&mut self) -> Option { let mut frame = self.take()?; - if self.chains.is_some() { + if self.batch.as_ref().is_some_and(|batch| batch.chains.is_some()) { self.join_chain(&mut frame); } Some(frame) @@ -3103,46 +3114,62 @@ impl Prefetch { #[inline] fn take(&mut self) -> Option { - if self.pos == self.len { + let batch = self.batch.as_mut()?; + if batch.pos == batch.len { return None; } - let frames = self.frames.as_mut()?; // SAFETY: `pos < len`, so this slot was written by `fill` and not yet taken. - let frame = unsafe { frames[self.pos].assume_init_read() }; - self.pos += 1; + let frame = unsafe { batch.frames[batch.pos].assume_init_read() }; + batch.pos += 1; Some(frame) } /// The segmented payload of the frame [`Self::take`] just returned. #[inline] fn take_chain(&mut self) -> Option { - self.chains.as_mut()?.get_mut(self.pos.checked_sub(1)?)?.take() + let batch = self.batch.as_mut()?; + batch.chains.as_mut()?.get_mut(batch.pos.checked_sub(1)?)?.take() } /// The relay metadata of the frame [`Self::pop`] just returned. fn take_fields(&mut self) -> Option> { - self.fields.as_mut()?.get_mut(self.pos.checked_sub(1)?)?.take() + let batch = self.batch.as_mut()?; + batch.fields.as_mut()?.get_mut(batch.pos.checked_sub(1)?)?.take() } /// Empty a drained batch (`pop` returned `None`) for a refill, which keeps inflated /// sizes beside its frames when it reads a compressed group's stored slices. fn reset(&mut self, inflated: bool) { - debug_assert_eq!(self.pos, self.len, "a refill of a non-empty batch would leak frames"); - self.pos = 0; - self.len = 0; + let Some(batch) = &mut self.batch else { + return; + }; + debug_assert_eq!(batch.pos, batch.len, "a refill of a non-empty batch would leak frames"); + batch.pos = 0; + batch.len = 0; if !inflated { - self.inflated = None; + batch.inflated = None; } } + fn len(&self) -> usize { + self.batch.as_ref().map_or(0, |batch| batch.len) + } + /// Buffer `entry` at the end of the batch, with its inflated size if `inflated`, /// returning where it ends on the group's pace scale. fn push(&mut self, entry: &Entry, inflated: bool) -> u64 { - let at = self.len; - let frames = self - .frames - .get_or_insert_with(|| Box::new([const { MaybeUninit::uninit() }; Self::CAP])); - let payload = match (&entry.frame.payload, &mut self.chains) { + let batch = self.batch.get_or_insert_with(|| { + Box::new(Batch { + frames: [const { MaybeUninit::uninit() }; Self::CAP], + chains: None, + fields: None, + inflated: None, + pos: 0, + len: 0, + }) + }); + let at = batch.len; + let payload = match (&entry.frame.payload, &mut batch.chains) { (Payload::Whole(bytes), Some(slots)) => { slots[at] = None; bytes.clone() @@ -3153,21 +3180,21 @@ impl Prefetch { Bytes::new() } }; - frames[at].write(frame::Frame { + batch.frames[at].write(frame::Frame { timestamp: entry.frame.timestamp, payload, arrived: entry.frame.arrived, }); // Plain frames never allocate the metadata table. - match (&entry.fields, &mut self.fields) { + match (&entry.fields, &mut batch.fields) { (Some(fields), slots) => slots.get_or_insert_default()[at] = Some(fields.clone()), (None, Some(slots)) => slots[at] = None, (None, None) => {} } if inflated { - self.inflated.get_or_insert_default()[at] = entry.inflated; + batch.inflated.get_or_insert_default()[at] = entry.inflated; } - self.len += 1; + batch.len += 1; entry.end } @@ -3238,33 +3265,33 @@ impl Prefetch { end = self.push(entry, compressed); } pace.took(end); - self.len + self.len() } /// Replace a filled batch's stored slices, from local frame `local`, with their inflated /// frames. A frame that fails to inflate ends the batch, failing it if first. fn inflate(&mut self, job: &Inflate, local: usize, meter: &stats::Meter) -> Result<()> { job.run(meter); - let Some(frames) = &mut self.frames else { + let Some(batch) = &mut self.batch else { return Ok(()); }; - for i in self.pos..self.len { + for i in batch.pos..batch.len { match job.get(local + i) { Ok(payload) => { // SAFETY: slots in `pos..len` are initialized (written by `fill`, not yet popped). - unsafe { frames[i].assume_init_mut() }.payload = payload; - if let Some(chains) = &mut self.chains { + unsafe { batch.frames[i].assume_init_mut() }.payload = payload; + if let Some(chains) = &mut batch.chains { chains[i] = None; } } Err(err) => { - for slot in &mut frames[i..self.len] { + for slot in &mut batch.frames[i..batch.len] { // SAFETY: as above, and `len` drops below them right after. unsafe { slot.assume_init_drop() }; } - self.len = i; + batch.len = i; job.plain.trim(); - return if i == self.pos { Err(err) } else { Ok(()) }; + return if i == batch.pos { Err(err) } else { Ok(()) }; } } } @@ -3277,30 +3304,27 @@ impl Prefetch { /// Read once per fill to bump the egress payload counters for the whole batch. fn buffered(&self) -> (u64, u64) { let mut bytes = 0u64; - let Some(frames) = &self.frames else { + let Some(batch) = &self.batch else { return (0, 0); }; - for slot in &frames[self.pos..self.len] { + for slot in &batch.frames[batch.pos..batch.len] { // SAFETY: slots in `pos..len` are initialized (written by `fill`, not yet popped). bytes += unsafe { slot.assume_init_ref() }.payload.len() as u64; } - if let Some(chains) = &self.chains { - bytes += chains[self.pos..self.len] + if let Some(chains) = &batch.chains { + bytes += chains[batch.pos..batch.len] .iter() .flatten() .map(|chain| chain.len() as u64) .sum::(); } - ((self.len - self.pos) as u64, bytes) + ((batch.len - batch.pos) as u64, bytes) } } -impl Drop for Prefetch { +impl Drop for Batch { fn drop(&mut self) { - let Some(frames) = &mut self.frames else { - return; - }; - for slot in &mut frames[self.pos..self.len] { + for slot in &mut self.frames[self.pos..self.len] { // SAFETY: slots in `pos..len` are initialized and were never taken. unsafe { slot.assume_init_drop() }; } @@ -3314,9 +3338,8 @@ pub struct Consumer { // Immutable stream state. info: Info, - // The parent track's info, inherited from the producer. Its `timescale` lets the - // wire publisher emit per-frame timestamps at the right scale for a fetched group. - track: track::Info, + // Cold metadata shares the allocation that deduplicates expiry across clones. + delivery: Arc, // Egress payload meter, set by a tagged track via [`Self::with_meter`]. Empty // (no-op) for an untagged group. Also owns unread content discarded by expiry. @@ -3329,9 +3352,6 @@ pub struct Consumer { // A replacement copy was adopted and has not yet returned a frame that matches. Its // ordinals match this group's only while its Object IDs do. adopted: bool, - // Cloned cursors are parallel views of one handed-out delivery. Whichever - // observes expiry first records its unread tail; the others must not repeat it. - stale_counted: Arc, // Handed out from a front's logical track: carries the read across route changes. // Boxed: it is the rare case. recover: Option>, @@ -3340,9 +3360,21 @@ pub struct Consumer { idle: Option>>, // The last frame read parked with no gate set up yet. parked: bool, - // When the group was created here. Kept across an adopted replacement copy, which the - // group arrived before. +} + +// Poll layers repeatedly move this value; keep cold state shared or lazy. +#[cfg(target_pointer_width = "64")] +const _: () = assert!(size_of::() <= 184); + +/// Per-delivery state shared by a consumer's clones and its frame readers. +/// Kept across replacement copies, so provenance and arrival remain the original's. +pub(super) struct Delivery { + timescale: Option, + #[cfg(feature = "async")] + ietf: Option, arrived: crate::time::Instant, + // Whichever reader observes expiry first records its unread tail only once. + pub(super) stale_counted: AtomicBool, } /// Subscriber-specific policy for a group after it was handed out: when it expires, and @@ -3419,19 +3451,17 @@ impl Clone for Consumer { Self { cursor: self.cursor.clone(), info: self.info, - track: self.track.clone(), + delivery: self.delivery.clone(), // Inherit the meter without re-counting the group: the original already // counted it when the track handed it out. stats: self.stats.clone(), expiry: self.expiry.clone(), expired: self.expired, adopted: self.adopted, - stale_counted: self.stale_counted.clone(), recover: self.recover.clone(), // A clone reads in parallel, so it parks for itself. idle: None, parked: false, - arrived: self.arrived, } } } @@ -3680,7 +3710,7 @@ impl Consumer { self.expired = true; // Another read, such as `poll_expired`, may judge a frame read parked before it. Idle::disarm(&mut self.idle); - if !self.stale_counted.swap(true, Ordering::Relaxed) { + if !self.delivery.stale_counted.swap(true, Ordering::Relaxed) { self.stats.stale(self.cursor.unread_content()); } } @@ -3705,7 +3735,7 @@ impl Consumer { /// The parent track's timescale, or `None` when it declared no timeline. pub fn timescale(&self) -> Option { - self.track.timescale + self.delivery.timescale } /// When the group was created at this hop, before its first frame, on the ambient clock: @@ -3715,14 +3745,14 @@ impl Consumer { /// from when it landed, as [`frame::Frame::arrived`] does for each frame. A consumer /// resumed onto a replacement copy keeps the original group's arrival. pub fn arrived(&self) -> crate::time::Instant { - self.arrived + self.delivery.arrived } /// The draft the parent track's moq-transport publisher spoke, which serialized every /// object's properties, or `None` for a moq-lite one. #[cfg(feature = "async")] pub(crate) fn ietf(&self) -> Option { - self.track.ietf() + self.delivery.ietf } /// The index of the next frame this consumer will return. @@ -3841,7 +3871,7 @@ impl Consumer { let expiry = self .expiry .as_ref() - .map(|policy| frame::Expiry::new(policy.clone(), stats.clone(), self.stale_counted.clone())); + .map(|policy| frame::Expiry::new(policy.clone(), stats.clone(), self.delivery.clone())); let res = self.cursor.poll_next_frame(waiter, &stats, expiry); self.poll_expired_if_blocked(waiter, res) } @@ -4251,7 +4281,7 @@ impl Cursor { // resets the batch, so an empty one leaves `len == 0` and the terminal checks // below resolve abort/fin/pending. let job = prefetch.fill(state, index, budget, raw)?; - if prefetch.len > 0 { + if prefetch.len() > 0 { // One stamp covers the whole batch: frames popped from the prefetch // don't re-stamp until the next refill, which `CAP` bounds. state.charge.refresh();