diff --git a/quest/m2/perf/uring-quiescence.md b/quest/m2/perf/uring-quiescence.md index cb29f6bb02e..878fca494c9 100644 --- a/quest/m2/perf/uring-quiescence.md +++ b/quest/m2/perf/uring-quiescence.md @@ -42,9 +42,10 @@ must not starve the socket. Decided in the 2026-09-30 audit: the egress requeue quest merged here, since both budgets need the same sweep. - Add `passes` per turn to the metrics beside `turns`. -- [Task capacity](/quest/m2/performance-survey/task-capacity.md) also - reshapes the `Tasks::poll` wake-word snapshot. Whichever of the two lands - second rebases its benchmark on the other. +- `Tasks::poll` now snapshots through a wake index + ([moq-dev/moq#7513](https://github.com/moq-dev/moq/pull/7513)), so each + re-snapshot costs the woken words, not the set's capacity. Baseline the pass + budget on that, using the `tasks_capacity_*` groups in kio's `tasks` bench. - Ingress fairness: a pass budget does not bound work inside one poll. The endpoint's `Inner::poll_run` calls `poll_recv` until `Pending`, so one poll drains every queued receive completion, and `Endpoint::demux` drains diff --git a/quest/m2/performance-survey/README.md b/quest/m2/performance-survey/README.md index c25cce3d3b9..194e646d3e9 100644 --- a/quest/m2/performance-survey/README.md +++ b/quest/m2/performance-survey/README.md @@ -173,7 +173,7 @@ No duplicate quest or workflow edit belongs here. ## Required -- [Task capacity after bursts](/quest/m2/performance-survey/task-capacity.md) - polling and memory after peak task occupancy subsides +- [Task capacity after bursts](/quest/m2/performance-survey/task-capacity.md) - memory a task set keeps after peak occupancy subsides - [QUIC receive-state reuse](/quest/m2/performance-survey/quic-receive-reuse.md) - reassembly metadata retained by recycled completed streams - [Cache metadata](/quest/m2/performance-survey/cache-metadata.md) - bookkeeping capacity after cached groups expire - [Cache backing amplification](/quest/m2/performance-survey/cache-backing.md) - small retained samples pinning large shared fragments diff --git a/quest/m2/performance-survey/task-capacity.md b/quest/m2/performance-survey/task-capacity.md index 708e03195dd..d0124bb56ad 100644 --- a/quest/m2/performance-survey/task-capacity.md +++ b/quest/m2/performance-survey/task-capacity.md @@ -1,59 +1,54 @@ -# [M] Task polling recovers after a burst +# [M] Task sets release memory after a burst ## Goal -Long-lived task sets do not pay an unjustified steady-state CPU or memory -cost for a burst that has already drained. +A long-lived task set does not keep an unjustified amount of memory for a +burst that has already drained, or the measurement shows that the retained +capacity is worth what it saves. ## Plan -[kio::Tasks](/rs/kio/src/task.rs) grows slots and per-slot wakers, retains -them after completion, and snapshots every allocated wake-bit word in each -`Tasks::poll`. Dispatch scans those words again. Work can depend on -historical peak/64 even when only one task remains. The io_uring worker and -session serving loops are current consumers. - -Extend [task benchmarks](/rs/kio/benches/tasks.rs) with peak slots × current -live/ready slots: burst then one survivor, sparse live slots, full occupancy, -and empty-but-polled. Report instructions and CPU per owner poll/per ready -task, allocations and retained bytes after draining. Compare a long-lived -worker or session before and after connection/subscription churn. - -The present sparse-wake case fixes 16 ready tasks among 1,000 residents; it -does not isolate historical capacity. Any reclamation or ready-word index -must preserve stale-waker safety, concurrent wakeups, coalescing and snapshot -fairness. Do not trade cheap burst reuse for churn without measuring both. - -A 2026-10-22 staging CDN profile (`f57c0aad`, DWARF stacks, 27-36% incomplete) -put `Tasks::poll` at 2.40% self-time on a 501-connection chat -shape and 1.90% on SD, the largest moq-net site on both, and kio -tasks/waiters at 7.5-11.5% of samples. Self-time only; it does not show how -much is historical capacity. - -Measured in the 2026-10-27 audit on main -[`651c30fee9`](https://github.com/moq-dev/moq/commit/651c30fee9d88fd56f4079433dd3e1f98d77f21c), -one live, non-ready task left after a burst (the task carries only a `bool`, -so real session tasks retain more per slot). Dropping the set returned the -heap to baseline, so this is retained capacity, not a leak. Timings are -unpinned wall clock on a shared host, so they show the slope only: - -| Historical peak | Retained heap | Median ns per owner poll | -|---|---|---| -| 1 | 912 B | 79.9 | -| 1,024 | 188,768 B | 86.6 | -| 16,384 | 3,019,184 B | 1,216.1 | -| 65,536 | 12,076,592 B | 4,830.8 | - -Candidates to evaluate: reclaimable slot chunks, and an index of nonempty -ready words so a poll does not scan historical capacity. Include owner polls -triggered by unrelated active work in the sweep. - -Related: [worker quiescence](/quest/m2/perf/uring-quiescence.md) and -[relay allocations](/quest/m2/perf/relay-allocs.md) own other costs. -Worker quiescence also reshapes the `Tasks::poll` snapshot with a pass -budget; whichever of the two lands second rebases its benchmark on the other. +[kio::Tasks](/rs/kio/src/task.rs) keeps every slot it ever grew: the slab +entry (`Option>`, whose `Park` holds an inline `Waiter` of about +100 B), the per-slot waker allocation, and the wake bitset. The CPU half of +this quest landed in [moq-dev/moq#7513](https://github.com/moq-dev/moq/pull/7513). +A three-level wake index means an owner poll visits only woken words, so poll +cost no longer depends on the historical peak. The memory half remains. + +Decided 2026-11-01 by the maintainer on +[moq-dev/moq#7513](https://github.com/moq-dev/moq/pull/7513): + +- ✅ (a) Land the CPU half as is, and keep this quest for the memory half: + reclaimable slot chunks, measured against repeat-burst cost and the real + consumers' task sizes. +- (b) Treat retained capacity as justified and delete this quest. +- (c) Implement chunk reclaim in the same PR. + +The [`task_capacity` bench](/rs/kio/benches/task_capacity.rs) reports +retained bytes after a burst. With an 8-byte task, the set keeps about +184 B per peak slot whatever is still live: 188,912 B after a 1,024-task +burst and 12,085,304 B after 65,536. A real task adds `size_of::()` +per slot, so measure the actual consumers (`GroupServe`, the io_uring worker's +`Task`, the session sets) before judging the cost. Dropping the set returns +the heap to baseline, so this is retained capacity, not a leak. + +The candidate is reclaimable slot chunks. Store slots, wakers and wake bits per +1,024-slot chunk, and free a chunk once it empties. That needs allocation to +prefer low free slots, so chunks can drain while one survivor stays behind, +plus some hysteresis so a set oscillating across a chunk boundary does not +reallocate 1,024 wakers each time. Today's LIFO free list and full reuse make +a repeated burst cheap. In the harness from +[moq-dev/moq#7513](https://github.com/moq-dev/moq/pull/7513), a repeated +1,024-task burst through a drained set costs about 840 instructions per task, +the task's own allocations included, and the set allocates nothing for it. Do +not trade that for reclaimed memory without measuring both, with an A/A control. + +Any reclamation must preserve stale-waker safety. A waker that outlives its +chunk must keep its chunk alive or point at nothing. It must also keep +concurrent wakeups, coalescing, snapshot fairness and the rotation order +(`concurrent_wakes_are_never_lost`, `wakes_past_64_chunks_run_in_slot_order`). Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md): -measure the baseline before optimizing, retain paired evidence, verify equivalent -work, schedule coverage nightly, and accept a measured no-win. Expected public -API and wire impact: none; report the actual impact in the implementation PR. +measure the baseline before optimizing, retain paired evidence, verify +equivalent work, schedule coverage nightly, and accept a measured no-win. +Expected public API and wire impact: none. diff --git a/rs/kio/Cargo.toml b/rs/kio/Cargo.toml index 1b457a46767..4e6291cebf9 100644 --- a/rs/kio/Cargo.toml +++ b/rs/kio/Cargo.toml @@ -68,6 +68,10 @@ tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread", "test name = "tasks" harness = false +[[bench]] +name = "task_capacity" +harness = false + [[bench]] name = "waiter" harness = false diff --git a/rs/kio/benches/task_capacity.rs b/rs/kio/benches/task_capacity.rs new file mode 100644 index 00000000000..81f7fcf35be --- /dev/null +++ b/rs/kio/benches/task_capacity.rs @@ -0,0 +1,119 @@ +//! Heap a [`kio::Tasks`] set keeps after a burst drains, swept over the burst's peak. +//! +//! Run `cargo bench -p kio --bench task_capacity` and compare the table across +//! revisions. Each row grows a set to `peak` parked tasks, removes all but +//! `live` of them, and reports the bytes the set still holds (its slots, wakers, +//! wake bitset and live tasks; the tasks' gates are outside the count), the +//! allocations per owner poll over 1,000 more polls, and the bytes left once +//! the set and its gates drop (nonzero would be a leak). Per-poll time for the +//! same shapes is in the `tasks` bench's `tasks_capacity_*` groups. + +use std::{ + alloc::{GlobalAlloc, Layout, System}, + hint::black_box, + sync::{ + Arc, Mutex, + atomic::{AtomicIsize, AtomicUsize, Ordering}, + }, + task::{Poll, Waker}, +}; + +struct Counter; + +static LIVE: AtomicIsize = AtomicIsize::new(0); +static ALLOCATIONS: AtomicUsize = AtomicUsize::new(0); + +#[global_allocator] +static ALLOCATOR: Counter = Counter; + +// Counting only. Every call forwards to the system allocator unchanged. +unsafe impl GlobalAlloc for Counter { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + LIVE.fetch_add(layout.size() as isize, Ordering::Relaxed); + ALLOCATIONS.fetch_add(1, Ordering::Relaxed); + unsafe { System.alloc(layout) } + } + + unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 { + LIVE.fetch_add(layout.size() as isize, Ordering::Relaxed); + ALLOCATIONS.fetch_add(1, Ordering::Relaxed); + unsafe { System.alloc_zeroed(layout) } + } + + unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 { + LIVE.fetch_add(new_size as isize - layout.size() as isize, Ordering::Relaxed); + ALLOCATIONS.fetch_add(1, Ordering::Relaxed); + unsafe { System.realloc(ptr, layout, new_size) } + } + + unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { + LIVE.fetch_sub(layout.size() as isize, Ordering::Relaxed); + unsafe { System.dealloc(ptr, layout) } + } +} + +/// A parked waker slot a task re-arms on every poll, so each task holds one +/// registration the way a channel read does. +type Gate = Arc>>; + +fn task(gate: Gate) -> impl FnMut(&kio::Waiter) -> Poll<()> { + move |waiter| { + *gate.lock().unwrap() = Some(waiter.waker().clone()); + Poll::Pending + } +} + +fn main() { + println!("| peak | live | retained bytes | bytes per peak slot | allocations per poll | bytes after drop |"); + println!("|---:|---:|---:|---:|---:|---:|"); + for peak in [1, 1024, 16_384, 65_536] { + let mut lives = vec![0, 1, peak / 64]; + lives.sort(); + lives.dedup(); + for live in lives { + let start = LIVE.load(Ordering::Relaxed); + let gates: Vec = (0..peak).map(|_| Gate::default()).collect(); + // Gates, keys and the owner's waiter precede the baseline, so only the set counts. + let mut keys = Vec::with_capacity(peak); + let waiter = kio::Waiter::noop(); + let base = LIVE.load(Ordering::Relaxed); + + let mut tasks = kio::Tasks::new(); + for gate in &gates { + keys.push(tasks.insert(task(gate.clone()))); + } + let _ = tasks.poll(&waiter); + // Keep every `peak / live`-th task, so live slots spread over the peak. + let stride = peak.checked_div(live).unwrap_or(0); + for (slot, key) in keys.drain(..).enumerate() { + if live == 0 || !slot.is_multiple_of(stride) { + drop(tasks.remove(key)); + // The removed task's registration goes with it, as a channel's would. + gates[slot].lock().unwrap().take(); + } + } + assert_eq!(tasks.len(), live); + let retained = LIVE.load(Ordering::Relaxed) - base; + + let before = ALLOCATIONS.load(Ordering::Relaxed); + for round in 0..1_000 { + if live > 0 { + let gate = &gates[(round % live) * stride]; + if let Some(waker) = gate.lock().unwrap().take() { + waker.wake(); + } + } + let _ = black_box(tasks.poll(&waiter)); + } + let allocations = (ALLOCATIONS.load(Ordering::Relaxed) - before) as f64 / 1_000.0; + + // Parked wakers hold their slot's waker state until the gates drop. + drop((tasks, gates, keys, waiter)); + let after = LIVE.load(Ordering::Relaxed) - start; + println!( + "| {peak} | {live} | {retained} | {:.1} | {allocations} | {after} |", + retained as f64 / peak as f64 + ); + } + } +} diff --git a/rs/kio/benches/tasks.rs b/rs/kio/benches/tasks.rs index accda24a341..054342a3e9e 100644 --- a/rs/kio/benches/tasks.rs +++ b/rs/kio/benches/tasks.rs @@ -23,6 +23,20 @@ //! which is the driver shape: `Tasks` reuses retired slots and their wakers, //! so steady-state churn stops allocating. //! +//! The `tasks_capacity_*` groups sweep a set's historical peak (1 to 65,536 +//! slots) against what is live and woken now, `kio::Tasks` alone. A burst grows +//! the set, and all but the named tasks are removed before measuring, so a cost +//! that rises with the peak is one the drained burst left behind: +//! +//! - `idle`: one live task, nothing woken. The owner polled for unrelated work. +//! - `survivor`: one live task, woken every poll. +//! - `sparse`: every 64th slot live, 16 of them woken per poll. +//! - `full`: every slot live, 16 woken per poll (no drained burst at all). +//! - `churn`: nothing live; 16 tasks pushed and drained per iteration, reusing +//! the slots the burst left. +//! +//! Retained bytes after the burst are in the `task_capacity` bench. +//! //! Run with `cargo bench -p kio`. use std::{ @@ -36,7 +50,7 @@ use std::{ task::{Context, Poll, Waker}, }; -use criterion::{Criterion, criterion_group, criterion_main}; +use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main}; use futures::stream::StreamExt; /// The shared wakeable primitive: a resettable flag plus a parked waker. @@ -236,5 +250,83 @@ fn spawn_drain_hot(c: &mut Criterion) { group.finish(); } -criterion_group!(benches, wake_sparse, spawn_drain, spawn_drain_hot); +/// Historical peaks: one task, one chunk, and 16 and 64 chunks (the exact reach +/// of the top wake-index word). +const PEAKS: [usize; 4] = [1, 1024, 16_384, 65_536]; + +/// A set that grew to `peak` tasks, every one parked, then removed all but the +/// slots `live` keeps. Returns the set and the live tasks' gates. +#[expect(clippy::type_complexity, reason = "an opaque task type cannot be aliased on stable")] +fn after_burst( + peak: usize, + live: impl Fn(usize) -> bool, +) -> (kio::Tasks Poll<()>>, Vec>) { + let gates: Vec<_> = (0..peak).map(|_| Gate::new()).collect(); + let mut tasks = kio::Tasks::new(); + let keys: Vec<_> = gates.iter().map(|gate| tasks.insert(forever(gate.clone()))).collect(); + let _ = tasks.poll(&kio::Waiter::noop()); + let mut kept = Vec::new(); + for (slot, (key, gate)) in keys.into_iter().zip(gates).enumerate() { + match live(slot) { + true => kept.push(gate), + false => drop(tasks.remove(key)), + } + } + (tasks, kept) +} + +fn capacity(c: &mut Criterion) { + // Live slot predicate and tasks woken per owner poll. + type Shape = (&'static str, fn(usize) -> bool, usize); + let shapes: [Shape; 4] = [ + ("idle", |slot| slot == 0, 0), + ("survivor", |slot| slot == 0, 1), + ("sparse", |slot| slot % 64 == 0, 16), + ("full", |_| true, 16), + ]; + for (name, live, ready) in shapes { + let mut group = c.benchmark_group(format!("tasks_capacity_{name}")); + for peak in PEAKS { + group.bench_with_input(BenchmarkId::from_parameter(peak), &peak, |b, &peak| { + let (mut tasks, gates) = after_burst(peak, live); + let ready = ready.min(gates.len()); + let waiter = kio::Waiter::noop(); + let mut next = 0; + b.iter(|| { + for _ in 0..ready { + gates[next].fire(); + next = (next + 1) % gates.len(); + } + let _ = black_box(tasks.poll(&waiter)); + }); + }); + } + group.finish(); + } + + let mut group = c.benchmark_group("tasks_capacity_churn"); + for peak in PEAKS { + group.bench_with_input(BenchmarkId::from_parameter(peak), &peak, |b, &peak| { + fn done(_: &kio::Waiter) -> Poll<()> { + Poll::Ready(()) + } + let waiter = kio::Waiter::noop(); + let mut tasks = kio::Tasks::new(); + for _ in 0..peak { + tasks.push(done); + } + while tasks.poll(&waiter).is_pending() {} + b.iter(|| { + for _ in 0..16 { + tasks.push(done); + } + while tasks.poll(&waiter).is_pending() {} + black_box(&tasks); + }); + }); + } + group.finish(); +} + +criterion_group!(benches, wake_sparse, spawn_drain, spawn_drain_hot, capacity); criterion_main!(benches); diff --git a/rs/kio/src/task.rs b/rs/kio/src/task.rs index 6c3e583baf7..df4fb3dfe89 100644 --- a/rs/kio/src/task.rs +++ b/rs/kio/src/task.rs @@ -29,15 +29,22 @@ //! [unicycle](https://github.com/udoprog/unicycle) proved out against //! `FuturesUnordered`'s intrusive list): a wake is one `fetch_or`, lock-free, //! and inherently deduplicated since setting a set bit is a no-op. Each poll -//! snapshots a word at a time with `swap(0)`, so a task woken mid-pass lands in -//! the next pass instead of spinning this one. Because tasks are closures, not -//! pinned futures, a slot's identity is stable and its [`Waker`] is minted once -//! and reused across occupants: the per-task allocation `FuturesUnordered` +//! snapshots the woken words with `swap(0)` before running anything, so a task +//! woken mid-pass lands in the next pass instead of spinning this one. Because +//! tasks are closures, not pinned futures, a slot's identity is stable and its +//! [`Waker`] is minted once and reused across occupants: the per-task allocation `FuturesUnordered` //! pays per future (and futures-buffered removes with an unsafe arena) is a //! per-slot cost here, amortized to zero in a long-lived set. An owner that //! wakes its own tasks mid-poll (a driver advancing its clock) runs that poll in //! [`Waiter::batch`], so the set's wake of the owner is not a second poll. Safe Rust //! throughout. +//! +//! The bitset is indexed so a poll visits only woken words, not every word the +//! set ever grew: a word's first bit also marks it in its chunk's summary, and a +//! chunk's first marked word also marks bit `chunk % 64` of one word shared by +//! the whole set. A poll with nothing woken is then one `swap`, however many +//! slots a past burst left behind. Up to 64 chunks (65,536 slots) the top word +//! is exact; past that a bit stands for every 64th chunk, one `swap` each. use std::{ cell::{Cell, RefCell}, @@ -57,11 +64,19 @@ const CHUNK_SLOTS: usize = WORD_BITS * CHUNK_WORDS; /// One fixed block of the wake bitset. Chunks never move or grow, so a waker /// can hold its chunk directly and set its bit without any lock. -struct Chunk([AtomicU64; CHUNK_WORDS]); +struct Chunk { + words: [AtomicU64; CHUNK_WORDS], + /// Bit `w` is set once word `w` went nonzero and stays set until a poll + /// takes the word, so a poll swaps only the words that hold wakes. + summary: AtomicU64, +} impl Chunk { fn new() -> Arc { - Arc::new(Self(std::array::from_fn(|_| AtomicU64::new(0)))) + Arc::new(Self { + words: std::array::from_fn(|_| AtomicU64::new(0)), + summary: AtomicU64::new(0), + }) } } @@ -83,10 +98,11 @@ pub struct Tasks { free: Vec, len: usize, shared: Arc, - /// The readiness snapshot each poll dispatches from, taken before any task - /// runs so a mid-pass wake lands in the next pass wherever its slot sits. + /// The readiness snapshot each poll dispatches from: the woken words as + /// `(word index, bits)` in ascending word order, taken before any task runs + /// so a mid-pass wake lands in the next pass wherever its slot sits. /// Retained so steady-state polls don't allocate. - scratch: Vec, + scratch: Vec<(usize, u64)>, /// The slot that leads the next readiness snapshot, rotated after every /// nonempty pass so wake-all resources cannot favor low-numbered tasks. next: usize, @@ -118,6 +134,9 @@ struct Shared { /// Set by the first wake since the last poll began, so a burst of wakes /// costs one parent wake, not one each. pending: AtomicBool, + /// The chunks with a marked summary: bit `c % 64` for chunk `c`. The top + /// level of the wake index, so a poll skips the chunks nothing woke. + ready: AtomicU64, } impl Shared { @@ -220,10 +239,12 @@ impl Polling { } /// The per-slot waker: sets the slot's wake bit and nudges the owner. +/// +/// One lives per slot the set ever grew, so it keeps only the slot index and +/// derives its bits from that. struct SlotWaker { chunk: Arc, - word: usize, - bit: u64, + slot: usize, shared: Arc, } @@ -235,9 +256,21 @@ impl Wake for SlotWaker { fn wake_by_ref(self: &Arc) { // An already-set bit means the slot is queued for the next pass and the // owner was already nudged: a burst of wakes is one bit and one nudge. - if self.chunk.0[self.word].fetch_or(self.bit, Ordering::AcqRel) & self.bit == 0 { - self.shared.wake_parent(); + let index = (self.slot % CHUNK_SLOTS) / WORD_BITS; + let bit = 1 << (self.slot % WORD_BITS); + let word = self.chunk.words[index].fetch_or(bit, Ordering::AcqRel); + if word & bit != 0 { + return; + } + // Only the wake that makes a level nonzero marks the level above. A poll + // clears each level before taking the one below it, so a later wake into a + // still-nonzero level is either taken by that poll or covered by the mark + // the level's first wake left (or is about to leave) for the next one. + if word == 0 && self.chunk.summary.fetch_or(1 << index, Ordering::AcqRel) == 0 { + let chunk = self.slot / CHUNK_SLOTS; + self.shared.ready.fetch_or(1 << (chunk % WORD_BITS), Ordering::AcqRel); } + self.shared.wake_parent(); } } @@ -253,6 +286,7 @@ impl Tasks { shared: Arc::new(Shared { parent: Mutex::new(None), pending: AtomicBool::new(false), + ready: AtomicU64::new(0), }), scratch: Vec::new(), next: 0, @@ -291,8 +325,7 @@ impl Tasks { self.children.push(None); self.wakers.push(Waker::from(Arc::new(SlotWaker { chunk: self.chunks[index / CHUNK_SLOTS].clone(), - word: (index % CHUNK_SLOTS) / WORD_BITS, - bit: 1 << (index % WORD_BITS), + slot: index, shared: self.shared.clone(), }))); index @@ -402,14 +435,31 @@ impl Tasks { // during the pass (by another task or a self-wake) lands in the next // pass wherever its slot sits, never this one. Dispatching straight off // the words instead would run a forward wake chain in a single pass, - // starving the caller's other arms. + // starving the caller's other arms. Each level is cleared before the + // level below it is read, the order the wakers' marks rely on. let mut scratch = std::mem::take(&mut self.scratch); scratch.clear(); - for chunk in &self.chunks { - for word in &chunk.0 { - scratch.push(word.swap(0, Ordering::AcqRel)); + let mut ready = self.shared.ready.swap(0, Ordering::AcqRel); + while ready != 0 { + let first = ready.trailing_zeros() as usize; + ready &= ready - 1; + for (chunk_index, chunk) in self.chunks.iter().enumerate().skip(first).step_by(WORD_BITS) { + let mut words = chunk.summary.swap(0, Ordering::AcqRel); + while words != 0 { + let word = words.trailing_zeros() as usize; + words &= words - 1; + let bits = chunk.words[word].swap(0, Ordering::AcqRel); + if bits != 0 { + scratch.push((chunk_index * CHUNK_WORDS + word, bits)); + } + } } } + // Up to 64 chunks the bits visit them in order; past that, a bit's chunks + // interleave with the next bit's. + if self.chunks.len() > WORD_BITS { + scratch.sort_unstable_by_key(|&(word, _)| word); + } let slots = self.children.len(); let mut advanced = false; @@ -421,8 +471,12 @@ impl Tasks { } let first_word = from / WORD_BITS; let last_word = (to - 1) / WORD_BITS; - for (word_index, word) in scratch.iter().enumerate().take(last_word + 1).skip(first_word) { - let mut bits = *word; + let start = scratch.partition_point(|&(word, _)| word < first_word); + for &(word_index, word) in &scratch[start..] { + if word_index > last_word { + break; + } + let mut bits = word; if word_index == first_word { bits &= u64::MAX << (from % WORD_BITS); } @@ -937,6 +991,127 @@ mod tests { assert_eq!(*order.lock().unwrap(), [0, 1, 2, 1, 2, 0, 2, 0, 1]); } + /// Past 64 chunks a bit of the top index word stands for every 64th chunk: + /// each woken task still runs exactly once, in rotated slot order, and a + /// chunk sharing a woken chunk's bit is not polled for it. + #[test] + #[cfg_attr(miri, ignore = "65 chunks of tasks take too long to interpret")] + fn wakes_past_64_chunks_run_in_slot_order() { + fn task(id: usize, order: Arc>>) -> impl FnMut(&Waiter) -> Poll<()> { + move |_| { + order.lock().unwrap().push(id); + Poll::Pending + } + } + + let order = Arc::new(Mutex::new(Vec::new())); + let mut tasks = Tasks::new(); + let slots = 66 * CHUNK_SLOTS; + for id in 0..slots { + tasks.push(task(id, order.clone())); + } + let waiter = Waiter::noop(); + assert!(tasks.poll(&waiter).is_pending()); + assert_eq!(order.lock().unwrap().len(), slots, "every new task ran once"); + order.lock().unwrap().clear(); + + // Chunks 1 and 65 share a top bit, as do 0 and 64; chunk 2 has its own. + let woken = [ + 65 * CHUNK_SLOTS + 3, + CHUNK_SLOTS + 7, + 2 * CHUNK_SLOTS, + 64 * CHUNK_SLOTS, + 5, + ]; + for &slot in &woken { + tasks.wakers[slot].wake_by_ref(); + } + assert!(tasks.poll(&waiter).is_pending()); + let mut sorted = woken; + sorted.sort(); + assert_eq!( + *order.lock().unwrap(), + sorted, + "ascending slot order from the rotation point" + ); + order.lock().unwrap().clear(); + + // The rotation point moved past the first slot run, which now runs last. + for &slot in &woken { + tasks.wakers[slot].wake_by_ref(); + } + assert!(tasks.poll(&waiter).is_pending()); + let mut rotated = sorted[1..].to_vec(); + rotated.push(sorted[0]); + assert_eq!(*order.lock().unwrap(), rotated); + + order.lock().unwrap().clear(); + assert!(tasks.poll(&waiter).is_pending()); + assert!(order.lock().unwrap().is_empty(), "nothing was woken"); + } + + /// Wakes from other threads racing the owner's polls are never lost: every + /// task woken after its last poll runs again. + #[test] + fn concurrent_wakes_are_never_lost() { + const TASKS: usize = 3 * CHUNK_SLOTS; + const THREADS: usize = 4; + const ROUNDS: usize = if cfg!(miri) { 16 } else { 2_000 }; + + // Each task records the highest wake count it has observed. + let counts: Arc> = Arc::new((0..TASKS).map(|_| AtomicUsize::new(0)).collect()); + let seen: Arc> = Arc::new((0..TASKS).map(|_| AtomicUsize::new(0)).collect()); + let mut tasks = Tasks::new(); + for id in 0..TASKS { + let (counts, seen) = (counts.clone(), seen.clone()); + tasks.push(move |_: &Waiter| { + seen[id].store(counts[id].load(Ordering::SeqCst), Ordering::SeqCst); + Poll::Pending + }); + } + + let wakes = Arc::new(Wakes(AtomicUsize::new(0))); + let waiter = Waiter::new(Waker::from(wakes.clone())); + assert!(tasks.poll(&waiter).is_pending()); + let wakers: Arc> = Arc::new(tasks.wakers.clone()); + + let threads: Vec<_> = (0..THREADS) + .map(|thread| { + let (counts, wakers) = (counts.clone(), wakers.clone()); + std::thread::spawn(move || { + // Spread each thread's wakes over every word and chunk. + let mut slot = thread * 997; + for _ in 0..ROUNDS { + for _ in 0..8 { + slot = (slot + 4099) % TASKS; + counts[slot].fetch_add(1, Ordering::SeqCst); + wakers[slot].wake_by_ref(); + } + } + }) + }) + .collect(); + // Poll while the wakers run: a wake that marks a word or chunk the + // running poll already passed must stay indexed for a later one. + while !threads.iter().all(|thread| thread.is_finished()) { + assert!(tasks.poll(&waiter).is_pending()); + } + for thread in threads { + thread.join().unwrap(); + } + assert!(wakes.0.load(Ordering::SeqCst) > 0, "no wake reached the owner"); + + // Every wake is in; one more poll must run each task woken since its last. + assert!(tasks.poll(&waiter).is_pending()); + for id in 0..TASKS { + assert_eq!( + seen[id].load(Ordering::SeqCst), + counts[id].load(Ordering::SeqCst), + "task {id} lost a wake" + ); + } + } + /// Drive a set to completion as a future. Generic on purpose: the two tests /// below hand this same code to `tokio::spawn` (which demands `Send`) and to /// `spawn_local` with `!Send` tasks. Send-ness is inferred from the task