Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions quest/m2/perf/uring-quiescence.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,10 @@ must not starve the socket.
Decided in the 2026-09-30 audit: the egress requeue quest merged here, since
both budgets need the same sweep.
- Add `passes` per turn to the metrics beside `turns`.
- [Task capacity](/quest/m2/performance-survey/task-capacity.md) also
reshapes the `Tasks::poll` wake-word snapshot. Whichever of the two lands
second rebases its benchmark on the other.
- `Tasks::poll` now snapshots through a wake index
([moq-dev/moq#7513](https://github.com/moq-dev/moq/pull/7513)), so each
re-snapshot costs the woken words, not the set's capacity. Baseline the pass
budget on that, using the `tasks_capacity_*` groups in kio's `tasks` bench.
- Ingress fairness: a pass budget does not bound work inside one poll.
The endpoint's `Inner::poll_run` calls `poll_recv` until `Pending`, so one
poll drains every queued receive completion, and `Endpoint::demux` drains
Expand Down
2 changes: 1 addition & 1 deletion quest/m2/performance-survey/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ No duplicate quest or workflow edit belongs here.

## Required

- [Task capacity after bursts](/quest/m2/performance-survey/task-capacity.md) - polling and memory after peak task occupancy subsides
- [Task capacity after bursts](/quest/m2/performance-survey/task-capacity.md) - memory a task set keeps after peak occupancy subsides
- [QUIC receive-state reuse](/quest/m2/performance-survey/quic-receive-reuse.md) - reassembly metadata retained by recycled completed streams
- [Cache metadata](/quest/m2/performance-survey/cache-metadata.md) - bookkeeping capacity after cached groups expire
- [Cache backing amplification](/quest/m2/performance-survey/cache-backing.md) - small retained samples pinning large shared fragments
Expand Down
97 changes: 46 additions & 51 deletions quest/m2/performance-survey/task-capacity.md
Original file line number Diff line number Diff line change
@@ -1,59 +1,54 @@
# [M] Task polling recovers after a burst
# [M] Task sets release memory after a burst

## Goal

Long-lived task sets do not pay an unjustified steady-state CPU or memory
cost for a burst that has already drained.
A long-lived task set does not keep an unjustified amount of memory for a
burst that has already drained, or the measurement shows that the retained
capacity is worth what it saves.

## Plan

[kio::Tasks](/rs/kio/src/task.rs) grows slots and per-slot wakers, retains
them after completion, and snapshots every allocated wake-bit word in each
`Tasks::poll`. Dispatch scans those words again. Work can depend on
historical peak/64 even when only one task remains. The io_uring worker and
session serving loops are current consumers.

Extend [task benchmarks](/rs/kio/benches/tasks.rs) with peak slots × current
live/ready slots: burst then one survivor, sparse live slots, full occupancy,
and empty-but-polled. Report instructions and CPU per owner poll/per ready
task, allocations and retained bytes after draining. Compare a long-lived
worker or session before and after connection/subscription churn.

The present sparse-wake case fixes 16 ready tasks among 1,000 residents; it
does not isolate historical capacity. Any reclamation or ready-word index
must preserve stale-waker safety, concurrent wakeups, coalescing and snapshot
fairness. Do not trade cheap burst reuse for churn without measuring both.

A 2026-10-22 staging CDN profile (`f57c0aad`, DWARF stacks, 27-36% incomplete)
put `Tasks<GroupServe>::poll` at 2.40% self-time on a 501-connection chat
shape and 1.90% on SD, the largest moq-net site on both, and kio
tasks/waiters at 7.5-11.5% of samples. Self-time only; it does not show how
much is historical capacity.

Measured in the 2026-10-27 audit on main
[`651c30fee9`](https://github.com/moq-dev/moq/commit/651c30fee9d88fd56f4079433dd3e1f98d77f21c),
one live, non-ready task left after a burst (the task carries only a `bool`,
so real session tasks retain more per slot). Dropping the set returned the
heap to baseline, so this is retained capacity, not a leak. Timings are
unpinned wall clock on a shared host, so they show the slope only:

| Historical peak | Retained heap | Median ns per owner poll |
|---|---|---|
| 1 | 912 B | 79.9 |
| 1,024 | 188,768 B | 86.6 |
| 16,384 | 3,019,184 B | 1,216.1 |
| 65,536 | 12,076,592 B | 4,830.8 |

Candidates to evaluate: reclaimable slot chunks, and an index of nonempty
ready words so a poll does not scan historical capacity. Include owner polls
triggered by unrelated active work in the sweep.

Related: [worker quiescence](/quest/m2/perf/uring-quiescence.md) and
[relay allocations](/quest/m2/perf/relay-allocs.md) own other costs.
Worker quiescence also reshapes the `Tasks::poll` snapshot with a pass
budget; whichever of the two lands second rebases its benchmark on the other.
[kio::Tasks](/rs/kio/src/task.rs) keeps every slot it ever grew: the slab
entry (`Option<Occupant<T>>`, whose `Park` holds an inline `Waiter` of about
100 B), the per-slot waker allocation, and the wake bitset. The CPU half of
this quest landed in [moq-dev/moq#7513](https://github.com/moq-dev/moq/pull/7513).
A three-level wake index means an owner poll visits only woken words, so poll
cost no longer depends on the historical peak. The memory half remains.

Decided 2026-11-01 by the maintainer on
[moq-dev/moq#7513](https://github.com/moq-dev/moq/pull/7513):

- ✅ (a) Land the CPU half as is, and keep this quest for the memory half:
reclaimable slot chunks, measured against repeat-burst cost and the real
consumers' task sizes.
- (b) Treat retained capacity as justified and delete this quest.
- (c) Implement chunk reclaim in the same PR.

The [`task_capacity` bench](/rs/kio/benches/task_capacity.rs) reports
retained bytes after a burst. With an 8-byte task, the set keeps about
184 B per peak slot whatever is still live: 188,912 B after a 1,024-task
burst and 12,085,304 B after 65,536. A real task adds `size_of::<T>()`
per slot, so measure the actual consumers (`GroupServe`, the io_uring worker's
`Task`, the session sets) before judging the cost. Dropping the set returns
the heap to baseline, so this is retained capacity, not a leak.

The candidate is reclaimable slot chunks. Store slots, wakers and wake bits per
1,024-slot chunk, and free a chunk once it empties. That needs allocation to
prefer low free slots, so chunks can drain while one survivor stays behind,
plus some hysteresis so a set oscillating across a chunk boundary does not
reallocate 1,024 wakers each time. Today's LIFO free list and full reuse make
a repeated burst cheap. In the harness from
[moq-dev/moq#7513](https://github.com/moq-dev/moq/pull/7513), a repeated
1,024-task burst through a drained set costs about 840 instructions per task,
the task's own allocations included, and the set allocates nothing for it. Do
not trade that for reclaimed memory without measuring both, with an A/A control.

Any reclamation must preserve stale-waker safety. A waker that outlives its
chunk must keep its chunk alive or point at nothing. It must also keep
concurrent wakeups, coalescing, snapshot fairness and the rotation order
(`concurrent_wakes_are_never_lost`, `wakes_past_64_chunks_run_in_slot_order`).

Follow the [survey's benchmark contract](/quest/m2/performance-survey/README.md):
measure the baseline before optimizing, retain paired evidence, verify equivalent
work, schedule coverage nightly, and accept a measured no-win. Expected public
API and wire impact: none; report the actual impact in the implementation PR.
measure the baseline before optimizing, retain paired evidence, verify
equivalent work, schedule coverage nightly, and accept a measured no-win.
Expected public API and wire impact: none.
4 changes: 4 additions & 0 deletions rs/kio/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,10 @@ tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread", "test
name = "tasks"
harness = false

[[bench]]
name = "task_capacity"
harness = false

[[bench]]
name = "waiter"
harness = false
Expand Down
119 changes: 119 additions & 0 deletions rs/kio/benches/task_capacity.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
//! Heap a [`kio::Tasks`] set keeps after a burst drains, swept over the burst's peak.
//!
//! Run `cargo bench -p kio --bench task_capacity` and compare the table across
//! revisions. Each row grows a set to `peak` parked tasks, removes all but
//! `live` of them, and reports the bytes the set still holds (its slots, wakers,
//! wake bitset and live tasks; the tasks' gates are outside the count), the
//! allocations per owner poll over 1,000 more polls, and the bytes left once
//! the set and its gates drop (nonzero would be a leak). Per-poll time for the
//! same shapes is in the `tasks` bench's `tasks_capacity_*` groups.

use std::{
alloc::{GlobalAlloc, Layout, System},
hint::black_box,
sync::{
Arc, Mutex,
atomic::{AtomicIsize, AtomicUsize, Ordering},
},
task::{Poll, Waker},
};

struct Counter;

static LIVE: AtomicIsize = AtomicIsize::new(0);
static ALLOCATIONS: AtomicUsize = AtomicUsize::new(0);

#[global_allocator]
static ALLOCATOR: Counter = Counter;

// Counting only. Every call forwards to the system allocator unchanged.
unsafe impl GlobalAlloc for Counter {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
LIVE.fetch_add(layout.size() as isize, Ordering::Relaxed);
ALLOCATIONS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc(layout) }
}

unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
LIVE.fetch_add(layout.size() as isize, Ordering::Relaxed);
ALLOCATIONS.fetch_add(1, Ordering::Relaxed);
unsafe { System.alloc_zeroed(layout) }
}

unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
LIVE.fetch_add(new_size as isize - layout.size() as isize, Ordering::Relaxed);
ALLOCATIONS.fetch_add(1, Ordering::Relaxed);
unsafe { System.realloc(ptr, layout, new_size) }
}

unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
LIVE.fetch_sub(layout.size() as isize, Ordering::Relaxed);
unsafe { System.dealloc(ptr, layout) }
}
}

/// A parked waker slot a task re-arms on every poll, so each task holds one
/// registration the way a channel read does.
type Gate = Arc<Mutex<Option<Waker>>>;

fn task(gate: Gate) -> impl FnMut(&kio::Waiter) -> Poll<()> {
move |waiter| {
*gate.lock().unwrap() = Some(waiter.waker().clone());
Poll::Pending
}
}

fn main() {
println!("| peak | live | retained bytes | bytes per peak slot | allocations per poll | bytes after drop |");
println!("|---:|---:|---:|---:|---:|---:|");
for peak in [1, 1024, 16_384, 65_536] {
let mut lives = vec![0, 1, peak / 64];
lives.sort();
lives.dedup();
for live in lives {
let start = LIVE.load(Ordering::Relaxed);
let gates: Vec<Gate> = (0..peak).map(|_| Gate::default()).collect();
// Gates, keys and the owner's waiter precede the baseline, so only the set counts.
let mut keys = Vec::with_capacity(peak);
let waiter = kio::Waiter::noop();
let base = LIVE.load(Ordering::Relaxed);

let mut tasks = kio::Tasks::new();
for gate in &gates {
keys.push(tasks.insert(task(gate.clone())));
}
let _ = tasks.poll(&waiter);
// Keep every `peak / live`-th task, so live slots spread over the peak.
let stride = peak.checked_div(live).unwrap_or(0);
for (slot, key) in keys.drain(..).enumerate() {
if live == 0 || !slot.is_multiple_of(stride) {
drop(tasks.remove(key));
// The removed task's registration goes with it, as a channel's would.
gates[slot].lock().unwrap().take();
}
}
assert_eq!(tasks.len(), live);
let retained = LIVE.load(Ordering::Relaxed) - base;

let before = ALLOCATIONS.load(Ordering::Relaxed);
for round in 0..1_000 {
if live > 0 {
let gate = &gates[(round % live) * stride];
if let Some(waker) = gate.lock().unwrap().take() {
waker.wake();
}
}
let _ = black_box(tasks.poll(&waiter));
}
let allocations = (ALLOCATIONS.load(Ordering::Relaxed) - before) as f64 / 1_000.0;

// Parked wakers hold their slot's waker state until the gates drop.
drop((tasks, gates, keys, waiter));
let after = LIVE.load(Ordering::Relaxed) - start;
println!(
"| {peak} | {live} | {retained} | {:.1} | {allocations} | {after} |",
retained as f64 / peak as f64
);
}
}
}
96 changes: 94 additions & 2 deletions rs/kio/benches/tasks.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,20 @@
//! which is the driver shape: `Tasks` reuses retired slots and their wakers,
//! so steady-state churn stops allocating.
//!
//! The `tasks_capacity_*` groups sweep a set's historical peak (1 to 65,536
//! slots) against what is live and woken now, `kio::Tasks` alone. A burst grows
//! the set, and all but the named tasks are removed before measuring, so a cost
//! that rises with the peak is one the drained burst left behind:
//!
//! - `idle`: one live task, nothing woken. The owner polled for unrelated work.
//! - `survivor`: one live task, woken every poll.
//! - `sparse`: every 64th slot live, 16 of them woken per poll.
//! - `full`: every slot live, 16 woken per poll (no drained burst at all).
//! - `churn`: nothing live; 16 tasks pushed and drained per iteration, reusing
//! the slots the burst left.
//!
//! Retained bytes after the burst are in the `task_capacity` bench.
//!
//! Run with `cargo bench -p kio`.

use std::{
Expand All @@ -36,7 +50,7 @@ use std::{
task::{Context, Poll, Waker},
};

use criterion::{Criterion, criterion_group, criterion_main};
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
use futures::stream::StreamExt;

/// The shared wakeable primitive: a resettable flag plus a parked waker.
Expand Down Expand Up @@ -236,5 +250,83 @@ fn spawn_drain_hot(c: &mut Criterion) {
group.finish();
}

criterion_group!(benches, wake_sparse, spawn_drain, spawn_drain_hot);
/// Historical peaks: one task, one chunk, and 16 and 64 chunks (the exact reach
/// of the top wake-index word).
const PEAKS: [usize; 4] = [1, 1024, 16_384, 65_536];

/// A set that grew to `peak` tasks, every one parked, then removed all but the
/// slots `live` keeps. Returns the set and the live tasks' gates.
#[expect(clippy::type_complexity, reason = "an opaque task type cannot be aliased on stable")]
fn after_burst(
peak: usize,
live: impl Fn(usize) -> bool,
) -> (kio::Tasks<impl FnMut(&kio::Waiter) -> Poll<()>>, Vec<Arc<Gate>>) {
let gates: Vec<_> = (0..peak).map(|_| Gate::new()).collect();
let mut tasks = kio::Tasks::new();
let keys: Vec<_> = gates.iter().map(|gate| tasks.insert(forever(gate.clone()))).collect();
let _ = tasks.poll(&kio::Waiter::noop());
let mut kept = Vec::new();
for (slot, (key, gate)) in keys.into_iter().zip(gates).enumerate() {
match live(slot) {
true => kept.push(gate),
false => drop(tasks.remove(key)),
}
}
(tasks, kept)
}

fn capacity(c: &mut Criterion) {
// Live slot predicate and tasks woken per owner poll.
type Shape = (&'static str, fn(usize) -> bool, usize);
let shapes: [Shape; 4] = [
("idle", |slot| slot == 0, 0),
("survivor", |slot| slot == 0, 1),
("sparse", |slot| slot % 64 == 0, 16),
("full", |_| true, 16),
];
for (name, live, ready) in shapes {
let mut group = c.benchmark_group(format!("tasks_capacity_{name}"));
for peak in PEAKS {
group.bench_with_input(BenchmarkId::from_parameter(peak), &peak, |b, &peak| {
let (mut tasks, gates) = after_burst(peak, live);
let ready = ready.min(gates.len());
let waiter = kio::Waiter::noop();
let mut next = 0;
b.iter(|| {
for _ in 0..ready {
gates[next].fire();
next = (next + 1) % gates.len();
}
let _ = black_box(tasks.poll(&waiter));
});
});
}
group.finish();
}

let mut group = c.benchmark_group("tasks_capacity_churn");
for peak in PEAKS {
group.bench_with_input(BenchmarkId::from_parameter(peak), &peak, |b, &peak| {
fn done(_: &kio::Waiter) -> Poll<()> {
Poll::Ready(())
}
let waiter = kio::Waiter::noop();
let mut tasks = kio::Tasks::new();
for _ in 0..peak {
tasks.push(done);
}
while tasks.poll(&waiter).is_pending() {}
b.iter(|| {
for _ in 0..16 {
tasks.push(done);
}
while tasks.poll(&waiter).is_pending() {}
black_box(&tasks);
});
});
}
group.finish();
}

criterion_group!(benches, wake_sparse, spawn_drain, spawn_drain_hot, capacity);
criterion_main!(benches);
Loading