Skip to content
2 changes: 1 addition & 1 deletion .github/relay-cpu-bisect.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Against `main` at [619df6e1b3](https://github.com/moq-dev/moq/commit/619df6e1b3)

`Cursor::probe` is the relay's check for groups a reader has missed. Each time a subscriber asks for its next group, the probe looks at what arrived on the track since its last look and decides whether any group the reader still wants has left the cache without the live feed delivering it (evicted, skipped, or from before the feed started), and asks upstream to fill those. A live reader that takes each group as it arrives never has such a gap, yet it paid for the whole check on every group: locking the subscription and the track's shared state, cloning the track's fills, walking the arrival ledger, checking each new group for holes, and running the ask-and-answer loop. Each time the cache aged out an old group, the eviction also landed with every reader, all of which had already taken it, and forced each one to re-run the full check. The fast path judges a group the arrival walk serves without the hole, backfill, and ledger passes, skips the subscription, fills, and answers when nothing is asked or owed, and starts each reader's eviction inbox above what it handed out. It recovers most of [moq-dev/moq#6171](https://github.com/moq-dev/moq/pull/6171)'s and [moq-dev/moq#7339](https://github.com/moq-dev/moq/pull/7339)'s cost.

kio's `WaiterList::register` is the session path's second-largest self cost: 7.2k per group. A waiter records 8 lists, and a relay session's control and serving tasks park on more, so every list past the eighth takes a duplicate entry on every poll. Remembering 16 cut the session by 4,424 (-3.9%) and was reverted: with it, two moq-net integration tests (`errored_end::an_errored_end_keeps_its_history_across_a_hop`, `quiet_catalog::a_quiet_catalog_reaches_an_unfloored_relay_reader_when_a_peer_resumes_past_it`) fail, and they pass with 8. A read whose waiter parks on more lists than it records is retired and walks again; with more records those reads stay parked, and something they depend on doesn't wake them. That is now an m0 correctness quest, [a waiter's record limit changes no behavior](../quest/m0/kio-waiter-records.md), and the per-group costs this report attributes and nothing here cut are accepted in the [release-perf epic](../quest/m1/release-perf/README.md).
kio's `WaiterList::register` was the session path's second-largest self cost: 7.2k per group. A waiter recorded 8 lists, and a relay session's control and serving tasks parked on more, so every list past the eighth took a duplicate entry on every poll. Remembering 16 cut the session by 4,424 (-3.9%) and was initially reverted after two moq-net integration tests failed. [moq-dev/moq#7789](https://github.com/moq-dev/moq/pull/7789) fixes the front's metadata/abort ordering and the quiet-catalog test's detached handshake, then raises the record limit to 16; its paired session measurement is recorded in that PR. Forcing every idle gate to walk still reproduced both failures, so the suspected idle-gate wake dependency did not explain them. The per-group costs this report attributes and nothing here cut are accepted in the [release-perf epic](../quest/m1/release-perf/README.md).

On the quiet test host (run.we4umM, one-minute load at most 4.7), chat over io\_uring went from 154.5k \[154.1-155.0] instructions per frame on `main` to 148.9k \[148.3-150.8] with the probe and window changes (-3.6%), four rounds each, every cell at 20,000 frames/s with no loss. The relay's QUIC, crypto, and syscalls are most of its cost, so the model's -23% shows as a few percent there.

Expand Down
1 change: 0 additions & 1 deletion quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,6 @@ edge, which embeds moq-relay; it moved there on 2026-09-28.
- [Backport the joined-head FETCH race fix](/quest/m0/backport-fill-head-race.md) - a `release` relay stops FETCHing a group head its own SUBSCRIBE is bringing, which left live readers hanging
- [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in GitHub's merge queue, one tested entry at a time, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last
- [More tests under load](/quest/m0/test-flakes-2/README.md) - load-only and queue failures, one quest per flake, fixed at the cause, since each one fails a whole batch
- [A waiter's record limit changes no behavior](/quest/m0/kio-waiter-records.md) - two moq-net tests stop failing when kio's waiters remember 16 lists, and the limit becomes a tuning knob worth 3.9% per delivered group
- [Backport the relay and auth hardening to release](/quest/m0/backport-release-hardening.md) - `release`'s relay refuses dot-segment paths, serves its HTTP routes with main's response hardening, and withdraws advertisements a peer starves of send credit; its auth client refuses redirects
- [Agent build output can't fill the desktop's disk](/quest/m0/agent-disk-budget.md) - finished agent checkouts reclaim their mbx targets automatically, so parallel agent builds stop filling the root disk
- [@moq/net's WebSocket receive windows match Rust](/quest/m0/qmux-receive-windows.md) - WebSocket JS throughput stops being capped below Rust's
35 changes: 0 additions & 35 deletions quest/m0/kio-waiter-records.md

This file was deleted.

7 changes: 7 additions & 0 deletions quest/m0/test-flakes-2/dispatch-query-terminate.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,10 @@ that PR does not touch `sh/`. It is skipped meanwhile, per AGENTS.md's rule
to disable a flaky test at once. Reproduce it under load (repeated runs beside
a CPU-heavy job), find the deadline or ordering it races on, and assert on
the event instead of a wait, as the epic's rules say.

A lead to verify: pullfrog reproduced a failure 1 in 420 runs on bash 5.3.20,
where the EXIT trap's `wait` returns a cancelled lane's 143 and `set -e` skips
`report`. Its fix, `wait || true` in `sh/dispatch.sh`'s EXIT trap
([e6690144ca](https://github.com/moq-dev/moq/commit/e6690144ca14f9ec8c48a56b4ed26c5a65a47708)),
passed 420/420 but was dropped from
[moq-dev/moq#7789](https://github.com/moq-dev/moq/pull/7789) as out of scope.
3 changes: 1 addition & 2 deletions quest/m1/release-perf/relay-cpu-bisect.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Decided 2026-11-03 (maintainer, on [moq-dev/moq#7605](https://github.com/moq-dev
- Accept all of them now.
- Keep investigating all of them.
2. The waiter's list records:
- ✅ A follow-up at m0, as a correctness bug: [a waiter's record limit changes no behavior](/quest/m0/kio-waiter-records.md).
- ✅ A correctness follow-up at m0, completed in [moq-dev/moq#7789](https://github.com/moq-dev/moq/pull/7789): the metadata/abort and test teardown races are fixed, and waiters remember 16 lists.
- Leave the limit at 8.
3. `group::Consumer` moves:
- ✅ A follow-up in this epic, implemented in [moq-dev/moq#7791](https://github.com/moq-dev/moq/pull/7791).
Expand All @@ -56,7 +56,6 @@ Public API: none expected. Wire: none.

## Related

- [A waiter's record limit changes no behavior](/quest/m0/kio-waiter-records.md) - found here; the session's waiter registration cost waits on it

- [Calibrate the paired nightly benchmarks](/quest/m1/bench-ci.md) - the nightly series that should have caught this growth
- [Performance audit](/quest/m2/audit-perf.md) - counts instructions when bisecting, for the same reason
4 changes: 2 additions & 2 deletions quest/m2/perf/kio-waiter-costs.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Three kio waiter costs are measured and fixed where they are real, each with before/after instruction counts:

- A long-lived waiter parked on more than `PARKED` (8) lists is no longer discarded and rebuilt on every poll, measured on a relay session driver.
- A long-lived waiter parked on more than `PARKED` (16) lists is no longer discarded and rebuilt on every poll, measured on a relay session driver.
- IETF fetch serving keeps one close watch per fetch stream instead of building one per chunk.
- A short-lived `kio::log` costs measurably less than about 1.3k instructions to set up and clean up, or the measurement shows why it can't.

Expand All @@ -14,7 +14,7 @@ From [moq-dev/moq#7556](https://github.com/moq-dev/moq/pull/7556) (a measured no

Decided 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run): m2, in the perf epic, one quest for the three, since they share kio's waiter and log internals and one measurement setup.

- **`Park::hold`'s retire rule** (`rs/kio/src/waiter.rs`): `hold` replaces a waiter that is not `reusable()`, and a waiter parked on more than `PARKED = 8` lists is not. #7556 traced about 4 `Arc<Waker>` per group on a session driver (`Handle::run` / `time::run`) to this: the new waiter's first registration lands on the session-close `Bridge` (moq-uring `adapter.rs` and moq-tokio `owned.rs` `poll_closed`). A relay session driver parks on many more lists, so measure there first (allocations and instructions per delivered group over `just bench-runtime` or a session bench). Then choose between raising `PARKED`, changing the retire rule, or leaving it, on the numbers.
- **`Park::hold`'s retire rule** (`rs/kio/src/waiter.rs`): `hold` replaces a waiter that is not `reusable()`, and a waiter parked on more than `PARKED = 16` lists is not ([moq-dev/moq#7789](https://github.com/moq-dev/moq/pull/7789) raised it from 8 to fix subscription ordering bugs). #7556 traced about 4 `Arc<Waker>` per group on a session driver (`Handle::run` / `time::run`) to this: the new waiter's first registration lands on the session-close `Bridge` (moq-uring `adapter.rs` and moq-tokio `owned.rs` `poll_closed`). A relay session driver parks on many more lists, so measure there first (allocations and instructions per delivered group over `just bench-runtime` or a session bench). Then choose between raising `PARKED`, changing the retire rule, or leaving it, on the numbers.
- **IETF fetch close watch** (`rs/moq-net/src/ietf/publisher.rs`): the fetch chunk loop pins a fresh `stream.closed()` per chunk (about lines 2387, 2450, and 2488), so each chunk builds a `kio::wait` whose waiter registers on the send stream's close `Bridge`. Hoist it to one per fetch stream. #7556 did not measure this path; a FETCH delivery bench with many chunks per stream shows the before/after.
- **`kio::log` lifetime cost**: #7531 attributed about 1.3k instructions per short-lived log to its epoch pin, flush, and collect, plus segment retirement on drop. That matters to any design holding one log per stream or per group. Profile a create/append/drop microbench, and cut what doesn't need to run for a log that never grew past its first segment.

Expand Down
6 changes: 3 additions & 3 deletions quest/m2/performance-survey/task-capacity.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ capacity is worth what it saves.

[kio::Tasks](/rs/kio/src/task.rs) keeps every slot it ever grew: the slab
entry (`Option<Occupant<T>>`, whose `Park` holds an inline `Waiter` of about
100 B), the per-slot waker allocation, and the wake bitset. The CPU half of
164 B), the per-slot waker allocation, and the wake bitset. The CPU half of
this quest landed in [moq-dev/moq#7513](https://github.com/moq-dev/moq/pull/7513).
A three-level wake index means an owner poll visits only woken words, so poll
cost no longer depends on the historical peak. The memory half remains.
Expand All @@ -26,8 +26,8 @@ Decided 2026-11-01 by the maintainer on

The [`task_capacity` bench](/rs/kio/benches/task_capacity.rs) reports
retained bytes after a burst. With an 8-byte task, the set keeps about
184 B per peak slot whatever is still live: 188,912 B after a 1,024-task
burst and 12,085,304 B after 65,536. A real task adds `size_of::<T>()`
248 B per peak slot whatever is still live: 254,448 B after a 1,024-task
burst and 16,279,608 B after 65,536. A real task adds `size_of::<T>()`
per slot, so measure the actual consumers (`GroupServe`, the io_uring worker's
`Task`, the session sets) before judging the cost. Dropping the set returns
the heap to baseline, so this is retained capacity, not a leak.
Expand Down
2 changes: 1 addition & 1 deletion rs/kio/benches/waiter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ fn bench_cancel(c: &mut Criterion) {
fn bench_park_cycle(c: &mut Criterion) {
let mut g = c.benchmark_group("waiter_park_cycle");
// Include broad polls that remain parked on many quiet lists.
for &lists in &[1usize, 2, 4, 8, 16] {
for &lists in &[1usize, 2, 4, 8, 16, 32] {
g.bench_with_input(BenchmarkId::from_parameter(lists), &lists, |b, &n| {
let cx = Context::from_waker(Waker::noop());
let mut park = Park::default();
Expand Down
6 changes: 3 additions & 3 deletions rs/kio/src/waiter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ pub struct Waiter {
noop: bool,
}

/// Lists a waiter remembers being parked on. A relay's busiest tasks park on up to 8.
const PARKED: usize = 8;
/// Lists a waiter remembers being parked on.
const PARKED: usize = 16;

impl Waiter {
/// Create a new waiter from an async [`Waker`].
Expand Down Expand Up @@ -82,7 +82,7 @@ impl Waiter {
/// Register this waiter with a [`WaiterList`] for future notification.
///
/// Delegates to [`WaiterList::register`], a no-op while the list still holds it.
/// Only the first 8 lists at once are remembered: a waiter kept across polls
/// Only the first 16 lists at once are remembered: a waiter kept across polls
/// that also sits on more appends a duplicate to those extras on every call.
pub fn register(&self, list: &mut WaiterList) {
list.register(self);
Expand Down
Loading
Loading