From 54fd21e081b7c223eab8ef8a03428cebe3361abc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 8 Oct 2026 16:25:35 -0700 Subject: [PATCH 1/8] quest: claim m0/serve-budget Co-Authored-By: Claude Opus 5.5 From 17dc4ef532023ab2c1e564493dd55bba36649f34 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 8 Oct 2026 18:44:38 -0700 Subject: [PATCH 2/8] feat(kio): bound a busy task's turn with a cooperative budget A serve loop that always has another group ready held its thread for as long as groups kept arriving, starving the QUIC driver on the same runtime. kio::coop gives each task a fixed budget per turn: a kio poll that would return Ready spends a unit, and once none are left it wakes the task and returns Pending. Tasks refills it per child; moq_net::time::run, the sim driver, and the uring worker refill it per runtime poll. Three moq-net sites read a yield as a verdict and are now plain reads or noop probes: the cursor's drift and staleness checks (dropped a taken group), the boundary probe after the groups finish (skipped SUBSCRIBE_END), and the expiry check's started test. Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 5 +- quest/m0/ffi-publisher-stall.md | 10 +- quest/m0/serve-budget.md | 98 ------------- quest/m1/audio-group-duration.md | 1 - quest/m1/ffi-runtime.md | 3 +- quest/m1/perf/uring-quiescence.md | 4 - quest/m1/track-tail-interop.md | 12 +- rs/kio/README.md | 2 + rs/kio/src/consumer.rs | 7 + rs/kio/src/coop.rs | 221 ++++++++++++++++++++++++++++++ rs/kio/src/lib.rs | 5 + rs/kio/src/producer.rs | 8 ++ rs/kio/src/queue.rs | 21 ++- rs/kio/src/shared.rs | 4 + rs/kio/src/task.rs | 50 ++++++- rs/kio/src/waiter.rs | 20 ++- rs/kio/src/weak.rs | 4 + rs/moq-net/benches/session.rs | 73 ++++++++-- rs/moq-net/sim/src/lib.rs | 31 +++-- rs/moq-net/src/lite/publisher.rs | 104 +++++++++++++- rs/moq-net/src/model/track.rs | 105 +++++++++++--- rs/moq-net/src/time.rs | 42 +++--- rs/moq-net/src/util.rs | 19 +++ rs/moq-uring/src/worker.rs | 22 +-- 24 files changed, 664 insertions(+), 207 deletions(-) delete mode 100644 quest/m0/serve-budget.md create mode 100644 rs/kio/src/coop.rs diff --git a/quest/m0/README.md b/quest/m0/README.md index 1a36af68210..80c24a72987 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -48,7 +48,7 @@ publishers mint epochs. Backport patches cut from `release` don't carry Liveness: a serve loop with work always ready never yields, which starved an FFI publisher's QUIC driver and fails hosted Interop's go lanes (found -2026-10-08 landing #4225). The [serve budget](/quest/m0/serve-budget.md) +2026-10-08 landing #4225). kio's cooperative budget (`kio::coop`) now bounds every kio task's loop. The Go and Python interop cells it fails moved here from m1 the same day, because the stall masks interop on every wire PR and hides as slow passes; the harness now fails a cell whose @@ -61,13 +61,12 @@ remain and no release waits on them. ## Required -- [Serve budget](/quest/m0/serve-budget.md) - a kio task that always has work ready yields after a budget, so a fast publisher can't starve its own QUIC driver - [Draft-22 LOCATION_FILTER](/quest/m0/ietf-location-filter-22.md) - moqt-22 LOCATION_FILTER carries its type instead of a Length in Rust and JS, so a draft-22 peer reads our Next Object correctly - [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.4, before Seattle - [Capped stream END_OF_GROUP](/quest/m0/ietf-end-of-track-location.md) - a stream capped by the subscription's end Location never claims END_OF_GROUP; moving End of Track's Location is deferred - [End of Group status](/quest/m0/ietf-end-of-group-status.md) - an End of Group status on a stream whose header already marks the group's end is accepted, so imquic's last object per group arrives - [Demand lost wake](/quest/m0/demand-lost-wake.md) - a reader that comes and goes between a demand poll and its re-read never leaves a front or `broadcast::Demand` without a wake -- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - every Go and Python publisher cell passes reliably once the serve budget lands, and a cell fails when a connection idles out +- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - every Go and Python publisher cell passes reliably now that kio's cooperative budget landed, and a cell fails when a connection idles out - [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to - [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines - [JS track takeover](/quest/m0/js-track-takeover.md) - JS `createTrack` answers a queued request and continues its sequences, as Rust does, so a re-announced `@moq/publish` catalog never restarts its groups diff --git a/quest/m0/ffi-publisher-stall.md b/quest/m0/ffi-publisher-stall.md index 1dd2616da1d..8584518b526 100644 --- a/quest/m0/ffi-publisher-stall.md +++ b/quest/m0/ffi-publisher-stall.md @@ -16,14 +16,14 @@ cell. The cause is known: a serve loop with work always ready never yields and starves moq-ffi's single runtime thread until the relay times the publisher -out (found 2026-10-08 landing #4225). [Serve budget](/quest/m0/serve-budget.md) -fixes it, so this quest starts once it lands and no bisect is needed. +out (found 2026-10-08 landing #4225). kio's cooperative budget (`kio::coop`) +fixes it, so no bisect is needed. Decided 2026-10-08: - In m0, moved from m1 and widened from the browser cells: the stall masks interop on every wire PR and may disconnect real FFI publishers. -- Re-check every Go and Python publisher cell once the serve budget lands. +- Re-check every Go and Python publisher cell now that the budget landed. The #4225 root-cause comment saw a 64-iteration yield still time the publisher out, so if a cell still stalls, find the rest of the cause. Never raise a timeout or add a retry. @@ -52,10 +52,6 @@ Harness facts (2026-10-08): Public API: none expected. Wire: none. -## Required - -- [Serve budget](/quest/m0/serve-budget.md) - fixes the stall's cause; this quest verifies the cells and hardens the harness - ## Related - [FFI runtime](/quest/m1/ffi-runtime.md) - FFI apps run on more than one thread, which also hides the stall diff --git a/quest/m0/serve-budget.md b/quest/m0/serve-budget.md deleted file mode 100644 index f76ec2bfd3f..00000000000 --- a/quest/m0/serve-budget.md +++ /dev/null @@ -1,98 +0,0 @@ -# [M] A busy task yields within a budget - -## Goal - -No kio task can hold its thread with an unbounded loop of async kio -readiness polls while work keeps arriving. Synchronous `try_*` and peek -loops, and code outside a `kio::coop::budget`, stay unbounded, as today. A serve loop that always has another group ready yields after a -fixed budget of progress and resumes on its next poll, so the QUIC driver, -timers, and the session's other tasks run in between, on every runtime -(tokio multi-thread and current-thread, wasm, and the io_uring worker). - -Found while landing #4225 (2026-10-08): the go interop publisher writes -2.5 ms Opus frames, one group each (about 400 groups/s). When a relay -subscribes, `RequestServe::poll_serve` (`rs/moq-net/src/lite/publisher.rs`) -keeps returning `Continue` and one poll ran over 4,096 iterations. moq-ffi's -current-thread runtime then starves noq's connection driver until the relay -times the publisher out after 10 s. Hosted Interop's `go -> *` lanes fail on -#4225 and stall for about 10 s on `main`. The gate is the mocked serve test -below, not those lanes: the m1 quests each hide the stall on their own, and -the #4225 root-cause comment saw a 64-iteration yield still time the -publisher out, so a yield alone may not be the whole fix. Re-check the lanes -and #4225 when this lands. - -Not here: moq-net spawning tasks (it stays poll-only by design, #2302, -#2736, #3825), the FFI runtime flavor, or the audio grouping default; those -are separate quests below. - -## Plan - -Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that -added this quest): - -- The bound is generic, modeled on tokio's cooperative budget, and lives in - kio, which takes no runtime dependency. Tokio's budget cannot see this loop, - because the work goes through kio primitives, never a tokio resource. - Rejected: per-loop budgets in the two serve loops (not generic), and - bridging kio to `tokio::task::coop` (leaves wasm and io_uring unbounded). -- Mechanics: - - a per-thread budget in kio. An async kio readiness poll that makes - progress spends one unit, and refunds it if the operation ends `Pending` - anyway; - - an exhausted budget wakes the current task - (`waiter.waker().wake_by_ref()`) and returns `Pending`; - - synchronous `try_*` and peek APIs, and any poll with - `kio::Waiter::noop()`, spend nothing, since their callers read "nothing - now" as empty, not as "come back". `Waiter` carries an explicit noop - flag for this, since comparing wakers is not reliable. -- A budget `Pending` must only postpone, never change a decision or lose - state. Spend only at state-safe boundaries, and audit every caller that - reads `Pending` as a value or commits state before a later poll. Known - sites: `TrackRun::start` reads Largest for SUBSCRIBE_START with - `Pending => None`; `poll_recv_next` probes `poll_finished` after - `poll_recv_group` ends, and a `Pending` there skips SUBSCRIBE_END; - `GroupServe` reads `poll_expired` as a bool; and `Cursor::poll_recv_group` - advances `index` or takes a parked group before `poll_stale`, so a - `Pending` there drops the group. Tests drive each site to exhaustion and - check every eligible group is delivered exactly once, fresh and parked. -- The budget is per kio task: `Tasks::poll` refills it for each child it - polls, and `moq_net::time::run` refills it once per `kio::wait` poll, - outside its loop, so an always-ready timer can't refill per iteration. A child that runs out lands in the next pass by - `Tasks`' one-snapshot rule (`a_self_waking_task_yields_to_the_owner`), and - the owner returns to its runtime after the pass. Work per turn scales with - the children that were ready, so a busy fan-out session is not throttled. - If [Run to quiescence](/quest/m1/perf/uring-quiescence.md) lets a turn run - several passes, each pass refills every child it polls, so a turn is - bounded by passes times ready children times the budget, and the owner - returns only after the last pass. Owners that poll their `Tasks` more than - once per poll (`Publisher::poll`, `SubscribeServe::poll_step` inside - `RequestServe::poll_serve`'s loop) compound the same way: a nested serve - can spend about the budget squared in one runtime poll. That is bounded, - and the bench decides whether it matters. - Rejected: one budget shared by the whole runtime task (tokio's model), - which caps a session with many subscriptions at one budget per turn. -- Refill mirrors tokio: `kio::coop::budget(f)` sets a fresh budget and - restores the previous value on exit, so nesting refills. Public, documented - as called only where a driver hands a task its turn. Code outside any - `budget` call is unconstrained, as today. The moq-uring worker's pass calls - it too if it polls anything outside `Tasks`. -- The size starts at 128 (tokio's) and is a constant, not configurable. A - bench picks it: sweep unbounded, 32, 128, and 512 against `main` and keep - the smallest whose throughput matches unbounded. -- Tests, with mocked time: a producer appends 10,000 groups during a serve - with many groups in flight, the serve poll returns `Pending` while a - sibling task and the transport make progress (fails on `main`), and the - units spent per runtime poll are counted against the bound; kio unit tests - for spend, refund, exhaustion, and nested refill. -- Bench: extend `rs/moq-net/benches/session.rs` swept over groups/s and - subscriptions per session, and kio's `channel` and `tasks` benches for the per-operation cost - of the thread-local. - -Public API: `kio::coop::budget` (new). Wire: none. - -## Related - -- [FFI runtime](/quest/m1/ffi-runtime.md) - FFI apps run on more than one thread -- [Audio group duration](/quest/m1/audio-group-duration.md) - small audio frames stop minting a group each -- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - verifies the Go and Python interop cells once this lands, and fails a cell whose connection idles out -- [Run to quiescence](/quest/m1/perf/uring-quiescence.md) - its pass count bounds passes per turn; this budget bounds one task's loop, so they compose diff --git a/quest/m1/audio-group-duration.md b/quest/m1/audio-group-duration.md index 2cb9203a659..925efdf99f3 100644 --- a/quest/m1/audio-group-duration.md +++ b/quest/m1/audio-group-duration.md @@ -32,5 +32,4 @@ Public API: a changed default in moq-audio and `@moq/publish`. Wire: none. ## Related -- [Serve budget](/quest/m0/serve-budget.md) - bounds the serve loop whatever the group rate - [FFI codec namespaces](/quest/m1/ffi-shape/codec.md) - owns the FFI encoder's frame duration default, a separate knob diff --git a/quest/m1/ffi-runtime.md b/quest/m1/ffi-runtime.md index ba15c0bd0c0..a5d59ae03e5 100644 --- a/quest/m1/ffi-runtime.md +++ b/quest/m1/ffi-runtime.md @@ -21,7 +21,7 @@ added this quest): - Use `new_multi_thread().enable_all()` with tokio's default worker count. No configuration knob until an embedder needs one. Rejected: a small fixed pool, and keeping `current_thread` behind the - [serve budget](/quest/m0/serve-budget.md) alone. + kio's cooperative budget (`kio::coop`) alone. - Keep the "foreign caller's thread never drives our futures" property: work still runs via `Task::run` and `detached`. Check the exported async fns that await in place (`subscribe_catalog`, `subscribe_track`, `fetch_group`, @@ -51,6 +51,5 @@ Public API: callbacks may fire concurrently on any runtime thread ## Related -- [Serve budget](/quest/m0/serve-budget.md) - the root fix for a task hogging its thread - [Kotlin/JVM exit](/quest/m2/kt-jvm-exit.md) - runtime-thread shutdown - [Generated C bindings](/quest/m1/c/README.md) - inherits the moq-ffi runtime, so hand-written moq-c needs no change diff --git a/quest/m1/perf/uring-quiescence.md b/quest/m1/perf/uring-quiescence.md index ee9c09ce2c1..42254f0849b 100644 --- a/quest/m1/perf/uring-quiescence.md +++ b/quest/m1/perf/uring-quiescence.md @@ -52,7 +52,3 @@ meaning with a budget of 1 and gains a sibling proving the budget bound. - [Hard fork](/quest/m1/quic/fork/README.md) - the noq driver this edits moves onto `moq-quic` (decided in the 2026-10-05 audit) - [One enter per turn](/quest/m1/perf/uring-one-enter.md) - the metrics and the submit placement this sweep is measured with - -## Related - -- [Serve budget](/quest/m0/serve-budget.md) - a per-task budget bounds one task's loop; this quest's pass count bounds passes per turn, and the sweep runs with both in place diff --git a/quest/m1/track-tail-interop.md b/quest/m1/track-tail-interop.md index 8531b9b21a4..ea101402a82 100644 --- a/quest/m1/track-tail-interop.md +++ b/quest/m1/track-tail-interop.md @@ -29,10 +29,10 @@ run 37139033242 (2026-10-03) failed `tail rust -> rust`, `rust -> js-native-node`, and `rust -> js-native-bun`. Merge `main` (after #4741 and #4813) into #4225, rerun, and record here whatever defect still fails. -Found 2026-10-08: #4225's `go -> *` lanes fail on the -[serve budget](/quest/m0/serve-budget.md) stall, where the go publisher's -serve loop starves its own QUIC driver until the relay times it out. Rerun -the lanes once that lands. +Found 2026-10-08: #4225's `go -> *` lanes fail on a stall where the go +publisher's serve loop starves its own QUIC driver until the relay times it +out. kio's cooperative budget (`kio::coop`) is meant to fix it; rerun the +lanes. Decided in the 2026-09-30 audit: the lite-07 drop case moved into [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md)'s tests, so the basic Rust and @@ -41,10 +41,6 @@ JS tail interop lands now instead of waiting on that [L] quest. QUIC on localhost rarely reorders, so this is a smoke check that the end is delivered and clean. The ordering race itself stays in the unit tests. -## Required - -- [Serve budget](/quest/m0/serve-budget.md) - is meant to fix the stall that fails #4225's go lanes; rerun them once it lands - ## Related - [SUBSCRIBE_DROP](/quest/m1/subscribe-drop.md) - owns the lite-07 drop case on top of this harness diff --git a/rs/kio/README.md b/rs/kio/README.md index bebde65b805..daa6172673e 100644 --- a/rs/kio/README.md +++ b/rs/kio/README.md @@ -18,6 +18,8 @@ The channel auto-closes when all producers are dropped. poll-native FIFO queue (bounded or unbounded) built in the same style. `Park` holds a waiter for as long as a poll stays pending, bridging a `std::task::Context` to kio's waiter-based polls when implementing `Future` or `poll_*` on top of kio channels. +`coop::budget` bounds the progress one task makes per turn, so a loop that always finds +work ready still yields to the rest of its runtime. It's used internally by [moq-net](https://github.com/moq-dev/moq/tree/main/rs/moq-net) and friends, but is generic enough to be useful on its own. diff --git a/rs/kio/src/consumer.rs b/rs/kio/src/consumer.rs index b9ba209ac1a..c6d4a5e6560 100644 --- a/rs/kio/src/consumer.rs +++ b/rs/kio/src/consumer.rs @@ -72,6 +72,9 @@ impl Consumer { /// channel is still open, registers the [`Waiter`] for notification. /// Returns `Err(`[`Ref`]`)` if the channel has been closed while the /// condition returned by `f` is still pending. + /// + /// A `Ready` from `f` spends a unit of the [cooperative budget](crate::coop); with + /// none left, its value is dropped and the task is woken to poll again. pub fn poll(&self, waiter: &Waiter, f: F) -> Poll>> where F: FnOnce(&Ref<'_, T>) -> Poll, @@ -80,6 +83,10 @@ impl Consumer { let consumer_state = Ref { state }; if let Poll::Ready(res) = f(&consumer_state) { + if !crate::coop::spend(waiter) { + drop(consumer_state); + return crate::coop::exhausted(waiter); + } return Poll::Ready(Ok(res)); } diff --git a/rs/kio/src/coop.rs b/rs/kio/src/coop.rs new file mode 100644 index 00000000000..cf0495cb36b --- /dev/null +++ b/rs/kio/src/coop.rs @@ -0,0 +1,221 @@ +//! Cooperative scheduling: bound how much one task does per turn. +//! +//! A task whose kio polls keep returning `Ready` never yields on its own: a serve +//! loop that always has another item ready holds its thread for as long as items +//! keep arriving, starving everything else on it. Tokio's own budget cannot see +//! that loop, since the work goes through kio rather than a tokio resource. +//! +//! [`budget`] gives a task a fixed number of units for its turn. Every kio poll +//! that would return `Ready` with progress (a value, a write guard, a queue item) +//! spends one. Once none are left, such a poll wakes the task and returns `Pending` +//! instead, so the task yields and picks up where it left off on its next turn. +//! A poll that returns `Pending` anyway spends nothing, and neither does one +//! passed [`Waiter::noop`], whose caller reads `Pending` as "nothing now", nor +//! the synchronous `try_*` and `read` methods. Waiting on a level (closure, demand) +//! is not progress, so those polls spend nothing either. +//! +//! Code outside any [`budget`] call is unconstrained. + +use std::cell::Cell; + +use crate::Waiter; + +/// Units a task gets per turn. The smallest a `session_delivery_burst` sweep over 32, +/// 128 (tokio's), and 512 found as fast as no budget at all. +pub(crate) const UNITS: u16 = 32; + +thread_local! { + /// Units left in the current turn, or `None` outside any [`budget`] call. + static LEFT: Cell> = const { Cell::new(None) }; +} + +/// Run `f` with a fresh budget, restoring the previous one afterwards. +/// +/// Call it only where a driver hands a task its turn: [`Tasks`](crate::Tasks) does +/// for each task it polls, and a runtime adapter does around each poll of its root. +/// Nested calls refill, so a child gets its own budget and its parent resumes with +/// whatever it had left. +pub fn budget(f: impl FnOnce() -> R) -> R { + struct Restore(Option); + + impl Drop for Restore { + fn drop(&mut self) { + LEFT.set(self.0); + } + } + + let _restore = Restore(LEFT.replace(Some(UNITS))); + f() +} + +/// Spend a unit for a poll about to return `Ready`. +/// +/// `false` means the budget ran out: the caller wakes `waiter`, once it released any +/// lock, and returns `Pending` without taking anything. +pub(crate) fn spend(waiter: &Waiter) -> bool { + if waiter.is_noop() { + return true; + } + LEFT.with(|left| match left.get() { + None => true, + Some(0) => false, + Some(units) => { + left.set(Some(units - 1)); + true + } + }) +} + +/// Yield for an exhausted budget: wake the task so it runs again next turn. +pub(crate) fn exhausted(waiter: &Waiter) -> std::task::Poll { + waiter.waker().wake_by_ref(); + std::task::Poll::Pending +} + +/// The units left in this turn, or `None` outside any [`budget`] call. +#[cfg(test)] +pub(crate) fn left() -> Option { + LEFT.get() +} + +#[cfg(all(test, not(loom)))] +mod tests { + use std::{ + sync::{ + Arc, + atomic::{AtomicUsize, Ordering}, + }, + task::{Poll, Wake, Waker}, + }; + + use super::*; + use crate::{Producer, Queue}; + + struct Count(AtomicUsize); + + impl Wake for Count { + fn wake(self: Arc) { + self.0.fetch_add(1, Ordering::SeqCst); + } + } + + fn counted() -> (Waiter, Arc) { + let count = Arc::new(Count(AtomicUsize::new(0))); + (Waiter::new(Waker::from(count.clone())), count) + } + + fn ready(value: &crate::Ref<'_, u32>) -> Poll { + Poll::Ready(**value) + } + + #[test] + fn unconstrained_outside_a_budget() { + let producer = Producer::new(1u32); + let consumer = producer.consume(); + let (waiter, _) = counted(); + for _ in 0..1_000 { + assert!(matches!(consumer.poll(&waiter, ready), Poll::Ready(Ok(1)))); + } + assert_eq!(left(), None); + } + + #[test] + fn ready_spends_and_pending_does_not() { + let producer = Producer::new(1u32); + let consumer = producer.consume(); + let (waiter, _) = counted(); + budget(|| { + assert!(consumer.poll(&waiter, ready).is_ready()); + assert_eq!(left(), Some(UNITS - 1)); + assert!(consumer.poll(&waiter, |_| Poll::<()>::Pending).is_pending()); + assert_eq!(left(), Some(UNITS - 1), "a pending poll must not spend"); + }); + } + + #[test] + fn exhaustion_yields_and_wakes() { + let producer = Producer::new(1u32); + let consumer = producer.consume(); + let (waiter, wakes) = counted(); + budget(|| { + for _ in 0..UNITS { + assert!(consumer.poll(&waiter, ready).is_ready()); + } + assert_eq!(wakes.0.load(Ordering::SeqCst), 0); + assert!(consumer.poll(&waiter, ready).is_pending()); + assert_eq!(wakes.0.load(Ordering::SeqCst), 1, "a yield must wake the task"); + + // Waiting on a level spends nothing, so it still answers. + producer.close().ok().expect("open"); + assert!(consumer.poll_closed(&waiter).is_ready()); + }); + } + + /// A queue item is never taken by a poll that then yields. + #[test] + fn exhaustion_takes_nothing() { + let queue = Queue::new(); + let (waiter, _) = counted(); + budget(|| { + for item in 0..=u32::from(UNITS) { + queue.try_push(item).expect("open"); + } + for item in 0..u32::from(UNITS) { + assert_eq!(queue.poll_pop(&waiter), Poll::Ready(Ok(item))); + } + assert!(queue.poll_pop(&waiter).is_pending()); + assert!(queue.poll_push_with(&waiter, || 99).is_pending()); + }); + assert_eq!( + queue.try_pop(), + Ok(Some(u32::from(UNITS))), + "the yield left the item queued" + ); + assert_eq!(queue.try_pop(), Ok(None), "the yield pushed nothing"); + } + + #[test] + fn noop_waiter_spends_nothing() { + let producer = Producer::new(1u32); + let consumer = producer.consume(); + let waiter = Waiter::noop(); + budget(|| { + for _ in 0..1_000 { + assert!(consumer.poll(&waiter, ready).is_ready()); + } + assert_eq!(left(), Some(UNITS)); + // A clone of the noop waiter is still one. + assert!(consumer.poll(&waiter.clone(), ready).is_ready()); + assert_eq!(left(), Some(UNITS)); + }); + } + + #[test] + fn nested_budgets_refill_and_restore() { + let producer = Producer::new(1u32); + let consumer = producer.consume(); + let (waiter, _) = counted(); + budget(|| { + assert!(consumer.poll(&waiter, ready).is_ready()); + budget(|| { + assert_eq!(left(), Some(UNITS), "a nested budget refills"); + for _ in 0..UNITS { + assert!(consumer.poll(&waiter, ready).is_ready()); + } + assert!(consumer.poll(&waiter, ready).is_pending()); + }); + assert_eq!(left(), Some(UNITS - 1), "the outer budget resumes where it was"); + }); + assert_eq!(left(), None); + } + + #[test] + fn a_panic_restores_the_outer_budget() { + budget(|| { + let inner = std::panic::catch_unwind(|| budget(|| panic!("boom"))); + assert!(inner.is_err()); + assert_eq!(left(), Some(UNITS)); + }); + assert_eq!(left(), None); + } +} diff --git a/rs/kio/src/lib.rs b/rs/kio/src/lib.rs index 40f597acbf7..b0a6785473f 100644 --- a/rs/kio/src/lib.rs +++ b/rs/kio/src/lib.rs @@ -12,6 +12,9 @@ //! [`Queue`] is a poll-native FIFO queue built in the same style: role-less //! clone-able handles, bounded or unbounded, with separate wake lists for the //! push and pop sides. +//! +//! [`coop`] bounds how much one task does per turn, so a loop that always finds +//! work ready still yields. use std::{ fmt, @@ -24,6 +27,8 @@ mod lock; mod sync; mod waiter; +pub mod coop; + mod consumer; mod pending; mod producer; diff --git a/rs/kio/src/producer.rs b/rs/kio/src/producer.rs index c1564d0293f..83199be73c3 100644 --- a/rs/kio/src/producer.rs +++ b/rs/kio/src/producer.rs @@ -135,6 +135,10 @@ impl Producer { let mut guard = Ref { state }; match f(&guard) { + Poll::Ready(()) if !crate::coop::spend(waiter) => { + drop(guard); + crate::coop::exhausted(waiter) + } // Upgrade the Ref to a Mut, keeping the same lock guard. Poll::Ready(()) => Poll::Ready(Ok(Mut::new(guard.state))), Poll::Pending => { @@ -158,6 +162,10 @@ impl Producer { let mut guard = Ref { state }; if let Poll::Ready(res) = f(&guard) { + if !crate::coop::spend(waiter) { + drop(guard); + return crate::coop::exhausted(waiter); + } return Poll::Ready(Ok(res)); } diff --git a/rs/kio/src/queue.rs b/rs/kio/src/queue.rs index ef16c941247..367a7dfdf2c 100644 --- a/rs/kio/src/queue.rs +++ b/rs/kio/src/queue.rs @@ -140,6 +140,10 @@ impl Queue { waiter.register(&mut state.waiters_push); return Poll::Pending; } + if !crate::coop::spend(waiter) { + drop(state); + return crate::coop::exhausted(waiter); + } let item = make(); state.queue.push_back(item); state.waiters_pop.take() @@ -183,14 +187,19 @@ impl Queue { pub fn poll_pop(&self, waiter: &Waiter) -> Poll> { let (item, mut waiters) = { let mut state = self.state.lock(); - match state.queue.pop_front() { - Some(item) => (item, state.waiters_push.take()), - None if state.closed => return Poll::Ready(Err(Closed)), - None => { - waiter.register(&mut state.waiters_pop); - return Poll::Pending; + if state.queue.is_empty() { + if state.closed { + return Poll::Ready(Err(Closed)); } + waiter.register(&mut state.waiters_pop); + return Poll::Pending; + } + if !crate::coop::spend(waiter) { + drop(state); + return crate::coop::exhausted(waiter); } + let item = state.queue.pop_front().expect("checked non-empty"); + (item, state.waiters_push.take()) }; waiters.wake(); Poll::Ready(Ok(item)) diff --git a/rs/kio/src/shared.rs b/rs/kio/src/shared.rs index b81123ef3c5..b60ce9ebb22 100644 --- a/rs/kio/src/shared.rs +++ b/rs/kio/src/shared.rs @@ -69,6 +69,10 @@ impl Shared { state: self.state.lock(), }; match f(&guard) { + Poll::Ready(()) if !crate::coop::spend(waiter) => { + drop(guard); + crate::coop::exhausted(waiter) + } // Upgrade the Ref to a Mut, keeping the same lock guard. Poll::Ready(()) => Poll::Ready(Mut::new(guard.state)), Poll::Pending => { diff --git a/rs/kio/src/task.rs b/rs/kio/src/task.rs index 0637818bfe4..a3948e41d23 100644 --- a/rs/kio/src/task.rs +++ b/rs/kio/src/task.rs @@ -245,6 +245,9 @@ impl> Tasks { /// such an arm with [`is_empty`](Self::is_empty). /// `waiter` is registered for the next task wake or push; the tasks /// themselves park on their own wakers. + /// + /// Each task polls under a fresh [cooperative budget](crate::coop), so one that + /// always finds work ready yields to the next pass instead of holding the owner. pub fn poll(&mut self, waiter: &Waiter) -> Poll<()> { { let mut parent = self.shared.parent.lock().unwrap(); @@ -305,7 +308,9 @@ impl> Tasks { } let cx = Context::from_waker(&self.wakers[index]); let child_waiter = occupant.park.hold(&cx); - if occupant.task.poll(child_waiter).is_ready() { + // Each task gets a turn of its own: one that spends its budget + // self-wakes into the next pass, after the owner's other arms run. + if crate::coop::budget(|| occupant.task.poll(child_waiter)).is_ready() { self.children[index] = None; self.free.push(index); self.len -= 1; @@ -583,6 +588,49 @@ mod tests { ); } + /// A task that always finds work ready yields once its budget runs out, lands in + /// the next pass, and leaves its sibling's budget untouched. + #[test] + fn a_busy_task_yields_within_its_budget() { + fn busy(work: crate::Consumer, done: Arc) -> impl FnMut(&Waiter) -> Poll<()> { + move |waiter| { + loop { + if done.load(Ordering::SeqCst) == 1_000 { + return Poll::Ready(()); + } + // Always ready: only the budget ends this loop. + std::task::ready!(work.poll(waiter, |value| Poll::Ready(**value))).ok(); + done.fetch_add(1, Ordering::SeqCst); + } + } + } + + let work = crate::Producer::new(0u32); + let (a, b) = (Arc::new(AtomicUsize::new(0)), Arc::new(AtomicUsize::new(0))); + let mut tasks = Tasks::new(); + tasks.push(busy(work.consume(), a.clone())); + tasks.push(busy(work.consume(), b.clone())); + + let waiter = Waiter::noop(); + assert!(tasks.poll(&waiter).is_pending()); + let units = crate::coop::UNITS as usize; + assert_eq!(a.load(Ordering::SeqCst), units, "a yields after its budget"); + assert_eq!(b.load(Ordering::SeqCst), units, "b gets a budget of its own"); + + let mut passes = 1; + loop { + passes += 1; + if tasks.poll(&waiter).is_ready() { + break; + } + } + assert_eq!( + passes, + 1_000usize.div_ceil(units), + "each pass serves one budget per task" + ); + } + /// Wake-all resources must not always hand the first chance to the lowest /// task slot. Rotating the leading task lets every contender make progress. #[test] diff --git a/rs/kio/src/waiter.rs b/rs/kio/src/waiter.rs index 0dcc4d8a5a7..ac3986df49d 100644 --- a/rs/kio/src/waiter.rs +++ b/rs/kio/src/waiter.rs @@ -42,6 +42,11 @@ pub struct Waiter { // A registration found no free slot in `parked`, so a list may hold this waiter // unrecorded and a later registration there would stack a duplicate. lost: AtomicBool, + + // Built by `Waiter::noop`, whose caller reads `Pending` as "nothing now", so the + // cooperative budget must never turn a `Ready` into one. Wakers can't be compared + // reliably, so the flag says it. + noop: bool, } /// Lists a waiter remembers being parked on. A relay's busiest tasks park on up to 8. @@ -55,12 +60,24 @@ impl Waiter { shared: OnceLock::new(), parked: Default::default(), lost: AtomicBool::new(false), + noop: false, } } /// Create a no-op waiter that discards registrations. + /// + /// Polls through it never yield to the [cooperative budget](crate::coop), since + /// nothing would wake them. pub fn noop() -> Self { - Self::new(Waker::noop().clone()) + Self { + noop: true, + ..Self::new(Waker::noop().clone()) + } + } + + /// Whether this is a [`Self::noop`] waiter, exempt from the cooperative budget. + pub(crate) fn is_noop(&self) -> bool { + self.noop } /// Register this waiter with a [`WaiterList`] for future notification. @@ -158,6 +175,7 @@ impl Clone for Waiter { shared: OnceLock::from(shared), parked: std::array::from_fn(|i| AtomicU64::new(self.parked[i].load(Ordering::Relaxed))), lost: AtomicBool::new(self.lost.load(Ordering::Relaxed)), + noop: self.noop, } } } diff --git a/rs/kio/src/weak.rs b/rs/kio/src/weak.rs index 2976e5e6a9f..b624ba422f5 100644 --- a/rs/kio/src/weak.rs +++ b/rs/kio/src/weak.rs @@ -277,6 +277,10 @@ impl ConsumerWeak { let consumer_state = Ref { state }; if let Poll::Ready(res) = f(&consumer_state) { + if !crate::coop::spend(waiter) { + drop(consumer_state); + return crate::coop::exhausted(waiter); + } return Poll::Ready(Ok(res)); } diff --git a/rs/moq-net/benches/session.rs b/rs/moq-net/benches/session.rs index c949251303b..ccd58c6cca2 100644 --- a/rs/moq-net/benches/session.rs +++ b/rs/moq-net/benches/session.rs @@ -81,6 +81,9 @@ struct Shape { frame: usize, /// Write one frame per round, as a live source does, instead of a whole group. paced: bool, + /// Whole groups each watched broadcast writes per round, so a serve loop finds a + /// backlog ready: the group rate a round stands in for. + burst: usize, } impl Shape { @@ -94,8 +97,15 @@ impl Shape { watch: 1, frame: 64, paced: false, + burst: 1, }; + /// What each viewer subscribes with: a burst is a backlog, which the default + /// subscription writes off as late. + fn subscription(&self) -> Option { + (self.burst > 1).then(|| track::Subscription::default().with_max_delay(Duration::from_secs(30))) + } + fn total(&self) -> usize { self.publishers * self.broadcasts } @@ -107,19 +117,28 @@ impl Shape { /// Payload bytes every viewer reads in one round. fn expected(&self) -> usize { - self.viewers * self.watch * self.frames() * self.frame + self.viewers * self.watch * self.burst * self.frames() * self.frame } fn id(&self, version: &str) -> String { format!( - "{version}/relays={}/publishers={}/broadcasts={}/viewers={}/watch={}/frame={}{}", + "{version}/relays={}/publishers={}/broadcasts={}/viewers={}/watch={}/frame={}{}{}", self.relays, self.publishers, self.broadcasts, self.viewers, self.watch, self.frame, - if self.paced { "/paced" } else { "" }, + if self.paced { + "/paced".to_string() + } else { + String::new() + }, + if self.burst > 1 { + format!("/burst={}", self.burst) + } else { + String::new() + }, ) } } @@ -289,11 +308,17 @@ struct Viewer { impl Viewer { /// Resolve `path` and subscribe to each of its `tracks`. - async fn watch(&mut self, path: &str, tracks: &[impl AsRef]) { + async fn watch(&mut self, path: &str, tracks: &[impl AsRef], subscription: Option) { let broadcast = self.origin.consume().routed_broadcast(path).await.unwrap(); for track in tracks { - self.subscribers - .push(broadcast.track(track.as_ref()).unwrap().subscribe(None).await.unwrap()); + self.subscribers.push( + broadcast + .track(track.as_ref()) + .unwrap() + .subscribe(subscription.clone()) + .await + .unwrap(), + ); } } } @@ -320,6 +345,7 @@ impl Room { shape.watch <= shape.total(), "a viewer can't watch more broadcasts than exist" ); + assert!(!shape.paced || shape.burst == 1, "a paced round writes one frame"); let mut cluster = Cluster::new(version, shape).await; // Viewer v watches a contiguous window starting at v * watch, so viewers @@ -337,7 +363,7 @@ impl Room { // publisher it watches whenever there is more than one. let mut viewer = cluster.join((viewer + 1) % shape.relays).await; for broadcast in broadcasts { - viewer.watch(&path(broadcast), &[TRACK]).await; + viewer.watch(&path(broadcast), &[TRACK], shape.subscription()).await; } viewers.push(viewer); } @@ -371,12 +397,17 @@ impl Room { return self.paced_round().await; } for &broadcast in &self.watched { - write_group(&self.cluster.tracks[broadcast], &self.payload, FRAMES); + for _ in 0..self.shape.burst { + write_group(&self.cluster.tracks[broadcast], &self.payload, FRAMES); + } } let mut bytes = 0; - for viewer in &mut self.viewers { - for subscriber in &mut viewer.subscribers { - bytes += read_group(subscriber).await; + // Round-robin, so no subscription's backlog waits on another's being read. + for _ in 0..self.shape.burst { + for viewer in &mut self.viewers { + for subscriber in &mut viewer.subscribers { + bytes += read_group(subscriber).await; + } } } assert_eq!(bytes, self.shape.expected()); @@ -472,7 +503,7 @@ fn join(c: &mut Criterion, name: &str, shapes: impl IntoIterator) let broadcast = hosted[iter % hosted.len()]; let start = Instant::now(); let mut viewer = cluster.join(relay).await; - viewer.watch(&path(broadcast), &[TRACK]).await; + viewer.watch(&path(broadcast), &[TRACK], None).await; let bytes = read_group(&mut viewer.subscribers[0]).await; elapsed += start.elapsed(); assert_eq!(bytes, FRAMES * shape.frame); @@ -560,7 +591,7 @@ impl DashRoom { let mut dash = cluster.join(0).await; for path in &paths { - dash.watch(path, &names).await; + dash.watch(path, &names, None).await; } let mut room = Self { @@ -696,7 +727,7 @@ impl ChurnRoom { let mut broadcasts = Vec::new(); for _ in 0..shape.sessions { let mut viewer = cluster.join(0).await; - viewer.watch(&path(0), &held).await; + viewer.watch(&path(0), &held, None).await; broadcasts.push(viewer.origin.consume().routed_broadcast(&path(0)).await.unwrap()); viewers.push(viewer); } @@ -852,6 +883,20 @@ fn session(c: &mut Criterion) { }), ); delivery(c, "frame", [64, 1024, 16 * 1024].map(|frame| Shape { frame, ..base })); + // One session serving a backlog on each of its subscriptions: the serve loop that + // kio's cooperative budget bounds per turn. + delivery( + c, + "burst", + [(1, 1), (1, 8), (1, 64), (8, 1), (8, 8)].map(|(watch, burst)| Shape { + publishers: 1, + broadcasts: 16, + viewers: 1, + watch, + burst, + ..base + }), + ); delivery( c, "relays", diff --git a/rs/moq-net/sim/src/lib.rs b/rs/moq-net/sim/src/lib.rs index 91ae336ae1a..629dfaa6de5 100644 --- a/rs/moq-net/sim/src/lib.rs +++ b/rs/moq-net/sim/src/lib.rs @@ -328,24 +328,27 @@ pub fn timeout(duration: Duration, future: F) -> impl Future(mut poll: impl FnMut(Instant, &kio::Waiter) -> Result, E> + Unpin) -> E { let runtime = current(); let mut timer: Option = None; + // One cooperative budget per poll, like `moq_net::time::run`. kio::wait(move |waiter| { - loop { - let at = match poll(runtime.now(), waiter) { - Ok(Some(at)) => at, - Ok(None) => { - timer = None; + kio::coop::budget(|| { + loop { + let at = match poll(runtime.now(), waiter) { + Ok(Some(at)) => at, + Ok(None) => { + timer = None; + return Poll::Pending; + } + Err(err) => return Poll::Ready(err), + }; + let sleep = timer.get_or_insert_with(|| sleep_until(at)); + if sleep.deadline() != at { + sleep.reset(at); + } + if waiter.poll_future(Pin::new(sleep)).is_pending() { return Poll::Pending; } - Err(err) => return Poll::Ready(err), - }; - let sleep = timer.get_or_insert_with(|| sleep_until(at)); - if sleep.deadline() != at { - sleep.reset(at); } - if waiter.poll_future(Pin::new(sleep)).is_pending() { - return Poll::Pending; - } - } + }) }) .await } diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index d777c681766..de04e242466 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -1882,6 +1882,31 @@ mod test { } } + /// A finished track still announces its boundary when the poll that finds the groups + /// over has run out of budget: a yield there must not read as "no boundary". + #[test] + fn a_budget_yield_keeps_the_boundary() { + let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); + for left in 0..4 { + let mut producer = track_producer("test"); + let mut subscriber = producer.subscribe(None); + producer.finish_at(0).unwrap(); + + let starved = kio::coop::budget(|| { + crate::util::leave_budget(left); + poll_recv_next(&mut subscriber, false, true, &waiter) + }); + let res = match starved { + Poll::Pending => kio::coop::budget(|| poll_recv_next(&mut subscriber, false, true, &waiter)), + res => res, + }; + assert!( + matches!(res, Poll::Ready(Ok(Recv::Boundary(0)))), + "left {left}: the boundary was skipped" + ); + } + } + /// A relay can ingest back-to-back groups micro-reordered (the upstream leg /// sends newest-first). The older group is cached and in demand, so serving /// must still deliver it; a sequence cursor would skip it permanently. @@ -2327,7 +2352,13 @@ fn poll_recv_next( } // No live data ready: report the boundary (if declared) before signalling Finished, so a // future boundary reaches the subscriber while the trailing groups are still in flight. - if emit_boundary && let Poll::Ready(res) = track.poll_finished(waiter) { + // Once the groups finished this is a probe whose answer decides between the two, so it + // must not yield to the cooperative budget; either way nothing is left to wait on. + let probe = match groups_finished { + true => &kio::Waiter::noop(), + false => waiter, + }; + if emit_boundary && let Poll::Ready(res) = track.poll_finished(probe) { return Poll::Ready(res.map(Recv::Boundary)); } if groups_finished { @@ -3649,6 +3680,77 @@ mod serve_group_test { group.finish().unwrap(); } + /// A serve loop that always has another group ready yields within kio's cooperative + /// budget, so a sibling (the transport's driver, say) runs between its turns. Without + /// it, one poll served every group the producer kept appending and starved the rest + /// of the runtime (a go publisher's 2.5 ms audio groups timed its session out). + #[test] + fn a_busy_serve_yields_to_its_siblings() { + /// kio's per-turn budget: each served group spends at least one unit. + const BUDGET: u64 = 32; + const GROUPS: u64 = 10_000; + const BATCH: u64 = 1_000; + + type Task = Box Poll<()>>; + + let track = track::Producer::new(Arc::new(broadcast::Info::default()), "test", None); + let subscriber = track.subscribe(track::Subscription::default().with_max_delay(Duration::from_secs(3600))); + // Unacknowledged FINs keep every served group in flight. + let log = Log::default(); + let (mut run, mut writer, _) = lite07_run(SinkSession::new(log.clone()).with_unacked_fin(), subscriber); + let opens = run.ctx.opens.clone(); + + let mut tasks: kio::Tasks = kio::Tasks::new(); + + // The producer appends a batch of ready groups on each of its turns. A clone, so the + // track outlives the task. + let mut producer = track.clone(); + let mut appended = 0; + tasks.push(Box::new(move |waiter: &kio::Waiter| { + for _ in 0..BATCH.min(GROUPS - appended) { + write_group(&mut producer, appended, appended); + appended += 1; + } + if appended == GROUPS { + return Poll::Ready(()); + } + waiter.waker().wake_by_ref(); + Poll::Pending + })); + + // Stands in for the transport's driver: it only needs a turn. + let turns = Arc::new(AtomicU64::new(0)); + let counted = turns.clone(); + tasks.push(Box::new(move |waiter: &kio::Waiter| { + counted.fetch_add(1, Ordering::Relaxed); + waiter.waker().wake_by_ref(); + Poll::Pending + })); + + tasks.push(Box::new(move |waiter: &kio::Waiter| { + run.poll(&mut writer, waiter).map(|res| res.expect("serve")) + })); + + let owner = kio::Waiter::noop(); + let mut polls = 0; + while opens.opened.load(Ordering::Relaxed) < GROUPS { + let before = opens.opened.load(Ordering::Relaxed); + assert!(tasks.poll(&owner).is_pending(), "the serve never ends"); + polls += 1; + let served = opens.opened.load(Ordering::Relaxed) - before; + assert!(served <= BUDGET, "one turn served {served} groups"); + assert_eq!(turns.load(Ordering::Relaxed), polls, "the sibling missed a turn"); + assert!(polls <= 2 * GROUPS, "the serve stalled"); + } + + // Every group was served exactly once, and nothing more is left to serve. + for _ in 0..4 { + assert!(tasks.poll(&owner).is_pending()); + } + assert_eq!(opens.opened.load(Ordering::Relaxed), GROUPS); + assert!(log.resets().is_empty(), "a group was reset: {:?}", log.resets()); + } + /// SUBSCRIBE_END counts the group streams opened, not the groups below the end: a /// group the track never produced has no stream and is not counted. #[moq_net_sim::test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index f63794fd6a1..4a181b556f5 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -3565,7 +3565,10 @@ impl group::Expiry for GroupExpiry { if let Some(reach) = successor.timestamp() { break reach; } - if successor.poll_started(waiter).is_pending() || successor.timestamp().is_none() { + // Registers for the first frame; the timestamp is the verdict, so a yield to the + // cooperative budget cannot read as "not started". + let _ = successor.poll_started(waiter); + if successor.timestamp().is_none() { return false; } }; @@ -3708,29 +3711,29 @@ impl Cursor { /// group is only ever judged against content that could actually be served in its /// place. Read fresh each poll, so a mid-stream [`Control::update`] applies /// to the very next group, and shared across every candidate that poll walks off, so - /// discarding a backlog of N groups costs one scan rather than N. Only ever - /// [`Poll::Ready`]; the track ending surfaces as the error the caller was going to - /// get anyway. - fn poll_drift(&self, cap: Option, waiter: &kio::Waiter) -> Poll> { + /// discarding a backlog of N groups costs one scan rather than N. A plain read, like + /// [`Self::is_stale`], so it never yields to the cooperative budget; the track ending + /// surfaces as the error the caller was going to get anyway. + fn drift(&self, cap: Option, waiter: &kio::Waiter) -> Drift { let mut max_delay = Duration::default(); let _ = self.subscription.poll(waiter, |subscription| { max_delay = subscription.max_delay; Poll::<()>::Pending }); - self.poll(waiter, |state| { - Poll::Ready(Ok(Drift { - budget: clamp_max_delay(max_delay, state.max_age_bound()), - edge: state.drift_edge(cap), - })) - }) + let state = self.state.read(); + Drift { + budget: clamp_max_delay(max_delay, state.max_age_bound()), + edge: state.drift_edge(cap), + } } /// Whether the drift budget says to skip `group`, against a [`Drift`] already resolved /// for this poll. - fn poll_stale(&self, group: &group::Consumer, drift: Drift, waiter: &kio::Waiter) -> Poll> { - self.poll(waiter, |state| { - Poll::Ready(Ok(state.is_stale(group.sequence, &drift.edge, drift.budget))) - }) + /// + /// A plain read, not a poll: the caller has already taken `group` off its cursor, so + /// yielding to the cooperative budget here would drop it. + fn is_stale(&self, group: &group::Consumer, drift: Drift) -> bool { + self.state.read().is_stale(group.sequence, &drift.edge, drift.budget) } fn with_expiry(&self, group: group::Consumer) -> group::Consumer { @@ -3765,7 +3768,7 @@ impl Cursor { .retain(|sequence, group| *sequence >= min_sequence && watch(group)); // One scan for the whole poll, so walking a backlog off stays linear in its size. - let drift = ready!(self.poll_drift(self.end_sequence, waiter))?; + let drift = self.drift(self.end_sequence, waiter); loop { // Re-offer the lowest parked group back inside the cap once it rises, @@ -3811,7 +3814,7 @@ impl Cursor { // Drop a group the drift budget has given up on and keep scanning, so one // poll walks a whole backlog off rather than handing it out group by group. - if ready!(self.poll_stale(&consumer, drift, waiter))? { + if self.is_stale(&consumer, drift) { self.stale.add(consumer.content()); continue; } @@ -3855,7 +3858,7 @@ impl Cursor { let mut floor = self.next_sequence.max(self.min_sequence); let end = self.end_sequence; // One scan for the whole poll, so walking a backlog off stays linear in its size. - let drift = ready!(self.poll_drift(end, waiter))?; + let drift = self.drift(end, waiter); let group = loop { let Some(producer) = ready!(self.poll(waiter, |state| state.poll_next_in_range(floor, end))?) else { @@ -3869,7 +3872,7 @@ impl Cursor { // Skip a group the budget has given up on and keep scanning, so one poll // walks a whole backlog off rather than handing it out group by group. - if ready!(self.poll_stale(&group, drift, waiter))? { + if self.is_stale(&group, drift) { // Not counted yet: a conviction is not permanent (the budget can widen, // the edge can be evicted), so the group may still be delivered. // Re-snapshot on every re-examination so the eventual count reflects the @@ -7985,6 +7988,70 @@ mod test { assert!(consumer.next_group().now_or_never().is_none(), "no more groups"); } + /// Read every group `poll` hands out, running out of budget at every position a + /// poll can: a yield must only postpone, so each group still arrives exactly once. + fn read_through_yields( + left: usize, + mut poll: impl FnMut(&kio::Waiter) -> Poll>>, + ) -> Vec { + let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); + let mut read = Vec::new(); + loop { + // Starved first, then a whole budget, so every read also makes progress. + let starved = kio::coop::budget(|| { + crate::util::leave_budget(left); + poll(&waiter) + }); + let res = match starved { + Poll::Pending => kio::coop::budget(|| poll(&waiter)), + res => res, + }; + match res { + Poll::Ready(Ok(Some(group))) => read.push(group.sequence), + Poll::Ready(Ok(None)) | Poll::Pending => return read, + Poll::Ready(Err(err)) => panic!("read failed: {err}"), + } + } + } + + /// Fresh and parked groups alike survive a poll that runs out of budget part way. + #[test] + fn a_budget_yield_never_drops_a_group() { + for left in 0..4 { + for ordered in [false, true] { + let producer = track_producer("test", None); + let mut subscriber = producer.subscribe(replay()); + subscriber.set_groups(..4); + for sequence in 0..8 { + let mut group = producer.create_group(group::Info { sequence }).unwrap(); + group + .write_frame(Timestamp::from_millis(sequence).unwrap(), b"x".as_slice()) + .unwrap(); + group.finish().unwrap(); + } + + let (mut plain, mut sequenced) = match ordered { + true => (None, Some(subscriber.ordered())), + false => (Some(subscriber), None), + }; + let mut read = Vec::new(); + // The first half is fresh, the rest parks above the cap until it lifts. + for lift in [false, true] { + if lift { + plain.iter_mut().for_each(|sub| sub.set_groups(..)); + sequenced.iter_mut().for_each(|sub| sub.set_groups(..)); + } + read.extend(read_through_yields(left, |waiter| match (&mut plain, &mut sequenced) { + (Some(sub), _) => sub.poll_recv_group(waiter), + (_, Some(sub)) => sub.poll_next_group(waiter), + _ => unreachable!(), + })); + } + assert_eq!(read, (0..8).collect::>(), "left {left}, ordered {ordered}"); + } + } + } + #[test] fn end_at_lower_than_cursor_parks_consumer() { let producer = track_producer("test", None); diff --git a/rs/moq-net/src/time.rs b/rs/moq-net/src/time.rs index 2bfcd5fffda..14a84b2516d 100644 --- a/rs/moq-net/src/time.rs +++ b/rs/moq-net/src/time.rs @@ -37,29 +37,33 @@ pub trait Driver { /// terminal error, [`Error::Closed`] for a clean finish. pub async fn run(mut driver: D) -> Error { let mut timer: Option>> = None; + // One budget per runtime poll, outside the loop: a timer that is always due must + // not refill it per pass. kio::wait(|waiter| { - loop { - let now = web_async::time::Instant::now(); - #[cfg(not(target_family = "wasm"))] - let now = now.into_std(); - let at = match driver.poll(now, waiter) { - Ok(Some(at)) => at, - Ok(None) => { - timer = None; + kio::coop::budget(|| { + loop { + let now = web_async::time::Instant::now(); + #[cfg(not(target_family = "wasm"))] + let now = now.into_std(); + let at = match driver.poll(now, waiter) { + Ok(Some(at)) => at, + Ok(None) => { + timer = None; + return Poll::Pending; + } + Err(err) => return Poll::Ready(err), + }; + #[cfg(not(target_family = "wasm"))] + let at = web_async::time::Instant::from_std(at); + let sleep = timer.get_or_insert_with(|| Box::pin(web_async::time::sleep_until(at))); + if sleep.deadline() != at { + sleep.as_mut().reset(at); + } + if waiter.poll_future(sleep.as_mut()).is_pending() { return Poll::Pending; } - Err(err) => return Poll::Ready(err), - }; - #[cfg(not(target_family = "wasm"))] - let at = web_async::time::Instant::from_std(at); - let sleep = timer.get_or_insert_with(|| Box::pin(web_async::time::sleep_until(at))); - if sleep.deadline() != at { - sleep.as_mut().reset(at); - } - if waiter.poll_future(sleep.as_mut()).is_pending() { - return Poll::Pending; } - } + }) }) .await } diff --git a/rs/moq-net/src/util.rs b/rs/moq-net/src/util.rs index c1eee584891..86eadb7de5b 100644 --- a/rs/moq-net/src/util.rs +++ b/rs/moq-net/src/util.rs @@ -258,6 +258,25 @@ impl Drop for TaskSet { } } +/// Spend all but `left` units of the current [`kio::coop::budget`] on reads that decide +/// nothing, so the next poll runs out wherever its `left + 1`th unit falls. +#[cfg(test)] +pub(crate) fn leave_budget(left: usize) { + let channel = kio::Producer::new(()); + let reader = channel.consume(); + // Not `noop`, which the budget exempts. + let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); + let mut units = 0usize; + kio::coop::budget(|| { + while reader.poll(&waiter, |_| Poll::Ready(())).is_ready() { + units += 1; + } + }); + for _ in 0..units.saturating_sub(left) { + let _ = reader.poll(&waiter, |_| Poll::Ready(())); + } +} + #[cfg(test)] mod tests { use std::sync::{ diff --git a/rs/moq-uring/src/worker.rs b/rs/moq-uring/src/worker.rs index 8573b50a4f5..d095d6e6f93 100644 --- a/rs/moq-uring/src/worker.rs +++ b/rs/moq-uring/src/worker.rs @@ -177,7 +177,8 @@ impl Worker { let cx = Context::from_waker(&waker); let waiter = self.park.hold(&cx); - if let Poll::Ready(value) = waiter.poll_future(future.as_mut()) { + // The root gets a turn like any task; `Tasks` budgets the spawned ones. + if let Poll::Ready(value) = kio::coop::budget(|| waiter.poll_future(future.as_mut())) { return Ok(value); } // `Ready` just means the set is drained; the waiter stays @@ -611,16 +612,19 @@ impl Handle { /// with its terminal error. pub async fn run(&self, mut driver: D) -> moq_net::Error { let mut timer = self.timer(); + // One cooperative budget per poll, outside the loop, like `moq_net::time::run`. kio::wait(|waiter| { - loop { - match driver.poll(Instant::now(), waiter) { - Ok(at) => timer.set(at), - Err(err) => return Poll::Ready(err), - } - if timer.poll(waiter).is_pending() { - return Poll::Pending; + kio::coop::budget(|| { + loop { + match driver.poll(Instant::now(), waiter) { + Ok(at) => timer.set(at), + Err(err) => return Poll::Ready(err), + } + if timer.poll(waiter).is_pending() { + return Poll::Pending; + } } - } + }) }) .await } From e7b967d5dbcd6223cef0ea49625b3053b7abc51c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 9 Oct 2026 00:29:42 -0700 Subject: [PATCH 3/8] fix(net): a cooperative-budget yield only postpones Every kio poll that yields to the budget drops a Ready it already had, so a caller that read the Pending as a verdict, or committed work in the poll closure, lost something. - kio::coop::poll_proceed: Pending (and a wake) once the turn's budget is spent, so a verdict read off a Pending waits for a fresh turn. - group reads: count a prefetched batch when it is filled, even on a yield, and never apply the drift budget or swap routes on a yielded read. - ordered reads: walk a stale backlog under one lock, so it spends one unit per delivered group instead of livelocking past 32 stale groups. - a source's serve loop (lite and ietf) no longer spins closing a source whose request poll keeps yielding. - the ietf track end, a trailing datagram, a tail settle, a fetch re-check, a resumed frame, a route fetch, and three timeouts no longer act on a yield. Co-Authored-By: Claude Opus 5.5 --- rs/kio/src/coop.rs | 35 ++++++++ rs/moq-net/src/ietf/control.rs | 2 + rs/moq-net/src/ietf/publisher.rs | 5 ++ rs/moq-net/src/ietf/subscriber.rs | 26 ++++++ rs/moq-net/src/lite/publisher.rs | 38 +++++++-- rs/moq-net/src/lite/subscriber.rs | 12 +++ rs/moq-net/src/model/frame.rs | 3 + rs/moq-net/src/model/group.rs | 67 +++++++++++++--- rs/moq-net/src/model/resume.rs | 2 + rs/moq-net/src/model/track.rs | 128 +++++++++++++++++++++++------- 10 files changed, 273 insertions(+), 45 deletions(-) diff --git a/rs/kio/src/coop.rs b/rs/kio/src/coop.rs index cf0495cb36b..03451870501 100644 --- a/rs/kio/src/coop.rs +++ b/rs/kio/src/coop.rs @@ -14,6 +14,9 @@ //! the synchronous `try_*` and `read` methods. Waiting on a level (closure, demand) //! is not progress, so those polls spend nothing either. //! +//! A yield only postpones, so a caller that reads a `Pending` as a verdict (blocked, +//! idle, not found) checks [`poll_proceed`] before acting on it. +//! //! Code outside any [`budget`] call is unconstrained. use std::cell::Cell; @@ -66,6 +69,20 @@ pub(crate) fn spend(waiter: &Waiter) -> bool { }) } +/// Hold off acting on a `Pending` while this turn's budget is spent. +/// +/// With the budget spent, a kio poll returns `Pending` even when it had an answer, so +/// a caller about to treat `Pending` as a verdict (blocked, idle, not found) checks this +/// first. `Pending` means the budget is spent: the task is woken, and the verdict waits +/// for a fresh turn. `Ready` means no poll this turn yielded, so its `Pending` was real. +/// Spends nothing. +pub fn poll_proceed(waiter: &Waiter) -> std::task::Poll<()> { + match waiter.is_noop() || LEFT.get() != Some(0) { + true => std::task::Poll::Ready(()), + false => exhausted(waiter), + } +} + /// Yield for an exhausted budget: wake the task so it runs again next turn. pub(crate) fn exhausted(waiter: &Waiter) -> std::task::Poll { waiter.waker().wake_by_ref(); @@ -174,6 +191,24 @@ mod tests { assert_eq!(queue.try_pop(), Ok(None), "the yield pushed nothing"); } + #[test] + fn proceed_holds_off_once_spent() { + let producer = Producer::new(1u32); + let consumer = producer.consume(); + let (waiter, wakes) = counted(); + assert!(poll_proceed(&waiter).is_ready(), "unconstrained outside a budget"); + budget(|| { + for _ in 0..UNITS { + assert!(poll_proceed(&waiter).is_ready()); + assert!(consumer.poll(&waiter, ready).is_ready()); + } + assert_eq!(left(), Some(0), "proceeding spends nothing"); + assert!(poll_proceed(&waiter).is_pending()); + assert_eq!(wakes.0.load(Ordering::SeqCst), 1, "holding off wakes the task"); + assert!(poll_proceed(&Waiter::noop()).is_ready(), "a noop waiter is exempt"); + }); + } + #[test] fn noop_waiter_spends_nothing() { let producer = Producer::new(1u32); diff --git a/rs/moq-net/src/ietf/control.rs b/rs/moq-net/src/ietf/control.rs index 3ba504d7e2b..031d9c37a09 100644 --- a/rs/moq-net/src/ietf/control.rs +++ b/rs/moq-net/src/ietf/control.rs @@ -138,6 +138,8 @@ impl Control { return Poll::Ready(Ok(state.request_id_next.increment())); } + // A check that yielded to the cooperative budget may have allowed it. + std::task::ready!(kio::coop::poll_proceed(waiter)); if timeout.poll(waiter).is_ready() { tracing::warn!("timed out waiting for MAX_REQUEST_ID"); return Poll::Ready(Err(Error::Cancel)); diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index b2ea00c2f8b..2ad7e6dc1d0 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -1618,6 +1618,8 @@ where if let Poll::Ready(joins) = joins { return Poll::Ready(Ok(joins.get(&subscribe_id).cloned().flatten())); } + // A lookup that yielded to the cooperative budget may have found it. + ready!(kio::coop::poll_proceed(waiter)); if deadline.poll(waiter).is_ready() { return Poll::Ready(if pending { Err(Error::Timeout) } else { Ok(None) }); } @@ -2717,6 +2719,9 @@ impl TrackServe { Poll::Ready(Ok(None)) => { // Datagrams written before the track finished still go out. self.poll_datagrams(waiter); + // Draining never looks again, so a yield to the cooperative budget (in + // the datagrams or the end) retries the whole step next turn. + ready!(kio::coop::poll_proceed(waiter)); self.draining = true; if let Poll::Ready(Ok(end)) = self.track.poll_finished(waiter) { self.end = Some(end); diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 6479900078f..53d0bd65b03 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -645,6 +645,8 @@ async fn resolve_track_alias( if let Poll::Ready(result) = resolved { return Poll::Ready(result.unwrap_or(Err(Error::Dropped))); } + // A lookup that yielded to the cooperative budget may have found it. + ready!(kio::coop::poll_proceed(waiter)); if timeout.poll(waiter).is_ready() { return Poll::Ready(Err(Error::NotFound)); } @@ -1742,6 +1744,8 @@ where if let Poll::Ready(next) = broadcast.poll_requested_track(waiter) { return Poll::Ready(Some(next)); } + // A yield to the cooperative budget, not an empty queue. + ready!(kio::coop::poll_proceed(waiter)); if route.as_ref().is_some_and(|route| route.poll_closed(waiter).is_ready()) { source.close(); continue; @@ -4372,6 +4376,28 @@ mod tests { assert_eq!(pending.await.unwrap(), RequestId(11)); } + /// An alias bound by the time the timeout fires still resolves when the lookup yields + /// to the cooperative budget: the yield is no "not found". + #[moq_net_sim::test] + async fn a_budget_yield_is_not_an_alias_timeout() { + let runtime = crate::time::Clock::sim(); + let aliases = TrackAliases::default(); + let pending = resolve_track_alias(&runtime, aliases.consume(), 7); + let mut pending = std::pin::pin!(pending); + assert!(poll!(&mut pending).is_pending()); + + insert_track_alias(&aliases, 7, RequestId(11)).unwrap(); + moq_net_sim::sleep(TRACK_ALIAS_TIMEOUT).await; + + let mut cx = std::task::Context::from_waker(std::task::Waker::noop()); + let starved = kio::coop::budget(|| { + crate::util::leave_budget(0); + pending.as_mut().poll(&mut cx) + }); + assert!(starved.is_pending(), "the lookup yields: {starved:?}"); + assert_eq!(pending.await.unwrap(), RequestId(11)); + } + /// SUBSCRIBE_OK has not accepted the track, so the map holds no producer. /// Abort still has to reject the parked origin request with the session error. #[moq_net_sim::test] diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index de04e242466..66ae41bb67b 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -1907,6 +1907,34 @@ mod test { } } + /// A finished track's trailing datagram still goes out when the poll that finds the + /// groups over runs out of budget: a yield there must not read as "no datagram". + #[test] + fn a_budget_yield_keeps_the_trailing_datagram() { + let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); + for left in 0..4 { + let mut producer = track_producer("test"); + let mut subscriber = producer.subscribe(None); + producer + .append_datagram(Timestamp::from_millis(1).unwrap(), &b"last"[..]) + .unwrap(); + producer.finish().unwrap(); + + let starved = kio::coop::budget(|| { + crate::util::leave_budget(left); + poll_recv_next(&mut subscriber, true, false, &waiter) + }); + let res = match starved { + Poll::Pending => kio::coop::budget(|| poll_recv_next(&mut subscriber, true, false, &waiter)), + res => res, + }; + assert!( + matches!(&res, Poll::Ready(Ok(Recv::Datagram(datagram))) if &datagram.payload[..] == b"last"), + "left {left}: the datagram was skipped" + ); + } + } + /// A relay can ingest back-to-back groups micro-reordered (the upstream leg /// sends newest-first). The older group is cached and in demand, so serving /// must still deliver it; a sequence cursor would skip it permanently. @@ -2350,15 +2378,11 @@ fn poll_recv_next( Poll::Pending => {} } } + // A read above that yielded to the cooperative budget is no answer: decide next turn. + ready!(kio::coop::poll_proceed(waiter)); // No live data ready: report the boundary (if declared) before signalling Finished, so a // future boundary reaches the subscriber while the trailing groups are still in flight. - // Once the groups finished this is a probe whose answer decides between the two, so it - // must not yield to the cooperative budget; either way nothing is left to wait on. - let probe = match groups_finished { - true => &kio::Waiter::noop(), - false => waiter, - }; - if emit_boundary && let Poll::Ready(res) = track.poll_finished(probe) { + if emit_boundary && let Poll::Ready(res) = track.poll_finished(waiter) { return Poll::Ready(res.map(Recv::Boundary)); } if groups_finished { diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index f7c8ed0470d..3e264719ad2 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -1494,6 +1494,8 @@ impl kio::Task for SourceServe { tracing::debug!(%err, "source closed"); self.ended = true; } + // A yield to the cooperative budget, not an empty queue. + Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => break, Poll::Pending => { // The route that minted the source went: close it now. if self.route.poll_closed(waiter).is_ready() { @@ -2730,6 +2732,14 @@ mod tests { let (_resolved, mut serve) = serve_one(&origin, &subscriber, &mut announced).await; assert!(kio::Task::poll(&mut serve, &kio::Waiter::noop()).is_pending()); announced.withdraw(&path); + // Out of cooperative budget, the close waits a turn rather than spinning on a + // request poll that keeps yielding. + let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); + let starved = kio::coop::budget(|| { + crate::util::leave_budget(0); + kio::Task::poll(&mut serve, &waiter) + }); + assert!(starved.is_pending()); assert!(kio::Task::poll(&mut serve, &kio::Waiter::noop()).is_ready()); } @@ -4425,6 +4435,8 @@ impl ServeLoop { { return Poll::Ready(ServeEnd::Finished); } + // Settling may have yielded to the cooperative budget instead. + ready!(kio::coop::poll_proceed(waiter)); if self.fetches.is_empty() && self.demand.poll_unused(waiter).is_ready() { return Poll::Ready(ServeEnd::Idle); } diff --git a/rs/moq-net/src/model/frame.rs b/rs/moq-net/src/model/frame.rs index b0acdcf9281..09a6ff51772 100644 --- a/rs/moq-net/src/model/frame.rs +++ b/rs/moq-net/src/model/frame.rs @@ -731,6 +731,8 @@ impl Consumer { let failed = match &res { Poll::Ready(Ok(_)) => return res, Poll::Ready(Err(err)) => Some(err.clone()), + // A yield to the cooperative budget is no stall. + Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => return res, Poll::Pending => None, }; let (recover, index) = &mut **recover; @@ -751,6 +753,7 @@ impl Consumer { Poll::Ready(Ok(Some(frame))) => frame, Poll::Ready(Ok(None)) => return Poll::Ready(Err(Error::WrongSize)), Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), + Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => return Poll::Pending, Poll::Pending => { return match self.poll_expired(waiter) { true => Poll::Ready(Err(Error::Old)), diff --git a/rs/moq-net/src/model/group.rs b/rs/moq-net/src/model/group.rs index 1ea02ec1748..be3869bc145 100644 --- a/rs/moq-net/src/model/group.rs +++ b/rs/moq-net/src/model/group.rs @@ -1375,6 +1375,8 @@ impl Consumer { return res; } Poll::Ready(Err(err)) => Some(err.clone()), + // A yield to the cooperative budget is no stall. + Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => return res, Poll::Pending => None, }; if !recover.wants(failed.as_ref(), waiter) { @@ -1419,7 +1421,8 @@ impl Consumer { if self.ended { return Some(false); } - if !self.poll_expired(waiter) { + // A read the cooperative budget turned into `Pending` may have had a frame ready. + if kio::coop::poll_proceed(waiter).is_pending() || !self.poll_expired(waiter) { return None; } let truncates = self.expired_truncates(); @@ -1874,21 +1877,23 @@ impl Cursor { state.poll_terminal(index) }); + // Count a fresh batch once here, under no lock, so the drained pops that follow + // stay free. Before `ready!`: a poll that yields to the cooperative budget keeps + // the batch it filled, and the next poll pops it on the fast path. + let (frames, bytes) = self.prefetch.buffered(); + if frames > 0 { + // The refill already updated the eviction rank under the group lock. + self.refreshed = self.cache.pool().now(); + stats.frames(frames); + stats.bytes(bytes); + } + match ready!(res) { Ok(Ok(())) => {} Ok(Err(err)) => return Poll::Ready(Err(err)), Err(state) => return Poll::Ready(Err(state.abort.clone().unwrap_or(Error::Dropped))), } - // The refill already updated the eviction rank under the group lock. - self.refreshed = self.cache.pool().now(); - - // A fresh batch was just filled (empty only on a clean end). Count the whole - // batch once here, under no lock, so the drained pops that follow stay free. - let (frames, bytes) = self.prefetch.buffered(); - stats.frames(frames); - stats.bytes(bytes); - Poll::Ready(Ok(self.prefetch.pop().inspect(|_| { self.index += 1; }))) @@ -2036,6 +2041,48 @@ mod test { assert_eq!(frame.payload, Bytes::from_static(b"hello")); } + /// A refill that yields to the cooperative budget keeps its batch, and still counts it. + #[test] + fn a_budget_yield_keeps_and_counts_the_prefetched_batch() { + let stats = crate::stats::Registry::new(crate::stats::Config::new()); + let meter = stats + .tier(crate::stats::Tier::default()) + .session("root") + .egress("demo") + .meter(); + + let mut producer = Info { sequence: 0 }.produce(); + for payload in [&b"one"[..], b"two", b"three"] { + producer.write_frame(Timestamp::ZERO, Bytes::from(payload)).unwrap(); + } + producer.finish().unwrap(); + let mut consumer = producer.consume().with_meter(meter); + + let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); + let starved = kio::coop::budget(|| { + crate::util::leave_budget(0); + consumer.poll_read_frame(&waiter) + }); + assert!(starved.is_pending(), "the refill yields"); + + let mut read = Vec::new(); + while let Poll::Ready(Ok(Some(frame))) = kio::coop::budget(|| consumer.poll_read_frame(&waiter)) { + read.push(frame.payload); + } + assert_eq!(read, [&b"one"[..], b"two", b"three"]); + + let traffic = stats.snapshot().traffic(); + let (_, _, egress) = traffic + .iter() + .find(|(_, role, _)| *role == crate::stats::Role::Publisher) + .expect("egress row"); + assert_eq!( + (egress.frames, egress.bytes), + (3, 11), + "the yielded batch is counted once" + ); + } + /// An untimed frame reads back untimed, and still counts as the group's first frame: /// an empty group and a group on an untimed track are different answers. #[test] diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 32fd3cffc85..5d45dafecbe 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -929,6 +929,8 @@ impl Recover { } } + // Not if the lookup above yielded to the cooperative budget: it may have found it. + ready!(kio::coop::poll_proceed(waiter)); // Otherwise ask for the rest. The subscription may still deliver it first. if self.fetch.as_ref().is_none_or(|(asked, _)| *asked != generation) { let options = group::Fetch::default().with_frame_start(index); diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 5b41fb0375a..4e4233f4f71 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -3424,8 +3424,9 @@ impl kio::Task for Fetching { Poll::Pending => return Poll::Pending, }; // An accept caches the group before it answers the channel, which may land - // between the track check above and this one: look again before failing. - match cached(waiter) { + // between the track check above and this one: look again before failing. A probe, + // so the cooperative budget can't turn a hit into that failure. + match cached(&kio::Waiter::noop()) { Poll::Ready(res) => Poll::Ready(res), Poll::Pending => Poll::Ready(Err(err)), } @@ -3692,7 +3693,7 @@ impl Cursor { // A helper to automatically apply Dropped if the state is closed without an error. fn poll(&self, waiter: &kio::Waiter, f: F) -> Poll> where - F: Fn(&kio::Ref<'_, TrackState>) -> Poll>, + F: FnOnce(&kio::Ref<'_, TrackState>) -> Poll>, { Poll::Ready(match ready!(self.state.poll(waiter, f)) { Ok(res) => res, @@ -3857,38 +3858,50 @@ impl Cursor { /// The lowest servable group past the last one returned, walking off everything the /// drift budget has convicted on the way. fn poll_next_group(&mut self, waiter: &kio::Waiter) -> Poll>> { - let mut floor = self.next_sequence.max(self.min_sequence); + let floor = self.next_sequence.max(self.min_sequence); let end = self.end_sequence; // One scan for the whole poll, so walking a backlog off stays linear in its size. let drift = self.drift(end, waiter); - let group = loop { - let Some(producer) = ready!(self.poll(waiter, |state| state.poll_next_in_range(floor, end))?) else { - // Deliberately no flush of `seek_pending` here: only a delivery commit - // may count a conviction. This `None` can be an artifact of a floor - // that will lower again. A conviction never committed is dropped - // uncounted with the cursor. - return Poll::Ready(Ok(None)); - }; - let group = producer.consume(); - - // Skip a group the budget has given up on and keep scanning, so one poll - // walks a whole backlog off rather than handing it out group by group. - if self.is_stale(&group, drift) { - // Not counted yet: a conviction is not permanent (the budget can widen, - // the edge can be evicted), so the group may still be delivered. - // Re-snapshot on every re-examination so the eventual count reflects the - // group's latest observed content. - self.seek_pending.insert(group.sequence, group.content()); - floor = group.sequence.saturating_add(1); - continue; + // Walk the backlog under one lock, so it spends one unit of the cooperative budget + // however many groups it skips: a poll per skip would yield partway through a long + // backlog and restart at `floor` every turn. + let mut convicted = Vec::new(); + let res = self.poll(waiter, |state| { + let mut floor = floor; + loop { + let Some(producer) = ready!(state.poll_next_in_range(floor, end))? else { + return Poll::Ready(Ok(None)); + }; + if !state.is_stale(producer.sequence, &drift.edge, drift.budget) { + return Poll::Ready(Ok(Some(producer))); + } + floor = producer.sequence.saturating_add(1); + convicted.push(producer); } + }); + + // Recorded even after a yield, since the next walk only re-snapshots them. Not + // counted yet: a conviction is not permanent (the budget can widen, the edge can be + // evicted), so the group may still be delivered. Consumed with the track guard + // released. + for producer in convicted { + let group = producer.consume(); + self.seek_pending.insert(group.sequence, group.content()); + } - // A conviction the budget walked back: the group is handed over after all, - // so it must never reach the stale count. - self.seek_pending.remove(&group.sequence); - break self.with_expiry(group); + let Some(producer) = ready!(res)? else { + // Deliberately no flush of `seek_pending` here: only a delivery commit + // may count a conviction. This `None` can be an artifact of a floor + // that will lower again. A conviction never committed is dropped + // uncounted with the cursor. + return Poll::Ready(Ok(None)); }; + let group = producer.consume(); + // A conviction the budget walked back: the group is handed over after all, + // so it must never reach the stale count. + self.seek_pending.remove(&group.sequence); + let group = self.with_expiry(group); self.next_sequence = group.sequence.saturating_add(1); // The delivery commits everything the seek stepped over to reach this group. @@ -8054,6 +8067,65 @@ mod test { } } + /// A stale backlog longer than a whole budget still reaches the deliverable group. + #[test] + fn an_ordered_walk_past_a_long_stale_backlog_spends_once() { + let mut producer = track_producer("test", None); + // Retains the backlog the reader under test walks off. + let _retain = producer.subscribe(replay()); + let mut ordered = producer.subscribe(None).ordered(); + let backlog = 64; + for millis in 0..=backlog { + append_at(&mut producer, millis); + } + + let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); + let res = kio::coop::budget(|| ordered.poll_next_group(&waiter)); + let Poll::Ready(Ok(Some(group))) = res else { + panic!("one turn delivers the newest group"); + }; + assert_eq!(group.sequence, backlog); + } + + /// A read the cooperative budget turns into `Pending` is not a stalled read, so the + /// drift budget leaves the group's buffered frame alone. + #[test] + fn a_budget_yield_is_not_a_stalled_read() { + for chunked in [false, true] { + let mut producer = track_producer("test", None); + let mut subscriber = producer.subscribe(None); + append_at(&mut producer, 0); + let mut group = subscriber.recv_group().now_or_never().unwrap().unwrap().expect("group"); + // The live edge moves a whole budget past the group's only frame. + append_at(&mut producer, 1000); + + let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); + let read = |group: &mut group::Consumer| -> Poll>> { + match chunked { + false => group + .poll_read_frame(&waiter) + .map_ok(|frame| frame.map(|frame| frame.payload)), + true => match ready!(group.poll_next_frame(&waiter)?) { + Some(mut frame) => frame.poll_read_chunk(&waiter), + None => Poll::Ready(Ok(None)), + }, + } + }; + let starved = kio::coop::budget(|| { + crate::util::leave_budget(0); + read(&mut group) + }); + assert!(starved.is_pending(), "chunked {chunked}: the read yields: {starved:?}"); + + let res = kio::coop::budget(|| read(&mut group)); + assert!( + matches!(&res, Poll::Ready(Ok(Some(payload))) if payload.as_ref() == b"x"), + "chunked {chunked}: the buffered frame is still delivered: {res:?}" + ); + assert!(!group.latency_expired()); + } + } + #[test] fn end_at_lower_than_cursor_parks_consumer() { let producer = track_producer("test", None); From 66aa426de1a0fb5c4b9d68271c8dceaedef8d0b4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 9 Oct 2026 00:52:10 -0700 Subject: [PATCH 4/8] docs(kio): a poll closure's side effects must survive a budget yield Co-Authored-By: Claude Opus 5.5 --- rs/kio/src/consumer.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/rs/kio/src/consumer.rs b/rs/kio/src/consumer.rs index c6d4a5e6560..493cf31836d 100644 --- a/rs/kio/src/consumer.rs +++ b/rs/kio/src/consumer.rs @@ -74,7 +74,10 @@ impl Consumer { /// condition returned by `f` is still pending. /// /// A `Ready` from `f` spends a unit of the [cooperative budget](crate::coop); with - /// none left, its value is dropped and the task is woken to poll again. + /// none left, its value is dropped and the task is woken to poll again. So anything + /// `f` commits outside its return value must stay valid when that value is dropped, + /// and a caller acting on `Pending` as a verdict checks + /// [`poll_proceed`](crate::coop::poll_proceed) first. pub fn poll(&self, waiter: &Waiter, f: F) -> Poll>> where F: FnOnce(&Ref<'_, T>) -> Poll, From ee11b57fc29e26d14e45dc6749128e21898f992f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 9 Oct 2026 01:10:04 -0700 Subject: [PATCH 5/8] fix(net): a yielded recovery or batch read waits for the next turn A route recovery whose serving lookup yielded read the yield as a failed recovery and returned the old copy's error. The publishers' chunked fallback popped a batch a yielded read had filled, so a long group drained in one turn regardless of the budget. Both check poll_proceed first. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/ietf/publisher.rs | 7 +++- rs/moq-net/src/lite/publisher.rs | 60 ++++++++++++++++++++++++++++++-- rs/moq-net/src/model/resume.rs | 48 +++++++++++++++++++++++++ 3 files changed, 112 insertions(+), 3 deletions(-) diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index 2ad7e6dc1d0..347c0d2bbae 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -1136,6 +1136,9 @@ where return Poll::Ready(Err(Error::Cancel)); } match group.poll_read_frames(waiter, &mut buf) { + // Out of cooperative budget, the batch it filled waits for the next turn + // rather than draining through the chunked read for free. + Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => Poll::Pending, Poll::Pending => match group.poll_next_frame(waiter) { Poll::Ready(Ok(Some(frame))) => Poll::Ready(Ok(FillStep::Partial(frame))), Poll::Ready(Ok(None)) => Poll::Ready(Ok(FillStep::Done)), @@ -2987,7 +2990,9 @@ impl GroupServe { } Poll::Ready(Ok(_)) => break 'serve Ok(()), Poll::Ready(Err(err)) => break 'serve Err(err), - Poll::Pending => {} + // Out of cooperative budget, the batch it filled waits for the next turn + // rather than draining through the chunked read for free. + Poll::Pending => ready!(kio::coop::poll_proceed(waiter)), } match self.group.poll_next_frame(waiter) { diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 66ae41bb67b..4f79161fc68 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -1698,7 +1698,9 @@ impl Request for FetchServe { } Poll::Ready(Ok(_)) => return Poll::Ready(Ok(ControlFlow::Break(()))), Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), - Poll::Pending => {} + // Out of cooperative budget, the batch it filled waits for the next turn + // rather than draining through the chunked read for free. + Poll::Pending => ready!(kio::coop::poll_proceed(waiter)), } match ready!(group.poll_next_frame(waiter))? { Some(next) => { @@ -3116,7 +3118,9 @@ impl GroupServe { } Poll::Ready(Ok(_)) => break 'serve Ok(()), Poll::Ready(Err(err)) => break 'serve Err(err), - Poll::Pending => {} + // Out of cooperative budget, the batch it filled waits for the next turn + // rather than draining through the chunked read for free. + Poll::Pending => ready!(kio::coop::poll_proceed(waiter)), } match self.group.poll_next_frame(waiter) { @@ -3775,6 +3779,58 @@ mod serve_group_test { assert!(log.resets().is_empty(), "a group was reset: {:?}", log.resets()); } + /// A group with more frames ready than a turn's budget drains over several turns: a + /// batch the budget cut short waits for the next turn instead of draining through the + /// chunked fallback for free. + #[test] + fn a_long_group_drains_within_the_budget() { + /// kio's per-turn budget; each refill of up to 8 frames spends a unit. + const BUDGET: usize = 32; + const FRAMES: usize = 4_000; + const PAYLOAD: usize = 100; + + type Task = Box Poll<()>>; + + let track = track::Producer::new(Arc::new(broadcast::Info::default()), "test", None); + let subscriber = track.subscribe(track::Subscription::default().with_max_delay(Duration::from_secs(3600))); + let log = Log::default(); + let (mut run, mut writer, _) = lite07_run(SinkSession::new(log.clone()).with_unacked_fin(), subscriber); + + let mut group = track.create_group(group::Info { sequence: 0 }).unwrap(); + for millis in 0..FRAMES as u64 { + group + .write_frame(Timestamp::from_millis(millis).unwrap(), vec![0u8; PAYLOAD]) + .unwrap(); + } + group.finish().unwrap(); + + let mut tasks: kio::Tasks = kio::Tasks::new(); + let turns = Arc::new(AtomicU64::new(0)); + let counted = turns.clone(); + tasks.push(Box::new(move |waiter: &kio::Waiter| { + counted.fetch_add(1, Ordering::Relaxed); + waiter.waker().wake_by_ref(); + Poll::Pending + })); + tasks.push(Box::new(move |waiter: &kio::Waiter| { + run.poll(&mut writer, waiter).map(|res| res.expect("serve")) + })); + + // A frame's header (size and timestamp delta) takes well under a payload more. + let most = (BUDGET + 1) * 8 * 2 * PAYLOAD; + let owner = kio::Waiter::noop(); + let mut polls = 0; + while log.writes.lock().unwrap().len() < FRAMES * PAYLOAD { + let before = log.writes.lock().unwrap().len(); + assert!(tasks.poll(&owner).is_pending(), "the serve never ends"); + polls += 1; + let wrote = log.writes.lock().unwrap().len() - before; + assert!(wrote <= most, "one turn wrote {wrote} bytes"); + assert_eq!(turns.load(Ordering::Relaxed), polls, "the sibling missed a turn"); + assert!(polls <= FRAMES as u64, "the serve stalled"); + } + } + /// SUBSCRIBE_END counts the group streams opened, not the groups below the end: a /// group the track never produced has no stream and is not counted. #[moq_net_sim::test] diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 5d45dafecbe..32958d27356 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -886,6 +886,10 @@ impl Recover { return Poll::Pending; } let res = self.poll_serving(generation, &serving, index, failed, waiter); + // A lookup that yielded to the cooperative budget is no verdict either way. + if res.is_pending() { + ready!(kio::coop::poll_proceed(waiter)); + } // The serving route failed too, and no front is left to replace it. if let (Poll::Pending, Some(err)) = (&res, failed) && (closed || end.is_some()) @@ -1174,6 +1178,50 @@ mod test { ); } + /// A failed copy whose replacement holds the group still continues from it when the + /// lookup yields to the cooperative budget: the yield is no failed recovery. + #[test] + fn a_budget_yield_is_not_a_failed_recovery() { + let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); + for left in 0..4 { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, Duration::from_secs(10)); + let mut open = a.create_group(group::Info { sequence: 0 }).unwrap(); + open.write_frame(ts(0), b"a".as_ref()).unwrap(); + let mut reading = recv(&mut sub); + assert_eq!(read(&mut reading), Some(b"a".to_vec())); + + // The replacement holds the whole group, then fails with nothing left to follow. + let b = copy(); + let mut whole = b.create_group(group::Info { sequence: 0 }).unwrap(); + for (ms, frame) in [(0, b"a"), (1, b"b")] { + whole.write_frame(ts(ms), frame.as_ref()).unwrap(); + } + whole.finish().unwrap(); + routes.serve(b.consume()); + b.abort(Error::Cancel).unwrap(); + routes.conclude(); + open.abort(Error::Dropped).unwrap(); + + let starved = kio::coop::budget(|| { + crate::util::leave_budget(left); + reading.poll_read_frame(&waiter) + }); + let res = match starved { + Poll::Pending => kio::coop::budget(|| reading.poll_read_frame(&waiter)), + res => res, + }; + assert!( + matches!(&res, Poll::Ready(Ok(Some(frame))) if frame.payload.as_ref() == b"b"), + "left {left}: {:?}", + res.map(|res| res.map(|frame| frame.map(|frame| frame.payload))) + ); + } + } + #[test] fn a_half_read_frame_continues_from_the_replacement() { let routes = Producer::new(); From 1e5553858efcd9984f807754568a2f3e0c5515e0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 9 Oct 2026 01:10:38 -0700 Subject: [PATCH 6/8] docs: poll_proceed's spare turn; hedge the FFI stall fix until main confirms it Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 2 +- quest/m0/ffi-publisher-stall.md | 3 ++- quest/m1/ffi-runtime.md | 4 ++-- rs/kio/src/coop.rs | 4 ++-- 4 files changed, 7 insertions(+), 6 deletions(-) diff --git a/quest/m0/README.md b/quest/m0/README.md index 28bb47c1fe6..3bc73b66a3a 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -64,7 +64,7 @@ remain and no release waits on them. - [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.4, before Seattle - [Capped stream END_OF_GROUP](/quest/m0/ietf-end-of-track-location.md) - a stream capped by the subscription's end Location never claims END_OF_GROUP; moving End of Track's Location is deferred - [End of Group status](/quest/m0/ietf-end-of-group-status.md) - an End of Group status on a stream whose header already marks the group's end is accepted, so imquic's last object per group arrives -- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - every Go and Python publisher cell passes reliably now that kio's cooperative budget landed, and a cell fails when a connection idles out +- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - every Go and Python publisher cell passes reliably with kio's cooperative budget, and a cell fails when a connection idles out - [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to - [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines - [JS track takeover](/quest/m0/js-track-takeover.md) - JS `createTrack` answers a queued request and continues its sequences, as Rust does, so a re-announced `@moq/publish` catalog never restarts its groups diff --git a/quest/m0/ffi-publisher-stall.md b/quest/m0/ffi-publisher-stall.md index 8584518b526..96629b2bb0b 100644 --- a/quest/m0/ffi-publisher-stall.md +++ b/quest/m0/ffi-publisher-stall.md @@ -17,7 +17,8 @@ cell. The cause is known: a serve loop with work always ready never yields and starves moq-ffi's single runtime thread until the relay times the publisher out (found 2026-10-08 landing #4225). kio's cooperative budget (`kio::coop`) -fixes it, so no bisect is needed. +is meant to fix it: interop run 37889112106 on #5088 passed every Go and +Python publisher cell. Confirm on `main` and nightly before closing. Decided 2026-10-08: diff --git a/quest/m1/ffi-runtime.md b/quest/m1/ffi-runtime.md index a5d59ae03e5..c1bc4734909 100644 --- a/quest/m1/ffi-runtime.md +++ b/quest/m1/ffi-runtime.md @@ -20,8 +20,8 @@ added this quest): #1071) with no stated reason. - Use `new_multi_thread().enable_all()` with tokio's default worker count. No configuration knob until an embedder needs one. Rejected: a small fixed - pool, and keeping `current_thread` behind the - kio's cooperative budget (`kio::coop`) alone. + pool, and keeping `current_thread` behind kio's cooperative budget + (`kio::coop`) alone. - Keep the "foreign caller's thread never drives our futures" property: work still runs via `Task::run` and `detached`. Check the exported async fns that await in place (`subscribe_catalog`, `subscribe_track`, `fetch_group`, diff --git a/rs/kio/src/coop.rs b/rs/kio/src/coop.rs index 03451870501..642e3410f64 100644 --- a/rs/kio/src/coop.rs +++ b/rs/kio/src/coop.rs @@ -74,8 +74,8 @@ pub(crate) fn spend(waiter: &Waiter) -> bool { /// With the budget spent, a kio poll returns `Pending` even when it had an answer, so /// a caller about to treat `Pending` as a verdict (blocked, idle, not found) checks this /// first. `Pending` means the budget is spent: the task is woken, and the verdict waits -/// for a fresh turn. `Ready` means no poll this turn yielded, so its `Pending` was real. -/// Spends nothing. +/// for a fresh turn. That is one spare turn when the `Pending` was real after all. +/// `Ready` means no poll this turn yielded, so its `Pending` was real. Spends nothing. pub fn poll_proceed(waiter: &Waiter) -> std::task::Poll<()> { match waiter.is_noop() || LEFT.get() != Some(0) { true => std::task::Poll::Ready(()), From 47d44a98c1fb7980a2bd103929c90c61717b3bd6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 9 Oct 2026 10:35:07 -0700 Subject: [PATCH 7/8] refactor(kio): yield only at owned budgets in the serve loops kio polls no longer turn a Ready into Pending once a per-thread budget runs out: that made every Pending a possible yield, and each review round found another caller that read it as a verdict and lost data. A loop that can keep finding work ready now owns a `kio::coop::Budget` and calls `poll_yield` at the head of each pass, sized by its own cost. The six publish serve loops take one (lite RequestServe and GroupServe, ietf TrackServe, GroupServe, read_fetch, and write_fetch_group), with a busy-track and a long-group regression for each protocol. The verdict guards, the thread-local, and the Tasks and runtime refills are gone. Co-Authored-By: Claude Opus 5.5 --- quest/m0/README.md | 6 +- quest/m0/ffi-publisher-stall.md | 7 +- quest/m1/README.md | 2 + quest/m1/ffi-runtime.md | 4 +- quest/m1/js-serve-yield.md | 20 +++ quest/m1/session-burst-hang.md | 8 +- quest/m1/spinning-loops.md | 33 ++++ quest/m1/track-tail-interop.md | 2 +- rs/kio/README.md | 2 +- rs/kio/src/consumer.rs | 10 -- rs/kio/src/coop.rs | 264 +++++----------------------- rs/kio/src/lib.rs | 2 +- rs/kio/src/producer.rs | 8 - rs/kio/src/queue.rs | 21 +-- rs/kio/src/shared.rs | 4 - rs/kio/src/task.rs | 50 +----- rs/kio/src/waiter.rs | 20 +-- rs/kio/src/weak.rs | 4 - rs/moq-net/benches/session.rs | 4 +- rs/moq-net/sim/src/lib.rs | 31 ++-- rs/moq-net/src/ietf/control.rs | 2 - rs/moq-net/src/ietf/publisher.rs | 155 +++++++++++++++-- rs/moq-net/src/ietf/subscriber.rs | 26 --- rs/moq-net/src/lite/publisher.rs | 279 +++++++++++++++--------------- rs/moq-net/src/lite/subscriber.rs | 12 -- rs/moq-net/src/model/frame.rs | 3 - rs/moq-net/src/model/group.rs | 67 ++----- rs/moq-net/src/model/resume.rs | 50 ------ rs/moq-net/src/model/track.rs | 233 +++++-------------------- rs/moq-net/src/time.rs | 42 ++--- rs/moq-net/src/util.rs | 19 -- rs/moq-uring/src/worker.rs | 22 +-- 32 files changed, 506 insertions(+), 906 deletions(-) create mode 100644 quest/m1/js-serve-yield.md create mode 100644 quest/m1/spinning-loops.md diff --git a/quest/m0/README.md b/quest/m0/README.md index 3bc73b66a3a..5a3a969fa67 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -48,8 +48,8 @@ publishers mint epochs. Backport patches cut from `release` don't carry Liveness: a serve loop with work always ready never yields, which starved an FFI publisher's QUIC driver and fails hosted Interop's go lanes (found -2026-10-08 landing #4225). kio's cooperative budget (`kio::coop`) now -bounds every kio task's loop. The Go and Python interop cells it fails +2026-10-08 landing #4225). The publish serve loops now yield through a +`kio::coop::Budget`. The Go and Python interop cells it fails moved here from m1 the same day, because the stall masks interop on every wire PR and hides as slow passes; the harness now fails a cell whose connection idles out. @@ -64,7 +64,7 @@ remain and no release waits on them. - [Draft-22 media on 0.17](/quest/m0/release-22/README.md) - a 0.17.x with the LOCATION_FILTER and FIRST_OBJECT fixes and moq-noq 1.3.4, before Seattle - [Capped stream END_OF_GROUP](/quest/m0/ietf-end-of-track-location.md) - a stream capped by the subscription's end Location never claims END_OF_GROUP; moving End of Track's Location is deferred - [End of Group status](/quest/m0/ietf-end-of-group-status.md) - an End of Group status on a stream whose header already marks the group's end is accepted, so imquic's last object per group arrives -- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - every Go and Python publisher cell passes reliably with kio's cooperative budget, and a cell fails when a connection idles out +- [FFI publisher stall](/quest/m0/ffi-publisher-stall.md) - every Go and Python publisher cell passes reliably with the serve budgets, and a cell fails when a connection idles out - [web-transport releases the qmux fixes](/quest/m0/qmux-credit-upstream.md) - waiting on moq-dev/web-transport#412 and #413 to merge and ship, which qmux credit bumps to - [qmux credit](/quest/m0/qmux-credit.md) - qmux returns connection credit for dropped and stopped streams and delivers its close frame, on both lines - [JS track takeover](/quest/m0/js-track-takeover.md) - JS `createTrack` answers a queued request and continues its sequences, as Rust does, so a re-announced `@moq/publish` catalog never restarts its groups diff --git a/quest/m0/ffi-publisher-stall.md b/quest/m0/ffi-publisher-stall.md index 96629b2bb0b..bc42972f0e3 100644 --- a/quest/m0/ffi-publisher-stall.md +++ b/quest/m0/ffi-publisher-stall.md @@ -16,9 +16,10 @@ cell. The cause is known: a serve loop with work always ready never yields and starves moq-ffi's single runtime thread until the relay times the publisher -out (found 2026-10-08 landing #4225). kio's cooperative budget (`kio::coop`) -is meant to fix it: interop run 37889112106 on #5088 passed every Go and -Python publisher cell. Confirm on `main` and nightly before closing. +out (found 2026-10-08 landing #4225). The publish serve loops now yield +through a `kio::coop::Budget` (#5088), which is meant to fix it: interop run +37889112106 passed every Go and Python publisher cell on an earlier design of +that PR. Confirm on `main` and nightly before closing. Decided 2026-10-08: diff --git a/quest/m1/README.md b/quest/m1/README.md index d5a22c1091f..b42e029e054 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -67,6 +67,8 @@ were done or not worth their cost. - [moq-uring tests under load](/quest/m1/uring-tests-under-load.md) - uring tests pass while parallel checks share locked memory - [FFI runtime](/quest/m1/ffi-runtime.md) - moq-ffi drives moq on a multi-thread runtime instead of one thread - [Audio group duration](/quest/m1/audio-group-duration.md) - audio groups span at least 20 ms by default, so small frames don't mint a group each +- [A spinning loop fails a sim test](/quest/m1/spinning-loops.md) - a sim test names any loop that holds a task poll too long, and each one yields through a budget +- [A busy js/net serve yields](/quest/m1/js-serve-yield.md) - js/net's serve loop leaves the browser its event loop under a fast publisher, if it does not already - [Wire compatibility](/quest/m1/wire-compat.md) - a nightly run tests this checkout against the last published release for tokens, session wire, and catalog/container - [Kotlin wrapper POMs](/quest/m1/kt-ffi-pom.md) - Maven builds of `dev.moq:moq` resolve a published moq-ffi instead of the missing `0.0.0-dev` - [Installable moq-gst](/quest/m1/gst-brew-path.md) - `brew install moq-gst` installs the plugin from the tarball's `lib/gstreamer-1.0/`, and the Nix package carries `x264enc`, `avenc_aac`, and `avdec_h264` diff --git a/quest/m1/ffi-runtime.md b/quest/m1/ffi-runtime.md index c1bc4734909..b7fe67151a2 100644 --- a/quest/m1/ffi-runtime.md +++ b/quest/m1/ffi-runtime.md @@ -20,8 +20,8 @@ added this quest): #1071) with no stated reason. - Use `new_multi_thread().enable_all()` with tokio's default worker count. No configuration knob until an embedder needs one. Rejected: a small fixed - pool, and keeping `current_thread` behind kio's cooperative budget - (`kio::coop`) alone. + pool, and keeping `current_thread` behind the serve budgets + (`kio::coop::Budget`) alone. - Keep the "foreign caller's thread never drives our futures" property: work still runs via `Task::run` and `detached`. Check the exported async fns that await in place (`subscribe_catalog`, `subscribe_track`, `fetch_group`, diff --git a/quest/m1/js-serve-yield.md b/quest/m1/js-serve-yield.md new file mode 100644 index 00000000000..127c56082c9 --- /dev/null +++ b/quest/m1/js-serve-yield.md @@ -0,0 +1,20 @@ +# [S] A busy js/net serve leaves the browser its event loop + +## Goal + +Find whether a js/net serve loop whose awaits keep resolving immediately (a +fast publisher with a backlog) runs only microtasks and starves the browser's +event loop: rendering, input, and WebTransport reads. Fix it at the cause if it +does, and keep a browser bench that shows it. + +## Plan + +Found 2026-10-09 while reworking #5088, which bounds the Rust serve loops with +`kio::coop::Budget`. Microtask starvation is the JS analog of a Rust task that +never returns `Pending`. Measure first: a browser bench publishing 2.5 ms +groups with a sibling timer, checking that the timer keeps firing. Node and Bun +only stand in for the browser. + +## Related + +- [A spinning loop fails a sim test](/quest/m1/spinning-loops.md) - the Rust side's guard diff --git a/quest/m1/session-burst-hang.md b/quest/m1/session-burst-hang.md index 6835a8149e4..bfda5fbc1ef 100644 --- a/quest/m1/session-burst-hang.md +++ b/quest/m1/session-burst-hang.md @@ -3,8 +3,8 @@ ## Goal `rs/moq-net/benches/session.rs`'s `burst` sweep runs to completion at 16 -subscriptions and 16 or more groups per round, with or without the kio -budget, and the sweep covers those shapes again. +subscriptions and 16 or more groups per round, with or without the serve +budgets, and the sweep covers those shapes again. ## Plan @@ -15,7 +15,3 @@ budget too, so the sweep was limited to shapes that complete. Decided 2026-10-08 treat it as a possible session stall, not a bench setup limit. Find whether it is cache size, stream credit, or a real stall, fix it at the cause, and add a regression test if the cause is in the session rather than the bench. - -## Required - -- [A busy task yields within a budget](/quest/m0/serve-budget.md) - adds the `burst` axis diff --git a/quest/m1/spinning-loops.md b/quest/m1/spinning-loops.md new file mode 100644 index 00000000000..efebde91bc1 --- /dev/null +++ b/quest/m1/spinning-loops.md @@ -0,0 +1,33 @@ +# [M] A spinning loop fails a sim test + +## Goal + +A moq-net sim test fails, naming the loop, when one task poll makes more than a +fixed number of kio `Ready` polls, and every loop it flags yields through a +`kio::coop::Budget`. Today a loop with no budget that keeps finding work ready +holds its thread with nothing to flag it, the way `RequestServe::poll_serve` +starved a go publisher's runtime (#5088). + +## Plan + +Decided 2026-10-09 while reworking #5088: + +- #5088 gives only the publish serve loops a `Budget`: lite `RequestServe` and + `GroupServe`, and ietf `TrackServe`, `GroupServe`, `read_fetch`, and + `write_fetch_group`. A search found about 31 more loops that can keep + finding work in one poll (receive, accept, announce, and request loops in + `lite/subscriber.rs`, `ietf/subscriber.rs`, `ietf/session.rs`, and + `model/origin.rs`). This guard decides which of them actually spin. +- The count is test-only, so kio's hot path stays untouched: a feature or + `cfg` that only the sim runtime enables, reset on each task poll. +- A loop takes a `Budget` only where its `Pending` goes straight up to the + task. Three known sites need care: lite `FetchServeRun` reads a + `FrameIngest` yield as "still short" and may `abort_unused`; + `Announced::poll_serve` must re-queue the route through `route_waiter` and + yield in the outer loop too; and ietf `poll_datagrams`'s drain arm would + skip datagrams after a yield unless it returns `Poll`. + +## Related + +- [Session burst hang](/quest/m1/session-burst-hang.md) - the same bench shapes that hang +- [FFI runtime](/quest/m1/ffi-runtime.md) - FFI apps run on more than one thread diff --git a/quest/m1/track-tail-interop.md b/quest/m1/track-tail-interop.md index ea101402a82..a3e8755b983 100644 --- a/quest/m1/track-tail-interop.md +++ b/quest/m1/track-tail-interop.md @@ -31,7 +31,7 @@ js-native-node`, and `rust -> js-native-bun`. Merge `main` (after #4741 and Found 2026-10-08: #4225's `go -> *` lanes fail on a stall where the go publisher's serve loop starves its own QUIC driver until the relay times it -out. kio's cooperative budget (`kio::coop`) is meant to fix it; rerun the +out. The serve budgets (`kio::coop::Budget`) are meant to fix it; rerun the lanes. Decided in the 2026-09-30 audit: the lite-07 drop case moved into diff --git a/rs/kio/README.md b/rs/kio/README.md index daa6172673e..be6159a765c 100644 --- a/rs/kio/README.md +++ b/rs/kio/README.md @@ -18,7 +18,7 @@ The channel auto-closes when all producers are dropped. poll-native FIFO queue (bounded or unbounded) built in the same style. `Park` holds a waiter for as long as a poll stays pending, bridging a `std::task::Context` to kio's waiter-based polls when implementing `Future` or `poll_*` on top of kio channels. -`coop::budget` bounds the progress one task makes per turn, so a loop that always finds +`coop::Budget` bounds the passes a loop makes per poll, so one that always finds work ready still yields to the rest of its runtime. It's used internally by [moq-net](https://github.com/moq-dev/moq/tree/main/rs/moq-net) and friends, but is generic enough to be useful on its own. diff --git a/rs/kio/src/consumer.rs b/rs/kio/src/consumer.rs index 493cf31836d..b9ba209ac1a 100644 --- a/rs/kio/src/consumer.rs +++ b/rs/kio/src/consumer.rs @@ -72,12 +72,6 @@ impl Consumer { /// channel is still open, registers the [`Waiter`] for notification. /// Returns `Err(`[`Ref`]`)` if the channel has been closed while the /// condition returned by `f` is still pending. - /// - /// A `Ready` from `f` spends a unit of the [cooperative budget](crate::coop); with - /// none left, its value is dropped and the task is woken to poll again. So anything - /// `f` commits outside its return value must stay valid when that value is dropped, - /// and a caller acting on `Pending` as a verdict checks - /// [`poll_proceed`](crate::coop::poll_proceed) first. pub fn poll(&self, waiter: &Waiter, f: F) -> Poll>> where F: FnOnce(&Ref<'_, T>) -> Poll, @@ -86,10 +80,6 @@ impl Consumer { let consumer_state = Ref { state }; if let Poll::Ready(res) = f(&consumer_state) { - if !crate::coop::spend(waiter) { - drop(consumer_state); - return crate::coop::exhausted(waiter); - } return Poll::Ready(Ok(res)); } diff --git a/rs/kio/src/coop.rs b/rs/kio/src/coop.rs index 642e3410f64..d7c2fa2293e 100644 --- a/rs/kio/src/coop.rs +++ b/rs/kio/src/coop.rs @@ -1,100 +1,49 @@ -//! Cooperative scheduling: bound how much one task does per turn. +//! Cooperative scheduling: bound how many passes a loop makes per poll. //! -//! A task whose kio polls keep returning `Ready` never yields on its own: a serve -//! loop that always has another item ready holds its thread for as long as items -//! keep arriving, starving everything else on it. Tokio's own budget cannot see -//! that loop, since the work goes through kio rather than a tokio resource. -//! -//! [`budget`] gives a task a fixed number of units for its turn. Every kio poll -//! that would return `Ready` with progress (a value, a write guard, a queue item) -//! spends one. Once none are left, such a poll wakes the task and returns `Pending` -//! instead, so the task yields and picks up where it left off on its next turn. -//! A poll that returns `Pending` anyway spends nothing, and neither does one -//! passed [`Waiter::noop`], whose caller reads `Pending` as "nothing now", nor -//! the synchronous `try_*` and `read` methods. Waiting on a level (closure, demand) -//! is not progress, so those polls spend nothing either. -//! -//! A yield only postpones, so a caller that reads a `Pending` as a verdict (blocked, -//! idle, not found) checks [`poll_proceed`] before acting on it. -//! -//! Code outside any [`budget`] call is unconstrained. +//! A task whose loop keeps finding work ready never yields on its own: a serve loop +//! that always has another item ready holds its thread for as long as items keep +//! arriving, starving everything else on it. Tokio's own budget cannot see that loop, +//! since the work goes through kio rather than a tokio resource. -use std::cell::Cell; +use std::task::Poll; use crate::Waiter; -/// Units a task gets per turn. The smallest a `session_delivery_burst` sweep over 32, -/// 128 (tokio's), and 512 found as fast as no budget at all. -pub(crate) const UNITS: u16 = 32; - -thread_local! { - /// Units left in the current turn, or `None` outside any [`budget`] call. - static LEFT: Cell> = const { Cell::new(None) }; -} - -/// Run `f` with a fresh budget, restoring the previous one afterwards. +/// Yields a loop after a fixed number of passes, sized by the loop's cost per pass. /// -/// Call it only where a driver hands a task its turn: [`Tasks`](crate::Tasks) does -/// for each task it polls, and a runtime adapter does around each poll of its root. -/// Nested calls refill, so a child gets its own budget and its parent resumes with -/// whatever it had left. -pub fn budget(f: impl FnOnce() -> R) -> R { - struct Restore(Option); - - impl Drop for Restore { - fn drop(&mut self) { - LEFT.set(self.0); - } - } - - let _restore = Restore(LEFT.replace(Some(UNITS))); - f() +/// Keep one per loop and call [`Self::poll_yield`] at the head of each pass. +#[derive(Debug, Clone)] +pub struct Budget { + passes: u32, + left: u32, } -/// Spend a unit for a poll about to return `Ready`. -/// -/// `false` means the budget ran out: the caller wakes `waiter`, once it released any -/// lock, and returns `Pending` without taking anything. -pub(crate) fn spend(waiter: &Waiter) -> bool { - if waiter.is_noop() { - return true; +impl Budget { + /// A budget allowing `passes` passes between yields. + pub const fn new(passes: u32) -> Self { + Self { passes, left: passes } } - LEFT.with(|left| match left.get() { - None => true, - Some(0) => false, - Some(units) => { - left.set(Some(units - 1)); - true - } - }) -} -/// Hold off acting on a `Pending` while this turn's budget is spent. -/// -/// With the budget spent, a kio poll returns `Pending` even when it had an answer, so -/// a caller about to treat `Pending` as a verdict (blocked, idle, not found) checks this -/// first. `Pending` means the budget is spent: the task is woken, and the verdict waits -/// for a fresh turn. That is one spare turn when the `Pending` was real after all. -/// `Ready` means no poll this turn yielded, so its `Pending` was real. Spends nothing. -pub fn poll_proceed(waiter: &Waiter) -> std::task::Poll<()> { - match waiter.is_noop() || LEFT.get() != Some(0) { - true => std::task::Poll::Ready(()), - false => exhausted(waiter), + /// Spend a pass, or wake the task and return `Pending` once none are left. + /// + /// The yield refills the budget for the next poll. Call it only where `Pending` goes + /// straight up to the task: a caller that reads `Pending` as "nothing now" would + /// mistake the yield for an answer. + pub fn poll_yield(&mut self, waiter: &Waiter) -> Poll<()> { + match self.left.checked_sub(1) { + Some(left) => { + self.left = left; + Poll::Ready(()) + } + None => { + self.left = self.passes; + waiter.waker().wake_by_ref(); + Poll::Pending + } + } } } -/// Yield for an exhausted budget: wake the task so it runs again next turn. -pub(crate) fn exhausted(waiter: &Waiter) -> std::task::Poll { - waiter.waker().wake_by_ref(); - std::task::Poll::Pending -} - -/// The units left in this turn, or `None` outside any [`budget`] call. -#[cfg(test)] -pub(crate) fn left() -> Option { - LEFT.get() -} - #[cfg(all(test, not(loom)))] mod tests { use std::{ @@ -102,11 +51,10 @@ mod tests { Arc, atomic::{AtomicUsize, Ordering}, }, - task::{Poll, Wake, Waker}, + task::{Wake, Waker}, }; use super::*; - use crate::{Producer, Queue}; struct Count(AtomicUsize); @@ -116,141 +64,17 @@ mod tests { } } - fn counted() -> (Waiter, Arc) { - let count = Arc::new(Count(AtomicUsize::new(0))); - (Waiter::new(Waker::from(count.clone())), count) - } - - fn ready(value: &crate::Ref<'_, u32>) -> Poll { - Poll::Ready(**value) - } - #[test] - fn unconstrained_outside_a_budget() { - let producer = Producer::new(1u32); - let consumer = producer.consume(); - let (waiter, _) = counted(); - for _ in 0..1_000 { - assert!(matches!(consumer.poll(&waiter, ready), Poll::Ready(Ok(1)))); - } - assert_eq!(left(), None); - } - - #[test] - fn ready_spends_and_pending_does_not() { - let producer = Producer::new(1u32); - let consumer = producer.consume(); - let (waiter, _) = counted(); - budget(|| { - assert!(consumer.poll(&waiter, ready).is_ready()); - assert_eq!(left(), Some(UNITS - 1)); - assert!(consumer.poll(&waiter, |_| Poll::<()>::Pending).is_pending()); - assert_eq!(left(), Some(UNITS - 1), "a pending poll must not spend"); - }); - } - - #[test] - fn exhaustion_yields_and_wakes() { - let producer = Producer::new(1u32); - let consumer = producer.consume(); - let (waiter, wakes) = counted(); - budget(|| { - for _ in 0..UNITS { - assert!(consumer.poll(&waiter, ready).is_ready()); - } - assert_eq!(wakes.0.load(Ordering::SeqCst), 0); - assert!(consumer.poll(&waiter, ready).is_pending()); - assert_eq!(wakes.0.load(Ordering::SeqCst), 1, "a yield must wake the task"); - - // Waiting on a level spends nothing, so it still answers. - producer.close().ok().expect("open"); - assert!(consumer.poll_closed(&waiter).is_ready()); - }); - } - - /// A queue item is never taken by a poll that then yields. - #[test] - fn exhaustion_takes_nothing() { - let queue = Queue::new(); - let (waiter, _) = counted(); - budget(|| { - for item in 0..=u32::from(UNITS) { - queue.try_push(item).expect("open"); - } - for item in 0..u32::from(UNITS) { - assert_eq!(queue.poll_pop(&waiter), Poll::Ready(Ok(item))); - } - assert!(queue.poll_pop(&waiter).is_pending()); - assert!(queue.poll_push_with(&waiter, || 99).is_pending()); - }); - assert_eq!( - queue.try_pop(), - Ok(Some(u32::from(UNITS))), - "the yield left the item queued" - ); - assert_eq!(queue.try_pop(), Ok(None), "the yield pushed nothing"); - } - - #[test] - fn proceed_holds_off_once_spent() { - let producer = Producer::new(1u32); - let consumer = producer.consume(); - let (waiter, wakes) = counted(); - assert!(poll_proceed(&waiter).is_ready(), "unconstrained outside a budget"); - budget(|| { - for _ in 0..UNITS { - assert!(poll_proceed(&waiter).is_ready()); - assert!(consumer.poll(&waiter, ready).is_ready()); + fn yields_after_its_passes_and_refills() { + let wakes = Arc::new(Count(AtomicUsize::new(0))); + let waiter = Waiter::new(Waker::from(wakes.clone())); + let mut budget = Budget::new(4); + for round in 1..=3 { + for _ in 0..4 { + assert!(budget.poll_yield(&waiter).is_ready()); } - assert_eq!(left(), Some(0), "proceeding spends nothing"); - assert!(poll_proceed(&waiter).is_pending()); - assert_eq!(wakes.0.load(Ordering::SeqCst), 1, "holding off wakes the task"); - assert!(poll_proceed(&Waiter::noop()).is_ready(), "a noop waiter is exempt"); - }); - } - - #[test] - fn noop_waiter_spends_nothing() { - let producer = Producer::new(1u32); - let consumer = producer.consume(); - let waiter = Waiter::noop(); - budget(|| { - for _ in 0..1_000 { - assert!(consumer.poll(&waiter, ready).is_ready()); - } - assert_eq!(left(), Some(UNITS)); - // A clone of the noop waiter is still one. - assert!(consumer.poll(&waiter.clone(), ready).is_ready()); - assert_eq!(left(), Some(UNITS)); - }); - } - - #[test] - fn nested_budgets_refill_and_restore() { - let producer = Producer::new(1u32); - let consumer = producer.consume(); - let (waiter, _) = counted(); - budget(|| { - assert!(consumer.poll(&waiter, ready).is_ready()); - budget(|| { - assert_eq!(left(), Some(UNITS), "a nested budget refills"); - for _ in 0..UNITS { - assert!(consumer.poll(&waiter, ready).is_ready()); - } - assert!(consumer.poll(&waiter, ready).is_pending()); - }); - assert_eq!(left(), Some(UNITS - 1), "the outer budget resumes where it was"); - }); - assert_eq!(left(), None); - } - - #[test] - fn a_panic_restores_the_outer_budget() { - budget(|| { - let inner = std::panic::catch_unwind(|| budget(|| panic!("boom"))); - assert!(inner.is_err()); - assert_eq!(left(), Some(UNITS)); - }); - assert_eq!(left(), None); + assert!(budget.poll_yield(&waiter).is_pending()); + assert_eq!(wakes.0.load(Ordering::SeqCst), round, "a yield must wake the task"); + } } } diff --git a/rs/kio/src/lib.rs b/rs/kio/src/lib.rs index b0a6785473f..50bb0def9a8 100644 --- a/rs/kio/src/lib.rs +++ b/rs/kio/src/lib.rs @@ -13,7 +13,7 @@ //! clone-able handles, bounded or unbounded, with separate wake lists for the //! push and pop sides. //! -//! [`coop`] bounds how much one task does per turn, so a loop that always finds +//! [`coop`] bounds the passes a loop makes per poll, so one that always finds //! work ready still yields. use std::{ diff --git a/rs/kio/src/producer.rs b/rs/kio/src/producer.rs index 83199be73c3..c1564d0293f 100644 --- a/rs/kio/src/producer.rs +++ b/rs/kio/src/producer.rs @@ -135,10 +135,6 @@ impl Producer { let mut guard = Ref { state }; match f(&guard) { - Poll::Ready(()) if !crate::coop::spend(waiter) => { - drop(guard); - crate::coop::exhausted(waiter) - } // Upgrade the Ref to a Mut, keeping the same lock guard. Poll::Ready(()) => Poll::Ready(Ok(Mut::new(guard.state))), Poll::Pending => { @@ -162,10 +158,6 @@ impl Producer { let mut guard = Ref { state }; if let Poll::Ready(res) = f(&guard) { - if !crate::coop::spend(waiter) { - drop(guard); - return crate::coop::exhausted(waiter); - } return Poll::Ready(Ok(res)); } diff --git a/rs/kio/src/queue.rs b/rs/kio/src/queue.rs index 367a7dfdf2c..ef16c941247 100644 --- a/rs/kio/src/queue.rs +++ b/rs/kio/src/queue.rs @@ -140,10 +140,6 @@ impl Queue { waiter.register(&mut state.waiters_push); return Poll::Pending; } - if !crate::coop::spend(waiter) { - drop(state); - return crate::coop::exhausted(waiter); - } let item = make(); state.queue.push_back(item); state.waiters_pop.take() @@ -187,19 +183,14 @@ impl Queue { pub fn poll_pop(&self, waiter: &Waiter) -> Poll> { let (item, mut waiters) = { let mut state = self.state.lock(); - if state.queue.is_empty() { - if state.closed { - return Poll::Ready(Err(Closed)); + match state.queue.pop_front() { + Some(item) => (item, state.waiters_push.take()), + None if state.closed => return Poll::Ready(Err(Closed)), + None => { + waiter.register(&mut state.waiters_pop); + return Poll::Pending; } - waiter.register(&mut state.waiters_pop); - return Poll::Pending; - } - if !crate::coop::spend(waiter) { - drop(state); - return crate::coop::exhausted(waiter); } - let item = state.queue.pop_front().expect("checked non-empty"); - (item, state.waiters_push.take()) }; waiters.wake(); Poll::Ready(Ok(item)) diff --git a/rs/kio/src/shared.rs b/rs/kio/src/shared.rs index b60ce9ebb22..b81123ef3c5 100644 --- a/rs/kio/src/shared.rs +++ b/rs/kio/src/shared.rs @@ -69,10 +69,6 @@ impl Shared { state: self.state.lock(), }; match f(&guard) { - Poll::Ready(()) if !crate::coop::spend(waiter) => { - drop(guard); - crate::coop::exhausted(waiter) - } // Upgrade the Ref to a Mut, keeping the same lock guard. Poll::Ready(()) => Poll::Ready(Mut::new(guard.state)), Poll::Pending => { diff --git a/rs/kio/src/task.rs b/rs/kio/src/task.rs index a3948e41d23..0637818bfe4 100644 --- a/rs/kio/src/task.rs +++ b/rs/kio/src/task.rs @@ -245,9 +245,6 @@ impl> Tasks { /// such an arm with [`is_empty`](Self::is_empty). /// `waiter` is registered for the next task wake or push; the tasks /// themselves park on their own wakers. - /// - /// Each task polls under a fresh [cooperative budget](crate::coop), so one that - /// always finds work ready yields to the next pass instead of holding the owner. pub fn poll(&mut self, waiter: &Waiter) -> Poll<()> { { let mut parent = self.shared.parent.lock().unwrap(); @@ -308,9 +305,7 @@ impl> Tasks { } let cx = Context::from_waker(&self.wakers[index]); let child_waiter = occupant.park.hold(&cx); - // Each task gets a turn of its own: one that spends its budget - // self-wakes into the next pass, after the owner's other arms run. - if crate::coop::budget(|| occupant.task.poll(child_waiter)).is_ready() { + if occupant.task.poll(child_waiter).is_ready() { self.children[index] = None; self.free.push(index); self.len -= 1; @@ -588,49 +583,6 @@ mod tests { ); } - /// A task that always finds work ready yields once its budget runs out, lands in - /// the next pass, and leaves its sibling's budget untouched. - #[test] - fn a_busy_task_yields_within_its_budget() { - fn busy(work: crate::Consumer, done: Arc) -> impl FnMut(&Waiter) -> Poll<()> { - move |waiter| { - loop { - if done.load(Ordering::SeqCst) == 1_000 { - return Poll::Ready(()); - } - // Always ready: only the budget ends this loop. - std::task::ready!(work.poll(waiter, |value| Poll::Ready(**value))).ok(); - done.fetch_add(1, Ordering::SeqCst); - } - } - } - - let work = crate::Producer::new(0u32); - let (a, b) = (Arc::new(AtomicUsize::new(0)), Arc::new(AtomicUsize::new(0))); - let mut tasks = Tasks::new(); - tasks.push(busy(work.consume(), a.clone())); - tasks.push(busy(work.consume(), b.clone())); - - let waiter = Waiter::noop(); - assert!(tasks.poll(&waiter).is_pending()); - let units = crate::coop::UNITS as usize; - assert_eq!(a.load(Ordering::SeqCst), units, "a yields after its budget"); - assert_eq!(b.load(Ordering::SeqCst), units, "b gets a budget of its own"); - - let mut passes = 1; - loop { - passes += 1; - if tasks.poll(&waiter).is_ready() { - break; - } - } - assert_eq!( - passes, - 1_000usize.div_ceil(units), - "each pass serves one budget per task" - ); - } - /// Wake-all resources must not always hand the first chance to the lowest /// task slot. Rotating the leading task lets every contender make progress. #[test] diff --git a/rs/kio/src/waiter.rs b/rs/kio/src/waiter.rs index ac3986df49d..0dcc4d8a5a7 100644 --- a/rs/kio/src/waiter.rs +++ b/rs/kio/src/waiter.rs @@ -42,11 +42,6 @@ pub struct Waiter { // A registration found no free slot in `parked`, so a list may hold this waiter // unrecorded and a later registration there would stack a duplicate. lost: AtomicBool, - - // Built by `Waiter::noop`, whose caller reads `Pending` as "nothing now", so the - // cooperative budget must never turn a `Ready` into one. Wakers can't be compared - // reliably, so the flag says it. - noop: bool, } /// Lists a waiter remembers being parked on. A relay's busiest tasks park on up to 8. @@ -60,24 +55,12 @@ impl Waiter { shared: OnceLock::new(), parked: Default::default(), lost: AtomicBool::new(false), - noop: false, } } /// Create a no-op waiter that discards registrations. - /// - /// Polls through it never yield to the [cooperative budget](crate::coop), since - /// nothing would wake them. pub fn noop() -> Self { - Self { - noop: true, - ..Self::new(Waker::noop().clone()) - } - } - - /// Whether this is a [`Self::noop`] waiter, exempt from the cooperative budget. - pub(crate) fn is_noop(&self) -> bool { - self.noop + Self::new(Waker::noop().clone()) } /// Register this waiter with a [`WaiterList`] for future notification. @@ -175,7 +158,6 @@ impl Clone for Waiter { shared: OnceLock::from(shared), parked: std::array::from_fn(|i| AtomicU64::new(self.parked[i].load(Ordering::Relaxed))), lost: AtomicBool::new(self.lost.load(Ordering::Relaxed)), - noop: self.noop, } } } diff --git a/rs/kio/src/weak.rs b/rs/kio/src/weak.rs index b624ba422f5..2976e5e6a9f 100644 --- a/rs/kio/src/weak.rs +++ b/rs/kio/src/weak.rs @@ -277,10 +277,6 @@ impl ConsumerWeak { let consumer_state = Ref { state }; if let Poll::Ready(res) = f(&consumer_state) { - if !crate::coop::spend(waiter) { - drop(consumer_state); - return crate::coop::exhausted(waiter); - } return Poll::Ready(Ok(res)); } diff --git a/rs/moq-net/benches/session.rs b/rs/moq-net/benches/session.rs index ccd58c6cca2..6078d923458 100644 --- a/rs/moq-net/benches/session.rs +++ b/rs/moq-net/benches/session.rs @@ -883,8 +883,8 @@ fn session(c: &mut Criterion) { }), ); delivery(c, "frame", [64, 1024, 16 * 1024].map(|frame| Shape { frame, ..base })); - // One session serving a backlog on each of its subscriptions: the serve loop that - // kio's cooperative budget bounds per turn. + // One session serving a backlog on each of its subscriptions: the serve loop that a + // `kio::coop::Budget` bounds per poll. delivery( c, "burst", diff --git a/rs/moq-net/sim/src/lib.rs b/rs/moq-net/sim/src/lib.rs index 629dfaa6de5..91ae336ae1a 100644 --- a/rs/moq-net/sim/src/lib.rs +++ b/rs/moq-net/sim/src/lib.rs @@ -328,27 +328,24 @@ pub fn timeout(duration: Duration, future: F) -> impl Future(mut poll: impl FnMut(Instant, &kio::Waiter) -> Result, E> + Unpin) -> E { let runtime = current(); let mut timer: Option = None; - // One cooperative budget per poll, like `moq_net::time::run`. kio::wait(move |waiter| { - kio::coop::budget(|| { - loop { - let at = match poll(runtime.now(), waiter) { - Ok(Some(at)) => at, - Ok(None) => { - timer = None; - return Poll::Pending; - } - Err(err) => return Poll::Ready(err), - }; - let sleep = timer.get_or_insert_with(|| sleep_until(at)); - if sleep.deadline() != at { - sleep.reset(at); - } - if waiter.poll_future(Pin::new(sleep)).is_pending() { + loop { + let at = match poll(runtime.now(), waiter) { + Ok(Some(at)) => at, + Ok(None) => { + timer = None; return Poll::Pending; } + Err(err) => return Poll::Ready(err), + }; + let sleep = timer.get_or_insert_with(|| sleep_until(at)); + if sleep.deadline() != at { + sleep.reset(at); } - }) + if waiter.poll_future(Pin::new(sleep)).is_pending() { + return Poll::Pending; + } + } }) .await } diff --git a/rs/moq-net/src/ietf/control.rs b/rs/moq-net/src/ietf/control.rs index 031d9c37a09..3ba504d7e2b 100644 --- a/rs/moq-net/src/ietf/control.rs +++ b/rs/moq-net/src/ietf/control.rs @@ -138,8 +138,6 @@ impl Control { return Poll::Ready(Ok(state.request_id_next.increment())); } - // A check that yielded to the cooperative budget may have allowed it. - std::task::ready!(kio::coop::poll_proceed(waiter)); if timeout.poll(waiter).is_ready() { tracing::warn!("timed out waiting for MAX_REQUEST_ID"); return Poll::Ready(Err(Error::Cancel)); diff --git a/rs/moq-net/src/ietf/publisher.rs b/rs/moq-net/src/ietf/publisher.rs index 989fcfe6244..cca598d6fd7 100644 --- a/rs/moq-net/src/ietf/publisher.rs +++ b/rs/moq-net/src/ietf/publisher.rs @@ -123,7 +123,10 @@ async fn read_fetch( let first = group.index(); let mut frames = Vec::new(); let mut complete = false; + // A long cached group is read a slice per poll, so the session's other tasks still run. + let mut budget = kio::coop::Budget::new(32); while until.is_none_or(|until| first + (frames.len() as u64) < until) { + kio::wait(|waiter| budget.poll_yield(waiter)).await; match group.read_frame().await? { Some(frame) => frames.push(frame), None => { @@ -1125,7 +1128,10 @@ where let mut first = true; let mut buf: frame::Buffer = frame::Buffer::new(); + // A long cached group is written a slice per poll, so the session's other tasks still run. + let mut budget = kio::coop::Budget::new(32); 'serve: loop { + kio::wait(|waiter| budget.poll_yield(waiter)).await; // The cap is the Largest Object snapshot: the group may keep growing, but // everything past the snapshot belongs to the subscription, not the fill. if until.is_some_and(|until| index >= until) { @@ -1139,9 +1145,6 @@ where return Poll::Ready(Err(Error::Cancel)); } match group.poll_read_frames(waiter, &mut buf) { - // Out of cooperative budget, the batch it filled waits for the next turn - // rather than draining through the chunked read for free. - Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => Poll::Pending, Poll::Pending => match group.poll_next_frame(waiter) { Poll::Ready(Ok(Some(frame))) => Poll::Ready(Ok(FillStep::Partial(frame))), Poll::Ready(Ok(None)) => Poll::Ready(Ok(FillStep::Done)), @@ -1701,8 +1704,6 @@ where if let Poll::Ready(joins) = joins { return Poll::Ready(Ok(joins.get(&subscribe_id).cloned().flatten())); } - // A lookup that yielded to the cooperative budget may have found it. - ready!(kio::coop::poll_proceed(waiter)); if deadline.poll(waiter).is_ready() { return Poll::Ready(if pending { Err(Error::Timeout) } else { Ok(None) }); } @@ -2659,6 +2660,8 @@ struct TrackServe { /// Serve the track's datagrams too, as OBJECT_DATAGRAMs. Off when the transport has no /// datagrams; there is no stream fallback. datagrams: bool, + /// A track that always has another group ready still lets the session's other tasks run. + budget: kio::coop::Budget, } impl TrackServe { @@ -2697,6 +2700,7 @@ impl TrackServe { draining: false, opened: Default::default(), end: None, + budget: kio::coop::Budget::new(32), } } @@ -2755,6 +2759,10 @@ impl TrackServe { let _ = self.children.poll(waiter); loop { + // Still serves the datagrams and the groups already started below. + if self.budget.poll_yield(waiter).is_pending() { + break; + } match self.track.poll_recv_group(waiter) { Poll::Ready(Ok(Some(group))) => { let sequence = group.sequence; @@ -2808,9 +2816,6 @@ impl TrackServe { Poll::Ready(Ok(None)) => { // Datagrams written before the track finished still go out. self.poll_datagrams(waiter); - // Draining never looks again, so a yield to the cooperative budget (in - // the datagrams or the end) retries the whole step next turn. - ready!(kio::coop::poll_proceed(waiter)); self.draining = true; if let Poll::Ready(Ok(end)) = self.track.poll_finished(waiter) { self.end = Some(end); @@ -2894,6 +2899,8 @@ struct GroupServe { version: Version, object_delta: u64, state: GroupState, + /// A long cached group drains a slice per poll, so its siblings still run. + budget: kio::coop::Budget, } // A state machine's enum is its storage: one transient instance per stream, so the @@ -2953,6 +2960,7 @@ impl GroupServe { version, object_delta, state: GroupState::Open, + budget: kio::coop::Budget::new(32), } } @@ -3009,6 +3017,7 @@ impl GroupServe { } let res = 'serve: { loop { + ready!(self.budget.poll_yield(waiter)); match writer.poll_flush(&mut cx) { Poll::Ready(Ok(())) => {} Poll::Ready(Err(err)) => break 'serve Err(err), @@ -3076,9 +3085,7 @@ impl GroupServe { } Poll::Ready(Ok(_)) => break 'serve Ok(()), Poll::Ready(Err(err)) => break 'serve Err(err), - // Out of cooperative budget, the batch it filled waits for the next turn - // rather than draining through the chunked read for free. - Poll::Pending => ready!(kio::coop::poll_proceed(waiter)), + Poll::Pending => {} } match self.group.poll_next_frame(waiter) { @@ -3332,6 +3339,132 @@ mod group_priority_test { ); } + /// Drive `serve` beside a sibling that only counts its turns, until `done`, checking every + /// poll stays under `most` by `progress` and leaves the sibling its turn. + fn serve_beside_a_sibling( + mut serve: TrackServe, + mut produce: impl FnMut() -> bool + 'static, + progress: impl Fn() -> usize, + most: usize, + done: usize, + ) { + type Task = Box Poll<()>>; + let mut tasks: kio::Tasks = kio::Tasks::new(); + tasks.push(Box::new(move |waiter: &kio::Waiter| { + if produce() { + return Poll::Ready(()); + } + waiter.waker().wake_by_ref(); + Poll::Pending + })); + let turns = std::sync::Arc::new(AtomicU64::new(0)); + let counted = turns.clone(); + tasks.push(Box::new(move |waiter: &kio::Waiter| { + counted.fetch_add(1, Ordering::Relaxed); + waiter.waker().wake_by_ref(); + Poll::Pending + })); + tasks.push(Box::new(move |waiter: &kio::Waiter| { + serve.poll(waiter).map(|res| res.expect("serve")) + })); + + let owner = kio::Waiter::noop(); + let mut polls = 0; + while progress() < done { + let before = progress(); + assert!(tasks.poll(&owner).is_pending(), "the serve never ends"); + polls += 1; + let made = progress() - before; + assert!(made <= most, "one poll made {made} progress"); + assert_eq!(turns.load(Ordering::Relaxed), polls, "the sibling missed a turn"); + assert!(polls <= 2 * done as u64, "the serve stalled"); + } + } + + /// A track that always has another group ready yields within its budget. + #[test] + fn a_busy_track_yields_to_its_siblings() { + /// `TrackServe`'s passes per poll; each starts at most one group. + const BUDGET: usize = 32; + const GROUPS: u64 = 10_000; + const BATCH: u64 = 1_000; + + let session = SinkSession::new(crate::lite::test_transport::Log::default()).with_unacked_fin(); + let track = track::Producer::new(std::sync::Arc::new(crate::broadcast::Info::default()), "test", None); + let subscriber = + track.subscribe(track::Subscription::default().with_max_delay(std::time::Duration::from_secs(3600))); + let serve = TrackServe::new( + session, + subscriber, + RequestId(0), + Version::Draft14, + ServeRange::default(), + None, + ); + let opened = serve.opened.clone(); + + let producer = track.clone(); + let mut appended = 0; + let produce = move || { + for _ in 0..BATCH.min(GROUPS - appended) { + let mut group = producer.create_group(group::Info { sequence: appended }).unwrap(); + group + .write_frame(crate::Timestamp::from_millis(appended).unwrap(), b"x".as_slice()) + .unwrap(); + group.finish().unwrap(); + appended += 1; + } + appended == GROUPS + }; + serve_beside_a_sibling( + serve, + produce, + || opened.load(Ordering::Relaxed) as usize, + BUDGET, + GROUPS as usize, + ); + } + + /// A group with more objects ready than one poll's budget drains over several polls. + #[test] + fn a_long_group_drains_within_the_budget() { + /// `GroupServe`'s passes per poll; each writes at most one object. + const BUDGET: usize = 32; + const FRAMES: usize = 4_000; + const PAYLOAD: usize = 100; + + let log = crate::lite::test_transport::Log::default(); + let session = SinkSession::new(log.clone()).with_unacked_fin(); + let track = track::Producer::new(std::sync::Arc::new(crate::broadcast::Info::default()), "test", None); + let subscriber = + track.subscribe(track::Subscription::default().with_max_delay(std::time::Duration::from_secs(3600))); + let mut group = track.create_group(group::Info { sequence: 0 }).unwrap(); + for millis in 0..FRAMES as u64 { + group + .write_frame(crate::Timestamp::from_millis(millis).unwrap(), vec![0u8; PAYLOAD]) + .unwrap(); + } + group.finish().unwrap(); + let serve = TrackServe::new( + session, + subscriber, + RequestId(0), + Version::Draft14, + ServeRange::default(), + None, + ); + + // An object's header takes well under a payload more, and a poll may also flush what + // the last one buffered. + serve_beside_a_sibling( + serve, + || true, + || log.writes.lock().unwrap().len(), + 2 * (BUDGET + 1) * 2 * PAYLOAD, + FRAMES * PAYLOAD, + ); + } + /// Serve one group at `version` with `timescale` and return the header plus what follows it. async fn serve_group(version: Version, timescale: Option) -> (ietf::GroupHeader, bytes::Bytes) { let log = crate::lite::test_transport::Log::default(); diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 07c1d76da5e..d57fc404199 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -645,8 +645,6 @@ async fn resolve_track_alias( if let Poll::Ready(result) = resolved { return Poll::Ready(result.unwrap_or(Err(Error::Dropped))); } - // A lookup that yielded to the cooperative budget may have found it. - ready!(kio::coop::poll_proceed(waiter)); if timeout.poll(waiter).is_ready() { return Poll::Ready(Err(Error::NotFound)); } @@ -1744,8 +1742,6 @@ where if let Poll::Ready(next) = broadcast.poll_requested_track(waiter) { return Poll::Ready(Some(next)); } - // A yield to the cooperative budget, not an empty queue. - ready!(kio::coop::poll_proceed(waiter)); if route.as_ref().is_some_and(|route| route.poll_closed(waiter).is_ready()) { source.close(); continue; @@ -4413,28 +4409,6 @@ mod tests { assert_eq!(pending.await.unwrap(), RequestId(11)); } - /// An alias bound by the time the timeout fires still resolves when the lookup yields - /// to the cooperative budget: the yield is no "not found". - #[moq_net_sim::test] - async fn a_budget_yield_is_not_an_alias_timeout() { - let runtime = crate::time::Clock::sim(); - let aliases = TrackAliases::default(); - let pending = resolve_track_alias(&runtime, aliases.consume(), 7); - let mut pending = std::pin::pin!(pending); - assert!(poll!(&mut pending).is_pending()); - - insert_track_alias(&aliases, 7, RequestId(11)).unwrap(); - moq_net_sim::sleep(TRACK_ALIAS_TIMEOUT).await; - - let mut cx = std::task::Context::from_waker(std::task::Waker::noop()); - let starved = kio::coop::budget(|| { - crate::util::leave_budget(0); - pending.as_mut().poll(&mut cx) - }); - assert!(starved.is_pending(), "the lookup yields: {starved:?}"); - assert_eq!(pending.await.unwrap(), RequestId(11)); - } - /// SUBSCRIBE_OK has not accepted the track, so the map holds no producer. /// Abort still has to reject the parked origin request with the session error. #[moq_net_sim::test] diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 4f79161fc68..af991a2c9bb 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -1024,6 +1024,8 @@ struct RequestServe> { // Log context, filled in after the decode. absolute: crate::PathOwned, track: String, + // A reply that always has another group ready still lets the session's other tasks run. + budget: kio::coop::Budget, } enum RequestState { @@ -1060,6 +1062,7 @@ impl> RequestServe { update: None, absolute: Default::default(), track: Default::default(), + budget: kio::coop::Budget::new(32), } } @@ -1095,6 +1098,7 @@ impl> RequestServe { fn poll_serve(&mut self, waiter: &kio::Waiter) -> Poll> { let mut cx = waiter.context(); loop { + ready!(self.budget.poll_yield(waiter)); // Once answered, the requester's FIN is the normal end, not a cancel. if matches!(self.state, RequestState::Resolve { .. } | RequestState::Serve(_)) && let Poll::Ready(res) = self.poll_requester(&mut cx) @@ -1698,9 +1702,7 @@ impl Request for FetchServe { } Poll::Ready(Ok(_)) => return Poll::Ready(Ok(ControlFlow::Break(()))), Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), - // Out of cooperative budget, the batch it filled waits for the next turn - // rather than draining through the chunked read for free. - Poll::Pending => ready!(kio::coop::poll_proceed(waiter)), + Poll::Pending => {} } match ready!(group.poll_next_frame(waiter))? { Some(next) => { @@ -1884,59 +1886,6 @@ mod test { } } - /// A finished track still announces its boundary when the poll that finds the groups - /// over has run out of budget: a yield there must not read as "no boundary". - #[test] - fn a_budget_yield_keeps_the_boundary() { - let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); - for left in 0..4 { - let mut producer = track_producer("test"); - let mut subscriber = producer.subscribe(None); - producer.finish_at(0).unwrap(); - - let starved = kio::coop::budget(|| { - crate::util::leave_budget(left); - poll_recv_next(&mut subscriber, false, true, &waiter) - }); - let res = match starved { - Poll::Pending => kio::coop::budget(|| poll_recv_next(&mut subscriber, false, true, &waiter)), - res => res, - }; - assert!( - matches!(res, Poll::Ready(Ok(Recv::Boundary(0)))), - "left {left}: the boundary was skipped" - ); - } - } - - /// A finished track's trailing datagram still goes out when the poll that finds the - /// groups over runs out of budget: a yield there must not read as "no datagram". - #[test] - fn a_budget_yield_keeps_the_trailing_datagram() { - let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); - for left in 0..4 { - let mut producer = track_producer("test"); - let mut subscriber = producer.subscribe(None); - producer - .append_datagram(Timestamp::from_millis(1).unwrap(), &b"last"[..]) - .unwrap(); - producer.finish().unwrap(); - - let starved = kio::coop::budget(|| { - crate::util::leave_budget(left); - poll_recv_next(&mut subscriber, true, false, &waiter) - }); - let res = match starved { - Poll::Pending => kio::coop::budget(|| poll_recv_next(&mut subscriber, true, false, &waiter)), - res => res, - }; - assert!( - matches!(&res, Poll::Ready(Ok(Recv::Datagram(datagram))) if &datagram.payload[..] == b"last"), - "left {left}: the datagram was skipped" - ); - } - } - /// A relay can ingest back-to-back groups micro-reordered (the upstream leg /// sends newest-first). The older group is cached and in demand, so serving /// must still deliver it; a sequence cursor would skip it permanently. @@ -2380,8 +2329,6 @@ fn poll_recv_next( Poll::Pending => {} } } - // A read above that yielded to the cooperative budget is no answer: decide next turn. - ready!(kio::coop::poll_proceed(waiter)); // No live data ready: report the boundary (if declared) before signalling Finished, so a // future boundary reaches the subscriber while the trailing groups are still in flight. if emit_boundary && let Poll::Ready(res) = track.poll_finished(waiter) { @@ -2912,6 +2859,8 @@ struct GroupServe { // subsequent delta is signed against the previous frame. prev_ts: u64, state: GroupState, + // A long cached group drains a slice per poll, so its siblings still run. + budget: kio::coop::Budget, } // A state machine's enum is its storage: one transient instance per stream, so the @@ -2954,6 +2903,7 @@ impl GroupServe { frame_start, prev_ts: 0, state: GroupState::Open, + budget: kio::coop::Budget::new(32), } } @@ -3051,6 +3001,7 @@ impl GroupServe { break 'serve Err(Error::Cancel); } loop { + ready!(self.budget.poll_yield(waiter)); match writer.poll_flush(&mut cx) { Poll::Ready(Ok(())) => {} Poll::Ready(Err(err)) => break 'serve Err(err), @@ -3118,9 +3069,7 @@ impl GroupServe { } Poll::Ready(Ok(_)) => break 'serve Ok(()), Poll::Ready(Err(err)) => break 'serve Err(err), - // Out of cooperative budget, the batch it filled waits for the next turn - // rather than draining through the chunked read for free. - Poll::Pending => ready!(kio::coop::poll_proceed(waiter)), + Poll::Pending => {} } match self.group.poll_next_frame(waiter) { @@ -3708,83 +3657,11 @@ mod serve_group_test { group.finish().unwrap(); } - /// A serve loop that always has another group ready yields within kio's cooperative - /// budget, so a sibling (the transport's driver, say) runs between its turns. Without - /// it, one poll served every group the producer kept appending and starved the rest - /// of the runtime (a go publisher's 2.5 ms audio groups timed its session out). - #[test] - fn a_busy_serve_yields_to_its_siblings() { - /// kio's per-turn budget: each served group spends at least one unit. - const BUDGET: u64 = 32; - const GROUPS: u64 = 10_000; - const BATCH: u64 = 1_000; - - type Task = Box Poll<()>>; - - let track = track::Producer::new(Arc::new(broadcast::Info::default()), "test", None); - let subscriber = track.subscribe(track::Subscription::default().with_max_delay(Duration::from_secs(3600))); - // Unacknowledged FINs keep every served group in flight. - let log = Log::default(); - let (mut run, mut writer, _) = lite07_run(SinkSession::new(log.clone()).with_unacked_fin(), subscriber); - let opens = run.ctx.opens.clone(); - - let mut tasks: kio::Tasks = kio::Tasks::new(); - - // The producer appends a batch of ready groups on each of its turns. A clone, so the - // track outlives the task. - let mut producer = track.clone(); - let mut appended = 0; - tasks.push(Box::new(move |waiter: &kio::Waiter| { - for _ in 0..BATCH.min(GROUPS - appended) { - write_group(&mut producer, appended, appended); - appended += 1; - } - if appended == GROUPS { - return Poll::Ready(()); - } - waiter.waker().wake_by_ref(); - Poll::Pending - })); - - // Stands in for the transport's driver: it only needs a turn. - let turns = Arc::new(AtomicU64::new(0)); - let counted = turns.clone(); - tasks.push(Box::new(move |waiter: &kio::Waiter| { - counted.fetch_add(1, Ordering::Relaxed); - waiter.waker().wake_by_ref(); - Poll::Pending - })); - - tasks.push(Box::new(move |waiter: &kio::Waiter| { - run.poll(&mut writer, waiter).map(|res| res.expect("serve")) - })); - - let owner = kio::Waiter::noop(); - let mut polls = 0; - while opens.opened.load(Ordering::Relaxed) < GROUPS { - let before = opens.opened.load(Ordering::Relaxed); - assert!(tasks.poll(&owner).is_pending(), "the serve never ends"); - polls += 1; - let served = opens.opened.load(Ordering::Relaxed) - before; - assert!(served <= BUDGET, "one turn served {served} groups"); - assert_eq!(turns.load(Ordering::Relaxed), polls, "the sibling missed a turn"); - assert!(polls <= 2 * GROUPS, "the serve stalled"); - } - - // Every group was served exactly once, and nothing more is left to serve. - for _ in 0..4 { - assert!(tasks.poll(&owner).is_pending()); - } - assert_eq!(opens.opened.load(Ordering::Relaxed), GROUPS); - assert!(log.resets().is_empty(), "a group was reset: {:?}", log.resets()); - } - - /// A group with more frames ready than a turn's budget drains over several turns: a - /// batch the budget cut short waits for the next turn instead of draining through the - /// chunked fallback for free. + /// A group with more frames ready than one poll's budget drains over several polls, + /// so a sibling (the transport's driver, say) runs in between. #[test] fn a_long_group_drains_within_the_budget() { - /// kio's per-turn budget; each refill of up to 8 frames spends a unit. + /// `GroupServe`'s passes per poll; each writes at most one frame. const BUDGET: usize = 32; const FRAMES: usize = 4_000; const PAYLOAD: usize = 100; @@ -3816,8 +3693,9 @@ mod serve_group_test { run.poll(&mut writer, waiter).map(|res| res.expect("serve")) })); - // A frame's header (size and timestamp delta) takes well under a payload more. - let most = (BUDGET + 1) * 8 * 2 * PAYLOAD; + // A frame's header takes well under a payload more, and a poll may also flush what + // the last one buffered. + let most = 2 * (BUDGET + 1) * 2 * PAYLOAD; let owner = kio::Waiter::noop(); let mut polls = 0; while log.writes.lock().unwrap().len() < FRAMES * PAYLOAD { @@ -3825,7 +3703,7 @@ mod serve_group_test { assert!(tasks.poll(&owner).is_pending(), "the serve never ends"); polls += 1; let wrote = log.writes.lock().unwrap().len() - before; - assert!(wrote <= most, "one turn wrote {wrote} bytes"); + assert!(wrote <= most, "one poll wrote {wrote} bytes"); assert_eq!(turns.load(Ordering::Relaxed), polls, "the sibling missed a turn"); assert!(polls <= FRAMES as u64, "the serve stalled"); } @@ -4737,6 +4615,129 @@ mod tests { assert_eq!(*run.track_priority_tx.read(), 7, "the update was lost"); } + /// A subscription that always has another group ready yields within its budget, so a + /// sibling (the transport's driver, say) runs between its polls. Without it, one poll + /// served every group the producer kept appending and starved the rest of the runtime + /// (a go publisher's 2.5 ms audio groups timed its session out). + #[moq_net_sim::test] + async fn a_busy_subscription_yields_to_its_siblings() { + /// `RequestServe`'s passes per poll; each serves at most one group. + const BUDGET: u64 = 32; + const GROUPS: u64 = 10_000; + const BATCH: u64 = 1_000; + + type Task = Box Poll<()>>; + + let origin = crate::origin::Config::new(crate::Hop::new(1).unwrap()).produce(); + let broadcast = origin.create_broadcast("room").unwrap(); + let mut dynamic = broadcast.dynamic(); + broadcast.announce(Default::default()).unwrap(); + + let peer_setup = crate::lite::PeerSetup::default(); + peer_setup.set(crate::lite::Setup::default()); + let (_, goaway) = crate::goaway::Handle::new(true); + let version = Version::Lite07; + let publisher = Publisher::new(PublisherConfig { + runtime: crate::time::Clock::sim(), + session: ScriptedSession::new(Vec::new()), + origin: origin.consume(), + version, + peer_setup, + goaway, + peer_hop: None, + subscriptions: Default::default(), + }); + + let mut script = Vec::new(); + lite::Subscribe { + epoch: None, + id: 0, + broadcast: crate::Path::new("room"), + track: "video".into(), + priority: 0, + max_delay: Duration::from_secs(3600), + start_group: None, + end_group: None, + start_frame: 0, + end_frame: None, + } + .encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) + .unwrap(); + let mut session = ScriptedSession::new(script); + let (send, recv) = futures::future::poll_fn(|cx| { + ::poll_open_bi(&mut session, cx) + }) + .await + .unwrap(); + let stream = Stream:: { + writer: Writer::new(send, version), + reader: crate::coding::Reader::new(recv, version), + }; + let mut serve = RequestServe::<_, SubscribeServe>::new(publisher.shared.clone(), stream); + assert!(!drive(&mut serve).await); + let track = dynamic + .requested_track() + .now_or_never() + .expect("track requested") + .unwrap() + .accept(None); + assert!(!drive(&mut serve).await, "the subscription ended"); + let RequestState::Serve(SubscribeServe::Run(run)) = &serve.state else { + panic!("the subscription is not running"); + }; + let opens = run.ctx.opens.clone(); + + let mut tasks: kio::Tasks = kio::Tasks::new(); + + // The producer appends a batch of ready groups on each of its turns. + let mut appended = 0; + tasks.push(Box::new(move |waiter: &kio::Waiter| { + for _ in 0..BATCH.min(GROUPS - appended) { + let mut group = track.create_group(group::Info { sequence: appended }).unwrap(); + group + .write_frame(crate::Timestamp::from_millis(appended).unwrap(), b"x".as_slice()) + .unwrap(); + group.finish().unwrap(); + appended += 1; + } + if appended == GROUPS { + return Poll::Ready(()); + } + waiter.waker().wake_by_ref(); + Poll::Pending + })); + + // Stands in for the transport's driver: it only needs a turn. + let turns = Arc::new(AtomicU64::new(0)); + let counted = turns.clone(); + tasks.push(Box::new(move |waiter: &kio::Waiter| { + counted.fetch_add(1, Ordering::Relaxed); + waiter.waker().wake_by_ref(); + Poll::Pending + })); + + tasks.push(Box::new(move |waiter: &kio::Waiter| { + serve.poll(waiter).map(|res| res.expect("serve")) + })); + + let owner = kio::Waiter::noop(); + let mut polls = 0; + while opens.opened.load(Ordering::Relaxed) < GROUPS { + let before = opens.opened.load(Ordering::Relaxed); + assert!(tasks.poll(&owner).is_pending(), "the serve never ends"); + polls += 1; + let served = opens.opened.load(Ordering::Relaxed) - before; + assert!(served <= BUDGET, "one poll served {served} groups"); + assert_eq!(turns.load(Ordering::Relaxed), polls, "the sibling missed a turn"); + assert!(polls <= 2 * GROUPS, "the serve stalled"); + } + assert!( + session.log.resets().is_empty(), + "a group was reset: {:?}", + session.log.resets() + ); + } + /// A SUBSCRIBE or a TRACK past the session's cap closes the session with /// TOO_MANY_REQUESTS. #[moq_net_sim::test] diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 3e264719ad2..f7c8ed0470d 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -1494,8 +1494,6 @@ impl kio::Task for SourceServe { tracing::debug!(%err, "source closed"); self.ended = true; } - // A yield to the cooperative budget, not an empty queue. - Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => break, Poll::Pending => { // The route that minted the source went: close it now. if self.route.poll_closed(waiter).is_ready() { @@ -2732,14 +2730,6 @@ mod tests { let (_resolved, mut serve) = serve_one(&origin, &subscriber, &mut announced).await; assert!(kio::Task::poll(&mut serve, &kio::Waiter::noop()).is_pending()); announced.withdraw(&path); - // Out of cooperative budget, the close waits a turn rather than spinning on a - // request poll that keeps yielding. - let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); - let starved = kio::coop::budget(|| { - crate::util::leave_budget(0); - kio::Task::poll(&mut serve, &waiter) - }); - assert!(starved.is_pending()); assert!(kio::Task::poll(&mut serve, &kio::Waiter::noop()).is_ready()); } @@ -4435,8 +4425,6 @@ impl ServeLoop { { return Poll::Ready(ServeEnd::Finished); } - // Settling may have yielded to the cooperative budget instead. - ready!(kio::coop::poll_proceed(waiter)); if self.fetches.is_empty() && self.demand.poll_unused(waiter).is_ready() { return Poll::Ready(ServeEnd::Idle); } diff --git a/rs/moq-net/src/model/frame.rs b/rs/moq-net/src/model/frame.rs index 09a6ff51772..b0acdcf9281 100644 --- a/rs/moq-net/src/model/frame.rs +++ b/rs/moq-net/src/model/frame.rs @@ -731,8 +731,6 @@ impl Consumer { let failed = match &res { Poll::Ready(Ok(_)) => return res, Poll::Ready(Err(err)) => Some(err.clone()), - // A yield to the cooperative budget is no stall. - Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => return res, Poll::Pending => None, }; let (recover, index) = &mut **recover; @@ -753,7 +751,6 @@ impl Consumer { Poll::Ready(Ok(Some(frame))) => frame, Poll::Ready(Ok(None)) => return Poll::Ready(Err(Error::WrongSize)), Poll::Ready(Err(err)) => return Poll::Ready(Err(err)), - Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => return Poll::Pending, Poll::Pending => { return match self.poll_expired(waiter) { true => Poll::Ready(Err(Error::Old)), diff --git a/rs/moq-net/src/model/group.rs b/rs/moq-net/src/model/group.rs index be3869bc145..1ea02ec1748 100644 --- a/rs/moq-net/src/model/group.rs +++ b/rs/moq-net/src/model/group.rs @@ -1375,8 +1375,6 @@ impl Consumer { return res; } Poll::Ready(Err(err)) => Some(err.clone()), - // A yield to the cooperative budget is no stall. - Poll::Pending if kio::coop::poll_proceed(waiter).is_pending() => return res, Poll::Pending => None, }; if !recover.wants(failed.as_ref(), waiter) { @@ -1421,8 +1419,7 @@ impl Consumer { if self.ended { return Some(false); } - // A read the cooperative budget turned into `Pending` may have had a frame ready. - if kio::coop::poll_proceed(waiter).is_pending() || !self.poll_expired(waiter) { + if !self.poll_expired(waiter) { return None; } let truncates = self.expired_truncates(); @@ -1877,23 +1874,21 @@ impl Cursor { state.poll_terminal(index) }); - // Count a fresh batch once here, under no lock, so the drained pops that follow - // stay free. Before `ready!`: a poll that yields to the cooperative budget keeps - // the batch it filled, and the next poll pops it on the fast path. - let (frames, bytes) = self.prefetch.buffered(); - if frames > 0 { - // The refill already updated the eviction rank under the group lock. - self.refreshed = self.cache.pool().now(); - stats.frames(frames); - stats.bytes(bytes); - } - match ready!(res) { Ok(Ok(())) => {} Ok(Err(err)) => return Poll::Ready(Err(err)), Err(state) => return Poll::Ready(Err(state.abort.clone().unwrap_or(Error::Dropped))), } + // The refill already updated the eviction rank under the group lock. + self.refreshed = self.cache.pool().now(); + + // A fresh batch was just filled (empty only on a clean end). Count the whole + // batch once here, under no lock, so the drained pops that follow stay free. + let (frames, bytes) = self.prefetch.buffered(); + stats.frames(frames); + stats.bytes(bytes); + Poll::Ready(Ok(self.prefetch.pop().inspect(|_| { self.index += 1; }))) @@ -2041,48 +2036,6 @@ mod test { assert_eq!(frame.payload, Bytes::from_static(b"hello")); } - /// A refill that yields to the cooperative budget keeps its batch, and still counts it. - #[test] - fn a_budget_yield_keeps_and_counts_the_prefetched_batch() { - let stats = crate::stats::Registry::new(crate::stats::Config::new()); - let meter = stats - .tier(crate::stats::Tier::default()) - .session("root") - .egress("demo") - .meter(); - - let mut producer = Info { sequence: 0 }.produce(); - for payload in [&b"one"[..], b"two", b"three"] { - producer.write_frame(Timestamp::ZERO, Bytes::from(payload)).unwrap(); - } - producer.finish().unwrap(); - let mut consumer = producer.consume().with_meter(meter); - - let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); - let starved = kio::coop::budget(|| { - crate::util::leave_budget(0); - consumer.poll_read_frame(&waiter) - }); - assert!(starved.is_pending(), "the refill yields"); - - let mut read = Vec::new(); - while let Poll::Ready(Ok(Some(frame))) = kio::coop::budget(|| consumer.poll_read_frame(&waiter)) { - read.push(frame.payload); - } - assert_eq!(read, [&b"one"[..], b"two", b"three"]); - - let traffic = stats.snapshot().traffic(); - let (_, _, egress) = traffic - .iter() - .find(|(_, role, _)| *role == crate::stats::Role::Publisher) - .expect("egress row"); - assert_eq!( - (egress.frames, egress.bytes), - (3, 11), - "the yielded batch is counted once" - ); - } - /// An untimed frame reads back untimed, and still counts as the group's first frame: /// an empty group and a group on an untimed track are different answers. #[test] diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 32958d27356..32fd3cffc85 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -886,10 +886,6 @@ impl Recover { return Poll::Pending; } let res = self.poll_serving(generation, &serving, index, failed, waiter); - // A lookup that yielded to the cooperative budget is no verdict either way. - if res.is_pending() { - ready!(kio::coop::poll_proceed(waiter)); - } // The serving route failed too, and no front is left to replace it. if let (Poll::Pending, Some(err)) = (&res, failed) && (closed || end.is_some()) @@ -933,8 +929,6 @@ impl Recover { } } - // Not if the lookup above yielded to the cooperative budget: it may have found it. - ready!(kio::coop::poll_proceed(waiter)); // Otherwise ask for the rest. The subscription may still deliver it first. if self.fetch.as_ref().is_none_or(|(asked, _)| *asked != generation) { let options = group::Fetch::default().with_frame_start(index); @@ -1178,50 +1172,6 @@ mod test { ); } - /// A failed copy whose replacement holds the group still continues from it when the - /// lookup yields to the cooperative budget: the yield is no failed recovery. - #[test] - fn a_budget_yield_is_not_a_failed_recovery() { - let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); - for left in 0..4 { - let routes = Producer::new(); - let logical = logical(&routes); - let a = copy(); - routes.serve(a.consume()); - let mut sub = subscribe(&logical, Duration::from_secs(10)); - let mut open = a.create_group(group::Info { sequence: 0 }).unwrap(); - open.write_frame(ts(0), b"a".as_ref()).unwrap(); - let mut reading = recv(&mut sub); - assert_eq!(read(&mut reading), Some(b"a".to_vec())); - - // The replacement holds the whole group, then fails with nothing left to follow. - let b = copy(); - let mut whole = b.create_group(group::Info { sequence: 0 }).unwrap(); - for (ms, frame) in [(0, b"a"), (1, b"b")] { - whole.write_frame(ts(ms), frame.as_ref()).unwrap(); - } - whole.finish().unwrap(); - routes.serve(b.consume()); - b.abort(Error::Cancel).unwrap(); - routes.conclude(); - open.abort(Error::Dropped).unwrap(); - - let starved = kio::coop::budget(|| { - crate::util::leave_budget(left); - reading.poll_read_frame(&waiter) - }); - let res = match starved { - Poll::Pending => kio::coop::budget(|| reading.poll_read_frame(&waiter)), - res => res, - }; - assert!( - matches!(&res, Poll::Ready(Ok(Some(frame))) if frame.payload.as_ref() == b"b"), - "left {left}: {:?}", - res.map(|res| res.map(|frame| frame.map(|frame| frame.payload))) - ); - } - } - #[test] fn a_half_read_frame_continues_from_the_replacement() { let routes = Producer::new(); diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 93adbe9facd..c6c0716d643 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -3433,9 +3433,8 @@ impl kio::Task for Fetching { Poll::Pending => return Poll::Pending, }; // An accept caches the group before it answers the channel, which may land - // between the track check above and this one: look again before failing. A probe, - // so the cooperative budget can't turn a hit into that failure. - match cached(&kio::Waiter::noop()) { + // between the track check above and this one: look again before failing. + match cached(waiter) { Poll::Ready(res) => Poll::Ready(res), Poll::Pending => Poll::Ready(Err(err)), } @@ -3577,10 +3576,7 @@ impl group::Expiry for GroupExpiry { if let Some(reach) = successor.timestamp() { break reach; } - // Registers for the first frame; the timestamp is the verdict, so a yield to the - // cooperative budget cannot read as "not started". - let _ = successor.poll_started(waiter); - if successor.timestamp().is_none() { + if successor.poll_started(waiter).is_pending() || successor.timestamp().is_none() { return false; } }; @@ -3702,7 +3698,7 @@ impl Cursor { // A helper to automatically apply Dropped if the state is closed without an error. fn poll(&self, waiter: &kio::Waiter, f: F) -> Poll> where - F: FnOnce(&kio::Ref<'_, TrackState>) -> Poll>, + F: Fn(&kio::Ref<'_, TrackState>) -> Poll>, { Poll::Ready(match ready!(self.state.poll(waiter, f)) { Ok(res) => res, @@ -3723,29 +3719,29 @@ impl Cursor { /// group is only ever judged against content that could actually be served in its /// place. Read fresh each poll, so a mid-stream [`Control::update`] applies /// to the very next group, and shared across every candidate that poll walks off, so - /// discarding a backlog of N groups costs one scan rather than N. A plain read, like - /// [`Self::is_stale`], so it never yields to the cooperative budget; the track ending - /// surfaces as the error the caller was going to get anyway. - fn drift(&self, cap: Option, waiter: &kio::Waiter) -> Drift { + /// discarding a backlog of N groups costs one scan rather than N. Only ever + /// [`Poll::Ready`]; the track ending surfaces as the error the caller was going to + /// get anyway. + fn poll_drift(&self, cap: Option, waiter: &kio::Waiter) -> Poll> { let mut max_delay = Duration::default(); let _ = self.subscription.poll(waiter, |subscription| { max_delay = subscription.max_delay; Poll::<()>::Pending }); - let state = self.state.read(); - Drift { - budget: clamp_max_delay(max_delay, state.max_age_bound()), - edge: state.drift_edge(cap), - } + self.poll(waiter, |state| { + Poll::Ready(Ok(Drift { + budget: clamp_max_delay(max_delay, state.max_age_bound()), + edge: state.drift_edge(cap), + })) + }) } /// Whether the drift budget says to skip `group`, against a [`Drift`] already resolved /// for this poll. - /// - /// A plain read, not a poll: the caller has already taken `group` off its cursor, so - /// yielding to the cooperative budget here would drop it. - fn is_stale(&self, group: &group::Consumer, drift: Drift) -> bool { - self.state.read().is_stale(group.sequence, &drift.edge, drift.budget) + fn poll_stale(&self, group: &group::Consumer, drift: Drift, waiter: &kio::Waiter) -> Poll> { + self.poll(waiter, |state| { + Poll::Ready(Ok(state.is_stale(group.sequence, &drift.edge, drift.budget))) + }) } fn with_expiry(&self, group: group::Consumer) -> group::Consumer { @@ -3780,7 +3776,7 @@ impl Cursor { .retain(|sequence, group| *sequence >= min_sequence && watch(group)); // One scan for the whole poll, so walking a backlog off stays linear in its size. - let drift = self.drift(self.end_sequence, waiter); + let drift = ready!(self.poll_drift(self.end_sequence, waiter))?; loop { // Re-offer the lowest parked group back inside the cap once it rises, @@ -3826,7 +3822,7 @@ impl Cursor { // Drop a group the drift budget has given up on and keep scanning, so one // poll walks a whole backlog off rather than handing it out group by group. - if self.is_stale(&consumer, drift) { + if ready!(self.poll_stale(&consumer, drift, waiter))? { self.stale.add(consumer.content()); continue; } @@ -3867,50 +3863,38 @@ impl Cursor { /// The lowest servable group past the last one returned, walking off everything the /// drift budget has convicted on the way. fn poll_next_group(&mut self, waiter: &kio::Waiter) -> Poll>> { - let floor = self.next_sequence.max(self.min_sequence); + let mut floor = self.next_sequence.max(self.min_sequence); let end = self.end_sequence; // One scan for the whole poll, so walking a backlog off stays linear in its size. - let drift = self.drift(end, waiter); - - // Walk the backlog under one lock, so it spends one unit of the cooperative budget - // however many groups it skips: a poll per skip would yield partway through a long - // backlog and restart at `floor` every turn. - let mut convicted = Vec::new(); - let res = self.poll(waiter, |state| { - let mut floor = floor; - loop { - let Some(producer) = ready!(state.poll_next_in_range(floor, end))? else { - return Poll::Ready(Ok(None)); - }; - if !state.is_stale(producer.sequence, &drift.edge, drift.budget) { - return Poll::Ready(Ok(Some(producer))); - } - floor = producer.sequence.saturating_add(1); - convicted.push(producer); - } - }); - - // Recorded even after a yield, since the next walk only re-snapshots them. Not - // counted yet: a conviction is not permanent (the budget can widen, the edge can be - // evicted), so the group may still be delivered. Consumed with the track guard - // released. - for producer in convicted { + let drift = ready!(self.poll_drift(end, waiter))?; + + let group = loop { + let Some(producer) = ready!(self.poll(waiter, |state| state.poll_next_in_range(floor, end))?) else { + // Deliberately no flush of `seek_pending` here: only a delivery commit + // may count a conviction. This `None` can be an artifact of a floor + // that will lower again. A conviction never committed is dropped + // uncounted with the cursor. + return Poll::Ready(Ok(None)); + }; let group = producer.consume(); - self.seek_pending.insert(group.sequence, group.content()); - } - let Some(producer) = ready!(res)? else { - // Deliberately no flush of `seek_pending` here: only a delivery commit - // may count a conviction. This `None` can be an artifact of a floor - // that will lower again. A conviction never committed is dropped - // uncounted with the cursor. - return Poll::Ready(Ok(None)); + // Skip a group the budget has given up on and keep scanning, so one poll + // walks a whole backlog off rather than handing it out group by group. + if ready!(self.poll_stale(&group, drift, waiter))? { + // Not counted yet: a conviction is not permanent (the budget can widen, + // the edge can be evicted), so the group may still be delivered. + // Re-snapshot on every re-examination so the eventual count reflects the + // group's latest observed content. + self.seek_pending.insert(group.sequence, group.content()); + floor = group.sequence.saturating_add(1); + continue; + } + + // A conviction the budget walked back: the group is handed over after all, + // so it must never reach the stale count. + self.seek_pending.remove(&group.sequence); + break self.with_expiry(group); }; - let group = producer.consume(); - // A conviction the budget walked back: the group is handed over after all, - // so it must never reach the stale count. - self.seek_pending.remove(&group.sequence); - let group = self.with_expiry(group); self.next_sequence = group.sequence.saturating_add(1); // The delivery commits everything the seek stepped over to reach this group. @@ -8012,129 +7996,6 @@ mod test { assert!(consumer.next_group().now_or_never().is_none(), "no more groups"); } - /// Read every group `poll` hands out, running out of budget at every position a - /// poll can: a yield must only postpone, so each group still arrives exactly once. - fn read_through_yields( - left: usize, - mut poll: impl FnMut(&kio::Waiter) -> Poll>>, - ) -> Vec { - let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); - let mut read = Vec::new(); - loop { - // Starved first, then a whole budget, so every read also makes progress. - let starved = kio::coop::budget(|| { - crate::util::leave_budget(left); - poll(&waiter) - }); - let res = match starved { - Poll::Pending => kio::coop::budget(|| poll(&waiter)), - res => res, - }; - match res { - Poll::Ready(Ok(Some(group))) => read.push(group.sequence), - Poll::Ready(Ok(None)) | Poll::Pending => return read, - Poll::Ready(Err(err)) => panic!("read failed: {err}"), - } - } - } - - /// Fresh and parked groups alike survive a poll that runs out of budget part way. - #[test] - fn a_budget_yield_never_drops_a_group() { - for left in 0..4 { - for ordered in [false, true] { - let producer = track_producer("test", None); - let mut subscriber = producer.subscribe(replay()); - subscriber.set_groups(..4); - for sequence in 0..8 { - let mut group = producer.create_group(group::Info { sequence }).unwrap(); - group - .write_frame(Timestamp::from_millis(sequence).unwrap(), b"x".as_slice()) - .unwrap(); - group.finish().unwrap(); - } - - let (mut plain, mut sequenced) = match ordered { - true => (None, Some(subscriber.ordered())), - false => (Some(subscriber), None), - }; - let mut read = Vec::new(); - // The first half is fresh, the rest parks above the cap until it lifts. - for lift in [false, true] { - if lift { - plain.iter_mut().for_each(|sub| sub.set_groups(..)); - sequenced.iter_mut().for_each(|sub| sub.set_groups(..)); - } - read.extend(read_through_yields(left, |waiter| match (&mut plain, &mut sequenced) { - (Some(sub), _) => sub.poll_recv_group(waiter), - (_, Some(sub)) => sub.poll_next_group(waiter), - _ => unreachable!(), - })); - } - assert_eq!(read, (0..8).collect::>(), "left {left}, ordered {ordered}"); - } - } - } - - /// A stale backlog longer than a whole budget still reaches the deliverable group. - #[test] - fn an_ordered_walk_past_a_long_stale_backlog_spends_once() { - let mut producer = track_producer("test", None); - // Retains the backlog the reader under test walks off. - let _retain = producer.subscribe(replay()); - let mut ordered = producer.subscribe(None).ordered(); - let backlog = 64; - for millis in 0..=backlog { - append_at(&mut producer, millis); - } - - let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); - let res = kio::coop::budget(|| ordered.poll_next_group(&waiter)); - let Poll::Ready(Ok(Some(group))) = res else { - panic!("one turn delivers the newest group"); - }; - assert_eq!(group.sequence, backlog); - } - - /// A read the cooperative budget turns into `Pending` is not a stalled read, so the - /// drift budget leaves the group's buffered frame alone. - #[test] - fn a_budget_yield_is_not_a_stalled_read() { - for chunked in [false, true] { - let mut producer = track_producer("test", None); - let mut subscriber = producer.subscribe(None); - append_at(&mut producer, 0); - let mut group = subscriber.recv_group().now_or_never().unwrap().unwrap().expect("group"); - // The live edge moves a whole budget past the group's only frame. - append_at(&mut producer, 1000); - - let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); - let read = |group: &mut group::Consumer| -> Poll>> { - match chunked { - false => group - .poll_read_frame(&waiter) - .map_ok(|frame| frame.map(|frame| frame.payload)), - true => match ready!(group.poll_next_frame(&waiter)?) { - Some(mut frame) => frame.poll_read_chunk(&waiter), - None => Poll::Ready(Ok(None)), - }, - } - }; - let starved = kio::coop::budget(|| { - crate::util::leave_budget(0); - read(&mut group) - }); - assert!(starved.is_pending(), "chunked {chunked}: the read yields: {starved:?}"); - - let res = kio::coop::budget(|| read(&mut group)); - assert!( - matches!(&res, Poll::Ready(Ok(Some(payload))) if payload.as_ref() == b"x"), - "chunked {chunked}: the buffered frame is still delivered: {res:?}" - ); - assert!(!group.latency_expired()); - } - } - #[test] fn end_at_lower_than_cursor_parks_consumer() { let producer = track_producer("test", None); diff --git a/rs/moq-net/src/time.rs b/rs/moq-net/src/time.rs index 14a84b2516d..2bfcd5fffda 100644 --- a/rs/moq-net/src/time.rs +++ b/rs/moq-net/src/time.rs @@ -37,33 +37,29 @@ pub trait Driver { /// terminal error, [`Error::Closed`] for a clean finish. pub async fn run(mut driver: D) -> Error { let mut timer: Option>> = None; - // One budget per runtime poll, outside the loop: a timer that is always due must - // not refill it per pass. kio::wait(|waiter| { - kio::coop::budget(|| { - loop { - let now = web_async::time::Instant::now(); - #[cfg(not(target_family = "wasm"))] - let now = now.into_std(); - let at = match driver.poll(now, waiter) { - Ok(Some(at)) => at, - Ok(None) => { - timer = None; - return Poll::Pending; - } - Err(err) => return Poll::Ready(err), - }; - #[cfg(not(target_family = "wasm"))] - let at = web_async::time::Instant::from_std(at); - let sleep = timer.get_or_insert_with(|| Box::pin(web_async::time::sleep_until(at))); - if sleep.deadline() != at { - sleep.as_mut().reset(at); - } - if waiter.poll_future(sleep.as_mut()).is_pending() { + loop { + let now = web_async::time::Instant::now(); + #[cfg(not(target_family = "wasm"))] + let now = now.into_std(); + let at = match driver.poll(now, waiter) { + Ok(Some(at)) => at, + Ok(None) => { + timer = None; return Poll::Pending; } + Err(err) => return Poll::Ready(err), + }; + #[cfg(not(target_family = "wasm"))] + let at = web_async::time::Instant::from_std(at); + let sleep = timer.get_or_insert_with(|| Box::pin(web_async::time::sleep_until(at))); + if sleep.deadline() != at { + sleep.as_mut().reset(at); + } + if waiter.poll_future(sleep.as_mut()).is_pending() { + return Poll::Pending; } - }) + } }) .await } diff --git a/rs/moq-net/src/util.rs b/rs/moq-net/src/util.rs index 86eadb7de5b..c1eee584891 100644 --- a/rs/moq-net/src/util.rs +++ b/rs/moq-net/src/util.rs @@ -258,25 +258,6 @@ impl Drop for TaskSet { } } -/// Spend all but `left` units of the current [`kio::coop::budget`] on reads that decide -/// nothing, so the next poll runs out wherever its `left + 1`th unit falls. -#[cfg(test)] -pub(crate) fn leave_budget(left: usize) { - let channel = kio::Producer::new(()); - let reader = channel.consume(); - // Not `noop`, which the budget exempts. - let waiter = kio::Waiter::new(std::task::Waker::noop().clone()); - let mut units = 0usize; - kio::coop::budget(|| { - while reader.poll(&waiter, |_| Poll::Ready(())).is_ready() { - units += 1; - } - }); - for _ in 0..units.saturating_sub(left) { - let _ = reader.poll(&waiter, |_| Poll::Ready(())); - } -} - #[cfg(test)] mod tests { use std::sync::{ diff --git a/rs/moq-uring/src/worker.rs b/rs/moq-uring/src/worker.rs index d095d6e6f93..8573b50a4f5 100644 --- a/rs/moq-uring/src/worker.rs +++ b/rs/moq-uring/src/worker.rs @@ -177,8 +177,7 @@ impl Worker { let cx = Context::from_waker(&waker); let waiter = self.park.hold(&cx); - // The root gets a turn like any task; `Tasks` budgets the spawned ones. - if let Poll::Ready(value) = kio::coop::budget(|| waiter.poll_future(future.as_mut())) { + if let Poll::Ready(value) = waiter.poll_future(future.as_mut()) { return Ok(value); } // `Ready` just means the set is drained; the waiter stays @@ -612,19 +611,16 @@ impl Handle { /// with its terminal error. pub async fn run(&self, mut driver: D) -> moq_net::Error { let mut timer = self.timer(); - // One cooperative budget per poll, outside the loop, like `moq_net::time::run`. kio::wait(|waiter| { - kio::coop::budget(|| { - loop { - match driver.poll(Instant::now(), waiter) { - Ok(at) => timer.set(at), - Err(err) => return Poll::Ready(err), - } - if timer.poll(waiter).is_pending() { - return Poll::Pending; - } + loop { + match driver.poll(Instant::now(), waiter) { + Ok(at) => timer.set(at), + Err(err) => return Poll::Ready(err), } - }) + if timer.poll(waiter).is_pending() { + return Poll::Pending; + } + } }) .await } From 4fc106b00d1f37c89a18c72176b0a9ea0ff80d9f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 9 Oct 2026 10:53:11 -0700 Subject: [PATCH 8/8] quest: list the ietf FETCH write loop under spinning loops Co-Authored-By: Claude Opus 5.5 --- quest/m1/spinning-loops.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/quest/m1/spinning-loops.md b/quest/m1/spinning-loops.md index 3ca96449577..a6ce372f850 100644 --- a/quest/m1/spinning-loops.md +++ b/quest/m1/spinning-loops.md @@ -18,6 +18,10 @@ Decided 2026-10-09 while reworking #5088: finding work in one poll (receive, accept, announce, and request loops in `lite/subscriber.rs`, `ietf/subscriber.rs`, `ietf/session.rs`, and `model/origin.rs`). This guard decides which of them actually spin. +- Also unbudgeted: ietf `run_fetch_stream`'s write loop over the frames + `read_fetch` collected, which sends a whole cached group in one poll over + always-ready writes. Add an end-to-end FETCH test that checks a sibling + still gets its turns (found by the OpenAI review of #5088). - The count is test-only, so kio's hot path stays untouched: a feature or `cfg` that only the sim runtime enables, reset on each task poll. - A loop takes a `Budget` only where its `Pending` goes straight up to the