diff --git a/quest/m0/README.md b/quest/m0/README.md index 123766472d8..d4988f624ac 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -42,6 +42,11 @@ gates the next release (decided 2026-10-03: viewers). #4741 can merge to main, but no release ships until first-party publishers mint epochs. +Liveness: a serve loop with work always ready never yields, which starved +an FFI publisher's QUIC driver and fails hosted Interop's go lanes (found +2026-10-08 landing #4225). The [serve budget](/quest/m0/serve-budget.md) +bounds every kio task's loop. + Audio playout: the jitter target is default in both languages (#4162); the [line](/quest/m1/audio-jitter-target/README.md) moved to m1 in the 2026-10-08 audit, since only a manual browser proof and a native trace replay @@ -49,6 +54,7 @@ remain and no release waits on them. ## Required +- [Serve budget](/quest/m0/serve-budget.md) - a kio task that always has work ready yields after a budget, so a fast publisher can't starve its own QUIC driver - [Draft-22 LOCATION_FILTER](/quest/m0/ietf-location-filter-22.md) - moqt-22 LOCATION_FILTER carries its type instead of a Length in Rust and JS, so a draft-22 peer reads our Next Object correctly - [Capped stream END_OF_GROUP](/quest/m0/ietf-end-of-track-location.md) - a stream capped by the subscription's end Location never claims END_OF_GROUP; moving End of Track's Location is deferred - [End of Group status](/quest/m0/ietf-end-of-group-status.md) - an End of Group status on a stream whose header already marks the group's end is accepted, so imquic's last object per group arrives diff --git a/quest/m0/serve-budget.md b/quest/m0/serve-budget.md new file mode 100644 index 00000000000..97365a7c1b3 --- /dev/null +++ b/quest/m0/serve-budget.md @@ -0,0 +1,98 @@ +# [M] A busy task yields within a budget + +## Goal + +No kio task can hold its thread with an unbounded loop of async kio +readiness polls while work keeps arriving. Synchronous `try_*` and peek +loops, and code outside a `kio::coop::budget`, stay unbounded, as today. A serve loop that always has another group ready yields after a +fixed budget of progress and resumes on its next poll, so the QUIC driver, +timers, and the session's other tasks run in between, on every runtime +(tokio multi-thread and current-thread, wasm, and the io_uring worker). + +Found while landing #4225 (2026-10-08): the go interop publisher writes +2.5 ms Opus frames, one group each (about 400 groups/s). When a relay +subscribes, `RequestServe::poll_serve` (`rs/moq-net/src/lite/publisher.rs`) +keeps returning `Continue` and one poll ran over 4,096 iterations. moq-ffi's +current-thread runtime then starves noq's connection driver until the relay +times the publisher out after 10 s. Hosted Interop's `go -> *` lanes fail on +#4225 and stall for about 10 s on `main`. The gate is the mocked serve test +below, not those lanes: the m1 quests each hide the stall on their own, and +the #4225 root-cause comment saw a 64-iteration yield still time the +publisher out, so a yield alone may not be the whole fix. Re-check the lanes +and #4225 when this lands. + +Not here: moq-net spawning tasks (it stays poll-only by design, #2302, +#2736, #3825), the FFI runtime flavor, or the audio grouping default; those +are separate quests below. + +## Plan + +Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that +added this quest): + +- The bound is generic, modeled on tokio's cooperative budget, and lives in + kio, which takes no runtime dependency. Tokio's budget cannot see this loop, + because the work goes through kio primitives, never a tokio resource. + Rejected: per-loop budgets in the two serve loops (not generic), and + bridging kio to `tokio::task::coop` (leaves wasm and io_uring unbounded). +- Mechanics: + - a per-thread budget in kio. An async kio readiness poll that makes + progress spends one unit, and refunds it if the operation ends `Pending` + anyway; + - an exhausted budget wakes the current task + (`waiter.waker().wake_by_ref()`) and returns `Pending`; + - synchronous `try_*` and peek APIs, and any poll with + `kio::Waiter::noop()`, spend nothing, since their callers read "nothing + now" as empty, not as "come back". `Waiter` carries an explicit noop + flag for this, since comparing wakers is not reliable. +- A budget `Pending` must only postpone, never change a decision or lose + state. Spend only at state-safe boundaries, and audit every caller that + reads `Pending` as a value or commits state before a later poll. Known + sites: `TrackRun::start` reads Largest for SUBSCRIBE_START with + `Pending => None`; `poll_recv_next` probes `poll_finished` after + `poll_recv_group` ends, and a `Pending` there skips SUBSCRIBE_END; + `GroupServe` reads `poll_expired` as a bool; and `Cursor::poll_recv_group` + advances `index` or takes a parked group before `poll_stale`, so a + `Pending` there drops the group. Tests drive each site to exhaustion and + check every eligible group is delivered exactly once, fresh and parked. +- The budget is per kio task: `Tasks::poll` refills it for each child it + polls, and `moq_net::time::run` refills it once per `kio::wait` poll, + outside its loop, so an always-ready timer can't refill per iteration. A child that runs out lands in the next pass by + `Tasks`' one-snapshot rule (`a_self_waking_task_yields_to_the_owner`), and + the owner returns to its runtime after the pass. Work per turn scales with + the children that were ready, so a busy fan-out session is not throttled. + If [Run to quiescence](/quest/m1/perf/uring-quiescence.md) lets a turn run + several passes, each pass refills every child it polls, so a turn is + bounded by passes times ready children times the budget, and the owner + returns only after the last pass. Owners that poll their `Tasks` more than + once per poll (`Publisher::poll`, `SubscribeServe::poll_step` inside + `RequestServe::poll_serve`'s loop) compound the same way: a nested serve + can spend about the budget squared in one runtime poll. That is bounded, + and the bench decides whether it matters. + Rejected: one budget shared by the whole runtime task (tokio's model), + which caps a session with many subscriptions at one budget per turn. +- Refill mirrors tokio: `kio::coop::budget(f)` sets a fresh budget and + restores the previous value on exit, so nesting refills. Public, documented + as called only where a driver hands a task its turn. Code outside any + `budget` call is unconstrained, as today. The moq-uring worker's pass calls + it too if it polls anything outside `Tasks`. +- The size starts at 128 (tokio's) and is a constant, not configurable. A + bench picks it: sweep unbounded, 32, 128, and 512 against `main` and keep + the smallest whose throughput matches unbounded. +- Tests, with mocked time: a producer appends 10,000 groups during a serve + with many groups in flight, the serve poll returns `Pending` while a + sibling task and the transport make progress (fails on `main`), and the + units spent per runtime poll are counted against the bound; kio unit tests + for spend, refund, exhaustion, and nested refill. +- Bench: extend `rs/moq-net/benches/session.rs` swept over groups/s and + subscriptions per session, and kio's `channel` and `tasks` benches for the per-operation cost + of the thread-local. + +Public API: `kio::coop::budget` (new). Wire: none. + +## Related + +- [FFI runtime](/quest/m1/ffi-runtime.md) - FFI apps run on more than one thread +- [Audio group duration](/quest/m1/audio-group-duration.md) - small audio frames stop minting a group each +- [Browser interop cells](/quest/m1/interop-browser-timeouts.md) - the `go -> js` and `python -> js` timeouts, likely the same stall +- [Run to quiescence](/quest/m1/perf/uring-quiescence.md) - its pass count bounds passes per turn; this budget bounds one task's loop, so they compose diff --git a/quest/m1/README.md b/quest/m1/README.md index 6c7da96751a..78f78f9df44 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -67,6 +67,8 @@ were done or not worth their cost. - [Media audio-tone check](/quest/m1/media-audio-tone.md) - the media lane's audio-tone check passes under load, fixed at its cause - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause - [moq-uring tests under load](/quest/m1/uring-tests-under-load.md) - uring tests pass while parallel checks share locked memory +- [FFI runtime](/quest/m1/ffi-runtime.md) - moq-ffi and moq-c drive moq on a multi-thread runtime instead of one thread +- [Audio group duration](/quest/m1/audio-group-duration.md) - audio groups span at least 20 ms by default, so small frames don't mint a group each - [Browser interop cells](/quest/m1/interop-browser-timeouts.md) - `go -> js` and `python -> js` pass, fixed at the cause rather than by a longer timeout - [Wire compatibility](/quest/m1/wire-compat.md) - a nightly run tests this checkout against the last published release for tokens, session wire, and catalog/container - [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives diff --git a/quest/m1/audio-group-duration.md b/quest/m1/audio-group-duration.md new file mode 100644 index 00000000000..2cb9203a659 --- /dev/null +++ b/quest/m1/audio-group-duration.md @@ -0,0 +1,36 @@ +# [S] Audio groups span at least 20 ms by default + +## Goal + +An audio publisher that never sets a group duration emits at most about 50 +groups per second, whatever its codec frame size. Today every producer +defaults `group_duration` to zero, a group per packet, so the go interop +client's 2.5 ms Opus frames mint about 400 groups/s, each a QUIC stream and a +serve. + +## Plan + +Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that +added this quest): + +- `moq_audio::encode::Options::group_duration` defaults to 20 ms + (`rs/moq-audio/src/encode/producer.rs`, documented today as "Defaults to + zero, a group per packet"). 20 ms frames behave as today; smaller frames + share a group up to 20 ms, so the 10 ms low-latency Opus preset now pairs + packets. +- Every producer follows it: moq-ffi `encode_audio`, moq-c, moq-boy, and the + CLI through `Options::default()`; JS publish's `groupDuration` + (`js/publish/src/audio/encoder.ts`); and the GStreamer sink, which today + cuts after every audio packet (`rs/moq-gst/src/sink/pad.rs`). Rejected: + only the FFI sets it, and keeping zero. +- Docs: `doc/lib/rs/moq-audio.md` and the JS publish docs drop "a group per + packet" as the default. +- Test: a 2.5 ms Opus encode yields groups of at least 20 ms in Rust and JS, + and the gst sink's grouping test. + +Public API: a changed default in moq-audio and `@moq/publish`. Wire: none. + +## Related + +- [Serve budget](/quest/m0/serve-budget.md) - bounds the serve loop whatever the group rate +- [FFI codec namespaces](/quest/m1/ffi-shape/codec.md) - owns the FFI encoder's frame duration default, a separate knob diff --git a/quest/m1/ffi-runtime.md b/quest/m1/ffi-runtime.md new file mode 100644 index 00000000000..368e49b467b --- /dev/null +++ b/quest/m1/ffi-runtime.md @@ -0,0 +1,52 @@ +# [S] FFI apps run on more than one thread + +## Goal + +moq-ffi and moq-c drive moq on a multi-thread tokio runtime instead of one +`current_thread` runtime thread, so noq's endpoint and connection drivers +run beside a busy session, and Go, Python, Swift, Kotlin, and Dart apps get +the parallelism their own runtimes already have. + +## Plan + +Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that +added this quest): + +- Today `runtime()` in `rs/moq-ffi/src/ffi.rs` builds + `new_current_thread().enable_all()` on one dedicated `moq-ffi` thread, and + `rs/moq-c/src/ffi.rs` does the same (`moq-c`). Everything shares it: noq's + endpoint and connection drivers, the moq-tokio reconnect loop, the session + driver, and the origin driver. The pattern was copied from libmoq (#732, + #1071) with no stated reason. +- Use `new_multi_thread().enable_all()` with tokio's default worker count. + No configuration knob until an embedder needs one. Rejected: a small fixed + pool, and keeping `current_thread` behind the + [serve budget](/quest/m0/serve-budget.md) alone. +- Keep the "foreign caller's thread never drives our futures" property: work + still runs via `Task::run` and `detached`. Check the exported async fns that + await in place (`subscribe_catalog`, `subscribe_track`, `fetch_group`, + `request_broadcast`, `decode_audio`, `decode_video`) and the sync methods + that enter the runtime context (`MoqAudioProducer::write`). +- Callbacks change contract. Today moq-c status callbacks and uniffi + callbacks fire one at a time on the single runtime thread; on workers they + fire on any thread, and callbacks for different handles run at once. + Document that in `doc/lib/` and audit binding code that assumes serialized + callbacks. +- Shutdown keeps its meaning: when `moq_ffi_shutdown` returns, + no task runs and no callback into the host is in flight, so Python's + `atexit` and JVM exit stay clean. `shutdown_background()` no longer gives + that with N workers; the shutdown must wait for them. moq-c exports no + shutdown and is being [retired](/quest/m1/c/retire.md), so it gets none. +- Watch [Kotlin/JVM exit](/quest/m2/kt-jvm-exit.md), which depends on how the + runtime thread stops. +- Test: the interop go and python publishers on a pinned single CPU and on + many, and every binding's test suite. Benchmark a binding publisher's + throughput before and after. + +Public API: callbacks may fire concurrently on any runtime thread +(documented); no signature changes. Wire: none. + +## Related + +- [Serve budget](/quest/m0/serve-budget.md) - the root fix for a task hogging its thread +- [Kotlin/JVM exit](/quest/m2/kt-jvm-exit.md) - runtime-thread shutdown diff --git a/quest/m1/interop-browser-timeouts.md b/quest/m1/interop-browser-timeouts.md index 6ae1156bb17..5b7c4aa26d6 100644 --- a/quest/m1/interop-browser-timeouts.md +++ b/quest/m1/interop-browser-timeouts.md @@ -21,5 +21,12 @@ FFI publisher's 2 s audio max age parked hundreds of group serves. - Find which step stalls and why, from the player trace and the publisher side. Check whether #5005 alone fixed it before changing anything else. - Never raise the timeout or add a retry. +- Found 2026-10-08 while landing #4225: an FFI publisher's serve loop can + starve moq-ffi's single runtime thread until the relay times it out. + Re-check these cells once [Serve budget](/quest/m0/serve-budget.md) lands. Public API: none expected. Wire: none. + +## Related + +- [Serve budget](/quest/m0/serve-budget.md) - likely the same stall diff --git a/quest/m1/perf/uring-quiescence.md b/quest/m1/perf/uring-quiescence.md index 42254f0849b..ee9c09ce2c1 100644 --- a/quest/m1/perf/uring-quiescence.md +++ b/quest/m1/perf/uring-quiescence.md @@ -52,3 +52,7 @@ meaning with a budget of 1 and gains a sibling proving the budget bound. - [Hard fork](/quest/m1/quic/fork/README.md) - the noq driver this edits moves onto `moq-quic` (decided in the 2026-10-05 audit) - [One enter per turn](/quest/m1/perf/uring-one-enter.md) - the metrics and the submit placement this sweep is measured with + +## Related + +- [Serve budget](/quest/m0/serve-budget.md) - a per-task budget bounds one task's loop; this quest's pass count bounds passes per turn, and the sweep runs with both in place