Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions quest/m0/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,19 @@ gates the next release (decided 2026-10-03:
viewers). #4741 can merge to main, but no release ships until first-party
publishers mint epochs.

Liveness: a serve loop with work always ready never yields, which starved
an FFI publisher's QUIC driver and fails hosted Interop's go lanes (found
2026-10-08 landing #4225). The [serve budget](/quest/m0/serve-budget.md)
bounds every kio task's loop.

Audio playout: the jitter target is default in both languages (#4162); the
[line](/quest/m1/audio-jitter-target/README.md) moved to m1 in the
2026-10-08 audit, since only a manual browser proof and a native trace replay
remain and no release waits on them.

## Required

- [Serve budget](/quest/m0/serve-budget.md) - a kio task that always has work ready yields after a budget, so a fast publisher can't starve its own QUIC driver
- [Draft-22 LOCATION_FILTER](/quest/m0/ietf-location-filter-22.md) - moqt-22 LOCATION_FILTER carries its type instead of a Length in Rust and JS, so a draft-22 peer reads our Next Object correctly
- [Capped stream END_OF_GROUP](/quest/m0/ietf-end-of-track-location.md) - a stream capped by the subscription's end Location never claims END_OF_GROUP; moving End of Track's Location is deferred
- [End of Group status](/quest/m0/ietf-end-of-group-status.md) - an End of Group status on a stream whose header already marks the group's end is accepted, so imquic's last object per group arrives
Expand Down
98 changes: 98 additions & 0 deletions quest/m0/serve-budget.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# [M] A busy task yields within a budget

## Goal

No kio task can hold its thread with an unbounded loop of async kio
readiness polls while work keeps arriving. Synchronous `try_*` and peek
loops, and code outside a `kio::coop::budget`, stay unbounded, as today. A serve loop that always has another group ready yields after a
fixed budget of progress and resumes on its next poll, so the QUIC driver,
timers, and the session's other tasks run in between, on every runtime
(tokio multi-thread and current-thread, wasm, and the io_uring worker).

Found while landing #4225 (2026-10-08): the go interop publisher writes
2.5 ms Opus frames, one group each (about 400 groups/s). When a relay
subscribes, `RequestServe::poll_serve` (`rs/moq-net/src/lite/publisher.rs`)
keeps returning `Continue` and one poll ran over 4,096 iterations. moq-ffi's
current-thread runtime then starves noq's connection driver until the relay
times the publisher out after 10 s. Hosted Interop's `go -> *` lanes fail on
#4225 and stall for about 10 s on `main`. The gate is the mocked serve test
below, not those lanes: the m1 quests each hide the stall on their own, and
the #4225 root-cause comment saw a 64-iteration yield still time the
publisher out, so a yield alone may not be the whole fix. Re-check the lanes
and #4225 when this lands.

Not here: moq-net spawning tasks (it stays poll-only by design, #2302,
#2736, #3825), the FFI runtime flavor, or the audio grouping default; those
are separate quests below.

## Plan

Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that
added this quest):

- The bound is generic, modeled on tokio's cooperative budget, and lives in
kio, which takes no runtime dependency. Tokio's budget cannot see this loop,
because the work goes through kio primitives, never a tokio resource.
Rejected: per-loop budgets in the two serve loops (not generic), and
bridging kio to `tokio::task::coop` (leaves wasm and io_uring unbounded).
- Mechanics:
- a per-thread budget in kio. An async kio readiness poll that makes
progress spends one unit, and refunds it if the operation ends `Pending`
anyway;
- an exhausted budget wakes the current task
(`waiter.waker().wake_by_ref()`) and returns `Pending`;
- synchronous `try_*` and peek APIs, and any poll with
`kio::Waiter::noop()`, spend nothing, since their callers read "nothing
now" as empty, not as "come back". `Waiter` carries an explicit noop
flag for this, since comparing wakers is not reliable.
- A budget `Pending` must only postpone, never change a decision or lose
state. Spend only at state-safe boundaries, and audit every caller that
reads `Pending` as a value or commits state before a later poll. Known
sites: `TrackRun::start` reads Largest for SUBSCRIBE_START with
`Pending => None`; `poll_recv_next` probes `poll_finished` after
`poll_recv_group` ends, and a `Pending` there skips SUBSCRIBE_END;
`GroupServe` reads `poll_expired` as a bool; and `Cursor::poll_recv_group`
advances `index` or takes a parked group before `poll_stale`, so a
`Pending` there drops the group. Tests drive each site to exhaustion and
check every eligible group is delivered exactly once, fresh and parked.
- The budget is per kio task: `Tasks::poll` refills it for each child it
polls, and `moq_net::time::run` refills it once per `kio::wait` poll,
outside its loop, so an always-ready timer can't refill per iteration. A child that runs out lands in the next pass by
`Tasks`' one-snapshot rule (`a_self_waking_task_yields_to_the_owner`), and
the owner returns to its runtime after the pass. Work per turn scales with
the children that were ready, so a busy fan-out session is not throttled.
If [Run to quiescence](/quest/m1/perf/uring-quiescence.md) lets a turn run
several passes, each pass refills every child it polls, so a turn is
bounded by passes times ready children times the budget, and the owner
returns only after the last pass. Owners that poll their `Tasks` more than
once per poll (`Publisher::poll`, `SubscribeServe::poll_step` inside
`RequestServe::poll_serve`'s loop) compound the same way: a nested serve
can spend about the budget squared in one runtime poll. That is bounded,
and the bench decides whether it matters.
Rejected: one budget shared by the whole runtime task (tokio's model),
which caps a session with many subscriptions at one budget per turn.
- Refill mirrors tokio: `kio::coop::budget(f)` sets a fresh budget and
restores the previous value on exit, so nesting refills. Public, documented
as called only where a driver hands a task its turn. Code outside any
`budget` call is unconstrained, as today. The moq-uring worker's pass calls
it too if it polls anything outside `Tasks`.
- The size starts at 128 (tokio's) and is a constant, not configurable. A
bench picks it: sweep unbounded, 32, 128, and 512 against `main` and keep
the smallest whose throughput matches unbounded.
- Tests, with mocked time: a producer appends 10,000 groups during a serve
with many groups in flight, the serve poll returns `Pending` while a
sibling task and the transport make progress (fails on `main`), and the
units spent per runtime poll are counted against the bound; kio unit tests
for spend, refund, exhaustion, and nested refill.
- Bench: extend `rs/moq-net/benches/session.rs` swept over groups/s and
subscriptions per session, and kio's `channel` and `tasks` benches for the per-operation cost
of the thread-local.

Public API: `kio::coop::budget` (new). Wire: none.

## Related

- [FFI runtime](/quest/m1/ffi-runtime.md) - FFI apps run on more than one thread
- [Audio group duration](/quest/m1/audio-group-duration.md) - small audio frames stop minting a group each
- [Browser interop cells](/quest/m1/interop-browser-timeouts.md) - the `go -> js` and `python -> js` timeouts, likely the same stall
- [Run to quiescence](/quest/m1/perf/uring-quiescence.md) - its pass count bounds passes per turn; this budget bounds one task's loop, so they compose
2 changes: 2 additions & 0 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ were done or not worth their cost.
- [Media audio-tone check](/quest/m1/media-audio-tone.md) - the media lane's audio-tone check passes under load, fixed at its cause
- [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause
- [moq-uring tests under load](/quest/m1/uring-tests-under-load.md) - uring tests pass while parallel checks share locked memory
- [FFI runtime](/quest/m1/ffi-runtime.md) - moq-ffi and moq-c drive moq on a multi-thread runtime instead of one thread
- [Audio group duration](/quest/m1/audio-group-duration.md) - audio groups span at least 20 ms by default, so small frames don't mint a group each
- [Browser interop cells](/quest/m1/interop-browser-timeouts.md) - `go -> js` and `python -> js` pass, fixed at the cause rather than by a longer timeout
- [Wire compatibility](/quest/m1/wire-compat.md) - a nightly run tests this checkout against the last published release for tokens, session wire, and catalog/container
- [Same-epoch importers](/quest/m1/hop-aligned-import.md) - importers sharing one `--epoch` and fed one stream publish identical groups and timestamps, so failover between a redundant pair survives
Expand Down
36 changes: 36 additions & 0 deletions quest/m1/audio-group-duration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# [S] Audio groups span at least 20 ms by default

## Goal

An audio publisher that never sets a group duration emits at most about 50
groups per second, whatever its codec frame size. Today every producer
defaults `group_duration` to zero, a group per packet, so the go interop
client's 2.5 ms Opus frames mint about 400 groups/s, each a QUIC stream and a
serve.

## Plan

Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that
added this quest):

- `moq_audio::encode::Options::group_duration` defaults to 20 ms
(`rs/moq-audio/src/encode/producer.rs`, documented today as "Defaults to
zero, a group per packet"). 20 ms frames behave as today; smaller frames
share a group up to 20 ms, so the 10 ms low-latency Opus preset now pairs
packets.
- Every producer follows it: moq-ffi `encode_audio`, moq-c, moq-boy, and the
CLI through `Options::default()`; JS publish's `groupDuration`
(`js/publish/src/audio/encoder.ts`); and the GStreamer sink, which today
cuts after every audio packet (`rs/moq-gst/src/sink/pad.rs`). Rejected:
only the FFI sets it, and keeping zero.
- Docs: `doc/lib/rs/moq-audio.md` and the JS publish docs drop "a group per
packet" as the default.
- Test: a 2.5 ms Opus encode yields groups of at least 20 ms in Rust and JS,
and the gst sink's grouping test.

Public API: a changed default in moq-audio and `@moq/publish`. Wire: none.

## Related

- [Serve budget](/quest/m0/serve-budget.md) - bounds the serve loop whatever the group rate
- [FFI codec namespaces](/quest/m1/ffi-shape/codec.md) - owns the FFI encoder's frame duration default, a separate knob
52 changes: 52 additions & 0 deletions quest/m1/ffi-runtime.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# [S] FFI apps run on more than one thread

## Goal

moq-ffi and moq-c drive moq on a multi-thread tokio runtime instead of one
`current_thread` runtime thread, so noq's endpoint and connection drivers
run beside a busy session, and Go, Python, Swift, Kotlin, and Dart apps get
the parallelism their own runtimes already have.

## Plan

Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that
added this quest):

- Today `runtime()` in `rs/moq-ffi/src/ffi.rs` builds
`new_current_thread().enable_all()` on one dedicated `moq-ffi` thread, and
`rs/moq-c/src/ffi.rs` does the same (`moq-c`). Everything shares it: noq's
endpoint and connection drivers, the moq-tokio reconnect loop, the session
driver, and the origin driver. The pattern was copied from libmoq (#732,
#1071) with no stated reason.
- Use `new_multi_thread().enable_all()` with tokio's default worker count.
No configuration knob until an embedder needs one. Rejected: a small fixed
pool, and keeping `current_thread` behind the
[serve budget](/quest/m0/serve-budget.md) alone.
- Keep the "foreign caller's thread never drives our futures" property: work
still runs via `Task::run` and `detached`. Check the exported async fns that
await in place (`subscribe_catalog`, `subscribe_track`, `fetch_group`,
`request_broadcast`, `decode_audio`, `decode_video`) and the sync methods
that enter the runtime context (`MoqAudioProducer::write`).
- Callbacks change contract. Today moq-c status callbacks and uniffi
callbacks fire one at a time on the single runtime thread; on workers they
fire on any thread, and callbacks for different handles run at once.
Document that in `doc/lib/` and audit binding code that assumes serialized
callbacks.
- Shutdown keeps its meaning: when `moq_ffi_shutdown` returns,
no task runs and no callback into the host is in flight, so Python's
`atexit` and JVM exit stay clean. `shutdown_background()` no longer gives
that with N workers; the shutdown must wait for them. moq-c exports no
shutdown and is being [retired](/quest/m1/c/retire.md), so it gets none.
- Watch [Kotlin/JVM exit](/quest/m2/kt-jvm-exit.md), which depends on how the
runtime thread stops.
- Test: the interop go and python publishers on a pinned single CPU and on
many, and every binding's test suite. Benchmark a binding publisher's
throughput before and after.

Public API: callbacks may fire concurrently on any runtime thread
(documented); no signature changes. Wire: none.

## Related

- [Serve budget](/quest/m0/serve-budget.md) - the root fix for a task hogging its thread
- [Kotlin/JVM exit](/quest/m2/kt-jvm-exit.md) - runtime-thread shutdown
7 changes: 7 additions & 0 deletions quest/m1/interop-browser-timeouts.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,12 @@ FFI publisher's 2 s audio max age parked hundreds of group serves.
- Find which step stalls and why, from the player trace and the publisher
side. Check whether #5005 alone fixed it before changing anything else.
- Never raise the timeout or add a retry.
- Found 2026-10-08 while landing #4225: an FFI publisher's serve loop can
starve moq-ffi's single runtime thread until the relay times it out.
Re-check these cells once [Serve budget](/quest/m0/serve-budget.md) lands.

Public API: none expected. Wire: none.

## Related

- [Serve budget](/quest/m0/serve-budget.md) - likely the same stall
4 changes: 4 additions & 0 deletions quest/m1/perf/uring-quiescence.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,7 @@ meaning with a budget of 1 and gains a sibling proving the budget bound.
- [Hard fork](/quest/m1/quic/fork/README.md) - the noq driver this edits moves onto `moq-quic` (decided in the 2026-10-05 audit)
- [One enter per turn](/quest/m1/perf/uring-one-enter.md) - the metrics and
the submit placement this sweep is measured with

## Related

- [Serve budget](/quest/m0/serve-budget.md) - a per-task budget bounds one task's loop; this quest's pass count bounds passes per turn, and the sweep runs with both in place
Loading