diff --git a/bench/ffi-gen/README.md b/bench/ffi-gen/README.md index 71d33e1f741..33bc31b8b65 100644 --- a/bench/ffi-gen/README.md +++ b/bench/ffi-gen/README.md @@ -217,7 +217,7 @@ and do not translate to BoltFFI's IR. | Call overhead | M: JNI or JNA direct mapping for Kotlin ([Kotlin calls](/quest/m2/kt-ffi-calls.md)); L: a C-extension Python backend, not yet planned | done | | Zero-copy byte returns | M: [Zero-copy byte returns](/quest/m2/ffi-bytes-return.md) | M: bytes cross its wire buffer, with no borrowed return | | Namespaced types | M: [Namespaced external types](/quest/m2/uniffi-namespaces.md) | M: same-name types collide (RFC #700, open) | -| Poll-native async | M: [Poll-native async](/quest/m2/ffi-poll.md) | M: the same poll-plus-continuation model, with per-method poll exports | +| Poll-native async | M: [Poll-native async](/quest/m2/ffi-poll-migrate.md) | M: the same poll-plus-continuation model, with per-method poll exports | | Binding moq-ffi as written | none | M: drop `Arc` returns, rework wasm gating, and fix the gaps above | Either way the fork owns a large codebase. BoltFFI's is about 134k lines, written mostly diff --git a/quest/README.md b/quest/README.md index 3ea2a59d5e3..bddb862eb1b 100644 --- a/quest/README.md +++ b/quest/README.md @@ -7,9 +7,10 @@ grouped into milestones ordered by priority. ## Plan -m0 holds urgent fixes and release-line maintenance. m1 is the finite next -breaking release: clustering stability first, the in-tree QUIC stack, and -planned public API and format breaks with their correctness and validation. +m0 holds urgent fixes and release-line maintenance. m1 is everything that +lands before the next cut of `main` into `release`, breaking or not: +clustering stability first, the in-tree QUIC stack, and planned public API and +format breaks with their correctness and validation. m2 holds independently shippable capabilities and optimizations, plus the publication and adoption steps that follow m1. m3 holds external conditions and speculative work. A milestone is a priority horizon; m0 maintenance does @@ -18,8 +19,9 @@ not make unrelated infrastructure a gate for the main-to-release cut. Milestones shift up after each major release (decided 2026-10-19): m0 is the current release line, m1 the next major, m2 the one after, and m3 stays the deferred bucket. The [release cut](/quest/m2/release/cut.md) recurs and -performs the shift, and the hygiene audits at the end of m0 recur once per -release behind it. +performs the shift. The hygiene audits at the end of each milestone recur +once per major release: each run adds its successor to the end of the next +milestone. The [release execution](/quest/m2/release/README.md) starts after m1 is ready. It publishes artifacts before downstream adoption and rollout can complete. @@ -28,6 +30,6 @@ Critical fixes continue to reach the existing release through backports. ## Required - [m0: immediate priorities](/quest/m0/README.md) - urgent correctness fixes and maintenance of the current release -- [m1: next wave](/quest/m1/README.md) - the agreed next breaking cut, with its required correctness and validation +- [m1: next wave](/quest/m1/README.md) - everything before the next release cut, breaking or not, with its required correctness and validation - [m2: later work](/quest/m2/README.md) - release execution first, then deferred features, optimization, and design studies - [m3: deferred](/quest/m3/README.md) - gated on the outside world (hardware, a partner, a consumer, a provider's offer, or an upstream release), or speculative with no named consumer diff --git a/quest/m0/README.md b/quest/m0/README.md index 3a1bde6e40f..6a2c86043a4 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -23,17 +23,6 @@ serve instead of enumerating broadcasts. Serving the relay's ingested-only view (`origin::Consumer::local()`) to localhost workers belongs to moq.pro's edge, which embeds moq-relay; it moved there on 2026-09-28. -Interop: Fastly's moq-relay-interop report (run of 2026-09-23, build -7ee2b02) was triaged against `main` on 2026-10-07. Its SETUP, UNSUBSCRIBE -and error-code items were already fixed. Fastly's reruns that day found -that a relay moves End of Track's Location and refuses imquic's End of -Group status. The fixes below LOCATION_FILTER come from them and go ahead -of Seattle. The cold-relay Largest stays compliant with INVALID_RANGE -(decided 2026-10-08), and the deviations doc landed in #5022. -An imquic draft-22 rig (2026-10-08) hit LOCATION_FILTER and a clear -FIRST_OBJECT at object 0 (#5027). Both shipped with moq-noq 1.3.5 in -moq-relay 0.17.3 and `@moq/net` 0.4.3. - Identity: the [broadcast epoch](/quest/m0/broadcast-epoch/README.md) line gates the next release (decided 2026-10-03: #4741 resumes an un-epoched republish into the old broadcast and stalls its @@ -41,40 +30,6 @@ viewers). #4741 can merge to main, but no release ships until first-party publishers mint epochs. Backport patches cut from `release` don't carry #4741 and aren't gated (decided 2026-10-08). -Liveness: a serve loop with work always ready never yields, which starved -an FFI publisher's QUIC driver and fails hosted Interop's go lanes (found -2026-10-08 landing #4225). The publish serve loops now yield through a -`kio::coop::Budget`. The Go and Python interop cells it fails -moved here from m1 the same day, because the stall masks interop on every -wire PR and hides as slow passes; the harness now fails a cell whose -connection idles out. - -Audio playout: the jitter target is default in both languages (#4162); the -[line](/quest/m1/audio-jitter-target/README.md) moved to m1 in the -2026-10-08 audit, since only a manual browser proof and a native trace replay -remain and the release only requires the production clock and playback contracts -that depend on them. - -Interop on main (2026-10-13): the cross-client matrix failed with `unroutable` -whenever WebSocket won the connect race: since #5294, #4180's upgrade -retired epochless handles before its held `Restart` reached anyone. Handles -now retire only once it is deliverable. First-party consumers following it -moved to m1 (maintainer, 2026-10-13), since the upgrade is removed. - -Crashes (2026-10-13, #5407): a publisher on a host whose wall clock reads -before 2020 panics, and `panic = "abort"` turned any library panic into a -dead host app. Both joined m0 (maintainer) and are fixed: the clock at its -source, and unwinding so a panic closes only its connection. - -Nightly on main (triaged 2026-10-13): two build breaks from 2026-10-11 get -quests here. The nixpkgs bump broke the Linux zigbuild release builds (its -GCC 16 miscompiles lld 21; fixed by bumping nixpkgs past the upstream fix), -and fat LTO in the release profile broke the fuzz build. Audio quality has been -red since it landed, against budgets recorded locally; re-measuring them on -the runner moved here from Watch (maintainer, 2026-10-13). The other red jobs -have fixes on `main`: macOS tests (#5335), drill-sensitivity (#5338), loom -(#5389) and Windows tests (#5339). - Flakes (2026-10-20, maintainer): the redundant failover drift moved here from the m1 [load round](/quest/m1/test-flakes-2/README.md). It fails loaded `just check` runs, the local gate every merge waits on, and may hide a TS @@ -87,14 +42,13 @@ Each lands its fixes in its own PR and plans only what needs a decision. ## Required -- [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in the merge queue as one `just ci` pass, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last +- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription - [Redundant failover drift](/quest/m0/redundant-failover-drift.md) - a stitching TS export switches back to the surviving importer behind bufferbloat instead of failing on a stalled clock - [qmux 0.5.3](/quest/m0/qmux-0.5.3.md) - crates.io publishes qmux 0.5.3 with the builder session config the release backport of #5609 needs - [Release shutdown backport](/quest/m0/ffi-shutdown-backport.md) - `release` publishes a non-breaking moq-ffi with an async `drain()` and qmux sessions on the QUIC stream limit - [Release fixed delay](/quest/m0/release-fixed-delay.md) - a fixed latency on `release`, which moq.dev runs, ignores the catalog's jitter and delay as `main` already does - [Publisher stall estimate](/quest/m0/publish-stall-estimate.md) - a live browser publish holds its viewers at the Auto target instead of ratcheting their buffer up with every stall -- [Broadcast epochs](/quest/m0/broadcast-epoch/README.md) - every first-party publisher that can restart mints a fresh route epoch, the newest wins a path, and only routes with the same epoch resume a subscription +- [CI cost and latency](/quest/m0/ci/README.md) - cloud CI runs only in the merge queue as one `just ci` pass, expensive checks run nightly and gate `release`, caches move to R2, and runners are chosen last - [Simplification audit](/quest/m0/audit-simplify.md) - overbuilt code and APIs are collapsed, merged, or narrowed into a net simpler whole - [Lint override and deprecation audit](/quest/m0/audit-lints.md) - every lint suppression is still needed and says why, nothing calls a deprecated API, and stale Rust overrides fail the build -- [Quest tree audit](/quest/m0/audit-quests.md) - the tree agrees with itself and with main - [Interop audit](/quest/m0/audit-interop.md) - our interop runner entries match what we ship, and every red interop cell with a moq.dev side is fixed, reported, or noted as down diff --git a/quest/m0/audit-quests.md b/quest/m0/audit-quests.md deleted file mode 100644 index 556e8afcf50..00000000000 --- a/quest/m0/audit-quests.md +++ /dev/null @@ -1,79 +0,0 @@ -# [M] Quest tree audit - -## Goal - -The quest tree agrees with itself and with `main`: done, stale, -conflicting, and misranked quests are fixed. - -## Plan - -Recurs once per major release (decided 2026-10-19). The PR that finishes -this run also adds the next one with the same file name at the end of the next -milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal -and this paragraph; the rest of its Plan is rewritten with what this run -learned: baselines, results, and what to check first. A successor placed in m2 -also lists the [release cut](/quest/m2/release/cut.md) under `Required`, so it -waits for the release in flight to ship. Only `/quest-delete` ends the -recurrence. Fixes land in the run's PR, one commit each, rather than as follow-ups. -Only a change that needs a maintainer decision (public API, scope) or is -independently landable and too large to review alongside the rest becomes a -follow-up through `/quest-plan`. - -Run `/quest-audit`. After a release cut shifts the milestones, also check -that the move left no broken links or misplaced quests. Until now, audits ran -ad hoc (2026-09-30, 10-05, 10-08, 10-10). - -Checkpoint, 2026-10-19, against `main` at `2e1ec2ebf`: - -- Read all 438 documents: m0, m3, and global structure in the coordinating - audit, with parallel reviewers for all 133 m1 and 222 m2 documents. - Structural validation found no broken links, cycles, or prerequisites in - a later milestone. -- Retired datagram priority: #5770 (`79005b06d`) and #5872 (`8e3966cd3`) - already contain the scheduler, publisher integration, saturation and - eviction tests, fan-out benchmarks, and draft updates. Broadcast-domain - exposure remains owned by track-priority-scope. -- Encoded the already-decided timestamp and E2EE naming-design - prerequisites. Corrected stale paths, epoch discovery, accepted refresh - recovery behavior, mobile priority, and the landed error, shard, benchmark - site, and wire-accounting work. Cut-through now follows the bufferbloat - policy. -- Four closed linked issues still have explicitly planned or in-flight - work. Their closure alone was not treated as evidence to retire a quest. - Open implementation PRs were not treated as merged work. -- Rechecked against `main` at `48c8a66dd` (#6017). The random-loss audit - (#6000) landed its own BBR, cut-through, and loss-input verdicts, so its - versions replace this run's; the 0-RTT window raise (#5937) also finished. - Quests added since the baseline (the resume line, egress held state, and - the request-behind-its-token race from #5972) were checked only against - this run's edits, not audited in full. -- Rechecked against `main` at `9b90cdd56` (#5967) for quests left behind by - their own merged PRs. Retired moq-uring on moq-clock (#5926), moving its - unmeasured `[vdso]` profile to an m2 perf quest, IETF max age - (#5993), the request-behind-its-token race (#5992), and the moq-quic - regression net with its last child (#5989). Trimmed the TRACK_STATUS race - (#5991) to its deferred upstream report, and recorded what #6006 and #5987 - left of the C++ package proof and the moq-clock crate migration. Check - first, next run: every PR merged from a `quest/...` branch whose quest file - still exists. -- Cause of those leftovers: no quest skill tells the PR that finishes a quest - to delete its file, or a partial PR to rewrite it to the remainder, and - `/quest-merge` enables auto-merge without checking either. Drafts whose - bodies said "the quest remains open" were merged with the quest unchanged. - -This run remains open, with no successor, until the unresolved conflicts are -settled. The unattended instruction was to skip uncertain choices: - -- [Auth issuer](/quest/m1/auth/issuer.md) specifies a shared handle and one - live session's grant, but not how independent concurrent sessions select - grants and refusals. Existing reconnect code only designates one active - generation alongside draining predecessors. -- [Binding linger](/quest/m1/linger-ffi.md) extends the hand-written C ABI, - while [generated C](/quest/m1/c/README.md) freezes it except claim safety. -- [Catalog HLS](/quest/m1/archive/catalog-hls.md) names `.hls.z` tracks, - while [compression migration](/quest/m1/compress/migrate.md) removes `.z` - names. Their shared timeline edits need a consistent final name and order. -- [Priority scope](/quest/m1/track-priority-scope.md) still preserves old - archive priority encoding, while [archive](/quest/m1/archive/README.md) - replaces the old layout without migration. The ladder also assumes a - publisher-priority tiebreak that priority scope still asks to decide. diff --git a/quest/m0/broadcast-epoch/reference-announcements.md b/quest/m0/broadcast-epoch/reference-announcements.md index cfde1ceeef0..f5b82e19b2f 100644 --- a/quest/m0/broadcast-epoch/reference-announcements.md +++ b/quest/m0/broadcast-epoch/reference-announcements.md @@ -22,6 +22,9 @@ text; this quest owns the following behavior it left out. today, as `failed_binding_does_not_retry_a_later_publisher` asserts; that changes. The "no retarget" rule applies only to readers that follow no announcement. + Decided 2026-10-20 (maintainer): this quest owns `Source::follow` and + [sink restart follow](/quest/m1/sink-restart-follow.md) reuses it, so + no reader hand-rolls its own follow loop. - `moqsrc`: Start binds the rendition, End idles its pad, and Restart restarts its pump. - Exporters: a late first announcement starts the rendition and End ends @@ -46,3 +49,4 @@ epoch, and concurrent old/new consumers, in the moq-mux unit tests and the - [Sink restart follow](/quest/m1/sink-restart-follow.md) - exporters and sinks that follow an epochless Restart of their own broadcast - [Export rendition rebind](/quest/m1/export-rendition-rebind.md) - exporters rebuild a rendition whose reference changes +- [Hang contribution references](/quest/m2/contribution-ref.md) - builds its lazy contribution entries on these reference rules diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 6ec3277c1ed..8a6ba2c2d2e 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -79,6 +79,6 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Pin Quest's gate](/quest/m0/ci/quest-gate-pin.md) - make shared command admission available - [Share local check capacity](/quest/m0/ci/local-gate.md) - gate expensive phases and bound internal fanout - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - condition: the maintainer bootstrap; then measure hits, restore time, and expiry -- [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - condition: the `nix.moq.dev` PR merges; then confirm CI and releases use it and retire Cachix +- [Verify the Nix cache](/quest/m0/ci/nix-cache-verify.md) - confirm CI and releases use `nix.moq.dev` and retire Cachix - [First shared FFI release](/quest/m0/ci/nightly-ffi-reuse.md) - condition: the first `moq-ffi-v*` tag that runs `Release FFI`, and a `Shipping sizes` run; verify the wrappers and size keys - [Faster runners](/quest/m0/ci/runners.md) - pick hosted, Blacksmith or the NixOS desktop from measurements of the optimized queue diff --git a/quest/m0/ci/nix-cache-verify.md b/quest/m0/ci/nix-cache-verify.md index 60aae1f4f35..403de9054f2 100644 --- a/quest/m0/ci/nix-cache-verify.md +++ b/quest/m0/ci/nix-cache-verify.md @@ -2,8 +2,14 @@ ## Goal -Condition: the PR that wired `nix.moq.dev` into CI has merged. The maintainer -ran the bootstrap in `infra/README.md` on 2026-10-19 and 2026-10-20 (the +`nix.moq.dev` serves CI and releases: Nix jobs substitute the dev shell +instead of building it, release tags substitute on a clean machine, its +bucket sizes hold under the 30-day expiry, and Cachix is retired. + +## Plan + +[#5693](https://github.com/moq-dev/moq/pull/5693) wired `nix.moq.dev` into +CI. The maintainer ran the bootstrap in `infra/README.md` on 2026-10-19 and 2026-10-20 (the release tag ruleset, both buckets and the Worker, the `cache-write` Environment's tag policies and secrets, and the signing key). Check it with `curl https://nix.moq.dev/nix-cache-info` and diff --git a/quest/m1/README.md b/quest/m1/README.md index 6050367ca70..7047a7cf789 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -1,21 +1,24 @@ -# m1: clustering stability and the breaking release +# m1: clustering stability and the next release ## Goal -The next breaking release delivers stable clustered routing and recovery, -switches to moq-quic, and consolidates the planned public API and format -changes. Complete this milestone before cutting main into release. +Everything that lands before `main` is next cut into `release`: stable +clustered routing and recovery, the switch to moq-quic, the planned public +API and format changes, and the non-breaking fixes and checks queued ahead of +the cut. Complete this milestone before cutting main into release. ## Plan -Decided with the maintainer on 2026-10-15: keep a finite next breaking -release, with clustering stability and the in-tree QUIC stack first. +Decided with the maintainer on 2026-10-15: clustering stability and the +in-tree QUIC stack first. Decided 2026-10-20 (maintainer): m1 is everything +before the cut, not only breaking work, so ready non-breaking quests stay +here instead of moving out; the cut is what ships them. - Preserve epoch correctness, routing recovery, outbound federation, and announcement takeover on both lite and IETF. Catalogs, long-lived readers, and late joiners obey the same identity contract. - Keep IETF passthrough, the public model audit, the broad clock conversion, - QUIC endpoint sharding, archive replay/DVR correctness, and the TS fidelity + archive replay/DVR correctness, and the TS fidelity expansion in m1. Keep their correctness, interoperability, memory, and performance evidence with them. - Defer P2P, watch-worker planning and implementation, AMD Vulkan encode and @@ -37,7 +40,10 @@ Existing upgrade and identity guides carry migrations; feature docs change inline. Keep generic proofs here and deployments and their operational acceptance with the downstream operator. -Keep one owner per shared code area and order prerequisites before consumers. +Keep one owner per shared code area and order prerequisites before consumers +(re-ranked in the 2026-10-20 audit: auth before cluster routing, the C line +before the binding breaks, and every lite-07 and public model prerequisite +before those quests). The moq-transport passthrough fields remain crate-private, the existing explicit exception to the public model audit. @@ -54,37 +60,39 @@ lite-07 stream type here. ## Required - [Epoch correctness](/quest/m0/broadcast-epoch/README.md) - publisher identity, duplicate suppression, and catalog pinning gate the breaking cut - - [Readers follow restarts](/quest/m1/sink-restart-follow.md) - every moq-cli and moq-gst reader, moqsrc and `moq play` included, follows instances through `announced_broadcast` and an epochless `Restart` where its output can mark it; otherwise it stays on the old instance and fails once that route dies +- [In-band auth](/quest/m1/auth/README.md) - a session tells its peer what it may publish and subscribe to, unions tokens presented in band, and fails loud on an out-of-scope publish - [Cluster routing](/quest/m1/cluster-routing/README.md) - prove shared routing contracts, implement per-origin routes, and connect outbound local clusters - [Routing guide](/quest/m1/routing-guide.md) - concepts and practical outbound federation setup - [Controlled time](/quest/m1/time/README.md) - every crate and package reads time through moq-clock or @moq/clock, with regression tests for the changed contracts - [Own the QUIC stack](/quest/m1/quic/README.md) - quinn hard-forked in-tree as `moq-quic`, carrying BBR, reliable reset, hierarchical scheduling, peer limits, and endpoint sharding - [qmux on the QUIC stream state machine](/quest/m1/quic-qmux.md) - qmux runs in-tree over `moq-quic`'s stream state machine, returning credit for every byte, delivering its close frame, and ending an unfinished read after its buffered data once the session closes - [lite-07 padding](/quest/m1/lite07-padding.md) - lite-07 reserves a padding stream type that receivers read and discard, for the browser's padding probes -- [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it +- [Rust request credit](/quest/m1/rs-request-credit.md) - a moq-net request fails at once without stream credit instead of waiting +- [Generated C bindings](/quest/m1/c/README.md) - C generated from moq-ffi ships as `moq-c` 0.8.0 and replaces the hand-written libmoq +- [Exact timestamps in every binding](/quest/m1/binding-timestamps.md) - bindings carry `{value, scale}` timestamps instead of lossy microseconds - [moq-transport passthrough](/quest/m1/ietf-extension/README.md) - object IDs, properties, end status, datagram objects, and unknown track properties survive moq-transport -> moq-relay (and a relay cluster) -> moq-transport; moxygen's moq-test sections 1-7 pass nightly, except cases that use a subgroup other than 0 - [Refresh](/quest/m1/refresh/README.md) - a join asks for a fresh group once the current group's deltas exceed a publisher-advertised byte ratio, at every hop; replaces `Demand::used`/`unused` and moq-json's `delta_ratio` +- [Announcement takeover](/quest/m1/announce-takeover.md) - a new independent route explicitly continues instances served through its predecessor, with shared Rust/JS semantics and lite-07 wire +- [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges with absolute or relative ends, in either order, and replaces FETCH; merged ranges never starve a subscriber; relays fill misses by range, including over moq-transport +- [A widened max age re-offers cached groups](/quest/m1/resolved-demand.md) - a warm relay serves a reader with a wider max age the cached groups a cold relay would, with no wire change +- [Drop set](/quest/m1/drop-set.md) - each cursor judges missing groups against its own budget, so a narrow reader drops promptly and a widening re-offers groups already reported dropped on lite-07 +- [Broadcast archives](/quest/m1/archive/README.md) - ANNOUNCE, TRACKS, and a SUBSCRIBE per track record every track under a prefix to any object_store, a host serves a stored broadcast back, and the catalog drops its archive pointer for an opt-in `hls` field - [Compressed tracks](/quest/m1/compress/README.md) - a track flagged `compress` is DEFLATEd per group by moq-net, forwarded compressed, and inflated once on the relay for peers that cannot signal it; `catalog.json.z` and moq-flate go away +- [Finalize moq-lite-07](/quest/m1/lite07-finalize.md) - when the maintainer cuts it, lite-07 negotiates as `moq-lite-07` and the next release ships it - [Moxygen minimal interop](/quest/m1/smoke-moxygen.md) - waiting on Moxygen to send a registered filter type, then the smoke lane stops hiding its failure - [IETF setup waits under validity](/quest/m1/ietf-setup-validity.md) - every IETF subscriber setup wait ends once its request's route retires or its grant is revoked -- [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges with absolute or relative ends, in either order, and replaces FETCH; merged ranges never starve a subscriber; relays fill misses by range, including over moq-transport - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause -- [A public model](/quest/m1/public-model/README.md) - every model item a built-in session reaches is public, unnecessary, or marked deliberately private, so a custom publisher, subscriber, or relay builds from the public model alone - [Lazy discovery](/quest/m1/lazy-discovery/README.md) - a `consume` session requests announcements only for prefixes the origin is watching, in JS then Rust; `follow`, `broadcasts`, and the `discovery` option go away -- [Exact timestamps in every binding](/quest/m1/binding-timestamps.md) - bindings carry `{value, scale}` timestamps instead of lossy microseconds +- [A public model](/quest/m1/public-model/README.md) - every model item a built-in session reaches is public, unnecessary, or marked deliberately private, so a custom publisher, subscriber, or relay builds from the public model alone - [JSON window timestamps](/quest/m1/json-window-timestamps.md) - Rust and JS window consumers return each event's frame timestamp like the snapshot and stream consumers - [JavaScript FETCH](/quest/m1/js-fetch.md) - browser publishers answer IETF FETCH through the JS ranges request surface - [Report relays that refuse TRACK_STATUS](/quest/m1/track-status-race.md) - moqx and any other relay refusing TRACK_STATUS with an undefined code get an upstream report - [IETF fill timeout](/quest/m1/ietf-fill-timeout.md) - native FILL_TIMEOUT bounds upstream waiting on drafts 18-22, alongside local content-age enforcement - [Rust identity enforcement](/quest/m1/catalog-identity-rs.md) - `moq-mux` refuses a redefined rendition, and every Rust publisher mints a new rendition name for a decoder identity change and updates everything else in place -- [Broadcast archives](/quest/m1/archive/README.md) - ANNOUNCE, TRACKS, and a SUBSCRIBE per track record every track under a prefix to any object_store, a host serves a stored broadcast back, and the catalog drops its archive pointer for an opt-in `hls` field -- [In-band auth](/quest/m1/auth/README.md) - a session tells its peer what it may publish and subscribe to, unions tokens presented in band, and fails loud on an out-of-scope publish -- [Announcement takeover](/quest/m1/announce-takeover.md) - a new independent route explicitly continues instances served through its predecessor, with shared Rust/JS semantics and lite-07 wire - [IETF takeover](/quest/m1/ietf-takeover.md) - carry the shared takeover contract through the negotiated IETF epoch extension - [Identity and recovery guide](/quest/m1/identity-guide.md) - explain pinned catalogs, lazy jobs, takeover, restart and version compatibility - [Reference epoch bindings](/quest/m1/reference-epoch-bindings.md) - the UniFFI bindings and C ABI expose and honor a catalog reference's optional `epoch` -- [Generated C bindings](/quest/m1/c/README.md) - C generated from moq-ffi ships as `moq-c` 0.8.0 and replaces the hand-written libmoq - [Scope track priority](/quest/m1/track-priority-scope.md) - priority orders one owner's streams, sent and received, and a shared cluster session is fair across tenants - [Session send target](/quest/m1/probe-target.md) - a session probes up to the sum of its publisher reservations through whichever sender its transport has - [An evicted group is judged stale](/quest/m1/evicted-group-stale.md) - a handed-out group the cache evicted goes stale once its successor's arrival plus the budget passes, so its stalled stream resets @@ -97,11 +105,8 @@ lite-07 stream type here. - [Relay signal install](/quest/m1/relay-signal-install.md) - a shutdown signal after the relay starts serving drains it, without leaking handlers into an embedder whose startup failed - [JS redirect keys strip userinfo](/quest/m1/js-reload-userinfo.md) - js/net keys a reconnect endpoint without URL userinfo, matching Rust - [Container reads drop backfill](/quest/m1/read-backfill.md) - the JS container consumer drops groups below its read cursor like Rust -- [Rust request credit](/quest/m1/rs-request-credit.md) - a moq-net request fails at once without stream credit instead of waiting - [qmux no-wait opens](/quest/m1/qmux-no-wait.md) - the in-tree @moq/qmux rejects an over-limit create when waitUntilAvailable is false, like Chrome - [TS fidelity and recovery](/quest/m1/ts-reliability/README.md) - both modes have explicit profiles, automated validation, loss tests, and a dedicated guide -- [A widened max age re-offers cached groups](/quest/m1/resolved-demand.md) - a warm relay serves a reader with a wider max age the cached groups a cold relay would, with no wire change -- [Drop set](/quest/m1/drop-set.md) - each cursor judges missing groups against its own budget, so a narrow reader drops promptly and a widening re-offers groups already reported dropped on lite-07 - [Resumed close](/quest/m1/resume-closed.md) - a resumed JS track keeps its declared end across copies, and its `closed` and `largest()` settle on a clean end - [Replica resume](/quest/m1/replica-resume.md) - a lagging same-epoch replica resumes with every frame exactly once and in order, on lite-07 and on moq-transport - [Resumed datagrams](/quest/m1/resume-datagrams.md) - a resumed track skips only datagrams it already delivered, not every lower sequence @@ -121,23 +126,17 @@ lite-07 stream type here. - [Load CMAF decode on demand](/quest/m1/hang-cmaf-lazy.md) - `@moq/watch` fetches the cml-based `@moq/hang/cmaf` only for CMAF renditions, and the unused public CMAF encoder is deleted - [lite-06 max sequence](/quest/m1/lite06-max-sequence.md) - group sequence 2^62-1 reaches a lite-06 subscriber or is refused loudly, instead of vanishing - [Track live signal](/quest/m1/track-live.md) - a reader can wait for its track's feed to be live, so a player skips old cached groups while a fresh upstream subscription starts - +- [moq-net compiles once](/quest/m1/net-erase-transport.md) - the session drivers stop monomorphizing per transport and per crate, saving several MiB of the relay, FFI library, `libmoq` and CLI - [Binding check cost](/quest/m1/binding-check-cost.md) - `moq.cpp` compiles once per C++ standard and Gradle stays warm, measured before and after - [Audio devices on ARM](/quest/m1/audio-devices-arm.md) - `just rs audio-devices` passes on the hosted ARM runner - [CI housekeeping](/quest/m1/ci-housekeeping.md) - queue nextest at 24 threads, stale Pages artifacts cleared, the actionlint ignore dropped - - [Interop audio delivery](/quest/m1/interop-audio.md) - every interop subscriber decodes and checks the publishers' 440 Hz tone, and the Rust publisher sends Opus - [AAC in the native JS fallback](/quest/m1/libav-aac.md) - Node and Bun players decode AAC, and the Rust interop publisher adds an AAC track - [Native audio decode conformance](/quest/m1/audio-decode-conformance.md) - Opus and AAC decode through every moq-audio backend, moq-c, and moq-ffi is checked against known tones and clips, like moq-video's suite - [Playwright browsers from Nix](/quest/m1/playwright-nix.md) - every Playwright check runs on Nix's Chromium locally and in CI, with npm pinned to match - - [Allocation-free single-range subscriber window](/quest/m1/window-alloc.md) - a one-range subscribe resolves its window without a heap allocation, recovering the join cost per-subscriber ranges added - -- [moq-net compiles once](/quest/m1/net-erase-transport.md) - the session drivers stop monomorphizing per transport and per crate, saving several MiB of the relay, FFI library, `libmoq` and CLI - [moq-net internals](/quest/m1/net-internals.md) - no per-group session clones, no transport type parameter in lite and IETF internals, and the `Local` check measured - - [Subscribe update churn](/quest/m1/subscribe-update-churn.md) - a viewer sends SUBSCRIBE_UPDATE only on a meaningful change, measured after the publisher stall fix - - [Dead code and API surface audit](/quest/m1/audit-dead-code.md) - repeat the removal audit before the next major release - [Dependency and alternatives audit](/quest/m1/audit-deps.md) - dependencies are current, deduplicated, needed, and the best maintained option - [Test cost audit](/quest/m1/audit-test-cost.md) - the slowest, flakiest, and least valuable tests are fixed, deleted, or planned @@ -147,6 +146,7 @@ lite-07 stream type here. - [Artifact size audit](/quest/m1/audit-size.md) - binary, FFI, WASM, and bundle growth since the m0 run is explained, fixed, or planned - [Compile time audit](/quest/m1/audit-compile-time.md) - build-time hotspots are measured against the m0 baselines and the worst are fixed or planned - [Performance audit](/quest/m1/audit-perf.md) - benchmarks that regressed since the m0 run are explained, fixed, or planned, and unbenched hot paths get coverage +- [Quest tree audit](/quest/m1/audit-quests.md) - the tree agrees with itself and with main ## Related diff --git a/quest/m1/archive/README.md b/quest/m1/archive/README.md index d528cdb82c3..0c2ae050934 100644 --- a/quest/m1/archive/README.md +++ b/quest/m1/archive/README.md @@ -4,9 +4,9 @@ `moq-archive` stores every track of every broadcast under a path prefix into any `object_store::ObjectStore` and serves a stored broadcast back as an -ordinary broadcast, with no catalog dependency. `archive::Producer` finds -broadcasts by ANNOUNCE, their tracks by TRACKS, and records each through its -own SUBSCRIBE; `archive::Consumer` serves a stored prefix, which a +ordinary broadcast, with no catalog dependency. A host finds +broadcasts by ANNOUNCE and runs one `archive::Producer` per broadcast, which +finds its tracks by TRACKS and records each through its own SUBSCRIBE; `archive::Consumer` serves a stored prefix, which a host publishes at a path of its choosing. Track, group, and frame structure and payloads survive byte for byte, so any consumer of the live broadcast, HLS included, works on the stored one. diff --git a/quest/m1/archive/catalog-hls.md b/quest/m1/archive/catalog-hls.md index 00059cc7fa6..1728ed95792 100644 --- a/quest/m1/archive/catalog-hls.md +++ b/quest/m1/archive/catalog-hls.md @@ -18,8 +18,11 @@ Decided by the maintainer on 2026-10-17: that stores them ([epic](/quest/m1/archive/README.md)). - **Renamed to `hls`** (catalog field, track naming, types, wording), since it exists only for HLS compatibility, not archiving or seeking; seeking uses - `Time(pts)` ranges. The track naming convention becomes `.hls.z`; - consumers still use the advertised name. + `Time(pts)` ranges. The track naming convention becomes `.hls`; + consumers still use the advertised name. Decided 2026-10-20 (maintainer): + no `.z` suffix. This quest publishes it uncompressed, and + [one compressed track per name](/quest/m1/compress/migrate.md), which + lands after it, sets `compress: true` on the same name. - **Opt-in.** `moq-mux` publishes it only when the application enables it, and nothing publishes it by default. Rejected: always-on timelines for every track, which cost every publisher for a feature most never use. diff --git a/quest/m1/audit-perf.md b/quest/m1/audit-perf.md index ea919752465..efd65f1efd9 100644 --- a/quest/m1/audit-perf.md +++ b/quest/m1/audit-perf.md @@ -128,8 +128,10 @@ Single-axis fan-out benches to widen: `session_delivery_paced` (viewers only), `bench/workloads/fanout*.toml` (connections only). Fan-out paths need a sweep over both axes (AGENTS.md). JS perf is judged in a -browser, not node or bun. PR-time gating belongs to -[benchmark regressions in CI](/quest/m2/bench-ci.md); don't duplicate it. +browser, not node or bun. Benchmarks never gate the merge queue (decided +2026-10-20, maintainer): each Nightly compares them into the git.moq.dev +report, which the [nightly benchmark regression report](/quest/m2/bench-ci.md) +owns; don't duplicate it. ## Related diff --git a/quest/m1/audit-quests.md b/quest/m1/audit-quests.md new file mode 100644 index 00000000000..9156e7567ab --- /dev/null +++ b/quest/m1/audit-quests.md @@ -0,0 +1,46 @@ +# [M] Quest tree audit + +## Goal + +The quest tree agrees with itself and with `main`: done, stale, +conflicting, and misranked quests are fixed. + +## Plan + +Recurs once per major release (decided 2026-10-19). The PR that finishes +this run also adds the next one with the same file name at the end of the next +milestone's `Required` (m0 to m1, m1 to m2). The successor keeps the Goal +and this paragraph; the rest of its Plan is rewritten with what this run +learned: baselines, results, and what to check first. A successor placed in m2 +also lists the [release cut](/quest/m2/release/cut.md) under `Required`, so it +waits for the release in flight to ship. Only `/quest-delete` ends the +recurrence. Fixes land in the run's PR, one commit each, rather than as follow-ups. +Only a change that needs a maintainer decision (public API, scope) or is +independently landable and too large to review alongside the rest becomes a +follow-up through `/quest-plan`. + +Run `/quest-audit`. After a release cut shifts the milestones, also check +that the move left no broken links or misplaced quests. Earlier runs: ad hoc +2026-09-30, 10-05, 10-08, 10-10; recurring 2026-10-19 and 10-20. + +Checkpoint, 2026-10-20, against `main` at `96d6b5bd4`: + +- Read all 426 documents and the 22 questline branches on origin with nine + parallel auditors, together with the moq.pro tree, since many of its quests + wait on ours. `quest check` was clean, so every finding was content. +- Settled the four conflicts the 10-19 run left, plus about 20 more with the + maintainer: e2ee publishes through a moq-net wrapper trait + ([net-wrap](/quest/m2/net-wrap/README.md)); m1 holds everything before the + cut, breaking or not; C halves wait on the generated package; the timeline + is `.hls` with `compress: true`; the publisher priority is the send + order tiebreak; an auth handle backs one live session; GroupState storage + is benchmarked before either redesign. +- Retired landed work whose PRs left the quest behind: goaway coverage + (#5998), the JS dead-route error (#5939), the uring stream header (#5690), + and the catalog-reader smoke. Deleted 29 merged questline branches. +- Most common findings, so check these first next run: ranks inside a + milestone README that put a quest above its own prerequisites; "after + #N" or "held until #N" wording after #N merged; questline branches on + origin whose directory is gone from `main`; and plans built on a design + another quest has since replaced (SUBSCRIBE_ALL, the e2ee bridge, the + per-PR CI model before #6038). diff --git a/quest/m1/auth/README.md b/quest/m1/auth/README.md index 24cf4f53a7b..cf9ff0086de 100644 --- a/quest/m1/auth/README.md +++ b/quest/m1/auth/README.md @@ -157,8 +157,7 @@ because `main` assigned 0x3A to NOT_FETCHABLE first. - [Request tokens beyond Rust](/quest/m3/request-token-surfaces.md) - JS, moq-ffi, and moq-cli request-token APIs, when a consumer needs one - [Token on FETCH](/quest/m3/auth-token-fetch.md) - TRACK_STATUS and FETCH carry - and honor a request token. Parked until the maintainer decides whether to - reverse the standalone-FETCH decision and bring it into this epic + and honor a request token, deferred until a consumer asks - [#5107](https://github.com/moq-dev/moq/pull/5107) - landed request-token decoding, including session-fatal REQUEST_UPDATE decode failures - [#5106](https://github.com/moq-dev/moq/pull/5106) - landed setup extension diff --git a/quest/m1/auth/issuer.md b/quest/m1/auth/issuer.md index 6acc18cf6f3..b7f2b50591c 100644 --- a/quest/m1/auth/issuer.md +++ b/quest/m1/auth/issuer.md @@ -28,6 +28,11 @@ connection makes, with no re-add or re-announce from the application. refusal and a final stop on `Unauthorized`). With the handle in moq-net, `Connection::auth()` returns that `moq_net::auth::Handle`, and `connection::auth::{Handle, Auth}` go away. +- Decided 2026-10-20 (maintainer): a handle backs one active session at a + time, plus the predecessors it is draining, matching the reconnect code's + single active generation. Attaching a handle to a second concurrent session + fails loudly; independent concurrent sessions use separate handles, so an + `Auth` never has to pick between two live grants. - An `Auth`'s `grant()` and `closed()` follow the session that is live now: the grant clears while disconnected and returns once the new session's peer answers, and `closed()` settles only on a refusal or when the handle ends. diff --git a/quest/m1/binding-timestamps.md b/quest/m1/binding-timestamps.md index 16cf4a3fee9..241adbc930b 100644 --- a/quest/m1/binding-timestamps.md +++ b/quest/m1/binding-timestamps.md @@ -30,9 +30,8 @@ Decided with the maintainer on 2026-10-18: - **Scope is the bindings only.** The core and JS types stay as they are. - **m1**, beside data consumer timestamps. -- **Lands after #5677**, which ships the data-value timestamps with the - interim microsecond fields; this quest converts every site at once, - `MoqJsonValue` included. +- **#5677 shipped** the data-value timestamps with the interim microsecond + fields; this quest converts every site at once, `MoqJsonValue` included. - **Shape mirrors the core:** one `{value, scale}` per value, `value` at most `2^62 - 1` and `scale` non-zero and at most `2^62 - 1`, matching `Timestamp::new` and `Timescale::new`. The FFI boundary refuses anything else with @@ -57,6 +56,14 @@ Verification: round-trip a 90 kHz tick, a present zero, and untimed through moq-ffi, moq-c, C++ and every wrapper's tests; refuse scale 0, an out-of-range scale, and an out-of-range value; `just test interop --all`. +Decided 2026-10-20 (maintainer): the moq-c half lands on the +[generated C package](/quest/m1/c/README.md), not the frozen hand-written +ABI, so C users break once. + +## Required + +- [Generated C bindings](/quest/m1/c/README.md) - `moq_timestamp` reaches C through the generated package, so C users break once + ## Related - [Controlled time](/quest/m1/time/README.md) - wall and monotonic time, not media timestamps; keep the types distinct diff --git a/quest/m1/c/README.md b/quest/m1/c/README.md index cdaa9b38596..9715f158b77 100644 --- a/quest/m1/c/README.md +++ b/quest/m1/c/README.md @@ -41,7 +41,7 @@ The line owns the pre-cut check: every `doc/lib/c` sample and the C interop client build and run against the built release archive, not only in-tree. [Published archive proof](/quest/m2/release/c-publish.md) repeats it after publication; [old-crate retirement](/quest/m2/release/c-retire.md) follows -its final notice release and is not a prerequisite of this preparation. +that publication and is not a prerequisite of this preparation. The hand-written crate gets no more feature work: its shutdown, CMake library, and fetch quests were abandoned for this line, and hidden is done. @@ -50,6 +50,15 @@ is a narrow correctness exception. Its existing claim API migrates with Rust and JS so C cannot retain the unsafe accept path while this line waits. That change owns its C tests and `MIGRATE.md` entries; other API reshaping stays here. +Decided 2026-10-20 (maintainer): m1 quests that change the C API +([exact binding timestamps](/quest/m1/binding-timestamps.md), +[scope track priority](/quest/m1/track-priority-scope.md), +[binding linger](/quest/m1/linger-ffi.md), +[reference epoch bindings](/quest/m1/reference-epoch-bindings.md), +[Rust identity enforcement](/quest/m1/catalog-identity-rs.md), and +[dynamic subscribe ranges](/quest/m1/subscribe-ranges/dynamic.md)) Require this +line and land their C half on the generated package, so C users break once. + ## Required - [moq-c package](/quest/m1/c/package.md) - the generated header ships as `moq-c` 0.8.0 with `moq::c`, pkg-config, and a release workflow diff --git a/quest/m1/c/package.md b/quest/m1/c/package.md index f263341e818..bf934d9f4da 100644 --- a/quest/m1/c/package.md +++ b/quest/m1/c/package.md @@ -19,7 +19,7 @@ under the same names. - `doc/index.md` says the C library ships static and shared; say what the package actually ships. - A release workflow and nightly/CI jobs follow the C++ package's, on the same - generator pin; `v0.11.0-kixelated.4+v0.32.2` already carries the C backend + generator pin; `v0.11.0-kixelated.5+v0.32.2` already carries the C backend (`--lang c`). - Ship a `MIGRATE.md` with the package, since it is what C users install. Carry over `rs/moq-c/MIGRATE.md`'s history and add an `## Unreleased` @@ -31,3 +31,7 @@ under the same names. - The Cross-Package Sync rows that name the hand-written `rs/moq-c` (as the target of `rs/moq-ffi` changes, and its `moq.h` C ABI) point at the generated package instead; update `AGENTS.md` in this quest. + +The hand-written `moq_session_failure` names a retrying first dial; the +generated package must keep that, through the moq-ffi report added in +[moq-dev/moq#6131](https://github.com/moq-dev/moq/pull/6131). diff --git a/quest/m1/catalog-identity-rs.md b/quest/m1/catalog-identity-rs.md index 9f14d21ce33..9c1e40a772c 100644 --- a/quest/m1/catalog-identity-rs.md +++ b/quest/m1/catalog-identity-rs.md @@ -67,6 +67,14 @@ Later, at lower priority: watch and HLS export consumer coverage for in-band resolution changes, rendition switches, reconnects, and late joiners against the contract. +Decided 2026-10-20 (maintainer): the moq-c half lands on the +[generated C package](/quest/m1/c/README.md), not the frozen hand-written +ABI, so C users break once. + +## Required + +- [Generated C bindings](/quest/m1/c/README.md) - the moq-c config re-declaration fix lands on the generated package, not the frozen hand-written ABI + ## Related - [Broadcast archives](/quest/m1/archive/README.md) - an archived catalog describes every stored group by the identity contract diff --git a/quest/m1/close-tail-drain.md b/quest/m1/close-tail-drain.md index 825ccfce7c0..47a23986e7f 100644 --- a/quest/m1/close-tail-drain.md +++ b/quest/m1/close-tail-drain.md @@ -62,7 +62,7 @@ Decided (maintainer, 2026-10-14): reader got the tail and fails before the fix. Cover both the buffered-tail case and one where a control or accept error wins the poll order. Give `older_lite_close_keeps_ack_drain` that reader assertion. -- Once #5530 has landed, switch `moq_net::transport::memory`'s +- #5530 landed the in-memory transport: switch `moq_net::transport::memory`'s `SendStream::poll_closed` to resolve on `finish` (delivery into the channel), dropping its "finished and empty" rule. Do this unless the memory bench (`rs/moq-net/benches/memory.rs`) shows a regression; if it diff --git a/quest/m1/cluster-routing/identity.md b/quest/m1/cluster-routing/identity.md index 37027d5368f..befa96364a8 100644 --- a/quest/m1/cluster-routing/identity.md +++ b/quest/m1/cluster-routing/identity.md @@ -41,13 +41,19 @@ its own is refused; a proof copied onto another session is refused; a session with no proof advertises no route of its own. Wire into CI with Rust/JS interop. +Decided 2026-10-20 (audit): this waits only on +[Token in band](/quest/m1/auth/token-in-band.md), not on the whole auth line. +That quest defines the AUTH stream (AUTH, AUTH_OK, AUTH_ERROR); this one adds +AUTH_CNF and the proof to it and specifies their order in the draft, so Rust +and JS implement one stream. + Wire: `drafts/draft-lcurley-moq-lite.md` gains the key and proof messages in lite-07-wip; `just drafts check` passes. Public API: likely none beyond connection setup; report any key handle that leaks into `moq-ffi`. ## Required -- [In-band auth](/quest/m1/auth/README.md) - the AUTH stream carries the key and its proof +- [Token in band](/quest/m1/auth/token-in-band.md) - defines the AUTH stream the key and its proof extend ## Related diff --git a/quest/m1/cluster-routing/routes.md b/quest/m1/cluster-routing/routes.md index e822cc63ea9..801db73b418 100644 --- a/quest/m1/cluster-routing/routes.md +++ b/quest/m1/cluster-routing/routes.md @@ -574,7 +574,7 @@ Open, for the implementer to settle and record: carry the old assumption into acceptance. - How much of today's route trie, fronts, and `route_order` survives intact. -The #4349 counterexamples are the `residual_*` tests in +The counterexamples from #4349 (closed; this quest still owns the fix) are the `residual_*` tests in `rs/moq-net/tests/integration/mesh_withdraw.rs`. They pin today's hunting on lite-06: a stale path outlasting the 300 ms hold, a revival the quarantine misses, and 179 hunting relays (3 reviving) over 200 seeded meshes. Turn each into a @@ -590,8 +590,8 @@ broadcast stays live. Pin today's sequence beside the `residual_*` tests; the epochless Restart rule above decides what it becomes. Tests, time mocked: the handoff fixtures and sweep in `handoff.rs`; -#4644's live-graph withdrawal and failover tests -(`quest/m0/path-hunting` branch) as acceptance, run on the wip version; a +#4644's live-graph withdrawal and failover tests (the PR closed and +its branch is gone; fetch `refs/pull/4644/head`) as acceptance, run on the wip version; a link flap with many broadcasts behind one origin costs one ROUTE change per link; a ring of three where a stale neighbour cannot revive an ended broadcast; a two-uplink mesh that never carries CDN routes back to the CDN; @@ -642,10 +642,6 @@ release. - [Identity proof](/quest/m1/cluster-routing/identity.md) - proven own identities and subject-qualified forwarded ones -## Closes - -- [#4349](https://github.com/moq-dev/moq/issues/4349) - close once the `residual_*` tests in `mesh_withdraw.rs` retract once everywhere - ## Related - [Route wakes](/quest/m2/route-wakes.md) - index work follows the settled source-selection model diff --git a/quest/m1/compress/migrate.md b/quest/m1/compress/migrate.md index 1cb2d576385..47addf6d628 100644 --- a/quest/m1/compress/migrate.md +++ b/quest/m1/compress/migrate.md @@ -4,8 +4,8 @@ Every first-party compressed track is one name with `compress: true`: `catalog.json` in moq-mux and js/publish, each moq-stats `.json` track, and -the hang timeline. `catalog.json.z`, the stats `.z` siblings, -`.timeline.z`, hang's catalog `compression` field, the "hangz" +the hang `.hls` track. `catalog.json.z`, the stats `.z` siblings, +hang's catalog `compression` field, the "hangz" consumer options, the moq-flate crate, `@moq/flate`, `moq-mux::binary`, moq-json's and `@moq/json`'s `Config::compression`, and the bindings' `flate` namespace are deleted. @@ -14,16 +14,23 @@ namespace are deleted. See the [epic](/quest/m1/compress/README.md) for the decisions. +Decided 2026-10-20 (maintainer): the timeline is `.hls` with +`compress: true`, never `.hls.z`. [Catalog HLS field](/quest/m1/archive/catalog-hls.md) +renames it and publishes it uncompressed, and the +[archive consumer](/quest/m1/archive/consumer.md) replaces the CLI's archive +reader, so this quest lands after both and only sets the flag. + - Delete the `.z` names outright, with no transition release: hangz is opt-in and `main` takes breaking changes. Add the MIGRATE.md entries. - Publishers: `rs/moq-mux/src/catalog/producer.rs`, `js/publish` (`broadcast.ts`, `catalog.ts`), `rs/moq-stats` (drop `TrackPair`), and - `rs/hang/src/timeline.rs` set `compress` on their one track. -- Consumers: `rs/moq-mux/src/catalog/consumer.rs`, `rs/moq-cli` - (`subscribe.rs`, `archive.rs`), and `js/watch` lose their hangz paths. - moq-mux's hang catalog adapter over `hang::catalog::Consumer` - ([#6114](https://github.com/moq-dev/moq/pull/6114)) drops its `format` - field and `CatalogFormat::HangZ` goes away; check whether + the `hls` track writer (`rs/hang/src/timeline.rs` today) set `compress` on + their one track. +- Consumers: `rs/moq-mux/src/catalog/consumer.rs`, `rs/moq-cli/src/subscribe.rs`, + and `js/watch` lose their hangz paths; the CLI's archive path is already on + the archive consumer. moq-mux's hang catalog adapter over + `hang::catalog::Consumer` ([#6114](https://github.com/moq-dev/moq/pull/6114)) + drops its `format` field and `CatalogFormat::HangZ` goes away; check whether `catalog::msf::Consumer` can reuse hang's escaping-path check instead of its own (decided 2026-10-20 instead of a separate quest). - moq-stats deltas: the `.z` side uses merge-patch deltas and the plain side @@ -54,3 +61,5 @@ Public API: removals listed above. Wire: the hang catalog loses ## Required - [JavaScript](/quest/m1/compress/js.md) - js/publish and js/watch need the JS side before they drop `.z` +- [Catalog HLS field](/quest/m1/archive/catalog-hls.md) - renames the timeline to `.hls`, which this compresses +- [Archive consumer](/quest/m1/archive/consumer.md) - replaces the CLI archive reader that reads `.z` timelines diff --git a/quest/m1/export-rendition-rebind.md b/quest/m1/export-rendition-rebind.md index 5eadf06b53f..0ef6263b186 100644 --- a/quest/m1/export-rendition-rebind.md +++ b/quest/m1/export-rendition-rebind.md @@ -42,6 +42,6 @@ that track ends) while the catalog names the new one. Raised by a Codex review of #5367, which added the per-rendition `track` field; the `broadcast` gap predates it. Deferred from that PR by the -maintainer's recommended option. Starts after #5367 merges. +maintainer's recommended option. Public API: none. Wire: none. diff --git a/quest/m1/ietf-extension/README.md b/quest/m1/ietf-extension/README.md index d438815430e..fa0d603ae5f 100644 --- a/quest/m1/ietf-extension/README.md +++ b/quest/m1/ietf-extension/README.md @@ -48,9 +48,10 @@ cannot answer unambiguously is refused. turns it on. A non-relay peer that negotiated it would receive properties and status objects it can only discard. - **Dynamic groups** are supported rather than treated as an opaque track - property. A private request path carries NEW_GROUP_REQUEST upstream and - coalesces redundant requests until the largest group advances. Plain - moq-lite does not advertise this capability. + property. The public `Demand::wanted` path (#5930, through relays in + #6030) carries NEW_GROUP_REQUEST upstream and coalesces redundant requests + until the largest group advances, so dynamic groups are no longer part of + the private carve-out below. - **Immutable properties** keep their serialized inner bytes, and the track its ingress moq-transport version, including across relay hops. Refuse forwarding between the absolute-key/QUIC-varint drafts 14/15, delta-key/QUIC-varint draft 16, diff --git a/quest/m1/lazy-discovery/README.md b/quest/m1/lazy-discovery/README.md index 44cf2ca8eb8..5caa447d073 100644 --- a/quest/m1/lazy-discovery/README.md +++ b/quest/m1/lazy-discovery/README.md @@ -32,8 +32,9 @@ Shared rules for the children: filters locally. - When the last watcher of a prefix leaves, its request lingers for the connection's existing `linger` (default 2s) before closing. No new option. -- A plain `request(path)` stays blind: it opens no announce request and - subscribes at once. +- A plain JS `request(path)` stays blind: it opens no announce request and + subscribes at once. Rust has no blind requests or answers (decided + 2026-10-20, recorded in [the Rust child](/quest/m1/lazy-discovery/rust.md)). - Docs and examples that a change makes stale are updated in that change. End to end, once all children land: the customer's flow (one connection, diff --git a/quest/m1/lazy-discovery/rust.md b/quest/m1/lazy-discovery/rust.md index 67b87a9dfa1..71791c22d58 100644 --- a/quest/m1/lazy-discovery/rust.md +++ b/quest/m1/lazy-discovery/rust.md @@ -46,8 +46,13 @@ closes 2s after nothing beneath it is watched (a fixed constant, since one-shot connections have no `linger`); a refused or reset request downgrades the whole session to blind answers. -Open for the plan round: the Rust equivalent of an announced request, how the -watch set crosses the poll-based session driver, and moq-ffi and binding +Decided 2026-10-20 (maintainer): no blind answers in Rust. moq-net has no +blind requests, and `origin::Consumer::announced_broadcast` +already waits on an announced route rather than falling back, so a refused or +reset announce request does not downgrade a Rust session; the request fails +like any other. This is the one place Rust does not mirror JS. + +Open for the plan round: how the watch set crosses the poll-based session driver, and moq-ffi and binding fallout (see the Cross-Package Sync table in `AGENTS.md`). Tests must pin the relay side: an origin in relay mode (cluster and upstream diff --git a/quest/m1/linger-ffi.md b/quest/m1/linger-ffi.md index 16df48346cc..5f2e783ac17 100644 --- a/quest/m1/linger-ffi.md +++ b/quest/m1/linger-ffi.md @@ -35,3 +35,11 @@ whether `cpp/obs` should finish on a clean stop. Test in `moq-ffi` with a mocked clock: a late subscriber within max-age gets the media, the broadcast closes once it passes, and cancelling closes early. Each binding needs at least a smoke test that the call exists and returns. + +Decided 2026-10-20 (maintainer): the moq-c half lands on the +[generated C package](/quest/m1/c/README.md), not the frozen hand-written +ABI, so C users break once. + +## Required + +- [Generated C bindings](/quest/m1/c/README.md) - the finish handle reaches C through the generated package, not the frozen hand-written ABI diff --git a/quest/m1/lite07-finalize.md b/quest/m1/lite07-finalize.md index db471527f73..2948209b5f4 100644 --- a/quest/m1/lite07-finalize.md +++ b/quest/m1/lite07-finalize.md @@ -54,13 +54,16 @@ The cut: rather than misparse; no compatibility shim. - The draft's lite-07 changelog matches the wire and `just drafts check` passes. -- Preserve SETUP's rule: "a receiver MUST treat a longer SETUP as a protocol - violation and MAY reject it based on the length prefix alone" (65,536 - bytes). The general cap proposed by request caps (#4820) only permits - rejection (MAY), so it does not replace SETUP's stronger requirement. - Remove the SETUP sentence only if the general rule requires the same - rejection. If SETUP keeps its own rule, the lite-07 changelog still names - that rejection, not only the cap. +- SETUP keeps its stronger rule: #4820 put it under the general Message + Length cap, which only permits rejection (MAY), while "a receiver MUST + treat a longer SETUP as a protocol violation" stays. The lite-07 changelog + names that rejection, not only the cap. + +Decided 2026-10-20 (maintainer): [Drop set](/quest/m1/drop-set.md) joins +Required, because it adds `Max Age` to lite-07's SUBSCRIBE_DROP. Only +[Token in band](/quest/m1/auth/token-in-band.md) gates the cut from the auth +line, because it is that line's lite-07 wire work; the rest is API work that +does not block the rename. Decided 2026-10-17: the archive line's discovery request joins Required, so lite-07 ships what the archive and the IETF SUBSCRIBE_TRACKS mapping need. @@ -85,9 +88,10 @@ is published; older versions are unchanged. ## Required -- [In-band auth](/quest/m1/auth/README.md) - lite-07 carries the Auth Stream and UNAUTHORIZED (0x3B) +- [Token in band](/quest/m1/auth/token-in-band.md) - lite-07 carries the Auth Stream and UNAUTHORIZED (0x3B) - [Identity proof](/quest/m1/cluster-routing/identity.md) - lite-07 carries the routing key and its per-session proof - [Routes and announces](/quest/m1/cluster-routing/routes.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the hop list gone +- [Drop set](/quest/m1/drop-set.md) - lite-07's SUBSCRIBE_DROP carries `Max Age`, so a widening re-offers groups reported dropped - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - SUBSCRIBE carries ranges and an order and lite FETCH is gone, in Rust and JS - [Announcement takeover](/quest/m1/announce-takeover.md) - ANNOUNCE_TAKEOVER preserves a proven instance under a new independent route - [Compressed tracks in js/net](/quest/m1/compress/js.md) - JS reads and writes compressed lite-07 tracks instead of refusing them diff --git a/quest/m1/public-model/README.md b/quest/m1/public-model/README.md index d4ba6348c62..019c8064fbd 100644 --- a/quest/m1/public-model/README.md +++ b/quest/m1/public-model/README.md @@ -28,8 +28,8 @@ Rules every child follows: enforces the marks): - "moq-transport passthrough only" (decided 2026-10-13; subgroups were dropped from it in #5704): object IDs and `skip`, object and unknown track properties, END_OF_GROUP status - objects, datagram object IDs, dynamic groups, and upstream request - signaling. Owned by [moq-transport passthrough](/quest/m1/ietf-extension/README.md). + objects, datagram object IDs, and upstream request signaling. Dynamic + groups left this list when #5930 made `Demand::wanted` public. Owned by [moq-transport passthrough](/quest/m1/ietf-extension/README.md). A relay built on the public model drops them, as plain moq-lite does. - "tunes the built-in sessions": constants and wake or allocation batching that change cost, never behavior a third party can observe. diff --git a/quest/m1/public-model/answer.md b/quest/m1/public-model/answer.md index 13f1adaab4b..c57e0f8ccb9 100644 --- a/quest/m1/public-model/answer.md +++ b/quest/m1/public-model/answer.md @@ -115,12 +115,13 @@ JS items (`js/net/src/`): - **Blind answers** (the same loop, raised in review of #5717). Every JS session answers each request no route covers by subscribing on its peer: always, and as the only path after a peer refuses or resets discovery. - [Lazy discovery](/quest/m1/lazy-discovery/README.md) keeps a plain - `request(path)` blind, and its Rust child adds the same downgrade, so - blind answering stays and needs a public shape before the loop goes. - **Decision** (settled 2026-10-16): - - ✅ A fallback answerer, `origin::Producer::fallback()` in Rust and - `Origin.Producer.fallback()` in JS. The name is still open: propose + [Lazy discovery](/quest/m1/lazy-discovery/README.md) keeps a plain JS + `request(path)` blind, so blind answering stays in JS and needs a public + shape before the loop goes. Rust has no blind answers (decided 2026-10-20, + maintainer, recorded in [Rust lazy discovery](/quest/m1/lazy-discovery/rust.md)), + so no Rust session needs this shape. + **Decision** (settled 2026-10-16, narrowed to JS 2026-10-20): + - ✅ A fallback answerer, `Origin.Producer.fallback()` in JS. The name is still open: propose alternatives in this quest's PR before shipping it. It returns the same `Dynamic` request stream, scoped to the handle's allowed paths. It receives only requests that no announced or served route covers. Once @@ -138,9 +139,8 @@ JS items (`js/net/src/`): announced route. This reuses one call but puts an unadvertised, claim-free entry in the route table, where selection and split-horizon must special-case it. - Land the Rust side with or after Rust lazy discovery (Required below), - which brings blind answers to Rust. Delete the JS loop only once the - fallback answers the same requests, including after a downgrade. + Delete the JS loop only once the fallback answers the same requests, + including after a downgrade. - `wire.ts` `OriginProducer.expect` (`connection/reload.ts`, keeps requests pending while a reconnect is in flight): public if Rust has the same need across a reconnect, else private to the connection. diff --git a/quest/m1/quic/README.md b/quest/m1/quic/README.md index bfce7cb65c1..fd36bfe609c 100644 --- a/quest/m1/quic/README.md +++ b/quest/m1/quic/README.md @@ -5,10 +5,12 @@ MoQ owns the QUIC features it needs in an in-tree stack, `moq-quic`, hard forked from quinn rather than noq, whose multipath, NAT traversal, and QAD MoQ never negotiates. One core serves the tokio backend, the thread-per-core -`moq-uring` backend, and qmux; iroh stays on upstream noq. The features are -BBR correctness, reliable stream resets, hierarchical stream scheduling with -per-broadcast fairness, bounded incomplete-message memory, wider limits for -relay peers, and endpoint sharding. +`moq-uring` backend, and qmux; iroh stays on upstream noq. BBR correctness, +reliable stream resets (#5690), hierarchical send groups (#5711, #5770), one +`Limits` type for relay peers (#5871), and endpoint sharding (#5688) are on +`main`. What remains is bounded incomplete-message memory; per-broadcast +fairness gives the crate-private send domains (#5872) their consumer in +[Scope track priority](/quest/m1/track-priority-scope.md). qmux on the shared stream state machine is its own m1 quest, [quic-qmux](/quest/m1/quic-qmux.md), outside this epic. Per-stream acknowledgment progress, per-stream deadlines, and the experiments @@ -27,8 +29,8 @@ moq-dev/noq is frozen to security patches for `release` and archived once no released MoQ crate depends on it. The seven BBR correctness fixes shipped in moq-noq 1.3.1 (#4206) are in -`moq-quic`'s BBR3. The remaining BBR quests here all edit -`bbr3/mod.rs`, so one owner should work there at a time. Controller-level +`moq-quic`'s BBR3. BBR quests all edit `bbr3/mod.rs`, so one owner should +work there at a time. Controller-level regressions extend the shared test `Sim` in `bbr3/mod.rs` with only what each needs, rather than adding another simulation loop; a fix at the transport boundary still needs a transport test through the real callbacks. The diff --git a/quest/m1/quic/receive-memory.md b/quest/m1/quic/receive-memory.md index 476565e9437..1e2c83477a9 100644 --- a/quest/m1/quic/receive-memory.md +++ b/quest/m1/quic/receive-memory.md @@ -18,6 +18,15 @@ peer-visible QUIC flow control instead of a separate header quota error. The native transport primitive is a separate prerequisite; do not activate whole-message retention without its recovery policy in this quest. +In flight as [#5878](https://github.com/moq-dev/moq/pull/5878) (draft); +the earlier drafts #5735 and #5752 were closed in its favor. + +Decided 2026-10-20 (maintainer): three quests break moq-net's +`transport::poll` traits, so they land in order and each rebases its trait +change on the last: this quest first, then +[Rust request credit](/quest/m1/rs-request-credit.md), then +[Scope track priority](/quest/m1/track-priority-scope.md). + That primitive landed as a provisional seam to finalize here: both native drivers' receive streams have a copying `poll_read_held` that yields a `Credit` for exactly the bytes read. Dropping a `Credit` returns it, so a diff --git a/quest/m1/reference-epoch-bindings.md b/quest/m1/reference-epoch-bindings.md index 3e426407772..7f8df4e4402 100644 --- a/quest/m1/reference-epoch-bindings.md +++ b/quest/m1/reference-epoch-bindings.md @@ -23,7 +23,15 @@ field: the first instance resolved, without its guarantee. `doc/lib`). Test a matching, mismatched and epochless reference through `moq-ffi` and -libmoq, and run `just test interop --all`. +the C package, and run `just test interop --all`. + +Decided 2026-10-20 (maintainer): the C half lands on the +[generated C package](/quest/m1/c/README.md), not the frozen hand-written +ABI, so C users break once. + +## Required + +- [Generated C bindings](/quest/m1/c/README.md) - the reference `epoch` reaches C through the generated package, not the frozen hand-written ABI ## Related diff --git a/quest/m1/refresh/README.md b/quest/m1/refresh/README.md index 5aae00de8ab..4b8a4fd781f 100644 --- a/quest/m1/refresh/README.md +++ b/quest/m1/refresh/README.md @@ -44,9 +44,10 @@ Decided with the maintainer on 2026-10-16. unit as moq-json's `delta_ratio`, which it replaces. `None` means joins never trigger groups. It is the only home of the ratio: producers that receive a created track read it from there. -- **Order.** Relay requests for dynamic groups landed first (#5708) as - relay-internal IETF support. The core quest makes its private request - path public, so the work is promoted rather than redone. +- **Order.** Relay requests for dynamic groups (#5708) were promoted to + the public `Demand::wanted` and `track::Info::refresh` in #5930, and + #6030 carries refresh demand through lite relays. The moq-json producers + remain. - **m1.** Replacing `Demand` is a breaking API change, and m1 finishes the planned breaking APIs before the cut. diff --git a/quest/m1/resolved-demand.md b/quest/m1/resolved-demand.md index 6c945e1b0e2..f7e01852efd 100644 --- a/quest/m1/resolved-demand.md +++ b/quest/m1/resolved-demand.md @@ -53,9 +53,9 @@ with no wire change. budget covers, in the subscription's order: newest first when descending (the default, so congestion sheds backlog rather than the live edge), and lowest first only when ascending. This extends the re-offer - [moq-dev/moq#5929](https://github.com/moq-dev/moq/pull/5929) adds for a + [moq-dev/moq#5929](https://github.com/moq-dev/moq/pull/5929) landed for a widened range (a group outside the ranges is parked and re-offered when they - widen); land after it and reuse that path rather than a second one. Only + widen); reuse that path rather than a second one. Only groups this cursor never delivered come back, keeping #5929's rule that nothing already served goes out twice; a group already delivered and since evicted downstream is a cache miss, which the relay fills upstream by range @@ -170,7 +170,7 @@ max age or the range. ## Required -- [Dynamic range requests](/quest/m1/subscribe-ranges/dynamic.md) - #5929 lands the re-offer and serving path and the range miss fill this reuses +- [Dynamic range requests](/quest/m1/subscribe-ranges/dynamic.md) - the range miss fill this reuses, beside #5929's landed re-offer and serving path ## Related diff --git a/quest/m1/rs-request-credit.md b/quest/m1/rs-request-credit.md index 9776c235a4f..b37aeb6e238 100644 --- a/quest/m1/rs-request-credit.md +++ b/quest/m1/rs-request-credit.md @@ -37,3 +37,12 @@ wait on the port. Say in the PR which backends fail fast. Propose the error variant's name in the PR. Public API: one new `Error` variant, and possibly a `transport::poll` session method. Wire: none. + +Decided 2026-10-20 (maintainer): the `transport::poll` trait breaks land in +order, [receive memory](/quest/m1/quic/receive-memory.md) first, then this +quest, then [Scope track priority](/quest/m1/track-priority-scope.md), so +implementers rebase on one shape at a time. + +## Required + +- [Bound message assembly](/quest/m1/quic/receive-memory.md) - lands its `transport::poll` change first; this quest builds its non-waiting open on that shape diff --git a/quest/m1/sink-restart-follow.md b/quest/m1/sink-restart-follow.md index 23d4db98a40..7c7dc82213b 100644 --- a/quest/m1/sink-restart-follow.md +++ b/quest/m1/sink-restart-follow.md @@ -47,9 +47,14 @@ Audit and fix the rest the same way: - `transcode`: its source broadcast; following means restarting the ladder on the new instance. -Share one helper (a `moq_mux` `Source::follow`-style API per -[reference announcements](/quest/m0/broadcast-epoch/reference-announcements.md)) -rather than one rule per reader, built on `announced_broadcast`. Each fix -lands with a test that replaces the publisher while the reader runs. +Reuse `moq_mux::Source::follow` from +[reference announcements](/quest/m0/broadcast-epoch/reference-announcements.md), +built on `announced_broadcast`, rather than one rule per reader (decided +2026-10-20, maintainer). Each fix lands with a test that replaces the +publisher while the reader runs. -Public API: possibly a shared follow helper in `moq-mux`. Wire: none. +Public API: none. Wire: none. + +## Required + +- [Reference announcements](/quest/m0/broadcast-epoch/reference-announcements.md) - owns `Source::follow`, built on moq-net's announced request diff --git a/quest/m1/test-flakes-2/README.md b/quest/m1/test-flakes-2/README.md index ad707232fed..da2650a0c96 100644 --- a/quest/m1/test-flakes-2/README.md +++ b/quest/m1/test-flakes-2/README.md @@ -26,17 +26,12 @@ Rules every child keeps: (the draft-16 cancel race was a retry sweep jittered past a fixed settle budget). Wait out the jitter's whole window, not a count of small steps. -Added 2026-10-15 (maintainer): three flakes seen while landing that day's -merge batch, ranked by how likely each hides a product bug. - This README's own work, after the children: run `just check --all` several times on a loaded machine, as the first round did. -Added 2026-10-19 (maintainer): three more, the redial pair first since CI -hit it. The same day's test cost audit added the macOS QUIC connects and the -[redundant failover drift](/quest/m0/redundant-failover-drift.md), which moved -to m0 on 2026-10-20 (maintainer): it may hide a TS export bug and fails loaded -`just check` runs. +The [redundant failover drift](/quest/m0/redundant-failover-drift.md) moved +to m0 on 2026-10-20 (maintainer), since it may hide a TS export bug and fails +loaded `just check` runs. Added 2026-10-20 (maintainer): three more from that day's merge batch, the draft-16 cancel race first since it fails on a mocked clock and so may hide diff --git a/quest/m1/track-priority-scope.md b/quest/m1/track-priority-scope.md index fa7da515250..75f8200e613 100644 --- a/quest/m1/track-priority-scope.md +++ b/quest/m1/track-priority-scope.md @@ -31,7 +31,8 @@ Where the model stands today, all in `rs/moq-net`: - Group order within a track follows each subscription's `order` (#5385), newest first by default. - A relay forwards the max of its downstream subscriber priorities upstream - (`model/subscription.rs`, `lite/subscriber.rs`), never the publisher's track + (`model/subscription.rs`, `lite/subscriber.rs`, and on moq-transport + upstreams `ietf/subscriber.rs` since #5922), never the publisher's track priority, so one viewer asking for 255 raises that track above every other tenant's on the cluster link (`rs/moq-relay/src/cluster.rs`: one bidirectional session per peer pair carries every broadcast). The draft @@ -69,15 +70,14 @@ Direction to settle in the draft first, then the code: or narrow the goal to subscription delivery. - Rank by track priority, then subscription, then the subscription's own group order (`Subscription::order`, newest first by default). -- Decide whether the publisher's `track::Info::priority` breaks a tie - between equal subscriber priorities in `Priority::cmp` - (`rs/moq-net/src/priority.rs`). The - [ladder controller](/quest/m3/ladder/controller.md), now in m3, wants that - tiebreak; this quest owns the answer and the code change, so the controller - only consumes it (2026-10-06 audit). Either way, make the docs agree: +- Decided 2026-10-20 (maintainer): the publisher's `track::Info::priority` + breaks a tie between equal subscriber priorities in `Priority::cmp` + (`rs/moq-net/src/priority.rs`), so the same number decides what to produce + and, among equal subscriber priorities, what to send first. The + [ladder controller](/quest/m3/ladder/controller.md) only consumes it. `Info::priority` in `rs/moq-net/src/model/track.rs` already claims the - tiebreak, and the allocator doc in `rs/moq-net/src/model/bandwidth.rs` - presents send order as unrelated. + tiebreak; fix the allocator doc in `rs/moq-net/src/model/bandwidth.rs`, + which presents send order as unrelated. - A per-session cap on distinct ranks is a scheduling detail; whatever replaces the 255-entry sort must stay O(log n) per group under chat-shaped churn. - moq-quic already has the fair tier: send domains sharing bytes by deficit @@ -130,13 +130,23 @@ Signed priority, decided with the maintainer: - Look for anything that does arithmetic on priority (the lite send queue, JS send-order packing, the bandwidth allocator, the relay's max-of-subscribers) and keep its ordering, not just its type. -- moq-archive's `Info::priority` follows. Its version-1 `.info` stores the - `u8`, so existing recordings must keep their meaning: store the moq-lite - byte (`p + 128`) or bump the format version, never reinterpret silently. +- moq-archive's `.info` priority follows and breaks in place with the + archive line's other formats (decided 2026-10-20, maintainer): no format + version or compatibility mapping for recordings made before the cut. - The moxygen default (127, #4273) already shipped, so the default byte moves a second time. Add an upgrade note to the release's upgrade page saying why. Rejected: a mapping that keeps today's bytes (lite byte = p + 127). +Decided 2026-10-20 (maintainer): the `transport::poll` trait breaks land in +order, [receive memory](/quest/m1/quic/receive-memory.md), then +[Rust request credit](/quest/m1/rs-request-credit.md), then this quest, which +adds the send domain last. The publisher serve loops it rewrites already +park one group per subscription for a stalled subscriber +([moq-dev/moq#6140](https://github.com/moq-dev/moq/pull/6140)); keep that +bound. The moq-c half lands on the +[generated C package](/quest/m1/c/README.md), not the frozen hand-written +ABI, so C users break once. + Prove scoping with the existing lite publisher tests extended to two broadcasts on one session with fairness on: neither starves, a downstream 255 does not change the upstream order, and with fairness off the strict order is @@ -153,6 +163,11 @@ Public API impact: `u8` priority fields become `i8` in every language, and in format, but the default byte moves from 127 to 128 on moq-lite and from 128 to 127 on IETF. Report both with the draft change. +## Required + +- [Generated C bindings](/quest/m1/c/README.md) - the `i8` priority reaches C through the generated package, so C users break once +- [Rust request credit](/quest/m1/rs-request-credit.md) - the previous `transport::poll` break; this one adds the send domain on top + ## Related - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - owns the group diff --git a/quest/m1/ts-reliability/README.md b/quest/m1/ts-reliability/README.md index f9efc3b07fd..25da24bab03 100644 --- a/quest/m1/ts-reliability/README.md +++ b/quest/m1/ts-reliability/README.md @@ -49,46 +49,10 @@ a coverage table linking each supported profile and loss guarantee to its test; unsupported or unmeasured cells stay explicit. Keep CLI instructions and the MPEG-TS draft consistent in each feature change. -Decided 2026-10-16 by the maintainer, for private PES timing and headers -(#5718): - -- **Carriage.** A PES without a PTS is not untimed. The multiplex places - every byte on the program clock; only its presentation time is absent. - So a private PES track stays timed on the legacy container, which keeps - the timestamp in its payload and so survives transports without - net-level timestamps (IETF drafts 14-16, lite-05/06's send-time stamps), - and never becomes an untimed track. Each frame's timestamp is the PES's - source placement: the program clock at its first byte, interpolated - between the PCRs on either side (ISO 13818-1 2.4.2.2) and moved onto the - catalog clock like the media. Its payload is the whole PES packet: PTS - and DTS rebased, every other header byte kept, so PTS presence, special - stream-ID forms and optional fields round-trip without per-stream-type - timing rules. A PES that carries a PTS also gets placement, not its PTS, - as its frame timestamp; the PTS lives in the bytes. -- **Signalling.** A new `pesPacket` member of the `mpegts.tracks` entry, - beside `verbatim` rather than a new `framing` value. Old consumers ignore - an entry with neither `verbatim` nor a described track, so a published - exporter silently drops these tracks (teletext, DVB subtitles, DVB AC-3, - KLV, unknown private types) from the rebuilt TS and its PMT, with no - error, instead of refusing the whole catalog. Chosen over the loud - refusal so old exporters keep the media and sections. -- **Legacy `pes`.** The `pes` framing is removed, not kept for a release: - export refuses a catalog that names it, and the AC-3 repack goes with - it. So a current exporter refuses broadcasts from published importers - and recordings that carry private PES, and draft -01 breaks -00 there. - The deletion lands with the importer switch, so main never writes a - catalog its exporter refuses. -- **Clock fields.** Only PTS and DTS are rebased. ESCR, `pack_header_field`, - TREF and PES-level scrambling refuse their stream with a warning and an - importer health count. TS-packet-layer details (continuity, adaptation - fields, packet boundaries) stay with the raw fallback. - -Rejected: an untimed track with a container that takes time from the net -frame, which loses timing on those transports and in archive replay and -needs timedness fixed before the first PES; a per-track "no PTS" flag, -which keeps the synthesized header and misleads old exporters; and -extrapolating placement from earlier PCRs without waiting, which is wrong -on variable-rate multiplexes. +Private PES (decided 2026-10-16, #5718) landed in #5832 and #5921: a +private PES rides whole as a timed `pesPacket` track, stamped with its +program-clock placement, and the draft -00 `pes` framing is refused. The +guide documents that contract and the clock fields that refuse a stream. The semantic fidelity audit (2026-10-20) maps each source mechanism to its contract in the semantic lane. The guide's profile table starts from it; diff --git a/quest/m2/README.md b/quest/m2/README.md index 9e68f74f899..31c3f6d91c2 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -52,7 +52,7 @@ ranked just below deterministic TS legs, which it requires. - [AudioToolbox decode](/quest/m2/audio-decode-audiotoolbox.md) - macOS and iOS decode HE-AAC, multichannel AAC, and what else the framework offers - [AudioToolbox encode](/quest/m2/audio-encode-audiotoolbox.md) - macOS and iOS encode AAC-LC - [CMAF surround Opus](/quest/m2/cmaf-opus-surround.md) - a released mp4-atom carries the `dOps` channel mapping table, and fMP4 import and export carry it through the OpusHead -- [GPU CI](/quest/m2/gpu-ci.md) - NVIDIA tests run nightly on real hardware, and `just rs nvidia` runs them locally instead of skipping +- [GPU CI](/quest/m2/gpu-ci.md) - NVIDIA tests gate the merge queue on a self-hosted GPU runner VM, and `just rs nvidia` runs them locally instead of skipping - [VA-API renderer tests](/quest/m2/vaapi-render-tests.md) - the ignored VA-API renderer import tests run in `just rs gpu` on AMD and Intel and fail instead of skipping - [Vendor-free CARLA bridge](/quest/m2/carla-bridge.md) - the CARLA demo's GPU bridge lets moq-video pick the encoder, and its stand-in test runs on AMD as well as NVIDIA - [V4L2 virtual camera](/quest/m2/v4l2-virtual-camera.md) - V4L2 capture runs nightly against the `vivid` virtual driver instead of only on a developer's camera @@ -61,8 +61,9 @@ ranked just below deterministic TS legs, which it requires. - [A finished PROBE stops reporting](/quest/m2/probe-end-stale.md) - the Rust receive estimate reads unknown once its PROBE stream ends - [Wire stats](/quest/m2/wire-stats/README.md) - `Presence` counts each root's transport bytes on the wire, `wire_tx` and `wire_rx`, from QUIC, WebSocket, and embedder gateway sources - [Media stats](/quest/m2/stats/README.md) - publishers announce a stats track in the catalog, viewers answer a soliciting catalog through a per-catalog `.echo` broadcast, and one model turns both into a health verdict and a preflight report +- [Group storage benchmark](/quest/m2/group-storage-bench.md) - numbers pick the append-only log, a ring buffer, or sorted runs for `GroupState` before either lands - [Perf](/quest/m2/perf/README.md) - eliminate measured hot-path costs across moq-uring, kio, and the moq-net model -- [Benchmark regressions in CI](/quest/m2/bench-ci.md) - PRs get a non-blocking comparison of the Criterion benches they affect, and a nightly trend on main alerts on regressions +- [Nightly benchmark regressions](/quest/m2/bench-ci.md) - each nightly compares the Criterion benches against the previous nightly's commit, publishes it to the git.moq.dev report, and alerts on a regression - [Relay session bench](/quest/m2/bench-relay.md) - the same scenario through moq-relay's own connection handling - [Transport x version benches](/quest/m2/bench-transport-sweep.md) - the session benches sweep transports as well as versions - [Read-only lookup](/quest/m2/read-only-lookup.md) - a live track lookup stops waking the front and every demand watcher @@ -75,8 +76,8 @@ ranked just below deterministic TS legs, which it requires. - [IETF lazy discovery](/quest/m2/ietf-lazy-discovery.md) - moq-transport sessions ask for announcements only under watched prefixes, starting with UNSUBSCRIBE_NAMESPACE on d14/15 - [Keyed prefix polling](/quest/m2/keyed-prefix-polling.md) - `SubscriberDriver` polls only the prefixes a wake concerns, if a bench shows the slope - [Route health](/quest/m2/route-health/README.md) - epoch-only active/passive track recovery over metadata-only subscriptions +- [Wrapper traits over moq-net](/quest/m2/net-wrap/README.md) - moq-net exports producer and consumer traits, and every writer and reader above it takes them, so a wrapper such as e2ee passes through unchanged - [E2EE](/quest/m2/e2ee/README.md) - TypeScript and Rust peers interoperate over encrypted broadcasts no relay can decrypt -- [Wrapper seam over moq-net](/quest/m2/net-wrap-trait.md) - a reviewed decision on how e2ee, json, and future wrappers sit between any moq-net writer or reader and the network, trait versus bridge - [iOS capture](/quest/m2/mobile-capture-ios.md) - Rust captures the camera and screen on iOS, setting the catalog rotation - [Android capture](/quest/m2/mobile-capture-android.md) - Rust captures through NDK/JNI on Android, reusing the existing codecs and setting the catalog rotation - [Time stretch](/quest/m2/watch-audio-time-stretch.md) - js/watch: the audio ring converges by time-stretching instead of skipping or going silent @@ -100,7 +101,11 @@ ranked just below deterministic TS legs, which it requires. - [QUIC caps](/quest/m2/rate-quic.md) - paced MAX_DATA credit in and a capped pacer out hold a QUIC session to its token's bitrate - [WebSocket caps](/quest/m2/rate-websocket.md) - paced reads and writes over bounded socket buffers hold WebSocket to the same caps - [Publishers learn their cap](/quest/m2/rate-grant.md) - the AUTH grant carries the caps and publishers clamp their encoder to them +- [moq-rtc REMB and TIAS](/quest/m2/rtc-remb-tias.md) - a capped WHIP session steers the browser's encoder under its cap and closes past it +- [moq-rtc pacing](/quest/m2/rtc-pacing.md) - a capped WHEP viewer never receives more than its cap plus burst, NACK resends included +- [SRT rate handle](/quest/m2/srt-rate-handle.md) - a per-connection cap paces SRT pulls, retransmits included, and closes SRT publishers past it - [One port on the io_uring workers](/quest/m2/uring-demux.md) - `moq-uring`'s workers host the UDP demux, so a ring relay keeps WebRTC media and SRT on its QUIC port +- [Hang contribution references](/quest/m2/contribution-ref.md) - a catalog names a whole contribution broadcast that a player opens only when selected, backing moq.pro's composed rooms - [Synced data playback](/quest/m2/watch-data-sync.md) - js/watch releases JSON and binary payloads on the media playhead, and a slow data track holds media back - [Watch decode gate](/quest/m2/watch-decode-gate.md) - video lookahead stays encoded until it is near presentation, like audio - [Drop announce="never"](/quest/m2/publish-announce-never.md) - `@moq/publish` offers only announce modes a watcher can reach @@ -185,7 +190,6 @@ ranked just below deterministic TS legs, which it requires. - [H.264 parsing with maintained crates](/quest/m2/h264-reader.md) - moq-mux parses SPS VUI with h264-reader and shares one bit reader across codec config parsers - [avcC and hvcC through mp4-atom](/quest/m2/mp4-atom-config.md) - moq-mux stops hand-writing the decoder config records its mp4-atom dependency already parses - [Relay metrics through prometheus-client](/quest/m2/relay-prometheus-client.md) - `/metrics` renders through the maintained client with the same series -- [Dead routes on IETF name their cause](/quest/m2/js-ietf-dead-route-error.md) - a JS subscription on a dead moq-transport session ends with the session error, not `unauthorized` - [IETF request order](/quest/m2/ietf-request-order.md) - d14/15 requests take their ID after opening, legacy `control.sending()` races are gone, and SUBSCRIBE_NAMESPACE never closes over its own close - [JS IETF request streams](/quest/m2/js-request-stream.md) - `RequestStream` owns deadlines and cancellation, and TRACK trailing bytes close the session like Rust - [Error cleanups](/quest/m2/error-cleanups.md) - delete the unused `Error::SessionClosed`; raw transport failures already map to `Unroutable` diff --git a/quest/m2/audio-decode-mediacodec.md b/quest/m2/audio-decode-mediacodec.md index 83d612ebb36..a4ca657d736 100644 --- a/quest/m2/audio-decode-mediacodec.md +++ b/quest/m2/audio-decode-mediacodec.md @@ -26,6 +26,10 @@ behind a new optional audio `mediacodec` feature and the decode seam, on `target - The binding ships in the moq-ffi Android slice, which is how Kotlin and Dart reach it. +## Required + +- [MediaCodec on an emulator](/quest/m2/android-mediacodec-ci.md) - the emulator lane this runtime proof runs in + ## Related - [Android capture](/quest/m2/mobile-capture-android.md) - the video MediaCodec family this sits beside diff --git a/quest/m2/audio-encode-mediacodec.md b/quest/m2/audio-encode-mediacodec.md index 1220119513b..9b785956a12 100644 --- a/quest/m2/audio-encode-mediacodec.md +++ b/quest/m2/audio-encode-mediacodec.md @@ -25,5 +25,6 @@ behind the `mediacodec` feature and the encode seam. ## Required +- [MediaCodec on an emulator](/quest/m2/android-mediacodec-ci.md) - the emulator lane this runtime proof runs in - [MediaCodec decode](/quest/m2/audio-decode-mediacodec.md) - the round-trip regression decodes through it - [AudioToolbox encode](/quest/m2/audio-encode-audiotoolbox.md) - owns the shared flush and zero-or-more packet encoder contract diff --git a/quest/m2/bench-ci.md b/quest/m2/bench-ci.md index 8534238356e..8ddb2020fc4 100644 --- a/quest/m2/bench-ci.md +++ b/quest/m2/bench-ci.md @@ -1,54 +1,51 @@ -# [M] Benchmark regressions in CI +# [M] Nightly benchmark regression report ## Goal -A pull request that touches a benched Rust crate, or anything it depends on, -gets a sticky comment comparing its Criterion benchmarks against the base. The -comment never fails the PR. A nightly run on `main` keeps a history of every -benchmark and fails when one regresses past a loose threshold against the -previous nightly's commit measured on the same runner, so the existing Alert -workflow posts it to Discord. +Each Nightly run on `main` compares every measured Rust Criterion benchmark +against the previous nightly's commit, built and run on the same runner, and +publishes the comparison to the git.moq.dev nightly report beside the +[regression table](https://git.moq.dev/bench/). A regression past a loose threshold +fails the job, so the existing Alert workflow posts it to Discord. -Rust Criterion targets only. Browser benchmarks stay under -`just bench-browser` (see `bench/README.md`), and relay load stays local under -`just bench BASE`. +Benchmarks never run in the merge queue or on PRs. Rust Criterion targets +only: browser benchmarks stay under `just bench-browser` (see +`bench/README.md`), and relay load stays local under `just bench BASE`. ## Plan +Decided 2026-10-20 (maintainer): benchmarks go into the git.moq.dev report +each night, not the merge queue, because the queue runs only what gates a +merge, and main PRs skip cloud CI since #6038. This drops the per-PR sticky +comment design and its `workflow_run` follow-up. + Researched 2026-09: CodSpeed's instruction counting skips every `iter_custom` bench (all of `moq-uring`, plus `moq-net` origin and track) and -can't see syscall time. Bencher compares a PR against history recorded on other -VMs, so its thresholds fire on runner noise. The chosen shape, free and without -a GitHub App: - -- PR: one job builds base and head on the same runner and runs only the - selected targets, saving and comparing Criterion baselines. Selection is the - changed crates plus their dependents, from the impact map in - `sh/dispatch.sh`. No selected bench - means no job. Extend `bench/run.sh` with a Criterion-only, crate-scoped mode - behind a recipe instead of writing a second runner. Hosted runners vary by - about 3%, so the comment highlights only changes Criterion calls - significant and beyond a noise floor you choose from A/A runs. Fork PRs - have a read-only token, so posting needs a `workflow_run` follow-up. -- Nightly: build the previous nightly's commit beside `main` on the same - runner and fail only on that paired comparison. Each hosted run is a fresh - VM, so the cross-run history is subject to the same noise as Bencher and - stays informational: record it with - [github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark) - on the [regression table](https://git.moq.dev/bench/)'s `gh-pages` branch, with its - own alert disabled. The `moq-uring` socket - benches run only here, and skip loudly when the runner's kernel is too old. Add the workflow's name to - `alert.yml` if it isn't `Nightly`. -- Decided 2026-10-16: moq-quic's Criterion targets (#5812) join the - selection. They time thread CPU rather than wall time, which already cut - most host noise; on a shared hybrid-core host, identical builds still - swing about 5% on small cases and 20-130% on large-working-set ones, so - the noise floor must come from A/A runs on the CI runner itself, never - from a laptop. +can't see syscall time. Bencher compares against history recorded on other +VMs, so its thresholds fire on runner noise. + +- Paired comparison: build the previous nightly's commit beside `main` on the + same runner, run the selected targets for both, and compare saved + Criterion baselines. Only that paired comparison fails the job; the + cross-run history stays informational on the regression table. +- Extend `bench/run.sh` and `sh/gh/nightly.sh` (`just gh nightly rust-bench`, + `rust-extra`, and `rust-uring`) with the paired mode instead of writing a second runner, and publish its + table through `just gh benchmark-report` into the report. `nightly.sh` + exits on a failure before it publishes today; publish first, so the table + exists when the regression alert fires. +- The nightly jobs measure in Criterion's quick mode for runtime; the paired + run needs enough samples for its threshold. Weigh a full run of the + compared targets against the nightly's time budget. +- Decided 2026-10-16: moq-quic's Criterion targets (#5812) are included. + They time thread CPU rather than wall time, which already cut most host + noise; on a shared hybrid-core host, identical builds still swing about 5% + on small cases and 20-130% on large-working-set ones, so the noise floor + must come from A/A runs on the CI runner itself, never from a laptop. +- The `moq-uring` socket benches skip loudly when the runner's kernel is too + old. - Validate with an A/A run (the same commit twice) and a deliberately slowed - bench before trusting either signal. -- Document the comment, the trend, and local reproduction in - `bench/README.md`. + bench before trusting the signal. +- Document the report and local reproduction in `bench/README.md`. ## Related diff --git a/quest/m2/bench-sources.md b/quest/m2/bench-sources.md index 49b1195a10a..414e66dac24 100644 --- a/quest/m2/bench-sources.md +++ b/quest/m2/bench-sources.md @@ -25,7 +25,7 @@ don't already emit one. - The relay load benches (`bench-audio`, `bench-profile`) and the audio-quality job's benchmarks can join later through the same input format. -- [Benchmark regressions in CI](/quest/m2/bench-ci.md) records its nightly +- [Nightly benchmark regression report](/quest/m2/bench-ci.md) records its nightly Criterion history on the same `gh-pages` branch instead of creating its own. Public API: none. Wire: none. diff --git a/quest/m2/capture-linux.md b/quest/m2/capture-linux.md index ee7993bfcf2..6b893de31ec 100644 --- a/quest/m2/capture-linux.md +++ b/quest/m2/capture-linux.md @@ -16,14 +16,9 @@ windows are reachable by explicit id even under Wayland. The portal path is what is left, and it is different in kind rather than merely unimplemented, because xdg-desktop-portal owns source selection: -- **Portal-picked windows reach the Rust API but not the CLI.** #5089 added - `capture::Source::Portal(portal::Selection)` with `portal::Kind::{Screen, - Window}`, so a caller can ask the picker for a window, and its restore token - stays with that selection. The portal still owns which window the user - picks, so there is no id to hand back. `moq` only takes an explicit window - id; wire a picker-driven window selection into the CLI (for example, a bare - `--window` going to the portal, mirroring an unqualified `--display`) and - update `doc/bin/cli.md`. +- Portal-picked windows reach the CLI: #5089 added + `capture::Source::Portal(portal::Selection)`, and #5566 sends a bare + `--window` to the portal (`doc/bin/cli.md`). - **Display selection stays picker-driven.** An unqualified `--display` still goes to the portal and ignores the selector. Since #5089 a `Selection` replays its restore token across reopens within one process, but the token is never diff --git a/quest/m2/carla-bridge.md b/quest/m2/carla-bridge.md index 65e0231f49b..87d55eaff81 100644 --- a/quest/m2/carla-bridge.md +++ b/quest/m2/carla-bridge.md @@ -26,8 +26,9 @@ probe opens the camera's own backend with a CPU picture. - Selection: switch `stream.rs` `config` from `Kind::Named("nvenc")` to `Kind::Auto`. A device no backend imports must be refused, never sent to - openh264; the workspace unifies `openh264` on, so check that refusal in a - test. + openh264. #5771 made openh264 opt-in, but the hand-written moq-c still + enables it, so a workspace build can unify it on; check that refusal in a + test built with `openh264`. - Probe: a CPU picture only works on a backend that also takes CPU input. Probe with the camera's surface type, or take parameter sets from the first keyframe instead, whichever backend picks. diff --git a/quest/m2/contribution-ref.md b/quest/m2/contribution-ref.md new file mode 100644 index 00000000000..40c074065a2 --- /dev/null +++ b/quest/m2/contribution-ref.md @@ -0,0 +1,56 @@ +# [M] Hang contribution references + +## Goal + +A Hang root catalog can name a whole contribution broadcast, relative to its +own path, without opening it: each guest feed in a composed room, for +example. Rust and JavaScript catalogs carry the entry, and `@moq/watch` +resolves one only when a consumer selects one of its renditions. This backs +moq.pro's overlay contribution reference, +whose composer never reads a contribution's catalog because subscribing is +the demand that starts the encoder. + +## Plan + +Decided 2026-10-20 (maintainer): an upstream quest backs this work, because +#5722 and #5298 closed unmerged for lack of one. Only the catalog-level +reference: the processor media contract that once carried it (#4946) stays +deleted. + +Salvage `refs/pull/5722/head` ("feat(hang): add whole contribution +references") and its decision notes rather than starting over: + +- A root `contributions` map, `{ [id]: { broadcast, epoch? } }`. `broadcast` + is a relative reference with the usual escape check, like the + per-rendition `broadcast` reference in `rs/hang/src/catalog/video/mod.rs` + and its audio, text, JSON, and binary counterparts. +- Listing an entry subscribes to nothing. `@moq/watch` opens one only + through `Broadcast.contribution(effect, id)`, a child broadcast scoped to + the caller's effect, resolved through the parent's `relativeBroadcast`. + The child's rendition IDs stay its own, so a contribution's rendition + never shadows a source rendition with the same ID. +- `epoch` follows the reference epochs that landed in + [#5699](https://github.com/moq-dev/moq/pull/5699): the same validation, known-mismatch refusal, and epochless best effort. A + lazy contribution has no anticipated epoch, so a composer leaves it out + until the contribution resolves. A reader that follows announcements + follows the contribution's Start, End, and Restart; one that follows none + pins the first resolution for its parent catalog instance. Reuse + `moq_mux::Source`'s pinning and the `Source::follow` from + [reference announcements](/quest/m0/broadcast-epoch/reference-announcements.md); + do not add a second one. +- #5722 was stacked on #5699's reference `epoch`, which has landed; rebuild + it on `main`. + +Per cross-package sync, land `rs/hang`, `js/hang`, `js/watch`, and +`drafts/draft-lcurley-moq-hang.md` in one PR. Cover lazy resolution (listing +the entry opens nothing), a missing contribution, a malformed entry refused, +a relative path that escapes, an ID collision with a source rendition, and a +contributor restart. + +Public API: a catalog `contributions` section in `hang` and `@moq/hang`, and +`Broadcast.contribution` in `@moq/watch`. Wire: a new optional catalog +field. + +## Required + +- [Reference announcements](/quest/m0/broadcast-epoch/reference-announcements.md) - owns `Source::follow`, which a contribution's announcement followers reuse diff --git a/quest/m2/e2ee/README.md b/quest/m2/e2ee/README.md index 5de9378fe31..656365acc0f 100644 --- a/quest/m2/e2ee/README.md +++ b/quest/m2/e2ee/README.md @@ -33,8 +33,8 @@ The Rust and TypeScript cores expose the same surface, and nothing else: - `Credential { context, kid, secret }` accepts the application-owned secret. The application generates and distributes it over its authenticated channel; the library does not mint a secret it cannot return. `credential.path(semantic)` derives the epoch-free opaque broadcast name. Credential is cheap to clone, never serializes the secret, and redacts it from `Debug`. - A core produces only under an epoch it minted itself (decided 2026-10-08). Only `credential.produce()` mints a producing generation, and it takes no epoch: it mints a fresh UUIDv7 and returns a `generation::Producer` owning `epoch()` for the route and `name(semantic)` for opaque track names. `credential.consume(epoch)` binds a discovered epoch as a consume-only `generation::Consumer` with `name(semantic)`. The two are separate types, so producing under a discovered or shared epoch does not compile (names decided 2026-10-11 in [#5313](https://github.com/moq-dev/moq/pull/5313)). Clones of a producing generation share publisher claims so reopening a track cannot reset its nonce counters. -- Applications only handle semantic track names (maintainer, 2026-10-17). `broadcast::Producer` wraps a `moq-net` broadcast under a producing generation with `create_track(semantic, info)` and `reserve_track(semantic)`; `broadcast::Consumer` wraps one under a consuming generation with `subscribe(semantic, subscription)`. Each derives the physical name, so only the CDN sees physical names. `generation` no longer opens tracks itself. -- `track::Producer` appends groups and datagrams and allocates identities; `track::Consumer` yields groups and datagram events. `group::Producer` and `group::Consumer` wrap the whole group lifecycle so every AEAD call has the canonical physical name and transport identity. +- Applications only handle semantic track names (maintainer, 2026-10-17). `broadcast::Producer` wraps a `moq-net` broadcast under a producing generation and `broadcast::Consumer` wraps one under a consuming generation; each derives the physical name, so only the CDN sees physical names. `generation` no longer opens tracks itself. The broadcast wrappers merged in [#5876](https://github.com/moq-dev/moq/pull/5876). +- The wrappers implement the [moq-net traits](/quest/m2/net-wrap/README.md) at the broadcast, track, group, and frame levels, datagrams included (decided 2026-10-20, maintainer, because one encrypted broadcast then works with every writer and reader). moq-mux, moq-json, hang, and the CLI publish and consume an encrypted broadcast unchanged, under semantic names, and every AEAD call still has the canonical physical name and transport identity. The e2ee mux bridge ([#5822](https://github.com/moq-dev/moq/pull/5822)) is closed and superseded. - Errors are the draft's typed codes plus the transport's. Nothing catalog-, hang-, or MSF-shaped lives here: a catalog is a track under a derived name, and compression is the catalog owner's job. - Stateless `seal`/`open` primitives with caller-chosen identities, HKDF labels and info builders, raw key bytes, and process-global claims are not public. The vectors are tested inside each core. @@ -46,18 +46,17 @@ The Rust and TypeScript cores expose the same surface, and nothing else: ### Dynamic names -Decided with the maintainer on 2026-10-17: physical names are a one-way hash, so a publisher cannot answer a request for a name it never declared. [Dynamic names](/quest/m2/e2ee/dynamic-names.md) designs that before any dynamic-name implementation. The broadcast wrappers merged in [#5876](https://github.com/moq-dev/moq/pull/5876) with `reserve_track`, which covers names declared in advance; that is every track the Rust publishers serve, so [rust-publish](/quest/m2/e2ee/rust-publish.md) does not wait on the design (maintainer, 2026-10-19). [Reopen](/quest/m2/e2ee/reopen.md) and the [consumer shape](/quest/m2/e2ee/consumer.md) still do. +Decided with the maintainer on 2026-10-17: physical names are a one-way hash, so a publisher cannot answer a request for a name it never declared. [Dynamic names](/quest/m2/e2ee/dynamic-names.md) designs that before any dynamic-name implementation. The broadcast wrappers merged in [#5876](https://github.com/moq-dev/moq/pull/5876) with `reserve_track`, which covers names declared in advance; that is every track the Rust publishers serve, so [rust-publish](/quest/m2/e2ee/rust-publish.md) does not wait on the design (maintainer, 2026-10-19). The [consumer shape](/quest/m2/e2ee/consumer.md) still does. ## Required -- [Rust protected publisher seams](/quest/m2/e2ee/rust-publish.md) - Rust media - and catalog publishers accept opaque physical names and emit no plaintext - semantic catalog in E2EE mode - [Dynamic names](/quest/m2/e2ee/dynamic-names.md) - a reviewed design for answering requests for undeclared track names, split into implementation quests -- [Refuse a re-created track](/quest/m2/e2ee/reopen.md) - a name the producer already carried fails loudly -- [Encrypted consumer shape](/quest/m2/e2ee/consumer.md) - one consumer shape, the producer's mirror, shared by browser, CLI, and FFI +- [Rust protected publisher seams](/quest/m2/e2ee/rust-publish.md) - Rust + publishers emit encrypted catalogs and no plaintext semantic catalog in + E2EE mode, through the moq-net traits +- [Encrypted consumer shape](/quest/m2/e2ee/consumer.md) - one consumer shape, the producer's mirror, shared by browser and CLI (moq-ffi is out of scope for this epic) - [Encrypted browser components](/quest/m2/e2ee/browser.md) - browser publish and watch components handle encrypted audio and video without leaking credentials into browser-owned surfaces @@ -75,4 +74,4 @@ Decided with the maintainer on 2026-10-17: physical names are a one-way hash, so ## Related - [archive](/quest/m1/archive/README.md) - protected broadcasts are deliberately outside recording and replay formats -- [Wrapper seam over moq-net](/quest/m2/net-wrap-trait.md) - whether a moq-net trait replaces the bridge the e2ee mux path uses +- [Wrapper traits over moq-net](/quest/m2/net-wrap/README.md) - the traits the e2ee wrappers implement, so every writer and reader takes an encrypted broadcast diff --git a/quest/m2/e2ee/browser.md b/quest/m2/e2ee/browser.md index 501974184d8..27e4edec344 100644 --- a/quest/m2/e2ee/browser.md +++ b/quest/m2/e2ee/browser.md @@ -8,7 +8,7 @@ Existing plaintext operation remains the default; an explicit application creden ## Plan -- Integrate the TypeScript E2EE layer before the existing Hang publish and after the existing Hang subscribe boundaries, keeping codec selection, synchronization, and rendering in the post-decryption pipelines. +- Hand `@moq/publish` and `@moq/watch` an encrypted broadcast through the [`@moq/net` interfaces](/quest/m2/net-wrap/js.md), keeping codec selection, synchronization, and rendering in the post-decryption pipelines. - Accept keys through a programmatic property/API only, never an HTML attribute, URL, persistent browser storage, analytics event, or log. Clear owned secret bytes when the component is replaced or disconnected; do not make extractability a hidden requirement. - Derive the invariant opaque broadcast path from the semantic name and resolve that path, taking the selected epoch from its route. On a wire diff --git a/quest/m2/e2ee/cli.md b/quest/m2/e2ee/cli.md index a6d004efc97..569277a17c8 100644 --- a/quest/m2/e2ee/cli.md +++ b/quest/m2/e2ee/cli.md @@ -8,7 +8,7 @@ Existing plaintext commands remain the default; an explicit application credenti ## Plan -- Integrate the Rust E2EE layer into `moq-cli` publication and playback before semantic mux output and after protected track input. Keep codec, capture, playback, and synchronization logic outside the crypto layer. +- Hand the [trait-generic CLI paths](/quest/m2/net-wrap/cli.md) an encrypted broadcast for publication and playback. Keep codec, capture, playback, and synchronization logic outside the crypto layer. - Accept the credential through a dedicated file descriptor, otherwise-unused stdin, or a permission-checked file (`0600` on Unix), never a command argument or environment variable. Redact errors and tracing, zeroize owned secret bytes, and document shell-safe invocation. - Follow the approved protected naming contract for mux, video, audio, timeline, and catalog construction. Take the semantic broadcast name from diff --git a/quest/m2/e2ee/consumer.md b/quest/m2/e2ee/consumer.md index 55ab44d3c8d..0292488606a 100644 --- a/quest/m2/e2ee/consumer.md +++ b/quest/m2/e2ee/consumer.md @@ -3,24 +3,33 @@ ## Goal Encrypted playback has one consumer shape, the mirror of the protected -producer, that the browser, the CLI, and the FFI bindings share, so each -player does not reinvent decryption and physical-name mapping. +producer, that the browser and the CLI share, so each player does not +reinvent decryption and physical-name mapping. ## Plan -#5822's CLI plan names the consumer as "the consuming mirror of that -producer in `moq-mux`", but nothing gives it a shape the browser and FFI -can mirror. Mock up the Rust shape first, then mirror the names in JS and -the bindings. Kept in m2 (decided 2026-10-17). +Decided 2026-10-20 (maintainer): the shape is the moq-net consumer traits. +`moq_e2ee::broadcast::Consumer` and its track and group consumers implement +them ([net-wrap](/quest/m2/net-wrap/README.md)), so a Rust player, the CLI, +and `@moq/watch` open an encrypted broadcast as they open any other, +because every reader already takes the traits. Kept in m2 (decided +2026-10-17). -The moq-ffi consumer follows the `rs/moq-ffi` row of the root `AGENTS.md` -cross-package table: wrappers, C and C++, OBS where it plays, and the -language docs land with it, or the FFI part waits for a quest that does. +What remains here: -Public API: a new e2ee consumer in `moq-mux`, `@moq/e2ee`, and moq-ffi with -its wrappers. Wire: none. +- Resolve the opaque path, take the epoch from its route (or from the + application on a wire without route epochs), and bind it with + `credential.consume(epoch)` in one call each player shares, in Rust and JS + with matching names. +- Decrypt the protected catalog before the existing selection. +- Rust and JS only (decided 2026-10-20, maintainer): the e2ee publisher has + no moq-ffi half either, so an FFI consumer waits for a quest that adds both. + +Public API: an e2ee consumer entry point in `moq-e2ee` and `@moq/e2ee`. +Wire: none. ## Required -- [Rust protected publisher seams](/quest/m2/e2ee/rust-publish.md) - the producer this mirrors - [Dynamic names](/quest/m2/e2ee/dynamic-names.md) - the approved naming design can change the physical-name mapping this fixes +- [Wrapper traits over moq-net](/quest/m2/net-wrap/README.md) - Rust and JS readers take the traits the e2ee consumer implements +- [Rust protected publisher seams](/quest/m2/e2ee/rust-publish.md) - the producer this mirrors diff --git a/quest/m2/e2ee/dynamic-names.md b/quest/m2/e2ee/dynamic-names.md index 1116777c747..fcebc3df2bd 100644 --- a/quest/m2/e2ee/dynamic-names.md +++ b/quest/m2/e2ee/dynamic-names.md @@ -23,9 +23,12 @@ Decided with the maintainer on 2026-10-17: - The Rust publishers do not wait on it (maintainer, 2026-10-19): the broadcast wrappers merged in [#5876](https://github.com/moq-dev/moq/pull/5876), and [rust-publish](/quest/m2/e2ee/rust-publish.md) declares every track it - serves. [Reopen](/quest/m2/e2ee/reopen.md) and the - [consumer shape](/quest/m2/e2ee/consumer.md) still wait, since the chosen - mapping can change the physical names they handle. + serves. The [consumer shape](/quest/m2/e2ee/consumer.md) still waits, + since the chosen mapping can change the physical names it handles. +- A dynamic name answers through the + [moq-net traits](/quest/m2/net-wrap/README.md) the wrappers implement + (`dynamic`, `track::Request`), so the design keeps their methods and + changes only the name mapping behind them (2026-10-20). Background: a physical track name is a one-way HKDF of the semantic name (the draft's Key Derivation section). Only the CDN sees physical names, and a diff --git a/quest/m2/e2ee/reopen.md b/quest/m2/e2ee/reopen.md deleted file mode 100644 index 3f916a905ef..00000000000 --- a/quest/m2/e2ee/reopen.md +++ /dev/null @@ -1,21 +0,0 @@ -# [XS] A re-created encrypted track is refused, not dropped - -## Goal - -Publishing a track under a name the e2ee producer already carried fails -loudly instead of being silently left out. - -## Plan - -`moq-mux`'s e2ee producer carries each name at most once, and `moq-e2ee` -never releases a claim because reopening would restart its nonce counters, -so a re-created track today is just not carried. Decided 2026-10-17 -(maintainer): refuse it. A track name always means the same content, so a -restarted encoder publishes a new name or a new broadcast. - -Public API: a new error from the e2ee producer. Wire: none. - -## Required - -- [Rust protected publisher seams](/quest/m2/e2ee/rust-publish.md) - the producer that claims names -- [Dynamic names](/quest/m2/e2ee/dynamic-names.md) - the approved naming design can change the physical-name mapping this fixes diff --git a/quest/m2/e2ee/rust-publish.md b/quest/m2/e2ee/rust-publish.md index 00eeae5805c..3c8e7581b28 100644 --- a/quest/m2/e2ee/rust-publish.md +++ b/quest/m2/e2ee/rust-publish.md @@ -1,22 +1,47 @@ -# [L] Rust protected publisher seams +# [M] Rust protected publisher seams ## Goal -Rust media and catalog publishers accept opaque physical names and emit no plaintext semantic catalog in E2EE mode. +Rust media and catalog publishers emit no plaintext semantic catalog in E2EE +mode, and a track name the e2ee producer already carried is refused loudly +rather than silently left out. -The reusable crypto crate remains independent of Hang and codec-specific publisher policy. +The reusable crypto crate remains independent of Hang and codec-specific +publisher policy. ## Plan Publishers use `Credential::produce()` and its `generation::Producer`; the producing/consuming split landed separately in #5313. -- Adapt mux, video, audio, timeline, and catalog publishers to the e2ee - broadcast wrappers (`create_track`, `reserve_track`). Every track these - publishers serve is declared in advance, so this does not wait on the - [dynamic names](/quest/m2/e2ee/dynamic-names.md) design (maintainer, - 2026-10-19). Applications keep semantic track names; the wrappers own - their translation to physical names, as the epic specifies. -- Derive the protected catalog name from its well-known logical role, then put media roles, codecs, quality, timelines, and custom-track mappings only inside the encrypted catalog. -- Encrypt both Hang catalog representations and encrypt or suppress the MSF catalog. Compress protected catalog bodies before encryption and retain ordinary plaintext publication unchanged outside E2EE mode. -- Cover every current Rust importer and capture path so a convenience API cannot silently recreate `.avc3`, `.opus`, timeline, or other semantic suffixes. +Decided 2026-10-20 (maintainer): the e2ee wrappers implement the +[moq-net traits](/quest/m2/net-wrap/README.md), and moq-mux and the +moq-video and moq-audio encoders take them, so mux, video, audio, and +timeline publishers already publish into an encrypted +broadcast under semantic names with no change here. This quest keeps only +what the traits do not cover. Every track these publishers serve is declared +in advance, so it does not wait on the +[dynamic names](/quest/m2/e2ee/dynamic-names.md) design (maintainer, +2026-10-19). + +- Catalog naming: derive the protected catalog name from its well-known + logical role, then put media roles, codecs, quality, timelines, and + custom-track mappings only inside the encrypted catalog. +- Encrypt both Hang catalog representations and encrypt or suppress the MSF + catalog. Compress protected catalog bodies before encryption and retain + ordinary plaintext publication unchanged outside E2EE mode. +- Cover every current Rust importer and capture path so a convenience API + cannot emit a plaintext semantic catalog. +- Refuse a re-created track (decided 2026-10-17, maintainer): moq-e2ee + never releases a name's claim, because reopening would restart its nonce + counters, so a track created again under a name the producer already + carried fails with a new error instead of being dropped. A track name + always means the same content, so a restarted encoder publishes a new name + or a new broadcast. Test it through moq-mux. + +Public API: a new error from the e2ee producer. Wire: none. + +## Required + +- [moq-e2ee implements the traits](/quest/m2/net-wrap/e2ee.md) - the encrypted broadcast moq-mux publishes into +- [moq-mux takes the traits](/quest/m2/net-wrap/mux.md) - the mux and capture publishers this relies on to stay unchanged diff --git a/quest/m2/ffi-size-profile.md b/quest/m2/ffi-size-profile.md index 665b0069e2b..6e6a966d93a 100644 --- a/quest/m2/ffi-size-profile.md +++ b/quest/m2/ffi-size-profile.md @@ -20,8 +20,9 @@ Guidance: everything at "s". - Measure what the bindings actually do: publish and subscribe throughput through moq-ffi on the [binding benchmark](/bench/README.md), plus - audio and video encode and decode. Wire the benchmark into CI, at least - nightly. + audio and video encode and decode. The binding benchmark already runs + nightly and feeds the [regression table](https://git.moq.dev/bench/); add the + "s" build as a series there. - If it's adopted, use a named profile (for example `[profile.ffi]`, inheriting release) that every binding build path selects, so the relay and self-builders keep opt-level 3. diff --git a/quest/m2/gpu-ci.md b/quest/m2/gpu-ci.md index 9d6f81995cf..1b29133d2b8 100644 --- a/quest/m2/gpu-ci.md +++ b/quest/m2/gpu-ci.md @@ -1,9 +1,9 @@ -# [S] NVIDIA tests run on real hardware +# [M] NVIDIA tests gate the merge queue on real hardware ## Goal -The NVDEC, NVENC, and CUDA tests run nightly on real NVIDIA hardware instead -of passing without a GPU on hosted runners, and a local +The NVDEC, NVENC, and CUDA tests gate the merge queue on real NVIDIA +hardware instead of passing without a GPU on hosted runners, and a local `just rs nvidia` runs them against the host driver instead of silently skipping inside the Nix shell. @@ -43,15 +43,20 @@ skipping inside the Nix shell. `vulkan_cuda_auto_external_encode` must pass its per-plane pixel checks against the CPU reference for both renditions, RGBA and BGRA, through NVENC and CUDA external-image import (the verification #4975 deferred here). -- Nightly: a job in `.github/workflows/nightly.yml` runs `nix develop - --command just rs nvidia`, a recipe and not a script path, like every other - workflow step: only `schedule` and `workflow_dispatch`, gated to - `refs/heads/main`, never `pull_request`, with read-only `permissions`. -- Open: where the GPU tests run. The self-hosted host this quest planned on - was dropped on 2026-10-13 with self-hosted CI. Recommendation: a - GitHub-hosted Linux GPU larger runner for the nightly job, measured for - cost, since larger runners bill even on public repositories; local - `just rs nvidia` runs cover development until then. +- Merge queue: a `merge_group` job in `.github/workflows/check.yml` (or + `video.yml`) runs `nix develop --command just rs nvidia`, a recipe and not + a script path, like every other workflow step, on a self-hosted runner VM + labeled for the NVIDIA host. Scope it like the other queue jobs: run when + the queue's diff touches moq-video, moq-nvenc, or what they depend on. + Read-only `permissions`, never `pull_request`. +- Decided 2026-10-20 (maintainer): GPU tests gate in the merge queue on the + NVIDIA card, because they gate correctness, not trends. The runner is a + persistent VM from the `github-runner-vm` module (#6040, + `nix/runner/README.md`). That module has no GPU passthrough today: add + VFIO passthrough of the NVIDIA card to the guest, with the driver in the + guest, and keep the runner registered only to this repository with + approval required for every external contributor's workflow, as the + README requires. Public API: none. Wire: none. diff --git a/quest/m2/group-storage-bench.md b/quest/m2/group-storage-bench.md new file mode 100644 index 00000000000..09403f3d30c --- /dev/null +++ b/quest/m2/group-storage-bench.md @@ -0,0 +1,41 @@ +# [M] Benchmark group storage before choosing it + +## Goal + +A benchmark picks the storage behind `GroupState`'s frames before either +proposal lands: the [append-only log](/quest/m2/perf/kio-channel-contention/log.md) +(lock-free cursor reads, front eviction by an atomic base), or a ring buffer +in its place, against the sorted runs of +[Front eviction](/quest/m2/unbounded-groups/front-eviction.md) (in-place +refill below `committed`). The outcome is numbers and a recommendation, and +both quests then build on the chosen shape. + +## Plan + +Decided 2026-10-20 (maintainer): benchmark first; the maintainer leans +toward the log, or a ring buffer, because hot-path performance matters more +than evicting the middle of a group, which moved to m3 +([Middle eviction](/quest/m3/middle-eviction.md)). + +- Prototype each shape behind `GroupState`'s current read and write + operations only, enough to bench; neither prototype merges. +- Workloads: append with 1, 16, and 1,024 cursor readers on 1, 4, and 16 + threads; one-frame and multi-frame (video-heavy) groups; a long-lived group + under front eviction; and a refill of an evicted front run while the tail + writer appends. Reuse `rs/kio/benches/contended.rs` and moq-net's + `group.rs`, and the multi-frame `just bench-profile lock` shape the log + quest needs anyway. +- Bench how each shape refills a front hole: runs patch in place; a log or + ring buffer needs a side run below its base. Rejecting the refill is not an + option: the [unbounded groups](/quest/m2/unbounded-groups/README.md) reader + contract refills an evicted run and returns `Error::Evicted` only when no + source has it. If the log wins, say what Front eviction's refill becomes. +- Record the numbers and the choice in the + [kio contention epic](/quest/m2/perf/kio-channel-contention/README.md) and + the [unbounded groups epic](/quest/m2/unbounded-groups/README.md). + +Public API: none. Wire: none. + +## Related + +- [Segmented frames](/quest/m2/unbounded-groups/segmented-frames.md) - changes what a frame entry holds, not how a group stores its frames diff --git a/quest/m2/interop-bindings.md b/quest/m2/interop-bindings.md index b65f042a610..a579d278412 100644 --- a/quest/m2/interop-bindings.md +++ b/quest/m2/interop-bindings.md @@ -4,7 +4,7 @@ `just test interop --all` publishes and subscribes through the Kotlin, Swift, and Dart bindings against every other client, so a wrapper regression in -network delivery fails CI. +network delivery fails the nightly interop run. ## Plan @@ -26,8 +26,11 @@ clients already cover. alternative is a macOS nightly job that runs only the Swift pairs. Recommendation: Linux, so one matrix covers every pair, unless the toolchain pushes the job past its time budget. -- Add `kt/`, `swift/`, and `dart/` to the interop workflow's PR path filter - once their clients are in the matrix. +- Interop runs nightly (`interop.yml`'s schedule), not through a PR path + filter (decided 2026-10-20, maintainer, because main PRs skip cloud CI + since #6038 and interop does not gate merges). The new clients join that + nightly matrix; add `kt/`, `swift/`, and `dart/` to the release-PR path + filter only if the release line backports them. Public API: none. Wire: none. diff --git a/quest/m2/js-ietf-dead-route-error.md b/quest/m2/js-ietf-dead-route-error.md deleted file mode 100644 index 96eaab2e1c7..00000000000 --- a/quest/m2/js-ietf-dead-route-error.md +++ /dev/null @@ -1,26 +0,0 @@ -# [XS] js/net names the real cause of a dead route on IETF - -## Goal - -When a moq-transport session dies under a JS subscription, the subscription -ends with the session's own error, as it does in Rust, instead of -`unauthorized: `, such as `unauthorized: room`. A grant -that lapses because its session closed is not a revocation. - -## Plan - -Decided with the maintainer on 2026-10-17: the goal as drafted, from the -follow-ups of [#5700](https://github.com/moq-dev/moq/pull/5700). - -The cause, as read at planning: `#close` in `js/net/src/ietf/connection.ts` -closes the session's auth first, which empties the grant union -(`#recompute` in `js/net/src/auth_session.ts`). The IETF subscriber's grant -watcher (`#denied` in `js/net/src/ietf/subscriber.ts`) then closes each live -subscription with `unauthorized(broadcast)` before anything ends it with the -session error. Rust has the same grant shape, but its IETF driver -(`rs/moq-net/src/ietf/session.rs`) aborts every received track with the -session's error in the same pass that closes auth, so the session error wins. - -Check the lite subscriber for the same race. Add a test that kills an IETF -session under a live subscription and asserts the session error, failing -without the fix. Wire: none. diff --git a/quest/m2/kt-ffi-calls.md b/quest/m2/kt-ffi-calls.md index 0ec76b1fb5c..20f5b0d56d1 100644 --- a/quest/m2/kt-ffi-calls.md +++ b/quest/m2/kt-ffi-calls.md @@ -28,8 +28,8 @@ Decided with the maintainer on 2026-10-15: unchanged. - [The BoltFFI spike](/bench/ffi-gen/README.md) measured the target: BoltFFI's JNI backend makes a no-op call in 6 ns and a 100 B `write_frame` in 134 ns, - against UniFFI's 3.2 and 12.8 us. If the maintainer picks BoltFFI, delete - this quest. + against UniFFI's 3.2 and 12.8 us. UniFFI stays the base (#5838), so the + fix lands in its Kotlin generator. Public API: none expected. Wire: none. diff --git a/quest/m2/latency-ledger.md b/quest/m2/latency-ledger.md index 7248e8e3ba8..2b776237279 100644 --- a/quest/m2/latency-ledger.md +++ b/quest/m2/latency-ledger.md @@ -36,3 +36,4 @@ probes, which was the right trade to get it running. This quest promotes them. ## Related - [QoS](/quest/m2/qos/README.md) - relay-side health, the same idea from the other end +- [Media stats](/quest/m2/stats/README.md) - the hang stats and `.echo` snapshots these stages join diff --git a/quest/m2/net-wrap-trait.md b/quest/m2/net-wrap-trait.md deleted file mode 100644 index a8f6ca35f28..00000000000 --- a/quest/m2/net-wrap-trait.md +++ /dev/null @@ -1,72 +0,0 @@ -# [M] Design a wrapper seam over moq-net - -## Goal - -Decide how a wrapper (e2ee, json, or a future one) sits between any moq-net -writer or reader and the network without rewriting each writer per wrapper. -Writers and readers include the moq-mux importers, json, and applications. -Compression is not a wrapper: it moves into moq-net as a track flag -([Compressed tracks](/quest/m1/compress/README.md)). - -The outcome is a reviewed decision record plus a prototype benchmark, not a -port. It ends by splitting the implementation into follow-up quests sized by -the decision. - -Non-goals: -- The runtime seam: [Runtime trait](/quest/m2/runtime-trait.md). -- The transport seam: [Drop web-transport-trait](/quest/m2/drop-web-transport-trait.md). - -## Plan - -Decided 2026-10-17: - -- **Why now.** E2EE is the first wrapper. moq-mux importers write into a - concrete `moq_net::broadcast::Producer` (about 108 `create_track` call - sites). json takes a concrete `track::Producer`, but already splits a codec - layer (`Encoder`/`Decoder`) from track ownership. -- **Interim design.** E2EE composes through a bridge: the - `moq_e2ee::broadcast::{Producer, Consumer}` wrappers (#5876) wrap a concrete - moq-net broadcast and see every track creation themselves. The e2ee mux path - (#5822) is on hold to be rebuilt on them. `moq-net` gains no track listing - for it (#5880 was abandoned). This quest compares the trait against that - bridge, not against a blank slate. -- **Design-only, with a gate.** Implementation splits into follow-up quests - only after the decision is reviewed. -- **Rust and JS together.** Names mirror across both. The browser e2ee - publisher has the same concrete-broadcast problem as moq-mux. -- **If a trait wins,** a follow-up quest moves the e2ee mux path onto it and - deletes the bridge. -- **Placement.** This is Related to the e2ee epic, not Required. E2EE ships on - the bridge. - -Open, for the design: - -- Trait versus bridge. Weigh the trait's generic spread or `dyn` cost against - the bridge's extra in-memory hop. -- Scope: which of the broadcast, track, and group levels it covers, for both - producers and consumers. Datagrams are in scope too: e2ee wraps them, and - they have their own append and receive APIs and drop rules. -- Generic or `dyn` in the frame write hot path. No `Send` bounds that native - needs and wasm can't satisfy (see `rs/AGENTS.md`). -- Group lifecycle hooks. e2ee's nonce binds the group and frame sequence, so - a per-frame map is not enough. -- Terminal propagation. Finish, typed aborts (such as e2ee's authentication - error), and handle teardown must cross the wrapper at the broadcast, track, - and group levels, not collapse into `Error::Dropped` or keep a producer - alive. -- Whether json becomes a wrapper or stays a codec above a track. -- Demand and preference propagation. moq-mux and the browser publisher gate - output on `track::Demand`, so a wrapper must not keep encoders active with - no downstream viewer or drop downstream preference updates. Compare both - designs on idle and active demand, not only throughput. -- Benchmark trait dispatch against the bridge hop on both sides: the write - bridge above, and its reverse for readers, where unwrapping once before a - local broadcast differs from unwrapping per reader. Include a datagram - workload. Sweep tracks × frames per group, and readers on the read side, so - a cost that grows with the table shows up as a slope. - -## Related - -- [E2EE](/quest/m2/e2ee/README.md) - the first wrapper; it ships on the bridge -- [Compressed tracks](/quest/m1/compress/README.md) - compression moves into moq-net as a track flag, so it is not a wrapper -- [Runtime trait](/quest/m2/runtime-trait.md) - a separate seam, out of scope here diff --git a/quest/m2/net-wrap/README.md b/quest/m2/net-wrap/README.md new file mode 100644 index 00000000000..a96b8941d86 --- /dev/null +++ b/quest/m2/net-wrap/README.md @@ -0,0 +1,80 @@ +# Wrapper traits over moq-net + +## Goal + +A wrapper such as e2ee sits between any moq-net writer or reader and the +network without changing the writer or reader. moq-net exports traits for +its producers and consumers at the broadcast, track, group, and frame levels, +datagrams included, and moq-net's own types implement them. Every writer and +reader above moq-net takes the traits, so moq-mux, moq-json, moq-stats, hang, +and the CLI can be handed an encrypted broadcast or track and publish or +consume it unchanged. Rust and JS use the same names. + +Compression is not a wrapper: it moves into moq-net as a track flag +([Compressed tracks](/quest/m1/compress/README.md)), which deletes +moq-flate, so moq-flate is out of scope. + +Non-goals: +- The runtime seam: [Runtime trait](/quest/m2/runtime-trait.md). +- The transport seam: [Drop web-transport-trait](/quest/m2/drop-web-transport-trait.md). + +## Plan + +Decided 2026-10-20 (maintainer), replacing the 2026-10-17 design-first plan +and its trait-versus-bridge comparison: + +- **A trait, implemented directly.** No bridge comparison, prototype + benchmark, or design gate. The bridge (#5822) is closed and superseded. +- **Generics, not `dyn`.** Static dispatch keeps the frame hot path free of + vtable calls, and moq-net's own types are the default type parameter, so + plaintext callers keep their source and pay nothing. The traits carry no + `Send` bounds that wasm can't meet (see `rs/AGENTS.md`); a native caller + that spawns adds the bound at its spawn site. +- **Mirror moq-net's current API exactly.** The trait methods are the + methods writers and readers call today, with the same names, arguments, + and errors: `create_track`, `reserve_track`, `dynamic`, `demand`, + `create_group(group::Info)`, `append_group`, `append_datagram`, + `insert_datagram`, `write_frame`, `create_frame`, `write`, `finish`, + `abort(self, Error)`, `closed`, `track(name)` then `.subscribe(..)`, + `recv_group`, `recv_datagram`, `read_frame`, `next_frame`, `read_chunk`, + and their `poll_*` forms. A wrapper adapts + to moq-net, not the other way around: moq-e2ee's error converts into + `moq_net::Error`. +- **Terminal propagation crosses the wrapper.** Finish, typed aborts (such + as e2ee's authentication error), and handle teardown reach the other side + at every level, rather than collapsing into `Error::Dropped` or keeping a + producer alive. +- **Demand and preference cross the wrapper.** moq-mux and the browser + publisher gate output on `track::Demand`, so a wrapper must not keep + encoders active with no downstream viewer or drop downstream preference + updates. Test idle and active demand through a wrapper. +- **Scope is every writer and reader above moq-net:** moq-mux (with the + moq-video and moq-audio encoders built on it), moq-json, moq-stats, hang, + and the CLI paths, plus the JS packages that need it. +- **Ships in m2 with e2ee**, not in m1's breaking release: it is a public + API break, and e2ee is its first consumer. + +Today moq-net has no model traits (only `Consume` in +`rs/moq-net/src/model/origin.rs`). moq-mux and moq-json take concrete types, +for example `container::Producer::new(track::Producer)` and moq-json's +snapshot, stream, and window `Producer::new(track::Producer)` and +`Consumer::new(track::Subscriber)`. + +Open: the trait names. Bring options with a recommendation in the +[moq-net traits](/quest/m2/net-wrap/traits.md) PR; the methods are fixed +above. + +## Required + +- [moq-net traits](/quest/m2/net-wrap/traits.md) - moq-net exports producer and consumer traits at every level, implemented by its own types +- [moq-e2ee implements the traits](/quest/m2/net-wrap/e2ee.md) - the e2ee broadcast, track, and group wrappers implement the traits, proving them before the adopters land +- [moq-json, moq-stats, and hang take the traits](/quest/m2/net-wrap/json.md) - json snapshots, streams, windows, the stats producer, and hang's tracks write and read through the traits +- [moq-mux takes the traits](/quest/m2/net-wrap/mux.md) - importers, catalog, container, timeline, json and binary tracks, `Source`, TS export, and the moq-video and moq-audio encoders are generic over the traits +- [CLI paths take the traits](/quest/m2/net-wrap/cli.md) - the CLI's publish and play paths hand moq-mux any broadcast that implements them +- [JS mirror](/quest/m2/net-wrap/js.md) - `@moq/net` exports the same interfaces, taken by `@moq/hang`, `@moq/publish`, and `@moq/watch`, and implemented by `@moq/e2ee` + +## Related + +- [E2EE](/quest/m2/e2ee/README.md) - the first wrapper, which publishes and consumes through these traits +- [Compressed tracks](/quest/m1/compress/README.md) - compression moves into moq-net as a track flag, so it is not a wrapper +- [Runtime trait](/quest/m2/runtime-trait.md) - a separate seam, out of scope here diff --git a/quest/m2/net-wrap/cli.md b/quest/m2/net-wrap/cli.md new file mode 100644 index 00000000000..6253c00cf9c --- /dev/null +++ b/quest/m2/net-wrap/cli.md @@ -0,0 +1,22 @@ +# [S] CLI paths take the traits + +## Goal + +`moq-cli`'s publish and play paths hand moq-mux a broadcast through the +[moq-net traits](/quest/m2/net-wrap/traits.md), so the +[encrypted CLI](/quest/m2/e2ee/cli.md) only swaps in an encrypted broadcast +and adds no crypto to the import, capture, export, or playback code. + +## Plan + +- Make the code between the session and moq-mux generic where it holds a + concrete moq-net broadcast or track: import, capture, export, and play. +- Paths that inspect a broadcast's content for the operator, such as stats + and echo, stay concrete; the encrypted CLI refuses them. +- No behavior change: the existing CLI tests pass unmodified. + +Public API: none outside moq-cli. Wire: none. + +## Required + +- [moq-mux takes the traits](/quest/m2/net-wrap/mux.md) - the CLI drives moq-mux diff --git a/quest/m2/net-wrap/e2ee.md b/quest/m2/net-wrap/e2ee.md new file mode 100644 index 00000000000..f159d63f10a --- /dev/null +++ b/quest/m2/net-wrap/e2ee.md @@ -0,0 +1,85 @@ +# [M] moq-e2ee implements the moq-net traits + +## Goal + +moq-e2ee's broadcast, track, and group wrappers implement the +[moq-net traits](/quest/m2/net-wrap/traits.md), so anything generic over them +publishes or consumes an encrypted broadcast unchanged, under semantic track +names. + +## Plan + +The wrappers merged in #5876 diverge from moq-net, and they adapt to it, not +the reverse (decided 2026-10-20, maintainer): + +- `finish(self)` and `abort(self)` without an error become moq-net's + `finish` and `abort(self, Error)`. +- `track::Producer::create_group(u64)` takes `group::Info`. +- `broadcast::Consumer::subscribe(semantic, ..)`, which returns a + `kio::Pending`, becomes `track(semantic)` then `.subscribe(..)`. +- `moq_e2ee::Error` converts into `moq_net::Error`, keeping the draft's + typed codes (such as an authentication failure) distinguishable rather + than collapsing into `Error::Dropped`. +- Datagrams: `append_datagram`, `insert_datagram`, and `recv_datagram` + match moq-net's signatures, so the trait read yields only authenticated + datagrams. A datagram that fails authentication, or a duplicate, is + dropped there and the track continues. The draft's `authentication` + event stays on moq-e2ee's own consumer, outside the trait, for callers + that hold the concrete type. A trait read can't carry it without + breaking the exact mirror. +- `track::Info.compress` is logical. Ciphertext doesn't compress, so the + wrapper compresses each group, and each datagram alone with a fresh + window, with moq-net's + [track compression](/quest/m1/compress/README.md) before AEAD, creates the + physical track uncompressed, and inflates after decryption. Sessions drop + a sealed datagram past the transport's datagram size as they drop an + oversized plaintext one, so document that the AEAD tag and any compression + growth count against the caller's payload budget. The physical size + describes ciphertext, so each sealed frame and datagram carries its + inflated size inside the authenticated payload: `frame::Info.size` stays + the logical size, the subscriber reserves only that declared size against + the same group and pool limits as a plain frame, and an inflate that + differs from it is an error, as moq-net's compression contract requires. + This keeps + [rust-publish](/quest/m2/e2ee/rust-publish.md)'s rule that protected + catalogs compress before encryption, and needs moq-net to expose its group + codec to the wrapper. `@moq/e2ee` does the same in + [JS](/quest/m2/net-wrap/js.md). +- The subscriber's wrapper learns the logical flag from authenticated E2EE + metadata, never from the physical track (decided 2026-10-20, maintainer): + the physical track's `compress` is always false, so it can't signal it. + The metadata lives in the E2EE track mapping, and a flag that fails + authentication fails the track loudly. Not the encrypted catalog: the + catalog track is itself compressed, so a subscriber would need the flag + before it could inflate the catalog that carries it. The + [E2EE epic](/quest/m2/e2ee/README.md) also keeps catalog formats out of + moq-e2ee, and [migrate](/quest/m1/compress/migrate.md) drops hang's + `compression` field. The subscriber needs the flag before inflating, + without changing moq-net's `track(name)`; propose the encoding in the PR. + Record it in `draft-lcurley-moq-e2ee.md`, replacing the Catalogs section's + `.z` sibling rule. +- Ranges: moq-net widens a range that starts mid-group on a compressed group + to frame 0 when the requester lacks the prefix, but it sees only the + uncompressed physical track. The wrapper's subscriber applies that rule + itself: it widens such a physical request to frame 0, inflates from + there, and yields only the requested logical range. Test a resumed + mid-group read, not only full round trips. +- Frames: AEAD authenticates a whole frame, so the wrapper's frame producer + buffers chunks from `write` and seals at `finish`, and its frame consumer + yields chunks only after the frame authenticates. +- Demand and preference cross the wrapper: an encrypted track's demand is + its physical track's, and preference updates reach the physical track. +- Keep the nonce rules the epic specifies: a group producer still owns its + frame numbering, and a re-created track name is refused, not dropped (see + [rust-publish](/quest/m2/e2ee/rust-publish.md)). +- Test the wrappers against the questline's pass-through test cases. The + moq-json round trip through an encrypted track waits for + [moq-json to take the traits](/quest/m2/net-wrap/json.md). + +Public API: the moq-e2ee wrapper signatures change to moq-net's. Wire: +`draft-lcurley-moq-e2ee.md` gains the authenticated logical `compress` flag +and each compressed object's authenticated inflated size. + +## Required + +- [moq-net traits](/quest/m2/net-wrap/traits.md) - the traits this implements diff --git a/quest/m2/net-wrap/js.md b/quest/m2/net-wrap/js.md new file mode 100644 index 00000000000..ee020adc72d --- /dev/null +++ b/quest/m2/net-wrap/js.md @@ -0,0 +1,36 @@ +# [L] JS mirror of the wrapper traits + +## Goal + +`@moq/net` exports interfaces with the Rust traits' names and methods, and +`@moq/hang`, `@moq/publish`, and `@moq/watch` accept them where they take a +concrete broadcast, track, or group today. `@moq/e2ee`'s wrappers implement +them, so a browser publisher or player handles an encrypted broadcast without +its own decryption path. + +## Plan + +- Mirror the [Rust traits](/quest/m2/net-wrap/traits.md) one to one, per the + cross-language naming rule. TypeScript's structural types make the default + free; there is no dispatch question. +- Take the interfaces only where a wrapper must pass through, as needed: + `@moq/hang`'s catalog and container code, `@moq/publish`'s broadcast, + and `@moq/watch`'s subscriptions. +- `@moq/e2ee`'s broadcast, track, and group wrappers adapt to `@moq/net`'s + method shapes, as moq-e2ee does in Rust, with its errors mapped to + `@moq/net`'s. Like moq-e2ee, it compresses a `compress` track before + encryption, publishes the physical track uncompressed, and carries the + logical flag in the same authenticated E2EE metadata + ([e2ee](/quest/m2/net-wrap/e2ee.md)). +- Demand and preference cross the wrapper: a publisher with no viewer + through an encrypted track stays idle. +- Test publish and watch round trips through a pass-through wrapper and + through `@moq/e2ee`. + +Public API: new `@moq/net` interfaces; `@moq/e2ee` wrapper signatures change. +Wire: the `draft-lcurley-moq-e2ee.md` compress flag from +[e2ee](/quest/m2/net-wrap/e2ee.md). + +## Required + +- [moq-net traits](/quest/m2/net-wrap/traits.md) - the names and methods this mirrors diff --git a/quest/m2/net-wrap/json.md b/quest/m2/net-wrap/json.md new file mode 100644 index 00000000000..a35c764e618 --- /dev/null +++ b/quest/m2/net-wrap/json.md @@ -0,0 +1,34 @@ +# [M] moq-json, moq-stats, and hang take the traits + +## Goal + +moq-json's snapshot, stream, and window producers and consumers, the +moq-stats producer and consumer, and hang's catalog, container, stats, echo, +and timeline tracks are generic over the +[moq-net traits](/quest/m2/net-wrap/traits.md), defaulting to moq-net's +types, so each publishes into and reads from a wrapped track unchanged. + +## Plan + +- moq-json takes the trait, not a wrapper role (decided 2026-10-20, + maintainer): its `Encoder`/`Decoder` codec layer stays as it is, and only + `Producer::new(track::Producer)` and `Consumer::new(track::Subscriber)` in + `snapshot`, `stream`, and `window` become generic. +- hang: `catalog::Consumer::new(track::Subscriber)`, + `Root::subscribe(&broadcast::Consumer)`, and + `container::Frame::write_to(&mut group::Producer)` are the entry points. +- moq-stats: the produce and consume paths that hold moq-net broadcasts and + groups. +- Group rolls these crates trigger themselves keep working through a + wrapper, as do demand-gated encoders. +- Existing tests keep passing with the defaults. Add one round trip per crate + through moq-net's shared pass-through test wrapper, and a moq-json snapshot + round trip through an encrypted track in moq-e2ee's tests. + +Public API: generic type parameters with moq-net defaults on moq-json, +moq-stats, and hang. Wire: none. + +## Required + +- [moq-net traits](/quest/m2/net-wrap/traits.md) - the traits these take +- [moq-e2ee implements the traits](/quest/m2/net-wrap/e2ee.md) - the encrypted track the snapshot round trip runs through diff --git a/quest/m2/net-wrap/mux.md b/quest/m2/net-wrap/mux.md new file mode 100644 index 00000000000..5028ee22b59 --- /dev/null +++ b/quest/m2/net-wrap/mux.md @@ -0,0 +1,38 @@ +# [L] moq-mux takes the traits + +## Goal + +moq-mux writes and reads through the +[moq-net traits](/quest/m2/net-wrap/traits.md), defaulting to moq-net's +types: the importers, catalog, container producers and consumers, timeline, +json and binary tracks, `Source`, and TS export. The moq-video and moq-audio +encode producers built on moq-mux take them too. Handed an encrypted +broadcast, every importer and capture encoder publishes it, and every +exporter reads one, without knowing. + +## Plan + +- Entry points include `container::Producer::new(track::Producer)`, the + importers' `broadcast::Producer` (`create_track` call sites throughout + `import/`), `catalog/`, `timeline.rs`, `json.rs`, `binary.rs`, + `source.rs`, and the TS exporter under `container/ts`. About 250 type + references across these modules. +- Capture: `moq_video::encode::Producer::new` takes a concrete + `moq_net::broadcast::Producer` and `moq_audio::encode::Producer::new` a + `&mut` one (`rs/moq-video/src/encode/producer.rs`, + `rs/moq-audio/src/encode/producer.rs`), so `moq import capture` could not + hand them an encrypted broadcast. Make both generic the same way. +- Demand gating stays as it is: importers and `Source` read + `track::Demand` and preferences through the trait. +- No behavior change with the defaults. Keep the `containers` Criterion + bench flat before and after. +- Test an import and export round trip (fMP4 and TS) through the moq-net + pass-through test wrapper, plus one through moq-e2ee's wrappers once + [they implement the traits](/quest/m2/net-wrap/e2ee.md). + +Public API: generic type parameters with moq-net defaults across moq-mux. +Wire: none. + +## Required + +- [moq-json, moq-stats, and hang take the traits](/quest/m2/net-wrap/json.md) - moq-mux writes catalogs and json tracks through them diff --git a/quest/m2/net-wrap/traits.md b/quest/m2/net-wrap/traits.md new file mode 100644 index 00000000000..ae8a649ae9f --- /dev/null +++ b/quest/m2/net-wrap/traits.md @@ -0,0 +1,60 @@ +# [L] moq-net producer and consumer traits + +## Goal + +moq-net exports traits for the broadcast, track, group, and frame producers +and consumers, datagrams included, and its own types implement them with no +change in behavior or cost. A crate generic over the traits compiles against +moq-net's types by default and against a wrapper's types when handed one. + +## Plan + +See the [questline](/quest/m2/net-wrap/README.md) for the decisions: generics +with moq-net's types as the default, no `Send` bounds wasm can't meet, and +the methods exactly as moq-net has them today. + +- Every handle a writer or reader can get from moq-net's public API, + including pending operations (`track::Subscribing`, `Querying`, + `Fetching`), requests, and adapters, has an associated type, so no trait + method returns a concrete moq-net type and a wrapper never leaks its + physical handle. Examples of what the rule covers: + - The producer chain: a broadcast producer's `create_track` returns its + own track producer, whose `create_group` returns its own group + producer, whose `create_frame` returns its own frame producer. + moq-mux's fMP4 and LOC importers stream chunks through `create_frame` + and `write`, and readers use `next_frame` and `read_chunk`, so without + the frame level they stay concrete or buffer whole frames. + `frame::Producer<'_>` borrows its group, so its associated type is + generic over that lifetime. + - Requests: a broadcast producer's `dynamic` returns its own + `broadcast::Dynamic`, whose `requested_track` yields its own + `track::Request`, so generic code serves undeclared names (as + [E2EE dynamic names](/quest/m2/e2ee/dynamic-names.md) needs). A track + dynamic's `requested_group` yields its own `group::Request`, whose + `accept` returns its own group producer. + - Pending operations: `track::Consumer::subscribe` resolves to the + wrapper's own subscriber, so E2EE decrypts behind it; `Querying` and + `Fetching` resolve to the wrapper's logical metadata and groups. + - Adapters: a subscriber's `ordered` returns its own `track::Ordered` + with the same group, datagram, and control surface. moq-json's snapshot + and window consumers, moq-mux, and moq-stats read through `ordered()`, + so without it they stay concrete. +- `poll_*` methods are the required ones; the async forms are provided + methods over them, so no `async fn` in a trait forces a `Send` bound. +- Demand (`demand`, `subscription_changed`, `Request::accept` and `reject`) + and terminal state (`finish`, `abort(self, Error)`, `closed`) are trait + methods, so a wrapper forwards them rather than reimplementing them. +- Inline moq-net's impls so a monomorphized caller compiles to today's code. + Prove it on `rs/moq-net/benches` (`group.rs`, `frame.rs`) before and after: + no regression on the plaintext path. +- A pass-through test wrapper implements every trait and checks that + finish, a typed abort, demand, and preference updates reach the other + side at each level. It sits behind a hidden, off-by-default moq-net + feature, like `fuzz`, so moq-json, moq-stats, and hang reuse it from + their dev-dependencies. +- Decide whether the existing `Consume` in `model/origin.rs` folds into + these traits or stays separate. +- Update `doc/concept` where it names the concrete types, and add the traits + to `MIGRATE.md`. + +Public API: new moq-net traits, plus the hidden test-wrapper feature. Wire: none. diff --git a/quest/m2/perf/kio-channel-contention/README.md b/quest/m2/perf/kio-channel-contention/README.md index dc549ddd394..3719411e8d0 100644 --- a/quest/m2/perf/kio-channel-contention/README.md +++ b/quest/m2/perf/kio-channel-contention/README.md @@ -150,6 +150,12 @@ Decided (maintainer, 2026-10-14, per-thread follow-up): - The append-only log waits on the spikes, since per-thread sessions would leave each replica's mutex uncontended. +Decided 2026-10-20 (maintainer): the +[group storage benchmark](/quest/m2/group-storage-bench.md) chooses between +the log (or a ring buffer) and unbounded groups' sorted runs for +`GroupState` before either lands, leaning log because performance matters +more than middle eviction. + ## Required - [Group expiry](/quest/m2/perf/kio-channel-contention/group-expiry.md) - a group poll decides expiry without locking the track state, the largest single wait site diff --git a/quest/m2/perf/kio-channel-contention/log.md b/quest/m2/perf/kio-channel-contention/log.md index 7cd841d2eff..102218d9ae0 100644 --- a/quest/m2/perf/kio-channel-contention/log.md +++ b/quest/m2/perf/kio-channel-contention/log.md @@ -5,7 +5,9 @@ kio exports an append-only log channel whose consumers read committed entries by cursor without taking a lock, and the hottest moq-net channels the [epic](/quest/m2/perf/kio-channel-contention/README.md)'s baseline names -(expected: `GroupState` frames, `TrackState` arrival order) move onto it. +(expected: `TrackState` arrival order, and `GroupState` frames if the +[group storage benchmark](/quest/m2/group-storage-bench.md) picks the log) +move onto it. Lands with the contended bench (`rs/kio/benches/contended.rs`) and `just bench-profile lock` before and after; a measured no-win abandons it. @@ -52,8 +54,18 @@ measured no-win abandons it. already provides. - Breaking moq-net's model internals is allowed; check whether any public moq-net API changes and sync `js/net` if so. +- Decided 2026-10-20 (maintainer): `GroupState` storage is chosen by the + [group storage benchmark](/quest/m2/group-storage-bench.md) against + [Front eviction](/quest/m2/unbounded-groups/front-eviction.md)'s sorted + runs before either lands. The maintainer leans toward this log, or a ring + buffer, because performance matters more than middle eviction. ## Required +- [Group storage benchmark](/quest/m2/group-storage-bench.md) - picks the log, a ring buffer, or sorted runs for `GroupState` frames - [Replica spike](/quest/m2/perf/kio-channel-contention/spike-replica.md) - if per-worker replicas win, each channel's cross-core readers collapse to one flow-controlled stream per worker and this log is likely abandoned - [Cluster spike](/quest/m2/perf/kio-channel-contention/spike-cluster.md) - the same question with nothing shared + +## Related + +- [Front eviction](/quest/m2/unbounded-groups/front-eviction.md) - the other proposal for `GroupState` storage, decided by the same benchmark diff --git a/quest/m2/read-only-lookup.md b/quest/m2/read-only-lookup.md index b26d244b930..5ab54fac910 100644 --- a/quest/m2/read-only-lookup.md +++ b/quest/m2/read-only-lookup.md @@ -9,9 +9,12 @@ mutates. ## Plan -Found by #5091: `track_inner` (`rs/moq-net/src/model/broadcast.rs`) mutates -the state on every lookup because `WeakCache::get` takes `&mut` to drop a -closed entry, and that mutation wakes every watcher. That accidental wake hid +Found by #5091: the broadcast lookup (`rs/moq-net/src/model/broadcast.rs`) +mutates the state on every lookup because `WeakCache::get` takes `&mut` to +drop a closed entry, and that mutation wakes every watcher. Since #5820 made +bare track handles lazy, the lookup is `Consumer::track`, `resolve_track` +(behind subscribe and fetch), and `peek_track`, all through +`BroadcastState::peek` or `tracks.get` under `lock()`. That accidental wake hid the front's lost wake; the front loom models in `rs/moq-net/tests/loom.rs` now guard against it returning. diff --git a/quest/m2/release/README.md b/quest/m2/release/README.md index da0317dde73..b7a3bb4a310 100644 --- a/quest/m2/release/README.md +++ b/quest/m2/release/README.md @@ -31,15 +31,12 @@ production stability proven merely because upstream artifacts published. - [API shape review](/quest/m2/release/api-review.md) - every new or changed public API reviewed for a better shape before the cut - [Feature review](/quest/m2/release/feature-review.md) - every major work line since the last release judged end to end before the cut - [Cut and publish](/quest/m2/release/cut.md) - independently executable after m1; release-triggered publication -- [Published catalog readers](/quest/m2/release/catalog-readers.md) - released Go and JS clients accept the current Rust publisher's live catalog in smoke; an earlier release-line patch may satisfy it - [Published assets](/quest/m2/release/assets-release.md) - npm carries hostable worklets - [C++ publication](/quest/m2/release/cpp-release.md) - archives and matching OBS plugin publish and install - [Generated C publication](/quest/m2/release/c-publish.md) - released archives run the C samples and interop client - [Dart publication](/quest/m2/release/dart-publish.md) - the wrapper and native artifacts install from the registry - [Final libmoq stub](/quest/m2/release/libmoq-final-release.md) - publish the migration notice before removing its source - [Retire libmoq](/quest/m2/release/libmoq-retire.md) - cleanup only after that publication -- [Final hand-written C notice](/quest/m2/release/c-final-release.md) - publish the migration notice before source retirement -- [Retire hand-written C](/quest/m2/release/c-retire.md) - complete the final old-package notice and source retirement +- [Retire hand-written C](/quest/m2/release/c-retire.md) - delete the hand-written crate once generated C publishes - [Retire the web-transport JS qmux](/quest/m2/release/qmux-js-retire.md) - once `@moq/qmux` publishes from `js/qmux`, deprecate `@moq/web-socket-stream` and stop the second publisher -- [Hosted worklet adoption](/quest/m2/release/dogfood-assets.md) - downstream consumers exercise the published assets - [moq.sh deployment](/quest/m2/release/moq-sh-deploy.md) - verify its release-triggered publication diff --git a/quest/m2/release/assets-release.md b/quest/m2/release/assets-release.md index 5a85130c425..07d09c89b25 100644 --- a/quest/m2/release/assets-release.md +++ b/quest/m2/release/assets-release.md @@ -10,7 +10,9 @@ this quest and every `Required` entry that links it. ## Plan -As of 2026-10-08 `main` carries `assets()` (#4518) but `release` does not, so -the newest releases (`@moq/watch` 0.6.2, `@moq/publish` 0.5.2) still lack it. Check with +As of 2026-10-20 `main` carries `assets()` (#4518) but `release` does not, so +the newest releases (`@moq/watch` 0.6.3, `@moq/publish` 0.5.3) still lack it; +`@moq/watch` 0.6.3's `exports` has no `./assets/*`. moq.pro's dashboard +adoption of the hosted files lives in moq.pro's package adoption, not here. Check with `npm view @moq/watch version` and `npm view @moq/publish version` for a newer version, then confirm its `exports` lists `./assets/*`. diff --git a/quest/m2/release/c-final-release.md b/quest/m2/release/c-final-release.md deleted file mode 100644 index e1e0ac80873..00000000000 --- a/quest/m2/release/c-final-release.md +++ /dev/null @@ -1,12 +0,0 @@ -# [XS] Publish the final hand-written C migration notice - -## Goal - -Waiting on publication: the final hand-written `moq-c` 0.7.x release tells -users to migrate to generated `moq-c` 0.8.0. Verify the published registry -artifact contains the notice before deleting the old source. Delete this -condition after publication is verified. - -## Related - -- [Retire hand-written C](/quest/m2/release/c-retire.md) - source removal follows the final notice and generated package publication diff --git a/quest/m2/release/c-publish.md b/quest/m2/release/c-publish.md index 5d5747d0fd4..a106eb2dc58 100644 --- a/quest/m2/release/c-publish.md +++ b/quest/m2/release/c-publish.md @@ -10,8 +10,9 @@ the C interop client build and run against it from a clean consumer checkout. The m1 C line proves the same cases with the built archive before the cut. This quest proves registry/release delivery after publication and records the artifact versions and results. It cannot block the cut that creates them. -Coordinate the final hand-written package notice before replacing its -publication; retain the source until its retirement condition is satisfied. +The hand-written crate never published, so there is no old package to +supersede; its source stays until +[c-retire](/quest/m2/release/c-retire.md). ## Required diff --git a/quest/m2/release/c-retire.md b/quest/m2/release/c-retire.md index 1c4f12b2e9b..1c02587a92f 100644 --- a/quest/m2/release/c-retire.md +++ b/quest/m2/release/c-retire.md @@ -2,19 +2,22 @@ ## Goal -The hand-written C crate is gone: its final `moq-c` 0.7.x release points -users at the generated `moq-c` 0.8.0, and the source, workflows, and docs that only served -it are deleted. The generated `moq-c` package's `MIGRATE.md` tells C users how to move. +The hand-written C crate is gone: its source, workflows, and docs that only +served it are deleted once the generated `moq-c` publishes. The generated +`moq-c` package's `MIGRATE.md` tells C users how to move. ## Plan - #4288 renamed libmoq to `moq-c` (`rs/moq-c`); this quest deletes that - hand-written crate. The code-free `rs/libmoq` stub left under the old name - is [Retire the libmoq stub](/quest/m2/release/libmoq-retire.md)'s, separately. + hand-written crate. +- Decided 2026-10-20 (maintainer): no final hand-written notice release, + because the hand-written `moq-c` never published to a registry; the + generated package is the first `moq-c` users install. +- The code-free `rs/libmoq` stub left under the old name is + [Retire the libmoq stub](/quest/m2/release/libmoq-retire.md)'s, separately. ## Required -- [Final hand-written C notice](/quest/m2/release/c-final-release.md) - published before deleting its source - [Generated C publication](/quest/m2/release/c-publish.md) - the replacement is available and verified - [C consumers](/quest/m1/c/consumers.md) - nothing in the repository still calls the hand-written ABI diff --git a/quest/m2/release/catalog-readers.md b/quest/m2/release/catalog-readers.md deleted file mode 100644 index 1cf8db7a533..00000000000 --- a/quest/m2/release/catalog-readers.md +++ /dev/null @@ -1,37 +0,0 @@ -# [S] Published catalog readers accept current live media - -## Goal - -The latest published Go and JavaScript clients read live media from the -current Rust publisher in test/smoke's Nix channel. Incompatible optional -archive metadata does not make an otherwise playable catalog unreadable. - -## Plan - -The [2026-10-15 Smoke run](https://github.com/moq-dev/smoke/actions/runs/38739639896) -fails only the Nix Rust publisher against Go and all five JS readers: -`archive.track` is missing because current catalogs use `archive.timelines`. -The released Rust, Python, Kotlin, C, and GStreamer readers pass. - -Parser tolerance landed on main in #5485 and on the Rust release line in -#5520. `@moq/hang` 0.5.3 now publishes the tolerant parser (verified -2026-10-19), while the Go module installed in that run required the old -archive shape. Verify which published artifacts carry the corrected parser and -advance any others through the normal release process. This quest does not -authorize a breaking release cut or version bumps outside that process. - -Complete against packages installed from their registries in smoke, keeping -every MoQ package at latest. Do not pin an older publisher, skip these cells, -or suppress catalog errors. Direct archive validation and invalid media -remain strict. The 2026-10-19 wire-compat remeasurement verified that -released `@moq/hang` 0.5.3 accepts current catalogs and removed the -`catalog-archive` exception. The registry-installed smoke matrix remains -this quest's acceptance check. - -The maintainer requested fixes or quests for the investigated failures on -2026-10-15. Publication is tracked separately from the completed parser fix, -in the existing release-execution act rather than as a prerequisite of m1. - -## Related - -- [Release execution](/quest/m2/release/README.md) - owns publication and installed-artifact verification diff --git a/quest/m2/release/cpp-release.md b/quest/m2/release/cpp-release.md index 7f5e5804ecd..6d181c642bd 100644 --- a/quest/m2/release/cpp-release.md +++ b/quest/m2/release/cpp-release.md @@ -14,8 +14,9 @@ once `main` reaches `release`; check with The version is `cpp/moq/VERSION`; the plugin's is `cpp/obs/VERSION`. Delete this quest once the release exists. -The OBS plugin now ships only with a C++ release, so `obs-moq` stays at its -last moq-c build until this tag. Decided when the C++ line landed on `main` +As of 2026-10-20 no `cpp-v*` release exists, and `obs-moq` still publishes +from the release line's C build (`obs-moq-v0.6.17`, 2026-10-17); the first +plugin built on the C++ package replaces it at this tag. Decided when the C++ line landed on `main` (#4079): the first plugin built on the C++ package must not drop the Advanced settings or the dock's protocol and reconnect reason that the moq-c build had, so those parity quests gate this release. diff --git a/quest/m2/release/dart-publish.md b/quest/m2/release/dart-publish.md index b1cd8ee3f94..82e1b2429be 100644 --- a/quest/m2/release/dart-publish.md +++ b/quest/m2/release/dart-publish.md @@ -7,8 +7,8 @@ A `moq-dart-v*` tag publishes `moq` to pub.dev with no manual step, as ## Plan -Found in the 2026-10-05 audit: both packages are on pub.dev. `moq_ffi` has -eight versions (0.4.3 to 0.4.10, published 2026-09-25 to 2026-10-03), so its +Both packages are on pub.dev (checked 2026-10-20). `moq_ffi` publishes from +every `moq-ffi-v*` tag (0.4.3 through 0.4.17, the latest 2026-10-17), so its tag-driven publish works. `moq` has only 0.1.0 (2026-09-25), the one `moq-dart-v*` tag so far, so whether its tag publishes unattended through trusted publishing is still unproven. diff --git a/quest/m2/release/dogfood-assets.md b/quest/m2/release/dogfood-assets.md deleted file mode 100644 index 300ff7b380d..00000000000 --- a/quest/m2/release/dogfood-assets.md +++ /dev/null @@ -1,30 +0,0 @@ -# [S] Dogfood hosted worklets - -## Goal - -The moq.pro dashboard (`app/`) runs on the `@moq/*` release that ships -`assets()`. It hosts the worklet and worker files and calls -`Watch.assets()` / `Publish.assets()`, so the hosted path gets real traffic. -CSP is out of scope: the dashboard adds no policy. - -## Plan - -Decided in planning: the maintainer wants hosted mode dogfooded, not just the -pin bump. No CSP. moq.dev's half moved to -[its own quest tree](https://github.com/moq-dev/moq.dev/blob/main/quest/m0/dogfood-assets.md) -on 2026-10-09. The moq.pro side is part of its package adoption. - -Guidance: - -- Copy `node_modules/@moq/{watch,publish}/assets/*` into a served `/moq/` - directory at build time. SvelteKit's Vite config takes - `vite-plugin-static-copy` directly. -- Call `assets("/moq/")` once, before any element or player starts. The - dashboard imports the element entries, so import `assets` from the package root. -- Verify in a browser that audio plays and publishes, and that the network - panel shows the worklets and the capture worker (Firefox) loading from - `/moq/`, not `blob:`. - -## Required - -- [A watch and publish release ships assets()](/quest/m2/release/assets-release.md) - the hosted files exist on npm to copy diff --git a/quest/m2/release/libmoq-final-release.md b/quest/m2/release/libmoq-final-release.md index fb89d3a5c38..23749330cbe 100644 --- a/quest/m2/release/libmoq-final-release.md +++ b/quest/m2/release/libmoq-final-release.md @@ -6,15 +6,14 @@ The newest `libmoq` on crates.io is the code-free stub that `main`'s `rs/libmoq` holds, whose README points at `moq-c`. Since #4738 publishing runs only from `release`, so the condition clears at the next `main`-to-`release` cut, which must take `main`'s stub for `rs/libmoq` at a version above -0.6.12. `main`'s stub is at 0.6.12, which crates.io already holds as a full -crate, so the cut bumps it or the stub never publishes. +0.6.17. `main`'s stub is at 0.6.12, below versions crates.io already holds as +full crates, so the cut bumps it or the stub never publishes. This quest tracks a condition outside the repository. When it holds, delete this quest and every `Required` entry that links it. ## Plan -As of 2026-10-06 the newest release is libmoq 0.6.12 (2026-10-06), still cut -from `release`'s full crate, not the stub; release PR #4763 merged and -published 0.6.11 the same way. Check with +As of 2026-10-20 the newest release is libmoq 0.6.17 (2026-10-17), still cut +from `release`'s full crate, not the stub. Check with `curl -s https://crates.io/api/v1/crates/libmoq` for a newer `max_version`. diff --git a/quest/m2/rtc-pacing.md b/quest/m2/rtc-pacing.md new file mode 100644 index 00000000000..037e3fd6327 --- /dev/null +++ b/quest/m2/rtc-pacing.md @@ -0,0 +1,48 @@ +# [M] moq-rtc paces WHEP viewers to a cap, NACKs included + +## Goal + +An embedder can give a `moq-rtc` WHEP session a bitrate cap, and moq-rtc +never sends that viewer more media payload than the cap plus a burst, +retransmits included. The cap counts payload, like the +[QUIC caps](/quest/m2/rate-quic.md). RTP, SRTP, and UDP overhead sits +outside it: moq-rtc picks the packet sizes, so the overhead per payload +byte is fixed, and the viewer can't inflate it. This backs moq.pro's +egress caps. + +## Plan + +Decided 2026-10-20 (maintainer): planned upstream now, because moq.pro's +WHEP enforcement needs it and nobody owned it. + +- Pace the RTP packets each frame packetizes into, charging their media + payload, rather than whole frames: bucket credit caps at the burst, so a + frame larger than the burst could never be released whole without + exceeding cap plus burst. Decide drops only before a group starts: past + cap plus burst, skip to the next group, the same whole-group drop + moq.pro's RTMP play uses. A group that has started is paced through to + its end even when it, or one of its frames, is larger than the burst, as + large frames finish gradually on [QUIC](/quest/m2/rate-quic.md), so an + oversized keyframe group still makes progress. + Dropping individual RTP packets would leave partial frames that corrupt + the picture and draw NACKs and PLIs nobody can satisfy. +- NACK resends draw from the same rate-plus-burst bucket as the frame pacer, + and a resend with no credit left is skipped, so faking loss only spends the + credit the viewer's next group would have used. str0m fulfils NACKs from + its video send buffer (`rs/moq-rtc/src/session.rs`), so charging needs a + hook on that path: find one in str0m 0.24's API, or propose one upstream + to str0m with the maintainer's approval. +- A per-session rate handle sets the cap and accepts live updates. The pacer + counts media; the docs advise headroom for RTP overhead. +- Tests: a capped viewer never receives more than cap plus burst, including + a single frame larger than the burst, which still arrives; a viewer + over the cap resumes cleanly on the next group; two viewers with different + caps, each NACKing every packet for several burst windows, stay under + their own cap plus burst. + +Public API: a rate handle on moq-rtc's WHEP server. Wire: none. + +## Related + +- [moq-rtc REMB and TIAS](/quest/m2/rtc-remb-tias.md) - the WHIP half of the same caps +- [SRT rate handle](/quest/m2/srt-rate-handle.md) - the same pacing and retransmit charging on SRT diff --git a/quest/m2/rtc-remb-tias.md b/quest/m2/rtc-remb-tias.md new file mode 100644 index 00000000000..f49293b72ef --- /dev/null +++ b/quest/m2/rtc-remb-tias.md @@ -0,0 +1,35 @@ +# [M] moq-rtc steers WHIP publishers to a bitrate cap + +## Goal + +An embedder can give a `moq-rtc` WHIP session a bitrate cap, and moq-rtc +steers the browser's encoder under it: a `b=TIAS` and `b=AS` line in the SDP +answer, and REMB where the publisher negotiates it. Media past the cap plus a +burst anyway (a client ignoring both) closes the session with a typed error. +This backs moq.pro's ingest caps. + +## Plan + +Decided 2026-10-20 (maintainer): planned upstream now, because moq.pro's +WHIP enforcement needs it and nobody owned it. + +- moq-rtc has no REMB or bandwidth-line support today. Add both on the WHIP + server: TIAS and AS from the cap in the answer, and periodic REMB at or + below the cap. TWCC carries arrival times, not an estimate, so there is + nothing to clamp there. +- A per-session rate handle sets the cap after signaling and accepts live + updates; REMB follows an update at once. Burst defaults to about one + second of the cap, as on QUIC. +- Meter received media against the cap plus burst and close past it with a + typed rate-limit error the embedder can log. +- Tests: an answer carries TIAS and AS for a capped session and neither for + an uncapped one; REMB never exceeds the cap; a publisher sending over the + cap plus burst is closed; under the cap nothing changes. + +Public API: a rate handle and a rate-limit error on moq-rtc's WHIP server. +Wire: SDP bandwidth lines and RTCP REMB. + +## Related + +- [moq-rtc pacing](/quest/m2/rtc-pacing.md) - the WHEP half of the same caps +- [SRT rate handle](/quest/m2/srt-rate-handle.md) - the same caps on SRT diff --git a/quest/m2/runtime-trait.md b/quest/m2/runtime-trait.md index 0c9aca6e90f..428bbca28f1 100644 --- a/quest/m2/runtime-trait.md +++ b/quest/m2/runtime-trait.md @@ -4,8 +4,8 @@ A MoQ runtime trait covers spawn and time (via `moq-clock`), with tokio, browser, io_uring, and sim implementations, and replaces web-async in every -crate that uses it (moq-ffi, moq-mux, moq-net, moq-room, moq-stats, and -moq-wasm if it still exists), so a crate runs on any of them unchanged. +crate that still uses it (moq-clock's `runtime` feature, moq-ffi, moq-stats, +and moq-wasm), so a crate runs on any of them unchanged. moq-relay spawns through it too, so its in-memory integration tests (`rs/moq-relay/tests/integration/connection.rs`) move from paused tokio onto diff --git a/quest/m2/site/README.md b/quest/m2/site/README.md index f42ccb9421c..f18bec1ebc1 100644 --- a/quest/m2/site/README.md +++ b/quest/m2/site/README.md @@ -26,4 +26,4 @@ Its history stays there, since the move is a fresh snapshot. ## Related -- [Hosted worklet adoption](/quest/m2/release/dogfood-assets.md) - links moq.dev's half, which the move ports back into this tree +- [Hosted worklet adoption](https://github.com/moq-dev/moq.dev/blob/main/quest/m0/dogfood-assets.md) - moq.dev's half, which the move ports back into this tree diff --git a/quest/m2/site/move.md b/quest/m2/site/move.md index 2a9dcbfa742..c9f4fc5dd44 100644 --- a/quest/m2/site/move.md +++ b/quest/m2/site/move.md @@ -57,8 +57,8 @@ Decided 2026-10-20 in planning: Reconcile with this tree rather than duplicate it: the npm `assets()` condition already exists as [Published assets](/quest/m2/release/assets-release.md), and - [Hosted worklet adoption](/quest/m2/release/dogfood-assets.md) links - moq.dev's half, which can fold back into it. Rank the voice demo quests in + moq.dev's [hosted worklet adoption](https://github.com/moq-dev/moq.dev/blob/main/quest/m0/dogfood-assets.md) + can fold back into it. Rank the voice demo quests in m2 on their own, not under this epic, so they do not hold it open. - **Port the open draft posts** by re-applying their MDX under `site/` as draft PRs here: moq-dev/moq.dev#148 (Update 02), #145 (Web Datagrams), #132 (AI diff --git a/quest/m2/srt-rate-handle.md b/quest/m2/srt-rate-handle.md new file mode 100644 index 00000000000..bc9a3a071fa --- /dev/null +++ b/quest/m2/srt-rate-handle.md @@ -0,0 +1,63 @@ +# [M] moq-srt per-connection rate handle + +## Goal + +An embedder can give each `moq-srt` connection a bitrate cap through a rate +handle that stays valid across live rate updates. An SRT pull never sends +more media payload than the cap plus a burst, retransmits included, and an +SRT publish past the cap plus burst is closed with a reason the embedder can +log. The cap counts payload, like the [QUIC caps](/quest/m2/rate-quic.md), +but which payload depends on who muxes. On a pull, it is media payload: TS, +SRT, and UDP overhead sits outside it, because moq-srt picks the muxing and +packet sizes, so the overhead per payload byte is fixed and a viewer can't +inflate it. On a publish, it is the MPEG-TS payload the SRT socket yields, +TS headers, adaptation fields, and padding included, because the publisher +picks the muxing and could otherwise pad without limit. SRT and UDP +headers stay outside both. This backs +moq.pro's ingest and egress caps. + +## Plan + +Decided 2026-10-20 (maintainer): planned upstream now, because moq.pro's SRT +enforcement needs it and nobody owned it. + +- Target the shared-socket integration that drives `srt-protocol` directly + ([SRT on the shared socket](/quest/m2/one-port/srt-demux.md)). One port + retires `srt-tokio` socket handling, so do not fork that backend to add + this. +- Pull: pace the TS packets each frame muxes into, charging their media + contribution, rather than whole frames, so a frame larger than the burst + is released gradually instead of waiting forever or exceeding cap plus + burst. Decide drops only before a group starts: past cap plus burst, skip + to the next group; dropping 188-byte TS packets would leave partial + frames. As in [moq-rtc pacing](/quest/m2/rtc-pacing.md), a group (or + frame) larger than the burst is paced through to its end. + Retransmissions draw from the same bucket, and one with no credit left is + skipped, which needs the hook `srt-protocol`'s sender exposes when driven + directly. +- Publish: SRT has no receiver backpressure, so meter the received TS bytes + before demuxing and close the established session past cap plus burst. A + CBR encoder's null packets count, so an embedder sizes the cap to the + publisher's mux rate, not its media bitrate. An established SRT + shutdown carries no reason, and `Forbidden` is a handshake rejection sent + only before `accept`, so it stays for admission refusals; the rate-limit + reason goes to the embedder instead. +- Burst defaults to about one second of the cap, as on QUIC. +- Tests: a capped pull stays under cap plus burst, including a single frame + larger than the burst, which still arrives; two pulls with different + caps, each reporting every packet lost for several burst windows, stay + under their own; a publisher over the cap is closed mid-session, apart from + a `Forbidden` admission refusal; a publisher whose TS mux rate, padding + included, stays under the cap is untouched; under the cap nothing changes. + +Public API: a rate handle and a rate-limit close reason on moq-srt's server. +Wire: none. + +## Required + +- [SRT on the shared socket](/quest/m2/one-port/srt-demux.md) - the direct `srt-protocol` integration this hooks into + +## Related + +- [moq-rtc pacing](/quest/m2/rtc-pacing.md) - the same pacing and retransmit charging on WHEP +- [moq-srt wire source](/quest/m2/wire-stats/srt.md) - per-connection byte counts from the same socket diff --git a/quest/m2/stats/README.md b/quest/m2/stats/README.md index afec56dcf89..f3fb1b3b6fa 100644 --- a/quest/m2/stats/README.md +++ b/quest/m2/stats/README.md @@ -126,3 +126,5 @@ connection view and the dashboard flow. combined per-broadcast verdict reading both lives downstream - [Encoder feedback](/quest/m3/stats-encoder-feedback.md) - a Rust encoder adapts its bitrate to what its viewers report +- [Latency ledger](/quest/m2/latency-ledger.md) - adds the viewer's latency + stages to the same stats and feedback snapshots diff --git a/quest/m2/unbounded-groups/README.md b/quest/m2/unbounded-groups/README.md index b6fe76b0422..1086db9fd01 100644 --- a/quest/m2/unbounded-groups/README.md +++ b/quest/m2/unbounded-groups/README.md @@ -44,14 +44,18 @@ Decided 2026-10-16 (maintainer interview): - QUIC receive credit is released per segment, so an arbitrary-size frame never needs a matching receive allowance. The aim is to avoid over-allocating, not to bound a frame. -- Refills patch the group's state in place, as sorted runs, written only - below `committed` so they never race the tail writer. They replace - today's separate `Slot.head` copy. +- Refills patch the group's state in place, written only below + `committed` so they never race the tail writer. They replace today's + separate `Slot.head` copy. Whether the state is sorted runs or the + [append-only log](/quest/m2/perf/kio-channel-contention/log.md) (or a ring + buffer) is the [group storage benchmark](/quest/m2/group-storage-bench.md)'s + call (decided 2026-10-20, maintainer, leaning log because hot-path + performance matters more than evicting the middle). - Readers pin nothing: reads refresh recency as today. A reader that falls into an evicted run refills like a joiner. With nothing to refill - from, it gets `Error::Evicted`. Within a group, victims go front to back - until [Middle eviction](/quest/m2/unbounded-groups/middle-eviction.md) - makes the head the last victim, however stale it is. + from, it gets `Error::Evicted`. Within a group, victims go front to back. + Keeping the head for joiners while evicting the middle moved to m3 + ([Middle eviction](/quest/m3/middle-eviction.md), decided 2026-10-20). - Whole cold groups are still evicted first. A group loses part of itself only when it is the protected live edge or is being read. - Over moq-transport, an object with a hole fails the reader with `Evicted` @@ -81,5 +85,8 @@ moq-transport, with memory held near the pool size. - [Segmented frames](/quest/m2/unbounded-groups/segmented-frames.md) - large frames store as sealed segments, preallocate at most one, and release receive credit per segment - [Front eviction](/quest/m2/unbounded-groups/front-eviction.md) - groups shed whole frames from their front under pressure and refill them in place; the group caps go - [Byte holes](/quest/m2/unbounded-groups/byte-holes.md) - segments inside a frame evict and refill by `(frame, byte)` on the next lite version; the frame cap goes -- [Middle eviction](/quest/m2/unbounded-groups/middle-eviction.md) - a group keeps its head for joiners and its tail for live readers, evicting between - [JS unbounded groups](/quest/m2/unbounded-groups/js.md) - `@moq/net` drops its caps, streams huge frames, and matches front eviction + +## Related + +- [Middle eviction](/quest/m3/middle-eviction.md) - a group keeps its head for joiners and its tail for live readers, evicting between diff --git a/quest/m2/unbounded-groups/front-eviction.md b/quest/m2/unbounded-groups/front-eviction.md index 260c33b889c..803cbb5b0e8 100644 --- a/quest/m2/unbounded-groups/front-eviction.md +++ b/quest/m2/unbounded-groups/front-eviction.md @@ -15,10 +15,15 @@ until [Byte holes](/quest/m2/unbounded-groups/byte-holes.md). See the [epic](/quest/m2/unbounded-groups/README.md) for the decisions. -- The group's state holds sorted runs of frames in place of one - `VecDeque` + `offset`. Readers keep one absolute cursor. A sequential - reader should hold its run so a lookup stays O(1). The common single-run - group must cost what it does today. +- Storage comes from the + [group storage benchmark](/quest/m2/group-storage-bench.md) (decided + 2026-10-20, maintainer): sorted runs of frames in place of one + `VecDeque` + `offset`, or the + [append-only log](/quest/m2/perf/kio-channel-contention/log.md) or a ring + buffer, which the maintainer leans toward because performance matters more + than middle eviction. Readers keep one absolute cursor. A sequential + reader should hold its position so a lookup stays O(1). The common + single-run group must cost what it does today. - Pool debt evicts whole cold groups first (`pay_debt` in `model/track.rs`, as today). Front frames are shed only from a group that is protected or being read. @@ -46,5 +51,10 @@ groups. A reader that used to see `GroupTooLarge` now refills or sees ## Required +- [Group storage benchmark](/quest/m2/group-storage-bench.md) - picks the group storage this builds on - [Segmented frames](/quest/m2/unbounded-groups/segmented-frames.md) - frame storage the cache can account for without whole-frame preallocation - [Dynamic range requests](/quest/m1/subscribe-ranges/dynamic.md) - `track::Dynamic` receives frame-precise range requests for misses + +## Related + +- [Append-only log](/quest/m2/perf/kio-channel-contention/log.md) - the other proposal for `GroupState` storage, decided by the same benchmark diff --git a/quest/m2/uniffi-fork.md b/quest/m2/uniffi-fork.md index 02c57f2ae38..4e0a15b30dc 100644 --- a/quest/m2/uniffi-fork.md +++ b/quest/m2/uniffi-fork.md @@ -14,18 +14,10 @@ are unchanged by the move itself. Decided with the maintainer on 2026-10-13, after a review of how moq uses UniFFI: -- **Base**: the maintainer picks UniFFI or BoltFFI from - [the BoltFFI spike](/bench/ffi-gen/README.md), which recommends UniFFI. - The plan below assumes UniFFI. If BoltFFI is chosen, re-plan this quest - against it (the repo name, the - crate names, porting the Go, Dart, and C++/C generators to BoltFFI) along - with the quests that name UniFFI internals: - [Payload bytes](/quest/m2/ffi-bytes.md), - [Poll-native async](/quest/m2/ffi-poll-migrate.md), - [Foreign poll for hot reads](/quest/m2/ffi-poll-protocol.md), - [Zero-copy byte returns](/quest/m2/ffi-bytes-return.md), - [Kotlin calls](/quest/m2/kt-ffi-calls.md), and - [Namespaced external types](/quest/m2/uniffi-namespaces.md). +- **Base**: UniFFI. [The BoltFFI spike](/bench/ffi-gen/README.md) + recommended it. Decided later in #5838 (2026-10-17, maintainer): + UniFFI-generated moq-ffi serves the browser too, over BoltFFI, so one generator serves every + language. - **Why**: every UniFFI minor release breaks external generators (0.31 changed `module_path`, 0.32 changed the metadata encoding without bumping diff --git a/quest/m2/wire-stats/srt.md b/quest/m2/wire-stats/srt.md index b464008a1bd..d733b1dd0a2 100644 --- a/quest/m2/wire-stats/srt.md +++ b/quest/m2/wire-stats/srt.md @@ -12,16 +12,19 @@ the same definition as QUIC's `udp_tx`. - srt-protocol's `SocketStatistics` carries `tx_all_bytes` / `rx_all_bytes`, which include control packets but also a fixed 20 B IPv4 + 8 B UDP estimate per packet. Subtract that estimate per packet so the - count is SRT packet bytes. The stats arrive on the socket's - `statistics()` watch. + count is SRT packet bytes. Today they arrive on `srt-tokio`'s + `SrtSocket::statistics()` stream. - `serve_subscribe` and the publish path own the socket today and expose nothing. Update a `moq_net::transport::Wire` per connection from each statistics update and hand it to the embedder, which registers it with `moq_net::stats::Handle::wire`. -- Target the shared-socket integration that drives `srt-protocol` directly, - not `srt-tokio` socket handling. +- Read `srt-tokio`'s statistics now rather than waiting on one port + (decided 2026-10-20, maintainer, because the source needs no socket ownership + change). When [SRT on the shared socket](/quest/m2/one-port/srt-demux.md) + replaces `srt-tokio`, it reads the same `SocketStatistics` from + `srt-protocol` and keeps this source. - Test: an SRT pull's reported tx covers at least the TS payload it sent. -## Required +## Related -- [Shared SRT socket](/quest/m2/one-port/srt-demux.md) - replaces today's srt-tokio ownership with the direct srt-protocol integration this source targets +- [SRT on the shared socket](/quest/m2/one-port/srt-demux.md) - carries this source over when it replaces `srt-tokio` diff --git a/quest/m3/3129-moq-uring-write-the-webtransport-stream-header-at-open.md b/quest/m3/3129-moq-uring-write-the-webtransport-stream-header-at-open.md deleted file mode 100644 index b2f87af98e9..00000000000 --- a/quest/m3/3129-moq-uring-write-the-webtransport-stream-header-at-open.md +++ /dev/null @@ -1,40 +0,0 @@ -# [M] moq-uring: write the WebTransport stream header at open time, so finish() never owes one - -## Goal - -A web-mode `SendStream` (`rs/moq-uring/src/quic/web.rs`) has its WebTransport -header on the wire before `poll_open_uni` hands it back, so `finish()` never -owes one and the `finishing` state machine goes away. Concurrent openers, -cancellation, and flow-control backpressure on the open have settled, -documented semantics. - -## Plan - -Follow-up to #3105 (item 4). - -##### Where it stands - -A web-mode `SendStream` carries its WebTransport header as an owed `prefix` and writes it lazily on the first `poll_write`. When `finish()` finds no credit for the header, it records `finishing` and returns `Ok`, and the FIN goes out on a later `poll_closed`, or on `Drop` if credit has returned by then. - -That covers every in-tree caller, because moq-net always pairs `finish()` with `poll_closed`. What it does not cover is a direct consumer that finishes and drops in the same breath: - -```rust -send.finish()?; // no credit: the header is owed, returns Ok -drop(send); // no ingress in between, so try_write still returns 0 -``` - -There is no moment for credit to return between those two calls, so `Drop` falls through to the mapped reset and the peer sees a cancellation despite `finish()` having reported success. `SendStream` documents this, but it is still a wart. - -##### The fix - -Option 1 from #3105: queue the prefix before handing the opened stream back, so nothing is ever owed at finish time. A WebTransport stream is arguably not open until its header is on the wire, so this is also the more honest shape. - -It is not a small change: `poll_open_uni` would have to hold a half-open stream across polls while the header drains, and `Session` clones share one `Rc`. A single slot for the in-flight open would have concurrent openers fighting over it, and a queue alone is not a chosen contract. Settle the contract first: - -- How independent concurrent openers keep ownership of a half-open stream across `Pending` and cancellation (a dropped open future). -- Whether an open-before-read caller can deadlock when the header needs flow-control credit. Making `open` block on credit moves backpressure earlier, which is more correct but changes when a caller learns about it. -- Resource bounds on half-open streams. - -Present the viable ownership choices and a recommendation to the maintainer before implementing. Regression cases: concurrent openers, credit starvation, dropped open futures, and finish then drop. - -Decided 2026-10-08: the open-contract planning quest merged in here, and the quest moved to m3. No in-tree caller hits the wart and io_uring ships in no package. diff --git a/quest/m3/README.md b/quest/m3/README.md index b97158d639d..50ac67acea0 100644 --- a/quest/m3/README.md +++ b/quest/m3/README.md @@ -32,10 +32,10 @@ m4's upstream waits in here and moved m2 work with no consumer. - [#3204](/quest/m3/3204-moq-uring-register-tx-pool-buffers-for-zero-copy-sends.md) - moq-uring: register TX-pool buffers for zero-copy sends - [Stream sessions on the ring](/quest/m3/uring-tcp/README.md) - WebSocket and HTTP on the io_uring workers, gated on the ablation; no fleet asks for ring TCP yet - [#3202](/quest/m3/3202-moq-uring-use-fixed-file-slots-for-worker-udp-sockets.md) - moq-uring: fixed-file slots for worker UDP sockets, if its own benchmark measures a win -- [#3129](/quest/m3/3129-moq-uring-write-the-webtransport-stream-header-at-open.md) - moq-uring writes the WebTransport stream header at open and settles the open contract, once a caller hits it - [L4S on the backbone](/quest/m3/quic-ecn.md) - an ECT(1) option in `moq-quic`, an `ecn` config knob, and a dualpi2 measurement, if the ECN measurement says marks survive - [Careful resume on reconnect](/quest/m3/quic-careful-resume.md) - a Rust redial starts at the previous connection's rate - [Cut-through](/quest/m3/cut-through/README.md) - a bufferbloat bench decides whether a relay should forward bytes behind a QUIC stream hole; the build is re-planned if it says go +- [Middle eviction](/quest/m3/middle-eviction.md) - a group too large to keep whole keeps its head for joiners and its tail for live readers, evicting between; speculative, after the GroupState storage benchmark - [Unreal prototype](/quest/m3/unreal.md) - a UE5 module on the C++ package with exceptions disabled, rendering a subscribed broadcast to a texture - [vcpkg registry](/quest/m3/cpp-vcpkg.md) - a registry we own serves the prebuilt package to `vcpkg` manifests, once a consumer asks - [Conan remote](/quest/m3/cpp-conan.md) - a remote we own serves the same tarball to `conan install` @@ -43,7 +43,7 @@ m4's upstream waits in here and moved m2 work with no consumer. - [Routing cost domains](/quest/m3/routing-cost-domains.md) - design operator boundaries and policy without adding incomparable costs - [Relay request tokens](/quest/m3/auth-request-lease.md) - the auth server sees a request token's lease end with its request - [Request tokens beyond Rust](/quest/m3/request-token-surfaces.md) - a JS, moq-ffi, or moq-cli request-token API, built only for the surface a consumer names -- [Token on FETCH](/quest/m3/auth-token-fetch.md) - a request token authorizes TRACK_STATUS and FETCH; parked on a maintainer decision, with m1 recommended +- [Token on FETCH](/quest/m3/auth-token-fetch.md) - a request token authorizes TRACK_STATUS and FETCH, once a consumer asks - [moq-transport cluster peers](/quest/m3/ietf-cluster-peers.md) - the cluster draft carries the route layer, so moq-transport relays peer again, once a moq-transport relay operator asks - [The moq.pro mesh runs lite-07](/quest/m3/lite07-mesh.md) - the deployment that makes the exemption dead code - [Drop the hidden cluster exemption](/quest/m3/hidden-exemption.md) - relays stop forcing hidden broadcasts on cluster peers once every peer opts in on the wire diff --git a/quest/m3/auth-token-fetch.md b/quest/m3/auth-token-fetch.md index 02382870403..f04f67213f0 100644 --- a/quest/m3/auth-token-fetch.md +++ b/quest/m3/auth-token-fetch.md @@ -16,22 +16,13 @@ PUBLISH_NAMESPACE. Fetch-only demand (TRACK_STATUS, then FETCH) and cache-miss group FETCHes still need the session scope. So a token-only peer can subscribe to a track but cannot look it up or fetch its history. -Open question for the maintainer. This reverses a settled decision: -"standalone FETCH gains no request-token authority" (2026-10-12, in -[#5148](https://github.com/moq-dev/moq/pull/5148)'s quest and -[the request gate refactor](https://github.com/moq-dev/moq/pull/5782)). - -1. (recommended) Do it in m1, after the request gate. Move this file under - [In-band auth](/quest/m1/auth/README.md) and list it in that epic's - Required. The gate's dispatcher then makes this one admission path for - two more request types, not new per-handler checks. Without it, the hook's - interoperability claim stops at SUBSCRIBE, while moq-transport peers - routinely look a track up and fetch its history. -2. Leave it parked here with the other request-token surfaces until a - consumer asks. - -It is parked in m3 until the maintainer answers, so an undecided scope does -not gate the breaking release (from this plan's review). +Decided 2026-10-20 (maintainer): stays in m3 with the other request-token +surfaces until a consumer asks, because it reverses "standalone FETCH gains +no request-token authority" (2026-10-12, in +[#5148](https://github.com/moq-dev/moq/pull/5148)'s quest) and nothing needs +it before the breaking release. The request gate it would build on landed in +[#5782](https://github.com/moq-dev/moq/pull/5782): its dispatcher makes this +one admission path for two more request types, not new per-handler checks. Keep #5148's boundaries (✅ Decided 2026-10-15, maintainer): a token on a request is one kind of auth on `requests()`, the request is held to its diff --git a/quest/m3/cut-through/README.md b/quest/m3/cut-through/README.md index 65fd23af677..f5b658ff20e 100644 --- a/quest/m3/cut-through/README.md +++ b/quest/m3/cut-through/README.md @@ -76,4 +76,4 @@ stats field, which its quest documents inline. ## Related - [QoS](/quest/m2/qos/README.md) - the loss-delay counter follows its per-broadcast ingress row conventions -- [Frame slots are cached for free](https://github.com/moq-dev/moq/pull/5239) - changes how the cache charges the in-flight frames a cut-through build would hold +- [Frame slot accounting](https://github.com/moq-dev/moq/pull/5239) - landed: the cache charges frame slot growth, which covers the in-flight frames a cut-through build would hold diff --git a/quest/m2/unbounded-groups/middle-eviction.md b/quest/m3/middle-eviction.md similarity index 70% rename from quest/m2/unbounded-groups/middle-eviction.md rename to quest/m3/middle-eviction.md index ec17ac7c8af..0cf9ebee7d4 100644 --- a/quest/m2/unbounded-groups/middle-eviction.md +++ b/quest/m3/middle-eviction.md @@ -9,9 +9,14 @@ it reads that far. ## Plan -See the [epic](/quest/m2/unbounded-groups/README.md) for the decisions. - -- This reuses the runs and refill from +See the [unbounded groups epic](/quest/m2/unbounded-groups/README.md) for +the decisions. Decided 2026-10-20 (maintainer): moved to m3, because group +storage is chosen for hot-path performance +([group storage benchmark](/quest/m2/group-storage-bench.md)), not for +evicting the middle; if that picks a log or ring buffer, keeping a head +apart from the tail is new storage work here. + +- This reuses the storage and refill from [Front eviction](/quest/m2/unbounded-groups/front-eviction.md). The new part is choosing what to keep. - Decide how much head to keep from what a join needs (e.g. the first frame, diff --git a/quest/m3/safari-webtransport.md b/quest/m3/safari-webtransport.md index a095ce1f69d..b8fb1a63fa9 100644 --- a/quest/m3/safari-webtransport.md +++ b/quest/m3/safari-webtransport.md @@ -20,11 +20,10 @@ this. Safari for macOS and on Chrome, Firefox and Edge for iOS, with desktop Chrome and Firefox as controls. - Update `doc/lib/js/index.md`. -- The gate needs the Safari version, and the iOS or iPadOS WebKit version - for the iOS browsers, out of the user-agent check. If - [JS bundle trims](https://github.com/moq-dev/moq/pull/5267) has replaced bowser in the - same file by then, its check exposes that version rather than a yes/no - WebKit flag (decided 2026-10-08). +- `parseUserAgent` in `js/net/src/connection/browser.ts` already returns + the Safari and iOS versions (bowser left in + [#5267](https://github.com/moq-dev/moq/pull/5267)); replace the + unconditional `webkit || ios` refusal with a version check. ## Required @@ -33,7 +32,3 @@ this. ## Closes - [#2388](https://github.com/moq-dev/moq/issues/2388) - close this issue when the quest finishes - -## Related - -- [JS bundle trims](https://github.com/moq-dev/moq/pull/5267) - replaces bowser in the same `browser.ts` user-agent check diff --git a/quest/m3/uring-tcp/relay.md b/quest/m3/uring-tcp/relay.md index f8640db2856..c4fee2ce2f2 100644 --- a/quest/m3/uring-tcp/relay.md +++ b/quest/m3/uring-tcp/relay.md @@ -2,18 +2,22 @@ ## Goal -The relay serves its WebSocket sessions, and its HTTP routes, from the -io_uring workers rather than the shared tokio runtime. A qmux media session is -then pinned to one thread for its whole life, exactly like a QUIC one. +The relay serves its TCP listeners (WebSocket sessions, qmux over TCP, and +the HTTP routes) from the io_uring workers rather than the shared tokio +runtime. A qmux media session over TCP is then pinned to one thread for its +whole life, exactly like a QUIC one. A `unix` listener stays on the shared +tokio runtime. ## Plan -`uring::Workers::bind` (`rs/moq-relay/src/uring.rs`) ignores `listen`'s -`tcp`/`unix` listeners and points the operator at a separate `init_streams` -tokio server; it refuses only settings a worker cannot serve -(`tls.generate`, `lb_id`, several certificates). Replace the silent skip with real support: each -worker binds its own listener in the reuseport group and runs the router from -[stream](/quest/m3/uring-tcp/stream.md) on it. +`uring::Workers::bind` (`rs/moq-relay/src/uring.rs`) serves QUIC only. The +relay serves `listen`'s `tcp`/`unix` listeners beside it from the shared tokio +runtime, through `init_streams` (`rs/moq-relay/src/relay.rs`). Move the TCP +listeners onto the workers: each worker binds its own listener in the +reuseport group and runs the router from +[stream](/quest/m3/uring-tcp/stream.md) on it. A Unix socket has no reuseport +group, so a `unix` listener keeps its one shared acceptor; the integration +test covers it too. The split of work stays what the `uring.rs` module docs already describe: the worker owns everything transport-shaped, while authentication and session