diff --git a/quest/m1/README.md b/quest/m1/README.md index 60e9861639c..b8381e248ce 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -82,7 +82,7 @@ above them. - [Live largest location](/quest/m1/largest.md) - `largest()` replaces `latest()` in Rust and JS as a live stream of the track's largest location, known before the first group from the publisher's report - [Live TS clock validation](/quest/m1/ts-live-clock.md) - resolve the local PCR-rate failure with reproducible timing evidence and CI controls - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause -- [Plan: a public model](/quest/m1/model-public-surface.md) - every crate-private model item is made public or unnecessary, so a custom publisher or subscriber builds from the public model alone +- [A public model](/quest/m1/public-model/README.md) - every model item a built-in session reaches is public, unnecessary, or marked deliberately private, so a custom publisher, subscriber, or relay builds from the public model alone - [Lazy discovery](/quest/m1/lazy-discovery/README.md) - a `consume` session requests announcements only for prefixes the origin is watching, in JS then Rust; `follow`, `broadcasts`, and the `discovery` option go away - [Announce caught up](/quest/m1/announce-live.md) - Rust and JS announce consumers expose when a session's initial announce snapshot has arrived, from the counts the wire already carries where a version has them - [A spinning loop fails a sim test](/quest/m1/spinning-loops.md) - a sim test names any loop that holds a task poll too long, and each one yields through a budget diff --git a/quest/m1/ietf-extension/README.md b/quest/m1/ietf-extension/README.md index b3082738d76..a1899a0f616 100644 --- a/quest/m1/ietf-extension/README.md +++ b/quest/m1/ietf-extension/README.md @@ -59,7 +59,7 @@ cannot answer unambiguously is refused. Rewriting them would violate immutability; forwarding unchanged across those families would change their meaning. - **Public model carve-out.** These fields stay crate-private on purpose, - an exception to [a public model](/quest/m1/model-public-surface.md). A + an exception to [a public model](/quest/m1/public-model/README.md). A third-party relay built on the public model drops them, as plain lite does. - **Verification:** Rust integration tests per feature in each child, plus @@ -90,4 +90,4 @@ Work owned by this README, after the children land: ## Related - [#4926](https://github.com/moq-dev/moq/issues/4926) - the declined request for an object-level API; this carries the fields for relays only -- [Plan: a public model](/quest/m1/model-public-surface.md) - these fields are its deliberate exception +- [A public model](/quest/m1/public-model/README.md) - these fields are its deliberate exception diff --git a/quest/m1/model-public-surface.md b/quest/m1/model-public-surface.md deleted file mode 100644 index 4d687dcd05f..00000000000 --- a/quest/m1/model-public-surface.md +++ /dev/null @@ -1,52 +0,0 @@ -# [M] Plan: a public model that builds any publisher or subscriber - -## Goal - -Every crate-private or hidden item in moq-net's `model` module, and every -model capability @moq/net keeps from its public API, is either made public or -made unnecessary, so a third party can build its own publisher or subscriber (a -custom transport, relay, or wire) from the public model alone. The output is -a decision per item and the implementation quests that carry them out, not -production code. - -Non-goals: changing the wire, or exposing session internals that are not part -of the model (`ietf/`, `lite/`, connection plumbing). - -## Plan - -Decided 2026-10-11 by the maintainer, in the quest-complete triage of #5283: -track metadata is never private. #5283 makes the exact END_OF_TRACK location -public instead of a crate-private field, and this quest finds the rest. - -- Inventory what the built-in sessions reach that a third party cannot. In - Rust: the roughly 290 `pub(crate)` items under `rs/moq-net/src/model/`, - plus any `#[doc(hidden)]` model item. In JS: everything `ietf/` and `lite/` - import that the package does not export, including `src/internal.ts`, - `src/wire.ts` (`OriginProducer.receive`/`answer`, - `Broadcast.subscribe`/`requested`), and `@internal`-stripped methods such - as `Subscriber.fork` and `Group.Producer.mirror`. Group them by what a custom publisher or subscriber would need - (receiving and revealing groups, finishing at a pending sequence, demand, - fetch, stats hooks, cache overhead constants). -- For each group, decide one of: make it public with a role-based name - mirrored in Rust and JS; replace it with a smaller public handle that makes - misuse unrepresentable; or remove the need. Constants and implementation - details that only tune the built-in sessions stay private. -- Prove the result by sketching a minimal publisher, subscriber, and - forwarding relay (route ingestion and answering included) over a toy - transport against only the public API, in Rust and JS. The inventory, not - the sketch, defines scope: every inventoried item gets a decision even if - the sketch never touches it. -- Decided 2026-10-13: the moq-transport passthrough fields (object - `skip`, object and unknown track properties, END_OF_GROUP status - objects, datagram object IDs) stay crate-private on purpose. Mark them - "deliberately private: moq-transport passthrough only". A relay built on - the public model drops them, as plain moq-lite does. - Dynamic-group capability and upstream request signaling are part of this - private relay-only carve-out too. -- Write the implementation quests (likely one Rust and one JS, or one per - group) under m1, each listing its public API additions for review. - -## Related - -- [moq-transport passthrough](/quest/m1/ietf-extension/README.md) - the deliberate exception to this rule -- [#5283](https://github.com/moq-dev/moq/pull/5283) - makes the END_OF_TRACK location public, the first item under this rule diff --git a/quest/m1/public-model/README.md b/quest/m1/public-model/README.md new file mode 100644 index 00000000000..127ded010d5 --- /dev/null +++ b/quest/m1/public-model/README.md @@ -0,0 +1,108 @@ +# A public model that builds any publisher or subscriber + +## Goal + +A third party builds its own publisher, subscriber, or forwarding relay (a +custom transport, relay, or wire) from the public moq-net and @moq/net model +alone. Every model item a built-in session reaches is public, made +unnecessary, or marked deliberately private with its reason. + +Non-goals: changing the wire, or exposing session internals that are not part +of the model (`ietf/`, `lite/`, connection plumbing, wire codecs). + +## Plan + +Planned 2026-10-16 from the inventory below; the per-item decisions live in +the children's Plans, and the maintainer settled the open ones in #5717. + +Rules every child follows: + +- Track metadata is never private (decided 2026-10-11 in the #5283 triage). +- Each item is made public under a role name mirrored in Rust and JS, + replaced by a smaller public handle that makes misuse unrepresentable, or + made unnecessary. Each child lists its public API additions and removals + in its PR for review. +- Deliberately private, marked in a comment on the item so the next + inventory skips it: + - "moq-transport passthrough only" (decided 2026-10-13; subgroups were + dropped from it in #5704): object IDs and `skip`, object and unknown track properties, END_OF_GROUP status + objects, datagram object IDs, dynamic groups, and upstream request + signaling. Owned by [moq-transport passthrough](/quest/m1/ietf-extension/README.md). + A relay built on the public model drops them, as plain moq-lite does. + - "tunes the built-in sessions": constants and wake or allocation batching + that change cost, never behavior a third party can observe. + - "capability gap": a built-in behavior a third party can observe but + not reproduce yet, kept private by a recorded decision that names what + would close it (the idle copy in [ingest](/quest/m1/public-model/ingest.md)). +- Items replaced by another quest's new API wait for that quest instead of + growing a public shape it deletes: fetch and per-group requests + ([Model ranges](/quest/m1/subscribe-ranges/model.md)), largest and + liveness ([Live largest location](/quest/m1/largest.md)), announce + interest ([Rust lazy discovery](/quest/m1/lazy-discovery/rust.md)), and + the received tail ([Tail arrivals](/quest/m1/tail-arrivals.md)). + +Inventory method, to rerun before each child starts (counts drift). Rust: +replace every `pub(crate)` under `rs/moq-net/src/model/` with +`pub(in crate::model)`, then run `cargo check -p moq-net --features fuzz` +and restore each item the compiler names until it builds; repeat with +`--tests` for test-only reach. JS: list what `src/{ietf,lite,connection}/` +import from `internal.ts`, `wire.ts`, and modules `index.ts` does not +export, plus `@internal` members (`stripInternal` hides them from the +published types). + +Inventory on `main` at b4eab0db1: + +- Rust has 347 `pub(crate)` items in `model/` and no `#[doc(hidden)]` model + item. 106 are reached by session code, 16 only by session tests, and 225 + only inside `model/`. +- The 106 split into [ingest](/quest/m1/public-model/ingest.md), + [serving](/quest/m1/public-model/serve.md), + [routes and answering](/quest/m1/public-model/answer.md), and the + passthrough carve-out (listed in each child where it shares a type). +- The 16 test-only items (`group::Info::produce`, `group::Producer::is_aborted` + and `is_finished`, `track::Info::timed`, `track::Producer::new`, + `publisher_priority` and `start_sequence`, `track::Consumer::poll_group` + and `resume_position`, `FetchOutcome::rejected`, + `origin::Producer::announce` with `AnnounceProducer` and `ProduceTest`, + `origin::Consumer::get_broadcast`) stay crate-private test helpers: a third + party tests through the public API it uses. +- The 225 model-internal items stay private. +- JS publishes only `src/index.ts`, with `stripInternal` on. Sessions reach + past it through `internal.ts` (helpers and 22 `hooks`, 7 of them called + by sessions: `endTrack`, `pendingTrackProducer`, `tryRecvGroup`, + `groupChanged`, `replaceGroups`, `guardGroup`, `readGroupFrame`), the + `wire.ts` views (`Broadcast`, `OriginProducer`, `OriginConsumer`, + `Established`), `@internal` members (`Track.Subscriber.fork`, + `Group.Producer.mirror`), and modules the index does not export + (`hop.ts` values, `announced.ts` `Producer`, `consume.ts`, `tail.ts`, + `priority.ts`). Model-internal hooks (sequence namespaces, timestamps, + arrival, eviction, announcer stamping) stay private. Each session-reached + item is assigned in a child, except `wire.ts` `Established` (`consume`, + `goaway`). It is connection plumbing outside the model, a session handle + the connection layer uses, and stays private like the other excluded + plumbing. + +Work owned by this README, after the children land: + +- Rerun the inventory and confirm every `pub(crate)` left in `model/` is a + marked passthrough, tuning, capability-gap, or test-only item, so the + "pending" marks from [Narrow the model](/quest/m1/public-model/narrow.md) + are all gone. +- Prove the result with a toy transport that uses only the public API: an + integration test under `rs/moq-net/tests/` (outside the crate, so the + compiler enforces it) and a JS test importing only the package entrypoint. + Each runs a minimal publisher, subscriber, and forwarding relay with route + ingestion and answering, on mock time, in CI. + +## Required + +- [Narrow the model](/quest/m1/public-model/narrow.md) - a new model item a session reaches fails to compile until it is made public or marked, while the other children are in flight +- [Public ingest](/quest/m1/public-model/ingest.md) - a custom subscriber writes received groups, frames, starts, and ends into a track through public calls +- [Public serving](/quest/m1/public-model/serve.md) - a custom publisher reads a track's start, cursor, and group expiry for its wire through public calls +- [Public routes and answering](/quest/m1/public-model/answer.md) - a custom relay ingests routes, answers broadcast and track requests on demand, and advertises without echoing a peer's routes back +- [Public model after ranges, handover, and the JS tail](/quest/m1/public-model/deferred.md) - fetch and track-info answering, the cache peeks, JS `fork`/`mirror`, and JS `endTrack`, once the quests that reshape them land + +## Related + +- [moq-transport passthrough](/quest/m1/ietf-extension/README.md) - the deliberate exception to this rule +- [#5283](https://github.com/moq-dev/moq/pull/5283) - made the END_OF_TRACK location public, the first item under this rule diff --git a/quest/m1/public-model/answer.md b/quest/m1/public-model/answer.md new file mode 100644 index 00000000000..0dba7b24eeb --- /dev/null +++ b/quest/m1/public-model/answer.md @@ -0,0 +1,192 @@ +# [L] Public routes and answering + +## Goal + +A custom relay or session does what the built-in ones do with an origin, +through public moq-net and @moq/net calls only: ingest a peer's routes under +the peer's identity, answer broadcast and track requests on demand and close +them once unused without racing a returning reader, and advertise routes to +a peer without echoing the peer's own routes back. + +## Plan + +Rules, carve-outs, and the inventory method are in the +[epic](/quest/m1/public-model/README.md). Rerun the inventory first; the +recommendations below are the planning quest's, and the ones marked +**decision** were settled by the maintainer on 2026-10-16 (#5717). + +Rust items (`rs/moq-net/src/model/`), as they stood at b4eab0db1: + +- `origin::Hop::from_wire` (IETF cluster): public as the decode-side + constructor that accepts 0 (`Hop::UNKNOWN`), beside `Hop::new`. +- `origin::Cost::charged` (both subscribers, IETF cluster): public. + `Cost::UNKNOWN` goes; it equals `Cost::default()`. +- **Announcing identity** (`origin::Route::via`, `Route::source`, + `Route::with_via`; both subscribers, server). **Decision**: + - ✅ Replace. The identity moves onto the origin handle a session + announces through, as `origin::Producer::via(hop)` beside the public + `Producer::peer` (the name `via` stays), so every route it ingests + carries it and a `Route` can never forward it. `Route.via` and + `Route.source` stay private fields the handle sets. + - A public `Route::with_via`, which a caller could forward by mistake. +- `origin::Consumer::excluding` (both publishers, client): public; + split-horizon toward a peer, the counterpart of the identity above. +- `origin::Producer::create_source` (both subscribers): public under a role + name (for example `answer_source`). It mints a broadcast with the origin's + cache policy and stats that answers a `Dynamic` request without entering + the route table, which `broadcast::Producer::new` cannot do. +- `broadcast::SourceGuard` (both subscribers): not model. Move it to the + sessions; a third party drops or closes its own source. +- **Request validity** (`broadcast::RequestValidity`, `origin::Selection`; + both subscribers). **Decision**: + - ✅ Remove the need. The source minted for a `Dynamic` request enforces + validity itself, so no session can serve a stale instance by forgetting + a check. Revised in review (#5717): once its route is replaced or its + selection period retires, the source refuses new track requests with + `Unroutable`, but tracks it already accepted keep running and finish. + `origin::Dynamic::update` and the subscriber drain paths rely on that, + so producer writes must not start failing. The answering session learns + it separately: the source's request stream (`poll_requested_track`) + ends, so it stops answering and drops the source once its accepted + tracks are done. + - A public validity handle every answerer must poll. +- **Unused races** (`broadcast::Producer::poll_unheld`, `close_unheld`, + `group::Producer::poll_unused`, `abort_unused`, + `track::Request::reject_unused`; both subscribers). **Decision**: + - ✅ Public, as one family matching the public + `track::Producer::abort_unused(self, err) -> Result<(), Self>`: each + answerer has `demand()` for the wait and an atomic `*_unused` that + refuses when a reader returned in the gap. Rename `close_unheld` and + `poll_unheld` to the `unused` vocabulary where their semantics match + `broadcast::Demand`, or document the difference. + - Keep them private. +- `origin::Dynamic::withdrawn` (lite subscriber): public, as the explicit + peer withdrawal that differs from a session simply dropping its handle. +- `origin::interest_prefixes` (both subscribers): remove the need through + [Rust lazy discovery](/quest/m1/lazy-discovery/rust.md), which gives Rust + the origin's watched interests the way JS has them; make that public in + both languages there or here, whichever lands second. +- `origin::Consumer::discovery`, `includes_hidden` (IETF publisher): fold + into the public `with_hidden` plus a getter, unless Lazy discovery deletes + them first. `beyond` (topping up a feed): re-inventory after Lazy + discovery; public only if it survives. +- `origin::Consumer::request` (lite publisher): remove the need; it is the + body of the public `request_broadcast`. +- `origin::Producer::empty`, `origin::Consumer::empty` (sessions): public. + A scope with no patterns is not an equivalent: `scope` refuses an empty + intersection with `Unauthorized`, which stays, since failing loud on a + scope that names nothing is the right default. The drivers fill an omitted + session half with an empty producer, and answer an out-of-scope announce + interest with an empty consumer, so a custom session needs the same + handles. Restructuring the drivers to run without a half is the + alternative, but it is a bigger change for no public gain. +- `origin::Producer::stats`, `origin::Consumer::stats` (sessions): remove + the need; the caller keeps the `stats::Session` it passed to the public + `with_stats`. +- `bandwidth::Producer::weak`, `bandwidth::Weak`, `Weak::consume` (session + handle): public. A session hands out estimate consumers without the + handle itself counting as demand, which + [Rust PROBE on demand](/quest/m1/probe-on-demand-rs.md) extends to the + receive side. +- Test-only: `origin::Producer::announce`, `AnnounceProducer`, + `ProduceTest`, `origin::Consumer::get_broadcast`. + +JS items (`js/net/src/`): + +- `wire.ts` `OriginProducer.receive` (`connection/forward.ts`): mirror the + identity decision. A received route is a public `Origin.Producer.dynamic` + on a handle marked with the peer's identity, so `receive` goes and + `Origin.Request.accept` stops special-casing session-delivered broadcasts + (`Broadcast.fromSession`). +- `wire.ts` `OriginProducer.requests`, `changed`, `blind`, `answer`, + `routes`, `origin.ts` `RequestSlot`, `internal.ts` `coveringPrefixes` + (the blind-answer loop in `connection/forward.ts`). **Decision**: + - ✅ Replace with the Rust shape. A session answers through the public + `Origin.Dynamic` requests and mints a source per request, as Rust + sessions do, so the slot loop and its view go. + - Make the `wire.ts` views public. +- **Blind answers** (the same loop, raised in review of #5717). Every JS + session answers each request no route covers by subscribing on its peer: + always, and as the only path after a peer refuses or resets discovery. + [Lazy discovery](/quest/m1/lazy-discovery/README.md) keeps a plain + `request(path)` blind, and its Rust child adds the same downgrade, so + blind answering stays and needs a public shape before the loop goes. + **Decision** (settled 2026-10-16): + - ✅ A fallback answerer, `origin::Producer::fallback()` in Rust and + `Origin.Producer.fallback()` in JS. The name is still open: propose + alternatives in this quest's PR before shipping it. It returns the same + `Dynamic` request stream, scoped to the handle's allowed paths. It + receives only requests that no announced or served route covers. Once + the table routes a path, the new route takes only unresolved and new + requests for it. A request the fallback already resolved ends with an + error, since a blind answer has no epoch to prove the route serves the + same instance (never stitch). It is never + advertised, since it claims nothing. The first fallback to answer a + request wins. Dropping the handle ends the requests it resolved with an + error and never moves them to another fallback, which may be another + publisher instance (never stitch). A standby answers only requests + still unresolved, and new ones. This is what `serveRequests` in + `connection/forward.ts` does today. + - A `dynamic(prefix, route)` flagged blind and ranked below every + announced route. This reuses one call but puts an unadvertised, + claim-free entry in the route table, where selection and split-horizon + must special-case it. + Land the Rust side with or after Rust lazy discovery (Required below), + which brings blind answers to Rust. Delete the JS loop only once the + fallback answers the same requests, including after a downgrade. +- `wire.ts` `OriginProducer.expect` (`connection/reload.ts`, keeps requests + pending while a reconnect is in flight): public if Rust has the same need + across a reconnect, else private to the connection. +- `wire.ts` `OriginProducer.interests`, `attach`, `accepts`: `interests` + is public with Lazy discovery; `attach(discovery)` goes with the + `discovery` option Lazy discovery removes; `accepts` is a check on the + public `allowed()` patterns. +- `wire.ts` `OriginConsumer.advertised`, `local`, `demand`, with + `Advertised`, `Advertisements`, `sameInstance`, and `internal.ts` + `presented`, `hiddenBelow`, `scopeHead`, `scopeCaptures`, `scopeOverlaps` + (both publishers and subscribers): replace with the Rust shape. A + publisher advertises from the public `announced()` stream of an + `excluding(peer)` consumer and resolves with the public request taking an + epoch, as Rust does. The helpers stay private to `origin.ts`. Check that + the public announce event carries what `sameInstance` and a scoped + route's claim need; if it does not, that is track metadata and goes public. +- `wire.ts` `Broadcast.requested` (with its `served` flag) and + `overrideBroadcastWire` (both subscribers). Part of the JS answering + decision above: ✅ public, mirroring Rust's + `broadcast::Producer::dynamic()`: a broadcast producer hands out track + requests on demand, and a track's info and fetches are answered through + its `Track.Request` and `Track.Dynamic`. A subclass of + `Broadcast.Consumer` cannot do either today. The track-request handout + lands here. Fetch and info answering through ranges (`overrideBroadcastWire`) + moved to [the deferred quest](/quest/m1/public-model/deferred.md), which + waits on Model ranges. +- `wire.ts` `OriginConsumer.broadcasts`: registered and never read; delete. +- `hop.ts` `HopSchema`, `randomHop`, `UNKNOWN_HOP`, `MAX_HOPS`, + `Cost.zero`, `Route.default`, `Route.normalize`, `routesEqual`: public + values under `Origin`, mirroring Rust's `Hop::new`, `Hop::UNKNOWN`, + `Route::default` and `PartialEq`. `Hop` is branded, so a third party + cannot build `Route.hops` without a cast today. Today `Route.default` is + one shared mutable object (`{ hops: [], cost: 0n }`), so a caller's + `hops.push` would pollute every later route. Publish it as a factory + returning a fresh route (`Route.empty()`, like Rust's `Route::default()`), + and move the internal readers onto it, rather than exporting the shared + object. +- `announced.ts` `Announce.Producer`: remove the need; a session's route + ingestion goes through the origin. +- `wireOf`, `registerWire`, `trackOf`, `registerTrackConsumer`: stay + package-private plumbing, shrinking as the views above empty out. Delete + `wire.ts` if nothing is left. + +Tests: each public call gets a mock-time test in its language, and the +built-in sessions move onto the public calls, so their existing suites +cover the behavior. + +## Required + +- [Narrow the model before it goes public](/quest/m1/public-model/narrow.md) - its visibility rewrite and marker lint guard every model API this child edits +- [Rust lazy discovery](/quest/m1/lazy-discovery/rust.md) - settles the interest and discovery items before they become public + +## Related + +- [Rust PROBE on demand](/quest/m1/probe-on-demand-rs.md) - the receive side of the bandwidth handle +- [Public model after ranges, handover, and the JS tail](/quest/m1/public-model/deferred.md) - the track info and fetch answers, split out of this quest diff --git a/quest/m1/public-model/deferred.md b/quest/m1/public-model/deferred.md new file mode 100644 index 00000000000..42f37a217e7 --- /dev/null +++ b/quest/m1/public-model/deferred.md @@ -0,0 +1,54 @@ +# [M] Public model after ranges, handover, and the JS tail + +## Goal + +The hidden model items that other quests reshape are public, unnecessary, +or marked private once those quests land: fetch ingest and answering, the +IETF cache peeks, JS `Track.Subscriber.fork` and `Group.Producer.mirror`, +and JS `hooks.endTrack`. A custom session then answers fetches and track +info, and declares a received end in JS, through public calls only. + +## Plan + +Split from the [ingest](/quest/m1/public-model/ingest.md), +[serving](/quest/m1/public-model/serve.md), and +[answering](/quest/m1/public-model/answer.md) quests in review of #5717, +so those three do not wait on the large quests that replace these items. +Rules and the inventory method are in the +[epic](/quest/m1/public-model/README.md); rerun the inventory first, since +the prerequisites will have deleted or renamed most of these. + +- Fetch ingest (Rust `group::Request` fields `state` and `result`, + `track::TrackState` with `TrackState::abort`, `track::FetchOutcome`, + `group::Request::finish_track_at`, `track::Consumer::poll_landed`): remove + the need. Model ranges replaces per-group requests with ranges, and its + public range answer is the ingest path. +- Cache peeks (Rust `track::Consumer::peek_latest`, `peek_before`, + `peek_group`, `is_live`; IETF publisher, for Largest Object and FETCH) + and JS `Track.Subscriber.fork`: remove the need, since IETF FETCH is served + from ranges and the largest location is public. If either still has a + caller, bring the shape back to the maintainer rather than publishing + peeks. +- JS `Group.Producer.mirror` (lite subscriber fetch coalescing): remove the + need through Model ranges and JS track handover, which owns the JS fan-out + copy. +- JS track info and fetch answering (`wire.ts` `overrideBroadcastWire`): + answered through the public `Track.Request` and `Track.Dynamic` range + requests that Model ranges defines, completing the answering quest's + `Broadcast.requested` decision. +- JS `hooks.endTrack` (the JS `finish_at_pending`): public under the + received-end shape the ingest quest settles in Rust, once JS pending tail + mirrors the hold. + +Tests: as in the other children, mock time in each language, with the +built-in sessions moved onto the public calls. + +## Required + +- [Model ranges](/quest/m1/subscribe-ranges/model.md) - replaces per-group fetch requests, the peeks, and `fork` +- [Live largest location](/quest/m1/largest.md) - the public largest-location stream that replaces `peek_latest` and `is_live` on the IETF Largest Object path +- [JS track handover](/quest/m1/js-group-handover.md) - owns the JS fan-out copy behind `Group.Producer.mirror` +- [JS pending tail](/quest/m1/js-pending-tail.md) - the JS received-end hold behind `hooks.endTrack` +- [Public ingest](/quest/m1/public-model/ingest.md) - settles the received-end shape JS mirrors +- [JS ranges](/quest/m1/subscribe-ranges/js.md) - the JS `Track.Request`/`Track.Dynamic` range requests that answer track info and fetches +- [Public routes and answering](/quest/m1/public-model/answer.md) - the public `Broadcast.requested` handout this completes diff --git a/quest/m1/public-model/ingest.md b/quest/m1/public-model/ingest.md new file mode 100644 index 00000000000..4484f5ce4f4 --- /dev/null +++ b/quest/m1/public-model/ingest.md @@ -0,0 +1,188 @@ +# [L] Public ingest + +## Goal + +A custom subscriber writes what it receives into a track through public +moq-net and @moq/net calls only: groups whose header lands before their first +frame, frames streamed across polls with peer-declared sizes, the start its +peer resolves, a received end with groups still in flight, and the session +ending. The built-in lite and IETF subscribers use the same calls. + +## Plan + +Rules, carve-outs, and the inventory method are in the +[epic](/quest/m1/public-model/README.md). Rerun the inventory first; the +recommendations below are the planning quest's, and the ones marked +**decision** were settled by the maintainer on 2026-10-16 (#5717). + +Rust items (`rs/moq-net/src/model/`), as they stood at b4eab0db1: + +- **Withheld groups** (`track::Producer::receive_group`, `reveal_group`, + `group::Producer::is_started`; lite subscriber). **Decision**, reopened + in review and settled 2026-10-16: + - ~~`create_group` withholds every group until its first frame or end.~~ + It deadlocks a publisher that creates a group and waits on + `group::Producer::demand()` before encoding, since no reader can reach + a withheld group to demand it. + - A public `receive_group` that reveals itself on its first frame or end. + - A public `receive_group` and `reveal_group` pair, as today. + - ✅ Drop withholding. Hiding a group from readers was never what fixed + #4914's rejoin bug: the model's judgments must not count a frameless + group as content. Derive that from each group's own state (first frame + opened, or ended) for every group, wherever the model asks it: the live + edge, successor and staleness, the rejoin wait (`shows`), `newest` and + `largest`, and `set_idle`. Every group joins arrival order when created, + so readers and group demand reach it before its first frame. + `create_group` is the only call; `receive_group`, `reveal_group`, and + `is_started` go. The group wakes its track once, on its first frame or + end, as `reveal` does today. Verified on a scratch branch at dd95d5804 + for received groups: with arrival order immediate and the judgments + gated, the whole moq-net suite passes except the two tests that assert + the old design. Consequences: + - `largest()` stops reporting a frameless newest group as (N, 0) and + reports the last frame that exists. This touches + [Live largest location](/quest/m1/largest.md); align with it there. + - Regression guard: `rejoin_waits_for_the_answers_first_frame` + (`rs/moq-net/tests/rejoin.rs`, fails when the judgments count a + frameless group) and + `revealing_a_presented_group_wakes_a_read_it_expires` stay; + `a_withheld_group_holds_the_end` and + `a_withheld_group_that_ends_empty_releases_the_end` are rewritten to + "a frameless group does not count as content" rather than "a reader + cannot see it". + - Unverified: app-created groups on the derived path, and the JS + mirror. Check both here before landing. +- **Owned frame writer** (`group::Producer::create_frame_owned`, + `frame::ProducerOwned`; both subscribers, `coding/reader.rs`). A + poll-driven wire driver cannot hold the borrowing `frame::Producer<'a>` + across polls. **Decision**, reopened in review and settled 2026-10-16: + - One owned `frame::Producer` guarded only by the runtime + `Error::FrameOpen` refusal. + - A second, public owned type (AGENTS.md's no-twins rule argues against + it). + - ✅ Make one open frame per group a compile-time guarantee. Drop public + `Clone` from `group::Producer`; the model and sessions keep a + crate-private duplicate. The chunked writer is owned: + `group::Producer::create_frame(self, info) -> Result` + aborts the group with the error on failure (an oversized frame, a + timestamp mismatch, or a closed group are bugs or protocol errors), + `frame::Producer::finish(self) -> Result` hands it + back, and `frame.group()` gives read access (demand, info, closed) + while the frame is open. `write_frame(&mut self)` stays for one-shot + frames. The borrowing `create_frame` and `ProducerOwned` go, with the + stale "caller's promise" doc; `FrameOpen` stays only as a backstop for + crate-internal duplicates. `track::Producer` keeps `Clone`: several + writers creating groups is legitimate, and `claim_sequence` already + refuses a duplicate sequence. + - Compile probe at dd95d5804: 26 `group::Producer` clone sites, 16 + model-internal and 7 in the sessions (kept by the crate-private + duplicate), and 3 outside the crate (2 in + `rs/moq-net/benches/track.rs`, 1 moq-ffi test), all trivial. + - The sessions clone to abort a group held elsewhere and to watch + demand or closure. If a third-party session needs to abort without + owning the writer, add a non-writing abort handle rather than + `Clone`. + - Pitfall: `group::Producer` derefs to the `Copy` `group::Info`, so a + downstream `producer.clone()` silently copies the `Info` and fails + later with a confusing "no method on `Info`" error. Say so in the + changelog, or drop the `Deref` for an `info()` getter. +- The frame writer's write without a wake (today + `frame::ProducerOwned::write`) and `notify`: stay private, "tunes the + built-in sessions" (batches wakes per poll turn). +- **Allocation budget** (`frame::Budget`; both subscribers): remove the + caller-supplied argument, but keep the aggregate bound. Today one 16 MiB + budget is shared by every frame a lite or IETF subscriber session + receives, so a peer that opens many streams declaring maximum-size frames + gets lazy growth past it, not memory. A per-producer budget would lose + that, and so would the origin's cache pool, which is unbounded by + default. Recommended: the budget lives on the origin producer handle a + session ingests through (the one that already carries its stats and + identity), with the same default, and is shared by every source, track, + and group minted from that handle. A frame whose declared size does not + fit grows with the bytes received, as today. The alternative is no + up-front allocation at all (always lazy), which needs no budget but pays + a reallocation per doubling on every large frame. Measure it against the + ingest benchmarks before choosing it. +- `group::Producer::info` (IETF subscriber): public, mirroring + `track::Producer::info`. +- **Received end** (`track::Producer::finish_at_pending`, `set_tail_pending`; + both subscribers). Without it, a custom subscriber over unordered streams + ends readers at a hole. **Decision**: + - Public after [Tail arrivals](/quest/m1/tail-arrivals.md), as one call + that returns a handle the session settles when its tail is in. + - Keep it private. + - ✅ Public after Tail arrivals, with the shape left to that quest. Once + the model records arrivals per track bounded by the subscription's + grace, it may own the grace timer itself, so a plain `finish_at` on a + received track could be enough and no settle handle is needed. +- **Session end** (`track::Producer::close`, `abort_session`; both + subscribers): public. Fold `abort_session` into the public `abort` (abort + open groups too) unless the implementer finds a reader-visible difference, + and expose `close` as "end without declaring an end, so a reader resumes + from another route". +- **Resolved start** (`track::Request::resolving_start`, + `track::Producer::request_start`; both subscribers): replace with one + public way to accept a request whose start the peer resolves later. It + must be set before any reader sees the track, so it lives on + `track::Request` (for example a pre-accept `request_start(floor)`), with + the public `start_at` resolving it. +- **Idle copy** (`track::Producer::set_idle`, `set_live`, `is_live`, + `idle_newest`, `track::IDLE_LINGER`, `poll_past`; lite subscriber, + `set_idle`, `set_live`, and `poll_past` also IETF). They keep a lingering copy's cache from serving stale groups + between subscriptions. **Decision**: + - ✅ Stay private, marked as a known capability gap rather than tuning: + the staleness they prevent is observable, so "tunes the built-in + sessions" does not fit. A custom subscriber that ends its track when + its upstream subscription ends is correct, but cannot keep a lingering + copy. When designing the resolved start below, check whether idle folds + into it (re-subscribing a lingering copy is a request whose start the + peer resolves later); if it does, the gap closes there. + - Public as an idle handle whose `live(largest)` resumes readers. + - `track::Producer::poll_past` (IETF resume: a group past the answer's + Largest marks the copy live) belongs to this item and shares its fate: + private while the idle copy is, public only with it. Fix it either way: + it reads `max_sequence`, which a frameless group header advances, so it + must wait for a later group that has started (a frame landed or it + ended), the gating rule above. +- Fetch ingest (`group::Request` fields `state` and `result`, + `track::TrackState`, `track::FetchOutcome`, `finish_track_at`, + `poll_landed`): moved to [the deferred quest](/quest/m1/public-model/deferred.md), + which waits on Model ranges. +- Passthrough, stay private and marked: `group::Producer::wire`, + `set_wire`, `mark_gap`, `set_end_object`, `group::Wire`, the `object` and + `datagram` modules, `track::Producer::insert_wire_datagram`, + `frame::Header`. +- Test-only: `track::Producer::new`, `publisher_priority`, `start_sequence`, + `group::Producer::is_aborted`, `is_finished`, `group::Info::produce`, + `track::Info::timed` (see the epic). + +JS items (`js/net/src/`): mirror each Rust decision under the same name. + +- `hooks.endTrack` (`internal.ts`, IETF subscriber; the JS + `finish_at_pending`): moved to the deferred quest, which waits on JS + pending tail. +- `hooks.pendingTrackProducer` (IETF subscriber) reaches the producer behind + an unaccepted request so objects landing before SUBSCRIBE_OK are kept. + Replace it with the resolved-start shape above: the request is accepted + with its start pending, so early data has a public producer to land in. +- `Group.Producer.mirror` (`@internal`, lite subscriber fetch coalescing): + moved to the deferred quest, which waits on Model ranges and JS track + handover. +- `internal.ts` `finalSequence` (IETF subscriber) is public with the serving + quest's `Position::final_sequence`. +- `tail.ts` `Tail` and `TAIL_GRACE_MS`, `consume.ts` `BroadcastCache`: stay + private. They are session plumbing in both languages (Rust's `tail.rs` and + consume cache sit outside `model/`). + +Tests: each public call gets a mock-time test in its language, and the +built-in subscribers move onto the public calls, so their existing suites +cover the behavior. + +## Required + +- [Narrow the model before it goes public](/quest/m1/public-model/narrow.md) - its visibility rewrite and marker lint guard every model API this child edits +- [Tail arrivals](/quest/m1/tail-arrivals.md) - settles the received-end semantics before they become public + +## Related + +- [Public model after ranges, handover, and the JS tail](/quest/m1/public-model/deferred.md) - the fetch ingest, `mirror`, and `endTrack` items split out of this quest diff --git a/quest/m1/public-model/narrow.md b/quest/m1/public-model/narrow.md new file mode 100644 index 00000000000..c6931d34603 --- /dev/null +++ b/quest/m1/public-model/narrow.md @@ -0,0 +1,61 @@ +# [S] Narrow the model before it goes public + +## Goal + +Every model-internal Rust item in `rs/moq-net/src/model/` is +`pub(in crate::model)` (or `pub(super)` outside `model/mod.rs`), so a `pub(crate)` left there is +either marked deliberately private (passthrough, tuning, capability gap, +test-only) or marked pending with the +[public model](/quest/m1/public-model/README.md) child that owns it. A new +model item a session reaches then fails to compile until someone makes it +public or marks it, instead of silently growing the private surface while +the other children are in flight. + +## Plan + +Run this before the other children so their work only shrinks the marked +set. Use the epic's inventory method: narrow every `pub(crate)` in +`model/`, then restore each item `cargo check -p moq-net --features fuzz` +(and `--tests`) names, adding a one-line comment that says which child +owns it or why it stays private. + +Enforce the marks in CI, or they drift. Narrowing makes the compiler +reject unmarked reach, but a contributor can still restore `pub(crate)` +without a comment. Add a check to the lint that `just check` and CI already +run. It is an allowlist, not a list of bad spellings (`pub(in +super::super)` from `model/group.rs` reaches `crate` too). Under +`rs/moq-net/src/model/`, every restricted visibility fails without a marker +unless it is `pub(self)`, `pub(super)` outside `model/mod.rs`, or a +`pub(in ...)` path starting with `crate::model`. A marker is a +comment from a fixed vocabulary (for example `// private: passthrough`, +`tuning`, `capability gap`, `test`, or `// pending: `). A +`pending` mark must name a quest that still exists. Existence is enough +because a finished quest is deleted in the PR that completes it (`quest +guide`: "Finished quests are deleted"). The check is a grep, recommended +over reading effective visibility from rustdoc JSON, which needs nightly +and an unstable format for the same answer. It +pairs with the narrowed visibility, so it does not need the compile probe. +Each child's PR then deletes its `pending` marks as items go public, and the +check fails if one is left behind. + +JS needs the same guard, since nothing compiles reach there. Custom +transports cannot see this surface for two different reasons: +`internal.ts` and `wire.ts` are not exported from `src/index.ts`, and +`stripInternal` drops `@internal` members of exported classes from the +declarations. Neither needs a new `@internal` tag. The classification +marker is a separate comment from the shared vocabulary above, not the +`@internal` tag. Add it to: every export of `js/net/src/internal.ts` +and `wire.ts` (each `hooks` entry and view member included), and every +`@internal` member in the model modules, plus every declaration session +code already imports from a module the index does not export (for example +`lite/subscriber.ts` reaching `BroadcastCache` in `consume.ts`, `Tail` in +`tail.ts`, and `sendOrder` in `priority.ts`), so the check passes on its +first run. Seed them by running the grep below and marking each hit. Then +add an import-edge grep to the +same lint. It fails when a non-test file under `src/{ietf,lite,connection}/` +imports a name from `internal.ts`, `wire.ts`, or a module the index does not +export, or reads a `hooks.` or view member, whose declaration has no +marker. Recommended over a custom Biome (GritQL) or ESLint rule: it is +one script beside the Rust grep and needs no type information. + +No public API or wire change. diff --git a/quest/m1/public-model/serve.md b/quest/m1/public-model/serve.md new file mode 100644 index 00000000000..a25d381a5a0 --- /dev/null +++ b/quest/m1/public-model/serve.md @@ -0,0 +1,117 @@ +# [L] Public serving + +## Goal + +A custom publisher reads a track for its wire through public moq-net and +@moq/net calls only: where the feed starts, a read cursor its peer can move in +either direction, the track's end in wire terms, and when a group it is still +writing has expired. The built-in lite and IETF publishers use the same calls. + +## Plan + +Rules, carve-outs, and the inventory method are in the +[epic](/quest/m1/public-model/README.md). Rerun the inventory first; the +recommendations below are the planning quest's, and the ones marked +**decision** were settled by the maintainer on 2026-10-16 (#5717). + +Rust items (`rs/moq-net/src/model/`), as they stood at b4eab0db1: + +- `track::Subscriber::poll_start` (lite publisher): public. A publisher + reports the resolved start to its peer (SUBSCRIBE_START, SUBSCRIBE_OK). +- `track::Subscriber::poll_live` (both publishers): remove the need through + [Live largest location](/quest/m1/largest.md), whose public + `poll_largest` answers "the largest position once the cache is live". + Check that it waits out an idle copy the way `poll_live` does. +- **Cursor** (`track::Subscriber::start_at`, `raise_start_to`, `end_at`, + `subscription::Cap`; both publishers). **Decision**: + - ✅ One public method assigns the floor and the cap in either direction, + renamed from `set_groups` to `set_bounds` (JS `setBounds`) on + `track::Subscriber` and `track::Ordered`, since it sets the reader's + bounds. Each reader already hands a sequence out once (#5446), so a + SUBSCRIBE_UPDATE that widens the start re-offers only unread groups. + `start_at`, `raise_start_to`, `end_at`, and `Cap` go. Precondition: + lowering the floor re-offers unread cached groups in every mode. Today + only the resume cursor does; the arrival-order (Plain) cursor walks a + monotonic arrival index, so it must rescan below its old floor for + groups it never handed out. Rewrite the doc ("without rewinding read + progress") to say what lowering does. JS's chaining form, + `withGroups` on `Track.Subscriber` and `Track.Ordered`, becomes + `withBounds`. + - A second public `replace_groups`. +- **Bounds naming** (decided 2026-10-16 in #5717): ✅ the siblings follow the + rename, so every range setter says bounds. `group::Consumer::set_frames` + becomes `set_bounds`, and `Subscription::with_groups` becomes + `with_bounds`, with the JS mirrors (the `Subscription.groups` field + becomes `bounds`, and any group-level frame bounds setter matches). +- `group::Consumer::start_at`, `end_at` (both publishers): remove the need; + the public `set_bounds` (today `set_frames`) and `skip_to` cover them. +- `subscription::Position::final_sequence` (lite publisher): public, the + one-line wire translation of an exclusive end that both languages + already duplicate. `track::Consumer::final_sequence` (IETF publisher) goes; + callers read `poll_finished` and translate. +- **Group expiry while writing** (`group::Consumer::poll_expired`, + `poll_expired_while_pending`; both publishers): public as one + `poll_expired`. A publisher that already read frames off a group must + still abandon its stream once the subscription's max delay expires the + group. Fold the `pending` flag in, or keep it private as tuning if it only + saves a poll. +- **Cache peeks** (`track::Consumer::peek_latest`, `peek_before`, + `peek_group`, `is_live`; IETF publisher, for Largest Object and FETCH): + moved to [the deferred quest](/quest/m1/public-model/deferred.md), which + waits on Model ranges. +- Passthrough, stay private and marked: `group::Consumer::wire`, + `end_object`, `last_object`, `first_frame`, `poll_read_object`, + `poll_read_objects`, `frame::Object`, `frame::Consumer::fields`, + `track::Subscriber::poll_recv_wire_datagram`. +- Test-only: `track::Consumer::poll_group`, `resume_position`. + +JS items (`js/net/src/`): + +- `hooks.replaceGroups` (lite publisher): mirror the cursor decision, so + `Track.Subscriber.setBounds` assigns the floor and the hook goes. +- `hooks.tryRecvGroup` with its `Recv` union, and `hooks.groupChanged` + (lite publisher). The public `Subscriber.tryRecvGroup()` returns + `undefined` for both a held boundary and a drained end, so a synchronous + publisher cannot tell them apart or park. **Decision**: + - ✅ Public. Return the `Recv` union from the public method and expose + the change wake (a signal or a promise, matching the package's style). + The union deliberately has no Rust type: Rust's lite publisher builds + its private `Recv` from the public `poll_recv_group` (Pending versus + `Ready(None)`) and `poll_finished`, which a third party composes the + same way. + - Mirror Rust exactly: a distinct drained result and a synchronous end + check instead of the union, plus the wake. +- `hooks.guardGroup`, `hooks.readGroupFrame` with `ReadGroupFrame` (both + publishers): mirror the Rust group expiry as one public call. Reading a + frame the publisher `complete()`s holds the group's `closed` until it is + written; keep that behavior behind the public read rather than a second + method if it can be. +- `Track.Subscriber.fork` (`@internal`, IETF publisher fill and cache + reads): moved to the deferred quest with the cache peeks. +- `wire.ts` `Broadcast.subscribe`, `fetchGroup`: remove the need; the public + `Broadcast.Consumer.track(name)` paths cover them. +- `wire.ts` `Broadcast.largest` and `hooks.cachedLargest`: remove the need + through Live largest location. +- `wire.ts` `Broadcast.resolveTrackInfo(name, hold)` (both publishers): + public as an option on the track's info lookup (an `AbortSignal` that keeps + the track wanted until it fires), mirrored by whatever Rust's `query` + holds. Check Rust parity first; drop the hold if Rust has no equivalent + need. +- `internal.ts` `groupBounds`: stays a private helper; a third party passes + the public `Groups`. +- `priority.ts` `sendOrder`, `Priority`: stay private. They rank streams on + a `stream.ts` writer, which is wire plumbing in both languages. + +Tests: lowering the floor re-offers an unread cached group in both cursor +modes. Each public call gets a mock-time test in its language, and the +built-in publishers move onto the public calls, so their existing suites +cover the behavior. + +## Required + +- [Narrow the model before it goes public](/quest/m1/public-model/narrow.md) - its visibility rewrite and marker lint guard every model API this child edits +- [Live largest location](/quest/m1/largest.md) - its `poll_largest` replaces `poll_live` and the cached-largest reads + +## Related + +- [Public model after ranges, handover, and the JS tail](/quest/m1/public-model/deferred.md) - the cache peeks and `fork`, split out of this quest