Repository navigation
quest(m1): settle dynamic range requests and split out fetch as a range #6123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
362b91b
01be8b6
3f3d671
7aa15d0
4a0620b
ea5166a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,10 +3,12 @@ | |
| ## Goal | ||
|
|
||
| A publisher's `track::Dynamic` receives range requests for misses instead of | ||
| one request per group, and a track serves a subscription's ranges from one | ||
| cursor, in its order, without duplicating a group, naming every sequence in | ||
| range it won't deliver. `requested_group` and `fetch_group` are replaced, not | ||
| kept beside it. | ||
| one request per group, and every sequence in a subscription's range either | ||
| arrives or is named as dropped, so a reader never waits on a miss. | ||
| `requested_group` is replaced by `requested_fill`, not kept beside it, in every | ||
| caller and binding (the frozen moq-c ABI excepted, per the moq-c decision | ||
| below). Removing `fetch_group` is the follow-up | ||
| [fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md). | ||
|
|
||
| ## Plan | ||
|
|
||
|
|
@@ -15,26 +17,79 @@ replaced. What remains is the miss: one below the cache becomes a range request | |
| on `Dynamic` rather than a group request. Time anchors land separately; see | ||
| [time anchors](/quest/m1/subscribe-ranges/time-anchors.md). | ||
|
|
||
| - Replace `requested_group` and `fetch_group` rather than keep both. Update | ||
| every caller (relay, moq-archive, hls, the ladder) in the same PR, | ||
| including the archive consumer's time resolution from its key listing if it | ||
| has landed, plus the | ||
| published surfaces built on them, per the cross-package table: moq-ffi | ||
| (`fetch_group`, `requested_group`), moq-c (`poll_requested_group`), every | ||
| binding wrapper and its `doc/lib` page, and `moq fetch` through | ||
| `moq_relay::fetch_group` (added in the 2026-10-05 audit). | ||
| - Every sequence in range either arrives or is named as dropped, so a reader | ||
| never waits on a miss. Share the accounting with | ||
| SUBSCRIBE_DROP rather than adding a second one: | ||
| the cursor's drop scan already names only sequences inside the reader's own | ||
| ranges. | ||
| - Replace `requested_group` rather than keep both. Update every caller | ||
| (sessions, relay, moq-archive, hls, the ladder) in the same PR, including | ||
| the archive consumer's time resolution from its key listing if it has | ||
| landed, plus the published surfaces built on it, per the cross-package | ||
| table: moq-ffi (`requested_group`), every binding wrapper and its `doc/lib` | ||
| page. Until the follow-up lands, a `fetch_group` miss queues a one-group | ||
| fill. | ||
| - Every sequence in range either arrives or is named as dropped. Share the | ||
| accounting with SUBSCRIBE_DROP rather than adding a second one: the cursor's | ||
| drop scan already names only sequences inside the reader's own ranges. | ||
| - Decided in the 2026-10-10 audit: `max_delay` caps every range, including | ||
| historical FETCH replacements, and publisher `max_age` always clamps it. | ||
| A FETCH with no reader delay uses publisher max-age alone, not the live | ||
| SUBSCRIBE zero-delay default. Preserve that distinction when translating | ||
| FETCH into a range request. Sharing a range never relaxes a caller's | ||
| SUBSCRIBE zero-delay default. Sharing a range never relaxes a caller's | ||
| budget or lets one expired caller cancel another still-valid reader. | ||
| Reuse FETCH's optional per-reader budget and publisher clamp rather than | ||
| restoring a historical exemption. | ||
| - Benchmark filling over range count and span as separate axes, beside | ||
| `track_range_serving`, which covers serving. | ||
|
|
||
| Decided by the maintainer on 2026-10-20, when the quest was first started | ||
| ([moq-dev/moq#6123](https://github.com/moq-dev/moq/pull/6123)): | ||
|
|
||
| - **Split.** This quest replaces `requested_group`; removing `fetch_group` | ||
| is [fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md), so each | ||
| PR breaks one API once. That half needs head fills, which | ||
| [frame bounds](/quest/m1/subscribe-ranges/frame-bounds.md) provides. | ||
| - **Handler shape.** `Dynamic::requested_fill` yields a `track::Fill` naming | ||
| one bounded run: `start: Position`, exclusive end group, priority, the most | ||
| tolerant budget among its readers, and order. Each reader's cursor still | ||
| skips what is stale under its own `max_delay`, so sharing never relaxes it. | ||
| The handler calls `accept(sequence, info)` for each group it has and | ||
| `drop_groups(range)` for the rest. Dropping the `Fill` | ||
| names whatever is left as dropped, in the track's existing `drops`, but only | ||
| while a reader still wants it: once demand leaves, the fill is withdrawn like | ||
| a `group::Request` today and records nothing, so `drops` (durable) never | ||
| hides still-fetchable history from a later reader. | ||
| Rejected: writing through `Producer::create_group`, since a fetch-only copy | ||
| or an archive track has no live producer, and backfill must stay out of the | ||
| arrival walk. | ||
| - **What a miss is.** A sequence in a reader's window, below its cap, that | ||
| the cache doesn't cover and the feed will never deliver: below the feed's | ||
| start, or at or below its newest group and not awaited by the arrival | ||
| ledger (evicted or skipped). Nothing above that frontier is a miss, since it | ||
| arrives live, so a fill is bounded by construction. A relay records the | ||
| span a SUBSCRIBE_START skipped past its requested floor as dropped (today it | ||
| only settles the tail), or it would FETCH what upstream already gave up. | ||
| - **Fetch budget.** `Subscription::max_delay` becomes `Option<Duration>`: | ||
| `None` means publisher max-age alone, the FETCH default above. This breaks | ||
| `Subscription` in every binding, with a `MIGRATE.md` entry in each affected | ||
|
Comment on lines
+63
to
+65
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
If the generated C package has not landed, which lines 85-90 explicitly allow, the current frozen AGENTS.md reference: AGENTS.md:L59-L60 Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in ea5166a, within the maintainer's moq-c decision (frozen ABI unchanged unless moq-dev/moq#6052 lands first): the frozen crate maps (Written by Claude Opus 5.5) |
||
| package. Rejected (recommended, but overruled): a `Duration::MAX` sentinel | ||
| that the wire saturates, which avoids the break but encodes the meaning in a | ||
| magic value. | ||
| - **Sessions.** A session serves a fill as it serves a group request today: | ||
| one FETCH per group, one in flight per fill, newest first for a descending | ||
| reader, withdrawing (not dropping) the fill when demand leaves. Once a | ||
| fetched group is stale under the fill's budget, the groups older than it | ||
| are too (a timed track's starts never rewind), so the fill stops fetching | ||
| them; in either order, newer groups still arrive. They are withdrawn, not | ||
| dropped, and each reader's cursor names them stale under its own budget, | ||
| since a later reader with a wider one may still fetch them. Only staleness | ||
| under publisher max-age, which binds every reader, goes in `drops`. | ||
| [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) turns lite-07 | ||
| fills into range subscriptions, and | ||
| [moq-transport ranges](/quest/m1/subscribe-ranges/ietf.md) into one FETCH | ||
| per run. Until then, one SUBSCRIBE range can cost up to frontier minus floor | ||
| serialized upstream FETCHes, each bounded by the fill's budget and stopped | ||
| when demand leaves, the same cost per-group requests carry today; accepted | ||
| rather than blocking on those quests. | ||
| - **moq-c.** Whichever lands first decides it. If the generated C package | ||
| ([moq-dev/moq#6052](https://github.com/moq-dev/moq/pull/6052)) has landed, | ||
| `poll_requested_group` leaves the hand-written ABI with a `MIGRATE.md` entry | ||
| and the fill reaches C through the generated API. Otherwise the frozen crate | ||
| splits each fill into its per-group requests internally, keeping its ABI | ||
| unchanged, and maps `moq_subscription.max_delay_us` to `Some`: a frozen C | ||
| subscriber cannot ask for max-age alone, as today. Not a blocker. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md) - removes `fetch_group` once fills land | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| # [M] Fetch as a range | ||
|
|
||
| ## Goal | ||
|
|
||
| `fetch_group` is gone: every caller reads a past group through a one-range | ||
| subscription, which ends once its range is accounted for. That covers the | ||
| relay, moq-archive, hls, moq-ffi and every binding wrapper with its `doc/lib` | ||
| page, and `moq fetch` through `moq_relay::fetch_group`. A benchmark covers | ||
| filling over range count and span as separate axes. | ||
|
|
||
| ## Plan | ||
|
|
||
| Split out of [dynamic range requests](/quest/m1/subscribe-ranges/dynamic.md) | ||
| by the maintainer on 2026-10-20 | ||
| ([moq-dev/moq#6123](https://github.com/moq-dev/moq/pull/6123)), so each PR | ||
| breaks one API once. | ||
|
|
||
| - **Ending a bounded read** (decided 2026-10-20). `recv_group` returns `None` | ||
| once every sequence in a bounded window is delivered, dropped, stale, or | ||
| unfillable. Today it parks past the window's end on purpose, so an update | ||
| can still widen it; a reader that wants more keeps an open range instead. | ||
| `Ok(None)` before any group arrives is a miss, which the moq-ffi fetch | ||
| wrapper reports as `NotFound`; after one, it is the range's end. A one-range | ||
| read surfaces no `NotFetchable` or `Old` result, and a peer's reset with | ||
| either names the group dropped. Every fetch boundary already treats | ||
| `NotFetchable` as not found (relay HTTP 404, hls `is_cache_miss`, moq-ffi | ||
| `map_fetch_error`, moq-c `NOT_FOUND`); a fetched `Old` becomes a miss too, | ||
| where relay HTTP answers 500 and moq-c reports `OLD` today. Both stay as wire | ||
| stream errors, and `Old` still aborts a cut group mid-read. | ||
| Each affected package gets a `MIGRATE.md` entry. | ||
| - A one-range subscription for a cached live group missing the frames its | ||
| range starts at needs a head fill, which is why this requires frame bounds. | ||
| - Benchmark filling beside `track_range_serving`, which covers serving. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Dynamic range requests](/quest/m1/subscribe-ranges/dynamic.md) - `requested_fill` and subscriptions filling their misses | ||
| - [Frame bounds](/quest/m1/subscribe-ranges/frame-bounds.md) - head fills for a range starting mid-group |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a requested range ends partway through a missing group, this shape conveys only the start frame and an end group, so the handler cannot distinguish
..Position { group: 5, frame: 1 }from a request for all of group 5. The planned per-group FETCH therefore over-fetches the terminal group, allowing a peer asking for one frame to induce delivery and caching of an entire group. Make the fill end aPositionso downstream demand bounds upstream work.AGENTS.md reference: AGENTS.md:L18-L18
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Keeping the end as an exclusive group, per the maintainer's handler-shape decision. A fill populates the shared cache, not one reader: a group fetched only up to some frame would sit in the cache missing its tail, which a later reader wanting the whole group cannot tell from a complete one, and there is no tail fill to repair it (frame bounds covers head fills only). The work is also bounded the same either way: a peer asking for frame 0 of group 5 could ask for all of group 5, so the terminal group costs at most one group it was entitled to request. Each reader still sees only its own frames through the cursor's frame bounds.
(Written by Claude Opus 5.5)